To Err is Human

When a Tesla automobile using the new “Autopilot” feature struck a semi trailer resulting in the death of the driver, the inevitable questions began: “Should Tesla disable the feature?”, “Are self-driving cars a good idea?” Of course, the driver was using the feature beyond its recommended envelope. But the incident highlights an interesting quirk of humans. We want to make our own mistakes. When human error – particularly our OWN human error – causes a problem, we are brimming with forgiveness. After all, it could happen to anybody, really. It was a momentary lapse of concentration. We were tired. The kids were acting up. The situation was just too complicated. We were unlucky. 

But when the mistake is made by someone else or – most importantly – by our technology, we are suddenly overwhelmed with righteous indignation. How could they have let that happen? What were the engineers thinking? Doesn’t anybody with half a brain DESIGN these things? 

As we engineer a world with more automation, where machines make more decisions autonomously, we are running headlong into an abyss of the entitlement of the uninformed. Unaware of the complexities and compromises inherent in any engineering solution, the public will always oversimplify the problem, propose outrageously unworkable alternative solutions, and condemn engineers whose work they do not understand. 

We are also faced with a public that is completely out of touch with the reality of large numbers and probabilities, but fed a constant stream of instant misinformation on every corner case or random event that occurs after rolling the dice tens of millions of times. Our “news” is filled with outliers, a steady stream of the extraordinary beamed 24/7 into our ordinary lives until the average citizen loses the ability to discern the difference. 

As engineers, this means simply that our work is held to much higher standards than are the humans it serves or offloads. In 2013, there were 1.25 million traffic deaths worldwide. That works out to one about every 25 seconds. The United States has a comparatively safe 12.9 fatalities per 100,000 motor vehicles. You may not want to drive in Guinea, with its 2013 death rate of 9,462 deaths per 100,000 vehicles. Finland and Sweden post world-leading safety rates with fewer than 5 deaths per 100,000.

Now bring autonomous cars onto the scene. What mortality rate will the public tolerate from our engineering marvels before they demand that humans take back the wheel? How will the companies developing technology protect themselves from potentially enormous liability claims? When a human makes a tragic mistake that results in loss of life, we can all sympathize and rationalize. When a computer makes one, there will almost always be a line of code that bears the blame. Who wrote that line? Who tested it? Who allowed it to go into production. Who hired those people? Who managed them? Who hired the managers? 

This kind of public outcry has already been experienced in much more limited venues. When Toyota replaced a simple throttle cable with a computer running software, corner cases reared their ugly heads and people died. Massive lawsuits ensued. Now, what happens when we replace not just a throttle cable, but the entire driver with software? The opportunities for failure multiply almost incomprehensibly.

The truth is, however, that even the primitive self-driving cars of today would probably be safer overall than those with human drivers. Tesla’s initial data would seem to indicate that. The number of accidents avoided by the technology must be much higher than the number of failures that lead to accidents. But there is no way to count the accidents that never happened, the tragedies avoided, and the life never lost. Those branches of the probability tree are nothing but the abstract offal of means and averages of equations with too many variables. What outrages the public are the things we know, the things that DID happen. This line of code cost these lives.

If a self-driving car is 1,000 times safer than its human-driven counterpart, will the public accept it? If 1.25 million annual traffic deaths were reduced by a factor of 1,000 with the introduction of autonomous cars, we would have 1,250 annual deaths CAUSED by autonomous cars. Public outrage would boil, even among the 1,248,750 people each year who would be unaware that they were actually SAVED by autonomous cars.

Automobile deaths in the US average about 1 per 66 million passenger miles. Commercial air travel, on the other hand, has a fatality rate of about 1 per 3.3 billion passenger miles. That makes commercial air travel about 50 times safer than riding in an automobile. Guess which one the public fears most? Why? Because they are not in control.

Modern software is the most complex thing ever created by humans. And the systems that allow automobiles to drive themselves will encompass some of the most remarkable fusions of multiple technical disciplines ever attempted, all controlled by innumerable layers of software from a wide variety of sources. Complete comprehension of such a system is far beyond the cognitive capabilities of even the most brilliant human. This will be collaborative engineering in the extreme. We will make errors – many, many errors. And our tests have no prayer of rooting out all the problems that will arise when millions and millions of automobiles start carrying families billions and billions of miles across vast stretches of inhospitable terrain year after year.

Even if we were able to produce perfect code, there will always be the threat of random mechanical and environmental events. A neutron flips a bit in an unprotected register. One of a billion transistors in a processor burns out unexpectedly. A camera lens is fogged. A connection corrodes. A Lidar signal is refracted in an unexpected way. The possibilities are endless, and we get very near the place where an infinite number of monkeys with an infinite number of typewriters produce a script to a tragedy. Or, in our earlier example, over a thousand tragedies a year. 

A single driver making an error can cause harm to a few people. A single engineer making an error could potentially harm thousands. With today’s ecosystem of re-used software and hardware, encapsulated third-party IP, APIs, subsystems, and COTS technology – our responsibility is more enormous than many of us realize. And the public is not predisposed to forgive our errors as “human.”

It’s a harsh and unforgiving world that we’re all working so diligently to improve. Be careful out there.