There will always be another software bug; never trust human life solely on software.
-- Dr. Nancy G. Leveson, MIT
John's literary agent Sam Fleishman asks about the Boeing 777 rogue computer you can read and watch here.
Sam: "You write, 'At 0:46, you hear a siren grind: RRRR, RRRR. The Trojan Horse bug escapes, sending your plane into convulsions. The bug jerks the nose up from four to eighteen degrees at three times normal gravity that CRUSHES you back in your seat. In six seconds, the software bug is skyrocketing the plane up at two miles a minute.' Why couldn't the pilots beat this renegade computer before things got out of control?"
John: "Imagine trying to beat your laptop from crashing; you can't. Dr. Lisanne Bainbridge explains that computers crash faster than humans can react, leaving pilots an impossible task. Yale's Charles Perrow says that computers crash incomprehensibly -- meaning these Boeing 777 pilots couldn't figure out fast enough what was going wrong. Three Mile Island nuclear plant computers crashed faster than humans could stop them."
Sam: "You write, 'You don't know it, but you've entered the first stage of a rare phugoid that will take you on a wild roller-coaster ride.' Tell me more about phugoids."
John: "Picture throwing a $1.39 balsa wood glider; that's a phugoid. The plane loses speed as it zooms, the wing quits flying, the nose drops, then the diving plane builds so much speed that it climbs again. Phugoids are deadly. A Japan Air 747's pilots fought a phugoid for thirty-two minutes till the plane hit a mountain, killing 520. Left alone, a phugoid stops by itself. But these Freddie Krueger autothrottles did exactly the wrong thing, amplified the cycle and made the phugoid worse."
Sam: "This computer bug hid in this 777's software for four years? How can that be?"
John: "MIT's Nancy Leveson explains that bugs can lay dormant for years. She writes, 'Several accident reports show that the computer kept secrets from pilots. We can't blame pilots if Designers didn't give them a way to detect the Computer's software errors.'"
Sam: "Scary stuff; so this 777 near-catastrophe isn't one-of-a-kind?"
John: "Far from it; Doc Leveson writes, "Most cockpit computer crashes involve software doing exactly what it was designed to do, but the design was flawed.'"
Sam: "I don't get it; don't years of crash-free flying mean that a plane is safe?"
John: "Not at all. Doc Leveson writes, 'Most of us assume that if a computer doesn't cause a plane crash, that risk is low. The opposite is true; danger actually increases over time, particularly in software-intensive systems.'"
Sam: "Hmmm . . . makes me glad you're writing this column. So this 777's autothrottles did the exact wrong thing and made the phugoid worse? I'm looking at the video. You write, 'At 1:28, the Three Stooges attempt computercide by yanking the throttles to idle.' Is this kind of bone-head mistake unusual?"
John: "Nope. Leveson explains that in complex systems like this autopilot--autothrottle combo, accidents have resulted from interactions across perfectly-functioning computers. She writes that cockpit computers are now so complex; they introduce errors that can crash the system. And the humans involved can't figure out quickly enough to stop what's going wrong."
Sam: "That's a recipe for disaster; do other scientists agree?"
John: "Yale's Charles Perrow argues that interactive complexity across computers like this 777 autopilot--autothrottle combo have caused 'inevitable' crashes. He writes that having computers control dangerous systems like cockpits increases danger. Doc Perrow explains that teeny software bugs snowball, then damage the larger system, leaving pilots not enough time to understand or control what's happening before there's a crash."
(Phugoid image by Jeff Scott. YouTube video courtesy of Aardvark2zz)
NEXT -- THE FREDDIE KRUEGER AUTOTHROTTLES