iPhone app iPad app Android phone app Android tablet app More

FAA Flight Delays: Glitch Causing Delays Fixed

HARRY R. WEBER and JOAN LOWY   11/19/09 07:12 PM ET   AP

Airport Delays

ATLANTA — For the second time in a little more than a year, a glitch at one of the two centers that handle flight plans for the nation's air travel system set off delays and cancellations for passengers around the country.

The snarl Thursday – traced to something as simple as a single circuit board – prompted calls for more money and manpower at the Federal Aviation Administration, which has struggled without success for years to overhaul the air traffic system.

The circuit board, at an FAA center in Salt Lake City, is part of a multibillion-dollar nationwide communications network that the agency has spent years installing as part of plans to modernize air traffic control.

A government watchdog said last year that the network was over budget and plagued by outages. On a single day in 2007 alone, the failure of parts of the network was responsible for 566 flight delays.

Aviation experts are unsure whether any system that relies on the interconnectedness of computers can prevent glitches from causing havoc unless there are sufficient backup systems to handle the thousands of flight plans filed each day in the U.S.

"A good communications system should have enough redundancy that a failure shouldn't hurt it that badly," said Michael Ball, a University of Maryland professor who specializes in aviation operations research.

Hundreds of flights were canceled or delayed from Atlanta to Houston to Phoenix after the problem began about 5 a.m. The glitch was fixed about four hours later, but scattered delays were reported throughout the day. Planes in the air were never in danger.

While the delays were not as bad as those caused by a major winter storm, passengers – already frustrated by add-on fees for checking bags and the other hassles of everyday air travel – were miffed.

"I am sitting here at the airport for an additional three hours when I could have been sleeping in," said Angelo Adams of Atlanta, waiting for a flight to Philadelphia.

Sisters Sharon Walker and Sheila James were taking their elderly mother, Rosa Washington, to see their other sister in St. Louis. Their 9:30 a.m. flight from Atlanta was delayed until 4 p.m.

"We were going to be there for a four-day weekend, but now it's getting cut short," James said. "It's just not a good day."

Lawmakers in Washington pounced. Sen. Charles Schumer, D-N.Y., said the country's aviation system is "in shambles" and the FAA needs more resources to prevent similar problems in the future.

"If we don't deliver the resources, manpower and technology (to) the FAA it needs to upgrade the system, these technical glitches that cause cascading delays and chaos across the country are going to become a very regular occurrence," he said in a statement.

Sen. Byron Dorgan, D-N.D., chairman of the Senate's aviation panel, said he plans to grill FAA Administrator Randy Babbitt about the issue at a Dec. 10 hearing. Rep. Jerry Costello, D-Ill., chairman of the House aviation panel, said he has asked the transportation inspector general to investigate and report to Congress within 60 days.

Airlines, reeling from the economic slowdown, were tallying their losses from the delays and cancellations. Several refused to estimate the cost. The Air Transport Association, an airline trade group, also could not give an estimate.

It was just a year and three months ago that the FAA had to deal with a similar headache. In August 2008, a software malfunction delayed hundreds of flights around the country.

In that episode, the Northeast was hit hardest by the delays, caused by a glitch at the Hampton, Ga., facility that processes flight plans for the eastern U.S.

FAA officials and an official for the union that represents the agency's technicians said Thursday's failure prevented air traffic control computers in different regions of the country from sending each other information about flights going back and forth.

The two large computer centers – in Salt Lake City and Hampton, an Atlanta suburb – were both affected, as were 21 regional radar centers around the country.

The problem began with the failure of a single small circuit board inside a router.

Air traffic controllers were forced to type in complicated flight plans themselves because they could not be transferred automatically from computers in one region of the country to computers in another, slowing down the whole system.

The equipment that failed was part of a telecommunication network owned and operated by FAA contractor Harris Corp. of Melbourne, Fla.

The FAA is investigating the cause of the failure and why a backup didn't immediately resolve the problem, FAA spokeswoman Laura Brown said.

Harris said in a statement that it is working with FAA to resolve the issue.

The union that represents the computer technicians that work for FAA said that the network is maintained by Harris, which didn't have its own technician on site when the equipment failed.

Tom Brantley, national president of Professional Aviation Safety Specialists, said Harris initially tried to troubleshoot the problem long distance without sending a technician to the FAA center in Salt Lake. When that didn't work, a technician was dispatched, Brantley said in a statement, but the delay extended the outage.

___

Lowy reported from Washington; Associated Press Writers Marcus Franklin in New York, David Koenig in Dallas, Joshua Freed in Minneapolis, Johnny Clark and Dionne Walker in Atlanta, Laurie Kellman in Washington and Matt Barakat in Chantilly, Va., contributed to this report.

FOLLOW HUFFPOST TECH

Filed by Nick Graham  | 
 
 
  • Comments
  • 115
  • Pending Comments
  • 0
  • View FAQ
Comments are closed for this entry
View All
Favorites
Recency  | 
Popularity
Page: 1 2 3  Next ›  Last »  (3 total)
photo
ohiomark
Rush Geek
10:01 PM on 11/19/2009
To error is human, to really screw things up requires a computer.
06:24 PM on 11/19/2009
Yup, the only cure is to rehire all those peeps fired by Ronnie Raygun many years ago.
photo
kk78
Someday Texas will be blue again
05:09 PM on 11/19/2009
they better have all this cr ap fixed by thanksgiving day!! ive got places to be! ha
03:18 PM on 11/19/2009
If all their computers were made in China, then that would explain the problem.
photo
HUFFPOST SUPER USER
cavegal
The Revolution Will Not Be Privatized
03:51 PM on 11/19/2009
My understanding is that much of the infrastructure hardware is from the 70's. So I kind of doubt it would be from China.
photo
HUFFPOST SUPER USER
ColleenHarper
Actions always have unintended consequences
04:41 PM on 11/19/2009
I worked on the system for the Navy 20 years ago. We were using Data General Nova 3/12 computers which were made originally for air conditioning systems for high-rise buildings. The "computer" was made up of huge circuit boards filled with AND, OR, NAND and NOR gates that were wired together. We weren't using some nice Intel or Motorola microprocessor.

And the oldest piece, which admittedly was the heart and soul of the system and usually worked wonderfully, used a 1024 bit ferrite core memory. That was 1024 tiny little iron rings with wires strung through the holes to make the ring magnetize one way or another. That piece was designed and built in 1963! I would not be surprised to walk into a modern air traffic control center radar room and find the same 1963 hardware still in use!

Now don't get me wrong. Some of the pieces of the system worked great. Other pieces were much more bothersome. But the basics of the aircraft transponder codes, the signals sent both ways to interrogate the transponders, etc, are all probably the same today as they were 20 years ago.

Yes, the FAA needs money to upgrade all this stuff. Will "fiscal conservatives" allow a Democratic Congress and President to fix it or will they stand on their out-of-power conversion back to principles they believe when they don't have power? They after all did NOTHING while they were in power!
This user has chosen to opt out of the Badges program
photo
Y3rMawm
veni, vidi, bibi.
05:25 PM on 11/19/2009
Meaning, your cell phone probably has more computing power than the FAA's system.

I wondered how long before this would be played as a Left vs. Right issue. Keep up with that illusion...let me know how that works for you.

This is an example of the failure of highly centralized systems...of any kind. Govt, money, computing, etc.

Cut off the head, and your are toast. Of course when most systems fail you don't fall out of the sky.
HUFFPOST SUPER USER
bettyx1138
02:58 PM on 11/19/2009
"glitch" is a meaningless word. when i hear someone say it its like nails on a chalkboard.

its a "bug", meaning something malfunctions under certain conditions. imho most bugs are known by developers and engineers but are not communicated effectively to end users/clients. or they are communicated, but the ppl in charge do not allow the fixes. they'd rather pay less $ and work w/ a work around like a bandaid on a massive wound. it happens every day in the software development industry.

imho ppl say "glitch" when they don't want to accept responsibility for it. like attributing it to "an act of god". imho.
photo
HUFFPOST SUPER USER
cavegal
The Revolution Will Not Be Privatized
03:09 PM on 11/19/2009
Exactly! Further down I commented that the hardware systems themselves are from the 70's, when you run continually more complex software something is eventually going to give. And the software applications they are running may have issues too. You are definitely right about management never wanting to pay for true fixes as that requires $$. They always prefer the cheap bandaid.
This user has chosen to opt out of the Badges program
03:17 PM on 11/19/2009
Actually, FAA systems, like a number of other critical infrastructure systems should run in what are called validated environments. That is, you can't just run anything you want on them. Every piece of software, and very often even every change in configuration, requires going through a pretty extensive (and often time consuming--think 3 to 18 months) validation process before they're allowed to go live.

So these systems tend to be pretty stable, per the specifications. The problem is that the specifications are often not robust enough. I posted a little about this earlier. In theory, most of these systems are supposed to be "highly available" with full redundancy. In practice, the implementation is sloppy because HA is typically slapped in after the fact and there are gaps in more places than people like to admit.

But that's the perennial balancing act. The impulse to keep a working system known and stable versus updated with the latest bells and whistles, realizing that as a practical matter all new software has bugs.
photo
HUFFPOST SUPER USER
MintysMom
02:56 PM on 11/19/2009
Just my luck. On the day when my husband is finally flying home after what seemed like the longest business trip ever. (~mope~)
HUFFPOST SUPER USER
bettyx1138
03:00 PM on 11/19/2009
:(
04:24 PM on 11/19/2009
Hope you met him at baggage claim with some Dom and wearing a mink....just a mink
02:42 PM on 11/19/2009
reminds me of that cyberterrorism special where a hacker shut a pump off from a computer and made a generator explode.
HUFFPOST SUPER USER
bettyx1138
03:00 PM on 11/19/2009
what kind of pump?
This user has chosen to opt out of the Badges program
photo
Y3rMawm
veni, vidi, bibi.
05:29 PM on 11/19/2009
Google SCADA. It's everywhere, Oil, gas, water, food.

Security standards are a little more strict, but still pretty discretionary.
HUFFPOST SUPER USER
sarabono
Oldie but Goody
02:38 PM on 11/19/2009
Every time you buy an airline ticket there is a Tax (Called Tarriff) on that Ticket that goes to the FAA to operate and upgrade the Air Traffic Control system.

The FAA has the authority to raise that tax/ tariff if they don't have enough funds to run and upgrade the system.

This is an off budget agency. There operations are supported by tax/ tariff's they impose on the industry.

No Stimulus or TARP funds needed here to upgrade the FAA system.
HUFFPOST SUPER USER
bettyx1138
02:59 PM on 11/19/2009
yeah they need to get their shitznit together.
photo
HUFFPOST SUPER USER
cavegal
The Revolution Will Not Be Privatized
03:05 PM on 11/19/2009
My understanding is that the systems themselves have not been upgraded since the 70's. They are using ancient technology to run continually more complex software programs. The infrastructure is actually dangerous. I'd like to know where the taxes/tariffs are being spent.
HUFFPOST SUPER USER
sarabono
Oldie but Goody
02:29 PM on 11/19/2009
Maybe they should switch to Apple's OS10. Les bugs, no viruses.
HUFFPOST SUPER USER
sarabono
Oldie but Goody
02:30 PM on 11/19/2009
That's Less
This user has chosen to opt out of the Badges program
02:51 PM on 11/19/2009
And this would prevent a circuit board from physically malfunctioning how again?
HUFFPOST SUPER USER
bettyx1138
02:55 PM on 11/19/2009
if it were as simple as a circuit bd frying, it wouldnt bring down a whole system. important machines always have a backup in case of such a frying. its called redundancy.
photo
HUFFPOST SUPER USER
cavegal
The Revolution Will Not Be Privatized
02:25 PM on 11/19/2009
My understanding is that the FAA is still using systems from the 70's. We are very lucky that we have not had some major disasters. Now would be a good time to upgrade with the stimulus package.
photo
HUFFPOST SUPER USER
JoeBlough
The Horror. . .The Horror. . .
02:53 PM on 11/19/2009
We need to upgrade Iraq and Afganistan first. America comes third. Ask the GOP.
photo
HUFFPOST SUPER USER
cavegal
The Revolution Will Not Be Privatized
03:17 PM on 11/19/2009
Well look at the companies that got those contracts and look at how little has been accomplished in either of those countries in terms of infrastructure upgrades. Iraq still has electricity issues, in that they still do not have the level of service they did under Saddam. We are woefully inadequate at nation building.
This user has chosen to opt out of the Badges program
photo
Y3rMawm
veni, vidi, bibi.
05:35 PM on 11/19/2009
I would, but they are irrelevant and certainly not in charge, so who is your second choice to blame?
HUFFPOST SUPER USER
sarabono
Oldie but Goody
02:22 PM on 11/19/2009
Those Red Chinese Hackers again! Darn!
02:20 PM on 11/19/2009
I smell another surcharge in the offing.
02:01 PM on 11/19/2009
Upgrading the FAA's computers is probably a good use of stimulus or bailout money. This system is BADLY in need of an upgrade.
HUFFPOST SUPER USER
bettyx1138
03:01 PM on 11/19/2009
agreed.
01:28 PM on 11/19/2009
It was the Microsoft 7 update!
They should have went for the Ultimate version instead of the Starter version...
photo
HUFFPOST SUPER USER
cavegal
The Revolution Will Not Be Privatized
02:26 PM on 11/19/2009
LMAO. They should have gone for Macs!
HUFFPOST SUPER USER
bettyx1138
02:56 PM on 11/19/2009
yeah.
photo
HUFFPOST SUPER USER
Alexandre Laudet
12:56 PM on 11/19/2009
so good this did not happen next week when so many people are trying to make it home for Thanksgiving.
photo
HUFFPOST SUPER USER
cavegal
The Revolution Will Not Be Privatized
02:22 PM on 11/19/2009
Agreed as I'm one of those people.