The FAA Follies

So on Monday, we were talking about the problems that ERAM is having. And then at work the other day, I heard about some issues that a test that ERAM had that wound up as an utter failure- and which illustrates something that we were talking about in the comments.

Seems they had the notion to test something about how ERAM works with the FDIO system. FDIO is the flight data system; it sends, well, flight plan data to all the “children” facilities in a given control center’s airspace.

(A jargon note… one project I’d love to have here would be a wiki for FAA stuff, so someone new coming to this blog could just click on these acronyms and have the wiki page pop up and they’d learn what ERAM and FDIO and so forth was. With a staff of a fulltimer or two, man, the things I could do with the Follies… sigh.)

So on Sunday morning, here’s the plan: They’re going to detail a trainee to fax flight plans to the facilities that would normally get them from Seattle Center, and then they can take FDIO down and run their tests.

Apparently, what they figured was they could do a two-hour-dump of the flight plans in the ETMS system (dammit, another acronym! ETMS is the system that our TMU, traffic management unit, uses to monitor supply and demand on our airports, and determines when to start delay programs due to overload) and then use the ETMS strips for the flight plan data to the outlying facilities.

Um, nope. The problem with this plan is that the ETMS strips aren’t really full flight plans. The data dump thing that ETMS does is just intended to serve as an emergency backup, and there’s no times, and the strips aren’t complete, and… well, honestly, I don’t really understand what all is missing from those strips. (Hey, I never work mids, so I never use them- which serves to illustrate how poorly trained some controllers are on the backup procedures, but that’s another post, so never mind.)

Anyway, what happens is that the facilities- like Seattle Approach and Portland Approach and so forth- have a few hundred flights and incomplete flight data. So, naturally, they start calling our TMU. Which has two people staffing it. To try and pass a couple hundred flight plans.

On top of THAT, we also distribute other types of information via FDIO, like significant weather reports. In fact, they’re called SIGMETs, and the approach controls and towers need them so they can inform pilots of weather that can be hazardous to flight.

There’s some SIGMETs out, but these facilities aren’t getting them, so they call for those, too. And they’re kind of long to read and describe and it takes a while, and remember- there’s only two people to do it. Those two people also have their REGULAR job duties to attend to, like playing Minesweeper and surfing the web. (Hey, I’m just kidding my TMU brothers and sisters… and checking to see if any of them are really reading the blog, or just tell me they love it but secretly ignore it. They’ll give me crap at work if they’re reading it!)

Anyway, what winds up happening is aircraft start getting their clearances later, and later, and then they have to start waiting for their clearances, but since the whole system is kind of dorked up we can’t really track the delays, and eventually someone with enough common sense says “screw this noise, end the ERAM test, turn the goddamn FDIO back on and let’s get back in business” and they do it and things get back to normal.

Meanwhile, airlines took delays. They might not have been “reportable”, meaning they were 14 minutes or less, but I’m told we definitely had airplanes waiting on the ground, burning fuel and costing the airlines money, while we couldn’t get our stuff straight.

For a TEST.

Now, the point here is this: We didn’t have backup. There’s systems that we use that simply MUST work, or else things are going to slow down. PERIOD. No way around it; the system, today, is built in such a way that if the main system fails, the backup cannot handle the same demand.

And this was a Sunday morning, the slowest daytime of the week, at Seattle Center, one of the slower centers in the nation for sheer traffic count. What if it’d been Chicago Center, midweek? What if the ERAM system’s link to FDIO failed then? What will we find next?

As people have pointed out in the comments, ERAM is a whole-ball-of-wax kind of deal. There is no separate program that’s a backup; ERAM is its own backup.

Imagine this for a minute: You take two completely identical computers from Dell, and you load them with exactly identical versions of Windows. You load them with exactly all the same programs, like Explorer and Firefox and Word and Minesweeper (gotta have something for the managers to do) and then you hook them into the internet in such a way that they get all of exactly the same information at exactly the same time.

They’re basically mirrors of each other. Computer A’s cooling fan dies, though, and the CPU overheats and the computer starts running all flakey and Minesweeper locks up. What do you do?

Well, you switch over to Computer B while the techs come and fix the fan, right? No problemo, you have a backup system, good move.

But a week later, something else happens. You surf one too many times to the porn page, or to the FAA Follies, and you get a nasty virus. It starts launching pop-up and pop-under windows like mad; each time you close one, another two pop up. It connects with a computer in China or Russia where some 7 year old kid who’s smarter than you and me put together has his game server, and he takes over your computer and uses it to store old episodes of Battlestar Galactica.

So your computer slows to a crawl and quits working entirely, just like it did the week before. What do you do?

You switch to Computer B, right?

But wait! You’re screwed! Computer B is a physically separate machine, sure, but remember- it’s running exactly all the same programs at the same time as A was, and it got all the same data at the same time… including the virus. And now it’s completely fuxored too.

What do you do? You turn ‘em both off and call the tech and hope that he can’t tell that you were reading a bad website when you downloaded the virus, right?

Except that when it’s ERAM, you just shut down the ATC system. And you have no backup.

THAT, folks, is why we’re so uptight about ERAM. There’s no backup. If the software dies, we’re screwed, blued, and tattooed (as they say in Enumclaw… well, someone said it once somewhere.) The hardware is separate, but the computers are running exactly the same thing, and if that software has a glitch, it’s going to take down EVERYTHING.

Now, during the transition time, we’ll have a backup system… but once we’re on ERAM full time, we’re going to dump the old system. And this is why ERAM needs to be seriously super-perfect, and why it needs to be tested like there’s no tomorrow… and why we need to start being seriously honest about how it’s screwed up right now instead of having puff pieces in the FAA’s internal communication channel.

Because if people don’t get honest about ERAM, and the big mucky-muck decision makers don’t hear the Truth, we’re liable to get this crap forced on us, and down the road we might have the complete failure scenario… for lack of honesty.