ERAM update time…
Posted by Paul Cox on April 9th, 2009
We’ve been watching ERAM for a while here on the Follies, as you know. For anyone new who’s tuning in… ERAM is the En Route Automation Modernization project. It’s basically like the entire central part of the nervous system for the USA’s air traffic control system; every approach control connects and uses it. It’s the brain.
ERAM is what tracks all airplanes while they’re enroute. It keeps all the flight plans, integrates all the radar information for display on enroute controllers’ scopes, everything. ERAM is being developed and is supposed to be turned on for live air traffic by the end of May.
Oh yeah- it doesn’t work.
Last night I spent a few (okay, several) minutes chatting with a couple of guys who are definitely “in the know” when it comes to ERAM. They agree that it is NOT going to be like ISSS; ERAM will happen. Eventually.
But they also agree that it’s just nowhere near ready for prime time. More bugs continue to pop up. One guy is an AF guy, and the other is an AT guy with a lot of staff experience (and he’s like a supergenius when it comes to automation, programming, etc.)
The AF guy said that ERAM keeps breaking. The developers will introduce a newer, updated patch, and some of the bugs are fixed… but new bugs appear. Right now, the latest build, when AF tries to “attach” printers to the system in the midst of their startup (or restart) sequence and print some info out, they can’t- which means they’re stuck at that point in the checklist.
In other words… not only does ERAM fail to run stable for an extended period, technically right now it can’t even START UP.
The AT guy said that part of the problem with ERAM is the architecture; it’s a ton of disparate parts making up the whole, and they’re all interrelated, so when they change (ie, fix) something in one part, it often has unintended consequences (read: new bugs) in other parts of ERAM. He thinks that they eventually WILL be able to get it going, but not for a good long while.
The warning signs have been there for a while, but the FAA accepted delivery of ERAM anyway- a political decision that we reported on here at the Follies. Lots of money in government contracts at stake, of course.
One of the best parts is that in this phase of the contract, when the FAA finds bugs and gets Lockheed Martin(the contractor for the ERAM program) to fix them, you’d think that since Lockheed developed ERAM, they would have to pay for those fixes, right?
Wrong. What are you, new? Lockheed’s been writing stuff into government contracts too long for that.
No, supposedly (I don’t have individual confirmation of this) now when the FAA finds a bug in the program that Lockheed developed and wrote, and the FAA says to Lockheed “hey, we need this fixed, because without it working we can’t use this software”, Lockheed says “sure, we’ll fix it”… and then they send a bill for that to the FAA.
In other words, the more bugs that are found in this software (that Lockheed wrote) the more money that Lockheed makes to fix the bugs!
Doesn’t this strike anyone as being fairly backwards? Maybe it’s just me.
So that’s your ERAM update, folks. Still doesn’t work, still has new bugs popping up, old bugs being fixed, many bugfixes wind up breaking more things…
They’re making progress. And I have to tell you, the people at the worker-bee level on this are excellent; there are some very sharp people working on the problems.
But the problems with the overall structure of the program, and the problems of ERAM not being ready overall… they continue.
As was expected, really.
April 9th, 2009 at 6:11 am
And meanwhile isn’t FAA steadily downsizing the HOST staff anticipating ERAM delivery? I have heard that some centers are day shift only on HOST now
April 9th, 2009 at 6:42 am
If you take a look at EVERY system at the FAA, (hell DOT) they run the same gimmick, FAA pays for the product, FAA pays for the testing of that product and then FAA pays for the bug fixes.
WHY?
Well ask yourself this. Why do so many early retirement upper
management FEDS get jobs on contracts?
If you play ball with the contracts.. you get a job.. And if you are a upper management FED, a contractor job is like hitting the lotto..
Also you have A TON of your middle management pushing this stuff to get in on time NO MATTER WHAT.. So they can show their projects go in on time, the PMO (Project Management Office) doesn’t record if a project is a failure, only if it met its “mile stone” dates.
April 9th, 2009 at 6:44 am
These steps are exactly how the FS21 computer came together for Flight Service.
Disparate programs, thrown together. The base computer software system was originally a stand alone airline dispatch system from so Eastern European airline. Then, they built on off of that, to make it a briefing and filing system for use in the United States.
When it didn’t work, they added products from ADDS (Government aviation weather site), WSI, then OPUS for NOTAMS, and many, many other programs.
The whole system was a failure, and only eventually worked (just enough) because the workers made it work with workarounds.
It was truly amazing when Lockheed won the AFSS contract how slick the little CDROMS they had describing this system were. I remember many features that were promised, but never delivered. Our system can’t even map TFRs.
Weather advisories are often a mess, and the thing still drops flight plans (which we got blamed for).
Tens of thousands of bugs. And now they are working on the replacement system, also delayed now, and one that is supposedly like the OASIS system the FAA used to have, and would have been included in the FAA MEO bid.
Oh well, Lockheed still gets its money. Even if briefing has fallen 50, 60, or 70% or whatever now.
April 9th, 2009 at 6:58 am
Sounds to me like a car mechanic. You take your car in to get the air filter fixed, they fix that and “accidentally” break the fuel pump. So now you have to take the car back and get them to fix that, only they “accidentally” break the fuel line…..
You get the picture…
April 9th, 2009 at 7:18 am
Snakey, from her pimp’s perch over at AIA, and like most Bush disciples, still defending the “legacy”, says ERAM was on time and on target.
She can’t help herself. She has to open that ugly piehole of hers every time a Nextgen story pops up somewhere in the press.
Marion C. Blakey, President/CEO Aerospace Industries Assn.
Arlington, Va.
“Adrian Schofield’s article on the en route automation modernization (ERAM) system was an excellent update on an important NextGen program (AW&ST Mar. 16, p. 44). However, I disagree with his description of ERAM as an on-schedule and on-budget program that is “something of a rarity for a major FAA modernization project.”
True, FAA’s legacy acquisition programs were often over budget and late. However, that is not the case today. Early this year, the Government Accountability Office (GAO) removed the FAA’s air traffic modernization efforts from its high-risk list.
As Bobby Sturgell, FAA’s former acting administrator, wrote in his commentary (AW&ST Feb. 16, p. 41), GAO’s report confirms that “FAA has reversed program management problems from previous decades” and that this “change was possible because of long-term leadership, the support of the industry and government, and the combination of private- and public-sector expertise.”
See, all is well at FAA, just ask Snakey and that cashiered punk Stoogell. Oh, I wonder what the C stands for in Blakey’s name ?
April 9th, 2009 at 7:34 am
Paul,you wrote:
One of the best parts is that in this phase of the contract, when the FAA finds bugs and gets Lockheed Martin(the contractor for the ERAM program) to fix them, you’d think that since Lockheed developed ERAM, they would have to pay for those fixes, right?
Wrong. What are you, new? Lockheed’s been writing stuff into government contracts too long for that.
—————————————————–
No surprise there, when LM specialists attend extra-ordinary events like airshows, seminars, balloon events, LM sends the extra bill to the FAA……travel, to/from, lodging, equipment, and time at the event. This adds up to thousands of dollars over the cost of the original contract per event. When it was federal, there was no additional cost to the taxpayer other than normal salary. Specialists were already stationed at the locations, so travel was null. Hell, you got volunteers to participate.
April 9th, 2009 at 8:12 am
Gee! That’s how we developed the 9020 and Host so what’s the problem?
April 9th, 2009 at 10:55 am
Paul wrote:
“Last night I spent a few (okay, several) minutes chatting with a couple of guys who are definitely “in the know” when it comes to ERAM. They agree that it is NOT going to be like ISSS; ERAM will happen. Eventually.”
I agree, ERAM will happen. Ever since the AAS debacle, the FAA has found that, politically, it cannot allow a procurement to fail. So, every project conceived and contracted for by the FAA WILL succeed. Failure is not an option. No apologies or excuses, just the FAA’s political reality.
Oh, and the contracts that allow LM, Raytheon, and others to bill the way they do are called Cost Plus Contracts. To my mind, they need to disappear. To much room for abuse. Do it more like the military used to do, and make a tight contract document, then hold the company to the letter of the contract unless significant reasons are presented for change.
April 9th, 2009 at 11:33 am
It sounds to me like the FAA hired the ERAM ‘off the street’ with no experience and then sits around scratching it’s head trying to figure out why it doesn’t work.
This is what you get when you completely shut out the only ones who REALLY know how ERAM should operate: the controllers!
April 9th, 2009 at 3:44 pm
When LM “developed” FS21, there was no actual FSS specialist input. Why should the controllers think it would be any different with ERAM??? The fix was in with the FSS privatization and it seems the fix was in for the ERAM contract. Gotta love that corporate welfare!!!
April 9th, 2009 at 6:32 pm
Ex-FAA’er Monte Belger, Jane Garvey’s FAA assistant, is a big player now with LM. I think he had zero AT experience before his top exec role in ATC. I wonder if he asks Jane for advice these days.
April 11th, 2009 at 6:59 am
No surprises.
I attended a VSCS Operational Testing and Evaluation in Atlantic City, maybe 15 years ago. As our team went through our programmed scenarios, there were several items that didn’t seem to behave the way they should. Sorry, I can’t recall them right now. Too many years of “workarounds” have cemented them into place. I do recall, though, that when I brought up these shortcomings in the debrief, I was informed that such was the way the system had been designed. The bug was acknowledged, but the spec was locked down, and any coding to change it would be at the vendee’s expense, and there was no funding in the project for changes. And so it goes…..