Stupid, or delusional (part II)
Posted by Paul Cox on December 4th, 2009
On Wednesday we looked at a manager who appeared to have deluded himself that banning visitors to the facility would somehow aid in the facility’s employees not getting the flu. The reality is that at least the two things are related (we DO get flu from other people), so it’s not totally delusional thought- just stupid.
Today, though, we have delusional thought, brought to us from the FAA’s “FAA Today” publication of November 23, 2009 discussing daily operations in the agency:
Tech Glitch Shows Need for New System
NextGov reports the technology problem that delayed flights nationwide last Thursday morning highlights the necessity of an ongoing air traffic control modernization effort, House lawmakers said. During the outage, air traffic controllers were forced to manage flight plan data manually. While they could communicate with pilots and see aircraft on the radar screen, they could not access electronic systems used to manage traffic flow or the National Airspace Data Interchange Network, which processes flight plans.
According to a statement from the FAA, a software configuration problem with a router in FAA’s Telecommunications Infrastructure in Salt Lake City was to blame. The incident marked the second major outage of the system in 15 months. The FAA has a solution in NextGen, an ambitious project to replace the radar-based system with a satellite-based setup. Originally NextGen was expected to be in place by 2025, but the FAA has changed its approach and is trying to roll out new systems and features on a more immediate basis.
This news blurb is kind of like saying “There was a traffic accident on the bridge that blocked several lanes, but when we buy a new camera, things will get much better.” No, they won’t; the two things are unrelated!
The system that failed is only a couple of years old. It’s part of our old friend, the FTI system, that we’ve discussed here in the past.
Like in this article, where I pointed out that when FTI fails it’s often a single point of failure and takes down an entire system.
Or here, where I pointed out that the FTI program was costing more and taking longer than had been advertised or planned by the FAA. That article also pointed out the Department of Transportation Inspector General’s report about the failures of the FTI program, and quoted this section from it…
Finally, FAA facilities using FTI have experienced outages of primary and back-up services, which have disrupted ATC operations. In addition, we found that, FTI services are not meeting availability requirements or being restored within contractual timeframes. As we reported in 2006, FAA must ensure that FTI services avoid these problems by meeting diversity requirements (adequate separation of primary and alternate services).
Or this article, from over a year ago, where I pointed out how the FAA was lying about the FTI program being a smashing success that went “without a blip” (according to agency propaganda). And it wasn’t just me ranting- my FAA employee brothers and sisters in PASS, the technician’s union, put out a press release (jointly with NATCA) talking about the failures and problems with FTI.
There’s a key phrase in that press release that I want to quote:
A similar episode occurred last month in Memphis in which repair work could not start on an outage until the FTI contractor responsible for the equipment arrived.
Now, why do I mention all of these old articles about FTI? Because of this, from November 20th, 2009:
WASHINGTON — Flights over much of the eastern United States were delayed Thursday by a pre-dawn failure in a fairly new communications system, which led to the shutdown of a computer that accepts flight plans from the airlines and feeds them to air traffic controllers.
It was the fourth major systemwide disruption attributed to the communications system, which the Federal Aviation Administration began putting into service earlier in this decade as a way to cut costs and assure reliability.
But when it failed, at about 5 a.m. Eastern time on Thursday, the airlines had to send flight plans by fax, and the controllers typed them into their computers, sort of a hunt-and-peck exercise that was so cumbersome that many planes were delayed more than an hour…
Now, what happened? Well, the FAA’s PR team is pitching this as a problem with the equipment, because gosh darn it, it’s just too old and needs to be replaced with NextGen, which will solve all our problems.
Lies, lies, lies. When will the FAA quit lying about things that are so easily exposed?
This article has a terrific rundown of the true story. The key part in the story is right here:
…When the router went offline, only the system maintainer—government telecommunications contractor Harris—knew that the backup card was not immediately available, and that one technician, who hadn’t come to work yet that day, had the key to the storage closet where the part was kept.
So the FAA had to wait until this technician was able to come to the site in Salt Lake City to replace the faulty card inside the router, reconfigure the software, and get the communications backbone back up and running so that the nation’s air traffic could get back to normal.
Huge. Absolutely huge. What we’ve been talking about- for over two YEARS now- is exactly what happened.
If you were delayed on November 19, 2009, and you’re pissed off about it, let me tell you the real story. The real story is simple: Your delays are directly attributable to the pro-big-corporation, pro-profit, anti-government, anti-worker political agenda of ex-President George W Bush.
Plain and simple.
It was Bush’s policy over the eight years of his presidency to move as many functions out of the federal government as possible, turning them over to private, for-profit corporations. These giant multi-national megacompanies have been making bigger and bigger profits.
The system that failed was NOT the “NADIN” system. What failed was a router that was part of the FTI system- which is a relatively NEW system. Remember, the FAA was touting the “proud success” of the implementation of this news system just 18 months ago!
So this system is brand-spanking new, but it failed. And when it did, why couldn’t we get back up and running without a glitch?
Because there’s no backup. Single point of failure. And because of Bush’s political ideology, which demands that private corporations can do the job better and more efficiently and cheaper than the government and its employees can.
Bush’s ideology led to this system having no backup, to “save money’.
But here’s the thing- when the government employees, the hard-working technical folks in the FAA, ran this system, they had technicians working 24 hours a day, 7 days a week, 365 days a year. They could have noticed the problem almost immediately, and they could have gone and gotten the replacement part from the closet immediately, and they could have fixed it immediately, and the millions of people who were delayed that day (and the tens or hundreds of millions of dollars lost that day) would have been saved and wouldn’t have ever noticed anything wrong.
Now the FAA claims that NextGen is going to cure this problem.
They’re delusional. NextGen has nothing to do with this. The system that failed is only a few years old. NextGen won’t do anything more to FIX this problem; it has nothing to do with the age of the technology.
The real problem is that the employee of the private company that had the key to the storage where the spare part was kept wasn’t due in to work for a few hours.
I guess we’re lucky he wasn’t on vacation for two weeks, or we’d have really been screwed.
Look, I know the conservative folks who read this blog don’t like me pointing this out, but this is on your ideology, guys. If you don’t contract out everything you possibly can, then we wouldn’t have seen this failure; there probably would have been backup systems. And even if we HAD seen this failure, an FAA tech would have been able to fix it within minutes instead of waiting for hours.
Let me cure you of you delusions: There are some things that the government does better than private, for-profit business. PERIOD.
December 4th, 2009 at 7:15 am
We can all understand your belief that contracting is evil, and hopefully you only believe that profits for corporations doing safety work for the government are evil when they fail.
I see it as, its here and it does suck. The real failure now is in the contracting office from the FAA. They didn’t mandate a reliable backup system like the previous system had. They didn’t mandate that the TechOps be able to have access to parts when the contractor’s employee was not at work. I could go on, and you could to.
That is what your focus should be on.
Yes make a point that contracting out essential government safety services is bad, but then nail them on their failure to contract for the public safety correctly.
Point out the flaws in the contract, and offer ways for them to fix the contract including punishment for the contractor when something does not work to spec. This will help the argument to not contract everything out. p.s. Great Blog!
December 4th, 2009 at 8:39 am
What’s most interesting is that no one from FAA management has the balls to tell it like it is. FTI has a few significant oversights which bring the NAS to a screeching halt in order to save a buck on personnel. A tech had to be called in from offsite and the flying public pays through the noise. Truly a company run by sycophants. The taxpayer pays through the nose.
December 4th, 2009 at 9:15 am
I don’t think the FAA is delusional, just manipulative and dishonest and deceptive…todays core values in the FAA.
December 4th, 2009 at 10:11 am
By FAA management “logic”, because 1,000 monkeys have difficulty writing “Hamlet” on Smith Corona typewriters, we should buy each monkey a Gateway 700XL instead.
December 4th, 2009 at 10:16 am
TowerGuy Said:
December 4th, 2009 at 7:15 am
“Yes make a point that contracting out essential government safety services is bad, but then nail them on their failure to contract for the public safety correctly.
Point out the flaws in the contract, and offer ways for them to fix the contract including punishment for the contractor when something does not work to spec. This will help the argument to not contract everything out.”
———————————————————
TowerGuy,
We did just what you suggested, and were told to shut up and sit down. Now we can say, “We told you so.”, but that does nothing to help the system. FTI will continue to be the weak point in the system unless the support and redundancy issues are addressed.
Consider this.
Theoretically, one of the backbone technologies to NEXGEN is ADS-B. Though essentially satellite based, ADS-B requires the services of a telecommunications infrastructure to communicate between ground stations. In the FAA that telecommunications infrastructure is FTI. So, as in any system, ADS-B (NEXGEN) is only as good as it’s weakest link, FTI.
Where does FTI leave NEXGEN as far as reliability and, ultimately, safety?
The FAA has a HUGE problem with FTI that it must fix. New technology requires that fix, and until the FAA addresses the FTI issue, safety takes a nose dive.
Just my humble opinion.
.
December 4th, 2009 at 10:54 am
News Flash:
from: https://employees.faa.gov/news/focusfaa/story/index.cfm?newsId=59956
——————————————————-
Jerry Lavey to Retire
December 4, 2009 –
Jerry Lavey, deputy assistant administrator for corporate communications, is retiring in early January after 35 years with the FAA.
Lavey oversaw decades of change in internal communication, from typewriters to twitter. He is perhaps best known for his ARA Highlights and, most recently, AOA Highlights — an early predictor of the blogs that are proliferating today.
In his position as deputy assistant administrator, he worked to improve communication within the agency. He formed VOICE, a panel of communicators from each line of business and staff office, to address internal communication issues and establish a network that reached across stove-piped organizations.
Lavey also was involved in the development of Focus FAA, the employee streaming news site that has experienced steady and rapid growth in readership since its inception.
For more news, features and employee opinion, access Focus FAA.
December 4th, 2009 at 2:16 pm
I am a conservative guy at heart and i agree with your take 100%. There is a difference between Republican and conservative, i don’t follow any party lock-step. Most people i work with are conservative, and can’t think of 1 that agrees with what Bush and the boys did here.
Contracting may have its place somewhere, but its not in an aviation safety environment in which level headed (or lack thereof) people can’t maintain a good redundent safety environment, while doling out these contracts.
Whoever wrote this terrible contract did not listen to our knowledable tech guys about redudency. Ths reads…MONEY. Save a few bucks. Low bidder.
Contracting is good for some services…lawn, cafeteria, cleaning, etc. Safety issues…no way. ATC is 100% inherently governmetal to me. And not just because i work in this field.
Ideology is a big problem on both sides of the isle.
December 4th, 2009 at 2:49 pm
Word is the Raytheon/training contract is way over budget. Look for big cut backs is facility training and in new hires going to OKC.
December 4th, 2009 at 3:06 pm
The issue is Harris did not live up to the contract and no one will make them. It will not get better…
December 4th, 2009 at 3:36 pm
Both Lavey and Osmus are getting the boot, and it sounds like Grizzle is taking modified duty so he can go try to re-open the Snopes trial.
I ask that all friends please post an appropriate farewell message to Lavey and Osmus at the below-linked “airplanepatches” comment section:
http://www.airplanepatches.com/2009/12/faa-veterans-leaving-agency/
FAA Veterans Leaving Agency
FAA had a couple of big retirement announcements today… two of its longest-serving senior officials will be leaving the agency next month.
Lynne Osmus is stepping down Jan. 3. She is currently acting deputy administrator, and served as acting administrator for a while after Bobby Sturgell left.
Osmus joined the agency in 1979, moving up through the FAA organization to positions as director of the agency’s Europe, Africa, and Middle East office in Brussels, Belgium; chief of staff for four years with Administrators David Hinson and Jane Garvey; deputy associate administrator of the civil aviation security program; and finally, assistant administrator for security and hazardous materials since July 1, 2003.
Effective today, Osmus returns to her permanent post as assistant administrator for security and hazardous materials, where she will finish her FAA tenure.
David Grizzle, FAA general counsel, is the new acting deputy administrator until a permanent appointment is made (we’re still expecting it to be Michael Huerta). James Whitlow will serve as the acting chief counsel.
Meanwhile, Jerry Lavey, FAA’s deputy assistant administrator for corporate communications, is also retiring in early January. Lavey has served 35 years with the agency.
http://www.airplanepatches.com/2009/12/faa-veterans-leaving-agency/
December 4th, 2009 at 4:54 pm
Let’s see. Corp. lawyers who spend their careers dealing with negotiating contracts vs faa employees with little or no knowledge or experience. Who wins?
December 4th, 2009 at 5:23 pm
Everything that directly affects aviation safety should be considered inherently governmental. Unfortunately it isn’t. Look how well the contracted out FSS’s are doing! They’re all going to be closed this Feb. with the exceptions of the 3 hubs, Princeton,Mn., Raleigh,NC, & Miami,Fl. You controllers think it’s hard to contact us now, just wait! We’re also implementing a new software system that’s slow and possibly unstable to replace the inadequate Flightwinds software we have for the crappy system we’re using now. Pilot on hold times will be very long. LM really wants to drive away the pilots!
December 4th, 2009 at 7:44 pm
By FAA management “logic”, because 1,000 monkeys have difficulty writing “Hamlet” on Smith Corona typewriters, we should buy each monkey a Gateway 700XL instead.
Nope, because the 1000 monkeys would not remember their password.
December 5th, 2009 at 9:36 am
“Most people i work with are conservative, and can’t think of 1 that agrees with what Bush and the boys did here.”
Really ? How many of you “conservatives” voted for Bush twice, his anti NATCA, union busting, controller hating, Blakey-Sturgell, contracting out reign of terror ?
December 5th, 2009 at 12:32 pm
“Everything that directly affects aviation safety should be considered inherently governmental. ”
Pilots and mechanics too?
December 5th, 2009 at 4:10 pm
“Most people i work with are conservative, and can’t think of 1 that agrees with what Bush and the boys did here.”
Really ? How many of you “conservatives” voted for Bush twice, his anti NATCA, union busting, controller hating, Blakey-Sturgell, contracting out reign of terror ?
41.7% (i didn’t…nor did i vote for McCain)
“Everything that directly affects aviation safety should be considered inherently governmental. ”
Pilots and mechanics too?
Should have been clearer…refering to AF/AT standpoint on that issue.
December 5th, 2009 at 11:08 pm
There can be an argument made for Mechanics as well to be inherently governmental. It is at least important as ATC. Maybe more.
December 6th, 2009 at 12:36 am
“Hello”, you’re right, I should have been clearer, but I since the subject was AF/AT related I thought it was understood. I guess WTF was trying to open a new line of discussion or was clueless on what I meant.
January 4th, 2010 at 10:55 pm
Part of the problem is that the FAA and Harris have a joint requirement for 10 days notification for emergency maintenance; So when this hardware starts to fail, if it gives any warning, the vendor responsible then has to schedule maintenance…and wait, and wait, and wait, until it either fails, justifying immediate replacement, or their 10 day waiting period was up.
End result, routine hardware failures, that give clear warning signs, are allowed to happen, rather than be replaced in a fast, seamless, and orderly fashion.
10 days for emergency hardware maintenance? Really, FAA?