One suspects that AOPA’s resistance to restarting NAOMS, a resistance apparently shared by the FAA, may stem from apprehension. If the surveys already conducted reflect great discomfort about safety among airline pilots, a comprehensive approach to compile data from other segments of the aviation system could well point to other problems of which the FAA is presently oblivious.
Indeed, it would seem that NAOMS would complement a little known FAA initiative to delve deeply into their data-mountain in an attempt to identify accident precursors.
In an era where accident investigators have scant opportunity to “kick tin” and forensically examine crash aftermaths, the FAA has belatedly discovered its latent ability to identify safety concerns from incident trends. Above and beyond the furor over the NAOMS air safety survey, the FAA is becoming aware of the vast amount of data available from FDM (Flight Data Monitoring, also known as FOQA, or Flight Operations Quality Assurance).
FDM data is accrued by downloading flight data recorders and QAR’s (Quick Access Recorders). Many safety experts believe that the data from these sources contain pointers to potential accidents and they are confident that high-speed computers and the right software can data-mine these sources for accident precursor trends. The FDM data initiative has grown so extensively in recent years that the FAA has launched its own effort to mine the information in search of precursors. Seven carriers have signed on to the initiative, which began last October. More carriers are expected to join. Jay Pardee, director of the FAA’s office of aviation safety, said, “We can use this tool to find the beginnings of a new safety threat that we don’t even perceive.”
That’s also the benefit of NAOMS. It can take the human element, not just bits and bytes gathered from flight data recorders, to identify threats to safety. The NAOMS responses gathered thus far do just that.
| Figure A |
| ‘We Are Being Criticized For What We Were Asked To Do’ NASA press conference 31 December 2007 on safety survey (extracts):Moderator J.D. Harrington: The goal was to release as much survey response information as possible before the end of the calendar year, but only to release information that does not contain confidential commercial information or information that could compromise the anonymity of individual pilots. …Reporter Mark Carreau, Houston Chronicle: I think this is for [NASA Administrator Michael] Griffin: I think in your testimony to Congress, you characterized this data as not as valid as you would prefer to have for a normal NASA report. I wonder if you might tell us if that is still your thought and what you think is missing that would be critical?Administrator Griffin: Let’s see. First of all, that is still my thought. One can’t retroactively peer-review a set of scientific or engineering technical work. The fundamental concern that I had at the time of my testimony and still have is that this research work was not properly peer-reviewed at its inception, and the data that was extracted from the survey was not properly validated at its conclusion … it was reported, for example, that the survey unearthed approximately four times as many engine failures as the FAA believes it has cognizance of. Engine failures, as I am sure you know, are a very high-profile item … if someone comes in and says we are seeing four times as many engine failures as are otherwise being reported, it calls into question the reporting mechanism rather than the underlying rate of engine failure which we believe we understand. … Reporter Matt Hosford, ABC News: Sir, what is the use [of this information you have now put up on the web] in trying to get a sense on aviation safety at this point? Administrator Griffin: Well, I don’t know. That would be up for you to determine. … Reporter Karl Hille, Baltimore Examiner: What does the administrator think this tells people about the safety of their personal experience in the air? Administrator Griffin: [The FAA has] over 150 different programs to provide survey data from pilots, mechanics, flight attendants, dispatchers that is part of a broader process in their Aviation Safety Action Program [ASAP] to capture the same kind of input that NAOMS was intended to capture. … What the public should understand is that they have approximately the same risk of dying from a lightning strike as they do of dying from an air transport accident in the United States, which means to say that this is one of the safest forms of travel that human beings have ever invented, and that no one should think otherwise, … Reporter Rita Beamish, Associated Press: We have only now been able to look at this data … and it is posted in a PDF format. I am aware that it is available elsewhere in Excel … format, which would be much less cumbersome to use and would provide an opportunity to compile and analyze the data. It just appears to me that you made an effort to obfuscate results. … Administrator Griffin: Our standard format for data release is PDF [so] that the data cannot be altered by others without our knowledge and still claim that it is NASA data. … Reporter Kate Tobin, CNN: Mr. Griffin, it sounds like you are not very proud of this study. It sounds really like a lot of taxpayers’ money has been wasted here. … Administrator Griffin: We consider that the study was not properly organized and not properly reviewed … and it makes the results very difficult to interpret and to use. … Reporter Tom Costello, NBC News: Just to reiterate, you were unable to draw any conclusions at all from this study, and am I correct in hearing you say this is simply raw data; you did not interpret it or find anything to interpret? … Administrator Griffin: It was always the purpose of this study to transition it to the larger aviation safety community. I would remind everyone again that NASA does not have in law, does not have a responsibility for operational aviation safety. We do conduct research; so this is one element of such research. We intended the data to be transitioned, again, to the larger safety community. … When NASA was conducting these surveys to try to obtain what many folks would characterize as ‘hangar talk’ among pilots and aviation professionals regarding the kinds of experience that might be risk precursors, the FAA established an Aviation Safety Action Program, or ASAP, which captures safety input data from pilots and all other kinds of aviation professionals, and they do it within 24 hours of the incident, and they offer immunity from prosecution. Those are our key points. They have more than 150 of these programs throughout the aviation industry. So my take on all this, Tom, is that the current practice is well beyond what was sought in the NAOMS project which, in fact, is why we brought it to an end. … Reporter Carter Yang, CBS Evening News: Given what you have said about the survey, does it cast any doubt on the safety of the aviation system in your mind or the statistics gathered on these same types of incidents by the FAA? Administrator Griffin: No, it doesn’t. Not in my mind. … Reporter Matt Wald, New York Times: Jon Krosnick was a consultant on this project, and he testified next to you in, I guess, the end of October. … Administrator Griffin: I don’t have his response in front of me … so … I am not going to be able to comment on your question today. Reporter Alan Levin, USA Today: The aviation safety experts that I have spoken with say that even though they are aware that there might be methodological problems with this, that there is great value to the unusual way this was conducted; in other words, going out and doing extensive interviews with people. Whereas, these other programs you mentioned, ASAP and whatnot, require the pilots, etc., to take the first step and go out and report, self-report, and even with anonymity, you know, there is going to be a self-selection process there. I gather from everything you’ve said here that you don’t see any value to this at all, and given the fact that you’ve … terminated the funding and all that, is there no value? If there is, what value is there here? Administrator Griffin: I think we have to allow the FAA, the Air Line Pilots Association, numerous other groups with an operational interest in aviation safety to determine what its value is. Reported Richard Harris, National Public Radio: You mentioned at the top, Dr. Griffin, that you waited about three weeks after the congressional testimony to get Bryan O’Connor [NASA chief of safety and mission assurance] and others actually on this task, and now you have released it on probably the slowest imaginable news day of the year. I wonder if you would like to comment about the timing of the release of this data. Administrator Griffin: Well, we didn’t wait three weeks, as your question implies. I will have to say for the record, I dislike the tone of your question. … Reporter Jim Swickard, Business & Commercial Aviation Magazine: Given the fact that the responsibility for aircraft safety always, without fail, rests with the pilot in command, whether it is a Piper Cub or your 747, it would seem to me that all perceptions of danger would start at the anecdotal or perceived level by the pilot in command, and therefore, what is wrong with that kind of data? Does hindsight by the FAA trump pilot’s judgment in the cockpit? That is a question that to me is easily answered … Can you comment on that? Administrator Griffin: Well, I can’t comment on that. … Reporter Trevor Thompson, Associated Press: We are just looking at the data now, and it is very difficult to ask some questions without having had a chance to read and review it ahead of time, but it looks like you have disaggregated from the main dataset, data about the flight hours and flight legs flown by pilots, and it is my understanding that information is essential to link back to the record to make appropriate estimates of the rate. Can you tell me why you did that and whether we can still make sense of these data because of that disaggregation? Administrator Griffin: Bryan, would you comment on that? NASA’s O’Connor: Yes. Again, what we were trying to do is make sure that we didn’t have information from these surveys that were lining up in a way that would threaten the anonymity of the pilot. But we tried to keep the information but disaggregate it, so you could see what the total hours and the total legs are, and we did that so you would not be coming up with a fingerprint for that pilot, if you follow my analogy. Moderator Harrington: Thanks, Bryan. … Reporter Susanna Ray, Bloomberg News: I am a little bit confused after all of this. If there never was an intent to analyze the results, then I really don’t understand why the study was undertaken. Administrator Griffin: What I said was there was not an intent by NASA to analyze the data; that NASA was funding the development of a potentially useful survey methodology … Reporter Laura Marquez, ABC News: I am still very confused … Why would you just dump 16,000 pages of responses without actually having some sort of ‘here is what we have found in regards to the original premise, which was to try and find a way to reduce accidents before they happen.’ Administrator Griffin: Well, we are being criticized for doing what we were asked to do, which was to release the raw data. So you are now characterizing the release of 16,000 pages of data as something that we shouldn’t have done, and yet that is what we were asked to do. Reporter Rita Beamish, Associated Press: There are documents, planning documents from NASA that were used in presentations during this program, as well as testimony from people before Congress who worked on the survey, that said it was intended as an ongoing tracking tool, and in fact, it was to include interviews not only with pilots, but with air traffic controllers, from flight attendants and mechanics, and that this would become something like NASA does with ASRS, an ongoing project that could … be used by the greater aviation community to help enhance safety. … Administrator Griffin: I did not say the survey results had no value. I said that the survey methodology had not been peer reviewed. … Reporter Beamish: Well, you said that the flying public would – there wouldn’t be nothing (sic) useful to yourself or the flying public. Administrator Griffin: I said that I did not – having looked at a snapshot of the data – I did not see anything that as a member of the flying public would affect me one way or the other. … Reporter Beamish: The thousands – Administrator Griffin: Rita, you need to stop. … Reporter Miles O’Brien, CNN: As I go through what I have seen here, I do see comments, a lot of talk about fatigue, a lot of talk about crowded airspace. Administrator Griffin: No. … Reporter Matt Wald, New York Times: Dr. Griffin, you have twice said that the National Academy of Sciences concluded that NAOMS should be terminated, but looking at their actual recommendations, they said, quote, ‘NASA should combine the National Aviation Monitoring Service methodology and resources with the Aviation Reporting System to identify aviation safety trends.’ It is also clear from their report that they envisioned the process being extended to cabin crews, maintenance personnel, and other categories that the initial research didn’t get to. Administrator Griffin: I don’t have those recommendations in front of me, and that is certainly not the verbiage that I have. … Reporter Alan Levin, USA Today: You know, in looking at these results very briefly, one of the things that pilots say is that they don’t trust the ASAP program. … They say that it is really not anonymous, that they sometimes fear they get punished as a result of that. Administrator Griffin: That calls for a conclusion that I am not professionally qualified to make. … I would remind you that we have an incredibly safe system today. … [End of NASA media briefing] |
| Figure B |
| The Pilots Speak Among the pilot responses in the heavily redacted material NASA released the day of Griffin’s press conference were these:“There are big problems with fatigue, especially at night.”“Ask if it’s a 2 or 3 man crew for monitoring purposes. How many pilot errors were made when inputting data into the flight management system.”“CRM (crew resource management), ability of crew members to act as a group and come up with proper safety results when operating in an unusual situation.” “Airplane maintenance is poor and getting worse. Cost is the bottom line, not safety, for airlines.” “Too many airplanes landing at airports.” “Change of runways within 10 miles is a problem; frequency congestion on domestic; on approach, being told to keep up speed; not having enough time to adjust to final instructions to land.” “NOTAMS [Notices to Airmen] system is inadequate – abbreviations and decipherings are impossible and worthless.” “The transition from the en route structure to the arrival phase – it’s a real problem nationwide regarding traffic separation on arrival. Almost universal everywhere you go.” |