Statistical Muddying

Former FAA Administrator Marion Blakey said the next goal regarding safety is to further reduce the number of fatalities. There are a few considerations, however, to what she is saying. At the 6 September town hall meeting with FAA employees, Blakey said:

“We’ll be using a new performance metric for commercial air carrier safety – fatalities per 100 million persons on board … We aim to cut this risk in half … We are currently at 8.8 fatalities per 100 million persons on board, and will focus our efforts to get to 4.4 fatalities by 2025.”

All well and good, but few people have a feel for a number as big as 100 million, so whether the fatality rate is 8.8 or 4.4 of 100 million really is an abstraction. Most people have their brains full with the notion of a mere 100,000 at a football game, so a base number that is 1,000 times greater basically has no practical meaning.

For another thing, whole numbers are preferable to eight-tenths or four-tenths of a person killed.

For yet another thing, achieving a rate of 4.4 persons killed per 100 million by 2025 is difficult to relate to the past goal of reducing the number of fatal airline accidents by 80% over the past decade. The past goal spans 10 years; the future goal covers a period of 18 years – the two are not comparable.

There is no mention by Blakey of a 3-year rolling average, or the fact that the FAA has kept book for the past decade by fiscal year, not calendar year. Fiscal year accords with the FAA’s budget cycle, but recording accidents from October through September, rather than January through December, just makes it difficult to compare the FAA rates to other databases.

Case in point, the accident and incident data maintained by the National Transportation Safety Board (NTSB). It is more detailed (and sensible) than the way the FAA keeps score on safety. For airlines, the NTSB data base tracks non-fatal as well as fatal accidents, thereby covering significant damage to aircraft and injuries. The database also tracks accidents annually, per departures, per flight hours, and per miles flown (see Figure A ).

It would be sensible if the FAA were to emulate the NTSB’s bookkeeping in its goal setting.

The same observation applies to general aviation (GA). Blakey has said, “There have been 284 fatal accidents, versus a not-to-exceed ceiling of 331 for the year.” The first problem is the dubious provenance of this not-to-exceed number. The original goal was to eliminate a year’s worth of fatal accidents. Tom McSweeny, then the FAA head of regulation and certification, said in March 2001 that “if we did nothing” there would be 437 GA accidents in 2007, and the goal was to cut that projection by 350 accidents and save more than 600 lives.

None of the FAA’s numbers can be related to the NTSB accident database for GA (see Figure B).

One thing is clear: the GA accident rate, at least in terms of gross numbers, is entirely too high. Blakey’s allusion to 284 accidents last year, while lower than the number shown on the NTSB database, reduced GA accidents from one every 29 hours implied in the not-to-exceed goal to one fatal accident every 31 hours. Compare this rate to that of two airline fatal accident rates per year, now achieve, and it is evident that reducing the GA fatal accident rate to even just one a week would be a major improvement.\

Irrespective of goals, the FAA should set its safety improvement goals in accordance with the NTSB bookkeeping of accidents. In this regard, setting a goal of reducing fractional fatalities per 100 million persons on board is an abstraction times two: the NTSB database counts whole deaths, and it does so in increments of 100,000 flight hours.

Figure A

Figure B