Abraham Wald and survivorship bias

      

Abraham Wald (1902-1950) was born in a Jewish family in a town in Austria – Hungarian empire.  He was home- schooled by his parents until college.  He studied mathematics in university, and graduated from University of Vienna with a PHD in mathematics. In 1938, he had to immigrated to the United States due to the discrimination and prosecution to Jews by the Nazi government.

During World War II, Wald was a member of the Statistical Research Group (SRG) at Columbia University, where he applied his statistical skills to various wartime problems. The SGR was most high powered and most influential for the wartime military decisions, it had a handful of most extraordinary statisticians, and the smartest person in the group was Abraham Wald.  

One of the problems that the SRG worked on was to examine the damage distribution on aircraft returning after their flying missions from Europe and to provide advice on how to minimize crew losses from enemy fire.  The military wanted to add extra armor to the vulnerable parts of the planes. If armoring too much would make the plane heavier hence less manoeuvrable and use more fuel.  The mathematicians needed to work out an optimum of how much the amour should add to the planes.  

The military came to the SRG with some data they thought might be useful.  The damages were not uniformly distributed across the aircraft, with more bullet holes in the fuselage (body part), not so many in the engines. The military officers suggested to concentrate the armor on the places with the greatest need, where the planes were getting hit the most.

However, Wald came with a very different answer: the armor shouldn’t go where the bullet holes were but quite opposite: on the engines.

Wald’s insight was that the planes came back with fewer hits to their engines, but the planes that got hit in the engines weren’t coming back.  In another words, the missing bullet holes were on the missing planes.  Returned planes with many bullet holes on their fuselages were the evidence that the planes could survived the hits on fuselages but not on the engines. Wald recognized this as biased sample that told a distorted and incomplete story.  The missing planes were not included in the whole sample set, only the survivors. Wald’s recommendations were quickly put into effect.  

Why could Wald see what the military officers, who had vast more knowledge and understanding of aerial combat, couldn't? It was owned to his mathematical thinking.  A mathematician is always asking: "what assumptions are you making? are they justified?"  To a mathematician, the structure underlying the bullet hole problem is a phenomenon called survivorship bias. 

Survivorship bias or survival bias is the logical error of concentrating on the people or things that made it passed some selection process and overlooking those that did not, typically because of their lack of visibility.  This can lead to false conclusions, it is a form of selection bias.