Invented by 18th-century English mathematician Thomas Bayes, Bayes’ theorem calculates the odds of one event happening given the odds of other related events. Some mathematicians refer to it simply as logical thinking, because Bayesian reasoning is something humans do naturally. If a husband tells his wife they didn’t eat the cake but their face is covered in chocolate, the probability of their guilt goes up. But when many factors are involved, a Bayesian calculation is a more precise way for statistics to estimate the probability of guilt or innocence. Its use in Court has not been without problems however.

To take a fictional example, consider the evidence of the victim of a bag snatcher and an eyewitness, who both describe the thief as a very tall man (over 2 metres in height), between 20 and 30 years old, with red hair and a pronounced limp. Later a man fitting this description is arrested but denied being the bag-snatcher.┬áIn court the prosecutor can’t produce any forensic evidence (the thief was wearing gloves,) but calls a mathematician (let us be kind, and say that statistics aren’t his primary field). The mathematician’s evidence is that, in London, the probability of:
Being male; 0.51
Being 2 metres tall or more; 0.025
Being between 20 and 30 years old; 0.25
Being red-headed; 0.037
Having a pronounced limp; 0.017

Because these probabilities are all independent of each other, the probability of one person having all these characteristics is the individual probabilities multiplied together, ie (0.51 x 0.025 x 0.25 x 0.037 x 0.017) = 0.000002. So the probability of only one person having all those characteristics is only 0.000002, or 2 in every 1 million people. On the basis of this evidence, the accused looked certain to be found guilty, but then the defence cross-examined the mathematician.

The solicitor for the defence asked him “Am I correct in saying that in the original trial you stated that the chance of any random individual having this set of characteristics was 0.000002, and that this grouping of characteristics is so rare that you will only come across it in one in half a million people?”

“Yes, I am, which is what points towards him being the perpetrator.”

“What is the population of London?”

He looked startled, but replied “I think it is about 10 million.”

“So how many people in London have this set of characteristics?”

He was forced to admit that there should be 20.

“Given that your evidence is based solely on a description and that you have admitted that there are 20 people in London who fit this description, this must mean that the probability of my client’s guilt is very small, only 1 in 20. Or to put it another way, the chance of his innocence is 19 in 20, not 1 in half-a-million.”

“Yes, I suppose so.”

In the absence of any other evidence against him the red-headed man was acquitted, and the police began to search for the other tall red-headed limping men.

The mathematician confused two things. The probability of an individual matching the description, and the probability of an individual who does match the description being guilty; they are not the same.

Although a probability might be small, when that is applied to a large population such as that of a whole city, the “suspect group” is suddenly much larger. Here, 1 in half a million translates into 20 possible suspects, and the accused is only one of this group. To convict safely, the other 19 also have to be excluded.

Of course, the fact that the accused is only 1 of 20 possible suspects does not mean that he is necessarily innocent; the guilty man is one of those 20. But it is a simple example of why the apparently tiny probabilities that are often quoted can be very misleading, and have to be used in conjunction with evidence from other sources to safely come to a conclusion.

The same principles also apply to other situations in which statistical methods are used in Court-forensic DNA testing in criminal trials, and paternity cases to name just two.

A real example of a miscarriage of justice attributable to the prosecutor’s fallacy is well-known. In 1974, Scotland Yard arrested a group of six men for two bomb explosions in Birmingham pubs attributed to the Provisional IRA. Their hands were swabbed for traces of nitro-glycerine and the swabs then analysed. Based on a positive test result, a forensic scientist testified at trial that he was 99% certain the defendants had handled explosives. However, it then transpired on appeal that many other substances can produce positive test results, including paint, lacquer, playing cards, soil, gasoline, cigarettes, and soap. The forensic scientist mistook the probability of a positive test result assuming that the suspects had handled explosives, with the probability that the suspects had handled explosives given a positive test. The convictions of the Birmingham Six were overturned on appeal as a result.

Although Bayes’ Theorem is well-understood and extremely powerful, it does depend on reliable and extensive datasets, and this can lead to other problems.

In 2010 a convicted killer, “T”, took his case to the Court of Appeal. The evidence against him included a shoeprint from a pair of Nike trainers, which seemed to match a pair found at his home. To apply Bayes’ Theorem correctly, the statistical analysis needed to consider the probability of the print at the crime scene coming from the same pair of Nike trainers as those found at the suspect’s house. To work out that probability, the calculation needed to factor in how common that precise kind of shoe was, the size of the shoe, how the sole had been worn down and any damage to it. But between 1996 and 2006 Nike distributed 786,000 pairs of trainers in the UK. In addition there are 1,200 different sole patterns of Nike trainers and around 42 million pairs of sports shoes sold every year. Not surprisingly, at the trial the expert couldn’t even say exactly how many of one particular type of Nike trainer there were in the country.

The Court of Appeal therefore decided that the conviction was unsafe, and in its judgment warned (not for the first time) that Bayes’ Theorem had to be used with considerable caution, and only when sufficient data was available. The judgment lead to some rather overblown commentary about how the Courts were rejecting the use of statistics; read properly, the Court of Appeal was simply saying that where it is used, statistics needs to be applied carefully and in the knowledge of its limitations. To rely on statistics alone, without any other corroborating evidence, is very risky.

for more information contact David Vaughan-Birch.

The prosecutor’s fallacy; statistics can be as misleading as damned lies in Court