One source of irritation for me is the way that news reports seem incapable of asking basic questions about numbers when reporting the news.
On 4th April 2016, another example caught my eye. According to the BBC, seizures of fake passports are at an ‘all time high’. Does that sounds worrying? Is it a cause for concern? What’s the story here?
If this report from the BBC is all there is, then the answer is, in order, ‘no’, ‘no’, and ‘not much’.
Bad Maths I – When is a difference significant?
Firstly, the news story contains the actual numbers of seizures –
- 2010 – 1,074
- 2011 – 1,024
- 2012 – 895
- 2013 – 1,082
- 2014 – 1,099
- 2015 (January to June) 528
The news story states that 2014 was an ‘all time high’. Two points to note here.
Firstly, yes, that’s true, but the idea that there ahs been a huge spike in the last few years isn’t borne out. The average number of seizures over those five years is 1,035. So in 2014 there were 64 more than that. Is that statistically significant? It doesn’t seem it given the fluctuations over the last few year. Looking at the actual data, there was a total number of 17 more false passports.
It’s an all time high, but if there’s five years of data then unless there is the same number of seizures each year (which would be suspicious quite frankly), one of those years has to be a ‘high’. An all time high even. If they were spread randomly over the time period, then there is a 20% chance that the last year was the highest.
There is second point. For the first six months of 2015, there were 528 false passports seized. These may not be distributed evenly over the course of a year, but if they were (and it’s unlikely to be much out) then 2015 would have seen 1,056, which would be a drop since 2014 – the year that the story is concerned with. Probably not a significant drop but, if that is the case, then the headline should be “false passport seizures return to normal after a bit of an increase“.
Bad Maths II – when is a number big?
In any news story, a number in isolation is useless unless you’ve got other information as to its context.
As an example, is 3 a big number? Well, if you’re talking about the number of limbs you have, then the answer is ‘no’. On the other hand (so to speak), if you’re talking about the number of houses that you own then three is quite a lot.
According to the Government, in 2015 there were 118.4 million journeys into the UK in the year ending March 2015.
That is a lot of people. Set against that, a figure of a coupe of thousand passports is a tiny, tiny fraction of the total number of passports and ID documents that are presented. To be specific, 0.00092820945% of visitors presented a false passport in 2014. On most measures, this would be rounded down to zero.
This also shows why the small changes in the numbers of passports is meaningless. The actual story here is that yes, there was an increase in 2014. But the news report could read that there was a “0.00001% increase in the number of people that are caught with false passports” from the year before.
If you want to be a bit sneaky, then you could just look at the total number of people who were refused entry (16,255 in 2015) as a more appropriate baseline. Here, the figures get a bit more murky. In 2014, 7.4% of people refused entry had presented false documents. In 2015 this went down to 6.8% (because there were fewer people refused entry). Well within the margin of error, but not really consistent with the figures being an ‘all time high’.
There may be more to the story than that, but it’s hard to see what.
The story here is that the number of people that present a false passport is pretty much the same year on year. Move on, there’s nothing to see. Not really news. It’s not yet been picked up by the Mail, Sun, etc, but you can imagine what they would/will make of it. Sometimes asking a few questions helps to show that the actual news is no news.