The importance of sampling – the devil is in the detail

10th Apr, 2015

As a long-time fan of Psychologies magazine, I love nothing better than a personality test.

Even better one that promises to reveal where I should move to live among like-minded people. Imagine my excitement on March 25th when the BBC launched its new research which maps the personality of 380 places in Great Britain. Where does my personality fit in?

The online survey was conducted over two years and nearly 400,000 people took part, covering the 380 local authority areas of England, Wales and Scotland. The project was conducted by Cambridge University and the BBC’s Lab UK project, with the results published in the journal PLOS ONE. Authoritative source, huge sample size: so far, so good.

Then I looked at the table of results….

Places and their personalities

  • Open: Hackney – Not so: Maldon, Essex
  • Conscientious: Isles of Scilly – Not so: Manchester
  • Extrovert: Hammersmith and Fulham: – Not: Boston, Lincolnshire
  • Agreeable: Isles of Scilly – Not so: City of London
  • Neurotic: Boston, Lincolnshire – Not so: Orkney

Not only does the Isles of Scilly have the most agreeable inhabitants but also the most conscientious too. On further investigation of the survey data (accessible to all in the PLOS ONE paper), the Isles of Scilly folk are also 2nd least likely to be neurotic and in the top 10% for being most open and within top 20% for being most extrovert. Of course, this could be the result of the temperature – the relative warmth of the South West of England – as suggested in the paper.

But alarm bells rang when I realised that the Isles of Scilly had the smallest sample at 27. So I then looked at the results for the largest sample – Leeds (at 5000 completed surveys). Here we see fewer extremes: the results for the Leed’s inhabitants are all within twelve percentage points of the average for the whole country.

This is when I started to suspect that what we are seeing here is just another example of Bernouilli’s Law of Large Numbers. For the best explanation of this, please read How Not To Be Wrong: The Hidden Maths of Everyday Life by Jordan Ellenberg – brilliant book on maths for non-mathematicians. In summary:

“The Law of Large Numbers will always push the big players’ scores toward 50%, [i.e. Leeds] while those of the Smalls [i.e. The Isles of Scilly] are apt to vary much more widely…..the smaller the sample size, the greater the variation.”

So despite the headline figure of a sample of 400,000, by looking at the results by Local Authority District the authors of the personality test report have ventured into dangerous territory and are making the error of reporting unreliable results due to small sample sizes

We can get around this problem here at Trajectory by using weighted averages – while it’s a bit of an art  we have many years of experience.

Charlotte Cornish is Research Director of Trajectory. According to the survey, her best fit for happiness is Hart, near Fleet in Hampshire – worst fit, the Outer Hebrides.