Trajectory CEO Paul Flatters examines what research questions are worth asking the public – and why we shouldn’t believe them when they say things were better in the old days…
Last year Ipsos Mori collaborated with Kings College London and the Royal Statistical Society on a truly excellent piece of research looking at the inaccuracy of public perceptions of major issues of the day. In a report titled “Perceptions are not reality: the top 10 we get wrong”, their survey revealed that, for example:
- on average, we think teenage pregnancy is 25 times higher than official estimates: we think that 15% of girls under 16 get pregnant each year, when official figures suggest it is around 0.6%
- 29% of people think we spend more on Job Seekers Allowance than pensions, when in fact we spend 15 times more on pensions (£4.9bn vs £74.2bn)
And so on, for the proportion of the population who are immigrants, crime levels, money spent on foreign aid and levels of turn out in general elections. In all these areas, the public display a poor grasp of reality (as revealed by the official statistics).
At the time I took this as an important reminder that in market research, social research and, especially, trends research it is only safe to ask people questions to which they can truly know the answer. We’ve all seen surveys where the public believe that things are awful nationally – that family life is in decline, NHS treatment is terrible, politicians are useless – and other surveys that show people’s own direct experiences of family, the NHS and politicians are much more positive. People can credibly answer these latter questions from their own direct experience – they do know about their own families. But in the former questions they are simply invited to pontificate on the state of the nation, based on the sort of ill-informed perceptions so expertly identified in IPSOS Mori’s 2013 study.
So it was surprising to see IPSOS Mori using some deeply flawed ‘pontification’ questions at the heart of their Global Trends 2014 Report. For example, The Huffington Post quoted the report as saying
“An average of two in three (64%) of the respondents around the world and 61% of Britons said they believe people led happier lives “in the old days when they had fewer problems to cope with”.
What on earth is a poor client supposed to do with this pearl of wisdom? This question not only asks people to pontificate about the state of life in their country now, but also pontificate about the state of life in the past. How reliable is that?
The only thing we can possibly glean from this question is that people today think that life was less problematic and happier in the past. Of course, it does not mean that it actually was. Only proper, longitudinal trends research can determine that, and the data tend to show that happiness levels are remarkably consistent over time.