WEBINAR HIGHLIGHTS: ‘Using Polls, Surveys and Research in your Journalism’
During the Covid-19 pandemic, statistics and research have become central to news reporting.
With this in mind, IMPRESS and the Market Research Society (MRS) co-organised a recent webinar, ‘Using Polls, Surveys and Research in your Journalism’ on how to report responsibly on statistical data, featuring insight from industry experts. This open event took place on 3 June.
Hosted by The Conversation’s Politics Editor, Laura Hood, the webinar featured Opinium Associate Director and Partner Adam Drummond, and Britain Thinks Associate Partner Cordelia Hay. This training session, open to journalists across the board, followed on from guidance on reporting on polls and statistics, co-created by IMPRESS and MRS that was released at the end of 2019.
Free and open to all, at the webinar Drummond and Hay spoke about the merits of using quantitative and qualitative research in journalistic reporting, as well as warning of the potential pitfalls of taking poor quality data “at face value”.
Highlights and top advice
Some useful pointers from Opinium’s Adam Drummond.
>> Three useful things to consider when looking at polls:
- Twyman’s Law:
- Public opinion changes more slowly than we think
- “Any figure that looks interesting or different is usually wrong” - William Twyman
- Questions to consider: Is the poll representative?
- Self-selecting polls are meaningless.
- Proper polls make sure that nobody can take part more than once. A representative cross-section of a population are asked and there aren’t any selection biases. People who have no interest in a subject should be just as likely to have taken part as people with strong opinions about it.
- Questions to consider: Who conducted the research?
- Is it a reputable organisation?
- Is the full data set available?
- How did they ask things?
- Is the story “a little too good” for one particular point of view?
>> Things to watch out for in polling about Covid-19:
- “Most people are not epidemiologists or public health professionals”:
- Their ability to answer questions about a subject will depend on the information available to them.
- Be wary of responses to detailed policy options, especially if presented in agree/disagree statements.
- “People take cues from their leaders”:
- People will typically split along party lines except on subjects where those parties or leaders largely agree e.g. lockdown in March 2020.
- Where leaders disagree, people will typically rely on the views of leaders they respect on subjects where they themselves may not have strong views.
- “Prompting makes a big difference on whether something has ‘cut through’”:
- People tend not to want to tick ‘don’t know’ and most responses to poll questions are a case of ‘well, since you ask…’
- “The way we knew the Cummings story had cut through was not just overwhelming answers to prompted questions but also in unprompted open text questions.”
>> Things to consider ahead of the 2020 US Election:
- “Our most useful predictor is less useful than before”:
- The relationship between the popular vote and the electoral college vote is untethering.
- Job approval is usually the most reliable metric of a president’s re-election chances.
- “Multiplying uncertainty”:
- There are lots of data points that go into a prediction of who will win; each with a degree of uncertainty. When the data points compound, this creates greater uncertainty.
- Addressing the impact of the pandemic on turnout:
- ~25% of voters cast ballots by mail in 2016. Some states (Michigan, Pennsylvania) have moved towards postal ballots but others (Texas) have moved away.
Notes from Cordelia Hay (Britain Thinks).
>> On top of adding ‘colour’, and the ‘why’ to the ‘what’, bringing qualitative methods into your journalism can tell you:
- What views are people bringing to the issues or voting decision?
- What are the issues that will matter to them when casting their vote?
- What is their starting view of the political parties and leaders?
- How firm are these views? How easily could they be swayed?
- Voter volatility is higher now than it has been since the 1930s
- Use qualitative insight to look into the ‘whites of people’s eyes’ to work out the strength and depth of their views
- What is cutting through to people (and, perhaps more importantly, what isn’t)?
- In most elections and on most issues, the public pick up only a fraction of news and political coverage
- Issues which feel seismic in the Westminster bubble sometimes barely get a mention by the general public
- The irrational, unspoken truths about an issue that sometimes just don’t surface in a poll.
- Use creative techniques to surface people’s deeper feelings about a topic
- Qualitative insight tells you how people are talking about an issue – the classic example being ‘Get Brexit Done’
>> How to incorporate qualitative research into your journalism:
- “It doesn’t need to be either/or”.
- Percentages will often make the best headlines, while qualitative insight adds nuance, texture and a human perspective under the byline.
- Qualitative data lends itself to richer, more detailed content: features, editorials, web content.
- In particular, qual can give you brilliant multimedia content including voxpops and case studies.
- Don’t get too het up on the sample size.
- Qualitative research is never representative, more doesn’t necessarily mean better, and great insight can come from just a few interviews.
- But, just as with the data, the devil is in the detail.
- Rather than sample size, you’ll want to check sampling approach and whether views are prompted (and if so, with what).