Q&A of the Day – How Reliable Are Political Polls? & How Samples are Chosen
Each day I feature a listener question sent by one of these methods.
Email: brianmudd@iheartmedia.com
Social: @brianmuddradio
iHeartRadio: Use the Talkback feature – the microphone button on our station’s page in the iHeart app.
Today’s Entry: Are political polls honest? I’m one of a large number of 60–70-year-old Floridians in my family who have never been polled? How are folks selected?
Please explain.
Bottom Line: The reliability, or perhaps better stated the perceived lack of reliability of political polls has increasingly been in focus in recent years. Skepticism over political polling reached a fever pitch after the 2016 presidential election in which pollsters seemed to under sample support for Donald Trump significantly. In reality, the polling industry wasn’t as far off as it may have seemed in the 2016 election cycle, but it was way off in 2020. I’ll start this exercise by breaking down the polling industry’s performance over the past decade and will then breakdown how pollsters are going about sampling for their surveys, which has been and is continuing to change considerably.
- 2014 Midterm Election: Polling avg. GOP +2.4; Actual GOP +5.7
- 2016 Presidential Election: Polling avg. Clinton +3.2; Actual Clinton +2.1
- 2018 Midterm Election: Polling avg. DEM +7.3; Actual DEM +8.4
- 2020 Presidential Election: Polling avg. Biden +7.2; Actual Biden +4.5
- 2022 Midterm Election: Polling avg. GOP +2.5; Actual GOP +2.8
You’ll notice a trend here. In all but one of the past five election cycles national political polling overrepresented support for Democrats with the lone exception being the 2018 midterm election cycle. Without further analysis or any explanations that seemingly lends credence to the concerns by those on the right that political polls are often rigged against Republicans. With that being said, the average result across those five cycles is an under sampling of Republican support by 1.3%. From an analytical perspective a miss by just over 1% on average per election is actually a pretty good result. Perhaps more importantly the overall outcome projected in the national polls has proved accurate in each of the previous five election cycles.
Republicans were projected to win the national popular vote in the 2014 midterm elections, and they did. Hillary Clinton was projected to win the 2016 popular vote and she did. Democrats were projected to win the 2018 national popular vote and they did. Joe Biden was projected to win the 2020 popular vote and he did. Republicans were projected to win the 2022 national popular vote and they did. Context is always key and in attempting to infer the validity of the polling industry that context is especially important. Also, notably, the most recent election cycle is the one that was polled most accurately. The previous election’s strong performance by the polling industry suggests that the changes the industry has made in recent years may be producing more accurate results and that takes me to the second part of your question. How samples are conducted and why some people are chosen while others aren’t.
Last year the Pew Research Center conducted a study of how political pollsters are conducting their polls. What they found is that the industry has rapidly been changing. The first big takeaway from their study is that over 60% of pollsters changed their sampling and their methodology between the 2016 and 2022 elections. The fastest growing change within the industry is the use of multiple methods to obtain samples for polls. The biggest change in the industry is a switch from call out surveys with live interviews to the use of digital sampling. Here’s the breakout of the percentage of pollsters using each of these methods:
- Online opt-in: 67%
- Probability-based panel (recurring panels of people who’ve agreed to be regularly polled): 33%
- Live phone interviews: 32%
- Texts: 13%
- Mailed surveys: 10%
- Interactive voice response (call-based voice automated polling system): 6%
So, with those six different methods currently in use I want to circle back to the statement in today’s note that you and your family members haven’t been polled. Is it the case that you’ve never been polled (or at least have been attempted to be polled), or is it that perhaps you just haven’t realized how pollsters are attempting to survey you and your family? For example, just two days ago I participated in a political poll conducted by text. Within the last two cycles I’ve participated in online surveys, texts and mailed in surveys. I think it’s likely that you and your family may have either been contacted by a pollster and/or may have come across opt ins for surveys without realizing it. In fact, you may have participated in a poll or polls without realizing it. Hopefully that’s helpful.