Q&A – How to Detect Legit Polls
Each day I feature a listener question sent by one of these methods.
Email: brianmudd@iheartmedia.com
Social: @brianmuddradio
iHeartRadio: Use the Talkback feature – the microphone button on our station’s page in the iHeart app.
Today’s Entry: @brianmuddradio What’s up with this poll saying that Trump’s lead is only 2 in Florida? Much different than what you’ve said.
Bottom Line: The short answer is that you’re probably a bit confused about what the poll represented. On Thursday, a new University of North Florida poll (which is the one that’s being questioned), was released. Due to the pollster, the University of North Florida, – it appears some in national news media misinterpreted who was being sampled in the poll. Headlines like this from The Hill is where the confusion came from: Trump tops Biden by 2 points in new poll | The Hill. The Hill originally, in their reporting, suggested that it was a Florida-based poll, it wasn’t. As they later came back to explain: Editor’s note: This story has been corrected to reflect the poll was a national survey. Had Trump only shown a lead of 2-points in Florida with a credible sample, that would have looked vastly different that the RealClear Politics polling average in Florida. Currently the RCP Florida average shows Trump up 9 on Biden – a number that’s well outside of a margin of error and is close to what Joe Biden’s lead over Donald Trump is in New York for additional context. It’s also a result, that if true, would seemingly have placed a whole host of elections down ballot that aren’t expected to be terribly close in play – none the least of which is this year’s senate race between likely candidates Rick Scott and Debbie Mucarsel-Powell. The polling result, if true would likely also have broad implications for Florida’s proposed constitutional amendments including the abortion and marijuana amendments.
The two-point advantage Biden is shown with in that poll is roughly in-line with the RCP national average I’ve cited in my weekly updates (though on that note – there's good news for Donald Trump over the past week as his advantage has appeared to grow since the start of the New York trial). But while confusion by news outlets like The Hill about the UNF poll led to confusion by their readers, this presents an opportunity to also discuss what to look for in a poll to detect a credible poll in Florida or in general for that matter.
The first thing I do when looking into polls is to go directly to the polls, as opposed to reading stories about them. If you go to the last page of the poll there’s often demographic information posted about who was polled. That’s key in understanding what the sample was of the people who were polled and can be used for additional context. When weighing the validity of a Florida poll I go straight to the voter registration information for the state. Here’s the current breakout of Florida’s registered voters:
- DEM: 32.3%
- GOP: 38.9%
- NPA/Other: 28.8%
With Florida’s Republicans currently holding a 6.6% advantage over Democrats, polling samples from within this state should mostly mirror that. When comparing samples nationally, it’s a much less exact science. Only 33 states have partisan voter registrations, so there’s no way to know precisely what a national partisan breakout should be. That means the best any pollster or analyst can do is to provide a guesstimate. The way I prefer to go about this is by using Gallup’s monthly tracking. For well over twenty years Gallup has tracked the partisan trends of voters monthly. The most recent data is from March. Here’s what it looked like (with partisan leaners included):
- 45% Democrat
- 44% Republican
- 11% Independent/Minor Party
(If partisan leaners aren’t included Republicans actually have a 2% advantage most recently). What this means is that decent national samples should have a slight oversampling of Democrats. So, with that in mind, since it brought about today’s Q&A, let’s look at what the UNF poll’s sample looked like:
- 40% Democrat
- 34% Republican
- 26% Independent/Minor Party
Using the Gallup comparison, Democrats were potentially oversampled by about 5% in the UNF poll which makes the 2% Trump lead nationally potentially even more impressive with that additional context. To UNF’s credit they denoted a 4.5% margin of error within this poll. That places the sample’s margin of error basically within the national Gallup data’s window with the notion being that if the poll is off, it’s probably under sampling Republicans by a bit.
Hopefully this exercise helped explain last week’s confusion with UNF’s poll and also what to look for if you want to wonk out on polling data this cycle!