The election this year has taught me many things, including this: Question everything. I already had a healthy dose of political skepticism heading into August 2015, but after closely following the primaries and their aftermaths, I have learned to look beyond what is spoon-fed to me. Polling, for instance, is a crucial part of every election, a way for the public (as well as the campaigns), to judge a candidate’s performance and chance to win the election. But when it comes to third parties, the picture is more complex.
Third party candidates have a troublesome relationship with the media. Usually, there is much less media coverage of third parties, which means the average American is less likely to know anything about third party candidates. But, because the average American doesn’t have the information about third party candidates, they’re less likely to be interested in learning about them.
More importantly, polling also determines who is let into the debates or not. In turn, the debates determine who the American public gets to know. The Commission on Presidential Debates currently sets the threshold for inclusion in the presidential debate at 15%. A candidate must poll at an average of 15% support in five polls nationwide. Gary Johnson must be frustrated indeed this year, as his support has been hovering around 8-12% for weeks.
How are the polls conducted? Could they be skewing the results in a way that negatively impacts third party candidates? This is the story of my journey as I examined the recent four-way general election polls.
Problems finding information
One of the first issues I ran into was the availability of full demographic information from the poll conductors. The first poll I tried to look into was a Rasmussen Reports poll. “Additional information from this survey and a full demographic breakdown are available to Platinum Members only.” To be a Platinum Member, you must pay a subscription fee of $20 a month or $200 a year. Not willing or desiring to give them money just to look at one poll, I moved on to the next poll, wondering what Rasmussen might be hiding.
Another poll I looked at was the latest NBC News|SurveyMonkey poll. The article reports on poll results from 8/29/2016 to 9/4/2016, but when you scroll to the bottom and click their link “for full results and methodology,” you are directed to outdated results from polls conducted 7/4/2016-7/10/2016.
There are other problems as well. Some polls don’t provide full demographic breakdowns and some only provide certain demographic categories. But while they do not always provide the demographic information, most polls do generally say they have weighted their results to correct for discrepancies in their demographics. I guess we just have to take their word for it.
Underrepresenting Independent voters
The main demographics that are interesting were the political party affiliations and age range. I would have looked into gender and race, but gender seems to be fairly well represented in the polls (as in it’s roughly 50/50) and a detailed race breakdown is much less commonly available. Thus it is harder to draw conclusions from these categories.
I have repeatedly heard and read over the past few months that around 40% of Americans consider themselves to be independents. Indeed, looking at Gallup’s polling results on party affiliation for 2016, the average does work out to be about 40%. Following this logic, political polls should reflect the trend and draw from a pool of respondents that is around 40% independent in party affiliation.
Yet, this is often not the case. Of the recent polls I surveyed, three had underrepresented sample sizes for independent voters. GWU/Battleground had just 14% (or 31% if you include the “lean” results as independent voters). The Reuters/Ipsos poll had just 206 independent voters, or about 13% of their sample (perhaps even a lower percentage if I read their data incorrectly). Fox News had only 16% of their sample identify as independent or other, with another 3% who didn’t know or refused. Many of the polls didn’t have party affiliation information.
Moving on to age, it was more difficult to find the breakdown by age group among polls. Plus, most of the polling methodologies say they weight their results to help smooth out any discrepancies in their age demographics. However, there was one poll in particular that stood out with respect to age: CNN/Opinion Research Corporation.
The CNN/ORC poll was the first poll that I encountered with a party affiliation breakdown that included 40% independent voters. I was initially excited to see how that would affect their results. This was the poll, after all, that showed trump up by 2% in the polls this week (note: that the margin of error was +/- 3%).
Yet, when I looked at their pages and pages of results with a breakdown by various demographics, I noticed something shocking. For their age groups, the 18-34-year-old group all said “N/A.” There were no data for this age group! Furthermore, it was the only demographic group in all of their tables that are excluded. I do not know whether they somehow had fewer 18-34-year-olds respond, but I doubt it as this seems to be consistent with their older polls as well. They exclude responses from this age bracket it seems.
Of course, this was immediately fishy to me because I know that one of the main demographics for Bernie Sanders was younger voters. If you exclude young voters from your results, you’re going to skew your whole results, possibly in a major way.
Why age matters
This suspicion was confirmed when looking at the age breakdowns in other polls. The Fox News poll, for instance, shows that 16% of those under the age of 35 would vote for Gary Johnson. 8% would vote for Jill Stein. This is very different from their overall results of 9% for Johnson and 4% for Stein.
The Investor’ Business Daily/TIPP is also interesting to look at. This poll, in particular, likes to brag about how it’s consistently rated as the most accurate political poll. Their breakdown by age shows a more dramatic picture. 33% of those aged 18-24 and 16% of those aged 25-44 said they would vote for Gary Johnson. 13% of those aged 18-24 and 4% of those aged 25-44 said they would vote for Jill Stein. (As a side note: only 17% of those aged 18-24 said they would vote for Trump.)
Meanwhile, the Quinnipiac University Poll (which also brags about its accuracy and prestige) shows that among its respondents aged 18-34 years, 16% would vote for Johnson and 11% would vote for Stein. This poll also shows that 48% of that age group would vote for Clinton, vs. 24% for Trump. Economist/YouGov showed similar results, with an under 30 age group breakdown of 44% for Clinton, 14% for Trump, 18% for Johnson, and 11% for Stein.
Conclusions and full results
Age certainly seems to affect the results of a political poll. If you exclude any age group, it could dramatically skew your results. On the other hand, the percentage of independents included in your sample seems to have a smaller effect on the results. This last point may be due to the fact that just because someone is an “independent,” doesn’t mean they won’t vote for a Democrat or Republican. This is why some people think that the independent voter is a “myth,” because most independents lean toward one party or the other. That being said, it is best to have the most representative sample possible.
The Commission on Presidential Debates states the following:
“Under the 2016 [Non-Partisan Candidate Selection] Criteria, in addition to being Constitutionally eligible, candidates must appear on a sufficient number of state ballots to have a mathematical chance of winning a majority vote in the Electoral College, and have a level of support of at least 15 percent of the national electorate as determined by five selected national public opinion polling organizations, using the average of those organizations’ most recently publicly-reported results at the time of the determination.”
What are the five polls they have selected? Washington Post-ABC News, CBS/New York Times, CNN/Opinion Research Corporation, Fox News, and NBC/Wall Street Journal. As I showed above, the CNN/OPC excludes young votes (those 18-34 years of age). Meanwhile, Fox News underrepresents independent voters. The Washington Post-ABC News poll methodological details state: “Demographics and religious identity questions are not shown.” The methodology then goes on to explain how the poll uses “statistical weighting procedures to account for deviations in the survey sample.” So, in other words, “Just trust us.” I did not look at the other two polls, but my guess is that they aren’t much better.
Below is a table of my full results from the polls I examined. It is a limited survey of the polls, I will admit, but I hope you find the numbers as enlightening as I did (click to enlarge):
Other coverage on the polls
If you’re interested in learning more about polls and how polling agencies may be skewing results away from third party candidates, check out the links and videos below:
Redacted Tonight agreed largely with my discoveries:
The Humanist Report, on the other hand, discussed the question of poll wording:
The Washington Post report: How Presidential ‘Non-Opinion’ Polls Drive Down Third Party Numbers And Facilitate Debate Exclusion