by Greg Enslen, 08/15/12, gregenslen.com:
This week, I was going to write another episode of ” what’s up with my wives magazines,” but then the release of this CNN poll prompted me to backburner my original column. I’ll get to it, I promise. I don’t usually talk about politics, but I found this poll to be very misleading and thought it would be a good opportunity for some critical thinking.
CNN and “Margin of Error”
This week, one of the biggest pieces of political news (before Romney’s VP announcement Saturday morning) was the release of several new polls. CNN announced them with “Recent Polls show President Barack Obama has Expanded his Margin” over Romney, going on to declare that “52% of registered voters say they would support the President, while 45% are backing Romney.”
CNN further stated “In the same survey last month, Obama held a three-point margin over Romney, 49%- 46%, a difference that fell within the survey’s sampling error.”
I have always been curious about polling methodologies, and in particular I was curious about Obama’s sudden surge in the polls—I had thought things were running neck and neck. I decided to go through the poll itself to understand what had changed. I found the PDF of the poll on the CNN website and downloaded it, studying the numbers. On the surface, it is a poll of voters to predict how they will vote, right?
Up front, the PDF states that the CNN/ORC poll (I wonder if it was conducted by Gandalf) says that they interviewed “1,010 adult Americans”via telephone on August 7-8, with a sampling error of +/- 3%. It also states that the sample “includes 911 interviews among registered voters” and explains that 760 of the interviews were conducted via landlines and 250 via cell phone.
“Registered voters” vs. “adults”
First, notice the 911 registered voters and 1,010 adult Americans. What does that mean? It means that 99 of those polled, or 9.8%, aren’t registered to vote. So why do we care about who they’re going to vote for if they’re not going to vote? If they can’t be bothered to register to vote, it’s much more likely that they won’t vote, so their opinion, while valuable, doesn’t matter.
It’s like asking someone if what flavor of ice cream they like to determine if strawberry is more popular than rocky road. A bunch of people like rocky road, but more like strawberry, so the reviewer determines that strawberry is more popular. But then, on further analysis, ten percent of those asked turn out to be lactose intolerant. They’re not going to buy ice cream anyway, so why do we care about their opinion? Are they picking their favorite flavor based on the box the ice cream came in? It doesn’t matter—they’re not buying any anyway.
Lesson Learned: when reading polls, look for “registered voters,” not “adults.”
Ah, this one is sneaky—be on the lookout for the Sample Makeup. You know how people always say “they can make numbers say whatever they like?” This is one way that can happen.
This CNN/ORC poll is of random Americans, so it should be approximately 50% liberal and 50% conservative, right? Or something else that approximates the overall population, like 40% liberal, 40% conservative, or 20% independent. (According to a Gallup poll conducted in January 2012, the split is 40% conservative, 35% moderate, and 21% liberal: http://www.gallup.com/poll/152021/conservatives-remain-largest-ideological-group.aspx).
In the case of this poll, they asked 1,010 “Adult Americans” who they wanted to see be the next president. As part of the polling, they also asked about political leaning. 419 of those polled identified themselves as republican or republican-leaning.
That means that the other 591 adults consider themselves to be liberal or liberal-leaning. Let’s calculate the percentages: of those polled, 58.5% labeled themselves as liberal and 41.5% were conservative.Let’s look at that number: 419 conservatives out of 1,010 adults.
Now, dear reader, you know what’s going to happen next—the pollster will ask them who they think will win. Based on their political leanings, and considering no other factors, Obama should get 58% of them and Romney 41%, assuming all the liberals voted for Obama and all the conservatives went for Romney. Right?
Lesson Learned: when reading polls, look for WHO they polled and make sure it’s an accurate, random sample.
Looking at the numbers in a new light, it looks like Obama should have received 58% of the votes and Romney 41%, based on the skewed makeup of the sample population. But Obama scored only 52%, and Romney 45%. That means that Obama is polling 6% UNDER where he should be in the poll and Romney is 4% ABOVE.
To assess the poll fairly, I removed some of the self-identified “liberals” from the polling data and compared 50%/50%. For the above sample, the 419 conservatives accounted for a 45% Romney result, so they were likely to vote for Romney 108% of the time. The 591 liberals were likely to vote for Obama 89% of the time.
Subtracting out 172 liberals, we get a sample of 419 conservatives and 419 liberals for a total of 838 respondents in this poll. Calculating using the “likely to vote” numbers above, we get a completely different result:
Romney 54%, Obama 44%.
So, according to the CNN/ORC poll conducted on August 7-8, 2012, Romney is ahead by 10 points.
Hmm. Now, I’m not saying my poll result is any more accurate than CNN’s—I’m using their poll data and guess on the number of respondents who didn’t give a party affiliation. But since 419 people self-identified as republican or republican-leaning, it’s safe to say that the remainder were liberal or liberal-leaning. And KNOWINGLY using unfair percentages, CNN managed to flip the outcome to Obama’s favor. A more accurate poll could be obtained by polling several thousand random registered voters.
My point is that polls can be “adjusted,” so numbers should be taken with a large grain of salt. The next time you hear a poll number from ANY source, take a moment to investigate the claims being presented, or you might be doing yourself a disservice.
Lesson Learned: Pay attention and Practice Critical Thinking.
Lastly, be aware of trends. CNN said that “In the same survey last month, Obama held a three-point margin over Romney, 49%- 46%, a difference that fell within the survey’s sampling error.” While that is true, they fail to mention that, in the same poll, it was Obama 53%/Romney 45% — BACK IN APRIL 2010.
Obama is down to 52% from a peak of 54% in March 2012, and Romney is down from a peak of 51% back in November 2011. Trends are illuminating over time, but month-to-month comparisons are less helpful than long-term trends. And according to the long-term trend, Obama and Romney are EXACTLY where they were over two years ago.
Of course, these numbers may be based on the faulty calculations such as those shown above. Reader, do you think it would affect the presidential race if CNN came out and said that Romney was ahead by 10 points, and had been for over two years? My point? Be sceptical, and assume every media source, from NPR to Fox News, has an agenda to promote. Until I find a truly objective source of news, I’ll continue to watch CNN and Fox and MSNBC—I guess I have to get the news from multiple sources if I want to get the real, unfiltered story.
Lesson Learned: Look at the numbers carefully-and investigate.
I’m writing this article on Monday morning, August 13, and wanted to cover some of the press coverage of the Ryan VP announcement, but I’m out of space. There was a whole host of “news reporters” or “commentators” — frankly, I can’t tell the difference anymore — holding forth on the Ryan pick. I’ll write it all down for next time. Feel free to drop me an email at email@example.com if you’d like to let me know more about creating accurate polls or past polling errors. Talk to you soon!