CITP Blog - Formerly Freedom to Tinker

Can you Hear me Now? In 2012, Some Political Pollsters Still Can’t

– by

Comments

Recently, I received a call from Gallup on our landline home phone, seeking to speak with my wife, presumably for a political poll. Because she was not at home at the time, Gallup’s representative told me he would call back later. To our knowledge that follow-up call never came. Gallup’s representative never asked me for my wife’s cell phone number, e-mail address, or any way to reach her beyond calling our home phone number again. Why not?

Apparently, some political polling efforts fail to recognize the variety of ways in which Americans communicate today. On his election season must-read blog FiveThirtyEight, Nate Silver wrote a post last month entitled “Obama’s Lead Looks Stronger in Polls That Include Cellphones.” Specifically, Mr. Silver observed that polls that use live interviewers and include cell phones show stronger results for President Obama than polls that use automated dialing methods or exclude cellphones. According to Mr. Silver, roughly one-third of American households are excluded by polls that call landlines only.

Failing to reach out to voters on their cell phones is a failure to connect with the way people use technology to participate in our democracy. A Pew study released several days ago points out that 88% of registered voters own a cell phone of some kind. Further, Pew found that registered voters who identify as Democrats and Republicans are equally likely to own a cell phone, own a smartphone, and to use text messaging or apps. Twenty-seven percent of registered voters have used their phone during this election cycle to keep up with the election or political issues.

The failure to include cell phones in political polling not only dims participation, but makes analyzing the meaning of a poll more difficult for members of the public. Mr. Silver can keep track of which polls likely include cellphones and which do not, but the average voter cannot do so easily. While Mr. Silver can develop a formula for weighting the value of polls that do not include cellphones in his election forecast, again, the public cannot do the same.

So, why would a political poll not include cell phones? If the purpose is not to skew the results, what’s the reason? Shouldn’t political pollsters be thinking about expanding their methodologies?

The public’s position on topics with less significance to the future of the United States than the coming Presidential election is polled using the Internet and wireless devices. For example, in a recent NY Times article, Arthur A. Stone explained, “Mondays Aren’t As Blue as We Think.” He helped reach his conclusions regarding people’s misperception of Mondays by analyzing a Gallop survey based on 1,000 live interviews a day on cell phones and land lines of people in every state, including querying Spanish speakers in Spanish. Beyond this serious academic study, visiting the web page for The Voice shows that there are seven ways to vote for artists: toll-free phone calls; on-line via NBC.com; the NBC Live app; the NBC app; Facebook; text messaging (Sprint customers only); and through certain iTunes purchases.

With fewer than three weeks until election day, where is the innovation or forward thinking in political polling? It’s past time for political pollsters to survey the entire electorate or immediately explain the practical, technical and financial limitations on expanding political polling beyond landline phones.

Comments

8 responses to “Can you Hear me Now? In 2012, Some Political Pollsters Still Can’t”

  1. Lior Silberman

    There is one significant difference between landlines and cell phones: receiving calls at a cell phone is not free.

    In countries where “caller pays” applies to cell phones like landlines no one minds getting called that way. In North America however the recipient pays for the airtime. Thus people don’t want to be called. This also affects the attitude towards commercial calls.

    Reply
    1. Bryan Feir

      The flip side of that is, if you need to have a cell phone anyway, why pay for a landline too? I know several people who don’t have landlines at all for the entire household, because they’re renting and a landline is an extra expense.

      And there are going to be skewing factors here. Homeowners are more likely to have landlines than people who move around a lot for their work. Older people who have always had landlines are more likely to have landlines than people who have grown up always knowing about cell phones. How much those correlate with someone’s likely politics is a question for someone else to discuss…

      Reply
  2. paul

    Although most people now have plans that offer enough prepaid minutes that answering a poll isn’t a big deal, there’s still a sense that cold-calling a mobile is a monetary imposition. But I’d say the problem goes well beyond this: thanks to decades of unrestricted telemarketing, many people simply don’t answer their cell or their landline (if they still have it) unless the caller is one the recognize. Like the internet, the phone network carries mostly spam at this point. So the samples pollsters get are the fairly small minority who bother to answer.

    Reply
  3. Anonymous

    I’m not sure if all polls are conducted the same, but I worked for a call center that conducted polls which seemed mostly biased in favor of republicans. Sometimes the surveys we would give would outright lie to the interviewee. Needless to say, I couldn’t stand working for the company with a blind conscious anymore, despite needing the money.

    But, as for what about my job is relevant here; we used an automated dialing system and were restricted to conducting the survey as it was presented on the screen. Additionally, it was a minimum wage job and we often had to deal with people that, understandably, didn’t want to be bothered. Sometimes, the interviewers honestly didn’t care much for the job. Imagine a job where you’re only allowed to get up twice during the two 10 minutes breaks you’re given, knowing you’re being payed minimum wage, hating the bias in the surveys, etc. etc.

    Reply
  4. Jeff Tignor

    At this point, the amount of data that a person can use monthly is more important to most people than the amount of monthly minutes in a cell phone plan, since texting has taken the place of much voice calling. Therefore, I don’t think the issue of the person receiving the call also paying for the minutes is as big an issue as it was several years ago. I do agree that most people do not want SPAM calls on their cell phones. However, I think pollsters need to think about what the next generation of polling is going to look like. For example, when Gallup called for my wife they could have asked me if they could text or call her. Pollsters need a more comprehensive strategy for reaching voters who do not rely on landline phones, but do have cell phones, e-mail addresses, social media accounts, etc.

    Reply
  5. Nathan T

    Thanks for the article I am glad other persons have a the sense to understand why such polls are inaccurate, I totally agree with you, though I would like to add even more reasons pollsters can’t hear me either.

    I had a similar situation with my wife being asked to take a poll. The poll was about filling out the census. As I was the one who filled out the census for our household my wife wouldn’t nor couldn’t have answered any questions about the census because she didn’t even see it, I filled it out–but the pollsters claimed SHE had to be the one to take the poll because that was what their computer called “random.” But calling landlines in not a statistically random sample for the very reasons you mention and more.

    Back in September a local news station (website) that I commonly comment on ran an article about Obama being up and Romney down in the polls and then questioned the polls’ validity. I commented on the fallacies of polling in general, and this idea of land-line telephone polling is one of the things I mentioned, though I was forced to keep my post to about 740 words so truncated a lot of my thought process.

    In it I talked about three things that make such political polls always inaccurate. I start with my experience in “polls” — I sat in on a college statistics class one day, and it covered why polling is supposedly accurate, but I spotted three fallacies.

    The college statistics class’s premise was that a small random selection (and it had to be random) polling was statistically accurate and they described how they came up to that conclusion. They used a huge container of marbles with a set of three or four colors. They knew how many of each color the container contained, and then they proceeded to do “random” counts of small portions of this container and graph how far off those random samples were. The graph they created they called a “bell curve” because it looked like a bell, with most of the time they conducted hundreds or thousands of these random counts to create this curve which suggested that most of the time the results were pretty accurate.

    Fallacies abound.

    1. Not every count was that accurate, only if you conducted hundreds or thousands of counts of the same container did you get enough accuracy to consider accurate. Polls however are only conducted once, which means in reality, just like many of their marble counts, any one single count may actually be at either end of the bell curve and be very inaccurate.

    2. Randomness is never true in the real world. Your post on telephones and landlines is part of why randomness isn’t real. No matter what they do they are actually getting a biased sample by biasing their results with A) only people with landlines, B) only certain people in the house they want to talk to, C) only those willing to even take the poll [which gets into a whole lot of psychology]. Further, pollsters commonly select even less randomness by asking “pre-qualification” questions to determine whether or not someone “qualifies” to take the poll and those often explicitly limit those who would poll contrary to the desired results.

    3. Real world polls use biases of language and thus the pollsters biases. The marble counts were more accurate because the question of the marble’s color is not ambiguous and neither is the answer of it’s color, and the marble can’t lie about it’s color either. But, real life polls don’t work that way. A snippet of an example I put in the post so I don’t have to recreate such example:

    “This language includes the very pollsters biases. Such as “if the election were today would you vote for Obama, Romney, or undecided” (with one of those answers being required to be marked). A person can’t therefore give a true answer of “I won’t vote” or “I don’t know” (which is significantly different than “undecided”), “neither, I will write in another name,” or perhaps the even more vague “if the unemployment drops to 7% by November, I vote for Obama, if it remains high I vote for Romney.” But, the polls won’t allow the more truthful answers, and therefore the answers they get are not accurate.”

    4. There political polling especially there isn’t a known population. Unlike the marbles where they knew exactly how many marbles were in the container, political polling doesn’t know how many people will vote, let alone who is actually even able to vote.

    For instance, I was asked in one such poll if who I would vote for in the primary, but in my state the primary (the one they were asking about) is “closed” so that I wasn’t even allowed to vote in it. Yet they were polling me “randomly” to see how I would vote in an election that I wasn’t even allowed to vote in. My answers if I had given any would have skewed the results.

    Reply
    1. Nathan T

      Oh and 100 points to whomever can spot the two “biases” in the seemingly unbiased question “if the election were today would you vote for Obama, Romney, or undecided”

      Reply
  6. Scott S

    Thank you for the article. I believe now on the eve of this election that any poll that does not include exclusive cell phone users to be inherently inaccurate. I strongly believe there are factors that differentiate people that still have land lines from those of us that gave them up years ago. The greatest being age, I literally know of NO ONE under the age of let say 40 living on their own that even has a land line. Over the past 2 years most my friends in their 40’s and 50’s have also dumped landlines, they do not need the useless expense or hassle of checking yet another message system. Yet all of our parents still seem to cling to them.

    I know there are problems with calling cell phones the most obvious being area codes may no longer have meaning (my personal phone number I have had since 1998 which was 3 moves and 2 states ago). It doesn’t matter, difficulty in gathering good data is NO EXCUSE FOR ACCEPTING BAD data.

    I will leave it to others to determine what it means, but I strongly suspect that younger voters are substantially underrepresented in such polls, and that this will become even more prevalent in future elections as the trend to no land lines accelerates.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

About

The CITP Blog is hosted by Princeton University’s Center for Information Technology Policy, a research center that studies digital technologies in public life. Here you’ll find comment and analysis from the digital frontier, written by the Center’s faculty, students, and friends.

Categories