ARG poll raises eyebrows — but is it reliable?

December 27, 2007

On Monday, the American Research Group released a very surprising new poll out of Iowa, which has caused quite a stir in the political world this week. The ARG results showed the Democratic presidential race shaping up this way:

Clinton 34% (last week 29%)
Edwards 20% (last week 18%)
Obama 19% (last week 25%)
Biden 8% (last week 8%)

The numbers for Republicans were also quirky, with John McCain doing far better than in other Iowa polls (he was third with 17%), and Fred Thompson doing far worse (a distant sixth with 3%).

But it was those Democratic results that really raised eyebrows. Some recent polls out of Iowa show Obama inching ahead of the field, though all of the top three candidates are within a few points of one another. And yet, ARG showed Clinton surging ahead with a whopping 15-point lead over Obama.

The results were released on Christmas Eve, which is generally a slow news day, but the ARG poll nevertheless drew some attention. CNN cited the data to argue that Clinton “appears to be breaking away from the pack.” Drudge gave the poll quite a bit of play yesterday, so politicos who might have missed it over the holiday quickly caught up with the news.

At first blush, I thought the poll might just be an outlier to be taken with a grain of salt, pending additional post-Christmas results. As it turns out, while the ARG poll is certainly dubious, there’s more to it than that.

The problem has to do with who pollsters can find via phone around the holidays. Pollster.com’s Mark Blumenthal noted that, historically, polling outlets didn’t even try to do surveys between Christmas and the New Year (indeed, Blumenthal said, “All of the polling firms I worked with rarely fielded surveys in the second half of December and typically shut down altogether between Christmas and New Years.”)

Of course, that was before the absurd race between early-voting states became comical, and the all-important Iowa caucuses were moved up to Jan. 3. Because the contest will have sweeping influence on who wins the parties’ nominations (whether it should or not), everyone is looking for polling data. As a result, pollsters have surveys in the field, even though it’s very difficult to get a balanced, representative sample.

And who are pollsters missing?

Three years ago, in a survey concluded a full week before the holidays (12/15-17/2004), the Gallup Organization asked a national sample of 1,002 adults whether they planned “to travel more than fifty miles from home this holiday season.” Twenty-eight percent (28%) said “yes.” More important, as the table below indicates (based on data drawn from the Roper Archives), those planning holiday travel had a very distinctive demographic profile. Holiday travelers were much more likely to be younger and better educated. Notice also that holiday travelers were not just college students. Adults between the ages of 30 and 44 are twice as likely to travel for the holidays than those over 65. […]

All of this brings us to the survey that the American Research Group released on Monday fielded between Thursday December 20 through Sunday December 23, a survey that shows Clinton gaining and Obama falling. Some will read this post as an attempt to debunk that result, and the findings above certainly argue for considerable caution in reading results from any survey this week. But the problem in trying to assess the ARG poll is that we know so little about it. Does ARG make call-backs to unavailable respondents? What was the sample composition on any ARG Iowa survey this year in terms of age and education level, and was this one suddenly different? Did ARG weight the results by age or education this time, and if so, by how much? We are in the dark on all of these questions.

It is also worth remembering, as some commenters noted yesterday, that real changes may be occurring in vote preference this week even if surveys may be severely challenged in their ability to measure it. Clinton may be gaining and Obama falling. So it is quite a leap for anyone to say they know conclusively that the ARG result is either right or wrong.

The same is probably true of almost any poll conducted this week. In other words, when it comes to poll results over the next week, caveat emptor.

ARGgggggh!

  • Pakistan opposition leader Benazir Bhutto was assassinated today. And Pakistan is our friend?! Arrrrrrgh!

  • Nine days from now I hope the headlines read:
    “A small number of Iowans have voted, now let’s see what America wants”.

    I’d be a lot more comfortable about the Iowa caucuses if there weren’t so many stories out about how they shape their votes to effect the rest of the primary season. With that distortion in place, I frankly could not care less about how Iowa or New Hampshire votes.

  • for those of us who watched with glee the initial exit poll results of the 2004 presidential election coming out strongly in favor of kerry (only to subsequently witness a roaring bear market), wisdom comes in the form of taking all polls with a grain of salt.

    as the old saying goes, fool me once shame on you, fool me twice shame on me (or as the winner of said election would say, “Fool me once, shame on — shame on you. Fool me — you can’t get fooled again.”)

  • I wonder how the early date is going to affect the college vote. My college is closed until mid-January which means that I would not have been able to attend the caucus. I really don’t know how big a block the college kids normally are but it will be far smaller than normal this year.

    I also wonder if the BCS Bowl Game on Thursday night will have any significant affect.

    However, the intrade.com is one of the few places where ALL of these factors are taking into consideration. I am really surprised that so few people are paying attention to one of the best indicators about the caucus and the election

  • These polls are making me crazy – I’m just sick of them and sick of how they do nothing but add to the horse-race coverage that has nothing to do with issues. I’m also leery of pre-election polls, because I do think people are prone to saying one thing and doing another. Exit polls, on the other hand, have traditionally been more reliable, because you are talking to people who aren’t just likely to vote or will possibly vote, but have actually just voted. The 2004 exit polls should have been a red flag for election fraud, as Kerry was clearly well ahead according to those polls.

    Anyway, that’s history that changed history, and there’s nothing to be done about it.

    I’ve also decided that the days of Iowa and New Hampshire having a lock on the first caucus/primary should be over, and some other system or schedule needs to be instituted, whether it is to rotate the order or do them on a regional basis – I don’t know.

  • Anne, it really doesn’t matter who it is that goes first – no two states are truly representative of the whole, and first will always be more newsworthy. That people are sick of it this year has little to do with the identities of the two states.

    I was polled last night by Zogby. The young, obviously out-of-state caller seemed to have no clue what it is like on the ground here. One of the questions was how many telephone polls I had been asked to participate in this cycle. When I said he was the fifth, he sounded like he thought I misunderstood the question, and kept asking “five?”

    Then he asked how many calls directly from candidate campaigns I had received. I said there is no way I could count them. He said “more than 6?” – which must have been the top of his scale. I told him we get 2-4 calls from candidates, committees, or representatives per day. Again he sounded shocked. I didn’t bother to tell him we had 8 pieces of caucus-related direct mail just yesterday.

    One thing I suspect happens about now is that some Iowans, out of frustration, lack of maturity, or both, get so bored and annoyed with the repeated calls that they start screwing around and not giving honest answers. Or, like my wife, they just refuse to take any more calls – which skews the representativeness of the sample.

  • Zeitgeist – I don’t envy you the onslaught of calls and mail you’re getting, and I can see how tempting it would be to have a little fun with the polling questions…

    If it doesn’t matter which state is first, why not change it up from cycle to cycle? Give the voters in other states what Iowans and New Hampshire-ites have had for years – access to the candidates. I can’t tell you how much I would appreciate being able to see all of them, ask them questions, engage in a more direct way. The TV candidate or the newspaper candidate or even the blog candidate is not the same thing as the up-close-and-in-the-flesh candidate – there’s a reality there that is missing for most people, and may explain why some people just think it doesn’t matter if they vote.

    In truth, it is because this kind of campaigning is so limited that we end up having to depend on the media to bring us closer to the candidates, and unfortunately, they are doing a really crappy job of it, with no signs of that changing anytime soon.

    I may just be getting antsy about the results from Iowa and NH, and what they are going to mean. I fear the inevitability of any candidacy, think Obama may not do as well as the polls indicate, and hope Edwards gets a big boost that will carry him on to the next phase.

  • There will be a very reliable poll in Iowa on January 3rd. Has anyone done a retro look at polls to see how accurate they have been in past elections? It is my distinct impression that nobody knows nothing from polls.

  • If I ever became dictator of a state I would pass a law that my state would have a primary the EXACT same day as the Iowa caucus.

    I loved being in Iowa and meeting all the candidates. I just think it is absurd that Iowa has about as much influence on the nominations as the other 48 states combined. (Yes, I am excluding NH)

    Why should Iowa and New Hampshire get all the power?

    It is wrong and it should be stopped.

  • Anne, you make a very important point. Like the Uncertainty Principle in physics writ large, our media measurements interfere with, even create, the “reality” they supposedly measure and report. Competent national, state and local political analysts used to give us the benefit of their knowledge, acquired “inside”, through contacts. Their journalistic reports didn’t interfere with the process so much as they revealed it.

    With the demise of those insiders and the news agencies they used to work with and for, an army of political pollsters and analysts pretend to knowledge they simply don’t have. If you ask most people a question they’ll answer it. Whether they ever thought about before the questioner showed up, or whether they give the questioner the answer they think is expected (or, just to be perverse, the opposite of that). Having nothing else to rely on, the public takes all those measurements (mostly of name recognition) as reality. And, real or not, it becomes the reality which influences still further polls and analyses.

    The voting public has issues it wants dealt with (war, health care, economic dark clouds). We nominally have a free and open democratic system. The expense of buying time on corporate-owned TeeVee distorts that system. But so also does this media-created reality.

    We could do something about the expense (take bake the media or force them run political programs at reduce or no cost, public financing of campaigns). We could also do something about the media circus called the “campaign season” — shorten it to the attention span of the unorganized American voter: like the World Series, about one week.

    Will we? Not while there’s a corporate buck to be made providing nothing but nonsense 24/7.

  • “The expense of buying media time” complained of by Ed (and others) is precisely why starting in small states without a major media market makes sense. If California went first, it would not be at all like Iowa with massive amounts of organizing and face-to-face small group meetings with real people. It would be the best-known, best-funded candidates killing the upstarts by buying prohibitive quantities of SD/LA/SF media. First because the population of California is such that media is really the only way to reach everyone, and second because the cost of large market media means only the “have’s” can afford to use it, so it becomes a barrier.

    In Iowa, the population is small enough that it is efficient to actually campaign in ways other than media, and the media is cheap enough that even Biden and Dodd are all over the airwaves. This would only be true in about half of the states, and no matter which among those were chosen to go first, the big states would complain about the “disproportionate impact” of a small state going first. But I happen to think there is real value in having campaigning start on this very human, door-to-door and town meeting type of scale. Someone, whether Iowans, Vermonters, Oklahomans, Montanans or whoever should be requiring these candidates to face the voters directly, in small groups, and talk to them face to face.

    And Lance, I have no idea where all of these stories are you refer to about Iowan’s using their caucus vote to “manipulate” subsequent state choices. I’ve never seen such a story, nor would I be inclined to believe it having worked in politics in Iowa for 25 years. First it is too difficult to do, second what is the difference really between voting for who I want – knowing it will impact subsequent states – and “manipulating” those states?

  • Sorry you haven’t seen them Zeit, but I have.

    I can see a process of starting with small states, then medium, then large. Give them three days, the first Tuesday of May, June and July to set their dates, and no others. There is no reason for Iowa to be first. We don’t need the filtering process to go through there. Let’s have some competition for God’s sake.

    Let’s have some candidates who DON’T think ethenol is a good thing.

  • How come the Brits can do it in two weeks? Technically the next election is held 17 working days after the date of the Proclamation dissolving the old Parliament.

    That declaration can be sprung at any time. By contrast we always know when the next election will be. Seriously, two weeks of campaigning? I think the public would buy that. Trouble is the professional pols, pollsters and TeeVee people (performers and ad salesmen alike) would be declared largely irrelevant … which they ought to be.

    “O, to be in England”

  • I can see a process of starting with small states, then medium, then large.

    I believe it was suggested at this blog quite a while ago that the order of the states in the primaries be based on the results of the last general election: the states with the closest result (e.g in 2004 ohio, new mexico, virginia (?), florida, missouri, etc.) are the first to hold a primary or caucus.

    That seems like a much better approach than our current process and yet still would avoid the problems associated with the states with large media markets and the establishment candidates. Not that I think this change will happen any time soon (if ever).

    thoughts?

  • Ed Stephan,

    How come the Brits can do it in two weeks?

    Sure, but they don’t actually get to vote for the Prime Minister. I’m not sure I want to go back to allowing the parties to select their nominees. As problematic as our system is, I think direct voting in the primaries and the general election is preferable to the British system.

  • Premise media is under the impression CBR readers are interested in an Intelligent Design propaganda vehicle? (ad in left margin)

    I like me some Ben Stein but I want this movie to flop big time.

    B.S. should be proven a limited marketability product outside the Bible belt.

  • Sure, but they [the Brits] don’t actually get to vote for the Prime Minister. I’m not sure I want to go back to allowing the parties to select their nominees. As problematic as our system is, I think direct voting in the primaries and the general election is preferable to the British system.

    Interesting idea. Now that you mention it, I tend to come down on the opposite side of the question. Given how such a huge number of people vote for a president based on his hair, his wife’s looks, the shininess of his halo, how much of a bully he seems, etc, at this point I’m ready for a system that doesn’t leave it up to the voters directly. On the other hand, given who our senators chose to lead them this time (Harry “Spineless” Reid), it looks like there’d be problems with that system too. Maybe a benevolent tyranny would work better. I volunteer!

  • Interesting take on the demographics available to answer the pollsters telephone call this late in December. Which brings up the question of cell phones. Do pollsters contact individual cell phones? I believe not as it is illegal to telemarket or survey to these devices. I would guess that the majority of young, educated voters use their cell phone exclusively. I believe Obama will win Iowa by better than six points. The momentum of this win, coupled with a win in New Hampshire, will be the catalyst needed to give this country new life, new liberty, new direction.

    • Comments are closed.