The Random Yak

Your Results May Vary

Filed under: News Yaks — Random Yak @ 10:20 am on August 9, 2006

A “top news” headline at CNN this morningreads “Poll: 60 percent of Americans oppose Iraq War.”

Myreaction: how did they ask the question?

I clicked on the story, curious to see precisely what wording the survey used. What I found didn’t really surprise. (You guessed it: they didn’t tell us.)

The first sentence of the news report reads: “Sixty percent of Americans oppose the U.S.war in Iraq, the highest number since polling on the subject began with the commencement of the war in March 2003, according to poll results and trends released Wednesday.”

The second sentence destroys the reliability of the first by stating thatthe results came from apoll which surveyed 1,047 adult Americans by telephone.

One of the first – andmost important – lessonsa good statistics teacher offers first-time students is the following: Statistics can be made to say anything you want them to say – which is why anyone studying statistics cannot accept the resultswithout studying the underlying dataand method of interpretation.

If you take CNN’s headline at face value, a significant majority of Americans don’t approve of “the U.S. war in Iraq.” So what does that actually mean – and more to the point, whatconclusion is CNNoffering itto support? (Because, in case you hadn’t noticed,statisticsare generally used to prove or support a conclusion, to suggest a preferred course of action or to reflect some reality aside from the mere existence of a given fact.)

For help, let’s consult The Yak’s Big Book of Moonbat Logic (I know, logic and statistics, and it’s not even noon yet. Bear with me anyway.):

Moonbat Logical Axiom #1:Peoplemust determine what is right for themselves.

Moonbat Logical Axiom #7: If a majority of people determine something is wrong, then it is wrong.

Moonbat Logical Axiom #8: If Axiom #7 conflicts with Axiom #1, go to Axiom #9.

Moonbat Logical Axiom #9: If we agree with the majority, Axiom #7 prevails. If not, go to Axiom #1.

Ergo: (according toMoonbat Logic) if 60% of people oppose the war in Iraq then the war in Iraq is wrong. Which is, of course, the point CNN is trying to make. The people don’t like the war, so the war should end.

All very well and good. And for the moment (and the sake ofmy originalpoint), I’m willing to disregard the larger Truth that one does not measure “right” and “wrong” by popular opinion. (Leave that 900-lb gorilla in the corner for now. I promise we’ll bring him out to play with the luggage later.) Let’s start with the (mistaken) assumption that CNN is right: ifa majority of people oppose U.S. intervention in Iraq, U.S. troops shouldn’t be there.

We must then ask: Doesthe data provethe conclusion? To rephrase: can we rely upon the conclusions stated in this poll and can we safely extrapolate the views of U.S. residents as a whole from the opinions of the poll respondents?

CNN seems to think so. After all, they reported it as headline news. In fairness, the article does initially qualify the conclusions by stating “poll respondents said” but later on states that “Americans were nearly evenly split on whether the U.S. would win the war in Iraq.” At this point, CNN extrapolates the poll results to cover the opinions of “Americans” – not just poll respondents, demonstrating a willingness to believe that the poll results in question are fairly representative of – and capable of telling us truthful information about – the opinions of Americans as a whole.

Let’s take a closer look and see whether CNN’s reliance is reasonable.

The poll surveyed 1.047 “Americans” by telephone.

Unfortunately, the article doesn’t tell us whether these “Americans” were citizens, legal residents or illegal aliens. Giving maximum benefit of the doubt, let’s say they were all citizens.

We don’t know whether they were all adults and whether those adults were sane, sober, registered voters. We don’t know whether they read the news. Let’s assumesane, competent, well-informed individuals. (Yes, I knowI’m already testing your ability to suspend disbelief. Stay with me.)

We don’t know what region of the country they respondents lived in. (Yes, that makes a difference – do you want California’s opinion making policy for the rest of you? Do you think, for a moment, that the average Texan sees the conflict in Iraq the same way a New Yorker does? ‘Nuff said.) Still, let’s assume the company did its work properly and called people across the United States.

We don’t know how many were registered Republicans, registered Democrats, registered Independents, Green Party Members, Communists, Fascists, cat loversor members of other political parties. (Note: this is perhaps the most telling piece of missing information, because a poll which surveys registered Democrats is going to read distinctly differently than one which calls Republicans). Let’s assume the company called equal numbers of Republicans and Democrats, and one supporter of anything done, said or proposed by Che Guevara.

We don’t know how they asked the question, or even what specific question was asked. “Do you support the United States going to war in Iraq” is very different from “Do you support the military action the United States has taken in Iraq” or even “Do you believe the United States was right to intervene in Iraq and help to remove Saddam Hussein from power.” (I could rephrase at least ten other ways, butas I discussed with David of Third World County just last week – anyone who’s not gettingthe ideawith three examples probably won’t get it with ten.) Any good attorney knows that you can manipulate answers by the way you phrase the question. Good polling companies know this too. CNN’s failure to report the precise question asked makes it impossible for readers toproperly analyzethe results – but for the sake of argument (and getting to that original point I mentioned some time ago) let’s assume the question was phrased in a content-neutral, non-weighted manner. (Stop laughing. Just because it’s never happened before doesn’t mean it couldn’t have happened now…)

But even eliminating all of those factors, any one of which coulddiscredit the survey if not properly handled or accounted for, we still have a problem.

The poll surveyed 1.047 “Americans” by telephone.

The current population of the United States is approximately 299,432,245.Which means that the research company made assertions about “Americans” as a whole based upon a survey of approximately .00035% of the population.

See the problem?

(If you don’t, I propose a new method of electingthe President of the United States. My blogroll will vote, and whoever we decide upon gets four years in the oval office.)

CNN reportedthe survey as if it wasas indicative of American opinion generally. And most of CNN’s readers will accept that report as truth, permitting the stated statistic to color their worldview (and perhaps their own opinions about U.S. involvement in Iraq).

Behold the power of statistics.

I could tell you that 100% of the peopleI surveyed supported U.S. involvement in Iraq – and I wouldn’t be lying. But unless you knew what questions to ask you might not know that I only asked three people, all of whom go to my church, vote Republican and have children in the military.Facts whichimpactthe statistics. Note that these facts don’t render the survey unreliable per se – they just limit the uses for which the data may be properly used.

Same with CNN and the reported poll on Iraq. The poll may or may not be reliable, as far as it goes.Thearticle doesn’t give me enough information about the underlying facts and data set to permit me to draw a conclusion one way or another. It does, however, give me plenty of information to determine that I’m unwilling to rely upon it as an indicator of “American opinion.” The sample is just too small (especially without other supporting facts about the data set and the way the questions were phrased).

Does that mean the poll isn’t worth the paper it’s printed on? For the purpose CNN claims, absolutely. You cannot make sound judgments about “American opinion” based upon 1,047 voices of unknown age, origin and political leaning – certainly not sound enough to support an editorial piece about a subject as serious as war. CNN could have bolstered the strength of its data by stating more information about the geographical andpoliticalspread of respondents, but did not do so (and we don’t know whether that information was even available to CNN, though if it wasn’t, it’s doubly irresponsible to rely on the study and report it as “news.”) Used as part of a larger or differently-focused article, the poll could have mentioned that “some Americans” hold these views – though that’s certainly not as punchy as the “60 Percent” headline.

CNN and other news agencies will continue to report flawed and manipulated statistics which don’t actually prove the conclusions they claim.The mission of individuals seeking to learn the truth, should they choose to accept it, is not to take the statistics at face value. To ask basic questions about the reliability of the underlying data and to interpret the conclusions for themselves.

Not that this is news. (You certainly won’t see it on CNN.) It forms but a small part of the larger search for absolute Truth and understanding which many people refuse to accept or undertake. On a day-to-day basis, it often seems easier to accept the view that “every man must decide what is right in his own eyes” and that majority rule means the majority is always right – as long as that majority follows the principle that “the individual is job #1.” The problem, of course, is that it doesn’t take long for these principles to conflict.And thestereotypical Moonbat response to this conflict is to deny the majority – along with a whole cartload oftruth and reality.

Case in point:Determining “right”and “wrong” andthe proper reactions to them.(Here comes the gorilla. Hold on to your Samsonite.) “Right” and “Wrong”cannotnecessarily bedetermined by majority rule. Some things are right (and others wrong) regardless ofwhether or not you have consensusin the room at the time. Both the Bible and Natural Law areclear on this point – but this post has already run long enough and I’ll defer further discussion on that topic for another day.

For now, I’ll be pleased if you just payattention to the statistics.

(Closing note: Statistically, 87% of you have noticed that I frequentlypost statistics about reader interpretation and understanding. Statistically, 23% of you already understood that doing so represents – and has always represented – a stab at the misinterpretation of statistics by the mainstream media. I think that number will go up now.)

Linked to the “Photoshopping for Reuters” post at Blue Star Chronicles and the Wednesday OTA/Fair Tax Blogburst at Third World County.

1 Comment

  1. Links and Minifeatures 08 17 Thursday…

    RINO Sightings

    Carnival of T……

    Trackback by Searchlight Crusade — August 17, 2006 @ 5:39 pm

RSS feed for comments on this post.

Sorry, the comment form is closed at this time.

Site Meter