Tortured Polling Logic
By Eastan McNeal on May 1, 2009 at 10:00 PM in Christianity, Current Affairs, Religion
On CNN today we see: Survey: Support for terror suspect torture differs among the faithful.
I suspect there may be some pushing involved in that Pew poll, as there may be a coordinated push to break down the cohesiveness of what the democrat party believes is the foundation of the republican party, White evangelical Protestants.
Years ago I worked with Dr. Marvin Kottke, a research professor at the University of Connecticut. Dr. Kottke adopted a couple of booklets I had written on survey statistical sampling methods and analysis presentation for his classes. We all understood the importance of how you select who you survey and how you phrase and order the questions. From him we learned the importance of preventing your desire for a pre-conceived outcome from influencing any of the process going into the survey, collecting the sample and analyzing the results.
Kottke lobbied the Council of American Research Organizations (CASRO) to include “in the box” guidance for researchers. He believed that no study should be released without a boxed description explaining how the survey / poll was conducted. You have seen them. Xyz research interviewed nnn people between this date and that date and the margin of error is x percent with a confidence interval of y. More importantly, he argued, was to state WHO commissioned the study and WHY.
My booklets were written to accompany a computer program I had just released that allowed organizations to tabulate their own surveys. I feared that if people put garbage into my statistical analysis program that they would get garbage out or that they would misinterpret the raw data and present leading findings. If the user of my software was challenged on their methodology they could easily blame the new software so, to shield myself from that end, I offered a free education through these little books. What I did not realize or expect was that people were sometimes using my guides as instructions on how to pre and post doctor the polls to further their agenda.
I was attending a National Ski Areas Association conference in Nashville when I heard Dr. Kottke warn an audience: “Do not tell your researchers what you expect to find with the study you are hiring them to conduct.” No matter how honest we are we fear our desire to satisfy the client, and we don’t really have accurate formulas for calculating out of your quantitative results, our qualitative bias.
I am seeing more and more surveys released without the “box” of information telling me about the research methodology. A more troubling omission is the who and why. Who paid Pew to conduct a study on torture that included questions about religious faith? Why did the client do that? What were they expecting to find?
If the survey would have shown that black agnostic eggheads supported torture more, would the survey results have been released?
That, by the way, is a good question. Most surveys are buried because the results did not support the client’s desired narrative. If the survey showed that religious and non-religious people felt the same way then who gave the researchers the suggestion to break out whites from blacks? And if that did not create a measurable enough difference, why was the evangelical group broken out, from the religious breakout, and why was that the only group breakout published by CNN?
Without answers to the above questions, I doubt the sincerity of the poll, the pollsters, the media that would publish the selected results and, most certainly, the mysterious client who commissioned the poll.
By the way, the information that should have been in the box:
Data from a Pew Research Center survey conducted April 14-21, 2009, under the direction of Princeton Survey Research Associates, among 742 American adults. Other religious groups are not reported due to small sample sizes.
Question wording: Do you think the use of torture against suspected terrorists in order to gain important information can often be justified, sometimes be justified, rarely be justified, or never be justified?
Pew
Pew’s own analysis of this poll showed that of the 742 respondents 188 were republican. They state that a number that low dropped the margin of error to 8% from 4% for the total sample. Also, using Pew’s own data from other studies, the number of people in the U.S. claiming to be evangelical is 19%, including black evangelicals. So the sample in this poll should have included at most 140 evangelical respondents and the margin of error should be stated as a number greater than 8%. These are assumptions because some of these sub-numbers are not published. The charts released indicate they surveyed 174 WEPs
The press is being biased and misleading in their treatment of the study. The above link goes to US News with a headline: Most Evangelicals and Catholics Condone Torture in Some Instances, when the headline actually could have read: Americans Split on Torture. According to the raw numbers 49% of the TOTAL sample says that torture can often or sometimes be justified. 47% say that torture can rarely or never be justified.
What was the intended goal when these cross-tabs were created and released to the public? Was this pure research or do some of you Christians out there now feel less likely to publicly embrace your religion?
(Release: November 13)
on sale for only $16.95.
has a spatula.)
8% Off at $11.95, with a rare five-star ranking by 294 customers: 
And check out these: 
("This crazy redhead is on to something. Her pink Rambler story alone is worth the price of the book," writes 
- “Richard Sale’s reportage on Washington, the CIA, the American armed forces, and foreign intrigue has created a perfect background for this account of the steady growth of President Clinton as a master of the game of nations.”--W. Patrick Lang, former head of Middle East Operations, DIA
"Anne C. Heller's stunning biography of Ayn Rand" uses a "plethora of newly unearthed research and fresh insight."
Experimenter Bias — E-bias
We were taught that we must be aware of E-bias — this was so strongly taught that I always check to see who (or what) paid for the research and try to determine the bias of the researchers. So often I see that the researchers didn’t make an effort to compensate for E-bias.
Perhaps this isn’t taught — perhaps polling is just plain manipulation.
Polling has an evil side effect — forcing individuals to make a decision before they have enough information to make an informed decision.
Spam monster ate my comment.
Thanks Eastan. Very interesting Post. Enjoyed it.
Your right, there doesn’t seem to be many of those box descriptions these days with polls or any research. I guess with all those misleading statements flying about, adding a few limiting facts might confuse the story they want tell.
And yes, its alway good to know who is paying for the poll. Because they will expect value for their money.
Interesting article indeed. Thanks for posting it.
I think it is possible that the motivation was to further stimatize conservative Christians.
On the other hand, Pew is a highly respected organization, so I would be disppointed if it got itself involved in any purposeful biasing.
I no longer trust polls, regardless of their source. Statistics lie and liars use statistics.
Informative article, Eastan. A good reminder, too: caveat emptor.
Well, that’s pretty scary. But polls and questions are not always clear, as we know. And…like the question asked, DOES NOT MAKE IT RIGHT, but still “if they are terrorists, is torture justified”. THEY DON;T KNOW THEY’RE TERRORISTS for ONE, it never will be limited to just “KNOWN” terrorists, and even if you could, it doesn’t make it right, good or smart.
ugh
Most Christians who responded in support probably see torture as legitimate force in national self-defense. To them, it’s an extension of capital
punishment, gun rights, or even domestic ‘
discipline’violence on the individual level.They don’t come with the background we have here to appreciate that when an enemy is believed to have information which, if extracted, would save innocent lives, then extreme measures are actually counterproductive for the greater good.
Until Pew uses context as an informational tool in their questioning to counter ultilitarian claims that torture is legal and has enhanced our safety, it is a push poll imho. Someone has a hidden agenda/motive against religious people, particularly towards Evangelical and Catholic Christians.
I will be one of the first to comment on my own post because I am still not sure about what may be going on with the people, press and politics – the Pew mantra.
I, too, find it hard to believe Pew would skew. But their clients may not be as pure at heart as we expect statistical scientists to be.
I have spoken to people of faith who believe that more than the cute “God is Dead” conversation we had in the 1960s today’s media seems to be attempting, in concert with Obama, to not just declare religion dead in America but to drive the nails in the coffin with their own pens.
I had not thought much about this subject until today’s poll release. I did some light research and found that, unfortunately, that theory may be supported by the sheer number of news stories and the quotes by Obama that may indicate that somebody out there has an agenda to put religion in a dark black box.
Somebody has this dark confused idea that conservative christian thinking = republican and, therefore, converting people into liberal agnostics should be a democrat party objective in their goal to hold power.
My thought is free thinking has no politics or religion. One’s beliefs should may be held up to critique, but the holder should never be condemned. I am seeing a lot of messenger shooting these days.
Eastan said:
“Somebody has this dark confused idea that conservative christian thinking = republican and, therefore, converting people into liberal agnostics should be a democrat party objective in their goal to hold power.”
Chilling. And I fear you may be onto something. I say this as a lifelong Dem, who no longer recognizes the party I grew up in.
There’s certainly a constituency within the Democratic Party (not Democrat Party, Eastan. That’s a silly slur perpetrated by some Republicans.) that pursues what we might call a “progressive agnostic” agenda. But there was also concerted effort in 2008 to broaden the party’s support among religious Americans. To my great delight, it succeeded.
Why is BO continuing — and funding — GWB’s Faith Based policies?
Read my comment below about O’s attempt to blur distinctions between religions. He knows most Americans have short attention spans. You got upset about the O’s continuing faith-based funding and connected it to the religious right. But did you blink or stop to think when Mr. Praise Jesus claimed to be Muslim and then held a Seder?
O plays a game of using a single moment to placate a certain single-issue segment of society and then using another single moment to placate a segment of society that may be opposed to the earlier one. There is NO cohesiveness between his various policies other than keeping the POWER of the Party (thus O) by manipulating the masses.
People should start building charts and keeping score to get the REAL O and the real Democratic Party nowadays. You have got to take them on their WHOLE “Weltanschauung,” not just by how he and they react to one’s single-issue mentality at any point in time.
should have been “may and should”
When I taught college freshmen the basic research class, I didn’t have expertise and didn’t have time to go into anything about research statistics or research methods. However, I always made it clear that one should consider research bias, to consider the source–for the fear of putting too much faith in research conducted with a priori reasoning.
I never participate in telephone polls. I agreed a few times and became so disgusted with the questions because I could tell that my answers might be interpreted a different way from the way I would want them interpreted, if only because the researcher could take one answer and combine it with another in a way I wouldn’t agree with. I also became suspicious because the people could give me no information really about who and why the poll was being taken.
Why would a Christian who is really a Christian condone torture? It makes no sense.
But then, how many times have I sat in meetings at church (I was an Elder at one time) and heard supposedly Christian people say clearly un-Christian things. And how many times have I also seen wonderful behavior from church goers. There are Christians and there are Christians. It’s like my Turkish daughter-in-law who considers herself Muslim though has never practiced religion. She claims to be but then makes clear she is “not religious.” There are people who claim to be Christian but who are also not religious. I also know people whom I consider very religious but who attend no church and claim no church affiliation. So making the leap to attribute a political belief to a person’s stated faith also makes no sense. It’s stereotyping.
I have simply come to the conclusion that I can never really put much faith in statistics behind the polls being reported by the popular press.
Eastan, without even examining the percentages provided, I believe you are absolutely correct in this suspician. I believe the media is showing its bias. And thank you for confirming my belief that many, many pols are conducted with predetermined results in mind.
Diana
Thank you for the time you gave to young learners. Teaching is, in my opinion, the highest valued profession.
I have worked in every aspect of polling. Most people have no idea how flawed a publishing can be if any and all of the factors are not protected by sincere intellectual honesty.
I hope you still teach.
Eastan,
I am retired–the new breed of teachers and I are on different pages, so to speak.
You must have been typing your comment while I was typing mine.
I am totally suspicious of Obama and his attitude toward religion. Here’s a man who claimed to spend twenty years “praising Jesus.” Then he claims to be a Muslim while visiting the Middle East. Then he holds a Seder at the White House. What next: a Wiccan ceremony, a Rosicrucian ceremony, something Hindu? He is trying to blur all religions into one. So your comments about an attempt to push “liberal agnostics” rings very true to me.
It’s a quandary for me, who was raised in a very religious (but not fundamental) home. I do believe that religion can do much to raise up ethical and moral children–if only because the stories from every religion are so morally instructive. Yet I too feel frustrated with fundamentalists. But liberal agnosticism smacks too much of the hubris the Greeks warned about.
Dear, I think the only option we have is “to own selves be true.”
I just talked to my friend Allen, who is a Christian and heads
http://christiansforthemountains.org/
Well. He agrees with me. Torture is not a good thing. Now why are we putting up with this crap from the mainstream media that says Christians want to act like government agents, pulling thumbnails from HufPost writers? (oh. Sorry. I like that idea)
Tell people that you do not like being classified as a non Obama American, just because you know a carpenter’s son. Don’t stand for this torture from the EXTREME left.
Be an American and state your position. Most of us are in the middle. We don’t have to listen to the kooks who sit on the extreme ends of political philosophy as we know it.
Speaking of “pushing”:
I suspect there may be some pushing involved in that Pew poll, as there may be a coordinated push to break down the cohesiveness of what the democrat party believes is the foundation of the republican party, White evangelical Protestants.
Eastan, the party is the Democratic Party. People in that party are Democrats.
This opening paragraph completely nullifies any attempt at thoughtful insight.
I do not understand your comment.
I read it several times and can’t make any sense out of JozeAl’s comment either. What is his/her point?
[...] right here and try to figure out why these results are as they are (setting aside for the moment Eastan McNeal’s recent excellent post about the survey’s methodology), the mind runs happily amok with what [...]
Excellent post. I don’t see how they can generalize from such a small sample and as was said the groups who opposed torture and those who tolerated it were almost the same size.
They don’t even generalize properly. When these agenda movers make their mental giant sanctifications, they generally reveal exactly what they’ve worked so hard prior to reveal. It usually works, too, since nearly everyone pays attention to their latest spin, and argues over that.
Case in point: 75% of the total in the survey has condoned torture, claiming it is justified obviously, because they believe it works.
There it is - the “geniuses” cleverly include “rarely” in the “NO!” column somehow.
But the “horrifying truth” they’ve worked so hard to hide for so many months and years is exposed in their antics.
SEVENTY FIVE PERCENT SAY TORTURE IS JUSTIFIED, THAT THEY THEREFORE BELIEVE IT WORKS, AND THAT WE SHOULD OBVIOUSLY MAKE USE OF IT WHEN NECCESSARY.
75%
3/4ths of the populace
.
See, they often have this problem that when they are scheming their little mind manipulation blame game, they let the other cat out of the bag, the GIANT CAT, that they had caged and hidden for so long.
Isn’t that wonderful - 75% for torture.
Glad they told us, finally.
Hmmmm. Were Jews surveyed? Muslims? Budhists? Why were Christians singled out? If any of the first three groups overwhelimingly favored torture (which I doubt), would any press outlet print it? Probably not, because then they would be accused of being “racist.” What a sad world.
Ok, never justified is only 25%, no matter what the religion or lack thereof.
I guess that settles it - 3/4ths of America believes in , 75%, enough for even a treaty based upon it, that torture is justified.
So what’s all this stuff about we shouldn’t torture ?
The Pew poll says 75% of Americans say do it.
Case closed.
Now, we’re doing the opposite because ?