Pollsters: Pentagon's Policy Survey May Be 'Overkill'

WASHINGTON (July 13) -- Talk about TMI.

The Pentagon's confidential, 103-question survey e-mailed to 400,000 active-duty and reserve troops last week -- and leaked to the media two days later -- is aimed at gauging their views on lifting the military's "don't ask, don't tell" policy on gays in the service. But it's being viewed in social science circles as the statistical equivalent of the Powell Doctrine, and at least one senator raised concerns today that survey respondents might feel they're empowered to dictate policy decisions.

"It's overkill," said Frank Luntz, the Republican pollster and focus group guru. "Everyone accepts a nationwide poll of 1,500 people as being representative of America," he said, noting the number of respondents typically surveyed by Gallup and other organizations. He said the views of the 2 million members of the armed forces could have been discerned by polling a mere 10,000 troops instead of surveying a prodigious one in five. "This is like dropping a nuclear bomb on a cockroach when a little Raid would do."

Defense Secretary Robert Gates, a former CIA director whose degrees are in history, took ownership last week for the decision to double the size of the survey from an already statistically robust 200,000. "I wanted a significant percentage of the force to have an opportunity to offer their views on this," he told reporters, noting the questionnaire had "been designed in partnership with a professional survey company and according to the best practices that they have for that industry."

The company, Westat, is based in suburban Rockville, Md., and has extensive experience conducting government surveys. It did not return a request from AOL News for comment.

"Westat is a well-respected research company with a long track record of doing government surveys," said Scott Keeter, director of survey research at the Pew Research Center. "They have very high standards, and they wouldn't take part in a study that was not designed with the best possible good-survey practices."

Westat's $4.5 million Pentagon contract covers more than the online survey. It is managing all facets of a nearly yearlong information gathering campaign that will help leaders implement the commander in chief's call to lift the policy that keeps gay service members in the closet. The fact-finding includes:

• More than 33,000 comments submitted so far in an online suggestion box.
• Information sessions of 250 to 300 people each at more than 30 military installations followed by smaller, voluntary focus groups.
• A separate survey, to be sent in early August, to 150,000 families of service members -- 70,000 to those of active-duty troops and 80,000 of Guard and Reserve families.

"There is no obvious statistical reason why it's necessary to do that many interviews," said Geoffrey Garin, president of Hart Research Associates, one of the country's top survey research firms, and a longtime Democratic pollster. "It does have the benefit of giving more people the feeling they've had a chance to say their peace and feel consulted. There's a benefit in that. Part of the way you create buy-in is you let people feel they've had a chance to participate and express their point of view."

But Sen. Carl Levin, chairman of the Senate Armed Services Committee, said today that the Pentagon may be sending the wrong signal by asking so many service members their views on an issue that will be decided above their pay grade.

Levin told reporters at a breakfast briefing he worries that such surveys could be construed as having some kind of "veto power" when it comes to military policy. "It would be really, really unacceptable that the greater use of surveys might lead people in the military to think it is a democracy,'' he said. "The military is not a democracy."

Levin, who noted that service members were not polled when President Truman ended racial segregation or when women were integrated into more roles in the military, said, "I can understand the resentment in the gay community" about putting the question to an opinion poll -- and one, he said, that included no questions about discrimination against gay and lesbian troops or how the discharge of more than 13,500 service members since 1994 has affected unit effectiveness.

The survey also doesn't ask participants if they themselves are gay, only whether they have family, friends or acquaintances who are.

"If it turned out that military personnel who are very angry about the proposed repeal were to participate at much higher levels than those indifferent or supportive of repeal, the picture the Pentagon gets back will be of a force more angry about repeal than it really truly is," Keeter of the Pew center said.

But like other social scientists, Keeter's main concern was the scope of the survey.

The Census Bureau's ongoing American Community Survey, which has replaced the decennial long form, goes to 250,000 people a month, or 3 million a year, in an effort to learn about the lives of 310 million Americans.

When the Pew Research Center's Forum on Religious and Public Life conducted its Religious Landscape Survey in 2007, it interviewed more than 35,000 people out of a population then of a little more than 300 million.

Keeter said that survey was larger than most that Pew conducts because of the need to include enough religious minorities, including Muslims and atheists, to make the results statistically valid. "It's expensive to increase your sample size," he said. "How much are you spending for what level of improvement? If the margin of error is plus or minus 3percentage points -- we can increase the sample size to 2 or 1 percent, but what do we gain from that added precision?"

Luntz says not much, even as he admitted to a huge case of poll envy.

"I'm jealous and angry I didn't get a crack at that. This is so unfair -- and I'm pro-military," he said of the mega-sized survey. "Give Secretary Gates my home number so next time he wants to do a survey he calls me. What the hell am I? Sliced bread?"