The Case of Louis Harris and Associates

The most important principle underlying democracy is that the majority should rule. But until relatively recently, Americans have been poorly equipped to communicate their wishes to elected representatives. The principal means for doing so has always been elections. But elections occur relatively infrequently, and they provide no means for citizens to indicate precisely which of a winning candidate’s stands they like and which they dislike. Especially passionate individuals have always expressed their views directly to their representatives, but these people convey the wishes of only a small segment of constituents.

The development of widespread national public opinion polling in the 1940’s was therefore a major step forward for the citizens of the United States. Opinion surveys have made it possible for the electorate to express detailed demands on policy matters to elected representatives in Washington. And when national crises occur, we can swiftly send pointed messages to politicians about what we want done. As a result, public opinion polls are tremendously valuable assets in the conduct of contemporary American democracy.

But as valuable as public opinion polls are, they are also easily misinterpreted. Public sentiment on most controversial issues is highly textured and conflicted. It is therefore difficult to describe the American electorate’s views on issues simply in terms of the proportions of people who favor and oppose a broad public policy such as legalized abortion. As is true for many issues, public opinion on abortion varies depending upon many details, including who the mother is, what the circumstances of the pregnancy are, and who would pay for and perform it.

Because public opinion is often so sensitive to detail, a slight change in the way a survey question is asked can sometimes greatly alter people’s answers. For example, consider the following question:

Which of these do you think is the main cause now holding back greater prosperity in this country: (a) Business Leaders, (b) The New Deal, or (c) Labor?

In a survey conducted in the early 1940’s, “Business Leaders” were selected as the chief obstruction to prosperity. However, when the third response choice in the question was reworded to read “Labor Unions” in a simultaneous survey of a comparable sample of respondents, answers changed significantly. In this case, “Labor Unions” were selected as the most important obstruction. Apparently, people’s beliefs about labor in general were different from their beliefs about labor unions in particular.

Pollsters have learned an important lesson from this example and many others like it: whenever the results of a survey are reported, the exact wording of the questions should be reported as well. Even slightly rephrasing a question can sometimes seriously mislead readers about what respondents were asked. However, despite the fact that paraphrasing a question clearly compromises the honesty and accuracy of a poll report, doing so is not uncommon. The culprits are sometimes journalists, who change wordings to make articles read more smoothly. But in one recent case, the culprit was a survey firm that conducts many important polls: Louis Harris and Associates.

During September and October of 1986, Louis Harris and Associates conducted a national study of public attitudes toward the civil justice system and tort law reform. The survey was commissioned by Aetna Life and Casualty, an insurance company that is often involved in civil litigation and that favors changes in some regulations governing civil lawsuits. The poll’s questions addressed topics ranging from citizens’ perceptions of current problems with the civil justice system to citizens’ attitudes toward proposed changes in the system.

Some results of this study were reported in an article in The New York Times on March 7, 1987. The survey was characterized as indicating that although the public was satisfied with some aspects of the civil justice system, it was dissatisfied with others and favored many reforms. Most importantly, spokesmen from Aetna, the Tort Reform Institute, and the Insurance Information Institute were quoted as saying that the poll demonstrates “broad public support for changes in the civil justice system” and that the poll’s findings “reflected the public’s demand for reform” (emphasis added). The findings of this survey were characterized similarly in a press release produced by Aetna and in the Harris report of the survey’s findings.

These characterizations are misleading for two reasons. First, the questions actually asked respondents how acceptable they found each proposed change: very acceptable, somewhat acceptable, not too acceptable, or not at all acceptable. But instead of describing the percentages of respondents who found each proposed change “acceptable,” The New York Times and the Aetna press release descriptions claimed to report the proportions of respondents who favored, supported, agreed with, or would allow each of them. Similarly, the Harris report offered conclusions about the number of Americans who supported and favored the changes.

The proportion of people who reported a proposed change to be acceptable might be different from the proportion who would have said they “favored,” “supported,” “agreed with,” or “would allow it.” The word “acceptable” seems to be a relatively mild word in comparison to “favor” or “support,” so many more people may find a given proposed change acceptable than would favor or support it. Therefore, the use of these last terms may have led to the appearance of inappropriately high levels of public endorsement.

Perhaps even more questionable than this word substitution is the method used to calculate the proportions of people who supposedly “favored,” “supported,” “agreed with,” or “would allow” each change. In all reports and summaries, this was done by combining the proportions of respondents who said they found each proposed change “very acceptable” or “somewhat acceptable.” It seems clearly inappropriate to describe some legal changes as receiving “very strong support” when a large proportion of respondents actually said those changes were only “somewhat acceptable.”

In order to explore whether using the words “favor” or “support” might have produced lower levels of apparent public endorsement of the proposed changes in the civil justice system, my students and I recently conducted a study in a regional telephone survey. Residents of the suburbs of Columbus, Ohio, were contacted by telephone. One-third of these people, selected randomly, were asked the nine questions in the Harris survey that asked “how acceptable” respondents found each proposed change to be. Another third of our respondents, again selected randomly, were asked instead whether they would “strongly favor,” “somewhat favor,” “favor a little,” or “not favor at all” each proposed change. And the final third of respondents were asked whether they would “strongly support,” “somewhat support,”, “support a little,” or “not support at all” each proposed change.

The results of this experiment were in part consistent with our expectations and in part surprising. As expected, many more respondents said they found the proposed changes acceptable than said they supported them. Thus, “support” seems to be stronger than “find acceptable” in many people’s minds. However, about the same proportion of respondents said they favored each proposed change as said they found it acceptable, so “favor” and “find acceptable” are apparently equally strong and can reasonably be substituted for one another.

This example of a misleading poll summary by Harris is not an isolated incident. For example, Harris conducted a 1981 survey of American parents that was sponsored by General Mills, Inc. The Harris report of the results said at one point:

Many (over 7 in 10 family members) believe that “even when they don’t work, parents today don’t give their children the time and attention they need.”

However, the figure of “7 in 10” is not actually the proportion of respondents who said they believed the statement. Instead, this is the proportion of people who “agreed strongly” or “agreed somewhat” with the statement. The Harris summary of answers to this question again seems to overstate public agreement with the statement.

Consider another example. On May 19, 1989, Louis Harris published an article opposite the editorial page in The New York Times describing the results of a survey his firm conducted for the NAACP Legal Defense and Educational Fund. At one point, Harris said:

More than 9 in 10 Americans would support special school programs for underclass children beginning at age 8. These classes would be designed to motivate the kids to stay in school and to convince them that they can extricate themselves from poverty. . . .

However, the survey question did not ask respondents whether they would support such programs. Instead, it asked whether respondents would “favor or oppose” such programs. Thus, Harris again probably overstated public support for these special school programs. Furthermore, Harris’s paraphrasing of the question is very different from the description of the special programs that respondents actually heard:

Starting special school programs with young underclass children when they are 8 years of age, designed to increase their motivation to stay in school-and to arouse hope within them that they can lift themselves out of their miserable life situation. . . .

The last phrase of the question was reworded in Harris’s summary in a way that changed its tone considerably.

Rephrasing questions and bunching together answer categories is not necessarily the rule in Harris’s reports of their polls. Indeed, the majority of Harris reports I examined did accurately characterize what respondents were asked and what answers they gave. Furthermore, Harris routinely publishes the entire questionnaire from each of its surveys in the final report, along with complete statistical breakdowns of answers to each question, so any careful reader can directly examine the actual results of a Harris survey. Nonetheless, the cases described above illustrate some problems.

There are at least three possible explanations for Harris’s overstatement. The first is simply carelessness accompanied by a motivation to make reports interesting and easy to read. The second is a desire to maximize the degree to which a poll appears to support the interests of the client who sponsored it. And the third is a political agenda.

I suspect instead, however, that the motivation for Harris’s occasional misrepresentations is none of these but rather a fourth possibility: the desire to make opinion poll results seem more important and compelling than they sometimes are. Most social and political issues are the subjects of lengthy and vociferous debates precisely because there are sharp disagreements among people about how to resolve them—as with abortion and gun control, and in fact for most big issues. But to report that the public is deadlocked in disagreement on an important issue is hardly newsworthy. Therefore Harris may exaggerate the extent of agreement in the American public on controversial issues in order to enhance the apparent strength of the public’s will and thereby increase the apparent significance of his findings.

Public opinion polls are clearly tremendously valuable in helping the American public keep its government in check when a majority of citizens agree on how an issue should be handled. But unless reports of surveys accurately describe their findings, the American public and our elected representatives can hardly be expected to take these surveys seriously. Making poll results more credible is an important goal for both survey researchers and journalists.