Lying with Numbers

July 10, 1994  ·  Michael Fumento  ·  The Washington Post  ·  Statistics

In an increasingly skeptical age, there’s one tool that continues to wow ’em: scientific data or data that’s been scientifically collected.

Wall Street Journal reporter Cynthia Crossen argues that there is indeed such a thing as objective truth, but that when the bottom line comes into play, such "scientific" studies may be the last place to find it.

Consider the oat bran mania, which was based originally on a study reflecting all of a 3.3 percent — statistically insignificant — drop in cholesterol among persons who had oatmeal or oat bran added to their diet. The grain that Dr. Johnson’s dictionary claimed was fit only for horses and Scotsmen became "the next miracle food," as one newspaper put it.

Indeed, the Quaker Oats Company aggressively pushed that line. Next thing we knew, oat bran had beenadded to more than 300 products, including potato chips, toothpaste, and licorice.

When another major study appeared that found no special cholesterol-lowering property in oat bran, Quaker Oats did practically everything to destroy the researchers’ credibility short of catapulting their spokesperson at them.

Food dis- and misinformation is bad enough, but what about pharmaceuticals?

Crossen notes that a researcher reviewed 107 published studies comparing a new drug and a traditional therapy and found "studies of new drugs sponsored by drug companies were more likely to favor those drugs than studies supported by noncommercial entities. In not a single case was a drug or treatment manufactured by the sponsoring company found inferior to another company’s product."

In all fairness, a company won’t subject a drug to a study that may be publicly reported if it doesn’t have confidence in that product. But these data underlie a powerful case. Crossen is on the mark when she writes, "Despite the many temptations modern scientists face, they still believe their discipline is self-correcting because it is open, verifiable and subject to close review by scientific peers. In American medicine, however, all three of these pillars are deteriorating."

Consumer survey methodology couldn’t get a worse whipping in Singapore than Crossen gives it. She shows how careful choice of survey respondents, selection of data, and control of other variables can make virtually anything taste better than anything else.

Thus, both Pepsi and Coca-Cola claimed taste test victories at the same time. Cigarette manufacturer Lorillard claimed that "TRIUMPH BEATS MERIT" because "an amazing 60 percent said Triumph tastes as good or better than Merit." Actually, 36 percent preferred Triumph, 24 percent said they were equal, and 40 percent preferred Merit.

Political polls fare somewhat better under Crossen’s microscope, but she identifies an amazing number of problems that pollsters need tobeware of, but often don’t. Unfortunately, her prime example is the polls surrounding the Clarence Thomas Supreme Court hearings. Noting that a year after the hearings the same pollsters that had reported Americans believing Thomas now had them believing his accuser, Anita Hill. She argues that the pollsters blew it, that the Senate acted on the polls, and that thus was changed the course of history. She never broaches the possibility that Americans were subject to new influences in that intervening year.

Yet this is what happened as feminists turned Hill into a symbol of victimization and Hollywood weighed in with at least three prime-time shows all depicting Hill as the hero and Thomas as the villain.

To strengthen her case as much as possible, Crossen occasionally seems to do her own invidious selecting of information. She rightly ridicules the cigarette industry’s pseudo-scientific claims regarding direct smoking, but then curtly dismisses the industry’s arguments against the EPA report implicating second-hand smoke in thousands of American lung cancers. Yet the EPA used the same technique of combining uncombinable studies that she had earlier condemned when Quaker Oats used it.

Even then, the agency had to change its usual method of determining statistical significance, essentially moving the goal posts back to the five yard line because the ball ended upon the three.

Nonetheless, such problems as these are not systemic. Tainted Truth is a terrific guide to a creeping disease that, as she puts it, will make it "almost inevitable that we will eventually lose our ability to cope with our problems." It should be read by every editor and reporter who deals with policy issues, along with everyone who still believes that truth is something to be sought, rather than manipulated for profit.

Crossen concludes by providing "solutions," some of which are quixotic, others more practical. One such: "American schools, which have largely ignored the explosion of quantitative information in daily life, can and should teach people how to tell whether particular sets of numbers are believable. Learning information skills should be as important to high school and college as a working knowledge of literature, science, economics or communications."

Add to that the need to teach formal logic and say "Amen!"