Lies, Damn Lies, and Economics

Have you heard about the recent scientific project that validated the Mediterranean diet? Researchers completed a massive five year study of over 7,000 people; they found that individuals can significantly decrease the probability of heart attacks and strokes when they focus their diets on olive oil, nuts, fish, fruits and vegetables, and (of course!) wine.

The study was particularly persuasive because it tracked three identical groups of people at the same time. They all lived in Spain and maintained similar life styles; the only differences that could have caused any variations in cardiovascular health were their (carefully controlled and measured) differences in dietary patterns.

When scientific medical studies are able to control for all relevant variables across very large groups of experimental participants, it’s easy to trust the results of the research activities. But should we place the same level of faith in the economic studies that are influencing the global fiscal policies of our political leaders?

Do Your Homework!

Three years ago, a pair of globally renowned economists from Harvard University published a study that demonstrated a significant relationship between levels of government debt and levels of economic growth. Specifically, they found that nations that run up massive deficits in order to “jump start” growth often learn that such strategies are counter-productive because the resulting debt obligations weigh down their economies.

At first, conservative Republican politicians in the United States embraced the findings as evidence that drastic austerity measures should be implemented to reduce the federal government’s budget deficit. And, after all, who could argue with them? If economists at a quintessentially progressive school like Harvard University could demonstrate that debt-financed government spending represents a counter-productive policy, how could any one continue to support economic stimulus activities?

Last week, however, a student at the University of Massachusetts named Thomas Herndon completed a homework assignment that required him to check the calculations of these two distinguished economists. Amazingly, he discovered that the economists made a number of simplistic mistakes and questionable judgment calls, actions that may have invalidated their conclusions.

The GIGO Principle

Most of the negative publicity regarding this controversy focused on a simple spreadsheet calculation error, one that significantly biased the economists’ results. That single error, on its own, attracted a significant amount of scorn towards the researchers and their findings.

But a closer reading of the student’s homework assignment, which is now being repositioned as an academic study, reveals an array of additional concerns concerning the validity of the original research activities. For instance, the Harvard researchers apparently ignored a number of highly indebted (and yet quickly growing) economies from the late 1940s, cases that would have served to contradict the findings of the original study.

Are you familiar with the phrase Garbage In, Garbage Out, or the relevant acronym GIGO? It refers to situations where faulty data is fed into information systems, which inevitably produce analyses and recommendations that are faulty as well. Can this GIGO principle, in fact, be applied to the original Harvard study?

Over a century ago, the great American humorist Mark Twain addressed such concerns. “Figures often beguile me, particularly when I have the arranging of them myself,” he wrote, adding “there are three kinds of lies: lies, damned lies, and statistics.” With this philosophy in mind, what are we to make of the brouhaha regarding the Harvard study?

The Nature of Economics

To be fair to the Harvard economists, one can argue that the excluded nations of the 1940s bear little resemblance to the nations of today. Considering that the purpose of their study was to provide information to contemporary political leaders about the impact of large budget deficits, it might indeed be reasonable to exclude such cases that have little in common with the modern world.

And yet, when making such exclusionary decisions, where does one draw the line? Are the nations of the 1950s, for instance, similar to the nations of today? Or the nations of the late twentieth century … or even those in the years prior to the 2008 / 09 global crash? Are any of those time periods — and the nations that existed during those periods — relevant to the economic challenges that face today’s political leaders?

Let’s think about the circumstances of the Mediterranean diet study. To create a highly reliable data set that is relevant to the contemporary world, the researchers closely followed the dietary habits and health outcomes of more than 7,000 people in “real time.” A macro-economic research study, though, cannot follow 7,000 nations in “real time” because the world contains fewer than 200 countries.

Thus, the only way to create a macro-economic data set with a significant sample size is to roll back through history and include periods of time that may (or may not) be relevant to the research question. The resulting uncertainty is, regrettably, an inevitable result of the nature of economics.