The proper role of statistical forecasting in the making of government policy is currently being debated in many parts of the United States, particularly with regard to U.S. energy consumption and the recent deregulation of the electricity industry. The following article is part of that debate.

In 1981, then vice-chairman of Chicago's Commonwealth Edison, Gordon Corey, stated “there is an unbreakable tie between economic prosperity and energy use.” Similarly, in its 1976 Energy Report, Chase Manhattan Bank stated, that “there is no documented evidence that indicates the long-lasting, consistent relationship between energy use and GDP will change in the future. There is no sound, proven basis for believing a billion dollars of GDP can be generated with less energy in the future.” In fact, between 1973 and 1986, U.S. primary energy consumption remained almost flat, while GDP rose 35% in inflation-adjusted terms. [Since 1986, energy consumption has again risen along with the GDP, although at less than 60% of the rate it did in the years prior to 1973—the editors of SecondMoment.]

The truth is, many forecasters use models that are based entirely on historical elasticities and other behavioral parameters to do long-term forecasts. While the relationships embodied in these parameters may be useful in the short run, they can easily become misleading when used to conduct long-run forecasts.

Physical Laws vs. Human Behavior
Physicists conduct replicable experiments to uncover fixed physical laws. A scientist measuring the speed of light would find the velocity the same in the United States or Tahiti. If another scientist conducted a similarly accurate experiment in Russia, the test would produce the same result. On the other hand, relationships between cause and effect for individuals and human institutions are dependent on institutional, social, and economic context. Furthermore, these relationships change over time. A market researcher attempting to predict consumer acceptance for a new toothpaste would find that a market test in San Francisco would likely yield quite different results than in Tahiti. In addition, if the same experiment had been conducted in the 1950s, the results would have presumably varied wildly from those of the current day.

This ostensibly obvious observation is often ignored. The energy forecasters cited above believed in an unbreakable link between energy use and GDP, assigning to it the immutability of a physical law. These believers forgot that people and institutions can adapt to new realities, and historically-derived relationships. The apparent link between energy use and GDP, which held up for more than two decades in the post World War II period, quickly become invalid in the wake of the 1973 oil embargo.

Computer Models and Forecasts
Most economic computer models embody historical experience in the form of statistical relationships, which they then use to forecast the future. Such models are often used to assess the potential effects of proposed changes in government policy or business strategy. Still, while these models embody history, they cannot give an accurate picture of a world in which the fundamental relationships on which they depend are in flux. If the statistically-derived relationships embedded in such models are the very ones that would be affected by choices or events, then those relationships must be modified in the analysis.

For example, many economists trying to assess the costs of reducing carbon emissions assume that the only way to reduce these emissions is to impose a large carbon tax, which would substantially raise the price of coal, oil, and natural gas. They then further compound this error by using models that embody historically derived elasticities to do long-term forecasts without altering those parameters to reflect the effects of near-term policy choices. Creating a world with vastly lower carbon emissions presupposes massive behavioral and institutional changes that would render past relationships between energy use and economic activity largely irrelevant, just as those that occurred after 1973. It is simply not possible to use computer models based on historical relationships to produce one hundred-year forecasts in the face of such massive changes.

Personal Perspective
In Autumn of 1997, the U.S. Environmental Protection Agency and Department of Energy asked my team at Lawrence Berkeley National Laboratory to conduct an analysis of the Clinton Administration’s soon-to-be proposed tax credits for energy efficient equipment. (For details on the calculations, download the analysis spreadsheets at http://enduse.lbl.gov/Projects/TaxCredits.html.) Our first attempts to model the effects of the credits involved running two engineering-economic models with a reduced price for the more efficient appliances. Soon, however, we realized that simply reducing the capital cost of more efficient equipment without changing the decision parameters affecting efficiency choices was simply another form of the same mistake.

After this realization, we went back to the best empirical data on responses to rebates, which had been created by Kenneth Train at University of California Berkeley, and used it to create a spreadsheet embodying those responses. There were two effects of a rebate from the Train data. The first was the “Direct Price Effect” on the market share of the more efficient product, which was what we first attempted to model. The second was the effect of a rebate that is independent of the size of the rebate, which we dubbed “The Announcement Effect.” Train’s analysis showed a change in market share for a rebate of zero—that is, the very fact that a rebate existed lent institutional credibility to a particular technology that the technology did not have before. In addition, the people selling the product changed their marketing strategy to use the existence of the rebate in their promotions, modifying markups and pricing to reflect the new strategy. Because it was based on data from only one utility service territory, Train’s analysis did not account for a third important effect relevant to national tax credits, that of the learning curve associated with increased production experience with a particular technology. As cumulative production experience for a product doubles, costs typically decline by 10 to 20% on a per unit basis. Many high-efficiency technologies are niche products with small sales, so it is easy to double cumulative production experience many times in the early stages of market acceptance. Cost reductions associated with increasing production experience are critically important for such technologies.

Our initial attempt to change capital costs in the engineering-economic models only addressed the direct price effect, which in our more sophisticated spreadsheet calculations accounted for only 10 to 30% of the total effect of the tax credits for HPWHs and CACs, respectively.

Incomplete Technology Portfolio
Another more subtle form of this same mistake occurs when forecasters conduct an analysis with an incomplete technology portfolio. For example, many large-scale models of the costs of reducing climate change contain relatively detailed representations of conventional electricity supply side technologies, but have little or no representation of efficiency technologies for end-users. Even on the supply side, these models often omit the technologies of the most interest from a long-term perspective—e.g. fuel cells, cogeneration, renewables. Yet these technologies are exactly the ones most likely to make a large difference in greenhouse gas emissions over the medium to longer term.

For example, Michael Kraus, who with his colleagues has performed a number of studies on European energy markets and reducing carbon emissions in Europe, accounts for the effects of policy choices on technology costs, on fuel prices, and on resource availability. He and his colleagues also include a detailed technology portfolio, designed to characterize both demand and supply-side technologies, conventional and advanced. These studies rely on empirical data to ground the scenario exercises in real-world experience. (See Krause, Florentin, Eric Haites, Richard Howarth, and Jonathan Koomey. 1993. Cutting Carbon Emissions—Burden or Benefit?: The Economics of Energy-Tax and Non-Price Policies.)

Another way to avoid some of the more common pitfalls in forecasting is to use a set of forecasts (i.e., scenarios), as described by Peter Schwartz in his 1996 book The Art of the Long View: Planning for the Future in an Uncertain World. Schwartz builds on the work of Pierre Wack, a planner in the London Offices of Royal Dutch/Shell whose own scenario analysis helped that company respond quickly and successfully to the Arab oil embargo following the Yom Kippur war in 1973. One of the central notions here is to vary key factors and investigate which of them to ignore and which to dissect further. All forecasts are wrong in some respect, but the very process of designing them can teach you something about the world and how events may unfold. Quantitative analysis can lend coherence and credence to scenario exercises by elaborating on consequences of future events, but modeling tools should support that process and not drive it, as is so often the case.

-------------

About The Author Dr. Jonathan Koomey is a Staff Scientist and Group Leader at Lawrence Berkeley National Laboratory. His new book, Turning Numbers into Knowledge: Mastering the Art of Problem Solving, was recently published by Analytics Press.

More Features