If you follow Itron’s forecasting work, you know that we’ve employed Statistically Adjusted End Use (SAE) models since the early 2000’s. To date, almost 60 companies across North America use the SAE models. These models are applied to both internal company forecasts and external regulatory filings.
With the widespread adoption of the SAE models, comes the question “are they accurate?”
Considering that SAE models are applied to long-term forecasts (10+ years), accuracy is not the primary reason why these models are used. The value of SAE models is developing an understanding of key energy drivers based on how customers use electricity.
Unlike traditional economic models which are based on macro-economic variables, the SAE model contains information about thermal shells, appliance saturations, and energy efficiency changes. This information allows the forecast to bend with known changes in end-use codes and standards. Explicit assumptions for energy efficiency trends create opportunities for developing scenarios and accounting for energy efficiency programs. And, detailed end-use projections open the door for understanding which end-uses are responsible for forecast growth.
Ultimately, trust in a forecast lies in its ability to present a plausible picture of the future. The greater the explanatory power of the forecast, the greater the trust.
But, is SAE accurate? If there’s one thing every forecaster knows, it’s that every forecast is wrong.
In 2001, the SAE model predicted increasing average use due to increasing saturations of central air conditioners, larger homes, and greater penetration of consumer electronics such as home office equipment and security systems. The figure below shows the 2001 indexed sales forecast (blue) compared against historical calculations from 2015 (red). In this figure, the forecast aligns well with history through 2005. Deviations begin to occur in 2006 with the first projected declines in average use due to higher energy prices and energy efficiency gains. And of course, the 2001 forecast missed the Great Recession, 2005 EPACT legislation, and 2007 EISA impacts.
The SAE model allows us to examine the end-use components and assess where the errors occur. In the figure below, the forecast of non-HVAC average usage shows consistency with history through 2006. After 2006, the deviation begins. Digging into the SAE data shows that improved lighting efficiency is the main factor driving non-HVAC usage down. Obviously, the advances in lighting standards and technology were not known in 2001.
Once the error is understood, future forecasts in the SAE model are improved by updating information about lighting efficiency.
So, are they accurate? Remember, the value of SAE models is not in accuracy but in information. Even so, Itron reviewed the accuracy of SAE models in its 2015 Benchmarking Survey. When Itron asked forecasters how well their 2013 forecast performed in 2014, SAE modelers (35 responses) showed an average MAPE of 1.46% while non-SAE modelers (32 responses) showed an average MAPE of 1.53%.
While accuracy is important, remember it is not the only factor in selecting a forecast.
Mark Quan is a Principal Forecast Consultant with Itron’s Forecasting Division. Since joining Itron in 1997, Quan has specialized in both short-term and long-term energy forecasting solutions as well as load research projects. Quan has developed and implemented several automated forecasting systems to predict next day system demand, load profiles, and retail consumption for companies throughout the United States and Canada. Short-term forecasting solutions include systems for the Midwest Independent System Operator (MISO) and the California Independent System Operator (CAISO). Long-term forecasting solutions include developing and supporting the long-term forecasts of sales and customers for clients such as Dairyland Power and Omaha Public Power District. These forecasts include end-use information and demand-side management impacts in an econometric framework. Finally, Quan has been involved in implementing Load Research systems such as at Snohomish PUD. Prior to joining Itron, Quan worked in the gas, electric, and corporate functions at Pacific Gas and Electric Company (PG&E), where he was involved in industry restructuring, electric planning, and natural gas planning. Quan received an M.S. in Operations Research from Stanford University and a B.S. in Applied Mathematics from the University of California at Los Angeles.
