6.4

MODEL RISK

A risk commonly faced by energy traders is model risk—the risk that either the methodology or assumptions used to value assets becomes invalid. Poor assumptions and incorrectly designed models cause risk management problems in every financial market. However, the complexity of energy models and their extended lifetimes, make these problems especially common in the energy markets.

The primary regulatory guidance around model risk comes from the U.S. Federal Reserve in its SR 11-7 guidance to banks. This guidance is generally appropriate to all types of models, including those used in the energy market:

The term model refers to a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates. A model consists of three components: an information input component, which delivers assumptions and data to the model; a processing component, which transforms inputs into estimates; and a reporting component, which translates the estimates into useful business information. Models meeting this definition might be used for analyzing business strategies, informing business decisions, identifying and measuring risks, valuing exposures, instruments or positions, conducting stress testing, assessing adequacy of capital, managing client assets, measuring compliance with internal limits, maintaining the formal control apparatus of the bank, or meeting financial or regulatory reporting requirements and issuing public disclosures. The definition of model also covers quantitative approaches whose inputs are partially or wholly qualitative or based on expert judgment, provided that the output is quantitative in nature.

Source: U.S. Federal Reserve, SR 11-7 attachment

All models are less complicated versions of reality. There is always a risk that something vital is left out of a model, or something unnecessary is incorporated. There is a trade-off between model risk and model complexity. Very simple models may not include enough factors to accurately describe reality. On the other hand, simple models are easy for people to understand. It is possible to address model risk by having a lot of people looking for problems. Complex models include more factors, but they are harder to understand. It is often unclear when a complex model stops working, and it is unrealistic to assume that people can verify a model that they don’t understand.

The Federal Reserve identifies two main causes of model risk—poorly built models and models being used incorrectly. In addition, there are several more practical problems that are commonly observed in the energy markets—bad assumptions, bad market data, and difficulty estimating model parameters.

Model risk occurs primarily for two reasons:

•   The model may have fundamental errors and may produce inaccurate outputs when viewed against the design objective and intended business uses. The mathematical calculation and quantification exercise underlying any model generally involves application of theory, choice of sample design and numerical routines, selection of inputs and estimation, and implementation in information systems. Errors can occur at any point from design through implementation. In addition, shortcuts, simplifications, or approximations used to manage complicated problems could compromise the integrity and reliability of outputs from those calculations. Finally, the quality of model outputs depends on the quality of input data and assumptions, and errors in inputs or incorrect assumptions will lead to inaccurate outputs.

•   The model may be used incorrectly or inappropriately. Even a fundamentally sound model producing accurate outputs consistent with the design objective of the model may exhibit high model risk if it is misapplied or misused. Models by their nature are simplifications of reality, and real-world events may prove those simplifications inappropriate. This is even more of a concern if a model is used outside the environment for which it was designed. Banks may do this intentionally as they apply existing models to new products or markets, or inadvertently as market conditions or customer behavior changes. Decision makers need to understand the limitations of a model to avoid using it in ways that are not consistent with the original intent. Limitations come in part from weaknesses in the model due to its various shortcomings, approximations, and uncertainties. Limitations are also a consequence of assumptions underlying a model that may restrict the scope to a limited set of specific circumstances and situations.

Source: U.S. Federal Reserve, SR 11-7 attachment

Model Documentation

Ensuring adequate model documentation is the first step toward managing modeling risk. This requires both organizing model information and requiring accountability from people using and developing models. For example, keeping a record of all of the models in an organization is a good start to this process. Ensuring that each model meets a minimum documentation standard and has someone assigned to keep the model updated are other essential steps. Another beneficial step is to have someone in a leadership position at the firm review each model. Accountability and a defined approval process is important (Figure 6.4.1). For example, there is a world of difference between “We use the model because a college intern that we hired over the summer five years ago said it was OK” and “A senior member of the firm, in consultation with the quantitative staff, approved the use of the model after the model was compared to three separate alternatives.”

Figure 6.4.1 Documentation questions

Some good practices to limit model risk are:

•   Ensure that the firm keeps a record of all models used in the firm and that this list is updated whenever models are put into use (or retired).

•   Assign each model a unique ID so that there is no confusion about which model is which when a model breaks or needs to be maintained.

•   Require each model to achieve a minimum level of documentation and be approved by an appropriate party before being used.

•   Assign a model owner to be responsible for maintaining the model and making sure the model is still in use at the end of each year.

Using Models Appropriately

One of the keys to ensuring that models are used appropriately is to have a plain English description of the model. This ensures that everyone who interacts with the model, from professional mathematicians, to traders, to software professionals and company leaders, understands the purpose of the model.

Models are typically composed of three parts—inputs, outputs, and a processing component that converts inputs to outputs. Examining each of these pieces will often indicate potential trouble areas. For example, a model might have initially relied on a data vendor to ensure that the input data were free of errors. If a cheaper data vendor was hired as a replacement, it would be necessary to ensure that similar checks on the data were being performed (perhaps the lack of checking is why the data are less expensive!).

Important information required to understand if a model is appropriately used is whether there were any restrictions placed on the model by the model developers. Other important information includes the theoretical foundation of the model, assumptions made by developers, and whether the people who have reviewed and developed the model were authorized and qualified to make that decision (Figure 6.4.2).

Figure 6.4.2 Model appropriateness questions

Testing the Model

It is often easier to watch a model for potential problems than it is to design a foolproof model. Observation is almost always easier than prevention. Having people pay attention to what is happening to a model means that problems can be found quickly. Most problems start to occur months or years before there is a financial loss. This makes observation a viable solution to most problems. People who depend on the model need to ensure that problems get reported and examined in a timely manner.

However, it is also possible to catch a lot of potential problems by testing models (Figure 6.4.3). Backtesting is the process of testing a model against as much historical data as possible. A related concept is stress testing. Stress testing simulates extreme conditions and unusual combinations of events. This is a way to catch combinations of events that might not have happened historically. Benchmarking involves comparing a model to alternative models and intuition. Stability testing ensures that small changes in model inputs give small changes to outputs.

Figure 6.4.3 Model testing questions

•   Backtesting. Backtesting involves testing a model to ensure that it works over some historical period.

•   Stress Testing. Stress testing involves understanding how models work under unusual market conditions like a financial crisis, the bankruptcy of a major market participant, an unusually cold winter or hot summer, or a natural disaster.

•   Benchmarking. Benchmarking involves comparing a model to alternative approaches or alternative models. For example, if an option model is being benchmarked, it might be compared to alternative option formulas, trader intuition, or historically observed payouts for similar options.

•   Stability Testing. Models convert inputs into outputs. Generally, models produce progressively more severe outputs when progressively more severe inputs are used in the model. This is a good property for a model since this allows the use of inputs that have never been previously observed. Models that are unstable or chaotic (models that produce large, unpredictable changes in outputs given small changes in inputs) are much more difficult to manage.

Maintaining Models

Many modeling problems are caused by the failure to maintain models over time. Some of these issues are caused by changes to model assumptions while others are due to errors introduced as the model is updated. Models are a simplification of real life. They are based on an analysis of typical behavior that may or may not be predictive of the future. Real life has exceptions to almost every rule, and it is not always clear where those exceptions are likely to occur. The most dangerous part of a model is getting basic assumptions about how the market operates wrong. If those assumptions are wrong, every conclusion made afterward is probably invalid too.

Models and the viability of the models’ assumptions change over time. If a model lasts 20 years, a lot of things can change. Investments like power plants, pipelines, and natural gas wells all have an extremely long life span. Even when a pricing model is sufficient to describe the investment initially, there is no guarantee that things won’t change. Assumptions made years earlier can be invalidated by regulatory changes, population shifts, and technological changes. Exacerbating this problem is the problem of employee turnover—commonly, the original developers of the models have moved to another job or retired before problems develop.

An example of a reality changing underneath a model is daylight savings time. When the energy savings from shifting the clock an hour was first calculated in the mid-1970s, air conditioners were still relatively uncommon. The 1975 study showed that energy use was cut slightly if the clock was shifted in the summer to have more sunlight in the evening. Thirty years later, Indiana finally switched over to daylight savings. Compared to the period before the switch, adopting daylight savings time caused a noticeable increase in that state’s power consumption. The widespread adoption of air conditioning in the intervening years was thought to be the culprit since air conditioning, rather than lighting, was now the major use of electricity in the region.

Models often exist for a long period of time. Typically, when models are first created, the team that creates the model understands the details and knows where all of the documents supporting the model are located. Over time, those team members often get promoted, leave the firm, or pass on the work to new team members. Those new team members weren’t part of the initial discussions and often will not know where to find calculations or supporting data unless that information is stored in the documentation (Figure 6.4.4).

Figure 6.4.4 Model maintenance questions

There are several procedures that model developers can use to reduce maintenance problems:

•   Model ID. Models need a name or other unique identifier. A modeling group might deal with hundreds of different models, and being able to uniquely identify models is the key to ensuring that they are all documented and the source code for the model can be found.

•   Model Owner. Models need to have someone responsible for their maintenance. Otherwise, they can get lost or outdated.

•   Change Log. Models will typically get updated many times over their lives. Every time a model is updated, it is necessary to document what changed and the reason for the change. That way, future model owners can understand why things have been done in a certain way.

•   Source Code Control. Storing old versions of the model (version control) also ensures that if an update accidently introduces an error to a model, it is possible to go back to the previously working version of the model.

•   Test Case. A test case is a known set of inputs and outputs that can be used to test that a model is still working as originally intended. If the model is tested against a known scenario every time it is updated, the likelihood of errors being introduced is minimized.

•   Interactions List. Models often interact with one another. For example, the inputs to a model may be the outputs of a different model. As a result, changing a model may result in unintended changes in related models. The only way to prevent this from happening is to keep track of which models might be affected by changes and may need to be updated. This list must also be updated to keep it current.

Bad Market Data

Another issue that commonly comes up in energy trading models is that many physical trading points aren’t very liquid. It is possible to trade at places that don’t usually have prices. Of course, the price at the time of trading is known. However, any future prices will depend on other people trading at the same location. If no one trades there regularly, it might be a while before another price exists. As a result, illiquid trading locations often need to be approximated by prices at a nearby liquid benchmark. Sometimes, this is a good assumption. At other times, the expected relationship between a benchmark and an illiquid location breaks down.

The problem of limited data is related to the problem of how to estimate parameters. For example, a spread option will depend on an estimate of the correlation between two assets. If there are no readily available prices, the correlation can’t be readily determined. As a result, it is fairly common for a couple of model parameters to be chosen (fairly arbitrarily) when a model is created. If any of those parameters turns out to be poorly chosen, the model might change dramatically (Figure 6.4.5).

Figure 6.4.5 Model input questions

Model Overlays and Reserves

If a model can’t be trusted—and no model can be completely trusted—it makes sense to limit the total amount of money lost due to a single problem. This can be done by limiting the total amount of risk that can be taken on by a specific type of trading. This enforced diversification is called a position limit. Essentially, it is a mandate that a trading desk not place all of its eggs in the same basket.

A second way of limiting risk is to delay recognizing profits from a valuation model. If a model is hemorrhaging money, a lot of attention will get focused on it. Because of the attention, any modeling problems are likely to be found quickly. A more dangerous situation is a model that shows a steady stream of profits. That model has a much higher probability of being ignored. To limit those problems, it is possible to set aside some of those profits to cover the possibility of future losses. This is called taking a modeling reserve.