10.3. Tests for Building Confidence in Models
The fluid nature of models in the social sciences has led system dynamicists (and other social system modellers) to adopt a broad and pragmatic view of model validity as a process of confidence building among those who will use the model (Forrester & Senge, 1980). Confidence building involves a variety of different tests to assess the quality of both the model and the model building process. Here I present three categories of tests that have proven particularly useful in practice: tests of model structure, tests of model behaviour and tests of learning. Other tests are also used, but they are beyond the scope of this book. Readers who wish to know more are referred to the comprehensive treatment of model validation and testing in Sterman (2000, Chapter 21).
Tests of model structure are intended to assess whether the feedback structure and equation formulations of the model are consistent with the available facts and descriptive knowledge of the real-world system. These tests apply to both the conceptual model and the algebraic model and are very important in system dynamics practice because they draw attention to whether modellers are making effective use of judgemental information for modelling. As Figure 10.3 illustrates, there is a vast amount of relevant data about social systems that resides in the minds of experienced people (their collective mental database). This information about the inner workings of the organisation, its procedures, priorities and even culture is rarely found in either formal numerical or written databases, yet is vital to a good representation of structure.
Figure 10.3. Sources of Information for Modelling
Source: Jay Forrester, 1994, Policies, Decisions and Information Sources for Modelling, in Modelling for Learning Organizations. Productivity Press. Reproduced by permission of Jay W. Forrester.
Tests of model behaviour are intended to assess the fit of simulations to observed real system behaviour. They serve a similar purpose to conventional goodness-of-fit tests in statistics and econometrics. However, behaviour tests are typically less formal than regression methods or statistical tests of significance in that they often rely on visual criteria to gauge goodness-of-fit such as the shape, scale and timing of simulated trajectories relative to actual time series data. Unlike regression methods, behaviour tests do not involve statistical estimation of parameter values to achieve a best fit with time series. The parameter values in the model are obtained from numerical facts in the case, independent of time series data. Any adjustments to parameter values made to improve fit with time series must remain plausibly close to the independent numerical facts (unless of course the facts themselves prove to be wrong or misleading).
Tests of learning are intended to assess whether model users have gained new insight about system structure or learned something new about real system behaviour. These tests draw attention to the interaction between mental models and formal models during a modelling project. They differ from the other tests because they focus on soft aspects of modelling – not so much the fit of the model with the real world but rather its ability to influence the way model users interpret their world. Learning can occur at any stage of the modelling process from problem articulation to model formulation and testing as Figure 10.4 illustrates. Here the modelling process interacts with a larger organisational process of strategic change involving mental models, decisions and outcome feedback. The learning cycle in the outer loop is slow and imperfect because organisational experiments take a long time to conduct and mental models are resistant to change. Moreover, outcome feedback is patchy and tricky to interpret. Formal models can accelerate this learning cycle by providing new insight into both system structure and likely dynamic behaviour.
Figure 10.4. Modelling for learning
Source: Sterman, J.D., Business Dynamics: Systems Thinking and Modeling for a Complex World, © 2000, Irwin McGraw-Hill, Boston, MA. Reproduced with permission of the McGraw-Hill Companies.