11.5.2.1 Calibration

One leg to the confidence table is the capability to reproduce past performance, that is to simulate a period of time over which there exists good data on the key stocks and flows in the system being modelled, whether the commodity price for an oil production model, sales history for a marketing model, production costs for a manufacturing model or the stock of vehicles in a country. If the model can be shown to replicate closely historical behaviour for the right reasons then the user will have greater confidence in the lessons learned from the simulator.

Calibration involves finding the values of model constants that make the model generate behaviour curves that best fit the real-world data. Manual calibration is a slow, painstaking process involving manipulation of the input assumptions, the running of the model, and the visual assessment of ‘goodness of fit’ for a range of performance indicators. Over the years, SD software tools have evolved in order to assist in this process, the most notable being the use of optimisation algorithms.

In the so-called ‘calibration optimisation’, the payoff is calculated as the accumulated differences between each historical and model-generated data point, the minimisation of which will result in a tendency to select model constant values minimising the difference between the historical data and the results generated by the model over the same historical period.

In the example in Figures 11.6 and 11.7 historical data exists for a number of key model variables, such as ‘vehicle penetration in region’. In the first instance, the software is informed of the variables for which historical data exists and, in the second, the range of values for which the selected assumptions may be searched in order to find a best fit.

Once a good fit has been achieved, the software provides a list of the constant values selected during calibration and these can be automatically used as input assumptions for future simulation runs. Figure 11.8 shows calibration output for the example model, where the calibration routine has found a good fit for vehicle penetration (in vehicles per thousand of population). Note that, in this case, some of the ‘historical’ data is actually ‘forecast’ data obtained from industry analysts.

11.5.2.2 Reality Checks

As models are built, there are various checks that must be done against reality. These checks may be explicit and take the form of tests of model behaviour or subsector behaviour under different assumptions, or they may be implicit mental simulations and analyses based on an understanding of models and the modelling process. In either case these checks are very important in ensuring that the models developed can adequately address the problems they are being applied to.

Reality checks provide a straightforward way to express statements that must be true about a model for it to be useful, and the machinery to test a model automatically for conformance with those statements. Reality Check is a technology that adds significantly to the ability to validate and defend models. It can also focus discussion away from specific assumptions made in models onto more solidly held beliefs about the nature of reality.

Models are representations of reality, or our perceptions of reality. In order to validate the usefulness of a model, it is important to determine whether things that are observed in reality also hold true in the model. This validation can be done using formal or informal methods to compare measurements and model behaviour. Comparison can be done by looking at time series data, seeing if conditions correspond to qualitative descriptions, testing sensitivity of assumptions in a model, and deriving reasonable explanations for model-generated behaviour and behaviour patterns.

Another important component in model validation is the detailed consideration of assumptions about structure. Agents should not require information that is not available to them to make decisions. There needs to be strict enforcement of causal connectedness. Material needs to be conserved.

Between the details of structure and the overwhelming richness of behaviour, there is a great deal that can be said about a model that is rarely acted upon. If you were to complete the sentence ‘For a model or submodel to be reasonable when I __ it should __’ you would see that there are many things that you could do to a model to find problems and build confidence in it.

In most cases, some of the things required for a model to be reasonable are so important that they get tested. In many cases, the things are said not about a complete model but about a small component of the model, or even an equation. In such cases the model builder can draw on experiences and the work of others relating to the behaviour of generic structures and specific formulations.

Ultimately, however, most of the things that need to be true for a model to be reasonable are never tested. Using traditional modelling techniques, the testing process requires cutting out sectors, driving them with different inputs, changing basic structure in selected locations, making lots of simulations, and reviewing the output. Even when this gets done, it is often done on a version of the model that is later revised, and the effect of the revisions not tested. Reality Check equations provide a language for specifying what is required for a model to be reasonable, and the machinery to go in and automatically test for conformance to those requirements. The specifications made are not tied to a version of a model. They are separate from the normal model equations and do not interfere with the normal function of the model. Pressing a button shows whether or not the model is in violation of the constraints that reality imposes.

The example in Figure 11.9 illustrates the use of reality checks in a policy model. There are two reality checks shown: ‘no demand, no production’ and ‘no demand, production capacity reduces to zero’. These are named in a fashion indicating the nature of the test inputs and the expected model behaviour. These equations are not part of the normal model structure and are not simulated during normal execution of the model. When a reality check experiment is called, however, each reality check is tested in turn, the software will force the test inputs to be true and it will compare model behaviour with the behaviour expected. For instance, in order to run these reality checks, the software forces ‘EU export sales’ and ‘EU domestic vehicle sales’ to be zero for the entire duration of the simulation run, and checks model behaviour, in this case ‘EU vehicle production capacity’ and ‘actual EU vehicle production’. If they conform to the expectation, the check is recorded as ‘passed’, while a ‘failure’ indicates a violation of this check.

Such reality checks are developed by those with the greatest knowledge of the real-world system being explored. Failure of any of these checks would need to be investigated and any structural failures in the model rectified. Passing checks designed by the policy makers and/or subject matter experts is a powerful validation of any model.

11.5.2.3 Illuminating Model Behaviour

Communication between the model end user (decision maker) and model developer (practitioner) is of paramount importance in ensuring models are useful. As previously noted, the software and hardware available in the twenty-first century have enabled increasingly complex representations of systems to be modelled. Increasing model complexity requires increasingly sophisticated analytical tools and methods in order to maintain the necessary level of communication and, therefore, understanding of and confidence in the model. Without the ability to communicate model structure and behaviour, the practitioner is unable to validate the model in the eyes of the policy maker and the project is likely doomed to fail. All SD packages now use diagramming tools in the creation of the model and most support both CLD and stock–flow diagram (SFD) representations. Furthermore, Vensim includes the ability to create multiple ‘views’ of the model structure serving to emphasise one structural relationship or another, a particular feedback loop or part of an SFD. Combined with the use of symbols, colour, annotations and ‘story-telling’, the practitioner is able to engage more effectively with the problem owner; this leads to a greater understanding of the modelling process by the latter and of the mental model of the problem owner by the former.

As an example, a manufacturing model, developed as a research tool, is shown in Figure 11.10. Although the project team are able to navigate the model quickly and understand where key influences are likely to be the causes of particular model behaviour, it is clear that those without a similar SD background, or having not been involved in the project development, would need support in order to aid understanding. A first step may be to create another model ‘view’ and simplify the representation (the diagram in Figure 11.10 represents all equations and influences using an SFD approach) through simply adding a high-level pictorial representation, as shown in Figure 11.11.

Here, the key influences only have been included and much of the intermediate calculation steps omitted for the sake of clarity. The diagram is still, unfortunately, overwhelming for many non-practitioners and so the user may choose to hide all but one small section of the model, the remainder to be progressively unveiled as the practitioner explains each stage of the diagram to the end user (story-telling). The use of font and colour changes further emphasises each section of the model and provides further visual support aiding understanding. In the recent past, this was achieved by copying the model and pasting multiple copies into, for example, Microsoft PowerPoint. Manipulation of each copy would enable story-telling as the reader progressed from slide to slide. This is now possible within many SD packages without the need to resort to copy and paste.

‘Story-telling’ is a powerful method for articulating the model structure as it relates to the behaviour of the system being modelled. Larger models tend to be made more readable through modularisation but decision makers often still have problems when faced with relatively simple modules, or ‘plates of spaghetti’. By allowing the practitioner to ‘hide’ model structure at multiple hide levels, model structure can be revealed in bite-sized, manageable portions. When coupled with explanatory notes appearing at each successive level, the practitioner is able to communicate more effectively with the end user.

Figures 11.12–11.16 show one such sequential ‘story-board’ where the practitioner is able to describe successfully a large proportion of the model with a series of building blocks beginning with a simple ‘call for maintenance’ to a maintenance service provider as a result of equipment failure at the manufacturer. In the next step (Figure 11.13), the impact of equipment failures at the manufacturer is explained as a reduction in plant productive time, further complicated by the availability or otherwise of spare parts to affect the repair.

Figure 11.14 introduces the concept of predictive and proactive maintenance. This requires the service provider to monitor the equipment for signs indicating potential failure in the future, and for those parts to be proactively replaced, thus reducing equipment failures and maintaining a higher plant productive time, even accounting for the scheduled downtime as a result of proactive maintenance, as such downtimes are predictable and can often be scheduled at times when the plant would not be operating anyway.

Other impacts on productivity are losses in production hours due to less than optimum operating as a result of worn parts. In Figure 11.15, the concepts of ‘speed loss’ and ‘quality loss’ are introduced in order to capture this effect on ‘plant productive time’.

Finally, Figure 11.16 includes the contracting issues as they affect the relationship between the manufacturer and its maintenance service provider. It is clear that insufficient contractual support may be the result of improper and infrequent contract reviews, especially if the manufacturing company is growing and, as a result, requesting greater contracted maintenance support.

Once a model is able to demonstrate it passes rudimentary tests, the syntax is correct, material is conserved, units are consistent, and so on, the practitioner will be able to start simulation testing in earnest. Observation of model-generated output and checking of the calculation made by each model equation form the next obvious steps in testing, and SD software analysis tools enable not only the verification that the model equations are calculating as intended, but that the output is valid under a range of assumptions. The ability to demonstrate why the model behaves in a particular way is a powerful means of gaining the confidence of the end user. As models increase in complexity, this ability to trace the cause-and-effect behaviour through layers of model structure becomes more difficult. Causal Tracing™ is a tool specifically designed to help the practitioner trace the causes of model behaviour, enabling a more rapid assessment of model output than would otherwise be possible. It is important to be able to interrogate the structure as well as the behaviour of the model, as the former determines the latter, while the latter should behave in a similar fashion to observed real-world behaviour and/or expected future behaviour.

In Figure 11.17, the simulated behaviour of ‘overall equipment effectiveness’ (OEE) is graphed and managers want to understand the reason for the observed cyclic behaviour. A causal tree for the output in question describes the immediate causal influences on this behaviour, ‘availability’, ‘performance’ and ‘quality’.

Immediately, the user can observe the direct structural influences on the output in question and follow this structural causality through multiple model layers. More powerfully, a causal ‘strip graph’ generates behavioural output for OEE (Figure 11.18, column ‘A’). It is immediately clear that the main influence on the OEE behavioural pattern is ‘availability’ and the user would naturally wish to pursue this avenue of investigation. By simply clicking on ‘availability’ in the displayed strip graph, a new strip graph is displayed showing graphical output for those immediate influences on ‘availability’ (column ‘B’).

Once again, the user is able to home in on the main cause of this behaviour and repeat the process (columns ‘C’ and ‘D’) and, in this particular case, track down the main cause of the oscillating OEE behaviour (in this complex model the causal tracing has to continue for some levels yet and, in this case, the oscillation was due to the feedback between usage of the manufacturing line and its rate of failure; the higher the number of failures, the less up-time leading to a change in the failure rate and greater up-time).

Causality can also be traced through tabular output of model values, assisting the developer in the verification process (Figure 11.19).

11.5.3.1 Model Documentation

Although seemingly of little importance to many model builders, a fully documented model is a must if it is to be peer reviewed and understood. Most of the software allows the model builder to enter textual descriptions for each equation and it is good practice to do so as the model is developed. Ventana has developed an advanced documentation add-on for Vensim, allowing model documentation to be created within Microsoft Excel. In Figure 11.22 below we see an example from the earlier example in this chapter. If the model builder has a written description of each variable in the ‘comments’ field of the equation editor, then this Excel documentation can be created with a few clicks of a mouse button.

Other tools can be added to Vensim by programmers with the necessary skills. For example, many end users would like to see more sophisticated documentation such as ‘data source’, ‘knowledge source’, ‘telecon information’, and so on, and a new application has been created to allow the model builder to enter documentation for a variable in the equation editor but under a range of user-defined fields.

For example, an application is created and set up to be activated upon a double click within the Vensim equation editor; this produces a ‘pop-up’ window, such as in Figure 11.20. The developer is then able to enter text under a series of user-defined fields.

Fields can be added, deleted and rearranged by the model developer (Figure 11.21) thus enabling a more sophisticated means of documenting each equation. In addition, the macro responsible for the Excel documentation in Figure 11.22 can be extended automatically to populate additional columns for each of the user-defined fields resulting in a fully documented list of variables indicating equation, type, group, causes, uses, units of measurement, subscript ranges, subscript elements and textual information under an unlimited number of user-defined headings.

11.5.3.2 Sensitivity Analysis

There are many occasions where uncertainty exists in the value of input assumptions for a model and one method of understanding the impact of this uncertainty is to vary the inputs and evaluate the impact on a number of key performance indicators (KPIs). In the past this would have involved time-consuming multiple experiments, although this has become automated in most software tools today, simply by listing the assumptions where uncertainty in the value exists and defining that uncertainty with the use of a random function from which to sample values. During sensitivity analysis, the software will repeat hundreds of simulations, each time sampling for the uncertain assumptions from their distributions, and the results stored as confidence bounds on the KPI outputs.

Figure 11.23 displays sensitivity output for ‘vehicle penetration in region’, a KPI from a vehicle ownership model trying to understand the possible future use of vehicles in an area of growing vehicle ownership. In this simple example, the developer wanted to assess the impact of uncertainty in a calibration estimate called ‘sensitivity new car sales to gdp’. This is added to the list of ‘currently active parameters’ (see Figure 11.24) and Vensim is instructed to select values from, in this case, a normal distribution function.

11.5.3.3 Policy Optimisation

A powerful weapon in the SD software armoury is the use of optimisation. That is, the ability to instruct the software to use the model to maximise a desired outcome, usually profit. By setting a payoff definition file (in much the same way as in the calibration setup at 11.5.2.1) and instructing Vensim to select from a range of values for a number of policy levers, the software will attempt to find the best combination in order to achieve the optimisation, running many hundreds or even tens of thousands of simulations in the process.

In the business game example, users are asked to choose from a multitude of decisions each quarter, in gaming mode. That is, the user makes decisions and simulates the next quarter in order to evaluate the outcome of those decisions. The decisions are evaluated using a simple balanced scorecard; points are allotted for good performance under a range of KPIs and the cumulative points show the overall performance of the individual user. It is useful to compare the performance of each user but also useful to see how close they come to an optimal solution.

Using optimisation, each policy lever was allowed to be adjusted between its maximum and minimum values at each decision point (i.e. after each quarter). The optimisation algorithm found a best solution which can then be presented to the users as a yardstick by which to compare their performance (Figure 11.25).

Use of optimisation features presents the end user with something tangible: a means to understand the implications of policy change in the improvement of performance.

11.5.3.4 User Interface

If there is one innovation surely capable of enhancing the usability of SD models, it has been the ability to create user-friendly graphical user interfaces (GUIs). Many of the SD software packages allow interfaces to be developed in languages such as Visual Basic and C++ and some allow interaction with the model through Excel. The latter has the advantage of familiarity while all enable the developer to create an environment designed to maximise the use of the model with minimal effort from the user.

In the latter half of the 1990s, a move was made to develop a GUI tool for Vensim which would be better looking and easier to use than the in-built capability of the software at the time. This effort, initially developed by practitioners for specific clients, became the software now known as Sable. Sable enables professional interfaces to be developed in hours and days rather than weeks and presents to policy makers an easier interaction with the underlying model.

Sable is a drag-and-drop object-based application enabling rapid and easy access to model inputs and outputs together with the majority of Vensim in-built analysis tools such as Causal Tracing. The example in Figure 11.26 shows a screen from a simple interface created for a small simulation investigating the growth in energy micro-generation, that is the generation of electricity and heat at the point of use. The interface has two slider bars only (there are many other assumptions) and is designed as a quick introduction to SD for energy industry participants who are non-users of SD. The interface developer also created a story-telling screen to explain the model and the scenario the users are faced with, simply by linking to the relevant Vensim view in the underlying model (Figure 11.27).

Interfaces have been created allowing multiple users simultaneous access to a single model on an organisation's intranet and enabling greater dissemination of the model and its results throughout the organisation. Forio (Forio Online Simulations, 2013) goes a step further by enabling Web-hosted interface applications compatible with many of the available SD software packages. Powersim and iThink are also examples of applications enabling interaction with models over the Internet, opening up SD use to the many.