Chapter 10. Model Validity, Mental Models and Learning
Models of Business and Social Systems
Tests for Building Confidence in Models
Model Confidence Building Tests in Action: A Case Study in Fast-moving Consumer Goods
Model Structure Tests and the Soap Industry Model
Equation Formulation Tests and the Soap Industry Model
Tests of Learning from Simulation
Summary of Confidence Building Tests
Conclusion – Model Fidelity and Usefulness
In an episode of a popular television programme called Changing Places a computer gaming enthusiast, who had clocked-up hundreds of simulated hours on a PlayStation driving imaginary high performance cars, was invited to drive a real racing car around Silverstone (a major race circuit in England, home of the British Grand Prix and the birthplace of Formula 1). The experience was sobering. He spun off. Even when he stayed on the track he failed to achieve competitive lap times.
This story is quite revealing about the purpose, limitations and use of models and simulators. A common view is that models are representations of reality intended to be useful to someone charged with managing and participating in that reality.[] In this case reality has a well-defined meaning (the real racing car on the track at Silverstone) and it is clear that the computer model falls short of reality in some important ways. The natural temptation for the model user is to demand a better model – one that represents a racing car more accurately. More realism is better.
[] In Chapter 1 of Systems Modelling (Pidd, 2004) Mike Pidd describes a spectrum of modelling approaches ranging from those where models are intended to be a shared representation of the real world to those where models are a representation of concepts relevant to the real world. The former characterise hard OR, which provides tools for routine (though important) decision making. The latter characterise soft OR, which provides tools for thinking and for making sense of messes/wicked problems.
However, there are several problems with high-fidelity modelling. The most obvious is that realism requires ever more detail. The model can become so large and complex that no one really understands it or has confidence in it. Slightly less obvious is that realism itself is often subject to debate if the system being modelled is ill-defined (suppose we're not really sure, before the event, whether the Silverstone challenge is to drive a car or a motorcycle). Finally, the elusive quest for realism can obscure the value of having some kind of tangible model (even if it is much simplified) versus no formal model at all.
To appreciate the value of a deliberately simplified model, it is useful to reconsider some positive aspects of the Silverstone racing experience. Most basically, the opportunity to 'change places' and drive a real racing car at Silverstone would never have arisen without the gaming simulator. The gaming enthusiast was passionate about motor racing and knew much more about the sport than the average person. He had learned a lot from hundreds of hours with the simulator. He was familiar with the car's instrumentation and controls, he knew Silverstone's layout, he had acquired some expertise in cornering technique (even though he later spun off), and he knew competitive lap times.