8.2.1 DES and SD Model Development Process

Considering the modelling process, as suggested in DES and SD textbooks teaching the art of modelling, one can identify similarities between the two approaches, especially in terms of the stages involved. It is clear that the main stages followed are equivalent to generic OR modelling (Hillier and Lieberman, 1990; Oral and Kettani, 1993; Willemain, 1995), which include: problem definition; conceptual modelling; model coding; model validity; model results and experimentation; implementation. For instance, see Figure 8.1 which shows a typical DES modelling process from Robinson (2004).

The main aspects pertaining to the model development process are considered next. First, it is claimed that during the DES model building process emphasis is put on the development of the model on the computer (model coding). In their experimental study, Baines et al. (1998) commented specifically that the time taken in building a DES model was considerably longer compared with a SD model. Furthermore, Artamonov (2002) developed two equivalent DES and SD models of the beer distribution game model (Senge, 1990) and commented on the difficulty involved in coding the model on the computer. He found the development of the model on the computer more difficult in the case of the DES approach, whereas the development of the SD model was less troublesome. Baines et al. (1998) attributes this to the fact that DES encourages the construction of a more lifelike representation of the real system compared with other techniques, hence resulting in a more detailed and complex model.

On the other hand, Meadows (1980) highlights that system dynamicists spend most of their modelling time specifying the model structure. Specification of the model structure consists of the representation of the causal relationships that generate the dynamic behaviour of the system. This is equivalent to the development of the conceptual model.

Another feature concerning DES and SD model development is the iterative nature of the modelling process. In DES and SD textbooks, it is highlighted that simulation modelling involves a number of repetitions and iterations (Randers, 1980; Sterman, 2000; Pidd, 2004; Robinson, 2004). Indeed, an iterative modelling process is depicted in Figure 8.1 for both DES and SD. Regardless of the modeller's experience, a number of repetitions occur from the creation of the first model, until a better understanding of the real-life system is achieved. So long as the number of iterations remains reasonable, these are in fact quite desirable (Randers, 1980).

8.2.2 Summary

The existing studies comparing DES and SD tackle three main areas of modelling: the practice of model development; modelling philosophy; and the use of models. These studies, however, are mostly opinion based and lack any empirical basis. We focus on the model development process in each field (DES and SD). The key aspects identified consist of the amount of attention paid to the different stages during modelling, the sequence of modelling stages followed, and the pattern of iterations followed. Taking an empirical perspective, we examine in this chapter the statements found in the literature by following the model development process adopted by expert DES and SD modellers. While it is expected that DES and SD modellers pay different levels of attention to different modelling stages, both DES and SD modellers are expected to follow iterative modelling processes. However, we do not have specific expectations about the exact pattern of iteration that the two groups of modellers will follow.

8.3.1 The Case Study

A suitable case study for this research needs to be sufficiently simple to enable the development of a model in a short period of time (60–90 minutes). In addition it needs to accommodate the development of models using both simulation techniques so that the specific features of each technique (randomness in DES vs deterministic models in SD, aggregated presentation of entities in SD vs individual representation of entities in DES, etc.) are present.

After considering a number of possible contexts, the prison population problem was selected. The prison population case study, where prisoners enter prison initially as first-time offenders and are then released or return to prison as recidivists, can be represented by simple simulation models using both DES and SD. Both approaches have been previously used for modelling the prison population system. DES models of the prison population have been developed by Kwak, Kuzdrall and Schniederjans (1984), Cox, Harrison and Dightman (1978) and Korporaal et al. (2000), while SD models have been developed by Bard (1978) and McKelvie et al. (2007); the UK prison population model of Grove, MacLeod and Godfrey (1998) is a flow model analogous to an SD model.

The UK prison population example used in this research is based on Grove, MacLeod and Godfrey (1998). The case study starts with a brief introduction to the prison population problem with particular attention to the issue of overcrowded prisons. Descriptions of the reasons for and impacts of the problem are provided. More specifically, two types of offenders are considered, petty and serious. There are in total 76 000 prisoners in the system, of which 50 000 are petty and 26 000 serious offenders. Offenders enter the system as first-time offenders and receive a sentence depending on the type of offence; on average 3000 petty offenders vs 650 serious offenders enter each year. Petty offenders receive a shorter sentence (on average 5 years vs 20 years for serious offenders). After serving time in prison they are released. A proportion of the released prisoners reoffend and go back to gaol (recidivists) after two years (on average), whereas the rest are rehabilitated. The figures and facts used in the case study are mostly based on reality, but slightly adapted for the purposes of the research. The modellers were provided with a basic conceptual model (Figure 8.2) and some initial data (i.e. the initial number of petty and serious offenders), while other data, namely statistical distributions, were intentionally omitted in order to observe modellers' reactions. The task for participating modellers was to develop a simulation model to be used as a decision-making tool by policy makers. For more details of the case study the reader is referred to Tako and Robinson (2009).

8.3.2 Verbal Protocol Analysis

VPA is a research method derived from psychology. It requires the subjects to ‘think aloud’ when making decisions or judgements during a problem-solving exercise. It relies on the participants' generated verbal protocols in order to understand in detail the mechanisms and the internal structure of cognitive processes that take place (Ericsson and Simon, 1984). Therefore, VPA as a process tracing method provides access to the activities that occur between the onset of a stimulus (case study) and the eventual response to it (model building) (Ericsson and Simon, 1984; Todd and Benbasat, 1987). Willemain (1994, 1995) was the first to use VPA in OR to document the thought processes of OR experts while building models.

VPA is considered to be an effective method for the comparison of the DES and SD model building process. It is useful because of the richness of information and the live accounts it provides on the experts' modelling process. Another potential research method would have been to observe real-life simulation projects, using DES and SD. This would, however, mean that only two situations could be used, which would result in a very small sample size. Additionally, for a valid comparison it is necessary to have comparable modelling situations, which would require two potential real-life modelling projects of equivalent problem situations. The potential for getting access to two modelling projects of a similar situation was deemed unfeasible. We also considered running interviews with modellers from the DES and SD fields. Given that the overall aim of this research is to go beyond opinions and to get an empirical view about model development, one can claim that modellers' reflections may not reflect correctly the processes followed during model building and this would therefore not represent a full picture of model building. Hence, interviews were not considered appropriate. VPA on the other hand mitigates the issues related to the aforementioned research methods, as it can capture modellers' thoughts in practical modelling sessions in a controlled experimental environment, using a common stimulus – the case study.

Protocol analysis as a technique has its own limitations. The verbal reports may omit important data (Willemain, 1995) because the experts, being under observation, may not behave as they would normally. The modellers are asked to work alone and this way of modelling may not reflect their usual practice of model building, where they would interact with the client, colleagues, and so on. In addition, there is a risk that participants do not ‘verbalise’ their actual thoughts, but are only ‘explaining’. To overcome this and to ensure that the experts speak their thoughts aloud, short verbalisation exercises, based on Ericsson and Simon (1984), were run at the beginning of each session.

8.3.3 The VPA Sessions

The subjects involved in this study were provided with the prison population case study at the start of the VPA session and were asked to build simulation models using their preferred simulation approach. During the modelling process the experts were asked to ‘think aloud’ as they model. The researcher (Tako) sat in the same room, but social interaction with the subjects was limited. She only intervened in the case when participants stopped talking for more than 20 seconds to tell them to ‘keep talking’. The researcher was also answering explanatory questions, providing participants with additional data inputs (if they asked for them) and prompting them to build a model on the computer in the case when they did not do so under their own initiative. The modelling sessions were held in an office environment with each individual participant. The sessions lasted approximately 60–90 minutes. The participants had access to writing paper and a computer with relevant simulation software (e.g. SIMUL8, Vensim, Witness, Powersim, etc.). The expert modellers chose to use the software they used as part of their work and hence were more familiar with. The protocols were recorded on audio tape and then transcribed.

8.3.4 The Subjects

The subjects involved in the modelling sessions were 10 simulation experts in DES and SD modelling, five in each area. The sample size of 10 participants is considered reasonable, although a larger sample would, of course, be better. According to Todd and Benbasat (1987), due to the richness of data found in one protocol, VPA samples tend to be small, between 2 and 20.

For reasons of confidentiality participants' names are not revealed. In order to distinguish each participant we use the symbol DES or SD, according to the simulation technique used, followed by a number. So DES modellers are called DES1, DES2, DES3, DES4 and DES5, while SD subjects are SD1, SD2, SD3, SD4 and SD5. All participants use simulation modelling (DES and SD) as part of their work, most of them holding consultant posts in different organisations. The companies they come from are established simulation software companies or consultancy companies based in the UK.

A mixture of backgrounds within each participant group (DES and SD) was sought. All participants have completed either doctorates or masters' degrees in engineering, computer science, OR or hold MBAs. Their experience in modelling ranges from at least 6 years up to 19 years. They have also acquired supplementary simulation training as part of their jobs. They boast an extensive experience of modelling in areas such as the National Health Service, criminal justice, the food and drinks sector, supply chains, and so on. A list of the expert modellers' experience and the software used in the modelling sessions is provided in Table 8.2.

8.3.5 The Coding Process

A coding scheme was designed in order to identify what the modellers were thinking about in the context of simulation model building. The coding scheme was devised following the stages of typical DES and SD simulation projects, as in Robinson (2004), Law (2007), Sterman (2000) and Randers (1980). Each modelling topic has been defined in the form of questions corresponding to the modelling stage considered. The modelling topics and their definitions are as follows:

1. Problem structuring: What is the problem? What are the objectives of the simulation task?
2. Conceptual modelling: Is a conceptual diagram drawn? What are the parts of the model? What should be included in the model? How to represent people? What variables are defined?
3. Model coding: What is the modeller entering on the screen? How is the initial condition of the system modelled? What units (time or measuring) are used? Does the modeller refer to documentation? How to model the user interface?
4. Data inputs: Do modellers refer to data inputs? How are the already provided data used? Are modellers interested in randomness? How are missing data derived?
5. Verification and validation: Is the model working as intended? Are the results correct? How is the model tested? Why is the model not working?
6. Model results and experimentation: What are the results of the model? What sort of results is the modeller interested in? What scenarios are run?
7. Implementation: How will the findings be used? What learning is achieved?

The coding process starts with the definition of a coding scheme. Initially, the recordings of each verbal protocol were transcribed and then divided into episodes or ‘thought’ fragments, where each fragment is the smallest unit of data meaningful to the research context. Each single episode was then coded into one of the seven modelling topics or an ‘other’ category for verbalisations that were not related to the modelling task. Some episodes, however, referred simultaneously to two modelling topics and were, therefore, coded as containing two modelling topics.

Regarding the nature of the coding process followed, a mix of top-down and bottom-up approaches to coding was taken (Ericsson and Simon, 1984; Patrick and James, 2004). A theoretical base was already established (the initially defined modelling topics), which enabled a top-down approach. Throughout the various checks of the coded protocols undertaken, the coding categories were further redefined through a bottom-up approach. An iterative coding process was followed, where the coding scheme was refined and reconsidered as more protocols were coded. Pilot studies were initially carried out to test the case study, use of VPA and the coding scheme. The coding scheme was refined during the pilots and to some extent during the coding of the 10 protocols.

The transcripts were coded manually using a standard word processor. According to Willemain (1995), the coding process requires attention to the context a phrase is used in and, therefore, subjectivity in the interpretation of the scripts is unavoidable. In order to deal with subjectivity, multiple independent codings were undertaken in two phases. In the first stage, one of the researchers (Tako) coded the transcripts twice with a gap of three months between coding. Overall, a 93% agreement between the two sets of coding was achieved, which was considered acceptable. The differences were examined and a combined coding was reached. Next, the coded transcripts with the combined codes were further blind-checked by a third party, knowledgeable in OR modelling and simulation. In the cases where the coding did not agree, the researcher who undertook the coding and the third party discussed the differences and re-examined the episodes to arrive at a consensus coding. Overall, a 90% agreement between the two codings was achieved, which was considered satisfactory compared with the minimum 80% match value suggested as acceptable by Chi (1997). A final examination of the coded transcripts was undertaken to check the consistency of the coded episodes. Some more changes were made to the definition of modelling topics, but these were fairly minor. The results from the coded protocols are now presented and discussed.

8.4.1 Attention Paid to Modelling Topics

In order to explore the distribution of attention by modelling topic, the number of words articulated is used as a measure of the amount of verbalisations made by the expert modellers. In turn, this is used to indicate the spread of modellers' attention to the different modelling topics. While Willemain used lines as a measure of verbalisation in his study of modellers' behaviour (Willemain, 1995; Willemain and Powell, 2006), we believe that word counts are a more accurate measure, since they avoid the misrepresentation caused by counting incomplete (half or three-quarter) lines as full lines. This is particularly relevant when measuring the amount of verbalisation in each episode, where the protocol is divided into smaller units, including more incomplete lines. The average number of words articulated in the DES and SD protocols by modelling topic is compared in order to establish significant differences between the two groups. Figure 8.3 shows the number of words verbalised by modelling topic by the two groups of modellers. The corresponding numerical values for the average number of words and the standard deviation for the two groups of modellers are also provided against each modelling topic. Comparing the total number of words verbalised in the overall DES and SD protocols, an average difference of 1751 words is identified, suggesting that DES modellers verbalise more than SD modellers. The equivalent box plots of the total number of words verbalised by DES and SD modellers in Figure 8.3 (bottom) show that, while the medians are similar, there is a bigger variation in the total number of words verbalised by DES modellers.

Considering each specific modelling topic, the biggest differences between the DES and SD protocols can be identified with regard to model coding, verification and validation and conceptual modelling (Figure 8.3). This suggests that DES modellers spend more effort in coding the model on the computer and testing it, while SD modellers spend more effort in conceptualising the mental model.

In order to test the significance of the differences identified, the Kolmogorov–Smirnov test, a non-parametric test, is used to compare two independent samples when it is believed that the hypothesis of normality does not hold (Sheskin, 2007). In this case, only five data points (word count for each modeller) are collected from the two groups of modellers (DES and SD). Due to the small sample size and the fact that count data are inherently not normal, it is considered that the assumption of normality is violated. The null hypothesis for the Kolmogorov–Smirnov test assumes that the verbalisations of the DES modellers follow the same distribution as the verbalisations of the SD modellers. The alternative hypothesis is that the data do not come from the same distribution. This test compares the cumulative probability distributions of the number of words verbalised by the modellers in the DES and SD groups for each modelling topic.

The statistical tests performed indicate significant differences, at a 10% level, in the amount of DES and SD modellers' verbalisations for the three modelling topics: conceptual modelling, model coding, and verification and validation (Table 8.3). This suggests that DES modellers verbalise more with respect to model coding and verification and validation, and thus spend more effort on these modelling topics compared with SD modellers. However, SD modellers verbalise more on conceptual modelling. In addition, the total verbalisations of the two groups of modellers are not found to be significantly different. Furthermore, the non-parametric Mann–Whitney test is used to compare the distributions of the verbalisations by modelling topic for the two groups of modellers. The test statistic W has a value of 70 for a p-value of 0.8748. At a 5% level of significance, this test suggests that the data for the two groups of modellers represent different distributions and hence that their verbalisations differ significantly.

8.4.2 The Sequence of Modelling Stages

This section focuses on the progression of modellers' attention during a simulation model development task using timeline plots. These plots show when modellers think about each modelling topic during the simulation modelling task (Willemain, 1995; Willemain and Powell, 2006). A timeline plot is created for each of the 10 verbal protocols.

As examples, Figures 8.4 and 8.5 show timeline plots for DES1 and SD1 respectively. The plots consist of matched sets of seven timelines showing which of the seven modelling topics the modeller attends to throughout the duration of the modelling exercise. The vertical axis takes three values: 1 when the specific modelling topic is attended to by the modeller; 0.5 when the modelling topic and another have been attended to at the same time; and 0 when the modelling topic is not mentioned. The horizontal axis represents the proportion of the verbal protocol, from 0 to 100% of the number of words. The proportion of the verbal protocol is counted as the fraction of the cumulative number of words for each consecutive episode over the total number of words in that protocol, expressed as a percentage.

The timeline plots (Figures 8.4 and 8.5) are representative of most of those generated with the exception of DES3 and SD5. The former modeller did not complete the model due to difficulties encountered with the large number of attributes and population size. The latter was reluctant to build a model on the computer and so attended to model coding only at the end of the protocol, after being prompted by the researcher.

Observing the DES and SD timeline plots, it is clear that modellers frequently switched their attention to the topics. Similar patterns of behaviour were observed by Willemain (1995), where expert modellers were asked to build models of a generic OR problem. Looking at the overall tendencies in the DES and SD timeline plots, it appears that the DES protocols might follow a more linear progression in the sequence of modelling topics. Linearity of thinking implies that modellers' thinking progresses from the first modelling topics at the top left of the graph to the next one down towards the bottom right corner of the graph, with modelling topics concentrated in the centre of the plot. Meanwhile, in the SD protocols, modellers' attention appears to be more scattered throughout the model building session (Figure 8.5). The transition of attention between modelling topics is further explored in the next subsection.

8.4.3 Pattern of Iterations Among Topics

In this subsection the iterations among modelling topics are explored using transition matrices, with a view to further understanding the pattern of iterations followed by DES and SD modellers. A transition matrix represents the cross-tabulation of the sequence of attention between successive pairs of episodes in a protocol. The total number of transitions occurring in the combined DES and SD protocols is displayed in Table 8.4. It can be observed that DES and SD modellers switched their attention from one topic to another almost to the same extent (505 times for DES modellers and 507 times for SD modellers). In order to explore the dominance of the modellers' thinking, the cells in the transition matrices have been highlighted according to the number of transitions counted. The darker shades represent the transitions that occur most frequently.

The main observations made based on the DES and SD transition matrices are as follows:

• Model coding is the topic DES modellers return to most often (185). Similarly, SD modellers return mostly to model coding.
• The modelling topics that DES modellers alternate between most often are conceptual modelling, model coding, data inputs, and verification and validation (shown by the dark grey highlighted cells in the DES transition matrix in Table 8.4). These transitions form the dominant loop in DES modellers' thinking. Among these transitions the highest are the ones between model coding and data inputs, and vice versa.
• SD modellers alternate mostly in a loop between conceptual modelling, model coding and data inputs. These transitions determine the dominant loop in their thinking process (dark grey highlighted cells in the SD transition matrix in Table 8.4).
• Comparing the dominant loops in the DES and SD transition matrices (Table 8.4), the pattern of the transitions for SD modellers follows a more horizontal progression, while a diagonal progression towards the bottom right-hand side of the matrix is observed for DES modellers. This serves as an indication that DES modellers' thinking process progresses more linearly compared with that of SD modellers.

The indication of linearity identified by comparing the dominant loops in DES and SD modellers' thinking is further verified using the total number of transitions of attention for the parallel linear strips in the two transition matrices. Each parallel strip includes the diagonal row of cells going from the top left to bottom right of the transition matrices (Table 8.4). The cells in each parallel strip have been highlighted in different shades for the DES and SD matrix separately (part (a), Table 8.5). So eight linear strips with different shades have been created, for which the total number of transitions is counted and summed (part (b), Table 8.5). In the case of absolute linear thinking, it is to be expected that all transitions would be concentrated in the central (darkest grey) strip. This would mean that the total number of transitions in the darkest grey strip would be equal to the total number of transitions, that is 505 for DES modellers and 507 for SD modellers. However, this is not the case with any of the DES or SD protocols. Nevertheless, the total number of transitions for all eight linear strips can provide an indication of the extent of linearity involved. The total number of transitions in each corresponding strip is compared for the DES and SD matrices. The highest total of transitions in the most central diagonal strips indicates a more linear modelling process. The highest total of transitions for the strips further away conveys a less linear process. The total number of transitions per linear strip for the DES and SD protocols is shown in Table 8.5(b). The reader should note that the totals shown in Table 8.5(b) differ from the totals in Table 8.4 due to the fact that the cells at the top right and bottom left corners have not been included in a parallel strip.

Comparing the total number of transitions in each parallel strip for the DES and SD modellers (Table 8.5b), it is observed that for DES modellers the higher numbers are at the top of the table, representing the most central strips in Table 8.5(a). This implies that the transition of attention for DES modellers focuses mainly in the most central strips in the matrix, hence representing a relatively more linear process compared with that for SD modellers. However, for SD modellers it can be observed that the higher totals are found at the bottom of the table, representing cells furthest away from the centre strips in Table 8.5(a). This suggests that the SD modellers switched their attention in a more vertical pattern compared with the DES modellers. It should be noted that an almost equal total number of transitions among topics has been found for DES and SD modellers. Based on the comparison of the transitions per linear strip, it can be concluded that DES modellers' attention progresses relatively more linearly among modelling topics than that of SD modellers.