CHAPTER 21

EVALUATING SCENARIOS BY SIMULATION

Professor Pericles Loucopoulos

Chair of Iinformation Systems Engineering, Manchester, UK

TYPE OF PROJECT

Organisational planning systems. The business context is that of Enterprise Integration to co-ordinate decisions and actions among subcontractors, individuals, and systems. The integration is along a process dimension. This implies a need to understand task-level interactions between people, departments, services, and companies to deliver appropriate system interconnectivity.

APPLICABILITY

The approach adopted in this project has general applicability. The modelling approach, using well-known principles and the simulation techniques related to these models, is suitable for most projects and particularly those that adopt a process-oriented, customer-driven philosophy.

POSITION IN THE LIFE CYCLE

ROLES PLAYED BY SCENARIOS

Scenarios were used extensively in stakeholder workshops. These workshops ranged from small, functional area-specific (typically 2–3 people) to participants from all functional areas (typically 30–50 people). The greatest contribution of scenarios was in identifying key factors influencing the performance of system components and of the overall system. Scenarios were used

• in validating stakeholders' assumptions relating to their requirements and
• in testing alternative realisations of system functionality.

STRENGTHS

Scenarios through simulation proved to be the only technique that worked satisfactorily in engaging all stakeholders to discuss and agree a complete set of requirements for each venue. Two other techniques, peer-to-peer knowledge transfer facilitated by expert consultants and business process modelling, were also tried but proved to be problematic.

WEAKNESSES

The approach required a careful construction of system models, prior to simulation, with the inclusion of relevant parameters to control system behaviour. Experts from different areas (e.g. security, accreditation etc) need to be involved in the modelling process and in defining these critical parameters. During scenario evaluation, the participation of such experts is critical in gaining confidence in the model and in defining realistic sets of requirements.

CASE STUDY

Background to the Case Study

The purpose of the RE project was for ATHOC to develop specifications for Venue Operations that would then serve as the basis for the delivery of systems by external contractors. Venue operations concerns the support components that need to be put in place at each venue so that it can function according to the specifications set by the International Olympic Committee. Therefore, the term venue operations system is defined as being composed of hardware, software, people, rules, and procedures or any combination of these components, interacting in space and time reflecting the dynamic behaviour of its underlying structure. The design of this system needs to address both its functional requirements (the resources and procedures for their management) and its non-functional requirements (the quality of service provision). Specifying the requirements prior to the design is the concern of stakeholders from 27 functional areas, the primary ones being accreditation, security, technology, transportation, spectator services, venue staffing, logistics, catering, sponsors, ticketing, and broadcasting.

As a general rule, Organising Committees for the staging of the Olympic Games are established a few years prior to the staging of the Games. Whilst at the outset the structure of an Organising Committee is strictly hierarchical and centred around individual functional areas, this has to be gradually transformed, as the Games approach, to a venue-based process orientation in order to shift the emphasis away from internal organisational efficiency towards venue operation efficiency. While designing venue operations, members of the Organising Committee from different functional areas form teams whose function is to manage the way that venues are run during the Games. This completes the transformation from a fully hierarchical, centralised structure to a venue-based, process-oriented, and distributed organisation.

Traditionally, the problem of designing venue operation specifications was approached by organising workshops, with the participation of representatives from various functional areas and experts from organising committees of previous Olympiads. These workshops were based on brainstorming sessions and focus group discussions, and the major information management tools used were text documents and architectural diagrams of the various venues. These workshops undeniably facilitated the exchange of knowledge among the functional area representatives involved, as well as the transfer of experience from one host city to the other. However, they were not sufficient for developing a common understanding among all participants, due to the lack of a common reference model and the significant variations in experience and background among the participants. The whole process under this approach is far too informal.

The closest substitutes for a common reference model available involved the use of architectural plans and representations of venue physical layouts. This practice imposed various constraints on the effectiveness of the workshops, the most important being the exclusive focus on specific operations taking place at specific venues. This fact prevented the participants from fully understanding the implications of service specifications in terms of resource requirements and resulting levels of service.

These shortcomings were overcome by adopting a systematic and systemic approach to requirements elicitation, definition, and validation. This approach was adopted in seven different applications within the broad area of venue operations, as shown in Table 21.1. The effort for these projects involved approximately 160 person/months effort from dedicated Requirements Modellers and Domain Experts.

The RE process was motivated by a desire to elicit, to model, and to reason about early requirements, requirements that exhibited the following characteristics:

• They focused on many co-operating systems rather than a single system.
• They were not confined to just software requirements.
• They were driven by a need to meet certain levels of desirable service to ‘customer groups’.
• They were owned by multiple stakeholders who set the levels of service and need to reach an agreed set of functionality due to limited resources and therefore inability to meet every single individual desirable level of service.

The applications shown in Table 21.1 focused on specific problems areas, such as printing and distribution of competition results, or athletes' transportation, but also on broader problem areas such as the dynamic profiling of an entire venue for the duration of a whole day. These applications ranged from simple processes involving a small set of actors and resources to complicated ones involving venues with multiple stadiums and many thousands of ‘customers’. Customers may be spectators, athletes, Games staff, Olympic family, broadcasters, journalists, and many others.

TABLE 21.1 RE Projects at ATHOC

Project	Description
Accreditation	Verification of accredited personnel during their arrival at various sites.
Venue staff arrival	Arrival of venue staff at site, security check, and check-in.
Results distribution	generation of competition results (hardcopy) and distribution within a venue.
Athletes' transport	Transportation of athletes from the Olympic village to and from various other venues.
Simple venue profile	Systems for a single venue for an entire Games day.
Complex venue profile	Systems for a multiple venue, including common areas, for an entire Games day.
City operations	Control of pedestrian flow around major venue sites during arrival and departure of spectators.

The RE Framework

The particular characteristics of the Olympic Games as a large-scale project and the overall complexity of designing venue operations create a number of issues in terms of both methodology and practice. Such issues included the systematic and reliable representation of a system of this magnitude, the ability for rapid exploration of multiple scenarios to test different hypotheses, and the cooperative and multifaceted environment in which these activities are to take place.

To support the process of RE, an approach that involved four interrelated activities, as shown in Figure 21.1, was adopted.

These activities were bound together by the use of a single generic approach capable of dealing with dynamic complexity of systems in order to understand the implications of interconnected components of venue operations. The approach was based on System Dynamics (Andersen, Richardson and Vennix 1997, Forrester 1961, Richmond 1997, Sterman 2000, Vennix 1999) as the underlying theoretical baseline. An advantage of the chosen paradigm is that the structure of a model of a system defines the system's behaviour that can be used to show how a change in any stage of the process can propagate to all subsequent stages. The language based on stocks and flows (Mass 1980, Richmond 1985, 1998), offered the possibility to develop maps that represented interdependent factors, causality, non-instantaneous impacts, and non-linear impacts. Furthermore, the models thus developed provided the ability to develop graphs of behaviour over time. The time frame for a single venue was a competition day and these were aggregated for the control system into a time frame spanning the entire duration of the Games.

Application Ontology Modelling

The basic concepts found in venue operations were grouped in the following four categories:

FIGURE 21.1 A requirements engineering framework

• Events: Events represent the temporal aspect of the ontology but also the one around which all the others revolve, as it is the trigger and the point of reference for the entire Games system.
• Venues: Venues represent the spatial aspect of the application. Venue knowledge is critical because topology heavily influences the way in which Games-related processes operate but also because planning traditionally revolves around architectural designs.
• Customers: There are 15 different customer groups that are involved in the venue operations system. A clear definition of each customer group and of its characteristics is necessary for understanding the different requirements that need to be satisfied by the organisers.
• Services: The services aspect details the functional areas involved and the resources required to support the various customer activities.

Application ontology modelling addressed the following questions:

1. What is the boundary of the system?
2. Who are the ‘beneficiaries’ of the system? In other words, who uses the system and for what purpose?
3. What are the different types of support that these users need in order to achieve their goals?

The answer to the first question defined the RE project space, which is presented as a conceptual model in Figure 21.2. This model defines the processes that the various functional areas will have to establish (lower shaded rectangle). These processes will manage resources established in order to provide a level of service previously agreed and that the implementation will have to achieve (middle shaded rectangle). The purpose of these resources is in supporting customer processes (upper shaded rectangle).

This brings forth the second question, namely defining the ‘beneficiaries’ of the system, the different customer groups of venue operations. A customer group is a specific category of Games participants, with a well-identified role and, therefore, characteristics that distinguish them from any other group. One such group is that of spectators, by far the largest and with a wide variety of needs, ranging from transportation to/from a venue to the ability for on-site purchase of tickets, food, and memorabilia. Another important customer group is of course that of athletes and team officials, a group that is at the focus of attention during the Games, and whose needs have a very high priority. There are 15 customer groups in total, including among others broadcasters, paid staff, volunteers, international federations, and the International Olympic Committee.

The answer to the third question defines the various types of service that ATHOC must put in place so that each customer group can successfully complete its mission in the Games. These services vary depending on the customer group for which they are intended, the time and location at which they are provided, and so on. One such service is security checking, which all customer groups must undergo before they can access any venue area. Because of the importance accorded by ATHOC to security issues, this is considered to be a key service and its successful implementation essential for the smooth functioning of the venue operations system. While security checking is uniformly provided to all customer groups, there are services that can be specific to one group only. The check-in of paid staff upon their arrival at a venue is such a service; it enables the implementation of the defined shift schedule and is therefore necessary for venue staff, yet it is invisible by all other customer groups.

FIGURE 21.2 The ATHOC RE project boundary

The notions of customer group and service type are closely interlinked because one helps define the other. Each customer group is primarily characterised by its needs and by the services that satisfy these needs, while the requirements (both functional and non-functional) of each service type are defined with reference to the respective customer group. The notion of customer, in particular, should not be perceived as purely passive. As all categories of Games participants are by definition involved in various activities, it is possible for a customer group to be a service receiver at a specific instant and subsequently become a service provider. Paid staff, for example, are serviced during their check-in, while later on they service spectators at various points inside the venue.

Stakeholders' Goal Elicitation

Goal modelling is about describing the causal structure of a system (be it a business system, or a software system, etc.), in terms of the goals-means relations from the “intentional” objectives that control and govern the system functions to the actual “physical” processes and activities available for achieving these objectives. Goal modelling aims at providing the means for describing the purpose of the system under consideration, why it came into being (Dardenne, Lamsweerde and Fickas 1993, van Lamsweerde 2001).

In eliciting the goals for the venue operations system, the aim was to understand what determined the successful operation of the system. This involved helping the various stakeholders externalise the (sometimes implicit) goals that they had, capturing these goals, and synthesising that knowledge with information from other sources, such as existing documentation, abstract descriptions of various systems and procedures, and so forth. Stakeholders' goals were thus an initial, high-level expression of system requirements viewed from the perspective of ATHOC, that is, the service provider (as opposed to that of the user).

With respect to goal categorisation, it was found that it was often relatively straightforward to capture goals about the functions that the system should provide (i.e. the functional requirements), while in most cases it was difficult to accurately define goals regarding the quality of venue operations. In both cases, the multitude of stakeholders (i.e. functional areas) involved in the requirements specification often resulted to competing, and sometimes clearly conflicting, goals about the system. Furthermore, it was especially difficult for stakeholders to express their goals in specific (i.e. measurable) terms. Indeed, while each functional area found it relatively easy to identify distinct functional/quality aspects of the system, it was much more difficult to quantify each of these aspects. This difficulty unfailingly complicated subsequent stages of system design because it had a decisive influence on the type and amount of resources required, and by extension on the final cost of the system.

To deal with the complex situation of goal elicitation a stepwise, cyclical approach was adopted, starting from high-level, sometimes fuzzy, goals. These goals were refined with the help of the affected functional areas. By modelling venue operations system processes, and by testing different scenarios on how quality goals can be implemented in each of these processes, we identified different ways of refining goals into specific quality requirements.

An example of a high-level goal that all functional areas expressed, irrespective of customer group and service type, was

This translates into goals such as

This type of goal does not translate very well into operational terms, because it does not specify a concrete target for the waiting time. To complicate matters further, there is not a single acceptable waiting time as that depends on the service type and the customer group for which it is intended. What is acceptable for spectators or staff, for instance, may not be acceptable for members of the Olympic family or for athletes. Therefore, the first question that had to be address in order to refine this goal was:

In other words, what is the level of service that each functional area is aiming to offer to the customers that will service? Will the functional area be happy with 30 seconds waiting time or with 15 minutes? In some cases even that answer was not ready, so it had to be negotiated.

A different type of high-level goal was expressed with respect to the overall presence of spectators in a venue. Given that a venue may hold more than one event (e.g. competition session) during a day, at any time there may be spectators arriving at the venue area for one of the upcoming sessions, spectators leaving the venue from one of the past sessions, and spectators participating in a current session. The total number of spectators present has to be somehow controlled for practical reasons such as the availability of resources (e.g. space), but also due to safety concerns. This translates into the goal

Again this is an abstract goal that needs to be made more specific; to refine it, the stakeholders examined the factors influencing the presence of spectators in the venue and their distribution in the various areas of which it consists. These factors included the competition schedule at each venue, the transportation capabilities to/from the venue, the availability of open spaces and/or service areas within the venue, and so forth. Addressing issues such as those concerning these two high-level goals was the first step towards visualising an operational system.

Process Modelling

During the RE stage, process modelling concerns the analysis of high-level goals into operational requirements. In our approach, this analysis engages the use of System Dynamics in describing the business processes and relating them to the stakeholder goals (Loucopoulos 2003).

To demonstrate the approach, consider again the example application.

There was a wide range of process-related problems to be studied while addressing the issue of venue operations. At one end of the spectrum, there were problems with ‘local’ impact, that is, affecting a single customer group, a small area of the venue, and a small part of venue resources (workforce, machinery, consumables). At the other end of the spectrum, there was the problem of the ‘behaviour’ of an entire venue as a complex, interconnected system. This corresponds to process models focusing on the dynamic profiling of all venue components, over an extended time frame (e.g. an entire day of the Games), possibly with respect to the needs of more than one customer group.

A distinguishing feature of this type of situation is the large number of different service types that the model must represent, since the behaviour of the venue operations system is affected by each of these service sub-components. As a result, the degree of complexity in the resulting process model rises dramatically.

Examples of the problems studied include

• Staff arrival at venue site and check-in
• Verification of accredited personnel at various sites
• Printing and distribution of competition results
• Transportation of athletes from the Olympic Village to various venue sites
• Spectator profiling at the main Olympic Complex.

The first facet of the venue operations system, that is, the behaviour of specific components of the system is examined by model components like the one presented in Figure 21.3. This fragment is about the various types of service facilities that are available to the spectators' customer group inside the Olympic Complex. These facilities service needs such as buying food or memorabilia, withdrawing money from an ATM, and so on.

The behaviour of the services component is determined by two issues, the demand for each type of service and the supply offered by the service provider. The demand is determined in part through the ‘pct of specs per service type’ variable, which expresses the number of customers expected at each type of service facility per unit of time as a percentage of total spectator presence. Total spectator presence depends on overall spectators' behaviour in the venue area, which interacts with this model fragment through a number of feedback loops (not shown here due to the complexity of the complete model).

The supply is determined by two parameters: the number of ‘Service Points’ available (e.g. 10 stands selling food), and the ‘specs per channel per minute’ service rate (e.g. two spectators serviced per service point per minute). According to this representation, spectators arrive at the service facility (‘going to facilities’), queue there for a while if no service point is available (‘Specs Queue at Facilities’), and eventually get serviced (‘servicing’).

FIGURE 21.3 Model fragment describing spectators' service facilities

Using this model fragment it is possible to elaborate on the way that stakeholder goals were refined through the use of process modelling. As stated already the high-level goal was to

The realisation of this goal for a given type of service facility, and for a given demand, depends on the availability of supply for that facility. Supply is composed of two independent factors, the number of service points and the service rate. Therefore, the initial goal was decomposed into two complementary (i.e. non-competing) goals,

These goals are more accurate than the initial one, however, they need to be analysed further in order to become quantifiable.

The second facet of the venue operations system, that is, the overall behaviour of the system, is the result of composing smaller system components (such as the services component) so as to build the complete system model. For instance, a summarised version of the model describing spectators' behaviour at the Olympic Complex is presented in Figure 21.4. The full version of the model contains a significant number of feedback loops and its behaviour is controlled by about 600 equations.

FIGURE 21.4 Model regarding overall spectators' behaviour in the Olympic Complex (summary)

Each of the main stages of the process (‘Entrances’, ‘Venues’, ‘Services’ etc.) corresponds to a detailed model component like the one presented in Figure 21.3, describing the sub-processes taking place at the respective venue area. One can see the interactions between these sub-processes. The behaviour of the ‘Services’ component, for instance, (i.e. its servicing capacity) influences the system as it determines the number of people that are queuing there and thus not participating in activities elsewhere. As another example, it is clear from the model that spectator arrival rate at the ‘Entrances’ and departure rate at the ‘Exits’, determines the number of people circulating in the common domain of the complex. Moreover, these rates affect the availability of spectators to fill the venues, and they also affect the demand at the service facilities. Therefore, the high-level goal

could be achieved (partly at least) through the more specific goal

Scenario Generation

The generation of different scenarios concerning each problem studied, and the simulation of these scenarios with the help of the process models developed, is an essential part of requirements definition. In the ATHOC case study, scenarios are an indispensable tool for truly understanding the implications of stakeholders in their deliberation of requirements. As the models were being developed and the stakeholders were becoming more aware of the different factors influencing each problem, the range of possible values for each of these factors became more evident, thus creating the initial ideas for different scenarios.

In the model of Figure 21.3, for example, scenario formulation focused on the three variables defining demand and supply for each service facility, namely the percentage of spectators expected for each type of service (demand), the number of service points per service type and the respective service rate (supply). Other relevant factors, such as spectators' arrival and departure patterns, were taken into account. The stakeholders involved in scenario generation investigated the range of probable values for each of these parameters, as well as some ‘extreme’ values that were less probable but worth investigating nonetheless. Each scenario was characterised by the values of all independent variables; the number of possible scenarios thus depended on the number of their feasible combinations.

Figure 21.5 presents the results of the simulation for a scenario concerning the ‘Merchandising’ service in one of the four areas of the Olympic Complex. The demand is set at 15% of spectators using merchandising services, while supply is provided by 32 points of service, and a service rate of 1.5 minutes per customer. Graphical results include total spectators waiting to be serviced at each moment (blue curve), and the corresponding waiting time (red curve), while numerical results include the mean and maximum waiting times, as well as the number of spectators served throughout the day. In this particular scenario, a total of 8526 spectators were serviced, with a mean waiting time of 4.7 minutes and a maximum waiting time of 29 minutes (numbers shown in the green numerical indicators).

FIGURE 21.5 Simulation results for the ‘merchandising’ service

The results of scenario simulation helped the stakeholders realise the implications of their design choices in terms of the service level provided to customers. This realisation in turn contributed to a quantification of the goals that each functional area set for the final system, for example,

Simulating the behaviour of the venue operations system overall yields results like those presented in Figure 21.6. The screenshot shows the profile of spectators' behaviour for the entire day at four key points of the complex: arrival, presence in the common area, presence inside venues and, finally, departure. According to the scenario presented here, spectators arrive to the Olympic Complex during the two hours preceding each competition session and leave the complex during the two hours following the session. This gives spectators time to stroll in the common domain of the complex, to visit service facilities, to be at a specified venue in time for the corresponding session, and to leave the complex in an orderly fashion. At the same time, a key goal was to ensure that the total number of spectators in the common domain of the complex was kept at a relatively comfortable level. In other words, the goal

FIGURE 21.6 Overall spectator profiling at the Olympic Complex

was quantified in terms of the two goals

The models were subjected to testing through simulation sessions, in workshops involving from 5 to as many as 40 participants. In all workshops, the models were presented to project stakeholders together with the corresponding scenarios and simulated runs. As most of the participants were not familiar either with RE methodologies or with the system dynamics way-of-thinking, their initial reactions ranged from excitement to disbelief. However, even sceptical participants soon realised the advantages of a visual yet operating model consisting of interacting components, and the power of rapid scenario development and simulation.

These features enabled stakeholders to reach a consensus about the underlying processes and the implications that each choice would have on overall system behaviour. The first type of result, that is, results concerning specific components of the system, helped to answer operational questions concerning the rational allocation of resources and the resulting service provision capabilities of the system. The second type of result proved useful for understanding the overall behaviour of a venue, thus answering higher-level, management questions concerning customer presence and distribution, arrival and departure patterns, and so on.

LESSONS LEARNT

Requirements engineering is considered by many as the most critical of all development activities for socio-technical systems. In most cases, different stakeholders are involved with different experiences, backgrounds, goals for the system, and so on.

Venue operations was a typical planning and design problem of a class of problems that has been termed “ill-structured problems” (Reitman 1965, Rittel and Webber 1984, Simon 1984). The problem state is not a-priori specified and there is no definitive formulation. Formulating the problem amounts, in large part, to solving it. While each group of stakeholders from each functional area at ATHOC had a general idea of their individual tasks, complexity arose from the need to coordinate activities across all the functional areas in a venue.

Initially, stakeholder workshops, facilitated on the basis of past experience, were only partially helpful. Architectural and topological designs imposed constraints on thinking about customer-oriented service provision. Textual requirements specification resulted in voluminous documentation with little chance for proper agreement, estimation of resources and planning for a coordinated implementation.

The advantages of a conceptual modelling language for representing system structures and behaviour over informal, natural language descriptions are well documented (Bubenko 1979, Loucopoulos 1992, Mylopoulos 1992). The goal of developing conceptual models of venue operations was to gain insights into the problem and through this to arrive at an agreed set of requirements. This was in the context of an organisational setting that, as well as the usual facets of time constraints, interpersonal conflicts, organisational politics, ambiguities, and so on, had the almost unique characteristic that the organisation itself was transient.

However, eliciting and developing maps of stakeholders' mental models were not sufficient by themselves for achieving stakeholders' agreement. Models needed to be subjected to ‘testing’ in order to understand the implications of changes to a system component on the overall behaviour of the system. Such a testing was achieved through simulation activities. Simulation of models was a necessary component for developing scenarios, and this way of working proved to be invaluable in experimenting with alternative solutions and to encourage cooperative design in multiple workshop sessions.

The field of scenarios has been a fertile one for many types of application, from industrial decision-making (Chindemi et al. 1998) to medical applications (Dangerfield, Fang and Roberts 2001, Georgantzas, Batista, Demos and Ames 2000), finance (LaRoche and Kohn 2000), human computer interaction (Carroll 2000), software development (Abdel-Hamid and Madnick 1991), and requirements engineering (Carroll 2002, Filippidou and Loucopoulos 1997, Potts, Takahashi and Anton 1994). A common feature of scenarios in all these domains is their use in examining alternative future situations. (See also the discussion of ‘Situation, Alternative World’ by Ian Alexander in Chapter 1, Introduction—Scenarios in System Development, this volume; and David Bush's account of Requirement Stability through Alternative Scenarios in Chapter 6 of this volume.)

According to Carroll, scenarios support the way experts work on ill-structured problem settings such as planning and design (Carroll 2002). In our application, scenarios encouraged group brainstorming through which participants could focus on alternative solutions and envision system behaviour prior to its implementation. Scenarios helped us define and agree desirable levels of service for venue operations, following much experimentation. This confirmed the findings of several empirical studies on the cognitive nature of design, that have shown that expert designers develop sub-solutions to help understand large problems (Darke 1978, Malhotra, Thomas, Carroll and Miller 1980, Rowe 1987). Our stakeholders first defined what they thought might be important aspects of the problem. They subsequently developed tentative designs in scenario analysis sessions to find out whether more could be discovered about the problem. The design paradigm resembled the ‘generator-conjecture-analysis’ paradigm (Hillier, Musgrove and O'Sullivan 1984).

This caused a fundamental change in the way stakeholders were working. The gradual shift of emphasis from informal to reflective was based on the realisation that there is no well-founded route from problem setting to problem solving, but there is a continuous interaction between the two.

Methodologically, the framework adopted in this project supports a ‘solution-first strategy’ (Carroll 2002) to requirements definition. Analysis guides design and design guides analysis—and all in an effort to gain an understanding of the problem, of the situation in hand.

KEYWORDS

Early Requirements

Simulation

Quantitative Models

Dynamic System Modelling

Behavioural Scenarios.

Alternative Scenarios

Alternative Future Situations

REFERENCES

Abdel-Hamid, T.K. and Madnick, S.E., Software Project Dynamics: An Integrated Approach, Prentice Hall, Englewood Cliffs, NJ, 1991.

Andersen, D.F., Richardson, G.P., and Vennix, J.A.M., Group model building: adding more science to the craft, System Dynamics Review, 13(2), 187–201, 1997.

Bubenko, J.A. Jr, On the role of understanding models in conceptual schema design, Proceedings of 5^th International Conference on Very Large Data Bases (VLDB), Morgan Kaufmann, Rio de Janeiro, Brazil, 1979.

Carroll, J.M., Making Use: Scenario-Based Design of Human-Computer Interactions, MIT Press, Cambridge, MA, 2000.

Carroll, J.M., Scenarios and design cognition, in E. Dubois and K. Pohl (Ed.), Proceedings of IEEE Joint International Conference on Requirements Engineering (RE'02), IEEE Computer Society, Essen, Germany, September 9–13, 2002, pp. 3–5.

Chindemi, M., Manca, S., Marcello, G., Turatto, R., and Ventola, N., Evolutionary scenarios in power generation. Modeling competitive effects in the Italian electricity market, Proceedings of 16^th International Conference of the System Dynamics Society, Quebec '98, System Dynamics Society, Quebec City, Canada, 1998, pp. 25.

Dangerfield, B.C., Fang, Y., nd Roberts, C.A., Model-based scenarios for the epidemiology of HIV/AIDS : the consequences of highly active antiretroviral therapy, System Dynamics Review, 17(2), 119–150, 2001.

Dardenne, A., van Lamsweerde, A., and Fickas, S., Goal-directed requirements acquisition, Science of Computer Programming, 20 (1-2), 3–50, 1993.

Darke, J., The primary generator and the design process, in W.E. Rogers and W.H. Ittelson (Eds.), Proceedings of New Directions in Environmental Design Research (EDRA9), EDRA, Washington, DC, 1978, pp. 325–337.

Filippidou, D. and Loucopoulos, P., Using scenarios to validate requirements in a plausibility-centred approach, in A. Olive (Ed.), Proceedings of 9^th International Conference on Advanced Information Systems Engineering (CaiSE '97), Springer-Verlag, Barcelona, June 16–20, 1997.

Forrester, J.W., Industrial Dynamics, Productivity Press, Cambridge, MA, 1961.

Georgantzas, N.C., Batista, A., Demos, D., and Ames, T., Renal care dynamics, Proceedings of 18^th International Conference of the System Dynamics Society, System Dynamics Society, Bergen, Norway, August 6–10, 2000, p. 78.

Hillier, B., Musgrove, J., and O'Sullivan, P., Knowledge and design, in N. Cross (Ed.), Developments in Design Methodology, John Wiley & Sons, 1984, pp. 245–264.

LaRoche, U. and Kohn, L., Prediction of exchange rates, Proceedings of 18^th International Conference of the System Dynamics Society, System Dynamics Society, Bergen, Norway, August 6–10, 2000, pp. 123–124.

Loucopoulos, P., Conceptual modelling, in P. Loucopoulos and R. Zicari (Eds.), Conceptual Modelling, Databases and CASE: An Integrated View of Information Systems Development, John Wiley & Sons, New York, 1992, pp. 1–26.

Loucopoulos, P., The S3 (strategy-service-support) framework for business process modelling, in J. Eder, R. Mittermeir, and B. Pernici (Eds.), Proceedings of Workshop on Requirements Engineering for Business Process Support (REBPS '03), Klagenfurt/Velden, Austria, June 17, 2003, pp. 378–382.

Malhotra, A., Thomas, J.C., Carroll, J.M., and Miller, L.A., Cognitive processes in design, International Journal of Man-Machine Studies, 12 (2), 119–140, 1980.

Mass, N.J., Stock and flow variables and the dynamics of supply and demand, in J. Randers (Ed.), Elements of the System Dynamics Method, Productivity Press, Cambridge, MA, 1980, pp. 95–112.

Mylopoulos, J. Conceptual modelling and telos, in P. Loucopoulos and R. Zicari (Ed.), Conceptual Modelling, Databases and CASE: An Integrated View of Information Systems Development, John Wiley & Sons, New York, 1992, pp. 49–68.

Potts, C., Takahashi, K., and Anton, A., Inquiry-Based Scenario Analysis of System Requirements, Technical Report GIT-CC-94/14, College of Computing, Georgia Institute of Technology, Atlanta, GA, January 1994.

Reitman, W.R., Cognition and Thought: An Information Processing Approach, John Wiley & Sons, New York, 1965.

Richmond, B.M., STELLA: Software for bringing system dynamics to the other 98%, Proceedings of the 1985 International Conference of the Systems Dynamics Society, Keystone, Colorado, 1985, pp. 706–718.

Richmond, B., The strategic forum: aligning objectives, strategy and process, System Dynamics Review, 13(2), 131–148, 1997.

Richmond, B., Operational thinking, The Systems Thinker, 9(2), 6–7, 1998.

Rittel, H.W. and Webber, M.M., Planning problems are wicked problems, in N. Cross (Ed.), Developments in Design Methodology, John Wiley & Sons, 1984, pp. 135–144.

Rowe, P.G., Design Thinking, MIT Press, Cambridge, MA, 1987.

Simon, H.A., The structure of ill-structured problems, in N. Cross (Ed.), Developments in Design Methodology, John Wiley & Sons, 1984, pp. 146–166.

Sterman, J.D., Business Dynamics: Systems Thinking and Modeling for a Complex World, Irwin/McGraw-Hill, Boston, MA, 2000.

van Lamsweerde, A., Goal-oriented requirements engineering: a guided tour, Proceedings of 5^th IEEE International Symposium on Requirements Engineering, RE '01, Springer, Toronto, Canada, 2001, pp. 249–263.

Vennix, J.A.M., Group model-building: tackling messy problems, System Dynamics Review, 15(4), 379–401, 1999.