Chapter 9

Conclusion: What Return on Investment Can We Expect from Simulation? 1

9.1. Returns on simulation for acquisition

Throughout the previous chapters, we have seen how simulation is used as much in the production of models as in their use for a given objective. Simulation allows us to reproduce characteristics of the environment, systems, and certain behaviors. We have gone into detail on techniques, specificities in terms of formats and standards, and the limitations and pitfalls of interpretation. In addition to this descriptive aspect, simulation allows us to control conditions and situations and thus to test solutions to given problems. This allows a certain level of flexibility, security, and cost reductions not possible using real experiments (e.g. 30 years ago, approximately a hundred real launches were required to test a missile; today, the number of real launches required is considerably smaller). Simulation thus provides precious help in planning equipment, doctrines of use and operation, as well as in training.

Beyond the capacity to model and to give life to these models, which sometimes requires considerable processing power (i.e. non-negligible levels of investment), simulation has become the key to success in the engineering process, a bridging tool for working in integrated multi-disciplinary teams on problems of integration of complex systems. Different actors are involved in defining, evaluating, producing, and supporting the project, and they must share information and data produced during tests and the simulations necessary for their own evaluation, validation, qualification, and preparation activities. To do this, they must also identify the information necessary in terms of tests and simulation as early in the process as possible. This is, moreover, the way simulation-based acquisition (SBA) was initially defined by the US Department of Defense: a complete acquisition process placed at the service of project owners and managers, with robust and collaborative use of simulation technologies in a coherent and integrated manner throughout the acquisition phases of programs.

The basic idea is to show that simulation is the essential tool for mastering the complexity of systems throughout their life cycle. This logic consists of using and reusing simulation tools and the available technologies on the entire operational breakdown of the system, during project phases and between various projects (particularly for systems of systems). This is paired with global management of modeling and simulation resources throughout the acquisition process: from simple occasional (in terms of time and space) support to program engineering, we may obtain a coherent and integrated process.

We can therefore put simulation into perspective: simulation is used in the acquisition of systems as parts of systems of systems, with a capacitive vision, not simply restricted to replacing components one by one. The virtual aspect is widely exploited, so certain architectural or technological choices can be left as late as possible in the project. To fully reap the benefits of using simulation in this way, a few principles should be considered:

– apply a descending approach to global design;

– verify conformity to specifications via an ascending approach focused on tests and integration issues;

– identify key milestones and plan in advance for the whole life cycle;

– use integrated multi-disciplinary teams with a user-oriented approach;

– manage the evolution of system configuration and roll out technico-economic analysis on all compromises;

– use modeling and simulation as a computer-assisted tool for all choices, accompanied by parametric analyses.

The approach as a whole is characterized by iterative adjustment of virtual prototypes immersed in realistic synthetic environments, which on the one hand helps to develop a shared vision of the emerging system and on the other hand provides appropriate means for a better understanding of complex interactions between elements of the system configuration. By having design, construction, and test engineers work together, the prototypes created will be more easily constructed and evaluated, leading to reductions in the global acquisition cost.

These gains may be quantified, and we shall discuss this fact in the following sections. Through the use of simulation in system acquisition, we can reduce a priori promises of investment based on decisions made (the Pareto curve is often used in this respect: it shows that after 20% of effective spending, decisions will be made which commit 80% of total investment). This results from the fact that simulation allows simultaneous management of a large number of technical alternatives, for which portions of the life cycle of varying lengths can be played out virtually, and the impact of an early decision can be measured later on in terms of performance and/or cost. From this point of view, simulation essentially contributes to the management of the project’s risk portfolio.

The other factor for economic gain lies in the reuse of simulations between systems within a meta-system; this reuse should not be limited to simple software “bricks”, but it should cover requirements, architectures, design patterns, interface models, test plans, data, documentation, and so on. Various cost studies over the last few years have allowed us to appreciate the true value of this factor.

We are thus able to show [LUZ 03] that well-managed simulation for acquisition is financially worthwhile from the moment the first incident occurs in the course of a project. Reuse between projects (when well managed) allows savings of a similar order of magnitude to the cost corresponding to the design of the system including individual systems. These first technico-economic analyses show strong reasons to invest, and the first figures produced by processes currently underway confirm the results of this theoretical analysis.

9.2. Economic analysis of gains from intelligent use of simulations

The first way in which simulation creates savings during system acquisition stems from the greater freedom we have in terms of outlay which is “frozen” by decisions made during the cycle. Simulation allows a wider range of technical alternatives to be considered: for example, successive portions of the cycle may be analyzed for several options simultaneously, and the impact of an early decision can be measured later on in terms of performance and/or cost.

This increased freedom should be compared with the added cost of developing and engineering the simulations themselves. To quantify this, we shall compare full project costs (from exploration of a concept to production) for a “standard” acquisition and an acquisition using an SBA process. We will then quantify the use of the increased degrees of freedom obtained through the use of an SBA process while considering the added acquisition costs linked to unpredicted events or planning errors.

Note that we have not included usage costs (launch and support), as these costs – particularly for defense systems, which are used over several decades – would skew the analysis; simulation creates savings in a clear and recognized manner, and usage costs form the largest part of global possession costs (the US Department of Defense estimates that the costs of acquisition, use, and support follow a ratio of 28-12-60). Savings from simulation use will therefore increase the longer the system is used. As the sections of the budget devoted to acquisition and use are often independent, we have chosen to maintain the separation between both these phases and concentrate on acquisition to demonstrate the interest of simulation use in this particular phase.

9.2.1. Additional costs of the SBA

We shall use the usual curve1, which is familiar to all project managers. This represents cumulative acquisition costs over different phases of a cycle.

The values shown in Figure 9.1 are merely indicative and may vary somewhat from one source to another, but these differences are a matter of units (this is the case, e.g. in the program director guides published by the French and US defense ministries; these values also depend on the milestone used to delimit different phases).

The part played by simulation in the acquisition process can vary greatly, as much in the feasibility phase as in development, depending on the use of different virtual prototypes and more generally on the degree to which a true SBA process is used. When a process of this type is applied, simulations involved in different acquisitions may be reused as much as possible and possibly included in a knowledge management infrastructure, with validation information for reuse in appropriate contexts without need for redevelopment. In an ideal example of SBA, the cycle can be even run virtually following a spiral logic, leaving the creation of solid prototypes as late as possible.

Figures 9.2 and 9.3 show the proportional contribution of simulation over different phases of the acquisition cycle: preparation, feasibility, definition, development, and construction. Figure 9.2 shows the data for an SBA process and is taken from the final report of the NMSG003 group, operating within the NATO/Research and Technology Organization (RTO) [NAT 03]. For Figure 9.3, we considered a standard process, and the data used is taken from normal practice.

In both cases, simulation is used almost exclusively during the preparation phase (analysis of needs and concept exploration) and in design (feasibility and definition). This is due to the use of virtual prototypes (which should include construction constraints from the beginning, as in the automobile industry), on the one hand, and to reuse on the other (models used during the preliminary and definition phases should be reused during development).

Before continuing with our demonstration, we shall make three remarks and clarify some aspects of notation.

REMARK 1.– The SBA requires added effort in the design phase (to deal with questions of reuse and to respect different standards and requirements imposed by capitalization repositories), in validation (to reduce costs of later reuse), and in integration (vertically: from components to sub-systems and systems and horizontally: throughout the cycle) of each phase.

Table 9.1 [PRI 01] shows the ratios of different activities (design; implementation; test, verification, and validation (TV&V); project management; documentation; and configuration) during each phase of the traditional software life cycle (concept exploration and analysis of needs, general design, creation, including detailed design, coding and unit tests, integration and corresponding tests) as shown in software engineering literature. The value on the left corresponds to small independent projects, whereas the value on the right refers to complex projects involving interactions with operators and/or other programs, and it gives a good indication of what SBA requires in comparison with a “standard” simulation (used sporadically and independently).

In what follows, to use this data in the context defined in previous figures, we shall assimilate:

– concept exploration and identification of needs with feasibility;

– general design with definition;

– creation and implementation with development;

– integration and TV&V with construction.

To check the coherence of this data with the estimations given for simulation, we shall calculate the TV&V activity throughout the cycle dedicated to simulation in a SBA process. By referring to Figure 9.2 and considering an average value for each phase, we calculate the relative part (expressed as a percentage) played by simulation in the SBA process during the four phases of feasibility, definition, development, and construction is 100, 95, 75, and 60, respectively. To calculate the global value of TV&V, we must add, for each phase, the product of the proportional contribution of simulation, the relative effort for the phase, and the proportional contribution of this activity during each phase: 1 × 0.07 × 0.16 + 0.95 × 0.18 × 0.18 + 0.75 × 0.44 × 0.2 + 0.6 × 0.3 1 × 0.25, giving 15.2%.

If we refer to [NAT 00], the cost of verification and validation is estimated at 15% of the global cost of the simulation, thus demonstrating the coherence of our data.

REMARK 2.– The figures above show the effort required for simulation use during the acquisition process. We shall suppose this effort to be directly linked to cost: this is true in terms of human resources, but ignores the cost of raw materials. Consequently, our estimation for SBA will be relatively pessimistic, as we shall not explicitly consider the fact that SBA reduces the amount of hardware equipment prototyping in a process in favor of “virtual” prototypes. It could, however, be argued that the human cost of material prototyping is lower than that involved in the creation of virtual prototypes, due to the different levels of qualification required of personnel and current market conditions.

A detailed study should analyze the exact balance between these various factors: materials including buying and initial treatment of raw materials, software including initial investment for high-performance calculation, qualification and market price of personnel concerned. From our own experience, SBA seems to be the best option for large-scale projects. However, as we do not have the necessary figures to prove this, we shall suppose that these factors are balanced in the following paragraphs; in other words, we ignore this question (if data was available, it could be included either in the cost coefficient, which we will discuss later, or in the γ coefficient which will also be introduced in later calculations).

In the same way, the translation of cost in terms of human resources can be a detailed process, as different abilities are needed for different activities and the level of expertise of the team working on the problem. Detailed cost models, such as those provided by Cocomo2.0 [BOE 00], cover these aspects. However, we shall ignore these aspects, and in what follows, the coefficient for the conversion of effort to cost shall be equal to 1.

REMARK 3.– To keep our calculations simple, we shall suppose that all constant costs and percentages remain constant throughout a given phase (so the functions illustrated in the figures above can be approximated by simple functions using the mean value for each phase). We could consider more complex curves, but this would change very little apart from higher level corrections.

We shall also assume that the cost of the simulation is independent of the phase in the cycle and depends exclusively on the percentage of simulation activity in this phase. This hypothesis ignores horizontal software integration costs (which may arise during different phases of the cycle) and all costs connected with validating implementations of the simulation. These two factors only contribute at a higher order of magnitude, as seen in the previous table. To take them into account would require more detailed analysis of the SBA process, including all efforts linked to the creation and management of a data repository, models and simulations, added verification and validation costs, and all savings generated by reuse. Only this last point will be considered in the next section, devoted to multi-project acquisition.

Notation:

– RelPartSimi,j is the proportional contribution of simulation to activity i during phase j;

– RelPartSimj is the proportional contribution of simulation to phase j;

– RelPartActi,j is the proportional contribution of activity i to phase j;

– RelPartPhasej is the proportional contribution of phase j to the global acquisition of the system;

– GlobAcqCost is the global acquisition cost of the system;

– For all variables, the exponent std (or SBA) represents the context: “standard” acquisition (or SBA);

– Coeffj and Coeffi,j are coefficients that convert relative efforts such as RelPartSimi,j and RelPartSimj to costs (see Remark 2).

Let us now return to the main calculation: for a given phase, simulation is part of the process, and its cost is directly proportional to the part played by simulation during this phase. This gives us a first formula for obtaining the global cost of simulation in each acquisition process, by adding this contribution over each phase. To refine this model, we can give greater detail on what happens within each phase and calculate how simulation contributes to each activity within a phase. This gives us a second formula for global acquisition costs, obtained by adding all activities in a phase, then all the phases.

Thus, global simulation costs are calculated in one of two ways, depending on whether we use the first or the second, more refined, model:

[9.1]

[9.2]

Clearly, the added cost due to the SBA process is the difference GlobSimCostSBA – GlobSimCoststd. To calculate this, we must identify the different terms in the previous equations.

The values of RelPartSimj are given in Figures 9.2 and 9.3 (remember that we are using the average value for each phase). The values of RelPartPhasej can be deduced from Figure 9.1 (note that the figure gives cumulative costs, integrating the costs of the previous phase(s) for each phase; it is easy, however, to obtain individual costs, and therefore, their average value, to simplify calculations as stated above). The values of RelPartActi,j are given in Table 9.1 for both acquisition processes. The values of RelPartSimi,j are deduced from the RelPartSimk values as follows: we carry out identification as explained in Remark 1, which tells us which k and l correspond to which i and j, and we take RelPartSimi,j = RelPartSimk × RelPartSim;. All Coeffj and CostCoeffi,j are presumed to be equal to 1, as stated in Remark 3.

Finally, we use GlobAcqCostSBA = γ × GlobAcqCoststd, where γ is a coefficient taking various investments into account (see Remark 2), alongside global savings due to improvements in systems engineering following a successful SBA process2. As this factor could be seen as giving an unfair advantage to the SBA process and as we lack access to widely validated data, we shall assume that γ = 1.

Equation [9.2] gives lower values than equation [9.1] as various non-simulation activities (such as project management, documentation, and configuration) are identified explicitly. If we look more closely at the values presented in Table 9.2, the greatest differences appear in the values corresponding to the construction phase. This is not surprising as this phase contributes a good deal to global acquisition costs, and the associated value is therefore a significant part of the global result.

Using equation [9.1], the ratio between simulation cost for the “standard” process and the full acquisition cost is 8; for an SBA process, it is 22. Using equation [9.2], the ratio for the “standard” process is 3 or 12 for SBA.

Depending on the model used, then the additional cost incurred by using SBA is estimated between 9% and 14% of the full acquisition cost.

9.2.2. Additional costs from unexpected events or bad planning

Our reference curve (Figure 9.4) is the classic Pareto curve3, which gives the proportion of expenditure which is tied up (or “frozen”, i.e. their use is pre-defined) by decisions made throughout the acquisition process.

Figure 9.5 should also be familiar to any project manager; it shows the cost derivative, in terms of cumulated additional expenses due to unplanned modifications. This is, in fact, a curve of the type shown in Figure 9.1 stacked on top of a curve of the type shown in Figure 9.6 from the moment of the unplanned event.

One consequence of the SBA is the reduction of risks by simultaneous study of several alternatives (and by indexing all intermediate results with adequate validation and traceability processes). Thus, final decisions may be taken later in the cycle, avoiding freezing expenses in the early phases. Consequently, the curve of expenses fixed a priori undergoes a vertical shift downwards. A fortiori, the cost derivative will also be reduced; an unplanned modification could be an alternative already studied – canceling the cost derivative – or may involve lower additional costs as it is not necessary to travel as far back in the cycle, meaning fewer elements need to be reworked.

If we compare both trajectories, one for a system acquired following the “standard” process and the other using SBA, three means of cost reduction become apparent:

– “frozen” costs are a priori lower;

– lower cost derivative;

– reduced expenses following the cost derivative.

The sum of these gains rapidly balances out all additional costs engendered by the SBA, as calculated above; with the qualitative tendencies already mentioned, this happens as soon as the development phase begins.

Actually, this calculation is too pessimistic regarding SBA: the added costs calculated earlier applied to the whole acquisition process, while the figure we need should only apply to the period preceding the cost derivative. If we look at these calculations in more detail, we see that development and construction are the main sources of added costs (as they involve higher acquisition costs, and all our calculations are proportional to the global cost of acquisition, although we pay particular attention to proportionality coefficients). In other terms, the major contribution is that SBA already reduces costs by controlling the cost derivative! A rapid analysis then shows that the gains in terms of the cost derivative and lower immediate outlay always balance out the added cost of SBA when compared with a “standard” acquisition.

To summarize, we have shown that the added costs of SBA are immediately balanced out and transformed into positive savings as soon as problems are encountered. This situation is analogous to what happens with total quality in other contexts.

This is not surprising as SBA increases the amount of systems engineering and effort in general in the initial phases of a project (analysis of requirements, compromises between requirements and technical solutions, and so on), and it is a well-known fact that most errors and avoidable added costs in projects can be traced back to these early phases (e.g. data from software engineering tells us that errors are distributed between requirement definition, design, coding, and integration activities with a ratio of 56-27-7-10, and the distribution of rework costs follows a ratio of 82-13-1-4). If we refer to Table 9.2, the costs added by SBA in the first three phases (i.e. ignoring the construction phase and concentrating on systems engineering) may be estimated approximately 5% of the global acquisition cost, a figure which should convince the reader of the value of SBA in risk reduction!

9.3. Multi-project acquisition

Let us now consider the acquisition process for a system of systems. We could carry out the analysis described above for each system and cumulate the gains. Savings of this kind are bound to appear because the probability of an uncontrolled risk does not increase in a linear fashion, but more rapidly with several different acquisition teams working on systems which are not temporally synchronized and must be integrated eventually.

We shall, in fact, propose a different economic analysis, with special focus on reuse. Reuse has been recognized over the last few years as a major key to cost reduction [EZR 99]. It is motivated by the search for standard solutions to recurring problems found in different contexts. Reuse is based on the availability of modules in standard source code, but it is not restricted to component level. It is also based on a pooling of needs and analogy of solution architectures. Consequently, requirements, architectures, design patterns, user interface models, test plans, data, and documentation must be shared. The immediate results of a correctly implemented reuse process may be measured in terms of:

– productivity: less aspects of a simulation system need to be produced and so less effort is required, reducing development costs and improving management of time to delivery;

– quality, which may be transferred from one project to another;

– performance, with reduced costs and time delays and better anticipation of the clients’ needs.

As simulations mainly consist of programs (including programs embedded in hybrid hardware-in-the-loop simulators), we shall use cost models for software reuse, like that of COCOMO2.0 (constructive cost model), initially introduced by Barry Boehm in 1981 and extended in 1987 and 1995; the latest versions, [BOE 95] and [BOE 00], include detailed analyses of commercial-off-the-shelf (COTS) use and software reuse.

The following curve, based on an analysis carried out by Richard Selby in 1988 on around 3,000 software modules developed by NASA, gives the cost of reuse as a function of the proportion of the program requiring modification [BOE 95].

This curve shows the following:

– savings of more than 50% when less than 10% of the program requires modification;

– savings of 25–50% for modifications up to 75%.

During the acquisition of a complex system, simulations developed during various phases become more and more specific. Thus, simulations may be reused easily during the feasibility phase, but much less during the development phase, and in all probability not at all during production.

Based on Figure 9.6, we can estimate that the cost savings generated by reuse are more than 50% during the feasibility phase and up to 25% during the definition and development phases. If we return to Table 9.2 and correct the costs associated with each phase using these figures, a global cost reduction for SBA can be estimated approximately 3–4% of the global acquisition cost (depending on the equation used).

Referring to Figures 9.2 and 9.3, we note that simulation plays a major role in the feasibility phase and a non-negligible role in definition and the first parts of development, whatever acquisition mode is used (standard or SBA).

When acquiring a system of systems, where several cycles overlap, reuse produces significant savings during the feasibility phase of recent systems and non-negligible savings during the definition phase. Savings in this area may be redirected to help with the integration of various systems (the feasibility and definition phases for the encompassing meta-system come for free), and furthermore, risk management is improved.

To conclude, this qualitative analysis4 shows that reuse, as promulgated in the SBA processes of several systems (on condition that they share a certain coherence and may potentially be combined within a system of systems), may be reinvested in the design of the meta-system, which brings together and organizes all these systems. By modifying the acquisition processes in this way, and by working at system of systems level rather than considering a disconnected group of systems, it is possible to reason at capacitive level (how to acquire a capacity which allows the production of a particular effect via clever use of available systems), within more or less consistent cost limits.

9.4. An (almost) definitive conclusion: conditions for success

From what we have seen above, the key information to share involves requirements and specifications, on the one hand, and integration tests and validation methods and data, on the other hand. This information must be exhaustive and thoroughly traced concerning its origin and configuration. It is through these two aspects of traceability and configuration management that we may obtain a vision of the whole system throughout its life. To share this view, we must have access to various architectural and technological choices, hence mastery of systems engineering. If a more mathematical reading of the global process was required, we might say that, in this case, we arrive at a canonical representation of which the execution corresponds to possible modifications of the system, with the capacity at any given moment to reconstruct the current state of the said system.

In addition to this information, which provides a set of discrete elements allowing us to reconstruct the system, it is useful to have a more continuous vision of the construction of each architectural element and level, hence the interest of a process involving tools, which gives immediate access to a global and behavioral vision of the system on condition that we have a capacity for vertical (from the meta-system to the component and vice versa, to provide an extreme example) and horizontal integration (ideally, over the whole life cycle, from first consideration of the system concept to its retirement from service). In this case, we are in fact defining the information system corresponding to the complex system in question, which would serve as a reference framework and memory and could be connected with other tools such as those linked to the way the project is run, its financial management, or even generic logistical assistance systems which might play a support role.

This was shown in Chapter 8 with the battle lab approach, which is based on the following:

– an engineering infrastructure, based on the coherent use of:

   - a tool for formalizing requirements, with traceability and configuration management functions for these requirements, along with connection to simulation capacities to explore and justify specification decisions,

   - a tool to assist in integration and validation tests, making an immediate link to requirements as handled by the previous tool, and once again, simulation capacities to actively contribute to validation,

   - a collaborative working environment, which may be distributed geographically, giving the various parties involved before and after each level access to information relating to this level, to facilitate returns and potentially accelerate progress between levels; in addition to this secure exchange function and management of information based on levels of confidentiality and user access rights, an environment of this kind becomes the key structure for dialog throughout the life of the system;

– a simulation infrastructure:

   - based on international standards,

   - offering the required level of interoperability between future or existing models,

   - providing the necessary service for designing global and often geographically distributed simulations (shared technical and methodological frameworks, libraries of models and tools, management of model configuration, support in the validation process).

We thus gain access to methods and tools for model engineering, the aim of which is to provide conceptual frameworks independent of specific implementations and to design models and simulations in general based on expressed needs, the final generation of code relating to a particular framework being carried out automatically. Following the iterative process described above, the battle lab (which is itself a system of systems!) alternatively plays the role of catalyzer and source for technical coherence of simulation resources acquired in the course of different projects and for the harmonization of processes in the new context of complex systems engineering.

Note, too, that the whole of the approach previously outlined should be accompanied by a normative approach, with the creation of adequate methodological frames of reference and determination, then appropriation, of the necessary standards, covering the entire chain of information being handled, including system and model description data, validation data for these various elements, data relating to the systems engineering system itself, and data (or metadata) concerning infrastructures.

This vision is certainly ambitious, but facilitated by very recent developments in software and systems engineering and by normative efforts and the desires of a vast international community, bringing together industry, academic researchers, public and private establishments, for a coherent and integrated vision of the various current norms (UML2 for model description data, for system modifications, ISO10303-AP233 for the exchange of description data, MDA for the transformation of models and meta-models, ISO 12207 for the software engineering process, and ISO 15288 for the systems engineering process, see [LUZ 10]).

It is at this cost, with the aid of these tools, methods, norms, and principles which guide and assist decision making, that the subtle art of systems architecture becomes a science which, if not accessible to others, will at least involve taking risks and considerable reductions in delays and costs.

9.5. Bibliography

[BER 50] VON BERTALANFFY L., Théorie générale des systèmes, Dunod, Paris (1993), 1950.

[BLA 98] BLANCHARD B.S., System Engineering Management, Wiley, New York, 1998.

[BOE 95] BOEHM B., CLARK B., HOROWITZ E., WESTLAND C., “Cost models for future software life cycle processes: COCOMO 2.0”, Annals of Software Engineering, Special Volume on Software Process and Product Measurement, J.D. ARTHUR and S.M. HENRY (Eds.), J.C. Baltzer AG, Science Publishers, Amsterdam, 1995.

[BOE 00] BOEHM B., ABTS C., WINSOR BROWN A., CHULANI S., CLARK B., HOROWITZ E., MADACHY R., REIFER D., STEECE B., Software Cost Estimation with COCOMO 2.0, Prentice Hall, Upper Saddle River, 2000.

[BRI 00] BRILL J.H., CHEVALLIER J., MERCHADOU J.-L., Manuel des meilleures pratiques pour le développement de systèmes spatiaux, Cépaduès-éditions, Toulouse, 2000.

[DON 02] DONNADIEU G., KARSKY M., La systémique: penser et agir dans la complexité, éditions Liaison, Rueil-Malmaison, 2002.

[ELM 08] ELM J.P., GOLDENSON D.R., EL EMAM K., DONATELL N., NELSA A., A survey of system engineering effectiveness – initial results (with detailed survey response data), Special Report CMU/SEI-2008-SR-034, Carnegie Mellon University, Pittsburgh, December 2008, http://www.sei.cmu.edu.

[EZR 99] EZRAN M., MORISIO M., TULLY C., Réutilisation logicielle: guide pratique et retours d’expérience, Eyrolles, Paris, 1999.

[HON 01] HONOUR E., MAR B., Value of systems engineering: SECOE Research Project Progress Report, 2001, http://www.secoe.org/0103.

[KAM 02] KAM L., LECINQ X., LUZEAUX D., CANTOT P., “ITCS: the technical M&S infrastructure for supporting the simulation-based acquisition process”, NATO Modeling and Simulation Group Conference, Paris, 2002.

[LEM 77] LE MOIGNE J.-L., La théorie du système général, PUF, Paris 1977.

[LUZ 03] LUZEAUX D., “Cost efficiency of simulation for complex system acquisition”, SPIE Aerosense’03, Conference on Simulation, Orlando, USA, 2003.

[LUZ 10] LUZEAUX D., RUAULT J.-R., Systems of Systems, ISTE, London, John Wiley & Sons, New York, 2010.

[MCG 07] MCGIBBON T., FERENS D., VIENNEAU R.-L., A business case for software improvement (2007 update): measuring return on investment from software engineering and management, A Data Analysis Center for Software (DACS) State-of-the-art report, DACS Report number 347616, Contract Number SP0700-98-D-4000, September 2007.

[MEI 98] MEINADIER J.-P., Ingénierie et intégration des systèmes, Hermès, Paris, 1998.

[MEI 02] MEINADIER J.-P., Le métier d’intégration des systèmes, Hermès, Paris, 2002.

[MOR 77] MORIN E., La méthode: la nature de la nature, Seuil, Paris, 1977.

[NAT 00] NATO Industrial Advisory Group, “Prefeasibility study on simulation based design and virtual prototyping”, NIAG-D(2000)9 AC/141(NG-6)D/25, September 2000.

[NAT 03] NATO Modeling and Simulation Group, “Feasibility study on M&S technology in support of SBA”, RTO-TR-064 AC/323(NMSG-003)TP/06, February 2003.

[NRC 02] NATIONAL RESEARCH COUNCIL, Modeling and simulation in manufacturing and defense systems acquisition, Washington DC, 2002.

[PRI 01] PRINTZ J., DEH C., MESDON B., TREVES N., Coûts et durée des projets informatiques, Hermès, Paris, 2001.

[YAT 99] YATCHINOVSKY A., L’approche systémique pour gérer l’incertitude et la complexité, ESF, Issy-les-Moulineaux, 1999.

---

1 Chapter written by Dominique LUZEAUX.

1 A Pareto type curve, with the 80-20 rule: 80% of financial outlay takes place in the last 20% of the project development.

2 [ELM 08] states that the adequate application of systems engineering allows cost reductions of 20–30% and productivity increases of almost 40%. Parametric studies [HON 01] also state that depending on the percentage accorded to systems engineering (typically closer to 15% than to 1%), risks of overrun are much better managed (typical overrun by a factor of 1 ± 0.1 instead of 1.4 ± 0.3). All these considerations seem to suggest that values of γ = 0.8 or 0.7 would be reasonable.

3 Once again, this follows the 80-20 rule, but it is applied in a different way: after 20% of effective expenditure, decisions have been made which a priori tie up 80% of the total budget.

4 We could also carry out an analytical analysis, giving, for example, the equational formulae for the various curves: thus, the Pareto type curve in Figure 9.1 may be approximated to the function x7 and that in Figure 9.6 by the function x1/7 (it is interesting to note that we thus obtain power laws; see Chapter 5). The interest of these formulae is that they are easier to handle when considering several systems in parallel, and we can even have fun increasing the number of systems considered to look for results at the limit.