2.1.1 Motivation and Description

2.1.2 Life‐Cycle Shapes

An interesting insight is that engineering project life‐cycles can have shapes. I will describe three of the most common such shapes: “U,” waterfall, and spiral.

I start with what I call the “U” diagram (Figure 2.4). The concept is clear: we start our project in the upper left, defining our requirements, and then creating our design. As noted above, these stages proceed in a hierarchy; we do requirements at the system level, and then for each smaller segment of our system. We do design at the system level, and then for each smaller segment of our system. That is, we perform the requirements and design stages by starting at the system level and working our way “down” the hierarchy (where “down” in this context signifies from the system level to ever smaller pieces of the system).

To make it clear what we mean by such a hierarchy, consider the example that we presented in Figure 2.2:

• The air transportation system can be considered to consist of the following major parts: airplanes, airports, and air traffic control. These items – airplanes, airports, and air traffic control – form the second level of this system's hierarchy.
• An airplane consists of parts too, such as the fuselage and the engine(s). Those parts form a third level of our hierarchy.
• An airplane engine also consists of parts, such as a compressor and a combustor. The compressor and combustor form a fourth level of our hierarchy.

We say that the requirements and design stages are performed top‐down, referring to the direction of motion through the hierarchies and through the “U” diagram.

Having completed our requirements and our design, we are ready to implement all the pieces of our system defined by our design. This portion of the diagram is drawn horizontally, because we may not use a hierarchy for this stage; we could just commission independent teams to implement each of the pieces we have defined.

We are then ready to move upward through the right‐hand side of the diagram. Having implemented all of the pieces, we start putting those pieces together, at first in small subsets of the system, and gradually progressing to larger subsets, until we finally have the entire system interconnected and operating to some initial degree of correctness; as noted above, I call this process integration. We then perform testing: the verification that our system is effective (e.g. satisfies all of its formal requirements) and the validation that our system is suitable (e.g. meets the needs and desires of its intended users). On this side of the diagram, we show the arrow pointing upward, because we progress from small pieces and subassemblies, through larger subassemblies, to the entire system; that is, in this portion of the diagram, we say that we are proceeding bottom‐up.

Many other people use a similar diagram, which they generally call the “V” diagram; it basically has the left and right sides of my “U.” But the “V” version of this diagram either omits the implementation steps, or it places that actual building of the pieces on the right‐hand side. I object to either of those approaches. Obviously, the implementation is important, and ought not to be omitted. Perhaps more subtle is the idea that the purpose of the left‐ and right‐hand sides of the “U” diagram is to show that these activities proceed in a hierarchy. The actual implementation of the pieces that we have defined through the decomposition on the left‐hand side, however, need not proceed in any sort of hierarchy; if we have decomposed our system into 500 little pieces, we might well build the 500 little pieces pretty much independent of each other. We resume working in a hierarchy (bottom‐up, on the right side of the “U”) when we start putting small numbers of those pieces together through the integration process. Hence my preference for the “U” shape over the “V.”

The “U” diagram does not depict the latter stages of the project life‐cycle (e.g. production, deployment, actual use of the system, retirement, and disposal); neither do the related “V” versions of this diagram. Instead, this “shape” concentrates on the stages that occur during the actual development of our system. Indeed, some of those later phases (e.g. use of the system in actual operations) are not actually “projects,” as I defined that term in Chapter 1; instead, they are what I defined as “continuous business operations.”

The next shape for a project life‐cycle that I will discuss is called the waterfall (see Figure 2.5). The so‐called waterfall method was introduced by Dr. Winston Royce in 1970.³ Dr. Royce's purpose was to bring some order to what he perceived as the chaos that seemed to him to be a recurring feature of large software development activities. His recommendation was for a series of particular steps to be undertaken in a particular order, while endeavoring to complete one step before beginning the next. He believed that the need to perform all of these steps was not universally recognized – he actually said that some customers believed that doing some analysis, and then doing the coding, was all that would be required to deliver a software product – and therefore, part of his goal was to advocate the use of the complete set of steps. Although his terminology is in some ways specific to software systems, you can see that, in concept, Dr. Royce's list of steps is not very different from the list of stages that I presented above.

It is difficult to overstate how important and influential Dr. Royce's work has been. Large companies, like TRW and IBM, created corporate software development policies that more or less adopted Dr. Royce's approach in toto. So too did the US Department of Defense, and through the Department of Defense's contractual terms that mandated that companies who were building software (and later, systems) for the US Government follow those standards, in fairly short order the entire world was following the waterfall method.

There is an additional insight that Dr. Royce provided: a recognition that things do go wrong, and one might at times have to go backwards (Figure 2.6).

The depiction of Figure 2.6 is often misunderstood. If one just looks at the drawing but does not read Dr. Royce's paper, one might get the impression that one is allowed – even encouraged – to back up as many steps as wished. This is not so! Dr. Royce is very clear that the desired approach is to be thorough enough at each step that one never has to back up more than a single step. His original caption for this figure actual reads “Hopefully, the iterative interaction between the successive phases is confined to successive steps.”

The waterfall method contributed huge value through its “normalization” of the necessity for steps other than analysis and coding, and through its promulgation of the idea that there should in fact be a planned sequence of steps.

In short, the waterfall method was aimed at introducing some organization and structure into what was perceived to be an overly chaotic approach to systems engineering and engineering project management. But the waterfall method also implied a rigorous sequencing of (i) doing all of the requirements, then (ii) doing all of the design, then (iii) doing all of the implementation, and so forth; in fact, many of the corporate and government development policies that were created in the wake of Dr. Royce's paper said explicitly: finish one step before you proceed to the next, and go through the life‐cycle exactly once.

Experience, however, soon showed that this was too constraining to be practical for many projects, especially those of larger scale and complexity. At times, people would achieve success through an incremental approach that involved a sort of successive approximation, by building a well‐defined partial version of the system, operate that version for a while to gain additional insight, then build a second well‐defined partial version of the system, operate that version for a while to gain additional insight, and so forth.

What to do? Live with the perceived inflexibility of the waterfall method, or return to the pre‐waterfall chaos? Neither of those choices seemed very good.

Fortunately, someone came along and proposed a new method – and a new project life‐cycle shape – that solved this dilemma, allowing the continued rigor and organization of the waterfall method, while creating a structured framework for successful development through incremental, successive approximations of the eventual system. The person who first put this forward as a candidate formal method was Dr. Barry Boehm,⁴ in 1986;⁵ he termed it the spiral model. The spiral model (Figure 2.7) forms a third shape for a project life‐cycle.

The point of the spiral method is that we develop a carefully thought out partial version of our system, and then we deliver that partial version to the users (Dr. Boehm calls this a prototype in Figure 2.7, but this partial version, ideally, is actually used to accomplish real work), who then use it for actual mission operations. The development team observes these operations, and thereby gains new insights into what those users actually need and want. The developers then incorporate those new insights into their plans for the next increment – which Dr. Boehm calls a spiral – of the system. This way, features or omissions that might cause the system to be unacceptable to the users get found and fixed along the way. It turns out (we quantify this in the next chapter) that fixing things earlier in the project development life‐cycle costs far less than fixing them later. Fixing them earlier, of course, also increases the user's satisfaction with the system, and their confidence in the development team.

Again, it is difficult to overstate how important and influential Dr. Boehm's work has been. All of the same organizations that in the 1970s and 1980s created policies and directives mandating the use of the waterfall method modified those policies and directives in the 1990s to allow and encourage the use of the spiral method. As a result, some variation of the spiral method is used nearly universally on engineering projects today, especially engineering projects with lots of software (which these days is most of them). Legendary computer scientist Dr. Fredrick P. Brooks says of Dr. Boehm's spiral model, “I strongly believe the way forward is to embrace and develop the Spiral Model.”⁶

Other shapes are possible, and have been propounded in the literature. But for the purposes of this book, we will limit ourselves to these three project life‐cycle shapes.

2.1.3 Progress Through the Stages

Whatever shape your engineering project employs, you will use some type of orderly method to determine when you are ready to move from one life‐cycle stage to the next (e.g. from requirements to design, and so forth). This method is centered around a review, which is the data‐gathering and data‐analysis exercise that forms the basis for a formal decision process (e.g. are we ready to move to the next life‐cycle stage or not?). If we determine that we are not yet ready, we then determine what are the things that remain for us to accomplish, so that we are ready to move to the next life‐cycle stage. Generally, you want a large segment of the stakeholders for your project (the development team, your company's management, the buying customer, the using customer, the paying customer, and so forth) to participate in these reviews and decisions. We include all of these people, both in order to get a full range of opinions to inform your decisions, but also to build a social consensus about the correctness of that decision.

This process is often referred to as decision gates. We call these reviews gates because we may be allowed to pass through them at this time … or we may not; that is, the gate may be either open or closed. The reviews are intended to determine the adequacy of the system to meet the known requirements (specifications and process guidelines) and constraints. Reviews become progressively more detailed and definitive as one advances through the program life‐cycle.

In toto, reviews provide a periodic assessment of your project's maturity, technical risk, and programmatic/execution risk. Equally importantly, they help one improve/build consensus around the go‐forward plans; this is why you involve so many of your stakeholders in the review process. Reviews provide you – the manager of this engineering project – with the data you need to make the decision about whether to proceed to the next phase, or return to the previous one in order to resolve some issues.

As noted above, an important characteristic of the systems method is that we strive to optimize at the systems level, not at the component level. Even if we make each component of our system the very best possible, they may interact in a fashion that provides less than the best possible performance; since we are striving to make the system as a whole the best possible, we must look at the interactions of the components, in addition to the performance of each individual component.

This may have other benefits. For example, I have frequently discovered that I could make do with a less capable version of some component in a system and still get the system‐level performance and capacity that I needed. It would be a waste of money to have paid for a better version of that particular component; limitations that arise out of the interactions of the components might prevent the improved version of that component from having a positive effect on the system as a whole.

This quest for system‐level optimization can lead to useful insights too. I once had a radio vendor on a military command‐and‐control system come to me and say that they had figured out (for a price!) how to make the signaling rate of their radio become twice as fast as it was at present. In the coordinate system of value for a radio designer, being twice as fast is of high value indeed. The radio signaling rate is an example of what I will later call a technical performance measure; this is an objective measurement within what I like to call the engineer's coordinate system of value. But before making a decision to pay that money for faster radio, I had my system‐level modelers insert a model of that faster radio into our system‐level performance model. We had created a set of metrics that measured the performance of the system not only in technical terms, but also in terms of the operational benefit to the intended users; this is what I will later call an operational performance measure, and it forms an objective measurement within what I like to call the customer's coordinate system of value (Figure 2.8). For example, we had created a metric that predicted the level of casualties on each of the opposing sides in a battle scenario; if our system design was going to add value to the customer (in this case, the US Army), the ratio of the number of enemy casualties to the number of US Army casualties should be higher than for the same battle scenario when our system was not used. We called this the loss‐exchange ratio; a higher loss‐exchange ratio indicated a better design, because that was a metric that the intended users of the system valued.

When we plugged the radio with the doubled signaling rate (a highly favorable technical performance measurement, in the engineer's coordinate system of value) into our system model, the result was no improvement at all in the system's overall performance, as measured by the loss‐exchange ratio! Therefore, in the customer's coordinate system of value, the faster radio had no value.

But we did not stop there; I asked the radio vendor and the modeling team to work together to figure out why the performance at the system level did not improve. They discovered that there was a subtle bottleneck, and this insight led to an idea for a different improvement that the radio vendor could make: leave the signaling rate the same, but decrease the time it took for the radio to acquire the channel and synchronize the encryption process. This was a far less expensive change to make than doubling the signaling rate, but led to a major improvement in system‐level performance.

My lesson: You should always ask why! And you cannot trust your intuition; the interactions inside of a complex system make it very difficult to intuit the system‐level impact of a change to a component.

We – not the Army – created the loss‐exchange ratio metric. But clearly, this metric would have more value to our design team if the customers and eventual users agreed that this metric actually reflected their coordinate system of value, that is, a better score on this metric would indicate a system that the users would actually find to be better. So, you need to take the effort to socialize your ideas for operational performance metrics with your customers, users, and other stakeholders. I like to achieve what I call the transfer of emotional ownership to the customers. This signifies that in some real sense, the metric has become theirs, rather than mine. For example, after socializing the loss‐exchange ratio metric with the US Army for several months, I learned that the Army modeling organization had started using that metric as the primary output of their own system performance models; our metric was now driving internal Army decision‐making about the future of our system. That is a transfer of emotional ownership! When you achieve that, you are building credibility with your customers, users, and other stakeholders.

Another aspect of the systems method is that we employ written guidance regarding our methods. We call such guidance processes. We have processes for every aspect of our engineering project: engineering, but also finance, hiring, managing our people, configuration control, contracting, procurement, testing, quality, safety, and many other items (many of which we will cover over the course of this book). Each process will specify in writing what is to be done, when, by whom, what the products will be, what artifacts will be created, how the work is measured and quality ensured, who approves the work and the artifacts, and many other aspects.

Figure 2.9 summarizes the range of processes that we use in an engineering project.

Why go to the trouble to create and employ such written process? Because engineering a complex system is hard. Dr. Eberhart Rechtin⁷ said that:

• Success comes from wisdom
• Wisdom comes from experience, and
• Experience comes from mistakes.

When possible, it is best to learn from the mistakes of others, rather than learning only from mistakes that one makes oneself. And that is the role of engineering processes – they allow us to learn from the mistakes of others. They are the “lessons learned” from past activities.

There is a caveat: processes are necessary – they help us be repeatable, and operate at scale. But processes by themselves are not sufficient to ensure a good design! We need both good processes and a good design. Good designs come from good designers, not from good processes.

Some companies have gone through a phase of assuming that good processes are in fact sufficient to ensure good design; the result was a series of expensive project failures.

We discuss how to achieve a good design later in this chapter.

Lastly, the systems method involves a lot of planning. We formulate and write plans about how we are going to perform each of the various aspects of the project, ranging from how we will validate the technical requirements, to how we will acquire the people with all of the specialized skills we need (and at the right time, and in sufficient quantity), to how we will keep our people (and the general public) safe as we perform this work.

2.3.1 The Design and its Process

The next phase in the development process of an engineering project is the design; we aspire to create a design that satisfies the requirements, but also one that is feasible and affordable to build.

We get a lot of “help” with the requirements; after all, our customers and our users understand well what they want the new system to do, and such what constitutes a major portion of the requirements. It is my experience that most systems eventually develop pretty good requirements, although it may take them longer to do so than they originally planned, and cost more money to do so than they planned.

But the design is a completely different matter; many systems simply have bad designs. Why might this be? For one thing, the customer and the users are generally not qualified to provide expert help with the design, in contrast to the way that they are qualified to provide expert help with the requirements.

How do I know that many systems actually have bad designs? More than once I have seen cases of two completed systems that do approximately the same thing, where one runs 100 times faster than the other. Similarly, I have seen cases of two completed systems that do approximately the same thing, where one is 1000 time more reliable than the other. I have then had the opportunity to examine these systems so as to find the root cause for the slower and less reliable performance, and therefore I can state with confidence that this gigantic gap in performance derived from specific (undesirable) features of their designs.

This finding has many interesting implications. First, having a 100× or 1000× range of outcomes for a critical parameter from an engineering project is shocking; mature disciplines do not have such large ranges of outcomes. Consider mid‐sized family sedans offered for sale that meet US emissions control requirements; the variation from best to worst in, for example, gas mileage is no more than 25%, not 100× (10 000%) or 1000× (100 000%). Something is going radically wrong inside the designs of the systems that exhibit such poor performance on such an important metric.

Second, a lot of engineering projects turn out to be problematic, in the sense that they end up far over budget, far behind their delivery schedule, and a shocking number (some studies say more than half of all engineering projects) are canceled before they complete, because of customer and user dissatisfaction with progress. The people who study these problem projects nearly all assign the blame to poor requirements. But I spent many years of my career as a designated fix‐it person for engineering projects that were in trouble, and I will tell you that they all had pretty good requirements. What they all lacked was a sensible, feasible design.

So, in light of the above, I have come to view the design as the critical portion of the engineering project development cycle. It is the stage that will likely make or break your project.

What is a design? In the previous section, we defined the requirements as the statements that tell us what the system is supposed to do, and how well it is supposed to do it; in contrast, the design tells us how the requirements are going to be accomplished. Consider a house: the requirements might tell us that the house needs to have four bedrooms and three bathrooms. The design tells us how we will satisfy those requirements: whether we will use wood or metal for the frame, whether we will use a raised structure or a concrete slab for the foundation, whether we will use casement or sliding windows, whether we will use wooden shakes or concrete tiles for the roofing materials, and so forth. We can build a house that meets the requirements – four bedrooms and three bathrooms – using either wood or metal as the framing material; both probably allow us to satisfy the requirements. But there may be other reasons for choosing one design approach or the other, reasons that have little to do with the requirements (e.g. “four bedrooms”). For example, if our house is going to be in a location with a really severe termite problem, we might not want to select wood for the framing material. But if wood is satisfactory, then using wood is probably a lot less expensive than using metal for the frame. A wood‐framed house can probably be built in less time than a metal‐framed one too. These are examples of alternative designs.

The process that we use to create a design centers around a method that we call the trade study. The trade study process helps us create a set of candidate alternative designs, helps us create a way to measure the “goodness” of each alternative, and finally allows us to select a preferred alternative, while also creating the data and the artifacts that will allow us to explain to our peers and stakeholders why we believe that it is the best possible alternative.

In the design process, there will seldom be a clear winner, in the sense that a particular alternative design is best in every category. That is why we call the process a trade; we make judgments (backed up by data and analyses) about which combination of positive and negative features achieves the best overall solution for our system. We strive not for perfection, but for a reasonable balance.

In this book, we are considering engineering projects, and therefore technology and technological concepts are central to the success of those projects. We therefore use engineering methods to guide project management decisions, and that in turn implies that we use data to help make decisions. The data that we use to measure the “goodness” of our candidate designs takes the form of two types of metrics: one that I call operational performance measures (OPMs), and one that I call technical performance measures (TPMs).

Why two types of metrics? Our degrees of freedom in creating alternative designs lie mostly in alternative technical concepts and approaches, and these are best measured using the technical performance measures. But we must also make sure that improved technical performance actually results in improved operational performance, as measured from the coordinate system of value relevant to the users, customers, and other stakeholders. We therefore also need to use the operational performance measures.

At first blush, one might believe that improved technical performance always leads to improved operational performance. That is simply not true. Remember the example a few pages back about the radio vendor who offered me a radio that sent and received data twice as fast as his current radio model? When we plugged twice the data rate into our system‐level model, the operational performance measures did not improve at all! This, then, is a real‐life example where dramatic improvement in a key technical performance measure did not result in any improvement in an operational performance measure.

In my experience, this is in fact a frequent occurrence; as a result, we must separately measure both technical performance measures and operational performance measures. We cannot abandon technical performance measures, as they are at the heart of our technical analyses that allow us to determine whether or not our design will in fact work. But we must somehow relate the effect of the technical performance measures to the operational performance measures. In the radio example cited above, we did that through a system‐level model. In other instances, we have done this through actual benchmark measurements. But however you choose to do it, it must be done.

Furthermore we must convince our stakeholders – most of whom are not engineers – that our predictions about operational performance measures are credible. So, we must be able to explain the connection between the technical performance measures and the operational performance measures in a credible and transparent fashion, even to our non‐technical stakeholders. This might be done by explaining the logic in the system‐level model, and then showing that we have calibrated that model by using it to predict the performance for a set of situations for which we can go out and make actual real‐world measurements. If the stakeholders understand and agree with the logic inside our system‐level model, and see that in a set of real‐world circumstances the system‐level model makes accurate predictions, then they may accept that when we use the model to make predictions about circumstances for which we cannot yet make measurements (e.g. how the new system will perform), the predictions may be believed.

I use the method depicted in Figure 2.15 in order to create a design. This figure introduces a few new terms, which will be described in the following text.

Let's discuss each of these.

By a pressure point that the design must actually address, I mean that we must use our operational knowledge of the users and the mission to determine what are the real design drivers for the system. Do not depend on your customers to do this for you! They are not engineers and designers, and while they understand their mission well, they often have an imperfect understanding of how technology interacts with their mission.

Think of the short‐range air defense system that I described earlier. If the Army were to notice that only about one‐third of the missiles they fire at airplanes in the sky actually hit the target airplane, they might well conclude that they need a better missile. While this may sound reasonable, that conclusion may be completely incorrect! When I was actually designing such a system many years ago, we discovered that the Army gunners were actually taking very few shots; they were shooting at a target only about 10% of the time that they could have. Most of the time, the very short nature of the shot opportunity (a high‐speed jet flying very close to the ground passes you by in just a few seconds) meant that they were not even shooting 90% of the time. Instead of building a better missile, we decided that the pressure point in the design was to help the gunner get ready, and to cue him when a shot opportunity was coming up soon, so that he would not miss so many shot opportunities. A few years later, after we had finished an automated system designed to help gunners achieve more shot opportunities, not only were they taking nearly 10 times as many shots per day, but most of those shots were hitting the target airplane. It turned out that by using automation in the system to help the gunner find and take his shot opportunities, we were not only creating more shot opportunities, but also creating better shot opportunities, ones where the target airplane was closer, or otherwise situated so as to make it easier for the missile actually to hit the target airplane. They didn't need a better missile at all!

Think about that: not only did we create a revolutionary improvement in the performance of the system (almost a 10× improvement!), but we did it without making any changes at all to the item that the customers and users initially might have thought was the problem. We had to discover what was the real pressure point in the design. In this case, that pressure point was improving the number of shot opportunities that complied with the rules of engagement (and also improving the quality of those shot opportunities), rather than improving the probability of kill once a missile was launched. This was despite the fact that the major observable of poor performance was that most of the time, the missile missed the intended target – which made it seem like the problem was with the missile itself.

In my experience, this sort of focusing on the wrong aspect of the problem takes place quite frequently. Therefore, my design methodology always starts with the assessment and analysis needed to determine where the actual pressure point is in the design of our new system. There may well be more than one, of course.

Once we know the pressure points, we can turn to the trade study. I use the following steps to perform a trade study (Figure 2.16):

• Use the knowledge and insight we acquired about the customer in order to create operational performance measures, and then discuss those with the customer. Of course, we actually started this process while we were creating the requirements.
• Use the operational sequences to firm up a list of all the independent stimuli that can activate a mission thread in your system, and also define what is produced as the output of each mission thread. We started this when we prepared the concept of operations document as part of the requirements, but now we need to do it in more detail.
• Use these items – the lists of stimuli and the partitioning of steps that could be in parallel and steps that must be in sequence – to define all of the independently schedulable entities within your system. These are the system activities that can be started in response any sort of asynchronous stimuli, and can therefore operate at the same time as other system activities.
• Use the knowledge and insight we acquired about the mission, together with the requirements, to create operational sequences that describe how they perform this mission, the mission threads. This is a mechanism that helps you to ensure that every requirement is addressed by the design. We started this while we were creating the requirements too. Now we have to do it at a finer level of detail, showing which steps on the mission threads can be performed in parallel, and which must be performed sequentially.
• Use the list of mission threads and the list of independently schedulable entities to create alternative groupings of those entities into actual implementation groupings (e.g. the items that will actually comprise the separately buildable elements of the system).
• For every alternative implementation grouping, postulate a set of candidate implementation methods (e.g. software, digital hardware, analog hardware, electromechanical, and so forth). Perhaps there will be more than one for some of these categories (e.g. several software‐based approaches, each using a different algorithmic method). These form the set of candidate designs; we call this set of candidate designs the trade space. The candidates that you select for the trade space ought to be informed, of course, by your previous selection of a set of pressure points that the design needs to consider.
• Perform an assessment of the technical feasibility of each candidate design and eliminate from further consideration those that are deemed not feasible, or too risky.
• Using the technical performance measures and the operational performance measures, create a methodology for assessing the “goodness” of each candidate design, by which I mean how well this design balances the two key goals of the design process: (i) satisfying all of the requirements, while also (ii) being technically feasible, and at the same time also satisfying any other goals that may apply to this specific project (such as unusually high levels of reliability, if our system is safety‐critical, and so forth). What will you measure? How will you measure it? How do you ensure validity of the measurements and the predictions that you make from those measurements? What are the desired values for each measure? What are the minimum acceptable values for each measure? How do you combine and process the measurements into an overall assessment of “goodness” for a candidate design? What are the relative weightings you will give to each measurement as a part of that overall assessment? I provide a list and description of common assessment methods in Figure 2.17.
• Use that measurement and assessment methodology in order to perform the actual assessment of the goodness of each candidate design. This will probably be done multiple times, eliminating a few of the worst‐performing candidates each time, adding additional measures for the remaining candidates, perhaps using the insight from the measurements in order to create new candidates. Keep careful records documenting the rationale behind all decisions, even for those candidates that are discarded.

Since we will be assessing several candidates during the trade study, the depth of each assessment is necessarily limited. Once we reach the point, however, where we believe that we have our single selected design (perhaps with alternatives for a small number of selected points within that design), we can afford to assess that final candidate in more depth. This involves assessing the final candidate design in several ways:

• The performance and capacity of the design. Many systems fail because they fall significantly short of promised performance rate and/or capacity, so we try to avoid that risk by a careful assessment against those dimensions of our candidate design. This assessment will be done using some combination of modeling and actual benchmarking.
• The stability of the design. We want the design to be relatively insensitive to small errors in inputs and assumptions, so we assess our design by subjecting our system model to such variations in inputs, initial conditions, operating conditions, and so forth, and check to see that small variations in these factors lead only to small variations in the performance of our system, rather than to catastrophic degradations in our system. See Figure 2.18.
• Design margin. We want the design to be relatively insensitive to small errors in implementation too. For example, if some part ends up weighing a small amount more than planned, we want the impact on our system to be small (e.g. a tiny decrease in fuel economy, etc.) rather than catastrophic (e.g. our satellite fails to reach the intended orbit). It is a major portion of our role as designers to provide design margin; this will enable the system to work, even if a few things don't work as well as planned. A classic example is managing the weight of a spacecraft; there is a hard limit for how much weight the launch vehicle can take to the designated orbit. It you allocate all of that weight, and something shows up late a few pounds over, you are in big trouble. So, instead you should keep some design margin (in this case, unallocated weight) in your “back pocket.” As you get closer to delivery, you can allocate from your design margin to solve problems. This is analogous to the way the program manager allocates from her/his management reserve of funding (which we will discuss in a later chapter). But, at the same time, you cannot keep an unreasonable amount of design margin; it costs too much, and detracts from operational performance.
• Avoidance of known design pitfalls. I have seen a lot of system designs. Those that fail often share a small number of characteristics. So, I advocate checking that your candidate design avoids those known design pitfalls. See Figure 2.19.

Of course, we may learn something from this assessment that causes us to adjust the design, or even to have to return to the trade study, create additional candidates, and reassess.

Once you have created a candidate design that appears to satisfy the requirements and meets your measures of goodness, then we need to do a more thorough analysis to show that this candidate actually meets all of the requirements. You inserted the functional requirements – the statements about what – into the mission threads, and used the mission threads to help create design candidates, so there is a reasonable chance that they are all addressed by the design. But we need to verify that. We do this by a traceability analysis, where we map every requirement to a section of the design that implements that requirement.

Furthermore, there are many other requirements: all of the how well requirements, which deal with reliability, capacity, timing, availability, and many other quantifiable matters. These are not usually addressed by a single step on a mission thread, but instead by a more subtle amalgam of elements within the design. Proving that they are addressed by the design is part of the traceability analysis too, but often the verification of the incorporation of the how well requirements must be verified through use of our system model and its predictions about the operational and technical performance measures.

2.3.2 The Design Hierarchy is Not the Same as the Requirements Hierarchy

A common mistake is to assume that the groupings created to organize the requirements – the functional groupings – are appropriate for organizing the design. My experience is that people who try to do this end up with bad designs. For the requirements, it is best to group things together that are functionally related, but we must group things differently for the design: when we design, it is best to group things together that support the implications of the steps that must be executed in sequence, and the freedom and performance improvement opportunities implied by steps that could be executed in parallel.

2.3.3 Modeling

Modeling appeared on the list in Figure 2.17, and in several other places I have made reference to the idea of a system‐level model that we use for making predictions about the behavior of our eventual system. Let me say a bit more now about modeling.

Modeling of some sort is used in most large system development efforts. These days, such modeling usually takes the form of a computer program that is designed to simulate selected aspects of the behavior of the eventual system. We use such a model to make predictions about how well the candidate design will work, and therefore hope to make better design decisions. Using such a model can both help you improve each design candidate, and also help you select the final design.

A typical computer‐based simulation model will simulate the components of a system, their actions and interactions, and the external interfaces (which provide stimuli to the system).

It is easy to code up a model; the first big issue is the credibility of the model. Why should anyone believe that the predictions from the model are reasonably accurate? Just because the computer says so is not good enough; computer models can at times make predictions which are wildly inaccurate. We use techniques such as the following to prove the validity and credibility of a model's predictions:

• Analytic validation. We check that the algorithms are coded correctly within the model.
• Calibration against benchmarks. We use the model to make predictions about things for which we already have actual measurements of a real system; we then believe that we can rely on the predictions of the model for additional situations.
• Assessment of the accuracy and risks of extrapolation beyond the benchmark data. Obviously, using the model to make predictions for situations where we do not have actual measurements is one of the primary purposes of employing a model. But such extrapolation necessarily entails some uncertainty; if you were doing an experiment where you were measuring the viscosity of water as you cooled it, and you made measurements at 60, 50, and 40 °F, you would conclude that cooling the water did not change the viscosity very much. You might therefore be tempted to extrapolate, and make a prediction about what would happen if you cooled the water to 30 °F.¹¹ Since such extrapolation is a major purpose of our models, we must make explicit efforts to look for the non‐linearities, phase changes (such as the fact that water will freeze if cooled to 32 °F!), turbulence, queuing, and other major disruptions that would limit the range of validity of such extrapolations.

There is another big issue with models: How accurate is the model? We use the term fidelity: How faithful to the real world are the predictions of the model? Nor do we always need 10 decimal places of accuracy: How accurate is good enough for my purposes?

We start with the question of determining the necessary fidelity with an error budget analysis of the system. What is an error budget analysis?

Recall the air defense system that we described earlier in this chapter. We said that the gunner could not shoot a missile at an airplane until after a trained operator had conducted a visual identification of the target through an appropriate magnifying optic. The airplane, of course, is constantly moving. The vehicle with the missiles and the trained operator is moving, too. In order to get the magnifying optic on that vehicle to point at the correct airplane, there are a series of steps that must be completed, all within some degree of accuracy; if we do not achieve at least that accuracy, the optic may be pointing in the wrong direction. To achieve the necessary degree of overall accuracy, we must specify the degree of accuracy for each step along the processing chain, and then combine all of those individual errors correctly to arrive at an estimate for the overall accuracy. That is an error budget analysis; it tells you how accurate each component of your system needs to be in order for the system to meet its performance requirements (how well). Obviously, then, our model needs to be slightly more accurate than the actual system will be (there are ways of calculating exactly how much more accurate the model needs to be), in order to make useful predictions about our system.

One can then verify the achievement of that accuracy (at least for some scenarios and ranges of data) through benchmarking (that is, making measurements under controlled circumstances, such as in a laboratory, or under constrained field conditions). Furthermore, as you build the actual system, you can continuously re‐benchmark the model against the emerging actual system over a larger and larger range of scenarios and data.

Models are often nested or chained. An example of such nesting might be:

• A physics model of radio‐frequency propagation, which feeds …
• A model of an antenna, which feeds …
• A model of the antenna mast height, which feeds …
• A model of the received signal quality, which feeds …
• A model of successful packet completion rate, which feeds …
• A model of message completion delay (average and variance), which feeds …
• A model of end‐to‐end completion time and accuracy for a specific capability, which feeds …
• A measure of some system operational performance measure!

There are, however, many different ways to interconnect these models. Sometimes, they are all put together into a model‐of‐models, with fully automated interactions between each model. Other times, they are run separately, but the outputs from one are automatically fed into the next model in the chain (these are usually called federated models). Other times, the models are completely disjoint, and the outputs from one are manually transferred into the next model in the chain.

It is my experience that it is important that each model be maintained and operated by its actual creator; that creator is the expert who knows the limits of credibility for their own model better than anyone else, and having them maintain and operate that model, in turn, contributes to achieving better and more credible predictions. This motivates me usually to prefer the use of separate models with manual transfer of data! That sounds old‐fashioned, but better accuracy and credibility in my view is more valuable than automated interconnection.

We use the models to analyze our system and its candidate designs; that is, to determine how well each of our candidate designs performs. Since we are concerned with whether it meets the needs of the users, the model must finally reach the level of being able to make predictions about the operational performance measures, not just about the technical performance measures. This is a common failing of system models; many are designed only to make technical predictions.

2.3.4 Design Patterns

In the building construction business, there is something called the “building code,” which actually defines how certain portions of the design for a building must be accomplished. This lowers risk, in that these model design segments can be created by experts, and reused by average designers. This is an example of what I call a design pattern: a vignette for a small portion of a design that has been codified by some authority as working well, under the appropriate conditions.

The last bit about appropriate conditions is important; a bit of building design that is great for a single‐story private residence might be a disaster for a 30‐story office building (and vice versa).

We too can make use of design patterns in designing our engineered systems. There are plenty of such design patterns for us to consider: things like EtherNet, TCP/IP (the network transport standard for the Internet), various standards for electronic circuit cards so that they can plug into a standardized backplane, and both the client‐server and HADOOP data storage architectures; these are all examples of such design patterns.

In the context of the engineered systems that we are considering herein, things can be much more subtle than when the local city building and safety department tells you that you must have studs every 18" in the subfloor for the new room in your house (which is, of course, another example of a design pattern). The problem lies not in being unable to understand client‐server or HADOOP; rather, the problem lies in understanding when each is appropriate or inappropriate for our particular system. The range of design variation in buildings is far smaller than the range of design variation in engineered systems. This is in part due to the intrinsic complexity of today's engineered systems; something with 100 000 000 lines of software code is far more complex than an office building. But remember the examples cited above about staggering levels of variation in quality between apparently similar engineered systems; I cited ranges of 100× in speed and 1000× in reliability. A major cause of this variation is poor matching of design patterns to the specific nature of the system being designed (e.g. using client‐server when that is just a poor choice for the nature of the system).

Good system design organizations keep libraries of past system developments (successful or otherwise), and you can peruse those in order to gain insight about which engineering design patterns are likely to be effective for your system.

I wish I had a better answer for selecting design patterns; we need to use design patterns in our systems (our customers will expect it), but if we choose patterns that are inappropriate for our particular system, we are likely to end up being the next example of a system failure. I advocate careful benchmarking and modeling of your design patterns in the context of your system before finalizing selection. There is, in my experience, too much reliance on the idea that “this is a standard approach, so it ought to work for our system, too” without understanding whether or not such use is appropriate to the specific nature of your system.

2.3.5 Do the Hard Parts First

There is a temptation to start design (and implementation too) with the easy, low‐risk parts of the system. For example, there may be portions of a system that are reused from previous systems with only minor adaptations. Another favorite for early design activity is the user interface software.

These are among the easier aspects of the design, and some people like to start with them so that the team can make a lot of quick progress. This might be thought to help the team learn to work together, and to establish credibility with the customer.

While one can in fact make quick progress by concentrating at the beginning of the design (and implementation) phase on the easy parts, there are, in my experience, far better reasons to start the design and development phases with the hardest and/or highest‐risk portions of the system. If nothing else, this approach provides you with more time to work on these harder/higher‐risk portions, and it is often the case that having more time is a pretty effective element of a risk‐mitigation strategy. Beware of those who would have you concentrate first on the “low‐hanging fruit”!

2.3.6 Designs and Your Team

A former radio personality¹² used to make his audience laugh by saying “… and all of the children are above average.” We laughed because we knew it could not be true – very few children can actually be materially above average. We also laugh because people recognize that they do tend to believe that too often about their own children.

There is actually a very important insight in this idea. You might be able to select the staff for a three‐person project so that everyone is above average. You cannot, however, realistically do that for a 30‐person project, nor for a 300‐person project. Most people on such larger projects will be … about average.

At the same time, it is often the case that a design requires that a small segment of very difficult and error‐prone work is spread across the tasks that many different people have to do. This was true, for example, in early implementations of cyber‐security techniques; every programmer had to implement a relatively small amount of very complex and highly error‐prone software code. Not surprisingly, the result was a lot of design and coding mistakes, causing very poor cyber‐security characteristics in the resulting system. If most of the designers and programmers on your project are average, you cannot expect them all to perform properly on very difficult tasks.

There is, however, a better alternative. It is possible to use the design process to isolate certain types of difficult and error‐prone tasks into a small segment of the implementation.¹³ You can then assign this isolated bit of work to a small set of experts. The overall difficulty of the entire project remains the same, but you have partitioned the design into (a small set of) difficult parts and (a large set of) less difficult parts. This matches the distribution of the difficulty of the work to the distribution of skills on your team: you have a small amount of difficult work, which is isolated into a small segment of the design and implementation, and you can therefore assign this difficult work to your small number of expert practitioners. You have a large amount of work of average difficulty, which you can assign to the remainder of the team. I have found that this approach significantly improves the outcome on real projects. This is a new and, I think, important design objective: matching the distribution of the difficulty of the work to the distribution of skills on your team.

A second example of how to use this technique involves the management of the system's dynamic behavior; I have found that this is a highly complex and error‐prone task, and one that has a significant impact on the overall success of the project. Remember what I stated above: most designers remember to design to implement the dynamic behavior they desire, but do not do nearly enough to prevent other (unplanned and undesirable) dynamic behavior. This is all very difficult work, especially the latter portion.

On many projects, responsibility for designing and implementing such controls is (perhaps inadvertently) spread out to every software package; because this is difficult work that ends up being performed by almost every member of the software team (most of whom are, after all, average in skills), this leads to poor technical performance, low system reliability, and lots of schedule and cost over‐runs. This was in fact the most common design flaw that I saw in all of those systems for which I was tasked to act as the fix‐it person.

To address this problem, I adapted technical concepts invented by some of my colleagues¹⁴ into a management and design methodology for isolating this complexity into a small segment of the design and software implementation, which allowed me to assign this difficult, error‐prone, but highly significant work to a small team of experts.

Good project managers have always tried to match people to assignments, but this quest was often confounded by the phenomena noted above: really difficult and risky tasks can creep into every single part of the system. Using the design process to avoid this problem by the sort of partitioning described above gives the project manager a new and powerful tool to improve the probability of a good outcome for their project. The design stage of your project is where you can do this.

2.3.7 Summary: Design

As I stated at the beginning of this chapter, in my experience, design is the make‐or‐break activity for most engineered systems. As you can tell from what we have discussed in this chapter, there are a lot of established practices that can get you going, but there is a lot that depends on judgment (which comes from experience) and art.

Design is also one of the most fun aspects of systems engineering. Learning to do it well will distinguish you from your colleagues.

In Figure 2.20 I summarize some of the key points we have discussed about the design activity in a format called IDEF‐0.