4. Requirements, Sin, and Contracts

Boeing-Sikorsky RAH-66 Comanche helicopter, originally the LHX
Sikorsky Aircraft/Richard Zellner/AP Wide World Photos

Any attempt to formulate all possible requirements at the start of a project will fail and would cause considerable delays.

PAHL AND BEITZ [2007], ENGINEERING DESIGN

The committee believes that getting to a state of clear and complete system-level requirements requires the interaction with potential contractors that occurs between Milestones A and B.

JAMES GARCIA, FOR THE AIR FORCE STUDIES
BOARD COMMITTEE ON PRE-MILESTONE A
AND EARLY-PHASE SYSTEMS ENGINEERING

A Horror Story

The general had spent a career in Marine Corps aviation and knew helicopters. He and I had been dispatched into the depths of the Pentagon as a subcommittee of the Defense Science Board. We listened intently as a colonel briefed us on the in-progress design of the LHX (the Comanche), the next-generation light attack helicopter on which billions would be spent, and upon which soldiers’ lives would depend. The helicopter was to be the successor to four different current helicopters, which had different missions.

The colonel outlined the requirements that had been developed by an inter-group committee representing the several using groups:

“Fly so fast, so far. Carry X armor; mount Y weapons; carry Z supply of ammunition; carry W fully equipped warriors, besides the crew.”

“Fly close to the ground, under the radar cover. Even on a dark and stormy night, jump up to avoid obstacles. Pop up to shoot; pop back down to evade returned fire.”

Then, without any change of inflection or expression, he said, “And it must ferry itself across the Atlantic [way beyond its normal operating range].”

The general and I were visibly shocked. The briefer quickly responded, “Oh, it will be designed to do that, by taking out all the ordnance and ammunition, and filling it with drums of gasoline.” But our incredulity bore on rationale, not feasibility. Even I, a computer designer, knew in my gut that one had to pay for such a capability—not in dollars but in reduced capacity elsewhere.

“Why must it do that? Surely, it will have to do that only twice, if you’re lucky.”

“We don’t have enough C-5 transport planes to carry them over.”

“Why not take a bit of the program money for the LHX and buy more C-5s, instead of compromising the LHX design?”

“That’s not possible.”

Our shock at the extreme requirement was fully matched by our shock that its extremity didn’t bother our briefer. Perhaps our instincts were wrong. Perhaps the marginal cost of the self-ferrying capability was indeed low. Perhaps knowledgeable designers had wrestled with that question.

But our ensuing conversation was not encouraging. If my recollection is accurate, the LHX Requirements Committee included neither an aircraft engineer nor a helicopter pilot—but rather mostly bureaucrats skilled at representing their groups in inter-group negotiations.¹

Unfortunately, Not Unique

Many readers will have no trouble conjuring up a vision of a meeting of the LHX Requirements Committee; we have been to such meetings.

Each player has a wish list garnered from his constituents and weighted by his personal experiences. Each has both an ego and a reputation that depend on how well he gets his list adopted. Logrolling is endemic—an inevitable consequence of the incentive structure. “I won’t naysay your wish, if you won’t naysay mine.”

Who advocates in the requirements process for the product itself—its conceptual integrity, its efficiency, its economy, its robustness? Often, no one. As often, an architect or engineer who can offer only opinion based on taste and instinct, unbuttressed as yet by facts.² For in a classical Waterfall Model product process, requirements are set before design is begun.

The result, of course, is a grossly obese set of requirements, the union of many wish lists, assembled without constraints.³ Usually, the list is neither prioritized nor weighted. The social forces in the committee forbid the painful conflicts occasioned by even weighting, much less prioritizing.

Eventually, wish list and constraints must be reconciled. In practice, the product designers implicitly weight the official requirements using their own personal user models.⁴ In many cases, the failure to provide weightings has decoupled the designers from deep user and application knowledge the several requirements setters do actually possess.

Generating requirements by committee, by the nature of the beast, tends to produce products that are too rich (Detroit cars, bloated software systems, unbuildable Internal Revenue and FBI systems). Perhaps committee specification is why large and super-ambitious software systems seem so prone to total disaster.

In the building of the IBM Operating System/360, requirements were initially set by a large committee from the Marketing Division, following a process and producing a result just like that described. As Project Manager, I had to reject the requirements document as totally impractical and have a quite small team of architects, marketers, and implementers extract the essence.

Fighting Requirements Bloat and Creep

Requirements proliferation must be fought, by both birth control and infanticide. A very insightful report from the National Research Council’s Air Force Studies Board addresses both attacks.⁵

The Committee on Pre-Milestone A and Early-Phase Systems Engineering starts with its own horror story. Major military systems 30 years ago had a development time of around five years. Now times from program initiation to system deployment are two to three times longer, even while the pace of both technology and threat accelerates.

The dramatically successful programs of yore typically had one or a few clear overriding objectives and schedule urgency. These projects were begun with a few top-level requirements. As development proceeded, these were indeed broken down into more specific sub-requirements and key performance parameters, all under hard-driving, capable managers who continually balanced system function against schedule and cost.

As for requirements creep, whether pressed by users or by internal inventors, schedule urgency had been the best defense. (This has been my own best defense, as well, in system building.) The committee perceives that Department of Defense acquisition has lost that permeating sense of urgency, replaced with ever-increasing layers of “oversight” mechanisms to avoid mistakes. Such a progression is also not unknown within technical corporations.

The committee recommends that an organized systems engineering effort begin well before even the focused technology development for a new system is begun. But they recommend not the initial specification of requirements before Milestone A, but that the initial statement of detailed requirements be defined during system development, between Milestone A and Milestone B:

The definition of clear key performance parameters by Milestone A and clear requirements by Milestone B that can remain stable through Initial Operational Capability can be essential to an efficient development phase.⁶

Requirements creep is addressed head-on in the most effective way. The committee’s top-priority recommendation is: Appoint early-on strong, seasoned, domain-knowledgeable managers who can stay with the program through initial systems delivery. Then empower them to “tailor standardized processes and procedures as they feel is necessary [emphasis added].”⁷

They also urge the use of a Requirements Traceability Matrix to ensure that each requirement detailed, defined, and laid on is indeed derived from one or more of the initial overall requirements—that it didn’t sneak in from a request by some user representative or a designer’s desire to do something clever, novel, and putatively useful.⁸

Sin

Suppose

• There is a client who is never greedy, but quite happy to pay his architect and builder fair prices for their expertise and labors (perhaps from enlightened self-interest, because he will want their help again).

• He has engaged an architect who always considers himself the client’s agent, eager to use his talents and professional skills to best uncover and serve the client’s true interests.

• He is contracting with a builder who sees and invariably performs his calling so as to produce high-quality products at the best possible value/cost ratio, within budget and on schedule.

• All the players are honest and truthful, and communication among them is excellent.

Then

I would argue that

• A cost-plus payment arrangement will give the client the most value per dollar.

• Design-build is the most rapid way to get a project built.

• An explicit Spiral Model process (next chapter) will yield the product best suited to the client’s needs.

If this last is true, how can we explain the persistence of the Waterfall Model, when the greater fidelity of the Spiral and Co-Evolutionary models has been seen for more than a quarter century?

The one-word answer is sin: pride, greed, and sloth. We all recognize the suppositions above as ideals. The reader may have snorted on reading them: “Fat chance of all those conditions prevailing!” Because humans are fallen, we cannot trust each other’s motivations. Because humans are fallen, we cannot communicate perfectly.

Contracts

For these reasons, “Get it in writing.” We need written agreements for clarity of communication; we need enforceable contracts for protection from misdeeds by others and temptations for ourselves. We need detailed enforceable contracts even more when the players are multi-person organizations, not just individuals. Organizations often behave worse than any member would.

Clearly, it is the necessity for contracts, whether within an organization or between organizations, that forces the too-early binding of goals, requirements, constraints. Everyone recognizes the fact that these must later be changed. (This opens new opportunities for wrongdoing: “Low-ball on the contract; make it up on the change orders.”) So it seems that the necessity for contracts best explains the persistence of the Waterfall Model for designing and building complex systems.

A Model for Contracting

The pressure for a complete and agreed-upon set of requirements comes ultimately from the desire—often, an institutional demand—for a fixed-price contract for a specific deliverable. Yet this demand runs head-on into the hard fact, argued in Chapter 3, that it is essentially impossible to specify a complete and accurate set of requirements for any complex system except in iterative interaction with the design process.

How have the centuries-old building design disciplines handled this perplexity? Fundamentally, by a quite different contracting model. Consider a normal building design process:

• The client develops a program, not a specification, for the building.

• He contracts with an architect, usually on an hourly or percentage basis, for services, not for a specified product.

• The architect elicits from the client, the users, and other stakeholders a more complete program, which does not pretend to be a rigid contractable product specification.

• The architect does a conceptual design that approximates the reconciliation of program and the constraints of budget, schedule, and code. This serves as a first prototype, to be conceptually tested by the stakeholders.

• After iteration, the architect performs design development, often producing more detailed drawings, a 3-D scale model, mock-ups, and so on. After stakeholder iteration, the architect produces construction drawings and specifications.

• The client uses these drawings and specifications to enter into a fixed-price contract for the product.

Notice how this long-evolved model separates the contract for design from the contract for construction. Even when both are performed by the same organization, this separation clarifies many things.

Of course, this model isn’t fully sequential, either. As anyone ever involved in even a modest construction project knows, practical construction problems and late-stage client changes in either needs or design evaluation will occasion design changes, which in turn will necessitate contract change orders.

The classical architectural process sketched above has its own drawbacks, not least an extended schedule. Building projects with

• Close client-architect-contractor trust relationships,

• Well-understood design challenges, or

• A pressing hurry, justifying higher risks,

often conflate the normal process into a concurrent, pipelined design-build process. The architects organize their work so that the detailed construction drawings are produced first for the parts the contractor will need first: long-lead-time steel, site work, foundations.

System projects that meet the bulleted conditions should similarly be able to proceed on a design-build basis, too. Here the challenge is for the computer and software builders to identify the build order and the long-lead-time components.

Much hard thinking remains to be done here. I challenge the community to engage in this dialog.^9,10

Notes and References

1. The Wikipedia [2002–2009] article on “RAH-66 Comanche” tells the history of the program. Its description of the helicopter’s properties confirms my recollection of the stated requirements:

The Comanche’s very sophisticated detection and navigation systems were intended to allow it to operate at night and in bad weather. Its airframe was designed to fit more easily than the Apache into transport aircraft or onto transport ships, enabling it to be deployed to hot spots quickly. If transport assets were not available, the Comanche’s ferry range of 1,260 nautical miles (2,330 km, 1,553 miles) would even allow it to fly to battlefields overseas on its own.

In the event, the LHX evolved from a light attack helicopter to a reconnaissance helicopter, the Comanche pictured in the chapter frontispiece. Two were built; the program was scrapped because unmanned drones had taken over the reconnaissance function.

2. Squires [1986], The Tender Ship, studied government acquisition of innovative technology. “A theme running through the book is that the key to success is allowing the designers to be faithful to the engineering integrity of the product” (Mary Shaw, reviewer’s comment). Squires urges designers to have a passion for the product’s integrity:

An applied scientist or engineer shall display utter probity toward the engineered object from the moment of its conception through its commissioning for use.

3. An anonymous reviewer correctly points out that one stakeholder’s frill is often another’s necessity. The effect I see, however, is that particular features have ardent champions. On the other hand, although everyone wants speed, smallness, robustness, ease of use, these requirements have no ardent champions in the requirements process, largely because the effect of a particular feature on them can’t be known so early.

4. Chapter 9 discusses designers’ mental models of users.

5. Air Force Studies Board [2008], Pre-Milestone A and Early-Phase Systems Engineering.

6. Air Force Studies Board [2008], Pre-Milestone A and Early-Phase Systems Engineering, 4. But see page 50, which might be misinterpreted:

One must clearly establish a complete and stable set of system-level requirements and products at Milestone A. While requirements creep is a real problem that must be addressed, some degree of requirements flexibility is also necessary as lessons involving feasibility and practicality are learned. ... Certainly control is necessary, but not an absolute freeze.

By communication with me, both the chair of the committee, Dr. Paul Kaminsky, and the NRC staff member responsible, Mr. James Garcia, clarified that the committee’s intent is as stated in the page 4 paragraph. Mr. Garcia says:

It was the committee’s intent to say that clear KPPs [key performance parameters] be developed by MS [milestone] A, and clear and complete requirements by MS B, as stated in the Summary and in Chapter 4. The committee believes that getting to a state of clear and complete system-level requirements requires the interaction with potential contractors that occurs between MS A and B.

7. John McManus, MBCS, is a leading practitioner in project management and software development methods. Dr. Trevor Wood-Harper is Professor of Systems Engineering at Salford University: “A project charter laying out the reasons and expectations for a new initiative, and a project manager’s vision for the task ahead are vital starting points for IT projects” (McManus [2003], Information Systems Project Management).

8. Boehm [1984], “Prototyping versus specifying,” describes a classroom experiment in which one team built from lovingly crafted design specifications and a second team essentially built straight from the requirements. The first team’s system suffered from feature bloat because the designers kept putting in things to “complete” the design or make it consistent. So it’s not just the wish-listers who cause bloat—designers themselves can do it. I’ve done it myself, on the IBM Stretch computer.

9. Jupp [2007] treats the novel contracting schemes that have been used in public-private partnerships for public works in the UK.

10. Muir Wood [2007], “Strategy for risk management,” is a conference paper that proposes how tunneling clients and contractors should deal with unforeseen risk in their contracting.