Market forces continue to pressure IT organizations to become as efficient as possible in order to stay competitive. As a cost-cutting maneuver, IT organizations were reorganized during the recent economic downturn and cut to the bone. Since no more obvious costs remain to be cut, more recent efficiency efforts focus on improving productivity instead of cutting costs. Although the situation sounds bleak, software development shops inside IT can take advantage of businesses’ drive to improve productivity in order to build better software. By harnessing productivity momentum, efforts to formalize software process improvement programs and achieve productivity goals are flourishing.

The regulatory and compliance environment is aligned with good security too, and in some cases, outweighs the productivity concerns. Many mid-level decision makers are very worried about compliance (or non-compliance, as the case may be). There’s lots of bad press out there, and they don’t want to be run over by it organizationally.

Any organization can initiate a change, but few have experience in sustaining change over time—the ultimate end state for any software security improvement program. So where to start? How can we define and manage a change program in today’s dynamic business environment? How can we prepare for and take advantage of natural change? How can we build a sustainable improvement program and a plan that is flexible enough to adapt over time?

Priority one is aligning software development and operational processes with strategic business objectives. Sometimes technologists forget why they are doing what they are doing. Yet most software today is created to service business. Software security practices and mechanisms will succeed only to the extent that they have clear and explicit connections to the business mission. Recall our discussion of the RMF in Chapter 2. The stakes are high. In terms of pure technology, what is at stake may be some new authentication feature versus avoiding attack 57. But translated into risk-related business terms, when the technologist says the fizzbob-authentificator is broken, mitigation becomes a decision between a $13 million PKI installation and a $10 million Directory service. All the poor, outgunned VP knows is that there is some technical problem with user identity. Making the right decision is essential. Those technologists who understand that security is a risk management process that unfolds over time will have little trouble understanding that business concerns are a fundamental driver in balancing and refining security best practices.

A well-architected vision and plan based on industry standards and best practices is essential to a successful software security program. Throughout this book, I have covered a number of software security touchpoints that are process agnostic and can thus be adopted regardless of an organization’s software development methodology. Because every organization is different, a software security improvement program plan that involves the adoption of these best practices must be tailored to the given business and technical situation. For example, organizations that focus more attention on code than on software architecture will likely benefit more quickly from the adoption of static analysis-based code review than they will from architectural analysis. First things first.

A well-defined roadmap lays out the specifics of how best to deploy software security best practices given a particular organization’s approach to building (and even buying and integrating) software. Explicit strategic objectives drive prioritization of change to ensure that only those program initiatives that will provide the biggest and/or quickest return are addressed first. Executing such a roadmap is carried out in five basic steps.

Every cultural change program requires buy-in from both management and tactical technical people. Improvement programs will fail if either group is left out or even underemphasized. Every organization, every group within an organization, and every stakeholder will have a different sensitivity toward change. These differences must be understood and accounted for because variances in sensitivity deeply affect expectations. Disconnects in expectation may eventually end up forcing an organization into a least common denominator approach that lacks impact. Some common pitfalls are described in the box Overcoming Common Pitfalls.

Keeping things simple is good because this enables people to understand and support a program—but don’t lose track of the big picture. Breaking a major change program down into logical segments of work, with specific deliverables tied to each segment (which we also call an initiative), is a proven tactical approach. In practice, we find that a reasonable time range for any given initiative is three to four months. A stepwise approach minimizes risk while enabling an organization to test the waters as it gauges receptivity to change.

In terms of breaking a program down, my approach recommends a mixed method of planning for dependencies blended with a sequence of initiatives that builds on itself. Dependencies can be used to adjust the general sequence to account for those items likely to require some dependent task prior to being kicked off. For example, building a set of measurement tools will be directly dependent on the software development methodology that is used. If an early segment includes the selection and/or adoption of a given methodology, tool choice issues should be deferred to a later segment because they require an in-place methodology to be effective.

A clear sequence of initiatives allows an organization to achieve a specific level of adoption, test the waters, measure and validate accomplishments, and set the stage for the next level. Cigital follows a change program maturity path sequence with the following six phases:

Phase 1: Stop the bleeding is targeted at those areas of software development programs that are known to be problem areas. If particular security bugs like buffer overflows are causing the biggest problem, a good phase 1 approach might involve the adoption of a code scanning tool and an associated process for its use. If there are tens of thousands of security-critical applications with unknown risks, a good phase 1 approach might be to carry out a flyover risk analysis process and organize the applications in order of criticality/security exposure so that the plan addresses those applications most at risk first.

Phase 2: Harvest the low-hanging fruit is focused on finding quick wins that are instrumental in getting buy-in from the organization and in helping a change program build momentum. Note that this phase and its predecessor are good barometers for determining the organization’s receptivity toward change.

Phase 3: Establish a foundation is about setting in place components that provide building blocks for future initiatives. Typical areas addressed in this phase include creating change control programs, building a root-cause analysis function, and setting up critical feedback loops. One such feedback loop identifies and cycles any security problems discovered through the application of best practices, such as code review, back into training (in order to teach developers how to avoid common security problems in the first place).

Phase 4: Craft core competencies is driven by both current strengths and desired strengths of the organization. If an organization has a strong reputation for creating solid architecture documentation, it will likely be more receptive to architectural risk analysis than it may be to abuse case development. This phase explicitly involves the adoption of software security best practices in a manner tailored to the strengths of the organization.

Phase 5: Develop differentiators in order to emphasize and highlight those capabilities that separate the organization from everyone else in the marketplace. Measurement and metrics systems put in place with a software security improvement program can be used to demonstrate how well things are going from a security perspective. This can serve as an important differentiator in the market.

Phase 6: Build out nice-to-haves involves adopting those capabilities that are not necessarily aligned to a given strategic business objective but bring value by achieving some improvement in productivity. These are left for last for obvious reasons.

Once a specific and actionable plan is set, a pragmatic approach should drive each initiative. Developing a clear understanding of what will be built during each part of the program; who will own it; and how they will build, deploy, and continue to improve it over time is essential.

The general framework and plan discussed earlier should include a number of factors, including (but not limited to):

All of these concerns should be related and described in terms of who, what, and when, especially in large organizations. Additionally, there are a number of drivers required that can help align the framework with the strategic business direction. These include current software architectures, security policies and guidelines, and regulatory requirements, to name just a few. An all-encompassing enterprise information architecture and associated enterprise architecture roadmap (including data sensitivity classifications and user/role/privilege maps across lines of business) is an absolutely essential anchor for framework-based adoption and change.

The most important decision for ensuring success in a cultural change program is the selection of champions—those individuals who will build, deploy, and own each initiative going forward. For example, should an initiative involve the adoption of static analysis tools for code review, a champion well versed in security analysis of implementations, the target language(s), and effective use of source code tools is necessary. Ideally, these individuals are not freshly trained in the area they are meant to own; rather, they should have a hand in developing the initiative and its components (including processes, success measures, and so on). A champion needs to be motivated; driven; and, most important, supported by the management team. Champions must be good communicators and part-time cheerleaders, and they must possess a strong capability to train and mentor others.

For each initiative, the assigned champion will drive the build, pilot, and deployment activities throughout the organization. The champion will also be responsible for monitoring, measuring, and improving the initiative over time. It’s important to understand and distinguish a technical champion from a business sponsor. In this case, I mean a technical champion.

The importance of measurement and metrics is hard to overstate. Measurement provides critical insight to management, allowing management to support strategic decision-making processes. Measures are numeric values assigned to a given artifact, software product, or process. A metric is a combination of two or more measures that together provide some business-relevant meaning. For example, when considered separately “lines of code” and “number of security breaches” are two distinct measures that provide very little business meaning because there is no context for their values. A metric made up as “number of breaches / lines of code” provides a more interesting relative value. A comparative metric like this can be used to compare and contrast a given system’s “security defect density” against a previous version or similar systems and thus provide management with useful data for decision making.

Ideally, metrics and measures will focus on four primary areas: project, process, product, and organization. The first three are specific to a given artifact or activity in a software development effort, while the purpose of the latter is to determine trends across the three other areas.

Establishing a metrics capability is a challenging undertaking. Early standard software process approaches focused on sequentially building a level of sufficiency in four areas and in a particular order: process, controls, metrics, and improvement. Unfortunately, following these basic steps in the prescribed order implies that metrics are not addressed until late in the program. By then it may be too late. In this case, processes and controls put in place early may not be properly designed to provide the kinds of metrics that are needed later. In those cases, some significant rework may be required to achieve business alignment.

All metrics should render decision criteria based on strategic business objectives. For that reason, business objectives must be articulated first and used to guide the entire program, from process and control development onward.

A Three-Step Enterprise Rollout

Figure 10-1 shows a simple three-step rollout plan for establishing an enterprise-wide, metrics-based software security program. This approach can be adapted for use in rolling out any large initiative. The three fundamental steps are (1) assess and plan, (2) build and pilot, and (3) propagate and improve.

Figure 10-1. A three-step rollout plan for enterprise adoption of software security best practices, based on establishing clear measurements and metrics up front.

Step 1 involves getting a handle on the current state of the business. This includes understanding the goals of the program writ large, collecting data to assess current state, and then comparing current state to goal state. This is in some sense a gap analysis. As an example, rollout step 1 in a large software security program will include understanding the in-place SDLC and assessing how well it covers the touchpoints discussed in this book. If code review is currently practiced only at the unit level by developers who are not using a static analysis tool, a clear gap has been identified between the goal (state-of-the-art software security) and reality. Since there are likely to be a large number of application development projects underway simultaneously in any large company, developing a rating system to assess each project is an important part of baselining. This leads to a measurement and metrics regimen that can be evenly applied throughout the rollout. Note that some of these measures can be taken from software artifacts as briefly described earlier.

Counterintuitively, it may be best to begin rollout step 2, build and pilot, by identifying a software project that is ahead of the game. That is, because you want to maximize the possibility of pilot success, starting with the project with the smallest gap may make things easier. For example, if the software project chosen for pilot is already using static analysis tools for reliability (looking for null pointers and other simple bugs), adopting a security-related source code analysis tool is likely to be fairly straightforward for the project team. Because we have a set of measurements in place, we can assess progress over time in rollout step 2 and refine our measurement system. Note that in almost all cases, training programs will need to be developed that clearly describe both the goals of the improvement program and how to actually carry out the new best practice. The material in Part II of this book should be particularly useful.

A successful pilot program provides an excellent real-world case study of the adoption of a best practice in one part of the enterprise. This success story provides “proof in the pudding” that a best practice, like code review with a source code security scanner, can be successful in the organization. Rollout step 3, propagate and improve, involves taking the best practice wide. By relying on the baseline gap analysis results from step 1, we can logically approach the problem of wide adoption. Our measurement program helps keep tabs on progress and is extremely useful in alerting us of adoption issues as they crop up. The training program developed in step 2 is a critical part of the widespread adoption of a best practice in a company. Also helpful is an information portal for software professionals to use as a resource as they adopt various software security touchpoints.

By following this straightforward rollout plan, a very large organization can transform its existing SDLC (or more likely SDLCs, plural) with the addition of best practices for software security. The idea of an SDL is thereby a combination of your SDLC with the software security touchpoints. This key point bears repeating. Presumably your organization already knows how to make software and has already been shipping it for years. There is no reason to throw out everything you’re already doing and start from scratch. Instead, your already-in-place SDLC can be adjusted by adding touchpoints. My process-agnostic approach, based around software artifacts, makes this possible.

The targeted end state for any improvement program (security or otherwise) is a sustainable ability to evolve and to change with the business climate. Improvement programs are focused on enabling an organization to develop consistent, effective standards that can be replicated throughout the organization over time.

As any organization carries out its day-to-day software development and maintenance functions, it will execute many processes. A critical foundation for continuous improvement is introspective in nature: Each process must be carefully analyzed, assessed with respect to the need for change, adjusted as appropriate, and reinstantiated after it is refreshed. This feedback cycle is critical for ensuring that any given initiative stays relevant. Process for process’ sake is a well-known pitfall that should be avoided. A feedback loop additionally helps to confirm that best practices like the touchpoints are in fact being followed. Unfortunately, many organizations have a tendency to become lazy and slip back into old habits. Control processes help counter this tendency.

A critical feature for the success of continuous improvement involves the periodic auditing and explicit reformulation of the organization’s strategic objectives to ensure that they have not changed too much over time. If business needs have moved far enough to push processes and procedures off track, then the entire software security initiative needs to be reevaluated.

All modern businesses are surrounded by change. Any business that does not embrace change will fall victim to the tides of the market. Businesses that successfully manage change become flexible and nimble enough to adjust to rapid market movement; to remain competitive; and, ultimately, to establish sustainable differentiators that improve their strategic position.

In this book, I am primarily interested in discussing how to build secure software by adjusting and updating existing software development processes. The touchpoints are described as simple best practices that can be added to any existing SDLC. That’s all fine and good for new projects, but to solve the software security problem, we need to think hard about existing applications, integrated software, and commercial off-the-shelf software (COTS) as well.

Fortunately, the problem of existing applications is very easily addressed. That’s because all of the touchpoints, as software security assurance activities, can be carried out against existing software applications as long as you have some artifacts. You’ll most certainly have executable binaries, so that’s a guaranteed starting place. But you’re likely to have other software artifacts, including source code, defect reports, architectural descriptions, and so on. By referring to these artifacts along with the running system itself, you can assess the security posture of an existing application and begin working toward making it more secure.

I am assuming that the existing application will be found lacking in its security. This is not a necessary condition, but it is one so commonly encountered that I treat it as a default. Getting a handle on a large pile of software applications (say, on the order of 1000 or so) is always an essential part of the baselining program described earlier. By measuring all of the applications with the same measuring stick (say, some kind of outside→in penetration testing or application security testing), we can get a rough estimate of which applications need the most help. In my experience, a cursory architectural risk analysis and a quick code review with a tool (the top two touchpoints) provide the most actionable data. Don’t forget that business impact is a critical factor here.

Assessing the state of existing software applications is a critical software security practice. The best approach is to perform a very high-level holistic analysis mixing both code review with a tool and architectural risk analysis to determine security posture (see Chapter 5). This can be done as part of routine maintenance schedules in order to be less disruptive in a large organization.

The COTS problem is more of a challenge. Because COTS vendors are often reluctant to provide the necessary software artifacts needed to determine security posture, using the touchpoints against COTS is not as effective.^[3] One approach is to attempt to bind the vendor legally (see Chapter 1). The Reagan-esque “trust but verify” idea applies in spades though, so taking a look yourself is almost always a good idea. You’ll always have executables, so that starting point exists. The hacker techniques discussed in Exploiting Software are also extremely useful when confronted with the COTS analysis problem [Hoglund and McGraw 2004].

A closely related issue involves the kind of modern enterprise software most commonly encountered in large businesses today: software that has both COTS components and middleware glue. In this common situation, it pays to have a coherent enterprise security framework in place that governs software security policy from development through integration and also covers purchasing.

An Enterprise Information Architecture

Figure 10-2 shows an overview of a generic enterprise information architecture (IA). Please note that any such IA must be adjusted to account for the software situation found in the large company where this architecture is to be applied. The IA is not a one-size-fits-all solution, but rather a large-scale “map” that describes goal state. The IA concept is designed to help large organizations think through what knowledge objects are necessary to apply for each touchpoint.

Figure 10-2. An enterprise information architecture, including critical software security documents, processes, and contracts.^[4] See the text on previous page for definitions of abbreviations.

The IA in Figure 10-2 includes several critical sections. Starting at the bottom, a number of essential documents labeled “Collateral” in the picture describe the security vision, document policy, and point to standards and requirements. For our purposes, the most interesting pieces of collateral are design patterns and code samples. These two essential knowledge categories are extremely useful in constructive software activities. Design patterns provide reference architectures (best tied back into policy and standards) and may address particular platforms such as .NET or J2EE. Code samples are similarly anchored in policy and standards, but they include actual usable code samples, possibly even to the level of class files with standardized APIs. These components allow software professionals to understand what their goal is (not the use of security features, per se, but the use of secure coding design and implementation techniques). Positive examples also make excellent fodder for later assessment and measurement activities.

Moving up the picture, the part labeled “Contractual Artifacts” describes aspects of software projects that allow outsourced development to make sense. By governing a software development project with a clear master services agreement (MSA) and statement of work (SoW), especially if the documents include service-level agreement (SLA) and quality-of-service (QoS) callouts to security, we can hold outsource vendors accountable for software security. Evidence can be provided in terms of risk analysis reviews, test results, code scanning results, and so on.

Moving one set of boxes higher, we encounter activities most useful when buying software. These are loosely analogous to the touchpoints described in this book, but they are more readily applicable to software purchase. Note that the black hat best practices are particularly useful when buying software. Developing threat models, abuse cases, and security requirements while thinking hard about attack patterns is possible even if you are standing outside the “box of code” with little wherewithal to get in the box.

Architectural review is also possible on COTS products, depending on the documentation provided with the code. In the case of open source adoption, you’re in better shape. But even when proprietary software is involved, a wealth of information useful to security analysis is often available. In any case, the level of analysis can be dialed in to reflect the information available for the product under review. Penetration testing (see Chapter 6) is almost always useful in assessing a COTS product, especially if it is based on risk analysis results.

Finally, at the top of Figure 10-2 we come to the all-familiar touch-points described in this book. These best practices are best applied when building software (or analyzing existing applications for which you have a nice set of software artifacts). As you can see, the IA as a whole is applicable to a very diverse set of software circumstances, ranging from code you build yourself all the way to off-the-shelf code that you simply integrate into your environment.

Figure 10-3 shows another view of the same IA, this time annotated with various labels. As you can see, the IA labels align with our previous description. The labels help to show how an overall security IA can be used to govern software processes, both for outsourcing situations and for bespoke building situations.

Figure 10-3. An IA annotated to describe different software situations, from buying software to building it. The tie to organizational policy is a key aspect of the IA but necessitates a very mature security organization that has described its vision and policy.^[5]

This IA is annotated to indicate the impact that collateral has on defining process and vice versa. It is meant to imply that process (like the famous touchpoints) can’t succeed without proper knowledge backup. The good news is that even if you find yourself in an organization with looming process problems, you can still enjoy initial success with the touchpoints. This success comes through building out the knowledge and sneaking the risk management methodology and activities into it.

I have found that those enterprises with more mature corporate governance policies and procedures are better able to comprehend and actualize an IA like this. Those software houses with a smaller handle on policy (and usually a correspondingly larger software clue) are best suited to approach software security through the touchpoints. Those enterprise development shops driven by a top-down governance structure are more likely to see success through clear description and imposition of a corporate IA.

An SDL is a combination of your existing SDLC and the best practices described in this book. There is no reason to wipe the software slate clean in order to adopt software security best practices. You know how to build software. The trick to effective software security is to adapt your current approach according to a process-agnostic plan.

A critical challenge facing software security today is the dearth of experienced practitioners. Approaches that rely solely on apprenticeship as a method of propagation are unlikely to scale quickly enough to address the increasingly more serious problem, so as the field evolves and best practices are established, business process engineering can play a central role in encapsulating and spreading the emerging discipline more efficiently.