Software development can affect all of the six fundamental principles of information security as described in Chapter 6 in this Handbook, but the most frequent problems caused by poor software involve integrity, availability, and utility.

Despite the ready availability of packaged software on the open market, such software frequently has to be customized to meet its users' particular needs. Where this is not possible, programs must be developed from scratch. Unfortunately, during any software development project, despite careful planning, unforeseen problems inevitably arise. Custom software and customized packages frequently are delivered late, are faulty, and do not meet specifications. Generally, software project managers tend to underestimate the impact of technical as well as nontechnical difficulties. Because of this experience, the field of software engineering has developed; its purpose is to find reasonable answers to questions that occur during software development projects.

39.2 GOALS OF SOFTWARE QUALITY ASSURANCE.

In the IEEE Glossary of Software Engineering Terminology, quality is defined as “the degree to which a system, component, or process meets customer or user needs or expectations.” In accordance with this definition, software should be measured primarily by the degree to which user needs are met. Because software frequently needs to be adapted to changing requirements, it should be adaptable at reasonable cost. Therefore, in addition to being concerned with correctness, reliability, and usability, the customer also is concerned with testability, maintainability, portability, and compliance with the established standards and procedures for the software and development process. Software quality assurance (SQA) is an element of software engineering that tries to ensure that software meets acceptable standards of completeness and quality. SQA acts as a watchdog overseeing all quality-related activities involved in software development.

The principal goals of SQA are:

Uncover all of a program's problems.
Reduce the likelihood that defective programs will enter production.
Safeguard the interests of users.
Safeguard the interests of the software producer.

39.2.1 Uncover All of a Program's Problems.

Quality must be built into a product. Testing can only reveal the presence of defects in the product. To ensure quality, SQA monitors both the development process and the behavior of software. The goal is not to pass or certify the software; the goal is to identify all the inadequacies and problems in the software so that they can be corrected.

39.2.2 Reduce the Likelihood that Defective Programs Will Enter Production.

All software development must comply with the standards and policies established within the organization to identify and eliminate errors before defective programs enter production.

39.2.3 Safeguard the Interests of Users.

The ultimate goal is to achieve software that meets the users' requirements and provides them with needed functionality. To meet these goals, SQA must review and audit the software development process and provide the results of those reviews and audits to management. SQA, as a functioning department, can be successful only if it has the support of management and if it reports to management at the same level as software development. Likewise, each SQA project will possess specific attributes, and the SQA program should be tailored to accommodate project needs. Characteristics to be considered include: mission criticality, schedule and budget, size and complexity of project, and size and competence of project staff organization.

39.2.4 Safeguard the Interests of Software Producers.

By ensuring that software meets requirements, SQA can help prevent legal conflicts that may arise if purchased software fails to meet contractual obligations. When software is developed in-house, SQA can prevent the finger-pointing that otherwise would damage relations between software developers and corporate users.

39.3 SOFTWARE DEVELOPMENT LIFE CYCLE.

Good software, whether developed in-house or bought from an external vendor, needs to be constructed using sound principles. Because software development projects often are large and have many people working on them for long periods of time, the development process needs to be monitored and controlled.

Although progress of such projects is difficult to measure, using a phased approach can control the projects. In a phased approach, a number of clearly identifiable milestones are established between the start and the finish of the project. A common analogy that is used is that of constructing a house, where the foundation is laid initially and each phase of construction is achieved in an orderly and controlled manner. Frequently, with both house construction and software development, payments for phases are dependent on reaching the designated milestones.

When developing systems, we refer to the phased process as the system development life cycle (SDLC). The SDLC is the process of developing information systems through investigation, analysis, design, coding and debugging, testing, implementation, and maintenance. These seven phases are common, although different models and techniques may contain more or fewer phases.

Generally, milestones identified in the SDLC correspond to the points in time when specified documents become available. Frequently the documents explain and reinforce the actions taken during the just-completed phase of the SDLC. We therefore say that traditional models for the phased development are document driven to a large extent.

There are problems and drawbacks inherent in the document-driven process. The method of viewing the development process via the SDLC is not totally realistic to the actual projects. In reality, errors found in earlier phases are noted, and fixes are developed, prototyping is introduced, and solutions are implemented. This, in effect, is more than what is assumed will be necessary in the debugging and maintenance phases. Also, often the problems are solved before those later phases are reached. Because of the recognition that much of what in traditional models is referred to as maintenance is really evolution, other models of the SDLC, known as evolutionary models, have been developed.

In traditional models, the initial development of a system is kept strictly separate from the maintenance phase. The major goal of the SDLC is to deliver a first production version of the software system to the user. It is not unusual for this approach to result in excessive maintenance costs to make the functioning or final system fit the needs of the real user. There are other models available, and it is necessary, for each project, to choose a specific systems development life cycle model. To do this, it is necessary to identify the need for a new system and to follow this by identifying the individual steps and phases, possible interaction of the phases, the necessary deliverables, and all related materials.

There are four major system development techniques:

Traditional systems development life cycle (SDLC)
Waterfall model (a variant of the traditional SDLC)
Rapid application development (RAD)
Joint application development (JAD)

Although these four development techniques frequently are seen as mutually exclusive, in truth they represent solutions that place different emphasis on common elements of systems design. Defined at different times, each methodology's strengths demonstrate the technology, economics, and organizational issues that were current at the time the methodology was first defined.

Software generally is constructed in phases, and tests should be conducted at the end of each phase before development continues in the next phase. Four major phases that are always present in software construction are:

The analysis phase, where software requirements are defined
The design phase, based on the previously described requirements
The construction, programming, or coding phase
The implementation phase, where software is actually installed on production hardware and finally tested before release to production

The programming phase always includes unit testing of individual modules and sys-tem testing of overall functions. Sometimes, in addition to testing during programming and implementation, a fifth phase, solely devoted to functional testing, is established. Review and modification generally follow all phases, where weaknesses and inadequacies are corrected.

39.3.1 Phases of the Traditional Software Development Life Cycle.

The seven phases discussed here comprise the most common traditional SDLC:

Investigation
Analysis
Design
Decoding and debugging
Testing
Implementation
Maintenance

From this traditional model, other models with fewer or more phases have been developed.

39.3.1.1 Investigation.

This phase involves the determination of the need for the system. It involves determining whether a business problem or opportunity exists and conducting a feasibility study to determine the cost effectiveness of the proposed solution.

39.3.1.2 Analysis.

Requirements analysis is the process of analyzing the end users' information needs, the environment of the organization and the current system, and developing functional requirements for a system to meet users' needs. This phase includes recording all of the requirements. The documentation must be referred to continually during the remainder of the system development process.

39.3.1.3 Design.

The architectural design phase lists and describes all of the necessary specifications for the hardware, software, people and data resources, and information products that will satisfy the proposed system's functional requirements. The design can best be described as a blueprint for the system. It is a crucial tool in detecting and eliminating problems and errors before they are built into the system.

39.3.1.4 Coding and Debugging.

This phase involves the actual creation of the system. It is done by information technology professionals, sometimes on staff within a company and sometimes from an external company that specializes in this type of work. The system is coded, and attempts are made to catch and eliminate all coding errors before the system is implemented.

39.3.1.5 Testing.

Once the system is created, testing is absolutely essential. Testing proves the functionality and reliability of the system and acts as another milestone for finding problems and errors before implementation. This phase is instrumental in determining whether the system will meet users' needs.

39.3.1.6 Implementation.

Once the previous five phases have been completed and accepted, with substantiating documentation, the system is implemented and users are permitted to utilize it.

39.3.1.7 Maintenance.

This phase is ongoing and takes place after the system is implemented. Because systems are subject to variances, flaws, and breakdowns, as well as difficulties when integrating with other systems and hardware, maintenance continues for as long as the system is in use.

39.3.2 Classic Waterfall Model.

Although phased approaches to software development appeared in the 1960s, the waterfall model, attributed to Roy Royce, appeared in the 1970s. The waterfall model demands a sequential approach to software development and contains only five phases:

Requirements analysis phase
Design
Implementation
Testing
Maintenance

39.3.2.1 Analysis or Requirements Analysis.

The waterfall model emphasizes analysis as part of the requirements analysis phase. Since software is always part of a larger system, requirements are first established for all system elements, and then some subset of these requirements is allocated to software. Identified requirements, for both the system and the software, are then documented in the requirements specification. A series of validation tests is required to ensure that, as the system is developed, it continues to meet these specifications.

The waterfall model includes validation and verification in each of its five phases. This means that in each phase of the software development process, it is necessary to compare the obtained results against the required results. Testing is done within every phase to answer this question and does not occur strictly in the testing phase that follows the implementation phase.

39.3.2.2 Design.

The design phase is actually a multistep process focusing on data structure, software architecture, procedural detail, and interface characterization. During the design phase, requirements are translated into a representation of the software that then can be assessed for quality before actual coding begins. Once again, documentation plays an important role because the documented design becomes a part of the software configuration. Coding is incorporated into the design phase instead of being a separate phase. During coding, the design is translated into machine-readable form.

39.3.2.3 Implementation.

During the implementation phase, the system is given to the user. Many developers feel that a large problem with this model is that the system is implemented before it actually is ready to be given to the user. However, waterfall model advocates claim that the consistent testing throughout the development process permits the system to be ready by the time this phase is reached. Note that in the waterfall model, the implementation phase precedes the testing phase.

39.3.2.4 Testing.

Although testing has been going on during the entire development process, the testing phase begins after code has been generated and the implementation phase has occurred. This phase focuses on both the logical internals of the software and the functional externals. Ideally, the end users and the process/business owners should be enlisted to assist in this phase, since they usually:

Are familiar with the business requirements
Know how the current software and hardware operates
Know what types of activity are anticipated and planned
Provide the ultimate acceptance approval of the software and hardware

Although not directly required by many of the regulatory reviews, involving the end users and process owners provides the IT area with additional resources, not only from a human resource perspective but also from a business knowledge and regulatory compliance perspective.

39.3.2.5 Maintenance.

The maintenance phase reapplies each of the preceding life cycle steps to existing programs. Maintenance is necessary because of errors that are detected, necessary adaptation to the external environment, and functional and performance enhancements required and requested by the customer.

The waterfall model is considered to be unreliable, partly due to failure to obey the strict sequence of phases advocated by the traditional model. In addition, the waterfall model often fails because it delivers products whose usefulness is limited when requirements have changed during the development process.

39.3.3 Rapid Application Development and Joint Application Design.

Rapid application development (RAD) supports the iteration and flexibility necessary for building robust business process support. RAD emphasizes user involvement and small development teams, prototyping of software, software reuse, and automated tools. Activities must be carried out within a specified time frame known as a time box. This approach differs from other development models where the requirements are fixed first and the time frame is decided later. RAD, in an effort to keep within the time box and its immovable deadline, may sacrifice some functionality.

RAD has four phases within its cycle:

Requirements planning
User design
Construction
Cutover

The main techniques used in RAD are joint requirements planning (JRP) and joint application design (JAD). The word “joint” refers to developers and users working together through the heavy use of workshops. JAD enables the identification, definition, and implementation of information infrastructures. The JAD technique is discussed with RAD because it enhances RAD.

39.3.4 Importance of Integrating Security at Every Phase.

Security should never be something added to software at the end of a project. Security must be considered continuously during an entire project in order to safeguard both the software and the entire system within which it functions. Regardless of which software development model is used, it is essential to integrate security within each phase of development and to include security in the testing at each phase. Rough estimates indicate that the cost of correcting an error rises tenfold with every additional phase. For example, catching an error at the analysis phase might require only several minutes—time for a user to correct the analyst—and therefore cost only a few dollars. Catching the same error once it has been incorporated into the specifications document might cost 10 times more; after implementation, 1,000 times more.

39.4 TYPES OF SOFTWARE ERRORS

39.4.1 Internal Design or Implementation Errors.

A general definition of a software error is a mismatch between a program and its specifications; a more specific definition is “the failure of a program to do what the end user reasonably expects.” There are many types of software errors. Some of the most important include:

Initialization
Logic flow
Calculation
Boundary condition violations
Parameter passing
Race condition
Load condition
Resource exhaustion
Resource, address, or program conflict with the operating system or application(s)
Regulatory compliance considerations
Other errors

39.4.1.1 Initialization.

Initialization errors are insidious and difficult to find. The most insidious programs save initialization information to disk and fail only the first time they are used—that is, before they create the initialization file. The second time a given user activates the program, there are no further initialization errors. Thus the bugs appear only to employees and customers when they activate a fresh copy or installation of the defective program. Other programs with initialization errors may show odd calculations or other flaws the first time they are used or initialized; because they do not store their initialization values, these initialization errors will continue to reappear each time the program is used.

39.4.1.2 Logic Flow.

Modules pass control to each other or to other programs. If execution passes to the wrong module, a logic-flow error has occurred. Examples include calling the wrong function, or branching to a subroutine that lacks a RETURN instruction, so that execution falls through the logical end of a module and begins executing some other code module.

39.4.1.3 Calculation.

When a program misinterprets complicated formulas and loses precision as it calculates, it is likely that a calculation error has occurred; for example, an intermediate value may be stored in an array with 16 bits of precision when it needs 32 bits. This category of errors also includes computational errors due to incorrect algorithms.

39.4.1.4 Boundary Condition Violations.

The term “boundaries” refers to the largest and smallest values with which a program can cope; for example, an array may be dimensioned with 365 values to account for days of the year and then fail in a leap year when the program increments the day-counter to 366 and thereby attempts to store a value in an illegal address. Programs that set variable ranges and memory allocation may work within the boundaries but, if incorrectly designed, may crash at or outside the boundaries. The first use of a program also can be considered a boundary condition.

39.4.1.5 Parameter Passing.

Sometimes there are errors in passing data back and forth among modules. For instance, a call to a function accidentally might pass the wrong variable name so that the function acts on the wrong values. When these parameter-passing errors occur, data may be corrupted and the execution path may be affected because of incorrect results of calculations or comparisons. As a result, the latest changes to the data might be lost or execution might fall into error-handling routines even though the intended data were correct.

39.4.1.6 Race Condition.

When a race occurs between event A and event B, a specific sequence of events is required for correct operation, but this sequence is not ensured by the program. For example, if process A locks resource 1 and waits for resource 2 to be unlocked while process B locks resource 2 and waits for resource 1 to be unlocked, there will be a deadly embrace that freezes the operations.

Race conditions can be expected in multiprocessing systems and interactive systems, but they can be difficult to replicate; for example, the deadly embrace just described might happen only once in 1,000 transactions if the average transaction time is very short. Consequently, race conditions are among the least tested.

39.4.1.7 Load Condition.

All programs and systems have limits to storage capacity, numbers of users, transactions, and throughput. Load errors can occur due to high volume, which includes a great deal of work over a long period of time, or high stress, which includes the maximum load all at one time.

39.4.1.8 Resource Exhaustion.

The program's running out of high-speed memory (RAM), mass storage (disk), central processing unit (CPU) cycles, operating system table entries, semaphores, or other resources can cause failure of the program. For example, inadequate main memory may cause swapping of data to disk, typically causing drastic reductions in throughput.

39.4.1.9 Interapplication Conflicts.

With operating systems (OS) as complex as they are, OS manufacturers routinely distribute the code requirements and certain parameters to the application software manufacturers, so that the likelihood of program conflicts or unexpected stoppages are minimized. Although this certainly helps reduce the number of problems and improves the forward and backward compatibility with previous OS versions, on occasion even the OS vendors experience or cause difficulties when they do not conform to the parameters established for their own programs.

39.4.1.10 Other Sources of Error.

It is not unusual for errors to occur where programs send bad data to devices, ignore error codes coming back, and even try to use devices that are busy or missing. The hardware might well be broken, but the software also is considered to be in error when it does not recover from such hardware conditions.

Additional errors can occur through improper builds of the executable; for example, if an old version of a module is linked to the latest version of the rest of the program, the wrong sign-on screens may pop up, the wrong copyright messages may be displayed, the wrong version numbers may appear, and various other inaccuracies may occur.

39.4.1.11 Regulatory Compliance Considerations.

If an organization is subject to Sarbanes-Oxley (SOX), then documentation of the errors encountered and the resulting internal reporting and remediation efforts is critical. The error should clearly indicate how it was identified, who identified it, how it was reported to management, what the remediation will be, and when it is anticipated to be completed. Without these details, management may encounter significant difficulties in confirming that adequate internal control mechanisms exist and that it is informed and involved appropriately and adequately. Also, an error could result in a control weakness being identified by the external auditors or, even worse, as a material weakness, depending on its nature and severity.

Additionally, if and when errors are noted that affect multilocation systems or applications, the significance and materiality must be considered from the aspect of both the subsidiary and the headquarters locales. Since laws differ among states and countries, what might be considered legally acceptable standards of privacy or accounting in one locale might not be acceptable elsewhere.

39.4.2 User Interface.

Generally speaking, the term “user interface” denotes all aspects of a system that are relevant to a user. It can be broadly described as the user virtual machine (UVM). This would include all screens, the mouse and keyboard, printed outputs, and all other elements with which the user interacts. A major problem arises when system designers cannot put themselves in the user's place and cannot foresee the problems that a technologically challenged user will have with an interface designed by a technologically knowledgeable person.

Documentation is a crucial part of every system. Each phase of development—requirements, analysis, development, coding, testing, errors, error solutions and modifications, implementation, and maintenance—needs to be documented. All documents and their various versions need to be retained for both future reference and auditing purposes. Additionally, it is important to document the correct use of the system and provide adequate instructional and reference materials to the user. Security policies and related enforcement and penalties also need to be documented. Ideally, the documentation should enable any technically qualified person to repair or modify any element, as long as the system remains operational.

39.4.2.1 Functionality.

A program has a functionality error if performance that can reasonably be expected is confusing, awkward, difficult, or impossible. Functionality errors often involve key features or functions that have never been implemented. Additional functionality errors exist when:

Features are not documented.
Required information is missing.
A program fails to acknowledge legitimate input.
There are factual errors or conflicting names for features.
There is information overload.
The material is written to an inappropriate reading level.
The cursor disappears or is in the wrong place.
Screen displays are wrong.
Instructions are obscured.
Identical functions require different operations in different screens.
Improperly formatted input screens exist.
Passwords or other confidential information are not obscured or protected adequately.
Tracing the user data entry or changes is unavailable or incomplete.
Segregation of duties is not enforced. (This can be particularly critical for organizations subject to HIPAA, SOX, ISO 17799, and/or GLBA.)

39.4.2.2 Control (Command) Structure.

Control structure errors can cause serious problems because they can result in:

Users getting lost in a program
Users wasting time because they must deal with confusing commands
Loss of data or the unwanted exposure of data
Work delay
Financial cost
Unanticipated exposure to data leakage or compromise; this can result in significant liability if consumers' personal identifying information (PII) is compromised
Data not being encrypted as intended or being visible to unauthorized users

Some common errors include:

Inability to move between menus
Confusing and repetitive menus
Failure to allow adequate command-line entries
Requiring command-line entries that are neither intuitive nor clearly defined on screen
Failure of the application program to follow the operating system's conventions
Failure to distinguish between source and parameter files, resulting in the wrong values being made available to the user through the interface, or failure to identify the source of the error
Inappropriate use of the keyboard when new programs do not meet the standard of a keyboard that has labeled function keys tied to standard meanings
Missing commands from the code and screens resulting in the user being unable to access information, to utilize programs, or to provide for the system to be backed up and recoverable, as well as a host of other commands that leave the system in a state of less-than-optimum operability
Inadequate privacy or security that can result in confidential information being divulged, the complete change or loss of data without recoverability, poor reporting, and even undesired access by outside parties

39.4.2.3 Performance.

Speed is important in interactive software. If a user feels that the program is working slowly, that can be an immediate problem. Slow operation can depend on (but is not limited to) the OS, the other applications running, memory allocation, memory leakage, and program conflicts. At another level, performance suffers when program designs make it difficult to change their functionality in response to changing requirements. Performance errors include slow response, unannounced case sensitivity, uncontrollable and excessively frequent automatic saves, inability to save, and limited scrolling speed.

39.4.2.4 Output Format.

Output format errors can be frustrating and time consuming. An error is considered to have occurred when the user cannot change fonts, underlining, boldface, and spacing that influence the final look of the output; alternatively, delays or errors when printing or saving document may occur. Errors occur when the user cannot control the content, scaling, and look of tables, figures, and graphs. Additionally, there are output errors that involve expression of the data to an inappropriate level of precision.

39.5 DESIGNING SOFTWARE TEST CASES

39.5.1 Good Tests.

No software program can ever be tested completely, since it would not be cost effective or efficient to test the validity, parameters, syntax, and boundaries of every line of code. Even if all valid inputs are defined and tested, there is no way to test all invalid inputs and all the variations on input timing. It is also difficult, if not impossible, to test every path the program might take, find every design error, and prove programs to be logically correct. Nevertheless, a good test procedure will find most of the problems that would occur, allowing the designers and developers to correct those problems and ensure that the software works properly. Generally, OS and application manufacturers classify the severity level of errors encountered, and many (as evidenced by the frequency and content of program patches) correct only the most serious ones that either they or the users notice.

Over the past several years, there has been an increasing debate on whether software vulnerabilities should be publicized immediately by researchers, or if the researchers should automatically notify and give the manufacturer time to correct the flaw. Generally, from a casual review of the vulnerabilities discovered concerning Microsoft's Internet Explorer version 6 over the past few years, the trend appears to have been that the researcher notifies Microsoft and allows Microsoft a varying period of time to create and distribute the patch (e.g., three to six months) before the researcher publicly announces the weakness. This trend is based on published reports in the technical news media.

Whenever you expect the same results from two tests, you consider the tests equivalent. A group of tests forms an equivalence class if the tester believes that the tests all test the same thing, and if one test catches or does not catch a bug, the others probably will do the same. Classical boundary tests check a program's response to input and output data; equivalence tests, however, teach a way of thinking about analyzing programs that enhances and strengthens test planning.

Finding equivalence classes is a subjective process. Different people analyzing the same program will come up with different lists of equivalence classes because of what the programs appear to achieve. Test cases often are lumped into the same equivalence class when they involve the same input variables, result in similar operations, affect the same output variables, or handle errors in the same manner.

Equivalence classes can be groups of tests dealing with ranges or multiple ranges of numbers, members of a group, and even time determined. Equivalence classes are generally the most extreme values, such as the biggest, smallest, fastest, and slowest. When testing, it is important to test each edge of an equivalence class, on all sides of each edge. Testers should use only one or two test cases from each equivalence class because a program that passes the tests generally will pass any test drawn from that class. Invalid input equivalence classes often allow program bugs to be overlooked during debugging.

39.5.2 Emphasize Boundary Conditions.

Boundaries are crucial for checking each program's response to input and output data. Equivalence class boundaries are the most extreme values of the class; for example, boundary conditions may consist of the biggest and smallest, soonest and latest, shortest and longest, or slowest and fastest members of an equivalence class.

Tests should include values below, at, and above boundary values. Additionally, they should be tested at various user levels (e.g., administrator, root or super user, data entry, and read-only access). This must ensure that there are no conditions at any level that permit unintended access or unauthorized escalation of privileges. When programs fail with nonboundary values, they generally fail at the boundaries too. Programs passing these tests probably also will pass any other test drawn from that class. Tests should also be able to generate the largest and smallest legitimate output values, remembering that input-boundary values may not generate output-boundary values.

Today, the most prevalent security breaches involve buffer overflows in active code (ActiveX and Java), in which inadequate bounds checking on input strings allows overflowing text to be interpreted as code that then can carry out improper operations. Restrictions on input length and type would prevent these exploits. Such overflows can result in the exposure of consumers' PII and incurring unwanted liability for the exposure; currently, the liability exists principally at the state level and may include criminal sanctions, depending on the state in which the consumers reside.

39.5.3 Check All State Transitions.

All interactive programs move from one state to another. A program's state is changed whenever something causes the program to alter output (e.g., to display something different on the screen) or to change the range of available choices (e.g., displaying a new menu of user options).

To test state transitions, the test designer should lay out a transition probability matrix to show all the paths people are likely to follow. State transitions can be very complex and often depend not on the simple choices provided but on the numbers the user enters. Testers often find it useful to construct menu maps that show exactly where to go from each choice available.

Menu maps also can show when and where users go when menu or keyboard commands take them to different states or dialogs. Maps are particularly handy when working with spaghetti code (code that has been poorly designed or badly maintained so that logical relationships are difficult to see); maps allow the designer or user to reach a dialog box in several ways and then proceed from the dialog box to several places. For spaghetti code, the menu maps afford a simpler method of spotting relationships between states than trying to work exclusively from the program itself because the map shows transition between states on paper or on screen. After mapping the relationships, the designer or user can check the program against the map for correctness.

Full testing theoretically requires that all possible paths are tested. However, in practice, complete testing may be unattainable for complex programs. Therefore, it makes sense to test the most frequently used paths first.

39.5.3.1 Test Every Limit.

It is necessary to test every limit on a program's behavior that is specified by any of the program's documents. Limits include the size of files, the number of terminals, the memory size the program can manage, the maximum size it requires, and the number of printers the program can drive. It is important to check on the ability of the program to handle large numbers of open files on an immediate basis and on a long-term, continuing basis as well. Load testing is actually boundary condition testing and should include running tests the program ought to pass and tests the program should not pass.

For Web applications, tools exist to simplify this task, as some will check the boundaries and limits of fields at varying depths of data entry to test the protections against buffer overflow, invalid data, and incorrect data syntax; such tools include but are not limited to WebInspect by Spi Dynamics.

39.5.3.2 Test for Race Conditions.

After testing the system under “normal” load, it should be tested for race conditions. A “race condition” is usually defined as anomalous behavior due to unexpected critical dependence on the relative timing of events. Race conditions generally involve one or more processes accessing a shared resource such as a file or variable, where this multiple access has not been properly controlled. Systems that are vulnerable to races, especially multiuser systems with concurrent access to resources, should undergo a full cycle of testing under load. Race conditions sometimes can be identified through testing under heavy load, light load, fast speed, slow speed, multiprocessors running concurrent programs, enhanced and more numerous input/output devices, frequent interrupts, less memory, slower memory, and related variables.

39.5.4 Use Test-Coverage Monitors.

For complex programs, path testing usually cannot or would not test every possible path throughout a program for practical as well as cost-effectiveness reasons. A more practical glass box approach (i.e., with knowledge of the internal design of a program) is to use the source code listing to force the program down every branch visible in the code. When programmers add special debugging code during development, a unique message will print out or be added to a log file whenever the program reaches a specified point. Typically such messages can be generated by calling a print routine with a unique parameter for each block of code. When source code is compiled, many languages permit a switch to be set to allow conditional compilation, so that a test version of the code contains active debugging statements whereas a production version does not. For interpreted code, such as most fourth-generation languages, similar switches allow for inclusion or activation of debugging instructions.

Since distinct messages are planted, it is possible to know exactly what point in the program the test has reached. Programmers specifically insert these messages at significant parts of the program. Once these messages have been added, anyone can run the program and conclude whether the different parts actually have run and been tested.

Special devices or tools also can be used to add these messages to the code automatically. Once source code is fed to such a coverage monitor, it analyzes the control structures in the code and adds probes for each branch of the program. Adding these probes or lines of code is called instrumenting the program. It is possible to tell that the program has been pushed down every branch when all the probes have been printed. Besides instrumenting code, a good coverage monitor can capture probe outputs, perform analyses, and summarize them. Some coverage monitors also log the time used for each routine, thus supporting performance analysis and optimization.

A coverage monitor is designed for glass box testing, but knowledge of the internals of the program under test is not needed in order to use the monitor. The coverage monitor counts the number of probe messages, reports on the number of probes triggered, and even reports on the thoroughness of the testing. Because it is possible for the coverage monitor to report on untriggered branches, the monitor can find and identify code that does not belong, such as routines included by default but never used or deliberately inserted undocumented code (a Trojan horse). Some commercially available coverage monitors are, unfortunately, themselves full of bugs. It is important to obtain a full list of the known bugs and patches from the developer. In addition, such tools should themselves be tested with programs that contain known errors, in the process known as seeding, to verify their correctness.

39.5.5 Seeding.

Quality assurance procedures themselves benefit from testing. One productive method, known as seeding, is to add known bugs to a program and measure how many of them are discovered through normal testing procedures. The success rate in identifying such known bugs can help estimate the proportion of unknown bugs left in a program. Such seeding is particularly important when establishing automated testing procedures and test-coverage monitors; also, it is useful for SOX or other compliance-related testing.

39.5.6 Building Test Data Sets.

One of the most serious errors in quality assurance is to allow programmers to use production data sets for testing. Production data may include confidential information that should not be accessible to programming staff, so access should be forbidden not only to production data but even to copies of production data, or even to copies of subsets or samples from production data. However, anonymizing processes applied to copies of production data (e.g., scrambling names and addresses of patients in a medical records database) may produce test data sets suitable for use in quality assurance.

Alternatively, if the organization has a test system that is completely (logically and physically) segregated from the production system, using historical data may be appropriate. In doing so, particular care and appropriate documentation must be maintained to ensure that there is no chance of the test system interfacing with the production system. A side benefit of such testing is that it may help the organization's SOX or HIPAA periodic testing and documentation regarding the availability, utility, and integrity of backup media and systems.

39.6 BEFORE GOING INTO PRODUCTION

39.6.1 Regression Testing.

Regression testing is fundamental work done by glass box and black box testers. The term is used in two different ways. The first definition involves those tests where an error is found and fixed and the test that exposed the problem is performed again. The second definition involves finding and fixing an error and then performing a standard series of tests to make certain the changes or fix made did not disturb anything else.

The first type of regression testing serves to test that a fix does what it is intended to do. The second type tests the fix and also tests the overall integrity of the program. It is not unusual for people who mention regression testing to be referring to both definitions, since both involve fixing and then retesting. It is recommended that both types of testing be done whenever errors are fixed. If an organization is subject to SOX or HIPAA, appropriate documentation should be maintained that details the test methodology, the sample selection, the frequency, the results, and the remediation (if any).

39.6.2 Automated Testing.

In some enterprises, every time a bug is fixed, every time a program is modified, and every time a new version is produced, a tester runs a regression test. All of this testing takes a considerable amount of time and consumes both personnel and machine resources. In addition, repetitive work can become mind-numbing, so testers may accidentally omit test cases or overlook erroneous results.

To cut down on the time consumption of personnel and the repetitive nature of the task, it is possible to program the computer to run acceptance and regression tests. This type of automation results in execution of the tests, collection of the results, comparison of the results with known good results, and a report of the results to the tester.

Early test automation consisted essentially of keyboard macros or scripts, with limited capacity for responding to errors; typically, an error would produce invalid results that would be fed into the next step of processing and lead to long chains of meaningless results. Slightly more advanced test harnesses would halt the test process at each error and require human intervention to resume. Both methods were of only modest help to the test team. However, today's test-automation software can include test databases that allow orderly configuration of tests, specific instructions for restarting test sequences after errors, and full documentation of inputs and results. For realistic load testing, some systems can be configured to simulate users on workstations, with scripts that define a range of randomly generated values, specific parameter limits, and even variable response times. Large-scale load testing can connect computers together through networks to simulate thousands of concurrent users and thus speed identification of resource exhaustion or race conditions.

Test automation can be well worth the expenditures required. Automated testing is usually more precise, more complete, faster, and less expensive than the tests done by human personnel. Typically, automated tests can accomplish tenfold or hundredfold increases in the number of tests achievable through manual methods. In addition to freeing personnel to do more rewarding work, such methods greatly reduce overall maintenance costs due to detection of errors before systems reach production. Additionally, for regulatory purposes, the automated testing requires significantly less sampling, documentation, and testing. Appropriate management review and reporting of results still are required, as is documentation of remediation follow-up.

39.6.3 Tracking Bugs from Discovery to Removal.

Finding a bug is not enough. Even eliminating a bug is insufficient. A system also must allow the causes of each bug to be identified, documented, and rectified. For these reasons, it is important to track and document all problems from the time of their discovery to their removal, to be certain that the problems have been resolved and will not affect the system. Too, this provides important historical records in diagnosing and remediating incidents and provides appropriate documentation for audit and regulatory purposes.

Problem-tracking systems must be used to report bugs, track solutions, and write summary reports about them. An organized system is essential to ensure accountability and communication regarding the bugs. Typical reports include: where bugs originate (e.g., which programmers and which teams are responsible for the greatest number of bugs); types of problems encountered (e.g., typographical errors, logic errors, boundary violations); and time to repair. However, problem-tracking systems can raise political issues, such as project accountability, personal monitoring, control issues, and issue remediation regarding the data in the database and who owns them.

Once a tracking system is established, a bug report is entered into the database system and a copy goes to the project manager, who either prioritizes it and passes it along or responds to it personally. Eventually the programmers will be brought into the loop, will fix the problem, and will mark the problem as fixed. The fixed problem then is tested and a status report is issued. If a fix proves not to work, that becomes a new problem that needs to be addressed.

39.7 MANAGING CHANGE.

Many people may be involved in creating and managing systems; whenever changes are made to the system, those changes need to be managed and monitored in an organized fashion to avoid chaos.

39.7.1 Change Request.

The change request is an important document that requests either a fix or some other change to a program or system. Information contained on the change request form generally includes who is requesting the change, the date the request is being made, the program and area affected, the date by which the change is needed, and authorization to go ahead and make the change.

Most companies have paper forms that are retained for monitoring and auditing purposes. As the popularity of digital signatures grows and society continues to move toward a paperless office, it seems logical that some of these change requests eventually will become totally automated.

39.7.2 Tracking System.

The tracking system may be manual, automated, or a combination of methods; for regulatory and control purposes, a system that incorporates automated tracking is best, since it reduces the chance for error, and it can enhance follow-up and remediation. Regardless of how it is kept, the original change form generally is logged into a system, either manually filed or entered into an automated database by some identifying aspect, such as date, type of change, or requesting department.

The system is used to track what happens to the change. When an action is taken, information must be entered into the system to show who worked on the request, what was done, when action was taken, what the result of the action was, who was notified of the actions taken, whether the change was accepted, and related information. This information is crucial in ensuring that issues are managed appropriately, escalated as needed to senior management, and tracked until remediated.

39.7.3 Regression Testing.

When a change or fix has been made, regression testing verifies that the change has indeed been made and that the program now works in the fashion desired, including all other functions. The successful completion of the testing should be documented in writing; this can be through the tracking system or in documentation appended to the tracking system entry.

39.7.4 Documentation.

Documentation is crucial when considering, approving, and implementing changes. If information is retained strictly in an individual's head, what happens when the individual goes on vacation, is out sick, or leaves the enterprise permanently? What if the individual simply does not remember what changes were made, or when; how does an organization know who was involved, what actually was done, and who signed off that the changes were made and accepted?

Undocumented changes can result in:

System crashes
Inconsistent data entry
Inappropriate segregation of duties
Data theft or corruption
Embezzlement
Other serious crimes

Lack of documentation can mean that unauthorized changes were made and likely can result in regulatory and audit violations. Undocumented changes may violate segregation of duties (e.g., a person might be informally approving changes to his or her own account, approving changes carried out by a supervisor, or authorizing changes by a person who is not under the responsibility of the approver).

Lack of documentation is often a violation of corporate policy and is often cited in audits; it often will cause significant difficulties regarding SOX, GLBA, and HIPAA compliance efforts and reviews, since management's assertions might not be supported adequately. Because constructing adequate documentation after the fact is almost impossible (and frequently can be considered to be falsification of records in sectors such as financial services), documentation must proceed in step with every phase of a system's development and operation. Thereafter, documentation must be retained according to company policy and legal requirements.

39.8 SOURCES OF BUGS AND PROBLEMS.

Locating and fixing bugs is an important task, but it is also essential to try to determine where and why the bugs and related problems originated. By finding and studying the source, often it is possible to prevent a recurrence of the same or similar type of problem.

39.8.1 Design Flaws.

Design flaws often occur because of poor communication between users and designers. Users are not always clear about what they want and need, while designers misunderstand, misinterpret, or simply ignore what the users relate to them. Within the design process, even when users' feelings and requirements are known and understood, design flaws can occur. However, they are easier to identify and remedy if the appropriate documentation is effected. Without the proper documentation, it may not be possible to identify the source of an error; this can result in patches or remedies that need to be patched. Often flaws result from attempts to comply with unrealistic delivery schedules. Managers should support their staff in resisting the pressure to rush through any of the design, development, and implementation stages.

39.8.2 Implementation Flaws.

Whenever a program is developed and implemented, there are time limits and deliverable dates. Most problems during development cause delay, so developers and testers often are rushed to get the job done. Sometimes they sacrifice the documentation, the review portion of the project, or even cut short testing, leaving unrecognized design and implementation flaws in place. Managers should emphasize the value of thorough review and testing, and allocate enough time to avoid such blunders.

39.8.3 Unauthorized Changes to Production Code.

If problems are traced to unauthorized changes, project managers should examine their policies. If the policies are clear, perhaps employee training and awareness programs need improvement. Managers also should examine their own behavior to ensure that they are not putting such pressure on their staff that cutting corners is perceived to be acceptable.

39.8.4 Insufficient or Substandard Programming Quality.

A programmer can make or break a program and a project. Programmers play essential roles in software development. Therefore, it is important that all programmers on a project be capable and reliable. Project programmers should be carefully screened before being placed on a software development project to ensure that their skills meet project requirements. Sometimes holes in programmers' skills can be filled with appropriate training and coaching. However, if problems appear to be consistent, management may want to check a programmer's background to verify that he or she did not falsify information when applying for the job. Truly incompetent programmers need to be removed from the project. Additionally, if management suspects that the programming quality is inadequate, then it should consider having an independent programmer review the code, performing particularly rigorous quality assurance (QA) testing.

39.8.5 Data Corruption.

Data corruption can occur because of poor programming, invalid data entry, inadequate locking during concurrent data access and modification, illegal access by one process to another process data stack, and hardware failures. Data corruption can occur even when a program is automatically tested or run without human intervention. In any event, when searching for the sources of bugs and other problems, it is important to do a careful review of the data after each round of testing, in order to identify deviations from the correct end state. The review should be documented appropriately and escalated to management; it then will assist the regulatory compliance efforts.

39.8.6 Hacking.

When bugs and problems do occur, hacking—both internal and external—should be considered a possibility and searched for by reviewing the logs of who worked on the software and when that work was done. Managers should be able to spot the use of legitimate IDs when the individuals involved were on vacation, out sick, or not available to log on for other reasons.

Archiving logs and retrieving them is an increasingly critical capability, aside from the regulatory requirements of SOX, HIPAA, GLBA, and so on. Logs are often used for:

Incident response activities, particularly in determining:
- If an incident occurred
- When an incident occurred
- What happened prior to and subsequent to when an incident is suspected to have occurred
Research; this can be to review performance capabilities or issues of hardware or software or networking

To ensure that the logs are available when needed, the organization needs a sound strategy that preserves and archives logs periodically, so that an attacker or a system anomaly does not wipe them from the system's hard drive. Although there are many means to accomplish this, a frequently used inexpensive method involves establishing a log server. For example, an organization might identify a discarded desktop computer, outfit it with a new, high-speed, high-capacity optical read-only memory (ROM) drive, and increase memory capacity (if needed). By routing the logs to the optical drive frequently and recording them throughout the day (perhaps multiple times during an hour) and changing the optical disk at least daily, the organization will create a forensically sound archive that can be duplicated easily for research, law enforcement, or legal use. Often the daily logs volume will not fill a DVD or a CD. However, having the regular routine of changing the media daily helps ensure that the media does not reach its capacity at an inopportune time—indeed, if a worm such as Blaster or Nimda infects the organization, then the logs likely will swell rapidly. Too, if an attacker within the system stops the logging, the organization should encounter less difficulty determining the date, time (or time period), and extent of the intrusion because ROM cannot be overwritten, in contrast with read-write (RW) media.

Unauthorized changes to code and data sometimes can be identified even if the perpetrator stops the logging and deletes or modifies log files. One method is to create checksums for production or other official versions of software and to protect the checksums against unauthorized access and modification using encryption and digital signatures. Similarly, unauthorized data modifications sometimes can be made more difficult by creating checksums for records and linking the checksums to time and data stamps and to the checksums for authorized programs. Under those conditions, unauthorized personnel and intruders find it difficult to create valid checksums for modified records. Other products, such as intrusion detection systems (IDS) or intrusion prevention systems (IPS), may be useful in an organization's protection and risk management strategy.

39.9 CONCLUSION.

This chapter has presented a comprehensive overview of the software development and quality assurance processes that must be utilized when developing, implementing, and modifying software. Software development involves more than simply selecting and utilizing an approach such as the traditional, waterfall, or RAD methodology. It means working as a team to develop, review, refine, and implement a viable working product. Many good techniques and products can be applied to both the SDLC and the quality assurance portions of producing software; some of the key elements are good documentation, allowing sufficient time for testing in the development and maintenance processes, building good tests, establishing test data, automating testing, and keeping track of change requests.

39.10 FURTHER READING

Campanella, J., ed. Principles of Quality Costs: Implementation and Use, 3rd ed. Milwaukee, WI: ASQC, Quality Press, 1998.

Institute of Electrical and Electronics Engineers. IEEE Standard for Software Quality Assurance Plans. ANSI/IEEE Std. 730-1998. New York: IEEE, 1998.

Institute of Electrical and Electronics Engineers. An American National Standard: IEEE Standard for Software Test Documentation. ANSI/IEEE Std. 829-1998. New York: IEEE 1998.

Fox, C., and P. Zonneveld. IT Control Objectives for Sarbanes-Oxley: The Role of IT in the Design and Implementation of Internal Control Over Financial Reporting, 2nd ed. IT Governance Institute, 2006.

NASA Software Assurance Technology Center: http://satc.gsfc.nasa.gov/.

Horton, W. K. Designing and Writing Online Documentation: Help Files to Hypertext. New York: John Wiley & Sons, 1990.

King, D. Current Practices in Software Development. New York: Yourdon Press, 1984.

Ould, M. A. Strategies for Software Engineering: The Management of Risk and Quality. New York: John Wiley & Sons, 1990.

Schwarz, D. “Some Words about Software Engineering,” in Spectral Envelopes in Sound Analysis and Synthesis, part 8.1, 1998, www.ircam.fr/anasyn/schwarz/da/specenv/8_1Some_Words_about_Softwar.html.

Whitten, N. Managing Software Development Projects, 2nd ed. New York: John Wiley & Sons, 1995.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.

Table of Contents for CHAPTER 39: SOFTWARE DEVELOPMENT AND QUALITY ASSURANCE

Create new playlist

Sign In

Sign Up

CHAPTER 39

SOFTWARE DEVELOPMENT AND QUALITY ASSURANCE

39.1 INTRODUCTION.