Appendix D Software Service History Questions
Chapter 24 provided the four categories of questions to consider when evaluating software service history. This appendix includes the specific questions from the Federal Aviation Administration’s Software Service History Handbook [1]. The handbook is available on the Federal Aviation Administration website (www.faa.gov); however, these questions are provided for convenience.
D.1 Questions Regarding Problem Reporting [1]
- Are the software versions tracked during the service history duration?
- Are problem reports tracked with respect to particular versions of software?
- Are problem reports associated with the solutions/patches and an analysis of change impact?
- Is revision/change history maintained for different versions of the software?
- Have change impact analyses been performed for changes?
- Were in-service problems reported?
- Were all reported problems recorded?
- Were these problem reports stored in a repository from which they can be retrieved?
- Were in-service problems thoroughly analyzed and/or those analyses included or appropriately referenced in the problem reports?
- Are problems within the problem report repository classified?
- If the same type of problem was reported multiple times, were there multiple entries or a single entry for a specific problem?
- If problems were found in the lab in executing copies of operational versions of software during the service history period, were these problems included in the problem reporting system?
- Is each problem report tracked with the status of whether it is fixed or open?
- If the problem was fixed, is there a record of how the problem was fixed (in requirements, design, code)?
- Is there a record of a new version of software with new release after the problem was fixed?
- Are there problems with no corresponding record of change in software version?
- Does the change history show that the software is currently stable and mature?
- Does the product have the property of exhibiting the error with a message to the user? (Some products may not have error trapping facilities, so they may just continue executing with wrong results and with no indication of failure.)
- Has the vendor (or the problem report collecting agency) made it clear to all users that problems are being collected and corrected?
- Are all problems within the problem report repository classified?
- Are safety-related problems identified as such? Can safety-related problems be retrieved?
- Is there a record of which safety problems are fixed and which problems are left open?
- Is there enough data after the last fix of safety-related problems to assess that the problem has been corrected and no new safety-related problems have surfaced?
- Do open problem reports have any safety impact?
- Is there enough data after the last fix of safety-related problems to assess that the problem is solved and no new safety-related problems have surfaced?
- Are the problem reports and their solutions classified to indicate how a fix was implemented?
- Is it possible to trace particular patches with release versions and infer from design and code fixes that the new versions correspond to these fixes?
- Is it possible to separate the problem reports that were fixed in the hardware or change of requirements?
- Are problem reports associated with the solutions/patches and an analysis of change?
- If the solutions indicated a change in the hardware or mode of usage or requirements, is there an analysis of whether these changes invalidate the service history data before that change?
- Is there a fix to a problem with changes to software but with no record of the change in the software version?
- Is the service period defined appropriate to the nature of software in question?
- How many copies of the software are in use and being tracked for problems?
- How many of these applications can be considered to be similar in operation and environment?
- Are the input/output domains the same between the service duration and the proposed usage?
- If the input/output domains are different, can they be amended using glue code?
- Does the service period include normal and abnormal operating conditions?
- Is there a record of the total number of service calls received during the period?
- Were warnings and service interruptions a part of this problemreporting system?
- Were warnings analyzed to assure that they were or were not problems?
- Was there a procedure used to log the problem reports as errors?
- What was the reasoning behind the contents of the procedure?
- Is there evidence that this procedure was enforced and used consistently throughout the service history period?
- Does the history of warranty claims made on the product match with the kind of problems seen in the service history?
- Have problem reports identified as a nonsafety problem in the original domain been reviewed to determine if they are safety related in the target domain?
D.2 Questions Regarding Operation [1]
- Is the intended software operation similar to the usage during the service history (i.e., its interface with the external world, people, and procedures)?
- Have the differences between service history usage and proposed usage been analyzed?
- Are there differences in the operating modes in the new usage?
- Are only some of the functions of the proposed application used in service usage?
- Is there a gap analysis of functions that are needed in the proposed application but have not been used in the service duration?
- Is the definition of normal operation and normal operation time appropriate to the product?
- Does the service period include normal and abnormal operating conditions?
- Is there a technology difference in the usage of the product from service history duration (manual vs automatic, user intercept of errors, used within a network vs standalone, etc.)?
- Was operator training on procedures required in the use of the product during the recorded service history time period?
- Is there a plan to provide similar training in the new operation?
- Will the software level for the new system be the same as it was in the old system?
D.3 Questions Regarding Environment [1]
- Are the hardware environment of service history and the target environment similar?
- Have the resource differences between the two computers been analyzed (time, memory, accuracy, precision, communication services, built-in tests, fault tolerance, channels and ports, queuing modes, priorities, error recovery actions, etc.)?
- Are safety requirements encountered by the product the same in both environments?
- Are exceptions encountered by the product the same in both environments?
- Is the data needed to analyze similarity of environment available? (Such data are not usually a part of problem data.)
- Does the analysis show which portions of the service history data are applicable to the proposed use?
- How much service history credit can be assigned to the product, as opposed to the fault-tolerant properties of the computer environment in the service history duration?
- Is the product compatible with the target computer without making modifications to the product software?
- If the hardware environments are different, have the differences been analyzed?
- Were there hardware modifications during the service history period?
- If there were hardware modifications, is it still appropriate to consider the service history duration before the modifications?
- Are software requirements and design data needed to analyze whether the configuration control of any hardware changes noted in the service history are acceptable?
D.4 Questions Regarding Time [1]
- What is the definition of service period?
- Is the service period defined appropriate to the nature of software in question?
- What is the definition of normal operation time?
- Does normal operation time used in service period include normal and abnormal operating conditions?
- Can contiguous operation time be derived from service history data?
- Is “applicable-service” portion recognized from the total service history data availability?
- What was the criterion for evaluating the service period duration?
- How many copies of the software are in use and being tracked for problems?
- What is the duration of applicable service?
- Is applicable-service definition appropriate?
- Is this the duration used for calculation of error rates?
- How reliable was the means of measuring time?
- How consistent was the means of measuring time throughout the service history duration?
- Do you have a proposed accepted error rate justifiable and appropriate for the level of safety of proposed usage, before analyzing the service history data?
- How do you propose that this error rate be calculated, before analyzing the service history data?
- Is the error rate computation (total errors divided by time duration, by number of execution cycles, by number of events such as landing, by flight hours, by flight distance, or by total population operating time) appropriate to the application in question? What was the total duration of time used for this computation? Has care been taken to consider only the appropriate durations?
- What is the actual error rate computed after analyzing the service history data?
- Is this error rate greater than the proposed acceptable error rate defined in PSAC [Plan for Software Aspects of Certification]?*
- If the error rate is greater, was analysis conducted to reassess the error rates?
References
1. U.D. Ferrell and T.K. Ferrell, Software Service History Handbook, DOT/FAA/ AR-01/116 (Washington, DC: Office of Aviation Research, January 2002).
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.