More and more areas are subject to standards and regulations. The standards that we will need to implement and/or use will focus on our field (e.g. medical, aeronautics, rail, nuclear, space, etc.), on the activities that we are required to perform (e.g. development, testing, etc.), the technologies and methodologies we implement (e.g. code design cycles such as Scrum or SAFe) as well as particular areas of systems-of-systems. Within the framework of systems-of-systems and preponderant software systems, we have among others:

– ISO/IEC/IEEE 12207:2017 Systems and software engineering – Software life-cycle processes;

– ISO/IEC/IEEE 15288:2008 Systems and software engineering – System life-cycle processes;

– ISO/IEC/IEEE 21839:2019 Information technology – Systems and software engineering – System-of-Systems (SoS) considerations in life-cycle stages of a system;

– ISO/IEC/IEEE 21840:2019 Systems and software engineering – Guidelines for the utilization of ISO/IEC/IEEE 15288 in the context of system-of-systems (SoS);

– ISO/IEC/IEEE 24748-1:2018 Systems and software engineering – Life-cycle management – Part 1: Guidelines for life-cycle management;

– ISO/IEC/IEEE 24748-2:2018 Systems and software engineering – Life-cycle management – Part 2: Guidelines for the application of ISO/IEC/IEEE 15288 (System life-cycle processes);

– ISO/IEC/IEEE 24748-3:2020 Systems and software engineering – Life-cycle management – Part 3: Guidelines for the application of ISO/IEC/IEEE 12207 (software life-cycle processes);

– ISO/IEC/IEEE 24748-4:2016 Systems and software engineering – Life-cycle management – Part 4: Systems engineering planning.

The first four are important in the context of systems-of-systems and describe, among other things, the four major process areas:

– agreement processes, including acquisition and provision;

– project processes with planning, evaluation and control, management and decisions, risk management, configuration management, information management and measurement processes;

– technical processes, including definition of stakeholder requirements, requirements analysis, architecture design, implementation, integration, verification, transition, validation, operations, maintenance and disposal;

– project organization processes, including model life-cycle management, infrastructure management, project portfolio management, human resources management and quality management.

Interpreting concepts for systems-of-systems purposes provides a standardized understanding.

Other standards are specific depending on the business area and can also be applied in the context of testing and qualification of software applications:

– space industry: the main standards are those of ESA (ECSS set) and NASA;

– aeronautics industry: RTCA standards DO178C/ED12C, EN9100, etc.;

– railway industry: BSI EN 50128:2011 standard on signaling, telecommunications and processing systems in railway applications;

– ISO/IEC/IEEE/DIS 21839 Information technology – Systems and software engineering – SoS considerations in life-cycle stages of a system;

– automotive industry: MISRA standards (Motor Industry Software Reliability Association);

– banking industry: Sarbanes Oxley, Bale III etc.;

– pharmaceutical and medical industry: FDA standards;

– all industries: standards relating to software testing (ISO/IEC/IEEE 29119 series), etc.

11.1. Definition of standards

The standards are mainly defined by international standardization bodies such as the IEEE (Institute of Electrical and Electronic Engineers) or the ISO (International Standard Organization); organizations representing the branches (e.g. ECSS for space) or national organizations (e.g. AFNOR in France, ANSI in the United States, BSI in the United Kingdom, DIN in Germany). Standards define a minimum level of quality or process in the production of hardware or software products.

The development time for a standard is relatively long (of the order of a few years) and its application is often of the order of 10 years. Minor or major changes to these standards are often spaced about 10 years apart. Some standards replace one or more standards, which they make obsolete (e.g. ISO/IEEE29119 which replaces IEEE 829, ISO 25010 replacing ISO 9126, etc.). It is everyone’s responsibility to ensure that the standards applicable in their field are respected and that proof of compliance is made available.

11.2. Usefulness and interest

In general, using a standard provides a reference framework defining the minimum actions to be implemented. We must pay attention to the reasons behind the implementation of standards in companies:

– Regulatory obligation: this is the case for companies in a field where regulations are applicable. In such a field of application, periodic process certifications are mandatory. These periodic certifications are carried out by specialized certification bodies, based on the description of the processes and the documentation produced by these processes.

– Marketing wish: this is the case for companies seeking to obtain ISO 9001, CMMI- or TMMI-type certification. The interest is to show to the outside that the company has a defined level of maturity to reassure customers or prospects about the company’s ability to carry out the project. It is necessary to pay attention here to the difference between an external evaluation (where the seriousness of the independent certification body is the basis of credibility to be granted) and a self-evaluation (evaluation carried out by the company itself).

– Desire for improvement: this is the best reason to implement a standard within a company. The choice comes from a reflection carried out internally by the company, with the aim of improving the quality of its products and processes.

Whatever the reason for implementing standards, it is important to understand the content of these standards and their interweaving in the families of standards. Then, it will be necessary to make sure to follow their evolutions.

11.3. Implementation

The implementation of one – or more – standard within a company begins with a management decision, sometimes following an audit, an evaluation of the benefits of such an implementation. It is necessary to realize that standards can sometimes be contradictory or very specific. We can indeed implement the standards partially, but this increases the risk of incorrect implementation.

The implementation of standards is often associated with the implementation of development quality management processes such as CMMI, ITIL, PRINCE2 or PMP. The organizations involved in the design of systems-of-systems often formalize their design processes (definitions of procedures, instructions, etc.) and refer to standards. This way of doing things makes it possible to frame the actions of test managers and testers in the implementation of demonstrated quality processes.

11.4. Demonstration of compliance – IADT

In industries subject to regulations or standards, it is necessary to demonstrate the compliance of the implementation of the applicable standards, in addition to providing proof of compliance with functional and technical requirements. Conformance must be developed throughout the system-of-systems design cycle, according to the acronym IADT:

– I for Inspection, where reviews determine whether compliance is achieved or not. These activities are mainly carried out in the downward phase of the V, the design phase;

– A for Analysis where the analysis activities (e.g. architecture, functionalities, etc.) make it possible to determine whether or not compliance has been achieved. These activities are also carried out in the descending phase of the V, the design phase;

– D for Demonstration, which roughly corresponds to the activities of software testing, use of prototypes and demonstration in the factory of the operation of the product or system. These activities are mainly carried out in the rising phase of the V, the integration and system test phases;

– T for Test, where we understand the test of the system-of-systems – and the software that is part of it – during a phase corresponding to the acceptance of the system-of-systems.

The results of each of these compliance research activities are recorded in reports which demonstrate – at each progress milestone in the design and construction of the system-of-systems – that the contractual and regulatory requirements have been correctly met (or not). It will therefore be necessary to have traceability from these requirements to the proofs (reports) of compliance.

This compliance can be carried out by audits – often external but sometimes internal – or by collecting evidence from suppliers and/or co-contractors.

Specialized bodies mandated for this purpose (e.g. FAA – Federal Aviation Authority, Directorate General of Civil Aviation, DoD, etc.) can provide certificates, approvals and other relevant approval documents, after execution of adequate audits and compliance checks.

11.5. Pseudo-standards and good practices

Several other references are useful in the development of systems-of-systems, including major software components. Some have shown their usefulness over the past decades, and are good practices, such as:

– IFPUG 4.3 and SNAP for measuring function points and therefore anticipating the size of systems, the number of anomalies to anticipate depending on the type of system developed;

– PMBOK from PMI (Project Management Institute), for the definition of good practices in project management. We could also cite CMMI (Capability Maturity Model integrated from the Carnegie Mellon Institute) although often limited to suppliers to the Department of Defense in the USA;

– PSP and TSP from the Carnegie Mellon Institute, for the definition of good software development practices.

Others include recommendations published in books (the majority are referenced in the bibliography of this book), standards or conferences without having statistically proven their real effectiveness or efficiency.

It is important to analyze the “good practices” that we could glean from conferences, on the Internet or in various books – including this one – in order to ensure that these “good practices” are applicable to our context. The efficiency and effectiveness described may very well not be achievable in our context or environment.

11.6. Adapting standards to needs

The standards are there to help us, not to impose a straitjacket on us. This assumes that our use of standards is to help us achieve a quality objective within our organization and that the use – or implementation – of standards is not mandatory. If the use of certain standards is mandatory (e.g. in regulated areas), the application of the recommendations can be felt as a straitjacket.

In general, each standard has mandatory or normative sections – which it is mandatory to apply – and others which are optional or informative. The level of compliance we want to achieve against a standard will define the sections we need to implement.

Even if we do not expect compliance with a standard, we can use it to achieve a level of compliance that will make it easier for us to comply with that standard or other standards in the future. Similarly, we can perform more advanced tasks or activities than what is required by the standard, in order to meet internal requirements of our organization.

Note that we cannot claim compliance with a standard if we do not strictly follow the mandatory aspects of that standard.

11.7. Standards and procedures

Our processes and ways of working are dictated – directly or indirectly – by the procedures and processes that apply to us. Any modification of a standard or a process must therefore be identified, even though we are not at the origin of this modification. We will therefore have to actively and continuously monitor the standards – and procedures – that apply to us, and adapt our processes, tasks and actions according to these developments.

When a standard becomes obsolete, it does not mean that we should no longer follow it. On the contrary, we must, for any project where this standard – in this version – was applicable, continue to apply this version. If the contract documents for the project do not mention a specific version of a standard, we should be able to upgrade to its new version. If the obsolete standard is replaced by another standard (e.g. replacement of the ISO9126 standard by ISO25010), our choice to respect the new standard implies that we will have to reorganize our entire project on these new bases, which may be costly.

11.8. Internal and external coherence of standards

The IEEE 21840-2019 standard describes the key concepts and differences between systems and systems-of-systems, among others, about managerial and operational independence. Many standards do not precisely define the scope – functional, organizational or technical – of their application. This therefore results in the possibility of different interpretations from one individual to another, from one organization to another. In the context of a system-of-systems, the recommended interpretation will vary depending on the tree – and dependencies – of the topmost system.

When we want to use standards, we will need to ensure that the terminology, life cycles, methods, etc. are correctly aligned. If this is not the case, we will have to explain what interpretation to give to the terms used and which ones have a preponderant place in our processes. This may be the subject of a separate glossary referenced by each document.

It should be noted that the terminology used in the standards is not consistent and may vary. So we must be circumspect and attentive to the impacts that a change in the standard can cause.