We have seen many aspects of systems-of-systems testing, although we have only scratched the surface. Many challenges lie ahead for the future and, as Niels Bohr mentioned, “Predictions are very difficult, especially if they concern the future.” However, let us try to do this perilous exercise and list some challenges below.

13.1. Technical debt

Technical debt is a reality in any development project. It appears because we make choices, sometimes take shortcuts or postpone updates. Technical debt is not just a problem for developers; it exists at all levels, including testing. Here is a list – not exhaustive – of usual technical debts:

– Specifications, requirements or user stories that are ambiguous, contradictory, outdated or missing, or even obsolete and which have not been reviewed (or removed if obsolete) before being provided to the development and test teams. These functionalities will have to be reviewed, corrected or – in the case of obsolete elements – removed and retested resulting in additional costs.

– Code, architecture of components and microservices, modularity and coupling of components, databases (relational or NoSQL), organization of data with or without orchestrators, distribution of components between the frontend, the backend and the cloud, etc. Any modification of the architecture and the organization which underlies the system will require modifications and tests, therefore additional costs.

– Absent test conditions or test cases for requirements considered minor, resulting in a discrepancy between the developed system and the test actions planned to validate the system. There will remain untested parts on this system.

– Lack of traceability from the requirements or user stories to the test cases that validate them, resulting in the design – and execution – of certain duplicate tests and the non-coverage of other functionalities, requirements or user stories.

– Non-automation of certain tests considered minor or relating to internal technical components. This can be intentional (e.g. difficulty in automating) or due to a lack of time or resources. These components will not be able to be tested automatically and the risk of regressions or false negatives is increased.

– Design of fragile tests (e.g. constants and hard data, non-parameterized replay capture, etc.). This is often due to a policy of automating tests – functional or not – that does not take into account the specific needs of automated tests (e.g. association of test cases to components within the framework of DevOps solutions, limitation of tests regression based on impact analyses, etc.). Test automation is to be considered exactly as a development project.

– Anomalies not retested after their corrections, and/or non-automation of these retests for inclusion in smoke test campaigns. Some anomalies are corrected and deployed without being retested (e.g. urgent correction or patch), which may cause side effects. It is important to maintain the level of test automation as close as possible to the functionalities of the developed system.

– Tests no longer working due to interface or code changes. These tests can be disabled during test execution campaigns, or generate incorrect results that are ignored but may hide other faulty tests. Often the absence of ascending traceability towards the requirements does not make it possible to identify those which are no longer validated.

13.1.3. Measuring technical debt

Technical debt is difficult to measure; often it is not even measured. Failure to take this into account will result in an accumulation of debts which will impact the product in the short term. Measuring the technical debt therefore makes it possible to identify the increase in this debt and to determine whether this debt is becoming too high.

Because technical debt can come from a variety of sources, it needs to be measured in multiple places. One way to measure the technical debt is to calculate the spent cost and the – planned – cost of correction and reconcile it with the optimized design cost to reach a Technical Debt Ratio (TDR) according to the formula:

A high ratio reflects a poor level of quality. This measurement should be done for the requirements (user stories, specifications, architecture, etc.), for the code and for the tests.

We can also identify the presence of technical debt by focusing on the following elements:

– increase in design and development costs;

– decline in quality and increase in the number of anomalies;

– increase in Time-to-Value, and increased delays between deliveries;

– reduced team velocity;

– too much cyclomatic complexity;

– etc.

Some tools pride themselves on measuring the technical debt of software (e.g. SQUORE, SonarQube, Kiuwan, etc.), but we also need to assess whether the technical debt is in the organization’s processes.

13.2. Systems-of-systems specific challenges

Systems-of-systems are subject to particular challenges including:

– long lifespan of the systems and the software composing them, involving:

 - the obligation to keep test environments and equipment in configurations that could become obsolete, in order to keep test environments representative of production,

 - the need to have maintenance and test teams that are kept for many years, so as not to lose the functional and technical knowledge related to these applications,

 - the need to have up-to-date and exhaustive documentation, in order to allow a transfer of knowledge to the personnel who will be involved in future maintenance;

– obligation to manage multiple software configurations (including OS compatibility, interactions between applications, flow definitions); these aspects of configuration management – both software and hardware (for EIS) – involve managing a very large number of versions, especially if continuous delivery methods are in place;

– evolution of requirements and risks, for each of the components and elements, each of the applications, each subsystem and each system;

– absence of a single referent with knowledge of the entire system-of-systems, specialization of knowledge on each of the systems without taking into account possible interactions;

– loss of consciousness by the departure of individuals without updating the documentation;

– obsolescence of components (HW/SW);

– management of interfaces and their evolutions;

– evolution of standards;

– ability to perform end-to-end tests (End-to-End or E2E);

– separate evolution of components (including hardware, operating systems, other software), replaceability of components; adding additional layers (display on tablets, evolution of multi-tiers systems).

13.3. Correct project management

For many years, the Standish Group has regularly published the “Chaos Report”, and the number of failed or seriously overdue or overbudget projects remains globally stable (close to 70%, of which 19% failed). The improvements identified over the years relate mainly to small projects – less than 10,000 function points – which can be properly mastered by Agile teams. We have confirmation – in Jones (2018) – that large projects (10,000 function points and more) such as systems-of-systems have a failure rate greater than 30%, which can go up to 47.5% for systems with 100,000 function points and more.

The need for good project management is obvious, but the organizational and technical complexity of systems-of-systems imposes a correct mastery of all aspects of project management, the use of appropriate project management tools, metrics and reliable periodic measurements to have an impartial view of the progress and quality of the project and of each of its components, and to be able to anticipate hazards. This implies that best practices in project management be put in place, such as those proposed by PMBOK (from the Project Management Institute) or PSP/TSP proposed by Carnegie Mellon University and Watts S. Humphrey (1998) and Humphrey and Thomas (2010) and to implement effective project management tools to take into account the management of critical paths and critical chains.

Some Agile advocates will consider that projects should be self-managing, but that does not seem appropriate for systems-of-systems projects given the interplay between a sequential view for higher levels and an agile and reactive view for lower levels. A SAFe type solution could be suitable by integrating periodic deliveries (Release Trains) comprising Agile sprints in a sequential environment. However, the requirements for hardware artifacts make implementing Agile solutions quite problematic.

The specifics related to testing software, systems with preponderant software and systems-of-systems are important and not covered in most project management frameworks, nor taught in most IT project managers’ courses. To limit the number of failures of large-scale projects, aspects related to testing (reviews, inspections, static analyses, coverage of quality characteristics, etc.) must be mastered, and this requires the adaptation of current initial training as well as specific continuing education.

13.4. DevOps

DevOps is a rising trend in the industry. As mentioned previously, it is a Lean version of agility whose objective is to streamline exchanges between development and operations via rapid and constant feedback to development teams (e.g. via the automation of tests) and automation of deployment activities (design of test environments and deployments).

The purpose of Lean is to limit loss of value. Among the losses, we have:

– loss of time, waiting times before an activity. Therefore, the grouping of several evolutions before their tests becomes a factor of loss of value; the sooner developers are made aware of their anomalies, the sooner they will be able to fix them and understand the source of these anomalies. This involves both build automation and test automation aspects;

– the accumulation of potential losses – technical debt – such as anomalies, even minor ones, not corrected and postponed to a later delivery;

– material losses, such as the need to rewrite tests or code.

One of the losses in value is the time lost between the creation of defects – during development – and the detection of these defects during testing. Early detection and correction of defects provides information to developers before costs rise too high.

In a systems-of-systems framework, the main DevOps challenges include:

– the quality and completeness of the documentation, whether we are talking about the description of requirements, functionalities, Epics, Features or User Stories, architecture or development documentation, or even the description of tests; this information – as well as the data allowing traceability from the requirements to the execution of the tests – is essential to maintain the consistency of the systems;

– execution of regression tests at each test level and for each iteration, taking into account the needs for automation of these tests, maintenance of previous tests (deletion or modification);

– the aspects of versioning and configuration management, in order to control the various hardware, parametric or software configurations of systems-of-systems at customers;

– the management of test data for each of the environments, and the potential coexistence of different versions of the same data;

– the need for multiple environments to run the tests for each test level, the maintenance of these environments and their upgrading;

– the difficulty of obtaining environments for the levels of system integration (SIT – System Integration Testing) and systems test of the system-of-systems insofar as these environments must be as representative as possible and their management – including aspects of configuration management, evolution management and anomaly management – are complex and costly.

13.5. IoT (Internet of Things)

The IoT represents an interesting challenge for systems-of-systems testing. Sometimes called “web 3.0”, the IoT covers telemetry and communications between connected objects for various uses, including in critical areas such as health and safety. We may also include areas such as home automation (any intrusion into our privacy may have an impact on our security and on the confidentiality of our personal data), production (e.g. refineries, power plants, etc.) or the transport of energy, without this list being exhaustive. For example, sensors placed on certain parts of our vehicle send information to a central unit to notify us in real time of its status, and also to notify the manufacturer or our mechanic in the event of a breakdown or failure. All these aspects of telemetry are grouped under the term IoT.

We clearly identify that objects will be developed by different organizations and that communications will use different standards or different versions of the same standards. The information exchanged from these objects will vary between object and connector types. As the objects are connected via radio connections (Wi-Fi, Bluetooth, ZigBee, Z-Wave, NFC, etc.), users do not always realize the information exchanged – sometimes without their knowledge – nor the impacts that these objects can have on their daily life (especially in case of dysfunction).

The identification of potential failures and how defects can be used for unauthorized purposes must be identified and analyzed as exhaustively as possible, and mitigation or even securing activities must be considered.

The combination of functional paths – valid and exceptional – is almost unlimited here, and the data exchange media are multiple and varied (RFID, QR Codes, ShotCodes, SMS, GPS-type virtual tags, etc.) when the sensors can provide medical data (blood sugar measurement for diabetics), geographical location or interact with social networks or not (e.g. sending targeted advertising messages, monitoring the quality of service in a store, etc.). This involves integrating components with varying levels of security into networks, which can weaken or compromise the level of security of network elements. Similarly, the use of voice detectors and analyzers (e.g. in vehicles or at home) implies that conversations can be listened to, analyzed and therefore data can be extracted and used well or badly.

As part of the end-to-end testing needed to validate systems-of-systems, one of the challenges will be to generate enough consistent test data to ensure that trends within a valid range of test data are detected (e.g. to detect leaks or wear), to identify any data outside of valid ranges, as well as to identify loss of connection with the sensors which could mean sensor failures or communication. It is indeed necessary to differentiate the display failure from the absence of data to display.

The rapid evolution of techniques, the appearance of new objects, new functionalities, and new exchanges of – confidential – data means that the identification of new attacks will be more difficult. Similarly, the use of central servers or the ability of manufacturers to impose – without the knowledge of system users – automatic updates including usage limitations – or changes to contractual conditions – tends to generalize and can be one of the vectors of a global attack on the systems-of-systems to which the IoT objects are connected, or even an open door to regressions or unwanted side effects.

13.6. Big Data

The concept of Big Data is often associated with the use of predictive analytics, behavioral analytics or other analytical methods that extract value from very large volumes of data. The use of the Internet, surveillance cameras (with facial recognition) and the storage of very large volumes of information on user behavior, their interactions in social networks, their browsing profiles, their consumption patterns, their purchases (including medicines), all this information can be used for analysis purposes by algorithms which will make it possible to obtain trends.

These algorithms should be checked and validated before being used. However, the only way to validate these algorithms, to ensure that they do not provide erroneous information, is to test them. To do this, it will be necessary to design datasets that represent behaviors that meet the profiles sought, and also behaviors that do not meet, or not completely, the profiles sought, in order to determine from which moment the algorithm is defaulted.

The solution of using mathematical analysis, statistics and other concepts to infer laws (e.g. regression, nonlinear relationships and causality of effects) is a way to reveal relationships and dependencies, even to predict results and behaviors. It seems risky to make such predictions if the sample size is not statistically representative, and – even though it is representative – such predictions are not certainties and come with no guarantees.

It is well realized that the level of complexity of the various algorithms will require testers with knowledge of statistics, human behaviors, etc. which must be at least equal to those of the designers of these algorithms. The creation of the test data could influence the results obtained.

13.7. Services and microservices

More and more systems are developed including microservices or APIs that allow communication between systems with data transfer. These exchanges use interfaces based on protocols, such as HTTP or HTTPS, to allow access to data servers, web servers or mobile applications. When a system uses data or content from multiple sources, it is sometimes referred to as a Mashup.

The challenges identified include, among others, the following aspects:

– evolutions of services, microservices or APIs, from the identification of these evolutions and the updating of systems, to the management of user regressions and security breaches;

– use of services in security-critical applications when the transmission medium (the web and TCP/IP) does not guarantee that all the data will be transmitted (asynchronous transmissions vs synchronous transmissions);

– increase in dependencies on services, functionalities or protocols developed by third parties for which the level of reliability is not adequately guaranteed, or even is not known (e.g. LOG4J flaw), which leads to fragility of the system-of-systems;

– confidentiality of transferred data, slowdown and security vulnerabilities due to coding format conversions.

The activities of refactoring or modifying the architectures of the systems-of-systems can lead to evolutions towards mergers or separation of components. This will have an impact on testing, among others in the context of DevOps (CI/CD) and the validation of functional and/or technical impacts (see also Ford et al. 2021).

13.8. Containers, Docker, Kubernetes, etc.

These solutions seem to allow for simple fixed configurations, but require significant software over-/under-layers that can slow down execution or increase dependencies, configuration management overhead, and development, maintenance and testing complexity. If these solutions make it possible to put code and unit tests in the same component, they will not cancel the need for complete functional tests of subsystems and systems.

Indeed, even though each element is stable and reliable, the evolution of its use or its environment in a system-of-systems, can lead to the appearance of risks or unforeseen behaviors during the initial design. We could take as an example the “Dirty Pipe” flaw (CVE-2022-0847) in Linux which allows a malicious process to modify the privileges of other containers using a shared image.

It is therefore, here too, necessary to ensure that the documentation of the components is up to date, whether it is the documentation of the interfaces, the functionalities, the code and the tests, and that everything is correctly traced from and towards requirements.

13.9. Artificial intelligence and machine learning (AI/ML)

Artificial intelligence has nothing to do with human intelligence but more with a machine repeating what it has learned (usually by repetition). Once the system has learned correctly (via the machine learning function), it must be able to use this learning to determine choices, make decisions, etc.

We understand that two aspects are fundamental: on the one hand, the quality of the data that will be used to teach the system (the sample and its size) and, on the other hand, the quality of the algorithm and its capacity to focus on the important elements of the samples by disregarding unnecessary elements. Let us compare with how a child learns a concept. What defines an automobile? Do we need wheels and if so how many? Do we need a roof, doors, steering system, etc.? Must the wheels be on the ground (is an overturned automobile still an automobile?)? We see that many parameters come into play. A machine should compare these two images below and determine that they are both automobiles.

Let us take an example to illustrate the difficulty linked to the validation of the algorithm: a tank detection system was shown (ML aspect) many images of allied or potentially enemy tanks. Image-based tests validated that the system correctly recognized “friends” from “enemies”. The system was therefore installed and tested on a shooting range. He considered ALL tanks identified as tanks to be “enemy”. Why? In fact, the images of “friendly” tanks were taken in different conditions than those of “enemy” tanks (closer to those of the terrain), and therefore, the system considered that the tanks on the ground were always enemy tanks. I do not know if this example is real or not, but it shows the difficulty in determining whether or not what the algorithm has selected is correct. In the context of automated driverless driving systems, these systems may be confronted with changes in environments that become challenges that are too complex to manage (see article from CNN¹ and YouTube²).

Questions for testers to ask themselves will include:

– How can we validate the proposed sample?

– Can we identify examples that invalidate the sample, or can we demonstrate a failure in the algorithm?

– Are the testers’ knowledge in statistics sufficient? How do we make sure that what we read is correct?

It is very likely that the validation of algorithms requires advanced knowledge in algorithms, statistics and/or mathematics, therefore collaboration with other specialists (see also Smith (2022)).

13.10. Multi-platforms, mobility and availability

In addition to the development of connected objects, we also have a multitude of smartphones and computers that can connect to applications – and therefore to systems and systems-of-systems – either directly (via a wired connection) or indirectly via radio connections, wired, etc. This forces applications to manage aspects of multiple remote connections (e.g. the same account is connected from a PC and a smartphone), and also connections that may change during the transaction (e.g. connection to different relays for a transaction carried out on a moving smartphone).

Applications and systems will have to take into account evolutions to be designed in parallel on various operating systems (e.g. Windows + Linux + Android + iOS + …) so that the same application, the same system-of-system, is accessible from these various platforms. If the user interfaces are carried out via a browser, it will then be necessary to ensure that the application or the system-of-systems generates the same user experience regardless of the platform and the browser, and also regardless of the geographical location and the network reception level. If the application is of the heavy client type, optional application modules – depending on the performance level of the network used – could be considered.

We realize that the test activities will be more complex as it will be necessary to combine functional aspects with elements of geolocation, variation in the quality of networks (including 3G, 4G, 5G, etc.), different browsers, responsive displays, etc.

We find ourselves faced with the complexity of systems-of-systems with software publishers and uncoordinated developments that can lead to side effects.

Regarding smartphones, they could be emulated and the tests of the applications they include could be carried out based on these emulators. However, emulators do not always behave identically to the system they emulate, so it will be necessary to also – in addition – perform tests on the physical systems themselves. Similarly, the writing modes – from left to right, from right to left, without forgetting the characters linked to Sanskrit, Chinese, Korean and Japanese among others – as well as their coding (via unicode) can generate behaviors specific to be tested. The translation of the “€” code seems to be “$” in some cases, which can lead to differences of values in invoices.

The availability of networks and features may also be limited depending on whether the user is roaming and depending on the country. Some countries limit the use of applications, or monitor the information exchanged. It is therefore recommended to act with caution regarding the use of connected systems and systems-of-systems which can become victims of attacks via uncontrolled or poorly controlled network connections.

13.11. Complexity

The number of conditions to be taken into account in systems-of-systems will increase more and more, and their complexity will grow more and more. The difficulty that our testers will have to face will relate to the creation of test environments that will be increasingly complex.

Separating systems into subsystems helps limit complexity, and also hides interactions between subsystems that may not have been anticipated. For example, positioning an antenna on an aircraft may be in the way of a door opening; the operation of one piece of equipment can interfere with another (one of the causes of the loss of HMS Sheffield during the Falklands conflict).

Complex systems are inherently more hazardous and inherently more dangerous. The frequency of the hazard can be modified, but the presence of these hazards characterizes such complex systems (and by extension systems-of-systems). In complex systems and systems-of-systems, the combination of minor failures leads to catastrophic failures. It is extremely rare that a single failure is sufficient to cause the entire system to fail. In the same way, complex systems, given their complexity, will never be tested exhaustively and will operate with latent failures, or in degraded mode. However, their multiple redundancies will allow nominal operation despite the presence of numerous failures.

More and more the systems, given their complexity, tend to be optimized. This optimization makes them more fragile and less robust. Increasing redundancy increases the robustness of these systems and therefore their resilience to failures.

13.12. Unknown dependencies

The LOG4J component is present in an Apache component included in the Apache logging services (Apache Logging Services). On December 9, 2021 a zero-day vulnerability was published by Alibaba Cloud Security Team regarding this LOG4J component, and it has since been widely publicized and used by many hackers. This vulnerability clearly shows we have less and less control over all the software we use, and we are dependent on an increasing number of other software that can open doors to malicious activities. The LOG4J vulnerability potentially affects millions of servers on which tens of millions of applications and users depend.

In the context of systems-of-systems, these vulnerabilities in relation to other software will increase, and may concern many systems. It is difficult to ensure the security testing of the servers and applications on which our systems depend, and it is complicated to listen to all the new vulnerabilities discovered every day. Ensuring that these new vulnerabilities are not present in our systems-of-systems and – in case they are present – ensuring that all systems are properly updated is a long-term job.

This is made doubly difficult because vulnerability patches must simultaneously fix the discovered flaw, and also avoid opening other ones. In the case of LOG4J, the corrections led to the discovery of other vulnerabilities (CVE-2021-44228, CVE-2021-45046 and CVE-2021-45105) on the published patches.

13.13. Automation of tests

Test automation is often perceived as the only solution to quickly perform many tests to validate all the combinations of values and actions to be tested. However, test automation brings many questions too frequently underestimated or evaded:

– What does test automation really bring?

– What return on investment can be expected from test automation?

– How to correctly implement end-to-end tests, often on heterogeneous environments, with multiple technologies?

– What maintenance costs keep automated tests up to date and what costs reduce technical debt?

– When the tests will be automated, how to ensure that the trust we place in them is well placed (how to avoid false negatives)?

– Are the scripts and data easily portable to another tool when the current tool becomes obsolete?

– Knowing that it is impossible to be exhaustive in the design and execution of tests, how can we ensure the quality of the deliverables provided?

In the following, we propose some lines of thought in response to these questions.

13.14. Security

Testing security in systems-of-systems and in software in general is difficult to implement because we will only seek to cover vulnerabilities that have already been found (e.g. among competitors, based on recognized repositories, etc.). Security vulnerabilities not known to date will not be tested, except by malicious hackers. A technical watch to identify “zero day” flaws and remedy them will be useful for each of the co-contractors of the system-of-systems.

It may also be necessary to think about security in terms of intellectual (or industrial) property security, national security and the defense of national interests. These are elements very often present in systems-of-systems. It is necessary in such cases to turn to our security officer.

13.15. Blindness or cognitive dissonance

Blindness happens when our beliefs collide with the reality on the ground. Technical or procedural beliefs are like religious beliefs and lead to the same reactions of denial or self-confirmation. It is unfortunate that, in a field that claims to be scientific, the use of statistical and tangible evidence is not more frequent and more widespread. We can visualize this aspect of beliefs in the use of methods or tools that claim to be applicable to all projects whatever they are (size, domain, criticality, etc.) when nothing proves it. We therefore come to refuse reality, to cling to certain non-representative cases where the belief applies, or to seek excuses to refute arguments contrary to our convictions, as explained by Syed (2015).

Blindness in software development and testing is generally characterized by persistent optimism and a refusal to see reality, to measure oneself against a known benchmark. In the case of systems-of-systems and software testing, this translates into various ways:

– Plan test campaigns without considering the needs for correcting detected anomalies and re-executing test scenarios that have suffered anomalies. This results in the execution of tests until the day before delivery, therefore in the potential identification of defects until this date. There will be no time left to test defect fixes and the risk of introducing anomalies into production will be increased.

– Not ensuring that the test execution prerequisites (datasets, environments and flows, requirements repositories, data expected for comparison, etc.) are present: this implies that the test activities will be an uninterrupted sequence jerks and stops. This will have the result that tests will start and then be forced to stop in the absence of one or more prerequisites, which will impact the delays.

– Thinking that the development teams can design code without defects, and therefore that it is impossible to predict the number of anomalies that the code could (should) contain. This common blindness can be removed quickly during testing by using metrics to identify and display trends.

– This same blindness also applies to testers: testers can also make mistakes, whether in test design, data selection or test execution and reporting. The implementation of traceability makes it possible to limit some of the oversights.

– Think that multitasking is a solution that allows us to focus on one task when another is blocked. This way of doing forgets that the individual needs to concentrate to be effective. Any change of task leads to a more or less long period during which the individual must change context and focus and is therefore no longer as effective.

– Thinking that the corrections will be made without generating new defects and therefore will not need to be retested. Each of us knows that it is when we are rushed and stressed, when we take shortcuts, when we make mistakes (an application of Murphy’s law). However, developers are often in a hurry to deliver bug fixes. This results in an increased risk of errors, and therefore of the introduction of other defects. This amounts to thinking that the reduced delays, the absence of detailed analyzes and the complexity of the defects to be corrected, will have no impact on the quality (more precisely the lack of quality) of the software corrections.

A potential solution to this aspect of blindness and cognitive dissonance can be to question yourself at every moment. That is, not to rely on opinions but to rely on statistically representative data, rather than trying to use this data to confirm a belief or an opinion.

13.16. Four truths

Among the elements that should not be lost sight of in the testing of complex systems and systems-of-systems, we also note four recurring truths:

– people are more important than material;

– quality is more important than quantity;

– good tests – and good testers – cannot be mass-produced;

– competent test teams cannot be created after the emergency arises.

How should we understand these truths?

13.17. Need to anticipate

More and more immediate answers are sought instead of anticipating problems upstream. New methodologies, such as Agility, take this to the extreme by offering sprint durations of two to three weeks. This implies that the majority of individuals will work with a two- or three-week horizon and will not anticipate much longer term.

As in the construction of buildings, it is necessary to design infrastructures and have an overall plan if we do not wish to have a system-of-systems unable to adapt or develop in the future. An optimal architecture, a well-organized system does not evolve organically and we do not have time to consider several parallel developments of systems to then select the best (see Darwinism).

Let us take an example: if we want to invest in an automation solution – an important job, sometimes profitable and always long – it is necessary to define the short-, medium- and long-term objectives that we want to achieve with this automation. This may be to save time when executing test campaigns, but will imply an additional workload for each sprint: it will be necessary to automate the tests that are designed currently, and also to automate all the old tests that have been designed in previous sprints. We realize that a horizon of two or three weeks will be too short. Similarly, timeframes for purchasing, sourcing, training and setting up automation tools and processes need to be considered. Here too, a three-week horizon may be unrealistic.

The purpose of anticipation is to make it possible to propose minor but early adjustments to avoid major adjustments later. It will therefore be necessary to have a long-term vision, both of the objectives and of the risks/constraints. This anticipation is often the prerogative of experienced individuals.

13.18. Always reinvent yourself

The evolutions of methodologies, the increase in interactions between the various systems and components as well as between individuals and companies, mean that the “good practices” identified here and in the works cited in reference will have to be adapted to meet specific needs of the project: a solution that worked in one environment may not be suitable for another environment.

This implies two things:

– First, intimately know all the techniques and methods, their advantages and their weaknesses, their prerequisites and their deliverables, etc. This involves continuing to learn, both in one’s main area (testing and quality), and also in all related areas (development, team management, hardware and software environments, software packages, methods, etc.) in order to control their impact on the proposed solution.

– Second, continually question yourself, trying to identify how the solutions implemented can be improved to achieve the intended objective. A one-time gain to achieve a particular goal can lead us to implement a sub-optimal solution but which is the best, given the constraints of the moment.

We will need to constantly measure failures and analyze their root causes, as independently as possible, to learn from them. We could take inspiration from aeronautics and the words of Chesley Sullenberger³: “Everything we know in aviation, every rule and every procedure we have, we know because someone died.” This solution allows us to improve ourselves, improve the profession and improve the products we test.

13.19. Last but not least

The contractual aspects and the contractual follow-up by both the customer and the supplier are important and will always be a serious challenge in the case of systems-of-systems.

We only need to look at major information system projects and their myriad of problems to realize that the solutions are not simple, despite the involvement of very large ESNs or leading manufacturers:

– Integration of CRM with complex legacy systems, in an international context, with multiple co-contractors where unit tests are not carried out – or not carried out correctly – by the offshore development teams, the integration focusing on fast releases and on the only correction of critical and blocking anomalies (by trying to push the others to a following iteration) which generates an increasingly important technical debt.

– Integration of a CRM in an insurance group, with different priorities and operating modes according to each insurance group, with pressure from decision-making bodies that push for rapid delivery and development and test teams that are not sized to deal with the complexity of interfaces, interactions, different management rules depending on the members of the group.

– Integration of an ERP in a complex system including Legacy, production, distribution, 3D modeling systems via the Internet and automatic tracking of order progress, as well as the entire logistics, supply and manufacturing. A known integrator, but no awareness of the impact or the complexity of the tests to be considered to ensure correct operation.

– The integration of several payroll systems for the military and civil servants, with the use of subcontracted personnel in countries where labor is cheaper, but without considering that these countries are not specially allies, etc. This initially resulted in payroll issues for the military (e.g. if the word “OPEX” – for external operation – is present then pay reduced to zero), then in the buyout of the master ESN work by its competitor (for a symbolic euro?).

– Design of military ships integrating systems and subsystems from national and international sources, covered by non-proliferation and/or export control treaties, where the co-contractors must ensure that they do not lose their advantages (properties intellectual property, trade secrets, etc.), and where the involvement of various government policies is important.

The common point of these projects seems to be the disconnect between the evaluations of the designers and the people in charge of the contractual aspect of the project. The blissful optimism of each other, the obstinacy in not wanting to consider the problems and the tendency to pass them on to another actor combine to generate impressive additional costs and fiascos.