Test automation is often perceived as “the” solution to facilitate and accelerate the execution of application tests, including for systems-of-systems. This vision must be modulated according to the objectives of each of the stakeholders on the system-of-systems.

We will define a test tool as a product to help the execution of one or more test activities. So, we will talk about tools that will help in test design, static test tools and dynamic test tools.

Like any testing activity, automation must demonstrate added value to justify its implementation. This value can be seen in the short term or in the longer term.

10.1. Objectives of test automation

Test automation is often seen only as a way to run more dynamic tests in a shorter amount of time, or “automatically” find more defects, or even reduce the cost of testing. It is illusory and unrealistic.

Having good goals is critical if we hope to achieve those goals. It is surprising how many test automation investment decisions are made without clearly defined goals or with vague justifications. It is important to make a distinction between our goals for testing (e.g. find as many defects as possible, reduce risk) and the goals we can achieve with dynamic test automation.

Realistic goals for test automation include:

– ensure the early detection of defects, from the drafting of requirements and codes, for example, by implementing static code analysis tools and TDD- or ATDD-type techniques;

– provide evidence of the coverage of the requirements by one or more tests according to their level of criticality, so as to provide confirmation of the correct implementation of the requirements;

– increase the coverage of environment configurations (e.g. browsers, OS, etc.) by automatically rerunning the same tests on different systems. This objective is similar to requirement coverage;

– reduce the duration of test campaigns (e.g. system tests and acceptance tests) by allowing concurrent and simultaneous execution of automated test campaigns;

– reduce test design times, by testing based on keywords (or action words) by designing tests in parallel with code development;

– automatically generate data and/or test environments and allow the execution of tests in each of these environments;

– etc.

Other frequent objectives are a reduction in the costs and delays of the project as a whole, including configuration or version management and the management of anomalies, even the follow-up of improvements, the taking of measures and the generation of reports or dashboards.

10.2. Test tool challenges

The use of testing tools allows us to automate a number of repetitive tasks, to provide coverage information – structural and functional – on the software under test. Some activities can reap many benefits from automation, while others cannot. The choice of activities to automate is made by measuring the investments to be made and the expected benefits.

The tooling and automation of tests must be considered on the whole project and not only on one part to the exclusion of the others.

10.3. What to automate?

The choice of activities to automate is the responsibility of the test manager. Since automation always requires a significant effort (therefore an investment), it is necessary to ensure that this effort will bring short-term value, and that this value will last throughout the life of the application.

If the choice of value is a financial objective, it will be necessary to measure the expenses and the point of equilibrium and the level of ROI that can be obtained.

If the reason for automating is to reduce the duration of test campaigns, or to ensure a defined level of quality (e.g. zero defects or six sigma), these objectives will have to be related to the costs and efforts necessary to reach them. In all cases, it will be necessary to take into account:

– scope – code or functionalities – to automate and how to measure it;

– objectives – including the test techniques to be implemented by the automated tests – and their applicability;

– maintainability and readability of the tests, knowing that these tests will have to be executed and maintained for several years.

It will also be necessary to measure if the elements considered are reached:

– choice of activities to automate (easily automatable activities, difficult to automate activities, traceability, measurements, statistics);

– measurement of investment, profitability.

Whatever choices we make, we will have to worry about false negatives, those anomalies that are not detected by the tests – automated or not – but which can appear in production. The use of a tooled solution always leads to confidence – sometimes misplaced – in this solution.

10.4. Test tooling

Whatever the development activities, tools can help with reviews and testing. Questions to address will focus on the benefits these tools can bring and the constraints they will impose. In the framework of systems-of-systems, these questions are to be dealt with in the long term and not only in the short term.

We recommend starting with an identification of the testing processes currently in place and the expected benefits. Ideally, we need at least the following tools:

– A requirement management tool, allowing them to be tracked from the contract to the delivery and maintenance of systems-of-systems in production. An open source tool is a possible short-term solution, but it seems more realistic to select a tool supported by a large customer base.

– A configuration management tool to manage software components as well as hardware components, here also a market tool, supported by a large customer base. This will make it easier to find qualified and experienced personnel.

Tooling chaotic processes is not useful and only serves to automate the chaos. We will therefore need to ensure beforehand that our processes are correctly defined and implemented, with means (metrics and periodic measurements) to ensure that they are correctly implemented. We are therefore talking here about a quality assurance policy (not to be confused with a testing policy).

Next, we need to identify the process or processes that can benefit from automation. This involves mastering our test processes, identifying the main phases where defects are introduced (often in the definition of requirements, in the architecture and the code) in order to determine the types of tools that will be most suitable, discovering these flaws. One thing to consider is that it’s best to find defects as early as possible and that means using those testing tools (if they exist) as early as possible in the design and development cycle. Another element is the expected time savings during the test execution phases (e.g. by parallelizing test activities or by executing them at night on several environments).

Another point to consider is the survivability of tools and artifacts created with these tools. When we decide to buy and implement a tool, the cost of the tool is an element to consider, but represents only a fraction – relatively small – of the total cost of ownership (TCO): we will also have to take into account the workload to design the test scripts, the test data associated with them, to maintain the tools and to run them, without forgetting to manage them in configuration, etc. Some tools have evolved over time, even if their publishers have been taken over by competitors. Other tools, mainly open source tools, have been abandoned (e.g. Open STA) or heavily modified during changes in the development teams (e.g. evolution from Selenium to Selenium 3), which makes all of them obsolete, or parts of the investments made. In addition to the survival of the tools and their editors, it is also necessary to take into account the artifacts (test scripts and test data) produced and used by these tools. If their organization is flawed, if their implementation is not documented, if these automated tests are too closely linked with user interfaces and have a strong coupling, they will quickly become obsolete and the investment made will have been wasted.

10.4.1. Selecting tools

The selection of tools must be carried out within the framework of the complete verification and validation of the system under development, so as to simultaneously avoid the introduction of faults, detect faults early and provide useful information to the project management. As the actors using the tools are not only limited to testers but also include developers, managers, as well as customers/product owners and the hierarchy, it is necessary to have a general vision in order to avoid an unnecessary workload or redundant activities.

Following an evaluation of the test processes to define the processes that will benefit the most from automation, the use of a technique such as GQM (Goal– Question–Metric) will make it possible to refine the objectives, to pose the good questions and to define the metrics that will have to be measured, as well as to set objectives (targets) to be achieved.

A first study – for example, via the publication of an RFP (Request for Proposal) – makes it possible to define a subset of products that will meet all or part of the requirements and needs. It is important to ensure that the tools work in our environments (they still work on the pre-sales consultant workstation). The points of attention here are:

– correctly identify the short- and long-term objectives that must be achieved with the use of tools;

– ensure that the test tools are compatible with other development tools and frameworks that are currently used by our teams (or that will be used in the future);

– check that the tools envisaged work on all the environments, materials and browsers for which the developments are carried out;

– qualify editors and suppliers to ensure their sustainability. Indeed, we will have to make sure that we have support for the entire life of the system. The choice of an open source solution may in this case be unwise.

The second phase, comparing the tools within the framework of our choice, aims to compare the tools pre-selected in the first phase in order to identify the one that will best meet the needs. It will therefore be necessary to have defined the metrics that we will be looking at (planned during the phase of defining the objectives to be achieved). During a tool comparison phase, the tools will be applied to a project representative of the company’s needs. At the end of this comparison phase, the choice of tool(s) is made. During this phase, we will need to focus on:

– selecting one or more tools that will correctly cover the objectives defined in the previous phase;

– selecting a pilot project that is representative of the problems that the tools will have to deal with. Choosing a pilot project that is too simple will not allow us to be sure of the tool’s ability to handle all cases;

– defining KPIs allowing impartial comparisons of tools.

The third phase consists of validating, on a pilot project (therefore not crucial for the organization), the operation of the tools selected in the previous phase; this involves, among other things:

– the installation and implementation of tools on the project environment;

– the modification – if necessary – of the processes and procedures for creating data and test cases, the applicable naming rules, the organization of directories and campaigns, etc.;

– the definition of metrics and means of measurement to ensure that the benefits (and costs) of the tools are measured;

– the use of tools, the backup of deliverables and of results.

At the end of this pilot project, it is necessary to provide feedback and ensure that the expected benefits with the tools are greater than the costs of these tools. From this moment, it is possible to begin to gradually generalize the use of the selected tools, by offering appropriate training and ensuring that the participants of the pilot project can act as coaches for the new teams.

10.4.2. Computing the return on investment (ROI)

Following the product comparison phase and at the end of the pilot project, the calculation of the ROI and the break-even point will be carried out. Things to consider will include:

– the initial cost of purchasing and installing the tool (or tools);

– staff training costs, whether staff assigned to the pilot project, or the design of specific training for the organization (post-pilot projects);

– the costs of tool management, annual license, user management and tool maintenance;

– the costs related to the design of the components supported by the tool (campaigns, sprints, anomalies, scripts, etc.), their configuration management, their maintenance;

– costs and execution times of test campaigns (per iteration, sprint and functionality);

– time and cost savings related to automated execution versus manual test execution.

10.4.3. Avoiding abandonment of tools and automation

A common concern is the abandonment of automation activities on a project. These abandonments have various causes, but result in a significant financial and technical loss for the organizations. Let us look at some of these causes:

– A cessation of investment in the project given the achievement of delivery milestones, for example, delivery to the customer or the cessation – temporary or permanent – of project financing. In this case, the people assigned to the test automation project are reassigned to other projects, which results in a loss of knowledge about the tools and the automation.

– A change in organization or priorities due to failure to achieve the initial automation objectives. This is mainly due to a poor identification of the constraints related to the introduction of tools or to objectives that are not (or not easily) achievable.

– A modification of the interfaces of the product to be tested without modification or adaptation of the automated test scripts. Test automation involves thinking about the maintenance of these tests, and therefore planning a test automation load when modifying the application to be tested.

– Failure to consider developments in test automation tools. Indeed, tools evolve, change and may no longer be supported by their development teams. This is especially true of “open source” tools that depend on teams of volunteers. Some tools like OpenSTA are no longer supported, and others like Selenium have evolved significantly between two versions.

When an automation effort is stopped, a resumption of this effort often requires a reassessment of the objectives, a new definition of the strategies, and sometimes the selection of new tools.

10.6.1. Mastering test automation

Test automation is only of interest if it is effective and efficient (profitable) and properly mastered. If the testing processes – manual – are not mastered, if the tests are chaotic, the use of one – or several – tools will only automate this chaos.

Test automation activities require additional actions compared to manual testing:

– in the context of sequential developments, the tests are based on the same requirements as the developments that they will verify and validate. They will therefore have to follow the manual activities of analysis, design and implementation of tests;

– in the context of Agile developments, the use of automated test tools is necessary – even mandatory – via the TDD, the ATDD and the BDD and this in addition to the analysis, design and implementation activities of test cases – nominal and exception – covering user stories;

– in the context of systems-of-systems with several levels of integration, it will be necessary to combine Agile and sequential test methods, identify which tests to execute at which level – to avoid redoing tests already carried out elsewhere – and then automate this which is the most useful.

In any case, it will be necessary to understand the origins of the defects – their root causes – and the ability of the tests to detect these defects. This will enable development process improvement (through root cause analysis and implementation of continuous improvements) and testing improvement (through the creation of targeted automated tests to identify defects to identify regression potential future).

Potential gains from implementing automated testing include:

– reduced automated test execution load because testers can be assigned to other tasks while automated tests are running;

– reducing automated test execution times because automated tests can run outside of normal tester working hours;

– reduction of the need for hardware and test environments to execute automated tests because these tests – if carried out after working hours – can use the workstations of the testers, or virtualized environments;

– control of the maintenance of automated tests which can be more expensive than for manual tests.

Designing automated tests brings several challenges, including:

– the different organizational models according to automators and tools, even according to individuals and their understanding of test automation (e.g. API tests, keyword-driven tests (KDT));

– the need to have several tools, each identifying different types of anomalies, different targets – applications, languages, etc. – and of course to have confidence in the quality of the tools. Testing tools are also prone to software flaws;

– the need to interface the tools with each other to link automated tests of one type with tests of another type or another environment. This may require monitoring tools or schedulers;

– the confidence that can be had in the test results of the lower levels;

– the maintainability of tests designed by others;

– monitoring of reporting and anomalies, etc.

One of the major challenges with test automation is the dropout rate of automation efforts after projects are initially set up. It is common for the release (delivery) of a software system to lead to a reduction in the size of the development and test teams, and therefore of the people in charge of test automation; However, it is at this moment that the benefits of automation can be the most important if this automation covers all the functional and technical areas of the system: it will indeed make it possible to verify the absence of regressions during future evolutions and corrections.

10.6.2. Preparing test automation

Developing automation requires seeing when and where automated testing can be introduced into the development cycle. This therefore implies close collaboration with the development teams.

The implementation of TDD implies that developers understand how to correctly test, in addition to the proper functioning of the application (or the feature or the user story), the non-passing cases, the cases at the limits of the partitions of equivalences, etc.

Similarly, the developed code must be maintainable, to limit future maintenance costs. Automated code analysis tools make it possible to control the quality of the code, from the moment the code is introduced into the repository.

In the case of automated functional tests, it is necessary – as in the case of manual tests – to ensure that the three phases are present: SETUP, RUN and TEARDOWN.

10.6.3. Defect injection/fault seeding

Defect injection is the introduction into the software of known quality level, and which is analyzed by an automated testing process, of known defects that the testing process is supposed to identify and filter.

It might seem incongruous to talk about the injection of defects in a work where the objective is to find the most defects as quickly as possible. However, since the implementation of automated tests aims to automatically find defects of various types, by limiting manual efforts, it is necessary to ensure that these automated mechanisms correctly identify defects, so that we can really have confidence in test machines. Otherwise, we will get a false sense of security.

The principle is simple: we have to set up a process – totally or partially automated – to find faults in the code. How can we be sure of the effectiveness of this process? By introducing faults that we control and ensuring that these faults are detected. If some of these defects are not identified, it shows failures in the process, therefore a need for improvement of this process.

This technique can be compared to vaccination in human beings. It makes it possible to validate that the body’s defense processes – here the verification/validation processes of the software being designed – work correctly and are capable of detecting pathologies (types of defects) previously identified.

10.7.1. Static test tools versus dynamic test tools

We have identified that static analysis and review activities make it possible to identify defects even before compiling the code. These static activities are effective and very efficient. The automation of dynamic tests often requires that the target of these tests (the developed and integrated applications) be of a sufficient level of quality and therefore these activities cannot really start in parallel with the development activities, unless they include the design – by the testers – of a model of the application to be tested. When the application to be tested is available, the fields of the model are mapped and replaced by the fields of the application, thus allowing the tests to be run on the target.

As far as static test support tools are concerned, we have:

– tools to manage workflows within the framework of reviews, in order to correctly manage the defects or questions identified during the reviews (and their corrections or answers), as well as their merging into a single document, then the management of the signature circuit;

– static analysis tools to identify the quality and complexity of documents or code (e.g. measurement of cyclomatic complexity, identification of non-compliance with coding rules, etc.).

As far as dynamic testing tools are concerned, we have:

– test execution tools, often including functionality for capturing user actions and programming; these tools may or may not be very sophisticated and apply to user interfaces, or focus on APIs;

– monitoring tools that can identify which instructions (or parts of the program) the dynamic tests have already covered, so as to allow the design of test cases specially targeted towards the parts not yet covered;

– tools focusing on the analysis of non-functional characteristics (security, memory leaks, performance, etc.) to provide information on these characteristics and sub-characteristics.

The groupings of tools proposed above are not exhaustive.

10.7.2. Data-driven testing (DDT)

Data-driven testing – or tests directed by data – is a method where we will record, on one side, a series of functional actions and, on the other side, the input data and the expected results of these functional actions on the input data. This makes it possible to cover a combination of important input data and therefore to implement the techniques of equivalence partitions and limit value analysis, without requiring too much workload. Input values and expected results can be stored in an Excel table or CSV file. The automation tool is used to record the functional activities, and then the automation script is modified to use the input variables from the file. At the end of the execution of the test script, the results obtained are compared with the expected results and the results (pass/fail + anomaly reference) are stored. Subsequent input values are read, and the test is rerun with the new values.

This approach makes it possible to cover a large volume of data combinations, with a reduced investment. It will be necessary to design scripts and data files to cover each functional area. This solution allows a simple and fast reuse of scripts for a reduced investment.

10.7.3. Keyword-driven testing (KDT)

Keyword-driven testing, sometimes called action word-driven testing, was proposed by Hans Buwalda in Buwalda et al. (2002). It is an extension of the concept of DDT in that each functional aspect (each recorded and variabilized test script) is referenced separately and can be called by another script. We can thus define a test script that calls other test scripts and executes the data that is passed as a parameter. This removes the tight coupling between automated tests and the interfaces they use to test the system.

Test files include only instructions (keywords), input data and expected results, as well as actions to perform if the result is “pass” or “fail”. This allows functional testers to define sequences of actions (instruction, input data, expected data) in an Excel or CSV file, without having to consider the scripting language of the test tool used. The decoupling obtained through the KDT makes it possible to keep business scenarios stable and to only must change the lower layers of automated tests: the scripts. The level of business coverage therefore remains guaranteed.

A small KDT execution engine therefore only has to read an instruction line from the Excel or CSV file and execute it, that is, identify the script associated with the instruction, replace the variables by the input data, run the script, compare the obtained data with the expected data and note the results.

A major advantage of this solution, in addition to saving time and efficiency, is that it makes the tests independent of the automation tools used. As the functional aspects evolve less quickly than the test tools, this solution allows scripts to be reused with another test tool. Some testing tools and frameworks are starting to offer KDT.

10.7.4. Model-based testing (MBT)

Testers design their tests based on requirements, specifications or their understanding of how the product under test should perform. It is a modeling of the behavior of the system – functional or non-functional – according to their vision and their understanding of the requirements. MBT formalizes this design method by using formal modeling languages to design the model from which the test cases will be defined.

MBT is the application of model-based design to testing. Test cases derived from models generally describe functional tests, but can also include other types of tests (e.g. security, performance, etc.). These tests derived from the analysis of models are mainly abstract tests often focusing on the nominal behaviors of the element to be tested. The testers will have, on the basis of these abstract test cases – test conditions – to associate with the element to be tested the real test cases which will include input data, references to real objects and actions on the developed components.

The representation of a model is often useful to allow testers to interface with users and business representatives. Such models are often also used by development and architecture teams to identify the objects to be developed, the classes, attributes and methods to be designed. The testers could be tempted to reuse the models developed by the designers and would then apply themselves to verifying that the designers have correctly translated the model. This is not what is requested, because there is no guarantee that the model initially developed correctly represents the needs of the users. Since models are often partial and simplified representations of the developed product, nominal behaviors will be identified while exceptional cases and error handling will frequently be forgotten.

Model-based testing obliges testers to learn other languages such as UML, B or Z, and to share this knowledge with users and customers to obtain from them the validation of the model from which will be deduced – manually or automatically – the test cases.

10.8.1. Regression tests in builds

An important moment for a realization team is the saving of the changes and the launch of the build which checks if the changes are correctly implemented. This is therefore a privileged moment to run regression tests, automated or not.

As the developers and designers act in a particular environment (the development environment) and as interactions can take place between the developments carried out by each of the actors, the development environment is not the most suitable for carrying out tests of regression. It will therefore be necessary, during builds, to allow the execution of regression tests in a more representative environment, without impacting the environments of the functional testers in charge of the integration test levels or the system test levels.

The moment of the build – especially if we want to do many builds daily – can be well chosen to perform other actions than regression tests: static analyses of the code could identify potential defects early.

10.8.2. Regression tests when environments change

The software execution environment includes both the other programs with which the software under test interfaces and the operating systems, databases, etc., necessary for the proper functioning of the software within the system-of-systems. If one or more of these elements is modified, the introduction of side effects – regressions – is possible. It is therefore necessary to ensure, through regression testing, that such regressions are identified and corrected. The same consideration applies when moving from one environment to another (e.g. moving from Windows 7 to Windows 10, moving from a 16-bit system to a 64-bit system, etc.).

During the first Ariane 5 flight (flight 501), the programs from the Ariane 4 system were reused. This was later justified by almost a hundred successful flights. Ariane 4 systems ran on 16-bit, and Ariane 5 systems ran on 64-bit. During calculations, a 16-bit digital zone recognized as “high value” was interpreted as a valid 64-bit value, leading to incorrect positioning of the thrusters and the loss of the machine. It is likely that running automated regression tests (during this environment change) could have identified this issue.

10.8.3. Prevalidation regression tests, sanity checks and smoke tests

At the start of the test campaign, it is important to ensure that the application to be tested is of adequate quality and will allow sufficient testing effort without having to stop too quickly. If the application under test does not run correctly, back and forth between test teams and development teams will occur, which will delay test execution. One way to ensure a sufficient level of quality is to carry out “smoke test”-type tests. These tests originated in electronics where, once the components were soldered onto the board, if plugging the board into power generated smoke (one or more components were electrically damaged), the board – in our case the application to test – was not of good quality.

Smoke tests should cover, in a light way, all areas of the application to be tested, all its modules and all its functionalities. An analysis of the results of these tests makes it possible to identify the classes, modules or functionalities which statistically have a higher percentage of failures. This will make it possible, on the one hand, to focus the tests (application of the principle of aggregation of defects) and, on the other hand, to check the general behavior of the application. Such a subset of tests can be automated since the execution of these tests will have to be performed many times.

“Health checks” or “sanity checks” are similar in principle to “smoke tests”, but are often limited to a subset of functional test cases to ensure the proper functioning of the main functionalities without which the app is irrelevant. Their automation is therefore justified.

10.8.4. What to automate?

One of the very frequently asked questions concerns the target of the automation and what it should be about. Like everything – and as Alice’s Cheshire Cat said – it depends on what we want to achieve. Among the different objectives, we have the following:

– identify the most failures;

– test faster;

– cover more combinations of values;

– be more efficient and less dependent on human actors.

However, it is necessary to remember that automation has associated costs and burdens, and that certain activities – requiring intuition and human intelligence – are difficult to automate or cannot be automated. For example:

– analyze the requirements or the user stories to find the combinations of actions that must be tested as well as the values to be selected. This involves analytical and intuitive skills as well as the ability to select the most appropriate test techniques;

– analyze the results obtained and ensure that they correspond to the expected results, even – optionally – create test oracles. This requires the ability to combine test techniques and select the combinations most likely to find failures;

– program test scripts and give them the ability to be variabilized to allow for DDT and KDT. This involves the identification of variables and test data, as well as programming skills;

– maintain the test scripts so that they remain in phase with the evolutions of the software and systems that they test, throughout the life of these. Without this constant maintenance, scripts will become obsolete and their value will rapidly diminish;

– design reports focused on the needs of the stakeholders and identify the necessary metrics as well as the frequency of measurement, and synthesize this information to give a realistic idea of the progress of the project, both at the level of the system under test and at the system-of-systems level.

A significant workload will therefore remain for human testers, and test automation will only add to this workload.

It is therefore important to define where automation saves time, effort or money, and to take a long-term view that involves more than just delivering the software or system. Indeed, especially in systems-of-systems, the durations of use being long, the needs for maintenance and evolutions justify a long-term vision. This analysis of potential gains should take place from the genesis of the project and focus on the early search for defects, rather than a late search often linked to the execution of functional tests during system test levels.

10.8.5. Test frameworks

A test framework is a set of rules that can be used to manage tests. In this book, we will consider that a framework is an overlay – or a set of overlays – which is positioned above automated test tools (generally a replay capture tool) and whose purpose is to separate the concepts business of purely IT aspects. The objective is to decorrelate the “business” functioning – often stable – from the user interfaces which are variable. Testing frameworks are often used to do DDT (Data-Driven Testing) or KDT (Keyword-Driven Testing).

Let’s take an example: in a bank we have business actions such as “add a customer”, “open an account”, “make a transfer”, etc. In general, the data to be entered for each of these actions is fixed and will not vary from one version of an application to another. The principle of KDT is that the KDT engine for application 1 will be developed based on the automation tool applicable to the application (here: C/R Tool 1). It is possible to map the screens and define the variables of interest on each of the screens. Similarly, it is possible to define, for each of the variables, the combinations of variables that may have to be used (thanks to the test techniques defined in Volume 1, Chapter 9). We can therefore define in a spreadsheet, for each action (keyword) to be performed, the variables to be used and assign them to the variables of the selected screen. The keyword engine can then provide the automation tool with information on the screen to use, the variables to use, the actions to perform and the expected results. Depending on the result, the next action to perform is reading and processing the next line (e.g. Nxt), skipping one or more lines (e.g. skip) or performing a series of actions (e.g. create an anomaly in case of expected result different from the obtained result).

We can use the same concept to cover several different applications and several different automation tools, so we could issue test case execution orders from one application to another (as long as the framework is compatible between the various systems.

Some languages and frameworks directly allow this type of operation to be designed (e.g. Robot Framework in PHP), and some commercial tools offer this method of developing automated test cases. It is of course possible to define – name – the fields of a screen even before the screen is available and to map the fields of the screen with the fields created by the developers. This solution makes it easier to identify missing (or additional) fields that will need to be processed before running the automated tests.

Such test frameworks could send orders from one application to another in a system-of-systems automatically and thus enable test suites that would automatically test important aspects such as identifying regressions.

10.8.6. E2E test cases

Testing a system-of-systems from end to end (hence the acronym E2E) generally involves ensuring that the test environments of the various systems are correctly connected and interfaced, that the settings are synchronized between systems and that consistent functional test cases are defined, with valid test data. Often, it is impossible to fully test a system-of-systems because certain aspects cannot be properly reproduced in test environments. For example, it is not possible to reproduce on Earth a functioning without gravity, such as behavior in space.

Test automation is generally considered on a system-by-system basis, as each system is tested separately, but verification and validation of system-of-systems behavior should also be considered. How to do?

A solution may be to carry out pilot tests in conditions close to reality, as is the case in the air (flight tests), in space (e.g. landing test of boosters for certain rockets), in the naval sector (sea trial campaign) or in other fields (nuclear, railway, automobile, etc.). These tests – with measurements of numerous parameters – allow an analysis of the recovered parameters and a comparison of these parameters with the anticipated values of these parameters. So, automation here will focus on the means of measurement and their analysis.

For software-intensive equipment, we may consider simulators to replace specific components. We will therefore be led to consider these simulators as stoppers in the context of system integration tests. If the systems – replaced by caps or simulators – evolve, the caps and simulators will have to evolve at the same time.

In the context of end-to-end testing of a system-of-systems, we will have to ensure the coordination of tests between the various systems. If a testing framework is used that allows communication between the various systems under test, end-to-end test automation should be facilitated. However, consider a higher layer that can handle synchronization and reporting at the system-of-systems level.

Since the various components of a system-of-systems are developed by different companies, it is likely that the issue tracking tools used will be different and the rules for naming or numbering issues will vary. Therefore, traceability must be put in place between the references of anomalies at the level of the system-of-systems and the same anomalies at the level of each of the other systems or components.

10.8.7. Automated test case maintenance or not?

As mentioned on several occasions, tests, automated or not, often have a life span at least equal to the life span of the software, component or system to be tested. It will be necessary to ensure that the evolutions of the software, component or system during its existence are considered and reflected in the tests, therefore maintaining these tests. This will ensure that all the evolutions will be correctly tested and that the functionalities which will disappear will be correctly removed from the list of tests to be carried out.

This synchronization between tests and developments will require the presence of a test team to:

– Analyze the impacts of requirement modifications, which will be facilitated by a tooled follow-up of the requirements and the traceability from the requirements to the developments, to the tests and their execution results. A configuration management tool will ensure the identification of developed elements, including test cases, test data and execution results. An analysis of the impacts of these modifications on the other components will make it possible to identify the full scope of the interventions to be considered.

– Design the new test cases, applicable to the new functionalities, and ensure full coverage – nominal behaviors and exceptional behaviors – of the functionalities, according to the new requirements. It will also be necessary to update the reports produced in order to take these new test cases into account.

– Define and implement new test cases – as well as test data – to ensure both requirements coverage and structure coverage (e.g. code, interfaces and messages, etc.), so as to adequately cover the scope of intervention identified during the analysis mentioned above.

– Identify test cases – and the data they use – that are no longer applicable (e.g. feature removed) and update product reports to reflect the removal of these test cases. Of course, it will be necessary to update the aspects of traceability from and to the requirements and the elements developed (which should also be modified).

– Create automated test scripts or update the identified automated test cases in order to reflect – in the test automation tools – the developments mentioned in the previous points (addition, modification or deletion of tests, adaptation of reports).

These activities are necessary and not carrying them out will expose the projects to the creation of technical debt which will accumulate quickly over time and will be difficult to reduce.

10.9.1. Automated reporting for the test manager

It is interesting for the test manager and for the test teams to have a real-time visualization of the progress of the tests on the various environments and for each of the characteristics – functional or not – being tested. This can be done by developing a small tool which, from the various tools used (test manager, test execution tracking, defect management, configuration management, requirements management, etc.), will extract information that can be aggregated and integrated in order to provide summary information for team members. This type of reporting, displayed on large screens visible to team members (and by management), shows everyone the level of progress of application testing.

Using a spreadsheet facilitates the extraction of data, and their use, independently of the tools used. This makes it possible to integrate information from different tools, without changing the reporting provided.