Acceptance Criteria

This is very true indeed.

Hopefully, this discussion leads to more questions for the product owner, which will help to understand better what should be tested. The answers will help to define the product more clearly.

In TFS or Azure DevOps, you can collect all this important information in a PBI (or user story or requirement, depending on the process template you use).

The PBI gives good traceability to follow the requirement to its acceptance criteria. Figure 6-1 shows an example of how the Visual Studio Scrum template in Azure DevOps displays this information.

Evolving Tests

In an Agile process, during which development is done in small increments, you also need to make sure the tests follow an iterative way of working. The tests need to sync with the flow of the application development; otherwise, you might run in to a lot of problems down the line.

Early in a project life cycle, little is known about a new feature, and you need to run tests against all acceptance criteria defined for the requirement. When a feature is completed, you should be confident it has been tested according to the test cases and that it works as expected. After that, you only need to run tests to validate changes in the requirement. This means you must have a process for knowing which tests to run.

Running all tests manually is tedious and takes a lot of effort in consideration. Instead, rethink how you design your test cases. You could, for example, think of your test base as a pyramid (Figure 6-2). Figure 6-2 shows how different types of tests can be put in proportion in a specific case.

At the bottom of the pyramid are the unit and component tests, which are the main part of the testing effort. These tests are relatively cheap to create and maintain; but, to test the system as a whole, you probably need to add regression tests that run end-to-end tests as well.

Some of the regression tests should be implemented as user interface tests to simulate how an end user uses the system. However, user interface tests are more complex to design and maintain, and often it isn’t practical to have more than a small set of them. Most of these tests can and should be automated to provide an efficient way of maintaining changes in the product.

With this way of designing tests, you end up with a small number of manual tests at the top of the pyramid. This means manual tests are always necessary to some extent.

Now let’s talk a little about how TFS can help us manage our tests.

Clients for Managing Tests

Test-Driven Development

TDD is a practice that originated with Kent Beck, who is credited with having developed or “rediscovered” the technique. TDD is one of the core practices in XP, but has created lots of general interest in its own right. So, even if you do not use XP, you can still use this practice as a way to help developers write better code.

TDD relies on the repetition of a very short development cycle. First, the developer writes an (initially failing) automated test case that defines a desired improvement or new function, then produces the minimum amount of code to pass that test, and then refactors the new code to acceptable standards. Kent Beck stated in 2003 that TDD encourages simple designs and inspires confidence.

Instead of designing a module, then coding it and then testing it, you turn the process around and do the testing first. To put it another way, you don’t write a single line of production code until you have a test that fails.

In traditional software development, tests were thought to verify that an existing bit of code was written correctly. When you do TDD, however, your tests are used to define the behavior of a class before you write it.

With TDD, you want your tests to run frequently to get continuous feedback about the written code. A change in code that breaks one or more tests is something that demands immediate notification. You can configure Visual Studio 2015 to run unit tests automatically after build so that as soon as the code is compiled, all tests the tests are run and feedback is generated regarding their results. This means the feedback loop is very short. The loop is the time it takes for (in this case) a developer to make a code change to when that developer gets feedback on whether the change was successful.

Working with Automated Tests

To achieve the goals of automated testing, you need to plan ahead and think about what you really want to get out of your automation efforts. Visual Studio 2015 helps you to set up your test environment and to select which test types to use.

The following sections on CI/CD are inspired by my boss, Mathias Olausson. He is an expert in this area, so make sure to check out his recent book about CD with Visual Studio ALM 2015 at http://www.apress.com/9781484212738?gtmf=s . It is a great reference for this topic.

Continuous Integration

CI is a practice in XP that has come to be the defacto standard in Agile projects. Martin Fowler was the person who introduced it to the more general public, but it was first named and proposed by Grady Booch in 1991.³ CI is a practice that integrates the code base frequently. In combination with running automated unit tests in the developer’s local environment and verifying they all pass before committing to the mainline, CI aims to make sure the developer is not checking in code that breaks any other developers’ code. Over time, this practice has evolved, and now builds run on build servers that run the unit tests automatically and periodically—or even after every commit—and report the results to the developers. CI has spread outside the Agile community and is now used frequently in other types of projects as well.

In addition to running the unit and integration tests, you can also run static and dynamic tests, measure and profile performance, extract and format documentation from the source code , and facilitate manual quality assurance processes. In this way, you get continuous quality control of your software as well.

CI can help resolve these issues. In fact, Scrum has a solution for this; use the retrospective to find ways to improve.

Why Continuous Integration?

Keep in mind that CI is not free of costs. You need to maintain your CI solution, including the build environment, over time. It can also take quite some effort to introduce it into your organization. And don’t forget that CI has costs for setting up the new build and CI infrastructure as well.

Figure 6-3 shows a process that is a complete CI solution. It should be what you strive to achieve.

Continuous Delivery

The problem with CI is that it can be a solution to a nonexistent problem. Deployment as part of the CI flow is not just about automating the build, test, and release process. You really need to think about delivery to add value to the deployment process .

CI is great, and it supplies a framework for producing software efficiently in a controlled fashion. But, to get the most out of it, you need to look at how it fits into the overall process of delivering software. In an Agile project, you want to deliver working software in every iteration. Unfortunately, this is easier said than done. It often turns out that, even if you implement CI and get the build process to produce a new installation package in a few minutes, it takes several days to get a new piece of software tested and released into production. So how can you make this process work better?

Let’s start by asking the following simple question:

How long does it take to release one changed line of code into production?

Probably, the answer is: Much longer than you would want to. Why is this? First, you must know more about how you release your product. Mathias Olausson says that even in organizations that follow good engineering practices, the release process is often neglected. A common reason why this happens is simply because releasing software needs collaboration across different disciplines in the process. To improve the situation, you need to sit down as a team and document the steps required to go from a code change to the software released into production. Figure 6-4 shows a typical delivery process. In practice, work happens sequentially, just like in the picture.

Having looked at the process and answered the previous questions, you should now have a better process, as shown in Figure 6-5.

With this model (Figure 6-5), the process is changed so that most steps are automated by implementing automated tests as well as automated build and deployment. Releasing to production automatically is not for the faint-hearted, so this would be done manually, but the same automated scripts as the automated deploy to test and staging environments should be used. It is possible to automate release to production, especially if you have had this step in place since the first iteration of the project. By doing so, you build confidence in the process and, having seen it work throughout the development cycle, you should trust the process at this critical stage. Also, in the model, parallelization the acceptance test and preparation of the production environment has occurred. By doing these steps in parallel, you can push the release to production as soon as the acceptance tests are green, instead of the traditional stage to production first after the acceptance tests have passed.

CD is a great practice to produce updates in a controlled and effective manner. But, without an intentional release management discipline, you can lose much of its value. What you need to add to the picture is how the release planning ties into the deployment process and then ensure you know which features you want to deploy where and when. This is not covered in the scope of this book, but please refer to Mathias Olausson’s book mentioned earlier for more details and best practices.

Azure Pipelines

With Azure DevOps, you can use the service Azure Pipelines to automate your CI/CD flow. You can build, test, and deploy Node.js, Python, Java, PHP, Ruby, C/C++, .NET, Android, and iOS apps. And, you can run in parallel on Linux, macOS, and Windows.

You can also build and push images to various container registries such as the Azure Container Registry or Docker Hub. Or, you can deploy your containers to individual hosts or Kubernetes, which is an open-source system for automating deployment, scaling, and management of “containerized” applications.

You can further use Azure Pipelines to implement your CD of software to any cloud, including Azure, Amazon Web Services (AWS), and Google Cloud Platform (GCP). Using Azure DevOps, you can visualize deployment to any number of interdependent stages as well.

SAFe in Azure DevOps

Because organizations are applying an Agile way of working more and more, many are asking themselves how they can get agility to work at scale. There are several frameworks available for scaling Agile such as LeSS and Nexus, but many organizations turn to SAFe. There is no process that supports SAFe out of the box in Azure DevOps, so you need to work with the ones you have got unless you want to create a custom process.

Let’s take a look at SAFe in Figure 6-6. This is the largest SAFe configuration, called the Full SAFe. There are four levels—Portfolio, Large Solution, Program, and Essential SAFe—that cover most organizational needs. Microsoft has four work item types that can be used to support this setup: Epics, Features, User Stories, and Tasks. There is a hierarchy to these work item types (Figure 6-7). The hierarchy originates from the Agile process, but works the same for Scrum and CMMI. So, if you’re interested in using SAFe, you can configure projects created with the Scrum, Agile, or CMMI processes to track SAFe criteria.

Figure 6-7 shows at what SAFe level these workt item types would be used. Epics support the Portfolio level; Features, the Program level; and User Stories and Tasks support the Team level. To support SAFe in Azure DevOps further, you need to work with your areas and iterations in Project Settings. Figure 6-8 shows one way of implementing SAFe in the iterations view.

You can see the overall Portfolio level (SAFe Demo Azure DevOps) with one value stream present (Value Stream 01). The value stream includes the ARTs (the Program Level)—in this case, ART 01 and ART 02—which lead to three program increments (PI 01, PI 02, and PI03), followed by the configured the sprints for each PI.

Because epics can span several release trains, the portfolio team probably wouldn’t be associated with any specific iterations. Program teams, on the other hand, track their features, which ship with a PI. The feature teams work in sprints to complete the stories chosen for the PI. Each team in turn chooses which iterations support them in tracking their deliverables.

When you are satisfied with the initial iteration setup, you can select which areas the different teams should have (Figure 6-9). You can always modify your iterations as time goes on, by, for instance, adding new PIs. Using areas, on the other hand, you determine which items the teams see on their backlogs and boards.

As you can see in Figure 6-9, the Portfolio team will track epics in each value stream and thus will keep track of higher level efforts. The team can also use a Kanban board (Figure 6-10).

The ARTs are tracked by the ART management teams (ART 01 MGMT). These teams can also follow everything that happens in each PI at the ART, PI, and team levels. Figure 6-11 shows the ART team tracking features on the Kanban board.

The individual teams (Team 01 and Team 02) will work with things on their own backlogs (subsets of the program backlog), so they can focus on their work. As seen in Figure 6-12, Team 01 can follow its PBIs on separate backlogs or Kanban boards.

This is one way of using Azure DevOps to support SAFe. Keep in mind that the flexibility of Azure DevOps and Azure DevOps Server makes it possible for to customize implementations. Before we leave this chapter, I want to say a few words about Nexus and Scaled Professional Scrum from Scrum.​org.

Nexus in Azure DevOps

This section is written in part by one of my excellent coworkers—Jesper Fernström—who works for Solidify in Stockholm. Solidify is a certified Scrum.​org training organization that is has considerable expertise with Azure DevOps. Let’s take another look at the Nexus framework (Figure 6-13).

If you know Scrum, Figure 6-13 should look very familiar to you. Unlike some other frameworks, when you scale Scrum using Nexus, Scrum is still Scrum. What you get with the Nexus framework are a few additions to Scrum to handle cross-team dependencies and issues, and improvement opportunities to ensure teams can deliver an integrated increment at the end of every sprint. Without going deeper into the Nexus framework, Figure 6-14 shows what is added to the mix.

The neat thing with Nexus is that it requires no change to your process template in Azure DevOps because you are still doing Scrum! The new role, events, and the Nexus goal do not need to be modeled into your process template. The Nexus sprint backlog is just an aggregated view of all the individual team sprint backlogs. So, the only thing you need to do is set up your teams so they have one common aggregated “view of everything” and their own filtered team views, which is basically what everyone wants for any multiteam effort—and is supported out of the box by Azure DevOps.

And you are done!

For “anything Nexus,” such as Nexus sprint planning and Nexus sprint backlog, use the view of Nexus team. For team-specific stuff, use the individual team views, just as you did when there was only one team.

Why Not a Dedicated Team for the Nexus Integration Team?

So, when trying to scale Scrum using Nexus, the Nexus integration team is a sort of semivirtual team with a “servant–leader” flavor. Although the last sentence of the quote says the team might perform work from the backlog, this should be rare and more of an emergency action than anything else. All in all, you are much better served by an aggregated “Nexus view” from which you can get an overview of the current situation.

Further Improvements

And with these words, we end this chapter and thank Jesper for his input.