Chapter 11. Automate Literally

After you worked hard on trawling together the examples for your application, it would be a waste of time if you directly threw them away without using them for your advantage—or even worse, you could put them in a document and wait for the dust to cover the print-out. Instead, a wiser approach for you would be to use your examples in order to drive your test automation efforts. If you can automate your examples to the degree that you can use it during an iteration demonstration or general product demonstration, your business representative will recognize the previously agreed upon examples.

Now you can achieve this recognition with a friendly automation approach. Friendly automation enables you to automate the examples as literally as possible. This means that you may reformat your examples in order to fit a certain table layout, but you can keep the textual portions of the examples almost to the letter.

In the airport parking lot example we saw literal automation. Tony reused the examples Bill, Phyllis, and he had agreed upon previously in the workshop. Using friendly automation he could automate these examples nearly to the letter.

One could argue that Tony had written down the examples in a format suitable for automation in first place. Indeed, he could have become biased by the tools that are around for Agile-friendly test automation. On the other hand, the tables also could have been influenced by traditional decision tables, assuming as a tester he knows about them.

Another aspect was the collaboration between programmers and testers in both examples in order to automate the given examples. In the airport example, Tony paired with Alex on the automation. In the second example, we switched our thinking hats between test generation and design thinking from time to time to reach similar results.

Finally, literal automation serves also the purpose of representing the business domain in your tests. In fact, in the second example we saw how the traffic lights were first represented in the tests, and how the tests drove the domain model code from there.

Around 2010 I heard the term domain-specific test language for the first time. My first reaction was that there shouldn’t be a discrepancy between the domain language that is used in your code and the one used in your tests at all. Sometimes—for example, in a legacy code system—you have no other choice but to keep the two separated when you begin your ATDD journey. Over time, though, you should unify the two into a single ubiquitous language used in your project. If you keep the two separated, you are growing and fostering miscommunication with your project stakeholders. One way to achieve this is to drive the domain code directly from your examples.

Let’s take a closer look on these three aspects: Agile- and ATDD-friendly test automation, collaboration between programmers and testers, and finally how to discover the domain and a sufficient and appropriate domain model from your examples.

Use Friendly Automation

I heard the term ATDD-friendly test automation for the first time when I attended a tutorial on ATDD held by Elisabeth Hendrickson in 2009. It indicates that there are test automation tools available to support ATDD and literal automation.

One piece of advice at this point: Although we will dive into a tool discussion here, ATDD really is not about tooling; ATDD is an approach that can be used with many tools. If you implement a tool, you will get an implementation of that tool, but not an application of a working approach. A tool is not a test strategy [KBP01].

That said, let your team make the decision about the tool. There are multiple considerations in picking the right tool for your approach. The programming language used might play a role in your decision as well as readability concerns of your Product Owner or on-site customer. Depending on the degree your whole team, including programmers, testers, and business representatives, will work with the tool, you will prefer different aspects in the software.

As of this writing, there is a variety of test automation tools that support ATDD in diverse ways. Keywords, data-driven tests, and table structures all provide a ground to get started with ATDD. Some prefer the ability to automate via natural language, so that you can keep your examples readable like a document. Other prefer table structures or tests expressed in text files with a certain structure. Over time, though, all these features of the different tools started to influence the other frameworks as well, so that you can now work with either framework while still getting most benefit from the other frameworks around.

Part of this success comes from the OpenSource community. With the source code available for the major acceptance test frameworks around, new features and bugfixes are released timely. If you have an issue with one of the tools, you can usually file a question to one of the supplier’s mailing lists or contact its support team. Most of the time you will get a helpful response within a day.

In the examples we already saw two such tools. Cucumber is a Ruby-based framework favoring tests in a behavior-driven development style. There are ports available for Java, Groovy, Scala, Clojure, and .NET, so that you can hook up your code with any of these languages. For more on Cucumber, see Appendix A and The Secret Ninja Cucumber Scrolls [dFAdF10].

The second example dealt with FitNesse. FitNesse is an acceptance test framework built in Java. It supports two test systems using either FIT and FitLibrary or SLiM. There are ports available for Java, .NET, Ruby, Python, and PHP. Since FitNesse is based upon a wiki, I found it suits distributed teams quite well. FitNesse itself is covered in more depth in Appendix B.

Other frameworks include Robot Framework, Concordion, JBehave, but also commercial ones such as Twist or Green Pepper. New frameworks may also be released in the next few years. These new tools might overcome some of the short-comings in the existing tools and reshape the tool landscape.

What all these tools have in common is an easy way to write text to express your requirements. Cucumber favors a Given-When-Then style, FitNesse has a table structure, and Robot Framework uses keywords. We saw examples of both styles in the examples.

What makes these tools friendly to ATDD is their ability to automate the examples given in a particular form with the usage of some code. The Cucumber steps use pattern matching with regular expression to decide which code to execute. For FitNesse with SLiM code, functions are called by convention. This creates a separation of concerns between the application that is tested and the textual representation of the tests.

When you use ATDD, you can derive the examples that you will test before writing a single line of code. Since the tools provide full test data separation from your application, you can grow the application alongside with the test examples, or even let your examples drive your implementation as we saw in the traffic lights example.

This separation of concerns between test data and application under test is the key to ATDD- and Agile-friendly test automation. When you adapt the ATDD approach, other tools will get in your way of a successful implementation. Test data separation is not a sufficient demand but a necessary one for a successful ATDD implementation.

Collaborate on Automation

Test automation is software development. Many experts have claimed this for decades. More interestingly, teams still fail to keep this in mind and end up completely lost with their test automation, eventually abandoning all their efforts for test automation at all. There is more to this particular topic, enough for me to include a separate chapter on test automation (see Chapter 12).

This section focuses on the collaboration part of test automation. As we saw in the airport example, tester Tony paired with Alex the test automation expert. Before that, Tony worked as far as he could on his own. When he got stuck completely, he knew he had to seek help.

One question that comes up in almost every class I teach on ATDD is the question whether the focus on test automation means that testers need to program. Many testers dropped out of programming because they did not feel passionate enough about programming. This sometimes leaves people with the impression that they are rather bad at programming.¹ Confronting them with ATDD can make them scared enough to resist the approach at all.

There is no need for testers to be afraid of ATDD. A focus on test automation does not mean that testers cannot collaborate with a programmer for test automation success. In fact, most programmers I paired with on test automation were more than eager to find out more about some of my skills as a tester as well. On the other hand, I learned new things about how an application was implemented as well as software design. Pairing on test automation is a win-win situation for the programmer and the tester.

When taking a closer look on the ATDD-friendly test automation frameworks, the frameworks decouple the tests from the tested application by introducing a mediation layer of code between these two aspects. In most teams starting with ATDD, testers maintain the test data while programmers are responsible for the system under test. But who should write the mediation code between these two aspects?

If the tester writes and maintains all the code that translates test examples to system calls in the application, then changes to the application might be overlooked if the programmer forgets to tell the tester. Once I worked with a company that had a separate testing department. The testers in that department were responsible for automating the tests on their own. Lack of communication forced a great deal of test automation rework.

On the other hand, if the programmer who maintains the portion of the application also writes the glue code to hook up the tests, changes in the application and the glue code may result in failing tests that were not adapted.

For both scenarios, the programmer and the tester have to communicate with each other. This is the lowest level of collaboration for test automation. I’d also claim that your team is not a team if it isn’t communicating most of the time with each other. An informative workspace with Information Radiators [Coc06] can help with that. For example, you can put a pairing matrix on your taskboard if you experience a lack of collaboration between programmers and testers.

If you want better results with your test automation efforts, then programmers and testers should work closely together. In the best case they work on the glue code together side by side in a pair programming setup. The amount of information sharing in such a setting is immense enough that experienced pair programmers claim that pair programming yields a more than doubled productivity for the pair. This has been also my experience, although I don’t have hard numbers on that.

Bringing both perspectives of the development project together at one desk to deal with the automation of part of the application also serves the purpose of exchange between the two specialists. The programmer learns a bit about testing and how the tester derives new test cases from the available information. The tester, on the other hand, learns more about programming, test-driven development, and can help the programmer to write maintainable code that the tester also understands.

Collaboration is a double-win situation for the whole project. Concerns about initial specialists in your team may be smoothed out through collaboration. Collaborating on test automation enables your team to implement collective test ownership, as Elisabeth Hendrickson calls it. With collective test ownership your whole team is empowered to write new tests and work on the automation code. This helps team members to cover for each other if progress is behind schedule. As a side effect, you will also get automated tests and a shared understanding of your test code and your tests.

Collaboration can come in many forms. As a minimum, testers and programmers should communicate about the glue code. Ideally, though, programmers and testers work together on critical parts of the automation code. The drawbacks of no collaboration at all are silo walls between programmers and testers, and in the worst case misunderstanding of the whole application. This can, in fact, destroy all your efforts for team building. By all means, I would rather avoid such a dysfunctional team.

Discover the Domain

In the traffic lights example we saw how to drive the domain code from the starting point of the tests. While driving the domain code, we also discovered some aspects about the domain itself. For example, the controller and the validator for valid light state combinations were not part of our initial design. The code we wrote in order to test the traffic lights motivated those concepts.

All our efforts as a software industry with domain-specific languages and behavior-driven development aside, there is a gap between the domain where our applications run and the domain model that is found in our code base. In our domain model we mostly add classes and constructs that are not existent in the real domain.

The Swedish Army has the following dictum [GW89]:

When the map and the territory don’t agree, always believe the territory.

Considering our domain model a map of the domain, if our domain model and our domain disagree, we should rather stick to the concepts in the domain, not the model.

On the other hand, as Michael Bolton put it,

When you’re lost, any old map will do.

This means that it is OK to get lost with your domain model from time to time, as long as you keep yourself in the position to adapt to the territory that you will find. You should work on the flexibility of your tests as well as the flexibility of your code. As long as you can adapt your code and your tests to the discoveries in the domain, you will be fine.

Tests and examples can guide our efforts to derive the domain model from the domain. They help us translate the real-world concepts and discover our model needs beyond them. Tests can help to elucidate missing elements in your domain model and code. Like the validator and the controller in the traffic lights example, these may not exist in the real world, but help to model the domain in the programming code.

As we saw in the traffic lights example, this technique works best if your programmers and testers work on the domain code and the automation code together. Not all teams work in this way. Driving the domain design and code from the outside-in does work poorly in these situations. Most teams starting with ATDD will not get this benefit right from the start. That’s why I consider this an advanced technique.

Prerequisites to this are a close collaboration between programmers and testers. Ideally, programmers know and maintain all the test examples together with the testers. In this case, they can choose to drive the domain code outside in if they get stuck on the design without it. Ideally, though, I would aim to work outside-in all the time.

Another prerequisite for outside-in development is some experience. Experience can be gained over time by applying it more and more and learning from it. You might not see how to drive your code from the outside right from the beginning, but you should push yourself forward to do this. Whenever you see a possibility to drive your code, take it.

If you want to reconsider your position on outside-in development, ask yourself, whether there are other possibilities other than the ones you see right now. Remember the Rule of Three interpretations, as Jerry Weinberg coined it [Wei93]. If you can’t think of three different approaches to drive your code, you haven’t thought enough about the problem—or you didn’t ask yourself the crucial questions on problem definitions: What is the problem?; Who has it?; Who should fix it?

Don’t decide that you can’t drive your code and your domain model from the outside-in until you have found at least three different ways to do it. At least three means that you might easily find five possibilities. Then you pick the one that looks most promising to you while still considering the constraint to drive the code from the outside-in. If you thought about three different possibilities to write that code, and still see no way to drive it outside-in, write it in the most promising way. Over time your experience with outside-in development will grow, and you will see more opportunities. Keep in mind that code is not cast in stone, and today’s integrated development environments (IDEs) provide a multitude of ways to restructure and rewrite your code later.

Summary

When you automate your examples, there are a few points to watch out for. First, make sure that you use the right tool to do it. Involve your whole team into that decision, because potentially everyone else has to work with it at some point in the future. Second, you want to avoid barriers for anyone on your team to this and also make sure that the tool serves your process, and not the other way around.

With tooling barriers removed, you should also be able to collaborate on automation. A tester can benefit from design knowledge of programmers, and programmers can benefit from the testing knowledge from the tester. As a side-effect this will also help to spread the tacit knowledge in the heads of your team, thereby helping you reach a common understanding of the problem you are solving for your customer.

Outside-in development can help with this, too. If the application domain code reflects your common understanding, then you might be able to have your customer at least read and understand your code. If you use your acceptance tests to discover the domain you are dealing with, you can drive your application, design, and architecture in completely different directions. This helps building applications that are easy to maintain in the long-run.