Feeling the Pain
We’ve identified four main types of pain your team might start to feel if their Cucumber goes bad. Take a look and see whether you recognize any of these:
| Symptom | Problem |
|---|---|
| Some of our tests fail randomly. |
Flickering scenarios |
| We keep breaking tests unintentionally. |
Brittle features |
| Our features take too long to run. |
Slow features |
| Our stakeholders don’t read our features. |
Bored stakeholders |
Let’s take a closer look at each of these symptoms.
Flickering Scenarios
When a scenario that was passing yesterday is failing today, with the same source code running in the same environment, you have what we call a flickering scenario. Here’s our definition of a flickering scenario:
- Flickering scenario
-
A flickering scenario fails occasionally and at random. The same scenario, run on the same codebase in the same environment, will mostly pass but sometimes fail. These apparently uncontrollable failures cause the team to lose confidence in their tests, in their code, and in themselves.
The worst thing about a flickering scenario is that as soon as you try to reproduce it so that you can fix it, it refuses to fail. Fixing a flickering scenario is one of the hardest tasks you can take on, yet it’s also one of the most important. For a suite of automated tests to be useful, the team must have absolute trust in it. When even just a single test is compromising that trust, it has a corrosive effect on how everyone feels about the whole test suite.
To fix a flickering scenario, you have to study the code and try to understand why it might be happening. This is a scientific process of making a hypothesis about the cause of the failure, creating an experiment to prove or disprove that hypothesis, and then running the experiment to see whether you were right. You might need to go around this loop several times before you crack the problem, and it might take several days to run an experiment, if the flickering scenario fails only intermittently. If you run out of ideas, consider simply deleting the test altogether rather than have it come back and fail on you at a time of its own choosing.
Flickering scenarios are caused by one of the following underlying problems:
Brittle Features
When you feel like you can hardly move in the test suite without making an apparently unrelated test fail for no good reason, you have what we call brittle features. Here’s our definition:
- Brittle feature
-
A brittle feature is easy to break. When features are brittle, a necessary change in one part of the test suite or main codebase causes apparently unrelated scenarios to break.
When you encounter a brittle scenario, it’s usually when you’re in the middle of doing something else. You’re interrupted by the unexpected failure and waste time dashing over to fix the unexpected broken test. On a bad day, this can happen several times before you emerge from the rabbit warren. Brittle features are self-fulfilling: when developers perceive their tests to be brittle, they tend to be less courageous about refactoring and cleaning up test code and instead try to get in and out as quickly as they can, leaving the tests and production codebase in an increasingly hard-to-maintain state.
Brittle features are caused by one of the following underlying problems:
Slow Features
Each time you add a new scenario to your test suite, you’re adding a few seconds to the total test run time. For a successful application whose users continue to demand new features, that test run time is only going to get longer. A long test run creeps up on you: first five minutes seems like an eternity to wait, then fifteen minutes seems bad, but you get used to going to grab a coffee while it runs. Pretty soon, you come back from your coffee and it still hasn’t finished, and fifteen minutes becomes twenty-five minutes. Before you know it, your features are taking more than an hour or even longer.
Once a new scenario is passing, the main reason to keep running it is for feedback: you want that scenario to warn you if you somehow accidentally break the functionality that it checks for. The value of that feedback diminishes as your test run starts taking longer and longer. When the build is slow, developers don’t run all the tests before committing code and will rely on the continuous integration server to give them feedback instead. If a few developers do this at the same time, the chances of all of their changes integrating successfully are slim, and a broken build becomes the norm.
Long test runs also mean people are scared to refactor and do other general maintenance on the Cucumber tests themselves. Refactoring the code in a step definition that’s used in 340 scenarios is scary, because you’ll need to run all 340 scenarios to tell you for certain whether your change broke anything.
A slow feature run is normally caused by a combination of the following:
Bored Stakeholders
“Our stakeholders don’t read our features.” This is a common complaint from teams that have tried Cucumber but failed to make it stick as anything other than a tool for automating their test scripts. Yet many other teams attest to the transformative effect Cucumber has had, helping development teams to collaborate much more effectively with their business stakeholders. What could be the difference between these two experiences?
The answer lies partly in starting with the right kind of collaborative relationship with those business stakeholders. If they think they’re too busy to help you understand exactly what they want, then you have a deeper team problem that Cucumber can’t help you solve. On the other hand, many teams who start out with keen and interested stakeholders waste the opportunity Cucumber gives them to build that collaborative relationship. When features are written by testers or developers working alone, they inevitably use technical terms that make the stakeholders feel marginalized when they read them. This becomes a vicious circle: as stakeholders lose interest, they spend less time helping write the features in a language that makes sense to them. Before you know it, the features have become nothing more than a testing tool.
This painful symptom is normally caused by a combination of the following underlying problems:
Once you’ve spotted any of these symptoms in your team, you need to know what to do. It’s time to look at the underlying problems that are at work and what you can do about them.