Returning Results
Cucumber is a testing tool, and it’s in the Ruby code of a step definition where our tests find out whether a step has succeeded in whatever it set out to do. So, how does a step definition tell Cucumber whether it passed or failed?
Like most other testing tools, Cucumber uses exceptions to communicate the failure of a test. As it executes a scenario, one step at a time, Cucumber assumes that a step has passed unless its step definition raises an exception. If a step passes, it moves on to the next step. The figure shows how this plays out.
In Cucumber, results are a little more sophisticated than a simple pass or fail. A scenario that’s been executed can end up in any of the following states:
- Undefined
- Pending
- Failed
- Passed
These states are designed to help indicate the progress that you make as you develop your tests. Let’s run through an example of automating the ATM withdrawal scenario to illustrate what we mean.
Undefined Steps
When Cucumber can’t find a step definition that matches a step, it marks the step as undefined (yellow) and stops the scenario. The rest of the steps in the scenario will be either skipped or marked as undefined too if they don’t have a matching step definition themselves.
To show you how this works, we can run our ATM withdrawal scenario. Create a file called features/cash_withdrawal.feature, and put the following into it:
| | Feature: Cash Withdrawal |
| | Scenario: Successful withdrawal from an account in credit |
| | Given I have deposited $100 in my account |
| | When I request $20 |
| | Then $20 should be dispensed |
We haven’t written any step definitions yet, so when we run this feature, we should see the steps all come up as undefined:
| | Feature: Cash Withdrawal |
| | |
| | Scenario: Successful withdrawal from an account in credit |
| | Given I have deposited $100 in my account |
| | When I request $20 |
| | Then $20 should be dispensed |
| | |
| | 1 scenario (1 undefined) |
| | 3 steps (3 undefined) |
| | 0m0.017s |
If you have color output in your terminal, you should see each step is yellow, indicating that it’s neither failing (red) nor passing (green) but somewhere in between. Notice also that Cucumber has printed out a snippet for each missing step definition. We can use these as a starting point for implementing our own step definitions.
Pending Steps
When Cucumber discovers a step definition that’s halfway through being implemented, it marks the step as pending (yellow). Again, the scenario will be stopped, and the rest of the steps will be skipped or marked as undefined.
How does Cucumber know whether a step definition has been implemented?
Of course, you have to tell it. A few special Cucumber methods are available from within your step definitions’ code blocks (which we will talk about more in Chapter 7, Step Definitions: On the Inside); one of them is pending.
When you call pending from within a step definition, it raises a Cucumber::Pending error. This tells Cucumber’s runtime that the step has failed but in a particular way: the step definition is still being worked on. You’ll probably have noticed that the snippets Cucumber generates for undefined steps have a call to pending in them; now you understand why.
Let’s get back to our ATM withdrawal scenario to show you what we mean.
Create a file called features/step_definitions/steps.rb and paste in this step definition:
| | Given(/^I have deposited $(d+) in my account$/) do |amount| |
| | pending("Need to design the Account interface") |
| | end |
Now when we run cucumber, we’ll see that it has only tried to execute the first step and has marked it as pending. All the rest are still undefined:
| | Feature: Cash Withdrawal |
| | |
| | Scenario: Successful withdrawal from an account in credit |
| | Given I have deposited $100 in my account |
| | Need to design the Account interface (Cucumber::Pending) |
| | ./features/step_definitions/steps.rb:2 |
| | features/cash_withdrawal.feature:3 |
| | When I request $20 |
| | Then $20 should be dispensed |
| | |
| | 1 scenario (1 pending) |
| | 3 steps (2 undefined, 1 pending) |
| | 0m0.021s |
The pending status is bit like those under construction signs you used to see all over the Internet in the 1990s. You can use it as a temporary signpost to your teammates that you’re in the middle of working on something.
When we’re developing a new scenario from the outside-in like this, we’ll tend to work right across the same layer before diving into the next one. Right now we’re concentrating on adding step definitions, so let’s do that for each of the other steps and add a pending call in each one:
| | Given(/^I have deposited $(d+) in my account$/) do |amount| |
| | pending("Need to design the Account interface") |
| | end |
| | When(/^I request $(d+)$/) do |amount| |
| | pending("How do we simulate cash being requested?") |
| | end |
| | Then(/^$(d+) should be dispensed$/) do |amount| |
| | pending("How do we validate that cash was dispensed?") |
| | end |
As we do this, we can check to see whether there are any existing step definitions we could use and, if not, just create a new one with a call to pending. The pending scenarios become our to-do list for the work we’ll do when we drop down to the next layer and start implementing the step definitions.
Failing Steps
If the block of code executed by a step definition raises an exception, Cucumber will mark that step as failed (red) and stop the scenario. The rest of the steps in the scenario will be skipped.
In practice, there are two reasons why a step definition will fail:
-
The scenario couldn’t finish because you have a bug in your step definition code, or in the system under test, which has caused it to throw an error. You’ll get used to seeing these failures all the time during development if you use Cucumber to drive your development from the outside-in. Each failure message tells you what you need to do next.
-
The step definition has used an assertion to check something about the state of the system, and the check didn’t pass. You’ll typically get these errors right at the end of your outside-in cycle or long after the feature has been implemented if someone accidentally introduces a bug.
An assertion is a check in your tests that describes some condition that you expect to be satisfied. Failures because of assertions tend to happen in Then steps, whose job is to check things about the state of the system. These are the alarm bells you’re fitting to the system that will go off if it starts to behave in unexpected ways. Either way, Cucumber will show you the exception message and backtrace in its output, so it’s up to you to investigate.
We’re almost done with this chapter, but we want to show you an example of a failing step first. We’re at the point where we need to start designing the interface between our tests and our system. Let’s start with something simple and imagine an Account class that we can use to create a bank account for the actor in our scenario. We can modify the step definition like this:
| | Given(/^I have deposited $(d+) in my account$/) do |amount| |
| | Account.new(amount.to_i) |
| | end |
What’s going to happen when we run this? We don’t actually have an Account class, so it’s going to blow up:
| | Feature: Cash Withdrawal |
| | |
| | Scenario: Successful withdrawal from an account in credit |
| | Given I have deposited $100 in my account |
| | uninitialized constant Account (NameError) |
| | ./features/step_definitions/steps.rb:3 |
| | features/cash_withdrawal.feature:3 |
| | When I request $20 |
| | Then $20 should be dispensed |
| | |
| | Failing Scenarios: |
| | cucumber features/cash_withdrawal.feature:2 |
| | |
| | 1 scenario (1 failed) |
| | 3 steps (1 failed, 2 skipped) |
| | 0m0.021s |
What’s this failure telling us? It’s telling us we need to create an Account class, of course! We’re not going to start building our banking system right now, but we’ll pick this example up again in Chapter 7, Step Definitions: On the Inside. As you’ve seen, Cucumber has caught the exception and displayed it to us just beneath the step. In the summary at the bottom we can see that the step failed, and the rest were skipped.