Chapter 4: Considerations for Good API Test Automation

In 1811, an angry group broke into a factory in the town of Nottingham in the UK. They started smashing machinery and equipment and destroyed much of the factory before fleeing. These men had been angry and upset for some time. They had issued manifestos and sent threatening letters, but on this day, they took the even more radical step of destroying the machines in this factory. Why were they so upset?

Well, to sum it up in a word: automation.

These men were skilled artisans. They were weavers and textile workers who had dedicated their lives to the craft of making clothes, and they did not like the new machines that were able to make cloth much more quickly and cheaply than they ever could. They claimed to be following the orders of a man named Ned Ludd and called themselves Luddites. This term has entered our modern vocabulary as a description of those who dislike new technology, but the origin of it goes back to the protest movement in which artisans were afraid of losing their jobs to automation.

The debate sounds similar to some of the debates in our society right now, although thankfully, we don't seem to have people going around and destroying property in relation to this issue. But the fact is, like it or not, our society has been on a steady path of increased automation for the past 200 years. We have added more and more automation to our factories, but we didn't stop there. We now live in a world that is saturated with cheap and accessible computing power. We have used this power to automate many things in life – I am typing this up on a computer, for example – and we continue to find more and more things to automate.

I rather enjoy the philosophy of automation – when is it helpful and when is it not? But that discussion is outside the scope of this book. What I want to focus on in this chapter is how we can best use test automation. The Luddites framed the debate in terms of a conflict between artisans and automation. I want to reframe that debate a bit. I want to help you become an artisan of automation. Automation is here to stay, and I think that, overall, is a good thing. It allows us to extend our capabilities as humans and to do more good than we could have otherwise, but that doesn't mean that all automation is good.

Automating something isn't just a matter of taking a manual task and turning it into a set of algorithmic steps that a computer can do. It requires careful consideration to ensure that it is actually helpful. Doing the wrong thing faster isn't helpful; it's destructive. Creating automation that takes more work to maintain than the work it saves isn't doing any good either. There are plenty of ways to create bad automation. In this chapter, I want to help you understand how to create good automation.

This chapter will lay some foundations that will be built on a lot in future chapters, but you will still find that you have a lot of practical skills to use by the time you get through the material in here. We will be covering the following topics in this chapter:

  • Exploring and automating
  • Organizing and structuring tests
  • Creating maintainable tests

By the end of this chapter, you will be able to create useful test automation, choose what to include (or not) in automated API tests, create well-structured tests suites, use variables, understand variable scopes in Postman, use best practices when deciding where to create Postman variables, and explain the importance of logging and reporting for creating maintainable tests.

Let's get started!

Technical requirements

The code that will be used in this chapter can be found at: https //github.com/PacktPublishing/API-Testing-and-Development-with-Postman/tree/master/Chapter04.

Exploring and automating

It is easy to think of test automation as a way to quickly do the things that we could do in a manual way. There can be times where this is true. If you are doing some tedious, repetitive work that requires doing the same thing over and over again, by all means automate it if you can! However, there is danger that comes with this kind of thinking as well. If you start to think of automation as replacing what humans do, you will end up making poor automation. Manual and automated testing may have the same goal of reducing the risk of shipping bugs to clients, but the way that they achieve this goal is radically different.

For example, a manual test that is meant to help you figure out if a change to the API has caused issues will be guided by what you, as the tester, know about the change you've made. It will also be influenced by the tester themselves. If you are familiar with the API, you will try and observe different things than someone who is not. You will notice things and possibly even slightly alter what you are doing. Even if you are trying to follow a strict script (which is not a form of manual testing I would generally encourage), there will be variations and changes in what you, as the tester, do from one time to the next. There might be minor changes in the system that you know are fine and you just skip over them. There might be other things that you see that aren't related to the change at all that you make note of. Even when scripted, testing being performed by humans in this way always has an element of exploration built into it.

On the other hand, an automated test that is meant to help you figure out if a change to the API has caused issues, will do exactly the same things it did last time and check all the same things in the same way. It will not notice any incidental changes that might be problematic. If there are minor changes that don't really matter, it won't know that and might fail for the wrong reason. Compared to exploratory testing, automated tests are less comprehensive and more brittle. But it's not all bad news. There is a reason we value automated tests as an industry. They can be much faster to run and can look at variations and details that it can be hard to see when you're doing exploratory testing.

I have summarized the differences between exploratory (or human) testing and automated (or machine) testing in the following table:

Table 4.1 – Exploratory/automated testing comparison

Table 4.1 – Exploratory/automated testing comparison

A good test strategy will include a good mix of automated and exploratory tests, but even when you're using a good balance of the two, it is important to understand how to create good test automation. By knowing the benefits and downfalls of automated testing, how do you go about making good test automation that will help reveal issues?

Exercise – considerations for good API test automation

In the next section, I will go over what I think are some approaches that you can use to create good test automation, but don't just take my word for it. Before you read through the next section, pause for a moment and think about what things you think are important for good test automation.

Write down a list of some factors (maybe two or three of them) that you think would be important when creating test automation. After reading through the next section, compare your list to the things I will be talking about. What is the same? What is different? What do you think of the differences? I will be sharing a few considerations here, but by no means are they the only possible things to think about. Entire books have been written on test automation. The quick review I will provide here certainly does not cover everything.

Writing good automation

When it comes to writing good test automation, you need to focus on a couple of things. The most important thing is to understand what you are trying to do. What kind of application are you testing and where might things go wrong? How bad is it if certain things go wrong? How often do you need to check certain things? All these things can help you make the decision of testing with an automated script or doing so by hand.

Once you have decided on the approach, though, there are a couple of other things to keep in mind. This book is about API testing, and APIs lend themselves very well to test automation. They are usually designed to be interacted with programmatically, but that doesn't mean there is nothing to think about when you're automating them. If you look at the benefits and drawbacks of using test automation in the preceding table, you can see some of the considerations we should be thinking about when it comes to test automation.

Automated testing is more brittle than manual testing. If you have ever worked with UI testing frameworks (for example, Selenium tests), you will be aware of this fact. Tests break – often for reasons that have nothing to do with finding bugs. Lower-level testing, such as calling API endpoints, has less dependencies and so are a bit less brittle, but they still break. Good test automation will create tests that are easy to fix when something changes. Software changes and good test automation will take that into account. I'll dive a bit deeper into how you can create maintainable tests later in this chapter.

Another important factor in creating good test automation is being organized. One of the benefits of test automation is that it can run the same tests over and over again. Automation can run much more quickly than a manual test, and computers don't get bored of doing repetitive tasks. You want to be able to leverage this power in your tests. In order to do this, you will need to have some kind of reporting in place that lets you know if there are test failures. You want these reports to be actionable so that they can help you pinpoint where there might be bugs, or where you might need to fix tests. This is much easier to do if your tests are well-structured.

Types of API tests

There are several types of API tests that lend themselves well to automation. Contract testing, which we will dive more deeply into in Chapter 13, Using Contract Testing to Verify an API, is a powerful form of API testing that you can use to verify that your API is meeting a given contract, which is usually specified in some sort of specification language such as OpenAPI.

You can also create integration or workflow style API tests where you are testing that you can accomplish a certain workflow. This kind of testing would typically involve calling multiple endpoints. For example, you might POST something to create a new item and then call another endpoint to verify that that item has been correctly created, and is also accessible in other parts of the system. Another example would be to GET an object such as a shopping cart and then use information in that response to GET some related information, such as details about the products in the cart.

Of course, you can also just create simple endpoint tests that call an endpoint with various kinds of inputs and verify the results. One important thing to consider here isn't so much a type of API testing as it is a style of test that should be run. You should check some negative test scenarios. What happens if you put in bad values or incorrect data? Like any good testing, API tests should not just consider what happens when things go right. They should also look at what happens when things go wrong.

One main category of tests where things can "go wrong" is security testing. I will talk a bit more about security testing in Chapter 5, Understanding Authorization Options, when I get into authorization and authentication, but it is an important style of testing to consider in APIs. APIs expose a lot of power and flexibility to end users (and that is partly why many companies make them), but in doing so, it is easy to miss things that could allow user access to things they should not have. Many security breaches in recent history have come from problems in APIs. I can't dive too deep into that topic in this book, but I will show you some basic things that you can do in Postman to check for API security.

Another type of testing that is worth mentioning here is performance testing. Postman isn't really designed as a performance testing tool, but you can get some basic performance data out of it. I will not be talking much about performance in this book, but it is an idea worth keeping in mind as you consider API quality. Performance can have a big impact on the customer's perception of quality, and APIs can often be performance bottlenecks as they involve network latency and other difficult to control factors.

As you can see, there are a lot of different ways to approach testing an API. One of the most important things you can do when trying to create good test automation is to be carefully structured in how you go about creating these tests.

Organizing and structuring tests

As the saying goes, an ounce of prevention is worth a pound of cure. This is good general life advice and is also true in test automation. Taking a bit of time to structure and organize your tests when you are starting will save you a lot of time later, when you are trying to understand test failures or reports on test effectiveness. Postman understands this philosophy and makes it easy to keep tests well-organized.

It is too easy to spout off a bunch of theory that you will skim over and not fully understand. In order to keep this practical, I will try to walk through a concrete example. I will once again use the Star Wars API for this (https://swapi.dev/). So, how would you go about structuring the tests for this API?

Creating the test structure

Let's start with the obvious thing – collections. One way you can think of a collection is as a folder that you can collect items, such as other folders and tests. You may already have a Star Wars API collection if you did the case study in Chapter 1, API Terminology and Types, but if not, I have provided a collection in the GitHub repository for this course. You can import that collection into Postman by following these steps:

  1. Go to this book's GitHub repository (https://github.com/PacktPublishing/API-Testing-and-Development-with-Postman) and click on the Code drop - down arrow and download or clone the code.
  2. Go to the File menu in the Postman application and choose the Import option.
  3. Click on Upload Files and navigate to where you downloaded the files from GitHub. Then, go to the Chapter04 folder.
  4. Choose the Stars Wars API_Chapter4_initial.postman_collection.json file and click Open and then Import.

This will create a collection for you that you can use for the rest of this section. You can find the collection in the collections navigation tree in Postman.

This API has six different resources. We could just create requests directly in the newly created collection for each of them, but that would get a bit messy. For example, the /people resource gets a list of all the people in the service, but you can also get information about specific people. If we want to be able to test different things, we will need to include several different calls to the /people resource.

Instead of creating a request for this resource, I will create a folder in the Star Wars API collection for the people resource. I can do this by clicking on the View more actions menu beside the collection and choosing the Add Folder option from that menu. I will then name that folder People. I can repeat this for the rest of the resources in this API (Films, Starships, Vehicles, Species, and Planets):

Figure 4.1 – Adding a folder to a collection

Figure 4.1 – Adding a folder to a collection

Now, with folders in place for each endpoint, I want to stop and think about what type of tests we are creating here. Although this service does provide a schema for the different resources that I could use to verify some of the results, it doesn't provide the kind of specification that I would need to create contract tests, so I won't worry about that kind of testing for now. I also won't worry about performance and security testing on this API. That leaves integration testing and endpoint testing. We will want to verify the various endpoints, so I will want endpoint tests. There are no direct workflows that we need to check here from a business sense, since this is just a "for fun" application, but there are still some interesting workflows that we might want to consider. For example, each of the resources cross-links to other resources. A call to a /planets endpoint gives links to people that live on that planet, along with other links. Each resource does this and provides links that point to other resources. This kind of linking creates some interesting possibilities for testing. You could check a planet's resources and see that it points to certain people, and you can check those people links to ensure that they have references to that planet. Tests like this could be a simple way to validate data integrity.

It seems like we want to create both kinds of tests, so the next thing I will do is create a folder in the collection called Integration Tests. Taken as a whole, these folders provide the basic structure for how we want to approach testing this API, but how do we organize these tests?

Organizing the tests

The first thing you need to do is to put the endpoint tests for each resource in the appropriate folder. I will start with the People folder. Click on the menu beside the folder and choose the Add Request option:

Figure 4.2 – Adding a request to a folder

Figure 4.2 – Adding a request to a folder

Name the request Get All People, click on the request in the tree, and set the request URL to https://swapi.dev/api/people/. This endpoint gives back the list of all people. In Chapter 6, Creating Test Validation Scripts, I will show you how to create checks to validate this data, but for now, I'll just focus on how to organize these tests. Each person in this list has their own endpoint, and we could create a request for each one. However, doing that could get very unwieldy. Instead, I will just sample one of the individual people endpoints. Once again, add a request to the folder and name this one Get Person Details. Set the request URL to https://swapi.dev/api/people/{{peopleId}}. Note that in this case, I have set the ID to be a variable enclosed in double curly braces. This is because I don't want to hardcode a particular value, but rather want to be able to try different characters at different times.

When it comes to organizing tests, sharing variables between them is very important. You have typed in a variable, but when you hover your mouse over it, you will see that it is an unresolved variable:

Figure 4.3 – Unresolved variable

Figure 4.3 – Unresolved variable

This warning is here because, by surrounding it with double curly braces, you told Postman that you want to use this peopleId variable, but you have not assigned that variable any value. There are a few different places in Postman where you can store variables. The first place is in the collection itself. These are aptly named collection variables, and we will discuss them in more detail later. In this case, I want you to save this variable as an environment variable, so let's look at how we can do that now.

Environments

In order to create an environment variable, you will need an environment that you can save the variable in. You can create a new environment in Postman by following these steps:

  1. Click on the New button and then choose the Environment option on the resulting popup.
  2. Name it SWAPI Environment.
  3. Add a new variable called peopleId with an initial value of 1:
    Figure 4.4 – Adding a new environment

    Figure 4.4 – Adding a new environment

  4. Click on Save to add this new environment and then close the Manage Environments tab.

Now that you have the variable defined in an environment, return to the Get Person Details request. The variable will still be unresolved, and that is because you need to tell Postman which environment you want to use. To do that, click on the dropdown near the top-right of the screen and choose SWAPI Environment from the list:

Figure 4.5 – Choosing an environment

Figure 4.5 – Choosing an environment

Now that you have set the environment, if you mouse over the peopleId variable, you should see that it is now set to 1:

Figure 4.6 – Variable defined

Figure 4.6 – Variable defined

Environments are a great way to organize data for your tests. In Chapter 6, Creating Test Validation Scripts, we will cover the different ways that we can share data and manipulate variables in the environment, which will make the concept even more powerful. Environments allow you to manage your variables in one place. This is very helpful if you need to make changes, or just need to understand what variables you have set up. Instead of looking through every endpoint to see what might be defined and trying to decide if you can change things in them, you can collect all the variables in one place and see what you can change at a glance.

Collection variables

I mentioned earlier that you can also store variables in the collection, so let's look at how that works. You have created a couple of requests already and each of them starts with the base URL https://swapi.dev/api. Instead of repeating that on every request that you put in the collection, let's turn it into a variable:

  1. Go to each of the requests that you created earlier and replace the https://swapi.dev/api URL with a variable called {{baseUrl}}.
  2. In the collection navigation pane, go to the View more actions menu for the collection and choose the Edit option.
  3. Go to the Variables tab and create the baseUrl variable, setting the initial value of it to the base URL (https://swapi.dev/api).
  4. Update the collection.

The variable will now be defined in all the requests that you are using it in. You can also use that variable when defining any new requests for this API.

Choosing a variable scope

You now know of two different places you can store variables in Postman (environments and collections). There are a couple of other places where you can store variables (for example, you can make global variables), but how do you decide where to store variables? How would you decide if you should put a variable into an environment, or in a collection, or somewhere else? Well, in order to answer that question, let's look at the different places where you can have variables in Postman and understand how they relate to each other.

How scopes work

When you store a variable in a collection or an environment, Postman will give that variable a scope that corresponds to where you have stored it. When you're trying to resolve a variable, Postman will look for it in a defined order in each of the scopes. Scopes allow a variable to be used in broader or narrower parts of the application. If you define a variable in the global scope, you can use it in any request anywhere in the application, but if you define it at the narrowest scope, it will only be available during one particular iteration of a test. So, what are the available variable scopes in Postman? In order from the broadest to the narrowest scopes, they are as follows:

  • Global
  • Collection
  • Environment
  • Data
  • Local

When resolving a variable, Postman will use the narrow scope in which that variable exists. Let's take a look at a concrete example illustrating this.

You have already created the baseUrl variable in the Collection scope. Now, we need to create a variable with the same name in the SWAPI Environment. You can do that by following these steps:

  1. Click on the eye icon (Environment Quick Look) beside the environment dropdown.
  2. Choose the Edit option to make changes to the environment:
    Figure 4.7 – Editing an environment

    Figure 4.7 – Editing an environment

  3. Add a new variable called baseUrl and give it silly value such as bob.
  4. Update the environment.
  5. Go to one of the requests and mouse over the baseUrl variable.

You can see that it now has a value of bob, and you can also see that the scope of this variable is Environment. Even though, in the collection, the variable with the same name contains the site URL, the value being used comes from the variable defined in the environment since an environment is a narrower scope than a collection:

Figure 4.8 – Variable values coming from the narrowest scope

Figure 4.8 – Variable values coming from the narrowest scope

Don't forget to go back into the environment and clean things up by deleting the silly variable that you made! To do that, edit the environment again and then click on the X that shows up beside the variable when you mouse over the field:

Figure 4.9 – Deleting a variable

Figure 4.9 – Deleting a variable

You now understand how Postman determines which value to use when variables are defined in different scopes, but how do you know which scope to create a variable in? Let's do a quick overview of the different scopes and discuss when you would want to use each of them.

Global scope

The first scope we will discuss is the Global scope, which defines variables that are accessible globally or anywhere in the application.

You may want to use global variables when experimenting with sharing data between different tests or collections, but in general, you want to try and avoid using global variables. They are, by definition, available everywhere and if you use them, it is likely that you will end up eventually giving a variable in another scope the same name as the global variable. Postman will, of course, use the narrower scope, but this can still lead to confusion, especially when you're trying to debug failures.

You can add and edit variables in the global scope by using Environment Quick Look, and then choosing the Edit option in the Globals section:

Figure 4.10 – Editing global variables

Figure 4.10 – Editing global variables

Variables in the global scope are the most widely available and the last ones that Postman uses when trying to resolve a variable value. Let's continue with the slightly narrower collection scope.

Collection scope

Variables defined in the collection are available for all the requests in that collection. They are not available in other collections, but any request in that collection can use them. You want to create collection variables when you need to share the data between multiple requests in the collection, and that data will stay the same. An example of this is an application URL. Every request will need it, and it should stay consistent. Another example would be doing some setup work with a pre-request script (I will show you how to do this in Chapter 6, Creating Test Validation Scripts) and then storing that data in a collection variable.

Environment scope

Environments allow you to define a set of variables that belong to a certain context. For example, many companies will have testing sites and production sites. You might also have a staging environment or want to run tests on a local build. Each of these environments might require some minor changes needing to be made to the variables being used, so in these cases, you would want to create variables that live in the different environments.

For example, if you had staging, testing, and local sites that you wanted to test against, you might define an environment in Postman for each of those sites, and then in each of those environments create a URL variable that points to the correct endpoint for each of those environments. It is important to note that in these cases, you should try not to also have the same variable defined in the collection. This will lead to confusion and is not a good practice.

Another time you might store variables in an environment is if you want to try some API endpoints as different users. You might want to try as an administrator and a regular user, for example, and so you could create environments for each of those users. These will store the login information for them and maybe some specific variables that store expected results for those different users.

If you find yourself with only one environment for a collection, you are probably better off just creating variables directly in the collection instead of adding the additional complexity of an environment.

Data scope

Most variables will be defined in collections and environments. However, in Chapter 7, Data-Driven Testing, I will show you how to do data-driven testing in Postman. This kind of testing imports data into Postman from a CSV or JSON file. Postman will create variables in the data scope from this imported data. You cannot create these kinds of variables yourself within Postman, but you should be aware of the kinds of variables that you have defined in your collections or environments when you are creating your input data files. The scope of data variables is narrower than that of collections or environments, so any variables defined in this scope will overwrite values defined in those broader scopes.

Local scope

The narrowest scope you can define variables in in Postman is the local scope. You can only create variables in this scope using request scripts (which I will talk about in Chapter 6, Creating Test Validation Scripts). They are temporary variables that do not persist between sessions, but since they are the narrowest scope, they do allow you to override values that have been set in collections or environments. Sometimes, this is desirable as it lets you override a variable in one request while still using it in all the other requests in a collection, but you should be careful that you are not accidentally overriding them either.

Variables and the different scope that they can be stored in give you a lot of power, control, and flexibility when it comes to creating and organizing tests. They also help with creating tests that continue to be valuable over time.

Exercise – using variables

Hopefully, at this point, you have a pretty good grasp of the various variable scopes, but I want you to try this out, both to cement these concepts in your mind and to prove to yourself the order in which Postman resolves the scopes:

  1. Create a global variable called base_url and set its value to https://swapi.dev/api.

    What are some potential problems with this?

  2. Create a new collection called jsonplaceholder.
  3. Add a request to that collection and set the URL of the request to {{base_url}}/todos/1.
  4. Mouse over the variable in the URL field.

    What value does it resolve to?

  5. Create an environment called jsonplaceholder Env.
  6. Add a variable to the environment called base_url and give it a value of https://jsonplaceholder.typicode.com/.
  7. Go to the request you made in the jsonplaceholder collection and ensure that the jsonplaceholder env environment is selected.
  8. Mouse over the base_url variable in the URL.

    What value does the variable resolve to? Why?

Effective use of variables is an important factor in test automation. Take some time and make sure that you understand this. Good management of this will help you create maintainable tests, but there are other factors that are important as well. Let's look at some of them.

Creating maintainable tests

One of the things that frequently gets forgotten in conversations about test automation is that they take time and work to maintain. The "sales pitch" for test automation is that we can run the same test over and over again "for free" but of course, that is not true. Leaving aside the hardware and software costs of running these tests, there are maintenance costs. Tests don't always pass. Sometimes, failures are due to finding a bug, but other times, it is just because the code has changed and the test needs to be updated, or because of some kind of flakiness in the system that we don't need to worry about too much. Well-written tests take these considerations into account. They assume that there will be failures in the future that need to be debugged. So, how do you set yourself up to make sure that your tests are maintainable?

Using logging

One of the ways in which you can make it easier to figure out failures is by having good logging options set up in your tests. You can view the logs in Postman using the console. You can either open a standalone console, by going to the View menu and choosing the Show Postman Console option, or you can open an in-app console by using the Console icon at the bottom of the application:

Figure 4.10 – Editing global variables

Figure 4.11 – Opening the Postman Console

The console will show you further details about the requests, along with any errors or warnings. It is also the place where any console.log() commands that you set up in your scripts will print to.

Maintainable tests also include some things that I have already talked about, such as writing good documentation and organizing your tests in a logical way, but another consideration for writing maintainable tests is having good test reports.

Test reports

Any automated test system needs to report on the results in some way so that you can find out if there are any issues to investigate. Postman will report on results directly in the application or command line. Many of the reporters built into Postman require that you have an Enterprise or Business plan.

However, there are some free reporting tools available, including a number of third-party reporters. newman-reporter-htmlextra (https://github.com/DannyDainton/newman-reporter-htmlextra), written by Danny Dainton, is probably the best one available. This reporter works with the Postman command-line runner known as Newman, which I will show you how to use in Chapter 8, Running API Tests in CI with Newman. In that chapter, I will also show you how to use the test reporter, but for now, keep in mind that good test reporting will help with creating maintainable tests.

Summary

This chapter has been filled with ideas on how to create long-lasting and valuable test automation. Over the next few chapters, I will take a lot of the foundations we've laid here and show you how to use various features in Postman. This will help you put the topics from this chapter into practice in your testing.

You have learned the strengths that test automation brings to the table and how to use those strengths to your advantage when creating a test strategy. You have also learned about different types of API tests that you can create, including contract tests, integration tests, endpoint tests, and performance and security tests.

I also showed you how to use Postman to organize and structure variables in ways that will help you understand what an API is doing and what you might need to do if tests fail. I also showed you how to use variables in Postman and where to store them. You now know which scope to use for different kinds of variables and understand how Postman will resolve variables values if there are conflicts between those scopes.

In addition, you have learned the importance of logging and test reporting when it comes to creating API tests. I gave you a few exercises to work on, but the truth is that this information might still feel a little abstract at this point, and that is ok. These are important principles to understand. I will build on them and make them much more concrete as we progress through this book. You will see how important these foundations are and how to use them in practice as you learn more about Postman.

In the next chapter, we will look at some concrete ways that you can use Postman to do authorization in APIs. We will see how authorization works in APIs, as well as how to set up and use various authorization options within Postman itself.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
44.193.29.184