In the previous chapter, you learned how to set up the Python 3 environment on Linux, macOS, and Windows computers. You also looked at a few popular IDEs for Python. In this chapter, you will get started with concepts of test automation. Then you will explore a light and easy way to learn the test automation framework in Python 3, called doctest.
A Brief Introduction to Software Testing Concepts
The textbook definition of software testing states that it’s the process of executing a program or application to find any bugs. Usually, there are multiple stakeholders in the process of software testing. The stakeholders include testers, the management team, consultants, business, customers, and end users. With medium- to large-scale projects, software testing is done to determine if the software behaves as intended under various sets of inputs and conditions.
Unit Testing
Unit testing is a software testing method in which individual components of the program, called units, are tested independently with all the required dependencies. Unit testing is mostly done by programmers, who write the programs for the units. In smaller projects, it is done informally. In most very large-scale projects, unit testing is part of a formal process of development, with proper documentation and proper schedule/efforts allocated to it.
Test Automation
Test automation is the automated execution and reporting of the outcome of test scenarios and cases. In most large and complex projects, many phases of the testing process are automated. Sometimes the effort of automating tests is so huge that there is a separate project for automation with a separate team dedicated to it, including a separate reporting structure with separate management. There are several areas and phases of testing that can be automated. Various tools like code libraries and third-party APIs are used for unit testing. Sometimes, the code for unit testing is also generated in an automated way. Unit testing is a prime candidate for automation.
The Benefits of Automated Unit Testing
There are many reasons to automate unit tests. Let’s consider them one by one.
Time and effort
As your codebase grows, the number of modules to be unit tested grows. Manual testing is very time-consuming. To reduce manual testing efforts, you can automate test cases, which then can be automated easily and quickly.
Accuracy
Test case execution is a rote and boring activity. Humans can make mistakes. However, an automated test suite will run and return correct results every time.
Early bug reporting
Automating unit test cases gives you the distinct advantage of early reporting of bugs and errors. When automated test suites are run by the scheduler, once the code freezes due to an error, all the logical bugs in the code are quickly discovered and reported, without much human intervention needed.
Built-in support for unit testing
There are many programming languages that provide built-in support for writing unit tests by means of libraries dedicated to unit testing. Examples include Python, Java, and PHP.
Using Docstrings
The focus of this chapter is on getting you started with unit test automation in Python. Let’s get started with the concept of docstrings and their implementation in Python. Docstrings are going to be immensely useful to you while learning doctest.
A docstring is a string literal that’s specified in the source code of a module. It is used to document a specific segment of code. Code comments are also used for documenting source code. However, there is a major difference between a docstring and a comment. When the source code is parsed, the comments are not included in the parsing tree as part of the code, whereas docstrings are included in the parsed code tree.
The major advantage of this is that the docstrings are available for use at runtime. Using the functionalities specific to the programming language, you can retrieve the docstring specific to a module. Docstrings are always retained through the entire runtime of the module instance.
Example of a Docstring in Python
Let’s see how the concept of the docstring is implemented in Python. A Python docstring is a string literal that occurs as the first statement in a module, function, class, or method definition. A docstring becomes the doc special attribute of that object.
Let’s take a look at a code example of a Python docstring. From this chapter onward, you will be programming quite a lot. I recommend that you create a directory on your computer and create chapter-specific subdirectories within it. As mentioned earlier, I am using a Linux OS. (My favorite computer, a Raspberry Pi 3 Model B.) I have created a directory called book and a directory called code under that. The code directory has chapter-specific directories containing the code of each chapter. Figure 2-1 shows a graphical representation of the directory structure in the form of a tree diagram.
Figure 2-1
The suggested directory structure for the book
Create chapter-specific subdirectories under the directory code, as shown in the tree diagram in Figure 2-1. We use the directory chapter02 for this chapter, chapter03 for the next chapter, and so on. Navigate to the chapter02 directory and save the following code (see Listing 2-1) as test_module01.py in that directory.
"""
This is test_module01.
This is example of multiline docstring. """
class TestClass01:
"""This is TestClass01."""
def test_case01(self):
"""This is test_case01()."""
def test_function01():
"""This is test_function01()."""
Listing 2-1
test_module01.py
In Listing 2-1, there is a test file called test_module01.py, which includes TestClass01 and test_function01(). TestClass01 has a method called test_ case01(). There is a docstring for all code units here. The first docstring is a multiline docstring. The rest are examples of single-line docstrings.
Let’s see how the docstrings work using the code in Listing 2-1 and an interactive Python session.
Navigate to the chapter02 directory and type python3 to invoke Python 3 in interpreter mode.
pi@raspberrypi:~/book/code/chapter02 $ pwd
/home/pi/book/code/chapter02
pi@raspberrypi:~/book/code/chapter02 $
python3 Python 3.4.2 (default, Oct 19 2014, 13:31:11)
[GCC 4.9.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
Import the test module you just created with the following statement:
>>> import test_module01
You can use the help() function to see the docstrings of the module and its members, as follows.
| list of weak references to the object (if defined)
FUNCTIONS
test_function01()
This is test_function01().
FILE
/home/pi/book/code/chapter02/test_module01.py
You can see the docstring of the individual members using help(). Run the following statements and see the output for yourself.
>>> help(test_module01.TestClass01)
>>> help(test_module01.TestClass01.test_case01)
>>> help(test_module01.test_function01)
As mentioned earlier, a docstring becomes the doc special attribute of that object. You can also use the print() function to see the docstring of a module and its members. The following interactive Python session demonstrates that.
In the next section, you learn to use docstrings to write simple test cases and execute them with doctest.
A Brief Introduction to doctest
doctest is the lightweight unit-testing framework in Python that uses docstrings to test automation. doctest is packaged with the Python interpreter, so you do not have to install anything separately to use it. It is part of Python’s standard library and adheres to Python’s “batteries-included” philosophy.
The code in Listing 2-2 is a simple example of a test module with two functions and two tests for each function.
"""
Sample doctest test module... test_module02
"""
def mul(a, b):
"""
>>> mul(2, 3)
6
>>> mul('a', 2)
'aa'
"""
return a * b
def add(a, b):
"""
>>> add(2, 3)
5
>>> add('a', 'b')
'ab'
"""
return a + b
Listing 2-2
test_module02.py
In Listing 2-2, the test cases are mentioned as docstrings for the modules and there is nothing specifically calling the doctest in the code itself. When the program is executed as a Python 3 program using the python3 test command, _module02.py does not produce any output at the command line. In order to see doctest in action, you have to run it using the following command at the command prompt:
python3 -m doctest -v test_module02.py
The output will be as follows:
Trying:
add(2, 3)
Expecting:
5
ok
Trying:
add('a', 'b')
Expecting:
'ab'
ok
Trying:
mul(2, 3)
Expecting:
6
ok
Trying:
mul('a', 2)
Expecting:
'aa'
ok
1. items had no tests:
test_module02
2. items passed all tests:
2 tests in test_module02.add
2 tests in test_module02.mul
4 tests in 3 items.
4 passed and 0 failed.
Test passed.
Let’s take a look at how doctest works. By comparing the code—specifically the commands for execution and output—you can figure out quite a few things. doctest works by parsing docstrings. Whenever doctest finds an interactive Python prompt in the doctest documentation of a module, it treats its output as the expected output. Then it runs the module and its members by referring to the docstrings. It compares the actual output against the output specified in the docstrings. Then it marks the test pass or fail. You have to use -m doctest while executing the module to let the interpreter know that you need to use the doctest module to execute the code.
The command-line argument -v stands for verbose mode. You must use it because, without it, the test will not produce any output unless it fails. Using verbose produces an execution log irrespective of whether the test passes or fails.
Failing Tests
In Listing 2-2, all the tests passed with no hassles. Now, let’s see how a test fails. In Listing 2-2, replace + on the last line of the code with an * (asterisk) and run the test again with the same command. You will get the following output:
You can clearly see two failures in the execution log. The tests usually fail due to one or more of the following reasons:
Faulty logic in the code
Faulty input into the code
Faulty test case
In this case, there are two failures in the test. The first one is due to faulty logic. The second failure is due to faulty logic in the code and the wrong type of input given to the function to be tested.
Correct the code by replacing the * in the last line with +. Then change the line that has 'aa' to aa and run the test again. This will demonstrate the third cause of test failure (a faulty test case).
Separate Test File
You can also write your tests in a separate test file and run them separately from the code to be tested. This helps maintain the test modules/code separately from the development code. Create a file called test_module03.txt in the same directory and add the code shown in Listing 2-3 to it.
>>> from test_module02 import *
>>> mul(2, 3)
6
>>> mul('a', 2)
'aa'
>>> add(2, 3)
5
>>> add('a', 'b')
'ab'
Listing 2-3
test_module03.txt
You can run this test in the usual way, by running the following command in the command prompt:
python3 -m doctest -v test_module03.txt
The output will be as follows:
Trying:
from test_module02 import *
Expecting nothing
ok
Trying:
mul(2, 3)
Expecting:
6
ok
Trying:
mul('a', 2)
Expecting:
'aa'
ok
Trying:
add(2, 3)
Expecting:
5
ok
Trying:
add('a', 'b')
Expecting:
'ab'
ok
1 items passed all tests:
5 tests in test_module03.txt
5 tests in 1 items.
5 passed and 0 failed.
Test passed.
Advantages and Disadvantages of doctest
As you have learned, doctest is a very simple and intuitive framework for novice-level testing in Python. It does not require any installation and you can quickly get started with it without needing to know any API. It is mostly used for the following purposes:
To verify if the code documentation is up to date and the interactive examples in the docstring still work after making changes to the code.
To perform module-wise basic regression testing.
To write illustrative tutorials and documentation that doubles as a test case for the package and module.
However, doctest has its own set of limitations. It does not have true API for testing.
doctest tests also tend to be static in nature and cannot be parameterized.
Just like doctest, there is another useful utility to view the documentation of a module. It comes with Python. It is known as Pydoc. On Linux, run the following command:
pydoc unittest
It will show the documentation of the unittest library. If you have created the documentation for your own custom module with docstrings, you can view it with the following command:
pydoc test_module01
This command displays the documentation on the terminal. You can save all this information in HTML files as follows:
pydoc -w unittest
pydoc -w test_module01
These commands will create unittest.html and test_module01.html documents in the directory where the commands are run. You can then open these files with a web browser of your choice.
On Windows, the commands can be run as follows:
python -m pydoc unittest
python -m pydoc -w unittest
Conclusion
In this chapter, you learned the basics of software testing. You explored a light testing framework, called doctest. It’s a good module for simple projects for novice Python users. However, due to its lack of advanced features like testrunner, test discovery, and test fixtures, doctest is not used in large projects. The next chapter discusses a built-in xUnit style test automation framework for Python, called unittest.
