The only real difference between an integration test and a unit test is that, in an integration test, you can break the code being tested into smaller meaningful chunks; in a unit test, however, if you divided the code any more, it wouldn't be meaningful. For this reason, the same tools that help you automate unit testing can be applied to integration testing. Since system testing is really the highest level of integration testing, the tools can be used for that as well.
The role of doctest in integration testing tends to be fairly limited: doctest's real strengths are in the early part of the development process. It's easy for a testable specification to stray into integration testing—as said before, that's fine as long as there are unit tests as well, but after that it's likely that you'll prefer unittest and Nose for writing your integration tests.
Integration tests need to be isolated from each other. Even though they contain multiple interacting units within themselves, you still benefit from knowing that nothing outside the test is affecting it. For this reason, unittest is a good choice for writing automated integration tests.
The integration diagram only provides a partial ordering of the integration tests, and there are several tests that could be the first one we write. Looking at the diagram, we can see that the Status and Activity classes are at the end of a lot of arrows, but not at the beginning of any. This makes them particularly good places to start writing integration tests, because it means that they don't call on anything outside themselves to operate. Since there's nothing to distinguish one of them as a better place to start than the other, we can choose between them arbitrarily. Let's start with Status, and then do Activity. We're going to write tests that exercise the whole class. At this low level, the integration tests will look a lot like the unit tests for the same class, but we're not going to use mock objects to represent other instances of the same class. We will use real instances. We're testing whether the class correctly interacts with itself.
Here is the test code for Status:
from unittest import TestCase
from planner.data import Status
from datetime import datetime
class statuses_integration_tests(TestCase):
def setUp(self):
self.A = Status('A',
datetime(year = 2012, month = 7, day = 15),
datetime(year = 2013, month = 5, day = 2))
def test_repr(self):
self.assertEqual(repr(self.A), '<A 2012-07-15T00:00:00 2013-05-02T00:00:00>')
def test_equality(self):
self.assertEqual(self.A, self.A)
self.assertNotEqual(self.A, Status('B',
datetime(year = 2012, month = 7, day = 15),
datetime(year = 2013, month = 5, day = 2)))
self.assertNotEqual(self.A, Status('A',
datetime(year = 2011, month = 7, day = 15),
datetime(year = 2013, month = 5, day = 2)))
self.assertNotEqual(self.A, Status('A',
datetime(year = 2012, month = 7, day = 15),
datetime(year = 2014, month = 5, day = 2)))
def test_overlap_begin(self):
status = Status('status name',
datetime(year = 2011, month = 8, day = 11),
datetime(year = 2012, month = 11, day = 27))
self.assertTrue(status.overlaps(self.A))
def test_overlap_end(self):
status = Status('status name',
datetime(year = 2012, month = 1, day = 11),
datetime(year = 2014, month = 4, day = 16))
self.assertTrue(status.overlaps(self.A))
def test_overlap_inner(self):
status = Status('status name',
datetime(year = 2011, month = 10, day = 11),
datetime(year = 2014, month = 1, day = 27))
self.assertTrue(status.overlaps(self.A))
def test_overlap_outer(self):
status = Status('status name',
datetime(year = 2012, month = 8, day = 12),
datetime(year = 2012, month = 9, day = 15))
self.assertTrue(status.overlaps(self.A))
def test_overlap_after(self):
status = Status('status name',
datetime(year = 2015, month = 2, day = 6),
datetime(year = 2019, month = 4, day = 27))
self.assertFalse(status.overlaps(self.A))Here is the test code for Activity:
from unittest import TestCase
from planner.data import Activity, TaskError
from datetime import datetime
class activities_integration_tests(TestCase):
def setUp(self):
self.A = Activity('A',
datetime(year = 2012, month = 7, day = 15),
datetime(year = 2013, month = 5, day = 2))
def test_repr(self):
self.assertEqual(repr(self.A), '<A 2012-07-15T00:00:00 2013-05-02T00:00:00>')
def test_equality(self):
self.assertEqual(self.A, self.A)
self.assertNotEqual(self.A, Activity('B',
datetime(year = 2012, month = 7, day = 15),
datetime(year = 2013, month = 5, day = 2)))
self.assertNotEqual(self.A, Activity('A',
datetime(year = 2011, month = 7, day = 15),
datetime(year = 2013, month = 5, day = 2)))
self.assertNotEqual(self.A, Activity('A',
datetime(year = 2012, month = 7, day = 15),
datetime(year = 2014, month = 5, day = 2)))
def test_overlap_begin(self):
activity = Activity('activity name',
datetime(year = 2011, month = 8, day = 11),
datetime(year = 2012, month = 11, day = 27))
self.assertTrue(activity.overlaps(self.A))
self.assertTrue(activity.excludes(self.A))
def test_overlap_end(self):
activity = Activity('activity name',
datetime(year = 2012, month = 1, day = 11),
datetime(year = 2014, month = 4, day = 16))
self.assertTrue(activity.overlaps(self.A))
self.assertTrue(activity.excludes(self.A))
def test_overlap_inner(self):
activity = Activity('activity name',
datetime(year = 2011, month = 10, day = 11),
datetime(year = 2014, month = 1, day = 27))
self.assertTrue(activity.overlaps(self.A))
self.assertTrue(activity.excludes(self.A))
def test_overlap_outer(self):
activity = Activity('activity name',
datetime(year = 2012, month = 8, day = 12),
datetime(year = 2012, month = 9, day = 15))
self.assertTrue(activity.overlaps(self.A))
self.assertTrue(activity.excludes(self.A))
def test_overlap_after(self):
activity = Activity('activity name',
datetime(year = 2015, month = 2, day = 6),
datetime(year = 2019, month = 4, day = 27))
self.assertFalse(activity.overlaps(self.A))Looking at our diagram, we can see that the next level out from either Status or Activity represents the integration of these classes with the Schedule class. Before we write this integration, we ought to write any tests that involve the Schedule class interacting with itself, without using mock objects:
from unittest import TestCase
from unittest.mock import Mock
from planner.data import Schedule
from datetime import datetime
class schedule_tests(TestCase):
def test_equality(self):
A = Mock(overlaps = Mock(return_value = False))
B = Mock(overlaps = Mock(return_value = False))
C = Mock(overlaps = Mock(return_value = False))
sched1 = Schedule()
sched2 = Schedule()
self.assertEqual(sched1, sched2)
sched1.add(A)
sched1.add(B)
sched2.add(A)
sched2.add(B)
sched2.add(C)
self.assertNotEqual(sched1, sched2)
sched1.add(C)
self.assertEqual(sched1, sched2)Now that the interactions within the Schedule class have been tested, we can write tests that integrate Schedule with either Status or Activity. Let's start with Status, then do Activity.
Here are the tests for Schedule and Status:
from planner.data import Schedule, Status
from unittest import TestCase
from datetime import datetime, timedelta
class test_schedules_and_statuses(TestCase):
def setUp(self):
self.A = Status('A',
datetime.now(),
datetime.now() + timedelta(minutes = 7))
self.B = Status('B',
datetime.now() - timedelta(hours = 1),
datetime.now() + timedelta(hours = 1))
self.C = Status('C',
datetime.now() + timedelta(minutes = 10),
datetime.now() + timedelta(hours = 1))
def test_usage_pattern(self):
sched = Schedule()
sched.add(self.A)
sched.add(self.C)
self.assertTrue(self.A in sched)
self.assertTrue(self.C in sched)
self.assertFalse(self.B in sched)
sched.add(self.B)
self.assertTrue(self.B in sched)
self.assertEqual(sched, sched)
sched.remove(self.A)
self.assertFalse(self.A in sched)
self.assertTrue(self.B in sched)
self.assertTrue(self.C in sched)
sched.remove(self.B)
sched.remove(self.C)
self.assertFalse(self.B in sched)
self.assertFalse(self.C in sched)Here are the tests for the interactions between real Schedule and Activity instances. Due to the similarity between Activity and Status, the tests are, not surprisingly, structured similarly:
from planner.data import Schedule, Activity, ScheduleError
from unittest import TestCase
from datetime import datetime, timedelta
class test_schedules_and_activities(TestCase):
def setUp(self):
self.A = Activity('A',
datetime.now(),
datetime.now() + timedelta(minutes = 7))
self.B = Activity('B',
datetime.now() - timedelta(hours = 1),
datetime.now() + timedelta(hours = 1))
self.C = Activity('C',
datetime.now() + timedelta(minutes = 10),
datetime.now() + timedelta(hours = 1))
def test_usage_pattern(self):
sched = Schedule()
sched.add(self.A)
sched.add(self.C)
self.assertTrue(self.A in sched)
self.assertTrue(self.C in sched)
self.assertFalse(self.B in sched)
self.assertRaises(ScheduleError, sched.add, self.B)
self.assertFalse(self.B in sched)
self.assertEqual(sched, sched)
sched.remove(self.A)
self.assertFalse(self.A in sched)
self.assertFalse(self.B in sched)
self.assertTrue(self.C in sched)
sched.remove(self.C)
self.assertFalse(self.B in sched)
self.assertFalse(self.C in sched)All right, it's finally time to put Schedule, Status, and
Activity together in the same test:
from planner.data import Schedule, Status, Activity, ScheduleError
from unittest import TestCase
from datetime import datetime, timedelta
class test_schedules_activities_and_statuses(TestCase):
def setUp(self):
self.A = Status('A',
datetime.now(),
datetime.now() + timedelta(minutes = 7))
self.B = Status('B',
datetime.now() - timedelta(hours = 1),
datetime.now() + timedelta(hours = 1))
self.C = Status('C',
datetime.now() + timedelta(minutes = 10),
datetime.now() + timedelta(hours = 1))
self.D = Activity('D',
datetime.now(),
datetime.now() + timedelta(minutes = 7))
self.E = Activity('E',
datetime.now() + timedelta(minutes = 30),
datetime.now() + timedelta(hours = 1))
self.F = Activity('F',
datetime.now() - timedelta(minutes = 20),
datetime.now() + timedelta(minutes = 40))
def test_usage_pattern(self):
sched = Schedule()
sched.add(self.A)
sched.add(self.B)
sched.add(self.C)
sched.add(self.D)
self.assertTrue(self.A in sched)
self.assertTrue(self.B in sched)
self.assertTrue(self.C in sched)
self.assertTrue(self.D in sched)
self.assertRaises(ScheduleError, sched.add, self.F)
self.assertFalse(self.F in sched)
sched.add(self.E)
sched.remove(self.D)
self.assertTrue(self.E in sched)
self.assertFalse(self.D in sched)
self.assertRaises(ScheduleError, sched.add, self.F)
self.assertFalse(self.F in sched)
sched.remove(self.E)
self.assertFalse(self.E in sched)
sched.add(self.F)
self.assertTrue(self.F in sched)The next thing we need to pull in is the File class but, before we integrate it with the rest of the system, we need to integrate it with itself and check its internal interactions without using mock objects:
from unittest import TestCase
from planner.persistence import File
from os import unlink
class test_file(TestCase):
def setUp(self):
storage = File('file_test.sqlite')
storage.store_object('tag1', 'A')
storage.store_object('tag2', 'B')
storage.store_object('tag1', 'C')
storage.store_object('tag1', 'D')
storage.store_object('tag3', 'E')
storage.store_object('tag3', 'F')
def tearDown(self):
try:
unlink('file_test.sqlite')
except OSError:
pass
def test_other_instance(self):
storage = File('file_test.sqlite')
self.assertEqual(set(storage.load_objects('tag1')),
set(['A', 'C', 'D']))
self.assertEqual(set(storage.load_objects('tag2')),
set(['B']))
self.assertEqual(set(storage.load_objects('tag3')),
set(['E', 'F']))Now we can write tests that integrate Schedules and File. Notice that, for this step, we still aren't involving Status or Activity, because they're outside the oval. We'll use mock objects in place of them, for now:
from unittest import TestCase
from unittest.mock import Mock
from planner.data import Schedule
from planner.persistence import File
from os import unlink
def unpickle_mocked_task(begins):
return Mock(overlaps = Mock(return_value = False), begins = begins)
class test_schedules_and_file(TestCase):
def setUp(self):
A = Mock(overlaps = Mock(return_value = False),
__reduce__ = Mock(return_value = (unpickle_mocked_task, (5,))),
begins = 5)
B = Mock(overlaps = Mock(return_value = False),
__reduce__ = Mock(return_value = (unpickle_mocked_task, (3,))),
begins = 3)
C = Mock(overlaps = Mock(return_value = False),
__reduce__ = Mock(return_value = (unpickle_mocked_task, (7,))),
begins = 7)
self.A = A
self.B = B
self.C = C
def tearDown(self):
try:
unlink('test_schedules_and_file.sqlite')
except OSError:
pass
def test_save_and_restore(self):
sched1 = Schedule()
sched1.add(self.A)
sched1.add(self.B)
sched1.add(self.C)
store1 = File('test_schedules_and_file.sqlite')
sched1.store(store1)
del sched1
del store1
store2 = File('test_schedules_and_file.sqlite')
sched2 = Schedule.load(store2)
self.assertEqual(set([x.begins for x in sched2.tasks]),
set([3, 5, 7]))We've built our way up to the outermost circle now, which means it's time to write tests that involve the whole system with no mock objects anywhere:
from planner.data import Schedule, Status, Activity, ScheduleError
from planner.persistence import File
from unittest import TestCase
from datetime import datetime, timedelta
from os import unlink
class test_system(TestCase):
def setUp(self):
self.A = Status('A',
datetime.now(),
datetime.now() + timedelta(minutes = 7))
self.B = Status('B',
datetime.now() - timedelta(hours = 1),
datetime.now() + timedelta(hours = 1))
self.C = Status('C',
datetime.now() + timedelta(minutes = 10),
datetime.now() + timedelta(hours = 1))
self.D = Activity('D',
datetime.now(),
datetime.now() + timedelta(minutes = 7))
self.E = Activity('E',
datetime.now() + timedelta(minutes = 30),
datetime.now() + timedelta(hours = 1))
self.F = Activity('F',
datetime.now() - timedelta(minutes = 20),
datetime.now() + timedelta(minutes = 40))
def tearDown(self):
try:
unlink('test_system.sqlite')
except OSError:
pass
def test_usage_pattern(self):
sched1 = Schedule()
sched1.add(self.A)
sched1.add(self.B)
sched1.add(self.C)
sched1.add(self.D)
sched1.add(self.E)
store1 = File('test_system.sqlite')
sched1.store(store1)
del store1
store2 = File('test_system.sqlite')
sched2 = Schedule.load(store2)
self.assertEqual(sched1, sched2)
sched2.remove(self.D)
sched2.remove(self.E)
self.assertNotEqual(sched1, sched2)
sched2.add(self.F)
self.assertTrue(self.F in sched2)
self.assertFalse(self.F in sched1)
self.assertRaises(ScheduleError, sched2.add, self.D)
self.assertRaises(ScheduleError, sched2.add, self.E)
self.assertTrue(self.A in sched1)
self.assertTrue(self.B in sched1)
self.assertTrue(self.C in sched1)
self.assertTrue(self.D in sched1)
self.assertTrue(self.E in sched1)
self.assertFalse(self.F in sched1)
self.assertTrue(self.A in sched2)
self.assertTrue(self.B in sched2)
self.assertTrue(self.C in sched2)
self.assertFalse(self.D in sched2)
self.assertFalse(self.E in sched2)
self.assertTrue(self.F in sched2)We've just integrated our whole code base, progressively constructing larger tests until we had tests encompassing the whole system. The whole time, we were careful to test one thing at a time. Because we took care to go step-by-step, we always knew where the newly discovered bugs originated, and we were able to fix them easily.
Speaking of which, if you were to run the tests for yourself while building this code structure, you would notice that some of them fail. All three of the failures point to the same problem: there's something wrong with the persistence database. This error doesn't show up in the unit tests for the File class, because it's only visible on a larger scale, when the database is used to communicate information between units.
Here's the error reported by the test_file.py tests:
Traceback (most recent call last): File "integration/integration_tests/test_file.py", line 26, in test_other_instance set(['A', 'C', 'D'])) AssertionError: Items in the second set but not the first: 'A' 'D' 'C'
The changes to the database aren't being committed to the file, and so they aren't visible outside the transaction where they were stored. Not testing the persistence code in separate transactions was not an oversight, but that's exactly the sort of mistake that we expect integration testing to catch.
We can fix the problem by altering the store_object method of the File class in persistence.py as follows:
def store_object(self, tag, object):
self.connection.execute('insert into objects values (?, ?)',
(tag, dumps(object)))
self.connection.commit()Another point of interest is the interaction between pickle and mock objects. There are a lot of things that mock objects do well, but accepting pickling is not one of them. Fortunately, that's relatively easy to work around is demonstrated in test integrating Schedule and File:
def unpickle_mocked_task(begins):
return Mock(overlaps = Mock(return_value = False), begins = begins)
class test_schedules_and_file(TestCase):
def setUp(self):
A = Mock(overlaps = Mock(return_value = False),
__reduce__ = Mock(return_value = (unpickle_mocked_task, (5,))),
begins = 5)
B = Mock(overlaps = Mock(return_value = False),
__reduce__ = Mock(return_value = (unpickle_mocked_task, (3,))),
begins = 3)
C = Mock(overlaps = Mock(return_value = False),
__reduce__ = Mock(return_value = (unpickle_mocked_task, (7,))),
begins = 7)The trick here is not really very tricky. We've just told the mock objects what return value to use for calls to the __reduce__ method. It so happens that the pickle dumping functions call __reduce__ to find out whether an object needs special handling when being pickled and unpickled. We told it that it did, and that it should call the unpickle_mocked_task function to reconstitute the mock object during unpickling. Now, our mock objects can be pickled and unpickled as well as the real objects can.
Another point of interest in the tests for Schedule and File is the tearDown test fixture method. The tearDown method will delete a database file, if it exists, but won't complain if it doesn't. The database is expected to be created within the test itself, and we don't want to leave it lying around; however, if it's not there, it's not a test fixture error:
def tearDown(self):
try:
unlink('test_schedules_and_file.sqlite')
except OSError:
passA lot of the test code in this chapter might seem redundant to you. That's because, in some sense, it is. Some things are repeatedly checked in different tests. Why bother?
The main reason for the redundancy is that each test is supposed to stand alone. We're not supposed to care what order they run in, or whether any other tests even exist. Each test is self-contained; thus, if it fails, we know exactly what needs to be fixed. Because each test is self-contained, some foundational things end up getting tested multiple times. In the case of this simple project, redundancy is even more pronounced than it would normally be.
Whether it's blatant or subtle, though, the redundancy isn't a problem. The so-called Don't Repeat Yourself (DRY) principle doesn't particularly apply to tests. There's not much downside to having something tested multiple times. This is not to say that it's a good idea to copy and paste tests, because it's very much not. Don't be surprised or alarmed to see similarities between your tests, but don't use that as an excuse. Every test that checks a particular thing is a test that needs to be changed if you change that thing, so it's still best to minimize redundancy where you can.