unittest: An under-appreciated gem

Andrew Bennetts

September 15, 2008

Python's standard library includes a unittest module. unittest belongs to the xUnit family of testing frameworks, and is also known as PyUnit.

There are now alternative test frameworks for Python, such as Nose¹ and py.test². The Python standard library even includes doctest, another testing framework. Because there is an abundance of newer alternatives, people often assume that unittest is obsolete, inflexible and verbose.

Actually, unittest's design remains sound. It provides plenty of flexibility to write good unit tests, and is surprisingly simple to extend. This paper gives a brief overview of the xUnit design, and then describes some useful real-world extensions used by projects such as Bazaar, Twisted and Launchpad.

Overview of xUnit's design

The run method on TestCase runs setUp, the test method and tearDown and reports the outcome to a TestResult object.

Test methods on TestCase subclasses in Python files are turned into TestCase instances by the TestLoader. TestRunners glue all of these together.

class TestFrobnicator(unittest.TestCase): def setUp(self): self.frobnicator = Frobnicator() self.frobnicator.initialise() def test_frob_one_word(self): input = "word" output = self.frobnicator.frob(input) self.assertEqual("frob", output) def test_frob_two_words(self): input = "two words" output = self.frobnicator.frob(input) self.assertEqual("frob frob", output)

Good test design that xUnit encourages

http://xunitpatterns.com/Custom xUnit is designed to allow you to write well-factored unit tests with minimum impedance. In many ways it encourages good style.

Any non-trivial test suite will need some structure to be manageable. xUnit provides that structure: TestCase subclasses group tests with common needs, facilitating code reuse in your tests. In addition to the built-in setUp and tearDown hooks, TestCase subclasses are a natural home for adding reusable domain-specific helpers, e.g. to add a ``assertUserHasPermission'' method³.

Because xUnit tests are methods they have names. This allows good reporting of exactly which tests are failing, and a way to run individual tests rather than the whole suite.

Because each test has its own TestCase instance, by default tests tend to be isolated from each other. Fixtures also tend to be minimal rather than large general-purpose fixtures that obscure intent and hinder debugging.

It's easy to reuse fixture setup code without sharing the fixture; just add another test method to the TestCase subclass. This in turn makes it easy to write single condition tests -- short tests, clear in intent, that test just a single condition of the system under test.

Extending PyUnit

PyUnit is easy to extend. The design allows you to use your own TestCase, TestResult, TestLoader, and so on. The interactions between the different xUnit components are simple and reasonably well-defined.

Here I present some examples of PyUnit extensions, some available as stand-alone libraries and some that have been implemented within the test suites of Bazaar or Twisted.

All of these extensions inter-operate cleanly, and no significant effort was required to achieve this.

addCleanup

This is a feature added to TestCase in the testtools library⁴ and the test suites of Bazaar and Twisted.

addCleanup is a robust way to arrange for a cleanup function to be called before tearDown. This is a powerful and simple alternative to putting cleanup logic in a try/finally block or tearDown method. For example:

The obvious alternative is to just use try/finally blocks, but they don't allow a helper method to acquire a resource and arrange for it to be cleaned up later. They also tend to lead to excessive indentation when nested.

Using addCleanup can often make tearDown methods unnecessary, or at least simpler.

Implementation of addCleanup

Test Parameterisation

This is a feature that has been implemented in testtools and Bazaar's test suite.

Often a test is applicable to multiple scenarios. For example, when there are multiple implementations of an interface and a suite of tests for that interface: all implementations should pass that same suite of tests.

For maintainability the same test code should be reused rather than duplicated for each scenario. A common way to do this with xUnit is make a base TestCase class with the test code, then a subclass for each scenario. This is a bit awkward to implement (you don't want the test loader to load the abstract base class, just its subclasses), and the intent of the subclassing is usually not obvious to a casual reader. It is also tedious if there are many TestCase classes to parameterise rather than just one.

Testtools provides a better way: multiply_test_suite_by_scenarios, a function that takes a test suite and list of scenarios. It returns a new test suite with multiple copies of each original test, one per scenario. Bazaar test suite also uses a variant of this idea.

Here's an example, using the testtools API for multiply_test_suite_by_scenarios:

# Simplified test case example based on a real test case from Twisted class LineReceiverTests(TestCase): def setUp(self): self.lineReceiver = self.makeLineReceiver(self.scenario.lineReceiverClass) def testLongLine(self): self.lineReceiver.MAX_LENGTH = 5 self.lineReceiver.dataReceived('123456n789n') # etc... # (See "Custom test loaders" section below.) def load_tests(standard_tests, module, loader): tests = testtools.multiply_test_suite_by_scenarios( standard_tests, Scenario('LineReceiver', lineReceiverClass=LineReceiver), Scenario('LineOnlyReceiver', lineReceiverClass=LineOnlyReceiver)) return unittest.TestSuite(tests)

Bazaar's test suite uses this idea extensively, with tests that are applied for each implementation of Transport, Branch, Repository, and so on. Bazaar plugins also benefit from this; for instance a plugin to provide WebDAV support for Bazaar can very easily have its new transport implementation included in the list of transports that must pass the Transport interface tests in bzr's test suite.

The new tests will have the scenario name appended to their name. Combined with a tool that filters test suites by test name this makes it easy to run all tests for a particular implementation. For instance, Bazaar's selftest can filter by regular expression, so to run all tests for the PyCurlTransport in Bazaar you can do:

There is an implementation of multiply_test_suite_by_scenarios available in testtools.

requireFeature

unittest by itself only has the notion of tests that succeeded or failed, but some test outcomes don't fit either category very well. For example Bazaar has tests for how it handles symlinks, but Windows doesn't support symlinks. Bazaar extended the standard TestCase class to allow a test to do:

Alternatively, individual test methods can call self.requireFeature(SymlinkFeature).

class _SymlinkFeature(Feature): def _probe(self): return hasattr(os, 'symlink') def feature_name(self): return 'symlinks' SymlinkFeature = _SymlinkFeature()

Tests that require an unavailable feature will be reported as such when run by Bazaar's test runner, which uses an extended TestResult class.

This extension is compatible with other runners that use the standard TestResult. With a standard TestResult an unavailable feature will cause a test outcome to be regarded as a success.

SubUnit

SubUnit, available from https://launchpad.net/subunit, is a library for running unit tests in separate processes to support test isolation.

Subunit comes in two parts: There is a parent-process component and a child process component. The parent component pretends to be a normal PyUnit test suite. Secondly there is the child component that runs in the child process and reports to the parent as tests begin and end. There are currently 3 implementations of the child - Python, C (with bindings to 'check' and 'cppunit' and shell.

It also includes a reporter for PyUnit to gather the results and expose them as PyUnit tests, allowing seamless integration of heterogenous tests into a single test run and UI.

testresources

testresources, available from https://launchpad.net/testresources, is a PyUnit library to manage the initialisation and lifetime of expensive test fixtures. For example reference databases often only need to be constructed once but standard test isolation causes them to be constructed for every fixture, making test execution very slow.

Custom test loaders (test_suite/load_tests hooks)

Plain unittest only provides simple APIs to load tests; the most powerful test-loading feature built-in is a method to take a single Python module and build tests from every TestCase defined in it.

Some test runners built on PyUnit provide a way for a test suite to have more control over test loading. For example, in Bazaar's test suite, if a module provides a load_tests function then Bazaar's test loader will invoke that rather than using the PyUnit default behaviour. The signature of it is load_tests(standard_tests, module, loader), where standard_tests are the tests found by the stock PyUnit test loader, module is the test module, and loader is Bazaar's test loader.

def load_tests(standard_tests, module, loader): result = loader.suiteClass() for test in iter_suite_tests(standard_tests): result.addTests([test, test]) return result

See the Test Parameterisation section for another example that dynamically generates tests.

PyUnit's weaknesses

The biggest annoyance is that there's no standard tool for loading and invoking a test suite. There are plenty of third-party answers to this, such as Twisted's trial command-line tool, but this is a surprising gap given Python's ``batteries included'' motto.

There are other problems too. Parts of the API could be better, for instance it is hard to change the id of a TestCase instance. The built-in set of assertion methods could be richer. The documentation for it bundled with Python could be better. And so on.

Conclusion

unittest is far from perfect. But it is still a very good API for writing clean, well-factored unit tests. It is very extensible, and it provides a reasonable API for creating inter-operable extensions. Because it is a standard library module, and has been since 2.1, it is a de facto standard that other testing frameworks are generally compatible with.

And there is an active community building and improving extensions for the unittest framework. This paper has presented some of those extensions, and I encourage you to use and improve these tools, and to share your own.