Sometimes Test the Framework

To every rule, there are exceptions. When we talked about frameworks, we talked about things like trust, having tests, guarantees, and correctness. Sometimes, those conditions are not true. Other times, you cannot afford to assume they are true. Let’s look at some times in which you need to test the framework.

Imagine you are part of a team working on a large software system, perhaps a product consisting of millions of lines of code, hundreds of developers, and using hundreds of third-party packages across the entire code base. In any given period of time between releases, perhaps dozens of those packages change and may need to be upgraded. How do you update the packages safely without breaking your system? You can rely on the tests you have written for your own use of the packages in part, but in other cases, particularly those in which you have chosen to stub or mock out the framework functionality (e.g., a service access library) or where it participates outside of the scope of your tests (like a dependency injection framework might, for example), your tests may not be sufficient to qualify the new version.

An open-source package without tests, especially one that is young, infrequently maintained, or with a history of incompatibility, also may not meet the standards of trust you need to include it without verification. Writing your own tests may be the most prudent course of action to ensure it meets your needs. You may even consider contributing your tests back to the project.

Frameworks with high-risk dependencies for your project probably also deserve greater attention to correctness. A simple framework used pervasively affects all parts of your system. A framework used comprehensively requires all parts of the framework to behave as expected.

In the end, testing third-party code entails a risk assessment of whether the cost of testing it justifies the benefits of reuse against the risk of failures. Consider it, and judge wisely.