How did we increase test coverage from less than 50% to 87% in just 10 months? Part of the answer is a tool called diff-cover.
In a typical workflow, a developer working in a large project might make a pull request that changes a few dozen lines of code. Before the change, the test coverage may have been 72%; afterwards, it could still be 72%. The size of the project makes it difficult to see the effect of a single pull request. Diff-cover lets you focus on the quality metrics of a single pull request instead of the project as a whole.
Diff-cover measures lines of code in a git diff. For a proposed change to the code, it will show you which of the changed lines are missing coverage. This is a simple idea, but it has powerful implications:
- For developers, diff-cover provides a clear and achievable metric: if you touch a line of code, it should be covered.
- For code-reviewers, diff-cover makes it easier to verify that developers are writing tests for all code changes.
By focusing on diff coverage, developers can make small, visible steps toward improving global coverage. A particular commit might increase global coverage by only a fraction of a percent but still have 95% diff coverage. Slowly but surely, as developers wrote tests for their code changes, global coverage began increasing as well. As a result, we were able catch certain kinds of bugs sooner.
More importantly, other developers began contributing to diff-cover itself and taking ownership over the tool. For example, Cale generalized the tool to support additional “quality” checks, and Sarina extended it to report pep8 and Pylint violations in a diff. Many other developers provided feedback and suggestions during an initial beta test of diff-cover. The tool became a starting point for a re-examination of our code review and testing standards, which led to a real change in our testing culture.
Of course, coverage measurement still has some important limitations. In particular, high diff coverage does not guarantee bug-free code: in a tightly coupled system, a change to one component could have far-reaching and unintended consequences on other parts of the system — even if the changed code is 100% covered. In such cases, integration tests can catch bugs that unit tests might miss.
If you think diff-cover could be useful to you, check out the project — it’s open-source and available on GitHub. The code is designed to be extensible to other version control systems and quality checkers, so feel free to add features and make a pull request!
Will Daly is a test engineer at edX. When he’s not advocating test-driven development or optimizing a Jenkins cluster, he enjoys running along rivers and minimizing the number of things in his apartment.