How does DeFlaker find flaky tests?
Recall that a test is flaky if it both passes and fails when the code that is executed by the test did not change; moreover, a test failure is new if the test passed on the previous version of code but fails in the current version. A straw-man technique to detect flaky tests is to collect statement coverage for each test (considering all code — of both the test code and the code under test), intersect coverage results with the changeshistory from a version-control system (VCS), and report as flaky new test failures that did not execute any changed code. However, collecting full statement coverage can be expensive. However, this technique is rather costly: industry reports suggest that collecting full coverage is often avoided due to the overhead imposed by coverage tools; our own study applying code coverage tools to open-source Java test suites showed that such tools can double the execution time.
Our key insight in DeFlaker is that one need not collect coverage of the entire codebase. Instead, one can collect only the coverage of the changed code, which we call differential coverage. Differential coverage first queries a version-control system (VCS) to detect code changes since the last version. It then analyzes the code and constructs an abstract-syntax tree for each changed file to determine where instrumentation needs to be inserted to track execution of each change. Finally, when tests run, it monitors change execution, generating an individual change-coverage report for each test.
DeFlaker detects flaky tests through lightweight differential coverage tracking. If a test fails but does not cover any changed code, DeFlaker reports the test as flaky without requiring any reruns.
While DeFlaker’s approach is generic and can apply to nearly any language or testing framework, we implemented our tool for Java, the Maven build system, and two testing frameworks (JUnit and TestNG). DeFlaker is available under an MIT license on GitHub, with binaries published on Maven Central. To add DeFlaker directly to your build, simply add it to your pom.xml:
<build>
...
<extensions>
<extension>
<groupId>org.deflaker</groupId>
<artifactId>deflaker-maven-extension</artifactId>
<version>1.4</version>
</extension>
</extensions>
...
</build>