DeFlaker

DeFlaker is a Maven build extension that identifies when a test is flaky. DeFlaker declares a test as likely flaky if it fails without executing any changed code. DeFlaker collects statement-level coverage for every test run. To reduce the overhead of collecting coverage that is imposed by existing tools, DeFlaker collects coverage only on the statements that have changed since the last successful build. When a test fails, DeFlaker checks if its coverage includes any changed code. To help debug these flaky tests, DeFlaker can also capture a complete core dump of the JVM at the time of the test failure. Finally, DeFlaker re-runs each likely flaky test at the end of the test execution, in an isolated JVM, to see if it fails again or passes (and records the output).

DeFlaker adds output like this after your tests run, if a likely flaky test is detected:

[WARNING] FLAKY>> Test FlakyTest.testRandom4 failed, but did not appear to run any changed code

This line indicates that a test failed, but didn’t run any changed code.

DeFlaker is deployed on Maven Central, and is compatible with JUnit 4 and TestNG tests that are executed with Maven, using the surefire or failsafe plugin.

To add DeFlaker to your build, add it to your pom.xml:

<build>
...
  <extensions>
    <extension>
      <groupId>org.deflaker</groupId>
      <artifactId>deflaker-maven-extension</artifactId>
      <version>1.4</version>
    </extension>
  </extensions>
...
</build>

 

By default, DeFlaker will rerun your failing tests up to 10 times (5 times in the same JVM where the test failed and then for 5 times in new, clean JVMs). The system property deflaker.rerunFlakies configures this behavior, specifying the number of times to rerun tests in each mode (in the same JVM and in the clean JVMs. For instance running mvn -Ddeflaker.rerunFlakies=1 test will cause failing tests to be re-run once in the JVM they failed in, then once in a clean JVM.

We are a group of individuals who are passionate about creating tools to improve software testing. If you have any feedback about the DeFlaker, please, email us at bellj@gmu.edu.