Mutation Testing: How Well Your Unit Tests Cover?

Unit testing is a crucial point for software development and maintenance. From design phase to maintenance phase, unit tests allow software developers to provide robust design, reduce code complexity, detect and fix a defect at an early stage, and extend software functionality with lower cost. Because of this kind of benefits, unit test metrics and results are integrated on release and build pipelines. One of the most popular metrics is code coverage. Unit test line coverage is helpful for quantity based evaluation but relying only on this metric makes developers miss quality of the unit tests.

Mutation testing gauges how well your unit tests are written and whether they cover all the cases. Mutation testing functions as changing the code behavior. Every modification is called as mutation. With this mutation, all of unit tests are run and if at least one of them is failed, this mutation is caught and killed by your unit tests. But if your unit tests still pass after the mutation, your unit tests are unable to catch the mutation and it survives. Survival of the mutation means your unit tests cannot cover all cases. The percentage of the killed mutations shows the quality of your unit cases.

In this article, I will mention about how to perform mutation testing on Spring Boot application. In my project, I wrote my unit tests with JUnit 5. To perform mutation tests, PITest library will be used. Maven dependency of PITest can be added as following.

<dependency>
    <groupId>org.pitest</groupId>
    <artifactId>pitest-parent</artifactId>
    <version>1.6.6</version>
    <type>pom</type>
</dependency>

And also pitest-maven plugin will be added in our pom.xml in order to use PITest library.

<plugin>
    <groupId>org.pitest</groupId>
    <artifactId>pitest-maven</artifactId>
    <version>1.6.6</version>
    <dependencies>
        <dependency>
            <groupId>org.pitest</groupId>
            <artifactId>pitest-junit5-plugin</artifactId>
            <version>0.14</version>
        </dependency>
    </dependencies>
    <configuration>
        <targetClasses>
            <param>com.sample.mutation.*</param>
        </targetClasses>
        <targetTests>
            <param>com.sample.mutation.*</param>
        </targetTests>
    </configuration>
</plugin>

pitest-junit5-plugin is required to perform mutation tests with your unit tests implemented with JUnit 5. With configuration part, mutation testing can be performed on specific packages that you deserve. Creating mutations and running all unit tests for each mutation is costly operation and filtering target packages and tests according to requirements helps to reduce this cost. After these preparations, it is ready to perform mutation test. Now, let’s execute the pitest plugin with the command below and check the mutation testing result:

mvn org.pitest:pitest-maven:mutationCoverage

After the execution is completed, there will be reports in HTML format in the target/pit-reports/YYYYMMDDHHMI directory.

For better understanding, let’s focus on an example. We have following code that checks whether the given number is the ith fibonacci number.

public class Fibonacci {

    private int getFibonacci(int index) {
        if(index == 0){
            return 0;
        }
        if(index == 1){
            return 1;
        }
        int firstPart = getFibonacci(index-1);
        int secondPart = getFibonacci(index-2);

        return firstPart + secondPart;
    }

    public boolean isCorrectFibonacci(int index, int number){
        return number == getFibonacci(index);
    }
}

For this class, the following unit code is implemented.

class FibonacciUnitTest {

    @Test
    void whenCorrectFibonacci_thenAccept() {
        Fibonacci fibonacciTester = new Fibonacci();
        assertTrue(fibonacciTester.isCorrectFibonacci(5, 5));
    }
}

When mutation testing is performed, the result report appears as following.

As the report says nine mutations are created but one of them can’t be covered by the unit test and it is survived. Although line coverage is 100%, unit test isn’t good enough to cover all cases. To increase quality, the unit test should be added to cover the case that resulted in a survived mutation. In this case, the mutation changed the line 19 as ‘return true;’ and our unit test only covers positive results. We can handle this mutation by adding unit test checks whether the method returns false when it is required.

void whenWrongFibonacci_thenReject() {
    Fibonacci fibonacciTester = new Fibonacci();
    assertFalse(fibonacciTester.isCorrectFibonacci(6, 11));
}

After adding this test case, mutation testing is performed again and the new report becomes as following. As the report states, all mutations are killed and our test strength is 100%.

Even though code coverage is an important metric, but sometimes it is not sufficient enough to guarantee a well-tested code. Mutation testing helps us to find possible missing test cases.