Mutation testing in Java using PIT
Mutation testing is a technique that evaluates quality of our tests by introducing small changes (mutations) into the code we are testing. As the code gets modified by PIT (PITest), the test should fail, if it does not fail, it means the test is not covering all the use cases and should be improved.
Setup for Java codebase
Here is the Gradle build file (in Kotlin).
plugins {
id("java")
jacoco
id("info.solidsoft.pitest") version "1.15.0"
}
group = "org.example"
version = "1.0-SNAPSHOT"
repositories {
mavenCentral()
}
dependencies {
testImplementation(platform("org.junit:junit-bom:5.9.1"))
testImplementation("org.junit.jupiter:junit-jupiter")
}
tasks.test {
useJUnitPlatform()
finalizedBy(tasks.jacocoTestReport)
finalizedBy(tasks.pitest)
}
tasks.jacocoTestReport {
dependsOn(tasks.test)
}
pitest {
setProperty("junit5PluginVersion", "1.2.1")
}
Code example to test
Here is a simple naive code to identify if a number is a prime number.
package org.example;
public class Calculator {
public boolean isPrime(int n) {
if (n <= 1) {
return false;
}
for (int i = 2; i < n; i++) {
if (n % i == 0) {
return false;
}
}
return true;
}
}
Crappy unit test
Here is a test that is point out the issue, that you can get some test coverage without any meaningful assertions.
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;
class CalculatorTest {
private final Calculator calculator = new Calculator();
@Test
void isPrime() {
calculator.isPrime(1);
calculator.isPrime(2);
assertTrue(true);
}
}
Here is what coverage we get with that.
Mutation tests (using PIT)
Let’s measuring the test quality with mutation testing for the crappy test we introduced. The result is, as expected, very unsatisfactory.
Fixing the code
Now let’s fix the test so it actually tests something, instead of reporting false test coverage.
package org.example;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;
class CalculatorTest {
private final Calculator calculator = new Calculator();
@Test
void twoIsPrime() {
boolean isPrime = calculator.isPrime(2);
assertTrue(isPrime);
}
}
When we run the PIT tests and look at the coverage report, we can see that the test improved its quality.
Let’s finish the test to cover all the mutations.
package org.example;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
import static org.junit.jupiter.api.Assertions.*;
class CalculatorTest {
private final Calculator calculator = new Calculator();
@ParameterizedTest
@ValueSource(ints = { -1, 0, 1, 4, 6, 8, 100 })
void numbersAreNotPrime(int value) {
assertFalse(calculator.isPrime(value));
}
@ParameterizedTest
@ValueSource(ints = { 2, 3, 5, 7, 11, 7919 })
void numbersArePrime(int value) {
assertTrue(calculator.isPrime(value));
}
}
When we run the tests again, we can observe that the code is covered by tests, using both JaCoCo and PIT.
Resources
- https://github.com/hcoles/pitest
- https://github.com/szpak/gradle-pitest-plugin
- https://gradle-pitest-plugin.solidsoft.info/
- https://github.com/pitest/pitest-junit5-plugin
- https://mvnrepository.com/artifact/info.solidsoft.pitest/info.solidsoft.pitest.gradle.plugin
- https://betterprogramming.pub/how-to-improve-the-quality-of-tests-using-mutation-testing-2346019829f1
- https://github.com/PoisonedYouth/kotlin-mutation-testing
- https://dev.to/rogervinas/mutation-testing-2cfd
- https://medium.com/seat-code/mutation-testing-in-kotlin-a8834771e85e