Mutation testing in Kotlin using PIT

This article is continuation of “Mutation testing in Java using PIT”.

The most important thing to look at when setting up PITest for Kotlin project is this repo’s readme: https://github.com/pitest/pitest-junit5-plugin. Otherwise you end up with errors like:

Exception in thread "main" org.pitest.util.PitError: 
Coverage generation minion exited abnormally! (UNKNOWN_ERROR)

Or this one, and many others:

Exception in thread "main" java.lang.IllegalArgumentException: 
  Unsupported class file major version 65
    at org.pitest.reloc.asm.ClassReader.<init>(ClassReader.java:199)

Here is what to pay attention to:

README.md in pitest-junit5-plugin

Gradle setup for Kotlin codebase

Here is the Gradle build file (in Kotlin). Pay extra attention to what version you set to “info.solidsoft.pitest” and to junit5PluginVersion, as mentioned earlier.

import org.gradle.api.tasks.testing.logging.TestExceptionFormat.FULL
import org.gradle.api.tasks.testing.logging.TestLogEvent.FAILED
import org.gradle.api.tasks.testing.logging.TestLogEvent.PASSED
import org.gradle.api.tasks.testing.logging.TestLogEvent.SKIPPED

plugins {
    jacoco
    kotlin("jvm") version "1.9.22"
    id("info.solidsoft.pitest") version "1.15.0"
}

repositories {
    mavenCentral()
}

dependencies {
    testImplementation(platform("org.junit:junit-bom:5.10.1"))
    testImplementation("org.junit.jupiter:junit-jupiter")
    testImplementation("org.junit.jupiter:junit-jupiter-params")
}

kotlin {
    jvmToolchain(21)
}

tasks.test {
    useJUnitPlatform()
    finalizedBy(tasks.jacocoTestReport)
    finalizedBy(tasks.pitest)
    
    testLogging {
        events(PASSED, SKIPPED, FAILED)
        exceptionFormat = FULL
        showExceptions = true
        showCauses = true
        showStackTraces = true
    }
}

tasks.jacocoTestReport {
    dependsOn(tasks.test)
}

pitest {
    junit5PluginVersion = "1.2.1"
    avoidCallsTo = setOf("kotlin.jvm.internal")
    mutators = setOf("STRONGER")
    targetClasses = setOf("com.example.*")
    targetTests = setOf("com.example.*")
    threads = Runtime.getRuntime().availableProcessors()
    outputFormats = setOf("XML", "HTML")
    mutationThreshold = 75
    coverageThreshold = 60
}

tasks.named("check") {
    dependsOn(":pitest")
}

Write and execute tests

Now we can write tests and implement the code accordingly.

package com.example

import org.junit.jupiter.api.Assertions.assertFalse
import org.junit.jupiter.api.Assertions.assertTrue
import org.junit.jupiter.params.ParameterizedTest
import org.junit.jupiter.params.provider.ValueSource

internal class CalculatorTest {

    private val calculator = Calculator()

    @ParameterizedTest
    @ValueSource(ints = [-1, 0, 1, 4, 6, 8, 100])
    fun `verifies non primary numbers`(value: Int) {
        assertFalse(calculator.isPrime(value))
    }

    @ParameterizedTest
    @ValueSource(ints = [2, 3, 5, 7, 11, 7919])
    fun `verifies primary numbers`(value: Int) {
        assertTrue(calculator.isPrime(value))
    }
}

And we can implement the code as well:

package com.example

class Calculator {

    fun isPrime(a: Int): Boolean {
        if (a <= 1) {
            return false
        }
        for (i in 2 until a) {
            if (a % i == 0) {
                return false
            }
        }
        return true
    }
}

When we run the tests, we will get JaCoCo report and also PITest coverage report.

JaCoCo coverage report

PITest coverage report

Resources

• https://github.com/hcoles/pitest
• https://github.com/szpak/gradle-pitest-plugin
• https://gradle-pitest-plugin.solidsoft.info/
• https://github.com/pitest/pitest-junit5-plugin
• https://mvnrepository.com/artifact/info.solidsoft.pitest/info.solidsoft.pitest.gradle.plugin
• https://betterprogramming.pub/how-to-improve-the-quality-of-tests-using-mutation-testing-2346019829f1
• https://github.com/PoisonedYouth/kotlin-mutation-testing
• https://dev.to/rogervinas/mutation-testing-2cfd
• https://medium.com/seat-code/mutation-testing-in-kotlin-a8834771e85e
• https://stackoverflow.com/questions/63248845/pit-coverage-generation-minion-exited-abnormally