Understanding OSR in JVM (On-Stack Replacement)

What is OSR?

On-Stack Replacement (OSR) is a JVM feature that allows it to replace currently executing code with a more optimized version mid-execution. Think of it as “changing the engine while the car is running.”

Why do we need OSR?

Consider this example:

public class OSRExample {
  public static void main(String[] args) {
      long sum = 0;
      // This loop will run for a very long time
      for (int i = 0; i < 10_000_000; i++) {
          sum += compute(i);
      }
      System.out.println(sum);
  }

  static int compute(int i) {
      return i * 2;
  }
}

Without OSR:

1. JVM starts executing the loop in interpreted mode
2. Realizes it’s hot code
3. Would have to wait for the loop to finish before using optimized code
4. Could be stuck with slow code for a long time

With OSR:

1. JVM starts executing in interpreted mode
2. Notices the loop is hot
3. Compiles the code while the loop is running
4. Switches to the optimized version mid-loop!

How to see OSR in action

Look for the % symbol in JVM compilation logs:

java -XX:+PrintCompilation OSRExample

Output might show:

27    7 %     3       OSRExample::main @ 4 (32 bytes)  // OSR compilation

The % indicates an OSR compilation, and @ 4 shows the bytecode index where OSR can occur.

Observing OSR in action

public class OSRDemo {

  static int counter = 0;

  public static void main(String[] args) throws Exception {
      long start = System.nanoTime();
      
      // Long-running loop that's a candidate for OSR
      for (int i = 0; i < 1_000_000; i++) {
          if (i % 10_000 == 0) {
              System.out.println("Progress: " + i);
              Thread.sleep(100);
          }
          counter += compute(i);
      }
      
      long end = System.nanoTime();
      System.out.println("Time: " + (end - start)/1_000_000 + "ms");
  }

  static int compute(int i) {
      return i % 2 == 0 ? i * 2 : i * 3;
  }
}

Run with:

java -XX:+PrintCompilation -XX:+UnlockDiagnosticVMOptions OSRDemo

You’ll see OSR compilations happening while the loop is still running!

Types of On-Stack Replacement

OSR Entry

Going from interpreter to compiled code.

public class OSREntryExample {
    public static void main(String[] args) {
        int sum = 0;
        
        // This loop starts in interpreter mode
        for (int i = 0; i < 1_000_000; i++) {
            // First few thousand iterations run interpreted
            sum += i;
            
            // JVM notices: "This loop is running a lot!"
            // At some point during execution (let's say i = 50000):
            // 1. JVM compiles this code
            // 2. Switches to compiled version
            // 3. Continues from i = 50000 but now running compiled code
        }
    }
}

OSR Exit

Going from compiled back to interpreter code.

public class OSRExitExample {
    public static void main(String[] args) {
        int[] array = new int[1000];
        
        // This loop is running compiled code
        for (int i = 0; i < array.length; i++) {
            // JVM compiled this assuming array is always int[]
            array[i] = array[i] * 2;
            
            if (i == 500) {
                // Suddenly change array type
                array = new double[1000];  // Oops! Assumption broken
                
                // JVM: "My assumption was wrong!"
                // 1. Must deoptimize
                // 2. Return to interpreter
                // 3. Continue safely but slower
            }
        }
    }
}

If we want to see the OSR exit, here is an examle that will show it.

public class CompleteOSRExample {
    static int processValue(int value) {
        int result = value;
        
        // Loop starts interpreted
        while (result < 1000000000) {
            if (result < 0) {
                // Rare case, if hit:
                // OSR Exit would happen here
                return -1;
            }
            
            result += 1;
            // If loop runs long enough:
            // OSR Entry happens here
            // Code gets compiled and switched
        }
        
        return result;
    }
    
    public static void main(String[] args) {
        // Normal case - will trigger OSR Entry
        System.out.println(processValue(1));  // Takes many iterations
        
        // Uncommon case - will trigger OSR Exit
        System.out.println(processValue(-5)); // Hits negative check
    }
}

We need to run this with the following options:

java  -XX:+PrintCompilation \
      -XX:+UnlockDiagnosticVMOptions \
      -XX:+LogCompilation \
      -XX:+TraceDeoptimization \
      CompleteOSRExample

We can lookup unstable_fused_if in the console output to see the reason behind the OSR exit. It marks the point when JVM decides to rool back the compiled code and run in the interpreted mode:

Benefits of OSR

Don’t wait for method exit

• Can optimize long-running loops immediately
• Don’t need to wait for current method to finish

Adaptive optimization

• Can switch to better code when available
• Can deoptimize if assumptions prove wrong

Better performance for long-running loops

• Loops get optimized while running
• Don’t stuck with interpreter performance

Common Scenarios

There are many scenarios that demonstrate OSR’s usefulness. Here are some most obvious ones.

// Web servers
while (serverIsRunning) {
    handleRequest();  // OSR will optimize this loop
}

// Data processing
for (Record record : hugeDataset) {
    processRecord(record);  // Gets OSR'd if dataset is large
}

// Game loops
while (gameIsRunning) {
    updateGame();  // Perfect OSR candidate
    renderFrame();
}