Making spring boot integration tests fast

A Spring Boot service with Kafka and MongoDB integration tests (Testcontainers) took 3m14s to run. Removing @DirtiesContext had been tried before and it broke the tests. This post explains why, what actually cost the time, and what fixed it.

1. Measure the gaps between test classes

JUnit XML reports only record time spent inside test methods. Context startup, context shutdown and container startup happen between classes and are not shown. Each <testsuite> element does carry a timestamp and a time, so the gaps can be reconstructed:

import re, glob, datetime as dt
rows = []
for f in glob.glob('build/test-results/test/*.xml'):
    s = open(f).read()
    m = re.search(r'<testsuite name="([^"]+)" tests="(\d+)".*?timestamp="([^"]+)".*?time="([\d.]+)"', s, re.S)
    rows.append((dt.datetime.fromisoformat(m.group(3)), float(m.group(4)), m.group(1).split('.')[-1]))
rows.sort(); prev_end = None
for ts, t, name in rows:
    gap = (ts - prev_end).total_seconds() if prev_end else 0
    if t > 1 or gap > 1: print(f"gap={gap:5.1f}s dur={t:5.1f}s {name}")
    end = ts + dt.timedelta(seconds=t)
    prev_end = max(prev_end, end) if prev_end else end

Findings for this suite:

• 31 Spring contexts were started, each in under one second. Context startup was not the problem.
• Three gaps of exactly 10 seconds with no test running.
• Kafka consumer tests took about 2 seconds per test.

2. Background: what a test context is and how Spring caches it

When a Spring Boot application starts, Spring creates every object the application needs (controllers, services, repositories, Kafka consumers, database clients) and wires them together. That set of live objects is the ApplicationContext. It is one running copy of the application.

A test annotated with @SpringBootTest needs such a copy to run against, so Spring starts one. When the test class finishes, Spring does not shut the copy down. It keeps it in a cache and reuses it for the next test class that asks for the same configuration (reference). This is what keeps a suite of 100 test classes from booting the application 100 times.

Two test classes get the same copy only if their configuration is identical. Any difference in @TestPropertySource, @DynamicPropertySource, @MockitoBean, @Import or @ActiveProfiles results in a separate copy. Both copies stay in the cache.

A cached copy is not paused. Its @KafkaListener containers keep polling and its database clients stay connected until the copy is closed, which happens either through @DirtiesContext or at JVM shutdown (reference).

3. What went wrong in this suite

The test classes differed from each other in small ways: one set a feature flag for tenant A, another for tenant B, another mocked an HTTP client, several used randomly generated consumer group ids. Each difference produced another copy. The result was 31 copies of the application running at the same time in one JVM.

Every copy’s Kafka consumers connected to the same Kafka container, subscribed to the same topics, and used the same consumer group names. Kafka delivers each record to exactly one member of a consumer group, and it has no way of knowing that these members belong to 31 different test setups (reference). So a record published by test class 17 could be picked up by a consumer belonging to copy 5, processed with copy 5’s configuration, and written to a different collection than test 17 was checking. Test 17 then timed out.

How @DirtiesContext was masking this

@DirtiesContext on a test class tells Spring to close that copy after the class finishes and remove it from the cache. With the previous copy gone, only the current copy’s consumers were alive, and records went to the right place. This is why removing the annotation broke the tests.

It was expensive for two reasons:

1. The next class needing that configuration had to boot a fresh copy.
2. Closing a copy stops all its Kafka listener containers, and each stop waits for the consumer thread to exit, up to shutdownTimeout, which defaults to 10 seconds (reference). Because the other 30 copies were constantly joining and leaving the same groups, consumers were usually mid-rebalance and could not exit promptly. Every close hit the full 10 seconds. Those were the 10-second gaps in the timeline.

@DirtiesContext was not fixing the underlying problem, which was that many copies were consuming at once. It was destroying copies so that only one was left, and paying 10 seconds each time.

4. Fix: one listening context at a time

Two changes.

Make fewer copies

Most of the differences between test classes were unnecessary. For example, one group of tests set feature.enabled-tenants=A and another set it to B, but the first group only ever used tenant A and the second only tenant B. Setting A,B once in the shared test application.yaml satisfies both groups with a single copy. Mocks of external HTTP clients were moved into the base class as shared @MockitoBean fields, so every class asks for the same set of mocks; Mockito beans are reset after each test by default (reference).

After this, almost every integration test asks for an identical configuration. The count went from 31 copies to 17, and the group of tests that dominated the runtime shares one copy. A few copies still legitimately differ, for example a test that mocks a domain service to force failures.

Let only one copy consume from Kafka

The remaining copies still cannot be allowed to consume at the same time. But they do not need to be destroyed for that. Only their Kafka consumers need to be stopped while another copy is in use. The rest of the copy (beans, database connections, configuration) can stay cached.

A TestExecutionListener (reference) on the base class does this. Before each test class runs:

1. Determine which copy the class needs.
2. If it is the copy whose consumers are currently running, do nothing.
3. Otherwise, stop the consumers of the currently active copy, start the consumers of the new copy, and remember the new copy as active.

KafkaListenerEndpointRegistry is the Spring Kafka bean that owns all listener containers of a context; stop(Runnable) stops them in parallel and start() starts them again (reference). Because copies are never destroyed, switching back to a copy used earlier only restarts its consumers.

public class KafkaListenersPerTestClass extends AbstractTestExecutionListener {

  private static ApplicationContext contextWithLiveListeners;

  @Override
  public void beforeTestClass(TestContext testContext) throws InterruptedException {
    var context = testContext.getApplicationContext();
    if (context == contextWithLiveListeners) return;
    stopListeners(contextWithLiveListeners);
    registry(context).start();
    contextWithLiveListeners = context;
  }

  private static void stopListeners(ApplicationContext context) throws InterruptedException {
    if (context == null) return;
    var stopped = new CountDownLatch(1);
    registry(context).stop(stopped::countDown);
    stopped.await(30, TimeUnit.SECONDS);
  }

  private static KafkaListenerEndpointRegistry registry(ApplicationContext context) {
    return context.getBean(KafkaListenerEndpointRegistry.class);
  }
}

@SpringBootTest
@TestExecutionListeners(
    listeners = KafkaListenersPerTestClass.class,
    mergeMode = TestExecutionListeners.MergeMode.MERGE_WITH_DEFAULTS)
public abstract class BaseIntegrationTest { ... }

@DirtiesContext is no longer required. The stop is also fast now: with one copy consuming at a time there is no rebalance in progress, so consumers exit on their next poll instead of waiting for the 10-second timeout.

This assumes test classes run sequentially within a JVM (Gradle’s default). With maxParallelForks each fork has its own state, which is fine. With JUnit parallel execution inside one JVM it would not be.

Two smaller items found on the way:

• 27 Awaitility.pollDelay(1, SECONDS) calls added a fixed second before the first check. Awaitility polls every 100 ms by default. For assertions that something does not happen, during(1, SECONDS) expresses the intent correctly (reference).
• A test topic was configured with 32 partitions, which only made rebalances slower.

5. Docker on macOS: the 256 file descriptor limit

After the refactor, full runs began to fail mid-way with MongoSocketReadException: Connection reset, Kafka AdminClient timeouts, and docker ps on the host reporting that the daemon was unreachable. Inside the Colima VM (colima ssh -- docker ps) all containers were healthy.

Sampling lsof during a run showed the cause. Colima forwards published container ports and the Docker socket through a single ssh ... ssh.sock [mux] process (reference). That process inherits the shell’s ulimit -n, which is 256 by default on macOS. One forwarded TCP connection uses one file descriptor. During the run the process sat at 256 to 260 descriptors, and new connections were reset.

The tests were part of the cause: each @BeforeEach created a new MongoClient and a new Kafka producer and never closed them. Peak usage was 146 connections to the Kafka port and about 96 to the Mongo port.

Two changes were needed:

• Tests: one MongoClient and one DefaultKafkaProducerFactory per JVM, held in static fields on the base class. Peak dropped to about 240.
• Machine: start Colima with a higher limit, for example colima stop; (ulimit -n 65536; colima start).

If a Testcontainers suite on macOS loses Docker mid-run, check lsof -p $(pgrep -f 'ssh.sock \[mux\]') | wc -l while it is running.

Results

Summary

• Context startup was cheap. Context shutdown, and 31 contexts consuming from the same topics, were the cost.
• @DirtiesContext was compensating for the interference. Keeping only one context’s listeners running removes both the interference and the need to close contexts.
• Unclosed clients in tests plus Colima’s 256 descriptor forwarder limit produce symptoms that look like Docker crashing.