Understanding and Optimizing the Context Cache in Spring Test

This is a personal note that I decided to share. It reflects my understanding of a subject and may contain errors and approximations. Feel free to contribute by contacting me here. Any help will be credited!

---

A few months ago, we decided to work on improving the performance of our test suites, comprising over 10,000 integration tests spread across around forty services and microservices. The aim was to reduce CI costs and improve the feedback loop so that we could deliver more, faster. We reduced build times with tests by around 40–60% across all services, with a record reduction of around 90% for one microservice. As the cost savings were proportional, we cut our bill by almost half.

For months, we focused on feature implementation and product improvement, relying on internal tooling for our tests: utility methods, custom annotations to launch MongoDB, S3, and Kafka in test containers, and ready-to-use configurations. To move quickly, we had deliberately set aside certain performance considerations, particularly regarding the context caching mechanism in the Spring testing framework.

In this post, I’d like to share a few considerations for carrying out this type of optimisation.

Thanks to Mickael LAMASUTA for his contribution to this work, particularly on the implementation of optimisation strategies and the associated performance tests.

The context cache mechanism

Spring’s documentation on this subject is very well written 1. I will mention here only a few points that I consider important.

In test suites that use Spring contexts, as is the case when using @SpringBootTest, Spring caches the contexts so that they can be reused from one test to another, and from one test class to another. This avoids having to create contexts from scratch, which is costly in terms of time and resources.

To achieve this, Spring uses a cache of 32 contexts by default and employs LRU (least recently used) management. You can force a context to be evicted from the cache by applying the @DirtiesContext annotation to a class or method.

public class DefaultContextCache implements ContextCache {
 /**  
  * Map of context keys to Spring {@code ApplicationContext} instances.  
  */
  private final Map<MergedContextConfiguration, ApplicationContext> contextMap =  
        Collections.synchronizedMap(new LruCache(32, 0.75f));
        
    ...
}

The identifier for a context in the cache is an instance of MergedContextConfiguration. This object acts as a context descriptor and therefore allows us to determine whether two Spring contexts are identical. By looking at the class’s hashcode, it is easy to see what defines a context.

@Override  
public int hashCode() {  
    int result = Arrays.hashCode(this.locations);  
    result = 31 * result + Arrays.hashCode(this.classes);  
    result = 31 * result + this.contextInitializerClasses.hashCode();  
    result = 31 * result + Arrays.hashCode(this.activeProfiles);  
    result = 31 * result + this.propertySourceDescriptors.hashCode();  
    result = 31 * result + Arrays.hashCode(this.propertySourceProperties);  
    result = 31 * result + this.contextCustomizers.hashCode();  
    result = 31 * result + (this.parent != null ? this.parent.hashCode() : 0);  
    result = 31 * result + nullSafeClassName(this.contextLoader).hashCode();  
    return result;  
}

• Parent Context: The parent configuration in a context hierarchy declared with @ContextHierarchy.
• Configuration Sources: Context definitions, primarily in XML files via @ContextConfiguration(locations = ...), and/or @Configuration classes via @ContextConfiguration(classes = ...). (These are the locations and classes fields.)
• Context Loader Type: The ContextLoader responsible for building the context, resolved in particular via @ContextConfiguration(loader = ...).
• Context Initialisers: The implementations of ApplicationContextInitializer registered via @ContextConfiguration(initialisers = ...).
• Context Customisers: Modifications applied to the context, used by features such as @DynamicPropertySource, @MockitoBean, or @MockitoSpyBean.
• Active Profiles: Spring profiles activated with @ActiveProfiles(...).
• Test Properties: The properties and property file locations provided by @TestPropertySource.
• Resource Path: The root path of the web resource provided by @WebAppConfiguration (as part of WebMergedContextConfiguration).

It is easy to see the simplest and most straightforward actions to take to maximise the reuse of cached contexts:

• The active profiles and the order in which they are activated ({profileA, profileB} and {profileB, profileA} will result in two different contexts in the cache).
• Declared mocks and spies. With the @MockitoBean and @MockitoSpyBean annotations, creating mocks is made extremely simple. If care is not taken, these mocks and spies generate new contexts and degrade the performance of test suites. One way to streamline this is to group mocks and configurations into shared abstract test classes; this involves the use of inheritance, which some advise against 2, but which effectively limits the proliferation of contexts.
• Standardise the management of property sources and resources.
• Avoid using the @DirtiesContext annotation.

Note that the cache size can be adjusted via a spring.properties file, which is useful if tests are run in constrained environments and to prevent too many third-party services from being launched in parallel.

spring.test.context.cache.maxSize=4

To understand how the cache is used in your test suite, you can log a range of information by changing the log level (see the documentation 1).

These initial steps deliver direct performance gains, provided that the contexts are streamlined and their number reduced.
However, as test suites become more complex, with the use of test containers and/or embedded third-party dependencies, other considerations come into play.

Considerations

Very often, test suites, particularly integration or end-to-end tests, require a number of third-party dependencies: a database, an authentication system, a message broker, a cloud service, and so on. Historically, developers often used ‘embedded’ versions of these services, which were more or less comprehensive (de.flapdoodle.embed.mongo, @EmbeddedKafka, etc.).
In recent years, containers have replaced them, offering greater portability, stability and reproducibility.

Whatever you use, it is important to understand the lifecycle of these services and how they integrate with test suites.

Let’s take Testcontainers as an example: they can be used via JUnit extensions, which are independent of the Spring context (@TestContainers), or within the Spring context (@ContextConfiguration(initialisers = …), @ServiceConnection, @DynamicPropertySource…).
If your MongoDB test container is configured within the Spring context and the cache size is set to 32, you can expect to have 32 MongoDB test containers running in parallel (be mindful of the resources required). For a database, it is feasible to reuse the container throughout the test suite, provided that data clearing is properly configured.

What applies to a database does not necessarily apply to a message broker. If you have a Kafka instance in a test container and wish to reuse it throughout your test suite, this presents numerous challenges. Clearing the data is not straightforward, as consumer groups remain active in cached contexts. This can lead to overlaps, messages being consumed across multiple contexts, and other issues. It may therefore be preferable to have one Kafka test container per Spring context, but be mindful of the number of contexts in the cache.

We ran into this problem. We started out with one Kafka instance per context, interspersed with regular @DirtiesContext annotations. We tried using a shared instance, as with MongoDB: this was unsuccessful. The impact on cached contexts was too high. We ultimately retained one Kafka instance per context, but removed the @DirtiesContext annotations by replacing our message production/consumption assertions with offset delta assertions, which are more tolerant of test sequences within the same context. To avoid having too many Kafka instances, we deliberately limited the cache size to a maximum of four contexts, without compromising performance on our CI runs.

These two examples illustrate that, to build a high-performing test suite without @DirtiesContext, while maximising cache utilisation without over-consuming resources, one must have a detailed understanding not only of one’s test configuration but also of the third-party services being used.

Conclusion

It is not easy to strike the right balance between performance, implementation costs, runtime costs, the resources required, and what we want to test. To save time, we are often tempted to create a customised setup for the tests we are writing. It is easy to add a mock, modify specific properties… Gradually, this multiplies the number of contexts and reduces the cache’s effectiveness. It is therefore worthwhile to streamline the definition of contexts, standardise test production using common mechanisms across the organisation, and understand what is happening within the framework and third-party dependencies.

In the era of AI-driven development, feedback loops are important, and their speed is crucial for maximising production capacity. To maintain good governance and avoid generating tests that are too disparate or underperforming, it may be worthwhile to understand the issues addressed in this article.

References

• Spring Framework Documentation
• Three Reasons Why We Should Not Use Inheritance In Our Tests - Petri Kainulainen
• Spring Framework - Github
• Spring Boot Test Context Caching: The Complete Guide - rieckpil.de