Caching de luxe with Spring and Guava

Summary

We generally don’t optimize expensive operations in code until they create a bottleneck. In some of these cases you could benefit a lot from caching such data. The Spring solution is non-intrusive, highly configurable yet easy to set up, and fully testable. But if your business domain is a bad fit, caching can do more harm than good. Rather than delve into the technical implementations details this post explores the functional ramifications of caching with some practical examples, available in a demo application on github: https://github.com/jaspersprengers/caching-demo.git

If you’re an experienced developer I assume you are familiar with the concept of caching. There are plenty of tutorials on the Spring caching framework, but to my taste they dive too quickly into the nitty-gritty of configuration without first distinguishing the good use cases from the less ideal candicates. Such decisions have everything to do with your system’s business rules. I will present three concrete and very different examples that at first glance are not ideal candidates for caching, but can still benefit from it if properly configured. Then we will look at ways to properly test a richly configured cache implementation. I deliberately leave out the finer details of advanced configuration. You can read all about them in the official Spring docs .

Make a spoonful of broth. Fifteen times.

Sometimes you have to take radical measures to convince your colleagues why some technology is useful and fun, so please bear with me when I start you off with a culinary analogy.

If you take your cooking seriously you’ll keep your pots, implements and jars (no, not jar files) within easy reach, especially when you are going to use them often. You don’t run back and forth to the cupboard – much less open and close it – every time you need to add a pinch of salt, do you now? To stretch the argument to breaking point: when you need to add a spoonful of broth every five minutes to your softly boiling risotto, do you boil a spoonful of water, make the broth, add it to the rice, clean the pan, put it away, and repeat this process fifteen times? Or do you prepare half a liter of broth before boiling the rice? A rhetorical question if ever these was one, yet this is exactly how we write our code most of the time: with repeated calls to relatively expensive operations that return exactly the same broth every time. All because we think in seconds instead of nanoseconds.

Crossing an A4 sheet at light speed

We are extravagantly wasteful with computer time because human consciousness operates in seconds, a pace many orders of magnitude slower than that of computers. Computers work in nanoseconds, which is hardly time at all. A nanosecond is a billion times faster than a second. It is to a second as a second is to thirty years. Light travels the length of an A4 sheet within a nanosecond. Got that?

Usability research shows that any response under 0.1 seconds (100 million nanoseconds) is perceived as instantaneous. We can’t tell if a web page returns in 10 microseconds or 50 milliseconds, and so notice no improvement. That’s how slow we are, even when sober. I recently started caching the results of  a common database query and even without network IO the performance increase was more than twentyfold:

1Local fetch from cassandra database: 2100 microseconds
2  Fetching from Guava cache:           78 microseconds

The figures are naturally much worse with a networked database (that is everywhere but in development) making the case for caching even greater. To make it visual:

In kitchen terms it’s having the pepper within reach (78 centimeter) or having to fetch it from the garden shed.

It’s tempting to ignore performance penalties just because you don’t notice them. It’s also tempting to over-use caching once you get a taste for it. The smart aleck who keeps insisting that premature optimization is the root of all evil has a point. So let’s look at sensible and not so sensible use cases for caching.

The use case from heaven

A little refresher: a cache sits between a source (database/webservice) and a client and builds a lookup table (usually hashmap) of unique keys and values, standing for the distinct input to the source and the return value. When the source is queried again with the exact same input, the cache intervenes and returns the saved value instead. Any non-void method could be enhanced by caching, but the ideal candidate would be a method that:

• behaves like a pure function: input A always returns B without side effects so cached entries never go stale.
• accepts a limited range of inputs (for example an enumeration of all countries), so the cache can never grow beyond the number of entries in that enumeration.
• is expensive to execute in terms of resources or duration and thus makes it worthwhile to cache in the first place.
• is queried often with an even distribution of arguments, so every cached entry is retrieved regularly and evenly.

To cache or not to cache

Real-world use cases are probably nothing like this. You typically cache calls to databases or web services whose return values have a use-by date and therefore should not live indefinitely in the cache. There must be an eviction policy for stale entries. When designing a cache you must know how often the source data is likely to change and – more importantly – whether it’s acceptable to return stale data. This depends on the type of data and who uses it. Accurate readings of physial phenomena change continuously, but if the increments are small it may be acceptable to cache up to a few minutes and return stale data.

Some operations never return stale data but maybe they allow a wide range of input, leading to a bloated cache with ditto memory consumption. What if the input values are not evenly distributed? Then some cache entries occupy precious memory but are never queried and you end up with an in-memory copy of your database. That’s when you know you’re doing it wrong. The Spring tutorial gives an example of a books cache identified by ISBN number . Good as a tutorial but probably not something to implement for real, given the millions of possible ISBN numbers.

A temperature cache

Let’s say that the Dutch Meteorological Office has a hundred online weather stations accessible over a web API that return an accurate temperature reading expressed as a floating point: 18.75° C.

• The readings of the thermometers change continuously, so the cache is always stale. Let’s say it is alright to return ten minute old readings. After that the entry should be evicted.
• There are a hundred possible input arguments (the weather station’s ID) , so the cache size never exceeds that number. No problem there.

A postcode cache

The service that will access our new temperature cache expects a Dutch postcode and finds the weather station nearest to it. A single database table maps all valid postcodes to the nearest weather station and we want to cache those requests. What’t different about this case?

• Postcode to weather station mappings never change, so the cache can never go stale. However…
• Dutch postcodes are expressed as four digits and two capital letters, meaning there are roughly 6,7 million possibilities (9999 * 26 * 26). A disgruntled employee could write a script to try them all out and cause some real OutOfMemory discomfort. Clearly with such a big input range we don’t want the cache to become a memory hog. Let’s assume that a little log analysis has shown that really 95% of queries are for 2000 distinct postal codes. We can then safely set the maximum cache size to 2000 entries and evict those that have not been read for a day.
• Most well-formed postal codes are not assigned to actual streets and therefore not in the database. The cache should be allowed to hold null values for these keys, so the database is not queried in vain  for the same key, whether valid or not.

A stock exchange cache

The last example is a service that queries a remote API to cache the current price for a given share.
DISCLAIMER: I know nothing about financial markets. For example’s sake let’s assume prices changes no more frequent than every five minutes.

• Stale values are not acceptable. A cached entry must be replaced as soon as the source changes.
• The input range (number of different shares) is limited, so no size restriction is necessary.

Can I please see some code???

I know you’ve been itching for this:

1git clone https://github.com/jaspersprengers/caching-demo.git
2cd caching-demo
3mvn clean install
4cd target
5java -jar caching-demo-1.0-SNAPSHOT.jar

This will start up the Springboot demo application, which exposes two endpoints. Supply a valid four digit/two letter postcode for {postcode} (e.g. 1000AA) and for {share} one of AKZO, SHELL, ASML, UNILEVER, GOOGLE or FACEBOOK.

1http://localhost:8080/temperature/{postcode}
2  http://localhost:8080/share/{share}

Spring provides a caching abstraction and leaves the actual storage implementation to third party providers. The default implementation (backed by a concurrent hashmap) is only useful for vanilla flavoured Hello-World-Foobar situations. Luckily Spring provides adaptors for more powerful cache implementations, such as Guava Cache, which we will use here.
The CacheManager is a bean that manages our three caches (key/value maps) and needs to be set up as follows (see nl.jsprengers.caching.CacheConfig)

1@Bean
2    public CacheManager cacheManager() {
3        SimpleCacheManager simpleCacheManager = new SimpleCacheManager();
4        simpleCacheManager.setCaches(Arrays.asList(
5                buildPostCodeCache(),
6                buildTemperatureCache(),
7                buildSharesCache()
8        ));
9        return simpleCacheManager;
10    }

The following three private methods create and configure our Guava caches. Note how all configuration parameters can – and probably be should – be made configurable using @Value annotations. These values are set once during configuration, but there’s nothing to stop you from accessing the CacheManager elsewhere in your code to retrieve and reconfigure the caches at runtime, as we’ll see in the section on integration testing.

1@Value("${cache.postcode.maximum.size:1000}")
2    private int postcodeMaxSize;
3    private GuavaCache buildPostCodeCache() {
4        return new GuavaCache(POSTCODE_CACHE, CacheBuilder
5                .newBuilder()
6                .maximumSize(postcodeMaxSize)
7                .expireAfterAccess(1, TimeUnit.DAYS)
8                .build(),
9                true);
10    }

The postcode cache entries never go stale, but neither should you keep them around if nobody needs them, so after a day Guava should evict them. The size of the cache is limited to a configurable number using Spring’s property injection (default 1000). Tip: if you set the maximumSize to zero you effectively disable the cache, which can be useful in a test run without rebuilding the source.

1@Value("${cache.expire.temperature.seconds:600}")
2    private int expiryTemperatureSeconds;
3    private GuavaCache buildTemperatureCache() {
4        return new GuavaCache(TEMPERATURE_CACHE, CacheBuilder
5                .newBuilder()
6                .expireAfterWrite(expiryTemperatureSeconds, TimeUnit.SECONDS)
7                .build(),
8                false);
9    }

Entries in the temperature cache must be evicted after ten minutes so the service can get fresh values from the weather station. There’s no need to set a cap on the number of entries.

1private GuavaCache buildSharesCache() {
2        return new GuavaCache(SHARES_CACHE,
3                CacheBuilder.newBuilder().build(), false);
4    }

The shares cache is the easiest to configure, because eviction of stale entries is not managed by Guava.

The cached resources

Caching in TemperatureService and PostcodeService is very simple. There’s really nothing more to it than the Cacheable annotation with a reference to the cache name:

From TemperatureService:

1@Cacheable(CacheConfig.TEMPERATURE_CACHE)
2    public float getTemperatureForCoordinate(int coordinate) {
3        return weatherStation.getForCoordinate(coordinate);
4    }

From PostcodeService:

1@Cacheable(CacheConfig.POSTCODE_CACHE)
2    public PostCode getPostcode(String code) {
3        return postcodeDao.findByCode(code);
4    }

The SharesService take a bit more planning because it has to notify the cache whenever fresh information about share prices comes in. The external notification occurs by calling the setNewSharePrice method annotated with @CachePut. At first sight this method doesn’t seem to do much, but Spring uses the share parameter (identified by the key property) and the return value to update the cache entry. Another option would be a void method annotated with @CacheEvict, providing only the share name. This would kick out the entry, after which a call to getValue queries the exchange service and updates the cache. It depends on your setup which is the suitable option. @CachePut probably generates less network traffic.

1@Service
2public class SharesService {
3    private static Logger LOGGER = LoggerFactory.getLogger(SharesService.class);
4    @Autowired
5    StockExchange exchange;
6 
7    @CachePut(cacheNames = CacheConfig.STOCKS_CACHE, key = "#share")
8    public float setNewSharePrice(String share, float nextValue) {
9        LOGGER.info("Share {} was updated to {}", share, nextValue);
10        return nextValue;
11    }
12 
13    @Cacheable(CacheConfig.SHARES_CACHE)
14    public float getValue(String stockName) {
15        LOGGER.info("Fetching stock {} from exchange", stockName);
16        return exchange.getValue(stockName);
17    }
18}

Caching in action

You can see caching in action if you run the application with the application property cache.expire.temperature.seconds to a value of, say, 15 seconds.

1cache.expire.temperature.seconds=15

Here’s a little excerpt from the log when hitting the REST server with two different postal codes at varying intervals. Every call is logged by the Controller class, but PostcodeService and TemperatureService only log when the actual method body is accessed. If a log line is missing, that means the response came from the cache.

Postcode 1000AA not yet cached, station 10 not yet cached:

108:39:41.915 Controller : GET temperature for postcode 1000AA
208:39:41.923 PostcodeService : Getting postcode 1000AA from dbase
308:39:42.070 TemperatureService : Getting temperature from weather station 10

Postcode 1000AB not yet cached, station 10 still in cache

108:39:52.130 Controller : GET temperature for postcode 1000AB
208:39:52.130 PostcodeService : Getting postcode 1000AB from dbase

Postcode 2000AA not yet cached, station 20 still in cache

108:40:04.075 Controller : GET temperature for postcode 2000AA
208:40:04.075 PostcodeService : Getting postcode 2000AA from dbase
308:40:04.077 TemperatureService : Getting temperature from weather station 20

Postcode 2000AB not yet cached, station 20 has expired (>15 seconds since last call)

108:40:22.677 Controller : GET temperature for postcode 2000AB
208:40:22.677 PostcodeService : Getting postcode 2000AB from dbase
308:40:22.692 TemperatureService : Getting temperature from weather station 20

Postcode 2000AB in cache, station 20 has expired

108:40:45.786 Controller : GET temperature for postcode 2000AB
208:40:45.787 TemperatureService : Getting temperature from weather station 20

Postcode 2000AB in cache, station 20 still in cache

108:40:56.426 Controller : GET temperature for postcode 2000AB

Postcode 2000AB in cache, station 20 has expired

108:41:02.293 Controller : GET temperature for postcode 2000AB
208:41:02.294 TemperatureService : Getting temperature from weather station 20

But how do I test all this?

Blimey, in all the excitement we have completely forgotten to test all this cool stuff!

Modern frameworks like Spring Boot remove lots of tedious boilerplate at the price of making your annotation-sprinkled code less deterministic. In short: you cannot unit-test caching behaviour. The @Cacheable annotated methods only work inside the container, so a plain JUnit doesn’t cut it.

In a production environment you need to test all this. You must make sure that your cache does not hog all memory and evicts entries when it needs to. Ideally we want to peek inside the cache to make sure that entries were properly added, evicted and updated. Fortunately you can do all that with Spring:

1@RunWith(SpringJUnit4ClassRunner.class)
2@SpringApplicationConfiguration(classes = {Application.class})
3@WebIntegrationTest
4public class SharesIntegrationTest {
5    @Autowired
6    CacheManager cacheManager;
7 
8    @Before
9    public void setup() {
10        sharesCache = getAndInvalidate(CacheConfig.SHARES_CACHE);
11    }
12    private Cache getAndInvalidate(String name) {
13        //retrieve a reference to the underlying guava cache
14        Cache guavaCache = (Cache) cacheManager.getCache(name)
15                                               .getNativeCache();
16        //clear all entries
17        guavaCache.invalidateAll();
18        return guavaCache;
19    }
20}

This test suite fires up a Spring container for nl.jsprengers.caching.Application. The CacheManager is a bean like any other and can be injected in our unit test. We can retrieve the underlying Guava cache and access the values as a map:

1@Test
2    public void testShares() {
3        float value = sharesService.getValue(Shares.AKZO.name());
4        //the cache should contain a key for AKZO
5        assertThat(sharesCache.asMap()).containsKey("AKZO");
6        //this will cause the cache to be updated with a new price        
7        stockExchange.invalidateAllPrices();
8        float updatedValue = sharesService.getValue(Shares.AKZO.name());
9        assertThat(value).isNotEqualTo(updatedValue);        
10    }

Conclusions

Adding caching to your application can make dramatic improvements in terms of bandwidth, I/O or processor resources, but you must ask yourself two very important questions.

1. Is it acceptable to return stale cache entries?
2. What input can I expect? How frequent and with what range?

The answer to the first question probably lies outside the IT-department.  For the second question a simple analysis of log data will go a long way. Caching is like most other frameworks and tools that promise to make our lives easier: give them a try, but if you don’t stand to gain from them, don’t bother.