Caching

The Idea

Your application is slow because data is calculated in real time and the calculations are expensive.¹ A simple way to improve performance is to hold the results in memory (the cache).² Now, when you get a second request for the same data, you just serve the results from the cache, instead of going through the calculations again. So your application is much faster!

But, is this actually a good idea?

Performance

For performance analysis we have to consider two cases: cache hits and cache misses.³ A cache hit is the case when we already have the result in the cache. We no longer have to go through the expensive calculations and just return the result from the cache. Obviously, accessing memory is a very quick operation, so clearly performance is great.

A cache miss is the case when we don’t have the results in the cache. We still have to do the calculations. Hence performance is just as bad as ever.

Now what does this mean for your application? It depends.

• If in your application the vast majority of requests are on a tiny subset of data⁴ and your application can live with the eventual slow response, caching can be a great strategy.
• If on the other hand the requests to your application differ greatly and frequently⁵ and you cannot live with the bad performance of misses, caching is not the right strategy for you.

Dirty Reads

A dirty read is what happens when you return results from the cache while the underlying data has changed already. So basically you return the results very quickly, but they are outdated.

Is this a problem? Again, it depends. If e.g. you are caching GPS data about plate tectonics for a couple of minutes you are probably fine. These plates tend to move slowly – so most likely your data still is accurate or very close to being accurate. Hence the data is “good enough”.

If on the other hand you cache price information about highly volatile stocks in a high frequency trading system, you might run into problems, as you take decisions based on wrong data.

The important part is to be aware of the challenge of dirty reads and make a conscious decision about it.

Invalidation

In almost all cases the data in your cache eventually will be outdated and should no longer be used. There are several possible strategies to ensure so and depending on the context they can be quite complex. However, the basic approaches are as follows:

• Validity period: e.g. consider the data in the cache valid for 30 minutes. If you hit a cache and the result is older than 30 minutes, you disregard the cached result, calculate a new result, and update the value in the cache.
• Invalidate caches when the underlying data changes. Whenever you update the underlying data, you also check which cached values are impacted and invalidate these values.
• Purging caches: whenever the underlying data changes, you just invalidate all cached data. This option is simpler to implement than the previous one. On the other hand it has a bigger performance impact, as you now have to rebuild the complete cache.

There is no single best strategy. It all depends on the problem at hand.

Cost Of Caching

Caching makes the code more complex, harder to change and maintain, and introduces its own challenges and pitfalls.⁶ So, caching always comes at an additional cost. Also, in all my years in software engineering I don’t think I’ve seen a single implementation of caching that did not lead into some kind of problems further down the line.

One pitfall I’ve seen particularly often is that at some point some part of the application decides to use real time data instead of using the cached data. Resulting in different data being used in different parts of the application, leading to all kinds of conflicts.

So, whenever you decide to use caching, ensure to make it a conscious decision and be fully aware of all the costs and risks associated with it.

Caching Checklist

I suggest following this checklist before introducing caching:

• Do you actually have a performance problem?⁷
• Does the cache actually address the performance bottle neck?
• Can you improve the underlying performance without resorting to caching?⁸
• Is worst case performance important to you? If so, caching is probably the wrong approach for your problem.
• Can your application deal with outdated data?
• What cache invalidation strategy is right for your application?
• Are you ready to pay the additional cost in development time and complexity associated with caching?

1. More generally caching can be applied in all cases where accessing data is expensive. E.g. when data is gathered through a slow web service or an expensive database query.
2. Alternatively the cached results can be held e.g. in dedicated hardware or a database. The important part is to use a technology that allows quick access.
3. Obviously this assumes we don’t have all our data in the cache. If you find yourself caching all your data, this might be a hint that something with your architecture is not quite right and you should reconsider your approach.
4. Hence you have a lot of hits and very few misses.
5. I.e., you have lot of misses.
6. As does pretty much any optimization, see also Performance.
7. I’ve seen so many optimizations introduced to resolve a suspected problem, which did not exist in the first place.
8. Sometimes you can e.g. just speed up the calculations by using a more efficient algorithm or speed up a database lookup by adding a couple of indexes.