Clients and eventual consistency

If you're working with a distributed system it is likely that you use eventual consistency to make your system scalable and robust but what does this means to clients?



In order to build a scalable fault tolerant distributed system you typically have to use eventual consistency. This means that your data is not guaranteed to be consistent all the time but you guarantee that the data eventually will be consistent. So far there should not be any surprises to anyone but what does this means for the clients of your system?

One approach is that your clients are aware of the fact that data will eventually be consistent. This means the client must be OK with inconsistent data. This is probably the easiest approach ignoring potential confusion the users might experience. And it clearly follows one of my guidelines; remember that the clients are also part of your distributed system.

Another approach is to detect an inconsistent state in your service and not present it to the user. The user would either get an error asking them to come back later or the client would see old data. Obviously the benefit is that you are not confusing your users at the expense of more complexity in your service.

Let's look at two examples of what this means. First let's pretend that you have built a banking system and when you transfer money from one account to another you only guarantee that the two accounts will eventually be consistent. If clients are OK with inconsistent data this means the balance of your two accounts might be displayed inaccurately for a short period of time. I.e. as if the transaction only happened on one account. With the other approach users would either get an error or progress bar waiting for transaction to complete.

Second let's pretend you have built a distributed cache where the same key exists on multiple nodes. If you allow clients to accept inconsistencies it means they will sometimes get old data after new data for a short period of time after an update, but then keeping the whole cache consistent or even detect inconsistency server side might be hard or even impossible.

As you can see in these examples there is no easy question. My recommendation would be that in general let clients allow for data to be temporarily inconsistent. This will make your life much easier. And then only for very specific scenarios where it is easy and relatively likely users will be confused by the inconsistency you make sure the client get a consistent view.