Data Consistency

In our monolithic application, all data can be stored in a single database, and even if we partitioned it, we probably partitioned the data in such a way that we could still use the database to enforce referential integrity and use transactions. In our microservices application, the data is probably distributed across multiple databases, and in addition, they may be entirely different types of databases.

As we decentralize our data and move to a model where each service has its own schema or database, this causes data consistency and integrity challenges that can be hard to solve. Data in one service can reference data in another service where maintaining integrity between both is important, and if data changes in one database, it must change in the other.

What if we reference some data in another service that is deleted, or maybe the data changes and we need to be made aware of this? If we replicate data, how do we keep it consistent and who owns it? If we cache it, how do we invalidate caches?

Dealing with these consistency challenges, and concepts like eventual consistency, can be extremely hard to get right. Although dealing with these data consistency and integrity concerns can be quite challenging, there are some techniques we can use. We can use a notification service to publish changes to data where consumers can subscribe to these changes and updates. Another approach would be to set proper TTLs where some defined and acceptable inconsistency can be tolerated. We can implement the services in a way that they can deal with eventually consistent or inconsistent data.

Many of these challenges are not new to microservices, as we commonly need to partition and replicate data in large systems as a way to increase scale and availability.