Now that we’ve learned what are the qualities that we are aiming for in a service and how we can connect them to each other, it is time for us to take a closer look at actual techniques, which will help us to migrate from our monolithic application to a microservice architecture. Please take note that the techniques described here are not silver bullets and you will definitely need to modify them for your own use-case, however, experience shows that these general methodologies give an excellent starting point for a successful migration. Keep in mind, some of the steps described in this chapter are parallelizable, so if you have more people to help you, so can speed things up a little.
Before you Start
By the time you’ve reached so far in the book, you probably understand that migrating your monolithic system to microservices is not going to be just a walk in the park. There will be serious human and financial costs of delivery. Even estimating delivery to your stakeholders might be a huge difficulty (at least in the beginning), so let’s run through the basic costs that you will need to calculate with.
Human Costs
Naturally, we are mostly talking about the costs of refactoring your codebase. In the early stages of the project, you will require much more effort than when you’ve migrated multiple components. Be very conservative with your estimations in the beginning and get a bit more strict with yourself and your team after you have the tooling in place that we will be talking here in chapter 5 and later in chapter 6.
- 1.
Operations-related - How to deploy and how to scale your new services can always be a critical and dividing question when moving to a new type of architecture. Deployment of your monolith might have been a person running a couple of scripts to rsync data to a remote server and then restarting the application, monitoring (which is a very important aspect when you’re changing your infrastructure in such a big way) might be rowing in the same boat. When you move to microservices, this might not cut it in the long run. At minimum, you will need to collect these executables and organize them in a usable way so that others in the company have access to it as well. We will cover more operations related topics in the next chapter.
- 2.
Code-related - You know when code is messy like a bowl of spaghetti and not neatly organized as a well sliced pizza? Keeping your codebase clean in a high velocity environment where there’s constant push for delivery can be a huge challenge. Unkempt code can be one of the other big costs when you would like to migrate.
Depending on the size of your company and monolithic application, it is a good idea to have a dedicated team that takes care of tooling, documentation, guidelines and best practices for the other teams that have the domain knowledge for the migration of their components. If you’re operating with hundreds of engineers with a monolith of millions of lines of code, this is practically a must have. If your scale is somewhat smaller, it could be a convenience.
Some companies like to implement emergency or “tiger” teams when a critical component needs to be migrated due to resiliency or other concerns. This could be a good way of moving a lot of software into different systems, however, it is highly recommended to pay attention to the handover of the code and implement intense knowledge sharing sessions between the migration and the maintenance teams.
Now let’s see the sort of hardware and infrastructure costs that we will need to implement.
Infrastructure Costs
Moving to microservices can have another expensive implication, which is the cost of having enough machines to run the new system (and, for a while, the old system as well). What does this mean exactly? Let’s consider the following scenario:
Our tizza application that runs on two 10 core machine with 128 gigabytes of RAM powering them. During the migration planning, we’ve identified 6 systems that we can logically shard our application into. Now, let’s do the math:
Depending on the load on the systems we will need either single core or dual core machines for the new services. Probably the system that handles authentication and likes will require 2 cores and 8 gigabytes of ram, whereas the pizza metadata storage might just take a single core and 4 gigabytes of RAM. We can average the number of CPUs for the entire cluster to 8 and the total memory costs to 32 gigabytes. Since, we used to work with 2 machines for the monolith, we should also raise the numbers here, we don’t want to decrease resiliency after all.
As you try to scale down your systems into smaller but more efficient pieces, it is a very human reaction to underscale your cluster and underestimate the amount of raw power that is required to run your software safely. A general rule of thumb that I like to follow when creating new microservies, is to run 3 copies of the service on different (virtual or physical) machines for high availability.
For confident people, the above statement can be eliminated with a great cloud provider, super lightweight applications and a well configured autoscaling system.
Note
Autoscaling is when you define rules about the number of servers you would like to run in your cluster. The rule can be a function of memory or CPU usage, number of live connections to your cluster, time of day, or other values that your cloud provider might allow you to use.
As you can see, we’ve raised the total number of cores in the system from 20 to 24 and the memory has remained at about 128 gigabytes, with a grand total of 96. You will quickly notice that these numbers tend to grow quicker than expected in a real life environment and depending on your provider, it might cause devastating costs to your business.
My advice is to overshoot in the beginning for safety and revisit your application every now and then to make sure that the hardware is not an overkill for the software.
I Made the Call, What’s Next?
Technical debt as a first-class project citizen: Oftentimes people think that these sorts of changes will require big projects where multiple people to collaborate at a grandiose scale. As you will see from this chapter, it’s not the case. The first advice I can give is to move technical debt and refactoring into feature projects that you’d need to deliver to the company. Make sure that you’re transparent and reasonable about this so you can do it in the future as well. Also, if you receive a no, that’s fine as well, just be persistent about it, it’s all about the conversation. Putting technical debt related tasks into feature projects enables engineers to be more efficient with both areas. There are less context switches, making both the feature development and the technical debt work more efficient. The engineers will also be happier, since they will leave higher quality work behind themselves.
Measure your results: If you were able to squeeze in some refactoring here and there, show your colleagues how much easier it is to use your new database interface or how much faster it is to deliver a new feature with the functions that you’ve extracted and make sure to tell your product owner or manager as well. If you have metrics to prove that your work has been worth it, it’s even better. These metrics are often hard to find and come up with, some of them could be related to application speed, some of them related to delivery speed (i.e. how fast does a new feature get shipped because we made this and this technical change in the service), or even the number of bug tickets that get addressed to your team.
Be candid: Make sure that you measure and explain the costs to all stakeholders and you do it honestly. This will be a big project, no point in making it look small. People do need to understand that feature development will slow down for a while and the processes around it will be different in the future.
Sometimes no is fine: It’s completely possible that your company is just not ready for a grand scale migration like this. In this case, make sure that you and your company are as successful as possible, so you can have resiliency problems in the near future. In the introductory chapter, we’ve seen the catastrophe scenarios for a streaming application. A shock like this can cause a company to change their mindset, you will, however, need to reach that scale first with the business. If you receive too many “no”s, then it’s probably time to rethink your own scope and reduce it to the smallest that you can do, to show the company how the process would look like and what the value of it is.
As you can see, making a call like this can be very difficult for a company in many ways. The best strategy is oftentimes to be patient, leave your frustrations behind and employ refactoring techniques and tools that you’ve acquired through this book, they will come in handy in the future.
Now that we are done with understanding the costs of the errand, it’s time to start migration our application, first, by preparing the data.
Data Preparation
Before we get into the juicy parts of refactoring our application, we need to make sure that the data we would like to transfer is, well, transferable, meaning that it is easy to copy it from one place to another and is not coupled too much with other data domains in our system. In addition to this, we need to find the clusters of data that seem to live together from a domain and business perspective.
Domain Sharding
User related information
Pizza and restaurant information
Likes and match groups
Events and event related integrations
Why is this beneficial? Well, now the objects are less coupled and more trust-based. Pizzas will trust the system that there’s a user with a given user profile id and can live in their own separate environment. There are no hard coded database rules that will bind pizzas and user profiles together anymore, which is very liberating, but also very scary at the same time.
Cascading deletes are gone. You will need to manually delete linked objects.
Some convenience methods provided by the Django ORM, such as select_related, are not available anymore.
Naturally, you can (and should) keep coupling between models that reside in the same database, so you retain the convenience methods and with it some speed and reliability for your queries.
Example migration from model to id
- 1.
We add a new field called owner_user_profile_id to the table. We set it as nullable, so the migrations can create it with no issues whatsoever.
- 2.We run a set of Python code that will set up the values for us accordingly:
- a.
The set_defaults function fetches all values from the already created pizzerias and adds them to the new field. Just what we need.
- b.
If we really need to, we can specify a reverse for this function. It will not be needed for now.
- a.
- 3.
We alter the owner_user_profile_id field to be non-nullable.
- 4.
We drop the owner field for good.
You can use the above template to almost all migration files. For tables with a high number of rows (i.e. it would be dangerous to load the entire database into memory), it is highly recommended to change the query in the set_defaults function to a bulk operation. Alternatively, for really big tables (we are talking about millions of business critical rows here), you might want to involve a database expert to aid with the migration.
Using properties as model fields
Using properties the above way can greatly speed up the migration process, however, it might cause issues in the long run, especially regarding performance, since we just moved from a very efficient database JOIN operation to another query that gets executed. However, you will later notice that this is not the biggest hit in speed that we will receive.Let’s take a look at the next steps of the migration, where we will make sure that data will be accessible to both the old and the new systems.
Database Duplication
After you’ve decided which part of your application you would like to migrate and modified the database accordingly, it is time to set up a migration plan on the database level. I.e. it is time to prepare your new database that will host your models.
Probably the easiest way to get started with this is to set up replicas of your main database. The idea is that all writes will be copied to the replica, which will be used as read-only. Don’t worry too much about setting up the replication for specific tables only, most of the time is only causes a headache and extra work. It’s usually easier to just set up a full replication and just drop the tables from the new database that you won’t need when the migration is ready.
Note
You can also set up master-master replication between the 2 databases, however, the technologies for this requires a lot of database expertise and give more room for error post-release.
Depending on the size and type of your database, the replication can take from minutes to days, so make sure to add this to your estimations when communicating to your line manager and team. To get started with this, you might want to take a look at how Amazon RDS does data replication. If you want to go a little bit deeper into the technology, there is great documentation on dev.mysql.com on how to setup replication for MySQL, and on the Postgres Wikipedia for Postgres.
Testing and Coverage
We’ve done some preparation. Now it’s time to copy all the code… Just kidding. Ideally, this is the point where you make sure that your application will not break when you migrate code from one system to another.
To achieve this, the single most useful tool that you can use is testing. Django comes with it’s own built in test framework with which you can test even the database level quite easily with it’s included in-memory database, however, any unittest framework will do the job, such as unittest2, pytest or nose.
When it comes to how you can measure if you’re doing well on the testing side, many teams and engineers recommend using tools like coverage, with which you can measure the number of lines of code you’ve tested in your application. However, this metric does not always measure the true value of your tests. The recommendation is that you cover the core business functionalities of your views and models. Ideally, if you’re running backend application with some external communication methods exposed, you can also implement integration tests which would test the functionalities provided by your entire endpoint or consumer. If you have the people, then you can also implement acceptance tests, which are usually very high level tests where an automated robot is clicking through your website and checking if the base user flows are successful or not. These are usually quite fragile and expensive to maintain, however, they can be life-savers as the last line of defence before a critical bug goes into production. An excellent acceptance testing framework is cucumber, you can read more about it at cucumber.io.
Now that we have covered our code with tests, it’s time to start working on some tools, so we can migrate the codebase from one place to another.
Moving the Service
So far we did some work on the models that we wanted to migrate and prepared a new database. It’s time to start working towards actually migrating the code.
Remote Models
Problematic function to extract
No matter which service we would like to extract the above code, we will face a dilemma. There are cross references to the models in the function which can be difficult to solve. If we want to avoid data duplication and clean the domain clean, we need to make sure that likes and user profiles don’t reside in both separate services and databases. For this, we can do a refactoring technique that we’re going to call remote models.
Remote models are a concept I’ve come across multiple times in my career and they are a real lifesaver. The idea is, that if your apis are uniform, you can very easily replace your database model calls with remote calls using a simple search and replace in your codebase (at least in most cases). See Listing 5-4 for an example remote model implementation.
Note
The code we will be looking at might not fit your needs perfectly, but it’s a good starting point and exercise to start thinking with remote models.
The basic remote model
That’s a lot of code. Let’s take a closer look at it. The first thing that you might notice is that the RemoteModel class’ interface exposes a mixture of Django models and standards that we’ve established during our exploration of the REST framework. The get, filter, delete, create, update methods expose a Django model like interface for the sake of simple refactoring and domain familiarity, however, the implementations themselves involve a lot of words that we’ve encountered when we were examining the REST paradigms.
The ENTITY_BASE_URL_MAP is a convenience map that you can create in your settings file to specify unique url bases for each entity that you’re working with.
All of this is quite simple so far. So where’s the trick? You might’ve noticed that the request object is a required parameter when you’re creating an instance of the remote model. Why is this? Simply put, we are using the request object to propagate the headers that we’ve received in the request itself. This way, if you’re using headers or cookies for authentication, everything will be propagated without any issues.
Simple remote pizza
Examples of remote pizza usage
Note
The filter function will require additional implementation on the server side, since the Django REST Framework does not support them by default.
Remote models can be slow - Depending on the network, the implementation, the hardware and sometimes the alignment of the stars, remote models can be much-much slower than their database counterparts. This slowness can also escalate as you start “chaining” remote methods over your architecture, by calling systems that call other systems that call other systems, etc.
It’s more fragile - In general, remote models are much more fragile than regular ones. Connections to the database are much more robust and enduring than connections that you do over HTTP.
Bulk operations and loops need to be reviewed thoroughly - Sometimes unideal code gets copied during the migration process and, lets say, a for loop that had a database call through models in it becomes a HTTP call through the remote models. Due to the first point, this can be devastating if we’re querying a large number of models.
There’s no serialization - If you’re using this simple model, you will definitely lose the power of serialization, meaning that you will only receive a dict back as a response and not necessarily a model or a lost of models that you’d be expecting. This is not an unsolvable problem, you can look into Python dataclasses and modules like dacite.
Another good topic that comes up during the implementation of remote models is caching. Caching is quite a difficult problem to solve, so I recommend you not to implement it in your first iteration. One easy and big win, that I’ve noticed over the years is to implement a request-level cache in your service. What this means, is that the result of each remote call is stored in some way on the request and does not need to be fetched again from the remote service. What this allows you, is to do multiple remote model calls to the same resource from your service in a view function, yet not actually use the network to get the resource. This can save a lot of network traffic even in the beginning.
Let’s take a look at exercise 5-1, 5-2 and 5-3 that will help us work with remote models a bit more.
Exercise 5-1: Remote Models for Service to Service
The above model solves the issue of header and cookie propagation well enough so that we can access data from various points of the system using authentication methods such as sessions or authentication headers that we’ve looked at in the previous chapters. However, this could cause issues if we would like to use different tokens when we would like to call services without user authentication. In this exercise, you’re encouraged to design an extension to the RemoteModel with which we can assign override authentication tokens properly. There’s already some code in place above which you can use.
Exercise 5-2: Remote or not Remote
Remote models seem like a very powerful tool already, but can we make them even more powerful? Try to extend the RemoteModel class to be able to handle database calls when the model is available in a local database. Doing this change can enable you to speed up migrations in the future.
Exercise 5-3: Request Level Cache
We’ve mentioned request level caching before, now it is time to test our Python and Django knowledge and implement it. Each time a remote model action is called, make sure to store the response in a cache that is tied to the request itself. You can use various caching libraries, like cachetools for this.
Working on our tooling was a lot of fun, time for the code migration.
The Code Migration
Keep it simple - At this point, no need to worry too much about code duplication between services (unless you already have some tooling in place for this). Just make sure that your application gets up and running as soon as possible. We are going to delete the code in the monolith anyway.
Follow the domain - Just like with data sharding, domain is key here as well. Make sure that the module that you’d like to move out is as isolated from the system as possible. What you’d like to aim for, is just copying an app from one codebase to another as-is.
Tests are key - Some microservices that you create are monsters in themselves. For example, you might have a payments service that has an internal state machine and multiple integrations to various payment providers, you’ve made the decision to extract the entire domain in one piece. Make sure that your tests are in place, running and eventually not breaking. Testing such massive systems by hand is nigh impossible. Tests can also aid your in your migration, if you’ve missed some code or functionality here and there.
Accept that you will lose speed first - It’s one thing that the migration takes a while, but applications usually become slower during the early stages of their lifecycle. This is caused by the above negatives that we’ve examined with remote models. What you will notice, is that on the long run the owner team will take good care of their application and implement various speed enhancing features as the most knowledgeable engineers in their domain.
Release
The code is copied, all the tests are passing, it’s time for the release.
Strategies
The first deployment of a new microservice is always a little messy and involves a lot of conversation around strategies and methodologies beforehand. Just as in most places in this book, there’s no silver bullet for the release process, however, there are a couple of methodologies you can choose from, depending on your preparation, tools and the willingness of your team to wake up early.
Read first, write later - This strategy means that the microservice will first only run in read-only mode, meaning that traffic on it will not modify the data owned by it. One of my favourite strategies, which allows you to use both the monolith and the new microservice to access data at the same time. If you’ve chosen to set up read replication to your new database, well, it should be quite safe to use the APIs from the new service the provide read functionalities, for example fetching pizza metadata. This way, you can make sure that your application is running in production and only start writing data in it when you’re confident that your infrastructure can handle it.
Rolling deployment - Basically means that you will send a percentage of your total traffic to the new microservice and leave the rest on the monolith, slowly but surely letting all traffic to be handled by the new system. With modern load-balancers and service meshes, this can be easily set up. This is not an option if you have chosen to create a read replica, since the writes that would happen on the new microservice’s database would not get registered in the monolith’s database.
Full traffic change - Probably the easiest to achieve and the fastest to revert. When you’re confident that your service works, you switch the traffic on a given url to the new service. The process should be simple and easily reversible, such as changing a configuration on a website or a file.
Note
Naturally, there are many other release strategies that we could be talking about here. The main idea is to have context around the options that you have regarding risk, difficulty and preparation time so you can make an educated decision on how you want to tackle this problem.
Now that we have an idea about what strategy we’d like to use to release our service, let’s take a look at how we are going to react when things inevitably break.
Handling Outages
Create a playbook for reverting - Probably the most important thing you can do. Make sure that you have a step by step guide for the engineers to revert the traffic to the monolithic application. It might seem trivial to do at first, but things can go really bad in live environments, especially in mission critical services, like pizza metadata. Make sure to practice the playbook as well, and involve other teams to review it.
Logging and monitoring should be in place - Your logging and metrics should be in place and properly monitored during the release time, both on the monolith, the new service and the databases as well.
Choose the time and place - Ideally, such a release should happen during a low traffic time, you know your application best, so choose the time accordingly. Monday mornings or Saturday mornings are usually a good choice for such migrations in general. If you have the chance, have people from the owner team and (if exists) the platform team on premise for efficient communication.
Practice on staging - Something that many teams forget is that there’s usually a pre-production or staging environment for their system. You can utilize this space to practice the release a couple of times, since there’s ideally no real customer data there.
Let the rest of the company know - This is a crucial step, make sure that public relations and the customer care team know about the maintenance that is coming up and the possible impact on customers. The more they know, the more effectively they can communicate if something goes bad.
Don’t forget about the data - Make sure that you have a plan for the data backfill as well, since it’s possible that during a problematic release, there would be data discrepancies between the monolith and the microservice database.
- 1.Prerequisites:
- a.
Make sure that you’re connected to the VPN.
- b.
Make sure that you have access to http://ci.tizza.io.
- c.
Make sure you have ssh access to the requested machines.
- d.
Have the latest https://github.com/tizza/tizza cloned on your machine.
- a.
- 2.
Announce on #alerts channel with @here that there’s an issue with your release and a revert is required.
- 3.
- 4.
Select the version of the application that you’d like to deploy and hit the green button.
- 5.
If the deployment tool reports failure, continue.
- 6.
ssh into the host machines, you can use ssh -A <host ip>
- 7.Run the following commands:
- a.
sudo su -
- b.
bash -x ./maintenance/set-application-version.sh <application version>
- c.
supervisorctl restart tizza-app tizza-nginx
- a.
- 8.
If the service still doesn’t respond, call +36123456789
This playbook is quite simple, however, it offers multiple options for the developer to fix the situation. It includes a prerequisites part, so the developer who runs these commands can make sure that they can do everything that the playbook requires. It also includes a catastrophe situation solution, where a phone number is provided, which is most likely linked to an experienced developer in the field.
There’s also a communication plan for the rest of the company as the second step. This is absolutely crucial, since the rest of your company will be interested if something went amiss.
We’ve done it! The application is migrated, however, we are not quite ready yet. The most fun part is still coming up. Let’s talk about how to make sure that we don’t leave a huge mess behind.
Cleanup
The graphs and logs look great. Customers are not complaining about any new issues, the system is stable. Congratulations, you’ve just released a new microservice! Now comes the most fun: cleaning up the mess that we’ve left behind.
Just like you would do with your kitchen, make sure that you don’t leave behind unwanted things in the old codebase. You can take your time with this, as a matter of fact, it is usually a good idea to leave the old code in place for 2-3 weeks, so if there are some issues you can still revert to the old logic using the playbooks that you’ve created.
Turn off replication between the monolith and the microservice databases - If you haven’t already, you can turn off the data replication between the two databases now.
Remove unused tables from the new service - If you went with a simple full database replication, you can now delete the tables from the microservice’s database that are not involved in the domain. This should free up plenty of storage.
Remove unused code from the monolith - Time to remove the modules that are not used. Make sure to do a clean sweep,utilize tools like pycodestyle to find unused code that can be removed.
Remove unused tables from the monolith - Now that you’re certain that no code accesses the tables that have been migrated to the new service, you can safely drop them. It might also be a good idea to archive this data and store it for a while, doesn’t cost much.
Conclusion
We’ve learned a lot in this chapter about small techniques that you can use to speed up your microservice migration. We’ve also made a huge mess in our new system in the meantime. There’s a lot of duplicated code and it’s still not clear who is owning what parts of the application. In the next chapter, we are going to dig deeper in this conversation and make sure that we can not just increase the number of services we have, but also scale our organization and the development for optimal efficiency with these systems.