In my professional life, I have worked in quite a few different companies, mainly in financial services, and all of the companies that I have worked for follow the same pattern as shown in the following:
- A company is set up by a couple of people with good domain knowledge: insurance, payments, credit cards, and so on.
- The company grows, demanding new business requirements that need to be satisfied quickly (regulation, big customers demanding silly things, and so on), which are built in a hurry with little to no planning.
- The company experiences another phase of growing, where the business transactions are clearly defined and poorly modelled by a hard-to-maintain monolithic software.
- The company increases the headcount that drives into growing pains and loss of efficiency due to restrictions imposed on how the software was built in the first instance.
This chapter is not only about how to avoid the previous flow (uncontrolled organic growth), but it is also about how to model a new system using microservices. This chapter is the soul of this book, as I will try to synthetize my experience in a few pages, setting up the principles to be followed in Chapter 4, Writing Your First Microservice in Node.js, where we will be building a full system based on microservices using the lessons learned in the previous chapters.
A huge percentage (my estimate is around 90%) of the modern enterprise software is built following a monolithic approach.
Large software components that run in a single container and have a well-defined development life cycle, which goes completely against the agile principles, deliver early and deliver often (https://en.wikipedia.org/wiki/Release_early,_release_often), as follows:
- Deliver early: The sooner you fail, the easier it is to recover. If you are working for two years on a software component and then it is released, there is a huge risk of deviation from the original requirements, which are usually wrong and change every few days.
- Deliver often: Delivering often, the stakeholders are aware of the progress and can see the changes reflected quickly in the software. Errors can be fixed in a few days and improvements are identified easily.
Companies build big software components instead of smaller ones that work together as it is the natural thing to do, as shown in the following:
- The developer has a new requirement.
- He builds a new method on an existing class on the service layer.
- The method is exposed on the API via HTTP, SOAP, or any other protocol.
Now, multiply it by the number of developers in your company, and you will obtain something called organic growth. Organic growth is a type of uncontrolled and unplanned growth on software systems under business pressure without an adequate long-term planning, and it is bad.
The first thing required to tackle organic growth is to make sure that the business and IT are aligned in the company. Usually, in big companies, IT is not seen as a core part of the business.
Organizations outsource their IT systems, keeping the price in mind, but not the quality so that the partners building these software components are focused on one thing: deliver on time and according to the specifications even if they are incorrect.
This produces a less-than-ideal ecosystem to respond to the business needs with a working solution for an existing problem. IT is lead by people who barely understand how the systems are built, and usually overlook the complexity of software development.
Fortunately, this is a changing tendency as IT systems become the driver of 99% of the businesses around the world, but we need to get smarter about how we build them.
The first measure to tackle organic growth is aligning IT and business stakeholders to work together: educating the nontechnical stakeholders is the key to success.
If we go back to the few big releases schema. Can we do it better?
Of course we can. Divide the work into manageable software artifacts that model a single, well-defined business activity and give it an entity.
It does not need to be a microservice at this stage, but keeping the logic inside of a separated, well-defined, easily testable, and decoupled module will give us a huge advantage for future changes in the application.
Let's consider the following example:
In this insurance system, you can see that someone was in a hurry. SMS and e-mail sender, although both are communication channels, they have a very different nature and you probably want them to act in different ways.
The calling services are grouped into the following two high-level entities:
At some point, the system needed to send SMSs and e-mails and someone created the communication service entity that handles all the third-party communications.
It looks like a good idea in the beginning. SMS or e-mail, at the end of the day, is only a channel, the communication mechanism will be 90% same and we can reuse plenty of functionality.
What happens if we suddenly want to integrate a third-party service that handles all the physical post?
What happens if we want to add a newsletter that goes out to the customers once a week with information that we consider interesting for our customers?
The service will grow out of control and it will become more difficult to test, release, and ensure that the changes in the SMS code won't affect sending the e-mail in any form.
This is organic growth and, in this case, it is related to a law called Conway's Law, which states the following:
Any organization that designs a system (defined more broadly here than just information systems) will inevitably produce a design whose structure is a copy of the organization's communication structure.
In this case, we are falling into a trap. We are trying to model the communication on a single software component that is probably too big and complex to react quickly to new business needs.
Let's take a look at the following diagram:
Now, we have encapsulated every communication channel on its own service (which, later on, will be deployed as a microservice) and we will do the same for future communication channels.
This is the first step to beat organic growth: create fine-grained services with well-defined boundaries and a single responsibility—do something small, but do it well.
Our brain can't handle complicated mechanisms. The abstraction capacity is one of the most recent human intelligence acquisitions.
In the example from the previous section, I've given something for good, which will upset half of the programmers in the world: eradicating the abstraction of our system.
The abstraction capacity is something that we learn along the years and, unlike intelligence, it can be trained. Not everyone can reach the same level of abstraction, and if we mix the specific and complex domain knowledge required by some industries with a high-level of abstraction, we have the perfect recipe for a disaster.
When building software, one of the golden rules that I always tried to follow (and try is the correct word, as I always find huge opposition to it) is to avoid premature abstraction.
How many times did you find yourself in a corner with a simple set of requirements: build a program to solve X. However, your team goes off and anticipates all the possible variations of X, without even knowing if they are plausible. Then, once the software is in production, one of the stakeholders comes with a variation of X that you could have never imagined (as the requirements were not even correct) and now, getting this variation to work will cost you a few days and a massive refactor.
The way to avoid this problem is simple: avoid abstraction without at least three use cases.
Do not factor in the possibility of sending the data through different types of channels as it might not happen and you are compromising the current feature with unnecessary abstractions. Once you have at least one other communication channel, it is time to start thinking about how these two software components can be designed better, and when the third use case shows up, refactor.
Remember that when building microservices, they should be small enough to be rewritten on a single sprint (around two weeks), so the benefits of having a working prototype in such a short period of time is worth the risk of having to rewrite it once the requirements are more concrete: something to show to the stakeholders is the quickest way to nail down the requirements.
Seneca is great in this regard as, through pattern matching, we can extend the API of a given microservice without affecting the existing calling code: our service is open for extension, but closed for modification (the SOLID principles) as we are adding functionality without affecting the existing one. We will see more complete examples of this behavior in Chapter 4, Writing Your First Microservice in Node.js.