Photo by Inja Pavlić on Unsplash

Microservices — your milage may vary*

I recently came across an old blog post where Istio — a service mesh for microservices — made a case for moving away from a microservice based design for their control plane to a monolithic single daemon approach — istiod. An article by Christian Posta on when not to do microservices provides more insight into this decision.

I believe that everyone reading is at different stages in their own microservice journey. As is the frequently the case with most thing in life, there is rarely a one-size-fit-all approach that works for everyone. Microservice are no different. How successful you are with your microservice implementation will be depends on variety of factors — least of it being the technical architecture you decided.

Below I share some insights I’ve gained running an Enterprise-scale microservice platform.

The Distributed Monolith!

As you start to chip away at your existing monolith by standing up multiple stand-alone services — you could end up building a Distributed Monolith if your are not deliberate in your decisions.

Ask yourself these questions to make sure you are on track.

A. Can each of your service scale and fail independently?

A big benefit that everyone trouts about the Microservice architecture is its ease of scalability. The core premise being the fact that since these services are much more light-weight compared the monolith counterpart, you can quickly & easily spin up additional instances as demand rises.

In reality, this requires a lot more effort and attention — as soon as you rollout out your second microservice.

Scaling up Service A mean all of its downstream dependent services have to support additional volume as well. An inability for any of the downstream service to handle this additional volume will limit ability of Service A.

Similarly, a failure in those downstream dependencies would cause cascading failure within other services.

As a result, if not designed for, under load the microservice architecture would perform worse than its monolith-peer.

There are various design approach that can be considered

1. Keep related data & endpoints (“bounded context”) together within a microservice so they can scale together.
2. Utilize back-pressure & fail-open strategies to prevent cascading failures.
3. Understand the runtime dependencies of your services, hotspots and singletons within your architecture (e.g. Kafka cluster, Redis cluster, etc)

B. Are your services too chatty?

Domain and service boundaries are difficult to ascertain easily. And as business needs change, new requirements pour in; these boundaries shift. I bring this up to make the point that getting a clean encapsulation around your domain data & its functions a.k.a your “microservice” is a hard problem.

If you have chattiness between 2 individual microservice it essentially means you have shared state/functionality that is escaping the boundaries of your service.

A tool to help you in effort is the utilize a Domain Driven Design approach to identify the “Bounded Context” within your application and the corresponding aggregate & aggregate root. What falls within the boundaries of this context are candidates to be kept together within a single microservice.

Also important for teams starting on their microservices journey is to be mindful of the temptation to create a new service for every new feature request. You might fall victim to drinking too much of the cool-aid syndrome and very easily end up with “Nano-services”. Besides just risking network chattiness you also end up incurring long term maintainability cost.

C. Do the microservice own their Data?

One of the key design aspect to consider when breaking down your monolith is having the microservice encapsulate and control data domain they are responsible for.

Though ideal, achieving this is not always easy. I have leant that in any application architecture it is always much easier to move service (and endpoints) than it is to move data. This is a result of other applications tightly coupled to the same data, ETL/batch process systems (backups, data loads, offline processing, recon jobs) or simply the uncertainty around usage knowledge of this data.

Not being able to encapsulate the data behind your microservice is far from a show-stopper. As a matter of fact, it is quite likely that you will encounter this often when transforming an enterprise application.

Below is the approach that I found useful when encountering this situation

1. Focus on the new microservice to support all the APIs handled by the corresponding monolith. Continue to use the existing data store. Once you have test, scaled and launched it to production focus on making sure all of the API clients have moved over to using your new microservice endpoints. Now you can go to step #2.
2. Identify all non-API based access to the domain data. These would end up being some form of offline batch processing, bulk read/ETL jobs, etc. Start to provide API endpoints where possible. You are essentially following a “Cap & Grow” approach at this stage. Also continue to make sure that all new request are always hitting your service endpoints.
   After you have a good handle on the above 2 points you can tackle #3
3. Start to introduce the domain within your microservice. Based on the architectural and functional needs you can consider non-relation vs relational store. At this point you will have to not just consider how you will load all existing data into this new domain store but consider if you need to keep legacy store updated (and if so how).
   A good design pattern to consider introducing at this point is CQRS Design Pattern. It helps with a clean delineation of usage pattern on your domain CRUD endpoints.
   I have used it to create a clean hydration architecture and to publish fine (& coarse) grain domain events for other to consume. You can read more about it at https://www.linkedin.com/pulse/reactive-%C2%B5services-cqrs-pankesh-contractor/

Leveraging Velocity

A primary driver of breaking down the monolith is the ability roll out new change frequently. For a vast majority of teams this is a 2 week cycle — from dev to production; some have it down to a day.

The ability to support this velocity requires an organization-wide change; not just with the development team. Your product & marketing team, business owners, test/quality assurance, SRE/DevOps teams will all need to scale to handle this velocity.

To be successful at this partner with the various team with your organization to answer the below question

1. Can your quality assurance team transition from test-cases/test-plans to a “Automation Driven Testing” approach?
2. What do the DevOps/SRE team need to support your platform deployments at your release velocity? Do they have capabilities to support canary, blue-green and/or zero-downtime deployments.
3. Do you Release Teams have the processes that can scale to support this velocity. What capabilities do you need to provide within the architecture for them — feature flags, dark launch, phase rollouts etc.

As an architect you will have to remember that moving to a Microservices based architecture is as much an organization transformation as it is a technical transformation. Being successful with it at enterprise-scale will requires a keen awareness on your part of various hidden pitfalls (some of which i touched upon above) and creating close partnerships with various other team within your organization.

In Part 2, I will cover some additional aspects related to developer habitability and tech debt when on this microservice journey.