Part 6. How to Work with Multiple Teams and Environments

You typically need a way to test changes to your software (a) before you expose those changes to your users and (b) in a way that limits the blast radius, so if something goes wrong during testing, the damage is constrained, and doesn’t affect users or your production environment.

To some extent, as soon as you deployed your app onto a server (in Part 1), you already had two environments: your local development environment (LDE), which is your own computer, and production, which is the server. Usually, the differences between your LDE and production are large enough that testing solely in the LDE is not sufficient. What you need is one or more environments that closely resemble production, but are completely isolated and only accessible to your team. A common setup you’ll see at many companies is to have the following three environments:

This trio of development, staging, and production, often shortened to dev, stage, and prod, shows up at most companies, although sometimes with slightly different names: e.g., stage is sometimes called QA, as that’s where the quality assurance (QA) team does testing before a release to production.

Larger companies often have multiple products and multiple product teams, and at a certain scale, having all of them work in the same environment or even the same set of environments can lead to a number of problems, as different products may have different requirements in terms of security, compliance, uptime, deployment frequency, and so on. Therefore, it’s common in larger companies for each team or product to have its own isolated set of environments.

For example, the search team might have their software deployed in the search-dev, search-stage, and search-prod environments, while the profile team might have their software deployed in the profile-dev, profile-stage, and profile-prod environments. This ensures that teams can customize their environments to their own needs, limits the blast radius if one team or product has issues, and allows teams to work mostly in isolation from each other.

If you have users in multiple locations around the world, you may want to run your software on servers (and data centers) that are geographically close to those users. One of the big reasons for this is latency: that is, the amount of time it takes to send data between your servers and your users' devices. This information is traveling at nearly the speed of light, but when you’re building software used across the globe, the speed of light can be too slow! Table 11 shows the latency of common computer operations:

These numbers are useful for doing back-of-the-envelope calculations. For example, you can estimate that having multiple data centers on multiple continents, versus one data center on one continent, will reduce latency to your users by around 100,000,000 ns (100 ms). This might not seem like much, but remember, this is the overhead for a single TCP packet, which is typically limited to 1 KB in size, and most web pages and mobile apps these days sends hundreds or thousands of KB of data, so the extra latency can add up to many seconds of additional overhead for every page load and button press.

Therefore, companies with a global reach often end up deploying software across multiple data centers across the globe. For example, you might have one production environment in Ireland (prod-ie) to give EU users lower latency and one production environment in the US (prod-us) to give your North American users lower latency.

If you operate in certain countries, work in certain industries, or work with certain customers, you may be subject to laws and regulations that require you to set up your environments in specific ways. For example, if you store and process credit card information, you may be subject to PCI DSS (Payment Card Industry Data Security Standard); if you store and process healthcare information, you may be subject to HIPAA (Health Insurance Portability and Accountability Act) and HITRUST (Health Information Trust Alliance); if you are building software for the US government, you may be subject to FedRAMP (Federal Risk and Authorization Management Program); and if you are building software in certain countries, you may be subject to data residency laws, such as the EU’s GDPR (Global Data Protection Regulation), which requires businesses that operate in an EU country, or have customers in an EU country, to store and process data on servers physically located within that country.

A common pattern is to set up a dedicated environment for complying with laws and regulations. For example, if you’re subject to PCI DSS, you might have one prod environment that meets all the PCI DSS requirements, and is used solely to run your payment processing software, and another prod environment that isn’t as locked down, and is used to run all your other software.

In Part 3, you saw that a single server can be a single point of failure; the solution was to deploy multiple servers. It turns out that, even if you have multiple servers, if all of them are in a single data center (a single environment), that one data center can be a single point of failure, too. It’s possible for a power outage, cooling problem, network connectivity issue, and a variety of other problems to disrupt the functionality of an entire data center, and all the servers within it. Companies that need a higher degree of resiliency deploy across multiple data centers that are in separate locations around the world (e.g., prod-ie and prod-us, as in the previous section).

Now that you’ve seen a few of the reasons to break up your deployment across multiple environments, let’s talk about how to actually do it.

Perhaps the most obvious challenge with multiple environments is that you now have more moving parts to set up and maintain. You may need to run more servers, set up more data centers, hire more people around the world, and so on. Using the cloud allows you to offload much of this overhead onto the cloud provider, but managing multiple AWS accounts still results in more overhead: each account needs its own authentication, authorization, networking, security tooling, and so on. But even this overhead may be just a drop in the bucket compared to the overhead of having to change your entire architecture to work across environments that are geographically separated, as discussed in the next section.

Having multiple data centers around the world, so they are closer to your users, reduces the latency between the data center and those users, but it may also increase the latency between the different parts of your software running in different data centers. This may force you to rework your software architecture completely, especially when it comes to data storage.

For example, let’s say you had a web app that needed to query a database. If the database you were talking to was in the same data center as the app, then as per Table 11, the networking overhead for the query would be roughly 500,000 ns (0.5 ms) for each packet round trip, which is negligible for most web apps. However, if you had multiple data centers around the world, and the database you were talking to was on a different continent, now the networking overhead could be as high as 183,000,000 ns (183 ms), a 366x increase for every single packet you send. Even a single database query will typically require multiple packets to make round trips, so this extra overhead adds up quickly, and it can make your app unusably slow.

No problem, you say, you’ll just ensure that the database is always in the same data center as the web app. But that means you now need one database per environment, rather than just one database total, and that may require you to radically change how you store and retrieve data, including how you generate primary keys (an auto incrementing primary key will not work with multiple data stores), how you handle data consistency (uniqueness constraints, foreign key constraints, and transactions all become difficult with multiple databases), how you look up data (querying and joining multiple databases is complicated), and so on.

Some companies choose to avoid these challenges by only running in active/standby mode: that is, one data center is active and is serving live traffic, and the other is a standby that only serves live traffic if the active data center goes down. That way, you are only ever reading/writing data in one location at a time. This is useful to boost resiliency, but doesn’t help with latency or local laws and regulations. If you have to have multiple data centers live at the same time, known as active/active, then you will most likely have to rearchitect your data storage patterns to work across multiple geographies. You’ll learn more about data storage in Part 9.

One of the unexpected costs of multiple environments is figuring out how to configure your application differently in each environment. In the early stages of a company, you typically only have a handful of configuration settings, so managing them is pretty straightforward. However, as your company grows, you typically end up with more environments, a more complex architecture, and more demanding security and performance requirements, and as a result, the number of configuration settings can explode. Here are just a few of the most common settings that may differ from environment to environment:

In a large company, it’s not unusual to have thousands of configuration settings to manage, and without the right tooling and processes, this is a common source of problems. In fact, based on Google’s analysis of thousands of postmortems, configuration changes are one of the biggest causes of outages, as shown in Table 12:

Google found that pushing configuration changes is just as risky as pushing code changes (pushing a new binary), and the longer a system has been around, the more configuration changes tend to become the dominant cause of outages.

So how should you manage application configuration to minimize these problems? Broadly speaking, there are two methods for configuring applications:

I recommend using build-time configuration for as much of your configuration as possible. That way, you can treat it just like the rest of your code: that is, every configuration change ends up in version control, gets code reviewed, and goes through your entire CI/CD pipeline (including all the automated tests). I only recommend using run-time configuration for use cases where the configuration changes frequently, such as service discovery and feature toggles, that having to deploy new code to get the latest configuration values would be too slow.

Now that you’ve seen the reasons to deploy into multiple environments, the different options for setting up multiple environments, and the challenges involved with multiple environments, let’s try an example: setting up multiple AWS accounts.

In Part 1, you created an AWS account by signing up on the AWS website. To create more AWS accounts, instead of signing up again and again on the website, let’s treat that initial AWS account you created as the management account, and use AWS Organizations to create all the other accounts as child accounts. This will give you centralized management and billing for of all your accounts, as you’ll be able to see and access all the child accounts from your management account, and any charges will go through the credit card in the management account (rather than a dozen cards if you have a dozen accounts).

Typically, the management account is only used to create and manage other AWS accounts. Since the account has such powerful permissions, you strictly limit who can access it, and do not run any other workloads or environments in that account. Therefore, as a first step, undeploy everything from your AWS account that you deployed in earlier blog posts: e.g., run tofu destroy on any OpenTofu modules you previously deployed and use the EC2 Console to manually undeploy anything you deployed via Ansible, Bash, etc. When you’re done, you should essentially have an empty AWS account, with your IAM user as the only one who can access it.

Next, you can use the aws-organizations module, which is in the blog post series’s sample code repo in the ch6/tofu/modules/aws-organizations folder, to create three child accounts (development, staging, and production) using AWS Organizations. Head into the folder where you’ve been working on the code samples for this blog post series and make sure you’re on the main branch, with the latest code:

Next, create a new ch6 folder for this blog post’s code examples, and within the ch6 folder, create a tofu/live/child-accounts folder:

Within management/child-accounts, create a main.tf file with the contents shown in Example 104:

The preceding code does the following:

Next, create an outputs.tf file with the output variables shown in Example 105:

The preceding code outputs the ARNs of IAM roles you can use to manage the dev, stage, and prod accounts. When you create child accounts using AWS Organizations, it automatically creates an IAM role named OrganizationAccountAccessRole within each child account, giving that IAM role admin permissions, and configuring it so you can assume it from the management account.

Deploy the child-accounts module as usual, authenticating to AWS as described in Authenticating to AWS on the command line, and running init and apply:

After apply completes, you should see your output variables:

Congrats, you just created new AWS accounts using code! And you can use code like this to create new accounts and manage them in the future.

You now have some new AWS accounts, but they aren’t useful until you deploy infrastructure into them, and to do that, you need to learn how to access them.

Now that you’ve created child accounts, to access them, you can assume the IAM role that AWS Organizations creates for you automatically in those accounts. There are many different ways to assume an IAM role. For example, follow these instructions to assume an IAM role in the AWS Web Console. To assume an IAM role in the terminal, one option is to configure an AWS profile for each child account. For example, to create a profile for your dev account, open up the AWS config file, which lives at ~/.aws/config (if the file doesn’t exist already, create it), and add the code shown in Example 106 to it:

The preceding code does the following:

Most tools that talk to AWS APIs give you a way to specify the profile to use. One way to do this is to use the AWS_PROFILE environment variable, as shown in Example 107:

The get-caller-identity command returns information about the authenticated user, so if you configured the profile correctly, ACCOUNT_ID should be the ID of the dev account, and ARN should be the ARN of the dev IAM role.

Create analogous profiles for the stage and prod accounts. In the next section, you’ll see how to use these profiles to deploy infrastructure into the dev, stage, and prod accounts.

Let’s now try to deploy the lambda-sample module from earlier blog posts into the dev, stage, and prod accounts. Copy the lambda-sample module from Part 5, as well as the test-endpoint module it relies on, into a new ch6/tofu/ folder:

Next, make two changes to the lambda-sample module:

In OpenTofu, you can use workspaces to manage multiple deployments of the same configuration. Each workspace has its own state file, so it represents a separate copy of all the infrastructure, and each workspace has a unique name, which is returned by terraform.workspace. If you don’t specify a workspace, as you’ve been doing so far throughout this blog post series, then you end up using a workspace called "default." In this blog post, let’s create a custom workspace per environment.

First, authenticate to your management account as usual (as described in Authenticating to AWS on the command line), and run tofu init to initialize the backend, modules, and providers:

Next, use the tofu workspace new command to create a new workspace:

The preceding command creates a workspace called "development," which you can use to store the state for your development environment. The idea is to deploy the development environment into the new development account you created earlier. You can do this by running tofu apply and telling OpenTofu to authenticate to your development account by setting the AWS_PROFILE environment variable to the name of the profile you created for the development account in the previous section:

You should see a plan output to create the Lambda function, API Gateway route, and so on. If everything looks good, type in yes and hit Enter. When apply completes, you should see the api_endpoint output variable, which contains a URL you can try to access the Lambda function. Try this URL out:

Congrats, you now have a serverless web app running in your development account! Let’s now try to deploy the same app into the staging account. First, create a new workspace for staging:

Next, run apply again, but this time, set AWS_PROFILE to the name of the profile you created for your staging account:

You should see a plan output that shows OpenTofu will create all the resources (Lambda function, API Gateway route, etc.) again from scratch. That’s because each workspace has its own state file, so when you’re in the staging workspace, OpenTofu doesn’t look at any of the infrastructure you deployed in the development workspace. If everything looks good with the plan, type in yes and hit Enter. When apply completes, you should have a different URL you can try:

And there you go, you now have a second environment running in a second AWS account! Complete the picture by deploying into the third environment, production:

When you’re done, you should see "Hello from production!" At this point, you have three environments, across three AWS accounts, with a separate copy of the serverless webapp in each one, and the OpenTofu code to manage it all.

You now have three copies of the serverless webapp running, all configured exactly the same way. Let’s see what it might look like to configure the app differently in each environment. To keep things simple, we’ll use JSON configuration files checked into version control. First, create a folder called config for the configuration files:

Within the config folder, create a file called development.json, with the contents shown in Example 110:

This file contains just a single config entry, text, which is the text the web app should return in that environment. Create analogous config/staging.json and config/production.json files, but with text updated to different values in each environment.

Next, update index.js to load the config file for the current environment and return the text value in the response, as shown in Example 111:

There are two updates to the app:

Now it’s time to deploy this change in each environment—that is, in each workspace. To see all your workspaces, use the workspace list command:

You can switch to any existing workspace using the workspace select command:

Now, your OpenTofu commands will run against the development workspace. Run apply with AWS_PROFILE set to the dev profile to deploy the changes to the development environment:

When apply completes, open the URL in the api_endpoint output variable, and you should see "Hello from dev config!" Use workspace select and apply (with AWS_PROFILE properly set) to deploy the changes in staging and production as well. When you test the URLs for those environments, you should see the text values you put into those configs: e.g., "Hello from stage config!" and "Hello from prod config!" Congrats, you’re now loading different configuration files in different environments!

When you’re done testing and experimenting with multiple AWS accounts, you may wish to close some or all of the new child accounts. Going forward, just about all the examples in this blog post series will deploy into just a single account (to keep things simple), so you don’t need all three running. Note that AWS does not charge anything extra for the accounts themselves, but you may want to clean them up to keep your security surface area smaller, and to ensure you don’t accidentally leave resources running in those accounts (e.g., EC2 instances), as AWS does charge for those as usual.

First, commit all your code changes to Git: that way, if you ever want to bring back the three accounts, you’ll have all the code to do it.

Second, undeploy the infrastructure in each workspace. To do that, use workspace select to select each environment and then run tofu destroy, making sure to set AWS_PROFILE to the profile you created for that environment. For example, here is how you undeploy the infrastructure in the development workspace:

Repeat the same workspace select and tofu destroy commands for the staging and production environments.

Third, run tofu destroy on the child-accounts module to start the process of closing the child accounts:

When you run destroy, AWS will initially mark the child accounts as "suspended" for 90 days, which is a fail-safe that gives you a chance to recover anything you may have forgotten in those accounts before they are closed forever. After 90 days, AWS will automatically close those accounts.

Once a codebase gets big enough, no one can understand all of it. There are just too many parts, too many interactions, and too many concepts to keep straight, and if you have to deal with all of them at once, your pace of development will drop, and the number of bugs will skyrocket. Consider Table 13, which is a table from the book Code Complete that shows the number of bugs in software projects of various sizes:

It’s no surprise that larger software projects have more bugs, but note that Table 13 shows that larger projects also have a higher bug density, or the number of bugs per 1,000 lines of code. To put this into perspective, take a developer, and have them add 100 lines of code to a small software project (<2K lines of code), and on average, you’ll find that new code has no new bugs, or maybe one or two. Take the same developer and have them add 100 lines of code to a large software project (>512K lines of code), and on average, you’ll find that they have introduced as many as ten new bugs. Same developer, same number of lines of new code, but 5-10x as many bugs. That’s the cost of complexity.

There is a limit to how much code complexity the human mind can handle. In fact, in that same Code Complete book, author Steve McConnell defines "managing complexity" as "the most important technical topic in software development." There are many techniques for managing complexity, but almost all of them come down to one basic principle: divide and conquer. That is, find a way to organize your code so that you can focus on one small part at a time, while being able to safely ignore the rest. One of the main goals of most software abstractions, including object-oriented programming, functional programming, libraries, and microservices is to break up the codebase into discrete pieces, so that you only need to think about the simple interface of that piece, rather than the full complexity of the underlying implementation details.

Another common reason to break up a codebase is to allow teams to work independently of each other and to have full ownership of their part of the product. As your company grows, different teams will start to develop preferences for different product development practices, such as how they design their systems and architecture, how they test and review their code, how often they deploy, and how much tolerance they have for bugs and outages.

If you do all your work in a single, tightly-coupled codebase, then a problem in any one team or product can affect all the other teams and products, and that’s not always desirable. For example, if you open a pull request, and an automated test fails in some totally unrelated product, should that block you from merging? If you deploy new code that includes changes to ten different products, and one of them has a bug, should you roll back the changes for the other nine? If one team wants to deploy dozens of times per day, but another team has a product in a regulated industry where they can only deploy once per quarter, should everyone be stuck with the slower deployment cadence? Splitting up the codebase allows you to set up separate processes for each team that meet their specific needs.

Note that teams working independently of each other doesn’t mean they never interact. It’s just that the interactions are now limited to well-defined interfaces: e.g., the API of a library or a web service. This lets you benefit from the output of that team’s work (e.g., the data returned by their API) without being subject to the particular inputs they need to make that work possible. In fact, you do this all the time, even in small companies whenever you add a dependency on a third party, such as an open source library or a vendor’s API. You’re able to benefit from the work they are doing, while keeping all your coding practices (testing, code reviews, deployment cadence, etc.) largely separate.

As your user base grows, you will hit more and more scaling challenges to handle the extra load. In some cases, you may find that some parts of your software have different scaling requirements than other parts. For example, one part of your code may benefit from distributing work across a large number of CPUs on many servers, whereas another part of your code may benefit from a large amount of memory on a single server. If everything is in one codebase—and more to the point, if everything is deployed together—meeting these conflicting scaling requirements can be difficult. As a result, many companies break up their codebase so that the different parts of the code can be deployed and scaled independently.

Most companies start with a single programming language, but as you grow, you may end up using multiple programming languages. Sometimes, this is because different developers at your company prefer different languages; sometimes, this is because you acquired a company that uses a different programming language; sometimes, this is because different languages may be a better fit for different problems (use the right tool for the job). Each time you introduce a new language, you have a new app to deploy, configure, update, and so on, and as you’ll see shortly, this typically means your codebase now consists of multiple services to manage.

Now that you’ve seen why you may want to break up a codebase, let’s talk about how to actually do it.

Just about all codebases are broken up into various abstractions, such as functions, interfaces, classes, and modules (depending on the programming language you’re using). However, if the codebase gets big enough, you may choose to break it up even further into libraries. An abstraction is a library if you no longer depend directly on the source code of that abstraction, but on a versioned artifact you publish for that abstraction. The exact type of artifact depends on the programming language: for example, in Java, that might be a .jar file; in Ruby, that might be a Ruby Gem; and in JavaScript, that might be an NPM module.

For example, you might start with a code base that has three parts, A, B, and C. Initially, part A depends directly on the source code of B and C, as shown in Figure 62:

You could break up this codebase by turning B and C into libraries that publish artifacts (e.g., if this was Java, the artifacts would be b.jar and c.jar), and update A to depend on a specific version of these artifacts, instead of the source code, as shown in Figure 63. Note that, as long as you use artifact dependencies, A, B, and C can all continue to live in a single repo, or be broken up across multiple repos. That said, multiple repos tends to be more common, as it ensures you don’t accidentally fall back to source code dependencies, and it gives teams more independence.

Breaking up your code into libraries has several advantages. First, it allows you to focus on one small part of your codebase (the library) at a time, while safely ignoring everything else. Second, each team can develop the internals of their libraries using whatever practices they want (e.g., for testing, code reviews, etc.). Third, teams can work more independently, as unlike source code dependencies, where every change immediately affects everyone who depends on your code, with libraries, your changes don’t affect anyone until (a) you’ve published a new versioned artifact and (b) users of your library have explicitly and deliberately chosen to pull in that new version.

Almost all software projects these days depend on libraries: namely, open source libraries. For example, the Node.js sample app you’ve been working on throughout this blog post series depends on Express.js, an open source web framework that you pull in through a versioned artifact (an NPM module). The maintainers of Express.js are able to develop this library completely independently of all the projects that depend on it, following their own coding conventions, testing practices, release cadence, and so on. The point is not that you need to open source your own code, but that if you break up your codebase into libraries, you can also benefit from being able to develop each piece independently.

If you do break your codebase up into libraries, I recommend following two practices: semantic versioning and automatic updates

Semantic versioning (SemVer) is a set of rules for how to assign version numbers to your code. The goal is to communicate to users if a new version of your library has backward incompatible changes: that is, changes that would require the user to update how they use your library in their code in order to make use of this new version. Typically, this happens when you make changes to the API: e.g., you remove something that was in the API before, or you add something new to the API that is now required. With SemVer, you use version numbers of the format MAJOR.MINOR.PATCH (e.g., 1.2.3), where you increment these three parts of the version number as follows:

For example, if your library is currently at version 1.2.3, and you have made a backward incompatible change to the API, then to communicate this to your users, the next release would be 2.0.0. On the other hand, if you made a backward compatible bug fix, the next release would be 1.2.4. It’s also worth mentioning that 1.0.0 is typically seen as the first release that provides compatibility promises, so if you just created something new, you can use 0.x.y to indicate that you’re not yet providing backward compatibility guarantees.

Automatic updates is a way to keep your dependencies up to date. One of the benefits of using library dependencies is that changes to that library only affect you when you explicitly and deliberately pull in a new version of that library. However, this strength is also a drawback: it’s easy to forget to update a library for a long time. This can be a problem, as the old version may have bugs or security vulnerabilities, and if you don’t update for a while, updating to the latest version to pick up a fix can be difficult, especially if there have been many breaking changes since your last update.

This is yet another place where, if it hurts, you need to do it more often. In particular, you want to set up a process where you automatically update your dependencies and roll those updates out to production (sometimes called software patching). This applies to all the different types of software you depend on, including open source libraries, internal libraries, operating systems, and so on. The automation you set up can either run on a schedule (e.g., update weekly) or in response to new versions being released.

You can set up automated updates using tools such as DependaBot, Renovate, Snyk, and Patcher, which can detect dependencies in your code, and automatically open pull requests to update you to new versions. That way, instead of having to remember to do updates yourself, the updates come to you, and all you have to do is check that they pass your suite of tests (as per Part 4), and if those pass, merge the pull request in, and let the code deploy automatically (as per Part 5).

Consider parts A, B, and C from the previous section: whether you use source code dependencies (Figure 62) or library dependencies (Figure 63), all the parts of your codebase run in a single process and communicate with each other via in-memory function calls. Another way to break up the codebase is to move from a single monolithic application into multiple services, where each service is a part of your code that you develop independently and deploy in a separate process, typically on a separate server, and all communication between services is done by sending messages over the network, as shown in Figure 64:

Over the years, there have been many different approaches to building services, and also many buzzwords and fads, which can make it hard to nail down concrete definitions. One approach is service-oriented architecture (SOA), which typically refers to building relatively large services that handle all the logic for an entire business line or product within your company; this was also sometimes called Web 2.0, when it referred to services exposed between different companies (e.g., APIs from Twitter, Facebook, Google Maps, etc). A slightly more recent approach that arose around the same time as DevOps is microservices, which typically refers to smaller, more fine-grained services that handle one domain within a company: e.g., one microservice to handle user profiles, one microservice to handle search, one microservice to do fraud detection, and so on. Yet another approach is event-driven architecture, where services communicate asynchronously; you’ll learn more about this approach in Part 9.

Whichever model of services you choose, there are typically three main advantages to breaking up your code into services:

Almost all large companies eventually move to services for these three advantages, but especially due to the ability to isolate teams. To some extent, using services allows each team to operate like its own, independent company, which is essential to scaling.

Moving to services can be an essential ingredient in helping a company scale, but beware: breaking up the codebase, whether into libraries or services, comes with a number of costs and challenges, so most companies should avoid it until they have no other choice, as described in the next section.

Both libraries and services consist of two parts: the public API and the internal implementation details. Breaking up your codebase allows you to make changes more quickly to the internal implementation details, as each team can maintain those however they want. However, making changes to the public API becomes slower and more difficult, as you now need to worry about backward compatibility. Making backward incompatible changes (AKA breaking changes) in a library or service can cause headaches, bugs, and outages for everyone who depends on your library or service, so you have to be careful in changes to the public API.

For example, imagine that in part B of your codebase, you have a function called foo that you want to rename to bar. This is easy to do if all the code that depends on B is in one codebase:

If B is a separate library, the process is more complicated:

If B is a separate service, the process is even more complicated:

Phew. That’s a lot of work. If you spend enough time maintaining a library or service, you quickly learn how important it is to get the public API right, and you’ll likely spend a lot of time obsessing over your public API design. But no matter how much time you spend on it, you’ll never get it exactly right, and you’ll always have to evolve it over time anyway, so expect public API maintenance to be one of the overheads of splitting up the codebase.

The reason it’s hard to maintain a public API in libraries and services is because that’s a place where you have to interact with many other teams at your company. As it turns out, this is just one specific type of change that becomes harder if you split up your codebase: the more general problem is that any global changes—changes that require updating multiple libraries or services—become considerably harder.

For example, LinkedIn, like almost all companies, started with a single monolithic application. It was called Leo and was written in Java. Eventually, Leo became a bottleneck to scaling, both in terms of scaling to handle more developers and more traffic, so we started to break it up into dozens of libraries and services. For the most part, this was a huge win, as each team was able to iterate on features within their library or service much faster than when those features were mixed with everyone else’s features within Leo. However, we also had to do the occasional global change. For example, almost every single service relied on some security utilities in a library called util-security.jar. When we found a vulnerability in that library, rolling out the new version to all services took a gargantuan effort:

What would’ve been a single commit and deploy within a monolith became a multi-week slog when dealing with dozens of microservices. To some extent, this is by design: the whole point of splitting up a codebase is to make it hard for changes in other parts of the codebase to affect you.

If you split up your codebase and find that the vast majority of the changes each team makes are within the part of the codebase owned by that team, then this split will allow you to go faster; but if you find that teams often have to make updates across multiple parts of the codebase, then this split will make you go slower. Unfortunately, knowing where to split up the codebase can be surprisingly challenging, as discussed in the next section.

One of the challenges of splitting up a codebase is knowing where to put the seams. If you get it wrong, then most changes become global changes, and that will slow you down. One place I see teams get this wrong all the time is splitting up the codebase way too early. It’s much easier to identify the seams in a codebase that has been around for a long time than it is to guess where to put the seams in something totally new. When you’ve been working with a codebase for years, you can usually look for the following patterns for where the codebase could be split:

Trying to predict any of these items ahead of time for a new codebase is futile. This is especially true of performance bottlenecks, which you can never really predict without running a profiler against real code and real data. The only way to get these seams right is to start with a monolith, grow it as far as you can, and only when you can’t scale it any further, do you break it up into smaller pieces.

This is one of the reasons that I shake my head when I see a tiny startup with a three-person engineering team launch their product with 12 microservices: not only are you going to pay a high price in terms of operational overhead (something you’ll learn more about shortly), but you almost certainly put the seams in the wrong places. Inevitably, these teams find that every time they go to make the slightest change in their product, they have to update 7 different microservices, and deploy them all in just the right order. Meanwhile, the startup who built on top of a Ruby on Rails or PHP monolith is running circles around them, shipping changes 10x faster.

In Part 5, you learned all about continuous integration, and its central role in helping teams move faster. Well, here’s a fun fact: splitting up your codebase into libraries and services is the opposite of continuous integration. What’s the difference between a long-lived feature branch that you only merge into main after 8 months versus a library dependency that you only update once every 8 months? Not much.

Once you’ve split up your codebase, what you’re effectively doing is late integration. And that’s by design: one of the main reasons to split up a codebase is to allow teams to work more independently from each other, which means you are going to be integrating your work together much less frequently.

This is a good trade-off to make if the different teams are truly decoupled from each other: e.g., they work on totally separate products within your company. However, if the teams are actually tightly coupled, and have to interact often, then splitting them up into separate codebases will lead to problems. Either the teams will try to work mostly independently, and due to the lack of integration and proper testing run into lots of conflicts, bugs, and outages, or the teams will try to integrate their work all the time, and due to the frequent need to make global changes across multiple parts of the codebase, they will find development is very slow.

One challenge that is unique to libraries is what’s sometimes referred to as dependency hell, which is where using versioned dependencies can lead to one of a number of frustrating situations, such as the following:

Just about all codebases run into these issues from time to time due to dependencies on open source libraries, but if you break your own codebase up into many libraries, these problems may become exponentially worse.

If you split a monolith into services, instead of having just a single type of app to manage, you now have many different types, possibly written in different languages, each with its own mechanisms for testing, deployment, monitoring, configuration, and so on. Think of all the work you’ve done so far in this blog post series to deploy a single app and a CI / CD pipeline for it, add all the work in upcoming posts, such as networking, data storage, and monitoring, and then multiply that by the number of services, and you’ll get an inkling of the operational overhead involved. But that’s not all. There is even more operational overhead from dependencies between services, debugging of multiple services, infrastructure for multiple services, and the performance impact from services, as discussed in the next several sections.

With N services, you not only have N things to deploy and manage, but you also have to consider the interactions between services, which grows at a rate of N². For example, let’s say you have a service A that depends on a service B. As part of developing a new feature, you add a new endpoint called foo to B, and you update the code in A to make calls to the foo endpoint. Now consider what happens at deployment time: if you deploy the new version of A before the new version of B is out, then when A tries to use the foo endpoint, it’ll fail, as the old version of B doesn’t have that endpoint yet. So now you have to enforce deployment ordering: B must be deployed before A. But of course, B itself may depend on new functionality in services C and D, and those services may depend on new functionality in other services, and so on. So now you have a deployment graph to maintain to ensure the right services are deployed in the right order. And this gets really messy if one of those services has a bug, and you have to rollback: for example, if the new version of C had to be rolled back, you’d also have to know to roll back the new versions of B and A.

One way to mitigate this problem is to ban deployment ordering entirely: that is, you require your code to be written so that services can be deployed in any order, and rolled back at any time. One way to do that is to use feature flags, which you saw in Part 5. You wrap the new functionality in A—the part of the code that calls the new foo endpoint in B—in an if-statement which is off by default. That way, you can deploy A and B at any time and in any order, as the new functionality won’t be visible to any users. When you’re sure both the new versions of A and B are deployed, you then turn the feature toggle on, and the new functionality should start working; if you hit any issue with the new functionality or B or one of its dependencies has to be rolled back, you can turn the feature toggle off again. Once again, you are using feature toggles to separate deployment from release, which is usually a more effective solution than trying to implement deployment ordering—but still nowhere near as simple as avoiding the problem entirely by sticking with a monolith as long as you can.

If you have a single monolithic app, and your users report a bug, you know the bug is in the app. If you have dozens of services, and your users report a bug, now you have to do an investigation to figure out which service is at fault. This is considerably harder for several reasons. One reason is the natural tendency for each team to immediately blame other teams, so no one will want to take ownership of the bug. Another reason is that when you have services that communicate over the network, rather than a monolith where everything happens in a single process, there are a large number of new, complicated failure conditions that are tricky to debug (you’ll learn more about this shortly).

One more reason is that, whereas debugging a single app can be hard, trying to track down a bug across dozens of separate services can be a nightmare. You can no longer look at the logs of a single app, and instead have to go look at logs from dozens of apps, each potentially in a different place and format; you can no longer reproduce the error by running a single app on your computer, and instead have to fire up dozens of services locally; you can no longer hook up a debugger to a single process and go through all the code step-by-step, and instead have to use all sorts of tracing tools to identify the dozens of services that end up processing a single request. A bug that could take an hour to figure out in a monolith can take weeks to track down in a microservices architecture.

Moving from a monolith to multiple services isn’t just about deploying a bunch of services: you typically also need to deploy a bunch of extra infrastructure to support the services themselves—and the more services you have, the more infrastructure you need to support them. For example, to help manage the deployments of 12 services, rather than 1 monolith, you may have to deploy a more complicated orchestration tool (e.g., Kubernetes); to help your services communicate with each other securely, you may have to deploy a service mesh tool (e.g., Istio); to help your services communicate with each other asynchronously, you may have to deploy a streaming platform (e.g. Kafka); to help with debugging and monitoring your microservices architecture, you may have to deploy a distributed tracing tool (e.g., Jaeger), and integrate a tracing library it into all your services (e.g., OpenTracing); and so on. All of this infrastructure takes a lot of time and money to deploy and manage, and you can avoid most of it by sticking with a monolith for as long as possible.

One of the benefits of services is that they help you deal with performance bottlenecks by allowing you to scale different parts of your codebase independently. One of the drawbacks of services is that, in almost every other way, they actually make performance considerably worse. This is due to the following reasons:

When you split a monolith into services, you often have to rewrite a lot of your code to use concurrency, caching, batching, and de-duping to minimize this performance overhead. However, this makes your code considerably more complicated, and it’ll still be orders of magnitude slower than keeping everything in a single process.

Splitting a monolith into services is a major shift: you’re turning a single app into a distributed system. Distributed systems are hard. Dealing with distributed systems introduces a number of new challenges, such as the following:

Now that you’ve seen all the challenges with splitting up your codebase, you might be feeling a little less excited about that sexy microservices architecture you saw at Google or Netflix. If so, that’s a good thing. You’re probably not working at Google or Netflix, so you shouldn’t blindly copy their architecture, as much of it was designed to handle problems of extraordinary scale, and if you don’t have those problems, then that architecture is more likely to slow you down than to help you.

Let’s assume that you’re at a company of a large enough scale to merit splitting up the codebase, and see what it looks like to run multiple services in Kubernetes.

As a first step, copy the Node.js sample app that you last saw in Part 5 into a new folder called sample-app-backend:

Since you’ll be deploying this backend into a Kubernetes cluster, you should also copy the Kubernetes Deployment and Service configurations from Part 3 into sample-app-backend:

Next, update the files in sample-app-backend as follows:

Build a Docker image for the backend app using the dockerize command you added back in Part 4:

This will create a new Docker image called "sample-app-backend" at version 0.0.1. Let’s deploy this Docker image into a Kubernetes cluster. The easiest one to test with is Kubernetes running locally in Docker Desktop, just as in Part 3. You can authenticate to the Kubernetes cluster in Docker Desktop as follows:

Now you can use kubectl apply to deploy the Deployment and Service:

If you run kubectl get services, you should see the Service for the backend:

Note the backend Service name; you’ll need this in the frontend app, which is the focus of the next section.

Create the frontend app using a process similar to the one you just used for the backend app. First, copy the Node.js sample app from Part 5 and the Kubernetes Deployment and Service configurations from Part 3 into a new folder called sample-app-frontend:

Next, update the files in sample-app-frontend as follows:

Build a Docker image for the frontend app using the dockerize command you added back in Part 4:

This will create a new Docker image called "sample-app-frontend" at version 0.0.1. To deploy this Docker image into a Kubernetes cluster, use kubectl apply:

If you run kubectl get services, you should now see the Services for both the backend and the frontend:

Notice how EXTERNAL-IP for the frontend is set to localhost and that it’s listening on port 80, so you can test it by going to http://localhost. If you open this URL in a web browser, you should see the HTML rendered, as shown in Figure 67:

Congrats, you’re now running two microservices in Kubernetes that are talking to each other! A separate team could own each service, developing, deploying, and scaling the service completely independently.

When you’re done testing, you may want to run kubectl delete on each of the Deployments and Services to undeploy them from your Kubernetes cluster. You should also commit your changes to Git, as you will continue to iterate on this code in subsequent blog posts.