Part 6. How to Work with Multiple Teams and Environments

You typically need a way to test changes to your software (a) before you expose those changes to your users and (b) in a way that limits the blast radius, so if something goes wrong during testing, the damage is constrained, and doesn’t affect users or your production environment.

To some extent, as soon as you deployed your app onto a server (in Part 1), you already had two environments: your local development environment (LDE), which is your own computer, and production, which is the server. This may be enough for testing some types of software, but in most cases, the differences between your LDE and production are so large, that only testing in the LDE is not sufficient. What you need is one or more environments that closely resemble production, but are completely isolated and only accessible to your team.

A common setup you’ll see at many companies is to have the following three environments:

This trio of development, staging, and production, often shortened to dev, stage, and prod, shows up at most companies, although sometimes with slightly different names: e.g., stage is sometimes called QA, as that’s where the quality assurance (QA) team does testing before a release to production.

Larger companies often have multiple products and multiple product teams, and at a certain scale, having all of them work in the same environment or even the same set of environments can lead to a number of problems: e.g., different products may have different requirements in terms of security, compliance, uptime, deployment frequency, and so on.

Therefore, it’s common in larger companies for each team or product to have its own isolated set of environments. For example, the search team might have their software deployed in the search-dev, search-stage, and search-prod environments, while the profile team might have their software deployed in the profile-dev, profile-stage, and profile-prod environments. This ensures that teams can customize their environments to their own needs, limits the blast radius if one team or product has issues, and allows teams to work mostly in isolation from each other.

If you have users in multiple locations around the world, you may want to run your software on servers (and data centers) that are geographically close to those users. One of the big reasons for this is latency: that is, the amount of time it takes to send data between your servers and your users' devices. This information is traveling at nearly the speed of light, but when you’re building software used across the globe, the speed of light can be too slow! If you’re going to build software, it’s a good idea to familiarize yourself with the latency of common computer operations, as shown in Table 9:

These numbers are useful for doing back-of-the-envelope calculations, such as estimating the impact of having one data center on one continent, where your users may end up connecting from a different continent, versus having multiple data centers on multiple continents, where your users always end connecting on the same continent: in this scenario, you can expect the multi-data center approach to reduce latency by around 100,000,000 ns (100 ms). This might not seem like much, but remember, this is the overhead for a single TCP packet: these are typically around 1 KB in size, and most web pages and mobile apps these days sends hundreds or thousands of KB of data, across many requests, so the extra latency can quickly add up to many seconds of additional overhead for every page load and button press.

Therefore, companies with a global reach often end up deploying software across multiple data centers across the globe: for example, you might have one production environment in Ireland (prod-ie) to give your European users lower latency and one production environment in the US (prod-us) to give your North American users lower latency.

If you operate in certain countries, work in certain industries, or work with certain customers, you may be subject to laws and regulations that require you to set up your environments in very specific ways. For example, if you store and process credit card information, you may be subject to PCI DSS (Payment Card Industry Data Security Standard); if you store and process healthcare information, you may be subject to HIPAA (Health Insurance Portability and Accountability Act) and HITRUST (Health Information Trust Alliance); if you are building software for the US government, you may be subject to FedRAMP (Federal Risk and Authorization Management Program); and if you are building software in certain countries, you may be subject to data residency laws, such as the EU’s GDPR (Global Data Protection Regulation), which require businesses that operate in that country, or have customers in that country, to store and process data on servers physically located within that country.

A common pattern is to set up a dedicated environment for complying with laws and regulations: for example, if you’re subject to PCI DSS, you might have one prod environment, perhaps called prod-pci, that meets all the PCI DSS requirements, and is used solely to run your payment processing software, and another prod environment, perhaps just called prod, that isn’t as locked down, and is used to run all your other software.

In Part 3, you saw that a single server can be a single point of failure; the solution was to deploy multiple servers. It turns out that, even if you have multiple servers, if all of them are in a single data center (a single environment), that one data center can be a single point of failure, too. It’s possible for a power outage, cooling problem, network connectivity issue, and a variety of other problems to disrupt the functionality of an entire data center, and all the servers within it.

The solution, for companies that need a higher degree of resiliency, is to deploy across multiple data centers that are in separate locations around the world (e.g., prod-ie and prod-us, as in the previous section), so that whatever causes an outage in one is unlikely to affect the others.

Now that you’ve seen a few of the most common reasons why you might want to break up your deployment across multiple environments, let’s talk about how to actually do it.

Perhaps the most obvious challenge with multiple environments is that you now have more moving parts to set up and maintain: you may need to run more servers, set up more data centers, hire more people around the world, and so on. Using the cloud allows you to offload much of this overhead onto the cloud provider, but even creating and managing multiple AWS accounts results in more overhead: each account needs its own authentication, authorization, networking, security tooling, audit logging, and so on.

But even this overhead may be just a drop in the bucket compared to the overhead of having to change your entire architecture to work across environments that are geographically separated, as discussed in the next section.

Having multiple data centers around the world, so they are closer to your users, reduces the latency between the data center and those users, but it may also increase the latency between the different parts of your software running in different data centers. This may force you to rework your software architecture completely, especially when it comes to data storage.

For example, let’s say you had a web app that needed to look up data in a database before sending a response. If the database you were talking to was in the same data center as the web app, then as per Table 9, the networking overhead for the database query would be roughly 500,000 ns (0.5 ms) for each packet round trip, which is negligible for most web apps. However, if you had multiple data centers around the world, and the database you were talking to was on a different continent, now the networking overhead could be as high as 183,000,000 ns (183 ms), a 366x increase for every single packet you send. Even a single database query will typically require multiple packets to make round trips, so this extra overhead adds up very quickly, and it can make your webapp unacceptably slow.

No problem, you say, you’ll just ensure that the database is always in the same data center as the web app. But that means you now need one database per environment, rather than just one database total, and that may require you to radically change how you store and retrieve data, including how you generate primary keys (the typical auto incrementing primary key will no longer work with multiple data stores), how you handle data consistency and concurrency (uniqueness constraints, foreign key constraints, locking, and transactions all become very difficult with multiple databases), how you look up data (querying and joining multiple databases is complicated), and so on.

Some companies choose to avoid these challenges by only running in active/standby mode: that is, one data center is active and is serving live traffic, and the other is a standby that only serves live traffic if the active data center goes down, in which case you make the standby data center the new active one. That way, you are only ever reading/writing data in one location at a time. This is useful to boost resiliency, but doesn’t help with latency or local laws and regulations. If you have to have multiple data centers live at the same time, known as active/active, then you will most likely have to rearchitect your data storage patterns to work across multiple geographies. You’ll learn more about data storage in Part 9 [coming soon].

One of the unexpected costs of multiple environments is figuring out how to configure your application differently in each environment. In the early stages of a company, you typically only have a handful of configuration settings, so managing them is pretty straightforward. However, as your company grows, you typically end up with more environments, a more complex architecture, and more demanding security and performance requirements, and as a result, the number of configuration settings can explode. Here are just a few of the most common settings that may differ from environment to environment:

In a large company, it’s not unusual to have thousands of configuration settings to manage, and without the right tooling and processes, this is a common source of problems. In fact, based on Google’s analysis of thousands of postmortems, configuration changes are one of the biggest causes of outages, as shown in Table 10:

Google found that pushing configuration changes is just as risky as pushing code changes (pushing a new binary), and the longer a system has been around, the more configuration changes tend to become the dominant cause of outages. That means it’s worth taking some time to think through how you’re going to manage and validate your application configuration.

Broadly speaking, there are two methods for configuring applications:

I recommend using build-time configuration for as much of your configuration as possible. That way, you can treat it just like the rest of your code: that is, every configuration change ends up in version control, gets code reviewed, and goes through your entire CI/CD pipeline (including all the automated tests), as per Part 4. I only recommend using run-time configuration for use cases where the configuration changes very frequently: e.g., service discovery and feature toggles. In those cases, having to do a commit and deploy to keep up is too slow, so using a data store designed for those use cases is the better fit.

Now that you’ve seen the reasons to deploy into multiple environments, the different options for setting up multiple environments, and the challenges involved with multiple environments, let’s try an example: using multiple AWS accounts as multiple environments.

In Part 1, you created an AWS account by signing up on the AWS website. To create more AWS accounts, instead of signing up again and again on the website, let’s treat that initial AWS account you created as the management account, and use AWS Organizations to create all the other accounts as child accounts. This will give you centralized management of all your accounts, as you’ll be able to see and access them all from your management account, and centralized billing, as all the charges will roll up to the management account, so everything goes through one credit card (rather than a dozen cards if you have a dozen accounts).

Typically, the management account is only used to create and manage other AWS accounts. Since the account has such powerful permissions, you strictly limit who can access it, and do not run any other workloads or environments in that account.

Therefore, as a first step, undeploy everything from your AWS account that you deployed in earlier chapters: e.g., run tofu destroy on any OpenTofu modules you previously deployed and use the EC2 Console to manually undeploy anything you deployed via Ansible, Bash, etc. When you’re done, you should essentially have an empty AWS account, with your IAM user as the only one who can access it.

Next, let’s use AWS Organizations to configure your current account as a management account and create three child accounts below it: development, staging, and production. Head into the folder where you’ve been working on the code samples for this blog post series and make sure you’re on the main branch, with the latest code:

Next, create a new ch6 folder for this blog post’s code examples, and within the ch6 folder, create a tofu/live/child-accounts folder:

Within the child-accounts folder, you will create an OpenTofu root module that can create the child accounts. Under the hood, this root module will use an OpenTofu module called aws-organizations that’s in the blog post series’s sample code repo in the ch6/tofu/modules/aws-organizations folder. The aws-organizations module uses AWS Organizations to create three child accounts: development, staging, and production.

Within management/child-accounts, create a main.tf file with the contents shown in Example 110:

The preceding code does the following:

You should also create an outputs.tf file with the output variables shown in Example 111:

The preceding code adds two types of output variables:

Deploy the child-accounts module as usual: authenticate to AWS as described in Authenticating to AWS on the command line, and run init and apply:

After apply completes, you should see your output variables:

Congrats, you just created new AWS accounts using code! And you can use code like this to create new accounts and manage them in the future.

Of course, these new AWS accounts aren’t useful until you deploy infrastructure into them, and to do that, you need to learn how to access them.

Now that you’ve created child accounts, to access them, you can assume the IAM role that AWS Organizations creates for you automatically in those accounts. There are many different ways to assume an IAM role. For example, in the AWS Web Console, you can click on your username in the top right corner and choose "Switch role," as shown in Figure 65:

On the next page, enter the ID of one of your child accounts (e.g., grab the dev account ID from the dev_account_id output variable), OrganizationAccountAccessRole as the role name, give it a display name (e.g., dev-admin) and optionally pick a color to use in the console, as shown in Figure 66:

Click the Switch Role button, and you should see the AWS Web Console for that child account. At this point, you have admin permissions in that child account, and you can do whatever you want. Feel free to repeat this process with the staging and production accounts, picking a different display name and display color for each one.

There are several ways to assume IAM roles in the terminal, too. One option is to configure an AWS profile for each child account. For example, to create a profile for your dev account, open up the AWS config file, which lives at ~/.aws/config (if the file doesn’t exist already, create it), and add the code shown in Example 112 to it:

The preceding code does the following:

Most tools that talk to AWS APIs give you a way to specify the profile to use. For example, with the AWS CLI, you can use the --profile argument, as shown in Example 113:

The get-caller-identity command returns information about the authenticated user, so if you configured the profile correctly, you should see your dev account’s ID in Account, and the ARN of the OrganizationAccountAccessRole dev IAM role in Arn.

Create analogous profiles for the stage and prod accounts. In the next section, you’ll see how to use these profiles to deploy infrastructure into the dev, stage, and prod accounts.

Let’s now try to deploy the lambda-sample module from earlier blog posts into the dev, stage, and prod accounts. Copy the lambda-sample module from Part 5, as well as the test-endpoint module it relies on, into a new ch6/tofu/ folder:

Next, make three changes to the lambda-sample module:

In OpenTofu, you can use workspaces to manage multiple deployments of the same configuration. Each workspace has its own state file, so it represents a separate copy of all the infrastructure, and each workspace has a unique name, which is returned by terraform.workspace. If you don’t specify a workspace explicitly, as you’ve been doing so far throughout this blog post series, then you end up using a workspace called "default." In this blog post, instead of using the default workspace, let’s create one workspace per environment.

First, authenticate to your management account as usual (as described in Authenticating to AWS on the command line), and run tofu init to initialize the backend, modules, and providers:

Next, use the tofu workspace new command to create a new workspace:

The preceding command creates a workspace called "development," which you can use to store the state for your development environment. The idea is to deploy the development environment into the new development account you created earlier. You can do this by running tofu apply and telling OpenTofu to authenticate to your development account by setting the aws_profile input variable to the name of the profile you created for the development account in the previous section:

You should see a plan output to create the Lambda function, API Gateway route, and so on. If everything looks good, type in yes and hit Enter. When apply completes, you should see the api_endpoint output variable, which contains a URL you can try to access the Lambda function. Try this URL out:

Congrats, you now have a serverless web app running in your development account! Let’s now try to deploy the same app into the staging account. First, create a new workspace for staging:

Next, run apply again, but this time, set aws_profile to the name of the profile you created for your staging account:

You should see a plan output that shows OpenTofu will create all the resources (Lambda function, API Gateway route, etc.) again from scratch. That’s because each workspace has its own state file, so when you’re in the staging workspace, OpenTofu doesn’t look at any of the infrastructure you deployed in the development workspace. If everything looks good with the plan, type in yes and hit Enter. When apply completes, you should have a different URL you can try:

And there you go, you now have a second environment running in a second AWS account! Complete the picture by deploying into the third environment, production (note the use of the full name "production" for this workspace, rather than "prod," as that results in NODE_ENV being set to "production," which is the value you want to use with Node.js apps in production):

When you’re done, you should see "Hello from production!" At this point, you have three environments, across three AWS accounts, with a separate copy of the serverless webapp in each one, and the OpenTofu code to manage it all.

You now have three copies of the serverless webapp running, but all three are configured exactly the same way. Let’s see what it might look like to configure the app differently in each environment. To keep things simple, we’ll use JSON configuration files checked into version control. First, create a folder called config for the configuration files:

Within the config folder, create a file called development.json, with the contents shown in Example 119:

This file contains just a single config entry, text, which is the text the web app should return in that environment. Create analogous config/staging.json and config/production.json files, but with text updated to different values in each environment.

Next, update index.js to load the config file for the current environment and return the text value in the response, as shown in Example 120:

There are two updates to the app:

Now it’s time to deploy this change in each environment—that is, in each workspace. To see all your workspaces, use the workspace list command:

You can switch to any existing workspace using the workspace select command:

Now, any OpenTofu command you run will against the development workspace. Run apply with the aws_profile variable set to the dev profile to deploy the changes to the development environment:

When apply completes, open the URL in the api_endpoint output variable, and you should see "Hello from dev config!" Use workspace select and apply (with the aws_profile variable properly set) to deploy the changes in staging and production as well. When you test the URLs for those environments, you should see the text values you put into those configs: e.g., "Hello from stage config!" and "Hello from prod config!" Congrats, you’re now loading different configuration files in different environments!

When you’re done testing and experimenting with multiple AWS accounts, you may wish to close some or all of the new child accounts. Going forward, just about all the examples in this blog post series will deploy into just a single account (to keep things simple), so you don’t need all three running. Note that AWS does not charge anything extra for the accounts themselves, but you may want to clean them up to keep your security surface area smaller, and to ensure you don’t accidentally leave resources running in those accounts (e.g., EC2 instances), as AWS does charge for those as usual.

First, commit all your code changes to Git: that way, if you ever want to bring back the three accounts, you’ll have all the code to do it.

Second, undeploy the infrastructure in each workspace. To do that, use workspace select to select each environment and then run tofu destroy, making sure to set the aws_profile variable to the profile you created for that environment. For example, here is how you undeploy the infrastructure in the development workspace:

Repeat the same workspace select and tofu destroy commands for the staging and production environments.

Third, run tofu destroy on the child-accounts module to start the process of closing the child accounts:

When you run destroy, AWS will initially mark the child accounts as "suspended" for 90 days, which is a fail-safe that gives you a chance to recover anything you may have forgotten in those accounts before they are closed forever. After 90 days, AWS will automatically close those accounts.

The most common reason to break up a codebase is that it doesn’t fit in your head. Once a codebase gets big enough, no one can understand all of it. There are just too many parts, too many interactions, and too many concepts to keep straight, and if you have to deal with all of them at once, your pace of development will slow to a crawl, and the number of bugs will explode. Consider Table 11, which is a table from the book Code Complete that shows the typical number of bugs in software projects of various sizes:

It’s no surprise that larger software projects have more bugs, but note that Table 11 shows that larger projects also have a higher bug density, or the number of bugs per 1,000 lines of code. To put this into perspective, take a developer, and have them add 100 lines of code to a small software project (<2K lines of code), and on average, you’ll find that new code has no new bugs, or maybe one or two. Take the same developer and have them add 100 lines of code to a large software project (>512K lines of code), and on average, you’ll find that they have introduced as many as ten new bugs. Same developer, same number of lines of code, but 5-10x as many bugs. That’s the cost of complexity.

There is a limit to how much code complexity the human mind can handle. In fact, in that same Code Complete book, author Steve McConnell defines "managing complexity" as "the most important technical topic in software development." There are many techniques for managing complexity, but almost all of them come down to one basic principle: divide and conquer. That is, find a way to organize your code so that you can focus on one small part at a time, while being able to safely ignore the rest. One of the main goals of most software abstractions, including object-oriented programming, functional programming, libraries, and microservices is to break up the codebase into discrete pieces, so that each piece can hide its implementation details, which are fairly complicated, and expose to you some sort of interface, which is much simpler.

Another common reason to break up a codebase has less to do with technical challenges, and more to do with social and human challenges: the desire to allow teams to work independently from each other and to have total ownership of their part of the product. As your company grows, different teams will start to develop preferences for different product development practices: that is, different practices around how they design their systems and architecture, how they test and review their code, how often they deploy, and how much tolerance they have for bugs and outages.

If you do all your work in a single, tightly-coupled codebase, then a problem in any one team or product can affect all the other teams and products, and that’s not always desirable. For example, if you open a pull request, and an automated test fails in some totally unrelated product, should that block you from merging? If you deploy new code that includes changes to ten different products, and one of them has a bug, should you roll back the changes for all ten products? If one team wants to deploy dozens of times per day, but another team has a product in a regulated industry where they can only deploy once per quarter, should everyone be stuck with the slower deployment cadence? Splitting up the codebase allows you to avoid difficult questions like these and set up separate processes for each team that meet their specific needs.

Note that teams working independently from each other doesn’t mean they never interact. It’s just that the interactions are now limited to well-defined interfaces: e.g., the API of a library or a web service. This lets you benefit from the output of that team’s work (e.g., the data returned by their API) without being subject to the particular inputs they need to make that work possible. In fact, you do this all the time, even in small companies whenever you add a dependency on a third party, such as an open source library or a vendor’s API: you’re able to benefit from the work they are doing, while keeping all your coding practices (testing, code reviews, deployment cadence, etc.) largely separate.

As your user base grows, you will hit more and more scaling challenges to handle the extra load. In some cases, you may find that some parts of your software have different scaling requirements than other parts: for example, one part of your code may benefit from distributing work across a large number of CPUs on many servers, whereas another part of your code may benefit from a large amount of memory on a single server. If everything is in one codebase—and more to the point, if everything is deployed together—meeting these conflicting scaling requirements can be difficult. As a result, many companies break up their codebase so that the different parts of the code can be deployed and scaled independently.

Most companies start with a single programming language, but as you grow, you may end up using multiple programming languages: sometimes, this is because different developers at your company prefer different languages; sometimes, this is because you acquired a company that uses a different programming language; sometimes, this is because different languages may be a better fit for different problems (use the right tool for the job). Each time you introduce a new language, you have a new app to deploy, configure, update, and so on, and as you’ll see shortly, this typically means your codebase now consists of multiple services to manage.

Now that you’ve seen the most common reasons to break up a codebase, let’s talk about how to actually do it.

Just about all codebases are broken up into various abstractions, such as functions, interfaces, classes, and modules (depending on the programming language you’re using). However, if the codebase gets big enough, you may choose to break it up even further into libraries. When I say library, what I mean is a unit of code that can be developed independently from the other units, as it meets the following properties:

For example, you might start with a code base that has three parts, A, B, and C. Initially, part A depends directly on the source code of B and C, as shown in Figure 67:

You could break up this codebase by turning B and C into libraries that publish artifacts (e.g., if this was Java, the artifacts would be b.jar and c.jar), and update A to depend on a specific version of these artifacts, instead of the source code, as shown in Figure 68:

The advantage of breaking up your code into libraries is that it allows you to focus on one library at a time—one small part of your codebase—and safely ignore everything else. Each team can develop the internals of a library using whatever practices they want (e.g., for testing, code reviews, etc.), so long as the public interface fulfills its promises. Moreover, whereas a source code change immediately affects everyone that depends directly on that source code, a change to a library only affects users after (a) you’ve released a new versioned artifact and (b) users of your library have explicitly and deliberately chosen to pull in that new version. This allows teams to work more independently from each other.

Almost all software projects these days depend on libraries: namely, open source libraries. For example, the Node.js sample app you’ve been working on throughout this blog post series now depends on Express.js, an open source web framework that you pull in through a versioned artifact (an NPM module). The maintainers of Express.js are able to develop this library completely independently from all the projects that depend on it, following their own coding conventions, testing practices, release cadence, and so on. That doesn’t mean you need to open source your own code, but if you break up your codebase into libraries, you can likewise benefit from being able to develop each piece independently.

If you do break your codebase up into libraries, I recommend following two best practices: semantic versioning and automatic updates

Semantic versioning (SemVer) is a set of rules for how to assign version numbers to your code. The goal is to communicate to users if a new version of your library has backward incompatible changes: that is, changes that would require the user to update how they use your library in their code in order to make use of this new version. Typically, this happens when you make changes to the API: e.g., you remove something that was in the API before, or you add something new to the API that is now required. With SemVer, you use version numbers of the format MAJOR.MINOR.PATCH (e.g., 1.2.3), where you increment these three parts of the version number as follows:

For example, if your library is currently at version 1.2.3, and you have made a backward incompatible change to the API, then to communicate this to your users, the next release would be 2.0.0. On the other hand, if you made a backward compatible bug fix, the next release would be 1.2.4. It’s also worth mentioning that 1.0.0 is typically seen as the first backward compatible release: new software can start at 0.x.y version numbers to indicate that it is not yet providing backward compatibility guarantees.

Automatic updates is a way to keep your dependencies up to date. One of the benefits of using library dependencies is that changes to that library only affect you when you explicitly and deliberately pull in a new version of that library. However, this strength is also a drawback: it’s easy to forget to update a library and to be stuck with an old version for years. This can be a problem as the old version may have bugs or security vulnerabilities, and if you don’t update for a while, updating to the latest version to pick up a fix can be difficult, especially if there have been many breaking changes since your last update.

This is yet another place where, if it hurts, you need to do it more often. In particular, you want to set up a process where you automatically update your dependencies and roll those updates out to production (sometimes called software patching). This applies to all the different types of software you depend on, including open source libraries, internal libraries, the operating systems your software runs on, software from cloud providers like AWS, GCP, and Azure, and so on. The automation you set up can either run on a schedule (e.g., update weekly) or in response to new versions being released.

You can set up automated updates using tools such as DependaBot, Renovate, Snyk, and Patcher, which can detect dependencies in your code, and automatically open pull requests to update you to new versions. That way, instead of having to remember to do updates yourself, the updates come to you, and all you have to do is check that they pass your suite of tests (as per Part 4), and if those pass, merge the pull request in, and let the code deploy automatically (as per Part 5).

Consider parts A, B, and C from the previous section: whether you use source code dependencies or library (artifact) dependencies, all the parts of your codebase run in a single process and communicate with each other via in-memory function calls, as shown in Figure 69:

Another way to break up the codebase is to run the different parts of your codebase as separate services: that is, you run each part in a separate process, typically on separate servers, and they communicate with each other by sending messages over the network, as shown in Figure 70:

Instead of a single, monolithic application, you move to multiple services, where each service meets the following properties:

Over the years, there have been many different approaches to building services—and also many buzzwords and fads, which can make it hard to nail down concrete definitions. One approach is service-oriented architecture (SOA), which typically refers to building relatively large services that handle all the logic for an entire business line or product within your company; this was also sometimes called Web 2.0, when it referred to services exposed between different companies (e.g., when you use APIs from services exposed by Twitter, Facebook, Google Maps, etc.). A slightly more recent approach that arose around the same time as DevOps is microservices, which typically refers to smaller, more fine-grained services that handle one domain within a company: e.g., one microservice to handle user profiles, one microservice to handle search, one microservice to do fraud detection, and so on. Yet another approach is event-driven architecture, where instead of services interacting synchronously by messaging each other and waiting for responses, they interact asynchronously, with each service listening for events, which are typically messages on some sort of event bus, processing those events, and creating new events by writing back to the event bus.

Whichever model of services you choose, there are typically three main advantages to breaking up your code into services:

Almost all large companies eventually move to services for these three advantages, but especially due to the ability to isolate teams. To some extent, using services allows each team to operate like its own, independent company, which is essential to scaling.

Moving to services can be an essential ingredient in helping a company scale, but beware: breaking up the codebase, whether into libraries or services, comes with a number of costs and challenges, so most companies should avoid it until they have no other choice, as described in the next section.

Both libraries and services consist of two parts: the public API and the internal implementation details. Breaking up your codebase allows you to make changes much more quickly to the internal implementation details, as each team can maintain those however they want. However, making changes to the public API becomes slower and more difficult, as you now need to worry about backward compatibility. Making backward incompatible changes (AKA breaking changes) in a library or service can cause headaches, bugs, and outages for everyone who depends on your library or service, so you have to be careful in changes to the public API.

For example, imagine that in part B of your codebase, you have a function called foo that you want to rename to bar. If all the code that depends on B is in one codebase, renaming is straightforward:

If B is a separate library, the process is more complicated:

If B is a separate service, the process is even more complicated:

Phew. That’s a lot of work. If you spend enough time maintaining a library or service, you quickly learn how important it is to get the public API right, and you’ll likely spend a lot of time obsessing over your public API design. But no matter how much time you spend on it, you’ll never get it exactly right, and you’ll always have to evolve it over time anyway, so expect public API maintenance to be one of the overheads of splitting up the codebase.

The reason it’s hard to maintain a public API in libraries and services is because that’s a place where you have to interact with many other teams at your company. As it turns out, this is just one specific type of change that becomes much harder if you split up your codebase: the more general problem is that any global changes—changes that require updating multiple libraries or services—become considerably harder.

For example, LinkedIn, like almost all companies, started with a single monolithic application: this one was written in Java and called Leo. Eventually, Leo became a bottleneck to scaling, both in terms of scaling to handle more developers, and scaling to handle more traffic, so we started to break it up into dozens of libraries and services. For the most part, this was a huge win, as each team was able to iterate on features within their library or service much faster than when those features were mixed with everyone else’s features within Leo. However, we also had to do the occasional global change: for example, almost every single service relied on some security utilities in a library called util-security.jar. When we found a vulnerability in that library, rolling out the new version to all services took a gargantuan effort:

What would’ve been a single commit and deploy within a monolith became a multi-week slog when dealing with dozens of microservices. To some extent, this is by design: the whole point of splitting up a codebase is to make it hard for changes in other parts of the codebase to affect you.

If you split up your codebase and find that the vast majority of the changes each team makes are within a single part of the codebase owned by that team, then this split will allow you to go much faster; but if you find that the majority of the changes require teams to make updates across multiple parts of the codebase, then this split will make you go much slower. Unfortunately, knowing where to spit up the codebase can be surprisingly challenging, as discussed in the next section.

One of the challenges of splitting up a codebase is knowing where to put the seams. You know you’ve split it correctly when the vast majority of the changes done by each team are within their own part of the codebase, as that will allow them to go much faster. However, it’s very easy to get the split wrong, and end up in a situation where most changes require updating multiple libraries or services, and as you learned in the previous section, these sorts of global changes will make you go much slower.

One place I see teams get this wrong all the time is splitting up the codebase way too early. It’s much easier to identify the seams in a codebase that has been around for a long time than it is to guess where to put the seams in something totally new. When you’ve been working with a codebase for years, you can usually identify several patterns that are good hints for where the codebase could be split:

Trying to predict any of these items ahead of time for a new codebase is futile. This is especially true of performance bottlenecks, which you can never really predict without running a profiler against real code and real data. The only way to get these seams right is to start with a monolith, grow it as far as you can, and only when you can’t scale it any further, do you break it up into smaller pieces.

This is one of the reasons that I shake my head when I see a tiny startup with a three-person engineering team launch their product with 12 microservices: not only are you going to pay a high price in terms of operational overhead (something you’ll learn more about shortly), but you almost certainly have put the seams in the wrong places. Inevitably, these teams find that every time they go to make the slightest change in their product, they have to update 7 different microservices, and deploy them all in just the right order. Meanwhile, the startup team who built on top of a Ruby on Rails or PHP monolith is running circles around them, shipping changes 10x faster.

In Part 5, you learned all about continuous integration, and its central role in helping teams move faster. Well, here’s a fun fact: splitting up your codebase into libraries and services is the opposite of continuous integration. What’s the difference between a long-lived feature branch that you only merge into main after 8 months versus a library dependency that you only update once every 8 months? Not much.

Once you’ve split up your codebase, what you’re effectively doing is late integration. And that’s by design: one of the main reasons to split up a codebase is to allow teams to work more independently from each other, which means you are going to be integrating your work together much less frequently.

This is a good trade-off to make if the different teams are truly decoupled from each other: e.g., they work on totally separate products within your company. However, if the teams are actually tightly coupled, and have to interact often, then splitting them up into separate codebases will lead to problems. Either the teams will try to work mostly independently, and due to the lack of integration and proper testing run into lots of conflicts, bugs, and outages, or the teams will try to integrate their work all the time, and due to the frequent need to make global changes across multiple parts of the codebase, they will find development is very slow.

One challenge that is unique to libraries is what’s sometimes referred to as dependency hell, which is where using versioned dependencies can lead to one of a number of frustrating situations, such as the following:

Just about all codebases run into these issues from time to time due to dependencies on open source libraries, but if you break your own codebase up into many libraries, these problems may become exponentially worse.

If you split a monolith into services, instead of having just a single type of app to manage, you now have many different types, possibly written in different languages, each with its own mechanisms for testing, deployment, monitoring, configuration, and so on. Think of all the work you’ve done so far in this blog post series to deploy a single app and a CI / CD pipeline for it, add all the work in upcoming posts, such as networking, secrets management, data storage, and monitoring, and then multiply that by the number of services, and you’ll get an inkling of the operational overhead involved. But that’s not all. There is even more operational overhead from dependencies between services, debugging of multiple services, infrastructure for multiple services, and the performance impact from services, as discussed in the next several sections.

With N services, you not only have N things to deploy and manage, but you also have to consider the interactions between services, which grows at a rate of N². For example, let’s say you have a service A that depends on a service B. As part of developing a new feature, you add a new endpoint called foo to B, and you update the code in A to make calls to the foo endpoint. Now consider what happens at deployment time: if you deploy the new version of A before the new version of B is out, then when A tries to use the foo endpoint, it’ll fail, as the old version of B doesn’t have that endpoint yet. So now you have a dependency on deployment ordering: B must be deployed before A. But of course, B itself may depend on new functionality in services C and D, and those services may depend on new functionality in other services, and so on. So now you have a deployment graph to maintain to ensure the right services are deployed in the right order. And this gets really messy if one of those services has a bug, and you have to rollback: for example, if the new version of C had to be rolled back, you’d also have to know to roll back the new versions of B and A.

One way to mitigate this problem is to ban deployment ordering entirely: that is, you require your code to be written so that services can be deployed in any order, and rolled back at any time. One way to do that is to use feature flags, which you saw in Part 5: you wrap the new functionality in A—the part of the code that calls the new foo endpoint in B—in an if-statement which is off by default. That way, you can deploy A and B at any time and in any order, as the new functionality won’t be visible to any users. When you’re sure both the new versions of A and B are deployed, you then turn the feature toggle on, and the new functionality should start working; if you hit any issue with the new functionality or B or one of its dependencies has to be rolled back, you can turn the feature toggle off again. Once again, you are using feature toggles to separate deployment from release, which is usually a more effective solution than trying to implement deployment ordering—but still nowhere near as simple as avoiding the problem entirely by sticking with a monolith as long as you can.

If you have a single monolithic app, and your users report a bug, you know the bug is in the app. If you have dozens of services, and your users report a bug, now you have to do an investigation to figure out which service is at fault. This is considerably harder for several reasons. One reason is the natural tendency for each team to immediately blame other teams, so no one will want to take ownership of the bug. Another reason is that when you have services that communicate over the network, rather than a monolith where everything happens in a single process, there are a large number of new, complicated failure conditions that are tricky to debug (you’ll learn more about this shortly).

One more reason is that, whereas debugging a single app can be hard, trying to track down a bug across dozens of separate services can be a nightmare: you can no longer look at the logs of a single app, and instead have to go look at logs from dozens of apps, each potentially in a different place and format; you can no longer reproduce the error by running a single app on your computer, and instead have to fire up dozens of services locally; you can no longer hook up a debugger to a single process and go through all the code step-by-step, and instead have to use all sorts of tracing tools identify the dozens of different processes that end up processing a single request. A bug that could take an hour to figure out in a monolith can take weeks to track down in a microservices architecture.

Moving from a monolith to multiple services isn’t just about deploying a bunch of services: you typically also need to deploy a bunch of extra infrastructure to support the services themselves—and the more services you have, the more infrastructure you need to support them. For example, to help manage the deployments of 12 services, rather than 1 monolith, you may have to deploy a more complicated orchestration tool (e.g., Kubernetes); to help your services communicate with each other securely, you may have to deploy a service mesh tool (e.g., Istio); to help your services communicate with each other asynchronously, you may have to deploy an event bus (e.g. Kafka); to help with debugging and monitoring your microservices architecture, you may have to deploy a distributed tracing tool (e.g., Jaeger), and integrate a tracing library it into all your services (e.g., OpenTracing); and so on. All of this infrastructure takes a lot of time and money to deploy and manage, and you can avoid most of it by sticking with a monolith for as long as possible.

One of the benefits of services is that they help you deal with performance bottlenecks by allowing you to scale different parts of your codebase independently. One of the drawbacks of services is that, in almost every other way, they actually make performance considerably worse. This is due to the following reasons:

When you split a monolith into services, you often have to rewrite a lot of your code to use concurrency, caching, batching, and de-duping to minimize this performance overhead. However, this makes your code considerably more complicated, and it’ll still be orders of magnitude slower than keeping everything in a single process.

Splitting a monolith into services is a major shift: you’re turning a single app into a distributed system. Distributed systems are hard. Dealing with distrubted systems introduces a number of new challenges, such as the following:

Now that you’ve seen all the challenges with splitting your codebase into libraries and services, you might be feeling a little less excited about that sexy microservices architecture you saw at Google or Netflix. If so, that’s a good thing. You’re probably not working at Google or Netflix, so you shouldn’t blindly copy their architecture, as much of it was designed to handle problems of extraordinary scale, and if you don’t have those problems, then that architecture is more likely to slow you down than to help you.

Let’s assume that you’re at a company of a large enough scale to merit splitting up the codebase, and see what it looks like to run multiple services in Kubernetes.

As a first step, copy the Node.js sample app that you last saw in Part 5 into a new folder called sample-app-backend:

Since you’ll be deploying this backend into a Kubernetes cluster, you should also copy the Kubernetes Deployment and Service configurations from Part 3 into sample-app-backend:

Next, update the files in sample-app-backend as follows:

Build a Docker image for the backend app using the dockerize command you added back in Part 4:

This should result in a new Docker image called "sample-app-backend" with version 0.0.1.

Let’s deploy this Docker image into a Kubernetes cluster. The easiest one to test with is Kubernetes running locally, as part of Docker Desktop, just as you did back in Part 3. You can authenticate to the Kubernetes cluster in Docker Desktop as follows:

Now you can use kubectl apply to deploy the Deployment and Service:

If you run kubectl get services, you should see the Service for the backend:

Note the backend Service name; you’ll need this in the frontend app, which is the focus of the next section.

Create the frontend app using a process similar to the one you just used for the backend app. First, copy the Node.js sample app from Part 5 and the Kubernetes Deployment and Service configurations from Part 3 into a new folder called sample-app-frontend:

Next, update the files in sample-app-frontend as follows:

Build a Docker image for the frontend app using the dockerize command you added back in Part 4:

This should result in a new Docker image called "sample-app-frontend" with version 0.0.1.

Let’s deploy this Docker image into a Kubernetes cluster. Use kubectl apply to deploy the Deployment and Service:

If you run kubectl get services, you should now see the Services for both the backend and the frontend:

Notice how EXTERNAL-IP for the frontend is set to localhost and that it’s listening on port 80, so you can test it by going to http://localhost. If you open this URL in a web browser, you should see the HTML rendered, as shown in Figure 73:

Congrats, you’re now running two microservices in Kubernetes that are talking to each other! A separate team could own each service, developing, deploying, and scaling the service completely independently.

When you’re done testing, you may want to run kubectl delete on each of the Deployments and Services to undeploy them from your Kubernetes cluster. You should also commit your changes to Git, as you will continue to iterate on this code in subsequent chapters.