Part 6. How to Work with Multiple Teams and Environments

You typically need a way to test changes to your software (a) before you expose those changes to your users and (b) in a way that limits the blast radius, so if something goes wrong during testing, the damage is constrained, and doesn’t affect users or your production environment.

To some extent, as soon as you deployed your app onto a server (in Part 1), you already had two environments: your local development environment (LDE), which is your own computer, and production, which is the server. Usually, the differences between your LDE and production are large enough that testing solely in the LDE is not sufficient. What you need is one or more environments that closely resemble production, but are completely isolated. A common setup you’ll see at many companies is to have the following three environments:

This trio of development, staging, and production, often shortened to dev, stage, and prod, shows up at most companies, although sometimes with slightly different names: e.g., stage is sometimes called QA, as that’s where the quality assurance (QA) team does testing, or UAT (user acceptance testing), as that’s where end-users and stakeholders test the application to make sure it meets expectations.

Larger companies often have multiple products and multiple product teams, and at a certain scale, having all of them work in the same environment or even the same set of environments can lead to a number of problems, as different products may have different requirements in terms of security, compliance, uptime, deployment frequency, and so on. Therefore, it’s common in larger companies for each team or product to have its own isolated set of environments.

For example, the search team might have their software deployed in the search-dev, search-stage, and search-prod environments, while the profile team might have their software deployed in the profile-dev, profile-stage, and profile-prod environments. This ensures that teams can customize their environments to their own needs, limits the blast radius if one team or product has issues, and allows teams to work mostly in isolation from each other.

If you have users in multiple locations around the world, you may want to run your software on data centers that are geographically close to those users. One of the big reasons for this is latency: that is, the amount of time it takes to send data between your data center and your users' devices. This information is traveling at nearly the speed of light, but when you’re building software used across the globe, the speed of light can be too slow! Table 10 shows the latency of common computer operations:

These numbers are useful for doing back-of-the-envelope calculations. For example, you can estimate that having data centers on multiple continents, rather than one continent, will reduce latency to your users by around 100,000,000 ns (100 ms). This might not seem like much, but remember, this is the overhead for a single TCP packet, which is typically limited to 1 KB in size, and most apps these days sends hundreds or thousands of KB of data, so the extra latency can add up to many seconds of additional overhead for every page load and button press.

Therefore, companies with a global reach often end up deploying software across multiple data centers across the globe. For example, you might have one production environment in Ireland (prod-ie) to give EU users lower latency and one production environment in the US (prod-us) to give your North American users lower latency.

If you operate in certain countries, work in certain industries, or work with certain customers, you may be subject to laws and regulations that require you to set up your environments in specific ways. For example, if you store and process credit card information, you may be subject to PCI DSS (Payment Card Industry Data Security Standard); if you store and process healthcare information, you may be subject to HIPAA (Health Insurance Portability and Accountability Act) and HITRUST (Health Information Trust Alliance); if you are building software for the US government, you may be subject to FedRAMP (Federal Risk and Authorization Management Program); and if you are building software in certain countries, you may be subject to data residency laws, such as the EU’s GDPR (Global Data Protection Regulation), which requires businesses that operate in an EU country, or have customers in an EU country, to store and process data on servers physically located within that country.

A common pattern is to set up a dedicated environment for complying with laws and regulations. For example, if you’re subject to PCI DSS, you might have one prod environment that meets all the PCI DSS requirements, and is used solely to run your payment processing software, and another prod environment that isn’t as locked down, and is used to run all your other software. Setting up multiple environments this way lets you minimize the surface area, and thus cost and time, of what has to be administered under the more stringent requirements of a compliance standard.

In Part 3, you saw that a single server can be a single point of failure; the solution was to deploy multiple servers. It turns out that, even if you have multiple servers, if all of them are in a single data center (a single environment), that one data center can be a single point of failure, too. It’s possible for a power outage, cooling problem, network connectivity issue, and a variety of other problems to disrupt the functionality of an entire data center, and all the servers within it. Companies that need a higher degree of resiliency deploy across multiple data centers that are in separate locations around the world (e.g., prod-ie and prod-us, as in the previous section).

Now that you’ve seen a few of the reasons to break up your deployment across multiple environments, let’s talk about how to actually do it.

Perhaps the most obvious challenge with multiple environments is that you now have more moving parts to set up and maintain. You may need to run more servers, set up more data centers, hire more people around the world, and so on. Using the cloud allows you to offload much of this overhead onto the cloud provider, but managing multiple AWS accounts still results in more overhead, as each account needs to be configured with authentication, authorization, networking, security tooling, and so on. But even this overhead may be just a drop in the bucket compared to the overhead of having to change your entire architecture to work across environments that are geographically separated, as discussed in the next section.

Having multiple data centers around the world, so they are closer to your users, reduces the latency between the data center and those users, but it may also increase the latency between the different parts of your software running in different data centers. This may force you to rethink your architecture, especially when it comes to data storage.

For example, let’s say you had a web app that needed to query a database. If the database you were talking to was in the same data center as the app, then as per Table 10, the networking overhead for the query would be roughly 500,000 ns (0.5 ms) for each packet round trip, which is negligible for most web apps. However, if you had multiple data centers around the world, and the database you were talking to was on a different continent, now the networking overhead could be as high as 183,000,000 ns (183 ms), a 366x increase for every single packet you send. Even a single database query will typically require multiple packets to make round trips, so this extra overhead adds up quickly, and it can make your app unusably slow.

No problem, you say, you’ll just ensure that the database is always in the same data center as the web app. But now you need multiple databases, and that may require you to change how you store and retrieve data, including how you generate primary keys (an auto incrementing primary key will not work with multiple data stores), how you handle data consistency (uniqueness constraints, foreign key constraints, and transactions all become difficult with multiple databases), how you look up data (querying and joining multiple databases is complicated), and so on (you’ll learn about these topics in Part 9).

Some companies choose to avoid these challenges by only running in active/standby mode, where one data center is active and serves live traffic, and the other is a standby that only serves live traffic if the active data center goes down. That way, you are only ever reading/writing data in one location at a time. This is useful to boost resiliency, but doesn’t help with latency or local laws and regulations. If you have to have multiple data centers live at the same time, known as active/active, then you will most likely have to rearchitect your data storage patterns to work across multiple geographies.

With multiple environments, you may have to configure your apps differently in each one. Some of the settings that may differ between environments include performance settings (e.g., CPU, memory, hard-drive, garbage collection), security settings (e.g., database passwords, API keys, TLS certificates), networking settings (e.g., IP addresses, ports, domain names), service discovery settings (e.g., IP addresses, ports, and domain names for the services you rely on), and feature settings (feature toggles to turn on and off).

In the early stages of a company, with a small number of apps and environments, managing these configurations is straightforward. As you grow, and you add more apps, environments, and complexity, the number of configuration settings can explode. For large companies, it’s not unusual to have thousands of settings to manage, which can be a significant challenge. In fact, Google found configuration changes to be one of their biggest causes of outages, as shown in Table 11:

Google found that pushing configuration changes is just as risky as pushing code changes (pushing a new binary), and the longer a system has been around, the more configuration changes tend to become the dominant cause of outages.

So how should you manage application configuration to minimize these problems? Broadly speaking, there are two methods for configuring applications:

I recommend using build-time configuration for as much of your configuration as possible. That way, you can treat it just like the rest of your code, ensuring every configuration change ends up in version control, gets code reviewed, and goes through your entire CI/CD pipeline (including all the automated tests). I only recommend using run-time configuration for use cases where the configuration changes frequently, such as service discovery and feature toggles, where having to deploy new code to get the latest configuration values would be too slow.

Now that you’ve seen the reasons to deploy into multiple environments, the different options for setting up multiple environments, and the challenges involved with multiple environments, let’s try an example: setting up multiple AWS accounts.

You probably created your first AWS account by signing up on the AWS website. To create more AWS accounts, instead of signing up again and again on the website, you can treat your initial AWS account as the management account, and use AWS Organizations to create all the other accounts as child accounts.

Typically, since the management account has powerful permissions, you only use it to create and manage other AWS accounts, and you do not run any other workloads in it. Therefore, as a first step, undeploy everything from your AWS account that you deployed in earlier blog posts: e.g., run tofu destroy on any OpenTofu modules you previously deployed and use the EC2 Console to manually undeploy anything you deployed via Ansible, Bash, etc (see also AWS Resource Explorer and Cloud Nuke). When you’re done, you should essentially have an empty AWS account.

Next, you can use the aws-organizations module, which is in the blog post series’s sample code repo in the ch6/tofu/modules/aws-organizations folder, to create three child accounts (development, staging, and production) using AWS Organizations. Head into the folder where you’ve been working on the code samples for this blog post series and make sure you’re on the main branch, with the latest code:

Next, create a new ch6 folder for this blog post’s code examples, and within the ch6 folder, create a tofu/live/child-accounts folder:

Within that folder, create main.tf with the contents shown in Example 101:

The preceding code uses the aws-organizations module as follows:

Next, create an outputs.tf file with the output variables shown in Example 102:

The preceding code adds an output variable called accounts which will contain data about your newly-created AWS accounts, as you’ll see shortly. Deploy the child-accounts module as usual, authenticating to AWS, and running init and apply:

After apply completes, you should see the accounts output variable, which will contain an id and iam_role_arn for each of your child accounts:

The id is the ID of the child account. The iam_role_arn is the ARN of an IAM role AWS Organizations automatically creates in each child account, which you can assume from the management account to get admin permissions in that child account (more on this in the next section). Congrats, you just created new AWS accounts using code!

You now have some new AWS accounts, but they aren’t useful until you deploy infrastructure into them, and to do that, you need to learn how to access them.

Now that you’ve created child accounts, there are several different ways to access them. I recommend using IAM Identity Center to grant users access to an appropriate permission set in your newly-created child accounts, and then using profiles in the AWS CLI to authenticate to those accounts in your terminal (see How to authenticate to AWS with IAM Identity Center for instructions). Alternatively, if you have an IAM user, you can create profiles for the AWS CLI that assume the IAM roles (based on the iam_role_arn values) that AWS organizations created in each account (see Switch to an IAM role (AWS CLI) for instructions).

I’m going to assume that, using either IAM Identity Center or IAM roles, you created profiles for the AWS CLI called dev-admin, stage-admin, and prod-admin to access the dev, stage, and prod accounts, respectively. Most tools that talk to AWS APIs give you a way to specify the profile to use. One way to do this is to use the AWS_PROFILE environment variable. For example, Example 103 shows how you can use the dev-admin profile to access the dev account:

The get-caller-identity command returns information about the authenticated user, so if you configured the dev-admin profile correctly, ACCOUNT_ID should be the ID of the dev account. In the next section, you’ll see how to use profiles to deploy infrastructure into the dev, stage, and prod accounts.

Let’s now try to deploy the lambda-sample module from earlier blog posts into the dev, stage, and prod accounts. Copy the lambda-sample module from Part 5 into a new ch6/tofu/ folder:

Next, make the following changes to the lambda-sample module:

You’re now ready to deploy this function across multiple environments. To configure each environment, you’ll create a variable definition file, which is a file with a .tfvars extension that sets the input variables for that environment. For example, to configure the dev environment, create dev.tfvars with the contents shown in Example 108:

This sets the name of the environment to "development" and configures the Lambda function to use 128 MB of memory. Create an analogous stage.tfvars file for the staging environment, as shown in Example 109:

And finally, create prod.tfvars, as shown in Example 110:

Notice how the production environment allocates more memory for the Lambda function. This is a common pattern where you configure pre-production environments with fewer resources (to save money) and production environments with more resources (to handle the load).

To deploy the dev environment, first, run tofu init, adding the -var-file flag to pass in the variables definition file for dev, and the -reconfigure flag to tell OpenTofu to configure a new backend, ignoring any previous backend configuration:

This will configure OpenTofu to store state in development.tfstate. Next, authenticate to your dev account by setting the AWS_PROFILE environment variable to the dev profile you created in the previous section, and run tofu apply:

You should see a plan output to create the Lambda function, Lambda function URL, and so on. If everything looks good, type in yes and hit Enter. When apply completes, you should see the function_url output variable, which contains a URL you can try to access the Lambda function. Try this URL out:

Congrats, you now have a serverless web app running in your dev account! To deploy into staging, repeat the process, but use the staging variable definition file, and authenticate to the staging account:

You should see a plan output that shows OpenTofu will create all the resources (Lambda function, Lambda function URL, etc.) again from scratch. That’s because you’re using a separate state file for each environment, so when you’re in the staging environment (staging.tfstate), OpenTofu doesn’t look at any of the infrastructure you deployed in the dev environment (development.tfstate). If everything looks good with the plan, type in yes and hit Enter. When apply completes, you should have a different URL you can try:

And there you go, you now have a second environment running in a second AWS account! Complete the picture by deploying into the third environment, production:

When you’re done, you should see "Hello from production!" At this point, you have three environments, across three AWS accounts, with a separate copy of the serverless webapp in each one, and the OpenTofu code to manage it all.

You now have three copies of the serverless webapp, all configured exactly the same way. In the next section, you’ll see how to configure each app differently.

Let’s configure the Lambda function in each environment differently. To keep things simple, we’ll use JSON configuration files checked into version control. First, create a folder called config for the configuration files:

Within the config folder, create development.json as shown in Example 111:

This file contains just a single config entry, text, which is the text the web app should return in that environment. Create analogous config/staging.json and config/production.json files, but with text updated to different values in each environment.

Next, update index.js to load the config file for the current environment and return the text value in the response, as shown in Example 112:

There are two updates to the app:

Now it’s time to deploy this change in each environment. Start with the dev environment, running init and apply with the -var-file and AWS_PROFILE settings:

When apply completes, open the URL in the function_url output variable, and you should see "Hello from development config!" Run the analogous init and apply commands to deploy the changes in staging and production as well. When you test the URLs for those environments, you should see the text values you put into those configs: e.g., "Hello from staging config!" and "Hello from production config!" Congrats, you’re now loading different configuration files in different environments!

When you’re done testing and experimenting with multiple AWS accounts, you may wish to close some or all of the new child accounts. Going forward, just about all the examples in this blog post series will deploy into just a single account (to keep things simple), so you don’t need all three running. Note that AWS does not charge anything extra for the accounts themselves, but you may want to clean them up to keep your security surface area smaller, and to ensure you don’t accidentally leave resources running in those accounts (e.g., EC2 instances), as AWS charges for those as usual.

First, commit all your code changes to Git. That way, if you ever want to bring back the three accounts, you’ll have all the code to do it.

Second, undeploy the infrastructure in each account. To do that, run tofu init and tofu destroy in each account, making sure to set the proper -var-file and AWS_PROFILE values. For example, here is how you undeploy the infrastructure in dev:

Repeat the analogous commands for the staging and production environments.

Third, run tofu destroy on the child-accounts module to start the process of closing the child accounts:

When you run destroy, AWS will initially mark the child accounts as "suspended" for 90 days, which is a fail-safe that gives you a chance to recover anything you may have forgotten in those accounts before they are closed forever. After 90 days, AWS will automatically close those accounts.

Once a codebase gets big enough, no one can understand all of it. There are just too many parts, too many interactions, and too many concepts to keep straight, and if you have to deal with all of them at once, your pace of development will drop, and the number of bugs will skyrocket. Consider Table 12, which shows the number of bugs in software projects of various sizes:

It’s no surprise that larger software projects have more bugs, but note that Table 12 shows that larger projects also have a higher bug density, or the number of bugs per 1,000 lines of code. To put this into perspective, take a developer, and have them add 100 lines of code to a small software project (<2K lines of code), and on average, you’ll find that new code has no new bugs, or maybe one or two. Take the same developer and have them add 100 lines of code to a large software project (>512K lines of code), and on average, you’ll find that they have introduced as many as ten new bugs. Same developer, same number of lines of new code, but 5-10x as many bugs. That’s the cost of complexity.

There is a limit to how much complexity the human mind can handle. In Steve McConnell’s book Code Complete (the source of Table 12), he defines managing complexity as "the most important technical topic in software development." There are many techniques for managing complexity, but almost all of them come down to one basic principle: divide and conquer. That is, find a way to organize your code so that you can focus on one small part at a time, while being able to safely ignore the rest. One of the main goals of most software abstractions, including object-oriented programming, functional programming, libraries, and microservices, is to break the codebase into discrete pieces, so that you only need to think about the simple interface of that piece, rather than the full complexity of the underlying implementation details.

Another common reason to break up a codebase is to allow teams to work more independently. As your company grows, different teams will start to develop preferences for different product development practices, such as how they design their systems and architecture, how they test and review their code, how often they deploy, and how much tolerance they have for bugs and outages.

If you do all your work in a single, tightly-coupled codebase, then a problem in any one team or product can affect all the other teams and products, and that’s not always desirable. For example, if you open a pull request, and an automated test fails in some totally unrelated product, should that block you from merging? If you deploy new code that includes changes to ten different products, and one of them has a bug, should you roll back the changes for the other nine? If one team wants to deploy dozens of times per day, but another team has a product in a regulated industry where they can only deploy once per quarter, should everyone be stuck with the slower deployment cadence? There are no easy answers to these questions in a single codebase, but if you split up the codebase, each team can use different processes that meet their specific needs.

Note that teams working more independently doesn’t mean they never interact or integrate their work together. It’s just that the integrations are now limited to well-defined interfaces: e.g., the API of a library or a web service. This lets you benefit from the output of that team’s work (e.g., the data returned by their API) without being subject to the particular inputs they need to make that work possible. In fact, you do this all the time whenever you add a dependency on a third party, such as an open source library or a vendor’s API. You’re able to benefit from the work they are doing, while keeping all your coding practices (testing, code reviews, deployment cadence, etc.) largely separate.

As your user base grows, you will hit more and more scaling challenges to handle the load, and you may find that some features have different scaling requirements than others. For example, one feature may benefit from distributing work across a large number of CPUs on many servers, whereas another feature may benefit from a large amount of memory on a single server. If everything is in one codebase—and more to the point, if everything is deployed together—meeting these conflicting scaling requirements can be difficult. As a result, many companies break up their codebase so that the different features can be deployed and scaled independently.

Most companies start with a single programming language, but as you grow, you may end up using multiple programming languages. Sometimes, this is because different developers at your company prefer different languages; sometimes, this is because you acquired a company that uses a different programming language; sometimes, this is because different languages may be a better fit for certain tasks. Each time you introduce a new language, you will have a new app to deploy, configure, update, and so on. In other words, you now have multiple services to manage, as you’ll see shortly.

Now that you’ve seen why you may want to break up a codebase, let’s talk about how to actually do it.

Just about all codebases are broken up into various abstractions, such as functions, interfaces, classes, and modules (depending on the programming language you’re using). However, if the codebase gets big enough, you may choose to break it up even further into libraries. An abstraction is a library if you no longer depend directly on the source code of that abstraction, but on a versioned artifact you publish for that abstraction. The exact type of artifact depends on the programming language: for example, in Java, that might be a .jar file; in Ruby, that might be a Ruby Gem; and in JavaScript, that might be an NPM module.

For example, you might start with a codebase that has three parts, A, B, and C. Initially, part A depends directly on the source code of B and C, as shown in Figure 50:

You could break up this codebase by turning B and C into libraries that publish artifacts (e.g., if this was Java, the artifacts would be b.jar and c.jar), and update A to depend on a specific version of these artifacts, instead of the source code, as shown in Figure 51.

Note that, as long as you use artifact dependencies, A, B, and C can all continue to live in a single repo, or you can split them into multiple repos. Multiple repos tends to be more common, as (a) it ensures you don’t accidentally fall back to source code dependencies, (b) it allows you to version the repos (e.g., using tags) independently, and (c) it gives teams more independence.

Breaking up your code into libraries has several advantages. First, it allows you to focus on one small part of your codebase (the library) at a time, while safely ignoring everything else. Second, each team can develop the internals of their libraries using whatever practices they want (e.g., for testing, code reviews, etc.). Third, teams can work more independently, as unlike source code dependencies, where every change immediately affects everyone who depends on your code, with libraries, your changes don’t affect anyone until (a) you’ve published a new versioned artifact and (b) users of your library have explicitly and deliberately chosen to pull in that new version.

Almost all software projects these days depend on libraries: namely, open source libraries. For example, the Node.js sample app you’ve been working on throughout this blog post series depends on Express.js, an open source web framework that you pull in through a versioned artifact (an NPM module). The maintainers of Express.js are able to develop this library completely independently of all the projects that depend on Express, following their own coding conventions, testing practices, release cadence, and so on. The point is not that you need to open source your own code, but that if you break up your codebase into libraries, you can benefit from being able to develop each library independently.

If you do split your codebase into libraries, I recommend the following practices:

Consider again parts A, B, and C from the previous section: whether you use source code dependencies (Figure 50) or library dependencies (Figure 51), all the parts of your codebase run in a single process and communicate with each other via in-memory function calls. Another way to break up the codebase is to move from a single monolithic application into multiple services, where each service is a part of your code that you develop independently and deploy in a separate process, typically on a separate server, and all communication between services is done by sending messages over the network, as shown in Figure 52:

Over the years, there have been many different approaches to building services, and also many buzzwords and fads, which can make it hard to nail down concrete definitions. One approach is service-oriented architecture (SOA), which typically refers to building relatively large services that handle all the logic for an entire business line or product within your company. This was also sometimes called Web 2.0, when it referred to services exposed between different companies (e.g., APIs from Twitter, Facebook, Google, etc). A slightly more recent approach that arose around the same time as DevOps is microservices, which typically refers to smaller, more fine-grained services that handle one domain within a company: e.g., one microservice to handle user profiles, one microservice to handle search, one microservice to do fraud detection, and so on. Yet another approach is event-driven architecture, where services communicate asynchronously (you’ll learn more about this approach in Part 9).

Whichever model of services you choose, there are typically three main advantages to breaking up your code into services:

Almost all large companies eventually move to services for these three advantages, but especially due to the ability to isolate teams. To some extent, using services allows each team to operate like an independent company, which is essential to scaling.

Moving to services can be an essential ingredient in helping a company scale, but beware, breaking up a codebase, whether into libraries or services, comes with a number of costs and challenges, so most companies should avoid it until the benefits clearly outweigh the costs described in the next section.

Both libraries and services consist of two parts: the public API and the internal implementation details. Breaking up your codebase allows you to make changes more quickly to the internal implementation details, as each team can maintain those however they want. However, making changes to the public API becomes slower and more difficult, as you now need to worry about backward compatibility. Making backward incompatible changes in a library or service can cause headaches, bugs, and outages for everyone who depends on your library or service, so you have to be extra careful with public API changes.

For example, imagine that in part B of your codebase, you have a function called foo that you want to rename to bar. If all the code that depends on B is in one codebase, this is easy:

If B is a separate library, the process is more complicated:

If B is a separate service, the process is even more complicated:

Phew. That’s a lot of work. If you spend enough time maintaining a library or service, you quickly learn how important it is to get the public API right, and you’ll likely spend a lot of time obsessing over your public API design. But no matter how much time you spend on it, you’ll never get it exactly right, so expect public API maintenance to be one of the overheads of splitting up the codebase.

The reason it’s hard to maintain a public API in libraries and services is because that’s a place where you have to interact with many other teams at your company. As it turns out, this is just one specific type of change that becomes harder if you split up your codebase. The more general problem is that any global changes—changes that require updating multiple libraries or services—become considerably harder.

For example, LinkedIn, like almost all companies, started with a single monolithic application. It was called Leo and was written in Java. Eventually, Leo became a bottleneck to scaling, both in terms of scaling to handle more developers and more traffic, so we started to break it up into dozens of libraries and services. For the most part, this was a huge win, as each team was able to iterate on features within their library or service much faster than when those features were mixed with everyone else’s features within Leo. However, we also had to do the occasional global change.

It was especially painful to make changes to a library called util-security.jar, which contained security utilities that almost every single service relied on. When we found a vulnerability in that library, rolling out the new version to all services took a gargantuan effort. First, we had to dig through dozens of services, many of which were defined in different repos, to find every service who depended on util-security.jar. Next, we had to update each of those services to the new version. Sometimes, this was a simple version number bump. Sometimes, the service was on an ancient version of util-security.jar, so we had to upgrade them through numerous breaking changes. Sometimes, the service depended on util-security.jar indirectly, through a chain of dependencies, so we had to update each of those dependencies in the chain first, and only then could we update the service. Every update required opening a pull request, waiting for code reviews, and bargaining with the team that owned the service to do a deployment.

What would’ve been a single commit and deploy within a monolith became a multi-week slog when dealing with dozens of microservices. To some extent, this is by design: the whole point of splitting up a codebase is to make it hard for changes in other parts of the codebase to affect you.

The challenge with splitting a codebase is knowing where to put the seams. If you get it right, then the majority of the changes you make will be within the part of the codebase owned by your team, which will allow you to go faster. If you get it wrong, then you’ll often have to make global changes across multiple parts of the codebase, which will make you go slower.

One place I see teams get this wrong all the time is splitting up the codebase too early. When you’ve been working with a codebase for years, you can usually look for the following patterns for where the codebase could be split:

It’s rare to be able to accurately predict any of these items ahead of time. This is especially true of performance bottlenecks, which are notoriously hard to predict without running a profiler against real code and real data. That’s why it’s almost always better to start with a monolith, grow it as far as you can, and only when you can’t scale it any further, and the seams are obvious, should you break it up into smaller pieces.

This is one of the reasons that I shake my head when I see a tiny startup with a three-person engineering team launch their product with 12 microservices. Not only are you going to pay a high price in terms of operational overhead (something you’ll learn more about shortly), but you almost certainly put the seams in the wrong places. Inevitably, these teams find that every time they go to make the slightest change in their product, they have to update 7 different microservices, and deploy them all in just the right order. Meanwhile, the startup who built on top of a Ruby on Rails monolith is running circles around them, shipping changes 10x faster.

In Part 5, you learned all about continuous integration, and its central role in helping teams move faster. Well, here’s a fun fact: splitting up your codebase into libraries and services is the opposite of continuous integration. What’s the difference between a long-lived feature branch that you only merge into main after 8 months versus a library dependency that you only update once every 8 months? Not much.

Once you’ve split up your codebase, what you’re effectively doing is late integration. And that’s by design: one of the main reasons to split up a codebase is to allow teams to work more independently from each other, which means you are going to be integrating your work together less frequently.

This is a good trade-off to make if the different teams are truly decoupled from each other: e.g., they work on totally separate products within your company. However, if the teams are actually tightly coupled, and have to interact often, then splitting them up into separate codebases will lead to problems. Either the teams will try to work mostly independently, and due to the lack of integration and proper testing run into lots of conflicts, bugs, and outages, or the teams will try to integrate their work all the time, and due to the frequent need to make global changes across multiple parts of the codebase, they will find development slows to a crawl.

An integration challenge that is unique to libraries is what’s sometimes referred to as dependency hell, which is where using versioned dependencies can lead problems such as the following:

Just about all codebases run into these issues from time to time due to dependencies on open source libraries, but if you break your own codebase up into many libraries, these problems may become considerably worse.

If you split a monolith into services, instead of having just a single type of app to manage, you now have many different types, possibly written in different languages, each with its own mechanisms for testing, deployment, monitoring, configuration, and so on. That’s already a lot of operational overhead, but there’s even more from the following items:

Now that you’ve seen all the challenges with splitting up your codebase, you might be feeling a little less excited about that sexy microservices architecture you saw at Google or Netflix. If so, that’s a good thing. You’re probably not working at Google or Netflix, so you shouldn’t blindly copy their architecture, as much of it was designed to handle problems of extraordinary scale, and if you don’t have those problems, then that architecture is more likely to slow you down than to help you.

Let’s assume that you’re at a company of a large enough scale to merit splitting up the codebase, and see what it looks like to run multiple services in Kubernetes.

As a first step, copy the Node.js sample app that you last saw in Part 5 into a new folder called sample-app-backend:

Since you’ll be deploying this backend into a Kubernetes cluster, you should also copy the Kubernetes Deployment and Service configurations from Part 3 into sample-app-backend:

Next, update the files in sample-app-backend as follows:

Build a Docker image for the backend app using the dockerize command you added back in Part 4:

This will create a new Docker image called "sample-app-backend" at version 0.0.1. Let’s deploy this Docker image into a Kubernetes cluster. The easiest one to test with is Kubernetes running locally in Docker Desktop, just as in Part 3. You can authenticate to the Kubernetes cluster in Docker Desktop as follows:

Now you can use kubectl apply to deploy the Deployment and Service:

If you run kubectl get services, you should see the Service for the backend:

Note the backend Service name. You’ll need this in the frontend app, which is the focus of the next section.

Create the frontend app using a process similar to the one you just used for the backend app. First, copy the Node.js sample app from Part 5 and the Kubernetes Deployment and Service configurations from Part 3 into a new folder called sample-app-frontend:

Next, update the files in sample-app-frontend as follows:

Build a Docker image for the frontend app using the dockerize command:

This will create a new Docker image called "sample-app-frontend" at version 0.0.1. To deploy this Docker image into a Kubernetes cluster, use kubectl apply:

If you run kubectl get services, you should now see the Services for both the backend and the frontend:

Notice how EXTERNAL-IP for the frontend is set to localhost and that it’s listening on port 80, so you can test it by going to http://localhost. If you open this URL in a web browser, you should see the HTML rendered, as shown in Figure 55:

Congrats, you’re now running two microservices in Kubernetes that are talking to each other! A separate team could own each service, developing, deploying, and scaling the service completely independently.

When you’re done testing, you may want to run kubectl delete on each of the Deployments and Services to undeploy them from your Kubernetes cluster. You should also commit your changes to Git, as you will continue to iterate on this code in subsequent blog posts.