Part 5. How to Set Up Continuous Integration (CI) and Continuous Delivery (CD)

Branch by abstraction is a technique that allows you to make large-scale changes to your code incrementally, across many commits, without ever risking breaking the build or releasing unfinished work to users. For example, let’s say you have hundreds of modules in your codebase that use Library X, as shown in Figure 42:

You want to replace Library X with Library Y, but this will require updating hundreds of modules, which could take months. If you do this work in a feature branch, by the time you merge it back, there’s a good chance you’ll have merge conflicts with many of the updated modules, and it’s possible new usages will have shown up in the meantime, so you’d have even more work to do.

Instead of a feature branch, the idea with branch by abstraction is to keep working on main, but to introduce a new abstraction into the codebase. What type of abstraction you use depends on your programming language: it might be an interface, a protocol, a class, etc. The important thing is that (a) the abstraction initially uses Library X under the hood, so there is no change in behavior and (b) it creates a layer of indirection between your modules and Library X, as shown in Figure 43:

You can update your modules to use the abstraction incrementally, across many commits to main. There’s no hurry or risk of breakage, as under the hood, the abstraction is still using Library X. Eventually, all modules should be using the abstraction; you could even add a test that fails if anyone tries to use Library X directly.

At this point, you can start updating the abstraction to use Library Y instead of Library X for some use cases, as shown in Figure 44:

You can roll out this change incrementally, across many commits to main, integrating your work regularly to minimize merge conflicts. You could also update your abstraction code to ensure that any new usages of the abstraction get Library Y under the hood by default. Eventually, when you’re done updating all module usages, you can remove Library X entirely, as shown in Figure 45:

Branch by abstraction is a great technique for doing large-scale refactors. But what if you need to introduce totally new functionality? If that functionality takes weeks or months to implement, how can you merge it regularly into main without accidentally releasing unfinished features to users? This is when you turn to the second approach, feature toggles, as described next.

The idea with feature toggles (AKA feature flags) is to wrap new functionality in conditionals that let you turn (toggle) those features on and off dynamically. For example, imagine that you wanted to take the Node.js sample app you’ve been using throughout this blog post series, and to update it to return a proper home page that is a little more interesting than the "Hello, World!" text. However, it’s going to take you several months to implement this new home page. The idea with a feature toggle is to add a conditional to your code as shown in Example 85 (you don’t need to actually make these code changes; this is just for demonstration purposes):

Here’s what this code does:

So what does the lookupFeatureToggle function do? Typically, this function will check if the feature toggle is enabled by querying a dedicated feature toggle service, which is a service that can do the following:

You could build your own feature toggle service around a database, or deploy an open source feature toggle service such as growthbook, Flagsmith, flagr, or OpenFeature, or you could use a managed feature toggle service such as Split, LaunchDarkly, ConfigCat, or Statsig.

It might not be obvious, but the humble if-statement, combined with a feature toggle check, unlocks a superpower: you can now commit and regularly merge code, even before it’s done. This is because of the following key property of feature toggles:

If you wrap new features in a feature toggle check, as long as the code is syntactically valid (which you can validate with simple automated tests), you can merge your new feature into main long before that feature is done, as by default, the new feature is off, so it will have no impact on other developers or your users. This is what allows you to develop large new features while still practicing continuous integration.

What’s even more surprising is that this is only one of the superpowers you get with feature toggles; you’ll see a number of others later in this blog post.

One way to allow automated tools to authenticate is to create a dedicated machine user, which is a user account that is only used for automation (not by any human user). You create the user, generate credentials for them (e.g., access keys), and copy those credentials into whatever tool you’re using (e.g., into GitHub).

Machine users have a number of advantages: they never depart your company; you can assign them just the permissions they need; no human ever logs in as a machine user, so they only show up in audit logs when used in your automations; and you can share access to a single machine user account across your team by using a secrets management tool (a topic you’ll learn more about in Part 8).

However, machine users also have two major drawbacks. First, you have to copy their credentials around manually, which is tedious and error-prone. Second, the credentials you’re copying are typically long-lived credentials, which don’t expire for a long time (if ever), so if these credentials are ever leaked, there is a potentially indefinite window of time during which they could be exploited.

With some tools, machine users are the best you can do, but these days, some systems support automatically-provisioned credentials, as described in the next section.

A second way to allow automated tools to authenticate is to use automatically-provisioned credentials, which are credentials the system can generate automatically, without any need for you to manually create machine users or copy/paste credentials. This requires that the system you’re authenticating from (e.g. a CI server) and the system you’re authenticating to (e.g., AWS) have an integration between them that supports automatically-provisioned credentials.

You’ve actually seen one form of automatically-provisioned credentials already earlier in this blog post series: IAM roles. Some of the resources you’ve deployed in AWS, such as the EKS cluster in Part 3, used IAM roles to authenticate and make API calls within your AWS account (e.g., to deploy EC2 instances as EKS worker nodes). You didn’t have to manually manage credentials to make this work: instead, under the hood, AWS automatically provisioned credentials that the EKS cluster could use.

With IAM roles, the thing you’re authenticating from and the thing you’re authenticating to are both AWS, but there are also systems that support automatically-provisioned credentials across different companies and services. One of the most common is Open ID Connect (OIDC), which is an open protocol for authentication. Not all services support OIDC, but in the cases where it is supported, it’s usually a more secure choice than machine user credentials, as OIDC gives you not only automatically-provisioned credentials (so no manual copy/paste), but also short-lived credentials that expire after a configurable period of time (e.g., one hour).

One place where OIDC is supported is between AWS and GitHub. To set up OIDC with AWS and GitHub, you configure your AWS account to trust an identity provider (IdP), such as GitHub, whose identity AWS can verify cryptographically, and then you can grant that provider permissions to assume specific IAM roles, subject to certain conditions: e.g., you can only use this IAM role from certain repos or branches.

Once you’ve set that up, Figure 49 shows the workflow for using OIDC to authenticate from GitHub to AWS:

Here are the steps in the workflow:

Since OIDC is a more secure option than machine user credentials, let’s try it out in the next section.

The following is a list of the most common deployment strategies you can use to deploy your apps:

Now that you’ve seen the core deployment strategies, how do they compare?

You can compare the core deployment strategies across the following dimensions:

Table 10 shows how the deployment strategies compare along these dimensions:

For deploying stateless apps, blue-green deployments are the gold standard. If blue-green deployments aren’t supported, then rolling deployments without replacement are the next best bet. For deploying stateful apps, where each replica has a unique set of data on its local hard-drive that needs to be persisted across the deployment (e.g., distributed data stores such as Consul, Elasticsearch, and ZooKeeper), rolling deployments with replacement are the gold standard. As for downtime deployments, the only time I’d use them is if you need to do a data migration, as taking a brief downtime during a deployment can sometimes make a data migration 10x cheaper and less error-prone (compared to trying to do it with no downtime, which requires complicated processes such as double writes).

All the deployment strategies you’ve seen so far can be use standalone. Let’s now turn our attention to strategies that are meant to be combined with other strategies.

The following is a list of the most common strategies you can combine with the core deployment strategies to make your deployments safer and more effective (note, these are not mutually exclusive; you can, and often do, use multiple add-ons together):

Now that you’ve seen all the add-on deployment strategies, how do they compare?

Canary deployments offer one way to reduce the blast radius if a deployment goes wrong. One of the reasons the companies you read about in Section 1.1 have a lower change failure rate is that they use canary deployments to catch issues early, before those issues escalate into major outages.

Feature toggles reduce the blast radius even further. In fact, feature toggles are such a powerful tool, that I often describe them as a "superpower" for DevOps. The following are just a few of the remarkable powers you get from using feature toggles with deployments:

The ability to separate deployment from release, carefully ramp new features, and quickly shut off features that are causing issues is such a huge advantage in agility, that once you get past a certain scale as a company, you should consider wrapping all new features in feature toggles. To be fair, feature toggles have some drawbacks, too. To use feature toggles, you have to run and maintain an extra feature toggle service, or pay for one from a 3rd party. Moreover, as you add more if-statements with feature toggle lookups, you get more forks in your code. This makes the code harder to maintain and test. If you’re going to use feature toggles, you’ll need to create the discipline (and automation) to ensure that you systematically remove if-statements for feature toggles that are unlikely to change again (e.g., feature toggles more than 1 year old). But this is a small price to pay for the superpowers you gain; the first time you’re paged at 3 AM, and you’re able to resolve the issue in minutes by shutting off a feature toggle, and going back to sleep, you’ll see what I mean.

Promotion workflows are also worth adopting at most companies, as they give you multiple chances to catch errors before those errors make it to prod. Moreover, whereas just about all the deployment strategies you’ve seen so far only work for deploying application code (i.e., apps written in Java, Ruby, Python, JavaScript, etc.), promotion workflows can also be used to deploy infrastructure code (i.e., OpenTofu, Pulumi, CloudFormation). With infrastructure code, deployments are fairly binary: either you make an infrastructure change, or you don’t; either you create (or delete!) that database, or you don’t. There’s no gradual rollout, no feature toggles, no canaries, etc. That makes infrastructure deployments harder and riskier. To mitigate these risks, you typically rely on promotion workflows, so you can try out the deployments in pre-production environments, and you also review execution plans (e.g., tofu plan output) before deploying; you’ll see an example of this later in this blog post.

Now that you’ve seen all the basic deployment strategies, let’s turn our attention to how to implement these strategies as code in the form of deployment pipelines.

To avoid introducing too many new tools, let’s stick to using GitHub Actions as the deployment server and GitHub Actions YAML workflows as the primary language for defining the pipeline. The goal is to implement the following pipeline for the lambda-sample module:

This type of pipeline, where you mostly drive actions through operations in Git (commits, branches, pull requests) is often referred to as a GitOps pipeline. You already implemented most of this GitOps pipeline as part of setting up automated tests in the CI section. The only items remaining are the following:

To add these items, you need to do the following steps:

The following three sections will cover each of these steps.

In Part 2, you learned that, by default, OpenTofu uses the local backend to store OpenTofu state in .tfstate files on your local hard drive. This works fine when you’re learning and working alone, but if you want to use OpenTofu as a team, you need a way to share these state files. You might be tempted to use version control, but that’s not a good idea for the following reasons:

This is why in Part 4, you added .tfstate files to .gitignore, so as not to accidentally check them in. Instead of using version control, the best way to share state files in a team is to use a supported remote backend, such as Amazon Storage Service (S3), Azure Storage, Google Cloud Storage (GCS), Consul, or Postgres. Remote backends solve the three issues just listed:

If you’re using OpenTofu with AWS, Amazon’s managed file store, S3, is typically your best bet as a remote backend for the following reasons:

To enable Amazon S3 as a remote backend, you must first create an S3 bucket and DynamoDB table. The blog post series’s sample code repo includes a module called state-bucket in the ch5/tofu/modules/state-bucket folder which can create an S3 bucket to store OpenTofu state, including:

The state-bucket module can also create a DynamoDB table. DynamoDB is Amazon’s distributed key-value store, and as it supports strongly consistent reads and conditional writes, OpenTofu can use it for locking (you’ll learn more about key-value stores in Part 9). Moreover, it’s a managed service, so you don’t have any infrastructure to deploy, and it’s inexpensive, with most OpenTofu usage easily fitting into the AWS free tier.

To use the state-bucket module, first check out the main branch of your own repo, and make sure you have the latest code:

Next, create a new folder called tofu-state to use as a root module:

Within the tofu-state folder, create a main.tf file with the contents shown in Example 95:

This code sets just one parameter, name, which will be used as the name of the S3 bucket and DynamoDB table. Note that S3 bucket names must be globally unique among all AWS customers. Therefore, you must change the name parameter from "fundamentals-of-devops-tofu-state" (which I already created) to your own name. Make sure to remember this name and take note of what AWS region you’re using, as you’ll need both pieces of information again a little later.

To create the S3 bucket and DynamoDB table, run init and apply as usual:

Once apply is done, you can start using the S3 bucket and DynamoDB table for state storage. To do that, you need to update your OpenTofu modules with a backend configuration. As a first step, add a backend.tf file to the tofu-state module with the contents shown in Example 96:

Here’s what this code does:

Run tofu init one more time, and you should see something like this:

OpenTofu will automatically detect that you already have a state file locally and prompt you to copy it to the new S3 backend. If you type yes and hit ENTER, you should see the following:

With this backend enabled, OpenTofu will automatically pull the latest state from this S3 bucket before running a command and automatically push the latest state to the S3 bucket after running a command, and it’ll use DynamoDB for locking.

You should make the same change in the lambda-sample module as well, adding the backend.tf file shown in Example 97:

This is identical to the backend.tf in the tofu-state module, but note three things:

To finish up the remote state setup, do the following two steps:

Now that you have a remote backend set up, you can move onto the next step, which is setting up IAM roles that will allow you to do deployments from GitHub Actions.

Earlier in this blog post, you configured an OIDC provider to give GitHub Actions access to your AWS account for running automated tests. Now you need a way to give GitHub Actions access to your AWS account for deployments. Normally, you would deploy to a totally separate environment (separate AWS account) from where you run automated tests, so you’d need to configure a new OIDC provider in your deployment environment. However, to keep things simple in this post, let’s use the same AWS account, and therefore the same OIDC provider, for both deployment and testing (you’ll learn how to set up additional environments in Part 6). Note that you still need to create new IAM roles for the following reasons:

Open up main.tf in the ci-cd-permissions module and add the code shown in Example 98 to enable creating IAM roles for both plan and apply:

This code does the following:

Next, update outputs.tf with two new output variables that contain the ARNs of the two new IAM roles, as shown in Example 99:

Run apply to create the new IAM roles and take note of the lambda_deploy_plan_role_arn and lambda_deploy_apply_role_arn outputs; you’ll need them shortly.

Commit your changes to the ci-cd-permissions module and push them to main. You’re now finally ready to define the deployment pipeline itself.

With all the prerequisites out of the way, you can finally implement a deployment pipeline for the lambda-sample module that will do the following:

Let’s first create a workflow for the plan portion. Create a new file called .github/workflows/tofu-plan.yml with the contents shown in Example 100:

This workflow has a few things you haven’t seen before:

Next, create a workflow for the apply portion in a new file called .github/workflows/tofu-apply.yml, with the contents shown in Example 101:

This workflow is similar the one for plan, but with a few key differences:

Commit these new workflow files directly to the main branch and then push them to GitHub:

Now, let’s give this deployment pipeline a shot. First, create a new branch called deployment-pipeline-test:

Make a change to the lambda-sample module, such as changing the text it returns, as shown in Example 102:

Make sure to also make the corresponding update to the assertion in the automated test in deploy.tftest.hcl, as shown in Example 103:

Commit both of these changes, push them to the deployment-pipeline-test branch, open a pull request, and you should see a page that looks like Figure 58:

You should see three things running in your pipeline:

When everything has finished, the PR should automatically be updated with a comment that shows the plan output, as shown in Figure 59:

Now you can review the code changes, test results, and plan output, and if everything looks good, merge the PR. This will kick off the apply workflow, and after a minute or two, it should post a comment with the apply output, as shown in Figure 60:

Congrats, you now have a basic deployment pipeline in place for your lambda-sample module! It runs tests, it runs plan, and it runs apply.

Now that you’ve seen the basics of deployment pipelines, and an example of how to implement one, let’s go through some recommendations for how to make your pipelines more effective.

Every deployment pipeline includes steps that must be done by humans, such as writing code, reviewing code, and perhaps manual testing and verification. All the other steps should be completely automated. Remember, it’s only continuous delivery if it is fast, reliable, and sustainable. These three things are precisely where computers excel over humans. Whereas humans are slow at performing a bunch of steps, computers can run automated processes extremely quickly; whereas humans make mistakes all the time, automated processes are predictable and repeatable; and whereas humans can get frustrated from repeating the same steps over and over again (especially if those steps sometimes cause an outage), computers never get tired or stressed.

Not only should most of your deployment pipeline be automated, but in most cases, all of that automation should only run on a dedicated deployment server and not any developer’s computer. There are three reasons for this:

To be able to do automated deployments from a server, you have to give the server access to sensitive permissions, such as AWS credentials. In fact, to deploy arbitrary infrastructure changes—e.g., to be able to run tofu apply on arbitrary OpenTofu modules—you need arbitrary permissions, which is just a fun way of saying "admin permissions." That means deployment servers (a) provide access to powerful, sensitive permissions, (b) are accessible to every developer in your company, and (c) are designed to execute arbitrary code. That is a terrifying combination from a security perspective, and it’s why deployment servers are particularly tempting targets for hackers.

Here are a few things you can do to protect your deployment server:

Creating reliable deployment pipelines is a lot of work. This shouldn’t be too surprising, as what you’re effectively trying to do is to capture your company’s processes, rules, and culture in the form of Bash scripts and YAML workflow files. That isn’t easy. But it’s worth it. Think of it this way: your infrastructure code (which you learned about in Part 2) and your CI / CD pipeline (which you learned about in this blog post) are essentially your company’s custom API for shipping software. Get the API right, and as you saw in Part 1, you can accelerate your company by a factor of 10x, 100x, or more.