Part 2. How to Manage Your Infrastructure as Code

As an example, let’s create a Bash script that automates all the manual steps you did in Part 1 to deploy a simple Node.js app in AWS. Head into the fundamentals-of-devops folder you created in Part 1 to work through the examples in this blog post series, and create a new subfolder for this part and the Bash script:

Copy the exact same user data script from Part 1 into a file called user-data.sh within the bash folder:

Next, create a Bash script called deploy-ec2-instance.sh, with the contents shown in Example 4:

If you’re not an expert in Bash syntax, all you have to know about this script is that it uses the AWS Command Line Interface (CLI) to do the following:

If you want to try the script out, you’ll first need to give the script execute permissions:

Next, authenticate to AWS as described in Authenticating to AWS on the command line, and run the script as follows:

After the script finishes, give the EC2 instance a minute or two to boot up, and then try opening up http://<Public IP> in your web browser, where <Public IP> is the IP address the script outputs at the very end. You should see:

Congrats, you are now managing your infrastructure as code! Well, sort of. This script, and most ad hoc scripts, have quite a few drawbacks in terms of using them to manage infrastructure, as discussed in the next section.

When you’re done experimenting with this script, you should manually undeploy the EC2 instance by finding it in the EC2 Console: check the top right corner to make sure you’re in the same region used by the script (us-east-2), then on the Instances page, look for the instance ID the script outputs at the end, click "Instance state," and choose "Terminate instance" in the drop down, as shown in Figure 18. This ensures that your account doesn’t start accumulating any unwanted charges.

Below is a list of criteria, which I’ll refer to as the IaC category criteria in this blog post, that you can use to compare different categories of IaC tools. In this section, I’ll flush out how ad hoc scripts stack up according to the IaC category criteria; in later sections, you’ll see how the other IaC categories perform along the same criteria, giving you a consistent way to compare the different options.

Ad hoc scripts have always been, and will always be, a big part of software delivery. They are the glue and duct tape of the DevOps world. However, they are not the best choice as a primary tool for managing infrastructure as code.

If you’re going to be managing all of your infrastructure as code, you should use an IaC tool that is purpose-built for the job, such as one of the ones discussed in the next several sections.

To be able to use configuration management, the first thing you need is a server. If you have an existing server you can use—e.g., a physical server on-prem or a virtual server in the cloud—and you have SSH access to that server, you can skip this section, and go to the next one.

If you don’t have a server you can use, this section will show you how to deploy an EC2 instance using Ansible. Note that deploying and managing servers (hardware) is not really what configuration management tools were designed to do—later in this blog post, you’ll see how provisioning tools are typically a better fit for this task—but for spinning up a single server for learning and testing, Ansible is good enough.

Create a new folder called ansible:

Inside the Ansible folder, create an Ansible playbook called create_ec2_instance_playbook.yml, with the contents shown in Example 5:

Instead of a general-purpose programming language (GPL), such as Bash or Ruby or Python, Ansible uses a domain specific language (DSL) defined on top of YAML. The YAML in the preceding playbook does the following:

To run this Ansible playbook, install Ansible, authenticate to AWS as described in Authenticating to AWS on the command line, and run the following command:

You should get log output for each step that looks something like this (truncated for readability):

Now that you have a server to work with, you can see what configuration management tools are really designed to do: configuring servers to run software.

In order for Ansible to be able to configure your servers, you have to provide an inventory, which is a file that specifies which servers you want configured, and how to connect to them. If you have a set of physical servers on-prem, you can put the IP addresses of those servers in an inventory file, as shown in Example 6:

The preceding file organizes your servers into groups: the webservers group has two servers in it and the dbservers group has three servers. You’ll then be able to write Ansible playbooks that target specific groups.

If you are running servers in the cloud, where servers come and go often, and IP addresses change more frequently, you’re better off using an inventory plugin that can dynamically discover your servers. For example, if you deployed an EC2 instance in AWS in the previous section, you can use the aws_ec2 inventory plugin by creating a file called inventory.aws_ec2.yml with the contents shown in Example 7:

This code does the following:

For each group in your inventory, you can also specify group variables to configure how to connect to the servers in that group. You define these variables in YAML files in the group_vars folder, with the name of the file set to the name of the group you’re configuring. For example, for the EC2 instance in the ch2_instances group, you should create a file in group_vars/ch2_instances.yml with the contents shown in Example 8:

The variables this file defines are:

Alright, with the inventory stuff out of the way, you can now create a playbook to configure your server to run the Node.js sample app. Create a file called configure_sample_app_playbook.yml with the contents shown in Example 9:

This playbook does two things:

An Ansible role is a structured way to organize tasks, templates, files, and other configuration you might want to apply to a server. The standard folder structure for Ansible roles looks like this:

Each folder has a specific purpose: e.g., the tasks folder defines tasks to run on a server; the files folder has files to copy to the server; the templates folder lets you use Jinja templates to dynamically fill in data in files; and so on. Having this standardized structure makes it easier to navigate and understand an Ansible code base.

To create the sample-app role for this playbook, create a roles/sample-app folder in the same directory as configure_sample_app_playbook.yml:

Within roles/sample-app, you should create files and tasks subfolders, which are the only parts of the standardized role folder structure you’ll need for this simple example. Copy the Node.js sample app you saw earlier in in Section 1.2.1 into files/app.js:

Next, create tasks/main.yml with the code shown in Example 10:

This code does the following:

To run this playbook, authenticate to AWS as described in Authenticating to AWS on the command line, and run the following command:

You should get log output for each step, including a recap at the end that looks something like this:

The value on the left, "xxx.us-east-2.compute.amazonaws.com," is a domain name you can use to access the instance. If you open http://xxx.us-east-2.compute.amazonaws.com:8080 (note it’s port 8080 this time, not 80) in your web browser, you should see the familiar "Hello, World!"

Congrats, you’re now using a configuration management tool to manage your infrastructure as code!

When you’re done experimenting with Ansible, you should manually undeploy the EC2 instance by finding it in the EC2 Console: check the top right corner to make sure you’re in the same region used by the Ansible playbook (us-east-2), then on the Instances page, look for the instance ID the script outputs at the end, click "Instance state," and choose "Terminate instance" in the drop down, as shown in Figure 18. This ensures that your account doesn’t start accumulating any unwanted charges.

Here is how configuration management tools stack up using the IaC category criteria from before:

Configuration management tools brought a number of advantages over ad hoc scripts, but they also introduced their own drawbacks. One big drawback is that some configuration management tools have a considerable setup cost: e.g., you may need to set up master servers and ways to connect to all your other servers (agents or SSH). A second big drawback is that most configuration management tools were designed for a mutable infrastructure paradigm: this can be problematic due to configuration drift, where over time, your long-running servers can build up unique histories of changes, so each server is subtly different from the others, which can make it hard to reason about what’s deployed and debug issues.

As cloud and virtualization becomes more and more ubiquitous, it’s becoming more common to use an immutable infrastructure paradigm, where instead of long-running physical servers, you use short-lived virtual servers that you replace every time you do an update. This is inspired by functional programming, where variables are immutable, so after you’ve set a variable to a value, you can never change that variable again, and if you need to update something, you create a new variable. Because variables never change, it’s a lot easier to reason about your code.

The idea behind immutable infrastructure is similar: once you’ve deployed a server, you never make changes to it again. If you need to update something, such as deploying a new version of your code, you deploy a new server. Because servers never change after being deployed, it’s a lot easier to reason about what’s deployed. The typical analogy used here (my apologies to vegetarians and animal lovers), is cattle vs pets: with mutable infrastructure, you treat your servers like pets, giving each one its own unique name, taking care of it, and trying to keep it alive as long as possible; with immutable infrastructure, you treat your servers like cattle, each one more or less indistinguishable from the others, with random or sequential IDs instead of names, and you kill them off and replace them regularly.

While it’s possible to use configuration management tools with immutable infrastructure patterns, it’s not what they were originally designed for, and that led to new approaches, as discussed in the next section.

As an example, let’s take a look at using Packer to create a VM image for AWS called an Amazon Machine Image (AMI) (you’ll see an example of how to create container images using Docker in Part 3). First, create a folder called packer:

Next, copy the Node.js sample app you saw earlier in Section 1.2.1 into the packer folder:

Create a Packer template called sample-app.pkr.hcl, with the contents shown in Example 11:

With Packer, you create templates using the HashiCorp Configuration Language (HCL) in files with a .hcl extension. The preceding Packer template does the following:

So this Packer template is nearly identical to the Bash script and Ansible playbook, except for the following two differences:

If you want to try the Packer template out, install Packer, authenticate to AWS as described in Authenticating to AWS on the command line, and run the following commands:

The first command, packer init, installs any plugins used in this Packer template: Packer can create images for many cloud providers—e.g., AWS, GCP, Azure, etc.—and the code for each of these providers lives not in the Packer binary itself, but in separate plugins that the init command can install. The second command, packer build, kicks off the build process. When the build is done, which typically takes 3-5 minutes, you should see some log output that looks like this:

Congrats, you’re now using a server templating tool to manage your server configuration as code! The ami-xxx value is the ID of the AMI that was created from this template. Save the value somewhere, as later in this post, you’ll see an example of how to deploy this AMI.

So, how do server templating tools stack up using the IaC category criteria from before?

Note that the different server templating tools have slightly different purposes. Packer is typically used to create images that you run directly on top of production servers, such as an AMI that you run in your production AWS account. Vagrant is typically used to create images that you run on your development computers, such as a VirtualBox image that you run on your Mac or Windows laptop. Docker is typically used to create images of individual applications. You can run the Docker images on production or development computers, as long as some other tool has configured that computer with the Docker Engine.

As I mentioned a few times, server templating tools are powerful, but they don’t work by themselves. You need another tool to actually deploy and manage the images you create, such as provisioning tools, which are the focus of the next section.

As an example of using a provisioning tool, lets create an OpenTofu module that can deploy an EC2 instance. You write OpenTofu modules in HCL (the same language you used with Packer), in configuration files with a .tf extension. OpenTofu will find all files with the .tf extension in a folder, so you can name the files whatever you want, but there are some standard conventions, such as putting the main resources in main.tf, input variables in variables.tf, and output variables in outputs.tf.

First, create a new tofu/ec2-instance folder for the module:

Within the tofu/ec2-instance folder, create a file called main.tf, with the contents shown in Example 12:

The code in main.tf does something very similar to the Bash script and Ansible playbook from earlier in the blog post:

In the same folder as main.tf, create a file called variables.tf to define input variables, as shown in Example 13:

As you’ll see shortly, this input variable will allow you to pass in the ID of a custom AMI to run in the EC2 instance: namely, the AMI you built from the Packer template in the previous section.

You should also create a file called user-data.sh, which contains the user data script shown in Example 14:

Note how this user data script is much shorter than the one you saw in the Bash code. That’s because all the dependencies (Node.js) and code (app.js) are already installed in the AMI by Packer. So the only thing this user data script does is start the sample app. This is a more idiomatic way to use user data.

Finally, create a file called outputs.tf with the contents shown in Example 15:

The preceding code defines output variables, which you can use to log and share values between modules. The preceding code defines output variables for the EC2 instance ID, security group ID, and EC2 instance public IP.

If you want to try the OpenTofu code out, install OpenTofu, authenticate to AWS as described in Authenticating to AWS on the command line, and run the following command:

The init command installs any providers used in this Tofu configuration: OpenTofu works with many cloud providers, including AWS, which is used in the preceding example, as well as Azure, GCP, Alibaba Cloud, OCI, and so on. The code for each provider doesn’t live in the tofu binary, but in separate provider binaries, which you download via the init command.

Once init has completed, run the apply command to start the deployment process:

The first thing the apply command will do is prompt you for the ami_id value:

You can paste in the ID of the AMI you built using Packer in the previous section and hit Enter. Alternatively, if you don’t want to be prompted interactively, you can instead use the -var flag when running apply:

You could also use an environment variable: you can set the value for any input variable foo using the environment variable TF_VAR_foo:

Another option is to define the values for your variables in a terraform.tfvars file, as shown in Example 16:

If you define this file and re-run tofu apply, it’ll automatically find the ami_id value, and won’t prompt you for it interactively.

The second thing the apply command will do is show you the execution plan (just plan for short), which will look something like this (truncated for readability):

The plan lets you see what OpenTofu will do before actually making any changes, and prompts you for confirmation before continuing. This is a great way to sanity-check your code before unleashing it onto the world. The plan output is similar to the output of the diff command that is part of Unix, Linux, and git: anything with a plus sign (+) will be created, anything with a minus sign (–) will be deleted, and anything with a tilde sign (~) will be modified in place. Every time you run apply, OpenTofu will show you this execution plan; you can also generate the execution plan without applying any changes by running tofu plan instead of tofu apply.

In the preceding plan output, you can see that OpenTofu is planning on creating an EC2 Instance, security group, and security group rule, which is exactly what you want. Type yes and hit Enter to let OpenTofu proceed. You should see log output that looks like this:

You can see the three output variables from outputs.tf at the end, including the public IP address in public_ip. Wait a minute or two for the EC2 instance to boot up, copy the public_ip, open http://<public_ip>:8080 in your web browser, and you should see the familiar "Hello, World!" yet again.

Congrats, you’re now using a provisioning tool to manage your infrastructure as code!

One of the big advantages of provisioning tools is that they support not just deploying infrastructure, but also updating and destroying it. For example, now that you’ve deployed an EC2 instance using OpenTofu, make a change to the configuration, such as adding a new Test tag with the value "update," as shown in Example 17:

Run the apply command again, and you should see output that looks like this (truncated for readability):

Every time you run OpenTofu, it records information about what infrastructure it created in an OpenTofu state file. OpenTofu manages state using backends; if you don’t specify a backend, the default is to use the local backend, which stores state locally in a terraform.tfstate file in the same folder as the OpenTofu module (you’ll see how to use other backends in Part 5). This file contains a custom JSON format that records a mapping from the OpenTofu resources in your configuration files to the representation of those resources in the real world.

For example, when you run apply the first on the tofu module, OpenTofu records in the state file the IDs of the EC2 instance, security group, security group rules, and any other resources it created. When you run apply again, you can see "Refreshing state" in the log output, which is OpenTofu updating itself on the latest status of the world. As a result, the new plan output that you see is the diff between what’s currently deployed in the real world and what’s in your OpenTofu code. The preceding diff shows that OpenTofu wants to create a single tag called Test, which is exactly what you want, so type yes and hit Enter, and you’ll see OpenTofu perform an update operation, updating the EC2 instance with your new tag.

When you’re done testing, you can run tofu destroy to have OpenTofu undeploy everything it deployed earlier, which should give you log output that looks something like this (log output truncated for readability):

When you run destroy, OpenTofu shows you a destroy plan, which tells you about all the resources it’s about to delete. This gives you one last chance to check that you really want to delete this stuff before you actually do it. It goes without saying that you should rarely, if ever, run destroy in a production environment—there’s no "undo" for the destroy command. If everything looks good, type yes and hit Enter, and in a minute or two, OpenTofu will clean up everything it deployed.

One of OpenTofu’s more powerful features is that the modules are reusable. In a general purpose programming language (e.g., JavaScript, Python, Java), you put reusable code in a function; in OpenTofu, you put reusable code in a module. You can then use that module multiple times to spin up many copies of the same infrastructure, without having to copy/paste the code.

So far, you’ve been using the ec2-instance module as a root module, which is any module on which you run apply directly. However, you can also use it as a resuable module, which is a module meant to be included in other modules (e.g., in other root modules) as a means of code re-use.

Let’s give it a shot. First, create a folder called modules to store your reusable modules:

Next, move the ec2-instance module into the modules folder:

Create a folder called live to store your root modules, as these modules configure your live environments:

Inside the live folder, create a new folder called sample-app, which will house the new root module you’ll use to deploy the sample app:

In the live/sample-app folder, create a main.tf file with the initial contents shown in Example 18:

To use one module from another, all you need is the following:

If you were to run apply on this code, it would use the ec2-instance module code to create a single EC2 instance. But the beauty of code reuse is that you can use the module multiple times, as shown in Example 19:

This code uses two module blocks, so if you run apply on it, it will create two EC2 instances. If you had three module blocks, it would create three EC2 instances; four module blocks would create four EC2 instances; and so on. And, of course, you can mix and match different modules, include modules in other modules, and so on. It’s not unusual for modules to be reused dozens or hundreds of times across a company, so that you put in the work to create a module that meets your company’s needs once, and then use it over and over again.

However, there are two changes you need to make to the ec2-instance module in order for it to work effectively as a reusable module.

The first change is to namespace all the resources created by the ec2-instance module. Currently, it hard-codes all names, such as the name of the security group, to "sample-app-tofu." AWS requires all security group names to be unique, so if you ran apply on these two module blocks, you’d get an error due to the name conflicts. To fix this, introduce a name input variable in modules/ec2-instance/variables.tf, as shown in Example 20:

Next, update the ec2-instance module to use the name input variable everywhere that was hard-coded to "sample-app-tofu," including the aws_security_group resource and the tags in the aws_instance resource, as shown in Example 21:

Now, back in sample-app/main.tf, you can set the name parameter to different values in each of the module blocks, as shown in Example 22:

Now you’ll get two EC2 instances, one with all resources named "sample-app-tofu-1," and the other with all resources named "sample-app-tofu-2."

The second change is to remove the provider block from the ec2-instance module. Having a provider block inside a module isn’t wrong per se, but typically, reusable modules do not declare provider blocks, and instead inherit those from the root module. This allows the provider block to be configured in different ways in different usages of the module: e.g., one usage might configure the provider to use a different region, another usage might configure it to a different AWS account, and so on. All you need to do is to move the provider block from the ec2-instance (reusable) module to the sample-app (root) module, as shown in Example 23:

One last step: create an outputs.tf file in the sample-app folder with the contents shown in Example 24:

The preceding code "proxies" the output variables from the underlying ec2-instance module usages so that you can see those outputs when you run apply on the sample-app module.

OK, you’re finally ready to run this code:

When apply completes, you should have two EC2 instances running, and the output variables should show their IPs and instance IDs. If you wait a minute or two for the instances to boot up, and open http://<IP>:8080 in your browser, where <IP> is the public IP of either instance, you should see the familiar "Hello, World!" text. When you’re done experimenting, run tofu destroy to clean everything up again.

There’s one more trick with OpenTofu modules: the source parameter can be set to not only a local file path, but also to a URL. For example, the blog post series’s sample code repo in GitHub includes an ec2-instance module that is more or less identical to your own ec2-instance module. You can use the module from the series’s sample code repo by updating the source URLs as shown in Example 25:

The preceding code sets the source URL to a GitHub URL. Note the intentional use of two slashes (//): the part to the left of the two slashes specifies the GitHub repo and the part to the right of the two slashes specifies the subfolder within that repo.

Run init on this code one more time:

The init command is responsible for downloading provider code and module code, and you can see in the preceding output that, this time, it downloaded the module code from GitHub.

If you now run apply, you should get the exact same two EC2 instances as before. When you’re done experimenting, run destroy to clean everything up.

You’ve now seen the power of reusable modules. A common pattern at many companies is for the Ops team to define and manage a library of vetted, reusable OpenTofu modules—e.g., one module to deploy servers, another to deploy databases, another to configure networking, and so on—and for the Dev teams to use these modules as a self-service way to deploy and manage the infrastructure they need for their apps.

This blog post series will make use of this pattern in future chapters: instead of writing every line of code from scratch, you’ll be able to use modules directly from this series’s sample code repo to deploy the infrastructure you need for each chapter.

So, how do provisioning tools stack up using the IaC category criteria from before?

Provisioning tools should be your go-to option for managing infrastructure. In fact these days, many provisioning tools can be used to not only manage traditional infrastructure (e.g., servers), but many other aspects of software delivery as well. For example, you can use OpenTofu to manage your version control system (e.g., using the GitHub provider), metrics and dashboards (e.g., using the Grafana provider), and your on-call rotation (e.g., using the PagerDuty provider), tying them all together with code.

Although I’ve been comparing IaC tools this entire blog post, the reality is that you’ll probably need to use multiple IaC tools together, as discussed in the next section.

Example: OpenTofu and Ansible. You use OpenTofu to deploy all the underlying infrastructure, including the network topology (i.e., virtual private clouds [VPCs], subnets, route tables), data stores (e.g., MySQL, Redis), load balancers, and servers. You then use Ansible to deploy your apps on top of those servers, as depicted in Figure 21.

This is an easy approach to get started with and there are many ways to get Ansible and OpenTofu to work together (e.g., OpenTofu adds tags to your servers, and Ansible uses an inventory plugin to automatically discover servers with those tags). The major downside is that using Ansible typically means mutable infrastructure, rather than immutable, so as your codebase, infrastructure, and team grow, maintenance and debugging can become more difficult.

Example: OpenTofu and Packer. You use Packer to package your apps as VM images. You then use OpenTofu to deploy servers with these VM images and the rest of your infrastructure, including the network topology (i.e., VPCs, subnets, route tables), data stores (e.g., MySQL, Redis), and load balancers, as illustrated in Figure 22.

This is also an easy approach to get started with: in fact, you already had a chance to try this combination out earlier in this post. Moreover, this is an immutable infrastructure approach, which will make maintenance easier. However, there are two major drawbacks. First, VMs can take a long time to build and deploy, which will slow down your iteration speed. Second, the deployment strategies you can implement with provisioning tools are typically limited (e.g., you can’t implement blue-green deployment natively in OpenTofu), so you either end up writing lots of complicated deployment scripts or you turn to orchestration tools.

Orchestration tools, such as Kubernetes, Nomad, and OpenShift, help you deploy and manage apps on top of your infrastructure. You’ll do a deep-dive on orchestration in Part 3.

Example: OpenTofu, Packer, Docker, and Kubernetes. You use Packer to create a VM image that has Docker and Kubernetes agents installed. You then use OpenTofu to deploy a cluster of servers, each of which runs this VM image, and the rest of your infrastructure, including the network topology (i.e., VPCs, subnets, route tables), data stores (e.g., MySQL, Redis), and load balancers. Finally, when the cluster of servers boots up, it forms a Kubernetes cluster that you use to run and manage your Dockerized applications, as shown in Figure 23.

The advantage of this approach is that Docker images build fairly quickly, you can run and test them on your local computer, and you can take advantage of all of the built-in functionality of Kubernetes, including various deployment strategies, auto healing, auto scaling, and so on. The drawback is the added complexity, both in terms of extra infrastructure to run (Kubernetes clusters are difficult and expensive to deploy and operate, though most major cloud providers now provide managed Kubernetes services, which can offload some of this work) and in terms of several extra layers of abstraction (Kubernetes, Docker, Packer) to learn, manage, and debug.