A comprehensive guide to managing secrets in your Terraform code

One of the most common questions we get about using Terraform to manage infrastructure as code is how to handle secrets such as passwords, API keys, and other sensitive data. For example, here’s a snippet of Terraform code that can be used to deploy MySQL using Amazon RDS:

Copy
resource "aws_db_instance" "example" {
engine               = "mysql"
engine_version       = "5.7"
instance_class       = "db.t2.micro"
name                 = "example"

Copy
# How should you manage the credentials for the master user?
username = "???"
password = "???"
}

Notice how Terraform requires you to set two secrets, username and password, which are the credentials for the master user of the database. In this blog post, I’ll go over the most common techniques you can use to safely and securely manage such secrets:

1. Pre-requisite #1: Don’t Store Secrets in Plain Text
2. Pre-requisite #2: Keep Your Terraform State Secure
3. Technique #1: Environment Variables
4. Technique #2: Encrypted Files (e.g., KMS, PGP, SOPS)
5. Technique #3: Secret Stores (e.g., Vault, AWS Secrets manager)

Pre-requisite #1: Don’t Store Secrets in Plain Text

The first rule of secrets management is:

Do not store secrets in plain text.

The second rule of secrets management is:

DO NOT STORE SECRETS IN PLAIN TEXT.

Seriously, don’t do it. For example, do NOT hard-code your database credentials directly in your Terraform code and check it into version control:

Copy
resource "aws_db_instance" "example" {
engine               = "mysql"
engine_version       = "5.7"
instance_class       = "db.t2.micro"
name                 = "example"

Copy
# DO NOT DO THIS!!!
username = "admin"
password = "password"
# DO NOT DO THIS!!!
}

Storing secrets in plain text in version control is a BAD IDEA. Here are just a few of the reasons why:

1. Anyone who has access to the version control system has access to that secret. In the example above, every single developer at your company has access to the master credentials for your database.
2. Every computer that has access to the version control system keeps a copy of that secret. Every single computer that has ever checked out that repo may still have a copy of that secret on its local hard drive. That includes the computer of every developer on your team, every computer involved in CI (e.g., Jenkins, CircleCi, GitLab, etc.), every computer involved in version control (e.g., GitHub, GitLab, BitBucket), every computer involved in deployment (e.g., all your pre-prod and prod environments), every computer involved in backup (e.g., CrashPlan, Time Machine, etc.), and so on.
3. Every piece of software you run has access to that secret. Because the secrets are sitting in plain text on so many hard-drives, every single piece of software running on any of those computers can potentially read that secret.
4. No way to audit or revoke access to that secret. When secrets are sitting on hundreds of hard drives in plain text, you have no way to know who accessed them (there’s no audit log) and no way to revoke access.

In short, if you store secrets in plain text, you are giving malicious actors (e.g., hackers, competitors, disgruntled former employees) countless ways to access your company’s most sensitive data—e.g., by compromising the version control system, or any of the computers you use, or any piece of software on any of those computers, etc—and you’ll have no idea if you were compromised or have any easy way to fix things if you were.

Therefore, I strongly recommend that you always store secrets in an encrypted format—and this applies to all secrets, and not just those used with Terraform! Later in this post, I’ll discuss several different techniques for encrypting and decrypting such secrets.

Pre-requisite #2: Keep Your Terraform State Secure

Hopefully, the previous section has convinced you to not store your secrets in plain text, and the subsequent sections will show you some techniques for encrypting your secrets. However, no matter which technique you use to encrypt the secrets on your end, there is still one place where they will end up in plain text: Terraform state.

Every time you deploy infrastructure with Terraform, it stores lots of data about that infrastructure, including all the parameters you passed in, in a state file. By default, this is a terraform.tfstate file that is automatically generated in the folder where you ran terraform apply. So even if you use one of the techniques mentioned later to safely pass in your secrets, such as the credentials for a database:

Copy
resource "aws_db_instance" "example" {
engine               = "mysql"
engine_version       = "5.7"
instance_class       = "db.t2.micro"
name                 = "example"

Copy
# Let's assume you found safe way to pass these in
username = <some secure mechanism>
password = <some secure mechanism>
}

These secrets will still end up in terraform.tfstate in plain text! This has been an open issue for more than 6 years now, with no clear plans for a first-class solution. There are some workarounds out there that can scrub secrets from your state files, but these are brittle and likely to break with each new Terraform release, so I don’t recommend them.

For the time being, no matter which of the techniques discussed below you end up using to manage secrets, you must also:

1. Store Terraform state in a backend that supports encryption. Instead of storing your state in a local terraform.tfstate file, Terraform natively supports a variety of backends, such as S3, GCS, and Azure Blob Storage. Many of these backends support encryption, so that instead of your state files being in plain text, they will always be encrypted, both in transit (e.g., via TLS) and on disk (e.g., via AES-256). Most backends also support collaboration features (e.g., automatically pushing and pulling state; locking), so using a backend is a must-have both from a security and teamwork perspective. See How to Manage Terraform State for more info.
2. Strictly control who can access your Terraform backend. Since Terraform state files may contain secrets, you’ll want to carefully control who has access to the backend you’re using to store your state files. For example, if you’re using S3 as a backend, you’ll want to configure an IAM policy that solely grants access to the S3 bucket for production to a small handful of trusted devs (or perhaps solely just the CI server you use to deploy to prod).

And now, without further ado, let’s discuss the various techniques available to you for managing secrets with Terraform.

Technique #1: Environment Variables

This first technique keeps plain text secrets out of your code by taking advantage of Terraform’s native support for reading environment variables.

Overview

To use this technique, declare variables for the secrets you wish to pass in:

Copy
variable "username" {
description = "The username for the DB master user"
type        = string
}

Copy
variable "password" {
description = "The password for the DB master user"
type        = string
}

Update, December 3, 2020: Terraform 0.14 has added the ability to mark variables as sensitive, which helps keep them out of your logs, so you should add sensitive = true to both variables above!

Next, pass the variables to the Terraform resources that need those secrets:

Copy
resource "aws_db_instance" "example" {
engine               = "mysql"
engine_version       = "5.7"
instance_class       = "db.t2.micro"
name                 = "example"

Copy
# Set the secrets from variables
username             = var.username
password             = var.password
}

You can now pass in a value for each variable foo by setting an environment variable called TF_VAR_foo. For example, here’s how you could set username and password via environment on Linux, Unix, or Mac, and run terraform apply to deploy the database:

Copy
# Set secrets via environment variables
export TF_VAR_username=(the username)
export TF_VAR_password=(the password)

Copy
# When you run Terraform, it'll pick up the secrets automatically
terraform apply

(Pro tip: if you have the HISTCONTROL environment variable set correctly in a Bash terminal, then any command with a leading space will not be stored in Bash history. Use this when setting environment variables with secrets to avoid having those secrets stored on disk.)

This technique helps you avoid storing secrets in plain text in your code, but it leaves the question of how to actually securely store and manage the secrets unanswered. So in a sense, this technique just kicks the can down the road, whereas the other techniques described later in this blog post are more prescriptive.

That said, so as not to leave you entirely hanging, if you do go with environment variables, the most common solution for storing and managing secrets is to use a password manager such as:

• 1Password: SaaS tool that offers (a) desktop and mobile apps for all platforms and (b) cloud sync so you can access your secrets across all devices and collaborate with a team.
• LastPass: SaaS tool that offers (a) desktop and mobile apps for all platforms and (b) cloud sync so you can access your secrets across all devices and collaborate with a team.
• pass: Open source tool that follows the Unix philosophy: it’s a CLI tool, it does input and output via text streams (i.e., stdin, stdout), and under the hood, it stores everything as files, with each secret in its own PGP-encrypted file (you can check these encrypted files into version for team collaboration).

These tools solve the “kick the can down the road” problem by relying on human memory: that is, your ability to memorize a password that gives you access to the password manager.

An example using pass

Let’s go through a quick example using pass. First, you’ll need to store your secrets by using the pass insert command:

Copy
$ pass insert db_username
Enter password for db_username: admin

Copy
$ pass insert db_password
Enter password for db_password: password

You can read a secret out to stdout by running pass <secret>:

Copy
$ pass db_username
admin

You can use this functionality in a subshell to set your secrets as environment variables and then call terraform apply:

Copy
# Read secrets from pass and set as environment variables
export TF_VAR_username=$(pass db_username)
export TF_VAR_password=$(pass db_password)

Copy
# When you run Terraform, it'll pick up the secrets automatically
terraform apply

Advantages of this technique

• Keep plain text secrets out of your code and version control system.
• Easy solution to get started with.
• Integrates with most other secrets management solutions: e.g., if your company already has a way to manage secrets, you can typically find a way to make it work with environment variables.
• Test friendly: when writing tests for your Terraform code (e.g., with Terratest), there are no dependencies to configure (e.g., secret stores), as you can easily set the environment variables to mock values.

Drawbacks to this technique

• Not everything is defined in the Terraform code itself. This makes understanding and maintaining the code harder.
• Everyone using your code has to know to take extra steps to either manually set these environment variables or run a wrapper script.
• No guarantees or opinions around security. Since all the secrets management happens outside of Terraform, the code doesn’t enforce any security properties, and it’s possible someone is still managing the secrets in an insecure way (e.g., storing them in plain text).

Technique #2: Encrypted Files (e.g., KMS, PGP, SOPS)

The second technique relies on encrypting the secrets, storing the cipher text in a file, and checking that file into version control.

Overview

To encrypt some data, such as some secrets in a file, you need an encryption key. This key is itself a secret! This creates a bit of a conundrum: how do you securely store that key? You can’t check the key into version control as plain text, as then there’s no point of encrypting anything with it. You could encrypt the key with another key, but then you then have to figure out where to store that second key. So you’re back to the “kick the can down the road problem,” as you still have to find a secure way to store your encryption key.

The most common solution to this conundrum is to store the key in a key service provided by your cloud provider, such as:

• AWS KMS
• GCP KMS
• Azure Key Vault

These key services solve the “kick the can down the road” problem by relying on human memory: in this case, your ability to memorize a password that gives you access to your cloud provider (or perhaps you store that password in a password manager and memorize the password to that instead).

An example using AWS KMS

Here’s an example of how you can use a key managed by AWS KMS to encrypt secrets. First, create a file called db-creds.yml with your secrets:

Copy
username: admin
password: password

Note: do NOT check this file into version control!

Next, encrypt this file by using the aws kms encrypt command and writing the resulting cipher text to db-creds.yml.encrypted:

Copy
aws kms encrypt \
--key-id <YOUR KMS KEY> \
--region <AWS REGION> \
--plaintext fileb://db-creds.yml \
--output text \
--query CiphertextBlob \
> db-creds.yml.encrypted

You can now safely check db-creds.yml.encrypted into version control.

To decrypt the secrets from this file in your Terraform code, you can use the aws_kms_secrets data source (for GCP KMS or Azure Key Vault, you’d instead use the google_kms_secret or azurerm_key_vault_secret data sources, respectively):

Copy
data "aws_kms_secrets" "creds" {
secret {
name    = "db"
payload = file("${path.module}/db-creds.yml.encrypted")
}
}

The code above will read db-creds.yml.encrypted from disk and, assuming you have permissions to access the corresponding key in KMS, decrypt the contents to get back the original YAML. You can parse the YAML as follows:

Copy
locals {
db_creds = yamldecode(data.aws_kms_secrets.creds.plaintext["db"])
}

And now you can read the username and password from that YAML and pass them to the aws_db_instance resource:

Copy
resource "aws_db_instance" "example" {
engine               = "mysql"
engine_version       = "5.7"
instance_class       = "db.t2.micro"
name                 = "example"

Copy
# Set the secrets from the encrypted file
username = local.db_creds.username
password = local.db_creds.password
}

One gotcha with this approach is that working with encrypted files is awkward. To make a change, you have to locally decrypt the file with a long aws kms decrypt command, make some edits, re-encrypt the file with another long aws kms encrypt command, and the whole time, be extremely careful to not accidentally check the plain text data into version control or leave it sitting behind forever on your computer. This is a tedious and error prone process—unless you use a tool like sops.

An example using AWS KMS with sops and Terragrunt

sops is an open source tool designed to make it easier to edit and work with files that are encrypted via AWS KMS, GCP KMS, Azure Key Vault, or PGP. sops can automatically decrypt a file when you open it in your text editor, so you can edit the file in plain text, and when you go to save those files, it automatically encrypts the contents again. This removes the need to run long aws kms commands to encrypt or decrypt data or worry about accidentally checking plain text secrets into version control. Here’s a .gif that shows sops in action:

Terraform does not yet have native support for decrypting files in the format used by sops. One solution is to install and use the custom provider for sops, terraform-provider-sops. Another option, which I’ll demonstrate here, is to use Terragrunt, which has native sops support built in. Terragrunt is a thin wrapper for Terraform that helps you keep your Terraform code DRY and maintainable (check out the Quick Start guide for an overview).

Let’s say you used sops to create an encrypted YAML file called db-creds.yml, as shown in the .gif above. Now, in your terragrunt.hcl config, you can use the sops_decrypt_file function built into Terragrunt to decrypt that file and yamldecode to parse it as YAML:

Copy
locals {
db_creds = yamldecode(sops_decrypt_file(("db-creds.yml")))
}

Next, you can pass username and password as inputs to your Terraform code:

Copy
inputs = {
username = local.db_creds.username
password = local.db_creds.password
}

Your Terraform code, in turn, can read these inputs via variables:

Copy
variable "username" {
description = "The username for the DB master user"
type        = string
}

Copy
variable "password" {
description = "The password for the DB master user"
type        = string
}

Update, December 3, 2020: Terraform 0.14 has added the ability to mark variables as sensitive, which helps keep them out of your logs, so you should add sensitive = true to both variables above!

And pass those variables through to aws_db_instance:

Copy
resource "aws_db_instance" "example" {
engine               = "mysql"
engine_version       = "5.7"
instance_class       = "db.t2.micro"
name                 = "example"

Copy
# Set the secrets from variables
username = var.username
password = var.password
}

Advantages of this technique

• Keep plain text secrets out of your code and version control system.
• Your secrets are stored in an encrypted format in version control, so they are versioned, packaged, and tested with the rest of your code. This helps reduce configuration errors, such as adding a new secret in one environment (e.g., staging) but forgetting to add it in another environment (e.g., production).
• Works with a variety of different encryption options: AWS KMS, GCP KMS, PGP, etc.
• Everything is defined in the code. There are no extra manual steps or wrapper scripts required (although sops integration does require either a custom provider or wrapper tool like Terragrunt).

Drawbacks to this technique

• Encrypting the data requires extra work. You either have to run lots of commands (e.g., aws kms encrypt) or use an external tool such as sops. There’s a learning curve to using these tools correctly and securely.
• The secrets are now encrypted, but as they are still stored in version control, rotating and revoking secrets is hard. If anyone ever compromises the encryption key, they can go back and decrypt all the secrets that were ever encrypted with it.
• Ability to audit who accessed secrets is minimal. If you’re using a cloud key management system (e.g., AWS KMS), it will likely maintain an audit log of who used a key to decrypt something, but you won’t be able to tell what was actually decrypted.
• Not as test friendly: when writing tests for your Terraform code (e.g., with Terratest), you will need to do extra work to encrypt data for your test environments.
• Most managed key services cost a small amount of money. For example, each key you store in AWS KMS costs $1/month.

Technique #3: Secret Stores (e.g., Vault, AWS Secrets Manager)

The third technique relies on storing your secrets in a dedicated secret store: that is, a database that is designed specifically for securely storing sensitive data and tightly controlling access to it.

Overview

Here a few of the more popular secret stores you can consider:

1. HashiCorp Vault: Open source, cross-platform secret store.
2. AWS Secrets Manager: AWS-managed secret store.
3. AWS Param Store: AWS-managed data store that supports encryption.
4. GCP Secret Manager: GCP-managed key/value store.

These secret stores solve the “kick the can down the road” problem by relying on human memory: in this case, your ability to memorize a password that gives you access to your cloud provider (or multiple passwords in the case of Vault, as it uses Shamir’s Secret Sharing).

An example using AWS Secrets Manager

First, login to the AWS Secrets Manager UI, click “store a new secret,” and enter the secrets you wish to store:

The default is to use a JSON format, as you can see in the screenshot above. Next, give the secret a unique name:

Click “next” and “store” to save the secret.

Now, in your Terraform code, you can use the aws_secretsmanager_secret_version data source to read this secret (for HashiCorp Vault, AWS SSM Param Store, or GCP Secret Store, you’d instead use the vault_generic_secret, aws_ssm_parameter, or google_secret_manager_secret_version data source):

Copy
data "aws_secretsmanager_secret_version" "creds" {
# Fill in the name you gave to your secret
secret_id = "db-creds"
}

If you stored the secret data as JSON, you can use jsondecode to parse it:

Copy
locals {
db_creds = jsondecode(
data.aws_secretsmanager_secret_version.creds.secret_string
)
}

And now you can use those secrets in the rest of your Terraform code:

Copy
resource "aws_db_instance" "example" {
engine               = "mysql"
engine_version       = "5.7"
instance_class       = "db.t2.micro"
name                 = "example"

Copy
# Set the secrets from AWS Secrets Manager
username = local.db_creds.username
password = local.db_creds.password
}

Advantages of this technique

• Keep plain text secrets out of your code and version control system.
• Your secrets are stored in a dedicated secret store that enforces encryption and strict access control.
• Everything is defined in the code itself. There are no extra manual steps or wrapper scripts required.
• Using a web UI to store secrets is a nice user experience with a minimal learning curve.
• Secret stores typically support rotating secrets, which is useful in case a secret got compromised. You can even enable rotation on a scheduled basis (e.g., every 30 days) as a preventative measure.
• Secret stores typically support detailed audit logs that show you exactly who accessed what data.
• Secret stores typically expose an API that can easily be used from all your applications, and not just Terraform code. AWS Secrets Manager even generates code snippets that show you exactly how to read your secrets from apps written in Java, Python, JavaScript, Ruby, Go, etc:

Drawbacks to this technique

• Since the secrets are not versioned, packaged, and tested with your code, configuration errors are more likely, such as adding a new secret in one environment (e.g., staging) but forgetting to add it in another environment (e.g., production).
• Most managed secret stores cost money. For example, AWS Secrets Manager charges $0.40 per month for each secret you store, plus $0.05 for every 10,000 API calls you make to store or retrieve data.
• If you’re using a self-managed secret store such as HashiCorp Vault, then you’re both spending money to run the store (e.g., paying AWS for 3–5 EC2 instances to run Vault in a highly available mode) and spending time and money to have your team deploy, configure, manage, update, and monitor the store.
• Not as test friendly: when writing tests for your Terraform code (e.g., with Terratest), you will need to do extra work to write data to your secret stores.

Conclusion

Here are your key takeaways from this blog post:

1. Do not store secrets in plain text.
2. Use a Terraform backend that supports encryption.
3. Use environment variables, encrypted files, or a secret store to securely pass secrets into your Terraform code. See the table below for the trade- offs between these options.

Your entire infrastructure. Defined as code. In about a day. Gruntwork.io.