README.md

Browse the Repo

.circleci

.github

_ci

_docs

examples

modules

aws-helpers

build-helpers

check-url

circleci-helpers

ec2-backup

ecs-deploy-runner-invoke-iam-policy

ecs-deploy-runner-standard-configuration

ecs-deploy-runner

git-helpers

gruntwork-module-circleci-helpers

iam-policies

infrastructure-deploy-script

infrastructure-deployer

install-jenkins

jenkins-server

kubernetes-circleci-helpers

monorepo-helpers

sign-binary-helpers

terraform-helpers

bin

README.md

install.sh

test

.gitignore

.pre-commit-config.yaml

CODEOWNERS

LICENSE.txt

README-CircleCI.adoc

README-Jenkins.adoc

README-Terraform-Terragrunt-Pipeline.adoc

README.adoc

setup.cfg

terraform-cloud-enterprise-private-module-...

Browse the Repo

.circleci

.github

_ci

_docs

examples

modules

aws-helpers

build-helpers

check-url

circleci-helpers

ec2-backup

ecs-deploy-runner-invoke-iam-policy

ecs-deploy-runner-standard-configuration

ecs-deploy-runner

git-helpers

gruntwork-module-circleci-helpers

iam-policies

infrastructure-deploy-script

infrastructure-deployer

install-jenkins

jenkins-server

kubernetes-circleci-helpers

monorepo-helpers

sign-binary-helpers

terraform-helpers

bin

README.md

install.sh

test

.gitignore

.pre-commit-config.yaml

CODEOWNERS

LICENSE.txt

README-CircleCI.adoc

README-Jenkins.adoc

README-Terraform-Terragrunt-Pipeline.adoc

README.adoc

setup.cfg

terraform-cloud-enterprise-private-module-...

EC2 backup

Snapshot your EC2 instances on a scheduled basis.

Terraform Helpers

This folder contains several helper scripts for working with Terraform. The scripts are:

terraform-update-variable: This script can automatically update a variable in a terraform.tfvars or terragrunt.hcl file and commit and push the changes to Git. It's common practice to store the version number of the currently deployed app in a variable and this script comes in handy for updating those variables automatically, such as in a CI job after building a new AMI or Docker image.
git-updated-files: This script can be used to detect all the files that have have been updated between two git refs (branch, tag, sha, etc). This can be used in a CI pipeline to find various scripts and files that were affected and only run actions against those files (e.g., build scripts).
git-updated-folders: This script can be used to detect all the folders that have have been updated between two git refs (branch, tag, sha, etc). This can be used in a CI pipeline to find the terraform and terragrunt modules that have changed, and only run plan or apply on the modules that has been affected.

Using the helper scripts in your code

You can install these scripts using the Gruntwork Installer:

gruntwork-install --module-name "terraform-helpers" --repo "https://github.com/gruntwork-io/terraform-aws-ci" --tag "0.0.1"

Note that terraform-update-variable depends on the git-helpers module being installed!

See the examples in the next section for how to use them.

Examples

The examples below should give you an idea of how to use the scripts. Run the scripts above with the --help flag to see full documentation.

terraform-update-variable
git-updated-files
git-updated-folders

terraform-update-variable

Terragrunt

Imagine you have a set of Terragrunt config in the live folder to deploy an app called rails-app. You create Amazon Machine Images (AMIs) for each new version of the app using Packer and the currently deployed version of the app is defined in a variable called rails_app_version, set to ami-12345 in live/rails-app/terragrunt.hcl.

If you just build a new version of the AMI, ami-67890, you could deploy it automatically, perhaps in a CI job, as follows:

cd live/rails-app
# Due to the existence of types in terragrunt.hcl, we need to be explicit about how the variable values are quoted.
terraform-update-variable --name rails_app_version --value "\"ami-67890\""
terragrunt apply

Running the commands above would:

Set the rails_app_version variable in the inputs map to the ID of the new AMI, ami-67890 in terragrunt.hcl.
commit and push the changes to terragrunt.hcl to Git.
Deploy the changes using Terragrunt.

Terraform

Imagine you have a set of Terraform templates in the templates folder to deploy an app called rails-app. You create Amazon Machine Images (AMIs) for each new version of the app using Packer and the currently deployed version of the app is defined in a variable called rails_app_version, set to ami-12345 in templates/terraform.tfvars.

If you just build a new version of the AMI, ami-67890, you could deploy it automatically, perhaps in a CI job, as follows:

cd templates
terraform-update-variable --vars-path terraform.tfvars --name rails_app_version --value "\"ami-67890\""
terraform apply

Running the commands above would:

Set the rails_app_version variable to the ID of the new AMI, ami-67890 in terraform.tfvars.
commit and push the changes to terraform.tfvars to Git.
Deploy the changes using Terraform.

git-updated-files

Build scripts

Consider the following typical terragrunt folder structure:

.
├── shared
│   └── us-east-1
│       └── _global
│           └── amis
│               └── build_web_server_ami.sh*
├── non-prod
│   ├── terragrunt.hcl
│   └── us-east-1
│       ├── qa
│       │   ├── env.yaml
│       │   ├── mysql
│       │   │   └── terragrunt.hcl
│       │   └── webserver-cluster
│       │       └── terragrunt.hcl*
│       ├── region.yaml
│       └── stage
│           ├── env.yaml
│           ├── mysql
│           │   └── terragrunt.hcl
│           └── webserver-cluster
│               └── terragrunt.hcl*
└── prod
    ├── terragrunt.hcl
    └── us-east-1
        ├── prod
        │   ├── env.yaml
        │   ├── mysql
        │   │   └── terragrunt.hcl
        │   └── webserver-cluster
        │       └── terragrunt.hcl*
        └── region.yaml*

Here we use * to indicate files that have been modified between the git refs development and master. To get the build scripts that have changed:

$ git-updated-files --source-ref development --target-ref master --ext sh
shared/us-east-1/_global/amis/build_web_server_ami.sh

This command looks for the files that changed with the extension sh. You can use this in your CI pipeline to act only on the files that changed in a particular commit. For example, in this case, you might want to invoke the build scripts that changed so that new AMIs can be built.

This command is not limited to a single extension. You can detect other files (e.g. yaml files) by passing additional --ext arguments. For example, to get the region.yaml file, you can run:

$ git-updated-files --source-ref development --target-ref master --ext sh --ext yaml
shared/us-east-1/_global/amis/build_web_server_ami.sh
prod/us-east-1/region.yaml

You can also do search for files that changed that do not match the extension. For example, to get all files that changes that are not terragrunt.hcl:

$ git-updated-files --source-ref development --target-ref master --exclude-ext terragrunt.hcl
shared/us-east-1/_global/amis/build_web_server_ami.sh
prod/us-east-1/region.yaml

git-updated-folders

Terragrunt

Consider the following typical terragrunt folder structure:

.
├── shared
│   └── us-east-1
│       └── _global
│           └── amis
│               └── build_web_server_ami.sh
├── non-prod
│   ├── terragrunt.hcl
│   └── us-east-1
│       ├── qa
│       │   ├── env.yaml
│       │   ├── mysql
│       │   │   └── terragrunt.hcl*
│       │   └── webserver-cluster
│       │       └── terragrunt.hcl
│       ├── region.yaml
│       └── stage
│           ├── env.yaml
│           ├── mysql
│           │   └── terragrunt.hcl*
│           └── webserver-cluster
│               └── terragrunt.hcl
└── prod
    ├── terragrunt.hcl
    └── us-east-1
        ├── prod
        │   ├── env.yaml
        │   ├── mysql
        │   │   └── terragrunt.hcl*
        │   └── webserver-cluster
        │       └── terragrunt.hcl
        └── region.yaml

Here we use * to indicate files that have been modified between the git refs development and master. To get the terragrunt modules that have changed:

$ git-updated-modules --source-ref development --target-ref master --terragrunt
non-prod/us-east-1/qa/mysql
non-prod/us-east-1/stage/mysql
prod/us-east-1/prod/mysql

Each of these folders were detected as terragrunt modules (because they contain a terragrunt.hcl file), and were marked as changed because the underlying terragrunt config was updated.

Note that this will only detect terragrunt.hcl files that have changed. For example, in the following scenario where the root terragrunt.hcl file and a few yaml files were changed (the files denoted with * are those that changed), only the root folder is returned:

.
├── non-prod
│   ├── terragrunt.hcl*
│   └── us-east-1
│       ├── qa
│       │   ├── env.yaml
│       │   ├── mysql
│       │   │   └── terragrunt.hcl
│       │   └── webserver-cluster
│       │       └── terragrunt.hcl
│       ├── region.yaml*
│       └── stage
│           ├── env.yaml
│           ├── mysql
│           │   └── terragrunt.hcl
│           └── webserver-cluster
│               └── terragrunt.hcl
└── prod
    ├── terragrunt.hcl
    └── us-east-1
        ├── prod
        │   ├── env.yaml
        │   ├── mysql
        │   │   └── terragrunt.hcl
        │   └── webserver-cluster
        │       └── terragrunt.hcl
        └── region.yaml*

$ git-updated-modules --source-ref development --target-ref master --terragrunt
non-prod

If you want to additionally pull in the folders with the yaml files that changed, you can pass in --ext .yaml:

$ git-updated-modules --source-ref development --target-ref master --terragrunt --ext .yaml
non-prod
non-prod/us-east-1
prod/us-east-1

Terraform

Consider the following project structure for a monorepo terraform project:

.
├── modules
│   ├── mysql
│   │   ├── main.tf
│   │   ├── outputs.tf*
│   │   └── variables.tf
│   └── webserver-cluster
│       ├── main.tf
│       ├── outputs.tf
│       └── variables.tf
└── live
    └── us-east-1
        ├── prod
        │   ├── terraform.tfvars
        │   ├── main.tf*
        │   ├── outputs.tf*
        │   └── variables.tf
        └── non-prod
            ├── terraform.tfvars*
            ├── main.tf
            ├── outputs.tf
            └── variables.tf

Here we use * to indicate files that have been modified between the git refs development and master. To get the terraform modules that have changed:

$ git-updated-modules --source-ref development --target-ref master --terraform
modules/mysql
live/us-east-1/prod
live/us-east-1/non-prod

Each of these were detected as terraform modules (because they contain .tf files), and were marked as changed because one of the underlying .tf or .tfvars files have changed. Specifically:

modules/mysql <= modules/mysql/outputs.tf
live/us-east-1/prod <= live/us-east-1/prod/main.tf and live/us-east-1/prod/outputs.tf
live/us-east-1/non-prod <= live/us-east-1/non-prod/terraform.tfvars

Deleted folders

By default, git-updated-folders will include deleted folders in the return list. This can cause problems in infrastructure pipelines as you don't want to run any plan or apply action on folders that don't exist.

To support destroy workflows, the git-updated-folders script supports two filters:

--exclude-deleted: This parameter will tell git-updated-folders to exclude folders that have been deleted when returning the updated folders.
--include-deleted-only: This parameter will tell git-updated-folders to only include folders that have been deleted when returning the updated folders.

Note that a folder is only considered deleted if all the tracked files in that folder is deleted. E.g., if you have a folder that only deleted the *.tf files, but keeps a README.md, then that folder is NOT considered deleted by the script because it still contains a tracked file.

In your infrastructure pipelines, you should use --exclude-deleted to filter out the deleted folders from your plan and apply actions so that Terraform/Terragrunt can run. If you wish to implement destroy workflows, you can use the --include-deleted-only parameter to setup a separate pipeline for handling destroy actions.

Questions? Ask away.

We're here to talk about our services, answer any questions, give advice, or just to chat.

Ready to hand off the Gruntwork?

Buy Now

"https://cdn.gruntwork.io/gruntwork-website/"

{"index":{"js":"https://cdn.gruntwork.io/gruntwork-website/index.bundle.c7884255553b53fbca3a.js","map":"https://cdn.gruntwork.io/gruntwork-website/index.bundle.1b14c1b7d19f1f5eb35d6e118e838255.map"},"styles":{"css":"https://cdn.gruntwork.io/gruntwork-website/styles.bundle.f22938926651ddec7c49.css","js":"https://cdn.gruntwork.io/gruntwork-website/styles.bundle.e782420e74a20dcb8691.js","map":"https://cdn.gruntwork.io/gruntwork-website/styles.bundle.d5e2af49807c6ca33f8367d621ece507.map"},"vendors":{"css":"https://cdn.gruntwork.io/gruntwork-website/vendors.bundle.29f7d0366a0978763f96.css","js":"https://cdn.gruntwork.io/gruntwork-website/vendors.bundle.fa8174a130797d75d12c.js","map":"https://cdn.gruntwork.io/gruntwork-website/vendors.bundle.57243d94deeeb29d5061288a338b4eb6.map"}}

EC2 backup

Code Preview

Preview the Code

Terraform Helpers

Using the helper scripts in your code

Examples

terraform-update-variable

Terragrunt

Terraform

git-updated-files

Build scripts

git-updated-folders

Terragrunt

Terraform

Deleted folders

Questions? Ask away.

Ready to hand off the Gruntwork?