Promotion Workflows with Terraform

How to configure GitOps-driven, immutable infrastructure workflows for Terraform using Gruntwork Patcher.

A couple of months ago we announced the beta release of Gruntwork Patcher, a tool to automatically keep your infrastructure code up-to-date, even with breaking changes.

I’m now happy to announce the beta release of Patcher promotion workflow. This gives you a way to automatically promote changes from environment to environment using a GitOps-driven, immutable infrastructure workflow: you can use it to roll out a new module version from dev to stage to prod, all automatically.

As part of this blog post we are publishing open source example workflows for GitHub Actions that you can download here.

In the rest of this blog post, I’ll walk you through how the promotion workflows work. If you’re a Gruntwork customer you can request access today. If interested in trying Patcher out and you’re not already a customer, please contact the Gruntwork sales team.

The Promotion Workflow

The video shows a 3-stage promotion workflow running against Gruntwork’s internal dogfood-infrastructure-live repo which contains the Terragrunt and Terraform infrastructure code for Gruntwork’s live environments and services (e.g., app.gruntwork.io).

1. We release a new version of a Terraform module. This triggers the start of the promotion workflow
2. The workflow picks up the new release event from the dogfood-infrastructure-modules repo, runs Patcher and creates a PR to update the module dependencies for our dev environment
3. Next, once the PR for dev has been merged (and successfully deployed — using Gruntwork Pipelines to run ‘plan’ and ‘apply’ on the PR), the changes are promoted to our staging environment by running Patcher to update the module dependencies for our stage environment and creating a PR with the updates
4. Next, once the PR for stage has been merged (and successfully deployed), the changes are promoted to our production environment by running Patcher to update the modules dependencies for our prod environment and creating a PR with the updates
5. Finally, the PR is merged and the changes deployed to our production environment

This workflow can be used in couple of different ways:

• It can be run immediately in response to a new release in an infra-modules repo your team maintains. This how the workflow is triggered in the video and is how we use the workflow internally at Gruntwork.
• It can also be scheduled to run periodically to systematically pick up all other types of releases, for example you can run it weekly to pick up updates to the Gruntwork Infrastructure as Code Library.

How Patcher is Different

Patcher is aware that it is updating infrastructure code, unlike tools like DependaBot and RenovateBot, and has special handling for breaking changes.

• Awareness of breaking changes: Patcher regards a breaking change as any change that cannot be satisfied with a simple version bump. Patcher detects breaking changes on a per module basis, rather than on a per repo basis.
• Automated patches: for some breaking changes we provide patches that automatically update your code — we currently provide patches for our CIS AWS Foundations Benchmark compliant modules.
• Clear instructions: for other breaking changes we generate a readme file with an extract of the release notes. The extract will be either a migration guide, or an explanation for the potential side effects of the new version, or sometimes both.
• Opinionated workflow: we expect you to read, action and then delete those readme files before merging. To help with this, the workflow includes a check that prevents merging the generated PR until you’ve followed acknowledged the readme files by deleting them.

Automating the Workflow with GitHub Actions

As part of this post we are publishing open source example workflows for GitHub Actions that implement the promotion of infrastructure changes one environment at time.

The GitHub Actions workflows used in the video are available here.

Our workflows for GitHub Actions include examples of how to trigger a promotion from an upstream module repo and also how to trigger using a cron job.

Using Patcher to Update an Environment

At their core, the GitHub Actions workflows are creating a PR based on the result of running a Patcher command similar to this:

Copy
patcher update --non-interactive --update-strategy=next-breaking dev

This command is intended to be run in a CI pipeline and will find and update all module dependencies in the dev folder and any sub-folders.

By default, Patcher runs in interactive mode where the user selects individual module dependencies in the UI. The --non-interactive flag tells Patcher to run in a mode that’s optimised for automation with all output streamed to stdout/stderr. In this mode Patcher updates all the module dependencies it finds using the specified update strategy.

Using --update-strategy=next-breaking causes Patcher to update each dependency it finds to the latest version of that module, stopping at the next-breaking changing it encounters. You can read more about Patcher’s update strategies on gruntwork.io.

Patcher only updates one breaking change at a time. In some cases the breaking change may have “side effects”, such as updating state, that need to happen before further updates can be applied. In other cases, there may be “edge cases”, that the Patcher needs the reviewer to be aware of before proceeding.

When the command finishes, it outputs a summary to stdout in YAML format describing each of the modules dependencies that were found, the updates that were performed plus any manual actions that need to be taken.

Copy
successful_updates:
- file_path: dev/us-east-1/dev/services/gruntwork-website-preview-environments/
alb-logs-s3-bucket/terragrunt.hcl
updated_modules:
- repo: terraform-aws-monitoring
module: logs/load-balancer-access-logs
previous_version: v0.36.1
updated_version: v0.36.1
- file_path: dev/_global/route53-public/terragrunt.hcl
updated_modules:
- repo: terraform-aws-service-catalog
module: networking/route53
previous_version: v0.102.0
updated_version: v0.104.14
- file_path: dev/us-east-1/dev/vpc/terragrunt.hcl
updated_modules:
- repo: terraform-aws-cis-service-catalog
module: networking/vpc
previous_version: v0.43.0
updated_version: v0.44.0
next_breaking_version:
version: v0.44.0
release_notes_url: https://github.com/gruntwork-io/
terraform-aws-cis-service-catalog/releases/tag/v0.44.0
manual_steps_you_must_follow:
- instructions_file_path: dev/us-east-1/dev/vpc/README-TO-COMPLETE-UPDATE.md

The example YAML above describes the following:

• terraform-aws-monitoring/logs/load-balancer-access-logs: the dependency was left unchanged, pinned at v0.36.1 — the latest version
• terraform-aws-service-catalog/networking/vpc: the dependency was updated from v0.102.0 and pinned at v0.104.12 — the latest version
• terrafrom-aws-cis-service-catalog/networking/vpc: Patcher encountered a breaking change at v0.44.0, pinned the dependency to v0.44.0 and stopped because a manual check is required
• terrafrom-aws-cis-service-catalog/networking/vpc: Patcher included a link to the release note for v0.44.0 of the terraform-aws-cis-service-catalog.
• dev/us-east-1/dev/vpc: Patcher created a README-TO-COMPLETE-UPDATE.md file for a dependency in this folder that needs manually checking.

In this example, the README-TO-COMPLETE-UPDATE.md file contains an extract of the release note for v0.44.0 of the terraform-aws-cis-service-catalog:

Copy
# networking/vpc v0.43.0 -> v0.44.0 (2023.06.16 10:51:01)

Updated dependency networking/vpc in dev/us-east-1/dev/vpc/terragrunt.hcl to
version v0.44.0, which contains breaking changes. You MUST follow the
instructions in the release notes to complete this update safely:
https://github.com/gruntwork-io/terraform-aws-cis-service-catalog/releases/tag/v0.44.0

Here are the release notes for version v0.44.0:

## Description

- Updated launch configurations to launch templates

## Migration Guide
Launch configurations have been replaced by launch templates, the default
behavior of subnet public IP assignment will no longer be followed. This
should require no migration in most cases, however subnet type and public IP
assignment should be verified if these modules have been used outside of a
standard reference architecture deployment. Migration is only necessary if
this behavior is relied upon and involves setting `associate_public_ip_address`
appropriately in the launch template.

network_interfaces { associate_public_ip_address = true }

Copy

The PR Process

As part of the promotion workflow, Patcher opens a PR in each of your environments. The screenshot below shows a newly created PR for the dev environment infrastructure.

The PR description starts with the event that triggered the workflow, in this case “a new release in dogfood-infrastructure-modules”.

This is followed by the YAML output from the patcher update --non-interactive command.

And then, a list of the README-TO-COMPLETE-UPDATE.md files the PR reviewer needs to read, action and then delete

After the PR was created, two additional workflows ran:

• The Labeler workflow which ensures the PR is correctly labelled
• The Patcher workflow which fails if the branch being merged contains any README-TO-COMPLETE-UPDATE.md files, which prevents the branch being accidentally merged whilst there are manual steps that remain to be actioned

The promotion workflow relies on PRs being correctly labelled with the environment that the PR updates, in this example the updates-dev label.

The Labeler workflow also added do-not-merge label which acts as an indicator to the person reviewing the PR and provides a way to filter the PRs awaiting attention.

Manual Review of Breaking Changes

Next, if the PR contains any manual steps the reviewer or reviewers should read the generated README-TO-COMPLETE-UPDATE.md file or files.

Having read the readme files and taken any required actions, the reviewer should delete the README-TO-COMPLETE-UPDATE.md files and commit the deletions. When a PR contains for than 2–3 README-TO-COMPLETE-UPDATE.md files, we recommend using separate commits for each file.

Once the branch being merged no longer contains any README-TO-COMPLETE-UPDATE.md files, the check in the Patcher GitHub Actions workflow will pass and the PR can be merged.

Again, as a visual indicator the reviewer, the do-not-merge label will also be removed once all the README-TO-COMPLETE-UPDATE.md files have been removed.

The video below shows the promotion of a breaking change from dev through stage and to production.

Try It Yourself

The GitHub Actions workflows used in the video are available here.

You need to be a customer of the Gruntwork Infrastructure as Code Library to use Patcher.

If you’re already part of the Gruntwork Patcher private beta then you already have access to the Patcher and you can download and use the GitHub Action workflows right now.

If you’re a Gruntwork customer but aren’t yet part of the Patcher private beta then email support@gruntwork.io. Once your added, you’ll be able use the GitHub Action workflows straightaway.

If you like what you’ve just read but aren’t yet a Gruntwork customer please contact the Gruntwork sales team or you can register for early access Patcher at gruntwork.io.