Browse the Repo

file-type-icon_docs
file-type-icon_images
file-type-icon01-architecture-overview.md
file-type-icon02-whats-deployed.md
file-type-icon03-security-compliance-compatibility.md
file-type-icon04-how-code-is-organized.md
file-type-icon05-dev-environment.md
file-type-icon06-ci-cd.md
file-type-icon07-monitoring-alerting-logging.md
file-type-icon08-ssh-vpn.md
file-type-icon09-accounts-and-auth.md
file-type-icon10-gruntwork-tools.md
file-type-icon11-deploying-a-docker-service.md
file-type-icon12-migration.md
file-type-icon13-deploying-the-reference-architecture-fr...
file-type-icon14-undeploying-the-reference-architecture.md
file-type-icon15-adding-new-environments-regions-and-acc...
file-type-iconREADME.md
file-type-iconmain
file-type-icon.gitignore
file-type-iconCODEOWNERS
file-type-iconREADME.md

Browse the Repo

file-type-icon_docs
file-type-icon_images
file-type-icon01-architecture-overview.md
file-type-icon02-whats-deployed.md
file-type-icon03-security-compliance-compatibility.md
file-type-icon04-how-code-is-organized.md
file-type-icon05-dev-environment.md
file-type-icon06-ci-cd.md
file-type-icon07-monitoring-alerting-logging.md
file-type-icon08-ssh-vpn.md
file-type-icon09-accounts-and-auth.md
file-type-icon10-gruntwork-tools.md
file-type-icon11-deploying-a-docker-service.md
file-type-icon12-migration.md
file-type-icon13-deploying-the-reference-architecture-fr...
file-type-icon14-undeploying-the-reference-architecture.md
file-type-icon15-adding-new-environments-regions-and-acc...
file-type-iconREADME.md
file-type-iconmain
file-type-icon.gitignore
file-type-iconCODEOWNERS
file-type-iconREADME.md
Single-account Reference Architecture

Single-account Reference Architecture

End-to-end tech stack designed to deploy into a single AWS account. Includes VPCs, EKS, ALBs, CI / CD, monitoring, alerting, VPN, DNS, and more.

Code Preview

Preview the Code

mobile file icon

06-ci-cd.md

down

Build, tests, and deployment (CI/CD)

In the previous section, you ran the app in the dev environment, made some changes to the code, and committed those changes to Git. Now it's time to see how you deploy your changes to AWS.

  • Deploying app changes: how to deploy changes to application code, such as a Ruby, Python, or Java web service packaged with Packer or Docker.
  • Deploying infrastructure changes: how to deploy changes to infrastructure code, such as Terraform modules that configure your VPCs, databases, DNS settings, etc.

Deploying app changes

Changes to your apps are deployed via an automated Continuous Integration (CI) and Continuous Delivery (CD) pipeline:

Trigger a build

  • We have configured CircleCi to run a build on every commit and new tag in your Git repos.

  • Note only changes in version control can kick off a CircleCi build; there is no way to trigger one manually!

Build and test the code

  • CircleCi is configured to build and test your code using a circle.yml file in the root of your repo. Check out the Configuring CircleCI docs for details on what you can configure with the circle.yml file.

  • Any build steps should typically go into compile section of circle.yml.

  • Test steps should typically go into the test section of circle.yml.

  • If any of the build steps fails, the entire build will fail, and the followers of that repo will get email notifications.

Package the code

We package the code as follows:

  • Build a new Docker image for the app
  • Tag the image with the Git revision (the sha1 hash) of the commit that triggered the build. This makes it easy to track Docker images back to the commits they contain.
  • Push the image to Amazon ECR
  • You can now deploy this new Docker image

Deploy the code

The deployment section of circle.yml configures the following deployment settings:

  • Commits to the master branch are automatically deployed to staging.
  • Tags of the format release-xxx are automatically deployed to production.

The automated deployment process works as follows:

  1. Git clone the infrastructure-live-acme repo.
  2. Update the version property in the service's terragrunt.hcl file to the new Docker image tag.
  3. git commit and git push the changes to the infrastructure-live-acme repo.
  4. Run terragrunt apply on the Terraform code to deploy the changes.

Deploying infrastructure changes

Deploying app changes is fairly easy, as all app deployments are roughly the same, and the cost of an error is fairly small. Deploying arbitrary infrastructure changes is more complicated. Due to the complexity and risks involved (e.g., an apply deployment going wrong usually does little damage, whereas an infrastructure deployment going wrong could, for example, delete a database!), we do not have a fully automated CI/CD pipeline for deploying infrastructure changes (note: Gruntwork is currently working on it; email support@gruntwork.io for details), so here's the process you should use:

Check out the infrastructure repos

All of the infrastructure code lives in the infrastructure-modules-acme and infrastructure-live-acme repos (see How the code is organized).

git clone git@github.com:gruntwork-io/infrastructure-modules-acme.git
git clone git@github.com:gruntwork-io/infrastructure-live-acme.git

Update the code

Make the appropriate changes in either infrastructure-modules-acme or infrastructure-live-acme.

  • If you want to change some settings for an existing module in an environment, make those changes in the terragrunt.hcl files in infrastructure-live-acme.

  • If you want to change how the underlying modules work or add a new module, make those changes in infrastructure-modules-acme. Note that after those changes are committed, you'll need to release a new version of infrastructure-modules-acme and update infrastructure-live-acme to use that new version (both of these steps are described below).

Test your changes in a sandbox environment

Every company should have a "sandbox" environment where developers can make Terraform changes without:

  1. Having to commit those changes to version control.
  2. Having to worry that if they break something, it will cause problems for the rest of the team.

While the company is small, a single shared sandbox environment is usually enough. As you grow bigger, you may need more to avoid conflicts. The gold standard is to have each developer spin up their own completely isolated environment for testing and to tear it down when they are done. Since all of your infrastructure is defined as code, you should be able to automate this process!

Once you've figured out where you are going to test, to apply your changes, do the following:

  1. Install Terragrunt. We use Terragrunt to keep the Terraform code as DRY as possible, so you'll need Terragrunt to make changes. See Keep your Terraform code DRY and Keep your remote state configuration DRY for background info on how we're taking advantage of Terragrunt.

  2. Configure your AWS credentials as environment variables.

  3. For some modules, you will also need to install kubergrunt and kubectl. kubergrunt is a collection of scripts packaged as a Go binary that provides certain functionality to make it easier to work with Kubernetes and EKS. kubectl is a comprehensive CLI for interacting with Kubernetes. You can check out the README for any infrastructure code to see if it requires them.

  4. Any modules that deploy applications to EKS will also need access to Helm. You can use the kubergrunt helm configure command to setup access on your machine. See the relevant section in section 8 "Accounts and Auth" of this document for more info. You can also look at the command documentation for more details on the command itself.

  5. Go into the appropriate folder in infrastructure-live-acme for your sandbox environment.

  6. Run terragrunt plan to see what impact your changes will have.

  7. If the plan looks good, run terragrunt apply to deploy those changes.

Note: if you've made changes in infrastructure-modules-acme, use the --terragrunt-source parameter to point Terragrunt at your local checkout of infrastructure-modules-acme. E.g., if you checked out infrastructure-modules-acme to ~/source/infrastructure-modules-acme, then you can test your changes locally, without creating a new release of infrastructure-modules-acme, by running terragrunt plan --terragrunt-source ~/source/infrastructure-modules-acme/my-module and terragrunt apply --terragrunt-source ~/source/infrastructure-modules-acme/my-module.

Note: you can also create automated tests for your modules using the Terratest library. Check out the test folder in any Gruntwork module for examples, such as the module-ecs tests and module-asg tests.

Submit a pull request

Once your code is working in the sandbox environment:

  • Commit your changes to a new branch

  • Submit a pull request so other team members can review your code.

  • If you only made changes in the sandbox environment in infrastructure-live-acme, then your pull request should include analogous changes in all the other environments in infrastructure-live-acme (but, of course, don't apply those changes yet!).

  • If you made changes in infrastructure-modules-acme, then submit the pull request in that repo first. After that pull request has been merged and a new version released, you can submit a pull request (or multiple pull requests) in infrastructure-live-acme updating all environments to use the new version. All of this is described in the next few sections.

Merge in your changes

If the code looks good, merge your changes into the master branch. The basic idea is that master should always reflect what's deployed in prod, so the next step is to deploy the code!

Release a new version

If your code changes included a change to infrastructure-modules-acme, you should release a new version by creating a new Git tag. If you use GitHub, you can do this by creating a new release on the releases page. Alternatively, you can do it with the Git CLI:

git tag -a v0.0.2 -m "added module foo"
git push --follow-tags

Promote the new version to prod

You can now begin deploying this new, immutable version of your infrastructure through all of your environments: e.g, qa -> stage -> prod. There are two ways to do this:

  1. You can submit a single pull request in infrastructure-live-acme to update all the environments. Once its merged, you can run terragrunt plan and terragrunt apply in each environment to deploy the changes.

  2. If you need to test the changes in each environment before deploying to the next environment, you may want to do a separate pull request for each environment, and merge and deploy them one at a time.

As you deploy each environment, make sure to carefully check that the plan output matches what you expect!

Next steps

Now that your code is built, tested, and deployed, it's time to take a look at Monitoring, Alerting, and Logging.

Questions? Ask away.

We're here to talk about our services, answer any questions, give advice, or just to chat.

Ready to hand off the Gruntwork?