Gruntwork Newsletter, July 2019

Once a month, we send out a newsletter to all Gruntwork customers that describes all the updates we’ve made in the last month, news in the DevOps industry, and important security updates. Note that many of the links below go to private repos in the Gruntwork Infrastructure as Code Library and Reference Architecture that are only accessible to customers.

Hello Grunts,

We’re excited to announce that it’s finally time to upgrade to Terraform 0.12! We’ve updated the entire Infrastructure as Code Library to be compatible with it and have included upgrade instructions in this blog post. In other news, the Early Release of Terraform: Up & Running, 2nd edition is now available (also fully updated for Terraform 0.12), Terragrunt now supports all Terraform built-in functions and has first-class support for GCS, and we’ve made several dozen other improvements and fixes.

As always, if you have any questions or need help, email us at support@gruntwork.io!

• Gruntwork Updates
• DevOps News
• Security Updates

Gruntwork Updates

It’s time to upgrade to Terraform 0.12!

‍

Motivation: Terraform 0.12 came out a couple months ago and we’ve all been waiting to use it.

Solution: We’ve updated the entire Infrastructure as Code Library to be compatible with Terraform 0.12! That means you should now upgrade to take advantage of first-class expressions, for and for_each, a more powerful type system, better error messages, and more.

What to do about it:

Prepare for the upgrade:

1. Read through the Terraform 0.12 upgrade guide so you can learn about the types of changes you’ll have to make.
2. Install Terraform 0.12.x. The latest version at the time of this writing was 0.12.4.
3. If you’re a Terragrunt user, install Terragrunt v0.19.x (or newer). The latest version at the time of this writing was v0.19.8. Make sure to read and follow the steps in the Terragrunt 0.19.x upgrade guide.
4. If you’re a Terratest user, install Terratest v0.16.x (or newer). The latest version at the time of this writing was v0.17.5.
5. Take a look at the Gruntwork module compatibility table. The right-most column lists the minimum version number that is compatible with Terraform 0.12.

Do the upgrade:

1. Find each of your Terraform modules (e.g., in your infrastructure-modules repo). We recommend updating and testing one module at a time. For the rest of this step-by-step guide, let’s assume you have a module ininfrastructure-modules/networking/vpc-app.
2. Go into the folder with the Terraform module:cd infrastructure-modules/networking/vpc-app
3. Search your code for any references to Gruntwork modules. Update each module to the version number specified in the right-most column of the Gruntwork module compatibility table (or newer). For example, infrastructure-modules/networking/vpc-app most likely makes use of module-vpc, which you’ll want to upgrade to v0.6.0 (or newer).
4. Make sure to check the release notes for the module to see any other changes you need to make. E.g., Here are the release notes for module-vpc, v0.6.0.
5. If you have a required_version constraint in your code, for the time being, remove it so the Terraform auto-upgrade step doesn’t complain. We’ll bring it back later.
6. Run terraform init -backend=false.
7. Run terraform 0.12upgrade.
8. Delete or edit the generated versions.tf file. We recommend specifying a strict required_version constraint in your Terraform code: e.g., required_version = "= 0.12.4".
9. There are a few gotchas you may have to handle manually. One gotcha is that Terraform no longer converts booleans to 1 or 0 automatically. For example, count = var.boolean_variable no longer works, so you have to explicitly change it to count = var.boolean_variable ? 1 : 0. A second gotcha is that 0.12upgrade occasionally duplicates comment blocks, so you’ll need to delete the extra ones.
10. Go through any variable declarations and add appropriate type constraints.
11. Try to run your code in a pre-prod environment. E.g., Go into infrastructure-live, cd into your dev environment, and run terragrunt apply --terragrunt-source /path/to/infrastructure-modules/networking/vpc-app.
12. Once everything is working, commit your changes, release a new version, and deploy the new version to all environments.
13. Repeat these steps with all of your other Terraform modules.

Terraform: Up & Running, 2nd edition is here!

‍

Motivation: In late 2016, we released the Comprehensive Guide to Terraform blog post series. In early 2017, we turned it into a book, Terraform: Up & Running. In the two years since, Terraform has changed considerably (4 major releases, a change to HCL2, a revamp of Terraform state, and much more), so both the blog post series and book were due for an update.

Solution: We’re happy to announce that (1) we’ve updated the Comprehensive Guide to Terraform blog post series all the way through Terraform 0.12 and (2) the Early Release of the 2nd edition of *Terraform: Up & Running* is now available! The 2nd edition of the book is nearly double the length of the 1st edition (~160 more pages), and it has been fully updated through Terraform 0.12, including two completely new chapters: Production-grade Terraform Code and How to Test Terraform Code.

What to do about it: Check out the Terraform: Up & Running, 2nd edition Early Release blog post to learn about the top 10 things that have changed in Terraform since the 1st edition, read the early release on O’Reilly, and let us know what you think!

Terragrunt now supports all Terraform built-in functions

Motivation: Last month, we upgraded Terragrunt to support Terraform 0.12 and use HCL2 syntax. This new syntax includes support for first-class functions and expressions, and users wanted to be able to take advantage of that more.

Solution: We’ve updated Terragrunt so that it now supports ALL built-in Terraform functions! You can use them to do math (e.g., ceil, min, max), manipulate strings (e.g., format, split, substr), interact with files (file, dirname), and much more. And since you can use expressions and functions everywhere in terragrunt.hcl files, this gives you a lot of expressive power.

‍

# Examples of using built-in functions throughout terragrunt.hcl

terraform {
  source = "../modules/${basename(get_terragrunt_dir())}"
}

remote_state {
  backend = "s3"
  config = {
    bucket = trimspace("   my-terraform-bucket     ")
    region = join("-", ["us", "east", "1"])
    key    = format("%s/foo.tfstate", path_relative_to_include())
  }
}

inputs = merge(
  yamldecode(file(find_in_parent_folders("common.yaml"))),
  {
      override = "some-value"
  },
)

‍

What to do about it: Upgrade to Terragrunt v0.19.4 and let us know what you think!

Terragrunt now supports GCS as a remote state backend

Motivation: In May, we announced a set of reusable, production-grade infrastructure modules for GCP. They work great with Terraform, however, we wanted to add support to Terragrunt for use with our GCP Reference Architecture.

Solution: We’ve updated Terragrunt so that it now supports storing remote state in a GCS bucket. It will also automatically create one with versioning enabled if it doesn’t exist already.

What to do about it: Upgrade to Terragrunt v0.19.8 and let us know what you think!

Open source updates

• terraform-google-gke, v0.3.1: Workaround for a bug in Terraform 0.12 which prevents the evaluation of data providers if they are used in a ternary operation.
• terraform-google-network, v0.2.2: The root example has been updated to pass through the project variable to all the resources.
• terraform-google-network, v0.2.3: The bastion-host module now outputs the instance private_ip address.
• terraform-google-network, v0.2.4: The bastion-host module google_compute_instance resource now uses the supplied project variable.
• terraform-google-network, v0.2.5: The vpc-network module now outputs the correct private subnetwork name.
• terraform-google-sql, v0.2.0: The cloud-sql module has now been upgraded to support Terraform v0.12.
• Terragrunt, v0.19.3: Terragrunt will now clean up .tf.json files in addition to .tf files in the download dir when fetching code from a source URL.
• Terragrunt, v0.19.5: Fix how include blocks are parsed so that path_relative_to_include, path_relative_from_include, and get_parent_terragrunt_dir work correctly when used directly in the child terragrunt.hcl.
• Terragrunt, v0.19.7: Introduce a new config option for the remote state backend in terragrunt.hcl(disable_init) which, when set to "true", will skip the backend initialization step.
• Terragrunt, v0.19.8: Ensures project and location are optional when using the GCS backend. Also fixes a bug where Terragrunt may create a GCS state bucket in the wrong location.
• Terratest, v0.17.2: Fixed the various terraform.Output commands (e.g., OutputE, OutputListE, OutputMapE, etc) to only read from stdout when parsing output. Before, they were also accidentally reading stderr. This worked fine with Terraform, as the output command wrote nothing to stderr, but did not work with Terragrunt.
• Terratest, v0.17.3: You can now specify a custom buffer size when running shell commands and terraform. The default scanners in golang have a limit of 64kb per line, which for some terraform modules is not enough. This allows you to extend the buffer so that you can test terraform modules that have long outputs.
• Terratest, v0.17.4: This release fixes an issue with dep, where the client-go library did not have a locked version of k8s.io/api. This caused problems when resolving dependencies because the latest version of k8s.io/api has breaking changes that are not compatible with version release-9.0 of client-go that we are using. This release locks down the k8s.io/api dependency so that users pulling in terratest will not run into this issue.
• Terratest, v0.17.5: This release improves support for testing multi nodegroup kubernetes clusters by exposing helper functions that can be used to get the endpoint from specific node groups. Check out the release notes for more details.
• terraform-aws-vault, v0.13.1: Fixed the syntax in the for_each loop used to add tags to the ASG from the cluster_extra_tagsargument.
• terraform-aws-consul, v0.7.1: Add Type=notify to systemd config for Consul so that any other systemd services that depend on Consul (e.g., via Requires=consul.service) will be started after Consul is fully up and running, rather than as soon as the Consul task is started.
• kubergrunt, v0.5.1: The TLS subject info JSON previously only allowed one name for some of the fields (e.g org for organization), but it now allows using alternate names. Checkout the release notes for more details.

Other updates

• package-openvpn, v0.8.1: install-openvpn now supports Ubuntu 18.04.
• package-openvpn, v0.8.2: This release introduces the ability to set an expiration lifecycle on the objects in the S3 backup bucket for the openvpn-server module.
• package-openvpn, v0.9.1: Fix bug where the IAM role for the openvpn server did not have a lifecycle config for create_before_destroy, leading to issues when trying to do a rolling update.
• package-openvpn, v0.9.2: Populate DNS server from proper location on Ubuntu 18.04. This should fix DNS resolution on client machines.
• package-elk, v0.3.0: This release fixes a bug where AWS region of the s3 bucket was hardcoded to us-east-1 in the elasticsearch-cluster-backup module. With this change, the elasticsearch-cluster-backup module now takes in a new required input variable region.
• package-elk, v0.3.1: This release fixes a bug in the install scripts where for some base AMIs, the install script could hang on a user prompt that will never be answered in the context of the automation script during apt-get upgrade.
• module-ci, v0.13.16: Add a variable for aws_alb_target_group.health_check.matcher to the jenkins-server module so that it can be configured.
• module-ci, v0.14.1: This release fixes a regression in the setup-minikube script in the kubernetes-circleci-helpers module, caused by the removal of the specific docker version we depended on from the ubuntu apt caches. Also, the install-jenkins module is now verified to work with Ubuntu 18.04. Finally, the jenkins-server module is now using v0.8.1 of module-asg/modules/server-group, which includes a fix for IAM timing issues.
• module-asg, v0.7.1: Fix bug where var.enable_elastic_ips was not properly used in the conditional logic to control Route 53 records in modules/server-group. This led to syntax errors on terraform 0.12 when you had the right inputs to enable the resource.
• module-asg, v0.8.0: All the module variables have been updated to use concrete types based on the new type system introduced in terraform 0.12.0. You can learn more about the types in the official documentation.
• module-asg, v0.8.1: There is a timing bug with IAM role and instance profile creation that causes the initial terraform apply to fail for code using the server-group module. This release addresses that by adding a sleep to wait for IAM resource propagation after creation.
• module-load-balancer, v0.14.1: Fixes a bug that arises when using terraform >=0.12.2 with the nlb module. Specifically, the access_logs subblock requires a valid bucket and prefix to be specified if the block is included, regardless of enabled flag. This release fixes it so that you can still pass in a null or empty bucket and prefix even if the access logs are disabled.
• terraform-aws-eks, v0.5.5: Fix bug where IAM to RBAC mapping did not work with capital letters in the entity name. This caused login issues because the script would naively use the IAM role / user name as the Kubernetes username, which were invalid when they contained upper case letters.
• module-security, v0.16.6: This release introduces a new module ssm-healthchecks-iam-permissions which provides IAM policies that you can attach to instance profiles that grants the EC2 instance the requisite permissions to run SSM healthchecks, which are enabled by default on many base AWS AMIs such as Ubuntu.
• module-security, v0.17.1: Fix a bug where the crontab configured by ssh-grunt install was missing the --force-user-deletion flag.
• module-aws-monitoring, v0.13.1: This release verifies compatibility of various module scripts in the repo with Ubuntu 18.04. Prior to this version, all modules except for logs/cloudwatch-log-aggregation-scripts worked with Ubuntu 18.04. This release fixes the logs/cloudwatch-log-aggregation-scripts module to also be compatible with Ubuntu 18.04.
• module-aws-monitoring, v0.13.2: Fix type constraint on the metrics variable of the cloudwatch-dashboard-metric-widget module to allow non-string types in the inner list, including map values.
• module-aws-monitoring, v0.13.3: You can now disable yellow cluster status alarms for Elasticsearch by setting var.disable_status_yellow_alarm to true.
• module-cache, v0.6.1: This release fixes a bug where the module errors on the output if you set both replication_group_size and cluster_modes input variables in the redis module.
• module-server, v0.7.1: Starting this release, all the scripts are now tested and verified to work with Ubuntu 18.04. Note that no change has been made to the scripts themselves in this release.
• module-server, v0.7.2: Fix mount-ebs-volume so it works properly on Amazon Linux.

DevOps News

NLB now supports UDP

What happened: The AWS Network Load Balancer (NLB) now supports load balancing UDP traffic.

Why it matters: Before, the NLB only supported TCP, but with UDP support, the NLB can now handle many new use cases, such as DNS, logging, and IoT.

What to do about it: Check out the documentation for details.

New Service: Amazon EventBridge

What happened: Amazon has launched a new service called EventBridge, which aims to be a central event bus for all of your SaaS applications and AWS Services.

Why it matters: You can use EventBridge to trigger one of many targets (e.g., Lambda function, ECS Task, Kinesis Stream, SNS message) from one of many sources (e.g., more than 90 AWS Services, such as EC2 Auto Scaling, CodeBuild, and AWS Step Functions and ~10 SaaS applications, including ZenDesk, DataDog, and PagerDuty). This allows you to connect all of your systems together: e.g., a support ticket in ZenDesk can trigger a Lambda function that issues a response.

What to do about it: Give EventBridge a shot and let us know what you think!

SSH to EC2 Instances with EC2 Instance Connect

What happened: AWS has launched a new feature called EC2 Instance Connect that allows you to SSH to your EC2 Instances using your IAM credentials for authentication.

Why it matters: The main advantages of EC2 Instance Connect are a) you don’t have to manage SSH keys, b) each developer connects with their IAM credentials, rather than a shared SSH Key, and c) it works from the AWS Web Console, in addition to your terminal. The main disadvantages of EC2 Instance Connect are that (a) it doesn’t use the SSH protocol, so you can’t use all the SSH port forwarding, tunneling, and other features you’re used to and (b) it doesn’t create OS users for you, so you can only connect as OS users you’ve created yourself through other mechanisms.

What to do about it: If you just need an easy way to connect to EC2 Instances, and don’t care much about the OS users or protocol, give EC2 Instance Connect a shot! However, if you want actual SSH connections and to auto sync OS users (and permissions) with your IAM users, you may want to stick with ssh-grunt.

VPC Traffic Mirroring

What happened: AWS has launched a new feature called VPC Traffic Mirroring that allows you to replicate the networking traffic to an EC2 Instance or EC2 Instances behind an NLB, filter the traffic, and forward it to other destinations.

Why it matters: This feature can be very useful for troubleshooting (e.g., send the traffic to an environment where devs can analyze it and debug issues), monitoring (e.g., measure and alert on certain types of network traffic), and security (e.g., detect an attack), all without adding any extra complexity or overhead to your EC2 Instances.

What to do about it: Check out the documentation for details on how to use this feature. Note that VPC Traffic Mirroring currently only works with Nitro instances.

Aurora Serverless now works with PostgreSQL

What happened: AWS has announced that Aurora Serverless now supports PostgreSQL.

Why it matters: Aurora Serverless allows you to have an on-demand database that would shut down when not in use and spin up quickly when you need it, which was great for saving money on infrequently used apps and pre-production environments. However, the original release only worked with MySQL. This new announcement means you can now take advantage of the serverless functionality with PostgreSQL too.

What to do about it: See the announcement blog post for details.

Security Updates

Below is a list of critical security updates that may impact your services. We notify Gruntwork customers of these vulnerabilities as soon as we know of them via the Gruntwork Security Alerts mailing list. It is up to you to scan this list and decide which of these apply and what to do about them, but most of these are severe vulnerabilities, and we recommend patching them ASAP.

Ruby

• strong_password Ruby Gem: The strong_password Ruby gem has been hacked, and replaced with a Gem that silently executes remote code in production. If you are using this Gem, make sure you are not using release v0.0.7!

Python

• CVE-2019–12308, ALAS-2019–1230: Several vulnerabilities were found in Python, plus one in Django, that could allow information disclosure, injection attacks, and XSS attacks.