Gruntwork Newsletter, November 2019

Once a month, we send out a newsletter to all Gruntwork customers that describes all the updates we’ve made in the last month, news in the DevOps industry, and important security updates. Note that many of the links below go to private repos in the Gruntwork Infrastructure as Code Library and Reference Architecture that are only accessible to customers.

Hello Grunts,

In the last month, we launched Gruntwork Compliance for the CIS AWS Foundations Benchmark, added cluster autoscaling support to our EKS modules, open sourced two new modules for setting up CI / CD with Google Cloud Build, used Terraform 0.12 features to consolidate and simplify the code for module-ecs, package-lambda, and module-load-balancer, and announced the dates for our winter holiday break. In other news, Helm 3 has been released, EKS now supports managed worker nodes, the ALB supports weighted routing, and Terraform 0.11 provider support is being deprecated.

As always, if you have any questions or need help, email us at support@gruntwork.io!

• Gruntwork Updates
• DevOps News

Gruntwork Updates

Announcing Gruntwork Compliance for the CIS AWS Foundations Benchmark

Motivation: The AWS Foundations Benchmark is an objective, consensus-driven guideline for establishing secure infrastructure on AWS from the Center for Internet Security. Several of our customers were interested in improving their security posture by achieving compliance with this benchmark, but meeting all the recommendations—from IAM to VPCs and CloudTrail to AWS Config—from scratch proved to be challenging.

Solution: We’re excited to announce Gruntwork Compliance for the CIS AWS Foundations Benchmark! We’ve made a number of changes that will make it possible for you to achieve compliance in days, not months:

1. We’ve extended the Infrastructure as Code Library to include compliance with the AWS Foundations Benchmark as a first class citizen. These compliance modules make it straightforward to implement the recommendations in the AWS Foundations Benchmark using infrastructure as code techniques for a repeatable compliance experience.
2. We’ve published the How to achieve compliance with the CIS AWS Foundations Benchmark deployment guide, which will walk you through the core concepts of the Benchmark, design guidelines for implementing the recommendations as code, and how to use the compliance modules.

What to do about it: Check out our announcement blog post and the compliance product page for all the details. If you’re interested in becoming a Gruntwork Compliance customer, contact us!

Cluster Autoscaler Support for EKS

Motivation: While Kubernetes natively supports autoscaling of the Pods that are deployed on a cluster, additional work was needed to setup autoscaling of the worker nodes. This meant that when you ran out of capacity on your EKS clusters such that you couldn’t schedule any more Pods, the only solution was to manually scale out more nodes in the cluster.

Solution: We’re happy to announce that we have added support for setting up cluster-autoscaler for use with EKS in terraform-aws-eks! This monitors Pod scheduling activities on your EKS cluster and automatically scales up more worker nodes if the cluster needs more capacity to schedule Pods. Additionally, the cluster-autoscaler application will automatically scale down worker nodes that are underutilized!

Other EKS updates:

• terraform-aws-eks, v0.9.8: You can now set enable detailed metrics gathering for the ASGs using the new var.enabled_metrics variable on the eks-cluster-workers module.
• terraform-aws-eks, v0.10.0: Updated Helm chart version from 0.1.6 to 0.1.11 in eks-alb-ingress-controller . You can now also set Pod priority class and configure liveness and readiness probes in the controller.

What to do about it: Update your EKS worker module calls to v0.9.7 and deploy the new eks-k8s-cluster-autoscaler module to try it out!

Consolidated modules for ecs-service

Motivation: Due to limitations with Terraform that prevented usage of dynamic inline blocks, we had to break up the ECS service code into 4 modules: one for use with either a CLB or no load balancer, one for use with an ALB or NLB, one for use with Service Discovery, and one for use with Fargate. If you needed some combination of these features, it wasn’t possible. Moreover, maintenance was more difficult, and some of the modules started to drift apart in terms of what features they offered.

Solution: With Terraform 0.12, we now have the ability to dynamically set inline blocks on resources. As a result we were able to consolidate all the separate ecs-service flavors into a single ecs-service module that can be configured for any of the different scenarios. Starting with v0.16.0, module-ecs will now only ship one ecs-service module.

A nice side effect of this is that all 4 use cases now have feature parity:

• You can now set up an App AutoScaling Target in all flavors, which will allow you to configure AutoScaling Policies on the ECS Service. Previously this was not available in ecs-service. Also, the module now outputs service_app_autoscaling_target_resource_id, which can be used for creating auto scaling policies.
• You can now fully setup the awsvpc network mode in all flavors. Specifically, the subnets and security groups of the ECS task are now configurable. Previously this was only available on ecs-fargate and ecs-service-with-discovery.
• You can now setup a canary deployment in all flavors. Previously this was only available on ecs-service and ecs-service-with-alb.
• You can now attach volumes on your ECS tasks in all flavors. Previously this was not available in ecs-service-with-discovery.

However, the consequence of this is that this is a massive backwards incompatible change. Due to the differences in naming of resources across the three modules (especially IAM resources), there is no way to avoid redeploying the ECS service when upgrading to the consolidated version.

Refer to the migration guide in the release notes for more information on how to upgrade.

What to do about it: Follow the migration guide in the release notes and update to module-ecs v0.16.1 to try out the consolidated ecs-service module!

Consolidated modules for package-lambda and module-load-balancer

Motivation: Just as with module-ecs, the lack of support for dynamic inline blocks in older Terraform versions led to a lot of code duplication in our package-lambda and module-load-balancer modules.

Solution: We’ve updated the modules to take advantage of Terraform 0.12’s support of dynamic inline blocks!

• module-load-balancer, v0.16.0: The two ALB resources used to switch on access logs have now been merged down to one resource. As a result of this consolidation, the following feature drift has been resolved on the ALB resource for the no logs flavor:idle_timeout was only defined on alb with logs; additional_security_group_ids was only being used on alb with logs
• package-lambda, v0.7.0: This consolidates the lambda resources in modules/lambda and modules/lambda-edge, taking advantage of the TF12 features that allow it. This allows for better maintainability of the modules.

What to do about it: Update to the new module versions above and let us know how they work for you!

New Open Source Modules for CI/CD on GCP using Google Cloud Build

Motivation: In May, in partnership with Google, we open-sourced a collection of reusable modules for Google Cloud Platform (GCP), including modules for Kubernetes (GKE), VPCs, Cloud SQL, Cloud Load Balancer, and more. These modules make it much easier to deploy and manage infrastructure on GCP, but lacked a way to setup an automated CI/CD pipeline.

Solution: We’ve open-sourced two new repos that show how you can set up an automated CI/CD pipeline using Google Cloud Build, a Google Kubernetes Engine (GKE) cluster and either a Google Cloud Source Repository or a GitHub repo.

• terraform-google-ci, v0.1.0: Terraform code, scripts and examples for deploying automated CI/CD pipelines on GCP.
• sample-app-docker, v0.1.0: A sample Docker application designed for use with Google Cloud Build.

What to do about it: Try out the two repos above and let us know how they work for you!

Winter break, 2019

Motivation: At Gruntwork, we are a human-friendly company, and we believe employees should be able to take time off to spend time with their friends and families, away from work.

Solution: The entire Gruntwork team will be on vacation December 23rd — January 3rd. During this time, there may not be anyone around to respond to support inquiries, so please plan accordingly.

What to do about it: We hope you’re able to relax and enjoy some time off as well. Happy holidays!

Open source updates

• terraform-kubernetes-helm, v0.5.3: Fixes a regression bug introduced in k8s-namespace and k8s-namespace-roles with the behavior of create_resources. Starting with terraform v0.12.11, referencing resource index 0 of those that were not created with count = 0 is an error where as before it was null. This means that we need to add a conditional for referencing those resources to only index into it when count > 0. This release also adds the create_resources parameter to k8s-service-account.
• terraform-kubernetes-helm, v0.6.0: The local-exec calls for k8s-tiller have been updated to be compatible with Windows. Note that this requires Powershell. Note that this is a backwards incompatible release. Refer to the migration guide in the release notes for more info.
• terraform-kubernetes-helm, v0.6.1: Fix regression bug from v0.6.0, where kubergrunt expects base64 encoded data, but we decode the data before passing it through.
• terratest, v0.22.1: helm.RenderTemplate now supports specifying the chart namespace. The function will now use the namespace of the provided KubectlOptions struct to render the template with a chosen namespace.
• terratest, v0.22.2: You can now specify the chart version in the helm functions.
• terratest, v0.22.3: Added docker.Build and docker.Run functions to run docker build and docker run, respectively, at test time.
• terratest, v0.23.0: helm.RenderTemplate now requires a releaseName to be passed. This allows you to check values that depend on the release name.
• terragrunt, v0.21.0: Changes --terragrunt-include-dir and --terragrunt-exclude-dir glob pattern matching to exact matching.
• terragrunt, v0.21.1: Fix missing dependencies in the lock file, which breaks certain downstream build processes.
• terragrunt, v0.21.2: Introduce synchronization logic to ensure that for a given dependency config, only have one terragrunt output call happening at once. This mitigates the issues #904 and #906.
• terragrunt, v0.21.3: DynamoDB tables generated by terragrunt to initialize state locks are now generated with PAY_PER_REQUEST billing mode.
• terragrunt, v0.21.4: You can now set the terragrunt download directory in the terragrunt.hcl config using the new download_dir attribute.
• terragrunt, v0.21.5: Add additional debug logging for helping to debug issues.
• terragrunt, v0.21.6: Support JSON format for terragrunt configuration files, as described in JSON configuration syntax for terraform.
• cloud-nuke, v0.1.12: Add --dry-run flag to list resources to nuke.

Other updates

• module-server, v0.7.5: Fix regression bug introduced in v0.7.4 with attach-eni, where some error messages were being swallowed in the script.
• module-ecs, v0.15.2: ecs-fargate now supports ECS Service Discovery.
• module-aws-monitoring, v0.14.2: This release updates the elasticsearch-alarms module to include all AWS recommended alarms, as documented at https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/cloudwatch-alarms.html.
• module-aws-monitoring, v0.14.3: The asg-disk-alarms module now allows you to configure how to handle the INSUFFICIENT_DATA state via the treat_missing_data input variable.
• module-cache, v0.7.0: The memcached version now sets the default version to 1.5.16. We were using 1.4.24 before, but that's no longer supported. If you wish to use a different version, use the memcached_version input variable. The redis module now sets the default version to 5.0.5. We were using 2.8.24 before, but that's now quite out of date. If you wish to use a different version, use the redis_version input variable.
• module-cache, v0.7.1: The redis module now allows custom names for subnet and security group.
• module-cache, v0.8.0: In the memcached and redis modules, we removed the bastion_host_security_group_id variable and added an allow_connections_from_security_groups variable, so you can now pass in a list of security group IDs that can connect to your cache, rather than just one.
• module-data-storage, v0.10.3: The aurora module now supports custom names for db subnets and security groups. We also made the tests more resilient to certain failures.

DevOps News

Helm 3 has been released

What happened: A major new release of Helm, version 3.0.0, is now generally available.

Why it matters: The major new features in Helm 3 are:

• There is no longer a need to run Tiller, the in-cluster component that was required by all previous versions of Helm. That means Helm is purely a client-side tool, which makes it much easier to use and more secure.
• Helm 3 includes support for a new version of Helm charts (Charts v2).
• Helm 3 also supports library charts — charts that are used primarily as a resource for other charts.
• A chart’s supplied values can now be validated against a JSON schema

What to do about it: We will be updating the Infrastructure as Code Library and Reference Architecture with support for Helm 3 as soon as we can. We will announce when this is ready.

EKS now supports managed worker nodes

What happened: AWS has added support for provisioning managed worker nodes for EKS.

Why it matters: Originally, EKS supported a fully managed control plane, but you had to run and manage the worker nodes yourself. Now, AWS can run the worker nodes for you automatically too, including handling node updates and terminations, gracefully draining nodes to ensure your applications stay available.

What to do about it: We will be updating terraform-aws-eks with support for managed worker nodes as soon as we can. We will announce when this is ready.

ALBs now support weighted target groups

What happened: Amazon’s Application Load Balancers (ALBs) now support weighted target groups. For example, if you have two target groups, you could assign one a weight of 8 and the other a weight of 2, and the load balancer will route 80% of the traffic to the first target group and 20% to the other.

Why it matters: This gives more fine-grained control over ALB routing, which can be used to implement features such as blue/green deployments and canary deployments. For example, to do a blue/green deployment, you would start with your old code (e.g., running in an ASG) in a “blue” target group with weight 1. You would then deploy your new code (e.g., into a new ASG) and register it with a new “green” target group with weight 0. Once the green target group is passing all health checks, you can switch its weight to 1 and the blue target group’s weight to 0, and all traffic will switch to your new code. You can then undeploy the old code.

What to do about it: Weighted target groups are not yet supported in Terraform; follow this issue for progress. We will be looking into this feature to add support for blue/green deployment to our ASG, ECS, and EKS code. We will announce when this is ready.

EC2 Instance Metadata Service v2

What happened: AWS has released version 2 of the EC2 Instance Metadata Service.

Why it matters: The new version of the EC2 Instance Metadata Service protects every request via session authentication. This helps to mitigate open firewalls, reverse proxies, and SSRF vulnerabilities.

What to do about it: Check out the announcement blog post for all the details. We’ll be looking into this functionality to see if/when we should support it in our modules.

Terraform 0.11 provider support is being deprecated

What happened: HashiCorp has announced that they will be deprecating Terraform 0.11 support for Terraform providers.

Why it matters: This change has two effects:

1. Provider bugs that are reproducible in Terraform 0.11, but not in Terraform 0.12 will be closed and left unfixed. So if you want fixes for those issues, you’ll have to upgrade to 0.12.
2. You’ll only be able to download the newest provider versions (via terraform init) with Terraform 0.12. So if you want any of the new features in your providers (e.g., support for a new AWS service announced at the upcoming re:Invent), you’ll have to upgrade to 0.12.

What to do about it: If you haven’t already, it’s time to upgrade to Terraform 0.12. Check out our migration guide for instructions.

Issue with destroying VPC using Terraform

What happened: On September 3rd, AWS announced improvements to the way AWS Lambda works with VPCs. These improvements introduced what is called a Hyperplane ENI that acts as a single entrypoint for multiple Lambda execution environments into your VPC. However, this change was not compatible with the way Terraform queries for ENIs when destroying network resources for Lambda. This caused terraform operations to fail with DependencyViolation errors. You can read more about this in the relevant Github issue for the AWS Provider.

What to do about it: This has been fixed in the Terraform AWS Provider version 2.31.0 or later. We recommend upgrading your AWS provider versions. You can use the version property on provider blocks to force terraform to use a version later than 2.31.0 :

Copy
provider "aws" {
version = "~> 2.31"
}