Gruntwork Newsletter, June 2022

Every few months, we send out a newsletter to all Gruntwork customers that describes all the updates we’ve made since the last newsletter and news from the DevOps industry. Note that many of the links below go to private repos in the Gruntwork Infrastructure as Code Library and Reference Architecture that are only accessible to customers.

Hello Grunts,

Since the last newsletter, we’ve added multiple new modules! These include modules for: Amazon MSK, which is a fully-managed Apache Kafka service; Steampipe Runner, which allows you to run Steampipe Mods (such as CIS, SOC 2, etc.) to continuously check your AWS accounts for compliance using EC2 Deploy Runner; AWS HTTP API Gateway, which allows you to create serverless apps using API Gateway v2; CloudWatch Logs to Slack, which can send filtered CloudWatch Logs to a Slack channel; and run-lambda-entrypoint which allows you to run Docker containers in Lambda with a secure, first-class integration with AWS Secrets Manager. We’ve also made many other updates and bug fixes, as detailed below.

As always, if you have any questions or need help, email us at support@gruntwork.io!

• Gruntwork Updates
• DevOps News

Gruntwork Updates

[NEW MODULE]: Amazon MSK (fully-managed Kafka)

Motivation: Running Apache Kafka (and ZooKeeper) yourself is a considerable undertaking, so a number of customers have asked us to add support for Amazon MSK, which is a fully-managed Kafka service.

Solution: We’ve added a new msk module to the Gruntwork IaC Library! You can now deploy a fully-managed, scalable, highly available Kafka cluster with IAM auth using just a few lines of code:

module "msk" {
  source = "git::git@github.com:gruntwork-io/terraform-aws-messaging.git//modules/msk?ref=v0.8.2"
  
  cluster_name  = "my-kafka-cluster"
  cluster_size  = 2
  instance_type = "kafka.t3.small"
  kafka_version = "2.8.1"
  
  # Enable IAM Auth
  enable_client_sasl_iam = true
  
  # Configure network access
  subnet_ids                         = var.subnet_ids
  vpc_id                             = var.vpc_id
  allowed_inbound_security_group_ids = var.allowed_inbound_sgs
}

‍

A note on deprecation: Please note that, with this release, we are putting our older, self-managed Kafka (terraform-aws-kafka) and ZooKeeper (terraform-aws-zookeeper) repos on hold. They have gotten relatively little customer interest and are much more complicated to use and maintain than MSK, so we are deprecating them in favor of this new MSK module.

What to do about it: Check out the new msk module and the msk-with-iam-auth example to get started, and let us know what you think!

[NEW MODULE]: Steampipe Runner for Gruntwork Compliance

NOTE: This module is only available to customers who purchased the CIS add-on.

Motivation: Gruntwork CIS customers received access to a module to deploy and configure AWS Security Hub for continuous monitoring of their environments against the CIS benchmark. Security Hub will periodically scan your environment for findings that violate the CIS benchmark and report on them in a dashboard that operators can use to assess and resolve those findings. However, AWS Security Hub only supports a limited number of benchmarks, and even the CIS benchmark is outdated (only supports version 1.2.0, whereas the latest benchmark is at 1.4.0).

Solution: We’ve introduced an extension to Gruntwork Pipelines that allows you to run Steampipe to scan your AWS accounts using the credentials granted to the ECS Deploy Runner! The Steampipe Runner is capable of running any checks that are implemented as Steampipe Mods, including the full range of compliance checks implemented in the steampipe-mod-aws-compliance mod.

Note on license: This repo includes code to install and run Steampipe in an AWS ECS Fargate task. The Steampipe CLI is AGPL Licensed. This repo contains a Dockerfile that installs the precompiled binary of Steampipe without any modification to the source code. All usage of Steampipe is directly with the precompiled binary, and complies with AGPL.

What to do about it: If you are a Gruntwork CIS subscriber, you should have gotten access to the new terraform-aws-ci-steampipe repo. Give the new**** ecs-deploy-runner-with-steampipe-runner module a try and let us know what you think!

[NEW MODULE]: AWS HTTP API Gateway Module

Motivation: For configuring API Gateway, we offered two options to customers — the now deprecated gruntsam utility for configuring custom routes, and api-gateway-proxy for exposing a full Lambda web app (e.g., built on Express.js). Both had limitations, with the former requiring code generation that was difficult to maintain over time, while the latter was too limiting in its use cases. Furthermore, both only supported API Gateway V1.

Solution: We’ve implemented a new module for configuring API Gateway V2: the lambda-http-api-gateway module! This module allows you to encode the function routes using the new routing rules of API Gateway V2, greatly improving the experience of configuring API Gateway. Using the new routing rules, you can easily configure your API Gateway to forward different requests to different Lambda functions, unlike the api-gateway-proxy module.

Here is an example to define an API Gateway that routes GET requests to the path /hello to the hello_lambda Lambda function, while POST requests to the path /world are routed to the world_lambda Lambda function:

module "api" {
  source = "git::git@github.com:gruntwork-io/terraform-aws-lambda.git//modules/lambda-http-api-gateway?ref=v0.19.0"
  
  name = "my-api"
  route_config = {
    "GET /hello" = {
      lambda_function_arn = module.hello_lambda.function_arn
    }
    "GET /world" = {
      lambda_function_arn = module.world_lambda.function_arn
    }
  }
}

‍

What to do about it: Give the new lambda-http-api-gateway module a try! (Released in version v0.19.0 of terraform-aws-lambda).

[NEW MODULE]: CloudWatch Logs to Slack

Motivation: CloudWatch Logs is great for aggregating logs, but sometimes you want to set up alerts to Slack when certain logs are posted from your apps. AWS does not provide any native integration with Slack, which forces you to implement your own.

Solution: We’ve added a new module for setting up a CloudWatch Log Group Subscription Filter against a Lambda function that will forward filtered logs to a Slack channel! This module allows you to configure a log filter for multiple Log Groups using the AWS native filter syntax.

What to do about it: Give the new log-filter-to-slack module a try! (Released in version v0.33.3 of terraform-aws-monitoring).

[NEW MODULE]: run-lambda-entrypoint Module

Motivation: In December 2020, AWS announced support for container based Lambda functions. Container based Lambda functions allow you to define your AWS Lambda functions using Docker container images, allowing you to use familiar tools to develop and manage your Lambda functions. However, one issue with container based Lambda (and AWS Lambda in general for that matter), is secrets management. Traditionally, your only choice for managing secrets with Lambda has been to either bake the logic to fetch secrets in the app, or to configure them as environment variables where they are recorded in plain text on the Lambda function metadata.

Solution: We’ve implemented a custom Lambda function entrypoint script for containerized functions that implement the logic of mapping Secrets Manager entries into runtime environment variables for the function! The run-lambda-entrypoint CLI will take in the ARN of a Secrets Manager entry via an environment variable SECRETS_MANAGER_ARN, and then automatically retrieve the secret and convert all the key-value pairs into environment variables for your Lambda function. In this way, the only configuration option that you need to set (and leak) is the ARN of the Secrets Manager entry, and not the actual secrets themselves.

Check out the new lambda-docker example to see an example of how to use the new entrypoint CLI.

What to do about it: Give the new run-lambda-entrypoint module a try! (Released in version v0.19.0 of terraform-aws-lambda).

Service Catalog Updates

terraform-aws-service-catalog

• v0.85.5: Added ability to bind a domain to database endpoints.
• v0.85.6: Exposed additional_security_group_ids which can be used to attach additional security groups to the lambda function when using VPC.
• v0.85.7: Added the ability to optionally create k8s PriorityClass resources in eks-core-services.
• v0.85.8: Exposed output for the CloudWatch Log Group name in lambda service. Exposed the ability to configure the Cluster Autoscaler log verbosity.
• v0.85.9: Updated for-production examples to the latest version of the Gruntwork Reference Architecture.
• v0.85.10: Added the ability to configure tags on the openvpn server module. Exposed variable auto_minor_version_upgrade in aurora module. Updated various dependencies — refer to the release notes for more details.
• v0.85.11: Updated tailscale packer template to support configuring the tailscale version; Updated core testing libraries (no impact on modules).
• v0.86.0: Exposed underlying lb_target_group_tags input in ecs-service module; Updated various dependencies — refer to the release notes for more details.
• v0.86.1: Exposed route table tagging variables in vpc module.
• v0.86.2: Added ability to create multiple subdomain records of different types for public zones in the route53 module.
• v0.87.0: Added support for Kubernetes 1.22. Fixed bug in multi region provider configuration which lead to extended periods of hanging. We recommend reviewing the providers.tf and terragrunt.hcl in our examples to get the latest version which sets the skip_get_ec2_platforms = false for opted out regions. Update examples to reflect latest best practices.
• v0.88.0: Updated Tailscale Subnet Router to not accept DNS. Tailscale recommends having AWS handle DNS configurations on EC2.
• v0.88.1: Added documentation of lb_listener authenticate_oidc options; Support enhanced monitoring in the rds module in service catalog.
• v0.88.2: Pass variables through for s3 object lock.
• v0.89.0: Added support for configuring error responses in Cloudfront for the public-static-website module. This also defaults to serving 404 error responses from a root document 404.html, and 500 responses from a root document 500.html; Updated various dependencies — refer to the Release Notes for more information.
• v0.89.1: Added the ability to configure the IMDS settings for EC2 Instances used as ECS workers in the ecs-cluster module. Exposed the ability to configure minimum protocol version for viewer certificates in the public-static-website module. Exposed the artifact_config variable in the ecs-deploy-runner module. Added ecr:ListImages to the list of IAM Permissions for read access in ecr-repos. Exposed the cors_rule variable in the public-static-website module. Updated dependencies.
• v0.89.2: Updated documentation for VPC filter in ec2-instance.pkr.hcl packer template. Added support for specifying the AWS Organizations and AWS Organizations Unit access for AMI access in all packer templates. Updated dependency terraform-aws-static-assets from v0.15.1 to v0.15.2.
• v0.89.3: Exposed parameter in public-static-websites to prevent perpetual diff in older AWS Accounts. Exposed forward_headers parameter in public-static-websites.
• v0.89.4: Updated dependency terraform-aws-security from v0.64.1 to v0.65.2.
• v0.90.0: Fixed bug where alb module may sometimes fail deploys due to race conditions in the S3 bucket. Updated the default version of Tailscale that is installed from v1.24.0 to v1.26.0. Exposed the ability to configure security response headers in the CloudFront distribution for the public-static-website module. Exposed the ability to link Lambda@Edge functions with the CloudFront distribution. Updated URL in documentation for AWS Load Balancer Controller Ingress Annotations. Updated dependencies - refer to the release notes for more info.
• v0.90.1: Updated eks-workers and eks-cluster to support log aggregation of server system logs (syslog and auth logs). This is different from the log aggregation managed by fluent-bit, which ships container level logs. All packer templates have been updated to require amazon plugin version at least 1.0.6.
• v0.90.2: Updated dependency terraform-aws-eks from v0.51.4 to v0.51.5. Exposed the ability to mirror tags on Managed Node Groups to the associated Auto Scaling Group.
• v0.90.3: Exposed variables to specify a custom s3 bucket for alb logs.
• v0.90.4: Exposed EKS Add-ons Variables. Update various dependencies — refer to the release notes for more details.
• v0.90.5: Locked version of helm provider to < 2.6.0 to work around an issue with the client authentication token. Updated the default version of tools installed in jenkins, and test dependencies.

Open Source Updates

Terratest

• v0.40.7: Fixed bug where the wrong ExpandQuery parameter was used for LoadBalancer module. Various enhancements to internal unit tests and documentation.
• v0.40.8: Fixed bug where FunctionError wasn't readable. Added new module with utility functions for validating Slack integrations.
• v0.40.9: Updated CopyTerraformFolderToDest to also copy .terraform.lock.hcl to the destination.
• v0.40.10: Added a new helper function version_checker.CheckVersion and version_checker.CheckVersionE which can be used to assert that installed tools satisfy a version constraint.
• v0.40.11: Added to HttpHelper functions that accept options structs. Fixed broken azure example.
• v0.40.12: Updated RenderTemplateE function to rebuild helm chart dependencies.
• v0.40.13: Updated dependency github.com/hashicorp/go-getter from 1.5.9 to 1.5.11
• v0.40.14: Updated GetResourceCount function to handle new plan output in Terraform 1.2.x.
• v0.40.15: Updated dependency hashicorp/go-getter from 1.5.11 to 1.6.1.
• v0.40.16: Added ShowWithStruct function that parses terraform plan output to PlanStruct.
• v0.40.17: Added aws.GetDefaultSubnetIDsForVpc to retrieve default subnet ids. Updated aws.IsPublicSubnet to check implicit association with the main route table in VPC when the subnet does not have any explicitly associated route tables.

kubergrunt

• v0.9.0: This adds support for Kubernetes version 1.22 in the sync-core-components command. In turn, support for Kubernetes 1.16, 1.17, and 1.18 has been dropped as they reached end of support in EKS.
• v0.9.1: Updated dependency aws-iam-authenticator so that v1beta1 versioned tokens can be generated.

git-xargs

• v0.0.15: This release enhances git-xargs with new functionality and flags to help you run git-xargs jobs that do not get rate limited in the first place, and to recover easily from rate limiting, all while honoring GitHub’s API guidelines and any slow-down headers GitHub may return. git-xargs will automatically retry pull requests that fail due to rate limiting with configurable back-off behavior.

cloud-nuke

• v0.11.8: This release adds a new non-destructive command, inspect-aws, which accepts many of the same flags as the nuke command. You can use it to explore your accounts and query for resources without fear of destroying anything. In addition, this release adds the ability to script against cloud-nuke as a library, which is useful in CI/CD jobs and in automation contexts. For more information, see Using cloud-nuke as a library.

Other updates

[NEW] terraform-aws-ci-steampipe

• v0.1.0: Initial release of Steampipe Runner for Gruntwork Pipelines (only available for customers with the CIS add-on.

terraform-aws-data-storage

• v0.23.3: Exposed restore_to_time parameter for point in time restore.
• v0.23.4: Updated ARNs to be partition-aware. Updated examples to use aws_subnets over aws_subnet_ids.
• v0.23.5: Updated backup-plan to attach S3 backup and restore policies to the Vault. Added the ability to specify custom access policies for the Backup Vault. This is useful for configuring cross account access.

terraform-aws-monitoring

• v0.33.3: Added new module for configuring a CloudWatch Log Group Subscription Filter that can stream filtered log entries to Slack.
• v0.33.4: Bumped internal dependencies and updated examples to be compatible with Terraform AWS Provider v4.
• v0.33.5: Similar to v0.33.4, and also exposed the ability to hook to the access logs S3 Bucket being fully configured in the logs/load-balancer-access-logs module. This is useful for ensuring the S3 Bucket configuration is set up to support linking to AWS ELB.
• v0.34.0: Functionally backward compatible upgrade that only requires that you run terraform init -upgrade to pull the latest Terraform AWS Provider. This release also adds a few upgrade tests to ensure backward compatibility.
• v0.34.1: In v0.34.0 we updated all but the logs/log-filter-to-slack module. This release makes sure that one gets attention!

terraform-aws-ci

• v0.47.3: Added retry logic in retrieving metadata of ECS tasks.
• v0.47.4: Updated ecs-deploy-runner to support repositories that has dockerfiles on the root of the repository.
• v0.47.5: Updated all places where ARNs are hardcoded to be partition-aware.
• v0.47.6: Fixed bug where the systemd file was unchanged for Jenkins, so all configurations were overwritten at boot time. Now we create a systemd override file so Jenkins uses the updated config setup at install time.
• v0.47.7: Updated infrastructure-deployer CLI to handle intermittent network connectivity errors when looking up the ECS task with retry logic.
• v0.47.8: Fixed regression where the logs from infrastructure-deployer became very chatty after v0.47.7.
• v0.47.9: Introduced new module sign-binary-helpers that can sign executable files for MacOS and Windows. Added new option --no-wait to infrastructure-deployer CLI. When passed in, it will instruct the infrastructure-deployer not to wait for the ECS task to finish and immediately exit without error.
• v0.47.10: Updated sign-binary utility to pass sensitive files through stdin; Updated the docker-image-builder component of ECS Deploy Runner to support assuming IAM roles for cross account docker image builds.
• v0.47.11: Moved --no-wait check to before waiting for ECS task to start. Now when you pass in --no-wait, the infrastructure-deployer will immediately exit after invoking the lambda function; Updated examples to be compatible with AWS Provider v4.
• v0.48.0: Removed some old tests that are no longer needed. Unlocked AWS provider v4. Require minimum 3.75.1. This update includes a few upgrade tests that make sure upgrading to this module from the last release is easy. However, you may need to bump your AWS provider version. See the migration guide.
• v0.48.1: Minor update, all related to testing module upgrades to make our builds more stable across Gruntwork’s IaC library!
• v0.48.2: Added a new go package for upgrade tests; Added the ability to install multiple terraform versions into the deploy-runner docker container by using the build arg additional_terraform_versions. Example: --build-arg additional_terraform_versions=0.12.31,0.15.1;Fixed bug where SSH key without trailing \n was being rejected by ssh-add when attempting to load into the deploy runner.
• v0.48.3: Made the repo name configurable in upgrade testing framework.
• v0.48.4: Updated upgrade testing framework to run apply -refresh-only to avoid terraform output changes causing resource counting to fail.
• v0.49.0: Updated dependency terraform-aws-asg from v0.13.0 to v0.18.0 in jenkins-server module.

terraform-aws-asg

• v0.18.0: Support for python2 has been dropped. All modules that depend on python now require python 3, and calls out to python3 directly. Most users should not be impacted by this change, as almost all operating systems ship with python3 now.

terraform-aws-security

• v0.64.0: Converted the previously inline IAM policies to managed IAM policies in a few different modules. Managed policies are recommended best practice by AWS and are friendlier with security compliance checkers.
• v0.64.1: Exposed the ability to configure the condition operator for GitHub Actions IAM role. This allows you to construct an IAM role that can be assumed by any repo in a particular org.
• v0.65.0: Unlock AWS provider v4. Require minimum 3.75.1. This update includes a few upgrade tests that make sure upgrading to this module from the last release is easy. However, you may need to bump your AWS provider version. See the migration guide notes below for more.
• v0.65.1: Updated to ignore changes to various S3 configuration.
• v0.65.2: Updated modules that creates IAM roles to expose the ability to set permission boundaries.
• v0.65.3: Added . to all permissions boundary variable descriptions. Addressed deprecation warning for object_lock_configuration on private s3 bucket. Added object_lock_configuration to the lifecycle ignore_changes to avoid perpetual diff.
• v0.65.4: Added a new output to private-s3-bucket that can be used to chain resources to the bucket being fully configured without using module depends_on (which has quirks that can lead to perpetual diffs). The primary use case would be when you are configuring an object upload in the same module that is creating the bucket, you would want to make sure all the configuration options are set on the bucket before uploading the first object.
• v0.65.5: Added option to enable delete_marker_replication in the replication config.
• v0.65.6: Removed hard-coded AWS Partitions from ARNs. Now the partition is dynamically computed based on the configured provider.

terraform-aws-eks

• v0.51.0: The default version of Kubernetes installed by the module has been updated to 1.22. As a result of this, the default version of addons were updated to support installation into 1.22. Refer to the release notes for more details.
• v0.51.1: Exposed the condition operator for service account selection as a configurable parameter in eks-iam-role-assume-role-policy-for-service-account.
• v0.51.2: Exposed advanced external-dns parameters to tweak syncing behavior. These parameters are useful for avoiding the Route 53 API limits. Refer to the new README section for more details.
• v0.51.3: Fix issue with autoscaler priority expander ConfigMap not rendered properly.
• v0.51.4: Updated package dependencies of eks-aws-auth-merger.
• v0.51.5: Added the ability to mirror the tags applied to Managed Node Groups to the underlying ASGs. Note that this feature depends on an assumption that there is only one ASG per MNG to work around an issue with Terraform for_each and count. If your environment has more than one ASG mapped to the MNG, then it is recommended to call the aws_autoscaling_group_tag resource outside the eks-cluster-managed-workers module.
• v0.51.6: Added better support for Windows to the local-exec calls in the eks-cluster-control-plane module.

terraform-aws-ecs

• v0.33.0: Support for python2.7 has been dropped from the modules where python was being used. You must have python3.5 or greater installed on the operator machine (where terraform is being called), and the python3 executable must be available on your PATH.
• v0.33.1: Added the ability to configure http_put_response_hop_limit on the metadata configuration.

terraform-aws-lambda

• v0.19.0: Added two new modules (lambda-http-api-gateway and run-lambda-entrypoint). Refer to the release notes for more details.
• v0.19.1: Updated release pipeline to build and publish run-lambda-entrypoint.
• v0.19.2: Added the ability to specify the architecture (x86_64 or arm64) for the lambda function.
• v0.19.3: Updated run-lambda-entrypoint to support wrapping the entrypoint call in the Runtime Interface Emulator for local testing. Note that you only need to use this feature if you are using a distroless container image for the Lambda function.

terraform-aws-static-assets

• v0.15.1: Updated dependency terraform-aws-security to v0.65.2.
• v0.15.2: Fixed bug where the S3 bucket configuration flip flopped due to missing lifecycle ignore rules.
• v0.15.3: Exposed the response_headers_policy_id attribute of the aws_cloudfront_distribution resource so that one could attach a custom response header policy to the CloudFront Distribution.
• v0.15.4: Fixed an outdated reference in a variable description. In the s3-static-website module, we fixed a bug in how routing_rule is parsed.
• v0.15.5: Added a new output to indicate the S3 Bucket is fully configured. This is useful for ensuring the Bucket is ready to link to CloudFront prior to setting up the CloudFront Distribution.

terraform-aws-openvpn

• v0.23.1: Updated openvpn-server to support running in various AWS partitions (e.g. GovCloud and other private partitions).
• v0.24.0: Unlocked AWS provider v4, requiring minimum 3.75.1. This update includes a few upgrade tests that make sure upgrading to this module from the last release is easy. However, you may need to bump your AWS provider version. See the migration guide.
• v0.24.1: Allowed specifying a prefix for the openvpn-server backup bucket server logs.

terraform-aws-utilities

• v0.9.0: Unlocked AWS provider v4, requiring minimum 3.75.1. This update includes a few upgrade tests that make sure upgrading to this module from the last release is easy. However, you may need to bump your AWS provider version. See the migration guide.

terraform-aws-vpc

• v0.22.0: Unlocked AWS provider v4, requiring minimum 3.75.1. This update includes a few upgrade tests that make sure upgrading to this module from the last release is easy. However, you may need to bump your AWS provider version. See the migration guide.

terraform-aws-asg

• v0.19.0: Unlocked AWS provider v4, requiring minimum 3.75.1. This update includes a few upgrade tests that make sure upgrading to this module from the last release is easy. However, you may need to bump your AWS provider version. See the migration guide.

DevOps News

Terraform 1.2 is out

What happened: HashiCorp has released Terraform 1.2.

Why it matters: Terraform 1.2 introduces several new features, including some improvements when using the CLI with Terraform Cloud and support for precondition and postcondition blocks. precondition blocks can be used to validate assumptions before running apply, such as checking that the EC2 instance type a user selected is in the AWS Free Tier; postcondition blocks can be used to enforce guarantees after running apply, such as validating an EBS volume you just created is encrypted.

What to do about it: We will start updating all our tests to use Terraform 1.2 in the next few months to ensure all our modules are compatible with it. Except for a few use cases detailed here, Terraform 1.2 should be fully backwards compatible, so it is most likely safe to start experimenting with it even now.