Gruntwork Newsletter, June 2018

Once a month, we send out a newsletter to all Gruntwork customers that describes all the updates we’ve made in the last month, news in the DevOps industry, and important security updates. Note that many of the links below go to private repos in the Gruntwork Infrastructure as Code Library and Reference Architecture that are only accessible to customers.

Hello Grunts,

Last month, we created and open sourced modules for running Couchbase in AWS, updated our security code to make it possible to assume IAM Roles for a longer time, added migration and deployment docs for the Reference Architecture, added a keep-warm module to fix the “cold start” problem with Lambda functions, and fixed a bunch of bugs. In other news, Amazon EKS is now generally available and Microsoft is acquiring GitHub.

As always, if you have any questions or need help, email us at support@gruntwork.io!

• Gruntwork Updates
• DevOps News
• Security Updates

Gruntwork Updates

New modules: Couchbase for AWS

Motivation: The Couchbase team wanted to make it 10x easier for users to get up and running with Couchbase on AWS.

Solution: We created the terraform-aws-couchbase module, which is open source on GitHub, and available in the Terraform Registry! This module makes it easy to spin up a Couchbase cluster in just a few lines of Terraform code:

Copy
provider "aws" {
region = "us-east-1"
}

Copy
module "couchbase" {
source  = "gruntwork-io/couchbase/aws"
version = "0.1.0"
}

Copy
output "couchbase_web_console_url" {
value = "${module.couchbase.couchbase_web_console_url}"
}

Copy
output "sync_gateway_url" {
value = "${module.couchbase.sync_gateway_url}"
}

Put the code above into a file called main.tf, run terraform init and terraform apply, and in a few minutes, you’ll have a Couchbase cluster running in AWS!

For more info and examples, check out Get Couchbase running on AWS in 5 minutes.

What to do: Give the Couchbase modules a try and let us know how they work for you!

Assume IAM Roles for a longer period of time!

Motivation: Many of our customers assume IAM Roles to access their AWS accounts. The default max expiration time for IAM Role credentials used to be just 1 hour, so you’d have to login again and again throughout the day. AWS has finally added support for configuring the max expiration time for IAM role credentials and our customers wanted to be able to take advantage of that.

Solution: We made two changes in module-security:

1. The cross-account-iam-roles module now sets a default max expiration of 12 hours for IAM Roles intended for human users (e.g., allow-read-only-access-from-other-accounts) and a default max expiration of 1 hour for IAM Roles intended for machine users (e.g., allow-auto-deploy-access-from-other-accounts). Both of these expiration values are configurable via the new input variables max_session_duration_human_users and max_session_duration_machine_users.
2. The aws-auth script now accepts optional --mfa-duration-seconds and --role-duration-seconds parameters that specify the session expiration for the creds you get back when authenticating with an MFA token or assuming an IAM role, respectively. The default for both of these has been set to 12 hours to be more human-friendly.

What to do about it: If you want to take advantage of these more friendly IAM Role expiration periods, update your AWS accounts to use module-security, v0.10.0, install the latest aws-auth, and enjoy all the time you’ve saved by not having to re-authenticate 8 times per day!

Migration and deployment docs for the Reference Architecture

Motivation: After deploying the Reference Architecture for our customers, many asked us for documentation on how to migrate their apps to it, and how to deploy it from scratch.

Solution: We’ve added the docs! All future Reference Architecture deployments will include detailed migration and deployment instructions. For existing customers, you can find the same docs as part of the Acme sample Reference Architecture repos. For example, here are the ones from the multi-account Acme Reference Architecture:

• Migrating to the Reference Architecture
• Deploying the Reference Architecture from scratch

What to do about it: We hope you find the new docs useful! If anything is missing or unclear, please let us know by emailing support@gruntwork.io, and we’ll improve it!

New module to fix the “cold start” problem with AWS Lambda

Motivation: Building infrastructure on top of AWS Lambda is great, as you don’t have to manage servers or worry much about scalability or high availability. However, if you have a Lambda function where latency is important (e.g., you’re using Lambda + API Gateway for a web service), and that function is not invoked for a while, the next invocation may be hit with the overhead of a “cold start,” adding from a few hundred milliseconds all the way up to 15 extra seconds to that function’s execution time!

Solution: We created a new module called keep-warm which you can use to invoke your Lambda functions on a scheduled basis, effectively keeping those functions “warm” so that you can avoid cold starts. For example, here’s how you can invoke a Lambda function called example-func every 5 minutes, passing it the event object {"foo": "bar"} on each invocation:

Copy
module "keep_warm" {
source = "git::git@github.com:gruntwork-io/package-lambda.git//modules/keep-warm?ref=v0.2.2"

Copy
name                = "keep-warm"
schedule_expression = "rate(5 minutes)"
concurrency         = 3

Copy
function_to_event_map = {
example-func = {
foo = "bar"
}
}
}

Note that the example above sets the concurrency level to 3. This tells the keep-warm module to invoke the example-func Lambda function 3 times concurrently each time it runs, thereby keeping 3 containers warm at the same time, rather than just one. See I’m afraid you’re thinking about AWS Lambda cold starts all wrong for more info on why concurrency is important.

What to do about it: If you have latency-sensitive Lambda functions (e.g., web services built with API Gateway and Lambda) with spiky, intermittent traffic, give the keep-warm module a shot for avoiding the cold start problem.

IAM Group Name fix in the Reference Architecture

Motivation: One of our customers was trying to assume an IAM Role in the Reference Architecture that allows them to request an OpenVPN Cert in another AWS account, but due to a bug, it wasn’t working.

Solution: We found a typo in the IAM Group names the Reference Architecture creates for using OpenVPN in other accounts. The IAM Role name was set to openvpn-server-allow-certificate-requests-for-external-accounts instead of openvpn-allow-certificate-requests-for-external-accounts.

What to do about it: If you are using the multi-account Reference Architecture, fix the IAM Role name in security/_global/iam-groups/terraform.tfvars. We’ve updated the Acme Multi-Account example with the fix, so you can use this commit as a reference.

Open source updates

• Terratest, v0.9.3: Added new methods for authenticating to AWS with an IAM role: NewAuthenticatedSessionFromRole and CreateAwsSessionFromRole.
• Terratest, v0.9.4: Fix a bug where the retry.DoWithRetry methods were not returning stdout/stderr. Fix a bug where the retry mechanism in terraform.Apply was checking the error message rather than stdout/stderr.
• Terratest, v0.9.5: Fixed a bug where test_structure.CopyTerraformFolderToTemp would fail if called from inside of a subtest, such as t.Run("SubtestcaseName", func(t *testing.T) {...})
• Terratest, v0.9.6: You can now set the Upgrade flag in terraform.Options to tell Terratest to set the -upgradeflag when running terraform init.
• pre-commit: We’ve moved our pre-commit hooks to a separate open source repo so that we can use those hooks with our open source projects. You can find the hooks here: https://github.com/gruntwork-io/pre-commit.
• terraform-aws-vault, v0.7.0: The vault-cluster module now requires a new parameter called allowed_inbound_security_group_count to be set. This is unfortunately required to work around a limitation in Terraform with the count parameter.
• terraform-aws-vault, v0.7.1: Vault will now exit with an error if cluster_addr == api_addr, so default api_addr to a different value, and make it configurable via a new --api-addr parameter.
• terraform-aws-vault, v0.7.2: Fix typo in the docs where we listed the parameter name as --s3-backend-enable but it should actually be --enable-s3-backend.
• terraform-aws-consul, v0.3.5: You can now use the --environment option with the run-consul script to set environment variables that should be exposed to Consul in its supervisord config.
• terraform-aws-nomad, v0.4.2: You can now use the --environment option with the run-nomad script to set environment variables that should be exposed to Nomad in its supervisord config.

Other fixes

• module-asg, v0.6.10: The server-group module now cleans up properly if a single server fails to pass health checks, exposes new vars to configure the rolling deploy timeout and add additional security groups, and updates to the latest version of Boto.
• module-asg, v0.6.11: Fix an intermittent bug in asg-rolling-deploy that would cause the error argument --tag-value: expected one argument.
• module-asg, v0.6.12: You can now specify a kms_key_id parameter for the server-group module to enable EBS Volume encryption with your own Customer Master Key (CMK). This will also automatically create an IAM Policy that gives the server access to that CMK.
• module-ci, v0.8.0: All the pre-commit hooks that were in modules/pre-commit are now in their own open source repo: https://github.com/gruntwork-io/pre-commit. Please update your .pre-commit-config.yml files to point to the new repo and its version numbers.
• module-ci, v0.9.0: git-add-commit-push has been moved from the gruntwork-module-circleci-helpers module to the git-helpers module. terraform-update-variable now depends on git-helpers being installed, as it uses git-add-commit-push under the hood to be able to more reliably commit and push changes.
• module-ci, v0.10.0: The git-add-commit-push script no longer defaults the branch name to $CIRCLE_BRANCH. Instead, it uses git to look up the name of the currently checked-out branch in pwd.
• module-data-storage, v0.6.5: In the aurora module, you can now use the db_instance_parameter_group_name param to set the parameter group for instances separately from the parameter group for the entire cluster (which can be set via the db_cluster_parameter_group_name param).
• module-data-storage, v0.6.6: The rds module now exposes a publicly_accessible parameter that you can set to true to make the DB accessible from the public Internet (NOT recommended for most use cases).
• module-load-balancer, v0.9.0: The nlb module now exposes a subnet_mapping_size parameter that you use to specify how many items there are in the subnet_mapping list. This fixes the issue that arises when using resources or module outputs as values in the maps within the subnet_mapping list.
• module-security, v0.9.0: The auto-update, ntp, fail2ban, and ip-lockdown modules now all use bash-commons under the hood. That means you must install bash-commons before installing any of those other modules. As a result, the auto-update and ntp modules now support Amazon Linux 2. We will add Amazon Linux 2 support for fail2ban and ip-lockdown modules in the future.
• module-ecs, v0.6.7: You can now optionally customize the ALB Target Group name in ecs-service-with-alb using the alb_target_group_name parameter.

DevOps News

Amazon EKS is now generally available

What happened: Amazon Elastic Kubernetes Service (EKS) is now generally available in us-east-1 and us-west-2. Moreover, HashiCorp has already launched Terraform support for EKS.

Why it matters: Running Kubernetes yourself is a significant undertaking, so being able to take advantage of a managed service that does it for you can be a huge win. Up until now, EKS was only available in early preview mode, but now everyone should be able to use it. However, it’s still very early days for EKS, and there are a number of limitations to be aware of: it costs a minimum of $150/month just for the master nodes, it’s only available in us-east-1 and us-west-2, the provisioning process is strange (using CloudFormation under the hood), PrivateLink is not supported yet, HIPAA compliance is not supported yet, no integration with Fargate yet, and so on (see reddit and HN for discussion). AWS will be fixing all of these in the future, but for now, buyer beware!

What to do about it: We will be diving deep into EKS over the next month or two and will report back on what we find. In the meantime, feel free to play around with it and let us know what you think!

Microsoft is acquiring GitHub

What happened: Microsoft has agreed to acquire GitHub for $7.5 billion.

Why it matters: A huge congrats to our friends at GitHub! We love the product, use it for all of our private and open source projects, and we are excited to see what the future holds.

What to do about it: Keep using GitHub and pushing Microsoft to make it even better.

Amazon ECS Service Discovery Supports Bridge and Host Container Networking Modes

What happened: You can now use ECS Service Discovery with ECS tasks that use the bridge and host networking modes.

Why it matters: Previously, ECS Service Discovery only supported tasks launched using the awsvpc networking mode, which assigns each running task a dedicated elastic networking interface.

What to do about it: Try it out on all your ECS Tasks! We’ll be adding support for it to module-ecs soon.

Amazon Aurora Backtrack

What happened: Amazon Aurora now has a new new “backtrack” feature that lets you roll back the database to any point in time, down to the granularity of seconds.

Why it matters: This feature is essentially an “undo” for when you screw something up in your database, such as deleting or corrupting critical data. Under the hood, Aurora stores every change to your database in a distributed, log-structured system, and is able to rollback to any previous log entry very quickly.

What to do about it: If you’re using Aurora, consider enabling this feature as an extra layer of protection (see the docs for more info). This functionality is now supported by Terraform and we will be updating module-data-storage with support for it soon.

ALB Authentication

What happened: The Application Load Balancer (ALB) now supports authentication of user requests when accessing web-based applications through social Identity Providers such as Facebook, Google, and Amazon or enterprise Identity Providers such as ActiveDirectory and PingFederate.

Why it matters: You can now offload authentication logic to the ALB and and Amazon Cognito instead of building it yourself. This allows your users to authenticate through any identity provider that supports OpenID Connect protocol or SAML2.0 or social identity providers such as Facebook, Google, and Amazon.

What to do about it: If you’d like to try out ALB user authentication, check out this blog post for a guide.

More CPU control for EC2 Instances

What happened: Amazon EC2 now allows you to control not only the instance type (e.g., m4.large), but also to customize how many vCPUs and how many threads per vCPU the instance will have access to.

Why it matters: You now have more flexibility and control over how many CPU cores and threads per CPU core your EC2 instances can access.

What to do about it: Check out the Optimize CPU docs for details.

Terraform FileSystem Provider

What happened: Terraform now has a new filesystem provider that provides a cross-platform way to read and write files.

Why it matters: Before, you could only use the file interpolation function, which only worked for files that already existed, and was only useful for reading files. With the filesystem_xxx resources, you can read and write files on the fly, create directories, set file permissions, and more.

What to do about it: Check out the terraform-provider-filesystem docs for details.

Security Updates

Below is a list of critical security updates that may impact your services. We notify Gruntwork customers of these vulnerabilities as soon as we know of them via the Gruntwork Security Alerts mailing list. It is up to you to scan this list and decide which of these apply and what to do about them, but most of these are severe vulnerabilities, and we recommend patching them ASAP.

Routers

• VPNFilter: A malware known as VPNFilter has infected hundreds of thousands of routers around the world. Many types of Linksys, Netgear, QNAP, and TP-Link routers are vulnerable. Check this article for the full list and if you’re using any of them in the office or at home, upgrade immediately!

PGP

• EFAIL: EFAIL attacks exploit vulnerabilities in the OpenPGP and S/MIME standards to reveal the plaintext of encrypted emails. If you are using PGP anywhere, you should try to update or migrate immediately. In particular, you should no longer decrypt PGP within any email client, and Keybase recommends removing any PGP keys you had associated with your Keybase profile.