aws

Gruntwork Newsletter, December 2017

Once a month, we send out a newsletter to all Gruntwork customers that describes all the updates we’ve made in the last month, news in the…
Gruntwork Newsletter, December 2017
YB
Yevgeniy Brikman
Co-Founder
Published November 6, 2017

Once a month, we send out a newsletter to all Gruntwork customers that describes all the updates we’ve made in the last month, news in the DevOps industry, and important security updates. Note that many of the links below go to private repos in the Gruntwork Infrastructure as Code Library and Reference Architecture that are only accessible to customers.

Hello Grunts,

Happy holidays! In the last month, we launched a Gruntwork Community Forum, created a Gruntwork CLI tool, and made important changes and fixes in OpenVPN, MongoDB, Terragrunt, and our code formatting. There were also some very exciting announcements from AWS (including managed Kubernetes, Aurora multi-master and serverless, managed GraphQL, and much more) and a new version of Terraform came out that is backwards incompatible. Read on to learn more!

As always, if you have any questions or need help, email us at support@gruntwork.io!

Gruntwork Updates

Gruntwork Community Forum

Motivation: All of our customers are using the same infrastructure, so we see the same questions come up time and again, but did not have a way to expose those repeated discussions to everyone.

Solution: We’ve launched the Gruntwork Community Forum as a place our customers can use for troubleshooting, design discussions, new infrastructure requests, and to discuss the Infrastructure as Code Library, Reference Architecture, Terraform, AWS, Docker, Packer, DevOps, etc.

What to do about it: Please head over to https://community.gruntwork.io/ and try it out for all your future questions! We will gently nudge any support question that looks like it could affect multiple customers towards the forum, but don’t worry, Gruntwork employees will be monitoring the forum regularly, and will respond with the same SLA as as always; it’s just that now, you’ll also be able to find answers from the community and via Google search! And, of course, you can still email us at support@gruntwork.io at any time!

Gruntwork CLI

Motivation: Many of our customers had to perform similar actions manually, such as creating the initial AWS accounts and Git repos for the Reference Architecture or granting Gruntwork employees access to your accounts for troubleshooting. These manual actions were tedious and error prone.

Solution: We’ve created a new cross-platform CLI tool to automate these actions. It’s called, not very creatively, gruntwork! It has commands built-in to create the AWS accounts and Git repos for the Reference Architecture, which is useful for new customers, as well as commands to grant Gruntwork employees access to your AWS accounts and Git repos, which is useful for all customers, especially when troubleshooting. Instead of creating an IAM user for each Gruntwork employee (which means juggling lots of usernames, passwords, API keys, etc), you can now give Gruntwork’s AWS account access to your AWS account, and we’ll be able to “switch” into your account with our existing credentials. For example, if you were troubleshooting an issue in your “prod” AWS account, a user in your “master” account could grant us read-only access with a single command:

gruntwork aws grant --managed-policy ReadOnlyAccess --account prod

When we were done troubleshooting, you could revoke all access just as easily:

gruntwork aws revoke --account prod

What to do about it:**** Try out the gruntwork CLI next time you need to give us access! Also, if Gruntwork employees already have IAM users in your AWS accounts, we recommend removing those IAM users, and switching to this model, as it’s more secure and easier to use.

Workaround for RDS and ElastiCache parameter_group bug

Motivation: A strange bug showed up recently where the parameter_group configuration used in ElastiCache and RDS would always show a diff when you ran plan. Worse yet, if you ran apply, you’d sometimes get an error trying to apply that phantom diff.

Solution: This appears to be a bug in Terraform (terraform-providers/terraform-provider-aws#2468), so for now, we’ve put in a workaround in both module-cache and module-data-storage where we set the default parameter_group value explicitly so there is no phantom diff.

What to do about it: Update to module-cache, v0.3.1 and module-data-storage, v0.4.0.

MongoDB improvements

Motivation: Creating a single module for MongoDB that worked well in both dev and prod was tricky, as you’d want to disable some clusters (e.g., backup) in dev, but not prod, and there’s no way to conditionally disable a module in Terraform. Also, the install-mongodb module broke due to a strange error with pip.

Solution: We’ve released a couple new versions of package-mongodb: v0.2.4 removes an earlierpip workaround, which should fix the install-mongodb module; v0.2.5 allows you to set the cluster_size parameter of mongodb-cluster to 0 to disable that module from creating any resources.

What to do about it: Upgrade to package-mongodb, v0.2.5 to take advantage of the new features/fixes.

OpenVPN fixes and improvements

Motivation: We discovered an issue where the Certificate Revocation List (CRL) in OpenVPN would expire after 30 days of not performing any revocations. This condition would prevent users from being able to connect to the VPN.

Solution: We’ve released version v0.5.0 of package-openvpn with a fix that sets the default CRL expiration date to 10 years in the future. We also added a new backup-openvpn-pki module and corresponding cron job to regularly backup the entire PKI infrastructure to the S3 bucket.

What to do about it: Upgrade to package-openvpn, v0.5.0 to take advantage of the new features/fixes.

Terraform fmt

Motivation: We were not formatting our Terraform code in a consistent manner, which made it harder to read.

Solution: We are updating each of our repos to (a) use a pre-commit hook (managed using pre-commit) to automatically run terraform fmt on the code and (b) fail the build if the code isn’t formatted. Best of all, this new functionality is available as part of the new precommit-hooks module, so you can enforce the same formatting on your own code!

What to do about it: Check out the precommit-hooks module for instructions.

Winter break

Motivation: We want to spend time with our families during the holidays.

Solution: The Gruntwork team will be on vacation December 25 — January 5. We will not be responding to support tickets or Slack inquiries during this time. Please plan accordingly.

What to do about it: We hope you’re able to take some time off and enjoy the holidays too!

Why prod is down

Motivation: Prod is down and you need to let your team know why.

Solution: Why Prod is Down.

What to do about it: Share it with your team!

Terragrunt improvements

We built Terragrunt to try to fill in gaps and weaknesses in Terraform. It’s a popular open source project, so this section will list the updates that happen each month.

  • v0.13.18: Fix a data race that could happen when you hit CTRL+C.
  • v0.13.19: Fix a concurrent write bug that would occasionally happen with the xxx-all commands (e.g., apply-all).
  • v0.13.20: You can now use the TF_INPUT environment variable to set Terragrunt to non-interactive mode. This is the same environment variable used by Terraform.
  • v0.13.21: The getAWSAccountId() helper now assumes the IAM role specified via --terragrunt-iam-role, if any.
  • v0.13.22: Fix a crash that could happen with local backends.

Other fixes

  • module-security, v0.6.3: You can now disable the full-access IAM group in the iam-groups module via the should_create_iam_group_full_access parameter. The iam-groups module now also outputs the ARNs and names of the ssh-iam groups.
  • module-security, v0.6.4: ssh-iam will now do a better job of reporting error messages if it reads an OS user that is missing the "comment field", which ssh-iam uses for storing the IAM user name.
  • package-zookeeper, v0.1.0: Update to ZooKeeper 3.4.11, as 3.4.9 was mysteriously deleted. Update to Oracle JDK 8u152, as 8u144 was mysteriously deleted.
  • package-kafka, v0.1.0: Update to the latest package-zookeeper.
  • gruntwork-installer, v0.0.20: The bootstrap script now supports 32-bit, 686 architecture.
  • module-load-balancer, v0.7.0: We’ve converted the https_listener_ports_and_ssl_certs and https_listener_ports_and_acm_ssl_certs input variables on the alb module from maps to lists of maps. The problem with using maps is that in Go—which Terraform uses under the hood—the iteration order for maps is (intentionally) randomized, so with multiple ports and certs in these variables, you would get spurious diffs in the plan command as Terraform would swap their order and assume listeners had to created/destroyed. By using lists, we can make the sort order consistent.
  • module-server, v0.2.2: You can now configure the source_dest_check param on the single-server module.

DevOps News

Terraform 0.11 and backwards incompatibility

What happened: Terraform 0.11 has been released. The major changes are:

  1. Improved integration with the Terraform Registry, including first-class support for versioning.
  2. You can configure providers on a per-module basis.
  3. The apply command now automatically runs plan and asks you to interactively approve the plan.
  4. If you use “singleton” syntax (e.g., TYPE.NAME.ATTR instead of TYPE.NAME.*.ATTR) on a resource with count, you will now get a runtime error if that count parameter is set to anything other than 1.

Items (3) and (4) are backwards incompatible (see the upgrade guide)!

Why it matters: If you tried to upgrade to Terraform 0.11 right now, you’d run into two issues:

  1. Since the apply command is now interactive, the automated deployment scripts in your CI servers will now be broken. You’ll have to add the -auto-approve flag to the apply command to make it non-interactive.
  2. A lot of Terraform code uses the singleton syntax with resources that have a count parameter, so many of the modules in your codebase and in the gruntwork-io repos do not work with Terraform 0.11! Worse yet, the errors only manifest at runtime and are hard to track down. We filed a bug about this and fortunately, HashiCorp released Terraform 0.11.1, which hasthe ability to issue a warning for all potentially invalid usages of the singleton syntax. This will make it possible for us to find all these errors and fix them.

What to do about it: We do NOT recommend upgrading to Terraform 0.11 at this time. We’ll start updating all of our modules next week to be compatible with 0.11, but it will take some time. If you absolutely must use 0.11 in the meantime, as a workaround, you can install Terraform 0.11.1 and set the TF_WARN_OUTPUT_ERRORS environment variable to 1. This will force Terraform to fall back to the old behavior with count (see 0.11.1 release notes for details).

AWS re:Invent

AWS had its annual re:Invent conference and announced a ton of new services, features, and improvements.

There are way more announcements than we can cover in this newsletter, so we’ll write a full blog post about this next week. In the meantime, here are a few of the highlights:

  • **Amazon Elastic Container Service for Kubernetes (EKS)**: A managed Kubernetes service on AWS! This allows you to use the highly popular, open source Docker orchestration tool without having to manage the cluster yourself. This could be a very compelling alternative to Amazon’s EC2 Container Service (ECS).
  • AWS Fargate: Although everyone has been waiting for EKS for a long time, Fargate may be an even more intriguing option, as it allows you to run Docker containers on AWS without worrying about clusters or servers at all! It’s a “serverless” offering similar to Lambda, except instead of deploying individual functions, you can deploy entire Docker containers, and there are no time limits.
  • Lambda improvements: AWS announced a number of improvements to Lambda, including doubling the maximum memory capacity, allowing you to control concurrency limits for lambda functions, adding support for shifting traffic between old and new versions of a lambda function, launching a marketplace for lambda functions called the Serverless Application Repository, and launching an online IDE for lambda functions called AWS Cloud9.
  • Aurora improvements: Aurora will add support for a multi-master configuration, allowing you to scale both reads and writes across multiple servers, as well as Aurora serverless, which allows you to use Aurora without having to manage any database instances at all, and to instead have the database start up, shut down, scale up, and scale down automatically based on your usage.
  • API Gateway improvements:**** You can now create private API Gateway endpoints that are only accessible within your VPC and do canary deployments of API Gateway changes.
  • Amazon Neptune: A managed, highly-available graph database.
  • Amazon MQ: A managed ActiveMQ .
  • Amazon AppSync: A managed GraphQL service, with real-time data sync and offline functionality.
  • Amazon TimeSync: Amazon’s replacement for NTP that is natively accessible from EC2 Instances that does a better job of handling certain types of errors (e.g., leap seconds). It works with standard NTP clients and does not cost anything extra.
  • Inter-region VPC peering: You can finally use VPC peering to connect VPCs in different AWS regions!
  • EC2 T2 Unlimited: You can now use T2 unlimited instances, which have the same, cheap baseline CPU performance, but allow you to “burst” above this baseline for as long as you need (i.e., instead of having a limited number of burst credits that could run out).

Security Updates

Below is a list of security updates that may impact your services. It is up to you to scan this list and decide which of these apply and what to do about them. Note that for critical security updates, we send out emails immediately to the Gruntwork Security Alerts list.

Python

  • USN-3496–1, USN-3496–2: A number of security vulnerabilities were found in Python, including some that may allow arbitrary code execution. We strongly recommend updating your Python version.

Bind

  • USN-3346–3: Bind could be made to serve incorrect information or expose sensitive information over the network.

Go

  • CVE-2017–15041, CVE-2017–15042: Several vulnerabilities have been discovered in Go, including one that could allow arbitrary code execution if yougo get a malicious repository.

Httpd

  • CVE-2017–12171: A regression was found in the Red Hat Enterprise Linux 6.9 version of httpd, causing comments in the “Allow” and “Deny” configuration lines to be parsed incorrectly. A web administrator could unintentionally allow any client to access a restricted HTTP resource.

OpenJDK

Jenkins

Linux Kernel

OpenSSL

  • advisory: A number of security vulnerabilities were found in OpenSSL, including several that allow arbitrary code execution.

Perl

MySQL

  • ALAS-2017-927, ALAS-2017-926: Vulnerability in the MySQL Server component of Oracle MySQL (subcomponent: Client programs). Easily exploitable vulnerability allows low privileged attacker with network access via multiple protocols to compromise MySQL Server. Successful attacks of this vulnerability can result in unauthorized access to critical data or complete access to all MySQL Server accessible data.

Postgres

Procmail

Ruby

  • DSA-4031: A number of security vulnerabilities were found in Ruby

Samba

  • usn-3426–1, usn-3486-2, dsa 4043–1, usn-3486–1: A number of security vulnerabilities were found in Samba, including several that allow man in the middle attacks or remote attacker could possibly use this issue to obtain server memory contents.

Got questions or feedback on the newsletter? Email us at support@gruntwork.io. Want to learn more about Gruntwork? Check out https://www.gruntwork.io/.