Gruntwork Newsletter, May 2018

Once a month, we send out a newsletter to all Gruntwork customers that describes all the updates we’ve made in the last month, news in the DevOps industry, and important security updates. Note that many of the links below go to private repos in the Gruntwork Infrastructure as Code Library and Reference Architecture that are only accessible to customers.

Hello Grunts,

April was a busy month! We refactored and open sourced our infrastructure testing library Terratest, created a new module called ip-lockdown that makes IAM role usage much more secure, created a new module for deploying Docker images using AWS Fargate, added a module for deploying the NLB, open sourced our collection of reusable Bash scripts in a repo called bash-commons, and made many other changes, fixes, and updates. In other news, AWS has launched two new services—Private Certificate Authority and Secrets Manager—and Russia has blocked AWS and GCP. Read on to learn more!

As always, if you have any questions or need help, email us at support@gruntwork.io!

• Gruntwork Updates
• DevOps News
• Security Updates

Gruntwork Updates

Terratest is open source!

Motivation: At Gruntwork, we maintain a library of over 250,000 lines of infrastructure code that’s used in production by hundreds of customers. To make it possible for our small team to maintain all of this code, we developed a Go library called Terratest, which provides a variety of tools that make it easier to write automated tests for Terraform, Packer, Docker, AWS, etc. Terratest has grown organically over the last few years as we figured out best practices for infrastructure testing, so we wanted to clean up the design and improve the documentation so everyone else could use it.

Solution: We have refactored Terratest, cleaned up the APIs, added lots of example code and documentation, and released it as an open source project on GitHub: https://github.com/gruntwork-io/terratest.

What to do about it: Read the Terratest announcement blog post, check out all the examples and docs, and start using Terratest to create automated tests for your infrastructure!

New module: ip-lockdown

Motivation:**** Every EC2 Instance has an EC2 Metadata endpoint (169.254.169.254) that gives you the credentials you need to assume any IAM role associated with that EC2 instance, and thereby, get all the permissions available in that IAM role. By default, that endpoint is accessible to anyone on the server, so we wanted a way to lock it down, ensuring that if a hacker gets access to the server, they can only make use of the IAM role if they have access to privileged user accounts.

Solution: We created a new module called [ip-lockdown](https://github.com/gruntwork-io/module-security/tree/master/modules/ip-lockdown) that can lock down IP addresses on a Linux server such that only specific OS users (say, only root) can access them. For example, you can use ip-lockdown to lock down the EC2 metadata endpoint to just the root user with the following command:

Copy
ip-lockdown 169.254.169.254 root

Now, if an attacker gets on your server, to be able to use the permissions in the IAM role, the attacker would also need to find a way to get root/sudo privileges, which significantly improves your security posture.

What to do: We strongly recommend using ip-lockdown on ALL of your EC2 Instances to lock down the EC2 metadata endpoint to the minimal set of OS users (e.g., just root) to protect your IAM roles. Check out module-security, v0.8.2 and the example usage for details.

New Module: Fargate

Motivation: AWS recently released a new way to deploy docker images to the cloud called Fargate. Fargate lets you deploy your containerized apps without being responsible for managing the underlying infrastructure; it’s a bit like AWS Lambda, but for Docker images instead of individual functions. You just hand AWS a Docker image, specify the CPU/memory/network/etc requirements, and AWS will automatically maintain and scale the underlying instances for you.

Solution: We added a new ecs-fargate module to allow you deploy your docker images with Fargate. A typical usage will look something like this:

Copy
module "fargate_service" {
source = "git::git@github.com:gruntwork-io/module-ecs.git//modules/ecs-fargate?ref=v0.6.5"

Copy
service_name          = "fargate-example"
container_definitions = "${data.template_file.container.rendered}"

Copy
vpc_id     = "${var.vpc_id}"
subnet_ids = "${var.subnet_ids}"

Copy
desired_number_of_tasks = 3

Copy
cpu    = 1024
memory = 2048
}

Copy
data "template_file" "container" {
template = <<EOF
[
{
"name": "fargate-example",
"image": "gruntwork-io:docker-test-webapp",
"cpu": 1024,
"memory": 2048,
"essential": true,
"portMappings": [{
"containerPort": 8080,
"protocol": "tcp"
}]
}
]
EOF
}

If you run terraform apply on the code above, AWS will automatically provision the hardware to run 3 copies of your Docker image. It doesn’t get much easier than that!

What to do: If you’d like to give Fargate a shot, use module-ecs, v0.6.5 and check out an example usage for details.

New Module: Network Load Balancer

Motivation: AWS recently added a new type of Load Balancer called the Network Load Balancer (NLB), which is a TCP (Layer 4) load balancer that supports very high throughput, rapid scaling, and static IP addresses.

Solution: We added a new nlb module to make it easier to deploy the NLB.

What to do: If you’d like to take advantage of the scalability and throughput benefits that an NLB brings, use module-load-balancer, v0.8.0 (example usage).

New open source project: Bash Commons

Motivation: At Gruntwork, we write a lot of Bash code that’s used in install scripts, run scripts, User Data scripts, and other glue code. We found that we were reimplementing the same functionality over and over again throughout our Bash scripts (e.g., basic assertions, logging, looking up data in EC2 metadata, etc.), and wanted a way to keep things more DRY and tested.

Solution: We’ve created a new open source repo called bash-commons! It’s a collection of reusable bash utilities, complete with documentation and automated tests. You can install bash-commons on your servers using the Gruntwork Installer (be sure to replace <VERSION> in the command below with the latest version from the releases page):

Copy
gruntwork-install \
--repo https://github.com/gruntwork-io/bash-commons \
--module-name bash-commons \
--tag <VERSION>

Now you can use the source command to import the modules and start using them in your code:

Copy
source /opt/gruntwork/bash-commons/log.sh
source /opt/gruntwork/bash-commons/assert.sh
source /opt/gruntwork/bash-commons/os.sh

log_info "Hello, World!"

assert_not_empty "--foo" "$foo"

if os_is_ubuntu "16.04"; then
log_info "This script is running on Ubuntu 16.04!"
elif os_is_centos; then
log_info "This script is running on CentOS!"
fi

What to do: Check out the bash-commons repo for more info and start using it in your Bash scripts!

Kafka Now Supports CentOS

Motivation: In our April Newsletter, we indicated that CentOS support would be added for Kafka shortly.

Solution: That work is now complete! In addition, we upgraded our Kafka package to support more best best practices.

What to do: If you’d like to use Kafka with CentOS or upgrade to the latest best practices, check out package-kafka, v0.3.0.

Open source updates

• Terragrunt, v0.14.7: Fix a bug where after hooks were not running unless run_on_error was set.
• Terragrunt, v0.14.8: Terragrunt will now merge hooks between parent and child configs.
• Terragrunt, v0.14.9: Improve the way Terragrunt parses source URLs so it should be able to handle both URLs with and without a //.
• Terragrunt, v0.14.10: Terragrunt will now properly show help text for Terraform commands (e.g., terragrunt plan --help).
• terraform-aws-vault, v0.5.1: The vault-cluster module now sets a Name tag on the Launch Configuration security group.
• terraform-aws-nomad, v0.4.0: Update Consul usage to v0.3.1 everywhere. Update all Git URLs to use HTTPS instead of SSH for consistency. Fix tags set on Nomad cluster in root example (Nomad clients should not get the same tags as servers).
• terraform-aws-consul, v0.3.2: The install-consul script now exposes the --ca-file-path, --cert-file-path, and --key-file-path options to make it easier to install TLS certs. The run-consul script now exposes --enable-gossip-encryption and --gossip-encryption-keyoptions to make it easier to enable gossip encryption, and --enable-rpc-encryption, --ca-path, --cert-file-path, and --key-file-path options to make it easier to enable RPC encryption.
• terraform-aws-consul, v0.3.3: The Security Group rules that open up ports for Consul Agents in client mode (that is, the ports for LAN gossip) have been extracted into a new consul-client-security-group-rules module. This allows you to attach those rules to any Security Group that has Consul clients on it (e.g., Vault).
• terraform-aws-consul, v0.3.4: Remove provider and aws_region from examples so you can use them following the instructions in the Terraform Registry.
• terraform-aws-vault, v0.5.2: All the vault examples now use the new consul-client-security-group-rules to open up ports required for Consul clients to communicate with each other (LAN gossip). We recommend you add this module to your deployments if you are using Vault with Consul.
• terraform-aws-vault, v0.6.0: Update to Vault 0.10.0. run-vault now configures api_addr instead of redirect_addr.
• terraform-aws-vault, v0.6.1: You can now specify additional security groups to attach to the ASG in the vault-clustermodule using the new additional_security_group_ids parameter.
• terraform-aws-vault, v0.6.2: Remove provider and aws_region from examples so you can use them following the instructions in the Terraform Registry.
• terraform-aws-nomad, v0.4.1: Remove provider and aws_region from examples so you can use them following the instructions in the Terraform Registry.

Other fixes

• module-data-storage, v0.6.2: The rds and aurora modules now allow you to enable enhanced monitoring and to specify a custom monitoring IAM role to associate with the DB.
• module-data-storage, v0.6.3: Explicitly disable snapshots for replicas so you can successfully destroy them without hitting errors.
• module-data-storage, v0.6.4: Explicitly set the minor_version_upgrade setting on rds read replicas so they use the same setting as the primary.
• module-vpc, v0.5.0: The vpc-mgmt-network-acls module now allows all inbound and outbound traffic within the private subnet and between the public and private subnet. Before, all inbound traffic was allowed, but outbound traffic was limited solely to TCP. The vpc-app-network-acls module now allows all inbound and outbound traffic from/to the mgmt VPC. Before, all inbound traffic was allowed, but outbound traffic was limited solely TCP.
• module-load-balancer, v0.8.1: Fix several bugs with count in the alb module that would crop up if allow_inbound_from_cidr_blocks or allow_inbound_from_security_group_ids were empty.
• module-ci, v0.7.2: Update the ALB version used in the jenkins-server module to v0.8.1 to pick up the fixes listed above.

DevOps News

AWS Certificate Manager Launches Private Certificate Authority

What happened: AWS has launched Private Certificate Authority (PCA), which is a new feature for AWS Certificate Manager (ACM) that you can use to manage private certificates (e.g., TLS certificates) within your AWS account.

Why it matters: Before, if you wanted to use private certificates, you’d have to create your own CA, sign the certs, manage renewing them, manage distributing the trusted cert files, manage certificate revocation lists (CRLs), and so on. Now, all of this is managed for you by AWS, and you can create new private certs with a few clicks or a few API calls.

What to do about it: Check out the PCA documentation to get started. We’ll be adding support for PCA to our Ref Arch (e.g., for certs for internal services, such as sample-app-backend and Jenkins) in the near future.

AWS Secrets Manager

What happened: AWS has launched a new product called the AWS Secrets Manager, which allows you to securely store, retrieve, and rotate secrets in AWS via API calls.

Why it matters: You now have an additional way to securely manage secrets and to grant access to them using IAM. Other options include using KMS to encrypt secrets and store them in files and tools such as HashiCorp Vault, which provide more powerful tools for secrets management, but require you to run your own Vault cluster (and a Consul cluster for HA).

What to do about it: Give the Secrets Manager a shot and let us know what you think!

HashiCorp Vault 0.10

What happened: Vault 0.10 has been released!

Why it matters: The key new features in Vault 0.10 are: secret versioning and check-and-set operations; the previously enterprise-only UI is now open source; root / admin credentials for DBs can now be securely rotated; Azure auth via Active Directory credentials; Vault can now create dynamic IAM credentials for accessing Google Cloud Platform environments.

What to do about it: Check out the announcement blog post for all the details. We hope to add support for all these features in our open source Vault module soon.

CloudWatch adds support for metric math

What happened: CloudWatch now allows you to perform mathematical operations on your metrics.

Why it matters: You can now create graphs for derived metrics, rather than being solely limited to the raw data. For example, instead of just seeing the total number of 5xx errors from your load balancer, you could use metric math to compute the error rate, which is the number of 5xx errors divided by the total number of requests.

What to do about it: Check out the metric math documentation and start using it with your CloudWatch dashboards! Unfortunately, it doesn’t seem like you can create alerts from computed metrics, but perhaps that will be available in the future.

AWS and GCP blocked in Russia

What happened: In a battle with encrypted messaging service Telegram, Russia has apparently taken the drastic measure of blocking all of AWS and GCP IPs.

Why it matters: If you are running your infrastructure on AWS or GCP, then users in Russia may not be able to access it.

What to do about it: Other than migrating off of AWS and GCP, there’s not much you can do other than wait it out, and hope Russia finds a less extreme way to deal with this situation.

Security Updates

Below is a list of critical security updates that may impact your services. We notify Gruntwork customers of these vulnerabilities as soon as we know of them via the Gruntwork Security Alerts mailing list. It is up to you to scan this list and decide which of these apply and what to do about them, but most of these are severe vulnerabilities, and we recommend patching them ASAP.

Drupal

• SA-CORE-2018–004: A remote code execution vulnerability exists within multiple subsystems of Drupal 7.x and 8.x which allows attackers to compromise the site. This vulnerability is already known to be exploited in the wild. We sent an email about this vulnerability to the Gruntwork Security Alerts Mailing List mailing list on April 27, 2018.

OpenJDK

• ALAS-2018–1002: A number of vulnerabilities were found in OpenJDK 8 that can be easily exploited by remote attackers. We sent an email about this vulnerability to the Gruntwork Security Alerts Mailing List on April 27, 2018.

CloudFront

• CloudFront Hijacking: CloudFront’s handling of DNS is not intuitive and may make it possible for a domain name to be hijacked. AWS is updating CloudFront to try to mitigate these issues. You can use cloudfrunt to check your domains for vulnerabilities. This is not a “critical” security issue, so we did not email it to the Gruntwork Security Alerts Mailing List, but something you may want to be aware of.

Intel CPUs

• Spectre-NG: Eight moreSpectre-like vulnerabilities have been found in Intel processors. Patches and full details are coming in the future. If there is immediate action to take, we will notify the Gruntwork Security Alerts Mailing List.