Browse the Repo

file-type-icon.circleci
file-type-icon_docs
file-type-iconexamples
file-type-iconmodules
file-type-iconauto-discovery
file-type-iconbeats-iam-policies
file-type-iconelastalert-iam-policies
file-type-iconelastalert-security-group-rules
file-type-iconelastalert
file-type-iconelasticsearch-cluster-backup
file-type-iconelasticsearch-cluster-restore
file-type-iconelasticsearch-cluster
file-type-iconREADME.md
file-type-iconmain.tf
file-type-iconoutputs.tf
file-type-iconvariables.tf
file-type-iconelasticsearch-iam-policies
file-type-iconelasticsearch-security-group-rules
file-type-iconinstall-collectd
file-type-iconinstall-elastalert
file-type-iconinstall-elasticsearch
file-type-iconinstall-filebeat
file-type-iconinstall-kibana
file-type-iconinstall-logstash
file-type-iconkibana-cluster
file-type-iconkibana-security-group-rules
file-type-iconload-balancer-alb-target-group
file-type-iconlogstash-cluster
file-type-iconlogstash-iam-policies
file-type-iconlogstash-security-group-rules
file-type-iconrun-collectd
file-type-iconrun-elastalert
file-type-iconrun-elasticsearch
file-type-iconrun-filebeat
file-type-iconrun-kibana
file-type-iconrun-logstash
file-type-icontest
file-type-icon.gitignore
file-type-icon.pre-commit-config.yaml
file-type-iconCODEOWNERS
file-type-iconCONTRIBUTING.md
file-type-iconLICENSE.md
file-type-iconREADME.md
file-type-iconterraform-cloud-enterprise-private-module-...

Browse the Repo

file-type-icon.circleci
file-type-icon_docs
file-type-iconexamples
file-type-iconmodules
file-type-iconauto-discovery
file-type-iconbeats-iam-policies
file-type-iconelastalert-iam-policies
file-type-iconelastalert-security-group-rules
file-type-iconelastalert
file-type-iconelasticsearch-cluster-backup
file-type-iconelasticsearch-cluster-restore
file-type-iconelasticsearch-cluster
file-type-iconREADME.md
file-type-iconmain.tf
file-type-iconoutputs.tf
file-type-iconvariables.tf
file-type-iconelasticsearch-iam-policies
file-type-iconelasticsearch-security-group-rules
file-type-iconinstall-collectd
file-type-iconinstall-elastalert
file-type-iconinstall-elasticsearch
file-type-iconinstall-filebeat
file-type-iconinstall-kibana
file-type-iconinstall-logstash
file-type-iconkibana-cluster
file-type-iconkibana-security-group-rules
file-type-iconload-balancer-alb-target-group
file-type-iconlogstash-cluster
file-type-iconlogstash-iam-policies
file-type-iconlogstash-security-group-rules
file-type-iconrun-collectd
file-type-iconrun-elastalert
file-type-iconrun-elasticsearch
file-type-iconrun-filebeat
file-type-iconrun-kibana
file-type-iconrun-logstash
file-type-icontest
file-type-icon.gitignore
file-type-icon.pre-commit-config.yaml
file-type-iconCODEOWNERS
file-type-iconCONTRIBUTING.md
file-type-iconLICENSE.md
file-type-iconREADME.md
file-type-iconterraform-cloud-enterprise-private-module-...
Elasticsearch (self-hosted)

Elasticsearch (self-hosted)

Deploy a self-hosted Elasticsearch cluster. Supports automatic bootstrap, zero-downtime rolling deployment, auto healing, backup, and recovery.

Code Preview

Preview the Code

mobile file icon

README.md

down

Elasticsearch Cluster

This folder contains a Terraform module to deploy an Elasticsearch cluster in AWS on top of an Auto Scaling Group. The idea is to create an Amazon Machine Image (AMI) that has Elasticsearch installed using the install-elasticsearch module.

In a non-production setting, you can install Elasticsearch tools such as Kibana and ElastAlert on the same AMI. In a production setting, Elasticsearch should be the sole service running on each Elasticsearch node.

How do you use this module?

This folder defines a Terraform module, which you can use in your code by adding a module configuration and setting its source parameter to URL of this folder:

module "elasticsearch_cluster" {
  # TODO: replace <VERSION> with the latest version from the releases page: https://github.com/gruntwork-io/terraform-aws-elk/releases
  source = "github.com/gruntwork-io/terraform-aws-elk//modules/elasticsearch-cluster?ref=<VERSION>"

  # Specify the ID of the Elasticsearch AMI. You should build this using the scripts in the install-elasticsearch (and 
  # in a non-production setting, the install-logstash, instgall-kibana, and install-elastalert modules).
  ami_id = "ami-abcd1234"
  
  # Configure and start Elasticsearch during boot. 
  user_data = <<-EOF
              #!/bin/bash
              /usr/share/elasticsearch/bin/run-elasticsearch --cluster-name dev --cluster-tag ClusterName=dev
              EOF
  
  # ... See variables.tf for the other parameters you must define for the elasticsearch-cluster module
}

Note the following parameters:

  • source: Use this parameter to specify the URL of the elasticsearch-cluster module. The double slash (//) is intentional and required. Terraform uses it to specify subfolders within a Git repo (see module sources). The ref parameter specifies a specific Git tag in this repo. That way, instead of using the latest version of this module from the master branch, which will change every time you run Terraform, you're using a fixed version of the repo.

  • ami_id: Use this parameter to specify the ID of an Elasticsearch Amazon Machine Image (AMI) to deploy on each server in the cluster. You should install Elasticsearch and optionally Elasticsearch tools on this AMI using the following modules:

    In a production setting, your AMI should only run Elasticsearch, and Elasticsearch tools should be built on a separate AMI. In a dev-only environment where parity to production doesn't matter, colocating Elasticsearch and its tools is ok.

  • user_data: Use this parameter to specify a User Data script that each server will run during boot. This is where you can use the run-elasticsearch and, if applicable, run-elastalert, run-kibana, and run-logstash scripts to configure and run Elasticsearch and its tools.

You can find the other parameters in variables.tf.

Check out the examples folder for fully-working sample code.

How do you connect to the Elasticsearch cluster?

Connecting to Elasticsearch via Official Elasticsearch Clients

The preferred way to connect to Elasticsearch is to use one of the official Elasticsearch clients. All official Elasticsearch clients are designed to discover multiple Elasticsearch nodes and distribute reuqests across the various nodes.

Therefore, using a Load Balancer to talk to Elasticsearch APIs (e.g., via an SDK) is NOT recommended, so you will need to get the IPs of the individual nodes and connect to them directly. Since those nodes run in an Auto Scaling Group (ASG) where servers can be added/replaced/removed at any time, you can't get their IP addresses from Terraform. Instead, you'll need to look up the IPs using the AWS APIs.

The easiest way to do that is to use the AWS SDK to look up the servers using EC2 Tags. Each server deployed by the elasticsearch-cluster module has its Name and aws:autoscaling:groupName tag set to the value you pass in via the cluster_name parameter. You can also specify custom tags via the tags parameter. You can use the AWS SDK to find the IPs of all servers with those tags.

For example, using the AWS CLI, you can get the IPs for servers in us-east-1 with the tag Name=elasticsearch-example as follows:

aws ec2 describe-instances \
    --region "us-east-1" \
    --filter \
      "Name=tag:Name,Values=elasticsearch-example" \
      "Name=instance-state-name,Values=running"

This will return a bunch of JSON that contains the IPs of the servers. You can then use the Elasticsearch client for your programming language to connect to these IPs.

Connecting via the REST API

Elasticsearch exposes a RESTful API that you can directly access using curl or any other programming language feature that makes HTTP requests.

What's included in this module?

This module creates the following:

What's Not Included

Auto Scaling Group

This module runs Elasticsearch on top of an Auto Scaling Group (ASG). Typically, you should run the ASG with multiple Instances spread across multiple Availability Zones. Each of the EC2 Instances should be running an AMI that has Elasticsearch and optional Elasticsearch tools installed via the install-elasticsearch, install-elastalert, install-kibana, and install-logstash scripts. You pass in the ID of the AMI to run using the ami_id input parameter.

Load Balancer

We use a Network Load Balancer (1) so that we can perform ongoing health checks on each Elasticsearch node, and (2) so that Kibana can be accessed via a single endpoint which will forward to a live Kibana endpoint at random.

Note that we do not need a Load Balancer to distribute traffic to Elasticsearch because all the official Elasticsearch clients are designed to discover all Elasticsearch nodes and distribute requests across the cluster. Using a Load Balancer for this reason would duplicate functionality Elasticsearch clients already give us.

Security Group

Each EC2 Instance in the ASG has a Security Group that allows minimal connectivity:

  • All outbound requests
  • Inbound SSH access from the CIDR blocks and security groups you specify

The ID of the security group is exported as an output variable, which you can use with the elasticsearch-security-group-rules, elastalert-security-group-rules, kibana-security-group-rules, and logstash-security-group-rules modules to open up all the ports necessary for Elasticsearch and the respective Elasticsearch tools.

Check out the Security section for more details.

IAM Role and Permissions

Each EC2 Instance in the ASG has an IAM Role attached. The IAM Role ARN and ID are exported as output variables if you need to add additional permissions.

EBS Volumes

Note that we do not use EBS Volumes, which are AWS's ultra-low-latency network-attached storage. Instead, per Elasticsearch docs on AWS Best Practices, we exclusively use Instance Stores.

Instance Stores have the major disadvantage that they do not survive the termination of an EC2 Instance. That is, when an EC2 Instance dies, all the data on an Instance Store dies with it and is unrecoverable. But Elasticsearch already has built in support for replica shards, so we already have redundancy available to us if an EC2 Instance should fail.

This enables us to take advantage of the benefits of Instance Stores, which are that they are significantly faster because I/O traffic is now all local. By contrast, I/O traffic with EBS Volumes must traverse the (admittedly ultra low- latency) network and are therefore much slower.

How do you roll out updates?

If you want to deploy a new version of Elasticsearch across the cluster, the best way to do that is to:

  1. Rolling deploy:

    1. Build a new AMI.

    2. Set the ami_id parameter to the ID of the new AMI.

    3. Run terraform apply.

    4. Because the elasticsearch-cluster module uses the Gruntwork server-group modules under the hood, running terraform apply will automatically perform a zero-downtime rolling deployment. Specifically, one EC2 Instance at a time will be terminated, a new EC2 Instance will spawn in its place, and only once the new EC2 Instance passes the Load Balancer Health Checks will the next EC2 Instance be rolled out.

      Note that there will be a brief period of time during which EC2 Instances based on both the old ami_id and new ami_id will be running. Rolling upgrades docs suggest that this is acceptable for Elasticsearch version 5.6 and greater.

      TODO: Add support for automatically disabling shard allocation and performing a synced flush on an Elasticsearch node prior to terminating it (docs).

  2. New cluster:

    1. Build a new AMI.
    2. Create a totally new ASG using the elasticsearch-cluster module with the ami_id set to the new AMI, but all other parameters the same as the old cluster.
    3. Wait for all the nodes in the new ASG to join the cluster and catch up on replication.
    4. Remove each of the nodes from the old cluster.
    5. Remove the old ASG by removing that elasticsearch-cluster module from your code.

Security

Here are some of the main security considerations to keep in mind when using this module:

  1. Encryption in transit
  2. Encryption at rest
  3. Dedicated instances
  4. Security groups
  5. SSH access

Encryption in transit

Elasticsearch can encrypt all of its network traffic. TODO: Should we recommend using X-Pack (official solution, but paid), an Nginx Reverse Proxy, a custom Elasticsearch plugin, or something else?

Encryption at rest

EC2 Instance Storage

The EC2 Instances in the cluster store their data in an EC2 Instance Store, which does not have native suport for encryption (unlike EBS Volume Encryption).

TODO: Should we implement encryption at rest uising the technique described at https://aws.amazon.com/blogs/security/how-to-protect-data-at-rest-with-amazon-ec2-instance-store-encryption/?

Elasticsearch Keystore

Some Elasticsearch settings may contain secrets and should be encrypted. You can use the Elasticsearch Keystore for such settings. The elasticsearch.keystore is created automatically upon boot of each node, and is available for use as described in the docs.

Dedicated instances

If you wish to use dedicated instances, you can set the tenancy parameter to "dedicated" in this module.

Security groups

This module attaches a security group to each EC2 Instance that allows inbound requests as follows:

SSH access

You can associate an EC2 Key Pair with each of the EC2 Instances in this cluster by specifying the Key Pair's name in the ssh_key_name variable. If you don't want to associate a Key Pair with these servers, set ssh_key_name to an empty string.

Questions? Ask away.

We're here to talk about our services, answer any questions, give advice, or just to chat.

Ready to hand off the Gruntwork?