Browse the Repo

file-type-icon.circleci
file-type-icon_docs
file-type-iconexamples
file-type-iconelasticsearch-docker
file-type-iconelasticsearch-only-cluster
file-type-iconelk-amis
file-type-iconelk-multi-cluster
file-type-iconelk-single-cluster
file-type-iconuser-data
file-type-iconREADME.md
file-type-iconmain.tf
file-type-iconoutputs.tf
file-type-iconvars.tf
file-type-iconmodules
file-type-icontest
file-type-icon.gitignore
file-type-iconCODEOWNERS
file-type-iconCONTRIBUTING.md
file-type-iconLICENSE.md
file-type-iconREADME.md

Browse the Repo

file-type-icon.circleci
file-type-icon_docs
file-type-iconexamples
file-type-iconelasticsearch-docker
file-type-iconelasticsearch-only-cluster
file-type-iconelk-amis
file-type-iconelk-multi-cluster
file-type-iconelk-single-cluster
file-type-iconuser-data
file-type-iconREADME.md
file-type-iconmain.tf
file-type-iconoutputs.tf
file-type-iconvars.tf
file-type-iconmodules
file-type-icontest
file-type-icon.gitignore
file-type-iconCODEOWNERS
file-type-iconCONTRIBUTING.md
file-type-iconLICENSE.md
file-type-iconREADME.md
Elasticsearch (self-hosted)

Elasticsearch (self-hosted)

Deploy a self-hosted Elasticsearch cluster. Supports automatic bootstrap, zero-downtime rolling deployment, auto healing, backup, and recovery.

Code Preview

Preview the Code

mobile file icon

README.md

down

ELK Single Cluster Example

In this example we demonstrate a way in which the entire ELK stack can be run on just one ASG by colocating all of the ELK components into the same AMI. While this is not the recommended approach for running a production ELK setup, it can be useful for non-production environments.

This example also deploys a Load Balancer in front of the entire cluster using the load-balancer module.

What resources are does this example deploy?

  1. A single all in one server behind an ASG where we run elasticsearch, kibana, logstash, filebeat and CollectD
  2. An Application Load Balancer
  3. A CloudWatch Log Group
  4. A CloudWatch Log Stream
  5. An S3 bucket for Cloudtrail logs

You will need to create Amazon Machine Images (AMIs) that have all of the ELK components installed. You can do this using:

How do I get this example deployed?

  1. git clone this repo to your computer.
  2. Build the ELK all-in-one AMI. See the all-in-one ami example documentation for instructions. Make sure to note down the ID of the AMI.
  3. Install Terraform.
  4. Open vars.tf, set the environment variables specified at the top of the file, and fill in any other variables that don't have a default, including putting your AMI ID into the ami_id variable.
  5. Run terraform init.
  6. Run terraform apply.

Connecting to the cluster

  • Once the cluster has successfully deployed, you should be able to see the DNS name of the load balancer in the alb_dns_name output variable.
  • To access the Kibana UI go to: http://[lb_dns_name]:[kibana_port]/. Your URL will look something like this: http://exampleescluster-alb-77641507.us-east-1.elb.amazonaws.com:5601/
  • Elasticsearch will be accessible at: http://[alb_dns_name]:[elasticsearch_api_port]/. Your URL will look something like this: http://exampleescluster-alb-77641507.us-east-1.elb.amazonaws.com:9200/
  • All components of the ELK stack communicate through the deployed load balancer, irrespective of the fact that they're all running on the same machine.

Questions? Ask away.

We're here to talk about our services, answer any questions, give advice, or just to chat.

Ready to hand off the Gruntwork?