Browse the Repo
Browse the Repo
Deploy a self-hosted Elasticsearch cluster. Supports automatic bootstrap, zero-downtime rolling deployment, auto healing, backup, and recovery.
In this example we demonstrate how to use our ELK modules to run an ELK stack consisting of several Auto Scaling Groups (ASGs) that each run an ELK component (Kibana, Logstash, Elasticsearch...)
git clonethis repo to your computer.
ami_ids of the AMIs you've built. Look here for detailed guides on how to build the AMIs
vars.tf, set the environment variables specified at the top of the file, and fill in any other variables that don't have a default, including putting the AMI Ids you noted down from above into the
alb_urloutput variable. Your URL will look something like this:
http://[SubdomainYouSet].[YourHostedZone]:[elasticsearch_api_port(9200 by default)].
We deploy an ALB in order to simplify load balancing and routing between the nodes of the ELK cluster. The ALB also helps solve the issues around resource discovery in a convenient and simple way. The ALB only deals with HTTP and HTTPS traffic. This is an important detail to keep in mind as ELK components that use custom (non-http) protocols, like Filebeat, cannot use the ALB for load balancing or discovery.
Given the ALB's HTTP only limitation, we require a different solution for dealing with discovery for Filebeat, a component which uses a custom protocol for sending messages to Elasticsearch.
A Network Load Balancer (NLB) was originally how we solved the load balancing and discovery problem for all ELK cluster members. In fact, we tried only using an ALB without having to deploy an ALB. Unfortunately we ran into some NLB Limitations which precluded its use:
To work around the issues we encountered we created a simple auto-discovery module that will use AWS APIs to get the IP addresses of all instances matching a given tag. The current implementation of this script will use a regular expression (passed as a parameter when invoking the script) to find/replace newly discovered IP addresses in a config file (also passed as a parameter). Finally, the script will automatically restart whatever application it is performing discovery for (the application's systemd service name is also passed as a script parameter)
LIMITATION: Unfortunately, the approach described above can't work if the application for which we are doing discovery is configured to do full SSL certificate verification (ie: hostname verification). Since our approach looks up randomly assigned IP addresses, there's no way we could generate an SSL certificate that encapsulates all of the possible IP addresses that may be found. For this reason, we have to disable SSL hostname verification when using auto-discovery.
To work around the limitation described above, we will be introducing updates to the auto-discovery module
that will update local
/etc/hosts or run a local instance of Dnsmasq
in order to make seamless updates to the DNS instead of updating an application's config file and then needing to restart
<a name="whyjustonenode">1</a>: We are using an ASG here even though we will only be running one node because we want the ASG to handle automatically restarting our EC2 instance if it ever crashes.
We're here to talk about our services, answer any questions, give advice, or just to chat.