In this example we demonstrate how to use our ELK modules to run an ELK stack consisting of several Auto Scaling Groups (ASGs)
that each run an ELK component (Kibana, Logstash, Elasticsearch...)
An SNS topic - this is where ElastAlert will send our sample alert notifications.
A Route53 Alias Record that will attach a subdomain name to an existing HostedZone.
How do I get this example deployed?
git clone this repo to your computer.
You'll need to build AMIs of each of the ELK components referenced above and note down the ami_ids of the AMIs
you've built. Look here for detailed guides on how to build the AMIs
Open vars.tf, set the environment variables specified at the top of the file, and fill in any other variables that
don't have a default, including putting the AMI Ids you noted down from above into the [elk-component]_ami_id variables.
Run terraform init.
Run terraform apply.
Connecting to the cluster
Once the cluster has successfully deployed, you should be able to see the main URL in the alb_url
output variable.
To access the Kibana UI go to the URL provided in alb_url output variable. Your URL will look something like
this: http://[SubdomainYouSet].[YourHostedZone]/
Elasticsearch will be accessible at: http://[SubdomainYouSet].[YourHostedZone]:[elasticsearch_api_port(9200 by default)].
Why deploy an ALB?
We deploy an ALB in order to simplify load balancing and routing between the nodes of the ELK cluster. The ALB also helps
solve the issues around resource discovery in a convenient and simple way. The ALB only deals with HTTP and HTTPS
traffic. This is an important detail to keep in mind as ELK components that use custom (non-http) protocols, like
Filebeat, cannot use the ALB for load balancing or discovery.
How we handle non-HTTP discovery?
Given the ALB's HTTP only limitation, we require a different solution for dealing with discovery for Filebeat, a component
which uses a custom protocol for sending messages to Elasticsearch.
Why we don't use an NLB
A Network Load Balancer (NLB) was originally how we solved the load balancing and discovery problem for all ELK cluster
members. In fact, we tried only using an ALB without having to deploy an ALB. Unfortunately we ran into some NLB Limitations
which precluded its use:
An internal NLB in a private subnet can't be accessed from a peered VPC. This limitation doesn't affect this example
code because we are deploying a public NLB into a public subnet and not trying to access it from a peered VPC. In a
production environment, this is no longer the case and makes it impossible to access the NLB.
Auto-discovery
Current implementation
To work around the issues we encountered we created a simple auto-discovery module that will
use AWS APIs to get the IP addresses of all instances matching a given tag. The current implementation of this script
will use a regular expression (passed as a parameter when invoking the script) to find/replace newly discovered IP addresses
in a config file (also passed as a parameter). Finally, the script will automatically restart whatever application it is
performing discovery for (the application's systemd service name is also passed as a script parameter)
LIMITATION: Unfortunately, the approach described above can't work if the application for which we are doing discovery
is configured to do full SSL certificate verification (ie: hostname verification). Since our approach looks up randomly
assigned IP addresses, there's no way we could generate an SSL certificate that encapsulates all of the possible IP addresses
that may be found. For this reason, we have to disable SSL hostname verification when using auto-discovery.
Future Improvements
To work around the limitation described above, we will be introducing updates to the auto-discovery module
that will update local /etc/hosts or run a local instance of Dnsmasq
in order to make seamless updates to the DNS instead of updating an application's config file and then needing to restart
that application.
<a name="whyjustonenode">1</a>: We are using an ASG here even though we will only be running one node because we want the
ASG to handle automatically restarting our EC2 instance if it ever crashes.
Questions? Ask away.
We're here to talk about our services, answer any questions, give advice, or just to chat.
{"treedata":{"name":"root","toggled":true,"children":[{"name":".circleci","children":[{"name":"config.yml","path":".circleci/config.yml","sha":"c2cff2a8764e0fabad7a8408ba9e42113cc23487"}]},{"name":".gitignore","path":".gitignore","sha":"f2438833ea6b1bc45e125b598cb1e08cbd274e1b"},{"name":"CODEOWNERS","path":"CODEOWNERS","sha":"cef671f8c0aee235f6d8cf886e421cfab3b51729"},{"name":"CONTRIBUTING.md","path":"CONTRIBUTING.md","sha":"492bef006a63f2c3189fd171116eb3f900c6cd02"},{"name":"LICENSE.md","path":"LICENSE.md","sha":"a2cf01ecdd725fddd718ab91c80c115882c94f3c"},{"name":"README.md","path":"README.md","sha":"ebe0be6e9b1f79275f4f56fb3b4205755b2ade16"},{"name":"_docs","children":[{"name":"elk-architecture.png","path":"_docs/elk-architecture.png","sha":"a7585c6f40eb0a7740b34f00d412bc2e37373266"}]},{"name":"examples","children":[{"name":"elasticsearch-docker","children":[{"name":"local-mocks","children":[{"name":"entrypoint.sh","path":"examples/elasticsearch-docker/local-mocks/entrypoint.sh","sha":"8e298671493441b6d7548f619a6155a408677d99"}]},{"name":"non-ssl","children":[{"name":".env","path":"examples/elasticsearch-docker/non-ssl/.env","sha":"bc82d24eea229df2ba77f0fdf1f6e1c6f6a8e532"},{"name":"README.md","path":"examples/elasticsearch-docker/non-ssl/README.md","sha":"ed4bee52f5093a864164067b57d54f41668e55f5"},{"name":"docker-compose.yml","path":"examples/elasticsearch-docker/non-ssl/docker-compose.yml","sha":"e7547917f574679857f800b24f86bd573671931e"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/elasticsearch-docker/non-ssl/user-data/user-data.sh","sha":"81df178b3080a40a9e03ff7e94f1ef5db7d57f98"}]}]},{"name":"ssl","children":[{"name":".env","path":"examples/elasticsearch-docker/ssl/.env","sha":"bc82d24eea229df2ba77f0fdf1f6e1c6f6a8e532"},{"name":"README.md","path":"examples/elasticsearch-docker/ssl/README.md","sha":"ed4bee52f5093a864164067b57d54f41668e55f5"},{"name":"docker-compose.yml","path":"examples/elasticsearch-docker/ssl/docker-compose.yml","sha":"25e2b036c7e672a8ffc4b9559ab53f8d77441f99"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/elasticsearch-docker/ssl/user-data/user-data.sh","sha":"fc95c5a93069b251aca3ec2ae70a2be90231fcd9"}]}]}]},{"name":"elasticsearch-only-cluster","children":[{"name":"README.md","path":"examples/elasticsearch-only-cluster/README.md","sha":"5bdbc120e354d55895febb193390b8ba4440e9ac"},{"name":"main.tf","path":"examples/elasticsearch-only-cluster/main.tf","sha":"491b09a2683ba577be29cd1ca226c3b75ed55183"},{"name":"outputs.tf","path":"examples/elasticsearch-only-cluster/outputs.tf","sha":"382f2cda2c87d824b274f986f5107fd90477544a"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/elasticsearch-only-cluster/user-data/user-data.sh","sha":"55678f9678e93453686c629e364f82ebacdb83df"}]},{"name":"vars.tf","path":"examples/elasticsearch-only-cluster/vars.tf","sha":"6c3db8f2b043c53d04f7278a076230b4ce06308a"}]},{"name":"elk-amis","children":[{"name":"README.md","path":"examples/elk-amis/README.md","sha":"4e58b3a2e6be086188621bee400ff750f02436e8"},{"name":"all-in-one","children":[{"name":"README.md","path":"examples/elk-amis/all-in-one/README.md","sha":"42799bc50b4f41eb9e27ed79f1b5624b812b81b2"},{"name":"all-in-one.json","path":"examples/elk-amis/all-in-one/all-in-one.json","sha":"e0b06562c958eae8d91c371c9ebd757446ed79cd"}]},{"name":"app-server","children":[{"name":"README.md","path":"examples/elk-amis/app-server/README.md","sha":"577966ff3d1aa65d69aa43d6b7e1e86968099335"},{"name":"app-server.json","path":"examples/elk-amis/app-server/app-server.json","sha":"d9f6b7c6009f24c60fe210b77f580b58b22562aa"}]},{"name":"collectd","children":[{"name":"README.md","path":"examples/elk-amis/collectd/README.md","sha":"a9fa0d44e853022868368cab446e4aaeb24997ab"},{"name":"collectd-install-steps.sh","path":"examples/elk-amis/collectd/collectd-install-steps.sh","sha":"1fa3c68c3a1318caab275a7eb4bbaf9533b75319"},{"name":"collectd.json","path":"examples/elk-amis/collectd/collectd.json","sha":"b3f883f1cfd1866f08f411ac302ba5db7bf2fde6"},{"name":"config","children":[{"name":"collectd-ssl.conf","path":"examples/elk-amis/collectd/config/collectd-ssl.conf","sha":"ca0469d729e643644f90118e5a639671a3f60d08"},{"name":"collectd.conf","path":"examples/elk-amis/collectd/config/collectd.conf","sha":"74c1b46937039070cd95f64269eef63aabd72805"}]}]},{"name":"elastalert","children":[{"name":"README.md","path":"examples/elk-amis/elastalert/README.md","sha":"9dd33dfd29ddafa1040213466560ffcf8a66944d"},{"name":"aws","children":[{"name":"elastalert-config","children":[{"name":"config-ssl.yml","path":"examples/elk-amis/elastalert/aws/elastalert-config/config-ssl.yml","sha":"f26f8404e40736c5b0b17bc23e6a5de119d0c80f"},{"name":"config.yml","path":"examples/elk-amis/elastalert/aws/elastalert-config/config.yml","sha":"6054d91b8e86df9102fd5fd1fa2c545f7a981d31"}]}]},{"name":"docker","children":[{"name":"elastalert-config","children":[{"name":"config-ssl.yml","path":"examples/elk-amis/elastalert/docker/elastalert-config/config-ssl.yml","sha":"f26f8404e40736c5b0b17bc23e6a5de119d0c80f"},{"name":"config.yml","path":"examples/elk-amis/elastalert/docker/elastalert-config/config.yml","sha":"6054d91b8e86df9102fd5fd1fa2c545f7a981d31"}]}]},{"name":"elastalert-install-steps.sh","path":"examples/elk-amis/elastalert/elastalert-install-steps.sh","sha":"0e5cca36e97e25543d28d2d4d346b80110ceef8d"},{"name":"elastalert-rules","children":[{"name":"example_change.yml","path":"examples/elk-amis/elastalert/elastalert-rules/example_change.yml","sha":"4020886483786ff9a8847159b5750014c5d2b0fb"}]},{"name":"elastalert.json","path":"examples/elk-amis/elastalert/elastalert.json","sha":"419a67dc9d900c05d480b60452dba7aea7a3ee92"}]},{"name":"elasticsearch","children":[{"name":"README.md","path":"examples/elk-amis/elasticsearch/README.md","sha":"f77a938e72424815565ffe16b9ae7787cb430cc7"},{"name":"aws","children":[{"name":"config","children":[{"name":"elasticsearch-ssl.yml","path":"examples/elk-amis/elasticsearch/aws/config/elasticsearch-ssl.yml","sha":"aab4150ebef375ddb962e17c679642f728df538d"},{"name":"elasticsearch.yml","path":"examples/elk-amis/elasticsearch/aws/config/elasticsearch.yml","sha":"0bf97273dce0664ca96ef87151f856162b1a2411"},{"name":"jvm.options","path":"examples/elk-amis/elasticsearch/aws/config/jvm.options","sha":"28dcf519ee339aba5dd15059c1acc4b97846d6ce"}]}]},{"name":"docker","children":[{"name":"config","children":[{"name":"elasticsearch-ssl.yml","path":"examples/elk-amis/elasticsearch/docker/config/elasticsearch-ssl.yml","sha":"705af52bc1e914df93faa919de8a23b973184938"},{"name":"elasticsearch.yml","path":"examples/elk-amis/elasticsearch/docker/config/elasticsearch.yml","sha":"2ac38b9c875dd6352b7a3d7d0fc2c02662d0e9cb"},{"name":"jvm.options","path":"examples/elk-amis/elasticsearch/docker/config/jvm.options","sha":"28dcf519ee339aba5dd15059c1acc4b97846d6ce"}]}]},{"name":"elasticsearch-install-steps.sh","path":"examples/elk-amis/elasticsearch/elasticsearch-install-steps.sh","sha":"7d766638a6a8368b56ae5b81c65f3d202c907811"},{"name":"elasticsearch.json","path":"examples/elk-amis/elasticsearch/elasticsearch.json","sha":"9c91629c2ba2b14d7647bf380532e442343228e9"},{"name":"plugins","children":[{"name":"readonlyrest-1.18.5_es6.8.3.zip","path":"examples/elk-amis/elasticsearch/plugins/readonlyrest-1.18.5_es6.8.3.zip","sha":"92de49ac4f2a7e90b58dd0d04effe068483a5002"}]},{"name":"readonlyrest-config","children":[{"name":"readonlyrest.yml","path":"examples/elk-amis/elasticsearch/readonlyrest-config/readonlyrest.yml","sha":"a5efa565eca562bcc9a33b959c668e5fd7087165"}]}]},{"name":"filebeat","children":[{"name":"README.md","path":"examples/elk-amis/filebeat/README.md","sha":"ce76c3cbb426dc4d5e82f04a85eb0efd043fe671"},{"name":"config","children":[{"name":"filebeat-ssl.yml","path":"examples/elk-amis/filebeat/config/filebeat-ssl.yml","sha":"24b163e671b4cc49af08726867d8869f105a8ecb"},{"name":"filebeat.yml","path":"examples/elk-amis/filebeat/config/filebeat.yml","sha":"ae4641ee77c89e389728836fca3cac1428214fe2"}]},{"name":"filebeat-install-steps.sh","path":"examples/elk-amis/filebeat/filebeat-install-steps.sh","sha":"71192c99d9e5a1207a7a0f6f0dfa107532a21fe0"},{"name":"filebeat.json","path":"examples/elk-amis/filebeat/filebeat.json","sha":"3d87c3b35089e41f432ff160593e8e2f7ad0854f"}]},{"name":"kibana","children":[{"name":"README.md","path":"examples/elk-amis/kibana/README.md","sha":"b8af363345ba445cb9fbc2b04ec0e7d6b0b08149"},{"name":"config","children":[{"name":"kibana-ssl.yml","path":"examples/elk-amis/kibana/config/kibana-ssl.yml","sha":"9d0761e628a703eec051b31e6cce9d39e62f5ade"},{"name":"kibana.yml","path":"examples/elk-amis/kibana/config/kibana.yml","sha":"852b146720a4ab3fc1380beacce6f77851017b51"}]},{"name":"kibana-install-steps.sh","path":"examples/elk-amis/kibana/kibana-install-steps.sh","sha":"1dc5d1dd094d0533947c53da510cb5d2139556bf"},{"name":"kibana.json","path":"examples/elk-amis/kibana/kibana.json","sha":"568c21c4d7c3eb056710155a3ab50a12b4d16378"}]},{"name":"logstash","children":[{"name":"README.md","path":"examples/elk-amis/logstash/README.md","sha":"b1035b8b3d34f28000e329171ba65275b1fac258"},{"name":"config","children":[{"name":"jvm.options","path":"examples/elk-amis/logstash/config/jvm.options","sha":"6017dd218501163070f950077ce6a1e9cff8f3a4"},{"name":"logstash-ssl.yml","path":"examples/elk-amis/logstash/config/logstash-ssl.yml","sha":"46e72285a2074ceba2313f726db5eb0db6dfa414"},{"name":"logstash.yml","path":"examples/elk-amis/logstash/config/logstash.yml","sha":"46e72285a2074ceba2313f726db5eb0db6dfa414"},{"name":"pipeline-ssl.conf","path":"examples/elk-amis/logstash/config/pipeline-ssl.conf","sha":"10c09944749f51350709b628eda2f84e8345e4c5"},{"name":"pipeline.conf","path":"examples/elk-amis/logstash/config/pipeline.conf","sha":"4a68c41fc5e86106915cb1c91ec39bd4c0cd2b55"}]},{"name":"logstash-install-steps.sh","path":"examples/elk-amis/logstash/logstash-install-steps.sh","sha":"7fd979a06f44c76a70745fd014895ead31f6b693"},{"name":"logstash.json","path":"examples/elk-amis/logstash/logstash.json","sha":"558130f7b83282583a5e32ce1b0dfe820f2caf18"}]}]},{"name":"elk-multi-cluster","children":[{"name":"README.md","path":"examples/elk-multi-cluster/README.md","sha":"ca696b69e95a04bbee5ea51c1b411131d0a47b33","toggled":true},{"name":"main.tf","path":"examples/elk-multi-cluster/main.tf","sha":"191b70e3e2f50c4b89fdc1ad5eef68117386c6c0"},{"name":"outputs.tf","path":"examples/elk-multi-cluster/outputs.tf","sha":"c43e3d202d3a8240c851aadba987c7a3f59908f1"},{"name":"user-data","children":[{"name":"app-server","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/app-server/user-data.sh","sha":"c369aff91cc88d849afc3eac2e00b641c8697e84"}]},{"name":"elastalert","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/elastalert/user-data.sh","sha":"48181b3b07a3aec5e5b808b4101a9ef447eed37b"}]},{"name":"elasticsearch","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/elasticsearch/user-data.sh","sha":"55678f9678e93453686c629e364f82ebacdb83df"}]},{"name":"kibana","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/kibana/user-data.sh","sha":"cb5337f5e4c5a8029554a1af33982f963e4b741b"}]},{"name":"logstash","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/logstash/user-data.sh","sha":"4a74a7ca4c180c4e53fa03e8d1d56184adedfff1"}]}]},{"name":"vars.tf","path":"examples/elk-multi-cluster/vars.tf","sha":"812eab62300f0fab7cd758b81bb7c830789e908e"}],"toggled":true},{"name":"elk-single-cluster","children":[{"name":"README.md","path":"examples/elk-single-cluster/README.md","sha":"f0842ece87a00338900f35f17b199604d4e46e0c"},{"name":"main.tf","path":"examples/elk-single-cluster/main.tf","sha":"5b6c0e2bfd58c7db42912edfb8eca7b95bb6dafa"},{"name":"outputs.tf","path":"examples/elk-single-cluster/outputs.tf","sha":"c12a0b236e6fcbf5881cb60276d62a432dadb87a"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/elk-single-cluster/user-data/user-data.sh","sha":"936dd8fb14577667d5aa4f1625a26ad7debf0ef5"}]},{"name":"vars.tf","path":"examples/elk-single-cluster/vars.tf","sha":"a67bfbaabbca1a1f25865570d9cc1a70d369485b"}]}],"toggled":true},{"name":"modules","children":[{"name":"auto-discovery","children":[{"name":"README.md","path":"modules/auto-discovery/README.md","sha":"f4de11c428f8fefd962220e7fbc4b976b4f838fe"},{"name":"bin","children":[{"name":"auto-discovery","path":"modules/auto-discovery/bin/auto-discovery","sha":"d3d8a818291a3e898b8802569a176ed1afe18725"}]},{"name":"install.sh","path":"modules/auto-discovery/install.sh","sha":"99a2f2fb2cbfaa31e752cbcc6495e5b755f034f2"}]},{"name":"beats-iam-policies","children":[{"name":"main.tf","path":"modules/beats-iam-policies/main.tf","sha":"2b3b4462aa4b3ed81068f4cea15e686d5a4b322c"},{"name":"vars.tf","path":"modules/beats-iam-policies/vars.tf","sha":"52873644e20b4cafdcc0f33dc11db99f1fa9b586"}]},{"name":"elastalert-iam-policies","children":[{"name":"README.md","path":"modules/elastalert-iam-policies/README.md","sha":"4dc09bec085bc821efa8052578b54cc4c321ea67"},{"name":"main.tf","path":"modules/elastalert-iam-policies/main.tf","sha":"989f18bad613464d4086df3305087650cf3a18bb"},{"name":"variables.tf","path":"modules/elastalert-iam-policies/variables.tf","sha":"bc55d045eba3b8df1b08f78a4657cae25ee0ea73"}]},{"name":"elastalert-security-group-rules","children":[{"name":"README.md","path":"modules/elastalert-security-group-rules/README.md","sha":"dc33e8391d9958ef1915f6a4e2104df391e2d320"},{"name":"main.tf","path":"modules/elastalert-security-group-rules/main.tf","sha":"519b25d67eb73a1587b8ad0556bba585230d1440"},{"name":"variables.tf","path":"modules/elastalert-security-group-rules/variables.tf","sha":"0d87d8de7864335c546406e1047a680f14e6ec73"}]},{"name":"elastalert","children":[{"name":"README.md","path":"modules/elastalert/README.md","sha":"8ee7b607def1754891d8ca65dde5a22552787ebb"},{"name":"main.tf","path":"modules/elastalert/main.tf","sha":"18ac92569874242356c83986ebab5f36df6f62bc"},{"name":"outputs.tf","path":"modules/elastalert/outputs.tf","sha":"aa5817acc77a75b75f4ea4acc686332c5b865618"},{"name":"vars.tf","path":"modules/elastalert/vars.tf","sha":"b51689b7b7dd8df9def896e3fb0ba1ad239a7fcc"}]},{"name":"elasticsearch-cluster-backup","children":[{"name":"README.md","path":"modules/elasticsearch-cluster-backup/README.md","sha":"6b3689afcf27cc5cde7a4048e125e97c7d82189b"},{"name":"backup","children":[{"name":"index.js","path":"modules/elasticsearch-cluster-backup/backup/index.js","sha":"6d3ff6a2a58d70dccd55399ba8bf0e0270140279"}]},{"name":"main.tf","path":"modules/elasticsearch-cluster-backup/main.tf","sha":"f0baf650b3eca0ed959b51eb2aba759da30b1ef0"},{"name":"notification","children":[{"name":"index.js","path":"modules/elasticsearch-cluster-backup/notification/index.js","sha":"8e92e6b387974c4f0c6f007b8e3555d64e984202"}]},{"name":"outputs.tf","path":"modules/elasticsearch-cluster-backup/outputs.tf","sha":"ce7c4ee206984e74253ad6bb0bf25cd4a3965053"},{"name":"vars.tf","path":"modules/elasticsearch-cluster-backup/vars.tf","sha":"93c34c8c22ecfa0c9498e450df4886996b4bcfa1"}]},{"name":"elasticsearch-cluster-restore","children":[{"name":"README.md","path":"modules/elasticsearch-cluster-restore/README.md","sha":"11f4f63105657b323a60828ff155c22aaa0ebfbf"},{"name":"main.tf","path":"modules/elasticsearch-cluster-restore/main.tf","sha":"d74982003b28e2dca1ef09d48a1acb612031b63b"},{"name":"notification","children":[{"name":"index.js","path":"modules/elasticsearch-cluster-restore/notification/index.js","sha":"4e441662f92edf384c0c92a96ea5e1a7e1b0b24d"}]},{"name":"outputs.tf","path":"modules/elasticsearch-cluster-restore/outputs.tf","sha":"92454e1cf45fcc57f60c09eb8b31064efb460726"},{"name":"restore","children":[{"name":"index.js","path":"modules/elasticsearch-cluster-restore/restore/index.js","sha":"7d84ea278f01b64310fbc33d18c86a2a73b60573"}]},{"name":"vars.tf","path":"modules/elasticsearch-cluster-restore/vars.tf","sha":"b1c5309ac8d80d86b7ad12912fbf003d7177028f"}]},{"name":"elasticsearch-cluster","children":[{"name":"README.md","path":"modules/elasticsearch-cluster/README.md","sha":"3af5458f9573916635427a990fe0855d4c008f47"},{"name":"main.tf","path":"modules/elasticsearch-cluster/main.tf","sha":"6cce0fa70166851bf8da3af72a04fa38f5f5b989"},{"name":"outputs.tf","path":"modules/elasticsearch-cluster/outputs.tf","sha":"0231e24415e13f62f29d8abefe0398ddb6d0fa04"},{"name":"variables.tf","path":"modules/elasticsearch-cluster/variables.tf","sha":"8b680c1329382792bee8415ff36bd1314675a4c9"}]},{"name":"elasticsearch-iam-policies","children":[{"name":"README.md","path":"modules/elasticsearch-iam-policies/README.md","sha":"57a7117828cff0d9d5bdf7c9fe3af050e55e4aa0"},{"name":"main.tf","path":"modules/elasticsearch-iam-policies/main.tf","sha":"cdb51d7b73acb7e3689e0fa5ab554abc8d576081"},{"name":"variables.tf","path":"modules/elasticsearch-iam-policies/variables.tf","sha":"c612a588acea6049cfdb930209854e253b549729"}]},{"name":"elasticsearch-security-group-rules","children":[{"name":"README.md","path":"modules/elasticsearch-security-group-rules/README.md","sha":"5f0b25abeaa80973e4839626ac96ff306291e759"},{"name":"main.tf","path":"modules/elasticsearch-security-group-rules/main.tf","sha":"fd06486758030a63372610631b3f88d25b7f0cd9"},{"name":"outputs.tf","path":"modules/elasticsearch-security-group-rules/outputs.tf","sha":"e69de29bb2d1d6434b8b29ae775ad8c2e48c5391"},{"name":"variables.tf","path":"modules/elasticsearch-security-group-rules/variables.tf","sha":"7f055aa865ca315861e7a69f2d7741ab7883fddf"}]},{"name":"install-collectd","children":[{"name":"README.md","path":"modules/install-collectd/README.md","sha":"3f9e7dedcca0cc1b67812309594e5d57f41535f0"},{"name":"install.sh","path":"modules/install-collectd/install.sh","sha":"4090c1e8b94dd1a4d30ce68da03b73c281e53a3a"}]},{"name":"install-elastalert","children":[{"name":"README.md","path":"modules/install-elastalert/README.md","sha":"d228d557e6792eba4813df9c0ac6e454fc7c53d7"},{"name":"install.sh","path":"modules/install-elastalert/install.sh","sha":"99f9f170e2c9f3fc31a0f5a92a6a83da5c2fe431"}]},{"name":"install-elasticsearch","children":[{"name":"README.md","path":"modules/install-elasticsearch/README.md","sha":"ff4e35e43bb505354640bca43a12ef4e6da886f0"},{"name":"install.sh","path":"modules/install-elasticsearch/install.sh","sha":"54db7090509f2e7eb56659de74480e2f20422870"}]},{"name":"install-filebeat","children":[{"name":"README.md","path":"modules/install-filebeat/README.md","sha":"cb64af493f471c37fe427070fce0a95617082dd7"},{"name":"install.sh","path":"modules/install-filebeat/install.sh","sha":"b38f01c2caff489023c1f9a29fa8ea062e17ee58"}]},{"name":"install-kibana","children":[{"name":"README.md","path":"modules/install-kibana/README.md","sha":"a0320bbaf4cf83b6133af8df620053420f0c9d25"},{"name":"install.sh","path":"modules/install-kibana/install.sh","sha":"2109aefeccc5c1028a881da3bf9b5fa36c62f6d8"}]},{"name":"install-logstash","children":[{"name":"README.md","path":"modules/install-logstash/README.md","sha":"fddf142ed7b9608f76eb9d07cbd9ffa0a5443dad"},{"name":"install.sh","path":"modules/install-logstash/install.sh","sha":"a6a44552a2ec903558d494bf4151f32f3374f786"}]},{"name":"kibana-cluster","children":[{"name":"README.md","path":"modules/kibana-cluster/README.md","sha":"4c4860d81ed7d630a77f8c7700da1e9588c82b5f"},{"name":"main.tf","path":"modules/kibana-cluster/main.tf","sha":"7f9a9ecd3b4b4aaf8efcf92108fc123f4fb7ee5d"},{"name":"outputs.tf","path":"modules/kibana-cluster/outputs.tf","sha":"f32e707371481d34b2b84d4ef7f205afaa04f2d4"},{"name":"variables.tf","path":"modules/kibana-cluster/variables.tf","sha":"8306891e82fd960d9587851d3e44c6c1579685c3"}]},{"name":"kibana-security-group-rules","children":[{"name":"README.md","path":"modules/kibana-security-group-rules/README.md","sha":"c35fc72e41001065470a412a0ad9756eb2e84310"},{"name":"main.tf","path":"modules/kibana-security-group-rules/main.tf","sha":"59a0df1a9f5ceb464efa6ee73b39ecaf11240c2c"},{"name":"outputs.tf","path":"modules/kibana-security-group-rules/outputs.tf","sha":"e69de29bb2d1d6434b8b29ae775ad8c2e48c5391"},{"name":"variables.tf","path":"modules/kibana-security-group-rules/variables.tf","sha":"b89d5c7f5f1d1b0e9b5e1d239a1dec9a4ea34486"}]},{"name":"load-balancer-alb-target-group","children":[{"name":"README.md","path":"modules/load-balancer-alb-target-group/README.md","sha":"66c957276dc5cf35a341e0fd563bf394fd4b3d22"},{"name":"main.tf","path":"modules/load-balancer-alb-target-group/main.tf","sha":"7cb9fede7e1bb3c06907f90a81c64b02d78900f4"},{"name":"outputs.tf","path":"modules/load-balancer-alb-target-group/outputs.tf","sha":"cdf904445cf5099ccea5cc8820d198a1ffba4283"},{"name":"variables.tf","path":"modules/load-balancer-alb-target-group/variables.tf","sha":"4a262b0abde2721f9e3a5edeb6a98046009b9df3"}]},{"name":"logstash-cluster","children":[{"name":"README.md","path":"modules/logstash-cluster/README.md","sha":"dbc17e8fc3c4d9a61661ca59b934295ce0bcc6a5"},{"name":"main.tf","path":"modules/logstash-cluster/main.tf","sha":"f138bbca6e0bc6858f8d323f134e95b71031747d"},{"name":"outputs.tf","path":"modules/logstash-cluster/outputs.tf","sha":"218a4f0042d52ea1673de0c4f78857a4831143a6"},{"name":"vars.tf","path":"modules/logstash-cluster/vars.tf","sha":"bff728633e0f853502c3a15b4f9311b5b9d2d41f"}]},{"name":"logstash-iam-policies","children":[{"name":"README.md","path":"modules/logstash-iam-policies/README.md","sha":"3714f39713885c9698daa2973a7891d0a6fb982f"},{"name":"main.tf","path":"modules/logstash-iam-policies/main.tf","sha":"7b553a45925c6ea587b95e8de3d05f546d03e278"},{"name":"vars.tf","path":"modules/logstash-iam-policies/vars.tf","sha":"865c74386084c6eff1f46cb103aa847a6ad0da68"}]},{"name":"logstash-security-group-rules","children":[{"name":"README.md","path":"modules/logstash-security-group-rules/README.md","sha":"e96766f09754cf166f53debb653ea278613a9c8e"},{"name":"main.tf","path":"modules/logstash-security-group-rules/main.tf","sha":"5ecab3d1c9d60cf246a98f9e36296bc518caa9bb"},{"name":"outputs.tf","path":"modules/logstash-security-group-rules/outputs.tf","sha":"3641c561d0e23ef02fa83319ddafa8d1f9f813d2"},{"name":"vars.tf","path":"modules/logstash-security-group-rules/vars.tf","sha":"9e26c219ddad783caf509581fae84953eae60fe9"}]},{"name":"run-collectd","children":[{"name":"README.md","path":"modules/run-collectd/README.md","sha":"783453e215c5907681e77ab4b48e64cad42ab989"},{"name":"bin","children":[{"name":"run-collectd","path":"modules/run-collectd/bin/run-collectd","sha":"7f7971d8b564a84a251d4c253c42f9025b5a1cf7"}]},{"name":"install.sh","path":"modules/run-collectd/install.sh","sha":"d2d8126f6fc5ba95a4e12242aa6ffe9bdc945ba6"}]},{"name":"run-elastalert","children":[{"name":"README.md","path":"modules/run-elastalert/README.md","sha":"e52cf808ab755b2d56a392ee8f525a0651963472"},{"name":"bin","children":[{"name":"run-elastalert","path":"modules/run-elastalert/bin/run-elastalert","sha":"cccab0f5cf41600c1b8ad48da5173db97965440f"}]},{"name":"install.sh","path":"modules/run-elastalert/install.sh","sha":"308f895a2366fc0cd8e6669229f6928a3d888f13"}]},{"name":"run-elasticsearch","children":[{"name":"README.md","path":"modules/run-elasticsearch/README.md","sha":"8a94a393f612da88f6b1b0a50a2e66743830abac"},{"name":"bin","children":[{"name":"run-elasticsearch","path":"modules/run-elasticsearch/bin/run-elasticsearch","sha":"e8e21b99ca07e03ce807ad537fbd9c3858912713"}]},{"name":"install.sh","path":"modules/run-elasticsearch/install.sh","sha":"c66e6da3d46195ac94d68eed18ed301ca7e0eb51"}]},{"name":"run-filebeat","children":[{"name":"README.md","path":"modules/run-filebeat/README.md","sha":"07fffbedecf8232d163a6ea985669497b11f3f7b"},{"name":"bin","children":[{"name":"run-filebeat","path":"modules/run-filebeat/bin/run-filebeat","sha":"3226cfa14477efe5c94c2a660b41ff792092d119"}]},{"name":"install.sh","path":"modules/run-filebeat/install.sh","sha":"cd890712803c3961ada86acf1e13b4c299402fb6"}]},{"name":"run-kibana","children":[{"name":"README.md","path":"modules/run-kibana/README.md","sha":"57d636cd26e4af0b51364b4ffb70b382f76314e4"},{"name":"bin","children":[{"name":"run-kibana","path":"modules/run-kibana/bin/run-kibana","sha":"0f60bca1be5eebc5594de9e8eab162aa979d4a17"}]},{"name":"install.sh","path":"modules/run-kibana/install.sh","sha":"e23b447832fce06ad32ce654f94a182a521e2e38"}]},{"name":"run-logstash","children":[{"name":"README.md","path":"modules/run-logstash/README.md","sha":"c3d92b73c36bf63b4bd84356f2f4b781aa34c172"},{"name":"bin","children":[{"name":"run-logstash","path":"modules/run-logstash/bin/run-logstash","sha":"60dc8d906d8181e5a94c3ad18b5b27e53d513cf1"}]},{"name":"install.sh","path":"modules/run-logstash/install.sh","sha":"0a30f2adc2d0765b5581274263e12ce2e688b623"}]}]},{"name":"test","children":[{"name":"Gopkg.lock","path":"test/Gopkg.lock","sha":"d726b0d9bf25eae861d500289fb65ec2ad356105"},{"name":"Gopkg.toml","path":"test/Gopkg.toml","sha":"630429bc686d90b570c81062c7b9c0c925c5fdd3"},{"name":"README.md","path":"test/README.md","sha":"819e790d15767c851bfc438f0a1413ed2bde7d3b"},{"name":"elasticsearch_aws_simple_test.go","path":"test/elasticsearch_aws_simple_test.go","sha":"18c119d40504e2066377d334739150dbc6711a4d"},{"name":"elasticsearch_docker_test.go","path":"test/elasticsearch_docker_test.go","sha":"2aad9485f5da9cd84305d427a5d5e4a615f7a889"},{"name":"elk_aio_test.go","path":"test/elk_aio_test.go","sha":"ef12f5aa440b29507c695c4e95a96df42e07cacf"},{"name":"elk_end_to_end_test.go","path":"test/elk_end_to_end_test.go","sha":"536f9e44142c9ae528d13c5bc01c22118f4f4d77"},{"name":"notes.md","path":"test/notes.md","sha":"664915d4b948ccd194ffda80854f47c191904e3d"},{"name":"test_helpers.go","path":"test/test_helpers.go","sha":"df39c94ac081dee152740bf5809a8950efa3370f"},{"name":"test_helpers_keystore.go","path":"test/test_helpers_keystore.go","sha":"1e189342859a6561ec9e75009ecb2e55dc42fc23"},{"name":"tls_helpers.go","path":"test/tls_helpers.go","sha":"dd131a0d95b024beffa40c3ae538333ef2663891"}]}]},"detailsContent":"<h1 class=\"preview__body--title\" id=\"elk-multi-cluster-example\">ELK Multi Cluster Example</h1><div class=\"preview__body--border\"></div><p>In this example we demonstrate how to use our ELK modules to run an ELK stack consisting of several Auto Scaling Groups (ASGs)\nthat each run an ELK component (Kibana, Logstash, Elasticsearch...)</p>\n<h2 class=\"preview__body--subtitle\" id=\"what-resources-does-this-example-deploy\">What resources does this example deploy?</h2>\n<ol>\n<li>An <a href=\"/repos/v0.5.0/package-elk/modules/elasticsearch-cluster\" class=\"preview__body--description--blue\">elasticsearch-cluster</a></li>\n<li>A <a href=\"/repos/v0.5.0/package-elk/modules/kibana-cluster\" class=\"preview__body--description--blue\">kibana-cluster</a></li>\n<li>A <a href=\"/repos/v0.5.0/package-elk/modules/logstash-cluster\" class=\"preview__body--description--blue\">logstash-cluster</a></li>\n<li>A simple <em>app-server</em> where we will run <a href=\"/repos/v0.5.0/package-elk/modules/run-filebeat\" class=\"preview__body--description--blue\">filebeat</a> and <a href=\"/repos/v0.5.0/package-elk/modules/run-collectd\" class=\"preview__body--description--blue\">CollectD</a></li>\n<li>An ASG of one node <sup><a href=\"#whyjustonenode\" class=\"preview__body--description--blue\">1</a></sup> running <a href=\"/repos/v0.5.0/package-elk/modules/elastalert\" class=\"preview__body--description--blue\">elastalert</a>.</li>\n<li>An <a href=\"/repos/module-load-balancer\" class=\"preview__body--description--blue\">Application Load Balancer</a></li>\n<li>A CloudWatch Log Group</li>\n<li>A CloudWatch Log Stream</li>\n<li>An S3 bucket for Cloudtrail logs</li>\n<li>An SNS topic - this is where ElastAlert will send our sample alert notifications.</li>\n<li>A Route53 Alias Record that will attach a subdomain name to an existing HostedZone.</li>\n</ol>\n<h2 class=\"preview__body--subtitle\" id=\"how-do-i-get-this-example-deployed\">How do I get this example deployed?</h2>\n<ol>\n<li><code>git clone</code> this repo to your computer.</li>\n<li>You'll need to build AMIs of each of the ELK components referenced above and note down the <code>ami_id</code>s of the AMIs\nyou've built. Look <a href=\"/repos/v0.5.0/package-elk/examples/elk-amis\" class=\"preview__body--description--blue\">here</a> for detailed guides on how to build the AMIs</li>\n<li>Install <a href=\"https://www.terraform.io/\" class=\"preview__body--description--blue\" target=\"_blank\">Terraform</a>.</li>\n<li>Open <code>vars.tf</code>, set the environment variables specified at the top of the file, and fill in any other variables that\ndon't have a default, including putting the AMI Ids you noted down from above into the <code>[elk-component]_ami_id</code> variables.</li>\n<li>Run <code>terraform init</code>.</li>\n<li>Run <code>terraform apply</code>.</li>\n</ol>\n<h2 class=\"preview__body--subtitle\" id=\"connecting-to-the-cluster\">Connecting to the cluster</h2>\n<ul>\n<li>Once the cluster has successfully deployed, you should be able to see the main URL in the <code>alb_url</code>\noutput variable.</li>\n<li>To access the Kibana UI go to the URL provided in <code>alb_url</code> output variable. Your URL will look something like\nthis: <code>http://[SubdomainYouSet].[YourHostedZone]/</code></li>\n<li>Elasticsearch will be accessible at: <code>http://[SubdomainYouSet].[YourHostedZone]:[elasticsearch_api_port(9200 by default)]</code>.</li>\n</ul>\n<h2 class=\"preview__body--subtitle\" id=\"why-deploy-an-alb\">Why deploy an ALB?</h2>\n<p>We deploy an ALB in order to simplify load balancing and routing between the nodes of the ELK cluster. The ALB also helps\nsolve the issues around resource discovery in a convenient and simple way. The ALB only deals with HTTP and HTTPS\ntraffic. This is an important detail to keep in mind as ELK components that use custom (non-http) protocols, like\nFilebeat, cannot use the ALB for load balancing or discovery.</p>\n<h2 class=\"preview__body--subtitle\" id=\"how-we-handle-non-http-discovery\">How we handle non-HTTP discovery?</h2>\n<p>Given the ALB's HTTP only limitation, we require a different solution for dealing with discovery for Filebeat, a component\nwhich uses a custom protocol for sending messages to Elasticsearch.</p>\n<h3 class=\"preview__body--subtitle\" id=\"why-we-dont-use-an-nlb\">Why we don't use an NLB</h3>\n<p>A Network Load Balancer (NLB) was originally how we solved the load balancing and discovery problem for <em>all</em> ELK cluster\nmembers. In fact, we tried only using an ALB without having to deploy an ALB. Unfortunately we ran into some NLB Limitations\nwhich precluded its use:</p>\n<ul>\n<li>The NLB can't <a href=\"https://forums.aws.amazon.com/thread.jspa?threadID=265344\" class=\"preview__body--description--blue\" target=\"_blank\">route back requests to the same node that initiated the request</a>.</li>\n<li>An internal NLB in a private subnet can't be accessed from a peered VPC. This limitation doesn't affect <strong>this</strong> example\ncode because we are deploying a public NLB into a public subnet and not trying to access it from a peered VPC. In a\nproduction environment, this is no longer the case and makes it impossible to access the NLB.</li>\n</ul>\n<h3 class=\"preview__body--subtitle\" id=\"auto-discovery\">Auto-discovery</h3>\n<h4 id=\"current-implementation\">Current implementation</h4>\n<p>To work around the issues we encountered we created a simple <a href=\"/repos/v0.5.0/package-elk/modules/auto-discovery\" class=\"preview__body--description--blue\">auto-discovery</a> module that will\nuse AWS APIs to get the IP addresses of all instances matching a given tag. The current implementation of this script\nwill use a regular expression (passed as a parameter when invoking the script) to find/replace newly discovered IP addresses\nin a config file (also passed as a parameter). Finally, the script will automatically restart whatever application it is\nperforming discovery for (the application's systemd service name is also passed as a script parameter)</p>\n<p><strong>LIMITATION:</strong> Unfortunately, the approach described above can't work if the application for which we are doing discovery\nis configured to do full SSL certificate verification (ie: hostname verification). Since our approach looks up randomly\nassigned IP addresses, there's no way we could generate an SSL certificate that encapsulates all of the possible IP addresses\nthat may be found. For this reason, we have to disable SSL hostname verification when using <a href=\"/repos/v0.5.0/package-elk/modules/auto-discovery\" class=\"preview__body--description--blue\">auto-discovery</a>.</p>\n<h4 id=\"future-improvements\">Future Improvements</h4>\n<p>To work around the limitation described above, we will be introducing updates to the <a href=\"/repos/v0.5.0/package-elk/modules/auto-discovery\" class=\"preview__body--description--blue\">auto-discovery</a> module\nthat will update local <code>/etc/hosts</code> or run a local instance of <a href=\"http://www.thekelleys.org.uk/dnsmasq/doc.html\" class=\"preview__body--description--blue\" target=\"_blank\">Dnsmasq</a>\nin order to make seamless updates to the DNS instead of updating an application's config file and then needing to restart\nthat application.</p>\n<p><a name="whyjustonenode">1</a>: We are using an ASG here even though we will only be running one node because we want the\nASG to handle automatically restarting our EC2 instance if it ever crashes.</p>\n","repoName":"package-elk","repoRef":"v0.5.1","serviceDescriptor":{"serviceName":"Elasticsearch (self-hosted)","serviceRepoName":"package-elk","serviceRepoOrg":"gruntwork-io","cloudProviders":["aws"],"description":"Deploy a self-hosted Elasticsearch cluster. Supports automatic bootstrap, zero-downtime rolling deployment, auto healing, backup, and recovery.","imageUrl":"elk.png","licenseType":"subscriber","technologies":["Terraform","Bash","JavaScript"],"compliance":[],"tags":[""]},"serviceCategoryName":"NoSQL","fileName":"README.md","filePath":"/examples/elk-multi-cluster","title":"Repo Browser: Elasticsearch (self-hosted)","description":"Browse the repos in the Gruntwork Infrastructure as Code Library."}