This folder defines a Terraform module, which you can use in your
code by adding a module configuration and setting its source parameter to URL of this folder:
module"logstash_cluster" {
# TODO: replace <VERSION> with the latest version from the releases page: https://github.com/gruntwork-io/package-elk/releases
source = "github.com/gruntwork-io/package-elk//modules/logstash-cluster?ref=<VERSION>"# Specify the ID of the Logstash AMI. You should build this using the scripts in the install-logstash module
ami_id = "ami-abcd1234"# Configure and start Logstash during boot.
user_data = <<-EOF
#!/bin/bash
/usr/share/logstash/bin/run-logstash
EOF
# ... See vars.tf for the other parameters you must define for the logstash-cluster module
}
Note the following parameters:
source: Use this parameter to specify the URL of the logstash-cluster module. The double slash (//) is
intentional and required. Terraform uses it to specify subfolders within a Git repo (see module
sources). The ref parameter specifies a specific Git tag in
this repo. That way, instead of using the latest version of this module from the master branch, which
will change every time you run Terraform, you're using a fixed version of the repo.
ami_id: Use this parameter to specify the ID of an Logstash Amazon Machine Image
(AMI) to deploy on each server in the cluster. You
should install Logstash on this AMI using the following install-logstash module.
user_data: Use this parameter to specify a User
Data script that each
server will run during boot. This is where you can use the run-logstash.
This module runs Logstash on top of an Auto Scaling Group (ASG). Typically,
you should run the ASG with multiple Instances spread across multiple Availability
Zones. Each of the EC2
Instances should be running an AMI that has Logstash installed via the install-logstash script.
You pass in the ID of the AMI to run using the ami_id input parameter.
Load Balancer
We use a Network Load Balancer (1)
so that we can perform ongoing health checks on each Logstash node, and (2) so that Filebeat can access the Logstash cluster
via a single endpoint which will forward to a live Kibana endpoint at random.
Security Group
Each EC2 Instance in the ASG has a Security Group that allows minimal connectivity:
All outbound requests
Inbound SSH access from the CIDR blocks and security groups you specify
The ID of the security group is exported as an output variable, which you can use with the logstash-security-group-rules module to open up all the ports necessary for Logstash.
If you want to deploy a new version of Logstash across the cluster, the best way to do that is to:
Rolling deploy:
Build a new AMI.
Set the ami_id parameter to the ID of the new AMI.
Run terraform apply.
Because the logstash-cluster module uses the Gruntwork server-group modules under the hood, running
terraform apply will automatically perform a zero-downtime rolling deployment. Specifically, one EC2 Instance at a time will be terminated, a new EC2 Instance will spawn in its place, and only once the new EC2 Instance passes the Load
Balancer Health Checks will the next EC2 Instance be rolled out.
Note that there will be a brief period of time during which EC2 Instances based on both the old ami_id and
new ami_id will be running.
New cluster:
Build a new AMI.
Create a totally new ASG using the logstash-cluster module with the ami_id set to the new AMI, but all
other parameters the same as the old cluster.
Wait for all the nodes in the new ASG to join the cluster and catch up on replication.
Remove each of the nodes from the old cluster.
Remove the old ASG by removing that logstash-cluster module from your code.
Security
Here are some of the main security considerations to keep in mind when using this module:
This module attaches a security group to each EC2 Instance that allows inbound requests as follows:
SSH: For the SSH port (default: 22), you can use the allowed_ssh_cidr_blocks parameter to control the list of CIDR blocks that will be allowed access. You can use
the allowed_inbound_ssh_security_group_ids parameter to control the list of source Security Groups that will be
allowed access.
The ID of the security group is exported as an output variable, which you can use with the logstash-security-group-rules modules to open up all the ports necessary for Logstash and the respective.
SSH access
You can associate an EC2 Key Pair with each
of the EC2 Instances in this cluster by specifying the Key Pair's name in the ssh_key_name variable. If you don't
want to associate a Key Pair with these servers, set ssh_key_name to an empty string.
Questions? Ask away.
We're here to talk about our services, answer any questions, give advice, or just to chat.
{"treedata":{"name":"root","toggled":true,"children":[{"name":".circleci","children":[{"name":"config.yml","path":".circleci/config.yml","sha":"c2cff2a8764e0fabad7a8408ba9e42113cc23487"}]},{"name":".gitignore","path":".gitignore","sha":"f2438833ea6b1bc45e125b598cb1e08cbd274e1b"},{"name":"CODEOWNERS","path":"CODEOWNERS","sha":"cef671f8c0aee235f6d8cf886e421cfab3b51729"},{"name":"CONTRIBUTING.md","path":"CONTRIBUTING.md","sha":"492bef006a63f2c3189fd171116eb3f900c6cd02"},{"name":"LICENSE.md","path":"LICENSE.md","sha":"a2cf01ecdd725fddd718ab91c80c115882c94f3c"},{"name":"README.md","path":"README.md","sha":"a528d869a6d4e60a4639921169d18041b0ce1a1e"},{"name":"_docs","children":[{"name":"elk-architecture.png","path":"_docs/elk-architecture.png","sha":"a7585c6f40eb0a7740b34f00d412bc2e37373266"}]},{"name":"examples","children":[{"name":"elasticsearch-docker","children":[{"name":"local-mocks","children":[{"name":"entrypoint.sh","path":"examples/elasticsearch-docker/local-mocks/entrypoint.sh","sha":"8e298671493441b6d7548f619a6155a408677d99"}]},{"name":"non-ssl","children":[{"name":".env","path":"examples/elasticsearch-docker/non-ssl/.env","sha":"bc82d24eea229df2ba77f0fdf1f6e1c6f6a8e532"},{"name":"README.md","path":"examples/elasticsearch-docker/non-ssl/README.md","sha":"ba7a1c035faf9791430ace13275d2d6b9b616b75"},{"name":"docker-compose.yml","path":"examples/elasticsearch-docker/non-ssl/docker-compose.yml","sha":"e7547917f574679857f800b24f86bd573671931e"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/elasticsearch-docker/non-ssl/user-data/user-data.sh","sha":"81df178b3080a40a9e03ff7e94f1ef5db7d57f98"}]}]},{"name":"ssl","children":[{"name":".env","path":"examples/elasticsearch-docker/ssl/.env","sha":"bc82d24eea229df2ba77f0fdf1f6e1c6f6a8e532"},{"name":"README.md","path":"examples/elasticsearch-docker/ssl/README.md","sha":"ba7a1c035faf9791430ace13275d2d6b9b616b75"},{"name":"docker-compose.yml","path":"examples/elasticsearch-docker/ssl/docker-compose.yml","sha":"25e2b036c7e672a8ffc4b9559ab53f8d77441f99"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/elasticsearch-docker/ssl/user-data/user-data.sh","sha":"fc95c5a93069b251aca3ec2ae70a2be90231fcd9"}]}]}]},{"name":"elasticsearch-only-cluster","children":[{"name":"README.md","path":"examples/elasticsearch-only-cluster/README.md","sha":"5bdbc120e354d55895febb193390b8ba4440e9ac"},{"name":"main.tf","path":"examples/elasticsearch-only-cluster/main.tf","sha":"491b09a2683ba577be29cd1ca226c3b75ed55183"},{"name":"outputs.tf","path":"examples/elasticsearch-only-cluster/outputs.tf","sha":"989bd7f7dfdb7fa4356d9926731f2849195ed345"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/elasticsearch-only-cluster/user-data/user-data.sh","sha":"55678f9678e93453686c629e364f82ebacdb83df"}]},{"name":"vars.tf","path":"examples/elasticsearch-only-cluster/vars.tf","sha":"6c3db8f2b043c53d04f7278a076230b4ce06308a"}]},{"name":"elk-amis","children":[{"name":"README.md","path":"examples/elk-amis/README.md","sha":"4e58b3a2e6be086188621bee400ff750f02436e8"},{"name":"all-in-one","children":[{"name":"README.md","path":"examples/elk-amis/all-in-one/README.md","sha":"42799bc50b4f41eb9e27ed79f1b5624b812b81b2"},{"name":"all-in-one.json","path":"examples/elk-amis/all-in-one/all-in-one.json","sha":"e0b06562c958eae8d91c371c9ebd757446ed79cd"}]},{"name":"app-server","children":[{"name":"README.md","path":"examples/elk-amis/app-server/README.md","sha":"577966ff3d1aa65d69aa43d6b7e1e86968099335"},{"name":"app-server.json","path":"examples/elk-amis/app-server/app-server.json","sha":"d9f6b7c6009f24c60fe210b77f580b58b22562aa"}]},{"name":"collectd","children":[{"name":"README.md","path":"examples/elk-amis/collectd/README.md","sha":"a9fa0d44e853022868368cab446e4aaeb24997ab"},{"name":"collectd-install-steps.sh","path":"examples/elk-amis/collectd/collectd-install-steps.sh","sha":"1fa3c68c3a1318caab275a7eb4bbaf9533b75319"},{"name":"collectd.json","path":"examples/elk-amis/collectd/collectd.json","sha":"b3f883f1cfd1866f08f411ac302ba5db7bf2fde6"},{"name":"config","children":[{"name":"collectd-ssl.conf","path":"examples/elk-amis/collectd/config/collectd-ssl.conf","sha":"ca0469d729e643644f90118e5a639671a3f60d08"},{"name":"collectd.conf","path":"examples/elk-amis/collectd/config/collectd.conf","sha":"74c1b46937039070cd95f64269eef63aabd72805"}]}]},{"name":"elastalert","children":[{"name":"README.md","path":"examples/elk-amis/elastalert/README.md","sha":"9dd33dfd29ddafa1040213466560ffcf8a66944d"},{"name":"aws","children":[{"name":"elastalert-config","children":[{"name":"config-ssl.yml","path":"examples/elk-amis/elastalert/aws/elastalert-config/config-ssl.yml","sha":"f26f8404e40736c5b0b17bc23e6a5de119d0c80f"},{"name":"config.yml","path":"examples/elk-amis/elastalert/aws/elastalert-config/config.yml","sha":"6054d91b8e86df9102fd5fd1fa2c545f7a981d31"}]}]},{"name":"docker","children":[{"name":"elastalert-config","children":[{"name":"config-ssl.yml","path":"examples/elk-amis/elastalert/docker/elastalert-config/config-ssl.yml","sha":"f26f8404e40736c5b0b17bc23e6a5de119d0c80f"},{"name":"config.yml","path":"examples/elk-amis/elastalert/docker/elastalert-config/config.yml","sha":"6054d91b8e86df9102fd5fd1fa2c545f7a981d31"}]}]},{"name":"elastalert-install-steps.sh","path":"examples/elk-amis/elastalert/elastalert-install-steps.sh","sha":"0e5cca36e97e25543d28d2d4d346b80110ceef8d"},{"name":"elastalert-rules","children":[{"name":"example_change.yml","path":"examples/elk-amis/elastalert/elastalert-rules/example_change.yml","sha":"4020886483786ff9a8847159b5750014c5d2b0fb"}]},{"name":"elastalert.json","path":"examples/elk-amis/elastalert/elastalert.json","sha":"419a67dc9d900c05d480b60452dba7aea7a3ee92"}]},{"name":"elasticsearch","children":[{"name":"README.md","path":"examples/elk-amis/elasticsearch/README.md","sha":"f77a938e72424815565ffe16b9ae7787cb430cc7"},{"name":"aws","children":[{"name":"config","children":[{"name":"elasticsearch-ssl.yml","path":"examples/elk-amis/elasticsearch/aws/config/elasticsearch-ssl.yml","sha":"aab4150ebef375ddb962e17c679642f728df538d"},{"name":"elasticsearch.yml","path":"examples/elk-amis/elasticsearch/aws/config/elasticsearch.yml","sha":"0bf97273dce0664ca96ef87151f856162b1a2411"},{"name":"jvm.options","path":"examples/elk-amis/elasticsearch/aws/config/jvm.options","sha":"28dcf519ee339aba5dd15059c1acc4b97846d6ce"}]}]},{"name":"docker","children":[{"name":"config","children":[{"name":"elasticsearch-ssl.yml","path":"examples/elk-amis/elasticsearch/docker/config/elasticsearch-ssl.yml","sha":"705af52bc1e914df93faa919de8a23b973184938"},{"name":"elasticsearch.yml","path":"examples/elk-amis/elasticsearch/docker/config/elasticsearch.yml","sha":"2ac38b9c875dd6352b7a3d7d0fc2c02662d0e9cb"},{"name":"jvm.options","path":"examples/elk-amis/elasticsearch/docker/config/jvm.options","sha":"28dcf519ee339aba5dd15059c1acc4b97846d6ce"}]}]},{"name":"elasticsearch-install-steps.sh","path":"examples/elk-amis/elasticsearch/elasticsearch-install-steps.sh","sha":"7d766638a6a8368b56ae5b81c65f3d202c907811"},{"name":"elasticsearch.json","path":"examples/elk-amis/elasticsearch/elasticsearch.json","sha":"9c91629c2ba2b14d7647bf380532e442343228e9"},{"name":"plugins","children":[{"name":"readonlyrest-1.18.5_es6.8.3.zip","path":"examples/elk-amis/elasticsearch/plugins/readonlyrest-1.18.5_es6.8.3.zip","sha":"92de49ac4f2a7e90b58dd0d04effe068483a5002"}]},{"name":"readonlyrest-config","children":[{"name":"readonlyrest.yml","path":"examples/elk-amis/elasticsearch/readonlyrest-config/readonlyrest.yml","sha":"a5efa565eca562bcc9a33b959c668e5fd7087165"}]}]},{"name":"filebeat","children":[{"name":"README.md","path":"examples/elk-amis/filebeat/README.md","sha":"6b4f4ff0ec66a8031cf4d5dcd86c26cb7b19bb50"},{"name":"config","children":[{"name":"filebeat-ssl.yml","path":"examples/elk-amis/filebeat/config/filebeat-ssl.yml","sha":"24b163e671b4cc49af08726867d8869f105a8ecb"},{"name":"filebeat.yml","path":"examples/elk-amis/filebeat/config/filebeat.yml","sha":"ae4641ee77c89e389728836fca3cac1428214fe2"}]},{"name":"filebeat-install-steps.sh","path":"examples/elk-amis/filebeat/filebeat-install-steps.sh","sha":"71192c99d9e5a1207a7a0f6f0dfa107532a21fe0"},{"name":"filebeat.json","path":"examples/elk-amis/filebeat/filebeat.json","sha":"3d87c3b35089e41f432ff160593e8e2f7ad0854f"}]},{"name":"kibana","children":[{"name":"README.md","path":"examples/elk-amis/kibana/README.md","sha":"b8af363345ba445cb9fbc2b04ec0e7d6b0b08149"},{"name":"config","children":[{"name":"kibana-ssl.yml","path":"examples/elk-amis/kibana/config/kibana-ssl.yml","sha":"9d0761e628a703eec051b31e6cce9d39e62f5ade"},{"name":"kibana.yml","path":"examples/elk-amis/kibana/config/kibana.yml","sha":"852b146720a4ab3fc1380beacce6f77851017b51"}]},{"name":"kibana-install-steps.sh","path":"examples/elk-amis/kibana/kibana-install-steps.sh","sha":"1dc5d1dd094d0533947c53da510cb5d2139556bf"},{"name":"kibana.json","path":"examples/elk-amis/kibana/kibana.json","sha":"568c21c4d7c3eb056710155a3ab50a12b4d16378"}]},{"name":"logstash","children":[{"name":"README.md","path":"examples/elk-amis/logstash/README.md","sha":"227bc8cdc3f55c5ae071bb3c2b5b544aacd1dcc5"},{"name":"config","children":[{"name":"jvm.options","path":"examples/elk-amis/logstash/config/jvm.options","sha":"6017dd218501163070f950077ce6a1e9cff8f3a4"},{"name":"logstash-ssl.yml","path":"examples/elk-amis/logstash/config/logstash-ssl.yml","sha":"46e72285a2074ceba2313f726db5eb0db6dfa414"},{"name":"logstash.yml","path":"examples/elk-amis/logstash/config/logstash.yml","sha":"46e72285a2074ceba2313f726db5eb0db6dfa414"},{"name":"pipeline-ssl.conf","path":"examples/elk-amis/logstash/config/pipeline-ssl.conf","sha":"10c09944749f51350709b628eda2f84e8345e4c5"},{"name":"pipeline.conf","path":"examples/elk-amis/logstash/config/pipeline.conf","sha":"4a68c41fc5e86106915cb1c91ec39bd4c0cd2b55"}]},{"name":"logstash-install-steps.sh","path":"examples/elk-amis/logstash/logstash-install-steps.sh","sha":"7fd979a06f44c76a70745fd014895ead31f6b693"},{"name":"logstash.json","path":"examples/elk-amis/logstash/logstash.json","sha":"558130f7b83282583a5e32ce1b0dfe820f2caf18"}]}]},{"name":"elk-multi-cluster","children":[{"name":"README.md","path":"examples/elk-multi-cluster/README.md","sha":"ca696b69e95a04bbee5ea51c1b411131d0a47b33"},{"name":"main.tf","path":"examples/elk-multi-cluster/main.tf","sha":"bea044afb4be911d51617264221f5a5cb46d449c"},{"name":"outputs.tf","path":"examples/elk-multi-cluster/outputs.tf","sha":"e0516b88a1243a674feadd292225318a2d865f05"},{"name":"user-data","children":[{"name":"app-server","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/app-server/user-data.sh","sha":"b9361ec30397e7e392a53af77ba7b8d1b50f451c"}]},{"name":"elastalert","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/elastalert/user-data.sh","sha":"48181b3b07a3aec5e5b808b4101a9ef447eed37b"}]},{"name":"elasticsearch","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/elasticsearch/user-data.sh","sha":"55678f9678e93453686c629e364f82ebacdb83df"}]},{"name":"kibana","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/kibana/user-data.sh","sha":"cb5337f5e4c5a8029554a1af33982f963e4b741b"}]},{"name":"logstash","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/logstash/user-data.sh","sha":"4a74a7ca4c180c4e53fa03e8d1d56184adedfff1"}]}]},{"name":"vars.tf","path":"examples/elk-multi-cluster/vars.tf","sha":"4e28eb820aa3fbbd19ce8958361e9a039a155439"}]},{"name":"elk-single-cluster","children":[{"name":"README.md","path":"examples/elk-single-cluster/README.md","sha":"f0842ece87a00338900f35f17b199604d4e46e0c"},{"name":"main.tf","path":"examples/elk-single-cluster/main.tf","sha":"5b6c0e2bfd58c7db42912edfb8eca7b95bb6dafa"},{"name":"outputs.tf","path":"examples/elk-single-cluster/outputs.tf","sha":"c12a0b236e6fcbf5881cb60276d62a432dadb87a"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/elk-single-cluster/user-data/user-data.sh","sha":"936dd8fb14577667d5aa4f1625a26ad7debf0ef5"}]},{"name":"vars.tf","path":"examples/elk-single-cluster/vars.tf","sha":"a67bfbaabbca1a1f25865570d9cc1a70d369485b"}]}]},{"name":"modules","children":[{"name":"auto-discovery","children":[{"name":"README.md","path":"modules/auto-discovery/README.md","sha":"f4de11c428f8fefd962220e7fbc4b976b4f838fe"},{"name":"bin","children":[{"name":"auto-discovery","path":"modules/auto-discovery/bin/auto-discovery","sha":"b8d119c933db64ee2fb5f960bb398316c53bf0eb"}]},{"name":"install.sh","path":"modules/auto-discovery/install.sh","sha":"99a2f2fb2cbfaa31e752cbcc6495e5b755f034f2"}]},{"name":"beats-iam-policies","children":[{"name":"main.tf","path":"modules/beats-iam-policies/main.tf","sha":"2b3b4462aa4b3ed81068f4cea15e686d5a4b322c"},{"name":"vars.tf","path":"modules/beats-iam-policies/vars.tf","sha":"52873644e20b4cafdcc0f33dc11db99f1fa9b586"}]},{"name":"elastalert-iam-policies","children":[{"name":"README.md","path":"modules/elastalert-iam-policies/README.md","sha":"4dc09bec085bc821efa8052578b54cc4c321ea67"},{"name":"main.tf","path":"modules/elastalert-iam-policies/main.tf","sha":"989f18bad613464d4086df3305087650cf3a18bb"},{"name":"variables.tf","path":"modules/elastalert-iam-policies/variables.tf","sha":"bc55d045eba3b8df1b08f78a4657cae25ee0ea73"}]},{"name":"elastalert-security-group-rules","children":[{"name":"README.md","path":"modules/elastalert-security-group-rules/README.md","sha":"dc33e8391d9958ef1915f6a4e2104df391e2d320"},{"name":"main.tf","path":"modules/elastalert-security-group-rules/main.tf","sha":"519b25d67eb73a1587b8ad0556bba585230d1440"},{"name":"variables.tf","path":"modules/elastalert-security-group-rules/variables.tf","sha":"0d87d8de7864335c546406e1047a680f14e6ec73"}]},{"name":"elastalert","children":[{"name":"README.md","path":"modules/elastalert/README.md","sha":"8ee7b607def1754891d8ca65dde5a22552787ebb"},{"name":"main.tf","path":"modules/elastalert/main.tf","sha":"18ac92569874242356c83986ebab5f36df6f62bc"},{"name":"outputs.tf","path":"modules/elastalert/outputs.tf","sha":"aa5817acc77a75b75f4ea4acc686332c5b865618"},{"name":"vars.tf","path":"modules/elastalert/vars.tf","sha":"b51689b7b7dd8df9def896e3fb0ba1ad239a7fcc"}]},{"name":"elasticsearch-cluster-backup","children":[{"name":"README.md","path":"modules/elasticsearch-cluster-backup/README.md","sha":"6b3689afcf27cc5cde7a4048e125e97c7d82189b"},{"name":"backup","children":[{"name":"index.js","path":"modules/elasticsearch-cluster-backup/backup/index.js","sha":"6d3ff6a2a58d70dccd55399ba8bf0e0270140279"}]},{"name":"main.tf","path":"modules/elasticsearch-cluster-backup/main.tf","sha":"f0baf650b3eca0ed959b51eb2aba759da30b1ef0"},{"name":"notification","children":[{"name":"index.js","path":"modules/elasticsearch-cluster-backup/notification/index.js","sha":"8e92e6b387974c4f0c6f007b8e3555d64e984202"}]},{"name":"outputs.tf","path":"modules/elasticsearch-cluster-backup/outputs.tf","sha":"ce7c4ee206984e74253ad6bb0bf25cd4a3965053"},{"name":"vars.tf","path":"modules/elasticsearch-cluster-backup/vars.tf","sha":"9c742c1e44a3c1c9ea7f7883a6da8a56dca3a217"}]},{"name":"elasticsearch-cluster-restore","children":[{"name":"README.md","path":"modules/elasticsearch-cluster-restore/README.md","sha":"11f4f63105657b323a60828ff155c22aaa0ebfbf"},{"name":"main.tf","path":"modules/elasticsearch-cluster-restore/main.tf","sha":"d74982003b28e2dca1ef09d48a1acb612031b63b"},{"name":"notification","children":[{"name":"index.js","path":"modules/elasticsearch-cluster-restore/notification/index.js","sha":"4e441662f92edf384c0c92a96ea5e1a7e1b0b24d"}]},{"name":"outputs.tf","path":"modules/elasticsearch-cluster-restore/outputs.tf","sha":"92454e1cf45fcc57f60c09eb8b31064efb460726"},{"name":"restore","children":[{"name":"index.js","path":"modules/elasticsearch-cluster-restore/restore/index.js","sha":"7d84ea278f01b64310fbc33d18c86a2a73b60573"}]},{"name":"vars.tf","path":"modules/elasticsearch-cluster-restore/vars.tf","sha":"2a3504c6b8f86e28a1659b336796bee04f48e2fa"}]},{"name":"elasticsearch-cluster","children":[{"name":"README.md","path":"modules/elasticsearch-cluster/README.md","sha":"f0dc0ff8093bd99055e2c6753e631337e0df2ec9"},{"name":"main.tf","path":"modules/elasticsearch-cluster/main.tf","sha":"6cce0fa70166851bf8da3af72a04fa38f5f5b989"},{"name":"outputs.tf","path":"modules/elasticsearch-cluster/outputs.tf","sha":"0231e24415e13f62f29d8abefe0398ddb6d0fa04"},{"name":"variables.tf","path":"modules/elasticsearch-cluster/variables.tf","sha":"8b680c1329382792bee8415ff36bd1314675a4c9"}]},{"name":"elasticsearch-iam-policies","children":[{"name":"README.md","path":"modules/elasticsearch-iam-policies/README.md","sha":"57a7117828cff0d9d5bdf7c9fe3af050e55e4aa0"},{"name":"main.tf","path":"modules/elasticsearch-iam-policies/main.tf","sha":"cdb51d7b73acb7e3689e0fa5ab554abc8d576081"},{"name":"variables.tf","path":"modules/elasticsearch-iam-policies/variables.tf","sha":"c612a588acea6049cfdb930209854e253b549729"}]},{"name":"elasticsearch-security-group-rules","children":[{"name":"README.md","path":"modules/elasticsearch-security-group-rules/README.md","sha":"5f0b25abeaa80973e4839626ac96ff306291e759"},{"name":"main.tf","path":"modules/elasticsearch-security-group-rules/main.tf","sha":"fd06486758030a63372610631b3f88d25b7f0cd9"},{"name":"outputs.tf","path":"modules/elasticsearch-security-group-rules/outputs.tf","sha":"e69de29bb2d1d6434b8b29ae775ad8c2e48c5391"},{"name":"variables.tf","path":"modules/elasticsearch-security-group-rules/variables.tf","sha":"7f055aa865ca315861e7a69f2d7741ab7883fddf"}]},{"name":"install-collectd","children":[{"name":"README.md","path":"modules/install-collectd/README.md","sha":"dea15e26259636922a06968d852635faa3d05024"},{"name":"install.sh","path":"modules/install-collectd/install.sh","sha":"4090c1e8b94dd1a4d30ce68da03b73c281e53a3a"}]},{"name":"install-elastalert","children":[{"name":"README.md","path":"modules/install-elastalert/README.md","sha":"8ff13a7658035994828103785d70cf4071b41e15"},{"name":"install.sh","path":"modules/install-elastalert/install.sh","sha":"99f9f170e2c9f3fc31a0f5a92a6a83da5c2fe431"}]},{"name":"install-elasticsearch","children":[{"name":"README.md","path":"modules/install-elasticsearch/README.md","sha":"ff4e35e43bb505354640bca43a12ef4e6da886f0"},{"name":"install.sh","path":"modules/install-elasticsearch/install.sh","sha":"54db7090509f2e7eb56659de74480e2f20422870"}]},{"name":"install-filebeat","children":[{"name":"README.md","path":"modules/install-filebeat/README.md","sha":"d9e46730117d2a71b4646f35267327270a7c5d56"},{"name":"install.sh","path":"modules/install-filebeat/install.sh","sha":"b38f01c2caff489023c1f9a29fa8ea062e17ee58"}]},{"name":"install-kibana","children":[{"name":"README.md","path":"modules/install-kibana/README.md","sha":"9489572678aabb69b2921e7a8b6be320b9ad1b5d"},{"name":"install.sh","path":"modules/install-kibana/install.sh","sha":"2109aefeccc5c1028a881da3bf9b5fa36c62f6d8"}]},{"name":"install-logstash","children":[{"name":"README.md","path":"modules/install-logstash/README.md","sha":"522ce13991cbac0f0d72073e913a7c81aabaf3a1"},{"name":"install.sh","path":"modules/install-logstash/install.sh","sha":"a6a44552a2ec903558d494bf4151f32f3374f786"}]},{"name":"kibana-cluster","children":[{"name":"README.md","path":"modules/kibana-cluster/README.md","sha":"6f95050e9979037eb28177d7aebd38ff908b385f"},{"name":"main.tf","path":"modules/kibana-cluster/main.tf","sha":"7f9a9ecd3b4b4aaf8efcf92108fc123f4fb7ee5d"},{"name":"outputs.tf","path":"modules/kibana-cluster/outputs.tf","sha":"f32e707371481d34b2b84d4ef7f205afaa04f2d4"},{"name":"variables.tf","path":"modules/kibana-cluster/variables.tf","sha":"8306891e82fd960d9587851d3e44c6c1579685c3"}]},{"name":"kibana-security-group-rules","children":[{"name":"README.md","path":"modules/kibana-security-group-rules/README.md","sha":"c35fc72e41001065470a412a0ad9756eb2e84310"},{"name":"main.tf","path":"modules/kibana-security-group-rules/main.tf","sha":"59a0df1a9f5ceb464efa6ee73b39ecaf11240c2c"},{"name":"outputs.tf","path":"modules/kibana-security-group-rules/outputs.tf","sha":"e69de29bb2d1d6434b8b29ae775ad8c2e48c5391"},{"name":"variables.tf","path":"modules/kibana-security-group-rules/variables.tf","sha":"b89d5c7f5f1d1b0e9b5e1d239a1dec9a4ea34486"}]},{"name":"load-balancer-alb-target-group","children":[{"name":"README.md","path":"modules/load-balancer-alb-target-group/README.md","sha":"5918b13e284a75e76bc34792789a44150bf0a12f"},{"name":"main.tf","path":"modules/load-balancer-alb-target-group/main.tf","sha":"7cb9fede7e1bb3c06907f90a81c64b02d78900f4"},{"name":"outputs.tf","path":"modules/load-balancer-alb-target-group/outputs.tf","sha":"cdf904445cf5099ccea5cc8820d198a1ffba4283"},{"name":"variables.tf","path":"modules/load-balancer-alb-target-group/variables.tf","sha":"4a262b0abde2721f9e3a5edeb6a98046009b9df3"}]},{"name":"logstash-cluster","children":[{"name":"README.md","path":"modules/logstash-cluster/README.md","sha":"dbc17e8fc3c4d9a61661ca59b934295ce0bcc6a5","toggled":true},{"name":"main.tf","path":"modules/logstash-cluster/main.tf","sha":"f138bbca6e0bc6858f8d323f134e95b71031747d"},{"name":"outputs.tf","path":"modules/logstash-cluster/outputs.tf","sha":"218a4f0042d52ea1673de0c4f78857a4831143a6"},{"name":"vars.tf","path":"modules/logstash-cluster/vars.tf","sha":"bff728633e0f853502c3a15b4f9311b5b9d2d41f"}],"toggled":true},{"name":"logstash-iam-policies","children":[{"name":"README.md","path":"modules/logstash-iam-policies/README.md","sha":"3714f39713885c9698daa2973a7891d0a6fb982f"},{"name":"main.tf","path":"modules/logstash-iam-policies/main.tf","sha":"7b553a45925c6ea587b95e8de3d05f546d03e278"},{"name":"vars.tf","path":"modules/logstash-iam-policies/vars.tf","sha":"865c74386084c6eff1f46cb103aa847a6ad0da68"}]},{"name":"logstash-security-group-rules","children":[{"name":"README.md","path":"modules/logstash-security-group-rules/README.md","sha":"e96766f09754cf166f53debb653ea278613a9c8e"},{"name":"main.tf","path":"modules/logstash-security-group-rules/main.tf","sha":"5ecab3d1c9d60cf246a98f9e36296bc518caa9bb"},{"name":"outputs.tf","path":"modules/logstash-security-group-rules/outputs.tf","sha":"3641c561d0e23ef02fa83319ddafa8d1f9f813d2"},{"name":"vars.tf","path":"modules/logstash-security-group-rules/vars.tf","sha":"9e26c219ddad783caf509581fae84953eae60fe9"}]},{"name":"run-collectd","children":[{"name":"README.md","path":"modules/run-collectd/README.md","sha":"783453e215c5907681e77ab4b48e64cad42ab989"},{"name":"bin","children":[{"name":"run-collectd","path":"modules/run-collectd/bin/run-collectd","sha":"7f7971d8b564a84a251d4c253c42f9025b5a1cf7"}]},{"name":"install.sh","path":"modules/run-collectd/install.sh","sha":"d2d8126f6fc5ba95a4e12242aa6ffe9bdc945ba6"}]},{"name":"run-elastalert","children":[{"name":"README.md","path":"modules/run-elastalert/README.md","sha":"e52cf808ab755b2d56a392ee8f525a0651963472"},{"name":"bin","children":[{"name":"run-elastalert","path":"modules/run-elastalert/bin/run-elastalert","sha":"cccab0f5cf41600c1b8ad48da5173db97965440f"}]},{"name":"install.sh","path":"modules/run-elastalert/install.sh","sha":"308f895a2366fc0cd8e6669229f6928a3d888f13"}]},{"name":"run-elasticsearch","children":[{"name":"README.md","path":"modules/run-elasticsearch/README.md","sha":"aed66a5572147d69f0633d20ba3131fd1e19f7c3"},{"name":"bin","children":[{"name":"run-elasticsearch","path":"modules/run-elasticsearch/bin/run-elasticsearch","sha":"e8e21b99ca07e03ce807ad537fbd9c3858912713"}]},{"name":"install.sh","path":"modules/run-elasticsearch/install.sh","sha":"c66e6da3d46195ac94d68eed18ed301ca7e0eb51"}]},{"name":"run-filebeat","children":[{"name":"README.md","path":"modules/run-filebeat/README.md","sha":"9ce2f4e4cba993bd49967d00338c985e2b4a188a"},{"name":"bin","children":[{"name":"run-filebeat","path":"modules/run-filebeat/bin/run-filebeat","sha":"61451fdb984b815a6747857d46e10c21b3e22939"}]},{"name":"install.sh","path":"modules/run-filebeat/install.sh","sha":"cd890712803c3961ada86acf1e13b4c299402fb6"}]},{"name":"run-kibana","children":[{"name":"README.md","path":"modules/run-kibana/README.md","sha":"57d636cd26e4af0b51364b4ffb70b382f76314e4"},{"name":"bin","children":[{"name":"run-kibana","path":"modules/run-kibana/bin/run-kibana","sha":"0f60bca1be5eebc5594de9e8eab162aa979d4a17"}]},{"name":"install.sh","path":"modules/run-kibana/install.sh","sha":"e23b447832fce06ad32ce654f94a182a521e2e38"}]},{"name":"run-logstash","children":[{"name":"README.md","path":"modules/run-logstash/README.md","sha":"04cd9e728caad40929b74d0f2f1e5d397a90ffe0"},{"name":"bin","children":[{"name":"run-logstash","path":"modules/run-logstash/bin/run-logstash","sha":"60dc8d906d8181e5a94c3ad18b5b27e53d513cf1"}]},{"name":"install.sh","path":"modules/run-logstash/install.sh","sha":"0a30f2adc2d0765b5581274263e12ce2e688b623"}]}],"toggled":true},{"name":"test","children":[{"name":"Gopkg.lock","path":"test/Gopkg.lock","sha":"d726b0d9bf25eae861d500289fb65ec2ad356105"},{"name":"Gopkg.toml","path":"test/Gopkg.toml","sha":"630429bc686d90b570c81062c7b9c0c925c5fdd3"},{"name":"README.md","path":"test/README.md","sha":"819e790d15767c851bfc438f0a1413ed2bde7d3b"},{"name":"elasticsearch_aws_simple_test.go","path":"test/elasticsearch_aws_simple_test.go","sha":"0362c99a0c22300c5723259b48acd75656acf280"},{"name":"elasticsearch_docker_test.go","path":"test/elasticsearch_docker_test.go","sha":"dcccc141f2d210b9f8c1a2df465a82f8396f16eb"},{"name":"elk_aio_test.go","path":"test/elk_aio_test.go","sha":"ef12f5aa440b29507c695c4e95a96df42e07cacf"},{"name":"elk_end_to_end_test.go","path":"test/elk_end_to_end_test.go","sha":"bf2f5c16ebdab4f83abf6cc84153e881c8a06580"},{"name":"notes.md","path":"test/notes.md","sha":"664915d4b948ccd194ffda80854f47c191904e3d"},{"name":"test_helpers.go","path":"test/test_helpers.go","sha":"227ab02ba9ad534bbd9106059f83a450e777cf52"},{"name":"test_helpers_keystore.go","path":"test/test_helpers_keystore.go","sha":"1e189342859a6561ec9e75009ecb2e55dc42fc23"},{"name":"tls_helpers.go","path":"test/tls_helpers.go","sha":"bf18a8f21521a483f1825c23b2d1a16718bf3565"}]}]},"detailsContent":"<h1 class=\"preview__body--title\" id=\"logstash-cluster\">Logstash Cluster</h1><div class=\"preview__body--border\"></div><p>This folder contains a <a href=\"https://www.terraform.io/\" class=\"preview__body--description--blue\" target=\"_blank\">Terraform</a> module to deploy an <a href=\"https://www.elastic.co/products/logstash\" class=\"preview__body--description--blue\" target=\"_blank\">Logstash</a> cluster in <a href=\"https://aws.amazon.com/\" class=\"preview__body--description--blue\" target=\"_blank\">AWS</a> on top of an Auto Scaling Group.\nThe idea is to create an <a href=\"http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html\" class=\"preview__body--description--blue\" target=\"_blank\">Amazon Machine Image (AMI)</a>\nthat has Logstash installed using the <a href=\"/repos/v0.5.1/package-elk/modules/install-logstash\" class=\"preview__body--description--blue\">install-logstash</a> modules.</p>\n<h2 class=\"preview__body--subtitle\" id=\"how-do-you-use-this-module\">How do you use this module?</h2>\n<p>This folder defines a <a href=\"https://www.terraform.io/docs/modules/usage.html\" class=\"preview__body--description--blue\" target=\"_blank\">Terraform module</a>, which you can use in your\ncode by adding a <code>module</code> configuration and setting its <code>source</code> parameter to URL of this folder:</p>\n<pre><span class=\"hljs-keyword\">module</span> <span class=\"hljs-string\">\"logstash_cluster\"</span> {\n <span class=\"hljs-comment\"># <span class=\"hljs-doctag\">TODO:</span> replace <VERSION> with the latest version from the releases page: https://github.com/gruntwork-io/package-elk/releases</span>\n source = <span class=\"hljs-string\">\"github.com/gruntwork-io/package-elk//modules/logstash-cluster?ref=<VERSION>\"</span>\n\n <span class=\"hljs-comment\"># Specify the ID of the Logstash AMI. You should build this using the scripts in the install-logstash module</span>\n ami_id = <span class=\"hljs-string\">\"ami-abcd1234\"</span>\n \n <span class=\"hljs-comment\"># Configure and start Logstash during boot. </span>\n user_data = <<-EOF\n <span class=\"hljs-comment\">#!/bin/bash</span>\n /usr/share/logstash/bin/run-logstash\n EOF\n \n <span class=\"hljs-comment\"># ... See vars.tf for the other parameters you must define for the logstash-cluster module</span>\n}\n</pre>\n<p>Note the following parameters:</p>\n<ul>\n<li>\n<p><code>source</code>: Use this parameter to specify the URL of the logstash-cluster module. The double slash (<code>//</code>) is\nintentional and required. Terraform uses it to specify subfolders within a Git repo (see <a href=\"https://www.terraform.io/docs/modules/sources.html\" class=\"preview__body--description--blue\" target=\"_blank\">module\nsources</a>). The <code>ref</code> parameter specifies a specific Git tag in\nthis repo. That way, instead of using the latest version of this module from the <code>master</code> branch, which\nwill change every time you run Terraform, you're using a fixed version of the repo.</p>\n</li>\n<li>\n<p><code>ami_id</code>: Use this parameter to specify the ID of an Logstash <a href=\"http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html\" class=\"preview__body--description--blue\" target=\"_blank\">Amazon Machine Image\n(AMI)</a> to deploy on each server in the cluster. You\nshould install Logstash on this AMI using the following <a href=\"/repos/v0.5.1/package-elk/modules/install-logstash\" class=\"preview__body--description--blue\">install-logstash</a> module.</p>\n</li>\n<li>\n<p><code>user_data</code>: Use this parameter to specify a <a href=\"http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html#user-data-shell-scripts\" class=\"preview__body--description--blue\" target=\"_blank\">User\nData</a> script that each\nserver will run during boot. This is where you can use the <a href=\"/repos/v0.5.1/package-elk/modules/run-logstash\" class=\"preview__body--description--blue\">run-logstash</a>.</p>\n</li>\n</ul>\n<p>You can find the other parameters in <a href=\"/repos/v0.5.1/package-elk/modules/logstash-cluster/vars.tf\" class=\"preview__body--description--blue\">vars.tf</a>.</p>\n<p>Check out the <a href=\"/repos/v0.5.1/package-elk/examples\" class=\"preview__body--description--blue\">examples folder</a> for fully-working sample code.</p>\n<h2 class=\"preview__body--subtitle\" id=\"whats-included-in-this-module\">What's included in this module?</h2>\n<p>This module creates the following:</p>\n<ul>\n<li><a href=\"#auto-scaling-group\" class=\"preview__body--description--blue\">Auto Scaling Group</a></li>\n<li><a href=\"#load-balancer\" class=\"preview__body--description--blue\">Load Balancer</a></li>\n<li><a href=\"#security-group\" class=\"preview__body--description--blue\">Security Group</a></li>\n</ul>\n<h3 class=\"preview__body--subtitle\" id=\"auto-scaling-group\">Auto Scaling Group</h3>\n<p>This module runs Logstash on top of an <a href=\"https://aws.amazon.com/autoscaling/\" class=\"preview__body--description--blue\" target=\"_blank\">Auto Scaling Group (ASG)</a>. Typically,\nyou should run the ASG with multiple Instances spread across multiple <a href=\"http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html\" class=\"preview__body--description--blue\" target=\"_blank\">Availability\nZones</a>. Each of the EC2\nInstances should be running an AMI that has Logstash installed via the <a href=\"/repos/v0.5.1/package-elk/modules/install-logstash\" class=\"preview__body--description--blue\">install-logstash</a> script.\nYou pass in the ID of the AMI to run using the <code>ami_id</code> input parameter.</p>\n<h3 class=\"preview__body--subtitle\" id=\"load-balancer\">Load Balancer</h3>\n<p>We use a <a href=\"https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html\" class=\"preview__body--description--blue\" target=\"_blank\">Network Load Balancer</a> (1)\nso that we can perform ongoing health checks on each Logstash node, and (2) so that Filebeat can access the Logstash cluster\nvia a single endpoint which will forward to a live Kibana endpoint at random.</p>\n<h3 class=\"preview__body--subtitle\" id=\"security-group\">Security Group</h3>\n<p>Each EC2 Instance in the ASG has a Security Group that allows minimal connectivity:</p>\n<ul>\n<li>All outbound requests</li>\n<li>Inbound SSH access from the CIDR blocks and security groups you specify</li>\n</ul>\n<p>The ID of the security group is exported as an output variable, which you can use with the <a href=\"/repos/v0.5.1/package-elk/modules/logstash-security-group-rules\" class=\"preview__body--description--blue\">logstash-security-group-rules</a> module to open up all the ports necessary for Logstash.</p>\n<p>Check out the <a href=\"#security\" class=\"preview__body--description--blue\">Security section</a> for more details.</p>\n<h2 class=\"preview__body--subtitle\" id=\"how-do-you-roll-out-updates\">How do you roll out updates?</h2>\n<p>If you want to deploy a new version of Logstash across the cluster, the best way to do that is to:</p>\n<ol>\n<li>\n<p>Rolling deploy:</p>\n<ol>\n<li>\n<p>Build a new AMI.</p>\n</li>\n<li>\n<p>Set the <code>ami_id</code> parameter to the ID of the new AMI.</p>\n</li>\n<li>\n<p>Run <code>terraform apply</code>.</p>\n</li>\n<li>\n<p>Because the <a href=\"/repos/v0.5.1/package-elk/modules/logstash-cluster\" class=\"preview__body--description--blue\">logstash-cluster module</a> uses the Gruntwork <a href=\"/repos/module-asg/modules/server-group\" class=\"preview__body--description--blue\">server-group</a> modules under the hood, running\n<code>terraform apply</code> will automatically perform a zero-downtime rolling deployment. Specifically, one EC2 Instance at a time will be terminated, a new EC2 Instance will spawn in its place, and only once the new EC2 Instance passes the Load\nBalancer Health Checks will the next EC2 Instance be rolled out.</p>\n<p>Note that there will be a brief period of time during which EC2 Instances based on both the old <code>ami_id</code> and\nnew <code>ami_id</code> will be running.</p>\n</li>\n</ol>\n</li>\n<li>\n<p>New cluster:</p>\n<ol>\n<li>Build a new AMI.</li>\n<li>Create a totally new ASG using the <code>logstash-cluster</code> module with the <code>ami_id</code> set to the new AMI, but all\nother parameters the same as the old cluster.</li>\n<li>Wait for all the nodes in the new ASG to join the cluster and catch up on replication.</li>\n<li>Remove each of the nodes from the old cluster.</li>\n<li>Remove the old ASG by removing that <code>logstash-cluster</code> module from your code.</li>\n</ol>\n</li>\n</ol>\n<h2 class=\"preview__body--subtitle\" id=\"security\">Security</h2>\n<p>Here are some of the main security considerations to keep in mind when using this module:</p>\n<ol>\n<li><a href=\"#security-groups\" class=\"preview__body--description--blue\">Security groups</a></li>\n<li><a href=\"#ssh-access\" class=\"preview__body--description--blue\">SSH access</a></li>\n</ol>\n<h3 class=\"preview__body--subtitle\" id=\"security-groups\">Security groups</h3>\n<p>This module attaches a security group to each EC2 Instance that allows inbound requests as follows:</p>\n<ul>\n<li>\n<p><strong>SSH</strong>: For the SSH port (default: 22), you can use the <code>allowed_ssh_cidr_blocks</code> parameter to control the list of<br>\n<a href=\"https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing\" class=\"preview__body--description--blue\" target=\"_blank\">CIDR blocks</a> that will be allowed access. You can use\nthe <code>allowed_inbound_ssh_security_group_ids</code> parameter to control the list of source Security Groups that will be\nallowed access.</p>\n<p>The ID of the security group is exported as an output variable, which you can use with the <a href=\"/repos/v0.5.1/package-elk/modules/logstash-security-group-rules\" class=\"preview__body--description--blue\">logstash-security-group-rules</a> modules to open up all the ports necessary for Logstash and the respective.</p>\n</li>\n</ul>\n<h3 class=\"preview__body--subtitle\" id=\"ssh-access\">SSH access</h3>\n<p>You can associate an <a href=\"http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html\" class=\"preview__body--description--blue\" target=\"_blank\">EC2 Key Pair</a> with each\nof the EC2 Instances in this cluster by specifying the Key Pair's name in the <code>ssh_key_name</code> variable. If you don't\nwant to associate a Key Pair with these servers, set <code>ssh_key_name</code> to an empty string.</p>\n","repoName":"package-elk","repoRef":"v0.5.0","serviceDescriptor":{"serviceName":"Elasticsearch (self-hosted)","serviceRepoName":"package-elk","serviceRepoOrg":"gruntwork-io","cloudProviders":["aws"],"description":"Deploy a self-hosted Elasticsearch cluster. Supports automatic bootstrap, zero-downtime rolling deployment, auto healing, backup, and recovery.","imageUrl":"elk.png","licenseType":"subscriber","technologies":["Terraform","Bash","JavaScript"],"compliance":[],"tags":[""]},"serviceCategoryName":"NoSQL","fileName":"README.md","filePath":"/modules/logstash-cluster","title":"Repo Browser: Elasticsearch (self-hosted)","description":"Browse the repos in the Gruntwork Infrastructure as Code Library."}