In a non-production setting, you can install Elasticsearch tools such as Kibana
and ElastAlert on the same AMI. In a production setting, Elasticsearch should
be the sole service running on each Elasticsearch node.
How do you use this module?
This folder defines a Terraform module, which you can use in your
code by adding a module configuration and setting its source parameter to URL of this folder:
module"elasticsearch_cluster" {
# TODO: replace <VERSION> with the latest version from the releases page: https://github.com/gruntwork-io/terraform-aws-elk/releases
source = "github.com/gruntwork-io/terraform-aws-elk//modules/elasticsearch-cluster?ref=<VERSION>"# Specify the ID of the Elasticsearch AMI. You should build this using the scripts in the install-elasticsearch (and # in a non-production setting, the install-logstash, instgall-kibana, and install-elastalert modules).
ami_id = "ami-abcd1234"# Configure and start Elasticsearch during boot.
user_data = <<-EOF
#!/bin/bash
/usr/share/elasticsearch/bin/run-elasticsearch --cluster-name dev --cluster-tag ClusterName=dev
EOF
# ... See variables.tf for the other parameters you must define for the elasticsearch-cluster module
}
Note the following parameters:
source: Use this parameter to specify the URL of the elasticsearch-cluster module. The double slash (//) is
intentional and required. Terraform uses it to specify subfolders within a Git repo (see module
sources). The ref parameter specifies a specific Git tag in
this repo. That way, instead of using the latest version of this module from the master branch, which
will change every time you run Terraform, you're using a fixed version of the repo.
ami_id: Use this parameter to specify the ID of an Elasticsearch Amazon Machine Image
(AMI) to deploy on each server in the cluster. You
should install Elasticsearch and optionally Elasticsearch tools on this AMI using the following
modules:
install-kibana: Optional. UI to explore Elasticsearch data.
install-logstash: Optional. Used to send Elasticsearch's own logs to Elasticsearch itself.
In a production setting, your AMI should only run Elasticsearch, and Elasticsearch tools should be built on a separate
AMI. In a dev-only environment where parity to production doesn't matter, colocating Elasticsearch and its tools is ok.
user_data: Use this parameter to specify a User
Data script that each
server will run during boot. This is where you can use the run-elasticsearch and, if
applicable, run-elastalert, run-kibana, and
run-logstash scripts to configure and run Elasticsearch and its tools.
You can find the other parameters in variables.tf.
Check out the examples folder for fully-working sample code.
How do you connect to the Elasticsearch cluster?
Connecting to Elasticsearch via Official Elasticsearch Clients
The preferred way to connect to Elasticsearch is to use one of the official Elasticsearch clients. All official Elasticsearch clients are designed to
discover multiple Elasticsearch nodes and distribute reuqests across the various nodes.
Therefore, using a Load Balancer to talk to Elasticsearch APIs (e.g., via an SDK) is NOT recommended, so you will need
to get the IPs of the individual nodes and connect to them directly. Since those nodes run in an Auto Scaling Group (ASG)
where servers can be added/replaced/removed at any time, you can't get their IP addresses from Terraform. Instead, you'll
need to look up the IPs using the AWS APIs.
The easiest way to do that is to use the AWS SDK to look up the servers using EC2 Tags. Each server deployed by
the elasticsearch-cluster module has its Name and aws:autoscaling:groupName tag set to the value you pass in via the
cluster_name parameter. You can also specify custom tags via the tags parameter. You can use the AWS SDK to find
the IPs of all servers with those tags.
For example, using the AWS CLI, you can get the IPs for servers in us-east-1 with
the tag Name=elasticsearch-example as follows:
This will return a bunch of JSON that contains the IPs of the servers. You can then use the Elasticsearch client for your programming language to connect
to these IPs.
Connecting via the REST API
Elasticsearch exposes a RESTful API that you can directly access using curl or any other programming language feature
that makes HTTP requests.
This module runs Elasticsearch on top of an Auto Scaling Group (ASG). Typically,
you should run the ASG with multiple Instances spread across multiple Availability
Zones. Each of the EC2
Instances should be running an AMI that has Elasticsearch and optional Elasticsearch tools installed via the
install-elasticsearch, install-elastalert, install-kibana, and install-logstash scripts. You pass in the ID of the AMI to
run using the ami_id input parameter.
Load Balancer
We use a Network Load Balancer (1)
so that we can perform ongoing health checks on each Elasticsearch node, and (2) so that Kibana can be accessed via a
single endpoint which will forward to a live Kibana endpoint at random.
Note that we do not need a Load Balancer to distribute traffic to Elasticsearch because all the official
Elasticsearch clients are designed to discover all
Elasticsearch nodes and distribute requests across the cluster. Using a Load Balancer for this reason would duplicate
functionality Elasticsearch clients already give us.
Security Group
Each EC2 Instance in the ASG has a Security Group that allows minimal connectivity:
All outbound requests
Inbound SSH access from the CIDR blocks and security groups you specify
Each EC2 Instance in the ASG has an IAM Role attached.
The IAM Role ARN and ID are exported as output variables if you need to add additional permissions.
Instance Stores have the major disadvantage that they do not survive the termination of an EC2 Instance. That is, when
an EC2 Instance dies, all the data on an Instance Store dies with it and is unrecoverable. But Elasticsearch already has
built in support for replica shards,
so we already have redundancy available to us if an EC2 Instance should fail.
This enables us to take advantage of the benefits of Instance Stores, which are that they are significantly faster
because I/O traffic is now all local. By contrast, I/O traffic with EBS Volumes must traverse the (admittedly ultra low-
latency) network and are therefore much slower.
How do you roll out updates?
If you want to deploy a new version of Elasticsearch across the cluster, the best way to do that is to:
Rolling deploy:
Build a new AMI.
Set the ami_id parameter to the ID of the new AMI.
Run terraform apply.
Because the elasticsearch-cluster module uses the Gruntwork server-group modules under the hood, running
terraform apply will automatically perform a zero-downtime rolling deployment. Specifically, one EC2 Instance at a time will be terminated, a new EC2 Instance will spawn in its place, and only once the new EC2 Instance passes the Load
Balancer Health Checks will the next EC2 Instance be rolled out.
Note that there will be a brief period of time during which EC2 Instances based on both the old ami_id and
new ami_id will be running. Rolling upgrades docs
suggest that this is acceptable for Elasticsearch version 5.6 and greater.
TODO: Add support for automatically disabling shard allocation and performing a synced flush on an Elasticsearch
node prior to terminating it (docs).
New cluster:
Build a new AMI.
Create a totally new ASG using the elasticsearch-cluster module with the ami_id set to the new AMI, but all
other parameters the same as the old cluster.
Wait for all the nodes in the new ASG to join the cluster and catch up on replication.
Remove each of the nodes from the old cluster.
Remove the old ASG by removing that elasticsearch-cluster module from your code.
Security
Here are some of the main security considerations to keep in mind when using this module:
Elasticsearch can encrypt all of its network traffic. TODO: Should we recommend using X-Pack (official solution, but
paid), an Nginx Reverse Proxy, a custom Elasticsearch plugin, or something else?
Encryption at rest
EC2 Instance Storage
The EC2 Instances in the cluster store their data in an EC2 Instance Store, which does not have native suport for
encryption (unlike EBS Volume Encryption).
TODO: Should we implement encryption at rest uising the technique described at https://aws.amazon.com/blogs/security/how-to-protect-data-at-rest-with-amazon-ec2-instance-store-encryption/?
Elasticsearch Keystore
Some Elasticsearch settings may contain secrets and should be encrypted. You can use the Elasticsearch Keystore for such settings. The
elasticsearch.keystore is created automatically upon boot of each node, and is available for use as described in the
docs.
Dedicated instances
If you wish to use dedicated instances, you can set the tenancy parameter to "dedicated" in this module.
Security groups
This module attaches a security group to each EC2 Instance that allows inbound requests as follows:
SSH: For the SSH port (default: 22), you can use the allowed_ssh_cidr_blocks parameter to control the list of CIDR blocks that will be allowed access. You can use
the allowed_inbound_ssh_security_group_ids parameter to control the list of source Security Groups that will be
allowed access.
You can associate an EC2 Key Pair with each
of the EC2 Instances in this cluster by specifying the Key Pair's name in the ssh_key_name variable. If you don't
want to associate a Key Pair with these servers, set ssh_key_name to an empty string.
Questions? Ask away.
We're here to talk about our services, answer any questions, give advice, or just to chat.
{"treedata":{"name":"root","toggled":true,"children":[{"name":".circleci","children":[{"name":"config.yml","path":".circleci/config.yml","sha":"9fec455dfc15ca063a578654ad29591ffd2252e1"}]},{"name":".gitignore","path":".gitignore","sha":"437200d1e2f9ac489a6f8febd3f3fa255a7fdfe9"},{"name":".pre-commit-config.yaml","path":".pre-commit-config.yaml","sha":"71195b5cefdf89443b15adbdfdc7735b11833a37"},{"name":"CODEOWNERS","path":"CODEOWNERS","sha":"19e775a4b681f94468d8fd2a6a2780cfbe91ecff"},{"name":"CONTRIBUTING.md","path":"CONTRIBUTING.md","sha":"492bef006a63f2c3189fd171116eb3f900c6cd02"},{"name":"LICENSE.md","path":"LICENSE.md","sha":"a2cf01ecdd725fddd718ab91c80c115882c94f3c"},{"name":"README.md","path":"README.md","sha":"887c739ec9651db0f4827445b56f975345b2820f"},{"name":"_docs","children":[{"name":"elk-architecture.png","path":"_docs/elk-architecture.png","sha":"a7585c6f40eb0a7740b34f00d412bc2e37373266"}]},{"name":"examples","children":[{"name":"elasticsearch-docker","children":[{"name":"local-mocks","children":[{"name":"entrypoint.sh","path":"examples/elasticsearch-docker/local-mocks/entrypoint.sh","sha":"a7405f93f4482217729e3694d14a59a331471750"}]},{"name":"non-ssl","children":[{"name":".env","path":"examples/elasticsearch-docker/non-ssl/.env","sha":"bc82d24eea229df2ba77f0fdf1f6e1c6f6a8e532"},{"name":"README.md","path":"examples/elasticsearch-docker/non-ssl/README.md","sha":"ed4bee52f5093a864164067b57d54f41668e55f5"},{"name":"docker-compose.yml","path":"examples/elasticsearch-docker/non-ssl/docker-compose.yml","sha":"e7547917f574679857f800b24f86bd573671931e"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/elasticsearch-docker/non-ssl/user-data/user-data.sh","sha":"81df178b3080a40a9e03ff7e94f1ef5db7d57f98"}]}]},{"name":"ssl","children":[{"name":".env","path":"examples/elasticsearch-docker/ssl/.env","sha":"bc82d24eea229df2ba77f0fdf1f6e1c6f6a8e532"},{"name":"README.md","path":"examples/elasticsearch-docker/ssl/README.md","sha":"ed4bee52f5093a864164067b57d54f41668e55f5"},{"name":"docker-compose.yml","path":"examples/elasticsearch-docker/ssl/docker-compose.yml","sha":"25e2b036c7e672a8ffc4b9559ab53f8d77441f99"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/elasticsearch-docker/ssl/user-data/user-data.sh","sha":"fc95c5a93069b251aca3ec2ae70a2be90231fcd9"}]}]}]},{"name":"elasticsearch-only-cluster","children":[{"name":"README.md","path":"examples/elasticsearch-only-cluster/README.md","sha":"e830acc97cf711f5087d03db0c61dc6ed82b1b8c"},{"name":"main.tf","path":"examples/elasticsearch-only-cluster/main.tf","sha":"16e304e12274a2c4cbc6c5b4c1d1758144ce259b"},{"name":"outputs.tf","path":"examples/elasticsearch-only-cluster/outputs.tf","sha":"382f2cda2c87d824b274f986f5107fd90477544a"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/elasticsearch-only-cluster/user-data/user-data.sh","sha":"1248e6d8692154f3d024ad95dab49a7a8646bc31"}]},{"name":"vars.tf","path":"examples/elasticsearch-only-cluster/vars.tf","sha":"6c3db8f2b043c53d04f7278a076230b4ce06308a"}]},{"name":"elk-amis","children":[{"name":"README.md","path":"examples/elk-amis/README.md","sha":"2d26c99b7a9a0d3a2124a771f0335eb5251f3b4c"},{"name":"all-in-one","children":[{"name":"README.md","path":"examples/elk-amis/all-in-one/README.md","sha":"42799bc50b4f41eb9e27ed79f1b5624b812b81b2"},{"name":"all-in-one.json","path":"examples/elk-amis/all-in-one/all-in-one.json","sha":"ecd2d44afe1d6040f90e304bc6cf042ec3668bc9"}]},{"name":"app-server","children":[{"name":"README.md","path":"examples/elk-amis/app-server/README.md","sha":"577966ff3d1aa65d69aa43d6b7e1e86968099335"},{"name":"app-server.json","path":"examples/elk-amis/app-server/app-server.json","sha":"6098834ef5ea26e93099f5131f4bd7c8ad77e82a"}]},{"name":"collectd","children":[{"name":"README.md","path":"examples/elk-amis/collectd/README.md","sha":"a9fa0d44e853022868368cab446e4aaeb24997ab"},{"name":"collectd-install-steps.sh","path":"examples/elk-amis/collectd/collectd-install-steps.sh","sha":"bfe1b037dcdee47751c588c4e431bfefec4661ee"},{"name":"collectd.json","path":"examples/elk-amis/collectd/collectd.json","sha":"236b08e2e60af5d34a9efabca7f49cf90772201f"},{"name":"config","children":[{"name":"collectd-ssl.conf","path":"examples/elk-amis/collectd/config/collectd-ssl.conf","sha":"0ccfc0f2aee65c0e17f5cd530c5edaccf247c3d8"},{"name":"collectd.conf","path":"examples/elk-amis/collectd/config/collectd.conf","sha":"74c1b46937039070cd95f64269eef63aabd72805"}]}]},{"name":"elastalert","children":[{"name":"README.md","path":"examples/elk-amis/elastalert/README.md","sha":"9dd33dfd29ddafa1040213466560ffcf8a66944d"},{"name":"aws","children":[{"name":"elastalert-config","children":[{"name":"config-ssl.yml","path":"examples/elk-amis/elastalert/aws/elastalert-config/config-ssl.yml","sha":"f26f8404e40736c5b0b17bc23e6a5de119d0c80f"},{"name":"config.yml","path":"examples/elk-amis/elastalert/aws/elastalert-config/config.yml","sha":"6054d91b8e86df9102fd5fd1fa2c545f7a981d31"}]}]},{"name":"docker","children":[{"name":"elastalert-config","children":[{"name":"config-ssl.yml","path":"examples/elk-amis/elastalert/docker/elastalert-config/config-ssl.yml","sha":"f26f8404e40736c5b0b17bc23e6a5de119d0c80f"},{"name":"config.yml","path":"examples/elk-amis/elastalert/docker/elastalert-config/config.yml","sha":"6054d91b8e86df9102fd5fd1fa2c545f7a981d31"}]}]},{"name":"elastalert-install-steps.sh","path":"examples/elk-amis/elastalert/elastalert-install-steps.sh","sha":"7da5b70623c85a0d61762a2ac5a2d91afd58d969"},{"name":"elastalert-rules","children":[{"name":"example_change.yml","path":"examples/elk-amis/elastalert/elastalert-rules/example_change.yml","sha":"4020886483786ff9a8847159b5750014c5d2b0fb"}]},{"name":"elastalert.json","path":"examples/elk-amis/elastalert/elastalert.json","sha":"762483ee0d35fdc4179cea051779f1076991fee5"}]},{"name":"elasticsearch","children":[{"name":"README.md","path":"examples/elk-amis/elasticsearch/README.md","sha":"f77a938e72424815565ffe16b9ae7787cb430cc7"},{"name":"aws","children":[{"name":"config","children":[{"name":"elasticsearch-ssl.yml","path":"examples/elk-amis/elasticsearch/aws/config/elasticsearch-ssl.yml","sha":"e6b0e5d3efd07f195aa18ae567789548a0eebc68"},{"name":"elasticsearch.yml","path":"examples/elk-amis/elasticsearch/aws/config/elasticsearch.yml","sha":"4918d1d6ec52dafa7741bda8212240d75b162f37"},{"name":"jvm.options","path":"examples/elk-amis/elasticsearch/aws/config/jvm.options","sha":"28dcf519ee339aba5dd15059c1acc4b97846d6ce"}]}]},{"name":"docker","children":[{"name":"config","children":[{"name":"elasticsearch-ssl.yml","path":"examples/elk-amis/elasticsearch/docker/config/elasticsearch-ssl.yml","sha":"705af52bc1e914df93faa919de8a23b973184938"},{"name":"elasticsearch.yml","path":"examples/elk-amis/elasticsearch/docker/config/elasticsearch.yml","sha":"2ac38b9c875dd6352b7a3d7d0fc2c02662d0e9cb"},{"name":"jvm.options","path":"examples/elk-amis/elasticsearch/docker/config/jvm.options","sha":"28dcf519ee339aba5dd15059c1acc4b97846d6ce"}]}]},{"name":"elasticsearch-install-steps.sh","path":"examples/elk-amis/elasticsearch/elasticsearch-install-steps.sh","sha":"a0de2d98730e3b1d0e3c5060dc80d925cb4043d3"},{"name":"elasticsearch.json","path":"examples/elk-amis/elasticsearch/elasticsearch.json","sha":"c8bfafebb715d5e1305f7c758274ce63de9f73ec"},{"name":"plugins","children":[{"name":"readonlyrest-1.22.1_es6.8.12.zip","path":"examples/elk-amis/elasticsearch/plugins/readonlyrest-1.22.1_es6.8.12.zip","sha":"93bdf77044e9ac16fb6eb8e091a1700419425ea9"}]},{"name":"readonlyrest-config","children":[{"name":"readonlyrest.yml","path":"examples/elk-amis/elasticsearch/readonlyrest-config/readonlyrest.yml","sha":"e056fad877791d455e337ff867845c26f514874a"}]}]},{"name":"filebeat","children":[{"name":"README.md","path":"examples/elk-amis/filebeat/README.md","sha":"ce76c3cbb426dc4d5e82f04a85eb0efd043fe671"},{"name":"config","children":[{"name":"filebeat-ssl.yml","path":"examples/elk-amis/filebeat/config/filebeat-ssl.yml","sha":"24b163e671b4cc49af08726867d8869f105a8ecb"},{"name":"filebeat.yml","path":"examples/elk-amis/filebeat/config/filebeat.yml","sha":"ae4641ee77c89e389728836fca3cac1428214fe2"}]},{"name":"filebeat-install-steps.sh","path":"examples/elk-amis/filebeat/filebeat-install-steps.sh","sha":"e7374df095835dddd4d89555d5529c7c3fdff175"},{"name":"filebeat.json","path":"examples/elk-amis/filebeat/filebeat.json","sha":"50b118a2452670dc1291f8e302a7045cba0c958a"}]},{"name":"kibana","children":[{"name":"README.md","path":"examples/elk-amis/kibana/README.md","sha":"b8af363345ba445cb9fbc2b04ec0e7d6b0b08149"},{"name":"config","children":[{"name":"kibana-ssl.yml","path":"examples/elk-amis/kibana/config/kibana-ssl.yml","sha":"7bcab515eb6cba2026899fb711541ebdecc97e3d"},{"name":"kibana.yml","path":"examples/elk-amis/kibana/config/kibana.yml","sha":"852b146720a4ab3fc1380beacce6f77851017b51"}]},{"name":"kibana-install-steps.sh","path":"examples/elk-amis/kibana/kibana-install-steps.sh","sha":"3be2e4579c93962ad8da899728ac3b496a5cd78e"},{"name":"kibana.json","path":"examples/elk-amis/kibana/kibana.json","sha":"6293086359621b813fc384d580598de122f29db0"}]},{"name":"logstash","children":[{"name":"README.md","path":"examples/elk-amis/logstash/README.md","sha":"b1035b8b3d34f28000e329171ba65275b1fac258"},{"name":"config","children":[{"name":"jvm.options","path":"examples/elk-amis/logstash/config/jvm.options","sha":"6017dd218501163070f950077ce6a1e9cff8f3a4"},{"name":"logstash-ssl.yml","path":"examples/elk-amis/logstash/config/logstash-ssl.yml","sha":"46e72285a2074ceba2313f726db5eb0db6dfa414"},{"name":"logstash.yml","path":"examples/elk-amis/logstash/config/logstash.yml","sha":"46e72285a2074ceba2313f726db5eb0db6dfa414"},{"name":"pipeline-ssl.conf","path":"examples/elk-amis/logstash/config/pipeline-ssl.conf","sha":"723780826865edbc2274837265a9059f9c7c1397"},{"name":"pipeline.conf","path":"examples/elk-amis/logstash/config/pipeline.conf","sha":"4a68c41fc5e86106915cb1c91ec39bd4c0cd2b55"}]},{"name":"logstash-install-steps.sh","path":"examples/elk-amis/logstash/logstash-install-steps.sh","sha":"e44ef50df6ac8989bfae91012cf04bbe386121b2"},{"name":"logstash.json","path":"examples/elk-amis/logstash/logstash.json","sha":"5ebdff0ca36a3bbe30043c3ef9b88959d1cdf3a8"}]}]},{"name":"elk-multi-cluster","children":[{"name":"README.md","path":"examples/elk-multi-cluster/README.md","sha":"8fd9aa87b62930d731636f72f6c0d22f900cb087"},{"name":"main.tf","path":"examples/elk-multi-cluster/main.tf","sha":"41e03942f41795d52e001ad11c5ea974abbf0dea"},{"name":"outputs.tf","path":"examples/elk-multi-cluster/outputs.tf","sha":"8bc4cde32815078921bc75375d963b533f6c50db"},{"name":"user-data","children":[{"name":"app-server","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/app-server/user-data.sh","sha":"e3ab01683b870b0bd96ad620d4b25860a7d7a833"}]},{"name":"elastalert","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/elastalert/user-data.sh","sha":"48181b3b07a3aec5e5b808b4101a9ef447eed37b"}]},{"name":"elasticsearch","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/elasticsearch/user-data.sh","sha":"7c7f36d79d716ff7e7618455097513aa936919bf"}]},{"name":"kibana","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/kibana/user-data.sh","sha":"b7f5e2b2d8c7091f1aa3e036a723913d9feca37b"}]},{"name":"logstash","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/logstash/user-data.sh","sha":"9e1af5d687be9c44149560d2a022b62b88743b44"}]}]},{"name":"vars.tf","path":"examples/elk-multi-cluster/vars.tf","sha":"43f00586a4b0fb616a2904f6bb11112ab334f36e"}]},{"name":"elk-single-cluster","children":[{"name":"README.md","path":"examples/elk-single-cluster/README.md","sha":"33320b782aca0aea4d298599e5d3d706c93a12cf"},{"name":"main.tf","path":"examples/elk-single-cluster/main.tf","sha":"1c58be21a641bc4c0bdb683fcc5afd37258a8649"},{"name":"outputs.tf","path":"examples/elk-single-cluster/outputs.tf","sha":"c12a0b236e6fcbf5881cb60276d62a432dadb87a"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/elk-single-cluster/user-data/user-data.sh","sha":"446a992b93c4a3a53a86377546b0cc520bc941d3"}]},{"name":"vars.tf","path":"examples/elk-single-cluster/vars.tf","sha":"a67bfbaabbca1a1f25865570d9cc1a70d369485b"}]}]},{"name":"modules","children":[{"name":"auto-discovery","children":[{"name":"README.md","path":"modules/auto-discovery/README.md","sha":"f4de11c428f8fefd962220e7fbc4b976b4f838fe"},{"name":"bin","children":[{"name":"auto-discovery","path":"modules/auto-discovery/bin/auto-discovery","sha":"fa1f2368e4ff71c550151ed9bd67107515d04873"}]},{"name":"install.sh","path":"modules/auto-discovery/install.sh","sha":"99a2f2fb2cbfaa31e752cbcc6495e5b755f034f2"}]},{"name":"beats-iam-policies","children":[{"name":"main.tf","path":"modules/beats-iam-policies/main.tf","sha":"e9a8c04747f61ad5aa049421312e955773f656e6"},{"name":"vars.tf","path":"modules/beats-iam-policies/vars.tf","sha":"52873644e20b4cafdcc0f33dc11db99f1fa9b586"}]},{"name":"elastalert-iam-policies","children":[{"name":"README.md","path":"modules/elastalert-iam-policies/README.md","sha":"012578675e314c2985c4c4c2ef7e58bd8eb37956"},{"name":"main.tf","path":"modules/elastalert-iam-policies/main.tf","sha":"e4f7ef96c2a388c60e6a6fb315f8cdac3256e280"},{"name":"variables.tf","path":"modules/elastalert-iam-policies/variables.tf","sha":"bc55d045eba3b8df1b08f78a4657cae25ee0ea73"}]},{"name":"elastalert-security-group-rules","children":[{"name":"README.md","path":"modules/elastalert-security-group-rules/README.md","sha":"050c72419243011fbcbdd5359764a2f9a22d0aaa"},{"name":"main.tf","path":"modules/elastalert-security-group-rules/main.tf","sha":"dae2fbeebd5a32a9dcd12194e7a433ea9762f52d"},{"name":"variables.tf","path":"modules/elastalert-security-group-rules/variables.tf","sha":"0d87d8de7864335c546406e1047a680f14e6ec73"}]},{"name":"elastalert","children":[{"name":"README.md","path":"modules/elastalert/README.md","sha":"ebc03e5af2cd6e03520a8e9ab126b34c9782acc6"},{"name":"main.tf","path":"modules/elastalert/main.tf","sha":"241f707360fec4cf509098fffe13f7f2bede372d"},{"name":"outputs.tf","path":"modules/elastalert/outputs.tf","sha":"aa5817acc77a75b75f4ea4acc686332c5b865618"},{"name":"vars.tf","path":"modules/elastalert/vars.tf","sha":"bd053dc8923e14a406bae848f75c92e4f0e8a822"}]},{"name":"elasticsearch-cluster-backup","children":[{"name":"README.md","path":"modules/elasticsearch-cluster-backup/README.md","sha":"2c3ca6928de82cfbdb8b96a4eaf5a2b86a0d39c7"},{"name":"backup","children":[{"name":"index.js","path":"modules/elasticsearch-cluster-backup/backup/index.js","sha":"6d3ff6a2a58d70dccd55399ba8bf0e0270140279"}]},{"name":"main.tf","path":"modules/elasticsearch-cluster-backup/main.tf","sha":"f6ed9528ec7b0f77513c638ad6410c118474353b"},{"name":"notification","children":[{"name":"index.js","path":"modules/elasticsearch-cluster-backup/notification/index.js","sha":"8e92e6b387974c4f0c6f007b8e3555d64e984202"}]},{"name":"outputs.tf","path":"modules/elasticsearch-cluster-backup/outputs.tf","sha":"ce7c4ee206984e74253ad6bb0bf25cd4a3965053"},{"name":"vars.tf","path":"modules/elasticsearch-cluster-backup/vars.tf","sha":"93c34c8c22ecfa0c9498e450df4886996b4bcfa1"}]},{"name":"elasticsearch-cluster-restore","children":[{"name":"README.md","path":"modules/elasticsearch-cluster-restore/README.md","sha":"754fa372fff69c4726499744ff41a83fe5c40446"},{"name":"main.tf","path":"modules/elasticsearch-cluster-restore/main.tf","sha":"f960c5c7e8298f62557bee5afef162721e7029f0"},{"name":"notification","children":[{"name":"index.js","path":"modules/elasticsearch-cluster-restore/notification/index.js","sha":"4e441662f92edf384c0c92a96ea5e1a7e1b0b24d"}]},{"name":"outputs.tf","path":"modules/elasticsearch-cluster-restore/outputs.tf","sha":"92454e1cf45fcc57f60c09eb8b31064efb460726"},{"name":"restore","children":[{"name":"index.js","path":"modules/elasticsearch-cluster-restore/restore/index.js","sha":"7d84ea278f01b64310fbc33d18c86a2a73b60573"}]},{"name":"vars.tf","path":"modules/elasticsearch-cluster-restore/vars.tf","sha":"b1c5309ac8d80d86b7ad12912fbf003d7177028f"}]},{"name":"elasticsearch-cluster","children":[{"name":"README.md","path":"modules/elasticsearch-cluster/README.md","sha":"96381fd14e010c475343d0eee79641c13ffe3e4e","toggled":true},{"name":"main.tf","path":"modules/elasticsearch-cluster/main.tf","sha":"13d41726d7cf5dc768c0414390d922bdd9da450f"},{"name":"outputs.tf","path":"modules/elasticsearch-cluster/outputs.tf","sha":"0231e24415e13f62f29d8abefe0398ddb6d0fa04"},{"name":"variables.tf","path":"modules/elasticsearch-cluster/variables.tf","sha":"8b680c1329382792bee8415ff36bd1314675a4c9"}],"toggled":true},{"name":"elasticsearch-iam-policies","children":[{"name":"README.md","path":"modules/elasticsearch-iam-policies/README.md","sha":"efe4837043dbce44ce15e2b9959d2cfa11983485"},{"name":"main.tf","path":"modules/elasticsearch-iam-policies/main.tf","sha":"55bf1f01c743d84523b3d24ab9e41787a1e28d39"},{"name":"variables.tf","path":"modules/elasticsearch-iam-policies/variables.tf","sha":"c612a588acea6049cfdb930209854e253b549729"}]},{"name":"elasticsearch-security-group-rules","children":[{"name":"README.md","path":"modules/elasticsearch-security-group-rules/README.md","sha":"3292b4b51529305c57cccd4687c64bfa2a2dd6ec"},{"name":"main.tf","path":"modules/elasticsearch-security-group-rules/main.tf","sha":"cdb770582d24b7a9fc7502dd395c6f98a54acbac"},{"name":"outputs.tf","path":"modules/elasticsearch-security-group-rules/outputs.tf","sha":"e69de29bb2d1d6434b8b29ae775ad8c2e48c5391"},{"name":"variables.tf","path":"modules/elasticsearch-security-group-rules/variables.tf","sha":"7f055aa865ca315861e7a69f2d7741ab7883fddf"}]},{"name":"install-collectd","children":[{"name":"README.md","path":"modules/install-collectd/README.md","sha":"ed7f82bbf5f6a3cd5042b8cf78a8b69292f6a929"},{"name":"install.sh","path":"modules/install-collectd/install.sh","sha":"4090c1e8b94dd1a4d30ce68da03b73c281e53a3a"}]},{"name":"install-elastalert","children":[{"name":"README.md","path":"modules/install-elastalert/README.md","sha":"ab3850634329401b2734c5ed46064f93afbb989e"},{"name":"install.sh","path":"modules/install-elastalert/install.sh","sha":"2428d52ac73bd555044c5848546aac8b564c3a5f"}]},{"name":"install-elasticsearch","children":[{"name":"README.md","path":"modules/install-elasticsearch/README.md","sha":"ca54645f1ea89c80bb6953a2e5d70abec68a5346"},{"name":"install.sh","path":"modules/install-elasticsearch/install.sh","sha":"89c6d78790ed08a49af5821e04a459ec4c6b89d6"}]},{"name":"install-filebeat","children":[{"name":"README.md","path":"modules/install-filebeat/README.md","sha":"e8aea474de93cfb6421e038bdfd0459bf9936055"},{"name":"install.sh","path":"modules/install-filebeat/install.sh","sha":"b7537d09a916f93ded0214c5386d897e2f38bf56"}]},{"name":"install-kibana","children":[{"name":"README.md","path":"modules/install-kibana/README.md","sha":"9517d9c1b2f73823a00f8c1b190bb90e5b90312e"},{"name":"install.sh","path":"modules/install-kibana/install.sh","sha":"35eeef5d8aa75b4a1dd54d2913acf6688429c780"}]},{"name":"install-logstash","children":[{"name":"README.md","path":"modules/install-logstash/README.md","sha":"e1a8bde78e9bae25b852d7e9e44d0c6f372c3294"},{"name":"install.sh","path":"modules/install-logstash/install.sh","sha":"a689e67980aa28d7d2f04f693ccb59138ccfe3e2"}]},{"name":"kibana-cluster","children":[{"name":"README.md","path":"modules/kibana-cluster/README.md","sha":"87b31b9825999d66d6d334b48dc73de56ba647f7"},{"name":"main.tf","path":"modules/kibana-cluster/main.tf","sha":"7eb4359660f7951c4551e44babd9649568f2be31"},{"name":"outputs.tf","path":"modules/kibana-cluster/outputs.tf","sha":"f32e707371481d34b2b84d4ef7f205afaa04f2d4"},{"name":"variables.tf","path":"modules/kibana-cluster/variables.tf","sha":"58d03200bff5a18c52347d42af2a6ac1a6452802"}]},{"name":"kibana-security-group-rules","children":[{"name":"README.md","path":"modules/kibana-security-group-rules/README.md","sha":"f8d904f11c6a2238c5be9100c7e0255de28e5e2d"},{"name":"main.tf","path":"modules/kibana-security-group-rules/main.tf","sha":"f890ae74fae825d5db87f5011ba6638af874fda2"},{"name":"outputs.tf","path":"modules/kibana-security-group-rules/outputs.tf","sha":"e69de29bb2d1d6434b8b29ae775ad8c2e48c5391"},{"name":"variables.tf","path":"modules/kibana-security-group-rules/variables.tf","sha":"b89d5c7f5f1d1b0e9b5e1d239a1dec9a4ea34486"}]},{"name":"load-balancer-alb-target-group","children":[{"name":"README.md","path":"modules/load-balancer-alb-target-group/README.md","sha":"0a95fccf4f76af1149a6ab10d2854156e28f03f3"},{"name":"main.tf","path":"modules/load-balancer-alb-target-group/main.tf","sha":"e96c609cc002f2a9de68147aeb5ea60b6d1eb108"},{"name":"outputs.tf","path":"modules/load-balancer-alb-target-group/outputs.tf","sha":"cdf904445cf5099ccea5cc8820d198a1ffba4283"},{"name":"variables.tf","path":"modules/load-balancer-alb-target-group/variables.tf","sha":"4a262b0abde2721f9e3a5edeb6a98046009b9df3"}]},{"name":"logstash-cluster","children":[{"name":"README.md","path":"modules/logstash-cluster/README.md","sha":"78e4b47484d9f68ad16ffe06fb1af15991282cfe"},{"name":"main.tf","path":"modules/logstash-cluster/main.tf","sha":"55219e71d6f49ec878fafebad547e2573e4449be"},{"name":"outputs.tf","path":"modules/logstash-cluster/outputs.tf","sha":"218a4f0042d52ea1673de0c4f78857a4831143a6"},{"name":"vars.tf","path":"modules/logstash-cluster/vars.tf","sha":"bff728633e0f853502c3a15b4f9311b5b9d2d41f"}]},{"name":"logstash-iam-policies","children":[{"name":"README.md","path":"modules/logstash-iam-policies/README.md","sha":"3e354879298dbe3456cf3bae79594ee815487ac4"},{"name":"main.tf","path":"modules/logstash-iam-policies/main.tf","sha":"0ce69e25d9c2f964dc35f41e10813af86f2a5a7c"},{"name":"vars.tf","path":"modules/logstash-iam-policies/vars.tf","sha":"865c74386084c6eff1f46cb103aa847a6ad0da68"}]},{"name":"logstash-security-group-rules","children":[{"name":"README.md","path":"modules/logstash-security-group-rules/README.md","sha":"7ca6878f6b4415297d21c96ca48a4d8aa0314095"},{"name":"main.tf","path":"modules/logstash-security-group-rules/main.tf","sha":"dd54760fec89fa4a83ed37835d9901256af24bf5"},{"name":"outputs.tf","path":"modules/logstash-security-group-rules/outputs.tf","sha":"3641c561d0e23ef02fa83319ddafa8d1f9f813d2"},{"name":"vars.tf","path":"modules/logstash-security-group-rules/vars.tf","sha":"9e26c219ddad783caf509581fae84953eae60fe9"}]},{"name":"run-collectd","children":[{"name":"README.md","path":"modules/run-collectd/README.md","sha":"2242bcaca7f50717b3d6154c0576847d7b08c12c"},{"name":"bin","children":[{"name":"run-collectd","path":"modules/run-collectd/bin/run-collectd","sha":"7f7971d8b564a84a251d4c253c42f9025b5a1cf7"}]},{"name":"install.sh","path":"modules/run-collectd/install.sh","sha":"d2d8126f6fc5ba95a4e12242aa6ffe9bdc945ba6"}]},{"name":"run-elastalert","children":[{"name":"README.md","path":"modules/run-elastalert/README.md","sha":"e52cf808ab755b2d56a392ee8f525a0651963472"},{"name":"bin","children":[{"name":"run-elastalert","path":"modules/run-elastalert/bin/run-elastalert","sha":"cccab0f5cf41600c1b8ad48da5173db97965440f"}]},{"name":"install.sh","path":"modules/run-elastalert/install.sh","sha":"308f895a2366fc0cd8e6669229f6928a3d888f13"}]},{"name":"run-elasticsearch","children":[{"name":"README.md","path":"modules/run-elasticsearch/README.md","sha":"598ecd294a3177b5f9791793f75cec95559ad886"},{"name":"bin","children":[{"name":"run-elasticsearch","path":"modules/run-elasticsearch/bin/run-elasticsearch","sha":"e8e21b99ca07e03ce807ad537fbd9c3858912713"}]},{"name":"install.sh","path":"modules/run-elasticsearch/install.sh","sha":"c66e6da3d46195ac94d68eed18ed301ca7e0eb51"}]},{"name":"run-filebeat","children":[{"name":"README.md","path":"modules/run-filebeat/README.md","sha":"7b20727ebd231d78a39f30eeb2bad79619447f0a"},{"name":"bin","children":[{"name":"run-filebeat","path":"modules/run-filebeat/bin/run-filebeat","sha":"2e323cd6b7550db8470085911fa372c4edad8ae6"}]},{"name":"install.sh","path":"modules/run-filebeat/install.sh","sha":"cd890712803c3961ada86acf1e13b4c299402fb6"}]},{"name":"run-kibana","children":[{"name":"README.md","path":"modules/run-kibana/README.md","sha":"f6aaf8c3bd3901d753551ab4c28c0772d73ed143"},{"name":"bin","children":[{"name":"run-kibana","path":"modules/run-kibana/bin/run-kibana","sha":"0f60bca1be5eebc5594de9e8eab162aa979d4a17"}]},{"name":"install.sh","path":"modules/run-kibana/install.sh","sha":"e23b447832fce06ad32ce654f94a182a521e2e38"}]},{"name":"run-logstash","children":[{"name":"README.md","path":"modules/run-logstash/README.md","sha":"e1b6d3752f9f2f917c4bb17283caabd163f4d589"},{"name":"bin","children":[{"name":"run-logstash","path":"modules/run-logstash/bin/run-logstash","sha":"60dc8d906d8181e5a94c3ad18b5b27e53d513cf1"}]},{"name":"install.sh","path":"modules/run-logstash/install.sh","sha":"0a30f2adc2d0765b5581274263e12ce2e688b623"}]}],"toggled":true},{"name":"terraform-cloud-enterprise-private-module-registry-placeholder.tf","path":"terraform-cloud-enterprise-private-module-registry-placeholder.tf","sha":"ae586c0fe830819580e1009d41a9074f16e65bed"},{"name":"test","children":[{"name":"README.md","path":"test/README.md","sha":"819e790d15767c851bfc438f0a1413ed2bde7d3b"},{"name":"elasticsearch_aws_simple_test.go","path":"test/elasticsearch_aws_simple_test.go","sha":"cccda6ac30573bc7fea6772f56f68a96ab1b0581"},{"name":"elasticsearch_docker_test.go","path":"test/elasticsearch_docker_test.go","sha":"9aab286b753db9127f7425acea12bcc93e5abdd1"},{"name":"elk_aio_test.go","path":"test/elk_aio_test.go","sha":"b1ca766cbee9d5b69e1028ef48e971f4d333232b"},{"name":"elk_end_to_end_test.go","path":"test/elk_end_to_end_test.go","sha":"fb74457130157625a2d817e0d54d6be55aa6a3c2"},{"name":"go.mod","path":"test/go.mod","sha":"b6d375ede5dfd840285c97e9e8c1e95a6b8cfb52"},{"name":"go.sum","path":"test/go.sum","sha":"c64c51d793c7c3ec7848f7998ddb9494a979ea59"},{"name":"notes.md","path":"test/notes.md","sha":"664915d4b948ccd194ffda80854f47c191904e3d"},{"name":"test_helpers.go","path":"test/test_helpers.go","sha":"0ba1e792e7a26419f43c55971e4910cbc242a009"},{"name":"test_helpers_keystore.go","path":"test/test_helpers_keystore.go","sha":"e3b1ed8963088a2bcbdc2863287e60d19d23cd47"},{"name":"tls_helpers.go","path":"test/tls_helpers.go","sha":"828f04c10b9ca6f4b6f8f82ddcd498bdb1c7dbb4"}]}]},"detailsContent":"<h1 class=\"preview__body--title\" id=\"elasticsearch-cluster\">Elasticsearch Cluster</h1><div class=\"preview__body--border\"></div><p>This folder contains a <a href=\"https://www.terraform.io/\" class=\"preview__body--description--blue\" target=\"_blank\">Terraform</a> module to deploy an <a href=\"https://www.elastic.co/products/elasticsearch\" class=\"preview__body--description--blue\" target=\"_blank\">Elasticsearch</a> cluster in <a href=\"https://aws.amazon.com/\" class=\"preview__body--description--blue\" target=\"_blank\">AWS</a> on top of an Auto Scaling Group.\nThe idea is to create an <a href=\"http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html\" class=\"preview__body--description--blue\" target=\"_blank\">Amazon Machine Image (AMI)</a>\nthat has Elasticsearch installed using the <a href=\"/repos/v0.11.1/package-elk/modules/install-elasticsearch\" class=\"preview__body--description--blue\">install-elasticsearch</a> module.</p>\n<p>In a non-production setting, you can install Elasticsearch tools such as <a href=\"https://www.elastic.co/products/kibana\" class=\"preview__body--description--blue\" target=\"_blank\">Kibana</a>\nand <a href=\"https://github.com/Yelp/elastalert\" class=\"preview__body--description--blue\" target=\"_blank\">ElastAlert</a> on the same AMI. In a production setting, Elasticsearch should\nbe the sole service running on each Elasticsearch node.</p>\n<h2 class=\"preview__body--subtitle\" id=\"how-do-you-use-this-module\">How do you use this module?</h2>\n<p>This folder defines a <a href=\"https://www.terraform.io/docs/modules/usage.html\" class=\"preview__body--description--blue\" target=\"_blank\">Terraform module</a>, which you can use in your\ncode by adding a <code>module</code> configuration and setting its <code>source</code> parameter to URL of this folder:</p>\n<pre><span class=\"hljs-keyword\">module</span> <span class=\"hljs-string\">\"elasticsearch_cluster\"</span> {\n <span class=\"hljs-comment\"># <span class=\"hljs-doctag\">TODO:</span> replace <VERSION> with the latest version from the releases page: https://github.com/gruntwork-io/terraform-aws-elk/releases</span>\n source = <span class=\"hljs-string\">\"github.com/gruntwork-io/terraform-aws-elk//modules/elasticsearch-cluster?ref=<VERSION>\"</span>\n\n <span class=\"hljs-comment\"># Specify the ID of the Elasticsearch AMI. You should build this using the scripts in the install-elasticsearch (and </span>\n <span class=\"hljs-comment\"># in a non-production setting, the install-logstash, instgall-kibana, and install-elastalert modules).</span>\n ami_id = <span class=\"hljs-string\">\"ami-abcd1234\"</span>\n \n <span class=\"hljs-comment\"># Configure and start Elasticsearch during boot. </span>\n user_data = <<-EOF\n <span class=\"hljs-comment\">#!/bin/bash</span>\n /usr/share/elasticsearch/bin/run-elasticsearch --cluster-name dev --cluster-tag ClusterName=dev\n EOF\n \n <span class=\"hljs-comment\"># ... See variables.tf for the other parameters you must define for the elasticsearch-cluster module</span>\n}\n</pre>\n<p>Note the following parameters:</p>\n<ul>\n<li>\n<p><code>source</code>: Use this parameter to specify the URL of the elasticsearch-cluster module. The double slash (<code>//</code>) is\nintentional and required. Terraform uses it to specify subfolders within a Git repo (see <a href=\"https://www.terraform.io/docs/modules/sources.html\" class=\"preview__body--description--blue\" target=\"_blank\">module\nsources</a>). The <code>ref</code> parameter specifies a specific Git tag in\nthis repo. That way, instead of using the latest version of this module from the <code>master</code> branch, which\nwill change every time you run Terraform, you're using a fixed version of the repo.</p>\n</li>\n<li>\n<p><code>ami_id</code>: Use this parameter to specify the ID of an Elasticsearch <a href=\"http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html\" class=\"preview__body--description--blue\" target=\"_blank\">Amazon Machine Image\n(AMI)</a> to deploy on each server in the cluster. You\nshould install Elasticsearch and optionally Elasticsearch tools on this AMI using the following\nmodules:</p>\n<ul>\n<li><a href=\"/repos/v0.11.1/package-elk/modules/install-elasticsearch\" class=\"preview__body--description--blue\">install-elasticsearch</a>: Required.</li>\n<li><a href=\"/repos/v0.11.1/package-elk/modules/install-elastalert\" class=\"preview__body--description--blue\">install-elastalert</a>: Optional. Enables alerts and notifications</li>\n<li><a href=\"/repos/v0.11.1/package-elk/modules/install-kibana\" class=\"preview__body--description--blue\">install-kibana</a>: Optional. UI to explore Elasticsearch data.</li>\n<li><a href=\"/repos/v0.11.1/package-elk/modules/install-logstash\" class=\"preview__body--description--blue\">install-logstash</a>: Optional. Used to send Elasticsearch's own logs to Elasticsearch itself.</li>\n</ul>\n<p>In a production setting, your AMI should only run Elasticsearch, and Elasticsearch tools should be built on a separate\nAMI. In a dev-only environment where parity to production doesn't matter, colocating Elasticsearch and its tools is ok.</p>\n</li>\n<li>\n<p><code>user_data</code>: Use this parameter to specify a <a href=\"http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html#user-data-shell-scripts\" class=\"preview__body--description--blue\" target=\"_blank\">User\nData</a> script that each\nserver will run during boot. This is where you can use the <a href=\"/repos/v0.11.1/package-elk/modules/run-elasticsearch\" class=\"preview__body--description--blue\">run-elasticsearch</a> and, if\napplicable, <a href=\"/repos/v0.11.1/package-elk/modules/run-elastalert\" class=\"preview__body--description--blue\">run-elastalert</a>, <a href=\"/repos/v0.11.1/package-elk/modules/run-kibana\" class=\"preview__body--description--blue\">run-kibana</a>, and\n<a href=\"/repos/v0.11.1/package-elk/modules/run-logstash\" class=\"preview__body--description--blue\">run-logstash</a> scripts to configure and run Elasticsearch and its tools.</p>\n</li>\n</ul>\n<p>You can find the other parameters in <a href=\"/repos/v0.11.1/package-elk/modules/elasticsearch-cluster/variables.tf\" class=\"preview__body--description--blue\">variables.tf</a>.</p>\n<p>Check out the <a href=\"/repos/v0.11.1/package-elk/examples\" class=\"preview__body--description--blue\">examples folder</a> for fully-working sample code.</p>\n<h2 class=\"preview__body--subtitle\" id=\"how-do-you-connect-to-the-elasticsearch-cluster\">How do you connect to the Elasticsearch cluster?</h2>\n<h3 class=\"preview__body--subtitle\" id=\"connecting-to-elasticsearch-via-official-elasticsearch-clients\">Connecting to Elasticsearch via Official Elasticsearch Clients</h3>\n<p>The preferred way to connect to Elasticsearch is to use one of the <a href=\"https://www.elastic.co/guide/en/elasticsearch/client/index.html\" class=\"preview__body--description--blue\" target=\"_blank\">official Elasticsearch clients</a>. All official Elasticsearch clients are designed to\ndiscover multiple Elasticsearch nodes and distribute reuqests across the various nodes.</p>\n<p>Therefore, using a Load Balancer to talk to Elasticsearch APIs (e.g., via an SDK) is NOT recommended, so you will need\nto get the IPs of the individual nodes and connect to them directly. Since those nodes run in an Auto Scaling Group (ASG)\nwhere servers can be added/replaced/removed at any time, you can't get their IP addresses from Terraform. Instead, you'll\nneed to look up the IPs using the AWS APIs.</p>\n<p>The easiest way to do that is to use the AWS SDK to look up the servers using EC2 Tags. Each server deployed by\nthe <code>elasticsearch-cluster</code> module has its <code>Name</code> and <code>aws:autoscaling:groupName</code> tag set to the value you pass in via the\n<code>cluster_name</code> parameter. You can also specify custom tags via the <code>tags</code> parameter. You can use the AWS SDK to find\nthe IPs of all servers with those tags.</p>\n<p>For example, using the <a href=\"https://aws.amazon.com/cli/\" class=\"preview__body--description--blue\" target=\"_blank\">AWS CLI</a>, you can get the IPs for servers in <code>us-east-1</code> with\nthe tag <code>Name=elasticsearch-example</code> as follows:</p>\n<pre>aws ec2 describe-instances <span class=\"hljs-string\">\\</span>\n --region <span class=\"hljs-string\">\"us-east-1\"</span> <span class=\"hljs-string\">\\</span>\n --filter <span class=\"hljs-string\">\\</span>\n <span class=\"hljs-string\">\"Name=tag:Name,Values=elasticsearch-example\"</span> <span class=\"hljs-string\">\\</span>\n <span class=\"hljs-string\">\"Name=instance-state-name,Values=running\"</span>\n</pre>\n<p>This will return a bunch of JSON that contains the IPs of the servers. You can then use the <a href=\"https://www.elastic.co/guide/en/elasticsearch/client/index.html\" class=\"preview__body--description--blue\" target=\"_blank\">Elasticsearch client</a> for your programming language to connect\nto these IPs.</p>\n<h3 class=\"preview__body--subtitle\" id=\"connecting-via-the-rest-api\">Connecting via the REST API</h3>\n<p>Elasticsearch exposes a RESTful API that you can directly access using <code>curl</code> or any other programming language feature\nthat makes HTTP requests.</p>\n<h2 class=\"preview__body--subtitle\" id=\"whats-included-in-this-module\">What's included in this module?</h2>\n<p>This module creates the following:</p>\n<ul>\n<li><a href=\"#auto-scaling-group\" class=\"preview__body--description--blue\">Auto Scaling Group</a></li>\n<li><a href=\"#load-balancer\" class=\"preview__body--description--blue\">Load Balancer</a></li>\n<li><a href=\"#security-group\" class=\"preview__body--description--blue\">Security Group</a></li>\n<li><a href=\"#iam-role-and-permissions\" class=\"preview__body--description--blue\">IAM Role and Permissions</a></li>\n</ul>\n<h3 class=\"preview__body--subtitle\" id=\"whats-not-included\">What's Not Included</h3>\n<ul>\n<li><a href=\"#ebs-volumes\" class=\"preview__body--description--blue\">EBS Volumes</a></li>\n</ul>\n<h3 class=\"preview__body--subtitle\" id=\"auto-scaling-group\">Auto Scaling Group</h3>\n<p>This module runs Elasticsearch on top of an <a href=\"https://aws.amazon.com/autoscaling/\" class=\"preview__body--description--blue\" target=\"_blank\">Auto Scaling Group (ASG)</a>. Typically,\nyou should run the ASG with multiple Instances spread across multiple <a href=\"http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html\" class=\"preview__body--description--blue\" target=\"_blank\">Availability\nZones</a>. Each of the EC2\nInstances should be running an AMI that has Elasticsearch and optional Elasticsearch tools installed via the\n<a href=\"/repos/v0.11.1/package-elk/modules/install-elasticsearch\" class=\"preview__body--description--blue\">install-elasticsearch</a>, <a href=\"/repos/v0.11.1/package-elk/modules/install-elastalert\" class=\"preview__body--description--blue\">install-elastalert</a>, <a href=\"/repos/v0.11.1/package-elk/modules/install-kibana\" class=\"preview__body--description--blue\">install-kibana</a>, and <a href=\"/repos/v0.11.1/package-elk/modules/install-logstash\" class=\"preview__body--description--blue\">install-logstash</a> scripts. You pass in the ID of the AMI to\nrun using the <code>ami_id</code> input parameter.</p>\n<h3 class=\"preview__body--subtitle\" id=\"load-balancer\">Load Balancer</h3>\n<p>We use a <a href=\"https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html\" class=\"preview__body--description--blue\" target=\"_blank\">Network Load Balancer</a> (1)\nso that we can perform ongoing health checks on each Elasticsearch node, and (2) so that Kibana can be accessed via a\nsingle endpoint which will forward to a live Kibana endpoint at random.</p>\n<p>Note that we do not need a Load Balancer to distribute traffic to Elasticsearch because all the <a href=\"https://www.elastic.co/guide/en/elasticsearch/client/index.html\" class=\"preview__body--description--blue\" target=\"_blank\">official\nElasticsearch clients</a> are designed to discover all\nElasticsearch nodes and distribute requests across the cluster. Using a Load Balancer for this reason would duplicate\nfunctionality Elasticsearch clients already give us.</p>\n<h3 class=\"preview__body--subtitle\" id=\"security-group\">Security Group</h3>\n<p>Each EC2 Instance in the ASG has a Security Group that allows minimal connectivity:</p>\n<ul>\n<li>All outbound requests</li>\n<li>Inbound SSH access from the CIDR blocks and security groups you specify</li>\n</ul>\n<p>The ID of the security group is exported as an output variable, which you can use with the <a href=\"/repos/v0.11.1/package-elk/modules/elasticsearch-security-group-rules\" class=\"preview__body--description--blue\">elasticsearch-security-group-rules</a>, <a href=\"/repos/v0.11.1/package-elk/modules/elastalert-security-group-rules\" class=\"preview__body--description--blue\">elastalert-security-group-rules</a>,\n<a href=\"/repos/v0.11.1/package-elk/modules/kibana-security-group-rules\" class=\"preview__body--description--blue\">kibana-security-group-rules</a>, and <a href=\"/repos/v0.11.1/package-elk/modules/logstash-security-group-rules\" class=\"preview__body--description--blue\">logstash-security-group-rules</a> modules to open up all the ports necessary for Elasticsearch and the respective\nElasticsearch tools.</p>\n<p>Check out the <a href=\"#security\" class=\"preview__body--description--blue\">Security section</a> for more details.</p>\n<h3 class=\"preview__body--subtitle\" id=\"iam-role-and-permissions\">IAM Role and Permissions</h3>\n<p>Each EC2 Instance in the ASG has an <a href=\"http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html\" class=\"preview__body--description--blue\" target=\"_blank\">IAM Role</a> attached.\nThe IAM Role ARN and ID are exported as output variables if you need to add additional permissions.</p>\n<h3 class=\"preview__body--subtitle\" id=\"ebs-volumes\">EBS Volumes</h3>\n<p>Note that we do not use <a href=\"https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumes.html\" class=\"preview__body--description--blue\" target=\"_blank\">EBS Volumes</a>, which are\nAWS's ultra-low-latency network-attached storage. Instead, per <a href=\"https://www.elastic.co/guide/en/elasticsearch/plugins/current/cloud-aws-best-practices.html\" class=\"preview__body--description--blue\" target=\"_blank\">Elasticsearch docs on AWS Best Practices</a>, we exclusively use <a href=\"https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html\" class=\"preview__body--description--blue\" target=\"_blank\">Instance\nStores</a>.</p>\n<p>Instance Stores have the major disadvantage that they do not survive the termination of an EC2 Instance. That is, when\nan EC2 Instance dies, all the data on an Instance Store dies with it and is unrecoverable. But Elasticsearch already has\nbuilt in support for <a href=\"https://www.elastic.co/guide/en/elasticsearch/guide/current/replica-shards.html\" class=\"preview__body--description--blue\" target=\"_blank\">replica shards</a>,\nso we already have redundancy available to us if an EC2 Instance should fail.</p>\n<p>This enables us to take advantage of the benefits of Instance Stores, which are that they are significantly faster\nbecause I/O traffic is now all local. By contrast, I/O traffic with EBS Volumes must traverse the (admittedly ultra low-\nlatency) network and are therefore much slower.</p>\n<h2 class=\"preview__body--subtitle\" id=\"how-do-you-roll-out-updates\">How do you roll out updates?</h2>\n<p>If you want to deploy a new version of Elasticsearch across the cluster, the best way to do that is to:</p>\n<ol>\n<li>\n<p>Rolling deploy:</p>\n<ol>\n<li>\n<p>Build a new AMI.</p>\n</li>\n<li>\n<p>Set the <code>ami_id</code> parameter to the ID of the new AMI.</p>\n</li>\n<li>\n<p>Run <code>terraform apply</code>.</p>\n</li>\n<li>\n<p>Because the <a href=\"/repos/v0.11.1/package-elk/modules/elasticsearch-cluster\" class=\"preview__body--description--blue\">elasticsearch-cluster module</a> uses the Gruntwork <a href=\"/repos/terraform-aws-asg/modules/server-group\" class=\"preview__body--description--blue\">server-group</a> modules under the hood, running\n<code>terraform apply</code> will automatically perform a zero-downtime rolling deployment. Specifically, one EC2 Instance at a time will be terminated, a new EC2 Instance will spawn in its place, and only once the new EC2 Instance passes the Load\nBalancer Health Checks will the next EC2 Instance be rolled out.</p>\n<p>Note that there will be a brief period of time during which EC2 Instances based on both the old <code>ami_id</code> and\nnew <code>ami_id</code> will be running. <a href=\"https://www.elastic.co/guide/en/elasticsearch/reference/current/rolling-upgrades.html\" class=\"preview__body--description--blue\" target=\"_blank\">Rolling upgrades docs</a>\nsuggest that this is acceptable for Elasticsearch version 5.6 and greater.</p>\n<p>TODO: Add support for automatically disabling shard allocation and performing a synced flush on an Elasticsearch\nnode prior to terminating it (<a href=\"https://www.elastic.co/guide/en/elasticsearch/reference/current/rolling-upgrades.html\" class=\"preview__body--description--blue\" target=\"_blank\">docs</a>).</p>\n</li>\n</ol>\n</li>\n<li>\n<p>New cluster:</p>\n<ol>\n<li>Build a new AMI.</li>\n<li>Create a totally new ASG using the <code>elasticsearch-cluster</code> module with the <code>ami_id</code> set to the new AMI, but all\nother parameters the same as the old cluster.</li>\n<li>Wait for all the nodes in the new ASG to join the cluster and catch up on replication.</li>\n<li>Remove each of the nodes from the old cluster.</li>\n<li>Remove the old ASG by removing that <code>elasticsearch-cluster</code> module from your code.</li>\n</ol>\n</li>\n</ol>\n<h2 class=\"preview__body--subtitle\" id=\"security\">Security</h2>\n<p>Here are some of the main security considerations to keep in mind when using this module:</p>\n<ol>\n<li><a href=\"#encryption-in-transit\" class=\"preview__body--description--blue\">Encryption in transit</a></li>\n<li><a href=\"#encryption-at-rest\" class=\"preview__body--description--blue\">Encryption at rest</a></li>\n<li><a href=\"#dedicated-instances\" class=\"preview__body--description--blue\">Dedicated instances</a></li>\n<li><a href=\"#security-groups\" class=\"preview__body--description--blue\">Security groups</a></li>\n<li><a href=\"#ssh-access\" class=\"preview__body--description--blue\">SSH access</a></li>\n</ol>\n<h3 class=\"preview__body--subtitle\" id=\"encryption-in-transit\">Encryption in transit</h3>\n<p>Elasticsearch can encrypt all of its network traffic. TODO: Should we recommend using X-Pack (official solution, but\npaid), an Nginx Reverse Proxy, a custom Elasticsearch plugin, or something else?</p>\n<h3 class=\"preview__body--subtitle\" id=\"encryption-at-rest\">Encryption at rest</h3>\n<h4 id=\"ec-2-instance-storage\">EC2 Instance Storage</h4>\n<p>The EC2 Instances in the cluster store their data in an <a href=\"https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html\" class=\"preview__body--description--blue\" target=\"_blank\">EC2 Instance Store</a>, which does not have native suport for\nencryption (unlike <a href=\"https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html\" class=\"preview__body--description--blue\" target=\"_blank\">EBS Volume Encryption</a>).</p>\n<p>TODO: Should we implement encryption at rest uising the technique described at https://aws.amazon.com/blogs/security/how-to-protect-data-at-rest-with-amazon-ec2-instance-store-encryption/?</p>\n<h4 id=\"elasticsearch-keystore\">Elasticsearch Keystore</h4>\n<p>Some Elasticsearch settings may contain secrets and should be encrypted. You can use the <a href=\"https://www.elastic.co/guide/en/elasticsearch/reference/current/secure-settings.html\" class=\"preview__body--description--blue\" target=\"_blank\">Elasticsearch Keystore</a> for such settings. The\n<code>elasticsearch.keystore</code> is created automatically upon boot of each node, and is available for use as described in the\ndocs.</p>\n<h3 class=\"preview__body--subtitle\" id=\"dedicated-instances\">Dedicated instances</h3>\n<p>If you wish to use dedicated instances, you can set the <code>tenancy</code> parameter to <code>"dedicated"</code> in this module.</p>\n<h3 class=\"preview__body--subtitle\" id=\"security-groups\">Security groups</h3>\n<p>This module attaches a security group to each EC2 Instance that allows inbound requests as follows:</p>\n<ul>\n<li>\n<p><strong>SSH</strong>: For the SSH port (default: 22), you can use the <code>allowed_ssh_cidr_blocks</code> parameter to control the list of<br>\n<a href=\"https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing\" class=\"preview__body--description--blue\" target=\"_blank\">CIDR blocks</a> that will be allowed access. You can use\nthe <code>allowed_inbound_ssh_security_group_ids</code> parameter to control the list of source Security Groups that will be\nallowed access.</p>\n<p>The ID of the security group is exported as an output variable, which you can use with the <a href=\"/repos/v0.11.1/package-elk/modules/elasticsearch-security-group-rules\" class=\"preview__body--description--blue\">elasticsearch-security-group-rules</a>, <a href=\"/repos/v0.11.1/package-elk/modules/elastalert-security-group-rules\" class=\"preview__body--description--blue\">elastalert-security-group-rules</a>,\n<a href=\"/repos/v0.11.1/package-elk/modules/kibana-security-group-rules\" class=\"preview__body--description--blue\">kibana-security-group-rules</a>, and <a href=\"/repos/v0.11.1/package-elk/modules/logstash-security-group-rules\" class=\"preview__body--description--blue\">logstash-security-group-rules</a> modules to open up all the ports necessary for Elasticsearch and the respective\nElasticsearch tools.</p>\n</li>\n</ul>\n<h3 class=\"preview__body--subtitle\" id=\"ssh-access\">SSH access</h3>\n<p>You can associate an <a href=\"http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html\" class=\"preview__body--description--blue\" target=\"_blank\">EC2 Key Pair</a> with each\nof the EC2 Instances in this cluster by specifying the Key Pair's name in the <code>ssh_key_name</code> variable. If you don't\nwant to associate a Key Pair with these servers, set <code>ssh_key_name</code> to an empty string.</p>\n","repoName":"package-elk","repoRef":"v0.9.0","serviceDescriptor":{"serviceName":"Elasticsearch (self-hosted)","serviceRepoName":"package-elk","serviceRepoOrg":"gruntwork-io","cloudProviders":["aws"],"description":"Deploy a self-hosted Elasticsearch cluster. Supports automatic bootstrap, zero-downtime rolling deployment, auto healing, backup, and recovery.","imageUrl":"elk.png","licenseType":"subscriber","technologies":["Terraform","Bash","JavaScript"],"compliance":[],"tags":[""]},"serviceCategoryName":"NoSQL","fileName":"README.md","filePath":"/modules/elasticsearch-cluster","title":"Repo Browser: Elasticsearch (self-hosted)","description":"Browse the repos in the Gruntwork Infrastructure as Code Library."}