This folder contains a Terraform module to take and backup snapshots of an Elasticsearch cluster to an S3 bucket. The module is a scheduled lambda function that calls the Elasticsearch API to perform snapshotting and backup related tasks documented here;
Terminologies
Snapshot: A snapshot represents the current state of the indices in an Elasticsearch cluster. This is the information stored in a backup repository.
Repository: A repository is an Elasticsearch abstraction over a storage medium like a Shared File System, S3 Bucket, HDFS etc. It's used to identify where snapshot files are stored and doesn't contain any snapshots itself.
How do you use this module?
The module works by deploying an AWS Lambda function that performs backups on a configurable schedule. This module saves all snapshots to S3 with the help of the repository-s3 plugin.
module"es_cluster_backup" {
# TODO: replace <VERSION> with the latest version from the releases page: https://github.com/gruntwork-io/package-elk/releases
source = "github.com/gruntwork-io/package-elk//modules/elasticsearch-cluster-backup?ref=<VERSION>"
name = "..."
schedule_expression = "..."
elasticsearch_dns = "..."
repository = "..."
bucket = "..."
region = "..."
}
Note the following parameters:
source: Use this parameter to specify the URL of this module. The double slash (//) is intentional and required. Terraform uses it to specify subfolders within a Git repo (see module sources). The ref parameter specifies a specific Git tag in this repo. That way, instead of using the latest version of this module from the master branch, which will change every time you run Terraform, you're using a fixed version of the repo.
name: This parameter specifies the name assigned to the lambda function.
schedule_expression: A CloudWatch schedule expression or valid cron expression to specify how often a backup should be done. E.g. cron(0 20 * * ? *), rate(5 minutes).
elasticsearch_dns: The DNS used to access the Elasticsearch cluster, must be reachable on port 9200 or use the elasticsearch_port parameter to configure a custom port.
respository: The name to assign to the repository that'll be created for all backups
bucket: The S3 bucket where snapshots will be stored. The bucket must exist as it will not be created.
Cluster snapshots are incremental. The first snapshot is always a full dump of the cluster and subsequent ones are a delta between the current state of the cluster and the previous snapshot. Snapshots are typically contained in .dat files stored in the storage medium (in this case S3) the repository points to.
CPU and Memory Usage
Snapshots are usually run on a single node which automatically co-ordinates with other nodes to ensure completenss of data. Backup of a cluster with a large volume of data will lead to high CPU and memory usage on the node performing the backup. This module makes backup requests to the cluster through the load balancer which routes the request to one of the nodes, during backup, if the selected node is unable to handle incoming requests the load balancer will direct the request to other nodes.
Frequency of Backups
How often you make backups depends entirely on the size of your deployment and the importance of your data. Larger clusters with high volume usage will typically need to be backed up more frequently than low volume clusters because of the amount of data change between snapshots. It's a safe bet to start off running backups on a nightly schedule and then continually tweak the schedule based on the demands of your cluster.
Backup Notification
The time it takes to backup a cluster is dependent on the volume of data. However, since the backup module is implemened as a Lambda function which has a maximum execution time of 5 minutes a separate notification Lambda is kicked off. A Cloudwatch metric is incremented any time the notification lambda confirms that a backup occured and an alarm connected to that metric notifies you where or not it was updated.
{"treedata":{"name":"root","toggled":true,"children":[{"name":".circleci","children":[{"name":"config.yml","path":".circleci/config.yml","sha":"c2cff2a8764e0fabad7a8408ba9e42113cc23487"}]},{"name":".gitignore","path":".gitignore","sha":"f2438833ea6b1bc45e125b598cb1e08cbd274e1b"},{"name":"CODEOWNERS","path":"CODEOWNERS","sha":"cef671f8c0aee235f6d8cf886e421cfab3b51729"},{"name":"CONTRIBUTING.md","path":"CONTRIBUTING.md","sha":"492bef006a63f2c3189fd171116eb3f900c6cd02"},{"name":"LICENSE.md","path":"LICENSE.md","sha":"a2cf01ecdd725fddd718ab91c80c115882c94f3c"},{"name":"README.md","path":"README.md","sha":"ebe0be6e9b1f79275f4f56fb3b4205755b2ade16"},{"name":"_docs","children":[{"name":"elk-architecture.png","path":"_docs/elk-architecture.png","sha":"a7585c6f40eb0a7740b34f00d412bc2e37373266"}]},{"name":"examples","children":[{"name":"elasticsearch-docker","children":[{"name":"local-mocks","children":[{"name":"entrypoint.sh","path":"examples/elasticsearch-docker/local-mocks/entrypoint.sh","sha":"8e298671493441b6d7548f619a6155a408677d99"}]},{"name":"non-ssl","children":[{"name":".env","path":"examples/elasticsearch-docker/non-ssl/.env","sha":"bc82d24eea229df2ba77f0fdf1f6e1c6f6a8e532"},{"name":"README.md","path":"examples/elasticsearch-docker/non-ssl/README.md","sha":"ed4bee52f5093a864164067b57d54f41668e55f5"},{"name":"docker-compose.yml","path":"examples/elasticsearch-docker/non-ssl/docker-compose.yml","sha":"e7547917f574679857f800b24f86bd573671931e"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/elasticsearch-docker/non-ssl/user-data/user-data.sh","sha":"81df178b3080a40a9e03ff7e94f1ef5db7d57f98"}]}]},{"name":"ssl","children":[{"name":".env","path":"examples/elasticsearch-docker/ssl/.env","sha":"bc82d24eea229df2ba77f0fdf1f6e1c6f6a8e532"},{"name":"README.md","path":"examples/elasticsearch-docker/ssl/README.md","sha":"ed4bee52f5093a864164067b57d54f41668e55f5"},{"name":"docker-compose.yml","path":"examples/elasticsearch-docker/ssl/docker-compose.yml","sha":"25e2b036c7e672a8ffc4b9559ab53f8d77441f99"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/elasticsearch-docker/ssl/user-data/user-data.sh","sha":"fc95c5a93069b251aca3ec2ae70a2be90231fcd9"}]}]}]},{"name":"elasticsearch-only-cluster","children":[{"name":"README.md","path":"examples/elasticsearch-only-cluster/README.md","sha":"5bdbc120e354d55895febb193390b8ba4440e9ac"},{"name":"main.tf","path":"examples/elasticsearch-only-cluster/main.tf","sha":"491b09a2683ba577be29cd1ca226c3b75ed55183"},{"name":"outputs.tf","path":"examples/elasticsearch-only-cluster/outputs.tf","sha":"382f2cda2c87d824b274f986f5107fd90477544a"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/elasticsearch-only-cluster/user-data/user-data.sh","sha":"55678f9678e93453686c629e364f82ebacdb83df"}]},{"name":"vars.tf","path":"examples/elasticsearch-only-cluster/vars.tf","sha":"6c3db8f2b043c53d04f7278a076230b4ce06308a"}]},{"name":"elk-amis","children":[{"name":"README.md","path":"examples/elk-amis/README.md","sha":"4e58b3a2e6be086188621bee400ff750f02436e8"},{"name":"all-in-one","children":[{"name":"README.md","path":"examples/elk-amis/all-in-one/README.md","sha":"42799bc50b4f41eb9e27ed79f1b5624b812b81b2"},{"name":"all-in-one.json","path":"examples/elk-amis/all-in-one/all-in-one.json","sha":"e0b06562c958eae8d91c371c9ebd757446ed79cd"}]},{"name":"app-server","children":[{"name":"README.md","path":"examples/elk-amis/app-server/README.md","sha":"577966ff3d1aa65d69aa43d6b7e1e86968099335"},{"name":"app-server.json","path":"examples/elk-amis/app-server/app-server.json","sha":"d9f6b7c6009f24c60fe210b77f580b58b22562aa"}]},{"name":"collectd","children":[{"name":"README.md","path":"examples/elk-amis/collectd/README.md","sha":"a9fa0d44e853022868368cab446e4aaeb24997ab"},{"name":"collectd-install-steps.sh","path":"examples/elk-amis/collectd/collectd-install-steps.sh","sha":"1fa3c68c3a1318caab275a7eb4bbaf9533b75319"},{"name":"collectd.json","path":"examples/elk-amis/collectd/collectd.json","sha":"b3f883f1cfd1866f08f411ac302ba5db7bf2fde6"},{"name":"config","children":[{"name":"collectd-ssl.conf","path":"examples/elk-amis/collectd/config/collectd-ssl.conf","sha":"ca0469d729e643644f90118e5a639671a3f60d08"},{"name":"collectd.conf","path":"examples/elk-amis/collectd/config/collectd.conf","sha":"74c1b46937039070cd95f64269eef63aabd72805"}]}]},{"name":"elastalert","children":[{"name":"README.md","path":"examples/elk-amis/elastalert/README.md","sha":"9dd33dfd29ddafa1040213466560ffcf8a66944d"},{"name":"aws","children":[{"name":"elastalert-config","children":[{"name":"config-ssl.yml","path":"examples/elk-amis/elastalert/aws/elastalert-config/config-ssl.yml","sha":"f26f8404e40736c5b0b17bc23e6a5de119d0c80f"},{"name":"config.yml","path":"examples/elk-amis/elastalert/aws/elastalert-config/config.yml","sha":"6054d91b8e86df9102fd5fd1fa2c545f7a981d31"}]}]},{"name":"docker","children":[{"name":"elastalert-config","children":[{"name":"config-ssl.yml","path":"examples/elk-amis/elastalert/docker/elastalert-config/config-ssl.yml","sha":"f26f8404e40736c5b0b17bc23e6a5de119d0c80f"},{"name":"config.yml","path":"examples/elk-amis/elastalert/docker/elastalert-config/config.yml","sha":"6054d91b8e86df9102fd5fd1fa2c545f7a981d31"}]}]},{"name":"elastalert-install-steps.sh","path":"examples/elk-amis/elastalert/elastalert-install-steps.sh","sha":"0e5cca36e97e25543d28d2d4d346b80110ceef8d"},{"name":"elastalert-rules","children":[{"name":"example_change.yml","path":"examples/elk-amis/elastalert/elastalert-rules/example_change.yml","sha":"4020886483786ff9a8847159b5750014c5d2b0fb"}]},{"name":"elastalert.json","path":"examples/elk-amis/elastalert/elastalert.json","sha":"419a67dc9d900c05d480b60452dba7aea7a3ee92"}]},{"name":"elasticsearch","children":[{"name":"README.md","path":"examples/elk-amis/elasticsearch/README.md","sha":"f77a938e72424815565ffe16b9ae7787cb430cc7"},{"name":"aws","children":[{"name":"config","children":[{"name":"elasticsearch-ssl.yml","path":"examples/elk-amis/elasticsearch/aws/config/elasticsearch-ssl.yml","sha":"aab4150ebef375ddb962e17c679642f728df538d"},{"name":"elasticsearch.yml","path":"examples/elk-amis/elasticsearch/aws/config/elasticsearch.yml","sha":"0bf97273dce0664ca96ef87151f856162b1a2411"},{"name":"jvm.options","path":"examples/elk-amis/elasticsearch/aws/config/jvm.options","sha":"28dcf519ee339aba5dd15059c1acc4b97846d6ce"}]}]},{"name":"docker","children":[{"name":"config","children":[{"name":"elasticsearch-ssl.yml","path":"examples/elk-amis/elasticsearch/docker/config/elasticsearch-ssl.yml","sha":"705af52bc1e914df93faa919de8a23b973184938"},{"name":"elasticsearch.yml","path":"examples/elk-amis/elasticsearch/docker/config/elasticsearch.yml","sha":"2ac38b9c875dd6352b7a3d7d0fc2c02662d0e9cb"},{"name":"jvm.options","path":"examples/elk-amis/elasticsearch/docker/config/jvm.options","sha":"28dcf519ee339aba5dd15059c1acc4b97846d6ce"}]}]},{"name":"elasticsearch-install-steps.sh","path":"examples/elk-amis/elasticsearch/elasticsearch-install-steps.sh","sha":"7d766638a6a8368b56ae5b81c65f3d202c907811"},{"name":"elasticsearch.json","path":"examples/elk-amis/elasticsearch/elasticsearch.json","sha":"9c91629c2ba2b14d7647bf380532e442343228e9"},{"name":"plugins","children":[{"name":"readonlyrest-1.18.5_es6.8.3.zip","path":"examples/elk-amis/elasticsearch/plugins/readonlyrest-1.18.5_es6.8.3.zip","sha":"92de49ac4f2a7e90b58dd0d04effe068483a5002"}]},{"name":"readonlyrest-config","children":[{"name":"readonlyrest.yml","path":"examples/elk-amis/elasticsearch/readonlyrest-config/readonlyrest.yml","sha":"a5efa565eca562bcc9a33b959c668e5fd7087165"}]}]},{"name":"filebeat","children":[{"name":"README.md","path":"examples/elk-amis/filebeat/README.md","sha":"ce76c3cbb426dc4d5e82f04a85eb0efd043fe671"},{"name":"config","children":[{"name":"filebeat-ssl.yml","path":"examples/elk-amis/filebeat/config/filebeat-ssl.yml","sha":"24b163e671b4cc49af08726867d8869f105a8ecb"},{"name":"filebeat.yml","path":"examples/elk-amis/filebeat/config/filebeat.yml","sha":"ae4641ee77c89e389728836fca3cac1428214fe2"}]},{"name":"filebeat-install-steps.sh","path":"examples/elk-amis/filebeat/filebeat-install-steps.sh","sha":"71192c99d9e5a1207a7a0f6f0dfa107532a21fe0"},{"name":"filebeat.json","path":"examples/elk-amis/filebeat/filebeat.json","sha":"3d87c3b35089e41f432ff160593e8e2f7ad0854f"}]},{"name":"kibana","children":[{"name":"README.md","path":"examples/elk-amis/kibana/README.md","sha":"b8af363345ba445cb9fbc2b04ec0e7d6b0b08149"},{"name":"config","children":[{"name":"kibana-ssl.yml","path":"examples/elk-amis/kibana/config/kibana-ssl.yml","sha":"9d0761e628a703eec051b31e6cce9d39e62f5ade"},{"name":"kibana.yml","path":"examples/elk-amis/kibana/config/kibana.yml","sha":"852b146720a4ab3fc1380beacce6f77851017b51"}]},{"name":"kibana-install-steps.sh","path":"examples/elk-amis/kibana/kibana-install-steps.sh","sha":"1dc5d1dd094d0533947c53da510cb5d2139556bf"},{"name":"kibana.json","path":"examples/elk-amis/kibana/kibana.json","sha":"568c21c4d7c3eb056710155a3ab50a12b4d16378"}]},{"name":"logstash","children":[{"name":"README.md","path":"examples/elk-amis/logstash/README.md","sha":"b1035b8b3d34f28000e329171ba65275b1fac258"},{"name":"config","children":[{"name":"jvm.options","path":"examples/elk-amis/logstash/config/jvm.options","sha":"6017dd218501163070f950077ce6a1e9cff8f3a4"},{"name":"logstash-ssl.yml","path":"examples/elk-amis/logstash/config/logstash-ssl.yml","sha":"46e72285a2074ceba2313f726db5eb0db6dfa414"},{"name":"logstash.yml","path":"examples/elk-amis/logstash/config/logstash.yml","sha":"46e72285a2074ceba2313f726db5eb0db6dfa414"},{"name":"pipeline-ssl.conf","path":"examples/elk-amis/logstash/config/pipeline-ssl.conf","sha":"10c09944749f51350709b628eda2f84e8345e4c5"},{"name":"pipeline.conf","path":"examples/elk-amis/logstash/config/pipeline.conf","sha":"4a68c41fc5e86106915cb1c91ec39bd4c0cd2b55"}]},{"name":"logstash-install-steps.sh","path":"examples/elk-amis/logstash/logstash-install-steps.sh","sha":"7fd979a06f44c76a70745fd014895ead31f6b693"},{"name":"logstash.json","path":"examples/elk-amis/logstash/logstash.json","sha":"558130f7b83282583a5e32ce1b0dfe820f2caf18"}]}]},{"name":"elk-multi-cluster","children":[{"name":"README.md","path":"examples/elk-multi-cluster/README.md","sha":"ca696b69e95a04bbee5ea51c1b411131d0a47b33"},{"name":"main.tf","path":"examples/elk-multi-cluster/main.tf","sha":"191b70e3e2f50c4b89fdc1ad5eef68117386c6c0"},{"name":"outputs.tf","path":"examples/elk-multi-cluster/outputs.tf","sha":"c43e3d202d3a8240c851aadba987c7a3f59908f1"},{"name":"user-data","children":[{"name":"app-server","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/app-server/user-data.sh","sha":"c369aff91cc88d849afc3eac2e00b641c8697e84"}]},{"name":"elastalert","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/elastalert/user-data.sh","sha":"48181b3b07a3aec5e5b808b4101a9ef447eed37b"}]},{"name":"elasticsearch","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/elasticsearch/user-data.sh","sha":"55678f9678e93453686c629e364f82ebacdb83df"}]},{"name":"kibana","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/kibana/user-data.sh","sha":"cb5337f5e4c5a8029554a1af33982f963e4b741b"}]},{"name":"logstash","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/logstash/user-data.sh","sha":"4a74a7ca4c180c4e53fa03e8d1d56184adedfff1"}]}]},{"name":"vars.tf","path":"examples/elk-multi-cluster/vars.tf","sha":"812eab62300f0fab7cd758b81bb7c830789e908e"}]},{"name":"elk-single-cluster","children":[{"name":"README.md","path":"examples/elk-single-cluster/README.md","sha":"f0842ece87a00338900f35f17b199604d4e46e0c"},{"name":"main.tf","path":"examples/elk-single-cluster/main.tf","sha":"5b6c0e2bfd58c7db42912edfb8eca7b95bb6dafa"},{"name":"outputs.tf","path":"examples/elk-single-cluster/outputs.tf","sha":"c12a0b236e6fcbf5881cb60276d62a432dadb87a"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/elk-single-cluster/user-data/user-data.sh","sha":"936dd8fb14577667d5aa4f1625a26ad7debf0ef5"}]},{"name":"vars.tf","path":"examples/elk-single-cluster/vars.tf","sha":"a67bfbaabbca1a1f25865570d9cc1a70d369485b"}]}]},{"name":"modules","children":[{"name":"auto-discovery","children":[{"name":"README.md","path":"modules/auto-discovery/README.md","sha":"f4de11c428f8fefd962220e7fbc4b976b4f838fe"},{"name":"bin","children":[{"name":"auto-discovery","path":"modules/auto-discovery/bin/auto-discovery","sha":"d3d8a818291a3e898b8802569a176ed1afe18725"}]},{"name":"install.sh","path":"modules/auto-discovery/install.sh","sha":"99a2f2fb2cbfaa31e752cbcc6495e5b755f034f2"}]},{"name":"beats-iam-policies","children":[{"name":"main.tf","path":"modules/beats-iam-policies/main.tf","sha":"2b3b4462aa4b3ed81068f4cea15e686d5a4b322c"},{"name":"vars.tf","path":"modules/beats-iam-policies/vars.tf","sha":"52873644e20b4cafdcc0f33dc11db99f1fa9b586"}]},{"name":"elastalert-iam-policies","children":[{"name":"README.md","path":"modules/elastalert-iam-policies/README.md","sha":"4dc09bec085bc821efa8052578b54cc4c321ea67"},{"name":"main.tf","path":"modules/elastalert-iam-policies/main.tf","sha":"989f18bad613464d4086df3305087650cf3a18bb"},{"name":"variables.tf","path":"modules/elastalert-iam-policies/variables.tf","sha":"bc55d045eba3b8df1b08f78a4657cae25ee0ea73"}]},{"name":"elastalert-security-group-rules","children":[{"name":"README.md","path":"modules/elastalert-security-group-rules/README.md","sha":"dc33e8391d9958ef1915f6a4e2104df391e2d320"},{"name":"main.tf","path":"modules/elastalert-security-group-rules/main.tf","sha":"519b25d67eb73a1587b8ad0556bba585230d1440"},{"name":"variables.tf","path":"modules/elastalert-security-group-rules/variables.tf","sha":"0d87d8de7864335c546406e1047a680f14e6ec73"}]},{"name":"elastalert","children":[{"name":"README.md","path":"modules/elastalert/README.md","sha":"8ee7b607def1754891d8ca65dde5a22552787ebb"},{"name":"main.tf","path":"modules/elastalert/main.tf","sha":"18ac92569874242356c83986ebab5f36df6f62bc"},{"name":"outputs.tf","path":"modules/elastalert/outputs.tf","sha":"aa5817acc77a75b75f4ea4acc686332c5b865618"},{"name":"vars.tf","path":"modules/elastalert/vars.tf","sha":"b51689b7b7dd8df9def896e3fb0ba1ad239a7fcc"}]},{"name":"elasticsearch-cluster-backup","children":[{"name":"README.md","path":"modules/elasticsearch-cluster-backup/README.md","sha":"6b3689afcf27cc5cde7a4048e125e97c7d82189b","toggled":true},{"name":"backup","children":[{"name":"index.js","path":"modules/elasticsearch-cluster-backup/backup/index.js","sha":"6d3ff6a2a58d70dccd55399ba8bf0e0270140279"}]},{"name":"main.tf","path":"modules/elasticsearch-cluster-backup/main.tf","sha":"f0baf650b3eca0ed959b51eb2aba759da30b1ef0"},{"name":"notification","children":[{"name":"index.js","path":"modules/elasticsearch-cluster-backup/notification/index.js","sha":"8e92e6b387974c4f0c6f007b8e3555d64e984202"}]},{"name":"outputs.tf","path":"modules/elasticsearch-cluster-backup/outputs.tf","sha":"ce7c4ee206984e74253ad6bb0bf25cd4a3965053"},{"name":"vars.tf","path":"modules/elasticsearch-cluster-backup/vars.tf","sha":"93c34c8c22ecfa0c9498e450df4886996b4bcfa1"}],"toggled":true},{"name":"elasticsearch-cluster-restore","children":[{"name":"README.md","path":"modules/elasticsearch-cluster-restore/README.md","sha":"11f4f63105657b323a60828ff155c22aaa0ebfbf"},{"name":"main.tf","path":"modules/elasticsearch-cluster-restore/main.tf","sha":"d74982003b28e2dca1ef09d48a1acb612031b63b"},{"name":"notification","children":[{"name":"index.js","path":"modules/elasticsearch-cluster-restore/notification/index.js","sha":"4e441662f92edf384c0c92a96ea5e1a7e1b0b24d"}]},{"name":"outputs.tf","path":"modules/elasticsearch-cluster-restore/outputs.tf","sha":"92454e1cf45fcc57f60c09eb8b31064efb460726"},{"name":"restore","children":[{"name":"index.js","path":"modules/elasticsearch-cluster-restore/restore/index.js","sha":"7d84ea278f01b64310fbc33d18c86a2a73b60573"}]},{"name":"vars.tf","path":"modules/elasticsearch-cluster-restore/vars.tf","sha":"b1c5309ac8d80d86b7ad12912fbf003d7177028f"}]},{"name":"elasticsearch-cluster","children":[{"name":"README.md","path":"modules/elasticsearch-cluster/README.md","sha":"3af5458f9573916635427a990fe0855d4c008f47"},{"name":"main.tf","path":"modules/elasticsearch-cluster/main.tf","sha":"6cce0fa70166851bf8da3af72a04fa38f5f5b989"},{"name":"outputs.tf","path":"modules/elasticsearch-cluster/outputs.tf","sha":"0231e24415e13f62f29d8abefe0398ddb6d0fa04"},{"name":"variables.tf","path":"modules/elasticsearch-cluster/variables.tf","sha":"8b680c1329382792bee8415ff36bd1314675a4c9"}]},{"name":"elasticsearch-iam-policies","children":[{"name":"README.md","path":"modules/elasticsearch-iam-policies/README.md","sha":"57a7117828cff0d9d5bdf7c9fe3af050e55e4aa0"},{"name":"main.tf","path":"modules/elasticsearch-iam-policies/main.tf","sha":"cdb51d7b73acb7e3689e0fa5ab554abc8d576081"},{"name":"variables.tf","path":"modules/elasticsearch-iam-policies/variables.tf","sha":"c612a588acea6049cfdb930209854e253b549729"}]},{"name":"elasticsearch-security-group-rules","children":[{"name":"README.md","path":"modules/elasticsearch-security-group-rules/README.md","sha":"5f0b25abeaa80973e4839626ac96ff306291e759"},{"name":"main.tf","path":"modules/elasticsearch-security-group-rules/main.tf","sha":"fd06486758030a63372610631b3f88d25b7f0cd9"},{"name":"outputs.tf","path":"modules/elasticsearch-security-group-rules/outputs.tf","sha":"e69de29bb2d1d6434b8b29ae775ad8c2e48c5391"},{"name":"variables.tf","path":"modules/elasticsearch-security-group-rules/variables.tf","sha":"7f055aa865ca315861e7a69f2d7741ab7883fddf"}]},{"name":"install-collectd","children":[{"name":"README.md","path":"modules/install-collectd/README.md","sha":"3f9e7dedcca0cc1b67812309594e5d57f41535f0"},{"name":"install.sh","path":"modules/install-collectd/install.sh","sha":"4090c1e8b94dd1a4d30ce68da03b73c281e53a3a"}]},{"name":"install-elastalert","children":[{"name":"README.md","path":"modules/install-elastalert/README.md","sha":"d228d557e6792eba4813df9c0ac6e454fc7c53d7"},{"name":"install.sh","path":"modules/install-elastalert/install.sh","sha":"99f9f170e2c9f3fc31a0f5a92a6a83da5c2fe431"}]},{"name":"install-elasticsearch","children":[{"name":"README.md","path":"modules/install-elasticsearch/README.md","sha":"ff4e35e43bb505354640bca43a12ef4e6da886f0"},{"name":"install.sh","path":"modules/install-elasticsearch/install.sh","sha":"54db7090509f2e7eb56659de74480e2f20422870"}]},{"name":"install-filebeat","children":[{"name":"README.md","path":"modules/install-filebeat/README.md","sha":"cb64af493f471c37fe427070fce0a95617082dd7"},{"name":"install.sh","path":"modules/install-filebeat/install.sh","sha":"b38f01c2caff489023c1f9a29fa8ea062e17ee58"}]},{"name":"install-kibana","children":[{"name":"README.md","path":"modules/install-kibana/README.md","sha":"a0320bbaf4cf83b6133af8df620053420f0c9d25"},{"name":"install.sh","path":"modules/install-kibana/install.sh","sha":"2109aefeccc5c1028a881da3bf9b5fa36c62f6d8"}]},{"name":"install-logstash","children":[{"name":"README.md","path":"modules/install-logstash/README.md","sha":"fddf142ed7b9608f76eb9d07cbd9ffa0a5443dad"},{"name":"install.sh","path":"modules/install-logstash/install.sh","sha":"a6a44552a2ec903558d494bf4151f32f3374f786"}]},{"name":"kibana-cluster","children":[{"name":"README.md","path":"modules/kibana-cluster/README.md","sha":"4c4860d81ed7d630a77f8c7700da1e9588c82b5f"},{"name":"main.tf","path":"modules/kibana-cluster/main.tf","sha":"7f9a9ecd3b4b4aaf8efcf92108fc123f4fb7ee5d"},{"name":"outputs.tf","path":"modules/kibana-cluster/outputs.tf","sha":"f32e707371481d34b2b84d4ef7f205afaa04f2d4"},{"name":"variables.tf","path":"modules/kibana-cluster/variables.tf","sha":"8306891e82fd960d9587851d3e44c6c1579685c3"}]},{"name":"kibana-security-group-rules","children":[{"name":"README.md","path":"modules/kibana-security-group-rules/README.md","sha":"c35fc72e41001065470a412a0ad9756eb2e84310"},{"name":"main.tf","path":"modules/kibana-security-group-rules/main.tf","sha":"59a0df1a9f5ceb464efa6ee73b39ecaf11240c2c"},{"name":"outputs.tf","path":"modules/kibana-security-group-rules/outputs.tf","sha":"e69de29bb2d1d6434b8b29ae775ad8c2e48c5391"},{"name":"variables.tf","path":"modules/kibana-security-group-rules/variables.tf","sha":"b89d5c7f5f1d1b0e9b5e1d239a1dec9a4ea34486"}]},{"name":"load-balancer-alb-target-group","children":[{"name":"README.md","path":"modules/load-balancer-alb-target-group/README.md","sha":"66c957276dc5cf35a341e0fd563bf394fd4b3d22"},{"name":"main.tf","path":"modules/load-balancer-alb-target-group/main.tf","sha":"7cb9fede7e1bb3c06907f90a81c64b02d78900f4"},{"name":"outputs.tf","path":"modules/load-balancer-alb-target-group/outputs.tf","sha":"cdf904445cf5099ccea5cc8820d198a1ffba4283"},{"name":"variables.tf","path":"modules/load-balancer-alb-target-group/variables.tf","sha":"4a262b0abde2721f9e3a5edeb6a98046009b9df3"}]},{"name":"logstash-cluster","children":[{"name":"README.md","path":"modules/logstash-cluster/README.md","sha":"dbc17e8fc3c4d9a61661ca59b934295ce0bcc6a5"},{"name":"main.tf","path":"modules/logstash-cluster/main.tf","sha":"f138bbca6e0bc6858f8d323f134e95b71031747d"},{"name":"outputs.tf","path":"modules/logstash-cluster/outputs.tf","sha":"218a4f0042d52ea1673de0c4f78857a4831143a6"},{"name":"vars.tf","path":"modules/logstash-cluster/vars.tf","sha":"bff728633e0f853502c3a15b4f9311b5b9d2d41f"}]},{"name":"logstash-iam-policies","children":[{"name":"README.md","path":"modules/logstash-iam-policies/README.md","sha":"3714f39713885c9698daa2973a7891d0a6fb982f"},{"name":"main.tf","path":"modules/logstash-iam-policies/main.tf","sha":"7b553a45925c6ea587b95e8de3d05f546d03e278"},{"name":"vars.tf","path":"modules/logstash-iam-policies/vars.tf","sha":"865c74386084c6eff1f46cb103aa847a6ad0da68"}]},{"name":"logstash-security-group-rules","children":[{"name":"README.md","path":"modules/logstash-security-group-rules/README.md","sha":"e96766f09754cf166f53debb653ea278613a9c8e"},{"name":"main.tf","path":"modules/logstash-security-group-rules/main.tf","sha":"5ecab3d1c9d60cf246a98f9e36296bc518caa9bb"},{"name":"outputs.tf","path":"modules/logstash-security-group-rules/outputs.tf","sha":"3641c561d0e23ef02fa83319ddafa8d1f9f813d2"},{"name":"vars.tf","path":"modules/logstash-security-group-rules/vars.tf","sha":"9e26c219ddad783caf509581fae84953eae60fe9"}]},{"name":"run-collectd","children":[{"name":"README.md","path":"modules/run-collectd/README.md","sha":"783453e215c5907681e77ab4b48e64cad42ab989"},{"name":"bin","children":[{"name":"run-collectd","path":"modules/run-collectd/bin/run-collectd","sha":"7f7971d8b564a84a251d4c253c42f9025b5a1cf7"}]},{"name":"install.sh","path":"modules/run-collectd/install.sh","sha":"d2d8126f6fc5ba95a4e12242aa6ffe9bdc945ba6"}]},{"name":"run-elastalert","children":[{"name":"README.md","path":"modules/run-elastalert/README.md","sha":"e52cf808ab755b2d56a392ee8f525a0651963472"},{"name":"bin","children":[{"name":"run-elastalert","path":"modules/run-elastalert/bin/run-elastalert","sha":"cccab0f5cf41600c1b8ad48da5173db97965440f"}]},{"name":"install.sh","path":"modules/run-elastalert/install.sh","sha":"308f895a2366fc0cd8e6669229f6928a3d888f13"}]},{"name":"run-elasticsearch","children":[{"name":"README.md","path":"modules/run-elasticsearch/README.md","sha":"8a94a393f612da88f6b1b0a50a2e66743830abac"},{"name":"bin","children":[{"name":"run-elasticsearch","path":"modules/run-elasticsearch/bin/run-elasticsearch","sha":"e8e21b99ca07e03ce807ad537fbd9c3858912713"}]},{"name":"install.sh","path":"modules/run-elasticsearch/install.sh","sha":"c66e6da3d46195ac94d68eed18ed301ca7e0eb51"}]},{"name":"run-filebeat","children":[{"name":"README.md","path":"modules/run-filebeat/README.md","sha":"07fffbedecf8232d163a6ea985669497b11f3f7b"},{"name":"bin","children":[{"name":"run-filebeat","path":"modules/run-filebeat/bin/run-filebeat","sha":"3226cfa14477efe5c94c2a660b41ff792092d119"}]},{"name":"install.sh","path":"modules/run-filebeat/install.sh","sha":"cd890712803c3961ada86acf1e13b4c299402fb6"}]},{"name":"run-kibana","children":[{"name":"README.md","path":"modules/run-kibana/README.md","sha":"57d636cd26e4af0b51364b4ffb70b382f76314e4"},{"name":"bin","children":[{"name":"run-kibana","path":"modules/run-kibana/bin/run-kibana","sha":"0f60bca1be5eebc5594de9e8eab162aa979d4a17"}]},{"name":"install.sh","path":"modules/run-kibana/install.sh","sha":"e23b447832fce06ad32ce654f94a182a521e2e38"}]},{"name":"run-logstash","children":[{"name":"README.md","path":"modules/run-logstash/README.md","sha":"c3d92b73c36bf63b4bd84356f2f4b781aa34c172"},{"name":"bin","children":[{"name":"run-logstash","path":"modules/run-logstash/bin/run-logstash","sha":"60dc8d906d8181e5a94c3ad18b5b27e53d513cf1"}]},{"name":"install.sh","path":"modules/run-logstash/install.sh","sha":"0a30f2adc2d0765b5581274263e12ce2e688b623"}]}],"toggled":true},{"name":"test","children":[{"name":"Gopkg.lock","path":"test/Gopkg.lock","sha":"d726b0d9bf25eae861d500289fb65ec2ad356105"},{"name":"Gopkg.toml","path":"test/Gopkg.toml","sha":"630429bc686d90b570c81062c7b9c0c925c5fdd3"},{"name":"README.md","path":"test/README.md","sha":"819e790d15767c851bfc438f0a1413ed2bde7d3b"},{"name":"elasticsearch_aws_simple_test.go","path":"test/elasticsearch_aws_simple_test.go","sha":"18c119d40504e2066377d334739150dbc6711a4d"},{"name":"elasticsearch_docker_test.go","path":"test/elasticsearch_docker_test.go","sha":"2aad9485f5da9cd84305d427a5d5e4a615f7a889"},{"name":"elk_aio_test.go","path":"test/elk_aio_test.go","sha":"ef12f5aa440b29507c695c4e95a96df42e07cacf"},{"name":"elk_end_to_end_test.go","path":"test/elk_end_to_end_test.go","sha":"536f9e44142c9ae528d13c5bc01c22118f4f4d77"},{"name":"notes.md","path":"test/notes.md","sha":"664915d4b948ccd194ffda80854f47c191904e3d"},{"name":"test_helpers.go","path":"test/test_helpers.go","sha":"df39c94ac081dee152740bf5809a8950efa3370f"},{"name":"test_helpers_keystore.go","path":"test/test_helpers_keystore.go","sha":"1e189342859a6561ec9e75009ecb2e55dc42fc23"},{"name":"tls_helpers.go","path":"test/tls_helpers.go","sha":"dd131a0d95b024beffa40c3ae538333ef2663891"}]}]},"detailsContent":"<h1 class=\"preview__body--title\" id=\"elasticsearch-cluster-backup\">Elasticsearch Cluster Backup</h1><div class=\"preview__body--border\"></div><p>This folder contains a <a href=\"https://www.terraform.io/\" class=\"preview__body--description--blue\" target=\"_blank\">Terraform</a> module to take and backup snapshots of an <a href=\"https://www.elastic.co/products/kibana\" class=\"preview__body--description--blue\" target=\"_blank\">Elasticsearch</a> cluster to an S3 bucket. The module is a scheduled lambda function that calls the Elasticsearch API to perform snapshotting and backup related tasks documented <a href=\"https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html\" class=\"preview__body--description--blue\" target=\"_blank\">here</a>;</p>\n<h2 class=\"preview__body--subtitle\" id=\"terminologies\">Terminologies</h2>\n<ul>\n<li><strong>Snapshot</strong>: A snapshot represents the current state of the indices in an Elasticsearch cluster. This is the information stored in a backup repository.</li>\n<li><strong>Repository</strong>: A repository is an Elasticsearch abstraction over a storage medium like a Shared File System, S3 Bucket, HDFS etc. It's used to identify where snapshot files are stored and doesn't contain any snapshots itself.</li>\n</ul>\n<h2 class=\"preview__body--subtitle\" id=\"how-do-you-use-this-module\">How do you use this module?</h2>\n<p>The module works by deploying an AWS Lambda function that performs backups on a configurable schedule. This module saves all snapshots to S3 with the help of the <a href=\"https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-s3-repository.html\" class=\"preview__body--description--blue\" target=\"_blank\">repository-s3</a> plugin.</p>\n<pre><span class=\"hljs-keyword\">module</span> <span class=\"hljs-string\">\"es_cluster_backup\"</span> {\n <span class=\"hljs-comment\"># <span class=\"hljs-doctag\">TODO:</span> replace <VERSION> with the latest version from the releases page: https://github.com/gruntwork-io/package-elk/releases</span>\n source = <span class=\"hljs-string\">\"github.com/gruntwork-io/package-elk//modules/elasticsearch-cluster-backup?ref=<VERSION>\"</span>\n\n name = <span class=\"hljs-string\">\"...\"</span>\n schedule_expression = <span class=\"hljs-string\">\"...\"</span>\n elasticsearch_dns = <span class=\"hljs-string\">\"...\"</span>\n repository = <span class=\"hljs-string\">\"...\"</span>\n bucket = <span class=\"hljs-string\">\"...\"</span>\n region = <span class=\"hljs-string\">\"...\"</span>\n}\n</pre>\n<p>Note the following parameters:</p>\n<ul>\n<li>\n<p><code>source</code>: Use this parameter to specify the URL of this module. The double slash (<code>//</code>) is intentional and required. Terraform uses it to specify subfolders within a Git repo (see <a href=\"https://www.terraform.io/docs/modules/sources.html\" class=\"preview__body--description--blue\" target=\"_blank\">module sources</a>). The <code>ref</code> parameter specifies a specific Git tag in this repo. That way, instead of using the latest version of this module from the <code>master</code> branch, which will change every time you run Terraform, you're using a fixed version of the repo.</p>\n</li>\n<li>\n<p><code>name</code>: This parameter specifies the name assigned to the lambda function.</p>\n</li>\n<li>\n<p><code>schedule_expression</code>: A CloudWatch schedule expression or valid cron expression to specify how often a backup should be done. E.g. <code>cron(0 20 * * ? *)</code>, <code>rate(5 minutes)</code>.</p>\n</li>\n<li>\n<p><code>elasticsearch_dns</code>: The DNS used to access the Elasticsearch cluster, must be reachable on port <code>9200</code> or use the <code>elasticsearch_port</code> parameter to configure a custom port.</p>\n</li>\n<li>\n<p><code>respository</code>: The name to assign to the repository that'll be created for all backups</p>\n</li>\n<li>\n<p><code>bucket</code>: The S3 bucket where snapshots will be stored. The bucket must exist as it will not be created.</p>\n</li>\n<li>\n<p><code>region</code>: The region where the S3 bucket exists.</p>\n</li>\n</ul>\n<p>You can find the other parameters in <a href=\"/repos/v0.6.0/package-elk/modules/elasticsearch-cluster-backup/vars.tf\" class=\"preview__body--description--blue\">vars.tf</a>.</p>\n<p>Check out the <a href=\"/repos/v0.6.0/package-elk/examples/elasticsearch-only-cluster\" class=\"preview__body--description--blue\">examples folder</a> for working sample code.</p>\n<h2 class=\"preview__body--subtitle\" id=\"taking-backups\">Taking Backups</h2>\n<p>Cluster snapshots are incremental. The first snapshot is always a full dump of the cluster and subsequent ones are a delta between the current state of the cluster and the previous snapshot. Snapshots are typically contained in <code>.dat</code> files stored in the storage medium (in this case S3) the repository points to.</p>\n<h3 class=\"preview__body--subtitle\" id=\"cpu-and-memory-usage\">CPU and Memory Usage</h3>\n<p>Snapshots are usually run on a single node which automatically co-ordinates with other nodes to ensure completenss of data. Backup of a cluster with a large volume of data will lead to high CPU and memory usage on the node performing the backup. This module makes backup requests to the cluster through the load balancer which routes the request to one of the nodes, during backup, if the selected node is unable to handle incoming requests the load balancer will direct the request to other nodes.</p>\n<h3 class=\"preview__body--subtitle\" id=\"frequency-of-backups\">Frequency of Backups</h3>\n<p>How often you make backups depends entirely on the size of your deployment and the importance of your data. Larger clusters with high volume usage will typically need to be backed up more frequently than low volume clusters because of the amount of data change between snapshots. It's a safe bet to start off running backups on a nightly schedule and then continually tweak the schedule based on the demands of your cluster.</p>\n<h3 class=\"preview__body--subtitle\" id=\"backup-notification\">Backup Notification</h3>\n<p>The time it takes to backup a cluster is dependent on the volume of data. However, since the backup module is implemened as a Lambda function which has a maximum execution time of 5 minutes a separate notification Lambda is kicked off. A Cloudwatch metric is incremented any time the notification lambda confirms that a backup occured and an alarm connected to that metric notifies you where or not it was updated.</p>\n<h2 class=\"preview__body--subtitle\" id=\"restoring-backups\">Restoring Backups</h2>\n<p>Restoring snapshots is handled by the <a href=\"/repos/v0.6.0/package-elk/modules/elasticsearch-cluster-restore\" class=\"preview__body--description--blue\">elasticsearch-cluster-restore module</a>.</p>\n","repoName":"package-elk","repoRef":"v0.5.1","serviceDescriptor":{"serviceName":"Elasticsearch (self-hosted)","serviceRepoName":"package-elk","serviceRepoOrg":"gruntwork-io","cloudProviders":["aws"],"description":"Deploy a self-hosted Elasticsearch cluster. Supports automatic bootstrap, zero-downtime rolling deployment, auto healing, backup, and recovery.","imageUrl":"elk.png","licenseType":"subscriber","technologies":["Terraform","Bash","JavaScript"],"compliance":[],"tags":[""]},"serviceCategoryName":"NoSQL","fileName":"README.md","filePath":"/modules/elasticsearch-cluster-backup","title":"Repo Browser: Elasticsearch (self-hosted)","description":"Browse the repos in the Gruntwork Infrastructure as Code Library."}
.circleci
main.tf
