This folder contains a Terraform module to restore backups of an Elasticsearch cluster from snapshots saved in S3. The module is a lambda function that calls the Elasticsearch API to perform cluster restore tasks documented here;
How do you use this module?
The module works by deploying an AWS Lambda function that can be manually invoked via the AWS Console. This module restores snapshots previously saved to S3 with the help of the repository-s3 plugin.
module"es_cluster_restore" {
# TODO: replace <VERSION> with the latest version from the releases page: https://github.com/gruntwork-io/terraform-aws-elk/releases
source = "github.com/gruntwork-io/terraform-aws-elk//modules/elasticsearch-cluster-restore?ref=<VERSION>"
name = "..."
elasticsearch_dns = "..."
repository = "..."
bucket = "..."
region = "..."
}
Note the following parameters:
source: Use this parameter to specify the URL of this module. The double slash (//) is intentional and required. Terraform uses it to specify subfolders within a Git repo (see module sources). The ref parameter specifies a specific Git tag in this repo. That way, instead of using the latest version of this module from the master branch, which will change every time you run Terraform, you're using a fixed version of the repo.
name: This parameter specifies the name assigned to the lambda function.
elasticsearch_dns: The DNS used to access the Elasticsearch cluster, must be reachable on port 9200 or use the elasticsearch_port parameter to configure a custom port.
respository: The name to assign to the repository where backups are stored
bucket: The existing S3 bucket where snapshots are stored.
All the above section does, is deploy the Lambda function that contains the cluster restore code. You'll need to actually invoke that function with the right snapshot ID to perform a restore. The backup module generates an ID for each snapshot it saves to S3 and this can be located in its CloudWatch logs; grep for string "Saving snapshot: <SNAPSHOT>". Snapshot index files stored along side the backup data in S3 also contain this information.
Performing a restore is quite straightforward at this point, it involves manually invoking the Lambda function via the web interface or AWS CLI. The ID of the snapshot to restore is specified in the event data passed to the Lambda:
{
"snapshotId": "<SNAPSHOT>"
}
Restoring to a different cluster
Snapshots created from a cluster can be restored to a completely different cluster, this module will transparently setup a backup repository (backed by the S3 cluster containing the snapshots) on the new cluster and the standard restore process described above will work.
You should be mindful of the difference in versions of the Elasticsearch cluster the snapshots were created with and the cluster it's being restored to. The documentation contains more information on the compatiblity matrix and how to upgrade snapshots created with older versions of Elasticsearch.
Restore Notification
The time it takes to restore a snapshot is dependent on the volume of data within that snapshot. However, since the restore module is implemened as a Lambda function which has a maximum execution time of 5 minutes a separate notification Lambda is kicked off. The notification Lambda will check the status of the restore operation and re-invoke itself until the operation is complete. The notification Lambda continiously logs the status of the restore operation to Cloudwatch.
Questions? Ask away.
We're here to talk about our services, answer any questions, give advice, or just to chat.
{"treedata":{"name":"root","toggled":true,"children":[{"name":".circleci","children":[{"name":"config.yml","path":".circleci/config.yml","sha":"9fec455dfc15ca063a578654ad29591ffd2252e1"}]},{"name":".gitignore","path":".gitignore","sha":"437200d1e2f9ac489a6f8febd3f3fa255a7fdfe9"},{"name":".pre-commit-config.yaml","path":".pre-commit-config.yaml","sha":"71195b5cefdf89443b15adbdfdc7735b11833a37"},{"name":"CODEOWNERS","path":"CODEOWNERS","sha":"19e775a4b681f94468d8fd2a6a2780cfbe91ecff"},{"name":"CONTRIBUTING.md","path":"CONTRIBUTING.md","sha":"492bef006a63f2c3189fd171116eb3f900c6cd02"},{"name":"LICENSE.md","path":"LICENSE.md","sha":"a2cf01ecdd725fddd718ab91c80c115882c94f3c"},{"name":"README.md","path":"README.md","sha":"887c739ec9651db0f4827445b56f975345b2820f"},{"name":"_docs","children":[{"name":"elk-architecture.png","path":"_docs/elk-architecture.png","sha":"a7585c6f40eb0a7740b34f00d412bc2e37373266"}]},{"name":"examples","children":[{"name":"elasticsearch-docker","children":[{"name":"local-mocks","children":[{"name":"entrypoint.sh","path":"examples/elasticsearch-docker/local-mocks/entrypoint.sh","sha":"a7405f93f4482217729e3694d14a59a331471750"}]},{"name":"non-ssl","children":[{"name":".env","path":"examples/elasticsearch-docker/non-ssl/.env","sha":"bc82d24eea229df2ba77f0fdf1f6e1c6f6a8e532"},{"name":"README.md","path":"examples/elasticsearch-docker/non-ssl/README.md","sha":"ed4bee52f5093a864164067b57d54f41668e55f5"},{"name":"docker-compose.yml","path":"examples/elasticsearch-docker/non-ssl/docker-compose.yml","sha":"e7547917f574679857f800b24f86bd573671931e"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/elasticsearch-docker/non-ssl/user-data/user-data.sh","sha":"81df178b3080a40a9e03ff7e94f1ef5db7d57f98"}]}]},{"name":"ssl","children":[{"name":".env","path":"examples/elasticsearch-docker/ssl/.env","sha":"bc82d24eea229df2ba77f0fdf1f6e1c6f6a8e532"},{"name":"README.md","path":"examples/elasticsearch-docker/ssl/README.md","sha":"ed4bee52f5093a864164067b57d54f41668e55f5"},{"name":"docker-compose.yml","path":"examples/elasticsearch-docker/ssl/docker-compose.yml","sha":"25e2b036c7e672a8ffc4b9559ab53f8d77441f99"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/elasticsearch-docker/ssl/user-data/user-data.sh","sha":"fc95c5a93069b251aca3ec2ae70a2be90231fcd9"}]}]}]},{"name":"elasticsearch-only-cluster","children":[{"name":"README.md","path":"examples/elasticsearch-only-cluster/README.md","sha":"e830acc97cf711f5087d03db0c61dc6ed82b1b8c"},{"name":"main.tf","path":"examples/elasticsearch-only-cluster/main.tf","sha":"16e304e12274a2c4cbc6c5b4c1d1758144ce259b"},{"name":"outputs.tf","path":"examples/elasticsearch-only-cluster/outputs.tf","sha":"382f2cda2c87d824b274f986f5107fd90477544a"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/elasticsearch-only-cluster/user-data/user-data.sh","sha":"1248e6d8692154f3d024ad95dab49a7a8646bc31"}]},{"name":"vars.tf","path":"examples/elasticsearch-only-cluster/vars.tf","sha":"6c3db8f2b043c53d04f7278a076230b4ce06308a"}]},{"name":"elk-amis","children":[{"name":"README.md","path":"examples/elk-amis/README.md","sha":"2d26c99b7a9a0d3a2124a771f0335eb5251f3b4c"},{"name":"all-in-one","children":[{"name":"README.md","path":"examples/elk-amis/all-in-one/README.md","sha":"42799bc50b4f41eb9e27ed79f1b5624b812b81b2"},{"name":"all-in-one.json","path":"examples/elk-amis/all-in-one/all-in-one.json","sha":"ecd2d44afe1d6040f90e304bc6cf042ec3668bc9"}]},{"name":"app-server","children":[{"name":"README.md","path":"examples/elk-amis/app-server/README.md","sha":"577966ff3d1aa65d69aa43d6b7e1e86968099335"},{"name":"app-server.json","path":"examples/elk-amis/app-server/app-server.json","sha":"6098834ef5ea26e93099f5131f4bd7c8ad77e82a"}]},{"name":"collectd","children":[{"name":"README.md","path":"examples/elk-amis/collectd/README.md","sha":"a9fa0d44e853022868368cab446e4aaeb24997ab"},{"name":"collectd-install-steps.sh","path":"examples/elk-amis/collectd/collectd-install-steps.sh","sha":"bfe1b037dcdee47751c588c4e431bfefec4661ee"},{"name":"collectd.json","path":"examples/elk-amis/collectd/collectd.json","sha":"236b08e2e60af5d34a9efabca7f49cf90772201f"},{"name":"config","children":[{"name":"collectd-ssl.conf","path":"examples/elk-amis/collectd/config/collectd-ssl.conf","sha":"0ccfc0f2aee65c0e17f5cd530c5edaccf247c3d8"},{"name":"collectd.conf","path":"examples/elk-amis/collectd/config/collectd.conf","sha":"74c1b46937039070cd95f64269eef63aabd72805"}]}]},{"name":"elastalert","children":[{"name":"README.md","path":"examples/elk-amis/elastalert/README.md","sha":"9dd33dfd29ddafa1040213466560ffcf8a66944d"},{"name":"aws","children":[{"name":"elastalert-config","children":[{"name":"config-ssl.yml","path":"examples/elk-amis/elastalert/aws/elastalert-config/config-ssl.yml","sha":"f26f8404e40736c5b0b17bc23e6a5de119d0c80f"},{"name":"config.yml","path":"examples/elk-amis/elastalert/aws/elastalert-config/config.yml","sha":"6054d91b8e86df9102fd5fd1fa2c545f7a981d31"}]}]},{"name":"docker","children":[{"name":"elastalert-config","children":[{"name":"config-ssl.yml","path":"examples/elk-amis/elastalert/docker/elastalert-config/config-ssl.yml","sha":"f26f8404e40736c5b0b17bc23e6a5de119d0c80f"},{"name":"config.yml","path":"examples/elk-amis/elastalert/docker/elastalert-config/config.yml","sha":"6054d91b8e86df9102fd5fd1fa2c545f7a981d31"}]}]},{"name":"elastalert-install-steps.sh","path":"examples/elk-amis/elastalert/elastalert-install-steps.sh","sha":"7da5b70623c85a0d61762a2ac5a2d91afd58d969"},{"name":"elastalert-rules","children":[{"name":"example_change.yml","path":"examples/elk-amis/elastalert/elastalert-rules/example_change.yml","sha":"4020886483786ff9a8847159b5750014c5d2b0fb"}]},{"name":"elastalert.json","path":"examples/elk-amis/elastalert/elastalert.json","sha":"762483ee0d35fdc4179cea051779f1076991fee5"}]},{"name":"elasticsearch","children":[{"name":"README.md","path":"examples/elk-amis/elasticsearch/README.md","sha":"f77a938e72424815565ffe16b9ae7787cb430cc7"},{"name":"aws","children":[{"name":"config","children":[{"name":"elasticsearch-ssl.yml","path":"examples/elk-amis/elasticsearch/aws/config/elasticsearch-ssl.yml","sha":"e6b0e5d3efd07f195aa18ae567789548a0eebc68"},{"name":"elasticsearch.yml","path":"examples/elk-amis/elasticsearch/aws/config/elasticsearch.yml","sha":"4918d1d6ec52dafa7741bda8212240d75b162f37"},{"name":"jvm.options","path":"examples/elk-amis/elasticsearch/aws/config/jvm.options","sha":"28dcf519ee339aba5dd15059c1acc4b97846d6ce"}]}]},{"name":"docker","children":[{"name":"config","children":[{"name":"elasticsearch-ssl.yml","path":"examples/elk-amis/elasticsearch/docker/config/elasticsearch-ssl.yml","sha":"705af52bc1e914df93faa919de8a23b973184938"},{"name":"elasticsearch.yml","path":"examples/elk-amis/elasticsearch/docker/config/elasticsearch.yml","sha":"2ac38b9c875dd6352b7a3d7d0fc2c02662d0e9cb"},{"name":"jvm.options","path":"examples/elk-amis/elasticsearch/docker/config/jvm.options","sha":"28dcf519ee339aba5dd15059c1acc4b97846d6ce"}]}]},{"name":"elasticsearch-install-steps.sh","path":"examples/elk-amis/elasticsearch/elasticsearch-install-steps.sh","sha":"a0de2d98730e3b1d0e3c5060dc80d925cb4043d3"},{"name":"elasticsearch.json","path":"examples/elk-amis/elasticsearch/elasticsearch.json","sha":"c8bfafebb715d5e1305f7c758274ce63de9f73ec"},{"name":"plugins","children":[{"name":"readonlyrest-1.22.1_es6.8.12.zip","path":"examples/elk-amis/elasticsearch/plugins/readonlyrest-1.22.1_es6.8.12.zip","sha":"93bdf77044e9ac16fb6eb8e091a1700419425ea9"}]},{"name":"readonlyrest-config","children":[{"name":"readonlyrest.yml","path":"examples/elk-amis/elasticsearch/readonlyrest-config/readonlyrest.yml","sha":"e056fad877791d455e337ff867845c26f514874a"}]}]},{"name":"filebeat","children":[{"name":"README.md","path":"examples/elk-amis/filebeat/README.md","sha":"ce76c3cbb426dc4d5e82f04a85eb0efd043fe671"},{"name":"config","children":[{"name":"filebeat-ssl.yml","path":"examples/elk-amis/filebeat/config/filebeat-ssl.yml","sha":"24b163e671b4cc49af08726867d8869f105a8ecb"},{"name":"filebeat.yml","path":"examples/elk-amis/filebeat/config/filebeat.yml","sha":"ae4641ee77c89e389728836fca3cac1428214fe2"}]},{"name":"filebeat-install-steps.sh","path":"examples/elk-amis/filebeat/filebeat-install-steps.sh","sha":"e7374df095835dddd4d89555d5529c7c3fdff175"},{"name":"filebeat.json","path":"examples/elk-amis/filebeat/filebeat.json","sha":"50b118a2452670dc1291f8e302a7045cba0c958a"}]},{"name":"kibana","children":[{"name":"README.md","path":"examples/elk-amis/kibana/README.md","sha":"b8af363345ba445cb9fbc2b04ec0e7d6b0b08149"},{"name":"config","children":[{"name":"kibana-ssl.yml","path":"examples/elk-amis/kibana/config/kibana-ssl.yml","sha":"7bcab515eb6cba2026899fb711541ebdecc97e3d"},{"name":"kibana.yml","path":"examples/elk-amis/kibana/config/kibana.yml","sha":"852b146720a4ab3fc1380beacce6f77851017b51"}]},{"name":"kibana-install-steps.sh","path":"examples/elk-amis/kibana/kibana-install-steps.sh","sha":"3be2e4579c93962ad8da899728ac3b496a5cd78e"},{"name":"kibana.json","path":"examples/elk-amis/kibana/kibana.json","sha":"6293086359621b813fc384d580598de122f29db0"}]},{"name":"logstash","children":[{"name":"README.md","path":"examples/elk-amis/logstash/README.md","sha":"b1035b8b3d34f28000e329171ba65275b1fac258"},{"name":"config","children":[{"name":"jvm.options","path":"examples/elk-amis/logstash/config/jvm.options","sha":"6017dd218501163070f950077ce6a1e9cff8f3a4"},{"name":"logstash-ssl.yml","path":"examples/elk-amis/logstash/config/logstash-ssl.yml","sha":"46e72285a2074ceba2313f726db5eb0db6dfa414"},{"name":"logstash.yml","path":"examples/elk-amis/logstash/config/logstash.yml","sha":"46e72285a2074ceba2313f726db5eb0db6dfa414"},{"name":"pipeline-ssl.conf","path":"examples/elk-amis/logstash/config/pipeline-ssl.conf","sha":"723780826865edbc2274837265a9059f9c7c1397"},{"name":"pipeline.conf","path":"examples/elk-amis/logstash/config/pipeline.conf","sha":"4a68c41fc5e86106915cb1c91ec39bd4c0cd2b55"}]},{"name":"logstash-install-steps.sh","path":"examples/elk-amis/logstash/logstash-install-steps.sh","sha":"e44ef50df6ac8989bfae91012cf04bbe386121b2"},{"name":"logstash.json","path":"examples/elk-amis/logstash/logstash.json","sha":"5ebdff0ca36a3bbe30043c3ef9b88959d1cdf3a8"}]}]},{"name":"elk-multi-cluster","children":[{"name":"README.md","path":"examples/elk-multi-cluster/README.md","sha":"8fd9aa87b62930d731636f72f6c0d22f900cb087"},{"name":"main.tf","path":"examples/elk-multi-cluster/main.tf","sha":"41e03942f41795d52e001ad11c5ea974abbf0dea"},{"name":"outputs.tf","path":"examples/elk-multi-cluster/outputs.tf","sha":"8bc4cde32815078921bc75375d963b533f6c50db"},{"name":"user-data","children":[{"name":"app-server","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/app-server/user-data.sh","sha":"e3ab01683b870b0bd96ad620d4b25860a7d7a833"}]},{"name":"elastalert","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/elastalert/user-data.sh","sha":"48181b3b07a3aec5e5b808b4101a9ef447eed37b"}]},{"name":"elasticsearch","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/elasticsearch/user-data.sh","sha":"7c7f36d79d716ff7e7618455097513aa936919bf"}]},{"name":"kibana","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/kibana/user-data.sh","sha":"b7f5e2b2d8c7091f1aa3e036a723913d9feca37b"}]},{"name":"logstash","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/logstash/user-data.sh","sha":"9e1af5d687be9c44149560d2a022b62b88743b44"}]}]},{"name":"vars.tf","path":"examples/elk-multi-cluster/vars.tf","sha":"43f00586a4b0fb616a2904f6bb11112ab334f36e"}]},{"name":"elk-single-cluster","children":[{"name":"README.md","path":"examples/elk-single-cluster/README.md","sha":"33320b782aca0aea4d298599e5d3d706c93a12cf"},{"name":"main.tf","path":"examples/elk-single-cluster/main.tf","sha":"1c58be21a641bc4c0bdb683fcc5afd37258a8649"},{"name":"outputs.tf","path":"examples/elk-single-cluster/outputs.tf","sha":"c12a0b236e6fcbf5881cb60276d62a432dadb87a"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/elk-single-cluster/user-data/user-data.sh","sha":"446a992b93c4a3a53a86377546b0cc520bc941d3"}]},{"name":"vars.tf","path":"examples/elk-single-cluster/vars.tf","sha":"a67bfbaabbca1a1f25865570d9cc1a70d369485b"}]}]},{"name":"modules","children":[{"name":"auto-discovery","children":[{"name":"README.md","path":"modules/auto-discovery/README.md","sha":"f4de11c428f8fefd962220e7fbc4b976b4f838fe"},{"name":"bin","children":[{"name":"auto-discovery","path":"modules/auto-discovery/bin/auto-discovery","sha":"fa1f2368e4ff71c550151ed9bd67107515d04873"}]},{"name":"install.sh","path":"modules/auto-discovery/install.sh","sha":"99a2f2fb2cbfaa31e752cbcc6495e5b755f034f2"}]},{"name":"beats-iam-policies","children":[{"name":"main.tf","path":"modules/beats-iam-policies/main.tf","sha":"e9a8c04747f61ad5aa049421312e955773f656e6"},{"name":"vars.tf","path":"modules/beats-iam-policies/vars.tf","sha":"52873644e20b4cafdcc0f33dc11db99f1fa9b586"}]},{"name":"elastalert-iam-policies","children":[{"name":"README.md","path":"modules/elastalert-iam-policies/README.md","sha":"012578675e314c2985c4c4c2ef7e58bd8eb37956"},{"name":"main.tf","path":"modules/elastalert-iam-policies/main.tf","sha":"e4f7ef96c2a388c60e6a6fb315f8cdac3256e280"},{"name":"variables.tf","path":"modules/elastalert-iam-policies/variables.tf","sha":"bc55d045eba3b8df1b08f78a4657cae25ee0ea73"}]},{"name":"elastalert-security-group-rules","children":[{"name":"README.md","path":"modules/elastalert-security-group-rules/README.md","sha":"050c72419243011fbcbdd5359764a2f9a22d0aaa"},{"name":"main.tf","path":"modules/elastalert-security-group-rules/main.tf","sha":"dae2fbeebd5a32a9dcd12194e7a433ea9762f52d"},{"name":"variables.tf","path":"modules/elastalert-security-group-rules/variables.tf","sha":"0d87d8de7864335c546406e1047a680f14e6ec73"}]},{"name":"elastalert","children":[{"name":"README.md","path":"modules/elastalert/README.md","sha":"ebc03e5af2cd6e03520a8e9ab126b34c9782acc6"},{"name":"main.tf","path":"modules/elastalert/main.tf","sha":"241f707360fec4cf509098fffe13f7f2bede372d"},{"name":"outputs.tf","path":"modules/elastalert/outputs.tf","sha":"aa5817acc77a75b75f4ea4acc686332c5b865618"},{"name":"vars.tf","path":"modules/elastalert/vars.tf","sha":"bd053dc8923e14a406bae848f75c92e4f0e8a822"}]},{"name":"elasticsearch-cluster-backup","children":[{"name":"README.md","path":"modules/elasticsearch-cluster-backup/README.md","sha":"2c3ca6928de82cfbdb8b96a4eaf5a2b86a0d39c7"},{"name":"backup","children":[{"name":"index.js","path":"modules/elasticsearch-cluster-backup/backup/index.js","sha":"6d3ff6a2a58d70dccd55399ba8bf0e0270140279"}]},{"name":"main.tf","path":"modules/elasticsearch-cluster-backup/main.tf","sha":"f6ed9528ec7b0f77513c638ad6410c118474353b"},{"name":"notification","children":[{"name":"index.js","path":"modules/elasticsearch-cluster-backup/notification/index.js","sha":"8e92e6b387974c4f0c6f007b8e3555d64e984202"}]},{"name":"outputs.tf","path":"modules/elasticsearch-cluster-backup/outputs.tf","sha":"ce7c4ee206984e74253ad6bb0bf25cd4a3965053"},{"name":"vars.tf","path":"modules/elasticsearch-cluster-backup/vars.tf","sha":"93c34c8c22ecfa0c9498e450df4886996b4bcfa1"}]},{"name":"elasticsearch-cluster-restore","children":[{"name":"README.md","path":"modules/elasticsearch-cluster-restore/README.md","sha":"754fa372fff69c4726499744ff41a83fe5c40446","toggled":true},{"name":"main.tf","path":"modules/elasticsearch-cluster-restore/main.tf","sha":"f960c5c7e8298f62557bee5afef162721e7029f0"},{"name":"notification","children":[{"name":"index.js","path":"modules/elasticsearch-cluster-restore/notification/index.js","sha":"4e441662f92edf384c0c92a96ea5e1a7e1b0b24d"}]},{"name":"outputs.tf","path":"modules/elasticsearch-cluster-restore/outputs.tf","sha":"92454e1cf45fcc57f60c09eb8b31064efb460726"},{"name":"restore","children":[{"name":"index.js","path":"modules/elasticsearch-cluster-restore/restore/index.js","sha":"7d84ea278f01b64310fbc33d18c86a2a73b60573"}]},{"name":"vars.tf","path":"modules/elasticsearch-cluster-restore/vars.tf","sha":"b1c5309ac8d80d86b7ad12912fbf003d7177028f"}],"toggled":true},{"name":"elasticsearch-cluster","children":[{"name":"README.md","path":"modules/elasticsearch-cluster/README.md","sha":"96381fd14e010c475343d0eee79641c13ffe3e4e"},{"name":"main.tf","path":"modules/elasticsearch-cluster/main.tf","sha":"13d41726d7cf5dc768c0414390d922bdd9da450f"},{"name":"outputs.tf","path":"modules/elasticsearch-cluster/outputs.tf","sha":"0231e24415e13f62f29d8abefe0398ddb6d0fa04"},{"name":"variables.tf","path":"modules/elasticsearch-cluster/variables.tf","sha":"8b680c1329382792bee8415ff36bd1314675a4c9"}]},{"name":"elasticsearch-iam-policies","children":[{"name":"README.md","path":"modules/elasticsearch-iam-policies/README.md","sha":"efe4837043dbce44ce15e2b9959d2cfa11983485"},{"name":"main.tf","path":"modules/elasticsearch-iam-policies/main.tf","sha":"55bf1f01c743d84523b3d24ab9e41787a1e28d39"},{"name":"variables.tf","path":"modules/elasticsearch-iam-policies/variables.tf","sha":"c612a588acea6049cfdb930209854e253b549729"}]},{"name":"elasticsearch-security-group-rules","children":[{"name":"README.md","path":"modules/elasticsearch-security-group-rules/README.md","sha":"3292b4b51529305c57cccd4687c64bfa2a2dd6ec"},{"name":"main.tf","path":"modules/elasticsearch-security-group-rules/main.tf","sha":"cdb770582d24b7a9fc7502dd395c6f98a54acbac"},{"name":"outputs.tf","path":"modules/elasticsearch-security-group-rules/outputs.tf","sha":"e69de29bb2d1d6434b8b29ae775ad8c2e48c5391"},{"name":"variables.tf","path":"modules/elasticsearch-security-group-rules/variables.tf","sha":"7f055aa865ca315861e7a69f2d7741ab7883fddf"}]},{"name":"install-collectd","children":[{"name":"README.md","path":"modules/install-collectd/README.md","sha":"ed7f82bbf5f6a3cd5042b8cf78a8b69292f6a929"},{"name":"install.sh","path":"modules/install-collectd/install.sh","sha":"4090c1e8b94dd1a4d30ce68da03b73c281e53a3a"}]},{"name":"install-elastalert","children":[{"name":"README.md","path":"modules/install-elastalert/README.md","sha":"ab3850634329401b2734c5ed46064f93afbb989e"},{"name":"install.sh","path":"modules/install-elastalert/install.sh","sha":"2428d52ac73bd555044c5848546aac8b564c3a5f"}]},{"name":"install-elasticsearch","children":[{"name":"README.md","path":"modules/install-elasticsearch/README.md","sha":"ca54645f1ea89c80bb6953a2e5d70abec68a5346"},{"name":"install.sh","path":"modules/install-elasticsearch/install.sh","sha":"89c6d78790ed08a49af5821e04a459ec4c6b89d6"}]},{"name":"install-filebeat","children":[{"name":"README.md","path":"modules/install-filebeat/README.md","sha":"e8aea474de93cfb6421e038bdfd0459bf9936055"},{"name":"install.sh","path":"modules/install-filebeat/install.sh","sha":"b7537d09a916f93ded0214c5386d897e2f38bf56"}]},{"name":"install-kibana","children":[{"name":"README.md","path":"modules/install-kibana/README.md","sha":"9517d9c1b2f73823a00f8c1b190bb90e5b90312e"},{"name":"install.sh","path":"modules/install-kibana/install.sh","sha":"35eeef5d8aa75b4a1dd54d2913acf6688429c780"}]},{"name":"install-logstash","children":[{"name":"README.md","path":"modules/install-logstash/README.md","sha":"e1a8bde78e9bae25b852d7e9e44d0c6f372c3294"},{"name":"install.sh","path":"modules/install-logstash/install.sh","sha":"a689e67980aa28d7d2f04f693ccb59138ccfe3e2"}]},{"name":"kibana-cluster","children":[{"name":"README.md","path":"modules/kibana-cluster/README.md","sha":"87b31b9825999d66d6d334b48dc73de56ba647f7"},{"name":"main.tf","path":"modules/kibana-cluster/main.tf","sha":"7eb4359660f7951c4551e44babd9649568f2be31"},{"name":"outputs.tf","path":"modules/kibana-cluster/outputs.tf","sha":"f32e707371481d34b2b84d4ef7f205afaa04f2d4"},{"name":"variables.tf","path":"modules/kibana-cluster/variables.tf","sha":"58d03200bff5a18c52347d42af2a6ac1a6452802"}]},{"name":"kibana-security-group-rules","children":[{"name":"README.md","path":"modules/kibana-security-group-rules/README.md","sha":"f8d904f11c6a2238c5be9100c7e0255de28e5e2d"},{"name":"main.tf","path":"modules/kibana-security-group-rules/main.tf","sha":"f890ae74fae825d5db87f5011ba6638af874fda2"},{"name":"outputs.tf","path":"modules/kibana-security-group-rules/outputs.tf","sha":"e69de29bb2d1d6434b8b29ae775ad8c2e48c5391"},{"name":"variables.tf","path":"modules/kibana-security-group-rules/variables.tf","sha":"b89d5c7f5f1d1b0e9b5e1d239a1dec9a4ea34486"}]},{"name":"load-balancer-alb-target-group","children":[{"name":"README.md","path":"modules/load-balancer-alb-target-group/README.md","sha":"0a95fccf4f76af1149a6ab10d2854156e28f03f3"},{"name":"main.tf","path":"modules/load-balancer-alb-target-group/main.tf","sha":"e96c609cc002f2a9de68147aeb5ea60b6d1eb108"},{"name":"outputs.tf","path":"modules/load-balancer-alb-target-group/outputs.tf","sha":"cdf904445cf5099ccea5cc8820d198a1ffba4283"},{"name":"variables.tf","path":"modules/load-balancer-alb-target-group/variables.tf","sha":"4a262b0abde2721f9e3a5edeb6a98046009b9df3"}]},{"name":"logstash-cluster","children":[{"name":"README.md","path":"modules/logstash-cluster/README.md","sha":"78e4b47484d9f68ad16ffe06fb1af15991282cfe"},{"name":"main.tf","path":"modules/logstash-cluster/main.tf","sha":"55219e71d6f49ec878fafebad547e2573e4449be"},{"name":"outputs.tf","path":"modules/logstash-cluster/outputs.tf","sha":"218a4f0042d52ea1673de0c4f78857a4831143a6"},{"name":"vars.tf","path":"modules/logstash-cluster/vars.tf","sha":"bff728633e0f853502c3a15b4f9311b5b9d2d41f"}]},{"name":"logstash-iam-policies","children":[{"name":"README.md","path":"modules/logstash-iam-policies/README.md","sha":"3e354879298dbe3456cf3bae79594ee815487ac4"},{"name":"main.tf","path":"modules/logstash-iam-policies/main.tf","sha":"0ce69e25d9c2f964dc35f41e10813af86f2a5a7c"},{"name":"vars.tf","path":"modules/logstash-iam-policies/vars.tf","sha":"865c74386084c6eff1f46cb103aa847a6ad0da68"}]},{"name":"logstash-security-group-rules","children":[{"name":"README.md","path":"modules/logstash-security-group-rules/README.md","sha":"7ca6878f6b4415297d21c96ca48a4d8aa0314095"},{"name":"main.tf","path":"modules/logstash-security-group-rules/main.tf","sha":"dd54760fec89fa4a83ed37835d9901256af24bf5"},{"name":"outputs.tf","path":"modules/logstash-security-group-rules/outputs.tf","sha":"3641c561d0e23ef02fa83319ddafa8d1f9f813d2"},{"name":"vars.tf","path":"modules/logstash-security-group-rules/vars.tf","sha":"9e26c219ddad783caf509581fae84953eae60fe9"}]},{"name":"run-collectd","children":[{"name":"README.md","path":"modules/run-collectd/README.md","sha":"2242bcaca7f50717b3d6154c0576847d7b08c12c"},{"name":"bin","children":[{"name":"run-collectd","path":"modules/run-collectd/bin/run-collectd","sha":"7f7971d8b564a84a251d4c253c42f9025b5a1cf7"}]},{"name":"install.sh","path":"modules/run-collectd/install.sh","sha":"d2d8126f6fc5ba95a4e12242aa6ffe9bdc945ba6"}]},{"name":"run-elastalert","children":[{"name":"README.md","path":"modules/run-elastalert/README.md","sha":"e52cf808ab755b2d56a392ee8f525a0651963472"},{"name":"bin","children":[{"name":"run-elastalert","path":"modules/run-elastalert/bin/run-elastalert","sha":"cccab0f5cf41600c1b8ad48da5173db97965440f"}]},{"name":"install.sh","path":"modules/run-elastalert/install.sh","sha":"308f895a2366fc0cd8e6669229f6928a3d888f13"}]},{"name":"run-elasticsearch","children":[{"name":"README.md","path":"modules/run-elasticsearch/README.md","sha":"598ecd294a3177b5f9791793f75cec95559ad886"},{"name":"bin","children":[{"name":"run-elasticsearch","path":"modules/run-elasticsearch/bin/run-elasticsearch","sha":"e8e21b99ca07e03ce807ad537fbd9c3858912713"}]},{"name":"install.sh","path":"modules/run-elasticsearch/install.sh","sha":"c66e6da3d46195ac94d68eed18ed301ca7e0eb51"}]},{"name":"run-filebeat","children":[{"name":"README.md","path":"modules/run-filebeat/README.md","sha":"7b20727ebd231d78a39f30eeb2bad79619447f0a"},{"name":"bin","children":[{"name":"run-filebeat","path":"modules/run-filebeat/bin/run-filebeat","sha":"2e323cd6b7550db8470085911fa372c4edad8ae6"}]},{"name":"install.sh","path":"modules/run-filebeat/install.sh","sha":"cd890712803c3961ada86acf1e13b4c299402fb6"}]},{"name":"run-kibana","children":[{"name":"README.md","path":"modules/run-kibana/README.md","sha":"f6aaf8c3bd3901d753551ab4c28c0772d73ed143"},{"name":"bin","children":[{"name":"run-kibana","path":"modules/run-kibana/bin/run-kibana","sha":"0f60bca1be5eebc5594de9e8eab162aa979d4a17"}]},{"name":"install.sh","path":"modules/run-kibana/install.sh","sha":"e23b447832fce06ad32ce654f94a182a521e2e38"}]},{"name":"run-logstash","children":[{"name":"README.md","path":"modules/run-logstash/README.md","sha":"e1b6d3752f9f2f917c4bb17283caabd163f4d589"},{"name":"bin","children":[{"name":"run-logstash","path":"modules/run-logstash/bin/run-logstash","sha":"60dc8d906d8181e5a94c3ad18b5b27e53d513cf1"}]},{"name":"install.sh","path":"modules/run-logstash/install.sh","sha":"0a30f2adc2d0765b5581274263e12ce2e688b623"}]}],"toggled":true},{"name":"terraform-cloud-enterprise-private-module-registry-placeholder.tf","path":"terraform-cloud-enterprise-private-module-registry-placeholder.tf","sha":"ae586c0fe830819580e1009d41a9074f16e65bed"},{"name":"test","children":[{"name":"README.md","path":"test/README.md","sha":"819e790d15767c851bfc438f0a1413ed2bde7d3b"},{"name":"elasticsearch_aws_simple_test.go","path":"test/elasticsearch_aws_simple_test.go","sha":"cccda6ac30573bc7fea6772f56f68a96ab1b0581"},{"name":"elasticsearch_docker_test.go","path":"test/elasticsearch_docker_test.go","sha":"9aab286b753db9127f7425acea12bcc93e5abdd1"},{"name":"elk_aio_test.go","path":"test/elk_aio_test.go","sha":"b1ca766cbee9d5b69e1028ef48e971f4d333232b"},{"name":"elk_end_to_end_test.go","path":"test/elk_end_to_end_test.go","sha":"fb74457130157625a2d817e0d54d6be55aa6a3c2"},{"name":"go.mod","path":"test/go.mod","sha":"b6d375ede5dfd840285c97e9e8c1e95a6b8cfb52"},{"name":"go.sum","path":"test/go.sum","sha":"c64c51d793c7c3ec7848f7998ddb9494a979ea59"},{"name":"notes.md","path":"test/notes.md","sha":"664915d4b948ccd194ffda80854f47c191904e3d"},{"name":"test_helpers.go","path":"test/test_helpers.go","sha":"0ba1e792e7a26419f43c55971e4910cbc242a009"},{"name":"test_helpers_keystore.go","path":"test/test_helpers_keystore.go","sha":"e3b1ed8963088a2bcbdc2863287e60d19d23cd47"},{"name":"tls_helpers.go","path":"test/tls_helpers.go","sha":"828f04c10b9ca6f4b6f8f82ddcd498bdb1c7dbb4"}]}]},"detailsContent":"<h1 class=\"preview__body--title\" id=\"elasticsearch-cluster-restore\">Elasticsearch Cluster Restore</h1><div class=\"preview__body--border\"></div><p>This folder contains a <a href=\"https://www.terraform.io/\" class=\"preview__body--description--blue\" target=\"_blank\">Terraform</a> module to restore backups of an <a href=\"https://www.elastic.co/products/kibana\" class=\"preview__body--description--blue\" target=\"_blank\">Elasticsearch</a> cluster from snapshots saved in S3. The module is a lambda function that calls the Elasticsearch API to perform cluster restore tasks documented <a href=\"https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html\" class=\"preview__body--description--blue\" target=\"_blank\">here</a>;</p>\n<h2 class=\"preview__body--subtitle\" id=\"how-do-you-use-this-module\">How do you use this module?</h2>\n<p>The module works by deploying an AWS Lambda function that can be manually invoked via the AWS Console. This module restores snapshots previously saved to S3 with the help of the <a href=\"https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-s3-repository.html\" class=\"preview__body--description--blue\" target=\"_blank\">repository-s3</a> plugin.</p>\n<pre><span class=\"hljs-keyword\">module</span> <span class=\"hljs-string\">\"es_cluster_restore\"</span> {\n <span class=\"hljs-comment\"># <span class=\"hljs-doctag\">TODO:</span> replace <VERSION> with the latest version from the releases page: https://github.com/gruntwork-io/terraform-aws-elk/releases</span>\n source = <span class=\"hljs-string\">\"github.com/gruntwork-io/terraform-aws-elk//modules/elasticsearch-cluster-restore?ref=<VERSION>\"</span>\n\n name = <span class=\"hljs-string\">\"...\"</span>\n elasticsearch_dns = <span class=\"hljs-string\">\"...\"</span>\n repository = <span class=\"hljs-string\">\"...\"</span>\n bucket = <span class=\"hljs-string\">\"...\"</span>\n region = <span class=\"hljs-string\">\"...\"</span>\n}\n</pre>\n<p>Note the following parameters:</p>\n<ul>\n<li>\n<p><code>source</code>: Use this parameter to specify the URL of this module. The double slash (<code>//</code>) is intentional and required. Terraform uses it to specify subfolders within a Git repo (see <a href=\"https://www.terraform.io/docs/modules/sources.html\" class=\"preview__body--description--blue\" target=\"_blank\">module sources</a>). The <code>ref</code> parameter specifies a specific Git tag in this repo. That way, instead of using the latest version of this module from the <code>master</code> branch, which will change every time you run Terraform, you're using a fixed version of the repo.</p>\n</li>\n<li>\n<p><code>name</code>: This parameter specifies the name assigned to the lambda function.</p>\n</li>\n<li>\n<p><code>elasticsearch_dns</code>: The DNS used to access the Elasticsearch cluster, must be reachable on port <code>9200</code> or use the <code>elasticsearch_port</code> parameter to configure a custom port.</p>\n</li>\n<li>\n<p><code>respository</code>: The name to assign to the repository where backups are stored</p>\n</li>\n<li>\n<p><code>bucket</code>: The <em>existing</em> S3 bucket where snapshots are stored.</p>\n</li>\n<li>\n<p><code>region</code>: The region where the S3 bucket exists.</p>\n</li>\n</ul>\n<p>You can find the other parameters in <a href=\"/repos/v0.11.0/package-elk/modules/elasticsearch-cluster-restore/vars.tf\" class=\"preview__body--description--blue\">vars.tf</a>.</p>\n<p>Check out the <a href=\"/repos/v0.11.0/package-elk/examples/elasticsearch-only-cluster\" class=\"preview__body--description--blue\">examples folder</a> for working sample code.</p>\n<h2 class=\"preview__body--subtitle\" id=\"restoring-snapshots\">Restoring snapshots</h2>\n<p>All the above section does, is deploy the Lambda function that contains the cluster restore code. You'll need to actually invoke that function with the right snapshot ID to perform a restore. The backup module generates an ID for each snapshot it saves to S3 and this can be located in its CloudWatch logs; grep for string <code>"Saving snapshot: <SNAPSHOT>"</code>. Snapshot index files stored along side the backup data in S3 also contain this information.</p>\n<p>Performing a restore is quite straightforward at this point, it involves manually invoking the Lambda function via the <a href=\"https://us-east-2.console.aws.amazon.com/lambda/home\" class=\"preview__body--description--blue\" target=\"_blank\">web interface</a> or <a href=\"https://docs.aws.amazon.com/lambda/latest/dg/with-on-demand-custom-android-example-upload-deployment-pkg.html#walkthrough-on-demand-custom-android-events-adminuser-create-test-function-upload-zip-test-manual-invoke\" class=\"preview__body--description--blue\" target=\"_blank\">AWS CLI</a>. The ID of the snapshot to restore is specified in the event data passed to the Lambda:</p>\n<pre>{\n <span class=\"hljs-attr\">\"snapshotId\"</span>: <span class=\"hljs-string\">\"<SNAPSHOT>\"</span>\n}\n</pre>\n<h3 class=\"preview__body--subtitle\" id=\"restoring-to-a-different-cluster\">Restoring to a different cluster</h3>\n<p>Snapshots created from a cluster can be restored to a completely different cluster, this module will transparently setup a backup repository (backed by the S3 cluster containing the snapshots) on the new cluster and the standard restore process described above will work.</p>\n<p>You should be mindful of the difference in versions of the Elasticsearch cluster the snapshots were created with and the cluster it's being restored to. The <a href=\"https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html#modules-snapshots\" class=\"preview__body--description--blue\" target=\"_blank\">documentation</a> contains more information on the compatiblity matrix and how to upgrade snapshots created with older versions of Elasticsearch.</p>\n<h3 class=\"preview__body--subtitle\" id=\"restore-notification\">Restore Notification</h3>\n<p>The time it takes to restore a snapshot is dependent on the volume of data within that snapshot. However, since the restore module is implemened as a Lambda function which has a maximum execution time of 5 minutes a separate notification Lambda is kicked off. The notification Lambda will check the status of the restore operation and re-invoke itself until the operation is complete. The notification Lambda continiously logs the status of the restore operation to Cloudwatch.</p>\n","repoName":"package-elk","repoRef":"v0.9.0","serviceDescriptor":{"serviceName":"Elasticsearch (self-hosted)","serviceRepoName":"package-elk","serviceRepoOrg":"gruntwork-io","cloudProviders":["aws"],"description":"Deploy a self-hosted Elasticsearch cluster. Supports automatic bootstrap, zero-downtime rolling deployment, auto healing, backup, and recovery.","imageUrl":"elk.png","licenseType":"subscriber","technologies":["Terraform","Bash","JavaScript"],"compliance":[],"tags":[""]},"serviceCategoryName":"NoSQL","fileName":"README.md","filePath":"/modules/elasticsearch-cluster-restore","title":"Repo Browser: Elasticsearch (self-hosted)","description":"Browse the repos in the Gruntwork Infrastructure as Code Library."}
.circleci
main.tf
