This folder contains a script that allows auto discovery of nodes in a cluster. This script has been tested on the following operating systems:
Ubuntu 16.04
Ubuntu 18.04
CentOS 7
Amazon Linux 2
Usage
Run auto-discovery --help to see all available arguments
Usage: auto-discovery [options]
This script can run on a scheduled basis to (a) automatically discover the IPs of other EC2 Instances (b) write these IPs to a config file, (c) restart the service using that config file.
Options:
--tag The tag key and value on the instances.
--port The port on which the server nodes listen on for connections.
--aws-region The AWS region where the instances are deployed.
--use-public-ips Settotrueto use public IPs andfalseto use private IPs. Default: true
--config-file The path to a configuration file.
--service-name The name of the systemd service to restart after updating the specified configuration file.
--pattern Regex pattern used to identify where to update in the specified configuration file.
--schedule A cron expression specifying the schedule this script will run with otherwise script will run only once.
--user The user to schedule the cron job for. Defaults to the user that executes the script
--help Show this help text and exit.
Example:
auto-discovery --tag 'Name=instance-0' --port 5044 --aws-region 'us-east-1' --config-file ./filebeat.yml --pattern 'hosts:.*' --service-name 'filebeat'
How it works
The auto-discovery script works in the following steps:
Makes a request to AWS to retrive all pending or running instances with a specific tag set to a the specified value.
Retrieves the public or private IP addresses of the returned instances depending on the value of the --use-public-ips flag.
Replaces a portion of the specified configuration file with a comma separated list of IP and port pairs (e.g "52.72.28.213:8065", "54.210.77.143:8065") using the specified regex pattern as a marker
Restarts the specified systemd service, which presumbly uses the just updated configuration file and needs to reload it.
If --schedule is set, this script schedules itself with cron to be run on the specified schedule.
LIMITATION: Unfortunately, the approach described above can't work if the application for which we are doing discovery
is configured to do full SSL certificate verification (ie: hostname verification). Since our approach looks up randomly
assigned IP addresses, there's no way we could generate an SSL certificate that encapsulates all of the possible IP addresses
that may be found. For this reason, we have to disable SSL hostname verification when using this module.
Future Improvements
To work around the limitation described above, we will be making changes that will update local /etc/hosts or run a
local instance of Dnsmasq in order to make seamless updates to the DNS
instead of updating an application's config file and then needing to restart that application.
Questions? Ask away.
We're here to talk about our services, answer any questions, give advice, or just to chat.
{"treedata":{"name":"root","toggled":true,"children":[{"name":".circleci","children":[{"name":"config.yml","path":".circleci/config.yml","sha":"9fec455dfc15ca063a578654ad29591ffd2252e1"}]},{"name":".gitignore","path":".gitignore","sha":"437200d1e2f9ac489a6f8febd3f3fa255a7fdfe9"},{"name":".pre-commit-config.yaml","path":".pre-commit-config.yaml","sha":"71195b5cefdf89443b15adbdfdc7735b11833a37"},{"name":"CODEOWNERS","path":"CODEOWNERS","sha":"19e775a4b681f94468d8fd2a6a2780cfbe91ecff"},{"name":"CONTRIBUTING.md","path":"CONTRIBUTING.md","sha":"492bef006a63f2c3189fd171116eb3f900c6cd02"},{"name":"LICENSE.md","path":"LICENSE.md","sha":"a2cf01ecdd725fddd718ab91c80c115882c94f3c"},{"name":"README.md","path":"README.md","sha":"887c739ec9651db0f4827445b56f975345b2820f"},{"name":"_docs","children":[{"name":"elk-architecture.png","path":"_docs/elk-architecture.png","sha":"a7585c6f40eb0a7740b34f00d412bc2e37373266"}]},{"name":"examples","children":[{"name":"elasticsearch-docker","children":[{"name":"local-mocks","children":[{"name":"entrypoint.sh","path":"examples/elasticsearch-docker/local-mocks/entrypoint.sh","sha":"a7405f93f4482217729e3694d14a59a331471750"}]},{"name":"non-ssl","children":[{"name":".env","path":"examples/elasticsearch-docker/non-ssl/.env","sha":"bc82d24eea229df2ba77f0fdf1f6e1c6f6a8e532"},{"name":"README.md","path":"examples/elasticsearch-docker/non-ssl/README.md","sha":"ed4bee52f5093a864164067b57d54f41668e55f5"},{"name":"docker-compose.yml","path":"examples/elasticsearch-docker/non-ssl/docker-compose.yml","sha":"e7547917f574679857f800b24f86bd573671931e"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/elasticsearch-docker/non-ssl/user-data/user-data.sh","sha":"81df178b3080a40a9e03ff7e94f1ef5db7d57f98"}]}]},{"name":"ssl","children":[{"name":".env","path":"examples/elasticsearch-docker/ssl/.env","sha":"bc82d24eea229df2ba77f0fdf1f6e1c6f6a8e532"},{"name":"README.md","path":"examples/elasticsearch-docker/ssl/README.md","sha":"ed4bee52f5093a864164067b57d54f41668e55f5"},{"name":"docker-compose.yml","path":"examples/elasticsearch-docker/ssl/docker-compose.yml","sha":"25e2b036c7e672a8ffc4b9559ab53f8d77441f99"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/elasticsearch-docker/ssl/user-data/user-data.sh","sha":"fc95c5a93069b251aca3ec2ae70a2be90231fcd9"}]}]}]},{"name":"elasticsearch-only-cluster","children":[{"name":"README.md","path":"examples/elasticsearch-only-cluster/README.md","sha":"e830acc97cf711f5087d03db0c61dc6ed82b1b8c"},{"name":"main.tf","path":"examples/elasticsearch-only-cluster/main.tf","sha":"16e304e12274a2c4cbc6c5b4c1d1758144ce259b"},{"name":"outputs.tf","path":"examples/elasticsearch-only-cluster/outputs.tf","sha":"382f2cda2c87d824b274f986f5107fd90477544a"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/elasticsearch-only-cluster/user-data/user-data.sh","sha":"1248e6d8692154f3d024ad95dab49a7a8646bc31"}]},{"name":"vars.tf","path":"examples/elasticsearch-only-cluster/vars.tf","sha":"6c3db8f2b043c53d04f7278a076230b4ce06308a"}]},{"name":"elk-amis","children":[{"name":"README.md","path":"examples/elk-amis/README.md","sha":"2d26c99b7a9a0d3a2124a771f0335eb5251f3b4c"},{"name":"all-in-one","children":[{"name":"README.md","path":"examples/elk-amis/all-in-one/README.md","sha":"42799bc50b4f41eb9e27ed79f1b5624b812b81b2"},{"name":"all-in-one.json","path":"examples/elk-amis/all-in-one/all-in-one.json","sha":"ecd2d44afe1d6040f90e304bc6cf042ec3668bc9"}]},{"name":"app-server","children":[{"name":"README.md","path":"examples/elk-amis/app-server/README.md","sha":"577966ff3d1aa65d69aa43d6b7e1e86968099335"},{"name":"app-server.json","path":"examples/elk-amis/app-server/app-server.json","sha":"6098834ef5ea26e93099f5131f4bd7c8ad77e82a"}]},{"name":"collectd","children":[{"name":"README.md","path":"examples/elk-amis/collectd/README.md","sha":"a9fa0d44e853022868368cab446e4aaeb24997ab"},{"name":"collectd-install-steps.sh","path":"examples/elk-amis/collectd/collectd-install-steps.sh","sha":"bfe1b037dcdee47751c588c4e431bfefec4661ee"},{"name":"collectd.json","path":"examples/elk-amis/collectd/collectd.json","sha":"236b08e2e60af5d34a9efabca7f49cf90772201f"},{"name":"config","children":[{"name":"collectd-ssl.conf","path":"examples/elk-amis/collectd/config/collectd-ssl.conf","sha":"0ccfc0f2aee65c0e17f5cd530c5edaccf247c3d8"},{"name":"collectd.conf","path":"examples/elk-amis/collectd/config/collectd.conf","sha":"74c1b46937039070cd95f64269eef63aabd72805"}]}]},{"name":"elastalert","children":[{"name":"README.md","path":"examples/elk-amis/elastalert/README.md","sha":"9dd33dfd29ddafa1040213466560ffcf8a66944d"},{"name":"aws","children":[{"name":"elastalert-config","children":[{"name":"config-ssl.yml","path":"examples/elk-amis/elastalert/aws/elastalert-config/config-ssl.yml","sha":"f26f8404e40736c5b0b17bc23e6a5de119d0c80f"},{"name":"config.yml","path":"examples/elk-amis/elastalert/aws/elastalert-config/config.yml","sha":"6054d91b8e86df9102fd5fd1fa2c545f7a981d31"}]}]},{"name":"docker","children":[{"name":"elastalert-config","children":[{"name":"config-ssl.yml","path":"examples/elk-amis/elastalert/docker/elastalert-config/config-ssl.yml","sha":"f26f8404e40736c5b0b17bc23e6a5de119d0c80f"},{"name":"config.yml","path":"examples/elk-amis/elastalert/docker/elastalert-config/config.yml","sha":"6054d91b8e86df9102fd5fd1fa2c545f7a981d31"}]}]},{"name":"elastalert-install-steps.sh","path":"examples/elk-amis/elastalert/elastalert-install-steps.sh","sha":"7da5b70623c85a0d61762a2ac5a2d91afd58d969"},{"name":"elastalert-rules","children":[{"name":"example_change.yml","path":"examples/elk-amis/elastalert/elastalert-rules/example_change.yml","sha":"4020886483786ff9a8847159b5750014c5d2b0fb"}]},{"name":"elastalert.json","path":"examples/elk-amis/elastalert/elastalert.json","sha":"762483ee0d35fdc4179cea051779f1076991fee5"}]},{"name":"elasticsearch","children":[{"name":"README.md","path":"examples/elk-amis/elasticsearch/README.md","sha":"f77a938e72424815565ffe16b9ae7787cb430cc7"},{"name":"aws","children":[{"name":"config","children":[{"name":"elasticsearch-ssl.yml","path":"examples/elk-amis/elasticsearch/aws/config/elasticsearch-ssl.yml","sha":"e6b0e5d3efd07f195aa18ae567789548a0eebc68"},{"name":"elasticsearch.yml","path":"examples/elk-amis/elasticsearch/aws/config/elasticsearch.yml","sha":"4918d1d6ec52dafa7741bda8212240d75b162f37"},{"name":"jvm.options","path":"examples/elk-amis/elasticsearch/aws/config/jvm.options","sha":"28dcf519ee339aba5dd15059c1acc4b97846d6ce"}]}]},{"name":"docker","children":[{"name":"config","children":[{"name":"elasticsearch-ssl.yml","path":"examples/elk-amis/elasticsearch/docker/config/elasticsearch-ssl.yml","sha":"705af52bc1e914df93faa919de8a23b973184938"},{"name":"elasticsearch.yml","path":"examples/elk-amis/elasticsearch/docker/config/elasticsearch.yml","sha":"2ac38b9c875dd6352b7a3d7d0fc2c02662d0e9cb"},{"name":"jvm.options","path":"examples/elk-amis/elasticsearch/docker/config/jvm.options","sha":"28dcf519ee339aba5dd15059c1acc4b97846d6ce"}]}]},{"name":"elasticsearch-install-steps.sh","path":"examples/elk-amis/elasticsearch/elasticsearch-install-steps.sh","sha":"a0de2d98730e3b1d0e3c5060dc80d925cb4043d3"},{"name":"elasticsearch.json","path":"examples/elk-amis/elasticsearch/elasticsearch.json","sha":"c8bfafebb715d5e1305f7c758274ce63de9f73ec"},{"name":"plugins","children":[{"name":"readonlyrest-1.22.1_es6.8.12.zip","path":"examples/elk-amis/elasticsearch/plugins/readonlyrest-1.22.1_es6.8.12.zip","sha":"93bdf77044e9ac16fb6eb8e091a1700419425ea9"}]},{"name":"readonlyrest-config","children":[{"name":"readonlyrest.yml","path":"examples/elk-amis/elasticsearch/readonlyrest-config/readonlyrest.yml","sha":"e056fad877791d455e337ff867845c26f514874a"}]}]},{"name":"filebeat","children":[{"name":"README.md","path":"examples/elk-amis/filebeat/README.md","sha":"ce76c3cbb426dc4d5e82f04a85eb0efd043fe671"},{"name":"config","children":[{"name":"filebeat-ssl.yml","path":"examples/elk-amis/filebeat/config/filebeat-ssl.yml","sha":"24b163e671b4cc49af08726867d8869f105a8ecb"},{"name":"filebeat.yml","path":"examples/elk-amis/filebeat/config/filebeat.yml","sha":"ae4641ee77c89e389728836fca3cac1428214fe2"}]},{"name":"filebeat-install-steps.sh","path":"examples/elk-amis/filebeat/filebeat-install-steps.sh","sha":"e7374df095835dddd4d89555d5529c7c3fdff175"},{"name":"filebeat.json","path":"examples/elk-amis/filebeat/filebeat.json","sha":"50b118a2452670dc1291f8e302a7045cba0c958a"}]},{"name":"kibana","children":[{"name":"README.md","path":"examples/elk-amis/kibana/README.md","sha":"b8af363345ba445cb9fbc2b04ec0e7d6b0b08149"},{"name":"config","children":[{"name":"kibana-ssl.yml","path":"examples/elk-amis/kibana/config/kibana-ssl.yml","sha":"7bcab515eb6cba2026899fb711541ebdecc97e3d"},{"name":"kibana.yml","path":"examples/elk-amis/kibana/config/kibana.yml","sha":"852b146720a4ab3fc1380beacce6f77851017b51"}]},{"name":"kibana-install-steps.sh","path":"examples/elk-amis/kibana/kibana-install-steps.sh","sha":"3be2e4579c93962ad8da899728ac3b496a5cd78e"},{"name":"kibana.json","path":"examples/elk-amis/kibana/kibana.json","sha":"6293086359621b813fc384d580598de122f29db0"}]},{"name":"logstash","children":[{"name":"README.md","path":"examples/elk-amis/logstash/README.md","sha":"b1035b8b3d34f28000e329171ba65275b1fac258"},{"name":"config","children":[{"name":"jvm.options","path":"examples/elk-amis/logstash/config/jvm.options","sha":"6017dd218501163070f950077ce6a1e9cff8f3a4"},{"name":"logstash-ssl.yml","path":"examples/elk-amis/logstash/config/logstash-ssl.yml","sha":"46e72285a2074ceba2313f726db5eb0db6dfa414"},{"name":"logstash.yml","path":"examples/elk-amis/logstash/config/logstash.yml","sha":"46e72285a2074ceba2313f726db5eb0db6dfa414"},{"name":"pipeline-ssl.conf","path":"examples/elk-amis/logstash/config/pipeline-ssl.conf","sha":"723780826865edbc2274837265a9059f9c7c1397"},{"name":"pipeline.conf","path":"examples/elk-amis/logstash/config/pipeline.conf","sha":"4a68c41fc5e86106915cb1c91ec39bd4c0cd2b55"}]},{"name":"logstash-install-steps.sh","path":"examples/elk-amis/logstash/logstash-install-steps.sh","sha":"e44ef50df6ac8989bfae91012cf04bbe386121b2"},{"name":"logstash.json","path":"examples/elk-amis/logstash/logstash.json","sha":"5ebdff0ca36a3bbe30043c3ef9b88959d1cdf3a8"}]}]},{"name":"elk-multi-cluster","children":[{"name":"README.md","path":"examples/elk-multi-cluster/README.md","sha":"8fd9aa87b62930d731636f72f6c0d22f900cb087"},{"name":"main.tf","path":"examples/elk-multi-cluster/main.tf","sha":"41e03942f41795d52e001ad11c5ea974abbf0dea"},{"name":"outputs.tf","path":"examples/elk-multi-cluster/outputs.tf","sha":"8bc4cde32815078921bc75375d963b533f6c50db"},{"name":"user-data","children":[{"name":"app-server","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/app-server/user-data.sh","sha":"e3ab01683b870b0bd96ad620d4b25860a7d7a833"}]},{"name":"elastalert","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/elastalert/user-data.sh","sha":"48181b3b07a3aec5e5b808b4101a9ef447eed37b"}]},{"name":"elasticsearch","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/elasticsearch/user-data.sh","sha":"7c7f36d79d716ff7e7618455097513aa936919bf"}]},{"name":"kibana","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/kibana/user-data.sh","sha":"b7f5e2b2d8c7091f1aa3e036a723913d9feca37b"}]},{"name":"logstash","children":[{"name":"user-data.sh","path":"examples/elk-multi-cluster/user-data/logstash/user-data.sh","sha":"9e1af5d687be9c44149560d2a022b62b88743b44"}]}]},{"name":"vars.tf","path":"examples/elk-multi-cluster/vars.tf","sha":"43f00586a4b0fb616a2904f6bb11112ab334f36e"}]},{"name":"elk-single-cluster","children":[{"name":"README.md","path":"examples/elk-single-cluster/README.md","sha":"33320b782aca0aea4d298599e5d3d706c93a12cf"},{"name":"main.tf","path":"examples/elk-single-cluster/main.tf","sha":"1c58be21a641bc4c0bdb683fcc5afd37258a8649"},{"name":"outputs.tf","path":"examples/elk-single-cluster/outputs.tf","sha":"c12a0b236e6fcbf5881cb60276d62a432dadb87a"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/elk-single-cluster/user-data/user-data.sh","sha":"446a992b93c4a3a53a86377546b0cc520bc941d3"}]},{"name":"vars.tf","path":"examples/elk-single-cluster/vars.tf","sha":"a67bfbaabbca1a1f25865570d9cc1a70d369485b"}]}]},{"name":"modules","children":[{"name":"auto-discovery","children":[{"name":"README.md","path":"modules/auto-discovery/README.md","sha":"f4de11c428f8fefd962220e7fbc4b976b4f838fe","toggled":true},{"name":"bin","children":[{"name":"auto-discovery","path":"modules/auto-discovery/bin/auto-discovery","sha":"fa1f2368e4ff71c550151ed9bd67107515d04873"}]},{"name":"install.sh","path":"modules/auto-discovery/install.sh","sha":"99a2f2fb2cbfaa31e752cbcc6495e5b755f034f2"}],"toggled":true},{"name":"beats-iam-policies","children":[{"name":"main.tf","path":"modules/beats-iam-policies/main.tf","sha":"e9a8c04747f61ad5aa049421312e955773f656e6"},{"name":"vars.tf","path":"modules/beats-iam-policies/vars.tf","sha":"52873644e20b4cafdcc0f33dc11db99f1fa9b586"}]},{"name":"elastalert-iam-policies","children":[{"name":"README.md","path":"modules/elastalert-iam-policies/README.md","sha":"012578675e314c2985c4c4c2ef7e58bd8eb37956"},{"name":"main.tf","path":"modules/elastalert-iam-policies/main.tf","sha":"e4f7ef96c2a388c60e6a6fb315f8cdac3256e280"},{"name":"variables.tf","path":"modules/elastalert-iam-policies/variables.tf","sha":"bc55d045eba3b8df1b08f78a4657cae25ee0ea73"}]},{"name":"elastalert-security-group-rules","children":[{"name":"README.md","path":"modules/elastalert-security-group-rules/README.md","sha":"050c72419243011fbcbdd5359764a2f9a22d0aaa"},{"name":"main.tf","path":"modules/elastalert-security-group-rules/main.tf","sha":"dae2fbeebd5a32a9dcd12194e7a433ea9762f52d"},{"name":"variables.tf","path":"modules/elastalert-security-group-rules/variables.tf","sha":"0d87d8de7864335c546406e1047a680f14e6ec73"}]},{"name":"elastalert","children":[{"name":"README.md","path":"modules/elastalert/README.md","sha":"ebc03e5af2cd6e03520a8e9ab126b34c9782acc6"},{"name":"main.tf","path":"modules/elastalert/main.tf","sha":"241f707360fec4cf509098fffe13f7f2bede372d"},{"name":"outputs.tf","path":"modules/elastalert/outputs.tf","sha":"aa5817acc77a75b75f4ea4acc686332c5b865618"},{"name":"vars.tf","path":"modules/elastalert/vars.tf","sha":"bd053dc8923e14a406bae848f75c92e4f0e8a822"}]},{"name":"elasticsearch-cluster-backup","children":[{"name":"README.md","path":"modules/elasticsearch-cluster-backup/README.md","sha":"2c3ca6928de82cfbdb8b96a4eaf5a2b86a0d39c7"},{"name":"backup","children":[{"name":"index.js","path":"modules/elasticsearch-cluster-backup/backup/index.js","sha":"6d3ff6a2a58d70dccd55399ba8bf0e0270140279"}]},{"name":"main.tf","path":"modules/elasticsearch-cluster-backup/main.tf","sha":"f6ed9528ec7b0f77513c638ad6410c118474353b"},{"name":"notification","children":[{"name":"index.js","path":"modules/elasticsearch-cluster-backup/notification/index.js","sha":"8e92e6b387974c4f0c6f007b8e3555d64e984202"}]},{"name":"outputs.tf","path":"modules/elasticsearch-cluster-backup/outputs.tf","sha":"ce7c4ee206984e74253ad6bb0bf25cd4a3965053"},{"name":"vars.tf","path":"modules/elasticsearch-cluster-backup/vars.tf","sha":"93c34c8c22ecfa0c9498e450df4886996b4bcfa1"}]},{"name":"elasticsearch-cluster-restore","children":[{"name":"README.md","path":"modules/elasticsearch-cluster-restore/README.md","sha":"754fa372fff69c4726499744ff41a83fe5c40446"},{"name":"main.tf","path":"modules/elasticsearch-cluster-restore/main.tf","sha":"f960c5c7e8298f62557bee5afef162721e7029f0"},{"name":"notification","children":[{"name":"index.js","path":"modules/elasticsearch-cluster-restore/notification/index.js","sha":"4e441662f92edf384c0c92a96ea5e1a7e1b0b24d"}]},{"name":"outputs.tf","path":"modules/elasticsearch-cluster-restore/outputs.tf","sha":"92454e1cf45fcc57f60c09eb8b31064efb460726"},{"name":"restore","children":[{"name":"index.js","path":"modules/elasticsearch-cluster-restore/restore/index.js","sha":"7d84ea278f01b64310fbc33d18c86a2a73b60573"}]},{"name":"vars.tf","path":"modules/elasticsearch-cluster-restore/vars.tf","sha":"b1c5309ac8d80d86b7ad12912fbf003d7177028f"}]},{"name":"elasticsearch-cluster","children":[{"name":"README.md","path":"modules/elasticsearch-cluster/README.md","sha":"96381fd14e010c475343d0eee79641c13ffe3e4e"},{"name":"main.tf","path":"modules/elasticsearch-cluster/main.tf","sha":"13d41726d7cf5dc768c0414390d922bdd9da450f"},{"name":"outputs.tf","path":"modules/elasticsearch-cluster/outputs.tf","sha":"0231e24415e13f62f29d8abefe0398ddb6d0fa04"},{"name":"variables.tf","path":"modules/elasticsearch-cluster/variables.tf","sha":"8b680c1329382792bee8415ff36bd1314675a4c9"}]},{"name":"elasticsearch-iam-policies","children":[{"name":"README.md","path":"modules/elasticsearch-iam-policies/README.md","sha":"efe4837043dbce44ce15e2b9959d2cfa11983485"},{"name":"main.tf","path":"modules/elasticsearch-iam-policies/main.tf","sha":"55bf1f01c743d84523b3d24ab9e41787a1e28d39"},{"name":"variables.tf","path":"modules/elasticsearch-iam-policies/variables.tf","sha":"c612a588acea6049cfdb930209854e253b549729"}]},{"name":"elasticsearch-security-group-rules","children":[{"name":"README.md","path":"modules/elasticsearch-security-group-rules/README.md","sha":"3292b4b51529305c57cccd4687c64bfa2a2dd6ec"},{"name":"main.tf","path":"modules/elasticsearch-security-group-rules/main.tf","sha":"cdb770582d24b7a9fc7502dd395c6f98a54acbac"},{"name":"outputs.tf","path":"modules/elasticsearch-security-group-rules/outputs.tf","sha":"e69de29bb2d1d6434b8b29ae775ad8c2e48c5391"},{"name":"variables.tf","path":"modules/elasticsearch-security-group-rules/variables.tf","sha":"7f055aa865ca315861e7a69f2d7741ab7883fddf"}]},{"name":"install-collectd","children":[{"name":"README.md","path":"modules/install-collectd/README.md","sha":"ed7f82bbf5f6a3cd5042b8cf78a8b69292f6a929"},{"name":"install.sh","path":"modules/install-collectd/install.sh","sha":"4090c1e8b94dd1a4d30ce68da03b73c281e53a3a"}]},{"name":"install-elastalert","children":[{"name":"README.md","path":"modules/install-elastalert/README.md","sha":"ab3850634329401b2734c5ed46064f93afbb989e"},{"name":"install.sh","path":"modules/install-elastalert/install.sh","sha":"2428d52ac73bd555044c5848546aac8b564c3a5f"}]},{"name":"install-elasticsearch","children":[{"name":"README.md","path":"modules/install-elasticsearch/README.md","sha":"ca54645f1ea89c80bb6953a2e5d70abec68a5346"},{"name":"install.sh","path":"modules/install-elasticsearch/install.sh","sha":"89c6d78790ed08a49af5821e04a459ec4c6b89d6"}]},{"name":"install-filebeat","children":[{"name":"README.md","path":"modules/install-filebeat/README.md","sha":"e8aea474de93cfb6421e038bdfd0459bf9936055"},{"name":"install.sh","path":"modules/install-filebeat/install.sh","sha":"b7537d09a916f93ded0214c5386d897e2f38bf56"}]},{"name":"install-kibana","children":[{"name":"README.md","path":"modules/install-kibana/README.md","sha":"9517d9c1b2f73823a00f8c1b190bb90e5b90312e"},{"name":"install.sh","path":"modules/install-kibana/install.sh","sha":"35eeef5d8aa75b4a1dd54d2913acf6688429c780"}]},{"name":"install-logstash","children":[{"name":"README.md","path":"modules/install-logstash/README.md","sha":"e1a8bde78e9bae25b852d7e9e44d0c6f372c3294"},{"name":"install.sh","path":"modules/install-logstash/install.sh","sha":"a689e67980aa28d7d2f04f693ccb59138ccfe3e2"}]},{"name":"kibana-cluster","children":[{"name":"README.md","path":"modules/kibana-cluster/README.md","sha":"87b31b9825999d66d6d334b48dc73de56ba647f7"},{"name":"main.tf","path":"modules/kibana-cluster/main.tf","sha":"7eb4359660f7951c4551e44babd9649568f2be31"},{"name":"outputs.tf","path":"modules/kibana-cluster/outputs.tf","sha":"f32e707371481d34b2b84d4ef7f205afaa04f2d4"},{"name":"variables.tf","path":"modules/kibana-cluster/variables.tf","sha":"58d03200bff5a18c52347d42af2a6ac1a6452802"}]},{"name":"kibana-security-group-rules","children":[{"name":"README.md","path":"modules/kibana-security-group-rules/README.md","sha":"f8d904f11c6a2238c5be9100c7e0255de28e5e2d"},{"name":"main.tf","path":"modules/kibana-security-group-rules/main.tf","sha":"f890ae74fae825d5db87f5011ba6638af874fda2"},{"name":"outputs.tf","path":"modules/kibana-security-group-rules/outputs.tf","sha":"e69de29bb2d1d6434b8b29ae775ad8c2e48c5391"},{"name":"variables.tf","path":"modules/kibana-security-group-rules/variables.tf","sha":"b89d5c7f5f1d1b0e9b5e1d239a1dec9a4ea34486"}]},{"name":"load-balancer-alb-target-group","children":[{"name":"README.md","path":"modules/load-balancer-alb-target-group/README.md","sha":"0a95fccf4f76af1149a6ab10d2854156e28f03f3"},{"name":"main.tf","path":"modules/load-balancer-alb-target-group/main.tf","sha":"e96c609cc002f2a9de68147aeb5ea60b6d1eb108"},{"name":"outputs.tf","path":"modules/load-balancer-alb-target-group/outputs.tf","sha":"cdf904445cf5099ccea5cc8820d198a1ffba4283"},{"name":"variables.tf","path":"modules/load-balancer-alb-target-group/variables.tf","sha":"4a262b0abde2721f9e3a5edeb6a98046009b9df3"}]},{"name":"logstash-cluster","children":[{"name":"README.md","path":"modules/logstash-cluster/README.md","sha":"78e4b47484d9f68ad16ffe06fb1af15991282cfe"},{"name":"main.tf","path":"modules/logstash-cluster/main.tf","sha":"55219e71d6f49ec878fafebad547e2573e4449be"},{"name":"outputs.tf","path":"modules/logstash-cluster/outputs.tf","sha":"218a4f0042d52ea1673de0c4f78857a4831143a6"},{"name":"vars.tf","path":"modules/logstash-cluster/vars.tf","sha":"bff728633e0f853502c3a15b4f9311b5b9d2d41f"}]},{"name":"logstash-iam-policies","children":[{"name":"README.md","path":"modules/logstash-iam-policies/README.md","sha":"3e354879298dbe3456cf3bae79594ee815487ac4"},{"name":"main.tf","path":"modules/logstash-iam-policies/main.tf","sha":"0ce69e25d9c2f964dc35f41e10813af86f2a5a7c"},{"name":"vars.tf","path":"modules/logstash-iam-policies/vars.tf","sha":"865c74386084c6eff1f46cb103aa847a6ad0da68"}]},{"name":"logstash-security-group-rules","children":[{"name":"README.md","path":"modules/logstash-security-group-rules/README.md","sha":"7ca6878f6b4415297d21c96ca48a4d8aa0314095"},{"name":"main.tf","path":"modules/logstash-security-group-rules/main.tf","sha":"dd54760fec89fa4a83ed37835d9901256af24bf5"},{"name":"outputs.tf","path":"modules/logstash-security-group-rules/outputs.tf","sha":"3641c561d0e23ef02fa83319ddafa8d1f9f813d2"},{"name":"vars.tf","path":"modules/logstash-security-group-rules/vars.tf","sha":"9e26c219ddad783caf509581fae84953eae60fe9"}]},{"name":"run-collectd","children":[{"name":"README.md","path":"modules/run-collectd/README.md","sha":"2242bcaca7f50717b3d6154c0576847d7b08c12c"},{"name":"bin","children":[{"name":"run-collectd","path":"modules/run-collectd/bin/run-collectd","sha":"7f7971d8b564a84a251d4c253c42f9025b5a1cf7"}]},{"name":"install.sh","path":"modules/run-collectd/install.sh","sha":"d2d8126f6fc5ba95a4e12242aa6ffe9bdc945ba6"}]},{"name":"run-elastalert","children":[{"name":"README.md","path":"modules/run-elastalert/README.md","sha":"e52cf808ab755b2d56a392ee8f525a0651963472"},{"name":"bin","children":[{"name":"run-elastalert","path":"modules/run-elastalert/bin/run-elastalert","sha":"cccab0f5cf41600c1b8ad48da5173db97965440f"}]},{"name":"install.sh","path":"modules/run-elastalert/install.sh","sha":"308f895a2366fc0cd8e6669229f6928a3d888f13"}]},{"name":"run-elasticsearch","children":[{"name":"README.md","path":"modules/run-elasticsearch/README.md","sha":"598ecd294a3177b5f9791793f75cec95559ad886"},{"name":"bin","children":[{"name":"run-elasticsearch","path":"modules/run-elasticsearch/bin/run-elasticsearch","sha":"e8e21b99ca07e03ce807ad537fbd9c3858912713"}]},{"name":"install.sh","path":"modules/run-elasticsearch/install.sh","sha":"c66e6da3d46195ac94d68eed18ed301ca7e0eb51"}]},{"name":"run-filebeat","children":[{"name":"README.md","path":"modules/run-filebeat/README.md","sha":"7b20727ebd231d78a39f30eeb2bad79619447f0a"},{"name":"bin","children":[{"name":"run-filebeat","path":"modules/run-filebeat/bin/run-filebeat","sha":"2e323cd6b7550db8470085911fa372c4edad8ae6"}]},{"name":"install.sh","path":"modules/run-filebeat/install.sh","sha":"cd890712803c3961ada86acf1e13b4c299402fb6"}]},{"name":"run-kibana","children":[{"name":"README.md","path":"modules/run-kibana/README.md","sha":"f6aaf8c3bd3901d753551ab4c28c0772d73ed143"},{"name":"bin","children":[{"name":"run-kibana","path":"modules/run-kibana/bin/run-kibana","sha":"0f60bca1be5eebc5594de9e8eab162aa979d4a17"}]},{"name":"install.sh","path":"modules/run-kibana/install.sh","sha":"e23b447832fce06ad32ce654f94a182a521e2e38"}]},{"name":"run-logstash","children":[{"name":"README.md","path":"modules/run-logstash/README.md","sha":"e1b6d3752f9f2f917c4bb17283caabd163f4d589"},{"name":"bin","children":[{"name":"run-logstash","path":"modules/run-logstash/bin/run-logstash","sha":"60dc8d906d8181e5a94c3ad18b5b27e53d513cf1"}]},{"name":"install.sh","path":"modules/run-logstash/install.sh","sha":"0a30f2adc2d0765b5581274263e12ce2e688b623"}]}],"toggled":true},{"name":"terraform-cloud-enterprise-private-module-registry-placeholder.tf","path":"terraform-cloud-enterprise-private-module-registry-placeholder.tf","sha":"ae586c0fe830819580e1009d41a9074f16e65bed"},{"name":"test","children":[{"name":"README.md","path":"test/README.md","sha":"819e790d15767c851bfc438f0a1413ed2bde7d3b"},{"name":"elasticsearch_aws_simple_test.go","path":"test/elasticsearch_aws_simple_test.go","sha":"cccda6ac30573bc7fea6772f56f68a96ab1b0581"},{"name":"elasticsearch_docker_test.go","path":"test/elasticsearch_docker_test.go","sha":"9aab286b753db9127f7425acea12bcc93e5abdd1"},{"name":"elk_aio_test.go","path":"test/elk_aio_test.go","sha":"b1ca766cbee9d5b69e1028ef48e971f4d333232b"},{"name":"elk_end_to_end_test.go","path":"test/elk_end_to_end_test.go","sha":"fb74457130157625a2d817e0d54d6be55aa6a3c2"},{"name":"go.mod","path":"test/go.mod","sha":"b6d375ede5dfd840285c97e9e8c1e95a6b8cfb52"},{"name":"go.sum","path":"test/go.sum","sha":"c64c51d793c7c3ec7848f7998ddb9494a979ea59"},{"name":"notes.md","path":"test/notes.md","sha":"664915d4b948ccd194ffda80854f47c191904e3d"},{"name":"test_helpers.go","path":"test/test_helpers.go","sha":"0ba1e792e7a26419f43c55971e4910cbc242a009"},{"name":"test_helpers_keystore.go","path":"test/test_helpers_keystore.go","sha":"e3b1ed8963088a2bcbdc2863287e60d19d23cd47"},{"name":"tls_helpers.go","path":"test/tls_helpers.go","sha":"828f04c10b9ca6f4b6f8f82ddcd498bdb1c7dbb4"}]}]},"detailsContent":"<h1 class=\"preview__body--title\" id=\"auto-discovery-script\">Auto Discovery Script</h1><div class=\"preview__body--border\"></div><p>This folder contains a script that allows auto discovery of nodes in a cluster. This script has been tested on the following operating systems:</p>\n<ul>\n<li>Ubuntu 16.04</li>\n<li>Ubuntu 18.04</li>\n<li>CentOS 7</li>\n<li>Amazon Linux 2</li>\n</ul>\n<h2 class=\"preview__body--subtitle\" id=\"usage\">Usage</h2>\n<p>Run <code>auto-discovery --help</code> to see all available arguments</p>\n<pre><span class=\"hljs-attribute\">Usage</span>: auto-discovery [options]\n\n<span class=\"routeros\">This<span class=\"hljs-built_in\"> script </span>can <span class=\"hljs-builtin-name\">run</span> on a scheduled basis <span class=\"hljs-keyword\">to</span> (a) automatically discover the IPs of other EC2 Instances (b) write these IPs <span class=\"hljs-keyword\">to</span> a<span class=\"hljs-built_in\"> config </span>file, (c) restart the<span class=\"hljs-built_in\"> service </span>using that<span class=\"hljs-built_in\"> config </span>file.\n\nOptions:\n\n --tag\t\t\t\t\t The tag key <span class=\"hljs-keyword\">and</span> value on the instances.\n --port\t\t\t\t The<span class=\"hljs-built_in\"> port </span>on which the<span class=\"hljs-built_in\"> server </span>nodes listen on <span class=\"hljs-keyword\">for</span> connections.\n --aws-region\t\t\t The AWS region where the instances are deployed.\n --use-public-ips <span class=\"hljs-builtin-name\">Set</span> <span class=\"hljs-keyword\">to</span> <span class=\"hljs-literal\">true</span> <span class=\"hljs-keyword\">to</span> use public IPs <span class=\"hljs-keyword\">and</span> <span class=\"hljs-literal\">false</span> <span class=\"hljs-keyword\">to</span> use private IPs. Default: <span class=\"hljs-literal\">true</span>\n --config-file\t\t\t The path <span class=\"hljs-keyword\">to</span> a configuration file.\n --service-name\t\t The name of the systemd<span class=\"hljs-built_in\"> service </span><span class=\"hljs-keyword\">to</span> restart after updating the specified configuration file.\n --pattern\t\t\t\t Regex pattern used <span class=\"hljs-keyword\">to</span> identify where <span class=\"hljs-keyword\">to</span> update <span class=\"hljs-keyword\">in</span> the specified configuration file.\n --schedule\t\t\t\t A cron expression specifying the schedule this<span class=\"hljs-built_in\"> script </span>will <span class=\"hljs-builtin-name\">run</span> with otherwise<span class=\"hljs-built_in\"> script </span>will <span class=\"hljs-builtin-name\">run</span> only once.\n --user\t\t\t \tThe<span class=\"hljs-built_in\"> user </span><span class=\"hljs-keyword\">to</span> schedule the cron job <span class=\"hljs-keyword\">for</span>. Defaults <span class=\"hljs-keyword\">to</span> the<span class=\"hljs-built_in\"> user </span>that executes the<span class=\"hljs-built_in\"> script\n</span> --help\t\t\t\t Show this help text <span class=\"hljs-keyword\">and</span> exit.\n\nExample:\n\n auto-discovery --tag <span class=\"hljs-string\">'Name=instance-0'</span> --port 5044 --aws-region <span class=\"hljs-string\">'us-east-1'</span> --config-file ./filebeat.yml --pattern <span class=\"hljs-string\">'hosts:.*'</span> --service-name <span class=\"hljs-string\">'filebeat'</span>\n</span></pre>\n<h3 class=\"preview__body--subtitle\" id=\"how-it-works\">How it works</h3>\n<p>The <code>auto-discovery</code> script works in the following steps:</p>\n<ol>\n<li>Makes a request to AWS to retrive all <code>pending</code> or <code>running</code> instances with a specific tag set to a the specified value.</li>\n<li>Retrieves the public or private IP addresses of the returned instances depending on the value of the <code>--use-public-ips</code> flag.</li>\n<li>Replaces a portion of the specified configuration file with a comma separated list of IP and port pairs (e.g <code>"52.72.28.213:8065", "54.210.77.143:8065"</code>) using the specified regex pattern as a marker</li>\n<li>Restarts the specified systemd service, which presumbly uses the just updated configuration file and needs to reload it.</li>\n<li>If <code>--schedule</code> is set, this script schedules itself with cron to be run on the specified schedule.</li>\n</ol>\n<p><strong>LIMITATION:</strong> Unfortunately, the approach described above can't work if the application for which we are doing discovery\nis configured to do full SSL certificate verification (ie: hostname verification). Since our approach looks up randomly\nassigned IP addresses, there's no way we could generate an SSL certificate that encapsulates all of the possible IP addresses\nthat may be found. For this reason, we have to disable SSL hostname verification when using this module.</p>\n<h3 class=\"preview__body--subtitle\" id=\"future-improvements\">Future Improvements</h3>\n<p>To work around the limitation described above, we will be making changes that will update local <code>/etc/hosts</code> or run a\nlocal instance of <a href=\"http://www.thekelleys.org.uk/dnsmasq/doc.html\" class=\"preview__body--description--blue\" target=\"_blank\">Dnsmasq</a> in order to make seamless updates to the DNS\ninstead of updating an application's config file and then needing to restart that application.</p>\n","repoName":"package-elk","repoRef":"v0.9.0","serviceDescriptor":{"serviceName":"Elasticsearch (self-hosted)","serviceRepoName":"package-elk","serviceRepoOrg":"gruntwork-io","cloudProviders":["aws"],"description":"Deploy a self-hosted Elasticsearch cluster. Supports automatic bootstrap, zero-downtime rolling deployment, auto healing, backup, and recovery.","imageUrl":"elk.png","licenseType":"subscriber","technologies":["Terraform","Bash","JavaScript"],"compliance":[],"tags":[""]},"serviceCategoryName":"NoSQL","fileName":"README.md","filePath":"/modules/auto-discovery","title":"Repo Browser: Elasticsearch (self-hosted)","description":"Browse the repos in the Gruntwork Infrastructure as Code Library."}
.circleci
install.sh
