This is a Gruntwork Script Module that installs
bash scripts used to boot a MongoDB Instance. It includes:
attach-ebs-volume: Search for an available EBS Volume to attach. If none is found, terminate
the EC2 Instance on which it runs.
attach-eni: Search for an available Elastic Network Interface to attach. If none is found, terminate
the EC2 Instance on which it runs.
See the comments at the top of each script for additional information.
These scripts are meant to execute in the User Data
of an EC2 Instance so that they are executed at boot time. Both scripts assume that a "pool" of available EBS Volumes and
Elastic Network Interfaces has already been created, out of band from this script. When an EC2 Instance boots up, it will
attach an Elastic Network Interface, and then attach, mount, and format an EBS Volume.
Why attach an EBS Volume?
...
Why attach an Elastic Network Interface (ENI)?
In a typical Auto Scaling Group, each EC2 Instance that boots has a unique private IP address. As long as there's some
kind of Service Discovery applications can reach these new EC2 Instances.
For example, an ELB or ALB associated with the Auto Scaling Group will automatically route to the new EC2 Instances.
But when we're dealing with clustered services, where each node is aware of the other and there is usually one node that
serves as the "primary" or "master" node, a load balancer usually isn't the right solution for routing a connectiong to
the right EC2 Instance. If the master node changes, we want to know about this instantly, whereas an ELB takes at
minimum 10 seconds to do such a switch. In addition, this adds a small amount of network latency and added cost to our
setup.
The alternative to using a load balancer is to rely on the MongoDB client to choose the right endpoint automatically.
In the case of MongoDB, clients specify a connection string
in the following format:
# MongoDB Client connection string format
mongodb://[username:password@]host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]]`
Since there are potentially many hosts, most Mongo clients that can't connect to host1 will automatically retry host2.
One elegant way to identify the hosts is to assign each of them a DNS name so that we could have a connection
string like:
# Example MongoDB Client connection string
mongodb://mongo1.gruntwork.io:27017,mongo2.gruntwork.io:27017,mongo3.gruntwork.io:27017`
But this suffers from a subtle problem: DNS addresses are cached at multiple layers. For example, the OS usually caches
the IP address to which a DNS record resolves, as does the MongoDB client. Even the DNS resolver used to resolve a DNS
query will cache responses!
In most cases, this isn't a problem. But if an EC2 Instance becomes unhealthy and reboots, one of our DNS records needs
to be updated. Each of our caches now hopefully honors the DNS Records Time-To-Live (TTL) property which can be as low
as 5 seconds but is likely higher. In addition, some DNS caches do not honor TTL's at all. Until the DNS record points to
the new EC2 Instance, the newly booted EC2 Instance is unreachable by MongoDB clients and therefore effectively out of
commission.
Admittedly, DNS caching issues causing a failure to connect to your database are unlikely. For example, if an EC2
Instance serving as the "primary" node is replaced, we would expect the MongoDB cluster to quickly promote a new primary
and MongoDB clients to quickly try a different host if the first one is unreachable. But issuesdohappen.
By using a second Elastic Network Interface (ENI),
we can assign a static private IP address to each EC2 Instance. When an EC2 Instance terminates, it automatically "detaches"
its ENI. When a new EC2 Instance boots up, it can assume the previously used ENI and thus be accessible at the same
private IP address. Now, our DNS records never need to change, and DNS caching issues are no longer a concern.
The attach-eni script makes this entire process automatic.
Questions? Ask away.
We're here to talk about our services, answer any questions, give advice, or just to chat.
{"treedata":{"name":"root","toggled":true,"children":[{"name":".circleci","children":[{"name":"config.yml","path":".circleci/config.yml","sha":"6d4d3bdc7870375ecc99f4f0ba5ffc2bce11f23c"}]},{"name":".gitignore","path":".gitignore","sha":"32845458602b36a63610885e236aecaf5d0cfb98"},{"name":"CODEOWNERS","path":"CODEOWNERS","sha":"4be01a6334d39aa5bf6abe6baae701f5e2a8c5ac"},{"name":"LICENSE.txt","path":"LICENSE.txt","sha":"f4e3d9bd4717a044ed31ad847a300eee74371a78"},{"name":"README.md","path":"README.md","sha":"7c75d82dc78347dc782c2de14fa7c0e2c45f0f56"},{"name":"_docs","children":[{"name":"Backup and Restore.md","path":"_docs/Backup and Restore.md","sha":"22cda43eb88ac930a2f5f9aad38ec405ba864428"},{"name":"Best Practices.md","path":"_docs/Best Practices.md","sha":"bb4f29e5974e2c17ce9b2487e02abba7e6ff5d56"},{"name":"Cluster Architecture and Design.md","path":"_docs/Cluster Architecture and Design.md","sha":"687d97b994a419b1e37b3a468a98bf38c9f01697"},{"name":"Gotchas.md","path":"_docs/Gotchas.md","sha":"39f7a49b558326c95c2a14dfa1aa19323c65bbf6"},{"name":"Setup a Cluster.md","path":"_docs/Setup a Cluster.md","sha":"98b2f5258bbacfd1d606cd66f44d32766fbdeb7e"}]},{"name":"examples","children":[{"name":"empty-cluster-test","children":[{"name":"README.md","path":"examples/empty-cluster-test/README.md","sha":"3405f2b25f0d80d80e69ee3c94ab84e704a18d1d"},{"name":"main.tf","path":"examples/empty-cluster-test/main.tf","sha":"3039577ff72ea219c1c9b672067fba809e11fe16"},{"name":"vars.tf","path":"examples/empty-cluster-test/vars.tf","sha":"1fe45398b3a22cbb1e0c432b91b519ee03990bd9"}]},{"name":"mongodb-ami","children":[{"name":"README.md","path":"examples/mongodb-ami/README.md","sha":"fb155ad2ef28d8b59c30a0a56f08dafc5934aa4d"},{"name":"files","children":[{"name":"mongodb.gpg","path":"examples/mongodb-ami/files/mongodb.gpg","sha":"1d681f820678e1e035ccf9a540b87e72afece678"},{"name":"mongodb.repo","path":"examples/mongodb-ami/files/mongodb.repo","sha":"29c8fb6b615b32cbf6fc41815e89d9f421983dcc"},{"name":"tls","children":[{"name":"README.md","path":"examples/mongodb-ami/files/tls/README.md","sha":"fc34588224aec934e1c55f2092715a2afc00ebd8"},{"name":"sample.ca.cert.pem","path":"examples/mongodb-ami/files/tls/sample.ca.cert.pem","sha":"8648019774aaea8bc490cea7a5a4ebc01fe90630"},{"name":"sample.key.pem","path":"examples/mongodb-ami/files/tls/sample.key.pem","sha":"f8c5c5f3b4eb969d87e5497adbdb7e8234f7a40d"}]}]},{"name":"mongodb.json","path":"examples/mongodb-ami/mongodb.json","sha":"55c5edce1a73677878be98a55de5d8b201ade070"}]},{"name":"mongodb-replica-set-cluster-with-backup","children":[{"name":"README.md","path":"examples/mongodb-replica-set-cluster-with-backup/README.md","sha":"eb3df05ad870b167a0c13a2df0c2062818d962b8"},{"name":"main.tf","path":"examples/mongodb-replica-set-cluster-with-backup/main.tf","sha":"7b0bb3f4dfe89e22e96ec9d75f0c935549973025"},{"name":"outputs.tf","path":"examples/mongodb-replica-set-cluster-with-backup/outputs.tf","sha":"b806f530f37ff6d44e47035efeef49a3f51a07e4"},{"name":"user-data","children":[{"name":"user-data-mongod-backup-node.sh","path":"examples/mongodb-replica-set-cluster-with-backup/user-data/user-data-mongod-backup-node.sh","sha":"48a4be7cf137c2e40426552be8e399c38f2f31b2"},{"name":"user-data-mongod.sh","path":"examples/mongodb-replica-set-cluster-with-backup/user-data/user-data-mongod.sh","sha":"37bcc2313f27b26e69bcad4b13af7f286faa07bf"}]},{"name":"vars.tf","path":"examples/mongodb-replica-set-cluster-with-backup/vars.tf","sha":"6f533b540c1d28d8e31428c2d30e9221dc49bb43"}]},{"name":"mongodb-sharded-cluster","children":[{"name":"README.md","path":"examples/mongodb-sharded-cluster/README.md","sha":"d0d6328765ea1bce6e914426b8927ecdc4d58979"},{"name":"main.tf","path":"examples/mongodb-sharded-cluster/main.tf","sha":"4a9bf3697b5808369693513c16761a83e9cacf1c"},{"name":"outputs.tf","path":"examples/mongodb-sharded-cluster/outputs.tf","sha":"1724634b121aa5a970e2b86e195e1d72bb291d03"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/mongodb-sharded-cluster/user-data/user-data.sh","sha":"29c2c548baf4341c220c6b42ba5199305f435e25"}]},{"name":"vars.tf","path":"examples/mongodb-sharded-cluster/vars.tf","sha":"bb4859b505f40b9f7d32865a231f674315758c46"}]},{"name":"mongodb-single-instance","children":[{"name":"README.md","path":"examples/mongodb-single-instance/README.md","sha":"161586396ea73da2ebdd5429f299d21cd58f87c6"},{"name":"main.tf","path":"examples/mongodb-single-instance/main.tf","sha":"fe795bb5665c84354435cd471a3a86d8d284ecac"},{"name":"outputs.tf","path":"examples/mongodb-single-instance/outputs.tf","sha":"0c204ee0280ee5e266a59f1fe45b3106e10c5094"},{"name":"user-data","children":[{"name":"user-data.sh","path":"examples/mongodb-single-instance/user-data/user-data.sh","sha":"f51c5b0d3c5180e885b0fba70030ac1453bf8a34"}]},{"name":"vars.tf","path":"examples/mongodb-single-instance/vars.tf","sha":"8b8411c1f9eae13a196645affa55c1a534a3f2f0"}]}]},{"name":"modules","children":[{"name":"backup-mongodb","children":[{"name":"README.md","path":"modules/backup-mongodb/README.md","sha":"807822fff41ac3767cae75f0917e1065d155d455"},{"name":"bin","children":[{"name":"backup-mongodb","path":"modules/backup-mongodb/bin/backup-mongodb","sha":"c86aa92f158f11dcf09d7b7f059123c295214bd8"}]},{"name":"install.sh","path":"modules/backup-mongodb/install.sh","sha":"5b5d58d45a914f252975c637627fb697b8e1f946"}]},{"name":"init-mongodb","children":[{"name":"README.md","path":"modules/init-mongodb/README.md","sha":"a1f9550b7609ca79bab4e7557bbbbe4ca2c10ece"},{"name":"bin","children":[{"name":"add-hidden-node","path":"modules/init-mongodb/bin/add-hidden-node","sha":"7d65492eb998974a8e12290d4e44c0d0d440c141"},{"name":"assert-cluster-ready","path":"modules/init-mongodb/bin/assert-cluster-ready","sha":"0c33101dea86592b7d71b406b1f9170117f04d2a"},{"name":"create-user","path":"modules/init-mongodb/bin/create-user","sha":"b6867a6a1e97a287ed2a59bfde2681ade7ae174a"},{"name":"init-replica-set","path":"modules/init-mongodb/bin/init-replica-set","sha":"3acca9a2a5ddff97fa62ffbabae53bd0961d72f5"}]},{"name":"install.sh","path":"modules/init-mongodb/install.sh","sha":"539b5d9b6eccf942fff14849a25c0e9ac9718983"}]},{"name":"install-mongodb","children":[{"name":"README.md","path":"modules/install-mongodb/README.md","sha":"992358d6e8e49530ed0d350398ff2fc236207e00"},{"name":"bin","children":[{"name":"install-mongodb","path":"modules/install-mongodb/bin/install-mongodb","sha":"61a241368bf4145341704a14bab0f1cc89b1c26f"}]},{"name":"files","children":[{"name":"supervisor-initd-script.sh","path":"modules/install-mongodb/files/supervisor-initd-script.sh","sha":"171b91613e98ab2bd10282025caff1707918c95a"},{"name":"supervisord.conf","path":"modules/install-mongodb/files/supervisord.conf","sha":"156eecf98d3bc37d390de5035c873ca0aa40aced"}]},{"name":"install.sh","path":"modules/install-mongodb/install.sh","sha":"159444b40aa70c8de0134405039d4e03616e901c"}]},{"name":"mongodb-backup","children":[{"name":"README.md","path":"modules/mongodb-backup/README.md","sha":"e438df72516e29e8b45c0454fc5c7a9d4943e41e"},{"name":"main.tf","path":"modules/mongodb-backup/main.tf","sha":"045729151906b532103939eb6b3992c09864ae28"},{"name":"vars.tf","path":"modules/mongodb-backup/vars.tf","sha":"8b492c54c4a5819446835fe4f8d8fec38ce04440"}]},{"name":"mongodb-cluster","children":[{"name":"README.md","path":"modules/mongodb-cluster/README.md","sha":"a8dfd88cde9517d4d680075aaefc684906bfa58e"},{"name":"main.tf","path":"modules/mongodb-cluster/main.tf","sha":"d4974c8ecae4d8c177c45f9f7adc39497f007d02"},{"name":"outputs.tf","path":"modules/mongodb-cluster/outputs.tf","sha":"8fecb95b45a8e65e0024e31032dbc341a639f569"},{"name":"user-data","children":[{"name":"default-user-data.sh","path":"modules/mongodb-cluster/user-data/default-user-data.sh","sha":"9640839c6b8f8d7458f2cf159ec0450555dcc56a"}]},{"name":"vars.tf","path":"modules/mongodb-cluster/vars.tf","sha":"7c2463df5cf3781e91499e66d6c56ce858b3cb7b"}]},{"name":"run-mongodb","children":[{"name":"README.md","path":"modules/run-mongodb/README.md","sha":"2840a280a4fae0852c62e3d4bd1057c00c1cefed"},{"name":"bin","children":[{"name":"run-mongodb","path":"modules/run-mongodb/bin/run-mongodb","sha":"b8507ab91cc09d821c9de91d674a51750897d46b"}]},{"name":"install.sh","path":"modules/run-mongodb/install.sh","sha":"30319b01091300027f9e0892640704bff75063d2"}]},{"name":"setup-ec2-instance","children":[{"name":"README.md","path":"modules/setup-ec2-instance/README.md","sha":"d0fa2a03eac9f78fb674e30da8feaa2fcb413dda","toggled":true},{"name":"bin","children":[{"name":"attach-ebs-volume","path":"modules/setup-ec2-instance/bin/attach-ebs-volume","sha":"08b9a5eb1a290437c304cda253dada75f9c7dba4"},{"name":"attach-eni","path":"modules/setup-ec2-instance/bin/attach-eni","sha":"39b9477a942425d94815895f66742549a4ab82b6"}]},{"name":"install.sh","path":"modules/setup-ec2-instance/install.sh","sha":"e9c4424f8c889e091c2be1136aab93e0d2c196bf"}],"toggled":true}],"toggled":true},{"name":"test","children":[{"name":"Gopkg.lock","path":"test/Gopkg.lock","sha":"c0a3e932f876c4d6fc932ac331743e3d721110dc"},{"name":"Gopkg.toml","path":"test/Gopkg.toml","sha":"043708efb2b6452474a0b7e453c194af93151cca"},{"name":"empty_cluster_example_test.go","path":"test/empty_cluster_example_test.go","sha":"a5de28a0543cef5a0aabb12d92742ed7c1e82e9c"},{"name":"helpers_aws.go","path":"test/helpers_aws.go","sha":"7d964cdaf0902f227e60255d2f25cf91d5ce3451"},{"name":"helpers_file.go","path":"test/helpers_file.go","sha":"139b02762a330ef5714823d7d89938039ac9d858"},{"name":"helpers_mongo.go","path":"test/helpers_mongo.go","sha":"447d5a6b1622b42eadc26c1201abefac3b730c46"},{"name":"helpers_tls.go","path":"test/helpers_tls.go","sha":"7ff58dd6b69f674b4f873a47a88a1a3c28e85a39"},{"name":"mongo_replica_set_with_backup_test.go","path":"test/mongo_replica_set_with_backup_test.go","sha":"8a5bbec42d2a2a8b29608c52060e78c6435f6803"},{"name":"mongo_sharded_cluster_test.go","path":"test/mongo_sharded_cluster_test.go","sha":"ac8986d1e71880f4b9016f9580193ba2449d52dd"},{"name":"mongodb_single_instance_test.go","path":"test/mongodb_single_instance_test.go","sha":"ef569feab3d2bc7c4492372e876307d5a2056e7c"},{"name":"packer_options.go","path":"test/packer_options.go","sha":"ad839abcac59b0cffd1f14d4c5708f200b5eb6d4"},{"name":"terratest_options.go","path":"test/terratest_options.go","sha":"bc7387b84ef2bb349b02d2b77abab271f0be89b3"}]}]},"detailsContent":"<h1 class=\"preview__body--title\" id=\"setup-ec-2-instance-scripts\">Setup EC2 Instance Scripts</h1><div class=\"preview__body--border\"></div><p>This is a <a href=\"/repos/gruntwork-installer#gruntwork-modules\" class=\"preview__body--description--blue\">Gruntwork Script Module</a> that installs\nbash scripts used to boot a MongoDB Instance. It includes:</p>\n<ul>\n<li>\n<p><a href=\"/repos/v0.4.1/package-mongodb/modules/setup-ec2-instance/bin/attach-ebs-volume\" class=\"preview__body--description--blue\">attach-ebs-volume</a>: Search for an available EBS Volume to attach. If none is found, terminate\nthe EC2 Instance on which it runs.</p>\n</li>\n<li>\n<p><a href=\"/repos/v0.4.1/package-mongodb/modules/setup-ec2-instance/bin/attach-eni\" class=\"preview__body--description--blue\">attach-eni</a>: Search for an available Elastic Network Interface to attach. If none is found, terminate\nthe EC2 Instance on which it runs.</p>\n</li>\n</ul>\n<p>See the comments at the top of each script for additional information.</p>\n<p>These scripts are meant to execute in the <a href=\"http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html\" class=\"preview__body--description--blue\" target=\"_blank\">User Data</a>\nof an EC2 Instance so that they are executed at boot time. Both scripts assume that a "pool" of available EBS Volumes and\nElastic Network Interfaces has already been created, out of band from this script. When an EC2 Instance boots up, it will\nattach an Elastic Network Interface, and then attach, mount, and format an EBS Volume.</p>\n<h2 class=\"preview__body--subtitle\" id=\"why-attach-an-ebs-volume\">Why attach an EBS Volume?</h2>\n<p>...</p>\n<h2 class=\"preview__body--subtitle\" id=\"why-attach-an-elastic-network-interface-eni\">Why attach an Elastic Network Interface (ENI)?</h2>\n<p>In a typical Auto Scaling Group, each EC2 Instance that boots has a unique private IP address. As long as there's some\nkind of <a href=\"https://en.wikipedia.org/wiki/Service_discovery\" class=\"preview__body--description--blue\" target=\"_blank\">Service Discovery</a> applications can reach these new EC2 Instances.\nFor example, an ELB or ALB associated with the Auto Scaling Group will automatically route to the new EC2 Instances.</p>\n<p>But when we're dealing with clustered services, where each node is aware of the other and there is usually one node that\nserves as the "primary" or "master" node, a load balancer usually isn't the right solution for routing a connectiong to\nthe right EC2 Instance. If the master node changes, we want to know about this instantly, whereas an ELB takes at\nminimum 10 seconds to do such a switch. In addition, this adds a small amount of network latency and added cost to our\nsetup.</p>\n<p>The alternative to using a load balancer is to rely on the MongoDB client to choose the right endpoint automatically.\nIn the case of MongoDB, clients specify a <a href=\"https://docs.mongodb.com/manual/reference/connection-string/\" class=\"preview__body--description--blue\" target=\"_blank\">connection string</a>\nin the following format:</p>\n<pre><span class=\"hljs-section\"># MongoDB Client connection string format</span>\nmongodb://[<span class=\"hljs-string\">username:password@</span>]host1[<span class=\"hljs-string\">:port1</span>][<span class=\"hljs-symbol\">,host2[:port2</span>],...[<span class=\"hljs-string\">,hostN[:portN</span>]]][<span class=\"hljs-string\">/[database</span>][<span class=\"hljs-symbol\">?options</span>]]`\n</pre>\n<p>Since there are potentially many hosts, most Mongo clients that can't connect to <code>host1</code> will automatically retry <code>host2</code>.\nOne elegant way to identify the hosts is to assign each of them a DNS name so that we could have a connection\nstring like:</p>\n<pre><span class=\"hljs-comment\"># Example MongoDB Client connection string</span>\nmongodb://mongo1.gruntwork.io:<span class=\"hljs-number\">27017</span>,mongo2.gruntwork.io:<span class=\"hljs-number\">27017</span>,mongo3.gruntwork.io:<span class=\"hljs-number\">27017</span>`\n</pre>\n<p>But this suffers from a subtle problem: DNS addresses are cached at multiple layers. For example, the OS usually caches\nthe IP address to which a DNS record resolves, as does the MongoDB client. Even the DNS resolver used to resolve a DNS\nquery will cache responses!</p>\n<p>In most cases, this isn't a problem. But if an EC2 Instance becomes unhealthy and reboots, one of our DNS records needs\nto be updated. Each of our caches now hopefully honors the DNS Records Time-To-Live (TTL) property which can be as low\nas 5 seconds but is likely higher. In addition, some DNS caches do not honor TTL's at all. Until the DNS record points to\nthe new EC2 Instance, the newly booted EC2 Instance is unreachable by MongoDB clients and therefore effectively out of\ncommission.</p>\n<p>Admittedly, DNS caching issues causing a failure to connect to your database are unlikely. For example, if an EC2\nInstance serving as the "primary" node is replaced, we would expect the MongoDB cluster to quickly promote a new primary\nand MongoDB clients to quickly try a different host if the first one is unreachable. But <a href=\"http://serverfault.com/questions/619319/mongodb-replica-failures-to-resolve-dns-when-primary-dns-is-stopped-while-second\" class=\"preview__body--description--blue\" target=\"_blank\">issues</a>\n<a href=\"https://jira.mongodb.org/browse/SERVER-4567\" class=\"preview__body--description--blue\" target=\"_blank\">do</a> <a href=\"https://news.ycombinator.com/item?id=8012675\" class=\"preview__body--description--blue\" target=\"_blank\">happen</a>.</p>\n<p>By using a second <a href=\"http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html\" class=\"preview__body--description--blue\" target=\"_blank\">Elastic Network Interface</a> (ENI),\nwe can assign a static private IP address to each EC2 Instance. When an EC2 Instance terminates, it automatically "detaches"\nits ENI. When a new EC2 Instance boots up, it can assume the previously used ENI and thus be accessible at the same\nprivate IP address. Now, our DNS records never need to change, and DNS caching issues are no longer a concern.</p>\n<p>The <code>attach-eni</code> script makes this entire process automatic.</p>\n","repoName":"package-mongodb","repoRef":"v0.4.1","serviceDescriptor":{"serviceName":"MongoDB","serviceRepoName":"package-mongodb","serviceRepoOrg":"gruntwork-io","cloudProviders":["aws"],"description":"Deploy a MongoDB cluster. Supports replica sets, sharding, automated bootstrapping, backup, recovery, and OS optimizations.","imageUrl":"mongodb.png","licenseType":"subscriber","technologies":["Terraform","Bash"],"compliance":[],"tags":[""]},"serviceCategoryName":"NoSQL","fileName":"README.md","filePath":"/modules/setup-ec2-instance","title":"Repo Browser: MongoDB","description":"Browse the repos in the Gruntwork Infrastructure as Code Library."}
.circleci
install.sh
