This module creates an ElastiCache cluster that runs Redis.
The redis cluster is managed by AWS and automatically detects and replaces failed nodes, streamlines software upgrades
and patches, enables easy scaling of the cluster both horizontally (add more nodes) and vertically (increase the power
of existing nodes), and simplifies backup and restore functionality.
About Amazon ElastiCache
What is Amazon ElastiCache?
Before Amazon ElastiCache existed, teams
would painstakingly configure the memcached or redis caching engines on their own. Setting up automatic failover, read
replicas, backups, and handling upgrades are all non-trivial and AWS recognized they could implement these features
according to best practices themselves, sparing customers the time and cost of doing it themselves. Behind the scenes,
ElastiCache runs on EC2 Instances located in subnets and protected by security groups you specify.
Structure of an ElastiCache Redis deployment
Nodes: The smallest unit of an ElastiCache Redis deployment is a node. It's basically the network-attached RAM on
which the cache engine (Redis in this case) runs
Shards: Also sometimes called "Node Group". A Shard is a replication-enabled collection of multiple nodes. Within
a Shard, one node is the primary read/write node while the rest are read-only replicas of the primary node.
Cluster: An ElastiCache cluster is a collection of one or more Shards. Somewhat confusingly, an ElastiCache
Cluster has a "cluster mode" property that allows a Cluster to distribute its data over multiple Shards.
Concept of Sharding & Replication in ElasticCache Redis
Redis implements replication in two ways:
Cluster Mode Disabled: With a single shard that contains all of the cluster's data in each node
Cluster Mode Enabled: With data partitioned across up to 500 shards
Each shard in a replication group has a single read/write primary node and up to 5 read-only replica nodes. You can
create a cluster with higher number of shards and lower number of replicas totaling up to 90 nodes per cluster. This
cluster configuration can range from 90 shards and 0 replicas to 15 shards and 5 replicas, which is the maximum number
of replicas allowed.
The node or shard limit can be increased to a maximum of 500 per cluster if the Redis engine version is 5.0.6 or higher.
For example, you can choose to configure a 500 node cluster that ranges between 83 shards (one primary and 5 replicas
per shard) and 500 shards (single primary and no replicas). Make sure there are enough available IP addresses to
accommodate the increase. Common pitfalls include the subnets in the subnet group have too small a CIDR range or the
subnets are shared and heavily used by other clusters.
You can find more information
here: https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/Replication.Redis.Groups.html
Different Modes of ElasticCache Redis deployment
There are different types of modes you can deploy ElasticCache Redis:
- enable_single_instance_mode = false <br/> - cluster_mode:num_node_groups = 1 <br/> <br/> Note: do not include .cluster.on suffix if you are setting parameter_group_name.
Single Instance
- no replication <br/> - no sharding
- enable_single_instance_mode = true
How to Enable Cluster Mode with Single Sharding
By default, if you set cluster_mode::num_node_groups = 1, you are deploying Redis with the cluster mode disabled. If
you want to enable cluster mode for redis with single sharding, you need to explicitly set the parameter group with
cluster-enabled=yes (e.g., default.redis5.0.cluster.on). You can find the list of default parameter group for
different redis version family group
here: https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/ParameterGroups.Redis.html
Choosing Cluster Mode vs. Single Instance
You can use var.enable_single_instance_mode=true to deploy a single node Redis instance. Refer
to examples/redis_single_instance as an example.
Here are some of the points you may consider while choosing which mode to run:
Scalability: Redis single instance is limited to a single server node, which can be scaled vertically, while Redis
cluster mode supports horizontal scaling by distributing data across multiple nodes, up to 1000 nodes. This allows for
increased capacity, availability, and fault tolerance.
High Availability: Redis single instance has limited high availability options, as it can only be replicated to a
slave node. In contrast, Redis cluster mode provides more robust high availability features by automatically
distributing data across multiple nodes and ensuring data replication, even in the case of node failures.
Performance: Redis cluster mode provides improved performance, as it allows for distributing the data across
multiple nodes, providing more CPU and memory resources. Additionally, Redis cluster mode uses sharding to distribute
keys across multiple nodes, allowing for parallel data processing and faster data access.
Complexity: Redis cluster mode introduces more complexity than Redis single instance. Configuring and managing a
Redis cluster requires more effort and expertise, as it involves setting up and managing multiple nodes, and
configuring the sharding and replication.
How do you connect to the Redis Cluster?
When connecting to Redis (cluster mode disabled) from your app, direct all operations to the Primary Endpoint of
the Cluster. This way, in the event of a failover, your app will be resolving a DNS record that automatically gets
updated to the latest primary node.
When connecting to Redis (cluster mode enabled) from your app, direct all reads/writes to the **Configuration Endpoint
** of the Cluster. Ensure you have a client that supports Redis Cluster (redis 3.2). You can still read from
individual node enpoints.
In both "cluster mode enabled" and "cluster mode disabled" deployment models you can still direct reads to any of the
Read Endpoints of the nodes in the Cluster, however you now risk reading a slightly out-of-date copy of the data
in the event that you read from a node before the primary's latest data has synced to it.
(Optional) It is possible to link a user_group_id and provide a list of user_id's to add additional layers of
security for your cluster or replication group. Refer to
the ElasticCache RBAC Access
documentation for further information.
(Optional) It is also possible to update the default user id used by AWS when creating a redis cluster and replace
it with your own default user. This is actually recommended by AWS as a first step to securing your Redis cluster.
Refer to
the Role-Based Access Control (RBAC)
documentation on how to do this. The new user_id can be passed into the Redis clusters user group via
the default_user_id variable.
This module outputs Terraform output variables that
contain the address of the primary endpoint and read endpoints. You can programmatically extract these variables in your
Terraform templates and pass them to other resources (e.g. as environment variables in an EC2 Instance) You'll also see
the variables at the end of each terraform apply call or if you run terraform output.
How do you scale the Redis Cluster?
You can scale your ElastiCache Cluster either horizontally (by adding more nodes) or vertically (by using more powerful
nodes), but the method depends on whether Cluster Mode is enabled or disabled.
When Cluster Mode is Disabled
This mode is useful when you'd prefer to have only a single point for data to be written to the redis database. All data
is written to the primary node of the single shard which is now replicated to the replica nodes. The advantage of this
approach is that you're sure that all your data is present at a single point which could make migrations and backups a
lot easier, if there's a problem with the primary write node however, all write attempts will fail.
Vertical: You can increase the type of the read replica nodes using the instance_type parameter (
see here for valid values).
Horizontal: You can add up to 5 replica nodes to a Redis Cluster using the cluster_size parameter. There is
always a primary node where all writes take place, but you can reduce load on this primary node by offloading reads to
the non-primary nodes, which are known as Read Replicas.
When cluster mode is enabled, data you write is split among the primary nodes in multiple Shards, and each data stored
in those Shards are replicated among the read replicas. If one Shard becomes unavailable for whatever reason, other
shards are still available for writing.
To deploy a Redis "cluster mode enabled" cluster you must ensure that the enable_automatic_failover parameter is set
to true and the cluster_mode variable has a single map with num_shards and replicas_per_shard parameters.
E.g. the following Terraform code specifies a "cluster mode enabled" Cluster with 2 Shards and 1 Read Replica per Shard
Horizontal: A "cluster mode enabled" cluster can be scaled horizontally by adding more Shards - also called
Resharding, the amount of read replicas present in the extra Shards is the same as the number specified when the
cluster was originally created. Resharding can take anywhere from a few minutes to several hours.
Vertical: A "cluster mode enabled" cluster can be scaled vertically by changing the node types within each Shard.
This method is offline only and the cluster has to be backed up, a new cluster created with the required node types
and the backup restored to that new cluster. During this process your entire cluster will experience a downtime which
could last for any length of time depending on how much data the original cluster contained.
auto_scale_dimension: choose one from the supported scalable dimension mentioned below
auto_scale_trigger_metric: choose one from the supported trigger metric mentioned below
auto_scale_min_capacity: choose the desired min capacity
auto_scale_max_capacity: choose the desired max capacity (Note: depends on the configuration of the Redis
cluster. (e.g., 5 would be the maximum for
replica auto-scaling)
Prerequisites
ElastiCache for Redis Auto Scaling is limited to the following:
Auto Scaling in ElastiCache for Redis is not supported for clusters running in Global datastores, Outposts or Local
Zones.
AWS Auto Scaling for ElastiCache for Redis is not available in the following regions: China (Beijing), China (
Ningxia), AWS GovCloud (US-West) and AWS GovCloud (US-East).
Supported Predefined Metrics
Here are some predefined metrics available for Redis: ElastiCachePrimaryEngineCPUUtilization (shards
auto-scale), ElastiCacheReplicaEngineCPUUtilization (replica auto-scale). You can find more information here:
Both pages also explain how to use a custom metric and you can find the list of CloudWatch metrics for Redis from
here: https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/CacheMetrics.Redis.html
Supported Scalable Dimensions
ElastiCache for Redis supports the following types of automatic scaling dimensions:
elasticache:replication-group:NodeGroups - The number of node groups for an Amazon ElastiCache replication group.
elasticache:replication-group:Replicas - The number of replicas per node group for an Amazon ElastiCache replication
group.
Prior to October 2016, ElastiCache for Redis did not support partitioning
(aka sharding) in Redis and as a result used a bunch of confusing terminologies to explain setting up a Redis cluster.
Cluster: In ElastiCache, each Redis node is labeled its own "cluster", despite that there is only a single node in
the cluster. This is an artifact of how ElastiCache implements memcached clusters, which can support multiple nodes.
Replication Group: The Redis "read/write primary cluster" will asynchronously copy its data to all other Redis
clusters (individual nodes) in the same Replication Group. In a sense, the "real" redis cluster in ElastiCache is
actually the Replication Group.
Primary Cluster: In a Replication Group, the primary cluster is where all write traffic is directed. Initially,
you can direct all your reads here as well.
Read Replicas: In a Replication Group, data written to the primary cluster is asynchronously copied to each Read
Replica. These are useful both to reduce the read load on the primary cluster and to faciliate fast failover.
Multi-AZ: Using the Multi-AZ option means that if the primary cluster fails, ElastiCache will auto failover to a
Read Replica. Without this option, the primary cluster node will simply replace itself, which will take longer and may
result in additional data loss.
Common Gotcha's
In the event of a Redis failover, you will experience a small period of time during which writes are not accepted,
so make sure your app can handle this gracefully. In fact, consider simulating Redis failovers on a regular basis or
with automated testing to validate that your app can handle it correctly.
Test that your app does not cache the DNS value of the Primary Endpoint! Java, in particular, has undesirable defaults
around DNS caching in many cases. If your code does not honor the TTL property of the Primary Endpoint's DNS record,
then your app may fail to reach the new primary node in the event of a failure.
The only way to add more storage space to nodes in a Redis Cluster (cluster mode disabled) is to scale up to a larger
node type.
Choosing Redis Cluster Mode vs. Single Instance
You can use var.disable_cluster_mode=true to deploy a single node Redis instance. Refer
to examples/redis_single_instance as an example.
Here are some of the points you may consider while choosing which mode to run:
Scalability: Redis single instance is limited to a single server node, which can be scaled vertically, while Redis
cluster mode supports horizontal scaling by distributing data across multiple nodes, up to 1000 nodes. This allows for
increased capacity, availability, and fault tolerance.
High Availability: Redis single instance has limited high availability options, as it can only be replicated to a
slave node. In contrast, Redis cluster mode provides more robust high availability features by automatically
distributing data across multiple nodes and ensuring data replication, even in the case of node failures.
Performance: Redis cluster mode provides improved performance, as it allows for distributing the data across
multiple nodes, providing more CPU and memory resources. Additionally, Redis cluster mode uses sharding to distribute
keys across multiple nodes, allowing for parallel data processing and faster data access.
Complexity: Redis cluster mode introduces more complexity than Redis single instance. Configuring and managing a
Redis cluster requires more effort and expertise, as it involves setting up and managing multiple nodes, and
configuring the sharding and replication.
Questions? Ask away.
We're here to talk about our services, answer any questions, give advice, or just to chat.
{"treedata":{"name":"root","toggled":true,"children":[{"name":".circleci","children":[{"name":"config.yml","path":".circleci/config.yml","sha":"1bc85ec3a200e8d740f424d7e6a4f8c1e869b1cf"},{"name":"post-upgrade-test-results.sh","path":".circleci/post-upgrade-test-results.sh","sha":"a4867e8fbdc334b7a90259568ee41ea577fbe764"},{"name":"set-upgrade-test-vars.sh","path":".circleci/set-upgrade-test-vars.sh","sha":"0d289ca9172f07aa927182f5caa7bbee5ed34f13"}]},{"name":".github","children":[{"name":"ISSUE_TEMPLATE","children":[{"name":"bug_report.md","path":".github/ISSUE_TEMPLATE/bug_report.md","sha":"d2e87e27c601e423865ed660ec697082470ca60f"},{"name":"feature_request.md","path":".github/ISSUE_TEMPLATE/feature_request.md","sha":"023a33099be2336476930c96e17ff1ba5dc55348"}]},{"name":"pull_request_template.md","path":".github/pull_request_template.md","sha":"6b100e40e323b5b07f40ed30616277c51c9f4b9e"}]},{"name":".gitignore","path":".gitignore","sha":"c816ce0f0e5ac5aeb14bcc0cc0ad002381f8bc99"},{"name":".pre-commit-config.yaml","path":".pre-commit-config.yaml","sha":"23068872ca60d7f40ae10c05ea8e5915d04056dc"},{"name":"CODEOWNERS","path":"CODEOWNERS","sha":"3bc8a8c8ee6e939522f988c5407a5d62b2d4ac26"},{"name":"LICENSE.txt","path":"LICENSE.txt","sha":"f4e3d9bd4717a044ed31ad847a300eee74371a78"},{"name":"README.md","path":"README.md","sha":"1521cd21f08f906dab626526813093bbbc158f5a"},{"name":"examples","children":[{"name":"elasticache_bastion_host","children":[{"name":"README.md","path":"examples/elasticache_bastion_host/README.md","sha":"2f79a6de2c864125ebc059ac08d9ef053306ecfc"},{"name":"main.tf","path":"examples/elasticache_bastion_host/main.tf","sha":"095255c2472c43aa031474553ee43500d56b87ff"},{"name":"outputs.tf","path":"examples/elasticache_bastion_host/outputs.tf","sha":"ed63b2572f3813bd183dc5c48d21ce606ac6392e"},{"name":"redis-cli-install.sh","path":"examples/elasticache_bastion_host/redis-cli-install.sh","sha":"2ccd53ce730eb639413f26422fdac7a7eaeedffa"},{"name":"vars.tf","path":"examples/elasticache_bastion_host/vars.tf","sha":"8c5aba4d0fd661eee1fcd6c6bd5cf3943ff4cd73"}]},{"name":"memcached","children":[{"name":"README.md","path":"examples/memcached/README.md","sha":"25f3220196a9df56803eb2beb96d7ecca03f71ac"},{"name":"main.tf","path":"examples/memcached/main.tf","sha":"ac9357c9fab554ac9978b1fa78147c1f87e529c3"},{"name":"outputs.tf","path":"examples/memcached/outputs.tf","sha":"bc090567ef708bed42c4a64aaef937e8ae27b67a"},{"name":"vars.tf","path":"examples/memcached/vars.tf","sha":"eadad613f832bc0d23b1e45cafe241f94135d12d"}]},{"name":"redis","children":[{"name":"README.md","path":"examples/redis/README.md","sha":"4fa42b734382a224e1b296e0dc1c7120a193211d"},{"name":"main.tf","path":"examples/redis/main.tf","sha":"c484c6d8daef3e516806e7bf6a114ced0f96eec5"},{"name":"outputs.tf","path":"examples/redis/outputs.tf","sha":"c52d0992fa05f0687d5583cb27e0b34f04c19f71"},{"name":"vars.tf","path":"examples/redis/vars.tf","sha":"85fec2b3d209b6186e32d580c63520a6de763ebe"}]},{"name":"redis_copy_snapshot","children":[{"name":"README.md","path":"examples/redis_copy_snapshot/README.md","sha":"ff193cc8f6e0d2a796bc6d7c6ba97c147a27a238"},{"name":"main.tf","path":"examples/redis_copy_snapshot/main.tf","sha":"b1dbc4793bdcfbdfaa7638e5fbc58636ea1c2f92"},{"name":"outputs.tf","path":"examples/redis_copy_snapshot/outputs.tf","sha":"92cacac6cb2652ac0b6b7d2ae572a1a460e41719"},{"name":"vars.tf","path":"examples/redis_copy_snapshot/vars.tf","sha":"a8b755e0fdb353054e1af2b6b15d557b03a37ff1"}]},{"name":"redis_single_instance","children":[{"name":"README.md","path":"examples/redis_single_instance/README.md","sha":"4fa42b734382a224e1b296e0dc1c7120a193211d"},{"name":"main.tf","path":"examples/redis_single_instance/main.tf","sha":"9623e58defa91203405eda521b940f184dc8ec9a"},{"name":"outputs.tf","path":"examples/redis_single_instance/outputs.tf","sha":"061e0694468400960c507c63c96562d4bc7c2137"},{"name":"vars.tf","path":"examples/redis_single_instance/vars.tf","sha":"7e05023e51ac194e4ab5ae523925b1b24b080ddf"}]},{"name":"redis_with_auth_token","children":[{"name":"README.md","path":"examples/redis_with_auth_token/README.md","sha":"d33e94e9a9cada95aaf64a4f001732a60ed273a6"},{"name":"main.tf","path":"examples/redis_with_auth_token/main.tf","sha":"493e62bce940089b92ed8b41ee009d4b4a28565e"},{"name":"outputs.tf","path":"examples/redis_with_auth_token/outputs.tf","sha":"c52d0992fa05f0687d5583cb27e0b34f04c19f71"},{"name":"vars.tf","path":"examples/redis_with_auth_token/vars.tf","sha":"cf892d3f32b73a6b2411087aa5441e56422c94d0"}]},{"name":"redis_with_auto_scaling","children":[{"name":"README.md","path":"examples/redis_with_auto_scaling/README.md","sha":"f9bcdbde9516b877426e7f6c2c1bcde177483bab"},{"name":"main.tf","path":"examples/redis_with_auto_scaling/main.tf","sha":"240956fc62422756a0399bee4564d6170d6701a2"},{"name":"outputs.tf","path":"examples/redis_with_auto_scaling/outputs.tf","sha":"571b69d7463ab7f9032ca2b5d70dbc33eb0b646e"},{"name":"vars.tf","path":"examples/redis_with_auto_scaling/vars.tf","sha":"ad07361697e2ebac23995ba1cb26689a44a00de7"}]},{"name":"redis_with_log_and_no_auto_upgrade","children":[{"name":"README.md","path":"examples/redis_with_log_and_no_auto_upgrade/README.md","sha":"95c7f5bdca5d8570071a501633e476d603ab01d9"},{"name":"main.tf","path":"examples/redis_with_log_and_no_auto_upgrade/main.tf","sha":"9085bd97c1d4383fbe641ed1dd6edaf4481ea665"},{"name":"outputs.tf","path":"examples/redis_with_log_and_no_auto_upgrade/outputs.tf","sha":"c52d0992fa05f0687d5583cb27e0b34f04c19f71"},{"name":"vars.tf","path":"examples/redis_with_log_and_no_auto_upgrade/vars.tf","sha":"74bc3a5b683effa50a2d8eb0088ade255feb44e2"}]},{"name":"redis_with_sharding","children":[{"name":"README.md","path":"examples/redis_with_sharding/README.md","sha":"15db6dcbe096133c0be14055a6ac7aa91d3092fa"},{"name":"main.tf","path":"examples/redis_with_sharding/main.tf","sha":"f852c4f006c0f50e965c883fec4a069efbd2e46d"},{"name":"outputs.tf","path":"examples/redis_with_sharding/outputs.tf","sha":"571b69d7463ab7f9032ca2b5d70dbc33eb0b646e"},{"name":"vars.tf","path":"examples/redis_with_sharding/vars.tf","sha":"38e5a2fdfe9234cb70c0d22e0f3781d4c3b6199d"}]},{"name":"redis_with_user_group_id","children":[{"name":"README.md","path":"examples/redis_with_user_group_id/README.md","sha":"43d59aaa30561d2e108a79632340da19c96e1989"},{"name":"main.tf","path":"examples/redis_with_user_group_id/main.tf","sha":"d6aa7b705aee734928c15083c2f397f6345646c8"},{"name":"outputs.tf","path":"examples/redis_with_user_group_id/outputs.tf","sha":"c52d0992fa05f0687d5583cb27e0b34f04c19f71"},{"name":"vars.tf","path":"examples/redis_with_user_group_id/vars.tf","sha":"715db05764e8a984b350748e3b812cf8f4bc5f02"}]}]},{"name":"modules","children":[{"name":"elastic-cache","children":[{"name":"README.md","path":"modules/elastic-cache/README.md","sha":"ddfa792dec8880597838c035637508b2107ca8ac"},{"name":"main.tf","path":"modules/elastic-cache/main.tf","sha":"a1297594e2bee37a6f23f3aae4d48d28c1803bc9"},{"name":"outputs.tf","path":"modules/elastic-cache/outputs.tf","sha":"d06f7754cd290a0564acf247c75616b7d9fb376d"},{"name":"vars.tf","path":"modules/elastic-cache/vars.tf","sha":"8f0c06e6f100d4e387c7c90583f2f2ca3ebc2629"}]},{"name":"memcached","children":[{"name":"README.md","path":"modules/memcached/README.md","sha":"c5d3f4dc902f551347f460c0442ae8a0991ede04"},{"name":"main.tf","path":"modules/memcached/main.tf","sha":"e0a6447fd8af83994325a0724c26ba032fe5a63b"},{"name":"outputs.tf","path":"modules/memcached/outputs.tf","sha":"bcfc047b98700eab8e90005207c56c76443ac889"},{"name":"vars.tf","path":"modules/memcached/vars.tf","sha":"effe6daff465c590227ad3542d874a3d22131276"}]},{"name":"redis","children":[{"name":"README.md","path":"modules/redis/README.md","sha":"8f8098a443520cae8b9921d67345ccfa2307e302","toggled":true},{"name":"main.tf","path":"modules/redis/main.tf","sha":"220afc1bb1d4ea2dcb846fc464f36d8ef54a6197"},{"name":"outputs.tf","path":"modules/redis/outputs.tf","sha":"991f74facba639eb60059df50581162311359472"},{"name":"vars.tf","path":"modules/redis/vars.tf","sha":"819e28143d70e427182ecd039f56f390749f41cd"}],"toggled":true},{"name":"redis_copy_snapshot","children":[{"name":"README.md","path":"modules/redis_copy_snapshot/README.md","sha":"1f5cbae6c9cd26fd5621cf42c6a4fbe6789bda0c"},{"name":"main.tf","path":"modules/redis_copy_snapshot/main.tf","sha":"c2896827bb4f41dea2accf429bebb9a7119a2838"},{"name":"outputs.tf","path":"modules/redis_copy_snapshot/outputs.tf","sha":"32f0c0ccd6524757bf1b4c02cb48d33090d6ddd6"},{"name":"redis-copy-snapshot","children":[{"name":"index.py","path":"modules/redis_copy_snapshot/redis-copy-snapshot/index.py","sha":"f6a211b527de2c0a1ad56681fb78216e531b1eb2"}]},{"name":"vars.tf","path":"modules/redis_copy_snapshot/vars.tf","sha":"bc38a3fd66ca21e6bbc1a1e7d638eff5efcaf7b6"}]}],"toggled":true},{"name":"terraform-cloud-enterprise-private-module-registry-placeholder.tf","path":"terraform-cloud-enterprise-private-module-registry-placeholder.tf","sha":"ae586c0fe830819580e1009d41a9074f16e65bed"},{"name":"test","children":[{"name":"README.md","path":"test/README.md","sha":"09098790510ea89ab30850b01fa25e69f2730457"},{"name":"example_memcached_connection.go","path":"test/example_memcached_connection.go","sha":"7e1e51e64346c2646ef01aafc376ea69239dd781"},{"name":"example_memcached_test.go","path":"test/example_memcached_test.go","sha":"1ba2b92d1d37aeb9b55e5c28d02ac359caed5e9e"},{"name":"example_redis_connection.go","path":"test/example_redis_connection.go","sha":"28442755bdcfe0eea2919554eb4c2b8b789a87a0"},{"name":"example_redis_test.go","path":"test/example_redis_test.go","sha":"1f491679c3c7610774cddf2499152f3719564a7b"},{"name":"example_util_functions.go","path":"test/example_util_functions.go","sha":"9deb8ef8a4a9621618fdf8e9512727e69b4f62e9"},{"name":"go.mod","path":"test/go.mod","sha":"da9ce050b81081dfbd91ea01bbe9050b75256bad"},{"name":"go.sum","path":"test/go.sum","sha":"7ef564d3e21815e6fabf3796378b8f14fa19a25e"},{"name":"upgrades","children":[{"name":"upgrade_test.go","path":"test/upgrades/upgrade_test.go","sha":"4a2a0d73b48f76e58ee242f071ecf5fcba4e883d"}]},{"name":"validation","children":[{"name":"validate_all_modules_and_examples_test.go","path":"test/validation/validate_all_modules_and_examples_test.go","sha":"33d73c385b64c4fc870033e99427e683c31dc45a"}]}]}]},"detailsContent":"<h1 class=\"preview__body--title\" id=\"redis-module\">Redis Module</h1><div class=\"preview__body--border\"></div><p>This module creates an ElastiCache cluster that runs <a href=\"http://redis.io/\" class=\"preview__body--description--blue\" target=\"_blank\">Redis</a>.</p>\n<p>The redis cluster is managed by AWS and automatically detects and replaces failed nodes, streamlines software upgrades\nand patches, enables easy scaling of the cluster both horizontally (add more nodes) and vertically (increase the power\nof existing nodes), and simplifies backup and restore functionality.</p>\n<h2 class=\"preview__body--subtitle\" id=\"about-amazon-elasti-cache\">About Amazon ElastiCache</h2>\n<h3 class=\"preview__body--subtitle\" id=\"what-is-amazon-elasti-cache\">What is Amazon ElastiCache?</h3>\n<p>Before <a href=\"http://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/WhatIs.html\" class=\"preview__body--description--blue\" target=\"_blank\">Amazon ElastiCache</a> existed, teams\nwould painstakingly configure the memcached or redis caching engines on their own. Setting up automatic failover, read\nreplicas, backups, and handling upgrades are all non-trivial and AWS recognized they could implement these features\naccording to best practices themselves, sparing customers the time and cost of doing it themselves. Behind the scenes,\nElastiCache runs on EC2 Instances located in subnets and protected by security groups you specify.</p>\n<h3 class=\"preview__body--subtitle\" id=\"structure-of-an-elasti-cache-redis-deployment\">Structure of an ElastiCache Redis deployment</h3>\n<ul>\n<li><strong>Nodes:</strong> The smallest unit of an ElastiCache Redis deployment is a node. It's basically the network-attached RAM on\nwhich the cache engine (Redis in this case) runs</li>\n<li><strong>Shards:</strong> Also sometimes called "Node Group". A Shard is a replication-enabled collection of multiple nodes. Within\na Shard, one node is the primary read/write node while the rest are read-only replicas of the primary node.</li>\n<li><strong>Cluster:</strong> An ElastiCache cluster is a collection of one or more Shards. Somewhat confusingly, an ElastiCache\nCluster has a "cluster mode" property that allows a Cluster to distribute its data over multiple Shards.</li>\n</ul>\n<h3 class=\"preview__body--subtitle\" id=\"concept-of-sharding-replication-in-elastic-cache-redis\">Concept of Sharding & Replication in ElasticCache Redis</h3>\n<p>Redis implements replication in two ways:</p>\n<ul>\n<li><strong>Cluster Mode Disabled</strong>: With a single shard that contains all of the cluster's data in each node</li>\n<li><strong>Cluster Mode Enabled</strong>: With data partitioned across up to 500 shards</li>\n</ul>\n<p>Each shard in a replication group has a single read/write primary node and up to 5 read-only replica nodes. You can\ncreate a cluster with higher number of shards and lower number of replicas totaling up to 90 nodes per cluster. This\ncluster configuration can range from 90 shards and 0 replicas to 15 shards and 5 replicas, which is the maximum number\nof replicas allowed.</p>\n<p>The node or shard limit can be increased to a maximum of 500 per cluster if the Redis engine version is 5.0.6 or higher.\nFor example, you can choose to configure a 500 node cluster that ranges between 83 shards (one primary and 5 replicas\nper shard) and 500 shards (single primary and no replicas). Make sure there are enough available IP addresses to\naccommodate the increase. Common pitfalls include the subnets in the subnet group have too small a CIDR range or the\nsubnets are shared and heavily used by other clusters.</p>\n<p>You can find more information\nhere: https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/Replication.Redis.Groups.html</p>\n<h3 class=\"preview__body--subtitle\" id=\"different-modes-of-elastic-cache-redis-deployment\">Different Modes of ElasticCache Redis deployment</h3>\n<p>There are different types of modes you can deploy ElasticCache Redis:</p>\n<table>\n<thead>\n<tr>\n<th>Mode</th>\n<th>Description</th>\n<th>Configuration</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><strong>Cluster Enabled</strong></td>\n<td>- supports sharding <br/> - supports replication</td>\n<td>- <code>enable_single_instance_mode = false</code> <br/> - <code>cluster_mode:num_node_groups > 1</code></td>\n</tr>\n<tr>\n<td><strong>Cluster Disabled</strong></td>\n<td>- no replication <br/> - supports sharding</td>\n<td>- <code>enable_single_instance_mode = false</code> <br/> - <code>cluster_mode:num_node_groups = 1</code> <br/> <br/> <strong>Note</strong>: do not include <code>.cluster.on</code> suffix if you are setting <code>parameter_group_name</code>.</td>\n</tr>\n<tr>\n<td><strong>Single Instance</strong></td>\n<td>- no replication <br/> - no sharding</td>\n<td>- <code>enable_single_instance_mode = true</code></td>\n</tr>\n</tbody>\n</table>\n<h4 id=\"how-to-enable-cluster-mode-with-single-sharding\">How to Enable Cluster Mode with Single Sharding</h4>\n<p>By default, if you set <code>cluster_mode::num_node_groups = 1</code>, you are deploying Redis with the cluster mode disabled. If\nyou want to enable cluster mode for redis with single sharding, you need to explicitly set the parameter group with\n<code>cluster-enabled=yes</code> (e.g., default.redis5.0.cluster.on). You can find the list of default parameter group for\ndifferent redis version family group\nhere: https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/ParameterGroups.Redis.html</p>\n<h4 id=\"choosing-cluster-mode-vs-single-instance\">Choosing Cluster Mode vs. Single Instance</h4>\n<p>You can use <code>var.enable_single_instance_mode=true</code> to deploy a single node Redis instance. Refer\nto <a href=\"/repos/v0.22.2/module-cache/examples/redis_single_instance\" class=\"preview__body--description--blue\">examples/redis_single_instance</a> as an example.</p>\n<p>Here are some of the points you may consider while choosing which mode to run:</p>\n<ul>\n<li><strong>Scalability</strong>: Redis single instance is limited to a single server node, which can be scaled vertically, while Redis\ncluster mode supports horizontal scaling by distributing data across multiple nodes, up to 1000 nodes. This allows for\nincreased capacity, availability, and fault tolerance.</li>\n<li><strong>High Availability</strong>: Redis single instance has limited high availability options, as it can only be replicated to a\nslave node. In contrast, Redis cluster mode provides more robust high availability features by automatically\ndistributing data across multiple nodes and ensuring data replication, even in the case of node failures.</li>\n<li><strong>Performance</strong>: Redis cluster mode provides improved performance, as it allows for distributing the data across\nmultiple nodes, providing more CPU and memory resources. Additionally, Redis cluster mode uses sharding to distribute\nkeys across multiple nodes, allowing for parallel data processing and faster data access.</li>\n<li><strong>Complexity</strong>: Redis cluster mode introduces more complexity than Redis single instance. Configuring and managing a\nRedis cluster requires more effort and expertise, as it involves setting up and managing multiple nodes, and\nconfiguring the sharding and replication.</li>\n</ul>\n<h3 class=\"preview__body--subtitle\" id=\"how-do-you-connect-to-the-redis-cluster\">How do you connect to the Redis Cluster?</h3>\n<ul>\n<li>\n<p>When connecting to Redis (cluster mode disabled) from your app, direct all operations to the <strong>Primary Endpoint</strong> of\nthe <strong>Cluster.</strong> This way, in the event of a failover, your app will be resolving a DNS record that automatically gets\nupdated to the latest primary node.</p>\n</li>\n<li>\n<p>When connecting to Redis (cluster mode enabled) from your app, direct all reads/writes to the **Configuration Endpoint\n** of the <strong>Cluster.</strong> Ensure you have a client that supports Redis Cluster (redis 3.2). You can still read from\nindividual node enpoints.</p>\n</li>\n<li>\n<p>In both "cluster mode enabled" and "cluster mode disabled" deployment models you can still direct reads to any of the\n<strong>Read Endpoints</strong> of the nodes in the Cluster, however you now risk reading a slightly out-of-date copy of the data\nin the event that you read from a node before the primary's latest data has synced to it.</p>\n</li>\n<li>\n<p>(Optional) It is possible to link a <code>user_group_id</code> and provide a list of user_id's to add additional layers of\nsecurity for your cluster or replication group. Refer to\nthe <a href=\"https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/Clusters.RBAC.html\" class=\"preview__body--description--blue\" target=\"_blank\">ElasticCache RBAC Access</a>\ndocumentation for further information.</p>\n</li>\n<li>\n<p>(Optional) It is also possible to update the <code>default</code> user id used by AWS when creating a redis cluster and replace\nit with your own <code>default</code> user. This is actually recommended by AWS as a first step to securing your Redis cluster.\nRefer to\nthe <a href=\"https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/Clusters.RBAC.html\" class=\"preview__body--description--blue\" target=\"_blank\">Role-Based Access Control (RBAC)</a>\ndocumentation on how to do this. The new user_id can be passed into the Redis clusters user group via\nthe <code>default_user_id</code> variable.</p>\n</li>\n</ul>\n<p>This module outputs <a href=\"https://www.terraform.io/intro/getting-started/outputs.html\" class=\"preview__body--description--blue\" target=\"_blank\">Terraform output variables</a> that\ncontain the address of the primary endpoint and read endpoints. You can programmatically extract these variables in your\nTerraform templates and pass them to other resources (e.g. as environment variables in an EC2 Instance) You'll also see\nthe variables at the end of each <code>terraform apply</code> call or if you run <code>terraform output</code>.</p>\n<h3 class=\"preview__body--subtitle\" id=\"how-do-you-scale-the-redis-cluster\">How do you scale the Redis Cluster?</h3>\n<p>You can scale your ElastiCache Cluster either horizontally (by adding more nodes) or vertically (by using more powerful\nnodes), but the method depends on whether Cluster Mode is enabled or disabled.</p>\n<h4 id=\"when-cluster-mode-is-disabled\">When Cluster Mode is Disabled</h4>\n<p>This mode is useful when you'd prefer to have only a single point for data to be written to the redis database. All data\nis written to the primary node of the single shard which is now replicated to the replica nodes. The advantage of this\napproach is that you're sure that all your data is present at a single point which could make migrations and backups a\nlot easier, if there's a problem with the primary write node however, all write attempts will fail.</p>\n<ul>\n<li>\n<p><strong>Vertical:</strong> You can increase the type of the read replica nodes using the <code>instance_type</code> parameter (\nsee <a href=\"https://aws.amazon.com/elasticache/details/#Available_Cache_Node_Types\" class=\"preview__body--description--blue\" target=\"_blank\">here</a> for valid values).</p>\n</li>\n<li>\n<p><strong>Horizontal:</strong> You can add up to 5 replica nodes to a Redis Cluster using the <code>cluster_size</code> parameter. There is\nalways a primary node where all writes take place, but you can reduce load on this primary node by offloading reads to\nthe non-primary nodes, which are known as Read Replicas.</p>\n</li>\n</ul>\n<p>For more info on both methods,\nsee <a href=\"https://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/Scaling.RedisStandalone.html\" class=\"preview__body--description--blue\" target=\"_blank\">Scaling Single-Node Redis (cluster mode disabled) Clusters</a>.</p>\n<h4 id=\"when-cluster-mode-is-enabled\">When Cluster Mode is Enabled</h4>\n<p>When cluster mode is enabled, data you write is split among the primary nodes in multiple Shards, and each data stored\nin those Shards are replicated among the read replicas. If one Shard becomes unavailable for whatever reason, other\nshards are still available for writing.</p>\n<p>To deploy a Redis "cluster mode enabled" cluster you must ensure that the <code>enable_automatic_failover</code> parameter is set\nto <code>true</code> and the <code>cluster_mode</code> variable has a single map with <code>num_shards</code> and <code>replicas_per_shard</code> parameters.</p>\n<p>E.g. the following Terraform code specifies a "cluster mode enabled" Cluster with 2 Shards and 1 Read Replica per Shard</p>\n<pre><span class=\"hljs-attr\">cluster_mode</span> = [\n {\n <span class=\"hljs-attr\">num_node_groups</span> = <span class=\"hljs-number\">2</span>\n <span class=\"hljs-attr\">replicas_per_node_group</span> = <span class=\"hljs-number\">1</span>\n }\n]\n</pre>\n<ul>\n<li>\n<p><strong>Horizontal:</strong> A "cluster mode enabled" cluster can be scaled horizontally by adding more Shards - also called\nResharding, the amount of read replicas present in the extra Shards is the same as the number specified when the\ncluster was originally created. Resharding can take anywhere from a few minutes to several hours.</p>\n</li>\n<li>\n<p><strong>Vertical:</strong> A "cluster mode enabled" cluster can be scaled vertically by changing the node types within each Shard.\nThis method is offline only and the cluster has to be backed up, a new cluster created with the required node types\nand the backup restored to that new cluster. During this process your entire cluster will experience a downtime which\ncould last for any length of time depending on how much data the original cluster contained.</p>\n</li>\n</ul>\n<p>For more info on scaling "cluster mode enabled" Redis clusters,\nsee <a href=\"https://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/scaling-redis-cluster-mode-enabled.html\" class=\"preview__body--description--blue\" target=\"_blank\">Scaling Multishard Redis (cluster mode disabled) Clusters</a></p>\n<h3 class=\"preview__body--subtitle\" id=\"how-do-you-use-the-auto-scaling-feature\">How do you use the auto-scaling feature?</h3>\n<p><a href=\"https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/AutoScaling.html\" class=\"preview__body--description--blue\" target=\"_blank\">Auto Scaling ElastiCache for Redis clusters</a>\nsummarizes how auto-scaling works for Redis clusters. You can enable auto-scaling via using the following variables:</p>\n<ul>\n<li><code>enable_auto_scaling</code>: true</li>\n<li><code>auto_scale_dimension</code>: choose one from the supported scalable dimension mentioned below</li>\n<li><code>auto_scale_trigger_metric</code>: choose one from the supported trigger metric mentioned below</li>\n<li><code>auto_scale_min_capacity</code>: choose the desired min capacity</li>\n<li><code>auto_scale_max_capacity</code>: choose the desired max capacity (<strong>Note</strong>: depends on the configuration of the Redis\ncluster. (e.g., 5 would be the maximum for\nreplica auto-scaling)</li>\n</ul>\n<h4 id=\"prerequisites\">Prerequisites</h4>\n<p>ElastiCache for Redis Auto Scaling is limited to the following:</p>\n<ul>\n<li>Redis (cluster mode enabled) clusters running Redis engine version 6.0 onwards</li>\n<li>Instance type families - R5, R6g, M5, M6g</li>\n<li>Instance sizes - Large, XLarge, 2XLarge</li>\n<li>Auto Scaling in ElastiCache for Redis is not supported for clusters running in Global datastores, Outposts or Local\nZones.</li>\n<li>AWS Auto Scaling for ElastiCache for Redis is not available in the following regions: China (Beijing), China (\nNingxia), AWS GovCloud (US-West) and AWS GovCloud (US-East).</li>\n</ul>\n<h4 id=\"supported-predefined-metrics\">Supported Predefined Metrics</h4>\n<p>Here are some predefined metrics available for Redis: <code>ElastiCachePrimaryEngineCPUUtilization</code> (shards\nauto-scale), <code>ElastiCacheReplicaEngineCPUUtilization</code> (replica auto-scale). You can find more information here:</p>\n<ul>\n<li><a href=\"https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/AutoScaling-Scaling-Policies-Replicas-Replicas.html\" class=\"preview__body--description--blue\" target=\"_blank\">Auto-Scaling with Replicas</a></li>\n<li><a href=\"https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/AutoScaling-Scaling-Policies-Target.html\" class=\"preview__body--description--blue\" target=\"_blank\">Auto-Scaling with Shards</a></li>\n</ul>\n<p>Both pages also explain how to use a custom metric and you can find the list of CloudWatch metrics for Redis from\nhere: https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/CacheMetrics.Redis.html</p>\n<h4 id=\"supported-scalable-dimensions\">Supported Scalable Dimensions</h4>\n<p>ElastiCache for Redis supports the following types of automatic scaling dimensions:</p>\n<ul>\n<li><code>elasticache:replication-group:NodeGroups</code> - The number of node groups for an Amazon ElastiCache replication group.</li>\n<li><code>elasticache:replication-group:Replicas</code> - The number of replicas per node group for an Amazon ElastiCache replication\ngroup.</li>\n</ul>\n<h2 class=\"preview__body--subtitle\" id=\"how-do-you-use-this-module\">How do you use this module?</h2>\n<p>Start by referencing one of the example modules:</p>\n<ul>\n<li>See the <a href=\"/repos/v0.22.2/module-cache/examples/redis\" class=\"preview__body--description--blue\">Redis Cluster Disabled Mode example</a> for an example.</li>\n<li>See the <a href=\"/repos/v0.22.2/module-cache/examples/redis_with_sharding\" class=\"preview__body--description--blue\">Redis CLuster Enabled Mode example</a> for an example.</li>\n<li>See the <a href=\"/repos/v0.22.2/module-cache/examples/redis_with_auth_token\" class=\"preview__body--description--blue\">Redis Auth Enabled Mode example</a> for an example.</li>\n<li>See the <a href=\"/repos/v0.22.2/module-cache/examples/redis_single_instance\" class=\"preview__body--description--blue\">Redis Single Instance Mode example</a> for an example.</li>\n<li>See the <a href=\"/repos/v0.22.2/module-cache/examples/redis_with_user_group_id\" class=\"preview__body--description--blue\">Redis User Group ID example</a> for an example.</li>\n</ul>\n<h3 class=\"preview__body--subtitle\" id=\"elasti-cache-old-terminology\">ElastiCache Old Terminology</h3>\n<p>Prior to October 2016, ElastiCache for Redis did not support <a href=\"http://redis.io/topics/partitioning\" class=\"preview__body--description--blue\" target=\"_blank\">partitioning</a>\n(aka sharding) in Redis and as a result used a bunch of confusing terminologies to explain setting up a Redis cluster.</p>\n<ul>\n<li>\n<p><strong>Cluster:</strong> In ElastiCache, each Redis node is labeled its own "cluster", despite that there is only a single node in\nthe cluster. This is an artifact of how ElastiCache implements memcached clusters, which can support multiple nodes.</p>\n</li>\n<li>\n<p><strong>Replication Group:</strong> The Redis "read/write primary cluster" will asynchronously copy its data to all other Redis\nclusters (individual nodes) in the same Replication Group. In a sense, the "real" redis cluster in ElastiCache is\nactually the Replication Group.</p>\n</li>\n<li>\n<p><strong>Primary Cluster:</strong> In a Replication Group, the primary cluster is where all write traffic is directed. Initially,\nyou can direct all your reads here as well.</p>\n</li>\n<li>\n<p><strong>Read Replicas:</strong> In a Replication Group, data written to the primary cluster is asynchronously copied to each Read\nReplica. These are useful both to reduce the read load on the primary cluster and to faciliate fast failover.</p>\n</li>\n<li>\n<p><strong>Multi-AZ:</strong> Using the Multi-AZ option means that if the primary cluster fails, ElastiCache will auto failover to a\nRead Replica. Without this option, the primary cluster node will simply replace itself, which will take longer and may\nresult in additional data loss.</p>\n</li>\n</ul>\n<h3 class=\"preview__body--subtitle\" id=\"common-gotchas\">Common Gotcha's</h3>\n<ul>\n<li>\n<p>In the event of a Redis failover, <em>you will experience a small period of time during which writes are not accepted</em>,\nso make sure your app can handle this gracefully. In fact, consider simulating Redis failovers on a regular basis or\nwith automated testing to validate that your app can handle it correctly.</p>\n</li>\n<li>\n<p>Test that your app does not cache the DNS value of the Primary Endpoint! Java, in particular, has undesirable defaults\naround DNS caching in many cases. If your code does not honor the TTL property of the Primary Endpoint's DNS record,\nthen your app may fail to reach the new primary node in the event of a failure.</p>\n</li>\n<li>\n<p>The only way to add more storage space to nodes in a Redis Cluster (cluster mode disabled) is to scale up to a larger\nnode type.</p>\n</li>\n</ul>\n<h3 class=\"preview__body--subtitle\" id=\"choosing-redis-cluster-mode-vs-single-instance\">Choosing Redis Cluster Mode vs. Single Instance</h3>\n<p>You can use <code>var.disable_cluster_mode=true</code> to deploy a single node Redis instance. Refer\nto <a href=\"/repos/v0.22.2/module-cache/examples/redis_single_instance\" class=\"preview__body--description--blue\">examples/redis_single_instance</a> as an example.</p>\n<p>Here are some of the points you may consider while choosing which mode to run:</p>\n<ul>\n<li><strong>Scalability</strong>: Redis single instance is limited to a single server node, which can be scaled vertically, while Redis\ncluster mode supports horizontal scaling by distributing data across multiple nodes, up to 1000 nodes. This allows for\nincreased capacity, availability, and fault tolerance.</li>\n<li><strong>High Availability</strong>: Redis single instance has limited high availability options, as it can only be replicated to a\nslave node. In contrast, Redis cluster mode provides more robust high availability features by automatically\ndistributing data across multiple nodes and ensuring data replication, even in the case of node failures.</li>\n<li><strong>Performance</strong>: Redis cluster mode provides improved performance, as it allows for distributing the data across\nmultiple nodes, providing more CPU and memory resources. Additionally, Redis cluster mode uses sharding to distribute\nkeys across multiple nodes, allowing for parallel data processing and faster data access.</li>\n<li><strong>Complexity</strong>: Redis cluster mode introduces more complexity than Redis single instance. Configuring and managing a\nRedis cluster requires more effort and expertise, as it involves setting up and managing multiple nodes, and\nconfiguring the sharding and replication.</li>\n</ul>\n","repoName":"module-cache","repoRef":"v0.22.4","serviceDescriptor":{"serviceName":"Redis","serviceRepoName":"module-cache","serviceRepoOrg":"gruntwork-io","serviceMainReadmePath":"/modules/redis","cloudProviders":["aws"],"description":"Run a Redis cluster using Amazon’s ElastiCache Service. Supports replication, multi-zone automatic failover, and automatic backup.","imageUrl":"redis.png","licenseType":"subscriber","technologies":["Terraform","Bash"],"compliance":[],"tags":[""]},"serviceCategoryName":"NoSQL","fileName":"README.md","filePath":"/modules/redis","title":"Repo Browser: Redis","description":"Browse the repos in the Gruntwork Infrastructure as Code Library."}
.circleci
main.tf
