Browse the Repo

file-type-icon.circleci
file-type-icon_docs
file-type-iconexamples
file-type-iconmodules
file-type-iconeks-alb-ingress-controller-iam-policy
file-type-iconeks-alb-ingress-controller
file-type-iconeks-cloudwatch-container-logs
file-type-iconeks-cluster-control-plane
file-type-iconeks-cluster-managed-workers
file-type-iconREADME.md
file-type-iconmain.tf
file-type-iconoutputs.tf
file-type-iconvariables.tf
file-type-iconeks-cluster-workers-cross-access
file-type-iconeks-cluster-workers
file-type-iconeks-iam-role-assume-role-policy-for-servic...
file-type-iconeks-k8s-cluster-autoscaler-iam-policy
file-type-iconeks-k8s-cluster-autoscaler
file-type-iconeks-k8s-external-dns-iam-policy
file-type-iconeks-k8s-external-dns
file-type-iconeks-k8s-role-mapping
file-type-iconeks-scripts
file-type-iconeks-vpc-tags
file-type-iconrfc
file-type-icontest
file-type-icon.gitignore
file-type-icon.pre-commit-config.yaml
file-type-iconCODEOWNERS
file-type-iconCONTRIBUTING.md
file-type-iconGRUNTWORK_PHILOSOPHY.md
file-type-iconLICENSE.md
file-type-iconREADME.adoc
file-type-iconcore-concepts.md
file-type-iconsetup.cfg

Browse the Repo

file-type-icon.circleci
file-type-icon_docs
file-type-iconexamples
file-type-iconmodules
file-type-iconeks-alb-ingress-controller-iam-policy
file-type-iconeks-alb-ingress-controller
file-type-iconeks-cloudwatch-container-logs
file-type-iconeks-cluster-control-plane
file-type-iconeks-cluster-managed-workers
file-type-iconREADME.md
file-type-iconmain.tf
file-type-iconoutputs.tf
file-type-iconvariables.tf
file-type-iconeks-cluster-workers-cross-access
file-type-iconeks-cluster-workers
file-type-iconeks-iam-role-assume-role-policy-for-servic...
file-type-iconeks-k8s-cluster-autoscaler-iam-policy
file-type-iconeks-k8s-cluster-autoscaler
file-type-iconeks-k8s-external-dns-iam-policy
file-type-iconeks-k8s-external-dns
file-type-iconeks-k8s-role-mapping
file-type-iconeks-scripts
file-type-iconeks-vpc-tags
file-type-iconrfc
file-type-icontest
file-type-icon.gitignore
file-type-icon.pre-commit-config.yaml
file-type-iconCODEOWNERS
file-type-iconCONTRIBUTING.md
file-type-iconGRUNTWORK_PHILOSOPHY.md
file-type-iconLICENSE.md
file-type-iconREADME.adoc
file-type-iconcore-concepts.md
file-type-iconsetup.cfg
EC2 Kubernetes Service (EKS) Cluster

EC2 Kubernetes Service (EKS) Cluster

Deploy a Kubernetes cluster on top of Amazon EC2 Kubernetes Service (EKS).

Code Preview

Preview the Code

mobile file icon

README.md

down

EKS Cluster Managed Workers Module

This module provisions EKS Managed Node Groups, as opposed to self managed ASGs. See the eks-cluster-workers module for a module to provision self managed worker groups.

This Terraform module launches worker nodes using EKS Managed Node Groups that you can use to run Kubernetes Pods and Deployments.

This module is responsible for the EKS Worker Nodes in the EKS cluster topology. You must launch a control plane in order for the worker nodes to function. See the eks-cluster-control-plane module for managing an EKS control plane.

How do you use this module?

  • See the root README for instructions on using Terraform modules.
  • See the examples folder for example usage.
  • See variables.tf for all the variables you can set on this module.
  • See outputs.tf for all the variables that are outputed by this module.

Differences with self managed workers

Managed Node Groups is a feature of EKS where you rely on EKS to manage the lifecycle of your worker nodes. This includes:

  • Automatic IAM role registration
  • Upgrades to platform versions and AMIs
  • Scaling up and down
  • Security Groups

Instead of manually managing Auto Scaling Groups and AMIs, you rely on EKS to manage those for you. This allows you to offload concerns such as upgrading and graceful scale out of your worker pools to AWS so that you don't have to manage them using tools like kubergrunt.

However, the trade off here is that managed node groups are more limited on the options for customizing the deployed servers. For example, you can not use any arbitrary AMI for managed node groups: they must be the officially published EKS optimized AMIs. You can't even use a custom AMI that is based off of the optimized AMIs. This means that you can't use utilities like ssh-grunt or ip-lockdown with Managed Node Groups.

Which flavor of worker pools to use depends on your infrastructure needs. Note that you can have both managed and self managed worker pools on a single EKS cluster, should you find the need for additional customizations.

Here is a list of additional tradeoffs to consider between the two flavors:

Managed Node Groups Self Managed Node Groups
Graceful Scale in and Scale out Supported automatically without needing additional tooling. Requires specialized tooling (e.g kubergrunt) to implement.
Boot scripts Not supported. Supported via user-data scripts in the ASG configuration.
OS Only supports Amazon Linux. Supports any arbitrary AMI, including Windows.
SSH access Only supports EC2 key pair, and restrictions by Security Group ID. Supports any PAM customized either in the AMI or boot scripts. Also supports any arbitrary security group configuration.
EBS Volumes Only supports adjusting the root EBS volume. Supports any EBS volume configuration, including attaching additional block devices.
ELB Supports automatic configuration via Kubernetes mechanisms. There is no way to manually register target groups to the ASG. Supports both automatic configuration by Kubernetes, and manual configuration with target group registration.
GPU support Supported via the GPU compatible EKS Optimized AMI. Supported via a GPU compatible AMI.

How do I enable cluster auto-scaling?

This module will not automatically scale in response to resource usage by default, the autoscaling_group_configurations.*.max_size option is only used to give room for new instances during rolling updates. To enable auto-scaling in response to resource utilization, deploy the Kubernetes Cluster Autoscaler module.

Note that the cluster autoscaler only supports ASGs that manage nodes in a single availability zone. This means that you need to carefully provision the managed node groups such that you have one group per AZ if you wish to use the cluster autoscaler. To accomplish this, ensure that the subnet_ids in each node_group_configurations input map entry come from the same AZ.

Refer to the Kubernetes Autoscaler documentation for more details.

How do I roll out an update to the instances?

Due to the way managed node groups work in Terraform, currently there is no way to rotate the instances without downtime when using terraform. Changes to the AMI or instance type will automatically cause the node group to be replaced. Additionally, the current resource does not support a mechanism to create the new group before destroying (the resource does not support name_prefix, and you can't create a new node group with the same name). As such, a naive update to the properties of the node group will likely lead to a period of reduced capacity as terraform replaces the groups.

To avoid downtime when updating your node groups, use a blue-green deployment:

  1. Provision a new node group with the updated, desired properties. You can do this by adding a new entry into the input map var.node_group_configurations.
  2. Apply the updated config using terraform apply to create the replacement node group.
  3. Once the new node group scales up, remove the old node group configuration from the input map.
  4. Apply the updated config using terraform apply to remove the old node group. The managed node group will gracefully scale down the nodes in Kubernetes (honoring PodDisruptionBudgets) before terminating them. During this process, the workloads will reschedule to the new nodes.

Questions? Ask away.

We're here to talk about our services, answer any questions, give advice, or just to chat.

Ready to hand off the Gruntwork?