Browse the Repo

file-type-icon.circleci
file-type-icon_docs
file-type-iconexamples
file-type-iconmodules
file-type-iconeks-alb-ingress-controller-iam-policy
file-type-iconeks-alb-ingress-controller
file-type-iconeks-aws-auth-merger
file-type-iconeks-cluster-control-plane
file-type-iconscripts
file-type-icontemplates
file-type-iconREADME.md
file-type-icondependencies.tf
file-type-iconmain.tf
file-type-iconoutputs.tf
file-type-iconvariables.tf
file-type-iconeks-cluster-managed-workers
file-type-iconeks-cluster-workers-cross-access
file-type-iconeks-cluster-workers
file-type-iconeks-container-logs
file-type-iconeks-iam-role-assume-role-policy-for-servic...
file-type-iconeks-k8s-cluster-autoscaler-iam-policy
file-type-iconeks-k8s-cluster-autoscaler
file-type-iconeks-k8s-external-dns-iam-policy
file-type-iconeks-k8s-external-dns
file-type-iconeks-k8s-role-mapping
file-type-iconeks-scripts
file-type-iconeks-vpc-tags
file-type-iconrfc
file-type-icontest
file-type-icon.gitignore
file-type-icon.pre-commit-config.yaml
file-type-iconCODEOWNERS
file-type-iconCONTRIBUTING.md
file-type-iconGRUNTWORK_PHILOSOPHY.md
file-type-iconLICENSE.md
file-type-iconREADME.adoc
file-type-iconcore-concepts.md
file-type-iconsetup.cfg
file-type-iconterraform-cloud-enterprise-private-module-...

Browse the Repo

file-type-icon.circleci
file-type-icon_docs
file-type-iconexamples
file-type-iconmodules
file-type-iconeks-alb-ingress-controller-iam-policy
file-type-iconeks-alb-ingress-controller
file-type-iconeks-aws-auth-merger
file-type-iconeks-cluster-control-plane
file-type-iconscripts
file-type-icontemplates
file-type-iconREADME.md
file-type-icondependencies.tf
file-type-iconmain.tf
file-type-iconoutputs.tf
file-type-iconvariables.tf
file-type-iconeks-cluster-managed-workers
file-type-iconeks-cluster-workers-cross-access
file-type-iconeks-cluster-workers
file-type-iconeks-container-logs
file-type-iconeks-iam-role-assume-role-policy-for-servic...
file-type-iconeks-k8s-cluster-autoscaler-iam-policy
file-type-iconeks-k8s-cluster-autoscaler
file-type-iconeks-k8s-external-dns-iam-policy
file-type-iconeks-k8s-external-dns
file-type-iconeks-k8s-role-mapping
file-type-iconeks-scripts
file-type-iconeks-vpc-tags
file-type-iconrfc
file-type-icontest
file-type-icon.gitignore
file-type-icon.pre-commit-config.yaml
file-type-iconCODEOWNERS
file-type-iconCONTRIBUTING.md
file-type-iconGRUNTWORK_PHILOSOPHY.md
file-type-iconLICENSE.md
file-type-iconREADME.adoc
file-type-iconcore-concepts.md
file-type-iconsetup.cfg
file-type-iconterraform-cloud-enterprise-private-module-...
EC2 Kubernetes Service (EKS) Cluster

EC2 Kubernetes Service (EKS) Cluster

Deploy a Kubernetes cluster on top of Amazon EC2 Kubernetes Service (EKS).

Code Preview

Preview the Code

mobile file icon

README.md

down

EKS Cluster Control Plane Module

This Terraform Module launches an Elastic Container Service for Kubernetes Cluster.

This module is responsible for the EKS Control Plane in the EKS cluster topology. You must launch worker nodes in order to be able to schedule pods on your cluster. See the eks-cluster-workers module for managing EKS worker nodes.

How do you use this module?

  • See the root README for instructions on using Terraform modules.
  • See the examples folder for example usage.
  • See variables.tf for all the variables you can set on this module.
  • See outputs.tf for all the variables that are outputed by this module.
  • This module depends on a packaged python binary, which requires a working python install.
  • This module depends on kubectl.
  • See How do I authenticate kubectl to the EKS cluster? for information on how to authenticate kubectl.
    • You will need to install kubectl to follow the instructions.
    • If you wish to use the automatic configuration, you will need kubergrunt. Refer to the kubergrunt documentation for installation instructions.

What is the EKS Control Plane?

The EKS Control Plane is a managed service entrirely managed by AWS. This contains the resources and endpoint to run and access the Kubernetes master components. The resources are deployed into your VPC so that they inherit the network rules you configure for your VPC.

Specifically, the control plane consists of:

  • etcd: A distributed key value store used by Kubernetes to hold the metadata and cluster state.
  • kube-apiserver: Web service that exposes the Kubernetes API. This is the main entrypoint for interacting with the Kubernetes cluster.
  • kube-scheduler: This component is responsible for watching for newly created Pods on the cluster, and scheduling them on to the available worker nodes.
  • kube-controller-manager: This component is responsible for executing the controller logic. Controllers are responsible for managing the Pods on the cluster. For example, you can use a Deployment controller to ensure that a specified number of replicas of a Pod is running on the cluster.
  • cloud-controller-manager: This component is responsible for managing cloud components that Kubernetes will manage. This includes resources like the LoadBalancers.

You can read more about the different components of EKS in the project README.

What security group rules are created?

This module will create a security group for the EKS cluster master nodes to allow them to function as a Kubernetes cluster. The rules are based on the recommendations provided by AWS for configuring an EKS cluster.

<a name="how-to-extend-security-group"></a>How do you add additional security group rules?

To add additional security group rules to the EKS cluster master nodes, you can use the aws_security_group_rule resource, and set its security_group_id argument to the Terraform output of this module called eks_control_plane_security_group_id. For example, here is how you can allow the master nodes in this cluster to allow incoming HTTPS requests on port 443 from an additional security group that is not the workers:

module "eks_cluster" {
  # (arguments omitted)
}

resource "aws_security_group_rule" "allow_inbound_http_from_anywhere" {
  type      = "ingress"
  from_port = 443
  to_port   = 443
  protocol  = "tcp"

  security_group_id        = module.eks_cluster.eks_control_plane_security_group_id
  source_security_group_id = var.source_aws_security_group_id
}

What IAM policies are attached to the EKS Cluster?

This module will create IAM roles for the EKS cluster master nodes with the minimum set of policies necessary for the cluster to function as a Kubernetes cluster. The policies attached to the roles are the same as those documented in the AWS getting started guide for EKS.

How do you add additional IAM policies?

To add additional IAM policies to the EKS cluster master nodes, you can use the aws_iam_role_policy or aws_iam_policy_attachment resources, and set the IAM role id to the Terraform output of this module called eks_control_plane_iam_role_name for the master nodes. For example, here is how you can allow the master nodes in this cluster to access an S3 bucket:

module "eks_cluster" {
  # (arguments omitted)
}

resource "aws_iam_role_policy" "access_s3_bucket" {
    name = "access_s3_bucket"
    role = module.eks_cluster.eks_control_plane_iam_role_name
    policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect":"Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::examplebucket/*"
    }
  ]
}
EOF
}

How do I associate IAM roles to the Pods?

NOTE: This configuration depends on kubergrunt, minimum version 0.5.3

This module will set up the OpenID Connect Provider that can be used with the IAM Roles for Service Accounts feature. When this feature is enabled, you can exchange the Kubernetes Service Account Tokens for IAM role credentials using the sts:AssumeRoleWithWebIdentity AWS API in the STS service.

To allow Kubernetes Service Accounts to assume the roles, you need to grant the proper assume role IAM policies to the role that is being assumed. Specifically, you need to:

  • Allow the OpenID Connect Provider to assume the role.
  • Specify any conditions on assuming the role. You can restrict by:
    • Service Accounts that can assume the role
    • Which Namespaces have full access to assume the role (meaning, all Service Accounts in the Namespace can assume that role).

You can use the eks-iam-role-assume-role-policy-for-service-account module to construct the policy using a more convenient interface. Refer to the module documentation for more info.

Once you have an IAM Role that can be assumed by the Kubernetes Service Account, you can configure your Pods to exchange them for IAM role credentials. EKS will automatically configure the correct environment variables that the SDK expects on the Pods when you annotate the associated Service Account with the role it should assume.

The following shows an example Kubernetes manifest that configures the Service Account to assume the IAM role arn:aws:iam::123456789012:role/myrole:

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/myrole

Note that the AWS SDK will automatically assume the role if you are using a compatible version. The following is a list of the minimum SDK version for various platforms that support the AWS_WEB_IDENTITY_TOKEN_FILE environment variable used by IRSA:

Java   1.11.623
Java2  2.7.36
Go     1.23.13
Python 1.9.220
Node   2.521.0
Ruby   2.11.345
PHP    3.110.7
.NET   3.3.580.0

How do I SSH into the nodes?

By design, AWS does not allow you to SSH into the master nodes of an EKS cluster.

API Access and Networking

By default the Kubernetes API is accessible from the internet. This is to make operations easier when making requests to the EKS cluster control plane. However, this module supports restricting access to only be allowed from within the VPC. You can restrict public access by setting the endpoint_public_access input variable to false.

Control Plane Logging

EKS supports exporting various logs to CloudWatch. By default, none of the logging options are enabled by this module. To enable logs, you can pass in the relevant type strings to the enabled_cluster_log_types input variable. For example, to enable API server and audit logs, you can pass in the list ["api", "audit"]. See the official documentation for a list of available log types.

How do I configure encryption at rest for Secrets?

Kubernetes Secrets are resources in the cluster designed to store and manage sensitive information. These behave like ConfigMaps, but have a few extra properties that enhance their security profile.

All EKS clusters encrypt Kubernetes Secrets at rest at the disk level using shared AWS managed KMS keys. Alternatively, you can provide your own KMS Customer Master Key (CMK) to use for envelope encryption. In envelope encryption, Kubernetes will use the provided CMK to encrypt the secret keys used to encrypt the Kubernetes Secrets. For each Secret, Kubernetes will dynamically generate a new data encryption key (DEK) for the purposes of encrypting and decrypting the secret. This key is then encrypted using the provided CMK before being stored in the cluster. In this way, you can manage access to the Secret (indirectly by restricting access to the DEK) through the KMS permissions. For example, you can disable all access to any Secrets in the EKS cluster by removing the permissions to encrypt/decrypt using the KMS key in case of a breach.

To enable envelope encryption, provide the KMS key ARN you would like to use using the variable secret_envelope_encryption_kms_key_arn. Note that if the KMS key belongs to another account, you will need to grant access to manage permissions for the key to the account holding the EKS cluster. See Allowing users in other accounts to use a CMK from the official AWS docs for more information.

How do I deploy Pods on Fargate?

AWS Fargate is an AWS managed infrastructure for running ECS Tasks and EKS Pods without any worker nodes. With Fargate, your EKS Pods will automatically be assigned a node from a shared pool of VMs that are fully managed by AWS. This means that you can focus entirely on the application you are deploying and not have to worry about servers, clusters, and the underlying infrastructure as a whole.

To use Fargate with your EKS Pods, you need to create a Fargate Profile to select the Pods that you want to run. You can use Namespaces and Labels to restrict which Pods of the EKS cluster will run on Fargate. This means that Pods that match the specifications of the Fargate Profile will automatically be deployed to Fargate without any further configuration.

Some additional notes on using Fargate:

  • Fargate Profiles require a Pod Execution Role, which is an IAM role that will be assigned to the underlying kubelet of the Fargate instance. At a minimum, this role must be given enough permissions to pull the images used by the Pod. Note that this role is NOT made available to the Pods! Use the IAM Role for Service Accounts (IRSA) feature of EKS to assign IAM roles for use by the Pods themselves.
  • If you set the input variable schedule_control_plane_services_on_fargate on this module, the module will automatically allocate a Fargate Profile that selects the core control plane services deployed in the kube-system Namespace (e.g., core-dns). This profile is highly selective and will most likely not match any other Pods in the cluster. To deploy additional Pods onto Fargate, you must manually create Fargate Profiles that select those Pods (use the aws_eks_fargate_profile resource to provision Fargate Profiles with Terraform). The Pod Execution Role created by the module may be reused for other Fargate Profiles.
  • Fargate does not support DaemonSets. This means that you can't rely on the eks-container-logs module to forward logs to CloudWatch. Instead, you need to manually configure a sidecar fluentd container that forwards the log entries to CloudWatch Logs. Refer to this AWS blog post for documentation on how to setup fluentd with Fargate.

How do I upgrade the Kubernetes Version of the cluster?

To upgrade the minor version of Kubernetes deployed on the EKS cluster, you need to update the kubernetes_version input variable. You must upgrade one minor version at a time, as EKS does not support upgrading by more than one minor version.

When you bump minor versions, the module will automatically update the deployed Kubernetes components as described in the official upgrade guide. This is handled by kubergrunt (minimum version 0.6.2) using the eks sync-core-components command, which will look up the deployed Kubernetes version and make the required kubectl calls to deploy the updated components.

Note that you must update the nodes to use the corresponding kubelet version as well. This means that when you update minor versions, you will also need to update the AMIs used by the worker nodes to match the version and rotate the workers. For more information on rotating worker nodes, refer to How do I roll out an update to the instances? in the eks-cluster-workers module README.

Here are detailed steps on how to update your cluster:

  1. Bump the kubernetes_version in the module variable to the next minor version in your module block for eks-cluster-control-plane.
  2. For self managed worker nodes (eks-cluster-workers module), build a new AMI for your worker nodes that depend on an EKS optimized AMI for the Kubernetes minor version. Update the asg_default_instance_ami variable to the new AMI in your module block for eks-cluster-workers.
  3. Apply the changes. This will update the Kubernetes version on the EKS control plane, and stage the updates for your workers. Note that your cluster will continue to function as Kubernetes supports worker nodes that are 1 minor version behind.
  4. Roll out the AMI update using kubergrunt eks deploy.

Python Helper Scripts

This module contains various python helper scripts that are used to achieve enhanced functionality not available in Terraform. All these scripts are intended to be used as part of local-exec provisioner. In order to support disabling the scripts, we use a null_resource to wrap the provisioner. See this module's main.tf file for example usage.

Since this is a python script, The operator machine must have a valid python interpreter available in the PATH under the name python. The scripts support python versions 2.7, 3.5, 3.6, 3.7, and 3.8, on Mac OSX or Linux.

Below is an overview of the scripts:

  • cleanup_cluster_resources: Clean up any residual resources managed by the EKS cluster that may be remaining after the cluster is destroyed. This is used to workaround terraform and kubernetes limitations that may leave managed resources behind.

Troubleshooting

AccessDenied when provisioning Services of LoadBalancer type

On brand new accounts, AWS needs to provision a new Service Linked Role for ELBs when an ELB is first provisioned. EKS automatically creates the Service Linked Role if it doesn't exist, but it needs more permissions than is provided by default. Since the permission is only needed as a one time thing, binding the necessary permissions would be a violation of least privileges.

As such, this module does not bind the requisite permissions, and instead we recommend taking one of the following approaches:

  • Create a one time wrapper module that appends the following IAM permissions to the control plane IAM role (the output eks_master_iam_role_arn), and deploy the EKS cluster with LoadBalancer service:

      ec2:DescribeAccountAttributes
      ec2:DescribeInternetGateways
    
  • Create an ELB using the AWS console, or the modules in terraform-aws-load-balancer.

  • Create the service linked role using the Landing Zone modules.

Questions? Ask away.

We're here to talk about our services, answer any questions, give advice, or just to chat.

Ready to hand off the Gruntwork?