Browse the Repo
.circleci_docschartsk8s-servicetemplates
.helmignoreChart.yamlREADME.mdlinter_values.yamlvalues.yamlexamplestest.gitignore.pre-commit-config.yamlCODEOWNERSCONTRIBUTING.mdGRUNTWORK_PHILOSOPHY.mdLICENSENOTICEREADME.adoccore-concepts.mdBrowse the Repo
.circleci_docschartsk8s-servicetemplates.helmignoreChart.yamlREADME.mdlinter_values.yamlvalues.yamlexamplestest.gitignore.pre-commit-config.yamlCODEOWNERSCONTRIBUTING.mdGRUNTWORK_PHILOSOPHY.mdLICENSENOTICEREADME.adoccore-concepts.mdKubernetes Service
Deploy a Kubernetes service with zero-downtime, rolling deployment, RBAC, auto scaling, secrets management, and more.
Code Preview
View on GitHub:
https://github.com/gruntwork-i...0/charts/k8s-service/README.mdPreview the Code
README.md
Kubernetes Service Helm Chart
This Helm Chart can be used to deploy your application container under a
Deployment resource onto your Kubernetes
cluster. You can use this Helm Chart to run and deploy a long-running container, such as a web service or backend
microservice. The container will be packaged into
Pods that are managed by the Deployment
controller.
This Helm Chart can also be used to front the Pods of the Deployment resource with a
Service to provide a stable endpoint to access the
Pods, as well as load balance traffic to them. The Helm Chart can also specify
Ingress rules to further configure complex routing
rules in front of the Service.
If you're using the chart to deploy to GKE, you can also use the chart to deploy a Google Managed SSL Certificate and associate it with the Ingress.
How to use this chart?
- See the root README for general instructions on using Gruntwork Helm Charts.
- See the examples folder for example usage.
- See the provided values.yaml file for the required and optional configuration values that you can set on this chart.
What resources does this Helm Chart deploy?
The following resources will be deployed with this Helm Chart, depending on which configuration values you use:
Deployment: The mainDeploymentcontroller that will manage the application container image specified in thecontainerImageinput value.- Secondary
Deploymentfor use as canary: An optionalDeploymentcontroller that will manage a canary deployment of the application container image specified in thecanary.containerImageinput value. This is useful for testing a new application tag, in parallel to your stable tag, prior to rolling the new tag out. Created only if you configure thecanary.containerImagevalues (and setcanary.enabled = true). Service: TheServiceresource providing a stable endpoint that can be used to address toPodscreated by theDeploymentcontroller. Created only if you configure theserviceinput (and setservice.enabled = true).ServiceMonitor: TheServiceMonitordescribes the set of targets to be monitored by Prometheus. Created only if you configure the service input and setserviceMonitor.enabled = true.Ingress: TheIngressresource providing host and path routing rules to theServicefor the deployedIngresscontroller in the cluster. Created only if you configure theingressinput (and setingress.enabled = true).Horizontal Pod Autoscaler: TheHorizontal Pod Autoscalerautomatically scales the number of pods in a replication controller, deployment, replica set or stateful set based on observed CPU or memory utilization. Created only if the user setshorizontalPodAutoscaler.enabled = true.PodDisruptionBudget: ThePodDisruptionBudgetresource that specifies a disruption budget for thePodsmanaged by theDeployment. This manages how many pods can be disrupted by a voluntary disruption (e.g node maintenance). Created if you specify a non-zero value for theminPodsAvailableinput value.ManagedCertificate: TheManagedCertificateis a GCP -specific resource that creates a Google Managed SSL certificate. Google-managed SSL certificates are provisioned, renewed, and managed for your domain names. Read more about Google-managed SSL certificates here. Created only if you configure thegoogle.managedCertificateinput (and setgoogle.managedCertificate.enabled = trueandgoogle.managedCertificate.domainName = your.domain.name).
How do I expose my application internally to the cluster?
In general, Pods are considered ephemeral in Kubernetes. Pods can come and go at any point in time, either because
containers fail or the underlying instances crash. In either case, the dynamic nature of Pods make it difficult to
consistently access your application if you are individually addressing the Pods directly.
Traditionally, this is solved using service discovery, where you have a stateful system that the Pods would register
to when they are available. Then, your other applications can query the system to find all the available Pods and
access one of the available ones.
Kubernetes provides a built in mechanism for service discovery in the Service resource. Services are an abstraction
that groups a set of Pods behind a consistent, stable endpoint to address them. By creating a Service resource, you
can provide a single endpoint to other applications to connect to the Pods behind the Service, and not worry about
the dynamic nature of the Pods.
You can read a more detailed description of Services in the official
documentation. Here we will cover just enough to
understand how to access your app.
By default, this Helm Chart will deploy your application container in a Pod that exposes ports 80. These will
be exposed to the Kubernetes cluster behind the Service resource, which exposes port 80. You can modify this behavior
by overriding the containerPorts input value and the service input value. See the corresponding section in the
values.yaml file for more details.
Once the Service is created, you can check what endpoint the Service provides by querying Kubernetes using
kubectl. First, retrieve the Service name that is outputted in the install summary when you first install the Helm
Chart. If you forget, you can get the same information at a later point using helm status. For example, if you had
previously installed this chart under the name edge-service, you can run the following command to see the created
resources:
$ helm status edge-service
LAST DEPLOYED: Fri Feb 8 16:25:49 2019
NAMESPACE: default
STATUS: DEPLOYED
RESOURCES:
==> v1/Service
NAME AGE
edge-service-nginx 24m
==> v1/Deployment
edge-service-nginx 24m
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
edge-service-nginx-844c978df7-f5wc4 1/1 Running 0 24m
edge-service-nginx-844c978df7-mln26 1/1 Running 0 24m
edge-service-nginx-844c978df7-rdsr8 1/1 Running 0 24m
This will show you some metadata about the release, the deployed resources, and any notes provided by the Helm Chart. In
this example, the service name is edge-service-nginx so we will use that to query the Service:
$ kubectl get service edge-service-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
edge-service-nginx ClusterIP 172.20.186.176 <none> 80/TCP 27m
Here you can see basic information about the Service. The important piece of information is the CLUSTER-IP and
PORT fields, which tell you the available endpoint for the Service, and any exposed ports. Given that, any Pod in
your Kubernetes cluster can access the Pods of this application by hitting {CLUSTER-IP}:{PORT}. So for this example,
that will be 172.20.186.176:80.
But what if you want to automatically find a Service by name? The name of the Service created by this Helm Chart is
always {RELEASE_NAME}-{applicationName}, where applicationName is provided in the input value and RELEASE_NAME is
set when you install the Helm Chart. This means that the name is predictable, while the allocated IP address may not be.
To address the Service by name, Kubernetes provides two ways:
- environment variables
- DNS
Addressing Service by Environment Variables
For each active Service that a Pod has access to, Kubernetes will automatically set a set of environment variables
in the container. These are {SVCNAME}_SERVICE_HOST and {SVCNAME}_SERVICE_PORT to get the host address (ip address)
and port respectively, where SVCNAME is the name of the Service. Note that SVCNAME will be the all caps version
with underscores of the Service name.
Using the previous example where we installed this chart with a release name edge-service and applicationName
nginx, we get the Service name edge-service-nginx. Kubernetes will expose the following environment variables to
all containers that can access the Service:
EDGE_SERVICE_NGINX_SERVICE_HOST=172.20.186.176
EDGE_SERVICE_NGINX_SERVICE_PORT=80
Note that environment variables are set when the container first boots up. This means that if you already had Pods
deployed in your system before the Service was created, you will have to cycle the Pods in order to get the
environment variables. If you wish to avoid ordering issues, you can use the DNS method to address the Service
instead, if that is available.
Addressing Service by DNS
If your Kubernetes cluster is deployed with the DNS add-on (this is automatically installed for EKS and GKE), then you
can rely on DNS to address your Service. Every Service in Kubernetes will register the domain
{SVCNAME}.{NAMESPACE}.svc.cluster.local to the DNS service of the cluster. This means that all your Pods in the
cluster can get the Service host by hitting that domain.
The NAMESPACE in the domain refers to the Namespace where the Service was created. By default, all resources are
created in the default namespace. This is configurable at install time of the Helm Chart using the --namespace
option.
In our example, we deployed the chart to the default Namespace, and the Service name is edge-service-nginx. So in
this case, the domain of the Service will be edge-service-nginx.default.svc.cluster.local. When any Pod addresses
that domain, it will get the address 172.20.186.176.
Note that DNS does not resolve ports, so in this case, you will have to know which port the Service uses. So in your
Pod, you will have to know that the Service exposes port 80 when you address it in your code for the container as
edge-service-nginx.default.svc.cluster.local:80. However, like the Service name, this should be predictable since it
is specified in the Helm Chart input value.
How do I expose my application externally, outside of the cluster?
Similar to the previous section (How do I expose my application internally to the
cluster?, you can use a Service resource to expose your
application externally. The primary service type that facilitates external access is the NodePort Service type.
The NodePort Service type will expose the Service by binding an available port on the network interface of the
physical machines running the Pod. This is different from a network interface internal to Kubernetes, which is only
accessible within the cluster. Since the port is on the host machine network interface, you can access the Service by
hitting that port on the node.
For example, suppose you had a 2 node Kubernetes cluster deployed on EC2. Suppose further that all your EC2 instances have public IP addresses that you can access. For the sake of this example, we will assign random IP addresses to the instances:
- 54.219.117.250
- 38.110.235.198
Now let's assume you deployed this helm chart using the NodePort Service type. You can do this by setting the
service.type input value to NodePort:
service:
enabled: true
type: NodePort
ports:
app:
port: 80
targetPort: 80
protocol: TCP
When you install this helm chart with this input config, helm will deploy the Service as a NodePort, binding an
available port on the host machine to access the Service. You can confirm this by querying the Service using
kubectl:
$ kubectl get service edge-service-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
edge-service-nginx NodePort 10.99.244.96 <none> 80:31035/TCP 33s
In this example, you can see that the Service type is NodePort as expected. Additionally, you can see that the there
is a port binding between port 80 and 31035. This port binding refers to the binding between the Service port (80 in
this case) and the host port (31035 in this case).
One thing to be aware of about NodePorts is that the port binding will exist on all nodes in the cluster. This means
that, in our 2 node example, both nodes now have a port binding of 31035 on the host network interface that routes to
the Service, regardless of whether or not the node is running the Pods backing the Service endpoint. This means
that you can reach the Service on both of the following endpoints:
54.219.117.250:3103538.110.235.198:31035
This means that no two Service can share the same NodePort, as the port binding is shared across the cluster.
Additionally, if you happen to hit a node that is not running a Pod backing the Service, Kubernetes will
automatically hop to one that is.
You might use the NodePort if you do not wish to manage load balancers through Kubernetes, or if you are running
Kubernetes on prem where you do not have native support for managed load balancers.
To summarize:
NodePortis the simplest way to expose yourServiceto externally to the cluster.- You have a limit on the number of
NodePortServicesyou can have in your cluster, imposed by the number of open ports available on your host machines. - You have potentially inefficient hopping if you happen to route to a node that is not running the
Podbacking theService.
Additionally, Kubernetes provides two mechanisms to manage an external load balancer that routes to the NodePort for
you. The two ways are:
LoadBalancer Service Type
The LoadBalancer Service type will expose the Service by allocating a managed load balancer in the cloud that is
hosting the Kubernetes cluster. On AWS, this will be an ELB, while on GCP, this will be a Cloud Load Balancer. When the
LoadBalancer Service is created, Kubernetes will automatically create the underlying load balancer resource in the
cloud for you, and create all the target groups so that they route to the Pods backing the Service.
You can deploy this helm chart using the LoadBalancer Service type by setting the service.type input value to
LoadBalancer:
service:
enabled: true
type: LoadBalancer
ports:
app:
port: 80
targetPort: 80
protocol: TCP
When you install this helm chart with this input config, helm will deploy the Service as a LoadBalancer, allocating
a managed load balancer in the cloud hosting your Kubernetes cluster. You can get the attached load balancer by querying
the Service using kubectl. In this example, we will assume we are using EKS:
$ kubectl get service edge-service-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
edge-service-nginx LoadBalancer 172.20.7.35 a02fef4d02e41... 80:32127/TCP 1m
Now, in this example, we have an entry in the EXTERNAL-IP field. This is truncated here, but you can get the actual
output when you describe the service:
$ kubectl describe service edge-service-nginx
Name: edge-service-nginx
Namespace: default
Labels: app.kubernetes.io/instance=edge-service
app.kubernetes.io/managed-by=Tiller
app.kubernetes.io/name=nginx
gruntwork.io/app-name=nginx
helm.sh/chart=k8s-service-0.1.0
Annotations: <none>
Selector: app.kubernetes.io/instance=edge-service,app.kubernetes.io/name=nginx,gruntwork.io/app-name=nginx
Type: LoadBalancer
IP: 172.20.7.35
LoadBalancer Ingress: a02fef4d02e4111e9891806271fc7470-173030870.us-west-2.elb.amazonaws.com
Port: app 80/TCP
TargetPort: 80/TCP
NodePort: app 32127/TCP
Endpoints: 10.0.3.19:80
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal EnsuringLoadBalancer 2m service-controller Ensuring load balancer
Normal EnsuredLoadBalancer 2m service-controller Ensured load balancer
In the describe output, there is a field named LoadBalancer Ingress. When you have a LoadBalancer Service type,
this field contains the public DNS endpoint of the associated load balancer resource in the cloud provider. In this
case, we have an AWS ELB instance, so this endpoint is the public endpoint of the associated ELB resource.
Note: Eagle eyed readers might also notice that there is an associated NodePort on the resource. This is because under the
hood, LoadBalancer Services utilize NodePorts to handle the connection between the managed load balancer of the
cloud provider and the Kubernetes Pods. This is because at this time, there is no portable way to ensure that the
network between the cloud load balancers and Kubernetes can be shared such that the load balancers can route to the
internal network of the Kubernetes cluster. Therefore, Kubernetes resorts to using NodePort as an abstraction layer to
connect the LoadBalancer to the Pods backing the Service. This means that LoadBalancer Services share the same
drawbacks as using a NodePort Service.
To summarize:
LoadBalancerprovides a way to set up a cloud load balancer resource that routes to the provisionedNodePorton each node in your Kubernetes cluster.LoadBalancercan be used to provide a persistent endpoint that is robust to the ephemeral nature of nodes in your cluster. E.g it is able to route to live nodes in the face of node failures.LoadBalancerdoes not support weighted balancing. This means that you cannot balance the traffic so that it prefers nodes that have more instances of thePodrunning.- Note that under the hood,
LoadBalancerutilizes aNodePortService, and thus shares the same limits asNodePort.
Ingress and Ingress Controllers
Ingress is a mechanism in Kubernetes that abstracts externally exposing a Service from the Service config itself.
Ingress resources support:
- assigning an externally accessible URL to a
Service - perform hostname and path based routing of
Services - load balance traffic using customizable balancing rules
- terminate SSL
You can read more about Ingress resources in the official
documentation. Here, we will cover the basics to
understand how Ingress can be used to externally expose the Service.
At a high level, the Ingress resource is used to specify the configuration for a particular Service. In turn, the
Ingress Controller is responsible for fulfilling those configurations in the cluster. This means that the first
decision to make in using Ingress resources, is selecting an appropriate Ingress Controller for your cluster.
Choosing an Ingress Controller
Before you can use an Ingress resource, you must install an Ingress Controller in your Kubernetes cluster. There are
many kinds of Ingress Controllers available, each with different properties. You can see a few examples listed in the
official documentation.
When you use an external cloud Ingress Controller such as the GCE Ingress
Controller or AWS ALB Ingress
Controller, Kubernetes will allocate an externally
addressable load balancer (for GCE this will be a Cloud Load Balancer and for AWS this will be an ALB) that fulfills the
Ingress rules. This includes routing the domain names and paths to the right Service as configured by the Ingress
rules. Additionally, Kubernetes will manage the target groups of the load balancer so that they are up to date with
the latest Ingress configuration. However, in order for this to work, there needs to be some way for the load balancer
to connect to the Pods servicing the Service. Since the Pods are internal to the Kubernetes network and the load
balancers are external to the network, there must be a NodePort that links the two together. As such, like the
LoadBalancer Service type, these Ingress Controllers also require a NodePort under the hood.
Alternatively, you can use an internal Ingress Controller that runs within Kubernetes as Pods. For example, the
official nginx Ingress Controller will launch nginx as Pods within your Kubernetes cluster. These nginx Pods
are then configured using Ingress resources, which then allows nginx to route to the right Pods. Since the nginx
Pods are internal to the Kubernetes network, there is no need for your Services to be NodePorts as they are
addressable within the network by the Pods. However, this means that you need some other mechanism to expose nginx
to the outside world, which will require a NodePort. The advantage of this approach, despite still requiring a
NodePort, is that you can have a single NodePort that routes to multiple services using hostnames or paths as
managed by nginx, as opposed to requiring a NodePort per Service you wish to expose.
Which Ingress Controller type you wish to use depends on your infrastructure needs. If you have relatively few
Services, and you want the simplicity of a managed cloud load balancer experience, you might opt for the external
Ingress Controllers such as GCE and AWS ALB controllers. On the other hand, if you have thousands of micro services
that push you to the limits of the available number of ports on a host machine, you might opt for an internal Ingress Controller approach. Whichever approach you decide, be sure to document your decision where you install the particular
Ingress Controller so that others in your team know and understand the tradeoffs you made.
Configuring Ingress for your Service
Once you have an Ingress Controller installed and configured on your Kuberentes cluster, you can now start creating
Ingress resources to add routes to it. This helm chart supports configuring an Ingress resource to complement the
Service resource that is created in the chart.
To add an Ingress resource, first make sure you have a Service enabled on the chart. Depending on the chosen
Ingress Controller, the Service type should be NodePort or ClusterIP. Here, we will create a NodePort
Service exposing port 80:
service:
enabled: true
type: NodePort
ports:
app:
port: 80
targetPort: 80
protocol: TCP
Then, we will add the configuration for the Ingress resource by specifying the ingress input value. For this
example, we will assume that we want to route /app to our Service, with the domain hosted on app.yourco.com:
ingress:
enabled: true
path: /app
servicePort: 80
hosts:
- app.yourco.com
This will configure the load balancer backing the Ingress Controller that will route any traffic with host and path
prefix app.yourco.com/app to the Service on port 80. If app.yourco.com is configured to point to the Ingress Controller load balancer, then once you deploy the helm chart you should be able to start accessing your app on that
endpoint.
Registering additional paths
Sometimes you might want to add additional path rules beyond the main service rule that is injected to the Ingress
resource. For example, you might want a path that routes to the sidecar containers, or you might want to reuse a single
Ingress for multiple different Service endpoints because to share load balancers. For these situations, you can use
the additionalPaths and additionalPathsHigherPriority input values.
Consider the following Service, where we have the app served on port 80, and the sidecarMonitor served on port
3000:
service:
enabled: true
type: NodePort
ports:
app:
port: 80
targetPort: 80
protocol: TCP
sidecarMonitor:
port: 3000
targetPort: 3000
protocol: TCP
To route /app to the app service endpoint and /sidecar to the sidecarMonitor service endpoint, we will configure
the app service path rules as the main service route and the sidecarMonitor as an additional path rule:
ingress:
enabled: true
path: /app
servicePort: 80
additionalPaths:
- path: /sidecar
servicePort: 3000
Now suppose you had a sidecar service that will return a fixed response indicating server maintainance and you want to
temporarily route all requests to that endpoint without taking down the pod. You can do this by creating a route that
catches all paths as a higher priority path using the additionalPathsHigherPriority input value.
Consider the following Service, where we have the app served on port 80, and the sidecarFixedResponse served on
port 3000:
service:
enabled: true
type: NodePort
ports:
app:
port: 80
targetPort: 80
protocol: TCP
sidecarFixedResponse:
port: 3000
targetPort: 3000
protocol: TCP
To route all traffic to the fixed response port:
ingress:
enabled: true
path: /app
servicePort: 80
additionalPathsHigherPriority:
- path: /*
servicePort: 3000
The /* rule which routes to port 3000 will always be used even when accessing the path /app because it will be
evaluated first when routing requests.
How do I deploy a worker service?
Worker services typically do not have a RPC or web server interface to access it. Instead, worker services act on their
own and typically reach out to get the data they need. These services should be deployed without any ports exposed.
However, by default k8s-service will deploy an internally exposed service with port 80 open.
To disable the default port, you can use the following values.yaml inputs:
containerPorts:
http:
disabled: true
service:
enabled: false
This will override the default settings such that only the Deployment resource is created, with no ports exposed on
the container.
How do I check the status of the rollout?
This Helm Chart packages your application into a Deployment controller. The Deployment controller will be
responsible with managing the Pods of your application, ensuring that the Kubernetes cluster matches the desired state
configured by the chart inputs.
When the Helm Chart installs, helm will mark the installation as successful when the resources are created. Under the
hood, the Deployment controller will do the work towards ensuring the desired number of Pods are up and running.
For example, suppose you set the replicaCount variable to 3 when installing this chart. This will configure the
Deployment resource to maintain 3 replicas of the Pod at any given time, launching new ones if there is a deficit or
removing old ones if there is a surplus.
To see the current status of the Deployment, you can query Kubernetes using kubectl. The Deployment resource of
the chart are labeled with the applicationName input value and the release name provided by helm. So for example,
suppose you deployed this chart using the following values.yaml file and command:
applicationName: nginx
containerImage:
repository: nginx
tag: stable
$ helm install -n edge-service gruntwork/k8s-service
In this example, the applicationName is set to nginx, while the release name is set to edge-service. This chart
will then install a Deployment resource in the default Namespace with the following labels that uniquely identifies
it:
app.kubernetes.io/name: nginx
app.kubernetes.io/instance: edge-service
So now you can query Kubernetes for that Deployment resource using these labels to see the state:
$ kubectl get deployments -l "app.kubernetes.io/name=nginx,app.kubernetes.io/instance=edge-service"
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
edge-service-nginx 3 3 3 1 24s
This includes a few useful information:
DESIREDlists the number ofPodsthat should be running in your cluster.CURRENTlists how manyPodsare currently created in the cluster.UP-TO-DATElists how manyPodsare running the desired image.AVAILABLElists how manyPodsare currently ready to serve traffic, as defined by thereadinessProbe.
When all the numbers are in sync and equal, that means the Deployment was rolled out successfully and all the Pods
are passing the readiness healthchecks.
In the example output above, note how the Available count is 1, but the others are 3. This means that all 3 Pods
were successfully created with the latest image, but only 1 of them successfully came up. You can dig deeper into the
individual Pods to check the status of the unavailable Pods. The Pods are labeled the same way, so you can pass in
the same label query to get the Pods managed by the deployment:
$ kubectl get pods -l "app.kubernetes.io/name=nginx,app.kubernetes.io/instance=edge-service"
NAME READY STATUS RESTARTS AGE
edge-service-nginx-844c978df7-f5wc4 1/1 Running 0 52s
edge-service-nginx-844c978df7-mln26 0/1 Pending 0 52s
edge-service-nginx-844c978df7-rdsr8 0/1 Pending 0 52s
This will show you the status of each individual Pod in your deployment. In this example output, there are 2 Pods
that are in the Pending status, meaning that they have not been scheduled yet. We can look into why the Pod failed
to schedule by getting detailed information about the Pod with the describe command. Unlike get pods, describe pod requires a single Pod so we will grab the name of one of the failing Pods above and feed it to describe pod:
$ kubectl describe pod edge-service-nginx-844c978df7-mln26
Name: edge-service-nginx-844c978df7-mln26
Namespace: default
Priority: 0
PriorityClassName: <none>
Node: <none>
Labels: app.kubernetes.io/instance=edge-service
app.kubernetes.io/name=nginx
gruntwork.io/app-name=nginx
pod-template-hash=4007534893
Annotations: <none>
Status: Pending
IP:
Controlled By: ReplicaSet/edge-service-nginx-844c978df7
Containers:
nginx:
Image: nginx:stable
Ports: 80/TCP
Host Ports: 0/TCP
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mgkr9 (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
default-token-mgkr9:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-mgkr9
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 1m (x25 over 3m) default-scheduler 0/2 nodes are available: 2 Insufficient pods.
This will output detailed information about the Pod, including an event log. In this case, the roll out failed because
there is not enough capacity in the cluster to schedule the Pod.
How do I set and share configurations with the application?
While you can bake most application configuration values into the application container, you might need to inject dynamic configuration variables into the container. These are typically values that change depending on the environment, such as the MySQL database endpoint. Additionally, you might also want a way to securely share secrets with the container such that they are not hard coded in plain text in the container or in the Helm Chart values yaml file. To support these use cases, this Helm Chart provides three ways to share configuration values with the application container:
Directly setting environment variables
The simplest way to set a configuration value for the container is to set an environment variable for the container runtime. These variables are set by Kubernetes before the container application is booted, which can then be looked up using the standard OS lookup functions for environment variables.
You can use the envVars input value to set an environment variable at deploy time. For example, the following entry in
a values.yaml file will set the DB_HOST environment variable to mysql.default.svc.cluster.local and the DB_PORT
environment variable to 3306:
envVars:
DB_HOST: "mysql.default.svc.cluster.local"
DB_PORT: 3306
One thing to be aware of when using environment variables is that they are set at start time of the container. This means that updating the environment variables require restarting the containers so that they propagate.
Using ConfigMaps
While environment variables are an easy way to inject configuration values, what if you want to share the configuration across multiple deployments? If you wish to use the direct environment variables approach, you would have no choice but to copy paste the values across each deployment. When this value needs to change, you are now faced with going through each deployment and updating the reference.
For this situation, ConfigMaps would be a better option. ConfigMaps help decouple configuration values from the
Deployment and Pod config, allowing you to share the values across the deployments. ConfigMaps are dedicated
resources in Kubernetes that store configuration values as key value pairs.
For example, suppose you had a ConfigMap to store the database information. You might store the information as two key
value pairs: one for the host (dbhost) and one for the port (dbport). You can create a ConfigMap directly using
kubectl, or by using a resource file.
To directly create the ConfigMap:
kubectl create configmap my-config --from-literal=dbhost=mysql.default.svc.cluster.local --from-literal=dbport=3306
Alternatively, you can manage the ConfigMap as code using a kubernetes resource config:
# my-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: my-config
data:
dbhost: mysql.default.svc.cluster.local
dbport: 3306
You can then apply this resource file using kubectl:
kubectl apply -f my-config.yaml
kubectl supports multiple ways to seed the ConfigMap. You can read all the different ways to create a ConfigMap in
the official
documentation.
Once the ConfigMap is created, you can access the ConfigMap within the Pod by configuring the access during
deployment. This Helm Chart provides the configMaps input value to configure what ConfigMaps should be shared with
the application container. There are two ways to inject the ConfigMap:
NOTE: It is generally not recommended to use ConfigMaps to store sensitive data. For those use cases, use
Secrets or an external secret store.
Accessing the ConfigMap as Environment Variables
You can set the values of the ConfigMap as environment variables in the application container. To do so, you set the
as attribute of the configMaps input value to environment. For example, to share the my-config ConfigMap above
using the same environment variables as the example in Directly setting environment
variables, you would set the configMaps as follows:
configMaps:
my-config:
as: environment
items:
dbhost:
envVarName: DB_HOST
dbport:
envVarName: DB_PORT
In this configuration for the Helm Chart, we specify that we want to share the my-config ConfigMap as environment
variables with the main application container. Additionally, we want to map the dbhost config value to the DB_HOST
environment variable, and similarly map the dbport config value to the DB_PORT environment variable.
Note that like directly setting environment variables, these are set at container start time, and thus the containers
need to be restarted when the ConfigMap is updated for the new values to be propagated. You can use files instead if
you wish the ConfigMap changes to propagate immediately.
Accessing the ConfigMap as Files
You can mount the ConfigMap values as files on the container filesystem. To do so, you set the as attribute of the
configMaps input value to volume.
For example, suppose you wanted to share the my-config ConfigMap above as the files /etc/db/host and
/etc/db/port. For this case, you would set the configMaps input value to:
configMaps:
my-config:
as: volume
mountPath: /etc/db
items:
dbhost:
filePath: host
dbport:
filePath: port
In the container, now the values for dbhost is stored as a text file at the path /etc/db/host and dbport is stored
at the path /etc/db/port. You can then read these files in in your application to get the values.
Unlike environment variables, using files has the advantage of immediately reflecting changes to the ConfigMap. For
example, when you update my-config, the files at /etc/db are updated automatically with the new values, without
needing a redeployment to propagate the new values to the container.
Using Secrets
In general, it is discouraged to store sensitive information such as passwords in ConfigMaps. Instead, Kubernetes
provides Secrets as an alternative resource to store sensitive data. Similar to ConfigMaps, Secrets are key value
pairs that store configuration values that can be managed independently of the Pod and containers. However, unlike
ConfigMaps, Secrets have the following properties:
- A secret is only sent to a node if a pod on that node requires it. They are automatically garbage collected when there
are no more
Podsreferencing it on the node. - A secret is stored in
tmpfson the node, so that it is only available in memory. - Starting with Kubernetes 1.7, they can be encrypted at rest in
etcd(note: this feature was in alpha state until Kubernetes 1.13).
You can read more about the protections and risks of using Secrets in the official
documentation.
Creating a Secret is very similar to creating a ConfigMap. For example, suppose you had a Secret to store the
database password. Like ConfigMaps, you can create a Secret directly using kubectl:
kubectl create secret generic my-secret --from-literal=password=1f2d1e2e67df
The generic keyword indicates the Secret type. Almost all use cases for your application should use this type. Other
types include docker-registry for specifying credentials for accessing a private docker registry, and tls for
specifying TLS certificates to access the Kubernetes API.
You can also manage the Secret as code, although you may want to avoid this for Secrets to avoid leaking them in
unexpected locations (e.g source control). Unlike ConfigMaps, Secrets require values to be stored as base64 encoded
values when using resource files. So the configuration for the above example will be:
# my-secret.yaml
apiVersion: v1
kind: Secret
type: Opaque
metadata:
name: my-secret
data:
password: MWYyZDFlMmU2N2Rm
Note that MWYyZDFlMmU2N2Rm is the base 64 encoded version of 1f2d1e2e67df. You can then apply this resource config
using kubectl:
kubectl apply -f my-secret.yaml
Similar to ConfigMaps, this Helm Chart supports two ways to inject Secrets into the application container: as
environment variables, or as files. The syntax to share the values is very similar to the configMaps input value, only
you use the secrets input value. The properties of each approach is very similar to ConfigMaps. Refer to the
previous section for more details on each approach. Here, we show you examples of the input values
to use for each approach.
Mounting secrets as environment variables: In this example, we mount the my-secret Secret created above as the
environment variable DB_PASSWORD.
secrets:
my-secret:
as: environment
items:
password:
envVarName: DB_PASSWORD
Mounting secrets as files: In this example, we mount the my-secret Secret as the file /etc/db/password.
secrets:
my-secret:
as: volume
mountPath: /etc/db
items:
password:
filePath: password
NOTE: The volumes are different between secrets and configMaps. This means that if you use the same mountPath
for different secrets and config maps, you can end up with only one. It is undefined which Secret or ConfigMap ends
up getting mounted. To be safe, use a different mountPath for each one.
Which configuration method should I use?
Which configuration method you should use depends on your needs. Here is a summary of the pro and con of each approach:
Directly setting environment variables
Pro:
- Simple setup
- Manage configuration values directly with application deployment config
- Most application languages support looking up environment variables
Con:
- Tightly couple configuration settings with application deployment
- Requires redeployment to update values
- Must store in plain text, and easy to leak into VCS
Best for:
- Iterating different configuration values during development
- Sotring non-sensitive values that are unique to each environment / deployment
Using ConfigMaps
Pro:
- Keep config DRY by sharing a common set of configurations
- Independently update config values from the application deployment
- Automatically propagate new values when stored as files
Con:
- More overhead to manage the configuration
- Stored in plain text
- Available on all nodes automatically
Best for:
- Storing non-sensitive common configuration that are shared across environments
- Storing non-sensitive dynamic configuration values that change frequently
Using Secrets
Pro:
- All the benefits of using
ConfigMaps - Can be encrypted at rest
- Opaque by default when viewing the values (harder to remember base 64 encoded version of "admin")
- Only available to nodes that use it, and only in memory
Con:
- All the challenges of using
ConfigMaps - Configured in plain text, making it difficult to manage as code securely
- Less safe than using dedicated secrets manager / store like HashiCorp Vault.
Best for:
- Storing sensitive configuration values
How do you update the application to a new version?
To update the application to a new version, you can upgrade the Helm Release using updated values. For example, suppose
you deployed nginx version 1.15.4 using this Helm Chart with the following values:
containerImage:
repository: nginx
tag: 1.15.4
applicationName: nginx
In this example, we will further assume that you deployed this chart with the above values using the release name
edge-service, using a command similar to below:
$ helm install -f values.yaml --name edge-service gruntwork/k8s-service
Now let's try upgrading nginx to version 1.15.8. To do so, we will first update our values file:
containerImage:
repository: nginx
tag: 1.15.8
applicationName: nginx
The only difference here is the tag of the containerImage.
Next, we will upgrade our release using the updated values. To do so, we will use the helm upgrade command:
$ helm upgrade -f values.yaml edge-service gruntwork/k8s-service
This will update the created resources with the new values provided by the updated values.yaml file. For this example,
the only resource that will be updated is the Deployment resource, which will now have a new Pod spec that points to
nginx:1.15.8 as opposed to nginx:1.15.4. This automatically triggers a rolling deployment internally to Kubernetes,
which will launch new Pods using the latest image, and shut down old Pods once those are ready.
You can read more about how changes are rolled out on Deployment resources in the official
documentation.
Deployment resource. For example, updating the
applicationName will cause the Deployment resource to be deleted, and then created. This can lead to down time
because the resources are replaced in an uncontrolled fashion.
How do I create a canary deployment?
You may optionally configure a canary deployment of an arbitrary tag that will run as an individual deployment behind your configured service. This is useful for ensuring a new application tag runs without issues prior to fully rolling it out.
To configure a canary deployment, set canary.enabled = true and define the containerImage values. Typically, you will want to specify the tag of your next release candidate:
canary:
enabled: true
containerImage:
repository: nginx
tag: 1.15.9
Once deployed, your service will route traffic across both your stable and canary deployments, allowing you to monitor for and catch any issues early.
How do I verify my canary deployment?
Canary deployment pods have the same name as your stable deployment pods, with the additional -canary appended to the end, like so:
$ kubectl get pods -l "app.kubernetes.io/name=nginx,app.kubernetes.io/instance=edge-service"
NAME READY STATUS RESTARTS AGE
edge-service-nginx-844c978df7-f5wc4 1/1 Running 0 52s
edge-service-nginx-844c978df7-mln26 0/1 Pending 0 52s
edge-service-nginx-844c978df7-rdsr8 0/1 Pending 0 52s
edge-service-nginx-canary-844c978df7-bsr8 0/1 Pending 0 52s
Therefore, in this example, you could monitor your canary by running kubectl logs -f edge-service-nginx-canary-844c978df7-bsr8
How do I roll back a canary deployment?
Update your values.yaml file, setting canary.enabled = false and then upgrade your helm installation:
$ helm upgrade -f values.yaml edge-service gruntwork/k8s-service
Following this update, Kubernetes will determine that your canary deployment is no longer desired and will delete it.
How do I ensure a minimum number of Pods are available across node maintenance?
Sometimes, you may want to ensure that a specific number of Pods are always available during voluntary
maintenance.
This chart exposes an input value minPodsAvailable that can be used to specify a minimum number of Pods to maintain
during a voluntary maintenance activity. Under the hood, this chart will create a corresponding PodDisruptionBudget to
ensure that a certain number of Pods are up before attempting to terminate additional ones.
You can read more about PodDisruptionBudgets in our blog post covering the
topic
and in the official
documentation.
Why does the Pod have a preStop hook with a Shutdown Delay?
When a Pod is removed from a Kubernetes cluster, the control plane notifies all nodes to remove the Pod from
registered addresses. This includes removing the Pod from the list of available Pods to service a Service
endpoint. However, because Kubernetes is a distributed system, there is a delay between the shutdown sequence and the
Pod being removed from available addresses. As a result, the Pod could still get traffic despite it having already
been shutdown on the node it was running on.
Since there is no way to guarantee that the deletion has propagated across the cluster, we address this eventual
consistency issue by adding an arbitrary delay between the Pod being deleted and the initiation of the Pod shutdown
sequence. This is accomplished by adding a sleep command in the preStop hook.
You can control the length of time to delay with the shutdownDelay input value. You can also disable this behavior by
setting the shutdownDelay to 0.
You can read more about this topic in our blog post "Delaying Shutdown to Wait for Pod Deletion Propagation".
What is a sidecar container?
In Kubernetes, Pods are one or more tightly coupled containers that are deployed together. The containers in the Pod
share, amongst other things, the network stack, the IPC namespace, and in some cases the PID namespace. You can read
more about the resources that the containers in a Pod share in the official
documentation.
Sidecar Containers are additional containers that you wish to deploy in the Pod housing your application container.
This helm chart supports deploying these containers by configuring the sideCarContainers input value. This input value
is a map between the side car container name and the values of the container spec. The spec is rendered directly into
the Deployment resource, with the name being set to the key. For example:
sideCarContainers:
datadog:
image: datadog/agent:latest
env:
- name: DD_API_KEY
value: ASDF-1234
- name: SD_BACKEND
value: docker
nginx:
image: nginx:1.15.4
This input will be rendered in the Deployment resource as:
apiVersion: apps/v1
kind: Deployment
metadata:
... Snipped for brevity ...
spec:
... Snipped for brevity ...
template:
spec:
containers:
... The first entry relates to the application ...
- name: datadog
image: datadog/agent:latest
env:
- name: DD_API_KEY
value: ASDF-1234
- name: SD_BACKEND
value: docker
- name: nginx
image: nginx:1.15.4
In this config, the side car containers are rendered as additional containers to deploy alongside the main application
container configured by the containerImage, ports, livenessProbe, etc input values. Note that the
sideCarContainers variable directly renders the spec, meaning that the additional values for the side cars such as
livenessProbe should be rendered directly within the sideCarContainers input value.
How do I use a private registry?
To pull container images from a private registry, the Kubernetes cluster needs to be able to authenticate to the docker registry with a registry key. On managed Kubernetes clusters (e.g EKS, GKE, AKS), this is automated through the server IAM roles that are assigned to the instance VMs. In most cases, if the instance VM IAM role has the permissions to access the registry, the Kubernetes cluster will automatically be able to pull down images from the respective managed registry (e.g ECR on EKS or GCR on GKE).
Alternatively, you can specify docker registry keys in the Kubernetes cluster as Secret resources. This is helpful in
situations where you do not have the ability to assign registry access IAM roles to the node itself, or if you are
pulling images off of a different registry (e.g accessing GCR from EKS cluster).
You can use kubectl to create a Secret in Kubernetes that can be used as a docker registry key:
kubectl create secret docker-registry NAME \
--docker-server=DOCKER_REGISTRY_SERVER \
--docker-username=DOCKER_USER \
--docker-password=DOCKER_PASSWORD \
--docker-email=DOCKER_EMAIL
This command will create a Secret resource named NAME that holds the specified docker registry credentials. You can
then specify the cluster to use this Secret when pulling down images for the service Deployment in this chart by
using the imagePullSecrets input value:
imagePullSecrets:
- NAME
You can learn more about using private registries with Kubernetes in the official documentation.
Questions? Ask away.
We're here to talk about our services, answer any questions, give advice, or just to chat.
