all blog posts


Assigning Roles to EC2 Kubernetes Nodes at Boot

In this post we'll describe how to leverage EC2 user data to assign labels to EKS worker nodes at boot. This will allow us to assign Kubernetes services and pods to specific instance types.

The Problem

A common architecture for Amazon EKS looks like this:

Common EKS Architecture

If you haven't applied any manual changes, the worker nodes created by the auto scaling group will show up like this in Kubernetes:

→ kubectl get nodes        
NAME                                        STATUS    ROLES     AGE       VERSION
ip-10-0-11-134.eu-west-1.compute.internal   Ready     <none>    2m25s     v1.12.7
ip-10-0-21-228.eu-west-1.compute.internal   Ready     <none>    12s       v1.12.7

This is fine if you have only one type of container and that type can run on any node, for example a pod containing NginX and your public website.

But what happens if you have different containers with different hardware requirements? You can add a second auto scaling group with a different instance type, and the architecture would look like this:

Two ASG EKS Architecture

In the diagram and the AWS console it will be abundantly clear which node belongs to which auto scaling group, and what the instance type is:

Two ASG EKS Console

However, to Kubernetes the nodes look exactly the same, and any pod might be deployed on any node.

→ kubectl get nodes
NAME                                        STATUS    ROLES     AGE       VERSION
ip-10-0-11-134.eu-west-1.compute.internal   Ready     <none>    19m       v1.12.7
ip-10-0-21-208.eu-west-1.compute.internal   Ready     <none>    38s       v1.12.7
ip-10-0-21-228.eu-west-1.compute.internal   Ready     <none>    16m       v1.12.7

The Solution

The solution is to add labels to the nodes. A special label is the Role label, which is displayed in the kubectl get nodes command. You could add labels to the roles manually, but not only is this tedious, it's also a problem when the auto scaling group scales out. For example, I've manually added labels to the web nodes:

→ kubectl get nodes
NAME                                        STATUS    ROLES     AGE       VERSION
ip-10-0-11-134.eu-west-1.compute.internal   Ready     <none>    19m       v1.12.7
ip-10-0-21-208.eu-west-1.compute.internal   Ready     <none>    38s       v1.12.7
ip-10-0-21-228.eu-west-1.compute.internal   Ready     <none>    16m       v1.12.7

→ kubectl label nodes ip-10-0-11-134.eu-west-1.compute.internal kubernetes.io/role=web
node/ip-10-0-11-134.eu-west-1.compute.internal labeled

→ kubectl label nodes ip-10-0-21-228.eu-west-1.compute.internal kubernetes.io/role=web
node/ip-10-0-21-228.eu-west-1.compute.internal labeled

→ kubectl get nodes                                                                   
NAME                                        STATUS    ROLES     AGE       VERSION
ip-10-0-11-134.eu-west-1.compute.internal   Ready     web       28m       v1.12.7
ip-10-0-21-208.eu-west-1.compute.internal   Ready     <none>    10m       v1.12.7
ip-10-0-21-228.eu-west-1.compute.internal   Ready     web       26m       v1.12.7

But now the auto scaling group has added an instance and the nodes look like this:

→ kubectl get nodes
NAME                                        STATUS    ROLES     AGE       VERSION
ip-10-0-1-115.eu-west-1.compute.internal    Ready     <none>    20s       v1.12.7
ip-10-0-11-134.eu-west-1.compute.internal   Ready     web       31m       v1.12.7
ip-10-0-21-208.eu-west-1.compute.internal   Ready     <none>    12m       v1.12.7
ip-10-0-21-228.eu-west-1.compute.internal   Ready     web       28m       v1.12.7

So what we really want is that any node labels itself at boot. We will see how to do that in the next section.

Adding kubelet_extra_args to bootstrap

The auto scaling groups for the EKS worker nodes run a script called bootstrap.sh. This bootstrap registers the worker nodes with the master nodes. The user data containing the script is stored in a launch configuration. By default, it looks like this:

#!/bin/bash
set -o xtrace
/etc/eks/bootstrap.sh demo-cluster

To add labels at boot, update the launch configuration like this:

#!/bin/bash
set -o xtrace
/etc/eks/bootstrap.sh demo-cluster --kubelet-extra-args '--node-labels=kubernetes.io/role=memory'

When the auto scaling group boots instances with this launch configuration, the instances will have the label memory applied to them immediately.

→ kubectl get nodes
NAME                                        STATUS    ROLES     AGE       VERSION
ip-10-0-1-115.eu-west-1.compute.internal    Ready     web       17m       v1.12.7
ip-10-0-11-134.eu-west-1.compute.internal   Ready     web       48m       v1.12.7
ip-10-0-11-242.eu-west-1.compute.internal   Ready     memory    26s       v1.12.7
ip-10-0-21-228.eu-west-1.compute.internal   Ready     web       45m       v1.12.7

Other flags for kubelet-extra-args

The announcement blog post for the kubelet-extra-args functionality can be found here. The source code for bootstrap.sh can be found on GitHub.

In the bootstrap source code, you can see that the parameters you provide in kubelet-extra-args are actually passed to the kubelet command (hence the name). This leads us to the official documentation for kubelet, which shows all the flags we can add to it.

A notable example is -register-with-taints, which you could add like this:

#!/bin/bash
set -o xtrace
/etc/eks/bootstrap.sh demo-cluster --kubelet-extra-args '--register-with-taints=dedicated=memory:NoSchedule'

Adding multiple labels

Adding multiple labels to a node is also possible:

#!/bin/bash
set -o xtrace
/etc/eks/bootstrap.sh demo-cluster --kubelet-extra-args '--node-labels=kubernetes.io/role=memory,dedicated=memory'

Adding multiple flags

And finally, adding multiple flags and multiple labels:

#!/bin/bash
set -o xtrace
/etc/eks/bootstrap.sh demo-cluster --kubelet-extra-args '--node-labels=kubernetes.io/role=memory,dedicated=memory --register-with-taints=dedicated=memory:NoSchedule'

Conclusion

This blog post has shown how to use the kubelet-extra-args flags for /etc/eks/bootstrap.sh to apply very powerful features to EKS Worker nodes at boot. With these options, managing and provisioning your Kubernetes should be a lot easier.

If you have any questions or remarks, please reach out to me on Twitter.


Related blog posts


all blog posts