> ## Documentation Index
> Fetch the complete documentation index at: https://docs.milkstraw.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Upgrade EC2 instance types for Amazon EKS
> Step-by-step guide to upgrade EC2 instance types for an existing Amazon EKS cluster.
This guide walks you through upgrading EC2 instance types for an existing Amazon EKS cluster by creating a new managed node group, migrating workloads, and then tearing down the old group.
If you're coming from the MilkStraw AI recommender, you'll plug the recommended instance types into the steps below.
***
## Before you start
Make sure:
* You can log in to the AWS Console and have IAM permissions for **EKS**, **EC2**, and **IAM**.
* `kubectl` is installed and configured for the correct cluster:
```bash theme={null}
aws eks update-kubeconfig --region --name
```
* You know whether your workloads require:
* GPU nodes, and/or
* Spot capacity (only use Spot if you understand interruption behavior and have workloads that tolerate it).
* If you use **public subnets** for nodes:
* `MapPublicIpOnLaunch=true` is set.
* Subnets have the standard EKS tags:
* `kubernetes.io/cluster/ = shared`
* `kubernetes.io/role/elb = 1` (for public subnets).
***
## Step 1 · Create a new node group with the target instance type
You'll first add a new node group using the new instance type(s). This lets you migrate workloads safely before deleting the old group.
### 1.1 Open the EKS cluster and add a node group
In the AWS Console:
1. Go to **Amazon EKS** → **Clusters**.
2. Select your cluster.
3. Open the **Compute** tab.
4. Choose **Add node group**.
### 1.2 Basics
Configure:
* **Name**
Use something descriptive and versioned, for example:
```text theme={null}
production-core-ng-v2
```
A good pattern is: `env-project-function-version`.
* **Node IAM role**
Choose the role with standard EKS worker node permissions, or create one if needed.
* **(Optional) Launch template**
Use a launch template if you need:
* Custom AMI or user data
* Specific disk types / encryption settings
* Extra EC2-level configuration
* **(Optional) Labels / Taints / Tags**
* Example label:
```text theme={null}
nodepool = new
```
* Example taint (useful for GPU-only workloads or to keep pods off until you're ready):
```text theme={null}
key = no-scheduling
effect = NoSchedule
```
### 1.3 Compute configuration
Under **Compute configuration**:
* **AMI type**
* General-purpose: **Amazon Linux 2** or **AL2023**.
AL2023 is newer and has different defaults; validate compatibility for your workloads.
* GPU: **Amazon Linux 2 GPU** (or a Bottlerocket GPU variant if you already use Bottlerocket).
* **Capacity type**
* **On-Demand** for predictable capacity.
* **Spot** only if your workloads tolerate interruption and you have a proper Spot strategy.
* **Instance types**
* Example general-purpose instance:
```text theme={null}
m6a.xlarge
```
* If you're using **MilkStraw AI recommender**, set this to the **recommended instance type(s)** from your report.
* **Disk size**
* For most applications: **50–100 GiB** root volume is sufficient.
* Increase if pods make heavy use of `emptyDir` volumes or local caching.
* **Scaling**
* Set **Desired**, **Min**, and **Max**.
For example:
```text theme={null}
Desired capacity: 2
Min size: 1
Max size: 10
```
* If you run Cluster Autoscaler, ensure these values work with your expected scale range.
### 1.4 Networking
Under **Networking**:
* **Subnets**
* Select the subnets where you want worker nodes to live.
* For public subnets, confirm:
* `MapPublicIpOnLaunch=true`
* Proper EKS tags as mentioned above.
* **(Optional) SSH key pair**
* Add an SSH key only if you need direct SSH access to nodes.
### 1.5 Create and verify node readiness
Create the node group and wait for nodes to join the cluster.
Watch nodes until they are `Ready`:
```bash theme={null}
kubectl get nodes --watch
```
#### GPU clusters
If these are **GPU nodes**, install the NVIDIA device plugin after the nodes are `Ready`.
In the NVIDIA `k8s-device-plugin` GitHub repo, locate the latest **DaemonSet** manifest and apply it:
```bash theme={null}
kubectl apply -f
```
***
## Step 2 · Move workloads to the new node group
You now have both old and new node groups attached to the cluster. The goal is to:
1. Stop new pods from landing on the old nodes.
2. Let the scheduler and Cluster Autoscaler move workloads to the new group.
3. Drain and empty the old nodes.
The examples below assume you're using **Managed Node Groups**.
### 2.1 Taint the old node group (EKS-managed taint)
First, add a taint to the old node group to prevent **new pods** from scheduling on it while existing pods continue running:
```bash theme={null}
aws eks update-nodegroup-config \
--cluster-name \
--nodegroup-name \
--taints addOrUpdateTaints='[{key=no-scheduling,effect=NO_SCHEDULE}]'
```
### 2.2 Cordon the old nodes
Cordoning marks nodes as unschedulable at the Kubernetes level:
```bash theme={null}
kubectl get nodes -l eks.amazonaws.com/nodegroup= -o name \
| xargs -n1 kubectl cordon
```
New pods will no longer be placed on these nodes.
### 2.3 Drain the old nodes
Drain nodes to evict pods safely and move them to the new node group:
```bash theme={null}
kubectl drain \
--ignore-daemonsets \
--delete-emptydir-data \
--grace-period=60 \
--timeout=10m
```
Flags:
* `--ignore-daemonsets`
DaemonSet pods are not evicted by `drain`. They are terminated when the node is deleted.
* `--delete-emptydir-data`
Only use if pods don't rely on data inside `emptyDir` volumes, as that data will be lost.
Additional considerations:
* **PodDisruptionBudgets (PDBs)**
If PDBs are strict, `kubectl drain` may block until disruptions are allowed.
You might temporarily increase `maxUnavailable` for smoother migrations.
* **Unmanaged pods**
If some pods are not controlled by a Deployment, ReplicaSet, or DaemonSet, you may need `--force`. Use this carefully and only if you understand the impact.
### 2.4 Watch workloads reschedule
Monitor pods as they are rescheduled onto the new node group:
```bash theme={null}
kubectl get pods -A -o wide --watch
```
If you set a label on the **new** node group (for example `nodepool=new`), make sure workloads can land there:
* Add a matching `nodeSelector` or node affinity to workloads that must move:
```yaml theme={null}
spec:
template:
spec:
nodeSelector:
nodepool: new
```
* Or skip tainting the new group so the scheduler naturally prefers it once old nodes are cordoned and drained.
If you run **Cluster Autoscaler**, it should automatically scale the new node group to fit the evicted pods.
### 2.5 Validate migration
Confirm the migration before deleting anything:
* All pods are `Running` and scheduled on **new nodes**:
```bash theme={null}
kubectl get pods -A -o wide
```
* All nodes in the **old** node group:
* Are `SchedulingDisabled`, and
* Either have no non-DaemonSet pods left or are fully drained.
***
## Step 3 · Remove the old node group
Once you are confident workloads are stable on the new instance types, remove the old group.
In the AWS Console:
1. Go to **Amazon EKS** → **Clusters** → select your cluster.
2. Open the **Compute** tab → **Node groups**.
3. Select the **old node group**.
4. Click **Delete**.
5. Type the node group name to confirm, then **Delete**.
If you are using **Cluster Autoscaler**:
* Make sure the old node group is no longer in its configuration, **or**
* Temporarily set its Desired/Min capacity to `0` before you delete it.
***
## Rollback · Move workloads back to the old node group
If something goes wrong after migration (for example, performance or compatibility issues), you can quickly roll workloads back to the old node group.
The high-level flow:
1. Scale the old group back up.
2. Pause scheduling onto the new group.
3. Re-enable scheduling on the old group.
4. Drain the new nodes so pods move back.
You keep the new group around until you decide next steps.
### Scale the old node group
In EC2 or EKS:
* Set the old node group's or Auto Scaling Group's **Desired/Min/Max** back to healthy values.
Example (Auto Scaling Group via console or CLI):
* Desired: previous steady-state value.
* Min: match Desired.
* Max: your upper bound.
### Pause scheduling on the new group
Add a taint to the **new** node group:
```bash theme={null}
aws eks update-nodegroup-config \
--cluster-name \
--nodegroup-name \
--taints addOrUpdateTaints='[{key=no-scheduling,effect=NO_SCHEDULE}]'
```
Cordon all new nodes:
```bash theme={null}
kubectl get nodes -l eks.amazonaws.com/nodegroup= -o name \
| xargs -n1 kubectl cordon
```
### Re-enable scheduling on the old group
Remove the taint from the **old** node group:
```bash theme={null}
aws eks update-nodegroup-config \
--cluster-name \
--nodegroup-name \
--taints removeTaints='[{key=no-scheduling,effect=NO_SCHEDULE}]'
```
Uncordon old nodes so they accept new pods again:
```bash theme={null}
kubectl get nodes -l eks.amazonaws.com/nodegroup= -o name \
| xargs -n1 kubectl uncordon
```
### Drain the new nodes
Now drain the **new** nodes so pods move back to the old group:
```bash theme={null}
kubectl drain \
--ignore-daemonsets \
--delete-emptydir-data \
--grace-period=60 \
--timeout=10m
```
Watch pods move back:
```bash theme={null}
kubectl get pods -A -o wide --watch
```
Once you stabilize and understand the root cause, you can try another migration with updated instance types or configuration.
***
## Extra resources
* **Install kubectl** — [Kubernetes documentation: Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/)
* **NVIDIA device plugin for Kubernetes** — [NVIDIA/k8s-device-plugin on GitHub](https://github.com/NVIDIA/k8s-device-plugin)
* **VPC and subnet requirements for EKS** — [Amazon EKS VPC and subnet requirements](https://docs.aws.amazon.com/eks/latest/userguide/network_reqs.html)