Karpenter node provisioning lab can fail under SCP due to c5.large in default NodePool

[w7089]

Workshop page

https://www.eksworkshop.com/docs/fundamentals/compute/karpenter/node-provisioning

What happened

While following the Karpenter node provisioning lab, the inflate pods stayed Pending and Karpenter did not provision nodes.

The cluster had a NodePool/default and EC2NodeClass/default, but the NodePool was not usable because the EC2NodeClass failed validation:

EC2NodeClass default: Ready=False
ValidationSucceeded=False
Reason=RunInstancesAuthCheckFailed
Message: Controller isn't authorized to call ec2:RunInstances

NodePool default: Ready=False
Reason=UnhealthyDependents
Message: NodeClassReady=False

Karpenter controller logs showed:

unauthorized to call ec2:RunInstances
... explicit deny in a service control policy ...
ignoring nodepool, not ready
no dynamic nodepools found

Decoding the authorization failure showed the SCP was denying invalid instance types. The account policy allowed patterns such as:

t*.nano
t*.micro
t*.small
t*.medium
t*.large
t*.xlarge
m*.medium
m*.large

But the workshop NodePool includes:

requirements:
- key: node.kubernetes.io/instance-type
  operator: In
  values:
  - c5.large
  - m5.large
  - r5.large
  - m5.xlarge

Karpenter attempted/validated c5.large, hit the SCP explicit deny, marked the EC2NodeClass not ready, and then ignored the whole NodePool instead of falling through to m5.large.

Workaround

Restricting the NodePool to m5.large made the EC2NodeClass and NodePool ready, created a NodeClaim, launched an EC2 instance, and scheduled the pending pods:

kubectl patch nodepool default --type=json \
  -p '[{"op":"replace","path":"/spec/template/spec/requirements/1/values","value":["m5.large"]}]'

After that:

nodepool/default      READY=True   NODES=1
ec2nodeclass/default  READY=True
nodeclaim/default-*   TYPE=m5.large READY=True
inflate-*             1/1 Running

Why this seems like a workshop issue

The lab assumes all listed instance types are allowed in the AWS environment. In environments with SCPs that constrain instance families/sizes, one denied candidate instance type can cause Karpenter's readiness validation to fail the entire NodePool, even when another listed type would be allowed.

A small note in the workshop, or using only broadly allowed workshop instance types, would make this easier to diagnose. For example, the page could mention checking EC2NodeClass conditions and adapting the NodePool instance-type list to match account/SCP restrictions.

Environment observed

• EKS: 1.33
• Karpenter controller image/chart: 1.9.0
• Region: us-west-2

[w7089]

Additional impact on the next consolidation lab: after working around the SCP by narrowing the NodePool to m5.large, the consolidation page cannot demonstrate replacement to c5.large.

The consolidation page expects that scaling inflate down to 1 replica lets Karpenter replace the remaining m5.large node with a cheaper c5.large. In this environment, c5.large is SCP-denied, so either:

1. keeping the original NodePool makes the EC2NodeClass fail validation with RunInstancesAuthCheckFailed, or
2. narrowing the NodePool to m5.large makes provisioning work, but consolidation naturally keeps m5.large because no smaller/cheaper allowed replacement remains in the NodePool.

Current observed state after the consolidation lab:

NodePool default requirements: node.kubernetes.io/instance-type In [m5.large]
Karpenter node: m5.large
inflate replicas: 1

So the consolidation behavior in the docs depends on c5.large being allowed by the account policy.