Skip to content

osdc: replicate -opt node-packing optimizations into the regular runner/nodepool defs#902

Open
jeanschmidt wants to merge 2 commits into
gh/jeanschmidt/82/basefrom
gh/jeanschmidt/82/head
Open

osdc: replicate -opt node-packing optimizations into the regular runner/nodepool defs#902
jeanschmidt wants to merge 2 commits into
gh/jeanschmidt/82/basefrom
gh/jeanschmidt/82/head

Conversation

@jeanschmidt

@jeanschmidt jeanschmidt commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Stack from ghstack (oldest at bottom):

Impact: every cluster on the regular arc-runners/nodepools modules (all staging + meta-prod-aws-ue2, lf-prod-aws-ue1/ue2, and meta-prod-aws-uw1 nodepools); no cluster is redeployed by this change
Risk: medium

What

Copy the packing optimizations that until now lived only in the -opt
modules into the regular arc-runners/nodepools defs: 21 runner defs gain
an explicit node_fleet on a right-sized, smaller instance type; nodepools
gain 11 per-size tier fleets, the c7i-runner pool switches to Graviton, and
the 6 superseded monolithic family fleets (c7a, c7i, m7g, m7i, m8g, r7i) are
removed. The -opt modules are left untouched here.

Why

Only meta-prod-aws-ue1 runs the -opt modules today; the pilot showed a
cluster-wide vCPUhours / CIhousr gain (≈14%) by packing each runner onto a
size-matched node tier instead of stranding a small job on a huge instance.
Landing the same defs in the regular modules is the first step of rolling
that win out fleet-wide.

[ghstack-poisoned]
@github-actions

github-actions Bot commented Jul 15, 2026

Copy link
Copy Markdown

Capacity report

commit 8c011ff1 · run log

✅ simulate-cluster
Installed 1 package in 56ms
�[1mMonte Carlo Cluster Simulation�[0m
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Seed: 42  |  MAPE threshold: 15%  |  Runners: 44  |  DaemonSets: 16
Peak target runner types: 30 (mapped from 38 old labels)

�[1m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━�[0m
�[1m�[0;36mCluster Simulation Results�[0m
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

�[1;33mSkipped labels (1):�[0m
  �[2ml-arm64g2-6-32: no runner def�[0m

�[1mNodes by instance type:�[0m

  Instance Type          Nodes  vCPU Used vCPU Total   Mem Used  Mem Total   GPU
  ──────────────────────────────────────────────────────────────────────────────
  c7a.12xlarge             772   27488.4c   36554.2c  53154.6Gi  65472.2Gi     -
  c7a.2xlarge                6      27.8c      44.7c     54.1Gi     76.2Gi     -
  c7a.48xlarge               2     188.6c     382.0c    379.0Gi    691.6Gi     -
  c7a.4xlarge             2054   17089.3c   31693.2c  33911.1Gi  52599.7Gi     -
  c7i.metal-24xl            37    3415.8c    3523.5c   6197.9Gi   6223.6Gi     -
  g4dn.12xlarge            162    7341.8c    7654.5c  27622.6Gi  27800.6Gi 648/648
  g4dn.8xlarge              89    2609.5c    2784.8c  10280.4Gi  10306.1Gi 89/89
  g4dn.metal                87    8205.8c    8276.3c  29624.3Gi  29737.0Gi 696/696
  g5.12xlarge               49    2220.7c    2315.2c   8110.0Gi   8144.0Gi 196/196
  g5.48xlarge               41    7762.1c    7826.5c  28720.9Gi  28749.8Gi 328/328
  g5.8xlarge               603   17680.0c   18867.9c  68446.4Gi  68686.6Gi 603/603
  g6.12xlarge               24    1087.7c    1134.0c   4092.2Gi   4118.6Gi 96/96
  g6.8xlarge               377   11053.6c   11796.3c  42793.2Gi  42943.4Gi 377/377
  m6i.32xlarge              26    3258.3c    3305.9c  11661.3Gi  12072.3Gi     -
  m7g.8xlarge               31     934.5c     973.1c   3447.1Gi   3546.7Gi     -
  m7g.metal                 30    1869.6c    1899.3c   6795.3Gi   6823.8Gi     -
  m7i.12xlarge             201    8192.0c    9517.4c  32363.6Gi  34895.3Gi     -
  m8g.8xlarge               34    1026.4c    1067.3c   3786.2Gi   3889.9Gi     -
  r7a.12xlarge             402   18415.0c   19034.7c 139543.9Gi 141185.6Gi     -
  r7a.16xlarge               4     216.3c     253.2c   1677.3Gi   1857.3Gi     -
  r7a.48xlarge              22    2075.0c    4201.8c  16291.2Gi  31050.9Gi     -
  r7g.16xlarge             122    7481.0c    7723.8c  54718.2Gi  56646.7Gi     -

�[1mDeployment accuracy:�[0m

  Total deployed: 6208 / 7294 target
  Weighted MAPE: 15.0%

  Runner                              Deployed   Target     Diff
  ───────────────────────────────────────────────────────────────
  �[1;33ml-arm64g3-16-62                           61       76      -15�[0m
  �[1;33ml-arm64g3-61-463                         122      153      -31�[0m
  �[0;32ml-arm64g4-16-62                           67       76       -9�[0m
  �[1;33ml-barm64g3-62-226                         30       39       -9�[0m
  �[1;33ml-bx86iamx-92-167                         37       45       -8�[0m
  �[0;32ml-bx86iavx512-94-344-t4-8                 87       91       -4�[0m
  �[0;32ml-x86aavx2-189-704-a10g-8                 41       42       -1�[0m
  �[0;32ml-x86aavx2-29-113-a10g                   603      695      -92�[0m
  �[0;32ml-x86aavx2-29-113-l4                     377      422      -45�[0m
  �[1;33ml-x86aavx2-45-167-a10g-4                  49       80      -31�[0m
  �[1;33ml-x86aavx2-45-172-l4-4                    24       29       -5�[0m
  �[0;32ml-x86aavx512-125-463                      26       24       +2�[0m
  �[1;33ml-x86iamx-32-128                         130      174      -44�[0m
  �[0;32ml-x86iamx-8-32                           354      384      -30�[0m
  �[1;33ml-x86iavx2-40-160                         22       30       -8�[0m
  �[0;32ml-x86iavx2-8-32                           19       18       +1�[0m
  �[1;33ml-x86iavx512-16-128                       68       89      -21�[0m
  �[1;33ml-x86iavx512-16-32                      1146     1384     -238�[0m
  �[1;33ml-x86iavx512-2-4                          12       15       -3�[0m
  �[0;32ml-x86iavx512-29-115-t4                    89      104      -15�[0m
  �[0;32ml-x86iavx512-32-256                       13       12       +1�[0m
  �[1;33ml-x86iavx512-37-68                        48       65      -17�[0m
  �[0;32ml-x86iavx512-45-172-t4-4                 162      183      -21�[0m
  �[1;33ml-x86iavx512-46-85                       151      189      -38�[0m
  �[0;32ml-x86iavx512-48-384                      366      417      -51�[0m
  �[0;32ml-x86iavx512-8-16                       2054     2400     -346�[0m
  �[0;32ml-x86iavx512-8-64                         26       28       -2�[0m
  �[0;32ml-x86iavx512-94-192                        2        2       +0�[0m
  �[1;33ml-x86iavx512-94-768                       22       28       -6�[0m

�[1mCluster-wide utilization:�[0m

  �[0;32mvCPU:    82.8%�[0m  (149640 / 180830 cores)
  �[0;32mMemory:  91.6%�[0m  (583671 / 637518 GiB)
  �[0;32mGPU:    100.0%�[0m  (3033 / 3033 GPUs across 1432 nodes)

  Total nodes: 5175
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ analyze-utilization
Installed 1 package in 1ms
�[1mNode Utilization Analysis�[0m
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Runner def dirs: /home/runner/work/ci-infra/ci-infra/osdc/modules/arc-runners/defs, /home/runner/work/ci-infra/ci-infra/osdc/modules/arc-runners-b200/defs, /home/runner/work/ci-infra/ci-infra/osdc/modules/arc-runners-h100/defs, /home/runner/work/ci-infra/ci-infra/osdc/modules/arc-runners-opt/defs
NodePool def dirs: /home/runner/work/ci-infra/ci-infra/osdc/modules/nodepools/defs, /home/runner/work/ci-infra/ci-infra/osdc/modules/nodepools-b200/defs, /home/runner/work/ci-infra/ci-infra/osdc/modules/nodepools-h100/defs
Utilization threshold: 90.0%

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: c7a.12xlarge�[0m
  Total: 48 vCPU, 96Gi advertised (88.8Gi actual)
  Kubelet reserved: 190m CPU, 2.9Gi RAM
  DaemonSet overhead: 460m CPU, 1.1Gi RAM
  �[0;32mAllocatable for runners: 47350m CPU (47.4 cores), 84.8Gi RAM�[0m

  �[1mRunners targeting this node:�[0m
    - l-x86iavx512-16-32: 16320m CPU, 32.5Gi RAM (job: 16c+32.0Gi, hooks: 320m+522Mi)
    - l-x86iavx512-37-68: 37320m CPU, 68.5Gi RAM (job: 37c+68.0Gi, hooks: 320m+522Mi)
    - l-x86iavx512-46-85: 46320m CPU, 83.5Gi RAM (job: 46c+83.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;31ml-x86iavx512-16-32�[0m: 2 pods
      CPU:  68.9% (32640m / 47350m) waste: 14710m (14.7 cores)
      MEM:  76.7% (65.0Gi / 84.8Gi) waste: 19.8Gi
      Bottleneck: CPU
    �[0;31ml-x86iavx512-37-68�[0m: 1 pods
      CPU:  78.8% (37320m / 47350m) waste: 10030m (10.0 cores)
      MEM:  80.8% (68.5Gi / 84.8Gi) waste: 16.3Gi
      Bottleneck: CPU
    �[0;32ml-x86iavx512-46-85�[0m: 1 pods
      CPU:  97.8% (46320m / 47350m) waste: 1030m (1.0 cores)
      MEM:  98.5% (83.5Gi / 84.8Gi) waste: 1.3Gi
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 3

    �[0;32mTop 3 most efficient:�[0m
      �[0;32m#1�[0m [1xl-x86iavx512-46-85]
         CPU:  97.8%  MEM:  98.5%  waste: 1.0c + 1.3Gi
      �[0;31m#2�[0m [1xl-x86iavx512-37-68]
         CPU:  78.8%  MEM:  80.8%  waste: 10.0c + 16.3Gi
      �[0;31m#3�[0m [2xl-x86iavx512-16-32]
         CPU:  68.9%  MEM:  76.7%  waste: 14.7c + 19.8Gi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: c7a.2xlarge�[0m
  Total: 8 vCPU, 16Gi advertised (14.8Gi actual)
  Kubelet reserved: 90m CPU, 993Mi RAM
  DaemonSet overhead: 460m CPU, 1.1Gi RAM
  �[0;32mAllocatable for runners: 7450m CPU (7.5 cores), 12.7Gi RAM�[0m

  �[1mRunners targeting this node:�[0m
    - l-x86iavx512-2-4: 2320m CPU, 4.5Gi RAM (job: 2c+4.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;31ml-x86iavx512-2-4�[0m: 2 pods
      CPU:  62.3% (4640m / 7450m) waste: 2810m (2.8 cores)
      MEM:  71.0% (9.0Gi / 12.7Gi) waste: 3.7Gi
      Bottleneck: MEM

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 1

    �[0;32mTop 1 most efficient:�[0m
      �[0;31m#1�[0m [2xl-x86iavx512-2-4]
         CPU:  62.3%  MEM:  71.0%  waste: 2.8c + 3.7Gi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: c7a.48xlarge�[0m
  Total: 192 vCPU, 384Gi advertised (355.2Gi actual)
  Kubelet reserved: 550m CPU, 8.3Gi RAM
  DaemonSet overhead: 460m CPU, 1.1Gi RAM
  �[0;32mAllocatable for runners: 190990m CPU (191.0 cores), 345.8Gi RAM�[0m

  �[1mRunners targeting this node:�[0m
    - l-x86iavx512-94-192: 94320m CPU, 189.5Gi RAM (job: 94c+189.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;31ml-x86iavx512-94-192�[0m: 1 pods
      CPU:  49.4% (94320m / 190990m) waste: 96670m (96.7 cores)
      MEM:  54.8% (189.5Gi / 345.8Gi) waste: 156.3Gi
      Bottleneck: MEM

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 1

    �[0;32mTop 1 most efficient:�[0m
      �[0;31m#1�[0m [1xl-x86iavx512-94-192]
         CPU:  49.4%  MEM:  54.8%  waste: 96.7c + 156.3Gi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: c7a.4xlarge�[0m
  Total: 16 vCPU, 32Gi advertised (29.6Gi actual)
  Kubelet reserved: 110m CPU, 2.9Gi RAM
  DaemonSet overhead: 460m CPU, 1.1Gi RAM
  �[0;32mAllocatable for runners: 15430m CPU (15.4 cores), 25.6Gi RAM�[0m

  �[1mRunners targeting this node:�[0m
    - l-x86iavx512-8-16: 8320m CPU, 16.5Gi RAM (job: 8c+16.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;31ml-x86iavx512-8-16�[0m: 1 pods
      CPU:  53.9% (8320m / 15430m) waste: 7110m (7.1 cores)
      MEM:  64.5% (16.5Gi / 25.6Gi) waste: 9.1Gi
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 1

    �[0;32mTop 1 most efficient:�[0m
      �[0;31m#1�[0m [1xl-x86iavx512-8-16]
         CPU:  53.9%  MEM:  64.5%  waste: 7.1c + 9.1Gi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: c7i.12xlarge�[0m
  Total: 48 vCPU, 96Gi advertised (88.8Gi actual)
  Kubelet reserved: 190m CPU, 2.9Gi RAM
  DaemonSet overhead: 460m CPU, 1.1Gi RAM
  �[0;32mAllocatable for runners: 47350m CPU (47.4 cores), 84.8Gi RAM�[0m

  �[1mRunners targeting this node:�[0m
    - l-x86iamx-14-27: 14320m CPU, 27.5Gi RAM (job: 14c+27.0Gi, hooks: 320m+522Mi)
    - l-x86iamx-22-41: 22320m CPU, 41.5Gi RAM (job: 22c+41.0Gi, hooks: 320m+522Mi)
    - l-x86iamx-46-84: 46320m CPU, 83.5Gi RAM (job: 46c+83.0Gi, hooks: 320m+522Mi)
    - l-x86iamx-8-16: 8320m CPU, 16.5Gi RAM (job: 8c+16.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;32ml-x86iamx-14-27�[0m: 3 pods
      CPU:  90.7% (42960m / 47350m) waste: 4390m (4.4 cores)
      MEM:  97.3% (82.5Gi / 84.8Gi) waste: 2.3Gi
      Bottleneck: CPU
    �[0;32ml-x86iamx-22-41�[0m: 2 pods
      CPU:  94.3% (44640m / 47350m) waste: 2710m (2.7 cores)
      MEM:  97.9% (83.0Gi / 84.8Gi) waste: 1.8Gi
      Bottleneck: CPU
    �[0;32ml-x86iamx-46-84�[0m: 1 pods
      CPU:  97.8% (46320m / 47350m) waste: 1030m (1.0 cores)
      MEM:  98.5% (83.5Gi / 84.8Gi) waste: 1.3Gi
      Bottleneck: CPU
    �[1;33ml-x86iamx-8-16�[0m: 5 pods
      CPU:  87.9% (41600m / 47350m) waste: 5750m (5.8 cores)
      MEM:  97.3% (82.5Gi / 84.8Gi) waste: 2.3Gi
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 8

    �[0;32mTop 5 most efficient:�[0m
      �[0;32m#1�[0m [1xl-x86iamx-46-84]
         CPU:  97.8%  MEM:  98.5%  waste: 1.0c + 1.3Gi
      �[0;32m#2�[0m [2xl-x86iamx-22-41]
         CPU:  94.3%  MEM:  97.9%  waste: 2.7c + 1.8Gi
      �[0;32m#3�[0m [3xl-x86iamx-14-27]
         CPU:  90.7%  MEM:  97.3%  waste: 4.4c + 2.3Gi
      �[1;33m#4�[0m [5xl-x86iamx-8-16]
         CPU:  87.9%  MEM:  97.3%  waste: 5.8c + 2.3Gi
      �[1;33m#5�[0m [1xl-x86iamx-14-27, 3xl-x86iamx-8-16]
         CPU:  83.0%  MEM:  90.8%  waste: 8.1c + 7.8Gi

    �[0;31mBottom 3 least efficient (money on the table):�[0m
      �[1;33m#1�[0m [1xl-x86iamx-22-41, 2xl-x86iamx-8-16]
         CPU:  82.3%  MEM:  87.9%  waste: 8.4c + 10.3Gi
      �[0;31m#2�[0m [2xl-x86iamx-14-27, 1xl-x86iamx-8-16]
         CPU:  78.1%  MEM:  84.3%  waste: 10.4c + 13.3Gi
      �[0;31m#3�[0m [1xl-x86iamx-14-27, 1xl-x86iamx-22-41]
         CPU:  77.4%  MEM:  81.4%  waste: 10.7c + 15.8Gi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: c7i.metal-24xl�[0m
  Total: 96 vCPU, 192Gi advertised (177.6Gi actual)
  Kubelet reserved: 310m CPU, 8.3Gi RAM
  DaemonSet overhead: 460m CPU, 1.1Gi RAM
  �[0;32mAllocatable for runners: 95230m CPU (95.2 cores), 168.2Gi RAM�[0m

  �[1mRunners targeting this node:�[0m
    - l-bx86iamx-92-167: 92320m CPU, 167.5Gi RAM (job: 92c+167.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;32ml-bx86iamx-92-167�[0m: 1 pods
      CPU:  96.9% (92320m / 95230m) waste: 2910m (2.9 cores)
      MEM:  99.6% (167.5Gi / 168.2Gi) waste: 712Mi
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 1

    �[0;32mTop 1 most efficient:�[0m
      �[0;32m#1�[0m [1xl-bx86iamx-92-167]
         CPU:  96.9%  MEM:  99.6%  waste: 2.9c + 712Mi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: g4dn.12xlarge�[0m
  Total: 48 vCPU, 192Gi advertised (177.6Gi actual), 4 GPU
  Kubelet reserved: 190m CPU, 2.9Gi RAM
  DaemonSet overhead: 560m CPU, 3.1Gi RAM
  �[0;32mAllocatable for runners: 47250m CPU (47.2 cores), 171.6Gi RAM, 4 GPU�[0m

  �[1mRunners targeting this node:�[0m
    - l-x86iavx512-45-172-t4-4: 45320m CPU, 170.5Gi RAM, 4 GPU (job: 45c+170.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;32ml-x86iavx512-45-172-t4-4�[0m: 1 pods
      CPU:  95.9% (45320m / 47250m) waste: 1930m (1.9 cores)
      MEM:  99.4% (170.5Gi / 171.6Gi) waste: 1.1Gi
      GPU: 100.0% (4 / 4)
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 1

    �[0;32mTop 1 most efficient:�[0m
      �[0;32m#1�[0m [1xl-x86iavx512-45-172-t4-4]
         CPU:  95.9%  MEM:  99.4%  GPU: 100.0%  waste: 1.9c + 1.1Gi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: g4dn.8xlarge�[0m
  Total: 32 vCPU, 128Gi advertised (118.4Gi actual), 1 GPU
  Kubelet reserved: 150m CPU, 993Mi RAM
  DaemonSet overhead: 560m CPU, 1.6Gi RAM
  �[0;32mAllocatable for runners: 31290m CPU (31.3 cores), 115.8Gi RAM, 1 GPU�[0m

  �[1mRunners targeting this node:�[0m
    - l-x86iavx512-29-115-t4: 29320m CPU, 115.5Gi RAM, 1 GPU (job: 29c+115.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;32ml-x86iavx512-29-115-t4�[0m: 1 pods
      CPU:  93.7% (29320m / 31290m) waste: 1970m (2.0 cores)
      MEM:  99.8% (115.5Gi / 115.8Gi) waste: 296Mi
      GPU: 100.0% (1 / 1)
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 1

    �[0;32mTop 1 most efficient:�[0m
      �[0;32m#1�[0m [1xl-x86iavx512-29-115-t4]
         CPU:  93.7%  MEM:  99.8%  GPU: 100.0%  waste: 2.0c + 296Mi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: g4dn.metal�[0m
  Total: 96 vCPU, 384Gi advertised (355.2Gi actual), 8 GPU
  Kubelet reserved: 310m CPU, 8.3Gi RAM
  DaemonSet overhead: 560m CPU, 5.1Gi RAM
  �[0;32mAllocatable for runners: 95130m CPU (95.1 cores), 341.8Gi RAM, 8 GPU�[0m

  �[1mRunners targeting this node:�[0m
    - l-bx86iavx512-94-344-t4-8: 94320m CPU, 340.5Gi RAM, 8 GPU (job: 94c+340.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;32ml-bx86iavx512-94-344-t4-8�[0m: 1 pods
      CPU:  99.1% (94320m / 95130m) waste: 810m (0.8 cores)
      MEM:  99.6% (340.5Gi / 341.8Gi) waste: 1.3Gi
      GPU: 100.0% (8 / 8)
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 1

    �[0;32mTop 1 most efficient:�[0m
      �[0;32m#1�[0m [1xl-bx86iavx512-94-344-t4-8]
         CPU:  99.1%  MEM:  99.6%  GPU: 100.0%  waste: 0.8c + 1.3Gi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: g5.12xlarge�[0m
  Total: 48 vCPU, 192Gi advertised (177.6Gi actual), 4 GPU
  Kubelet reserved: 190m CPU, 8.3Gi RAM
  DaemonSet overhead: 560m CPU, 3.1Gi RAM
  �[0;32mAllocatable for runners: 47250m CPU (47.2 cores), 166.2Gi RAM, 4 GPU�[0m

  �[1mRunners targeting this node:�[0m
    - l-x86aavx2-45-167-a10g-4: 45320m CPU, 165.5Gi RAM, 4 GPU (job: 45c+165.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;32ml-x86aavx2-45-167-a10g-4�[0m: 1 pods
      CPU:  95.9% (45320m / 47250m) waste: 1930m (1.9 cores)
      MEM:  99.6% (165.5Gi / 166.2Gi) waste: 712Mi
      GPU: 100.0% (4 / 4)
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 1

    �[0;32mTop 1 most efficient:�[0m
      �[0;32m#1�[0m [1xl-x86aavx2-45-167-a10g-4]
         CPU:  95.9%  MEM:  99.6%  GPU: 100.0%  waste: 1.9c + 712Mi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: g5.48xlarge�[0m
  Total: 192 vCPU, 768Gi advertised (710.4Gi actual), 8 GPU
  Kubelet reserved: 550m CPU, 4.1Gi RAM
  DaemonSet overhead: 560m CPU, 5.1Gi RAM
  �[0;32mAllocatable for runners: 190890m CPU (190.9 cores), 701.2Gi RAM, 8 GPU�[0m

  �[1mRunners targeting this node:�[0m
    - l-x86aavx2-189-704-a10g-8: 189320m CPU, 700.5Gi RAM, 8 GPU (job: 189c+700.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;32ml-x86aavx2-189-704-a10g-8�[0m: 1 pods
      CPU:  99.2% (189320m / 190890m) waste: 1570m (1.6 cores)
      MEM:  99.9% (700.5Gi / 701.2Gi) waste: 723Mi
      GPU: 100.0% (8 / 8)
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 1

    �[0;32mTop 1 most efficient:�[0m
      �[0;32m#1�[0m [1xl-x86aavx2-189-704-a10g-8]
         CPU:  99.2%  MEM:  99.9%  GPU: 100.0%  waste: 1.6c + 723Mi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: g5.8xlarge�[0m
  Total: 32 vCPU, 128Gi advertised (118.4Gi actual), 1 GPU
  Kubelet reserved: 150m CPU, 2.9Gi RAM
  DaemonSet overhead: 560m CPU, 1.6Gi RAM
  �[0;32mAllocatable for runners: 31290m CPU (31.3 cores), 113.9Gi RAM, 1 GPU�[0m

  �[1mRunners targeting this node:�[0m
    - l-x86aavx2-29-113-a10g: 29320m CPU, 113.5Gi RAM, 1 GPU (job: 29c+113.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;32ml-x86aavx2-29-113-a10g�[0m: 1 pods
      CPU:  93.7% (29320m / 31290m) waste: 1970m (2.0 cores)
      MEM:  99.7% (113.5Gi / 113.9Gi) waste: 408Mi
      GPU: 100.0% (1 / 1)
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 1

    �[0;32mTop 1 most efficient:�[0m
      �[0;32m#1�[0m [1xl-x86aavx2-29-113-a10g]
         CPU:  93.7%  MEM:  99.7%  GPU: 100.0%  waste: 2.0c + 408Mi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: g6.12xlarge�[0m
  Total: 48 vCPU, 192Gi advertised (177.6Gi actual), 4 GPU
  Kubelet reserved: 190m CPU, 2.9Gi RAM
  DaemonSet overhead: 560m CPU, 3.1Gi RAM
  �[0;32mAllocatable for runners: 47250m CPU (47.2 cores), 171.6Gi RAM, 4 GPU�[0m

  �[1mRunners targeting this node:�[0m
    - l-x86aavx2-45-172-l4-4: 45320m CPU, 170.5Gi RAM, 4 GPU (job: 45c+170.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;32ml-x86aavx2-45-172-l4-4�[0m: 1 pods
      CPU:  95.9% (45320m / 47250m) waste: 1930m (1.9 cores)
      MEM:  99.4% (170.5Gi / 171.6Gi) waste: 1.1Gi
      GPU: 100.0% (4 / 4)
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 1

    �[0;32mTop 1 most efficient:�[0m
      �[0;32m#1�[0m [1xl-x86aavx2-45-172-l4-4]
         CPU:  95.9%  MEM:  99.4%  GPU: 100.0%  waste: 1.9c + 1.1Gi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: g6.8xlarge�[0m
  Total: 32 vCPU, 128Gi advertised (118.4Gi actual), 1 GPU
  Kubelet reserved: 150m CPU, 2.9Gi RAM
  DaemonSet overhead: 560m CPU, 1.6Gi RAM
  �[0;32mAllocatable for runners: 31290m CPU (31.3 cores), 113.9Gi RAM, 1 GPU�[0m

  �[1mRunners targeting this node:�[0m
    - l-x86aavx2-29-113-l4: 29320m CPU, 113.5Gi RAM, 1 GPU (job: 29c+113.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;32ml-x86aavx2-29-113-l4�[0m: 1 pods
      CPU:  93.7% (29320m / 31290m) waste: 1970m (2.0 cores)
      MEM:  99.7% (113.5Gi / 113.9Gi) waste: 408Mi
      GPU: 100.0% (1 / 1)
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 1

    �[0;32mTop 1 most efficient:�[0m
      �[0;32m#1�[0m [1xl-x86aavx2-29-113-l4]
         CPU:  93.7%  MEM:  99.7%  GPU: 100.0%  waste: 2.0c + 408Mi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: m6i.32xlarge�[0m
  Total: 128 vCPU, 512Gi advertised (473.7Gi actual)
  Kubelet reserved: 390m CPU, 8.3Gi RAM
  DaemonSet overhead: 460m CPU, 1.1Gi RAM
  �[0;32mAllocatable for runners: 127150m CPU (127.2 cores), 464.3Gi RAM�[0m

  �[1mRunners targeting this node:�[0m
    - l-x86aavx512-125-463: 125320m CPU, 448.5Gi RAM (job: 125c+448.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;32ml-x86aavx512-125-463�[0m: 1 pods
      CPU:  98.6% (125320m / 127150m) waste: 1830m (1.8 cores)
      MEM:  96.6% (448.5Gi / 464.3Gi) waste: 15.8Gi
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 1

    �[0;32mTop 1 most efficient:�[0m
      �[0;32m#1�[0m [1xl-x86aavx512-125-463]
         CPU:  98.6%  MEM:  96.6%  waste: 1.8c + 15.8Gi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: m7g.8xlarge�[0m
  Total: 32 vCPU, 128Gi advertised (118.4Gi actual)
  Kubelet reserved: 150m CPU, 2.9Gi RAM
  DaemonSet overhead: 460m CPU, 1.1Gi RAM
  �[0;32mAllocatable for runners: 31390m CPU (31.4 cores), 114.4Gi RAM�[0m

  �[1mRunners targeting this node:�[0m
    - l-arm64g3-16-62: 15320m CPU, 56.5Gi RAM (job: 15c+56.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;32ml-arm64g3-16-62�[0m: 2 pods
      CPU:  97.6% (30640m / 31390m) waste: 750m (0.8 cores)
      MEM:  98.8% (113.0Gi / 114.4Gi) waste: 1.4Gi
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 1

    �[0;32mTop 1 most efficient:�[0m
      �[0;32m#1�[0m [2xl-arm64g3-16-62]
         CPU:  97.6%  MEM:  98.8%  waste: 0.8c + 1.4Gi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: m7g.metal�[0m
  Total: 64 vCPU, 256Gi advertised (236.9Gi actual)
  Kubelet reserved: 230m CPU, 8.3Gi RAM
  DaemonSet overhead: 460m CPU, 1.1Gi RAM
  �[0;32mAllocatable for runners: 63310m CPU (63.3 cores), 227.5Gi RAM�[0m

  �[1mRunners targeting this node:�[0m
    - l-barm64g3-62-226: 62320m CPU, 226.5Gi RAM (job: 62c+226.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;32ml-barm64g3-62-226�[0m: 1 pods
      CPU:  98.4% (62320m / 63310m) waste: 990m (1.0 cores)
      MEM:  99.6% (226.5Gi / 227.5Gi) waste: 974Mi
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 1

    �[0;32mTop 1 most efficient:�[0m
      �[0;32m#1�[0m [1xl-barm64g3-62-226]
         CPU:  98.4%  MEM:  99.6%  waste: 1.0c + 974Mi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: m7i.12xlarge�[0m
  Total: 48 vCPU, 192Gi advertised (177.6Gi actual)
  Kubelet reserved: 190m CPU, 2.9Gi RAM
  DaemonSet overhead: 460m CPU, 1.1Gi RAM
  �[0;32mAllocatable for runners: 47350m CPU (47.4 cores), 173.6Gi RAM�[0m

  �[1mRunners targeting this node:�[0m
    - l-x86iamx-32-128: 32320m CPU, 128.5Gi RAM (job: 32c+128.0Gi, hooks: 320m+522Mi)
    - l-x86iamx-8-32: 8320m CPU, 32.5Gi RAM (job: 8c+32.0Gi, hooks: 320m+522Mi)
    - l-x86iavx2-40-160: 40320m CPU, 160.5Gi RAM (job: 40c+160.0Gi, hooks: 320m+522Mi)
    - l-x86iavx2-8-32: 8320m CPU, 32.5Gi RAM (job: 8c+32.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;31ml-x86iamx-32-128�[0m: 1 pods
      CPU:  68.3% (32320m / 47350m) waste: 15030m (15.0 cores)
      MEM:  74.0% (128.5Gi / 173.6Gi) waste: 45.1Gi
      Bottleneck: CPU
    �[1;33ml-x86iamx-8-32�[0m: 5 pods
      CPU:  87.9% (41600m / 47350m) waste: 5750m (5.8 cores)
      MEM:  93.6% (162.5Gi / 173.6Gi) waste: 11.1Gi
      Bottleneck: CPU
    �[1;33ml-x86iavx2-40-160�[0m: 1 pods
      CPU:  85.2% (40320m / 47350m) waste: 7030m (7.0 cores)
      MEM:  92.5% (160.5Gi / 173.6Gi) waste: 13.1Gi
      Bottleneck: CPU
    �[1;33ml-x86iavx2-8-32�[0m: 5 pods
      CPU:  87.9% (41600m / 47350m) waste: 5750m (5.8 cores)
      MEM:  93.6% (162.5Gi / 173.6Gi) waste: 11.1Gi
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 9

    �[0;32mTop 5 most efficient:�[0m
      �[1;33m#1�[0m [5xl-x86iamx-8-32]
         CPU:  87.9%  MEM:  93.6%  waste: 5.8c + 11.1Gi
      �[1;33m#2�[0m [4xl-x86iamx-8-32, 1xl-x86iavx2-8-32]
         CPU:  87.9%  MEM:  93.6%  waste: 5.8c + 11.1Gi
      �[1;33m#3�[0m [3xl-x86iamx-8-32, 2xl-x86iavx2-8-32]
         CPU:  87.9%  MEM:  93.6%  waste: 5.8c + 11.1Gi
      �[1;33m#4�[0m [2xl-x86iamx-8-32, 3xl-x86iavx2-8-32]
         CPU:  87.9%  MEM:  93.6%  waste: 5.8c + 11.1Gi
      �[1;33m#5�[0m [1xl-x86iamx-8-32, 4xl-x86iavx2-8-32]
         CPU:  87.9%  MEM:  93.6%  waste: 5.8c + 11.1Gi

    �[0;31mBottom 4 least efficient (money on the table):�[0m
      �[1;33m#1�[0m [5xl-x86iavx2-8-32]
         CPU:  87.9%  MEM:  93.6%  waste: 5.8c + 11.1Gi
      �[1;33m#2�[0m [1xl-x86iamx-32-128, 1xl-x86iamx-8-32]
         CPU:  85.8%  MEM:  92.7%  waste: 6.7c + 12.6Gi
      �[1;33m#3�[0m [1xl-x86iamx-32-128, 1xl-x86iavx2-8-32]
         CPU:  85.8%  MEM:  92.7%  waste: 6.7c + 12.6Gi
      �[1;33m#4�[0m [1xl-x86iavx2-40-160]
         CPU:  85.2%  MEM:  92.5%  waste: 7.0c + 13.1Gi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: m8g.16xlarge�[0m
  Total: 64 vCPU, 256Gi advertised (236.9Gi actual)
  Kubelet reserved: 230m CPU, 8.3Gi RAM
  DaemonSet overhead: 460m CPU, 1.1Gi RAM
  �[0;32mAllocatable for runners: 63310m CPU (63.3 cores), 227.5Gi RAM�[0m

  �[1mRunners targeting this node:�[0m
    - l-barm64g4-62-226: 62320m CPU, 226.5Gi RAM (job: 62c+226.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;32ml-barm64g4-62-226�[0m: 1 pods
      CPU:  98.4% (62320m / 63310m) waste: 990m (1.0 cores)
      MEM:  99.6% (226.5Gi / 227.5Gi) waste: 974Mi
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 1

    �[0;32mTop 1 most efficient:�[0m
      �[0;32m#1�[0m [1xl-barm64g4-62-226]
         CPU:  98.4%  MEM:  99.6%  waste: 1.0c + 974Mi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: m8g.24xlarge�[0m
  Total: 96 vCPU, 384Gi advertised (355.2Gi actual)
  Kubelet reserved: 310m CPU, 8.3Gi RAM
  DaemonSet overhead: 460m CPU, 1.1Gi RAM
  �[0;32mAllocatable for runners: 95230m CPU (95.2 cores), 345.8Gi RAM�[0m

  �[1mRunners targeting this node:�[0m
    - l-barm64g4-94-344: 94320m CPU, 344.5Gi RAM (job: 94c+344.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;32ml-barm64g4-94-344�[0m: 1 pods
      CPU:  99.0% (94320m / 95230m) waste: 910m (0.9 cores)
      MEM:  99.6% (344.5Gi / 345.8Gi) waste: 1.3Gi
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 1

    �[0;32mTop 1 most efficient:�[0m
      �[0;32m#1�[0m [1xl-barm64g4-94-344]
         CPU:  99.0%  MEM:  99.6%  waste: 0.9c + 1.3Gi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: m8g.48xlarge�[0m
  Total: 192 vCPU, 768Gi advertised (710.4Gi actual)
  Kubelet reserved: 550m CPU, 8.3Gi RAM
  DaemonSet overhead: 460m CPU, 1.1Gi RAM
  �[0;32mAllocatable for runners: 190990m CPU (191.0 cores), 701.0Gi RAM�[0m

  �[1mRunners targeting this node:�[0m
    - rel-l-arm64g4-16-62: 16320m CPU, 62.5Gi RAM (job: 16c+62.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;32mrel-l-arm64g4-16-62�[0m: 11 pods
      CPU:  94.0% (179520m / 190990m) waste: 11470m (11.5 cores)
      MEM:  98.1% (687.6Gi / 701.0Gi) waste: 13.4Gi
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 1

    �[0;32mTop 1 most efficient:�[0m
      �[0;32m#1�[0m [11xrel-l-arm64g4-16-62]
         CPU:  94.0%  MEM:  98.1%  waste: 11.5c + 13.4Gi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: m8g.8xlarge�[0m
  Total: 32 vCPU, 128Gi advertised (118.4Gi actual)
  Kubelet reserved: 150m CPU, 2.9Gi RAM
  DaemonSet overhead: 460m CPU, 1.1Gi RAM
  �[0;32mAllocatable for runners: 31390m CPU (31.4 cores), 114.4Gi RAM�[0m

  �[1mRunners targeting this node:�[0m
    - l-arm64g4-16-62: 15320m CPU, 56.5Gi RAM (job: 15c+56.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;32ml-arm64g4-16-62�[0m: 2 pods
      CPU:  97.6% (30640m / 31390m) waste: 750m (0.8 cores)
      MEM:  98.8% (113.0Gi / 114.4Gi) waste: 1.4Gi
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 1

    �[0;32mTop 1 most efficient:�[0m
      �[0;32m#1�[0m [2xl-arm64g4-16-62]
         CPU:  97.6%  MEM:  98.8%  waste: 0.8c + 1.4Gi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: p4d.24xlarge�[0m
  Total: 96 vCPU, 1152Gi advertised (1065.2Gi actual), 8 GPU
  Kubelet reserved: 310m CPU, 3.0Gi RAM
  DaemonSet overhead: 560m CPU, 5.1Gi RAM
  �[0;32mAllocatable for runners: 95130m CPU (95.1 cores), 1057.0Gi RAM, 8 GPU�[0m

  �[1mRunners targeting this node:�[0m
    - l-bx86iavx512-88-1000-a100-8: 88320m CPU, 1000.5Gi RAM, 8 GPU (job: 88c+1000.0Gi, hooks: 320m+522Mi)
    - l-x86iavx512-11-125-a100: 11320m CPU, 125.5Gi RAM, 1 GPU (job: 11c+125.0Gi, hooks: 320m+522Mi)
    - l-x86iavx512-22-250-a100-2: 22320m CPU, 250.5Gi RAM, 2 GPU (job: 22c+250.0Gi, hooks: 320m+522Mi)
    - l-x86iavx512-44-500-a100-4: 44320m CPU, 500.5Gi RAM, 4 GPU (job: 44c+500.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;32ml-bx86iavx512-88-1000-a100-8�[0m: 1 pods
      CPU:  92.8% (88320m / 95130m) waste: 6810m (6.8 cores)
      MEM:  94.7% (1000.5Gi / 1057.0Gi) waste: 56.5Gi
      GPU: 100.0% (8 / 8)
      Bottleneck: CPU
    �[0;32ml-x86iavx512-11-125-a100�[0m: 8 pods
      CPU:  95.2% (90560m / 95130m) waste: 4570m (4.6 cores)
      MEM:  95.0% (1004.1Gi / 1057.0Gi) waste: 53.0Gi
      GPU: 100.0% (8 / 8)
      Bottleneck: CPU
    �[0;32ml-x86iavx512-22-250-a100-2�[0m: 4 pods
      CPU:  93.9% (89280m / 95130m) waste: 5850m (5.8 cores)
      MEM:  94.8% (1002.0Gi / 1057.0Gi) waste: 55.0Gi
      GPU: 100.0% (8 / 8)
      Bottleneck: CPU
    �[0;32ml-x86iavx512-44-500-a100-4�[0m: 2 pods
      CPU:  93.2% (88640m / 95130m) waste: 6490m (6.5 cores)
      MEM:  94.7% (1001.0Gi / 1057.0Gi) waste: 56.0Gi
      GPU: 100.0% (8 / 8)
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 10

    �[0;32mTop 5 most efficient:�[0m
      �[0;32m#1�[0m [8xl-x86iavx512-11-125-a100]
         CPU:  95.2%  MEM:  95.0%  GPU: 100.0%  waste: 4.6c + 53.0Gi
      �[0;32m#2�[0m [6xl-x86iavx512-11-125-a100, 1xl-x86iavx512-22-250-a100-2]
         CPU:  94.9%  MEM:  94.9%  GPU: 100.0%  waste: 4.9c + 53.5Gi
      �[0;32m#3�[0m [4xl-x86iavx512-11-125-a100, 2xl-x86iavx512-22-250-a100-2]
         CPU:  94.5%  MEM:  94.9%  GPU: 100.0%  waste: 5.2c + 54.0Gi
      �[0;32m#4�[0m [4xl-x86iavx512-11-125-a100, 1xl-x86iavx512-44-500-a100-4]
         CPU:  94.2%  MEM:  94.8%  GPU: 100.0%  waste: 5.5c + 54.5Gi
      �[0;32m#5�[0m [2xl-x86iavx512-11-125-a100, 3xl-x86iavx512-22-250-a100-2]
         CPU:  94.2%  MEM:  94.8%  GPU: 100.0%  waste: 5.5c + 54.5Gi

    �[0;31mBottom 5 least efficient (money on the table):�[0m
      �[0;32m#1�[0m [2xl-x86iavx512-11-125-a100, 1xl-x86iavx512-22-250-a100-2, 1xl-x86iavx512-44-500-a100-4]
         CPU:  93.9%  MEM:  94.8%  GPU: 100.0%  waste: 5.8c + 55.0Gi
      �[0;32m#2�[0m [4xl-x86iavx512-22-250-a100-2]
         CPU:  93.9%  MEM:  94.8%  GPU: 100.0%  waste: 5.8c + 55.0Gi
      �[0;32m#3�[0m [2xl-x86iavx512-22-250-a100-2, 1xl-x86iavx512-44-500-a100-4]
         CPU:  93.5%  MEM:  94.7%  GPU: 100.0%  waste: 6.2c + 55.5Gi
      �[0;32m#4�[0m [2xl-x86iavx512-44-500-a100-4]
         CPU:  93.2%  MEM:  94.7%  GPU: 100.0%  waste: 6.5c + 56.0Gi
      �[0;32m#5�[0m [1xl-bx86iavx512-88-1000-a100-8]
         CPU:  92.8%  MEM:  94.7%  GPU: 100.0%  waste: 6.8c + 56.5Gi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: p5.48xlarge�[0m
  Total: 192 vCPU, 2048Gi advertised (1894.4Gi actual), 8 GPU
  Kubelet reserved: 550m CPU, 2.5Gi RAM
  DaemonSet overhead: 560m CPU, 5.1Gi RAM
  �[0;32mAllocatable for runners: 190890m CPU (190.9 cores), 1886.8Gi RAM, 8 GPU�[0m

  �[1mRunners targeting this node:�[0m
    - l-bx86iamx-176-1800-h100-8: 176320m CPU, 1800.5Gi RAM, 8 GPU (job: 176c+1800.0Gi, hooks: 320m+522Mi)
    - l-x86iamx-22-225-h100: 22320m CPU, 225.5Gi RAM, 1 GPU (job: 22c+225.0Gi, hooks: 320m+522Mi)
    - l-x86iamx-44-450-h100-2: 44320m CPU, 450.5Gi RAM, 2 GPU (job: 44c+450.0Gi, hooks: 320m+522Mi)
    - l-x86iamx-88-900-h100-4: 88320m CPU, 900.5Gi RAM, 4 GPU (job: 88c+900.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;32ml-bx86iamx-176-1800-h100-8�[0m: 1 pods
      CPU:  92.4% (176320m / 190890m) waste: 14570m (14.6 cores)
      MEM:  95.4% (1800.5Gi / 1886.8Gi) waste: 86.3Gi
      GPU: 100.0% (8 / 8)
      Bottleneck: CPU
    �[0;32ml-x86iamx-22-225-h100�[0m: 8 pods
      CPU:  93.5% (178560m / 190890m) waste: 12330m (12.3 cores)
      MEM:  95.6% (1804.1Gi / 1886.8Gi) waste: 82.7Gi
      GPU: 100.0% (8 / 8)
      Bottleneck: CPU
    �[0;32ml-x86iamx-44-450-h100-2�[0m: 4 pods
      CPU:  92.9% (177280m / 190890m) waste: 13610m (13.6 cores)
      MEM:  95.5% (1802.0Gi / 1886.8Gi) waste: 84.8Gi
      GPU: 100.0% (8 / 8)
      Bottleneck: CPU
    �[0;32ml-x86iamx-88-900-h100-4�[0m: 2 pods
      CPU:  92.5% (176640m / 190890m) waste: 14250m (14.2 cores)
      MEM:  95.5% (1801.0Gi / 1886.8Gi) waste: 85.8Gi
      GPU: 100.0% (8 / 8)
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 10

    �[0;32mTop 5 most efficient:�[0m
      �[0;32m#1�[0m [8xl-x86iamx-22-225-h100]
         CPU:  93.5%  MEM:  95.6%  GPU: 100.0%  waste: 12.3c + 82.7Gi
      �[0;32m#2�[0m [6xl-x86iamx-22-225-h100, 1xl-x86iamx-44-450-h100-2]
         CPU:  93.4%  MEM:  95.6%  GPU: 100.0%  waste: 12.7c + 83.2Gi
      �[0;32m#3�[0m [4xl-x86iamx-22-225-h100, 2xl-x86iamx-44-450-h100-2]
         CPU:  93.2%  MEM:  95.6%  GPU: 100.0%  waste: 13.0c + 83.7Gi
      �[0;32m#4�[0m [4xl-x86iamx-22-225-h100, 1xl-x86iamx-88-900-h100-4]
         CPU:  93.0%  MEM:  95.5%  GPU: 100.0%  waste: 13.3c + 84.2Gi
      �[0;32m#5�[0m [2xl-x86iamx-22-225-h100, 3xl-x86iamx-44-450-h100-2]
         CPU:  93.0%  MEM:  95.5%  GPU: 100.0%  waste: 13.3c + 84.2Gi

    �[0;31mBottom 5 least efficient (money on the table):�[0m
      �[0;32m#1�[0m [2xl-x86iamx-22-225-h100, 1xl-x86iamx-44-450-h100-2, 1xl-x86iamx-88-900-h100-4]
         CPU:  92.9%  MEM:  95.5%  GPU: 100.0%  waste: 13.6c + 84.8Gi
      �[0;32m#2�[0m [4xl-x86iamx-44-450-h100-2]
         CPU:  92.9%  MEM:  95.5%  GPU: 100.0%  waste: 13.6c + 84.8Gi
      �[0;32m#3�[0m [2xl-x86iamx-44-450-h100-2, 1xl-x86iamx-88-900-h100-4]
         CPU:  92.7%  MEM:  95.5%  GPU: 100.0%  waste: 13.9c + 85.3Gi
      �[0;32m#4�[0m [2xl-x86iamx-88-900-h100-4]
         CPU:  92.5%  MEM:  95.5%  GPU: 100.0%  waste: 14.2c + 85.8Gi
      �[0;32m#5�[0m [1xl-bx86iamx-176-1800-h100-8]
         CPU:  92.4%  MEM:  95.4%  GPU: 100.0%  waste: 14.6c + 86.3Gi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: p6-b200.48xlarge�[0m
  Total: 192 vCPU, 2048Gi advertised (1894.4Gi actual), 8 GPU
  Kubelet reserved: 550m CPU, 2.5Gi RAM
  DaemonSet overhead: 560m CPU, 5.1Gi RAM
  �[0;32mAllocatable for runners: 190890m CPU (190.9 cores), 1886.8Gi RAM, 8 GPU�[0m

  �[1mRunners targeting this node:�[0m
    - l-bx86iamx-176-1800-b200-8: 176320m CPU, 1800.5Gi RAM, 8 GPU (job: 176c+1800.0Gi, hooks: 320m+522Mi)
    - l-x86iamx-22-225-b200: 22320m CPU, 225.5Gi RAM, 1 GPU (job: 22c+225.0Gi, hooks: 320m+522Mi)
    - l-x86iamx-44-450-b200-2: 44320m CPU, 450.5Gi RAM, 2 GPU (job: 44c+450.0Gi, hooks: 320m+522Mi)
    - l-x86iamx-88-900-b200-4: 88320m CPU, 900.5Gi RAM, 4 GPU (job: 88c+900.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;32ml-bx86iamx-176-1800-b200-8�[0m: 1 pods
      CPU:  92.4% (176320m / 190890m) waste: 14570m (14.6 cores)
      MEM:  95.4% (1800.5Gi / 1886.8Gi) waste: 86.3Gi
      GPU: 100.0% (8 / 8)
      Bottleneck: CPU
    �[0;32ml-x86iamx-22-225-b200�[0m: 8 pods
      CPU:  93.5% (178560m / 190890m) waste: 12330m (12.3 cores)
      MEM:  95.6% (1804.1Gi / 1886.8Gi) waste: 82.7Gi
      GPU: 100.0% (8 / 8)
      Bottleneck: CPU
    �[0;32ml-x86iamx-44-450-b200-2�[0m: 4 pods
      CPU:  92.9% (177280m / 190890m) waste: 13610m (13.6 cores)
      MEM:  95.5% (1802.0Gi / 1886.8Gi) waste: 84.8Gi
      GPU: 100.0% (8 / 8)
      Bottleneck: CPU
    �[0;32ml-x86iamx-88-900-b200-4�[0m: 2 pods
      CPU:  92.5% (176640m / 190890m) waste: 14250m (14.2 cores)
      MEM:  95.5% (1801.0Gi / 1886.8Gi) waste: 85.8Gi
      GPU: 100.0% (8 / 8)
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 10

    �[0;32mTop 5 most efficient:�[0m
      �[0;32m#1�[0m [8xl-x86iamx-22-225-b200]
         CPU:  93.5%  MEM:  95.6%  GPU: 100.0%  waste: 12.3c + 82.7Gi
      �[0;32m#2�[0m [6xl-x86iamx-22-225-b200, 1xl-x86iamx-44-450-b200-2]
         CPU:  93.4%  MEM:  95.6%  GPU: 100.0%  waste: 12.7c + 83.2Gi
      �[0;32m#3�[0m [4xl-x86iamx-22-225-b200, 2xl-x86iamx-44-450-b200-2]
         CPU:  93.2%  MEM:  95.6%  GPU: 100.0%  waste: 13.0c + 83.7Gi
      �[0;32m#4�[0m [4xl-x86iamx-22-225-b200, 1xl-x86iamx-88-900-b200-4]
         CPU:  93.0%  MEM:  95.5%  GPU: 100.0%  waste: 13.3c + 84.2Gi
      �[0;32m#5�[0m [2xl-x86iamx-22-225-b200, 3xl-x86iamx-44-450-b200-2]
         CPU:  93.0%  MEM:  95.5%  GPU: 100.0%  waste: 13.3c + 84.2Gi

    �[0;31mBottom 5 least efficient (money on the table):�[0m
      �[0;32m#1�[0m [2xl-x86iamx-22-225-b200, 1xl-x86iamx-44-450-b200-2, 1xl-x86iamx-88-900-b200-4]
         CPU:  92.9%  MEM:  95.5%  GPU: 100.0%  waste: 13.6c + 84.8Gi
      �[0;32m#2�[0m [4xl-x86iamx-44-450-b200-2]
         CPU:  92.9%  MEM:  95.5%  GPU: 100.0%  waste: 13.6c + 84.8Gi
      �[0;32m#3�[0m [2xl-x86iamx-44-450-b200-2, 1xl-x86iamx-88-900-b200-4]
         CPU:  92.7%  MEM:  95.5%  GPU: 100.0%  waste: 13.9c + 85.3Gi
      �[0;32m#4�[0m [2xl-x86iamx-88-900-b200-4]
         CPU:  92.5%  MEM:  95.5%  GPU: 100.0%  waste: 14.2c + 85.8Gi
      �[0;32m#5�[0m [1xl-bx86iamx-176-1800-b200-8]
         CPU:  92.4%  MEM:  95.4%  GPU: 100.0%  waste: 14.6c + 86.3Gi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: r7a.12xlarge�[0m
  Total: 48 vCPU, 384Gi advertised (355.2Gi actual)
  Kubelet reserved: 190m CPU, 2.9Gi RAM
  DaemonSet overhead: 460m CPU, 1.1Gi RAM
  �[0;32mAllocatable for runners: 47350m CPU (47.4 cores), 351.2Gi RAM�[0m

  �[1mRunners targeting this node:�[0m
    - l-x86iavx512-16-128: 15320m CPU, 116.5Gi RAM (job: 15c+116.0Gi, hooks: 320m+522Mi)
    - l-x86iavx512-32-256: 32320m CPU, 256.5Gi RAM (job: 32c+256.0Gi, hooks: 320m+522Mi)
    - l-x86iavx512-48-384: 46320m CPU, 350.5Gi RAM (job: 46c+350.0Gi, hooks: 320m+522Mi)
    - rel-l-x86iavx512-44-340: 44320m CPU, 340.5Gi RAM (job: 44c+340.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;32ml-x86iavx512-16-128�[0m: 3 pods
      CPU:  97.1% (45960m / 47350m) waste: 1390m (1.4 cores)
      MEM:  99.5% (349.5Gi / 351.2Gi) waste: 1.7Gi
      Bottleneck: CPU
    �[0;31ml-x86iavx512-32-256�[0m: 1 pods
      CPU:  68.3% (32320m / 47350m) waste: 15030m (15.0 cores)
      MEM:  73.0% (256.5Gi / 351.2Gi) waste: 94.7Gi
      Bottleneck: CPU
    �[0;32ml-x86iavx512-48-384�[0m: 1 pods
      CPU:  97.8% (46320m / 47350m) waste: 1030m (1.0 cores)
      MEM:  99.8% (350.5Gi / 351.2Gi) waste: 715Mi
      Bottleneck: CPU
    �[0;32mrel-l-x86iavx512-44-340�[0m: 1 pods
      CPU:  93.6% (44320m / 47350m) waste: 3030m (3.0 cores)
      MEM:  97.0% (340.5Gi / 351.2Gi) waste: 10.7Gi
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 4

    �[0;32mTop 4 most efficient:�[0m
      �[0;32m#1�[0m [1xl-x86iavx512-48-384]
         CPU:  97.8%  MEM:  99.8%  waste: 1.0c + 715Mi
      �[0;32m#2�[0m [3xl-x86iavx512-16-128]
         CPU:  97.1%  MEM:  99.5%  waste: 1.4c + 1.7Gi
      �[0;32m#3�[0m [1xrel-l-x86iavx512-44-340]
         CPU:  93.6%  MEM:  97.0%  waste: 3.0c + 10.7Gi
      �[0;31m#4�[0m [1xl-x86iavx512-32-256]
         CPU:  68.3%  MEM:  73.0%  waste: 15.0c + 94.7Gi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: r7a.16xlarge�[0m
  Total: 64 vCPU, 512Gi advertised (473.7Gi actual)
  Kubelet reserved: 230m CPU, 8.3Gi RAM
  DaemonSet overhead: 460m CPU, 1.1Gi RAM
  �[0;32mAllocatable for runners: 63310m CPU (63.3 cores), 464.3Gi RAM�[0m

  �[1mRunners targeting this node:�[0m
    - l-x86iavx512-8-64: 8320m CPU, 64.5Gi RAM (job: 8c+64.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;32ml-x86iavx512-8-64�[0m: 7 pods
      CPU:  92.0% (58240m / 63310m) waste: 5070m (5.1 cores)
      MEM:  97.3% (451.6Gi / 464.3Gi) waste: 12.7Gi
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 1

    �[0;32mTop 1 most efficient:�[0m
      �[0;32m#1�[0m [7xl-x86iavx512-8-64]
         CPU:  92.0%  MEM:  97.3%  waste: 5.1c + 12.7Gi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: r7a.48xlarge�[0m
  Total: 192 vCPU, 1536Gi advertised (1420.8Gi actual)
  Kubelet reserved: 550m CPU, 8.3Gi RAM
  DaemonSet overhead: 460m CPU, 1.1Gi RAM
  �[0;32mAllocatable for runners: 190990m CPU (191.0 cores), 1411.4Gi RAM�[0m

  �[1mRunners targeting this node:�[0m
    - l-x86iavx512-94-768: 94320m CPU, 740.5Gi RAM (job: 94c+740.0Gi, hooks: 320m+522Mi)
    - rel-l-x86iavx512-8-64: 8320m CPU, 64.5Gi RAM (job: 8c+64.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;31ml-x86iavx512-94-768�[0m: 1 pods
      CPU:  49.4% (94320m / 190990m) waste: 96670m (96.7 cores)
      MEM:  52.5% (740.5Gi / 1411.4Gi) waste: 670.9Gi
      Bottleneck: MEM
    �[0;32mrel-l-x86iavx512-8-64�[0m: 21 pods
      CPU:  91.5% (174720m / 190990m) waste: 16270m (16.3 cores)
      MEM:  96.0% (1354.7Gi / 1411.4Gi) waste: 56.7Gi
      Bottleneck: MEM

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 1

    �[0;32mTop 1 most efficient:�[0m
      �[0;32m#1�[0m [1xl-x86iavx512-94-768, 10xrel-l-x86iavx512-8-64]
         CPU:  92.9%  MEM:  98.2%  waste: 13.5c + 25.8Gi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: r7g.12xlarge�[0m
  Total: 48 vCPU, 384Gi advertised (355.2Gi actual)
  Kubelet reserved: 190m CPU, 2.9Gi RAM
  DaemonSet overhead: 460m CPU, 1.1Gi RAM
  �[0;32mAllocatable for runners: 47350m CPU (47.4 cores), 351.2Gi RAM�[0m

  �[1mRunners targeting this node:�[0m
    - rel-l-arm64g3-44-340: 44320m CPU, 340.5Gi RAM (job: 44c+340.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;32mrel-l-arm64g3-44-340�[0m: 1 pods
      CPU:  93.6% (44320m / 47350m) waste: 3030m (3.0 cores)
      MEM:  97.0% (340.5Gi / 351.2Gi) waste: 10.7Gi
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 1

    �[0;32mTop 1 most efficient:�[0m
      �[0;32m#1�[0m [1xrel-l-arm64g3-44-340]
         CPU:  93.6%  MEM:  97.0%  waste: 3.0c + 10.7Gi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: r7g.16xlarge�[0m
  Total: 64 vCPU, 512Gi advertised (473.7Gi actual)
  Kubelet reserved: 230m CPU, 8.3Gi RAM
  DaemonSet overhead: 460m CPU, 1.1Gi RAM
  �[0;32mAllocatable for runners: 63310m CPU (63.3 cores), 464.3Gi RAM�[0m

  �[1mRunners targeting this node:�[0m
    - l-arm64g3-61-463: 61320m CPU, 448.5Gi RAM (job: 61c+448.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;32ml-arm64g3-61-463�[0m: 1 pods
      CPU:  96.9% (61320m / 63310m) waste: 1990m (2.0 cores)
      MEM:  96.6% (448.5Gi / 464.3Gi) waste: 15.8Gi
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 1

    �[0;32mTop 1 most efficient:�[0m
      �[0;32m#1�[0m [1xl-arm64g3-61-463]
         CPU:  96.9%  MEM:  96.6%  waste: 2.0c + 15.8Gi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: r7i.12xlarge�[0m
  Total: 48 vCPU, 384Gi advertised (355.2Gi actual)
  Kubelet reserved: 190m CPU, 2.9Gi RAM
  DaemonSet overhead: 460m CPU, 1.1Gi RAM
  �[0;32mAllocatable for runners: 47350m CPU (47.4 cores), 351.2Gi RAM�[0m

  �[1mRunners targeting this node:�[0m
    - l-x86iamx-16-128: 15320m CPU, 116.5Gi RAM (job: 15c+116.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;32ml-x86iamx-16-128�[0m: 3 pods
      CPU:  97.1% (45960m / 47350m) waste: 1390m (1.4 cores)
      MEM:  99.5% (349.5Gi / 351.2Gi) waste: 1.7Gi
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 1

    �[0;32mTop 1 most efficient:�[0m
      �[0;32m#1�[0m [3xl-x86iamx-16-128]
         CPU:  97.1%  MEM:  99.5%  waste: 1.4c + 1.7Gi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: r7i.16xlarge�[0m
  Total: 64 vCPU, 512Gi advertised (473.7Gi actual)
  Kubelet reserved: 230m CPU, 8.3Gi RAM
  DaemonSet overhead: 460m CPU, 1.1Gi RAM
  �[0;32mAllocatable for runners: 63310m CPU (63.3 cores), 464.3Gi RAM�[0m

  �[1mRunners targeting this node:�[0m
    - l-x86iamx-8-64: 8320m CPU, 64.5Gi RAM (job: 8c+64.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[0;32ml-x86iamx-8-64�[0m: 7 pods
      CPU:  92.0% (58240m / 63310m) waste: 5070m (5.1 cores)
      MEM:  97.3% (451.6Gi / 464.3Gi) waste: 12.7Gi
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 1

    �[0;32mTop 1 most efficient:�[0m
      �[0;32m#1�[0m [7xl-x86iamx-8-64]
         CPU:  92.0%  MEM:  97.3%  waste: 5.1c + 12.7Gi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1m�[0;36mNode Type: t4g.2xlarge�[0m
  Total: 8 vCPU, 32Gi advertised (29.6Gi actual)
  Kubelet reserved: 90m CPU, 993Mi RAM
  DaemonSet overhead: 460m CPU, 1.1Gi RAM
  �[0;32mAllocatable for runners: 7450m CPU (7.5 cores), 27.5Gi RAM�[0m

  �[1mRunners targeting this node:�[0m
    - l-arm64g2-6-25: 6320m CPU, 25.5Gi RAM (job: 6c+25.0Gi, hooks: 320m+522Mi)

  �[1mHomogeneous packing (single runner type fills the node):�[0m
    �[1;33ml-arm64g2-6-25�[0m: 1 pods
      CPU:  84.8% (6320m / 7450m) waste: 1130m (1.1 cores)
      MEM:  92.8% (25.5Gi / 27.5Gi) waste: 2.0Gi
      Bottleneck: CPU

  �[1mMaximal mixed combos (node fully packed, no room for another pod):�[0m
    Total maximal combos: 1

    �[0;32mTop 1 most efficient:�[0m
      �[1;33m#1�[0m [1xl-arm64g2-6-25]
         CPU:  84.8%  MEM:  92.8%  waste: 1.1c + 2.0Gi

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[0;31m�[1mFound 13 runner type(s) with homogeneous utilization below 90.0%�[0m

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
�[1mUnused resource headroom per node (homogeneous packing only):�[0m

  Node Type                 Min CPU    Max CPU    Min MEM    Max MEM
  ────────────────────────────────────────────────────────────────
  c7a.12xlarge              1030m     14710m      1.3Gi     19.8Gi
  c7a.2xlarge               2810m      2810m      3.7Gi      3.7Gi
  c7a.48xlarge             96670m     96670m    156.3Gi    156.3Gi
  c7a.4xlarge               7110m      7110m      9.1Gi      9.1Gi
  c7i.12xlarge              1030m      5750m      1.3Gi      2.3Gi
  c7i.metal-24xl            2910m      2910m      712Mi      712Mi
  g4dn.12xlarge             1930m      1930m      1.1Gi      1.1Gi
  g4dn.8xlarge              1970m      1970m      296Mi      296Mi
  g4dn.metal                 810m       810m      1.3Gi      1.3Gi
  g5.12xlarge               1930m      1930m      712Mi      712Mi
  g5.48xlarge               1570m      1570m      723Mi      723Mi
  g5.8xlarge                1970m      1970m      408Mi      408Mi
  g6.12xlarge               1930m      1930m      1.1Gi      1.1Gi
  g6.8xlarge                1970m      1970m      408Mi      408Mi
  m6i.32xlarge              1830m      1830m     15.8Gi     15.8Gi
  m7g.8xlarge                750m       750m      1.4Gi      1.4Gi
  m7g.metal                  990m       990m      974Mi      974Mi
  m7i.12xlarge              5750m     15030m     11.1Gi     45.1Gi
  m8g.16xlarge               990m       990m      974Mi      974Mi
  m8g.24xlarge               910m       910m      1.3Gi      1.3Gi
  m8g.48xlarge             11470m     11470m     13.4Gi     13.4Gi
  m8g.8xlarge                750m       750m      1.4Gi      1.4Gi
  p4d.24xlarge              4570m      6810m     53.0Gi     56.5Gi
  p5.48xlarge              12330m     14570m     82.7Gi     86.3Gi
  p6-b200.48xlarge         12330m     14570m     82.7Gi     86.3Gi
  r7a.12xlarge              1030m     15030m      715Mi     94.7Gi
  r7a.16xlarge              5070m      5070m     12.7Gi     12.7Gi
  r7a.48xlarge             16270m     96670m     56.7Gi    670.9Gi
  r7g.12xlarge              3030m      3030m     10.7Gi     10.7Gi
  r7g.16xlarge              1990m      1990m     15.8Gi     15.8Gi
  r7i.12xlarge              1390m      1390m      1.7Gi      1.7Gi
  r7i.16xlarge              5070m      5070m     12.7Gi     12.7Gi
  t4g.2xlarge               1130m      1130m      2.0Gi      2.0Gi
  ────────────────────────────────────────────────────────────────
  �[1mWORST CASE            �[0m     750m     96670m      296Mi    670.9Gi

  The tightest node has only �[1m750m CPU�[0m and �[1m296Mi RAM�[0m free.
  Any new DaemonSet must fit within these limits or runners will fail to schedule.

@github-actions

github-actions Bot commented Jul 15, 2026

Copy link
Copy Markdown

tofu plan — meta-prod-aws-uw1

✅ Plan succeeded · commit 8c011ff1 · run log

Plan output
Installed 1 package in 2ms
{
    "BucketArn": "arn:aws:s3:::ciforge-tfstate-arc-cbr-prod-uw1",
    "BucketRegion": "us-west-2",
    "AccessPointAlias": false
}
━━━ PLAN: Base (meta-prod-aws-uw1) ━━━
There are some problems with the CLI configuration:
╷
│ Error: The specified plugin cache dir /home/runner/work/ci-infra/ci-infra/osdc/.terraform.d/plugin-cache cannot be opened: stat /home/runner/work/ci-infra/ci-infra/osdc/.terraform.d/plugin-cache: no such file or directory
│
╵

As a result of the above problems, OpenTofu may not behave as intended.


data.aws_availability_zones.available: Reading...
module.eks.data.aws_caller_identity.current: Reading...
module.eks.data.aws_ami.eks_optimized_al2023: Reading...
module.eks.aws_iam_role.node: Refreshing state... [id=meta-prod-aws-uw1-node-role]
module.eks.aws_kms_key.eks_secrets[0]: Refreshing state... [id=249b363f-50d9-4d71-8c19-33c90089adc4]
module.harbor.aws_iam_user.harbor_s3: Refreshing state... [id=meta-prod-aws-uw1-harbor-s3]
module.vpc.aws_vpc.this: Refreshing state... [id=vpc-0c7a0f5c4f16f50dd]
module.harbor.aws_s3_bucket.harbor_registry: Refreshing state... [id=meta-prod-aws-uw1-harbor-registry]
module.eks.aws_iam_role.cluster: Refreshing state... [id=meta-prod-aws-uw1-cluster-role]
module.eks.data.aws_caller_identity.current: Read complete after 0s [id=308535385114]
module.harbor.aws_iam_access_key.harbor_s3: Refreshing state... [id=AKIAUPVRELQNNXRVD65N]
module.eks.aws_iam_role_policy_attachment.vpc_resource_controller: Refreshing state... [id=meta-prod-aws-uw1-cluster-role/arn:aws:iam::aws:policy/AmazonEKSVPCResourceController]
module.eks.aws_iam_role_policy_attachment.cluster_policy: Refreshing state... [id=meta-prod-aws-uw1-cluster-role/arn:aws:iam::aws:policy/AmazonEKSClusterPolicy]
module.eks.aws_iam_role_policy_attachment.node_policy: Refreshing state... [id=meta-prod-aws-uw1-node-role/arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy]
module.eks.aws_iam_role_policy.node_cni_ipv6: Refreshing state... [id=meta-prod-aws-uw1-node-role:meta-prod-aws-uw1-node-cni-ipv6]
module.eks.aws_iam_role_policy_attachment.ssm_policy: Refreshing state... [id=meta-prod-aws-uw1-node-role/arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore]
data.aws_availability_zones.available: Read complete after 0s [id=us-west-1]
module.eks.aws_iam_role_policy_attachment.ecr_policy: Refreshing state... [id=meta-prod-aws-uw1-node-role/arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly]
module.eks.aws_iam_role_policy_attachment.cni_policy: Refreshing state... [id=meta-prod-aws-uw1-node-role/arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy]
module.eks.aws_kms_alias.eks_secrets[0]: Refreshing state... [id=alias/meta-prod-aws-uw1-eks-secrets]
module.eks.data.aws_ami.eks_optimized_al2023: Read complete after 1s [id=ami-07fd8394a1d58b614]
module.vpc.aws_egress_only_internet_gateway.this: Refreshing state... [id=eigw-0063ee624cd2732d8]
module.vpc.aws_internet_gateway.this: Refreshing state... [id=igw-0c3a516c5f092e482]
module.vpc.aws_subnet.private[1]: Refreshing state... [id=subnet-0e9dc0314cc83122d]
module.vpc.aws_subnet.private[0]: Refreshing state... [id=subnet-0ec6555eaab36cb0f]
module.vpc.aws_route_table.public: Refreshing state... [id=rtb-09ba42112ad2f3498]
module.vpc.aws_eip.nat_secondary["us-west-1c-2"]: Refreshing state... [id=eipalloc-02f6fb7e715ebf888]
module.vpc.aws_eip.nat_secondary["us-west-1a-2"]: Refreshing state... [id=eipalloc-09e48c6512d9e612f]
module.vpc.aws_subnet.public[1]: Refreshing state... [id=subnet-000973dc2a1d7e619]
module.vpc.aws_subnet.public[0]: Refreshing state... [id=subnet-09e992b7d08e2e934]
module.vpc.aws_eip.nat[1]: Refreshing state... [id=eipalloc-04dd3f923b70d7177]
module.vpc.aws_eip.nat_secondary["us-west-1a-6"]: Refreshing state... [id=eipalloc-0bf3c5405c4b04b9c]
module.vpc.aws_eip.nat_secondary["us-west-1c-0"]: Refreshing state... [id=eipalloc-0aa6be0f303fb413d]
module.vpc.aws_eip.nat_secondary["us-west-1c-3"]: Refreshing state... [id=eipalloc-0b5f261d2d97bcdc2]
module.vpc.aws_eip.nat_secondary["us-west-1a-3"]: Refreshing state... [id=eipalloc-0396526b463284581]
module.vpc.aws_eip.nat_secondary["us-west-1c-5"]: Refreshing state... [id=eipalloc-0def1ec3fa43dac4b]
module.vpc.aws_eip.nat_secondary["us-west-1a-4"]: Refreshing state... [id=eipalloc-052349005feef822c]
module.vpc.aws_eip.nat_secondary["us-west-1a-0"]: Refreshing state... [id=eipalloc-08ee495e2612f9854]
module.vpc.aws_eip.nat_secondary["us-west-1c-1"]: Refreshing state... [id=eipalloc-0b7ef04c32820eb2b]
module.vpc.aws_eip.nat_secondary["us-west-1a-5"]: Refreshing state... [id=eipalloc-0686d617611f65881]
module.vpc.aws_eip.nat[0]: Refreshing state... [id=eipalloc-062b73a22caa683b7]
module.vpc.aws_eip.nat_secondary["us-west-1c-6"]: Refreshing state... [id=eipalloc-047cd622afc027ee1]
module.vpc.aws_eip.nat_secondary["us-west-1a-1"]: Refreshing state... [id=eipalloc-07fc0a4a8844d68ef]
module.vpc.aws_eip.nat_secondary["us-west-1c-4"]: Refreshing state... [id=eipalloc-026da1516572eec86]
module.harbor.aws_iam_policy.harbor_registry: Refreshing state... [id=arn:aws:iam::308535385114:policy/meta-prod-aws-uw1-harbor-registry]
module.harbor.aws_s3_bucket_server_side_encryption_configuration.harbor_registry: Refreshing state... [id=meta-prod-aws-uw1-harbor-registry]
module.harbor.aws_s3_bucket_public_access_block.harbor_registry: Refreshing state... [id=meta-prod-aws-uw1-harbor-registry]
module.vpc.aws_route_table_association.public[1]: Refreshing state... [id=rtbassoc-021106d76c25ab7dd]
module.vpc.aws_route_table_association.public[0]: Refreshing state... [id=rtbassoc-0ea3c0815876de4c6]
module.harbor.aws_iam_user_policy_attachment.harbor_s3: Refreshing state... [id=meta-prod-aws-uw1-harbor-s3/arn:aws:iam::308535385114:policy/meta-prod-aws-uw1-harbor-registry]
module.eks.aws_eks_cluster.this: Refreshing state... [id=meta-prod-aws-uw1]
module.eks.aws_eks_access_entry.cluster_admin["osdc_gha_prod"]: Refreshing state... [id=meta-prod-aws-uw1:arn:aws:iam::308535385114:role/osdc_gha_prod]
module.eks.aws_eks_addon.vpc_cni: Refreshing state... [id=meta-prod-aws-uw1:vpc-cni]
module.eks.aws_launch_template.base: Refreshing state... [id=lt-09e8b410d5601fb81]
module.eks.aws_eks_addon.kube_proxy: Refreshing state... [id=meta-prod-aws-uw1:kube-proxy]
module.eks.data.tls_certificate.cluster[0]: Reading...
module.eks.aws_eks_node_group.base: Refreshing state... [id=meta-prod-aws-uw1:meta-prod-aws-uw1-base-nodes]
module.eks.data.tls_certificate.cluster[0]: Read complete after 0s [id=ab5db6c82031e2d229412c67921160a3b3af073b]
module.eks.aws_iam_openid_connect_provider.cluster[0]: Refreshing state... [id=arn:aws:iam::308535385114:oidc-provider/oidc.eks.us-west-1.amazonaws.com/id/049195ED3247B03B610AB041768C99D9]
module.eks.data.aws_iam_policy_document.ebs_csi_assume_role[0]: Reading...
module.harbor.aws_iam_role.harbor_registry: Refreshing state... [id=meta-prod-aws-uw1-harbor-registry]
module.eks.data.aws_iam_policy_document.ebs_csi_assume_role[0]: Read complete after 0s [id=1948150697]
module.eks.aws_iam_role.ebs_csi_driver[0]: Refreshing state... [id=meta-prod-aws-uw1-ebs-csi-driver-role]
module.eks.aws_eks_access_policy_association.cluster_admin["osdc_gha_prod"]: Refreshing state... [id=meta-prod-aws-uw1#arn:aws:iam::308535385114:role/osdc_gha_prod#arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy]
module.eks.aws_eks_addon.coredns: Refreshing state... [id=meta-prod-aws-uw1:coredns]
module.eks.aws_iam_role_policy_attachment.ebs_csi_driver[0]: Refreshing state... [id=meta-prod-aws-uw1-ebs-csi-driver-role/arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy]
module.harbor.aws_iam_role_policy_attachment.harbor_registry: Refreshing state... [id=meta-prod-aws-uw1-harbor-registry/arn:aws:iam::308535385114:policy/meta-prod-aws-uw1-harbor-registry]
module.eks.aws_eks_addon.ebs_csi_driver: Refreshing state... [id=meta-prod-aws-uw1:aws-ebs-csi-driver]
module.vpc.aws_nat_gateway.this[0]: Refreshing state... [id=nat-0c4f5e0d136a43bc1]
module.vpc.aws_nat_gateway.this[1]: Refreshing state... [id=nat-0f09a57f0a682972e]
module.vpc.aws_route_table.private[0]: Refreshing state... [id=rtb-0252a95ace300ec13]
module.vpc.aws_route_table.private[1]: Refreshing state... [id=rtb-096d5785b7f057456]
module.vpc.aws_route_table_association.private[0]: Refreshing state... [id=rtbassoc-03da4cbd6db56b03b]
module.vpc.aws_route_table_association.private[1]: Refreshing state... [id=rtbassoc-073892599db845ebe]

No changes. Your infrastructure matches the configuration.

OpenTofu has compared your real infrastructure against your configuration and
found no differences, so no changes are needed.

━━━ PLAN: Module karpenter (meta-prod-aws-uw1) ━━━
data.terraform_remote_state.base: Reading...
aws_cloudwatch_event_rule.scheduled_change: Refreshing state... [id=meta-prod-aws-uw1-karpenter-scheduled-change]
aws_cloudwatch_event_rule.instance_state_change: Refreshing state... [id=meta-prod-aws-uw1-karpenter-instance-state-change]
aws_sqs_queue.karpenter: Refreshing state... [id=https://sqs.us-west-1.amazonaws.com/308535385114/meta-prod-aws-uw1-karpenter]
aws_cloudwatch_event_rule.rebalance: Refreshing state... [id=meta-prod-aws-uw1-karpenter-rebalance]
aws_cloudwatch_event_rule.spot_interruption: Refreshing state... [id=meta-prod-aws-uw1-karpenter-spot-interruption]
aws_sqs_queue_policy.karpenter: Refreshing state... [id=https://sqs.us-west-1.amazonaws.com/308535385114/meta-prod-aws-uw1-karpenter]
aws_cloudwatch_event_target.scheduled_change: Refreshing state... [id=meta-prod-aws-uw1-karpenter-scheduled-change-KarpenterScheduledChange]
aws_cloudwatch_event_target.instance_state_change: Refreshing state... [id=meta-prod-aws-uw1-karpenter-instance-state-change-KarpenterInstanceStateChange]
aws_cloudwatch_event_target.spot_interruption: Refreshing state... [id=meta-prod-aws-uw1-karpenter-spot-interruption-KarpenterSpotInterruption]
aws_cloudwatch_event_target.rebalance: Refreshing state... [id=meta-prod-aws-uw1-karpenter-rebalance-KarpenterRebalance]
data.terraform_remote_state.base: Read complete after 1s
aws_ec2_tag.cluster_sg_karpenter: Refreshing state... [id=sg-07b2ae82341ea99c4,karpenter.sh/discovery]
aws_iam_role.karpenter_controller: Refreshing state... [id=meta-prod-aws-uw1-karpenter-controller]
aws_ec2_tag.subnet_karpenter_discovery["subnet-0ec6555eaab36cb0f"]: Refreshing state... [id=subnet-0ec6555eaab36cb0f,karpenter.sh/discovery]
aws_ec2_tag.subnet_karpenter_discovery["subnet-0e9dc0314cc83122d"]: Refreshing state... [id=subnet-0e9dc0314cc83122d,karpenter.sh/discovery]
aws_iam_policy.karpenter_controller: Refreshing state... [id=arn:aws:iam::308535385114:policy/meta-prod-aws-uw1-karpenter-controller]
aws_iam_role_policy_attachment.karpenter_controller: Refreshing state... [id=meta-prod-aws-uw1-karpenter-controller-20260625071532734400000001]

No changes. Your infrastructure matches the configuration.

OpenTofu has compared your real infrastructure against your configuration and
found no differences, so no changes are needed.

━━━ PLAN: Module hf-cache (meta-prod-aws-uw1) ━━━
data.terraform_remote_state.base: Reading...
aws_iam_policy.hf_cache: Refreshing state... [id=arn:aws:iam::308535385114:policy/meta-prod-aws-uw1-hf-cache-s3]
aws_s3_bucket.hf_cache: Refreshing state... [id=pytorch-hf-model-cache-meta-prod-aws-uw1]
data.terraform_remote_state.base: Read complete after 1s
aws_iam_role.hf_cache: Refreshing state... [id=meta-prod-aws-uw1-hf-cache-role]
aws_iam_role_policy_attachment.hf_cache: Refreshing state... [id=meta-prod-aws-uw1-hf-cache-role-20260701020207578300000001]
aws_s3_bucket_public_access_block.hf_cache: Refreshing state... [id=pytorch-hf-model-cache-meta-prod-aws-uw1]
aws_s3_bucket_server_side_encryption_configuration.hf_cache: Refreshing state... [id=pytorch-hf-model-cache-meta-prod-aws-uw1]
aws_s3_bucket_lifecycle_configuration.hf_cache: Refreshing state... [id=pytorch-hf-model-cache-meta-prod-aws-uw1]

No changes. Your infrastructure matches the configuration.

OpenTofu has compared your real infrastructure against your configuration and
found no differences, so no changes are needed.

@github-actions

github-actions Bot commented Jul 15, 2026

Copy link
Copy Markdown

tofu plan — meta-prod-aws-ue1

✅ Plan succeeded · commit 8c011ff1 · run log

Plan output
Installed 1 package in 2ms
{
    "BucketArn": "arn:aws:s3:::ciforge-tfstate-arc-cbr-prod-ue1",
    "BucketRegion": "us-west-2",
    "AccessPointAlias": false
}
━━━ PLAN: Base (meta-prod-aws-ue1) ━━━
There are some problems with the CLI configuration:
╷
│ Error: The specified plugin cache dir /home/runner/work/ci-infra/ci-infra/osdc/.terraform.d/plugin-cache cannot be opened: stat /home/runner/work/ci-infra/ci-infra/osdc/.terraform.d/plugin-cache: no such file or directory
│
╵

As a result of the above problems, OpenTofu may not behave as intended.


data.aws_availability_zones.available: Reading...
module.eks.data.aws_caller_identity.current: Reading...
module.eks.data.aws_ami.eks_optimized_al2023: Reading...
module.eks.aws_kms_key.eks_secrets[0]: Refreshing state... [id=03f0ec26-a6da-43fa-b637-0f60858b706f]
module.eks.aws_iam_role.node: Refreshing state... [id=meta-prod-aws-ue1-node-role]
module.vpc.aws_vpc.this: Refreshing state... [id=vpc-046818728dce02486]
module.eks.aws_iam_role.cluster: Refreshing state... [id=meta-prod-aws-ue1-cluster-role]
module.harbor.aws_s3_bucket.harbor_registry: Refreshing state... [id=meta-prod-aws-ue1-harbor-registry]
module.harbor.aws_iam_user.harbor_s3: Refreshing state... [id=meta-prod-aws-ue1-harbor-s3]
module.eks.data.aws_caller_identity.current: Read complete after 0s [id=308535385114]
module.harbor.aws_iam_access_key.harbor_s3: Refreshing state... [id=AKIAUPVRELQNOISM5G6N]
data.aws_availability_zones.available: Read complete after 0s [id=us-east-1]
module.eks.aws_iam_role_policy_attachment.cluster_policy: Refreshing state... [id=meta-prod-aws-ue1-cluster-role/arn:aws:iam::aws:policy/AmazonEKSClusterPolicy]
module.eks.aws_iam_role_policy_attachment.vpc_resource_controller: Refreshing state... [id=meta-prod-aws-ue1-cluster-role/arn:aws:iam::aws:policy/AmazonEKSVPCResourceController]
module.eks.aws_kms_alias.eks_secrets[0]: Refreshing state... [id=alias/meta-prod-aws-ue1-eks-secrets]
module.eks.aws_iam_role_policy_attachment.node_policy: Refreshing state... [id=meta-prod-aws-ue1-node-role/arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy]
module.eks.aws_iam_role_policy.node_cni_ipv6: Refreshing state... [id=meta-prod-aws-ue1-node-role:meta-prod-aws-ue1-node-cni-ipv6]
module.eks.aws_iam_role_policy_attachment.cni_policy: Refreshing state... [id=meta-prod-aws-ue1-node-role/arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy]
module.eks.aws_iam_role_policy_attachment.ssm_policy: Refreshing state... [id=meta-prod-aws-ue1-node-role/arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore]
module.eks.aws_iam_role_policy_attachment.ecr_policy: Refreshing state... [id=meta-prod-aws-ue1-node-role/arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly]
module.eks.data.aws_ami.eks_optimized_al2023: Read complete after 1s [id=ami-0dafeb02304897431]
module.harbor.aws_iam_policy.harbor_registry: Refreshing state... [id=arn:aws:iam::308535385114:policy/meta-prod-aws-ue1-harbor-registry]
module.harbor.aws_s3_bucket_public_access_block.harbor_registry: Refreshing state... [id=meta-prod-aws-ue1-harbor-registry]
module.harbor.aws_s3_bucket_server_side_encryption_configuration.harbor_registry: Refreshing state... [id=meta-prod-aws-ue1-harbor-registry]
module.harbor.aws_iam_user_policy_attachment.harbor_s3: Refreshing state... [id=meta-prod-aws-ue1-harbor-s3/arn:aws:iam::308535385114:policy/meta-prod-aws-ue1-harbor-registry]
module.vpc.aws_internet_gateway.this: Refreshing state... [id=igw-0cf3d9cf37ee998b6]
module.vpc.aws_egress_only_internet_gateway.this: Refreshing state... [id=eigw-04023bafa7a35ed32]
module.vpc.aws_route_table.public: Refreshing state... [id=rtb-0c553a216ffcfbc6e]
module.vpc.aws_eip.nat_secondary["us-east-1c-4"]: Refreshing state... [id=eipalloc-08c22d102baccc560]
module.vpc.aws_eip.nat_secondary["us-east-1b-5"]: Refreshing state... [id=eipalloc-0891e3c936177ca1f]
module.vpc.aws_subnet.private[0]: Refreshing state... [id=subnet-0d65ec2dd49f0d87c]
module.vpc.aws_subnet.private[1]: Refreshing state... [id=subnet-0348c5058db524cd2]
module.vpc.aws_eip.nat_secondary["us-east-1b-0"]: Refreshing state... [id=eipalloc-0d3046f70c5e06703]
module.vpc.aws_eip.nat_secondary["us-east-1b-1"]: Refreshing state... [id=eipalloc-0d7540e64f41d03ed]
module.vpc.aws_eip.nat_secondary["us-east-1a-1"]: Refreshing state... [id=eipalloc-04d8d726f10c423a7]
module.vpc.aws_subnet.private[2]: Refreshing state... [id=subnet-02ce11d6646870431]
module.vpc.aws_eip.nat_secondary["us-east-1c-5"]: Refreshing state... [id=eipalloc-08b6b96db8427fe7c]
module.vpc.aws_eip.nat_secondary["us-east-1b-3"]: Refreshing state... [id=eipalloc-080f1de6aadf86bfe]
module.vpc.aws_eip.nat_secondary["us-east-1c-1"]: Refreshing state... [id=eipalloc-0f0532f5d59cb48d4]
module.vpc.aws_eip.nat_secondary["us-east-1a-0"]: Refreshing state... [id=eipalloc-0b13ecc2db20a0a08]
module.vpc.aws_eip.nat_secondary["us-east-1c-2"]: Refreshing state... [id=eipalloc-07835c4a6798d0cba]
module.vpc.aws_eip.nat_secondary["us-east-1b-4"]: Refreshing state... [id=eipalloc-033e293e0db093eb5]
module.vpc.aws_eip.nat_secondary["us-east-1c-6"]: Refreshing state... [id=eipalloc-0cfb63f74c1bfe868]
module.vpc.aws_eip.nat_secondary["us-east-1b-2"]: Refreshing state... [id=eipalloc-082b7b67fc91bddea]
module.vpc.aws_eip.nat_secondary["us-east-1a-5"]: Refreshing state... [id=eipalloc-0b1c40b234bf34782]
module.vpc.aws_eip.nat_secondary["us-east-1a-2"]: Refreshing state... [id=eipalloc-085f1255da01c4a74]
module.vpc.aws_eip.nat_secondary["us-east-1c-0"]: Refreshing state... [id=eipalloc-04e4ed5d389da6ee8]
module.vpc.aws_eip.nat_secondary["us-east-1c-3"]: Refreshing state... [id=eipalloc-0345e227c85668435]
module.vpc.aws_eip.nat_secondary["us-east-1a-6"]: Refreshing state... [id=eipalloc-0f4b7b7794e7ddcab]
module.vpc.aws_eip.nat_secondary["us-east-1a-3"]: Refreshing state... [id=eipalloc-01fe44e4ddcf970a4]
module.vpc.aws_eip.nat_secondary["us-east-1b-6"]: Refreshing state... [id=eipalloc-0626688dd96fd72b1]
module.vpc.aws_eip.nat_secondary["us-east-1a-4"]: Refreshing state... [id=eipalloc-00b67915425a445ad]
module.vpc.aws_subnet.public[2]: Refreshing state... [id=subnet-0b79a2a6a8c2d4c93]
module.vpc.aws_subnet.public[0]: Refreshing state... [id=subnet-07f0242f48547edf9]
module.vpc.aws_subnet.public[1]: Refreshing state... [id=subnet-092d6ffaad52de297]
module.vpc.aws_eip.nat[2]: Refreshing state... [id=eipalloc-0e51c52fc16722bf1]
module.vpc.aws_eip.nat[0]: Refreshing state... [id=eipalloc-02e910dda876f6868]
module.vpc.aws_eip.nat[1]: Refreshing state... [id=eipalloc-0c54492cc83f6a297]
module.eks.aws_eks_cluster.this: Refreshing state... [id=meta-prod-aws-ue1]
module.vpc.aws_route_table_association.public[1]: Refreshing state... [id=rtbassoc-0092c387bef531804]
module.vpc.aws_route_table_association.public[2]: Refreshing state... [id=rtbassoc-0ec09f9cebd9e03d7]
module.vpc.aws_route_table_association.public[0]: Refreshing state... [id=rtbassoc-002be7d0c2cd21897]
module.vpc.aws_nat_gateway.this[1]: Refreshing state... [id=nat-0017452213c04953c]
module.vpc.aws_nat_gateway.this[0]: Refreshing state... [id=nat-051257023fad95bd5]
module.vpc.aws_nat_gateway.this[2]: Refreshing state... [id=nat-017b6d671f099d80b]
module.eks.aws_eks_addon.kube_proxy: Refreshing state... [id=meta-prod-aws-ue1:kube-proxy]
module.eks.aws_eks_addon.vpc_cni: Refreshing state... [id=meta-prod-aws-ue1:vpc-cni]
module.eks.aws_eks_access_entry.cluster_admin["osdc_gha_prod"]: Refreshing state... [id=meta-prod-aws-ue1:arn:aws:iam::308535385114:role/osdc_gha_prod]
module.eks.aws_launch_template.base: Refreshing state... [id=lt-010f6dcef487af1c3]
module.eks.data.tls_certificate.cluster[0]: Reading...
module.vpc.aws_route_table.private[2]: Refreshing state... [id=rtb-010e846a3ffc4e7f9]
module.vpc.aws_route_table.private[0]: Refreshing state... [id=rtb-01284f97ba53d4b6a]
module.vpc.aws_route_table.private[1]: Refreshing state... [id=rtb-08bf0f3dfffae56cd]
module.eks.aws_eks_node_group.base: Refreshing state... [id=meta-prod-aws-ue1:meta-prod-aws-ue1-base-nodes]
module.vpc.aws_route_table_association.private[1]: Refreshing state... [id=rtbassoc-08adf22b49bcc40a4]
module.vpc.aws_route_table_association.private[0]: Refreshing state... [id=rtbassoc-016c3961a7fb4ded9]
module.vpc.aws_route_table_association.private[2]: Refreshing state... [id=rtbassoc-0ec9f3350a410730f]
module.eks.data.tls_certificate.cluster[0]: Read complete after 0s [id=b1b539daa206035ae3c3e28288b0681fa1b462f3]
module.eks.aws_iam_openid_connect_provider.cluster[0]: Refreshing state... [id=arn:aws:iam::308535385114:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/B24010B528DA0FC03A2C455E74946D6B]
module.eks.data.aws_iam_policy_document.ebs_csi_assume_role[0]: Reading...
module.harbor.aws_iam_role.harbor_registry: Refreshing state... [id=meta-prod-aws-ue1-harbor-registry]
module.eks.data.aws_iam_policy_document.ebs_csi_assume_role[0]: Read complete after 0s [id=4151242138]
module.eks.aws_iam_role.ebs_csi_driver[0]: Refreshing state... [id=meta-prod-aws-ue1-ebs-csi-driver-role]
module.eks.aws_eks_access_policy_association.cluster_admin["osdc_gha_prod"]: Refreshing state... [id=meta-prod-aws-ue1#arn:aws:iam::308535385114:role/osdc_gha_prod#arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy]
module.eks.aws_eks_addon.coredns: Refreshing state... [id=meta-prod-aws-ue1:coredns]
module.eks.aws_iam_role_policy_attachment.ebs_csi_driver[0]: Refreshing state... [id=meta-prod-aws-ue1-ebs-csi-driver-role/arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy]
module.harbor.aws_iam_role_policy_attachment.harbor_registry: Refreshing state... [id=meta-prod-aws-ue1-harbor-registry/arn:aws:iam::308535385114:policy/meta-prod-aws-ue1-harbor-registry]
module.eks.aws_eks_addon.ebs_csi_driver: Refreshing state... [id=meta-prod-aws-ue1:aws-ebs-csi-driver]

No changes. Your infrastructure matches the configuration.

OpenTofu has compared your real infrastructure against your configuration and
found no differences, so no changes are needed.

━━━ PLAN: Module karpenter (meta-prod-aws-ue1) ━━━
data.terraform_remote_state.base: Reading...
aws_cloudwatch_event_rule.instance_state_change: Refreshing state... [id=meta-prod-aws-ue1-karpenter-instance-state-change]
aws_cloudwatch_event_rule.rebalance: Refreshing state... [id=meta-prod-aws-ue1-karpenter-rebalance]
aws_sqs_queue.karpenter: Refreshing state... [id=https://sqs.us-east-1.amazonaws.com/308535385114/meta-prod-aws-ue1-karpenter]
aws_cloudwatch_event_rule.spot_interruption: Refreshing state... [id=meta-prod-aws-ue1-karpenter-spot-interruption]
aws_cloudwatch_event_rule.scheduled_change: Refreshing state... [id=meta-prod-aws-ue1-karpenter-scheduled-change]
aws_sqs_queue_policy.karpenter: Refreshing state... [id=https://sqs.us-east-1.amazonaws.com/308535385114/meta-prod-aws-ue1-karpenter]
aws_cloudwatch_event_target.instance_state_change: Refreshing state... [id=meta-prod-aws-ue1-karpenter-instance-state-change-KarpenterInstanceStateChange]
aws_cloudwatch_event_target.rebalance: Refreshing state... [id=meta-prod-aws-ue1-karpenter-rebalance-KarpenterRebalance]
aws_cloudwatch_event_target.spot_interruption: Refreshing state... [id=meta-prod-aws-ue1-karpenter-spot-interruption-KarpenterSpotInterruption]
aws_cloudwatch_event_target.scheduled_change: Refreshing state... [id=meta-prod-aws-ue1-karpenter-scheduled-change-KarpenterScheduledChange]
data.terraform_remote_state.base: Read complete after 1s
aws_ec2_tag.subnet_karpenter_discovery["subnet-0348c5058db524cd2"]: Refreshing state... [id=subnet-0348c5058db524cd2,karpenter.sh/discovery]
aws_ec2_tag.subnet_karpenter_discovery["subnet-0d65ec2dd49f0d87c"]: Refreshing state... [id=subnet-0d65ec2dd49f0d87c,karpenter.sh/discovery]
aws_ec2_tag.cluster_sg_karpenter: Refreshing state... [id=sg-016f4a0d209f3e4a9,karpenter.sh/discovery]
aws_ec2_tag.subnet_karpenter_discovery["subnet-02ce11d6646870431"]: Refreshing state... [id=subnet-02ce11d6646870431,karpenter.sh/discovery]
aws_iam_policy.karpenter_controller: Refreshing state... [id=arn:aws:iam::308535385114:policy/meta-prod-aws-ue1-karpenter-controller]
aws_iam_role.karpenter_controller: Refreshing state... [id=meta-prod-aws-ue1-karpenter-controller]
aws_iam_role_policy_attachment.karpenter_controller: Refreshing state... [id=meta-prod-aws-ue1-karpenter-controller-20260627083405030800000001]

No changes. Your infrastructure matches the configuration.

OpenTofu has compared your real infrastructure against your configuration and
found no differences, so no changes are needed.

━━━ PLAN: Module hf-cache (meta-prod-aws-ue1) ━━━
data.terraform_remote_state.base: Reading...
aws_iam_policy.hf_cache: Refreshing state... [id=arn:aws:iam::308535385114:policy/meta-prod-aws-ue1-hf-cache-s3]
aws_s3_bucket.hf_cache: Refreshing state... [id=pytorch-hf-model-cache-meta-prod-aws-ue1]
data.terraform_remote_state.base: Read complete after 1s
aws_iam_role.hf_cache: Refreshing state... [id=meta-prod-aws-ue1-hf-cache-role]
aws_s3_bucket_public_access_block.hf_cache: Refreshing state... [id=pytorch-hf-model-cache-meta-prod-aws-ue1]
aws_s3_bucket_server_side_encryption_configuration.hf_cache: Refreshing state... [id=pytorch-hf-model-cache-meta-prod-aws-ue1]
aws_s3_bucket_lifecycle_configuration.hf_cache: Refreshing state... [id=pytorch-hf-model-cache-meta-prod-aws-ue1]
aws_iam_role_policy_attachment.hf_cache: Refreshing state... [id=meta-prod-aws-ue1-hf-cache-role-20260702211726957200000001]

No changes. Your infrastructure matches the configuration.

OpenTofu has compared your real infrastructure against your configuration and
found no differences, so no changes are needed.

@github-actions

github-actions Bot commented Jul 15, 2026

Copy link
Copy Markdown

tofu plan — lf-prod-aws-ue1

✅ Plan succeeded · commit 8c011ff1 · run log

Plan output
Installed 1 package in 1ms
{
    "BucketArn": "arn:aws:s3:::lf-osdc-tfstate-prod-ue1",
    "BucketRegion": "us-west-2",
    "AccessPointAlias": false
}
━━━ PLAN: Base (lf-prod-aws-ue1) ━━━
There are some problems with the CLI configuration:
╷
│ Error: The specified plugin cache dir /home/runner/work/ci-infra/ci-infra/osdc/.terraform.d/plugin-cache cannot be opened: stat /home/runner/work/ci-infra/ci-infra/osdc/.terraform.d/plugin-cache: no such file or directory
│
╵

As a result of the above problems, OpenTofu may not behave as intended.


data.aws_availability_zones.available: Reading...
module.eks.data.aws_caller_identity.current: Reading...
module.harbor.aws_iam_user.harbor_s3: Refreshing state... [id=lf-prod-aws-ue1-harbor-s3]
module.vpc.aws_vpc.this: Refreshing state... [id=vpc-06f350eae88f37700]
module.eks.data.aws_ami.eks_optimized_al2023: Reading...
module.eks.aws_iam_role.cluster: Refreshing state... [id=lf-prod-aws-ue1-cluster-role]
module.eks.aws_kms_key.eks_secrets[0]: Refreshing state... [id=e5e45db6-94ad-4dfd-8a1a-213730256a9c]
module.eks.aws_iam_role.node: Refreshing state... [id=lf-prod-aws-ue1-node-role]
module.harbor.aws_s3_bucket.harbor_registry: Refreshing state... [id=lf-prod-aws-ue1-harbor-registry]
module.eks.data.aws_caller_identity.current: Read complete after 0s [id=391835788720]
module.harbor.aws_iam_access_key.harbor_s3: Refreshing state... [id=AKIAVWOZ3UWYJZNKMI7G]
data.aws_availability_zones.available: Read complete after 0s [id=us-east-1]
module.eks.aws_iam_role_policy_attachment.vpc_resource_controller: Refreshing state... [id=lf-prod-aws-ue1-cluster-role/arn:aws:iam::aws:policy/AmazonEKSVPCResourceController]
module.eks.aws_iam_role_policy_attachment.cluster_policy: Refreshing state... [id=lf-prod-aws-ue1-cluster-role/arn:aws:iam::aws:policy/AmazonEKSClusterPolicy]
module.eks.aws_kms_alias.eks_secrets[0]: Refreshing state... [id=alias/lf-prod-aws-ue1-eks-secrets]
module.eks.aws_iam_role_policy_attachment.ecr_policy: Refreshing state... [id=lf-prod-aws-ue1-node-role/arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly]
module.eks.aws_iam_role_policy.node_cni_ipv6: Refreshing state... [id=lf-prod-aws-ue1-node-role:lf-prod-aws-ue1-node-cni-ipv6]
module.eks.aws_iam_role_policy_attachment.cni_policy: Refreshing state... [id=lf-prod-aws-ue1-node-role/arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy]
module.eks.aws_iam_role_policy_attachment.node_policy: Refreshing state... [id=lf-prod-aws-ue1-node-role/arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy]
module.eks.aws_iam_role_policy_attachment.ssm_policy: Refreshing state... [id=lf-prod-aws-ue1-node-role/arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore]
module.eks.data.aws_ami.eks_optimized_al2023: Read complete after 0s [id=ami-0dafeb02304897431]
module.harbor.aws_iam_policy.harbor_registry: Refreshing state... [id=arn:aws:iam::391835788720:policy/lf-prod-aws-ue1-harbor-registry]
module.harbor.aws_s3_bucket_public_access_block.harbor_registry: Refreshing state... [id=lf-prod-aws-ue1-harbor-registry]
module.harbor.aws_s3_bucket_server_side_encryption_configuration.harbor_registry: Refreshing state... [id=lf-prod-aws-ue1-harbor-registry]
module.vpc.aws_egress_only_internet_gateway.this: Refreshing state... [id=eigw-03548aa6de237de4c]
module.vpc.aws_internet_gateway.this: Refreshing state... [id=igw-089c5123e6da8d43c]
module.harbor.aws_iam_user_policy_attachment.harbor_s3: Refreshing state... [id=lf-prod-aws-ue1-harbor-s3/arn:aws:iam::391835788720:policy/lf-prod-aws-ue1-harbor-registry]
module.vpc.aws_eip.nat[1]: Refreshing state... [id=eipalloc-0f16b1a5ecd405405]
module.vpc.aws_eip.nat[2]: Refreshing state... [id=eipalloc-0fe9cf01a4661b360]
module.vpc.aws_subnet.public[1]: Refreshing state... [id=subnet-06e680510bc45584b]
module.vpc.aws_eip.nat[0]: Refreshing state... [id=eipalloc-087f338c446cffe5d]
module.vpc.aws_eip.nat_secondary["us-east-1a-5"]: Refreshing state... [id=eipalloc-0936080d9155a0306]
module.vpc.aws_subnet.private[2]: Refreshing state... [id=subnet-01ca3df6137b445c0]
module.vpc.aws_subnet.public[0]: Refreshing state... [id=subnet-0afe958a38da9f46c]
module.vpc.aws_subnet.private[0]: Refreshing state... [id=subnet-07234379e7833a398]
module.vpc.aws_subnet.private[1]: Refreshing state... [id=subnet-0fa332056910f46b2]
module.vpc.aws_subnet.public[2]: Refreshing state... [id=subnet-0f7184dc74425b3ca]
module.vpc.aws_eip.nat_secondary["us-east-1b-3"]: Refreshing state... [id=eipalloc-042e254711d0d3dda]
module.vpc.aws_eip.nat_secondary["us-east-1c-1"]: Refreshing state... [id=eipalloc-01b2275a4f494fe58]
module.vpc.aws_eip.nat_secondary["us-east-1c-4"]: Refreshing state... [id=eipalloc-0db3b4c44cbf47d5a]
module.vpc.aws_eip.nat_secondary["us-east-1b-0"]: Refreshing state... [id=eipalloc-0533e7098b8548fda]
module.vpc.aws_eip.nat_secondary["us-east-1c-6"]: Refreshing state... [id=eipalloc-088384cdd02d04bce]
module.vpc.aws_eip.nat_secondary["us-east-1b-6"]: Refreshing state... [id=eipalloc-085e43aacdd3b5c5f]
module.vpc.aws_eip.nat_secondary["us-east-1b-1"]: Refreshing state... [id=eipalloc-0f13b0cd68a133531]
module.vpc.aws_eip.nat_secondary["us-east-1b-2"]: Refreshing state... [id=eipalloc-0c66936151cceca74]
module.vpc.aws_eip.nat_secondary["us-east-1a-3"]: Refreshing state... [id=eipalloc-01a2c9bf10e45099b]
module.vpc.aws_eip.nat_secondary["us-east-1a-1"]: Refreshing state... [id=eipalloc-034348675ffacd849]
module.vpc.aws_eip.nat_secondary["us-east-1a-0"]: Refreshing state... [id=eipalloc-03131d115b478f7c3]
module.vpc.aws_eip.nat_secondary["us-east-1c-3"]: Refreshing state... [id=eipalloc-03ec1105bba33668b]
module.vpc.aws_eip.nat_secondary["us-east-1b-4"]: Refreshing state... [id=eipalloc-0bd28bd3991297f4a]
module.vpc.aws_eip.nat_secondary["us-east-1c-0"]: Refreshing state... [id=eipalloc-051a8792e8dad6c5b]
module.vpc.aws_eip.nat_secondary["us-east-1a-6"]: Refreshing state... [id=eipalloc-046fa83874bde66b5]
module.vpc.aws_route_table.public: Refreshing state... [id=rtb-051217c40b1d02b3a]
module.vpc.aws_eip.nat_secondary["us-east-1a-4"]: Refreshing state... [id=eipalloc-0581e7f2f8194266e]
module.vpc.aws_eip.nat_secondary["us-east-1c-2"]: Refreshing state... [id=eipalloc-0a4874208e55dfb7b]
module.vpc.aws_eip.nat_secondary["us-east-1c-5"]: Refreshing state... [id=eipalloc-07612f4e715e508ae]
module.vpc.aws_eip.nat_secondary["us-east-1b-5"]: Refreshing state... [id=eipalloc-00cd91c376a1f197d]
module.vpc.aws_eip.nat_secondary["us-east-1a-2"]: Refreshing state... [id=eipalloc-0625b097098b1ac2a]
module.vpc.aws_route_table_association.public[0]: Refreshing state... [id=rtbassoc-025d5eab2da94f8e6]
module.vpc.aws_route_table_association.public[1]: Refreshing state... [id=rtbassoc-0a6049a78a7428383]
module.vpc.aws_route_table_association.public[2]: Refreshing state... [id=rtbassoc-025956fd021d43094]
module.eks.aws_eks_cluster.this: Refreshing state... [id=lf-prod-aws-ue1]
module.vpc.aws_nat_gateway.this[1]: Refreshing state... [id=nat-09c6f66e50dce1835]
module.vpc.aws_nat_gateway.this[2]: Refreshing state... [id=nat-0cf871626838e4133]
module.vpc.aws_nat_gateway.this[0]: Refreshing state... [id=nat-0d0e9d964d1cd8a9e]
module.vpc.aws_route_table.private[2]: Refreshing state... [id=rtb-027811d9ba750a284]
module.vpc.aws_route_table.private[1]: Refreshing state... [id=rtb-0264db606b7f24bb6]
module.vpc.aws_route_table.private[0]: Refreshing state... [id=rtb-02536bbe724eaaa2f]
module.eks.aws_eks_addon.kube_proxy: Refreshing state... [id=lf-prod-aws-ue1:kube-proxy]
module.eks.data.tls_certificate.cluster[0]: Reading...
module.eks.aws_eks_addon.vpc_cni: Refreshing state... [id=lf-prod-aws-ue1:vpc-cni]
module.eks.aws_eks_access_entry.cluster_admin["lf_osdc_admin"]: Refreshing state... [id=lf-prod-aws-ue1:arn:aws:iam::391835788720:role/lf_osdc_admin]
module.eks.aws_launch_template.base: Refreshing state... [id=lt-0e3be05a985acc61c]
module.vpc.aws_route_table_association.private[2]: Refreshing state... [id=rtbassoc-009170e5cc902aa3e]
module.vpc.aws_route_table_association.private[1]: Refreshing state... [id=rtbassoc-0c6e77db648d5279c]
module.vpc.aws_route_table_association.private[0]: Refreshing state... [id=rtbassoc-00cd583d71292870b]
module.eks.aws_eks_node_group.base: Refreshing state... [id=lf-prod-aws-ue1:lf-prod-aws-ue1-base-nodes]
module.eks.data.tls_certificate.cluster[0]: Read complete after 0s [id=b1b539daa206035ae3c3e28288b0681fa1b462f3]
module.eks.aws_iam_openid_connect_provider.cluster[0]: Refreshing state... [id=arn:aws:iam::391835788720:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/E8EF4A6C55DB9699E53A54DA444C21A3]
module.harbor.aws_iam_role.harbor_registry: Refreshing state... [id=lf-prod-aws-ue1-harbor-registry]
module.eks.data.aws_iam_policy_document.ebs_csi_assume_role[0]: Reading...
module.eks.data.aws_iam_policy_document.ebs_csi_assume_role[0]: Read complete after 0s [id=717515682]
module.eks.aws_iam_role.ebs_csi_driver[0]: Refreshing state... [id=lf-prod-aws-ue1-ebs-csi-driver-role]
module.eks.aws_eks_access_policy_association.cluster_admin["lf_osdc_admin"]: Refreshing state... [id=lf-prod-aws-ue1#arn:aws:iam::391835788720:role/lf_osdc_admin#arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy]
module.eks.aws_eks_addon.coredns: Refreshing state... [id=lf-prod-aws-ue1:coredns]
module.eks.aws_iam_role_policy_attachment.ebs_csi_driver[0]: Refreshing state... [id=lf-prod-aws-ue1-ebs-csi-driver-role/arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy]
module.harbor.aws_iam_role_policy_attachment.harbor_registry: Refreshing state... [id=lf-prod-aws-ue1-harbor-registry/arn:aws:iam::391835788720:policy/lf-prod-aws-ue1-harbor-registry]
module.eks.aws_eks_addon.ebs_csi_driver: Refreshing state... [id=lf-prod-aws-ue1:aws-ebs-csi-driver]

No changes. Your infrastructure matches the configuration.

OpenTofu has compared your real infrastructure against your configuration and
found no differences, so no changes are needed.

━━━ PLAN: Module karpenter (lf-prod-aws-ue1) ━━━
data.terraform_remote_state.base: Reading...
aws_cloudwatch_event_rule.scheduled_change: Refreshing state... [id=lf-prod-aws-ue1-karpenter-scheduled-change]
aws_cloudwatch_event_rule.instance_state_change: Refreshing state... [id=lf-prod-aws-ue1-karpenter-instance-state-change]
aws_cloudwatch_event_rule.rebalance: Refreshing state... [id=lf-prod-aws-ue1-karpenter-rebalance]
aws_sqs_queue.karpenter: Refreshing state... [id=https://sqs.us-east-1.amazonaws.com/391835788720/lf-prod-aws-ue1-karpenter]
aws_cloudwatch_event_rule.spot_interruption: Refreshing state... [id=lf-prod-aws-ue1-karpenter-spot-interruption]
aws_sqs_queue_policy.karpenter: Refreshing state... [id=https://sqs.us-east-1.amazonaws.com/391835788720/lf-prod-aws-ue1-karpenter]
aws_cloudwatch_event_target.rebalance: Refreshing state... [id=lf-prod-aws-ue1-karpenter-rebalance-KarpenterRebalance]
aws_cloudwatch_event_target.instance_state_change: Refreshing state... [id=lf-prod-aws-ue1-karpenter-instance-state-change-KarpenterInstanceStateChange]
aws_cloudwatch_event_target.spot_interruption: Refreshing state... [id=lf-prod-aws-ue1-karpenter-spot-interruption-KarpenterSpotInterruption]
aws_cloudwatch_event_target.scheduled_change: Refreshing state... [id=lf-prod-aws-ue1-karpenter-scheduled-change-KarpenterScheduledChange]
data.terraform_remote_state.base: Read complete after 0s
aws_ec2_tag.subnet_karpenter_discovery["subnet-07234379e7833a398"]: Refreshing state... [id=subnet-07234379e7833a398,karpenter.sh/discovery]
aws_ec2_tag.cluster_sg_karpenter: Refreshing state... [id=sg-07a769a5e8a93a444,karpenter.sh/discovery]
aws_iam_policy.karpenter_controller: Refreshing state... [id=arn:aws:iam::391835788720:policy/lf-prod-aws-ue1-karpenter-controller]
aws_ec2_tag.subnet_karpenter_discovery["subnet-01ca3df6137b445c0"]: Refreshing state... [id=subnet-01ca3df6137b445c0,karpenter.sh/discovery]
aws_ec2_tag.subnet_karpenter_discovery["subnet-0fa332056910f46b2"]: Refreshing state... [id=subnet-0fa332056910f46b2,karpenter.sh/discovery]
aws_iam_role.karpenter_controller: Refreshing state... [id=lf-prod-aws-ue1-karpenter-controller]
aws_iam_role_policy_attachment.karpenter_controller: Refreshing state... [id=lf-prod-aws-ue1-karpenter-controller-20260605165913470400000001]

No changes. Your infrastructure matches the configuration.

OpenTofu has compared your real infrastructure against your configuration and
found no differences, so no changes are needed.

@github-actions

github-actions Bot commented Jul 15, 2026

Copy link
Copy Markdown

tofu plan — lf-prod-aws-ue2

✅ Plan succeeded · commit 8c011ff1 · run log

Plan output
Installed 1 package in 2ms
{
    "BucketArn": "arn:aws:s3:::lf-osdc-tfstate-prod-ue2",
    "BucketRegion": "us-west-2",
    "AccessPointAlias": false
}
━━━ PLAN: Base (lf-prod-aws-ue2) ━━━
There are some problems with the CLI configuration:
╷
│ Error: The specified plugin cache dir /home/runner/work/ci-infra/ci-infra/osdc/.terraform.d/plugin-cache cannot be opened: stat /home/runner/work/ci-infra/ci-infra/osdc/.terraform.d/plugin-cache: no such file or directory
│
╵

As a result of the above problems, OpenTofu may not behave as intended.


module.harbor.aws_iam_user.harbor_s3: Refreshing state... [id=lf-prod-aws-ue2-harbor-s3]
module.eks.aws_iam_role.cluster: Refreshing state... [id=lf-prod-aws-ue2-cluster-role]
module.eks.data.aws_ami.eks_optimized_al2023: Reading...
module.eks.aws_kms_key.eks_secrets[0]: Refreshing state... [id=27a9b8e9-2509-43ce-ac8e-cfc320b65fe2]
data.aws_availability_zones.available: Reading...
module.eks.data.aws_caller_identity.current: Reading...
module.vpc.aws_vpc.this: Refreshing state... [id=vpc-0f7d54e3accfbe3e4]
module.eks.aws_iam_role.node: Refreshing state... [id=lf-prod-aws-ue2-node-role]
module.harbor.aws_s3_bucket.harbor_registry: Refreshing state... [id=lf-prod-aws-ue2-harbor-registry]
module.eks.data.aws_caller_identity.current: Read complete after 0s [id=391835788720]
module.harbor.aws_iam_access_key.harbor_s3: Refreshing state... [id=AKIAVWOZ3UWYMGG4LIHB]
module.eks.aws_iam_role_policy_attachment.cluster_policy: Refreshing state... [id=lf-prod-aws-ue2-cluster-role/arn:aws:iam::aws:policy/AmazonEKSClusterPolicy]
module.eks.aws_iam_role_policy_attachment.vpc_resource_controller: Refreshing state... [id=lf-prod-aws-ue2-cluster-role/arn:aws:iam::aws:policy/AmazonEKSVPCResourceController]
module.eks.aws_iam_role_policy_attachment.ssm_policy: Refreshing state... [id=lf-prod-aws-ue2-node-role/arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore]
module.eks.aws_iam_role_policy_attachment.ecr_policy: Refreshing state... [id=lf-prod-aws-ue2-node-role/arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly]
module.eks.aws_iam_role_policy.node_cni_ipv6: Refreshing state... [id=lf-prod-aws-ue2-node-role:lf-prod-aws-ue2-node-cni-ipv6]
data.aws_availability_zones.available: Read complete after 1s [id=us-east-2]
module.eks.aws_iam_role_policy_attachment.node_policy: Refreshing state... [id=lf-prod-aws-ue2-node-role/arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy]
module.eks.aws_iam_role_policy_attachment.cni_policy: Refreshing state... [id=lf-prod-aws-ue2-node-role/arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy]
module.eks.aws_kms_alias.eks_secrets[0]: Refreshing state... [id=alias/lf-prod-aws-ue2-eks-secrets]
module.eks.data.aws_ami.eks_optimized_al2023: Read complete after 1s [id=ami-009f1fe7d56695348]
module.harbor.aws_s3_bucket_public_access_block.harbor_registry: Refreshing state... [id=lf-prod-aws-ue2-harbor-registry]
module.harbor.aws_iam_policy.harbor_registry: Refreshing state... [id=arn:aws:iam::391835788720:policy/lf-prod-aws-ue2-harbor-registry]
module.harbor.aws_s3_bucket_server_side_encryption_configuration.harbor_registry: Refreshing state... [id=lf-prod-aws-ue2-harbor-registry]
module.harbor.aws_iam_user_policy_attachment.harbor_s3: Refreshing state... [id=lf-prod-aws-ue2-harbor-s3/arn:aws:iam::391835788720:policy/lf-prod-aws-ue2-harbor-registry]
module.vpc.aws_egress_only_internet_gateway.this: Refreshing state... [id=eigw-061f8f7ac8b40d720]
module.vpc.aws_internet_gateway.this: Refreshing state... [id=igw-042c4d31ed557eaa4]
module.vpc.aws_subnet.private[0]: Refreshing state... [id=subnet-0515848329e5dc53a]
module.vpc.aws_route_table.public: Refreshing state... [id=rtb-0508ab6e3db7ccf08]
module.vpc.aws_subnet.private[2]: Refreshing state... [id=subnet-06a9b2e4ea40968b6]
module.vpc.aws_subnet.private[1]: Refreshing state... [id=subnet-0ae8d251d3a0336ca]
module.vpc.aws_eip.nat[2]: Refreshing state... [id=eipalloc-079ed57d9de06fd9b]
module.vpc.aws_subnet.public[1]: Refreshing state... [id=subnet-0e53846501278171e]
module.vpc.aws_eip.nat[0]: Refreshing state... [id=eipalloc-0e0efd2a8ef20d72e]
module.vpc.aws_eip.nat_secondary["us-east-2c-4"]: Refreshing state... [id=eipalloc-005e21ac878c4db34]
module.vpc.aws_eip.nat[1]: Refreshing state... [id=eipalloc-08c041a7cb9147705]
module.vpc.aws_subnet.public[2]: Refreshing state... [id=subnet-080bfdf02da937445]
module.vpc.aws_subnet.public[0]: Refreshing state... [id=subnet-016d460df617c0e2c]
module.vpc.aws_eip.nat_secondary["us-east-2b-5"]: Refreshing state... [id=eipalloc-06f0755f7542d77fa]
module.vpc.aws_eip.nat_secondary["us-east-2a-3"]: Refreshing state... [id=eipalloc-0737f1fdf35a0f975]
module.vpc.aws_eip.nat_secondary["us-east-2b-4"]: Refreshing state... [id=eipalloc-08683a31d5967bff6]
module.vpc.aws_eip.nat_secondary["us-east-2a-6"]: Refreshing state... [id=eipalloc-077cbc910a56d08fd]
module.vpc.aws_eip.nat_secondary["us-east-2b-0"]: Refreshing state... [id=eipalloc-055182abe5c634ddc]
module.vpc.aws_eip.nat_secondary["us-east-2c-1"]: Refreshing state... [id=eipalloc-0241d507f34cdb0b5]
module.vpc.aws_eip.nat_secondary["us-east-2a-0"]: Refreshing state... [id=eipalloc-0a9078e90b80cc1de]
module.vpc.aws_eip.nat_secondary["us-east-2c-6"]: Refreshing state... [id=eipalloc-057df768d859ed17e]
module.vpc.aws_eip.nat_secondary["us-east-2c-2"]: Refreshing state... [id=eipalloc-08e66df79eddc18b5]
module.vpc.aws_eip.nat_secondary["us-east-2a-1"]: Refreshing state... [id=eipalloc-0a90e8e5b75a3fe45]
module.vpc.aws_eip.nat_secondary["us-east-2b-2"]: Refreshing state... [id=eipalloc-0403ed9359182b72c]
module.vpc.aws_eip.nat_secondary["us-east-2a-5"]: Refreshing state... [id=eipalloc-095865342b4c692ac]
module.vpc.aws_eip.nat_secondary["us-east-2b-1"]: Refreshing state... [id=eipalloc-0c8d74e3dcfb2dad0]
module.vpc.aws_eip.nat_secondary["us-east-2c-3"]: Refreshing state... [id=eipalloc-04d97b3aec8f5fb8a]
module.vpc.aws_eip.nat_secondary["us-east-2c-5"]: Refreshing state... [id=eipalloc-06c020042f283554a]
module.vpc.aws_eip.nat_secondary["us-east-2a-4"]: Refreshing state... [id=eipalloc-0bd8a5e170892bb0b]
module.vpc.aws_eip.nat_secondary["us-east-2b-3"]: Refreshing state... [id=eipalloc-0b95441aa4e161db2]
module.vpc.aws_eip.nat_secondary["us-east-2b-6"]: Refreshing state... [id=eipalloc-09c38605941dbbaac]
module.vpc.aws_eip.nat_secondary["us-east-2c-0"]: Refreshing state... [id=eipalloc-09bd4b74b1a8ca6ac]
module.vpc.aws_eip.nat_secondary["us-east-2a-2"]: Refreshing state... [id=eipalloc-0e53d306d25151b0e]
module.vpc.aws_route_table_association.public[1]: Refreshing state... [id=rtbassoc-028a6f03785f6bca2]
module.vpc.aws_route_table_association.public[2]: Refreshing state... [id=rtbassoc-005f847cdca1f2143]
module.vpc.aws_route_table_association.public[0]: Refreshing state... [id=rtbassoc-0d0f31615161dab0f]
module.eks.aws_eks_cluster.this: Refreshing state... [id=lf-prod-aws-ue2]
module.vpc.aws_nat_gateway.this[1]: Refreshing state... [id=nat-0288194135c91a55d]
module.vpc.aws_nat_gateway.this[0]: Refreshing state... [id=nat-095cd56cd812b4931]
module.vpc.aws_nat_gateway.this[2]: Refreshing state... [id=nat-0caff297f1b93f0c7]
module.vpc.aws_route_table.private[1]: Refreshing state... [id=rtb-0d0497dd1d2a111f5]
module.vpc.aws_route_table.private[0]: Refreshing state... [id=rtb-0ce64842bfadf32b0]
module.vpc.aws_route_table.private[2]: Refreshing state... [id=rtb-0d7230758d05b4f20]
module.eks.data.tls_certificate.cluster[0]: Reading...
module.eks.aws_eks_addon.kube_proxy: Refreshing state... [id=lf-prod-aws-ue2:kube-proxy]
module.eks.aws_eks_access_entry.cluster_admin["lf_osdc_admin"]: Refreshing state... [id=lf-prod-aws-ue2:arn:aws:iam::391835788720:role/lf_osdc_admin]
module.eks.aws_launch_template.base: Refreshing state... [id=lt-062d0b42e1b1ca1af]
module.eks.aws_eks_addon.vpc_cni: Refreshing state... [id=lf-prod-aws-ue2:vpc-cni]
module.vpc.aws_route_table_association.private[0]: Refreshing state... [id=rtbassoc-0cbaf74e1bd57a865]
module.vpc.aws_route_table_association.private[1]: Refreshing state... [id=rtbassoc-05de05c204a439484]
module.vpc.aws_route_table_association.private[2]: Refreshing state... [id=rtbassoc-0feb6707491379e22]
module.eks.aws_eks_node_group.base: Refreshing state... [id=lf-prod-aws-ue2:lf-prod-aws-ue2-base-nodes]
module.eks.data.tls_certificate.cluster[0]: Read complete after 0s [id=033a163afb2babc26f7883e642621ac361c93d61]
module.eks.aws_iam_openid_connect_provider.cluster[0]: Refreshing state... [id=arn:aws:iam::391835788720:oidc-provider/oidc.eks.us-east-2.amazonaws.com/id/43EEAC690CC76E15781134A4FC06EDCE]
module.eks.data.aws_iam_policy_document.ebs_csi_assume_role[0]: Reading...
module.harbor.aws_iam_role.harbor_registry: Refreshing state... [id=lf-prod-aws-ue2-harbor-registry]
module.eks.data.aws_iam_policy_document.ebs_csi_assume_role[0]: Read complete after 0s [id=796338164]
module.eks.aws_iam_role.ebs_csi_driver[0]: Refreshing state... [id=lf-prod-aws-ue2-ebs-csi-driver-role]
module.eks.aws_eks_addon.coredns: Refreshing state... [id=lf-prod-aws-ue2:coredns]
module.harbor.aws_iam_role_policy_attachment.harbor_registry: Refreshing state... [id=lf-prod-aws-ue2-harbor-registry/arn:aws:iam::391835788720:policy/lf-prod-aws-ue2-harbor-registry]
module.eks.aws_iam_role_policy_attachment.ebs_csi_driver[0]: Refreshing state... [id=lf-prod-aws-ue2-ebs-csi-driver-role/arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy]
module.eks.aws_eks_access_policy_association.cluster_admin["lf_osdc_admin"]: Refreshing state... [id=lf-prod-aws-ue2#arn:aws:iam::391835788720:role/lf_osdc_admin#arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy]
module.eks.aws_eks_addon.ebs_csi_driver: Refreshing state... [id=lf-prod-aws-ue2:aws-ebs-csi-driver]

No changes. Your infrastructure matches the configuration.

OpenTofu has compared your real infrastructure against your configuration and
found no differences, so no changes are needed.

━━━ PLAN: Module karpenter (lf-prod-aws-ue2) ━━━
data.terraform_remote_state.base: Reading...
aws_cloudwatch_event_rule.scheduled_change: Refreshing state... [id=lf-prod-aws-ue2-karpenter-scheduled-change]
aws_sqs_queue.karpenter: Refreshing state... [id=https://sqs.us-east-2.amazonaws.com/391835788720/lf-prod-aws-ue2-karpenter]
aws_cloudwatch_event_rule.instance_state_change: Refreshing state... [id=lf-prod-aws-ue2-karpenter-instance-state-change]
aws_cloudwatch_event_rule.rebalance: Refreshing state... [id=lf-prod-aws-ue2-karpenter-rebalance]
aws_cloudwatch_event_rule.spot_interruption: Refreshing state... [id=lf-prod-aws-ue2-karpenter-spot-interruption]
aws_sqs_queue_policy.karpenter: Refreshing state... [id=https://sqs.us-east-2.amazonaws.com/391835788720/lf-prod-aws-ue2-karpenter]
aws_cloudwatch_event_target.rebalance: Refreshing state... [id=lf-prod-aws-ue2-karpenter-rebalance-KarpenterRebalance]
aws_cloudwatch_event_target.instance_state_change: Refreshing state... [id=lf-prod-aws-ue2-karpenter-instance-state-change-KarpenterInstanceStateChange]
aws_cloudwatch_event_target.spot_interruption: Refreshing state... [id=lf-prod-aws-ue2-karpenter-spot-interruption-KarpenterSpotInterruption]
aws_cloudwatch_event_target.scheduled_change: Refreshing state... [id=lf-prod-aws-ue2-karpenter-scheduled-change-KarpenterScheduledChange]
data.terraform_remote_state.base: Read complete after 1s
aws_ec2_tag.cluster_sg_karpenter: Refreshing state... [id=sg-06c1f2ed8ffb1ddfa,karpenter.sh/discovery]
aws_ec2_tag.subnet_karpenter_discovery["subnet-06a9b2e4ea40968b6"]: Refreshing state... [id=subnet-06a9b2e4ea40968b6,karpenter.sh/discovery]
aws_ec2_tag.subnet_karpenter_discovery["subnet-0515848329e5dc53a"]: Refreshing state... [id=subnet-0515848329e5dc53a,karpenter.sh/discovery]
aws_ec2_tag.subnet_karpenter_discovery["subnet-0ae8d251d3a0336ca"]: Refreshing state... [id=subnet-0ae8d251d3a0336ca,karpenter.sh/discovery]
aws_iam_role.karpenter_controller: Refreshing state... [id=lf-prod-aws-ue2-karpenter-controller]
aws_iam_policy.karpenter_controller: Refreshing state... [id=arn:aws:iam::391835788720:policy/lf-prod-aws-ue2-karpenter-controller]
aws_iam_role_policy_attachment.karpenter_controller: Refreshing state... [id=lf-prod-aws-ue2-karpenter-controller-20260608235145776400000001]

No changes. Your infrastructure matches the configuration.

OpenTofu has compared your real infrastructure against your configuration and
found no differences, so no changes are needed.

@huydhn

huydhn commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

@claude Double check to see if all the changes are moved correctly from arc-runners-opt to arc-runners. Any recent changes on arc-runners to call out?

@claude

claude Bot commented Jul 15, 2026

Copy link
Copy Markdown

Claude finished @huydhn's task in 4m 20s —— View job


Verifying arc-runners-opt → arc-runners migration

  • Compare PR diff against the -opt module definitions
  • Verify runner defs (node_fleet / instance types) match
  • Verify nodepool tier fleets match
  • Check recent history on arc-runners for changes not reflected in -opt
  • Summarize findings

✅ Everything moved correctly

I diffed the regular modules against the -opt modules file-by-file. All packing-related defs are now byte-identical to their -opt counterparts.

Runner defs (21 changed) — all identical to arc-runners-opt:
Every shared runner def in arc-runners/defs/ now byte-matches arc-runners-opt/defs/. The node_fleet + right-sized instance_type were copied correctly, e.g.:

  • l-x86iavx512-16-32: c7a.48xlargec7a.12xlarge + node_fleet: c7a-12xl
  • l-x86iamx-8-32: m7i.48xlargem7i.12xlarge + node_fleet: m7i-12xl

Nodepool tier fleets (11 added/renamed) — all identical to nodepools-opt:
c7a-12xl, c7a-2xl, c7a-4xl, c7i-12xl, m7g-8xl, m7i-12xl, m8g-8xl, r7a-12xl, r7a-16xl, r7i-12xl, r7i-16xl — all byte-identical. ✓

c7i-runner Graviton switch — identical to opt: arch: amd64arm64, instances → m7g.12xlarge/m7g.8xlarge/m8g.12xlarge/m8g.8xlarge, and the taint-decoupling comment all match nodepools-opt/c7i-runner.yaml exactly. ✓

Monolithic fleets removed (6): c7a→renamed to c7a-12xl, c7ic7i-12xl, m7im7i-12xl, r7ir7i-12xl; m7g.yaml and m8g.yaml deleted. Matches the PR description. ✓

⚠️ Intentional divergences from -opt to call out

These are not migration mistakes — they're places where the regular modules deliberately differ from -opt, and the PR correctly leaves them alone:

  1. Release-runner design differs between the two modules (and the regular one is the newer design). The regular module ships:

    • rel-l-arm64g3-44-340.yamlr7g.12xlarge, one-runner-per-node, sized to the manywheel linux.arm64.r7g.12xlarge.memory build spec
    • rel-l-x86iavx512-44-340.yamlr7a.12xlarge, sized to linux.12xlarge.memory

    whereas -opt still has the older rel-l-arm64g4-16-62 (m8g.48xlarge) / rel-l-x86iavx512-8-64 (r7a.48xlarge) packing approach. Correspondingly, regular r7a.yaml/r7g.yaml carry a release: block on *.12xlarge, -opt carries m8g.yaml + r7a.48xlarge release. None of these rel-* defs or release blocks are touched by this PR — confirmed no rel-* file and no r7g.yaml appears in the diff. So the regular module is actually ahead of -opt on release runners; that gap is out of scope here and is worth a follow-up to reconcile -opt in the other direction.

  2. PR diff on r7a.yaml is only the instances: removal — the pre-existing release: block is retained. Correct: the monolithic packing fleet is dropped, release isolation stays.

Recent arc-runners changes

This is a ghstack PR, so history on the branch shows only the two stack commits (b5b8423, 37da338 base update). There are no independent recent commits to arc-runners/nodepools defs outside this stack that would conflict with the copy. The one standing difference from -opt (the release-runner defs above) predates this PR and is intentionally untouched.

Bottom line: the copy is complete and accurate — all 21 runner defs and 11 tier fleets match -opt exactly, the c7i-runner Graviton switch matches, and the 6 monolithic fleets are removed as described. The only divergences from -opt are the deliberately-different release-runner defs, which this PR correctly does not modify.

[ghstack-poisoned]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants