Skip to content

Commit 91eea39

Browse files
ravisoundarravisoundar
committed
test(e2e): e2e k8s environment tests using chainsaw
Signed-off-by: Ravi Shankar <ravish@nvidia.com>
1 parent beb745e commit 91eea39

20 files changed

Lines changed: 1097 additions & 2 deletions

File tree

‎.claude/CLAUDE.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@ internal/ # Shared utilities not part of the public API
4242
charts/topograph/ # Helm chart (with node-data-broker subchart); tests/ holds the helm-unittest suites + snapshots
4343
docs/ # Public-facing docs — overview.md, architecture.md, api.md + providers/, engines/, reference/ subdirectories
4444
tests/models/ # YAML simulation fixtures
45+
tests/chainsaw/ # Chainsaw E2E test suites (label-application, label-truncation, node-observer, slinky)
4546
config/ # Sample topograph-config.yaml
4647
scripts/ # Build scripts (deb, rpm, SSL, clean)
4748
localdev/ # Developer-local workspace — not tracked; personal scratch files
@@ -94,6 +95,20 @@ make coverage # human-readable per-package summary
9495

9596
Run `make qualify` before pushing. The individual targets are available if you want to run a single check during iteration. Run `make chart-test` when you change `charts/topograph/` or its subcharts; CI runs it on every workflow trigger.
9697

98+
### E2E tests (Chainsaw)
99+
100+
Chainsaw conformance tests live in `tests/chainsaw/` and exercise the full Helm deploy → generate → assert cycle against a real cluster.
101+
102+
```bash
103+
make e2e-local # build image, create kind cluster, run all suites, delete cluster
104+
make kind-load KIND_CLUSTER=<name> # load image into an existing kind cluster (run before make e2e)
105+
make e2e # run suites against current KUBECONFIG context
106+
```
107+
108+
`make e2e` uses `E2E_IMAGE_TAG` (defaults to the short commit SHA) as the image tag. For a local kind cluster, run `make image-build && make kind-load KIND_CLUSTER=<name>` before each `make e2e` — the tag changes with every commit, so both steps are needed after any new commit. Prerequisites: `chainsaw`, `kind`, `helm`, `kubectl`, `docker`. See `tests/chainsaw/README.md` for details.
109+
110+
These tests are triggered manually via `.github/workflows/e2e.yml` (`workflow_dispatch`). Run them before merging changes to the Helm chart, Node Observer, or engine output.
111+
97112
### Coverage policy
98113

99114
From `codecov.yml`:
@@ -109,6 +124,7 @@ Coverage checks run on pull requests. A drop below target with no matching uplif
109124
- `.github/workflows/docker.yml` — container image build (manual trigger)
110125
- `.github/workflows/docker-ib.yml` — InfiniBand-variant container (manual trigger)
111126
- `.github/workflows/helm-release.yaml` — Helm chart release (manual trigger)
127+
- `.github/workflows/e2e.yml` — Chainsaw E2E suite against a kind cluster (manual trigger via `workflow_dispatch`)
112128

113129
### Deployment surfaces
114130

‎.github/workflows/e2e.yml‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
# Copyright 2026 NVIDIA CORPORATION
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
name: E2E
5+
6+
on:
7+
workflow_dispatch:
8+
inputs:
9+
chainsaw_version:
10+
description: "Chainsaw version to install (e.g. v0.2.12)"
11+
required: false
12+
default: "latest"
13+
14+
env:
15+
KIND_CLUSTER: topograph-e2e
16+
IMAGE_REPO: ghcr.io/nvidia/topograph
17+
CHAINSAW_VERSION: ${{ github.event.inputs.chainsaw_version || 'latest' }}
18+
19+
jobs:
20+
e2e:
21+
name: Chainsaw E2E
22+
runs-on: ubuntu-latest
23+
permissions:
24+
contents: read
25+
26+
steps:
27+
- uses: actions/checkout@v5
28+
29+
- name: Set up Go
30+
uses: actions/setup-go@v6
31+
with:
32+
go-version: '1.25.9'
33+
34+
- name: Install kind
35+
run: go install sigs.k8s.io/kind@latest
36+
37+
- name: Install Chainsaw
38+
run: |
39+
if [ "$CHAINSAW_VERSION" = "latest" ]; then
40+
TAG=$(curl -s https://api.github.com/repos/kyverno/chainsaw/releases/latest \
41+
| grep '"tag_name"' | cut -d'"' -f4)
42+
else
43+
TAG="$CHAINSAW_VERSION"
44+
fi
45+
echo "Installing Chainsaw $TAG"
46+
BASE_URL="https://github.com/kyverno/chainsaw/releases/download/${TAG}"
47+
curl -fsSL "${BASE_URL}/chainsaw_linux_amd64.tar.gz" -o chainsaw.tar.gz
48+
curl -fsSL "${BASE_URL}/chainsaw_checksums.txt" -o chainsaw_checksums.txt
49+
grep "chainsaw_linux_amd64.tar.gz" chainsaw_checksums.txt | sha256sum -c -
50+
tar xz -f chainsaw.tar.gz chainsaw
51+
sudo mv chainsaw /usr/local/bin/
52+
rm -f chainsaw.tar.gz chainsaw_checksums.txt
53+
chainsaw version
54+
55+
- name: Create kind cluster
56+
run: |
57+
kind create cluster \
58+
--name "$KIND_CLUSTER" \
59+
--config tests/chainsaw/kind-config.yaml \
60+
--wait 120s
61+
62+
- name: Build Linux/amd64 image
63+
run: make build-linux-amd64
64+
65+
- name: Build container image
66+
env:
67+
GOOS: linux
68+
GOARCH: amd64
69+
run: |
70+
# Use the short commit SHA as the image tag: always a valid Docker tag,
71+
# works regardless of branch naming conventions.
72+
IMAGE_TAG=$(git rev-parse --short HEAD)
73+
make image-build IMAGE_TAG="$IMAGE_TAG"
74+
echo "IMAGE_TAG=$IMAGE_TAG" >> "$GITHUB_ENV"
75+
76+
- name: Load image into kind
77+
run: |
78+
kind load docker-image "${IMAGE_REPO}:${IMAGE_TAG}" \
79+
--name "$KIND_CLUSTER"
80+
81+
- name: Run E2E tests
82+
env:
83+
TOPOGRAPH_IMAGE_REPO: ${{ env.IMAGE_REPO }}
84+
TOPOGRAPH_IMAGE_PULL_POLICY: Never
85+
run: |
86+
make e2e E2E_IMAGE_TAG="$IMAGE_TAG"
87+
88+
- name: Collect diagnostic logs on failure
89+
if: failure()
90+
run: |
91+
echo "=== kind nodes ==="
92+
kubectl get nodes -o wide
93+
echo "=== all pods ==="
94+
kubectl get pods -A -o wide
95+
echo "=== recent events ==="
96+
kubectl get events -A --sort-by='.lastTimestamp' | tail -50
97+
98+
- name: Delete kind cluster
99+
if: always()
100+
run: kind delete cluster --name "$KIND_CLUSTER"

‎AGENTS.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@ internal/ # Shared utilities not part of the public API
4242
charts/topograph/ # Helm chart (with node-data-broker subchart); tests/ holds the helm-unittest suites + snapshots
4343
docs/ # Public-facing docs — overview.md, architecture.md, api.md + providers/, engines/, reference/ subdirectories
4444
tests/models/ # YAML simulation fixtures
45+
tests/chainsaw/ # Chainsaw E2E test suites (label-application, label-truncation, node-observer, slinky)
4546
config/ # Sample topograph-config.yaml
4647
scripts/ # Build scripts (deb, rpm, SSL, clean)
4748
localdev/ # Developer-local workspace — not tracked; personal scratch files
@@ -94,6 +95,20 @@ make coverage # human-readable per-package summary
9495

9596
Run `make qualify` before pushing. The individual targets are available if you want to run a single check during iteration. Run `make chart-test` when you change `charts/topograph/` or its subcharts; CI runs it on every workflow trigger.
9697

98+
### E2E tests (Chainsaw)
99+
100+
Chainsaw conformance tests live in `tests/chainsaw/` and exercise the full Helm deploy → generate → assert cycle against a real cluster.
101+
102+
```bash
103+
make e2e-local # build image, create kind cluster, run all suites, delete cluster
104+
make kind-load KIND_CLUSTER=<name> # load image into an existing kind cluster (run before make e2e)
105+
make e2e # run suites against current KUBECONFIG context
106+
```
107+
108+
`make e2e` uses `E2E_IMAGE_TAG` (defaults to the short commit SHA) as the image tag. For a local kind cluster, run `make image-build && make kind-load KIND_CLUSTER=<name>` before each `make e2e` — the tag changes with every commit, so both steps are needed after any new commit. Prerequisites: `chainsaw`, `kind`, `helm`, `kubectl`, `docker`. See `tests/chainsaw/README.md` for details.
109+
110+
These tests are triggered manually via `.github/workflows/e2e.yml` (`workflow_dispatch`). Run them before merging changes to the Helm chart, Node Observer, or engine output.
111+
97112
### Coverage policy
98113

99114
From `codecov.yml`:
@@ -109,6 +124,7 @@ Coverage checks run on pull requests. A drop below target with no matching uplif
109124
- `.github/workflows/docker.yml` — container image build (manual trigger)
110125
- `.github/workflows/docker-ib.yml` — InfiniBand-variant container (manual trigger)
111126
- `.github/workflows/helm-release.yaml` — Helm chart release (manual trigger)
127+
- `.github/workflows/e2e.yml` — Chainsaw E2E suite against a kind cluster (manual trigger via `workflow_dispatch`)
112128

113129
### Deployment surfaces
114130

‎Makefile‎

Lines changed: 45 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ OUTPUT_DIR := ./bin
2323

2424
IMAGE_REPO ?=ghcr.io/nvidia/topograph
2525
GIT_REF ?=$(shell git rev-parse --abbrev-ref HEAD)
26-
IMAGE_TAG ?=$(GIT_REF)
26+
IMAGE_TAG ?=$(shell git rev-parse --short HEAD)
2727

2828
.PHONY: build
2929
build:
@@ -102,7 +102,7 @@ coverage: test
102102

103103
.PHONY: image-build
104104
image-build:
105-
$(DOCKER_BIN) build --build-arg TARGETOS=$(GOOS) --build-arg TARGETARCH=$(GOARCH) -t $(IMAGE_REPO):$(IMAGE_TAG) -f ./Dockerfile .
105+
$(DOCKER_BIN) build --build-arg TARGETOS=linux --build-arg TARGETARCH=$(GOARCH) -t $(IMAGE_REPO):$(IMAGE_TAG) -f ./Dockerfile .
106106

107107
.PHONY: image-push
108108
image-push: image-build
@@ -115,6 +115,49 @@ docker-buildx:
115115
$(DOCKER_BIN) buildx build --platform $(PLATFORMS) -t $(IMAGE_REPO):$(IMAGE_TAG) -f ./Dockerfile --push .
116116
- $(DOCKER_BIN) buildx rm topograph-builder
117117

118+
CHAINSAW_BIN ?= chainsaw
119+
KIND_CLUSTER ?= topograph-e2e
120+
E2E_IMAGE_TAG ?= $(IMAGE_TAG)
121+
122+
# Check that chainsaw is installed; print install hint if not.
123+
.PHONY: chainsaw-install
124+
chainsaw-install:
125+
@which $(CHAINSAW_BIN) >/dev/null 2>&1 || \
126+
(echo "chainsaw not found — install from https://kyverno.github.io/chainsaw/latest/quick-start/install/"; exit 1)
127+
128+
# Load the locally-built image into an existing kind cluster with the correct
129+
# E2E_IMAGE_TAG. Use this before running make e2e against a local kind cluster:
130+
# make kind-load KIND_CLUSTER=topograph-test && make e2e
131+
.PHONY: kind-load
132+
kind-load:
133+
kind load docker-image $(IMAGE_REPO):$(E2E_IMAGE_TAG) --name $(KIND_CLUSTER)
134+
135+
# Run all Chainsaw E2E suites against the current KUBECONFIG context.
136+
# For a pre-pushed registry image: set TOPOGRAPH_IMAGE_REPO and TOPOGRAPH_IMAGE_TAG.
137+
# For a local kind cluster: run "make kind-load KIND_CLUSTER=<cluster>" first.
138+
.PHONY: e2e
139+
e2e: chainsaw-install
140+
TOPOGRAPH_IMAGE_REPO=$(IMAGE_REPO) \
141+
TOPOGRAPH_IMAGE_TAG=$(E2E_IMAGE_TAG) \
142+
$(CHAINSAW_BIN) test --test-dir tests/chainsaw
143+
144+
# Build the image, create a 4-worker kind cluster, load the image, run all
145+
# Chainsaw suites, and destroy the cluster. Requires kind and chainsaw.
146+
.PHONY: e2e-local
147+
e2e-local: chainsaw-install image-build
148+
kind create cluster --name $(KIND_CLUSTER) \
149+
--config tests/chainsaw/kind-config.yaml --wait 120s \
150+
|| kind get clusters | grep -q "^$(KIND_CLUSTER)$$"
151+
kind load docker-image $(IMAGE_REPO):$(E2E_IMAGE_TAG) --name $(KIND_CLUSTER)
152+
KUBECONFIG="$$(kind get kubeconfig --name $(KIND_CLUSTER))" \
153+
TOPOGRAPH_IMAGE_REPO=$(IMAGE_REPO) \
154+
TOPOGRAPH_IMAGE_TAG=$(E2E_IMAGE_TAG) \
155+
TOPOGRAPH_IMAGE_PULL_POLICY=Never \
156+
$(CHAINSAW_BIN) test --test-dir tests/chainsaw; \
157+
E2E_STATUS=$$?; \
158+
kind delete cluster --name $(KIND_CLUSTER); \
159+
exit $$E2E_STATUS
160+
118161
.PHONY: ssl
119162
ssl:
120163
SSL_DIR=ssl ./scripts/configure-ssl.sh

‎docs/engines/k8s.md‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -344,6 +344,52 @@ tests:
344344
enabled: false
345345
```
346346

347+
### Conformance testing with Chainsaw
348+
349+
`helm test` verifies that a deployed instance is healthy. To verify that the engine actually **applies correct topology labels** to nodes, use the Chainsaw E2E suite in `tests/chainsaw/`.
350+
351+
[Chainsaw](https://kyverno.github.io/chainsaw/) is Kyverno's declarative E2E framework. Each suite drives `apply → wait → assert → cleanup` against a real cluster using the built-in **test provider** — no cloud credentials required.
352+
353+
#### Test suites
354+
355+
| Suite | What it checks |
356+
|---|---|
357+
| `k8s/label-application` | `leaf`, `spine`, and `accelerator` labels applied to nodes after generation |
358+
| `k8s/label-truncation` | Switch names >63 chars replaced with an FNV64a hash (valid label value) |
359+
| `k8s/node-observer` | Node Observer auto-triggers generation when it starts and detects the fake node objects |
360+
| `slinky/configmap-output` | Slinky engine writes correct `topology.conf` into a ConfigMap |
361+
362+
#### How suites map topology to nodes
363+
364+
Each suite ships a `topology-model.yaml` in its directory. The suite creates
365+
fake K8s Node objects whose names match the node IDs in that file, loads the
366+
file into a ConfigMap, and mounts it at `/etc/topograph/models/` inside the
367+
pod. The `/v1/generate` request passes `modelFileName` pointing at the mounted
368+
file. No node annotations are required.
369+
370+
#### Running locally
371+
372+
```bash
373+
# Prerequisites: chainsaw, kind, helm, kubectl, docker
374+
375+
# Full lifecycle — build, create cluster, run all suites, delete cluster:
376+
make e2e-local
377+
378+
# Against an existing local kind cluster (repeat after each commit):
379+
make image-build # rebuild with the current commit SHA tag
380+
make kind-load KIND_CLUSTER=<cluster-name> # load into the cluster
381+
make e2e
382+
383+
# Single suite only:
384+
chainsaw test --test-dir tests/chainsaw/k8s/label-application
385+
```
386+
387+
See `tests/chainsaw/README.md` for full prerequisites and environment variable reference.
388+
389+
#### Running in CI
390+
391+
The `.github/workflows/e2e.yml` workflow runs on `workflow_dispatch`. Trigger it manually from the GitHub UI before merging changes to the Helm chart, Node Observer, or engine output code paths.
392+
347393
### Chart README
348394

349395
For installation, prerequisites, values reference, and configuration examples, see [`charts/topograph/README.md`](../../charts/topograph/README.md) — also surfaced via `helm show readme topograph/topograph`.

0 commit comments

Comments
 (0)