Skip to content

Commit 4c3c0ac

Browse files
taratoriotaratorio
authored
tools, makefile, .github/workflows: extract only the shard's EEST fixture set (erigontech#21771)
## Problem `eest-spec-enginextests-benchmark-30m-parallel` failed on erigontech#21764 with the Go linker dying mid-build ([run](https://github.com/erigontech/erigon/actions/runs/27421370747/job/81047607869)): ``` link: mapping output file failed: no space left on device ``` GitHub's `ubuntu-24.04` hosted pool currently serves two disk flavors — 72G and 145G root volumes. The same shard on the same branch passed 45 minutes earlier on a 145G runner (110G free after cleanup) and failed on a 72G one (36G free after cleanup). The job's disk demand sits right at that 36G line because every `eest-spec-*` shard provisions **all three** EEST corpora through the blanket `test-fixtures-eest` Make target, while `tools/run-eest-spec-test.sh` only ever reads one of them: | corpus | tarball | extracted | |---|---|---| | eest_benchmark | 1.0G | 5.3G | | eest_devnet | 0.6G | 10G | | eest_stable | 0.4G | 8.1G | 36G free − ~8G (Go toolchain + mod/build cache restore) − 2G tarballs − 23.4G extraction ≈ 2.6G left when `go build` starts → ENOSPC at link. The ±1-2G run-to-run variance is why it flakes instead of failing deterministically. ## Fix - Move fixture provisioning into `tools/run-eest-spec-test.sh`, which already resolves the shard's corpus from its name — each shard now downloads/extracts only the set it reads. Worst-case footprint drops from ~35G to ~22G (devnet shards), ~17G for the benchmark shard that flaked. - The zkevm/non-zkevm Makefile rule split existed solely to pick the fixture prerequisite, so the four pattern rules collapse to two (race/non-race). - Add `--download-only` to `tools/test-fixtures.sh` and use it in `cache-warming-eest-fixtures.yml`. The warmer caches only the `.tar.gz` files but was extracting all four corpora (~27G) — on a runner that doesn't even run the cleanup-space step, i.e. the same ENOSPC waiting to happen on a 72G runner whenever the probe misses. Local `make eest-spec-*` behaviour is unchanged: provisioning is a sentinel-guarded no-op when the corpus is already extracted, and `make test-fixtures-eest`/`test-fixtures-zkevm` remain for prefetching everything. ## Verification TDD note: infra/shell change with no Go behaviour; verified functionally instead of via Go tests: - Synthetic end-to-end test of `test-fixtures.sh` (tiny `file://` manifest in a temp dir): download-only cold / idempotent re-run / normal-mode extract from existing tarball / sentinel no-op / download-only re-download after tarball deletion — all pass. - `make -n` for one shard per class (plain, race, zkevm-race, benchmark): correct binary (`evm` vs `evm.race`), no blanket fixture extraction in any recipe. - Smoke run of `run-eest-spec-test.sh statetests-stable` against a populated cache: in-script provisioning hits the cached fast-path and proceeds into the run stage. - `actionlint` on both workflows; `shellcheck` adds no new findings; `make lint` clean twice.
1 parent db998ff commit 4c3c0ac

5 files changed

Lines changed: 65 additions & 44 deletions

File tree

‎.github/workflows/cache-warming-eest-fixtures.yml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -60,10 +60,12 @@ jobs:
6060
lookup-only: true
6161

6262
# Download all four tarballs in one invocation so the single cached entry
63-
# is complete (a shard run only fetches the subset it needs).
63+
# is complete (a shard run only extracts the set it needs). Download-only:
64+
# the cache stores just the tarballs, and extracting all corpora (20G+)
65+
# doesn't fit on the smaller hosted runners.
6466
- name: Download EEST tarballs
6567
if: steps.probe.outputs.cache-hit != 'true'
66-
run: bash tools/test-fixtures.sh test-fixtures.json test-fixtures-cache eest_stable eest_devnet eest_benchmark eest_zkevm
68+
run: bash tools/test-fixtures.sh --download-only test-fixtures.json test-fixtures-cache eest_stable eest_devnet eest_benchmark eest_zkevm
6769

6870
- name: Save EEST cache
6971
if: steps.probe.outputs.cache-hit != 'true'

‎.github/workflows/test-eest-spec.yml‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -84,9 +84,10 @@ jobs:
8484
if: '!inputs.cache-warming-only'
8585
uses: actions/cache/restore@v5
8686
with:
87-
# Only the .tar.gz files are cached; tools/test-fixtures.sh re-extracts
88-
# them each run. cl_mainnet is intentionally excluded (owned by
89-
# test-integration-caplin.yml's separate cache).
87+
# Only the .tar.gz files are cached; each shard re-extracts just the
88+
# fixture set it needs (tools/run-eest-spec-test.sh). cl_mainnet is
89+
# intentionally excluded (owned by test-integration-caplin.yml's
90+
# separate cache).
9091
path: |
9192
test-fixtures-cache/eest_stable.tar.gz
9293
test-fixtures-cache/eest_devnet.tar.gz

‎Makefile‎

Lines changed: 8 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -275,9 +275,9 @@ test-fixtures-zkevm:
275275
# .github/workflows/test-eest-spec.yml's load-matrix job and
276276
# tools/run-eest-spec-test.sh's runtime lookup). Shards whose names contain
277277
# "-race" dispatch through the race-instrumented evm.race binary so race
278-
# coverage works without polluting the non-race shards. zkevm-* shards provision
279-
# the eest_zkevm corpus (test-fixtures-zkevm); all others provision the
280-
# eest_{stable,devnet,benchmark} corpora (test-fixtures-eest).
278+
# coverage works without polluting the non-race shards. Each shard provisions
279+
# only its own fixture set (via tools/run-eest-spec-test.sh); all corpora
280+
# together are 20G+ extracted and don't fit on the smaller CI runner disks.
281281
.PHONY: evm.race
282282
evm.race:
283283
$(GO_BUILD_ENV) $(GO) build -race $(GO_FLAGS) -tags $(BUILD_TAGS) -o $(GOBIN)/evm.race ./cmd/evm
@@ -287,23 +287,15 @@ evm.race:
287287
ifneq ($(filter eest-spec-%,$(MAKECMDGOALS)),)
288288
$(call check_tools,yq jq,eest-spec targets)
289289

290-
EEST_SPEC_RACE_SHARDS := $(shell yq -o=json '.' tools/eest-spec-shards.yml | jq -r '.[].shard | select(test("-race")) | select(test("^zkevm") | not)')
291-
EEST_SPEC_SHARDS := $(shell yq -o=json '.' tools/eest-spec-shards.yml | jq -r '.[].shard | select(test("-race") | not) | select(test("^zkevm") | not)')
292-
EEST_SPEC_ZKEVM_RACE_SHARDS := $(shell yq -o=json '.' tools/eest-spec-shards.yml | jq -r '.[].shard | select(test("^zkevm")) | select(test("-race"))')
293-
EEST_SPEC_ZKEVM_SHARDS := $(shell yq -o=json '.' tools/eest-spec-shards.yml | jq -r '.[].shard | select(test("^zkevm")) | select(test("-race") | not)')
290+
EEST_SPEC_RACE_SHARDS := $(shell yq -o=json '.' tools/eest-spec-shards.yml | jq -r '.[].shard | select(test("-race"))')
291+
EEST_SPEC_SHARDS := $(shell yq -o=json '.' tools/eest-spec-shards.yml | jq -r '.[].shard | select(test("-race") | not)')
294292

295-
.PHONY: $(addprefix eest-spec-,$(EEST_SPEC_SHARDS)) $(addprefix eest-spec-,$(EEST_SPEC_RACE_SHARDS)) $(addprefix eest-spec-,$(EEST_SPEC_ZKEVM_SHARDS)) $(addprefix eest-spec-,$(EEST_SPEC_ZKEVM_RACE_SHARDS))
293+
.PHONY: $(addprefix eest-spec-,$(EEST_SPEC_SHARDS)) $(addprefix eest-spec-,$(EEST_SPEC_RACE_SHARDS))
296294

297-
$(addprefix eest-spec-,$(EEST_SPEC_SHARDS)): eest-spec-%: test-fixtures-eest evm
295+
$(addprefix eest-spec-,$(EEST_SPEC_SHARDS)): eest-spec-%: evm
298296
@bash tools/run-eest-spec-test.sh "$*"
299297

300-
$(addprefix eest-spec-,$(EEST_SPEC_RACE_SHARDS)): eest-spec-%: test-fixtures-eest evm.race
301-
@EVM_BIN=$(GOBIN)/evm.race bash tools/run-eest-spec-test.sh "$*"
302-
303-
$(addprefix eest-spec-,$(EEST_SPEC_ZKEVM_SHARDS)): eest-spec-%: test-fixtures-zkevm evm
304-
@bash tools/run-eest-spec-test.sh "$*"
305-
306-
$(addprefix eest-spec-,$(EEST_SPEC_ZKEVM_RACE_SHARDS)): eest-spec-%: test-fixtures-zkevm evm.race
298+
$(addprefix eest-spec-,$(EEST_SPEC_RACE_SHARDS)): eest-spec-%: evm.race
307299
@EVM_BIN=$(GOBIN)/evm.race bash tools/run-eest-spec-test.sh "$*"
308300
endif
309301

‎tools/run-eest-spec-test.sh‎

Lines changed: 13 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -66,15 +66,16 @@ for tool in yq jq; do
6666
command -v "$tool" >/dev/null 2>&1 || { echo "run-eest-spec-test: required tool '$tool' not found in PATH" >&2; exit 1; }
6767
done
6868

69-
# Resolve fixtures base from the shard name. blocktests-{stable,devnet}-race-*
69+
# Resolve the fixture set from the shard name. blocktests-{stable,devnet}-race-*
7070
# inherit from the parent shard (stable/devnet).
7171
case "$shard" in
72-
*zkevm*) base=test-fixtures-cache/eest_zkevm/fixtures ;;
73-
*-stable*) base=test-fixtures-cache/eest_stable/fixtures ;;
74-
*-devnet*) base=test-fixtures-cache/eest_devnet/fixtures ;;
75-
*-benchmark*) base=test-fixtures-cache/eest_benchmark/fixtures ;;
72+
*zkevm*) fixtures=eest_zkevm ;;
73+
*-stable*) fixtures=eest_stable ;;
74+
*-devnet*) fixtures=eest_devnet ;;
75+
*-benchmark*) fixtures=eest_benchmark ;;
7676
*) echo "cannot resolve fixtures for shard: $shard" >&2; exit 2 ;;
7777
esac
78+
base=test-fixtures-cache/$fixtures/fixtures
7879

7980
# Resolve workers + failure budget + exec3-parallel flag from the single-source
8081
# manifest. Both this script and the test-eest-spec.yml load-matrix job read
@@ -146,10 +147,14 @@ if [[ ! -x "$evm_bin" ]]; then
146147
echo "$evm_bin not found or not executable; run 'make evm' first" >&2
147148
exit 2
148149
fi
150+
151+
# Provision only this shard's fixture set (no-op when already extracted) —
152+
# all corpora together are 20G+ extracted and don't fit on the smaller CI
153+
# runner disks.
154+
bash tools/test-fixtures.sh test-fixtures.json test-fixtures-cache "$fixtures"
155+
149156
if [[ ! -d "$path" ]]; then
150-
fixtures_target=test-fixtures-eest
151-
case "$shard" in *zkevm*) fixtures_target=test-fixtures-zkevm ;; esac
152-
echo "fixture path $path does not exist; run 'make $fixtures_target' first" >&2
157+
echo "fixture path $path does not exist in the $fixtures corpus" >&2
153158
exit 2
154159
fi
155160

‎tools/test-fixtures.sh‎

Lines changed: 36 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -12,12 +12,19 @@
1212
# 2x local disk usage vs. extracted-only; CI caches only *.tar.gz to stay
1313
# under cache budget and re-extracts on each run.
1414
#
15-
# Usage: tools/test-fixtures.sh [MANIFEST [CACHE_DIR [KEY...]]]
15+
# Usage: tools/test-fixtures.sh [--download-only] [MANIFEST [CACHE_DIR [KEY...]]]
16+
# --download-only download & sha256-verify the tarballs, skip extraction
1617
# MANIFEST defaults to test-fixtures.json
1718
# CACHE_DIR defaults to test-fixtures-cache
1819
# KEY... optional manifest keys; default = all keys
1920
set -euo pipefail
2021

22+
download_only=false
23+
if [[ "${1:-}" == "--download-only" ]]; then
24+
download_only=true
25+
shift
26+
fi
27+
2128
manifest="${1:-test-fixtures.json}"
2229
cache="${2:-test-fixtures-cache}"
2330
keys=("${@:3}")
@@ -68,30 +75,44 @@ tar_path=""
6875
out_dir=""
6976
trap 'rm -f "${tar_path}.tmp" 2>/dev/null; rm -rf "${out_dir}.tmp" 2>/dev/null' EXIT
7077

78+
# ensure_tarball NAME URL WANT: download & verify $tar_path unless it already
79+
# matches WANT.
80+
ensure_tarball() {
81+
if [[ -f "$tar_path" ]] && [[ "$(sha256 "$tar_path")" == "$3" ]]; then
82+
echo "$1: tarball cached (sha256 ${3:0:12})"
83+
return 0
84+
fi
85+
echo "$1: downloading from $2"
86+
# Ride out transient upstream 5xx (release CDN) with exponential backoff over a 5-min window.
87+
curl -fsSL --retry 10 --retry-all-errors --retry-delay 0 \
88+
--retry-max-time 300 --connect-timeout 30 \
89+
-o "$tar_path.tmp" "$2"
90+
local got
91+
got=$(sha256 "$tar_path.tmp")
92+
if [[ "$got" != "$3" ]]; then
93+
rm -f "$tar_path.tmp"
94+
echo "test-fixtures: $1: sha256 mismatch — want $3 got $got" >&2
95+
exit 1
96+
fi
97+
mv "$tar_path.tmp" "$tar_path"
98+
}
99+
71100
while IFS=$'\t' read -r name url want; do
72101
tar_path="$cache/${name}.tar.gz"
73102
out_dir="$cache/${name}"
74103
sentinel="$out_dir/.sha256"
75104

105+
if [[ "$download_only" == true ]]; then
106+
ensure_tarball "$name" "$url" "$want"
107+
continue
108+
fi
109+
76110
if [[ -f "$sentinel" ]] && [[ "$(cat "$sentinel")" == "$want" ]]; then
77111
echo "$name: cached (sha256 ${want:0:12})"
78112
continue
79113
fi
80114

81-
if [[ ! -f "$tar_path" ]] || [[ "$(sha256 "$tar_path")" != "$want" ]]; then
82-
echo "$name: downloading from $url"
83-
# Ride out transient upstream 5xx (release CDN) with exponential backoff over a 5-min window.
84-
curl -fsSL --retry 10 --retry-all-errors --retry-delay 0 \
85-
--retry-max-time 300 --connect-timeout 30 \
86-
-o "$tar_path.tmp" "$url"
87-
got=$(sha256 "$tar_path.tmp")
88-
if [[ "$got" != "$want" ]]; then
89-
rm -f "$tar_path.tmp"
90-
echo "test-fixtures: $name: sha256 mismatch — want $want got $got" >&2
91-
exit 1
92-
fi
93-
mv "$tar_path.tmp" "$tar_path"
94-
fi
115+
ensure_tarball "$name" "$url" "$want"
95116

96117
echo "$name: extracting"
97118
rm -rf "$out_dir.tmp" "$out_dir"

0 commit comments

Comments
 (0)