Skip to content

Shard the backend integration tests and add a per-test timeout - #592

Merged
haksungjang merged 5 commits into
mainfrom
ci/shard-backend-integration
Oct 24, 2026
Merged

haksungjang merged 5 commits into
mainfrom
ci/shard-backend-integration

Conversation

@haksungjang

@haksungjang haksungjang commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

test (backend-integration) took 22 to 37 minutes and was the critical path of every pull request. The integration suite now runs as three jobs, test (backend-integration, shard K), each with its own Postgres and Redis service containers. pytest --shard K/N (tests/_shard.py) splits by whole file, longest first onto the least loaded shard, using weights in tests/integration/shard_durations.json; files keep their collection order inside a shard. I chose a file-level split over pytest-split because it keeps module fixtures and in-file ordering intact, depends only on file paths (parametrised ids need not be stable across processes), and adds no dependency. N is strategy.job-total. Three is enough: the tests add up to about 31 minutes, so a shard is about 10 minutes of tests plus 2 of setup, which is below the unit leg (12 to 21 minutes), now the critical path.

The required check keeps its name. It belongs to an aggregate job (needs: the shards, if: always()) that is red unless the shards' result is success and python -m tests._shard verify confirms that the shards ran every collected file exactly once with the same collection. It also uploads merged durations for refreshing the weights. coverage-gate now needs test and the aggregate, and combines .coverage.backend-unit with .coverage.backend-integration-K-of-N after checking that every shard's file is present. The combined total matches main (91%, 3131 missed statements vs 3135), and 2724 tests pass with 16 skipped (main: 2722 and 16, plus the two tests added here).

For #591, CI passes --timeout=300 --timeout-method=thread (pytest-timeout 2.4.0, hashed in requirements-dev.lock). The slowest test takes 70 s. The thread method fires even when the main thread is blocked in C code, and a conftest hook puts the test id in the main thread's name, so the stack dump is headed MainThread running <test id>, including for hangs in fixtures. This PR does not look for the lock itself.

Verification on this branch, all in throwaway commits that have since been dropped: a test hung on a lock was killed after 20 s with its stack and id printed, and the aggregate went red (failure). A shard cancelled by a one-minute timeout-minutes also turned the aggregate red (cancelled) while the other shards passed. Breaking the audit export's 413 mapping made the new 413 test fail.

Tests that relied on state left by other files, found by running the partitions:

  • test_seed_verify_baseline_webhook_secrets.py failed in a shard where test_scan_scheduler.py (whose TRUNCATE ... CASCADE reaches projects) ran before it with no fresh seed in between. It now drops and reseeds the demo itself.
  • Two code paths were covered only by accumulated rows: the audit CSV export's 413 (the unfiltered export in test_demo_admin_scope_db.py passed 100,000 audit rows) and the catalog refresh skipping a clean row. Each now has a test that reaches it on purpose.

Closes #575
Refs #591

The integration suite runs as three jobs, one whole file per job, weighted
by measured durations. The required check test (backend-integration) is now
an aggregate that is green only when every shard succeeded and the shards
ran every collected file exactly once. pytest-timeout turns a hang into a
failure with every thread's stack, headed by the test id.
Shard weights now come from the first sharded CI run, and the per-test
timeout is set to 300 s, about four times the slowest test. Two paths were
only covered by rows other files left in the shared database (the audit
export 413 and the catalog refresh skipping a clean row); each now has a test
that reaches it on purpose.
@haksungjang
haksungjang force-pushed the ci/shard-backend-integration branch from 77d8eea to 815d6a0 Compare October 24, 2026 17:56
Only the first seed creates scan-pipeline. Once test_scan_scheduler's
TRUNCATE CASCADE has removed the projects, a top-up seed recreates fx-appr
alone, so the test failed in any shard that ran it without a fresh seed in
between. It now drops the demo and seeds it itself.
A file's duration depends on what ran before it in the same database, so
weights from the first run's partition left one shard five minutes shorter
than the others under the second. Reweighting evens the estimate to about
ten minutes of tests per shard.
@haksungjang
haksungjang force-pushed the ci/shard-backend-integration branch from 3cea3f9 to 09c6327 Compare October 24, 2026 18:36
@haksungjang
haksungjang marked this pull request as ready for review October 24, 2026 18:56
A failed shard verify leaves every shard green, so nightly-alert also needs
the aggregator job to report it. Fix the conftest note on where the timeout
method is set.
@haksungjang
haksungjang merged commit c57f3d2 into main Oct 24, 2026
34 checks passed
@haksungjang
haksungjang deleted the ci/shard-backend-integration branch October 24, 2026 19:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Shard the backend integration tests to shorten PR feedback

1 participant