Repository navigation
Shard the backend integration tests and add a per-test timeout - #592
Merged
Merged
Conversation
The integration suite runs as three jobs, one whole file per job, weighted by measured durations. The required check test (backend-integration) is now an aggregate that is green only when every shard succeeded and the shards ran every collected file exactly once. pytest-timeout turns a hang into a failure with every thread's stack, headed by the test id.
Shard weights now come from the first sharded CI run, and the per-test timeout is set to 300 s, about four times the slowest test. Two paths were only covered by rows other files left in the shared database (the audit export 413 and the catalog refresh skipping a clean row); each now has a test that reaches it on purpose.
haksungjang
force-pushed
the
ci/shard-backend-integration
branch
from
October 24, 2026 17:56
77d8eea to
815d6a0
Compare
Only the first seed creates scan-pipeline. Once test_scan_scheduler's TRUNCATE CASCADE has removed the projects, a top-up seed recreates fx-appr alone, so the test failed in any shard that ran it without a fresh seed in between. It now drops the demo and seeds it itself.
A file's duration depends on what ran before it in the same database, so weights from the first run's partition left one shard five minutes shorter than the others under the second. Reweighting evens the estimate to about ten minutes of tests per shard.
haksungjang
force-pushed
the
ci/shard-backend-integration
branch
from
October 24, 2026 18:36
3cea3f9 to
09c6327
Compare
haksungjang
marked this pull request as ready for review
October 24, 2026 18:56
A failed shard verify leaves every shard green, so nightly-alert also needs the aggregator job to report it. Fix the conftest note on where the timeout method is set.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test (backend-integration)took 22 to 37 minutes and was the critical path of every pull request. The integration suite now runs as three jobs,test (backend-integration, shard K), each with its own Postgres and Redis service containers.pytest --shard K/N(tests/_shard.py) splits by whole file, longest first onto the least loaded shard, using weights intests/integration/shard_durations.json; files keep their collection order inside a shard. I chose a file-level split over pytest-split because it keeps module fixtures and in-file ordering intact, depends only on file paths (parametrised ids need not be stable across processes), and adds no dependency. N isstrategy.job-total. Three is enough: the tests add up to about 31 minutes, so a shard is about 10 minutes of tests plus 2 of setup, which is below the unit leg (12 to 21 minutes), now the critical path.The required check keeps its name. It belongs to an aggregate job (
needs:the shards,if: always()) that is red unless the shards' result issuccessandpython -m tests._shard verifyconfirms that the shards ran every collected file exactly once with the same collection. It also uploads merged durations for refreshing the weights. coverage-gate now needstestand the aggregate, and combines.coverage.backend-unitwith.coverage.backend-integration-K-of-Nafter checking that every shard's file is present. The combined total matches main (91%, 3131 missed statements vs 3135), and 2724 tests pass with 16 skipped (main: 2722 and 16, plus the two tests added here).For #591, CI passes
--timeout=300 --timeout-method=thread(pytest-timeout 2.4.0, hashed in requirements-dev.lock). The slowest test takes 70 s. The thread method fires even when the main thread is blocked in C code, and a conftest hook puts the test id in the main thread's name, so the stack dump is headedMainThread running <test id>, including for hangs in fixtures. This PR does not look for the lock itself.Verification on this branch, all in throwaway commits that have since been dropped: a test hung on a lock was killed after 20 s with its stack and id printed, and the aggregate went red (
failure). A shard cancelled by a one-minutetimeout-minutesalso turned the aggregate red (cancelled) while the other shards passed. Breaking the audit export's 413 mapping made the new 413 test fail.Tests that relied on state left by other files, found by running the partitions:
test_seed_verify_baseline_webhook_secrets.pyfailed in a shard wheretest_scan_scheduler.py(whoseTRUNCATE ... CASCADEreachesprojects) ran before it with no fresh seed in between. It now drops and reseeds the demo itself.test_demo_admin_scope_db.pypassed 100,000 audit rows) and the catalog refresh skipping a clean row. Each now has a test that reaches it on purpose.Closes #575
Refs #591