Skip to content

fix(tests): order-dependent parser-test isolation — the bare-name import-state leak (#627) - #662

Merged
gadievron merged 2 commits into
masterfrom
fix/627-parser-test-isolation
Sep 21, 2026
Merged

gadievron merged 2 commits into
masterfrom
fix/627-parser-test-isolation

Conversation

@gadievron

@gadievron gadievron commented Sep 21, 2026 •

Copy link
Copy Markdown
Collaborator

What

pytest tests/parsers failed 12 tests on a clean master (4 in test_issue309_py_callgraph_constructs.py, 8 in test_issue440_mro_import_bases.py) while every file alone, the two together, and the full suite passed — the exact receipt from the issue. Closes #627.

Root cause (bisected)

The poisoner is tests/parsers/c/test_empty_seed_keep_all.py. It execs parsers/c/test_pipeline.py, which — like parsers/python/parse_repository.py and the PHP and Ruby pipeline scripts — imports its parser machinery under bare module names (function_extractor, call_graph_builder, repository_scanner, unit_generator) resolved via a sys.path entry. The exec leaves the C versions in sys.modules; a later parse_repository(language="python") then gets the C FunctionExtractor (first-import wins), whose statistics lack standalone_functions — the KeyError behind all 12 failures. The full suite passes only because some earlier test imports the python bare names first: order decides the winner, which is exactly the flakiness class.

The bisection: directory halving with a failing file as the detector → tests/parsers/c → single-file scan → test_empty_seed_keep_all.py; the pair (poisoner + both failing files) reproduces 12/12 deterministically.

The fix (isolation, not a reorder)

Snapshot sys.path and sys.modules at loader ENTRY and restore around the exec — new keys removed, replaced keys restored. Applied to the loaders that exec bare-importing pipelines: c, zig, php, ruby (Go, JavaScript, and Swift use qualified imports and do not poison the bare-name cache; their loaders still leak sys.path state — a broader sweep is noted as follow-up).

Testing

  • The regression test (test_issue627_import_state_isolation.py) reproduces the poisoning within one test — no invocation order needed: exec the C loader, then run a python parse_repository resolution; it asserts the pkg-defined re-export resolves and that the loader leaves no import state.
  • Stash-RED verified: with the fix reverted the regression test fails (2 failed), restored it passes.
  • The receipt inverts: pytest tests/parsers → 918 passed, 1 skipped, 0 failed (was 12 failed, 805 passed).
  • Full suite: 4337 passed, 34 skipped, 2 failed — both test_llm_sdk_contract_floor pins, a local-environment artifact (installed anthropic 1.3.0 vs the requirements.txt pin 1.7.0; CI installs the pin), not master failures. Corrected 2026-09-21.

Coordination

…ort-state leak (#627)

pytest tests/parsers failed 12 tests on a clean master (4 in the
python callgraph-constructs file, 8 in the MRO import-bases file) while
every file alone, the pair together, and the full suite passed. Bisected
to tests/parsers/c/test_empty_seed_keep_all.py: it execs
parsers/c/test_pipeline.py, which (like parsers/python/parse_repository.py
and the PHP and Ruby pipeline scripts) imports its machinery under BARE
module names via a sys.path entry — the exec leaves the C versions in
sys.modules, and a later python parse_repository then gets the C
FunctionExtractor (first-import wins), whose statistics lack
standalone_functions — the KeyError behind all 12 failures. The full
suite passes only because some earlier test imports the python bare
names first: order decides the winner.

The fix is loader isolation, not a test reorder: snapshot sys.path and
sys.modules at ENTRY and restore around the exec (new keys removed,
replaced keys restored) — applied to the loaders that exec
bare-importing pipelines (c, zig, php, ruby; Go/JavaScript/Swift use
qualified imports and do not poison the bare-name cache). The
self-contained regression test reproduces the poisoning WITHIN one
test (no invocation order needed) and pins the no-leak invariant.
The regression test's docstring claimed every parser's pipeline script
uses bare-name imports and that the fix covers two exec sites. Neither
is accurate: Go, JavaScript, and Swift use qualified imports (their
loaders leak sys.path state but do not poison the bare-name cache), and
the fix covers c, zig, php, ruby — the loaders whose pipelines import
bare names.
@gadievron
gadievron merged commit abd1dcf into master Sep 21, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Order-dependent parser-test failures: pytest tests/parsers fails 12 on a clean master while every file, pair, and the full suite pass

1 participant