Skip to content

Fix five JS/Py differential-corpus divergences (P1–P5) - #40

Open
mikkel wants to merge 4 commits into
mainfrom
fix/differential-corpus-parity
Open

mikkel wants to merge 4 commits into
mainfrom
fix/differential-corpus-parity

Conversation

@mikkel

@mikkel mikkel commented Nov 20, 2026

Copy link
Copy Markdown
Member

What this is

The Python engine diverged from the JS reference implementation (255BITS/patchling) on five cases in JS's own benchmark corpus (128 examples / 135 file-patches, deterministic apply). This PR fixes all five. Each bug has a regression test in tests/test_differential_corpus_regressions.py that fails on the old code and passes on the new code (verified per commit).

Differential corpus: Py 109 correct / 3 wrong / 23 fallback → 112 / 0 / 23 — exact parity with JS, including identical fallback and wrong sets. A follow-up differential fuzz round (800 generated cases: markdown-list deletions, ++/-- body lines, fence content, no-newline markers) shows 0 divergences.

The bugs

  • P2 — silent truncation (worst). Deleting a markdown list item produces the body line -- [ ] Write applier. parse_diff_per_file's fallback strategy treated any -{2,3} /+{2,3} line as a file header, split the patch there, and discarded the tail — the change itself. The deterministic applier then returned the nearly-original file as a successful apply: the edit silently never happened.
  • P5 — wrong apply on a -- body line. Same split: the patch head alone applied to a wrong non-null result instead of refusing.
  • P4 — patch re-keyed by body text (3 cases). A ++ console.error(msg); body line was treated as a to-header, so the patch came back keyed console.error(msg); instead of src/logger.js.
  • P3 — duplicated last line. _parse_hunks discarded \ No newline at end of file markers entirely, so git's "last line loses its newline" pattern (kept line as context + restating add) duplicated const Retries = 3.
  • P1 — needless LLM fallback. _body_op skipped every line starting with ``` — including content lines of a markdown file — desyncing the hunk's context block, so a pure append at EOF returned None where JS applies deterministically.

The fixes (one commit each, mirroring the JS sibling's mechanisms)

  1. P3: _parse_hunks_detailed keeps per-op no_newline flags and the @@ declared new-line count; the collapse rule drops a redundant restating add when the declared count confirms it; assembly tracks the last emitted line's newline state. Two test_smartapply new-file expectations were updated to the JS sibling's byte-exact deterministic output (created files end with \n — the JS test for the identical case asserts exactly that).
  2. P2/P5: a from-header counts only when the next line is a to-header; the pair is consumed together; a standalone +++ line is never a header (the parseDiffPerFile rule).
  3. P4: regression pins for the parser half of the same port.
  4. P1: fence lines inside a hunk are content and classify by their diff prefix; wrapping fences are already stripped by extract_diff_blocks.

Verification

  • Py suite: 268 passed / 2 skipped (baseline 263 passed; +7 new regression tests, +2 updated expectations). Two failures in this sandbox are pre-existing environment flakes, proven to fail identically on the unmodified tree: test_cli_version (the CLI's cold import takes ~10.9s here vs the test's 10s subprocess timeout) and test_relevance_cache_concurrency (multiprocess timing assertion). Neither touches code changed here.
  • JS suite (untouched): 274 passed / 0 failed / 5 skipped.
  • Speed (same workloads, before → after): corpus median 12.42 → 8.57 ms/pass; 20k-line/100-hunk parse+apply median 606.7 → 469.8 ms. No regression; the profile-guided parser optimization is a separate follow-up.

Not merged — awaiting review.

Nova (Muse) added 4 commits November 20, 2026 18:31
Differential corpus (255BITS/patchling benchmark, 128 examples / 135
file-patches) divergence P3: for git's 'last line loses its newline'
pattern written with the kept line as context plus a restating add
(no-trailing-newline-delete-trailing-line), Python duplicated the last
kept line ('const Retries = 3' twice) where JS applies cleanly.

_parse_hunks discarded '\ No newline at end of file' markers entirely,
and apply_patch_to_content assembled output using only the original's
trailing-newline bit. Port the JS sibling's (applyPatchToFile) state:
- _parse_hunks_detailed keeps per-op no_newline flags and the @@
  header's declared new-line count; _parse_hunks keeps its old shape.
- The collapse rule: drop a final no-newline add that restates the last
  context line across only deletions when the declared count confirms
  the collapsed reading.
- Assembly tracks whether the last emitted line ends without a newline
  (marker, or the original's own unmodified last line), so an append
  after a newline-less last line gains the trailing newline JS emits.

Two test_smartapply new-file expectations updated to the JS sibling's
byte-exact deterministic output (created files end with '\n'; the JS
test for the identical case asserts the trailing newline).

Corpus after: Py 110 correct / 2 wrong / 23 fallback (was 109/3/23).
Regression test fails on the old code, passes on the new code.
Differential corpus divergences P2 and P5 share one root in
parse_diff_per_file's fallback strategy: any line matching '-{2,3} ' or
'+{2,3} ' was treated as a file header, including hunk *body* lines.

P2 (silent truncation, worst of the five): deleting a markdown list
item produces the body line '-- [ ] Write applier'. The parser split
the patch there, discarded the tail (the change itself), and the
deterministic applier returned the nearly-original file as a
successful apply — the edit silently never happened.

P5: the body line '--  debug: true,' split the patch the same way and
the head alone applied to a wrong, non-null result.

Port the JS sibling's rule (parseDiffPerFile): a from-header counts
only when the next line is a to-header; the pair is consumed together
and a standalone '+++' line is never a header. That also removes the
standalone to-header branch that re-keyed patches by body text (P4's
parser half; its regression pins land separately).

Corpus after: Py 111 correct / 0 wrong / 24 fallback (was 110/2/23).
Both new regression tests fail on the old parser, pass on the new one.
Differential corpus divergence P4 (three cases): a doubled addition
marker in a hunk body ('++  console.error(msg);') was treated as a
to-header by parse_diff_per_file's fallback strategy, so the patch
came back keyed by the body text ('console.error(msg);',
'return res.json();', 'sum += item.price * item.qty;') instead of the
real path — smartapply then looked up a file that does not exist.

The mechanism fix landed with the P2/P5 header-pairing port (a
standalone '+++' line is never a header); this commit pins it: the
patch stays keyed by the real path and apply_patch_to_content refuses
the ambiguous body line (None), matching JS. All three tests fail on
the pre-fix parser and pass now.
Differential corpus divergence P1 (append-prepend-readme-footer):
_body_op skipped every line starting with ```, including *content*
lines of the file being patched — the context lines ' ```bash' and
' ```' of a README. Dropping them desynced the hunk's context block
from the file, so a pure append at EOF returned None (LLM fallback)
where JS applies deterministically.

Wrapping fences are already stripped before this point by
extract_diff_blocks in parse_diff_per_file; inside a hunk, a fence
line classifies by its diff prefix like any other line, as in the JS
sibling.

Corpus after: Py 112 correct / 0 wrong / 23 fallback — exact parity
with JS, including identical fallback and wrong sets.
Regression test fails on the old _body_op, passes on the new one.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant