Skip to content

Widen live CI, and fix two proxy-failure defects it turned up - #16

Merged
jehrr merged 2 commits into
mainfrom
p2-ci-expansion
Sep 21, 2026
Merged

jehrr merged 2 commits into
mainfrom
p2-ci-expansion

Conversation

@jehrr

@jehrr jehrr commented Sep 21, 2026

Copy link
Copy Markdown
Collaborator

The remaining half of the audit's "CI is narrow" finding, plus two defects
that looking for its "proxy failure semantics" test turned up.

The two defects

The proxy-failure classifier was declared three times. Each engine
carried its own copy of the Chromium marker tuple and inlined its own match
against it. They were byte-identical when this was written, which is the
state a drift starts from rather than a defence against one. It is now
page_flow.proxy_failure(), which takes the exception rather than
pre-extracted text so no caller can forget to stringify a driver's own
exception type.

And the responses had already drifted. A dead proxy and a timeout want
opposite answers — another try at the same exit, versus a different exit —
and pyppeteer rotated on any load failure:

playwright / selenium   if exit_failed and block_attempt < block_retries
pyppeteer               if load_failed and block_attempt < block_retries

So an ordinary network flap spent a --proxy-block-retries budget and
re-fetched the page through a fresh browser on another exit, while its twins
gave up and reported page_load_timeout. Three engines disagreeing about
what a timeout means, on the one invariant that names this exact confusion.

The CI half

weekly.yml covers what the daily canary does not: every mode (product,
reviews, a best-seller grid), a second and third marketplace, and the two
non-primary engines over two pages each. The assertions are chosen to be the
thing each one can be wrong about while everything else still looks right —
the currency per marketplace, page+position uniqueness per engine,
record_type and reviews that actually carry an id and a body.

The query is per-marketplace, because a German term sent to amazon.co.uk
still returns something, and a run that looks fine while exercising a query
nobody would type is worse than one that fails.

Dispatch-only, for the same reason the variation canary was: nobody has
watched these go green. The evidence that does exist is written into the
header along with what it does not cover, and the cron line sits
commented beneath it.

What is promoted

variation-canary moves to the daily schedule, on evidence: two dispatches
from GitHub runners with no proxy and no secret, both green.

Those two runs are also why its assertion stays a floor rather than the
measured figure. The same product reported 776 variants from a
workstation, then 744, then 745 from the runners ten minutes apart.
One product, three numbers, two hours — a canary pinned to any of them would
go red for the catalogue behaving normally.

Verification

  • smoke_test.py: OK. .github/ci_checks.py --all: clean.
  • Every new check controlled. Restoring pyppeteer's old rotation condition
    reddened both of its parity checks and named it.
  • The inline-Python check added in the previous PR picked up all three of
    the new heredocs on its first run, which is that guard doing the job it
    was written for a day earlier.

🤖 Generated with Claude Code

…timeout

Looking for the audit's "proxy failure semantics" test turned up two things
neither the audit nor any check here had noticed.

The classifier was declared THREE times: each engine carried its own copy of
the Chromium marker tuple and inlined its own match against it. They were
byte-identical when this was written, which is the state a drift starts
from, not a defence against one — the same argument that already put the
retry and page-state policy in page_flow. It is now
page_flow.proxy_failure(), which takes the exception rather than
pre-extracted text so no caller can forget to stringify a driver's own
exception type.

And the responses HAD already drifted. A dead proxy and a timeout want
opposite answers — another try at the same exit, versus a different exit —
and pyppeteer rotated on ANY load failure:

    playwright / selenium   if exit_failed and block_attempt < block_retries
    pyppeteer               if load_failed and block_attempt < block_retries

So an ordinary network flap spent a --proxy-block-retries budget and
re-fetched the page through a fresh browser on another exit, while its twins
gave up and reported page_load_timeout. Three engines disagreeing about what
a timeout means, on the one invariant that names this exact confusion.
pyppeteer now records which failure it was and rotates only for a proxy one,
matching its twins line for line.

Checks: the classifier against real Chromium error text for each marker and
against four things that are NOT a proxy problem (a timeout,
ERR_NAME_NOT_RESOLVED, ERR_CONNECTION_RESET, the empty string), that an
exception object classifies like its text, and per engine that it calls the
shared helper, keeps no private tuple, and rotates on exit_failed rather
than load_failed. That last pair is the one that would have caught the
drift. Controlled by restoring pyppeteer's old condition: both of its
checks went red and named it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… cron

The daily canary covers one path — a listing run on amazon.com through
Playwright. That is the right thing to run every morning, but it is also why
both of the defects fixed this week were invisible to CI: no detail page was
ever fetched, so `variations` could return null on every run for months, and
no engine but Playwright was ever run live, so pyppeteer could rotate a
proxy exit on an ordinary timeout without anyone noticing.

weekly.yml is the broader, slower half: every mode (product, reviews, a
best-seller grid), a second and third marketplace, and the two non-primary
engines over two pages each. The assertions are chosen to be the thing that
mode can be wrong about while everything else looks right —

  reviews        record_type, and reviews that actually carry an id and a body
  bestsellers    row count, and page+position unique across the run
  marketplaces   the CURRENCY, per host, because a defaulted "USD" on
                 amazon.de is the exact shape of that bug
  engines        two pages completed, and page+position unique, which is how
                 a page number that never reaches the parser shows up

— and the query is per-marketplace, because a German term sent to
amazon.co.uk still returns something, and a run that looks fine while
exercising a query nobody would type is worse than one that fails.

Dispatch-only, for the same reason the variation canary was: nobody has
watched these go green, and a live job nobody has watched does not belong on
a cron. The evidence that exists is written into the header, along with what
it does NOT cover, and the promotion steps are the cron line sitting
commented beneath them.

The variation canary itself IS promoted to the schedule now, on evidence:
two dispatches from GitHub runners with no proxy and no secret, both green.
Those two runs are also why its assertion stays a floor — the same product
reported 776 variants from a workstation, then 744, then 745 from the
runners ten minutes apart. One product, three numbers, two hours.

The inline-Python check added earlier picked up all three of the new
heredocs on its first run, which is the guard doing the job it was written
for a day ago.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@jehrr
jehrr merged commit 1e594ec into main Sep 21, 2026
7 checks passed
@jehrr
jehrr deleted the p2-ci-expansion branch September 21, 2026 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants