Widen live CI, and fix two proxy-failure defects it turned up - #16
Merged
Merged
Conversation
…timeout
Looking for the audit's "proxy failure semantics" test turned up two things
neither the audit nor any check here had noticed.
The classifier was declared THREE times: each engine carried its own copy of
the Chromium marker tuple and inlined its own match against it. They were
byte-identical when this was written, which is the state a drift starts
from, not a defence against one — the same argument that already put the
retry and page-state policy in page_flow. It is now
page_flow.proxy_failure(), which takes the exception rather than
pre-extracted text so no caller can forget to stringify a driver's own
exception type.
And the responses HAD already drifted. A dead proxy and a timeout want
opposite answers — another try at the same exit, versus a different exit —
and pyppeteer rotated on ANY load failure:
playwright / selenium if exit_failed and block_attempt < block_retries
pyppeteer if load_failed and block_attempt < block_retries
So an ordinary network flap spent a --proxy-block-retries budget and
re-fetched the page through a fresh browser on another exit, while its twins
gave up and reported page_load_timeout. Three engines disagreeing about what
a timeout means, on the one invariant that names this exact confusion.
pyppeteer now records which failure it was and rotates only for a proxy one,
matching its twins line for line.
Checks: the classifier against real Chromium error text for each marker and
against four things that are NOT a proxy problem (a timeout,
ERR_NAME_NOT_RESOLVED, ERR_CONNECTION_RESET, the empty string), that an
exception object classifies like its text, and per engine that it calls the
shared helper, keeps no private tuple, and rotates on exit_failed rather
than load_failed. That last pair is the one that would have caught the
drift. Controlled by restoring pyppeteer's old condition: both of its
checks went red and named it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… cron
The daily canary covers one path — a listing run on amazon.com through
Playwright. That is the right thing to run every morning, but it is also why
both of the defects fixed this week were invisible to CI: no detail page was
ever fetched, so `variations` could return null on every run for months, and
no engine but Playwright was ever run live, so pyppeteer could rotate a
proxy exit on an ordinary timeout without anyone noticing.
weekly.yml is the broader, slower half: every mode (product, reviews, a
best-seller grid), a second and third marketplace, and the two non-primary
engines over two pages each. The assertions are chosen to be the thing that
mode can be wrong about while everything else looks right —
reviews record_type, and reviews that actually carry an id and a body
bestsellers row count, and page+position unique across the run
marketplaces the CURRENCY, per host, because a defaulted "USD" on
amazon.de is the exact shape of that bug
engines two pages completed, and page+position unique, which is how
a page number that never reaches the parser shows up
— and the query is per-marketplace, because a German term sent to
amazon.co.uk still returns something, and a run that looks fine while
exercising a query nobody would type is worse than one that fails.
Dispatch-only, for the same reason the variation canary was: nobody has
watched these go green, and a live job nobody has watched does not belong on
a cron. The evidence that exists is written into the header, along with what
it does NOT cover, and the promotion steps are the cron line sitting
commented beneath them.
The variation canary itself IS promoted to the schedule now, on evidence:
two dispatches from GitHub runners with no proxy and no secret, both green.
Those two runs are also why its assertion stays a floor — the same product
reported 776 variants from a workstation, then 744, then 745 from the
runners ten minutes apart. One product, three numbers, two hours.
The inline-Python check added earlier picked up all three of the new
heredocs on its first run, which is the guard doing the job it was written
for a day ago.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The remaining half of the audit's "CI is narrow" finding, plus two defects
that looking for its "proxy failure semantics" test turned up.
The two defects
The proxy-failure classifier was declared three times. Each engine
carried its own copy of the Chromium marker tuple and inlined its own match
against it. They were byte-identical when this was written, which is the
state a drift starts from rather than a defence against one. It is now
page_flow.proxy_failure(), which takes the exception rather thanpre-extracted text so no caller can forget to stringify a driver's own
exception type.
And the responses had already drifted. A dead proxy and a timeout want
opposite answers — another try at the same exit, versus a different exit —
and pyppeteer rotated on any load failure:
So an ordinary network flap spent a
--proxy-block-retriesbudget andre-fetched the page through a fresh browser on another exit, while its twins
gave up and reported
page_load_timeout. Three engines disagreeing aboutwhat a timeout means, on the one invariant that names this exact confusion.
The CI half
weekly.ymlcovers what the daily canary does not: every mode (product,reviews, a best-seller grid), a second and third marketplace, and the two
non-primary engines over two pages each. The assertions are chosen to be the
thing each one can be wrong about while everything else still looks right —
the currency per marketplace,
page+positionuniqueness per engine,record_typeand reviews that actually carry an id and a body.The query is per-marketplace, because a German term sent to amazon.co.uk
still returns something, and a run that looks fine while exercising a query
nobody would type is worse than one that fails.
Dispatch-only, for the same reason the variation canary was: nobody has
watched these go green. The evidence that does exist is written into the
header along with what it does not cover, and the cron line sits
commented beneath it.
What is promoted
variation-canarymoves to the daily schedule, on evidence: two dispatchesfrom GitHub runners with no proxy and no secret, both green.
Those two runs are also why its assertion stays a floor rather than the
measured figure. The same product reported 776 variants from a
workstation, then 744, then 745 from the runners ten minutes apart.
One product, three numbers, two hours — a canary pinned to any of them would
go red for the catalogue behaving normally.
Verification
smoke_test.py: OK..github/ci_checks.py --all: clean.reddened both of its parity checks and named it.
the new heredocs on its first run, which is that guard doing the job it
was written for a day earlier.
🤖 Generated with Claude Code