Scraper API: send waitFor as an object, read the target status from http_code - #19
Merged
Merged
Conversation
…ttp_code
Measured 2026-09-23 against the live /tasks/sync endpoint: waitFor sent
as a JSON-encoded string is answered HTTP 422 "params.waitFor must be an
object" and still billed ($0.0005); as an object it is answered 200.
The response's `status` is the API's own verdict ("success"); the
target's HTTP code is `http_code`, which is now what reaches the page
classifier (fallback to `status` only when it is an int).
Adds one offline regression check, controlled: with the old client the
suite goes red on exactly that check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two defects in
scraper_api_client.pyMeasured 2026-09-23 against
https://scraper.2captcha.com/tasks/sync:waitForsent as a JSON-encoded string (what_build_wait_forreturned) is answered HTTP 422ScrapeParser: params.waitFor must be an object— and the call is still billed ($0.0005). The same request with an object answers 200. So every run with--wait-text/--wait-element/--wait-statefailed with exit 5 and cost money._build_wait_fornow returns a dict (logged throughjson.dumps), and the docstring that said the API wanted a double-encoded string now says what was measured.status, which is the API's own verdict string ("success"). The target site's HTTP code ishttp_code. This client LOGS that value (it does not classify on it), so a target 403/503 read "success" in the log. It now readshttp_code(int), falling back tostatusonly if that is itself an int.Regression check
test_scraper_api_sends_waitfor_as_an_object_and_reads_http_codeinsmoke_test.py: drives the real function withrequests.poststubbed (no network), returning{"status":"success","http_code":403,"body":"<html></html>"}, and asserts (a) the payload'swaitForis a dict and (b) the status it reads is403(int).Control: (registered with
ok &= …inmain()) with origin/main'sscraper_api_client.pyswapped into this branch's tree (file hash confirmed changed), the suite went red with exactly 2 failures, both this check's messages (waitForcame back as the string'{"text": "results"}', the logged status as['success']). Restored:smoke_test: OK. This client'sfetch_htmlreturns only the HTML and LOGS the status, so part (b) asserts on the captured log record rather than a return value.Live result
One call, key from the environment:
scraper_api_client.py --url 'https://www.amazon.com/s?k=wireless+headphones' --wait-text results→ API HTTP 200 (no 422), logUpstream page HTTP status 200, 16 rows, exit 0, sidecar status complete.Not changed
Unlike its siblings, this client does not hand the target status to a page classifier at all — it only logs it — and this PR does not change that (it would be a behaviour change beyond this fix; worth its own PR). The module docstring's API-surface block now shows the object form and the real response shape. The browser engines and
product_parser.pyare untouched. No version bump — the CHANGELOG entry is under[Unreleased]. Reference fix: zimmo-scraper PR #1.🤖 Generated with Claude Code