Scraper API: send waitFor as an object, read the target status from http_code

[jehrr]

Two defects in scraper_api_client.py, both measured 2026-09-23 against https://scraper.2captcha.com/tasks/sync.

1. waitFor was sent as a JSON-encoded string. The live API now answers that with HTTP 422 ScrapeParser: params.waitFor must be an object — and still bills the task ($0.0005). The same request with waitFor as an object is answered 200. _build_wait_for now returns a dict (Optional[dict]), logged via json.dumps; its docstring, which said the string form was required, now records the measurement instead.

2. The target site's status was never read. The response is {"status": "success", "http_code": <int>, "headers": …, "body": …}. The client read status — the API's own verdict string — so detect_page_state got "success" instead of the target's 403/503, and this repo's "Wellfound refused the Scraper API's exit" branch could not fire on a status. It now reads http_code, falling back to status only if that is an int.

Check

check_scraper_api_waitfor_is_object_and_status_is_http_code in smoke_test.py (auto-discovered check_ function, defined before the CHECKS list is built) drives the real fetch_html with requests.post stubbed to capture the payload and return {"status":"success","http_code":403,…}, and asserts (a) waitFor is a dict for --wait-text, (b) the status handed onward is 403 (int).

Control: a copy of this branch with scraper_api_client.py restored from origin/main (asserted that the file actually changed) → suite exit 1, both of the new check's messages FAILED (got '{"text": "Python"}' and got 'success'). Real branch → exit 0, 365 checks passed.

Live (one call, key via env)

scraper_api_client.py --url https://wellfound.com/role/r/software-engineer --wait-state load --retries 0 → API HTTP 200 (no 422), Upstream page status (http_code) 200, 567146 bytes, 38 jobs parsed, exit 0.

Worth noting: the module docstring says (measured 2026-09-17) that Wellfound answers the Scraper API's datacentre exits with 403. On this one call on 2026-09-23 it served the page. One call is not enough to rewrite that measurement, so the docstring is left as it is; it deserves a re-measure.

Not changed

• The docstring's 2026-09-17 access measurement (see above).
• No version bump; the entry is under [Unreleased].
• The browser engines and the rest of the client are untouched.

🤖 Generated with Claude Code