feat(batch): verify webhook signatures and match the API's result shapes

[emirlan-zen]

What this changes

This PR brings the Batch client in line with what the Batch API returns, and adds a webhook signature check. It is additive: no exported name, field, type, signature or accepted input is removed or narrowed. See "Compatibility" for how that was checked.

• New: verify_webhook(raw_body, signature_header, secrets) checks the X-Signature header (t=...,v1=...,kid=...) on a signed delivery.

  • It base64-decodes the secret the API showed you and picks it by kid, so it keeps working during a key rotation.
  • It computes HMAC-SHA256 over <t>.<raw body> and compares in constant time.
  • It rejects deliveries more than 300 seconds old. Only tolerance_seconds=0 disables the age check.
  • A negative, non-finite or oversized tolerance rejects every delivery, and so does a NaN, infinite or non-numeric now, even with the age check off.
  • A secret that doesn't decode to exactly 32 bytes (the length the API issues) never verifies. An empty secret would otherwise let anyone forge a valid signature.
  • It never raises on bad input. A malformed header, a non-hex or non-ASCII digest, a bad timestamp, a missing kid, an unreadable secrets mapping, a body that can't be encoded and wrong-typed arguments all return False.

• Bulk downloads no longer stop part-way. download_to_dir and download_to_memory used to raise ValueError on the first row with no result_url.

  • Rows with no body to fetch are now skipped:
    • failed tasks
    • results past the 14-day visibility window, which the API returns as expired
    • rows not finished yet (pending / processing)
  • One WARNING on zenrows.batch.download gives the counts per status. The return value is still the number of rows downloaded.
  • A successful row with no result_url is not skipped. Its body exists, but the API could not sign a link on that read.
    • The helper fetches that results page once more.
    • If the link is still missing, it raises the new ResultURLUnavailable (with task_ids) instead of dropping a result silently.
    • ResultURLUnavailable subclasses ValueError, so an existing except ValueError around a bulk download still catches it.
  • The single-task helpers still raise ValueError for a row with no URL, as before.

• TaskStatus.EXPIRED is added (an enum widening; see Compatibility).

• add_tasks(..., idempotency_key=...) is a new optional keyword.

  • With a key, a repeat with the same key and body replays the first answer instead of creating or counting the tasks twice, and the transport retries the call on 429/502/503/504.
  • Without a key, behaviour is unchanged: the call is sent once.

• Run.trigger (manual or scheduled) is a new optional field. The API already sends it.

• estimate_cost reads boolean flags the way the API does.

  • The API lowercases boolean params and accepts only "true" / "false". It rejects 1, "1", "yes" and "on" with a 400.
  • The estimator used to price those spellings (and padded strings like " TRUE ") as on, for jobs the API would refuse.
  • Now only True or a string equal to "true" ignoring case turns js_render / premium_proxy on.

• TaskResult.spend and TaskSpend are kept and documented as deprecated.

  • Result rows never carry per-task spend, so the field is always None.
  • Reading it does not warn, so projects that turn warnings into errors keep working.
  • run.stats.spend is the run-level figure.

• README webhook wording. It now says:

  • You get one event per run that completes or fails on its own, and none on /stop.
  • Delivery is best-effort: a failed delivery is retried with backoff for a bounded window, so an event can repeat.
  • Dedupe on X-ZenRows-Event-Id.

  It used to say "at-least-once".

• Docs and docstrings:

  • No fixed export cap; the server's failure reads "results are larger than N gb".
  • Client-side downloads are bound only by the SDK's own max_files / max_bytes_per_file.
  • The non-existent ZENROWS_BATCH_PROGRESS setting is no longer documented.
  • extract_fields works in Batch.
  • Per-task params are zenrows_params.
  • run.stats.spend is indicative recorded spend; the account's usage and invoices are authoritative.
  • Result URLs are valid for 2 hours; /content is a separate endpoint that needs the API key.
  • Target 404 and 410 responses are billed.
  • The API limits task URLs to 2,048 bytes once UTF-8 encoded.

• Spec refresh in docs/openapi.yaml:

  • Task limits of 100,000 per submit and 10,000 per add-tasks call. This widens the model's max_length.
  • 429 quota_exceeded, including when a job stops counting as active: once it is closed and its latest run has finished, or at once when it is deleted.
  • Add-tasks idempotency.
  • The stale "scheduled returns 501" note is removed.

  The models and docs/batch-client-reference.md are regenerated, and the reference gains a "Webhook signatures" section.

Compatibility

Nothing a current caller relies on changes. TaskInput.url and TaskResult.url stay AnyUrl, so .host and passing an AnyUrl still work. TaskSpend and TaskResult.spend stay. Every exported name and every existing signature is unchanged.

griffe check zenrows -s src --against origin/main reports no findings: it prints nothing and exits 0.

Widenings and behaviour changes griffe does not see, disclosed here:

• TaskStatus gains EXPIRED. Unknown values already parsed as UNKNOWN, so code matching on statuses keeps working, but it may now see EXPIRED.
• Request size limits: submit and add-tasks request models accept larger task lists (100,000 and 10,000), matching the API.
• Bulk downloads skip rows with no body instead of raising. They raise ResultURLUnavailable (a ValueError) when a successful row's link stays missing.
• estimate_cost prices 1, "1", "yes" and "on" at the base tier. The API rejects a job sending them.

Testing

All of the following were run at the final head under nice -n 19, one suite at a time.

• make test: 278 passed, 1 deselected (the live e2e test). New tests cover:
  • verify_webhook:
    • a genuine delivery, a tampered body, and the undecoded-secret mistake
    • kid rotation
    • malformed headers and digests
    • non-ASCII and oversized timestamps
    • bad now and tolerance values, including with the age check off
    • an empty-key forgery and wrong-length secrets
    • wrong-typed arguments, an unreadable secrets mapping, and a body that can't be encoded
    • an uppercase digest
    • the tolerance window
  • Bulk downloads:
    • skipping expired, failed, pending and processing rows, serially and with concurrency=4
    • per-status counts
    • re-fetching a successful row with no URL
    • ResultURLUnavailable when the second read still has no URL, and that it is a ValueError
    • the file cap still applying after skips
  • Models:
    • expired parsing with url still an AnyUrl
    • Run.trigger
    • deprecated TaskSpend still importing and parsing without a warning
  • add_tasks: with and without an idempotency key.
  • Estimate: boolean spellings.
• Coverage: pytest --cov=zenrows.batch --cov-report=term-missing gives 88% for the package. webhooks.py is at 100%. No changed executable line in src/zenrows/batch/ is left uncovered; this was checked by intersecting the diff against coverage's missing lines.
• make check (ty, ruff check, ruff format --check): passes.
• Generated files: make generate and make docs were run, and their output is committed.

Needs a human decision

• Billing wording: the README says target 404 and 410 responses are billed. Please confirm before release.
• Lenient task-error parsing, not done: a failed task's error that lacks type, title, status or code still fails validation of the whole results page, as on main. Making those fields optional would fix that, but it changes their types to str | None and breaks strict type checking for current callers, so it is left out here.
• Verbatim URLs, not done: AnyUrl normalises submitted URLs (https://Example.com becomes https://example.com/), so results can't always be matched to inputs by URL string. Fixing this without breaking callers would need a new field, such as a raw URL next to url. It is left out here.
• Release: it goes in as a feat, so the next release is a minor bump.