Repository navigation
docs(crawl): say that a crawl out of credits ends stopped, with error set - #56
Open
zenrows-dev-pool[bot] wants to merge 5 commits into
Open
zenrows-dev-pool[bot] wants to merge 5 commits into
zenrows-dev-pool[bot] wants to merge 5 commits into
Conversation
… a page as a file A crawl reports `discovery`, the strategies it finds URLs with on each page it walks, and a create can ask for them. `links` follows the links in the page's HTML, and is what a crawl runs with when the create leaves it out. The field reads `["links"]` when the server does not send it, so the model parses an older answer too. `content(..., download=True)` asks for the page as a file to save rather than one to show. The text is the same; the server sends a file name built from the content id. `CrawlErrorCode` gains `api_key_cap_reached`, which a crawl ends with when the key that created it reaches one of its credit caps. It parsed as an unknown value before.
… stopped one has none The check that reads a page as a file borrowed the crawl another check stops mid-flight. That stop can land before any page is fetched, and the check then skipped instead of covering anything. It now falls back to a one-item crawl of its own and waits for it, as the check beside it already did.
The lock names the API description this surface was synced to, by its blob id, so a later drift check compares against it instead of reporting the surface as never synced. Two operations in that description are not served to callers and no surface exposes them; the lock lists them as pending, so a change to either does not read as drift here.
… set The Crawl API now ends a crawl that runs out of credits, or whose API key hits a credit cap, as `stopped` with `error.code` `insufficient_credits` or `api_key_cap_reached` and no `stop_reason`, instead of `failed`. The pages it fetched before that stay readable. The docstrings, README and example said `error` appears only on a `failed` crawl and that `stopped` always means the caller stopped it. They now describe both cases. A unit test pins that `wait` returns such a crawl with its error and no stop reason.
The fixtures now carry the error.detail the server returns when credits end a crawl, one per error code.
Base automatically changed from
feat/crawl-discovery-and-page-download
to
main
October 16, 2026 07:04
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before
The docstrings on
RunError,CrawlErrorCodeandStopReason, thewaitdocstring, the README andexamples/11_crawl.pysaiderroris present only when a crawl isfailed, and thatstoppedmeans the caller stopped it.After
They also cover a crawl that ran out of credits, or whose API key hit a credit cap, while it ran: it ends
stopped, witherror.codeinsufficient_creditsorapi_key_cap_reachedand nostop_reason, and the pages it fetched stay readable. The example prints the reason for such a stop and still reads the results. No model changes: both codes are already inCrawlErrorCode, anderroris already optional on every status.Checks
uv run pytest(251 passed),uv run ty check src,uv run ruff check src/ tests/ acceptance/anduv run ruff format --checkpass.test_wait_returns_crawl_stopped_on_creditscovers both codes.Merge order
This builds on #49. Merge it after #49, and only once the Crawl API ends crawls that run out of credits as
stopped. Until then those crawls still endfailed.