feat(crawl): add zenrows crawl for the Crawl API

[StefanoGuerrini]

This PR replaces #30. The code is the same. The history drops the AI co-author trailer. The review of the first version is here.

What this adds

A zenrows crawl command for the Crawl API (https://api.zenrows.com/v1/crawls). Crawl is new. Crawl takes one start URL, follows the links on each page up to depth on the start URL's registrable domain (subdomains count), and returns the URLs it keeps, optionally with each page's HTML.

The subcommands and flags follow the shared Crawl client interface, so the CLI uses the same operation and parameter names as the Node, Python and Go SDKs and the MCP server.

zenrows crawl create <url> --depth N [--max-items N] [--max-pages N]
                     [--include-pattern P]... [--exclude-pattern P]... [--output-format html]
                     [--follow] [--timeout s] [--idempotency-key K] [--no-signup]
zenrows crawl get <id> [--cursor C] [--limit N]
zenrows crawl list [--cursor C] [--limit N]
zenrows crawl results <id> [--limit N] [--out file]
zenrows crawl content <id> <content_id|content_url> [--out file]
zenrows crawl download <id> [--out file]
zenrows crawl stop <id>
zenrows crawl wait <id> [--timeout s]

Every subcommand prints JSON with --json, in the same {ok, ..., error} shape the other commands use. All keys are snake_case (crawl_id, content_id, content_type), as in the MCP server. create, get, wait and stop print {ok, crawl_id, status, crawl}; the crawl object, with coverage, stop_reason and error, is under crawl. get adds results and next_cursor. results --out with --json prints the envelope with file instead of results. The run artifact keeps crawlId, as Batch keeps jobId.

• src/core/crawl-api.ts: the client. createCrawl, getCrawl, listCrawls, stopCrawl, getCrawlContent, downloadCrawl (returns the X-Crawl-Status status with the NDJSON), listAllResults (follows next_cursor; on a running crawl it returns what is kept so far and ends at the first empty page, with partial: true), waitForCrawl (polls with limit=1, 2s backoff x1.5 capped at 15s, 600 s default; when the time runs out it returns the crawl, still running). Auth is the X-API-Key header. ZENROWS_CRAWL_API_BASE overrides the base; otherwise it follows the configured apiBase. Transient failures are retried as the Crawl SDKs do: up to 3 retries, waiting the Retry-After seconds or 250 ms x 2^n (+/-20% jitter, capped at 10 s). GET requests retry on 429, 502, 503, 504 and network errors (a per-request timeout counts). create retries only with an Idempotency-Key, and only on 502, 503, 504 and network errors, never on 429. stop never retries. Batch does not change.
• create sends only the fields the caller set. --output-format html sets output_format: "html" and is the only value it takes. Without it, the crawl returns URLs only. With it, each kept page is one more fetch that --max-pages counts.
• get reads one API page: the crawl and one page of its results, with next_cursor. list reads one page of crawls. results follows every page; --limit is the page size. download writes the NDJSON export to --out (default <id>.jsonl) and reports status (null when the API sends no X-Crawl-Status) and partial.
• create --follow and wait poll until the crawl ends or --timeout seconds run out (default 600). When the time runs out, the command exits 0 and prints the crawl with status running (ok: true under --json) and the hint zenrows crawl wait <id> (under --json, in note, the key the MCP server uses). The crawl keeps running. A crawl that ends failed exits 1. Ctrl-C stops the wait, not the crawl: the CLI says the crawl is still running, prints zenrows crawl wait <id> and exits 130.
• stop on a crawl that already ended prints the status it ended with, not "Stopped".
• Error codes: CRAWL_NOT_ENABLED (403 REQS008), AUTH_INVALID (401), CRAWL_QUOTA_EXCEEDED and CRAWL_KEY_CAP_REACHED (402; AUTH014 is the key cap; the messages keep the claim-URL, top-up and key-cap advice of the shared credit errors), CRAWL_INVALID_REQUEST (400 and 422; for idempotency_key_reused the advice is to send a new key or no key), CRAWL_REQUEST_IN_FLIGHT (409, retry once the first request ends), CRAWL_NOT_FOUND and CRAWL_CONTENT_NOT_FOUND (404), CRAWL_TOO_MANY_CRAWLS (429: the account has reached its limit of active jobs, 3 by default, shared with its Batch jobs; structured retry_after in seconds), CRAWL_FAILED (a crawl that ended failed, any other 403, or another error) and BACKEND_UNAVAILABLE (network). Every error carries the API's code as server_code when the API sent one, and crawl_id on any call about one crawl.
• create checks the domain policy and max_pages_per_run (default 1000) before any network call, and writes a run artifact. The help says that this local policy caps --max-pages.
• The crawl capability (available), a crawl skill (available) with its registry entry, crawl lines in the zenrows and trace-debug skills, a README section and table row, and a docs/capabilities.md row. The help, README and skills say New. The README Crawl section and the capability notes carry one line that says Crawl is still evolving.

Test guards changed on purpose

tests/production-surface.test.ts asserted that crawl is an unknown command, and tests/registry.test.ts asserted that there is no crawl skill. Both now check only discover and monitor, and also check that crawl is registered. tests/capabilities.test.ts pins the crawl status to available.

Tests

• npm run typecheck: clean.
• npm test: 338 passed, 0 failed. tests/crawl-api.test.ts covers request shape, response parsing, error mapping (403 REQS008 and other 403, 402 quota and key cap, 404 both codes, 400, 409, 422 including idempotency_key_reused, 429 with retry_after, 401, 500, a body without a code), the retry policy (a GET retried on 503 with backoff and then successful, a GET that gives up after 3 retries, Retry-After honored, a transport failure retried on a GET, create without a key not retried on 503, create with a key retried on 503, create never retried on 429 even with a key, stop not retried), server_code and crawl_id on errors, paging with a page size on ended and running crawls, and the waiter's end and timeout. tests/crawl-command.test.ts covers the command: undeclared and removed flags (--html, --include, --exclude, --wait, results --download, results --cursor) and the removed start and status subcommands are rejected before any request, --output-format and --timeout validation, get with a cursor, partial results, download with and without X-Crawl-Status, the JSON keys of create, get, stop and content, the --out --json envelope, stop on an ended crawl, a wait that runs out (exit 0, status running, resume hint, note under --json) and Ctrl-C during a wait.
• E2E, tests/e2e/crawl.e2e.test.ts: drives the built CLI binary. It runs only with npm run test:e2e, and it skips unless ZENROWS_E2E=1, ZENROWS_API_KEY and ZENROWS_E2E_CRAWL_URL are set. ZENROWS_E2E_CRAWL_INCLUDE and ZENROWS_CRAWL_API_BASE are optional. docs/contributing.md has a "Running the Crawl e2e test" section.

export ZENROWS_API_KEY="$(tr -d '\n' < path/to/api.key)"
ZENROWS_E2E=1 ZENROWS_CRAWL_API_BASE=$ZENROWS_CRAWL_API_BASE \
  ZENROWS_E2E_CRAWL_URL=<start-url> ZENROWS_E2E_CRAWL_INCLUDE=<pattern> npm run test:e2e

I ran it against a live ZenRows API deployment on the first commit, with the target set through ZENROWS_E2E_CRAWL_URL: 1 passed, 0 failed. The run started a crawl and waited for it to complete. Every result matched the include pattern. It read one page's HTML, and the NDJSON export had one line per result. The crawl appeared in the list, stop on the ended crawl returned its terminal status, and an unknown id gave CRAWL_NOT_FOUND. The e2e test now uses the new subcommands and flags; it did not run again after these changes.