feat(crawl): add the zenrows crawl command and its acceptance test

README.md

@@ -39,6 +39,7 @@ that endpoint with extraction parameters, not a separate product.
 | `zenrows fetch` | Fetch — retrieve a protected page | **available** — `GET https://api.zenrows.com/v1/` |
 | `zenrows extract` | Extract — `extract=auto` (domain-gated open beta; falls back to Autoparse) / CSS / Markdown | **beta** — same `/v1/` |
 | `zenrows batch` | Batch — fan out over many URLs | beta — cloud works with beta access; local validate/estimate always |
+| `zenrows crawl` | Crawl (new) — collect the URLs behind one start page, optionally with each page's HTML | **available** — `https://api.zenrows.com/v1/crawls` |
 | `zenrows browser` | Browser Sessions REST API (same backend as MCP `browser_*`) | **available** — escalation-only; bills by bandwidth + time |
 | `zenrows mcp` | MCP server config (remote + local) | **available** |
 | Zenrows CLI | this repo | available |
@@ -156,7 +157,79 @@ An Extract task costs the same as a regular one (1 credit at base tier), so
 keys: `html` (the raw page) and `parsed` (the structured data) — validate a
 sample with `zenrows extract <url>` before running the full batch.
 
-## 10. Browser Sessions
+## 10. Crawl (new)
+
+Crawl is still evolving: new features are coming, limits may be tuned, and the changelog announces each change.
+
+Crawl (`https://api.zenrows.com/v1/crawls`)
+takes one start URL and returns the URLs it finds behind it, optionally with
+each page's HTML. Use it when you know the listing or section but not the item
+URLs. Crawl follows the links on each page, up to `--depth` hops when you set
+it, and stays on the start URL's registrable domain (subdomains count). Without
+`--output-format html` it returns URLs only. An account without Crawl access gets
+`CRAWL_NOT_ENABLED` ("Crawl is not enabled for this account", HTTP 403
+`REQS008`).
+
+```bash
+zenrows crawl create https://example.com/products/ --depth 1 \
+  --max-items 20 --include-patterns /product/ --follow  # create and wait for the end
+zenrows crawl results <crawl-id> --out urls.jsonl       # every kept URL (paginated)
+zenrows crawl create <url> --depth 2 --output-format html  # also fetch each kept page's HTML
+zenrows crawl content <crawl-id> <content-id>           # one page's HTML
+zenrows crawl download <crawl-id>                       # NDJSON export (URLs + HTML) → <crawl-id>.jsonl
+zenrows crawl get <crawl-id>                            # status, coverage, stop reason, one page of results
+zenrows crawl wait <crawl-id>                           # wait for the end
+zenrows crawl list                                      # your crawls, newest first
+zenrows crawl stop <crawl-id>                           # stop a running crawl
+```
+
+`--depth` (link hops, 1 to 100000) is optional: leave it out and the crawl
+follows links until `--max-items` or `--max-pages` stops it, or nothing is left
+to open. `--max-items` and `--max-pages` cap the crawl (API default 100 each),
+and `--max-pages` bounds the cost, since each page is one fetch. The local policy `max_pages_per_run` caps `--max-pages` (default
+1000). `--output-format html` uses up the page budget: each kept page is one
+more fetch, and `--max-pages` counts it. `--include-patterns` /
+`--exclude-patterns` are repeatable substring filters on the URL. `get` and
+`list` read one page and take `--cursor` and `--limit`. `results` follows every
+page (`--limit` is the page size); on a running crawl it returns the URLs kept
+so far, with `partial: true`. `download` reports the crawl's `status` (`null`
+when the API sends none), and `partial: true` while it runs.
+
+With `--json`, `create`, `get`, `wait` and `stop` print
+`{ok, crawl_id, status, crawl}`. The crawl object (with `coverage`,
+`stop_reason` and `error`) is under `crawl`. `get` adds `results` and
+`next_cursor`. All keys are snake_case.
+
+`wait`, and `create --follow`, poll until the crawl ends or `--timeout` seconds
+run out (default 600). When the time runs out, the command exits 0 and prints
+the crawl with status `running` and the hint `zenrows crawl wait <crawl-id>`
+(under `--json`, in `note`); the crawl keeps running. Ctrl-C stops the wait, not the crawl, and exits 130;
+resume with `zenrows crawl wait <crawl-id>`. A crawl that ends `failed` exits 1
+with `CRAWL_FAILED`.
+
+Errors carry the API's code as `server_code`, `crawl_id` on any call about one
+crawl, and `retry_after` on a 429. When the account has reached its limit of
+active jobs (3 by default), shared with its Batch jobs, a create gets 429
+`too_many_crawls`, mapped to `CRAWL_TOO_MANY_CRAWLS`, and creates nothing; retry
+after `retry_after` seconds. Any other 429 is `CRAWL_RATE_LIMITED`: requests
+are being throttled; retry after `retry_after` seconds. Other codes: `CRAWL_INSUFFICIENT_CREDITS` and `CRAWL_KEY_CAP_REACHED` (402, with the
+same claim, top-up and key-cap advice as other commands),
+`CRAWL_INVALID_REQUEST` (400/422, do not retry as is), `CRAWL_REQUEST_IN_FLIGHT`
+(409, retry once the first request ends), `CRAWL_NOT_FOUND`,
+`CRAWL_CONTENT_NOT_FOUND`, `CRAWL_BUSY` (503 on a stop that was not saved yet;
+run it again, it is safe) and `CRAWL_FAILED`. Set `ZENROWS_CRAWL_API_BASE` to
+point the CLI at another deployment.
+
+The CLI retries transient failures, as the Crawl SDKs do: up to 3 retries,
+waiting the `Retry-After` seconds (at most 30) or 250 ms x 2^n (+/-20% jitter,
+at most 10 s).
+Reads retry on 429, 502, 503, 504 and network errors (a timeout counts).
+`create` retries only with `--idempotency-key`, and only on 502, 503, 504 and
+network errors, never on 429. `stop` retries on 429, 502, 503 (a `crawl_busy`
+means the stop was not saved and the crawl still runs) and 504, waiting the
+`Retry-After` seconds (at most 30) or 1 s; repeating a stop is safe.
+
+## 11. Browser Sessions
 
 Escalation only — **prefer `fetch`/`extract` for the vast majority of cases**;
 they cost less. Use the browser for logins, forms, and multi-step JS flows that
@@ -170,7 +243,7 @@ bring-your-own Playwright/Puppeteer.
 close interactive sessions with `zenrows browser close`. On by default; opt out
 with `zenrows policy set allow_browser false`.
 
-## 11. MCP
+## 12. MCP
 
 ```bash
 zenrows mcp status
@@ -188,7 +261,7 @@ both servers in `mcp.json` (no secrets) and ships MCP-native skills under
 `agent-plugin/skills/`. It is included in the published `@zenrows/cli` npm
 tarball (`node_modules/@zenrows/cli/agent-plugin`).
 
-## 12. Plugins
+## 13. Plugins
 
 ```bash
 zenrows plugin list
@@ -201,7 +274,7 @@ MCP-native skills). See [`agent-plugin/README.md`](agent-plugin/README.md).
 Legacy per-client snippets from `zenrows plugin install` remain available.
 CLI-oriented skills stay under repo-root `skills/`.
 
-## 13. Skills
+## 14. Skills
 
 Agent-readable playbooks that teach agents how to choose primitives.
 
@@ -213,28 +286,28 @@ zenrows skill validate zenrows
 
 The master skill `skills/zenrows/SKILL.md` teaches the full decision tree.
 
-## 14. Templates
+## 15. Templates
 
 ```bash
 zenrows template list
 zenrows template create protected-fetch-node --output ./my-project
 ```
 
-## 15. Workflows
+## 16. Workflows
 
 ```bash
 zenrows workflow list
 zenrows workflow explain competitor-intelligence
 ```
 
-## 16. Recipes
+## 17. Recipes
 
 ```bash
 zenrows recipe list
 zenrows recipe run fetch-protected-page
 ```
 
-## 17. Evals
+## 18. Evals
 
 Reproducible, transparent benchmarks. **No competitor keys are bundled** and
 nothing is hardcoded to make Zenrows win — comparison evals require you to supply
@@ -249,7 +322,7 @@ zenrows eval report protected-fetch-smoke
 Reports write `input.json`, `results.json`, `report.md`, `failures.jsonl`,
 `cost.json`, and `traces/` under `.zenrows/evals/<run-id>/`.
 
-## 18. Security
+## 19. Security
 
 - API keys are never printed or written into run artifacts/assets.
 - Secrets live in `.zenrows/secrets.json` (0600, gitignored); logs are redacted.
@@ -259,12 +332,12 @@ Reports write `input.json`, `results.json`, `report.md`, `failures.jsonl`,
 - Destructive `uninstall` requires `--yes`. Browser is on by default; opt out with
   `zenrows policy set allow_browser false`. Experimental features are off by default.
 
-## 19. Capability matrix
+## 20. Capability matrix
 
 See [`registry/capabilities.json`](registry/capabilities.json). `zenrows status
 --json` emits it.
 
-## 20. Contributing
+## 21. Contributing
 
 The CLI is TypeScript with **zero runtime dependencies** (native `fetch`,
 `node:util` `parseArgs`, `node:test`). Node 20+ runs the published build;

acceptance/crawl/.gitignore

@@ -0,0 +1 @@
+out/

acceptance/crawl/README.md

@@ -0,0 +1,113 @@
+# Crawl first-install acceptance test: CLI
+
+This folder tests the CLI (`@zenrows/cli`) the way a new customer meets it. It builds the release
+artifact, installs it in a clean container with nothing else, follows the README's Crawl
+quickstart, and checks every Crawl feature against the live API. The result is one report.
+
+It tests this repo only, so it runs on its own.
+
+## What it needs
+
+- bash, git and Docker. Nothing else on the host.
+- Network access to GitHub, the package registries and the Crawl API.
+- A Zenrows API key with Crawl access, in `ZENROWS_API_KEY`.
+
+## How to run it
+
+From the repo root:
+
+```bash
+export ZENROWS_API_KEY="$(cat path/to/key-file)"   # keep it out of your shell history
+bash acceptance/crawl/run.sh              # build, run every check, write the report
+bash acceptance/crawl/run.sh --no-key     # only the checks that need no key; the live checks are SKIP
+bash acceptance/crawl/run.sh --report     # rebuild the report from out/
+REF=v1.2.3 bash acceptance/crawl/run.sh   # test a tag, branch or SHA from GitHub instead of this checkout
+```
+
+Without `ZENROWS_API_KEY`, `run.sh` stops and asks for it. The report is
+`acceptance/crawl/out/crawl-first-install-report.md`; `out/` is not committed.
+
+Every input is an environment variable with a default in `versions.env`: `CRAWL_URL`
+(default `https://www.scrapingcourse.com/ecommerce/`), `CRAWL_INCLUDE` (default `/product/`),
+`REF`, `CRAWL_API_BASE`, the images (`node:20`, `python:3.12-slim`, `golang:1.25`, pinned by
+digest) and the tool versions. `OUT_DIR` moves the output.
+
+## What a run does
+
+1. **Source.** By default, this checkout, including changes not yet committed. With `REF`, the
+   repo at that ref from GitHub.
+2. **Build.** In its own container, the release artifact: `npm install -g` of the `npm pack` tarball.
+3. **Clean test container.** It gets only the artifact (read-only), the checks in `surface/`
+   (read-only), the inputs, and the key by name (`-e ZENROWS_API_KEY`). It has no source, no
+   caches from the build, and no tokens.
+4. **README first.** It runs the README's Crawl quickstart as written. The only changes are the
+   ones the cost rules need: the key, `CRAWL_URL`, `CRAWL_INCLUDE`, `max_pages` 10 or less, and
+   no html. The report lists each change.
+5. **Checks 1 to 21 and the extras.** Each writes one JSON result:
+   `{surface, check, title, status, expected, actual, evidence}`, with `status` PASS, FAIL or
+   SKIP. The evidence is the command and its trimmed output, with the key redacted.
+6. **Clean-up.** It stops every crawl the run started and reads each one's final status and
+   `coverage.pages_fetched`. It runs even when a check fails, or on Ctrl-C.
+7. **Report.** Verdict, environment, install steps, feature matrix, findings, cost and
+   clean-up. Last, the run checks that the key appears in no file under `out/`.
+
+Some checks don't use the live API, so they behave the same on every run. They point the
+client at a small local stand-in server that records what it sends. These are: "only the
+fields you set are sent", "the error code is empty when the body has none", and the timing
+and signal checks.
+
+## The checks
+
+| # | Feature | Expected |
+| --- | --- | --- |
+| 1 | Create | returns at once: `running`, a `crawl_id`, the echoed parameters; only the fields set are sent (no `--depth` sends no `depth`) |
+| 2 | Create with html | results get `content_status` and, once fetched, `content_url` |
+| 3 | Idempotency | same key and body → same `crawl_id`; different body → 422 `idempotency_key_reused` (MCP, CLI: `CRAWL_INVALID_REQUEST` with the advice to send a new key) |
+| 4 | Wait | returns the ended crawl; `failed` is returned, not raised (CLI: exit 1) |
+| 5 | Wait runs out | a 1 s timeout on a running crawl returns it `running`, no error (CLI: exit 0; MCP, CLI: a `note`) |
+| 6 | Get | `limit` 2, then the cursor: one page each; `next_cursor` moves on and is null on the last page |
+| 7 | List | `limit` 2, then the next page: newest first; `next_cursor` absent or null on the last page |
+| 8 | Results | every kept URL once, each matching `CRAWL_INCLUDE`; on a running crawl, what is kept so far (MCP, CLI: `partial: true`) |
+| 9 | Content | by content id and by `content_url`: the same HTML |
+| 10 | Download | a status, and one line per result with `url`, `content_status`, `content` |
+| 11 | Stop | running → `stopped`, `stop_reason: user`; ended → answers as it ended |
+| 12 | Not found | 404 `crawl_not_found` (MCP, CLI: `CRAWL_NOT_FOUND` with `crawl_id` and `server_code`) |
+| 13 | Invalid input | `depth` 0 → 422 `invalid_parameter` (MCP, CLI: `CRAWL_INVALID_REQUEST`) |
+| 14 | Gated option | `output_format` json → 422 `invalid_parameter`; if it is accepted, the key is in a beta segment, not a bug |
+| 15 | Bad key | the key `invalid` → 401 (MCP, CLI: `AUTH_INVALID`) |
+| 16 | Errors carry the server code | SDKs: `code` holds it, empty when the body has none; MCP, CLI: `server_code` |
+| 17 | Naming | the API's operation and parameter names, in this surface's casing |
+| 18 | Copy | Crawl is "new", never "beta"; one "still evolving" line; the 429 limit reads as the account's active jobs, shared with Batch |
+| 19 | Failed crawl | an include pattern that matches nothing ends `failed` with `no_items_found`; SDKs return it, not raise it; MCP and CLI say `ok: false` (CLI: exit 1, `CRAWL_FAILED`) |
+| 20 | Results after stop | the html crawl is stopped while its pages are still being fetched: reading its results ends at once (MCP and CLI: `partial: false`, no note to call again), and the same URLs come back once the pages in flight finish |
+| 21 | Documented defaults | a create without `max_pages` and `max_items` gets the default the docs state (100) |
+
+Extra checks for this surface: every subcommand prints JSON with `--json` (`create`, `get`, `wait` and `stop` print `{ok, crawl_id, status, crawl, note?}`), Ctrl-C during `crawl wait` prints the resume hint and exits 130, and `--wait` and `--html` are rejected as unknown flags.
+
+## What it costs
+
+Every crawl uses `depth` 1, and `max_pages` 10 or less except check 19's, which leaves `max_pages`
+and `max_items` at the API default (100) for check 21; at `depth` 1 without html it fetches one
+page. A run starts five crawls: the README run,
+one URLs-only crawl, one idempotency crawl that is stopped at once, one html crawl that is
+stopped with its pages in flight (check 20), and one that keeps nothing (check 19). On the default `CRAWL_URL`, a URLs-only crawl fetches 1 page and the
+html crawl 6, so a run costs about **10 pages**, and at most 50 if every crawl reaches its
+`max_pages`. Check 9 starts a sixth crawl, one html item, when the stop in check 20 lands
+before any page is fetched. It adds at most 10 pages. The report lists each crawl id and its `pages_fetched`.
+
+An account can run 3 active jobs at once, shared with Batch jobs. A run has at most 2 crawls
+running at a time. If a create still gets a 429, the run waits `Retry-After` and tries again.
+It never sends requests to trigger the limit. Don't run two copies against one account at the
+same time.
+
+## How to read the report
+
+- **Verdict:** ready, unless there is a blocker or major finding.
+- **Feature matrix:** PASS, FAIL or SKIP, with a short reason. SKIP means the check does not
+  apply to this surface, the client rejects the input before the API, a crawl ended before
+  the check could act on it, or there was no key.
+- **Findings:** one entry per FAIL, most severe first, with the command and output. An install
+  or README failure is a blocker; a wrong answer from a feature is major; naming, copy, the
+  defaults and the extras are minor.
+- `out/results/checks/*.json` holds every result; `out/results/container.log` is the full
+  output.

acceptance/crawl/builders/build-npm.sh

@@ -0,0 +1,16 @@
+#!/usr/bin/env bash
+# Runs in the Node image. Builds the npm tarball a release would publish: the repo's own
+# install and build, then npm pack. Only artifact/*.tgz leaves this container.
+set -euo pipefail
+cp -a /out/source /build && cd /build
+if [ -f pnpm-lock.yaml ]; then
+  npm install -g --no-audit --no-fund "pnpm@$PNPM_VERSION" >/dev/null
+  pnpm install --frozen-lockfile --store-dir /tmp/pnpm-store
+  pnpm run build
+else
+  npm ci --no-audit --no-fund
+  npm run build
+fi
+rm -rf /out/artifact && mkdir -p /out/artifact
+npm pack --pack-destination /out/artifact 2>&1 | tail -n 60
+ls -l /out/artifact

acceptance/crawl/builders/fetch-source.sh

@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+# Runs in the Node image when REF is set: downloads ZenRows/$REPO at $REF as GitHub's source
+# tarball and reads the full commit SHA from it.
+set -euo pipefail
+cd /out
+curl -fsSL --retry 4 -o src.tar.gz "https://codeload.github.com/ZenRows/$REPO/tar.gz/$REF"
+# git get-tar-commit-id stops reading after the header, so the pipe breaks: ignore that.
+(gunzip -c src.tar.gz | git get-tar-commit-id > sha) || true
+[ -s sha ] || { echo "no commit id in the tarball of $REPO@$REF" >&2; exit 1; }
+rm -rf source && mkdir -p source && tar -xzf src.tar.gz -C source --strip-components 1 && rm src.tar.gz
+echo "fetched $REPO@$REF = $(cat sha)"