StefanoGuerrini · Oct 8, 2026 · Oct 9, 2026 · Oct 9, 2026
diff --git a/README.md b/README.md
@@ -39,6 +39,7 @@ that endpoint with extraction parameters, not a separate product.
 | `zenrows fetch` | Fetch — retrieve a protected page | **available** — `GET https://api.zenrows.com/v1/` |
 | `zenrows extract` | Extract — `extract=auto` (domain-gated open beta; falls back to Autoparse) / CSS / Markdown | **beta** — same `/v1/` |
 | `zenrows batch` | Batch — fan out over many URLs | beta — cloud works with beta access; local validate/estimate always |
+| `zenrows crawl` | Crawl (beta) — collect the URLs behind one start page, optionally with each page's HTML | beta — `https://api.zenrows.com/v1/crawls` |
 | `zenrows browser` | Browser Sessions REST API (same backend as MCP `browser_*`) | **available** — escalation-only; bills by bandwidth + time |
 | `zenrows mcp` | MCP server config (remote + local) | **available** |
 | Zenrows CLI | this repo | available |
@@ -156,7 +157,48 @@ An Extract task costs the same as a regular one (1 credit at base tier), so
 keys: `html` (the raw page) and `parsed` (the structured data) — validate a
 sample with `zenrows extract <url>` before running the full batch.
 
-## 10. Browser Sessions
+## 10. Crawl (beta)
+
+Zenrows **Crawl** is in beta. Crawl (`https://api.zenrows.com/v1/crawls`)
+takes one start URL and returns the URLs it finds behind it, optionally with
+each page's HTML. Use it when you know the listing or section but not the item
+URLs. Crawl follows the links on each page up to `--depth` hops and stays on
+the start URL's registrable domain (subdomains count). Without `--html` it returns URLs only. An account without Crawl access gets
+`CRAWL_NOT_ENABLED` ("Crawl is not enabled for this account", HTTP 403
+`REQS008`).
+
+```bash
+zenrows crawl start https://example.com/products/ --depth 1 \
+  --max-items 20 --include /product/ --follow    # start and wait for the end
+zenrows crawl results <crawl-id> --out urls.jsonl # every kept URL (paginated)
+zenrows crawl start <url> --depth 2 --html        # also fetch each kept page's HTML
+zenrows crawl content <crawl-id> <content-id>     # one page's HTML
+zenrows crawl results <crawl-id> --download       # NDJSON export (URLs + HTML) → <crawl-id>.jsonl
+zenrows crawl status <crawl-id>                   # status, coverage, stop reason
+zenrows crawl list                                # your crawls, newest first
+zenrows crawl stop <crawl-id>                     # stop a running crawl
+```
+
+`--depth` (link hops) is required; `--max-items` and `--max-pages` cap the
+crawl (API default 10 each), and `--max-pages` bounds the cost, since each page
+is one fetch. The local policy `max_pages_per_run` caps `--max-pages` (default
+1000). `--html` uses up the page budget: each kept page is one more fetch, and
+`--max-pages` counts it. `--include` / `--exclude` are repeatable substring
+filters on the URL. `results` on a running crawl returns the URLs kept so far,
+with `partial: true`. `--follow` gives up after `--timeout` (default 600000 ms)
+with `CRAWL_WAIT_TIMEOUT`; the crawl keeps running. Ctrl-C stops the wait, not
+the crawl, and exits 130; resume with `zenrows crawl wait <crawl-id>`.
+
+Errors carry the API's code as `server_code`, and `crawl_id` once the crawl
+exists. When the account has reached its limit of active jobs (3 by default),
+shared with its Batch jobs, a start gets `CRAWL_TOO_MANY_CRAWLS` and creates
+nothing; retry after `retry_after` seconds. Other codes: `CRAWL_INVALID_REQUEST`
+(400/422, do not retry as is), `CRAWL_REQUEST_IN_FLIGHT` (409, retry once the
+first request ends), `CRAWL_NOT_FOUND`, `CRAWL_CONTENT_NOT_FOUND` and
+`CRAWL_FAILED`. Set `ZENROWS_CRAWL_API_BASE` to point the CLI at another
+deployment.
+
+## 11. Browser Sessions
 
 Escalation only — **prefer `fetch`/`extract` for the vast majority of cases**;
 they cost less. Use the browser for logins, forms, and multi-step JS flows that
@@ -170,7 +212,7 @@ bring-your-own Playwright/Puppeteer.
 close interactive sessions with `zenrows browser close`. On by default; opt out
 with `zenrows policy set allow_browser false`.
 
-## 11. MCP
+## 12. MCP
 
 ```bash
 zenrows mcp status
@@ -188,7 +230,7 @@ both servers in `mcp.json` (no secrets) and ships MCP-native skills under
 `agent-plugin/skills/`. It is included in the published `@zenrows/cli` npm
 tarball (`node_modules/@zenrows/cli/agent-plugin`).
 
-## 12. Plugins
+## 13. Plugins
 
 ```bash
 zenrows plugin list
@@ -201,7 +243,7 @@ MCP-native skills). See [`agent-plugin/README.md`](agent-plugin/README.md).
 Legacy per-client snippets from `zenrows plugin install` remain available.
 CLI-oriented skills stay under repo-root `skills/`.
 
-## 13. Skills
+## 14. Skills
 
 Agent-readable playbooks that teach agents how to choose primitives.
 
@@ -213,28 +255,28 @@ zenrows skill validate zenrows
 
 The master skill `skills/zenrows/SKILL.md` teaches the full decision tree.
 
-## 14. Templates
+## 15. Templates
 
 ```bash
 zenrows template list
 zenrows template create protected-fetch-node --output ./my-project
 ```
 
-## 15. Workflows
+## 16. Workflows
 
 ```bash
 zenrows workflow list
 zenrows workflow explain competitor-intelligence
 ```
 
-## 16. Recipes
+## 17. Recipes
 
 ```bash
 zenrows recipe list
 zenrows recipe run fetch-protected-page
 ```
 
-## 17. Evals
+## 18. Evals
 
 Reproducible, transparent benchmarks. **No competitor keys are bundled** and
 nothing is hardcoded to make Zenrows win — comparison evals require you to supply
@@ -249,7 +291,7 @@ zenrows eval report protected-fetch-smoke
 Reports write `input.json`, `results.json`, `report.md`, `failures.jsonl`,
 `cost.json`, and `traces/` under `.zenrows/evals/<run-id>/`.
 
-## 18. Security
+## 19. Security
 
 - API keys are never printed or written into run artifacts/assets.
 - Secrets live in `.zenrows/secrets.json` (0600, gitignored); logs are redacted.
@@ -259,12 +301,12 @@ Reports write `input.json`, `results.json`, `report.md`, `failures.jsonl`,
 - Destructive `uninstall` requires `--yes`. Browser is on by default; opt out with
   `zenrows policy set allow_browser false`. Experimental features are off by default.
 
-## 19. Capability matrix
+## 20. Capability matrix
 
 See [`registry/capabilities.json`](registry/capabilities.json). `zenrows status
 --json` emits it.
 
-## 20. Contributing
+## 21. Contributing
 
 The CLI is TypeScript with **zero runtime dependencies** (native `fetch`,
 `node:util` `parseArgs`, `node:test`). Node 20+ runs the published build;

diff --git a/docs/capabilities.md b/docs/capabilities.md
@@ -22,5 +22,6 @@ Classification is based on the public Zenrows documentation:
 | `protected_fetch` | Fetch `GET https://api.zenrows.com/v1/` with `mode`, `js_render`, `premium_proxy`, `proxy_country`, `wait`/`wait_for`, `js_instructions`, `response_type`, `screenshot`, `original_status`, … | available |
 | `extract` | Extract `GET https://api.zenrows.com/v1/` via `extract=auto` (domain-gated open beta; CLI falls back to `autoparse`), plus `autoparse`, `css_extractor`, `outputs`, `response_type=markdown\|plaintext` | beta |
 | `batch` | Batch `https://async.api.zenrows.com/v1` (separate host, `X-API-Key` header) — open beta. Local JSONL spec validation + credit estimation work with no key. | beta |
+| `crawl` | Crawl `https://api.zenrows.com/v1/crawls` (`X-API-Key` header; same host as Fetch/Extract). Follows the links on each page from one start URL up to `depth`, on the start URL's registrable domain (subdomains count); returns URLs, and with `output_format: "html"` each page's HTML. Without Crawl access the API returns 403 `REQS008` → `CRAWL_NOT_ENABLED`. | beta |
 | `browser` | Browser Sessions REST API `https://mcp.zenrows.com/browser/sessions/*` (Bearer; same backend as `@zenrows/mcp` `browser_*`); CDP via `zenrows browser connect`. Escalation-only (prefer fetch/extract); on by default, opt out via `policy.allow_browser`; bills by bandwidth + session time (15-min max) | available |
 | `mcp` | Hosted `https://mcp.zenrows.com/mcp` + local `npx -y @zenrows/mcp` | available |
diff --git a/docs/contributing.md b/docs/contributing.md
@@ -37,6 +37,39 @@ npm run build          # tsc → dist/ (for publishing)
 4. Add a skill + recipe/eval and declare `requires_backend_capabilities`.
 5. Add tests. Never fake backend behavior; return a normalized error instead.
 
+## Running the Crawl e2e test
+
+`tests/e2e/crawl.e2e.test.ts` drives the built CLI (`zenrows crawl …`) against a
+live Zenrows API: it starts one small crawl of the target you give it (depth 1,
+3 items, 5 pages, HTML output), waits for it, reads its results, one page's
+HTML and the NDJSON export, lists crawls, stops the ended crawl and checks a
+missing id. It is not part of `npm test`, and it skips unless `ZENROWS_E2E=1`,
+`ZENROWS_API_KEY` and `ZENROWS_E2E_CRAWL_URL` are all set. It bills a few
+pages on the key's account.
+
+| Variable | Value |
+| --- | --- |
+| `ZENROWS_E2E` | `1` to run the test |
+| `ZENROWS_API_KEY` | a key with Crawl access |
+| `ZENROWS_E2E_CRAWL_URL` | the start URL to crawl, e.g. `https://example.com/products/` |
+| `ZENROWS_E2E_CRAWL_INCLUDE` | optional include pattern, e.g. `/product/`; every result URL must contain it |
+| `ZENROWS_CRAWL_API_BASE` | optional API base, default `https://api.zenrows.com/v1` |
+
+Read the key from a file rather than typing it, so it stays out of your shell
+history:
+
+```bash
+export ZENROWS_API_KEY="$(tr -d '\n' < path/to/api.key)"
+ZENROWS_E2E=1 ZENROWS_E2E_CRAWL_URL=https://example.com/products/ \
+  ZENROWS_E2E_CRAWL_INCLUDE=/product/ npm run test:e2e
+```
+
+To target a local or staging deployment, set `ZENROWS_CRAWL_API_BASE` to its
+`/v1` base. When the account has reached its limit of active jobs (3 by
+default, shared with its Batch jobs), the test waits `retry_after` seconds and
+retries the start for up to 5 minutes, so run concurrent e2e
+runs on one account one at a time.
+
 ## Rules
 
 - Never print or persist API keys. Redact secrets in logs and artifacts.

diff --git a/package.json b/package.json
@@ -48,6 +48,7 @@
     "zenrows": "node bin/zenrows.ts",
     "dev": "node bin/zenrows.ts",
     "test": "npm run build && node --test --import ./dist/tests/setup.js dist/tests/*.test.js",
+    "test:e2e": "npm run build && node --test dist/tests/e2e/*.test.js",
     "typecheck": "tsc --noEmit",
     "build": "rm -rf dist && tsc && chmod +x dist/bin/zenrows.js",
     "prepack": "npm run build",

diff --git a/registry/capabilities.json b/registry/capabilities.json
@@ -28,6 +28,15 @@
       "requiresAuth": true,
       "notes": "Open beta. Cloud subcommands (create/status/results/cancel/wait/retry-failed) call the Batch API; `zenrows batch estimate` works locally with no API key."
     },
+    "crawl": {
+      "key": "crawl",
+      "label": "Crawl",
+      "status": "beta",
+      "command": "zenrows crawl",
+      "backend": "Crawl — https://api.zenrows.com/v1/crawls (X-API-Key header; same host as Fetch/Extract)",
+      "requiresAuth": true,
+      "notes": "Beta. Follows the links on each page from one start URL up to depth, on the start URL's registrable domain (subdomains count), and returns the URLs it keeps, optionally with each page's HTML (start/status/wait/results/content/list/stop). An account without Crawl access gets 403 REQS008 (CRAWL_NOT_ENABLED). When the account has reached its limit of active jobs (3 by default), shared with its Batch jobs, a start gets 429 too_many_crawls (CRAWL_TOO_MANY_CRAWLS); retry after retry_after seconds."
+    },
     "browser": {
       "key": "browser",
       "label": "Browser Sessions",

diff --git a/registry/skills.json b/registry/skills.json
@@ -44,6 +44,17 @@
       "path": "skills/batch-jobs",
       "tags": ["batch", "scale", "jsonl"]
     },
+    {
+      "name": "crawl",
+      "type": "skill",
+      "description": "Collect the URLs behind one start page with Crawl (beta): start, wait, read results and page HTML, stop.",
+      "status": "beta",
+      "requires_backend_capabilities": ["crawl"],
+      "requires_auth": true,
+      "version": "0.1.0",
+      "path": "skills/crawl",
+      "tags": ["crawl", "links", "urls"]
+    },
     {
       "name": "interact-browser",
       "type": "skill",

diff --git a/skills/crawl/SKILL.md b/skills/crawl/SKILL.md
@@ -0,0 +1,98 @@
+---
+name: crawl
+description: Collect the URLs behind one start page with Crawl (beta) — start a crawl, wait for it, read the kept URLs and optionally each page's HTML, stop it. Use when the user has a listing or section URL but not the item URLs.
+version: 0.1.0
+requires_backend_capabilities: [crawl]
+---
+
+# Crawl (beta)
+
+Crawl is in beta. Give Crawl one start URL; it follows the links on that page (and, with a
+higher depth, on the pages behind it) and returns the URLs it keeps. Use it
+when you know where the items live (a category page, a blog index, a docs
+section) but not the item URLs themselves. Once you have the URLs, read pages
+with `zenrows fetch` / `zenrows extract`, or ask Crawl for each page's HTML
+with `--html`.
+
+Crawl stays on the start URL's registrable domain; subdomains count. An account without Crawl access gets
+403 `REQS008` → `CRAWL_NOT_ENABLED` ("Crawl is not enabled for this account");
+do not retry it.
+
+## Start, wait, read
+
+```
+zenrows crawl start https://example.com/products/ --depth 1 \
+  --max-items 20 --max-pages 25 --include /product/ --follow --json
+zenrows crawl results <id> --json            # every kept URL
+zenrows crawl results <id> --out urls.jsonl  # same, as JSONL
+```
+
+- `--depth N` is required: link hops from the start URL. `1` = the start
+  page's links; `2` = also the links on each of those pages.
+- `--max-items` (kept URLs) and `--max-pages` (fetched pages) cap the crawl;
+  both default to 10. `--max-pages` bounds the cost: each page is one fetch.
+  The local policy `max_pages_per_run` caps `--max-pages` (default 1000);
+  above it the CLI refuses with `POLICY_LIMIT_EXCEEDED` before any request.
+- `--include` / `--exclude` are substring matches on the normalized URL and
+  repeat: `--include /product/ --exclude ?add-to-cart`.
+- `start` returns at once. `--follow` (or `zenrows crawl wait <id>`) polls
+  until the crawl ends (default timeout 600000 ms). On `--timeout` the crawl
+  keeps running (it is not stopped) and you get `CRAWL_WAIT_TIMEOUT` with
+  `crawl_id`. Ctrl-C also stops the wait, not the crawl, and exits 130;
+  resume with `zenrows crawl wait <id>`.
+- `results` on a running crawl returns the URLs kept so far, with
+  `partial: true`. Read again once the crawl ends.
+
+## Page HTML
+
+```
+zenrows crawl start <url> --depth 1 --html --max-pages 20 --follow
+zenrows crawl results <id> --json        # fetched results carry content_url
+zenrows crawl content <id> <content_id>  # one page's HTML (or pass the content_url)
+zenrows crawl results <id> --download --out pages.jsonl  # every URL + HTML, NDJSON
+```
+
+`--html` uses up the page budget: each kept page is one more fetch, and
+`--max-pages` counts it. Raise `--max-pages` to keep the same number of URLs.
+A result's `content_status` is `pending`, `fetched` or `failed`; only
+`fetched` results have content (else `CRAWL_CONTENT_NOT_FOUND`).
+
+## While it runs, and after
+
+```
+zenrows crawl status <id>                       # status, coverage, stop_reason / error
+zenrows crawl results <id> --cursor <c> --json  # one page; keep polling with next_cursor
+zenrows crawl list                              # your crawls, newest first
+zenrows crawl stop <id>                         # stop it; kept URLs stay readable
+```
+
+A crawl ends `completed` (nothing left, or a cap hit: `stop_reason`
+`max_items` / `max_pages`), `stopped` (you stopped it) or `failed` (`error.code`
+and `error.detail` say why). Only `failed` exits non-zero (`CRAWL_FAILED`).
+`stop` on a crawl that already ended prints the status it ended with.
+
+## Limits
+
+When the account has reached its limit of active jobs (3 by default), shared
+with its Batch jobs, a start gets 429 `too_many_crawls` →
+`CRAWL_TOO_MANY_CRAWLS`; nothing was created. Retry after `retry_after`
+seconds (about 30), or stop one of your crawls or Batch jobs first.
+
+## Errors
+
+Every Crawl error carries the API's code as `server_code` (when the API sent
+one), and `crawl_id` once the crawl exists.
+
+| Code | Meaning | Retry? |
+| --- | --- | --- |
+| `CRAWL_NOT_ENABLED` | 403 `REQS008`: Crawl is not enabled for this account | no |
+| `CRAWL_INVALID_REQUEST` | 400 or 422: fix the request. For `idempotency_key_reused`, send a new key or no key | no, not as is |
+| `CRAWL_REQUEST_IN_FLIGHT` | 409: the same Idempotency-Key is still in flight | yes, once the first request ends |
+| `CRAWL_NOT_FOUND` / `CRAWL_CONTENT_NOT_FOUND` | 404: wrong id, or the page has no content | no |
+| `CRAWL_TOO_MANY_CRAWLS` | 429: the account's active-job limit | yes, after `retry_after` |
+| `CRAWL_WAIT_TIMEOUT` | the wait ran out; the crawl keeps running | wait again |
+| `CRAWL_FAILED` | the crawl ended `failed`, or another error | 5xx: yes, with backoff |
+
+Rules: start small (`--depth 1`, low caps) and check the results before a
+deep crawl. Mind the cost ([[cost-control]]). Read errors with
+[[trace-debug]].
diff --git a/skills/trace-debug/SKILL.md b/skills/trace-debug/SKILL.md
@@ -45,6 +45,15 @@ zenrows trace export <run-id>     # JSON for sharing
   auto-created Free plan.
 - `BATCH_FAILED` → the batch run ended `failed`; `failure_reason` and
   `failure_detail` in `zenrows batch status <id> --json` say why.
+- `CRAWL_NOT_ENABLED` → the API answered 403 `REQS008`: Crawl is not
+  enabled for this account. Do not retry; ask Zenrows support for access.
+- `CRAWL_TOO_MANY_CRAWLS` → 429 `too_many_crawls`: the account has reached
+  its limit of active jobs (3 by default), shared with its Batch jobs.
+  Nothing was created; retry after `retry_after` seconds (see
+  `zenrows crawl list`).
+- `CRAWL_FAILED` → the crawl ended `failed`; `error.code` in
+  `zenrows crawl status <id> --json` says why, with `error.detail`. See
+  [[crawl]].
 - `PARAM_CONFLICT_AUTO_MANUAL` → drop the managed flags or add `--manual`.
 - `CAPABILITY_UNAVAILABLE` → the primitive is not available on this account (e.g. beta/invite-only); use the local-spec path where offered.
 

diff --git a/skills/zenrows/SKILL.md b/skills/zenrows/SKILL.md
@@ -34,6 +34,10 @@ If the user has a known URL and wants page content:
 If the user has a known URL and wants structured data:
   → Use Extract.                    (zenrows extract <url> | --autoparse | --css)
 
+If the user has a start page (listing, index, section) but not the item URLs:
+  → Use Crawl.                      (zenrows crawl start <url> --depth 1 --follow,
+                                     then `zenrows crawl results <id>`)
+
 If the user has many URLs (bulk):
   → Use Batch.                      (validate one page with fetch/extract first,
                                      then `zenrows batch create <jobs.jsonl>`)
@@ -70,6 +74,7 @@ Run `zenrows status` for the live capability matrix. As of this toolkit:
 | Protected Fetch | `zenrows fetch` | available (`GET /v1/`) |
 | Extract (extract=auto / Autoparse/CSS/Markdown) | `zenrows extract` | beta (same `/v1/`; extract=auto falls back to autoparse) |
 | Batch | `zenrows batch` | beta — runs cloud jobs (create/status/results); cheapest path at scale. Estimate specs locally with no key |
+| Crawl | `zenrows crawl` | beta — collect the URLs behind one start page, optionally with each page's HTML |
 | Browser | `zenrows browser` | available (Browser Sessions REST API / MCP) |
 | MCP | `zenrows mcp` | available (remote + local server) |