Skip to content
mrscraper-comPublic

About

No description, website, or topics provided.

Resources

Stars

4 stars

Watchers

0 watching

Forks

Latest commit

 

History

68 Commits

Folders and files

Repository files navigation

MrScraper CLI

MrScraper

Command-line access to the seven MrScraper services, with browser sign-in and optional agent skill setup.

Overview

Seven services share the same descriptions, parameter meanings, and defaults across the CLI, SDKs, and integrations. Names, types, authentication, and code examples follow each surface's conventions. See the API Reference.

Service Public name Description
Status status Return the raw account status API response without summarizing, redacting body fields, or adding calculated fields.
Scrape HTML scrape-html Retrieve the raw HTML of a page. Browser rendering is optional.
Scrape Markdown scrape-markdown Retrieve a page as Markdown. Including HTML also enables browser rendering and returns both outputs as JSON.
Get screenshot get-screenshot Capture a base64 screenshot of the full page or top viewport. Browser rendering is always enabled.
Scrape brand scrape-brand Extract a website's colors, typography, logos, and social links as JSON.
Scrape JSON scrape-json Extract structured JSON from a page using a natural-language prompt. Append Json Schema: and a JSON schema to shape the output.
Web search web-search Search the web and return ranked results with titles, URLs, descriptions, and sources.

Installation

Requires Node.js 20 or newer.

npm install -g @mrscraper/cli@latest

For the CLI and optional agent skill pack:

npx -y @mrscraper/cli@latest init --all

For automated setup, use npx -y @mrscraper/cli@latest init --all --yes --skip-auth, then run browser login explicitly when needed.

Authentication

Obtain an API key from the MrScraper app. API key used to authenticate the request.

mrscraper login
mrscraper auth status

Browser sign-in saves a dedicated API key in ~/.mrscraper/auth.json. Keep mrscraper login running until browser approval completes. --no-open prints the sign-in URL. --no-browser is an explicit interactive API-key prompt.

For automation, set MRSCRAPER_API_KEY. --token overrides configured authentication for one service call; it is sent in headers, never in the request URL.

export MRSCRAPER_API_KEY="YOUR_TOKEN"

Quickstart

mrscraper scrape-html https://example.com --browser-rendering --warm-start \
  --wait-for-selector 'body' --timeout 120 --max-retries 0

Examples use example.com as the sample target. Replace it for your use case. Live scraping and search calls consume account quota.

Response examples show selected keys and types observed in live API responses. Values are placeholders; page content, base64 images, and arrays are abbreviated. Actual responses may contain additional fields.

Shared parameters

Page controls

HTML, Markdown, screenshots, and JSON extraction share url, timeout, geoCode, proxyCountry, super, waitUntil, proxy, maxRetries, and saveResult (using the naming style shown below). Page defaults are 300 seconds, US geolocation/proxy country, real-device routing off, and dashboard saving off. Omitted retry limits mean no limit. Exact types, defaults, and standard descriptions appear in each service table.

Loading controls

HTML, Markdown, and screenshots additionally accept blockResources, waitForSelector, and returnCookie. JSON extraction does not accept these controls. Browser rendering is optional for HTML, Markdown, and JSON; screenshots always enable it. Markdown with HTML also enables browser rendering.

Service reference

Status

Description

Return the raw account status API response without summarizing, redacting body fields, or adding calculated fields.

Parameters

This service takes no service parameters. Authentication is configured separately.

The HTTP deadline is fixed at 600 seconds. It is not a service parameter.

Example

mrscraper status

Response

The API body contains message and data. Account details remain nested in data, including subscriptionItem and scraperAllowance. The response envelope therefore exposes account details at data.data. Sensitive response headers are removed.

Success response

{
  "status_code": 200,
  "data": {
    "message": "Example response",
    "data": {
      "id": "ACCOUNT_ID",
      "tokenLimit": 100000,
      "tokenUsage": 2500,
      "stripeStatus": "active",
      "user": {
        "name": "Example user",
        "email": "user@example.com",
        "isVerified": true
      },
      "subscriptionItem": {
        "name": "Example plan",
        "tokenLimit": 100000
      },
      "scraperAllowance": {
        "remaining": 97500,
        "plans": [
          {
            "remaining": 97500
          }
        ]
      }
    }
  },
  "headers": {
    "content-type": "application/json; charset=utf-8"
  }
}

Scrape HTML

Description

Retrieve the raw HTML of a page. Browser rendering is optional.

Parameters

Parameter HTTP field Type Default Description
<url> url string Required Absolute HTTP or HTTPS URL of the page to process.
--timeout timeout integer 300 Page load timeout in seconds, from 1 to 300. The HTTP request allows 30 additional seconds.
--geo-code geoCode string "us" Two-letter country code to load the page from, sent in lowercase.
--proxy-country proxyCountry string "us" Two-letter country code of the proxy server, sent in lowercase.
--super super boolean false Whether to route the request through real devices.
--wait-until waitUntil string: domcontentloaded, load, networkidle0, networkidle2 "domcontentloaded" Navigation completion event to wait for: domcontentloaded, load, networkidle0, or networkidle2.
--proxy proxy string Omitted Custom proxy URL used to load the page.
--max-retries maxRetries integer Omitted Maximum number of retries, as a nonnegative integer. Omit this parameter for no limit.
--save-result saveResult boolean false Whether to save the result to the dashboard Result tab.
--block-resources blockResources boolean false Whether to block images, fonts, and stylesheets while loading the page.
--wait-for-selector waitForSelector string Omitted Wait for the page to contain the specified CSS selector before returning the result.
--return-cookie returnCookie boolean false Whether to include browser session cookies in the response.
--browser-rendering browserRendering boolean false Whether to render the page in a browser before returning the result.
--warm-start warmStart boolean false Whether to enable warm start for more reliable scraping, at the cost of additional latency.

Example

mrscraper scrape-html https://example.com --browser-rendering --warm-start \
  --wait-for-selector 'body' --timeout 120 --max-retries 0

Response

The default response body is HTML text. Additional controls can cause the API to return JSON; the returned structure is preserved.

Success response

{
  "status_code": 200,
  "data": "<html><body>Example page content...</body></html>",
  "headers": {
    "content-type": "text/html; charset=utf-8"
  }
}

Scrape Markdown

Description

Retrieve a page as Markdown. Including HTML also enables browser rendering and returns both outputs as JSON.

Parameters

Parameter HTTP field Type Default Description
<url> url string Required Absolute HTTP or HTTPS URL of the page to process.
--timeout timeout integer 300 Page load timeout in seconds, from 1 to 300. The HTTP request allows 30 additional seconds.
--geo-code geoCode string "us" Two-letter country code to load the page from, sent in lowercase.
--proxy-country proxyCountry string "us" Two-letter country code of the proxy server, sent in lowercase.
--super super boolean false Whether to route the request through real devices.
--wait-until waitUntil string: domcontentloaded, load, networkidle0, networkidle2 "domcontentloaded" Navigation completion event to wait for: domcontentloaded, load, networkidle0, or networkidle2.
--proxy proxy string Omitted Custom proxy URL used to load the page.
--max-retries maxRetries integer Omitted Maximum number of retries, as a nonnegative integer. Omit this parameter for no limit.
--save-result saveResult boolean false Whether to save the result to the dashboard Result tab.
--block-resources blockResources boolean false Whether to block images, fonts, and stylesheets while loading the page.
--wait-for-selector waitForSelector string Omitted Wait for the page to contain the specified CSS selector before returning the result.
--return-cookie returnCookie boolean false Whether to include browser session cookies in the response.
--browser-rendering browserRendering boolean false Whether to render the page in a browser before returning the result.
--warm-start warmStart boolean false Whether to enable warm start for more reliable scraping, at the cost of additional latency.
--html html boolean false Whether to include the page HTML in the response.

Example

mrscraper scrape-markdown https://example.com --html --warm-start

Response

The default response body is Markdown text. Enabling HTML returns JSON containing both HTML and Markdown.

With html enabled

{
  "status_code": 200,
  "data": {
    "html": "<html>...</html>",
    "markdown": "# Example page\n\n..."
  },
  "headers": {
    "content-type": "application/json"
  }
}

Default Markdown text

{
  "status_code": 200,
  "data": "# Example page\n\nExample Markdown content...",
  "headers": {
    "content-type": "text/markdown; charset=utf-8"
  }
}

Get screenshot

Description

Capture a base64 screenshot of the full page or top viewport. Browser rendering is always enabled.

Parameters

Parameter HTTP field Type Default Description
<url> url string Required Absolute HTTP or HTTPS URL of the page to process.
--timeout timeout integer 300 Page load timeout in seconds, from 1 to 300. The HTTP request allows 30 additional seconds.
--geo-code geoCode string "us" Two-letter country code to load the page from, sent in lowercase.
--proxy-country proxyCountry string "us" Two-letter country code of the proxy server, sent in lowercase.
--super super boolean false Whether to route the request through real devices.
--wait-until waitUntil string: domcontentloaded, load, networkidle0, networkidle2 "domcontentloaded" Navigation completion event to wait for: domcontentloaded, load, networkidle0, or networkidle2.
--proxy proxy string Omitted Custom proxy URL used to load the page.
--max-retries maxRetries integer Omitted Maximum number of retries, as a nonnegative integer. Omit this parameter for no limit.
--save-result saveResult boolean false Whether to save the result to the dashboard Result tab.
--block-resources blockResources boolean false Whether to block images, fonts, and stylesheets while loading the page.
--wait-for-selector waitForSelector string Omitted Wait for the page to contain the specified CSS selector before returning the result.
--return-cookie returnCookie boolean false Whether to include browser session cookies in the response.
--screenshot screenshot string: full, top "full" Screenshot area to capture: full for the entire page or top for the top viewport.
--no-html (sets html=false) html boolean true Whether to include the page HTML in the response.
--markdown markdown boolean false Whether to include the page Markdown in the response.

Example

mrscraper get-screenshot https://example.com --screenshot top --no-html --markdown

Response

The response body contains a base64 screenshot and requested page outputs. The API can still include HTML when HTML output is disabled.

Success response

{
  "status_code": 200,
  "data": {
    "html": "<html>...</html>",
    "screenshot": "BASE64_PNG_DATA",
    "markdown": "# Example page\n\n..."
  },
  "headers": {
    "content-type": "application/json; charset=utf-8"
  }
}

Scrape brand

Description

Extract a website's colors, typography, logos, and social links as JSON.

Parameters

Parameter HTTP field Type Default Description
<url> url string Required Absolute HTTP or HTTPS URL of the page to process.

The HTTP deadline is fixed at 120 seconds. It is not a service parameter.

The request uses action=brand-scraping and bundle=none; page controls are not accepted.

Example

mrscraper scrape-brand https://example.com

Response

The API body contains message and a data object with job_id, status, url, final_url, brand, and error. Brand details are available at data.data.brand in the response envelope.

Success response

{
  "status_code": 200,
  "data": {
    "data": {
      "job_id": "JOB_ID",
      "status": "succeeded",
      "url": "https://example.com",
      "final_url": "https://example.com",
      "brand": {
        "name": "Example brand",
        "colors": {
          "background": "#ffffff",
          "text": "#222222"
        },
        "typography": {
          "body": {
            "font_family": "Example font"
          }
        }
      },
      "error": null,
      "bundle_url": null
    },
    "message": "Example response"
  },
  "headers": {
    "content-type": "application/json"
  }
}

Scrape JSON

Description

Extract structured JSON from a page using a natural-language prompt. Append Json Schema: and a JSON schema to shape the output.

Parameters

Parameter HTTP field Type Default Description
<url> url string Required Absolute HTTP or HTTPS URL of the page to process.
--timeout timeout integer 300 Page load timeout in seconds, from 1 to 300. The HTTP request allows 30 additional seconds.
--geo-code geoCode string "us" Two-letter country code to load the page from, sent in lowercase.
--proxy-country proxyCountry string "us" Two-letter country code of the proxy server, sent in lowercase.
--super super boolean false Whether to route the request through real devices.
--wait-until waitUntil string: domcontentloaded, load, networkidle0, networkidle2 "domcontentloaded" Navigation completion event to wait for: domcontentloaded, load, networkidle0, or networkidle2.
--proxy proxy string Omitted Custom proxy URL used to load the page.
--max-retries maxRetries integer Omitted Maximum number of retries, as a nonnegative integer. Omit this parameter for no limit.
--save-result saveResult boolean false Whether to save the result to the dashboard Result tab.
--prompt prompt string "Extract all data as complete as possible." Instructions describing the data to extract. Append Json Schema: and a JSON schema to shape the output.
--browser-rendering browserRendering boolean false Whether to render the page in a browser before returning the result.
--html html boolean false Whether to include the page HTML in the response.
--screenshot screenshot boolean false Whether to include a base64 screenshot in the response.

The request uses agent=general and GET. If the encoded URL exceeds 8,000 characters, it switches to POST with url, prompt, and agent in the body; page controls remain in the query.

Example

mrscraper scrape-json https://example.com --browser-rendering \
  --prompt 'Extract the first 2 items with title and price. Json Schema: {"type":"array","items":{"type":"object","properties":{"title":{"type":"string"},"price":{"type":"string"}},"required":["title","price"]}}'

Response

Extraction results are in the API body's data field and follow the requested schema. Response metadata includes thread_id, token_usage, status, error, runtime, and cookies. html, markdown, and screenshot can be empty strings when not requested.

Success response

{
  "status_code": 200,
  "data": {
    "success": true,
    "data": [
      {
        "title": "Example item",
        "price": "10.00"
      }
    ],
    "thread_id": "THREAD_ID",
    "token_usage": 1,
    "html": "",
    "markdown": "",
    "screenshot": "",
    "status": "Results ready",
    "error": "",
    "runtime": 1.23,
    "cookies": []
  },
  "headers": {
    "content-type": "application/json"
  }
}

Web search

Description

Search the web and return ranked results with titles, URLs, descriptions, and sources.

Parameters

Parameter HTTP field Type Default Description
<query> query string Required Search query text, trimmed before sending, containing 1 to 1000 characters.
--country country string "us" Two-letter country code to search from, sent in lowercase.
--num-results numResults integer 10 Number of search results to request, as an integer from 10 to 100. Fewer results may be returned.

The HTTP deadline is fixed at 120 seconds. It is not a service parameter.

Example

mrscraper web-search "site:example.com documentation" --country id --num-results 20

Response

The API body contains success, message, a data array, and tokenUsage. Each result includes rank, title, url, description, and source.

Success response

{
  "status_code": 200,
  "data": {
    "success": true,
    "message": "Example response",
    "data": [
      {
        "rank": 1,
        "title": "Example result",
        "url": "https://example.com",
        "description": "Example result description.",
        "source": "example.com"
      }
    ],
    "tokenUsage": 1
  },
  "headers": {
    "content-type": "application/json; charset=utf-8"
  }
}

Response format

Successful services use an envelope containing status_code, data, and headers. The data field contains the API response body; its field names are not converted to the SDK's naming style. Status preserves the complete body without adding summary fields.

{
  "status_code": 200,
  "data": "<html><body>Example page content...</body></html>",
  "headers": {
    "content-type": "text/html; charset=utf-8"
  }
}

The CLI prints the envelope as JSON to stdout. It filters credential-bearing response headers and redacts known credential metadata in scraping/search JSON; status body fields are preserved.

Error handling

Failed API calls print a JSON envelope with error and set a nonzero exit code. HTTP status remains in status_code; network failures use null. Progress is written to stderr so stdout stays machine-readable. Invalid arguments fail before the request.

mrscraper scrape-html https://example.com > result.json

Integration features

Configuration

The CLI also provides init, login, auth status, logout, and setup skills. Service options use kebab-case. status --json is accepted for compatibility; JSON is already the default.

Agent setup supports Claude Code, Cursor, Codex, Grok Build, Hermes Agent, OpenCode, OpenClaw, Pi, and Oh My Pi. init --all targets detected harnesses; init --agent NAME targets one. --yes, --skip-auth, and noninteractive input leave authentication for an explicit login.

mrscraper init --agent codex --yes --skip-auth
mrscraper init --all --yes --skip-auth --dry-run
mrscraper setup skills
mrscraper setup skills --agent hermes

Advanced usage

Three repository-hosted skills cover setup and account usage, page retrieval, and search:

For programmatic JavaScript usage, import scrapeHtml, scrapeMarkdown, getScreenshot, scrapeBrand, scrapeJson, webSearch, or getSubscriptionAccountApi from @mrscraper/cli. These low-level exports accept options objects; the internal JavaScript option for super is superMode.

import { scrapeHtml } from "@mrscraper/cli";
const result = await scrapeHtml({
  token: process.env.MRSCRAPER_API_KEY,
  url: "https://example.com",
});
console.log(result.data);

Hosted MCP setup is separate from the CLI and agent skill bootstrap. MrScraper MCP exposes status, scrapeHtml, scrapeMarkdown, getScreenshot, scrapeBrand, scrapeJson, and webSearch. Native agent plugins are maintained separately for ChatGPT and Codex and Claude.

Troubleshooting

For missing credentials, run mrscraper login or configure MRSCRAPER_API_KEY. For slow pages, increase the page timeout up to 300 seconds. Brand and search have fixed 120-second transport deadlines. Use --help on a command to inspect its accepted options. Save large output with shell redirection.

Migration

Replace account with status. Replace combined scraperAPI / scraper-api calls with scrape-html, scrape-markdown, or get-screenshot. Search uses web-search with country and num-results. Legacy locale, header, token-budget, result, rerun, and bundle controls are outside the seven-service interface. super and saveResult default to false; omitted maxRetries means no limit.

Development and testing

Automated checks

The automated suite checks service behavior without production API calls.

npm install
npm test
node bin/mrscraper.js --help

Manual checks

The following commands call the live API using your credentials and consume quota.

mrscraper status
mrscraper scrape-html https://example.com
mrscraper web-search "site:example.com documentation"

License

MIT. See LICENSE.

About

No description, website, or topics provided.

Resources

Stars

4 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages