ebay-scrapper

FastAPI service that scrapes eBay product data with anti-blocking baked in. Built on Scrapling (StealthyFetcher) and exposes a small REST API designed to be driven by a cron job (or any external scheduler).

Features

• 3 endpoints - keyword search, single item, category browse
• Pagination handled - auto-paginates eBay search/category results using _pgn=N, detects the last page via the pagination__next button, throttles between pages
• Anti-blocking - stealthy headless browser, randomized delays, ad/resource blocking, Cloudflare-aware fetcher, optional proxy, captcha/block-page detection
• Caching - SQLite-backed TTL cache so repeated scrapes don't hammer eBay
• Layered architecture - api → services → scrapers → schemas/db, easy to swap or add another marketplace later

Project layout

app/
├── main.py                 # FastAPI app + lifespan
├── core/                   # config, logging, exceptions
├── api/
│   ├── deps.py             # dependency wiring
│   └── v1/                 # versioned endpoints (health, products)
├── schemas/                # Pydantic request/response models
├── services/               # orchestration + cache
├── scrapers/               # Scrapling-backed scrapers (BaseScraper, EbayScraper)
└── db/                     # SQLAlchemy async engine + ORM models
scripts/
├── run_dev.sh              # uvicorn --reload
└── cron_scrape.sh          # example cron entry that hits the API

Setup

python3.13 -m venv .venv
.venv/bin/pip install -r requirements.txt

# Download Playwright browsers (one-time, ~500MB)
.venv/bin/scrapling install

cp .env.example .env  # tweak as needed

Run

./scripts/run_dev.sh
# or: .venv/bin/uvicorn app.main:app --reload

Visit:

• Swagger UI - http://localhost:8000/docs
• OpenAPI JSON - http://localhost:8000/openapi.json

API

All endpoints live under /api/v1.

GET /products/search

param	type	default	notes
q	string	-	required, 1–300 chars
max_pages	int	5	clamped to PAGINATION_HARD_MAX_PAGES
per_page	int	-	eBay _ipg, e.g. 60, 120, 240
sort	string	-	eBay _sop, e.g. 12 best, 15 ending soonest
use_cache	bool	true	set false to force a fresh scrape

curl 'http://localhost:8000/api/v1/products/search?q=iphone%2015&max_pages=3'

GET /products/{item_id}

curl 'http://localhost:8000/api/v1/products/123456789012'

GET /products/category/{category_id}

curl 'http://localhost:8000/api/v1/products/category/9355?max_pages=3'

Pagination behaviour

• Walks pages by appending _pgn=N to the URL.
• Stops early when the "next" button is missing or aria-disabled="true".
• Stops if a page returns zero items (eBay's drift past the last real page).
• Throttles between pages with a random delay in [PAGINATION_MIN_DELAY_SEC, PAGINATION_MAX_DELAY_SEC].
• If a page mid-crawl fails, returns the items already collected with partial=true in the response (the partial result is not cached).

Anti-blocking

Out of the box:

• TLS-impersonated HTTP via AsyncFetcher (curl_cffi with Chrome fingerprint). This is the default path - eBay's CDN (Akamai) hard-blocks Playwright/patchright browser fingerprints with a 403 "Access Denied", so the stealth browser path is off by default. Toggle via SCRAPE_USE_STEALTH=true if needed for another site.
• google_search=True → sets a Google referrer so requests look organic.
• Randomized 3–7s delays between paginated requests.
• Polite retry-on-block: if a fetch is detected as blocked, sleep RETRY_BLOCK_BACKOFF_SEC and retry once.
• Block detection covers: 403/429/500 responses, "Access Denied" pages, the splashui/challenge JS interstitial, "Pardon Our Interruption" pages, and generic captcha markup. Any of these surface to the API as HTTP 503.

Geo / regional blocks

eBay's edge applies different policies per region. From outside the US, www.ebay.com/sch/... returns a hard 403 immediately. Use a regional eBay that's reachable from your IP:

Region	Base URL	Currency parsed as
Australia	https://www.ebay.com.au	AUD
UK	https://www.ebay.co.uk	GBP
Germany	https://www.ebay.de	EUR
US	https://www.ebay.com	USD (needs US IP)

Set EBAY_BASE_URL in .env to switch.

When eBay still blocks you

After enough requests from the same IP, eBay starts serving the SplashUI JS challenge that the plain HTTP fetcher can't solve. Mitigations:

1. Slow down - bump PAGINATION_MIN/MAX_DELAY_SEC and run cron less often.
2. Use a residential proxy - set PROXY_URL in .env (HTTP or SOCKS). Long term, swap a single proxy for a rotating pool - the seam is in base.py.
3. Wait it out - the SplashUI rate limit usually clears in 10–30 minutes.

Cron

Use external cron (system cron, GitHub Actions, cloud scheduler) to keep the API stateless. Example:

0 * * * * /full/path/to/scripts/cron_scrape.sh "iphone 15" 5 >> /var/log/ebay_cron.log 2>&1

The script writes one JSON snapshot per run to data/snapshots/.

Configuration

All knobs in .env (defaults in config.py). Highlights:

key	purpose
SCRAPE_HEADLESS	run browser headless
SCRAPE_TIMEOUT_MS	per-page timeout
PAGINATION_HARD_MAX_PAGES	absolute ceiling regardless of API param
PAGINATION_MIN/MAX_DELAY_SEC	throttle window between pages
CACHE_TTL_SECONDS	how long results live in SQLite
PROXY_URL	optional proxy for all fetches

Notes & caveats

• eBay rotates HTML class names. If parsing breaks, update the _extract_* helpers in ebay.py - pagination, throttling, and block detection are selector-agnostic.
• Use this responsibly. Respect eBay's Terms of Service and robots.txt for your use case.

Made with ❤️ by rbayuokt