Skip to content

Commit a1dda87

Browse files
zydozydo
committed
Document worker↔engine binding & on-demand-search concurrency
Design record capturing the exploration: why each engine is bound to one warm worker/context today, the sequential-serialization limit that surfaces for on-demand web search (and the live diagnosis that it's capacity, not cooldown or parallelism), the Layer-1 backpressure fix shipped (max_in_flight semaphore → 429, skip-stale jobs, planned observability), and the future direction — decoupling workers from engines into a warm-session pool, with the persistent per-(engine × exit-IP) rate constraint and shared pacing/cooldown cost spelled out.
1 parent 59acd42 commit a1dda87

1 file changed

Lines changed: 142 additions & 0 deletions

File tree

‎docs/WORKER_ENGINE_BINDING.md‎

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
# Worker ↔ engine binding & on-demand-search concurrency
2+
3+
> Design record (2026-06-21). Why each search engine is bound to one worker /
4+
> warm browser session, the concurrency limit that surfaces for **on-demand web
5+
> search**, the backpressure fix we shipped, and the future direction
6+
> (decoupling workers from engines into a warm-session pool). Companion to
7+
> [TASK_SCHEDULER.md](TASK_SCHEDULER.md), [WEB_SEARCH_WARMUP.md](WEB_SEARCH_WARMUP.md),
8+
> and [SCRAPING_BEHAVIOR.md](SCRAPING_BEHAVIOR.md).
9+
10+
## TL;DR
11+
12+
- **Today:** 1 engine = 1 worker = 1 warm browser context (single thread). Simple
13+
and lock-free, but on-demand web searches for a given engine are served
14+
**strictly sequentially** (~one warm session, ~5-6s/serve).
15+
- **Symptom:** concurrent `/api/v1/search` requests mostly `timeout` (only ~4 fit
16+
in the 25s window); it was *not* cooldown and *not* parallel-but-slow.
17+
- **Shipped (Layer 1):** per-engine **in-flight cap** (`max_in_flight`,
18+
default 4) → the N+1th request is rejected fast with **HTTP 429 + Retry-After**
19+
instead of a silent 25s hang; the worker also skips serving jobs older than the
20+
request timeout. Plus a planned web-search **observability** panel on `/monitor`.
21+
- **Future:** for real search concurrency, **decouple workers from engines** — a
22+
*pool* of warm "generalist" sessions pulling from a shared cross-engine queue.
23+
The catch that doesn't go away: the rate limit is **per-(engine × exit-IP)**, so
24+
a pool needs **shared per-engine pacing + cooldown** (reintroducing the
25+
cross-context coordination the current model avoids).
26+
27+
## Why engines are bound to workers today
28+
29+
Introduced with the per-engine parallel workers (2026-06-19; see
30+
`scraper/worker.py`, `scraper/main.py`):
31+
32+
1. **Cross-engine parallelism is detection-neutral.** Engines can't see each
33+
other's traffic, so running google/bing/yahoo/brave workers in parallel adds
34+
no per-engine detection risk — the risk is per-engine request *rate*.
35+
2. **Per-engine cooldown maps 1:1 to a single-writer worker** → no locking around
36+
the cooldown/pacing state (`scraper/cooldown.py`); each worker is the sole
37+
writer of its own engine's state.
38+
3. **One stable identity per engine.** Each worker has its own persistent profile
39+
dir (cookies/warmth isolated), and Chromium holds a `SingletonLock` per
40+
profile — one Chromium process per profile.
41+
4. **Continuous news scraping** per engine fits a dedicated long-lived session.
42+
43+
On-demand web search was layered on by **reusing** each engine's existing warm
44+
news session (see [WEB_SEARCH_WARMUP.md](WEB_SEARCH_WARMUP.md)) rather than
45+
spinning up new sessions — one request stream per engine per exit IP.
46+
47+
## The concurrency limit (the bug report)
48+
49+
On-demand web search (`GET /api/v1/search`, Google-only for now) mostly returned
50+
`timeout`/`blocked` under concurrent load; only some succeeded.
51+
52+
**Live diagnosis (2026-06-21):**
53+
54+
- **Not cooldown.** Google had `failures=0`, `next_probe_at=NULL` throughout; a
55+
benched engine returns `unavailable`, not `timeout`.
56+
- **Not "parallel but slow."** One worker on one context in one thread → two
57+
Google searches *cannot* run at once. They're served **strictly sequentially,
58+
FCFS** (`claim_web_search_job ORDER BY created_at`), and the worker drains the
59+
web queue **before** news (web is polled first each turn).
60+
- **It's capacity.** A single isolated search serves in ~4s, but the `request_timeout`
61+
is 25s and each serve is ~5-6s (incl. the ~2s pace floor), so only ~4 concurrent
62+
fit; the rest wait past 25s and time out. A web job that lands behind an
63+
in-flight ~15s news scrape eats into the window too.
64+
65+
So the behavior already matched "served if your turn comes in time, else timeout"
66+
— there was no logic bug, just an unbounded queue with a silent failure mode.
67+
68+
## What we shipped — Layer 1 (backpressure + skip-stale)
69+
70+
Commit `59acd42`. The chosen mechanism was a **semaphore (max in-flight = N)**,
71+
*not* a strict "singleton acquire-or-fail":
72+
73+
- A strict singleton is just **N=1** — it rejects any request arriving while one
74+
is serving, throwing away cheap useful queueing.
75+
- The semaphore admits up to **N** in-flight (pending+claimed) jobs per engine
76+
(they wait their turn, served sequentially) and rejects the **N+1th fast**.
77+
Singleton is the degenerate N=1 case.
78+
79+
Implementation:
80+
- `scraper.web_search.max_in_flight` (default 4) — sized to how many serves fit
81+
in `request_timeout` (~6s each → ~4).
82+
- **Capacity-checked enqueue** (`db.enqueue_web_search` with `max_in_flight`): an
83+
atomic `INSERT … SELECT … WHERE count(in-flight) < N` that returns no row when
84+
full. Soft cap (a hair of over-admission under heavy races is fine).
85+
- Dispatcher (`api/websearch.py`): on a full engine, record status **`busy`** and
86+
fall back to an engine with room (if fan-out is enabled); the endpoint maps
87+
`busy` → **HTTP 429 + `Retry-After`**. Scoped to **web search only** — news is
88+
internal/scheduled with no caller to fail-fast.
89+
- **Skip stale jobs:** the worker no longer claims jobs older than the request
90+
timeout (`claim_web_search_job(max_age_seconds=…)`) — a requester that already
91+
gave up shouldn't burn the single serve slot.
92+
- **Observability (planned, same Layer 1):** web one-offs bypass
93+
`scraper_logs`/`scraper_cycles` (news-only) and the job row is deleted after the
94+
API reads it, so failures leave no durable trace and `/monitor` shows nothing.
95+
Add per-request logging (`query, engine, outcome, latency, attempts`) + a
96+
web-search panel. (Top of [TODO](../TODO.md) "NEXT UP".)
97+
98+
Verified live: 8 concurrent requests → 4 admitted, 4 immediate `429` (~50ms).
99+
100+
## The architecture discussion (future directions)
101+
102+
### Option A — multiple workers for *one* engine (same IP) → rejected
103+
Two contexts hitting one engine from the same exit IP **doubles that engine's
104+
request rate from the IP** → more blocks, and two identities for one engine looks
105+
suspicious + splits warmth. The right way to scale one engine's throughput is
106+
**more exit IPs** (separate machines) — which the saturation signal
107+
(`scraper/saturation.py`) already flags. More contexts per IP fights the rate we
108+
pace against.
109+
110+
### Option B — multi-engine fan-out → partial, deferred
111+
Spread concurrent searches across the existing per-engine workers (N engines =
112+
N concurrent). Real, and it's the "add other engines later" already planned — but
113+
currently web search is Google-only by request (`web_search.engines = [google]`),
114+
so this is parked.
115+
116+
### Option C — decouple workers from engines (warm-session pool) → the real future fix
117+
A **pool of M warm "generalist" sessions**, each able to serve *any* engine,
118+
pulling from a **shared cross-engine task queue**. Benefits:
119+
- **M-way concurrency for on-demand search** — a Google burst spreads across M
120+
contexts (load-balanced), directly solving the limit above.
121+
- Arguably **more realistic** (one browser visiting multiple engines) — though
122+
this is a bonus, not the main driver.
123+
124+
**The constraint that does NOT disappear:** the rate limit is
125+
**per-(engine × exit-IP)** regardless of which context sends the request. So a
126+
pool needs **shared per-engine pacing + cooldown across the whole pool** (e.g.,
127+
M contexts must not all hit Google at once) — reintroducing the cross-context
128+
coordination/locking that today's single-writer-per-engine model avoids. It also
129+
**dilutes each engine's warmth** across more contexts (each visits a given engine
130+
less often). Manageable, but real.
131+
132+
**Trade summary:** simple, lock-free, engine-serialized (today) ↔ concurrent,
133+
load-balanced, needs shared per-engine rate/cooldown state (pool).
134+
135+
## Recommendation / when to revisit
136+
137+
- Keep the **per-engine worker** model for now — it's simple and correct, and
138+
Layer 1 backpressure makes the concurrency limit graceful and honest.
139+
- When on-demand search volume justifies real concurrency, build **Option C**:
140+
warm-session pool + shared cross-engine queue + shared per-engine pacer/cooldown
141+
(a generalization of fan-out). Don't scale a single engine by adding contexts on
142+
one IP (Option A) — add IPs instead.

0 commit comments

Comments
 (0)