Skip to content

feat(client): opt-in retries that follow the shared retry policy, off by default - #58

Open
gonzalo-zenrows wants to merge 1 commit into
mainfrom
feat/client-retry-jitter
Open

gonzalo-zenrows wants to merge 1 commit into
mainfrom
feat/client-retry-jitter

Conversation

@gonzalo-zenrows

@gonzalo-zenrows gonzalo-zenrows commented Oct 16, 2026 •

Copy link
Copy Markdown

Merge after #65, which adds the changelog section this PR's changelog lines use.

This brings fetch(), get() and extract() in line with the shared retry policy. retries still defaults to 0, so a caller who never sets it sees no change: every request is sent once, with the same headers, answers and errors.

When retries are on (retries above 0)

  1. Retried: a 429, 502, 503 or 504; a 422 other than no_data_extracted; a connection that failed before the request was sent (ECONNREFUSED, ENOTFOUND, EAI_AGAIN, ENETUNREACH, EHOSTUNREACH, UND_ERR_CONNECT_TIMEOUT, or ETIMEDOUT from connect; an AggregateError only when every inner error is one of these).
  2. Never retried: post(); a 500 or any other status; a network error after sending, a response timeout or an abort; any status with original_status, or one listed in allowed_status_codes.
  3. Wait: Retry-After in whole seconds, clamped to 0.25-30 s; otherwise a random 0.25 s to min(retryMaxWait, 2^attempt) s. New retryMaxWait, default 30.
  4. Budget: no retry starts once the time since the first try plus the next wait would pass retryBudget (new, default 90 s).
  5. client.batch, client.crawl, prepareDomain() and preparedDomainStatus() are untouched.

If you already set retries

  1. A 429 and a 502 are now retried. A 422 other than no_data_extracted is still retried.
  2. post() is never retried.
  3. A network error after sending, a response timeout or an abort is no longer retried. A connect timeout still is.
  4. A status from the target site (original_status, allowed_status_codes) is no longer retried.
  5. Waits follow Retry-After (capped at 30 s) or full jitter instead of a fixed 1, 2, 4 s, within the 90 s budget.
  6. Every retries value 2.6.0 accepted still builds a client with the same count (a numeric string is its number, a fraction rounds up, 0 or less is off). NaN, Infinity or another non-number now turns retries off with one warning; before, it retried without limit.

Tests

  1. The shared conformance vectors run as tables: Retry-After values, jitter bounds, statuses and network errors, plus one test per rule (target-site statuses, budget, post(), prepared-domain calls).
  2. A count table builds a client for each retries value 2.6.0 accepted and checks the requests sent and the warning; it matches a probe run against 2.6.0.
  3. A refused connection and a socket closed after sending run through Node's real fetch against a local server; with no retries, the refused connection is sent once.
  4. pnpm check, pnpm test -- --run, pnpm build and tsc --noEmit pass; the built .d.ts adds the two options and one private field, nothing else.

BEGIN_COMMIT_OVERRIDE
feat(client): opt-in retries following the shared retry policy, with Retry-After, full jitter and a 90 s budget; still off by default

behaviour(client): with retries set, a 429 and a 502 are now retried; post(), errors after sending, response timeouts, aborts and target-site statuses no longer are; waits follow Retry-After or full jitter; NaN or Infinity turns retries off
END_COMMIT_OVERRIDE

@gonzalo-zenrows

Copy link
Copy Markdown
Author

@jonathan-zenrows jonathan-zenrows left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't be merged as is.

1. Conflicts with #49, and the fallback in this branch is now dead code

Main is on 2.5.0 with #49, which removed the extract() AutoParse fallback and added prepareDomain/preparedDomainStatus. This branch still has fallbackToAutoparse = true and the shared clock (src/index.ts:189-208), and src/index.ts conflicts. If the conflict is resolved by keeping this side, #49 gets reverted without anyone noticing: the fallback comes back on and the prepared-domain exports disappear. After the rebase, delete the CallClock fallback path, the tests at tests/retries.test.ts:375-415, and the README sentence about the fallback sharing the budget.

2. Retries hold queue slots

The backoff sleep runs inside the fastq worker (src/index.ts:130, 306). With the default concurrency: 5, five calls that get 429 + Retry-After: 30 block every other call on that client for 30 s. Before this PR nobody hit it, because retries defaulted to 0. Repro: concurrency 5, a server that answers 429 Retry-After: 30 to the first 5 requests, and fire a 6th call: it waits 30 s before it even leaves. It should release the slot while it waits, or at minimum document this.

3. Release note

0 to 2 retries, plus RangeError on bad options (src/retry.ts:57-76), ships as 2.6.0 with no note under feat. It needs feat!/BREAKING CHANGE: or a written changelog line.

Small things:

  • {"Idempotency-Key": 123} from plain JS throws a TypeError at value.trim() (src/retry.ts:116). Use String(value).trim().
  • ENETUNREACH/EHOSTUNREACH are retried here but not in the other SDKs. They're fine to retry, but let's make all 5 the same.
  • localhost (happy-eyeballs → AggregateError) isn't tested, only 127.0.0.1. On Node 26 the code arrives correctly; add a test so we don't depend on that.

@gonzalo-zenrows gonzalo-zenrows changed the title feat(client): retry uncharged failures by default, with full-jitter backoff and a time budget feat(client): opt-in retries with full-jitter backoff and a time budget, off by default Nov 2, 2026
… by default

With retries above 0, fetch(), get() and extract() retry a 429, 502, 503,
504, a 422 other than no_data_extracted, and a connect failure before the
request was sent. post(), a network error after sending, a response timeout,
an abort and target-site statuses are never retried. Waits follow Retry-After
or full jitter (new retryMaxWait), within a 90 s budget (new retryBudget).
Every retries value 2.6.0 accepted still builds a client with the same count;
NaN and Infinity now turn retries off with a warning.
@gonzalo-zenrows
gonzalo-zenrows force-pushed the feat/client-retry-jitter branch from 5a582d8 to d10c3cb Compare November 3, 2026 15:02
@gonzalo-zenrows gonzalo-zenrows changed the title feat(client): opt-in retries with full-jitter backoff and a time budget, off by default feat(client): opt-in retries that follow the shared retry policy, off by default Nov 3, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants