Skip to content

concurrent-oauth-refresh-invalid-grant.md

Latest commit

 

History

History
437 lines (348 loc) · 24 KB

concurrent-oauth-refresh-invalid-grant.md

File metadata and controls

437 lines (348 loc) · 24 KB
Raw

invalid_grant / refresh_token_reused under concurrent OAuth2 token refresh: cause & fix

A vendor-neutral engineering guide. Copy-paste code, no fabricated benchmarks. Web version (no trackers): https://refresh-guard-guide.pages.dev/

TL;DR

The cause: modern refresh tokens are single-use and rotating — the first successful refresh invalidates the token it was sent with and hands back a new one. When N requests all notice the access token expired at the same instant, they each POST to the token endpoint with the same refresh token. One wins; the losers present a token that has just been revoked and get invalid_grant — or, on providers with reuse detection (Okta, Auth0, Salesforce), refresh_token_reused, which can revoke the whole token family and log the user out everywhere.

The fixes (apply all that match your topology):

  • In-process single-flight — the first caller starts the refresh and stores the in-flight Promise; everyone else awaits the same promise; clear it in finally. Exactly one refresh per expiry, within one process.
  • Cross-process lock — when several processes (CLIs, workers, agents) share one credential file: take an exclusive lock (lock file / O_EXCL), re-read the token after acquiring the lock and short-circuit if it was already rotated, then write atomically (temp file + rename).
  • Rotation-merge — when the refresh response omits a new refresh_token (Google does this), keep the previous one; only replace on actual rotation.
  • Re-read before failing — on invalid_grant, re-read the stored token once: a sibling may have already refreshed it, turning a hard failure into a retry that succeeds.

Important distinction: in-process single-flight does not fix the cross-process case, and a cross-process lock alone is heavier than you need inside one process. They solve two different problems.

The race, step by step

An OAuth2 client holds an access token (short-lived) and a refresh token (long-lived). When the access token expires you POST the refresh token to the token endpoint and get a fresh access token back. With refresh-token rotation — now the default at Okta, Auth0, Microsoft, Salesforce, and recommended by the OAuth 2.0 Security BCP for public clients — that refresh token is single-use: the response also carries a new refresh token, and the one you just sent is immediately invalidated.

The bug appears when more than one request needs a token at the same time. This is the norm, not the exception: a page that fires several API calls in parallel, a worker pool, a job that fans out, or several CLIs/agents sharing one stored credential. Here is the timeline with two callers, A and B:

t0   access token is expired (or within the skew window)
t1   caller A reads creds, sees "expired", POSTs refresh_token = R0
t2   caller B reads creds, sees "expired", POSTs refresh_token = R0   // same token!
t3   provider processes A: issues access A1 + rotates R0 -> R1, REVOKES R0
t4   provider processes B: R0 is revoked  ->  400 invalid_grant
                                            (reuse-detecting providers: refresh_token_reused,
                                            may revoke the whole R-family)

Both callers did exactly what the textbook flow says. The defect is that they did it concurrently with a token that only tolerates being used once. The symptoms you actually see: random invalid_grant, surprise re-authentication, a corrupted token file (two writers racing a read-modify-write), and the classic "works locally, fails under load."

Why do I get invalid_grant under concurrent refresh?

Because the loser of the race presents a refresh token that the winner already rotated away. The provider invalidated R0 the moment it issued R1 to the first caller; the second caller is now sending a revoked grant, and invalid_grant is the spec's "this grant is no longer valid" answer.

invalid_grant is overloaded; the fix depends on which cause you actually have:

Cause of invalid_grant Concurrency-related? Fix
Refresh token was rotated by a concurrent refresh and you sent the old one Yes — the focus here Single-flight + (if multi-process) a lock + re-read
You overwrote/lost the refresh token (Google omitted it, you saved the response as-is) Often surfaces under concurrency Rotation-merge on persist
User revoked access, changed password, or the token genuinely expired/was idle too long No Re-authenticate; terminal
Clock skew, or wrong redirect_uri/code on an auth-code exchange No Refresh with a skew margin; fix the request

The trap: invalid_grant reads like "the user is logged out, give up and re-auth." Under concurrency it usually isn't — it's "a sibling request already refreshed; your copy is stale." Re-authenticating on every concurrency-induced invalid_grant produces exactly the "surprise re-login" symptom. Re-read before failing (below).

refresh_token_reused and reuse detection

Some providers go further than rejecting the old token. Refresh-token reuse detection (Okta, Auth0, Salesforce) treats the presentation of an already-rotated refresh token as a likely token-theft replay — the legitimate pattern (a stolen token used after the real client rotated) is indistinguishable from your innocent race. So the provider does the safe thing: it revokes the entire refresh-token family, invalidating both the thief's copy and yours, and the user is logged out everywhere.

That is why a concurrency bug here is more dangerous than a plain invalid_grant: a single racing refresh can nuke a working session. On these providers, serializing refresh is not an optimization, it is a correctness requirement.

Definitions: refresh-token rotation means each refresh returns a new refresh token and invalidates the old one. Reuse detection means presenting an already-rotated refresh token is treated as a security event that revokes the whole token family. Rotation makes concurrency throw errors; reuse detection makes those errors expensive.

How do you serialize OAuth token refresh?

The whole fix reduces to one rule: make exactly one refresh happen, and have every other caller use its result instead of starting their own. There are two scopes, and you may need both:

  1. In-process (single process, many concurrent tasks): use single-flight — coalesce concurrent callers onto one shared Promise.
  2. Cross-process (many processes share one credential): add an inter-process lock with a re-read after acquiring it, plus atomic writes so no reader ever sees a half-written token.

And independently of locking, always merge rotation correctly when persisting, and refresh proactively a little before expiry (a small skew, e.g. 30–60s) so requests don't all hit the cliff at the same instant.

In-process single-flight (with code)

Single-flight is the smallest correct fix for the common case (one process, concurrent calls). The first caller to see expiry starts the refresh and stores the in-flight Promise; every other caller awaits the same promise rather than kicking off its own; the promise is cleared in a finally so the next expiry can refresh again. JavaScript's single-threaded event loop makes "check the flag, set the promise" atomic — there is no lock to take.

let inflight = null;            // the single shared refresh promise (null when idle)
let creds = loadCreds();        // { access_token, refresh_token, expires_at }
const SKEW_MS = 60_000;         // refresh ~1 min before real expiry

function isValid(c) {
  return c?.access_token && c.expires_at - SKEW_MS > Date.now();
}

async function getValidToken() {
  if (isValid(creds)) return creds.access_token;     // fast path, no refresh

  // SINGLE-FLIGHT: if a refresh is already running, await THAT one.
  if (!inflight) {
    inflight = doRefresh(creds)
      .then(next => { creds = next; return next; })
      .finally(() => { inflight = null; });          // clear so the next expiry can refresh
  }
  const refreshed = await inflight;                  // every concurrent caller awaits the SAME promise
  return refreshed.access_token;
}

async function doRefresh(prev) {
  const res = await fetch(TOKEN_URL, { method: "POST", body: form(prev.refresh_token) });
  if (!res.ok) throw new Error(`refresh failed: ${res.status}`);
  const body = await res.json();                     // { access_token, expires_in, refresh_token? }
  return persist(mergeRotation(prev, body));         // merge + write atomically (see below)
}

With 50 callers all hitting getValidToken() on an expired token, exactly one doRefresh runs; the other 49 await its result. The two details that are easy to get wrong:

  • Clear the promise in finally, not then — otherwise a failed refresh leaves a rejected promise wedged in inflight forever, and every future call re-rejects with the stale error (a "stuck promise"). finally clears it on both success and failure so the next call retries cleanly.
  • Store the promise before the first await — assign inflight synchronously so a second caller arriving on the next microtask sees it.

This alone fixes most reports. If your token lives in one process — a server, a single worker, a browser tab — single-flight + rotation-merge is the complete fix. You do not need a lock file.

Does an in-process lock fix it?

No — not if more than one process shares the same refresh token. An in-process single-flight (a shared promise, an async mutex, a library's internal lock) coalesces refreshes within one event loop. Two separate processes each have their own memory and their own inflight variable, so they cannot see each other's in-flight refresh. If two CLIs, two workers, two containers, or two agents read the same credential file and both refresh, you are right back in the race — single-flight did nothing for them.

Your topology What you need
One process, concurrent async calls / request fan-out In-process single-flight (+ rotation-merge)
One token file shared by multiple CLIs / workers / agents / containers Single-flight per process + a cross-process lock + re-read + atomic write
Many machines sharing one credential (distributed) A distributed lock (Redis/DB) or a single token-broker service; a local file lock is not enough

Using single-flight where you needed a lock is the most common reason the bug "comes back" after you thought you fixed it. Be explicit about topology.

Cross-process lock (with code)

When several processes share one stored credential, you need three things together: an exclusive lock, a re-read after acquiring it (so the process that waited doesn't refresh a token a sibling already rotated), and an atomic write (so a racing reader never sees a half-written file).

1. An exclusive lock via O_EXCL

Opening a lock file with the exclusive-create flag (wx in Node, O_CREAT | O_EXCL at the syscall level) succeeds for exactly one process and fails with EEXIST for the rest — a portable mutex. Write the holder's PID and timestamp so a stale lock from a crashed process can be reclaimed.

import { open, readFile, rename, writeFile, unlink } from "node:fs/promises";

const LOCK = TOKEN_PATH + ".lock";
const STALE_MS = 30_000;

async function withTokenLock(fn) {
  for (;;) {
    let fh;
    try {
      fh = await open(LOCK, "wx");                    // O_CREAT|O_EXCL: only ONE process wins
      await fh.writeFile(JSON.stringify({ pid: process.pid, at: Date.now() }));
      try { return await fn(); }
      finally { await fh.close(); await unlink(LOCK).catch(() => {}); }
    } catch (e) {
      if (e.code !== "EEXIST") throw e;
      if (await lockIsStale(LOCK)) { await unlink(LOCK).catch(() => {}); continue; }
      await sleep(50 + Math.random() * 100);          // someone else holds it; back off and retry
    }
  }
}

2. Re-read after acquiring the lock — the step everyone forgets

By the time you get the lock, the process that held it before you may have already done the refresh. If you blindly refresh anyway, you send a token that was just rotated — the very invalid_grant you were trying to avoid, only now serialized. Re-read the token from disk after acquiring the lock and short-circuit if it's already fresh.

async function getValidTokenMultiProcess() {
  let creds = await readToken();
  if (isValid(creds)) return creds.access_token;      // fast path, no lock

  return withTokenLock(async () => {
    creds = await readToken();                         // *** RE-READ inside the lock ***
    if (isValid(creds)) return creds.access_token;     // a sibling already refreshed -> done

    const next = mergeRotation(creds, await doRefresh(creds));
    await writeTokenAtomic(next);                      // atomic persist (below)
    return next.access_token;
  });
}

3. Atomic persistence (temp file + rename)

A token write must be all-or-nothing. Writing in place lets a concurrent reader catch a truncated file (the read-modify-write corruption). Write to a temp file, then rename it over the target: rename is atomic on the same filesystem, so readers see either the old file or the new file, never a half-written one.

async function writeTokenAtomic(creds) {
  const tmp = TOKEN_PATH + "." + process.pid + "." + Date.now() + ".tmp";
  await writeFile(tmp, JSON.stringify(creds), { mode: 0o600 });  // 0600: owner-only
  await rename(tmp, TOKEN_PATH);                                 // atomic swap
}

Order matters: lock -> re-read -> refresh only if still stale -> atomic write -> release. Drop the re-read and the lock just serializes the same bug. Drop the atomic write and you trade the network race for a file-corruption race.

Rotation-merge & the "Google dropped my refresh_token" bug

Independent of locking, how you persist the refresh response is its own source of invalid_grant. Providers disagree on what they return:

  • Rotating providers (Okta, Auth0, Microsoft, Salesforce) return a new refresh_token on every refresh — you must save it, or your next refresh uses a revoked token.
  • Google returns a refresh_token only on the first authorization; refresh responses omit it entirely. If you overwrite stored credentials with the refresh response as-is, you erase the refresh token and force a full re-consent.

The single correct rule that handles both — rotation-merge: if the response carries a refresh_token, use it; if it doesn't, keep the previous one.

function mergeRotation(prev, res) {
  const merged = { ...res };
  // provider omitted a new refresh_token (Google) -> keep the old one
  if (!merged.refresh_token && prev?.refresh_token) {
    merged.refresh_token = prev.refresh_token;
  }
  // normalize expires_in (seconds) -> expires_at (ms epoch)
  if (merged.expires_in && !merged.expires_at) {
    merged.expires_at = Date.now() + merged.expires_in * 1000;
  }
  return merged;
}

Naive overwrite (save(res)) silently works for rotating providers and silently breaks Google. Naive "always keep the old refresh_token" silently works for Google and silently breaks rotation. Merge is the only rule that is correct for both.

Re-read before failing on invalid_grant

Even with single-flight, a race can still slip through across processes, after an invalidate(), or at a deploy boundary. So make invalid_grant handling self-healing: before you surface it to the user as "log in again," re-read the stored token once — a sibling may have just refreshed it.

try {
  return await getValidToken();
} catch (e) {
  if (isInvalidGrant(e)) {
    const fresh = await readToken();                 // did someone else just refresh?
    if (isValid(fresh)) return fresh.access_token;   // yes -> recover silently
  }
  throw e;                                           // genuinely revoked -> now re-auth
}

This converts the common concurrency-induced invalid_grant into a transparent retry, and reserves the disruptive re-login for the case where the grant is actually gone (user revoked, password changed, token idle-expired).

Provider quirks (verify against current docs)

Provider Rotates refresh token? Reuse detection? The gotcha
Google No No Omits refresh_token on refresh -> you must keep the old one (merge), or you force re-consent.
Okta Yes (default) Yes Each refresh returns a new token; reusing the old (a race) -> invalid_grant. Single-flight essential.
Auth0 Yes Yes Rotation + automatic reuse detection; a concurrent refresh trips it and can revoke the family.
Salesforce Yes Yes A reused token can revoke the whole chain. Never refresh concurrently.
Microsoft (Entra) Yes No Identity platform rotates refresh tokens; persist the new one atomically every time.
GitHub OAuth apps: no No OAuth-app tokens are non-expiring and the same token comes back; GitHub Apps use expiring user tokens with rotation.

Behaviour reflects documented defaults as of mid-2026 and can be reconfigured per tenant/app; verify against each provider's current docs before relying on it.

Fix checklist

In order of leverage. Items 1–3 fix the single-process case (most reports); 4–6 add the multi-process case; 7–8 are correctness hygiene that prevents the other invalid_grant causes.

  • Refresh proactively with a skew (30–60s before expiry) so callers don't all hit the cliff at once.
  • Single-flight in-process — one shared in-flight Promise; all concurrent callers await it.
  • Clear the in-flight promise in finally so a failed refresh doesn't wedge a stuck/rejected promise.
  • If a credential is shared across processes, take an exclusive lock (lock file / O_EXCL) around refresh+persist.
  • Re-read the token after acquiring the lock and short-circuit if a sibling already rotated it.
  • Persist atomically — temp file + rename, mode 0600; never write the token file in place.
  • Rotation-merge on persist — keep the previous refresh_token when the response omits one (Google); save the new one when it rotates.
  • Re-read before failing on invalid_grant — recover silently if a sibling already refreshed; only re-auth when the grant is genuinely gone.

One library option (honest)

You can implement everything above yourself — the patterns are small and the code on this page is complete enough to copy. If you'd rather not re-derive it, refresh-guard is a small, MIT, zero-dependency library that packages the in-process single-flight + correct rotation-merge + atomic file persistence as one installable primitive, with a typed provider-quirks table for the gotchas above.

import { createTokenManager, fileStore } from "refresh-guard";

const tokens = createTokenManager({
  provider: "google",                                // optional: picks a quirks profile
  store: fileStore("~/.myapp/creds.json"),           // atomic temp-file + rename persistence
  refresh: async (prev) => {
    const r = await fetch(TOKEN_URL, { method: "POST", body: form(prev.refresh_token) });
    return await r.json();                           // { access_token, expires_in, refresh_token? }
  }
});

// Call from anywhere, as often as you like — exactly ONE refresh happens:
const accessToken = await tokens.getValidToken();
npm i refresh-guard

Disclosure & honest scope: refresh-guard is built by the same people who wrote this guide, and we list it because it's exactly the shape this page recommends — not because it's the only way. It solves the in-process case (single-flight) plus rotation-merge and atomic persistence. It does not ship a cross-process lock — if you share one credential across processes, combine it with the lock-file pattern above, or use a self-written cron/broker. Source: github.com/wartzar-bee/refresh-guard · package: refresh-guard on npm. The patterns work with any OAuth client or no library at all.

FAQ

Why do I get invalid_grant under concurrent OAuth2 token refresh? Your refresh token is single-use and rotating. When several requests hit an expired access token at once, each POSTs the same refresh token. The first refresh succeeds and the provider rotates (revokes) that token; the other in-flight requests then present the now-revoked token and get invalid_grant. On Okta/Auth0/Salesforce, reuse detection may report refresh_token_reused and revoke the whole token family. Fix: make exactly one refresh happen and have everyone else await its result.

How do you serialize OAuth token refresh? In one process, single-flight: the first caller starts the refresh and stores the in-flight Promise; every other caller awaits that same Promise; clear it in finally. Across processes (CLIs/workers/agents sharing a credential file), add an exclusive lock (O_EXCL lock file or flock), re-read the token after acquiring the lock and short-circuit if it was already rotated, refresh only if still stale, and write atomically (temp file + rename).

Does an in-process lock fix concurrent refresh across processes? No. In-process single-flight only coalesces refreshes within one process and event loop. Two separate processes each have their own memory and their own in-flight Promise, so they can't see each other's refresh. For the multi-process case you need an inter-process lock plus a re-read after acquiring it and atomic writes. They solve two different problems; use both when one credential is shared across processes.

Why does Google drop my refresh_token after a refresh? Google returns a refresh_token only on the first authorization; refresh responses omit it. If you overwrite stored credentials with the refresh response as-is, you erase the refresh token and force a re-consent. Fix: rotation-merge — when the response omits refresh_token, keep the previous one; replace it only when the provider actually rotates.

What is refresh_token_reused and how do I avoid it? It means the provider saw an already-rotated refresh token presented again — a pattern that also describes a stolen, replayed token — so it treats it as a security event and often revokes the whole refresh-token family. The usual innocent cause is concurrency. Avoid it by serializing refresh (single-flight in-process, a lock across processes), persisting the rotated token atomically before any other caller can read a stale copy, and re-reading after acquiring a cross-process lock.

Should I refresh the token proactively before it expires? Yes — refresh a little early using a small skew (30–60s). Treating the token as expired slightly ahead avoids the cliff where many requests discover expiry at the same instant, and absorbs clock skew. Proactive refresh reduces but doesn't remove the need for single-flight: you still want exactly one refresh when the skew window opens.

Will a mutex or async-lock library work instead of a shared Promise? Yes, for the in-process case any of them work — a mutex/async-lock guarding the refresh-and-persist section achieves the same coalescing as the shared-Promise pattern; the Promise approach is just the lightest with zero dependencies. None of them help across processes; that still needs a file/OS/distributed lock.

Is single-flight enough on its own? Inside one process, single-flight plus rotation-merge is the complete fix and you don't need a lock file. Add the cross-process lock only if a credential is genuinely shared across processes, and a distributed lock or a token-broker service if it's shared across machines.

The Concurrent OAuth Refresh Guide — vendor-neutral, no trackers, no fabricated benchmarks. Free to read and cite. Maintained by the team behind refresh-guard, one option discussed above. Provider names are referenced for accurate technical comparison; this page is not affiliated with or endorsed by any of them.