[Bug]: Authorization policy lookup uses a background-task timeout profile on the synchronous request path

[behzad-taghipour-hs]

Description

A per-request authorization check reads role-based access policies from a database lookup as part of every query's processing. That lookup was configured with a timeout/retry profile meant for background or asynchronous work — a long timeout with multiple retries — even though it sits directly in the synchronous path every request has to pass through. Under any slowness or transient issue on that lookup, a single request could consume a large chunk of, or exceed, the entire request's time budget on this one step alone, well before ever reaching the actual query.

Environment

• Commit ID: 42b5c6b (v0.2.2)
• Environment: prod

Step to reproduce

• Introduce latency or transient unavailability on the database backing the authorization policy lookup
• Run any query that goes through the standard authorization check
• Observe the request's time budget consumed disproportionately by the policy lookup step, independent of the actual query's own performance

A root cause

The client used for this lookup was constructed with a general-purpose, background-task timeout/retry configuration (long timeout, multiple retries) instead of a fail-fast configuration meant for steps directly on the customer-facing request path. The codebase already has a dedicated fail-fast configuration for exactly this kind of synchronous, latency-sensitive call; this lookup was never switched to use it.

Context

The system distinguishes background/ingestion-style database calls (tolerant of slower dependencies) from customer-facing, synchronous calls (expected to fail fast so a hung dependency can't tie up request capacity). This authorization lookup sits on the synchronous side but was wired with the background-side profile.

Risks & Known Unknowns

• This was not the cause of the specific incident under investigation, but is a latent risk that could independently exhaust a request's time budget under database slowness.
• Any other synchronous-path lookup using the same general-purpose client construction should be audited for the same mismatch.

[noah-paige]

Fix status

Pending Mirror

Root-cause fix: DynamoDBDAO now accepts an optional botocore Config. The Cedar policy loader (_DdbPolicyLoader in cedar_authorizer.py) — the lookup this issue describes — now passes the repo's existing fail-fast profile (sync_boto_config(): 8s read timeout, 2 max attempts) instead of inheriting the DAO's background profile (async_boto_config(): 30s, 3 attempts). Backward-compatible: the new config param defaults to None, so every other DynamoDBDAO caller keeps its existing behavior.

Audit requested by this issue, done: per the "Risks & Known Unknowns" note above, I audited the other ~38 DynamoDBDAO call sites repo-wide for the same mismatch and found three more on the same class of synchronous, authorization-critical request path that hadn't been switched over:

• role_resolver.resolve_profile() — the serve invoke() entrypoint's role/grant resolution.
• policy_loader.load_policies_for_roles() — shared by the control-plane authorizer Lambda and the MCP server's grant resolver.
• The control-plane API Gateway custom authorizer Lambda itself (authorization/handler.py) — 4 call sites (_get_cache_invalidation_dao, _get_namespaces_dao, and the per-request rrm_dao/roles_dao). This one is the highest-impact of the three: it's invoked on every gated request (the authorizer result cache is disabled), so a DynamoDB brownout here could stall authorization for up to ~90s (30s × 3 attempts) with no fallback path, versus the fixed profile's ~16s worst case (8s × 2).

All three now pass sync_boto_config() as well, with unit tests asserting the fail-fast config at each site. 287 tests pass across the four affected packages; lint/format clean.