feat: make gateway Postgres connection pool size configurable (hardcoded to 10)

Problem Statement

The gateway's Postgres pool is hardcoded to max_connections(10) (crates/openshell-server/src/persistence/postgres.rs), with no config or env path. It's a single pool shared by every DB-backed RPC, so 10 is a hard, client-side ceiling on gateway↔DB concurrency — independent of how large the database is.

In cluster mode with external Postgres (workload.kind=deployment + server.externalDbSecret), a fleet of sandbox supervisors polling GetSandboxConfig / GetInferenceBundle / provider-env easily exceeds 10 concurrent connections. Symptoms:

sqlx::pool::acquire slow-threshold warnings; multi-second acquire latency on top of every DB-backed RPC.
/readyz becomes latency-sensitive to pool pressure, since it also acquires from this pool (PostgresStore::ping()). (This surfaced as readiness flapping in our case, but the root cause there was the default readiness-probe timeout being too short to tolerate a slow acquire — operator-tunable — not the pool itself. Noted only because it shows readiness is coupled to pool saturation.)

Scaling the database up doesn't help: the pool never opens more than 10 connections, leaving the extra capacity idle.

Proposed Design

Make the pool options configurable via environment variables, each defaulting to today's value (no change for existing deployments). Every one is read at startup and drops straight into the Helm chart as a container env: value — no image rebuild or config-file edit — consistent with OPENSHELL_DB_URL:

OPENSHELL_DB_MAX_CONNECTIONS — the pool cap (default 10). The primary fix.
OPENSHELL_DB_ACQUIRE_SLOW_THRESHOLD_SECS — the sqlx slow-acquire warn threshold (currently the sqlx default of 2s), so operators can tune or quiet that log line independently of the pool size.
Optionally OPENSHELL_DB_MIN_CONNECTIONS and OPENSHELL_DB_ACQUIRE_TIMEOUT_SECS.

Thread these into the pool builder (Store::connect → PostgresStore::connect → PgPoolOptions). They may also be mirrored in gateway.toml [openshell.gateway.database] (as log_level is today), with the env vars taking precedence.

Operator note: total connections = pool_size × replica_count, which must stay under the server's max_connections.

Agent Investigation

Single shared pool: Store::connect (crates/openshell-server/src/lib.rs) builds one PostgresStore/PgPool at startup.
Hardcoded cap: PgPoolOptions::new().max_connections(10).connect(url) (persistence/postgres.rs) — connect takes only the URL, no pool knob.
Shared by all handlers: e.g. handle_get_sandbox_config (grpc/policy.rs) does several store reads per request, each from the same 10 slots.

Checklist

I've reviewed existing issues and the architecture docs
This is a design proposal, not a "please build this" request

📋 triage-agent

Triage Assessment

Classification: validated-feature

Summary

Technically valid and feasible. Confirmed directly in the codebase: the gateway Postgres pool is hardcoded and the proposed env/TOML plumbing follows an existing precedent. High-confidence assessment.

Investigation

Hardcoded cap confirmed: crates/openshell-server/src/persistence/postgres.rs:30-31 — PgPoolOptions::new().max_connections(10). PostgresStore::connect(url) (line 29) takes only the URL, so no pool knob is threaded through.

Single shared pool confirmed: built once at startup via Store::connect; every DB-backed RPC draws from the same 10 slots.

Precedent exists: OPENSHELL_DB_URL is already wired as both a clap arg (cli.rs:94) and a config-file field with an env: binding (config_file.rs:302). New pool options (OPENSHELL_DB_MAX_CONNECTIONS, etc.) can follow the same env-first, TOML-mirrored pattern.

sqlx support: PgPoolOptions exposes max_connections/min_connections/acquire_timeout/acquire_slow_threshold — no new deps.

Duplicates: none found (searched open/closed for postgres/pool/max_connections).

Scope note: touches openshell-server (config + pool builder), the Helm chart env: surface (deploy/), and gateway config docs (docs/reference/gateway-config.mdx if TOML keys are added). Defaults preserve today's behavior (10), so existing deployments are unaffected.

Impact Signals

Affected users/scope: cluster-mode operators with external Postgres and a fleet of supervisors polling DB-backed RPCs; hard client-side ceiling of 10 conns regardless of DB size.

Regression: no — longstanding hardcoded default.

Workaround: unavailable — no config/env path exists; scaling the DB does not help.

Evidence quality: high — exact code paths cited by reporter verified; behavior is unambiguous.

Human Decision Required

Decide whether OpenShell should address this issue. If yes, replace state:validated with state:accepted, associate it with a roadmap item, and decide whether the work remains human-owned. To queue investigation or planning for an unattended agent, also apply agent:plan-requested. You can instead directly ask an agent to use create-spike or build-from-issue on this issue. If no, close it as not planned and record the rationale. Roadmap association is independent sequencing metadata.

I'd like to take this one

@letv1nnn

thanks @letv1nnn I made an MR that is auto closed as i am not yet vouched.

its based off 0.0.85 but is what worked for us running the gw with around 1000 sandboxes, its a hot fix patch on my end for now

just added in case you find it a useful reference

if you would like to vouch me i do have a request open and have opened 3 issues into this repo and would really appreciate being able to contribute in a greater capacity #2610

@bjw123

Thanks for the patch and context @bjw123.
I can't vouch you myself, since I'm a contributor here, not a maintainer, and the vouch gate only accepts /vouch from maintainers/collaborators on a Vouch Request discussion.

no worries been waiting for a vouch for a while, guess i should reach out to a maintainer :)

feat(server): make the database pool ceiling configurable

feat(server): make the Postgres pool ceiling configurable

feat(server): make the database pool ceiling configurable

Navigation Menu

Repository navigation

feat: make gateway Postgres connection pool size configurable (hardcoded to 10) #2561

Problem Statement

Proposed Design

Agent Investigation

Checklist

Triage Assessment

Summary

Investigation

Impact Signals

Human Decision Required

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

feat: make gateway Postgres connection pool size configurable (hardcoded to 10) #2561

Description

Problem Statement

Proposed Design

Agent Investigation

Checklist

Activity

letv1nnn commented on Aug 10, 2026

Triage Assessment

Summary

Investigation

Impact Signals

Human Decision Required

letv1nnn commented on Aug 10, 2026

bjw123 commented on Aug 11, 2026

letv1nnn commented on Aug 11, 2026

bjw123 commented on Aug 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions