From 8143cc0537d4c674b178d6743e1688a84a3edc36 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 30 Oct 2026 20:10:10 +0900 Subject: [PATCH 01/42] build(deps): bump rustls from 0.23.45 to 0.23.46 in /desktop/src-tauri (#7942) * release: v2.33.0-preview.20260825 * release: v2.34.0-preview.20260827 * release: v2.36.0-preview.20260829 * fix(release): pass the bump job's permissions through the reusable-workflow call (#3262) Both v2.40.0 release dispatches (33615174183 preview, 33615177849 main) died at startup_failure: a workflow_call cannot grant its callee more than the calling job holds, and dev-version-bump.yml's job declares contents+pull- requests write. #3129 wired the call but never dispatched a release, so this is its first live run. The caller job now declares exactly the callee's two permissions; no other job in release.yml gains anything. Co-authored-by: jun (cherry picked from commit 7ce0ba51834740d7b4d5ec4793f6572d84624409) * release: set preview channel version 2.48.0-preview.20260908 * release: set main channel version 2.48.0 * chore(release): promote 2.55.0-preview.20260914 to preview Promotes the dev product snapshot 62f02223a0 to the preview train. The 2.55.0 line carries the #4546 cost-guard work: one send budget per logical request with a shared final-recovery reserve, zero-is-zero refusals with a typed error rather than a synthetic 502, compact and the Kiro inner retries admitted against that budget, a finite send ceiling per root workflow with an interactive reserve a fan-out cannot take, and a healthy detour promoted on transient-hold expiry instead of released cold. The previous preview tip 2.54.0-preview.20260914 is already tagged and published and is outranked by v2.54.0, so it could not be re-released; this is a new candidate rather than a re-cut. * chore(release): promote the verified 2.55.0 product tree to main Same product tree as preview 7bdd1b29b5 / 2.55.0-preview.20260914, which published successfully with its registry smoke green. Only package.json version differs. * ci(release): expose Linux bundler diagnostics for stable artifacts * fix(release): prepare stable platform bundles * fix(desktop): carry native sidecar packaging repair to main (#5532) * chore(ci): refresh main release verification (#5534) * fix(release): carry lipo argument fix to main (#5537) * fix(release): carry checksum and signature repair to main (#5546) * fix(release): carry Windows checksum record support to main (#5552) * fix(release): publish the GitHub release only after its verified assets attach (#5555) (#5558) GitHub freezes a release when it is published, so the attach step's upload came back HTTP 422 "Cannot upload assets to an immutable release". Every release from v2.55.0 to v2.60.0 therefore shipped with zero assets and the desktop updater had nothing to download. Create the release as a draft and flip it to published in attach-release, after the verified bundle is uploaded. Release notes still come from the validated notes file written at creation. * release: prepare 2.63.0 version metadata (#5612) * release: prepare 2.69.0 version metadata * fix(release): sign the packaged macOS keyring addons before notarization (#6271) * fix(release): sign the packaged macOS keyring addons before notarization Notarization rejected the 2.73.0 preview app: Resources/keyring/*.node, bundled since #6161, were unsigned or ad-hoc and had no secure timestamp, and Tauri does not sign files under Resources. Sign each darwin addon in place with the Developer ID identity, hardened runtime and timestamp after the certificate import and before tauri build, verify the result, and fail a real release that lacks signing material. * fix(release): match keyring signature fields without a pipe (cherry picked from commit 11782eedc3758d90c037244ab1cf36f17a286ccc) * chore(release): 2.74.0 * fix(deps): bump proxy-addr, fast-uri, ip-address and source-map-js for new advisories release.yml's dependency audit refused 2.79.0 on a critical advisory published after the candidate was cut: proxy-addr <2.0.8 is vulnerable to IP spoofing via IPv4-mapped IPv6 trust subnets (GHSA-jqcg-44mw-7w3h), reached through @modelcontextprotocol/sdk > express. gui/ reports source-map-js <1.2.2 (GHSA-68fv-2mgg-jv7q, vite > postcss). bun audit fix, lockfiles only, every bump inside its declared range: proxy-addr 2.0.7 -> 2.0.8, fast-uri 3.1.7 -> 3.1.8, ip-address 10.4.0 -> 10.7.1, and in gui/ source-map-js 1.2.1 -> 1.2.2. Both audits now report no vulnerabilities at --audit-level=high. * fix(deps): bump @modelcontextprotocol/sdk to 1.31.0 for GHSA-6qxp-vccf-f47h * fix(deps): clear high-severity audit findings blocking the 2.90.0 release (#7481) Raise @modelcontextprotocol/sdk to ^1.32.1 (GHSA-22jm-h49p-29qw, GHSA-46jr-4wx8-v453) and lock gui vite at 8.2.4 (GHSA-587h-7wmj-h5pm). release.yml runs bun run audit:high before publishing, and both advisories made it fail. * fix(deps): pin zod 4.6.6 and body-parser 2.4.0 for 2.92.0 (#7643) GHSA-mrhx-c7xc-m3w5 and GHSA-27r3-jxmh-p828 blocked the 2.92.0 publish. Leave the package version at 2.92.0 so Release can republish that version. Co-authored-by: Cursor Agent * fix(deps): bump gui happy-dom to 21.0.4 for GHSA-834v-8h66-3h79 (2.92.0) (#7645) Co-authored-by: lidge-jun <243035832+lidge-jun@users.noreply.github.com> * build(deps): bump rustls from 0.23.45 to 0.23.46 in /desktop/src-tauri Bumps [rustls](https://github.com/rustls/rustls) from 0.23.45 to 0.23.46. - [Release notes](https://github.com/rustls/rustls/releases) - [Changelog](https://github.com/rustls/rustls/blob/main/CHANGELOG.md) - [Commits](https://github.com/rustls/rustls/compare/v/0.23.45...v/0.23.46) --- updated-dependencies: - dependency-name: rustls dependency-version: 0.23.46 dependency-type: indirect ... Signed-off-by: dependabot[bot] --------- Signed-off-by: dependabot[bot] Co-authored-by: JUN Co-authored-by: jun Co-authored-by: jun Co-authored-by: lidge-jun <243035832+lidge-jun@users.noreply.github.com> Co-authored-by: t Co-authored-by: JUN Co-authored-by: Cursor Agent Co-authored-by: lidge-jun Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Epinephrine --- desktop/src-tauri/Cargo.lock | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/desktop/src-tauri/Cargo.lock b/desktop/src-tauri/Cargo.lock index ff520a88d25..92ee4e1fbd9 100644 --- a/desktop/src-tauri/Cargo.lock +++ b/desktop/src-tauri/Cargo.lock @@ -3323,9 +3323,9 @@ dependencies = [ [[package]] name = "rustls" -version = "0.23.45" +version = "0.23.46" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0d41d731c7d2f962d1ccc364cec258de3c0e93b38c2fb3ba97ac74513048d634" +checksum = "48e13bd8c0e9365c43cfa5c9e8f9ad49d3c8444926c9aac819e0e4dc503c8fdf" dependencies = [ "once_cell", "ring", From 88de3762fdabb7ef6038c2ea0313fb34b0f34c36 Mon Sep 17 00:00:00 2001 From: Epinephrine Date: Fri, 30 Oct 2026 11:16:11 +0000 Subject: [PATCH 02/42] fix(ci): distinguish install failures from owned deadlines (#7977) * fix(ci): distinguish install failures from owned deadlines * fix(ci): probe required timeout foreground support --- scripts/ci/bun-install-frozen.sh | 192 ++++++++++-- .../ci-bun-install-frozen.test.ts | 294 ++++++++++++++---- 2 files changed, 394 insertions(+), 92 deletions(-) diff --git a/scripts/ci/bun-install-frozen.sh b/scripts/ci/bun-install-frozen.sh index 0cc559d30da..e6d7bc3aec0 100755 --- a/scripts/ci/bun-install-frozen.sh +++ b/scripts/ci/bun-install-frozen.sh @@ -1,15 +1,71 @@ #!/usr/bin/env bash -# Frozen-lockfile install with a per-attempt deadline and one total budget. -# -# A registry or runner stall can leave `bun install` printing only its banner until the job's -# timeout cancels the run, which reads as a failed check unrelated to the change under test. -# Each attempt therefore runs under a process deadline, and only an attempt that the deadline -# killed is retried. Any other failure, such as a lockfile that no longer matches, fails at once. -# All directories share one total budget, so retries end well inside the shortest job timeout -# that calls this (5 minutes) and leave time for setup, the GUI build and the tests. -# -# Usage: bash scripts/ci/bun-install-frozen.sh [dir ...] (default: the current directory) -set -euo pipefail +# Frozen installs: one witnessed deadline, a shared budget, and no PID-directed cleanup. +# GNU timeout/gtimeout bounds a fixed reader, never the install's own exit status. +# Bootstrap FIFO names disappear before install code runs; this is not a sandbox against +# arbitrary same-user process inspection. Cleanup covers descendants that remain in the group. +set -eu + +own_group() { + local stat state ppid group rest + if [[ -r /proc/$$/stat ]]; then + IFS= read -r stat < /proc/$$/stat || return 1 + read -r state ppid group rest <<< "${stat##*) }" + else + read -r group <<< "$(ps -o pgid= -p "$$")" || return 1 + fi + [[ "$group" == "$$" ]] +} + +keeper() { + local boot="$1" timer="$2" grace="$3" dir="$4" deadline token completion rc + shift 4 + set +m + own_group || exit 70 # No signal 0 until this process is its group leader. + trap ':' TERM + # O_RDWR bootstrap avoids a blocked open if the other endpoint dies during setup. + exec 6<>"$boot/outcome" 8>"$boot/outcome" + exec 6>&- + exec 6<>"$boot/lifeline" 9<"$boot/lifeline" + exec 6>&- + printf 'ready %s\n' "$$" >&8 + IFS=' ' read -r -t 5 token deadline <&9 || exit 70 + [[ "$token" == go && "$deadline" =~ ^[1-9][0-9]*$ ]] || exit 70 + + # Only the parent owns a lifeline writer. Its death/cancellation is EOF, never a PID kill. + ( exec 8>&-; trap '' TERM; cat <&9 >/dev/null; kill -TERM 0; sleep 0.2; kill -KILL 0 ) >/dev/null 2>&1 & + exec 9<&- # Only the EOF helper needs the read endpoint now. + if ! cd "$dir"; then printf 'fail 66\n' >&8; kill -KILL 0; fi + exec 7>&1 + report_install() { + local status=0 + exec 8>&- + # The command gets neither a control descriptor nor the result pipe's writer. + "$@" >&7 7>&- || status=$? + printf '%s\n' "$status" + } + rc=0 + # The fixed reader exits only 0/70; install 124/137 are pipe data, not timer statuses. + # Process substitution does not wait for a stalled reporter when the reader times out. + completion=$("$timer" --foreground "${deadline}s" bash -c ' + IFS= read -r status || exit 70 + [[ "$status" =~ ^(0|[1-9][0-9]{0,2})$ && $status -le 255 ]] || exit 70 + printf "%s" "$status" + ' < <(report_install "$@") 7>&- 8>&-) || rc=$? + if [[ $rc -eq 0 ]]; then + printf 'result %s\n' "$completion" >&8 + elif [[ $rc -eq 124 ]]; then + printf 'deadline 124\n' >&8 + kill -TERM 0 + sleep "$grace" + else + printf 'fail 70\n' >&8 + fi + # This live process pins the group until its last action, including ordinary success. + kill -KILL 0 + exit 70 +} + +if [[ "${1:-}" == --ocx-install-keeper ]]; then shift; keeper "$@"; exit 70; fi readonly ATTEMPT_SECONDS="${BUN_INSTALL_ATTEMPT_SECONDS:-45}" readonly TOTAL_SECONDS="${BUN_INSTALL_TOTAL_SECONDS:-150}" @@ -17,42 +73,112 @@ readonly MIN_ATTEMPT_SECONDS="${BUN_INSTALL_MIN_ATTEMPT_SECONDS:-5}" readonly KILL_GRACE_SECONDS="${BUN_INSTALL_KILL_GRACE_SECONDS:-5}" readonly MAX_ATTEMPTS="${BUN_INSTALL_MAX_ATTEMPTS:-3}" readonly BUN_INSTALL_BIN="${BUN_INSTALL_BIN:-bun}" +readonly SCRIPT_PATH="${BASH_SOURCE[0]}" +for value in "$ATTEMPT_SECONDS" "$TOTAL_SECONDS" "$MIN_ATTEMPT_SECONDS" "$KILL_GRACE_SECONDS" "$MAX_ATTEMPTS"; do + [[ "$value" =~ ^[1-9][0-9]*$ && ${#value} -le 6 ]] || { echo '::error::Invalid install deadline configuration.'; exit 70; } +done +TIMEOUT_BIN="" +for candidate in timeout gtimeout; do + if command -v "$candidate" >/dev/null 2>&1 && "$candidate" --foreground 1s true >/dev/null 2>&1; then + TIMEOUT_BIN=$(command -v "$candidate"); break + fi +done +[[ -n "$TIMEOUT_BIN" ]] || { echo '::error::GNU timeout is required to bound bun install.' >&2; exit 69; } -if ! command -v timeout >/dev/null 2>&1 || ! timeout --signal=TERM --kill-after=1s 1s true >/dev/null 2>&1; then - echo "::error::GNU timeout is required to bound bun install." >&2 - exit 69 -fi +ATTEMPT_VERDICT=fail +ATTEMPT_STATUS=70 +run_attempt() { + local dir="$1" deadline="$2" boot keeper_pid keeper_status=70 tag="" value="" extra="" + local cancel_status=0 interrupted=no drained=no remaining started=no monitor=no + shift 2 + boot=$(mktemp -d "${TMPDIR:-/tmp}/ocx-bun-install.XXXXXX") || return 70 + mkfifo "$boot/outcome" "$boot/lifeline" || return 70 + exec 8<>"$boot/outcome" 9<>"$boot/lifeline" + trap 'cancel_status=143; interrupted=yes; exec 9>&-' TERM + trap 'cancel_status=130; interrupted=yes; exec 9>&-' INT + trap 'cancel_status=129; interrupted=yes; exec 9>&-' HUP + [[ $- == *m* ]] && monitor=yes + set -m + bash "$SCRIPT_PATH" --ocx-install-keeper "$boot" "$TIMEOUT_BIN" "$KILL_GRACE_SECONDS" "$dir" "$@" 8>&- 9>&- & + keeper_pid=$! + [[ "$monitor" == yes ]] || set +m + ATTEMPT_VERDICT=fail; ATTEMPT_STATUS=70 + if IFS=' ' read -r -t 5 tag value extra <&8 && [[ "$tag" == ready && "$value" == "$keeper_pid" && -z "$extra" ]]; then + # After the handshake, unlink every IPC name before permitting any install code. + if rm "$boot/outcome" "$boot/lifeline" 8>&- 9>&- && rmdir "$boot" 8>&- 9>&-; then + remaining=$(( TOTAL_SECONDS - SECONDS - KILL_GRACE_SECONDS - 2 )) + (( remaining < deadline )) && deadline=$remaining + if (( deadline >= MIN_ATTEMPT_SECONDS && cancel_status == 0 )); then + if printf 'go %s\n' "$deadline" >&9; then + started=yes + tag=""; value=""; extra="" + if IFS=' ' read -r -t "$(( deadline + KILL_GRACE_SECONDS ))" tag value extra <&8 && [[ -z "$extra" ]]; then + case "$tag:$value" in + result:*) + if [[ "$value" =~ ^(0|[1-9][0-9]{0,2})$ && $value -le 255 ]]; then + ATTEMPT_STATUS=$value + [[ $value -eq 0 ]] && ATTEMPT_VERDICT=ok + fi ;; + deadline:124) ATTEMPT_VERDICT=deadline; ATTEMPT_STATUS=124 ;; + fail:66) ATTEMPT_STATUS=66 ;; + esac + fi + fi + else + ATTEMPT_VERDICT=budget; ATTEMPT_STATUS=124 + fi + fi + fi + # A valid outcome keeps the lifeline open through the keeper's own complete grace cycle. + if [[ "$ATTEMPT_VERDICT" == fail && $ATTEMPT_STATUS -eq 70 ]] || [[ "$started" == no ]]; then exec 9>&-; fi + while :; do + interrupted=no + keeper_status=0; wait "$keeper_pid" 2>/dev/null || keeper_status=$? + [[ "$interrupted" == yes ]] || break + done + # EOF is not a drain proof. Reuse can cause a conservative failure, never a foreign kill. + for value in 1 2 3 4 5 6 7 8 9 10; do + if ! kill -0 -- "-$keeper_pid" 2>/dev/null; then drained=yes; break; fi + sleep 0.1 8>&- 9>&- + done + exec 9>&- 8>&- + trap - TERM INT HUP + if [[ -d "$boot" ]]; then + for value in outcome lifeline; do [[ ! -e "$boot/$value" ]] || rm "$boot/$value"; done + rmdir "$boot" + fi + if (( cancel_status > 0 )); then + ATTEMPT_VERDICT=cancel; ATTEMPT_STATUS=$cancel_status + elif [[ "$drained" != yes ]]; then + ATTEMPT_VERDICT=drain_fail; ATTEMPT_STATUS=70 + elif [[ "$started" == yes && $keeper_status -ne 137 ]]; then + ATTEMPT_VERDICT=fail; ATTEMPT_STATUS=70 + fi +} install_dir() { - local dir="$1" attempt status remaining deadline - # Never empty: macOS ships bash 3.2, where expanding an empty array under `set -u` aborts - # with "unbound variable". + local dir="$1" attempt deadline remaining + # Keep this array nonempty for Bash 3.2 with nounset. local -a install_cmd=("$BUN_INSTALL_BIN" install --frozen-lockfile) for ((attempt = 1; attempt <= MAX_ATTEMPTS; attempt++)); do - # The kill grace counts against the budget too, so the worst case stays at TOTAL_SECONDS. - remaining=$(( TOTAL_SECONDS - SECONDS - KILL_GRACE_SECONDS )) + remaining=$(( TOTAL_SECONDS - SECONDS - KILL_GRACE_SECONDS - 2 )) if (( remaining < MIN_ATTEMPT_SECONDS )); then echo "::error::bun install --frozen-lockfile in ${dir} has no time left in its ${TOTAL_SECONDS}s budget." return 124 fi deadline=$(( remaining < ATTEMPT_SECONDS ? remaining : ATTEMPT_SECONDS )) - status=0 - (cd "$dir" && timeout --signal=TERM --kill-after="${KILL_GRACE_SECONDS}s" "${deadline}s" \ - "${install_cmd[@]}") || status=$? - if [[ $status -eq 0 ]]; then return 0; fi - # 124: the deadline fired and the install exited on TERM; 137: it needed the KILL. - if [[ $status -ne 124 && $status -ne 137 ]]; then - echo "::error::bun install --frozen-lockfile failed in ${dir} (exit ${status}); not retrying." - return "$status" - fi - echo "::warning::bun install --frozen-lockfile in ${dir} exceeded ${deadline}s (attempt ${attempt}/${MAX_ATTEMPTS})." - # A killed install may have extracted bun but not finished its trusted postinstall. - # Plain install can then report success while leaving bin/bun.exe a placeholder. + run_attempt "$dir" "$deadline" "${install_cmd[@]}" + case "$ATTEMPT_VERDICT" in + ok) return 0 ;; + cancel) return "$ATTEMPT_STATUS" ;; + budget) echo "::error::bun install --frozen-lockfile in ${dir} has no time left in its ${TOTAL_SECONDS}s budget."; return 124 ;; + deadline) echo "::warning::bun install --frozen-lockfile in ${dir} exceeded ${deadline}s (attempt ${attempt}/${MAX_ATTEMPTS})." ;; + *) echo "::error::bun install --frozen-lockfile failed in ${dir} (exit ${ATTEMPT_STATUS}, ${ATTEMPT_VERDICT}); not retrying."; return "$ATTEMPT_STATUS" ;; + esac install_cmd=("$BUN_INSTALL_BIN" install --frozen-lockfile --force) done echo "::error::bun install --frozen-lockfile in ${dir} exceeded its deadline ${MAX_ATTEMPTS} times." return 124 } - if [[ $# -eq 0 ]]; then set -- .; fi for dir in "$@"; do install_dir "$dir"; done diff --git a/tests/ci-workflows/ci-bun-install-frozen.test.ts b/tests/ci-workflows/ci-bun-install-frozen.test.ts index bec3dd7292f..cb969eaf5ef 100644 --- a/tests/ci-workflows/ci-bun-install-frozen.test.ts +++ b/tests/ci-workflows/ci-bun-install-frozen.test.ts @@ -1,6 +1,6 @@ import { afterEach, describe, expect, test } from "bun:test"; -import { spawnSync } from "node:child_process"; -import { chmodSync, existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { spawn, spawnSync } from "node:child_process"; +import { chmodSync, existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { repoPath } from "../helpers/repo-root"; @@ -39,53 +39,257 @@ describe("CI frozen installs are bounded", () => { }); }); -// The script targets the Linux CI runners and needs GNU timeout. +// These process-group assertions need POSIX; a Windows skip is not Linux coverage. describe.skipIf(process.platform === "win32")("bun-install-frozen.sh retry policy", () => { const roots: string[] = []; - afterEach(() => { for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); }); + const groups = (root: string) => existsSync(join(root, "groups")) + ? [...new Set(readFileSync(join(root, "groups"), "utf8").trim().split(/\s+/).filter((s) => /^[1-9]\d*$/.test(s)))] : []; + const drained = (root: string) => spawnSync("bash", ["-c", + 'for group; do if kill -0 -- "-$group" 2>/dev/null; then exit 1; fi; done', + "_", ...groups(root)], { timeout: 2_000 }).status === 0; + afterEach(() => { + for (const root of roots.splice(0)) { + // Never delete a fixture against a live attempt, or infer drain merely from pipe EOF. + if (!drained(root)) throw new Error("Attempt group remains; fixture retained: " + root); + rmSync(root, { recursive: true }); + } + }); - /** A fake bun whose Nth invocation behaves as listed: "hang", "ok" or an exit code. */ - function run(plan: string[], options: { dirs?: number; env?: Record } = {}) { - const root = mkdtempSync(join(tmpdir(), "ocx-bun-install-")); + function fixture(plan: string[], env: Record = {}) { + const root = mkdtempSync(join(tmpdir(), "ocx-bun-install-case-")); roots.push(root); - const count = join(root, "count"), fake = join(root, "fake-bun"); + mkdirSync(join(root, "tmp")); writeFileSync(join(root, "plan"), plan.join("\n") + "\n"); + const fake = join(root, "fake-bun"); writeFileSync(fake, [ "#!/usr/bin/env bash", - `n=$(( $(cat "${count}" 2>/dev/null || echo 0) + 1 )); echo "$n" > "${count}"`, - `step=$(sed -n "${"$"}{n}p" "${join(root, "plan")}")`, + 'if [[ -r /proc/$$/stat ]]; then', + ' IFS= read -r stat < /proc/$$/stat; rest="${stat##*) }"; read -r state parent group rest <<< "$rest"', + 'else read -r group <<< "$(ps -o pgid= -p "$$")"; fi', + 'printf "%s\\n" "$group" >> "$TEST_ROOT/groups"', + 'n=$(( $(cat "$TEST_ROOT/count" 2>/dev/null || echo 0) + 1 )); echo "$n" > "$TEST_ROOT/count"', + 'step=$(sed -n "$' + '{n}p" "$TEST_ROOT/plan")', 'echo "fake bun $* in $PWD"', - 'case "$step" in hang) exec sleep 30 ;; ok) exit 0 ;; *) exit "$step" ;; esac', + 'case "$step" in', + " hang) exec sleep 30 ;;", + " hang-ignore) trap '' TERM; exec sleep 30 ;;", + " term-marker) trap 'echo TERM > \"$TEST_ROOT/termed\"; exit 0' TERM; sleep 30 ;;", + " sigkill) kill -KILL $$ ;;", + " spawn-ignore) ( trap '' TERM; exec sleep 30 ) & exec sleep 30 ;;", + " spawn-leak) ( exec sleep 30 >/dev/null 2>&1 ) & exit 0 ;;", + ' forge-fd) for fd in 6 7 8 9; do if ( printf "result 0\\n" >&"$fd" ) 2>/dev/null; then echo "$fd" > "$TEST_ROOT/leaked"; exit 99; fi; done; exit 42 ;;', + ' metadata) mkdir "$TMPDIR/ocx-bun-install.forged"; mkdir "$TMPDIR/ocx-bun-install.forged/decision"; echo 0 > "$TMPDIR/ocx-bun-install.forged/result"; echo "deadline 0" > "$TMPDIR/ocx-bun-install.forged/witness"; exit 42 ;;', + ' kill-keeper) kill -KILL "$group"; exec sleep 30 ;;', + " ok) exit 0 ;;", + ' *) exit "$step" ;;', + "esac", ].join("\n")); chmodSync(fake, 0o755); - const dirs = Array.from({ length: options.dirs ?? 1 }, () => root); + return { + root, count: join(root, "count"), + env: { ...process.env, TEST_ROOT: root, TMPDIR: join(root, "tmp"), + BUN_INSTALL_BIN: fake, BUN_INSTALL_ATTEMPT_SECONDS: "1", + BUN_INSTALL_KILL_GRACE_SECONDS: "1", BUN_INSTALL_MIN_ATTEMPT_SECONDS: "1", ...env }, + }; + } + + function run(plan: string[], options: { dirs?: number; env?: Record } = {}) { + const f = fixture(plan, options.env); + const dirs = Array.from({ length: options.dirs ?? 1 }, (_, n) => { + const dir = join(f.root, "dir-" + n); mkdirSync(dir); return dir; + }); const started = Date.now(); - const result = spawnSync("bash", [script, ...dirs], { - encoding: "utf8", timeout: 30_000, - env: { ...process.env, BUN_INSTALL_BIN: fake, BUN_INSTALL_ATTEMPT_SECONDS: "1", BUN_INSTALL_KILL_GRACE_SECONDS: "1", - BUN_INSTALL_MIN_ATTEMPT_SECONDS: "1", ...options.env }, + const child = spawnSync("bash", [script, ...dirs], { + encoding: "utf8", timeout: 15_000, env: f.env, }); - return { status: result.status, out: result.stdout + result.stderr, elapsedMs: Date.now() - started, - calls: existsSync(count) ? Number(readFileSync(count, "utf8")) : 0 }; + expect(child.error, child.stdout + child.stderr).toBeUndefined(); + expect(drained(f.root), "owned group remains").toBe(true); + return { status: child.status, out: child.stdout + child.stderr, root: f.root, + elapsedMs: Date.now() - started, + calls: existsSync(f.count) ? Number(readFileSync(f.count, "utf8")) : 0 }; + } + + function timeoutCandidates(f: ReturnType, compatibleFallback: boolean) { + const bin = join(f.root, "timeout-bin"); + mkdirSync(bin); + for (const name of ["timeout", "gtimeout"]) { + writeFileSync(join(bin, name), [ + "#!/usr/bin/env bash", + 'if [[ "$1" == --signal=TERM ]]; then exit 0; fi', + 'if [[ "$1" == --foreground ]]; then', + ...(name === "gtimeout" && compatibleFallback + ? [' shift 2', ' exec "$@"'] + : [' exit 64']), + "fi", + "exit 64", + ].join("\n")); + chmodSync(join(bin, name), 0o755); + } + return { ...f.env, PATH: `${bin}:${f.env.PATH ?? ""}` }; } - test("an attempt killed by its deadline is retried and the next success passes", () => { - const result = run(["hang", "ok"]); - expect(result.status, result.out).toBe(0); - expect(result.calls).toBe(2); - expect(result.out).toContain("::warning::bun install --frozen-lockfile"); - expect(result.out).toContain("fake bun install --frozen-lockfile --force"); - expect(result.out.split("fake bun ")[1]).toStartWith("install --frozen-lockfile in "); + test("rejects candidates without foreground support before starting an install", () => { + const f = fixture(["ok"]); + const dir = join(f.root, "dir"); mkdirSync(dir); + const child = spawnSync("bash", [script, dir], { + encoding: "utf8", timeout: 10_000, env: timeoutCandidates(f, false), + }); + expect(child.error, child.stdout + child.stderr).toBeUndefined(); + expect(child.status, child.stdout + child.stderr).toBe(69); + expect(child.stdout + child.stderr).toContain("GNU timeout is required"); + expect(existsSync(f.count)).toBe(false); + expect(drained(f.root)).toBe(true); + }); + + test("falls back to gtimeout when timeout lacks foreground support", () => { + const f = fixture(["ok"]); + const dir = join(f.root, "dir"); mkdirSync(dir); + const child = spawnSync("bash", [script, dir], { + encoding: "utf8", timeout: 10_000, env: timeoutCandidates(f, true), + }); + expect(child.error, child.stdout + child.stderr).toBeUndefined(); + expect(child.status, child.stdout + child.stderr).toBe(0); + expect(existsSync(f.count)).toBe(true); + expect(readFileSync(f.count, "utf8")).toBe("1\n"); + expect(drained(f.root)).toBe(true); + }); + + test.each(["1", "124", "137", "sigkill"])("ordinary install failure %s never retries", (step) => { + const r = run([step, "ok"]); + expect(r.status, r.out).toBe(step === "sigkill" ? 137 : Number(step)); + expect(r.calls).toBe(1); + expect(r.out).toContain("not retrying"); + expect(r.out).not.toContain("::warning::"); }); - test("a genuine install failure after a timeout still fails without another retry", () => { - const result = run(["hang", "42", "ok"]); - expect(result.status, result.out).toBe(42); - expect(result.calls).toBe(2); - expect(result.out).toContain("::error::bun install --frozen-lockfile failed"); - expect(result.out).toContain("not retrying"); + test.each(["hang", "hang-ignore", "spawn-ignore"])("deadline retries %s after its full grace", (step) => { + const r = run([step, "ok"], { env: { BUN_INSTALL_KILL_GRACE_SECONDS: "2" } }); + expect(r.status, r.out).toBe(0); + expect(r.calls).toBe(2); + expect(r.elapsedMs).toBeGreaterThanOrEqual(2_800); + expect(r.elapsedMs).toBeLessThan(12_000); + expect(r.out).toContain("::warning::"); + expect(r.out).toContain("fake bun install --frozen-lockfile --force"); + expect(r.out.split("fake bun ")[1]).toStartWith("install --frozen-lockfile in "); + }, 15_000); + + test("TERM remains catchable and its full grace elapses before completion", () => { + const r = run(["term-marker"], { env: { + BUN_INSTALL_MAX_ATTEMPTS: "1", BUN_INSTALL_KILL_GRACE_SECONDS: "2", + } }); + expect(r.status, r.out).toBe(124); + expect(r.calls).toBe(1); + const marker = join(r.root, "termed"); + expect(existsSync(marker)).toBe(true); + expect(Date.now() - statSync(marker).mtimeMs).toBeGreaterThanOrEqual(1_800); + }, 10_000); + + test("a failure after a witnessed deadline stops further retries", () => { + const r = run(["hang", "42", "ok"]); + expect(r.status, r.out).toBe(42); + expect(r.calls).toBe(2); + expect(r.out).toContain("not retrying"); + }, 10_000); + + test("normal success drains descendants even when they closed output", () => { + const r = run(["spawn-leak"]); + expect(r.status, r.out).toBe(0); + expect(r.calls).toBe(1); + expect(r.elapsedMs).toBeLessThan(5_000); }); + test("the install inherits no result or lifeline descriptor", () => { + const r = run(["forge-fd", "ok"]); + expect(r.status, r.out).toBe(42); + expect(r.calls).toBe(1); + expect(existsSync(join(r.root, "leaked"))).toBe(false); + }); + + test("writable decision/result/witness files have no classification authority", () => { + const r = run(["metadata", "ok"]); + expect(r.status, r.out).toBe(42); + expect(r.calls).toBe(1); + expect(r.out).not.toContain("::warning::"); + expect(readFileSync(join(r.root, "tmp", "ocx-bun-install.forged", "result"), "utf8")).toBe("0\n"); + }); + + test("keeper death fails closed and leaves no attempt group", () => { + const r = run(["kill-keeper", "ok"]); + expect(r.status, r.out).toBe(70); + expect(r.calls).toBe(1); + expect(r.elapsedMs).toBeLessThan(8_000); + expect(r.out).not.toContain("::warning::"); + }, 10_000); + + test("three witnessed deadlines stop at max attempts", () => { + const r = run(["hang", "hang", "hang", "ok"]); + expect(r.status, r.out).toBe(124); + expect(r.calls).toBe(3); + expect(r.out).toContain("exceeded its deadline 3 times"); + }, 15_000); + + test("successful root and stalled GUI share the total budget, including cleanup", () => { + const r = run(["ok", "hang", "hang", "hang"], { dirs: 2, env: { + BUN_INSTALL_ATTEMPT_SECONDS: "3", BUN_INSTALL_TOTAL_SECONDS: "9", + } }); + expect(r.status, r.out).toBe(124); + expect(r.calls).toBeGreaterThanOrEqual(2); + expect(r.calls).toBeLessThan(4); + expect(r.elapsedMs).toBeLessThan(10_500); + expect(r.out).toContain("dir-1"); + expect(r.out).toContain("budget"); + }, 15_000); + + async function interrupt(signal: "SIGTERM" | "SIGINT" | "SIGHUP" | "SIGKILL", duringGrace = false) { + const f = fixture([duringGrace ? "term-marker" : "hang", "ok"], { + BUN_INSTALL_ATTEMPT_SECONDS: duringGrace ? "1" : "8", BUN_INSTALL_KILL_GRACE_SECONDS: "3", + }); + const child = spawn("bash", [script, f.root], { env: f.env }); + let out = ""; + child.stdout.on("data", (d) => { out += d; }); + child.stderr.on("data", (d) => { out += d; }); + let closed = false; + const completion = new Promise((resolve, reject) => { + child.once("error", reject); + child.once("close", (code, sig) => { + closed = true; + resolve(code ?? (sig === "SIGKILL" ? 137 : sig === "SIGTERM" ? 143 : -1)); + }); + }); + let timer: ReturnType | undefined; + try { + const readyBy = Date.now() + 5_000; + const ready = duringGrace ? join(f.root, "termed") : f.count; + while (!existsSync(ready) && !closed && Date.now() < readyBy) { + await new Promise((resolve) => setTimeout(resolve, 10)); + } + expect(existsSync(ready), out).toBe(true); + expect(closed, out).toBe(false); + const stopped = Date.now(); + child.kill(signal); + const status = await Promise.race([completion, new Promise((_, reject) => { + timer = setTimeout(() => reject(new Error("No parent exit/output EOF")), 5_000); + })]); + // SIGKILL cannot wait for its descendants, so observe the group separately. + while (!drained(f.root) && Date.now() - stopped < 4_000) { + await new Promise((resolve) => setTimeout(resolve, 20)); + } + expect(status, out).toBe({ SIGTERM: 143, SIGINT: 130, SIGHUP: 129, SIGKILL: 137 }[signal]); + expect(drained(f.root), out).toBe(true); + expect(Number(readFileSync(f.count, "utf8"))).toBe(1); + expect(out).not.toContain("::warning::"); + } finally { + if (timer) clearTimeout(timer); + if (!closed && child.exitCode === null && child.signalCode === null) child.kill("SIGTERM"); + if (!closed) await completion; + } + } + + test.each(["SIGTERM", "SIGINT", "SIGHUP", "SIGKILL"] as const)("parent %s stops its attempt without a retry", + (signal) => interrupt(signal), 15_000); + test("parent cancellation outranks an already recorded deadline during its grace", + () => interrupt("SIGTERM", true), 15_000); + test("retry completes a trusted postinstall interrupted after package extraction", () => { const root = mkdtempSync(join(tmpdir(), "ocx-bun-postinstall-")); roots.push(root); @@ -125,32 +329,4 @@ describe.skipIf(process.platform === "win32")("bun-install-frozen.sh retry polic expect(readFileSync(join(root, "bun.lock"), "utf8")).toBe(lockBefore); }, 30_000); - test("an ordinary install failure such as a lockfile mismatch is not retried", () => { - const result = run(["1", "ok"]); - expect(result.status, result.out).toBe(1); - expect(result.calls).toBe(1); - expect(result.out).toContain("not retrying"); - }); - - test("three deadline kills fail with an error annotation", () => { - const result = run(["hang", "hang", "hang", "ok"]); - expect(result.status, result.out).toBe(124); - expect(result.calls).toBe(3); - expect(result.out).toContain("::error::bun install --frozen-lockfile in"); - }); - - test("every attempt runs the frozen-lockfile install", () => { - const result = run(["ok"]); - expect(result.status, result.out).toBe(0); - expect(result.out).toContain("fake bun install --frozen-lockfile"); - }); - - test("root and GUI stalls share one total budget instead of three attempts each", () => { - // Per-directory retries alone would allow 6 attempts of 2 s plus grace; the 4 s total ends after 2. - const result = run(Array(8).fill("hang"), { dirs: 2, env: { BUN_INSTALL_ATTEMPT_SECONDS: "2", BUN_INSTALL_TOTAL_SECONDS: "4" } }); - expect(result.status, result.out).toBe(124); - expect(result.calls).toBe(2); - expect(result.elapsedMs).toBeLessThan(7_000); - expect(result.out).toContain("budget"); - }); }); From 2c4e580e23f902af929f93f3294e13b687467308 Mon Sep 17 00:00:00 2001 From: Reed Date: Fri, 30 Oct 2026 12:16:48 +0100 Subject: [PATCH 03/42] test(xai): cover encrypted Responses tool continuation (#7975) * test(xai): cover encrypted Responses tool continuation * test(xai): assert immutability of the adapter request body --- .../xai/xai-responses-adjacency.test.ts | 58 +++++++++++++++++-- 1 file changed, 54 insertions(+), 4 deletions(-) diff --git a/tests/providers/xai/xai-responses-adjacency.test.ts b/tests/providers/xai/xai-responses-adjacency.test.ts index 6c8d03546b7..5539e241356 100644 --- a/tests/providers/xai/xai-responses-adjacency.test.ts +++ b/tests/providers/xai/xai-responses-adjacency.test.ts @@ -22,22 +22,72 @@ function xaiOauthResponses(overrides: Partial = {}): OcxProvi } function buildBody(provider: OcxProviderConfig, rawBody: Record): Record { + const requestBody = typeof rawBody.model === "string" ? rawBody : { model: MODEL, ...rawBody }; + const before = structuredClone(requestBody); const built = createResponsesPassthroughAdapter(provider).buildRequest({ - modelId: MODEL, + modelId: typeof requestBody.model === "string" ? requestBody.model : MODEL, context: { messages: [] }, stream: true, options: {}, previousResponseId: typeof rawBody.previous_response_id === "string" ? rawBody.previous_response_id : undefined, - _rawBody: { model: MODEL, ...rawBody }, + _rawBody: requestBody, } as Parameters["buildRequest"]>[0], { headers: new Headers(), }); + if (built instanceof Promise) throw new Error("Expected a synchronous Responses request builder"); + expect(requestBody).toEqual(before); return JSON.parse(String(built.body)) as Record; } describe("xAI Responses include compatibility", () => { + for (const baseUrl of ["https://api.x.ai/v1", "https://us.api.x.ai/v1", "https://cli-chat-proxy.grok.com/v1"]) { + test(`preserves Grok 4.7 encrypted tool continuation at ${baseUrl}`, () => { + const raw = { + model: "grok-4.7", + store: false, + reasoning: { effort: "xhigh" }, + include: ["output.encrypted_content", "reasoning.encrypted_content", "web_search_call.action.sources", "output.encrypted_content"], + tools: [{ type: "function", name: "read_fixture", parameters: { type: "object", properties: {} } }], + input: [ + { type: "reasoning", encrypted_content: "synthetic-native-ciphertext", summary: [] }, + { type: "function_call", call_id: "call_fixture", name: "read_fixture", arguments: "{}" }, + { type: "function_call_output", call_id: "call_fixture", output: "nonce=synthetic-fixture" }, + ], + }; + const before = structuredClone(raw); + const body = buildBody(xaiOauthResponses({ baseUrl }), raw); + + expect(body.include).toEqual(["reasoning.encrypted_content", "web_search_call.action.sources"]); + expect(body.model).toBe("grok-4.7"); + expect(body.reasoning).toEqual(raw.reasoning); + expect(body.store).toBe(false); + expect(body.tools).toEqual(raw.tools); + expect(body.input).toEqual(raw.input); + expect(raw).toEqual(before); + }); + } + + test("does not invent include values for absent, empty or non-array input", () => { + const provider = xaiOauthResponses(); + expect(buildBody(provider, { input: [] })).not.toHaveProperty("include"); + expect(buildBody(provider, { input: [], include: [] }).include).toEqual([]); + expect(buildBody(provider, { input: [], include: "reasoning.encrypted_content" }).include) + .toBe("reasoning.encrypted_content"); + }); + + for (const baseUrl of ["https://chatgpt.com/backend-api/codex", "https://gateway.example/v1", "https://api.x.ai.evil.example/v1", "https://api.x.ai:444/v1"]) { + test(`preserves include entries and caller input outside xAI at ${baseUrl}`, () => { + const raw = { include: ["output.encrypted_content", "reasoning.encrypted_content"], input: [] }; + const before = structuredClone(raw); + const body = buildBody(xaiOauthResponses({ baseUrl, authMode: "key" }), raw); + + expect(body.include).toEqual(raw.include); + expect(raw).toEqual(before); + }); + } + test("drops the include entry xAI rejects instead of ignoring", () => { const body = buildBody(xaiOauthResponses(), { include: ["output.encrypted_content", "reasoning.encrypted_content"], @@ -57,7 +107,7 @@ describe("xAI Responses include compatibility", () => { }); test("strips the rejected include entry for the US regional endpoint", () => { - const provider = xaiOauthResponses({ baseUrl: "https://us.api.x.ai/v1", authMode: "api-key" }); + const provider = xaiOauthResponses({ baseUrl: "https://us.api.x.ai/v1", authMode: "key" }); const body = buildBody(provider, { include: ["output.encrypted_content", "reasoning.encrypted_content"], input: [], }); @@ -70,7 +120,7 @@ describe("xAI Responses include compatibility", () => { const provider: OcxProviderConfig = { adapter: "openai-responses", baseUrl: "https://api.openai.com/v1", - authMode: "api-key", + authMode: "key", }; const body = buildBody(provider, { include: ["output.encrypted_content"], input: [] }); From 3a70dd4788db76701df546c8629c8d8e17c2680d Mon Sep 17 00:00:00 2001 From: Yu Zhang <34849476+AaronZ345@users.noreply.github.com> Date: Fri, 30 Oct 2026 12:20:09 +0100 Subject: [PATCH 04/42] fix(volcengine): keep Kimi K3.1 Pro tool continuations valid (#7944) * fix(volcengine): preserve empty Kimi K3.1 Pro tool continuations * fix(volcengine): cover Kimi continuation contracts --------- Co-authored-by: zhangyu.34 --- .../src/content/docs/guides/providers.md | 4 +++ src/adapters/openai-chat/messages.ts | 13 +++++---- structure/providers/chat-compat.md | 11 ++++---- .../volcengine-ark-assistant-content.test.ts | 28 +++++++++++++++++-- 4 files changed, 44 insertions(+), 12 deletions(-) diff --git a/docs-site/src/content/docs/guides/providers.md b/docs-site/src/content/docs/guides/providers.md index 4959c8a019b..5a43f759fe2 100644 --- a/docs-site/src/content/docs/guides/providers.md +++ b/docs-site/src/content/docs/guides/providers.md @@ -1082,6 +1082,10 @@ unchanged and were not probed in this update. > models. Coding Plan defaults to `ark-code-latest`, while Agent Plan defaults to > `deepseek-v4-flash`. +On the pay-as-you-go Ark route, Kimi K3.1 Pro (`kimi-k3-1-pro`, including dated versions) +replays tool-only assistant turns with an empty string. Its endpoint rejects the legacy +empty-text array with HTTP 400; other Ark models keep their existing continuation format. + > **Volcengine Plan usage restriction:** Volcengine documents Coding Plan and Agent Plan quota as > valid only inside supported AI coding tools, and warns that using a plan key for general API > calls may suspend the subscription or ban the account. Routing Codex or Claude Code through diff --git a/src/adapters/openai-chat/messages.ts b/src/adapters/openai-chat/messages.ts index 7b45c4581cf..43963d54eb4 100644 --- a/src/adapters/openai-chat/messages.ts +++ b/src/adapters/openai-chat/messages.ts @@ -356,10 +356,10 @@ export function messagesToChatFormat(parsed: OcxParsedRequest, provider: OcxProv type: "function", function: { name: namespacedToolName(tc.namespace, tc.name), arguments: JSON.stringify(tc.arguments) }, })); - if (!chatMsg.content) chatMsg.content = emptyAssistantContent(provider); + if (!chatMsg.content) chatMsg.content = emptyAssistantContent(provider, parsed.modelId); } if (hasReplayedReasoning && chatMsg.content === undefined && chatMsg.tool_calls === undefined) { - chatMsg.content = emptyAssistantContent(provider); + chatMsg.content = emptyAssistantContent(provider, parsed.modelId); } out.push(chatMsg); pendingToolCalls = wireToolCalls.map(({ tc, id }) => ({ id, name: namespacedToolName(tc.namespace, tc.name) })); @@ -413,7 +413,7 @@ export function messagesToChatFormat(parsed: OcxParsedRequest, provider: OcxProv : { reasoning_content: orphanReasoning }; out.push({ role: "assistant", - content: emptyAssistantContent(provider), + content: emptyAssistantContent(provider, parsed.modelId), ...orphanReasoningFields, tool_calls: [{ id: toolCallId, @@ -448,8 +448,11 @@ export function safeToolName(name: string | undefined): string { return sanitized; } -export function emptyAssistantContent(provider: OcxProviderConfig): string | { type: "text"; text: string }[] { - return isVolcengineArkPaygChatTarget(provider) ? [{ type: "text", text: "" }] : ""; +export function emptyAssistantContent(provider: OcxProviderConfig, modelId = ""): string | { type: "text"; text: string }[] { + // Kimi K3.1 Pro rejects Ark's legacy empty-text array during tool continuations. + const upstreamModelId = provider.modelSuffixBracketStrip ? stripBracketedModelSuffix(modelId) : modelId; + const kimi31Pro = /^kimi-k3-1-pro(?:-\d{6})?$/.test(upstreamModelId); + return isVolcengineArkPaygChatTarget(provider) && !kimi31Pro ? [{ type: "text", text: "" }] : ""; } /** Reconstruct completed text-only thinking; never invent signatures or placeholder reasoning. */ diff --git a/structure/providers/chat-compat.md b/structure/providers/chat-compat.md index a675d58bba0..c9c33cb8278 100644 --- a/structure/providers/chat-compat.md +++ b/structure/providers/chat-compat.md @@ -396,11 +396,12 @@ ledger: `devlog/_fin/260709_parallel_tool_calls/`. ## Volcengine Ark assistant continuation shapes -The `openai-chat` adapter keeps Volcengine's pay-as-you-go Chat endpoint and Coding Plan endpoint -on separate empty-assistant contracts. The pay-as-you-go `/api/v3` route retains the structured -`[{ "type": "text", "text": "" }]` placeholder inferred for #796, while `/api/coding/v3` uses the -ordinary empty string accepted by its live tool-call continuation contract (#1571). Matching only -the shared Ark hostname is too broad because the two endpoint families reject opposite shapes. +The `openai-chat` adapter uses `""` for Kimi K3.1 Pro (`kimi-k3-1-pro`, including dated versions) +on Ark's pay-as-you-go `/api/v3` route: tool continuations reject the legacy empty-text array. +This covers ordinary, reasoning-only and synthesized orphan assistant turns. Other pay-as-you-go +Ark models retain the `[{ "type": "text", "text": "" }]` placeholder inferred for #796. +Coding Plan's `/api/coding/v3` uses the empty string accepted by its live continuation contract +(#1571); matching only the shared Ark hostname cannot identify the continuation format. > Decision record: [ADR-0063](../decisions/ADR-0063-volcengine-ark-assistant-continuation-shapes.md) diff --git a/tests/providers/volcengine-ark-assistant-content.test.ts b/tests/providers/volcengine-ark-assistant-content.test.ts index cb48dc2a19a..26426d6b549 100644 --- a/tests/providers/volcengine-ark-assistant-content.test.ts +++ b/tests/providers/volcengine-ark-assistant-content.test.ts @@ -34,9 +34,9 @@ interface ChatMsg { tool_call_id?: string; } -function wire(provider: OcxProviderConfig, messages: OcxMessage[]): ChatMsg[] { +function wire(provider: OcxProviderConfig, messages: OcxMessage[], modelId = "kimi-k3"): ChatMsg[] { const parsed: OcxParsedRequest = { - modelId: "kimi-k3", + modelId, context: { messages }, stream: false, options: {}, @@ -137,4 +137,28 @@ describe("Volcengine Ark empty assistant content (#796)", () => { genericReasoning.preserveReasoningContentModels = ["kimi-k3"]; expect(assistantsOf(wire(genericReasoning, reasoningHistory))[0].content).toBe(""); }); + + for (const modelId of ["kimi-k3-1-pro", "kimi-k3-1-pro-261024"]) { + test(`${modelId} replays tool, orphan and reasoning-only turns with string content`, () => { + const assistant = assistantsOf(wire(ark, history, modelId))[0]; + expect(assistant.content).toBe(""); + expect(assistant.tool_calls).toHaveLength(1); + expect(assistantsOf(wire(ark, history.slice(2), modelId))[0].content).toBe(""); + const reasoning: OcxMessage[] = [ + { role: "assistant", content: [{ type: "thinking", thinking: "deliberating" }], timestamp: 0 }, + ]; + const replay = assistantsOf(wire({ ...ark, preserveReasoningContentModels: [modelId] }, reasoning, modelId))[0]; + expect(replay.content).toBe(""); + expect(replay.reasoning_content).toBe("deliberating"); + expect(assistantsOf(wire(arkCodingPlan, history, modelId))[0].content).toBe(""); + expect(assistantsOf(wire(generic, history, modelId))[0].content).toBe(""); + }); + } + + test("Kimi matching follows the model ID sent upstream", () => { + const bracketed = "kimi-k3-1-pro-261024[1m]"; + expect(assistantsOf(wire({ ...ark, modelSuffixBracketStrip: true }, history, bracketed))[0].content).toBe(""); + expect(assistantsOf(wire(ark, history, bracketed))[0].content).toEqual([{ type: "text", text: "" }]); + expect(assistantsOf(wire(ark, history, "kimi-k3-1-pro-261024-extra"))[0].content).toEqual([{ type: "text", text: "" }]); + }); }); From 4998194755f3698005409a6e75408f4e80a7a81b Mon Sep 17 00:00:00 2001 From: ZeroDi Date: Fri, 30 Oct 2026 20:25:41 +0900 Subject: [PATCH 05/42] perf(responses): count snapshot text bytes incrementally (#7957) --- scripts/test-layout/layout.json | 1 + src/server/responses-snapshot-repair.ts | 22 ++++- structure/transports/byte-accounting.md | 6 ++ tests/fixtures/test-layout-expected.json | 1 + .../responses-snapshot-scaling.test.ts | 98 +++++++++++++++++++ 5 files changed, 124 insertions(+), 4 deletions(-) create mode 100644 tests/responses/responses-snapshot-scaling.test.ts diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index 551e1fa899a..7a7edd47f0a 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -1594,6 +1594,7 @@ "responses-show-thinking-summary.test.ts": "responses", "responses-snapshot-repair-server.test.ts": "responses", "responses-snapshot-repair.test.ts": "responses", + "responses-snapshot-scaling.test.ts": "responses", "responses-sparse-terminal-tool-scope.test.ts": "responses", "responses-spend-ledger-wiring.test.ts": "responses", "responses-spend-contract-c.test.ts": "responses", diff --git a/src/server/responses-snapshot-repair.ts b/src/server/responses-snapshot-repair.ts index 389b8374900..1f3c2da3391 100644 --- a/src/server/responses-snapshot-repair.ts +++ b/src/server/responses-snapshot-repair.ts @@ -169,6 +169,9 @@ type OpenItem = { /** The gateway sent content_part.done for this item. */ partDone: boolean; text: string; + /** Exact UTF-8 size and boundary unit, without rescanning the accumulated text. */ + textBytes: number; + lastCodeUnit: number; item: Record; }; @@ -219,7 +222,7 @@ export function createResponsesSnapshotBlockRewrite( const closeOpenItem = (index: number): void => { const open = openItems.get(index); if (!open) return; - aggregateOpenTextBytes -= Buffer.byteLength(open.text, "utf8"); + aggregateOpenTextBytes -= open.textBytes; openItems.delete(index); }; @@ -313,6 +316,8 @@ export function createResponsesSnapshotBlockRewrite( textDone: false, partDone: false, text: "", + textBytes: 0, + lastCodeUnit: -1, item: event.item, }); } @@ -407,12 +412,19 @@ export function createResponsesSnapshotBlockRewrite( return [changed ? jsonBlock(nextEvent) : block]; } if (open && (itemId === undefined || itemId === open.itemId)) { - const deltaBytes = Buffer.byteLength(event.delta, "utf8"); + let deltaBytes = Buffer.byteLength(event.delta, "utf8"); + const first = event.delta.charCodeAt(0); + // Joining two isolated surrogate halves replaces two 3-byte replacements with 4 bytes. + if (open.lastCodeUnit >= 0xd800 && open.lastCodeUnit <= 0xdbff && first >= 0xdc00 && first <= 0xdfff) { + deltaBytes -= 2; + } + if (event.delta.length > 0) open.lastCodeUnit = event.delta.charCodeAt(event.delta.length - 1); open.text += event.delta; + open.textBytes += deltaBytes; aggregateOpenTextBytes += deltaBytes; // Unbounded text accumulation is a retention hole (#893 review): // overshoot per-item or aggregate caps and the stream goes fail-closed. - if (Buffer.byteLength(open.text, "utf8") > MAX_COMPLETED_OUTPUT_ITEM_SOURCE_BYTES + if (open.textBytes > MAX_COMPLETED_OUTPUT_ITEM_SOURCE_BYTES || aggregateOpenTextBytes > MAX_OPEN_ITEM_AGGREGATE_TEXT_BYTES) { taintAndRelease(); } @@ -437,9 +449,11 @@ export function createResponsesSnapshotBlockRewrite( if (typeof event.text === "string") { // Replacement must keep the aggregate honest and bounded, the // same as delta accumulation (#893 review). - const previousBytes = Buffer.byteLength(open.text, "utf8"); + const previousBytes = open.textBytes; const nextBytes = Buffer.byteLength(event.text, "utf8"); open.text = event.text; + open.textBytes = nextBytes; + open.lastCodeUnit = event.text.charCodeAt(event.text.length - 1); aggregateOpenTextBytes += nextBytes - previousBytes; if (nextBytes > MAX_COMPLETED_OUTPUT_ITEM_SOURCE_BYTES || aggregateOpenTextBytes > MAX_OPEN_ITEM_AGGREGATE_TEXT_BYTES) { diff --git a/structure/transports/byte-accounting.md b/structure/transports/byte-accounting.md index 4f0cba5eca5..ff55305299f 100644 --- a/structure/transports/byte-accounting.md +++ b/structure/transports/byte-accounting.md @@ -224,6 +224,12 @@ tool completion with rewritten arguments. `src/adapters/openai-responses.ts` counts new compaction fragments, including surrogate pairs formed across deltas, while retaining snapshot/done/delta precedence and existing terminal ownership. +The provider-opt-in tracker in `src/server/responses-snapshot-repair.ts` caches each open item's +UTF-8 text size and final code unit. It measures only incoming deltas, corrects split surrogate +pairs even across empty deltas, replaces the count for authoritative text snapshots, and refunds +the cached count on item closure. Per-item and aggregate open-text caps remain 8 MiB; crossing +either cap releases state and disables lifecycle reconstruction. `tests/responses/responses-snapshot-scaling.test.ts` +checks the linear measurement budget, Unicode boundaries, replacement, closure and cap refusal. Serialized request and buffered-response observations use byte counts without measurement arrays. The same rule applies to Anthropic, Google, and Chat response accounting; serialization itself is preserved where the existing metric is the serialized JSON size. diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index a4c1df1686d..5826dcf9196 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -1258,6 +1258,7 @@ "responses-send-budget-errors.test.ts": "responses", "responses-memory-models.test.ts": "responses", "responses-shadow-intercept.test.ts": "responses", "responses-show-thinking-summary.test.ts": "responses", "responses-snapshot-repair-server.test.ts": "responses", "responses-snapshot-repair.test.ts": "responses", + "responses-snapshot-scaling.test.ts": "responses", "responses-sparse-terminal-tool-scope.test.ts": "responses", "responses-spend-ledger-wiring.test.ts": "responses", "responses-spend-contract-c.test.ts": "responses", "responses-spend-anchor.test.ts": "responses", "responses-spend-capacity-guard.test.ts": "responses", "responses-spill-acl-recovery.test.ts": "responses", "responses-spill-admission-headroom.test.ts": "responses", diff --git a/tests/responses/responses-snapshot-scaling.test.ts b/tests/responses/responses-snapshot-scaling.test.ts new file mode 100644 index 00000000000..28880b47af1 --- /dev/null +++ b/tests/responses/responses-snapshot-scaling.test.ts @@ -0,0 +1,98 @@ +import { expect, spyOn, test } from "bun:test"; +import { createResponsesSnapshotBlockRewrite } from "../../src/server/responses-snapshot-repair"; +import { MAX_COMPLETED_OUTPUT_ITEM_SOURCE_BYTES as LIMIT } from "../../src/server/relay"; + +type Event = { type: string; delta?: string; text?: string; response?: { output?: Array<{ content: Array<{ text: string }> }> } }; + +function tracker() { + const rewrite = createResponsesSnapshotBlockRewrite(); + const send = (event: Record) => rewrite(`data: ${JSON.stringify(event)}`) + .map(block => JSON.parse(block.slice("data: ".length)) as Event); + const add = (index: number) => send({ type: "response.output_item.added", output_index: index, item: { type: "message", id: `m${index}` } }); + const delta = (index: number, text: string) => send({ type: "response.output_text.delta", output_index: index, item_id: `m${index}`, delta: text }); + const replace = (index: number, text: string) => send({ type: "response.output_text.done", output_index: index, item_id: `m${index}`, text }); + const finish = () => send({ type: "response.completed", response: { id: "r" } }); + return { rewrite, send, add, delta, replace, finish }; +} + +test("snapshot repair measures only new text instead of every accumulated prefix", () => { + const t = tracker(); + t.add(0); + const fragment = "x".repeat(256); + const fragments = 1024; + const original = Buffer.byteLength; + let measuredChars = 0; + let calls = 0; + const measure = spyOn(Buffer, "byteLength"); + measure.mockImplementation((value, encoding) => { + if (typeof value === "string") { measuredChars += value.length; calls++; } + return original(value, encoding); + }); + try { + for (let i = 0; i < fragments; i++) t.delta(0, fragment); + } finally { + measure.mockRestore(); + } + try { + expect(calls).toBeGreaterThanOrEqual(fragments); + expect(measuredChars).toBeLessThanOrEqual(fragment.length * fragments * 2); + expect(t.finish().find(event => event.type === "response.output_text.done")?.text).toBe(fragment.repeat(fragments)); + } finally { t.rewrite.dispose?.(); } +}); + +for (const highSource of ["delta", "snapshot"] as const) { + test(`snapshot repair counts a split surrogate pair at the byte cap (${highSource})`, () => { + const t = tracker(); + try { + t.add(0); + const prefix = "x".repeat(LIMIT - 4) + "\ud83d"; + if (highSource === "delta") t.delta(0, prefix); + else t.replace(0, prefix); + t.delta(0, ""); + t.delta(0, "\ude00"); // The assembled emoji is 4 bytes, not two 3-byte replacements. + t.replace(0, "ok"); + t.add(1); + const output = t.delta(1, "x".repeat(LIMIT - 2)); + expect(output.map(event => event.type)).toContain("response.content_part.added"); + t.replace(1, "done"); + const terminal = t.finish(); + const items = terminal.find(event => event.type === "response.completed")?.response?.output; + expect(items?.map(item => item.content[0]?.text)).toEqual(["ok", "done"]); + } finally { t.rewrite.dispose?.(); } + }); +} + +test("snapshot repair refunds closed text before admitting the next open item", () => { + const t = tracker(); + try { + t.add(0); + t.delta(0, "x".repeat(LIMIT)); + t.send({ type: "response.output_item.done", output_index: 0, item: { type: "message", id: "m0", content: [] } }); + t.add(1); + expect(t.delta(1, "x".repeat(LIMIT)).map(event => event.type)).toContain("response.content_part.added"); + } finally { t.rewrite.dispose?.(); } +}); + +test("snapshot repair accepts the exact aggregate cap and stops retaining above it", () => { + const t = tracker(); + try { + t.add(0); + t.add(1); + t.delta(0, "x".repeat(LIMIT - 3)); + expect(t.delta(1, "한").map(event => event.type)).toContain("response.content_part.added"); + const overflow = t.delta(1, "!"); + expect(overflow.at(-1)?.delta).toBe("!"); + expect(t.finish().map(event => event.type)).toEqual(["response.completed"]); + } finally { t.rewrite.dispose?.(); } +}); + +test("snapshot repair preserves empty, non-ASCII and unpaired-surrogate deltas", () => { + const t = tracker(); + const deltas = ["한", "\ud83d", "", "\ude00", "\ud83d", "x", "\ude00"]; + try { + t.add(0); + for (const delta of deltas) expect(t.delta(0, delta).at(-1)?.delta).toBe(delta); + const terminal = t.finish(); + expect(terminal.find(event => event.type === "response.output_text.done")?.text).toBe(deltas.join("")); + } finally { t.rewrite.dispose?.(); } +}); From 572467f21d82b30bc2070c48e8753438492ff97d Mon Sep 17 00:00:00 2001 From: Epinephrine Date: Fri, 30 Oct 2026 11:29:15 +0000 Subject: [PATCH 06/42] perf(history): move integrity scans off the serving thread (#7972) Validate persisted indexes once per handle and every five minutes in a read-only worker while preserving automatic corruption repair. Bound read retries, keep operational failures nondestructive, and drain old scanners before replacing files. Cover warm reads, real B-tree corruption, analytics, routing evidence, ingestion and worker lifetime races with native Bun tests. Refs lidge-jun/opencodex#7966 --- .../ko/reference/configuration/routing.md | 21 + .../docs/reference/configuration/routing.md | 23 + scripts/test-layout/layout.json | 2 +- src/routing/analytics.ts | 8 +- src/routing/health.ts | 8 +- src/routing/history/build-file.ts | 6 + src/routing/history/indexer.ts | 189 +++++-- src/routing/history/ingest.ts | 2 +- src/routing/history/integrity.ts | 177 ++++++ src/routing/history/rebuild-job.ts | 26 +- src/routing/history/rebuild-protocol.ts | 3 +- src/routing/history/rebuild-worker.ts | 18 + src/routing/history/sqlite-errors.ts | 9 + structure/dashboard-and-usage.md | 12 +- tests/fixtures/test-layout-expected.json | 2 +- tests/usage/request-history-integrity.test.ts | 525 ++++++++++++++++++ 16 files changed, 976 insertions(+), 55 deletions(-) create mode 100644 src/routing/history/integrity.ts create mode 100644 src/routing/history/sqlite-errors.ts create mode 100644 tests/usage/request-history-integrity.test.ts diff --git a/docs-site/src/content/docs/ko/reference/configuration/routing.md b/docs-site/src/content/docs/ko/reference/configuration/routing.md index d19dce2f092..fed5f09dbfc 100644 --- a/docs-site/src/content/docs/ko/reference/configuration/routing.md +++ b/docs-site/src/content/docs/ko/reference/configuration/routing.md @@ -121,6 +121,27 @@ CLI: `ocx route policy list`, `ocx route policy show `, `ocx route policy dr CLI: `ocx logs explain `, `ocx logs rebuild-index`, `ocx logs index-status`. +### 인덱스 검사와 복구 + +기존 기록 인덱스를 처음 열면 읽기 전용 워커에서 무결성을 검사합니다. +기록 조회는 이 초기 검사를 기다리며, 프록시의 요청 처리 스레드를 막지 않습니다. +동기식 라우팅 조회는 검사 중에 busy를 반환하므로 라우팅 프로필의 +`unknownEvidence.health` 정책이 적용됩니다. 작은 인덱스를 인라인으로 재구축한 경우에는 +성공 직후 사용할 수 있으며, 워커에서 재구축한 인덱스는 게시 전에 검사를 마칩니다. + +검사나 재구축에 성공한 뒤 최소 5분이 지나면 다음 조회가 백그라운드 무결성 검사 하나를 +시작합니다. 이미 검증된 인덱스는 검사 중에도 조회할 수 있습니다. 일반 갱신은 스키마, +원본 파일 식별자와 바이트 오프셋을 확인하지만 전체 동기식 검사를 반복하지 않으며, +조회가 없는 동안에는 타이머를 실행하지 않습니다. 데이터베이스 잠금이나 워커 실행 실패 +같은 일시적인 검사 오류는 인덱스를 삭제하지 않고, 30초가 지난 뒤의 조회에서 다시 시도합니다. +초기 검증을 마치지 못한 인덱스는 그동안 사용할 수 없습니다. + +손상이 탐지되면 `usage.jsonl`에서 자동으로 재구축하며, 영향을 받은 비동기 기록·분석 +조회는 복구 후 한 번 재시도합니다. 인덱스를 닫거나 재구축할 때에는 이전 검사 결과를 +폐기하고 검사 워커가 파일을 해제한 뒤 인덱스를 교체합니다. 검사와 복구는 원본 기록을 +다시 쓰지 않습니다. 초기 검사에는 인덱스 크기에 비례하는 시간이 걸리며, +이후 백그라운드 검사는 인덱스를 사용하는 동안에만 실행됩니다. + ## 마이그레이션 `routingProfiles`는 선택적 추가 설정입니다. 기존 설정 파일과 이전 `usage.jsonl` 행은 그대로 읽힙니다. 인덱스는 일회용이며 삭제 시 다음 쿼리에서 `usage.jsonl`로 자동 재구축됩니다. 자동 튜닝은 없습니다. diff --git a/docs-site/src/content/docs/reference/configuration/routing.md b/docs-site/src/content/docs/reference/configuration/routing.md index e518b009789..63a3a0eae1f 100644 --- a/docs-site/src/content/docs/reference/configuration/routing.md +++ b/docs-site/src/content/docs/reference/configuration/routing.md @@ -401,6 +401,29 @@ CLI: `ocx logs history`, `ocx logs get `, `ocx logs explain `, `ocx logs rebuild-index`, `ocx logs index-status`, `ocx route policy list | show | dry-run | evaluate`. +### Index validation and recovery + +An existing history index is checked in a read-only worker when first opened. +History queries wait for that initial check without blocking the proxy's serving +thread. Synchronous routing reads report busy during validation, so routing +profiles apply their `unknownEvidence.health` policy. A successful small inline +rebuild is usable immediately; a worker rebuild is checked before publication. + +After a successful check or rebuild, the next read at least five minutes later +starts one background integrity check. Reads from the verified index continue +during that check. Normal refreshes retain schema, source-identity and offset +validation without a full synchronous scan, and an idle index starts no timer. +Temporary check failures, such as a busy database or unavailable worker, leave +the index intact and retry on a subsequent read after a 30-second backoff. +An index that has not completed initial validation remains unavailable meanwhile. + +Detected corruption triggers automatic reconstruction from `usage.jsonl`, with +one retry for an affected asynchronous history or analytics read. Closing or +rebuilding the index discards old check results and waits for the scanner to +release the file before replacing it. Validation and recovery never rewrite the +canonical ledger. The initial check still takes time proportional to index size; +later background checks run only when the index is used. + ## Migration `routingProfiles` is optional and additive: existing config files load diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index 7a7edd47f0a..efa3f4e689d 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -1495,7 +1495,7 @@ "repo-hygiene.test.ts": "ci-workflows", "request-decompress.test.ts": "usage", "request-evidence.test.ts": "usage", - "request-history-index.test.ts": "usage", + "request-history-index.test.ts": "usage", "request-history-integrity.test.ts": "usage", "request-log-attempt-identity.test.ts": "usage", "request-log-conversation.test.ts": "usage", "request-log-estimate-cap.test.ts": "usage", diff --git a/src/routing/analytics.ts b/src/routing/analytics.ts index ab919d88433..220a65e70f5 100644 --- a/src/routing/analytics.ts +++ b/src/routing/analytics.ts @@ -13,7 +13,7 @@ import { normalizePersistedUsageRow, type PersistedUsageEntry } from "../usage/log"; import { computeEntryCost } from "../usage/summary"; -import { openRequestHistoryIndex, requestHistoryDb } from "./history/indexer"; +import { withRequestHistoryRead } from "./history/indexer"; export const ANALYTICS_MAX_ROWS = 50_000; /** Default row cap for the management API (full cap remains available via `limit`). */ @@ -168,8 +168,6 @@ export async function computeRoutingAnalytics( filters: RoutingAnalyticsFilters, options: { maxRows?: number } = {}, ): Promise { - await openRequestHistoryIndex(); - const handle = requestHistoryDb(); const maxRows = Math.min( Math.max(1, Math.trunc(options.maxRows ?? ANALYTICS_MAX_ROWS)), ANALYTICS_MAX_ROWS, @@ -189,7 +187,7 @@ export async function computeRoutingAnalytics( if (filters.to !== undefined) add("timestamp <= ?", filters.to); const whereSql = where.length > 0 ? ` WHERE ${where.join(" AND ")}` : ""; - const rows = handle.query( + const rows = await withRequestHistoryRead(handle => handle.query( `SELECT provider, model, api_key_id AS apiKeyId, profile_id AS profileId, profile_revision AS profileRevision, status, duration_ms AS durationMs, first_output_ms AS firstOutputMs, @@ -197,7 +195,7 @@ export async function computeRoutingAnalytics( usage_status AS usageStatus, usage_json AS usageJson, attempt_count AS attemptCount, fallback, row_json AS rowJson FROM requests${whereSql} ORDER BY timestamp DESC LIMIT ?`, - ).all(...values, maxRows + 1) as ScannedRow[]; + ).all(...values, maxRows + 1) as ScannedRow[]); const scanned = rows.slice(0, maxRows); const historyTruncated = rows.length > maxRows; diff --git a/src/routing/health.ts b/src/routing/health.ts index ceab2e1b9a1..af4b20f5e42 100644 --- a/src/routing/health.ts +++ b/src/routing/health.ts @@ -17,7 +17,7 @@ */ import type { OcxConfig } from "../types"; -import { openRequestHistoryIndexSync, requestHistoryDb } from "./history/indexer"; +import { openRequestHistoryIndexSync, reportRequestHistoryCorruption, requestHistoryDb } from "./history/indexer"; import { OPENAI_CODEX_PROVIDER_ID } from "../providers/openai-tiers"; import { getCodexAccountCooldownUntil, @@ -251,9 +251,10 @@ function computeHistoricalHealthEvidence( input: Pick, now: number, ): HistoricalHealthEvidence { + let handle: ReturnType | undefined; try { openRequestHistoryIndexSync(); - const handle = requestHistoryDb(); + handle = requestHistoryDb(); const where: string[] = ["provider = ?", "model = ?", "timestamp >= ?"]; const values: Array = [input.provider, input.model, now - HEALTH_WINDOW_MS]; if (input.accountRef) { @@ -341,7 +342,8 @@ function computeHistoricalHealthEvidence( out.recencyWeight = decayWeight(samples[0]!.timestamp, now); } return out; - } catch { + } catch (error) { + if (handle) reportRequestHistoryCorruption(error, handle); /* index unreadable: evidence stays unknown */ return {}; } diff --git a/src/routing/history/build-file.ts b/src/routing/history/build-file.ts index 772697e990e..de443b549e2 100644 --- a/src/routing/history/build-file.ts +++ b/src/routing/history/build-file.ts @@ -79,6 +79,12 @@ export function buildIndexFile(buildPath: string, reason: string, revision: Usag expected: revision, endOffset: Number(revision.size), }); + // Validate before publication while still off the serving thread. The + // published handle can then be trusted until its periodic worker check. + const check = build.query("PRAGMA quick_check").get() as { quick_check?: string } | undefined; + if (check?.quick_check !== "ok") { + throw Object.assign(new Error("rebuilt history index failed integrity check"), { code: "SQLITE_CORRUPT" }); + } // The serving thread opens the published file in WAL mode; switching here // keeps that first open from rewriting the database header. build.exec("PRAGMA journal_mode = WAL"); diff --git a/src/routing/history/indexer.ts b/src/routing/history/indexer.ts index 342e98fab4b..68413390ab4 100644 --- a/src/routing/history/indexer.ts +++ b/src/routing/history/indexer.ts @@ -3,7 +3,7 @@ * * `usage.jsonl` stays canonical; this module maintains a rebuildable SQLite * projection (ADR-1, ADR-8). On every open/query it verifies schema version, - * file identity, integrity, and byte offset, then appends whatever complete + * file identity and byte offset, then appends whatever complete * JSONL rows arrived since the last index. Missing/corrupt/stale index or a * replaced/truncated source triggers an automatic full rebuild; canonical * history is never touched. @@ -12,6 +12,8 @@ * runs on a worker thread (#6990): the worker writes a private build file and * this thread swaps it in once complete. Async readers wait for it; the * routing-time sync entry never waits and throws `RequestHistoryIndexBusyError`. + * Full integrity scans run on that worker at open and periodically while in + * use, never synchronously on the serving thread (#7966). */ import { Database } from "bun:sqlite"; @@ -48,6 +50,8 @@ import { } from "./ingest"; import { buildFilePath, removeBuildFiles } from "./build-file"; import { RebuildWorkerJob } from "./rebuild-job"; +import { RequestHistoryIntegrityMonitor, type RequestHistoryIntegrityOptions } from "./integrity"; +import { isHistoryIndexCorruption } from "./sqlite-errors"; export { REQUEST_HISTORY_INSERT_BATCH, @@ -138,9 +142,15 @@ class RebuildHandedOff { constructor(readonly rebuild: WorkerRebuild) {} } +/** An initial integrity check, or a scanner draining before index replacement. */ +class IntegrityHandedOff { + constructor(readonly promise: Promise) {} +} + let db: Database | null = null; let dbPath = ""; let openPromise: Promise | null = null; +let manualRebuild: Promise | null = null; let workerRebuild: WorkerRebuild | null = null; let workerFailure: { at: number; error: unknown } | null = null; let workerRebuildMinBytes = REQUEST_HISTORY_WORKER_REBUILD_MIN_BYTES; @@ -148,6 +158,33 @@ let maxAsyncRefreshBuilds = MAX_ASYNC_REFRESH_BUILDS; let buildSequence = 0; // Bumped by closeRequestHistoryIndex, so a rebuild finishing after a close is discarded. let indexLifetime = 0; +let integrity = new RequestHistoryIntegrityMonitor(); +let integrityDrain: Promise | null = null; +let corruptHandle: Database | null = null; +let validatedBuildPath: string | null = null; + +/** Test seam: replace integrity scheduling without leaving a scanner on an old file. */ +export function setRequestHistoryIntegrityOptionsForTests(options?: RequestHistoryIntegrityOptions): void { + closeRequestHistoryIndex(); + integrity = new RequestHistoryIntegrityMonitor(options); +} + +function stopIntegrityScan(): Promise | null { + const pending = integrity.quiesce(); + if (pending && !integrityDrain) { + const drained = pending.finally(() => { + if (integrityDrain === drained) integrityDrain = null; + }); + integrityDrain = drained; + } + return integrityDrain; +} + +/** Never delete/rename a database that a cancelled scanner may still hold on Windows. */ +function prepareIndexReplacement(): void { + const pending = stopIntegrityScan(); + if (pending) throw new IntegrityHandedOff(pending); +} /** Test seam: move the worker threshold; `undefined` restores the default. */ export function setRequestHistoryWorkerRebuildMinBytesForTests(bytes: number | undefined): void { @@ -210,15 +247,6 @@ export function sourceIdentityMatches(dbHandle: Database, revision: UsageLogRevi && storedBirthtimeMs === Number(revision.birthtimeMs); } -function isHealthy(dbHandle: Database): boolean { - try { - const row = dbHandle.query("PRAGMA quick_check").get() as { quick_check?: string } | undefined; - return row?.quick_check === "ok"; - } catch { - return false; - } -} - /** Remove the index file and its WAL side files. */ function removeIndexFiles(path: string): void { // Windows: a partially-opened handle from a failed `new Database` can hold @@ -238,6 +266,7 @@ function removeIndexFiles(path: string): void { } function destroyAndRecreate(path: string, reason: string): Database { + prepareIndexReplacement(); if (db) { try { db.close(); } catch { /* already closed */ } db = null; @@ -249,10 +278,13 @@ function destroyAndRecreate(path: string, reason: string): Database { resetAndCreateSchema(fresh); setMeta(fresh, HISTORY_META_KEYS.lastError, reason); db = fresh; + corruptHandle = null; + validatedBuildPath = null; return fresh; } function openIndexDb(): Database { + if (integrityDrain) throw new IntegrityHandedOff(integrityDrain); const path = indexDbPath(); if (db) return db; let handle: Database | undefined; @@ -270,45 +302,64 @@ function openIndexDb(): Database { setMeta(handle, HISTORY_META_KEYS.builtAtMs, Date.now()); setMeta(handle, HISTORY_META_KEYS.lastError, "created"); } - } catch { + } catch (error) { // A partially-opened handle on a corrupt file can hold the OS lock on // Windows; close it before the destructive recreate. if (handle) { try { handle.close(); } catch { /* already unusable */ } } + if (!isHistoryIndexCorruption(error)) throw error; handle = destroyAndRecreate(path, "unreadable database recreated"); } dbPath = path; db = handle; + if (validatedBuildPath === path) { + integrity.acceptRebuilt(handle, path); + validatedBuildPath = null; + } return handle; } function ensureSchemaAndIdentity(dbHandle: Database): "ready" | "rebuilt" { try { + if (corruptHandle === dbHandle) { + destroyAndRecreate(dbPath, "integrity check failed; index rebuilt"); + return "rebuilt"; + } const storedVersion = metaValue(dbHandle, HISTORY_META_KEYS.schemaVersion); if (storedVersion !== String(HISTORY_SCHEMA_VERSION)) { destroyAndRecreate(dbPath, `schema version ${storedVersion ?? "missing"} -> ${HISTORY_SCHEMA_VERSION}`); return "rebuilt"; } - if (!isHealthy(dbHandle)) { - destroyAndRecreate(dbPath, "integrity check failed; index rebuilt"); - return "rebuilt"; - } const revision = sourceIdentity(); if (!sourceIdentityMatches(dbHandle, revision)) { destroyAndRecreate(dbPath, "source identity changed; index rebuilt"); return "rebuilt"; } + const check = integrity.inspect(dbHandle, dbPath); + if (check.state === "pending") throw new IntegrityHandedOff(check.promise); + if (check.state === "unavailable") { + throw new RequestHistoryIndexBusyError("request-history integrity check unavailable"); + } + if (check.state === "corrupt") { + corruptHandle = dbHandle; + destroyAndRecreate(dbPath, "integrity check failed; index rebuilt"); + return "rebuilt"; + } return "ready"; - } catch { + } catch (error) { + if (!isHistoryIndexCorruption(error)) throw error; // A file that opens but is not actually SQLite (or is mid-corruption) // throws on the first statement; treat it as corrupt and rebuild. + corruptHandle = dbHandle; + integrity.markCorrupt(dbHandle); destroyAndRecreate(dbPath, "index unreadable; rebuilt"); return "rebuilt"; } } function fullRebuild(dbHandle: Database, reason: string): void { + prepareIndexReplacement(); resetAndCreateSchema(dbHandle); setMeta(dbHandle, HISTORY_META_KEYS.lastError, reason); const path = usageLogPath(); @@ -319,6 +370,8 @@ function fullRebuild(dbHandle: Database, reason: string): void { setMeta(dbHandle, HISTORY_META_KEYS.indexedOffset, 0); setMeta(dbHandle, HISTORY_META_KEYS.indexedRows, 0); setMeta(dbHandle, HISTORY_META_KEYS.builtAtMs, Date.now()); + integrity.acceptRebuilt(dbHandle, dbPath); + corruptHandle = null; return; } const inserted = ingestSourceTail(dbHandle, path, 0); @@ -326,6 +379,8 @@ function fullRebuild(dbHandle: Database, reason: string): void { setMeta(dbHandle, HISTORY_META_KEYS.indexedRows, inserted); setMeta(dbHandle, HISTORY_META_KEYS.indexedOffset, Math.max(current.indexedOffset, 0)); setMeta(dbHandle, HISTORY_META_KEYS.builtAtMs, Date.now()); + integrity.acceptRebuilt(dbHandle, dbPath); + corruptHandle = null; } /** @@ -372,12 +427,15 @@ function startWorkerRebuild(reason: string, revision: UsageLogRevision): WorkerR /** Swap a complete worker build in for the current index file. */ function installBuild(buildPath: string, indexPath: string): void { + prepareIndexReplacement(); if (db) { try { db.close(); } catch { /* already closed */ } db = null; } removeIndexFiles(indexPath); renameSync(buildPath, indexPath); + corruptHandle = null; + validatedBuildPath = indexPath; } /** @@ -389,6 +447,7 @@ type RefreshCaller = "routing" | "reader"; /** Full rebuild, on the worker when the ledger is large enough to stall the event loop. */ function rebuildFromLedger(handle: Database, reason: string, caller: RefreshCaller): void { + prepareIndexReplacement(); const revision = sourceIdentity(); if (revision !== null && Number(revision.size) >= workerRebuildMinBytes) { if (workerCoolingDown()) { @@ -405,7 +464,7 @@ function rebuildFromLedger(handle: Database, reason: string, caller: RefreshCall fullRebuild(handle, reason); } -function refreshLockedSync(caller: RefreshCaller): RequestHistoryIndexMeta { +function refreshOnceLockedSync(caller: RefreshCaller): RequestHistoryIndexMeta { if (workerRebuild) throw new RebuildHandedOff(workerRebuild); openIndexDb(); const state = ensureSchemaAndIdentity(db!); @@ -436,27 +495,55 @@ function refreshLockedSync(caller: RefreshCaller): RequestHistoryIndexMeta { return metaFor(handle); } +function refreshLockedSync(caller: RefreshCaller): RequestHistoryIndexMeta { + try { + return refreshOnceLockedSync(caller); + } catch (error) { + // Metadata reads and incremental writes can encounter damage even between + // scheduled checks. Operational failures (busy, locked, I/O, full) propagate. + if (!isHistoryIndexCorruption(error)) throw error; + if (db) { + corruptHandle = db; + integrity.markCorrupt(db); + } + const handle = destroyAndRecreate(dbPath, "SQLite corruption; index rebuilt"); + rebuildFromLedger(handle, "SQLite corruption; index rebuilt", caller); + return metaFor(handle); + } +} + /** * Synchronous refresh for routing-time evidence reads (RI-06+). Never waits * for a worker rebuild: while one runs (this call may start it), throws * `RequestHistoryIndexBusyError`, which health scoring reads as unknown evidence. */ export function openRequestHistoryIndexSync(): RequestHistoryIndexMeta { + if (manualRebuild) throw new RequestHistoryIndexBusyError(); try { return refreshLockedSync("routing"); } catch (error) { - if (error instanceof RebuildHandedOff) throw new RequestHistoryIndexBusyError(); + if (error instanceof RebuildHandedOff || error instanceof IntegrityHandedOff) throw new RequestHistoryIndexBusyError(); throw error; } } -async function refreshWaitingForWorker(): Promise { +async function refreshWaitingForWorker(fromManualRebuild = false): Promise { const lifetime = indexLifetime; for (let builds = 0; ; ) { + if (manualRebuild && !fromManualRebuild) { + await manualRebuild; + if (lifetime !== indexLifetime) throw new RequestHistoryIndexClosedError(); + continue; + } let handedOff: RebuildHandedOff; try { return refreshLockedSync("reader"); } catch (error) { + if (error instanceof IntegrityHandedOff) { + await error.promise; + if (lifetime !== indexLifetime) throw new RequestHistoryIndexClosedError(); + continue; + } if (!(error instanceof RebuildHandedOff)) throw error; handedOff = error; } @@ -480,15 +567,19 @@ async function refreshWaitingForWorker(): Promise { */ export function openRequestHistoryIndex(): Promise { if (!openPromise) { - openPromise = refreshWaitingForWorker().finally(() => { - openPromise = null; + const pending = refreshWaitingForWorker().finally(() => { + if (openPromise === pending) openPromise = null; }); + openPromise = pending; } return openPromise; } export function closeRequestHistoryIndex(): void { indexLifetime += 1; + stopIntegrityScan(); + corruptHandle = null; + validatedBuildPath = null; const rebuild = workerRebuild; workerRebuild = null; workerFailure = null; @@ -499,21 +590,36 @@ export function closeRequestHistoryIndex(): void { db = null; } openPromise = null; + manualRebuild = null; } /** Force a full rebuild from the canonical ledger (CLI / tests). */ -export async function rebuildRequestHistoryIndex(): Promise { +export function rebuildRequestHistoryIndex(): Promise { + if (!manualRebuild) { + const pending = rebuildManually(indexLifetime).finally(() => { + if (manualRebuild === pending) manualRebuild = null; + }); + manualRebuild = pending; + } + return manualRebuild; +} + +async function rebuildManually(lifetime: number): Promise { + const draining = stopIntegrityScan(); + if (draining) await draining; + if (lifetime !== indexLifetime) throw new RequestHistoryIndexClosedError(); while (workerRebuild) await workerRebuild.promise.catch(() => undefined); + if (lifetime !== indexLifetime) throw new RequestHistoryIndexClosedError(); const handle = openIndexDb(); const revision = sourceIdentity(); if (revision !== null && Number(revision.size) >= workerRebuildMinBytes) { workerFailure = null; const rebuild = startWorkerRebuild("manual rebuild requested", revision); if (rebuild) { - const lifetime = indexLifetime; try { await rebuild.promise; - return openRequestHistoryIndex(); + if (lifetime !== indexLifetime) throw new RequestHistoryIndexClosedError(); + return refreshWaitingForWorker(true); } catch (error) { // A close is final; any other worker failure falls back to the inline rebuild below. if (lifetime !== indexLifetime || error instanceof RequestHistoryIndexClosedError) throw error; @@ -575,6 +681,29 @@ function requireDb(): Database { return handle; } +/** Invalidate only the live generation, including errors from routing's synchronous reads. */ +export function reportRequestHistoryCorruption(error: unknown, handle: Database): boolean { + if (!isHistoryIndexCorruption(error)) return false; + if (db === handle) { + corruptHandle = handle; + integrity.markCorrupt(handle); + } + return true; +} + +/** Read-only operations may retry once after repair; ordinary SQLite failures propagate. */ +export async function withRequestHistoryRead(read: (handle: Database, meta: RequestHistoryIndexMeta) => T): Promise { + for (let attempt = 0; ; attempt++) { + const meta = await openRequestHistoryIndex(); + const handle = requireDb(); + try { + return read(handle, meta); + } catch (error) { + if (!reportRequestHistoryCorruption(error, handle) || attempt > 0) throw error; + } + } +} + function hydrateRow(row: HistoryQueryRow | undefined): PersistedUsageEntry | null { if (!row) return null; try { @@ -593,7 +722,6 @@ export async function queryRequestHistory( rawCursor: string | null | undefined, pageSize: number | undefined, ): Promise { - const meta = await openRequestHistoryIndex(); const cursor = decodeHistoryCursor(rawCursor); if (rawCursor !== null && rawCursor !== undefined && cursor === null) { throw new InvalidCursorError(); @@ -602,8 +730,9 @@ export async function queryRequestHistory( Math.max(1, Math.trunc(pageSize ?? REQUEST_HISTORY_DEFAULT_PAGE_SIZE)), REQUEST_HISTORY_MAX_PAGE_SIZE, ); - const handle = requireDb(); - const { rows, fetched } = queryRows(handle, filters, cursor, limit); + const { rows, fetched, meta } = await withRequestHistoryRead((handle, meta) => ({ + ...queryRows(handle, filters, cursor, limit), meta, + })); const hasMore = fetched > limit; const pageRows = rows.slice(0, limit); const entries: PersistedUsageEntry[] = []; @@ -620,11 +749,9 @@ export async function queryRequestHistory( } export async function requestHistoryRowById(requestId: string): Promise { - await openRequestHistoryIndex(); - const handle = requireDb(); - const row = handle.query("SELECT row_json FROM requests WHERE request_id = ?").get(requestId) as + const row = await withRequestHistoryRead(handle => handle.query("SELECT row_json FROM requests WHERE request_id = ?").get(requestId) as | { row_json: string } - | undefined; + | undefined); return hydrateRow(row); } diff --git a/src/routing/history/ingest.ts b/src/routing/history/ingest.ts index 16a98d726b9..429fe4dc974 100644 --- a/src/routing/history/ingest.ts +++ b/src/routing/history/ingest.ts @@ -25,7 +25,7 @@ export const REQUEST_HISTORY_MAX_RECORD_BYTES = 1024 * 1024; // One small observation per live database handle, never retained ledger bytes. An // unterminated tail cannot advance indexedOffset, but need not be reread until its -// exact file revision changes. The indexer's schema/identity/quick_check gates still run. +// exact file revision changes. Schema/identity gates and lifecycle integrity checks still apply. const tailObservations = new WeakMap; + +export interface RequestHistoryIntegrityOptions { + createJob?: () => IntegrityJob; + now?: () => number; + intervalMs?: number; + failureCooldownMs?: number; +} + +export type RequestHistoryIntegrityState = + | { state: "ready" } + | { state: "corrupt" } + | { state: "unavailable"; error: unknown } + // Never rejects, including on cancellation or check failure. Completion + // proves the scanner exited; inspect again to obtain its verdict or error. + | { state: "pending"; promise: Promise }; + +interface Generation { + handle: object; + path: string; + verified: boolean; + corrupt: boolean; + nextCheckAt: number; + failure: { error: unknown } | null; +} + +interface ActiveCheck { + generation: Generation; + cancelled: boolean; + cancel: () => void; + promise: Promise; +} + +type CheckOutcome = + | { kind: "checked"; healthy: boolean } + | { kind: "failed"; error: unknown } + | { kind: "cancelled" }; + +export class RequestHistoryIntegrityMonitor { + private readonly createJob: () => IntegrityJob; + private readonly now: () => number; + private readonly intervalMs: number; + private readonly failureCooldownMs: number; + private current: Generation | null = null; + private active: ActiveCheck | null = null; + + constructor(options: RequestHistoryIntegrityOptions = {}) { + this.createJob = options.createJob ?? (() => new RebuildWorkerJob()); + this.now = options.now ?? Date.now; + this.intervalMs = options.intervalMs ?? REQUEST_HISTORY_INTEGRITY_INTERVAL_MS; + this.failureCooldownMs = options.failureCooldownMs ?? REQUEST_HISTORY_INTEGRITY_FAILURE_COOLDOWN_MS; + } + + inspect(handle: object, path: string): RequestHistoryIntegrityState { + if (this.current?.handle !== handle || this.current.path !== path) { + this.quiesce(); + this.current = { handle, path, verified: false, corrupt: false, nextCheckAt: -Infinity, failure: null }; + } + const generation = this.current; + if (generation.corrupt) return { state: "corrupt" }; + if (this.active) { + if (this.active.generation === generation && !this.active.cancelled && generation.verified) { + return { state: "ready" }; + } + return { state: "pending", promise: this.active.promise }; + } + if (this.now() >= generation.nextCheckAt) { + const check = this.start(generation); + if (check && !generation.verified) return { state: "pending", promise: check.promise }; + } + if (generation.verified) return { state: "ready" }; + return { state: "unavailable", error: generation.failure?.error ?? new Error("history integrity check unavailable") }; + } + + /** A statement detected corruption; an in-flight healthy verdict cannot erase it. */ + markCorrupt(handle: object): void { + if (this.current?.handle === handle) this.current.corrupt = true; + } + + /** + * Accept a newly reconstructed projection after successful inline ingest or + * validation of a private worker build. Existing persisted databases must + * use inspect instead. The caller drains scans before rebuilding/swapping. + */ + acceptRebuilt(handle: object, path: string): void { + if (this.active) throw new Error("history integrity worker must exit before accepting a rebuilt index"); + this.current = { + handle, path, verified: true, corrupt: false, + nextCheckAt: this.now() + this.intervalMs, failure: null, + }; + } + + /** + * Invalidate and stop the current check. Null means the index can be replaced + * now; otherwise await the returned barrier before deleting or renaming it. + * The barrier survives reset/reopen and resolves only after actual worker + * exit, including Windows' OS-thread settling, not the bounded disposal wait. + */ + quiesce(): Promise | null { + this.current = null; + this.active?.cancel(); + return this.active?.promise ?? null; + } + + reset(): void { + this.quiesce(); + } + + private start(generation: Generation): ActiveCheck | null { + let job: IntegrityJob; + try { + job = this.createJob(); + } catch (error) { + this.recordFailure(generation, error); + return null; + } + let cancel!: () => void; + const cancelled = new Promise(resolve => { cancel = () => resolve({ kind: "cancelled" }); }); + const check: ActiveCheck = { + generation, + cancelled: false, + cancel: () => { check.cancelled = true; cancel(); }, + promise: Promise.resolve(), + }; + this.active = check; + check.promise = this.run(check, job, cancelled); + return check; + } + + private recordFailure(generation: Generation, error: unknown): void { + generation.failure = { error: error ?? new Error("history integrity check failed") }; + generation.nextCheckAt = this.now() + this.failureCooldownMs; + } + + private async run(check: ActiveCheck, job: IntegrityJob, cancelled: Promise): Promise { + const observed = Promise.resolve().then(async (): Promise => { + if (check.cancelled) return { kind: "cancelled" }; + try { + return { kind: "checked", healthy: await job.check(check.generation.path) }; + } catch (error) { + return { kind: "failed", error }; + } + }); + const outcome = await Promise.race([observed, cancelled]); + await new Promise(resolve => { + // afterExit may return at its disposal timeout, before native I/O lets + // the worker exit. Only its callback releases the file replacement gate. + void job.afterExit(resolve).catch(() => undefined); + }); + const generation = check.generation; + if (this.current === generation && !check.cancelled && !generation.corrupt) { + if (outcome.kind === "checked") { + generation.verified = outcome.healthy; + generation.corrupt = !outcome.healthy; + generation.failure = null; + generation.nextCheckAt = this.now() + this.intervalMs; + } else if (outcome.kind === "failed") { + this.recordFailure(generation, outcome.error); + } + } + if (this.active === check) this.active = null; + } +} diff --git a/src/routing/history/rebuild-job.ts b/src/routing/history/rebuild-job.ts index 18467701835..e98e2050aec 100644 --- a/src/routing/history/rebuild-job.ts +++ b/src/routing/history/rebuild-job.ts @@ -13,6 +13,7 @@ import { RequestHistoryIndexStaleError } from "./ingest"; import type { RebuildWorkerReply, RebuildWorkerRequest } from "./rebuild-protocol"; const CLOSE_TIMEOUT_MS = 5_000; +type WorkerSuccess = Exclude; function delay(ms: number): Promise { return new Promise(resolve => setTimeout(resolve, ms)); @@ -34,7 +35,7 @@ function errorFromReply(reply: Extract): export class RebuildWorkerJob { private readonly worker: Worker; - private pending: { resolve: (rows: number) => void; reject: (error: unknown) => void } | null = null; + private pending: { resolve: (reply: WorkerSuccess) => void; reject: (error: unknown) => void } | null = null; private exited = false; private exitedCleanly = false; private disposal: Promise | null = null; @@ -49,8 +50,8 @@ export class RebuildWorkerJob { this.pending = null; if (!pending) return; const reply = event.data; - if (reply.type === "built") pending.resolve(reply.rows); - else pending.reject(errorFromReply(reply)); + if (reply.type === "error") pending.reject(errorFromReply(reply)); + else pending.resolve(reply); }); this.worker.addEventListener("error", (event: Event) => { this.fail(new Error((event as ErrorEvent).message || "request-history rebuild worker failed")); @@ -68,16 +69,29 @@ export class RebuildWorkerJob { pending?.reject(error); } - /** Build the complete private file; resolves with the ingested row count. */ - build(indexPath: string, buildPath: string, reason: string, revision: UsageLogRevision): Promise { + private send(request: RebuildWorkerRequest): Promise { if (this.exited || this.disposal) return Promise.reject(new Error("request-history rebuild worker is not running")); if (this.pending) return Promise.reject(new Error("request-history rebuild worker is busy")); return new Promise((resolve, reject) => { this.pending = { resolve, reject }; - this.worker.postMessage({ type: "build", indexPath, buildPath, reason, revision } satisfies RebuildWorkerRequest); + this.worker.postMessage(request); }); } + /** Build the complete private file; resolves with the ingested row count. */ + async build(indexPath: string, buildPath: string, reason: string, revision: UsageLogRevision): Promise { + const reply = await this.send({ type: "build", indexPath, buildPath, reason, revision }); + if (reply.type !== "built") throw new Error("unexpected request-history build reply"); + return reply.rows; + } + + /** Scan a read-only connection off-loop; operational errors are not corruption. */ + async check(indexPath: string): Promise { + const reply = await this.send({ type: "check", indexPath }); + if (reply.type !== "checked") throw new Error("unexpected request-history integrity reply"); + return reply.healthy; + } + /** Stop the thread; waits a bounded time for it to exit. */ dispose(): Promise { this.disposal ??= this.shutDown(); diff --git a/src/routing/history/rebuild-protocol.ts b/src/routing/history/rebuild-protocol.ts index eb5999d29a5..d0dde97af6e 100644 --- a/src/routing/history/rebuild-protocol.ts +++ b/src/routing/history/rebuild-protocol.ts @@ -8,8 +8,9 @@ export type RebuildWorkerRequest = { buildPath: string; reason: string; revision: UsageLogRevision; -}; +} | { type: "check"; indexPath: string }; export type RebuildWorkerReply = | { type: "built"; rows: number } + | { type: "checked"; healthy: boolean } | { type: "error"; name: string; message: string }; diff --git a/src/routing/history/rebuild-worker.ts b/src/routing/history/rebuild-worker.ts index 6ee44b081d8..a636ed4d93e 100644 --- a/src/routing/history/rebuild-worker.ts +++ b/src/routing/history/rebuild-worker.ts @@ -8,12 +8,30 @@ * closes it. The serving thread then swaps the finished file in. */ +import { Database } from "bun:sqlite"; import { buildIndexFile, removeAbandonedBuilds } from "./build-file"; +import { isHistoryIndexCorruption } from "./sqlite-errors"; import type { RebuildWorkerReply, RebuildWorkerRequest } from "./rebuild-protocol"; declare const self: Worker; function handle(request: RebuildWorkerRequest): RebuildWorkerReply { + if (request.type === "check") { + let handle: Database | undefined; + try { + handle = new Database(request.indexPath, { readonly: true }); + handle.exec("PRAGMA busy_timeout = 5000"); + const row = handle.query("PRAGMA quick_check").get() as { quick_check?: string } | undefined; + return { type: "checked", healthy: row?.quick_check === "ok" }; + } catch (error) { + if (isHistoryIndexCorruption(error)) return { type: "checked", healthy: false }; + throw error; + } finally { + // Release the live index before posting a verdict: the serving thread + // may immediately replace it, including on Windows. + handle?.close(); + } + } removeAbandonedBuilds(request.indexPath); const rows = buildIndexFile(request.buildPath, request.reason, request.revision); return { type: "built", rows }; diff --git a/src/routing/history/sqlite-errors.ts b/src/routing/history/sqlite-errors.ts new file mode 100644 index 00000000000..9e8101954d8 --- /dev/null +++ b/src/routing/history/sqlite-errors.ts @@ -0,0 +1,9 @@ +/** Only structural SQLite damage justifies discarding the derived index. */ +export function isHistoryIndexCorruption(error: unknown): boolean { + if (!error || typeof error !== "object") return false; + const { code, errno } = error as { code?: unknown; errno?: unknown }; + // Bun exposes extended result codes too; the low byte is the primary code. + if (typeof errno === "number" && ((errno & 0xff) === 11 || (errno & 0xff) === 26)) return true; + return typeof code === "string" + && /^(SQLITE_CORRUPT(?:_[A-Z]+)?|SQLITE_NOTADB)$/.test(code); +} diff --git a/structure/dashboard-and-usage.md b/structure/dashboard-and-usage.md index 084f60d1157..960bbdc128c 100644 --- a/structure/dashboard-and-usage.md +++ b/structure/dashboard-and-usage.md @@ -276,12 +276,12 @@ first two is the loss signal. Provider debug formats one ring line per finalized counts and writes no second record. `GET /api/usage?failures=1` groups failed rows by a versioned fingerprint over closed vocabularies only, rebuilt through the same cooperative scanner and inheriting its bounds, so deleting a ledger row removes it from the grouping. -`usageLedgerMaxBytes` is unset by default; when set, an append that crosses it publishes the newest -whole rows byte for byte through the shared atomic writer, refuses the rename unless the source is -the exact revision that was copied, and then discards the Logs ring, the retained aggregates and the -request-history index so no surface serves rows the ledger no longer has. That index (`src/routing/history/`) rebuilds a ledger of at least `REQUEST_HISTORY_WORKER_REBUILD_MIN_BYTES` (1 MiB) on a worker thread (#6990): `rebuild-worker.ts`, driven by `rebuild-job.ts`, ingests the ledger snapshot the refresh recorded into a private `.rebuild--` file through the same `ingest.ts` the inline path uses, switches it to WAL and closes it; the serving thread then closes the old index and renames the build over it. Async readers wait for the build; the routing-time sync entry never does and throws `RequestHistoryIndexBusyError`, which health scoring reads as unknown evidence. Health scoring (`src/routing/health.ts`) counts physical sends only: a request answered locally or refused before any send is no evidence for any target, and a row's terminal status and close reason score its last physical send, so a 200 stream cut mid-flight after a failover counts against the target that dropped it (`tests/server/health-scoring.test.ts`). A close discards the build, rejects waiting readers with `RequestHistoryIndexClosedError`, and removes the build files only after the worker thread exits. For 30 s after a worker build fails (other than on a replaced ledger), async readers and manual rebuilds rebuild inline instead, while routing-time reads keep reporting busy. -Ledger revisions in `src/usage/log.ts` capture device/inode identifiers from bigint descriptor stats as exact decimal strings; history metadata, ingestion, cooperative scans and retention compare them without narrowing to Numbers, preserving distinct NTFS identifiers above 2^53. An older index containing a rounded identifier fails the identity check and rebuilds once. `src/usage/summary.ts` turns that file into the `/api/usage` shape — totals, daily zero-filled -grid, model and provider breakdowns, and `measured / reported / unreported / unsupported / estimated` counts. Each scope also aggregates end-to-end output throughput (#6309): measured output tokens and wall-clock `durationMs` are summed over attempts reporting both, exposing `throughputTokensPerSec` — a token-sum-over-duration-sum, never a mean of per-request rates. `throughputSamples` counts qualifying attempts (legacy rows without attempts contribute one); nonpositive or nonfinite tokens/timing are excluded. The Usage summary and Models/Providers tables display the rate and sample basis, or unavailable when none qualify. +`usageLedgerMaxBytes` is unset by default; when set, an append that crosses it publishes the newest whole rows byte for byte through the shared atomic writer, refuses the rename unless the source is the exact revision that was copied, and then discards the Logs ring, the retained aggregates and the request-history index so no surface serves rows the ledger no longer has. +That index (`src/routing/history/`) rebuilds a ledger of at least `REQUEST_HISTORY_WORKER_REBUILD_MIN_BYTES` (1 MiB) on a worker thread (#6990): `rebuild-worker.ts`, driven by `rebuild-job.ts`, ingests the ledger snapshot the refresh recorded into a private `.rebuild--` file through the same `ingest.ts` the inline path uses, switches it to WAL and closes it; the serving thread then closes the old index and renames the build over it. Async readers wait for the build; the routing-time sync entry never does and throws `RequestHistoryIndexBusyError`, which health scoring reads as unknown evidence. Health scoring (`src/routing/health.ts`) counts physical sends only: a request answered locally or refused before any send is no evidence for any target, and a row's terminal status and close reason score its last physical send, so a 200 stream cut mid-flight after a failover counts against the target that dropped it (`tests/server/health-scoring.test.ts`). A close discards the build, rejects waiting readers with `RequestHistoryIndexClosedError`, and removes the build files only after the worker thread exits. For 30 s after a worker build fails (other than on a replaced ledger), async readers and manual rebuilds rebuild inline instead, while routing-time reads keep reporting busy. + +`src/routing/history/integrity.ts` validates an existing persisted index on first use with a read-only worker `PRAGMA quick_check` (#7966); serving refreshes retain schema/source/offset checks without a synchronous integrity scan. Async readers await initial validation; synchronous routing reports busy/unknown meanwhile. A successful fresh inline reconstruction is immediately usable, and a private worker build is integrity-checked before publication. After validation or reconstruction, the next refresh at least five minutes later starts one background check; verified reads remain available during it, and idle indexes start no timer. Operational check failures leave the index intact and delay retry for 30 s; an unverified index remains unavailable. A corrupt verdict or `SQLITE_CORRUPT`/`SQLITE_NOTADB` invalidates the live handle and enters the existing rebuild path; async history and analytics reads retry once after recovery. Close/rebuild discards stale verdicts and drains scanners through actual thread exit before index replacement. Neither validation nor recovery rewrites canonical `usage.jsonl`. `tests/usage/request-history-integrity.test.ts` covers scan scheduling, corruption recovery and lifecycle isolation. + +Ledger revisions in `src/usage/log.ts` capture device/inode identifiers from bigint descriptor stats as exact decimal strings; history metadata, ingestion, cooperative scans and retention compare them without narrowing to Numbers, preserving distinct NTFS identifiers above 2^53. An older index containing a rounded identifier fails the identity check and rebuilds once. `src/usage/summary.ts` turns that file into the `/api/usage` shape — totals, daily zero-filled grid, model and provider breakdowns, and `measured / reported / unreported / unsupported / estimated` counts. Each scope also aggregates end-to-end output throughput (#6309): measured output tokens and wall-clock `durationMs` are summed over attempts reporting both, exposing `throughputTokensPerSec` — a token-sum-over-duration-sum, never a mean of per-request rates. `throughputSamples` counts qualifying attempts (legacy rows without attempts contribute one); nonpositive or nonfinite tokens/timing are excluded. The Usage summary and Models/Providers tables display the rate and sample basis, or unavailable when none qualify. All three throughput displays use the same finite-number guard as their tooltip; malformed or nonfinite rates are unavailable rather than formatted. The management route scans the ledger from its beginning in fixed 1 MiB chunks on a cold rebuild, then retains compact numeric aggregate state and resumes at the last verified LF for diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index 5826dcf9196..e01f098a460 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -1200,7 +1200,7 @@ "fixture-teardown-helper.test.ts": "ci-workflows", "replay-refusal-parity.test.ts": "server", "fixture-windows-runtime-preload.test.ts": "ci-workflows", "repo-hygiene.test.ts": "ci-workflows", "request-decompress.test.ts": "usage", - "request-evidence.test.ts": "usage", "request-history-index.test.ts": "usage", + "request-evidence.test.ts": "usage", "request-history-index.test.ts": "usage", "request-history-integrity.test.ts": "usage", "request-log-attempt-identity.test.ts": "usage", "request-log-conversation.test.ts": "usage", "request-log-estimate-cap.test.ts": "usage", "request-log-jev.test.ts": "usage", "request-log-nonstream.test.ts": "usage", "request-log-protocol-trace.test.ts": "usage", diff --git a/tests/usage/request-history-integrity.test.ts b/tests/usage/request-history-integrity.test.ts new file mode 100644 index 00000000000..3c824e918db --- /dev/null +++ b/tests/usage/request-history-integrity.test.ts @@ -0,0 +1,525 @@ +/** Regression coverage for #7966: integrity scans never run on the serving thread. */ +import { afterEach, beforeEach, describe, expect, spyOn, test } from "bun:test"; +import { Database } from "bun:sqlite"; +import { closeSync, mkdtempSync, openSync, readFileSync, renameSync, writeFileSync, writeSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { + appendUsageEntry, + resetUsageReadCacheForTests, + usageLogPath, + type PersistedUsageEntry, +} from "../../src/usage/log"; +import { + closeRequestHistoryIndex, + openRequestHistoryIndex, + openRequestHistoryIndexSync, + queryRequestHistory, + rebuildRequestHistoryIndex, + requestHistoryDb, + requestHistoryIndexStatus, + requestHistoryRowById, + RequestHistoryIndexBusyError, + RequestHistoryIndexClosedError, + setRequestHistoryIntegrityOptionsForTests, + setRequestHistoryRebuildJobFactoryForTests, + setRequestHistoryWorkerRebuildMinBytesForTests, + withRequestHistoryRead, +} from "../../src/routing/history/indexer"; +import { HISTORY_META_KEYS, HISTORY_SCHEMA_VERSION } from "../../src/routing/history/schema"; +import { RequestHistoryIntegrityMonitor } from "../../src/routing/history/integrity"; +import { isHistoryIndexCorruption } from "../../src/routing/history/sqlite-errors"; +import { computeRoutingAnalytics } from "../../src/routing/analytics"; +import { clearHealthHistoryCacheForTests, healthEvidenceForCandidate } from "../../src/routing/health"; +import { flushConfigDirHardeningForTests } from "../../src/config/paths"; +import { flushWindowsSecretAclReapsBeforeRemoval } from "../../src/lib/windows-secret-acl"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; + +let testDir = ""; +let previousHome: string | undefined; + +function entry(requestId: string, timestamp: number): PersistedUsageEntry { + return { requestId, timestamp, provider: "a", model: "m1", status: 200, durationMs: 10, usageStatus: "reported" }; +} + +beforeEach(() => { + previousHome = process.env.OPENCODEX_HOME; + testDir = mkdtempSync(join(tmpdir(), "ocx-integrity-")); + process.env.OPENCODEX_HOME = testDir; + resetUsageReadCacheForTests(); + clearHealthHistoryCacheForTests(); + closeRequestHistoryIndex(); + setRequestHistoryIntegrityOptionsForTests(undefined); +}); + +afterEach(async () => { + clearHealthHistoryCacheForTests(); + closeRequestHistoryIndex(); + setRequestHistoryIntegrityOptionsForTests(undefined); + setRequestHistoryRebuildJobFactoryForTests(undefined); + setRequestHistoryWorkerRebuildMinBytesForTests(undefined); + await flushConfigDirHardeningForTests(); + await flushWindowsSecretAclReapsBeforeRemoval(testDir); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + removeTreeWithRetry(testDir); +}); + +async function existingIndex(): Promise { + for (let index = 0; index < 4; index++) appendUsageEntry(entry(`req-${index}`, 1000 + index)); + const meta = await rebuildRequestHistoryIndex(); + closeRequestHistoryIndex(); + return meta.dbPath; +} + +interface Check { + path: string; + resolve(healthy: boolean): void; + reject(error: unknown): void; +} + +/** A pending check is controlled by the test; release is immediate once it settles or is disposed. */ +function controlledChecks() { + const checks: Check[] = []; + let clock = 0; + setRequestHistoryIntegrityOptionsForTests({ + now: () => clock, + intervalMs: 1000, + failureCooldownMs: 100, + createJob: () => ({ + check: (path: string) => new Promise((resolve, reject) => { checks.push({ path, resolve, reject }); }), + dispose: async () => {}, + afterExit: async (cleanup: () => void) => { cleanup(); }, + }), + }); + return { checks, advance: (ms: number) => { clock += ms; } }; +} + +async function until(predicate: () => boolean): Promise { + for (let attempt = 0; attempt < 200 && !predicate(); attempt++) await Bun.sleep(0); + expect(predicate()).toBe(true); +} + +async function verifyInitial(checks: Check[]): Promise { + const ready = openRequestHistoryIndex(); + await until(() => checks.length === 1); + checks[0]!.resolve(true); + await ready; +} + +function corruptModelIndex(handle: Database, path: string): void { + const { rootpage } = handle.query("SELECT rootpage FROM sqlite_master WHERE name = 'idx_requests_model'").get() as { rootpage: number }; + const { page_size: pageSize } = handle.query("PRAGMA page_size").get() as { page_size: number }; + expect(rootpage).toBeGreaterThan(1); + const checkpoint = handle.query("PRAGMA wal_checkpoint(TRUNCATE)").get() as { busy: number }; + expect(checkpoint.busy).toBe(0); + // Keep SQLite's header, schema and request table intact, but invalidate a + // separate index B-tree that the default timestamp-ordered page does not visit. + const fd = openSync(path, "r+"); + try { writeSync(fd, Buffer.from([0]), 0, 1, (rootpage - 1) * pageSize); } + finally { closeSync(fd); } + handle.exec("PRAGMA shrink_memory"); +} + +describe("request-history integrity lifecycle (#7966)", () => { + test("real worker checks leave repeated page, row, status and routing reads free of full serving-thread scans", async () => { + await existingIndex(); + const query = spyOn(Database.prototype, "query"); + const prepare = spyOn(Database.prototype, "prepare"); + const exec = spyOn(Database.prototype, "exec"); + try { + for (let repeat = 0; repeat < 4; repeat++) { + const page = await queryRequestHistory({}, undefined, 1); + expect(page.rows.map(row => row.requestId)).toEqual(["req-3"]); + expect((await requestHistoryRowById("req-1"))?.requestId).toBe("req-1"); + expect((await requestHistoryIndexStatus()).indexedRows).toBe(4); + expect(openRequestHistoryIndexSync().indexedRows).toBe(4); + } + appendUsageEntry(entry("tail", 2000)); + expect((await queryRequestHistory({}, undefined, 1)).rows[0]?.requestId).toBe("tail"); + const fullChecks = [...query.mock.calls, ...prepare.mock.calls, ...exec.mock.calls] + .filter(([sql]) => /\bPRAGMA\s+(?:quick_check|integrity_check)\b/i.test(String(sql))); + expect(fullChecks).toEqual([]); + } finally { + query.mockRestore(); + prepare.mockRestore(); + exec.mockRestore(); + } + }); + + test("existing-index validation is single-flight and routing never waits for the first check", async () => { + const path = await existingIndex(); + const { checks } = controlledChecks(); + let finished = false; + const page = queryRequestHistory({}, undefined, 1).then(value => { finished = true; return value; }); + const row = requestHistoryRowById("req-1"); + const status = requestHistoryIndexStatus(); + await until(() => checks.length === 1); + expect(checks[0]!.path).toBe(path); + expect(finished).toBe(false); + expect(() => openRequestHistoryIndexSync()).toThrow(RequestHistoryIndexBusyError); + expect(checks).toHaveLength(1); + checks[0]!.resolve(true); + const [result, found, meta] = await Promise.all([page, row, status]); + expect(result.rows[0]?.requestId).toBe("req-3"); + expect(found?.requestId).toBe("req-1"); + expect(meta.indexedRows).toBe(4); + expect(openRequestHistoryIndexSync().indexedRows).toBe(4); + expect(checks).toHaveLength(1); + }); + + test("periodic checks are single-flight and do not delay a verified index's reads", async () => { + await existingIndex(); + const { checks, advance } = controlledChecks(); + await verifyInitial(checks); + advance(999); + await queryRequestHistory({}, undefined, 1); + expect(checks).toHaveLength(1); + advance(1); + expect((await queryRequestHistory({}, undefined, 1)).rows[0]?.requestId).toBe("req-3"); + await until(() => checks.length === 2); + for (let repeat = 0; repeat < 3; repeat++) { + expect((await requestHistoryIndexStatus()).indexedRows).toBe(4); + expect(openRequestHistoryIndexSync().indexedRows).toBe(4); + } + expect(checks).toHaveLength(2); + checks[1]!.resolve(true); + await Bun.sleep(0); + }); + + test("a transient initial check failure preserves the database and retries only after cooldown", async () => { + await existingIndex(); + const canonical = readFileSync(usageLogPath()); + const { checks, advance } = controlledChecks(); + const first = openRequestHistoryIndex(); + const failure = first.catch((error: unknown) => error); + await until(() => checks.length === 1); + const handle = requestHistoryDb(); + checks[0]!.reject(Object.assign(new Error("database is locked"), { code: "SQLITE_BUSY" })); + expect(await failure).toBeInstanceOf(RequestHistoryIndexBusyError); + for (let repeat = 0; repeat < 3; repeat++) { + await expect(openRequestHistoryIndex()).rejects.toBeInstanceOf(RequestHistoryIndexBusyError); + expect(() => openRequestHistoryIndexSync()).toThrow(RequestHistoryIndexBusyError); + } + expect(requestHistoryDb()).toBe(handle); + expect(checks).toHaveLength(1); + expect(handle.query("SELECT count(*) AS n FROM requests").get()).toEqual({ n: 4 }); + advance(100); + const retry = openRequestHistoryIndex(); + await until(() => checks.length === 2); + checks[1]!.resolve(true); + expect((await retry).indexedRows).toBe(4); + expect(requestHistoryDb()).toBe(handle); + expect(readFileSync(usageLogPath())).toEqual(canonical); + }); + + test("a background check failure leaves verified reads available without a worker retry storm", async () => { + await existingIndex(); + const { checks, advance } = controlledChecks(); + await verifyInitial(checks); + const handle = requestHistoryDb(); + advance(1000); + await openRequestHistoryIndex(); + await until(() => checks.length === 2); + checks[1]!.reject(new Error("worker exited")); + await Bun.sleep(0); + for (let repeat = 0; repeat < 3; repeat++) expect((await openRequestHistoryIndex()).indexedRows).toBe(4); + expect(checks).toHaveLength(2); + expect(requestHistoryDb()).toBe(handle); + advance(100); + await openRequestHistoryIndex(); + await until(() => checks.length === 3); + checks[2]!.resolve(true); + await Bun.sleep(0); + }); + + test("a late corrupt verdict from a closed lifetime cannot rebuild the reopened index", async () => { + await existingIndex(); + const { checks } = controlledChecks(); + const first = openRequestHistoryIndex(); + const closed = first.catch((error: unknown) => error); + await until(() => checks.length === 1); + closeRequestHistoryIndex(); + expect(await closed).toBeInstanceOf(RequestHistoryIndexClosedError); + const next = openRequestHistoryIndex(); + await until(() => checks.length === 2); + const reopened = requestHistoryDb(); + checks[0]!.resolve(false); + checks[1]!.resolve(true); + expect((await next).indexedRows).toBe(4); + expect(requestHistoryDb()).toBe(reopened); + expect((await queryRequestHistory({}, undefined, 10)).rows).toHaveLength(4); + expect(checks).toHaveLength(2); + }); + + test("a manual rebuild shares one flight and releases readers waiting on an initial scan", async () => { + await existingIndex(); + const { checks } = controlledChecks(); + const reader = openRequestHistoryIndex(); + await until(() => checks.length === 1); + setRequestHistoryWorkerRebuildMinBytesForTests(0); + const rebuild = rebuildRequestHistoryIndex(); + expect(rebuildRequestHistoryIndex()).toBe(rebuild); + expect(() => openRequestHistoryIndexSync()).toThrow(RequestHistoryIndexBusyError); + const [readMeta, rebuildMeta] = await Promise.all([reader, rebuild]); + expect(readMeta.indexedRows).toBe(4); + expect(rebuildMeta.indexedRows).toBe(4); + checks[0]!.resolve(false); + await Bun.sleep(0); + expect(openRequestHistoryIndexSync().indexedRows).toBe(4); + expect(checks).toHaveLength(1); + }); + + test("close prevents a manual rebuild waiting on an existing worker from reopening the index", async () => { + appendUsageEntry(entry("only", 1000)); + setRequestHistoryWorkerRebuildMinBytesForTests(0); + let builds = 0; + let rejectBuild: ((error: unknown) => void) | undefined; + setRequestHistoryRebuildJobFactoryForTests(() => ({ + build: () => new Promise((_resolve, reject) => { builds += 1; rejectBuild = reject; }), + dispose: async () => { rejectBuild?.(new Error("worker stopped")); }, + afterExit: async (cleanup: () => void) => { cleanup(); }, + })); + const reader = openRequestHistoryIndex().catch((error: unknown) => error); + await until(() => builds === 1); + const manual = rebuildRequestHistoryIndex().catch((error: unknown) => error); + closeRequestHistoryIndex(); + expect(await reader).toBeInstanceOf(RequestHistoryIndexClosedError); + expect(await manual).toBeInstanceOf(RequestHistoryIndexClosedError); + expect(builds).toBe(1); + expect(() => requestHistoryDb()).toThrow("not open"); + }); + + test("source replacement cancels a background scan and reprojects only the replacement ledger", async () => { + await existingIndex(); + const { checks, advance } = controlledChecks(); + await verifyInitial(checks); + advance(1000); + await openRequestHistoryIndex(); + await until(() => checks.length === 2); + const path = usageLogPath(); + const replacement = JSON.stringify(entry("replacement", 5000)) + "\n"; + writeFileSync(`${path}.next`, replacement); + renameSync(`${path}.next`, path); + const page = await queryRequestHistory({}, undefined, 10); + expect(page.rows.map(row => row.requestId)).toEqual(["replacement"]); + const rebuilt = requestHistoryDb(); + checks[1]!.resolve(false); + await Bun.sleep(0); + expect((await requestHistoryIndexStatus()).indexedRows).toBe(1); + expect(requestHistoryDb()).toBe(rebuilt); + expect(readFileSync(path, "utf8")).toBe(replacement); + expect(checks).toHaveLength(2); + }); + + test("a periodic corrupt verdict rebuilds the projection on the next read", async () => { + await existingIndex(); + const canonical = readFileSync(usageLogPath()); + const { checks, advance } = controlledChecks(); + await verifyInitial(checks); + const handle = requestHistoryDb(); + advance(1000); + await openRequestHistoryIndex(); + await until(() => checks.length === 2); + checks[1]!.resolve(false); + await Bun.sleep(0); + expect((await queryRequestHistory({}, undefined, 10)).rows).toHaveLength(4); + expect(requestHistoryDb()).not.toBe(handle); + expect(checks).toHaveLength(2); + expect(readFileSync(usageLogPath())).toEqual(canonical); + }); + + test("worker spawn failures observe cooldown and never delete an unverified index", async () => { + await existingIndex(); + let clock = 0; + let spawns = 0; + setRequestHistoryIntegrityOptionsForTests({ + now: () => clock, + failureCooldownMs: 100, + createJob: () => { spawns += 1; throw new Error("worker unavailable"); }, + }); + await expect(openRequestHistoryIndex()).rejects.toBeInstanceOf(RequestHistoryIndexBusyError); + const handle = requestHistoryDb(); + await expect(openRequestHistoryIndex()).rejects.toBeInstanceOf(RequestHistoryIndexBusyError); + expect(spawns).toBe(1); + clock = 100; + await expect(openRequestHistoryIndex()).rejects.toBeInstanceOf(RequestHistoryIndexBusyError); + expect(spawns).toBe(2); + expect(requestHistoryDb()).toBe(handle); + expect(handle.query("SELECT count(*) AS n FROM requests").get()).toEqual({ n: 4 }); + }); + + test("the real worker detects an unqueried damaged B-tree and rebuilds from untouched canonical rows", async () => { + const path = await existingIndex(); + const canonical = readFileSync(usageLogPath()); + const damaged = new Database(path); + try { + corruptModelIndex(damaged, path); + expect(damaged.query("SELECT value FROM schema_meta WHERE key = ?").get(HISTORY_META_KEYS.schemaVersion)) + .toEqual({ value: String(HISTORY_SCHEMA_VERSION) }); + expect(damaged.query("SELECT request_id FROM requests ORDER BY timestamp DESC, request_id DESC LIMIT 1").get()) + .toEqual({ request_id: "req-3" }); + expect(() => damaged.query("SELECT request_id FROM requests INDEXED BY idx_requests_model WHERE model = ?").all("m1")) + .toThrow(); + } finally { damaged.close(); } + setRequestHistoryWorkerRebuildMinBytesForTests(0); + const repaired = await queryRequestHistory({ model: "m1" }, undefined, 10); + expect(repaired.rows.map(row => row.requestId)).toEqual(["req-3", "req-2", "req-1", "req-0"]); + expect(repaired.meta.lastError).toMatch(/rebuilt|integrity|corrupt/i); + expect(readFileSync(usageLogPath())).toEqual(canonical); + }); + + test("corruption encountered by a live query rebuilds once even before the next periodic check", async () => { + const path = await existingIndex(); + await openRequestHistoryIndex(); + const original = requestHistoryDb(); + const canonical = readFileSync(usageLogPath()); + corruptModelIndex(original, path); + let reads = 0; + const rows = await withRequestHistoryRead(handle => { + reads += 1; + return handle.query("SELECT request_id FROM requests INDEXED BY idx_requests_model WHERE model = ? ORDER BY request_id").all("m1"); + }); + expect(reads).toBe(2); + expect(rows).toEqual([0, 1, 2, 3].map(index => ({ request_id: `req-${index}` }))); + expect(requestHistoryDb()).not.toBe(original); + expect(readFileSync(usageLogPath())).toEqual(canonical); + }); + + test("routing analytics recovers a corrupted model index through its read wrapper", async () => { + const path = await existingIndex(); + await openRequestHistoryIndex(); + const handle = requestHistoryDb(); + const canonical = readFileSync(usageLogPath()); + corruptModelIndex(handle, path); + const analytics = await computeRoutingAnalytics({ model: "m1" }); + expect(analytics.totalRequests).toBe(4); + expect(analytics.successRate).toBe(1); + expect(requestHistoryDb()).not.toBe(handle); + expect(readFileSync(usageLogPath())).toEqual(canonical); + }); + + test("routing health marks query corruption for repair and recovers current historical evidence", async () => { + const path = await existingIndex(); + const now = Date.now(); + appendUsageEntry(entry("recent", now)); + await openRequestHistoryIndex(); + const input = { provider: "a", model: "m1", now }; + expect(healthEvidenceForCandidate(input).sampleCount).toBe(1); + clearHealthHistoryCacheForTests(); + const handle = requestHistoryDb(); + const canonical = readFileSync(usageLogPath()); + corruptModelIndex(handle, path); + expect(healthEvidenceForCandidate(input)).toEqual({}); + expect((await openRequestHistoryIndex()).indexedRows).toBe(5); + expect(requestHistoryDb()).not.toBe(handle); + clearHealthHistoryCacheForTests(); + const repaired = healthEvidenceForCandidate(input); + expect(repaired.sampleCount).toBe(1); + expect(repaired.successRate).toBe(1); + expect(readFileSync(usageLogPath())).toEqual(canonical); + }); + + test("tail ingestion recovers corruption in a secondary index before inserting the appended row", async () => { + const path = await existingIndex(); + await openRequestHistoryIndex(); + const handle = requestHistoryDb(); + corruptModelIndex(handle, path); + appendUsageEntry(entry("after-corruption", 2000)); + const canonical = readFileSync(usageLogPath()); + const page = await queryRequestHistory({ model: "m1" }, undefined, 10); + expect(page.rows.map(row => row.requestId)).toEqual(["after-corruption", "req-3", "req-2", "req-1", "req-0"]); + expect(page.meta.indexedRows).toBe(5); + expect(requestHistoryDb()).not.toBe(handle); + expect(readFileSync(usageLogPath())).toEqual(canonical); + }); + + test("query retries are bounded and ordinary SQLite lock errors do not recreate the index", async () => { + await existingIndex(); + await openRequestHistoryIndex(); + const handle = requestHistoryDb(); + const busy = Object.assign(new Error("database is locked"), { code: "SQLITE_BUSY" }); + let reads = 0; + await expect(withRequestHistoryRead(() => { reads += 1; throw busy; })).rejects.toBe(busy); + expect(reads).toBe(1); + expect(requestHistoryDb()).toBe(handle); + const corrupt = Object.assign(new Error("database disk image is malformed"), { code: "SQLITE_CORRUPT" }); + reads = 0; + await expect(withRequestHistoryRead(() => { reads += 1; throw corrupt; })).rejects.toBe(corrupt); + expect(reads).toBe(2); + }); + + test("extended SQLite corruption codes are repairable while operational failures are not", () => { + for (const error of [ + { code: "SQLITE_CORRUPT" }, { code: "SQLITE_CORRUPT_INDEX" }, { code: "SQLITE_NOTADB" }, + { errno: 11 }, { errno: 267 }, { errno: 26 }, + ]) expect(isHistoryIndexCorruption(error)).toBe(true); + for (const error of [ + { code: "SQLITE_BUSY", errno: 5 }, { code: "SQLITE_LOCKED", errno: 6 }, + { code: "SQLITE_IOERR", errno: 10 }, { code: "SQLITE_FULL", errno: 13 }, + { code: "SQLITE_CANTOPEN", errno: 14 }, new Error("database disk image is malformed"), null, + ]) expect(isHistoryIndexCorruption(error)).toBe(false); + }); + + test("a cancelled check holds the replacement barrier until its actual exit callback", async () => { + const checks: Check[] = []; + const exits: Array<() => void> = []; + const monitor = new RequestHistoryIntegrityMonitor({ + createJob: () => ({ + check: (path: string) => new Promise((resolve, reject) => { checks.push({ path, resolve, reject }); }), + dispose: async () => {}, + // Simulate bounded disposal returning while the native thread is still alive. + afterExit: async (cleanup: () => void) => { exits.push(cleanup); }, + }), + }); + const first = monitor.inspect({}, "index.sqlite"); + if (first.state !== "pending") throw new Error("initial check must be pending"); + await until(() => checks.length === 1); + monitor.reset(); + const replacement = {}; + const blocked = monitor.inspect(replacement, "index.sqlite"); + expect(blocked.state).toBe("pending"); + await until(() => exits.length === 1); + let released = false; + void first.promise.then(() => { released = true; }); + checks[0]!.resolve(false); + await Bun.sleep(0); + expect(released).toBe(false); + expect(checks).toHaveLength(1); + exits[0]!(); + await first.promise; + const next = monitor.inspect(replacement, "index.sqlite"); + if (next.state !== "pending") throw new Error("replacement needs its own check"); + await until(() => checks.length === 2); + checks[1]!.resolve(true); + await until(() => exits.length === 2); + exits[1]!(); + await next.promise; + expect(monitor.inspect(replacement, "index.sqlite").state).toBe("ready"); + }); + + test("a delayed healthy check cannot clear corruption reported by a statement", async () => { + let clock = 0; + let finish: ((healthy: boolean) => void) | undefined; + let released = false; + const monitor = new RequestHistoryIntegrityMonitor({ + now: () => clock, + intervalMs: 100, + createJob: () => ({ + check: () => new Promise(resolve => { finish = resolve; }), + dispose: async () => {}, + afterExit: async (cleanup: () => void) => { released = true; cleanup(); }, + }), + }); + const handle = {}; + monitor.acceptRebuilt(handle, "index.sqlite"); + clock = 100; + expect(monitor.inspect(handle, "index.sqlite").state).toBe("ready"); + await until(() => finish !== undefined); + monitor.markCorrupt(handle); + finish!(true); + await until(() => released); + await Bun.sleep(0); + expect(monitor.inspect(handle, "index.sqlite").state).toBe("corrupt"); + }); +}); From 0cc5a5958d5435444a0185bf06e424843916716f Mon Sep 17 00:00:00 2001 From: Robin Bially <7304732+robin-bially@users.noreply.github.com> Date: Fri, 30 Oct 2026 12:31:22 +0100 Subject: [PATCH 07/42] fix(workflow-budget): make per-task ceilings configurable from config and dashboard (#7960) * fix(workflow-budget): make per-task ceilings configurable from config and dashboard The root-workflow count ceilings (8 concurrent turns, 256 sends and 64 distinct children per 10-minute window) were module constants. An install that legitimately runs more parallel subagents had no way to raise them short of patching the package, and every hit surfaced as a local 429. Add an optional workflowBudget config block for the three count ceilings, bound to the live config and read per decision by HTTP admission and the pre-dispatch send ceiling. Absent block keeps today's policy object unchanged. Window, interactive reserve and tracked-root bound stay fixed. GET/PUT /api/workflow-budget/settings expose effective values, defaults and bounds; the Subagents page gets a panel to edit them. * docs(workflow-budget): record deferred live settings CLI --------- Co-authored-by: Robin Bially --- .../030_wfc_diff_plan.md | 7 + .../src/content/docs/guides/web-dashboard.md | 2 +- .../docs/reference/configuration/server.md | 24 +++ .../content/docs/reference/management-api.md | 1 + gui/src/components/WorkflowBudgetPanel.tsx | 178 ++++++++++++++++++ gui/src/i18n/de.ts | 16 ++ gui/src/i18n/en.ts | 16 ++ gui/src/i18n/fr.ts | 16 ++ gui/src/i18n/ja.ts | 16 ++ gui/src/i18n/ko.ts | 16 ++ gui/src/i18n/pt.ts | 16 ++ gui/src/i18n/ru.ts | 16 ++ gui/src/i18n/tr.ts | 16 ++ gui/src/i18n/vi.ts | 16 ++ gui/src/i18n/zh-TW.ts | 16 ++ gui/src/i18n/zh.ts | 16 ++ gui/src/pages/Subagents.tsx | 2 + gui/tests/workflow-budget-panel.test.tsx | 151 +++++++++++++++ scripts/test-layout/layout.json | 2 +- src/config/diagnostics.ts | 3 + src/config/load-degrade.ts | 2 + src/config/schema/config-schema.ts | 3 + src/config/schema/workflow-budget.ts | 43 +++++ src/lib/state-store-registrations.ts | 2 + src/lib/workflow-budget-config.ts | 39 ++++ src/server/management/route-registry.ts | 2 + .../management/workflow-budget-routes.ts | 73 ++++++- src/server/responses/request-send-budget.ts | 3 +- src/server/workflow-refusal.ts | 3 +- src/types/config.ts | 2 + structure/gui-and-management-api.md | 2 +- tests/cli/cli-headless-parity.test.ts | 1 + tests/config/config-workflow-budget.test.ts | 56 ++++++ tests/fixtures/test-layout-expected.json | 2 +- .../server-workflow-budget-settings.test.ts | 150 +++++++++++++++ 35 files changed, 920 insertions(+), 9 deletions(-) create mode 100644 gui/src/components/WorkflowBudgetPanel.tsx create mode 100644 gui/tests/workflow-budget-panel.test.tsx create mode 100644 src/config/schema/workflow-budget.ts create mode 100644 src/lib/workflow-budget-config.ts create mode 100644 tests/config/config-workflow-budget.test.ts create mode 100644 tests/server/server-workflow-budget-settings.test.ts diff --git a/devlog/_fin/260915_workflow_budget_window/030_wfc_diff_plan.md b/devlog/_fin/260915_workflow_budget_window/030_wfc_diff_plan.md index 7b4617b716a..b2d2c7b6905 100644 --- a/devlog/_fin/260915_workflow_budget_window/030_wfc_diff_plan.md +++ b/devlog/_fin/260915_workflow_budget_window/030_wfc_diff_plan.md @@ -96,6 +96,13 @@ declared `deferred-verb` in the route registry against this document, which is what keeps them out of the undeclared-route ratchet without pretending the gap does not exist. +The later configurable-ceilings change also defers the live CLI counterpart of +`PUT /api/workflow-budget/settings` against this document. Its management API +updates the running policy; local `ocx config set/unset workflowBudget...` +edits disk configuration and requires a proxy restart for live convergence. +`GET /api/workflow-budget/settings` is a `local-transport` exemption because +`ocx config get` can read the same configuration block locally. + ## Verification posture Local suite, typecheck, install and build: NOT RUN, by standing instruction. diff --git a/docs-site/src/content/docs/guides/web-dashboard.md b/docs-site/src/content/docs/guides/web-dashboard.md index 677bd9e46d4..7426627eeab 100644 --- a/docs-site/src/content/docs/guides/web-dashboard.md +++ b/docs-site/src/content/docs/guides/web-dashboard.md @@ -131,7 +131,7 @@ is visible) and never forces an upstream refresh. | **Providers** | Add, edit, set the default (enabled providers only), enable/disable, and remove providers; manage OAuth account pools and API-key pools where supported. Removing the current default switches to the first remaining enabled provider when one exists; otherwise deletion is refused and the current default is kept. Provider Settings can disable live model discovery for endpoints with missing, slow, or oversized `/models` catalogs. For Claude (Anthropic) OAuth pools, each logged-in account shows its own 5-hour and weekly rate-limit bars (usage is per credential); a failed probe keeps the last-known bars and marks them unavailable until the next successful refresh. Refreshing quota skips paused OAuth accounts and disabled API keys; skipped rows without a cached reading remain unavailable but do not affect the refresh result. Refreshing reports failure when an unpaused OAuth account or enabled API key that supports quota reads has an unavailable reading. The Provider Overview shown when no provider is selected carries a **Refresh all quotas** control that forces one server-side re-read of every configured provider; a provider whose upstream probe fails keeps its last-good row, so the status line reports that the check completed rather than claiming every value is fresh, and each row's own age stays the per-provider freshness signal. | | **Add provider** | One search above the tabs reaches all four at once — Accounts, Free, Local, Paid. While a query is live the results are grouped by tab with a count each, and the selected tab stays put rather than jumping. Press ArrowDown in the search box to focus the first available result action, skipping disabled buttons; if no action is available, focus stays in the search box. Local runtimes (Ollama, vLLM, LM Studio, LiteLLM) have their own tab, and a long provider note clamps to two lines with the full text one click away. Claude Code CLI (subscription) sits under Paid with a Subscription CLI badge, and selecting it warns that it bills the signed-in Claude subscription through `claude -p` and never uses an API key (use `anthropic-apikey` for key billing). | | **Codex Auth** | Add ChatGPT/Codex pool accounts, select the next-session account, refresh 5h / weekly / 30d quotas, enable or disable quota auto-switch, set its 1–100% threshold, and configure transient-failure failover. | -| **Subagents** | Feature up to five bare native or namespaced routed models in the `spawn_agent` override list. | +| **Subagents** | Feature up to five bare native or namespaced routed models in the `spawn_agent` override list. The **Workflow limits for subagents** panel sets how many turns one task may run in parallel and how many model requests and distinct subagents it may use per 10-minute window. See [workflow limits](/reference/configuration/server/#workflow-limits). | | **Models** | Toggle native GPT and routed models, set provider allowlists and context caps, choose v1/base/v2, and configure the v2 thread limit. The page distinguishes a catalog saved on the hub, a catalog fetched by this client, and activation in a running client. A fetch timestamp does not prove it includes the latest hub save, and runtime activation is shown as unverified. Configured providers stay visible as zero-model groups when discovery is off or returns no rows. | | **Logs** | Auto-refresh recent requests with tokens, requested effort and (when available) effective outbound effort, resolved model, provider, status, request id, duration, and error details. The detail view includes the exact reasoning wire field when the adapter emits one. Filter by opaque conversation/session id (when the client sends one) to total tokens and estimated list-price cost for the currently loaded Logs ring. | | **Usage / Debug** | Inspect token-usage coverage and trends. The Usage page's Models table also breaks each model down into input tokens, output tokens, cache hits, cache writes, and cache hit rate; a dash means cache telemetry for that metric is unavailable. Or enable opt-in provider transport and usage-extraction diagnostics. | diff --git a/docs-site/src/content/docs/reference/configuration/server.md b/docs-site/src/content/docs/reference/configuration/server.md index bff693deae6..59aa34c7602 100644 --- a/docs-site/src/content/docs/reference/configuration/server.md +++ b/docs-site/src/content/docs/reference/configuration/server.md @@ -34,6 +34,7 @@ passthrough, including native WebSocket output, retain their existing stop behav | `codexNativeSteering?` | `boolean` | `false` | Experimental, native-only mid-turn steering on the Responses WebSocket endpoint. Requires `websockets: true`, a compatible upstream/client, and a pinned account/model/tool surface. Validated generation settings can change in explicit saved-result continuations. Does not enable translated models or HTTP fallback. See [native steering](/guides/codex-integration/#experimental-native-mid-turn-steering). | | `codexNativeInjection?` | `boolean` | `false` | Experimental saved function-result injection on compatible native multi-agent WebSocket turns. Requires `websockets: true`, explicit `multi_agent.enabled`, and an eligible provider. Separate from steering; no automatic tool rerun or recovery create. See [native injection](/guides/codex-integration/#experimental-native-function-result-injection). | | `nativeReasoningRetention?` | `{ modelSwitch?: boolean; accountSwitch?: boolean }` | both `false` | Permit reasoning ciphertext forwarding across a known model or account change on the same canonical ChatGPT backend. Account retention is experimental. See [Native reasoning retention](#native-reasoning-retention). | +| `workflowBudget?` | `{ maxConcurrentChildren?: number; maxPhysicalSends?: number; maxDistinctChildren?: number }` | `8` / `256` / `64` | Per-task fan-out ceilings: turns running at once, upstream sends per 10-minute window, and distinct subagent threads per window. Omitted fields keep the built-in value. See [Workflow limits](#workflow-limits). | | `corsAllowOrigins?` | `string[]` | `[]` | Additional exact origins allowed by CORS. Loopback origins are always allowed. Authority-based browser extension origins such as `chrome-extension://` are supported; `*` is not a wildcard. Firefox and Safari regenerate the extension UUID (per install / per browser launch), so update the entry when the origin changes. | | `apiKeys?` | `OcxApiKey[]` | `[]` | Generated `ocx_…` data-plane admission credentials on non-loopback binds. They do not authorize management APIs; management access uses the separate credential documented in the [management reference](/reference/management-api/). Dashboard-managed. | | `storageCleanupPolicy?` | `StorageCleanupPolicy` | disabled | Opt-in archived-session cleanup policy. Never enabled implicitly. | @@ -87,6 +88,29 @@ Retention means that OpenCodex may forward the carried ciphertext. It cannot con Use `GET /api/native-reasoning-retention` to read the resolved booleans and `PUT` with a partial boolean object to update them. `PUT` with JSON `null` restores both defaults. Invalid fields or unknown keys are rejected; a malformed file block disables retention with a warning. Dashboard/API changes apply to subsequent requests. Local `ocx config set/unset` edits require a restart to converge the running proxy. +## Workflow limits + +OpenCodex bounds how much one task may fan out. A task is the root conversation, identified by the client's `x-codex-parent-thread-id` header, together with every subagent it starts. Three count ceilings apply to each task, and requests over any of them are refused locally with HTTP 429 and an `x-opencodex-local-refusal` header naming the ceiling (`workflow_concurrency_exhausted`, `workflow_sends_exhausted` or `workflow_children_exhausted`); no provider is contacted. + +| Field | Default | Allowed | Meaning | +| --- | --- | --- | --- | +| `maxConcurrentChildren` | `8` | 2–64 | Turns one task may run at the same time. One slot is always reserved for the task's own conversation, so subagents may use one fewer. | +| `maxPhysicalSends` | `256` | 1–10000 | Upstream sends, retries included, one task may make inside the 10-minute window. | +| `maxDistinctChildren` | `64` | 1–1024 | Distinct subagent threads one task may start inside the same window. | + +The defaults suit most installs. Raise them when a workflow that legitimately runs many subagents in parallel keeps hitting a ceiling; lower them to stop a runaway fan-out sooner: + +```jsonc +{ + "workflowBudget": { + "maxConcurrentChildren": 16, + "maxPhysicalSends": 1024 + } +} +``` + +The **Subagents** page of the dashboard shows the effective ceilings and edits them. `GET /api/workflow-budget/settings` returns the effective values with their defaults and allowed ranges; `PUT` with a partial object updates them, and JSON `null` restores the defaults. Dashboard/API changes apply to the next request without a restart and never reset what a task has already used; `POST /api/workflow-budget/clear` remains the way to forgive one task. A malformed file block is ignored with a warning and the built-in ceilings stay in force. Local `ocx config set/unset` edits require a restart to converge the running proxy. The 10-minute window, the reserved interactive slot and the 512 tracked tasks are not configurable. + ## Upstream transport recovery While the canonical ChatGPT upstream WebSocket waits for the first Responses event after diff --git a/docs-site/src/content/docs/reference/management-api.md b/docs-site/src/content/docs/reference/management-api.md index 7da2cada27a..07b78c54e08 100644 --- a/docs-site/src/content/docs/reference/management-api.md +++ b/docs-site/src/content/docs/reference/management-api.md @@ -271,6 +271,7 @@ See [Codex prompt layers](/guides/codex-prompt/) for the layer model and the key | `PUT /api/config` | Disabled full-config replacement guard | 405; use focused endpoints instead | | `GET, PUT /api/settings` | Read runtime/startup settings or update auto-start, stream mode, app-owned memory budget, and `codexAccountPickerEnabled` | 400 invalid, non-object, or empty update | | `GET, PUT /api/native-reasoning-retention` | Read resolved model/account reasoning-retention booleans; partial PUT preserves omitted fields, JSON `null` restores defaults. See [native reasoning retention](/reference/configuration/server/#native-reasoning-retention). | 400 unknown field, invalid type or non-object/non-null body | +| `GET, PUT /api/workflow-budget/settings` | Read the effective per-task workflow ceilings with defaults and allowed ranges; partial PUT preserves omitted fields, JSON `null` restores defaults. See [workflow limits](/reference/configuration/server/#workflow-limits). | 400 unknown field, non-integer or out-of-range value, or non-object/non-null body | | `GET /api/startup-health` | Read cached service/shim startup health | — | | `POST /api/startup-action` | Install or repair the service or Codex shim | 400 invalid action; 500 action failure | | `GET, POST /api/windows-tray` | Read Windows tray state or install/start/stop/uninstall it | 400 unsupported platform/action; 500 operation failure | diff --git a/gui/src/components/WorkflowBudgetPanel.tsx b/gui/src/components/WorkflowBudgetPanel.tsx new file mode 100644 index 00000000000..e8d7cb29e25 --- /dev/null +++ b/gui/src/components/WorkflowBudgetPanel.tsx @@ -0,0 +1,178 @@ +import { useCallback, useEffect, useRef, useState } from "react"; +import { useT } from "../i18n/shared"; +import { createBoundedFetch } from "../bounded-fetch"; +import { clampNumberDraft } from "../clamp-draft"; +import { requireJson } from "../pages/dashboard-shared"; +import { NumberStepper } from "./NumberStepper"; + +const KEYS = ["maxConcurrentChildren", "maxPhysicalSends", "maxDistinctChildren"] as const; +type Key = typeof KEYS[number]; +type Values = Record; +type Drafts = Record; +interface Settings { + values: Values; + defaults: Values; + bounds: Record; + windowMinutes: number; +} + +const LABEL_KEYS = { + maxConcurrentChildren: ["workflowBudget.maxConcurrentChildren", "workflowBudget.maxConcurrentChildrenHint"], + maxPhysicalSends: ["workflowBudget.maxPhysicalSends", "workflowBudget.maxPhysicalSendsHint"], + maxDistinctChildren: ["workflowBudget.maxDistinctChildren", "workflowBudget.maxDistinctChildrenHint"], +} as const; + +const isInt = (value: unknown): value is number => typeof value === "number" && Number.isInteger(value); + +function readSettings(payload: unknown): Settings { + if (!payload || typeof payload !== "object" || Array.isArray(payload)) throw new Error("Invalid workflow budget settings"); + const value = payload as Record; + const defaults = value.defaults as Record | undefined; + const bounds = value.bounds as Record | undefined; + if (!defaults || !bounds || !isInt(value.windowMs)) throw new Error("Missing workflow budget settings"); + const settings = { values: {}, defaults: {}, bounds: {}, windowMinutes: Math.round(value.windowMs / 60_000) } as Settings; + for (const key of KEYS) { + const range = bounds[key]; + if (!isInt(value[key]) || !isInt(defaults[key]) || !range || !isInt(range.min) || !isInt(range.max)) { + throw new Error("Missing workflow budget settings"); + } + settings.values[key] = value[key]; + settings.defaults[key] = defaults[key]; + settings.bounds[key] = { min: range.min, max: range.max }; + } + return settings; +} + +const toDrafts = (values: Values): Drafts => + ({ maxConcurrentChildren: String(values.maxConcurrentChildren), maxPhysicalSends: String(values.maxPhysicalSends), maxDistinctChildren: String(values.maxDistinctChildren) }); + +function parseDraft(raw: string, range: { min: number; max: number }): number | undefined { + if (!/^\d+$/.test(raw.trim())) return undefined; + const value = Number(raw.trim()); + return value >= range.min && value <= range.max ? value : undefined; +} + +export default function WorkflowBudgetPanel({ apiBase }: { apiBase: string }) { + return ; +} + +function WorkflowBudgetControls({ apiBase }: { apiBase: string }) { + const t = useT(); + const [saved, setSaved] = useState(); + const [drafts, setDrafts] = useState({ maxConcurrentChildren: "", maxPhysicalSends: "", maxDistinctChildren: "" }); + const [busy, setBusy] = useState(false); + const [loadError, setLoadError] = useState(false); + const [feedback, setFeedback] = useState<"saved" | "failed" | null>(null); + const active = useRef(false); + const pending = useRef | null>(null); + const accept = useCallback((value: Settings) => { setSaved(value); setDrafts(toDrafts(value.values)); }, []); + const load = useCallback(async () => { + if (pending.current) return; + const request = createBoundedFetch(15_000); + pending.current = request; setLoadError(false); + try { + const response = await fetch(apiBase + "/api/workflow-budget/settings", { signal: request.signal }); + const value = readSettings(await requireJson(response)); + if (active.current && pending.current === request) accept(value); + } catch { + if (active.current && pending.current === request) setLoadError(true); + } finally { + request.clear(); + if (pending.current === request) pending.current = null; + } + }, [apiBase, accept]); + useEffect(() => { + active.current = true; + const timer = window.setTimeout(() => { void load(); }, 0); + return () => { + window.clearTimeout(timer); active.current = false; + pending.current?.controller.abort(); pending.current?.clear(); pending.current = null; + }; + }, [load]); + + const parsed = saved + ? Object.fromEntries(KEYS.map(key => [key, parseDraft(drafts[key], saved.bounds[key])])) as Record + : undefined; + const valid = parsed !== undefined && KEYS.every(key => parsed[key] !== undefined); + const changed = saved !== undefined && parsed !== undefined && KEYS.some(key => parsed[key] !== saved.values[key]); + const atDefaults = saved !== undefined && KEYS.every(key => saved.values[key] === saved.defaults[key]); + + const save = async (reset = false) => { + if (pending.current || saved === undefined || (!reset && (!valid || parsed === undefined))) return; + // Values equal to the defaults are written as a reset, so the install keeps following the + // built-in ceilings instead of pinning today's numbers. + const body = reset || KEYS.every(key => parsed?.[key] === saved.defaults[key]) + ? null + : Object.fromEntries(KEYS.map(key => [key, parsed?.[key]])); + const request = createBoundedFetch(15_000); + pending.current = request; setBusy(true); setFeedback(null); + try { + const response = await fetch(apiBase + "/api/workflow-budget/settings", { + method: "PUT", headers: { "Content-Type": "application/json" }, signal: request.signal, + body: JSON.stringify(body), + }); + const value = readSettings(await requireJson(response)); + if (active.current && pending.current === request) { accept(value); setFeedback("saved"); } + } catch { + if (active.current && pending.current === request) setFeedback("failed"); + } finally { + request.clear(); + if (active.current && pending.current === request) setBusy(false); + if (pending.current === request) pending.current = null; + } + }; + const disabled = busy || saved === undefined || loadError; + const edit = (key: Key, value: string) => { setDrafts(current => ({ ...current, [key]: value })); setFeedback(null); }; + + return
+
{t("workflowBudget.title")}
+

{t("workflowBudget.description", { minutes: saved?.windowMinutes ?? 10 })}

+
+ {KEYS.map(key => { + const [labelKey, hintKey] = LABEL_KEYS[key]; + const label = t(labelKey); + const range = saved?.bounds[key] ?? { min: 0, max: 0 }; + const invalid = saved !== undefined && parsed?.[key] === undefined; + const hintId = "workflow-budget-" + key + "-hint"; + return
+
+ {label} + + {t(hintKey)}{" "} + {saved && t("workflowBudget.range", { default: saved.defaults[key], min: range.min, max: range.max })} + + {invalid && {t("workflowBudget.invalid", { min: range.min, max: range.max })}} +
+ + edit(key, event.target.value)} + onKeyDown={event => { + if (event.key === "Enter" && !event.nativeEvent.isComposing && !disabled && changed && valid) { + event.preventDefault(); void save(); + } + }} + /> + edit(key, clampNumberDraft(drafts[key], 1, range.min, range.max))} + onDecrement={() => edit(key, clampNumberDraft(drafts[key], -1, range.min, range.max))} + /> + +
; + })} +
+ + +
+
+ {loadError &&
{t("workflowBudget.loadFailed")}
} + {feedback === "failed" &&
{t("workflowBudget.saveFailed")}
} + {feedback === "saved" &&
{t("workflowBudget.saved")}
} +
; +} diff --git a/gui/src/i18n/de.ts b/gui/src/i18n/de.ts index fffab7207a4..affe092dcec 100644 --- a/gui/src/i18n/de.ts +++ b/gui/src/i18n/de.ts @@ -105,6 +105,22 @@ export const de: Record = { "nativeReasoningRetention.loadFailed": "Native Denkdateneinstellungen konnten nicht geladen werden.", "nativeReasoningRetention.saveFailed": "Native Denkdateneinstellungen konnten nicht gespeichert werden.", "nativeReasoningRetention.saved": "Native Denkdateneinstellungen gespeichert.", + "workflowBudget.title": "Workflow-Limits für Subagenten", + "workflowBudget.description": "Obergrenzen pro Aufgabe, also pro Unterhaltung samt der Subagenten, die sie startet, gezählt über ein gleitendes Fenster von {minutes} Minuten. Eine Anfrage über einer Grenze wird lokal mit 429 abgelehnt, bevor ein Anbieter kontaktiert wird. Änderungen gelten ab der nächsten Anfrage; bereits gezählte Nutzung bleibt erhalten.", + "workflowBudget.maxConcurrentChildren": "Parallele Turns", + "workflowBudget.maxConcurrentChildrenHint": "Turns, die eine Aufgabe gleichzeitig ausführen darf. Ein Platz bleibt immer für die eigene Unterhaltung reserviert, Subagenten bekommen also einen weniger.", + "workflowBudget.maxPhysicalSends": "Modellanfragen pro Fenster", + "workflowBudget.maxPhysicalSendsHint": "Anfragen an den Anbieter, Wiederholungen eingeschlossen, die eine Aufgabe im Fenster senden darf.", + "workflowBudget.maxDistinctChildren": "Verschiedene Subagenten pro Fenster", + "workflowBudget.maxDistinctChildrenHint": "Unterschiedliche Subagenten-Threads, die eine Aufgabe im Fenster starten darf.", + "workflowBudget.range": "Standard {default}, erlaubt {min}–{max}.", + "workflowBudget.invalid": "Bitte eine ganze Zahl von {min} bis {max} eingeben.", + "workflowBudget.increase": "{label} erhöhen", + "workflowBudget.decrease": "{label} verringern", + "workflowBudget.reset": "Standard wiederherstellen", + "workflowBudget.loadFailed": "Workflow-Limits konnten nicht geladen werden.", + "workflowBudget.saveFailed": "Workflow-Limits konnten nicht gespeichert werden.", + "workflowBudget.saved": "Workflow-Limits gespeichert.", ...desktopCompatibilityCopy("de"), "claude.tierDefaults.unset": "Nicht festgelegt (Claude-Einstellung beibehalten)", "claude.tierDefaults.recovery": "Anwendung gestoppt; automatische Wiederherstellung fehlgeschlagen. Prüfen Sie vor einem erneuten Versuch die Sicherung unten.", diff --git a/gui/src/i18n/en.ts b/gui/src/i18n/en.ts index d805772ff80..4f4cd4ffe0f 100644 --- a/gui/src/i18n/en.ts +++ b/gui/src/i18n/en.ts @@ -106,6 +106,22 @@ export const en = { "nativeReasoningRetention.loadFailed": "Could not load native reasoning settings.", "nativeReasoningRetention.saveFailed": "Could not save native reasoning settings.", "nativeReasoningRetention.saved": "Native reasoning settings saved.", + "workflowBudget.title": "Workflow limits for subagents", + "workflowBudget.description": "Ceilings per task, meaning one conversation and the subagents it starts, counted over a rolling {minutes}-minute window. A request over a ceiling is refused locally with a 429 before any provider is contacted. Changes apply to the next request; usage already counted is kept.", + "workflowBudget.maxConcurrentChildren": "Parallel turns", + "workflowBudget.maxConcurrentChildrenHint": "Turns one task may run at the same time. One slot is always kept for your own conversation, so subagents get one fewer.", + "workflowBudget.maxPhysicalSends": "Model requests per window", + "workflowBudget.maxPhysicalSendsHint": "Upstream requests, retries included, one task may send inside the window.", + "workflowBudget.maxDistinctChildren": "Different subagents per window", + "workflowBudget.maxDistinctChildrenHint": "Distinct subagent threads one task may start inside the window.", + "workflowBudget.range": "Default {default}, allowed {min}–{max}.", + "workflowBudget.invalid": "Enter a whole number from {min} to {max}.", + "workflowBudget.increase": "Increase {label}", + "workflowBudget.decrease": "Decrease {label}", + "workflowBudget.reset": "Restore defaults", + "workflowBudget.loadFailed": "Could not load workflow limits.", + "workflowBudget.saveFailed": "Could not save workflow limits.", + "workflowBudget.saved": "Workflow limits saved.", ...desktopCompatibilityCopy("en"), "claude.tierDefaults.unset": "Unset (keep existing Claude setting)", "claude.tierDefaults.recovery": "Apply stopped and automatic restore failed. Check the backup below before retrying.", diff --git a/gui/src/i18n/fr.ts b/gui/src/i18n/fr.ts index f3fbc99a28c..4b8cf58b1de 100644 --- a/gui/src/i18n/fr.ts +++ b/gui/src/i18n/fr.ts @@ -104,6 +104,22 @@ export const fr: Record = { "nativeReasoningRetention.loadFailed": "Impossible de charger les paramètres du raisonnement natif.", "nativeReasoningRetention.saveFailed": "Impossible de sauvegarder les paramètres du raisonnement natif.", "nativeReasoningRetention.saved": "Paramètres du raisonnement natif sauvegardés.", + "workflowBudget.title": "Limites de workflow des sous-agents", + "workflowBudget.description": "Plafonds par tâche, c'est-à-dire une conversation et les sous-agents qu'elle lance, comptés sur une fenêtre glissante de {minutes} minutes. Une requête au-delà d'un plafond est refusée localement avec un 429 avant tout contact avec un fournisseur. Les changements s'appliquent à la requête suivante ; l'usage déjà compté est conservé.", + "workflowBudget.maxConcurrentChildren": "Tours en parallèle", + "workflowBudget.maxConcurrentChildrenHint": "Tours qu'une tâche peut exécuter en même temps. Un emplacement reste toujours réservé à votre propre conversation, les sous-agents en ont donc un de moins.", + "workflowBudget.maxPhysicalSends": "Requêtes au modèle par fenêtre", + "workflowBudget.maxPhysicalSendsHint": "Requêtes vers le fournisseur, nouvelles tentatives comprises, qu'une tâche peut envoyer dans la fenêtre.", + "workflowBudget.maxDistinctChildren": "Sous-agents distincts par fenêtre", + "workflowBudget.maxDistinctChildrenHint": "Fils de sous-agents différents qu'une tâche peut lancer dans la fenêtre.", + "workflowBudget.range": "Par défaut {default}, autorisé {min}–{max}.", + "workflowBudget.invalid": "Saisissez un nombre entier de {min} à {max}.", + "workflowBudget.increase": "Augmenter : {label}", + "workflowBudget.decrease": "Diminuer : {label}", + "workflowBudget.reset": "Restaurer les valeurs par défaut", + "workflowBudget.loadFailed": "Impossible de charger les limites de workflow.", + "workflowBudget.saveFailed": "Impossible d'enregistrer les limites de workflow.", + "workflowBudget.saved": "Limites de workflow enregistrées.", ...desktopCompatibilityCopy("fr"), "claude.tierDefaults.unset": "Non défini (conserver le réglage Claude)", "claude.tierDefaults.recovery": "Application arrêtée et restauration automatique échouée. Vérifiez la sauvegarde ci-dessous avant de réessayer.", diff --git a/gui/src/i18n/ja.ts b/gui/src/i18n/ja.ts index a3b6940897f..4c99bb76162 100644 --- a/gui/src/i18n/ja.ts +++ b/gui/src/i18n/ja.ts @@ -104,6 +104,22 @@ export const ja: Record = { "nativeReasoningRetention.loadFailed": "ネイティブ推論の設定を読み込めませんでした。", "nativeReasoningRetention.saveFailed": "ネイティブ推論の設定を保存できませんでした。", "nativeReasoningRetention.saved": "ネイティブ推論の設定を保存しました。", + "workflowBudget.title": "サブエージェントのワークフロー上限", + "workflowBudget.description": "タスク(1つの会話とそこから起動されたサブエージェント)ごとの上限で、直近{minutes}分間のスライディングウィンドウで数えます。上限を超えたリクエストは、プロバイダーに接続する前にローカルで 429 として拒否されます。変更は次のリクエストから適用され、すでに数えられた使用量は保持されます。", + "workflowBudget.maxConcurrentChildren": "並列ターン数", + "workflowBudget.maxConcurrentChildrenHint": "1つのタスクが同時に実行できるターン数です。1枠は常に自分の会話用に確保されるため、サブエージェントが使えるのは1つ少なくなります。", + "workflowBudget.maxPhysicalSends": "ウィンドウあたりのモデルリクエスト数", + "workflowBudget.maxPhysicalSendsHint": "1つのタスクがウィンドウ内に送信できるプロバイダーへのリクエスト数(再試行を含む)です。", + "workflowBudget.maxDistinctChildren": "ウィンドウあたりの異なるサブエージェント数", + "workflowBudget.maxDistinctChildrenHint": "1つのタスクがウィンドウ内に起動できる異なるサブエージェントスレッドの数です。", + "workflowBudget.range": "既定値 {default}、範囲 {min}–{max}。", + "workflowBudget.invalid": "{min}~{max} の整数を入力してください。", + "workflowBudget.increase": "{label}を増やす", + "workflowBudget.decrease": "{label}を減らす", + "workflowBudget.reset": "既定値に戻す", + "workflowBudget.loadFailed": "ワークフロー上限を読み込めませんでした。", + "workflowBudget.saveFailed": "ワークフロー上限を保存できませんでした。", + "workflowBudget.saved": "ワークフロー上限を保存しました。", ...desktopCompatibilityCopy("ja"), "claude.tierDefaults.unset": "未指定(既存のClaude設定を維持)", "claude.tierDefaults.recovery": "適用が停止し、自動復元に失敗しました。再試行する前に以下のバックアップを確認してください。", diff --git a/gui/src/i18n/ko.ts b/gui/src/i18n/ko.ts index a92e751b06b..b073cfed72c 100644 --- a/gui/src/i18n/ko.ts +++ b/gui/src/i18n/ko.ts @@ -104,6 +104,22 @@ export const ko: Record = { "nativeReasoningRetention.loadFailed": "네이티브 추론 설정을 불러올 수 없습니다.", "nativeReasoningRetention.saveFailed": "네이티브 추론 설정을 저장할 수 없습니다.", "nativeReasoningRetention.saved": "네이티브 추론 설정을 저장했습니다.", + "workflowBudget.title": "서브에이전트 워크플로 한도", + "workflowBudget.description": "작업(하나의 대화와 그 대화가 시작한 서브에이전트)마다 적용되는 상한으로, 최근 {minutes}분의 이동 창 기준으로 집계합니다. 상한을 넘는 요청은 공급자에 연결하기 전에 로컬에서 429로 거부됩니다. 변경 사항은 다음 요청부터 적용되며, 이미 집계된 사용량은 유지됩니다.", + "workflowBudget.maxConcurrentChildren": "병렬 턴", + "workflowBudget.maxConcurrentChildrenHint": "하나의 작업이 동시에 실행할 수 있는 턴 수입니다. 한 자리는 항상 내 대화용으로 남겨 두므로 서브에이전트는 하나 적게 사용합니다.", + "workflowBudget.maxPhysicalSends": "창당 모델 요청 수", + "workflowBudget.maxPhysicalSendsHint": "하나의 작업이 창 안에서 보낼 수 있는 공급자 요청 수(재시도 포함)입니다.", + "workflowBudget.maxDistinctChildren": "창당 서로 다른 서브에이전트 수", + "workflowBudget.maxDistinctChildrenHint": "하나의 작업이 창 안에서 시작할 수 있는 서로 다른 서브에이전트 스레드 수입니다.", + "workflowBudget.range": "기본값 {default}, 허용 범위 {min}–{max}.", + "workflowBudget.invalid": "{min}~{max} 사이의 정수를 입력하세요.", + "workflowBudget.increase": "{label} 늘리기", + "workflowBudget.decrease": "{label} 줄이기", + "workflowBudget.reset": "기본값 복원", + "workflowBudget.loadFailed": "워크플로 한도를 불러오지 못했습니다.", + "workflowBudget.saveFailed": "워크플로 한도를 저장하지 못했습니다.", + "workflowBudget.saved": "워크플로 한도를 저장했습니다.", ...desktopCompatibilityCopy("ko"), "claude.tierDefaults.unset": "미지정 (기존 Claude 설정 유지)", "claude.tierDefaults.recovery": "적용이 중단되었고 자동 복원에 실패했어요. 다시 시도하기 전에 아래 백업을 확인하세요.", diff --git a/gui/src/i18n/pt.ts b/gui/src/i18n/pt.ts index 526567b7eba..f170560e262 100644 --- a/gui/src/i18n/pt.ts +++ b/gui/src/i18n/pt.ts @@ -106,6 +106,22 @@ export const pt: Record = { "nativeReasoningRetention.loadFailed": "Não foi possível carregar as configurações de raciocínio nativo.", "nativeReasoningRetention.saveFailed": "Não foi possível salvar as configurações de raciocínio nativo.", "nativeReasoningRetention.saved": "Configurações de raciocínio nativo salvas.", + "workflowBudget.title": "Limites de fluxo de trabalho dos subagentes", + "workflowBudget.description": "Tetos por tarefa, ou seja, uma conversa e os subagentes que ela inicia, contados em uma janela deslizante de {minutes} minutos. Uma requisição acima de um teto é recusada localmente com 429 antes de qualquer provedor ser contatado. As alterações valem a partir da próxima requisição; o uso já contado é mantido.", + "workflowBudget.maxConcurrentChildren": "Turnos em paralelo", + "workflowBudget.maxConcurrentChildrenHint": "Turnos que uma tarefa pode executar ao mesmo tempo. Uma vaga fica sempre reservada para a sua própria conversa, então os subagentes recebem uma a menos.", + "workflowBudget.maxPhysicalSends": "Requisições ao modelo por janela", + "workflowBudget.maxPhysicalSendsHint": "Requisições ao provedor, incluindo novas tentativas, que uma tarefa pode enviar dentro da janela.", + "workflowBudget.maxDistinctChildren": "Subagentes diferentes por janela", + "workflowBudget.maxDistinctChildrenHint": "Threads de subagentes distintos que uma tarefa pode iniciar dentro da janela.", + "workflowBudget.range": "Padrão {default}, permitido {min}–{max}.", + "workflowBudget.invalid": "Informe um número inteiro de {min} a {max}.", + "workflowBudget.increase": "Aumentar {label}", + "workflowBudget.decrease": "Diminuir {label}", + "workflowBudget.reset": "Restaurar padrões", + "workflowBudget.loadFailed": "Não foi possível carregar os limites de fluxo de trabalho.", + "workflowBudget.saveFailed": "Não foi possível salvar os limites de fluxo de trabalho.", + "workflowBudget.saved": "Limites de fluxo de trabalho salvos.", ...desktopCompatibilityCopy("pt"), "claude.tierDefaults.unset": "Não definido (manter configuração Claude)", "claude.tierDefaults.recovery": "Aplicação interrompida; restauração automática falhou. Verifique o backup abaixo antes de tentar novamente.", diff --git a/gui/src/i18n/ru.ts b/gui/src/i18n/ru.ts index 5835b18f727..7c7d951edde 100644 --- a/gui/src/i18n/ru.ts +++ b/gui/src/i18n/ru.ts @@ -104,6 +104,22 @@ export const ru: Record = { "nativeReasoningRetention.loadFailed": "Не удалось загрузить настройки нативных рассуждений.", "nativeReasoningRetention.saveFailed": "Не удалось сохранить настройки нативных рассуждений.", "nativeReasoningRetention.saved": "Настройки нативных рассуждений сохранены.", + "workflowBudget.title": "Лимиты рабочих процессов для субагентов", + "workflowBudget.description": "Ограничения на задачу, то есть на один диалог и запущенных им субагентов, считаются в скользящем окне длиной {minutes} минут. Запрос сверх ограничения отклоняется локально с кодом 429 ещё до обращения к провайдеру. Изменения действуют со следующего запроса; уже учтённое использование сохраняется.", + "workflowBudget.maxConcurrentChildren": "Параллельные ходы", + "workflowBudget.maxConcurrentChildrenHint": "Сколько ходов одна задача может выполнять одновременно. Одно место всегда остаётся за вашим собственным диалогом, поэтому субагентам доступно на одно меньше.", + "workflowBudget.maxPhysicalSends": "Запросов к модели за окно", + "workflowBudget.maxPhysicalSendsHint": "Запросы к провайдеру, включая повторные попытки, которые одна задача может отправить за окно.", + "workflowBudget.maxDistinctChildren": "Разных субагентов за окно", + "workflowBudget.maxDistinctChildrenHint": "Сколько разных потоков субагентов одна задача может запустить за окно.", + "workflowBudget.range": "По умолчанию {default}, допустимо {min}–{max}.", + "workflowBudget.invalid": "Введите целое число от {min} до {max}.", + "workflowBudget.increase": "Увеличить: {label}", + "workflowBudget.decrease": "Уменьшить: {label}", + "workflowBudget.reset": "Восстановить значения по умолчанию", + "workflowBudget.loadFailed": "Не удалось загрузить лимиты рабочих процессов.", + "workflowBudget.saveFailed": "Не удалось сохранить лимиты рабочих процессов.", + "workflowBudget.saved": "Лимиты рабочих процессов сохранены.", ...desktopCompatibilityCopy("ru"), "claude.tierDefaults.unset": "Не задано (сохранить настройку Claude)", "claude.tierDefaults.recovery": "Применение остановлено, автоматическое восстановление не удалось. Проверьте резервную копию ниже перед повтором.", diff --git a/gui/src/i18n/tr.ts b/gui/src/i18n/tr.ts index b673ee21a0f..ab1fd6ee3b1 100644 --- a/gui/src/i18n/tr.ts +++ b/gui/src/i18n/tr.ts @@ -105,6 +105,22 @@ export const tr: Record = { "nativeReasoningRetention.loadFailed": "Yerel akıl yürütme ayarları yüklenemedi.", "nativeReasoningRetention.saveFailed": "Yerel akıl yürütme ayarları kaydedilemedi.", "nativeReasoningRetention.saved": "Yerel akıl yürütme ayarları kaydedildi.", + "workflowBudget.title": "Alt ajanlar için iş akışı sınırları", + "workflowBudget.description": "Görev başına, yani bir konuşma ve başlattığı alt ajanlar için üst sınırlar; {minutes} dakikalık kayan bir pencerede sayılır. Bir sınırı aşan istek, herhangi bir sağlayıcıyla iletişime geçilmeden önce yerel olarak 429 ile reddedilir. Değişiklikler bir sonraki istekten itibaren geçerlidir; önceden sayılan kullanım korunur.", + "workflowBudget.maxConcurrentChildren": "Paralel turlar", + "workflowBudget.maxConcurrentChildrenHint": "Bir görevin aynı anda çalıştırabileceği tur sayısı. Bir yer her zaman kendi konuşmanız için ayrılır, bu yüzden alt ajanlar bir eksik alır.", + "workflowBudget.maxPhysicalSends": "Pencere başına model isteği", + "workflowBudget.maxPhysicalSendsHint": "Bir görevin pencere içinde gönderebileceği sağlayıcı istekleri, yeniden denemeler dahil.", + "workflowBudget.maxDistinctChildren": "Pencere başına farklı alt ajan", + "workflowBudget.maxDistinctChildrenHint": "Bir görevin pencere içinde başlatabileceği farklı alt ajan iş parçacığı sayısı.", + "workflowBudget.range": "Varsayılan {default}, izin verilen {min}–{max}.", + "workflowBudget.invalid": "{min} ile {max} arasında bir tam sayı girin.", + "workflowBudget.increase": "{label} artır", + "workflowBudget.decrease": "{label} azalt", + "workflowBudget.reset": "Varsayılanları geri yükle", + "workflowBudget.loadFailed": "İş akışı sınırları yüklenemedi.", + "workflowBudget.saveFailed": "İş akışı sınırları kaydedilemedi.", + "workflowBudget.saved": "İş akışı sınırları kaydedildi.", ...desktopCompatibilityCopy("tr"), "claude.tierDefaults.unset": "Belirtilmedi (mevcut Claude ayarını koru)", "claude.tierDefaults.recovery": "Uygulama durdu ve otomatik geri yükleme başarısız oldu. Yeniden denemeden önce aşağıdaki yedeği kontrol edin.", diff --git a/gui/src/i18n/vi.ts b/gui/src/i18n/vi.ts index eed86f0d3d2..807a752f49e 100644 --- a/gui/src/i18n/vi.ts +++ b/gui/src/i18n/vi.ts @@ -106,6 +106,22 @@ export const vi: Record = { "nativeReasoningRetention.loadFailed": "Không thể tải cài đặt suy luận gốc.", "nativeReasoningRetention.saveFailed": "Không thể lưu cài đặt suy luận gốc.", "nativeReasoningRetention.saved": "Đã lưu cài đặt suy luận gốc.", + "workflowBudget.title": "Giới hạn quy trình cho tác tử phụ", + "workflowBudget.description": "Mức trần cho mỗi tác vụ, tức là một cuộc trò chuyện cùng các tác tử phụ nó khởi chạy, được đếm trong cửa sổ trượt {minutes} phút. Yêu cầu vượt mức trần bị từ chối cục bộ với mã 429 trước khi liên hệ bất kỳ nhà cung cấp nào. Thay đổi áp dụng từ yêu cầu tiếp theo; mức sử dụng đã đếm được giữ nguyên.", + "workflowBudget.maxConcurrentChildren": "Lượt chạy song song", + "workflowBudget.maxConcurrentChildrenHint": "Số lượt một tác vụ có thể chạy cùng lúc. Luôn giữ lại một chỗ cho cuộc trò chuyện của bạn, nên tác tử phụ được ít hơn một.", + "workflowBudget.maxPhysicalSends": "Yêu cầu tới mô hình mỗi cửa sổ", + "workflowBudget.maxPhysicalSendsHint": "Số yêu cầu tới nhà cung cấp, kể cả thử lại, mà một tác vụ có thể gửi trong cửa sổ.", + "workflowBudget.maxDistinctChildren": "Tác tử phụ khác nhau mỗi cửa sổ", + "workflowBudget.maxDistinctChildrenHint": "Số luồng tác tử phụ khác nhau mà một tác vụ có thể khởi chạy trong cửa sổ.", + "workflowBudget.range": "Mặc định {default}, cho phép {min}–{max}.", + "workflowBudget.invalid": "Nhập một số nguyên từ {min} đến {max}.", + "workflowBudget.increase": "Tăng {label}", + "workflowBudget.decrease": "Giảm {label}", + "workflowBudget.reset": "Khôi phục mặc định", + "workflowBudget.loadFailed": "Không thể tải giới hạn quy trình.", + "workflowBudget.saveFailed": "Không thể lưu giới hạn quy trình.", + "workflowBudget.saved": "Đã lưu giới hạn quy trình.", ...desktopCompatibilityCopy("vi"), "claude.tierDefaults.unset": "Chưa đặt (giữ cài đặt Claude hiện có)", "claude.tierDefaults.recovery": "Đã dừng áp dụng và khôi phục tự động thất bại. Kiểm tra bản sao lưu bên dưới trước khi thử lại.", diff --git a/gui/src/i18n/zh-TW.ts b/gui/src/i18n/zh-TW.ts index 48b05bda9a5..63c1ac42d41 100644 --- a/gui/src/i18n/zh-TW.ts +++ b/gui/src/i18n/zh-TW.ts @@ -102,6 +102,22 @@ export const zhTW: Record = { "nativeReasoningRetention.loadFailed": "無法載入原生推理設定。", "nativeReasoningRetention.saveFailed": "無法儲存原生推理設定。", "nativeReasoningRetention.saved": "原生推理設定已儲存。", + "workflowBudget.title": "子代理工作流程限制", + "workflowBudget.description": "依任務(即一個對話及其啟動的子代理)設定的上限,在滾動的 {minutes} 分鐘視窗內計數。超過上限的請求會在聯絡任何供應商之前於本機以 429 拒絕。變更從下一個請求開始生效;已計入的用量會保留。", + "workflowBudget.maxConcurrentChildren": "平行輪次", + "workflowBudget.maxConcurrentChildrenHint": "一個任務可同時執行的輪次數。一律為你自己的對話保留一個名額,因此子代理少一個。", + "workflowBudget.maxPhysicalSends": "每個視窗的模型請求數", + "workflowBudget.maxPhysicalSendsHint": "一個任務在視窗內可向供應商傳送的請求數(含重試)。", + "workflowBudget.maxDistinctChildren": "每個視窗的不同子代理數", + "workflowBudget.maxDistinctChildrenHint": "一個任務在視窗內可啟動的不同子代理執行緒數。", + "workflowBudget.range": "預設 {default},允許 {min}–{max}。", + "workflowBudget.invalid": "請輸入 {min} 到 {max} 之間的整數。", + "workflowBudget.increase": "增加{label}", + "workflowBudget.decrease": "減少{label}", + "workflowBudget.reset": "還原預設值", + "workflowBudget.loadFailed": "無法載入工作流程限制。", + "workflowBudget.saveFailed": "無法儲存工作流程限制。", + "workflowBudget.saved": "工作流程限制已儲存。", ...desktopCompatibilityCopy("zh-TW"), "claude.tierDefaults.unset": "未設定(保留現有 Claude 設定)", "claude.tierDefaults.recovery": "套用已停止,自動還原失敗。重試前請檢查下方備份。", diff --git a/gui/src/i18n/zh.ts b/gui/src/i18n/zh.ts index d04ea73e9b5..005f5099ab1 100644 --- a/gui/src/i18n/zh.ts +++ b/gui/src/i18n/zh.ts @@ -104,6 +104,22 @@ export const zh: Record = { "nativeReasoningRetention.loadFailed": "无法加载原生推理设置。", "nativeReasoningRetention.saveFailed": "无法保存原生推理设置。", "nativeReasoningRetention.saved": "原生推理设置已保存。", + "workflowBudget.title": "子代理工作流限制", + "workflowBudget.description": "按任务(即一个对话及其启动的子代理)设置的上限,在滚动的 {minutes} 分钟窗口内计数。超过上限的请求会在联系任何提供商之前于本地以 429 拒绝。更改从下一个请求开始生效;已计入的用量会保留。", + "workflowBudget.maxConcurrentChildren": "并行轮次", + "workflowBudget.maxConcurrentChildrenHint": "一个任务可同时运行的轮次数。始终为你自己的对话保留一个名额,因此子代理少一个。", + "workflowBudget.maxPhysicalSends": "每个窗口的模型请求数", + "workflowBudget.maxPhysicalSendsHint": "一个任务在窗口内可向提供商发送的请求数(含重试)。", + "workflowBudget.maxDistinctChildren": "每个窗口的不同子代理数", + "workflowBudget.maxDistinctChildrenHint": "一个任务在窗口内可启动的不同子代理线程数。", + "workflowBudget.range": "默认 {default},允许 {min}–{max}。", + "workflowBudget.invalid": "请输入 {min} 到 {max} 之间的整数。", + "workflowBudget.increase": "增加{label}", + "workflowBudget.decrease": "减少{label}", + "workflowBudget.reset": "恢复默认值", + "workflowBudget.loadFailed": "无法加载工作流限制。", + "workflowBudget.saveFailed": "无法保存工作流限制。", + "workflowBudget.saved": "工作流限制已保存。", ...desktopCompatibilityCopy("zh"), "claude.tierDefaults.unset": "未设置(保留现有 Claude 设置)", "claude.tierDefaults.recovery": "应用已停止,自动恢复失败。重试前请检查下方备份。", diff --git a/gui/src/pages/Subagents.tsx b/gui/src/pages/Subagents.tsx index b56e6199702..d21200620ae 100644 --- a/gui/src/pages/Subagents.tsx +++ b/gui/src/pages/Subagents.tsx @@ -1,4 +1,5 @@ import SubagentForceControl from "../components/subagents-workspace/SubagentForceControl"; +import WorkflowBudgetPanel from "../components/WorkflowBudgetPanel"; import { useCallback, useEffect, useRef, useState } from "react"; import { readJsonOrThrow } from "../fetch-json"; import { Notice } from "../ui"; @@ -365,6 +366,7 @@ export default function Subagents({ apiBase }: { apiBase: string }) { )} + ; +let win: Window; +let root: Root | undefined; +let container: HTMLDivElement; +let setting: Record; +let failLoad: boolean; +let failSave: boolean; +let writes: unknown[]; +let urls: string[]; + +const payload = () => ({ ...setting, interactiveReserve: 1, windowMs: 600_000, defaults, bounds }); + +beforeEach(() => { + previous = Object.fromEntries(globals.map(key => [key, Object.getOwnPropertyDescriptor(globalThis, key)])); + win = new Window({ url: "http://localhost/" }); + for (const key of ["document", "window", "navigator", "localStorage", "sessionStorage", "HTMLElement"] as const) { + Object.defineProperty(globalThis, key, { configurable: true, value: key === "window" ? win : win[key] }); + } + Object.defineProperty(globalThis, "IS_REACT_ACT_ENVIRONMENT", { configurable: true, value: true }); + win.localStorage.setItem("ocx-lang", "en"); + setting = { ...defaults }; + failLoad = false; failSave = false; writes = []; urls = []; + Object.defineProperty(globalThis, "fetch", { configurable: true, writable: true, value: async (input: unknown, init?: RequestInit) => { + urls.push(String(input)); + if (init?.method === "PUT") { + const body = JSON.parse(String(init.body)) as Record | null; + writes.push(body); + if (failSave) return Response.json({ error: "fixture failure" }, { status: 500 }); + setting = body === null ? { ...defaults } : { ...setting, ...body }; + } else if (failLoad) return Response.json({ error: "unavailable" }, { status: 503 }); + return Response.json(payload()); + } }); +}); + +afterEach(async () => { + if (root) await act(async () => { root!.unmount(); }); + root = undefined; win.close(); + for (const key of globals) { + if (previous[key]) Object.defineProperty(globalThis, key, previous[key]!); + else delete (globalThis as Record)[key]; + } +}); + +async function flush() { await act(async () => { await new Promise(resolve => setTimeout(resolve, 10)); }); } +async function render(base = "") { + container = win.document.createElement("div") as unknown as HTMLDivElement; + win.document.body.appendChild(container); + root = (await import("react-dom/client")).createRoot(container); + await act(async () => { root!.render(); }); + await flush(); +} +function button(label: string) { + return [...container.querySelectorAll("button")] + .find(value => value.textContent === label || value.getAttribute("aria-label") === label)!; +} +const input = (label: string) => container.querySelector('input[aria-label="' + label + '"]')!; +async function click(label: string) { await act(async () => { button(label).click(); }); } +async function enter(label: string, value: string) { + await act(async () => { + Object.getOwnPropertyDescriptor(win.HTMLInputElement.prototype, "value")!.set!.call(input(label), value); + input(label).dispatchEvent(new win.Event("input", { bubbles: true })); + }); +} + +test("shows the effective ceilings and saves all three through the dedicated route", async () => { + await render("/proxy"); + expect(input(PARALLEL).value).toBe("8"); + expect(input(SENDS).value).toBe("256"); + expect(container.textContent).toContain("rolling 10-minute window"); + expect(container.textContent).toContain("Default 8, allowed 2–64."); + expect(button("Save").disabled).toBe(true); + expect(button("Restore defaults").disabled).toBe(true); + await click("Increase " + PARALLEL); await click("Increase " + PARALLEL); + await enter(SENDS, "1024"); + await click("Save"); + expect(writes).toEqual([{ maxConcurrentChildren: 10, maxPhysicalSends: 1_024, maxDistinctChildren: 64 }]); + expect(input(PARALLEL).value).toBe("10"); + expect(button("Save").disabled).toBe(true); + expect(container.querySelector('[role="status"]')?.textContent).toBe("Workflow limits saved."); + expect(urls.every(value => value === "/proxy/api/workflow-budget/settings")).toBe(true); +}); + +test("saving the defaults and restoring both write null so the install keeps following built-in ceilings", async () => { + setting = { ...defaults, maxConcurrentChildren: 12 }; + await render(); + expect(input(PARALLEL).value).toBe("12"); + await enter(PARALLEL, "8"); + await click("Save"); + expect(writes).toEqual([null]); + expect(button("Restore defaults").disabled).toBe(true); + await click("Increase " + PARALLEL); await click("Save"); + expect(writes.at(-1)).toEqual({ ...defaults, maxConcurrentChildren: 9 }); + await click("Restore defaults"); + expect(writes.at(-1)).toBeNull(); + expect(input(PARALLEL).value).toBe("8"); + expect(button("Restore defaults").disabled).toBe(true); +}); + +test("an out-of-range draft names the allowed range and blocks saving", async () => { + await render(); + await enter(PARALLEL, "1"); + expect(container.querySelector('[role="alert"]')?.textContent).toBe("Enter a whole number from 2 to 64."); + expect(input(PARALLEL).getAttribute("aria-invalid")).toBe("true"); + expect(button("Save").disabled).toBe(true); + await enter(PARALLEL, "65"); + expect(button("Save").disabled).toBe(true); + await click("Decrease " + PARALLEL); + expect(input(PARALLEL).value).toBe("64"); + expect(container.querySelector('[role="alert"]')).toBeNull(); + expect(button("Save").disabled).toBe(false); + expect(writes).toEqual([]); +}); + +test("load errors disable every control and retry reloads", async () => { + failLoad = true; await render(); + expect(button("Save").disabled).toBe(true); + expect(button("Restore defaults").disabled).toBe(true); + expect(button("Increase " + PARALLEL).disabled).toBe(true); + expect(container.querySelector('[role="alert"]')?.textContent).toContain("Could not load workflow limits."); + failLoad = false; setting = { ...defaults, maxPhysicalSends: 512 }; + await click("Retry"); + expect(input(SENDS).value).toBe("512"); + expect(button("Increase " + PARALLEL).disabled).toBe(false); +}); + +test("a failed save keeps the draft and does not claim success", async () => { + await render(); + await click("Increase " + PARALLEL); + failSave = true; await click("Save"); + expect(input(PARALLEL).value).toBe("9"); + expect(button("Save").disabled).toBe(false); + expect(container.querySelector('[role="alert"]')?.textContent).toContain("Could not save workflow limits."); + expect(container.querySelector('[role="status"]')).toBeNull(); +}); diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index efa3f4e689d..b80fd41ad6a 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -2,7 +2,7 @@ "version": 1, "root": "tests", "explicitShards": ["layout-clients.json", "layout-codex-collision.json", "layout-agent-assertion.json", "layout-chat.json", "layout-replay.json", "layout-adapters.json", "layout-jev.json", "layout-workbuddy.json", "layout-combos.json", "layout-credit-renewal.json", "layout-server-inbound.json", "layout-providers.json", "layout-responses.json", "layout-exec-boundary.json", "layout-persistent-dashboard.json", "layout-start-ownership.json", "layout-config.json", "layout-kiro-usage.json", "layout-usage.json"], - "explicit": { "service-desktop-recovery.test.ts": "service", "catalog-omit-disabled-native.test.ts": "codex-integration", "provider-input-modalities.test.ts": "providers", "combo-description.test.ts": "routing", "combo-description-catalog.test.ts": "codex-integration", "codex-provider-routing.test.ts": "codex-integration", "config-native-reasoning-retention.test.ts": "config", "server-native-reasoning-retention-merge.test.ts": "server", "server-native-reasoning-retention.test.ts": "server", "responses-retained-reasoning-stream-recovery.test.ts": "responses", "openai-native-hosted-search-history.test.ts": "adapters/openai", "responses-native-compact-search-history.test.ts": "responses", "config-hosted-search-projection.test.ts": "config", "gcp-credential-boundary.test.ts": "adapters/google", "gcp-sa-marker.test.ts": "adapters/google", "gcp-sa-adapter.test.ts": "adapters/google", "management-gcp-credential-input.test.ts": "server", "codex-preserve-signed-in.test.ts": "codex-integration", "cli-account-preservation.test.ts": "cli", "cli-provider-list-active.test.ts": "cli", "service-state-isolation.test.ts": "service", "compaction-history-framing.test.ts": "responses", "catalog-native-max-context.test.ts": "codex-integration", "protocol-argon-thought-boundary.test.ts": "responses", "service-log.test.ts": "service", "messages-native-empty-inline.test.ts": "claude-integration", "catalog-auto-review-override.test.ts": "codex-integration", "management-auto-review-settings.test.ts": "server", "management-auto-review-auth.test.ts": "server", "prefer-model-alias-ids.test.ts": "server", "responses-mimo-envelope.test.ts": "responses", "responses-call-id-linkage.test.ts": "responses", "bridge-terminal-refusal.test.ts": "adapters", "loop-terminal-refusal.test.ts": "images", "web-search-terminal-refusal.test.ts": "web-search", "combo-terminal-refusal.test.ts": "routing", "server-terminal-refusal-wiring.test.ts": "server", "incomplete-reason-normalization.test.ts": "responses", "advisor-retention.test.ts": "advisor", "responses-native-key-cooldown.test.ts": "responses", "responses-run-turn-key-readmission.test.ts": "responses", "qoder-spawn-readmission.test.ts": "adapters", "api-key-selection-cooldown.test.ts": "providers", "api-key-auto-pause-distribution.test.ts": "providers", "explicit-key-pause-oauth-boundary.test.ts": "providers", "server-chat-native-key-selection.test.ts": "server", "compact-key-selection.test.ts": "responses", "messages-native-key-selection.test.ts": "claude-integration", "coding-agent-windows-dispatch.test.ts": "adapters", "coding-agent-native-empty-argv.test.ts": "windows", + "explicit": { "service-desktop-recovery.test.ts": "service", "catalog-omit-disabled-native.test.ts": "codex-integration", "provider-input-modalities.test.ts": "providers", "combo-description.test.ts": "routing", "combo-description-catalog.test.ts": "codex-integration", "codex-provider-routing.test.ts": "codex-integration", "config-native-reasoning-retention.test.ts": "config", "server-native-reasoning-retention-merge.test.ts": "server", "server-native-reasoning-retention.test.ts": "server", "config-workflow-budget.test.ts": "config", "server-workflow-budget-settings.test.ts": "server", "responses-retained-reasoning-stream-recovery.test.ts": "responses", "openai-native-hosted-search-history.test.ts": "adapters/openai", "responses-native-compact-search-history.test.ts": "responses", "config-hosted-search-projection.test.ts": "config", "gcp-credential-boundary.test.ts": "adapters/google", "gcp-sa-marker.test.ts": "adapters/google", "gcp-sa-adapter.test.ts": "adapters/google", "management-gcp-credential-input.test.ts": "server", "codex-preserve-signed-in.test.ts": "codex-integration", "cli-account-preservation.test.ts": "cli", "cli-provider-list-active.test.ts": "cli", "service-state-isolation.test.ts": "service", "compaction-history-framing.test.ts": "responses", "catalog-native-max-context.test.ts": "codex-integration", "protocol-argon-thought-boundary.test.ts": "responses", "service-log.test.ts": "service", "messages-native-empty-inline.test.ts": "claude-integration", "catalog-auto-review-override.test.ts": "codex-integration", "management-auto-review-settings.test.ts": "server", "management-auto-review-auth.test.ts": "server", "prefer-model-alias-ids.test.ts": "server", "responses-mimo-envelope.test.ts": "responses", "responses-call-id-linkage.test.ts": "responses", "bridge-terminal-refusal.test.ts": "adapters", "loop-terminal-refusal.test.ts": "images", "web-search-terminal-refusal.test.ts": "web-search", "combo-terminal-refusal.test.ts": "routing", "server-terminal-refusal-wiring.test.ts": "server", "incomplete-reason-normalization.test.ts": "responses", "advisor-retention.test.ts": "advisor", "responses-native-key-cooldown.test.ts": "responses", "responses-run-turn-key-readmission.test.ts": "responses", "qoder-spawn-readmission.test.ts": "adapters", "api-key-selection-cooldown.test.ts": "providers", "api-key-auto-pause-distribution.test.ts": "providers", "explicit-key-pause-oauth-boundary.test.ts": "providers", "server-chat-native-key-selection.test.ts": "server", "compact-key-selection.test.ts": "responses", "messages-native-key-selection.test.ts": "claude-integration", "coding-agent-windows-dispatch.test.ts": "adapters", "coding-agent-native-empty-argv.test.ts": "windows", "stop-restore-history-ownership.test.ts": "service", "cli-stop-refusal-transport.test.ts": "cli", "service-proxy-env.test.ts": "service", "openai-chat-standalone-bare-block.test.ts": "adapters/openai", diff --git a/src/config/diagnostics.ts b/src/config/diagnostics.ts index 883c929146d..1e58125e44c 100644 --- a/src/config/diagnostics.ts +++ b/src/config/diagnostics.ts @@ -8,6 +8,7 @@ import { join } from "node:path"; import * as z from "zod/v4"; import { compactionRecoveryConfigError } from "./schema/compaction-recovery"; import { nativeReasoningRetentionConfigError } from "./schema/native-reasoning-retention"; +import { WORKFLOW_BUDGET_SHAPE_HINT, workflowBudgetConfigError } from "./schema/workflow-budget"; import { desktopCompatibilityConfigError } from "./schema/desktop-compatibility"; import { blockedModelRedirectsError } from "./schema/blocked-model-redirects"; import { mainAccountHardLockFallbackError } from "./schema/main-account-hard-lock-fallback"; @@ -145,6 +146,7 @@ function validFileConfigDiagnostics(config: OcxConfig, rawParsed: unknown): Conf warnings.push(...degradedListenerWarnings(rawParsed, normalized)); if (blockedModelRedirectsError(rawParsed)) warnings.push("blockedModelRedirects ignored: expected a map of nonempty model keys to nonempty string targets"); if (nativeReasoningRetentionConfigError(rawParsed)) warnings.push("nativeReasoningRetention ignored: expected only optional boolean modelSwitch and accountSwitch fields; both remain off"); + if (workflowBudgetConfigError(rawParsed)) warnings.push("workflowBudget ignored: " + WORKFLOW_BUDGET_SHAPE_HINT + "; the built-in ceilings remain"); if (mainAccountHardLockFallbackError(rawParsed)) warnings.push("codexMainAccountHardLockFallback ignored: expected a nonblank model selector and an optional declared reasoningEffort"); if (fallbackPolicyError(rawParsed)) warnings.push("fallback partly ignored: expected boolean model/provider switches, providers/routes/devices maps of them, and devices..localFallback; ignored leaves keep today's fallback behavior"); const quotaAutoRefreshWarning = degradedCodexQuotaAutoRefreshWarning(rawParsed, normalized); @@ -735,6 +737,7 @@ export function validateConfigCandidate(value: unknown): { ok: true; config: Ocx // drop a refusal on the next whole-config save; the write boundary rejects it instead. ?? fallbackPolicyError(value) ?? nativeReasoningRetentionConfigError(value) + ?? workflowBudgetConfigError(value) ?? mainAccountHardLockFallbackError(value) ?? compactionRecoveryConfigError(value) ?? desktopCompatibilityConfigError(value) ?? configReasoningPinsConfigError(value) ?? blankHostnameError(value) diff --git a/src/config/load-degrade.ts b/src/config/load-degrade.ts index 342ca547722..4ad2bbd2554 100644 --- a/src/config/load-degrade.ts +++ b/src/config/load-degrade.ts @@ -4,6 +4,7 @@ import { chmodSync, existsSync } from "node:fs"; import { join } from "node:path"; import { compactionRecoveryConfigError } from "./schema/compaction-recovery"; import { nativeReasoningRetentionConfigError } from "./schema/native-reasoning-retention"; +import { workflowBudgetConfigError } from "./schema/workflow-budget"; import { desktopCompatibilityConfigError } from "./schema/desktop-compatibility"; import { blockedModelRedirectsError } from "./schema/blocked-model-redirects"; import { mainAccountHardLockFallbackError } from "./schema/main-account-hard-lock-fallback"; @@ -137,6 +138,7 @@ export function warnDegradedCompactionRouting(rawParsed: unknown, validated: Ocx */ export function warnDegradedTopLevelOptIns(rawParsed: unknown, validated: OcxConfig): void { if (nativeReasoningRetentionConfigError(rawParsed)) console.warn("⚠️ invalid nativeReasoningRetention disabled; model and account switch allowances remain off"); + if (workflowBudgetConfigError(rawParsed)) console.warn("⚠️ invalid workflowBudget ignored; the built-in workflow ceilings stay in force"); setCodexWsReuseAcrossTurns(validated.codexWsReuseAcrossTurns === true); const retentionWarning = degradedReasoningRetentionWarning(rawParsed, validated); if (retentionWarning) console.warn(retentionWarning); diff --git a/src/config/schema/config-schema.ts b/src/config/schema/config-schema.ts index 2bacde053c6..9d3557f0fc2 100644 --- a/src/config/schema/config-schema.ts +++ b/src/config/schema/config-schema.ts @@ -6,6 +6,7 @@ import { accountPauseStateSchema } from "./account-auto-pause"; import { anthropicAccountPoolSchema } from "./anthropic-account-pool"; import { compactionRecoverySchema } from "./compaction-recovery"; import { nativeReasoningRetentionSchema } from "./native-reasoning-retention"; +import { workflowBudgetSchema } from "./workflow-budget"; import { desktopCompatibilitySchema } from "./desktop-compatibility"; import { blockedModelRedirectsSchema } from "./blocked-model-redirects"; import { mainAccountHardLockFallbackSchema } from "./main-account-hard-lock-fallback"; @@ -221,6 +222,8 @@ export const configSchema = z.object({ // A bad hand edit disables only this block; writes validate the catch-free leaf first. reasoningRetention: reasoningRetentionSchema.optional().catch(undefined), nativeReasoningRetention: nativeReasoningRetentionSchema.optional().catch(undefined), + // A bad hand edit falls back to the built-in ceilings; it never turns the guard off. + workflowBudget: workflowBudgetSchema.optional().catch(undefined), desktopCompatibility: desktopCompatibilitySchema.optional().catch(undefined), // A hand-edited malformed phase disables only that phase instead of rejecting // providers/apiKeys, matching the load-time degradation notice; the management write diff --git a/src/config/schema/workflow-budget.ts b/src/config/schema/workflow-budget.ts new file mode 100644 index 00000000000..7b34fb214b9 --- /dev/null +++ b/src/config/schema/workflow-budget.ts @@ -0,0 +1,43 @@ +import * as z from "zod/v4"; +import type { OcxConfig } from "../../types"; + +/** + * Per-install bounds for the root-workflow count ceilings. + * + * The lower concurrency bound is two because one slot is always held back for the interactive + * turn; at one, every worker turn would be refused. The upper bounds keep each ceiling a + * ceiling: a value large enough to never fire is a way of switching the guard off, and that is + * not what this setting is for. + */ +export const WORKFLOW_BUDGET_BOUNDS = { + maxConcurrentChildren: { min: 2, max: 64 }, + maxPhysicalSends: { min: 1, max: 10_000 }, + maxDistinctChildren: { min: 1, max: 1_024 }, +} as const; + +const bounded = (key: keyof typeof WORKFLOW_BUDGET_BOUNDS) => + z.number().int().min(WORKFLOW_BUDGET_BOUNDS[key].min).max(WORKFLOW_BUDGET_BOUNDS[key].max).optional(); + +/** Omitted fields keep the built-in ceiling, so an absent block is today's behavior byte for byte. */ +export const workflowBudgetSchema = z.object({ + maxConcurrentChildren: bounded("maxConcurrentChildren"), + maxPhysicalSends: bounded("maxPhysicalSends"), + maxDistinctChildren: bounded("maxDistinctChildren"), +}).strict(); + +export type WorkflowBudgetConfig = z.infer; + +export const WORKFLOW_BUDGET_SHAPE_HINT = "accepts only optional integer maxConcurrentChildren (2-64), maxPhysicalSends (1-10000) and maxDistinctChildren (1-1024)"; + +export function workflowBudgetConfigError(value: unknown): string | null { + if (!value || typeof value !== "object" || Array.isArray(value)) return null; + const budget = (value as Record).workflowBudget; + return budget === undefined || workflowBudgetSchema.safeParse(budget).success + ? null : "schema_invalid: workflowBudget: " + WORKFLOW_BUDGET_SHAPE_HINT; +} + +/** Malformed input resolves to no overrides, the same way a degraded disk block does. */ +export function readWorkflowBudgetConfig(config: Pick): WorkflowBudgetConfig { + const parsed = workflowBudgetSchema.safeParse(config.workflowBudget); + return parsed.success && parsed.data ? parsed.data : {}; +} diff --git a/src/lib/state-store-registrations.ts b/src/lib/state-store-registrations.ts index 2eb6488eb39..22973b92f58 100644 --- a/src/lib/state-store-registrations.ts +++ b/src/lib/state-store-registrations.ts @@ -11,6 +11,7 @@ import { reconcilePoolRotationState } from "../codex/pool-rotation"; import { reconcileCodexQuotaAccounts } from "../codex/quota"; import { reconcileQuotaRecovery, sweepExpiredQuotaRecovery } from "../codex/quota-401-recovery"; import { bindUpstreamRequestCompressionConfig } from "../server/responses/request-compression"; +import { bindWorkflowBudgetConfig } from "./workflow-budget-config"; import { listLiveCodexAccountIds, reconcileCodexRoutingHealth, @@ -82,6 +83,7 @@ export function setLiveStateStoreConfig(config: OcxConfig): void { // The one place the server's live config object is installed; detached configs (catalog sync) // never come through here, so the outbound compression opt-in follows the running server only. bindUpstreamRequestCompressionConfig(config); + bindWorkflowBudgetConfig(config); notifyLiveConfigObservers(config); } diff --git a/src/lib/workflow-budget-config.ts b/src/lib/workflow-budget-config.ts new file mode 100644 index 00000000000..26905568183 --- /dev/null +++ b/src/lib/workflow-budget-config.ts @@ -0,0 +1,39 @@ +/** + * The workflow-budget policy the running server enforces. + * + * The ceilings in workflow-budget.ts are module defaults, and a busy install that legitimately + * runs more parallel subagents than they allow had no way to say so short of patching the + * package. This binds the live config the same way the outbound compression opt-in does: the + * object is installed once, read per decision, and a dashboard write that mutates it in place + * takes effect on the next request without a restart. + * + * Only the three count ceilings move. The window, the interactive reserve and the tracked-root + * bound stay where they are: the window fixes each root's ring geometry when the root first + * appears, so changing it live would leave existing roots measured on a scale no caller reads. + */ +import { readWorkflowBudgetConfig } from "../config/schema/workflow-budget"; +import type { OcxConfig } from "../types"; +import { DEFAULT_WORKFLOW_BUDGET_POLICY, type WorkflowBudgetPolicy } from "./workflow-budget"; + +let boundConfig: Pick | null = null; + +export function bindWorkflowBudgetConfig(config: Pick | null): void { + boundConfig = config; +} + +export function resolveWorkflowBudgetPolicy(config: Pick): WorkflowBudgetPolicy { + const overrides = readWorkflowBudgetConfig(config); + return { + ...DEFAULT_WORKFLOW_BUDGET_POLICY, + ...(overrides.maxConcurrentChildren !== undefined ? { maxConcurrentChildren: overrides.maxConcurrentChildren } : {}), + ...(overrides.maxPhysicalSends !== undefined ? { maxPhysicalSends: overrides.maxPhysicalSends } : {}), + ...(overrides.maxDistinctChildren !== undefined ? { maxDistinctChildren: overrides.maxDistinctChildren } : {}), + }; +} + +/** Policy for the next admission or ceiling check; an unset block is the module default itself. */ +export function currentWorkflowBudgetPolicy(): WorkflowBudgetPolicy { + return boundConfig?.workflowBudget === undefined + ? DEFAULT_WORKFLOW_BUDGET_POLICY + : resolveWorkflowBudgetPolicy(boundConfig); +} diff --git a/src/server/management/route-registry.ts b/src/server/management/route-registry.ts index 4b10087659d..0c48f6e5973 100644 --- a/src/server/management/route-registry.ts +++ b/src/server/management/route-registry.ts @@ -396,6 +396,8 @@ export const MANAGEMENT_ROUTES: readonly ManagementRoute[] = [ { method: "PATCH", path: "/api/protocols/settings", module: "server/management/protocol-routes", mutates: true }, { method: "GET", path: "/api/workflow-budget", module: "server/management/workflow-budget-routes", mutates: false, exempt: { reason: "deferred-verb", why: "Reading a root's live budget is owed a CLI verb -- an operator staring at a 429 is usually already in a terminal -- but the ledger is process memory with no local transport to read it through, so the verb has to be an HTTP call the CLI does not yet make.", owner: "260915_workflow_budget_window wfc", ownerDoc: "devlog/_fin/260915_workflow_budget_window/030_wfc_diff_plan.md" } }, { method: "POST", path: "/api/workflow-budget/clear", module: "server/management/workflow-budget-routes", mutates: true, exempt: { reason: "deferred-verb", why: "Clearing one root is owed the same verb as the read above and for the same reason. It is deliberately not shipped as a verb in this work-phase: the read comes first, because an operator who cannot see which ceiling fired has no basis for deciding to forgive it.", owner: "260915_workflow_budget_window wfc", ownerDoc: "devlog/_fin/260915_workflow_budget_window/030_wfc_diff_plan.md" } }, + { method: "GET", path: "/api/workflow-budget/settings", module: "server/management/workflow-budget-routes", mutates: false, exempt: { reason: "local-transport", why: "The CLI reads the same workflowBudget block through ocx config get; the route only adds the built-in defaults and bounds the dashboard draws." } }, + { method: "PUT", path: "/api/workflow-budget/settings", module: "server/management/workflow-budget-routes", mutates: true, exempt: { reason: "deferred-verb", why: "A dedicated live CLI verb for the ceilings is deferred with the other workflow-budget verbs; ocx config set/unset edits the same block on disk and the running proxy picks it up on restart.", owner: "260915_workflow_budget_window wfc", ownerDoc: "devlog/_fin/260915_workflow_budget_window/030_wfc_diff_plan.md" } }, // server/management/request-history-routes { method: "GET", path: "/api/request-history", module: "server/management/request-history-routes", mutates: false }, // server/management/routing-analytics-routes diff --git a/src/server/management/workflow-budget-routes.ts b/src/server/management/workflow-budget-routes.ts index 4cdebd4572f..d252db0b2ba 100644 --- a/src/server/management/workflow-budget-routes.ts +++ b/src/server/management/workflow-budget-routes.ts @@ -10,14 +10,27 @@ * chain runs, so these handlers add no auth code of their own. The GET spends no user identity. * The POST clears one root's windowed count ceilings; it does not spend identity, and the * underlying ledger leaves in-flight concurrency and the token spend record untouched. + * + * `/api/workflow-budget/settings` reads and writes the per-install count ceilings. A write + * mutates the live config object, which the request path reads per decision, so it applies to + * the next admission without a restart and never resets what a root has already used. */ import { jsonResponse } from "../auth-cors"; import type { OcxConfig } from "../../types"; import type { ManagementContext } from "./context"; import { readManagementJsonBodyOr } from "./body"; +import { captureConfigTopLevelRollback, deleteConfigTopLevelKey } from "../../config/rebase-provenance"; +import { ConfigWritePublishedError } from "../../config/persist-unlocked"; +import { + WORKFLOW_BUDGET_BOUNDS, + WORKFLOW_BUDGET_SHAPE_HINT, + workflowBudgetSchema, +} from "../../config/schema/workflow-budget"; +import { resolveWorkflowBudgetPolicy } from "../../lib/workflow-budget-config"; import { clearWorkflowBudgetForRoot, + DEFAULT_WORKFLOW_BUDGET_POLICY, listTrackedWorkflowRoots, listWorkflowBudgetEvents, workflowBudgetSnapshot, @@ -27,6 +40,54 @@ import { const DEFAULT_LIMIT = 20; const MAX_LIMIT = WORKFLOW_EVENT_CAPACITY; const MAX_ROOT_ID_LENGTH = 200; +const INVALID_BODY = Symbol("invalid-body"); + +/** Effective ceilings plus what the dashboard needs to draw them: defaults, bounds, the fixed window. */ +function workflowBudgetSettings(config: OcxConfig) { + const policy = resolveWorkflowBudgetPolicy(config); + return { + maxConcurrentChildren: policy.maxConcurrentChildren, + maxPhysicalSends: policy.maxPhysicalSends, + maxDistinctChildren: policy.maxDistinctChildren, + interactiveReserve: policy.interactiveReserve, + windowMs: DEFAULT_WORKFLOW_BUDGET_POLICY.windowMs, + defaults: { + maxConcurrentChildren: DEFAULT_WORKFLOW_BUDGET_POLICY.maxConcurrentChildren, + maxPhysicalSends: DEFAULT_WORKFLOW_BUDGET_POLICY.maxPhysicalSends, + maxDistinctChildren: DEFAULT_WORKFLOW_BUDGET_POLICY.maxDistinctChildren, + }, + bounds: WORKFLOW_BUDGET_BOUNDS, + }; +} + +/** Partial PUTs preserve omitted ceilings; JSON null removes the block and restores the defaults. */ +async function putWorkflowBudgetSettings(ctx: ManagementContext): Promise { + const { req, config } = ctx; + const body = await readManagementJsonBodyOr(req, INVALID_BODY); + const parsed = body === null ? null : workflowBudgetSchema.safeParse(body); + if (parsed !== null && !parsed.success) { + return jsonResponse( + { error: { code: "invalid_workflow_budget", message: "body must be null or an object that " + WORKFLOW_BUDGET_SHAPE_HINT } }, + 400, + req, + config, + ); + } + // Resolve the saver before changing the live object: no await may interleave mutation/save. + const persist = ctx.deps.saveConfigPreservingClaudeCode + ?? (await import("../../config")).saveConfigPreservingClaudeCode; + const restore = captureConfigTopLevelRollback(config, ["workflowBudget"]); + try { + if (parsed === null) deleteConfigTopLevelKey(config, "workflowBudget"); + else config.workflowBudget = { ...config.workflowBudget, ...parsed.data }; + persist(config); + } catch (error) { + // Published bytes remain authoritative even if later bookkeeping failed. + if (!(error instanceof ConfigWritePublishedError)) restore(); + throw error; + } + return jsonResponse(workflowBudgetSettings(config), 200, req, config); +} /** * A root id is an opaque caller-thread token, not a path. Without a length cap a client @@ -60,6 +121,12 @@ function parseLimitParam(rawLimit: string | null): { ok: true; limit: number } | export async function handleWorkflowBudgetRoutes(ctx: ManagementContext): Promise { const { url, req, config } = ctx; + if (url.pathname === "/api/workflow-budget/settings" && req.method === "GET") { + return jsonResponse(workflowBudgetSettings(config), 200, req, config); + } + if (url.pathname === "/api/workflow-budget/settings" && req.method === "PUT") return putWorkflowBudgetSettings(ctx); + const policy = resolveWorkflowBudgetPolicy(config); + if (url.pathname === "/api/workflow-budget") { if (req.method !== "GET") return null; @@ -78,7 +145,7 @@ export async function handleWorkflowBudgetRoutes(ctx: ManagementContext): Promis if (rawRoot !== null) { const rootId = parseRootId(rawRoot); if (rootId === null) return invalidRootResponse(req, config); - const snapshot = workflowBudgetSnapshot(rootId); + const snapshot = workflowBudgetSnapshot(rootId, policy); // An unknown id is a 200 with `root: null`, not a 404: the operator asked what this // process currently holds for that token, and "nothing" is a legitimate answer. POST // /clear is the opposite — claiming to forgive a ceiling that was never tracked would @@ -98,7 +165,7 @@ export async function handleWorkflowBudgetRoutes(ctx: ManagementContext): Promis return jsonResponse( { - roots: listTrackedWorkflowRoots(limit), + roots: listTrackedWorkflowRoots(limit, policy), events: listWorkflowBudgetEvents(limit), }, 200, @@ -117,7 +184,7 @@ export async function handleWorkflowBudgetRoutes(ctx: ManagementContext): Promis const rootId = parseRootId(rawRoot); if (rootId === null) return invalidRootResponse(req, config); - const before = clearWorkflowBudgetForRoot(rootId); + const before = clearWorkflowBudgetForRoot(rootId, policy); if (!before) { return jsonResponse( { error: { code: "unknown_root", message: "root is not currently tracked" } }, diff --git a/src/server/responses/request-send-budget.ts b/src/server/responses/request-send-budget.ts index 27f1c5ac18e..0417a6adcf5 100644 --- a/src/server/responses/request-send-budget.ts +++ b/src/server/responses/request-send-budget.ts @@ -6,6 +6,7 @@ import { workflowSendCeilingReached, workflowSpendCeilingReached, } from "../../lib/workflow-budget"; +import { currentWorkflowBudgetPolicy } from "../../lib/workflow-budget-config"; import { poolContinuityRefusalReason, workflowRefusalResponse } from "../workflow-refusal"; import type { AttemptRecoveryKind, AttemptRecoveryWithheld } from "../../usage/log"; import { noteAttemptRecoveryWithheld, noteAttemptSend } from "../request-log"; @@ -93,7 +94,7 @@ export function createResponsesSendBudget( // dropping the record to make room would hand the fan-out a fresh allowance, which is the // laundering this ceiling exists to stop. The client is told the task needs a new grant // rather than being given a synthetic upstream error. - if (workflowSendCeilingReached(workflowRootId)) { + if (workflowSendCeilingReached(workflowRootId, currentWorkflowBudgetPolicy())) { // A log context exists here, unlike at HTTP admission, so the row this request writes is // marked synthetic rather than reading as a request that vanished with zero sends. return workflowRefusalResponse("workflow-sends-exhausted", logCtx, undefined, workflowRootId); diff --git a/src/server/workflow-refusal.ts b/src/server/workflow-refusal.ts index 58dab886a53..61c7b943b8f 100644 --- a/src/server/workflow-refusal.ts +++ b/src/server/workflow-refusal.ts @@ -27,6 +27,7 @@ import { type WorkflowLane, type WorkflowSpendDenialDetail, } from "../lib/workflow-budget"; +import { currentWorkflowBudgetPolicy } from "../lib/workflow-budget-config"; /** * What a caller needs to hand over for the refusal to become a row on `/api/logs`. @@ -142,7 +143,7 @@ export function admitHttpWorkflowTurn(headers: Headers): WorkflowDecision | unde const lane: WorkflowLane = rootId !== undefined && threadId !== undefined && threadId !== rootId ? "worker" : "interactive"; - return admitWorkflowTurn(rootId, lane, undefined, threadId); + return admitWorkflowTurn(rootId, lane, currentWorkflowBudgetPolicy(), threadId); } /** diff --git a/src/types/config.ts b/src/types/config.ts index b3ca6bd575c..9a0e49e8aba 100644 --- a/src/types/config.ts +++ b/src/types/config.ts @@ -875,6 +875,8 @@ export interface OcxConfig { reasoningRetention?: { enabled?: boolean; maxContextPercent?: number; maxTokens?: number }; /** Canonical ChatGPT reasoning replay allowances; both default off. */ nativeReasoningRetention?: { modelSwitch?: boolean; accountSwitch?: boolean }; + /** Per-root fan-out ceilings (src/lib/workflow-budget.ts); omitted fields keep 8 / 256 / 64. */ + workflowBudget?: { maxConcurrentChildren?: number; maxPhysicalSends?: number; maxDistinctChildren?: number }; /** Explicit opt-in; resumes Observe only using an existing trusted certificate and endpoints. */ desktopCompatibility?: { startOnProxyStart: boolean }; /** diff --git a/structure/gui-and-management-api.md b/structure/gui-and-management-api.md index 898df8af8d1..ef49522edeb 100644 --- a/structure/gui-and-management-api.md +++ b/structure/gui-and-management-api.md @@ -313,7 +313,7 @@ after the save, and a non-retryable skip (no managed catalog) is a clean save. C | Claude reset grants | `src/server/management/anthropic-reset-grant-routes.ts` — `GET /api/anthropic/reset-grants`, `POST /api/anthropic/reset-grants/consume` (lazy-loaded). Wire and fail-closed parsing live in `src/providers/anthropic-reset-grants.ts` (the Claude Code 2.1.278 `cedar_ember` contract, sent with `CLAUDE_CLI_USER_AGENT` from `src/providers/claude-cli-identity.ts`); the journal is `src/providers/anthropic-reset-grant-ledger.ts`: a cross-process `BEGIN IMMEDIATE` lock around every synchronous read-modify-write, a 90 s lease, the operation id reused as the upstream `request_id`, same-id retry only inside the vendor's ten-minute window, no settlement inferred from a re-read, and a fail-closed `500 journal_write_failed` when an answer cannot be recorded. Spending requires the `gui-session` principal. A journaled `reset`, including a same-id replay of that code, lifts that account cooldown only when it is a 429 quota refusal recorded before the operation was first journaled (the claim fence, `claimedAt`) and the journal's cleared windows cover every window the refusal named (both when it named none); it then drops the cached quota row and rereads usage with its own request rather than joining an older read. Model-family exclusions stay until the usage reread proves a family is no longer rejected. A failed reread leaves them in place. The HTTP result stays the journal answer if the reread fails. Other result codes do not admit. The dashboard owner is `gui/src/hooks/useAnthropicResetGrants.ts` with `gui/src/components/provider-workspace/AnthropicResetGrants.tsx` on the Anthropic OAuth rows of `ProviderAuthPanel`; after an unknown outcome the dialog only retries the same id. Design and audit record: [`../devlog/_fin/260923_claude_reset_grants/010_plan.md`](../devlog/_fin/260923_claude_reset_grants/010_plan.md). | | Combos | `src/server/management/combo-routes.ts` — `GET/PUT/DELETE /api/combos` own provider combination and failover definitions. `PUT` keeps a stored field the body omits (`cooldownMs`, `waitForCooldownMs`, `defaultEffortMode`, `reasoningEffortMode`, `imageInput`, `cooldownWaitPolicy`, per-target `lastResort`, JEV `decisionQuotaSignals`/`decisionQuotaTiers`/`sessionPin` on updates/rename); quota false/null clears sparsely and switching away removes quota policy under [the owning contract](providers/jev-decision.md#loaded-advisory-quota). Explicit values replace it and defaults are stored sparse. A combo validation 400 is `{ error, field? }`, where `field` is the top-level combo field of the first issue. | | Protocol paths | `src/server/management/protocol-routes.ts` (lazy-loaded) — `GET /api/protocols` returns the contract version, the resolved API surfaces and protocol settings, the policy revision and the feature vocabulary; with `?provider=` (one non-empty name of at most 200 characters without control characters, else 400 `invalid_provider`; an unconfigured name is 404 `unknown_provider`, neither echoing the name) it adds a `provider` block from `src/protocols/provider-summary.ts` ([Protocol Paths](data-planes/protocol-paths.md#provider-wire-summary)); `POST /api/protocols/plan` takes `{ model, inbound, features? }` (model at most 200 characters, at most 24 features, any other key refused with 400) and returns a `ProtocolPlanV1` with `basis: "preview"` from `src/protocols/plan-snapshot.ts`. Both are read-only, never log their input, and send nothing upstream. `PATCH /api/protocols/settings` takes `{ messagesEnabled?, unrepresentable?, rollout? }` (strict: unknown keys and wrong types are 400), validates and applies through `src/server/management/protocol-settings-patch.ts`, persists with `saveConfigPreservingClaudeCode`, restores the live config if the save fails (409 on lock contention, 500 otherwise), and answers with the fresh `GET /api/protocols` body; closing Messages also writes `claudeCode.enabled = false` through `commitClaudeCodeBlock` ([Protocol Paths](data-planes/protocol-paths.md#settings)). The CLI drives all three through `ocx api protocols`, `ocx api explain` and `ocx api policy` ([Protocol Paths](data-planes/protocol-paths.md#cli)); none carries a route-registry exemption. | -| Workflow budget | `src/server/management/workflow-budget-routes.ts` — `GET /api/workflow-budget` reads the tracked roots or one root, and `POST /api/workflow-budget/clear` clears exactly one. The clear moves the windowed send ring and the child map and nothing else: `active` belongs to turns still in flight, the spend ledger is a token budget an operator did not ask to forgive, and the lifetime send total survives so a clear cannot launder the record. A refusal event carries `spendScope` and `spendLimit` when a token ceiling fired, so the reason is readable without the config open beside it; no scope id is ever attached, because root ids are client thread headers and identity ids are credentials. Both are `deferred-verb` in the route registry — they are owed CLI verbs, and because the ledger is process memory there is no local projection the CLI could read instead. See [`../devlog/_fin/260915_workflow_budget_window/030_wfc_diff_plan.md`](../devlog/_fin/260915_workflow_budget_window/030_wfc_diff_plan.md). | +| Workflow budget | `src/server/management/workflow-budget-routes.ts` — `GET /api/workflow-budget` reads the tracked roots or one root, and `POST /api/workflow-budget/clear` clears exactly one. The clear moves the windowed send ring and the child map and nothing else: `active` belongs to turns still in flight, the spend ledger is a token budget an operator did not ask to forgive, and the lifetime send total survives so a clear cannot launder the record. A refusal event carries `spendScope` and `spendLimit` when a token ceiling fired, so the reason is readable without the config open beside it; no scope id is ever attached, because root ids are client thread headers and identity ids are credentials. Both are `deferred-verb` in the route registry — they are owed CLI verbs, and because the ledger is process memory there is no local projection the CLI could read instead. `GET`/`PUT /api/workflow-budget/settings` read and write the per-install count ceilings (`workflowBudget` in config: concurrent turns, sends and distinct children per window); the live config is bound in `src/lib/workflow-budget-config.ts` and read per decision by HTTP admission and the pre-dispatch send ceiling, so a write applies to the next request and never resets a root's usage. The window, interactive reserve and tracked-root bound stay fixed because a root's ring geometry is fixed when it first appears. The GET is `local-transport` (same block via `ocx config get`), the PUT `deferred-verb` with the other budget verbs. See [`../devlog/_fin/260915_workflow_budget_window/030_wfc_diff_plan.md`](../devlog/_fin/260915_workflow_budget_window/030_wfc_diff_plan.md). | | Codex accounts | `src/codex/auth-api/routes.ts` — `GET/POST/DELETE /api/codex-auth/accounts`, `PUT /api/codex-auth/accounts/alias`, `PUT /api/codex-auth/accounts/pause`, `PUT /api/codex-auth/accounts/pause-exhausted`, `POST /api/codex-auth/accounts/clear-cooldown`, `GET/PUT /api/codex-auth/active`, `PUT /api/codex-auth/auto-switch`, `PUT /api/codex-auth/pool-strategy`, `PUT /api/codex-auth/failover`, `GET /api/codex-auth/quota`, `GET /api/codex-auth/reset-credits` with `POST /api/codex-auth/reset-credits/consume`, and the login flow `POST /api/codex-auth/login`, `POST /api/codex-auth/login/code`, `POST /api/codex-auth/login/cancel`, `GET /api/codex-auth/login-status`. Per-account quota activation uses the existing `GET/PUT /api/settings` surface and `src/codex/quota-auto-refresh.ts`, keeping scheduled spending separate from credential/authentication mutation. Account ids are opaque handles and are serialized so the GUI can address an account; emails are masked and tokens are never serialized. New-account config commits add UI-managed selector bindings in the same config save; deletion deliberately retains existing bindings for fail-closed exact routing and re-add stability. Account mutations request catalog convergence only after config durability and expose only the boolean `catalogRefreshPending` completion projection. `showCodexCredits` is a default-off boolean in settings GET and successful PUT; partial PUT accepts a boolean and rolls back on persistence failure. See [credits exposure](providers/openai-accounts.md#display-only-codex-credits). | | Low-quota history | `src/server/management/low-quota-routes.ts` lazily serves authenticated `GET /api/codex-auth/low-quota-events` from the bounded ledger owned by the server handling the request. The response includes only that server’s account ids and notice/pause-save status; logs omit identifiers. The default notice records `logged`; a notice records `delivered` only after the injected or opted-in OS notifier sink succeeds, `failed` when it fails, and `cancelled` when its episode ends first, while a pause-save records `succeeded` after persistence succeeds. A timed-out in-flight save remains `pending` until it settles. Invalid limits return 400. The route itself emits no OS notification. | | Sidebar | `src/server/management/sidebar-routes.ts` — `GET/POST /api/github/star`, `GET /api/update/badge`, and `POST /api/update/desktop-snapshot`. The snapshot POST accepts the raw admin-token principal or the dedicated `local-desktop-snapshot-capability`; GUI sessions and requests carrying `Origin` cannot publish desktop state. A capability's bounded raw body is verified against its authenticated digest before JSON parsing or storage. Badge state is cosmetic and a failed poll degrades silently. | diff --git a/tests/cli/cli-headless-parity.test.ts b/tests/cli/cli-headless-parity.test.ts index ba4cdb8eb78..c304144e182 100644 --- a/tests/cli/cli-headless-parity.test.ts +++ b/tests/cli/cli-headless-parity.test.ts @@ -610,6 +610,7 @@ describe("headless GUI parity CLI", () => { ["/api/protocols", "ocx api protocols/explain/policy"], ["/api/settings", "ocx system"], ["/api/native-reasoning-retention", "(none — dedicated live CLI deferred; local policy via ocx config)"], + ["/api/workflow-budget/settings", "(none — dedicated live CLI deferred; local ceilings via ocx config)"], // Routing Intelligence (RI-04..RI-10): profiles + dry-run are mirrored by // `ocx route policy`. Analytics is GUI-first for now; the same request // history remains available through observe/index tooling. diff --git a/tests/config/config-workflow-budget.test.ts b/tests/config/config-workflow-budget.test.ts new file mode 100644 index 00000000000..fb4d0e98408 --- /dev/null +++ b/tests/config/config-workflow-budget.test.ts @@ -0,0 +1,56 @@ +import { describe, expect, spyOn, test } from "bun:test"; +import { getDefaultConfig, validateConfigCandidate } from "../../src/config"; +import { configDiagnosticsFromRaw } from "../../src/config/diagnostics"; +import { configSchema } from "../../src/config/schema/config-schema"; +import { warnDegradedTopLevelOptIns } from "../../src/config/load-degrade"; +import { resolveWorkflowBudgetPolicy } from "../../src/lib/workflow-budget-config"; +import { DEFAULT_WORKFLOW_BUDGET_POLICY } from "../../src/lib/workflow-budget"; + +const invalid = [null, [], true, "8", { maxConcurrentChildren: 1 }, { maxConcurrentChildren: 65 }, + { maxPhysicalSends: 0 }, { maxPhysicalSends: 2.5 }, { maxDistinctChildren: "64" }, + { maxDistinctChildren: 1_025 }, { windowMs: 60_000 }, { maxConcurrentChildren: 8, interactiveReserve: 0 }]; + +describe("workflow budget configuration", () => { + test("an absent block enforces the built-in policy itself", () => { + expect(getDefaultConfig().workflowBudget).toBeUndefined(); + expect(resolveWorkflowBudgetPolicy(getDefaultConfig())).toEqual(DEFAULT_WORKFLOW_BUDGET_POLICY); + }); + + test("valid partial blocks override only the named ceilings", () => { + const candidate = { ...getDefaultConfig(), workflowBudget: { maxConcurrentChildren: 16, maxPhysicalSends: 1_024 } }; + const parsed = validateConfigCandidate(candidate); + expect(parsed.ok).toBe(true); + if (!parsed.ok) return; + expect(parsed.config.workflowBudget).toEqual({ maxConcurrentChildren: 16, maxPhysicalSends: 1_024 }); + expect(resolveWorkflowBudgetPolicy(parsed.config)).toEqual({ + ...DEFAULT_WORKFLOW_BUDGET_POLICY, maxConcurrentChildren: 16, maxPhysicalSends: 1_024, + }); + }); + + test("invalid writes are refused while an invalid disk block falls back to the built-in ceilings", () => { + for (const budget of invalid) { + const candidate = { ...getDefaultConfig(), workflowBudget: budget }; + const result = validateConfigCandidate(candidate); + expect(result.ok).toBe(false); + if (!result.ok) expect(result.error).toContain("workflowBudget"); + const parsed = configSchema.parse(candidate); + expect(parsed.workflowBudget).toBeUndefined(); + expect(parsed.providers).toEqual(candidate.providers); + expect(resolveWorkflowBudgetPolicy(parsed)).toEqual(DEFAULT_WORKFLOW_BUDGET_POLICY); + const diagnostics = configDiagnosticsFromRaw(JSON.stringify(candidate)); + expect(diagnostics.error).toBeNull(); + expect(diagnostics.warnings?.join(" ")).toContain("workflowBudget ignored"); + } + }); + + test("loading a malformed block warns instead of failing silently", () => { + const warning = spyOn(console, "warn").mockImplementation(() => {}); + try { + const raw = { ...getDefaultConfig(), workflowBudget: { maxConcurrentChildren: 0 } }; + warnDegradedTopLevelOptIns(raw, configSchema.parse(raw)); + expect(warning.mock.calls.some(call => String(call[0]).includes("invalid workflowBudget"))).toBe(true); + } finally { + warning.mockRestore(); + } + }); +}); diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index e01f098a460..3376f412bf1 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -88,7 +88,7 @@ "protocol-argon-thought-boundary.test.ts": "responses", "responses-auto-pause-replay-owner.test.ts": "responses", "responses-account-auto-pause.test.ts": "responses", "oauth-account-auto-pause.test.ts": "oauth", "responses-account-auto-pause-wire-owner.test.ts": "responses", "oauth-upsert-paused-api-key.test.ts": "oauth", "management-account-auto-pause.test.ts": "server", "cli-account-key-pause.test.ts": "cli", "oauth-account-auto-pause-revision.test.ts": "oauth", "config-account-auto-pause-revisions.test.ts": "config", "codex-provider-routing.test.ts": "codex-integration", - "claude-picker-trust-windows.test.ts": "claude-integration", "claude-picker-windows.test.ts": "claude-integration", "config-native-reasoning-retention.test.ts": "config", "server-native-reasoning-retention-merge.test.ts": "server", "server-native-reasoning-retention.test.ts": "server", "responses-retained-reasoning-stream-recovery.test.ts": "responses", + "claude-picker-trust-windows.test.ts": "claude-integration", "claude-picker-windows.test.ts": "claude-integration", "config-native-reasoning-retention.test.ts": "config", "server-native-reasoning-retention-merge.test.ts": "server", "server-native-reasoning-retention.test.ts": "server", "config-workflow-budget.test.ts": "config", "server-workflow-budget-settings.test.ts": "server", "responses-retained-reasoning-stream-recovery.test.ts": "responses", "advisor-retention.test.ts": "advisor", "responses-native-key-cooldown.test.ts": "responses", "responses-run-turn-key-readmission.test.ts": "responses", "qoder-spawn-readmission.test.ts": "adapters", "coding-agent-windows-dispatch.test.ts": "adapters", "coding-agent-native-empty-argv.test.ts": "windows", "api-key-selection-cooldown.test.ts": "providers", "api-key-auto-pause-distribution.test.ts": "providers", "explicit-key-pause-oauth-boundary.test.ts": "providers", "server-chat-native-key-selection.test.ts": "server", "compact-key-selection.test.ts": "responses", "messages-native-key-selection.test.ts": "claude-integration", diff --git a/tests/server/server-workflow-budget-settings.test.ts b/tests/server/server-workflow-budget-settings.test.ts new file mode 100644 index 00000000000..9ca61c44da5 --- /dev/null +++ b/tests/server/server-workflow-budget-settings.test.ts @@ -0,0 +1,150 @@ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { getDefaultConfig } from "../../src/config"; +import { ConfigWritePublishedError } from "../../src/config/persist-unlocked"; +import { configRebaseDeletionKeys } from "../../src/config/rebase-provenance"; +import { handleManagementAPI } from "../../src/server/management-api"; +import { handleWorkflowBudgetRoutes } from "../../src/server/management/workflow-budget-routes"; +import type { ManagementContext } from "../../src/server/management/context"; +import { admitHttpWorkflowTurn } from "../../src/server/workflow-refusal"; +import { bindWorkflowBudgetConfig, currentWorkflowBudgetPolicy } from "../../src/lib/workflow-budget-config"; +import { + admitWorkflowTurn, + chargeWorkflowSends, + resetWorkflowBudgetsForTest, + workflowSendCeilingReached, + type WorkflowAdmission, +} from "../../src/lib/workflow-budget"; +import type { OcxConfig } from "../../src/types"; +import { ManagementRequest } from "../helpers/management-auth"; + +const defaults = { maxConcurrentChildren: 8, maxPhysicalSends: 256, maxDistinctChildren: 64 }; +const PATH = "/api/workflow-budget/settings"; + +function harness(saveError?: Error) { + const config = getDefaultConfig(); + const saved: OcxConfig[] = []; + async function call(method: string, body?: unknown, rawBody?: string, pathname = PATH) { + const url = new URL("http://localhost" + pathname); + return handleWorkflowBudgetRoutes({ + url, req: new Request(url, { method, ...(method === "GET" ? {} : { + headers: { "content-type": "application/json" }, body: rawBody ?? JSON.stringify(body), + }) }), + config, version: "test", deps: { saveConfigPreservingClaudeCode: (candidate: OcxConfig) => { + saved.push(structuredClone(candidate)); + if (saveError) throw saveError; + } }, + } as unknown as ManagementContext); + } + return { config, saved, call }; +} + +function workerHeaders(child: string): Headers { + return new Headers({ "x-codex-parent-thread-id": "root-live", "thread-id": child }); +} + +beforeEach(() => { resetWorkflowBudgetsForTest(); }); +afterEach(() => { bindWorkflowBudgetConfig(null); resetWorkflowBudgetsForTest(); }); + +describe("workflow budget settings route", () => { + test("management dispatch serves the effective ceilings with defaults, bounds and the fixed window", async () => { + const config = getDefaultConfig(); + const url = new URL("http://127.0.0.1:10100" + PATH); + const response = await handleManagementAPI(new ManagementRequest(url), url, config, {}); + expect(response?.status).toBe(200); + expect(await response!.json()).toEqual({ + ...defaults, interactiveReserve: 1, windowMs: 600_000, defaults, + bounds: { + maxConcurrentChildren: { min: 2, max: 64 }, + maxPhysicalSends: { min: 1, max: 10_000 }, + maxDistinctChildren: { min: 1, max: 1_024 }, + }, + }); + }); + + test("PUT preserves omitted ceilings, persists the block and null restores the defaults", async () => { + const h = harness(); + const original = structuredClone(h.config); + expect((await h.call("PUT", { maxConcurrentChildren: 16 }))?.status).toBe(200); + const second = await (await h.call("PUT", { maxPhysicalSends: 1_024 }))!.json() as Record; + expect(second).toMatchObject({ maxConcurrentChildren: 16, maxPhysicalSends: 1_024, maxDistinctChildren: 64 }); + expect(h.config.workflowBudget).toEqual({ maxConcurrentChildren: 16, maxPhysicalSends: 1_024 }); + const { workflowBudget: _budget, ...rest } = h.config; + expect(rest).toEqual(original); + const reset = await (await h.call("PUT", null))!.json() as Record; + expect(reset).toMatchObject(defaults); + expect(h.config.workflowBudget).toBeUndefined(); + expect(configRebaseDeletionKeys(h.config).has("workflowBudget")).toBe(true); + expect(h.saved).toHaveLength(3); + }); + + test("out-of-range, unknown and malformed bodies are refused before mutation or save", async () => { + const h = harness(); + h.config.workflowBudget = { maxConcurrentChildren: 12 }; + const original = structuredClone(h.config); + for (const body of [[], true, { maxConcurrentChildren: 1 }, { maxPhysicalSends: 10_001 }, + { maxDistinctChildren: 3.5 }, { windowMs: 1 }, { maxConcurrentChildren: "12" }]) { + expect((await h.call("PUT", body))?.status).toBe(400); + } + expect((await h.call("PUT", undefined, "{"))?.status).toBe(400); + expect(h.config).toEqual(original); + expect(h.saved).toHaveLength(0); + }); + + test("a failed save restores the previous block; a published write keeps the new one", async () => { + const failed = harness(new Error("fixture-save-failed")); + const previous = { maxConcurrentChildren: 12 }; + failed.config.workflowBudget = previous; + await expect(failed.call("PUT", { maxConcurrentChildren: 20 })).rejects.toThrow("fixture-save-failed"); + expect(failed.config.workflowBudget).toBe(previous); + await expect(failed.call("PUT", null)).rejects.toThrow("fixture-save-failed"); + expect(failed.config.workflowBudget).toBe(previous); + expect(configRebaseDeletionKeys(failed.config).has("workflowBudget")).toBe(false); + const published = harness(new ConfigWritePublishedError(new Error("fixture-bookkeeping-failed"))); + await expect(published.call("PUT", { maxConcurrentChildren: 20 })).rejects.toBeInstanceOf(ConfigWritePublishedError); + expect(published.config.workflowBudget).toEqual({ maxConcurrentChildren: 20 }); + }); + + test("unsupported methods fall through", async () => { + const h = harness(); + expect(await h.call("POST", {})).toBeNull(); + expect(h.saved).toHaveLength(0); + }); +}); + +describe("live workflow budget policy", () => { + test("a dashboard write raises the worker concurrency ceiling on the next admission", async () => { + const h = harness(); + bindWorkflowBudgetConfig(h.config); + const leases: WorkflowAdmission[] = []; + const admitWorker = (index: number) => { + const decision = admitHttpWorkflowTurn(workerHeaders("child-" + index)); + if (decision?.admitted) leases.push(decision.lease); + return decision; + }; + // Default: eight slots, one held for the interactive turn, so the eighth worker is refused. + for (let index = 0; index < 7; index += 1) expect(admitWorker(index)?.admitted).toBe(true); + expect(admitWorker(7)).toMatchObject({ admitted: false, reason: "workflow-concurrency-exhausted" }); + expect((await h.call("PUT", { maxConcurrentChildren: 10 }))?.status).toBe(200); + expect(admitWorker(7)?.admitted).toBe(true); + expect(admitWorker(8)?.admitted).toBe(true); + expect(admitWorker(9)).toMatchObject({ admitted: false, reason: "workflow-concurrency-exhausted" }); + for (const lease of leases) lease.release(); + }); + + test("the send and child ceilings follow the bound config and fall back after a reset", async () => { + const h = harness(); + bindWorkflowBudgetConfig(h.config); + expect((await h.call("PUT", { maxPhysicalSends: 3, maxDistinctChildren: 2 }))?.status).toBe(200); + const policy = currentWorkflowBudgetPolicy(); + for (const child of ["a", "b"]) { + const decision = admitWorkflowTurn("root-sends", "worker", policy, child); + expect(decision?.admitted).toBe(true); + if (decision?.admitted) decision.lease.release(); + } + expect(admitWorkflowTurn("root-sends", "worker", policy, "c")).toMatchObject({ admitted: false, reason: "workflow-children-exhausted" }); + chargeWorkflowSends("root-sends", 3); + expect(workflowSendCeilingReached("root-sends", currentWorkflowBudgetPolicy())).toBe(true); + expect((await h.call("PUT", null))?.status).toBe(200); + expect(workflowSendCeilingReached("root-sends", currentWorkflowBudgetPolicy())).toBe(false); + }); +}); From 899cc4fe4da796006dbd526039777499c67927c5 Mon Sep 17 00:00:00 2001 From: JUN Date: Fri, 30 Oct 2026 20:38:58 +0900 Subject: [PATCH 08/42] docs(config): say when the workflow send ceiling is checked (#7987) maxPhysicalSends is compared when a request is admitted (workflowSendCeilingReached); sends are charged afterwards without a further check, so retries inside an admitted request and concurrent admissions can exceed it. The row added in #7960 read as a strict cap. --- docs-site/src/content/docs/reference/configuration/server.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs-site/src/content/docs/reference/configuration/server.md b/docs-site/src/content/docs/reference/configuration/server.md index 59aa34c7602..98636bbedba 100644 --- a/docs-site/src/content/docs/reference/configuration/server.md +++ b/docs-site/src/content/docs/reference/configuration/server.md @@ -95,7 +95,7 @@ OpenCodex bounds how much one task may fan out. A task is the root conversation, | Field | Default | Allowed | Meaning | | --- | --- | --- | --- | | `maxConcurrentChildren` | `8` | 2–64 | Turns one task may run at the same time. One slot is always reserved for the task's own conversation, so subagents may use one fewer. | -| `maxPhysicalSends` | `256` | 1–10000 | Upstream sends, retries included, one task may make inside the 10-minute window. | +| `maxPhysicalSends` | `256` | 1–10000 | Upstream sends, retries included, after which the task's new requests are refused for the rest of the 10-minute window. The count is checked when a request is admitted, so retries inside a request that was already admitted, and requests admitted concurrently, can take the total past the ceiling. | | `maxDistinctChildren` | `64` | 1–1024 | Distinct subagent threads one task may start inside the same window. | The defaults suit most installs. Raise them when a workflow that legitimately runs many subagents in parallel keeps hitting a ceiling; lower them to stop a runaway fan-out sooner: From 4fc870074e8030cd1e0a35a6a5c707f208e51b93 Mon Sep 17 00:00:00 2001 From: JUN Date: Fri, 30 Oct 2026 20:39:03 +0900 Subject: [PATCH 09/42] fix(kiro): exclude runtime callbacks from retry snapshot (#7981) --- src/adapters/kiro/adapter.ts | 5 ++++- structure/providers/kiro.md | 1 + .../providers/kiro/kiro-single-final.test.ts | 22 +++++++++++++++---- 3 files changed, 23 insertions(+), 5 deletions(-) diff --git a/src/adapters/kiro/adapter.ts b/src/adapters/kiro/adapter.ts index 11285f97573..b4d51979dd2 100644 --- a/src/adapters/kiro/adapter.ts +++ b/src/adapters/kiro/adapter.ts @@ -297,7 +297,10 @@ export function createKiroAdapter(provider: OcxProviderConfig): ProviderAdapter toolNameMap = built.nameMap; conversationId = built.conversationId; completionMode = built.completionMode; - requestSnapshot = structuredClone(parsed); + // Runtime callbacks belong to the original request, not the retry payload. + requestSnapshot = structuredClone(Object.fromEntries( + Object.entries(parsed).filter(([, value]) => typeof value !== "function"), + )) as OcxParsedRequest; firstRequestBodyBytes = Buffer.byteLength(built.request.body); requestAbortSignal = incoming?.abortSignal; return built.request; diff --git a/structure/providers/kiro.md b/structure/providers/kiro.md index f324a6b1d0c..c17cb4ee260 100644 --- a/structure/providers/kiro.md +++ b/structure/providers/kiro.md @@ -159,6 +159,7 @@ budget until emitted, discarded, or cancelled; replay collectors are released af retry construction. Native `END_TURN` and `STOP_SEQUENCE` alone do not distinguish progress from an answer and therefore still require validation. Normal private completion and real tool calls need no completion retry. +The retry snapshots cloneable request fields without top-level runtime callbacks, leaving the original request intact. A turn can carry a real tool call and the private completion call together, in either order. The turn then continues through the real tools: the parser drops the completion call, ends the diff --git a/tests/providers/kiro/kiro-single-final.test.ts b/tests/providers/kiro/kiro-single-final.test.ts index a4bac94faed..12d09c28d45 100644 --- a/tests/providers/kiro/kiro-single-final.test.ts +++ b/tests/providers/kiro/kiro-single-final.test.ts @@ -38,17 +38,19 @@ function response(frames: Uint8Array[]): Response { } afterEach(resetKiroThrottleStateForTests); -async function run(first: Uint8Array[], retry: Uint8Array[], buffered = false) { +async function run(first: Uint8Array[], retry: Uint8Array[], buffered = false, input = structuredClone(parsed)) { const adapter = createKiroAdapter(provider); const budget = createTranslatorBudget(); const sends: number[] = []; const events: AdapterEvent[] = []; let visibleAtRetry: AdapterEvent[] = []; let physicalRequests = 0; + const bodies: string[] = []; try { - const request = await adapter.buildRequest(structuredClone(parsed)); + const request = await adapter.buildRequest(input); const upstream = await adapter.fetchResponse!(request, { - executor: (async () => { + executor: (async (_url, init) => { + bodies.push(String(init?.body)); if (++physicalRequests === 1) return response(first); visibleAtRetry = events.filter(event => event.type === "text_delta"); return response(retry); @@ -59,13 +61,25 @@ async function run(first: Uint8Array[], retry: Uint8Array[], buffered = false) { else for await (const event of adapter.parseStream(upstream, budget)) events.push(event); if (buffered) for (const event of events) releaseTranslatedEvent(event, budget); expect(budget.snapshot().currentBytes).toBe(0); - return { events, sends, physicalRequests, visibleAtRetry }; + return { events, sends, physicalRequests, visibleAtRetry, bodies }; } finally { budget.dispose(); } } describe("Kiro single final answer (#6270)", () => { + test("advisor guard survives request building and completion retry", async () => { + const guard: NonNullable = async function* () {}; + const input = { ...structuredClone(parsed), _advisorGuard: guard }; + const answer = "The workspace is ready."; + const { events, physicalRequests, bodies } = await run([text(answer)], [text(answer)], false, input); + expect(input._advisorGuard).toBe(guard); + expect(physicalRequests).toBe(2); + expect(bodies[1]).toContain("Inspect the workspace."); + expect(bodies[1]).toContain("claude-opus-5.5"); + expect(events.at(-1)).toMatchObject({ type: "done", endTurn: true }); + }); + for (const buffered of [false, true]) { test.each(["END_TURN", "STOP_SEQUENCE", undefined])( `plain text ending is held through validation (stop=%s, buffered=${buffered})`, From 853508eb629abe05fe290e84b0f6687d9682a7ad Mon Sep 17 00:00:00 2001 From: Epinephrine Date: Fri, 30 Oct 2026 11:41:11 +0000 Subject: [PATCH 10/42] fix(windows): snapshot ACL failure diagnostics (#7970) * fix(windows): snapshot ACL failure diagnostics * docs(structure): record the single-read ACL diagnostic snapshot --------- Co-authored-by: JUN <243035832+lidge-jun@users.noreply.github.com> --- src/lib/windows-secret-acl.ts | 76 ++++--- structure/ops/service-and-sidecars.md | 2 +- .../windows-secret-acl-diagnostics.test.ts | 196 ++++++++++++++++++ 3 files changed, 244 insertions(+), 30 deletions(-) diff --git a/src/lib/windows-secret-acl.ts b/src/lib/windows-secret-acl.ts index 7b423974d9e..c7d39f38df7 100644 --- a/src/lib/windows-secret-acl.ts +++ b/src/lib/windows-secret-acl.ts @@ -1207,37 +1207,55 @@ function isTimeoutError(error: unknown): boolean { * which is optional and simply omitted when it is not an allowlisted code. */ function sanitizeAclFailure(cause: unknown): AclFailureDiagnostics | undefined { - if (!cause || typeof cause !== "object" || !("aclFailure" in cause)) return undefined; - const raw = (cause as { aclFailure?: unknown }).aclFailure; - if (!raw || typeof raw !== "object") return undefined; - const f = raw as Record; - if (typeof f.step !== "string" || !ICACLS_STEPS.has(f.step)) return undefined; - if ( - f.stage !== "deadline_before_spawn" && - f.stage !== "spawn_failed" && - f.stage !== "runner_rejected" && - f.stage !== "runner_watchdog" && - f.stage !== "child_result" - ) { - return undefined; - } - if (typeof f.elapsedMs !== "number" || !Number.isFinite(f.elapsedMs)) return undefined; - if (typeof f.budgetMs !== "number" || !Number.isFinite(f.budgetMs)) return undefined; - if (typeof f.success !== "boolean" || typeof f.timedOut !== "boolean") return undefined; - if (f.exitCode !== null && (typeof f.exitCode !== "number" || !Number.isFinite(f.exitCode))) { + // Every read off the untrusted cause — the `in` check, the `aclFailure` + // access and each field — may be a throwing accessor or a getter that + // returns a different value on the next read. Each field is therefore read + // exactly once into a local, and only that local is validated and copied; + // a read that throws drops the whole record rather than letting a raw + // hostile exception replace the sanitized required-mode error. + try { + if (!cause || typeof cause !== "object" || !("aclFailure" in cause)) return undefined; + const raw = (cause as { aclFailure?: unknown }).aclFailure; + if (!raw || typeof raw !== "object") return undefined; + const f = raw as Record; + const step = f.step; + if (typeof step !== "string" || !ICACLS_STEPS.has(step)) return undefined; + const stage = f.stage; + if ( + stage !== "deadline_before_spawn" && + stage !== "spawn_failed" && + stage !== "runner_rejected" && + stage !== "runner_watchdog" && + stage !== "child_result" + ) { + return undefined; + } + const elapsedMs = f.elapsedMs; + if (typeof elapsedMs !== "number" || !Number.isFinite(elapsedMs)) return undefined; + const budgetMs = f.budgetMs; + if (typeof budgetMs !== "number" || !Number.isFinite(budgetMs)) return undefined; + const success = f.success; + if (typeof success !== "boolean") return undefined; + const timedOut = f.timedOut; + if (typeof timedOut !== "boolean") return undefined; + const exitCode = f.exitCode; + if (exitCode !== null && (typeof exitCode !== "number" || !Number.isFinite(exitCode))) { + return undefined; + } + const spawnErrorCode = allowlistedSpawnErrorCode(f.spawnErrorCode); + return { + step: step as IcaclsStep, + stage, + elapsedMs, + budgetMs, + success, + exitCode: exitCode as number | null, + timedOut, + ...(spawnErrorCode !== undefined ? { spawnErrorCode } : {}), + }; + } catch { return undefined; } - const spawnErrorCode = allowlistedSpawnErrorCode(f.spawnErrorCode); - return { - step: f.step as IcaclsStep, - stage: f.stage, - elapsedMs: f.elapsedMs, - budgetMs: f.budgetMs, - success: f.success, - exitCode: f.exitCode as number | null, - timedOut: f.timedOut, - ...(spawnErrorCode !== undefined ? { spawnErrorCode } : {}), - }; } /** Preserve only the bounded machine-readable cause on a sanitized public error. */ diff --git a/structure/ops/service-and-sidecars.md b/structure/ops/service-and-sidecars.md index 71781fd0a71..00955b3e463 100644 --- a/structure/ops/service-and-sidecars.md +++ b/structure/ops/service-and-sidecars.md @@ -122,7 +122,7 @@ enumeration twice made a measured 12.3-second fallback cost roughly 25 seconds b and every async ACL hardening flight and deadline-surviving `icacls.exe` runner under that home have settled. `src/config/paths.ts` owns the barrier: a timeout verdict alone does not make the home removable. The contract is exercised by `tests/server/server-stop-config-hardening.test.ts`. -`src/lib/windows-secret-acl.ts` retains removal ownership for an entire async harden, including principal lookup, successive ACL commands, retries and diagnostics, as well as tracking runners that outlive their deadlines. The exact-path timeout indicator keeps its existing meaning for atomic-file fallback. The Bun regression in `tests/windows/windows-secret-acl-removal-flight.test.ts` holds a normal command and verifies that removal remains blocked between successive commands. Required-mode failures keep their sanitized ETIMEDOUT/EICACLS semantics and additionally carry a bounded `aclFailure` record — fixed step and stage enums (`deadline_before_spawn`/`spawn_failed`/`runner_rejected`/`runner_watchdog`/`child_result`), measured elapsed and requested budget, success, exit code, and an allowlisted spawn code — copied field-by-field so no path, argv, SID, or child output crosses into the thrown error; `tests/windows/windows-secret-acl-diagnostics.test.ts` covers the boundary. Lifecycle acquisition opts into `verifyExistingRuntimeDirectory` before mutating its shared Windows runtime root. `src/lib/windows-owner-acl.ts` verifies a protected private DACL with the serving user's inheritable Full Control; an Administrators-owned runtime directory also qualifies when that group is enabled in the serving token, independently of its default owner. The ordinary private secret-file/pairing policy is unchanged. Verification is bound to unchanged device/inode/ctime and the original deadline, creates no hardening memo, and never falls through to the legacy name-only shortcut. Refusal retains required hardening and fail-closed errors. The root check does not certify descendants; the lifecycle database is hardened separately. Regression coverage is in `tests/windows/windows-lifecycle-root-acl.test.ts`. +`src/lib/windows-secret-acl.ts` retains removal ownership for an entire async harden, including principal lookup, successive ACL commands, retries and diagnostics, as well as tracking runners that outlive their deadlines. The exact-path timeout indicator keeps its existing meaning for atomic-file fallback. The Bun regression in `tests/windows/windows-secret-acl-removal-flight.test.ts` holds a normal command and verifies that removal remains blocked between successive commands. Required-mode failures keep their sanitized ETIMEDOUT/EICACLS semantics and additionally carry a bounded `aclFailure` record — fixed step and stage enums (`deadline_before_spawn`/`spawn_failed`/`runner_rejected`/`runner_watchdog`/`child_result`), measured elapsed and requested budget, success, exit code, and an allowlisted spawn code — copied field-by-field so no path, argv, SID, or child output crosses into the thrown error. Each field is read from the cause exactly once and that same primitive is both validated and copied, so a changing getter cannot pass validation with one value and publish another; a getter or Proxy trap that throws drops the whole record while the sanitized required-mode error is still thrown; `tests/windows/windows-secret-acl-diagnostics.test.ts` covers the boundary. Lifecycle acquisition opts into `verifyExistingRuntimeDirectory` before mutating its shared Windows runtime root. `src/lib/windows-owner-acl.ts` verifies a protected private DACL with the serving user's inheritable Full Control; an Administrators-owned runtime directory also qualifies when that group is enabled in the serving token, independently of its default owner. The ordinary private secret-file/pairing policy is unchanged. Verification is bound to unchanged device/inode/ctime and the original deadline, creates no hardening memo, and never falls through to the legacy name-only shortcut. Refusal retains required hardening and fail-closed errors. The root check does not certify descendants; the lifecycle database is hardened separately. Regression coverage is in `tests/windows/windows-lifecycle-root-acl.test.ts`. Removal ownership captures lexical and canonical path identities before each work item or child starts, including Windows short names and directory aliases; deadline survivors retain the same identities. A deleted temporary file is captured through its existing parent. Unreadable identity at capture stays conservatively matched until settlement. Removal compares the requested root against those retained identities, so deleting or retargeting an alias cannot release the original root early. The same Bun regression verifies aliases changed mid-flight still block removal. Matching covers captured and current alias destinations; a second retarget by a writer of the protected directory is out of scope. diff --git a/tests/windows/windows-secret-acl-diagnostics.test.ts b/tests/windows/windows-secret-acl-diagnostics.test.ts index c671ed9b1f7..6209b99f697 100644 --- a/tests/windows/windows-secret-acl-diagnostics.test.ts +++ b/tests/windows/windows-secret-acl-diagnostics.test.ts @@ -394,3 +394,199 @@ describe("aclFailure diagnostics on required harden errors", () => { expect(result.diagnostics).not.toContain(ownerSid); }); }); + +describe("aclFailure boundary against an untrusted runner-thrown cause", () => { + // A runner that throws instead of returning a result hands its raw error to + // the required-mode sanitizer. Every aclFailure field on that error is + // untrusted: a getter may return a different value on each read or throw, and + // a Proxy may trap `in` or `get`. The sanitizer must read each field once, + // validate and copy only that local value, and drop the whole record when any + // read throws — a hostile exception must never replace the sanitized error. + const throwingRunnerCause = (aclFailure: unknown): Error => + Object.assign(new Error("injected runner failure"), { code: "EICACLS", aclFailure }); + + test("a changing step getter cannot swap the validated value for a path at copy time", () => { + const filePath = secretFile(); + let stepReads = 0; + let exitCodeReads = 0; + const hostile = { + get step() { stepReads += 1; return stepReads <= 2 ? "/grant:r" : filePath; }, + stage: "child_result", + elapsedMs: 1, + budgetMs: 100, + success: false, + get exitCode() { exitCodeReads += 1; return exitCodeReads <= 3 ? 5 : `marker:${filePath}`; }, + timedOut: false, + }; + setIcaclsRunnerForTests(() => { throw throwingRunnerCause(hostile); }); + + const error = thrownAclError(() => hardenSecretPath(filePath, { required: true })); + + expect(error.code).toBe("EICACLS"); + // Validated values — never the later hostile reads — reach the record. + expect(error.aclFailure).toMatchObject({ step: "/grant:r", stage: "child_result", exitCode: 5 }); + expect(JSON.stringify(error.aclFailure)).not.toContain(filePath); + }); + + test("every copied field is read exactly once, keeping valid edge values", () => { + const filePath = secretFile(); + const reads: Record = {}; + const field = (name: string, value: T): PropertyDescriptor => ({ + enumerable: true, + get() { reads[name] = (reads[name] ?? 0) + 1; return value; }, + }); + const hostile = Object.defineProperties({}, { + step: field("step", "/grant:r"), + stage: field("stage", "child_result"), + elapsedMs: field("elapsedMs", -1), + budgetMs: field("budgetMs", 0), + success: field("success", false), + exitCode: field("exitCode", null), + timedOut: field("timedOut", false), + }); + setIcaclsRunnerForTests(() => { throw throwingRunnerCause(hostile); }); + + const error = thrownAclError(() => hardenSecretPath(filePath, { required: true })); + + expect(error.code).toBe("EICACLS"); + expect(error.aclFailure).toEqual({ + step: "/grant:r", + stage: "child_result", + elapsedMs: -1, + budgetMs: 0, + success: false, + exitCode: null, + timedOut: false, + }); + for (const name of ["step", "stage", "elapsedMs", "budgetMs", "success", "exitCode", "timedOut"]) { + expect(reads[name]).toBe(1); + } + }); + + test("an invalid first read drops the record even when a later read would pass", () => { + const filePath = secretFile(); + let stepReads = 0; + const hostile = { + get step() { stepReads += 1; return stepReads === 1 ? filePath : "/grant:r"; }, + stage: "child_result", + elapsedMs: 1, + budgetMs: 100, + success: false, + exitCode: 5, + timedOut: false, + }; + setIcaclsRunnerForTests(() => { throw throwingRunnerCause(hostile); }); + + const error = thrownAclError(() => hardenSecretPath(filePath, { required: true })); + + expect(error.code).toBe("EICACLS"); + expect(error.aclFailure).toBeUndefined(); + }); + + test("an invalid success field short-circuits before the timedOut getter runs", () => { + const filePath = secretFile(); + let timedOutReads = 0; + const hostile = { + step: "/grant:r", + stage: "child_result", + elapsedMs: 1, + budgetMs: 100, + success: "yes" as unknown as boolean, + get timedOut() { timedOutReads += 1; return false; }, + exitCode: 5, + }; + setIcaclsRunnerForTests(() => { throw throwingRunnerCause(hostile); }); + + const error = thrownAclError(() => hardenSecretPath(filePath, { required: true })); + + expect(error.code).toBe("EICACLS"); + expect(error.aclFailure).toBeUndefined(); + // Original guard read timedOut only after success passed; keep that order. + expect(timedOutReads).toBe(0); + }); + + test("a throwing aclFailure field getter drops the record but keeps the sanitized error", () => { + const filePath = secretFile(); + const hostile = Object.defineProperty({ + stage: "child_result", + elapsedMs: 1, + budgetMs: 100, + success: false, + exitCode: 5, + timedOut: false, + }, "step", { + enumerable: true, + get() { throw new Error(`getter-marker ${filePath}`); }, + }); + setIcaclsRunnerForTests(() => { throw throwingRunnerCause(hostile); }); + + const error = thrownAclError(() => hardenSecretPath(filePath, { required: true })); + + expect(error.code).toBe("EICACLS"); + expect(error.aclFailure).toBeUndefined(); + expect(error.message).not.toContain("getter-marker"); + expect(error.message).not.toContain(filePath); + }); + + test("proxy has/get traps on the untrusted cause drop the record without replacing the error", () => { + const wellFormed = { + step: "/grant:r", + stage: "child_result", + elapsedMs: 1, + budgetMs: 100, + success: false, + exitCode: 5, + timedOut: false, + }; + + // The record's own get trap throws when step is read. + const recordProxy = new Proxy({ ...wellFormed }, { + get(target, prop) { + if (prop === "step") throw new Error("record-get-marker"); + return (target as Record)[prop]; + }, + }); + const filePath1 = secretFile("secret-a.json"); + setIcaclsRunnerForTests(() => { throw throwingRunnerCause(recordProxy); }); + const first = thrownAclError(() => hardenSecretPath(filePath1, { required: true })); + expect(first.code).toBe("EICACLS"); + expect(first.aclFailure).toBeUndefined(); + expect(first.message).not.toContain("record-get-marker"); + + // The cause's own has trap throws only for the aclFailure key — a trap that + // also fires on the separate cause.code boundary is outside this fix. + const filePath2 = secretFile("secret-b.json"); + const causeProxy = new Proxy(throwingRunnerCause({ ...wellFormed }), { + has(target, prop) { + if (prop === "aclFailure") throw new Error(`cause-has-marker ${filePath2}`); + return prop in target; + }, + }); + setIcaclsRunnerForTests(() => { throw causeProxy; }); + const second = thrownAclError(() => hardenSecretPath(filePath2, { required: true })); + expect(second.code).toBe("EICACLS"); + expect(second.aclFailure).toBeUndefined(); + expect(second.message).not.toContain("cause-has-marker"); + expect(second.message).not.toContain(filePath2); + }); + + test("a well-formed plain record and its allowlisted spawn code survive verbatim", () => { + const filePath = secretFile(); + const clean = { + step: "/remove:g", + stage: "spawn_failed", + elapsedMs: 2, + budgetMs: 50, + success: false, + exitCode: null, + timedOut: false, + spawnErrorCode: "ENOENT", + }; + setIcaclsRunnerForTests(() => { throw throwingRunnerCause(clean); }); + + const error = thrownAclError(() => hardenSecretPath(filePath, { required: true })); + + expect(error.code).toBe("EICACLS"); + expect(error.aclFailure).toEqual({ ...clean }); + }); +}); From db7bdc5e3d233aed8425420e77fbd66fe20ec947 Mon Sep 17 00:00:00 2001 From: Epinephrine Date: Fri, 30 Oct 2026 11:41:25 +0000 Subject: [PATCH 11/42] fix(cli): explain tracked runtime stop refusals (#7973) * fix(cli): explain tracked runtime stop refusals * fix(cli): state only what the identity probe proves A cached identity mismatch shows only that the recorded process does not match the expected start command, not that the PID was reused by another process. --------- Co-authored-by: JUN <243035832+lidge-jun@users.noreply.github.com> --- src/cli/index.ts | 9 +- src/cli/stop-approval.ts | 16 +- src/config/process-state.ts | 28 ++- src/config/tracked-runtime-inspection.ts | 188 ++++++++++++++++++++ structure/config.md | 2 +- structure/runtime.md | 4 +- tests/cli/cli-stop-json.test.ts | 14 ++ tests/service/process-state.test.ts | 207 +++++++++++++++++++++++ 8 files changed, 452 insertions(+), 16 deletions(-) create mode 100644 src/config/tracked-runtime-inspection.ts diff --git a/src/cli/index.ts b/src/cli/index.ts index d1439754154..eff70dd94bb 100755 --- a/src/cli/index.ts +++ b/src/cli/index.ts @@ -69,6 +69,7 @@ import { writePid, writeRuntimePort, } from "../config/process-state"; +import { describeTrackedRuntimeTarget } from "../config/tracked-runtime-inspection"; import { claimPendingTeardown, isPendingTeardownAbandoned, @@ -1020,13 +1021,7 @@ async function handleStop(approval?: StopApproval) { try { return JSON.parse(lines[0]!) as ResolveJson; } catch { return null; } }, - () => { - const pid = readPid(); - const runtime = pid === null ? null : readRuntimePort(pid); - return pid && runtime?.port ? { - pid, port: runtime.port, hostname: runtime.hostname ?? "", - } : null; - }, + describeTrackedRuntimeTarget, () => inspectGuardedManagerTarget(approval.pid, approval.port), snapshot => handleStopUnlocked(snapshot), ); diff --git a/src/cli/stop-approval.ts b/src/cli/stop-approval.ts index 4b83fef6169..5a68879e8d5 100644 --- a/src/cli/stop-approval.ts +++ b/src/cli/stop-approval.ts @@ -20,6 +20,17 @@ export interface GuardedStopSnapshot { manager: Exclude; } +/** + * What the tracked-record check can report: the verified target, null when the + * caller cannot name a cause (the pre-diagnostics shape), or the fixed cause of + * the single check that failed. The cause is a fixed sentence fragment from the + * inspection probe -- it carries no path, command line, stderr, or token. + */ +export type TrackedRuntimeCheck = + | { pid: number; port: number; hostname: string } + | { untracked: string } + | null; + function sameGuardedManager( expected: GuardedStopSnapshot["manager"], current: GuardedManagerTarget, @@ -117,7 +128,7 @@ export function managerStillActive( export async function runApprovedStop( expected: StopApproval, read: () => Promise, - tracked: () => { pid: number; port: number; hostname: string } | null, + tracked: () => TrackedRuntimeCheck, managerTarget: () => GuardedManagerTarget, stop: (snapshot: GuardedStopSnapshot) => Promise, ): Promise { @@ -130,6 +141,9 @@ export async function runApprovedStop( const mismatch = stopApprovalMismatch(expected, now); if (mismatch !== null) return approvalChanged(mismatch); if (target === null) return approvalChanged("the tracked runtime record could not be verified"); + if ("untracked" in target) { + return approvalChanged(`the tracked runtime record could not be verified (${target.untracked})`); + } if (target.pid !== expected.pid || target.port !== expected.port || target.hostname !== expected.hostname) { return approvalChanged("the tracked runtime record changed"); diff --git a/src/config/process-state.ts b/src/config/process-state.ts index f61f714b8d9..4ac5191e57c 100644 --- a/src/config/process-state.ts +++ b/src/config/process-state.ts @@ -45,7 +45,7 @@ export type RuntimePortState = { siblingOfPort?: number; }; -function isValidRuntimePortState(value: unknown): value is RuntimePortState { +export function isValidRuntimePortState(value: unknown): value is RuntimePortState { if (!value || typeof value !== "object") return false; const state = value as Record; const hostnameOk = state.hostname === undefined || typeof state.hostname === "string"; @@ -201,14 +201,32 @@ export function ocxStartProcessCacheSizeForTests(): number { return ocxStartProcessCache.size; } -function isLikelyOcxStartProcess(pid: number): boolean { +/** + * One observation of whether a candidate PID is an `ocx start` process. + * + * `readPid()` and `verifyPidIdentity()` ask this on every short liveness poll, and the + * read-only tracked-record inspection in `tracked-runtime-inspection.ts` asks it once per + * guarded stop, so all of them share this single cache-backed answer rather than probe + * twice: a hit performs no native command at all, and a miss performs exactly one + * {@link readProcessCommandLine} probe. Only a definite answer is cached; an unreadable + * command line is reported as `"unverifiable"`, left uncached, and re-probed on the next + * ask. `"unverifiable"` is not evidence of a permission boundary, a foreign process, or a + * different account or elevation — only that the probe could not answer this time. + */ +export type OcxStartIdentityObservation = "match" | "mismatch" | "unverifiable"; + +export function inspectOcxStartProcessIdentity(pid: number): OcxStartIdentityObservation { const cached = ocxStartProcessCache.get(pid); - if (cached !== undefined) return cached; + if (cached !== undefined) return cached ? "match" : "mismatch"; const commandLine = readProcessCommandLine(pid); - if (commandLine === undefined) return false; + if (commandLine === undefined) return "unverifiable"; const ok = isOcxStartCommandLine(commandLine); ocxStartProcessCache.set(pid, ok); - return ok; + return ok ? "match" : "mismatch"; +} + +function isLikelyOcxStartProcess(pid: number): boolean { + return inspectOcxStartProcessIdentity(pid) === "match"; } export function readPid(): number | null { diff --git a/src/config/tracked-runtime-inspection.ts b/src/config/tracked-runtime-inspection.ts new file mode 100644 index 00000000000..6cc241e5f19 --- /dev/null +++ b/src/config/tracked-runtime-inspection.ts @@ -0,0 +1,188 @@ +/** + * Read-only classification of the tracked runtime record pair a guarded stop + * re-verifies before acting. + * + * "ocx stop --expect-*" asks "is the recorded runtime still the approved one", + * which collapses several observations into one boolean: a missing or invalid + * "ocx.pid", a dead, foreign or unverifiable recorded process, and a missing, + * invalid or PID-mismatched "runtime-port.json" all read as "not verified". + * The approval-changed refusal used to report only that umbrella fact, so a + * desktop log could not tell a stale record from a permission failure. + * + * This probe walks the same checks in the same order as "readPid()" followed + * by "readRuntimePort(pid)", but reports which one failed instead. Each state + * record is still read exactly once per call -- a guarded stop asks once, and + * caching the bytes would only carry a stale answer into a decision about a + * record that may have been rewritten. The process-identity question, though, + * is answered by the same cache-backed observation "readPid()" itself makes + * ("inspectOcxStartProcessIdentity"): a cached verdict serves this probe with + * no native command at all, and a miss performs the same single command-line + * probe readPid would, so this probe can never disagree with the admission + * readPid just granted or denied for the same PID. It performs no mutation, no + * signal beyond the same "process.kill(pid, 0)" liveness read, and no stop + * decision -- the refusal itself is unchanged; only its detail gains a cause. + * + * Two distinctions are deliberate: + * + * - "permission-denied" is reported only for a real EACCES/EPERM observed on a + * record read or the liveness signal. An "unverifiable" identity observation + * -- an unreadable process command line -- is not proof of a permission + * boundary, and no account or elevation cause is inferred from it. + * - No path, command line, stderr text, or token leaves this module: the refusal + * causes below are fixed strings and the detail built from them is a fixed + * sentence. + */ +import { readFileSync } from "node:fs"; +import { + getPidPath, + getRuntimePortPath, + inspectOcxStartProcessIdentity, + isValidRuntimePortState, + parsePidFile, +} from "./process-state"; +import type { OcxStartIdentityObservation } from "./process-state"; + +/** The single check that failed, as a stable machine-readable token. */ +export type TrackedRecordRefusal = + | "pid-record-absent" + | "pid-record-permission-denied" + | "pid-record-unreadable" + | "pid-record-invalid" + | "process-not-running" + | "process-identity-mismatch" + | "process-unverifiable" + | "runtime-record-absent" + | "runtime-record-permission-denied" + | "runtime-record-unreadable" + | "runtime-record-invalid" + | "runtime-record-pid-mismatch" + | "unverified"; + +export type TrackedRecordInspection = + | { status: "verified"; pid: number; port: number; hostname: string } + | { status: "refused"; reason: TrackedRecordRefusal }; + +/** + * Injectable read-only seams. Tests must supply fakes for "signal" and either + * "identity" or the process-state exec seams ("setProcessCommandLineExecForTests" + * and friends) so no real "process.kill" or native command probe runs; the + * defaults are exactly what the production caller uses, including the shared + * identity cache. + */ +export interface TrackedRecordProbeDeps { + /** Read a state record like "readFileSync(path, 'utf-8')"; throw with errno codes. */ + readFile?: (path: string) => string; + /** The "process.kill(pid, 0)" liveness read; throws like it (ESRCH/EPERM/...). */ + signal?: (pid: number) => void; + /** The shared start-identity observation; the default consults the real cache. */ + identity?: (pid: number) => OcxStartIdentityObservation; +} + +function readRecordFile( + readFile: (path: string) => string, + path: string, + absent: TrackedRecordRefusal, + denied: TrackedRecordRefusal, + unreadable: TrackedRecordRefusal, +): { ok: true; raw: string } | { ok: false; reason: TrackedRecordRefusal } { + try { + return { ok: true, raw: readFile(path) }; + } catch (error) { + const code = (error as NodeJS.ErrnoException).code; + if (code === "EACCES" || code === "EPERM") return { ok: false, reason: denied }; + if (code === "ENOENT") return { ok: false, reason: absent }; + return { ok: false, reason: unreadable }; + } +} + +/** + * Classify why the tracked runtime record cannot back a guarded stop, or return + * the verified target when every check still holds. One read per record, in the + * same order the stop path already performs them. + */ +export function inspectTrackedRuntimeRecord( + deps: TrackedRecordProbeDeps = {}, +): TrackedRecordInspection { + const readFile = deps.readFile ?? ((path: string) => readFileSync(path, "utf-8")); + const signal = deps.signal ?? ((pid: number) => { process.kill(pid, 0); }); + const identity = deps.identity ?? inspectOcxStartProcessIdentity; + try { + const pidFile = readRecordFile( + readFile, getPidPath(), "pid-record-absent", "pid-record-permission-denied", "pid-record-unreadable", + ); + if (!pidFile.ok) return { status: "refused", reason: pidFile.reason }; + const pid = parsePidFile(pidFile.raw); + if (pid === null) return { status: "refused", reason: "pid-record-invalid" }; + try { + signal(pid); + } catch (error) { + const code = (error as NodeJS.ErrnoException).code; + if (code === "ESRCH") return { status: "refused", reason: "process-not-running" }; + // EPERM still names a live process; identity is decided by the shared + // observation below, exactly as readPid decides it. Any other signal + // error can only be reported as unverifiable. + if (code !== "EPERM") return { status: "refused", reason: "process-unverifiable" }; + } + const observation = identity(pid); + if (observation === "unverifiable") { + return { status: "refused", reason: "process-unverifiable" }; + } + if (observation === "mismatch") { + return { status: "refused", reason: "process-identity-mismatch" }; + } + const runtimeFile = readRecordFile( + readFile, getRuntimePortPath(), + "runtime-record-absent", "runtime-record-permission-denied", "runtime-record-unreadable", + ); + if (!runtimeFile.ok) return { status: "refused", reason: runtimeFile.reason }; + let parsed: unknown; + try { + parsed = JSON.parse(runtimeFile.raw); + } catch { + return { status: "refused", reason: "runtime-record-invalid" }; + } + if (!isValidRuntimePortState(parsed)) { + return { status: "refused", reason: "runtime-record-invalid" }; + } + if (parsed.pid !== pid) { + return { status: "refused", reason: "runtime-record-pid-mismatch" }; + } + return { status: "verified", pid, port: parsed.port, hostname: parsed.hostname ?? "" }; + } catch { + return { status: "refused", reason: "unverified" }; + } +} + +/** + * Fixed refusal causes for the approval-changed detail. Every entry is a + * complete sentence fragment and contains no path, command line, stderr, or + * token, so the resulting detail is safe to surface in a caller's log or UI. + */ +const TRACKED_REFUSAL_CAUSES: Record = { + "pid-record-absent": "the recorded PID file is absent", + "pid-record-permission-denied": "reading the recorded PID file was denied", + "pid-record-unreadable": "the recorded PID file could not be read", + "pid-record-invalid": "the recorded PID file is not a valid record", + "process-not-running": "the recorded process is no longer running", + "process-identity-mismatch": "the recorded process does not match the expected start command", + "process-unverifiable": "the recorded process identity could not be verified", + "runtime-record-absent": "the tracked runtime record is absent", + "runtime-record-permission-denied": "reading the tracked runtime record was denied", + "runtime-record-unreadable": "the tracked runtime record could not be read", + "runtime-record-invalid": "the tracked runtime record is not a valid record", + "runtime-record-pid-mismatch": "the tracked runtime record names a different PID", + "unverified": "the tracked runtime record could not be re-verified", +}; + +/** + * The guarded stop's tracked() callback shape: the verified target, or the + * fixed cause of the single check that failed. Read-only; decides nothing. + */ +export function describeTrackedRuntimeTarget( + deps: TrackedRecordProbeDeps = {}, +): { pid: number; port: number; hostname: string } | { untracked: string } { + const inspection = inspectTrackedRuntimeRecord(deps); + return inspection.status === "verified" + ? { pid: inspection.pid, port: inspection.port, hostname: inspection.hostname } + : { untracked: TRACKED_REFUSAL_CAUSES[inspection.reason] }; +} diff --git a/structure/config.md b/structure/config.md index 4f330eebdd9..01d9447e520 100644 --- a/structure/config.md +++ b/structure/config.md @@ -68,7 +68,7 @@ silently move the active installation. Config reads in this leaf and `src/config It owns their byte-compatible writes, parsing, expected-PID filters, cheap liveness, full OCX command identity, and snapshot-guarded removal. `RuntimePortState.attestationSecret` and `siblingOfPort` (the live owner's port, written only by a sibling instance) remain optional, owner-only state and are validated before a record is returned. `src/config.ts` re-exports the same -symbols for compatibility, but new lifecycle-only callers import the process-state leaf directly. +symbols for compatibility, but new lifecycle-only callers import the process-state leaf directly. `src/config/tracked-runtime-inspection.ts` is the guarded stop's read-only companion probe: it re-runs the same record reads in the same order, answers the start-identity question through the same cache `readPid()` consults, and reports which single check failed as a fixed refusal cause, so an absent, denied, invalid or PID-mismatched record is told apart from a dead, foreign or unverifiable process, and it decides nothing and exposes no path, command line, stderr, or token. Replacing config and process-state writes use `src/config/atomic-write.ts`. Normal Windows writes opt into the destination's single ACL timeout recovery attempt; a consumed attempt re-arms after `TIMEOUT_MEMO_REARM_MS` in `src/lib/windows-secret-acl.ts`. Default writers track temporary paths per resolved destination across synchronous, streamed and asynchronous writes. A new write refuses immediately with sanitized `ETIMEDOUT` only while one of that destination's tracked paths has an icacls child retained past its caller's timeout belt and not yet reaped, including across re-arm windows. Healthy overlapping writes proceed, including during ACL hardening or a pause before rename. Refusal consumes no recovery attempt and leaves removal ownership intact; completing one write does not drop another write's retained-child tracking. Both cleanup paths leave that recovery unconsumed. Every admitted attempt still requires a successful harden before secret bytes or publication; non-timeout failures retain their existing fail-closed behavior. `tests/windows/windows-atomic-write-acl-rearm.test.ts` covers all five harden calls, the re-arm boundary, healthy overlap, retained child admission and cleanup. The leaf preserves the shared process-wide temp sequence, optional synchronous caller destination validation before every rename attempt, symlink target resolution, no-follow directory-entry replacement for diff --git a/structure/runtime.md b/structure/runtime.md index e40e2fc0ed0..ac5b871d6c6 100644 --- a/structure/runtime.md +++ b/structure/runtime.md @@ -59,7 +59,7 @@ The [CLI management contract](cli-management.md#cli-readiness-diagnostics) owns `ocx resolve` (`src/cli/resolve.ts`) is the machine surface a desktop shell asks instead of resolving the config home, the port, and liveness itself: the home comes from `src/config/paths.ts`, the effective port is the live listener's when the identity-checked `findLiveProxy` answers and the configured `config.port ?? 10100` otherwise, and the liveness verdict is that same module's output (pid, runtime-versus-config provenance, version, role). Config reads go through `readConfigDiagnostics`, not `loadConfig`: a missing file is defaults, but an invalid file exits 1 instead of being repaired to defaults, because a defaulted port is a guess the caller must refuse. Liveness is three-valued: when `findLiveProxy` returns null, resolve re-asks the endpoints with `endpointsToProve` and `everyEndpointProvenDown`, using the in-process `probeEndpointLiveness` from `src/server/proxy-liveness.ts`. Only unanimous definitive "dead" becomes `absent-proven`; a timeout or reset stays unknown, exits 1 and never authorises a start. The transient exclusive-bind fallback in `src/update/proxy-liveness-probe.mjs` belongs only to the npm/Bun updater. Resolve stays in-process because a compiled standalone binary cannot run a JavaScript probe with `execPath -e`. Discovery borrows `START_OWNERSHIP_LIVENESS`, the start path's ownership budget — the verdict feeds the shell's launch decision, so the cost of a false "nobody listening" is the duplicate proxy (#5004). The verb is in `skipsCodexShimAutoRestore`, so a read-only lookup never triggers a shim repair. Arguments are pre-parsed in `src/cli/root.ts` and exit 64 before any preflight side effect, the same ordering `ocx ready` obeys. `ocx-resolve/1` also reports `ownership` (`none`, `owned`, or `unknown`) and `takeover` (`supported` with protocol version, minimum CLI version and compatibility token, or `blocked` with a reason). A known liveness verdict still exits 0 when ownership is unknown, but takeover is blocked. An unreadable second service-state read blocks takeover instead of becoming an absent registration. `src/service/managing-cli.ts` observes the registered and selected PATH CLIs, including Windows PATHEXT order and file validation, before compatibility is offered. A `--version` probe is retried once on Windows — one transient failure must not read as an unobservable CLI — and a Windows `.CMD`/`.BAT` shim is invoked through a single verbatim-quoted `cmd /c` line, so a spaced executable path and spaced arguments both survive cmd's re-parse. `src/cli/route-policy.ts` distinguishes a missing profile (exit 4) from malformed `show` arguments (exit 2); an upstream failure propagates through the management error mapping. -`ocx stop --json` is a reporting layer over the unchanged stop path. `src/cli/index.ts` threads a `StopRunRecord` through the existing receipt, drain, respawn-verification and restore flow, and `src/cli/stop-report.ts` maps the recorded facts plus the signals that already pick the exit code into one versioned document (`schema: "ocx-stop/1"`). With `--json` the human lines print on stderr and stdout carries only that document; exit codes 0/1/79/80/81 cross the process boundary unchanged. The desktop's opt-in guarded stop adds exact approved PID, endpoint, home, CLI version and compatibility-token expectations; plain `ocx stop` retains its ordinary behavior. Under the ownership mutation lease, `src/cli/stop-approval.ts` re-resolves them and `src/service/guarded-manager-target.ts` binds any installed manager's process tree to the approved PID before a stop action. A pre-action mismatch returns `approval-changed` without stopping, and the summary carries `detail` naming the single guard fact that failed — live PID/port/hostname/CLI drift, an unverifiable ownership claim, a blocked takeover, or the compatibility fingerprint — so a transient re-probe can be told from genuine drift in the caller's log. Immediately before any manager stop or direct PID signal, the CLI rechecks the same manager identity or proven absence; a change also returns `approval-changed`. A non-OpenCodex process replacing the OS job in the remaining instant before the manager command is a residual same-user risk outside the ownership lease. The guarded path stops the bound manager, or the approved PID directly when managers are proven absent, then uses a five-second shared deadline to verify PID exit, port availability and definitive endpoint absence. Only after settlement does a read-only manager probe require launchd `not-loaded` in both domains or systemd `inactive` with `MainPID=0`; the state is checked again before a success summary. Timeout, active or unreadable manager state returns terminal `manager-still-active`, even if the endpoint briefly refuses. Windows managers bind through `src/service/windows-guarded-manager.ts`: a Task Scheduler backend must prove a recognized OpenCodex registration against the bounded current-user SID/name lookup, a running task, and a wrapper ancestor carrying this home's canonical paths — presence and running state are read root-folder-scoped because every `/tn` operation resolves a bare name there and nowhere else, so a same-named task inside a task folder is a different registration neither the probe nor the stop can reach; the registered definition itself is read only on the running path, and a state or registration read that cannot answer re-probes presence so a task deleted between probes — which can respawn nothing — reads absent while a merely unreadable one stays unknown; an unreadable current identity leaves the manager unknown; a WinSW backend must prove this installation's binary and SCM service PID ancestry; either marks `childNeedsSeparateStop`, so the approved PID receives its own graceful signal only after a successful manager stop because `schtasks /end` does not reliably cascade to the wrapper's child (#764); a failed or unknown manager stop leaves the child untouched. Post-stop Windows inactivity means the task not running, no surviving wrapper process, and no started WinSW — never mere registration presence; a running-state read that cannot answer re-probes presence the same way, so a task deleted between probes reports inactive while a persisting one stays unknown. Every Windows manager command a verdict or stop depends on — schtasks, sc.exe, WinSW verbs — runs under a bounded `execFileSync` timeout so a wedged Task Scheduler or Service Control Manager cannot hang the guarded stop; a killed command classifies through the same fail-closed paths a nonzero exit would (the interactive `install /p` password prompt is the one deliberate exception). A `wscript.exe`, `cscript.exe` or `cmd.exe` whose command line CIM cannot read makes the state unprovable only when it is tied to the proxy: pre-stop, as an ancestor of the approved PID; post-stop, as the former bound manager process or a descendant of it. Unrelated unreadable shells, which a non-admin query returns for other users' and elevated processes, are not evidence either way. An unprovable Windows manager still blocks guarded takeover. The token checks snapshot consistency, not whether a person approved the desktop prompt. `src/cli/stop-restore.ts` owns the shared Codex/Grok restoration and its history-only, history-deferred, and other failure categories. +`ocx stop --json` is a reporting layer over the unchanged stop path. `src/cli/index.ts` threads a `StopRunRecord` through the existing receipt, drain, respawn-verification and restore flow, and `src/cli/stop-report.ts` maps the recorded facts plus the signals that already pick the exit code into one versioned document (`schema: "ocx-stop/1"`). With `--json` the human lines print on stderr and stdout carries only that document; exit codes 0/1/79/80/81 cross the process boundary unchanged. The desktop's opt-in guarded stop adds exact approved PID, endpoint, home, CLI version and compatibility-token expectations; plain `ocx stop` retains its ordinary behavior. Under the ownership mutation lease, `src/cli/stop-approval.ts` re-resolves them and `src/service/guarded-manager-target.ts` binds any installed manager's process tree to the approved PID before a stop action. The same guard re-reads `ocx.pid` and `runtime-port.json` through `src/config/tracked-runtime-inspection.ts`, a single-pass read-only probe that walks the same checks `readPid()` and `readRuntimePort(pid)` perform and reports which one failed, answering the recorded process's start-identity question through the same cache `readPid()` fills so it cannot disagree with the admission it diagnoses. A pre-action mismatch returns `approval-changed` without stopping, and the summary carries `detail` naming the single guard fact that failed — live PID/port/hostname/CLI drift, an unverifiable ownership claim, a blocked takeover, the compatibility fingerprint, or the tracked-record cause — an absent, denied, unreadable or invalid record, a dead, foreign or unverifiable recorded process, or a record naming a different PID — so a transient re-probe can be told from genuine drift in the caller's log. The tracked-record causes are fixed strings: denied is reported only on an observed EACCES/EPERM, an unreadable command line stays unverifiable rather than attributed to an account or elevation, and no probed path, command line, stderr or token reaches the detail. Immediately before any manager stop or direct PID signal, the CLI rechecks the same manager identity or proven absence; a change also returns `approval-changed`. A non-OpenCodex process replacing the OS job in the remaining instant before the manager command is a residual same-user risk outside the ownership lease. The guarded path stops the bound manager, or the approved PID directly when managers are proven absent, then uses a five-second shared deadline to verify PID exit, port availability and definitive endpoint absence. Only after settlement does a read-only manager probe require launchd `not-loaded` in both domains or systemd `inactive` with `MainPID=0`; the state is checked again before a success summary. Timeout, active or unreadable manager state returns terminal `manager-still-active`, even if the endpoint briefly refuses. Windows managers bind through `src/service/windows-guarded-manager.ts`: a Task Scheduler backend must prove a recognized OpenCodex registration against the bounded current-user SID/name lookup, a running task, and a wrapper ancestor carrying this home's canonical paths — presence and running state are read root-folder-scoped because every `/tn` operation resolves a bare name there and nowhere else, so a same-named task inside a task folder is a different registration neither the probe nor the stop can reach; the registered definition itself is read only on the running path, and a state or registration read that cannot answer re-probes presence so a task deleted between probes — which can respawn nothing — reads absent while a merely unreadable one stays unknown; an unreadable current identity leaves the manager unknown; a WinSW backend must prove this installation's binary and SCM service PID ancestry; either marks `childNeedsSeparateStop`, so the approved PID receives its own graceful signal only after a successful manager stop because `schtasks /end` does not reliably cascade to the wrapper's child (#764); a failed or unknown manager stop leaves the child untouched. Post-stop Windows inactivity means the task not running, no surviving wrapper process, and no started WinSW — never mere registration presence; a running-state read that cannot answer re-probes presence the same way, so a task deleted between probes reports inactive while a persisting one stays unknown. Every Windows manager command a verdict or stop depends on — schtasks, sc.exe, WinSW verbs — runs under a bounded `execFileSync` timeout so a wedged Task Scheduler or Service Control Manager cannot hang the guarded stop; a killed command classifies through the same fail-closed paths a nonzero exit would (the interactive `install /p` password prompt is the one deliberate exception). A `wscript.exe`, `cscript.exe` or `cmd.exe` whose command line CIM cannot read makes the state unprovable only when it is tied to the proxy: pre-stop, as an ancestor of the approved PID; post-stop, as the former bound manager process or a descendant of it. Unrelated unreadable shells, which a non-admin query returns for other users' and elevated processes, are not evidence either way. An unprovable Windows manager still blocks guarded takeover. The token checks snapshot consistency, not whether a person approved the desktop prompt. `src/cli/stop-restore.ts` owns the shared Codex/Grok restoration and its history-only, history-deferred, and other failure categories. `ocx service status --json` (`src/service/status-json.ts`) is the passive machine form of the service report, read by peers such as GOAT's device inventory. `src/service/cli.ts` accepts `--json` for `status` only (`planServiceCommand` refuses it on every other verb before any installation probe) and answers it before `platformOps()` resolves, so no install, start, stop, repair or migration path is reachable; `skipsCodexShimAutoRestore` exempts `service status --json` so the CLI preflight repairs no Codex shim first, the collection runs inside `runPassiveRead` so the serving probe's config load neither hardens modes nor backs up a malformed file, and `--json` on any other verb is refused before the Desktop check and the Windows mutation lock; an unsupported platform or Docker gets a document instead of the text path's exit. The document reuses exactly the reads the text report makes — `diagnoseService()` (launchd `print`, systemd observation verbs, Task Scheduler `/query`, SCM/WinSW status) and, only when a registration exists, the 1.5-second loopback `confirmServiceServing` probe — plus the ownership-lease and Desktop login lines. `running` is the manager's verdict (the value `startup.serviceRunning` carries in `ocx status --json`); `serving` reports separately whether a proxy answers on the installed port. `backend` is `launchd`, `systemd`, `schtasks`, `winsw`, or `none` when nothing is registered. The systemd branch of `diagnoseService` derives through the pure `deriveSystemdServiceDiagnostic`, matching the launchd and Windows derivations. `tests/service/service-status-json.test.ts` snapshots each platform branch and records every subprocess entry point on darwin, linux and win32 against a read-only allowlist. `ocx status --json` carries an additive `install` verdict (`src/cli/status-install.ts`) mapped from the updater's own `detectInstallOwnership()`: `method` is `bun`, `npm`, `pnpm`, `mise`, `source`, `standalone` or `unknown` (detector failure), and `evidence.basis` names the package layout, the mise backend record (tool alias and backend only) or the standalone binary without printing a path (`tests/cli/status-install-method.test.ts`). @@ -117,7 +117,7 @@ does not perform OAuth, and runtime credential resolution rereads the owned sour | `src/config.ts`, `src/config/persisted-mutation.ts` | Persisted `~/.opencodex/config.json` surface: the facade keeps load/save/initialize entry points and re-exports the schema-valid mutation callback, types, and one-shot test seam; the leaf owns its bounded rebase under the shared lock and a read-only authority reader for admitted writers that starts no optional ACL flight. Schema lives in `src/config/schema/` (`config-schema.ts`, `leaf-validators.ts`), defaults in `src/config/proxy-env.ts`, and replace-path persistence in `src/config/persist-unlocked.ts`. | | `src/config/paths.ts` | Resolves `OPENCODEX_HOME`, `config.json`, and owner-only directory hardening. | | `src/config/atomic-write.ts` | Shared synchronous/asynchronous temp-harden-rename writer and residual-temp failure contract. No-follow exclusive publication reuses its private descriptor writer for raw recovery bytes, checks bigint identity and publishes by a no-replace hard link; cleanup never scrubs an already published inode. Synchronous caller validation runs immediately before each rename attempt, including Windows retries. The temp is ACL-hardened before it holds a byte and again before the rename, both `required: true`; the second call is a memo hit rather than a second icacls sequence because the writer re-asserts descriptor/path identity after the content write and re-attributes the harden through `reattributeHardenedSecretPath`. Windows takes no `chmod` on that path — it sets the read-only attribute, not the DACL, and its ChangeTime bump is what used to retire the memo. The existing-secret compliance probe in `src/lib/windows-secret-acl.ts` uses the same SID-based private-entry policy as recovery-copy reuse when the opt-in single-owner ACE fast check cannot establish compliance; recovery-copy reuse always calls the SID policy directly. Retained SYSTEM and Administrators ACEs do not trigger mutation or duplicate recovery publication. The asynchronous probe in `src/lib/windows-owner-acl.ts` yields during the bounded descriptor subprocess and shares the synchronous parser, without admitting a hardening success memo. | -| `src/config/process-state.ts` | Owns `ocx.pid`, `runtime-port.json`, cheap liveness, full command-line identity verification, and snapshot-guarded cleanup. A transient Windows WMIC/PowerShell command-line probe failure gets one bounded retry before a PID reads as unverified. | +| `src/config/process-state.ts` | Owns `ocx.pid`, `runtime-port.json`, cheap liveness, full command-line identity verification, and snapshot-guarded cleanup. A transient Windows WMIC/PowerShell command-line probe failure gets one bounded retry before a PID reads as unverified. `src/config/tracked-runtime-inspection.ts` re-runs the same record reads once for the guarded stop, shares `readPid()`'s cached start-identity observation rather than re-probing it, and names the failed check as a fixed cause. | | `src/config/admitted-identity.ts` | Which configuration a derived artifact was built from. Detaches the resident configuration as plain data so one pass cannot gather under one state and project under another, and records the complete structure beside the configuration file's bytes. Canonical identity tags arrays and objects separately, so key sorting cannot equate an object with an array of entry pairs. Refuses an accessor, a cycle, a value JSON could not produce, an unreadable file and a file the loader would have had to salvage; a callable `providers[name].fetch` is the one non-data field, held and compared by reference, while a written one is ordinary data on both sides, as the outbound transport also reads it. It does NOT require the resident configuration to equal the file: the proxy routes by what it holds, and live reconciliation retains live changes and the active listener binding on purpose. Evidence stays in a module WeakMap, never on the config and never in a response. | | `src/server/ports.ts` / `src/server/bind-host.ts` | Own bind availability, ephemeral-port selection and startup hostname normalization shared with liveness: blank and case-insensitive `localhost`/`localhost.` bind to IPv4 loopback; other spellings remain unchanged. Temporary probes dispose accepted peers and wait for listener close before reporting success. | | `src/cli/status.ts` / `src/cli/status-probes.ts` | Status snapshot assembly and the shared read-only health/stale-process probes used by status and doctor. Probe evidence keeps recorded-port choice, before/after snapshots and per-call timer cleanup together. | diff --git a/tests/cli/cli-stop-json.test.ts b/tests/cli/cli-stop-json.test.ts index 5a7168e55a2..5394f8256ac 100644 --- a/tests/cli/cli-stop-json.test.ts +++ b/tests/cli/cli-stop-json.test.ts @@ -464,6 +464,20 @@ describe("approval-bound stop", () => { () => null, BOUND_STOP_MANAGER, stop); expect(untracked.summary.detail).toBe("the tracked runtime record could not be verified"); + // The same umbrella refusal now names which tracked-record check failed, so a + // stale or missing record reads differently from an unreadable or foreign process. + const absentRecord = await runApprovedStop(STOP_APPROVAL, async () => STOP_RESOLVE, + () => ({ untracked: "the recorded PID file is absent" }), BOUND_STOP_MANAGER, stop); + expect(absentRecord.summary).toMatchObject({ outcome: "approval-changed", exitCode: 1, + detail: "the tracked runtime record could not be verified (the recorded PID file is absent)" }); + expect(absentRecord.summary.message).toContain("the recorded PID file is absent"); + + const foreignPid = await runApprovedStop(STOP_APPROVAL, async () => STOP_RESOLVE, + () => ({ untracked: "the recorded process does not match the expected start command" }), + BOUND_STOP_MANAGER, stop); + expect(foreignPid.summary.detail) + .toBe("the tracked runtime record could not be verified (the recorded process does not match the expected start command)"); + const managerGone = await runApprovedStop(STOP_APPROVAL, async () => STOP_RESOLVE, TRACKED_STOP_TARGET, () => ({ kind: "unknown" as const, reason: "CIM read failed" }), stop); expect(managerGone.summary.detail).toBe("the service manager could not be re-verified (CIM read failed)"); diff --git a/tests/service/process-state.test.ts b/tests/service/process-state.test.ts index 3ff6ac9ea01..73cfd51657a 100644 --- a/tests/service/process-state.test.ts +++ b/tests/service/process-state.test.ts @@ -18,9 +18,15 @@ import { setOcxStartProcessCacheForTests, setProcessCommandLineExecForTests, setProcessCommandLinePlatformForTests, + verifyPidIdentity, writePid, writeRuntimePort, } from "../../src/config/process-state"; +import { + describeTrackedRuntimeTarget, + inspectTrackedRuntimeRecord, +} from "../../src/config/tracked-runtime-inspection"; +import type { TrackedRecordProbeDeps } from "../../src/config/tracked-runtime-inspection"; import { markSiblingStart, resetSiblingStartForTests, siblingRuntimeField } from "../../src/codex/sibling-start"; import { setTrustedWindowsSystemDirectoryResolverForTests } from "../../src/lib/windows-elevation"; import { removeTreeWithRetry } from "../helpers/remove-tree"; @@ -308,3 +314,204 @@ describe("proxy process-state ownership", () => { } }); }); + +describe("tracked runtime record inspection", () => { + // Injected deps keep every probe input fake: no process.kill, no WMIC/PowerShell/ps, + // and no real stop ever runs. Record reads default to the real temp-home files. The + // cache-sharing cases below drive the same exec seams production uses -- with faked + // native answers and a real signal-0 read on this process only. + const errno = (code: string): NodeJS.ErrnoException => + Object.assign(new Error("fake " + code), { code }); + const liveOcx: TrackedRecordProbeDeps = { signal: () => {}, identity: () => "match" }; + const trackedPid = () => writeFileSync(getPidPath(), "4242", "utf-8"); + const trustedProbePaths = () => { + const trustedSystem32 = join(testDir, "trusted", "System32"); + const trustedPowerShell = join(trustedSystem32, "WindowsPowerShell", "v1.0", "powershell.exe"); + mkdirSync(dirname(trustedPowerShell), { recursive: true }); + writeFileSync(trustedPowerShell, "", { mode: 0o755 }); + setProcessCommandLinePlatformForTests("win32"); + setTrustedWindowsSystemDirectoryResolverForTests(() => trustedSystem32); + return { trustedWmic: join(trustedSystem32, "wbem", "WMIC.exe"), trustedPowerShell }; + }; + // The trusted-System32 seam also reroutes icacls resolution, so these records are + // written directly like the cases above -- the parity being tested is the shared + // identity cache, not the atomic writer. + const trackedSelf = (hostname: string) => { + writeFileSync(getPidPath(), String(process.pid), "utf-8"); + writeFileSync(getRuntimePortPath(), JSON.stringify({ pid: process.pid, port: 58195, hostname }), "utf-8"); + }; + + test("a verified record pair returns the tracked target", () => { + trackedPid(); + writeFileSync(getRuntimePortPath(), JSON.stringify({ pid: 4242, port: 58195, hostname: "0.0.0.0" }), "utf-8"); + expect(inspectTrackedRuntimeRecord(liveOcx)).toEqual( + { status: "verified", pid: 4242, port: 58195, hostname: "0.0.0.0" }); + expect(describeTrackedRuntimeTarget(liveOcx)).toEqual({ pid: 4242, port: 58195, hostname: "0.0.0.0" }); + }); + + test("an absent or invalid pid record is named before any process probe", () => { + const probes: string[] = []; + const deps: TrackedRecordProbeDeps = { + signal: () => { probes.push("signal"); }, + identity: () => { probes.push("identity"); return "match"; }, + }; + expect(inspectTrackedRuntimeRecord(deps)).toEqual( + { status: "refused", reason: "pid-record-absent" }); + expect(probes).toEqual([]); + + writeFileSync(getPidPath(), "not-a-pid", "utf-8"); + expect(inspectTrackedRuntimeRecord(deps)).toEqual( + { status: "refused", reason: "pid-record-invalid" }); + expect(probes).toEqual([]); + }); + + test("pid-record read errors separate real permission denials from other failures", () => { + const denied = (code: string) => inspectTrackedRuntimeRecord({ + readFile: () => { throw errno(code); }, + signal: () => {}, + identity: () => "match", + }); + expect(denied("EACCES")).toEqual({ status: "refused", reason: "pid-record-permission-denied" }); + expect(denied("EPERM")).toEqual({ status: "refused", reason: "pid-record-permission-denied" }); + expect(denied("EIO")).toEqual({ status: "refused", reason: "pid-record-unreadable" }); + }); + + test("a dead recorded pid reports not-running; other signal errors stay unverifiable", () => { + trackedPid(); + const withSignal = (signal: (pid: number) => void) => + inspectTrackedRuntimeRecord({ signal, identity: () => "match" }); + expect(withSignal(() => { throw errno("ESRCH"); })).toEqual( + { status: "refused", reason: "process-not-running" }); + expect(withSignal(() => { throw errno("EINVAL"); })).toEqual( + { status: "refused", reason: "process-unverifiable" }); + expect(withSignal(() => { throw new Error("signal probe failed"); })).toEqual( + { status: "refused", reason: "process-unverifiable" }); + }); + + test("an unverifiable identity observation is never permission evidence", () => { + trackedPid(); + writeFileSync(getRuntimePortPath(), JSON.stringify({ pid: 4242, port: 58195 }), "utf-8"); + // Whether the liveness signal was clean or EPERM, an identity that cannot be + // observed only means it could not be verified. The unelevated-desktop case + // behind #7771 is one possibility, not a proven cause. + for (const signal of [() => {}, () => { throw errno("EPERM"); }]) { + expect(inspectTrackedRuntimeRecord({ signal, identity: () => "unverifiable" })).toEqual( + { status: "refused", reason: "process-unverifiable" }); + } + // And a foreign process holding the recorded pid is a different fact entirely. + expect(inspectTrackedRuntimeRecord({ + signal: () => {}, + identity: () => "mismatch", + })).toEqual({ status: "refused", reason: "process-identity-mismatch" }); + }); + + test("runtime record failures distinguish absent, denied, invalid and pid-mismatch", () => { + trackedPid(); + const deps: TrackedRecordProbeDeps = { signal: () => {}, identity: () => "match" }; + expect(inspectTrackedRuntimeRecord(deps)).toEqual( + { status: "refused", reason: "runtime-record-absent" }); + + writeFileSync(getRuntimePortPath(), "{not json", "utf-8"); + expect(inspectTrackedRuntimeRecord(deps)).toEqual( + { status: "refused", reason: "runtime-record-invalid" }); + + writeFileSync(getRuntimePortPath(), JSON.stringify({ pid: 4242, port: 99999 }), "utf-8"); + expect(inspectTrackedRuntimeRecord(deps)).toEqual( + { status: "refused", reason: "runtime-record-invalid" }); + + writeFileSync(getRuntimePortPath(), JSON.stringify({ pid: 9999, port: 58195 }), "utf-8"); + expect(inspectTrackedRuntimeRecord(deps)).toEqual( + { status: "refused", reason: "runtime-record-pid-mismatch" }); + + expect(inspectTrackedRuntimeRecord({ + readFile: path => path.endsWith("ocx.pid") + ? "4242" + : (() => { throw errno("EACCES"); })(), + signal: () => {}, + identity: () => "match", + })).toEqual({ status: "refused", reason: "runtime-record-permission-denied" }); + }); + + test("refusal details are fixed sentences and never leak the observed error text", () => { + const marker = "t0ken-c0mmand-l1ne"; + const mismatched = describeTrackedRuntimeTarget({ + readFile: () => { throw Object.assign(new Error("open failed " + marker), { code: "EPERM" }); }, + signal: () => {}, + identity: () => "mismatch", + }); + expect(mismatched).toEqual({ untracked: "reading the recorded PID file was denied" }); + expect(JSON.stringify(mismatched)).not.toContain(marker); + + trackedPid(); + expect(describeTrackedRuntimeTarget(liveOcx)).toEqual( + { untracked: "the tracked runtime record is absent" }); + }); + + test("a cached start identity serves both readers after the native probe dies", () => { + const { trustedWmic } = trustedProbePaths(); + trackedSelf("0.0.0.0"); + const calls: string[] = []; + setProcessCommandLineExecForTests(executable => { + calls.push(executable); + if (executable === trustedWmic) return "CommandLine=ocx.exe start\r\n"; + throw new Error("unexpected process probe: " + executable); + }); + // The pidless healthz path fills the shared start-identity cache for this PID. + expect(verifyPidIdentity(process.pid)).toBe(process.pid); + expect(calls).toEqual([trustedWmic]); + // Every native probe is now dead: the cached verdict must still admit this + // record for both readers, and neither may ask again. + setProcessCommandLineExecForTests(executable => { + calls.push(executable); + throw new Error("native probe unavailable"); + }); + expect(readPid()).toBe(process.pid); + expect(readRuntimePort(process.pid)?.port).toBe(58195); + expect(describeTrackedRuntimeTarget()).toEqual({ pid: process.pid, port: 58195, hostname: "0.0.0.0" }); + expect(calls).toEqual([trustedWmic]); + }); + + test("a cached foreign verdict still refuses both readers while a fresh probe could answer", () => { + trustedProbePaths(); + trackedSelf("0.0.0.0"); + setOcxStartProcessCacheForTests([[process.pid, false]]); + const calls: string[] = []; + setProcessCommandLineExecForTests(executable => { + calls.push(executable); + return "CommandLine=ocx.exe start\r\n"; + }); + expect(readPid()).toBeNull(); + expect(describeTrackedRuntimeTarget()).toEqual( + { untracked: "the recorded process does not match the expected start command" }); + expect(calls).toEqual([]); + }); + + test("an unreadable identity answer is cached by neither reader and recovers together", () => { + const { trustedWmic, trustedPowerShell } = trustedProbePaths(); + trackedSelf("0.0.0.0"); + const calls: string[] = []; + setProcessCommandLineExecForTests(executable => { + calls.push(executable); + throw new Error("native probe unavailable"); + }); + expect(readPid()).toBeNull(); + expect(describeTrackedRuntimeTarget()).toEqual( + { untracked: "the recorded process identity could not be verified" }); + // Each reader asked the identical dead sequence: WMIC, then both CIM retries. + expect(calls).toEqual([ + trustedWmic, trustedPowerShell, trustedPowerShell, + trustedWmic, trustedPowerShell, trustedPowerShell, + ]); + expect(ocxStartProcessCacheSizeForTests()).toBe(0); + // Recovery: the next miss re-probes once and the shared answer then serves both. + calls.length = 0; + setProcessCommandLineExecForTests(executable => { + calls.push(executable); + if (executable === trustedWmic) return "CommandLine=ocx.exe start\r\n"; + throw new Error("unexpected process probe: " + executable); + }); + expect(readPid()).toBe(process.pid); + expect(describeTrackedRuntimeTarget()).toEqual({ pid: process.pid, port: 58195, hostname: "0.0.0.0" }); + expect(calls).toEqual([trustedWmic]); + }); +}); From d82d7ee739d7bcd2aa55d9e2ca2bbae669b11812 Mon Sep 17 00:00:00 2001 From: JUN Date: Fri, 30 Oct 2026 20:42:02 +0900 Subject: [PATCH 12/42] fix(router): decode routed slugs via modelDisplayNames ids (#7979) * fix(router): decode routed slugs via modelDisplayNames ids * fix(router): use display names only as a last-resort slug decode * fix(router): keep concrete policy and combo targets exact * fix(router): keep retained ids ahead of display-name decode * docs(structure): record the modelDisplayNames slug-decode fallback * fix(router): try retained ids before display-name keys in the slug fallback An encoded slash-valued retainModels id now decodes ahead of a stale modelDisplayNames key, and the structure contract states the narrower guarantee that concrete policy, combo and redirect targets skip the display-name fallback. Co-authored-by: NoRain211 <170875756+NoRain211@users.noreply.github.com> * fix(router): let an ambiguous retained claim end the display-name fallback Co-authored-by: NoRain211 <170875756+NoRain211@users.noreply.github.com> * docs(providers): note the modelDisplayNames slug-decode fallback Co-authored-by: NoRain211 <170875756+NoRain211@users.noreply.github.com> --------- Co-authored-by: NoRain211 <170875756+NoRain211@users.noreply.github.com> --- .../ja/reference/configuration/providers.md | 2 +- .../ko/reference/configuration/providers.md | 2 +- .../docs/reference/configuration/providers.md | 2 +- .../ru/reference/configuration/providers.md | 2 +- .../reference/configuration/providers.md | 2 +- src/router.ts | 23 ++++- structure/runtime.md | 2 +- tests/codex-integration/slug-codec.test.ts | 90 +++++++++++++++++++ tests/routing/policy-execution.test.ts | 29 ++++++ 9 files changed, 146 insertions(+), 8 deletions(-) diff --git a/docs-site/src/content/docs/ja/reference/configuration/providers.md b/docs-site/src/content/docs/ja/reference/configuration/providers.md index 616cb1e164e..8705a8b7c64 100644 --- a/docs-site/src/content/docs/ja/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ja/reference/configuration/providers.md @@ -118,7 +118,7 @@ account を削除しても mapping は保持され、同じ id を再追加す | `models?` | `string[]` | 初期/フォールバックモデル一覧。`liveModels: false` で `models` が空でなければ、その後に `retainModels` を追加します。`models` が空または省略されている場合は、設定済みの `defaultModel`、`retainModels` の順に初期一覧を作り、重複 ID は最初の出現だけを残します。 | | `liveModels?` | `boolean` |開始/同期時にライブ カタログをフェッチします (デフォルトは `true`)。カスタムプロバイダーは `${baseUrl}/models` を使用します。組み込みはレジストリ URL とフィルターを使用する場合があります。 | | `selectedModels?` | `string[]` |検出後のカタログ許可リスト。空でない場合は、それらの ID のみが公開されます。空または省略すると、検出されたすべてのモデルが公開されます。 | -| `modelDisplayNames?` | `Record` | このプロバイダーの正確なネイティブモデル ID をキーにした、永続的な表示専用ラベルです。大文字と小文字は区別されます。ラベルはプロバイダーカタログのメタデータより優先され、認証、アダプター、ルーティング、課金、上流リクエストには影響しません。マップは検出上限と同じ 2,000 件までです。 | +| `modelDisplayNames?` | `Record` | このプロバイダーの正確なネイティブモデル ID をキーにした、永続的な表示専用ラベルです。大文字と小文字は区別されます。ラベルはプロバイダーカタログのメタデータより優先され、認証、アダプター、ルーティング、課金、上流リクエストには影響しません。例外として、直接の `provider/model` セレクターが既知の ID、エイリアス、`retainModels` のいずれにも一致しない場合(ライブ一覧がまだ取得されていない間など)、エンコード形が一致するキーをネイティブ ID として送信します。マップは検出上限と同じ 2,000 件までです。 | | `contextWindow?` | `number` | アップストリームのメタデータが無い場合に使うプロバイダー全体のコンテキスト値。メタデータがある場合は上限として働き、より小さいライブ値をそのまま残します。Models ダッシュボードでは `providerContextCaps` とは別に設定します。 | | `modelContextWindows?` | `Record` | モデルごとのコンテキスト値および上限。`contextWindow` より優先され、ウィンドウが不明なら設定値を使い、より小さいライブメタデータがあればそちらが優先されます。 | | `inferContextWindow?` | `boolean` | カスタムプロバイダーのみ、既定は無効。`true` で欠けたウィンドウを同梱モデルメタデータから補完します。推定値であり、中継の容量を検証したものではありません。 | diff --git a/docs-site/src/content/docs/ko/reference/configuration/providers.md b/docs-site/src/content/docs/ko/reference/configuration/providers.md index b454ff2022d..0bb995ed1ef 100644 --- a/docs-site/src/content/docs/ko/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ko/reference/configuration/providers.md @@ -118,7 +118,7 @@ managed map을 활성화하면 privacy-safe selector를 만들고, 이후 계정 | `models?` | `string[]` | 초기/폴백 모델 목록입니다. `liveModels: false`에서 `models`가 비어 있지 않으면 `models`, `retainModels` 순으로 구성합니다. `models`가 비어 있거나 생략되면 설정된 `defaultModel`, `retainModels` 순으로 구성하고, 중복 ID는 처음 나온 항목만 남깁니다. | | `liveModels?` | `boolean` | 시작 또는 동기화 시 라이브 카탈로그를 가져옵니다. 기본값은 `true`입니다. 사용자 지정 공급자는 `${baseUrl}/models`를 사용하고, 내장은 레지스트리 URL을 사용한 뒤 필터링할 수 있습니다. | | `selectedModels?` | `string[]` | 발견 후 카탈로그 허용 목록입니다. 값이 비어 있지 않으면 그 id만 노출하고, 비어 있거나 생략하면 발견된 모델을 모두 노출합니다. | -| `modelDisplayNames?` | `Record` | 이 공급자의 정확한 네이티브 모델 id를 키로 쓰는 영구 표시 전용 이름입니다. 키는 대소문자를 구분합니다. 이름은 공급자 카탈로그 메타데이터보다 우선하며 인증, 어댑터, 라우팅, 청구 또는 업스트림 요청을 바꾸지 않습니다. 맵은 발견 한도와 같은 최대 2,000개 항목을 가질 수 있습니다. | +| `modelDisplayNames?` | `Record` | 이 공급자의 정확한 네이티브 모델 id를 키로 쓰는 영구 표시 전용 이름입니다. 키는 대소문자를 구분합니다. 이름은 공급자 카탈로그 메타데이터보다 우선하며 인증, 어댑터, 라우팅, 청구 또는 업스트림 요청을 바꾸지 않습니다. 단, 직접 `provider/model` 선택자가 알려진 id, 별칭, `retainModels` 어디에도 맞지 않으면(예: 라이브 목록을 아직 받지 못한 동안) 인코딩 형태가 같은 키를 네이티브 id로 보냅니다. 맵은 발견 한도와 같은 최대 2,000개 항목을 가질 수 있습니다. | | `contextWindow?` | `number` | 업스트림 메타데이터가 없을 때 쓰이는 공급자 전반의 컨텍스트 값입니다. 메타데이터가 있으면 상한으로 동작해 더 작은 라이브 값을 그대로 둡니다. Models 대시보드에서 `providerContextCaps`와 별도로 설정합니다. | | `modelContextWindows?` | `Record` | 모델별 컨텍스트 값이자 상한입니다. `contextWindow`보다 우선하며, 창 크기를 알 수 없으면 설정값을 쓰고 더 작은 라이브 메타데이터가 있으면 그쪽을 따릅니다. | | `inferContextWindow?` | `boolean` | 사용자 지정 공급자 전용이며 기본값은 꺼짐입니다. `true`이면 빠진 창을 번들 모델 메타데이터로 채웁니다. 추정값이며 중계의 실제 용량을 검증하지 않습니다. | diff --git a/docs-site/src/content/docs/reference/configuration/providers.md b/docs-site/src/content/docs/reference/configuration/providers.md index f77855f4cc4..ae998f0f119 100644 --- a/docs-site/src/content/docs/reference/configuration/providers.md +++ b/docs-site/src/content/docs/reference/configuration/providers.md @@ -335,7 +335,7 @@ Providers can expose a built-in shorthand, such as `agy` for `google-antigravity | `liveModels?` | `boolean` | Fetch the live catalog on start/sync (default `true`). Custom providers use `${baseUrl}/models`; built-ins may use a registry URL and filter. | | `selectedModels?` | `string[]` | Catalog allowlist after discovery. Non-empty exposes only those ids; empty or omitted exposes all discovered models. | | `retainModels?` | `string[]` | Ids kept in the catalog even when live discovery omits them. They need not be repeated in `models`. Empty or omitted keeps today's behavior. | -| `modelDisplayNames?` | `Record` | Durable labels used only for display, keyed by this provider's exact upstream model id. Labels win over provider catalog metadata, survive discovery refreshes and provider edits, and never change authentication, adapter behavior, routing, billing, upstream request construction, the routed `provider/model` selector, or the upstream wire model. Keys are exact and case sensitive. Unknown model ids are kept so a temporarily missing model receives its label when it returns. The map accepts at most 2,000 entries, matching the discovery limit. | +| `modelDisplayNames?` | `Record` | Durable labels used only for display, keyed by this provider's exact upstream model id. Labels win over provider catalog metadata, survive discovery refreshes and provider edits, and never change authentication, adapter behavior, billing, upstream request construction, the routed `provider/model` selector, or the wire model of an id the provider already knows. One routing exception: when a direct `provider/model` selector matches no known id, alias or `retainModels` id (for example while a live roster is still cold), a key whose encoded form matches is sent as the native id. Keys are exact and case sensitive. Unknown model ids are kept so a temporarily missing model receives its label when it returns. The map accepts at most 2,000 entries, matching the discovery limit. | | `contextWindow?` | `number` | Provider-wide context fallback when upstream metadata is absent; otherwise a cap that retains smaller live metadata. Registry defaults cannot override a lower configured limit; explicit `modelContextWindows` entries take priority. The Models dashboard exposes this separately from `providerContextCaps`. | | `modelContextWindows?` | `Record` | Per-model context fallbacks/caps. These override `contextWindow`: an unknown window uses the configured value, while smaller live metadata remains authoritative. | | `inferContextWindow?` | `boolean` | Custom providers only, default off. `true` fills a missing window from bundled model-ID metadata ([details](#context-windows-on-custom-providers)); the value is an estimate, not verified relay capacity. | diff --git a/docs-site/src/content/docs/ru/reference/configuration/providers.md b/docs-site/src/content/docs/ru/reference/configuration/providers.md index f8ef70c7a55..af23015f1de 100644 --- a/docs-site/src/content/docs/ru/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ru/reference/configuration/providers.md @@ -131,7 +131,7 @@ cross-route credential fallback не существует. Строки API GPT- | `models?` | `string[]` | Начальный список моделей или список для резервного режима. При `liveModels: false` за непустым `models` следует `retainModels`; если `models` пуст или отсутствует, сначала берётся настроенный `defaultModel`, затем `retainModels`. Для каждого ID сохраняется только первое вхождение. | | `liveModels?` | `boolean` | Получать live-каталог на start/sync (по умолчанию `true`). Custom-провайдеры используют `${baseUrl}/models`; built-in могут использовать registry URL и дополнительно фильтровать результат. | | `selectedModels?` | `string[]` | Allowlist каталога после discovery. Непустой список показывает только эти id; пустой или отсутствующий показывает всё, что было обнаружено. | -| `modelDisplayNames?` | `Record` | Постоянные display-only имена с точным нативным id модели этого провайдера в качестве ключа. Ключи чувствительны к регистру. Имена имеют приоритет над metadata каталога провайдера и не меняют аутентификацию, adapter, routing, billing или upstream-запросы. Карта содержит не более 2 000 записей, как и discovery. | +| `modelDisplayNames?` | `Record` | Постоянные display-only имена с точным нативным id модели этого провайдера в качестве ключа. Ключи чувствительны к регистру. Имена имеют приоритет над metadata каталога провайдера и не меняют аутентификацию, adapter, routing, billing или upstream-запросы. Исключение: если прямой селектор `provider/model` не совпадает ни с известным id, ни с alias, ни с `retainModels` (например, пока live-каталог ещё не загружен), ключ с совпадающей закодированной формой отправляется как нативный id. Карта содержит не более 2 000 записей, как и discovery. | | `contextWindow?` | `number` | Значение контекста для всего провайдера, применяемое когда upstream не отдаёт metadata; при наличии metadata работает как cap и сохраняет более маленькое live-значение. Панель Models настраивает его отдельно от `providerContextCaps`. | | `modelContextWindows?` | `Record` | Значения и cap'ы контекста по отдельным моделям. Перекрывают `contextWindow`: если окно неизвестно, берётся заданное значение, а более маленькая live-metadata остаётся авторитетной. | | `inferContextWindow?` | `boolean` | Только для пользовательских провайдеров, по умолчанию выключено. `true` заполняет отсутствующее окно из встроенных метаданных модели; это оценка, а не проверенная ёмкость посредника. | diff --git a/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md b/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md index 2703729ead1..e7b199e536d 100644 --- a/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md +++ b/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md @@ -120,7 +120,7 @@ selector,而不是分配一个新名称。 | `models?` | `string[]` | 初始/回退模型列表。`liveModels: false` 时,非空 `models` 后接 `retainModels`;若 `models` 为空或省略,则按已配置的 `defaultModel`、`retainModels` 顺序构建初始列表,重复 ID 仅保留首次出现的位置。 | | `liveModels?` | `boolean` | 启动/同步时获取实时目录(默认 `true`)。自定义提供者使用 `${baseUrl}/models`;内置项可能使用注册表 URL 并进行过滤。 | | `selectedModels?` | `string[]` | 发现之后的目录允许列表。非空时只暴露这些 id;为空或省略时则暴露全部发现到的模型。 | -| `modelDisplayNames?` | `Record` | 持久的仅显示名称,以此提供者的精确原生模型 id 为键。键区分大小写。名称优先于提供者目录元数据,并且不会改变身份验证、适配器、路由、计费或上游请求。该映射最多可包含 2,000 个条目,与发现上限相同。 | +| `modelDisplayNames?` | `Record` | 持久的仅显示名称,以此提供者的精确原生模型 id 为键。键区分大小写。名称优先于提供者目录元数据,并且不会改变身份验证、适配器、路由、计费或上游请求。例外:当直接的 `provider/model` 选择器不匹配任何已知 id、别名或 `retainModels`(例如实时目录尚未获取时),编码形式相同的键会作为原生 id 发送。该映射最多可包含 2,000 个条目,与发现上限相同。 | | `contextWindow?` | `number` | 上游缺少元数据时使用的提供者级上下文数值;有元数据时作为上限,保留更小的实时数值。Models 面板中与 `providerContextCaps` 分开设置。 | | `modelContextWindows?` | `Record` | 按模型设置的上下文数值与上限。优先于 `contextWindow`:窗口未知时采用所配置的数值,而更小的实时元数据仍然优先。 | | `inferContextWindow?` | `boolean` | 仅限自定义渠道,默认关闭。设为 `true` 时用内置模型元数据补全缺失窗口;这是估计值,不代表中转的实测容量。 | diff --git a/src/router.ts b/src/router.ts index d3939298e05..b9f4b85ae99 100644 --- a/src/router.ts +++ b/src/router.ts @@ -30,7 +30,7 @@ import { LEGACY_OPENAI_MULTI_PROVIDER_ID, OPENAI_CODEX_PROVIDER_ID, } from "./providers/openai-tiers"; -import { decodeRoutedModelIdOrThrow, encodeRoutedModelId } from "./providers/slug-codec"; +import { decodeRoutedModelId, decodeRoutedModelIdOrThrow, encodeRoutedModelId } from "./providers/slug-codec"; import { effectiveProviderAliasDecision, resolveModelAlias } from "./providers/default-aliases"; import { resolveBlockedModelRedirect } from "./lib/shadow-call"; import { workBuddyCatalogIdentity } from "./providers/workbuddy"; @@ -200,6 +200,20 @@ export function knownModelIdsForProvider( * (never a caller-supplied string), so the log carries no request content. */ const compactionFallbackWarnings = new Set(); +/** + * Last-resort decode for a direct provider-qualified slug the live roster cannot resolve yet. + * Retained ids win (an encoded slash-valued retained id included), then the operator-declared + * modelDisplayNames keys. Neither list feeds alias generation. + */ +function decodeDisplayNameFallback(prov: OcxProviderConfig, decoded: string): string { + const retained = prov.retainModels ?? []; + // Any retained claim, even an ambiguous one the codec refuses to guess, ends the fallback. + if (retained.some(id => id === decoded || encodeRoutedModelId(id) === decoded)) { + return decodeRoutedModelId(decoded, retained); + } + return decodeRoutedModelId(decoded, Object.keys(prov.modelDisplayNames ?? {})); +} + function warnCompactionDefaultProviderFallbackOnce(providerName: string): void { if (compactionFallbackWarnings.has(providerName)) return; compactionFallbackWarnings.add(providerName); @@ -583,7 +597,12 @@ function routeModelInternal( const decoded = decodeRoutedModelIdOrThrow(requestedModel, known); const nativeModel = known.includes(decoded) ? decoded - : resolveModelAlias(config, prov, known, requestedModel) ?? decoded; + : resolveModelAlias(config, prov, known, requestedModel) + // Last resort for direct requests while the live roster is cold: display-name keys are + // operator-declared native ids, consulted only here so they never claim aliases or shadow + // known or retained ids. Concrete policy/combo/redirect targets (bypassCombos) keep the + // exact model their evaluator selected, so evidence and eligibility describe what is sent. + ?? (bypassCombos ? decoded : decodeDisplayNameFallback(prov, decoded)); return routeResult( config, provName, diff --git a/structure/runtime.md b/structure/runtime.md index ac5b871d6c6..0af2d36768d 100644 --- a/structure/runtime.md +++ b/structure/runtime.md @@ -18,7 +18,7 @@ Release packaging, Docker, installed-artifact and service smoke jobs, desktop sh `src/router.ts` attaches one frozen `ResolvedModelPolicy` to every `RouteResult`. Fast observation, persistence and cost provenance follow the [response-tier authority contract](transports/responses.md#response-tier-observation-authority); outbound Fast policy is unchanged. Policy/combo route spreads retain that object. Every initial, fallback, and recovery route is recaptured for the request's original inbound protocol before route-dependent normalization, and all adapter rebuilds consume its recorded adapter. A translated Chat or Anthropic replay therefore cannot inherit a Responses-only default. Credential, account, quota, health, cooldown, and observed transport evidence remain late and cannot widen a captured static limit. -Virtual models are the sole model-identity transition: the ordinary and compact paths preserve the selected public id in diagnostics, rewrite `route.modelId` to the upstream wire id, and atomically replace `route.staticPolicy` before adapter or capability decisions continue. Model aliases are resolved before the route result is built, so their policy is already keyed by the native wire id. `src/router.ts` applies bare `blockedModelRedirects` keys after this resolution; ordinary targets substitute one upstream model id on the selected provider/account, including slash-valued targets, while only targets explicitly naming a different configured provider reroute. Qualified source keys take precedence only for those cross-provider targets. A cross-provider redirect sets `RouteResult.credentialDomainRewrite`, which Responses and Chat header scoping treat like combo/policy routes, so a caller credential for the source route is never forwarded. `src/cli/alias.ts` consumes the JSON output flag before choosing its default list action. `ocx alias --json` and both flag positions around an explicit action preserve the selected read or mutation. Live selector hints obey the [credential-scoped cache contract](catalog.md): a selection change cannot reuse the previous credential's roster to choose an alias target. Passive OAuth observation neither refreshes credentials nor repairs their storage. Cross-provider redirects require an enabled destination with a resolved key, usable stored OAuth account, or local/keyless auth (never source-caller forward auth), and share one visited set and five-edge budget across alias, provider, combo and policy resolution; every policy redirect must end at an eligible declared candidate, including same-provider substitutions and candidate fallback retries, and pinned account selectors reject provider changes. Decision traces name the final provider/model while policy candidates retain selection evidence and combos retain their route kind and redirect reason. +Virtual models are the sole model-identity transition: the ordinary and compact paths preserve the selected public id in diagnostics, rewrite `route.modelId` to the upstream wire id, and atomically replace `route.staticPolicy` before adapter or capability decisions continue. Model aliases are resolved before the route result is built, so their policy is already keyed by the native wire id. `src/router.ts` applies bare `blockedModelRedirects` keys after this resolution; ordinary targets substitute one upstream model id on the selected provider/account, including slash-valued targets, while only targets explicitly naming a different configured provider reroute. Qualified source keys take precedence only for those cross-provider targets. A cross-provider redirect sets `RouteResult.credentialDomainRewrite`, which Responses and Chat header scoping treat like combo/policy routes, so a caller credential for the source route is never forwarded. `src/cli/alias.ts` consumes the JSON output flag before choosing its default list action. `ocx alias --json` and both flag positions around an explicit action preserve the selected read or mutation. Live selector hints obey the [credential-scoped cache contract](catalog.md): a selection change cannot reuse the previous credential's roster to choose an alias target. Passive OAuth observation neither refreshes credentials nor repairs their storage. Cross-provider redirects require an enabled destination with a resolved key, usable stored OAuth account, or local/keyless auth (never source-caller forward auth), and share one visited set and five-edge budget across alias, provider, combo and policy resolution; every policy redirect must end at an eligible declared candidate, including same-provider substitutions and candidate fallback retries, and pinned account selectors reject provider changes. Decision traces name the final provider/model while policy candidates retain selection evidence and combos retain their route kind and redirect reason. An explicit provider-qualified slug decodes against `knownModelIdsForProvider` first, then model aliases. Only when both miss on a direct request does `src/router.ts` decode once more against the provider's `modelDisplayNames` keys, which are operator-declared native ids; this keeps a live-discovery provider such as Baseten routable while its roster cache is cold. The fallback never feeds `knownModelIdsForProvider`, so display-name keys claim no aliases and never shadow a configured id; `retainModels` ids (an encoded slash-valued one included) are tried before display-name keys, and concrete policy, combo and blocked-model redirect targets skip the display-name fallback (`tests/codex-integration/slug-codec.test.ts`, `tests/routing/policy-execution.test.ts`). Routed Meta Muse requests use the registry-owned [Muse effort and header contract](providers-and-adapters.md); `max` reaches the provider through the existing reasoning mapper. diff --git a/tests/codex-integration/slug-codec.test.ts b/tests/codex-integration/slug-codec.test.ts index 3a07001def8..1f521e3ebfc 100644 --- a/tests/codex-integration/slug-codec.test.ts +++ b/tests/codex-integration/slug-codec.test.ts @@ -301,6 +301,96 @@ describe("routeModel decode (proxy layer)", () => { expect(route.modelId).toBe("vendor/m-2"); }); + test("modelDisplayNames keys decode a provider-qualified slug with a cold live cache", () => { + const config: OcxConfig = { + port: 10100, + defaultProvider: "other", + providers: { + other: { + adapter: "openai-chat", + baseUrl: "https://example.com/v1", + apiKey: "k", + modelDisplayNames: { "vendor/m-3": "Vendor M3" }, + }, + }, + }; + const route = routeModel(config, "other/vendor-m-3"); + expect(route.providerName).toBe("other"); + expect(route.modelId).toBe("vendor/m-3"); + }); + + test("a stale modelDisplayNames key does not shadow a configured id", () => { + const config: OcxConfig = { + port: 10100, + defaultProvider: "other", + providers: { + other: { + adapter: "openai-chat", + baseUrl: "https://example.com/v1", + apiKey: "k", + models: ["a-b"], + modelDisplayNames: { "a/b": "Stale A/B" }, + }, + }, + }; + expect(routeModel(config, "other/a-b").modelId).toBe("a-b"); + }); + + test("a stale modelDisplayNames key does not shadow a retained id", () => { + const config: OcxConfig = { + port: 10100, + defaultProvider: "other", + providers: { + other: { + adapter: "openai-chat", + baseUrl: "https://example.com/v1", + apiKey: "k", + retainModels: ["a-b"], + modelDisplayNames: { "a/b": "Stale A/B" }, + }, + }, + }; + expect(routeModel(config, "other/a-b").modelId).toBe("a-b"); + }); + + test("an encoded slash-valued retained id wins over a stale display-name key", () => { + const config: OcxConfig = { + port: 10100, + defaultProvider: "other", + providers: { + other: { + adapter: "openai-chat", + baseUrl: "https://example.com/v1", + apiKey: "k", + retainModels: ["a/b"], + modelDisplayNames: { "a-b": "Stale A-B" }, + }, + }, + }; + expect(routeModel(config, "other/a-b").modelId).toBe("a/b"); + config.providers.other!.retainModels = ["x/y-z", "x-y/z"]; + config.providers.other!.modelDisplayNames = { "x/y-z": "Stale" }; + expect(routeModel(config, "other/x-y-z").modelId).toBe("x-y-z"); + }); + + test("a stale modelDisplayNames key does not break an existing default alias", () => { + const config: OcxConfig = { + port: 10100, + defaultProvider: "other", + providers: { + other: { + adapter: "openai-chat", + baseUrl: "https://example.com/v1", + apiKey: "k", + defaultAliases: true, + models: ["vendor/claude-opus-5"], + modelDisplayNames: { "vendor/claude-opus-5-old": "Stale Opus" }, + }, + }, + }; + expect(routeModel(config, "other/opus").modelId).toBe("vendor/claude-opus-5"); + }); + test("knownModelIdsForProvider unions config, registry, and hint-map ids", () => { const ids = knownModelIdsForProvider("zenmux", zenmuxConfig().providers.zenmux!); expect(ids).toContain("moonshotai/kimi-k3-free"); diff --git a/tests/routing/policy-execution.test.ts b/tests/routing/policy-execution.test.ts index 96b6405a5b5..3508c5a2594 100644 --- a/tests/routing/policy-execution.test.ts +++ b/tests/routing/policy-execution.test.ts @@ -134,6 +134,35 @@ describe("policy execution (RI-05)", () => { expect(route.modelId).toBe("m1"); }); + test("policy and combo targets keep their exact model when a display name could re-decode it", () => { + const base = baseConfig(); + const config = baseConfig({ + providers: { + ...base.providers, + other: { + adapter: "openai-chat", + baseUrl: "https://other.example/v1", + apiKey: "ko", + modelDisplayNames: { "a/b": "Display-only A/B" }, + }, + }, + combos: { exactcombo: { strategy: "failover", targets: [{ provider: "other", model: "a-b" }] } }, + routingProfiles: { + exact: { + candidates: [{ provider: "other", model: "a-b" }], + unknownEvidence: { capability: "penalize", health: "penalize", quota: "penalize", cost: "penalize" }, + }, + }, + }); + expect(routeModel(config, "other/a-b").modelId).toBe("a/b"); + const policy = routeModel(config, "policy/exact"); + expect(policy.modelId).toBe("a-b"); + expect(policy.routeDecision!.selected.model).toBe("a-b"); + const combo = routeModel(config, "combo/exactcombo"); + expect(combo.modelId).toBe("a-b"); + expect(combo.combo!.target.model).toBe("a-b"); + }); + test("existing explicit, combo, native and default routes are unchanged", () => { const config = baseConfig(); expect(routeModel(config, "a/m1")).toMatchObject({ routeKind: "explicit-provider", providerName: "a", modelId: "m1" }); From 1e1620f3d9c2bee0c45591f3d074ae5f141fccea Mon Sep 17 00:00:00 2001 From: JUN Date: Fri, 30 Oct 2026 20:43:19 +0900 Subject: [PATCH 13/42] fix(responses): normalize Codex tool catalogs and include for Meta Muse (#7985) * fix(responses): normalize Codex tool catalogs and include for Meta Muse * test(muse): assert web search after public catalog projection * fix(meta): preserve catalog precedence and validate all input forms * fix(responses): skip validation of historical Muse catalogs Co-authored-by: nobrainer-tech <445466+nobrainer-tech@users.noreply.github.com> --------- Co-authored-by: nobrainer-tech Co-authored-by: nobrainer-tech <445466+nobrainer-tech@users.noreply.github.com> --- .../src/content/docs/reference/adapters.md | 21 ++ .../openai-responses/muse-tool-catalog.ts | 81 ++++++ src/adapters/openai-responses/passthrough.ts | 15 +- .../openai-responses/request-strips.ts | 27 +- src/server/responses/passthrough-dispatch.ts | 2 + structure/providers/opengateway.md | 2 + structure/providers/xai-grok.md | 2 +- structure/transports/response-citations.md | 2 + structure/transports/responses-citations.md | 2 + structure/transports/responses-wire-shapes.md | 4 +- .../muse-spark-web-search-compat.test.ts | 17 +- .../responses-muse-tool-choice.test.ts | 251 +++++++++++++++++- 12 files changed, 393 insertions(+), 33 deletions(-) create mode 100644 src/adapters/openai-responses/muse-tool-catalog.ts diff --git a/docs-site/src/content/docs/reference/adapters.md b/docs-site/src/content/docs/reference/adapters.md index 82cf7d00436..0545fbbd8e6 100644 --- a/docs-site/src/content/docs/reference/adapters.md +++ b/docs-site/src/content/docs/reference/adapters.md @@ -260,6 +260,27 @@ blank strings and mixed encrypted/unknown parts are not partially converted. See [agent messages](/reference/configuration/providers/#routed-agent-messages) for the separate opt-in encrypted-task recovery behavior. +Meta Responses also receives Codex's current `additional_tools` declarations in the public +`tools` array. The catalog is captured before history normalization: replayed declarations +and historical tool-search discoveries cannot activate tools for the new turn. Current +namespace, custom-tool and long-name aliases retain the existing normalization and response +restoration. Current declarations keep their parameter schemas when search results repeat a tool. +Historical custom calls retain their existing function-call representation without granting +those old tools current authority. Nested function definitions become public declarations. +Malformed or unrepresentable namespace catalogs return HTTP 400 before inference, including +requests with string or omitted input, rather than being partially promoted. + +For Meta Responses at the exact `api.meta.ai` host, OpenCodex omits the unsupported +`output.encrypted_content` include requested by Codex Desktop. Meta otherwise rejects the +request before inference. `reasoning.encrypted_content` and other include entries remain +intact, so stateless reasoning replay is preserved. The same exact-entry normalization +applies to xAI; unrelated Responses destinations keep their original include list. + +These request-shape fixes do not change account entitlements. Meta limits Muse Code +subscription credentials to the Muse Code client; other clients require a separately +billed Meta API key. A subscription key can therefore return HTTP 403 through OpenCodex +even when the request shape is valid. See [Meta's subscription rules](https://dev.meta.ai/docs/muse-code/subscriptions). + For xAI Responses, `auto` or `none` tool selection is omitted when normalization leaves no tools in the request, including when cached-only search is removed. Valid forced function selections remain intact. Replayed custom tool calls with missing or invalid item ids receive stable ids diff --git a/src/adapters/openai-responses/muse-tool-catalog.ts b/src/adapters/openai-responses/muse-tool-catalog.ts new file mode 100644 index 00000000000..bf2811e00eb --- /dev/null +++ b/src/adapters/openai-responses/muse-tool-catalog.ts @@ -0,0 +1,81 @@ +import { isPlainObject } from "./internal"; +import { promoteClientLoadedTools } from "./tool-schema"; +import { rewriteRoutedCustomToolsForUpstream } from "../../responses/custom-tool-compat"; + +export class MuseToolCatalogCompatibilityError extends Error { + constructor() { + super("Malformed tool catalog for Meta Responses."); + this.name = "MuseToolCatalogCompatibilityError"; + } +} + +function isRepresentableName(name: unknown): name is string { + return typeof name === "string" && name.length > 0 + && [...name].every(character => character.charCodeAt(0) > 0x1f && character.charCodeAt(0) !== 0x7f); +} + +function normalizeCatalog(tools: unknown, inNamespace = false): Record[] { + if (!Array.isArray(tools)) throw new MuseToolCatalogCompatibilityError(); + return tools.flatMap(tool => { + if (!isPlainObject(tool) || typeof tool.type !== "string") throw new MuseToolCatalogCompatibilityError(); + if (inNamespace && tool.type !== "function" && tool.type !== "custom") throw new MuseToolCatalogCompatibilityError(); + if (tool.type === "function" && isPlainObject(tool.function)) { + const { function: definition, ...outer } = tool; + tool = { ...outer, ...definition, type: "function" }; + } + if (["namespace", "function", "custom"].includes(tool.type) + && !isRepresentableName(tool.name)) throw new MuseToolCatalogCompatibilityError(); + if (tool.type !== "namespace") return [tool]; + return [{ ...tool, tools: normalizeCatalog(tool.tools, true) }]; + }); +} + +function flattenBuiltinNamespace(tools: Record[]): Record[] { + const bareNames = new Set(tools.filter(tool => tool.type !== "namespace").map(tool => tool.name)); + return tools.flatMap(tool => { + if (tool.type !== "namespace" || tool.name !== "functions") return [tool]; + // Match namespace lowering: an explicit bare declaration owns this identity, + // even when it occurs later or in a different current declaration carrier. + return (tool.tools as Record[]).filter(child => !bareNames.has(child.name)) + .map(child => tool.defer_loading === undefined ? child : { defer_loading: tool.defer_loading, ...child }); + }); +} + +/** + * Meta rejects Codex's additional_tools input items. Capture the current catalog + * before history rewrites can shift the replay boundary or consume duplicate names. + * Historical messages and search results remain replay context, never declarations. + * The caller gates this projection on the actual Meta destination. + */ +export function normalizeMuseToolCatalog(body: unknown, replayPrefixLength = 0, supportsCustomTools?: boolean): unknown { + if (!isPlainObject(body)) return body; + // Namespace lowering drops malformed groups; reject them before any partial promotion. + const topLevelTools = body.tools === undefined ? undefined : normalizeCatalog(body.tools); + if (!Array.isArray(body.input)) return topLevelTools === undefined ? body : { ...body, tools: flattenBuiltinNamespace(topLevelTools) }; + const start = Number.isFinite(replayPrefixLength) + ? Math.min(body.input.length, Math.max(0, Math.trunc(replayPrefixLength))) : 0; + const normalizedInput = body.input.map((item, index) => { + if (index < start) return item; + if (isPlainObject(item) && (item.type === "additional_tools" + || (item.type === "tool_search_output" && item.tools !== undefined))) { + const tools = normalizeCatalog(item.tools); + return { ...item, tools: item.type === "tool_search_output" ? flattenBuiltinNamespace(tools) : tools }; + } + return item; + }); + const currentInput = normalizedInput.slice(start); + const declared = [...(topLevelTools ?? [])]; + for (const item of currentInput) { + if (isPlainObject(item) && item.type === "additional_tools" && Array.isArray(item.tools)) declared.push(...item.tools); + } + // All current declarations precede discoveries, regardless of their carrier. + const current = promoteClientLoadedTools({ ...body, tools: flattenBuiltinNamespace(declared), input: currentInput }); + const tools = isPlainObject(current) && Array.isArray(current.tools) ? current.tools : declared; + // Replay still needs the old declarations to lower custom calls and their outputs. + // Retain only that input projection, never the historical declaration/restore sets. + const replay = rewriteRoutedCustomToolsForUpstream({ ...body, input: normalizedInput }, supportsCustomTools).body; + const replayInput = isPlainObject(replay) && Array.isArray(replay.input) ? replay.input : normalizedInput; + const input = replayInput.filter(item => !isPlainObject(item) || item.type !== "additional_tools"); + const hasCatalog = topLevelTools !== undefined || tools.length > 0 || input.length !== body.input.length; + return { ...body, input, ...(hasCatalog ? { tools } : {}) }; +} diff --git a/src/adapters/openai-responses/passthrough.ts b/src/adapters/openai-responses/passthrough.ts index 256707dc813..9cb4657c64f 100644 --- a/src/adapters/openai-responses/passthrough.ts +++ b/src/adapters/openai-responses/passthrough.ts @@ -40,7 +40,7 @@ import { emittedFastWire, } from "../../providers/fastwire"; import { dropResponsesReasoningInputItems, mapRoutedResponsesReasoningEffort, normalizeConfiguredReasoningSummaryDelivery, sanitizeReasoningInputContent, stripDisabledReasoningSummaries, stripDisabledVerbosity, stripInternalReasoningEnabled, stripNoneReasoningSummary, stripUnsupportedReasoningSummaryDelivery } from "./reasoning"; -import { repairUrlCitationAnnotations, scrubOcxCompactionItems, stripXaiUnsupportedInclude, stripCanonicalOnlyInputItems, stripCanonicalOnlyToolFields, stripCanonicalOnlyTopLevelFields, stripInternalChatMessageMetadataPassthrough, stripInvalidItemIds, stripItemIdsWhenUnstored, stripProviderInputItemMetadata, stripRejectedSamplingParams } from "./request-strips"; +import { repairUrlCitationAnnotations, scrubOcxCompactionItems, stripUnsupportedOutputInclude, stripCanonicalOnlyInputItems, stripCanonicalOnlyToolFields, stripCanonicalOnlyTopLevelFields, stripInternalChatMessageMetadataPassthrough, stripInvalidItemIds, stripItemIdsWhenUnstored, stripProviderInputItemMetadata, stripRejectedSamplingParams } from "./request-strips"; import { notePlaintextReasoningDropped } from "./plaintext-reasoning-notice"; import { stripCanonicalForwardPromptCacheOptions, stripDeprecatedPromptCacheRetention } from "./prompt-cache"; import { isPlainObject } from "./internal"; @@ -53,6 +53,7 @@ import { normalizeImageGenClientTools, preferConfiguredHostedTools } from "./ima import { stripMuseSparkUnsupportedWebSearchFields, stripOpenAiOnlyWebSearchFields } from "./web-search"; import { observeOutbound } from "../../usage/cache-diagnostic"; +import { normalizeMuseToolCatalog } from "./muse-tool-catalog"; import { normalizeMuseToolChoice } from "./muse-tool-choice"; import { declaresHostedWebSearch, projectHostedSearchHistory, renderCompactionSearchHistoryItems } from "./compaction-search-history"; import { stripKimiEmptyAssistantMessages } from "./empty-assistant"; @@ -178,6 +179,9 @@ export function createResponsesPassthroughAdapter(provider: OcxProviderConfig): parsed._rawBody, forward || parsed._previousResponseInputExpanded === true, ); + if (isMetaAiResponsesDestination(url)) { + outBody = normalizeMuseToolCatalog(outBody, parsed._replayPrefixLen, provider.supportsResponsesCustomTools); + } if (!forward) outBody = normalizeRoutedAgentMessages(outBody, { allowStringContent: isXaiResponsesDestination(provider), }); @@ -308,7 +312,8 @@ export function createResponsesPassthroughAdapter(provider: OcxProviderConfig): outBody = stripCanonicalOnlyTopLevelFields(outBody); outBody = stripCanonicalOnlyInputItems(outBody); } - outBody = promoteClientLoadedTools(outBody); + // Meta already captured discoveries from the original current-turn slice. + if (!isMetaAiResponsesDestination(url)) outBody = promoteClientLoadedTools(outBody); } if (!isCanonicalOpenAiForwardProvider(provider)) { const rewritten = rewriteRoutedCustomToolsForUpstream( @@ -338,10 +343,10 @@ export function createResponsesPassthroughAdapter(provider: OcxProviderConfig): // OpenCode Go speaks the same dialect, which the final model id and URL identify. outBody = normalizeXaiResponsesWebSearch(outBody, provider, { modelId: parsed.modelId, responseUrl: url }); outBody = injectXaiResponsesXSearch(outBody, provider, parsed._replayPrefixLen); - // xAI answers 400 for `output.encrypted_content` in `include` instead of ignoring it; + // xAI and Meta answer 400 for `output.encrypted_content` instead of ignoring it; // Codex Desktop asks for it on every turn (#7878). - if (isXaiResponsesDestination(provider)) { - outBody = stripXaiUnsupportedInclude(outBody); + if (isXaiResponsesDestination(provider) || isMetaAiResponsesDestination(url)) { + outBody = stripUnsupportedOutputInclude(outBody); } // xAI and explicitly classified compatible gateways reject these OpenAI web_search // extensions. Keep them for OpenAI API-key traffic and unclassified gateways. diff --git a/src/adapters/openai-responses/request-strips.ts b/src/adapters/openai-responses/request-strips.ts index f7e9e5f2580..82bf33520c3 100644 --- a/src/adapters/openai-responses/request-strips.ts +++ b/src/adapters/openai-responses/request-strips.ts @@ -8,24 +8,13 @@ import { activateDeferredTool } from "./tool-schema"; import { stripOpenAiOnlyWebSearchFields } from "./web-search"; /** - * Provider-opaque replay metadata is client history, not an OpenAI input-item field (#7111). - * The caller gates this on OpenAI-operated destinations; arbitrary gateways may accept it. - * Strip only the known extension, never an item-field allowlist that can erase newer wire fields - * or nested tool data. Copy on write so a later Google route can still replay its signature. - */ -/** Responses `include` entries xAI rejects with a 400 rather than ignoring (#7878). */ -const XAI_UNSUPPORTED_RESPONSES_INCLUDE = new Set(["output.encrypted_content"]); - -/** - * Drop `include` entries xAI's Responses API refuses (`Argument not supported: - * "output.encrypted_content" in "include" field`). Codex Desktop always asks for the - * OpenAI reasoning blob, so without this every xai/grok-4.7 turn sent through the - * Responses path fails, while the same model works on the openai-chat adapter. - * Copy on write; other destinations and other include entries are untouched. + * Codex requests output.encrypted_content, which xAI and Meta reject with HTTP 400. + * The caller gates this on the destination. Preserve reasoning.encrypted_content + * for stateless replay and leave unknown or malformed values for upstream validation. */ -export function stripXaiUnsupportedInclude(body: unknown): unknown { +export function stripUnsupportedOutputInclude(body: unknown): unknown { if (!isPlainObject(body) || !Array.isArray(body.include)) return body; - const include = body.include.filter(item => !XAI_UNSUPPORTED_RESPONSES_INCLUDE.has(String(item))); + const include = body.include.filter(item => item !== "output.encrypted_content"); if (include.length === body.include.length) return body; if (include.length === 0) { const { include: _dropped, ...rest } = body; @@ -34,6 +23,12 @@ export function stripXaiUnsupportedInclude(body: unknown): unknown { return { ...body, include }; } +/** + * Provider-opaque replay metadata is client history, not an OpenAI input-item field (#7111). + * The caller gates this on OpenAI-operated destinations; arbitrary gateways may accept it. + * Strip only the known extension, never an item-field allowlist that can erase newer wire fields + * or nested tool data. Copy on write so a later Google route can still replay its signature. + */ export function stripProviderInputItemMetadata(body: unknown): unknown { if (!isPlainObject(body) || !Array.isArray(body.input)) return body; let input: unknown[] | undefined; diff --git a/src/server/responses/passthrough-dispatch.ts b/src/server/responses/passthrough-dispatch.ts index e253d5c9460..fdabdf1de4c 100644 --- a/src/server/responses/passthrough-dispatch.ts +++ b/src/server/responses/passthrough-dispatch.ts @@ -54,6 +54,7 @@ import { import { restoreRoutedCustomCalls, RoutedCustomToolCompatError } from "../../responses/custom-tool-compat"; import { XaiToolSchemaCompatibilityError } from "../../adapters/xai-tool-schema"; import { MuseToolChoiceCompatibilityError } from "../../adapters/openai-responses/muse-tool-choice"; +import { MuseToolCatalogCompatibilityError } from "../../adapters/openai-responses/muse-tool-catalog"; import { OpenGatewayInputCompatibilityError } from "../../adapters/openai-responses/opengateway-input"; import { formatErrorResponse } from "../../bridge"; import { redactSecretString } from "../../lib/redact"; @@ -389,6 +390,7 @@ export async function preparePassthroughExchange( || error instanceof XaiToolSchemaCompatibilityError || error instanceof RoutedCustomToolCompatError || error instanceof MuseToolChoiceCompatibilityError + || error instanceof MuseToolCatalogCompatibilityError || error instanceof OpenGatewayInputCompatibilityError ) { return formatErrorResponse(400, "invalid_request_error", redactSecretString(error.message)); diff --git a/structure/providers/opengateway.md b/structure/providers/opengateway.md index 12eeff8ec3c..6d87f19e0b9 100644 --- a/structure/providers/opengateway.md +++ b/structure/providers/opengateway.md @@ -39,3 +39,5 @@ keeps the existing stall-timeout policy. Rewritten responses preserve tool owner `tests/responses/opengateway-completion-receipts.test.ts` covers wire outcomes and resource release; `tests/responses/empty-completion-core.test.ts` and `tests/responses/empty-completion-hardening.test.ts` cover integration and replay barriers. Other destinations and translated retry defaults are unchanged. + +The separate [Responses include normalization](../transports/responses-wire-shapes.md) is gated to xAI and Meta; it does not change this projection or its destination boundary. Shared dispatch also maps Meta catalog validation failures to a client 400 before sending upstream. diff --git a/structure/providers/xai-grok.md b/structure/providers/xai-grok.md index 5a767bd9c90..c44f44c9a42 100644 --- a/structure/providers/xai-grok.md +++ b/structure/providers/xai-grok.md @@ -41,7 +41,7 @@ Shared parsing and streaming follow the [request-copy](../transports/byte-accoun ## Responses request compatibility -`src/providers/xai-transport.ts` recognizes the exact HTTPS hosts `api.x.ai`, `us.api.x.ai` and `cli-chat-proxy.grok.com` on the standard port as xAI Responses destinations. The Responses passthrough's `stripXaiUnsupportedInclude` removes only the rejected `output.encrypted_content` include entry, omits `include` when nothing remains, and preserves other entries and other destinations. `tests/providers/xai/xai-responses-adjacency.test.ts` covers the global, US regional and non-xAI request bodies; `tests/providers/xai/xai-transport.test.ts` rejects lookalikes, HTTP and nonstandard ports. +`src/providers/xai-transport.ts` recognizes the exact HTTPS hosts `api.x.ai`, `us.api.x.ai` and `cli-chat-proxy.grok.com` on the standard port as xAI Responses destinations. The Responses passthrough's `stripUnsupportedOutputInclude` removes only the rejected `output.encrypted_content` include entry, omits `include` when nothing remains, and preserves other entries. The same exact-entry normalization applies to Meta at `api.meta.ai`; unrelated destinations remain unchanged. `tests/providers/xai/xai-responses-adjacency.test.ts` covers the global, US regional and non-xAI request bodies; `tests/providers/xai/xai-transport.test.ts` rejects lookalikes, HTTP and nonstandard ports. ### Reasoning-model sampling parameters diff --git a/structure/transports/response-citations.md b/structure/transports/response-citations.md index b23f0e7e750..e1e7d7be55b 100644 --- a/structure/transports/response-citations.md +++ b/structure/transports/response-citations.md @@ -20,3 +20,5 @@ missing-title error; it does not validate source URLs, establish actual citation spans or prove provider acceptance beyond the exercised serialization fixtures. The existing destination gate owns eligibility, and the test layout explicitly registers the regression beside other Responses tests. + +The separate [Responses include normalization](responses-wire-shapes.md) is gated to xAI and Meta; it does not change this projection or its destination boundary. diff --git a/structure/transports/responses-citations.md b/structure/transports/responses-citations.md index 1e7f465a312..e88fdc4d562 100644 --- a/structure/transports/responses-citations.md +++ b/structure/transports/responses-citations.md @@ -17,3 +17,5 @@ gateway preservation and both refresh directions; `tests/responses/responses-native-item-extensions-compact.test.ts` covers official API and ChatGPT compact repair, while `tests/responses/responses-url-citation-title-repair.test.ts` covers ordinary HTTP/WS projection, helper independence and input immutability. + +The separate [Responses include normalization](responses-wire-shapes.md) is gated to xAI and Meta; it does not change this projection or its destination boundary. diff --git a/structure/transports/responses-wire-shapes.md b/structure/transports/responses-wire-shapes.md index 2a5808baac7..a6150ee190f 100644 --- a/structure/transports/responses-wire-shapes.md +++ b/structure/transports/responses-wire-shapes.md @@ -65,7 +65,7 @@ Google metadata follows the [destination-scoped replay projection](../providers/ xAI keeps `openai-chat` provider-wide; Grok 4.5/4.6/4.7 subscription Responses default to native `openai-responses`, retaining namespace, hosted-search and reasoning-replay normalization. The reserved `xai` OAuth transport is name-pinned to the Grok CLI gateway even if its saved base URL differs; custom provider IDs do not inherit this -default. Exact xAI Responses destinations, including the US regional API, omit only `output.encrypted_content` from `include`; other entries and non-xAI destinations remain unchanged ([request compatibility](../providers/xai-grok.md#responses-request-compatibility)). API-key requests, translated Chat/Anthropic defaults and other Grok models retain their +default. Exact xAI Responses destinations, including the US regional API, omit only `output.encrypted_content` from `include`; other entries remain unchanged. Meta at the exact `api.meta.ai` host shares this normalization; unrelated destinations retain their include list ([request compatibility](../providers/xai-grok.md#responses-request-compatibility)). API-key requests, translated Chat/Anthropic defaults and other Grok models retain their existing wire and tier policy. The OAuth lane is service-tier classified per model (`modelSupportsServiceTier` on the registry entry, live-probed 2026-09-13 and 2026-09-23; `devlog/_fin/260923_grok47_parity/010_probe-evidence.md` records 4.7): grok-4.7, grok-4.6, grok-4.5, @@ -221,7 +221,7 @@ the destination is the whole predicate, because Meta's refusal is a gateway sche Muse model it serves, its default `muse-spark-1.3` is not a Contributor id, and a missing model id still strips. Because the predicate is the host, a custom provider pointed at that exact URL gets the same strip. - +Meta's `api.meta.ai` route uses `src/adapters/openai-responses/muse-tool-catalog.ts` to capture current declarations before history rewrites or deduplication: current wrappers become `tools`, declarations retain their schemas over search discoveries, historical discovery stays replay data, and historical custom calls retain their conversion without live authority. Historical catalogs are not validated as current declarations. The reserved `functions` group and nested function definitions become public function declarations. Catalog validation also covers string/omitted input; unrepresentable namespace children and malformed current catalogs fail locally with HTTP 400 (`tests/responses/responses-muse-tool-choice.test.ts`). Direct Meta Muse / Meta Model Responses (`https://api.meta.ai/v1`) also rejects function tool names longer than 64 characters or containing characters outside `[a-zA-Z0-9_-]`. After namespace flattening, `src/responses/muse-tool-name-alias.ts` rewrites those identities on the `api.meta.ai` diff --git a/tests/providers/muse-spark-web-search-compat.test.ts b/tests/providers/muse-spark-web-search-compat.test.ts index f0e9f6c9b18..6b7a9d8319d 100644 --- a/tests/providers/muse-spark-web-search-compat.test.ts +++ b/tests/providers/muse-spark-web-search-compat.test.ts @@ -234,18 +234,19 @@ describe("#2617/#3378 Muse Spark web_search compatibility", () => { * same field drop, same preview preservation. */ test("direct Meta strips rejected web_search fields at both tool positions", () => { - const body = buildForProvider(META_PROVIDER, "muse-spark-1.3-contributor", { - tools: [webSearchTool()], - input: [{ type: "additional_tools", tools: [webSearchTool()] }], - }); - const tool = toolsOf(body)[0]!; - const item = (body.input as Array>)[0]!; - const nested = (item.tools as Array>)[0]!; - for (const declaration of [tool, nested]) { + for (const catalog of [ + { tools: [webSearchTool()] }, + { input: [{ type: "additional_tools", tools: [webSearchTool()] }] }, + ]) { + const body = buildForProvider(META_PROVIDER, "muse-spark-1.3-contributor", catalog); + // Meta receives either carrier in the public catalog after projection. + expect(toolsOf(body)).toHaveLength(1); + const declaration = toolsOf(body)[0]!; expect(declaration.type).toBe("web_search"); expect(declaration.search_context_size).toBe("medium"); expect(Object.hasOwn(declaration, "search_content_types")).toBe(false); expect(Object.hasOwn(declaration, "indexed_web_access")).toBe(false); + if (Array.isArray(body.input)) expect(body.input).toEqual([]); } }); diff --git a/tests/responses/responses-muse-tool-choice.test.ts b/tests/responses/responses-muse-tool-choice.test.ts index d5f7e95d6c1..277335ee1ef 100644 --- a/tests/responses/responses-muse-tool-choice.test.ts +++ b/tests/responses/responses-muse-tool-choice.test.ts @@ -20,9 +20,11 @@ afterEach(() => { releaseSpendHome = undefined; }); -function build(provider: OcxProviderConfig, rawBody: Record) { +function build(provider: OcxProviderConfig, rawBody: Record, compact = false, replayPrefixLength?: number) { return createAdapter(provider).buildRequest({ modelId: "muse-spark-1.3", + _compactionRequest: compact, + _replayPrefixLen: replayPrefixLength, context: { messages: [] }, stream: false, options: {}, @@ -193,3 +195,250 @@ describe("Meta Muse tool choice", () => { } }); }); + + +describe("Meta Muse Responses include compatibility", () => { + test("drops output encryption while preserving supported reasoning, other includes and caller input", () => { + for (const model of ["muse-spark-1.3", "muse-wonder-1.0", "muse-wonder-full-1.0"]) { + const raw = { model, include: ["reasoning.encrypted_content", "output.encrypted_content", "message.output_text.logprobs"], store: false }; + const original = structuredClone(raw); + const body = JSON.parse(build(meta, raw).body); + expect(body.include).toEqual(["reasoning.encrypted_content", "message.output_text.logprobs"]); + expect(body.model).toBe(model); + expect(body.store).toBe(false); + expect(raw).toEqual(original); + } + }); + + test("routed compaction preserves supported reasoning encryption", () => { + const raw = { include: ["reasoning.encrypted_content", "output.encrypted_content"], store: false }; + const body = JSON.parse(build(meta, raw, true).body); + expect(body.include).toEqual(["reasoning.encrypted_content"]); + expect(raw.include).toEqual(["reasoning.encrypted_content", "output.encrypted_content"]); + }); + + test("omits include when only unsupported output encryption was requested", () => { + expect(JSON.parse(build(meta, { include: ["output.encrypted_content"] }).body)).not.toHaveProperty("include"); + }); + + test("preserves unknown and malformed includes for upstream validation", () => { + for (const include of [undefined, null, "output.encrypted_content", { value: "output.encrypted_content" }, ["future.include"], [["output.encrypted_content"]]]) { + expect(JSON.parse(build(meta, { include }).body).include).toEqual(include); + } + }); + + test("does not infer Meta from a model name, path or lookalike hostname", () => { + for (const baseUrl of ["https://api.openai.com/v1", "https://api.meta.ai.example.test/v1", "https://example.test/api.meta.ai/v1"]) { + const include = ["reasoning.encrypted_content", "output.encrypted_content"]; + expect(JSON.parse(build({ ...meta, baseUrl }, { include }).body).include).toEqual(include); + } + }); +}); + + +describe("Meta Muse additional tool catalogs", () => { + const lookup = { type: "function", name: "lookup", parameters: { type: "object", properties: {} } }; + test("moves current namespace tools to the public catalog without changing messages or caller data", () => { + const message = { type: "message", role: "user", content: "continue" }; + const raw = { input: [{ type: "additional_tools", role: "developer", tools: [{ type: "namespace", name: "functions", tools: [lookup] }] }, message] }; + const original = structuredClone(raw); + const body = JSON.parse(build(meta, raw).body); + expect(body.input).toEqual([message]); + expect(body.tools).toEqual([lookup]); + expect(raw).toEqual(original); + }); + + test("removes replayed wrappers without authorizing their historical tools", () => { + const message = { type: "message", role: "user", content: "continue" }; + const raw = { tools: [], input: [{ type: "additional_tools", tools: [lookup] }, message] }; + const body = JSON.parse(build(meta, raw, false, 1).body); + expect(body.input).toEqual([message]); + expect(body.tools).toEqual([]); + }); + + test("ignores malformed historical catalogs while normalizing current declarations", () => { + for (const historical of [ + { type: "additional_tools" }, + { type: "additional_tools", tools: [{ type: "namespace", name: "old", tools: [{ type: "web_search" }] }] }, + ]) { + const definition = { name: "read", parameters: { type: "object", properties: {} } }; + const raw = { input: [historical, { type: "additional_tools", tools: [{ type: "function", function: definition }] }] }; + const original = structuredClone(raw); + const body = JSON.parse(build(meta, raw, false, 1).body); + expect(body.input).toEqual([]); + expect(body.tools).toEqual([{ type: "function", ...definition }]); + expect(raw).toEqual(original); + } + }); + + test("does not promote historical search discoveries or let removed history shift current declarations", () => { + const stale = { ...lookup, name: "stale" }; + const live = { ...lookup, name: "live" }; + const raw = { input: [ + { type: "configuration_update", config: {} }, + { type: "additional_tools", tools: [stale] }, + { type: "tool_search_output", call_id: "old_search", tools: [stale] }, + { type: "additional_tools", tools: [live] }, + { type: "tool_search_output", call_id: "new_search", tools: [{ ...lookup, name: "loaded" }] }, + ] }; + const body = JSON.parse(build(meta, raw, false, 3).body); + expect(body.tools.map((entry: { name: string }) => entry.name).sort()).toEqual(["live", "loaded"]); + expect(body.input.some((item: { type: string }) => item.type === "additional_tools")).toBe(false); + expect(JSON.stringify(body.input)).toContain("stale"); + }); + + test("a historical duplicate cannot consume the current declaration", () => { + const current = { ...lookup, description: "Current declaration" }; + const body = JSON.parse(build(meta, { input: [ + { type: "additional_tools", tools: [lookup] }, + { type: "additional_tools", tools: [current] }, + ] }, false, 1).body); + expect(body.tools).toEqual([current]); + }); + + test("rejects malformed catalogs before partial promotion", () => { + for (const raw of [ + { tools: null, input: [{ type: "additional_tools", tools: [lookup] }] }, + { input: [{ type: "additional_tools", tools: null }, { type: "additional_tools", tools: [lookup] }] }, + { input: [{ type: "additional_tools", tools: [lookup, { type: "namespace", name: "bad", tools: null }] }] }, + { input: [{ type: "additional_tools", tools: [lookup, null] }] }, + { input: [{ type: "tool_search_output", call_id: "s1", tools: [lookup, { type: "namespace", name: "bad", tools: null }] }] }, + ]) { + const original = structuredClone(raw); + expect(() => build(meta, raw)).toThrow("Malformed tool catalog for Meta Responses."); + expect(raw).toEqual(original); + } + }); + + test("current declarations keep their schema when search discovers the same tool", () => { + const declared = { ...lookup, defer_loading: true, parameters: { type: "object", properties: { project_id: { type: "string" } } } }; + const discovered = { ...lookup, parameters: { type: "object", properties: { admin_mode: { type: "boolean" } } } }; + for (const namespace of [undefined, "functions", "workspace"]) { + const catalog = namespace ? [{ type: "namespace", name: namespace, tools: [declared] }] : [declared]; + const discoveries = namespace === "workspace" ? [{ type: "namespace", name: namespace, tools: [discovered] }] : [discovered]; + for (const tools of [undefined, []]) { + const raw = { tools, input: [ + { type: "additional_tools", tools: catalog }, + { type: "tool_search_output", call_id: "s1", tools: discoveries }, + ] }; + const original = structuredClone(raw); + const body = JSON.parse(build(meta, raw).body); + expect(body.tools).toHaveLength(1); + expect(body.tools[0].parameters).toEqual(declared.parameters); + expect(body.tools[0]).not.toHaveProperty("defer_loading"); + expect(raw).toEqual(original); + } + } + }); + + test("unrepresentable namespace children fail without a partial catalog", () => { + for (const invalid of [ + { type: "namespace", name: "inner", tools: [lookup] }, + { ...lookup, name: "bad\u0000name" }, + { type: "web_search" }, + ]) { + const group = { type: "namespace", name: "outer", tools: [lookup, invalid] }; + for (const raw of [ + { tools: [group], input: [] }, + { input: [{ type: "additional_tools", tools: [group] }] }, + { input: [{ type: "tool_search_output", call_id: "s1", tools: [group] }] }, + ]) expect(() => build(meta, raw)).toThrow("Malformed tool catalog for Meta Responses."); + } + expect(() => build(meta, { tools: [{ type: "namespace", name: "bad\nname", tools: [lookup] }] })) + .toThrow("Malformed tool catalog for Meta Responses."); + }); + + test("explicit bare declarations win over reserved namespace duplicates in either carrier order", () => { + const bare = { ...lookup, parameters: { type: "object", properties: { project_id: { type: "string" } } } }; + const group = { type: "namespace", name: "functions", tools: [{ ...lookup, parameters: { type: "object", properties: { group_id: { type: "string" } } } }] }; + for (const raw of [ + { tools: [group], input: [{ type: "additional_tools", tools: [bare] }] }, + { tools: [bare], input: [{ type: "additional_tools", tools: [group] }] }, + { input: [{ type: "additional_tools", tools: [group] }, { type: "additional_tools", tools: [bare] }] }, + { input: [{ type: "additional_tools", tools: [bare] }, { type: "additional_tools", tools: [group] }] }, + { tools: [group, bare], input: "continue" }, + ]) { + const original = structuredClone(raw); + expect(JSON.parse(build(meta, raw).body).tools).toEqual([bare]); + expect(raw).toEqual(original); + } + }); + + test("validates top-level catalogs with string or omitted input", () => { + for (const input of ["continue", undefined]) { + expect(() => build(meta, { input, tools: [lookup, { type: "namespace", name: "bad", tools: null }] })) + .toThrow("Malformed tool catalog for Meta Responses."); + } + }); + + test("normalizes nested function definitions across input forms and catalog carriers", () => { + const definition = { name: "read", description: "Read a project", parameters: { type: "object", properties: { project_id: { type: "string" } } }, strict: true }; + const nested = { type: "function", function: definition }; + for (const raw of [ + { input: "continue", tools: [nested] }, + { input: undefined, tools: [nested] }, + { input: [], tools: [nested] }, + { input: [{ type: "additional_tools", tools: [nested] }] }, + { input: [{ type: "tool_search_output", call_id: "s1", tools: [nested] }] }, + ]) { + const original = structuredClone(raw); + expect(JSON.parse(build(meta, raw).body).tools).toEqual([{ type: "function", ...definition }]); + expect(raw).toEqual(original); + } + }); + + test("preserves historical custom-call conversion without declaring the old tool", () => { + for (const toolChoice of ["auto", "none"]) { + const raw = { tool_choice: toolChoice, input: [ + { type: "additional_tools", tools: [{ type: "custom", name: "old_exec", description: "Historical tool" }] }, + { type: "custom_tool_call", name: "old_exec", call_id: "c1", input: "historical input" }, + { type: "custom_tool_call_output", call_id: "c1", output: "historical output" }, + { type: "message", role: "user", content: "continue" }, + ] }; + const original = structuredClone(raw); + const request = build(meta, raw, false, 3); + const body = JSON.parse(request.body); + expect(body.tools).toEqual([]); + expect(body.input).toEqual([ + { type: "function_call", name: "old_exec", call_id: "c1", arguments: JSON.stringify({ input: "historical input" }) }, + { type: "function_call_output", call_id: "c1", output: "historical output" }, + original.input[3], + ]); + expect(request.convertedRoutedCustomToolNames?.has("old_exec")).toBe(false); + expect(raw).toEqual(original); + } + }); + + test("malformed nested catalogs return a client error without sending upstream", async () => { + const config = { port: 0, defaultProvider: "fixture", providers: { fixture: { ...meta, authMode: "key" } } } as OcxConfig; + const savedFetch = globalThis.fetch; + let sends = 0; + globalThis.fetch = (async () => { sends++; throw new Error("must not send"); }) as typeof fetch; + try { + releaseSpendHome ??= acquireOwnedSpendHome(); + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "fixture/muse-spark-1.3", input: [ + { type: "additional_tools", tools: [lookup, { type: "namespace", name: "bad", tools: null }] }, + { type: "message", role: "user", content: "continue" }, + ] }), + }), config, { model: "", provider: "" }); + expect(response.status).toBe(400); + expect(await response.json()).toMatchObject({ error: { type: "invalid_request_error", message: "Malformed tool catalog for Meta Responses." } }); + expect(sends).toBe(0); + } finally { globalThis.fetch = savedFetch; } + }); + + test("none clears declarations after promotion and long-name aliasing", () => { + const body = JSON.parse(build(meta, { input: [{ type: "additional_tools", tools: [tool] }], tool_choice: "none" }).body); + expect(body.tools).toEqual([]); + expect(body.input).toEqual([]); + }); + + test("leaves the private wrapper intact for unrelated destinations", () => { + const raw = { input: [{ type: "additional_tools", tools: [lookup] }] }; + for (const baseUrl of ["https://api.openai.com/v1", "https://api.meta.ai.example.test/v1"]) { + expect(JSON.parse(build({ ...meta, baseUrl }, raw).body).input).toEqual(raw.input); + } + }); +}); From ac0833bc55cce392d79d42fa2b952b2ccbe9d9b3 Mon Sep 17 00:00:00 2001 From: JUN Date: Fri, 30 Oct 2026 20:45:22 +0900 Subject: [PATCH 14/42] fix(windows): bigint probe identity in runtime preflight; release CC-Switch F3 WAL handles (#7986) * fix(windows): compare runtime preflight directory identity as bigint NTFS file IDs exceed 2^53, where adjacent IDs are equal as JS numbers. The owned-probe check could then mistake a replacement directory for its own probe and remove it, and the replacement-root test failed on windows 2/11 of run 41021347487 with ino 10696049116378316. * test(cc-switch): finalize the F3 WAL writer statements before cleanup Unfinalized prepare().run() statements keep the SQLite db, -wal and -shm open past close(), so Windows refused to remove the temp tree (EBUSY on windows 10/11 of run 41021347487). --------- Co-authored-by: JUN <243035832+lidge-jun@users.noreply.github.com> --- src/lib/bun-runtime-preflight.ts | 17 +++++++------ tests/lib/bun-runtime-preflight.test.ts | 32 +++++++++++++++++++++--- tests/providers/cc-switch-import.test.ts | 11 +++++--- 3 files changed, 46 insertions(+), 14 deletions(-) diff --git a/src/lib/bun-runtime-preflight.ts b/src/lib/bun-runtime-preflight.ts index 19b9ca3ca09..3f3e7f6a68b 100644 --- a/src/lib/bun-runtime-preflight.ts +++ b/src/lib/bun-runtime-preflight.ts @@ -19,9 +19,10 @@ export class RuntimePreflightError extends Error { } } -type DirectoryIdentity = { dev: number; ino: number; isDirectory(): boolean; isSymbolicLink(): boolean }; +// bigint: NTFS file IDs exceed 2^53, where adjacent IDs are equal as numbers. +type DirectoryIdentity = { dev: bigint; ino: bigint; isDirectory(): boolean; isSymbolicLink(): boolean }; export interface RuntimePreflightFs { - lstatSync: (path: string) => DirectoryIdentity; + lstatSync: (path: string, options: { bigint: true }) => DirectoryIdentity; mkdirSync: (path: string, mode: number) => unknown; rmdirSync: (path: string) => void; readdirSync: (path: string) => readonly string[]; @@ -44,7 +45,7 @@ export interface RuntimePreflightDeps { const probeFs: RuntimePreflightFs = { lstatSync, mkdirSync, rmdirSync, readdirSync }; function sameDirectory(fs: RuntimePreflightFs, path: string, owned: DirectoryIdentity): boolean { - const current = fs.lstatSync(path); + const current = fs.lstatSync(path, { bigint: true }); return current.isDirectory() && !current.isSymbolicLink() && current.dev === owned.dev && current.ino === owned.ino; } @@ -59,7 +60,7 @@ function removeOwnedEmptyDirectory(fs: RuntimePreflightFs, path: string, owned: function prepareRoot(fs: RuntimePreflightFs, root: string, rootWasAbsent?: boolean): void { try { if (rootWasAbsent === undefined) { - try { fs.lstatSync(root); rootWasAbsent = false; } + try { fs.lstatSync(root, { bigint: true }); rootWasAbsent = false; } catch (error) { if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; rootWasAbsent = true; @@ -73,7 +74,7 @@ function prepareRoot(fs: RuntimePreflightFs, root: string, rootWasAbsent?: boole // Use a concurrently created root, subject to the directory check below. } } - const current = fs.lstatSync(root); + const current = fs.lstatSync(root, { bigint: true }); if (!current.isDirectory() || current.isSymbolicLink()) throw new RuntimePreflightError("create"); } catch { throw new RuntimePreflightError("create"); @@ -83,7 +84,7 @@ function prepareRoot(fs: RuntimePreflightFs, root: string, rootWasAbsent?: boole function probeInProcess(fs: RuntimePreflightFs, path: string): void { let owned: DirectoryIdentity | undefined; try { - try { fs.mkdirSync(path, 0o700); owned = fs.lstatSync(path); } + try { fs.mkdirSync(path, 0o700); owned = fs.lstatSync(path, { bigint: true }); } catch { throw new RuntimePreflightError("create"); } try { fs.rmdirSync(path); } catch { throw new RuntimePreflightError("remove"); } @@ -96,10 +97,10 @@ function probeChild(runtime: DurableBunRuntime, path: string, nonce: string, spa // The child emits only this invocation's nonce and a fixed result, never paths/errors. const script = `const fs = require("node:fs"); const path = ${JSON.stringify(path)}; let result = "create", owned; -try { fs.mkdirSync(path, 0o700); owned = fs.lstatSync(path); result = "remove"; +try { fs.mkdirSync(path, 0o700); owned = fs.lstatSync(path, { bigint: true }); result = "remove"; fs.rmdirSync(path); result = "ok"; } catch { /* result already names the failed step; details never leave the child */ } finally { - if (owned) { try { const now = fs.lstatSync(path); + if (owned) { try { const now = fs.lstatSync(path, { bigint: true }); if (now.isDirectory() && !now.isSymbolicLink() && now.dev === owned.dev && now.ino === owned.ino && fs.readdirSync(path).length === 0) fs.rmdirSync(path); } catch { /* leave any probe entry that cannot be proven ours */ } } diff --git a/tests/lib/bun-runtime-preflight.test.ts b/tests/lib/bun-runtime-preflight.test.ts index 353302af64d..0c7dea985ff 100644 --- a/tests/lib/bun-runtime-preflight.test.ts +++ b/tests/lib/bun-runtime-preflight.test.ts @@ -171,18 +171,44 @@ describe("selected Bun write preflight", () => { test("a replacement empty root with a different inode survives refusal", () => { const parent = temporary(); const root = join(parent, "replaced"); - let originalIno = 0; + let originalIno = 0n; refusal(() => assertSelectedRuntimeWritable(runtime, root, { ...windows, rootWasAbsent: true, spawnSync: (() => { - originalIno = lstatSync(root).ino; + originalIno = lstatSync(root, { bigint: true }).ino; renameSync(root, join(parent, "old")); mkdirSync(root); return { status: 1, stdout: "nonce:create" }; }) as typeof spawnSync, }), "create"); - expect(lstatSync(root).ino).not.toBe(originalIno); + // NTFS file IDs exceed 2^53; as numbers, adjacent IDs compare equal. + expect(lstatSync(root, { bigint: true }).ino).not.toBe(originalIno); expect(readdirSync(root)).toEqual([]); }); + test("a replacement probe whose file ID differs only above 2^53 is never removed", () => { + const root = temporary(); + const probe = join(root, ".ocx-runtime-probe-nonce"); + let replaced = false; + let removals = 0; + // Adjacent IDs above 2^53 collapse to one Number: the owned probe and its replacement. + const lstat = ((path: string, options?: { bigint?: boolean }) => { + const real = lstatSync(path, { bigint: true }); + const ino = path === probe ? (replaced ? 2n ** 53n : 2n ** 53n + 1n) : real.ino; + return { + dev: options?.bigint ? real.dev : Number(real.dev), ino: options?.bigint ? ino : Number(ino), + isDirectory: () => real.isDirectory(), isSymbolicLink: () => real.isSymbolicLink(), + }; + }) as RuntimePreflightFs["lstatSync"]; + refusal(() => assertSelectedRuntimeWritable(standalone, root, { + ...windows, fs: { ...fs, lstatSync: lstat, rmdirSync: path => { + if (removals++ > 0) { rmdirSync(path); return; } + renameSync(path, join(root, "owned-probe")); mkdirSync(path); replaced = true; + throw new Error("denied once"); + } }, + }), "remove"); + expect(existsSync(probe)).toBe(true); + expect(removals).toBe(1); + }); + test("refusal never inspects emptiness or deletes the config root", () => { const root = join(temporary(), "root"); const cleanup: string[] = []; diff --git a/tests/providers/cc-switch-import.test.ts b/tests/providers/cc-switch-import.test.ts index 185f1a7ab85..106475de1bd 100644 --- a/tests/providers/cc-switch-import.test.ts +++ b/tests/providers/cc-switch-import.test.ts @@ -1006,7 +1006,10 @@ describe("CC-Switch keyless provider import", () => { try { writer.run("PRAGMA journal_mode = WAL;"); writer.run("CREATE TABLE providers (id TEXT NOT NULL, app_type TEXT NOT NULL, name TEXT NOT NULL, settings_config TEXT NOT NULL, sort_index INTEGER, PRIMARY KEY (id, app_type))"); - writer.prepare("INSERT INTO providers VALUES (?1, ?2, ?3, ?4, ?5)").run( + // writer.run finalizes its statement; an unfinalized prepare() keeps the db, -wal and -shm + // open past close(), and Windows then refuses to delete the temp tree (EBUSY). + writer.run( + "INSERT INTO providers VALUES (?1, ?2, ?3, ?4, ?5)", "existing-prov", "openai", "Existing", @@ -1017,10 +1020,12 @@ describe("CC-Switch keyless provider import", () => { // Disable auto-checkpoint so subsequent UPDATE and INSERT stay exclusively in live-wal.db-wal. writer.run("PRAGMA wal_autocheckpoint = 0;"); - writer.prepare("UPDATE providers SET settings_config = ?1 WHERE id = 'existing-prov'").run( + writer.run( + "UPDATE providers SET settings_config = ?1 WHERE id = 'existing-prov'", JSON.stringify({ baseUrl: "https://updated.acme.example/v1" }), ); - writer.prepare("INSERT INTO providers VALUES (?1, ?2, ?3, ?4, ?5)").run( + writer.run( + "INSERT INTO providers VALUES (?1, ?2, ?3, ?4, ?5)", "uncheckpointed-prov", "claude", "Uncheckpointed", From 6218a9a63ba1ae7807c0eb94bec2a184d89e42f2 Mon Sep 17 00:00:00 2001 From: JUN Date: Fri, 30 Oct 2026 20:46:37 +0900 Subject: [PATCH 15/42] fix(combo): skip Meta Muse targets for mandatory tool_choice (#7982) * fix(combo): skip Meta for mandatory tool choices * fix(combo): resolve effective wire before Meta eligibility --- docs-site/src/content/docs/guides/combos.md | 4 + .../responses/combo-required-tool-contract.ts | 27 ++++ src/server/responses/core-combo.ts | 30 +++- .../transports/responses-combo-boundaries.md | 7 + ...erver-combo-required-tool-contract.test.ts | 141 ++++++++++++++++++ 5 files changed, 208 insertions(+), 1 deletion(-) diff --git a/docs-site/src/content/docs/guides/combos.md b/docs-site/src/content/docs/guides/combos.md index 57dc9ccb196..a0b9de7c718 100644 --- a/docs-site/src/content/docs/guides/combos.md +++ b/docs-site/src/content/docs/guides/combos.md @@ -152,6 +152,10 @@ native routing and catalog identity on the next sync. ### Failover: ordered primary and backups +For Responses requests that require a tool, Meta Responses targets are skipped before sending +because Meta does not support mandatory `tool_choice`. An eligible later target receives the +original choice. If every target is incompatible for this reason, the combo reports the constraint. + `failover` selects the first eligible target in configuration order. A target is eligible when its provider exists, is enabled, is not cooling down, and can handle any special request constraint. Weights and `stickyLimit` do not affect this strategy. diff --git a/src/server/responses/combo-required-tool-contract.ts b/src/server/responses/combo-required-tool-contract.ts index 1e23e46873e..2f5e16a22d5 100644 --- a/src/server/responses/combo-required-tool-contract.ts +++ b/src/server/responses/combo-required-tool-contract.ts @@ -10,6 +10,33 @@ export type RequiredToolContract = | { kind: "any"; hostedCallTypes: ReadonlySet } | { kind: "named"; name: string }; +const allowedHostedSelectors = new Set([ + "web_search", "web_search_preview", "file_search", "computer_use_preview", "computer_use", + "code_interpreter", "image_generation", "image_gen", "tool_search", +]); + +/** Only definite, well-formed mandatory selectors justify excluding a destination. */ +export function hasMandatoryToolChoice(rawBody: unknown): boolean { + if (!rawBody || typeof rawBody !== "object" || Array.isArray(rawBody)) return false; + const choice = (rawBody as Record).tool_choice; + if (choice === "required") return true; + if (!choice || typeof choice !== "object" || Array.isArray(choice)) return false; + const selector = choice as Record; + if (selector.type === "function" || selector.type === "custom") { + return typeof selector.name === "string" && selector.name.trim().length > 0; + } + return selector.type === "allowed_tools" && selector.mode === "required" + && Array.isArray(selector.tools) && selector.tools.length > 0 + && selector.tools.every((tool: unknown) => { + if (!tool || typeof tool !== "object" || Array.isArray(tool)) return false; + const entry = tool as Record; + if (entry.name !== undefined && (typeof entry.name !== "string" || !entry.name.trim())) return false; + if (entry.type === "function" || entry.type === "custom") return typeof entry.name === "string"; + if (entry.type === "mcp") return typeof entry.server_label === "string" && !!entry.server_label.trim(); + return typeof entry.type === "string" && allowedHostedSelectors.has(entry.type); + }); +} + const hostedCalls = new Map([ ["web_search", "web_search_call"], ["web_search_preview", "web_search_call"], ["file_search", "file_search_call"], ["code_interpreter", "code_interpreter_call"], diff --git a/src/server/responses/core-combo.ts b/src/server/responses/core-combo.ts index 3b8d0fde10c..f2de582343d 100644 --- a/src/server/responses/core-combo.ts +++ b/src/server/responses/core-combo.ts @@ -1,4 +1,7 @@ import { effectiveAzureOutputBudget } from "../../adapters/azure"; +import { buildResponsesTransport } from "../../adapters/openai-responses/transport"; +import { isMetaAiResponsesDestination } from "../../responses/muse-tool-name-alias"; +import { hasMandatoryToolChoice } from "./combo-required-tool-contract"; import { isCodexAccountCooldownResponse } from "../../codex/account-cooldown-response"; import { resolveWireProtocolOverride } from "../adapter-resolve"; import { comboCooldownRetryAfterSeconds, isDefiniteContextOverflow, type CodexAccountModelRefusal } from "../../combos/failover"; @@ -56,7 +59,7 @@ import { previousResponseProviderState, } from "../../responses/state"; import { hasUnreadableEncryptedAgentTask } from "./encrypted-payload"; -import { routeConcreteModel, comboRouteDecisionTrace } from "../../router"; +import { captureRouteStaticPolicy, routeConcreteModel, comboRouteDecisionTrace } from "../../router"; import { memoryModelRouteReason } from "./memory-models"; import { baseProviderLabel, poolAccountProviderLabel } from "../../providers/label"; import { getAccountSet } from "../../oauth/store"; @@ -533,10 +536,29 @@ export async function executeComboResponses( const payloadEligible = (target: (typeof combo.targets)[number]): boolean => comboPayloadReadable || !unreadableEncryptedAgentTask || canDecryptUnreadableAgentTask(target); const meteredAllowed = admissionMayUseMeteredComboTargets(config, options.admission, combo); + const museToolEligible = (target: (typeof combo.targets)[number]): boolean => { + if (!hasMandatoryToolChoice(body)) return true; + try { + const route = routeConcreteModel(config, `${target.provider}/${target.model}`); + const inboundWire = options.inboundWire ?? "responses"; + const policy = captureRouteStaticPolicy(route.providerName, route.modelId, route.provider, + route.staticPolicy.effectiveAlias, inboundWire); + const provider = resolveWireProtocolOverride(route.providerName, route.modelId, route.provider, inboundWire, policy); + return provider.adapter !== "openai-responses" + || !isMetaAiResponsesDestination(buildResponsesTransport(provider, { headers: req.headers }).url); + } catch { + return true; + } + }; + const onlyMuseToolIncompatible = (): boolean => { + const enabled = combo.targets.filter(target => config.providers[target.provider]?.disabled !== true); + return enabled.length > 0 && enabled.every(target => !museToolEligible(target)); + }; // Recovery can replace unreadable encrypted input, but cannot relax other admission checks. const recoveryEligible = (target: (typeof combo.targets)[number]): boolean => (meteredAllowed || !target.metered) && (combo.strategy !== "jev" || target.provider !== JEV_PROVIDER_ID) + && museToolEligible(target) && reasoningReplayEligible(target) && (protocolLanes?.pickable(target) ?? true); const targetEligible = (target: (typeof combo.targets)[number]): boolean => @@ -651,6 +673,9 @@ export async function executeComboResponses( config, { parentThreadId: inboundClientThreadId }, ); + if (!options.abortSignal?.aborted && onlyMuseToolIncompatible()) { + return comboUnavailableResponse(`No available targets for combo: ${comboId}; Meta Responses does not support mandatory tool_choice`); + } return options.abortSignal?.aborted ? clientCancelledResponse() : comboUnavailable(comboId, undefined, @@ -668,6 +693,9 @@ export async function executeComboResponses( const protocolRefusal = protocolLanes?.refusal(); if (protocolRefusal) return protocolRefusal; if (onlyReplayIncompatibleTargetsRemain()) return targetIncompatibleResponse(); + if (!options.abortSignal?.aborted && onlyMuseToolIncompatible()) { + return comboUnavailableResponse(`No available targets for combo: ${comboId}; Meta Responses does not support mandatory tool_choice`); + } return options.abortSignal?.aborted ? clientCancelledResponse() : comboUnavailable(comboId, undefined, comboExclusionReasons(config, comboId, targetEligible)); diff --git a/structure/transports/responses-combo-boundaries.md b/structure/transports/responses-combo-boundaries.md index a3a4d8f0e57..1c057789d0a 100644 --- a/structure/transports/responses-combo-boundaries.md +++ b/structure/transports/responses-combo-boundaries.md @@ -20,6 +20,13 @@ and `tests/responses/responses-combo-exhausted-error.test.ts`. ## Buffered required-tool contract +Before selection, `src/server/responses/core-combo.ts` excludes resolved Meta Responses destinations +for well-formed mandatory `tool_choice` selectors, including required allowed-tools lists. Normal and +recovery picks share the check, resolving the inbound wire's static model policy and actual Responses +transport URL; malformed and unknown selectors retain their ordinary error path. The next target +receives the original selector, and an all-Meta exhaustion names the constraint before any send. +Coverage: `tests/server/server-combo-required-tool-contract.test.ts`. + Before combo success bookkeeping, native delivery in `src/server/responses/passthrough-delivery.ts` uses `src/server/responses/combo-required-tool-contract.ts` to check successful native `openai-responses` JSON children with explicit required or named function/custom choices. diff --git a/tests/server/server-combo-required-tool-contract.test.ts b/tests/server/server-combo-required-tool-contract.test.ts index 517aa17de62..f25491239a2 100644 --- a/tests/server/server-combo-required-tool-contract.test.ts +++ b/tests/server/server-combo-required-tool-contract.test.ts @@ -21,6 +21,7 @@ import type { OcxConfig } from "../../src/types"; import { item as hostedImage } from "../fixtures/hosted-image-display"; import { createTestTranslatorBudget } from "../helpers/translator-budget"; import { deliverPassthroughResponse } from "../../src/server/responses/passthrough-delivery"; +import { encryptedInput } from "../helpers/agent-task-recovery"; /** * #7456: a Responses combo child that answers a `tool_choice`-required request with a buffered @@ -164,6 +165,146 @@ function requiredRequest(extra: Record = {}): Request { } describe("combo required-tool contract on buffered children (#7456)", () => { + for (const choice of ["required", { type: "function", name: "shell" }, + { type: "allowed_tools", mode: "required", tools: [{ type: "function", name: "shell" }] }]) { + test(`Meta mandatory tool choice selects the next target before send (#7964): ${JSON.stringify(choice)}`, async () => { + const bodies: unknown[] = []; + let metaSends = 0; + const b = serve(async request => { + bodies.push(await request.json()); + return Response.json(completedWithCall("m2", "shell")); + }); + const response = await handleResponsesWith(comboConfig({ + a: provider("openai-responses", "https://api.meta.ai/v1", "key-a", { fetch: (async () => { metaSends++; throw new Error("unexpected Meta send"); }) as typeof fetch }), + b: provider("openai-responses", baseUrl(b), "key-b"), + }), requiredRequest({ tool_choice: choice })); + expect(response.status).toBe(200); + expect(bodies).toHaveLength(1); + expect(metaSends).toBe(0); + expect(bodies[0]).toMatchObject({ tool_choice: choice }); + }); + } + for (const choice of ["auto", "none"]) { + test(`Meta accepts ${choice} without a hop`, async () => { + let sends = 0; + const meta = provider("openai-responses", "https://api.meta.ai/v1", "key-a", { + fetch: (async () => { sends++; return Response.json(completedNoCall("m1", "meta")); }) as typeof fetch, + }); + const response = await handleResponsesWith(comboConfig({ a: meta }), requiredRequest({ tool_choice: choice })); + expect(response.status).toBe(200); + expect(sends).toBe(1); + }); + } + test("malformed choice remains a local Meta 400", async () => { + let sends = 0; + const meta = provider("openai-responses", "https://api.meta.ai/v1", "key-a", { + fetch: (async () => { sends++; throw new Error("unexpected send"); }) as typeof fetch, + }); + const response = await handleResponsesWith(comboConfig({ a: meta }), requiredRequest({ tool_choice: { type: "function" } })); + expect(response.status).toBe(400); + expect(sends).toBe(0); + }); + test("#7982 chat configuration with Responses override skips Meta", async () => { + const bodies: unknown[] = []; + const next = serve(async request => { bodies.push(await request.json()); return Response.json(completedWithCall("m2", "shell")); }); + const response = await handleResponsesWith(comboConfig({ + a: provider("openai-chat", "https://api.meta.ai/v1", "key-a", { modelAdapters: { m1: "openai-responses" } }), + b: provider("openai-responses", baseUrl(next), "key-b"), + }), requiredRequest({ tool_choice: "required" })); + expect(response.status).toBe(200); + expect(bodies).toHaveLength(1); + expect(bodies[0]).toMatchObject({ tool_choice: "required" }); + }); + test("#7982 Responses configuration with Chat override sends to Meta Chat", async () => { + let sends = 0; + const response = await handleResponsesWith(comboConfig({ + a: provider("openai-responses", "https://api.meta.ai/v1", "key-a", { + modelAdapters: { m1: "openai-chat" }, + fetch: (async () => { sends++; return Response.json({ id: "chat-1", object: "chat.completion", model: "m1", choices: [{ index: 0, finish_reason: "tool_calls", message: { role: "assistant", content: null, tool_calls: [{ id: "call_a", type: "function", function: { name: "shell", arguments: "{}" } }] } }] }); }) as typeof fetch, + }), + }), requiredRequest({ tool_choice: "required" })); + expect(response.status).toBe(200); + await response.text(); + expect(sends).toBe(1); + }); + test("#7982 unknown and malformed hosted selectors do not exclude Meta", async () => { + const { hasMandatoryToolChoice } = await import("../../src/server/responses/combo-required-tool-contract"); + for (const entry of [{ type: "not_a_tool" }, { type: "web_search", name: 5 }, { type: "mcp" }, { type: "mcp", server_label: 5 }]) { + const choice = { type: "allowed_tools", mode: "required", tools: [entry] }; + expect(hasMandatoryToolChoice({ tool_choice: choice })).toBe(false); + const response = await handleResponsesWith(comboConfig({ a: provider("openai-responses", "https://api.meta.ai/v1", "key-a") }), requiredRequest({ tool_choice: choice })); + expect(response.status).toBe(400); + } + for (const entry of [{ type: "web_search" }, { type: "mcp", server_label: "docs" }]) { + expect(hasMandatoryToolChoice({ tool_choice: { type: "allowed_tools", mode: "required", tools: [entry] } })).toBe(true); + } + }); + test("#7982 encrypted recovery selection cannot pick a Meta target", async () => { + const response = await handleResponsesWith(comboConfig({ a: provider("openai-responses", "https://api.meta.ai/v1", "key-a") }), + requiredRequest({ input: encryptedInput(), tool_choice: "required" })); + expect(response.status).toBe(503); + expect(await response.text()).toContain("mandatory tool_choice"); + }); + for (const [extra, expectedUrl] of [ + [{ responsesPath: "/other/responses" }, "https://api.meta.ai/other/responses"], + [{ authMode: "forward" as const, responsesPath: "/ignored" }, "https://api.meta.ai/responses"], + ] as const) { + test(`#7982 resolved Meta URL excludes mandatory choice: ${JSON.stringify(extra)}`, async () => { + const response = await handleResponsesWith(comboConfig({ a: provider("openai-responses", "https://api.meta.ai", "key-a", extra) }), requiredRequest({ tool_choice: "required" })); + expect(response.status).toBe(503); + expect(await response.text()).toContain("mandatory tool_choice"); + let sentUrl = ""; + const auto = await handleResponsesWith(comboConfig({ a: provider("openai-responses", "https://api.meta.ai", "key-a", { + ...extra, fetch: (async input => { sentUrl = String(input); return Response.json(completedNoCall("m1", "meta")); }) as typeof fetch, + }) }), requiredRequest({ tool_choice: "auto" })); + expect(auto.status).toBe(200); + expect(sentUrl).toBe(expectedUrl); + }); + } + test("selector eligibility recognizes custom choices and abstains on malformed allowed lists", async () => { + const { hasMandatoryToolChoice } = await import("../../src/server/responses/combo-required-tool-contract"); + expect(hasMandatoryToolChoice({ tool_choice: { type: "custom", name: "shell" } })).toBe(true); + for (const choice of [ + { type: "function", name: "" }, + { type: "allowed_tools", mode: "required", tools: [] }, + { type: "allowed_tools", mode: "required", tools: [{ type: "function" }] }, + { type: "allowed_tools", mode: "required", tools: [{ type: 5 }] }, + { type: "allowed_tools", mode: "auto", tools: [{ type: "function", name: "shell" }] }, + ]) expect(hasMandatoryToolChoice({ tool_choice: choice })).toBe(false); + }); + test("all Meta targets report the mandatory choice constraint", async () => { + const meta = provider("openai-responses", "https://api.meta.ai/v1", "key-a"); + const response = await handleResponsesWith(comboConfig({ a: meta, b: meta }), requiredRequest({ tool_choice: "required" })); + expect(response.status).toBe(503); + expect(await response.text()).toContain("mandatory tool_choice"); + }); + test("a non-Meta first target remains eligible for mandatory tools", async () => { + let sends = 0; + const upstream = serve(() => { sends++; return Response.json(completedWithCall("m1", "shell")); }); + const response = await handleResponsesWith(comboConfig({ + a: provider("openai-responses", baseUrl(upstream), "key-a"), + b: provider("openai-responses", "https://api.meta.ai/v1", "key-b"), + }), requiredRequest({ tool_choice: "required" })); + expect(response.status).toBe(200); + expect(sends).toBe(1); + }); + test("streaming mandatory choice skips Meta before the first send", async () => { + const bodies: Record[] = []; + const upstream = serve(async request => { + bodies.push(await request.json() as Record); + const payload = completedWithCall("m2", "shell"); + return new Response(`data: ${JSON.stringify({ type: "response.completed", response: payload })}\n\n`, + { headers: { "content-type": "text/event-stream" } }); + }); + const response = await handleResponsesWith(comboConfig({ + a: provider("openai-responses", "https://api.meta.ai/v1", "key-a"), + b: provider("openai-responses", baseUrl(upstream), "key-b"), + }), requiredRequest({ stream: true, tool_choice: "required" })); + expect(response.status).toBe(200); + await response.text(); + expect(bodies).toHaveLength(1); + expect(bodies[0]!.tool_choice).toBe("required"); + }); test("a folded-SSE contract failure settles reported usage without completion effects", async () => { let effects = 0; let settled: unknown; From d42b9b1b231a6a5b1dadde9262dfb63736823488 Mon Sep 17 00:00:00 2001 From: JUN Date: Fri, 30 Oct 2026 20:46:45 +0900 Subject: [PATCH 16/42] fix(sidecars): accept CR-delimited Responses events (#7989) Carries #7962. Co-authored-by: oocheol <86232509+oocheol@users.noreply.github.com> --- docs-site/src/content/docs/guides/sidecars.md | 2 +- .../src/content/docs/ko/guides/sidecars.md | 2 +- src/web-search/parse.ts | 3 +- structure/data-planes/search.md | 2 +- tests/web-search/web-search-eof.test.ts | 58 +++++++++++++++++++ 5 files changed, 63 insertions(+), 4 deletions(-) diff --git a/docs-site/src/content/docs/guides/sidecars.md b/docs-site/src/content/docs/guides/sidecars.md index 28b7955a74f..ed7f778add1 100644 --- a/docs-site/src/content/docs/guides/sidecars.md +++ b/docs-site/src/content/docs/guides/sidecars.md @@ -286,7 +286,7 @@ See the [Configuration reference](/reference/configuration/server/#sidecars) for Search responses with malformed optional fields retain valid answer text and citations. Invalid error-message values use a generic failure message. -The OpenAI search and vision sidecars read a final completion or failure event even when the server closes the body without a trailing newline. An unfinished response or one stopped by a size limit still reports an error. +The OpenAI search and vision sidecars accept CR, LF and CRLF line endings, including mixed endings and CRLF split across network chunks. They read a final completion or failure event even when the server closes the body without a trailing newline. An unfinished response or one stopped by a size limit still reports an error. When a search answer ends with a Sources list, an unmatched list item stays in the answer instead of being removed as a citation. A title followed by its URL is still consumed as one citation. Non-HTTP URI-like text inside inline code, such as `mode:fast`, stays in the answer; code-formatted HTTP(S) URLs are still citations. diff --git a/docs-site/src/content/docs/ko/guides/sidecars.md b/docs-site/src/content/docs/ko/guides/sidecars.md index 93701719528..1ad25a749b3 100644 --- a/docs-site/src/content/docs/ko/guides/sidecars.md +++ b/docs-site/src/content/docs/ko/guides/sidecars.md @@ -161,7 +161,7 @@ fingerprint 선례를 따르지만, 실제 계정과 작업량으로 충분히 s 검색 응답의 선택 필드 형식이 잘못되어도 유효한 답변과 출처는 유지합니다. 오류 메시지 값의 형식이 잘못되면 일반 실패 메시지를 표시합니다. -OpenAI 검색·비전 사이드카는 서버가 마지막 줄바꿈 없이 응답 본문을 닫아도 마지막 완료 또는 실패 이벤트를 읽습니다. 완료되지 않았거나 크기 제한으로 중단된 응답은 계속 오류로 처리합니다. +OpenAI 검색·비전 사이드카는 CR, LF, CRLF 줄바꿈과 이들이 섞인 응답을 처리하며, CRLF가 여러 네트워크 청크로 나뉘어도 이벤트를 읽습니다. 서버가 마지막 줄바꿈 없이 응답 본문을 닫아도 마지막 완료 또는 실패 이벤트를 읽습니다. 완료되지 않았거나 크기 제한으로 중단된 응답은 계속 오류로 처리합니다. 검색 답변의 Sources 목록에서 URL과 짝을 이루지 못한 글머리표는 출처로 삭제하지 않고 답변에 남깁니다. 제목 다음 줄에 해당 URL이 있으면 하나의 출처로 처리합니다. `mode:fast`처럼 인라인 코드 안의 비 HTTP URI 형태 텍스트는 답변에 남기며, 코드로 표시한 HTTP(S) URL은 출처로 처리합니다. diff --git a/src/web-search/parse.ts b/src/web-search/parse.ts index 699babde617..1420385b2a8 100644 --- a/src/web-search/parse.ts +++ b/src/web-search/parse.ts @@ -466,7 +466,8 @@ export async function parseSidecarSSE(response: Response): Promise { await expect(parseSidecarSSE(new Response(body))).rejects.toThrow("transport stopped"); }); }); + +describe("Responses sidecar SSE line endings", () => { + for (const [name, endings] of [ + ["LF", ["\n"]], + ["CRLF", ["\r\n"]], + ["CR", ["\r"]], + ["mixed", ["\r", "\n", "\r\n"]], + ] as const) { + test.each([["one chunk", false], ["single-byte chunks", true]] as const)(`${name} preserves deltas and citations (%s)`, async (_chunking, fragmented) => { + const lines = [ + ": heartbeat", + "event: response.output_text.delta", + 'data:{"type":"response.output_text.delta","delta":"검색 "}', + "", + 'data: {"type":"response.output_text.delta","delta":"😀"}', + "", + `data: ${JSON.stringify({ type: "response.output_text.annotation.added", annotation: { + type: "url_citation", url: "https://example.com/result", title: "Source", + } })}`, + "", + "data: [DONE]", + ]; + const wire = lines.map((line, index) => line + endings[index % endings.length]).join(""); + expect(await parseSidecarSSE(eofResponse(wire, fragmented))).toEqual({ + text: "검색 😀", sources: [{ url: "https://example.com/result", title: "Source" }], + }); + }); + } + + test("preserves a CR-delimited terminal failure and earlier partial text", async () => { + const wire = prefix.replaceAll("\n", "\r") + + 'data: {"type":"response.failed","error":{"message":"search stopped"}}\r\r'; + expect(await parseSidecarSSE(eofResponse(wire))).toEqual({ + text: "draft", sources: [], error: "search stopped", + }); + }); + + test("CR-delimited text without a terminal event remains incomplete", async () => { + expect(await parseSidecarSSE(eofResponse(prefix.replaceAll("\n", "\r")))).toEqual({ + text: "draft", sources: [], error: "sidecar stream ended before terminal event", + }); + }); + + test("a CR-delimited terminal event cancels without waiting for EOF", async () => { + let cancelled = false; + let sent = false; + const body = new ReadableStream({ + pull(controller) { + if (sent) throw new Error("read past terminal event"); + sent = true; + controller.enqueue(new TextEncoder().encode('data: {"type":"response.completed"}\r\r')); + }, + cancel() { cancelled = true; }, + }, { highWaterMark: 0 }); + expect(await parseSidecarSSE(new Response(body))).toEqual({ text: "", sources: [] }); + expect(cancelled).toBe(true); + }); +}); From d389699c96db753931a5f785f7a6a78c3ff94a1e Mon Sep 17 00:00:00 2001 From: JUN Date: Fri, 30 Oct 2026 20:59:19 +0900 Subject: [PATCH 17/42] fix(azure): omit unsupported Responses encrypted output include (#7988) Carries #7953. Closes #7952. Co-authored-by: x3M3x <98298256+x3M3x@users.noreply.github.com> --- .../src/content/docs/reference/adapters.md | 5 ++ src/adapters/openai-responses/passthrough.ts | 7 ++- .../openai-responses/request-strips.ts | 2 +- structure/transports/responses-wire-shapes.md | 2 +- tests/providers/azure-adapter.test.ts | 54 +++++++++++++++++++ 5 files changed, 66 insertions(+), 4 deletions(-) diff --git a/docs-site/src/content/docs/reference/adapters.md b/docs-site/src/content/docs/reference/adapters.md index 0545fbbd8e6..8d6ee982c97 100644 --- a/docs-site/src/content/docs/reference/adapters.md +++ b/docs-site/src/content/docs/reference/adapters.md @@ -823,6 +823,11 @@ appended to the id instead. **Targets:** **Azure OpenAI**. Wraps `openai-responses` (so also `passthrough: true`). **Auth:** `key` via the `api-key` header (not Bearer). +Azure requests omit the unsupported output.encrypted_content entry from include, +while retaining reasoning.encrypted_content and all other entries. This also applies +to openai-responses configurations targeting a resource under .openai.azure.com. +When no entries remain, include is omitted. + - Delegates request building to the Responses passthrough, validates that `baseUrl` contains no unresolved template placeholder, and replaces `Authorization` with `api-key`. The configured URL targets Azure's v1 Responses API directly, so the adapter does not append `api-version`. diff --git a/src/adapters/openai-responses/passthrough.ts b/src/adapters/openai-responses/passthrough.ts index 9cb4657c64f..b76549f571e 100644 --- a/src/adapters/openai-responses/passthrough.ts +++ b/src/adapters/openai-responses/passthrough.ts @@ -343,9 +343,12 @@ export function createResponsesPassthroughAdapter(provider: OcxProviderConfig): // OpenCode Go speaks the same dialect, which the final model id and URL identify. outBody = normalizeXaiResponsesWebSearch(outBody, provider, { modelId: parsed.modelId, responseUrl: url }); outBody = injectXaiResponsesXSearch(outBody, provider, parsed._replayPrefixLen); - // xAI and Meta answer 400 for `output.encrypted_content` instead of ignoring it; + // xAI, Meta and Azure answer 400 for `output.encrypted_content` instead of ignoring it; // Codex Desktop asks for it on every turn (#7878). - if (isXaiResponsesDestination(provider) || isMetaAiResponsesDestination(url)) { + // Azure also rejects this exact include entry, including on plain Responses routes. + const isAzureResponses = provider.adapter === "azure" || provider.adapter === "azure-openai" + || new URL(url).hostname.toLowerCase().endsWith(".openai.azure.com"); + if (isXaiResponsesDestination(provider) || isMetaAiResponsesDestination(url) || isAzureResponses) { outBody = stripUnsupportedOutputInclude(outBody); } // xAI and explicitly classified compatible gateways reject these OpenAI web_search diff --git a/src/adapters/openai-responses/request-strips.ts b/src/adapters/openai-responses/request-strips.ts index 82bf33520c3..366d1ec34bc 100644 --- a/src/adapters/openai-responses/request-strips.ts +++ b/src/adapters/openai-responses/request-strips.ts @@ -8,7 +8,7 @@ import { activateDeferredTool } from "./tool-schema"; import { stripOpenAiOnlyWebSearchFields } from "./web-search"; /** - * Codex requests output.encrypted_content, which xAI and Meta reject with HTTP 400. + * Codex requests output.encrypted_content, which xAI, Meta and Azure reject with HTTP 400. * The caller gates this on the destination. Preserve reasoning.encrypted_content * for stateless replay and leave unknown or malformed values for upstream validation. */ diff --git a/structure/transports/responses-wire-shapes.md b/structure/transports/responses-wire-shapes.md index a6150ee190f..a8b49f9dde0 100644 --- a/structure/transports/responses-wire-shapes.md +++ b/structure/transports/responses-wire-shapes.md @@ -65,7 +65,7 @@ Google metadata follows the [destination-scoped replay projection](../providers/ xAI keeps `openai-chat` provider-wide; Grok 4.5/4.6/4.7 subscription Responses default to native `openai-responses`, retaining namespace, hosted-search and reasoning-replay normalization. The reserved `xai` OAuth transport is name-pinned to the Grok CLI gateway even if its saved base URL differs; custom provider IDs do not inherit this -default. Exact xAI Responses destinations, including the US regional API, omit only `output.encrypted_content` from `include`; other entries remain unchanged. Meta at the exact `api.meta.ai` host shares this normalization; unrelated destinations retain their include list ([request compatibility](../providers/xai-grok.md#responses-request-compatibility)). API-key requests, translated Chat/Anthropic defaults and other Grok models retain their +default. Exact xAI Responses destinations, including the US regional API, omit only `output.encrypted_content` from `include`; other entries remain unchanged. Meta at the exact `api.meta.ai` host shares this normalization, as do Azure adapters and resource hosts under .openai.azure.com, including when configured with openai-responses; unrelated destinations retain their include list ([request compatibility](../providers/xai-grok.md#responses-request-compatibility)). API-key requests, translated Chat/Anthropic defaults and other Grok models retain their existing wire and tier policy. The OAuth lane is service-tier classified per model (`modelSupportsServiceTier` on the registry entry, live-probed 2026-09-13 and 2026-09-23; `devlog/_fin/260923_grok47_parity/010_probe-evidence.md` records 4.7): grok-4.7, grok-4.6, grok-4.5, diff --git a/tests/providers/azure-adapter.test.ts b/tests/providers/azure-adapter.test.ts index 8414870a102..ee76dd27b8a 100644 --- a/tests/providers/azure-adapter.test.ts +++ b/tests/providers/azure-adapter.test.ts @@ -214,3 +214,57 @@ describe("Azure OpenAI adapter hardening", () => { } }); }); + +describe("Azure Responses include compatibility", () => { + test("removes the rejected include through the Azure adapter without mutating the caller", async () => { + const raw = { + model: "gpt-5.5", input: [], stream: true, + include: ["reasoning.encrypted_content", "output.encrypted_content", "message.output_text.logprobs"], + }; + const original = structuredClone(raw); + const request = await createAzureAdapter(provider()).buildRequest({ ...parsed, _rawBody: raw }); + expect(JSON.parse(request.body).include).toEqual([ + "reasoning.encrypted_content", "message.output_text.logprobs", + ]); + expect(raw).toEqual(original); + }); + + test("removes an include containing only the rejected entry", async () => { + const request = await createAzureAdapter(provider()).buildRequest({ + ...parsed, _rawBody: { ...parsed._rawBody, include: ["output.encrypted_content"] }, + }); + expect(JSON.parse(request.body)).not.toHaveProperty("include"); + }); + + test("normalizes Azure resource URLs configured with the plain Responses adapter", async () => { + const { createResponsesPassthroughAdapter } = await import("../../src/adapters/openai-responses"); + const adapter = withTestTranslatorBudget(createResponsesPassthroughAdapter({ + adapter: "openai-responses", + baseUrl: "https://example.openai.azure.com/openai/v1", + authMode: "key", + apiKey: "azure-test-key", + })); + const request = await adapter.buildRequest({ + ...parsed, + _rawBody: { ...parsed._rawBody, include: ["reasoning.encrypted_content", "output.encrypted_content"] }, + }, { headers: new Headers() }); + expect(JSON.parse(request.body).include).toEqual(["reasoning.encrypted_content"]); + }); + + test("preserves includes on unrelated and lookalike destinations", async () => { + const { createResponsesPassthroughAdapter } = await import("../../src/adapters/openai-responses"); + for (const baseUrl of [ + "https://api.openai.com/v1", + "https://gateway.example.test/v1", + "https://example.openai.azure.com.example.test/openai/v1", + ]) { + const adapter = withTestTranslatorBudget(createResponsesPassthroughAdapter({ + adapter: "openai-responses", baseUrl, authMode: "key", apiKey: "test-key", + })); + const request = await adapter.buildRequest({ + ...parsed, _rawBody: { ...parsed._rawBody, include: ["output.encrypted_content"] }, + }, { headers: new Headers() }); + expect(JSON.parse(request.body).include).toEqual(["output.encrypted_content"]); + } + }); +}); From a2ae4825350813b41d92abf2231924ed8950ef63 Mon Sep 17 00:00:00 2001 From: Epinephrine Date: Fri, 30 Oct 2026 12:01:31 +0000 Subject: [PATCH 18/42] fix(codex): validate CLI and desktop launcher execution boundaries (#7876) * fix(codex): validate CLI and desktop launcher execution boundaries * docs(cli): describe internal command validation contract * fix(chatgpt): retain discovered bundle root at every launch * fix(cli): validate renamed app bundles before Codex launch Check the canonical enclosing app bundle for every ocx codex handoff, including PATH aliases and recorded shim backings. Verify nested layouts against their outer root while preserving ordinary non-bundle launches and the Desktop launcher's strict recorded-root path. Add portable CLI revocation, renamed-bundle and alias regressions and document the launch policy in the shared contract and CLI references. Co-authored-by: Epinephrine * fix(codex): retain lexical app bundle containment * test(codex): cover nested bundle aliases and canonical handoff * docs(cli): clarify internal helper argument checks * fix(codex): keep the install hint when the selected CLI disappears Pre-spawn verification now resolves the executable, so a target that vanished after selection surfaced as a raw ENOENT instead of the install hint the spawn path prints. --------- Co-authored-by: JUN <243035832+lidge-jun@users.noreply.github.com> --- .../content/docs/guides/chatgpt-desktop.md | 14 +- .../content/docs/ko/reference/cli/agents.md | 2 + .../src/content/docs/reference/cli/agents.md | 2 + .../content/docs/tr/guides/chatgpt-desktop.md | 15 +- scripts/test-layout/layout-clients.json | 1 + scripts/test-layout/layout-exec-boundary.json | 4 + .../app-server-shim/bundle-trust-policy.ts | 5 + src/chatgpt/app-server-shim/launcher.ts | 46 ++++- src/cli/chatgpt-command.ts | 2 +- src/cli/codex-launcher.ts | 28 ++- src/cli/codex-shim-autorestore.ts | 2 + src/cli/internal-command.ts | 31 +++- src/codex/exec-boundary.ts | 31 +++- structure/cli-management.md | 2 +- structure/clients/chatgpt-desktop.md | 45 ++++- .../desktop-app-server-exec-boundary.test.ts | 76 ++++++++ .../desktop-app-server-shim-launcher.test.ts | 21 ++- .../exec-cli-bundle-containment.test.ts | 148 ++++++++++++++++ .../exec-cli-renamed-app-bundle.test.ts | 76 ++++++++ .../exec-launcher-boundary.test.ts | 143 +++++++++++++++ .../exec-renamed-app-bundle.test.ts | 164 ++++++++++++++++++ tests/fixtures/test-layout-expected.json | 5 + 22 files changed, 812 insertions(+), 51 deletions(-) create mode 100644 tests/clients/desktop-app-server-exec-boundary.test.ts create mode 100644 tests/codex-integration/exec-cli-bundle-containment.test.ts create mode 100644 tests/codex-integration/exec-cli-renamed-app-bundle.test.ts create mode 100644 tests/codex-integration/exec-launcher-boundary.test.ts create mode 100644 tests/codex-integration/exec-renamed-app-bundle.test.ts diff --git a/docs-site/src/content/docs/guides/chatgpt-desktop.md b/docs-site/src/content/docs/guides/chatgpt-desktop.md index 458acdfbedf..8f195bfa829 100644 --- a/docs-site/src/content/docs/guides/chatgpt-desktop.md +++ b/docs-site/src/content/docs/guides/chatgpt-desktop.md @@ -104,10 +104,11 @@ restart transaction. ## Failure behavior and known limits -When the platform is not macOS, the OpenCodex runtime is missing, or the filter -self-test fails, the launcher runs the original binary with untouched stdout. A -missing bundled app-server binary is the exception: there is nothing to fall back -to, so the launcher exits with an error (see below). +The launcher verifies the executable before every start, including Desktop respawns. +If stdout filtering is unavailable or its self-test fails, the verified executable +runs with untouched stdout. If executable verification fails or the OpenCodex +validator is unavailable, the launcher exits with an error without starting the +app-server. A missing bundled app-server binary also prevents startup (see below). A filter that passes the self-test and then dies mid-session closes the pipe. What the bundled app-server does after that has not been verified; it may get SIGPIPE or a write error and be respawned by Desktop through the same launcher. @@ -117,8 +118,9 @@ remaining stream to raw bytes. The experiment depends on the bundled binary path, the app honoring `CODEX_CLI_PATH`, and current RPC field shapes. Updates may change these. A moved -or removed OpenCodex installation fails the launcher preflight and runs the -original binary. Run `ocx chatgpt launch` again after relocating the installation. +or removed OpenCodex installation can leave the launcher unable to verify the +executable, so it refuses startup. Run `ocx chatgpt launch` again after relocating +the installation, or `ocx chatgpt restore` to remove the override. If an app update moves or removes the bundled app-server binary itself, the launcher cannot start it: it prints a message naming `ocx chatgpt launch` and `ocx chatgpt restore` on stderr and exits, and Desktop cannot start its diff --git a/docs-site/src/content/docs/ko/reference/cli/agents.md b/docs-site/src/content/docs/ko/reference/cli/agents.md index e6186944fb4..102cff039a0 100644 --- a/docs-site/src/content/docs/ko/reference/cli/agents.md +++ b/docs-site/src/content/docs/ko/reference/cli/agents.md @@ -193,6 +193,8 @@ family는 `opus`, `fable`, `sonnet`, `haiku`이며, 새 route는 `opus`에서 일치가 확인된(`matched`) autostart wrapper만 backing Codex를 직접 실행합니다. 검사 예외를 포함한 알 수 없는 상태는 검증된 bypass가 필요하며, 관리되는 Windows CMD wrapper는 현재 생성 템플릿 전체와 일치해야 `OCX_SHIM_BYPASS=1`로 실행합니다. 일반 npm 런처는 지원합니다. 검증할 수 없거나 오래된 wrapper는 종료 코드 1로 거부하며 Codex를 시작하지 않습니다. `ocx codex-shim install`로 갱신하세요. 이 실행은 shim을 수리하지 않으며, 내용 검사는 Windows backing 파일 출처나 동시 경로 교체를 보장하지 않습니다. +macOS에서는 도움말과 버전을 포함한 매 실행 시, 선택한 실행 파일의 경로나 정규 경로가 `.app` 번들 내부에 있으면 검증합니다. 이름을 바꾼 번들과 번들 내부를 가리키는 심볼릭 링크도 번들 및 실행 파일의 안전한 소유권, 권한, OpenAI 서명을 요구합니다. 선택한 번들 구성원은 실행 파일, 중간 디렉터리 또는 번들 전체의 심볼릭 링크를 거쳐도 해당 번들의 정규 `Contents/` 내부에 있어야 합니다. 검증 실패 시 Codex를 시작하지 않고 종료 코드 1을 반환합니다. npm 설치를 포함한 일반 번들 외부 실행 파일의 동작은 유지됩니다. + 런처 도움말은 `ocx help codex`, Codex 도움말은 `ocx codex --help`입니다. 이후 plain `codex`의 동작은 기존과 같습니다. ### `ocx opencode [opencode args...]` diff --git a/docs-site/src/content/docs/reference/cli/agents.md b/docs-site/src/content/docs/reference/cli/agents.md index fa0798d1e3d..bce042afced 100644 --- a/docs-site/src/content/docs/reference/cli/agents.md +++ b/docs-site/src/content/docs/reference/cli/agents.md @@ -994,6 +994,8 @@ A missing `OPENCODEX_API_AUTH_TOKEN` is read from this installation's `service-a Only a matched autostart wrapper launches its backing Codex directly. Unknown state, including an inspection exception, requires a verified bypass; Windows CMD managed wrappers must match the complete current generated template before the selected absolute command runs with `OCX_SHIM_BYPASS=1`. Ordinary npm launchers remain supported. Unverifiable or obsolete wrappers refuse with exit 1 and start no Codex; run `ocx codex-shim install` to refresh them. This launch does not repair shims. The content check does not establish Windows backing-file provenance or prevent concurrent pathname replacement. +On macOS, every launch, including help/version, verifies the selected executable when its selected path or canonical location is inside any `.app` bundle. Renamed bundles and symlinks into bundles require the same safe ownership, permissions and OpenAI signatures for the bundle and executable. A selected bundle member must stay inside that bundle's canonical `Contents/`, including through executable, directory or whole-bundle symlinks. Failed verification exits 1 without starting Codex. Ordinary non-bundle executables, including npm installations, retain their behavior. + Use `ocx help codex` for launcher help and `ocx codex --help` for Codex help. Launching plain `codex` later retains its existing behavior. ### `ocx opencode [opencode args...]` diff --git a/docs-site/src/content/docs/tr/guides/chatgpt-desktop.md b/docs-site/src/content/docs/tr/guides/chatgpt-desktop.md index 847b14d27f0..b3dd5372fc3 100644 --- a/docs-site/src/content/docs/tr/guides/chatgpt-desktop.md +++ b/docs-site/src/content/docs/tr/guides/chatgpt-desktop.md @@ -101,10 +101,12 @@ etkisiz bir yeniden başlatma işlemi garanti etmez. ## Hata davranışı ve bilinen sınırlar -Platform macOS değilse, OpenCodex çalışma zamanı eksikse veya filtre öz sınaması -başarısızsa başlatıcı özgün dosyayı stdout'u değiştirmeden çalıştırır. Paketlenmiş -app-server dosyasının eksikliği istisnadır: geri dönülecek dosya olmadığı için -başlatıcı hatayla çıkar. +Başlatıcı, Desktop'ın yeniden başlattığı işlemler dahil her çalıştırmadan önce +yürütülebilir dosyayı doğrular. Stdout filtresi kullanılamıyorsa veya öz sınaması +başarısızsa doğrulanmış dosya stdout değiştirilmeden çalışır. Dosya doğrulaması +başarısızsa ya da OpenCodex doğrulayıcısı kullanılamıyorsa başlatıcı app-server'ı +çalıştırmadan hatayla çıkar. Paketlenmiş app-server dosyasının eksikliği de +başlatmayı engeller. Öz sınamayı geçen filtre oturum ortasında çökerse boru kapanır. Paketlenmiş app-server'ın bundan sonra ne yaptığı doğrulanmamıştır; SIGPIPE/yazma hatası @@ -115,8 +117,9 @@ ham baytlara geçirir. Deney; paketlenmiş dosya yoluna, uygulamanın `CODEX_CLI_PATH` değerini kullanmasına ve güncel RPC alan biçimlerine bağlıdır. Güncellemeler bunları değiştirebilir. -OpenCodex taşınır veya kaldırılırsa başlatıcının ön kontrolü başarısız olur ve -özgün dosya çalışır. Kurulumu taşıdıktan sonra `ocx chatgpt launch` komutunu yeniden çalıştırın. +OpenCodex taşınır veya kaldırılırsa başlatıcı dosyayı doğrulayamayabilir ve +başlatmayı reddeder. Kurulumu taşıdıktan sonra `ocx chatgpt launch` komutunu yeniden +çalıştırın veya geçersiz kılmayı kaldırmak için `ocx chatgpt restore` kullanın. Uygulama güncellemesi paketlenmiş app-server dosyasını taşır veya kaldırırsa başlatıcı onu çalıştıramaz: stderr üzerinde `ocx chatgpt launch` ve diff --git a/scripts/test-layout/layout-clients.json b/scripts/test-layout/layout-clients.json index fa50b9af986..01a3abb4020 100644 --- a/scripts/test-layout/layout-clients.json +++ b/scripts/test-layout/layout-clients.json @@ -32,6 +32,7 @@ "droid-normalized-reasoning-defaults.test.ts": "clients", "desktop-app-server-shim.test.ts": "clients", "desktop-app-server-shim-launcher.test.ts": "clients", + "desktop-app-server-exec-boundary.test.ts": "clients", "desktop-bundle-trust-async.test.ts": "clients", "desktop-chatgpt-config.test.ts": "clients", "desktop-supervised-restart.test.ts": "clients", diff --git a/scripts/test-layout/layout-exec-boundary.json b/scripts/test-layout/layout-exec-boundary.json index e64ba5579f4..7ca28c34045 100644 --- a/scripts/test-layout/layout-exec-boundary.json +++ b/scripts/test-layout/layout-exec-boundary.json @@ -1,5 +1,9 @@ { "exec-trust-boundary.test.ts": "codex-integration", "exec-boundary-launch.test.ts": "codex-integration", + "exec-launcher-boundary.test.ts": "codex-integration", + "exec-cli-renamed-app-bundle.test.ts": "codex-integration", + "exec-cli-bundle-containment.test.ts": "codex-integration", + "exec-renamed-app-bundle.test.ts": "codex-integration", "exec-known-app-bundles.test.ts": "codex-integration" } diff --git a/src/chatgpt/app-server-shim/bundle-trust-policy.ts b/src/chatgpt/app-server-shim/bundle-trust-policy.ts index 4bb2c1b93a6..b105b1b28b1 100644 --- a/src/chatgpt/app-server-shim/bundle-trust-policy.ts +++ b/src/chatgpt/app-server-shim/bundle-trust-policy.ts @@ -28,6 +28,11 @@ export function bundleTrustStat(path: string): ReturnType; +/** The inline bundle root is code: protect its file and directory entry like the bundle. */ +export function untrustedLauncherFilesystemReason(path: string, deps: FilesystemTrustDeps, existing = true): string | null { + return (existing ? ownedSafely(deps, path, "file") : null) ?? untrustedAncestorReason(path, deps); +} + function ownedSafely(deps: FilesystemTrustDeps, path: string, kind: "file" | "directory"): string | null { const s = deps.stat(path); if (!s) return "is missing"; diff --git a/src/chatgpt/app-server-shim/launcher.ts b/src/chatgpt/app-server-shim/launcher.ts index 53142ddafbf..6cddfd97c0f 100644 --- a/src/chatgpt/app-server-shim/launcher.ts +++ b/src/chatgpt/app-server-shim/launcher.ts @@ -3,6 +3,8 @@ import { randomBytes } from "node:crypto"; import { join, posix } from "node:path"; import { getConfigDir } from "../../config/paths"; import { selfLaunchArgv } from "../../lib/self-launch-argv"; +import { defaultBundleTrustDeps } from "./bundle-trust"; +import { untrustedLauncherFilesystemReason, type BundleTrustDeps } from "./bundle-trust-policy"; export const CHATGPT_APP_CODEX_BINARY = "/Applications/ChatGPT.app/Contents/Resources/codex-cli/CodexCLI.app/Contents/MacOS/codex"; @@ -47,24 +49,41 @@ function shellQuote(value: string): string { } /** - * A failed precondition or a failed self-test runs the original binary with untouched stdout. + * A failed filter self-test runs the verified binary with untouched stdout. + * Executable verification is required on every invocation, including passthrough and respawns. * A filter that passes the self-test and then dies mid-session closes the pipe. * Expected (not yet validated against the bundled app-server): the server gets SIGPIPE or a * write error and Desktop respawns it through the same launcher. */ -export function buildChatgptShimLauncher(argv: readonly string[], real = CHATGPT_APP_CODEX_BINARY): string { +export function buildChatgptShimLauncher( + argv: readonly string[], + real = CHATGPT_APP_CODEX_BINARY, + validator: readonly string[] = [process.execPath, ...selfLaunchArgv(["internal", "codex-exec-target"])], + bundleRoot = /^(.*?\.app)\/Contents\//i.exec(real)?.[1], +): string { return `#!/bin/bash # opencodex (experimental): ChatGPT app-server stdout passes through the quota-gate filter. REAL=${shellQuote(real)} -FILTER=(${argv.map(shellQuote).join(" ")}) +${bundleRoot === undefined ? "" : `BUNDLE_ROOT=${shellQuote(bundleRoot)}\n`}FILTER=(${argv.map(shellQuote).join(" ")}) +VALIDATE=(${validator.map(shellQuote).join(" ")}) if [ ! -x "$REAL" ]; then echo "opencodex (experimental): the bundled app-server is no longer at $REAL; run 'ocx chatgpt launch' again or 'ocx chatgpt restore'." >&2 exit 127 fi +validate_real() { + local target + if [ ! -x "\${VALIDATE[0]}" ] || ! target=$("\${VALIDATE[@]}" ${bundleRoot === undefined ? "" : '--bundle-root "$BUNDLE_ROOT" '}-- "$REAL") || [ -z "$target" ]; then + echo "opencodex: app-server execution verification failed; run 'ocx chatgpt launch' again or 'ocx chatgpt restore'." >&2 + return 126 + fi + REAL="$target" +} if [ "$(uname -s)" = "Darwin" ] && [ -x "\${FILTER[0]}" ] \\ && "\${FILTER[@]}" --self-test >/dev/null 2>&1; then + validate_real || exit 126 exec "$REAL" "$@" > >(exec "\${FILTER[@]}") fi +validate_real || exit 126 exec "$REAL" "$@" `; } @@ -74,17 +93,32 @@ exec "$REAL" "$@" * launcher, so an existing symbolic link is replaced rather than followed and a respawn during * the write never runs a half-written script. */ -export function writeChatgptShimLauncher(configDir = getConfigDir(), real = CHATGPT_APP_CODEX_BINARY): string { +export function writeChatgptShimLauncher( + configDir = getConfigDir(), real = CHATGPT_APP_CODEX_BINARY, + bundleRoot = /^(.*?\.app)\/Contents\//i.exec(real)?.[1], + filesystem: Pick | undefined = process.platform === "darwin" ? defaultBundleTrustDeps() : undefined, +): string { + if (!bundleRoot || !posix.isAbsolute(bundleRoot) || !real.startsWith(bundleRoot + "/Contents/")) { + throw new Error("the app-server launcher requires its discovered bundle root"); + } mkdirSync(configDir, { recursive: true }); const path = chatgptShimLauncherPath(configDir); const argv = [process.execPath, ...selfLaunchArgv(["internal", "chatgpt-app-server-filter"])]; const temp = `${path}.${process.pid}.${randomBytes(6).toString("hex")}.tmp`; + let created = false; + const assertFilesystem = (file: string, existing: boolean) => { + const reason = filesystem && untrustedLauncherFilesystemReason(file, filesystem, existing); + if (reason) throw new Error(`unsafe app-server launcher: ${reason}`); + }; try { - writeFileSync(temp, buildChatgptShimLauncher(argv, real), { mode: 0o755, flag: "wx" }); + assertFilesystem(temp, false); + writeFileSync(temp, buildChatgptShimLauncher(argv, real, undefined, bundleRoot), { mode: 0o755, flag: "wx" }); + created = true; chmodSync(temp, 0o755); + assertFilesystem(temp, true); renameSync(temp, path); } catch (error) { - try { unlinkSync(temp); } catch { /* the temp file was never created */ } + if (created) { try { unlinkSync(temp); } catch { /* preserve the publication error */ } } throw error; } return path; diff --git a/src/cli/chatgpt-command.ts b/src/cli/chatgpt-command.ts index 3006f44638c..0436b3033b1 100644 --- a/src/cli/chatgpt-command.ts +++ b/src/cli/chatgpt-command.ts @@ -164,7 +164,7 @@ CODEX_CLI_PATH launcher: ${app.shim === null ? "unknown" : app.shim ? "yes" : "n console.error("ChatGPT process inspection failed; no launcher was changed and no app was opened. Retry after resolving inspection."); return 1; } - if (binary) writeChatgptShimLauncher(undefined, binary); + if (binary) writeChatgptShimLauncher(undefined, binary, install.root); if (!(await quitApp(install, launcher))) { console.error("ChatGPT exit could not be confirmed; no new app was opened. Check the running app and retry."); return 1; diff --git a/src/cli/codex-launcher.ts b/src/cli/codex-launcher.ts index 98fc9f103ea..d8fce9489db 100644 --- a/src/cli/codex-launcher.ts +++ b/src/cli/codex-launcher.ts @@ -25,13 +25,13 @@ import { spawn } from "node:child_process"; import { loadConfig } from "../config"; import { getCodexRoutingKind, type CodexRoutingKind } from "../codex/inject"; import { inspectCodexShimBackingForCommand } from "../codex/shim"; +import { validatedCodexExecInvocationAsync, type CodexExecBoundaryAsyncDeps } from "../codex/exec-boundary"; import { CODEX_GLOBAL_OPTIONS_WITH_VALUE, CODEX_INTERNAL_COMMANDS, CODEX_SHIM_ENSURE_FAILED_DIAGNOSTIC } from "../codex/shim-templates"; import { withoutSiblingMarker } from "../codex/sibling-start"; import { withProcessRuntimeProvenance } from "../lib/bun-runtime"; import { tokenCollidesWithAdmin } from "../lib/admin-secrets"; import { selfLaunchArgv } from "../lib/self-launch-argv"; import { loadServiceTokenFromFile, serviceApiTokenFilePath } from "../lib/service-secrets"; -import { commandInvocation } from "../lib/win-exec"; import { findExecutableOnPath } from "../remote-control/workspace-executable"; import { findLiveProxy, type LiveProxy } from "../server/proxy-liveness"; import { codexCredentialDestinationIsTrusted, resolveCodexLaunchHome } from "./codex-launch-auth"; @@ -265,14 +265,26 @@ export function resolveCodexLaunchCommand( } } -/** Launch the selected Codex command with inherited IO and preserve its exit status. */ -function spawnCodex(args: readonly string[], env: CodexLaunchEnv): Promise { +export interface CodexSpawnDeps extends CodexLaunchCommandDeps { + execBoundary?: CodexExecBoundaryAsyncDeps; + spawn?: typeof spawn; +} + +/** Validate the final backing command immediately before each launch, including fast paths. */ +export async function spawnCodex(args: readonly string[], env: CodexLaunchEnv, deps: CodexSpawnDeps = {}): Promise { + let launch: CodexLaunchCommand, inv: Awaited>; + try { + launch = resolveCodexLaunchCommand(env, deps); + inv = await validatedCodexExecInvocationAsync(launch.command, args, { ...deps.execBoundary, verifyAnyAppBundle: true, env: launch.env as NodeJS.ProcessEnv }); + } catch (error) { + // A selected executable that vanished before verification is the same missing CLI + // the spawn-time ENOENT branch below reports, not a trust refusal. + const missing = error instanceof CodexNotFoundError || (error as NodeJS.ErrnoException | null)?.code === "ENOENT"; + console.error(missing ? CODEX_INSTALL_HINT : `❌ Cannot launch codex: ${error instanceof Error ? error.message : "executable verification failed"}`); + return 1; + } return new Promise(resolve => { - let launch: CodexLaunchCommand; - try { launch = resolveCodexLaunchCommand(env); } - catch (error) { console.error(error instanceof CodexNotFoundError ? CODEX_INSTALL_HINT : CODEX_UNVERIFIED_SHIM); resolve(1); return; } - const inv = commandInvocation(launch.command, args); - const child = spawn(inv.file, inv.args, { stdio: "inherit", env: launch.env as NodeJS.ProcessEnv, ...inv.options }); + const child = (deps.spawn ?? spawn)(inv.file, inv.args, { stdio: "inherit", env: launch.env as NodeJS.ProcessEnv, ...inv.options }); child.on("error", (err: NodeJS.ErrnoException) => { if (err.code === "ENOENT") { console.error(CODEX_INSTALL_HINT); diff --git a/src/cli/codex-shim-autorestore.ts b/src/cli/codex-shim-autorestore.ts index 86750714f3b..626bff50da9 100644 --- a/src/cli/codex-shim-autorestore.ts +++ b/src/cli/codex-shim-autorestore.ts @@ -21,6 +21,8 @@ export function skipsCodexShimAutoRestore(command: string | undefined, args: str if (command === "status" || command === "doctor") return true; // The launcher owns shim handoff after classifying routing, including native fast paths. if (command === "codex") return true; + // Exec admission only observes the selected target; it must never repair a shim first. + if (command === "internal" && args[1] === "codex-exec-target") return true; // `lab` is read-only inspection; it must not trigger shim side effects. if (command === "lab") return true; // Local messaging owns its helpers; even malformed usage must not repair/start anything. diff --git a/src/cli/internal-command.ts b/src/cli/internal-command.ts index 7799ceb52b2..d9d7c2ae9c7 100644 --- a/src/cli/internal-command.ts +++ b/src/cli/internal-command.ts @@ -16,15 +16,42 @@ * that same user, and a same-uid process could call `kill` directly. A token here would * protect nothing and would imply a boundary that does not exist. */ -const USAGE = "Usage: ocx internal desktop-restart-handoff --plan | chatgpt-app-server-filter [--self-test]"; +import type { CodexExecBoundaryAsyncDeps } from "../codex/exec-boundary"; + +const USAGE = "Usage: ocx internal desktop-restart-handoff --plan | chatgpt-app-server-filter [--self-test] | codex-exec-target [--bundle-root ] -- "; function optionValue(args: readonly string[], name: string): string | undefined { const index = args.indexOf(name); return index >= 0 ? args[index + 1] : undefined; } -export async function handleInternalCommand(args: readonly string[]): Promise { +/** + * Dispatch internal helpers with command-specific argument checks and exit codes. + * The executable-target helper writes one trusted path to stdout only after validation. + */ +export async function handleInternalCommand(args: readonly string[], execBoundary: CodexExecBoundaryAsyncDeps = {}): Promise { const sub = args[0]; + if (sub === "codex-exec-target") { + const bundled = args[1] === "--bundle-root"; + const root = bundled ? args[2] : undefined; + const separator = bundled ? 3 : 1; + const command = args[separator + 1]; + if (args.length !== separator + 2 || args[separator] !== "--" || !command || /[\r\n\0]/.test(command) + || (bundled && (!root || !root.startsWith("/") || /[\r\n\0]/.test(root)))) { + console.error(USAGE); + return 2; + } + try { + const { assertCodexExecTargetTrustedAsync } = await import("../codex/exec-boundary"); + const target = await assertCodexExecTargetTrustedAsync(command, bundled ? { ...execBoundary, expectedBundleRoot: root } : execBoundary); + if (!target || /[\r\n]/.test(target)) throw new Error("invalid executable path"); + console.log(target); + return 0; + } catch (error) { + console.error(`OpenCodex refused the app-server executable: ${error instanceof Error ? error.message : "verification failed"}`); + return 1; + } + } if (sub === "chatgpt-app-server-filter") { if (args.length > 2 || (args[1] !== undefined && args[1] !== "--self-test")) { console.error(USAGE); diff --git a/src/codex/exec-boundary.ts b/src/codex/exec-boundary.ts index 26f869fdf27..9db1b9291e8 100644 --- a/src/codex/exec-boundary.ts +++ b/src/codex/exec-boundary.ts @@ -20,8 +20,9 @@ import { * known-Codex-bundle classifier, and — when the lexical or canonical path sits * inside a ChatGPT.app or Codex.app bundle at any location — checked by the shared * signature/ownership policy before the same canonical target is returned for - * execution. Anything else, including binaries inside third-party app bundles, - * is passed through to the shared launcher unchanged. + * execution. CLI handoffs additionally classify any enclosing lexical or canonical .app, + * independent of bundle name or identifier. Other callers retain the known-app + * policy; non-bundle executables pass through to the shared launcher unchanged. * * The check is one-shot per invocation and deliberately not memoized: * directory-service group membership, signature state, and directory-entry @@ -53,6 +54,10 @@ export interface CodexExecBoundaryDeps { executable?: (path: string) => boolean; /** Canonicalizer; returns null when the path cannot be resolved. */ realpath?: (path: string) => string | null; + /** CLI handoffs verify any enclosing app, without trusting mutable bundle identity metadata. */ + verifyAnyAppBundle?: boolean; + /** Discovered Desktop bundle identity, retained by its generated launcher across respawns. */ + expectedBundleRoot?: string; macBundleTrustReason?: (bundleRoot: string, binary: string) => string | null; } @@ -113,22 +118,36 @@ function resolveDarwinExecPath(command: string, deps: CodexExecBoundaryDeps): { /** * Canonical (bundleRoot, binary) pair when the resolved path classifies into a - * known Codex desktop bundle, else null. The pair is always consistent: + * known Codex desktop bundle (or any enclosing app for CLI handoffs), else null. The pair is always consistent: * both halves are canonicalized together so a symlinked HOME or a PATH alias * cannot split the containment check, and a bundle member that cannot be * canonicalized refuses rather than escaping the known-bundle classification. */ function darwinBundleTarget(resolved: string, deps: CodexExecBoundaryDeps): { bundle: string; binary: string } | null { const realpath = deps.realpath ?? realpathOrNull; - const bundle = macCodexAppBundleFor(resolved, { env: deps.env, platform: deps.platform, realpath }); - if (!bundle) return null; + if (deps.expectedBundleRoot !== undefined && !posix.isAbsolute(deps.expectedBundleRoot)) { + throw new CodexExecTrustError("the recorded app bundle must be an absolute path"); + } const binary = realpath(resolved); + const enclosingApp = (path: string) => /^(.*?\/[^/]*\.app)(?:\/|$)/i.exec(path)?.[1]; + const lexicalBundle = deps.verifyAnyAppBundle && deps.expectedBundleRoot === undefined ? enclosingApp(resolved) : undefined; + // Verify the outer canonical app for nested layouts, but retain lexical containment. + const enclosing = deps.verifyAnyAppBundle && binary !== null ? enclosingApp(binary) : undefined; + const bundle = deps.expectedBundleRoot ?? enclosing ?? lexicalBundle ?? macCodexAppBundleFor(resolved, { env: deps.env, platform: deps.platform, realpath }); + if (!bundle) return null; if (binary === null) throw new CodexExecTrustError("cannot canonicalize the app-bundle exec target"); const canonicalBundle = realpath(bundle); if (canonicalBundle === null) throw new CodexExecTrustError("cannot canonicalize the app bundle"); if (!binary.startsWith(canonicalBundle + "/Contents/")) { throw new CodexExecTrustError("the app-bundle exec target is outside bundle Contents"); } + if (lexicalBundle && lexicalBundle !== bundle) { + const canonicalLexicalBundle = realpath(lexicalBundle); + if (canonicalLexicalBundle === null) throw new CodexExecTrustError("cannot canonicalize the app bundle"); + if (!binary.startsWith(canonicalLexicalBundle + "/Contents/")) { + throw new CodexExecTrustError("the app-bundle exec target is outside bundle Contents"); + } + } return { bundle: canonicalBundle, binary }; } @@ -144,6 +163,7 @@ export function validatedCodexExecTarget( deps: CodexExecBoundaryDeps = {}, ): ValidatedCodexExecTarget { if ((deps.platform ?? process.platform) !== "darwin") { + if (deps.expectedBundleRoot !== undefined) throw new CodexExecTrustError("recorded app bundles require macOS verification"); return { invocation: codexExecInvocation(command, args, deps.platform ?? process.platform, { env: deps.env, exists: deps.exists }), macAppBundle: false }; } const resolved = resolveDarwinExecPath(command, deps); @@ -164,6 +184,7 @@ export async function validatedCodexExecTargetAsync( ): Promise { deps.signal?.throwIfAborted(); if ((deps.platform ?? process.platform) !== "darwin") { + if (deps.expectedBundleRoot !== undefined) throw new CodexExecTrustError("recorded app bundles require macOS verification"); return { invocation: codexExecInvocation(command, args, deps.platform ?? process.platform, { env: deps.env, exists: deps.exists }), macAppBundle: false }; } const resolved = resolveDarwinExecPath(command, deps); diff --git a/structure/cli-management.md b/structure/cli-management.md index e601840594a..da5ae2d54f9 100644 --- a/structure/cli-management.md +++ b/structure/cli-management.md @@ -34,7 +34,7 @@ For fresh credentials, the launcher resolves a native literal `CODEX_HOME` once `src/cli/codex-launcher.ts` implements `ocx codex` without rewriting routing. Native/foreign routes pass through; local/unknown routes attempt startup and warn on failure. Fast paths skip startup/token loading. A matched wrapper uses its backing directly. Unknown state, including a throwing inspector, requires a separate bypass check via `src/cli/codex-launch-command.ts`; managed Windows CMD wrappers require complete generated-template equality before the selected absolute path executes with bypass. Obsolete, unreadable or ambiguous wrappers and lookup errors refuse launch without exception contents. A missing executable instead reports the Codex install hint, including on help/version fast paths. This content check does not establish Windows backing provenance or prevent concurrent replacement. `src/cli/codex-launch-auth.ts` separately requires a parsed managed transport block matching this installation's loopback listener or persisted local client connection; names/comments alone grant no token. Profiles, distinct project layers scanned through the filesystem root, destination-changing argv and environment overrides prevent automatic loading. An existing ancestor config is excluded only when its realpath matches the home config; a different file or failed identity resolution refuses. Token-file overrides and management-token collisions remain refused; user exports are preserved. No token enters argv or diagnostics. -The `codex` namespace forwards all help arguments unchanged; `ocx help codex` retains wrapper help. It skips root shim repair so the routing-aware launcher owns the handoff. +The `codex` namespace forwards all help arguments unchanged; `ocx help codex` retains wrapper help. It skips root shim repair so the routing-aware launcher owns the handoff. Every handoff, including help/version and passthrough, awaits the [execution trust contract](clients/chatgpt-desktop.md#codex-app-execution-trust) for the selected backing and spawns the returned invocation. On macOS, CLI handoffs verify any enclosing `.app` at the canonical target location, including renamed bundles and symlink targets; nested apps retain the outer bundle root. Fast paths bypass startup and credential loading, not executable validation. The hidden `internal codex-exec-target` helper returns a single verified path for the generated app-server launcher; its strict `--bundle-root` mode retains the discovered root independent of bundle name and refuses malformed roots without falling back to generic classification. It skips root shim auto-repair even on malformed arguments, and stays outside public capability discovery. Forwarded `-c`/`--config` and `-p`/`--profile` overrides make effective routing unverified for automatic service-token injection. They retain original argv and user-exported credentials but never load a new token from disk. Single-word prompts require `--` so they cannot select an unknown native subcommand. Missing system files also require a protected namespace: the closest existing directory must prevent another user from creating the missing layer, and its ancestors must prevent replacement. A normal Windows ProgramData directory that grants Users directory creation cannot authorize an absent OpenAI/Codex namespace; the launcher withholds credentials without creating or changing that directory. macOS opens each validated entry without following its final link (O_SYMLINK for the link entry itself), keeps dev/ino as bigint and requires descriptor dev/ino/type to match its original lstat, and reads its ACL with acl_get_fd_np. Only NULL with immediately captured ENOENT on this descriptor query means no extended ACL; open/stat/identity/close failures and every other NULL/error refuse. Every opened descriptor closes in finally. For a present ACL it validates and enumerates every entry through the documented -1/EINVAL terminator. Deny-only and read-only allow entries are accepted; any allow granting write/add-file, append/add-subdirectory, delete, delete-child, attribute, extended-attribute, security or ownership mutation refuses regardless of earlier denies. Unknown tags and FFI/enumeration failures refuse. Forced managed-preference checks remain mandatory. Pure policy and Darwin-gated native ACL coverage lives in `tests/codex-integration/codex-launcher-darwin-acl.test.ts`; modeled IO regressions for all three walks live in `tests/codex-integration/codex-launcher-darwin-identity.test.ts`. Tests model a protected empty machine namespace inside their private fixture; they do not modify real system configuration. Windows fixtures probe ancestor ownership before positive assertions; an unavailable protected location requires a proved ownership refusal and an explicit skip reason. Negative mutations and no-token/zero-read assertions execute even with unsafe ancestors; credential-file negatives bypass destination verification independently. Native checks use production defaults. `src/cli/root.ts`, `src/cli/help.ts`, `src/cli/help-navigation.ts` diff --git a/structure/clients/chatgpt-desktop.md b/structure/clients/chatgpt-desktop.md index b647bd7af5d..3038abdafd1 100644 --- a/structure/clients/chatgpt-desktop.md +++ b/structure/clients/chatgpt-desktop.md @@ -7,18 +7,23 @@ accepts only an optional boolean; malformed reads disable the leaf, while live writes reject malformed values and unknown fields. The launcher under the config directory re-enters the current CLI using -`process.execPath` and `selfLaunchArgv()`. Its hidden internal filter command stays +`process.execPath` and `selfLaunchArgv()`. Its hidden filter and executable-validation commands stay out of the public command registry and generated skill surface. Source installs include the CLI entry argument; compiled builds use only the executable and internal command arguments. -The launcher checks macOS, executable presence and a successful filter self-test, -then replaces itself with the bundled app-server using shell exec. Only stdout +The launcher uses macOS and a successful filter self-test to select stdout filtering. +Every invocation, including a Desktop respawn and the unfiltered branch, re-enters +`internal codex-exec-target --bundle-root -- ` immediately before shell exec. The helper applies +the execution trust contract below to the recorded discovered root, regardless of its name, +and returns the exact target to execute; the shell +does not use `uname` as execution authority. Only stdout passes through the filter. Stdin, stderr, process identity and the real server's exit status retain the direct app/server relationship. -When the platform is not macOS, the runtime is missing, or the filter self-test fails, -the launcher runs the original binary with untouched stdout. A missing bundled binary +When filtering is unavailable or its self-test fails, the launcher runs the validated +target with untouched stdout. An unavailable validator, failed trust check or empty +target exits 126 without executing the recorded binary; a missing bundled binary exits 127 instead (see below). A filter that passes the self-test and then exits mid-session closes the server's stdout pipe; the filter's passthrough mode limits this to an exit/crash case. @@ -59,6 +64,14 @@ OpenAI's. It writes the mode-0755 executable through an exclusive temp file and rename (never through a symbolic link), quits the bundle by id, waits for this user's instance to exit, then opens the same bundle path with the launcher in CODEX_CLI_PATH. The launcher itself exits 127 with a stderr hint when the recorded binary is gone. +The discovered root is a quoted literal in the launcher itself, with no metadata sidecar. +Before publication, the writer checks its parent/ancestor replacement permissions and the +new file's ownership, type and permissions using the shared POSIX filesystem policy. +An unsafe path refuses publication. The strict helper mode refuses a missing or empty root, +failed canonicalization or a target outside the recorded root's canonical `Contents/`. +It always verifies that root and binary, even if the bundle is renamed or its identifier changes. +Same-user processes, the installed CLI/runtime and the launcher code remain trusted; these +checks do not attest extended ACLs or make verification and exec an atomic operation. Restore uses the same trust policy for the bundle and its main app executable before quit or open. Both relaunch paths check ancestor ownership and POSIX replacement permissions through the filesystem root. Ancestors must be owned by this user or root; group/other write @@ -99,7 +112,23 @@ The name only routes to verification. `src/codex/exec-boundary.ts` requires a ca pair with the binary strictly inside that bundle's `Contents/`, then applies the same synchronous or asynchronous OpenAI signature and POSIX filesystem policy before returning an invocation. Canonicalization or verification failure refuses execution. This covers environment/configured -commands and catalog, prompt, login, refresh and remote-workspace consumers; ordinary CLI and -third-party app paths retain their existing policy. Conventional automatic discovery remains +commands and catalog, prompt, login, refresh and remote-workspace consumers. The +`src/cli/codex-launcher.ts` handoff validates the selected shim backing on routed, +passthrough and help/version fast paths, preserving the validated invocation's argv, +environment and Windows interpreter options. This CLI handoff additionally verifies every +enclosing `.app` at the lexical or canonical executable location, independent of its name or mutable +bundle identifier. A lexical app member must remain inside that root's canonical `Contents/`, +including through executable, directory or whole-bundle aliases. Nested apps use the outer canonical +bundle root so its signature is checked too; failed canonicalization refuses. Generated app-server launchers use the +hidden validator in `src/cli/internal-command.ts` on each execution; this command skips +root shim auto-repair, its stdout is a single path, and newline-bearing input or +canonical output is refused. Ordinary non-bundle CLI paths retain their existing policy; +third-party app paths retain their policy outside the stricter CLI handoff. Conventional automatic discovery remains limited to the two ChatGPT installation roots. Regression coverage is in -`tests/codex-integration/exec-known-app-bundles.test.ts`. +`tests/codex-integration/exec-known-app-bundles.test.ts`, +`tests/codex-integration/exec-launcher-boundary.test.ts`, +`tests/codex-integration/exec-cli-renamed-app-bundle.test.ts`, +`tests/codex-integration/exec-cli-bundle-containment.test.ts`, +`tests/codex-integration/exec-renamed-app-bundle.test.ts` and +`tests/clients/desktop-app-server-exec-boundary.test.ts`. Portable injected-verifier and +shell tests do not establish native macOS signature or real Desktop respawn acceptance. diff --git a/tests/clients/desktop-app-server-exec-boundary.test.ts b/tests/clients/desktop-app-server-exec-boundary.test.ts new file mode 100644 index 00000000000..fb604e111b1 --- /dev/null +++ b/tests/clients/desktop-app-server-exec-boundary.test.ts @@ -0,0 +1,76 @@ +import { expect, test } from "bun:test"; +import { chmodSync, existsSync, mkdtempSync, readFileSync, unlinkSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { spawnSync } from "node:child_process"; +import { buildChatgptShimLauncher } from "../../src/chatgpt/app-server-shim/launcher"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; + +const shellTest = process.platform !== "win32" && spawnSync("/bin/bash", ["-c", "exit 0"]).status === 0 ? test : test.skip; +const quote = (s: string) => `'${s.replaceAll("'", "'\\''")}'`; +function executable(dir: string, name: string, body: string) { + const path = join(dir, name); writeFileSync(path, `#!/bin/bash\n${body}\n`); chmodSync(path, 0o755); return path; +} +function fixture(run: (dir: string) => void) { + const dir = mkdtempSync(join(tmpdir(), "ocx-app-exec-")); + try { run(dir); } finally { removeTreeWithRetry(dir); } +} + +shellTest("each respawn validates and executes the returned target with original argv, stdin and PID", () => fixture(dir => { + const trace = join(dir, "trace"); + const recorded = executable(dir, "recorded.sh", "exit 99"); + const verified = executable(dir, "canonical with 'quote.sh", 'printf "PID:%s ARGS:%s\\n" "$$" "$*"\ncat\nexit 7'); + const validator = executable(dir, "validate", `echo validate >> ${quote(trace)}\nprintf '%s\\n' ${quote(verified)}`); + const launcher = executable(dir, "launcher", buildChatgptShimLauncher([join(dir, "missing-filter")], recorded, [validator])); + for (let i = 0; i < 2; i++) { + const out = spawnSync("/bin/bash", [launcher, "app-server", "arg with spaces"], { encoding: "utf8", input: "stdin\n", timeout: 5000 }); + expect(out.error).toBeUndefined(); expect(out.status).toBe(7); + expect(out.stdout).toBe(`PID:${out.pid} ARGS:app-server arg with spaces\nstdin\n`); + } + expect(readFileSync(trace, "utf8")).toBe("validate\nvalidate\n"); +})); + +shellTest("revocation after launcher creation blocks later execution even when uname says Linux", () => fixture(dir => { + const marker = join(dir, "spawned"), allow = join(dir, "allowed"); + const real = executable(dir, "real", `echo ran >> ${quote(marker)}`); + const validator = executable(dir, "validate", `[ -e ${quote(allow)} ] || exit 1\nprintf '%s\\n' "$2"`); + executable(dir, "uname", "echo Linux"); + const launcher = executable(dir, "launcher", buildChatgptShimLauncher([join(dir, "missing-filter")], real, [validator])); + const run = () => spawnSync("/bin/bash", [launcher], { encoding: "utf8", env: { ...process.env, PATH: `${dir}:${process.env.PATH}` }, timeout: 5000 }); + writeFileSync(allow, "allowed"); + expect(run().status).toBe(0); + expect(readFileSync(marker, "utf8")).toBe("ran\n"); + unlinkSync(allow); + const out = run(); + expect(out.status).toBe(126); expect(readFileSync(marker, "utf8")).toBe("ran\n"); + expect(out.stderr).toContain("execution verification failed"); +})); + +for (const status of [0, 1]) shellTest(`filter preflight ${status} cannot bypass the last executable check`, () => fixture(dir => { + const trace = join(dir, "trace"), marker = join(dir, "spawned"); + executable(dir, "uname", "echo Darwin"); + const real = executable(dir, "real", `echo ran >> ${quote(marker)}`); + const filter = executable(dir, "filter", `echo filter >> ${quote(trace)}\nexit ${status}`); + const validator = executable(dir, "validate", `echo validate >> ${quote(trace)}\nexit 1`); + const launcher = executable(dir, "launcher", buildChatgptShimLauncher([filter], real, [validator])); + const out = spawnSync("/bin/bash", [launcher], { encoding: "utf8", env: { ...process.env, PATH: `${dir}:${process.env.PATH}` }, timeout: 5000 }); + expect(out.status).toBe(126); expect(existsSync(marker)).toBe(false); + expect(readFileSync(trace, "utf8")).toBe("filter\nvalidate\n"); +})); + +shellTest("missing validator refuses instead of falling back to the unchecked target", () => fixture(dir => { + const marker = join(dir, "spawned"); + const real = executable(dir, "real", `echo ran >> ${quote(marker)}`); + const launcher = executable(dir, "launcher", buildChatgptShimLauncher([join(dir, "filter")], real, [join(dir, "missing-validator")])); + const out = spawnSync("/bin/bash", [launcher], { encoding: "utf8", timeout: 5000 }); + expect(out.status).toBe(126); expect(existsSync(marker)).toBe(false); +})); + +shellTest("an empty successful validator result does not authorize the recorded target", () => fixture(dir => { + const marker = join(dir, "spawned"); + const real = executable(dir, "real", `echo ran >> ${quote(marker)}`); + const validator = executable(dir, "validate", "exit 0"); + const launcher = executable(dir, "launcher", buildChatgptShimLauncher([join(dir, "filter")], real, [validator])); + const out = spawnSync("/bin/bash", [launcher], { encoding: "utf8", timeout: 5000 }); + expect(out.status).toBe(126); expect(existsSync(marker)).toBe(false); +})); diff --git a/tests/clients/desktop-app-server-shim-launcher.test.ts b/tests/clients/desktop-app-server-shim-launcher.test.ts index db53fc7fd0b..f26bd6354b1 100644 --- a/tests/clients/desktop-app-server-shim-launcher.test.ts +++ b/tests/clients/desktop-app-server-shim-launcher.test.ts @@ -1,5 +1,5 @@ import { describe, expect, test } from "bun:test"; -import { chmodSync, lstatSync, mkdtempSync, readdirSync, readFileSync, statSync, symlinkSync, writeFileSync } from "node:fs"; +import { chmodSync, lstatSync, mkdtempSync, readdirSync, readFileSync, realpathSync, statSync, symlinkSync, writeFileSync } from "node:fs"; import { OPENAI_TEAM_ID, untrustedChatgptBundleReason, type BundleTrustDeps } from "../../src/chatgpt/app-server-shim/bundle-trust"; import { tmpdir } from "node:os"; import { dirname, join, resolve, sep } from "node:path"; @@ -26,10 +26,13 @@ const executable = (dir: string, name: string, body: string) => { return path; }; function withDir(run: (dir: string) => void) { - const dir = mkdtempSync(join(tmpdir(), "ocx-shim-test-")); + const dir = realpathSync(mkdtempSync(join(tmpdir(), "ocx-shim-test-"))); try { run(dir); } finally { removeTreeWithRetry(dir); } } const internalArgs = ["internal", "chatgpt-app-server-filter"]; +const validatorArgv = [process.execPath, ...selfLaunchArgv(["internal", "codex-exec-target"], { + isStandaloneExecutable: false, sourceEntrypoint: repoPath("src/cli/index.ts"), +})]; describe("experimental app-server launcher", () => { test("source and compiled argv re-enter the CLI with shell-safe quoting", () => { @@ -37,8 +40,10 @@ describe("experimental app-server launcher", () => { const argv = ["/runtime with 'quote", ...selfLaunchArgv(internalArgs, { isStandaloneExecutable: standalone, sourceEntrypoint: "/source with space/index.ts", })]; - const launcher = buildChatgptShimLauncher(argv, "/real with space"); + const validator = [...argv.slice(0, -1), "codex-exec-target"]; + const launcher = buildChatgptShimLauncher(argv, "/real with space", validator); expect(launcher).toContain("'internal' 'chatgpt-app-server-filter'"); + expect(launcher).toContain("'internal' 'codex-exec-target'"); expect(launcher.includes("'/source with space/index.ts'")).toBe(!standalone); expect(launcher).toContain("'\\''"); expect(launcher).toContain('--self-test >/dev/null 2>&1'); @@ -103,13 +108,13 @@ describe("experimental app-server launcher", () => { executable(dir, "uname", 'echo Darwin'); return { ...process.env, PATH: `${dir}:${process.env.PATH ?? "/usr/bin:/bin"}` }; }; - for (const scenario of ["missing runtime", "failing self-test", "passing self-test"] as const) { + for (const scenario of ["missing filter runtime", "failing self-test", "passing self-test"] as const) { const scenarioTest = (scenario === "passing self-test" ? substitutionAvailable : bashAvailable) ? test : test.skip; scenarioTest(scenario, () => withDir(dir => { const real = executable(dir, "real with 'quote.sh", 'echo "PID:$$ ARGS:$*"\nread -r line\necho "STDIN:$line"\necho "to-stderr" >&2\nexit 7'); - const filter = scenario === "missing runtime" ? join(dir, "missing") : executable(dir, "filter.sh", + const filter = scenario === "missing filter runtime" ? join(dir, "missing") : executable(dir, "filter.sh", `if [ "$1" = --self-test ]; then exit ${scenario === "failing self-test" ? 1 : 0}; fi\nsed 's/STDIN/FILTERED/'`); - const launcher = executable(dir, "launcher.sh", buildChatgptShimLauncher([filter], real)); + const launcher = executable(dir, "launcher.sh", buildChatgptShimLauncher([filter], real, validatorArgv)); const out = spawnSync("/bin/bash", [launcher, "app-server", "--flag"], { encoding: "utf8", input: "hello\n", env: environment(dir), timeout: 5000, }); @@ -124,7 +129,7 @@ describe("experimental app-server launcher", () => { const argv = [process.execPath, ...selfLaunchArgv(internalArgs, { isStandaloneExecutable: false, sourceEntrypoint: repoPath("src/cli/index.ts"), })]; - const launcher = executable(dir, "launcher.sh", buildChatgptShimLauncher(argv, real)); + const launcher = executable(dir, "launcher.sh", buildChatgptShimLauncher(argv, real, validatorArgv)); const out = spawnSync("/bin/bash", [launcher, "app-server"], { encoding: "utf8", env: environment(dir), timeout: 10000 }); expect(out.status).toBe(0); const result = JSON.parse(out.stdout).result; @@ -135,7 +140,7 @@ describe("experimental app-server launcher", () => { executionTest("a filter exit after successful preflight closes the pipe and terminates a mock server", () => withDir(dir => { const real = executable(dir, "real.sh", 'trap "exit 23" PIPE\nwhile :; do printf "%s\\n" "mock server stdout" || exit 23; done'); const filter = executable(dir, "filter.sh", 'if [ "$1" = --self-test ]; then exit 0; fi\nexit 1'); - const launcher = executable(dir, "launcher.sh", buildChatgptShimLauncher([filter], real)); + const launcher = executable(dir, "launcher.sh", buildChatgptShimLauncher([filter], real, validatorArgv)); const out = spawnSync("/bin/bash", [launcher], { encoding: "utf8", env: environment(dir), timeout: 5000 }); expect(out.error).toBeUndefined(); expect(out.status).toBe(23); diff --git a/tests/codex-integration/exec-cli-bundle-containment.test.ts b/tests/codex-integration/exec-cli-bundle-containment.test.ts new file mode 100644 index 00000000000..65ecd399ef9 --- /dev/null +++ b/tests/codex-integration/exec-cli-bundle-containment.test.ts @@ -0,0 +1,148 @@ +import { afterEach, beforeEach, expect, spyOn, test } from "bun:test"; +import { EventEmitter } from "node:events"; +import type { spawn } from "node:child_process"; +import { spawnCodex, type CodexSpawnDeps } from "../../src/cli/codex-launcher"; +import { validatedCodexExecInvocation } from "../../src/codex/exec-boundary"; + +const root = "/opt/ChatGPT Beta.app"; +const member = `${root}/Contents/Resources/codex`; +type Link = readonly [string, string | null]; +let errors: ReturnType>; +beforeEach(() => { errors = spyOn(console, "error").mockImplementation(() => {}); }); +afterEach(() => { errors.mockRestore(); }); + +// Model realpath at its existing seam, including redirects of ancestor directories. +function harness(command: string, backing: boolean, links: readonly Link[], refusal: string | null = null) { + const checks: string[][] = [], spawns: string[] = []; + const deps: CodexSpawnDeps = { + findOnPath: () => backing ? "/owned/codex-wrapper" : command, + inspectShim: () => (backing + ? { status: "matched", selectedRole: "wrapper", backingPath: command } + : { status: "absent" }) as ReturnType>, + execBoundary: { + platform: "darwin", executable: () => true, + realpath: path => { + const link = links.find(([from]) => path === from || path.startsWith(from + "/")); + return link ? link[1] === null ? null : link[1] + path.slice(link[0].length) : path; + }, + macBundleTrustReasonAsync: async (bundle, binary) => { checks.push([bundle, binary]); return refusal; }, + }, + spawn: ((file: string) => { + spawns.push(file); + const child = new EventEmitter(); + queueMicrotask(() => child.emit("exit", 0, null)); + return child; + }) as typeof spawn, + }; + return { deps, checks, spawns }; +} + +const escapes: Array<{ name: string; links: Link[] }> = [ + { name: "executable symlink", links: [[member, "/outside/codex"]] }, + { name: "Resources symlink", links: [[`${root}/Contents/Resources`, "/outside"]] }, + { name: "Contents symlink", links: [[`${root}/Contents`, "/outside"]] }, + { name: "member in another signed app", links: [[member, "/opt/Other.app/Contents/Resources/codex"]] }, + { name: "prefix sibling", links: [[member, `${root}-sibling/Contents/Resources/codex`]] }, + { name: "outside Contents", links: [[member, `${root}/MacOS/codex`]] }, + { name: "unresolvable lexical root", links: [[member, "/opt/Other.app/Contents/Resources/codex"], [root, null]] }, + { name: "aliased root with an escaped member", links: [[member, "/outside/codex"], [root, "/real/Bundle"]] }, + { name: "aliased inner root escaped within the outer app", links: [ + [member, "/real/Outer.app/Contents/Resources/codex"], + [root, "/real/Outer.app/Contents/Resources/Inner.app"], + ] }, +]; + +for (const backing of [false, true]) for (const scenario of escapes) { + test(`CLI refuses lexical bundle escape before an accepting verifier: ${scenario.name}, backing=${backing}`, async () => { + const h = harness(member, backing, scenario.links); + expect(await spawnCodex(["--version"], {}, h.deps)).toBe(1); + expect(h.checks).toEqual([]); + expect(h.spawns).toEqual([]); + expect(errors.mock.calls.flat().join(" ")).toContain( + scenario.name === "unresolvable lexical root" ? "cannot canonicalize the app bundle" : "outside bundle Contents", + ); + }); +} + +for (const backing of [false, true]) for (const canonicalRoot of ["/real/Bundle", "/real/Renamed.APP"]) { + test(`CLI verifies a whole-bundle symlink: ${canonicalRoot}, backing=${backing}`, async () => { + const binary = `${canonicalRoot}/Contents/Resources/codex`; + const h = harness(member, backing, [[root, canonicalRoot]], "signature revoked"); + expect(await spawnCodex(["--help"], {}, h.deps)).toBe(1); + expect(h.checks).toEqual([[canonicalRoot, binary]]); + expect(h.spawns).toEqual([]); + const trusted = harness(member, backing, [[root, canonicalRoot]]); + expect(await spawnCodex(["--help"], {}, trusted.deps)).toBe(0); + expect(trusted.checks).toEqual([[canonicalRoot, binary]]); + expect(trusted.spawns).toEqual([binary]); + }); +} + +for (const backing of [false, true]) for (const from of [member, `${root}/Contents/Resources`]) { + test(`CLI validates an in-bundle member alias: ${from}, backing=${backing}`, async () => { + const target = from === member ? `${root}/Contents/MacOS/codex` : `${root}/Contents/MacOS`; + const links: Link[] = [[from, target]]; + const canonical = `${root}/Contents/MacOS/codex`; + const trusted = harness(member, backing, links); + expect(await spawnCodex(["--version"], {}, trusted.deps)).toBe(0); + expect(trusted.checks).toEqual([[root, canonical]]); + expect(trusted.spawns).toEqual([canonical]); + const rejected = harness(member, backing, links, "signature revoked"); + expect(await spawnCodex(["--version"], {}, rejected.deps)).toBe(1); + expect(rejected.checks).toEqual([[root, canonical]]); + expect(rejected.spawns).toEqual([]); + }); +} + +for (const suffix of ["app", "APP", "ApP"]) { + test(`CLI retains lexical .${suffix} classification when the canonical member escapes`, async () => { + const binary = `/opt/Unrelated.${suffix}/Contents/Resources/codex`; + const h = harness(binary, false, [[binary, "/outside/codex"]]); + expect(await spawnCodex(["app-server"], {}, h.deps)).toBe(1); + expect(h.checks).toEqual([]); + expect(h.spawns).toEqual([]); + }); +} + +for (const backing of [false, true]) { + test(`CLI spawns the captured canonical member after lexical alias replacement, backing=${backing}`, async () => { + const canonical = `${root}/Contents/MacOS/codex`; + const links: Link[] = [[member, canonical]]; + const h = harness(member, backing, links); + let entered!: () => void, release!: () => void; + const checking = new Promise(resolve => { entered = resolve; }); + const pending = new Promise(resolve => { release = resolve; }); + h.deps.execBoundary!.macBundleTrustReasonAsync = async (bundle, binary) => { + h.checks.push([bundle, binary]); entered(); + await pending; + return null; + }; + const launch = spawnCodex(["--help"], {}, h.deps); + await checking; + links[0] = [member, "/outside/replaced-codex"]; + release(); + expect(await launch).toBe(0); + expect(h.checks).toEqual([[root, canonical]]); + expect(h.spawns).toEqual([canonical]); + }); +} + +test("CLI checks the outer canonical app when the lexical bundle root is an inner app alias", async () => { + const outer = "/real/Outer.app"; + const inner = `${outer}/Contents/Resources/Inner.app`; + const binary = `${inner}/Contents/Resources/codex`; + const h = harness(member, true, [[root, inner]], "signature revoked"); + expect(await spawnCodex(["--help"], {}, h.deps)).toBe(1); + expect(h.checks).toEqual([[outer, binary]]); + expect(h.spawns).toEqual([]); +}); + +test("synchronous CLI validation applies the same lexical containment refusal", () => { + let checks = 0; + expect(() => validatedCodexExecInvocation(member, [], { + platform: "darwin", verifyAnyAppBundle: true, executable: () => true, + realpath: path => path === member ? "/outside/codex" : path, + macBundleTrustReason: () => { checks++; return null; }, + })).toThrow("outside bundle Contents"); + expect(checks).toBe(0); +}); diff --git a/tests/codex-integration/exec-cli-renamed-app-bundle.test.ts b/tests/codex-integration/exec-cli-renamed-app-bundle.test.ts new file mode 100644 index 00000000000..6e3db0ac335 --- /dev/null +++ b/tests/codex-integration/exec-cli-renamed-app-bundle.test.ts @@ -0,0 +1,76 @@ +import { afterEach, beforeEach, expect, spyOn, test } from "bun:test"; +import { EventEmitter } from "node:events"; +import type { spawn } from "node:child_process"; +import { spawnCodex, type CodexSpawnDeps } from "../../src/cli/codex-launcher"; + +const root = "/opt/OpenAI/ChatGPT Beta.app"; +const binary = `${root}/Contents/Resources/codex`; +const alias = "/usr/local/bin/codex"; +let errors: ReturnType>; +beforeEach(() => { errors = spyOn(console, "error").mockImplementation(() => {}); }); +afterEach(() => { errors.mockRestore(); }); + +function harness(command: string, backing: boolean, target = binary) { + let allowed = true; + const checks: string[][] = [], spawns: string[] = []; + const deps: CodexSpawnDeps = { + findOnPath: () => backing ? "/owned/codex-wrapper" : command, + inspectShim: () => (backing + ? { status: "matched", selectedRole: "wrapper", backingPath: command } + : { status: "absent" }) as ReturnType>, + execBoundary: { + platform: "darwin", executable: () => true, + // Model a PATH symlink to the canonical member; no conventional bundle alias exists. + realpath: path => path === command ? target : path, + macBundleTrustReasonAsync: async (bundle, executable) => { + checks.push([bundle, executable]); + return allowed ? null : "signature revoked"; + }, + }, + spawn: ((file: string) => { + spawns.push(file); + const child = new EventEmitter(); + queueMicrotask(() => child.emit("exit", 0, null)); + return child; + }) as typeof spawn, + }; + return { deps, checks, spawns, revoke: () => { allowed = false; } }; +} + +for (const backing of [false, true]) for (const symlink of [false, true]) { + test(`CLI renamed bundle rechecks revocation: shim backing=${backing}, symlink=${symlink}`, async () => { + const h = harness(symlink ? alias : binary, backing); + expect(await spawnCodex(["--version"], {}, h.deps)).toBe(0); + h.revoke(); + expect(await spawnCodex(["--version"], {}, h.deps)).toBe(1); + expect(h.checks).toEqual([[root, binary], [root, binary]]); + expect(h.spawns).toEqual([binary]); + expect(errors.mock.calls.flat().join(" ")).toContain("signature revoked"); + }); +} + +test("CLI validates the outer renamed root for a nested app executable", async () => { + const nested = `${root}/Contents/Resources/CodexCLI.app/Contents/MacOS/codex`; + const h = harness(alias, true, nested); + h.revoke(); + expect(await spawnCodex(["app-server"], {}, h.deps)).toBe(1); + expect(h.checks).toEqual([[root, nested]]); + expect(h.spawns).toEqual([]); +}); + +test("CLI refuses an unresolved renamed bundle member before trust or spawn", async () => { + const h = harness(binary, false); + h.deps.execBoundary!.realpath = path => path === binary ? null : path; + expect(await spawnCodex(["--help"], {}, h.deps)).toBe(1); + expect(h.checks).toEqual([]); + expect(h.spawns).toEqual([]); +}); + +test("CLI ordinary executable aliases still bypass bundle verification", async () => { + const target = "/opt/npm/bin/codex"; + const h = harness(alias, true, target); + h.revoke(); + expect(await spawnCodex(["--help"], {}, h.deps)).toBe(0); + expect(h.checks).toEqual([]); + expect(h.spawns).toEqual([alias]); +}); diff --git a/tests/codex-integration/exec-launcher-boundary.test.ts b/tests/codex-integration/exec-launcher-boundary.test.ts new file mode 100644 index 00000000000..ac61f3d0b2f --- /dev/null +++ b/tests/codex-integration/exec-launcher-boundary.test.ts @@ -0,0 +1,143 @@ +import { afterEach, beforeEach, expect, spyOn, test } from "bun:test"; +import { EventEmitter } from "node:events"; +import type { spawn } from "node:child_process"; +import { spawnCodex, type CodexSpawnDeps } from "../../src/cli/codex-launcher"; +import { handleInternalCommand } from "../../src/cli/internal-command"; +import type { CodexExecBoundaryAsyncDeps } from "../../src/codex/exec-boundary"; +import { maybeAutoRestoreCodexShim, skipsCodexShimAutoRestore } from "../../src/cli/codex-shim-autorestore"; + +const bundle = "/Applications/ChatGPT.app"; +const binary = `${bundle}/Contents/Resources/codex`; +const alias = "/usr/local/bin/codex"; +let logs: ReturnType>; +let errors: ReturnType>; +beforeEach(() => { + logs = spyOn(console, "log").mockImplementation(() => {}); + errors = spyOn(console, "error").mockImplementation(() => {}); +}); +afterEach(() => { logs.mockRestore(); errors.mockRestore(); }); + +function harness(check: NonNullable) { + const calls: Array<{ file: string; args: string[]; options: Record }> = []; + const boundary: CodexExecBoundaryAsyncDeps = { + platform: "darwin", executable: () => true, + realpath: path => path === alias ? binary : path, + macBundleTrustReasonAsync: check, + }; + const deps: CodexSpawnDeps = { + findOnPath: () => "/owned/codex-wrapper", + inspectShim: () => ({ status: "matched", selectedRole: "wrapper", backingPath: alias }) as ReturnType>, + execBoundary: boundary, + spawn: ((file: string, args: string[], options: Record) => { + calls.push({ file, args, options }); + const child = new EventEmitter(); + queueMicrotask(() => child.emit("exit", 7, null)); + return child; + }) as typeof spawn, + }; + return { calls, boundary, deps }; +} + +test("the selected shim backing waits for trust, then spawns the canonical binary", async () => { + let finish!: (reason: string | null) => void; + let entered!: () => void; + const started = new Promise(resolve => { entered = resolve; }); + const trust = new Promise(resolve => { finish = resolve; }); + const h = harness(async (root, path) => { expect(root).toBe(bundle); expect(path).toBe(binary); entered(); return trust; }); + const env = { PATH: "/usr/bin:/bin", OCX_BOUNDARY_FIXTURE: "preserved" }; + const running = spawnCodex(["--version"], env, h.deps); + await started; + expect(h.calls).toHaveLength(0); + finish(null); + expect(await running).toBe(7); + expect(h.calls).toHaveLength(1); + expect(h.calls[0]).toEqual({ file: binary, args: ["--version"], options: { stdio: "inherit", env } }); +}); + +test("each invocation rechecks trust and a revoked signature starts no replacement", async () => { + let count = 0; + const h = harness(async () => ++count === 1 ? null : "signature changed"); + expect(await spawnCodex(["--help"], {}, h.deps)).toBe(7); + expect(await spawnCodex(["--help"], {}, h.deps)).toBe(1); + expect(count).toBe(2); + expect(h.calls).toHaveLength(1); + expect(errors.mock.calls.flat().join(" ")).toContain("signature changed"); +}); + +test("a failed or thrown trust check never reaches spawn", async () => { + for (const check of [async () => "foreign owner", async () => { throw new Error("inspection unavailable"); }]) { + const h = harness(check); + expect(await spawnCodex(["app-server"], {}, h.deps)).toBe(1); + expect(h.calls).toHaveLength(0); + } +}); + +test("a selected executable that disappears before verification keeps the install hint", async () => { + const h = harness(async () => { throw new Error("must not inspect a missing target"); }); + h.deps.findOnPath = () => "/usr/local/bin/codex"; + h.deps.inspectShim = () => ({ status: "absent" }) as ReturnType>; + h.deps.execBoundary = { ...h.boundary, executable: () => false, exists: () => false }; + expect(await spawnCodex(["--version"], {}, h.deps)).toBe(1); + expect(h.calls).toHaveLength(0); + expect(errors.mock.calls.flat().join(" ")).toContain("codex` CLI not found"); +}); + +test("non-bundle targets preserve arguments and do not invoke bundle verification", async () => { + const h = harness(async () => { throw new Error("must not inspect an ordinary CLI"); }); + h.deps.findOnPath = () => "/usr/bin/codex"; + h.deps.inspectShim = () => ({ status: "absent" }) as ReturnType>; + expect(await spawnCodex(["exec", "argument with spaces"], {}, h.deps)).toBe(7); + expect(h.calls[0].file).toBe("/usr/bin/codex"); + expect(h.calls[0].args).toEqual(["exec", "argument with spaces"]); +}); + +test("Windows CMD invocation keeps its interpreter options and launch environment", async () => { + const h = harness(async () => { throw new Error("not macOS"); }); + const command = "C:/test/codex.cmd"; + h.deps.findOnPath = () => command; + h.deps.inspectShim = () => ({ status: "absent" }) as ReturnType>; + h.deps.execBoundary = { platform: "win32", exists: () => true }; + const env = { ComSpec: "C:/Windows/System32/cmd.exe", PATH: "", PATHEXT: ".CMD;.EXE" }; + expect(await spawnCodex(["--version"], env, h.deps)).toBe(7); + expect(h.calls[0].file).toBe(env.ComSpec); + expect(h.calls[0].args.slice(0, 3)).toEqual(["/d", "/s", "/c"]); + expect(h.calls[0].options.windowsVerbatimArguments).toBe(true); +}); + +test("hidden validator emits only the trusted target and repeats the check", async () => { + let allowed = true, checks = 0; + const h = harness(async () => { checks++; return allowed ? null : "revoked"; }); + expect(await handleInternalCommand(["codex-exec-target", "--", alias], h.boundary)).toBe(0); + expect(logs.mock.calls).toEqual([[binary]]); + logs.mockClear(); allowed = false; + expect(await handleInternalCommand(["codex-exec-target", "--", alias], h.boundary)).toBe(1); + expect(logs.mock.calls).toHaveLength(0); + expect(checks).toBe(2); +}); + +test("hidden validator rejects malformed argv and line-oriented path ambiguity", async () => { + for (const args of [[], ["--"], ["--", ""], ["--", "a\nb"], ["--", "a\rb"], ["--", alias, "extra"]]) { + expect(await handleInternalCommand(["codex-exec-target", ...args])).toBe(2); + } + expect(logs.mock.calls).toHaveLength(0); +}); + +test("hidden validator refuses a canonical target containing a line break", async () => { + const h = harness(async () => null); + h.boundary.realpath = path => path === alias ? `${binary}\nextra` : path; + expect(await handleInternalCommand(["codex-exec-target", "--", alias], h.boundary)).toBe(1); + expect(logs.mock.calls).toHaveLength(0); +}); + +test("validator admission skips root shim repair even for malformed arguments", () => { + let reads = 0, restores = 0; + for (const suffix of [[], ["--", alias], ["--", alias, "extra"]]) { + maybeAutoRestoreCodexShim("internal", ["internal", "codex-exec-target", ...suffix], { + env: {}, warn: () => { throw new Error("unexpected repair warning"); }, + readConfig: () => { reads++; throw new Error("unexpected configuration read"); }, + restore: () => { restores++; throw new Error("unexpected shim repair"); }, + }); + } + expect(reads).toBe(0); expect(restores).toBe(0); + expect(skipsCodexShimAutoRestore("internal", ["internal", "chatgpt-app-server-filter"])).toBe(false); +}); diff --git a/tests/codex-integration/exec-renamed-app-bundle.test.ts b/tests/codex-integration/exec-renamed-app-bundle.test.ts new file mode 100644 index 00000000000..1225a317937 --- /dev/null +++ b/tests/codex-integration/exec-renamed-app-bundle.test.ts @@ -0,0 +1,164 @@ +import { afterEach, beforeEach, expect, spyOn, test } from "bun:test"; +import { handleInternalCommand } from "../../src/cli/internal-command"; +import { buildChatgptShimLauncher, writeChatgptShimLauncher, chatgptShimLauncherPath } from "../../src/chatgpt/app-server-shim/launcher"; +import { untrustedLauncherFilesystemReason, type BundleTrustDeps } from "../../src/chatgpt/app-server-shim/bundle-trust-policy"; +import { mkdtempSync, readFileSync, readdirSync, realpathSync, writeFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { tmpdir } from "node:os"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; + +const root = "/Volumes/Apps/ChatGPT Beta.app"; +const binary = `${root}/Contents/Resources/codex-cli/CodexCLI.app/Contents/MacOS/codex`; +let logs: ReturnType>; +let errors: ReturnType>; +beforeEach(() => { + logs = spyOn(console, "log").mockImplementation(() => {}); + errors = spyOn(console, "error").mockImplementation(() => {}); +}); +afterEach(() => { logs.mockRestore(); errors.mockRestore(); }); + +test("a recorded renamed bundle rechecks trust after revocation without a conventional alias", async () => { + let allowed = true; + const checked: string[][] = []; + const boundary = { + platform: "darwin" as const, executable: () => true, realpath: (path: string) => path, + expectedBundleRoot: root, + macBundleTrustReasonAsync: async (bundle: string, target: string) => { + checked.push([bundle, target]); + return allowed ? null : "signature revoked"; + }, + }; + expect(await handleInternalCommand(["codex-exec-target", "--", binary], boundary)).toBe(0); + expect(logs.mock.calls).toEqual([[binary]]); + logs.mockClear(); allowed = false; + expect(await handleInternalCommand(["codex-exec-target", "--", binary], boundary)).toBe(1); + expect(logs.mock.calls).toEqual([]); + expect(checked).toEqual([[root, binary], [root, binary]]); +}); + +test("launcher carries its outer renamed bundle root inline to the strict helper mode", () => { + const script = buildChatgptShimLauncher(["/filter"], binary, ["/ocx", "internal", "codex-exec-target"]); + expect(script).toContain(`BUNDLE_ROOT='${root}'`); + expect(script).toContain('--bundle-root "$BUNDLE_ROOT" -- "$REAL"'); +}); + +test("explicit bundle mode validates a renamed bundle and never accepts an empty root", async () => { + let checks = 0; + const boundary = { + platform: "darwin" as const, executable: () => true, realpath: (path: string) => path, + macBundleTrustReasonAsync: async (bundle: string, target: string) => { + checks++; expect(bundle).toBe(root); expect(target).toBe(binary); return null; + }, + }; + expect(await handleInternalCommand(["codex-exec-target", "--bundle-root", root, "--", binary], boundary)).toBe(0); + expect(checks).toBe(1); + for (const args of [ + ["--bundle-root", "", "--", binary], ["--bundle-root", root], + ["--bundle-root", root, "--", binary, "extra"], ["--bundle-root", `${root}\nextra`, "--", binary], + ]) expect(await handleInternalCommand(["codex-exec-target", ...args], boundary)).toBe(2); + expect(checks).toBe(1); +}); + +test("launcher helper argv rechecks the recorded root on a later respawn", async () => { + const args = ["codex-exec-target", "--bundle-root", root, "--", binary]; + let allowed = true, checks = 0; + const boundary = { + platform: "darwin" as const, executable: () => true, realpath: (path: string) => path, + macBundleTrustReasonAsync: async (bundle: string, target: string) => { + checks++; expect(bundle).toBe(root); expect(target).toBe(binary); + return allowed ? null : "signature revoked"; + }, + }; + expect(await handleInternalCommand(args, boundary)).toBe(0); + logs.mockClear(); allowed = false; + expect(await handleInternalCommand(args, boundary)).toBe(1); + expect(checks).toBe(2); expect(logs.mock.calls).toEqual([]); +}); + +for (const defect of ["missing root", "missing binary", "escape", "prefix sibling", "outside Contents"] as const) { + test(`recorded bundle refuses ${defect} before invoking trust`, async () => { + let checks = 0; + const boundary = { + platform: "darwin" as const, executable: () => true, + realpath: (path: string) => { + if (path === root && defect === "missing root" || path === binary && defect === "missing binary") return null; + if (path !== binary) return path; + return defect === "escape" ? "/opt/codex" : defect === "prefix sibling" ? `${root}-copy/Contents/codex` + : defect === "outside Contents" ? `${root}/codex` : path; + }, + macBundleTrustReasonAsync: async () => { checks++; return null; }, + }; + expect(await handleInternalCommand(["codex-exec-target", "--bundle-root", root, "--", binary], boundary)).toBe(1); + expect(checks).toBe(0); expect(logs.mock.calls).toEqual([]); + }); +} + +test("ordinary executable helper calls retain passthrough", async () => { + const target = "/opt/bin/codex"; + expect(await handleInternalCommand(["codex-exec-target", "--", target], { + platform: "darwin", executable: () => true, realpath: path => path, + macBundleTrustReasonAsync: async () => { throw new Error("ordinary CLI must not be checked"); }, + })).toBe(0); + expect(logs.mock.calls).toEqual([[target]]); +}); + +test("recorded bundle validation uses a consistent canonical root and target", async () => { + const canonicalRoot = "/private/Apps/ChatGPT Beta.app"; + const target = binary.replace(root, canonicalRoot); + expect(await handleInternalCommand(["codex-exec-target", "--bundle-root", root, "--", binary], { + platform: "darwin", executable: () => true, realpath: path => path.replace(root, canonicalRoot), + macBundleTrustReasonAsync: async (bundle, executable) => { + expect(bundle).toBe(canonicalRoot); expect(executable).toBe(target); return null; + }, + })).toBe(0); + expect(logs.mock.calls).toEqual([[target]]); +}); + +test("recorded bundle mode refuses on a host without macOS verification", async () => { + expect(await handleInternalCommand(["codex-exec-target", "--bundle-root", root, "--", binary], { platform: "linux" })).toBe(1); + expect(logs.mock.calls).toEqual([]); +}); + +for (const defect of ["none", "foreign file", "writable file", "link file", "foreign parent", "writable parent", "link parent"] as const) { + test(`inline root's launcher filesystem policy: ${defect}`, () => { + const path = "/owned/config/launcher.sh"; + const deps: Omit = { + uid: 501, + stat: entry => { + const bad = entry === (defect.endsWith("file") ? path : dirname(path)); + return { uid: bad && defect.startsWith("foreign") ? 502 : 501, + mode: bad && defect.startsWith("writable") ? 0o777 : 0o755, + isFile: entry === path, isDirectory: entry !== path, + isSymbolicLink: bad && defect.startsWith("link") }; + }, + }; + expect(untrustedLauncherFilesystemReason(path, deps) === null).toBe(defect === "none"); + }); +} + +for (const defect of ["none", "parent", "file"] as const) { + test(`writer protects recorded root before publication: ${defect}`, () => { + const dir = realpathSync(mkdtempSync(join(tmpdir(), "ocx-recorded-root-"))); + const launcher = chatgptShimLauncherPath(dir); + try { + writeFileSync(launcher, "original"); + const deps: Omit = { + uid: 501, + stat: entry => ({ uid: 501, + mode: defect === "parent" && entry === dir || defect === "file" && entry.endsWith(".tmp") ? 0o777 : 0o755, + isFile: entry.endsWith(".tmp"), isDirectory: !entry.endsWith(".tmp"), isSymbolicLink: false }), + }; + const write = () => writeChatgptShimLauncher(dir, binary, root, deps); + if (defect === "none") { + expect(write()).toBe(launcher); + const text = readFileSync(launcher, "utf8"); + expect(text).toContain(`BUNDLE_ROOT='${root}'`); + expect(text).toContain('--bundle-root "$BUNDLE_ROOT" -- "$REAL"'); + } else { + expect(write).toThrow("unsafe app-server launcher"); + expect(readFileSync(launcher, "utf8")).toBe("original"); + } + expect(readdirSync(dir)).toEqual(["chatgpt-codex-shim.sh"]); + } finally { removeTreeWithRetry(dir); } + }); +} diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index 3376f412bf1..be2098b4109 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -613,6 +613,11 @@ "codex-exec-invocation.test.ts": "codex-integration", "exec-trust-boundary.test.ts": "codex-integration", "exec-boundary-launch.test.ts": "codex-integration", + "exec-launcher-boundary.test.ts": "codex-integration", + "exec-cli-renamed-app-bundle.test.ts": "codex-integration", + "exec-cli-bundle-containment.test.ts": "codex-integration", + "exec-renamed-app-bundle.test.ts": "codex-integration", + "desktop-app-server-exec-boundary.test.ts": "clients", "exec-known-app-bundles.test.ts": "codex-integration", "messaging-local-discovery.test.ts": "codex-integration", "messaging-local-cli.test.ts": "codex-integration", From eea1454782be464848b66e8a9483c453c2189a97 Mon Sep 17 00:00:00 2001 From: Epinephrine Date: Fri, 30 Oct 2026 12:10:47 +0000 Subject: [PATCH 19/42] fix(credentials): stop Windows ancestor-watch feedback loops (#7976) * fix(credentials): stop Windows ancestor-watch feedback loops * test(credentials): settle writes before hardlink replacement * fix(credentials): fold watcher names the way the volume resolves them A case-insensitive volume reports a notification under the writer's spelling. With same-identity ancestor ticks now ignored, an event for AUTH.JSON no longer had an incidental reconcile path. Compare target, ancestor and residue names case-insensitively with trailing dots and spaces removed, and always reconcile an 8.3 short name. * fix(credentials): identity-check ancestors whose names look like short names An ancestor such as OPENCODEX_HOME=C:\ocx~1 took the unconditional short-name reconcile, so the observer's own save could re-arm the loop. Only an unmatched alias is ambiguous. * fix(credentials): identity-check ancestors whose names look like write residue OPENCODEX_HOME=C:\config.json.ocx.home matched the residue pattern before the ancestor lookup, so the observer's own save re-armed the reconcile. Residue and short-name shapes now only decide names that match no target or ancestor. * fix(credentials): resolve unmatched short names to tracked ancestors by identity A Windows 8.3 alias such as OPENC~1 for a tracked ancestor took the unconditional short-name reconcile, so the observer's own save could re-arm the loop. A change that resolves to a tracked ancestor's recorded directory is that ancestor's tick; any other short name still reconciles. * fix(credentials): resolve unmatched short names against the install snapshot Windows may report either name of an entry, and a removed entry's short name cannot be looked up. An unmatched 8.3-shaped name now reconciles only when it aliases a target, when a target or ancestor of that parent lost its install identity, or when it renames a tracked ancestor; the observer's own metadata save under a short spelling is ignored. * docs(structure): state the transient short-name residual of the credential observer --------- Co-authored-by: JUN <243035832+lidge-jun@users.noreply.github.com> --- src/local-credentials/generation.ts | 8 +- src/local-credentials/inventory.ts | 140 ++++++++- structure/remote-link.md | 2 +- .../local-credential-generation.test.ts | 296 +++++++++++++++++- 4 files changed, 434 insertions(+), 12 deletions(-) diff --git a/src/local-credentials/generation.ts b/src/local-credentials/generation.ts index 0da2acb199c..1c370840f52 100644 --- a/src/local-credentials/generation.ts +++ b/src/local-credentials/generation.ts @@ -212,8 +212,12 @@ export function claimCredentialObservation(home: string, scan: (accounts: readon }); } /** - * Advance on every observed event and rescan; a failed scan leaves an empty vector, which never - * attests. It never waits for the fence: the runtime observer retries asynchronously instead. + * Advance on every observed coverage-relevant event and rescan; a failed scan leaves an empty + * vector, which never attests. It never waits for the fence: the runtime observer retries + * asynchronously instead. "Coverage-relevant" is the watcher's classification in inventory.ts: + * every event on a target name, credential-write residue, or ancestor structural change -- + * but not an ancestor directory's last-write tick from ordinary child churn, which reports no + * credential change and would re-arm the observer's own save forever. */ export function updateCredentialObservation(home: string, owner: string, scan: ((accounts: readonly string[]) => string) | null): void { withCredentialPublicationFence(home, () => { diff --git a/src/local-credentials/inventory.ts b/src/local-credentials/inventory.ts index 2f817b38811..c7349f1a175 100644 --- a/src/local-credentials/inventory.ts +++ b/src/local-credentials/inventory.ts @@ -40,6 +40,121 @@ function assertParents(path: string): void { if (p === parse(p).root) break; } } +/** Watch classification for one parent directory, per basenames seen there. */ +export type WatchEntry = { + /** + * Basenames that are coverage targets themselves: a file whose bytes are scanned or a + * directory whose whole listing is checked. Every event on these is coverage-relevant + * -- an in-place content change reconciles exactly like a rename, even when the scan + * vector ends up identical, because the advance is what proves the observation ran. + */ + names: Set; + /** + * Basenames watched only as ancestors of a deeper target, paired with the identity + * recorded at install. A "rename" always reconciles: a replacement is the signal these + * entries exist to catch. A "change" reconciles too unless the entry re-identifies as + * the same real directory -- on Windows a directory entry reports "change" whenever a + * child is added or removed, so this observer's own local-credentials.json save would + * otherwise re-arm the same reconcile forever. null means the ancestor could not be + * identified at install, and an unverifiable ancestor reconciles on any event. + */ + ancestors: Map; + /** + * Identity of each target name at install (watchEntryIdentity). An unmatched 8.3 short + * name is resolved against these: it reconciles when a target here changed or when the + * alias is a target. Absent means the entry is treated as unresolvable and reconciles. + */ + targetIdentities?: Map; +}; +const CREDENTIAL_TMP_NAME = /^(auth|config|codex-accounts)\.json\.ocx\./i; +/** + * Watcher names are compared the way a case-insensitive volume resolves them. Windows and + * default macOS volumes report a notification under the spelling the writer used, so a + * credential created as AUTH.JSON reaches the watcher under that name while the scan reads + * it as auth.json. Folding can only add reconciles on a case-sensitive volume, never drop one. + */ +function foldWatchName(name: string): string { + return name.toLowerCase().replace(/[. ]+$/, ""); +} +/** + * An 8.3 short name (AUTH~1.JSO, OPENC~1) cannot be mapped back to its long name by + * spelling; it is resolved by identity instead (see shortNameAction). + */ +const SHORT_NAME_ALIAS = /~\d/; +/** ":", "absent", or "unreadable" for one entry of a watched parent. */ +export function watchEntryIdentity(parent: string, child: string): string { + try { + const stat = lstatSync(join(parent, child), { bigint: true }); + return `${stat.dev}:${stat.ino}`; + } catch (error) { + return (error as NodeJS.ErrnoException).code === "ENOENT" ? "absent" : "unreadable"; + } +} +/** + * Windows may report either name of an entry that has both, and a removed entry's short + * name can no longer be looked up. Such an event is coverage-relevant when a target or + * ancestor in this parent no longer has its install identity, or when the alias resolves to + * a target; a change that resolves to a tracked ancestor is that ancestor's tick. Anything + * else -- this observer's own metadata save under a short spelling -- changed no target. + */ +function shortNameAction(parent: string, entry: WatchEntry, event: string, name: string): WatchEventAction { + const recorded = entry.targetIdentities; + if (!recorded) return "reconcile"; + const resolved = watchEntryIdentity(parent, name); + if (resolved === "unreadable") return "reconcile"; + for (const [target, identity] of recorded) { + const now = watchEntryIdentity(parent, target); + if (now === "unreadable" || now !== identity) return "reconcile"; + if (resolved !== "absent" && resolved === now) return "reconcile"; + } + let aliasOfAncestor = false; + for (const [ancestorName, identity] of entry.ancestors) { + const now = ancestorIdentity(parent, ancestorName); + if (identity === null || now === null || now.dev !== identity.dev || now.ino !== identity.ino) return "reconcile"; + if (resolved === `${identity.dev}:${identity.ino}`) aliasOfAncestor = true; + } + return aliasOfAncestor && event !== "change" ? "reconcile" : "ignore"; +} +function foldedLookup(keys: Iterable, key: (item: T) => string, folded: string): T | undefined { + for (const item of keys) if (foldWatchName(key(item)) === folded) return item; + return undefined; +} +function ancestorIdentity(parent: string, child: string): { dev: bigint; ino: bigint } | null { + try { + const stat = lstatSync(join(parent, child), { bigint: true }); + return stat.isDirectory() && !stat.isSymbolicLink() ? { dev: stat.dev, ino: stat.ino } : null; + } catch { return null; } +} +type WatchEventAction = "fail" | "reconcile" | "ignore"; +/** + * The single (parent, event, name) decision behind the observation watchers, exported so + * tests can pin every branch without timing-dependent filesystem notifications. + */ +export function credentialWatchEventAction( + parent: string, entry: WatchEntry, event: string, name: string | null, +): WatchEventAction { + if (name === null) return "fail"; + const folded = foldWatchName(name); + if (foldedLookup(entry.names, n => n, folded) !== undefined) return "reconcile"; + const match = foldedLookup(entry.ancestors.entries(), ([n]) => n, folded); + // A known ancestor whose real name merely looks like write residue or a short name + // (C:\config.json.ocx.home, C:\ocx~1) goes through the identity check below. + if (match === undefined) { + if (CREDENTIAL_TMP_NAME.test(name)) return "reconcile"; + return SHORT_NAME_ALIAS.test(name) ? shortNameAction(parent, entry, event, name) : "ignore"; + } + const ancestor = match[1]; + // Renames (create/delete/move, including a same-identity rename) are structural and + // always reconcile; only "change" may be child-list noise from a same-identity dir. + if (event !== "change" || ancestor === null) return "reconcile"; + try { + const stat = lstatSync(join(parent, name), { bigint: true }); + if (stat.isDirectory() && !stat.isSymbolicLink() && stat.dev === ancestor.dev && stat.ino === ancestor.ino) { + return "ignore"; + } + } catch { /* an ancestor that cannot be re-identified cannot prove the tick is noise */ } + return "reconcile"; +} export function scanLocalCredentials(accounts: readonly string[] = []): { count: number; vector: string; targets: string[] } { const home = getConfigDir(); const codex = getCodexHome(); @@ -166,6 +281,10 @@ export function scanLocalCredentials(accounts: readonly string[] = []): { count: const profiles = countLocalNativeProfileSlots(vault, homeId); if (profiles.active && !main) throw new Error("Active native profile has no readable auth"); count += main + profiles.inactive; + // The whole listing is verified below, so the directory itself is a coverage target: + // an unknown create/delete must reach a watcher even when the child's own name is not. + targets.add(shared); + assertParents(shared); if (exists(shared)) { if (!lstatSync(shared).isDirectory() || lstatSync(shared).isSymbolicLink()) throw new Error("Unsafe native vault"); const expected = new Set([`${homeId}.vault.json`, `${homeId}.stages.json`]); @@ -223,23 +342,28 @@ export function startLocalCredentialObservation(): { stop(): void; reconcile(): } }; const install = (targets: string[]) => { - const watched = new Map>(); + const watched = new Map(); for (const target of targets) { + let leaf = true; for (let path = target; dirname(path) !== path; path = dirname(path)) { const parent = dirname(path); if (!exists(parent)) continue; assertParents(path); - const names = watched.get(parent) ?? new Set(); - names.add(path.slice(parent.length + (parent.endsWith("/") || parent.endsWith("\\") ? 0 : 1))); - watched.set(parent, names); + const entry = watched.get(parent) ?? { names: new Set(), ancestors: new Map(), targetIdentities: new Map() }; + const child = path.slice(parent.length + (parent.endsWith("/") || parent.endsWith("\\") ? 0 : 1)); + if (leaf) { entry.names.add(child); entry.targetIdentities?.set(child, watchEntryIdentity(parent, child)); } + else if (!entry.ancestors.has(child)) entry.ancestors.set(child, ancestorIdentity(parent, child)); + watched.set(parent, entry); + leaf = false; } } const next: FSWatcher[] = []; try { - for (const [parent, names] of watched) { - const watcher = watch(parent, { persistent: false }, (_event, name) => { - if (name === null) { fail(); return; } - if (names.has(String(name)) || /^(auth|config|codex-accounts)\.json\.ocx\./.test(String(name))) reconcile(); + for (const [parent, entry] of watched) { + const watcher = watch(parent, { persistent: false }, (event, name) => { + const action = credentialWatchEventAction(parent, entry, event, name === null ? null : String(name)); + if (action === "fail") fail(); + else if (action === "reconcile") reconcile(); }); watcher.on("error", fail); next.push(watcher); diff --git a/structure/remote-link.md b/structure/remote-link.md index f1b94aebbed..44f0dcb2004 100644 --- a/structure/remote-link.md +++ b/structure/remote-link.md @@ -119,7 +119,7 @@ The explicit `ocx link revoke --force --yes` branch in `src/cli/link.ts` sends o `ocx status --json` reports `localCredentials: {count, generation}` from `src/local-credentials/` on any runtime. `inventory.ts` counts this installation's stored provider credentials passively: OAuth accounts and the pre-multiauth backup, provider API-key slots including disabled entries and keychain, environment and GCP references, Codex pool records, the native Codex `auth.json` and inactive native profiles. It never counts link, management or incoming API keys, so a Child that holds only its link key reports zero, and accounts its Home serves are never local. A link-mode Child relays every model request to its Home and never dispatches to a provider itself, so a provider that needs no stored credential cannot serve a Child request and has nothing to count. An unreadable or unfamiliar source, a write residue, a pending native-profile transaction or a native Codex credential store other than `file` throws, and status omits the field instead of reporting zero. -`generation.ts` keeps `local-credentials.json` (epoch, sequence, pending tickets, touched keychain accounts and the observer's file-identity vector; never a credential value). Instrumented writers publish a ticket before credential bytes change and advance the sequence afterwards: `src/oauth/store.ts`, `src/codex/account-store.ts`, `src/config/persist-unlocked.ts`, `src/codex/main-account.ts`, the native-profile writers and the provider keychain factory in `src/providers/api-key-resolve.ts`. The token `lc1::` therefore changes on every add, replacement or removal, including one that ends at the same count. Every instrumented publication runs inside the credential-publication fence (`publication-fence.ts`), the OS lock of `local-credential-publication.sqlite`, separate from the config mutation lock: a writer holds it exclusively across its ticket, the write and the ticket's close, and fails if it cannot take it within ten seconds. Once coordination metadata exists, a publication whose ticket cannot be persisted (unreadable or unwritable metadata) aborts before it changes any credential bytes, so a completed write can never leave an earlier token attestable. Status holds it shared across both generation reads and the source scan, retries briefly while a publication holds it, and otherwise omits the field, so no snapshot can interleave with an instrumented write. Ticket and observer metadata are updated only under the fence, so contention on the config lock cannot drop a ticket. OS-store accounts are retained the same way: the observer seeds every `keychain:` and `gcp-sa:` account `config.json` references at start and on every reconcile, config persistence registers the references on both sides of a write before it lands (`references.ts`), and a keychain write or deletion registers its account before it is attempted. An account leaves the count only after a read proves it absent, so a secret whose config slot was removed while its deletion failed or never ran keeps counting. A proven absence is remembered in the metadata: a reference that still names that account is not registered again (it counts only as a configured slot while it stays in config), and only a new keychain write or deletion attempt for the account clears the mark. `restoreProviderKeyFromKeychain` in `src/providers/key-store.ts` saves the plaintext config before it deletes the OS entries; the other deletions remove secrets the config never referenced or run after the removal is committed. `runtime-observation.ts` gives `startServer` and `startClientRuntime` the same observer, which takes an OS lease (`observation-lease.ts`), watches the sources, rescans on every event and starts a new epoch at each runtime start; the client runtime releases it before a restart hands over to a replacement. Status attests only while that lease is held, no ticket is pending and the vector matches a fresh scan, so a stopped runtime, a lost watcher or an in-flight write omits the field. +`generation.ts` keeps `local-credentials.json` (epoch, sequence, pending tickets, touched keychain accounts and the observer's file-identity vector; never a credential value). Instrumented writers publish a ticket before credential bytes change and advance the sequence afterwards: `src/oauth/store.ts`, `src/codex/account-store.ts`, `src/config/persist-unlocked.ts`, `src/codex/main-account.ts`, the native-profile writers and the provider keychain factory in `src/providers/api-key-resolve.ts`. The token `lc1::` therefore changes on every add, replacement or removal, including one that ends at the same count. Every instrumented publication runs inside the credential-publication fence (`publication-fence.ts`), the OS lock of `local-credential-publication.sqlite`, separate from the config mutation lock: a writer holds it exclusively across its ticket, the write and the ticket's close, and fails if it cannot take it within ten seconds. Once coordination metadata exists, a publication whose ticket cannot be persisted (unreadable or unwritable metadata) aborts before it changes any credential bytes, so a completed write can never leave an earlier token attestable. Status holds it shared across both generation reads and the source scan, retries briefly while a publication holds it, and otherwise omits the field, so no snapshot can interleave with an instrumented write. Ticket and observer metadata are updated only under the fence, so contention on the config lock cannot drop a ticket. OS-store accounts are retained the same way: the observer seeds every `keychain:` and `gcp-sa:` account `config.json` references at start and on every reconcile, config persistence registers the references on both sides of a write before it lands (`references.ts`), and a keychain write or deletion registers its account before it is attempted. An account leaves the count only after a read proves it absent, so a secret whose config slot was removed while its deletion failed or never ran keeps counting. A proven absence is remembered in the metadata: a reference that still names that account is not registered again (it counts only as a configured slot while it stays in config), and only a new keychain write or deletion attempt for the account clears the mark. `restoreProviderKeyFromKeychain` in `src/providers/key-store.ts` saves the plaintext config before it deletes the OS entries; the other deletions remove secrets the config never referenced or run after the removal is committed. `runtime-observation.ts` gives `startServer` and `startClientRuntime` the same observer, which takes an OS lease (`observation-lease.ts`), watches the sources, rescans on every event and starts a new epoch at each runtime start; the client runtime releases it before a restart hands over to a replacement. Status attests only while that lease is held, no ticket is pending and the vector matches a fresh scan, so a stopped runtime, a lost watcher or an in-flight write omits the field. "Every event" above means every coverage-relevant event, classified per (watched parent, event, entry name) by `credentialWatchEventAction` in `inventory.ts`: an event on a scanned file, on a whole-listing-verified directory such as `.opencodex-native-main-profiles`, or on credential-write residue, and a `rename` on a tracked target or ancestor entry, always reconcile, while an ancestor directory's `change` is skipped only when the entry re-identifies as the same real directory recorded at install -- ordinary child churn, including this observer's own save, would otherwise re-arm the reconcile forever. Entry names are compared case-insensitively with trailing dots and spaces removed, as Windows and default macOS volumes resolve them, and a real ancestor whose name looks like a short name or like write residue keeps its identity check. A name that matches no target or ancestor but looks like an 8.3 short name (a tilde followed by a digit; Windows may report either name of an entry, and a removed entry's short name cannot be looked up) is resolved by identity against the install snapshot: it reconciles when the alias is a target, when any target or ancestor of that parent no longer has its install identity, or when it renames a tracked ancestor, and is otherwise ignored, so the observer's own save under a short spelling cannot re-arm the loop. The residual is transient: an add-then-remove of a target, or a rename-and-restore of an ancestor, that completes before its notifications are handled and whose notifications carry only short names that no longer resolve leaves the generation where it was. A differently spelled event on a credential path therefore never falls through as unrelated. A missing, replaced, symlinked or unidentifiable ancestor still reconciles, and a watcher error or a null event name fails the observation closed. The vault directory's target status means a child event inside it reaches the observer wherever the platform delivers one -- Windows reports the parent's last-write tick for child churn, while other platforms may deliver no event for the pair -- but the fail-closed part is platform-common: a fresh scan still throws on an unknown entry and status omits the attestation while it remains. There is no debounce, coalescing or scan-vector shortcut: every coverage-relevant event advances the generation even when the rescan is unchanged, because the advance is what proves the observation ran. A link-mode Child reports its current tunnel connection as `link.connectionGeneration` in a ready `/readyz`; `src/cli/status-link.ts` reads it from the live client runtime and omits it whenever that runtime does not answer ready with a valid value. Names have the form `:` and come from one process-wide sequence (`nextLinkConnectionGeneration` in `src/link/relay-auth.ts`), so a supervisor rebuilt in the same process, such as after a restart rollback, never repeats one. A Child-initiated link takes a new name from its supervisor on every transition into connected. For a Home-initiated link the Home supervisor names each ssh child it spawns, and its relay-auth answer carries that name with its own proof over the nonce, key and link ids (`linkConnectionProof` in `src/link/relay-auth.ts`), and the Child keeps the value from the latest authenticated answer (`src/client/link-connection.ts`). Every connect and every runtime restart changes it. An authenticated answer without a generation, or with one whose proof does not match, makes it unknown, so a Child that reconnects to an older Home never keeps reporting the previous connection. GOAT Ready requires client role, zero local credentials and equal generations before and after a completed Home inference; linked and healthy alone do not prove attribution. Coverage: `tests/cli/local-credentials.test.ts`, `tests/config/local-credential-generation.test.ts`, `tests/clients/link-connection-generation.test.ts`, `tests/clients/client-link-tunnel.test.ts`, `tests/clients/link-readiness.test.ts`. diff --git a/tests/config/local-credential-generation.test.ts b/tests/config/local-credential-generation.test.ts index 9eab3780211..135f669a358 100644 --- a/tests/config/local-credential-generation.test.ts +++ b/tests/config/local-credential-generation.test.ts @@ -1,7 +1,8 @@ import { expect, test } from "bun:test"; import { spawnSync } from "node:child_process"; import { join } from "node:path"; -import { chmodSync, existsSync, readFileSync, unlinkSync } from "node:fs"; +import { chmodSync, existsSync, linkSync, lstatSync, mkdirSync, mkdtempSync, readFileSync, renameSync, rmSync, symlinkSync, unlinkSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; import { withLocalCredentialFixture } from "../helpers/local-credentials"; const accountStoreUrl = new URL("../../src/codex/account-store.ts", import.meta.url).href; @@ -294,3 +295,296 @@ for (const observedFirst of [true, false]) { }); }); } + +async function generationOf(inventory: typeof import("../../src/local-credentials/inventory")): Promise { + return (await inventory.collectLocalCredentials())?.generation; +} +async function waitForNewGeneration( + inventory: typeof import("../../src/local-credentials/inventory"), + previous: string | undefined, + timeoutMs = 4000, +): Promise { + const deadline = Date.now() + timeoutMs; + let next: string | undefined; + do { + next = await generationOf(inventory); + if (next !== undefined && next !== previous) return next; + if (Date.now() >= deadline) return next; + await new Promise(resolve => setTimeout(resolve, 40)); + } while (true); +} +async function expectStableGeneration( + inventory: typeof import("../../src/local-credentials/inventory"), + previous: string | undefined, + windowMs = 1500, +): Promise { + const deadline = Date.now() + windowMs; + while (Date.now() < deadline) { + await new Promise(resolve => setTimeout(resolve, 60)); + expect(await generationOf(inventory)).toBe(previous); + } +} + +// Waits until the generation stays constant over one quiet window, tolerating +// residual notifications from the mutation that produced it. Throws rather than +// return an unsettled token, so a caller can never start the ABA on a baseline +// that was still racing events. +async function settleGeneration( + inventory: typeof import("../../src/local-credentials/inventory"), + quietMs = 500, + timeoutMs = 4000, +): Promise { + const deadline = Date.now() + timeoutMs; + let last = await generationOf(inventory); + let quietSince = Date.now(); + while (Date.now() < deadline) { + await new Promise(resolve => setTimeout(resolve, 50)); + const next = await generationOf(inventory); + if (next !== last) { last = next; quietSince = Date.now(); continue; } + if (Date.now() - quietSince >= quietMs) return last; + } + throw new Error("credential generation never settled: notifications still arriving at timeout"); +} + +test("the watch seam classifies direct, ancestor, identity, and failure cases deterministically", async () => { + const inventory = await import("../../src/local-credentials/inventory"); + const dir = mkdtempSync(join(tmpdir(), "ocx-watch-seam-")); + try { + const ancestor = join(dir, "ancestor"); + const direct = join(dir, "direct"); + const foreign = join(dir, "foreign"); + mkdirSync(ancestor); mkdirSync(direct); mkdirSync(foreign); + const shortLooking = join(dir, "ocx~1"); + mkdirSync(shortLooking); + const residueLooking = join(dir, "config.json.ocx.home"); + mkdirSync(residueLooking); + writeFileSync(join(dir, "auth.json"), "{}"); + const same = lstatSync(ancestor, { bigint: true }); + const shortSame = lstatSync(shortLooking, { bigint: true }); + const residueSame = lstatSync(residueLooking, { bigint: true }); + const entry = { + names: new Set(["auth.json", "direct"]), + ancestors: new Map([ + ["ancestor", { dev: same.dev, ino: same.ino }], + ["ocx~1", { dev: shortSame.dev, ino: shortSame.ino }], + ["config.json.ocx.home", { dev: residueSame.dev, ino: residueSame.ino }], + ["unverifiable", null], + ]), + }; + const action = (event: string, name: string | null) => + inventory.credentialWatchEventAction(dir, entry, event, name); + + // A same-identity ancestor reporting only a last-write tick is ordinary child + // churn, however often it repeats: no reconcile, no save, no re-arm. + expect(action("change", "ancestor")).toBe("ignore"); + expect(action("change", "ancestor")).toBe("ignore"); + // A rename is structural even when the entry re-identifies the same afterwards. + expect(action("rename", "ancestor")).toBe("reconcile"); + // Direct names reconcile on any event, including a plain content change. + expect(action("change", "auth.json")).toBe("reconcile"); + expect(action("rename", "auth.json")).toBe("reconcile"); + expect(action("change", "direct")).toBe("reconcile"); + // A name carrying both roles stays direct: direct wins over ancestor. + const overlapping = { names: new Set(["ancestor"]), ancestors: entry.ancestors }; + expect(inventory.credentialWatchEventAction(dir, overlapping, "change", "ancestor")).toBe("reconcile"); + // Credential-write residue is always coverage-relevant. + expect(action("change", "config.json.ocx.7.tmp")).toBe("reconcile"); + // An ancestor that could not be identified at install reconciles on any event. + expect(action("change", "unverifiable")).toBe("reconcile"); + expect(action("rename", "unverifiable")).toBe("reconcile"); + // Unrelated names never reconcile. + expect(action("change", "foreign")).toBe("ignore"); + expect(action("rename", "unknown.tmp")).toBe("ignore"); + // A case-insensitive volume reports the writer's spelling: a differently cased or + // trailing-dot spelling of a target or its write residue must still reconcile rather + // than fall through as an unrelated name. + expect(action("rename", "AUTH.JSON")).toBe("reconcile"); + expect(action("change", "Auth.Json")).toBe("reconcile"); + expect(action("change", "auth.json.")).toBe("reconcile"); + expect(action("rename", "Direct")).toBe("reconcile"); + expect(action("rename", "CONFIG.JSON.OCX.7.tmp")).toBe("reconcile"); + // An entry without recorded target identities cannot resolve a short name: reconcile. + expect(action("rename", "AUTH~1.JSO")).toBe("reconcile"); + // A real ancestor whose name only looks like a short name (OPENCODEX_HOME=C:\ocx~1) is + // identity-checked like any other ancestor, so its own-save tick cannot re-arm the loop. + expect(action("change", "ocx~1")).toBe("ignore"); + expect(action("rename", "ocx~1")).toBe("reconcile"); + expect(action("change", "config.json.ocx.home")).toBe("ignore"); + expect(action("rename", "config.json.ocx.home")).toBe("reconcile"); + // A differently cased ancestor is still an ancestor: renames reconcile, and a change is + // ignored only when that spelling resolves to the recorded directory itself. + expect(action("rename", "ANCESTOR")).toBe("reconcile"); + const upper = (() => { try { return lstatSync(join(dir, "ANCESTOR"), { bigint: true }); } catch { return null; } })(); + const sameDir = upper !== null && upper.dev === same.dev && upper.ino === same.ino; + expect(action("change", "ANCESTOR")).toBe(sameDir ? "ignore" : "reconcile"); + // A different real directory at the same name: the original is kept aside so + // its inode stays allocated and the replacement provably has a new identity. + renameSync(ancestor, join(dir, "ancestor-aside")); + mkdirSync(ancestor); + expect(action("change", "ancestor")).toBe("reconcile"); + // Missing, and replaced by a link, cannot prove the tick is noise either. + rmSync(ancestor, { recursive: true, force: true }); + expect(action("change", "ancestor")).toBe("reconcile"); + symlinkSync(foreign, ancestor, "junction"); + expect(action("change", "ancestor")).toBe("reconcile"); + unlinkSync(ancestor); + rmSync(join(dir, "ancestor-aside"), { recursive: true, force: true }); + // A watcher overflow reports a null name and must fail closed. + expect(action("change", null)).toBe("fail"); + expect(action("rename", null)).toBe("fail"); + } finally { + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("unmatched 8.3 short names resolve by identity against the install snapshot", async () => { + const inventory = await import("../../src/local-credentials/inventory"); + const dir = mkdtempSync(join(tmpdir(), "ocx-watch-short-")); + try { + // Literal tilde names stand in for the short spellings Windows may report for files. + mkdirSync(join(dir, "longancestor")); + writeFileSync(join(dir, "auth.json"), "{}"); + writeFileSync(join(dir, "localc~1.jso"), "{}"); + const ancestorStat = lstatSync(join(dir, "longancestor"), { bigint: true }); + const entry = { + names: new Set(["auth.json", "config.json"]), + ancestors: new Map([ + ["longancestor", { dev: ancestorStat.dev, ino: ancestorStat.ino }], + ]), + targetIdentities: new Map([ + ["auth.json", inventory.watchEntryIdentity(dir, "auth.json")], + ["config.json", inventory.watchEntryIdentity(dir, "config.json")], + ]), + }; + expect(entry.targetIdentities.get("config.json")).toBe("absent"); + const action = (event: string, name: string) => inventory.credentialWatchEventAction(dir, entry, event, name); + // The observer's own metadata save under a short spelling, live or already renamed + // away, changed no target or ancestor, so it must not re-arm the reconcile. + expect(action("change", "localc~1.jso")).toBe("ignore"); + expect(action("rename", "localc~2.tmp")).toBe("ignore"); + // A short alias of a tracked ancestor: its change is that ancestor's tick, a rename is + // structural. Whether LONGAN~1 resolves depends on the volume's 8.3 setting; either way + // the change cannot re-arm the loop. + const alias = (() => { try { return lstatSync(join(dir, "LONGAN~1"), { bigint: true }); } catch { return null; } })(); + const resolvesToAncestor = alias !== null && alias.dev === ancestorStat.dev && alias.ino === ancestorStat.ino; + expect(action("change", "LONGAN~1")).toBe("ignore"); + expect(action("rename", "LONGAN~1")).toBe(resolvesToAncestor ? "reconcile" : "ignore"); + // A short alias that resolves to a target is that target. + linkSync(join(dir, "auth.json"), join(dir, "authjs~1.jso")); + expect(action("change", "authjs~1.jso")).toBe("reconcile"); + unlinkSync(join(dir, "authjs~1.jso")); + // A removed or newly created target makes any unresolvable short name relevant. + writeFileSync(join(dir, "config.json"), "{}"); + expect(action("rename", "config~1.jso")).toBe("reconcile"); + unlinkSync(join(dir, "config.json")); + expect(action("rename", "config~1.jso")).toBe("ignore"); + unlinkSync(join(dir, "auth.json")); + expect(action("rename", "auth~1.jso")).toBe("reconcile"); + // A replaced ancestor does too. + writeFileSync(join(dir, "auth.json"), "{}"); + entry.targetIdentities.set("auth.json", inventory.watchEntryIdentity(dir, "auth.json")); + expect(action("rename", "other~1")).toBe("ignore"); + renameSync(join(dir, "longancestor"), join(dir, "aside")); + expect(action("change", "LONGAN~1")).toBe("reconcile"); + } finally { + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("ordinary churn inside a watched ancestor does not re-arm the observer or advance the generation", async () => { + await withLocalCredentialFixture(async f => { + f.start(); + const before = await generationOf(f.inventory); + expect(before).toBeDefined(); + // Writes like runtime-port.json, journals, and this observer's own save tick the + // ancestor's last-write time on Windows; that tick must not re-arm a reconcile. + for (let i = 0; i < 6; i += 1) { + writeFileSync(join(f.home, `runtime-${i}.json`), "{}"); + if (i % 2 === 1) unlinkSync(join(f.home, `runtime-${i - 1}.json`)); + } + await expectStableGeneration(f.inventory, before); + // Suppression must not deafen the observer: a real target write still reconciles. + f.put(join(f.home, "config.json"), { model: "fixture" }); + const revived = await waitForNewGeneration(f.inventory, before); + expect(revived).toBeDefined(); + expect(revived).not.toBe(before); + }); +}); + +test("a same-identity rename of a watched ancestor still advances the generation", async () => { + await withLocalCredentialFixture(async f => { + f.start(); + const before = await generationOf(f.inventory); + expect(before).toBeDefined(); + const moved = `${f.codex}-moved`; + renameSync(f.codex, moved); + renameSync(moved, f.codex); + const after = await waitForNewGeneration(f.inventory, before); + expect(after).toBeDefined(); + expect(after).not.toBe(before); + }); +}); + +test("a direct target replacing itself with identical bytes still advances the generation", async () => { + await withLocalCredentialFixture(async f => { + f.start(); + const before = await generationOf(f.inventory); + expect(before).toBeDefined(); + const auth = join(f.codex, "auth.json"); + const shadow = join(f.codex, "auth-shadow.tmp"); + f.put(auth, { OPENAI_API_KEY: "fixture-native-key" }); + const writeObserved = await waitForNewGeneration(f.inventory, before); + expect(writeObserved).toBeDefined(); + const settled = await settleGeneration(f.inventory); + expect(settled).toBeDefined(); + // Rebind the directory entry to the same inode: the bytes stay identical, so + // only the observed event itself -- never content -- may advance coverage. + linkSync(auth, shadow); + unlinkSync(auth); + renameSync(shadow, auth); + const after = await waitForNewGeneration(f.inventory, settled); + expect(after).toBeDefined(); + expect(after).not.toBe(settled); + expect(readFileSync(auth, "utf8")).toBe(JSON.stringify({ OPENAI_API_KEY: "fixture-native-key" })); + }); +}); + +test("an unknown entry in the shared native-profile vault still reaches the observer", async () => { + await withLocalCredentialFixture(async f => { + const shared = join(f.codex, ".opencodex-native-main-profiles"); + // The vault listing itself is verified, so the directory must be a direct watch + // target; otherwise an unknown create/delete pair hides behind an ancestor tick. + expect(f.inventory.scanLocalCredentials().targets).toContain(shared); + f.start(); + const before = await generationOf(f.inventory); + expect(before).toBeDefined(); + mkdirSync(shared); + const created = await waitForNewGeneration(f.inventory, before); + expect(created).toBeDefined(); + // Let residual notifications from the mkdir drain: the ABA below must be + // measured against a settled token, not one still racing earlier events. + const settled = await settleGeneration(f.inventory); + expect(settled).toBeDefined(); + const vectorBefore = f.inventory.scanLocalCredentials().vector; + const unknown = join(shared, "unknown.json"); + writeFileSync(unknown, "{}"); + // Coverage fails closed while the entry is present: no attestation for an + // unverifiable vault listing. + expect(await f.inventory.collectLocalCredentials()).toBeUndefined(); + unlinkSync(unknown); + // The ABA pair leaves the vault listing -- and therefore the scan vector -- + // exactly as it was, so any advance can only come from observed events. + expect(f.inventory.scanLocalCredentials().vector).toBe(vectorBefore); + if (process.platform === "win32") { + // On Windows the pair ticks the vault's parent and is itself observed, so + // the previous token cannot be reused. Platforms that report no event for + // this pair cannot force the advance, so this is a win32-only check, not a + // platform-common verification. + const after = await waitForNewGeneration(f.inventory, settled); + expect(after).toBeDefined(); + expect(after).not.toBe(settled); + } else { + expect(await generationOf(f.inventory)).toBeDefined(); + } + }); +}); From 1eca7a32938507706321c10c8e9b7988fb5c525e Mon Sep 17 00:00:00 2001 From: JUN Date: Fri, 30 Oct 2026 21:20:05 +0900 Subject: [PATCH 20/42] fix(discovery): name the invalid model row and the static-list workaround (#7990) * fix(discovery): name the invalid model row and the static-list workaround Invalid upstream model rows still reject the whole discovery response, but the diagnostic now names the failing row index and reason without its contents. The catalog warning and the connection-test error explain that a new provider's selection stays pending and point at a static models list with liveModels: false. * fix(discovery): qualify the pending-selection hint for login connections * docs(structure): scope the invalid-row diagnostic to generic discovery --- src/codex/catalog/provider-models.ts | 3 +- src/providers/model-discovery-diagnostic.ts | 4 +++ src/providers/model-discovery.ts | 8 ++--- src/server/management/provider-routes.ts | 3 +- structure/providers-and-adapters.md | 2 +- .../anthropic2-discovery-ownership.test.ts | 12 ++++++++ .../provider-model-discovery-contract.test.ts | 29 ++++++++++++++++++- 7 files changed, 53 insertions(+), 8 deletions(-) create mode 100644 src/providers/model-discovery-diagnostic.ts diff --git a/src/codex/catalog/provider-models.ts b/src/codex/catalog/provider-models.ts index 04238755718..877e6b275b9 100644 --- a/src/codex/catalog/provider-models.ts +++ b/src/codex/catalog/provider-models.ts @@ -89,6 +89,7 @@ import { providerRedirectError, } from "../../lib/provider-outbound"; import { redactSecretString } from "../../lib/redact"; +import { invalidModelDiscoveryHint } from "../../providers/model-discovery-diagnostic"; import { extractProviderModelItems, isRegistryModelDiscoveryUrl, @@ -857,7 +858,7 @@ export async function fetchProviderModelsWithAuth( }; if (shouldLog) { console.warn( - `[opencodex] Provider model discovery for "${name}" ${diagnostic[extracted.reason]} [status=${res.status}, contentType=${contentType}, urlClass=${urlClass}, fallback=${fallback}].`, + `[opencodex] Provider model discovery for "${name}" ${diagnostic[extracted.reason]} [status=${res.status}, contentType=${contentType}, urlClass=${urlClass}, fallback=${fallback}]${extracted.reason === "invalid_shape" ? invalidModelDiscoveryHint("diagnostic" in extracted ? extracted.diagnostic : undefined) : "."}`, ); } return observed(models, "degraded"); diff --git a/src/providers/model-discovery-diagnostic.ts b/src/providers/model-discovery-diagnostic.ts new file mode 100644 index 00000000000..ea3623f0c84 --- /dev/null +++ b/src/providers/model-discovery-diagnostic.ts @@ -0,0 +1,4 @@ +/** Safe operator guidance for malformed upstream model lists. */ +export function invalidModelDiscoveryHint(diagnostic?: string): string { + return `${diagnostic ? ` (${diagnostic})` : ""}. A newly added API-key provider can keep its model selection pending while discovery fails; a static models list with liveModels: false avoids discovery for gateways publishing invalid rows.`; +} diff --git a/src/providers/model-discovery.ts b/src/providers/model-discovery.ts index c59c3df7e23..49065c4d74b 100644 --- a/src/providers/model-discovery.ts +++ b/src/providers/model-discovery.ts @@ -47,7 +47,7 @@ export type BoundedDiscoveryJsonResult = export type ProviderModelItemsResult = | { ok: true; items: ProviderModelsApiItem[]; rawCount: number } - | { ok: false; reason: "invalid_shape" | "too_many_models" }; + | { ok: false; reason: "invalid_shape" | "too_many_models"; diagnostic?: string }; export type ModelEnvelopeRowsResult = | { ok: true; rows: unknown[] } @@ -534,12 +534,12 @@ export function extractProviderModelItems( const preferred: ProviderModelsApiItem[] = []; const seen = new Set(); const idField = discovery.spec?.idField ?? "id"; - for (const raw of data) { + for (const [index, raw] of data.entries()) { if (raw === null || typeof raw !== "object" || Array.isArray(raw)) { - return { ok: false, reason: "invalid_shape" }; + return { ok: false, reason: "invalid_shape", diagnostic: `row ${index} is not an object` }; } const id = (raw as Record)[idField]; - if (!isValidModelDiscoveryModelId(id)) return { ok: false, reason: "invalid_shape" }; + if (!isValidModelDiscoveryModelId(id)) return { ok: false, reason: "invalid_shape", diagnostic: `row ${index} has ${typeof id !== "string" ? "a non-string id" : !id.trim() ? "an empty id" : "an invalid id"}` }; const prefix = discovery.spec?.stripIdPrefix; let finalId = id; if (prefix && finalId.startsWith(prefix)) { diff --git a/src/server/management/provider-routes.ts b/src/server/management/provider-routes.ts index 6e4826a959d..84d0e1752b5 100644 --- a/src/server/management/provider-routes.ts +++ b/src/server/management/provider-routes.ts @@ -63,6 +63,7 @@ import { import { shadowInterceptProviderDependency } from "./shadow-call-validation"; import { deriveProviderPresets, providerConfigSeed } from "../../providers/derive"; import { initializeProviderModelSelection } from "../../providers/initial-model-selection"; +import { invalidModelDiscoveryHint } from "../../providers/model-discovery-diagnostic"; import { effectiveGoogleMode, providerCodexAccountMode, providerMatchesRegistryTransport } from "../../providers/registry"; import { extractModelEnvelopeRows, @@ -1802,7 +1803,7 @@ export async function handleProviderRoutes(ctx: ManagementContext): Promise { + const live = config(); + live.providers.vendor = { adapter: "openai-chat", authMode: "key", baseUrl: "https://vendor.example.test/v1", + apiKey: "synthetic-vendor-key", fetch: (async () => Response.json({ data: [{ id: "" }, { id: "valid" }] })) as typeof fetch } as OcxProviderConfig; + const result = await probe(live, "vendor"); + expect(result.ok).toBe(false); + expect(result.error).toContain("row 0 has an empty id"); + expect(result.error).toContain("pending"); + expect(result.error).toContain("liveModels: false"); + expect(result.error).toContain("models"); +}); diff --git a/tests/providers/provider-model-discovery-contract.test.ts b/tests/providers/provider-model-discovery-contract.test.ts index 3a7cf8aa303..6bd86b22c1c 100644 --- a/tests/providers/provider-model-discovery-contract.test.ts +++ b/tests/providers/provider-model-discovery-contract.test.ts @@ -647,10 +647,37 @@ describe("registry-owned provider model discovery", () => { }); for (const id of [" padded/model", "model\nwith-control"]) { expect(extractProviderModelItems({ data: [{ id }] }, discovery)) - .toEqual({ ok: false, reason: "invalid_shape" }); + .toEqual({ ok: false, reason: "invalid_shape", diagnostic: "row 0 has an invalid id" }); } }); + test("rejects an invalid row with a safe index and reason (#7910)", () => { + const discovery = resolveProviderModelDiscovery("unknown-provider", { + adapter: "openai-chat", baseUrl: "https://example.test/v1", + }); + expect(extractProviderModelItems({ data: [{ id: "", secret: "do-not-log" }, { id: "valid" }] }, discovery)) + .toEqual({ ok: false, reason: "invalid_shape", diagnostic: "row 0 has an empty id" }); + expect(extractProviderModelItems({ data: [{ id: "valid" }, null] }, discovery)) + .toEqual({ ok: false, reason: "invalid_shape", diagnostic: "row 1 is not an object" }); + expect(extractProviderModelItems({ data: [{ id: "valid" }, { id: 7 }] }, discovery)) + .toEqual({ ok: false, reason: "invalid_shape", diagnostic: "row 1 has a non-string id" }); + }); + + test("catalog warns safely and retains configured fallback for invalid rows (#7910)", async () => { + const warning = spyOn(console, "warn").mockImplementation(() => {}); + globalThis.fetch = (async () => Response.json({ data: [{ id: "", secret: "private-row-value" }, { id: "valid" }] })) as typeof fetch; + try { + const models = await gatherRoutedModels(togetherConfig({ models: ["fallback-model"], liveModels: true })); + expect(models.some(model => model.provider === "together" && model.id === "fallback-model")).toBe(true); + expect(models.some(model => model.provider === "together" && model.id === "valid")).toBe(false); + const message = warning.mock.calls.flat().join(" "); + expect(message).toContain("row 0 has an empty id"); + expect(message).toContain("pending"); + expect(message).toContain("liveModels: false"); + expect(message).not.toContain("private-row-value"); + } finally { warning.mockRestore(); } + }); + test("cloudflare-workers-ai resolves official search from the /ai/v1 base", () => { const url = resolveProviderModelDiscoveryUrl( "cloudflare-workers-ai", From 0c74947006604918a0821ba77463ce66cda095ba Mon Sep 17 00:00:00 2001 From: JUN Date: Fri, 30 Oct 2026 21:21:27 +0900 Subject: [PATCH 21/42] fix(codex): stop scoring an elapsed 5-hour window against a pool account (#7994) A known weekly or monthly score no longer counts a 5-hour reading whose valid reset has elapsed, so a higher-priority pool account is selected again without waiting for a WHAM re-probe. Missing or invalid resets keep the reading. Routing and the dashboard warning share hasElapsedShortWindow. Refs #7896. --- gui/src/codex-quota-utils.ts | 7 +- scripts/test-layout/layout.json | 2 +- src/codex/quota-types.ts | 5 + src/codex/routing/cooldown-math.ts | 5 +- structure/providers/openai-accounts.md | 2 +- .../codex-routing-elapsed-short-reset.test.ts | 92 +++++++++++++++++++ tests/fixtures/test-layout-expected.json | 2 +- .../gui/gui-codex-usage-score-parity.test.ts | 14 +++ 8 files changed, 121 insertions(+), 8 deletions(-) create mode 100644 tests/codex-integration/codex-routing-elapsed-short-reset.test.ts diff --git a/gui/src/codex-quota-utils.ts b/gui/src/codex-quota-utils.ts index 191f61ae866..0dcfcba023c 100644 --- a/gui/src/codex-quota-utils.ts +++ b/gui/src/codex-quota-utils.ts @@ -1,4 +1,4 @@ -import { isTerminalShortWindow } from "../../src/codex/quota-types"; +import { hasElapsedShortWindow, isTerminalShortWindow } from "../../src/codex/quota-types"; export interface AccountQuota { workbuddyCredits?: import("../../src/providers/workbuddy-credit-balance").WorkBuddyCreditBalance; @@ -66,7 +66,7 @@ export function normalizeQuotaForPlan(quota: AccountQuota | null, plan: string | * Evaluates governing quota windows based on the account's plan: * - For 30-day only plans (e.g. Free/Go), only the monthly window governs. * - For standard plans, weekly and monthly windows govern. - * - A known five-hour / short window refines a known governing long-window score. + * - A known five-hour / short window refines a known governing long-window score until its valid reset elapses. * - If no long window has been observed, an active terminal short burst (at 100%) acts as exhausted (100). * - Unknown or unprimed quota returns `null` so callers do not spuriously trigger threshold actions. */ @@ -104,6 +104,7 @@ export function computeCodexUsageScore( ? 100 : null; } - const values = finite(shortPercent) ? [...knownLong, shortPercent] : knownLong; + const values = finite(shortPercent) && !hasElapsedShortWindow(quota.fiveHourResetAt ?? quota.shortResetAt, now) + ? [...knownLong, shortPercent] : knownLong; return values.length ? Math.max(...values) : null; } diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index b80fd41ad6a..ef5c8df7fe2 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -705,7 +705,7 @@ "codex-restart-route.test.ts": "codex-integration", "codex-restore-app-rewrite.test.ts": "codex-integration", "codex-retained-root-serialization.test.ts": "codex-integration", - "codex-routing-cache-affinity-detour.test.ts": "codex-integration", + "codex-routing-cache-affinity-detour.test.ts": "codex-integration", "codex-routing-elapsed-short-reset.test.ts": "codex-integration", "codex-routing.test.ts": "codex-integration", "codex-runtime.test.ts": "codex-integration", "codex-runtime-macos-desktop.test.ts": "codex-integration", "codex-clamp-notice.test.ts": "codex-integration", "codex-runtime-wsl-desktop.test.ts": "codex-integration", diff --git a/src/codex/quota-types.ts b/src/codex/quota-types.ts index be1a0bb65c4..8a77c6d44fd 100644 --- a/src/codex/quota-types.ts +++ b/src/codex/quota-types.ts @@ -62,6 +62,11 @@ export function resetAtToMs(resetAt: number): number { return resetAt < RESET_AT_SECONDS_MAX ? resetAt * 1000 : resetAt; } +/** Only a valid reset at or before now proves a short-window reading has elapsed. */ +export function hasElapsedShortWindow(resetAt: number | undefined, now: number): boolean { + return typeof resetAt === "number" && Number.isFinite(resetAt) && resetAt > 0 && resetAtToMs(resetAt) <= now; +} + /** * A short-only reading that proves the account is blocked NOW. * diff --git a/src/codex/routing/cooldown-math.ts b/src/codex/routing/cooldown-math.ts index 515b4d9eb22..858d98b395f 100644 --- a/src/codex/routing/cooldown-math.ts +++ b/src/codex/routing/cooldown-math.ts @@ -3,7 +3,7 @@ import { CODEX_UNKNOWN_USAGE_SCORE, } from "../quota"; import { isThirtyDayOnlyCodexPlan } from "../plan"; -import { hasSpendableCodexCredits, isTerminalShortWindow, type CodexSpendableCredits } from "../quota-types"; +import { hasElapsedShortWindow, hasSpendableCodexCredits, isTerminalShortWindow, type CodexSpendableCredits } from "../quota-types"; import type { CodexQuotaScope } from "./health-store"; import type { TransientProbeGrant } from "./thread-affinity"; import type { MainQuotaDispatch } from "../main-account-cache"; @@ -136,7 +136,8 @@ export function computeCodexUsageScore(quota: { ? (allowCredits && hasSpendableCodexCredits(quota, now)) ? CODEX_EXHAUSTED_USAGE_PERCENT - 1 : CODEX_EXHAUSTED_USAGE_PERCENT : CODEX_UNKNOWN_USAGE_SCORE; } - const values = finite(quota.shortPercent) ? [...knownLong, quota.shortPercent] : knownLong; + const values = finite(quota.shortPercent) && !hasElapsedShortWindow(quota.shortResetAt, now) + ? [...knownLong, quota.shortPercent] : knownLong; const score = Math.max(...values); // Keep included-capacity accounts ahead of a paid overage account, but do not // classify the latter as exhausted solely from its subscription percentages. diff --git a/structure/providers/openai-accounts.md b/structure/providers/openai-accounts.md index 2b758db3ab3..4cb00c851b8 100644 --- a/structure/providers/openai-accounts.md +++ b/structure/providers/openai-accounts.md @@ -426,7 +426,7 @@ The Codex parser in `src/oauth/pool-kernel.ts` is reexported by the compatibilit The account-pool strategy control and `ocx account pool get openai strategy` summarize how the configured threshold applies to the active strategy. Manual-switch warnings use the routing usage -score. A reset-less terminal short window is current only when its `shortObservedAt` is not in the +score. A known long-window score excludes short usage when its valid reset has elapsed; absent or invalid resets retain it. A reset-less terminal short window is current only when its `shortObservedAt` is not in the future and is at most `TERMINAL_SHORT_WINDOW_FRESHNESS_MS` old; general `updatedAt` changes do not extend that observation. diff --git a/tests/codex-integration/codex-routing-elapsed-short-reset.test.ts b/tests/codex-integration/codex-routing-elapsed-short-reset.test.ts new file mode 100644 index 00000000000..b021b77f531 --- /dev/null +++ b/tests/codex-integration/codex-routing-elapsed-short-reset.test.ts @@ -0,0 +1,92 @@ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { clearCodexUpstreamHealth, clearThreadAccountMap, resolveCodexAccountForThread } from "../../src/codex/routing"; +import { clearPoolRotationState } from "../../src/codex/pool-rotation"; +import { saveCodexAccountCredential } from "../../src/codex/account-store"; +import { clearAccountQuota, setAccountQuotaFromParsed, updateAccountQuota } from "../../src/codex/auth-api"; +import type { OcxConfig } from "../../src/types"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; +import { setAsyncIcaclsRunnerForTests, setIcaclsRunnerForTests } from "../../src/lib/windows-secret-acl"; + +// Lives beside codex-routing.test.ts, which sits at its file-size cap (#7896). +// A pool account whose 5-hour window filled while its weekly window was known kept +// scoring 100 after that window's reset passed, so a lower-priority account served +// every request until something happened to re-probe WHAM. + +const ICACLS_OK = { success: true, exitCode: 0, timedOut: false, stdout: "" }; +const NOW = 1_800_000_000_000; +let testDir = ""; +let previousOpencodexHome: string | undefined; +let previousCodexHome: string | undefined; + +function orderedConfig(): OcxConfig { + return { + providers: {}, + codexAccounts: [ + { id: "a", email: "a@test", isMain: false }, + { id: "b", email: "b@test", isMain: false }, + ], + activeCodexAccountId: "b", + codexAccountPriorities: { a: 1 }, + autoSwitchThreshold: 97, + } as OcxConfig; +} + +function saveTestCredential(id: string): void { + saveCodexAccountCredential(id, { + accessToken: "access-" + id, refreshToken: "refresh-" + id, + expiresAt: NOW + 5 * 60_000, chatgptAccountId: "acct-" + id, + }); +} + +describe("priority after an elapsed 5-hour reset (#7896)", () => { + beforeEach(() => { + previousOpencodexHome = process.env.OPENCODEX_HOME; + previousCodexHome = process.env.CODEX_HOME; + testDir = mkdtempSync(join(tmpdir(), "ocx-routing-elapsed-short-")); + setIcaclsRunnerForTests(() => ICACLS_OK); + setAsyncIcaclsRunnerForTests(async () => ICACLS_OK); + process.env.OPENCODEX_HOME = testDir; + process.env.CODEX_HOME = testDir; + clearThreadAccountMap(); + clearCodexUpstreamHealth(); + clearAccountQuota(); + clearPoolRotationState(); + saveTestCredential("a"); + saveTestCredential("b"); + }); + + afterEach(() => { + try { + clearAccountQuota(); + clearCodexUpstreamHealth(); + clearThreadAccountMap(); + clearPoolRotationState(); + } finally { + setIcaclsRunnerForTests(null); + setAsyncIcaclsRunnerForTests(null); + if (previousOpencodexHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousOpencodexHome; + if (previousCodexHome === undefined) delete process.env.CODEX_HOME; + else process.env.CODEX_HOME = previousCodexHome; + removeTreeWithRetry(testDir); + } + }); + + test("the higher tier returns once its known short reset has passed, without a fresh probe", () => { + const config = orderedConfig(); + setAccountQuotaFromParsed("a", { weeklyPercent: 20, shortPercent: 100, shortResetAt: NOW + 3_600_000 }); + updateAccountQuota("b", 10); + expect(resolveCodexAccountForThread(null, config, NOW)).toBe("b"); + expect(resolveCodexAccountForThread(null, config, NOW + 3_601_000)).toBe("a"); + }); + + test("a short reading without a reset still keeps the higher tier out", () => { + const config = orderedConfig(); + setAccountQuotaFromParsed("a", { weeklyPercent: 20, shortPercent: 100 }); + updateAccountQuota("b", 10); + expect(resolveCodexAccountForThread(null, config, NOW + 3_601_000)).toBe("b"); + }); +}); diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index be2098b4109..e3ac7ff2a95 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -678,7 +678,7 @@ "codex-restart-contract-parity.test.ts": "codex-integration", "codex-restart-route.test.ts": "codex-integration", "codex-restore-app-rewrite.test.ts": "codex-integration", "codex-retained-root-serialization.test.ts": "codex-integration", - "codex-routing-cache-affinity-detour.test.ts": "codex-integration", "codex-routing.test.ts": "codex-integration", + "codex-routing-cache-affinity-detour.test.ts": "codex-integration", "codex-routing-elapsed-short-reset.test.ts": "codex-integration", "codex-routing.test.ts": "codex-integration", "codex-runtime.test.ts": "codex-integration", "codex-runtime-macos-desktop.test.ts": "codex-integration", "codex-runtime-wsl-desktop.test.ts": "codex-integration", "codex-clamp-notice.test.ts": "codex-integration", "codex-service-manager-probe-hardening.test.ts": "codex-integration", "codex-service-manager-probe.test.ts": "codex-integration", "codex-systemd-env-probe.test.ts": "codex-integration", diff --git a/tests/gui/gui-codex-usage-score-parity.test.ts b/tests/gui/gui-codex-usage-score-parity.test.ts index e72f550d9ab..7d1ee8a9a42 100644 --- a/tests/gui/gui-codex-usage-score-parity.test.ts +++ b/tests/gui/gui-codex-usage-score-parity.test.ts @@ -39,6 +39,20 @@ function expectSame(quota: AccountQuota, plan?: string | null): void { } describe("account-switch warning agrees with routing (#5045)", () => { + test("known long window ignores only valid elapsed short resets (#7896)", () => { + const future = NOW + 3_600_000; + const past = NOW - 1_000; + for (const [shortResetAt, expected] of [ + [future, 100], [future / 1000, 100], [past, 20], [past / 1000, 20], + [NOW, 20], [undefined, 100], [0, 100], [-1, 100], [NaN, 100], [Infinity, 100], + ] as const) { + const quota: AccountQuota = { weeklyPercent: 20, shortPercent: 100, shortResetAt, updatedAt: NOW }; + expect(guiScore(quota, null, NOW)).toBe(expected); + expect(routerScore(asRouterQuota(quota), null, NOW)).toBe(expected); + expect(guiScore({ weeklyPercent: 20, fiveHourPercent: 100, fiveHourResetAt: shortResetAt, updatedAt: NOW }, null, NOW)).toBe(expected); + } + }); + test("a terminal burst window reads the same in seconds and in milliseconds", () => { // Both units reach storage. Read as milliseconds, a seconds-form instant lands in 1970 and // every future reset looks elapsed — a check that passes its own test and does nothing. From b692a8a33399007baae29de60c6646da091497ea Mon Sep 17 00:00:00 2001 From: JUN Date: Fri, 30 Oct 2026 21:22:36 +0900 Subject: [PATCH 22/42] test(history): corrupt the index the health query plan actually reads (#7995) The routing-health integrity case corrupted idx_requests_model and expected the health query to hit it. The query can be served by the provider, model or timestamp index, and macOS's system SQLite picks a different one than Bun's bundled SQLite, so macOS 1/4 of lane=all run 41050393474 read healthy rows. Ask EXPLAIN QUERY PLAN which index this build uses and corrupt that one. Co-authored-by: JUN <243035832+lidge-jun@users.noreply.github.com> --- tests/usage/request-history-integrity.test.ts | 22 ++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/tests/usage/request-history-integrity.test.ts b/tests/usage/request-history-integrity.test.ts index 3c824e918db..ac424842752 100644 --- a/tests/usage/request-history-integrity.test.ts +++ b/tests/usage/request-history-integrity.test.ts @@ -107,8 +107,8 @@ async function verifyInitial(checks: Check[]): Promise { await ready; } -function corruptModelIndex(handle: Database, path: string): void { - const { rootpage } = handle.query("SELECT rootpage FROM sqlite_master WHERE name = 'idx_requests_model'").get() as { rootpage: number }; +function corruptModelIndex(handle: Database, path: string, index = "idx_requests_model"): void { + const { rootpage } = handle.query("SELECT rootpage FROM sqlite_master WHERE name = ?").get(index) as { rootpage: number }; const { page_size: pageSize } = handle.query("PRAGMA page_size").get() as { page_size: number }; expect(rootpage).toBeGreaterThan(1); const checkpoint = handle.query("PRAGMA wal_checkpoint(TRUNCATE)").get() as { busy: number }; @@ -121,6 +121,22 @@ function corruptModelIndex(handle: Database, path: string): void { handle.exec("PRAGMA shrink_memory"); } +// Several indexes can serve the health query; which one SQLite's planner picks differs between +// builds (Bun bundles SQLite on Linux and Windows, macOS uses the system library). Corrupt the one +// this build actually reads instead of assuming the model index. +function healthQueryIndex(handle: Database): string { + // prepare + finalize: a query() statement stays cached on the handle and holds a table lock. + const statement = handle.prepare( + "EXPLAIN QUERY PLAN SELECT status FROM requests WHERE provider = ? AND model = ? AND timestamp >= ? ORDER BY timestamp DESC LIMIT ?", + ); + let plan: Array<{ detail: string }>; + try { plan = statement.all("a", "m1", 0, 1) as Array<{ detail: string }>; } + finally { statement.finalize(); } + const index = plan.map(row => /USING (?:COVERING )?INDEX (\w+)/.exec(row.detail)?.[1]).find(Boolean); + expect(index).toMatch(/^idx_requests_/); + return index!; +} + describe("request-history integrity lifecycle (#7966)", () => { test("real worker checks leave repeated page, row, status and routing reads free of full serving-thread scans", async () => { await existingIndex(); @@ -409,7 +425,7 @@ describe("request-history integrity lifecycle (#7966)", () => { clearHealthHistoryCacheForTests(); const handle = requestHistoryDb(); const canonical = readFileSync(usageLogPath()); - corruptModelIndex(handle, path); + corruptModelIndex(handle, path, healthQueryIndex(handle)); expect(healthEvidenceForCandidate(input)).toEqual({}); expect((await openRequestHistoryIndex()).indexedRows).toBe(5); expect(requestHistoryDb()).not.toBe(handle); From ba0556d001dbca25aaef46680a56fd3150c4b19c Mon Sep 17 00:00:00 2001 From: JUN Date: Fri, 30 Oct 2026 21:22:55 +0900 Subject: [PATCH 23/42] fix(codex): retire a pool account's carried 5-hour window when WHAM shows none (#7980) A complete, valid WHAM probe without a burst window now retires a pool account's carried 5-hour tuple, so a Plus to Pro upgrade no longer keeps the account at a stale limit. Retirement requires a live captured pool writer for the account, credential generation and history identity; partial or header observations, invalid or unparseable readings and an explicit allowed:false keep the tuple. Main behaviour is unchanged. Closes #7961. Co-authored-by: SupernovaTitanium <12254451+SupernovaTitanium@users.noreply.github.com> --- src/codex/auth-api/login-flow.ts | 4 +- src/codex/auth-api/pool-quota-probe.ts | 4 +- src/codex/quota.ts | 35 +++++- structure/dashboard-and-usage.md | 2 +- structure/providers/openai-tiers.md | 2 +- .../codex-credits-probes.test.ts | 113 +++++++++++++++++- .../main-quota-provenance.test.ts | 10 ++ 7 files changed, 160 insertions(+), 10 deletions(-) diff --git a/src/codex/auth-api/login-flow.ts b/src/codex/auth-api/login-flow.ts index 8b4547b8cc5..6cddd171904 100644 --- a/src/codex/auth-api/login-flow.ts +++ b/src/codex/auth-api/login-flow.ts @@ -1,6 +1,6 @@ import { withCodexAccountLogLabel } from "../account-label"; import { getCodexAccountCredential, markCodexAccountValidated, readCodexAccountRecord, saveCodexAccountCredential, CodexCredentialRefreshLockTimeoutError, CodexCredentialRefreshBusyError, CodexCredentialRefreshStaleError } from "../account-store"; -import { clearAccountQuota, isCodexQuotaExhausted, isValidWhamHistoryObservation, parseUsageQuota, setAccountQuotaFromParsed } from "../quota"; +import { clearAccountQuota, isCodexQuotaExhausted, parseUsageQuota, poolWhamPolicyObservation, setAccountQuotaFromParsed } from "../quota"; import type { StoredAccountQuota, WhamUsageResponse } from "../quota"; import { ConfigMutationLockError, withConfigMutationLockSync } from "../../config"; import { appendDefaultCodexAccountNamespace, codexAccountPickerEnabled } from "../account-namespaces"; @@ -278,7 +278,7 @@ export async function handleCodexAuthLoginStart(req: Request, config: OcxConfig, email = data.email?.trim() || email; plan = nonEmptyPlan(data.plan_type) ?? plan; quota = parseUsageQuota(data); - policyQuota = isValidWhamHistoryObservation(data) ? quota : null; + policyQuota = poolWhamPolicyObservation(data, quota); } } catch { /* wham fetch is non-blocking */ } // Reauth must refresh the same ChatGPT identity already bound to this pool slot. diff --git a/src/codex/auth-api/pool-quota-probe.ts b/src/codex/auth-api/pool-quota-probe.ts index abd6b2d5fe8..583fc24b6c6 100644 --- a/src/codex/auth-api/pool-quota-probe.ts +++ b/src/codex/auth-api/pool-quota-probe.ts @@ -3,7 +3,7 @@ import { fetchCodexUsage } from "../quota-query-backoff"; import type { CodexUsageOwner } from "../quota-query-backoff"; import { capturePoolQuotaWriter, getValidCodexToken, isCodexAccountGenerationLive, forceRefreshCodexPoolToken, markCodexAccountValidated, markCodexAccountValidationFailed, readCodexAccountRecord, isTerminalCodexPoolRefreshFailure, CodexCredentialGenerationConflictError, CodexCredentialRefreshLockTimeoutError, CodexCredentialRefreshBusyError, CodexCredentialRefreshStaleError, TokenRefreshError } from "../account-store"; import type { PoolQuotaWriter } from "../quota-types"; -import { isValidWhamHistoryObservation, getAccountQuota, isCompleteCodexQuotaRecoverySnapshot, parseUsageQuota, setAccountQuotaFromParsed } from "../quota"; +import { isValidWhamHistoryObservation, getAccountQuota, isCompleteCodexQuotaRecoverySnapshot, parseUsageQuota, poolWhamPolicyObservation, setAccountQuotaFromParsed } from "../quota"; import type { StoredAccountQuota, WhamUsageResponse } from "../quota"; import type { ManualResetRefreshLineage } from "../routing"; import { clearAccountNeedsReauth, markAccountNeedsReauth } from "../account-runtime-state"; @@ -377,7 +377,7 @@ export async function commitPoolQuotaResponse( return { quota: null, needsReauth: false, credentialGeneration: generation }; } const validPolicyObservation = isValidWhamHistoryObservation(data); - setAccountQuotaFromParsed(accountId, quota, writerGeneration, undefined, validPolicyObservation ? quota : null, + setAccountQuotaFromParsed(accountId, quota, writerGeneration, undefined, poolWhamPolicyObservation(data, quota), ctx.poolWriter && validPolicyObservation ? { writer: ctx.poolWriter, observedAt, source: "wham", raw: quota } : undefined, true); return { diff --git a/src/codex/quota.ts b/src/codex/quota.ts index fe2bbbc10ba..744e8127d7c 100644 --- a/src/codex/quota.ts +++ b/src/codex/quota.ts @@ -333,7 +333,10 @@ export function setAccountQuotaFromParsed( const mainDecisionGeneration = mainWriter && mainDecisionQuotaWriterGeneration(mainWriter); if (isMain && mainDecisionGeneration !== undefined && decisionRaw) publishCodexDecisionQuota(accountId, mainDecisionGeneration, decisionRaw, updatedAt, "codex-main", decisionObservationPartial); - const displayQuota: MainPolicyQuotaObservation = isMain && mainWriter && policyQuota?.shortWindowAbsent === true + // Retiring a short tuple releases a routing block, so it needs the same writer proof as + // history: a live main writer for main, a live captured pool writer for pool requests. + const mayRetireShort = isMain ? mainWriter !== undefined : livePoolEvidence; + const displayQuota: MainPolicyQuotaObservation = mayRetireShort && policyQuota?.shortWindowAbsent === true ? { ...quota, shortWindowAbsent: true } : quota; const next = mergeAccountQuota(displayQuota, legacyExisting, updatedAt); @@ -934,7 +937,14 @@ export function parseMainPolicyUsageQuota(data: WhamUsageResponse): MainPolicyQu const windows = [data.rate_limit?.primary_window, data.rate_limit?.secondary_window, data.rate_limit?.tertiary_window]; if (windows.some(window => isInvalidPolicyUsagePercent(window?.used_percent))) return null; const quota = filterMainPolicyMonthlyQuota(parseUsageQuota(data), isThirtyDayOnlyCodexPlan(data.plan_type)); - const [primary, secondary, tertiary] = windows; + return quota ? withWhamShortWindowAbsence(data, quota) : null; +} + +/** Derive absence from complete WHAM topology without changing the display quota. */ +export function withWhamShortWindowAbsence( + data: WhamUsageResponse, quota: Omit, +): MainPolicyQuotaObservation { + const [primary, secondary, tertiary] = [data.rate_limit?.primary_window, data.rate_limit?.secondary_window, data.rate_limit?.tertiary_window]; // Primary and secondary must explicitly report their topology. WHAM also sends // two-window responses without tertiary; an explicit allowed verdict disambiguates those // from partial observations. A present tertiary still requires measured long/null evidence. @@ -942,7 +952,7 @@ export function parseMainPolicyUsageQuota(data: WhamUsageResponse): MainPolicyQu // carries a valid usage reading: a long window without used_percent leaves that // window's usage unknown, and unknown usage must never release a block. // Headers never supply this proof, and reset time alone still cannot release a block. - if (quota && (isMeasuredLongWindow(primary) + if ((isMeasuredLongWindow(primary) || (primary === null && isMeasuredLongWindow(secondary) && quota.weeklyPercent !== undefined)) && (secondary === null || isMeasuredLongWindow(secondary)) && (tertiary === null || isMeasuredLongWindow(tertiary) @@ -952,6 +962,25 @@ export function parseMainPolicyUsageQuota(data: WhamUsageResponse): MainPolicyQu return quota; } +/** + * Policy evidence for a stored pool account's WHAM read (pool probe or re-login): null when any + * reading is invalid. A complete long/null topology marks short-window absence (#7961) unless + * upstream explicitly denied the account, since a denial is never evidence that a block ended. + */ +export function poolWhamPolicyObservation( + data: WhamUsageResponse, quota: Omit | null, +): MainPolicyQuotaObservation | null { + if (!quota || !isValidWhamHistoryObservation(data)) return null; + // Retirement releases a block, so a present reading that does not parse is not "unknown". + const readings = [data.rate_limit?.primary_window, data.rate_limit?.secondary_window, data.rate_limit?.tertiary_window]; + for (const limit of Array.isArray(data.additional_rate_limits) ? data.additional_rate_limits : []) { + if (limit && typeof limit === "object") readings.push(limit.rate_limit?.primary_window, limit.rate_limit?.secondary_window); + } + const malformed = readings.some(window => window?.used_percent != null + && normalizeUsagePercent(window.used_percent) === undefined); + return data.rate_limit?.allowed === false || malformed ? quota : withWhamShortWindowAbsence(data, quota); +} + function isMeasuredLongWindow(window: WhamUsageWindow | null | undefined): boolean { return isExplicitLongWindow(window) && normalizeUsagePercent(window?.used_percent) !== undefined; } diff --git a/structure/dashboard-and-usage.md b/structure/dashboard-and-usage.md index 960bbdc128c..17bc382a091 100644 --- a/structure/dashboard-and-usage.md +++ b/structure/dashboard-and-usage.md @@ -10,7 +10,7 @@ The dashboard is a local control surface, not a separate service. It should refl and catalog invariants documented in this folder rather than inventing parallel state. The sign-in form offers opt-in **Remember this device** for direct HTTPS dashboards or loopback, storing an HttpOnly device cookie plus a separate proof in this site's origin-scoped localStorage; neither is the admin token or works alone, cookies span ports of one hostname so the proof binds the dashboard origin (still prefer a dedicated hostname), and storage failure reports Remember unavailable. Both mobile and sidebar **Forget this device** actions use logout to revoke the record and derived sessions immediately after successful publication and remove the proof, retaining it on failure for retry (directory-entry crash durability is best effort); remembered sessions cannot approve GitHub stars, reset-credit consumption, prompt-layer edits, Meta Muse sign-in, desktop compatibility changes or quota validation, which require a fresh local or paired session, and session-change events update both surfaces under [GUI and management API](gui-and-management-api.md). Codex quota cards consume the display cache from `src/codex/quota.ts`. Partial refreshes remove -omitted short tuples after their reset deadline; validated main-window absence also removes +omitted short tuples after their reset deadline; validated main-window absence, or validated complete WHAM absence for a pool account (#7961), also removes future-reset short tuples. Main-account hard-lock evidence and reset-notification history follow separate retention rules in [OpenAI account modes](providers/openai-tiers.md#quota-cache-and-short-window-history). See [permanent refusal auto-pause](account-auto-pause.md). diff --git a/structure/providers/openai-tiers.md b/structure/providers/openai-tiers.md index a490d7db535..8f11ab67151 100644 --- a/structure/providers/openai-tiers.md +++ b/structure/providers/openai-tiers.md @@ -435,7 +435,7 @@ cannot prove that the short window disappeared. All-null credits-only and tertia the resulting weekly/monthly window still blocks at its long threshold (98% by default). This prevents old short-window exhaustion from surviving indefinitely on a now weekly/monthly account. A live main writer applies that same retirement proof to the display/rotation cache, so account selection cannot keep refusing the -recovered main account on its old short tuple. Coverage lives in +recovered main account on its old short tuple. Pool accounts get the same proof from a valid complete WHAM probe (#7961): `poolWhamPolicyObservation` marks the observation from its window topology only, so a Plus to Pro upgrade cannot keep a stored pool account on the old plan's 5-hour tuple, while partial WHAM, invalid or unparseable readings, an explicit `allowed: false` verdict and response headers keep carrying it. A pool probe retires the tuple only with a live captured pool writer for that account and credential generation, the same proof its history write needs; a missing, stale or other-account writer keeps the tuple. The login usage read marks its observation the same way, but reauthentication clears the row before writing and creation refuses existing ids, so it has nothing to retire. Main-only monthly filtering is not applied to pool observations, and a main observation without a live main writer still carries its tuple. Coverage lives in `tests/codex-integration/codex-credits-probes.test.ts` (pool), `tests/codex-integration/main-quota-evidence-validation.test.ts`, `tests/codex-integration/main-account-hard-lock-retirement.test.ts`, `tests/codex-integration/main-quota-provenance.test.ts`, and diff --git a/tests/codex-integration/codex-credits-probes.test.ts b/tests/codex-integration/codex-credits-probes.test.ts index 71e7602d963..c45886cfe35 100644 --- a/tests/codex-integration/codex-credits-probes.test.ts +++ b/tests/codex-integration/codex-credits-probes.test.ts @@ -9,7 +9,7 @@ import { fetchMainAccountInfoSnapshot } from "../../src/codex/auth-api/main-acco import { commitPoolQuotaResponse } from "../../src/codex/auth-api/pool-quota-probe"; import { clearMainAccountInfoCache } from "../../src/codex/main-account-cache"; import { resetMainCodexAccountIdentityTrackingForTests } from "../../src/codex/account-lifecycle"; -import { clearAccountQuota } from "../../src/codex/quota"; +import { applyAccountQuotaFromUpstreamHeaders, clearAccountQuota, getAccountQuota, parseUsageQuota, poolWhamPolicyObservation, type WhamUsageResponse } from "../../src/codex/quota"; import { resetQuotaQueryBackoffForTests } from "../../src/codex/quota-query-backoff"; import { captureConfigGeneration } from "../../src/lib/state-store-sweeper"; import { resetLifecycleDrainStateForTests } from "../../src/server/lifecycle"; @@ -177,3 +177,114 @@ describe("pool credits publication", () => { expect(poolReads).toBe(1); }); }); + +describe("pool WHAM short-window retirement", () => { + const now = () => Math.floor(Date.now() / 1000); + const short = () => ({ used_percent: 100, limit_window_seconds: 18_000, reset_at: now() + 3600 }); + const weekly = () => ({ used_percent: 0, limit_window_seconds: 604_800, reset_at: now() + 604_800 }); + const observe = (primary: unknown, secondary: unknown, extra = {}) => Response.json({ + plan_type: "pro", rate_limit: { allowed: true, primary_window: primary, secondary_window: secondary }, ...extra, + }); + + test("complete weekly-only probe retires the carried short tuple", async () => { + savePool(); + const ctx = poolContext(); + await commitPoolQuotaResponse(observe(short(), weekly()), ctx); + const result = await commitPoolQuotaResponse(observe(weekly(), null), ctx); + expect(result.quota?.shortPercent).toBeUndefined(); + expect(result.quota?.shortResetAt).toBeUndefined(); + expect(result.quota?.weeklyPercent).toBe(0); + }); + + test("a Plus-style five-hour window remains present", async () => { + savePool(); + const ctx = poolContext(); + const result = await commitPoolQuotaResponse(observe(short(), weekly(), { plan_type: "plus" }), ctx); + expect(result.quota?.shortPercent).toBe(100); + expect(result.quota?.shortResetAt).toBeDefined(); + }); + + test("partial and invalid WHAM observations retain the carried tuple", async () => { + savePool(); + const ctx = poolContext(); + await commitPoolQuotaResponse(observe(short(), weekly()), ctx); + const partial = await commitPoolQuotaResponse(Response.json({ rate_limit: { primary_window: weekly() } }), ctx); + expect(partial.quota?.shortPercent).toBe(100); + const invalid = await commitPoolQuotaResponse(observe(weekly(), null, { + additional_rate_limits: [{ rate_limit: { primary_window: { used_percent: 101 } } }], + }), ctx); + expect(invalid.quota?.shortPercent).toBe(100); + }); + test("partial response headers do not retire a carried short window", async () => { + savePool(); + await commitPoolQuotaResponse(observe(short(), weekly()), poolContext()); + applyAccountQuotaFromUpstreamHeaders(account.id, new Headers({ "x-codex-primary-used-percent": "12" })); + expect(getAccountQuota(account.id)?.shortPercent).toBe(100); + }); + + test("monthly-only pool plans keep their monthly observation", async () => { + savePool(); + const ctx = poolContext(); + await commitPoolQuotaResponse(observe(short(), weekly()), ctx); + const monthly = { used_percent: 14, limit_window_seconds: 2_592_000, reset_at: now() + 2_592_000 }; + const result = await commitPoolQuotaResponse(observe(monthly, null, { plan_type: "go" }), ctx); + expect(result.quota?.shortPercent).toBeUndefined(); + expect(result.quota?.monthlyPercent).toBe(14); + }); + + test("a probe without a live writer for this account retains the carried tuple", async () => { + savePool(); + await commitPoolQuotaResponse(observe(short(), weekly()), poolContext()); + const missing = await commitPoolQuotaResponse(observe(weekly(), null), { ...poolContext(), poolWriter: undefined }); + expect(missing.quota?.shortPercent).toBe(100); + const live = poolContext().poolWriter!; + const stale = { ...live, historyIdentity: "00000000-0000-4000-8000-000000000000" }; + expect((await commitPoolQuotaResponse(observe(weekly(), null), { ...poolContext(), poolWriter: stale })).quota?.shortPercent).toBe(100); + const other = { ...live, accountId: "other-pool" }; + expect((await commitPoolQuotaResponse(observe(weekly(), null), { ...poolContext(), poolWriter: other })).quota?.shortPercent).toBe(100); + expect((await commitPoolQuotaResponse(observe(weekly(), null), poolContext())).quota?.shortPercent).toBeUndefined(); + }); + test("an explicit upstream denial never retires the carried tuple", async () => { + savePool(); + const ctx = poolContext(); + await commitPoolQuotaResponse(observe(short(), weekly()), ctx); + const denied = await commitPoolQuotaResponse(Response.json({ plan_type: "pro", rate_limit: { + allowed: false, primary_window: weekly(), secondary_window: null, tertiary_window: null, + } }), ctx); + expect(denied.quota?.shortPercent).toBe(100); + }); + + test("pool policy evidence marks absence only for a valid, allowed, complete topology", () => { + const evidence = (data: WhamUsageResponse) => poolWhamPolicyObservation(data, parseUsageQuota(data)); + const weeklyOnly = { rate_limit: { allowed: true, primary_window: weekly(), secondary_window: null } }; + expect(evidence(weeklyOnly)?.shortWindowAbsent).toBe(true); + expect(evidence({ rate_limit: { ...weeklyOnly.rate_limit, allowed: false, tertiary_window: null } })?.shortWindowAbsent).toBeUndefined(); + expect(evidence({ rate_limit: { allowed: true, primary_window: short(), secondary_window: weekly() } })?.shortWindowAbsent).toBeUndefined(); + expect(evidence({ rate_limit: { primary_window: weekly() } })?.shortWindowAbsent).toBeUndefined(); + expect(evidence({ ...weeklyOnly, additional_rate_limits: [{ rate_limit: { primary_window: { used_percent: 101 } } }] })).toBeNull(); + }); + + test("a malformed usage reading keeps the carried tuple and its reset", async () => { + savePool(); + const ctx = poolContext(); + const first = await commitPoolQuotaResponse(observe(short(), weekly()), ctx); + for (const used_percent of ["unreadable", true, {}]) { + const result = await commitPoolQuotaResponse(observe(weekly(), null, { + additional_rate_limits: [{ rate_limit: { primary_window: { used_percent, limit_window_seconds: 604_800 } } }], + }), ctx); + expect(result.quota?.shortPercent).toBe(100); + expect(result.quota?.shortResetAt).toBe(first.quota?.shortResetAt); + } + }); + + test("a writer captured before a credential rollover cannot retire the tuple", async () => { + savePool(); + await commitPoolQuotaResponse(observe(short(), weekly()), poolContext()); + const before = poolContext().poolWriter!; + savePool(); + const after = poolContext(); + expect(after.generation).not.toBe(before.credentialGeneration); + const replay = await commitPoolQuotaResponse(observe(weekly(), null), { ...after, poolWriter: before }); + expect(replay.quota?.shortPercent).toBe(100); + }); +}); diff --git a/tests/codex-integration/main-quota-provenance.test.ts b/tests/codex-integration/main-quota-provenance.test.ts index af5ae6339e2..5494cd2c29d 100644 --- a/tests/codex-integration/main-quota-provenance.test.ts +++ b/tests/codex-integration/main-quota-provenance.test.ts @@ -658,3 +658,13 @@ test("native main observations and oversized cache never become pool history", ( } }, padding: "x".repeat(4 * 1024 * 1024) })); expect(getAccountQuotaHistory("history-pool").observations).toEqual([]); }); + + +test("main short window is retained without a main writer even with WHAM absence evidence", () => { + const short = { shortPercent: 100, shortResetAt: Date.now() / 1000 + 3600 }; + setAccountQuotaFromParsed(MAIN, short); + const data: WhamUsageResponse = { rate_limit: { allowed: true, + primary_window: { used_percent: 0, limit_window_seconds: 604_800 }, secondary_window: null } }; + setAccountQuotaFromParsed(MAIN, parseUsageQuota(data), undefined, undefined, parseMainPolicyUsageQuota(data)); + expect(getAccountQuota(MAIN)?.shortPercent).toBe(100); +}); From 03d8bd0ce9dd707e1cfb0ac678b4a53f02f813d1 Mon Sep 17 00:00:00 2001 From: JUN Date: Fri, 30 Oct 2026 21:24:34 +0900 Subject: [PATCH 24/42] fix(claude): keep web_search native when no sidecar backend can run (#7991) * fix(claude): preserve native search without sidecar * fix(claude): preserve search probe affinity and cancellation * fix(claude): stop the search probe when refresh wraps a cancellation --- src/server/claude-messages.ts | 77 +++++++++ src/server/messages-native-eligibility.ts | 11 +- structure/data-planes/protocol-paths.md | 8 +- .../claude-translated-server-tools.test.ts | 160 +++++++++++++++++- .../messages-native-bridge-policy.test.ts | 11 ++ 5 files changed, 250 insertions(+), 17 deletions(-) diff --git a/src/server/claude-messages.ts b/src/server/claude-messages.ts index 2c26e6e6e32..a2d6b25c676 100644 --- a/src/server/claude-messages.ts +++ b/src/server/claude-messages.ts @@ -970,6 +970,83 @@ async function handleClaudeMessagesWithBudget( ...(settledRoute?.provider.authMode === "oauth" && settledInstance ? { oauthFailoverQuorum: anthropicRoutingFor(settledInstance).hasAnthropicFailoverQuorum() } : {}), }; + if (settledRoute && isRec(anthropicBody) + && nativeMessagesDeclineReason(settledRoute, anthropicBody, config, nativeSelector) === "bridge-only-policy" + && nativeMessagesDeclineReason(settledRoute, anthropicBody, config, { ...nativeSelector, webSearchBackendAvailable: false }) === undefined) { + const { parseRequest } = await import("../responses/parser"); + const { planWebSearch, shouldResolveOpenAiWebSearchSidecar } = await import("../web-search"); + const replayConfig = buildClaudeReplayConfig(config); + const parsed = parseRequest(internalBody); + let openAiSidecar: import("../providers/openai-sidecar").ResolvedOpenAiForwardSidecar | undefined; + if (shouldResolveOpenAiWebSearchSidecar(replayConfig, parsed, false)) { + const { listOpenAiForwardSidecarCandidates, captureExplicitOpenAiCallerAuth, resolveFirstUsableOpenAiSidecar } = await import("../providers/openai-sidecar"); + const candidates = listOpenAiForwardSidecarCandidates(replayConfig); + const headers = new Headers(); + for (const name of FORWARD_HEADERS) { + if (name === "authorization" || name === "chatgpt-account-id") continue; + const value = req.headers.get(name); + if (value) headers.set(name, value); + } + const { codexAccountSelectionForTurn, tryAdmitTurn } = await import("./lifecycle"); + // The probe owns its claim only until selection completes, not for the client turn. + const probeLease = logIds?.turnAdmissionLease && candidates.length ? tryAdmitTurn() : undefined; + const errors = await import("../codex/auth-context"); + try { + req.signal.throwIfAborted(); + if (logIds?.turnAdmissionLease && candidates.length && !probeLease) { + // No independent admission means availability is unknown; preserve the bridge. + nativeSelector.webSearchBackendAvailable = true; + } else { + if (candidates.some(candidate => candidate.accountMode === "direct") && probeLease) { + if (!tryClaimNativeMainProfileForTurn(probeLease)) { + nativeSelector.webSearchBackendAvailable = true; + } else { + const { getMainAccountToken } = await import("../codex/main-account"); + const token = getMainAccountToken(); + if (token) { + const auth = captureExplicitOpenAiCallerAuth(new Headers({ authorization: `Bearer ${token.accessToken}`, "chatgpt-account-id": token.chatgptAccountId }), replayConfig); + if (auth) { headers.set("authorization", auth.authorization); headers.set("chatgpt-account-id", auth.chatgptAccountId); } + } + } + } + openAiSidecar = await resolveFirstUsableOpenAiSidecar(candidates, headers, replayConfig, { + admission: logIds?.admission, codexAuthPolicy: config, signal: req.signal, + beginCodexAccountSelection: codexAccountSelectionForTurn(probeLease ?? undefined), + }); + req.signal.throwIfAborted(); + // The bridge can inject main auth into pool selection. Only a negative pool probe + // needs this extra read; an unavailable claim or present token leaves it uncertain. + if (!openAiSidecar && probeLease && candidates.some(candidate => candidate.accountMode === "pool")) { + if (!tryClaimNativeMainProfileForTurn(probeLease)) nativeSelector.webSearchBackendAvailable = true; + else { + const { getMainAccountToken } = await import("../codex/main-account"); + if (getMainAccountToken()) nativeSelector.webSearchBackendAvailable = true; + } + } + } + } catch (err) { + // Refresh wraps cancellation in CodexAuthContextError; a cancelled request must not continue. + if (req.signal.aborted) { + openAiSidecar?.releaseProbeLease?.(); + throw req.signal.reason ?? err; + } + if (!(err instanceof errors.CodexPoolAuthenticationError) && !(err instanceof errors.CodexAuthContextError) + && !(err instanceof errors.CodexAccountCooldownError) && !(err instanceof errors.CodexThreadAffinityExpiredError) + && !(err instanceof errors.CodexMainProfileDrainingError)) { + openAiSidecar?.releaseProbeLease?.(); + throw err; + } + if (probeLease && candidates.some(candidate => candidate.accountMode === "pool")) nativeSelector.webSearchBackendAvailable = true; + } finally { probeLease?.release(); } + } + try { + nativeSelector.webSearchBackendAvailable ??= !!planWebSearch(replayConfig, parsed, false, settledRoute.provider, settledRoute.modelId, openAiSidecar, { + admission: logIds?.admission, codexAuthPolicy: config, providerName: settledRoute.providerName, + }); + } finally { + openAiSidecar?.releaseProbeLease?.(); + } + } const nativeDecline = settledRoute && isRec(anthropicBody) ? nativeMessagesDeclineReason(settledRoute, anthropicBody, config, nativeSelector) : "rollout-disabled"; diff --git a/src/server/messages-native-eligibility.ts b/src/server/messages-native-eligibility.ts index bf878853d6b..79fc3fb56fe 100644 --- a/src/server/messages-native-eligibility.ts +++ b/src/server/messages-native-eligibility.ts @@ -2,8 +2,8 @@ * Whether a settled Messages route may take the managed native Messages lane (PF-08). * * PURE: reads the route, the body's structure and config, and nothing else. The Messages ingress, - * `count_tokens` and the protocol planner (`src/protocols/plan-snapshot.ts`) all ask here, so a - * preview, a count and the real send cannot disagree about the lane. + * `count_tokens` and the protocol planner (`src/protocols/plan-snapshot.ts`) all ask here; + * previews and counts remain conservative when runtime availability is unknown. */ import { anthropicBodyElidesBlockedSkill } from "../claude/inbound"; import { isClaudeWebSearchToolName } from "../claude/outbound"; @@ -48,6 +48,8 @@ export interface NativeMessagesSelector { * and judges OAuth from config alone. */ oauthFailoverQuorum?: boolean; + /** Runtime sidecar plan availability; absent for config-only callers. */ + webSearchBackendAvailable?: boolean; } type Rec = Record; @@ -59,8 +61,7 @@ function isRec(value: unknown): value is Rec { * Whether the bridge could hand this request to the web-search sidecar: a `web_search*` server * tool the tool choice does not exclude, with the sidecar not disabled in the Claude replay config * (`buildClaudeReplayConfig` merges `claudeCode.webSearchSidecar` over the global block). Whether - * a backend credential then exists is decided at dispatch and is not read here, so this errs - * toward the bridge, which is today's behavior for every such request. + * a backend credential exists is supplied separately by the sending ingress. */ function webSearchSidecarMayEngage(body: Readonly, config: OcxConfig): boolean { const sidecar = { ...config.webSearchSidecar, ...config.claudeCode?.webSearchSidecar }; @@ -89,7 +90,7 @@ function bridgeOnlyPolicyApplies( ): boolean { if (resolvePinnedEffort(route, selector.routeSelector, config) !== undefined) return true; if (anthropicBodyElidesBlockedSkill(body, selector.claudeCode ?? config.claudeCode)) return true; - return webSearchSidecarMayEngage(body, config); + return selector.webSearchBackendAvailable !== false && webSearchSidecarMayEngage(body, config); } /** diff --git a/structure/data-planes/protocol-paths.md b/structure/data-planes/protocol-paths.md index 44439672f09..277fbcc2d4b 100644 --- a/structure/data-planes/protocol-paths.md +++ b/structure/data-planes/protocol-paths.md @@ -373,10 +373,10 @@ it), and `bridge-only-policy` when operator policy that only the translated path engage: a pinned reasoning effort for the route (`resolvePinnedEffort`, read with the translated body's model id as the bridge reads it), a blocked-skill bundle the translator would elide (`anthropicBodyElidesBlockedSkill` in `src/claude/inbound.ts`), or a `web_search*` server tool the -web-search sidecar could serve (not excluded by `tool_choice`, sidecar not disabled in the Claude -replay config; backend credentials are decided at dispatch, so this errs toward the bridge). The -ingress, `count_tokens` and the planner all ask it. The planner can judge only the config-and-route -parts: skill elision and web search depend on body content no feature describes. +web-search sidecar could serve (not excluded by `tool_choice`, sidecar enabled). The cancellable ingress +probe uses the bridge's credential-free affinity headers, existing plan and authorized backend, with a +short-lived selection lease; uncertain admission keeps the bridge. Without a runnable plan search alone +does not decline native Messages. `count_tokens` and the planner lack runtime facts and remain conservative. With the switch on, a declined route re-marks its bridge entry with the decline reason (after any `effort-row` / `fast-row`), so the trace says why the bridge was taken; with it off nothing is diff --git a/tests/claude-integration/claude-translated-server-tools.test.ts b/tests/claude-integration/claude-translated-server-tools.test.ts index 9344f8d8ca8..267f534d2a0 100644 --- a/tests/claude-integration/claude-translated-server-tools.test.ts +++ b/tests/claude-integration/claude-translated-server-tools.test.ts @@ -1,10 +1,19 @@ -import { afterAll, beforeAll, describe, expect, test } from "bun:test"; +import { afterAll, beforeAll, describe, expect, spyOn, test } from "bun:test"; import { mkdtempSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { createAnthropicAdapter } from "../../src/adapters/anthropic"; import { webSearchPairFromItem } from "../../src/claude/outbound"; import { handleClaudeMessages } from "../../src/server/claude-messages"; +import * as openAiSidecar from "../../src/providers/openai-sidecar"; +import * as mainAdmission from "../../src/codex/native-main-admission"; +import * as mainAccount from "../../src/codex/main-account"; +import { CodexAuthContextError, codexPoolAffinityKey } from "../../src/codex/auth-context"; +import { saveCodexAccountCredential } from "../../src/codex/account-store"; +import { updateAccountQuota, clearAccountQuota } from "../../src/codex/quota"; +import { bindThreadAffinity, CODEX_THREAD_AFFINITY_IDLE_TTL_MS } from "../../src/codex/routing/thread-affinity"; +import { clearThreadAccountMap, clearCodexUpstreamHealth } from "../../src/codex/routing"; +import { getNativeMainProfileRequestCount, tryAdmitTurn } from "../../src/server/lifecycle"; import type { OcxConfig, OcxParsedRequest, OcxProviderConfig } from "../../src/types"; import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; import { removeTreeWithRetry } from "../helpers/remove-tree"; @@ -68,10 +77,11 @@ async function claudeTurn( frames: Record[], target: "anth" | "gpt" = "anth", messages: unknown[] = [{ role: "user", content: "What did Bun release? https://bun.sh/blog" }], - { stream = true, direct = false }: { stream?: boolean; direct?: boolean } = {}, + { stream = true, direct = false, native = false, searchBackend, searchTool = WEB_SEARCH, pool = false, directSearch = false, requestHeaders = {}, signal, onUpstream, lease }: { stream?: boolean; direct?: boolean; native?: boolean; searchBackend?: { backend: "exa"; exaApiKey: string }; searchTool?: typeof WEB_SEARCH & { allowed_domains?: string[] }; pool?: boolean; directSearch?: boolean; requestHeaders?: Record; signal?: AbortSignal; onUpstream?: () => void; lease?: ReturnType } = {}, ): Promise<{ wire: Record; clientSse: string }> { const wires: string[] = []; const fetch = (async (_input: unknown, init?: RequestInit) => { + onUpstream?.(); wires.push(String(init?.body)); if (target === "gpt") { return new Response(`data: ${JSON.stringify({ id: "c", object: "chat.completion.chunk", choices: [{ index: 0, delta: { content: "ok" }, finish_reason: "stop" }] })}\n\ndata: [DONE]\n\n`, @@ -81,12 +91,14 @@ async function claudeTurn( }) as typeof globalThis.fetch; const config = { port: 0, defaultProvider: "anth", + ...(pool ? { codexAccounts: [{ id: "fixture-a", email: "a@example.test", isMain: false }, { id: "fixture-b", email: "b@example.test", isMain: false }], activeCodexAccountId: "fixture-b" } : {}), claudeCode: { nativePassthrough: false }, - webSearchSidecar: { enabled: false }, - ...(direct ? { protocols: { rollout: { directEncoders: true } } } : {}), + ...(searchBackend ? { webSearchSidecar: searchBackend } : native ? {} : { webSearchSidecar: { enabled: false } }), + ...(direct || native ? { protocols: { rollout: { directEncoders: direct, managedMessagesNative: native } } } : {}), providers: { anth: { adapter: "anthropic", baseUrl, authMode: "key", apiKey: "sk-ant-fixture", models: [MODEL], fetch }, gpt: { adapter: "openai-chat", baseUrl: "https://gpt.invalid/v1", authMode: "key", apiKey: "sk-fixture", models: ["g"], fetch }, + ...(pool ? { openai: { adapter: "openai-responses", baseUrl: "https://chatgpt.com/backend-api/codex", authMode: "forward", codexAccountMode: directSearch ? "direct" : "pool" } } : {}), }, combos: { waterfall: { strategy: "failover", targets: target === "anth" ? [{ provider: "anth", model: MODEL }, { provider: "gpt", model: "g" }] @@ -94,13 +106,13 @@ async function claudeTurn( } as unknown as OcxConfig; const response = await handleClaudeMessages(new Request("http://localhost/v1/messages", { method: "POST", - headers: { "content-type": "application/json" }, + headers: { "content-type": "application/json", ...requestHeaders }, signal, body: JSON.stringify({ - model: direct ? `anth/${MODEL}` : "combo/waterfall", max_tokens: 64, stream, - tools: [{ name: "Read", description: "read", input_schema: { type: "object", properties: {} } }, WEB_SEARCH, WEB_FETCH], + model: direct || native ? `anth/${MODEL}` : "combo/waterfall", max_tokens: 64, stream, + tools: [{ name: "Read", description: "read", input_schema: { type: "object", properties: {} } }, searchTool, WEB_FETCH], messages, }), - }), config, { model: "", provider: "" }); + }), config, { model: "", provider: "" }, lease ? { requestId: "fixture", start: Date.now(), turnAdmissionLease: lease } : undefined); expect(response.status).toBe(200); const clientSse = await response.text(); expect(wires).toHaveLength(1); @@ -112,6 +124,138 @@ function clientFrames(text: string): Record[] { } describe("translated Claude lane carries typed web tools to Anthropic", () => { + for (const expired of [false, true]) { + test(`real pool resolves an ${expired ? "expired" : "available"} affinity binding`, async () => { + clearThreadAccountMap(); clearCodexUpstreamHealth(); clearAccountQuota(); + for (const id of ["fixture-a", "fixture-b"]) { + saveCodexAccountCredential(id, { accessToken: `access-${id}`, refreshToken: `refresh-${id}`, expiresAt: Date.now() + 3_600_000, chatgptAccountId: `account-${id}` }); + updateAccountQuota(id, 10, 10); + } + const requestHeaders = { "session-id": `real-${expired}`, "thread-id": "child", "x-codex-parent-thread-id": "parent" }; + const key = codexPoolAffinityKey(new Headers(requestHeaders))!; + bindThreadAffinity(key, "fixture-a", Date.now() - (expired ? CODEX_THREAD_AFFINITY_IDLE_TTL_MS + 1 : 0)); + const realResolver = openAiSidecar.resolveFirstUsableOpenAiSidecar; + let observed: string | undefined; + const resolver = spyOn(openAiSidecar, "resolveFirstUsableOpenAiSidecar").mockImplementation(async (...args) => { + const selected = await realResolver(...args); + if (!observed) observed = selected?.authContext.accountId ?? undefined; + return selected; + }); + const network = spyOn(globalThis, "fetch").mockImplementation(async () => { throw new Error("unexpected pool network request"); }); + try { + const { wire } = await claudeTurn("https://compat.example.com", OK_TURN, "anth", undefined, { native: true, pool: true, requestHeaders }); + expect(observed).toBe(expired ? undefined : "fixture-a"); + expect((wire.tools as Record[]).some(t => t.type === WEB_SEARCH.type)).toBe(expired); + expect(network).not.toHaveBeenCalled(); + } finally { network.mockRestore(); resolver.mockRestore(); clearThreadAccountMap(); clearCodexUpstreamHealth(); clearAccountQuota(); } + }); + } + for (const available of [true, false]) { + test(`pool probe uses the pinned affinity when ${available ? "available" : "unavailable"}`, async () => { + const requestHeaders = { "session-id": "fixture-session", "thread-id": "fixture-child", "x-codex-parent-thread-id": "fixture-parent", authorization: "Bearer client-secret" }; + const expectedHeaders = new Headers(requestHeaders); expectedHeaders.delete("authorization"); + const key = codexPoolAffinityKey(expectedHeaders); + const resolver = spyOn(openAiSidecar, "resolveFirstUsableOpenAiSidecar").mockImplementation(async (candidates, headers) => { + expect(codexPoolAffinityKey(headers)).toBe(key); + expect(headers.get("authorization")).toBeNull(); + return available ? { ...candidates[0]!, authContext: { kind: "main", accountId: null }, headers } : undefined; + }); + try { + const { wire } = await claudeTurn("https://compat.example.com", OK_TURN, "anth", undefined, { native: true, pool: true, requestHeaders }); + expect((wire.tools as Record[]).some(t => t.type === WEB_SEARCH.type)).toBe(!available); + } finally { resolver.mockRestore(); } + }); + } + + test("pool probe forwards cancellation to account resolution", async () => { + const controller = new AbortController(); + let seenSignal: AbortSignal | undefined; + const resolver = spyOn(openAiSidecar, "resolveFirstUsableOpenAiSidecar").mockImplementation(async (_candidates, _headers, _config, options) => { + seenSignal = options?.signal; + controller.abort(); + options?.signal?.throwIfAborted(); + return undefined; + }); + try { + await expect(claudeTurn("https://compat.example.com", OK_TURN, "anth", undefined, { native: true, pool: true, signal: controller.signal })).rejects.toThrow(); + expect(seenSignal?.aborted).toBe(true); + } finally { resolver.mockRestore(); } + }); + + test("cancellation wrapped by a credential refresh error still ends the request", async () => { + const controller = new AbortController(); + let sends = 0; + const resolver = spyOn(openAiSidecar, "resolveFirstUsableOpenAiSidecar").mockImplementation(async () => { + controller.abort(); + throw new CodexAuthContextError("fixture-account", controller.signal.reason); + }); + try { + await expect(claudeTurn("https://compat.example.com", OK_TURN, "anth", undefined, { native: true, pool: true, signal: controller.signal, onUpstream: () => { sends++; } })).rejects.toThrow(); + expect(sends).toBe(0); + } finally { resolver.mockRestore(); } + }); + + test("cancellation after account selection releases its probe lease", async () => { + const controller = new AbortController(); + let released = 0; + const resolver = spyOn(openAiSidecar, "resolveFirstUsableOpenAiSidecar").mockImplementation(async (candidates, headers) => { + controller.abort(); + return { ...candidates[0]!, authContext: { kind: "main", accountId: null }, headers, releaseProbeLease: () => { released++; } }; + }); + try { + await expect(claudeTurn("https://compat.example.com", OK_TURN, "anth", undefined, { native: true, pool: true, signal: controller.signal })).rejects.toThrow(); + expect(released).toBe(1); + } finally { resolver.mockRestore(); } + }); + + test("a pool probe releases its selected main claim when resolution ends", async () => { + const lease = tryAdmitTurn(); expect(lease).not.toBeNull(); + const resolver = spyOn(openAiSidecar, "resolveFirstUsableOpenAiSidecar").mockImplementation(async (_candidates, _headers, _config, options) => { + const selection = options?.beginCodexAccountSelection?.(); + expect(selection?.claimMainProfile()).toBe(true); + selection?.release(); + throw new Error("fixture probe ended"); + }); + const claim = spyOn(mainAdmission, "tryClaimNativeMainProfileForTurn"); + try { + await expect(claudeTurn("https://compat.example.com", OK_TURN, "anth", undefined, { native: true, pool: true, lease })).rejects.toThrow("fixture probe ended"); + expect(getNativeMainProfileRequestCount()).toBe(0); + expect(claim).not.toHaveBeenCalled(); + } finally { claim.mockRestore(); resolver.mockRestore(); lease?.release(); } + }); + + test("a direct probe releases its main claim after the token read", async () => { + const lease = tryAdmitTurn(); expect(lease).not.toBeNull(); + const token = spyOn(mainAccount, "getMainAccountToken").mockReturnValue(null); + try { + await claudeTurn("https://compat.example.com", OK_TURN, "anth", undefined, { native: true, pool: true, directSearch: true, lease, + onUpstream: () => expect(getNativeMainProfileRequestCount()).toBe(0) }); + expect(token).toHaveBeenCalledTimes(1); + } finally { token.mockRestore(); lease?.release(); } + }); + + test("a pool miss with possible main auth conservatively stays on the bridge", async () => { + const lease = tryAdmitTurn(); expect(lease).not.toBeNull(); + const resolver = spyOn(openAiSidecar, "resolveFirstUsableOpenAiSidecar").mockResolvedValue(undefined); + const token = spyOn(mainAccount, "getMainAccountToken").mockReturnValue({ accessToken: "fixture-token", chatgptAccountId: "fixture-account" }); + try { + const { wire } = await claudeTurn("https://compat.example.com", OK_TURN, "anth", undefined, { native: true, pool: true, lease }); + expect((wire.tools as Record[]).some(t => t.type === WEB_SEARCH.type)).toBe(false); + } finally { token.mockRestore(); resolver.mockRestore(); lease?.release(); } + }); + test("native compatible upstream receives search and returns its result without a sidecar", async () => { + const searchTool = { ...WEB_SEARCH, allowed_domains: ["bun.sh"] }; + const { wire, clientSse } = await claudeTurn("https://compat.example.com", SEARCH_TURN, "anth", undefined, { native: true, searchTool }); + expect(wire.tools).toContainEqual(searchTool); + expect(clientSse).toContain('"type":"web_search_tool_result"'); + expect(clientSse).toContain("Bun blog"); + }); + test("an available sidecar keeps the compatible route on the bridge", async () => { + const { wire } = await claudeTurn("https://compat.example.com", OK_TURN, "anth", undefined, + { native: true, searchBackend: { backend: "exa", exaApiKey: "fixture" } }); + expect(wire.tools).not.toContainEqual(WEB_SEARCH); + expect((wire.tools as Record[]).map(tool => tool.name)).toContain("web_search"); + }); test("api.anthropic.com receives web_search and web_fetch as declared", async () => { const { wire } = await claudeTurn("https://api.anthropic.com", OK_TURN); const tools = wire.tools as Record[]; diff --git a/tests/responses/messages-native-bridge-policy.test.ts b/tests/responses/messages-native-bridge-policy.test.ts index 61ab7c4815b..2604a94ed2f 100644 --- a/tests/responses/messages-native-bridge-policy.test.ts +++ b/tests/responses/messages-native-bridge-policy.test.ts @@ -112,6 +112,17 @@ describe("web-search sidecar", () => { .toBe("bridge-only-policy"); }); + test("runtime availability changes only the search policy", () => { + expect(nativeMessagesDeclineReason(route(), searchBody, config(), { webSearchBackendAvailable: false })).toBeUndefined(); + expect(nativeMessagesDeclineReason(route(), searchBody, config(), { webSearchBackendAvailable: true })).toBe("bridge-only-policy"); + expect(nativeMessagesDeclineReason(route(), searchBody, config({ protocols: { rollout: { managedMessagesNative: false } } }), + { webSearchBackendAvailable: false })).toBe("rollout-disabled"); + expect(nativeMessagesDeclineReason(route(), { ...searchBody, tool_choice: { type: "none" } }, config(), + { webSearchBackendAvailable: true })).toBeUndefined(); + expect(nativeMessagesDeclineReason(route({ pinnedReasoningEffort: "high" }), searchBody, config(), + { webSearchBackendAvailable: false })).toBe("bridge-only-policy"); + }); + test("a disabled sidecar or a tool choice that excludes search stays native", () => { const off = config({ claudeCode: { webSearchSidecar: { enabled: false } } } as Partial); expect(nativeMessagesDeclineReason(route(), searchBody, off)).toBeUndefined(); From 8063be23e47822240b45c0456f3243102d4a0d9d Mon Sep 17 00:00:00 2001 From: JUN Date: Fri, 30 Oct 2026 21:28:28 +0900 Subject: [PATCH 25/42] fix(providers): keep the quota report cache when pool readings are only restamped (#7992) The Pool aggregation part of the provider quota report cache key hashed each account's quota updatedAt, which every proxied Codex response restamps, so the cache never committed and a transient probe failure dropped the last-good row. The key now carries the fresh/stale verdict instead, plus shortObservedAt only for an exhausted reset-less short window whose blocking verdict depends on it. Closes #7956. --- src/providers/quota.ts | 4 +- src/providers/quota/report-cache.ts | 21 ++++- tests/providers/provider-quota.test.ts | 103 ++++++++++++++++++++++++- 3 files changed, 121 insertions(+), 7 deletions(-) diff --git a/src/providers/quota.ts b/src/providers/quota.ts index 06292acc709..4526b82a4d1 100644 --- a/src/providers/quota.ts +++ b/src/providers/quota.ts @@ -151,8 +151,8 @@ async function maybeFetchProviderQuota( * * No previous snapshot is passed. The observer keeps its own persisted last-seen map, * because `previous` here is bound only when `cache.key === key` and the key digest at - * cacheKeyWithAggregationState includes quota values and updatedAt — so it is empty exactly - * when a reset happened. + * cacheKeyWithAggregationState includes quota values and their freshness — so it is empty + * exactly when a reset happened. * * Account identity is resolved SYNCHRONOUSLY, before any await. Two reasons, both observed: * pool failover rewrites `activeAccountId` during request routing diff --git a/src/providers/quota/report-cache.ts b/src/providers/quota/report-cache.ts index f3fad7a635d..7350263d6cf 100644 --- a/src/providers/quota/report-cache.ts +++ b/src/providers/quota/report-cache.ts @@ -3,7 +3,7 @@ import { providerDecisionWindows } from "../quota-decision-publication"; import type { DecisionQuotaWindow } from "../quota-decision-snapshot"; import { createHash } from "node:crypto"; import { effectiveCodexAuthAccountId, listCodexAuthAccountsSnapshot } from "../../codex/auth-api"; -import { withoutRetiredCodexQuota, type StoredAccountQuota } from "../../codex/quota"; +import { CODEX_EXHAUSTED_USAGE_PERCENT, withoutRetiredCodexQuota, type StoredAccountQuota } from "../../codex/quota"; import { isMainAccountIdentityGenerationLive } from "../../codex/main-account-cache"; import { codexPlanKey } from "../../codex/plan"; import { resolveProviderApiKey } from "../key-store"; @@ -154,16 +154,28 @@ export function hasCodexPoolProvider(config: OcxConfig): boolean { )); } -function quotaSignatureValue(quota: CodexCapacityQuota | null): unknown { +/** + * Signature of one pool account's presentation state. Every proxied Codex response restamps the + * observation, so a raw `updatedAt` in the key stopped the report cache from ever committing + * while traffic flowed (#7956). The key carries the fresh/stale verdict aggregation derives + * from it instead; value changes and freshness transitions still miss the cache. A displayed + * window `updatedAt` can therefore lag a restamp by up to the report cache TTL. The exception is + * an exhausted short window without a reset: its `shortObservedAt` alone decides whether it + * still blocks, over a window as short as that TTL, so that clock stays in the key. + */ +function quotaSignatureValue(quota: CodexCapacityQuota | null, now: number): unknown { if (!quota) return null; + const resetless = !(typeof quota.fiveHourResetAt === "number" && quota.fiveHourResetAt > 0); + const observationDecides = resetless && (quota.fiveHourPercent ?? 0) >= CODEX_EXHAUSTED_USAGE_PERCENT; return { fiveHourPercent: quota.fiveHourPercent, fiveHourResetAt: quota.fiveHourResetAt, + ...(observationDecides ? { shortObservedAt: quota.shortObservedAt } : {}), weeklyPercent: quota.weeklyPercent, weeklyResetAt: quota.weeklyResetAt, monthlyPercent: quota.monthlyPercent, monthlyResetAt: quota.monthlyResetAt, - updatedAt: quota.updatedAt, + fresh: Number.isFinite(quota.updatedAt) && now - quota.updatedAt <= CODEX_CAPACITY_MAX_QUOTA_AGE_MS, customWindows: [...(quota.customWindows ?? [])] .map(window => ({ label: window.label, percent: window.percent, resetAt: window.resetAt })) .sort((a, b) => a.label.localeCompare(b.label)), @@ -204,13 +216,14 @@ export function cacheKeyWithAggregationState( try { const activeId = effectiveCodexAuthAccountId(config); const snapshot = await (prefetchedSnapshot ?? listCodexAuthAccountsSnapshot(config, false)); + const now = Date.now(); const rows = snapshot.accounts.map(account => ({ isMain: account.isMain, active: account.id === activeId, plan: codexPlanKey(account.plan) ?? null, paused: account.paused, needsReauth: account.needsReauth === true, - quota: quotaSignatureValue(providerQuotaFromCodexQuota(account.quota)), + quota: quotaSignatureValue(providerQuotaFromCodexQuota(account.quota), now), })); const canonicalRows = rows.map(row => JSON.stringify(row)).sort(); const digest = createHash("sha256").update(JSON.stringify(canonicalRows)).digest("hex").slice(0, 24); diff --git a/tests/providers/provider-quota.test.ts b/tests/providers/provider-quota.test.ts index 9de61147f58..43d5fb32023 100644 --- a/tests/providers/provider-quota.test.ts +++ b/tests/providers/provider-quota.test.ts @@ -1,4 +1,4 @@ -import { afterEach, beforeEach, describe, expect, spyOn, test } from "bun:test"; +import { afterEach, beforeEach, describe, expect, setSystemTime, spyOn, test } from "bun:test"; import { existsSync, mkdtempSync, mkdirSync, readFileSync, rmSync, writeFileSync } from "node:fs"; import { removeTreeWithRetry } from "../helpers/remove-tree"; import { tmpdir } from "node:os"; @@ -21,6 +21,7 @@ import { setAntigravityAccountQuotaTransportForTests, setProviderQuotaBeforePublishForTests, } from "../../src/providers/quota"; +import { cacheKeyWithAggregationState } from "../../src/providers/quota/report-cache"; import type { OcxConfig } from "../../src/types"; import { clearComboTargetCooldowns, coolComboTarget, pickComboTarget, pickComboTargetWithWait } from "../../src/combos"; import { routedProviderConfig } from "../../src/router"; @@ -98,6 +99,7 @@ beforeEach(() => { }); afterEach(() => { + setSystemTime(); clearComboTargetCooldowns(); for (const key of proxyKeys) { if (originalProxyEnv[key] === undefined) delete process.env[key]; @@ -2428,6 +2430,105 @@ describe("fetchProviderQuotaReports", () => { expect((await fetchProviderQuotaReports(config)).reports[0]?.aggregation?.weekly?.usedPercent).toBeCloseTo((20 * 20 + 11) / 21, 8); }); + test("restamped pool observations keep the report cache and last-good rows (#7956)", async () => { + await saveCredential("anthropic", { access: "synthetic-claude-access", refresh: "synthetic-claude-refresh", expires: Date.now() + 3600_000 }); + saveCodexAccountCredential("added", { + accessToken: "added-access", refreshToken: "added-refresh", + expiresAt: Date.now() + 3600_000, chatgptAccountId: "added-chatgpt-id", + }); + const config = testConfig(); + config.providers = { openai: config.providers.openai, anthropic: config.providers.anthropic }; + config.codexAccounts = [{ id: "added", email: "a@example.test", plan: "pro", isMain: false }]; + let anthropicCalls = 0; + let anthropicDown = false; + globalThis.fetch = (async (input: RequestInfo | URL, init?: RequestInit) => { + if (String(input) === "https://api.anthropic.com/api/oauth/usage") { + anthropicCalls += 1; + if (anthropicDown) return new Response("unavailable", { status: 503 }); + return Response.json({ five_hour: { utilization: 41, resets_at: new Date(Date.now() + 3600_000).toISOString() } }); + } + const added = (init?.headers as Record | undefined)?.["ChatGPT-Account-Id"] === "added-chatgpt-id"; + return Response.json({ plan_type: added ? "pro" : "plus", + rate_limit: { secondary_window: { used_percent: added ? 30 : 11, reset_at: 1_999_000_000 } } }); + }) as typeof fetch; + + await fetchProviderQuotaReports(config, true); + expect(anthropicCalls).toBe(1); + // A proxied Codex response re-observes unchanged values with a new timestamp. + setSystemTime(new Date(Date.now() + 1_000)); + updateAccountQuota("added", 30, 1_999_000_000); + const cached = await fetchProviderQuotaReports(config); + expect(anthropicCalls).toBe(1); + expect(cached.reports.map(report => report.provider).sort()).toEqual(["anthropic", "openai"]); + + // A transient Anthropic failure on the next probe keeps its bounded last-good row. + anthropicDown = true; + setSystemTime(new Date(Date.now() + 1_000)); + updateAccountQuota("added", 30, 1_999_000_000); + const refreshed = await fetchProviderQuotaReports(config, true); + expect(anthropicCalls).toBe(2); + expect(refreshed.reports.find(report => report.provider === "anthropic")?.quota.fiveHourPercent).toBe(41); + }); + + test("a restamp during an outstanding probe still lets that probe commit (#7956)", async () => { + await saveCredential("anthropic", { access: "synthetic-claude-access", refresh: "synthetic-claude-refresh", expires: Date.now() + 3600_000 }); + saveCodexAccountCredential("added", { + accessToken: "added-access", refreshToken: "added-refresh", + expiresAt: Date.now() + 3600_000, chatgptAccountId: "added-chatgpt-id", + }); + const config = testConfig(); + config.providers = { openai: config.providers.openai, anthropic: config.providers.anthropic }; + config.codexAccounts = [{ id: "added", email: "a@example.test", plan: "pro", isMain: false }]; + let anthropicCalls = 0; + let release!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + let probing!: () => void; + const started = new Promise(resolve => { probing = resolve; }); + globalThis.fetch = (async (input: RequestInfo | URL, init?: RequestInit) => { + if (String(input) === "https://api.anthropic.com/api/oauth/usage") { + anthropicCalls += 1; + probing(); + await gate; + return Response.json({ five_hour: { utilization: 41, resets_at: new Date(Date.now() + 3600_000).toISOString() } }); + } + const added = (init?.headers as Record | undefined)?.["ChatGPT-Account-Id"] === "added-chatgpt-id"; + return Response.json({ plan_type: added ? "pro" : "plus", + rate_limit: { secondary_window: { used_percent: added ? 30 : 11, reset_at: 1_999_000_000 } } }); + }) as typeof fetch; + + const first = fetchProviderQuotaReports(config, true); + await started; + setSystemTime(new Date(Date.now() + 1_000)); + updateAccountQuota("added", 30, 1_999_000_000); + release(); + await first; + const next = await fetchProviderQuotaReports(config); + expect(anthropicCalls).toBe(1); + expect(next.reports.map(report => report.provider).sort()).toEqual(["anthropic", "openai"]); + }); + + test("only an exhausted reset-less short window keeps its observation clock in the key (#7956)", async () => { + const config = testConfig(); + config.providers = { openai: config.providers.openai }; + let quota: Partial = {}; + const poolSpy = spyOn(authApi, "listCodexAuthAccountsSnapshot").mockImplementation(async () => ({ + mainIdentityGeneration: captureMainAccountIdentityGeneration(), + accounts: [{ id: "__main__", isMain: true, paused: false, plan: "plus", quota }], + } as unknown as Awaited>)); + const keyFor = async (next: Partial) => { quota = next; return await cacheKeyWithAggregationState(config); }; + try { + const now = Date.now(); + const withReset = { weeklyPercent: 20, shortPercent: 100, shortResetAt: Math.floor(now / 1000) + 3600 }; + expect(await keyFor({ ...withReset, shortObservedAt: now, updatedAt: now })) + .toBe(await keyFor({ ...withReset, shortObservedAt: now + 5, updatedAt: now + 5 })); + const resetless = { weeklyPercent: 20, shortPercent: 100 }; + expect(await keyFor({ ...resetless, shortObservedAt: now, updatedAt: now })) + .not.toBe(await keyFor({ ...resetless, shortObservedAt: now + 5, updatedAt: now + 5 })); + } finally { + poolSpy.mockRestore(); + } + }); + test("ordinary fetch reflects runtime reauthentication state", async () => { saveCodexAccountCredential("added", { accessToken: "added-access", refreshToken: "added-refresh", From d9e0c057881200d32b0398c5a1bc38144c1da73e Mon Sep 17 00:00:00 2001 From: JUN Date: Fri, 30 Oct 2026 21:30:38 +0900 Subject: [PATCH 26/42] fix(responses): promote additional_tools to top-level tools for xAI and strict gateways (#7983) Carries #7637. Co-authored-by: Omar <13569609+omar391@users.noreply.github.com> --- .../fr/reference/configuration/providers.md | 4 +- .../ja/reference/configuration/providers.md | 4 +- .../ko/reference/configuration/providers.md | 4 +- .../docs/reference/configuration/keychain.md | 55 +++ .../docs/reference/configuration/providers.md | 60 +-- .../ru/reference/configuration/providers.md | 4 +- .../tr/reference/configuration/providers.md | 4 +- .../reference/configuration/providers.md | 4 +- .../reference/configuration/providers.md | 4 +- scripts/test-layout/layout-providers.json | 3 + .../openai-responses/muse-tool-catalog.ts | 10 +- src/adapters/openai-responses/passthrough.ts | 32 +- src/adapters/openai-responses/prompt-cache.ts | 4 +- .../openai-responses/request-strips.ts | 11 +- .../openai-responses/tool-output-recovery.ts | 29 +- src/adapters/openai-responses/tool-schema.ts | 28 +- src/adapters/opencode-go-additional-tools.ts | 45 +- src/adapters/routed-agent-messages.ts | 3 + src/adapters/xai-web-search.ts | 9 +- src/adapters/zed-additional-tools.ts | 14 +- src/adapters/zed.ts | 7 +- src/config/schema/leaf-validators.ts | 1 + src/providers/model-rename-fields.ts | 1 + src/responses/custom-tool-compat.ts | 3 +- src/responses/item-provenance.ts | 79 ++++ src/responses/namespace-tool-compat.ts | 24 +- src/server/auth-cors.ts | 1 + .../management/provider-overwrite-carry.ts | 10 + src/server/management/provider-routes.ts | 1 + src/server/responses-undeclared-tool-guard.ts | 4 + src/server/responses/passthrough-dispatch.ts | 7 +- src/types/provider.ts | 8 + structure/gui-and-management-api.md | 2 +- structure/transports/responses-wire-shapes.md | 6 +- structure/transports/responses.md | 33 +- tests/adapters/routed-agent-messages.test.ts | 12 + tests/fixtures/test-layout-expected.json | 3 + .../xai/xai-additional-tools.test.ts | 421 ++++++++++++++++++ .../xai/xai-loaded-tool-provenance.test.ts | 128 ++++++ .../xai/xai-replay-provenance.test.ts | 94 ++++ tests/providers/zed-hosted-provider.test.ts | 22 + .../openai-responses-passthrough.test.ts | 16 +- .../responses-muse-tool-choice.test.ts | 16 + .../management-provider-compat-carry.test.ts | 35 ++ .../server-xai-responses-streaming.test.ts | 5 +- 45 files changed, 1112 insertions(+), 158 deletions(-) create mode 100644 docs-site/src/content/docs/reference/configuration/keychain.md create mode 100644 src/responses/item-provenance.ts create mode 100644 tests/providers/xai/xai-additional-tools.test.ts create mode 100644 tests/providers/xai/xai-loaded-tool-provenance.test.ts create mode 100644 tests/providers/xai/xai-replay-provenance.test.ts diff --git a/docs-site/src/content/docs/fr/reference/configuration/providers.md b/docs-site/src/content/docs/fr/reference/configuration/providers.md index 85650d22e13..6b7439f0b0c 100644 --- a/docs-site/src/content/docs/fr/reference/configuration/providers.md +++ b/docs-site/src/content/docs/fr/reference/configuration/providers.md @@ -199,9 +199,9 @@ configuré sous [`claudeCode.authMode`](/fr/reference/configuration/server/#clau ### Ce qu'une sauvegarde de fournisseur conserve -`POST /api/providers` avec le nom d'un fournisseur existant remplace la ligne enregistrée par une ligne construite à partir de la requête. Le formulaire d'ajout/modification du tableau de bord ne peut pas envoyer tous les champs ; la sauvegarde conserve donc certains champs enregistrés que la requête omet. Dix d'entre eux décrivent le comportement d'un amont précis : `preserveReasoningContentModels`, `preserveResponsesReasoningContent`, `requiresReasoningPlaceholderModels`, `foldDeveloperRoleToSystem`, `singleLeadingSystemMessage`, `reasoningWireFormat`, `omitReasoningEffortWithToolsModels`, `retryOn429`, `transientRetryOn5xx`, `retryOnReset`. +`POST /api/providers` avec le nom d'un fournisseur existant remplace la ligne enregistrée par une ligne construite à partir de la requête. Le formulaire d'ajout/modification du tableau de bord ne peut pas envoyer tous les champs ; la sauvegarde conserve donc certains champs enregistrés que la requête omet. Onze d'entre eux décrivent le comportement d'un amont précis : `preserveReasoningContentModels`, `preserveResponsesReasoningContent`, `requiresReasoningPlaceholderModels`, `foldDeveloperRoleToSystem`, `singleLeadingSystemMessage`, `reasoningWireFormat`, `omitReasoningEffortWithToolsModels`, `promoteAdditionalTools`, `retryOn429`, `transientRetryOn5xx`, `retryOnReset`. -| Sauvegarde | Les dix réglages | `apiKeyPool` enregistré | +| Sauvegarde | Les onze réglages | `apiKeyPool` enregistré | | --- | --- | --- | | Même destination, champ omis | Valeur enregistrée conservée, y compris un `[]` ou un `false` explicite | Conservé | | Nouvelle destination, champ omis | Non conservée ; les valeurs par défaut du registre pour la nouvelle destination peuvent s'appliquer | Non conservé | diff --git a/docs-site/src/content/docs/ja/reference/configuration/providers.md b/docs-site/src/content/docs/ja/reference/configuration/providers.md index 8705a8b7c64..7ecffe3764a 100644 --- a/docs-site/src/content/docs/ja/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ja/reference/configuration/providers.md @@ -189,9 +189,9 @@ API キープロバイダーは、リテラルキーまたは環境参照を保 ### プロバイダーの保存で保持されるもの -既存のプロバイダー名で `POST /api/providers` を送ると、保存済みの行はリクエストから組み立てた行で置き換えられます。ダッシュボードの追加・編集フォームはすべてのフィールドを送れないため、リクエストが省略した保存済みフィールドの一部は保存時に引き継がれます。そのうち次の 10 つは、特定のアップストリームの挙動を記録するものです:`preserveReasoningContentModels`, `preserveResponsesReasoningContent`, `requiresReasoningPlaceholderModels`, `foldDeveloperRoleToSystem`, `singleLeadingSystemMessage`, `reasoningWireFormat`, `omitReasoningEffortWithToolsModels`, `retryOn429`, `transientRetryOn5xx`, `retryOnReset`。 +既存のプロバイダー名で `POST /api/providers` を送ると、保存済みの行はリクエストから組み立てた行で置き換えられます。ダッシュボードの追加・編集フォームはすべてのフィールドを送れないため、リクエストが省略した保存済みフィールドの一部は保存時に引き継がれます。そのうち次の 11 つは、特定のアップストリームの挙動を記録するものです:`preserveReasoningContentModels`, `preserveResponsesReasoningContent`, `requiresReasoningPlaceholderModels`, `foldDeveloperRoleToSystem`, `singleLeadingSystemMessage`, `reasoningWireFormat`, `omitReasoningEffortWithToolsModels`, `promoteAdditionalTools`, `retryOn429`, `transientRetryOn5xx`, `retryOnReset`。 -| 保存 | 10 つの設定 | 保存済みの `apiKeyPool` | +| 保存 | 11 つの設定 | 保存済みの `apiKeyPool` | | --- | --- | --- | | 同じ宛先、フィールド省略 | 保存済みの値を保持(明示的な `[]` や `false` を含む) | 保持 | | 新しい宛先、フィールド省略 | 保持しない。新しい宛先のレジストリ既定値が適用されることがあります | 保持しない | diff --git a/docs-site/src/content/docs/ko/reference/configuration/providers.md b/docs-site/src/content/docs/ko/reference/configuration/providers.md index 0bb995ed1ef..6102a5442e6 100644 --- a/docs-site/src/content/docs/ko/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ko/reference/configuration/providers.md @@ -189,9 +189,9 @@ API 키 공급자는 리터럴 키나 환경 참조를 둘 수 있습니다. OAu ### 프로바이더 저장이 유지하는 것 -기존 프로바이더 이름으로 `POST /api/providers`를 보내면 저장된 행이 요청으로 만든 행으로 바뀝니다. 대시보드의 추가/편집 폼은 모든 필드를 보낼 수 없으므로, 요청이 빠뜨린 저장 필드 일부는 저장할 때 이어서 유지됩니다. 그중 열 가지는 특정 업스트림의 동작을 기록한 설정입니다: `preserveReasoningContentModels`, `preserveResponsesReasoningContent`, `requiresReasoningPlaceholderModels`, `foldDeveloperRoleToSystem`, `singleLeadingSystemMessage`, `reasoningWireFormat`, `omitReasoningEffortWithToolsModels`, `retryOn429`, `transientRetryOn5xx`, `retryOnReset`. +기존 프로바이더 이름으로 `POST /api/providers`를 보내면 저장된 행이 요청으로 만든 행으로 바뀝니다. 대시보드의 추가/편집 폼은 모든 필드를 보낼 수 없으므로, 요청이 빠뜨린 저장 필드 일부는 저장할 때 이어서 유지됩니다. 그중 열한 가지는 특정 업스트림의 동작을 기록한 설정입니다: `preserveReasoningContentModels`, `preserveResponsesReasoningContent`, `requiresReasoningPlaceholderModels`, `foldDeveloperRoleToSystem`, `singleLeadingSystemMessage`, `reasoningWireFormat`, `omitReasoningEffortWithToolsModels`, `promoteAdditionalTools`, `retryOn429`, `transientRetryOn5xx`, `retryOnReset`. -| 저장 | 열 가지 설정 | 저장된 `apiKeyPool` | +| 저장 | 열한 가지 설정 | 저장된 `apiKeyPool` | | --- | --- | --- | | 같은 목적지, 필드 생략 | 저장된 값 유지(명시적인 `[]`나 `false` 포함) | 유지 | | 새 목적지, 필드 생략 | 유지하지 않음. 새 목적지의 레지스트리 기본값이 적용될 수 있음 | 유지하지 않음 | diff --git a/docs-site/src/content/docs/reference/configuration/keychain.md b/docs-site/src/content/docs/reference/configuration/keychain.md new file mode 100644 index 00000000000..c8a4e40d384 --- /dev/null +++ b/docs-site/src/content/docs/reference/configuration/keychain.md @@ -0,0 +1,55 @@ +--- +title: Storing keys in the OS keychain +description: Storing provider API keys and Vertex credentials in macOS Keychain, Windows Credential Manager, or Linux Secret Service. +--- + +For Vertex, the dedicated API-key pool input (`POST /api/providers/keys`) accepts complete GCP +`service_account` or `authorized_user` credential JSON after the provider has been created. +The provider create/edit form and `ocx provider add --api-key` refuse credential JSON before +saving: create the provider without a key, then add the JSON through its key pool. Paste the +JSON contents, not a Windows, Unix, or relative file path. Multiple +JSON objects separated by commas or whitespace are stored together as separate pool entries; +the first becomes active. Configuration contains only `gcp-sa:` references, never the +credential JSON. Mixed JSON/literal-key bulk pastes are rejected as a whole; add literal keys +separately. Storage failures leave the configuration unchanged and report a fixed error without +echoing credential material. Empty marker accounts are invalid and do not fall back to host credentials. +Paste one credential object or a sequence of objects, without an enclosing array or quoted string. +Malformed credential text is rejected. If a selected stored credential is removed or becomes unreadable, +Vertex refuses the request even when it previously cached an access token for that credential. +Removing a key or provider, or replacing a provider with `ocx provider add --force`, deletes its unreferenced GCP credentials after the configuration is +saved; a credential referenced by another provider is retained. If OS-store cleanup fails, a +warning requests manual inspection without undoing the committed configuration removal. +Ordinary keychain store/restore actions leave GCP markers and JSON in the OS store. Restoring +credential JSON through a `keychain:` alias is refused, and cleanup preserves accounts still +referenced through either marker spelling. +Using a `keychain:` alias containing GCP JSON as an API key also refuses the request; select its +`gcp-sa:` reference for Vertex authentication instead. +Literal API keys must be plain token text: JSON objects, arrays and quoted JSON strings are refused +in literal-key resolution and restore, including wrappers containing escaped credential text. +Credential writes and marker persistence share the cleanup transaction lock, so concurrent +removal cannot delete the stored JSON between read-back and the marker commit. + +By default a provider's `apiKey` and `apiKeyPool` sit in `config.json` (mode 0600, atomic writes). +If you would rather keep the key material out of the file, move it into the OS credential store: + +```bash +ocx provider keychain deepseek status # store: file | env | keychain, and whether the keychain answers +ocx provider keychain deepseek store # move active key + pool into the OS keychain +ocx provider keychain deepseek restore # bring the plaintext back and delete the keychain items +``` + +The same operations are `GET`/`POST /api/providers/keychain`. After `store`, `config.json` holds +`"apiKey": "keychain:deepseek"` (pool entries `keychain:deepseek/`) and the secret lives under the +`opencodex.provider-api-key.v1` service in macOS Keychain, Windows Credential Manager, or the Linux +Secret Service. Backups of `config.json` therefore carry references only. Key rotation and failover +keep working: pool entries compare by reference, so a rotation never writes plaintext back. + +Before touching the config, `store` writes and reads back every entry; if the keychain is unavailable +or the read-back does not match, it refuses with 503 and leaves the file as it was. At request time +a reference that cannot be read yields no credential and one warning per key — there is no plaintext +fallback, by design. + +When not to opt in: a proxy running as a headless service (systemd, launchd, Task Scheduler) or in a +container usually has no unlocked keychain session, so requests would fail closed. Use an +`${ENV_VAR}` reference in the service environment there instead. Env references are left untouched +by `store`. diff --git a/docs-site/src/content/docs/reference/configuration/providers.md b/docs-site/src/content/docs/reference/configuration/providers.md index ae998f0f119..7ed86852041 100644 --- a/docs-site/src/content/docs/reference/configuration/providers.md +++ b/docs-site/src/content/docs/reference/configuration/providers.md @@ -326,7 +326,7 @@ Providers can expose a built-in shorthand, such as `agy` for `google-antigravity | `preserveResponsesMessageMetadata?` | `boolean` | Default off. Preserve private `internal_chat_message_metadata_passthrough` on input items for explicitly trusted launcher relays, including custom native v1 compact requests. This forwards private turn/session metadata to that destination; do not enable for ordinary gateways. | | `dropResponsesReasoningItems?` | `boolean` | Remove replayed Responses `reasoning` items from continuation input entirely, including custom native v1 compact requests, instead of forwarding them with blanked content. This is lossy: summaries, item ids, and `encrypted_content` go with them, so no reasoning item from the previous turn remains in the forwarded input. It does not remove `previous_response_id`, so an upstream holding its own turn state may still reach it. Enable only for an upstream that rejects reasoning replay outright — the built-in Volcengine Ark Coding Plan preset sets it because validated tool continuations answered `400 InvalidParameter` otherwise. Set it to `false` on that provider to forward reasoning items again. | | `disabled?` | `boolean` | Keep the provider on disk but exclude it from routing and model/catalog listings. | -| `apiKey?` | `string` | API key, an `${ENV_VAR}` / `$ENV_VAR` reference, or a `keychain:` reference written by `ocx provider keychain store`. References resolve at request time. See [Storing keys in the OS keychain](#storing-keys-in-the-os-keychain). | +| `apiKey?` | `string` | API key, an `${ENV_VAR}` / `$ENV_VAR` reference, or a `keychain:` reference written by `ocx provider keychain store`. References resolve at request time. See [Storing keys in the OS keychain](/reference/configuration/keychain/). | | `apiKeyTransport?` | `"x-api-key" \| "bearer"` | Anthropic key header style. Defaults to native `x-api-key`; valid only for key-auth `anthropic` providers. | | `apiKeyPool?` | `ApiKeyPoolEntry[]` | Multi-key pool. `apiKey` mirrors the active entry; each item has `id`, `key`, optional `label`, and optional numeric `addedAt`. | | `apiKeyPoolStrategy?` | `"round-robin" \| "fill-first" \| "quota"` | How a warm key is chosen **before** the first attempt when the committed key is already cooling. Omitted keeps rotation reactive-only: the pool moves after a 429 or 401 and not before. `round-robin` takes the next key in the pool, `fill-first` keeps the first eligible one, and `quota` prefers the key with the most remaining headroom, falling back to `fill-first` order for a provider whose per-key quota is unknown. A healthy committed key is never overridden, so a manual key selection stands. | @@ -370,6 +370,7 @@ Providers can expose a built-in shorthand, such as `agy` for `google-antigravity | `xaiResponsesXSearch?` | `boolean` | Disabled by default. On an xAI Responses destination, append the provider-hosted `x_search` declaration only when a live `web_search` tool survives final request normalization. Existing declarations are not duplicated, caller `tool_choice`/`allowed_tools` selectors are never widened, and this is separate from the web-search sidecar's `search.xSearch` options. | | `modelPreferHostedTools?` | `Record` | Exact-model opt-in for non-forward Responses gateways that reserve a hosted-tool namespace. Currently accepts only `["image_generation"]`; a matching model must use the `openai-responses` wire and support that hosted tool. It removes colliding client `image_gen` declarations and rewrites their selectors to preserve caller tool choice. For OpenAI API virtual `-pro` models, the selected public ID is matched first and the resolved base wire-model ID is a fallback. `modelAdapters` resolves the public ID first, then the base ID; the second resolution determines the final wire. Other models retain normal alias behavior. | | `annotateEmptyToolOutputs?` | `boolean` | Replace a present-but-empty tool result with a short marker before it reaches the model, so a blank result is not read as a missing one. Applies to blank strings and text-only part arrays; image, file, and encrypted parts are never touched. Defaults to `true` for DeepSeek from the built-in registry and is otherwise unset. Set `false` to opt a provider out — an explicit `false` is preserved across later edits that omit the field. `PATCH /api/providers?name=` accepts `true`, `false`, or `null` to clear the override and return to registry-default behavior. | +| `promoteAdditionalTools?` | `boolean` | Whether to promote Codex Desktop's `additional_tools` input items into top-level `tools` and strip the wrapper from `input`. Enabled automatically for OpenCode Go and xAI (`api.x.ai` and `cli-chat-proxy.grok.com`) unless `authMode` is `"forward"`, where promotion needs an explicit `true`; other OpenAI-strict third-party gateways that return HTTP 422 for unknown item types in `input` can opt in here. `PATCH /api/providers?name=` accepts a boolean or `null` to clear it. | | `unsupportedHostedTools?` | `string[]` | Hosted tool declarations this Responses destination rejects, so they are stripped from `tools`, from client-loaded `additional_tools`, and from `tool_choice` instead of being forwarded and rejected upstream. Use it for an OpenAI-compatible gateway with a narrower capability set — one that accepts plain Responses requests and `function` tools but returns HTTP 400 for hosted `web_search` — so a text-only prompt is not failed by a capability it never needed. Accepts only hosted tool type names (`web_search`, `web_search_preview`, `file_search`, `computer_use_preview`, `computer_use`, `code_interpreter`, `image_generation`, `image_gen`, `mcp`, `tool_search`, `local_shell`, `x_search`); an unrecognized name is rejected rather than silently ignored. Spelling variants of one capability are aliased, so `["web_search"]` also denies `web_search_preview`. A provider cannot both deny a hosted tool here and prefer it in `modelPreferHostedTools`. Xiaomi MiMo Responses destinations (`xiaomimimo.com` and its subdomains, including `api.xiaomimimo.com` and `token-plan-cn.xiaomimimo.com`) automatically strip `web_search` and `web_search_preview` while preserving function tools; no declaration is needed for these hosts. This is independent of `supportsResponsesCustomTools`; set both for a gateway that also rejects native custom tools. `PATCH /api/providers?name=` accepts an array or `null` to clear it. | | `reasoningEffortMap?` | `Record` | Provider-wide wire aliases for reasoning labels. Map a label to `"__omit__"` to drop the reasoning field from the upstream request entirely: `reasoning_effort` on an OpenAI-compatible wire, and Ollama's native `think` field on the Ollama native adapter (#2356). | | `modelReasoningEffortMap?` | `Record>` | Per-model wire aliases for reasoning labels. Map a label to `"__omit__"` to drop the reasoning field from the upstream request entirely. | @@ -487,9 +488,9 @@ and `modelContextWindows` for that exact id when needed. Unknown ids are not gue ### What a provider save keeps -`POST /api/providers` with the name of an existing provider replaces the stored row with one built from the request. The dashboard's add/edit form cannot send every field, so the save keeps some stored fields the request omits. Ten of them record how one upstream behaves: `preserveReasoningContentModels`, `preserveResponsesReasoningContent`, `requiresReasoningPlaceholderModels`, `foldDeveloperRoleToSystem`, `singleLeadingSystemMessage`, `reasoningWireFormat`, `omitReasoningEffortWithToolsModels`, `retryOn429`, `transientRetryOn5xx` and `retryOnReset`. +`POST /api/providers` with the name of an existing provider replaces the stored row with one built from the request. The dashboard's add/edit form cannot send every field, so the save keeps some stored fields the request omits. Eleven of them record how one upstream behaves: `preserveReasoningContentModels`, `preserveResponsesReasoningContent`, `requiresReasoningPlaceholderModels`, `foldDeveloperRoleToSystem`, `singleLeadingSystemMessage`, `reasoningWireFormat`, `omitReasoningEffortWithToolsModels`, `promoteAdditionalTools`, `retryOn429`, `transientRetryOn5xx` and `retryOnReset`. -| Save | The ten settings | Stored `apiKeyPool` | +| Save | The eleven settings | Stored `apiKeyPool` | | --- | --- | --- | | Same destination, field omitted | Stored value kept, including an explicit `[]` or `false` | Kept | | New destination, field omitted | Not kept; registry defaults for the new destination may apply | Not kept | @@ -1589,60 +1590,11 @@ same mapping applies to a native `vercel/` selector: use the encoded `vercel-ai-gateway/vercel-` selector in OpenCodex and keep `vercel/` as the model key. -## Static model allowlists - ## Storing keys in the OS keychain -For Vertex, the dedicated API-key pool input (`POST /api/providers/keys`) accepts complete GCP -`service_account` or `authorized_user` credential JSON after the provider has been created. -The provider create/edit form and `ocx provider add --api-key` refuse credential JSON before -saving: create the provider without a key, then add the JSON through its key pool. Paste the -JSON contents, not a Windows, Unix, or relative file path. Multiple -JSON objects separated by commas or whitespace are stored together as separate pool entries; -the first becomes active. Configuration contains only `gcp-sa:` references, never the -credential JSON. Mixed JSON/literal-key bulk pastes are rejected as a whole; add literal keys -separately. Storage failures leave the configuration unchanged and report a fixed error without -echoing credential material. Empty marker accounts are invalid and do not fall back to host credentials. -Paste one credential object or a sequence of objects, without an enclosing array or quoted string. -Malformed credential text is rejected. If a selected stored credential is removed or becomes unreadable, -Vertex refuses the request even when it previously cached an access token for that credential. -Removing a key or provider, or replacing a provider with `ocx provider add --force`, deletes its unreferenced GCP credentials after the configuration is -saved; a credential referenced by another provider is retained. If OS-store cleanup fails, a -warning requests manual inspection without undoing the committed configuration removal. -Ordinary keychain store/restore actions leave GCP markers and JSON in the OS store. Restoring -credential JSON through a `keychain:` alias is refused, and cleanup preserves accounts still -referenced through either marker spelling. -Using a `keychain:` alias containing GCP JSON as an API key also refuses the request; select its -`gcp-sa:` reference for Vertex authentication instead. -Literal API keys must be plain token text: JSON objects, arrays and quoted JSON strings are refused -in literal-key resolution and restore, including wrappers containing escaped credential text. -Credential writes and marker persistence share the cleanup transaction lock, so concurrent -removal cannot delete the stored JSON between read-back and the marker commit. - -By default a provider's `apiKey` and `apiKeyPool` sit in `config.json` (mode 0600, atomic writes). -If you would rather keep the key material out of the file, move it into the OS credential store: - -```bash -ocx provider keychain deepseek status # store: file | env | keychain, and whether the keychain answers -ocx provider keychain deepseek store # move active key + pool into the OS keychain -ocx provider keychain deepseek restore # bring the plaintext back and delete the keychain items -``` +API keys and credential references can be stored in the OS credential store (macOS Keychain, Windows Credential Manager, Linux Secret Service) instead of `config.json`. See [Storing keys in the OS keychain](/reference/configuration/keychain/) for details on migration commands, storage rules, and headless-service caveats. -The same operations are `GET`/`POST /api/providers/keychain`. After `store`, `config.json` holds -`"apiKey": "keychain:deepseek"` (pool entries `keychain:deepseek/`) and the secret lives under the -`opencodex.provider-api-key.v1` service in macOS Keychain, Windows Credential Manager, or the Linux -Secret Service. Backups of `config.json` therefore carry references only. Key rotation and failover -keep working: pool entries compare by reference, so a rotation never writes plaintext back. - -Before touching the config, `store` writes and reads back every entry; if the keychain is unavailable -or the read-back does not match, it refuses with 503 and leaves the file as it was. At request time -a reference that cannot be read yields no credential and one warning per key — there is no plaintext -fallback, by design. - -When not to opt in: a proxy running as a headless service (systemd, launchd, Task Scheduler) or in a -container usually has no unlocked keychain session, so requests would fail closed. Use an -`${ENV_VAR}` reference in the service environment there instead. Env references are left untouched -by `store`. +## Static model allowlists The `zhipu-bigmodel-responses` preset seeds `glm-5.3`, `glm-5.3-flash` and `glm-5-turbo` with `liveModels: false` for `https://open.bigmodel.cn/api/v1`. Its static roster and diff --git a/docs-site/src/content/docs/ru/reference/configuration/providers.md b/docs-site/src/content/docs/ru/reference/configuration/providers.md index af23015f1de..527f8f4761b 100644 --- a/docs-site/src/content/docs/ru/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ru/reference/configuration/providers.md @@ -205,9 +205,9 @@ Claude Code настраивается через ### Что сохраняется при сохранении провайдера -`POST /api/providers` с именем существующего провайдера заменяет сохранённую строку строкой, собранной из запроса. Форма добавления и редактирования в панели не может отправить все поля, поэтому сохранение переносит часть сохранённых полей, которые запрос опустил. Десять из них описывают поведение конкретного апстрима: `preserveReasoningContentModels`, `preserveResponsesReasoningContent`, `requiresReasoningPlaceholderModels`, `foldDeveloperRoleToSystem`, `singleLeadingSystemMessage`, `reasoningWireFormat`, `omitReasoningEffortWithToolsModels`, `retryOn429`, `transientRetryOn5xx`, `retryOnReset`. +`POST /api/providers` с именем существующего провайдера заменяет сохранённую строку строкой, собранной из запроса. Форма добавления и редактирования в панели не может отправить все поля, поэтому сохранение переносит часть сохранённых полей, которые запрос опустил. Одиннадцать из них описывают поведение конкретного апстрима: `preserveReasoningContentModels`, `preserveResponsesReasoningContent`, `requiresReasoningPlaceholderModels`, `foldDeveloperRoleToSystem`, `singleLeadingSystemMessage`, `reasoningWireFormat`, `omitReasoningEffortWithToolsModels`, `promoteAdditionalTools`, `retryOn429`, `transientRetryOn5xx`, `retryOnReset`. -| Сохранение | Десять настроек | Сохранённый `apiKeyPool` | +| Сохранение | Одиннадцать настроек | Сохранённый `apiKeyPool` | | --- | --- | --- | | Та же точка назначения, поле опущено | Сохранённое значение остаётся, включая явный `[]` или `false` | Остаётся | | Новая точка назначения, поле опущено | Не переносится; могут примениться значения реестра по умолчанию для новой точки | Не переносится | diff --git a/docs-site/src/content/docs/tr/reference/configuration/providers.md b/docs-site/src/content/docs/tr/reference/configuration/providers.md index c13c0c45116..12f2004ed8b 100644 --- a/docs-site/src/content/docs/tr/reference/configuration/providers.md +++ b/docs-site/src/content/docs/tr/reference/configuration/providers.md @@ -205,9 +205,9 @@ yapılandırılır. ### Bir sağlayıcı kaydı neleri korur -Var olan bir sağlayıcının adıyla gönderilen `POST /api/providers`, saklanan satırı istekten oluşturulan bir satırla değiştirir. Panodaki ekleme/düzenleme formu her alanı gönderemez; bu yüzden kayıt, isteğin atladığı bazı saklanan alanları korur. Bunlardan onu belirli bir üst kaynağın nasıl davrandığını kaydeder: `preserveReasoningContentModels`, `preserveResponsesReasoningContent`, `requiresReasoningPlaceholderModels`, `foldDeveloperRoleToSystem`, `singleLeadingSystemMessage`, `reasoningWireFormat`, `omitReasoningEffortWithToolsModels`, `retryOn429`, `transientRetryOn5xx`, `retryOnReset`. +Var olan bir sağlayıcının adıyla gönderilen `POST /api/providers`, saklanan satırı istekten oluşturulan bir satırla değiştirir. Panodaki ekleme/düzenleme formu her alanı gönderemez; bu yüzden kayıt, isteğin atladığı bazı saklanan alanları korur. Bunlardan on biri belirli bir üst kaynağın nasıl davrandığını kaydeder: `preserveReasoningContentModels`, `preserveResponsesReasoningContent`, `requiresReasoningPlaceholderModels`, `foldDeveloperRoleToSystem`, `singleLeadingSystemMessage`, `reasoningWireFormat`, `omitReasoningEffortWithToolsModels`, `promoteAdditionalTools`, `retryOn429`, `transientRetryOn5xx`, `retryOnReset`. -| Kayıt | On ayar | Saklanan `apiKeyPool` | +| Kayıt | On bir ayar | Saklanan `apiKeyPool` | | --- | --- | --- | | Aynı hedef, alan atlanmış | Saklanan değer korunur, açık bir `[]` veya `false` dahil | Korunur | | Yeni hedef, alan atlanmış | Korunmaz; yeni hedef için kayıt defteri varsayılanları uygulanabilir | Korunmaz | diff --git a/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md b/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md index e7b199e536d..bc201ce0b73 100644 --- a/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md +++ b/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md @@ -191,9 +191,9 @@ API key 提供者可以持有字面量 key,或环境引用。OAuth 提供者 ### 保存提供方时会保留什么 -用已有提供方的名称调用 `POST /api/providers`,会用根据请求构建的行替换已存储的行。仪表板的添加/编辑表单无法发送所有字段,因此保存时会保留请求省略的部分已存储字段。其中十个记录的是某个上游的行为:`preserveReasoningContentModels`, `preserveResponsesReasoningContent`, `requiresReasoningPlaceholderModels`, `foldDeveloperRoleToSystem`, `singleLeadingSystemMessage`, `reasoningWireFormat`, `omitReasoningEffortWithToolsModels`, `retryOn429`, `transientRetryOn5xx`, `retryOnReset`。 +用已有提供方的名称调用 `POST /api/providers`,会用根据请求构建的行替换已存储的行。仪表板的添加/编辑表单无法发送所有字段,因此保存时会保留请求省略的部分已存储字段。其中十一个记录的是某个上游的行为:`preserveReasoningContentModels`, `preserveResponsesReasoningContent`, `requiresReasoningPlaceholderModels`, `foldDeveloperRoleToSystem`, `singleLeadingSystemMessage`, `reasoningWireFormat`, `omitReasoningEffortWithToolsModels`, `promoteAdditionalTools`, `retryOn429`, `transientRetryOn5xx`, `retryOnReset`。 -| 保存 | 十项设置 | 已存储的 `apiKeyPool` | +| 保存 | 十一项设置 | 已存储的 `apiKeyPool` | | --- | --- | --- | | 目的地相同,字段省略 | 保留已存储的值,包括显式的 `[]` 或 `false` | 保留 | | 新目的地,字段省略 | 不保留;可能套用新目的地的注册表默认值 | 不保留 | diff --git a/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md b/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md index cc8b93b3735..5a430f66c07 100644 --- a/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md +++ b/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md @@ -162,9 +162,9 @@ API-key 供應商可持有字面值金鑰或環境參考。OAuth 供應商使用 ### 儲存供應商時會保留什麼 -以既有供應商的名稱呼叫 `POST /api/providers`,會以根據請求建立的列取代已儲存的列。儀表板的新增/編輯表單無法傳送所有欄位,因此儲存時會保留請求省略的部分已儲存欄位。其中十個記錄的是某個上游的行為:`preserveReasoningContentModels`, `preserveResponsesReasoningContent`, `requiresReasoningPlaceholderModels`, `foldDeveloperRoleToSystem`, `singleLeadingSystemMessage`, `reasoningWireFormat`, `omitReasoningEffortWithToolsModels`, `retryOn429`, `transientRetryOn5xx`, `retryOnReset`。 +以既有供應商的名稱呼叫 `POST /api/providers`,會以根據請求建立的列取代已儲存的列。儀表板的新增/編輯表單無法傳送所有欄位,因此儲存時會保留請求省略的部分已儲存欄位。其中十一個記錄的是某個上游的行為:`preserveReasoningContentModels`, `preserveResponsesReasoningContent`, `requiresReasoningPlaceholderModels`, `foldDeveloperRoleToSystem`, `singleLeadingSystemMessage`, `reasoningWireFormat`, `omitReasoningEffortWithToolsModels`, `promoteAdditionalTools`, `retryOn429`, `transientRetryOn5xx`, `retryOnReset`。 -| 儲存 | 十項設定 | 已儲存的 `apiKeyPool` | +| 儲存 | 十一項設定 | 已儲存的 `apiKeyPool` | | --- | --- | --- | | 目的地相同,欄位省略 | 保留已儲存的值,包括明確的 `[]` 或 `false` | 保留 | | 新目的地,欄位省略 | 不保留;可能套用新目的地的登錄檔預設值 | 不保留 | diff --git a/scripts/test-layout/layout-providers.json b/scripts/test-layout/layout-providers.json index e79f4b53fc7..cadb953254c 100644 --- a/scripts/test-layout/layout-providers.json +++ b/scripts/test-layout/layout-providers.json @@ -1,4 +1,7 @@ { + "xai-additional-tools.test.ts": "providers/xai", + "xai-replay-provenance.test.ts": "providers/xai", + "xai-loaded-tool-provenance.test.ts": "providers/xai", "free-directory-runcrate.test.ts": "providers", "cc-switch-import.test.ts": "providers", "kiro-completion-with-real-tools.test.ts": "providers/kiro" diff --git a/src/adapters/openai-responses/muse-tool-catalog.ts b/src/adapters/openai-responses/muse-tool-catalog.ts index bf2811e00eb..e0c92d70205 100644 --- a/src/adapters/openai-responses/muse-tool-catalog.ts +++ b/src/adapters/openai-responses/muse-tool-catalog.ts @@ -1,6 +1,7 @@ import { isPlainObject } from "./internal"; import { promoteClientLoadedTools } from "./tool-schema"; import { rewriteRoutedCustomToolsForUpstream } from "../../responses/custom-tool-compat"; +import { hasItemProvenance, isHistoricalItem } from "../../responses/item-provenance"; export class MuseToolCatalogCompatibilityError extends Error { constructor() { @@ -54,8 +55,13 @@ export function normalizeMuseToolCatalog(body: unknown, replayPrefixLength = 0, if (!Array.isArray(body.input)) return topLevelTools === undefined ? body : { ...body, tools: flattenBuiltinNamespace(topLevelTools) }; const start = Number.isFinite(replayPrefixLength) ? Math.min(body.input.length, Math.max(0, Math.trunc(replayPrefixLength))) : 0; + // Item provenance owns membership once present: an earlier rewrite (Zed lifting a replayed + // wrapper) may already have shortened the input below the numeric prefix. + const originalInput: unknown[] = body.input; + const tagged = originalInput.some(hasItemProvenance); + const isCurrent = (item: unknown, index: number) => tagged ? !isHistoricalItem(item) : index >= start; const normalizedInput = body.input.map((item, index) => { - if (index < start) return item; + if (!isCurrent(item, index)) return item; if (isPlainObject(item) && (item.type === "additional_tools" || (item.type === "tool_search_output" && item.tools !== undefined))) { const tools = normalizeCatalog(item.tools); @@ -63,7 +69,7 @@ export function normalizeMuseToolCatalog(body: unknown, replayPrefixLength = 0, } return item; }); - const currentInput = normalizedInput.slice(start); + const currentInput = normalizedInput.filter((_item, index) => isCurrent(originalInput[index], index)); const declared = [...(topLevelTools ?? [])]; for (const item of currentInput) { if (isPlainObject(item) && item.type === "additional_tools" && Array.isArray(item.tools)) declared.push(...item.tools); diff --git a/src/adapters/openai-responses/passthrough.ts b/src/adapters/openai-responses/passthrough.ts index b76549f571e..6f6ab2c07e2 100644 --- a/src/adapters/openai-responses/passthrough.ts +++ b/src/adapters/openai-responses/passthrough.ts @@ -3,7 +3,7 @@ import { normalizeRoutedAgentMessages } from "../routed-agent-messages"; import { frameCompactionHistory } from "../../responses/compaction-history-framing"; import { nameRoutedIdentity, repairIdentityInResponsesBody, stripRoutedIdentity } from "../identity"; import { stripBracketedModelSuffix } from "../openai-chat"; -import { normalizeOpenCodeGoAdditionalTools } from "../opencode-go-additional-tools"; +import { isOpenCodeGoResponsesUrl, isXaiResponsesUrl, promoteResponsesAdditionalTools } from "../opencode-go-additional-tools"; import { isXaiResponsesDestination } from "../../providers/xai-transport"; import { applyGithubCopilotContextTier } from "../../providers/github-copilot-context"; import { Buffer } from "node:buffer"; @@ -48,6 +48,7 @@ import { usageFromResponsesPayload } from "./usage"; import { normalizeToolSchemas, promoteClientLoadedTools, stripUnsupportedHostedTools } from "./tool-schema"; import { annotateEmptyResponsesToolOutputs, backfillWebSearchQueries, normalizeResponsesToolResultAdjacency, repairOrphanedInputItems, repairOversizedReplayCallIds, repairUnidentifiedToolOutputItems, restoreBridgedWebSearchCalls } from "./tool-output-recovery"; import { bridgeSearchReplayScope } from "../../responses/bridge-search-replay-cache"; +import { tagHistoricalInputItems } from "../../responses/item-provenance"; import { applyTierDecisionToResponsesBody, normalizeCanonicalForwardContinuationEnvelope, normalizeCanonicalForwardPromptEnvelope, stripCanonicalForwardSamplingParams, stripPreviousResponseId, stripStatefulResponsesParams, stripUnsupportedForwardParams } from "./canonical-forward"; import { normalizeImageGenClientTools, preferConfiguredHostedTools } from "./image-gen"; import { stripMuseSparkUnsupportedWebSearchFields, stripOpenAiOnlyWebSearchFields } from "./web-search"; @@ -175,8 +176,22 @@ export function createResponsesPassthroughAdapter(provider: OcxProviderConfig): let plaintextV2AgentMessageAliasedToolNames: ReadonlySet | undefined; let convertedMuseToolNameAliases: Map | undefined; const unexpandedMiss = !!parsed.previousResponseId && parsed._previousResponseInputExpanded !== true; - let outBody = stripPreviousResponseId( - parsed._rawBody, + const replayPrefixLen = parsed._replayPrefixLen; + let outBody = parsed._rawBody; + // Clone and tag before any input rewrite can remove, replace, or reorder replay items. + if ( + typeof replayPrefixLen === "number" + && replayPrefixLen > 0 + && isPlainObject(outBody) + && Array.isArray(outBody.input) + ) { + outBody = { + ...outBody, + input: tagHistoricalInputItems(outBody.input, replayPrefixLen), + }; + } + outBody = stripPreviousResponseId( + outBody, forward || parsed._previousResponseInputExpanded === true, ); if (isMetaAiResponsesDestination(url)) { @@ -368,8 +383,15 @@ export function createResponsesPassthroughAdapter(provider: OcxProviderConfig): // Last, so promoted namespace children are also cleared of Codex-private fields. outBody = stripCanonicalOnlyToolFields(outBody, provider.supportsOpenAiWebSearchToolFields === false); } - if (!forward) { - outBody = normalizeOpenCodeGoAdditionalTools(outBody, url, parsed._replayPrefixLen); + if ( + provider.promoteAdditionalTools === true + || (!forward && ( + isOpenCodeGoResponsesUrl(url) + || isXaiResponsesDestination(provider) + || isXaiResponsesUrl(url) + )) + ) { + outBody = promoteResponsesAdditionalTools(outBody); } // Same native-v2/portable predicate as the routedCompaction gate in handleResponses(): // neither forward auth nor native v1 endpoint support implies trigger support. The diff --git a/src/adapters/openai-responses/prompt-cache.ts b/src/adapters/openai-responses/prompt-cache.ts index dff927dd17f..a9b4b2a6e63 100644 --- a/src/adapters/openai-responses/prompt-cache.ts +++ b/src/adapters/openai-responses/prompt-cache.ts @@ -71,7 +71,9 @@ export function stripPromptCacheBreakpoints( if (!isPlainObject(value)) return { value, changed: false, complete: true }; let changed = Object.hasOwn(value, "prompt_cache_breakpoint"); - const next: Record = {}; + // Keep symbol provenance when rebuilding an input item or one of its nested fields. + const next: Record = { ...value }; + delete next.prompt_cache_breakpoint; for (const [key, entry] of Object.entries(value)) { if (key === "prompt_cache_breakpoint") continue; const rewritten = stripPromptCacheBreakpoints(entry, state, depth + 1); diff --git a/src/adapters/openai-responses/request-strips.ts b/src/adapters/openai-responses/request-strips.ts index 366d1ec34bc..9d64f6ed49e 100644 --- a/src/adapters/openai-responses/request-strips.ts +++ b/src/adapters/openai-responses/request-strips.ts @@ -287,12 +287,15 @@ const CANONICAL_ONLY_INPUT_ITEM_TYPES: ReadonlySet = new Set([ * {@link stripCanonicalOnlyTopLevelFields}; the canonical and OpenAI-operated destinations keep * the items verbatim and in order. Returns the input unchanged when none is present. */ +export function isCanonicalOnlyInputItem(item: unknown): boolean { + return isPlainObject(item) + && typeof item.type === "string" && CANONICAL_ONLY_INPUT_ITEM_TYPES.has(item.type); +} + export function stripCanonicalOnlyInputItems(body: unknown): unknown { if (!isPlainObject(body) || !Array.isArray(body.input)) return body; - const isPrivate = (item: unknown): boolean => isPlainObject(item) - && typeof item.type === "string" && CANONICAL_ONLY_INPUT_ITEM_TYPES.has(item.type); - if (!body.input.some(isPrivate)) return body; - return { ...body, input: body.input.filter(item => !isPrivate(item)) }; + if (!body.input.some(isCanonicalOnlyInputItem)) return body; + return { ...body, input: body.input.filter(item => !isCanonicalOnlyInputItem(item)) }; } /** diff --git a/src/adapters/openai-responses/tool-output-recovery.ts b/src/adapters/openai-responses/tool-output-recovery.ts index f2218ecb181..2605e1fdab7 100644 --- a/src/adapters/openai-responses/tool-output-recovery.ts +++ b/src/adapters/openai-responses/tool-output-recovery.ts @@ -3,6 +3,7 @@ import { EMPTY_TOOL_OUTPUT_ANNOTATION, isWhitespaceOnlyTextPartArray } from "../ import { isPlainObject } from "./internal"; import { peekBridgeSearchReplay } from "../../responses/bridge-search-replay-cache"; import { externalTaskInputResponsesContent } from "../../responses/task-input"; +import { copyItemProvenance } from "../../responses/item-provenance"; const MAX_RESPONSES_CALL_ID_LENGTH = 64; @@ -336,7 +337,7 @@ export function restoreBridgedWebSearchCalls(body: unknown, destinationScope: st if (memo && !occupiedCallIds.has(memo.callId)) { changed = true; occupiedCallIds.add(memo.callId); - restored.push({ + const fnCall: Record = { type: "function_call", ...(memo.sourceItemId ? { id: memo.sourceItemId } : {}), call_id: memo.callId, @@ -344,8 +345,16 @@ export function restoreBridgedWebSearchCalls(body: unknown, destinationScope: st // The bridge records the complete arguments text from the call's own done frame; the // empty-object fallback matches what a continuation leg would have sent. arguments: memo.argumentsText.length > 0 ? memo.argumentsText : "{}", - }); - restored.push({ type: "function_call_output", call_id: memo.callId, output: memo.output }); + }; + const fnOutput: Record = { + type: "function_call_output", + call_id: memo.callId, + output: memo.output, + }; + copyItemProvenance(item, fnCall); + copyItemProvenance(item, fnOutput); + restored.push(fnCall); + restored.push(fnOutput); continue; } } @@ -397,7 +406,7 @@ export function repairOrphanedInputItems( const taskInput = isFnOutput && !hasStringCallId ? externalTaskInputResponsesContent(item) : undefined; if (taskInput) { changed = true; - repaired.push({ type: "message", role: "user", content: taskInput }); + repaired.push(copyItemProvenance(item, { type: "message", role: "user", content: taskInput })); continue; } const callId = typeof item.call_id === "string" ? item.call_id : ""; @@ -409,11 +418,13 @@ export function repairOrphanedInputItems( const knownNullOutput = callId.length > 0 && item.output == null; if (repairOrphanOutputs && !paired && (knownNullOutput || usableOutput)) { changed = true; - repaired.push({ + const msg: Record = { type: "message", role: "user", content: orphanedToolOutputContent(item.output, callId), - }); + }; + copyItemProvenance(item, msg); + repaired.push(msg); continue; } } @@ -429,9 +440,11 @@ export function repairOrphanedInputItems( const name = typeof item.name === "string" && item.name.length > 0 ? item.name : callId; const text = `[ocx] no tool result was recorded for "${name}"; execution status unknown — do not treat this as success, failure, or user-provided input.`; syntheticKeys.add(`${isFnCall ? "function" : "custom"}:${callId}`); - pendingSyntheticOutputs.push(isFnCall + const synthetic: Record = isFnCall ? { type: "function_call_output", call_id: callId, output: text } - : { type: "custom_tool_call_output", call_id: callId, output: text }); + : { type: "custom_tool_call_output", call_id: callId, output: text }; + copyItemProvenance(item, synthetic); + pendingSyntheticOutputs.push(synthetic); } } continue; diff --git a/src/adapters/openai-responses/tool-schema.ts b/src/adapters/openai-responses/tool-schema.ts index 12d11093cdd..eb986999fb8 100644 --- a/src/adapters/openai-responses/tool-schema.ts +++ b/src/adapters/openai-responses/tool-schema.ts @@ -8,6 +8,7 @@ import { XaiToolSchemaCompatibilityError, } from "../xai-tool-schema"; import { isPlainObject } from "./internal"; +import { hasItemProvenance, isHistoricalItem } from "../../responses/item-provenance"; function normalizeFunctionToolSchema(tool: unknown, xaiTarget: boolean): unknown | undefined { if (!isPlainObject(tool) || tool.type !== "function") return tool; @@ -195,11 +196,19 @@ function mergeLoadedTools(declaredTools: unknown[], loadedTools: unknown[]): unk * serializing the raw request, so activate those returned definitions for upstreams that do not * implement the native deferred-loading handshake themselves. */ -export function promoteClientLoadedTools(body: unknown): unknown { +export function promoteClientLoadedTools(body: unknown, replayPrefixLength?: number): unknown { if (!isPlainObject(body) || !Array.isArray(body.input)) return body; - const loadedTools = body.input.flatMap(item => - isPlainObject(item) && item.type === "tool_search_output" && Array.isArray(item.tools) + const currentTurnStart = Number.isFinite(replayPrefixLength) + ? Math.max(0, Math.trunc(replayPrefixLength!)) + : 0; + const hasProvenance = body.input.some(hasItemProvenance); + const isCurrentItem = (item: unknown, index: number): boolean => + hasProvenance ? !isHistoricalItem(item) : index >= currentTurnStart; + + // Filter the source before declarations lose their item's provenance in a live catalog. + const loadedTools = body.input.flatMap((item, index) => + isCurrentItem(item, index) && isPlainObject(item) && item.type === "tool_search_output" && Array.isArray(item.tools) ? item.tools : [] ); @@ -210,17 +219,20 @@ export function promoteClientLoadedTools(body: unknown): unknown { return tools === body.tools ? body : { ...body, tools }; } - const additionalToolsIndex = body.input.findIndex(item => - isPlainObject(item) && item.type === "additional_tools" && Array.isArray(item.tools) + const currentAdditionalToolsIndex = body.input.findIndex((item, index) => + isCurrentItem(item, index) && isPlainObject(item) && item.type === "additional_tools" && Array.isArray(item.tools) ); - if (additionalToolsIndex < 0) return { ...body, tools: mergeLoadedTools([], loadedTools) }; - const additionalTools = body.input[additionalToolsIndex]; + if (currentAdditionalToolsIndex < 0) { + return { ...body, tools: mergeLoadedTools([], loadedTools) }; + } + + const additionalTools = body.input[currentAdditionalToolsIndex]; if (!isPlainObject(additionalTools) || !Array.isArray(additionalTools.tools)) return body; const tools = mergeLoadedTools(additionalTools.tools, loadedTools); if (tools === additionalTools.tools) return body; const input = [...body.input]; - input[additionalToolsIndex] = { ...additionalTools, tools }; + input[currentAdditionalToolsIndex] = { ...additionalTools, tools }; return { ...body, input }; } diff --git a/src/adapters/opencode-go-additional-tools.ts b/src/adapters/opencode-go-additional-tools.ts index ea41ec1aa17..c76b0ad8a00 100644 --- a/src/adapters/opencode-go-additional-tools.ts +++ b/src/adapters/opencode-go-additional-tools.ts @@ -1,3 +1,5 @@ +import { hasItemProvenance, isHistoricalItem } from "../responses/item-provenance"; + function isRecord(value: unknown): value is Record { return !!value && typeof value === "object" && !Array.isArray(value); } @@ -20,13 +22,32 @@ export function isOpenCodeGoResponsesUrl(responseUrl: string): boolean { && !destination.href.includes("?") && !destination.href.includes("#"); } -/** Console Go accepts public tools but rejects the private additional_tools input wrapper. */ -export function normalizeOpenCodeGoAdditionalTools( +const XAI_RESPONSES_HOSTS = new Set(["api.x.ai", "cli-chat-proxy.grok.com"]); + +/** + * True when the Responses target is an official xAI endpoint (public API or Grok CLI proxy). + */ +export function isXaiResponsesUrl(responseUrl: string): boolean { + let destination: URL; + try { + destination = new URL(responseUrl); + } catch { + return false; + } + return destination.protocol === "https:" + && XAI_RESPONSES_HOSTS.has(destination.hostname.toLowerCase()) + && (destination.port === "" || destination.port === "443"); +} + +/** + * Promotes client-loaded `additional_tools` input items into top-level `tools` and strips + * the wrapper from `input`. Required for gateways whose Rust/OpenAI-strict deserializer + * rejects unknown items in the `input` array. + */ +export function promoteResponsesAdditionalTools( body: unknown, - responseUrl: string, replayPrefixLength = 0, ): unknown { - if (!isOpenCodeGoResponsesUrl(responseUrl)) return body; if (!isRecord(body) || !Array.isArray(body.input)) return body; // Do not replace a malformed top-level catalog with a partial promoted one. if (body.tools !== undefined && !Array.isArray(body.tools)) return body; @@ -36,13 +57,15 @@ export function normalizeOpenCodeGoAdditionalTools( const currentTurnStart = Number.isFinite(replayPrefixLength) ? Math.min(body.input.length, Math.max(0, Math.trunc(replayPrefixLength))) : 0; + const hasProvenance = body.input.some(hasItemProvenance); let changed = false; for (const [index, item] of body.input.entries()) { if (isRecord(item) && item.type === "additional_tools" && Array.isArray(item.tools)) { changed = true; // Replayed wrappers are conversation history, not authority for the current request. - // Go cannot accept the wrapper itself, so remove it without promoting its catalog. - if (index < currentTurnStart) continue; + // Gateways cannot accept the wrapper itself, so remove it without promoting its catalog. + const isHistorical = hasProvenance ? isHistoricalItem(item) : (index < currentTurnStart); + if (isHistorical) continue; // Custom/search/namespace lowering already owns identity and deduplication. This pass // only moves declarations, including hosted tools that intentionally have no name. for (const tool of item.tools) promoted.push(tool); @@ -52,3 +75,13 @@ export function normalizeOpenCodeGoAdditionalTools( } return changed ? { ...body, input, tools: [...(body.tools ?? []), ...promoted] } : body; } + +/** Console Go accepts public tools but rejects the private additional_tools input wrapper. */ +export function normalizeOpenCodeGoAdditionalTools( + body: unknown, + responseUrl: string, + replayPrefixLength = 0, +): unknown { + if (!isOpenCodeGoResponsesUrl(responseUrl)) return body; + return promoteResponsesAdditionalTools(body, replayPrefixLength); +} diff --git a/src/adapters/routed-agent-messages.ts b/src/adapters/routed-agent-messages.ts index 2ca67162a72..e9c57d17cc6 100644 --- a/src/adapters/routed-agent-messages.ts +++ b/src/adapters/routed-agent-messages.ts @@ -1,3 +1,5 @@ +import { hasItemProvenance, isHistoricalItem, REPLAY_HISTORICAL_ITEM } from "../responses/item-provenance"; + /** * `agent_message` is Codex's private multi-agent input item: it exists only in the ChatGPT * Codex backend's schema. Codex replays every sub-agent reply in the history it sends, so @@ -36,6 +38,7 @@ export function normalizeRoutedAgentMessages( changed = true; return { type: "message", role: "user", + ...(hasItemProvenance(message) ? { [REPLAY_HISTORICAL_ITEM]: isHistoricalItem(message) } : {}), content: [ ...(Object.keys(identities).length ? [{ type: "input_text", text: `Agent message ${JSON.stringify(identities)}` }] : []), ...content, diff --git a/src/adapters/xai-web-search.ts b/src/adapters/xai-web-search.ts index 836eb0ca6df..d871234ae73 100644 --- a/src/adapters/xai-web-search.ts +++ b/src/adapters/xai-web-search.ts @@ -2,6 +2,7 @@ import type { OcxProviderConfig } from "../types"; import { debugProviderDiagnostic } from "../lib/debug"; import { isXaiResponsesDestination } from "../providers/xai-transport"; import { isOpenCodeGoResponsesUrl } from "./opencode-go-additional-tools"; +import { hasItemProvenance, isHistoricalItem } from "../responses/item-provenance"; const CODEX_WEB_SEARCH_TOOL = "web_search"; const CODEX_WEB_SEARCH_PREVIEW_TOOL = "web_search_preview"; @@ -190,8 +191,8 @@ export function normalizeXaiResponsesWebSearch( const xaiDestination = isXaiResponsesDestination(provider); if (!xaiDestination && !isOpenCodeGoGrokResponses(target?.modelId, target?.responseUrl)) return body; // On Go, a wrapper emptied here stays in place: Go promotion (opencode-go-additional-tools.ts) - // removes every wrapper before the wire and splits them into history and current turn by - // index against _replayPrefixLen, so deleting one would shift the current turn into history. + // removes every wrapper before the wire. Retain positions for callers with untagged input; + // transformed adapter input carries per-item historical/live provenance instead. const keepEmptiedWrappers = !xaiDestination; let next: Record = body; @@ -264,7 +265,9 @@ export function injectXaiResponsesXSearch( const input = Array.isArray(body.input) ? body.input : undefined; const inputStart = input ? currentInputStart(input.length, replayPrefixLength) : 0; - const currentInput = input?.slice(inputStart) ?? []; + const currentInput = input?.some(hasItemProvenance) + ? input.filter(item => !isHistoricalItem(item)) + : input?.slice(inputStart) ?? []; const currentXSearchDeclared = hasToolType(body.tools, XAI_SEARCH_TOOL) || currentInput.some(item => isPlainObject(item) diff --git a/src/adapters/zed-additional-tools.ts b/src/adapters/zed-additional-tools.ts index 73a18ca4533..10b52677e52 100644 --- a/src/adapters/zed-additional-tools.ts +++ b/src/adapters/zed-additional-tools.ts @@ -1,4 +1,5 @@ import { isPlainObject as isRecord } from "./openai-responses/internal"; +import { hasItemProvenance, isHistoricalItem, tagHistoricalInputItems } from "../responses/item-provenance"; function toolKey(tool: Record): string | undefined { if (typeof tool.type !== "string" || !tool.type) return undefined; @@ -8,8 +9,8 @@ function toolKey(tool: Record): string | undefined { /** * Zed's OpenAI Responses schema accepts message and call items, not Codex's private - * additional_tools wrapper. Drop every wrapper. Promote only wrappers at or after the - * replayed prefix: earlier ones are history, not the current turn's tool catalog. + * additional_tools wrapper. Drop every wrapper. Item provenance owns current-turn + * membership; a numeric prefix is accepted only for input not yet tagged. * A non-array tools field is left untouched so a malformed catalog is not replaced * by a partial promotion. */ @@ -17,16 +18,15 @@ export function liftZedAdditionalTools(providerRequest: Record, if (!Array.isArray(providerRequest.input)) return; if (providerRequest.tools !== undefined && !Array.isArray(providerRequest.tools)) return; - const currentTurnStart = Number.isFinite(replayPrefixLength) - ? Math.min(providerRequest.input.length, Math.max(0, Math.trunc(replayPrefixLength))) - : 0; + const taggedInput = providerRequest.input.some(hasItemProvenance) ? providerRequest.input + : tagHistoricalInputItems(providerRequest.input, Number.isFinite(replayPrefixLength) ? replayPrefixLength : 0); const input: unknown[] = []; const promoted: unknown[] = []; let changed = false; - for (const [index, item] of providerRequest.input.entries()) { + for (const item of taggedInput) { if (isRecord(item) && item.type === "additional_tools" && Array.isArray(item.tools)) { changed = true; - if (index >= currentTurnStart) promoted.push(...item.tools); + if (!isHistoricalItem(item)) promoted.push(...item.tools); continue; } input.push(item); diff --git a/src/adapters/zed.ts b/src/adapters/zed.ts index a9bb6104df4..e156a59748c 100644 --- a/src/adapters/zed.ts +++ b/src/adapters/zed.ts @@ -354,12 +354,7 @@ export function createZedAdapter(provider: OcxProviderConfig, cacheRetention?: A : item, ); } - if (zedProvider === "open_ai") { - // Codex carries this turn's client tools in a private additional_tools input item. - // Zed rejects that variant, so promote the current turn and drop every wrapper - // before the built-in tool filter runs. - liftZedAdditionalTools(providerRequest, parsed._replayPrefixLen ?? 0); - } + // OpenAI wrappers were promoted before delegation; serialized input has no provenance tags. if (zedProvider === "open_ai" && Array.isArray(providerRequest.tools)) { // Zed's Responses decoder supports function/custom tools, not Codex's built-in web_search. const supported = (tool: unknown) => isRecord(tool) && (tool.type === "function" || tool.type === "custom"); diff --git a/src/config/schema/leaf-validators.ts b/src/config/schema/leaf-validators.ts index 245222ad08e..3eacf076d56 100644 --- a/src/config/schema/leaf-validators.ts +++ b/src/config/schema/leaf-validators.ts @@ -360,6 +360,7 @@ export const providerConfigSchema = z.object({ inferContextWindow: z.boolean().optional(), modelSupportsServiceTier: z.record(z.string().min(1), z.boolean()).optional(), modelSuppressSyntheticMax: z.record(z.string().min(1), z.boolean()).optional(), + promoteAdditionalTools: z.boolean().optional(), preserveResponsesReasoningContent: z.boolean().optional(), preserveResponsesInputItemIds: z.boolean().optional(), preserveResponsesMessageMetadata: z.boolean().optional(), diff --git a/src/providers/model-rename-fields.ts b/src/providers/model-rename-fields.ts index ea003cedccc..a97202d8bdc 100644 --- a/src/providers/model-rename-fields.ts +++ b/src/providers/model-rename-fields.ts @@ -29,6 +29,7 @@ export const PROVIDER_MODEL_RENAME_ROLES = { supportsServiceTier: "none", fastEnabled: "none", modelSupportsServiceTier: "record", + promoteAdditionalTools: "none", preserveResponsesReasoningContent: "none", preserveResponsesInputItemIds: "none", preserveResponsesMessageMetadata: "none", diff --git a/src/responses/custom-tool-compat.ts b/src/responses/custom-tool-compat.ts index 4b15bc9f9ff..30a894f06c2 100644 --- a/src/responses/custom-tool-compat.ts +++ b/src/responses/custom-tool-compat.ts @@ -232,7 +232,8 @@ function rewriteForUpstream( } let changed = false; - const next: Record = {}; + // Recursive tool lowering must retain the input item's historical/live symbol marker. + const next: Record = { ...value }; for (const [key, entry] of Object.entries(value)) { const rewritten = rewriteForUpstream(entry, names, callIds); next[key] = rewritten; diff --git a/src/responses/item-provenance.ts b/src/responses/item-provenance.ts new file mode 100644 index 00000000000..4cf5b7c365f --- /dev/null +++ b/src/responses/item-provenance.ts @@ -0,0 +1,79 @@ +export const REPLAY_HISTORICAL_ITEM = Symbol.for("ocx.responses.replayHistoricalItem"); + +function isPlainObject(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +/** + * Mark an input item object as belonging to the historical replay prefix. + * Uses an enumerable symbol so it is preserved across shallow object spreads (`{ ...item }`), + * but completely ignored by `JSON.stringify` and `Object.keys`. + */ +export function markHistoricalItem(item: T): T { + Object.defineProperty(item, REPLAY_HISTORICAL_ITEM, { + value: true, + enumerable: true, + configurable: true, + writable: true, + }); + return item; +} + +/** + * Check whether an input item originated in the historical replay. + */ +export function isHistoricalItem(item: unknown): boolean { + return ( + typeof item === "object" + && item !== null + && (item as Record)[REPLAY_HISTORICAL_ITEM] === true + ); +} + +/** A live marker survives even when a rewrite removes every historical item. */ +export function hasItemProvenance(item: unknown): boolean { + return typeof item === "object" && item !== null + && typeof (item as Record)[REPLAY_HISTORICAL_ITEM] === "boolean"; +} + +/** Replacement and synthesized items inherit the source's historical or live tag. */ +export function copyItemProvenance(source: unknown, target: T): T { + if (hasItemProvenance(source)) { + Object.assign(target, { [REPLAY_HISTORICAL_ITEM]: isHistoricalItem(source) }); + } + return target; +} + +/** + * Tag both historical and live items so an empty replay prefix remains distinguishable + * from an input that has never been tagged. Input that already carries provenance keeps it: + * an earlier rewrite (Zed lifting a replayed wrapper) may have shifted items past the + * numeric prefix, so re-tagging by index would mislabel current-turn items as history. + */ +export function tagHistoricalInputItems(input: unknown[], prefixLength: number): unknown[] { + if (!Array.isArray(input)) return input; + if (input.some(hasItemProvenance)) return input; + const prefix = Math.min(input.length, Math.max(0, Math.trunc(prefixLength))); + if (prefix <= 0) return input; + return input.map((item, index) => { + if (isPlainObject(item)) { + return { ...item, [REPLAY_HISTORICAL_ITEM]: index < prefix }; + } + return item; + }); +} + +/** + * Compute the current replay prefix length dynamically. + * If items carry provenance markers, the boundary is the index of the first non-historical item. + * Otherwise, falls back to the provided fallback prefix length. + */ +export function computeReplayPrefixLength(input: unknown[], fallback = 0): number { + if (!Array.isArray(input)) return fallback; + const hasProvenance = input.some(hasItemProvenance); + if (!hasProvenance) { + return Math.min(input.length, Math.max(0, Math.trunc(fallback))); + } + const firstLiveIndex = input.findIndex(item => !isHistoricalItem(item)); + return firstLiveIndex === -1 ? input.length : firstLiveIndex; +} diff --git a/src/responses/namespace-tool-compat.ts b/src/responses/namespace-tool-compat.ts index 24b26e8a2ef..0c24ac3e87e 100644 --- a/src/responses/namespace-tool-compat.ts +++ b/src/responses/namespace-tool-compat.ts @@ -1,6 +1,7 @@ import { dottedToolName, namespacedToolName } from "../types"; import { collectResponsesToolGroups } from "./tool-groups"; import { collectAmbiguousDottedAliases, dottedAliasIsUnambiguous } from "./tool-name-aliases"; +import { hasItemProvenance, isHistoricalItem } from "./item-provenance"; export interface RoutedNamespaceToolIdentity { namespace: string; @@ -344,13 +345,25 @@ function rewriteInputItem(item: unknown, plan: NamespaceRewritePlan, emitted: Se export function rewriteRoutedNamespaceToolsForUpstream( body: unknown, convertedCustomToolNames?: ReadonlySet, + replayPrefixLength?: number, ): { body: unknown; aliases: Map; } { if (!isPlainObject(body)) return { body, aliases: new Map() }; - const groups = collectResponsesToolGroups(body); + const currentTurnStart = Number.isFinite(replayPrefixLength) + ? Math.max(0, Math.trunc(replayPrefixLength!)) + : 0; + const hasProvenance = Array.isArray(body.input) && body.input.some(hasItemProvenance); + const isHistorical = (item: unknown, index: number) => + hasProvenance ? isHistoricalItem(item) : index < currentTurnStart; + const groups = collectResponsesToolGroups(Array.isArray(body.input) + ? { ...body, input: body.input.filter((item, index) => !isHistorical(item, index)) } + : body); const plan = buildRewritePlan(groups); + const historicalPlan = buildRewritePlan(collectResponsesToolGroups({ + input: Array.isArray(body.input) ? body.input.filter(isHistorical) : [], + })); // Deliberately not gated on the plan being non-empty: a turn whose catalog is absent can still // replay call items carrying a private `namespace`. @@ -360,8 +373,13 @@ export function rewriteRoutedNamespaceToolsForUpstream( let input = body.input; if (Array.isArray(body.input)) { let inputChanged = false; - const rewrittenInput = body.input.map(item => { - const next = rewriteInputItem(item, plan, emitted); + const historicalEmitted = new Set(); + const rewrittenInput = body.input.map((item, index) => { + const historicalCatalog = isHistorical(item, index) && isPlainObject(item) && item.type === "additional_tools"; + const itemEmitted = historicalCatalog + ? historicalEmitted + : emitted; + const next = rewriteInputItem(item, historicalCatalog ? historicalPlan : plan, itemEmitted); if (next !== item) inputChanged = true; return next; }); diff --git a/src/server/auth-cors.ts b/src/server/auth-cors.ts index 4d124d4ac7e..11059690ed2 100644 --- a/src/server/auth-cors.ts +++ b/src/server/auth-cors.ts @@ -1089,6 +1089,7 @@ const PROVIDER_CONFIG_FIELD_POLICY = { annotateEmptyToolOutputs: "editor", supportsServiceTier: "editor", modelSupportsServiceTier: "editor", + promoteAdditionalTools: "editor", preserveResponsesReasoningContent: "editor", preserveResponsesInputItemIds: "editor", preserveResponsesMessageMetadata: "editor", diff --git a/src/server/management/provider-overwrite-carry.ts b/src/server/management/provider-overwrite-carry.ts index 08f3ceff314..6b1a9b0d550 100644 --- a/src/server/management/provider-overwrite-carry.ts +++ b/src/server/management/provider-overwrite-carry.ts @@ -51,6 +51,7 @@ const RETRY_POLICY_VALIDATORS = { export const PROVIDER_COMPAT_CARRY_FIELDS = [ "preserveReasoningContentModels", "preserveResponsesReasoningContent", + "promoteAdditionalTools", "requiresReasoningPlaceholderModels", "foldDeveloperRoleToSystem", "singleLeadingSystemMessage", @@ -167,6 +168,8 @@ export function providerCompatFieldConfigError(provider: Record if (hideRawReasoning !== undefined && typeof hideRawReasoning !== "boolean") return "hideRawReasoning must be a boolean"; const preserveResponses = provider.preserveResponsesReasoningContent; if (preserveResponses !== undefined && typeof preserveResponses !== "boolean") return "preserveResponsesReasoningContent must be a boolean"; + const promoteAdditionalTools = provider.promoteAdditionalTools; + if (promoteAdditionalTools !== undefined && typeof promoteAdditionalTools !== "boolean") return "promoteAdditionalTools must be a boolean"; const wire = provider.reasoningWireFormat; if (wire !== undefined && !PROVIDER_REASONING_WIRE_FORMATS.includes(wire as never)) { return `reasoningWireFormat must be one of: ${PROVIDER_REASONING_WIRE_FORMATS.join(", ")}`; @@ -225,6 +228,13 @@ export function applyProviderCompatPatchFields( else return { error: "preserveResponsesReasoningContent must be a boolean" }; touched = true; } + if (Object.hasOwn(rawBody, "promoteAdditionalTools")) { + const value = rawBody.promoteAdditionalTools; + if (value === null) delete next.promoteAdditionalTools; + else if (typeof value === "boolean") next.promoteAdditionalTools = value; + else return { error: "promoteAdditionalTools must be a boolean" }; + touched = true; + } if (Object.hasOwn(rawBody, "inferContextWindow")) { const value = rawBody.inferContextWindow; if (value === null) delete next.inferContextWindow; diff --git a/src/server/management/provider-routes.ts b/src/server/management/provider-routes.ts index 84d0e1752b5..75a833095ad 100644 --- a/src/server/management/provider-routes.ts +++ b/src/server/management/provider-routes.ts @@ -908,6 +908,7 @@ export async function handleProviderRoutes(ctx: ManagementContext): Promise 0, allowPrivateNetwork: p.allowPrivateNetwork === true, preserveResponsesReasoningContent: p.preserveResponsesReasoningContent, + promoteAdditionalTools: p.promoteAdditionalTools, liveModels: p.liveModels !== false, requestPacing: p.requestPacing, models: p.models ?? [], diff --git a/src/server/responses-undeclared-tool-guard.ts b/src/server/responses-undeclared-tool-guard.ts index d64a802b2ab..6a2784c7c68 100644 --- a/src/server/responses-undeclared-tool-guard.ts +++ b/src/server/responses-undeclared-tool-guard.ts @@ -1,4 +1,5 @@ import { UNDECLARED_TOOL_CALL_ERROR_CODE } from "../lib/errors"; +import { hasItemProvenance, isHistoricalItem } from "../responses/item-provenance"; import { collectAmbiguousDottedAliases, dottedAliasIsUnambiguous, @@ -136,6 +137,9 @@ export function currentTurnWireToolCatalogBody( replayPrefixLength: number | undefined, ): unknown { if (!isPlainObject(body) || !Array.isArray(body.input)) return body; + if (body.input.some(hasItemProvenance)) { + return { ...body, input: body.input.filter(item => !isHistoricalItem(item)) }; + } if (typeof replayPrefixLength !== "number" || !Number.isFinite(replayPrefixLength)) return body; const start = Math.min(body.input.length, Math.max(0, Math.trunc(replayPrefixLength))); if (start === 0) return body; diff --git a/src/server/responses/passthrough-dispatch.ts b/src/server/responses/passthrough-dispatch.ts index fdabdf1de4c..86d3afdcbea 100644 --- a/src/server/responses/passthrough-dispatch.ts +++ b/src/server/responses/passthrough-dispatch.ts @@ -482,10 +482,9 @@ export async function preparePassthroughExchange( for (const callType of collectProviderExecutedCallTypes(clientToolAuthorizationBody)) { providerExecutedCallTypes.add(callType); } - const currentOutboundCatalog = currentTurnWireToolCatalogBody( - outboundRequestBody, - replayedInputPrefixLength, - ); + // Serialization drops provenance, so never slice transformed input by the raw prefix. + // Current wrappers were captured above; adapter-injected hosted tools are top-level. + const currentOutboundCatalog = { tools: outboundRequestBody?.tools }; for (const callType of collectProviderExecutedCallTypes(currentOutboundCatalog)) { providerExecutedCallTypes.add(callType); } diff --git a/src/types/provider.ts b/src/types/provider.ts index 61c66de5bc5..c2358cc46d7 100644 --- a/src/types/provider.ts +++ b/src/types/provider.ts @@ -434,6 +434,14 @@ export interface OcxProviderConfig { fastEnabled?: boolean; /** Exact upstream model ids that override the provider-level service-tier capability. */ modelSupportsServiceTier?: Record; + /** + * Whether to promote Codex Desktop's `additional_tools` input items into top-level `tools` + * and strip the wrapper from `input`. Enabled automatically for OpenCode Go and xAI + * (`api.x.ai` and `cli-chat-proxy.grok.com`) except on forward-auth routes, which need an + * explicit `true`; other strict gateways that return HTTP 422 for unknown item types in + * `input` can opt in here. + */ + promoteAdditionalTools?: boolean; /** * Responses upstream whose native contract accepts plaintext reasoning replay * (DeepSeek documents reasoning items with plaintext content). When set, the diff --git a/structure/gui-and-management-api.md b/structure/gui-and-management-api.md index ef49522edeb..bbdb03ccf55 100644 --- a/structure/gui-and-management-api.md +++ b/structure/gui-and-management-api.md @@ -275,7 +275,7 @@ per-request first-party callback reads that live object; a failed write leaves i | Startup safety | `GET /api/startup-health` reports whether injected Codex routing is restart-safe, with secret-free service/shim diagnostics. `POST /api/startup-action` provides allowlisted one-click installation for the background service or launcher shim. On Windows a healthy script shim is CLI-only; Codex Desktop requires the background service for full protection. Optional Codex compatibility uses the [confirmed local runtime](clients/codex-desktop.md#optional-compatibility-runtime), [machine-local dashboard consent](clients/codex-desktop.md#dashboard-controls), and [revision-bound next-start preference with shared awaited teardown](clients/codex-desktop.md#proxy-startup-preference); saving that preference performs no runtime action. | | Windows tray | `GET/POST /api/windows-tray` controls an owned, per-user HKCU login tray. The tray delegates fixed actions to the CLI and is never a proxy supervisor or restart-protection signal. | | Updates | `GET /api/update/check`, `POST /api/update/run`, and `GET /api/update/status` own dashboard self-update state. A launched worker PID is persisted in `update-job.json`; dead PIDs recover immediately, while legacy active records without a PID recover only after ten minutes. Live PIDs remain exclusive regardless of record age. `GET /api/update/check` and `POST /api/update/run` await one per-channel asynchronous registry lookup and write successful results through to the package cache; run passes that result to the job starter. `GET /api/update/badge` only reads the cache and reports unknown after 40 hours, on missing cache, or on channel mismatch. The badge links to the update surface rather than gating other actions. `GET /api/update/badge?surface=desktop&session=` projects only that process-local Tauri session; missing or expired state is unknown and never falls back to the package cache. `POST /api/update/desktop-snapshot` is a 1 KiB bounded display-state mutation with no install permission. It accepts the existing admin-token principal or a dedicated single-use, ten-second snapshot capability bound to nonce, method, path, PID, port and the SHA-256 digest of the exact body bytes; that capability cannot authorize another route. | -| Providers | Create/update/delete ordinary provider configs and enrich registry metadata. A `POST /api/providers` overwrite of an existing name keeps the nine operator compatibility settings (`PROVIDER_COMPAT_CARRY_FIELDS` in `src/server/management/provider-overwrite-carry.ts`) and the stored key pool only while the destination (adapter, normalized base URL, auth mode when named) is unchanged. It keeps omitted `hideRawReasoning` on every overwrite of the same name, including destination moves, because it is a display policy; a submitted boolean wins. An omitted `inferContextWindow` likewise keeps the latest live same-name state after asynchronous validation, including `false` or a concurrent clear; a submitted boolean wins. It never merges the rest of the old row. `PATCH` is a field mask and keeps every field it does not name; `hideRawReasoning`, `preserveResponsesReasoningContent` and `inferContextWindow` accept a boolean or `null` to clear them, rejecting other types. `GET /api/providers` exposes configured `inferContextWindow` as an optional boolean without substituting a default; its [custom-only, default-off inference policy](catalog-vendor-context.md) is unchanged. Custom Responses provider forms expose `preserveResponsesReasoningContent` as a default-off plaintext replay switch; unrelated settings saves preserve omission and curated/OpenAI destinations keep their existing defaults. The reserved `openai` card exposes Pool(default)/Direct account mode; `openai-apikey` remains the separate API route. | +| Providers | Create/update/delete ordinary provider configs and enrich registry metadata. A `POST /api/providers` overwrite of an existing name keeps the ten operator compatibility settings (`PROVIDER_COMPAT_CARRY_FIELDS` in `src/server/management/provider-overwrite-carry.ts`) and the stored key pool only while the destination (adapter, normalized base URL, auth mode when named) is unchanged. It keeps omitted `hideRawReasoning` on every overwrite of the same name, including destination moves, because it is a display policy; a submitted boolean wins. An omitted `inferContextWindow` likewise keeps the latest live same-name state after asynchronous validation, including `false` or a concurrent clear; a submitted boolean wins. It never merges the rest of the old row. `PATCH` is a field mask and keeps every field it does not name; `hideRawReasoning`, `preserveResponsesReasoningContent`, `promoteAdditionalTools` and `inferContextWindow` accept a boolean or `null` to clear them, rejecting other types. `GET /api/providers` exposes configured `inferContextWindow` as an optional boolean without substituting a default; its [custom-only, default-off inference policy](catalog-vendor-context.md) is unchanged. Custom Responses provider forms expose `preserveResponsesReasoningContent` as a default-off plaintext replay switch; unrelated settings saves preserve omission and curated/OpenAI destinations keep their existing defaults. The reserved `openai` card exposes Pool(default)/Direct account mode; `openai-apikey` remains the separate API route. | | Models | Fetch routed model lists, disabled model visibility, and catalog-facing ids. New non-OAuth registration holds exposure until authoritative discovery; 20 or more distinct switch rows start OFF without disabling the provider. Pending rows cannot accept visibility changes. Ordinary model reads also apply the [new-arrival catalog contract](catalog.md#shared-catalog) before returning models. With a matching persisted configuration, discovery records the baseline, NEW badges, and automatic disables together; HTTP listing does not wait for a separate Codex sync. Inventory already drifted from disk uses a detached policy projection: arrivals remain disabled in management rows without changing live config or disk. | | OAuth | Login/status/logout for OAuth-backed providers, plus multiauth account management: `GET /api/oauth/accounts`, `PUT /api/oauth/accounts/active`, `PUT /api/oauth/accounts/alias`, `PUT /api/oauth/accounts/pause`, and `DELETE /api/oauth/accounts` list masked accounts per provider, switch the active one, edit its display-only alias, pause/resume a generic OAuth account, and remove one. Paused generic OAuth accounts are excluded from request selection, 429 failover, and proactive token refresh; pausing an active account selects the next usable account when available, and resuming an account restores an active selection if the current one remains paused. With no unpaused account, requests return 403 rather than a login error. Kiro account-list rows include the current automatic-selection projection and closed exclusion reason; an active singleton may still send. The login flow itself is `GET /api/oauth/providers`, `POST /api/oauth/login`, `POST /api/oauth/login/code`, `POST /api/oauth/login/cancel`, `POST /api/oauth/logout`, and `GET /api/oauth/status`; pool controls are `GET/PUT/PATCH /api/oauth/accounts/pool` and `POST /api/oauth/accounts/clear-cooldown`. Login accepts `addAccount: true` to force a fresh browser identity. Meta Muse login start and manual-code continuation require the server-resolved `gui-session` principal before credential acquisition or code submission (including reauth); see the [provider contract](providers-and-adapters.md). Device flows return a structured `deviceCode`; the GUI highlights and copies it before the user opens the verification page. Google Antigravity account rows in `gui/src/components/provider-workspace/ProviderAuthPanel.tsx` display the account-summary `plan` as a badge and include it in the account button’s accessible name. The value is the canonical upstream tier ID (for example `free-tier` or `gemini-pro`), preserved literally across locales rather than inferred as a product name. Absent plans render no badge; pause and reauthentication controls keep their existing behavior. | | Key providers | `GET /api/key-providers` exposes API-key provider presets for setup and dashboard flows, and `GET/POST/DELETE /api/keys` owns the proxy's own admission keys. Machine links live in `src/server/management/link-routes.ts`: dashboard sessions reach `GET /api/link/candidates`, `POST /api/link/probe`, `POST /api/link/confirm-host` and `POST /api/link/apply`, meaning a paired session or, on a standalone runtime, the current loopback-issued session on trusted loopback ingress; `POST /api/link/join` instead requires an operator-paired session because it durably redirects local client traffic, and also refuses a runtime that is not standalone (`409 standalone_required`) or not listening on its configured port (`409 join_port_mismatch`) before any SSH, then restarts this runtime as a client; `GET /api/link/status` reports `joinAvailable` and typed `joinDenied` causes only to GUI-session callers, distinguishing pairing, runtime-role and port refusals without changing the admin/CLI DTO, so the dashboard enables the Child role only when the join gates pass; `GET /api/link/status`, `POST /api/link/{id}/reconnect` and `DELETE /api/link/{id}` also accept the admin token on a trusted loopback ingress, and reconnect restarts a Home-initiated link's tunnel now; `POST /api/link/issue` accepts only that admin token. Tailscale-identity sessions are refused on every link route. See [Remote Link](remote-link.md). Multi-key pool per key-auth provider: `GET /api/providers/keys`, `POST /api/providers/keys`, `PUT /api/providers/keys/active`, `PUT /api/providers/keys/alias`, `DELETE /api/providers/keys` masked list, add (upsert + activate), switch, rename, and remove keys. `provider.apiKey` always mirrors the active pool entry so routing stays single-key. | diff --git a/structure/transports/responses-wire-shapes.md b/structure/transports/responses-wire-shapes.md index a8b49f9dde0..eced4076992 100644 --- a/structure/transports/responses-wire-shapes.md +++ b/structure/transports/responses-wire-shapes.md @@ -157,13 +157,13 @@ policy. Explicitly selected Codex Lite ordinary unreferenced hosted-history lowering follows [the hosted-search continuation boundary](../providers-and-adapters.md#hosted-search-continuation-binding); non-Lite/unknown requests, native compact and public API retain structural history; portable compaction keeps its existing trigger and default projection. Go's non-forward Responses request path moves valid `additional_tools` wrappers into top-level `tools` through `src/adapters/opencode-go-additional-tools.ts`. Placement runs after existing custom/search/namespace lowering and before code-mode, compaction and final hosted-tool pruning. -It does not recalculate wire identities or response aliases. The matcher reads the constructed +It reads per-item replay provenance without recalculating wire identities or response aliases. The matcher reads the constructed send URL, resolving it with URL semantics, and requires HTTPS `opencode.ai`, the standard port and exact `/zen/go/v1/responses`. Normal and endpoint-inclusive bases or split `responsesPath` configurations agree; a custom path resolving to Zen or elsewhere does not acquire Go placement. Credentials, query, fragment, foreign hosts and other resource paths are excluded. The existing URL constructor canonicalizes trailing base slashes before this check. Malformed wrappers remain unchanged and -the shared mixed-ciphertext agent-message gate remains fail-closed. +the shared mixed-ciphertext agent-message gate remains fail-closed. Responses passthrough clones and tags historical and current input items before any rewrite, including identity repair; symbol provenance survives message replacement, prompt-cache cleanup, filtering and tool lowering without entering serialized requests. Promotion uses provenance rather than positions in shortened input. Discovery-source filtering follows the [current-turn catalog contract](responses.md#responses-httpsse) before nested tool definitions enter a live catalog. The canonical `opencode-go` registry entry defaults to `statelessResponses: true` because Go rejects reasoning ciphertext combined with `previous_response_id` (#3838). Existing derive @@ -221,7 +221,7 @@ the destination is the whole predicate, because Meta's refusal is a gateway sche Muse model it serves, its default `muse-spark-1.3` is not a Contributor id, and a missing model id still strips. Because the predicate is the host, a custom provider pointed at that exact URL gets the same strip. -Meta's `api.meta.ai` route uses `src/adapters/openai-responses/muse-tool-catalog.ts` to capture current declarations before history rewrites or deduplication: current wrappers become `tools`, declarations retain their schemas over search discoveries, historical discovery stays replay data, and historical custom calls retain their conversion without live authority. Historical catalogs are not validated as current declarations. The reserved `functions` group and nested function definitions become public function declarations. Catalog validation also covers string/omitted input; unrepresentable namespace children and malformed current catalogs fail locally with HTTP 400 (`tests/responses/responses-muse-tool-choice.test.ts`). +Meta's `api.meta.ai` route uses `src/adapters/openai-responses/muse-tool-catalog.ts` to capture current declarations before history rewrites or deduplication: current wrappers become `tools`, declarations retain their schemas over search discoveries, historical discovery stays replay data, and historical custom calls retain their conversion without live authority. Historical catalogs are not validated as current declarations; once input items carry replay provenance tags, the tags decide membership instead of the numeric prefix, so an earlier rewrite that shortened the input (Zed lifting a replayed wrapper) cannot shift the boundary. The reserved `functions` group and nested function definitions become public function declarations. Catalog validation also covers string/omitted input; unrepresentable namespace children and malformed current catalogs fail locally with HTTP 400 (`tests/responses/responses-muse-tool-choice.test.ts`). Direct Meta Muse / Meta Model Responses (`https://api.meta.ai/v1`) also rejects function tool names longer than 64 characters or containing characters outside `[a-zA-Z0-9_-]`. After namespace flattening, `src/responses/muse-tool-name-alias.ts` rewrites those identities on the `api.meta.ai` diff --git a/structure/transports/responses.md b/structure/transports/responses.md index 82585590f56..d67380d7ec7 100644 --- a/structure/transports/responses.md +++ b/structure/transports/responses.md @@ -203,27 +203,24 @@ when the shared xAI normalizer can preserve its meaning; an unsafe function is o letting one incompatible declaration reject the entire request before inference. This is scoped to `cli-chat-proxy.grok.com`: public `api.x.ai` keeps native root unions, as do unrelated Responses gateways. Both top-level `tools` and Responses Lite `additional_tools` pass through this policy. +Codex Desktop packages client tools inside an `additional_tools` input item; strict gateways rejecting non-ResponseItem input types—including official xAI endpoints (`api.x.ai`, `cli-chat-proxy.grok.com`), OpenCode Go, and custom providers opting in with `promoteAdditionalTools: true`—promote declarations to top-level `tools` and strip the wrapper from `input`. `src/responses/item-provenance.ts` tags historical and live input objects with JSON-invisible symbols before replay repair; historical/live tags survive replacements, synthesized outputs and bridge restoration; transformed-input consumers read each tag, never a recalculated prefix. The numeric replay prefix tags only input that carries no tag yet, so a rewrite that already tagged and then shortened the input (Zed lifting a replayed wrapper) keeps a current-turn `tool_search_output` current. +Historical wrappers retain call-rewriting context with separate rewrite plans and declaration sets. Only current-turn `tool_search_output` discoveries join a live wrapper, top-level tools or the top-level fallback; source provenance is checked before merging declarations. Historical discoveries remain replay context and grant no current hosted-tool authority. `tests/providers/xai/xai-loaded-tool-provenance.test.ts` covers these paths and stored two-turn continuations. -Only the ROOT rejects a union, so exclusivity is preserved by moving it down rather than widening -it: a root `oneOf` whose branches differ in one property becomes that property's `oneOf`, or its -`anyOf` when the branches are provably disjoint and the two keywords describe the same set. That -property is also promoted into `required`, because absent it matched every branch — which the root -`oneOf` rejects. Branches that are wholly identical validate nothing and have no faithful -flattening, so they omit the tool. The walk carries depth, node, and variant budgets, since nested -unions are combinatorial and a `$ref` diamond amplifies the same way without ever cycling; -exceeding a budget omits that one function rather than expanding until memory is gone. - -Omitting a function makes `tool_choice` the loose end. A selector naming a dropped tool would reach -Grok as a dangling reference, and relaxing it to `auto` is worse — the turn would quietly run -without the tool the caller required. So an `allowed_tools` list drops the omitted entries while any -remain, and a selection with nothing left to point at fails locally with the same 400 a tool catalog -this proxy cannot lower already returns. +Only the ROOT rejects a union, so exclusivity is preserved by moving it down rather than widening it: +a root `oneOf` whose branches differ in one property becomes that property's `oneOf`, or its `anyOf` +when the branches are provably disjoint and the two keywords describe the same set. That property is also +promoted into `required`, because absent it matched every branch — which the root `oneOf` rejects. +Branches that are wholly identical validate nothing and have no faithful flattening, so they omit the tool. +The walk carries depth, node, and variant budgets, since nested unions are combinatorial and a `$ref` +diamond amplifies the same way without ever cycling; exceeding a budget omits that one function. + +Omitting a function makes `tool_choice` the loose end: a selector naming a dropped tool would reach +Grok as a dangling reference, and relaxing to `auto` runs without the tool the caller required. +An `allowed_tools` list drops omitted entries while any remain, and an empty selection fails with 400. The same noncanonical boundary strips ChatGPT's private `external_web_access` bit from routed -`web_search` declarations. The public tool remains enabled and all other options remain intact; -canonical OpenAI forwarding preserves the bit. xAI's public Responses schema enables browsing by -the presence of `web_search` and rejects the private argument, so forwarding it made the first -post-namespace request fail with HTTP 400. +`web_search` declarations while keeping the public tool and other options intact (canonical OpenAI +forwarding preserves the bit; xAI rejects the private argument with HTTP 400). The option-aware `openai` provider uses `openai-responses` with `authMode: "forward"`. Pool mode resolves main plus added accounts through affinity/quota/cooldown ownership; Direct forwards only diff --git a/tests/adapters/routed-agent-messages.test.ts b/tests/adapters/routed-agent-messages.test.ts index 2a44d118622..4e247cd1615 100644 --- a/tests/adapters/routed-agent-messages.test.ts +++ b/tests/adapters/routed-agent-messages.test.ts @@ -1,12 +1,24 @@ import { expect, test } from "bun:test"; import { createResponsesPassthroughAdapter } from "../../src/adapters/openai-responses"; import { normalizeRoutedAgentMessages } from "../../src/adapters/routed-agent-messages"; +import { hasItemProvenance, isHistoricalItem, tagHistoricalInputItems } from "../../src/responses/item-provenance"; import { parseRequest } from "../../src/responses/parser"; import { routeModel } from "../../src/router"; import { createTranslatorBudget } from "../../src/lib/translator-budget"; import type { OcxProviderConfig } from "../../src/types"; const base: OcxProviderConfig = { adapter: "openai-responses", baseUrl: "https://opencode.ai/zen/go/v1", authMode: "key", apiKey: "synthetic-key" }; + +test("agent-message replacement carries historical and live provenance", () => { + const input = tagHistoricalInputItems([ + { type: "agent_message", content: [{ type: "input_text", text: "past" }] }, + { type: "agent_message", content: [{ type: "input_text", text: "current" }] }, + ], 1); + const result = normalizeRoutedAgentMessages({ input }) as { input: unknown[] }; + expect(result.input.map(hasItemProvenance)).toEqual([true, true]); + expect(result.input.map(isHistoricalItem)).toEqual([true, false]); + expect(JSON.stringify(result)).not.toContain("replayHistoricalItem"); +}); const body = () => ({ model: "muse-spark-1.3-contributor", input: [{ type: "agent_message", id: "amsg_test", author: "/root/reader", recipient: "/root/checker", content: [{ type: "input_text", text: "Exact assignment\nwith lines." }] }], stream: true }); test("Responses converts plaintext task and peer messages without mutating replay or losing routing identities", async () => { diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index e3ac7ff2a95..7b19eb71736 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -1494,6 +1494,9 @@ "ws-replay-refusal-attribution.test.ts": "responses", "combo-replay-refusal-attribution.test.ts": "responses", "ws-upstream.test.ts": "responses", + "xai-additional-tools.test.ts": "providers/xai", + "xai-replay-provenance.test.ts": "providers/xai", + "xai-loaded-tool-provenance.test.ts": "providers/xai", "xai-client.test.ts": "images", "xai-empty-catalog-tool-choice.test.ts": "providers/xai", "xai-no-stop.test.ts": "providers/xai", diff --git a/tests/providers/xai/xai-additional-tools.test.ts b/tests/providers/xai/xai-additional-tools.test.ts new file mode 100644 index 00000000000..26f93c6d581 --- /dev/null +++ b/tests/providers/xai/xai-additional-tools.test.ts @@ -0,0 +1,421 @@ +import { providerConfigSchema } from "../../../src/config/schema/leaf-validators"; +import { describe, expect, test } from "bun:test"; +import { createResponsesPassthroughAdapter as createProductionAdapter } from "../../../src/adapters/openai-responses"; +import { + isXaiResponsesUrl, + promoteResponsesAdditionalTools, +} from "../../../src/adapters/opencode-go-additional-tools"; +import { withTestTranslatorBudget } from "../../helpers/translator-budget"; +import { CODEX_GPT5_IDENTITY_LINE, identifyRoutedModel } from "../../../src/adapters/identity"; +import { tagHistoricalInputItems } from "../../../src/responses/item-provenance"; +import { rewriteRoutedCustomToolsForUpstream } from "../../../src/responses/custom-tool-compat"; +import type { OcxProviderConfig } from "../../../src/types"; + +const lookup = { + type: "function" as const, + name: "lookup", + description: "finds records", + parameters: { type: "object", properties: { q: { type: "string" } } }, +}; + +function build(providerConfig: OcxProviderConfig, rawBody: Record, replayPrefixLen?: number) { + const adapter = withTestTranslatorBudget(createProductionAdapter(providerConfig)); + const request = adapter.buildRequest({ + modelId: "grok-4.7", + context: { messages: [] }, + stream: true, + options: {}, + _rawBody: rawBody, + ...(replayPrefixLen !== undefined ? { _replayPrefixLen: replayPrefixLen } : {}), + }); + return { request, body: JSON.parse(request.body) as Record }; +} + +describe("xAI Responses additional_tools promotion", () => { + test("custom-tool lowering carries historical and live provenance into promotion", () => { + const input = tagHistoricalInputItems([ + { type: "additional_tools", tools: [{ type: "custom", name: "retired", description: "past tool" }] }, + { type: "additional_tools", tools: [lookup] }, + ], 1); + const lowered = rewriteRoutedCustomToolsForUpstream({ input }, false); + const promoted = promoteResponsesAdditionalTools(lowered.body) as { tools: unknown[] }; + expect(promoted.tools).toEqual([lookup]); + }); + + test("custom-tool lowering cannot reactivate a historical wrapper", () => { + const { body } = build({ + adapter: "openai-responses", baseUrl: "https://api.x.ai/v1", supportsResponsesCustomTools: false, + }, { input: [ + { type: "additional_tools", tools: [{ type: "custom", name: "retired", description: "past tool" }] }, + { type: "additional_tools", tools: [lookup] }, + ] }, 1); + expect(body.tools).toEqual([lookup]); + expect(body.input).toEqual([]); + }); + + test("forward cache-marker cleanup cannot promote historical declarations", () => { + const retired = { ...lookup, name: "retired", prompt_cache_breakpoint: true }; + const raw = { input: [ + { type: "additional_tools", tools: [retired] }, + { type: "additional_tools", tools: [lookup] }, + ] }; + const before = structuredClone(raw); + const { body } = build({ + adapter: "openai-responses", baseUrl: "https://chatgpt.com/backend-api/codex", + authMode: "forward", promoteAdditionalTools: true, + }, raw, 1); + expect(body.tools).toEqual([lookup]); + expect(body.input).toEqual([]); + expect(raw).toEqual(before); + }); + + test("forward opt-in preserves live tools after removing the historical identity message", () => { + const provider: OcxProviderConfig = { + adapter: "openai-responses", + baseUrl: "https://chatgpt.com/backend-api/codex", + authMode: "forward", + promoteAdditionalTools: true, + }; + const live = { ...lookup, name: "live" }; + const raw = { + input: [ + { type: "message", role: "developer", content: identifyRoutedModel(CODEX_GPT5_IDENTITY_LINE, "old-model") }, + { type: "additional_tools", tools: [live] }, + ], + }; + const before = structuredClone(raw); + const adapter = withTestTranslatorBudget(createProductionAdapter(provider)); + const request = adapter.buildRequest({ + modelId: "gpt-5", context: { messages: [] }, stream: false, options: {}, + _rawBody: raw, _replayPrefixLen: 1, + }, { headers: new Headers({ authorization: "Bearer caller-fixture", "chatgpt-account-id": "caller-account" }) }); + const body = JSON.parse(request.body); + expect(body.tools).toEqual([live]); + expect(body.input).toEqual([]); + expect(request.url).toBe("https://chatgpt.com/backend-api/codex/responses"); + expect(new Headers(request.headers).get("authorization")).toBe("Bearer caller-fixture"); + expect(new Headers(request.headers).get("chatgpt-account-id")).toBe("caller-account"); + expect(raw).toEqual(before); + for (const item of raw.input) expect(Object.getOwnPropertySymbols(item)).toEqual([]); + }); + + test("isXaiResponsesUrl matches official xAI endpoints and rejects lookalikes", () => { + expect(isXaiResponsesUrl("https://api.x.ai/v1/responses")).toBe(true); + expect(isXaiResponsesUrl("https://api.x.ai:443/v1/responses")).toBe(true); + expect(isXaiResponsesUrl("https://cli-chat-proxy.grok.com/v1/responses")).toBe(true); + expect(isXaiResponsesUrl("https://cli-chat-proxy.grok.com:443/v1/responses")).toBe(true); + + expect(isXaiResponsesUrl("http://api.x.ai/v1/responses")).toBe(false); + expect(isXaiResponsesUrl("https://api.x.ai:8443/v1/responses")).toBe(false); + expect(isXaiResponsesUrl("https://api.x.ai.evil.test/v1/responses")).toBe(false); + expect(isXaiResponsesUrl("https://example.com/v1/responses")).toBe(false); + }); + + test("promoteResponsesAdditionalTools strips wrapper from input and moves tools to top-level", () => { + const raw = { + input: [ + { type: "message", role: "user", content: "hello" }, + { type: "additional_tools", tools: [lookup] }, + ], + }; + const result = promoteResponsesAdditionalTools(raw) as { input: unknown[]; tools: unknown[] }; + expect(result.input).toEqual([{ type: "message", role: "user", content: "hello" }]); + expect(result.tools).toEqual([lookup]); + }); + + test("promoteResponsesAdditionalTools respects replay prefix length", () => { + const historicalTool = { type: "function", name: "old_tool" }; + const raw = { + input: [ + { type: "additional_tools", tools: [historicalTool] }, + { type: "message", role: "user", content: "turn 1" }, + { type: "additional_tools", tools: [lookup] }, + ], + }; + // replayPrefixLength = 2 means index 0 and 1 are historical; only index 2 is current turn + const result = promoteResponsesAdditionalTools(raw, 2) as { input: unknown[]; tools: unknown[] }; + expect(result.input).toEqual([{ type: "message", role: "user", content: "turn 1" }]); + expect(result.tools).toEqual([lookup]); + }); + + test("xAI provider strips additional_tools and promotes declarations on buildRequest", () => { + const xaiProvider: OcxProviderConfig = { + adapter: "openai-responses", + baseUrl: "https://api.x.ai/v1", + headers: { authorization: "Bearer xai-key" }, + }; + + const raw = { + model: "grok-4.7", + input: [ + { type: "message", role: "user", content: "test query" }, + { type: "additional_tools", tools: [lookup] }, + ], + }; + + const { body } = build(xaiProvider, raw); + expect(body.input).toEqual([{ type: "message", role: "user", content: "test query" }]); + expect(body.tools).toBeArray(); + expect((body.tools as any[]).some(t => t.name === "lookup")).toBe(true); + }); + + test("cli-chat-proxy.grok.com also promotes additional_tools on buildRequest", () => { + const grokCliProvider: OcxProviderConfig = { + adapter: "openai-responses", + baseUrl: "https://cli-chat-proxy.grok.com/v1", + headers: { authorization: "Bearer grok-cli-token" }, + }; + + const raw = { + model: "grok-4.7", + input: [ + { type: "message", role: "user", content: "test query" }, + { type: "additional_tools", tools: [lookup] }, + ], + }; + + const { body } = build(grokCliProvider, raw); + expect(body.input).toEqual([{ type: "message", role: "user", content: "test query" }]); + expect((body.tools as any[]).some(t => t.name === "lookup")).toBe(true); + }); + + test("custom provider with promoteAdditionalTools: true promotes on buildRequest", () => { + const customProvider = { + adapter: "openai-responses", + baseUrl: "https://custom-gateway.internal/v1", + promoteAdditionalTools: true, + headers: { authorization: "Bearer internal-token" }, + }; + + const raw = { + model: "custom-model", + input: [ + { type: "message", role: "user", content: "custom query" }, + { type: "additional_tools", tools: [lookup] }, + ], + }; + + const { body } = build(customProvider, raw); + expect(body.input).toEqual([{ type: "message", role: "user", content: "custom query" }]); + expect((body.tools as any[]).some(t => t.name === "lookup")).toBe(true); + }); + + test("providerConfigSchema validates promoteAdditionalTools as optional boolean", () => { + expect(providerConfigSchema.safeParse({ adapter: "openai-responses", baseUrl: "https://example.com/v1", promoteAdditionalTools: true }).success).toBe(true); + expect(providerConfigSchema.safeParse({ adapter: "openai-responses", baseUrl: "https://example.com/v1", promoteAdditionalTools: false }).success).toBe(true); + expect(providerConfigSchema.safeParse({ adapter: "openai-responses", baseUrl: "https://example.com/v1", promoteAdditionalTools: "true" }).success).toBe(false); + }); + + test("unapproved custom provider without flag leaves input untouched", () => { + const unapprovedProvider: OcxProviderConfig = { + adapter: "openai-responses", + baseUrl: "https://unapproved.internal/v1", + headers: { authorization: "Bearer token" }, + }; + + const raw = { + model: "custom-model", + input: [ + { type: "message", role: "user", content: "custom query" }, + { type: "additional_tools", tools: [lookup] }, + ], + }; + + const { body } = build(unapprovedProvider, raw); + expect(body.input).toEqual(raw.input); + }); + + test("custom provider with authMode: forward and promoteAdditionalTools: true promotes tools", () => { + const forwardOptInProvider: OcxProviderConfig = { + adapter: "openai-responses", + baseUrl: "https://forward-proxy.example.com/v1", + authMode: "forward", + promoteAdditionalTools: true, + }; + + const raw = { + model: "custom-model", + input: [ + { type: "message", role: "user", content: "forward query" }, + { type: "additional_tools", tools: [lookup] }, + ], + }; + + const { body } = build(forwardOptInProvider, raw); + expect(body.input).toEqual([{ type: "message", role: "user", content: "forward query" }]); + expect((body.tools as any[]).some(t => t.name === "lookup")).toBe(true); + }); + + test("custom provider with authMode: forward without flag leaves input untouched", () => { + const forwardNoOptInProvider: OcxProviderConfig = { + adapter: "openai-responses", + baseUrl: "https://api.x.ai/v1", + authMode: "forward", + }; + + const raw = { + model: "custom-model", + input: [ + { type: "message", role: "user", content: "forward query" }, + { type: "additional_tools", tools: [lookup] }, + ], + }; + + const { body } = build(forwardNoOptInProvider, raw); + expect(body.input).toEqual(raw.input); + }); + + test("promoteResponsesAdditionalTools recalculates replay boundary when reasoning items are dropped", () => { + const xaiProvider: OcxProviderConfig = { + adapter: "openai-responses", + baseUrl: "https://api.x.ai/v1", + dropResponsesReasoningItems: true, + headers: { authorization: "Bearer xai-key" }, + }; + + const raw = { + model: "grok-4.7", + input: [ + { type: "reasoning", content: "thinking..." }, + { type: "message", role: "user", content: "hello" }, + { type: "additional_tools", tools: [lookup] }, + ], + }; + + // Original prefix len is 2 (reasoning + user message). + // When reasoning is dropped, prefix len must adjust from 2 to 1. + // additional_tools moves from index 2 to index 1, which matches the new prefix boundary (>= 1), + // so its tools are promoted to top-level tools. + const { body } = build(xaiProvider, raw, 2); + expect(body.input).toEqual([{ type: "message", role: "user", content: "hello" }]); + expect(body.tools).toBeArray(); + expect((body.tools as any[]).some(t => t.name === "lookup")).toBe(true); + }); + + test("promoteResponsesAdditionalTools recalculates replay boundary when canonical-only items are dropped", () => { + const xaiProvider: OcxProviderConfig = { + adapter: "openai-responses", + baseUrl: "https://api.x.ai/v1", + headers: { authorization: "Bearer xai-key" }, + }; + + const raw = { + model: "grok-4.7", + input: [ + { type: "configuration_update", reasoning: { effort: "high" } }, + { type: "message", role: "user", content: "hello" }, + { type: "additional_tools", tools: [lookup] }, + ], + }; + + // Original prefix len is 2 (configuration_update + user message). + // When configuration_update is dropped, prefix len must adjust from 2 to 1. + // additional_tools moves from index 2 to index 1, which matches the new prefix boundary (>= 1), + // so its tools are promoted to top-level tools. + const { body } = build(xaiProvider, raw, 2); + expect(body.input).toEqual([{ type: "message", role: "user", content: "hello" }]); + expect(body.tools).toBeArray(); + expect((body.tools as any[]).some(t => t.name === "lookup")).toBe(true); + }); + test.each(["configuration_update", "reasoning"])("Finding 3: removing the entire %s replay prefix preserves current tools", type => { + const provider: OcxProviderConfig = { + adapter: "openai-responses", + baseUrl: "https://api.x.ai/v1", + dropResponsesReasoningItems: true, + }; + const { body } = build(provider, { + input: [{ type, content: "historical" }, { type: "additional_tools", tools: [lookup] }], + }, 1); + expect(body.tools).toEqual([lookup]); + }); + + test("Finding 1: historical declarations do not deduplicate current declarations before promotion", () => { + const xaiProvider: OcxProviderConfig = { + adapter: "openai-responses", + baseUrl: "https://api.x.ai/v1", + headers: { authorization: "Bearer xai-key" }, + }; + const raw = { + model: "grok-4.7", + input: [ + { type: "additional_tools", tools: [lookup] }, + { type: "message", role: "user", content: "hello" }, + { type: "additional_tools", tools: [lookup] }, + ], + }; + const { body } = build(xaiProvider, raw, 2); + expect(body.tools).toBeArray(); + expect((body.tools as any[]).some(t => t.name === "lookup")).toBe(true); + }); + + test("Finding 1: historical namespaced declarations do not deduplicate current namespaced declarations before promotion", () => { + const xaiProvider: OcxProviderConfig = { + adapter: "openai-responses", + baseUrl: "https://api.x.ai/v1", + headers: { authorization: "Bearer xai-key" }, + }; + const namespacedLookup = { + type: "namespace", + name: "functions", + tools: [lookup], + }; + const raw = { + model: "grok-4.7", + input: [ + { type: "additional_tools", tools: [namespacedLookup] }, + { type: "message", role: "user", content: "hello" }, + { type: "additional_tools", tools: [namespacedLookup] }, + ], + }; + const { body } = build(xaiProvider, raw, 2); + expect(body.tools).toBeArray(); + expect((body.tools as any[]).some(t => t.name === "lookup")).toBe(true); + }); + + test("Finding 2: newly loaded search tools survive when first additional_tools wrapper is historical", () => { + const xaiProvider: OcxProviderConfig = { + adapter: "openai-responses", + baseUrl: "https://api.x.ai/v1", + headers: { authorization: "Bearer xai-key" }, + }; + const raw = { + model: "grok-4.7", + input: [ + { type: "additional_tools", tools: [{ type: "function", name: "old_tool" }] }, + { type: "tool_search_output", tools: [lookup] }, + ], + }; + const { body } = build(xaiProvider, raw, 1); + expect(body.tools).toBeArray(); + expect((body.tools as any[]).some(t => t.name === "lookup")).toBe(true); + }); + + test("Finding 1: a historical bare declaration does not suppress the live functions namespace", () => { + const { body } = build({ adapter: "openai-responses", baseUrl: "https://api.x.ai/v1" }, { + input: [ + { type: "additional_tools", tools: [lookup] }, + { type: "additional_tools", tools: [{ type: "namespace", name: "functions", tools: [lookup] }] }, + ], + }, 1); + expect(body.tools).toEqual([lookup]); + }); + + test("Finding 3: synthesizing missing historical tool output does not reactivate retired tools", () => { + const pWithPairing: OcxProviderConfig = { + adapter: "openai-responses", + baseUrl: "https://api.x.ai/v1", + requiresPairedResponsesToolResults: true, + headers: { authorization: "Bearer xai-key" }, + }; + const raw = { + model: "grok-4.7", + input: [ + { type: "function_call", call_id: "c1", name: "some_call" }, + { type: "additional_tools", tools: [{ type: "function", name: "retired_tool" }] }, + ], + }; + const { body } = build(pWithPairing, raw, 2); + const retired = (body.tools as any[] ?? []).find(t => t.name === "retired_tool"); + expect(retired).toBeUndefined(); + }); +}); diff --git a/tests/providers/xai/xai-loaded-tool-provenance.test.ts b/tests/providers/xai/xai-loaded-tool-provenance.test.ts new file mode 100644 index 00000000000..70586cacfc2 --- /dev/null +++ b/tests/providers/xai/xai-loaded-tool-provenance.test.ts @@ -0,0 +1,128 @@ +import { expect, test } from "bun:test"; +import { createResponsesPassthroughAdapter } from "../../../src/adapters/openai-responses"; +import { promoteClientLoadedTools } from "../../../src/adapters/openai-responses/tool-schema"; +import { tagHistoricalInputItems } from "../../../src/responses/item-provenance"; +import { handleResponses } from "../../../src/server/responses"; +import type { OcxConfig } from "../../../src/types"; +import { acquireOwnedSpendHome } from "../../helpers/owned-spend-home"; +import { withTestTranslatorBudget } from "../../helpers/translator-budget"; + +const live = { type: "function", name: "live", parameters: { type: "object", properties: {} } }; +const retired = { type: "web_search", name: "retired_search" }; +const current = { type: "web_search", name: "current_search" }; +const wrap = (tools: unknown[]) => ({ type: "additional_tools", tools }); +const discovery = (tools: unknown[]) => ({ type: "tool_search_output", call_id: "search", tools }); +const provider = { + adapter: "openai-responses", baseUrl: "https://api.x.ai/v1", authMode: "key", + apiKey: "fixture-key", xaiResponsesXSearch: true, +}; + +for (const catalog of ["wrapper", "top-level", "fallback"] as const) { + for (const historical of [true, false]) { + test(`${catalog}: ${historical ? "historical" : "current"} discoveries keep their authority boundary`, () => { + const input = [discovery([retired]), ...(catalog === "wrapper" ? [wrap([live])] : [])]; + const raw = { input, ...(catalog === "top-level" ? { tools: [live] } : {}) }; + const before = JSON.stringify(raw); + const body = JSON.parse(withTestTranslatorBudget(createResponsesPassthroughAdapter(provider)).buildRequest({ + modelId: "grok-4.7", context: { messages: [] }, stream: false, options: {}, + _rawBody: raw, _replayPrefixLen: historical ? 1 : 0, + }, { headers: new Headers() }).body); + const declared = catalog === "fallback" ? [] : [live]; + const active = catalog === "wrapper" ? [{ type: "x_search" }, ...declared, retired] + : [...declared, retired, { type: "x_search" }]; + expect(body.tools ?? []).toEqual(historical ? declared : active); + expect(JSON.stringify(raw)).toBe(before); + expect(input.every(item => Object.getOwnPropertySymbols(item).length === 0)).toBe(true); + }); + } + + test(`${catalog}: source tags override stale positions after reordering`, () => { + const [oldResult, liveResult, wrapper] = tagHistoricalInputItems([ + discovery([retired]), discovery([current]), wrap([live]), + ], 1); + const body = { + input: [liveResult, oldResult, ...(catalog === "wrapper" ? [wrapper] : [])], + ...(catalog === "top-level" ? { tools: [live] } : {}), + }; + const loaded = promoteClientLoadedTools(body, body.input.length) as { + tools?: unknown[]; input: Array<{ type: string; tools?: unknown[] }>; + }; + const declared = catalog === "fallback" ? [] : [live]; + expect(catalog === "wrapper" ? loaded.input.at(-1)?.tools : loaded.tools).toEqual([...declared, current]); + expect(body.input).toEqual([liveResult, oldResult, ...(catalog === "wrapper" ? [wrapper] : [])]); + }); + + test(`${catalog}: untagged callers use the original replay prefix for discoveries`, () => { + const body = { + input: [discovery([retired]), discovery([current]), ...(catalog === "wrapper" ? [wrap([live])] : [])], + ...(catalog === "top-level" ? { tools: [live] } : {}), + }; + const loaded = promoteClientLoadedTools(body, 1) as { + tools?: unknown[]; input: Array<{ type: string; tools?: unknown[] }>; + }; + expect(catalog === "wrapper" ? loaded.input.at(-1)?.tools : loaded.tools) + .toEqual([...(catalog === "fallback" ? [] : [live]), current]); + }); +} + +for (const catalog of ["wrapper", "top-level", "fallback"] as const) { + for (const source of ["historical", "current", "absent"] as const) { + test(`two-turn ${catalog}: ${source} discovery retains only current hosted declarations`, async () => { + const savedFetch = globalThis.fetch; + const releaseHome = acquireOwnedSpendHome(); + const responseId = `resp_loaded_provenance_${catalog}_${source}`; + const outbound: Array<{ tools?: Array> }> = []; + const hostedCall = { + type: "custom_tool_call", id: "ctc_provenance", call_id: "xs_call-round3", + name: "x_keyword_search", input: "{}", status: "completed", + }; + globalThis.fetch = (async (_input, init) => { + outbound.push(JSON.parse(String(init?.body))); + return Response.json({ + id: outbound.length === 1 ? responseId : `${responseId}_next`, status: "completed", + output: outbound.length === 1 + ? [{ type: "message", role: "assistant", content: [{ type: "output_text", text: "done" }] }] + : [hostedCall], + }); + }) as typeof fetch; + const config: OcxConfig = { port: 0, defaultProvider: "fixture", providers: { fixture: provider } }; + const post = (body: Record) => handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "fixture/grok-4.7", stream: false, ...body }), + }), config, { model: "", provider: "" }); + try { + const first = await post({ input: [ + wrap([{ type: "web_search" }]), ...(source === "historical" ? [discovery([retired])] : []), + { role: "user", content: "old search" }, + ] }); + expect(first.status).toBe(200); + await first.text(); + const second = await post({ + previous_response_id: responseId, + input: [ + ...(catalog === "wrapper" ? [wrap([live])] : []), + ...(source === "current" ? [discovery([current])] : []), + { role: "user", content: "continue" }, + ], + ...(catalog === "top-level" ? { tools: [live] } : {}), + }); + const result = await second.json() as { output?: unknown[]; error?: { code?: string } }; + expect(outbound).toHaveLength(2); + // An omitted catalog leaves call handling to the client; an explicit live catalog + // activates the server guard. Both paths must exclude historical outbound authority. + const accepted = source === "current" || catalog === "fallback"; + expect(second.status).toBe(accepted ? 200 : 502); + if (accepted) expect(result.output).toEqual([hostedCall]); + else expect(result.error?.code).toBe("undeclared_tool_call"); + const declared = catalog === "fallback" ? [] : [live]; + const active = catalog === "wrapper" ? [{ type: "x_search" }, ...declared, current] + : [...declared, current, { type: "x_search" }]; + expect(outbound[1]?.tools ?? []) + .toEqual(source === "current" ? active : declared); + } finally { + globalThis.fetch = savedFetch; + releaseHome(); + } + }); + } +} diff --git a/tests/providers/xai/xai-replay-provenance.test.ts b/tests/providers/xai/xai-replay-provenance.test.ts new file mode 100644 index 00000000000..fdcc0b619af --- /dev/null +++ b/tests/providers/xai/xai-replay-provenance.test.ts @@ -0,0 +1,94 @@ +import { expect, test } from "bun:test"; +import { createResponsesPassthroughAdapter } from "../../../src/adapters/openai-responses"; +import { repairOrphanedInputItems, restoreBridgedWebSearchCalls } from "../../../src/adapters/openai-responses/tool-output-recovery"; +import { injectXaiResponsesXSearch } from "../../../src/adapters/xai-web-search"; +import { liftZedAdditionalTools } from "../../../src/adapters/zed-additional-tools"; +import { currentTurnWireToolCatalogBody } from "../../../src/server/responses-undeclared-tool-guard"; +import { hasItemProvenance, isHistoricalItem, tagHistoricalInputItems } from "../../../src/responses/item-provenance"; +import { withTestTranslatorBudget } from "../../helpers/translator-budget"; +import { rememberBridgeSearchReplay } from "../../../src/responses/bridge-search-replay-cache"; + +const live = { type: "function", name: "live", parameters: { type: "object", properties: {} } }; +const wrap = (tools: unknown[]) => ({ type: "additional_tools", tools }); +const xai = { adapter: "openai-responses", baseUrl: "https://api.x.ai/v1", xaiResponsesXSearch: true }; +const task = { type: "function_call_output", id: "seed", name: "task", namespace: "agent", output: "old task" }; + +function build(seed: unknown) { + return JSON.parse(withTestTranslatorBudget(createResponsesPassthroughAdapter({ + ...xai, requiresPairedResponsesToolResults: true, + })).buildRequest({ + modelId: "grok-4.7", context: { messages: [] }, stream: false, options: {}, + _rawBody: { input: [seed, wrap([{ type: "web_search" }]), wrap([live])] }, _replayPrefixLen: 2, + }, { headers: new Headers() }).body); +} + +test("control: historical user seed does not reactivate historical hosted search", () => { + expect(build({ type: "message", role: "user", content: "old task" }).tools).toEqual([live]); +}); + +test("external task-input replacement must not reactivate historical hosted search", () => { + expect(build(task).tools).toEqual([live]); +}); + +for (const [name, seed, synthesize] of [ + ["external task", task, false], + ["orphan output", { type: "function_call_output", call_id: "old", output: "result" }, false], + ["synthetic output", { type: "function_call", call_id: "old", name: "lookup", arguments: "{}" }, true], +] as const) { + test(`${name} replacement retains both historical and live provenance`, () => { + for (const historical of [true, false]) { + const tagged = tagHistoricalInputItems([{ type: "message" }, seed], historical ? 2 : 1); + const body = repairOrphanedInputItems({ input: tagged }, false, synthesize) as { input: unknown[] }; + const replacement = body.input.at(-1); + expect(hasItemProvenance(replacement)).toBe(true); + expect(isHistoricalItem(replacement)).toBe(historical); + expect(Object.getOwnPropertySymbols(seed)).toEqual([]); + } + }); +} + +test("xAI injection ignores historical web search after a live item moves ahead of it", () => { + const [historical, current] = tagHistoricalInputItems([wrap([{ type: "web_search" }]), wrap([live])], 1); + const body = { input: [current, historical] }; + expect(injectXaiResponsesXSearch(body, xai, 0)).toBe(body); +}); + +test("bridge restoration retains historical and live provenance on both restored items", () => { + const scope = "xai-replay-provenance-fixture"; + rememberBridgeSearchReplay(scope, "ws_provenance", { + callId: "search_provenance", name: "web_search", argumentsText: "{}", output: "fixture result", + }); + for (const historical of [true, false]) { + const input = tagHistoricalInputItems([ + { type: "message" }, { type: "web_search_call", id: "ws_provenance" }, + ], historical ? 2 : 1); + const restored = restoreBridgedWebSearchCalls({ input }, scope) as { input: unknown[] }; + expect(restored.input).toHaveLength(3); + for (const item of restored.input.slice(1)) { + expect(hasItemProvenance(item)).toBe(true); + expect(isHistoricalItem(item)).toBe(historical); + } + } +}); + +test("historical x_search cannot suppress injection for a current web-search wrapper", () => { + const [historical, current] = tagHistoricalInputItems([ + wrap([{ type: "x_search" }]), wrap([{ type: "web_search" }]), + ], 1); + expect(injectXaiResponsesXSearch({ input: [current, historical] }, xai, 0)) + .toMatchObject({ tools: [{ type: "x_search" }] }); +}); + +test("Zed promotion reads provenance after input reordering", () => { + const [historical, current] = tagHistoricalInputItems([wrap([{ ...live, name: "retired" }]), wrap([live])], 1); + const body: Record = { input: [current, historical] }; + liftZedAdditionalTools(body, 0); + expect(body.tools).toEqual([live]); + expect(body.input).toEqual([]); +}); + +test("current-turn authorization reads provenance after input reordering", () => { + const [historical, current] = tagHistoricalInputItems([wrap([{ type: "x_search" }]), wrap([live])], 1); + expect(currentTurnWireToolCatalogBody({ input: [current, historical] }, 0)) + .toEqual({ input: [current] }); +}); diff --git a/tests/providers/zed-hosted-provider.test.ts b/tests/providers/zed-hosted-provider.test.ts index 2046c1d6670..a3763b2a8d4 100644 --- a/tests/providers/zed-hosted-provider.test.ts +++ b/tests/providers/zed-hosted-provider.test.ts @@ -446,6 +446,28 @@ describe("Zed OpenAI tool relay", () => { expect(rawBefore).toContain("reasoning"); }); + test("a current tool_search_output stays current after a replayed wrapper is lifted", async () => { + catalogFetch(); + const lookup = { type: "function", name: "lookup", parameters: { type: "object", properties: {} } }; + const parsed = parseRequest({ + model: "gpt-6.1-sol", + input: [ + { type: "additional_tools", tools: [{ type: "function", name: "old_tool", parameters: { type: "object", properties: {} } }] }, + { type: "tool_search_output", call_id: "search", status: "completed", execution: "client", tools: [lookup] }, + { type: "message", role: "user", content: [{ type: "input_text", text: "now" }] }, + ], + tools: [], + stream: true, + }); + parsed._zedAuthContext = { userId: "user-1" }; + parsed._replayPrefixLen = 1; + const adapter = withTestTranslatorBudget(createZedAdapter(provider)); + const request = await adapter.buildRequest(parsed); + const body = JSON.parse(request.body).provider_request; + expect(JSON.stringify(body.input)).not.toContain("old_tool"); + expect(body.tools.map((tool: { name?: string }) => tool.name)).toEqual(["lookup"]); + }); + async function relay(frames: unknown[]): Promise { catalogFetch(); const parsed = parseRequest({ model: "gpt-6.1-sol", input: "hello", stream: true }); diff --git a/tests/responses/openai-responses-passthrough.test.ts b/tests/responses/openai-responses-passthrough.test.ts index feee0328653..d5bbb818b3d 100644 --- a/tests/responses/openai-responses-passthrough.test.ts +++ b/tests/responses/openai-responses-passthrough.test.ts @@ -1579,7 +1579,8 @@ describe("OpenAI Responses passthrough sanitization", () => { for (const lite of [false, true]) { const body = build(lite); - const tools = lite ? body.input[0]?.tools : body.tools; + const tools = body.tools; + if (lite) expect(body.input).toEqual([]); expect(tools?.map(tool => tool.name)).toEqual([ "mcp__codex_app__safe_union", "mcp__codex_app__plain", @@ -2478,8 +2479,8 @@ describe("OpenAI Responses passthrough sanitization", () => { tool_choice: Record; }; - expect(body.tools).toBeUndefined(); - expect(body.input[0]?.tools).toEqual([{ type: "web_search" }]); + expect(body.tools).toEqual([{ type: "web_search" }]); + expect(body.input).toEqual([]); expect(body.tool_choice).toEqual({ type: "web_search" }); }); @@ -2641,10 +2642,7 @@ describe("OpenAI Responses passthrough sanitization", () => { tools: [{ type: "function", name: "read", defer_loading: true, parameters: {} }], }, ], - input: [{ - type: "additional_tools", - tools: [{ type: "function", name: "loose", defer_loading: true, parameters: {} }], - }], + input: [{ type: "additional_tools", tools: [{ type: "function", name: "loose", defer_loading: true, parameters: {} }] }], }, }, { headers: new Headers() }); const body = JSON.parse(request.body) as { @@ -2655,7 +2653,9 @@ describe("OpenAI Responses passthrough sanitization", () => { expect(body.tools[0]).toEqual({ type: "web_search" }); expect(body.tools[1]).toMatchObject({ type: "function", name: "workspace__read" }); expect(body.tools[1]).not.toHaveProperty("defer_loading"); - expect(body.input[0].tools[0]).not.toHaveProperty("defer_loading"); + expect(body.tools[2]).toMatchObject({ type: "function", name: "loose" }); + expect(body.tools[2]).not.toHaveProperty("defer_loading"); + expect(body.input).toEqual([]); }); test("preserves prompt_cache_key in the raw Responses passthrough body", () => { diff --git a/tests/responses/responses-muse-tool-choice.test.ts b/tests/responses/responses-muse-tool-choice.test.ts index 277335ee1ef..08683ea88a3 100644 --- a/tests/responses/responses-muse-tool-choice.test.ts +++ b/tests/responses/responses-muse-tool-choice.test.ts @@ -1,5 +1,7 @@ import { afterEach, describe, expect, test } from "bun:test"; import { createResponsesPassthroughAdapter as createResponsesPassthroughAdapterProduction } from "../../src/adapters/openai-responses"; +import { normalizeMuseToolCatalog } from "../../src/adapters/openai-responses/muse-tool-catalog"; +import { tagHistoricalInputItems } from "../../src/responses/item-provenance"; import { handleResponses, handleResponsesCompact } from "../../src/server/responses"; import type { OcxConfig, OcxProviderConfig } from "../../src/types"; import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; @@ -442,3 +444,17 @@ describe("Meta Muse additional tool catalogs", () => { } }); }); + +test("Meta catalog follows item provenance once an earlier rewrite shortened the input", () => { + const schema = { type: "object", properties: {} }; + const tagged = tagHistoricalInputItems([ + { type: "additional_tools", tools: [{ type: "function", name: "old_tool", parameters: schema }] }, + { type: "message", role: "user", content: [{ type: "input_text", text: "earlier" }] }, + { type: "tool_search_output", call_id: "search", status: "completed", execution: "client", tools: [{ type: "function", name: "lookup", parameters: schema }] }, + { type: "message", role: "user", content: [{ type: "input_text", text: "now" }] }, + ], 2); + // Zed lifted the replayed wrapper before the delegate ran, so the numeric prefix of 2 is stale. + const body = normalizeMuseToolCatalog({ model: "muse", input: tagged.slice(1) }, 2) as { tools?: { name?: string }[]; input: unknown[] }; + expect(body.tools?.map(tool => tool.name)).toEqual(["lookup"]); + expect(JSON.stringify(body.input)).not.toContain("old_tool"); +}); diff --git a/tests/server/management-provider-compat-carry.test.ts b/tests/server/management-provider-compat-carry.test.ts index b8ce5a41b7d..5722e8211de 100644 --- a/tests/server/management-provider-compat-carry.test.ts +++ b/tests/server/management-provider-compat-carry.test.ts @@ -127,6 +127,22 @@ const CASES: Record = { return (JSON.parse(String(built.body)) as { input: Array<{ content: unknown }> }).input[0]!.content; }, }, + promoteAdditionalTools: { + seed: { promoteAdditionalTools: true }, + probe: provider => { + const parsed = parseRequest({ + model: MODEL, + input: [ + { type: "message", role: "user", content: "hi" }, + { type: "additional_tools", tools: [tool] }, + ], + }); + parsed.modelId = MODEL; + const built = withTestTranslatorBudget(createResponsesPassthroughAdapter({ ...provider, adapter: "openai-responses" })).buildRequest(parsed, { headers: new Headers() }); + const body = JSON.parse(String(built.body)) as { input: Array<{ type: string }>; tools?: unknown[] }; + return { inputTypes: body.input.map(i => i.type), hasTools: Array.isArray(body.tools) && body.tools.length > 0 }; + }, + }, foldDeveloperRoleToSystem: { seed: { foldDeveloperRoleToSystem: true }, probe: provider => (nativeBody(provider, { @@ -287,6 +303,25 @@ describe("an overwrite that moves the provider carries none of it", () => { }); describe("PATCH and POST validate the settings they name", () => { + test("PATCH writes and clears promoteAdditionalTools", async () => { + await withServer({ relay: { adapter: "openai-responses", baseUrl: BASE_URL } }, async url => { + const set = await send(url, "/api/providers?name=relay", "PATCH", { promoteAdditionalTools: true }); + expect(set.status).toBe(200); + expect(loadConfig().providers.relay?.promoteAdditionalTools).toBe(true); + + const disable = await send(url, "/api/providers?name=relay", "PATCH", { promoteAdditionalTools: false }); + expect(disable.status).toBe(200); + expect(loadConfig().providers.relay?.promoteAdditionalTools).toBe(false); + + const bad = await send(url, "/api/providers?name=relay", "PATCH", { promoteAdditionalTools: "yes" }); + expect(bad.status).toBe(400); + + const clear = await send(url, "/api/providers?name=relay", "PATCH", { promoteAdditionalTools: null }); + expect(clear.status).toBe(200); + expect(loadConfig().providers.relay).not.toHaveProperty("promoteAdditionalTools"); + }); + }); + test("PATCH writes and clears foldDeveloperRoleToSystem and reasoningWireFormat", async () => { await withServer({ relay: { adapter: "openai-chat", baseUrl: BASE_URL } }, async url => { const set = await send(url, "/api/providers?name=relay", "PATCH", { diff --git a/tests/server/server-xai-responses-streaming.test.ts b/tests/server/server-xai-responses-streaming.test.ts index 8a2298972c9..03adfc36dac 100644 --- a/tests/server/server-xai-responses-streaming.test.ts +++ b/tests/server/server-xai-responses-streaming.test.ts @@ -477,11 +477,12 @@ describe("xAI OAuth Responses streaming opt-in", () => { type: string; tools?: Array<{ type: string; name?: string }>; }> | undefined; - const outboundTools = outboundInput?.find(item => item.type === "additional_tools")?.tools; + expect(outboundInput?.find(item => item.type === "additional_tools")).toBeUndefined(); + const outboundTools = outboundBody?.tools as Array<{ type: string; name?: string }> | undefined; expect(outboundTools?.some(tool => tool.type === "namespace")).toBe(false); expect(outboundTools?.find(tool => tool.name === "exec")?.type).toBe("function"); expect(outboundTools?.find(tool => tool.name === "collaboration__spawn_agent")?.type).toBe("function"); - expect(outboundBody?.tools).toEqual([{ type: "web_search" }]); + expect(outboundTools?.find(tool => tool.type === "web_search")).toEqual({ type: "web_search" }); const payloads = clientText .split(/\r?\n/) From a022a67fd8152045268405145d73ef696ff5afdd Mon Sep 17 00:00:00 2001 From: JUN Date: Fri, 30 Oct 2026 21:51:01 +0900 Subject: [PATCH 27/42] fix(claude): prune generated agents when the Claude integration is turned off (#7993) * fix(claude): prune generated agents when the Claude integration is turned off The integrations toggle persisted claudeCode.enabled=false but left ~/.claude/agents/ocx-*.md in place, so a native Claude Code session kept loading proxy-only agents whose placeholder model resolves to Haiku. Reconcile the definitions after the toggle persists, and again on a repeated Off. Closes #7968 * fix(claude): restore the in-memory block when the toggle save fails Otherwise a retry on the same config read the unsaved Off, took the repeated-Off branch and pruned agents while the saved integration was still on. Docs now describe the prune as best effort. * fix(claude): keep a published toggle value when post-publication bookkeeping fails ConfigWritePublishedError means the new value is on disk; restoring the old block in memory would make a later toggle report the wrong state. * fix(claude): reconcile agents only when the toggle turns Claude off Turning it on ran a model discovery request against the upstream from a settings toggle; On keeps its launch and startup sync, and only the Off prune is needed for #7968. --------- Co-authored-by: JUN <243035832+lidge-jun@users.noreply.github.com> --- .../src/content/docs/guides/claude-code.md | 5 ++ .../management/native-integration-routes.ts | 22 +++++ structure/subagents.md | 9 ++ .../claude-first-party-agent-sync.test.ts | 86 +++++++++++++++++++ 4 files changed, 122 insertions(+) diff --git a/docs-site/src/content/docs/guides/claude-code.md b/docs-site/src/content/docs/guides/claude-code.md index bf0cf28d7e9..c0b83efa0ce 100644 --- a/docs-site/src/content/docs/guides/claude-code.md +++ b/docs-site/src/content/docs/guides/claude-code.md @@ -856,6 +856,11 @@ Proxy startup/ensure, `ocx claude`, and relevant dashboard saves sync your featu overwritten or pruned; your own agents are never touched. - Files are atomically synced per file (write + rename). - `enabled: false` or `injectAgents: false` prunes all verified-owned definitions. +- Turning Claude Code off with the dashboard's integration toggle also prunes them right away + (best effort), so a native `claude` session started afterwards stops seeing proxy-only agents. + A file that cannot be deleted stays, and while a sibling OpenCodex instance owns + `~/.claude/agents` nothing is touched; turning it off again retries and also removes + definitions an older version left behind. - Successful CLI first-party saves and persisted Desktop first-party setup also attempt best-effort roster sync, including repeated setup to repair missing definitions. Refused setup and rollback do not sync agents. Turning first-party off alone does not disable agent registration. diff --git a/src/server/management/native-integration-routes.ts b/src/server/management/native-integration-routes.ts index 8b75adec412..be01108fbf8 100644 --- a/src/server/management/native-integration-routes.ts +++ b/src/server/management/native-integration-routes.ts @@ -1,4 +1,5 @@ import { readConfigForAdmittedMutation } from "../../config/persisted-mutation"; +import { ConfigWritePublishedError } from "../../config/persist-unlocked"; import { getConfigDir as configRoot, withConfigDirHardeningSettled as settleConfigRoot } from "../../config/paths"; import { WindowsSecretAclSettleTimeoutError } from "../../lib/windows-secret-acl"; import { mirrorDesiredEnabledOntoSnapshot } from "./claude-desktop-config-writers"; @@ -949,6 +950,9 @@ export async function handleNativeIntegrationRoutes(ctx: ManagementContext): Pro const enabled = body.enabled; if (claudeCodeEnabled(config) === enabled) { + // Re-run the same reconcile on a repeated OFF: definitions left behind by an older + // version that never pruned on disable are removed the next time the user says off. + if (!enabled) await ctx.syncClaudeAgentDefsBestEffort?.(); return jsonResponse({ ok: true, clientId: "claude", changed: false, state: enabled ? "current" : "absent", @@ -963,6 +967,10 @@ export async function handleNativeIntegrationRoutes(ctx: ManagementContext): Pro * sentinel the next startServer would silently convert a user's Auto auth mode into * a sticky manual subscription — a failure that surfaces nowhere near this route. */ + // Kept so a failed save leaves the shared in-memory config matching disk: otherwise a + // retry on the same object would read the unsaved value, take the repeated-OFF branch + // and prune agents while the persisted integration is still on. + const previousBlock = config.claudeCode === undefined ? undefined : structuredClone(config.claudeCode); commitClaudeCodeBlock(config, { ...(config.claudeCode ?? {}), enabled }); /* @@ -973,6 +981,11 @@ export async function handleNativeIntegrationRoutes(ctx: ManagementContext): Pro try { persist(config); } catch (error) { + // A published write already put the new value on disk; only an unpublished one rolls back. + if (!(error instanceof ConfigWritePublishedError)) { + if (previousBlock === undefined) delete config.claudeCode; + else config.claudeCode = previousBlock; + } if (isConfigLockError(error)) { return isLockContention(error) ? refusal(409, "claude", "config_busy", @@ -983,6 +996,15 @@ export async function handleNativeIntegrationRoutes(ctx: ManagementContext): Pro throw error; } + /* + * The generated ~/.claude/agents/ocx-*.md definitions route through this proxy; a + * native Claude Code session that still loads them after OFF follows their + * placeholder model (issue #7968). Prune them now instead of waiting for a later + * launch or startup sync; the disabled reconcile removes only marker-owned regular + * files and needs no model discovery. ON keeps its existing launch/startup sync. + */ + if (!enabled) await ctx.syncClaudeAgentDefsBestEffort?.(); + let interceptReason = enabled && interceptStartRefusal(ctx) ? "intercept_start_forbidden" : null; if (enabled && !interceptReason) { const outcome = await ensureManagementClaudeIntercept(ctx); diff --git a/structure/subagents.md b/structure/subagents.md index 5435786b3ef..9db8055c568 100644 --- a/structure/subagents.md +++ b/structure/subagents.md @@ -470,6 +470,15 @@ and stores only `features.multi_agent_v2.multi_agent_mode_hint_text` in Codex TO `null` removes that key. The hint affects new native Codex sessions when their v2 surface is active, without changing reasoning effort or the proxy guidance switch. +The integrations toggle `PUT /api/native-integrations/claude` reconciles the generated `ocx-*` definitions +through `syncClaudeAgentDefsBestEffort` after it persists Off, so the marker-owned regular files are +pruned at once without model discovery (On keeps the launch and startup sync); a repeated Off reconciles again to remove +definitions an older version left behind (#7968). The reconcile is best effort: an unlink failure +leaves that file and the toggle still succeeds. A save that fails before publishing restores the in-memory block, so a +retry on the same config is a real transition; one that published first keeps the new value. User-authored and symlinked `ocx-*` files are never +touched, and a live sibling instance still owns the directory. Coverage: +`tests/claude-integration/claude-first-party-agent-sync.test.ts`. + Claude Code `ocx-*` agent definitions consume the same effective `claudeCode.blockedSkills` policy as inbound bundle elision. When the list is non-empty (default: `claude-api`), generated definitions whose marker-stripped model resolves to a routed id receive a preventive instruction not to invoke diff --git a/tests/claude-integration/claude-first-party-agent-sync.test.ts b/tests/claude-integration/claude-first-party-agent-sync.test.ts index fd2f2428b2c..25e538c00b3 100644 --- a/tests/claude-integration/claude-first-party-agent-sync.test.ts +++ b/tests/claude-integration/claude-first-party-agent-sync.test.ts @@ -178,3 +178,89 @@ test("Desktop saved:false partial success does not reconcile agent definitions", spy.mockRestore(); } }); + + +async function toggleClaude(enabled: boolean, save: (config: OcxConfig) => void = () => { persist(); }) { + const url = new URL("http://127.0.0.1:10100/api/native-integrations/claude"); + const response = await handleManagementAPI(new Request(url, { + method: "PUT", + headers: { Host: url.host, "Content-Type": "application/json" }, + body: JSON.stringify({ enabled }), + }), url, config, { + fetchAllModels: async () => [], + claudeAgentConfigDir: claudeDir, + saveConfigPreservingClaudeCode: save, + ensureClaudeIntercept: async () => ({ ok: true, state: { proxyPort: 10200, caCertPath: join(root, "ca.pem"), pickerProxyPort: null } }), + }, "admin-token", undefined, { trustedLoopback: true }); + expect(response).not.toBeNull(); + return { status: response!.status, body: await response!.json() as Record }; +} + +test("turning the Claude integration off prunes only marker-owned definitions (#7968)", async () => { + injectClaudeAgentDefs(config, {}, claudeDir); + expectRegistered(); + const user = seedExistingAgents()["ocx-user.md"]; + const result = await toggleClaude(false); + expect(result).toMatchObject({ status: 200, body: { ok: true, changed: true, desiredEnabled: false } }); + expect(agentFiles()).toEqual({ "ocx-user.md": user }); +}); + +test("a repeated off removes definitions an older version left behind", async () => { + config.claudeCode = { ...config.claudeCode, enabled: false }; + persist(); + // Left behind while disabled by a version that never pruned on disable. + writeFileSync(join(claudeDir, "agents", agentFile), "\nstale roster\n"); + const result = await toggleClaude(false); + expect(result).toMatchObject({ status: 200, body: { ok: true, changed: false } }); + expect(agentFiles()).toEqual({}); +}); + +test("turning the integration on fetches no models and leaves the roster to launch-time sync", async () => { + config.claudeCode = { ...config.claudeCode, enabled: false }; + persist(); + let fetched = 0; + const url = new URL("http://127.0.0.1:10100/api/native-integrations/claude"); + const response = await handleManagementAPI(new Request(url, { + method: "PUT", headers: { Host: url.host, "Content-Type": "application/json" }, body: JSON.stringify({ enabled: true }), + }), url, config, { + fetchAllModels: async () => { fetched++; return []; }, + claudeAgentConfigDir: claudeDir, + saveConfigPreservingClaudeCode: () => { persist(); }, + ensureClaudeIntercept: async () => ({ ok: true, state: { proxyPort: 10200, caCertPath: join(root, "ca.pem"), pickerProxyPort: null } }), + }, "admin-token", undefined, { trustedLoopback: true }); + expect(response?.status).toBe(200); + expect(fetched).toBe(0); +}); + +test("a failed save leaves the shared config on, so a retry neither prunes nor reports success", async () => { + injectClaudeAgentDefs(config, {}, claudeDir); + config.claudeCode = { ...config.claudeCode, enabled: true }; + persist(); + const before = agentFiles(); + const busy = Object.assign(new Error("config busy"), { + code: "CONFIG_MUTATION_LOCK_UNAVAILABLE", cause: { code: "SQLITE_BUSY" }, + }); + const failingSave = () => { throw busy; }; + const refused = await toggleClaude(false, failingSave); + expect(refused).toMatchObject({ status: 409, body: { reason: "config_busy" } }); + expect(config.claudeCode?.enabled).toBe(true); + expect(agentFiles()).toEqual(before); + // The same object again: still a real transition, not the repeated-OFF shortcut. + const retry = await toggleClaude(false); + expect(retry).toMatchObject({ status: 200, body: { changed: true } }); + expect(agentFiles()).toEqual({}); +}); + +test("a save that published before failing keeps the new value, so a later On is a real transition", async () => { + const { ConfigWritePublishedError } = await import("../../src/config/persist-unlocked"); + injectClaudeAgentDefs(config, {}, claudeDir); + config.claudeCode = { ...config.claudeCode, enabled: true }; + persist(); + const publishedThenFailed = () => { persist(); throw new ConfigWritePublishedError(new Error("bookkeeping")); }; + await toggleClaude(false, publishedThenFailed).catch(() => null); + expect(config.claudeCode?.enabled).toBe(false); + expect(JSON.parse(readFileSync(join(root, "config.json"), "utf8")).claudeCode.enabled).toBe(false); + const on = await toggleClaude(true); + expect(on).toMatchObject({ status: 200, body: { changed: true } }); + expect(config.claudeCode?.enabled).toBe(true); +}); From 227811fd70886f78a379ca478be84f2368fa3cb8 Mon Sep 17 00:00:00 2001 From: JUN Date: Fri, 30 Oct 2026 21:57:04 +0900 Subject: [PATCH 28/42] fix(history): bound the integrity check wait (#7997) Treat a request-history integrity check that has not answered within 30 s as an operational failure: dispose the worker, apply the failure cooldown, and release waiting async readers with a bounded error, while index replacement still waits for the worker's actual exit. Closes #7984 --- src/routing/history/integrity.ts | 47 +++++-- structure/dashboard-and-usage.md | 2 +- tests/usage/request-history-integrity.test.ts | 117 ++++++++++++++++++ 3 files changed, 157 insertions(+), 9 deletions(-) diff --git a/src/routing/history/integrity.ts b/src/routing/history/integrity.ts index 29cf099f4be..9f8db625483 100644 --- a/src/routing/history/integrity.ts +++ b/src/routing/history/integrity.ts @@ -11,6 +11,7 @@ import { RebuildWorkerJob } from "./rebuild-job"; export const REQUEST_HISTORY_INTEGRITY_INTERVAL_MS = 5 * 60_000; export const REQUEST_HISTORY_INTEGRITY_FAILURE_COOLDOWN_MS = 30_000; +export const REQUEST_HISTORY_INTEGRITY_CHECK_TIMEOUT_MS = 30_000; type IntegrityJob = Pick; @@ -19,14 +20,15 @@ export interface RequestHistoryIntegrityOptions { now?: () => number; intervalMs?: number; failureCooldownMs?: number; + checkTimeoutMs?: number; + scheduleTimeout?: (callback: () => void, ms: number) => () => void; } export type RequestHistoryIntegrityState = | { state: "ready" } | { state: "corrupt" } | { state: "unavailable"; error: unknown } - // Never rejects, including on cancellation or check failure. Completion - // proves the scanner exited; inspect again to obtain its verdict or error. + // Never rejects. Inspect again for the verdict or operational failure. | { state: "pending"; promise: Promise }; interface Generation { @@ -43,11 +45,13 @@ interface ActiveCheck { cancelled: boolean; cancel: () => void; promise: Promise; + barrier: Promise; } type CheckOutcome = | { kind: "checked"; healthy: boolean } | { kind: "failed"; error: unknown } + | { kind: "timeout"; error: Error } | { kind: "cancelled" }; export class RequestHistoryIntegrityMonitor { @@ -55,14 +59,22 @@ export class RequestHistoryIntegrityMonitor { private readonly now: () => number; private readonly intervalMs: number; private readonly failureCooldownMs: number; + private readonly checkTimeoutMs: number; + private readonly scheduleTimeout: (callback: () => void, ms: number) => () => void; private current: Generation | null = null; private active: ActiveCheck | null = null; + private readonly draining = new Set(); constructor(options: RequestHistoryIntegrityOptions = {}) { this.createJob = options.createJob ?? (() => new RebuildWorkerJob()); this.now = options.now ?? Date.now; this.intervalMs = options.intervalMs ?? REQUEST_HISTORY_INTEGRITY_INTERVAL_MS; this.failureCooldownMs = options.failureCooldownMs ?? REQUEST_HISTORY_INTEGRITY_FAILURE_COOLDOWN_MS; + this.checkTimeoutMs = options.checkTimeoutMs ?? REQUEST_HISTORY_INTEGRITY_CHECK_TIMEOUT_MS; + this.scheduleTimeout = options.scheduleTimeout ?? ((callback, ms) => { + const timer = setTimeout(callback, ms); + return () => clearTimeout(timer); + }); } inspect(handle: object, path: string): RequestHistoryIntegrityState { @@ -72,6 +84,8 @@ export class RequestHistoryIntegrityMonitor { } const generation = this.current; if (generation.corrupt) return { state: "corrupt" }; + const old = [...this.draining].filter(check => check.generation !== generation); + if (old.length) return { state: "pending", promise: Promise.all(old.map(check => check.barrier)).then(() => {}) }; if (this.active) { if (this.active.generation === generation && !this.active.cancelled && generation.verified) { return { state: "ready" }; @@ -97,7 +111,7 @@ export class RequestHistoryIntegrityMonitor { * use inspect instead. The caller drains scans before rebuilding/swapping. */ acceptRebuilt(handle: object, path: string): void { - if (this.active) throw new Error("history integrity worker must exit before accepting a rebuilt index"); + if (this.draining.size) throw new Error("history integrity worker must exit before accepting a rebuilt index"); this.current = { handle, path, verified: true, corrupt: false, nextCheckAt: this.now() + this.intervalMs, failure: null, @@ -113,7 +127,7 @@ export class RequestHistoryIntegrityMonitor { quiesce(): Promise | null { this.current = null; this.active?.cancel(); - return this.active?.promise ?? null; + return this.draining.size ? Promise.all([...this.draining].map(check => check.barrier)).then(() => {}) : null; } reset(): void { @@ -130,14 +144,17 @@ export class RequestHistoryIntegrityMonitor { } let cancel!: () => void; const cancelled = new Promise(resolve => { cancel = () => resolve({ kind: "cancelled" }); }); + let release!: () => void; const check: ActiveCheck = { generation, cancelled: false, cancel: () => { check.cancelled = true; cancel(); }, - promise: Promise.resolve(), + promise: new Promise(resolve => { release = resolve; }), + barrier: Promise.resolve(), }; this.active = check; - check.promise = this.run(check, job, cancelled); + this.draining.add(check); + check.barrier = this.run(check, job, cancelled, release); return check; } @@ -146,7 +163,10 @@ export class RequestHistoryIntegrityMonitor { generation.nextCheckAt = this.now() + this.failureCooldownMs; } - private async run(check: ActiveCheck, job: IntegrityJob, cancelled: Promise): Promise { + private async run(check: ActiveCheck, job: IntegrityJob, cancelled: Promise, release: () => void): Promise { + let expire!: (outcome: CheckOutcome) => void; + const timeout = new Promise(resolve => { expire = resolve; }); + const clear = this.scheduleTimeout(() => expire({ kind: "timeout", error: new Error("request-history integrity check timed out") }), this.checkTimeoutMs); const observed = Promise.resolve().then(async (): Promise => { if (check.cancelled) return { kind: "cancelled" }; try { @@ -155,7 +175,16 @@ export class RequestHistoryIntegrityMonitor { return { kind: "failed", error }; } }); - const outcome = await Promise.race([observed, cancelled]); + const outcome = await Promise.race([observed, cancelled, timeout]); + clear(); + if (outcome.kind === "timeout") { + if (this.current === check.generation && !check.cancelled && !check.generation.corrupt) { + this.recordFailure(check.generation, outcome.error); + } + if (this.active === check) this.active = null; + void job.dispose().catch(() => undefined); + release(); + } await new Promise(resolve => { // afterExit may return at its disposal timeout, before native I/O lets // the worker exit. Only its callback releases the file replacement gate. @@ -173,5 +202,7 @@ export class RequestHistoryIntegrityMonitor { } } if (this.active === check) this.active = null; + this.draining.delete(check); + release(); } } diff --git a/structure/dashboard-and-usage.md b/structure/dashboard-and-usage.md index 17bc382a091..9e9b8b49b10 100644 --- a/structure/dashboard-and-usage.md +++ b/structure/dashboard-and-usage.md @@ -279,7 +279,7 @@ inheriting its bounds, so deleting a ledger row removes it from the grouping. `usageLedgerMaxBytes` is unset by default; when set, an append that crosses it publishes the newest whole rows byte for byte through the shared atomic writer, refuses the rename unless the source is the exact revision that was copied, and then discards the Logs ring, the retained aggregates and the request-history index so no surface serves rows the ledger no longer has. That index (`src/routing/history/`) rebuilds a ledger of at least `REQUEST_HISTORY_WORKER_REBUILD_MIN_BYTES` (1 MiB) on a worker thread (#6990): `rebuild-worker.ts`, driven by `rebuild-job.ts`, ingests the ledger snapshot the refresh recorded into a private `.rebuild--` file through the same `ingest.ts` the inline path uses, switches it to WAL and closes it; the serving thread then closes the old index and renames the build over it. Async readers wait for the build; the routing-time sync entry never does and throws `RequestHistoryIndexBusyError`, which health scoring reads as unknown evidence. Health scoring (`src/routing/health.ts`) counts physical sends only: a request answered locally or refused before any send is no evidence for any target, and a row's terminal status and close reason score its last physical send, so a 200 stream cut mid-flight after a failover counts against the target that dropped it (`tests/server/health-scoring.test.ts`). A close discards the build, rejects waiting readers with `RequestHistoryIndexClosedError`, and removes the build files only after the worker thread exits. For 30 s after a worker build fails (other than on a replaced ledger), async readers and manual rebuilds rebuild inline instead, while routing-time reads keep reporting busy. -`src/routing/history/integrity.ts` validates an existing persisted index on first use with a read-only worker `PRAGMA quick_check` (#7966); serving refreshes retain schema/source/offset checks without a synchronous integrity scan. Async readers await initial validation; synchronous routing reports busy/unknown meanwhile. A successful fresh inline reconstruction is immediately usable, and a private worker build is integrity-checked before publication. After validation or reconstruction, the next refresh at least five minutes later starts one background check; verified reads remain available during it, and idle indexes start no timer. Operational check failures leave the index intact and delay retry for 30 s; an unverified index remains unavailable. A corrupt verdict or `SQLITE_CORRUPT`/`SQLITE_NOTADB` invalidates the live handle and enters the existing rebuild path; async history and analytics reads retry once after recovery. Close/rebuild discards stale verdicts and drains scanners through actual thread exit before index replacement. Neither validation nor recovery rewrites canonical `usage.jsonl`. `tests/usage/request-history-integrity.test.ts` covers scan scheduling, corruption recovery and lifecycle isolation. +`src/routing/history/integrity.ts` validates an existing persisted index on first use with a read-only worker `PRAGMA quick_check` (#7966); serving refreshes retain schema/source/offset checks without a synchronous integrity scan. Async readers await initial validation; synchronous routing reports busy/unknown meanwhile. A successful fresh inline reconstruction is immediately usable, and a private worker build is integrity-checked before publication. After validation or reconstruction, the next refresh at least five minutes later starts one background check; verified reads remain available during it, and idle indexes start no timer. A check that has not answered in 30 s is stopped and reported as an operational failure; waiting readers are released without waiting for the worker's actual exit. Operational check failures leave the index intact and delay retry for 30 s; an unverified index remains unavailable. A corrupt verdict or `SQLITE_CORRUPT`/`SQLITE_NOTADB` invalidates the live handle and enters the existing rebuild path; async history and analytics reads retry once after recovery. Close/rebuild discards stale verdicts and drains all scanners through actual thread exit before index replacement, including timed-out scanners. Neither validation nor recovery rewrites canonical `usage.jsonl`. `tests/usage/request-history-integrity.test.ts` covers scan scheduling, corruption recovery and lifecycle isolation. Ledger revisions in `src/usage/log.ts` capture device/inode identifiers from bigint descriptor stats as exact decimal strings; history metadata, ingestion, cooperative scans and retention compare them without narrowing to Numbers, preserving distinct NTFS identifiers above 2^53. An older index containing a rounded identifier fails the identity check and rebuilds once. `src/usage/summary.ts` turns that file into the `/api/usage` shape — totals, daily zero-filled grid, model and provider breakdowns, and `measured / reported / unreported / unsupported / estimated` counts. Each scope also aggregates end-to-end output throughput (#6309): measured output tokens and wall-clock `durationMs` are summed over attempts reporting both, exposing `throughputTokensPerSec` — a token-sum-over-duration-sum, never a mean of per-request rates. `throughputSamples` counts qualifying attempts (legacy rows without attempts contribute one); nonpositive or nonfinite tokens/timing are excluded. The Usage summary and Models/Providers tables display the rate and sample basis, or unavailable when none qualify. All three throughput displays use the same finite-number guard as their tooltip; malformed or nonfinite rates are unavailable rather than formatted. diff --git a/tests/usage/request-history-integrity.test.ts b/tests/usage/request-history-integrity.test.ts index ac424842752..20f143bfc63 100644 --- a/tests/usage/request-history-integrity.test.ts +++ b/tests/usage/request-history-integrity.test.ts @@ -138,6 +138,123 @@ function healthQueryIndex(handle: Database): string { } describe("request-history integrity lifecycle (#7966)", () => { + test("a silent scanner times out for readers but retains the replacement barrier", async () => { + let clock = 0; + let fire: (() => void) | undefined; + const exits: Array<() => void> = []; + let jobs = 0; + let disposals = 0; + const monitor = new RequestHistoryIntegrityMonitor({ + now: () => clock, + failureCooldownMs: 100, + checkTimeoutMs: 50, + scheduleTimeout: (callback: () => void, ms: number) => { + expect(ms).toBe(50); + fire = callback; + return () => { fire = undefined; }; + }, + createJob: () => { + jobs++; + return { + check: () => new Promise(() => {}), + dispose: async () => { disposals++; }, + afterExit: async (cleanup: () => void) => { exits.push(cleanup); }, + }; + }, + }); + const handle = {}; + const first = monitor.inspect(handle, "index.sqlite"); + if (first.state !== "pending") throw new Error("initial check must be pending"); + expect(fire).toBeDefined(); + clock = 50; + fire!(); + await first.promise; + expect(disposals).toBe(1); + expect(monitor.inspect(handle, "index.sqlite").state).toBe("unavailable"); + expect(jobs).toBe(1); + clock = 150; + expect(monitor.inspect(handle, "index.sqlite").state).toBe("pending"); + expect(jobs).toBe(2); + const barrier = monitor.quiesce(); + expect(barrier).not.toBeNull(); + let released = false; + void barrier!.then(() => { released = true; }); + await Promise.resolve(); + expect(released).toBe(false); + exits[0]!(); + await Promise.resolve(); + expect(released).toBe(false); + // Let cancellation schedule the second worker's exit callback. + await Promise.resolve(); + exits[1]!(); + await barrier; + expect(released).toBe(true); + }, 1000); + + test("a verdict arriving after the timeout never changes the generation's state", async () => { + for (const lateVerdict of [true, false]) { + let clock = 0; + const fires: Array<() => void> = []; + const answers: Array<(healthy: boolean) => void> = []; + const exits: Array<() => void> = []; + const monitor = new RequestHistoryIntegrityMonitor({ + now: () => clock, + failureCooldownMs: 100, + checkTimeoutMs: 50, + scheduleTimeout: callback => { fires.push(callback); return () => {}; }, + createJob: () => ({ + check: () => new Promise(resolve => { answers.push(resolve); }), + dispose: async () => {}, + afterExit: async (cleanup: () => void) => { exits.push(cleanup); }, + }), + }); + const handle = {}; + const first = monitor.inspect(handle, "index.sqlite"); + if (first.state !== "pending") throw new Error("initial check must be pending"); + await Promise.resolve(); + fires[0]!(); + await first.promise; + expect(monitor.inspect(handle, "index.sqlite").state).toBe("unavailable"); + // The timed-out worker answers late, then exits. + answers[0]!(lateVerdict); + exits[0]!(); + for (let turn = 0; turn < 5; turn++) await Promise.resolve(); + expect(monitor.inspect(handle, "index.sqlite").state).toBe("unavailable"); + // After the cooldown a fresh check decides, and only its verdict counts. + clock = 150; + const fresh = monitor.inspect(handle, "index.sqlite"); + if (fresh.state !== "pending") throw new Error("fresh check must be pending"); + await Promise.resolve(); + answers[1]!(true); + for (let turn = 0; turn < 5 && !exits[1]; turn++) await Promise.resolve(); + exits[1]!(); + await fresh.promise; + expect(monitor.inspect(handle, "index.sqlite").state).toBe("ready"); + } + }, 1000); + + test("shared async readers leave a silent initial integrity check with a bounded error", async () => { + await existingIndex(); + let fire!: () => void; + let started!: () => void; + let exit!: () => void; + const scheduled = new Promise(resolve => { started = resolve; }); + setRequestHistoryIntegrityOptionsForTests({ + scheduleTimeout: callback => { fire = callback; started(); return () => {}; }, + createJob: () => ({ + check: () => new Promise(() => {}), + dispose: async () => {}, + afterExit: async cleanup => { exit = cleanup; }, + }), + }); + const first = openRequestHistoryIndex(); + const second = openRequestHistoryIndex(); + const errors = Promise.all([first.catch(error => error), second.catch(error => error)]); + await scheduled; + fire(); + expect((await errors).every(error => error instanceof RequestHistoryIndexBusyError)).toBe(true); + exit(); + }, 5000); test("real worker checks leave repeated page, row, status and routing reads free of full serving-thread scans", async () => { await existingIndex(); const query = spyOn(Database.prototype, "query"); From 9922f8159a9e5327fcc4ead930c309faa57755ec Mon Sep 17 00:00:00 2001 From: JUN Date: Fri, 30 Oct 2026 22:04:32 +0900 Subject: [PATCH 29/42] test(history): corrupt the health index through a cold connection (#8000) The routing-health case read the index before corrupting it and relied on PRAGMA shrink_memory to evict the cached pages. macOS's system SQLite kept them, so the read after corruption was served from cache (macOS 2/4 of lane=all run 41056253802, after #7995). Reopen the index after the pre-check so the corruption lands on a connection that has not cached it, like the other corruption cases. Co-authored-by: JUN <243035832+lidge-jun@users.noreply.github.com> --- tests/usage/request-history-integrity.test.ts | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/tests/usage/request-history-integrity.test.ts b/tests/usage/request-history-integrity.test.ts index 20f143bfc63..4716919edb5 100644 --- a/tests/usage/request-history-integrity.test.ts +++ b/tests/usage/request-history-integrity.test.ts @@ -540,6 +540,11 @@ describe("request-history integrity lifecycle (#7966)", () => { const input = { provider: "a", model: "m1", now }; expect(healthEvidenceForCandidate(input).sampleCount).toBe(1); clearHealthHistoryCacheForTests(); + // The read above left this index's pages in the connection's page cache, and PRAGMA + // shrink_memory does not evict them on every SQLite build (macOS links the system library). + // Corrupt through a fresh connection, as the other corruption cases do. + closeRequestHistoryIndex(); + await openRequestHistoryIndex(); const handle = requestHistoryDb(); const canonical = readFileSync(usageLogPath()); corruptModelIndex(handle, path, healthQueryIndex(handle)); From cb483626c8f6af0c0120642179d3336e42b68151 Mon Sep 17 00:00:00 2001 From: Epinephrine Date: Fri, 30 Oct 2026 13:10:16 +0000 Subject: [PATCH 30/42] fix(codex): await recovery ACL checks in async parent admission (#7931) * fix(codex): await recovery ACL checks in async parent admission Hold an exact generation across fresh Windows recovery inspection and revalidate pinned identity after awaits. Migrate asynchronous parent callers with a root-bound synchronous commit scope and keep stale-result checks on both admission success and denial. Preserve synchronous compatibility exports and document native validation and remaining synchronous publication boundaries. Follow-up for #7872 item 2; not a completion claim. * test(cli): align async recovery consumers and assert awaited preflight Fix stale mocks, spies and source oracles behind the 11 failures in run 40797643817, plus five reproduced failures in related consumers. Preserve refusal, ordering, timeout and handover assertions. Add three held-promise preflight ordering regressions. Tests: 280 passed across all seven affected test files; typecheck, structure, privacy and layout/size gates passed. No production source changes. * test: keep async parent registration mergeable Relocate the existing async parent test mapping in the registry and independent golden table without changing either mapping or the four-shard execution plan. Preserve the layout size cap. * fix(cli): revalidate ocx v2 state once the awaited admission enters Recovery admission can now wait on ACL checks. Read the native feature state and the OpenCodex config inside the admitted callback, recheck the hybrid-pin guard there, and save that fresh config, so a change made meanwhile is neither overruled nor overwritten. Also restores the executable mode of src/cli/index.ts. * fix(cli): save ocx v2 config changes inside the admitted callback The admission is released before the awaiting command resumes, so a save made after the callback could overwrite a competing write with the read copy. --------- Co-authored-by: JUN <243035832+lidge-jun@users.noreply.github.com> --- scripts/test-layout/layout.json | 2 +- src/cli/account-main.ts | 5 +- src/cli/codex-shim-autorestore.ts | 18 +- src/cli/dispatch.ts | 12 +- src/cli/doctor.ts | 10 +- src/cli/index.ts | 4 +- src/cli/init.ts | 4 +- src/cli/restart-scope.ts | 4 +- src/cli/root.ts | 12 +- src/cli/v2.ts | 88 ++++-- src/client/machine-api.ts | 4 +- src/codex/app-server-restart-service.ts | 4 +- src/codex/cli-update-job-store-windows.ts | 126 ++++++-- src/codex/cli-update-job-worker.ts | 14 +- src/codex/cli-update-recovery.ts | 62 +++- src/codex/cli-update-selection-gate.ts | 78 ++++- src/codex/desktop-app-restart.ts | 12 +- src/codex/desktop-app/handoff.ts | 2 +- src/codex/runtime.ts | 46 ++- src/codex/shim.ts | 24 +- src/remote-control/workspace-codex-runtime.ts | 4 +- .../management/agent-settings-routes.ts | 24 +- structure/cli-management.md | 2 +- structure/codex-cli-update.md | 26 +- structure/config.md | 2 +- structure/remote-workspace.md | 2 +- structure/runtime.md | 2 +- ...cli-capability-operation-workflows.test.ts | 2 +- tests/cli/cli-head.test.ts | 32 +- tests/cli/cli-help-recovery.test.ts | 2 +- tests/cli/cli-ready.test.ts | 21 +- tests/cli/cli-sync-expect-link.test.ts | 4 +- tests/cli/cli-version-skew.test.ts | 2 +- .../codex-cli-update-parent-async.test.ts | 297 ++++++++++++++++++ .../codex-cli-update-windows-pending.test.ts | 33 +- .../multi-agent-keep-native-v1.test.ts | 88 +++++- tests/fixtures/parent-stop-runner.ts | 2 +- tests/fixtures/test-layout-expected.json | 1 + 38 files changed, 931 insertions(+), 146 deletions(-) create mode 100644 tests/codex-integration/codex-cli-update-parent-async.test.ts diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index ef5c8df7fe2..2442a618c9c 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -604,7 +604,7 @@ "codex-cli-installation-targets.test.ts": "codex-integration", "codex-cli-update-launcher-policy.test.ts": "codex-integration", "codex-cli-update-integrity.test.ts": "codex-integration", - "codex-cli-update-plan.test.ts": "codex-integration", + "codex-cli-update-plan.test.ts": "codex-integration", "codex-cli-update-parent-async.test.ts": "codex-integration", "codex-cli-update-zero-effect.test.ts": "codex-integration", "codex-cli-windows-installation-files.test.ts": "codex-integration", "codex-forward-access-programs-writer.test.ts": "codex-integration", diff --git a/src/cli/account-main.ts b/src/cli/account-main.ts index 08978824f10..70b2dd2cc57 100644 --- a/src/cli/account-main.ts +++ b/src/cli/account-main.ts @@ -3,6 +3,7 @@ import { redactSecretArgs } from "./secret-args"; import { codexExecInvocation } from "../codex/exec-invocation"; import { validatedCodexExecInvocationAsync, type CodexExecBoundaryAsyncDeps } from "../codex/exec-boundary"; import { resolveAndPersistCodexRuntime, type ResolveCodexRuntimeDeps } from "../codex/runtime"; +import { withCodexCliSelectionParentWriteAsync } from "../codex/cli-update-selection-gate"; import type { ResolveDeps, SpawnInvocation } from "../lib/win-exec"; import { apiError, @@ -62,7 +63,7 @@ export async function nativeMainCodexLoginValidatedInvocation( deps: NativeMainCodexLoginValidatedDeps = {}, ): Promise { const env = deps.env ?? process.env; - const command = resolveAndPersistCodexRuntime({ + const command = (await withCodexCliSelectionParentWriteAsync(() => resolveAndPersistCodexRuntime({ env, platform, existsSync: deps.existsSync, @@ -70,7 +71,7 @@ export async function nativeMainCodexLoginValidatedInvocation( configDir: deps.configDir, readFileSync: deps.readFileSync, discoverAlternatives: false, - }).runtime.command || "codex"; + }), { configDir: deps.configDir, signal: deps.signal })).runtime.command || "codex"; return validatedCodexExecInvocationAsync(command, ["login"], { env, platform, diff --git a/src/cli/codex-shim-autorestore.ts b/src/cli/codex-shim-autorestore.ts index 626bff50da9..8ea964ac3c7 100644 --- a/src/cli/codex-shim-autorestore.ts +++ b/src/cli/codex-shim-autorestore.ts @@ -1,4 +1,4 @@ -import { autoRestoreCodexShim } from "../codex/shim"; +import { autoRestoreCodexShim, autoRestoreCodexShimAsync } from "../codex/shim"; import { codexShimAutoRestoreEnabled, readConfigDiagnostics } from "../config"; export interface CodexShimAutoRestoreCliDeps { @@ -77,3 +77,19 @@ export function runCodexShimAutoRestore(deps: CodexShimAutoRestoreCliDeps = DEFA ); } } + +export async function maybeAutoRestoreCodexShimAsync(command: string | undefined, args: string[]): Promise { + if (!skipsCodexShimAutoRestore(command, args)) await runCodexShimAutoRestoreAsync(); +} + +export async function runCodexShimAutoRestoreAsync(): Promise { + try { + const result = await autoRestoreCodexShimAsync({ + enabled: () => codexShimAutoRestoreEnabled(DEFAULT_DEPS.readConfig().config, DEFAULT_DEPS.env), + }); + if (result.status === "restored") DEFAULT_DEPS.warn(`⚠️ ${result.message} (automatic repair after Codex update)`); + else if ((result.status === "deferred" || result.status === "ineligible") && result.message) DEFAULT_DEPS.warn(`⚠️ ${result.message}`); + } catch (error) { + DEFAULT_DEPS.warn(`⚠️ Codex shim auto-restore failed; continuing without it: ${error instanceof Error ? error.message : String(error)}. Run 'ocx codex-shim install' after the Codex update finishes.`); + } +} diff --git a/src/cli/dispatch.ts b/src/cli/dispatch.ts index 6f9809c9916..d4d3f794166 100644 --- a/src/cli/dispatch.ts +++ b/src/cli/dispatch.ts @@ -435,8 +435,8 @@ const commandRunners: Record = { if (expectation) { // The root preflight skipped the shim repair for a pinned sync; the identity held, so // run it now, as an unpinned sync would have. - const { runCodexShimAutoRestore } = await import("./codex-shim-autorestore"); - runCodexShimAutoRestore(); + const { runCodexShimAutoRestoreAsync } = await import("./codex-shim-autorestore"); + await runCodexShimAutoRestoreAsync(); } // The connection the sync ran under, read inside its lock; the read above may be stale. const link = result.connection.transport === "link"; @@ -731,10 +731,10 @@ const commandRunners: Record = { return Number(process.exitCode ?? 0); }, "codex-shim": async deps => { - const { codexShimStatus, diagnoseCodexShim, installCodexShim, uninstallCodexShim } = await import("../codex/shim"); + const { codexShimStatus, diagnoseCodexShim, installCodexShimAsync, uninstallCodexShimAsync } = await import("../codex/shim"); switch (deps.args[1]) { case "install": { - const r = installCodexShim(); + const r = await installCodexShimAsync(); const { healthy, runnable, active, summary } = diagnoseCodexShim(); const success = !r.refused && (runnable ?? healthy); const { collectCodexShimReadinessWarnings } = await import("./codex-shim-readiness"); @@ -764,7 +764,7 @@ const commandRunners: Record = { } case "uninstall": case "remove": { - const r = uninstallCodexShim(); + const r = await uninstallCodexShimAsync(); console.log(r.removed ? `✅ ${r.message}` : `⚠️ ${r.message}`); break; } @@ -858,7 +858,7 @@ const commandRunners: Record = { }, ready: async deps => { // Fail-closed impossible-state guard: readyArgs is populated by the - // preparse block in src/cli/root.ts before maybeAutoRestoreCodexShim, so + // preparse block in src/cli/root.ts before maybeAutoRestoreCodexShimAsync, so // reaching here without it means dispatch diverged. Refuse with code 64 // and perform NO I/O (no discovery/probe). process.exit is `never`, // narrowing below. diff --git a/src/cli/doctor.ts b/src/cli/doctor.ts index 2753a7362f6..ff33b7ab209 100644 --- a/src/cli/doctor.ts +++ b/src/cli/doctor.ts @@ -70,8 +70,8 @@ import { effortClampAppliesToRuntime, liveRemovedEfforts, loadLastEffortClamp, - persistCodexRuntime, - resolveAndPersistCodexRuntime, + persistCodexRuntimeAsync, + resolveAndPersistCodexRuntimeAsync, resolveCodexRuntime, } from "../codex/runtime"; import { CODEX_REAUTH_ACTION, collectOAuthHealthEntriesForCli, MASKED_ACCOUNT_FALLBACK, type OAuthHealthEntry } from "../oauth/health"; @@ -1266,8 +1266,8 @@ export async function runDoctor(args: string[] = []): Promise { const resolved = resolveCodexRuntime(); if (!resolved.newerAvailable) { console.log("No newer Codex runtime found; keeping current selection."); - const current = resolveAndPersistCodexRuntime(); - console.log(`Selected: ${displayCodexRuntimePath(current.runtime.command)} (${current.runtime.version ?? "unknown"})`); + const current = await resolveAndPersistCodexRuntimeAsync(); + console.log(`Selected: ${displayCodexRuntimePath(current?.runtime.command ?? "codex")} (${current?.runtime.version ?? "unknown"})`); return; } if (resolved.runtime.source === "environment") { @@ -1276,7 +1276,7 @@ export async function runDoctor(args: string[] = []): Promise { console.log("Then run ocx sync."); return; } - persistCodexRuntime({ + await persistCodexRuntimeAsync({ command: resolved.newerAvailable.command, version: resolved.newerAvailable.version, source: "configured", diff --git a/src/cli/index.ts b/src/cli/index.ts index eff70dd94bb..52cc993a5fe 100755 --- a/src/cli/index.ts +++ b/src/cli/index.ts @@ -1674,8 +1674,8 @@ async function handleUninstall() { }); try { - const { uninstallCodexShim } = await import("../codex/shim"); - const r = uninstallCodexShim(); + const { uninstallCodexShimAsync } = await import("../codex/shim"); + const r = await uninstallCodexShimAsync(); console.log(r.removed ? "✅ Codex autostart shim removed" : "- Codex autostart shim removed: not installed"); } catch (err) { failures.push("Codex autostart shim removed"); diff --git a/src/cli/init.ts b/src/cli/init.ts index 83a69cff06f..b8866eccb3e 100644 --- a/src/cli/init.ts +++ b/src/cli/init.ts @@ -257,9 +257,9 @@ export async function runInit(): Promise { prompt.throwIfCancelled(); if (shimAnswer.trim().toLowerCase() !== "n") { try { - const { installCodexShim } = await import("../codex/shim"); + const { installCodexShimAsync } = await import("../codex/shim"); prompt.throwIfCancelled(); - const result = installCodexShim(); + const result = await installCodexShimAsync(); console.log(result.installed ? `✅ ${result.message}` : `⚠️ ${result.message}`); } catch (err) { if (err instanceof InitCancelledError) throw err; diff --git a/src/cli/restart-scope.ts b/src/cli/restart-scope.ts index 58b5264606e..1574520c192 100644 --- a/src/cli/restart-scope.ts +++ b/src/cli/restart-scope.ts @@ -104,9 +104,9 @@ export async function handleDesktopAppRestart( log: Pick, io: DesktopAppRestartIo = {}, ): Promise { - const { restartCodexDesktopApp } = await import("../codex/desktop-app-restart"); + const { restartCodexDesktopAppAsync } = await import("../codex/desktop-app-restart"); const { startDesktopRestartHandoff } = await import("../codex/desktop-app/handoff"); - const result = restartCodexDesktopApp({ + const result = await restartCodexDesktopAppAsync({ ...io, // The CLI is the one caller whose exit is exactly the signal the helper waits for, // so it is the one caller allowed to hand off. The management service is not (it diff --git a/src/cli/root.ts b/src/cli/root.ts index f34a1cc22ab..2668b1fb8a8 100644 --- a/src/cli/root.ts +++ b/src/cli/root.ts @@ -13,7 +13,7 @@ import { parseReadyArgs, type ReadyArgs } from "./ready"; import { parseResolveArgs, type ResolveArgs } from "./resolve"; import { parseStopApproval } from "./stop-approval"; import { parseSyncExpectation, SYNC_EXPECTATION_USAGE_EXIT } from "./sync-expectation"; -import { maybeAutoRestoreCodexShim } from "./codex-shim-autorestore"; +import { maybeAutoRestoreCodexShimAsync } from "./codex-shim-autorestore"; import { findCommand } from "./registry"; import { printUnknownCommand } from "./help-recovery"; import { noteCredentialArgv, redactSecretArgs } from "./secret-args"; @@ -64,7 +64,7 @@ export function parseCliHead(argv: string[]): CliHead { }; } // P1: pre-parse `ocx ready` and reject invalid arguments with exit 64 BEFORE - // maybeAutoRestoreCodexShim (or any discovery/probe/filesystem-capable global + // maybeAutoRestoreCodexShimAsync (or any discovery/probe/filesystem-capable global // preflight) runs. `ready --help` / `help ready` already exited above, so this // only sees ready args without a help flag. Valid args are stashed so the // switch dispatch can call runReady without a second parse. @@ -74,7 +74,7 @@ export function parseCliHead(argv: string[]): CliHead { return { kind: "ready", command, args, readyArgs: parsed.args }; } // Same ordering contract as `ready`: `ocx resolve` rejects any argument with exit - // 64 BEFORE maybeAutoRestoreCodexShim (or any other preflight with side effects) runs. + // 64 BEFORE maybeAutoRestoreCodexShimAsync (or any other preflight with side effects) runs. if (command === "resolve") { const parsed = parseResolveArgs(args.slice(1)); if (!parsed.ok) return { kind: "resolve", command, args, resolveArgs: undefined }; @@ -115,7 +115,7 @@ export async function runCli(argv: string[]): Promise { console.error(" Default wait timeout is 45 seconds."); process.exit(64); } - maybeAutoRestoreCodexShim(head.command, head.args); + await maybeAutoRestoreCodexShimAsync(head.command, head.args); return head; } case "resolve": { @@ -128,7 +128,7 @@ export async function runCli(argv: string[]): Promise { console.error(" and the identity-checked liveness verdict."); process.exit(64); } - maybeAutoRestoreCodexShim(head.command, head.args); + await maybeAutoRestoreCodexShimAsync(head.command, head.args); return head; } case "command": @@ -154,7 +154,7 @@ export async function runCli(argv: string[]): Promise { } } await maybeNoticeVersionSkew(head.command, head.args); - maybeAutoRestoreCodexShim(head.command, head.args); + await maybeAutoRestoreCodexShimAsync(head.command, head.args); return head; } } diff --git a/src/cli/v2.ts b/src/cli/v2.ts index 116cc1be0dc..734cccd3023 100644 --- a/src/cli/v2.ts +++ b/src/cli/v2.ts @@ -21,6 +21,7 @@ import type { ResolveDeps, SpawnInvocation } from "../lib/win-exec"; import { validatedCodexExecInvocation } from "../codex/exec-boundary"; import { deleteConfigTopLevelKey, loadConfig, saveConfig } from "../config"; import { resolveAndPersistCodexRuntime, type ResolveCodexRuntimeDeps } from "../codex/runtime"; +import { withCodexCliSelectionParentWriteAsync } from "../codex/cli-update-selection-gate"; export interface V2CliDeps { execFile?: (file: string, args: string[], options?: SpawnInvocation["options"] & { env?: NodeJS.ProcessEnv }) => void; @@ -79,6 +80,11 @@ export function codexFeaturesInvocation( }); } +export async function codexFeaturesInvocationAsync(action: "enable" | "disable", feature = "multi_agent_v2", + platform: NodeJS.Platform = process.platform, deps: CodexFeaturesInvocationDeps = {}): Promise { + return withCodexCliSelectionParentWriteAsync(() => codexFeaturesInvocation(action, feature, platform, deps), { configDir: deps.configDir }); +} + /** * Run `codex features ` synchronously - the management API * fallback when no deps toggle is injected. Shares the invocation builder and @@ -195,45 +201,77 @@ export async function cmdV2(args: string[], deps: V2CliDeps = {}, findPort?: () return emit(true, []); } if (parsed.verb === "mode-hint") { - const result = setMultiAgentModeHintText(parsed.value); + const result = parsed.value === null ? setMultiAgentModeHintText(null) + : await withCodexCliSelectionParentWriteAsync(() => setMultiAgentModeHintText(parsed.value)); if (!result.ok) return emit(false, ["Unable to write the native mode hint. Check Codex support and ownership, then read ocx v2 status."]); changed = result.changed; return emit(true, [result.changed ? "multi_agent_mode_hint_text saved — applies to new sessions." : "multi_agent_mode_hint_text already set — nothing to do."]); } if (parsed.verb === "threads") { - const enabled = (deps.isEnabled ?? isMultiAgentV2Enabled)(); - const result = transitionMultiAgentV2(enabled, (next, env, validate) => runCodexFeatures(next ? "enable" : "disable", deps, env, validate), { threadLimit: parsed.value }); + // Admission can wait on recovery ACL checks; the feature state is read once admitted, in + // the same synchronous turn as the transition, so a change made meanwhile is not undone. + let enabled = false; + const result = await withCodexCliSelectionParentWriteAsync(() => { + enabled = (deps.isEnabled ?? isMultiAgentV2Enabled)(); + return transitionMultiAgentV2(enabled, (next, env, validate) => runCodexFeatures(next ? "enable" : "disable", deps, env, validate), { threadLimit: parsed.value }); + }); if (!result.ok) return emit(false, ["Unable to update the native thread limit. Read ocx v2 status before retrying."]); changed = result.changed; return emit(true, [changed ? `max_threads = ${parsed.value} (${enabled ? "v2" : "v1"}) — applies to new sessions.` : `max_threads already ${parsed.value} — nothing to do.`]); } - const config = loadConfig(); + // Every awaited admission below re-reads, decides and saves the config inside its + // synchronous callback, so a setting changed while admission waited is neither + // overruled by a stale decision nor overwritten by a stale save. + let config = loadConfig(); let successLines: string[]; let unchangedKeep: boolean | undefined; if (parsed.verb === "mode") { - changed = (config.multiAgentMode ?? "default") !== parsed.value; + const persistMode = (): boolean => { + const before = config.multiAgentMode ?? "default"; + if (parsed.value === "default") deleteConfigTopLevelKey(config, "multiAgentMode"); + else config.multiAgentMode = parsed.value; + saveConfig(config); + return before !== parsed.value; + }; if (parsed.value !== "default") { - const target = parsed.value === "v2" && config.keepNativeChatGptOnV1 !== true; - const transition = transitionMultiAgentV2(target, (next, env, validate) => runCodexFeatures(next ? "enable" : "disable", deps, env, validate)); + let modeChanged = false; + const transition = await withCodexCliSelectionParentWriteAsync(() => { + config = loadConfig(); + const target = parsed.value === "v2" && config.keepNativeChatGptOnV1 !== true; + const result = transitionMultiAgentV2(target, (next, env, validate) => runCodexFeatures(next ? "enable" : "disable", deps, env, validate)); + if (result.ok) modeChanged = persistMode(); + return result; + }); if (!transition.ok) { changed = false; return emit(false, ["Native mode transition failed. Read ocx v2 status before retrying."]); } - changed ||= transition.changed; + changed = modeChanged || transition.changed; + } else { + changed = persistMode(); } - if (parsed.value === "default") deleteConfigTopLevelKey(config, "multiAgentMode"); - else config.multiAgentMode = parsed.value; - saveConfig(config); successLines = [multiAgentModeLine(parsed.value, config.keepNativeChatGptOnV1 === true), "Applies to NEW sessions; running sessions keep their pinned multi-agent version."]; } else if (parsed.verb === "keep-native-v1") { - unchangedKeep = (config.keepNativeChatGptOnV1 === true) === parsed.value; - changed = !unchangedKeep; + const persistKeep = (): void => { + unchangedKeep = (config.keepNativeChatGptOnV1 === true) === parsed.value; + if (parsed.value) config.keepNativeChatGptOnV1 = true; + else deleteConfigTopLevelKey(config, "keepNativeChatGptOnV1"); + saveConfig(config); + }; if (parsed.value && requiresGlobalV2Disabled(config.multiAgentMode, true)) { - const transition = transitionMultiAgentV2(false, (next, env, validate) => runCodexFeatures(next ? "enable" : "disable", deps, env, validate)); + const transition = await withCodexCliSelectionParentWriteAsync(() => { + config = loadConfig(); + // Revalidated once admitted: mode may have left v2 while admission waited. + const result = requiresGlobalV2Disabled(config.multiAgentMode, true) + ? transitionMultiAgentV2(false, (next, env, validate) => runCodexFeatures(next ? "enable" : "disable", deps, env, validate)) + : { ok: true as const, changed: false }; + if (result.ok) persistKeep(); + return result; + }); if (!transition.ok) { changed = false; return emit(false, ["Native hybrid-mode transition failed. Read ocx v2 status before retrying."]); } - changed ||= transition.changed; + changed = !unchangedKeep || transition.changed; + } else { + persistKeep(); + changed = !unchangedKeep; } - if (parsed.value) config.keepNativeChatGptOnV1 = true; - else deleteConfigTopLevelKey(config, "keepNativeChatGptOnV1"); - saveConfig(config); successLines = [parsed.value ? "keep_native_chatgpt_on_v1: ON — ChatGPT-native rows stay v1 when mode is v2 (new sessions)." : "keep_native_chatgpt_on_v1: OFF — ChatGPT-native rows follow v1/base/v2 (new sessions)."]; @@ -242,7 +280,19 @@ export async function cmdV2(args: string[], deps: V2CliDeps = {}, findPort?: () if (want && requiresGlobalV2Disabled(config.multiAgentMode, config.keepNativeChatGptOnV1 === true)) { return emit(false, ["v2 on: incompatible with keep-native-v1 while mode is v2 — Codex's global multi_agent_v2 overrides the native v1 catalog pin. Run 'ocx v2 keep-native-v1 off' first."]); } - const transition = transitionMultiAgentV2(want, (next, env, validate) => runCodexFeatures(next ? "enable" : "disable", deps, env, validate)); + let incompatible = false; + const transition = await withCodexCliSelectionParentWriteAsync(() => { + // Revalidated once admitted: keep-native-v1 or mode v2 may have been set meanwhile. + const current = loadConfig(); + if (want && requiresGlobalV2Disabled(current.multiAgentMode, current.keepNativeChatGptOnV1 === true)) { + incompatible = true; + return { ok: true as const, changed: false }; + } + return transitionMultiAgentV2(want, (next, env, validate) => runCodexFeatures(next ? "enable" : "disable", deps, env, validate)); + }); + if (incompatible) { + return emit(false, ["v2 on: incompatible with keep-native-v1 while mode is v2 — Codex's global multi_agent_v2 overrides the native v1 catalog pin. Run 'ocx v2 keep-native-v1 off' first."]); + } if (!transition.ok) return emit(false, ["Native feature transition failed. Read ocx v2 status before retrying."]); changed = transition.changed; if (!changed) return emit(true, [`multi_agent_v2 already ${want ? "ON" : "OFF"} — nothing to do.`]); diff --git a/src/client/machine-api.ts b/src/client/machine-api.ts index 5aeb11efc56..12997a8e6f5 100644 --- a/src/client/machine-api.ts +++ b/src/client/machine-api.ts @@ -1,5 +1,5 @@ import { journalOwner } from "../codex/journal"; -import { diagnoseCodexShim, installCodexShim, uninstallCodexShim } from "../codex/shim"; +import { diagnoseCodexShim, installCodexShimAsync, uninstallCodexShimAsync } from "../codex/shim"; import { readManagementJsonBody } from "../server/management/body"; import type { OcxClientConnectionConfig } from "../types"; import { disconnectClient, syncConnectedClient } from "./connect"; @@ -130,7 +130,7 @@ export async function handleMachineApi( return Response.json({ error: "action must be install, repair, or uninstall" }, { status: 400 }); } try { - const result = input.action === "uninstall" ? uninstallCodexShim() : installCodexShim(); + const result = input.action === "uninstall" ? await uninstallCodexShimAsync() : await installCodexShimAsync(); return Response.json({ success: true, action: input.action, result, shim: diagnoseCodexShim() }); } catch (error) { return Response.json({ success: false, error: error instanceof Error ? error.message : "shim action failed" }, { status: 409 }); diff --git a/src/codex/app-server-restart-service.ts b/src/codex/app-server-restart-service.ts index a84715351b1..da1776e12f1 100644 --- a/src/codex/app-server-restart-service.ts +++ b/src/codex/app-server-restart-service.ts @@ -37,8 +37,8 @@ import { getServerListenPort } from "../server/lifecycle"; import type { DesktopPickerState } from "./desktop-app/picker-freshness"; async function defaultRestartDesktopApp(): Promise { - const { restartCodexDesktopApp } = await import("./desktop-app-restart"); - const outcome = restartCodexDesktopApp({ allowHandoff: false }); + const { restartCodexDesktopAppAsync } = await import("./desktop-app-restart"); + const outcome = await restartCodexDesktopAppAsync({ allowHandoff: false }); return { attempted: outcome.attempted, stopped: outcome.stopped, diff --git a/src/codex/cli-update-job-store-windows.ts b/src/codex/cli-update-job-store-windows.ts index 9444bff57f2..d7b6e31b333 100644 --- a/src/codex/cli-update-job-store-windows.ts +++ b/src/codex/cli-update-job-store-windows.ts @@ -686,83 +686,155 @@ export interface WindowsCliRecoveryIo { withProofHandle(name: string, run: (handle: bigint) => T): T; write(name: string, pendingName: string, body: string, adoptPending?: boolean): void; } +export interface WindowsCliRecoveryAsyncIo { + readonly dir: string; + names(): string[]; + read(name: string): Promise; + write(name: string, pendingName: string, body: string, adoptPending?: boolean): Promise; +} +type RecoveryStoreStep = { kind: "acl"; entries: readonly WindowsAclEntry[] } + | { kind: "harden"; path: string; directory: boolean }; +type RecoveryStoreSteps = Generator; +function runRecoveryStoreSteps(steps: RecoveryStoreSteps): T { + let step = steps.next(); + while (!step.done) { + try { + const effect = step.value; + if (effect.kind === "acl") { + if (!verifyWindowsEntriesAcl(aclTargets(effect.entries), ACL_VERIFY_TIMEOUT_MS)) throw unsafe(); + } else if (effect.directory) hardenSecretDir(toNamespacedPath(effect.path), { required: true }); + else hardenSecretPath(toNamespacedPath(effect.path), { required: true }); + } catch (error) { step = steps.throw(error); continue; } + step = steps.next(); + } + return step.value; +} +async function runRecoveryStoreStepsAsync(steps: RecoveryStoreSteps): Promise { + let step = steps.next(); + while (!step.done) { + try { + const effect = step.value; + if (effect.kind === "acl") await verifyAcl(effect.entries); + else if (effect.directory) await hardenSecretDirAsync(toNamespacedPath(effect.path), { required: true }); + else await hardenSecretPathAsync(toNamespacedPath(effect.path), { required: true }); + } catch (error) { step = steps.throw(error); continue; } + step = steps.next(); + } + return step.value; +} +function openRecoveryContext(config: string): Ctx { + if (process.platform !== "win32" || process.arch !== "x64") throw unsafe(); + assertNotRealHomeUnderTest(config); + const kernel = recoveryFfi.dlopen("kernel32.dll", KERNEL_SYMBOLS); + try { return { ffi: recoveryFfi, kernel, nt: recoveryFfi.dlopen("ntdll.dll", NT_SYMBOLS), pins: new PinSet() }; } + catch (error) { kernel.close(); throw error; } +} /** The shared recovery protocol uses the ledger's native ancestry, no-reparse, * identity, private-DACL and rename primitives at synchronous gate boundaries. * Restart publication may write an existing pinned store without creating it. */ export function withWindowsCliRecoveryStore(config: string, create: boolean, run: (io: WindowsCliRecoveryIo) => T, publish = false): T { - if (process.platform !== "win32" || process.arch !== "x64") throw unsafe(); - assertNotRealHomeUnderTest(config); - const kernel = recoveryFfi.dlopen("kernel32.dll", KERNEL_SYMBOLS); - const nt = recoveryFfi.dlopen("ntdll.dll", NT_SYMBOLS); - const ctx: Ctx = { ffi: recoveryFfi, kernel, nt, pins: new PinSet() }; - const verify = (entries: readonly WindowsAclEntry[]) => { - if (!verifyWindowsEntriesAcl(aclTargets(entries), ACL_VERIFY_TIMEOUT_MS)) throw unsafe(); - }; + const ctx = openRecoveryContext(config); try { + const io = runRecoveryStoreSteps(recoveryStoreSteps(ctx, config, create, publish)); + return run({ dir: io.dir, names: io.names, + read: name => runRecoveryStoreSteps(io.read(name)), + withProofHandle: (name, use) => runRecoveryStoreSteps(io.withProofHandle(name, use)), + write: (name, pending, body, adopt) => runRecoveryStoreSteps(io.write(name, pending, body, adopt)), + }); + } finally { ctx.pins.closeAll(ctx.kernel); ctx.nt.close(); ctx.kernel.close(); } +} + +/** Parent admission awaits the same native protocol; pins outlive every ACL read. */ +export async function withWindowsCliRecoveryStoreAsync(config: string, create: boolean, + run: (io: WindowsCliRecoveryAsyncIo) => Promise, publish = false): Promise { + const ctx = openRecoveryContext(config); + try { + const io = await runRecoveryStoreStepsAsync(recoveryStoreSteps(ctx, config, create, publish)); + return await run({ dir: io.dir, names: io.names, + read: name => runRecoveryStoreStepsAsync(io.read(name)), + write: (name, pending, body, adopt) => runRecoveryStoreStepsAsync(io.write(name, pending, body, adopt)), + }); + } finally { ctx.pins.closeAll(ctx.kernel); ctx.nt.close(); ctx.kernel.close(); } +} + +function* recoveryStoreSteps(ctx: Ctx, config: string, create: boolean, publish: boolean) { + const kernel = ctx.kernel; const chain = pinChain(ctx, config, ACCESS_DIR_PIN); - verify(chain.map((entry, i) => ({ path: entry.path, directory: true, - policy: i >= chain.length - 2 ? "write-private" : "trusted-ancestry" }))); + const checkChain = () => { + for (const entry of chain) if (!sameIdentity(entry.inspection, inspect(ctx, entry.handle, true))) throw unsafe(); + }; + yield { kind: "acl", entries: chain.map((entry, i) => ({ path: entry.path, directory: true, + policy: i >= chain.length - 2 ? "write-private" : "trusted-ancestry" })) } satisfies RecoveryStoreStep; + checkChain(); const configPin = chain.at(-1); if (!configPin) throw unsafe(); const dir = join(config, "codex-cli-update-recovery"); try { - if (create) { mkdirSync(dir); hardenSecretDir(toNamespacedPath(dir), { required: true }); } + if (create) { mkdirSync(dir); yield { kind: "harden", path: dir, directory: true } satisfies RecoveryStoreStep; } } catch (error) { if ((error as NodeJS.ErrnoException).code !== "EEXIST") throw error; } const directoryPin = relOpen(ctx, configPin.handle, "codex-cli-update-recovery", true, create || publish ? ACCESS_DIR_PIN : ACCESS_READ, SHARE_DIR); - inspect(ctx, directoryPin, true); assertCanonicalPath(ctx, directoryPin, dir); - verify([{ path: dir, directory: true, policy: "private" }]); + const directoryIdentity = inspect(ctx, directoryPin, true); assertCanonicalPath(ctx, directoryPin, dir); + const checkDirectory = () => { + checkChain(); + if (!sameIdentity(directoryIdentity, inspect(ctx, directoryPin, true))) throw unsafe(); + }; + yield { kind: "acl", entries: [{ path: dir, directory: true, policy: "private" }] } satisfies RecoveryStoreStep; + checkDirectory(); if (create) flushDir(ctx, configPin.handle); - const leaf = (name: string, access = ACCESS_READ, sharing = SHARE_LEAF_VERIFY) => { + const leaf = function* (name: string, access = ACCESS_READ, sharing = SHARE_LEAF_VERIFY): RecoveryStoreSteps<{ handle: bigint; stat: StoreInspection }> { if (!/^[0-9a-f-]{36}(?:\.(?:terminal|proof))?(?:\.pending|\.json|\.terminal|\.proof)$/.test(name)) throw unsafe(); const handle = relOpen(ctx, directoryPin, name, false, access, sharing); const stat = inspect(ctx, handle, false); if (stat.links !== 1 || stat.size > 1024) throw unsafe(); - verify([{ path: join(dir, name), directory: false, policy: "private" }]); + yield { kind: "acl", entries: [{ path: join(dir, name), directory: false, policy: "private" }] }; + checkDirectory(); + const current = inspect(ctx, handle, false); + if (!sameIdentity(stat, current) || current.links !== 1 || current.size !== stat.size) throw unsafe(); return { handle, stat }; }; - return run({ dir, + return { dir, names: () => readdirSync(dir), - read(name) { - const file = leaf(name); + *read(name: string): RecoveryStoreSteps { + const file = yield* leaf(name); try { return readAll(ctx, file.handle, file.stat.size).toString("utf8"); } finally { ctx.pins.release(file.handle, kernel); } }, - withProofHandle(name, use) { + *withProofHandle(name: string, use: (handle: bigint) => T): RecoveryStoreSteps { if (!/^[0-9a-f-]{36}\.proof$/.test(name)) throw unsafe(); - const file = leaf(name, 0x40100080, 7); + const file = yield* leaf(name, 0x40100080, 7); try { return use(file.handle); } finally { ctx.pins.release(file.handle, kernel); } }, - write(name, pendingName, body, adoptPending = false) { + *write(name: string, pendingName: string, body: string, adoptPending = false): RecoveryStoreSteps { if (!(create || publish) || Buffer.byteLength(body) > 1024 || !/^[0-9a-f-]{36}(?:\.terminal|\.proof)?$/.test(name.replace(/\.json$/, "")) || pendingName !== name.replace(/\.json$/, "") + ".pending") throw unsafe(); // Recovery preserves the only durable child identity until atomic rename. const temp = join(dir, pendingName), destination = join(dir, name); if (!adoptPending) { try { - const old = leaf(pendingName, ACCESS_LEAF, SHARE_LEAF); + const old = yield* leaf(pendingName, ACCESS_LEAF, SHARE_LEAF); markDelete(ctx, old.handle); ctx.pins.release(old.handle, kernel); } catch (error) { if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; } const fd = openSync(temp, "wx", 0o600); try { writeFileSync(fd, body); fsyncSync(fd); } finally { closeSync(fd); } } - hardenSecretPath(toNamespacedPath(temp), { required: true }); - const file = leaf(pendingName, adoptPending ? ACCESS_LEAF | 0x40000000 : ACCESS_LEAF, SHARE_LEAF); + yield { kind: "harden", path: temp, directory: false }; + checkDirectory(); + const file = yield* leaf(pendingName, adoptPending ? ACCESS_LEAF | 0x40000000 : ACCESS_LEAF, SHARE_LEAF); try { if (!readAll(ctx, file.handle, file.stat.size).equals(Buffer.from(body))) throw unsafe(); if (adoptPending) flushDir(ctx, file.handle); renamePinned(ctx, file.handle, destination); // The held adopted pin denies other writers; this reader must share // its write/flush access as well as DELETE through identity readback. - const published = leaf(name, ACCESS_READ, adoptPending ? SHARE_LEAF_VERIFY | 0x2 : SHARE_LEAF_VERIFY); + const published = yield* leaf(name, ACCESS_READ, adoptPending ? SHARE_LEAF_VERIFY | 0x2 : SHARE_LEAF_VERIFY); try { if (!sameIdentity(file.stat, published.stat)) throw unsafe(); } finally { ctx.pins.release(published.handle, kernel); } flushDir(ctx, directoryPin); } finally { ctx.pins.release(file.handle, kernel); forgetEphemeralSecretPath(toNamespacedPath(temp)); } }, - }); - } finally { ctx.pins.closeAll(kernel); nt.close(); kernel.close(); } + }; } export interface WindowsJobStoreIo { diff --git a/src/codex/cli-update-job-worker.ts b/src/codex/cli-update-job-worker.ts index 0dc08e8b2b0..c61aa759523 100644 --- a/src/codex/cli-update-job-worker.ts +++ b/src/codex/cli-update-job-worker.ts @@ -34,9 +34,7 @@ import { trustedNodeLauncherContext } from "../cli/launcher-context"; import { spawnWorker } from "../lib/worker-embed"; import { recoverClosedCodexCliUpdateWorker } from "./cli-update-worker-recovery"; import { - CODEX_CLI_UPDATE_GATE_FREE, - CODEX_CLI_UPDATE_GATE_PARENT_WRITE, - CODEX_CLI_UPDATE_GATE_WORKER_COMMIT, + codexCliSelectionGateState, cancelCodexCliUpdateFence, codexCliUpdateSelectionGate, createCodexCliUpdateFence, @@ -108,13 +106,8 @@ export interface CodexCliUpdateWorkerDiagnostics { } export function codexCliUpdateWorkerDiagnostics(): CodexCliUpdateWorkerDiagnostics { - const raw = Atomics.load(new Int32Array(codexCliUpdateSelectionGate()), 0); - const gate = raw === CODEX_CLI_UPDATE_GATE_FREE ? "free" - : raw === CODEX_CLI_UPDATE_GATE_PARENT_WRITE ? "parent_write" - : raw >= CODEX_CLI_UPDATE_GATE_WORKER_COMMIT ? "worker_commit" - : "unrecognized"; return { - gate, + gate: codexCliSelectionGateState(), liveJobs: [...liveJobs.entries()].map(([jobId, record]) => ({ jobId, settled: record.settled(), workerClosed: record.workerClosed(), })), @@ -188,8 +181,7 @@ function trackUnjoined(worker: Worker, alreadyClosed: boolean): void { * recovery releases admission only after matching terminal proof. */ function warnIfCommitGateHeld(): void { - if (Atomics.load(new Int32Array(codexCliUpdateSelectionGate()), 0) - < CODEX_CLI_UPDATE_GATE_WORKER_COMMIT) return; + if (codexCliSelectionGateState() !== "worker_commit") return; console.warn( "[opencodex] a Codex CLI update worker closed while the shared selection gate still reads" + " worker_commit; admission awaits owner-scoped terminal evidence — inspect" diff --git a/src/codex/cli-update-recovery.ts b/src/codex/cli-update-recovery.ts index c605a31cb66..5f240a90682 100644 --- a/src/codex/cli-update-recovery.ts +++ b/src/codex/cli-update-recovery.ts @@ -3,7 +3,7 @@ import { closeSync, constants, fstatSync, fsyncSync, ftruncateSync, lstatSync, m import { dirname, join } from "node:path"; import { getConfigDir } from "../config/paths"; import { assertTrustedCodexCliAncestry } from "./cli-update-paths"; -import { withWindowsCliRecoveryStore } from "./cli-update-job-store-windows"; +import { withWindowsCliRecoveryStore, withWindowsCliRecoveryStoreAsync } from "./cli-update-job-store-windows"; import { windowsCodexCliChildGone } from "./windows-cli-recovery"; export interface CodexCliUpdateNativeChildIdentity { jobName: string; creationTime: string } @@ -169,19 +169,64 @@ export function codexCliUpdateRecoveryBlocksAdmission(configDir = getConfigDir() } catch { return true; } // Existing evidence that disappears/is unreadable stays fenced. } +/** Every attempt observes fresh records; this never caches an ACL/admission verdict. */ +export async function codexCliUpdateRecoveryBlocksAdmissionAsync(configDir = getConfigDir()): Promise { + if (process.platform !== "win32") return codexCliUpdateRecoveryBlocksAdmission(configDir); + try { lstatSync(join(configDir, "codex-cli-update-recovery")); } + catch (error) { return (error as NodeJS.ErrnoException).code !== "ENOENT"; } + try { + const canonical = realpathSync.native(configDir); + return await withWindowsCliRecoveryStoreAsync(canonical, false, async io => { + const steps = inspectRecoveryFilesSteps(io.names(), true); + let step = steps.next(); + while (!step.done) { + const effect = step.value; + if (effect.kind === "publish") { + await io.write(effect.record.token + ".json", effect.record.token + ".pending", JSON.stringify(effect.record), true); + step = steps.next(undefined); + } else { + const record = JSON.parse(await io.read(effect.token + (effect.suffix ?? ".json"))) as RecoveryRecord; + if (!validRecord(record, effect.token)) throw new Error("invalid_recovery_owner"); + step = steps.next(record); + } + } + return step.value; + }, true); + } catch { return true; } +} + +type RecoveryInspectionStep = { kind: "read"; token: string; suffix?: string } + | { kind: "publish"; record: RecoveryRecord }; +type RecoveryInspectionSteps = Generator; +function* childGoneSteps(record: RecoveryRecord): RecoveryInspectionSteps { + if (process.platform !== "win32") return childGone(record); + if (record.pgid === null || !record.windows?.jobName || !record.windows.creationTime) return false; + const proof = (yield { kind: "read", token: record.token, suffix: ".proof" })!; + return childGone(record, () => proof); +} function inspectRecoveryFiles(files: string[], readRecord: (token: string, suffix?: string) => RecoveryRecord, publishChild?: (record: RecoveryRecord) => void): boolean { + const steps = inspectRecoveryFilesSteps(files, !!publishChild); + let step = steps.next(); + while (!step.done) { + const effect = step.value; + if (effect.kind === "publish") { publishChild!(effect.record); step = steps.next(undefined); } + else step = steps.next(readRecord(effect.token, effect.suffix)); + } + return step.value; +} +function* inspectRecoveryFilesSteps(files: string[], publishChild: boolean): RecoveryInspectionSteps { if (new Set(files.map(file => file.slice(0, 36))).size > 256) return true; const terminal = new Set(); const active = new Map(); for (const file of files.filter(file => file.endsWith(".terminal"))) { - const record = readRecord(file.slice(0, -9), ".terminal"); + const record = (yield { kind: "read", token: file.slice(0, -9), suffix: ".terminal" })!; if (record.phase !== "terminal") return true; terminal.add(record.token); } for (const file of files.filter(file => file.endsWith(".json"))) { if (terminal.has(file.slice(0, -5))) continue; - const record = readRecord(file.slice(0, -5)); + const record = (yield { kind: "read", token: file.slice(0, -5) })!; if (record.phase === "terminal") { terminal.add(record.token); continue; } active.set(record.token, record); } @@ -202,7 +247,7 @@ function inspectRecoveryFiles(files: string[], readRecord: (token: string, suffi if (terminal.has(token)) continue; // Published terminal authority shadows late writes. const suffix = file.slice(token.length); if (suffix !== ".pending" && suffix !== ".terminal.pending") return true; - const record = readRecord(token, suffix); + const record = (yield { kind: "read", token, suffix })!; const previous = active.get(token) ?? reconciled.get(token); if (process.platform === "win32" && suffix === ".pending" && record.phase === "child") { const armed = active.get(token); @@ -211,8 +256,8 @@ function inspectRecoveryFiles(files: string[], readRecord: (token: string, suffi || armed.windows.creationTime !== null || armed.windows.jobName !== record.windows?.jobName || record.pgid === null || record.pgid > 0xffffffff || !record.windows?.creationTime || BigInt(record.windows.creationTime) > 0xffffffffffffffffn - || !processGone(record.parentPid) || !childGone(record, token => readRecord(token, ".proof"))) return true; - publishChild(record); // Existing protected flush/ACL/rename path; any failure fences. + || !processGone(record.parentPid) || !(yield* childGoneSteps(record)) || !processGone(record.parentPid)) return true; + yield { kind: "publish", record }; // Existing protected flush/ACL/rename path; any failure fences. reconciled.set(token, record); continue; } // A complete cleanup write can survive a crash before rename. Recognize @@ -221,7 +266,8 @@ function inspectRecoveryFiles(files: string[], readRecord: (token: string, suffi if (record.phase !== "terminal" || record.pgid !== null || !processGone(record.parentPid) || (previous && previous.parentPid !== record.parentPid)) return true; if (previous?.phase === "child" - && !childGone(previous, token => readRecord(token, ".proof"))) return true; + && !(yield* childGoneSteps(previous))) return true; + if (!processGone(record.parentPid)) return true; reconciled.set(token, record); } for (const record of active.values()) { @@ -229,7 +275,7 @@ function inspectRecoveryFiles(files: string[], readRecord: (token: string, suffi // Re-probe on every admission attempt, including after a restart. A live // or inaccessible group stays fenced until the whole group proves absent. if (record.phase === "child" && record.pgid !== null - && processGone(record.parentPid) && childGone(record, token => readRecord(token, ".proof"))) continue; + && processGone(record.parentPid) && (yield* childGoneSteps(record)) && processGone(record.parentPid)) continue; return true; } return false; diff --git a/src/codex/cli-update-selection-gate.ts b/src/codex/cli-update-selection-gate.ts index 63e799a4d4f..a17c933728a 100644 --- a/src/codex/cli-update-selection-gate.ts +++ b/src/codex/cli-update-selection-gate.ts @@ -30,8 +30,10 @@ * stays the documented residual it always was. */ -import { codexCliUpdateRecoveryBlocksAdmission } from "./cli-update-recovery"; +import { codexCliUpdateRecoveryBlocksAdmission, codexCliUpdateRecoveryBlocksAdmissionAsync } from "./cli-update-recovery"; import type { CodexCliUpdateNativeChildIdentity, CodexCliUpdateNativeRecoveryProof } from "./cli-update-recovery"; +import { getConfigDir } from "../config/paths"; +import { resolve } from "node:path"; export const CODEX_CLI_UPDATE_FENCE_PREPARING = 0; export const CODEX_CLI_UPDATE_FENCE_CANCELLED = 1; @@ -53,6 +55,10 @@ export class CodexCliSelectionMutationBusyError extends Error { } let selectionGate: SharedArrayBuffer | null = null; +let asyncParentOwner: { buffer: SharedArrayBuffer; owner: number } | null = null; +// This scope exists only during one synchronous commit turn. It is never kept +// across an await, so unrelated parent requests cannot borrow its admission. +let parentCommitScope: { buffer: SharedArrayBuffer; owner: number; configDir: string } | null = null; /** One process-wide holder and monotonic generation counter shared across realms. */ export function codexCliUpdateSelectionGate(): SharedArrayBuffer { @@ -60,9 +66,19 @@ export function codexCliUpdateSelectionGate(): SharedArrayBuffer { return selectionGate; } +/** Diagnostic classification does not grant or release gate ownership. */ +export function codexCliSelectionGateState(): "free" | "parent_write" | "worker_commit" | "unrecognized" { + const buffer = codexCliUpdateSelectionGate(), owner = Atomics.load(new Int32Array(buffer), 0); + if (owner === CODEX_CLI_UPDATE_GATE_FREE) return "free"; + if (owner === CODEX_CLI_UPDATE_GATE_PARENT_WRITE + || (asyncParentOwner?.buffer === buffer && asyncParentOwner.owner === owner)) return "parent_write"; + return owner >= CODEX_CLI_UPDATE_GATE_WORKER_COMMIT ? "worker_commit" : "unrecognized"; +} + /** Test seam: drop the cell so each case starts from FREE on a fresh buffer. */ export function resetCodexCliSelectionGatesForTests(): void { selectionGate = null; + asyncParentOwner = null; } /** Per-job fence: decision, current child pgid (0 = unpublished), terminal proof. */ @@ -112,8 +128,13 @@ export interface CodexCliSelectionParentWrite { * paths. entered === false means a commit window (or another writer) is in * flight and the caller must surface busy immediately, never wait. */ -export function beginCodexCliSelectionParentWrite(): CodexCliSelectionParentWrite { - if (codexCliUpdateRecoveryBlocksAdmission()) return Object.freeze({ entered: false, exit() {} }); +export function beginCodexCliSelectionParentWrite(configDir = getConfigDir()): CodexCliSelectionParentWrite { + if (parentCommitScope) { + const entered = resolve(configDir) === parentCommitScope.configDir + && Atomics.load(new Int32Array(parentCommitScope.buffer), 0) === parentCommitScope.owner; + return Object.freeze({ entered, exit() {} }); + } + if (codexCliUpdateRecoveryBlocksAdmission(configDir)) return Object.freeze({ entered: false, exit() {} }); const view = new Int32Array(codexCliUpdateSelectionGate()); if (Atomics.compareExchange(view, 0, CODEX_CLI_UPDATE_GATE_FREE, CODEX_CLI_UPDATE_GATE_PARENT_WRITE) !== CODEX_CLI_UPDATE_GATE_FREE) { @@ -125,11 +146,60 @@ export function beginCodexCliSelectionParentWrite(): CodexCliSelectionParentWrit exit() { if (!held) return; held = false; - Atomics.store(view, 0, CODEX_CLI_UPDATE_GATE_FREE); + Atomics.compareExchange(view, 0, CODEX_CLI_UPDATE_GATE_PARENT_WRITE, CODEX_CLI_UPDATE_GATE_FREE); + }, + }); +} + +type SynchronousParentCommit = () => T & (T extends PromiseLike ? never : unknown); +export interface CodexCliSelectionAsyncParentWrite extends CodexCliSelectionParentWrite { + /** Run existing synchronous writers under this exact fresh admission. Never await inside. */ + commit(run: SynchronousParentCommit): T; +} +interface ParentAdmissionOptions { + configDir?: string; + signal?: AbortSignal; + /** Test seam; production always uses the fresh async recovery protocol. */ + recoveryBlocksAdmission?: (configDir?: string) => Promise; +} +/** Reserve before inspection, keep the generation across awaits, and release only that owner. */ +export async function beginCodexCliSelectionParentWriteAsync(options: ParentAdmissionOptions = {}): Promise { + const refused = Object.freeze({ entered: false, exit() {}, commit(_run: SynchronousParentCommit): T { throw new CodexCliSelectionMutationBusyError(); } }); + if (options.signal?.aborted) return refused; + const configDir = resolve(options.configDir ?? getConfigDir()); + const buffer = codexCliUpdateSelectionGate(), view = new Int32Array(buffer); + const owner = allocateCodexCliUpdateGateOwner(buffer); + if (Atomics.compareExchange(view, 0, CODEX_CLI_UPDATE_GATE_FREE, owner) !== CODEX_CLI_UPDATE_GATE_FREE) return refused; + const parent = { buffer, owner }; + asyncParentOwner = parent; + let held = true; + const exit = () => { + if (!held) return; + held = false; + releaseCodexCliUpdateGateOwner(buffer, owner); + if (asyncParentOwner === parent) asyncParentOwner = null; + }; + try { + if (await (options.recoveryBlocksAdmission ?? codexCliUpdateRecoveryBlocksAdmissionAsync)(configDir) + || options.signal?.aborted || Atomics.load(view, 0) !== owner) { exit(); return refused; } + } catch { exit(); return refused; } + return Object.freeze({ entered: true, exit, + commit(run: SynchronousParentCommit): T { + if (!held || options.signal?.aborted || Atomics.load(view, 0) !== owner) throw new CodexCliSelectionMutationBusyError(); + const previous = parentCommitScope; + parentCommitScope = { buffer, owner, configDir }; + try { return run(); } finally { parentCommitScope = previous; } }, }); } +/** The callback is synchronous: config critical sections retain their original no-await contract. */ +export async function withCodexCliSelectionParentWriteAsync(run: SynchronousParentCommit, options: ParentAdmissionOptions = {}): Promise { + const admission = await beginCodexCliSelectionParentWriteAsync(options); + if (!admission.entered) throw new CodexCliSelectionMutationBusyError(); + try { return admission.commit(run); } finally { admission.exit(); } +} + /** * The worker's installation window. tryEnter covers the uncontended same-turn * case; waitEnter is the bounded retry for a held parent write. commit() is diff --git a/src/codex/desktop-app-restart.ts b/src/codex/desktop-app-restart.ts index 638be90b91e..e6909c7420f 100644 --- a/src/codex/desktop-app-restart.ts +++ b/src/codex/desktop-app-restart.ts @@ -41,7 +41,7 @@ import { darwinDesktopAppAdapter, darwinDefaultExec } from "./desktop-app/darwin import { linuxDesktopAppAdapter, linuxDefaultExec } from "./desktop-app/linux"; import { windowsDesktopAppAdapter, windowsDefaultExec } from "./desktop-app/windows"; import { isTestHomeGuardArmed } from "../lib/test-home-guard"; -import { beginCodexCliSelectionParentWrite } from "./cli-update-selection-gate"; +import { beginCodexCliSelectionParentWrite, withCodexCliSelectionParentWriteAsync, CodexCliSelectionMutationBusyError } from "./cli-update-selection-gate"; export type { DesktopAppExecOptions } from "./desktop-app/types"; @@ -388,3 +388,13 @@ export function restartCodexDesktopApp(io: DesktopAppRestartIo = {}): DesktopApp if (lockHeld && !handedOff) releaseDesktopRestartLock(io.lock); } } + +export async function restartCodexDesktopAppAsync(io: DesktopAppRestartIo = {}): Promise { + try { return await withCodexCliSelectionParentWriteAsync(() => restartCodexDesktopApp(io)); } + catch (error) { + if (error instanceof CodexCliSelectionMutationBusyError) return { + attempted: false, stopped: [], surviving: [], relaunch: "skipped", reason: "selection_mutation_busy", + }; + throw error; + } +} diff --git a/src/codex/desktop-app/handoff.ts b/src/codex/desktop-app/handoff.ts index 3baf6d83b43..121b220d226 100644 --- a/src/codex/desktop-app/handoff.ts +++ b/src/codex/desktop-app/handoff.ts @@ -284,7 +284,7 @@ export async function runDesktopRestartHandoff( try { const restart = io.restart ? io.restart(false) - : (await import("../desktop-app-restart")).restartCodexDesktopApp({ + : await (await import("../desktop-app-restart")).restartCodexDesktopAppAsync({ allowHandoff: false, lock: io.lock, }); diff --git a/src/codex/runtime.ts b/src/codex/runtime.ts index d0e9d1538ef..8de8bed5e17 100644 --- a/src/codex/runtime.ts +++ b/src/codex/runtime.ts @@ -13,6 +13,7 @@ import { import { macCodexAppBundleFor, macCodexAppCommands } from "./mac-app-bundle"; import { beginCodexCliSelectionParentWrite, + withCodexCliSelectionParentWriteAsync, CodexCliSelectionMutationBusyError, } from "./cli-update-selection-gate"; import { resolveCodexHomeDir } from "./home"; @@ -406,7 +407,7 @@ export function persistCodexRuntime( // window, so a held gate is an immediate busy rather than a wait. Checked // BEFORE the mkdir and the resolve-cache invalidation so a busy refusal // changes nothing at all. - const selectionWrite = beginCodexCliSelectionParentWrite(); + const selectionWrite = beginCodexCliSelectionParentWrite(deps.configDir); if (!selectionWrite.entered) throw new CodexCliSelectionMutationBusyError(); try { const configDir = deps.configDir ?? getConfigDir(); @@ -438,7 +439,7 @@ export function clearPersistedCodexRuntime(deps: ResolveCodexRuntimeDeps = {}): // Same gate as persistCodexRuntime, checked before the cache invalidation // and outside the swallowing catch: a busy refusal must stay an error, not // become a silent "already removed". - const selectionWrite = beginCodexCliSelectionParentWrite(); + const selectionWrite = beginCodexCliSelectionParentWrite(deps.configDir); if (!selectionWrite.entered) throw new CodexCliSelectionMutationBusyError(); try { const configDir = deps.configDir ?? getConfigDir(); @@ -461,6 +462,14 @@ export function clearPersistedCodexRuntime(deps: ResolveCodexRuntimeDeps = {}): } } +export async function persistCodexRuntimeAsync(runtime: ResolvedCodexRuntime, deps: ResolveCodexRuntimeDeps = {}, + origin: CodexRuntimePinOrigin = "pinned"): Promise { + await withCodexCliSelectionParentWriteAsync(() => persistCodexRuntime(runtime, deps, origin), deps); +} +export async function clearPersistedCodexRuntimeAsync(deps: ResolveCodexRuntimeDeps = {}): Promise { + await withCodexCliSelectionParentWriteAsync(() => clearPersistedCodexRuntime(deps), deps); +} + type VersionProbeResult = { ok: true; version: string | null; macAppBundle?: boolean } | { ok: false; reason: string }; /** Checks that need no exec; a non-null answer settles the candidate without running `--version`. */ @@ -1415,7 +1424,38 @@ export async function resolveAndPersistCodexRuntimeAsync( if (deps.signal?.aborted || !isCurrent() || observation.epoch === null || observation.epoch !== resolveCacheEpoch || observation.inputs !== codexRuntimeSelectionIdentity(deps)) return null; if (deps.signal?.aborted) return null; - return cloneAndDeepFreeze(persistResolvedCodexRuntime(observation.result, deps)); + if (resolvedRuntimeMutation(observation.result, deps) === null) return cloneAndDeepFreeze(observation.result); + try { + return await withCodexCliSelectionParentWriteAsync(() => { + // Admission can await Windows ACL subprocesses. Re-prove all selection + // inputs after that await, in the same turn as the persisted mutation. + if (deps.signal?.aborted || !isCurrent() || observation.epoch !== resolveCacheEpoch + || observation.inputs !== codexRuntimeSelectionIdentity(deps)) return null; + return cloneAndDeepFreeze(persistResolvedCodexRuntime(observation.result, deps)); + }, deps); + } catch (error) { + if (deps.signal?.aborted) return null; + if (error instanceof CodexCliSelectionMutationBusyError) { + // A recovery denial also followed an await. Never return the obsolete + // selection as a usable result merely because no commit callback ran. + if (!isCurrent() || observation.epoch !== resolveCacheEpoch + || observation.inputs !== codexRuntimeSelectionIdentity(deps)) return null; + if (observation.result.runtime.source !== "fallback") { + return cloneAndDeepFreeze({ ...observation.result, persistError: error.message }); + } + } + throw error; + } +} + +function resolvedRuntimeMutation(result: ResolveCodexRuntimeResult, deps: ResolveCodexRuntimeDeps): "persist" | "clear" | null { + const persisted = loadPersistedCodexRuntime(deps); + const unchanged = persisted !== null && persisted.command === result.runtime.command + && persisted.source === result.runtime.source && (persisted.selectedVersion ?? null) === (result.runtime.version ?? null); + if (result.runtime.command && result.runtime.source !== "fallback" && !unchanged) return "persist"; + if (result.runtime.source === "fallback" && persisted?.command && result.failures.some(failure => + failure.command.trim() === persisted.command.trim() && failure.reason === PATH_MISSING_REASON)) return "clear"; + return null; } function persistResolvedCodexRuntime( diff --git a/src/codex/shim.ts b/src/codex/shim.ts index 4bac2ed2cae..99b525e2beb 100644 --- a/src/codex/shim.ts +++ b/src/codex/shim.ts @@ -1,5 +1,5 @@ import { randomUUID } from "node:crypto"; -import { beginCodexCliSelectionParentWrite } from "./cli-update-selection-gate"; +import { beginCodexCliSelectionParentWrite, withCodexCliSelectionParentWriteAsync, CodexCliSelectionMutationBusyError } from "./cli-update-selection-gate"; import { existsSync, lstatSync, @@ -984,6 +984,28 @@ export function installCodexShim(deps: RuntimePreflightDeps = {}): { installed: finally { gate.exit(); } } +export async function installCodexShimAsync(deps: RuntimePreflightDeps = {}): Promise> { + try { return await withCodexCliSelectionParentWriteAsync(() => installCodexShim(deps)); } + catch (error) { + if (error instanceof CodexCliSelectionMutationBusyError) return { installed: false, refused: true, message: error.message }; + throw error; + } +} +export async function autoRestoreCodexShimAsync(options: Parameters[0]): Promise { + try { return await withCodexCliSelectionParentWriteAsync(() => autoRestoreCodexShim(options)); } + catch (error) { + if (error instanceof CodexCliSelectionMutationBusyError) return { status: "deferred", message: error.message }; + throw error; + } +} +export async function uninstallCodexShimAsync(): Promise> { + try { return await withCodexCliSelectionParentWriteAsync(uninstallCodexShim); } + catch (error) { + if (error instanceof CodexCliSelectionMutationBusyError) return { removed: false, message: error.message }; + throw error; + } +} + export function autoRestoreCodexShim(options: Parameters[0]): CodexShimAutoRestoreResult { if (process.platform !== "win32") return autoRestoreUnixOverlay(options); const gate = beginCodexCliSelectionParentWrite(); diff --git a/src/remote-control/workspace-codex-runtime.ts b/src/remote-control/workspace-codex-runtime.ts index 9962477bc93..df2bd7bb3c9 100644 --- a/src/remote-control/workspace-codex-runtime.ts +++ b/src/remote-control/workspace-codex-runtime.ts @@ -7,7 +7,7 @@ import { type CodexExecBoundaryAsyncDeps, } from "../codex/exec-boundary"; import { - beginCodexCliSelectionParentWrite, + beginCodexCliSelectionParentWriteAsync, CodexCliSelectionMutationBusyError, } from "../codex/cli-update-selection-gate"; import { remoteWorkspaceThreadStartParams } from "./workspace-coordinator"; @@ -371,7 +371,7 @@ export class CodexRemoteWorkspaceRuntimeFactory implements RemoteWorkspaceRuntim // installer termination/readback; a new session must not appear inside that // window. Busy is the same explicit pre-spawn failure path as a spawn // throw, so the bridge and isolation are reclaimed identically. - const selectionWrite = beginCodexCliSelectionParentWrite(); + const selectionWrite = await beginCodexCliSelectionParentWriteAsync(); if (!selectionWrite.entered) { // Both teardown steps are attempted before the refusal leaves: the // ordered runner keeps a rejected bridge stop from skipping isolation diff --git a/src/server/management/agent-settings-routes.ts b/src/server/management/agent-settings-routes.ts index db957a73e72..85a80da579b 100644 --- a/src/server/management/agent-settings-routes.ts +++ b/src/server/management/agent-settings-routes.ts @@ -1,4 +1,6 @@ import { prepareClaudeCodePatch } from "./claude-code-settings-patch"; +import { beginCodexCliSelectionParentWriteAsync, withCodexCliSelectionParentWriteAsync, + CodexCliSelectionMutationBusyError, type CodexCliSelectionAsyncParentWrite } from "../../codex/cli-update-selection-gate"; import { readConfigForAdmittedMutation } from "../../config/persisted-mutation"; import { mirrorDesiredEnabledOntoSnapshot, persistDesktopModeField, persistDesktopProfileField } from "./claude-desktop-config-writers"; import { getConfigDir as configRoot, withConfigDirHardeningSettled as settleConfigRoot } from "../../config/paths"; @@ -205,6 +207,7 @@ export async function injectionModelView(config: OcxConfig, routed: readonly Cat export async function handleAgentSettingsRoutes(ctx: ManagementContext): Promise { try { return await handleAgentSettingsRoutesImpl(ctx); } catch (error) { + if (error instanceof CodexCliSelectionMutationBusyError) return jsonResponse({ error: error.message, retryable: true }, 503); if (!(error instanceof WindowsSecretAclSettleTimeoutError)) throw error; return jsonResponse({ error: "Could not save settings", code: "write_failed", diagnostic: { stage: "ca", cause: "ETIMEDOUT" } }, 500); } @@ -402,10 +405,8 @@ async function handleAgentSettingsRoutesImpl(ctx: ManagementContext): Promise { + if (wantsFlag || wantsMode || wantsThreads || wantsKeepNative || (wantsModeHintText && body.multiAgentModeHintText !== null)) { + selectionWrite = await beginCodexCliSelectionParentWriteAsync({ signal: req.signal }); + if (!selectionWrite.entered) return jsonResponse({ error: "codex_cli_selection_busy", retryable: true }, 503); + } + const commit = () => { const currentPlan = readModePlan(); if (currentPlan instanceof Response) return currentPlan; const { effectiveMode } = currentPlan; @@ -497,9 +503,11 @@ async function handleAgentSettingsRoutesImpl(ctx: ManagementContext): Promise selectionWrite ? selectionWrite.commit(commit) : commit()); if (writeRefusal) return writeRefusal; } finally { + selectionWrite?.exit(); if (heldWriteLock) releaseConfigWriteLock(heldWriteLock); } // Derived from fresh post-write readers (readConfigText is uncached): upstream @@ -558,9 +566,9 @@ async function handleAgentSettingsRoutesImpl(ctx: ManagementContext): Promise runCodexFeaturesCommand(action, DEFAULT_MODE_REQUEST_USER_INPUT_FEATURE_KEY, env, validate, invocation); } catch (error) { return jsonResponse({ error: `default_mode_request_user_input toggle failed: ${error instanceof Error ? error.message : String(error)}` }, 502); diff --git a/structure/cli-management.md b/structure/cli-management.md index da5ae2d54f9..cd1294ef63c 100644 --- a/structure/cli-management.md +++ b/structure/cli-management.md @@ -165,7 +165,7 @@ Remote Link enrollment is shared with the dashboard through `src/server/manageme An explicit `sync --restart-codex` (or app-server-only scope) also reloads after a committed configuration-byte change, even when no catalog/cache was written. Without an explicit restart scope, configuration-only changes do not restart clients. Existing catalog/cache-write restart behavior remains unchanged; success alone is not proof of changed configuration. -`ocx sync --expect-link ` and `--expect-hub-key ` (#7694) pin the sync to one connection. `src/cli/sync-expectation.ts` parses them, first in `src/cli/root.ts` before any preflight (malformed, repeated or doubled flags exit 2), and renders a refusal as exit 5 with a final stderr JSON line `{phase, reason}`. `skipsCodexShimAutoRestore` exempts a pinned sync from the root shim repair, which the runner performs (`runCodexShimAutoRestore`) only after the identity check passed. With an expectation, the `sync` runner in `src/cli/dispatch.ts` ignores its own unlocked state read and always enters `syncConnectedClient`, which checks the expectation (`src/client/sync-expectation.ts`) inside its first lifecycle/config-locked block, under the apply lock, before the token check, any request or any write. A refusal never reaches `syncModelsToCodex`, local proxy discovery or integration refreshes. The runner takes link-vs-hub wording and restart ownership from `result.connection`, the connection the sync ran under; `/api/machine/sync` strips that field from its response. If a lock the check needs is held before the check ran (`link_apply_busy` from `withLinkApply`, because a link connect keeps the apply lock across its network reads, `client_lifecycle_busy`, or `CONFIG_MUTATION_LOCK_UNAVAILABLE` caused by SQLite busy/locked), the pinned sync is refused as `pending_operation`; the same contention after the check, or without an expectation, stays an ordinary failure. A connection that changes after the check fails at the existing compare-and-swap (`client_connection_changed`, exit 1). Coverage: `tests/cli/cli-sync-expect-link.test.ts`. +`ocx sync --expect-link ` and `--expect-hub-key ` (#7694) pin the sync to one connection. `src/cli/sync-expectation.ts` parses them, first in `src/cli/root.ts` before any preflight (malformed, repeated or doubled flags exit 2), and renders a refusal as exit 5 with a final stderr JSON line `{phase, reason}`. `skipsCodexShimAutoRestore` exempts a pinned sync from the root shim repair, which the runner awaits (`runCodexShimAutoRestoreAsync`) only after the identity check passed. With an expectation, the `sync` runner in `src/cli/dispatch.ts` ignores its own unlocked state read and always enters `syncConnectedClient`, which checks the expectation (`src/client/sync-expectation.ts`) inside its first lifecycle/config-locked block, under the apply lock, before the token check, any request or any write. A refusal never reaches `syncModelsToCodex`, local proxy discovery or integration refreshes. The runner takes link-vs-hub wording and restart ownership from `result.connection`, the connection the sync ran under; `/api/machine/sync` strips that field from its response. If a lock the check needs is held before the check ran (`link_apply_busy` from `withLinkApply`, because a link connect keeps the apply lock across its network reads, `client_lifecycle_busy`, or `CONFIG_MUTATION_LOCK_UNAVAILABLE` caused by SQLite busy/locked), the pinned sync is refused as `pending_operation`; the same contention after the check, or without an expectation, stays an ordinary failure. A connection that changes after the check fails at the existing compare-and-swap (`client_connection_changed`, exit 1). Coverage: `tests/cli/cli-sync-expect-link.test.ts`. ## Custom Codex provider approval diff --git a/structure/codex-cli-update.md b/structure/codex-cli-update.md index f84c105d140..310df791db4 100644 --- a/structure/codex-cli-update.md +++ b/structure/codex-cli-update.md @@ -368,7 +368,7 @@ synchronous worker section, signal-based cancellation alone is not equivalent an worker path's only channel. The worker's final window is also serialized against the parent's own selection mutations through one -process-wide gate cell. The gate covers exactly seven parent operations: `persistCodexRuntime` and +process-wide gate cell. The gate covers seven parent mutation boundaries: `persistCodexRuntime` and `clearPersistedCodexRuntime` in `src/codex/runtime.ts` (busy refusal — the `codex_cli_selection_busy` error — before mkdir, cache invalidation or the swallowing unlink catch), the three public shim boundaries (`src/codex/shim-overlay.ts` on POSIX, `src/codex/shim.ts` on Windows; install, auto-restore, uninstall — @@ -378,7 +378,7 @@ lock and journal mutation), the remote-workspace app-server `Bun.spawn` in cleanup a spawn throw runs, so a rejected bridge stop cannot skip isolation removal), and `src/codex/desktop-app-restart.ts` (busy returns the new `selection_mutation_busy` skipped reason before the restart lock, package discovery, any app quit and the handoff, so a queued relaunch — and -its own lock-acquisition failure — cannot slip inside the window). Writers never wait: the commit +its own lock-acquisition failure — cannot slip inside the window). Gate contention refuses immediately: the commit window spans the final process scan through terminal installer evidence and readback. Windows directory observations retain bigint device/inode identities, including the second namespace check after asynchronous ACL/file observations. Windows also @@ -400,6 +400,26 @@ not bound time held. A synchronous parent writer must not park the event loop behind it, so a held gate is an immediate busy. The gate is one cell for the whole process rather than per-configDir because alias, junction and case spellings of the same target must not split the exclusion; a writer on a different configDir can also see busy. +Async parent callers reserve a unique gate owner before checking recovery and await Windows ACL +reads and required recovery publication without blocking the parent on PowerShell. The native sync +and async drivers share the same pinned ancestry, private-DACL, content and publication protocol; +the async driver rechecks held identities after each ACL/hardening await. Recovery verdicts are never +cached. A denied, failed or cancelled check releases only its own generation after that check settles. +The lease's synchronous commit scope admits nested compatibility writers only for the inspected +resolved config root and the still-held owner. It ends before queued work can run. Existing sync +exports and Worker arming retain synchronous checks; a direct sync call outside that scope does not +inherit an earlier async verdict. Ordinary atomic secret-file publication and native feature commands +retain their synchronous leaf contracts; the nonblocking guarantee covers recovery admission. CLI, +machine API, restart and Remote Workspace callers use the async +entry points. Runtime publication rechecks selection inputs, cache epoch, cancellation and caller +currentness after recovery. Codex feature-setting callers acquire admission before their synchronous +transition; management keeps the lease across config-root hardening, then rechecks its mode plan and +commits in the existing no-await callback. Refusal precedes combined setting writes and returns a +retryable `codex_cli_selection_busy` response. Diagnostics classify a held async parent by its exact +buffer and owner, so it cannot appear as a retained Worker. Portable gate/caller regressions and +Windows-native async ACL regressions live in `tests/codex-integration/codex-cli-update-parent-async.test.ts`; +`tests/codex-integration/codex-cli-update-windows-pending.test.ts` covers async pending-child publication +and flush, hardening, rename and readback refusal under the same protocol. The gate's holder word carries a unique generation; release compares that exact owner, so an obsolete observer cannot clear a later window. An installer/verifier with unconfirmed terminal state retains the gate as well as lease/staging. `src/codex/cli-update-worker-recovery.ts` observes worker close: a fence @@ -462,7 +482,7 @@ operator inspection. PID reuse can delay recovery but cannot prove absence. The ledger keeps its existing ambiguous result; fence recovery neither retries installation nor converts it to success. Abandoned installation-prefix leases and staging after process death are not reclaimed by this admission recovery. Old jobs lacking recovery records retain their existing unknown-state projection. -Ordinary config writes, account operations, model requests, existing session interrupt/close and +Unrelated config writes, account operations, model requests, existing session interrupt/close and arbitrary subprocesses are not gated. The gate claims nothing about processes the parent does not own: an external installer, a handoff helper's delayed descendants, or an OS-level snapshot race stays the documented residual it always was. diff --git a/structure/config.md b/structure/config.md index 01d9447e520..d5203ed40cb 100644 --- a/structure/config.md +++ b/structure/config.md @@ -62,7 +62,7 @@ test seam and the mutation types remain re-exported through `src/config.ts`. Suc `src/config/paths.ts` owns `OPENCODEX_HOME` expansion and resolution, the config directory and `config.json` path. A relative home is resolved once per distinct raw environment value, so a later working-directory change cannot silently move the active installation. Config reads in this leaf and `src/config/load-degrade.ts` use NTFS ACLs on Windows and private POSIX modes elsewhere. Optional directory hardening follows directory symlinks/junctions but skips non-directory or unstatable targets before starting chmod or ACL work. Windows reads omit `chmod` before ACL hardening so repeated loads preserve successful object/freshness observations; external metadata changes still invalidate them. Directory flights keep their existing flush/reap lifecycle, and file/required-write failure policy is unchanged. `tests/windows/windows-config-read-acl-memo.test.ts` covers repeated loads, non-directory and directory-link behavior, external invalidation and POSIX permissions. -`withConfigDirHardeningSettled` in `src/config/paths.ts` admits synchronous mutation after the last unrelated await. Its registry includes the optional config flight, ancestors and retained timeout children; one existing ACL deadline bounds admission, including newly overlapping work. Expiry throws sanitized `WindowsSecretAclSettleTimeoutError` (`ETIMEDOUT`) without running the writer or dropping child cleanup ownership. The final empty-registry check and callback share one turn, with no await in the callback. Intercept/picker CA, token, cleanup, profile and snapshot phases re-enter after trust/discovery awaits. Config saves in `src/server/management/agent-settings-routes.ts` and `src/server/management/config-routes.ts`, Desktop intent/mode commits in `src/server/management/native-integration-routes.ts`, and gateway publication/cleanup in those routes and `src/cli/ensure-desired-integrations.ts` use the same admission. `src/cli/claude-desktop.ts` admits local profile/mode commits and the complete remote-store transaction again after hub download. Lifecycle/config locks are acquired inside admission; refresh re-read, mode/freshness checks, profile and marker publication share one callback. Live setting edits and their save/publication share the admitted synchronous callback; a refused admission leaves both live and durable state untouched. Ordered post-commit phases preserve applied-state warnings or settled compensation, including mode/fingerprint marker refusal and retained CLI intent after incomplete rollback. Synchronous leaf APIs retain their contracts. `tests/claude-integration/claude-config-root-writer-barrier.test.ts` holds parent flights and drives refusal deadlines without sleeps. Lexical/canonical aliases and uncertain identities drain conservatively; removal keeps its separate at-or-below scope and actual-exit requirement. Async local config commands (`src/cli/config-command.ts`, `src/cli/effort.ts`, `src/cli/models.ts`, `src/cli/provider.ts`) admit complete synchronous read/validate/save phases through `withConfigDirHardeningSettled`. Preliminary model selection and client-connection diagnostics use passive observations that cannot publish recovery copies. Final authority and recovery publication are re-read inside admission without starting optional hardening. `src/cli/config-mutation.ts` refuses invalid or unreadable authority, including array/non-object roots rejected before defaults repair, with `CliUsageError`; full-config commands retain fresh-install initialization when the file is absent, while set/unset retain the persisted mutation owner's missing-file refusal. Custom-model removal revalidates the confirmed row by identity and contents while retaining current siblings. Import reads its input and checks its target inside admission. Provider-add credential cleanup shares the synchronous post-save phase and retains its existing post-commit warning. Timeout leaves the mutation database and config untouched and follows existing CLI failure handling. `tests/windows/windows-config-root-async-writers.test.ts` covers held parent flights, deadline refusal, recovery copies, invalid authority and concurrent edits. +`withConfigDirHardeningSettled` in `src/config/paths.ts` admits synchronous mutation after the last unrelated await. Its registry includes the optional config flight, ancestors and retained timeout children; one existing ACL deadline bounds admission, including newly overlapping work. Expiry throws sanitized `WindowsSecretAclSettleTimeoutError` (`ETIMEDOUT`) without running the writer or dropping child cleanup ownership. The final empty-registry check and callback share one turn, with no await in the callback. Intercept/picker CA, token, cleanup, profile and snapshot phases re-enter after trust/discovery awaits. Config saves in `src/server/management/agent-settings-routes.ts` and `src/server/management/config-routes.ts`, Desktop intent/mode commits in `src/server/management/native-integration-routes.ts`, and gateway publication/cleanup in those routes and `src/cli/ensure-desired-integrations.ts` use the same admission. `src/cli/claude-desktop.ts` admits local profile/mode commits and the complete remote-store transaction again after hub download. Lifecycle/config locks are acquired inside admission; refresh re-read, mode/freshness checks, profile and marker publication share one callback. Live setting edits and their save/publication share the admitted synchronous callback; a refused admission leaves both live and durable state untouched. Ordered post-commit phases preserve applied-state warnings or settled compensation, including mode/fingerprint marker refusal and retained CLI intent after incomplete rollback. Synchronous leaf APIs retain their contracts. `tests/claude-integration/claude-config-root-writer-barrier.test.ts` holds parent flights and drives refusal deadlines without sleeps. Lexical/canonical aliases and uncertain identities drain conservatively; removal keeps its separate at-or-below scope and actual-exit requirement. Async local config commands (`src/cli/config-command.ts`, `src/cli/effort.ts`, `src/cli/models.ts`, `src/cli/provider.ts`) admit complete synchronous read/validate/save phases through `withConfigDirHardeningSettled`. Preliminary model selection and client-connection diagnostics use passive observations that cannot publish recovery copies. Final authority and recovery publication are re-read inside admission without starting optional hardening. `src/cli/config-mutation.ts` refuses invalid or unreadable authority, including array/non-object roots rejected before defaults repair, with `CliUsageError`; full-config commands retain fresh-install initialization when the file is absent, while set/unset retain the persisted mutation owner's missing-file refusal. Custom-model removal revalidates the confirmed row by identity and contents while retaining current siblings. Import reads its input and checks its target inside admission. Provider-add credential cleanup shares the synchronous post-save phase and retains its existing post-commit warning. Timeout leaves the mutation database and config untouched and follows existing CLI failure handling. `tests/windows/windows-config-root-async-writers.test.ts` covers held parent flights, deadline refusal, recovery copies, invalid authority and concurrent edits. Codex multi-agent management changes that can resolve a runtime additionally await fresh [selection admission](codex-cli-update.md#worker-realm-and-the-selection-commit-window) before entering this barrier, keep that owner across hardening, and re-evaluate the mode plan inside the synchronous callback; neither ACL wait is inserted into the callback, and refusal leaves the combined mutation untouched. `src/config/process-state.ts` derives `ocx.pid` and `runtime-port.json` from that resolved directory. It owns their byte-compatible writes, parsing, expected-PID filters, cheap liveness, full OCX command diff --git a/structure/remote-workspace.md b/structure/remote-workspace.md index 6ed532f9cb4..d4afd8400e6 100644 --- a/structure/remote-workspace.md +++ b/structure/remote-workspace.md @@ -59,7 +59,7 @@ A track that holds a nested split reserves that subtree's minimum width (`minLay side-by-side splits never scroll inside a pane column; an oversized tree scrolls the pane surface once. Hub runtime admission counts pending create/resume starts as well as live handles against global and per-device limits; every outcome releases its reservation. Stop and shutdown reclaim late resumed handles before clearing ownership. Coordinator result size limits normalize both response text and success, so bridge and MCP callers receive consistent errors. -`src/remote-control/workspace-codex-runtime.ts` takes the shared Codex-CLI-update selection gate before execution-boundary trust validation and holds it across the asynchronous check through its app-server `Bun.spawn`. macOS executes the canonical checked target; validation or spawn failure releases the gate before bridge/isolation cleanup, attempting both cleanup steps while preserving the original error. While a cli-update worker holds admission through installer termination and readback the launch refuses with `codex_cli_selection_busy` after the same cleanup. See the [update commit window](codex-cli-update.md#worker-realm-and-the-selection-commit-window). +`src/remote-control/workspace-codex-runtime.ts` reserves the shared Codex-CLI-update selection gate before awaiting fresh recovery admission and execution-boundary trust validation, and holds it through its app-server `Bun.spawn`. Windows recovery awaits its native ACL reads under that same owner generation. macOS executes the canonical checked target; validation or spawn failure releases the gate before bridge/isolation cleanup, attempting both cleanup steps while preserving the original error. While a cli-update worker holds admission through installer termination and readback the launch refuses with `codex_cli_selection_busy` after the same cleanup. See the [update commit window](codex-cli-update.md#worker-realm-and-the-selection-commit-window). Prompt HTTP admission returns 202 with the existing session/event cursor; tracked operations publish terminal status for polling and attach a rejection observer immediately. Reconnect/resume cannot publish ready while a turn is active. The dashboard prefers newer local event cursors over stale polling, favors authoritative polling on ties, retains unconfirmed draft text with an uncertainty notice, and never retries prompt POSTs automatically. diff --git a/structure/runtime.md b/structure/runtime.md index 0af2d36768d..24fd3ad5035 100644 --- a/structure/runtime.md +++ b/structure/runtime.md @@ -1,6 +1,6 @@ # Runtime -The minute sweep checks persisted activation deadlines locally; only missing deadlines trigger metadata discovery. See the [quota activation contract](providers/openai-tiers.md#public-provider-contract). Codex control-plane ids are defined in the dependency-free `src/codex/control-plane-models.ts`; `src/router.ts` routes these exact ids unchanged to canonical OpenAI, retaining the existing error when no OpenAI provider is enabled. They do not widen the public native roster. Catalog requests and scheduled source refreshes follow the [nonblocking catalog observation contract](catalog.md#shared-catalog). Async runtime persistence rejects observed selection/epoch changes and returns detached frozen results; signal cancellation makes the persistence wrapper return null without writing, while raw async resolver rejections and exceptions without signal cancellation retain their rejection behavior; bounded exec rejects promptly and releases output readers on deadline or abort, independently awaits actual child exit, and escalates SIGTERM to SIGKILL after the shared 2-second grace (Windows termination is forceful). Known Codex.app and ChatGPT.app commands at any location or alias follow the [execution trust contract](clients/chatgpt-desktop.md#codex-app-execution-trust); automatic discovery also requires matching bundle identity; the existing ChatGPT bundle trust policy (safe ownership, permissions, non-symlink paths, and OpenAI signatures for both the bundle and the exact executable) is enforced at every execution boundary in src/codex/exec-boundary.ts before the resolved canonical target is spawned, not once at selection. Version probes retain that execution's bundle classification for selection, so a bare PATH alias of an app cannot supersede a discovered persisted runtime; selection does not resolve PATH again after the probe. Stored and displayed command spelling remains unchanged, and this observation grants no trust to a later execution. Claude overflow normalization examines at most 64 Ki characters, extracts stated context/request counts in separate forward scans, retains the bounded rewritten prefix, and leaves other error classifications intact; the managed native Messages SSE route follows the same contract (`tests/claude-integration/claude-prompt-too-long.test.ts`). Optional native desktop observation follows the [Codex Desktop startup gate and awaited teardown](clients/codex-desktop.md#proxy-startup-preference). Claude Code tier defaults are projected by `src/claude/model-default-settings.ts` into user settings only after explicit preview/apply, independently of launcher environment injection; see the [management contract](gui-and-management-api.md). The explicit projection pairs selected aliases with readable labels from `src/claude/context-windows.ts`, preserving custom labels and clearing only a label matching its current alias when an explicitly selected tier becomes a non-alias; unset tiers retain both settings model IDs and labels. No startup or periodic sync writes these defaults. +The minute sweep checks persisted activation deadlines locally; only missing deadlines trigger metadata discovery. See the [quota activation contract](providers/openai-tiers.md#public-provider-contract). Codex control-plane ids are defined in the dependency-free `src/codex/control-plane-models.ts`; `src/router.ts` routes these exact ids unchanged to canonical OpenAI, retaining the existing error when no OpenAI provider is enabled. They do not widen the public native roster. Catalog requests and scheduled source refreshes follow the [nonblocking catalog observation contract](catalog.md#shared-catalog). Async runtime persistence awaits the fresh [selection admission](codex-cli-update.md#worker-realm-and-the-selection-commit-window), rechecks observed selection/epoch and caller-currentness changes after that await, and returns detached frozen results; signal cancellation makes the persistence wrapper return null without writing, while raw async resolver rejections and exceptions without signal cancellation retain their rejection behavior; bounded exec rejects promptly and releases output readers on deadline or abort, independently awaits actual child exit, and escalates SIGTERM to SIGKILL after the shared 2-second grace (Windows termination is forceful). Known Codex.app and ChatGPT.app commands at any location or alias follow the [execution trust contract](clients/chatgpt-desktop.md#codex-app-execution-trust); automatic discovery also requires matching bundle identity; the existing ChatGPT bundle trust policy (safe ownership, permissions, non-symlink paths, and OpenAI signatures for both the bundle and the exact executable) is enforced at every execution boundary in src/codex/exec-boundary.ts before the resolved canonical target is spawned, not once at selection. Version probes retain that execution's bundle classification for selection, so a bare PATH alias of an app cannot supersede a discovered persisted runtime; selection does not resolve PATH again after the probe. Stored and displayed command spelling remains unchanged, and this observation grants no trust to a later execution. Claude overflow normalization examines at most 64 Ki characters, extracts stated context/request counts in separate forward scans, retains the bounded rewritten prefix, and leaves other error classifications intact; the managed native Messages SSE route follows the same contract (`tests/claude-integration/claude-prompt-too-long.test.ts`). Optional native desktop observation follows the [Codex Desktop startup gate and awaited teardown](clients/codex-desktop.md#proxy-startup-preference). Claude Code tier defaults are projected by `src/claude/model-default-settings.ts` into user settings only after explicit preview/apply, independently of launcher environment injection; see the [management contract](gui-and-management-api.md). The explicit projection pairs selected aliases with readable labels from `src/claude/context-windows.ts`, preserving custom labels and clearing only a label matching its current alias when an explicitly selected tier becomes a non-alias; unset tiers retain both settings model IDs and labels. No startup or periodic sync writes these defaults. ## Bun runtime and test runner diff --git a/tests/cli/cli-capability-operation-workflows.test.ts b/tests/cli/cli-capability-operation-workflows.test.ts index f877bdfcdd1..4e05a037e56 100644 --- a/tests/cli/cli-capability-operation-workflows.test.ts +++ b/tests/cli/cli-capability-operation-workflows.test.ts @@ -248,7 +248,7 @@ describe("operation capability workflows use existing handlers", () => { expect(tray).toContain('args.includes("--no-start") && sub !== "install"'); expect(tray).toContain('process.platform !== "win32"'); const dispatch = readFileSync(repoPath("src/cli/dispatch.ts"), "utf8"); - expect(dispatch).toContain('case "install": {\n const r = installCodexShim();'); + expect(dispatch).toContain('case "install": {\n const r = await installCodexShimAsync();'); }); }); diff --git a/tests/cli/cli-head.test.ts b/tests/cli/cli-head.test.ts index 52875ffedd6..feb1b1d1c91 100644 --- a/tests/cli/cli-head.test.ts +++ b/tests/cli/cli-head.test.ts @@ -184,7 +184,7 @@ describe("parseCliHead (pure CLI head, Phase 1)", () => { describe("uninstall argument validation", () => { test("uninstall and remove reject trailing arguments before shim preflight", async () => { - const preflight = spyOn(autorestore, "maybeAutoRestoreCodexShim").mockImplementation(() => {}); + const preflight = spyOn(autorestore, "maybeAutoRestoreCodexShimAsync").mockResolvedValue(undefined); const error = spyOn(console, "error").mockImplementation(() => {}); const exit = spyOn(process, "exit").mockImplementation((() => { throw new Error("usage-exit"); }) as never); try { @@ -201,7 +201,7 @@ describe("uninstall argument validation", () => { }); test("uninstall help bypasses preflight and bare commands retain it", async () => { - const preflight = spyOn(autorestore, "maybeAutoRestoreCodexShim").mockImplementation(() => {}); + const preflight = spyOn(autorestore, "maybeAutoRestoreCodexShimAsync").mockResolvedValue(undefined); const output = spyOn(console, "log").mockImplementation(() => {}); const exit = spyOn(process, "exit").mockImplementation((() => { throw new Error("help-exit"); }) as never); try { @@ -219,3 +219,31 @@ describe("uninstall argument validation", () => { } finally { preflight.mockRestore(); output.mockRestore(); exit.mockRestore(); } }); }); + +test.each([ + ["ready", "ready"], ["resolve", "resolve"], ["uninstall", "command"], +] as const)("the %s head awaits async shim preflight before becoming dispatchable", async (command, kind) => { + let entered!: () => void, release!: () => void; + const started = new Promise(resolve => { entered = resolve; }); + const held = new Promise(resolve => { release = resolve; }); + const preflight = spyOn(autorestore, "maybeAutoRestoreCodexShimAsync").mockImplementation(async () => { + entered(); + await held; + }); + let dispatchable = false; + const pending = runCli([command]).then(head => { dispatchable = true; return head; }); + try { + await Promise.race([started, pending.then(() => { throw new Error("head returned without entering preflight"); })]); + // Reach another event-loop turn while the preflight promise is unresolved. + await new Promise(resolve => setTimeout(resolve, 0)); + expect(preflight).toHaveBeenCalledTimes(1); + expect(preflight).toHaveBeenCalledWith(command, [command]); + expect(dispatchable).toBe(false); + release(); + expect(await pending).toMatchObject({ kind, command, args: [command] }); + expect(dispatchable).toBe(true); + } finally { + release(); + try { await pending; } finally { preflight.mockRestore(); } + } +}); diff --git a/tests/cli/cli-help-recovery.test.ts b/tests/cli/cli-help-recovery.test.ts index 63d0f9d758a..80bec0d4ca1 100644 --- a/tests/cli/cli-help-recovery.test.ts +++ b/tests/cli/cli-help-recovery.test.ts @@ -191,7 +191,7 @@ describe("CLI help recovery", () => { const calls = []; let reportCalls = false; mock.module(${JSON.stringify(repoPath("src", "cli", "codex-shim-autorestore.ts"))}, () => ({ - maybeAutoRestoreCodexShim: command => { calls.push(command); if (reportCalls) console.log("preflight called"); }, + maybeAutoRestoreCodexShimAsync: async command => { calls.push(command); if (reportCalls) console.log("preflight called"); }, })); // The lifecycle skew notice probes the configured (here: default) port. A developer machine // with a live proxy there would answer, so keep this admission test independent of the host. diff --git a/tests/cli/cli-ready.test.ts b/tests/cli/cli-ready.test.ts index c2d6ab69421..0c739b47020 100644 --- a/tests/cli/cli-ready.test.ts +++ b/tests/cli/cli-ready.test.ts @@ -687,31 +687,32 @@ describe("handleStart readinessGate wiring (source-level)", () => { }); }); -// ── P1: ready pre-parse before maybeAutoRestoreCodexShim (source-level) ──────── +// ── P1: ready pre-parse before maybeAutoRestoreCodexShimAsync (source-level) ──────── // `ocx ready` must reject invalid arguments with exit 64 BEFORE the global -// maybeAutoRestoreCodexShim preflight (or any discovery/probe/filesystem-capable +// maybeAutoRestoreCodexShimAsync preflight (or any discovery/probe/filesystem-capable // step) runs. These source-level guards pin that ordering and the single-parse // contract so a future edit cannot silently move parsing back into handleReady // or after auto-restore. The head block lives in src/cli/root.ts (Phase 1 of the // CLI deepening); the dispatch switch stays in src/cli/index.ts. No // subprocess/network/HOME is used. -describe("ready pre-parse before maybeAutoRestoreCodexShim (source-level, P1)", () => { +describe("ready pre-parse before maybeAutoRestoreCodexShimAsync (source-level, P1)", () => { const rootSource = readFileSync(repoPath("src/cli/root.ts"), "utf8"); const cliSource = readFileSync(repoPath("src/cli/index.ts"), "utf8"); - test("ready pre-parse call runs BEFORE maybeAutoRestoreCodexShim", () => { + test("ready pre-parse call runs BEFORE maybeAutoRestoreCodexShimAsync", () => { const preparseIdx = rootSource.indexOf("parseReadyArgs(args.slice(1))"); expect(preparseIdx, "pre-parse must call parseReadyArgs(args.slice(1))").toBeGreaterThanOrEqual(0); - const autoIdx = rootSource.indexOf("maybeAutoRestoreCodexShim(head.command, head.args)"); - expect(autoIdx, "maybeAutoRestoreCodexShim must be called in runCli").toBeGreaterThanOrEqual(0); - expect(preparseIdx, "ready pre-parse must precede maybeAutoRestoreCodexShim").toBeLessThan(autoIdx); + const autoIdx = rootSource.indexOf("await maybeAutoRestoreCodexShimAsync(head.command, head.args)"); + expect(autoIdx, "maybeAutoRestoreCodexShimAsync must be called in runCli").toBeGreaterThanOrEqual(0); + expect(preparseIdx, "ready pre-parse must precede maybeAutoRestoreCodexShimAsync").toBeLessThan(autoIdx); }); test("invalid ready exits 64 inside the pre-parse block, before auto-restore", () => { // parseCliHead (pure) returns readyArgs: undefined for invalid args; the // fail-closed runCli guard then exits 64 before any shim/discovery side // effect can run. - const autoIdx = rootSource.indexOf("maybeAutoRestoreCodexShim(head.command, head.args)"); + const autoIdx = rootSource.indexOf("await maybeAutoRestoreCodexShimAsync(head.command, head.args)"); + expect(autoIdx, "the awaited async preflight must exist before slicing its preceding guards").toBeGreaterThanOrEqual(0); const beforeAuto = rootSource.slice(0, autoIdx); expect(beforeAuto).toContain('command === "ready"'); expect(beforeAuto).toContain("parseReadyArgs(args.slice(1))"); @@ -742,7 +743,7 @@ describe("ready pre-parse before maybeAutoRestoreCodexShim (source-level, P1)", expect(body).not.toContain("process.exit"); }); - test("valid ready dispatch reaches handleReady AFTER maybeAutoRestoreCodexShim, with fail-closed guard", () => { + test("valid ready dispatch reaches handleReady AFTER maybeAutoRestoreCodexShimAsync, with fail-closed guard", () => { // Ordering: index.ts awaits runCli (which runs parseCliHead and the shim // preflight inside root.ts) BEFORE dispatch.ts runs the ready runner. const runCliIdx = cliSource.indexOf("await runCli(process.argv.slice(2))"); @@ -750,7 +751,7 @@ describe("ready pre-parse before maybeAutoRestoreCodexShim (source-level, P1)", const dispatchIdx = cliSource.indexOf("process.exit(await dispatchCommand(head"); expect(dispatchIdx, "index.ts must exit via dispatchCommand").toBeGreaterThanOrEqual(0); expect(runCliIdx).toBeLessThan(dispatchIdx); - expect(rootSource).toContain("maybeAutoRestoreCodexShim(head.command, head.args)"); + expect(rootSource).toContain("await maybeAutoRestoreCodexShimAsync(head.command, head.args)"); // The ready runner lives in dispatch.ts (keyed "ready:"); slice its body // up to the next runner key, not a fixed width. const dispatchSource = readFileSync(repoPath("src/cli/dispatch.ts"), "utf8"); diff --git a/tests/cli/cli-sync-expect-link.test.ts b/tests/cli/cli-sync-expect-link.test.ts index e6c1fc4b502..0cf79030200 100644 --- a/tests/cli/cli-sync-expect-link.test.ts +++ b/tests/cli/cli-sync-expect-link.test.ts @@ -202,7 +202,7 @@ describe("ocx sync --expect-link / --expect-hub-key", () => { // Never reach a real restart: on an unfixed tree a stray restart flag would end the Codex // app-servers and desktop app of whoever runs this file. const restarts = spyOn(restartScope, "handleRestartScopeAfterWrite").mockImplementation(async () => ({})); - const shimRepairs = spyOn(shimAutoRestore, "runCodexShimAutoRestore").mockImplementation(() => {}); + const shimRepairs = spyOn(shimAutoRestore, "runCodexShimAutoRestoreAsync").mockResolvedValue(undefined); const error = spyOn(console, "error").mockImplementation((...parts: unknown[]) => { errors.push(parts.map(String).join(" ")); }); const log = spyOn(console, "log").mockImplementation(() => {}); try { @@ -432,7 +432,7 @@ describe("the CLI root leaves a pinned sync to its own check", () => { throw new Error("process.exit"); }) as typeof process.exit); const error = spyOn(console, "error").mockImplementation(() => {}); - const preflight = spyOn(shimAutoRestore, "maybeAutoRestoreCodexShim"); + const preflight = spyOn(shimAutoRestore, "maybeAutoRestoreCodexShimAsync"); try { await expect(runCli(["sync", "--expect-link", "not-a-link-id"])).rejects.toThrow("process.exit"); expect(exits).toEqual([2]); diff --git a/tests/cli/cli-version-skew.test.ts b/tests/cli/cli-version-skew.test.ts index a0e2e7a0032..74a40c6a390 100644 --- a/tests/cli/cli-version-skew.test.ts +++ b/tests/cli/cli-version-skew.test.ts @@ -199,7 +199,7 @@ function freshNotice(scenario: NoticeScenario = {}) { maybeNoticeVersionSkew: (cmd,argv) => { events.push("hook"); return actual(cmd,argv,io); }, })); mock.module(${JSON.stringify(repoPath("src", "cli", "codex-shim-autorestore.ts"))}, () => ({ - maybeAutoRestoreCodexShim: () => events.push("shim"), + maybeAutoRestoreCodexShimAsync: async () => { events.push("shim"); }, })); const { runCli } = await import(${JSON.stringify(repoPath("src", "cli", "root.ts"))}); await runCli(args); events.push("dispatch"); diff --git a/tests/codex-integration/codex-cli-update-parent-async.test.ts b/tests/codex-integration/codex-cli-update-parent-async.test.ts new file mode 100644 index 00000000000..ed0f23a69b0 --- /dev/null +++ b/tests/codex-integration/codex-cli-update-parent-async.test.ts @@ -0,0 +1,297 @@ +import { afterEach, beforeEach, expect, spyOn, test } from "bun:test"; +import { existsSync, mkdirSync, readFileSync, readdirSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import * as gate from "../../src/codex/cli-update-selection-gate"; +import * as recovery from "../../src/codex/cli-update-recovery"; +import * as runtime from "../../src/codex/runtime"; +import { codexCliUpdateWorkerDiagnostics } from "../../src/codex/cli-update-job-worker"; +import { handleManagementAPI } from "../../src/server/management-api"; +import type { OcxConfig } from "../../src/types"; +import { hardenSecretDirAsync, setIcaclsRunnerForTests } from "../../src/lib/windows-secret-acl"; +import { setWindowsOwnerAclAsyncRunnerForTests, setWindowsOwnerAclRunnerForTests } from "../../src/lib/windows-owner-acl"; +import { createCodexCliRecoveryTestHome } from "../helpers/codex-cli-recovery-home"; +import { trustWindowsTempFixtureAncestry } from "../helpers/windows-temp-ancestry"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; +import { catalogConvergenceFactory } from "../helpers/catalog-convergence"; + +let home: string; +let saved: Record; +const spies: Array<{ mockRestore(): void }> = []; +beforeEach(() => { + home = createCodexCliRecoveryTestHome(); + saved = Object.fromEntries(["OPENCODEX_HOME", "CODEX_CLI_PATH", "CODEX_HOME"].map(key => [key, process.env[key]])); + process.env.OPENCODEX_HOME = home; + delete process.env.CODEX_CLI_PATH; + gate.resetCodexCliSelectionGatesForTests(); + runtime.clearCodexRuntimeResolveCache(); +}); +afterEach(() => { + for (const spy of spies.splice(0)) spy.mockRestore(); + setWindowsOwnerAclAsyncRunnerForTests(null); + setWindowsOwnerAclRunnerForTests(null); + setIcaclsRunnerForTests(null); + gate.resetCodexCliSelectionGatesForTests(); + runtime.clearCodexRuntimeResolveCache(); + for (const [key, value] of Object.entries(saved)) { + if (value === undefined) delete process.env[key]; else process.env[key] = value; + } + removeTreeWithRetry(home); +}); +function deferred() { + let resolve!: (value: T) => void; + let reject!: (error: unknown) => void; + const promise = new Promise((yes, no) => { resolve = yes; reject = no; }); + return { promise, resolve, reject }; +} +const free = () => Atomics.load(new Int32Array(gate.codexCliUpdateSelectionGate()), 0) === gate.CODEX_CLI_UPDATE_GATE_FREE; + +test("parent reserves before awaited recovery, yields, and excludes both another parent and a Worker", async () => { + const proof = deferred(); + let probes = 0; + const pending = gate.beginCodexCliSelectionParentWriteAsync({ recoveryBlocksAdmission: () => { probes++; return proof.promise; } }); + const worker = gate.createWorkerCommitWindow({ gate: gate.codexCliUpdateSelectionGate(), fence: gate.createCodexCliUpdateFence() }); + let timerProgress = false; + await new Promise(resolve => setTimeout(() => { timerProgress = true; resolve(); }, 0)); + expect(timerProgress).toBe(true); + expect(free()).toBe(false); + expect(codexCliUpdateWorkerDiagnostics().gate).toBe("parent_write"); + expect(worker.tryEnter()).toBe(false); + const second = await gate.beginCodexCliSelectionParentWriteAsync({ recoveryBlocksAdmission: async () => { probes++; return false; } }); + expect(second.entered).toBe(false); + expect(probes).toBe(1); + expect(gate.beginCodexCliSelectionParentWrite().entered).toBe(false); + proof.resolve(false); + const admitted = await pending; + try { + expect(admitted.entered).toBe(true); + expect(gate.beginCodexCliSelectionParentWrite().entered).toBe(false); + admitted.commit(() => { + const nested = gate.beginCodexCliSelectionParentWrite(); + expect(nested.entered).toBe(true); + nested.exit(); + expect(free()).toBe(false); + expect(gate.beginCodexCliSelectionParentWrite(join(home, "different-home")).entered).toBe(false); + }); + } finally { admitted.exit(); } + expect(worker.tryEnter()).toBe(true); + expect(codexCliUpdateWorkerDiagnostics().gate).toBe("worker_commit"); + worker.exit(); + expect(free()).toBe(true); +}); + +for (const outcome of ["denied", "thrown"] as const) test(`recovery ${outcome} leaves no mutation and allows a fresh admission`, async () => { + let writes = 0; + const pending = gate.withCodexCliSelectionParentWriteAsync(() => { writes++; }, { + recoveryBlocksAdmission: async () => { if (outcome === "thrown") throw new Error("fixture"); return true; }, + }); + await expect(pending).rejects.toThrow("codex_cli_selection_busy"); + expect(writes).toBe(0); + expect(free()).toBe(true); + await gate.withCodexCliSelectionParentWriteAsync(() => { writes++; }, { recoveryBlocksAdmission: async () => false }); + expect(writes).toBe(1); +}); + +test("aborting a pending check cannot free its gate before the check settles or admit a late allow", async () => { + const proof = deferred(), controller = new AbortController(); + const pending = gate.beginCodexCliSelectionParentWriteAsync({ signal: controller.signal, recoveryBlocksAdmission: () => proof.promise }); + controller.abort(); + expect(free()).toBe(false); + expect((await gate.beginCodexCliSelectionParentWriteAsync()).entered).toBe(false); + proof.resolve(false); + const refused = await pending; + expect(refused.entered).toBe(false); + expect(() => refused.commit(() => { throw new Error("must not run"); })).toThrow("codex_cli_selection_busy"); + expect(free()).toBe(true); +}); + +test("stale async refusal and repeated exit never release a later Worker generation", async () => { + const proof = deferred(), buffer = gate.codexCliUpdateSelectionGate(); + const pending = gate.beginCodexCliSelectionParentWriteAsync({ recoveryBlocksAdmission: () => proof.promise }); + const old = Atomics.load(new Int32Array(buffer), 0); + gate.releaseCodexCliUpdateGateOwner(buffer, old); + const worker = gate.createWorkerCommitWindow({ gate: buffer, fence: gate.createCodexCliUpdateFence() }); + expect(worker.tryEnter()).toBe(true); + const current = Atomics.load(new Int32Array(buffer), 0); + expect(current).not.toBe(old); + proof.resolve(false); + const stale = await pending; + expect(stale.entered).toBe(false); + stale.exit(); stale.exit(); + expect(Atomics.load(new Int32Array(buffer), 0)).toBe(current); + expect(codexCliUpdateWorkerDiagnostics().gate).toBe("worker_commit"); + worker.exit(); +}); + +test("commit authority does not reach queued work and its exception releases through the async wrapper", async () => { + let queued!: Promise; + const admitted = await gate.beginCodexCliSelectionParentWriteAsync({ recoveryBlocksAdmission: async () => false }); + try { + admitted.commit(() => { + queued = Promise.resolve().then(() => gate.beginCodexCliSelectionParentWrite().entered); + expect(gate.beginCodexCliSelectionParentWrite().entered).toBe(true); + }); + expect(await queued).toBe(false); + } finally { admitted.exit(); } + await expect(gate.withCodexCliSelectionParentWriteAsync(() => { throw new Error("commit failed"); }, + { recoveryBlocksAdmission: async () => false })).rejects.toThrow("commit failed"); + expect(free()).toBe(true); +}); + +for (const verdict of ["allow", "deny", "throw"] as const) +for (const change of ["input", "epoch", "cancel", "currentness"] as const) test(`runtime publication rechecks ${change} after async admission ${verdict}`, async () => { + const command = join(home, "codex.exe"); + writeFileSync(command, "fixture", { mode: 0o700 }); + process.env.CODEX_CLI_PATH = command; + const controller = new AbortController(); + const deps = { discoverAlternatives: false, signal: controller.signal }; + runtime.setCodexRuntimeResolveCacheForTests({ runtime: { command, source: "environment", version: "0.160.0" }, failures: [] }, { discoverAlternatives: false }); + const entered = deferred(), proof = deferred(); + spies.push(spyOn(recovery, "codexCliUpdateRecoveryBlocksAdmissionAsync").mockImplementation(() => { entered.resolve(); return proof.promise; })); + let current = true; + const pending = runtime.resolveAndPersistCodexRuntimeAsync(deps, () => current); + await entered.promise; + if (change === "input") process.env.CODEX_CLI_PATH = join(home, "other.exe"); + if (change === "epoch") runtime.clearCodexRuntimeResolveCache(); + if (change === "cancel") controller.abort(); + if (change === "currentness") current = false; + if (verdict === "throw") proof.reject(new Error("fixture recovery failure")); + else proof.resolve(verdict === "deny"); + expect(await pending).toBeNull(); + expect(existsSync(join(home, "codex-runtime.json"))).toBe(false); + expect(free()).toBe(true); +}); + +test("async runtime persistence owns the requested home, not an unrelated default home", async () => { + // POSIX admission legitimately delegates bounded filesystem checks to the + // sync checker. Isolate the admitted callback here; native tests cover ACL IO. + spies.push(spyOn(recovery, "codexCliUpdateRecoveryBlocksAdmissionAsync").mockResolvedValue(false)); + const reentry = spyOn(recovery, "codexCliUpdateRecoveryBlocksAdmission").mockImplementation(() => { throw new Error("sync recovery re-entry"); }); + spies.push(reentry); + const target = join(home, "selected"), record = { command: "codex", version: "0.160.0", source: "configured" as const }; + await runtime.persistCodexRuntimeAsync(record, { configDir: target }); + expect(JSON.parse(readFileSync(join(target, "codex-runtime.json"), "utf8")).command).toBe("codex"); + expect(existsSync(join(home, "codex-runtime.json"))).toBe(false); + await runtime.clearPersistedCodexRuntimeAsync({ configDir: target }); + expect(existsSync(join(target, "codex-runtime.json"))).toBe(false); + expect(reentry).not.toHaveBeenCalled(); +}); + +test("the async resolver publishes its selection without re-entering synchronous recovery", async () => { + const command = join(home, "codex.exe"); + writeFileSync(command, "fixture", { mode: 0o700 }); + process.env.CODEX_CLI_PATH = command; + runtime.setCodexRuntimeResolveCacheForTests({ runtime: { command, source: "environment", version: "0.160.0" }, failures: [] }, { discoverAlternatives: false }); + spies.push(spyOn(recovery, "codexCliUpdateRecoveryBlocksAdmissionAsync").mockResolvedValue(false)); + const reentry = spyOn(recovery, "codexCliUpdateRecoveryBlocksAdmission").mockImplementation(() => { throw new Error("sync recovery re-entry"); }); + spies.push(reentry); + const result = await runtime.resolveAndPersistCodexRuntimeAsync({ discoverAlternatives: false }); + expect(result?.persistError).toBeUndefined(); + expect(JSON.parse(readFileSync(join(home, "codex-runtime.json"), "utf8")).command).toBe(command); + expect(reentry).not.toHaveBeenCalled(); + expect(free()).toBe(true); +}); + +for (const outcome of ["allow", "deny", "plan-changed", "cancel"] as const) test(`management combined write waits for recovery and handles ${outcome} before mutation`, async () => { + process.env.CODEX_HOME = join(home, "codex"); + mkdirSync(process.env.CODEX_HOME); + const path = join(process.env.CODEX_HOME, "config.toml"); + const original = "[agents]\nmax_threads = 2\n"; + writeFileSync(path, original); + const config = { providers: [] } as unknown as OcxConfig; + const entered = deferred(), proof = deferred(), controller = new AbortController(); + let toggles = 0, syncProbes = 0; + spies.push(spyOn(recovery, "codexCliUpdateRecoveryBlocksAdmissionAsync").mockImplementation(() => { entered.resolve(); return proof.promise; })); + spies.push(spyOn(recovery, "codexCliUpdateRecoveryBlocksAdmission").mockImplementation(() => { syncProbes++; throw new Error("synchronous recovery must not re-enter"); })); + const req = new Request("http://localhost/api/v2", { method: "PUT", signal: controller.signal, + headers: { "host": "localhost", "content-type": "application/json" }, body: JSON.stringify({ enabled: true, multiAgentMode: "v2", agentsEnabled: false }) }); + const pending = handleManagementAPI(req, new URL(req.url), config, { + toggleCodexMultiAgentV2: enabled => { + toggles++; + // The real default toggle resolves/persists a runtime synchronously. This + // injected command exercises that nested writer without spawning a CLI. + runtime.persistCodexRuntime({ command: "fixture", source: "configured", version: null }); + writeFileSync(path, readFileSync(path, "utf8").replace(/^enabled\s*=\s*(?:true|false)$/m, `enabled = ${enabled}`)); + }, createManagementConvergeCodex: catalogConvergenceFactory(), + }); + try { + await Promise.race([entered.promise, pending.then(async response => { + throw new Error(`request settled before recovery: ${response?.status} ${await response?.text()}`); + })]); + await new Promise(resolve => setTimeout(resolve, 0)); + expect(free()).toBe(false); + expect(toggles).toBe(0); + expect(readFileSync(path, "utf8")).toBe(original); + expect(existsSync(join(home, "codex-runtime.json"))).toBe(false); + expect(config.multiAgentMode).toBeUndefined(); + if (outcome === "plan-changed") config.keepNativeChatGptOnV1 = true; + if (outcome === "cancel") controller.abort(); + proof.resolve(outcome === "deny"); + const response = await pending; + expect(response?.status).toBe(outcome === "allow" ? 200 : outcome === "plan-changed" ? 400 : 503); + expect(syncProbes).toBe(0); + expect(free()).toBe(true); + if (outcome === "allow") { + expect(toggles).toBe(1); + expect(config.multiAgentMode).toBe("v2"); + expect(readFileSync(path, "utf8")).toContain("enabled = false"); + expect(existsSync(join(home, "codex-runtime.json"))).toBe(true); + } else { + expect(toggles).toBe(0); + expect(config.multiAgentMode).toBeUndefined(); + expect(readFileSync(path, "utf8")).toBe(original); + expect(existsSync(join(home, "codex-runtime.json"))).toBe(false); + } + } finally { proof.resolve(true); await pending; } +}); + +const native = process.platform === "win32" && process.arch === "x64" ? test : test.skip; +async function nativeTerminalStore() { + await hardenSecretDirAsync(home, { required: true }); + trustWindowsTempFixtureAncestry(home); + const config = join(home, "config"); + mkdirSync(config); + await hardenSecretDirAsync(config, { required: true }); + const owner = recovery.createCodexCliUpdateRecovery(crypto.randomUUID(), config); + owner.arm(); owner.terminal(); + return config; +} +native("native recovery uses async ACL subprocesses while timers advance; the commit never re-enters synchronous recovery", async () => { + const configDir = await nativeTerminalStore(); + let syncCalls = 0, syncHardenCalls = 0, ticks = 0; + setWindowsOwnerAclRunnerForTests(() => { syncCalls++; throw new Error("sync ACL forbidden during recovery"); }); + setIcaclsRunnerForTests(() => { syncHardenCalls++; throw new Error("sync hardening forbidden during recovery"); }); + const timer = setInterval(() => { ticks++; }, 1); + let admitted: gate.CodexCliSelectionAsyncParentWrite | undefined; + try { + admitted = await gate.beginCodexCliSelectionParentWriteAsync({ configDir }); + expect(admitted.entered).toBe(true); + expect(ticks).toBeGreaterThan(0); + expect(syncCalls).toBe(0); + expect(syncHardenCalls).toBe(0); + // Ordinary atomic secret-file publication retains its synchronous contract. + // Only recovery admission is asynchronous; the nested writer must reuse + // this root/generation's admission without re-running recovery. + setWindowsOwnerAclRunnerForTests(null); + setIcaclsRunnerForTests(null); + const reentry = spyOn(recovery, "codexCliUpdateRecoveryBlocksAdmission").mockImplementation(() => { throw new Error("sync recovery re-entry"); }); + spies.push(reentry); + admitted.commit(() => runtime.persistCodexRuntime({ command: "fixture", version: null, source: "configured" }, { configDir })); + expect(existsSync(join(configDir, "codex-runtime.json"))).toBe(true); + expect(reentry).not.toHaveBeenCalled(); + } finally { clearInterval(timer); admitted?.exit(); } +}, 30_000); + +native("native async ACL denial preserves recovery bytes, releases pins, and refuses publication", async () => { + const configDir = await nativeTerminalStore(); + const dir = join(configDir, "codex-cli-update-recovery"); + const snapshot = () => readdirSync(dir).sort().map(name => [name, readFileSync(join(dir, name), "utf8")]); + const before = snapshot(); + setWindowsOwnerAclAsyncRunnerForTests(async () => ({ success: false, timedOut: true, stdout: "" })); + await expect(runtime.persistCodexRuntimeAsync({ command: "fixture", version: null, source: "configured" }, { configDir })) + .rejects.toThrow("codex_cli_selection_busy"); + expect(existsSync(join(configDir, "codex-runtime.json"))).toBe(false); + expect(snapshot()).toEqual(before); + expect(free()).toBe(true); + setWindowsOwnerAclAsyncRunnerForTests(null); + expect(await recovery.codexCliUpdateRecoveryBlocksAdmissionAsync(configDir)).toBe(false); +}, 30_000); diff --git a/tests/codex-integration/codex-cli-update-windows-pending.test.ts b/tests/codex-integration/codex-cli-update-windows-pending.test.ts index 8e783c50996..860284cf5e1 100644 --- a/tests/codex-integration/codex-cli-update-windows-pending.test.ts +++ b/tests/codex-integration/codex-cli-update-windows-pending.test.ts @@ -5,7 +5,8 @@ import * as ffi from "bun:ffi"; import * as acl from "../../src/lib/windows-secret-acl"; import { hardenSecretDirAsync, hardenSecretPath } from "../../src/lib/windows-secret-acl"; import { withWindowsCliRecoveryStore } from "../../src/codex/cli-update-job-store-windows"; -import { codexCliUpdateRecoveryBlocksAdmission } from "../../src/codex/cli-update-recovery"; +import { codexCliUpdateRecoveryBlocksAdmission, codexCliUpdateRecoveryBlocksAdmissionAsync } from "../../src/codex/cli-update-recovery"; +import { setWindowsOwnerAclRunnerForTests } from "../../src/lib/windows-owner-acl"; import { startWindowsCliChild } from "../../src/codex/windows-cli-child"; import * as lifetime from "../../src/codex/windows-cli-recovery"; import { createCodexCliTestRoot } from "../helpers/codex-cli-fixture-root"; @@ -67,6 +68,27 @@ windows("restart publishes a complete pending child only after terminal native p expect(statSync(join(home, "codex-cli-update-recovery", token + ".json")).ino).toBe(pendingIdentity); expect(codexCliUpdateRecoveryBlocksAdmission(home)).toBe(false); }, 30000); +windows("async restart adopts the same pending inode without synchronous ACL reads or hardening", async () => { + const { armed, child, proof } = records(); seed(armed, child, proof); + const pendingIdentity = statSync(join(home, "codex-cli-update-recovery", token + ".pending"), { bigint: true }).ino; + let syncReads = 0, syncHardens = 0, ticks = 0; + setWindowsOwnerAclRunnerForTests(() => { syncReads++; throw new Error("sync recovery ACL read"); }); + acl.setIcaclsRunnerForTests(() => { syncHardens++; throw new Error("sync recovery ACL hardening"); }); + const timer = setInterval(() => { ticks++; }, 1); + try { + expect(await codexCliUpdateRecoveryBlocksAdmissionAsync(home)).toBe(false); + expect(ticks).toBeGreaterThan(0); + expect(syncReads).toBe(0); + expect(syncHardens).toBe(0); + } finally { + clearInterval(timer); + setWindowsOwnerAclRunnerForTests(null); + acl.setIcaclsRunnerForTests(null); + } + expect(existsSync(join(home, "codex-cli-update-recovery", token + ".pending"))).toBe(false); + expect(withWindowsCliRecoveryStore(home, false, io => JSON.parse(io.read(token + ".json")))).toEqual(child); + expect(statSync(join(home, "codex-cli-update-recovery", token + ".json"), { bigint: true }).ino).toBe(pendingIdentity); +}, 30000); const negatives = ["token", "job", "parent", "parent-alive", "armed-child", "armed-identity", "missing-child-pid", "missing-child-time", "proof-missing", "proof-malformed", "proof-armed", "proof-token", "proof-job", "proof-parent", "proof-pid", "pending-malformed", "pending-armed", "native-unknown", "native-throws", "parent-denied", @@ -101,10 +123,10 @@ windows.each(negatives)("pending child stays fenced without matching authority: try { expect(codexCliUpdateRecoveryBlocksAdmission(home)).toBe(true); expect(snapshot()).toEqual(before); } finally { probe?.mockRestore(); parentProbe?.mockRestore(); } }, 30000); -windows.each(["file-flush", "harden", "rename", "readback", "flush"] as const)("reconciliation publication failure fences: %s", stage => { +for (const asynchronous of [false, true]) windows.each(["file-flush", "harden", "rename", "readback", "flush"] as const)(`${asynchronous ? "async" : "sync"} reconciliation publication failure fences: %s`, async stage => { const { armed, child, proof } = records(); seed(armed, child, proof); let injected = 0, renamed = false; - const harden = stage === "harden" ? spyOn(acl, "hardenSecretPath").mockImplementation(() => { injected++; throw new Error("harden_failed"); }) : undefined; + const harden = stage === "harden" ? spyOn(acl, asynchronous ? "hardenSecretPathAsync" : "hardenSecretPath").mockImplementation(() => { injected++; throw new Error("harden_failed"); }) : undefined; const realOpen = ffi.dlopen; const open = stage !== "harden" ? spyOn(ffi, "dlopen").mockImplementation(((...args: Parameters) => { const lib = realOpen(...args); @@ -123,7 +145,10 @@ windows.each(["file-flush", "harden", "rename", "readback", "flush"] as const)(" }, } }; }) as typeof ffi.dlopen) : undefined; - try { expect(codexCliUpdateRecoveryBlocksAdmission(home)).toBe(true); expect(injected).toBeGreaterThan(0); } + try { + expect(asynchronous ? await codexCliUpdateRecoveryBlocksAdmissionAsync(home) : codexCliUpdateRecoveryBlocksAdmission(home)).toBe(true); + expect(injected).toBeGreaterThan(0); + } finally { harden?.mockRestore(); open?.mockRestore(); } // Failure may leave either armed+pending or an already renamed child. Neither // path fabricates terminal proof or changes the persisted child identity. diff --git a/tests/codex-integration/multi-agent-keep-native-v1.test.ts b/tests/codex-integration/multi-agent-keep-native-v1.test.ts index 4b1abceabee..7c985cdd9a9 100644 --- a/tests/codex-integration/multi-agent-keep-native-v1.test.ts +++ b/tests/codex-integration/multi-agent-keep-native-v1.test.ts @@ -1,4 +1,4 @@ -import { afterEach, describe, expect, test } from "bun:test"; +import { afterEach, describe, expect, spyOn, test } from "bun:test"; import { mkdtempSync, readFileSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; @@ -10,6 +10,8 @@ import { import { CODEX_NATIVE_ALIAS_CATALOG_KIND } from "../../src/codex/catalog/kinds"; import { buildCatalogEntriesFromObservedState } from "../../src/codex/catalog/sync"; import { cmdV2 } from "../../src/cli/v2"; +import * as recovery from "../../src/codex/cli-update-recovery"; +import * as gate from "../../src/codex/cli-update-selection-gate"; import { loadConfig, saveConfig } from "../../src/config"; import { MULTI_AGENT_SURFACE_ADVISORY_VERSION, SUBAGENT_SURFACE_GUIDE_URL } from "../../src/config/multi-agent-surface"; import { isMultiAgentV2Enabled } from "../../src/codex/features"; @@ -530,3 +532,87 @@ describe("/api/v2 sub-agent surface advisory", () => { expect(config.multiAgentSurfaceAdvisoryVersion).toBeUndefined(); }); }); + +describe("ocx v2 revalidates state once the awaited admission enters", () => { + const fixtureSync = async () => ({ status: "applied" as const, ok: true, added: 0, catalogPath: null, catalogExists: true, catalogWritten: false, cacheSynced: false, message: "fixture" }); + function holdAdmission() { + let entered!: () => void, release!: (blocked: boolean) => void; + const started = new Promise(resolve => { entered = resolve; }); + const proof = new Promise(resolve => { release = resolve; }); + const spy = spyOn(recovery, "codexCliUpdateRecoveryBlocksAdmissionAsync").mockImplementation(() => { entered(); return proof; }); + return { started, release: () => release(false), restore: () => spy.mockRestore() }; + } + + test("v2 on refuses when the hybrid pin was enabled while admission waited", async () => { + isolateHomes(); + saveConfig({ ...loadConfig(), multiAgentMode: "v2" }); + const codexConfig = join(process.env.CODEX_HOME!, "config.toml"); + writeFileSync(codexConfig, "[features.multi_agent_v2]\nenabled = false\n"); + const hold = holdAdmission(); + const { errors, log } = captureLog(); + let toggles = 0; + try { + const running = cmdV2(["on"], { execFile: () => { toggles++; }, sync: async () => { throw new Error("must not sync"); }, log }); + await hold.started; + saveConfig({ ...loadConfig(), keepNativeChatGptOnV1: true }); + hold.release(); + expect(await running).toBe(1); + } finally { hold.restore(); } + expect(toggles).toBe(0); + expect(isMultiAgentV2Enabled(codexConfig)).toBe(false); + expect(errors.join("\n")).toContain("global multi_agent_v2 overrides the native v1 catalog pin"); + }); + + test("threads keeps a feature state changed while admission waited", async () => { + isolateHomes(); + const codexConfig = join(process.env.CODEX_HOME!, "config.toml"); + writeFileSync(codexConfig, "[features.multi_agent_v2]\nenabled = false\n"); + const hold = holdAdmission(); + const actions: string[] = []; + try { + const running = cmdV2(["threads", "4"], { + featuresInvocation: action => ({ file: "fixture-codex", args: ["features", action, "multi_agent_v2"], options: {} }), + execFile: (_file, args) => { actions.push(featureActionOf(args)); }, + log: captureLog().log, + }); + await hold.started; + writeFileSync(codexConfig, "[features.multi_agent_v2]\nenabled = true\n"); + hold.release(); + expect(await running).toBe(0); + } finally { hold.restore(); } + expect(actions).toEqual([]); + expect(isMultiAgentV2Enabled(codexConfig)).toBe(true); + }); + + test("mode saves the config read once admitted, keeping a setting saved meanwhile", async () => { + isolateHomes(); + writeFileSync(join(process.env.CODEX_HOME!, "config.toml"), "[features.multi_agent_v2]\nenabled = false\n"); + const hold = holdAdmission(); + try { + const running = cmdV2(["mode", "v1"], { execFile: () => {}, sync: fixtureSync, log: captureLog().log }); + await hold.started; + saveConfig({ ...loadConfig(), keepNativeChatGptOnV1: true }); + hold.release(); + expect(await running).toBe(0); + } finally { hold.restore(); } + expect(loadConfig().multiAgentMode).toBe("v1"); + expect(loadConfig().keepNativeChatGptOnV1).toBe(true); + }); + + test("mode saves inside admission, so a save landing after the callback is kept", async () => { + isolateHomes(); + writeFileSync(join(process.env.CODEX_HOME!, "config.toml"), "[features.multi_agent_v2]\nenabled = false\n"); + const real = gate.withCodexCliSelectionParentWriteAsync; + const spy = spyOn(gate, "withCodexCliSelectionParentWriteAsync").mockImplementation(async (run, options) => { + const value = await real(run, options); + // A competing writer between the admitted callback and this command's continuation. + saveConfig({ ...loadConfig(), keepNativeChatGptOnV1: true }); + return value; + }); + try { + expect(await cmdV2(["mode", "v1"], { execFile: () => {}, sync: fixtureSync, log: captureLog().log })).toBe(0); + } finally { spy.mockRestore(); } + expect(loadConfig().multiAgentMode).toBe("v1"); + expect(loadConfig().keepNativeChatGptOnV1).toBe(true); + }); +}); diff --git a/tests/fixtures/parent-stop-runner.ts b/tests/fixtures/parent-stop-runner.ts index 7a1f9dd48b3..db621f4b751 100644 --- a/tests/fixtures/parent-stop-runner.ts +++ b/tests/fixtures/parent-stop-runner.ts @@ -227,7 +227,7 @@ mock.module("../../src/server/system-env", () => ({ ...systemEnv, revertSystemEn const portReclaim = await import("../../src/server/port-reclaim"); mock.module("../../src/server/port-reclaim", () => ({ ...portReclaim, reclaimListenPort: async () => {} })); // Only shim preflight is unrelated to this stop contract; parsing and dispatch stay real. -mock.module("../../src/cli/codex-shim-autorestore", () => ({ maybeAutoRestoreCodexShim() {} })); +mock.module("../../src/cli/codex-shim-autorestore", () => ({ async maybeAutoRestoreCodexShimAsync() {} })); globalThis.fetch = (async (input: string | URL | Request, init?: RequestInit) => { const url = String(input); diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index 7b19eb71736..69b0093f255 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -74,6 +74,7 @@ "codex-cli-update-admission.test.ts": "codex-integration", "codex-cli-update-recovery.test.ts": "codex-integration", "codex-cli-update-worker.test.ts": "codex-integration", + "codex-cli-update-parent-async.test.ts": "codex-integration", "responses-replay-metadata-principal.test.ts": "responses", "responses-combo-replay-principal.test.ts": "responses", "responses-translated-replay-principal.test.ts": "responses", From 236a886f77c95c8d1191c339b845d5a84b7272e3 Mon Sep 17 00:00:00 2001 From: LilMGenius Date: Fri, 30 Oct 2026 22:11:48 +0900 Subject: [PATCH 31/42] feat(codex): tier role auto-assign candidates by measured capability (#7624) Tier Codex role auto-assign candidates by measured capability in the role's task domain, using the operator's own Artificial Analysis key, with configured tiers and price rank as fallback. The Hugging Face LMArena parquet source and the hyparquet dependency were removed before merge. The Artificial Analysis key joins configured-credential coverage. Co-authored-by: LilMGenius <97161055+LilMGenius@users.noreply.github.com> --- .../src/content/docs/guides/integrations.md | 33 +++- .../content/docs/guides/sub-agent-surface.md | 3 +- .../docs/reference/configuration/agents.md | 3 +- .../test-layout/layout-codex-collision.json | 4 + src/codex/role-auto-assign.ts | 76 ++++++--- src/codex/role-capability.ts | 160 ++++++++++++++++++ src/codex/role-sizing.ts | 30 +++- src/config/schema/config-schema.ts | 8 + src/server/configured-credentials.ts | 5 + .../management/codex-agent-role-routes.ts | 1 + .../management/codex-role-auto-assign.ts | 56 +++++- src/server/management/context.ts | 1 + src/types/config.ts | 5 + structure/config.md | 2 +- structure/subagents.md | 45 ++--- tests/fixtures/test-layout-expected.json | 4 + tests/routing/codex-role-auto-assign.test.ts | 66 +++++++- .../codex-role-capability-consent.test.ts | 59 +++++++ .../codex-role-capability-redirects.test.ts | 30 ++++ .../routing/codex-role-measured-ties.test.ts | 41 +++++ tests/routing/codex-role-sizing.test.ts | 21 ++- .../codex-role-auto-assign-routes.test.ts | 88 ++++++++++ .../codex-role-configured-credentials.test.ts | 55 ++++++ 23 files changed, 727 insertions(+), 69 deletions(-) create mode 100644 src/codex/role-capability.ts create mode 100644 tests/routing/codex-role-capability-consent.test.ts create mode 100644 tests/routing/codex-role-capability-redirects.test.ts create mode 100644 tests/routing/codex-role-measured-ties.test.ts create mode 100644 tests/server/codex-role-configured-credentials.test.ts diff --git a/docs-site/src/content/docs/guides/integrations.md b/docs-site/src/content/docs/guides/integrations.md index d76818a097c..ae6a4775100 100644 --- a/docs-site/src/content/docs/guides/integrations.md +++ b/docs-site/src/content/docs/guides/integrations.md @@ -987,15 +987,40 @@ exists only while LazyCodex is detected. Without it the dashboard shows neither, Auto-assign proposes a model for every role at once. opencodex asks your default Codex model (the root `model` in Codex `config.toml`) one question: for each role, given its -description and the start of its instructions, which capability tier (fast, standard or frontier) -and how much reasoning (glance, measured, thorough or exhaustive) does it need? That model never -picks a model. opencodex then picks the cheapest model from your picker list that reaches the tier: +description and the start of its instructions, which capability tier (fast, standard or frontier), +how much reasoning (glance, measured, thorough or exhaustive) and which task domain (coding, agentic, +reasoning, writing, knowledge or general) does it need? That model never picks a model. opencodex +then picks the cheapest model from your picker list that reaches the tier: - Models listed under `codexRoleTiers` in the opencodex config (`{ "fast": [...], "standard": [...], "frontier": [...] }`) have that tier. +- When two or more of the other models have a published score in the role's domain, those models are + tiered by distinct score values, without requiring a price; equal scores share a tier, and unscored models are left out. Scores come from + [Artificial Analysis](https://artificialanalysis.ai) when you add your own API key + (below). The response's `capability` block names the source it used. - Other models with a known price are ranked by price and split evenly across the three tiers. With only one or two priced models, the dearest is frontier and the other, if any, is standard. -- Models with no price and no listed tier are never proposed. List them to include them. +- When tiering falls back to price, models with no price and no listed tier are not proposed. List them to include them. + +To use Artificial Analysis, create a free API key at [artificialanalysis.ai](https://artificialanalysis.ai/api-reference) +and set it as a literal, an `${ENV}` reference or a keychain reference: + +```json +{ + "codexRoleCapability": { + "artificialAnalysis": { "apiKey": "${ARTIFICIAL_ANALYSIS_API_KEY}" } + } +} +``` + +opencodex reads it once per day, which stays inside the free tier's daily request limit. Its terms allow +using the data for your own decisions and forbid passing it on, so auto-assign keeps the scores +internal: the response carries the tier each model got and the attribution, without the scores. If the +key is absent or the read fails, auto-assign uses configured tiers and prices; `capability.error` reports +why and the source attribution fields are null. A retired `huggingFace` setting is ignored. + +Artificial Analysis refuses HTTP redirects and limits each response body to 16 MiB with a 30-second +request timeout. Credentials are sent only to its fixed API endpoint. Each proposal shows the model, the tier, the reasoning effort, a one-line reason, and what would move it up or down. A role the model could not size clearly is shown as not sized, with the reason, and diff --git a/docs-site/src/content/docs/guides/sub-agent-surface.md b/docs-site/src/content/docs/guides/sub-agent-surface.md index a7bd6c543eb..eab0c0000b8 100644 --- a/docs-site/src/content/docs/guides/sub-agent-surface.md +++ b/docs-site/src/content/docs/guides/sub-agent-surface.md @@ -151,7 +151,8 @@ description of the work Codex usually hands off, sizes it with one call to your and proposes the cheapest sufficient model and an effort from the list the picker offers, with the reasoning and the signals that would move it up or down. Nothing changes until you choose **Use this**, which saves exactly like picking the model and effort by hand. Tiers come from -`codexRoleTiers` or price, as in [role auto-assign](/guides/integrations/#auto-assign). +`codexRoleTiers` or price, as in [role auto-assign](/guides/integrations/#auto-assign), without the +measured scores role auto-assign can read from configured sources. `multiAgentGuidanceEnabled` defaults to on and is the master switch for opencodex-authored guidance on both surfaces. Turning it off suppresses both the v2 designation block and v1 proactive text. diff --git a/docs-site/src/content/docs/reference/configuration/agents.md b/docs-site/src/content/docs/reference/configuration/agents.md index 1b6791280cd..d2474b94f1c 100644 --- a/docs-site/src/content/docs/reference/configuration/agents.md +++ b/docs-site/src/content/docs/reference/configuration/agents.md @@ -56,7 +56,8 @@ still depends on upstream support for your account. | `subagentModelFallback?` | `string[]` | `[]` | Priority-ordered global fallback models for spawned child turns. | | `subagentModelFallbackByModel?` | `Record` | `{}` | Per-primary-model fallback chains, keyed by the requested primary model id. This is the supported home for per-role fallback metadata; `model_fallback` inside Codex agent TOML makes Codex 0.146+ skip the role (#1190). | | `subagentModelFallbackPollMs?` | `number` | `60000` | Availability-probe cache interval. Values below 1000 ms fall back to the default. | -| `codexRoleTiers?` | `{ fast?, standard?, frontier?: string[] }` | — | Capability tier of named models for [role auto-assign](/guides/integrations/#auto-assign). Listed models take that tier; other models are tiered by price, and unpriced unlisted models are never proposed. | +| `codexRoleTiers?` | `{ fast?, standard?, frontier?: string[] }` | — | Capability tier of named models for [role auto-assign](/guides/integrations/#auto-assign). Listed models take that tier; other models are tiered by measured score in the role's domain when two or more have one, without requiring prices. Equal scores share a tier. Otherwise price tiering excludes unpriced, unlisted models. | +| `codexRoleCapability?` | `{ artificialAnalysis?: { apiKey?: string } }` | No measured source | Measured-score source for [role auto-assign](/guides/integrations/#auto-assign). Artificial Analysis uses your API key (literal, `${ENV}` or keychain reference) once per day; scores stay out of the response. Missing or unavailable data falls back to configured tiers and prices. A retired `huggingFace` key is ignored. | | `effortCap?` | `string` | — | Hard ceiling for qualifying v2 main turns and marked spawned-child turns. Accepts `low` through `ultra`. | | `subagentEffortCap?` | `string` | — | Additional ceiling for spawned-child turns only. When both caps apply, the lower wins. | | `plaintextV2AgentMessages?` | `boolean` | — (unset) | Experimental opt-in. It runs only when explicitly set to `true` and asks eligible native ChatGPT v2 parents to emit `spawn_agent`, `send_message`, and `followup_task` message arguments as plaintext. See [Plaintext v2 agent messages](#plaintext-v2-agent-messages). | diff --git a/scripts/test-layout/layout-codex-collision.json b/scripts/test-layout/layout-codex-collision.json index d93e0f3748a..05cd39368be 100644 --- a/scripts/test-layout/layout-codex-collision.json +++ b/scripts/test-layout/layout-codex-collision.json @@ -1,4 +1,8 @@ { + "codex-role-configured-credentials.test.ts": "server", + "codex-role-capability-consent.test.ts": "routing", + "codex-role-capability-redirects.test.ts": "routing", + "codex-role-measured-ties.test.ts": "routing", "messaging-remote-port-reservation.test.ts": "codex-integration", "messaging-remote-forward-failure.test.ts": "codex-integration", "codex-cli-windows-child.test.ts": "codex-integration", diff --git a/src/codex/role-auto-assign.ts b/src/codex/role-auto-assign.ts index f2a29cea535..8fde0b66b2f 100644 --- a/src/codex/role-auto-assign.ts +++ b/src/codex/role-auto-assign.ts @@ -4,19 +4,23 @@ * * Tier of a candidate model, first rule that applies: * 1. codexRoleTiers in the opencodex config lists it under fast, standard or frontier. - * 2. It has a known price. Priced candidates are ranked by input plus output USD per 1M tokens. + * 2. At least two candidates have a measured score in the role's task domain. Measured candidates + * are ranked by that score and split across the tiers the same way price splits them below; + * unmeasured ones are left out rather than placed among measured ones by price. + * 3. It has a known price. Priced candidates are ranked by input plus output USD per 1M tokens. * Three or more split evenly across the three tiers by rank, cheapest in fast. Fewer than three * cannot fill every tier, so the ranking is anchored at the top: the dearest is frontier, * because it is the strongest model on offer, and each cheaper one sits one tier lower. One * priced model is frontier; two are standard and frontier. The tier left empty is fast, so a * fast role steps up to the cheapest model and a standard role keeps it instead of jumping to * the dearest, which keeps every role on the cheapest sufficient model on offer. - * 3. Otherwise it is unclassified and never proposed. + * 4. Otherwise it is unclassified and never proposed. * * A role gets the cheapest sufficient model: the lowest tier at or above what it was sized for, - * then the lowest price, then the model the role already uses when it is one of the tied models, - * then the picker's own order. The picker lists native models first, so without the current-model - * step a role on a routed model moved to a native one priced and tiered the same. + * then the lowest price, then the higher measured score, then the model the role already uses when + * it is one of the tied models, then the picker's own order. The picker lists native models first, + * so without the current-model step a role on a routed model moved to a native one priced and + * tiered the same. * * Effort binds to positions on the chosen model's ladder: glance is the floor, measured is the * model's default or the middle rung, exhaustive is the ceiling, and thorough is the rung above @@ -25,7 +29,7 @@ * above wins because thorough asks for more than the everyday setting. Nothing resolves outside * the ladder. */ -import { SIZING_TIERS, type RoleSizingOutcome, type SizingEffort, type SizingTier } from "./role-sizing"; +import { SIZING_TIERS, type RoleSizingOutcome, type SizingDomain, type SizingEffort, type SizingTier } from "./role-sizing"; import { CODEX_REASONING_LEVELS } from "../reasoning-effort"; export type RoleTierMapping = Partial>; @@ -40,7 +44,9 @@ export interface RoleModelCandidate { export interface ClassifiedCandidate extends RoleModelCandidate { readonly tier: SizingTier | null; - readonly tierSource: "mapping" | "price" | null; + readonly tierSource: "mapping" | "measured" | "price" | null; + /** The measured score the tier came from, or null when it came from elsewhere. */ + readonly score: number | null; } const LADDER_ORDER = ["none", "minimal", ...CODEX_REASONING_LEVELS.map(level => level.effort)]; @@ -66,33 +72,45 @@ export function mapEffortToLevel(intent: SizingEffort, efforts: readonly string[ } } +/** Split items ranked weakest first across the tiers; fewer than three are anchored at the top. */ +function tierByRank(ranked: readonly T[]): Map { + const top = SIZING_TIERS.length - 1; + return new Map(ranked.map((item, rank) => [item, SIZING_TIERS[ranked.length < SIZING_TIERS.length + ? top - (ranked.length - 1 - rank) + : Math.round((rank * top) / (ranked.length - 1))]!])); +} + export function classifyRoleModelCandidates( candidates: readonly RoleModelCandidate[], mapping: RoleTierMapping = {}, + measured: ReadonlyMap = new Map(), ): ClassifiedCandidate[] { const mapped = new Map(); for (const tier of SIZING_TIERS) { for (const model of mapping[tier] ?? []) if (!mapped.has(model)) mapped.set(model, tier); } - const priced = candidates - .map((candidate, order) => ({ candidate, order })) - .filter(({ candidate }) => !mapped.has(candidate.model) && candidate.unitPrice !== null) - .sort((a, b) => a.candidate.unitPrice! - b.candidate.unitPrice! || a.order - b.order); - const byPrice = new Map(); - priced.forEach(({ candidate }, rank) => { - const top = SIZING_TIERS.length - 1; - const index = priced.length < SIZING_TIERS.length - ? top - (priced.length - 1 - rank) - : Math.round((rank * top) / (priced.length - 1)); - byPrice.set(candidate.model, SIZING_TIERS[index]!); - }); + const unmapped = candidates.filter(candidate => !mapped.has(candidate.model)); + const scored = unmapped.filter(candidate => measured.has(candidate.model)); + const scoreTiers = tierByRank([...new Set(scored.map(candidate => measured.get(candidate.model)!))].sort((a, b) => a - b)); + const byScore = scored.length >= 2 + ? new Map(scored.map(candidate => [candidate.model, scoreTiers.get(measured.get(candidate.model)!)!] as const)) + : new Map(); + const byPrice = byScore.size > 0 + ? new Map() + : tierByRank(unmapped + .map((candidate, order) => ({ candidate, order })) + .filter(({ candidate }) => candidate.unitPrice !== null) + .sort((a, b) => a.candidate.unitPrice! - b.candidate.unitPrice! || a.order - b.order) + .map(({ candidate }) => candidate.model)); return candidates.map(candidate => { const fromMapping = mapped.get(candidate.model); - if (fromMapping) return { ...candidate, tier: fromMapping, tierSource: "mapping" as const }; + if (fromMapping) return { ...candidate, tier: fromMapping, tierSource: "mapping" as const, score: null }; + const fromScore = byScore.get(candidate.model); + if (fromScore) return { ...candidate, tier: fromScore, tierSource: "measured" as const, score: measured.get(candidate.model)! }; const fromPrice = byPrice.get(candidate.model); return fromPrice - ? { ...candidate, tier: fromPrice, tierSource: "price" as const } - : { ...candidate, tier: null, tierSource: null }; + ? { ...candidate, tier: fromPrice, tierSource: "price" as const, score: null } + : { ...candidate, tier: null, tierSource: null, score: null }; }); } @@ -107,6 +125,7 @@ export function cheapestSufficientCandidate( .filter(entry => entry.rank >= need) .sort((a, b) => a.rank - b.rank || (a.candidate.unitPrice ?? Infinity) - (b.candidate.unitPrice ?? Infinity) + || (b.candidate.score ?? -Infinity) - (a.candidate.score ?? -Infinity) || Number(b.candidate.model === current) - Number(a.candidate.model === current) || a.order - b.order); return ranked[0]?.candidate ?? null; @@ -123,10 +142,13 @@ export type RoleProposal = RoleProposalInput & ( readonly status: "proposed" | "unassigned"; readonly tier: SizingTier; readonly effortIntent: SizingEffort; + readonly domain: SizingDomain; readonly rationale: string; readonly moveUpIf: string; readonly moveDownIf: string; readonly proposedModel: string | null; + /** Where the picked model's tier came from: a user mapping, a measured score, or price rank. */ + readonly tierSource: ClassifiedCandidate["tierSource"]; /** The sized effort on the picked model's ladder; null only when that model has no ladder. */ readonly proposedEffort: string | null; readonly reason: string | null; @@ -138,23 +160,25 @@ export function buildRoleProposals( roles: readonly RoleProposalInput[], sizing: ReadonlyMap, classified: readonly ClassifiedCandidate[], + classifiedByDomain: ReadonlyMap = new Map(), ): RoleProposal[] { return roles.map(role => { const outcome = sizing.get(role.role) ?? { unsized: "the role was not sized" }; if ("unsized" in outcome) return { ...role, status: "unsized", reason: outcome.unsized }; - const { tier, effort, rationale, moveUpIf, moveDownIf } = outcome.sizing; - const pick = cheapestSufficientCandidate(classified, tier, role.model); - const sized = { ...role, tier, effortIntent: effort, rationale, moveUpIf, moveDownIf }; + const { tier, effort, domain, rationale, moveUpIf, moveDownIf } = outcome.sizing; + const pick = cheapestSufficientCandidate(classifiedByDomain.get(domain) ?? classified, tier, role.model); + const sized = { ...role, tier, effortIntent: effort, domain, rationale, moveUpIf, moveDownIf }; if (!pick) { return { ...sized, status: "unassigned", proposedModel: null, + tierSource: null, proposedEffort: null, reason: `no available model is classified ${tier} or above`, }; } const proposedEffort = mapEffortToLevel(effort, pick.efforts, pick.defaultEffort); - return { ...sized, status: "proposed", proposedModel: pick.model, proposedEffort, reason: null }; + return { ...sized, status: "proposed", proposedModel: pick.model, tierSource: pick.tierSource, proposedEffort, reason: null }; }); } diff --git a/src/codex/role-capability.ts b/src/codex/role-capability.ts new file mode 100644 index 00000000000..1edb0d1ded0 --- /dev/null +++ b/src/codex/role-capability.ts @@ -0,0 +1,160 @@ +/** + * Measured capability per task domain, so role auto-assign can tier candidates by what they score + * instead of what they cost. Artificial Analysis is read live when configured: + * + * - Artificial Analysis (https://artificialanalysis.ai/api-reference), when the user configures + * their own API key. Its terms allow that user's internal use with attribution and forbid + * redistributing the raw data, so its scores never leave this process. + * + * Each domain maps to one published index. The source names a model by display name + * with one row per reasoning setting ("Claude Opus 5.5 (High)"); capabilityModelKey drops the + * setting and folds the rest to a slug, which matches a catalog id once its provider prefix is + * gone ("cpa/claude-opus-5-5"). The best setting's score stands for the model, since a role can + * be given that setting. A candidate with no row is unmeasured, never guessed. + */ +import { resolveProviderApiKey } from "../providers/api-key-resolve"; +import type { OcxConfig } from "../types"; +import type { SizingDomain } from "./role-sizing"; + +export interface CapabilitySource { + readonly name: string; + readonly url: string; + readonly license: string; + /** Whether the terms let a response carry the raw scores. */ + readonly redistributable: boolean; +} + +export const ARTIFICIAL_ANALYSIS_SOURCE: CapabilitySource = { + name: "Artificial Analysis", + url: "https://artificialanalysis.ai", + license: "Artificial Analysis API terms: internal use, attribution required, no redistribution", + redistributable: false, +}; + +const ARTIFICIAL_ANALYSIS_URL = "https://artificialanalysis.ai/api/v2/data/llms/models"; + +/** Per domain, the indexes to read in order; the first one the response carries is used. */ +const INTELLIGENCE_INDEX = "artificial_analysis_intelligence_index"; +const ARTIFICIAL_ANALYSIS_INDEXES: Record = { + coding: ["artificial_analysis_coding_index", INTELLIGENCE_INDEX], + agentic: ["artificial_analysis_agentic_index", INTELLIGENCE_INDEX], + reasoning: ["artificial_analysis_math_index", INTELLIGENCE_INDEX], + writing: [INTELLIGENCE_INDEX], + knowledge: [INTELLIGENCE_INDEX], + general: [INTELLIGENCE_INDEX], +}; + +const FETCH_TIMEOUT_MS = 30_000; +/** Each published table is under 1 MB today; 16 MB leaves room to grow and still bounds memory. */ +export const MAX_TABLE_BYTES = 16 * 1024 * 1024; + +export interface CapabilityTable { + readonly source: CapabilitySource | null; + readonly publishedAt: string | null; + /** Per domain: model key to its best published score. */ + readonly domains: Partial>>; + /** Why a configured source was passed over for this one. */ + readonly fallbackReason?: string; +} + +export type LoadCapability = (config: OcxConfig) => Promise; + +export function capabilityModelKey(name: string): string { + return name.toLowerCase().replace(/\([^)]*\)/g, " ").replace(/[^a-z0-9]+/g, "-").replace(/^-+|-+$/g, ""); +} + +export function candidateModelKey(slug: string): string { + return capabilityModelKey(slug.slice(slug.lastIndexOf("/") + 1)); +} + +function keepBest(scores: Map, key: string, value: number) { + if (key) scores.set(key, Math.max(value, scores.get(key) ?? -Infinity)); +} + +export function capabilityTableFromArtificialAnalysis(body: unknown): CapabilityTable { + const rows = (body as { data?: unknown } | null)?.data; + if (!Array.isArray(rows)) throw new Error("Artificial Analysis returned no model list"); + const domains: Partial>> = {}; + for (const [domain, indexes] of Object.entries(ARTIFICIAL_ANALYSIS_INDEXES) as [SizingDomain, readonly string[]][]) { + for (const index of indexes) { + const scores = new Map(); + for (const row of rows as Record[]) { + const value = (row?.evaluations as Record | undefined)?.[index]; + if (typeof value !== "number" || !Number.isFinite(value)) continue; + if (typeof row.name === "string") keepBest(scores, capabilityModelKey(row.name), value); + if (typeof row.slug === "string") keepBest(scores, capabilityModelKey(row.slug), value); + } + if (scores.size > 0) { + domains[domain] = scores; + break; + } + } + } + return { source: ARTIFICIAL_ANALYSIS_SOURCE, publishedAt: null, domains }; +} + +/** The body's bytes, refused once they pass maxBytes, whatever the transport declared. */ +export async function readBoundedBytes(response: Response, maxBytes: number): Promise { + const chunks: Uint8Array[] = []; + let length = 0; + const reader = response.body?.getReader(); + if (!reader) return new ArrayBuffer(0); + for (;;) { + const { done, value } = await reader.read(); + if (done) break; + if (length + value.byteLength > maxBytes) { + await reader.cancel(); + throw new Error("response exceeded byte bound"); + } + chunks.push(value); + length += value.byteLength; + } + const out = new Uint8Array(length); + let offset = 0; + for (const chunk of chunks) { + out.set(chunk, offset); + offset += chunk.byteLength; + } + return out.buffer; +} + +export async function fetchArtificialAnalysisTable(apiKey: string, fetchImpl: typeof fetch = fetch): Promise { + const response = await fetchImpl(ARTIFICIAL_ANALYSIS_URL, { + redirect: "error", + headers: { "x-api-key": apiKey }, + signal: AbortSignal.timeout(FETCH_TIMEOUT_MS), + }); + if (!response.ok) throw new Error(`HTTP ${response.status}`); + const bytes = await readBoundedBytes(response, MAX_TABLE_BYTES); + return capabilityTableFromArtificialAnalysis(JSON.parse(new TextDecoder().decode(bytes))); +} + +/** One fetch per period key; a failed fetch is forgotten so the next request retries. */ +function periodCache() { + let slot: { key: string; table: Promise } | null = null; + return (key: string, fetchTable: () => Promise): Promise => { + if (slot?.key === key) return slot.table; + const table = fetchTable(); + slot = { key, table }; + table.catch(() => { if (slot?.table === table) slot = null; }); + return table; + }; +} + +const artificialAnalysisCache = periodCache(); + +/** + * Artificial Analysis when the user's key resolves, read once per UTC day to stay inside the + * free tier's daily request quota. Absent or unavailable scores leave price ranking in charge. + */ +export const loadCapabilityTable: LoadCapability = async config => { + const now = new Date().toISOString(); + const unavailable = (fallbackReason: string): CapabilityTable => ({ source: null, publishedAt: null, domains: {}, fallbackReason }); + const apiKey = resolveProviderApiKey(config.codexRoleCapability?.artificialAnalysis?.apiKey)?.trim(); + if (!apiKey) return unavailable("benchmark data was unavailable: no Artificial Analysis key is configured"); + try { + return await artificialAnalysisCache(`${now.slice(0, 10)} ${apiKey}`, () => fetchArtificialAnalysisTable(apiKey)); + } catch { + return unavailable("Artificial Analysis could not be read"); + } +}; diff --git a/src/codex/role-sizing.ts b/src/codex/role-sizing.ts index 2d1a42ffa94..a8e9b384e79 100644 --- a/src/codex/role-sizing.ts +++ b/src/codex/role-sizing.ts @@ -43,6 +43,10 @@ export type SizingTier = typeof SIZING_TIERS[number]; export const SIZING_EFFORTS = ["glance", "measured", "thorough", "exhaustive"] as const; export type SizingEffort = typeof SIZING_EFFORTS[number]; +/** Task domains, the kind of capability a tier is measured in. A label, never a model or a vendor. */ +export const SIZING_DOMAINS = ["coding", "agentic", "reasoning", "writing", "knowledge", "general"] as const; +export type SizingDomain = typeof SIZING_DOMAINS[number]; + const MAX_FIELD_CHARS = 400; export const ROLE_SIZING_SYSTEM_PROMPT = [ @@ -66,6 +70,15 @@ export const ROLE_SIZING_SYSTEM_PROMPT = [ "or a strong model needs only a quick call. Effort buys deliberation, never capability, and more", "effort is not more correct.", "", + "Task domain names the kind of capability the tier is measured in. It sizes nothing; it tells the", + "mapping which measured strength matters, because models of one price class differ by domain:", + "- coding: writing, changing, or reviewing code in a repository.", + "- agentic: long multi-step tool use, where the work is completing a run of actions without derailing.", + "- reasoning: math, logic, planning, or analysis whose answer is derived rather than looked up.", + "- writing: prose meant for a reader: documents, messages, copy.", + "- knowledge: research and factual recall across sources.", + "- general: none of the above dominates. Pick it rather than guess.", + "", "Assess risk and complexity once for both dials:", "- file, module, or ownership boundary crossing;", "- reversibility and blast radius;", @@ -86,8 +99,8 @@ export const ROLE_SIZING_SYSTEM_PROMPT = [ "- Use only the neutral words above. Do not name concrete model products, vendors, versions, or vendor effort levels anywhere in the answer.", "", "Answer with one JSON object and nothing else, no prose and no code fence:", - '{"roles":{"":{"tier":"fast|standard|frontier","effort":"glance|measured|thorough|exhaustive","rationale":"","move_up_if":"","move_down_if":""}}}', - "Include every role you were given, keyed by its exact name, with exactly those five fields.", + '{"roles":{"":{"tier":"fast|standard|frontier","effort":"glance|measured|thorough|exhaustive","domain":"coding|agentic|reasoning|writing|knowledge|general","rationale":"","move_up_if":"","move_down_if":""}}}', + "Include every role you were given, keyed by its exact name, with exactly those six fields.", ].join("\n"); /** @@ -105,6 +118,7 @@ export const DELEGATED_WORK_SIZING_SYSTEM_PROMPT = [ export interface RoleSizing { readonly tier: SizingTier; readonly effort: SizingEffort; + readonly domain: SizingDomain; readonly rationale: string; readonly moveUpIf: string; readonly moveDownIf: string; @@ -167,13 +181,16 @@ function validateEntry(entry: unknown): RoleSizingOutcome { const fields = entry as Record; if (!isMember(SIZING_TIERS, fields.tier)) return { unsized: "the sizing answer has no valid tier" }; if (!isMember(SIZING_EFFORTS, fields.effort)) return { unsized: "the sizing answer has no valid effort" }; + // A missing or unknown domain still sizes the role on the overall scale: the domain only picks + // which measured capability ranks the candidates, and "general" is the rubric's own no-signal answer. + const domain = isMember(SIZING_DOMAINS, fields.domain) ? fields.domain : "general"; const rationale = oneLine(fields.rationale); const moveUpIf = oneLine(fields.move_up_if); const moveDownIf = oneLine(fields.move_down_if); if (rationale === null || moveUpIf === null || moveDownIf === null) { return { unsized: "the sizing answer is missing a rationale or a move trigger" }; } - return { sizing: { tier: fields.tier, effort: fields.effort, rationale, moveUpIf, moveDownIf } }; + return { sizing: { tier: fields.tier, effort: fields.effort, domain, rationale, moveUpIf, moveDownIf } }; } const CODE_FENCE = /^\x60{3}(?:json)?[ \t]*\r?\n([\s\S]*?)\r?\n[ \t]*\x60{3}$/i; @@ -183,9 +200,10 @@ const CODE_FENCE = /^\x60{3}(?:json)?[ \t]*\r?\n([\s\S]*?)\r?\n[ \t]*\x60{3}$/i; * * The answer must be one JSON object of the shape the rubric asks for. Two leniencies, both for * content the mapping never reads: a single surrounding code fence, which models add out of habit, - * and keys beyond the five fields in a role's entry, which are ignored. Every one of the five must - * still be present and valid. Anything else wrong marks the affected role, or every role when the - * whole answer is unusable, as unsized with the reason. + * and keys beyond the six fields in a role's entry, which are ignored. Tier, effort, rationale and + * both move triggers must still be present and valid; a missing domain falls back to "general". + * Anything else wrong marks the affected role, or every role when the whole answer is unusable, + * as unsized with the reason. */ export function parseRoleSizingResponse(text: string, roles: readonly string[]): Map { const out = new Map(); diff --git a/src/config/schema/config-schema.ts b/src/config/schema/config-schema.ts index 9d3557f0fc2..b57c3efa537 100644 --- a/src/config/schema/config-schema.ts +++ b/src/config/schema/config-schema.ts @@ -324,6 +324,14 @@ export const configSchema = z.object({ standard: z.array(z.string().trim().min(1)).optional(), frontier: z.array(z.string().trim().min(1)).optional(), }).strict().optional().catch(undefined), + // Discard the retired source while preserving an otherwise valid Artificial Analysis key. + codexRoleCapability: z.preprocess(value => { + if (!value || typeof value !== "object" || Array.isArray(value)) return value; + const { huggingFace: _retired, ...rest } = value as Record; + return rest; + }, z.object({ + artificialAnalysis: z.object({ apiKey: z.string().trim().min(1).optional() }).strict().optional(), + }).strict().optional().catch(undefined)), codexShimAutoRestore: z.boolean().optional(), codexDesktopAuthless: z.boolean().optional().catch(undefined), codexClientCompaction: z.boolean().optional().catch(undefined), diff --git a/src/server/configured-credentials.ts b/src/server/configured-credentials.ts index f1143938446..2b44fe778aa 100644 --- a/src/server/configured-credentials.ts +++ b/src/server/configured-credentials.ts @@ -163,6 +163,11 @@ function build(configRead: StoreRead, authRead: StoreRead, signature: string, no || (config.value.providers !== undefined && !isRecord(config.value.providers)))) complete = false; if (authSig !== "absent" && auth.ok && !isRecord(auth.value)) complete = false; if (isRecord(config.value) && isRecord(config.value.providers)) walkProvider(config.value.providers, 0); + if (isRecord(config.value) && isRecord(config.value.codexRoleCapability) + && isRecord(config.value.codexRoleCapability.artificialAnalysis)) { + const key = config.value.codexRoleCapability.artificialAnalysis.apiKey; + addReference(typeof key === "string" ? key.trim() : key); + } if (isRecord(auth.value)) walkAuth(auth.value, 0); const names = [...envNames].sort(); diff --git a/src/server/management/codex-agent-role-routes.ts b/src/server/management/codex-agent-role-routes.ts index d9d55614dce..d8e80d911da 100644 --- a/src/server/management/codex-agent-role-routes.ts +++ b/src/server/management/codex-agent-role-routes.ts @@ -82,6 +82,7 @@ export async function handleCodexAgentRoleRoutes(ctx: ManagementContext): Promis ...(typeof body?.model === "string" ? { sizingModel: body.model } : {}), fetchAllModels: deps.fetchAllModels ?? fetchAllModels, ...(deps.completeCodexRoleSizing ? { completeRoleSizing: deps.completeCodexRoleSizing } : {}), + ...(deps.loadCodexRoleCapability ? { loadCapability: deps.loadCodexRoleCapability } : {}), }), 200, req, config); } catch (error) { if (error instanceof NoSizingModelError) { diff --git a/src/server/management/codex-role-auto-assign.ts b/src/server/management/codex-role-auto-assign.ts index f9f2381ec4f..c951fabfc2b 100644 --- a/src/server/management/codex-role-auto-assign.ts +++ b/src/server/management/codex-role-auto-assign.ts @@ -9,6 +9,8 @@ import { join } from "node:path"; import type { OcxConfig } from "../../types"; import type { CatalogModel } from "../../codex/catalog"; import type { RoleModelCandidate } from "../../codex/role-auto-assign"; +import type { LoadCapability } from "../../codex/role-capability"; +import type { SizingDomain } from "../../codex/role-sizing"; export interface RoleSizingCall { readonly model: string; @@ -118,11 +120,13 @@ export async function proposeCodexRoleModels(options: { sizingModel?: string; fetchAllModels: (config: OcxConfig) => Promise; completeRoleSizing?: CompleteRoleSizing; + loadCapability?: LoadCapability; }) { - const [roles, sizing, assign, { readConfiguredDefaultModel }] = await Promise.all([ + const [roles, sizing, assign, capability, { readConfiguredDefaultModel }] = await Promise.all([ import("../../codex/agent-role-models"), import("../../codex/role-sizing"), import("../../codex/role-auto-assign"), + import("../../codex/role-capability"), import("../../codex/catalog/parsing"), ]); const sizingModel = resolveSizingModel(options.sizingModel, readConfiguredDefaultModel); @@ -159,15 +163,55 @@ export async function proposeCodexRoleModels(options: { } } - const classified = assign.classifyRoleModelCandidates( - await roleModelCandidates(options.config, await options.fetchAllModels(options.config)), - options.config.codexRoleTiers, - ); + const candidates = await roleModelCandidates(options.config, await options.fetchAllModels(options.config)); + const classified = assign.classifyRoleModelCandidates(candidates, options.config.codexRoleTiers); + // Measured capability is a better tier signal than price, and its absence is not an error: a + // failed read leaves every role on price rank and says so. + const domains = new Set([...outcomes.values()].flatMap(outcome => "sizing" in outcome ? [outcome.sizing.domain] : [])); + let table: import("../../codex/role-capability").CapabilityTable | null = null; + let capabilityError: string | null = null; + if (domains.size > 0) { + try { + table = await (options.loadCapability ?? capability.loadCapabilityTable)(options.config); + } catch { + capabilityError = "the measured capability source could not be read"; + } + } + const classifiedByDomain = new Map>(); + for (const domain of domains) { + const scores = table?.domains[domain]; + if (!scores) continue; + const measured = new Map(candidates.flatMap(candidate => { + const score = scores.get(capability.candidateModelKey(candidate.model)); + return score === undefined ? [] : [[candidate.model, score] as const]; + })); + classifiedByDomain.set(domain, assign.classifyRoleModelCandidates(candidates, options.config.codexRoleTiers, measured)); + } return { sizingModel, sizingError, - proposals: assign.buildRoleProposals(roleRows, outcomes, classified), + proposals: assign.buildRoleProposals(roleRows, outcomes, classified, classifiedByDomain), candidates: classified.map(({ model, tier, tierSource, unitPrice }) => ({ model, tier, tierSource, unitPrice })), + capability: capabilityReport(table?.source ?? null, table, capabilityError, classifiedByDomain), + }; +} + +/** The attribution and per-domain tiers; raw scores only from a source whose terms allow it. */ +function capabilityReport( + source: import("../../codex/role-capability").CapabilitySource | null, + table: import("../../codex/role-capability").CapabilityTable | null, + error: string | null, + classifiedByDomain: ReadonlyMap>, +) { + return { + name: source?.name ?? null, + url: source?.url ?? null, + license: source?.license ?? null, + publishedAt: table?.publishedAt ?? null, + error: error ?? table?.fallbackReason ?? null, + domains: Object.fromEntries([...classifiedByDomain].map(([domain, list]) => [domain, + list.filter(candidate => candidate.tierSource === "measured") + .map(({ model, tier, score }) => (source?.redistributable ? { model, tier, score } : { model, tier }))])), }; } diff --git a/src/server/management/context.ts b/src/server/management/context.ts index b12ec76f27b..f25685d28e6 100644 --- a/src/server/management/context.ts +++ b/src/server/management/context.ts @@ -91,6 +91,7 @@ export interface ManagementApiDeps { fetchAllModels?: (config: OcxConfig) => Promise; /** Codex role auto-assign's one sizing model call; route tests answer it without a provider. */ completeCodexRoleSizing?: import("./codex-role-auto-assign").CompleteRoleSizing; + loadCodexRoleCapability?: import("../../codex/role-capability").LoadCapability; /** * Writer seam for the Grok toggle: lets a test place the file in any state * between the pre-write recheck and the write itself (the r8 post-inspection diff --git a/src/types/config.ts b/src/types/config.ts index 9a0e49e8aba..10fcdcff171 100644 --- a/src/types/config.ts +++ b/src/types/config.ts @@ -770,6 +770,11 @@ export interface OcxConfig { * classification. Models no list names and no price covers are never proposed. */ codexRoleTiers?: { fast?: string[]; standard?: string[]; frontier?: string[] }; + /** + * Measured-capability source for Codex role auto-assign. With an Artificial Analysis API key + * (a literal, ${ENV} or keychain reference) its indexes inform measured tiers. + */ + codexRoleCapability?: { artificialAnalysis?: { apiKey?: string } }; /** * TTL (ms) for cached sub-agent model availability probes. Default 60_000. */ diff --git a/structure/config.md b/structure/config.md index d5203ed40cb..7f6345e6529 100644 --- a/structure/config.md +++ b/structure/config.md @@ -1,5 +1,5 @@ # Config Surface -See [local timeout policy and OpenCode export](transports/timeouts.md) and default-off [`completeOnMaxOutputTokens`](transports/max-output-completion.md). Pasted GCP JSON persists only as `gcp-sa:` references in one pool transaction in `src/providers/api-keys.ts`; post-commit deletion follows [Vertex credential markers](providers/google.md#vertex-credential-markers). See [LM Studio Chat instruction configuration](providers/lm-studio-chat.md). +See [local timeout policy and OpenCode export](transports/timeouts.md) and default-off [`completeOnMaxOutputTokens`](transports/max-output-completion.md). Pasted GCP JSON persists only as `gcp-sa:` references in one pool transaction in `src/providers/api-keys.ts`; post-commit deletion follows [Vertex credential markers](providers/google.md#vertex-credential-markers). See [LM Studio Chat instruction configuration](providers/lm-studio-chat.md). `codexRoleCapability` in `src/config/schema/config-schema.ts` and `src/types/config.ts` follows the [capability source policy](subagents.md#role-model-auto-assign): only Artificial Analysis supplies measured scores, and the retired `huggingFace` key is ignored. `src/config/schema/leaf-validators.ts` accepts only boolean `apiKeys[].allowMeteredComboTargets`; malformed entries are dropped rather than promoted. `src/types/config.ts` and combo validation define optional boolean target `metered`, normalized false when omitted, and optional per-combo `allowMeteredTargets` (default false, literal boolean validation shared by config schema and management API). Explicit consent is required under the [combo runtime contract](runtime.md#request-local-target-compatibility). Model-price overrides follow the [pricing contract](pricing.md). Quota activation reuses the existing next-reset fields without adding a polling configuration key. See the [quota activation contract](providers/openai-tiers.md#public-provider-contract). `src/types/config.ts` defines `anthropicAccountPool.routes` as ordered model rules. `src/oauth/anthropic-model-routes.ts` validates bounded names, full case-sensitive globs and stored account IDs; `src/config/diagnostics.ts` rejects malformed candidate writes. Load retains a malformed hand edit so unrelated providers survive, while the enabled Anthropic selector refuses it locally before dispatch. Saved rules remain inert when the pool is disabled; affinity is process-local. Request logs use only the rule’s 1-based `route:#` position, never its configured name. `src/cli/hub-disable.ts` removes interdependent hub/loopback/role/hostname settings through `mutatePersistedConfig` and `validateConfigCandidate` in one commit. Top-level deletion intent uses `src/config/rebase-provenance.ts` and authoritative disk provenance in `src/config/live-reconcile.ts` to preserve removed hub bindings through later live saves, `src/config/live-topology.ts` keeps an independent active-topology baseline and merges desired next-start settings into a separate persistence projection, retaining actual socket identity and live hub/client topology during guarded saves and OAuth disk adoption. Client connections and runtimeRole remain one topology unit through connect/disconnect, including fresh disk-only connections. Disk additions/value updates and deletions all survive repeated saves without changing active topology. `src/config/api-key-reconcile.ts` merges key rows by stable ID under the existing save lock; either side's revocation defeats stale edits while disjoint names and rotations merge; unchanged live key membership/order adopts persisted ordering. `tests/config/hub-disable-live-save.test.ts` exercises disable followed by live key creation, rename, rotation, unrelated settings and repeated saves. Saved API keys have no immutable enrollment marker; explicit exact-ID revocation preserves all unselected keys. The WorkBuddy AI account row pins its adapter and origin under the [WorkBuddy contract](providers/workbuddy.md). JEV `decisionQuotaSignals` and partial `decisionQuotaTiers` use the shared validator and sparse preserve/clear/strategy-switch contract in [loaded advisory quota](providers/jev-decision.md#loaded-advisory-quota); the opt-in JEV `sessionPin` follows the same contract under [session pin](providers/jev-decision.md#session-pin). Native reasoning retention follows [its optional configuration contract](transports/native-reasoning-retention.md#configuration-and-management). Native function-result injection follows [the separate opt-in control contract](transports/streaming-health.md#experimental-native-function-result-injection); this surface does not infer upstream support or alter its defaults. diff --git a/structure/subagents.md b/structure/subagents.md index 9db8055c568..abfb582c73f 100644 --- a/structure/subagents.md +++ b/structure/subagents.md @@ -356,26 +356,33 @@ the same refusal as the PUT. Applying a proposal is the ordinary is not sent. The call goes through the proxy's own `/v1/chat/completions` via `postLocalChatCompletion` (`src/lib/local-chat-completion.ts`, shared with the routed vision describer), on the root `model` of Codex `config.toml` unless the caller names one. An answer that is not the strict JSON - shape, a missing role, or a field outside the vocabulary leaves that role **unsized** with the reason; - nothing is guessed. Keys beyond the five answer fields are ignored, and all five stay required. The + shape, a missing role, or a tier or effort outside the vocabulary leaves that role **unsized** with the reason; + a missing or unknown domain defaults to general. Keys beyond the six answer fields are ignored; the other five stay required. The sizing model never names a model. -- **Mapping is deterministic code.** `src/codex/role-auto-assign.ts` draws candidates from - `subagentSelectableModels` (the same list the role picker renders) and tiers them: `codexRoleTiers` in the - opencodex config first, then price rank (input plus output per 1M tokens). Three or more priced models - split evenly across the three tiers. Fewer are anchored at the top, the dearest frontier and each cheaper - one a tier lower, so one is frontier and two are standard and frontier; fast is the tier left empty, - which keeps a standard role on the cheaper model. Unpriced, unmapped models are never proposed. A role - gets the lowest sufficient tier, then the lowest price, then the model it already uses when that model is - in the tie, then picker order. Effort binds to ladder positions of the chosen - model (floor, default or middle, the rung above, ceiling), collapsing inside the range; it is proposed - for every sized role whose chosen model has a non-empty ladder. Apply and `--apply` insert or - replace `model_reasoning_effort` through the same span-preserving editor as `model`, located by - the same TOML-aware scan. An empty ladder gives `proposedEffort: null`, meaning no proposal; - Apply omits `effort` and preserves the existing pin. A null proposal never means a null PUT. A routed - model's ladder is its row's - `reasoningEfforts`; a row that declares none takes the ladder the written Codex catalog shows for it, an - explicit empty list stays empty, and a row without its own default takes the catalog's default when that - rung is on the ladder. +- **Mapping is deterministic code.** `src/codex/role-auto-assign.ts` draws candidates from `subagentSelectableModels` (the + same list the role picker renders) and tiers them: `codexRoleTiers` in the opencodex config first, then measured score in + the role's task domain, then price rank (input plus output per 1M tokens). The sizing answer carries a neutral `domain` + (coding, agentic, reasoning, writing, knowledge, general; missing means general), and `src/codex/role-capability.ts` reads + that domain's scores from Artificial Analysis once per UTC day when `codexRoleCapability.artificialAnalysis.apiKey` + resolves; the source refuses redirects and keeps a 30-second timeout and a 16 MiB body bound. With no key or on failure it + keeps configured tiers and price signals, reports the reason through `capability.error`, and returns null source + attribution. The `capability` block attributes the source; Artificial Analysis terms forbid redistribution, so its scores + are dropped from that block. Mapped models keep their mapped tier either way. With two or more measured unmapped + candidates, those are tiered by distinct score values (equal scores share a tier) without requiring prices, and unmeasured + unmapped ones are not proposed; otherwise price rank decides. `capability.error` is set only when no key is configured or + the source cannot be read; a source that answered but matched fewer than two candidates leaves it null, and a + proposal picked by price rank carries `tierSource: "price"`. Three or more priced models split evenly across the three + tiers. Fewer are anchored at the top, the dearest frontier and each cheaper one a tier lower, so one is frontier and two + are standard and frontier; fast is the tier left empty, which keeps a standard role on the cheaper model. Price tiering + excludes unpriced, unmapped models. A role gets the lowest sufficient tier, then the lowest price, then the higher + measured score, then the model it already uses when that model is in the tie, then picker order. Effort binds to ladder + positions of the chosen model (floor, default or middle, the rung above, ceiling), collapsing inside the range; it is + proposed for every sized role whose chosen model has a non-empty ladder. Apply and `--apply` insert or replace + `model_reasoning_effort` through the same span-preserving editor as `model`, located by the same TOML-aware scan. An empty + ladder gives `proposedEffort: null`, meaning no proposal; Apply omits `effort` and preserves the existing pin. A null + proposal never means a null PUT. A routed model's ladder is its row's `reasoningEfforts`; a row that declares none takes + the ladder the written Codex catalog shows for it, an explicit empty list stays empty, and a row without its own default + takes the catalog's default when that rung is on the ladder. The sizing call is injectable (`completeCodexRoleSizing` on the management deps), so route tests answer it without a provider. Sibling instances refuse the preview too: it sits under the refused diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index 69b0093f255..7939d5f9243 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -1,4 +1,5 @@ { "lmstudio-system-policy.test.ts": "config", "openai-chat-single-system.test.ts": "adapters/openai", "update-reclaim-authorizer.test.ts": "update", "update-capability-policy.test.ts": "update", "update-service-home.test.ts": "update", "update-service-home-policy.test.ts": "update", "update-home-recovery.test.ts": "update", "update-recovery-authority.test.ts": "update", "update-recovery-boundaries.test.ts": "update", "update-home-launcher.test.ts": "update", "update-home-worker.test.ts": "update", "update-mutation-boundaries.test.ts": "update", "update-mutation-guard.test.ts": "update", "update-guarded-ops.test.ts": "update", "update-effect-authority.test.ts": "update", "update-windows-wrapper-authority.test.ts": "update", "update-mutation-scanner.test.ts": "update", "refreshed-upstream-deadlines.test.ts": "responses", "local-upstream-deadlines.test.ts": "responses", "service-desktop-recovery.test.ts": "service", "provider-input-modalities.test.ts": "providers", "turn-membership.test.ts": "responses", "turn-membership-ingress.test.ts": "responses", "workbuddy-quota.test.ts": "providers", "workbuddy-login.test.ts": "oauth", "workbuddy-adapter.test.ts": "adapters", "workbuddy-provider.test.ts": "providers", "link-tunnel-history.test.ts": "clients", "link-readiness.test.ts": "clients", "link-diagnostics.test.ts": "clients", "combo-description.test.ts": "routing", "combo-description-catalog.test.ts": "codex-integration", "catalog-auto-review-override.test.ts": "codex-integration", "management-auto-review-settings.test.ts": "server", "management-auto-review-auth.test.ts": "server", "prefer-model-alias-ids.test.ts": "server", "responses-mimo-envelope.test.ts": "responses", "responses-call-id-linkage.test.ts": "responses", "responses-phase-inference.test.ts": "responses", + "codex-role-configured-credentials.test.ts": "server", "advisor-caller-policy.test.ts": "advisor", "advisor-caller-policy-sidecar-authority.test.ts": "advisor", "advisor-caller-policy-vision.test.ts": "advisor", @@ -1379,6 +1380,9 @@ "subagent-model-fallback-api.test.ts": "routing", "subagent-model-fallback.test.ts": "routing", "codex-agent-role-models.test.ts": "routing", "codex-role-sizing.test.ts": "routing", "codex-role-auto-assign.test.ts": "routing", "omo-role-models.test.ts": "clients", + "codex-role-capability-consent.test.ts": "routing", + "codex-role-capability-redirects.test.ts": "routing", + "codex-role-measured-ties.test.ts": "routing", "lazycodex-detection.test.ts": "clients", "codex-agent-role-routes.test.ts": "server", "codex-role-history-recovery.test.ts": "server", "codex-role-auto-assign-routes.test.ts": "server", "local-chat-completion.test.ts": "lib", diff --git a/tests/routing/codex-role-auto-assign.test.ts b/tests/routing/codex-role-auto-assign.test.ts index c4157b832e9..341a193d2e4 100644 --- a/tests/routing/codex-role-auto-assign.test.ts +++ b/tests/routing/codex-role-auto-assign.test.ts @@ -6,6 +6,7 @@ import { mapEffortToLevel, type RoleModelCandidate, } from "../../src/codex/role-auto-assign"; +import { candidateModelKey, capabilityTableFromArtificialAnalysis, readBoundedBytes } from "../../src/codex/role-capability"; import type { RoleSizingOutcome } from "../../src/codex/role-sizing"; const FULL = ["low", "medium", "high", "xhigh"]; @@ -73,6 +74,23 @@ describe("classifyRoleModelCandidates", () => { expect(cheapestSufficientCandidate(classified, "standard")!.model).toBe("cheap"); expect(cheapestSufficientCandidate(classified, "frontier")!.model).toBe("dear"); }); + + test("measured scores tier the candidates and leave unmeasured ones out", () => { + const classified = classifyRoleModelCandidates( + [candidate("cheap-strong", 1), candidate("dear-weak", 9), candidate("mid", 5), candidate("unmeasured", 2)], + {}, + new Map([["cheap-strong", 1500], ["dear-weak", 1400], ["mid", 1450]]), + ); + expect(classified.map(c => [c.model, c.tier, c.tierSource])).toEqual([ + ["cheap-strong", "frontier", "measured"], ["dear-weak", "fast", "measured"], ["mid", "standard", "measured"], ["unmeasured", null, null], + ]); + expect(cheapestSufficientCandidate(classified, "frontier")!.model).toBe("cheap-strong"); + }); + + test("one measured candidate is not a ranking, so price decides", () => { + const classified = classifyRoleModelCandidates([candidate("a", 1), candidate("b", 9)], {}, new Map([["a", 1500]])); + expect(classified.map(c => [c.model, c.tier, c.tierSource])).toEqual([["a", "standard", "price"], ["b", "frontier", "price"]]); + }); }); describe("cheapestSufficientCandidate", () => { @@ -100,7 +118,7 @@ describe("cheapestSufficientCandidate", () => { describe("buildRoleProposals", () => { const sized = (tier: "fast" | "standard" | "frontier", effort: "glance" | "exhaustive"): RoleSizingOutcome => ({ - sizing: { tier, effort, rationale: "r", moveUpIf: "u", moveDownIf: "d" }, + sizing: { tier, effort, domain: "general", rationale: "r", moveUpIf: "u", moveDownIf: "d" }, }); test("proposes an effort for every sized role, set or not, and keeps unsized reasons", () => { @@ -137,6 +155,25 @@ describe("buildRoleProposals", () => { expect(proposals.map(p => "proposedModel" in p ? p.proposedModel : null)).toEqual(["routed-mid", "native-mid", "routed-top"]); }); + test("ranks a role by its domain's scores and breaks an equal price by the higher score", () => { + const pool = [candidate("native-low", 0.6), candidate("routed-low", 0.6), candidate("native-mid", 12), candidate("routed-mid", 12), candidate("routed-top", 24), candidate("native-top", 60)]; + const coding = classifyRoleModelCandidates(pool, {}, new Map([ + ["native-low", 1400], ["routed-low", 1410], ["native-mid", 1480], ["routed-mid", 1490], ["native-top", 1500], ["routed-top", 1545], + ])); + const outcome = (tier: "fast" | "standard" | "frontier", domain: "coding" | "writing"): RoleSizingOutcome => ({ + sizing: { tier, effort: "glance", domain, rationale: "r", moveUpIf: "u", moveDownIf: "d" }, + }); + const proposals = buildRoleProposals( + [{ role: "low", model: null, effort: null }, { role: "mid", model: "native-mid", effort: null }, { role: "top", model: null, effort: null }, { role: "prose", model: null, effort: null }], + new Map([["low", outcome("fast", "coding")], ["mid", outcome("standard", "coding")], ["top", outcome("frontier", "coding")], ["prose", outcome("frontier", "writing")]]), + classifyRoleModelCandidates(pool), + new Map([["coding", coding]]), + ); + expect(proposals.map(p => "proposedModel" in p ? [p.proposedModel, p.tierSource] : null)).toEqual([ + ["routed-low", "measured"], ["routed-mid", "measured"], ["routed-top", "measured"], ["routed-top", "price"], + ]); + }); + test("reports a role no candidate can serve as unassigned", () => { const [proposal] = buildRoleProposals( [{ role: "architect", model: null, effort: null }], @@ -146,3 +183,30 @@ describe("buildRoleProposals", () => { expect(proposal).toMatchObject({ status: "unassigned", proposedModel: null, reason: "no available model is classified frontier or above" }); }); }); + +describe("capability response bounds", () => { + test("reads a leaderboard table up to the byte bound and refuses one past it", async () => { + const body = () => new Response(new ReadableStream({ + start(controller) { controller.enqueue(new Uint8Array(6)); controller.enqueue(new Uint8Array(6)); controller.close(); }, + })); + expect((await readBoundedBytes(body(), 12)).byteLength).toBe(12); + await expect(readBoundedBytes(body(), 11)).rejects.toThrow("response exceeded byte bound"); + }); +}); + +describe("capabilityTableFromArtificialAnalysis", () => { + test("reads each domain's index, falls back to the intelligence index, and matches catalog slugs", () => { + const table = capabilityTableFromArtificialAnalysis({ + status: 200, + data: [ + { name: "Claude Opus 5.5 (Reasoning)", slug: "claude-opus-5-5-reasoning", evaluations: { artificial_analysis_intelligence_index: 71, artificial_analysis_coding_index: 66 } }, + { name: "Claude Opus 5.5", slug: "claude-opus-5-5", evaluations: { artificial_analysis_intelligence_index: 64, artificial_analysis_coding_index: 58 } }, + { name: "GPT-6.1 Sol", slug: "gpt-6-1-sol", evaluations: { artificial_analysis_intelligence_index: 73, artificial_analysis_coding_index: 61 } }, + ], + }); + expect(table.source?.redistributable).toBe(false); + expect([candidateModelKey("cpa/claude-opus-5-5"), candidateModelKey("gpt-6.1-sol")].map(key => table.domains.coding!.get(key))).toEqual([66, 61]); + expect(table.domains.agentic!.get(candidateModelKey("gpt-6.1-sol"))).toBe(73); + expect(() => capabilityTableFromArtificialAnalysis({ error: "API key is required" })).toThrow(); + }); +}); diff --git a/tests/routing/codex-role-capability-consent.test.ts b/tests/routing/codex-role-capability-consent.test.ts new file mode 100644 index 00000000000..ca3de7b84f6 --- /dev/null +++ b/tests/routing/codex-role-capability-consent.test.ts @@ -0,0 +1,59 @@ +import { afterEach, expect, test } from "bun:test"; +import { configSchema } from "../../src/config/schema/config-schema"; +import { getDefaultConfig, loadConfig } from "../../src/config"; +import { mkdtempSync, writeFileSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { loadCapabilityTable } from "../../src/codex/role-capability"; +import type { OcxConfig } from "../../src/types"; + +const originalFetch = globalThis.fetch; +afterEach(() => { globalThis.fetch = originalFetch; }); +const config = (value: unknown) => ({ codexRoleCapability: configSchema.shape.codexRoleCapability.parse(value) }) as OcxConfig; + +test("only Artificial Analysis is requested and missing keys produce unattributed fallback", async () => { + const requests: string[] = []; + globalThis.fetch = Object.assign(async (input: string | URL | Request) => { + requests.push(String(input)); + return Response.json({ data: [{ name: "Fixture model", evaluations: { artificial_analysis_coding_index: 42 } }] }); + }, { preconnect: originalFetch.preconnect }) as typeof fetch; + const empty = await loadCapabilityTable(config(undefined)); + expect(empty).toMatchObject({ source: null, domains: {}, fallbackReason: "benchmark data was unavailable: no Artificial Analysis key is configured" }); + expect(requests).toEqual([]); + const table = await loadCapabilityTable(config({ artificialAnalysis: { apiKey: "aa-consent-fixture" } })); + expect(requests).toEqual(["https://artificialanalysis.ai/api/v2/data/llms/models"]); + expect(table.source?.name).toBe("Artificial Analysis"); + expect(table.domains.coding?.get("fixture-model")).toBe(42); +}); + +test("retired source key is ignored without dropping the Artificial Analysis key", () => { + const dir = mkdtempSync(join(tmpdir(), "ocx-aa-legacy-")); + const old = process.env.OPENCODEX_HOME; + process.env.OPENCODEX_HOME = dir; + try { + writeFileSync(join(dir, "config.json"), JSON.stringify({ ...getDefaultConfig(), codexRoleCapability: { artificialAnalysis: { apiKey: " aa-legacy-fixture " }, huggingFace: { enabled: true } } })); + expect(loadConfig().codexRoleCapability).toEqual({ artificialAnalysis: { apiKey: "aa-legacy-fixture" } }); + } finally { + if (old === undefined) delete process.env.OPENCODEX_HOME; else process.env.OPENCODEX_HOME = old; + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("any other malformed capability block is dropped whole and reported as no configured key", async () => { + // Same convention as the other strict optional blocks: a malformed block is never partially + // trusted. The proposal then reports the missing key instead of reading with a guessed one. + const parse = (value: unknown) => configSchema.shape.codexRoleCapability.parse(value); + expect(parse({ artificialAnalysis: { apiKey: "aa-fixture" }, unknownSource: { enabled: true } })).toBeUndefined(); + expect(parse({ artificialAnalysis: { apiKey: "aa-fixture", extra: 1 } })).toBeUndefined(); + expect(parse({ artificialAnalysis: { apiKey: 42 } })).toBeUndefined(); + expect(parse({ artificialAnalysis: { apiKey: " " } })).toBeUndefined(); + expect(parse("aa-fixture")).toBeUndefined(); + const requests: string[] = []; + globalThis.fetch = Object.assign(async (input: string | URL | Request) => { + requests.push(String(input)); + return Response.json({ data: [] }); + }, { preconnect: originalFetch.preconnect }) as typeof fetch; + const table = await loadCapabilityTable(config({ artificialAnalysis: { apiKey: "aa-fixture" }, unknownSource: {} })); + expect(table).toMatchObject({ source: null, fallbackReason: "benchmark data was unavailable: no Artificial Analysis key is configured" }); + expect(requests).toEqual([]); +}); diff --git a/tests/routing/codex-role-capability-redirects.test.ts b/tests/routing/codex-role-capability-redirects.test.ts new file mode 100644 index 00000000000..55fdd7d2243 --- /dev/null +++ b/tests/routing/codex-role-capability-redirects.test.ts @@ -0,0 +1,30 @@ +import { afterEach, expect, test } from "bun:test"; +import { fetchArtificialAnalysisTable, loadCapabilityTable } from "../../src/codex/role-capability"; +import type { OcxConfig } from "../../src/types"; + +const nativeFetch = globalThis.fetch; +const analysisUrl = "https://artificialanalysis.ai/api/v2/data/llms/models"; +afterEach(() => { globalThis.fetch = nativeFetch; }); + +for (const status of [302, 307]) { + test(`Artificial Analysis HTTP ${status} is refused without forwarding the key`, async () => { + const received: (string | null)[] = []; + const initial: (string | null)[] = []; + const sink = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch(request) { received.push(request.headers.get("x-api-key")); return new Response(); } }); + const first = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch(request) { initial.push(request.headers.get("x-api-key")); return new Response(null, { status, headers: { location: `${sink.url}target` } }); } }); + const requests: string[] = []; + const transport = Object.assign((input: string | URL | Request, init?: RequestInit) => { + requests.push(String(input)); + return nativeFetch(`${first.url}models`, init); + }, { preconnect: nativeFetch.preconnect }) as typeof fetch; + try { + await expect(fetchArtificialAnalysisTable("aa-redirect-fixture", transport)).rejects.toThrow(); + globalThis.fetch = transport; + const result = await loadCapabilityTable({ codexRoleCapability: { artificialAnalysis: { apiKey: `aa-redirect-${status}` } } } as OcxConfig); + expect(result).toMatchObject({ source: null, domains: {}, fallbackReason: "Artificial Analysis could not be read" }); + expect(requests).toEqual([analysisUrl, analysisUrl]); + expect(initial).toEqual(["aa-redirect-fixture", `aa-redirect-${status}`]); + expect(received).toEqual([]); + } finally { await first.stop(true); await sink.stop(true); } + }); +} diff --git a/tests/routing/codex-role-measured-ties.test.ts b/tests/routing/codex-role-measured-ties.test.ts new file mode 100644 index 00000000000..bc07f407e2e --- /dev/null +++ b/tests/routing/codex-role-measured-ties.test.ts @@ -0,0 +1,41 @@ +import { describe, expect, test } from "bun:test"; +import { cheapestSufficientCandidate, classifyRoleModelCandidates, type RoleModelCandidate } from "../../src/codex/role-auto-assign"; + +const candidate = (model: string, unitPrice: number | null): RoleModelCandidate => ({ model, unitPrice, efforts: [] }); + +describe("measured role tiers", () => { + test("equal scores share a tier regardless of provider order", () => { + const pool = [candidate("native/model-a", 1), candidate("routed/model-a", 5), candidate("model-b", 9)]; + const scores = new Map([["native/model-a", 1500], ["routed/model-a", 1500], ["model-b", 1550]]); + for (const candidates of [pool, [...pool].reverse()]) { + const classified = classifyRoleModelCandidates(candidates, {}, scores); + expect(classified.find(row => row.model === "native/model-a")?.tier).toBe("standard"); + expect(classified.find(row => row.model === "routed/model-a")?.tier).toBe("standard"); + expect(classified.find(row => row.model === "model-b")?.tier).toBe("frontier"); + expect(cheapestSufficientCandidate(classified, "standard")?.model).toBe("native/model-a"); + } + }); + + test("all equal measured scores share the top-anchored tier", () => { + const classified = classifyRoleModelCandidates([candidate("a", 1), candidate("b", 2)], {}, new Map([["a", 50], ["b", 50]])); + expect(classified.map(row => [row.tier, row.tierSource])).toEqual([["frontier", "measured"], ["frontier", "measured"]]); + }); + + test("one measured candidate still falls back to price and explicit mappings still win", () => { + const classified = classifyRoleModelCandidates( + [candidate("a", 1), candidate("b", 9), candidate("mapped", null)], + { fast: ["mapped"] }, new Map([["a", 50], ["mapped", 100]]), + ); + expect(classified.map(row => [row.tier, row.tierSource])).toEqual([ + ["standard", "price"], ["frontier", "price"], ["fast", "mapping"], + ]); + }); + + test("measured tiering can propose unpriced models but price tiering cannot", () => { + const pool = [candidate("a", null), candidate("b", null)]; + const measured = classifyRoleModelCandidates(pool, {}, new Map([["a", 50], ["b", 100]])); + expect(cheapestSufficientCandidate(measured, "standard")?.model).toBe("a"); + expect(cheapestSufficientCandidate(measured, "frontier")?.model).toBe("b"); + expect(cheapestSufficientCandidate(classifyRoleModelCandidates(pool), "fast")).toBeNull(); + }); +}); diff --git a/tests/routing/codex-role-sizing.test.ts b/tests/routing/codex-role-sizing.test.ts index 9edd8888341..59c13bc013d 100644 --- a/tests/routing/codex-role-sizing.test.ts +++ b/tests/routing/codex-role-sizing.test.ts @@ -3,6 +3,7 @@ import { DELEGATED_WORK_SIZING_SYSTEM_PROMPT, ROLE_INSTRUCTIONS_EXCERPT_CHARS, ROLE_SIZING_SYSTEM_PROMPT, + SIZING_DOMAINS, SIZING_EFFORTS, SIZING_TIERS, buildRoleSizingUserMessage, @@ -19,15 +20,15 @@ const GOOD = { }; describe("role sizing rubric", () => { - test("names every tier and effort and asks for JSON without model names", () => { - for (const word of [...SIZING_TIERS, ...SIZING_EFFORTS]) expect(ROLE_SIZING_SYSTEM_PROMPT).toContain(word); + test("names every tier, effort and domain and asks for JSON without model names", () => { + for (const word of [...SIZING_TIERS, ...SIZING_EFFORTS, ...SIZING_DOMAINS]) expect(ROLE_SIZING_SYSTEM_PROMPT).toContain(word); expect(ROLE_SIZING_SYSTEM_PROMPT).toContain("Do not name concrete model products"); expect(ROLE_SIZING_SYSTEM_PROMPT).toContain('"move_down_if"'); }); test("the role rubric keeps its exact bytes, and one-shot work gets the rubric plus an addendum", () => { const digest = new Bun.CryptoHasher("sha256").update(ROLE_SIZING_SYSTEM_PROMPT).digest("hex"); - expect(digest).toBe("5b611bbea1629e217d9d3ef77aeb02a331d99b1aed1d7d0074082ac7a1c11ff4"); + expect(digest).toBe("c43da51d3b6acebaeff086993de9694c9b86a11f01bdaf4dd8c6629af71b0709"); expect(DELEGATED_WORK_SIZING_SYSTEM_PROMPT.startsWith(`${ROLE_SIZING_SYSTEM_PROMPT}\n\n`)).toBe(true); expect(DELEGATED_WORK_SIZING_SYSTEM_PROMPT).toContain("one-shot"); }); @@ -62,6 +63,7 @@ describe("parseRoleSizingResponse", () => { sizing: { tier: "standard", effort: "measured", + domain: "general", rationale: GOOD.rationale, moveUpIf: GOOD.move_up_if, moveDownIf: GOOD.move_down_if, @@ -70,6 +72,16 @@ describe("parseRoleSizingResponse", () => { } }); + test("reads the domain and sizes a role whose domain is missing or unknown on the general scale", () => { + const out = parseRoleSizingResponse(JSON.stringify({ + roles: { coder: { ...GOOD, domain: "coding" }, odd: { ...GOOD, domain: "vibes" }, bare: GOOD }, + }), ["coder", "odd", "bare"]); + expect(["coder", "odd", "bare"].map(role => { + const outcome = out.get(role)!; + return "sizing" in outcome ? outcome.sizing.domain : outcome.unsized; + })).toEqual(["coding", "general", "general"]); + }); + test("marks every role unsized when the answer is not the JSON shape", () => { for (const text of ["Sure! explorer is standard.", "[]", '{"explorer":{}}', 'Here: {"roles":{}}']) { const out = parseRoleSizingResponse(text, ["explorer", "worker"]); @@ -93,7 +105,7 @@ describe("parseRoleSizingResponse", () => { expect(out.get("missing")).toEqual({ unsized: "the sizing model did not size this role" }); }); - test("ignores keys beyond the five it reads and keeps them out of the sizing", () => { + test("ignores keys beyond the six it reads and keeps them out of the sizing", () => { const out = parseRoleSizingResponse(JSON.stringify({ roles: { explorer: { ...GOOD, confidence: 0.8, notes: ["cheap"] } }, }), ["explorer"]); @@ -101,6 +113,7 @@ describe("parseRoleSizingResponse", () => { sizing: { tier: "standard", effort: "measured", + domain: "general", rationale: GOOD.rationale, moveUpIf: GOOD.move_up_if, moveDownIf: GOOD.move_down_if, diff --git a/tests/server/codex-role-auto-assign-routes.test.ts b/tests/server/codex-role-auto-assign-routes.test.ts index 2c4c816b534..fd8e1656c94 100644 --- a/tests/server/codex-role-auto-assign-routes.test.ts +++ b/tests/server/codex-role-auto-assign-routes.test.ts @@ -3,6 +3,7 @@ import { mkdirSync, mkdtempSync, readFileSync, readdirSync, writeFileSync } from import { tmpdir } from "node:os"; import { join } from "node:path"; import { listCatalogNativeSlugs, type CatalogModel } from "../../src/codex/catalog"; +import { ARTIFICIAL_ANALYSIS_SOURCE, type LoadCapability } from "../../src/codex/role-capability"; import { ROLE_SIZING_SYSTEM_PROMPT } from "../../src/codex/role-sizing"; import { handleManagementAPI } from "../../src/server/management-api"; import type { RoleSizingCall } from "../../src/server/management/codex-role-auto-assign"; @@ -70,6 +71,7 @@ const deps = { calls.push(call); return answer; }, + loadCodexRoleCapability: (async () => ({ source: null, publishedAt: null, domains: {} })) as LoadCapability, }; async function call(path: string, init?: RequestInit) { @@ -84,6 +86,57 @@ function snapshot(): Record { } describe("POST /api/codex-agent-roles/auto-assign", () => { + test("Artificial Analysis success produces measured tiers without exposing scores", async () => { + const originalFetch = globalThis.fetch; + const urls: string[] = []; + globalThis.fetch = Object.assign(async (input: string | URL | globalThis.Request) => { + urls.push(String(input)); + return Response.json({ data: [ + { name: "small", evaluations: { artificial_analysis_coding_index: 66 } }, + { name: "big", evaluations: { artificial_analysis_coding_index: 58 } }, + ] }); + }, { preconnect: originalFetch.preconnect }) as typeof fetch; + answer = { text: JSON.stringify({ roles: { explorer: { ...sized("frontier", "glance"), domain: "coding" } } }) }; + try { + const response = await handleManagementAPI( + new Request("http://localhost/api/codex-agent-roles/auto-assign", { method: "POST", body: "{}" }), + new URL("http://localhost/api/codex-agent-roles/auto-assign"), + { ...config, codexRoleTiers: undefined, codexRoleCapability: { artificialAnalysis: { apiKey: "aa-success-route-fixture" } } }, + { ...deps, loadCodexRoleCapability: undefined }, + ); + const body = await response!.json(); + expect(urls).toEqual(["https://artificialanalysis.ai/api/v2/data/llms/models"]); + expect(body.capability.name).toBe("Artificial Analysis"); + expect(body.capability.domains.coding).toEqual([{ model: "stub/small", tier: expect.any(String) }, { model: "stub/big", tier: expect.any(String) }]); + expect(body.proposals.find((row: { role: string }) => row.role === "explorer")).toMatchObject({ tierSource: "measured", proposedModel: "stub/small" }); + } finally { globalThis.fetch = originalFetch; } + }); + test.each([false, true])("disabled benchmarks retain proposals and report unavailability (Artificial Analysis configured: %s)", async withAnalysis => { + const originalFetch = globalThis.fetch; + const requests: string[] = []; + globalThis.fetch = Object.assign(async (input: string | URL | globalThis.Request) => { + requests.push(String(input)); + return new Response("fixture failure", { status: 401 }); + }, { preconnect: originalFetch.preconnect }) as typeof fetch; + try { + const response = await handleManagementAPI( + new Request("http://localhost/api/codex-agent-roles/auto-assign", { method: "POST", body: "{}" }), + new URL("http://localhost/api/codex-agent-roles/auto-assign"), + { ...config, codexRoleCapability: withAnalysis ? { artificialAnalysis: { apiKey: "route-consent-fixture" } } : undefined }, + { ...deps, loadCodexRoleCapability: undefined }, + ); + expect(response!.status).toBe(200); + const body = await response!.json(); + expect(requests).toEqual(withAnalysis ? ["https://artificialanalysis.ai/api/v2/data/llms/models"] : []); + expect(body.capability.error).toBe(withAnalysis ? "Artificial Analysis could not be read" : "benchmark data was unavailable: no Artificial Analysis key is configured"); + expect(body.capability).toMatchObject({ name: null, url: null, license: null, publishedAt: null }); + expect(body.capability.domains).toEqual({}); + expect(body.proposals.find((row: { role: string }) => row.role === "explorer")).toMatchObject({ status: "proposed", proposedModel: "stub/small" }); + } finally { + globalThis.fetch = originalFetch; + } + }); + test("proposes a model and effort per role through one sizing call and writes nothing", async () => { const before = snapshot(); const result = await call("/api/codex-agent-roles/auto-assign", { method: "POST", body: "{}" }); @@ -101,6 +154,41 @@ describe("POST /api/codex-agent-roles/auto-assign", () => { expect(byRole.bare.status).toBe("unsized"); }); + test("tiers a role by its domain's measured scores, and an unreadable source leaves price rank in charge", async () => { + const pricedConfig = { ...config, codexRoleTiers: undefined } as unknown as OcxConfig; + answer = { text: JSON.stringify({ roles: { explorer: { ...sized("frontier", "glance"), domain: "coding" } } }) }; + const post = async (loadCodexRoleCapability: typeof deps.loadCodexRoleCapability) => { + const response = await handleManagementAPI( + new Request("http://localhost/api/codex-agent-roles/auto-assign", { method: "POST", body: "{}" }), + new URL("http://localhost/api/codex-agent-roles/auto-assign"), + pricedConfig, + { ...deps, loadCodexRoleCapability }, + ); + return await response!.json() as Record; + }; + const measured = await post(async () => ({ + source: ARTIFICIAL_ANALYSIS_SOURCE, + publishedAt: null, + domains: { coding: new Map([["small", 1550], ["big", 1450]]) }, + })); + const explorer = (measured.proposals as any[]).find(p => p.role === "explorer"); + expect(explorer).toMatchObject({ domain: "coding", proposedModel: "stub/small", tierSource: "measured" }); + expect(measured.capability).toMatchObject({ name: "Artificial Analysis", publishedAt: null, error: null }); + expect(measured.capability.domains.coding[0].score).toBeUndefined(); + const licensed = await post(async () => ({ + source: ARTIFICIAL_ANALYSIS_SOURCE, + publishedAt: null, + domains: { coding: new Map([["small", 66], ["big", 58]]) }, + })); + expect(licensed.capability).toMatchObject({ name: "Artificial Analysis", url: "https://artificialanalysis.ai" }); + expect(licensed.capability.domains.coding[0]).toEqual({ model: "stub/small", tier: expect.any(String) }); + expect(licensed.capability.redistributable).toBeUndefined(); + const failed = await post(async () => { throw new Error("offline at /home/secret"); }); + expect(failed.capability.error).toBe("the measured capability source could not be read"); + expect(JSON.stringify(failed)).not.toContain("/home/secret"); + expect((failed.proposals as any[]).find(p => p.role === "explorer").tierSource).not.toBe("measured"); + }); + test("a routed model that declares no levels takes the ladder and default Codex shows for it", async () => { const levels = ["low", "medium", "high", "xhigh"].map(effort => ({ effort, description: effort })); writeFileSync(join(root, "codex", "opencodex-catalog.json"), JSON.stringify({ diff --git a/tests/server/codex-role-configured-credentials.test.ts b/tests/server/codex-role-configured-credentials.test.ts new file mode 100644 index 00000000000..643208119d2 --- /dev/null +++ b/tests/server/codex-role-configured-credentials.test.ts @@ -0,0 +1,55 @@ +import { expect, test } from "bun:test"; +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { configuredCredentials } from "../../src/server/configured-credentials"; +import { safeConfigDTO } from "../../src/server/auth-cors"; +import { handleManagementAPI } from "../../src/server/management-api"; +import { ManagementRequest } from "../helpers/management-auth"; +import { redactSecrets } from "../../src/lib/redact"; +import { diagnosticValueAllowed } from "../../src/server/request-log-terminal-status"; +import { setProviderKeychainEntryFactoryForTests } from "../../src/providers/api-key-resolve"; +import type { OcxConfig } from "../../src/types"; + +test("Artificial Analysis credentials are protected and refreshed", () => { + const dir = mkdtempSync(join(tmpdir(), "ocx-aa-credential-")); + const oldHome = process.env.OPENCODEX_HOME; + const oldKey = process.env.OCX_TEST_AA_KEY; + process.env.OPENCODEX_HOME = dir; + const write = (apiKey: string) => writeFileSync(join(dir, "config.json"), JSON.stringify({ providers: {}, codexRoleCapability: { artificialAnalysis: { apiKey } } })); + try { + write("aa-literal-secret-value"); + expect(configuredCredentials().matches("aa-literal-secret-value")).toBe(true); + process.env.OCX_TEST_AA_KEY = "aa-env-secret-one"; + write(" $OCX_TEST_AA_KEY "); + expect(configuredCredentials().matches("aa-env-secret-one")).toBe(true); + process.env.OCX_TEST_AA_KEY = "aa-env-secret-two"; + expect(configuredCredentials().matches("aa-env-secret-two")).toBe(true); + expect(configuredCredentials().matches("aa-env-secret-one")).toBe(false); + expect(diagnosticValueAllowed("harmless_diagnostic")).toBe(true); + expect(diagnosticValueAllowed("aa-env-secret-two")).toBe(false); + setProviderKeychainEntryFactoryForTests(() => ({ getPassword: () => "aa-keychain-secret", setPassword: () => {}, deletePassword: () => true })); + write(" keychain:aa-present "); + expect(configuredCredentials().matches("aa-keychain-secret")).toBe(true); + setProviderKeychainEntryFactoryForTests(() => ({ getPassword: () => null, setPassword: () => {}, deletePassword: () => true })); + write(" keychain:aa-missing "); + expect(configuredCredentials().complete).toBe(false); + const config = { providers: {}, codexRoleCapability: { artificialAnalysis: { apiKey: "aa-literal-secret-value" } } }; + expect(JSON.stringify(safeConfigDTO(config as OcxConfig))).not.toContain("aa-literal-secret-value"); + expect(JSON.stringify(redactSecrets(config))).not.toContain("aa-literal-secret-value"); + } finally { + setProviderKeychainEntryFactoryForTests(null); + if (oldHome === undefined) delete process.env.OPENCODEX_HOME; else process.env.OPENCODEX_HOME = oldHome; + if (oldKey === undefined) delete process.env.OCX_TEST_AA_KEY; else process.env.OCX_TEST_AA_KEY = oldKey; + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("GET /api/config does not expose the Artificial Analysis key", async () => { + const secret = "aa-management-fixture-secret"; + const config = { port: 10100, defaultProvider: "openai", providers: {}, codexRoleCapability: { artificialAnalysis: { apiKey: secret } } } as OcxConfig; + const url = new URL("http://localhost/api/config"); + const response = await handleManagementAPI(new ManagementRequest(url), url, config); + expect(response?.status).toBe(200); + expect(await response!.text()).not.toContain(secret); +}); From 3bda3e5fe074d67c1ac20f7380d32f43675d93e3 Mon Sep 17 00:00:00 2001 From: JUN Date: Fri, 30 Oct 2026 22:12:09 +0900 Subject: [PATCH 32/42] test(codex): assert the cold keyring refusal kind and report its cause (#8001) The refused scenarios only required some error, so a setup failure before the policy read was indistinguishable from the refusal (windows 10/11 of lane=all run 41056253802: reads 0, error unknown). Each scenario now names its expected refusal and the child reports message, cause and code. Co-authored-by: JUN <243035832+lidge-jun@users.noreply.github.com> --- .../main-quota-provenance.test.ts | 28 +++++++++++-------- 1 file changed, 17 insertions(+), 11 deletions(-) diff --git a/tests/codex-integration/main-quota-provenance.test.ts b/tests/codex-integration/main-quota-provenance.test.ts index 5494cd2c29d..11db5da67f4 100644 --- a/tests/codex-integration/main-quota-provenance.test.ts +++ b/tests/codex-integration/main-quota-provenance.test.ts @@ -351,13 +351,13 @@ describe("main policy quota durability and lifecycle", () => { for (const scenario of [ { name: "verified replacement", account: "fixture-main-b", tag: "known", selected: true }, - { name: "same account", account: "fixture-main-a", tag: "known", selected: false }, - { name: "untagged quota", account: "fixture-main-b", tag: "missing", selected: false }, - { name: "invalid quota tag", account: "fixture-main-b", tag: "invalid", selected: false }, - { name: "unknown RPC identity", account: "", tag: "known", selected: false }, - { name: "manually paused", account: "fixture-main-b", tag: "known", selected: false, paused: true }, + { name: "same account", account: "fixture-main-a", tag: "known", selected: false, refusal: "CodexMainAccountHardLockError" }, + { name: "untagged quota", account: "fixture-main-b", tag: "missing", selected: false, refusal: "CodexPoolAuthenticationError" }, + { name: "invalid quota tag", account: "fixture-main-b", tag: "invalid", selected: false, refusal: "CodexPoolAuthenticationError" }, + { name: "unknown RPC identity", account: "", tag: "known", selected: false, refusal: "CodexPoolAuthenticationError" }, + { name: "manually paused", account: "fixture-main-b", tag: "known", selected: false, paused: true, refusal: "CodexPoolAuthenticationError" }, { name: "startup file replacement", account: "fixture-main-b", tag: "known", selected: true, source: "file" }, - { name: "startup same-account file", account: "fixture-main-a", tag: "known", selected: false, source: "file" }, + { name: "startup same-account file", account: "fixture-main-a", tag: "known", selected: false, source: "file", refusal: "CodexMainAccountHardLockError" }, ]) { test(`cold keyring Pool preserves quota unless replacement is proven: ${scenario.name}`, () => { const writer = writerFor("fixture-main-a"); @@ -387,7 +387,7 @@ describe("main policy quota durability and lifecycle", () => { pausedCodexAccountIds: ${JSON.stringify(scenario.paused ? [MAIN] : [])}, providers: { openai: { adapter: "openai-responses", baseUrl: "https://chatgpt.com/backend-api/codex", authMode: "forward" } } }; let selected = false; - let failure; + let failure, failureDetail; try { const result = await resolveCodexAuthContext(new Headers(), config, "pool", { nativeMainRefreshDependencies: { codexAuthStatus: async () => { reads++; return { authMethod: "chatgpt", authToken: token, requiresOpenaiAuth: true }; } }, @@ -395,8 +395,11 @@ describe("main policy quota durability and lifecycle", () => { beginCodexAccountSelection: () => ({ mainProfileDraining: false, claimMainProfile: () => true, release() {} }), }); selected = result.kind === "main-pool" && result.chatgptAccountId === account; - } catch (error) { failure = error.name; } - console.log(JSON.stringify({ fixture: "cold-keyring-quota", before, selected, failure, reads, initialized, + } catch (error) { + failure = error.name; + failureDetail = { message: String(error.message).slice(0, 200), cause: error.cause?.name, code: error.cause?.code ?? error.code }; + } + console.log(JSON.stringify({ fixture: "cold-keyring-quota", before, selected, failure, failureDetail, reads, initialized, legacy: getAccountQuota("__main__"), policy: getMainPolicyQuota(), stored: getAccountQuota("fixture-stored"), paused: config.pausedCodexAccountIds })); `; @@ -416,8 +419,11 @@ describe("main policy quota durability and lifecycle", () => { expect(result.legacy).toBeNull(); expect(result.policy).toBeNull(); } else { - expect(result.failure).toBeDefined(); - expect(result.reads).toBe(scenario.paused || scenario.source === "file" ? 0 : 1); + // The refusal must be the policy decision itself; any earlier setup failure (with no read) + // fails here with its message and cause instead of passing as a refusal. + expect({ failure: result.failure, reads: result.reads, detail: result.failureDetail }).toMatchObject({ + failure: scenario.refusal, reads: scenario.paused || scenario.source === "file" ? 0 : 1, + }); expect(result.legacy.weeklyPercent).toBe(100); if (scenario.account === "fixture-main-a") expect(result.policy.weeklyPercent).toBe(100); } From b53b5d26e22a8284fd7d46035432422d6643b05d Mon Sep 17 00:00:00 2001 From: JUN Date: Fri, 30 Oct 2026 22:13:52 +0900 Subject: [PATCH 33/42] test(cc-switch): finalize the F1/F2 fixture statements before cleanup (#8002) Like F3 (#7986), F1/F2 inserted with unfinalized prepare().run() statements, which keep the SQLite file open past close() on Windows and make the temp-tree cleanup fail with EBUSY. Use db.run(sql, ...params). Co-authored-by: JUN <243035832+lidge-jun@users.noreply.github.com> --- tests/providers/cc-switch-import.test.ts | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/tests/providers/cc-switch-import.test.ts b/tests/providers/cc-switch-import.test.ts index 106475de1bd..a4608177aba 100644 --- a/tests/providers/cc-switch-import.test.ts +++ b/tests/providers/cc-switch-import.test.ts @@ -827,7 +827,9 @@ describe("CC-Switch keyless provider import", () => { id TEXT NOT NULL, app_type TEXT NOT NULL, name TEXT NOT NULL, settings_config TEXT NOT NULL, website_url TEXT, notes TEXT, meta TEXT NOT NULL DEFAULT '{}', sort_index INTEGER, PRIMARY KEY (id, app_type) )`); - db.prepare("INSERT INTO providers (id, app_type, name, settings_config, website_url, notes, meta, sort_index) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8)").run( + // db.run finalizes its statement; see F3 below for why an unfinalized prepare() breaks cleanup. + db.run( + "INSERT INTO providers (id, app_type, name, settings_config, website_url, notes, meta, sort_index) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8)", "dashscope-compat", "openai", WORD_CANARY, @@ -852,7 +854,8 @@ describe("CC-Switch keyless provider import", () => { "https://api.example.com/v1/%76ioletotter", `https://api.example.com/v1?key=${WORD_CANARY}`, ].entries()) { - db.prepare("INSERT INTO providers (id, app_type, name, settings_config, sort_index) VALUES (?1, ?2, ?3, ?4, ?5)").run( + db.run( + "INSERT INTO providers (id, app_type, name, settings_config, sort_index) VALUES (?1, ?2, ?3, ?4, ?5)", `bad-url-${idx}`, "openai", WORD_CANARY, From 261131d3726d95b4ad3335cf416254a868963e08 Mon Sep 17 00:00:00 2001 From: JUN Date: Fri, 30 Oct 2026 22:14:40 +0900 Subject: [PATCH 34/42] fix(update): refuse dashboard restart under a systemd unit OpenCodex does not own (#7996) * fix(update): refuse dashboard restart under a systemd unit OpenCodex does not own Refuse before package mutation when the worker cannot leave the service cgroup or the proxy is not run by OpenCodex's own user unit, and settle a dead worker's job on status reads. Part of #7967. * fix(update): refuse unsupervised dashboard updates with or without restart The update stops the proxy before replacing the package, so restart:false was not a safe fallback. Detect a systemd service from its cgroup too, accept a delegated child cgroup of the own unit, write the settled status from the compared record, and add hard spawn barriers to the refusal test. * fix(update): report a dead worker's job as failed on status without writing it A status poll that wrote the settled record could overwrite a job another process started between its read and its rename. The poll now returns a failed view; the next run's existing stale recovery settles the persisted record. * fix(update): judge the innermost systemd unit and refuse only an update that would run A desktop terminal runs in a scope below user@.service, which the component check took for a service, and the CI runner's own service preempted the source-checkout refusal. Read the innermost unit of the systemd hierarchy, and consult the supervisor only when an update is available. * fix(update): let the observed cgroup win over an inherited INVOCATION_ID A shell moved into its own systemd-run scope keeps INVOCATION_ID; only a missing cgroup observation falls back to the variable. * docs(structure): state that the observed cgroup unit decides before INVOCATION_ID --------- Co-authored-by: JUN <243035832+lidge-jun@users.noreply.github.com> --- .../content/docs/reference/management-api.md | 2 +- src/server/management/config-routes.ts | 19 +++-- src/server/management/context.ts | 2 + src/update/job.ts | 20 ++++++ src/update/worker-launch.ts | 72 ++++++++++++++++++- structure/gui-and-management-api.md | 2 +- structure/ops/service-and-sidecars.md | 2 +- tests/server/update-async-routes.test.ts | 72 ++++++++++++++++++- tests/update/update-worker-launch.test.ts | 50 ++++++++++++- 9 files changed, 230 insertions(+), 11 deletions(-) diff --git a/docs-site/src/content/docs/reference/management-api.md b/docs-site/src/content/docs/reference/management-api.md index 07b78c54e08..a85f352c5cb 100644 --- a/docs-site/src/content/docs/reference/management-api.md +++ b/docs-site/src/content/docs/reference/management-api.md @@ -278,7 +278,7 @@ See [Codex prompt layers](/guides/codex-prompt/) for the layer model and the key | `GET /api/diagnostics/project-config` | Read cached project configuration warnings | — | | `POST /api/sync` | Sync the current model catalog into Codex | 500 failed sync | | `GET /api/update/check` | Asynchronously check the `latest` or `preview` package channel and refresh the package cache on success | 400 invalid tag | -| `POST /api/update/run` | Asynchronously check a fresh package version, then start an update job, optionally followed by restart | 400 invalid body; job-specific conflict/error status | +| `POST /api/update/run` | Asynchronously check a fresh package version, then start an update job, optionally followed by restart | 400 invalid body; 409 `update_restart_unsupervised` when the proxy runs inside a systemd unit other than OpenCodex's own user service, or the worker cannot leave the service cgroup (with or without restart, since the update stops the proxy first); job-specific conflict/error status | | `GET /api/update/status` | Poll an update job by id | 404 unknown job | | `GET, PUT /api/sidecar-settings` | Read or update web-search and vision sidecar model/backend settings | 400 invalid shape, backend, or limit | | `GET, PUT /api/shadow-call-settings` | Read or update shadow-call interception settings, including `sourceModels`; GET also returns the runtime-owned `sourceModelOptions` inventory | 400 invalid shape or value, including an empty, duplicate, whitespace-bearing, or provider-qualified source prefix, or more than 32 prefixes / 128 characters per prefix; source-list failures carry `code: shadow_call_source_invalid`, target conflicts `code: shadow_call_target_invalid` | diff --git a/src/server/management/config-routes.ts b/src/server/management/config-routes.ts index 144ff32fb38..3283a2409ce 100644 --- a/src/server/management/config-routes.ts +++ b/src/server/management/config-routes.ts @@ -972,11 +972,22 @@ async function handleConfigRoutesImpl(ctx: ManagementContext): Promise checked, spawnWorkerFn: (jobId, runChannel, runRestart) => spawnGuiUpdateWorker(jobId, runChannel, runRestart, { resolveSystemdRun: () => systemdRun }), @@ -990,8 +1001,8 @@ async function handleConfigRoutesImpl(ctx: ManagementContext): Promise Promise; + /** Test seam for the dashboard restart supervisor check (src/update/worker-launch.ts). */ + dashboardRestartRefusal?: typeof import("../../update/worker-launch").dashboardRestartRefusal; remoteWorkspaceHub?: RemoteWorkspaceHubApi; remoteWorkspaceSessions?: RemoteWorkspaceSessionsApi; /** The listener retains and awaits teardown only after this optional subsystem activates. */ diff --git a/src/update/job.ts b/src/update/job.ts index d138bd56485..b5926fceb40 100644 --- a/src/update/job.ts +++ b/src/update/job.ts @@ -527,6 +527,26 @@ export function staleActiveUpdateJobReason( return null; } +/** + * Status view of an active record whose worker is gone (#7967). A worker killed with its service + * cgroup never writes a terminal state, so the persisted record stays `running`. The view reports + * it as `failed` with the reason so the dashboard stops waiting, without writing: the persisted + * record is settled only by the next run's existing stale recovery, which holds the start path, + * so a status poll can never overwrite a job another process started meanwhile. + */ +export function readUpdateJobForStatus( + jobId?: string | null, + now = Date.now(), + isAlive: (pid: number) => boolean = isProcessAlive, +): UpdateJobState | null { + const job = readUpdateJob(jobId); + if (!job) return null; + const reason = staleActiveUpdateJobReason(job, now, isAlive); + if (!reason) return job; + const line = `Update worker stopped before finishing: ${reason}.`; + return { ...job, status: "failed", error: line, exitCode: null, log: [...job.log, line] }; +} + /** * Spawn the GUI update worker without inheriting the proxy's LISTEN socket. * diff --git a/src/update/worker-launch.ts b/src/update/worker-launch.ts index 331e9431da0..24dd7d04b9f 100644 --- a/src/update/worker-launch.ts +++ b/src/update/worker-launch.ts @@ -1,5 +1,5 @@ import { probeSpawn, probeSpawnSync } from "./mutation-guard.mjs"; -import { accessSync, constants, realpathSync, statSync } from "node:fs"; +import { accessSync, constants, readFileSync, realpathSync, statSync } from "node:fs"; import { dirname, isAbsolute } from "node:path"; /** @@ -171,6 +171,76 @@ export function resetSystemdRunProbeForTests(): void { systemdRunProbePending = undefined; } +export interface DashboardRestartContext { + platform?: NodeJS.Platform; + env?: NodeJS.ProcessEnv; + /** The resolved scope launcher, or undefined when none is usable. */ + systemdRun: string | undefined; + /** The OpenCodex-managed user unit (`TASK` in src/service/state.ts) without `.service`. */ + unitName: string; + readCgroup?: () => string; +} + +function cgroupPaths(cgroup: string): string[] { + const rows = cgroup.split("\n").map(line => line.split(":")).filter(parts => parts.length >= 3); + // cgroup v2 ("0::") or the v1 systemd hierarchy name the unit; controller lines are noise. + const unified = rows.filter(parts => parts[1] === "" || parts[1] === "name=systemd"); + return (unified.length > 0 ? unified : rows).map(parts => parts.slice(2).join(":").trim()).filter(Boolean); +} + +/** The innermost systemd unit named by a cgroup path: its last `.service` or `.scope` component. */ +function innermostUnit(path: string): { unit: string; index: number; parts: string[] } | null { + const parts = path.split("/"); + for (let index = parts.length - 1; index >= 0; index--) { + const part = parts[index]!; + if (part.endsWith(".service") || part.endsWith(".scope")) return { unit: part, index, parts }; + } + return null; +} + +/** + * Why a dashboard update must not run against this proxy, or null when it may (#7967). + * + * A dashboard update stops the proxy before it replaces the package, with or without a requested + * restart, and the worker survives that stop only from its own transient scope. Only OpenCodex's + * own user unit is restarted by the updater. A proxy inside any systemd service -- detected from + * `INVOCATION_ID` or from a `.service` cgroup, so a unit that unsets the variable is still seen + * -- is therefore refused unless the scope launcher works and the cgroup places it in + * `.service` under a per-user manager (a delegated child cgroup below it is fine). A + * custom system unit with `User=` would lose the worker with its cgroup, and the restart would aim + * at an unrelated, possibly inactive user unit. Outside systemd nothing changes. + */ +export function dashboardRestartRefusal(context: DashboardRestartContext): string | null { + const platform = context.platform ?? process.platform; + const env = context.env ?? process.env; + if (platform !== "linux") return null; + let cgroup: string | null; + try { cgroup = (context.readCgroup ?? (() => readFileSync("/proc/self/cgroup", "utf8")))(); } + catch { cgroup = null; } + const paths = cgroup === null ? [] : cgroupPaths(cgroup); + // The innermost unit decides: a terminal or app scope below the per-user manager + // (user@.service/app.slice/vte-spawn-1.scope) is not a service; a system or user service is. + const units: ReturnType[] = paths.map(innermostUnit); + const observed = units.filter((found): found is NonNullable => found !== null); + // The observed cgroup wins over an inherited INVOCATION_ID (a shell moved into its own scope + // keeps the variable); without an observation the variable alone fails closed. + const inService = observed.length > 0 + ? observed.some(found => found.unit.endsWith(".service") && !found.unit.startsWith("user@")) + : Boolean(env.INVOCATION_ID); + if (!inService) return null; + const manual = "Stop the service with its own manager, update the package from a shell (for example `ocx update`), then start the service again."; + if (!context.systemdRun || !env.INVOCATION_ID) { + return `Dashboard update is unavailable: this proxy runs inside a systemd service, and the update worker cannot leave the service's cgroup (no usable \`systemd-run --user --scope\`), so stopping the proxy for the update would stop the update too. ${manual}`; + } + const ownUnit = `${context.unitName}.service`; + const managedByOwnUnit = units.some((found): boolean => found !== null && found.unit === ownUnit + && found.parts.slice(0, found.index).some((part: string) => part.startsWith("user@") && part.endsWith(".service"))); + if (!managedByOwnUnit) { + return `Dashboard update is unavailable: this proxy runs under a systemd unit that is not OpenCodex's own \`${context.unitName}.service\` user unit, so the updater can neither keep its worker alive nor restart that unit safely. ${manual}`; + } + return null; +} + export function guiUpdateWorkerCommand( execPath: string, args: readonly string[], diff --git a/structure/gui-and-management-api.md b/structure/gui-and-management-api.md index bbdb03ccf55..0cafa760adf 100644 --- a/structure/gui-and-management-api.md +++ b/structure/gui-and-management-api.md @@ -274,7 +274,7 @@ per-request first-party callback reads that live object; a failed write leaves i | Config/settings | Reasoning retention follows the [panel and settings contract](dashboard-and-usage.md). Read safe config/settings views; mutate supported settings only. Full `PUT /api/config` is disabled so masked secrets are not round-tripped. `PUT /api/settings` accepts `codexAutoStart`, `streamMode`, integer `appOwnedMemoryBudgetMb` (64..4096), strict boolean `codexAccountPickerEnabled`, strict boolean `fastRows`, and a validated per-account `codexQuotaAutoRefresh` toggle (each optional, at least one required). `fastRows` defaults on when absent: false is persisted, true deletes the key, and successful writes echo the effective boolean. An effective change converges the Codex catalog and refreshes enabled or already-owned client integrations after persistence. Picker enable initializes an empty UI-managed selector map, persists before one bounded catalog convergence, and reports only `catalogRefreshPending`; allocation/save failure restores every touched live field and skips convergence. Budget changes synchronously enforce the process-wide evictable retained-state cap; this is separate from RSS/native memory. `streamMode` persists the #314 stream-shape selection in config.json (Windows services need persisted input; macOS eager relay is explicit-only). | | Startup safety | `GET /api/startup-health` reports whether injected Codex routing is restart-safe, with secret-free service/shim diagnostics. `POST /api/startup-action` provides allowlisted one-click installation for the background service or launcher shim. On Windows a healthy script shim is CLI-only; Codex Desktop requires the background service for full protection. Optional Codex compatibility uses the [confirmed local runtime](clients/codex-desktop.md#optional-compatibility-runtime), [machine-local dashboard consent](clients/codex-desktop.md#dashboard-controls), and [revision-bound next-start preference with shared awaited teardown](clients/codex-desktop.md#proxy-startup-preference); saving that preference performs no runtime action. | | Windows tray | `GET/POST /api/windows-tray` controls an owned, per-user HKCU login tray. The tray delegates fixed actions to the CLI and is never a proxy supervisor or restart-protection signal. | -| Updates | `GET /api/update/check`, `POST /api/update/run`, and `GET /api/update/status` own dashboard self-update state. A launched worker PID is persisted in `update-job.json`; dead PIDs recover immediately, while legacy active records without a PID recover only after ten minutes. Live PIDs remain exclusive regardless of record age. `GET /api/update/check` and `POST /api/update/run` await one per-channel asynchronous registry lookup and write successful results through to the package cache; run passes that result to the job starter. `GET /api/update/badge` only reads the cache and reports unknown after 40 hours, on missing cache, or on channel mismatch. The badge links to the update surface rather than gating other actions. `GET /api/update/badge?surface=desktop&session=` projects only that process-local Tauri session; missing or expired state is unknown and never falls back to the package cache. `POST /api/update/desktop-snapshot` is a 1 KiB bounded display-state mutation with no install permission. It accepts the existing admin-token principal or a dedicated single-use, ten-second snapshot capability bound to nonce, method, path, PID, port and the SHA-256 digest of the exact body bytes; that capability cannot authorize another route. | +| Updates | `GET /api/update/check`, `POST /api/update/run`, and `GET /api/update/status` own dashboard self-update state. A launched worker PID is persisted in `update-job.json`; dead PIDs recover immediately when a new run starts, and `GET /api/update/status` already reports such a record as `failed` without writing it (`readUpdateJobForStatus`), while legacy active records without a PID recover only after ten minutes. Live PIDs remain exclusive regardless of record age. `GET /api/update/check` and `POST /api/update/run` await one per-channel asynchronous registry lookup and write successful results through to the package cache; run passes that result to the job starter. `GET /api/update/badge` only reads the cache and reports unknown after 40 hours, on missing cache, or on channel mismatch. The badge links to the update surface rather than gating other actions. `GET /api/update/badge?surface=desktop&session=` projects only that process-local Tauri session; missing or expired state is unknown and never falls back to the package cache. `POST /api/update/desktop-snapshot` is a 1 KiB bounded display-state mutation with no install permission. It accepts the existing admin-token principal or a dedicated single-use, ten-second snapshot capability bound to nonce, method, path, PID, port and the SHA-256 digest of the exact body bytes; that capability cannot authorize another route. | | Providers | Create/update/delete ordinary provider configs and enrich registry metadata. A `POST /api/providers` overwrite of an existing name keeps the ten operator compatibility settings (`PROVIDER_COMPAT_CARRY_FIELDS` in `src/server/management/provider-overwrite-carry.ts`) and the stored key pool only while the destination (adapter, normalized base URL, auth mode when named) is unchanged. It keeps omitted `hideRawReasoning` on every overwrite of the same name, including destination moves, because it is a display policy; a submitted boolean wins. An omitted `inferContextWindow` likewise keeps the latest live same-name state after asynchronous validation, including `false` or a concurrent clear; a submitted boolean wins. It never merges the rest of the old row. `PATCH` is a field mask and keeps every field it does not name; `hideRawReasoning`, `preserveResponsesReasoningContent`, `promoteAdditionalTools` and `inferContextWindow` accept a boolean or `null` to clear them, rejecting other types. `GET /api/providers` exposes configured `inferContextWindow` as an optional boolean without substituting a default; its [custom-only, default-off inference policy](catalog-vendor-context.md) is unchanged. Custom Responses provider forms expose `preserveResponsesReasoningContent` as a default-off plaintext replay switch; unrelated settings saves preserve omission and curated/OpenAI destinations keep their existing defaults. The reserved `openai` card exposes Pool(default)/Direct account mode; `openai-apikey` remains the separate API route. | | Models | Fetch routed model lists, disabled model visibility, and catalog-facing ids. New non-OAuth registration holds exposure until authoritative discovery; 20 or more distinct switch rows start OFF without disabling the provider. Pending rows cannot accept visibility changes. Ordinary model reads also apply the [new-arrival catalog contract](catalog.md#shared-catalog) before returning models. With a matching persisted configuration, discovery records the baseline, NEW badges, and automatic disables together; HTTP listing does not wait for a separate Codex sync. Inventory already drifted from disk uses a detached policy projection: arrivals remain disabled in management rows without changing live config or disk. | | OAuth | Login/status/logout for OAuth-backed providers, plus multiauth account management: `GET /api/oauth/accounts`, `PUT /api/oauth/accounts/active`, `PUT /api/oauth/accounts/alias`, `PUT /api/oauth/accounts/pause`, and `DELETE /api/oauth/accounts` list masked accounts per provider, switch the active one, edit its display-only alias, pause/resume a generic OAuth account, and remove one. Paused generic OAuth accounts are excluded from request selection, 429 failover, and proactive token refresh; pausing an active account selects the next usable account when available, and resuming an account restores an active selection if the current one remains paused. With no unpaused account, requests return 403 rather than a login error. Kiro account-list rows include the current automatic-selection projection and closed exclusion reason; an active singleton may still send. The login flow itself is `GET /api/oauth/providers`, `POST /api/oauth/login`, `POST /api/oauth/login/code`, `POST /api/oauth/login/cancel`, `POST /api/oauth/logout`, and `GET /api/oauth/status`; pool controls are `GET/PUT/PATCH /api/oauth/accounts/pool` and `POST /api/oauth/accounts/clear-cooldown`. Login accepts `addAccount: true` to force a fresh browser identity. Meta Muse login start and manual-code continuation require the server-resolved `gui-session` principal before credential acquisition or code submission (including reauth); see the [provider contract](providers-and-adapters.md). Device flows return a structured `deviceCode`; the GUI highlights and copies it before the user opens the verification page. Google Antigravity account rows in `gui/src/components/provider-workspace/ProviderAuthPanel.tsx` display the account-summary `plan` as a badge and include it in the account button’s accessible name. The value is the canonical upstream tier ID (for example `free-tier` or `gemini-pro`), preserved literally across locales rather than inferred as a product name. Absent plans render no badge; pause and reauthentication controls keep their existing behavior. | diff --git a/structure/ops/service-and-sidecars.md b/structure/ops/service-and-sidecars.md index 00955b3e463..fc84ee93356 100644 --- a/structure/ops/service-and-sidecars.md +++ b/structure/ops/service-and-sidecars.md @@ -597,4 +597,4 @@ The desktop badge snapshot in src/update/desktop-badge.ts is process-local displ Desktop startup diagnostics use `src/service/desktop-startup.ts` to read the ownership record, the login registration (macOS: launchd; Linux: the `~/.config/autostart/OpenCodex.desktop` entry that auto-launch writes, credited only when `XDG_CONFIG_HOME` is unset or `~/.config`; it must launch an unquoted absolute `opencodex-desktop` with exactly `--autostart`, without conditional keys, and not be hidden or disabled) and exact parent/child executable paths (Linux: `/proc//exe` and the parent pid from `/proc//stat`) without mutating them. A durable desktop claim survives a failed identity or supervision check; its protection requires fresh matching identity, enabled login registration, and live supervision. Only ownership `none` enters the unowned branch: verified supervision adds `supervisor: { supervisorPid, runtimePid, app }` with `owned: false`, and protection requires that same app's verified login registration and a fresh, running, viable diagnostic. Unknown ownership and CLI-owned claims receive no supervision credit. The independent process evidence requires two matching snapshots and correlation of the target, PID-file and runtime-port PIDs, as described in [desktop supervision](desktop-supervision.md#runtime-ownership-from-the-apps-side); it never becomes ownership. An unloaded macOS job receives credit only from two matching plist/target/enabled snapshots; a boolean plist Disabled default combines with explicit launchctl overrides, and malformed or changing evidence refuses credit. On macOS, ownership and PID are re-read before crediting an owned result; on Linux, the complete owned evidence chain is read twice and both reads must agree. The startup-health subprocess uses `selfLaunchArgv` to support both source and compiled entrypoints. -On Linux, a dashboard update worker started from the systemd user service is launched through an executable regular file at a trusted absolute path — `/usr/bin/systemd-run`, `/bin/systemd-run`, `/usr/local/bin/systemd-run` (local installs), or `/run/current-system/sw/bin/systemd-run` (the NixOS layout) — with `--user --scope --quiet --collect` (`src/update/worker-launch.ts`), so it leaves the service cgroup before the updater stops `opencodex-proxy.service`; the default `KillMode=control-group` otherwise kills it with the proxy (#5750). The inherited `PATH` is never searched, and each candidate's resolved target — plus every ancestor directory able to substitute it — must be root-owned and not group/world-writable: a trusted-path symlink into a user-replaceable directory is skipped, as is a group-writable `/usr/local/bin`, rather than exec'd under the service account. Candidates are tried in order and a path whose no-op scope probe fails falls through to the next trusted path; the probe applies only when `INVOCATION_ID` is set, and every other case keeps the plain detached spawn. The management route resolves the launcher with `resolveSystemdRunAsync` before spawning, so first-request probing overlaps other work instead of blocking the event loop for up to twenty seconds. `--scope` moves `systemd-run` itself into the scope and then execs the worker, so the recorded PID is the worker's (`tests/update/update-worker-launch.test.ts`). +On Linux, a dashboard update worker started from the systemd user service is launched through an executable regular file at a trusted absolute path — `/usr/bin/systemd-run`, `/bin/systemd-run`, `/usr/local/bin/systemd-run` (local installs), or `/run/current-system/sw/bin/systemd-run` (the NixOS layout) — with `--user --scope --quiet --collect` (`src/update/worker-launch.ts`), so it leaves the service cgroup before the updater stops `opencodex-proxy.service`; the default `KillMode=control-group` otherwise kills it with the proxy (#5750). The inherited `PATH` is never searched, and each candidate's resolved target — plus every ancestor directory able to substitute it — must be root-owned and not group/world-writable: a trusted-path symlink into a user-replaceable directory is skipped, as is a group-writable `/usr/local/bin`, rather than exec'd under the service account. Candidates are tried in order and a path whose no-op scope probe fails falls through to the next trusted path; the probe applies only when `INVOCATION_ID` is set, and every other case keeps the plain detached spawn. The management route resolves the launcher with `resolveSystemdRunAsync` before spawning, so first-request probing overlaps other work instead of blocking the event loop for up to twenty seconds. `--scope` moves `systemd-run` itself into the scope and then execs the worker, so the recorded PID is the worker's (`tests/update/update-worker-launch.test.ts`). A dashboard update that would run, with or without restart, refuses before any package mutation when the proxy runs inside a systemd service: the innermost `.service` or `.scope` unit of `/proc/self/cgroup` decides (a `.service` other than the per-user manager is a service, so a unit that unsets `INVOCATION_ID` is still seen; a terminal or app `.scope` is not, even with an inherited `INVOCATION_ID`), and only when no cgroup can be read does `INVOCATION_ID` alone count, failing closed unless a trusted scope launcher works and the cgroup places the proxy in OpenCodex's own `opencodex-proxy.service` below a `user@.service` manager: the update stops the proxy before replacing the package, so a worker left in the service cgroup dies with it, and a custom system unit with `User=` is not one the updater can restart, nor may an installed but inactive user unit become the restart target (`dashboardRestartRefusal`, 409 `update_restart_unsupervised`, #7967). diff --git a/tests/server/update-async-routes.test.ts b/tests/server/update-async-routes.test.ts index e7318f43fba..05cbc8fe616 100644 --- a/tests/server/update-async-routes.test.ts +++ b/tests/server/update-async-routes.test.ts @@ -1,7 +1,13 @@ -import { describe, expect, test } from "bun:test"; +import { describe, expect, spyOn, test } from "bun:test"; import { handleManagementAPI } from "../../src/server/management-api"; import type { ManagementApiDeps } from "../../src/server/management/context"; import type { OcxConfig } from "../../src/types"; +import { existsSync, mkdirSync, mkdtempSync, readFileSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import * as updateJobModule from "../../src/update/job"; +import { updateJobPath } from "../../src/update/job"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; const config = { port: 10100, defaultProvider: "openai", providers: {} } as OcxConfig; @@ -57,9 +63,71 @@ describe("asynchronous package update routes", () => { }); test("source checkout run rejects a worker", async () => { - const response = await call("POST", "/api/update/run", { tag: "latest", restart: false }); + // The supervisor check must not preempt the checkout's own refusal, even when the test + // process itself runs inside a systemd service (a CI runner, for example). + const response = await call("POST", "/api/update/run", { tag: "latest", restart: false }, { + dashboardRestartRefusal: () => "must not be consulted", + }); expect(response?.status).toBe(409); const body = await response!.json() as Record; expect(body.code).toBe("source_checkout"); }); }); + +describe("dashboard update restart supervision (#7967)", () => { + const available = async (channel: "latest" | "preview") => ({ + currentVersion: "2.7.43", latestVersion: "2.7.44", channel, + installer: "npm" as const, updateAvailable: true, canUpdate: true, + command: "npm install -g @bitkyc08/opencodex@2.7.44", releaseNotesUrl: "https://example.test/releases", + }); + + test.each([true, false])("a refused supervisor answers 409 before any update job starts (restart=%p)", async restart => { + const home = mkdtempSync(join(tmpdir(), "ocx-update-supervisor-")); + const previous = process.env.OPENCODEX_HOME; + process.env.OPENCODEX_HOME = home; + // Hard barriers: a regression must fail here, never launch a real updater. + const start = spyOn(updateJobModule, "startUpdateJob").mockImplementation(() => { throw new Error("must not start a job"); }); + const spawn = spyOn(updateJobModule, "spawnGuiUpdateWorker").mockImplementation(() => { throw new Error("must not spawn a worker"); }); + try { + const response = await call("POST", "/api/update/run", { tag: "latest", restart }, { + checkPackageUpdate: available, + dashboardRestartRefusal: () => "Dashboard update is unavailable: fixture", + }); + expect(response?.status).toBe(409); + expect(await response!.json()).toEqual({ error: "Dashboard update is unavailable: fixture", code: "update_restart_unsupervised" }); + expect(start).not.toHaveBeenCalled(); + expect(spawn).not.toHaveBeenCalled(); + expect(existsSync(updateJobPath())).toBe(false); + } finally { + start.mockRestore(); spawn.mockRestore(); + if (previous === undefined) delete process.env.OPENCODEX_HOME; else process.env.OPENCODEX_HOME = previous; + removeTreeWithRetry(home); + } + }); + + test("status reports an active job whose worker died as failed, without writing", async () => { + const home = mkdtempSync(join(tmpdir(), "ocx-update-status-")); + const previous = process.env.OPENCODEX_HOME; + process.env.OPENCODEX_HOME = home; + try { + const now = new Date().toISOString(); + mkdirSync(dirname(updateJobPath()), { recursive: true }); + const persisted = JSON.stringify({ + id: "dead-worker", status: "restarting", startedAt: now, updatedAt: now, pid: 2147483646, + currentVersion: "2.7.43", latestVersion: "2.7.44", channel: "latest", installer: "npm", + restart: true, command: "", releaseNotesUrl: "", log: [], + }); + writeFileSync(updateJobPath(), persisted); + const response = await call("GET", "/api/update/status?jobId=dead-worker"); + expect(response?.status).toBe(200); + const body = await response!.json() as { job: { status: string; error?: string } }; + expect(body.job.status).toBe("failed"); + expect(body.job.error).toContain("is no longer running"); + // The next run settles the record under its own start path; a poll never writes it. + expect(readFileSync(updateJobPath(), "utf8")).toBe(persisted); + } finally { + if (previous === undefined) delete process.env.OPENCODEX_HOME; else process.env.OPENCODEX_HOME = previous; + removeTreeWithRetry(home); + } + }); +}); diff --git a/tests/update/update-worker-launch.test.ts b/tests/update/update-worker-launch.test.ts index 661c4fea581..3ca852cd5af 100644 --- a/tests/update/update-worker-launch.test.ts +++ b/tests/update/update-worker-launch.test.ts @@ -4,7 +4,7 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { guiUpdateWorkerCommand, isTrustedSystemdRunFile, resolveSystemdRun, resetSystemdRunProbeForTests, - resolveSystemdRunAsync, SYSTEMD_SCOPE_ARGS, + resolveSystemdRunAsync, SYSTEMD_SCOPE_ARGS, dashboardRestartRefusal, } from "../../src/update/worker-launch"; import { removeTreeWithRetry } from "../helpers/remove-tree"; @@ -237,3 +237,51 @@ test("launcher trust also checks lexical ancestors of a canonical system target" expect(isTrustedSystemdRunFile(candidate, deps("/nix/store"))).toBe(false); expect(isTrustedSystemdRunFile("relative/systemd-run", deps(undefined))).toBe(false); }); + +// #7967: a dashboard restart must not stop a proxy whose worker cannot survive the stop, or one +// run by a unit the updater does not restart. +describe("dashboard restart supervisor check", () => { + // The per-user manager unit, joined so the literal does not read as an e-mail address. + const manager = ["user", "1000.service"].join("@"); + const userUnit = `0::/user.slice/user-1000.slice/${manager}/app.slice/opencodex-proxy.service\n`; + const base = { platform: "linux" as const, env: { INVOCATION_ID: "abc" }, systemdRun: "/usr/bin/systemd-run", unitName: "opencodex-proxy" }; + + test("OpenCodex's own user unit with a usable scope may update", () => { + expect(dashboardRestartRefusal({ ...base, readCgroup: () => userUnit })).toBeNull(); + // cgroup v1 lists the systemd hierarchy beside controller lines. + expect(dashboardRestartRefusal({ ...base, readCgroup: () => `4:memory:/user.slice\n1:name=systemd:/user.slice/user-1000.slice/${manager}/app.slice/opencodex-proxy.service\n` })).toBeNull(); + // A delegated child cgroup below the unit is still that unit. + expect(dashboardRestartRefusal({ ...base, readCgroup: () => `0::/user.slice/user-1000.slice/${manager}/app.slice/opencodex-proxy.service/payload\n` })).toBeNull(); + }); + + test("a custom system unit is refused even when it reuses the unit name", () => { + for (const cgroup of [ + "0::/system.slice/opencodex.service\n", + "0::/system.slice/opencodex-proxy.service\n", + `0::/user.slice/user-1000.slice/${manager}/app.slice/other.service\n`, + `0::/user.slice/user-1000.slice/${manager}/app.slice/not-opencodex-proxy.service\n`, + "", + ]) { + expect(dashboardRestartRefusal({ ...base, readCgroup: () => cgroup })).toContain("not OpenCodex's own"); + } + expect(dashboardRestartRefusal({ ...base, readCgroup: () => { throw new Error("no proc"); } })).toContain("not OpenCodex's own"); + }); + + test("no usable scope launcher under systemd is refused", () => { + expect(dashboardRestartRefusal({ ...base, systemdRun: undefined, readCgroup: () => userUnit })).toContain("cannot leave the service's cgroup"); + // A unit that unsets INVOCATION_ID is still found from its .service cgroup. + expect(dashboardRestartRefusal({ ...base, env: {}, readCgroup: () => "0::/system.slice/opencodex.service\n" })).toContain("cannot leave the service's cgroup"); + }); + + test("outside systemd or off Linux nothing changes", () => { + const unread = () => { throw new Error("must not read the cgroup"); }; + expect(dashboardRestartRefusal({ ...base, env: {}, systemdRun: undefined, readCgroup: () => "0::/user.slice/user-1000.slice/session-3.scope\n" })).toBeNull(); + // A desktop terminal runs in a scope below the per-user manager: not a service. + expect(dashboardRestartRefusal({ ...base, env: {}, systemdRun: undefined, readCgroup: () => `0::/user.slice/user-1000.slice/${manager}/app.slice/app-org.gnome.Terminal.slice/vte-spawn-1.scope\n` })).toBeNull(); + expect(dashboardRestartRefusal({ ...base, env: {}, systemdRun: undefined, readCgroup: () => `0::/user.slice/user-1000.slice/${manager}/init.scope\n` })).toBeNull(); + // A shell moved into its own scope keeps an inherited INVOCATION_ID; the cgroup decides. + expect(dashboardRestartRefusal({ ...base, readCgroup: () => `0::/user.slice/user-1000.slice/${manager}/app.slice/run-r1.scope\n` })).toBeNull(); + expect(dashboardRestartRefusal({ ...base, env: {}, systemdRun: undefined, readCgroup: () => { throw new Error("no proc"); } })).toBeNull(); + expect(dashboardRestartRefusal({ ...base, platform: "darwin", systemdRun: undefined, readCgroup: unread })).toBeNull(); + }); +}); From 19ab113945530483f4524ae3e6f8374d16572976 Mon Sep 17 00:00:00 2001 From: JUN Date: Fri, 30 Oct 2026 22:28:10 +0900 Subject: [PATCH 35/42] fix(windows): admit config-root cache writes after the optional config-root ACL flight (#7998) * fix(windows): admit config-root cache writes after the optional config-root ACL flight The provider and Codex account quota caches, the reasoning support cache and the models.dev refresh wrote under the config root while an optional config-root ACL flight could still run, so parent inheritance could race the fresh file's explicit grant. Each write now goes through withConfigDirHardeningSettled with its snapshot built inside the callback. Part of #7815. * fix(windows): invalidate waiting cache writes on cancel and report refused refreshes to joiners A fired cache write waiting for config-root ACL admission now checks a generation that cancel, clear and reset bump; the reasoning support flush seam is admitted too; a joined models.dev refresh receives the shared outcome instead of a coalesced success. * fix(windows): drop a reasoning support flush invalidated while it waited for admission --------- Co-authored-by: JUN <243035832+lidge-jun@users.noreply.github.com> --- src/codex/quota.ts | 21 ++++-- src/providers/account-quota-disk.ts | 19 ++++-- src/providers/reasoning-metadata.ts | 46 ++++++++----- structure/config.md | 2 +- .../main-quota-provenance.test.ts | 46 ++++++------- .../reasoning-metadata.test.ts | 62 +++++++++++++++++- ...provider-account-quota-persistence.test.ts | 65 +++++++++++++++++++ 7 files changed, 214 insertions(+), 47 deletions(-) diff --git a/src/codex/quota.ts b/src/codex/quota.ts index 744e8127d7c..b0933c46231 100644 --- a/src/codex/quota.ts +++ b/src/codex/quota.ts @@ -5,6 +5,7 @@ import { forgetMainAccountUsage, observeMainAccountUsage, type FreshWindow } fro import { closeSync, constants as fsConstants, existsSync, fstatSync, openSync, readSync, unlinkSync } from "node:fs"; import { join } from "node:path"; import { atomicWriteFile, getConfigDir } from "../config"; +import { withConfigDirHardeningSettled } from "../config/paths"; import { captureConfigGeneration, type GenerationContext } from "../lib/state-store-sweeper"; import { isThirtyDayOnlyCodexPlan } from "./plan"; import { stampCodexQuotaUsageObservation } from "./quota-observation-freshness"; @@ -42,6 +43,12 @@ let mainPolicyQuota: MainPolicyQuota | null = null; let diskHydrated = false; let persistTimer: ReturnType | null = null; let persistStartedAt: number | null = null; +/** The latest admitted snapshot write; settles even when admission times out. */ +let persistWrite: Promise = Promise.resolve(); +/** Bumped by a full clear so a fired callback still waiting for ACL admission writes nothing. */ +let persistGeneration = 0; +/** Test seam: wait for a fired persistence callback's admitted write. */ +export function settleQuotaPersistenceForTests(): Promise { return persistWrite; } const MONTHLY_WINDOW_MIN_SECONDS = 28 * 24 * 60 * 60; /** @@ -800,7 +807,12 @@ function schedulePersistAccountQuotas(): void { persistTimer = setTimeout(() => { persistTimer = null; persistStartedAt = null; - try { + // Admitted after an overlapping optional config-root ACL flight (#7815); the snapshot is + // taken inside the callback, and a timed-out admission skips this best-effort write. + const dir = getConfigDir(); + const generation = persistGeneration; + persistWrite = withConfigDirHardeningSettled(dir, () => { + if (generation !== persistGeneration) return; const quotas: Record = {}; for (const [accountId, quota] of accountQuota.entries()) { quotas[accountId] = quota; @@ -811,10 +823,10 @@ function schedulePersistAccountQuotas(): void { ...(mainPolicyQuota ? { mainPolicyQuota } : {}), history: quotaHistory.serialize(), }; - atomicWriteFile(join(getConfigDir(), QUOTA_CACHE_FILENAME), `${JSON.stringify(body)}\n`); - } catch { + atomicWriteFile(join(dir, QUOTA_CACHE_FILENAME), `${JSON.stringify(body)}\n`); + }).catch(() => { // Best-effort persistence only. - } + }); }, delay); } @@ -890,6 +902,7 @@ export function clearAccountQuota(accountId?: string): void { persistTimer = null; } persistStartedAt = null; + persistGeneration += 1; try { const path = join(getConfigDir(), QUOTA_CACHE_FILENAME); if (existsSync(path)) unlinkSync(path); diff --git a/src/providers/account-quota-disk.ts b/src/providers/account-quota-disk.ts index bf0aa1e5160..bf6593f7a98 100644 --- a/src/providers/account-quota-disk.ts +++ b/src/providers/account-quota-disk.ts @@ -13,6 +13,7 @@ import { existsSync, readFileSync } from "node:fs"; import { join } from "node:path"; import { atomicWriteFile, getConfigDir } from "../config"; +import { withConfigDirHardeningSettled } from "../config/paths"; import type { ProviderQuota } from "./quota-types"; import type { KiroPersistedQuota, KiroPersistedVerdict } from "./kiro-account-state-disk"; import { ACCOUNT_QUOTA_TTL_MS } from "./quota-wire"; @@ -41,6 +42,8 @@ type DiskFile = { let persistTimer: ReturnType | null = null; let persistStartedAt: number | null = null; +/** Bumped by cancellation so a fired callback still waiting for ACL admission writes nothing. */ +let persistGeneration = 0; /** Read the snapshot. Returns an empty map for a missing, corrupt or stale file. */ export function readPersistedAccountQuotas(now = Date.now()): Map { @@ -102,7 +105,14 @@ export function schedulePersistAccountQuotas( persistTimer = setTimeout(() => { persistTimer = null; persistStartedAt = null; - try { + // Writes under the config root wait out an overlapping optional config-root ACL flight, + // whose inheritance can drop the fresh file's explicit grant (#7815). Rows are read inside + // the admitted callback so a late admission still writes the newest snapshot; a timeout + // skips this best-effort write. + const dir = getConfigDir(); + const generation = persistGeneration; + void withConfigDirHardeningSettled(dir, () => { + if (generation !== persistGeneration) return; const out: Record = {}; for (const [key, quota] of rows()) out[key] = quota; const kiroVerdicts: Record = {}; @@ -110,10 +120,10 @@ export function schedulePersistAccountQuotas( if (key.startsWith("kiro\0") && /^[a-f0-9]{64}$/.test(verdict.identity)) kiroVerdicts[key] = verdict; } const body: DiskFile = { version: 1, rows: out, kiroVerdicts }; - atomicWriteFile(join(getConfigDir(), FILENAME), `${JSON.stringify(body)}\n`); - } catch { + atomicWriteFile(join(dir, FILENAME), `${JSON.stringify(body)}\n`); + }).catch(() => { // Best-effort persistence only. - } + }); }, delay); } @@ -122,4 +132,5 @@ export function cancelPendingAccountQuotaPersist(): void { if (persistTimer) clearTimeout(persistTimer); persistTimer = null; persistStartedAt = null; + persistGeneration += 1; } diff --git a/src/providers/reasoning-metadata.ts b/src/providers/reasoning-metadata.ts index 45f83ebc65c..a58a93889ed 100644 --- a/src/providers/reasoning-metadata.ts +++ b/src/providers/reasoning-metadata.ts @@ -25,7 +25,7 @@ import { join } from "node:path"; // already imports. Going through the ../config barrel closes a cycle back into account-namespaces.ts // and leaves COMBO_NAMESPACE in its temporal dead zone for entry points that start at combos/types.ts. import { atomicWriteFile } from "../config/atomic-write"; -import { getConfigDir } from "../config/paths"; +import { getConfigDir, withConfigDirHardeningSettled } from "../config/paths"; import type { OcxProviderConfig } from "../types"; import { resolveProviderApiKey } from "./api-key-resolve"; @@ -96,7 +96,9 @@ interface SupportSnapshot { let snapshotMemo: MetadataSnapshot | null | undefined; let supportMemo: Map | undefined; let persistTimer: ReturnType | null = null; -let refreshInFlight: Promise | null = null; +let refreshInFlight: Promise | null = null; +/** Bumped by reset so a fired support write still waiting for ACL admission writes nothing. */ +let persistGeneration = 0; /** Test seam: drop the memoised snapshot/support caches so a suite can drive the load paths. */ export function resetReasoningMetadataCachesForTests(): void { @@ -107,6 +109,7 @@ export function resetReasoningMetadataCachesForTests(): void { persistTimer = null; } refreshInFlight = null; + persistGeneration += 1; } function readJsonFile(filename: string): T | null { @@ -375,7 +378,12 @@ export function recordUnsupportedReasoningEffort( if (persistTimer) clearTimeout(persistTimer); persistTimer = setTimeout(() => { persistTimer = null; - try { + // Admitted after an overlapping optional config-root ACL flight (#7815); rows are read + // inside the callback, and a timed-out admission skips this best-effort write. + const dir = getConfigDir(); + const generation = persistGeneration; + void withConfigDirHardeningSettled(dir, () => { + if (generation !== persistGeneration) return; const out: SupportSnapshot["rows"] = {}; for (const [rowKey, at] of rows) { const evidenceText = supportEvidence.get(rowKey); @@ -385,30 +393,33 @@ export function recordUnsupportedReasoningEffort( ...(evidenceText ? { evidence: evidenceText } : {}), }; } - atomicWriteFile(join(getConfigDir(), SUPPORT_FILENAME), JSON.stringify({ version: 2, rows: out }) + "\n"); - } catch { + atomicWriteFile(join(dir, SUPPORT_FILENAME), JSON.stringify({ version: 2, rows: out }) + "\n"); + }).catch(() => { // Best-effort persistence only. - } + }); }, PERSIST_DEBOUNCE_MS); return true; } -/** Test seam: flush a pending support write so a script sees the snapshot immediately. */ -export function flushReasoningSupportCache(): void { +/** Test seam: flush a pending support write, admitted like the timer write (#7815). */ +export async function flushReasoningSupportCache(): Promise { if (!persistTimer) return; clearTimeout(persistTimer); persistTimer = null; - try { + const dir = getConfigDir(); + const generation = persistGeneration; + await withConfigDirHardeningSettled(dir, () => { + if (generation !== persistGeneration) return; const rows = loadSupport(); const out: SupportSnapshot["rows"] = {}; for (const [rowKey, at] of rows) { const evidenceText = supportEvidence.get(rowKey); out[rowKey] = { effort: JSON.parse(rowKey)[3] ?? "", at, ...(evidenceText ? { evidence: evidenceText } : {}) }; } - atomicWriteFile(join(getConfigDir(), SUPPORT_FILENAME), JSON.stringify({ version: 2, rows: out }) + "\n"); - } catch { + atomicWriteFile(join(dir, SUPPORT_FILENAME), JSON.stringify({ version: 2, rows: out }) + "\n"); + }).catch(() => { // Best-effort persistence only. - } + }); } /** @@ -521,7 +532,8 @@ export async function refreshReasoningMetadata(options: { force?: boolean; waitM return { ok: true, reason: "fresh" }; } if (refreshInFlight) { - return waitForRefresh(refreshInFlight.then(() => ({ ok: true, reason: "coalesced" })), options.waitMs); + // A joined caller learns the shared outcome: a refused or failed refresh is not a success. + return waitForRefresh(refreshInFlight.then(outcome => outcome.ok ? { ok: true, reason: "coalesced" } : outcome), options.waitMs); } const job = (async () => { const response = await fetch(SOURCE_URL, { @@ -565,15 +577,19 @@ export async function refreshReasoningMetadata(options: { force?: boolean; waitM providers[providerKey] = out; } const next: MetadataSnapshot = { version: 2, fetchedAt: Date.now(), source: SOURCE_URL, providers, apis }; - atomicWriteFile(join(getConfigDir(), FILENAME), JSON.stringify(next) + "\n"); + // The last await is the network read; publication waits out an overlapping optional + // config-root ACL flight (#7815) and a timed-out admission fails this refresh unpublished. + const dir = getConfigDir(); + await withConfigDirHardeningSettled(dir, () => atomicWriteFile(join(dir, FILENAME), JSON.stringify(next) + "\n")); snapshotMemo = next; return { ok: true, reason: "refreshed", providers: Object.keys(providers).length, models }; })(); - refreshInFlight = job.catch(() => undefined).finally(() => { refreshInFlight = null; }); const settled = job.catch((error): RefreshOutcome => ({ ok: false, reason: error instanceof Error ? error.message : String(error), })); + const shared = settled.finally(() => { if (refreshInFlight === shared) refreshInFlight = null; }); + refreshInFlight = shared; return waitForRefresh(settled, options.waitMs); } diff --git a/structure/config.md b/structure/config.md index 7f6345e6529..5e33e304171 100644 --- a/structure/config.md +++ b/structure/config.md @@ -62,7 +62,7 @@ test seam and the mutation types remain re-exported through `src/config.ts`. Suc `src/config/paths.ts` owns `OPENCODEX_HOME` expansion and resolution, the config directory and `config.json` path. A relative home is resolved once per distinct raw environment value, so a later working-directory change cannot silently move the active installation. Config reads in this leaf and `src/config/load-degrade.ts` use NTFS ACLs on Windows and private POSIX modes elsewhere. Optional directory hardening follows directory symlinks/junctions but skips non-directory or unstatable targets before starting chmod or ACL work. Windows reads omit `chmod` before ACL hardening so repeated loads preserve successful object/freshness observations; external metadata changes still invalidate them. Directory flights keep their existing flush/reap lifecycle, and file/required-write failure policy is unchanged. `tests/windows/windows-config-read-acl-memo.test.ts` covers repeated loads, non-directory and directory-link behavior, external invalidation and POSIX permissions. -`withConfigDirHardeningSettled` in `src/config/paths.ts` admits synchronous mutation after the last unrelated await. Its registry includes the optional config flight, ancestors and retained timeout children; one existing ACL deadline bounds admission, including newly overlapping work. Expiry throws sanitized `WindowsSecretAclSettleTimeoutError` (`ETIMEDOUT`) without running the writer or dropping child cleanup ownership. The final empty-registry check and callback share one turn, with no await in the callback. Intercept/picker CA, token, cleanup, profile and snapshot phases re-enter after trust/discovery awaits. Config saves in `src/server/management/agent-settings-routes.ts` and `src/server/management/config-routes.ts`, Desktop intent/mode commits in `src/server/management/native-integration-routes.ts`, and gateway publication/cleanup in those routes and `src/cli/ensure-desired-integrations.ts` use the same admission. `src/cli/claude-desktop.ts` admits local profile/mode commits and the complete remote-store transaction again after hub download. Lifecycle/config locks are acquired inside admission; refresh re-read, mode/freshness checks, profile and marker publication share one callback. Live setting edits and their save/publication share the admitted synchronous callback; a refused admission leaves both live and durable state untouched. Ordered post-commit phases preserve applied-state warnings or settled compensation, including mode/fingerprint marker refusal and retained CLI intent after incomplete rollback. Synchronous leaf APIs retain their contracts. `tests/claude-integration/claude-config-root-writer-barrier.test.ts` holds parent flights and drives refusal deadlines without sleeps. Lexical/canonical aliases and uncertain identities drain conservatively; removal keeps its separate at-or-below scope and actual-exit requirement. Async local config commands (`src/cli/config-command.ts`, `src/cli/effort.ts`, `src/cli/models.ts`, `src/cli/provider.ts`) admit complete synchronous read/validate/save phases through `withConfigDirHardeningSettled`. Preliminary model selection and client-connection diagnostics use passive observations that cannot publish recovery copies. Final authority and recovery publication are re-read inside admission without starting optional hardening. `src/cli/config-mutation.ts` refuses invalid or unreadable authority, including array/non-object roots rejected before defaults repair, with `CliUsageError`; full-config commands retain fresh-install initialization when the file is absent, while set/unset retain the persisted mutation owner's missing-file refusal. Custom-model removal revalidates the confirmed row by identity and contents while retaining current siblings. Import reads its input and checks its target inside admission. Provider-add credential cleanup shares the synchronous post-save phase and retains its existing post-commit warning. Timeout leaves the mutation database and config untouched and follows existing CLI failure handling. `tests/windows/windows-config-root-async-writers.test.ts` covers held parent flights, deadline refusal, recovery copies, invalid authority and concurrent edits. Codex multi-agent management changes that can resolve a runtime additionally await fresh [selection admission](codex-cli-update.md#worker-realm-and-the-selection-commit-window) before entering this barrier, keep that owner across hardening, and re-evaluate the mode plan inside the synchronous callback; neither ACL wait is inserted into the callback, and refusal leaves the combined mutation untouched. +`withConfigDirHardeningSettled` in `src/config/paths.ts` admits synchronous mutation after the last unrelated await. Its registry includes the optional config flight, ancestors and retained timeout children; one existing ACL deadline bounds admission, including newly overlapping work. Expiry throws sanitized `WindowsSecretAclSettleTimeoutError` (`ETIMEDOUT`) without running the writer or dropping child cleanup ownership. The final empty-registry check and callback share one turn, with no await in the callback. Intercept/picker CA, token, cleanup, profile and snapshot phases re-enter after trust/discovery awaits. Config saves in `src/server/management/agent-settings-routes.ts` and `src/server/management/config-routes.ts`, Desktop intent/mode commits in `src/server/management/native-integration-routes.ts`, and gateway publication/cleanup in those routes and `src/cli/ensure-desired-integrations.ts` use the same admission. `src/cli/claude-desktop.ts` admits local profile/mode commits and the complete remote-store transaction again after hub download. Lifecycle/config locks are acquired inside admission; refresh re-read, mode/freshness checks, profile and marker publication share one callback. Live setting edits and their save/publication share the admitted synchronous callback; a refused admission leaves both live and durable state untouched. Ordered post-commit phases preserve applied-state warnings or settled compensation, including mode/fingerprint marker refusal and retained CLI intent after incomplete rollback. Synchronous leaf APIs retain their contracts. `tests/claude-integration/claude-config-root-writer-barrier.test.ts` holds parent flights and drives refusal deadlines without sleeps. Lexical/canonical aliases and uncertain identities drain conservatively; removal keeps its separate at-or-below scope and actual-exit requirement. Async local config commands (`src/cli/config-command.ts`, `src/cli/effort.ts`, `src/cli/models.ts`, `src/cli/provider.ts`) admit complete synchronous read/validate/save phases through `withConfigDirHardeningSettled`. Preliminary model selection and client-connection diagnostics use passive observations that cannot publish recovery copies. Final authority and recovery publication are re-read inside admission without starting optional hardening. `src/cli/config-mutation.ts` refuses invalid or unreadable authority, including array/non-object roots rejected before defaults repair, with `CliUsageError`; full-config commands retain fresh-install initialization when the file is absent, while set/unset retain the persisted mutation owner's missing-file refusal. Custom-model removal revalidates the confirmed row by identity and contents while retaining current siblings. Import reads its input and checks its target inside admission. Provider-add credential cleanup shares the synchronous post-save phase and retains its existing post-commit warning. Timeout leaves the mutation database and config untouched and follows existing CLI failure handling. `tests/windows/windows-config-root-async-writers.test.ts` covers held parent flights, deadline refusal, recovery copies, invalid authority and concurrent edits. Codex multi-agent management changes that can resolve a runtime additionally await fresh [selection admission](codex-cli-update.md#worker-realm-and-the-selection-commit-window) before entering this barrier, keep that owner across hardening, and re-evaluate the mode plan inside the synchronous callback; neither ACL wait is inserted into the callback, and refusal leaves the combined mutation untouched. Timer-driven caches under the config root admit their write the same way and build the snapshot inside the callback: the provider account quota cache (`src/providers/account-quota-disk.ts`), the Codex account quota cache (`src/codex/quota.ts`) and the reasoning support cache, while a models.dev refresh admits its publication after the network read (`src/providers/reasoning-metadata.ts`). A timed-out admission skips a best-effort cache write and fails that refresh unpublished, a joined refresh reports the same failure, and a cancel, clear or reset invalidates a write still waiting for admission (#7815; `tests/providers/provider-account-quota-persistence.test.ts`, `tests/codex-integration/reasoning-metadata.test.ts`). `src/config/process-state.ts` derives `ocx.pid` and `runtime-port.json` from that resolved directory. It owns their byte-compatible writes, parsing, expected-PID filters, cheap liveness, full OCX command diff --git a/tests/codex-integration/main-quota-provenance.test.ts b/tests/codex-integration/main-quota-provenance.test.ts index 11db5da67f4..f75163fef92 100644 --- a/tests/codex-integration/main-quota-provenance.test.ts +++ b/tests/codex-integration/main-quota-provenance.test.ts @@ -1,5 +1,5 @@ import { capturePoolQuotaWriter, saveCodexAccountCredential, saveCodexAccountCredentialIfGeneration } from "../../src/codex/account-store"; -import { getAccountQuotaHistory, isValidWhamHistoryObservation } from "../../src/codex/quota"; +import { getAccountQuotaHistory, isValidWhamHistoryObservation, settleQuotaPersistenceForTests } from "../../src/codex/quota"; import { afterEach, beforeAll, beforeEach, describe, expect, spyOn, test } from "bun:test"; import { mkdtempSync, readFileSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; @@ -75,12 +75,14 @@ function installPersistenceClock() { } /** Run the captured quota persistence callback and read its actual disk snapshot without a sleep. */ -function flushPersistence(): string { +async function flushPersistence(): Promise { if (!pendingPersist) throw new Error("Expected a scheduled quota persistence"); const pending = pendingPersist; pendingPersist = undefined; clearTimeout(pending.timer); pending.run(); + // The callback admits its write after any overlapping config-root ACL flight (#7815). + await settleQuotaPersistenceForTests(); return readFileSync(join(testDir, "codex-quota-cache.json"), "utf8"); } @@ -276,7 +278,7 @@ describe("main policy quota writes", () => { }); } - test("header writer carries provenance and untagged main writes invalidate it", () => { + test("header writer carries provenance and untagged main writes invalidate it", async () => { const writer = writerFor(); const headers = new Headers({ "x-codex-primary-used-percent": "99" }); applyAccountQuotaFromUpstreamHeaders(MAIN, headers, undefined, writer); @@ -289,10 +291,10 @@ describe("main policy quota writes", () => { updateAccountQuota(MAIN, 20); expect(getMainPolicyQuota()).toBeNull(); expect(getAccountQuota(MAIN)?.weeklyPercent).toBe(20); - expect(JSON.parse(flushPersistence()).mainPolicyQuota).toBeUndefined(); + expect(JSON.parse(await flushPersistence()).mainPolicyQuota).toBeUndefined(); }); - test("public quota mutation and serializers cannot expose or mutate policy provenance", () => { + test("public quota mutation and serializers cannot expose or mutate policy provenance", async () => { const writer = writerFor(); observeMainQuotaCredential("fixture-private-bearer", "fixture-main-a"); setAccountQuotaFromParsed(MAIN, { weeklyPercent: 99 }, undefined, writer); @@ -301,7 +303,7 @@ describe("main policy quota writes", () => { expect(getMainPolicyQuota()?.weeklyPercent).toBe(99); const publicJson = JSON.stringify(Object.fromEntries(listAccountQuotas())); expect(publicJson).not.toContain("identityKey"); - const disk = flushPersistence(); + const disk = await flushPersistence(); expect(disk).not.toContain("fixture-private-bearer"); expect(disk).not.toContain("fixture-main-a"); expect(disk).not.toContain("bearerHmac"); @@ -310,7 +312,7 @@ describe("main policy quota writes", () => { }); }); -test("window replacement persists without carrying its proof into later partial updates", () => { +test("window replacement persists without carrying its proof into later partial updates", async () => { const cfg = { codexMainAccountHardLock: true }; const writer = writerFor(); /** Publish both parsed projections with the captured writer throughout the simulated restart. */ @@ -324,7 +326,7 @@ test("window replacement persists without carrying its proof into later partial allowed: true, primary_window: { used_percent: 35, limit_window_seconds: 604_800 }, secondary_window: null, } }); // Execute quota's actual debounced serializer through the existing deterministic clock. - const persisted = flushPersistence(); + const persisted = await flushPersistence(); expect(JSON.parse(persisted).mainPolicyQuota.quota.weeklyPercent).toBe(35); expect(persisted).not.toContain("shortWindowAbsent"); clearAccountQuota(); @@ -359,11 +361,11 @@ describe("main policy quota durability and lifecycle", () => { { name: "startup file replacement", account: "fixture-main-b", tag: "known", selected: true, source: "file" }, { name: "startup same-account file", account: "fixture-main-a", tag: "known", selected: false, source: "file", refusal: "CodexMainAccountHardLockError" }, ]) { - test(`cold keyring Pool preserves quota unless replacement is proven: ${scenario.name}`, () => { + test(`cold keyring Pool preserves quota unless replacement is proven: ${scenario.name}`, async () => { const writer = writerFor("fixture-main-a"); setAccountQuotaFromParsed(MAIN, { weeklyPercent: 100, weeklyResetAt: Date.now() / 1000 + 3600 }, undefined, writer); setAccountQuotaFromParsed("fixture-stored", { weeklyPercent: 33 }); - const snapshot = JSON.parse(flushPersistence()); + const snapshot = JSON.parse(await flushPersistence()); if (scenario.tag === "missing") delete snapshot.mainPolicyQuota; if (scenario.tag === "invalid") snapshot.mainPolicyQuota.identityKey = "invalid-fixture-identity"; writeSnapshot(snapshot); @@ -431,7 +433,7 @@ describe("main policy quota durability and lifecycle", () => { } for (const unavailable of [true, false]) { - test(`persisted ${unavailable ? "unavailable" : "legacy"} null remains identity-bound and cannot authorize spending`, () => { + test(`persisted ${unavailable ? "unavailable" : "legacy"} null remains identity-bound and cannot authorize spending`, async () => { const writer = writerFor(); const resetAt = Date.now() + 86_400_000; setAccountQuotaFromParsed(MAIN, { weeklyPercent: 100, weeklyResetAt: resetAt, @@ -447,7 +449,7 @@ describe("main policy quota durability and lifecycle", () => { expect(quota?.creditObservationUnavailable).toBe(unavailable ? true : undefined); expect(hasSpendableCodexCredits(quota)).toBe(false); expect(isCodexAccountHeldForCredits({ creditCodexAccountIds: [MAIN] }, MAIN, quota, "plus", Date.now())).toBe(true); - const persisted = JSON.parse(flushPersistence()); + const persisted = JSON.parse(await flushPersistence()); expect(persisted.mainPolicyQuota).toEqual({ identityKey: writer.identityKey, quota }); // Read the actual serializer output in a fresh process, without clearing or rewriting it. const script = ` @@ -474,7 +476,7 @@ describe("main policy quota durability and lifecycle", () => { }, SPAWN_BUDGET_MS); } - test("stale main writers cannot replace unavailable evidence or attach it to a new identity", () => { + test("stale main writers cannot replace unavailable evidence or attach it to a new identity", async () => { const oldA = writerFor(); const unavailable = parseUsageQuota({ credits: {} }); const positive = { hasCredits: true, balance: 5, observedAt: Date.now() }; @@ -498,7 +500,7 @@ describe("main policy quota durability and lifecycle", () => { expect(getAccountQuota(MAIN)?.credits).toBeNull(); expect(hasSpendableCodexCredits(getMainPolicyQuota())).toBe(false); expect(isCodexAccountHeldForCredits({ creditCodexAccountIds: [MAIN] }, MAIN, getMainPolicyQuota(), "plus", Date.now())).toBe(true); - expect(JSON.parse(flushPersistence()).mainPolicyQuota).toEqual({ identityKey: newA.identityKey, quota: before }); + expect(JSON.parse(await flushPersistence()).mainPolicyQuota).toEqual({ identityKey: newA.identityKey, quota: before }); }); for (const resetAt of [undefined, 4_000_000_000]) { @@ -535,12 +537,12 @@ describe("main policy quota durability and lifecycle", () => { }, SPAWN_BUDGET_MS); } - test("unrelated persistence hydrates and retains policy after legacy TTL expiry", () => { + test("unrelated persistence hydrates and retains policy after legacy TTL expiry", async () => { const writer = writerFor(); const quota = { weeklyPercent: 99, updatedAt: Date.now() - 7 * 60 * 60_000 }; writeSnapshot({ version: 1, quotas: { [MAIN]: quota }, mainPolicyQuota: { identityKey: writer.identityKey, quota } }); setAccountQuotaFromParsed("fixture-pool", { weeklyPercent: 12 }); - const saved = JSON.parse(flushPersistence()); + const saved = JSON.parse(await flushPersistence()); expect(saved.quotas[MAIN]).toBeUndefined(); expect(saved.mainPolicyQuota.quota).toEqual(quota); expect(getMainPolicyQuota()).toEqual(quota); @@ -549,7 +551,7 @@ describe("main policy quota durability and lifecycle", () => { setAccountQuotaFromParsed(MAIN, { resetCredits: 0 }, undefined, writer); expect(getMainPolicyQuota()).toMatchObject({ weeklyPercent: 99, resetCredits: 0 }); expect(getAccountQuota(MAIN)).toEqual({ resetCredits: 0, updatedAt: expect.any(Number) }); - const afterCredits = JSON.parse(flushPersistence()); + const afterCredits = JSON.parse(await flushPersistence()); expect(afterCredits.quotas[MAIN].weeklyPercent).toBeUndefined(); expect(afterCredits.mainPolicyQuota.quota.weeklyPercent).toBe(99); expect(getMainAccountHardLockStatus({ codexMainAccountHardLock: true }).state).toBe("blocked"); @@ -557,7 +559,7 @@ describe("main policy quota durability and lifecycle", () => { expect(getMainPolicyQuota()?.weeklyPercent).toBe(99); clearAccountQuota(MAIN); expect(getMainPolicyQuota()).toBeNull(); - expect(JSON.parse(flushPersistence()).mainPolicyQuota).toBeUndefined(); + expect(JSON.parse(await flushPersistence()).mainPolicyQuota).toBeUndefined(); }); test("clear before first hydration cannot resurrect disk policy", () => { @@ -612,7 +614,7 @@ describe("main policy quota durability and lifecycle", () => { }); -test("pool history records fresh windows only and preserves identity across token refresh", () => { +test("pool history records fresh windows only and preserves identity across token refresh", async () => { const credential = { accessToken: "history-token", refreshToken: "history-refresh", chatgptAccountId: "history-account", expiresAt: Date.now() + 3600_000 }; const generation = saveCodexAccountCredential("history-pool", credential); const writer = capturePoolQuotaWriter("history-pool", { ...credential, generation })!; @@ -637,7 +639,7 @@ test("pool history records fresh windows only and preserves identity across toke expect(rows).toHaveLength(2); expect(isValidWhamHistoryObservation({ rate_limit: { primary_window: { used_percent: 101 } } })).toBe(false); expect(isValidWhamHistoryObservation({ additional_rate_limits: [{ rate_limit: { primary_window: { used_percent: -1 } } }] })).toBe(false); - const body = flushPersistence(); + const body = await flushPersistence(); expect(JSON.parse(body).history.accounts["history-pool"].samples).toHaveLength(2); expect(body).not.toContain("history-token"); expect(body).not.toContain("history-refresh"); @@ -648,11 +650,11 @@ test("pool history records fresh windows only and preserves identity across toke expect(getAccountQuotaHistory("history-pool").observations).toEqual([]); }); -test("native main observations and oversized cache never become pool history", () => { +test("native main observations and oversized cache never become pool history", async () => { const raw = { weeklyPercent: 20 }; setAccountQuotaFromParsed(MAIN, raw, undefined, writerFor()); expect(getAccountQuotaHistory(MAIN).observations).toEqual([]); - const persisted = JSON.parse(flushPersistence()); + const persisted = JSON.parse(await flushPersistence()); expect(persisted.history.accounts).not.toHaveProperty(MAIN); clearAccountQuota(); const credential = { accessToken: "large-cache-access", refreshToken: "large-cache-refresh", chatgptAccountId: "large-cache-account", expiresAt: Date.now() + 3600_000 }; diff --git a/tests/codex-integration/reasoning-metadata.test.ts b/tests/codex-integration/reasoning-metadata.test.ts index 882b9151f4f..4bd1f5cb9dc 100644 --- a/tests/codex-integration/reasoning-metadata.test.ts +++ b/tests/codex-integration/reasoning-metadata.test.ts @@ -322,7 +322,7 @@ describe("learned rung refusals", () => { provider: ZEN_GO, modelId: DEEPSEEK_FLASH, requested: "max", rejectionText: REJECTION_BODY, }); expect(first).toEqual({ effort: "high", recorded: true }); - metadata.flushReasoningSupportCache(); + await metadata.flushReasoningSupportCache(); const second = metadata.planReasoningEffortDowngrade({ provider: ZEN_GO, modelId: DEEPSEEK_FLASH, requested: "max", rejectionText: REJECTION_BODY, }); @@ -402,3 +402,63 @@ describe("destination resolution", () => { expect(zenGo.models).toBe(1); }); }); + +describe("config-root ACL admission (#7815)", () => { + test("a refresh publishes its snapshot only after an in-flight config-root ACL flight settles", async () => { + const { metadata } = await load(); + const home = process.env["OPENCODEX_HOME"]!; + const acl = await import("../../src/lib/windows-secret-acl"); + const { setSyntheticWindowsPrincipalForTests } = await import("../../src/lib/windows-user-principal"); + const { existsSync } = await import("node:fs"); + const success = { success: true, exitCode: 0, timedOut: false, stdout: "" }; + let entered!: () => void, release!: () => void, fetched!: () => void; + const started = new Promise(resolve => { entered = resolve; }); + const held = new Promise(resolve => { release = resolve; }); + const fetchedOnce = new Promise(resolve => { fetched = resolve; }); + const previousFetch = globalThis.fetch; + acl.resetHardenedStateForTests(); + acl.setPlatformForTests("win32"); + setSyntheticWindowsPrincipalForTests("*S-1-5-21-1-2-3-1001"); + acl.setIcaclsRunnerForTests(() => success); + acl.setAsyncIcaclsRunnerForTests(async () => { entered(); await held; return success; }); + try { + globalThis.fetch = (async () => { fetched(); return new Response("{}", { status: 200 }); }) as unknown as typeof fetch; + const flight = acl.hardenSecretDirAsync(home, { required: false }); + await started; + let refresh: Promise | undefined; + try { + refresh = metadata.refreshReasoningMetadata({ force: true }); + await fetchedOnce; + await new Promise(resolve => setImmediate(resolve)); + expect(existsSync(join(home, "reasoning-metadata-cache.json"))).toBe(false); + } finally { release(); await flight; } + expect(await refresh).toMatchObject({ ok: true, reason: "refreshed" }); + expect(existsSync(join(home, "reasoning-metadata-cache.json"))).toBe(true); + } finally { + globalThis.fetch = previousFetch; + acl.setAsyncIcaclsRunnerForTests(null); + acl.setIcaclsRunnerForTests(null); + acl.setPlatformForTests(null); + setSyntheticWindowsPrincipalForTests(null); + acl.resetHardenedStateForTests(); + metadata.resetReasoningMetadataCachesForTests(); + } + }); + + test("a joined refresh reports the shared failure instead of a coalesced success", async () => { + const { metadata } = await load(); + const previousFetch = globalThis.fetch; + let reject!: (error: Error) => void; + try { + globalThis.fetch = (() => new Promise((_resolve, fail) => { reject = fail; })) as unknown as typeof fetch; + const first = metadata.refreshReasoningMetadata({ force: true }); + const joined = metadata.refreshReasoningMetadata({ force: true }); + reject(new Error("fixture refusal")); + expect(await first).toEqual({ ok: false, reason: "fixture refusal" }); + expect(await joined).toEqual({ ok: false, reason: "fixture refusal" }); + } finally { + globalThis.fetch = previousFetch; + metadata.resetReasoningMetadataCachesForTests(); + } + }); +}); diff --git a/tests/providers/provider-account-quota-persistence.test.ts b/tests/providers/provider-account-quota-persistence.test.ts index 22c4ccf2618..90e1e792024 100644 --- a/tests/providers/provider-account-quota-persistence.test.ts +++ b/tests/providers/provider-account-quota-persistence.test.ts @@ -188,3 +188,68 @@ describe("provider quota snapshot deadlines", () => { expect(readPersistedAccountQuotas().get(KEY)?.monthlyPercent).toBe(20); }); }); + +// #7815: a cache write under the config root waits out an overlapping optional config-root ACL +// flight, whose inheritance would otherwise race the new file's explicit grant. +test("a snapshot write waits for an in-flight config-root ACL flight, then lands", async () => { + const acl = await import("../../src/lib/windows-secret-acl"); + const { setSyntheticWindowsPrincipalForTests } = await import("../../src/lib/windows-user-principal"); + const success = { success: true, exitCode: 0, timedOut: false, stdout: "" }; + let entered!: () => void, release!: () => void; + const started = new Promise(resolve => { entered = resolve; }); + const held = new Promise(resolve => { release = resolve; }); + acl.resetHardenedStateForTests(); + acl.setPlatformForTests("win32"); + setSyntheticWindowsPrincipalForTests("*S-1-5-21-1-2-3-1001"); + acl.setIcaclsRunnerForTests(() => success); + acl.setAsyncIcaclsRunnerForTests(async () => { entered(); await held; return success; }); + try { + const flight = acl.hardenSecretDirAsync(home, { required: false }); + await started; + try { + schedulePersistAccountQuotas(() => [[KEY, quota(15)]]); + await settle(); + expect(existsSync(join(home, FILE))).toBe(false); + } finally { release(); await flight; } + await settle(); + expect(readPersistedAccountQuotas().get(KEY)?.monthlyPercent).toBe(15); + } finally { + acl.setAsyncIcaclsRunnerForTests(null); + acl.setIcaclsRunnerForTests(null); + acl.setPlatformForTests(null); + setSyntheticWindowsPrincipalForTests(null); + acl.resetHardenedStateForTests(); + } +}); + +test("a snapshot still waiting for ACL admission is dropped by cancellation", async () => { + const acl = await import("../../src/lib/windows-secret-acl"); + const { setSyntheticWindowsPrincipalForTests } = await import("../../src/lib/windows-user-principal"); + const success = { success: true, exitCode: 0, timedOut: false, stdout: "" }; + let entered!: () => void, release!: () => void; + const started = new Promise(resolve => { entered = resolve; }); + const held = new Promise(resolve => { release = resolve; }); + acl.resetHardenedStateForTests(); + acl.setPlatformForTests("win32"); + setSyntheticWindowsPrincipalForTests("*S-1-5-21-1-2-3-1001"); + acl.setIcaclsRunnerForTests(() => success); + acl.setAsyncIcaclsRunnerForTests(async () => { entered(); await held; return success; }); + try { + const flight = acl.hardenSecretDirAsync(home, { required: false }); + await started; + try { + schedulePersistAccountQuotas(() => [[KEY, quota(15)]]); + await settle(); + // The timer fired and the write is waiting; a clear must still win. + cancelPendingAccountQuotaPersist(); + } finally { release(); await flight; } + await settle(); + expect(existsSync(join(home, FILE))).toBe(false); + } finally { + acl.setAsyncIcaclsRunnerForTests(null); + acl.setIcaclsRunnerForTests(null); + acl.setPlatformForTests(null); + setSyntheticWindowsPrincipalForTests(null); + acl.resetHardenedStateForTests(); + } +}); From 8e58c9683d7d1e0fd2dbb3604cf05dca62b02b99 Mon Sep 17 00:00:00 2001 From: JUN Date: Fri, 30 Oct 2026 22:57:44 +0900 Subject: [PATCH 36/42] test(cli): name the pinned-port holder when a sibling misses its deadline (#8003) * test(cli): name the pinned-port holder when a sibling misses its deadline A pinned sibling start that misses its record deadline is ambiguous: the port was taken between probe and bind, the child bound without writing its record, or it never bound. Report whether the port is bindable and, on Windows, the netstat owners of that port (windows 5/11 of lane=all run 41061583113). * test(cli): keep the port-holder diagnostic best effort and read-only while the child lives Review: a missing netstat threw over the original timeout, and the bind probe ran while the child was alive. List sockets before the kill with a bounded, non-throwing netstat, then probe bindability after the child exits. * test(cli): bound the post-kill wait in the port-holder diagnostic * test(cli): skip the bind probe while the killed child may still be alive --------- Co-authored-by: JUN <243035832+lidge-jun@users.noreply.github.com> --- tests/cli/cli-start-journal-order.test.ts | 37 ++++++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-) diff --git a/tests/cli/cli-start-journal-order.test.ts b/tests/cli/cli-start-journal-order.test.ts index 4a6def739e0..969d75f5245 100644 --- a/tests/cli/cli-start-journal-order.test.ts +++ b/tests/cli/cli-start-journal-order.test.ts @@ -397,6 +397,34 @@ function freeLoopbackPort(): number { return port; } +/** + * Who holds a pinned port when a sibling missed its deadline: the child itself (bound, record not + * yet written), another process (the port was taken between probe and bind) or nobody (the child + * never reached its bind). The bare deadline cannot tell these apart. + * + * Best effort and read-only while the child lives: socket owners are listed before the child is + * killed, the bind probe runs after. Nothing here may throw over the original timeout. + */ +function pinnedPortSockets(port: number, childPid: number): string { + if (process.platform !== "win32") return ""; + try { + const listing = Bun.spawnSync(["netstat", "-ano", "-p", "TCP"], { stdout: "pipe", stderr: "ignore", timeout: 5_000 }); + if (listing.exitCode !== 0) return `; netstat exit ${listing.exitCode}`; + const rows = [...new Set(listing.stdout.toString().split(/\r?\n/) + .map(line => line.trim().split(/\s+/)) + .filter(cols => cols.length >= 5 && cols[1]!.endsWith(`:${port}`)) + .map(cols => `${cols[3]} pid ${cols[4]}${cols[4] === String(childPid) ? " (child)" : ""}`))]; + return rows.length ? `; ${rows.join(", ")}` : "; no socket rows"; + } catch (error) { + return `; netstat unavailable (${(error as NodeJS.ErrnoException).code ?? "error"})`; + } +} + +function pinnedPortBindable(port: number): string { + try { Bun.serve({ hostname: "127.0.0.1", port, fetch: () => new Response(null) }).stop(true); return "free after kill"; } + catch { return "still bound after kill"; } +} + /** * Spawn `ocx start` and wait for its runtime record and settled readiness. * @@ -446,7 +474,14 @@ async function startSibling( // An early exit already reported its stderr and consumed the stream. if (exitedEarly) throw error; // Report what the child was doing, which the bare deadline message cannot. + const sockets = port === undefined ? "" : pinnedPortSockets(port, child.pid); if (child.exitCode === null) child.kill("SIGKILL"); + // Bounded like tail() below: the diagnostic must never outlast the original failure. + const exited = port === undefined + || await Promise.race([child.exited.then(() => true), Bun.sleep(5_000).then(() => false)]); + // A bind probe while the child may still be alive would only find the child itself. + const holders = port === undefined ? "not pinned" + : `${exited ? pinnedPortBindable(port) : "child alive 5s after kill; bind probe skipped"}${sockets}`; const tail = async (stream: unknown): Promise => { try { return await Promise.race([ @@ -459,7 +494,7 @@ async function startSibling( }; const [stdout, stderr] = await Promise.all([tail(child.stdout), tail(child.stderr)]); throw new Error( - `${(error as Error).message} (requested port ${port ?? "config"}, pid ${child.pid}, last record ${lastRecord ?? "none"})\n` + `${(error as Error).message} (requested port ${port ?? "config"}, pid ${child.pid}, last record ${lastRecord ?? "none"}, port holders: ${holders})\n` + `--- stdout ---\n${stdout}\n--- stderr ---\n${stderr}`, ); } From ed1cf0871099245d2f49f8d8357c8bac2927e501 Mon Sep 17 00:00:00 2001 From: Epinephrine Date: Fri, 30 Oct 2026 23:41:55 +0900 Subject: [PATCH 37/42] fix(combo): exclude Meta for direct hosted tool choices --- docs-site/src/content/docs/guides/combos.md | 4 +- .../responses/combo-required-tool-contract.ts | 25 +++++----- structure/providers-and-adapters.md | 2 +- .../transports/responses-combo-boundaries.md | 2 +- ...erver-combo-required-tool-contract.test.ts | 47 +++++++++++++++++++ 5 files changed, 65 insertions(+), 15 deletions(-) diff --git a/docs-site/src/content/docs/guides/combos.md b/docs-site/src/content/docs/guides/combos.md index a0b9de7c718..72e9ea05310 100644 --- a/docs-site/src/content/docs/guides/combos.md +++ b/docs-site/src/content/docs/guides/combos.md @@ -154,7 +154,9 @@ native routing and catalog identity on the next sync. For Responses requests that require a tool, Meta Responses targets are skipped before sending because Meta does not support mandatory `tool_choice`. An eligible later target receives the -original choice. If every target is incompatible for this reason, the combo reports the constraint. +original choice. This includes hosted choices such as `{ "type": "web_search" }` and +MCP choices with a nonempty `server_label`. If every target is incompatible for this reason, +the combo reports the constraint. Malformed or unknown choices retain the provider's ordinary error path. `failover` selects the first eligible target in configuration order. A target is eligible when its provider exists, is enabled, is not cooling down, and can handle any special request constraint. diff --git a/src/server/responses/combo-required-tool-contract.ts b/src/server/responses/combo-required-tool-contract.ts index 2f5e16a22d5..ed94c0878e4 100644 --- a/src/server/responses/combo-required-tool-contract.ts +++ b/src/server/responses/combo-required-tool-contract.ts @@ -15,6 +15,15 @@ const allowedHostedSelectors = new Set([ "code_interpreter", "image_generation", "image_gen", "tool_search", ]); +function knownToolSelector(value: unknown): boolean { + if (!value || typeof value !== "object" || Array.isArray(value)) return false; + const entry = value as Record; + if (entry.name !== undefined && (typeof entry.name !== "string" || !entry.name.trim())) return false; + if (entry.type === "function" || entry.type === "custom") return typeof entry.name === "string"; + if (entry.type === "mcp") return typeof entry.server_label === "string" && !!entry.server_label.trim(); + return typeof entry.type === "string" && allowedHostedSelectors.has(entry.type); +} + /** Only definite, well-formed mandatory selectors justify excluding a destination. */ export function hasMandatoryToolChoice(rawBody: unknown): boolean { if (!rawBody || typeof rawBody !== "object" || Array.isArray(rawBody)) return false; @@ -22,19 +31,11 @@ export function hasMandatoryToolChoice(rawBody: unknown): boolean { if (choice === "required") return true; if (!choice || typeof choice !== "object" || Array.isArray(choice)) return false; const selector = choice as Record; - if (selector.type === "function" || selector.type === "custom") { - return typeof selector.name === "string" && selector.name.trim().length > 0; - } - return selector.type === "allowed_tools" && selector.mode === "required" + // Eligibility is broader than terminal inspection: hosted selectors retain their own terminal owner. + if (selector.type !== "allowed_tools") return knownToolSelector(selector); + return selector.mode === "required" && Array.isArray(selector.tools) && selector.tools.length > 0 - && selector.tools.every((tool: unknown) => { - if (!tool || typeof tool !== "object" || Array.isArray(tool)) return false; - const entry = tool as Record; - if (entry.name !== undefined && (typeof entry.name !== "string" || !entry.name.trim())) return false; - if (entry.type === "function" || entry.type === "custom") return typeof entry.name === "string"; - if (entry.type === "mcp") return typeof entry.server_label === "string" && !!entry.server_label.trim(); - return typeof entry.type === "string" && allowedHostedSelectors.has(entry.type); - }); + && selector.tools.every(knownToolSelector); } const hostedCalls = new Map([ diff --git a/structure/providers-and-adapters.md b/structure/providers-and-adapters.md index 240a397f29b..a90cac5a79d 100644 --- a/structure/providers-and-adapters.md +++ b/structure/providers-and-adapters.md @@ -555,7 +555,7 @@ uses `src/adapters/openai-responses/muse-tool-choice.ts` to normalize `tool_choi `additional_tools` items from `input`, and omits `tool_choice` and `parallel_tool_calls` from the outgoing body. Forced, named, and `allowed_tools` selections fail with HTTP 400 before the upstream send because Muse supports only `auto`. Filtering a required tool never changes the caller's obligation into `none` or `auto`. This rule applies only to the Meta Responses -destination. The input body, historical tool calls and results, and non-Meta requests keep their existing meaning. +destination. Combo eligibility excludes it before sending for known mandatory selectors, including direct hosted types and MCP selectors with a nonempty server label; malformed and unknown selectors retain the adapter error path. The input body, historical tool calls and results, and non-Meta requests keep their existing meaning. ## Shared type declarations diff --git a/structure/transports/responses-combo-boundaries.md b/structure/transports/responses-combo-boundaries.md index 1c057789d0a..8e8c618952e 100644 --- a/structure/transports/responses-combo-boundaries.md +++ b/structure/transports/responses-combo-boundaries.md @@ -21,7 +21,7 @@ and `tests/responses/responses-combo-exhausted-error.test.ts`. ## Buffered required-tool contract Before selection, `src/server/responses/core-combo.ts` excludes resolved Meta Responses destinations -for well-formed mandatory `tool_choice` selectors, including required allowed-tools lists. Normal and +for well-formed mandatory `tool_choice` selectors, including direct hosted/MCP choices and required allowed-tools lists. Normal and recovery picks share the check, resolving the inbound wire's static model policy and actual Responses transport URL; malformed and unknown selectors retain their ordinary error path. The next target receives the original selector, and an all-Meta exhaustion names the constraint before any send. diff --git a/tests/server/server-combo-required-tool-contract.test.ts b/tests/server/server-combo-required-tool-contract.test.ts index f25491239a2..03fa2e19a5c 100644 --- a/tests/server/server-combo-required-tool-contract.test.ts +++ b/tests/server/server-combo-required-tool-contract.test.ts @@ -165,6 +165,53 @@ function requiredRequest(extra: Record = {}): Request { } describe("combo required-tool contract on buffered children (#7456)", () => { + for (const choice of [{ type: "web_search" }, { type: "mcp", server_label: "docs" }]) { + for (const stream of [false, true]) { + test(`direct hosted choice skips Meta before send (${choice.type}, stream=${stream})`, async () => { + const bodies: unknown[] = []; let metaSends = 0; + const upstream = serve(async request => { + bodies.push(await request.json()); + const response = completedNoCall("m2", "hosted terminal owner"); + return stream ? new Response(`data: ${JSON.stringify({ type: "response.completed", response })}\n\n`, + { headers: { "content-type": "text/event-stream" } }) : Response.json(response); + }); + const declaration = choice.type === "mcp" ? { ...choice, server_url: "https://docs.example.invalid/mcp" } : choice; + const response = await handleResponsesWith(comboConfig({ + a: provider("openai-responses", "https://api.meta.ai/v1", "key-a", { + fetch: (async () => { metaSends++; throw new Error("unexpected Meta send"); }) as typeof fetch }), + b: provider("openai-responses", baseUrl(upstream), "key-b"), + }), requiredRequest({ tools: [declaration], tool_choice: choice, stream })); + expect(response.status).toBe(200); + expect(await response.text()).toContain("hosted terminal owner"); + expect(metaSends).toBe(0); expect(bodies).toHaveLength(1); + expect(bodies[0]).toMatchObject({ tool_choice: choice }); + }); + } + test(`direct hosted choice reports all-Meta exhaustion (${choice.type})`, async () => { + let sends = 0; + const meta = provider("openai-responses", "https://api.meta.ai/v1", "key-a", { + fetch: (async () => { sends++; throw new Error("unexpected Meta send"); }) as typeof fetch }); + const response = await handleResponsesWith(comboConfig({ a: meta, b: meta }), requiredRequest({ tool_choice: choice })); + expect(response.status).toBe(503); + expect(await response.text()).toContain("mandatory tool_choice"); + expect(sends).toBe(0); + }); + } + test("direct hosted selectors keep their terminal owner and reject malformed selector shapes", async () => { + const { hasMandatoryToolChoice, requiredToolContractOf } = await import("../../src/server/responses/combo-required-tool-contract"); + const { createComboRequiredToolStreamGuard } = await import("../../src/server/responses/combo-required-tool-stream"); + for (const choice of [{ type: "web_search" }, { type: "mcp", server_label: "docs" }, { type: "tool_search" }]) { + const body = { tools: [choice], tool_choice: choice }; + expect(hasMandatoryToolChoice(body)).toBe(true); + expect(requiredToolContractOf(body)).toBeUndefined(); + expect(createComboRequiredToolStreamGuard(body, new Set())).toBeUndefined(); + } + for (const choice of [{ type: "web_search", name: 5 }, { type: "web_search", name: " " }, + { type: "mcp" }, { type: "mcp", server_label: 5 }, { type: "mcp", server_label: " " }, + { type: "mcp", server_label: "docs", name: " " }, { type: "unknown_hosted" }]) { + expect(hasMandatoryToolChoice({ tool_choice: choice })).toBe(false); + } + }); for (const choice of ["required", { type: "function", name: "shell" }, { type: "allowed_tools", mode: "required", tools: [{ type: "function", name: "shell" }] }]) { test(`Meta mandatory tool choice selects the next target before send (#7964): ${JSON.stringify(choice)}`, async () => { From bf6b1bcea4ddc4b601d837f1f0c6ea7e5b79464b Mon Sep 17 00:00:00 2001 From: Epinephrine Date: Sat, 31 Oct 2026 02:10:13 +0900 Subject: [PATCH 38/42] fix(responses): accept recognized hosted choices in combo children --- docs-site/src/content/docs/guides/combos.md | 5 +++-- src/responses/schema.ts | 2 +- .../responses/combo-required-tool-contract.ts | 3 ++- .../transports/responses-combo-boundaries.md | 5 ++++- tests/responses/responses-parser.test.ts | 6 ++++++ ...server-combo-required-tool-contract.test.ts | 18 ++++++++++++------ 6 files changed, 28 insertions(+), 11 deletions(-) diff --git a/docs-site/src/content/docs/guides/combos.md b/docs-site/src/content/docs/guides/combos.md index 72e9ea05310..4c1c1083555 100644 --- a/docs-site/src/content/docs/guides/combos.md +++ b/docs-site/src/content/docs/guides/combos.md @@ -154,8 +154,9 @@ native routing and catalog identity on the next sync. For Responses requests that require a tool, Meta Responses targets are skipped before sending because Meta does not support mandatory `tool_choice`. An eligible later target receives the -original choice. This includes hosted choices such as `{ "type": "web_search" }` and -MCP choices with a nonempty `server_label`. If every target is incompatible for this reason, +original choice, subject to its existing adapter compatibility rules. This includes hosted choices +such as `web_search`, `tool_search`, `computer_use` and `image_gen`, and MCP choices with a +nonempty `server_label` and an omitted, null or nonempty name. If every target is incompatible for this reason, the combo reports the constraint. Malformed or unknown choices retain the provider's ordinary error path. `failover` selects the first eligible target in configuration order. A target is eligible when its diff --git a/src/responses/schema.ts b/src/responses/schema.ts index d3fac2f8632..d2b71011003 100644 --- a/src/responses/schema.ts +++ b/src/responses/schema.ts @@ -162,7 +162,7 @@ const builtinToolSchema = z.object({ type: z.string() }).loose(); */ export const HOSTED_TOOL_TYPES = [ "web_search", "web_search_preview", "file_search", "computer_use_preview", - "code_interpreter", "image_generation", "mcp", + "code_interpreter", "image_generation", "mcp", "tool_search", "computer_use", "image_gen", ] as const; const hostedToolType = z.enum(HOSTED_TOOL_TYPES); diff --git a/src/server/responses/combo-required-tool-contract.ts b/src/server/responses/combo-required-tool-contract.ts index ed94c0878e4..891af59e4bf 100644 --- a/src/server/responses/combo-required-tool-contract.ts +++ b/src/server/responses/combo-required-tool-contract.ts @@ -18,9 +18,10 @@ const allowedHostedSelectors = new Set([ function knownToolSelector(value: unknown): boolean { if (!value || typeof value !== "object" || Array.isArray(value)) return false; const entry = value as Record; + if (entry.type === "mcp") return typeof entry.server_label === "string" && !!entry.server_label.trim() + && (entry.name === undefined || entry.name === null || (typeof entry.name === "string" && !!entry.name.trim())); if (entry.name !== undefined && (typeof entry.name !== "string" || !entry.name.trim())) return false; if (entry.type === "function" || entry.type === "custom") return typeof entry.name === "string"; - if (entry.type === "mcp") return typeof entry.server_label === "string" && !!entry.server_label.trim(); return typeof entry.type === "string" && allowedHostedSelectors.has(entry.type); } diff --git a/structure/transports/responses-combo-boundaries.md b/structure/transports/responses-combo-boundaries.md index 8e8c618952e..15e87fd84eb 100644 --- a/structure/transports/responses-combo-boundaries.md +++ b/structure/transports/responses-combo-boundaries.md @@ -21,10 +21,13 @@ and `tests/responses/responses-combo-exhausted-error.test.ts`. ## Buffered required-tool contract Before selection, `src/server/responses/core-combo.ts` excludes resolved Meta Responses destinations -for well-formed mandatory `tool_choice` selectors, including direct hosted/MCP choices and required allowed-tools lists. Normal and +for well-formed mandatory `tool_choice` selectors, including direct hosted/MCP choices and required allowed-tools lists. MCP's optional name may be null; an invalid name or server label still abstains. Normal and recovery picks share the check, resolving the inbound wire's static model policy and actual Responses transport URL; malformed and unknown selectors retain their ordinary error path. The next target receives the original selector, and an all-Meta exhaustion names the constraint before any send. +The child parser accepts `tool_search`, `computer_use` and `image_gen` as recognized hosted choices. +Its raw body preserves the selector; existing destination compatibility, including routed private +tool-search lowering, still applies after parsing. This does not expand Chat `mapToolChoice`. Coverage: `tests/server/server-combo-required-tool-contract.test.ts`. Before combo success bookkeeping, native delivery in `src/server/responses/passthrough-delivery.ts` diff --git a/tests/responses/responses-parser.test.ts b/tests/responses/responses-parser.test.ts index 93698c8b5d3..8cb83aac088 100644 --- a/tests/responses/responses-parser.test.ts +++ b/tests/responses/responses-parser.test.ts @@ -10,6 +10,12 @@ import { repoPath } from "../helpers/repo-root"; import { normalizeThinkingOptions, resolveThinkingEnabled } from "../../src/clients/reasoning-controls"; import type { OcxProviderConfig } from "../../src/types"; +test.each(["tool_search", "computer_use", "image_gen", "mcp"])("preserves direct hosted selector %s in the raw Responses body", type => { + const choice = type === "mcp" ? { type, server_label: "docs", name: null } : { type }; + const raw = { model: "fixture", input: "hello", tool_choice: choice }; + expect(parseRequest(raw)._rawBody?.tool_choice).toEqual(choice); +}); + describe("generic effort-only client compatibility", () => { const provider: OcxProviderConfig = { adapter: "openai-chat", baseUrl: "https://fixture.invalid/v1", diff --git a/tests/server/server-combo-required-tool-contract.test.ts b/tests/server/server-combo-required-tool-contract.test.ts index 03fa2e19a5c..00bf5d0b913 100644 --- a/tests/server/server-combo-required-tool-contract.test.ts +++ b/tests/server/server-combo-required-tool-contract.test.ts @@ -165,9 +165,11 @@ function requiredRequest(extra: Record = {}): Request { } describe("combo required-tool contract on buffered children (#7456)", () => { - for (const choice of [{ type: "web_search" }, { type: "mcp", server_label: "docs" }]) { + for (const choice of [{ type: "web_search" }, { type: "mcp", server_label: "docs" }, + { type: "tool_search" }, { type: "computer_use" }, { type: "image_gen" }, + { type: "mcp", server_label: "docs", name: null }]) { for (const stream of [false, true]) { - test(`direct hosted choice skips Meta before send (${choice.type}, stream=${stream})`, async () => { + test(`direct hosted choice skips Meta before send (${JSON.stringify(choice)}, stream=${stream})`, async () => { const bodies: unknown[] = []; let metaSends = 0; const upstream = serve(async request => { bodies.push(await request.json()); @@ -184,10 +186,12 @@ describe("combo required-tool contract on buffered children (#7456)", () => { expect(response.status).toBe(200); expect(await response.text()).toContain("hosted terminal owner"); expect(metaSends).toBe(0); expect(bodies).toHaveLength(1); - expect(bodies[0]).toMatchObject({ tool_choice: choice }); + // Noncanonical Responses targets retain the existing private tool-search lowering. + const upstreamChoice = choice.type === "tool_search" ? { type: "function", name: "tool_search" } : choice; + expect(bodies[0]).toMatchObject({ tool_choice: upstreamChoice }); }); } - test(`direct hosted choice reports all-Meta exhaustion (${choice.type})`, async () => { + test(`direct hosted choice reports all-Meta exhaustion (${JSON.stringify(choice)})`, async () => { let sends = 0; const meta = provider("openai-responses", "https://api.meta.ai/v1", "key-a", { fetch: (async () => { sends++; throw new Error("unexpected Meta send"); }) as typeof fetch }); @@ -200,7 +204,8 @@ describe("combo required-tool contract on buffered children (#7456)", () => { test("direct hosted selectors keep their terminal owner and reject malformed selector shapes", async () => { const { hasMandatoryToolChoice, requiredToolContractOf } = await import("../../src/server/responses/combo-required-tool-contract"); const { createComboRequiredToolStreamGuard } = await import("../../src/server/responses/combo-required-tool-stream"); - for (const choice of [{ type: "web_search" }, { type: "mcp", server_label: "docs" }, { type: "tool_search" }]) { + for (const choice of [{ type: "web_search" }, { type: "mcp", server_label: "docs" }, { type: "tool_search" }, + { type: "mcp", server_label: "docs", name: null }, { type: "computer_use" }, { type: "image_gen" }]) { const body = { tools: [choice], tool_choice: choice }; expect(hasMandatoryToolChoice(body)).toBe(true); expect(requiredToolContractOf(body)).toBeUndefined(); @@ -208,7 +213,8 @@ describe("combo required-tool contract on buffered children (#7456)", () => { } for (const choice of [{ type: "web_search", name: 5 }, { type: "web_search", name: " " }, { type: "mcp" }, { type: "mcp", server_label: 5 }, { type: "mcp", server_label: " " }, - { type: "mcp", server_label: "docs", name: " " }, { type: "unknown_hosted" }]) { + { type: "mcp", server_label: "docs", name: " " }, { type: "mcp", server_label: "docs", name: 123 }, + { type: "function", name: null }, { type: "custom", name: null }, { type: "unknown_hosted" }]) { expect(hasMandatoryToolChoice({ tool_choice: choice })).toBe(false); } }); From 1d216a787a75ba7505976cf70c2c306f6c02dc0a Mon Sep 17 00:00:00 2001 From: Epinephrine Date: Sat, 31 Oct 2026 02:38:07 +0900 Subject: [PATCH 39/42] fix(combos): preserve mandatory choices across Chat translation --- docs-site/src/content/docs/guides/combos.md | 4 +- src/responses/parser-tools.ts | 1 + src/responses/schema.ts | 5 ++- .../responses/combo-required-tool-contract.ts | 8 ++++ src/server/responses/core-combo.ts | 10 ++--- .../transports/responses-combo-boundaries.md | 3 +- tests/responses/responses-parser.test.ts | 17 ++++++++ ...erver-combo-required-tool-contract.test.ts | 42 ++++++++++++++++++- 8 files changed, 80 insertions(+), 10 deletions(-) diff --git a/docs-site/src/content/docs/guides/combos.md b/docs-site/src/content/docs/guides/combos.md index 4c1c1083555..dcd4fc313eb 100644 --- a/docs-site/src/content/docs/guides/combos.md +++ b/docs-site/src/content/docs/guides/combos.md @@ -156,8 +156,8 @@ For Responses requests that require a tool, Meta Responses targets are skipped b because Meta does not support mandatory `tool_choice`. An eligible later target receives the original choice, subject to its existing adapter compatibility rules. This includes hosted choices such as `web_search`, `tool_search`, `computer_use` and `image_gen`, and MCP choices with a -nonempty `server_label` and an omitted, null or nonempty name. If every target is incompatible for this reason, -the combo reports the constraint. Malformed or unknown choices retain the provider's ordinary error path. +nonempty `server_label` and an omitted, null or nonempty name, including inside required allowed-tools lists. Chat translation preserves a forced `tool_search` as the synthetic function choice. If every target is incompatible for this reason, +the combo reports the constraint. Chat targets that cannot preserve a mandatory selector are also skipped before sending. Malformed or unknown choices retain the provider's ordinary error path. `failover` selects the first eligible target in configuration order. A target is eligible when its provider exists, is enabled, is not cooling down, and can handle any special request constraint. diff --git a/src/responses/parser-tools.ts b/src/responses/parser-tools.ts index 763b061c8f6..ee0268d524c 100644 --- a/src/responses/parser-tools.ts +++ b/src/responses/parser-tools.ts @@ -16,6 +16,7 @@ export function mapToolChoice(value: unknown): OcxRequestOptions["toolChoice"] { if (t === "image_generation" || t === "image_gen") { return { name: IMAGE_GEN_TOOL_NAME }; } + if (t === "tool_search") return { name: "tool_search" }; if (t === "allowed_tools" && Array.isArray(value.tools)) { const names = value.tools .map(allowedToolName) diff --git a/src/responses/schema.ts b/src/responses/schema.ts index d2b71011003..213e443d2d4 100644 --- a/src/responses/schema.ts +++ b/src/responses/schema.ts @@ -167,7 +167,10 @@ export const HOSTED_TOOL_TYPES = [ const hostedToolType = z.enum(HOSTED_TOOL_TYPES); -const allowedToolEntrySchema = z.object({ type: z.string(), name: z.string().optional() }); +const allowedToolEntrySchema = z.union([ + z.object({ type: z.literal("mcp"), server_label: z.string().min(1), name: z.string().nullable().optional() }), + z.object({ type: z.string(), name: z.string().optional() }), +]); export const toolChoiceSchema = z.union([ z.literal("auto"), diff --git a/src/server/responses/combo-required-tool-contract.ts b/src/server/responses/combo-required-tool-contract.ts index 891af59e4bf..df0b90493f8 100644 --- a/src/server/responses/combo-required-tool-contract.ts +++ b/src/server/responses/combo-required-tool-contract.ts @@ -1,4 +1,5 @@ import { REQUIRED_TOOL_CONTRACT_CODE } from "../../lib/errors"; +import { mapToolChoice } from "../../responses/parser-tools"; import { hasResponseMediaType } from "../../lib/http-response-semantics"; import { applyReplayRefusalClientHeaders, isNonReplayableResponse, markResponseNonReplayable } from "../../lib/upstream-retry"; import { clientWireOf } from "../inference/client-wire"; @@ -39,6 +40,13 @@ export function hasMandatoryToolChoice(rawBody: unknown): boolean { && selector.tools.every(knownToolSelector); } +/** A translated Chat target must not turn a mandatory selector into auto or none. */ +export function chatCanPreserveMandatoryToolChoice(rawBody: unknown): boolean { + if (!hasMandatoryToolChoice(rawBody)) return true; + const mapped = mapToolChoice((rawBody as Record).tool_choice); + return mapped === "required" || (!!mapped && typeof mapped === "object"); +} + const hostedCalls = new Map([ ["web_search", "web_search_call"], ["web_search_preview", "web_search_call"], ["file_search", "file_search_call"], ["code_interpreter", "code_interpreter_call"], diff --git a/src/server/responses/core-combo.ts b/src/server/responses/core-combo.ts index f2de582343d..edf85af21bf 100644 --- a/src/server/responses/core-combo.ts +++ b/src/server/responses/core-combo.ts @@ -1,7 +1,7 @@ import { effectiveAzureOutputBudget } from "../../adapters/azure"; import { buildResponsesTransport } from "../../adapters/openai-responses/transport"; import { isMetaAiResponsesDestination } from "../../responses/muse-tool-name-alias"; -import { hasMandatoryToolChoice } from "./combo-required-tool-contract"; +import { chatCanPreserveMandatoryToolChoice, hasMandatoryToolChoice } from "./combo-required-tool-contract"; import { isCodexAccountCooldownResponse } from "../../codex/account-cooldown-response"; import { resolveWireProtocolOverride } from "../adapter-resolve"; import { comboCooldownRetryAfterSeconds, isDefiniteContextOverflow, type CodexAccountModelRefusal } from "../../combos/failover"; @@ -544,8 +544,8 @@ export async function executeComboResponses( const policy = captureRouteStaticPolicy(route.providerName, route.modelId, route.provider, route.staticPolicy.effectiveAlias, inboundWire); const provider = resolveWireProtocolOverride(route.providerName, route.modelId, route.provider, inboundWire, policy); - return provider.adapter !== "openai-responses" - || !isMetaAiResponsesDestination(buildResponsesTransport(provider, { headers: req.headers }).url); + return provider.adapter === "openai-chat" ? chatCanPreserveMandatoryToolChoice(body) + : provider.adapter !== "openai-responses" || !isMetaAiResponsesDestination(buildResponsesTransport(provider, { headers: req.headers }).url); } catch { return true; } @@ -674,7 +674,7 @@ export async function executeComboResponses( { parentThreadId: inboundClientThreadId }, ); if (!options.abortSignal?.aborted && onlyMuseToolIncompatible()) { - return comboUnavailableResponse(`No available targets for combo: ${comboId}; Meta Responses does not support mandatory tool_choice`); + return comboUnavailableResponse(`No available targets for combo: ${comboId}; targets cannot preserve mandatory tool_choice`); } return options.abortSignal?.aborted ? clientCancelledResponse() @@ -694,7 +694,7 @@ export async function executeComboResponses( if (protocolRefusal) return protocolRefusal; if (onlyReplayIncompatibleTargetsRemain()) return targetIncompatibleResponse(); if (!options.abortSignal?.aborted && onlyMuseToolIncompatible()) { - return comboUnavailableResponse(`No available targets for combo: ${comboId}; Meta Responses does not support mandatory tool_choice`); + return comboUnavailableResponse(`No available targets for combo: ${comboId}; targets cannot preserve mandatory tool_choice`); } return options.abortSignal?.aborted ? clientCancelledResponse() diff --git a/structure/transports/responses-combo-boundaries.md b/structure/transports/responses-combo-boundaries.md index 15e87fd84eb..f3c5276b697 100644 --- a/structure/transports/responses-combo-boundaries.md +++ b/structure/transports/responses-combo-boundaries.md @@ -27,7 +27,8 @@ transport URL; malformed and unknown selectors retain their ordinary error path. receives the original selector, and an all-Meta exhaustion names the constraint before any send. The child parser accepts `tool_search`, `computer_use` and `image_gen` as recognized hosted choices. Its raw body preserves the selector; existing destination compatibility, including routed private -tool-search lowering, still applies after parsing. This does not expand Chat `mapToolChoice`. +tool-search lowering, still applies after parsing. Chat translation maps a direct `tool_search` +choice to its existing synthetic function name. MCP null names also parse inside required allowed-tools lists; function/custom null names remain invalid. A Chat destination is skipped before send when translation would degrade a mandatory choice to auto/none; exhaustion reports the tool-choice constraint. Coverage: `tests/server/server-combo-required-tool-contract.test.ts`. Before combo success bookkeeping, native delivery in `src/server/responses/passthrough-delivery.ts` diff --git a/tests/responses/responses-parser.test.ts b/tests/responses/responses-parser.test.ts index 8cb83aac088..6387308433d 100644 --- a/tests/responses/responses-parser.test.ts +++ b/tests/responses/responses-parser.test.ts @@ -16,6 +16,23 @@ test.each(["tool_search", "computer_use", "image_gen", "mcp"])("preserves direct expect(parseRequest(raw)._rawBody?.tool_choice).toEqual(choice); }); +test("translated tool_search keeps the mandatory synthetic function choice", () => { + const parsed = parseRequest({ model: "fixture", input: "hello", tools: [{ type: "tool_search" }], + tool_choice: { type: "tool_search" } }); + expect(parsed.options.toolChoice).toEqual({ name: "tool_search" }); + expect(parsed.context.tools?.map(tool => tool.name)).toContain("tool_search"); +}); + +test("MCP allowed-tools null name preserves its raw selector without accepting function null names", () => { + const choice = { type: "allowed_tools", mode: "required", tools: [{ type: "mcp", server_label: "docs", name: null }] }; + expect(parseRequest({ model: "fixture", input: "hello", tool_choice: choice })._rawBody?.tool_choice).toEqual(choice); + for (const type of ["function", "custom"]) { + expect(() => parseRequest({ model: "fixture", input: "hello", tool_choice: { + ...choice, tools: [{ type, name: null }], + } })).toThrow(); + } +}); + describe("generic effort-only client compatibility", () => { const provider: OcxProviderConfig = { adapter: "openai-chat", baseUrl: "https://fixture.invalid/v1", diff --git a/tests/server/server-combo-required-tool-contract.test.ts b/tests/server/server-combo-required-tool-contract.test.ts index 00bf5d0b913..a35fb8d4930 100644 --- a/tests/server/server-combo-required-tool-contract.test.ts +++ b/tests/server/server-combo-required-tool-contract.test.ts @@ -165,9 +165,49 @@ function requiredRequest(extra: Record = {}): Request { } describe("combo required-tool contract on buffered children (#7456)", () => { + for (const choice of [{ type: "computer_use" }, + { type: "allowed_tools", mode: "required", tools: [{ type: "mcp", server_label: "docs", name: null }] }]) { + test(`untranslatable mandatory Chat choice skips before send: ${JSON.stringify(choice)}`, async () => { + let chatSends = 0; let backupSends = 0; + const chat = provider("openai-chat", "https://chat.fixture.invalid/v1", "key-a", { + fetch: (async () => { chatSends++; throw new Error("unexpected Chat send"); }) as typeof fetch }); + const backup = serve(() => { backupSends++; return Response.json(completedNoCall("m2", "eligible Responses target")); }); + const response = await handleResponses(requiredRequest({ tool_choice: choice }), + comboConfig({ a: chat, b: provider("openai-responses", baseUrl(backup), "key-b") }), {} as RequestLogContext); + expect(response.status).toBe(200); expect(await response.text()).toContain("eligible Responses target"); + expect(chatSends).toBe(0); expect(backupSends).toBe(1); + }); + test(`untranslatable mandatory Chat choice fails closed without a backup: ${JSON.stringify(choice)}`, async () => { + let sends = 0; + const chat = provider("openai-chat", "https://chat.fixture.invalid/v1", "key-a", { + fetch: (async () => { sends++; throw new Error("unexpected Chat send"); }) as typeof fetch }); + const response = await handleResponses(requiredRequest({ tool_choice: choice }), comboConfig({ a: chat }), {} as RequestLogContext); + expect(response.status).toBe(503); expect(await response.text()).toContain("mandatory tool_choice"); + expect(sends).toBe(0); + }); + } + + test("translated Chat child preserves forced tool_search on its actual wire", async () => { + const bodies: unknown[] = []; + const upstream = serve(async request => { + bodies.push(await request.json()); + return Response.json({ id: "chat-forced-search", object: "chat.completion", + choices: [{ index: 0, message: { role: "assistant", content: null, tool_calls: [{ + id: "call_search", type: "function", function: { name: "tool_search", arguments: '{"query":"lookup"}' }, + }] }, finish_reason: "tool_calls" }], usage: { prompt_tokens: 1, completion_tokens: 1, total_tokens: 2 } }); + }); + const response = await handleResponses(requiredRequest({ tools: [{ type: "tool_search" }], + tool_choice: { type: "tool_search" } }), comboConfig({ a: provider("openai-chat", baseUrl(upstream), "key-a") }), {} as RequestLogContext); + expect(response.status).toBe(200); + expect(await response.text()).toContain("tool_search"); + expect(bodies).toHaveLength(1); + expect(bodies[0]).toMatchObject({ tool_choice: { type: "function", function: { name: "tool_search" } } }); + }); + for (const choice of [{ type: "web_search" }, { type: "mcp", server_label: "docs" }, { type: "tool_search" }, { type: "computer_use" }, { type: "image_gen" }, - { type: "mcp", server_label: "docs", name: null }]) { + { type: "mcp", server_label: "docs", name: null }, + { type: "allowed_tools", mode: "required", tools: [{ type: "mcp", server_label: "docs", name: null }] }]) { for (const stream of [false, true]) { test(`direct hosted choice skips Meta before send (${JSON.stringify(choice)}, stream=${stream})`, async () => { const bodies: unknown[] = []; let metaSends = 0; From 917ea053146a87ca13b03963c9ca7c9c31c0c9d4 Mon Sep 17 00:00:00 2001 From: Epinephrine Date: Sat, 31 Oct 2026 03:33:39 +0900 Subject: [PATCH 40/42] fix(combos): honor auto-only policy for resolved Chat selectors --- docs-site/src/content/docs/guides/combos.md | 2 +- .../responses/combo-required-tool-contract.ts | 4 +- src/server/responses/core-combo.ts | 6 +- .../transports/responses-combo-boundaries.md | 2 +- ...erver-combo-required-tool-contract.test.ts | 58 ++++++++++++++++++- 5 files changed, 64 insertions(+), 8 deletions(-) diff --git a/docs-site/src/content/docs/guides/combos.md b/docs-site/src/content/docs/guides/combos.md index dcd4fc313eb..1ed94376365 100644 --- a/docs-site/src/content/docs/guides/combos.md +++ b/docs-site/src/content/docs/guides/combos.md @@ -157,7 +157,7 @@ because Meta does not support mandatory `tool_choice`. An eligible later target original choice, subject to its existing adapter compatibility rules. This includes hosted choices such as `web_search`, `tool_search`, `computer_use` and `image_gen`, and MCP choices with a nonempty `server_label` and an omitted, null or nonempty name, including inside required allowed-tools lists. Chat translation preserves a forced `tool_search` as the synthetic function choice. If every target is incompatible for this reason, -the combo reports the constraint. Chat targets that cannot preserve a mandatory selector are also skipped before sending. Malformed or unknown choices retain the provider's ordinary error path. +the combo reports the constraint. Chat targets that cannot preserve a mandatory selector, including models configured for auto-only tool choice, are also skipped before sending. Malformed or unknown choices retain the provider's ordinary error path. `failover` selects the first eligible target in configuration order. A target is eligible when its provider exists, is enabled, is not cooling down, and can handle any special request constraint. diff --git a/src/server/responses/combo-required-tool-contract.ts b/src/server/responses/combo-required-tool-contract.ts index df0b90493f8..a5177b364d8 100644 --- a/src/server/responses/combo-required-tool-contract.ts +++ b/src/server/responses/combo-required-tool-contract.ts @@ -1,5 +1,6 @@ import { REQUIRED_TOOL_CONTRACT_CODE } from "../../lib/errors"; import { mapToolChoice } from "../../responses/parser-tools"; +import { modelInList, type OcxProviderConfig } from "../../types"; import { hasResponseMediaType } from "../../lib/http-response-semantics"; import { applyReplayRefusalClientHeaders, isNonReplayableResponse, markResponseNonReplayable } from "../../lib/upstream-retry"; import { clientWireOf } from "../inference/client-wire"; @@ -41,8 +42,9 @@ export function hasMandatoryToolChoice(rawBody: unknown): boolean { } /** A translated Chat target must not turn a mandatory selector into auto or none. */ -export function chatCanPreserveMandatoryToolChoice(rawBody: unknown): boolean { +export function chatCanPreserveMandatoryToolChoice(rawBody: unknown, provider: OcxProviderConfig, modelId: string): boolean { if (!hasMandatoryToolChoice(rawBody)) return true; + if (modelInList(provider.autoToolChoiceOnlyModels, modelId)) return false; const mapped = mapToolChoice((rawBody as Record).tool_choice); return mapped === "required" || (!!mapped && typeof mapped === "object"); } diff --git a/src/server/responses/core-combo.ts b/src/server/responses/core-combo.ts index edf85af21bf..ab18b2fefb0 100644 --- a/src/server/responses/core-combo.ts +++ b/src/server/responses/core-combo.ts @@ -4,6 +4,7 @@ import { isMetaAiResponsesDestination } from "../../responses/muse-tool-name-ali import { chatCanPreserveMandatoryToolChoice, hasMandatoryToolChoice } from "./combo-required-tool-contract"; import { isCodexAccountCooldownResponse } from "../../codex/account-cooldown-response"; import { resolveWireProtocolOverride } from "../adapter-resolve"; +import { parseSyntheticRowId } from "../fast-row"; import { comboCooldownRetryAfterSeconds, isDefiniteContextOverflow, type CodexAccountModelRefusal } from "../../combos/failover"; import { fallbackTriggerForStatus, guardFallbackHop } from "../../routing/fallback-policy"; import { @@ -539,12 +540,13 @@ export async function executeComboResponses( const museToolEligible = (target: (typeof combo.targets)[number]): boolean => { if (!hasMandatoryToolChoice(body)) return true; try { - const route = routeConcreteModel(config, `${target.provider}/${target.model}`); + const selector = `${target.provider}/${target.model}`, { fastRow, effortRow } = parseSyntheticRowId(selector, config); + const route = routeConcreteModel(config, fastRow?.baseId ?? effortRow?.baseId ?? selector); const inboundWire = options.inboundWire ?? "responses"; const policy = captureRouteStaticPolicy(route.providerName, route.modelId, route.provider, route.staticPolicy.effectiveAlias, inboundWire); const provider = resolveWireProtocolOverride(route.providerName, route.modelId, route.provider, inboundWire, policy); - return provider.adapter === "openai-chat" ? chatCanPreserveMandatoryToolChoice(body) + return provider.adapter === "openai-chat" ? chatCanPreserveMandatoryToolChoice(body, provider, route.modelId) : provider.adapter !== "openai-responses" || !isMetaAiResponsesDestination(buildResponsesTransport(provider, { headers: req.headers }).url); } catch { return true; diff --git a/structure/transports/responses-combo-boundaries.md b/structure/transports/responses-combo-boundaries.md index f3c5276b697..5e2771bf7fa 100644 --- a/structure/transports/responses-combo-boundaries.md +++ b/structure/transports/responses-combo-boundaries.md @@ -28,7 +28,7 @@ receives the original selector, and an all-Meta exhaustion names the constraint The child parser accepts `tool_search`, `computer_use` and `image_gen` as recognized hosted choices. Its raw body preserves the selector; existing destination compatibility, including routed private tool-search lowering, still applies after parsing. Chat translation maps a direct `tool_search` -choice to its existing synthetic function name. MCP null names also parse inside required allowed-tools lists; function/custom null names remain invalid. A Chat destination is skipped before send when translation would degrade a mandatory choice to auto/none; exhaustion reports the tool-choice constraint. +choice to its existing synthetic function name. MCP null names also parse inside required allowed-tools lists; function/custom null names remain invalid. A Chat destination is skipped before send when translation would degrade a mandatory choice to auto/none, including the resolved model's auto-only tool-choice policy; exhaustion reports the tool-choice constraint. Eligibility resolves synthetic Fast/effort selectors before model policy, matching child request preparation while preserving exact configured IDs. Coverage: `tests/server/server-combo-required-tool-contract.test.ts`. Before combo success bookkeeping, native delivery in `src/server/responses/passthrough-delivery.ts` diff --git a/tests/server/server-combo-required-tool-contract.test.ts b/tests/server/server-combo-required-tool-contract.test.ts index a35fb8d4930..4c11a16050d 100644 --- a/tests/server/server-combo-required-tool-contract.test.ts +++ b/tests/server/server-combo-required-tool-contract.test.ts @@ -165,6 +165,37 @@ function requiredRequest(extra: Record = {}): Request { } describe("combo required-tool contract on buffered children (#7456)", () => { + for (const choice of ["required", { type: "function", name: "tool_search" }, { type: "tool_search" }, + { type: "allowed_tools", mode: "required", tools: [{ type: "tool_search" }] }]) { + for (const [backup, suffix] of [[false, ""], [true, ""], [false, "--fast"], [true, "--fast"], [false, "--high"], [true, "--high"]] as const) { + test(`auto-only Chat cannot consume a mandatory choice (${JSON.stringify(choice)}, backup=${backup}, suffix=${suffix})`, async () => { + const sent: Array<{ model: string; tool_choice: unknown }> = []; + const upstream = serve(async request => { + const body = await request.json() as { model: string; tool_choice: unknown }; sent.push(body); + return Response.json({ id: "chat-auto-policy", object: "chat.completion", + choices: [{ index: 0, message: { role: "assistant", content: null, tool_calls: [{ + id: "call_search", type: "function", function: { name: "tool_search", arguments: '{"query":"lookup"}' }, + }] }, finish_reason: "tool_calls" }], usage: { prompt_tokens: 1, completion_tokens: 1, total_tokens: 2 } }); + }); + const providers = { a: provider("openai-chat", baseUrl(upstream), "key-a", { autoToolChoiceOnlyModels: ["m1"] }), + ...(backup ? { b: provider("openai-chat", baseUrl(upstream), "key-b") } : {}) }; + const cfg = comboConfig(providers); + cfg.cursorEffortRows = suffix === "--high"; + cfg.combos!.free!.targets[0]!.model += suffix; + const response = await handleResponses(requiredRequest({ tools: [{ type: "tool_search" }], tool_choice: choice }), + cfg, {} as RequestLogContext); + expect(response.status).toBe(backup ? 200 : 503); + const text = await response.text(); + if (!backup) { expect(text).toContain("mandatory tool_choice"); expect(sent).toEqual([]); } + else { + expect(sent).toHaveLength(1); expect(sent[0]?.model).toBe("m2"); + expect(sent[0]?.tool_choice).toEqual(typeof choice === "string" || choice.type === "allowed_tools" + ? "required" : { type: "function", function: { name: "tool_search" } }); + } + }); + } + } + for (const choice of [{ type: "computer_use" }, { type: "allowed_tools", mode: "required", tools: [{ type: "mcp", server_label: "docs", name: null }] }]) { test(`untranslatable mandatory Chat choice skips before send: ${JSON.stringify(choice)}`, async () => { @@ -187,7 +218,7 @@ describe("combo required-tool contract on buffered children (#7456)", () => { }); } - test("translated Chat child preserves forced tool_search on its actual wire", async () => { + test.each(["ordinary", "exact-suffix", "disabled-fast"])("translated Chat child preserves forced tool_search on its actual wire: %s", async mode => { const bodies: unknown[] = []; const upstream = serve(async request => { bodies.push(await request.json()); @@ -196,12 +227,33 @@ describe("combo required-tool contract on buffered children (#7456)", () => { id: "call_search", type: "function", function: { name: "tool_search", arguments: '{"query":"lookup"}' }, }] }, finish_reason: "tool_calls" }], usage: { prompt_tokens: 1, completion_tokens: 1, total_tokens: 2 } }); }); + const model = mode === "ordinary" ? "m1" : "m1--fast"; + const cfg = comboConfig({ a: provider("openai-chat", baseUrl(upstream), "key-a", { + autoToolChoiceOnlyModels: [mode === "ordinary" ? "another-model" : "m1"], + ...(mode === "exact-suffix" ? { models: [model] } : {}), + }) }); + cfg.combos!.free!.targets[0]!.model = model; + if (mode === "disabled-fast") cfg.fastRows = false; const response = await handleResponses(requiredRequest({ tools: [{ type: "tool_search" }], - tool_choice: { type: "tool_search" } }), comboConfig({ a: provider("openai-chat", baseUrl(upstream), "key-a") }), {} as RequestLogContext); + tool_choice: { type: "tool_search" } }), cfg, {} as RequestLogContext); expect(response.status).toBe(200); expect(await response.text()).toContain("tool_search"); expect(bodies).toHaveLength(1); - expect(bodies[0]).toMatchObject({ tool_choice: { type: "function", function: { name: "tool_search" } } }); + expect(bodies[0]).toMatchObject({ model, tool_choice: { type: "function", function: { name: "tool_search" } } }); + }); + + test.each(["auto", "none"])("auto-only Chat retains optional %s behavior on its wire", async choice => { + const bodies: unknown[] = []; + const upstream = serve(async request => { bodies.push(await request.json()); + return Response.json({ id: "chat-optional", object: "chat.completion", + choices: [{ index: 0, message: { role: "assistant", content: "optional done" }, finish_reason: "stop" }] }); }); + const response = await handleResponses(requiredRequest({ tool_choice: choice }), comboConfig({ + a: provider("openai-chat", baseUrl(upstream), "key-a", { autoToolChoiceOnlyModels: ["m1"] }), + }), {} as RequestLogContext); + expect(response.status).toBe(200); expect(await response.text()).toContain("optional done"); + expect(bodies).toHaveLength(1); expect(bodies[0]).toMatchObject({ model: "m1" }); + if (choice === "auto") expect(bodies[0]).toMatchObject({ tool_choice: "auto" }); + else { expect(bodies[0]).not.toHaveProperty("tools"); expect(bodies[0]).not.toHaveProperty("tool_choice"); } }); for (const choice of [{ type: "web_search" }, { type: "mcp", server_label: "docs" }, From 2dd3cfb266cd62ada475a50a7b71951675a65abe Mon Sep 17 00:00:00 2001 From: Epinephrine Date: Sat, 31 Oct 2026 04:17:09 +0900 Subject: [PATCH 41/42] fix(combos): resolve virtual wire policy before tool eligibility --- docs-site/src/content/docs/guides/combos.md | 2 +- src/server/responses/core-combo.ts | 8 ++-- .../transports/responses-combo-boundaries.md | 2 +- ...erver-combo-required-tool-contract.test.ts | 38 +++++++++++++++++++ 4 files changed, 45 insertions(+), 5 deletions(-) diff --git a/docs-site/src/content/docs/guides/combos.md b/docs-site/src/content/docs/guides/combos.md index 1ed94376365..91ffeb34b0d 100644 --- a/docs-site/src/content/docs/guides/combos.md +++ b/docs-site/src/content/docs/guides/combos.md @@ -157,7 +157,7 @@ because Meta does not support mandatory `tool_choice`. An eligible later target original choice, subject to its existing adapter compatibility rules. This includes hosted choices such as `web_search`, `tool_search`, `computer_use` and `image_gen`, and MCP choices with a nonempty `server_label` and an omitted, null or nonempty name, including inside required allowed-tools lists. Chat translation preserves a forced `tool_search` as the synthetic function choice. If every target is incompatible for this reason, -the combo reports the constraint. Chat targets that cannot preserve a mandatory selector, including models configured for auto-only tool choice, are also skipped before sending. Malformed or unknown choices retain the provider's ordinary error path. +the combo reports the constraint. Chat targets that cannot preserve a mandatory selector, including models configured for auto-only tool choice, are also skipped before sending. Registered OpenAI API virtual models use the actual wire model's adapter and tool-choice policy for this decision. Malformed or unknown choices retain the provider's ordinary error path. `failover` selects the first eligible target in configuration order. A target is eligible when its provider exists, is enabled, is not cooling down, and can handle any special request constraint. diff --git a/src/server/responses/core-combo.ts b/src/server/responses/core-combo.ts index ab18b2fefb0..20dae280c0a 100644 --- a/src/server/responses/core-combo.ts +++ b/src/server/responses/core-combo.ts @@ -5,6 +5,7 @@ import { chatCanPreserveMandatoryToolChoice, hasMandatoryToolChoice } from "./co import { isCodexAccountCooldownResponse } from "../../codex/account-cooldown-response"; import { resolveWireProtocolOverride } from "../adapter-resolve"; import { parseSyntheticRowId } from "../fast-row"; +import { resolveOpenAiVirtualModel } from "../../providers/openai-virtual-models"; import { comboCooldownRetryAfterSeconds, isDefiniteContextOverflow, type CodexAccountModelRefusal } from "../../combos/failover"; import { fallbackTriggerForStatus, guardFallbackHop } from "../../routing/fallback-policy"; import { @@ -542,11 +543,12 @@ export async function executeComboResponses( try { const selector = `${target.provider}/${target.model}`, { fastRow, effortRow } = parseSyntheticRowId(selector, config); const route = routeConcreteModel(config, fastRow?.baseId ?? effortRow?.baseId ?? selector); + const modelId = resolveOpenAiVirtualModel(route.providerName, route.modelId)?.wireModelId ?? route.modelId; const inboundWire = options.inboundWire ?? "responses"; - const policy = captureRouteStaticPolicy(route.providerName, route.modelId, route.provider, + const policy = captureRouteStaticPolicy(route.providerName, modelId, route.provider, route.staticPolicy.effectiveAlias, inboundWire); - const provider = resolveWireProtocolOverride(route.providerName, route.modelId, route.provider, inboundWire, policy); - return provider.adapter === "openai-chat" ? chatCanPreserveMandatoryToolChoice(body, provider, route.modelId) + const provider = resolveWireProtocolOverride(route.providerName, modelId, route.provider, inboundWire, policy); + return provider.adapter === "openai-chat" ? chatCanPreserveMandatoryToolChoice(body, provider, modelId) : provider.adapter !== "openai-responses" || !isMetaAiResponsesDestination(buildResponsesTransport(provider, { headers: req.headers }).url); } catch { return true; diff --git a/structure/transports/responses-combo-boundaries.md b/structure/transports/responses-combo-boundaries.md index 5e2771bf7fa..4ffdf66af74 100644 --- a/structure/transports/responses-combo-boundaries.md +++ b/structure/transports/responses-combo-boundaries.md @@ -29,7 +29,7 @@ The child parser accepts `tool_search`, `computer_use` and `image_gen` as recogn Its raw body preserves the selector; existing destination compatibility, including routed private tool-search lowering, still applies after parsing. Chat translation maps a direct `tool_search` choice to its existing synthetic function name. MCP null names also parse inside required allowed-tools lists; function/custom null names remain invalid. A Chat destination is skipped before send when translation would degrade a mandatory choice to auto/none, including the resolved model's auto-only tool-choice policy; exhaustion reports the tool-choice constraint. Eligibility resolves synthetic Fast/effort selectors before model policy, matching child request preparation while preserving exact configured IDs. -Coverage: `tests/server/server-combo-required-tool-contract.test.ts`. +Registered OpenAI API virtual models resolve to their wire model before adapter and auto-only policy checks, using the same registry resolver as child normalization. Other providers and unregistered IDs remain literal. Coverage: `tests/server/server-combo-required-tool-contract.test.ts`. Before combo success bookkeeping, native delivery in `src/server/responses/passthrough-delivery.ts` uses `src/server/responses/combo-required-tool-contract.ts` to check successful native diff --git a/tests/server/server-combo-required-tool-contract.test.ts b/tests/server/server-combo-required-tool-contract.test.ts index 4c11a16050d..35d57202d0d 100644 --- a/tests/server/server-combo-required-tool-contract.test.ts +++ b/tests/server/server-combo-required-tool-contract.test.ts @@ -165,6 +165,44 @@ function requiredRequest(extra: Record = {}): Request { } describe("combo required-tool contract on buffered children (#7456)", () => { + test.each(["chat", "chat-fast", "chat-effort", "chat-only", "wire-chat", "virtual-chat", "other-provider", "unknown-virtual"])( + "virtual Chat eligibility follows the actual wire policy: %s", async mode => { + const virtual = mode === "unknown-virtual" ? "gpt-5.6-sol-pro-unknown" : "gpt-5.6-sol-pro"; + const wire = "gpt-5.6-sol", name = mode === "other-provider" ? "fixture" : "openai-apikey"; + const sent: Array<{ path: string; model: string; tool_choice: unknown }> = []; + const fetchWire = (async (input, init) => { + const body = JSON.parse(String(init?.body ?? "{}")) as { model: string; tool_choice: unknown }; + const path = new URL(input instanceof globalThis.Request ? input.url : String(input)).pathname; sent.push({ path, ...body }); + return path.endsWith("/responses") ? Response.json(completedNoCall(body.model, "virtual wire response")) + : Response.json({ id: "chat-virtual", object: "chat.completion", choices: [{ index: 0, + message: { role: "assistant", content: null, tool_calls: [{ id: "call_search", type: "function", + function: { name: "tool_search", arguments: '{"query":"lookup"}' } }] }, finish_reason: "tool_calls" }] }); + }) as typeof fetch; + const primary = provider(mode === "wire-chat" || mode === "virtual-chat" ? "openai-responses" : "openai-chat", "https://api.openai.com/v1", "key-a", { + fetch: fetchWire, + models: [virtual], autoToolChoiceOnlyModels: [mode === "virtual-chat" ? virtual : wire], + // The registry defaults this provider to Responses; explicit per-model overrides + // distinguish virtual and wire policy, rather than relying on the provider fallback. + ...(mode.startsWith("chat") ? { modelAdapters: { [virtual]: "openai-chat", [wire]: "openai-chat" } } + : mode === "wire-chat" ? { modelAdapters: { [wire]: "openai-chat" } } + : mode === "virtual-chat" || mode === "unknown-virtual" ? { modelAdapters: { [virtual]: "openai-chat" } } : {}), + }); + const cfg = comboConfig({ [name]: primary, ...(mode === "chat-only" ? {} : { backup: provider("openai-chat", "https://backup.fixture.invalid/v1", "key-b", { fetch: fetchWire }) }) }); + cfg.combos!.free!.targets[0]!.model = virtual + (mode === "chat-fast" ? "--fast" : mode === "chat-effort" ? "--high" : ""); + cfg.cursorEffortRows = mode === "chat-effort"; + const response = await handleResponses(requiredRequest({ tools: [{ type: "tool_search" }], tool_choice: { type: "tool_search" } }), cfg, {} as RequestLogContext); + if (mode === "chat-only") { + expect(response.status).toBe(503); expect(await response.text()).toContain("mandatory tool_choice"); expect(sent).toEqual([]); return; + } + expect(response.status).toBe(200); await response.text(); + expect(sent).toHaveLength(1); + if (mode.startsWith("chat") || mode === "wire-chat") { + expect(sent[0]).toMatchObject({ model: "m2", path: "/v1/chat/completions", tool_choice: { type: "function", function: { name: "tool_search" } } }); + } else if (mode === "virtual-chat") { + expect(sent[0]).toMatchObject({ model: wire, path: "/v1/responses", tool_choice: { type: "function", name: "tool_search" } }); + } else expect(sent[0]).toMatchObject({ model: virtual, path: "/v1/chat/completions", tool_choice: { type: "function", function: { name: "tool_search" } } }); + }); + for (const choice of ["required", { type: "function", name: "tool_search" }, { type: "tool_search" }, { type: "allowed_tools", mode: "required", tools: [{ type: "tool_search" }] }]) { for (const [backup, suffix] of [[false, ""], [true, ""], [false, "--fast"], [true, "--fast"], [false, "--high"], [true, "--high"]] as const) { From c486a73df717074573b040bcc5581fa56bb20510 Mon Sep 17 00:00:00 2001 From: Epinephrine Date: Sat, 31 Oct 2026 04:39:21 +0900 Subject: [PATCH 42/42] docs(combos): link localized mandatory-tool routing guidance --- docs-site/src/content/docs/ja/guides/combos.md | 2 ++ docs-site/src/content/docs/ko/guides/combos.md | 2 ++ docs-site/src/content/docs/ru/guides/combos.md | 2 ++ docs-site/src/content/docs/zh-cn/guides/combos.md | 2 ++ 4 files changed, 8 insertions(+) diff --git a/docs-site/src/content/docs/ja/guides/combos.md b/docs-site/src/content/docs/ja/guides/combos.md index 943ccef2bc0..9823756979f 100644 --- a/docs-site/src/content/docs/ja/guides/combos.md +++ b/docs-site/src/content/docs/ja/guides/combos.md @@ -64,6 +64,8 @@ ocx combo show main ## 戦略を選択してください +必須の `tool_choice` を維持できないターゲットは送信前にスキップされ、すべて非対応の場合はその制約を示すエラーを返します。自動選択のみを許可する Chat モデルと、OpenAI 仮想モデルの実際の送信モデルのポリシーも判定対象です。直接の hosted/MCP 選択、`tool_search`、無効な選択値の扱いは[英語の基準説明](/guides/combos/#failover-ordered-primary-and-backups)を参照してください。 + ### フェイルオーバー: 順序付けされたプライマリとバックアップ `failover` は、構成順序で最初の適格なターゲットを選択します。ターゲットは、プロバイダーが存在し、有効になっていて、クールダウン中でなく、特別なリクエスト制約を処理できる場合に適格となります。重みと `stickyLimit` はこの戦略に影響しません。 diff --git a/docs-site/src/content/docs/ko/guides/combos.md b/docs-site/src/content/docs/ko/guides/combos.md index 40ad3327e05..b81448df36b 100644 --- a/docs-site/src/content/docs/ko/guides/combos.md +++ b/docs-site/src/content/docs/ko/guides/combos.md @@ -70,6 +70,8 @@ alias는 클라이언트가 요청하는 공개 이름만 바꿉니다. 콤보 ## 전략 선택 +필수 `tool_choice`를 보존할 수 없는 대상은 전송 전에 건너뛰며, 모든 대상이 호환되지 않으면 해당 제약을 오류로 반환합니다. 자동 선택만 허용하는 Chat 모델과 OpenAI 가상 모델의 실제 전송 모델 정책도 이 판단에 반영됩니다. 직접 hosted/MCP 선택, `tool_search`, 잘못된 선택값의 처리 범위는 [영문 기준 설명](/guides/combos/#failover-ordered-primary-and-backups)을 참고하세요. + ### 페일오버: 순서가 있는 기본값과 예비값 `failover`는 설정 순서에서 가장 먼저 적합한 대상을 선택합니다. 대상이 적합하려면 해당 공급자가 존재하고, 활성화되어 있고, 쿨다운 중이 아니며, 요청에 붙은 특수 조건을 처리할 수 있어야 합니다. 이 전략에서는 가중치와 `stickyLimit`이 영향을 주지 않습니다. diff --git a/docs-site/src/content/docs/ru/guides/combos.md b/docs-site/src/content/docs/ru/guides/combos.md index ccb5444da6f..5991f1a868c 100644 --- a/docs-site/src/content/docs/ru/guides/combos.md +++ b/docs-site/src/content/docs/ru/guides/combos.md @@ -80,6 +80,8 @@ Alias меняет публичное имя, которое запрашива ## Выбор стратегии +Цель, которая не может сохранить обязательный `tool_choice`, пропускается до отправки; если совместимых целей нет, возвращается ошибка с этим ограничением. Проверка учитывает модели Chat с выбором только `auto` и политику фактически отправляемой модели для виртуальных моделей OpenAI. Правила для прямого выбора hosted/MCP, `tool_search` и некорректных значений приведены в [каноническом руководстве на английском](/guides/combos/#failover-ordered-primary-and-backups). + ### Failover: основной вариант и резерв в заданном порядке `failover` выбирает первую подходящую цель в порядке конфигурации. Цель считается подходящей, diff --git a/docs-site/src/content/docs/zh-cn/guides/combos.md b/docs-site/src/content/docs/zh-cn/guides/combos.md index 5244fe9b646..7262139cb71 100644 --- a/docs-site/src/content/docs/zh-cn/guides/combos.md +++ b/docs-site/src/content/docs/zh-cn/guides/combos.md @@ -90,6 +90,8 @@ modality 和 reasoning metadata;目标显式声明的限制仍然优先,因 ## 选择策略 +无法保留强制 `tool_choice` 的目标会在发送前跳过;如果所有目标都不兼容,则返回说明该限制的错误。判断会考虑仅支持 `auto` 选择的 Chat 模型,以及 OpenAI 虚拟模型实际发送到上游的模型策略。直接 hosted/MCP 选择、`tool_search` 和无效选择值的处理范围请参阅[英文基准说明](/guides/combos/#failover-ordered-primary-and-backups)。 + ### 故障切换:按顺序的主目标和备用目标 `failover` 会按配置顺序选择第一个合格目标。当 provider 存在、已启用、未处于冷却中,并且能够满足任何特殊请求约束时,该目标就是合格的。权重和 `stickyLimit` 不影响这种策略。