diff --git a/.gitignore b/.gitignore index e233f05..851c639 100644 --- a/.gitignore +++ b/.gitignore @@ -2,6 +2,7 @@ node_modules/ .env .env.* +llm-proxy-passthrough.json !.env.example !.env.*.example promotion.md diff --git a/README.md b/README.md index b43817f..825dfab 100644 --- a/README.md +++ b/README.md @@ -71,6 +71,7 @@ See also: [Security](docs/security.md) · [Comparisons](docs/comparisons.md) · - [Quickstart](#quickstart) - [Install](#install) - [Configuration](#configuration) + - [Direct upstream passthrough](#direct-upstream-passthrough) - [Tool calling](#tool-calling) - [Using with SDKs and tools](#using-with-sdks-and-tools) - [n8n](#n8n) @@ -190,7 +191,8 @@ curl -o .opencode/plugins/llm-proxy.js \ | `OPENCODE_LLM_PROXY_ALLOW_PRIVATE_NETWORK` | `false` | Set to `true` to allow browser Private Network Access preflights. | | `OPENCODE_LLM_PROXY_REQUEST_TIMEOUT_MS` | `120000` | Total request timeout, from 1 to 3,600,000 ms. | | `OPENCODE_LLM_PROXY_IDLE_TIMEOUT_SECONDS` | request timeout + 5 s | Seconds a connection may stay silent before the server closes it, from 0 (disabled) to 255. Defaults to the request timeout plus 5 s, or 0 when that exceeds 255 s. | -| `OPENCODE_LLM_PROXY_MAX_REQUEST_BYTES` | `1048576` | Maximum JSON request body and embedded data-URL size, up to 100 MiB. | +| `OPENCODE_LLM_PROXY_MAX_REQUEST_BYTES` | `33554432` | Maximum JSON request body and embedded data-URL size, up to 100 MiB. The 32 MiB default accommodates whole-session compaction prompts. | +| `OPENCODE_LLM_PROXY_PASSTHROUGH_CONFIG` | `~/.config/opencode/llm-proxy-passthrough.json` | Path to a JSON passthrough config (`baseUrl`, optional `prefix`, `unprefixed`, `apiKey`, `authHeader`). Missing or invalid config disables passthrough. Keep this file local. | | `OPENCODE_LLM_PROXY_MAX_CONCURRENT_REQUESTS` | `8` | Maximum active POST requests. | | `OPENCODE_LLM_PROXY_MAX_QUEUED_REQUESTS` | `32` | Maximum POST requests waiting for capacity; excess requests receive `503`. | | `OPENCODE_LLM_PROXY_TOOL_BRIDGE_POOL_SIZE` | `8` | Max concurrent in-flight requests using [tool calling](#tool-calling). | @@ -202,7 +204,7 @@ curl -o .opencode/plugins/llm-proxy.js \ | `OPENCODE_LLM_PROXY_METRICS_ENABLED` | `false` | Set to `true` to expose the authenticated Prometheus endpoint at `GET /metrics`. | | `OPENCODE_LLM_PROXY_REMOTE_MEDIA_ENABLED` | `false` | Set to `true` to fetch remote media URLs and convert them to embedded data URLs. Leave disabled unless required. | | `OPENCODE_LLM_PROXY_REMOTE_MEDIA_ALLOWED_SCHEMES` | `["https"]` | JSON array of allowed remote URL schemes (`https` and, if explicitly enabled, `http`). HTTPS-only is strongly recommended. | -| `OPENCODE_LLM_PROXY_REMOTE_MEDIA_MAX_BYTES` | value of `OPENCODE_LLM_PROXY_MAX_REQUEST_BYTES` (`1048576` by default) | Maximum downloaded bytes per remote media item, up to 100 MiB. | +| `OPENCODE_LLM_PROXY_REMOTE_MEDIA_MAX_BYTES` | `1048576` | Maximum downloaded bytes per remote media item, up to 100 MiB. | | `OPENCODE_LLM_PROXY_REMOTE_MEDIA_MAX_ITEMS` | `4` | Maximum remote media downloads in one request, from 0 to 10,000. | | `OPENCODE_LLM_PROXY_MAX_MEDIA_ITEMS` | `64` | Maximum total embedded and remote media items in one request. | | `OPENCODE_LLM_PROXY_REMOTE_MEDIA_MAX_REDIRECTS` | `3` | Maximum redirects per remote media download, from 0 to 100. | @@ -217,6 +219,35 @@ The proxy returns a single choice and cannot forward `stop`, `seed`, `frequency_ When an upstream call fails, the error names the reason the provider gave (for example `Upstream request failed: Free usage limit reached`) instead of a bare `Upstream request failed.` Credentials that a provider echoes back are redacted, and the text is truncated to 500 characters, so the detail is useful without exposing a key or token. +### Direct upstream passthrough + +For clients that already speak an upstream's native API, the proxy can forward requests **verbatim** instead of running them through an OpenCode session and tool bridge. This preserves the upstream's own streaming, tool-call, and error behavior. + +Create a local config file (`~/.config/opencode/llm-proxy-passthrough.json` by default, or point `OPENCODE_LLM_PROXY_PASSTHROUGH_CONFIG` at another path): + +```json +{ + "baseUrl": "https://upstream.example.com", + "prefix": "upstream/", + "unprefixed": false, + "apiKey": "optional-upstream-key", + "authHeader": "authorization" +} +``` + +- `POST /v1/chat/completions`, `POST /v1/responses`, and `POST /v1/messages` are eligible. +- A model that starts with `prefix` is forwarded to `baseUrl` with the prefix removed (`upstream/model-a` → `model-a`). Prefix matching always wins. +- Unprefixed discovery is **opt-in**: with `"unprefixed": true`, a model that the upstream lists in `GET /v1/models` is forwarded unchanged, even without the prefix. Because this can silently shadow a local OpenCode model with the same id, it is off by default. +- The proxy's own bearer auth and CORS checks still apply. The caller's `Authorization`, `x-api-key`, and `anthropic-api-key` headers are removed before forwarding. +- If `apiKey` is set, it replaces the caller's credentials. `authHeader` defaults to `authorization`, which is sent as `Bearer `; any other header name (for example `x-api-key`) receives the raw key. The same credential is used for the `GET /v1/models` probe. +- Hop-by-hop headers (`connection`, `keep-alive`, `proxy-authenticate`, `proxy-authorization`, `te`, `trailer`, `transfer-encoding`, `upgrade`), plus any header named in `Connection`, `host`, `content-length`, and the caller's auth headers are stripped before forwarding. +- The request body is read with the `OPENCODE_LLM_PROXY_MAX_REQUEST_BYTES` cap **before** forwarding (HTTP 413 when exceeded), including for requests that end up on the normal OpenCode-backed path. +- Passthrough requests bypass the proxy's concurrency queue and request timeout; the upstream's own limits apply. +- Responses carry `x-opencode-passthrough: true`. +- The file is read at plugin start. Keep it local and mode `600`; it may contain an internal endpoint or credential. The default path is outside the repository, and `llm-proxy-passthrough.json` is git-ignored. + +Requests for models that match neither rule continue through the normal OpenCode-backed path. + ### Session lifecycle (safe teardown) Each proxied request runs in a throwaway OpenCode session titled `Proxy: ` (**"Proxy: " is a reserved title prefix** — do not name your own sessions with it). When a request completes normally, the session is deleted immediately. When a request is abandoned mid-turn (client disconnect, timeout, or error), the session is **intentionally left in place**: deleting it immediately races the OpenCode server's final persist of the aborted turn, which surfaces as `FOREIGN KEY constraint failed` errors in the server log (observed at thousands per day under bulk clients that cancel slow streams, e.g. RAG pipelines with aggressive timeouts). diff --git a/dist/llm-proxy.js b/dist/llm-proxy.js index 1e4505b..ff0c484 100644 --- a/dist/llm-proxy.js +++ b/dist/llm-proxy.js @@ -2,6 +2,9 @@ import { fileURLToPath } from "node:url"; import { Buffer as Buffer2 } from "node:buffer"; import { timingSafeEqual } from "node:crypto"; +import { readFileSync } from "node:fs"; +import { homedir } from "node:os"; +import { join } from "node:path"; // canonical-messages.js function isObject(value) { @@ -1432,7 +1435,10 @@ function getState() { } var DEFAULTS = Object.freeze({ requestTimeoutMs: 12e4, - maxRequestBytes: 1024 * 1024, + // Coding agents routinely send whole-session compaction prompts above 1 MiB. + // Keep a bounded default, but do not reject an otherwise valid request at + // the old 1 MiB mark. Override with OPENCODE_LLM_PROXY_MAX_REQUEST_BYTES. + maxRequestBytes: 32 * 1024 * 1024, maxConcurrentRequests: 8, maxQueuedRequests: 32, bridgeAcquireTimeoutMs: 1e4, @@ -1516,7 +1522,9 @@ function loadConfig() { remoteMedia: { enabled: booleanEnv("OPENCODE_LLM_PROXY_REMOTE_MEDIA_ENABLED"), allowedSchemes: jsonArrayEnvDefault("OPENCODE_LLM_PROXY_REMOTE_MEDIA_ALLOWED_SCHEMES", ["https"]), - maxBytes: integerEnv("OPENCODE_LLM_PROXY_REMOTE_MEDIA_MAX_BYTES", maxRequestBytes || 1024 * 1024, { min: 1, max: 100 * 1024 * 1024 }), + // Keep the media download budget independent: raising the JSON body cap + // must not silently raise how much the proxy will fetch from remote URLs. + maxBytes: integerEnv("OPENCODE_LLM_PROXY_REMOTE_MEDIA_MAX_BYTES", 1024 * 1024, { min: 1, max: 100 * 1024 * 1024 }), maxItems: integerEnv("OPENCODE_LLM_PROXY_REMOTE_MEDIA_MAX_ITEMS", 4, { min: 0, max: 1e4 }), maxTotalItems: integerEnv("OPENCODE_LLM_PROXY_MAX_MEDIA_ITEMS", 64, { min: 1, max: 1e4 }), maxRedirects: integerEnv("OPENCODE_LLM_PROXY_REMOTE_MEDIA_MAX_REDIRECTS", 3, { min: 0, max: 100 }), @@ -1524,6 +1532,77 @@ function loadConfig() { } }; } +var HOP_BY_HOP_HEADERS = [ + "connection", + "keep-alive", + "proxy-authenticate", + "proxy-authorization", + "te", + "trailer", + "transfer-encoding", + "upgrade" +]; +var RESERVED_AUTH_HEADERS = /* @__PURE__ */ new Set([ + ...HOP_BY_HOP_HEADERS, + "host", + "content-length", + "content-type" +]); +function loadPassthroughConfig() { + const path = process.env.OPENCODE_LLM_PROXY_PASSTHROUGH_CONFIG ?? join(homedir(), ".config", "opencode", "llm-proxy-passthrough.json"); + try { + const parsed = JSON.parse(readFileSync(path, "utf8")); + if (typeof parsed?.baseUrl !== "string" || parsed.baseUrl.trim() === "") return null; + const url = new URL(parsed.baseUrl); + if (url.protocol !== "http:" && url.protocol !== "https:") return null; + const prefix = typeof parsed.prefix === "string" ? parsed.prefix : ""; + const unprefixed = parsed.unprefixed === true; + if (!prefix && !unprefixed) return null; + const apiKey = typeof parsed.apiKey === "string" && parsed.apiKey.trim() !== "" ? parsed.apiKey : void 0; + const authHeader = typeof parsed.authHeader === "string" && parsed.authHeader.trim() !== "" ? parsed.authHeader.trim().toLowerCase() : "authorization"; + if (RESERVED_AUTH_HEADERS.has(authHeader)) return null; + try { + new Headers({ [authHeader]: apiKey ?? "probe" }); + } catch { + return null; + } + return { + prefix, + baseUrl: parsed.baseUrl.replace(/\/+$/, ""), + unprefixed, + apiKey, + authHeader + }; + } catch { + return null; + } +} +function applyUpstreamAuth(headers, config) { + if (!config.apiKey) return headers; + headers.set( + config.authHeader, + config.authHeader === "authorization" ? `Bearer ${config.apiKey}` : config.apiKey + ); + return headers; +} +function stripHopByHop(headers, source) { + for (const token of (source.get("connection") ?? "").split(",")) { + const name = token.trim().toLowerCase(); + if (name) headers.delete(name); + } + for (const name of HOP_BY_HOP_HEADERS) headers.delete(name); + return headers; +} +function passthroughHeaders(request, config) { + const headers = stripHopByHop(new Headers(request.headers), request.headers); + headers.delete("host"); + headers.delete("content-length"); + headers.delete("authorization"); + headers.delete("x-api-key"); + headers.delete("anthropic-api-key"); + headers.set("content-type", "application/json"); + return applyUpstreamAuth(headers, config); +} function commonHeaders(request, config) { return { "cache-control": "no-store", @@ -1629,7 +1708,7 @@ function isAuthorized(request, config = loadConfig()) { function isPlainObject(value) { return Boolean(value && typeof value === "object" && !Array.isArray(value)); } -async function readJsonBody(request, maxBytes, signal) { +async function readTextBody(request, maxBytes, signal) { const declared = Number(request.headers.get("content-length")); if (Number.isFinite(declared) && declared > maxBytes) { throw new ProxyError("Request body is too large.", 413, "request_too_large"); @@ -1667,9 +1746,7 @@ async function readJsonBody(request, maxBytes, signal) { } chunks.push(value); } - const body = JSON.parse(Buffer2.concat(chunks.map((chunk) => Buffer2.from(chunk))).toString("utf8")); - if (!isPlainObject(body)) throw new ProxyError("Request body must be a JSON object.", 400, "invalid_json"); - return body; + return Buffer2.concat(chunks.map((chunk) => Buffer2.from(chunk))).toString("utf8"); } catch (error) { if (error instanceof ProxyError) throw error; throw new ProxyError("Request body must be valid JSON.", 400, "invalid_json"); @@ -1678,6 +1755,17 @@ async function readJsonBody(request, maxBytes, signal) { reader.releaseLock(); } } +async function readJsonBody(request, maxBytes, signal) { + const text2 = await readTextBody(request, maxBytes, signal); + try { + const body = JSON.parse(text2); + if (!isPlainObject(body)) throw new ProxyError("Request body must be a JSON object.", 400, "invalid_json"); + return body; + } catch (error) { + if (error instanceof ProxyError) throw error; + throw new ProxyError("Request body must be valid JSON.", 400, "invalid_json"); + } +} function createRequestSignal(request, timeoutMs) { const controller = new AbortController(); const onAbort = () => controller.abort(new ProxyError("Request was cancelled.", 499, "cancelled")); @@ -3105,8 +3193,60 @@ function geminiModelFromPath(pathname) { const match = pathname.match(/^\/v1beta\/models\/(.+):(?:generateContent|streamGenerateContent)$/); return match ? decodeURIComponent(match[1]) : null; } +var PASSTHROUGH_PATHS = /* @__PURE__ */ new Set(["/v1/chat/completions", "/v1/responses", "/v1/messages"]); +var UPSTREAM_MODELS_TTL_MS = 6e4; function createProxyFetchHandler(client) { const config = loadConfig(); + const passthrough = loadPassthroughConfig(); + let upstreamModelsCache = null; + async function isUpstreamModel(model) { + if (!passthrough.unprefixed) return false; + const now = Date.now(); + if (!upstreamModelsCache || now - upstreamModelsCache.at > UPSTREAM_MODELS_TTL_MS) { + let ids = /* @__PURE__ */ new Set(); + try { + const response = await fetch(`${passthrough.baseUrl}/v1/models`, { + headers: applyUpstreamAuth(new Headers(), passthrough), + signal: AbortSignal.timeout(3e3) + }); + if (response.ok) { + const data = await response.json(); + ids = new Set((data?.data ?? []).map((entry) => entry?.id).filter((id) => typeof id === "string")); + } + } catch { + } + upstreamModelsCache = { at: now, ids }; + } + return upstreamModelsCache.ids.has(model); + } + async function proxyPassthrough(request, url, body, upstreamModel) { + const upstreamBody = JSON.stringify({ ...body, model: upstreamModel }); + const headers = passthroughHeaders(request, passthrough); + let upstream; + try { + upstream = await fetch(`${passthrough.baseUrl}${url.pathname}${url.search}`, { + method: "POST", + headers, + body: upstreamBody, + signal: request.signal + }); + } catch (error) { + const message = error instanceof Error ? error.message : String(error); + await safeLog(client, "error", "Passthrough request failed", { + error: message, + path: url.pathname + }); + return badRequest(`Passthrough request failed: ${message}`, 502, request); + } + const responseHeaders = stripHopByHop(new Headers(upstream.headers), upstream.headers); + responseHeaders.delete("content-encoding"); + responseHeaders.delete("content-length"); + responseHeaders.set("x-opencode-passthrough", "true"); + for (const [key, value] of Object.entries(commonHeaders(request, config))) { + responseHeaders.set(key, value); + } + return new Response(upstream.body, { status: upstream.status, headers: responseHeaders }); + } const handleRequest = async (request) => { const url = new URL(request.url); const origin = request.headers.get("origin"); @@ -3126,6 +3266,44 @@ function createProxyFetchHandler(client) { if (!isAuthorized(request, config)) { return unauthorized(request); } + if (passthrough && request.method === "POST" && PASSTHROUGH_PATHS.has(url.pathname)) { + let rawBody; + try { + rawBody = await readTextBody(request, config.maxRequestBytes, request.signal); + } catch (error) { + if (error instanceof ProxyError) { + return badRequest(error.message, error.status, request, error.code); + } + return badRequest("Failed to read request body.", 400, request); + } + let parsedBody; + try { + parsedBody = JSON.parse(rawBody); + } catch { + parsedBody = void 0; + } + if (typeof parsedBody?.model === "string") { + if (passthrough.prefix && parsedBody.model.startsWith(passthrough.prefix)) { + return proxyPassthrough( + request, + url, + parsedBody, + parsedBody.model.slice(passthrough.prefix.length) + ); + } + if (passthrough.unprefixed && await isUpstreamModel(parsedBody.model)) { + return proxyPassthrough(request, url, parsedBody, parsedBody.model); + } + } + const replayHeaders = new Headers(request.headers); + replayHeaders.delete("content-length"); + request = new Request(request.url, { + method: request.method, + headers: replayHeaders, + body: rawBody, + signal: request.signal + }); + } const started = Date.now(); const context = createRequestSignal(request, config.requestTimeoutMs); let releaseSlot = () => { diff --git a/docs/security.md b/docs/security.md index cc0a1d1..5e8bc1a 100644 --- a/docs/security.md +++ b/docs/security.md @@ -68,6 +68,21 @@ When enabled, the fetcher defaults to HTTPS only and converts successful downloa Do not add `http` to `OPENCODE_LLM_PROXY_REMOTE_MEDIA_ALLOWED_SCHEMES` unless transport security is provided by a trusted environment and the risk is understood. These controls reduce SSRF risk but do not make arbitrary untrusted URL fetching preferable to leaving the feature disabled. +## Direct upstream passthrough + +Passthrough is a second, separate egress path. When enabled, requests for matching models are forwarded verbatim to the `baseUrl` in `OPENCODE_LLM_PROXY_PASSTHROUGH_CONFIG` (default `~/.config/opencode/llm-proxy-passthrough.json`). The proxy does not validate that endpoint for SSRF or allowlist reasons: it is an operator-supplied upstream, not user-supplied remote media. + +Treat the config file as a secret and keep it local, mode `600`, and outside version control: + +- It can contain an internal endpoint and an `apiKey`. +- The default path is outside the repository, and `llm-proxy-passthrough.json` is git-ignored. +- The proxy applies the configured credential to both the upstream request and the `GET /v1/models` probe. Caller credentials are never forwarded. +- Hop-by-hop headers, `host`, `content-length`, and caller auth headers are stripped before forwarding. +- The request body is read with the `OPENCODE_LLM_PROXY_MAX_REQUEST_BYTES` cap before forwarding, so a client cannot make the proxy buffer an unbounded body. +- Unprefixed model discovery is opt-in (`"unprefixed": true`) because it can shadow local OpenCode models with the same ids. + +Enabling passthrough grants every authenticated proxy client the same upstream access that the configured credential has. Use a dedicated, least-privilege key where the upstream supports one, and keep the proxy bound to localhost or behind a trusted network and firewall. + ## Checklist - [ ] Localhost binding unless network access is genuinely required @@ -77,4 +92,5 @@ Do not add `http` to `OPENCODE_LLM_PROXY_REMOTE_MEDIA_ALLOWED_SCHEMES` unless tr - [ ] Firewall restricts inbound access to known hosts - [ ] Tool-using clients are trusted and reviewed - [ ] Remote media remains disabled, or its HTTPS-only limits are reviewed +- [ ] Passthrough config stays local (mode `600`) and uses a least-privilege upstream key - [ ] Tokens never written to logs diff --git a/eslint.config.js b/eslint.config.js index c785fac..998b403 100644 --- a/eslint.config.js +++ b/eslint.config.js @@ -17,6 +17,7 @@ export default [ // Web API globals available in both Node and Bun Request: "readonly", Response: "readonly", + Headers: "readonly", URL: "readonly", TextEncoder: "readonly", ReadableStream: "readonly", diff --git a/index.js b/index.js index 8bc3978..43c2fbe 100644 --- a/index.js +++ b/index.js @@ -1,6 +1,9 @@ import { fileURLToPath } from "node:url" import { Buffer } from "node:buffer" import { timingSafeEqual } from "node:crypto" +import { readFileSync } from "node:fs" +import { homedir } from "node:os" +import { join } from "node:path" import { adaptAnthropic, adaptGemini, @@ -24,7 +27,10 @@ function getState() { const DEFAULTS = Object.freeze({ requestTimeoutMs: 120000, - maxRequestBytes: 1024 * 1024, + // Coding agents routinely send whole-session compaction prompts above 1 MiB. + // Keep a bounded default, but do not reject an otherwise valid request at + // the old 1 MiB mark. Override with OPENCODE_LLM_PROXY_MAX_REQUEST_BYTES. + maxRequestBytes: 32 * 1024 * 1024, maxConcurrentRequests: 8, maxQueuedRequests: 32, bridgeAcquireTimeoutMs: 10000, @@ -126,7 +132,9 @@ function loadConfig() { remoteMedia: { enabled: booleanEnv("OPENCODE_LLM_PROXY_REMOTE_MEDIA_ENABLED"), allowedSchemes: jsonArrayEnvDefault("OPENCODE_LLM_PROXY_REMOTE_MEDIA_ALLOWED_SCHEMES", ["https"]), - maxBytes: integerEnv("OPENCODE_LLM_PROXY_REMOTE_MEDIA_MAX_BYTES", maxRequestBytes || 1024 * 1024, { min: 1, max: 100 * 1024 * 1024 }), + // Keep the media download budget independent: raising the JSON body cap + // must not silently raise how much the proxy will fetch from remote URLs. + maxBytes: integerEnv("OPENCODE_LLM_PROXY_REMOTE_MEDIA_MAX_BYTES", 1024 * 1024, { min: 1, max: 100 * 1024 * 1024 }), maxItems: integerEnv("OPENCODE_LLM_PROXY_REMOTE_MEDIA_MAX_ITEMS", 4, { min: 0, max: 10000 }), maxTotalItems: integerEnv("OPENCODE_LLM_PROXY_MAX_MEDIA_ITEMS", 64, { min: 1, max: 10000 }), maxRedirects: integerEnv("OPENCODE_LLM_PROXY_REMOTE_MEDIA_MAX_REDIRECTS", 3, { min: 0, max: 100 }), @@ -135,6 +143,101 @@ function loadConfig() { } } +const HOP_BY_HOP_HEADERS = [ + "connection", + "keep-alive", + "proxy-authenticate", + "proxy-authorization", + "te", + "trailer", + "transfer-encoding", + "upgrade", +] + +// An authHeader names the header that carries the upstream credential. Setting +// a framing or hop-by-hop header there would corrupt the request, so reject +// those at load time instead of failing every forwarded call. +const RESERVED_AUTH_HEADERS = new Set([ + ...HOP_BY_HOP_HEADERS, + "host", + "content-length", + "content-type", +]) + +// Optional direct-upstream passthrough. The config file is local and is never +// committed: it names an upstream base URL and an optional model prefix. +// Requests whose model matches are forwarded verbatim (minus the prefix), with +// no OpenCode session and no tool bridge, so streaming, tools, and large bodies +// behave exactly like the upstream API. +function loadPassthroughConfig() { + const path = process.env.OPENCODE_LLM_PROXY_PASSTHROUGH_CONFIG + ?? join(homedir(), ".config", "opencode", "llm-proxy-passthrough.json") + try { + const parsed = JSON.parse(readFileSync(path, "utf8")) + if (typeof parsed?.baseUrl !== "string" || parsed.baseUrl.trim() === "") return null + const url = new URL(parsed.baseUrl) + if (url.protocol !== "http:" && url.protocol !== "https:") return null + const prefix = typeof parsed.prefix === "string" ? parsed.prefix : "" + const unprefixed = parsed.unprefixed === true + // An empty prefix with unprefixed discovery disabled would make every + // request read and replay its body for no reason. + if (!prefix && !unprefixed) return null + const apiKey = typeof parsed.apiKey === "string" && parsed.apiKey.trim() !== "" + ? parsed.apiKey + : undefined + const authHeader = typeof parsed.authHeader === "string" && parsed.authHeader.trim() !== "" + ? parsed.authHeader.trim().toLowerCase() + : "authorization" + if (RESERVED_AUTH_HEADERS.has(authHeader)) return null + // Reject an invalid header name or value at load time instead of failing a + // request: Headers rejects both, so a key with a newline cannot slip in. + try { + new Headers({ [authHeader]: apiKey ?? "probe" }) + } catch { + return null + } + return { + prefix, + baseUrl: parsed.baseUrl.replace(/\/+$/, ""), + unprefixed, + apiKey, + authHeader, + } + } catch { + return null + } +} + +function applyUpstreamAuth(headers, config) { + if (!config.apiKey) return headers + headers.set( + config.authHeader, + config.authHeader === "authorization" ? `Bearer ${config.apiKey}` : config.apiKey, + ) + return headers +} + +function stripHopByHop(headers, source) { + // RFC 9110: a Connection header can list additional hop-by-hop fields. + for (const token of (source.get("connection") ?? "").split(",")) { + const name = token.trim().toLowerCase() + if (name) headers.delete(name) + } + for (const name of HOP_BY_HOP_HEADERS) headers.delete(name) + return headers +} + +function passthroughHeaders(request, config) { + const headers = stripHopByHop(new Headers(request.headers), request.headers) + headers.delete("host") + headers.delete("content-length") + headers.delete("authorization") + headers.delete("x-api-key") + headers.delete("anthropic-api-key") + headers.set("content-type", "application/json") + return applyUpstreamAuth(headers, config) +} + function commonHeaders(request, config) { return { "cache-control": "no-store", @@ -254,7 +357,7 @@ function isPlainObject(value) { return Boolean(value && typeof value === "object" && !Array.isArray(value)) } -async function readJsonBody(request, maxBytes, signal) { +async function readTextBody(request, maxBytes, signal) { const declared = Number(request.headers.get("content-length")) if (Number.isFinite(declared) && declared > maxBytes) { throw new ProxyError("Request body is too large.", 413, "request_too_large") @@ -291,9 +394,7 @@ async function readJsonBody(request, maxBytes, signal) { } chunks.push(value) } - const body = JSON.parse(Buffer.concat(chunks.map((chunk) => Buffer.from(chunk))).toString("utf8")) - if (!isPlainObject(body)) throw new ProxyError("Request body must be a JSON object.", 400, "invalid_json") - return body + return Buffer.concat(chunks.map((chunk) => Buffer.from(chunk))).toString("utf8") } catch (error) { if (error instanceof ProxyError) throw error throw new ProxyError("Request body must be valid JSON.", 400, "invalid_json") @@ -303,6 +404,18 @@ async function readJsonBody(request, maxBytes, signal) { } } +async function readJsonBody(request, maxBytes, signal) { + const text = await readTextBody(request, maxBytes, signal) + try { + const body = JSON.parse(text) + if (!isPlainObject(body)) throw new ProxyError("Request body must be a JSON object.", 400, "invalid_json") + return body + } catch (error) { + if (error instanceof ProxyError) throw error + throw new ProxyError("Request body must be valid JSON.", 400, "invalid_json") + } +} + function createRequestSignal(request, timeoutMs) { const controller = new AbortController() const onAbort = () => controller.abort(new ProxyError("Request was cancelled.", 499, "cancelled")) @@ -2138,8 +2251,75 @@ function geminiModelFromPath(pathname) { return match ? decodeURIComponent(match[1]) : null } +const PASSTHROUGH_PATHS = new Set(["/v1/chat/completions", "/v1/responses", "/v1/messages"]) +const UPSTREAM_MODELS_TTL_MS = 60_000 + export function createProxyFetchHandler(client) { const config = loadConfig() + const passthrough = loadPassthroughConfig() + let upstreamModelsCache = null + + // Accept both prefixed model ids and the upstream's own ids, so a client + // that omits the prefix still gets a transparent passthrough instead of the + // OpenCode session path. + async function isUpstreamModel(model) { + if (!passthrough.unprefixed) return false + const now = Date.now() + if (!upstreamModelsCache || now - upstreamModelsCache.at > UPSTREAM_MODELS_TTL_MS) { + let ids = new Set() + try { + const response = await fetch(`${passthrough.baseUrl}/v1/models`, { + headers: applyUpstreamAuth(new Headers(), passthrough), + signal: AbortSignal.timeout(3000), + }) + if (response.ok) { + const data = await response.json() + ids = new Set((data?.data ?? []) + .map((entry) => entry?.id) + .filter((id) => typeof id === "string")) + } + } catch { + // Model discovery is best-effort; prefixed requests still work. + } + upstreamModelsCache = { at: now, ids } + } + return upstreamModelsCache.ids.has(model) + } + + async function proxyPassthrough(request, url, body, upstreamModel) { + const upstreamBody = JSON.stringify({ ...body, model: upstreamModel }) + const headers = passthroughHeaders(request, passthrough) + + let upstream + try { + upstream = await fetch(`${passthrough.baseUrl}${url.pathname}${url.search}`, { + method: "POST", + headers, + body: upstreamBody, + signal: request.signal, + }) + } catch (error) { + const message = error instanceof Error ? error.message : String(error) + await safeLog(client, "error", "Passthrough request failed", { + error: message, + path: url.pathname, + }) + return badRequest(`Passthrough request failed: ${message}`, 502, request) + } + + // fetch has already decoded the body, so content-encoding no longer + // applies. Strip hop-by-hop fields in both directions as well: Bun's server + // handles connection framing itself. + const responseHeaders = stripHopByHop(new Headers(upstream.headers), upstream.headers) + responseHeaders.delete("content-encoding") + responseHeaders.delete("content-length") + responseHeaders.set("x-opencode-passthrough", "true") + for (const [key, value] of Object.entries(commonHeaders(request, config))) { + responseHeaders.set(key, value) + } + return new Response(upstream.body, { status: upstream.status, headers: responseHeaders }) + } + const handleRequest = async (request) => { const url = new URL(request.url) const origin = request.headers.get("origin") @@ -2164,6 +2344,51 @@ export function createProxyFetchHandler(client) { return unauthorized(request) } + if (passthrough && request.method === "POST" && PASSTHROUGH_PATHS.has(url.pathname)) { + let rawBody + try { + rawBody = await readTextBody(request, config.maxRequestBytes, request.signal) + } catch (error) { + if (error instanceof ProxyError) { + return badRequest(error.message, error.status, request, error.code) + } + return badRequest("Failed to read request body.", 400, request) + } + let parsedBody + try { + parsedBody = JSON.parse(rawBody) + } catch { + // Leave malformed JSON to the endpoint-specific handler so each API + // keeps its native error shape (for example Anthropic's `type: error`). + parsedBody = undefined + } + + if (typeof parsedBody?.model === "string") { + if (passthrough.prefix && parsedBody.model.startsWith(passthrough.prefix)) { + return proxyPassthrough( + request, + url, + parsedBody, + parsedBody.model.slice(passthrough.prefix.length), + ) + } + if (passthrough.unprefixed && await isUpstreamModel(parsedBody.model)) { + return proxyPassthrough(request, url, parsedBody, parsedBody.model) + } + } + + // Not a passthrough model: replay the consumed body for the regular + // OpenCode-backed handler. + const replayHeaders = new Headers(request.headers) + replayHeaders.delete("content-length") + request = new Request(request.url, { + method: request.method, + headers: replayHeaders, + body: rawBody, + signal: request.signal, + }) + } + const started = Date.now() const context = createRequestSignal(request, config.requestTimeoutMs) let releaseSlot = () => {} diff --git a/index.test.js b/index.test.js index c7cb9c6..70b4972 100644 --- a/index.test.js +++ b/index.test.js @@ -2,6 +2,9 @@ import test, { describe, it, after, beforeEach } from "node:test" import assert from "node:assert/strict" import { setTimeout as delay } from "node:timers/promises" import { PassThrough } from "node:stream" +import { mkdtempSync, rmSync, writeFileSync } from "node:fs" +import { tmpdir } from "node:os" +import { join } from "node:path" import * as mod from "./index.js" import { @@ -355,6 +358,354 @@ test("any configured bearer token is accepted", async () => { } }) +// --------------------------------------------------------------------------- +// Integration: direct upstream passthrough +// --------------------------------------------------------------------------- + +async function withPassthroughConfig(config, fn) { + const dir = mkdtempSync(join(tmpdir(), "llm-proxy-passthrough-")) + const file = join(dir, "passthrough.json") + writeFileSync(file, JSON.stringify(config)) + process.env.OPENCODE_LLM_PROXY_PASSTHROUGH_CONFIG = file + try { + return await fn() + } finally { + delete process.env.OPENCODE_LLM_PROXY_PASSTHROUGH_CONFIG + rmSync(dir, { recursive: true, force: true }) + } +} + +async function withMockUpstream(fn, { responseHeaders = {} } = {}) { + const calls = [] + const originalFetch = globalThis.fetch + globalThis.fetch = async (url, init = {}) => { + const body = typeof init.body === "string" ? JSON.parse(init.body) : null + calls.push({ + url: String(url), + body, + headers: new Headers(init.headers ?? {}), + }) + if (String(url).endsWith("/v1/models")) { + return Response.json({ data: [{ id: "model-a" }, { id: "model-b" }] }) + } + return new Response("upstream-body", { + status: 200, + headers: { "content-type": "text/plain", "x-upstream": "yes", ...responseHeaders }, + }) + } + try { + return await fn(calls) + } finally { + globalThis.fetch = originalFetch + } +} + +test("passthrough forwards a prefixed model verbatim and strips proxy auth", async () => { + await withPassthroughConfig({ baseUrl: "http://upstream.test/", prefix: "up/" }, async () => { + await withMockUpstream(async (calls) => { + const handler = createProxyFetchHandler(createClient()) + const response = await handler(new Request("http://127.0.0.1:4010/v1/responses", { + method: "POST", + headers: { + "content-type": "application/json", + authorization: "Bearer proxy-token", + "x-request-id": "req-1", + }, + body: JSON.stringify({ model: "up/model-a", input: "hi", stream: true }), + })) + + assert.equal(response.status, 200) + assert.equal(response.headers.get("x-opencode-passthrough"), "true") + assert.equal(response.headers.get("x-upstream"), "yes") + assert.equal(await response.text(), "upstream-body") + + const call = calls.find((entry) => entry.url.endsWith("/v1/responses")) + assert.equal(call.url, "http://upstream.test/v1/responses") + assert.equal(call.body.model, "model-a") + assert.equal(call.headers.get("authorization"), null) + assert.equal(call.headers.get("x-request-id"), "req-1") + }) + }) +}) + +test("passthrough accepts an upstream model id without the prefix when opted in", async () => { + await withPassthroughConfig({ baseUrl: "http://upstream.test", prefix: "up/", unprefixed: true }, async () => { + await withMockUpstream(async (calls) => { + const handler = createProxyFetchHandler(createClient()) + const response = await handler(new Request("http://127.0.0.1:4010/v1/chat/completions", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ + model: "model-b", + messages: [{ role: "user", content: "hi" }], + }), + })) + + assert.equal(response.status, 200) + assert.equal(response.headers.get("x-opencode-passthrough"), "true") + assert.equal(calls.filter((entry) => entry.url.endsWith("/v1/models")).length, 1) + const call = calls.find((entry) => entry.url.endsWith("/v1/chat/completions")) + assert.equal(call.body.model, "model-b") + }) + }) +}) + +test("passthrough does not take over an unprefixed model unless opted in", async () => { + await withPassthroughConfig({ baseUrl: "http://upstream.test", prefix: "up/" }, async () => { + await withMockUpstream(async (calls) => { + const handler = createProxyFetchHandler(createModelsClient([])) + const response = await handler(new Request("http://127.0.0.1:4010/v1/chat/completions", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ + model: "model-b", + messages: [{ role: "user", content: "hi" }], + }), + })) + + assert.equal(response.status, 400) + assert.equal(response.headers.get("x-opencode-passthrough"), null) + assert.equal(calls.filter((entry) => entry.url.endsWith("/v1/models")).length, 0) + }) + }) +}) + +test("passthrough applies a configured upstream bearer credential", async () => { + await withPassthroughConfig({ baseUrl: "http://upstream.test", prefix: "up/", apiKey: "upstream-secret" }, async () => { + await withMockUpstream(async (calls) => { + const handler = createProxyFetchHandler(createClient()) + const response = await handler(new Request("http://127.0.0.1:4010/v1/responses", { + method: "POST", + headers: { "content-type": "application/json", authorization: "Bearer proxy-token" }, + body: JSON.stringify({ model: "up/model-a", input: "hi" }), + })) + + assert.equal(response.status, 200) + const call = calls.find((entry) => entry.url.endsWith("/v1/responses")) + assert.equal(call.headers.get("authorization"), "Bearer upstream-secret") + }) + }) +}) + +test("passthrough applies a configured upstream custom auth header", async () => { + await withPassthroughConfig({ baseUrl: "http://upstream.test", prefix: "up/", apiKey: "upstream-secret", authHeader: "x-api-key" }, async () => { + await withMockUpstream(async (calls) => { + const handler = createProxyFetchHandler(createClient()) + const response = await handler(new Request("http://127.0.0.1:4010/v1/responses", { + method: "POST", + headers: { "content-type": "application/json", authorization: "Bearer proxy-token" }, + body: JSON.stringify({ model: "up/model-a", input: "hi" }), + })) + + assert.equal(response.status, 200) + const call = calls.find((entry) => entry.url.endsWith("/v1/responses")) + assert.equal(call.headers.get("x-api-key"), "upstream-secret") + assert.equal(call.headers.get("authorization"), null) + }) + }) +}) + +test("passthrough authenticates the model probe and never sends caller credentials", async () => { + await withPassthroughConfig({ + baseUrl: "http://upstream.test", + prefix: "up/", + unprefixed: true, + apiKey: "upstream-secret", + authHeader: "x-api-key", + }, async () => { + await withMockUpstream(async (calls) => { + const handler = createProxyFetchHandler(createClient()) + const response = await handler(new Request("http://127.0.0.1:4010/v1/chat/completions", { + method: "POST", + headers: { "content-type": "application/json", authorization: "Bearer proxy-token" }, + body: JSON.stringify({ model: "model-b", messages: [{ role: "user", content: "hi" }] }), + })) + + assert.equal(response.status, 200) + const probe = calls.find((entry) => entry.url.endsWith("/v1/models")) + assert.ok(probe, "the unprefixed model probe must run") + assert.equal(probe.headers.get("x-api-key"), "upstream-secret") + assert.equal(probe.headers.get("authorization"), null) + }) + }) +}) + +test("a reserved passthrough auth header disables passthrough", async () => { + for (const authHeader of ["host", "content-length", "content-type", "connection", "bad header"]) { + await withPassthroughConfig({ baseUrl: "http://upstream.test", prefix: "up/", apiKey: "secret", authHeader }, async () => { + await withMockUpstream(async (calls) => { + const handler = createProxyFetchHandler(createModelsClient([])) + const response = await handler(new Request("http://127.0.0.1:4010/v1/responses", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "up/model-a", input: "hi" }), + })) + + assert.notEqual(response.headers.get("x-opencode-passthrough"), "true", `${authHeader} must disable passthrough`) + assert.equal(calls.length, 0) + }) + }) + } +}) + +test("passthrough strips upstream hop-by-hop response headers", async () => { + await withPassthroughConfig({ baseUrl: "http://upstream.test", prefix: "up/" }, async () => { + await withMockUpstream(async () => { + const handler = createProxyFetchHandler(createClient()) + const response = await handler(new Request("http://127.0.0.1:4010/v1/responses", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "up/model-a", input: "hi" }), + })) + + assert.equal(response.status, 200) + for (const name of ["connection", "keep-alive", "transfer-encoding", "te", "upgrade", "trailer", "x-remove-me"]) { + assert.equal(response.headers.get(name), null, `${name} must not reach the client`) + } + assert.equal(response.headers.get("x-upstream"), "yes") + }, { + responseHeaders: { + connection: "keep-alive, x-remove-me", + "x-remove-me": "yes", + "keep-alive": "timeout=5", + "transfer-encoding": "chunked", + te: "trailers", + upgrade: "h2c", + trailer: "x-checksum", + }, + }) + }) +}) + +test("passthrough strips caller hop-by-hop headers", async () => { + await withPassthroughConfig({ baseUrl: "http://upstream.test", prefix: "up/" }, async () => { + await withMockUpstream(async (calls) => { + const handler = createProxyFetchHandler(createClient()) + const response = await handler(new Request("http://127.0.0.1:4010/v1/responses", { + method: "POST", + headers: { + "content-type": "application/json", + connection: "x-remove-me", + "x-remove-me": "yes", + "transfer-encoding": "chunked", + te: "trailers", + upgrade: "h2c", + "keep-alive": "timeout=5", + "x-keep": "yes", + }, + body: JSON.stringify({ model: "up/model-a", input: "hi" }), + })) + + assert.equal(response.status, 200) + const call = calls.find((entry) => entry.url.endsWith("/v1/responses")) + for (const name of ["connection", "x-remove-me", "transfer-encoding", "te", "upgrade", "keep-alive"]) { + assert.equal(call.headers.get(name), null, `${name} must not be forwarded`) + } + assert.equal(call.headers.get("x-keep"), "yes") + }) + }) +}) + +test("passthrough enforces the request-body cap before buffering", async () => { + process.env.OPENCODE_LLM_PROXY_MAX_REQUEST_BYTES = "1024" + try { + await withPassthroughConfig({ baseUrl: "http://upstream.test", prefix: "up/" }, async () => { + await withMockUpstream(async (calls) => { + const handler = createProxyFetchHandler(createClient()) + for (const model of ["up/model-a", "other/model-x"]) { + const response = await handler(new Request("http://127.0.0.1:4010/v1/chat/completions", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ + model, + messages: [{ role: "user", content: "x".repeat(4096) }], + }), + })) + + assert.equal(response.status, 413, `${model} must be rejected before forwarding`) + } + assert.equal(calls.length, 0) + }) + }) + } finally { + delete process.env.OPENCODE_LLM_PROXY_MAX_REQUEST_BYTES + } +}) + +test("passthrough forwards bodies above the old 1 MiB default", async () => { + await withPassthroughConfig({ baseUrl: "http://upstream.test", prefix: "up/" }, async () => { + await withMockUpstream(async (calls) => { + const handler = createProxyFetchHandler(createClient()) + const input = "x".repeat(2 * 1024 * 1024) + const response = await handler(new Request("http://127.0.0.1:4010/v1/responses", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "up/model-a", input }), + })) + + assert.equal(response.status, 200) + const call = calls.find((entry) => entry.url.endsWith("/v1/responses")) + assert.equal(call.body.input.length, input.length) + }) + }) +}) + +test("a non-upstream model still uses the normal OpenCode path", async () => { + await withPassthroughConfig({ baseUrl: "http://upstream.test", prefix: "up/", unprefixed: true }, async () => { + await withMockUpstream(async (calls) => { + const handler = createProxyFetchHandler(createModelsClient([])) + const response = await handler(new Request("http://127.0.0.1:4010/v1/responses", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "other/model-x", input: "hi" }), + })) + + assert.equal(response.status, 400) + assert.equal(response.headers.get("x-opencode-passthrough"), null) + assert.equal(calls.some((entry) => entry.url.endsWith("/v1/responses")), false) + assert.equal(calls.filter((entry) => entry.url.endsWith("/v1/models")).length, 1) + }) + }) +}) + +test("passthrough still enforces the proxy bearer token", async () => { + process.env.OPENCODE_LLM_PROXY_TOKEN = "secret-token" + try { + await withPassthroughConfig({ baseUrl: "http://upstream.test", prefix: "up/" }, async () => { + await withMockUpstream(async (calls) => { + const handler = createProxyFetchHandler(createClient()) + const response = await handler(new Request("http://127.0.0.1:4010/v1/responses", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "up/model-a", input: "hi" }), + })) + + assert.equal(response.status, 401) + assert.equal(calls.length, 0) + }) + }) + } finally { + delete process.env.OPENCODE_LLM_PROXY_TOKEN + } +}) + +test("an invalid passthrough base URL disables passthrough", async () => { + await withPassthroughConfig({ baseUrl: "file:///etc/passwd", prefix: "up/" }, async () => { + await withMockUpstream(async (calls) => { + const handler = createProxyFetchHandler(createModelsClient([])) + const response = await handler(new Request("http://127.0.0.1:4010/v1/responses", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "up/model-a", input: "hi" }), + })) + + assert.equal(response.status, 400) + assert.equal(response.headers.get("x-opencode-passthrough"), null) + assert.equal(calls.length, 0) + }) + }) +}) + // --------------------------------------------------------------------------- // Integration: /v1/chat/completions error handling // --------------------------------------------------------------------------- @@ -590,6 +941,24 @@ test("HTTP metrics record exact auth, CORS, route, and stream statuses", async ( assert.match(output, /method="POST",route="\/v1\/chat\/completions",status="200"\} 1/) }) +test("passthrough responses are recorded in HTTP metrics", async () => { + resetMetrics() + await withPassthroughConfig({ baseUrl: "http://upstream.test", prefix: "up/" }, async () => { + await withMockUpstream(async () => { + const handler = createProxyFetchHandler(createClient()) + const response = await handler(new Request("http://127.0.0.1:4010/v1/responses", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "up/model-a", input: "hi" }), + })) + + assert.equal(response.status, 200) + const output = getMetrics().metrics() + assert.match(output, /method="POST",route="\/v1\/responses",status="200"\} 1/) + }) + }) +}) + test("metrics endpoint is available only when enabled", async () => { let response = await createProxyFetchHandler(createClient())(new Request("http://127.0.0.1:4010/metrics")) assert.equal(response.status, 404)