Skip to content

fix(model,doctor,netutil): guard occupied GPUs and harden HTTP MCP dialing - #656

Open
toasterbook88 wants to merge 7 commits into
mainfrom
fix/doctor-dial-gpu-guard-20261117
Open

toasterbook88 wants to merge 7 commits into
mainfrom
fix/doctor-dial-gpu-guard-20261117

Conversation

@toasterbook88

@toasterbook88 toasterbook88 commented Nov 17, 2026 •

Copy link
Copy Markdown
Owner

Explicit model starts could overlap an occupied GPU because --main-gpu was treated as device isolation. Doctor's HTTP MCP probes could also resolve a hostname once for validation and again for dialing, bypassing the checked address policy and the endpoint's DNS deadline.

The four bugfix commits resolve those findings and preserve address fallback:

  • Fixed in 43daf820: reject any observed resident GPU occupancy before a llama-server launch. A different --main-gpu cannot bypass the refusal. Missing GPU-index observations do not establish occupancy; CPU models on distinct ports remain permitted.
  • Fixed in 8e7253e8: resolve and validate the exact addresses dialed under the endpoint context and five-second budget. Preserve explicit internal-host allowlists, Host/TLS identity, configured headers, same-origin redirects, and the ten-redirect bound.
  • Fixed in 321945f7: snapshot the resolver before asynchronous transport work and race checked literal addresses when the preferred address stalls. Cancel losing attempts and close unused connections without another hostname resolution.
  • Fixed in 997b568a: interleave address families so an alternate family starts after one fallback delay even when many preferred-family records precede it. Cover IPv4-first and IPv6-first answers.

Validation at 997b568a44281c4f8703a7c90be06976ba5648e0: full make ci-preflight passed, including hermetic tests, race tests, install tests, lint, build, coverage, public boundary, repository truth, documentation facts, workflow checks, and lifecycle checks. Regression probes reproduced the earlier GPU-overlap, DNS-deadline, rebinding, stalled-address, and alternate-family starvation failures; focused and race runs pass after the fixes. A separate review-only TLS probe also confirmed that endpoint cancellation closes a stalled handshake connection. The final fresh-context diff review approved with no findings or open questions.

Ripwire was assessed separately from executable test results. Quality-delta exited 2 with four gating heuristics: three test-duplication pairs involving renamed tests and the HTTP probe growing from 56 to 71 lines. Declined: the test pairs share setup while asserting distinct behavior, and the added probe lines keep its context, transport, TLS, and proxy policy together. Its other 14 rows concern test entry points marked as dead code and complexity in two regression tests; Go executes those entry points and the tests pass. Test-gate exited 4 with 69 test-file obligations and 13 untested symbols or documentation sections; the full test suite passed, while the latter are unchanged transitive graph results. No new regression-test gap was established, and coverage of every graph-reported symbol is not claimed. No heuristic acknowledgements were added.

HTTP MCP probes connect directly and ignore environment proxies so a proxy cannot resolve a different destination. An endpoint reachable only through a proxy can therefore produce an advisory probe failure. These probes establish listener reachability, not MCP protocol readiness. Validation used hermetic tests and local fixtures; live GPU workloads, operator-configured endpoints, and deployed behavior remain unverified.


Devin Review

@toasterbook88 toasterbook88 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AXIS Grok gate

Verdict: PASS-WITH-RISK

PR #656: fix(model,doctor,netutil): guard occupied GPUs and harden HTTP MCP dialing
Head: b5b53e8
Files touched (10): cmd/axis/doctor.go, cmd/axis/doctor_http_dns_test.go (new), cmd/axis/doctor_http_fallback_linux_test.go (new), cmd/axis/model.go, cmd/axis/model_start_gpu_guard_test.go (new), cmd/axis/model_start_test.go, internal/netutil/dial.go, internal/netutil/dial_fallback_linux_test.go, internal/netutil/dial_test.go, and related test updates.

Invariant hits: none.

  • GPU guard now refuses any observed ResidentModels GPU occupancy before llama-server launch. --main-gpu is correctly treated as non-isolating. Missing index observations do not invent occupancy. Receipts carry snapshot provenance (daemon-cache). Aligns with observed-state truth boundary.
  • HTTP MCP probes now resolve once under the endpoint context (5s budget), dial only checked literal addresses via NewOutboundDialer (snapshot resolver, no rebinding, no proxy bypass), preserve Host/TLS identity, same-origin redirects, and ten-redirect bound. Interleave + raced fallback preserves alternate-family reachability without a second hostname resolve. Direct connect is documented; proxy-only endpoints produce advisory failure.
  • No HITL/dispatch lock changes, no cache silent fallback, no secret_read / ssh_bypass / fleet_exec / spawn_subagent / self-authorize / unattended config write. Agent self-model untouched.

CI status (at review): Test & Build in_progress, govulncheck in_progress, Analyze (actions) in_progress, auto-merge skipped, Devin Review pending. mergeable_state: blocked. PR body reports local make ci-preflight green at prior SHA 997b568; full suite + race claimed green after fixes. Live GPU workloads and operator endpoints remain unverified (explicit in body).

Recommended operator action: Wait for CI green on this head. Confirm GPU-overlap rejection and MCP dial behavior on a live node with occupied GPU + dual-stack DNS if possible. No APPROVE. Do not merge until CI succeeds and residual live verification is acceptable.

Advisory only. Charter and invariants unchanged.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@testdriverai

testdriverai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Re-reviewed commit c3a8807 following the synchronization with main. The changes remain correct and clean with no regressions: llama-server start exclusivity is guarded against observed resident GPU occupancy, and Doctor HTTP MCP probes remain safely hardened with direct context-bound dialing, DNS rebinding mitigation, TLS identity preservation, and address fallback.

📋 Walkthrough (10 files)
File Change
cmd/axis/doctor.go Enforces 5-second context timeout, direct dialing via NewOutboundDialer, and same-origin redirect bounding for doctor HTTP MCP probes.
cmd/axis/doctor_http_dns_test.go Tests covering DNS timeout enforcement, rebinding defense, mixed address rejection, and TLS SNI preservation.
cmd/axis/doctor_http_fallback_linux_test.go Linux-specific test verifying doctor HTTP probe address fallback when preferred addresses stall.
cmd/axis/model.go Replaces --main-gpu isolation logic with unconditional rejection when resident GPU occupancy is observed before starting llama-server.
cmd/axis/model_start_gpu_guard_test.go Tests verifying that observed GPU occupancy causes a rejected model start with a gpu_occupied receipt.
cmd/axis/model_start_test.go Updates start tests to match the removal of --main-gpu isolation hints and verify CPU-only model coexistence.
docs/current-state.md Documents the hardened HTTP MCP probe dialing behavior and the strict GPU resident conflict guard policy.
internal/netutil/dial.go Provides NewOutboundDialer with context-bound resolution, pre-dial IP address validation, and family-interleaved concurrent dialing with fallback.
internal/netutil/dial_fallback_linux_test.go Tests verifying address family interleaving and fallback under backlog saturation.
internal/netutil/dial_test.go Unit tests for outbound dialer internal address allowlisting and context cancellation.

Was this review useful? React with 👍 or 👎 to help TestDriver improve.


Want help from the team that built TestDriver? Explore TestDriver for Business

@testdriverai

testdriverai Bot commented Nov 18, 2026

Copy link
Copy Markdown

⏸️ Skipped this review: 3/3 PR reviews used this hour for this organization. Push a new commit after Wed, 18 Nov 2026 00:50:25 GMT and I'll review it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants