Skip to content

SWI-10226 feat: OAuth2, live voice/messaging callbacks, hosted-mode + host resolution#8

Merged
kshahbw merged 65 commits into
mainfrom
feat/host-resolution-and-hosted-defaults
Jun 11, 2026
Merged

SWI-10226 feat: OAuth2, live voice/messaging callbacks, hosted-mode + host resolution#8
kshahbw merged 65 commits into
mainfrom
feat/host-resolution-and-hosted-defaults

Conversation

@kshahbw

@kshahbw kshahbw commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

BLUF

main is a thin OpenAPI→MCP wrapper: Basic Auth, one-shot API calls, no way to receive anything back. This branch turns it into a usable product — an AI agent can authenticate, provision from zero, place a call or send a message, and respond to live call/message webhooks in real time.

Supersedes #6 (the old feat/express-registration branch — same feature area, three months stale: still Express-branded, still Basic Auth, no host resolution, no audit fixes). Close #6 in favor of this. Carries SWI-10226.

What changed (by capability, not commit firehose)

Auth — OAuth2 client credentials (replaces Basic Auth)

  • src/oauth.py: exchanges BW_CLIENT_ID/BW_CLIENT_SECRET for a Bearer token, auto-discovers the account ID from the JWT accounts claim. No more BW_ACCOUNT_ID. Mirrors the band CLI.
  • setCredentials / clearCredentials (src/tools/credentials.py) for mid-session auth — the zero-to-one path.

Live voice + messaging (the headline main can't do)

  • Callback HTTP routes (src/callbacks.py) served on the same transport as MCP, for inbound messages and voice answer/gather/disconnect/continue.
  • In-memory EventStore (src/event_store.py) bridges webhooks (HTTP side) to the agent (getCallbackEvents/getInboundMessages).
  • generateBXML + respondToCallback (src/tools/voice.py) — pre-queue BXML before the call is answered, drive a two-way conversation.

Discovery (hand-written XML tools from_openapi can't generate)

  • listApplications, listPhoneNumbers, createApplication, configureCallbacks against the Dashboard XML API.

Context-window sanity

  • Tool profiles (src/profiles.py): curated ~27-tool default instead of dumping all 400+. BW_MCP_PROFILE=full for everything.
  • Dynamic, tool-aware instructions (src/instructions.py).

Build Registration (renamed from Express)

  • Single createRegistration tool. SMS/email verification happen in the user's browser — the API only kicks off signup. Verification ops removed (they broke the browser flow).

Hosted-mode capability + host resolution

  • streamable-http/sse transports, BW_MCP_BASE_URL, loopback-default bind, setCredentials withheld over HTTP.
  • BW_ENVIRONMENT=test/uat + per-host overrides, centralized in src/urls.py (swap_host so OpenAPI tools honor it too).
  • Opt-in cloudflared dev tunnel (BW_MCP_DEV_TUNNEL) for local callback testing.

Validation

  • 116 unit tests pass.
  • End-to-end smoke against stage + prod with real creds: auth, voice CRUD, messaging send, media, lookup, discovery all validated green within each credential's permission scope. Five real bugs found and fixed during smoke (host override reach, full profile, Content-Type on non-JSON bodies, /tns role fallback, lookup spec patch).
  • Not yet validated (blocked by credential roles / account entitlements, not code): listPhoneNumbers (Numbers role), MFA send (MFA role), prod lookup (account entitlement), stage messaging (separate auth realm), recordings (need a seeded recording). Handing these to someone with the right roles.

Not in scope

This makes the server hosting-capable; it does not deploy it. No Dockerfile/deploy target yet, and credentials are process-global (fine for local single-user, needs a per-tenant model before a shared hosted instance). Follow-up.

kshahbw added 30 commits March 27, 2026 00:52
Adds requires_auth parameter to _create_server, registers Express Registration API (no-auth) in api_server_info, and adds Express-specific tests. Total tool count updated to 49 (Express adds 2 net new tools; verifyCode deduplicates with MFA).
… collision

The Express and MFA APIs both had operationId: verifyCode, causing FastMCP
to silently drop one during import. Renamed to verifyRegistrationCode so
all 50 tools are accessible.
…fidelity

- Add idempotency guard to _reload_authenticated_servers (prevents tool
  duplication on repeated setCredentials calls)
- Fix mutable default config={} in _create_server and create_bandwidth_mcp
- Fix Express tests to use requires_auth=False (matches production config)
- Move warnings import to top of config.py
Adds AGENTS.md (structured reference for AI agents using the bw CLI)
and exposes it as resource://cli_agent_reference. Covers command
semantics, prerequisite graph, workflows, error patterns, and
limitations so agents can plan multi-step operations without
trial and error.
Adds AGENTS.md as a comprehensive reference for AI agents using the MCP server:
tool groups, required env vars, dependency order, common workflows, error patterns,
and limitations. Also registers it as a FunctionResource at resource://mcp_agent_reference
so agents can read it programmatically mid-session.
Adds src/instructions.py with build_instructions() that generates
context-aware MCP instructions based on loaded tools and config presence.
Also adds pythonpath config to pyproject.toml so src imports resolve
consistently across all test files.
Introduces resolve_profile() with named presets (messaging, voice, onboarding, lookup, full) and wires --profile CLI flag + BW_MCP_PROFILE env var into get_enabled_tools() as a fallback when no explicit tool list is provided.
Config resource was added after the test was written. Fixes 5 pre-existing failures.
Writes cleaned specs to ~/.bw-mcp/spec-cache after a successful fetch and
falls back to the cached version when the network is unavailable.
Calls build_instructions after setup() and _reload_authenticated_servers()
so mcp.instructions reflects the actual loaded tool set at runtime.
Ring-buffer backed EventStore with TTL, per-session read cursors for multi-session
safety, and first-write-wins BXML locking on CallState objects.
Implements BXML generation from verb descriptors and a callback-response
flow for active voice calls, with full test coverage across all supported
verb types and edge cases.
…ader

Express Registration spec isn't published to dev.bandwidth.com yet.
Bundle it in src/specs/ and load directly from disk. fetch_openapi_spec
now handles local paths before attempting HTTP fetch.
FastMCP.mount() expects another FastMCP server, not a raw Starlette app.
Refactored to use @mcp.custom_route() decorator which registers routes
directly on the MCP server's HTTP transport. Callback routes are now
registered during setup() alongside tools, always available in HTTP mode.
Agent can authenticate with just username/password and discover
the account ID from the API afterward.
setCredentials now takes client_id and client_secret, exchanges them
for a Bearer token via POST /api/v1/oauth2/token, and extracts account
IDs from JWT claims — same flow as the Bandwidth CLI. Account ID is
discovered automatically, not required from the user.

Startup OAuth: if BW_CLIENT_ID and BW_CLIENT_SECRET env vars are set,
token exchange happens during setup(). All API requests use Bearer auth.
Adds createCall and 27 other voice tools, numbers management,
and toll-free verification to the server. All specs fetched from
dev.bandwidth.com at startup.
Test now asserts tools > 0 and validates filtering behavior directly
instead of predicting a total count that breaks when specs change.
kshahbw and others added 15 commits April 2, 2026 23:44
When no BW_MCP_BASE_URL is set and transport is HTTP, the server
automatically starts a Cloudflare quick tunnel. Zero user config —
callbacks just work. Production deploys set BW_MCP_BASE_URL instead.

Verified end-to-end: tunnel URL → configureCallbacks → createCall.
respondToCallback now auto-creates call state if it doesn't exist,
allowing agents to queue BXML before the callee answers. The answer
callback checks for pre-queued BXML and serves it immediately instead
of redirecting. Instructions updated with correct sequence:
generate BXML → createCall → respondToCallback (before answer).
custom_route must be called before mcp.run() so Starlette includes
the routes in its app. Moving from lifespan to module level fixes
404 on callback URLs. Verified: Bandwidth callback received and
returned pre-queued BXML — voice call works end to end.
Agent needs to know exactly where to find accountId, applicationId,
from number, and answerUrl. Instructions now point to specific config
keys and give the exact answerUrl pattern. Added auto_gather guidance
for one-shot vs conversational calls.
…eApplication

Agent can now discover phone numbers and voice applications from the
account instead of requiring pre-configured env vars. createApplication
handles the edge case where no Voice-V2 app exists — it creates one
with callback URLs auto-pointed at the server.

Instructions updated: agent reads config, discovers resources,
creates app if needed, then makes the call. Zero pre-configuration
beyond credentials.
…rtup

configureCallbacks was hitting the Voice v2 JSON API which doesn't
actually update app callback URLs. Switched to Dashboard XML API
(GET current app, PUT with updated URLs). Server auto-configures
voice app callbacks on startup when BW_VOICE_APPLICATION_ID and
BW_MCP_BASE_URL are set.

Added gatherUrl to auto-generated Gather BXML so Bandwidth knows
where to POST speech results. Conversation flow verified: greeting
→ speech capture → response.
Drop requirements.txt and dev-requirements.txt in favor of pyproject.toml
with [project.optional-dependencies]. Update CI to pip install ".[dev]".
Remove root AGENTS.md (stale duplicate of src/specs/AGENTS.md). Clean up
5 unused imports across source files.
All host strings in src/ now flow through src/urls.py. Production hosts
are the defaults; each is overridable via its own env var (BW_API_URL,
BW_VOICE_URL, BW_DASHBOARD_URL, BW_MESSAGING_URL).
- setCredentials accepts secret material as tool arguments and is only
  registered under stdio transport.
- BW_MCP_HOST defaults to 127.0.0.1; production deploy sets it
  explicitly.
- The development tunnel (cloudflared) is now opt-in via
  BW_MCP_DEV_TUNNEL and prints a clear stderr notice when it engages.
Match the band CLI's host resolution model:

- BW_ENVIRONMENT=test|uat flips the API and Voice hosts to the test
  environment in one shot. Per-host overrides (BW_API_URL, BW_VOICE_URL,
  BW_MESSAGING_URL) still win.
- Drop BW_DASHBOARD_URL. The Dashboard XML API is served from the API
  gateway under {api_base}/api/v2, same shape the CLI uses, so a single
  BW_API_URL override now reaches both APIs.
- README tools list now mirrors src/profiles.py — previous version
  referenced operationIds (createLookup, getReports, Address/Compliance/
  Requirements-Packages) that no longer exist in any current profile.
- README env var section adds BW_ENVIRONMENT, BW_API_URL, BW_VOICE_URL,
  BW_MESSAGING_URL, BW_MCP_PROFILE.
- AGENTS.md host table reflects the new BW_ENVIRONMENT + single API
  gateway model.
- AGENTS.md surface description no longer overclaims parity with the
  band CLI command surface (lookup/MFA are MCP-only; 10DLC/TFV/numbers/
  topology are CLI-only). Limitations section spells out what's not
  exposed and where to find it.
Bandwidth's public name for the free voice-first trial is Build, not
Express. All user-visible references rename — filenames, profile key,
docs, instructions, the spec title. The API path /v1/express stays (it's
the actual URL, not a name).

Build Registration now exposes only createRegistration. The CLI's
band account register works the same way: kick off signup, then SMS
verification, password set, and API credential generation happen in
pages Bandwidth links the user to. If the MCP server consumes the SMS
OTP via API, the user's browser flow breaks — so sendVerificationCode
and verifyRegistrationCode are removed. instructions.py and AGENTS.md
spell this out and tell the agent to hand off to the user after the
kickoff.

Also nudges the agent to proactively offer Build Registration when the
user asks how to sign up, says they have no credentials, or wants to
test things out — most users won't know the flow by name.

Other cleanups in the same pass:
- Drop the @bandwidth.com email restriction in the spec (leftover from
  an internal-test version of the API). Example emails are generic now.
- Remove unused BW_MCP_AUTH_TOKEN from config.py. Easy to re-add once
  hosted auth is real.
Live-API smoke runs against stage.api.bandwidth.com and api.bandwidth.com
surfaced five real issues. All fixed; 116 unit tests still green.

1. BW_*_URL overrides only reached hand-coded tools — OpenAPI-derived
   tools used the spec's hardcoded prod host. urls.swap_host() rewrites
   the spec server URL by hostname; servers._create_server applies it
   before building the httpx client. Adds MFA + Insights to the host
   map so BW_MFA_URL / BW_INSIGHTS_URL exist and BW_ENVIRONMENT=test
   leaves them on prod (matching CLI messaging behavior).

2. BW_MCP_PROFILE=full silently loaded the curated 27 instead of all
   tools. resolve_profile() returns None for both "no profile set" and
   "profile=full"; get_enabled_tools couldn't tell them apart. Now it
   peeks at the raw string for "full" before delegating.

3. Non-JSON request bodies (BXML on updateCallBxml, raw media on
   uploadMedia) went out with no Content-Type because FastMCP from_openapi
   uses httpx content= when the body isn't a dict, and httpx doesn't
   auto-set the header for content=. Bandwidth then rejected with 415.
   Added a one-shot event_hook in _create_server that sniffs the body
   and injects application/xml / json / octet-stream as appropriate.

4. listPhoneNumbers hit /accounts/{id}/inserviceNumbers, which requires
   the inservice role — but the prod cred we tested with only has the
   Numbers role (which gates /tns), and the stage cred has neither.
   Try /tns first (mirrors the CLI's choice), fall back to inservice on
   403 so creds with the older role still work.

5. The upstream phone-number-lookup-v2 spec ships with empty request
   body schemas. FastMCP couldn't surface the body field to the agent,
   so the lookup tools were unusable. _patch_phone_number_lookup injects
   the real shape (phoneNumbers, confirmed against live API).

Smoke validation in this session:
- Stage cred: lookup returns real T-Mobile data; voice list/call CRUD
  works; messaging/MFA blocked by separate stage auth realm and missing
  roles (not MCP issues).
- Prod cred: messaging + media full surface (real SMS, real call);
  number list + lookup gated by account/role perms (not MCP issues).
The cloudflared dev tunnel was undiscoverable — no mention in the README,
AGENTS.md, or the served instructions, so neither a human nor an agent
would know it exists. An agent helping a user wire up local callbacks
would reach for ngrok instead.

Add a "Local Callback Tunnel (dev only)" section to README and a matching
subsection to AGENTS.md (served as resource://mcp_agent_reference). Both
cover the BW_MCP_DEV_TUNNEL flag, the cloudflared dependency, and the
dev-only caveat. AGENTS.md adds an explicit nudge: suggest the flag when
a user's local webhooks aren't arriving.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kshahbw kshahbw requested review from a team as code owners June 3, 2026 21:53
@bwappsec

bwappsec commented Jun 3, 2026

Copy link
Copy Markdown

Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
Open Source Security 0 0 0 0 0 issues
Licenses 0 0 0 0 0 issues
Code Security 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

ckoegel
ckoegel previously approved these changes Jun 5, 2026
The MFA API (mfa.bandwidth.com) doesn't accept OAuth2 client-credential
bearer tokens — it's Basic Auth only. Since this branch dropped Basic Auth
entirely in favor of OAuth2, the MFA tools (generateMessagingCode,
generateVoiceCode, verifyCode) can't authenticate and are dead weight.

Remove them from the default surface and docs until the MFA API supports
OAuth2. The tools are preserved intact on the feat/mfa-tools branch for a
clean re-add once the API catches up.

Removed:
- multi-factor-auth spec from servers.api_server_info
- the `mfa` profile + its slot in DEFAULT_TOOLS (now voice+messaging+lookup)
- MFA_SECTION + trigger from instructions
- mfa profile docs from AGENTS.md and README; MFA mentions in account-type
  and env-var sections
- the MFA use-case from common_use_cases.md
- MFA test fixture, the MFA spec_list case, the MFA instructions test, and
  multi-factor-auth from the integration/server mock loops

Kept: the BW_MFA_URL / mfa host mapping in urls.py (internal, undocumented
host-resolution plumbing — makes MFA a drop-in when it returns).

114 tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@kshahbw

kshahbw commented Jun 10, 2026

Copy link
Copy Markdown
Contributor Author

Update — MFA tools pulled (2026-06-10). The MFA API (mfa.bandwidth.com) doesn't accept OAuth2 client-credential bearer tokens — it's Basic Auth only, which this branch removed. generateMessagingCode, generateVoiceCode, and verifyCode are dropped from the default surface and docs until the MFA API supports OAuth2. They're preserved intact on feat/mfa-tools for a clean re-add. (The BW_MFA_URL host plumbing stays in urls.py so MFA is a drop-in when it returns.) 114 tests pass.

Folds in PR #7's security fix the right way. #7 bumped fastmcp in
requirements.txt — a file this branch deleted — so it couldn't merge.
Bump the pin in pyproject.toml instead: fastmcp~=3.2 (resolves 3.4.2,
fixes SNYK-PYTHON-FASTMCP-15871014/29/30 — SSRF + command injection)
and mcp~=1.24 (fastmcp 3.x requires mcp>=1.24).

fastmcp 3.x moved/renamed APIs we use; migrated all call sites:
- fastmcp.server.openapi → fastmcp.server.providers.openapi (MCPType) and
  fastmcp.utilities.openapi (HTTPRoute)
- get_tools()/get_resources() (dict) → list_tools()/list_resources()
  (lists of objects); source reads .name, tests use a tool_map() helper
- import_server() → mount() (sync; no namespace = bare tool names, so
  profile filtering by operationId is unaffected)
- dropped the obsolete FASTMCP_EXPERIMENTAL_ENABLE_NEW_OPENAPI_PARSER env
  flag (the new parser is the only one in 3.x)
- the per-tool httpx client moved from server._client to tool._client;
  added a server_client() test helper

114 tests pass; fastmcp deprecation warnings cleared (52 → 3, the
remaining 3 are our own intentional UserWarnings + a Starlette internal).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
configureCallbacks 400'd on every real call. Root cause: the flow
defaulted to types=["messaging","voice"] and set BOTH families' callback
fields, but the Dashboard API rejects messaging fields on a Voice-V2 app
and voice fields on a Messaging-V2 app:

  ErrorCode 12962: CallbackUrl is set but it is only allowed with
  ServiceType Messaging-V2

The function already fetched the app's ServiceType but never used it to
pick fields. Reordered to GET first, then derive the callback fields from
the actual ServiceType (Voice-V2 → CallInitiated/CallStatus; Messaging-V2
→ Callback/StatusCallback). The `types` arg is now advisory — an app has
exactly one ServiceType, so the app decides. Unknown service types return
a clear error instead of a 400.

Verified live against stage/prod: both Voice-V2 and Messaging-V2 apps now
return "configured" even when called with the old both-types default.

Also fixed a latent deprecation in the same function (Element truth-value
test → explicit `is None`).

Adds 3 regression tests (voice-only fields, messaging-only fields, unknown
type errors). 117 tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@kshahbw kshahbw merged commit 5c75bf8 into main Jun 11, 2026
14 checks passed
@kshahbw kshahbw deleted the feat/host-resolution-and-hosted-defaults branch June 11, 2026 18:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants