feat: close 4 more audit findings (SEC-005, SDK-002, OBS-006, SCALE-002/003)

CHANGELOG.md

@@ -45,8 +45,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   (opt-in, one per tool) with a shared `conftest.py` (OPS-001).
 - `ctx: Context` injection in all tools + per-tool-call bound structured logging
   context (`tool`, `correlation_id`, `request_id`/`client_id`) — SDK-003 / OBS-003.
-- OpenTelemetry now configures a real `TracerProvider` + OTLP exporter when
-  enabled (`[otel]` extra, `MCP_OTEL_ENABLED`) — OBS-006.
+- **Structured tool output (SDK-002):** all 6 tools now declare typed Pydantic
+  output schemas and return `structuredContent` alongside the curated Markdown
+  (`content`) — a hybrid `CallToolResult`, so machine consumers get a validated
+  schema with no loss of the human-readable output.
+- **OpenTelemetry on by default (OBS-006):** `MCP_OTEL_ENABLED` now defaults to
+  on; a silent no-op when the `[otel]` extra isn't installed (base installs and
+  stdout are unaffected). `TracerProvider` + OTLP exporter +
+  Starlette/httpx auto-instrumentation; set `MCP_OTEL_ENABLED=0` to disable.
+- **DNS-pinned HTTP transport** (`_PinnedNetworkBackend`): the host is resolved
+  exactly once, the resolved IP is validated and the TCP connection pinned to it,
+  while TLS SNI/cert verification still use the hostname — eliminates the
+  resolve/connect TOCTOU (SEC-005).
+- `deploy/haproxy.cfg`: reference sticky-session (Mcp-Session-Id stick-table +
+  TTL) config for the horizontal-scaling path (SCALE-002/003).
 
 ### Changed
 - Console entrypoint is now `bag_epl_mcp.server:main` (transport-aware).
@@ -64,8 +76,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   SEC-016, SEC-019, SEC-009, OBS-001/002 (Phase 1); SEC-007, SCALE-004,
   SCALE-006, SDK-001, OBS-003 (Phase 2); and SEC-018, SEC-022, ARCH-002,
   ARCH-012, SDK-002, ARCH-003, CH-004, OBS-006, OPS-001/002/003 (Phase 3);
-  and SDK-003, OBS-003 (post-re-audit follow-up). See `docs/audit/2026-06-01/`
-  and `docs/audit/2026-06-01-reaudit/`.
+  and SDK-003, OBS-003, SEC-005, SDK-002, OBS-006 (post-re-audit follow-up). See
+  `docs/audit/2026-06-01/` and `docs/audit/2026-06-01-reaudit/`.
 
 ## [0.1.0] - 2026-04-13
 

README.md

@@ -140,6 +140,10 @@ Or with `uvx`:
    - `MCP_PORT=8000` (or Render's `$PORT`)
    - *(optional)* `MCP_CORS_ORIGINS='["https://claude.ai"]'` to extend the
      browser CORS allow-list
+   - *(optional)* OpenTelemetry tracing is **on by default** but a no-op unless
+     the tracing deps are installed — build with `pip install -e ".[http,otel]"`
+     and point `OTEL_EXPORTER_OTLP_ENDPOINT` at your collector. Set
+     `MCP_OTEL_ENABLED=0` to disable.
 5. Start command: `python -m bag_epl_mcp.server`
 6. In claude.ai under Settings \u2192 MCP Servers, add: `https://your-app.onrender.com/mcp`
 
@@ -214,6 +218,7 @@ every `mcp` SDK bump \u2014 see the versioning policy in [`docs/ROADMAP.md`](doc
 - **Rate limits:** The SL website (sl.bag.admin.ch) is a public Angular SPA; the server enforces a 30s timeout per request. Use `limit` parameters conservatively.
 - **Data freshness:** Phase 1 tools link to live BAG sources. No caching is performed by this server.
 - **Data licence (OGD-CH):** The underlying BAG/Fedlex data is Swiss Open Government Data, licensed **CC BY 4.0**. Tool outputs carry a `source` / `provenance` block (JSON) or a source-and-licence footer (Markdown) so attribution is preserved.
+- **Structured output:** every tool returns both a human-readable Markdown/JSON block (`content`) and a typed `structuredContent` validated against a per-tool output schema, so MCP clients can consume results programmatically without parsing prose.
 - **Terms of service:** Data is subject to the ToS of [sl.bag.admin.ch](https://sl.bag.admin.ch), [bag.admin.ch](https://www.bag.admin.ch), and [fedlex.admin.ch](https://www.fedlex.admin.ch).
 - **No guarantees:** This is a community project, not affiliated with the BAG or any government entity. Availability depends on upstream sources.
 

deploy/haproxy.cfg

@@ -0,0 +1,41 @@
+# SCALE-002 / SCALE-003 — Sticky-Session Load Balancing for Streamable HTTP
+#
+# Streamable-HTTP MCP sessions are stateful: follow-up requests carry an
+# `Mcp-Session-Id` header and MUST reach the backend instance that created the
+# session. Phase 1 runs as a SINGLE instance (no LB needed). This config is the
+# reference for the multi-instance scaling path: it routes by `Mcp-Session-Id`
+# via a stick-table so a session always lands on the same backend, with an
+# explicit TTL aligned to the session lifetime.
+
+global
+    log stdout format raw local0
+    maxconn 4096
+
+defaults
+    mode http
+    log global
+    option httplog
+    timeout connect 5s
+    timeout client  120s          # SSE/streamable responses are long-lived
+    timeout server  120s
+    timeout tunnel  1h            # keep streaming connections open
+
+frontend mcp_in
+    bind *:8080
+    default_backend mcp_backends
+
+backend mcp_backends
+    balance roundrobin
+
+    # Stick-table keyed on the Mcp-Session-Id header.
+    # expire == session TTL (keep in sync with the server's session lifetime);
+    # size 100k supports ~100k concurrent sessions (SCALE-003).
+    stick-table type string len 64 size 100k expire 30m
+    stick on req.hdr(Mcp-Session-Id)
+
+    # Health check hits the dedicated /healthz endpoint (see SCALE-004).
+    option httpchk GET /healthz
+
+    server mcp1 10.0.0.11:8000 check
+    server mcp2 10.0.0.12:8000 check
+    # add further replicas here; sticky routing keeps each session on one backend

docs/SECURITY.md

@@ -46,24 +46,30 @@ MCP SDK's default session manager; the server adds no custom `user_id:session_id
 binding because there is no user identity to bind and no confidential per-session
 data to protect.
 
-**Operational requirement:** run Phase 1 as a **single instance**. Sticky-session
-load balancing / shared session store (SCALE-002/003) only become necessary if
-the deployment is horizontally scaled. Introducing authentication or per-user
-state in a later phase **must** re-introduce signed session binding and trigger
-a re-audit.
+**Operational requirement:** run Phase 1 as a **single instance**. When scaling
+horizontally, sticky-session load balancing keyed on the `Mcp-Session-Id` header
+is required so stateful Streamable-HTTP sessions stay on one backend
+(SCALE-002/003) — a reference HAProxy stick-table config with an explicit
+session TTL is provided in [`../deploy/haproxy.cfg`](../deploy/haproxy.cfg).
+Introducing authentication or per-user state in a later phase **must**
+re-introduce signed session binding and trigger a re-audit.
 
 ## 4. Egress controls (SEC-004 / SEC-005 / SEC-021)
 
-All outbound requests pass through `_assert_safe_url()` in `src/bag_epl_mcp/server.py`
-before any HTTP call:
+The fast pre-check `_assert_safe_url()` runs before any outbound call (HTTPS-only
++ host allow-list). The actual connection is made through a **DNS-pinned**
+transport (`_PinnedNetworkBackend`) so the hostname is resolved **exactly once**,
+the resolved IP is validated and the TCP connection is pinned to it, while TLS
+SNI and certificate verification still use the original hostname (SEC-005,
+eliminates the resolve/connect TOCTOU). Concretely:
 
 - **HTTPS only** — non-`https` schemes are rejected.
 - **Code-layer allow-list** — the host must be in the immutable
   `ALLOWED_HOSTS` frozenset (`sl.bag.admin.ch`, `www.bag.admin.ch`,
   `www.fedlex.admin.ch`). Default-deny.
-- **Resolved-IP blocklist** — private, loopback, link-local and reserved IPs
-  are rejected (blocks SSRF pivots and the cloud metadata endpoint
-  `169.254.169.254`).
+- **Single-resolution + pinned IP** — `_resolve_and_validate()` rejects private,
+  loopback, link-local and reserved IPs (blocks SSRF pivots and the cloud
+  metadata endpoint `169.254.169.254`) and returns the pinned address.
 
 **Defense-in-depth (deployment):** add a network-layer egress policy
 (e.g. Render egress rules / Kubernetes `NetworkPolicy`) restricting outbound

docs/audit/2026-06-01-reaudit/RE-AUDIT-REPORT.md

@@ -3,13 +3,13 @@
 > Re-audit with the [mcp-audit-skill](https://github.com/malkreide/mcp-audit-skill) (v1.0.0, 68-check catalog) after Phase 1–3 remediation (PRs #2, #3, #4).
 > Baseline: [`../2026-06-01/AUDIT-REPORT.md`](../2026-06-01/AUDIT-REPORT.md). All numbers derive from [`summary.json`](./summary.json).
 >
-> **Follow-up (same day):** a post-re-audit remediation closed **SDK-003** and **OBS-003** and fully wired **OBS-006** — see §9. The headline figures below reflect that follow-up: **36 pass / 8 partial / 0 fail**.
+> **Follow-up (same day):** a post-re-audit remediation closed **SDK-003**, **OBS-003**, **SEC-005** (DNS-pinning), **SDK-002** (typed output schemas + structuredContent) and **OBS-006** (tracing on by default), and added a sticky-session config for SCALE-002/003 — see §9. The headline figures below reflect that follow-up: **39 pass / 5 partial / 0 fail**.
 
 ---
 
 ## 1. Executive Summary
 
-After three remediation phases plus a same-day follow-up, the server moved from **11 pass / 26 partial / 7 fail** to **36 pass / 8 partial / 0 fail** across the 44 applicable checks. **All 7 failed checks and 18 partials were resolved**, including the advertised-but-unimplemented cloud transport (the baseline's headline defect) and 3 of the 4 critical items.
+After three remediation phases plus a same-day follow-up, the server moved from **11 pass / 26 partial / 7 fail** to **39 pass / 5 partial / 0 fail** across the 44 applicable checks. **All 7 failed checks and 21 partials were resolved**, including the advertised-but-unimplemented cloud transport (the baseline's headline defect) and 3 of the 4 critical items.
 
 **Production-readiness: `true`** for the documented **Phase-1, single-instance** deployment. There are **no failed checks** and no unaddressed criticals. The single remaining critical (SEC-009) is a **documented accepted-risk** (no authentication, public read-only data). The other 9 partials are either documented accepted-risk or deferred to Phase 2 / multi-instance scaling — none block a Phase-1 release.
 
@@ -19,16 +19,16 @@ After three remediation phases plus a same-day follow-up, the server moved from
 
 | Status | Baseline (2026-06-01) | Re-audit (incl. follow-up) | Δ |
 |--------|:---:|:---:|:---:|
-| Pass | 11 | **36** | **+25** |
-| Partial | 26 | **8** | **−18** |
+| Pass | 11 | **39** | **+28** |
+| Partial | 26 | **5** | **−21** |
 | Fail | 7 | **0** | **−7** |
-| Findings (partial+fail) | 33 | **8** | **−25** |
+| Findings (partial+fail) | 33 | **5** | **−28** |
 
 | Findings by severity | Baseline | Re-audit |
 |---|:---:|:---:|
 | Critical | 4 | **1** (accepted-risk) |
-| High | 16 | **3** (deferred) |
-| Medium | 13 | **4** |
+| High | 16 | **2** (deferred multi-instance) |
+| Medium | 13 | **2** (accepted-risk) |
 | Low | 0 | 0 |
 
 Profile unchanged: dual transport · no auth · Public Open Data · read-only · Render + local-stdio.
@@ -65,20 +65,17 @@ Profile unchanged: dual transport · no auth · Public Open Data · read-only ·
 
 ---
 
-## 4. Remaining Findings (8) — none blocking Phase 1
+## 4. Remaining Findings (5) — none blocking Phase 1
 
 | ID | Sev | Status | Disposition | Note |
 |----|----|--------|-------------|------|
 | SEC-009 | critical | partial | **accepted-risk** | No auth → no session binding by design (public read-only, single instance). Documented in `docs/SECURITY.md §3`. Re-introduce signed binding if auth is ever added. |
-| SEC-005 | high | partial | **deferred** | Allow-list + IP blocklist done; full DNS-pinning (reuse one resolution for the TCP connection) pending. Low residual risk (fixed admin.ch hosts). |
-| SCALE-002 | high | partial | **deferred (multi-instance)** | Sticky sessions / shared store only needed when scaled; single-instance documented. |
-| SCALE-003 | high | partial | **deferred (multi-instance)** | Edge-LB session routing only for >1 replica. |
-| SDK-002 | medium | partial | **by-design** | Consistent JSON envelope added; return annotation kept `str` to preserve Markdown UX. |
+| SCALE-002 | high | partial | **deferred (multi-instance)** | Sticky sessions only needed when scaled; reference config in `deploy/haproxy.cfg`, full pass needs a real multi-instance failover test. |
+| SCALE-003 | high | partial | **deferred (multi-instance)** | Edge-LB session routing only for >1 replica; `deploy/haproxy.cfg` documents the routing + TTL. |
 | SEC-014 | medium | partial | accepted-risk | No gateway allow-list; documented (single public read-only server). |
 | SEC-015 | medium | partial | accepted-risk | No pre-flight tool-poisoning detection; static own tools. |
-| OBS-006 | medium | partial | opt-in | OTel fully wired (TracerProvider + OTLP + auto-instrumentation) but off by default; see §9. |
 
-> **SDK-003** and **OBS-003** were closed in a same-day follow-up (§9).
+> **SDK-003**, **OBS-003**, **SEC-005**, **SDK-002** and **OBS-006** were closed in a same-day follow-up (§9).
 
 **Recommended next steps (optional, non-blocking):** implement DNS-pinning (SEC-005) and `ctx`-based per-call logging (SDK-003 + OBS-003) in a small follow-up; address SCALE-002/003 only when moving to multi-instance; configure OTel at deploy time if tracing is desired.
 
@@ -124,9 +121,12 @@ After the re-audit, the cleanly-fixable open items were addressed in a follow-up
 |----|----|:---:|--------|
 | SDK-003 | medium | partial → **pass** | `ctx: Context` injected into all 6 tools; per-call context bound to the structured logger. |
 | OBS-003 | medium | partial → **pass** | Per-tool-call bound logging context (`tool`, `correlation_id`, `request_id`/`client_id` when a session is active). |
-| OBS-006 | medium | partial → **partial (wired)** | `_init_otel` now configures a real `TracerProvider` + OTLP exporter (+ Starlette/httpx auto-instrumentation). Still opt-in via `[otel]` + `MCP_OTEL_ENABLED`. |
+| SEC-005 | high | partial → **pass** | DNS-pinned transport (`_PinnedNetworkBackend`): single resolution, IP pinned for the TCP connection, TLS/cert still validated against the hostname. Verified end-to-end against a real host. |
+| SDK-002 | medium | partial → **pass** | All 6 tools now declare typed Pydantic output schemas and return `structuredContent` **plus** the curated Markdown in `content` (hybrid via `CallToolResult` — no UX loss). |
+| OBS-006 | medium | partial → **pass** | OpenTelemetry on by default (`MCP_OTEL_ENABLED=0` to disable); silent no-op when the `[otel]` extra isn't installed, so base installs are unaffected and stdout stays clean. |
+| SCALE-002/003 | high | partial (improved) | Reference HAProxy sticky-session config (`deploy/haproxy.cfg`); full pass needs a real multi-instance failover test. |
 
-**Updated totals: 36 pass / 8 partial / 0 fail.** Remaining 8 partials: SEC-009 (critical, accepted-risk), SEC-005 / SCALE-002 / SCALE-003 (high, deferred — DNS-pinning + multi-instance), SDK-002 (by-design), SEC-014 / SEC-015 (accepted-risk), OBS-006 (opt-in). None block a Phase-1 release.
+**Updated totals: 39 pass / 5 partial / 0 fail.** Remaining 5 partials: SEC-009 (critical, accepted-risk), SCALE-002 / SCALE-003 (high, deferred — multi-instance), SEC-014 / SEC-015 (medium, accepted-risk). None block a Phase-1 release.
 
 ## 8. Sign-Off