dashboard: something with hot reloading #114

Activity

1. quinn-dougherty commented on Jun 11, 2026

   

   quinn-dougherty

   on Jun 11, 2026

   MemberAuthor

   More actions

   Design notes: a hot-reload research dashboard

   Proposed shape: bun create next-app → ./dashboard at monorepo root — a live job monitor, "like inspect view but fuzzing-shaped" (CPU/mem, exec/s, coverage & crash curves). Captuing the design discussion here.

   TL;DR — this is a data problem wearing a UI costume

   The stack choice is fine; the blocker is that we don't emit the live, fuzzing-shaped telemetry a hot-reload view needs. Fix the data feed first, the frontend is then a thin reader.

   What's missing on the Python side

   1. No live/incremental stream. save_manifest only fires at run-level transitions (mark_run_started/completed/failed), so polling the manifest gives "12 of 36 grid cells done," not the in-progress curves that are the point of this issue. Mid-run FuzzingState lives in Inspect's TaskState.store and never hits disk.
   2. No CPU/mem data at all. Today FuzzingStats captures execs_per_sec, total_execs, corpus_size, crashes_found, plus the oom_killed/timed_out flags — and they're point-in-time per cycle, not a time-series. The "CPU stats" ambition isn't backed by collected data; it needs docker stats sampling on the sandbox container inside the campaign loop.
   3. Don't couple to .eval. The Inspect ZIP log format is internal, and the pydantic-ai/logfire migration (Instrument Logfire on top of Inspect-AI (observability) #138) may delete it. A dashboard that parses .eval breaks on that migration.

   The load-bearing decision: a project-owned feed

   Define a JSON/JSONL "dashboard feed" contract that Python writes and TS reads, decoupled from both Inspect and logfire. The runner + campaign loop append one event per cycle; the dashboard is a thin reader (Next route handler tails the file server-side, pushes via SSE — or dumb 1s polling, since presentation is explicitly out of scope). This survives #138.

   Strawman event (per cycle, append-only results/<exp>/events.jsonl):

   { "run_id", "ts", "cycle",
     "execs_per_sec", "total_execs", "corpus_size", "unique_crashes",
     "line_coverage_pct", "branch_coverage_pct",
     "cpu_pct", "rss_bytes", "oom_killed", "timed_out",
     "tokens", "fuzz_seconds" }      // last two feed the #135 efficiency metrics
   

   The real fork (decide before scaffolding): vs. Logfire

   #138 adopts Logfire, which is itself a live job monitor (timeline, token usage, span drill-down). So:

   • Logfire-native — push fuzzing metrics as span attributes, build Logfire dashboards. Near-zero infra, but hosted/external and awkward for fuzzing-domain views (coverage-over-time, CASR crash clusters, swarm marginal-crash curve).
   • Bespoke ./dashboard — full control of fuzzing-shaped views, runs locally/offline against the results dir, but we build/maintain it.

   Recommendation: both, split by job. Logfire owns agent-trace + token drill-down (link out, don't reimplement). ./dashboard owns what Logfire can't show well: exec/s & coverage curves, crash dedup / CASR clusters, the #135 efficiency metrics (vulns_per_fuzz_hour, vulns_per_mtok_equiv), CPU/mem, and SwarmResult.marginal_crash_curve.

   Fit & footguns

   • ./dashboard parallels scaffold/ and comms/ cleanly. It subsumes the Streamlit viz/ (slated for overhaul) and stays distinct from comms/site — which is exactly the "build-up-front website" this issue says not to do. Clean split: dashboard = live research, comms/site = presentation.
   • Polyglot monorepo: the existing .github/workflows/ci.yml is scaffold-scoped (ruff/ty/pytest) and won't touch the TS app — add a separate dashboard job for lint/build.

   Thinnest de-risking slice

   1. Emit events.jsonl from the runner per cycle.
   2. Next route handler tails it + SSE.
   3. One page: run grid + per-run exec/s & crashes sparkline.

   CPU/mem is a fast-follow once docker stats sampling lands. This sequence proves the live-data pipe before any UI investment.

   Open decisions

   • Logfire-native vs bespoke vs both (recommend both, split by job)
   • Lock the events.jsonl schema + the runner/campaign emission points
   • Confirm ./dashboard replaces viz/
2. quinn-dougherty

   added a commit that references this issue on Jun 11, 2026

   Bridge Inspect sample lifecycle into Logfire spans (#138, route 2)

   ...

   fbe9c0c
3. quinn-dougherty commented on Jun 16, 2026

   

   quinn-dougherty

   on Jun 16, 2026

   MemberAuthor

   More actions

   Updated stance: lean on Logfire heavily; shrink the bespoke surface

   Since the "both, split by job" comment above, the ground has shifted in Logfire's favor:

   • Instrument Logfire on top of Inspect-AI (observability) #138 instruments Anthropic/OpenAI/httpx + bridges Inspect sample lifecycle into OTel spans (target/score/crash/usage attributes already on each eval sample span).
   • Replace all vanilla Python logging with Logfire logging #145 / Replace vanilla logging with a Logfire-backed logging facade (#145) #146 route all our application logging through a Logfire-backed facade. Once we do the structured-attribute follow-up (deferred out of Replace vanilla logging with a Logfire-backed logging facade (#145) #146), our domain logs — exec/s, crash counts, coverage-profile counts, the develop metric for vulns per mtok and/or vulns per CPU-time (or vulns per some combination of mtok and time) #135 efficiency metrics — become first-class numeric attributes Logfire can chart and alert on natively. That's most of what the bespoke ./dashboard was being proposed to own.

   Revised recommendation: Logfire is the primary research dashboard. Build bespoke UI only for the residue Logfire genuinely can't express, and keep that residue as small as possible:

   • Likely-still-bespoke: crash dedup / CASR cluster views, SwarmResult.marginal_crash_curve (a derived shape, not a raw time series).
   • Move to Logfire: per-run/per-target exec/s & coverage curves, token drill-down, vulns_per_fuzz_hour / vulns_per_mtok_equiv (develop metric for vulns per mtok and/or vulns per CPU-time (or vulns per some combination of mtok and time) #135), error rates — all become Logfire queries/charts off structured attributes + the sample-span attributes from Instrument Logfire on top of Inspect-AI (observability) #138.

   This also collapses the live-data plumbing: no events.jsonl + SSE tail to build/maintain for anything Logfire already ingests. We get hot-reload-ish live views for free from Logfire's live tail.

   On viz/

   Maintainer says they don't care about viz/. So drop the "overhaul Streamlit viz" framing entirely — Logfire replaces viz/, it doesn't get rebuilt. comms/site stays as the separate presentation layer (the "build-up-front website" this issue says not to do for research).

   Updated open decisions

   • Logfire-native vs bespoke vs both → Logfire-native primary, bespoke only for CASR-cluster / marginal-crash views
   • Finish the structured-attribute follow-up to Replace all vanilla Python logging with Logfire logging #145/Replace vanilla logging with a Logfire-backed logging facade (#145) #146 so domain metrics land as Logfire attributes (prerequisite for Logfire owning the curves)
   • viz/ → deprecate/remove rather than overhaul
   • Decide whether the few bespoke views justify any standalone app at all, or just live as saved Logfire dashboards
4. quinn-dougherty

   added 4 commits that reference this issue on Jun 16, 2026

   Harden logging facade for structured kwargs + structure fuzzing hot p…

   ...

   c072c82

   Emit per-cycle fuzzing metrics to Logfire (#147)

   ...

   d415f9e

   Emit `model`/run identity so Logfire owns the research curves (#114)

   ...

   ea8de6c

   Add importable Logfire dashboard JSON for the #114 research views

   ...

   e072e2e
5. quinn-dougherty

   linked a pull request that will close this issueFuzzing harnesses link the real target library (fixes #151) #152on Jun 18, 2026
6. quinn-dougherty

   closed this as completedin #152on Jun 23, 2026