Skip to content

MANIFEST.md

Latest commit

 

History

History
90 lines (88 loc) · 12.6 KB

MANIFEST.md

File metadata and controls

90 lines (88 loc) · 12.6 KB
Raw

Manifest

Key Files

  • PLAN.md — Execution plan: research deltas, dependency DAG, parallel partition, build order, done-criteria.
  • AGENTS.md — Shared agent contract (Karpathy rules + deterministic-gate hard rule). Source of truth.
  • CLAUDE.md — Claude-specific notes; imports AGENTS.md.
  • README.md — Public pitch + architecture diagram + quickstart.
  • ROADMAP.md — Public-v1 maintainer backlog, current baseline, Codex/API-credit use plan, and non-goals.
  • DEMO.md — Local demo runbook, click path, API/MCP checks, and Alpic handoff.
  • CODE_OF_CONDUCT.md — Contributor conduct expectations and escalation path for unsafe or abusive behavior.
  • SUPPORT.md — Best-effort support scope, routing, and maintainer priorities.
  • .github/workflows/ci.yml — GitHub Actions gate for backend tests, seed eval report, frontend audit, lint, and build on push/PR/manual runs.
  • .github/workflows/codeql.yml — GitHub CodeQL scanning for Python and JavaScript/TypeScript on push/PR/weekly/manual runs.
  • .github/dependabot.yml — Weekly dependency and GitHub Actions update checks for backend, frontend, and workflow actions.
  • .github/ISSUE_TEMPLATE/ — Bug report and public-v1 task templates that require reproduction, tests, acceptance evidence, and public-truth impact.
  • .github/pull_request_template.md — PR review checklist for public claims, deterministic-gate safety, verification commands, and integration truth.
  • docs/PUBLIC_V1_TDD_FLYWHEEL.md — Public-v1 execution loop: TDD lanes, tracking rules, sub-agent packets, release checks, and maintenance cadence.
  • docs/CODEX_APP_PUBLIC_V1_OPERATING_MODEL.md — How to run Sentinel public-v1 through Codex app: orchestrator/worker/explorer roles, automations, plugins, waves, and governance.
  • scripts/smoke_mcp_http.py — Streamable HTTP MCP smoke script for local and deployed /mcp endpoints.
  • sentinel/contracts.py — Frozen interface: AdRequest, Claim, PipelineResult, Attestation, Verdict. Import; don't edit without a sync.
  • sentinel/config.py — Env-backed settings (pydantic-settings). All secrets via .env.
  • sentinel/main.py — FastAPI app; /health + /v1/analyze wired to the deterministic pipeline.
  • sentinel/mcp_server.py — FastMCP verify tool over streamable HTTP; wraps the same deterministic pipeline.
  • sentinel/tracing.py — Local audit JSONL persistence plus optional Overmind span emission.
  • sentinel/integrations/thrad_client.py — Thrad staging adapter with live-shaped payload normalization and deterministic mock fallback.
  • sentinel/pipeline/ — Thrad DistilBERT context classifier with heuristic fallback, claim extractor, Tavily-backed rating verifier with fixture fallback, safety judge, and deterministic decide_placement.
  • sentinel/pipeline/thrad_context_classifier.py — Lazy ONNX wrapper for Thrad/thrad-distilbert-conversation-classifier, pinned to a Hugging Face revision and mapped to A-M intent labels.
  • sentinel/attest/ — ed25519 attestation sign/verify helpers; signs from a PEM file (ATTESTATION_PRIVATE_KEY_PATH) or PEM env secret (ATTESTATION_PRIVATE_KEY_PEM, for hosted deploys).
  • alpic.json — Alpic deploy manifest with uv venv install/start commands for hosting the MCP verify tool (#12).
  • ui/ — Vanilla split-screen demo UI served by FastAPI at /demo/.
  • docs/assets/gifs/ — README demo GIFs (+ MP4 source) animated from retina /demo captures: console-tour, pipeline, block-verdict, approve-receipt.
  • docs/assets/layne-ui/v2/ — Retina (3840×2160) source captures of Layne's Next.js UI: landing + 4 post-verdict trace states (approve/block-anxiety/block-rating/block-urgency).
  • data/overmind_seed_cases.json — 25 {input, expected_output, expected_rule_fired} eval cases for Overmind optimizer seeding.
  • sentinel/eval.py — Runs seed (gated) + adversarial (measurement) splits; prints accuracy+95% CI, confusion matrix, per-verdict P/R/F1, and weighted cost (python -m sentinel.eval).
  • sentinel/eval_stats.py — Pure-stdlib eval statistics: Wilson 95% CI (plain + Yates), exact McNemar, Cohen's weighted kappa, Pearson. No numpy/scipy.
  • sentinel/eval_metrics.py — Auxiliary metrics: 3x3 confusion matrix, per-verdict precision/recall/f1, safety-first weighted cost.
  • sentinel/calibrate.py — Phase 5 L4 judge calibration: per-dimension quadratic-weighted kappa (judge vs human gold, target >=0.6) + length-bias probe, over a swappable JudgeFn seam (python -m sentinel.calibrate).
  • data/safety_gold.jsonl — Bootstrap human-label gold set for judge calibration (1-5 per dimension); seed only, needs >=200 traces / 2-3 annotators.
  • data/eval_cost_matrix.json — Cost[truth][prediction]; safety-first asymmetry (missed BLOCK = 10, ESCALATE = cheap valve).
  • data/adversarial_cases.json — Held-out split (negation/paraphrase/obfuscation/benign-trigger/minor/homoglyph) with correct verdict + currently_passes snapshot.
  • tests/test_eval.py — Parametrized regression over all 25 seed cases (verdict + rule_fired).
  • tests/test_eval_stats.py — Pins Wilson/McNemar/kappa/Pearson to published reference values.
  • tests/test_calibrate.py — Calibration harness tests: judge seam catches perfect vs miscalibrated judges; characterizes today's heuristic judge.
  • tests/test_eval_metrics.py — Confusion matrix, P/R, and safety-first cost-matrix invariants.
  • tests/test_adversarial.py — Characterization of the held-out split; breaks loudly when a case's behaviour changes; asserts held-out accuracy trails fitted.
  • tests/test_gate_robustness.py — Score-jitter sweep proving the gate never skips a band (no silent BLOCK<->APPROVE) and boundaries match policy.
  • data/policy.json — Ineligible contexts, score dimensions, block/escalate thresholds.
  • data/scenarios.json — 4 seed scenarios = the acceptance test for the demo.
  • tests/test_smoke.py — Health, API scenario, attestation, and MCP wrapper smoke tests.
  • tests/test_fact_verifier.py — Mocked Tavily match/mismatch, no-key fallback, failure fallback, source-hash, and deterministic-gate coverage for rating claims.
  • tests/test_public_api_contract.py — Public API/MCP contract tests for route envelopes, bounded audit queries, escalation decisions, and signed receipt verification.
  • tests/test_tracing.py — Local audit and optional Overmind span tests proving export success/failure never changes API verdicts.
  • tests/test_thrad_client.py — Thrad live-shaped bid normalization and timeout/500/malformed fallback tests.
  • tests/test_gate.py — Exhaustive deterministic-gate branch tests; proves no LLM verdict override input exists.

Recent Changes

  • 2026-06-01: Opened and documented the next Codex/API-credit hardening backlog (#23 claim-type expansion, #24 adversarial eval remediation, #25 hosted MCP controls).
  • 2026-06-01: Implemented issue #13 Tavily rating-claim verification behind TAVILY_API_KEY, with mocked match/mismatch tests, no-key/failure fixture fallback tests, source hashes on claims, and deterministic-gate coverage.
  • 2026-06-01: Added README known limits and release checks for issue #16, including hosted MCP smoke guidance for deployed /mcp URLs.
  • 2026-06-01: Hardened Thrad live-shaped bid normalization for issue #15 with OpenRTB-style payload support and timeout/500/malformed fallback tests.
  • 2026-06-01: Added trace tests for issue #14: local audit persistence without Overmind, mocked Overmind span attributes when configured, and API success when Overmind raises.
  • 2026-06-01: Hardened /v1/audit/latest with bounded limit validation and added public API/MCP contract tests for issue #16.
  • 2026-06-01: Added CODE_OF_CONDUCT.md, SUPPORT.md, and a general issue template to improve public maintainer readiness and contributor routing.
  • 2026-06-01: Truth-synced README and frontend copy so live Tavily, fixture fallback, optional Overmind export, and local audit traces are described according to the current code paths.
  • 2026-06-01: Synced README public verification copy to the current 115-test suite, added release badge, and documented active maintenance gates: CI, CodeQL, Dependabot security updates, secret scanning, push protection, and frontend audit.
  • 2026-06-01: Expanded CI to cover frontend npm audit, lint, and build; added CodeQL scanning for Python and JavaScript/TypeScript; added Dependabot schedules for pip, npm, and GitHub Actions.
  • 2026-06-01: Added GitHub issue templates and a PR template so future external contributions capture reproduction steps, test evidence, public-claim impact, and deterministic-gate safety before review.
  • 2026-06-01: Added GitHub Actions CI for backend tests plus the seed eval report, and added ROADMAP.md to make the public-v1 maintainer backlog and Codex/API-credit use plan reviewer-visible.
  • 2026-05-30: Added docs/CODEX_APP_PUBLIC_V1_OPERATING_MODEL.md — the Codex-app execution model for public-v1 work, covering orchestrator vs worker agents, plugin boundaries, recurring automation prompts, parallel waves, and production governance.
  • 2026-05-30: Added docs/PUBLIC_V1_TDD_FLYWHEEL.md — an executable public-v1 plan covering ground-truth integration states, red/green/refactor lane gates, what to track vs ignore, sub-agent packets for Tavily/Overmind/Thrad/LLM/public docs, release checks, and maintenance rhythm.
  • 2026-05-28: Added 4 README demo GIFs (docs/assets/gifs/) + retina source captures (docs/assets/layne-ui/v2/). GIFs are eased pan/zoom camera moves over real /demo post-verdict states (console tour, deterministic pipeline span tree, BLOCK+signed receipt, APPROVE+signed receipt); embedded in a new README "See it run" section. Full motion-graphics launch video deferred per Ved.
  • 2026-05-28: Built Phase 5 L4 judge calibration harness (calibrate.py, data/safety_gold.jsonl) — per-dimension quadratic-weighted Cohen's kappa vs human gold (target >=0.6) + length-bias probe over a swappable JudgeFn seam (defaults to the current pipeline's L4 scores). Seed gold: overall kappa 0.85, honestly flags annotator floor not met. Added kappa+Pearson to eval_stats.py. Not yet a CI gate (no LLM judge wired; gold is a seed).
  • 2026-05-28: Upgraded evals to May-2026 standard — added Wilson CI + confusion matrix + per-verdict P/R/F1 + safety-first cost matrix (eval_stats.py, eval_metrics.py); split-aware runner; held-out adversarial split (fitted 100% vs held-out 30%); gate-jitter robustness tests. Zero new deps. Sources in PLAN.md / memory.
  • 2026-05-28: Created PLAN.md — reviewed plan + research findings (Tavily/Overmind/MCP/Alpic/adtech-standards) + parallel partition.
  • 2026-05-28: Created MANIFEST.md — per CLAUDE.md file-manifest rule.
  • 2026-05-28: Updated sentinel/config.py — added overmind_service_name/overmind_environment (real SDK vars); marked overmind_project_id console-only.
  • 2026-05-28: Updated requirements.txt — added mcp[cli] for the MCP server (#8).
  • 2026-05-28: Wired /v1/analyze to the deterministic offline MVP pipeline; all seed scenarios now return expected verdicts.
  • 2026-05-28: Added attestation sign/verify helpers and tests proving valid signatures pass while tampering fails.
  • 2026-05-28: Added local FastMCP verify tool using streamable-http for the Alpic deployment path.
  • 2026-05-28: Aligned decide_placement with the policy-backed gate contract and exhaustive branch tests.
  • 2026-05-28: Made attestation deploy-ready — signing key loads from ATTESTATION_PRIVATE_KEY_PEM env when no key file exists; added alpic.json. Deployed verify() now returns signed (not silently unsigned) receipts.
  • 2026-05-28: Added local audit tracing, seeded optimizer cases, Thrad mock fallback, and /demo/ UI with escalation actions.
  • 2026-05-28: Added DEMO.md runbook for the presentation path and hosted deploy handoff.
  • 2026-05-28: Tightened Alpic config against current docs and enabled stateless JSON Streamable HTTP on the MCP server; env vars stay in dashboard settings.
  • 2026-05-28: Added a README Deploy on Alpic button to reduce the final dashboard deploy step.
  • 2026-05-28: Added Streamable HTTP MCP smoke script so local and Alpic /mcp endpoints can be verified with the same command.
  • 2026-05-28: Added sentinel/eval.py and tests/test_eval.py — 25-case regression suite with expected_rule_fired pins.
  • 2026-05-28: Updated AGENTS.md — continual-learning learned preferences/facts; gitignored .cursor/hooks/state/.
  • 2026-05-28: Integrated Thrad's open-source DistilBERT conversation classifier as an optional context-gate signal; policy-threshold D/J/M labels map into deterministic ineligible context flags, direct illegal/harmful asks stay heuristic-blocked, and missing model/deps fall back to heuristics.