From 0d6ebe7af5a74a551eaab33de784bf62a278a429 Mon Sep 17 00:00:00 2001 From: Joshua Davis Date: Sat, 31 Oct 2026 17:59:19 -0400 Subject: [PATCH] State the project's status and known limits The README now says what this is today and links the threat model, and the roadmap shows its progress and the latest findings. Co-Authored-By: Claude Opus 5.5 --- README.md | 16 ++++++++++++++++ ROADMAP.md | 7 +++++-- 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 355c717..cbe7e7b 100644 --- a/README.md +++ b/README.md @@ -21,6 +21,21 @@ it was tested, and which beacons didn't answer. Use it as a web app, a CLI, a batch log scanner, a headless log watcher or an MCP server for AI assistants. Where it's headed: [ROADMAP.md](ROADMAP.md), on unmasking borrowed legitimacy. +## Status and known limits + +This is a working beta: a triage aid for an analyst to check, not something to act on without a person. Today: + +- **Accuracy rests on a small set.** Calibration uses 55 labeled IPs. Most paradoxes on known-malicious IPs fall away + once VirusTotal is on, as [the note on the results](#calibration) explains. +- **Evidence from OTX alone can be gamed.** In synthetic tests, 5 new OTX accounts take a clean IP to Likely malicious, + and OTX's own whitelist or padded pulses can hide real OTX evidence. Independent beacons hold up against the same + attacks. See [the adversarial results](ROADMAP.md#35-adversarial-test-suite--m). +- **Not yet tested:** use by real analysts, prompt injection against the summary, and attackers hiding on CDNs, cloud + hosts or Tor. +- **VirusTotal's free tier** rules out commercial use, and its limits make a 25-IP scan take at least 6 minutes. + +The threat model and its open gaps are in [SECURITY.md](SECURITY.md). + ## Install ```bash @@ -469,6 +484,7 @@ paradox/cli.py the `paradox` command: app + headless subcommands paradox/sources.py beacon adapters -> Signal paradox/scoring.py tier scores, paradox, threat paradox/engine.py concurrent scan + caching +paradox/quota.py keeps VirusTotal inside its free limits paradox/history.py SQLite scan history + lag measurement paradox/logscan.py batch mode: extract IPs from logs, scan, rank paradox/watch.py headless sensor: follow logs, alert, periodic rescans diff --git a/ROADMAP.md b/ROADMAP.md index 50e1057..a699881 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -24,13 +24,14 @@ Effort: **S** is about a day, **M** a few days, **L** a week or more. Each item has a permanent story ID (`PDX-n`) and a "done when" line for the [Muster board](https://github.com/Castor-o7/muster/blob/main/docs/board.md). A checked box means shipped, not verified. -The watchman keeps each story's state. +The watchman keeps each story's state. **Progress: 21 of 40 stories shipped (52%),** counting each story the same +whatever its size, and the total grows as work finds new stories. --- ## Where we stand -What the 2026-10-29 calibration (run 5) and the live tests showed, which sets the priorities below. A ✅ marks a shipped fix; the rescore after 1.1, 1.2 and 3.2 is in the README: +What the 2026-10-29 calibration (run 5), the live tests and later measurements showed, which sets the priorities below. A ✅ marks a shipped fix; the rescore after 1.1, 1.2 and 3.2 is in the README: | Finding | Evidence | Addressed by | |---|---|---| @@ -40,6 +41,8 @@ What the 2026-10-29 calibration (run 5) and the live tests showed, which sets th | Blocklists used as labels include research scanners | Shodan and Palo Alto Xpanse IPs were in the "malicious" set | 3.2 ✅ | | We assume which beacons are fast and slow, and never measured it | Tiers are hard-coded in `BEACONS` | 3.4 | | Veneer has no ground truth, so we can't state its precision | No labeled veneer cases exist | 3.3 | +| Evidence from OTX alone can be gamed | Synthetic attacks: 5 new accounts take a clean IP to Likely malicious; OTX's whitelist or padding can hide real evidence | 3.5 ✅ measured; 3.8, 3.9, 3.10 to fix | +| Most paradoxes on malicious IPs reflected the missing VirusTotal key | With it on, malicious at paradox 35+ fell from 28 to 4 of 30 | Open: how to frame the score | ---