modelscan-eval

A small meta-benchmark for static ML supply-chain scanners. It runs the same labeled, synthetic corpus of 47 files through picklescan, modelscan and a naive in-repo opcode baseline, then reports detection rate, false-positive rate and robustness to five pickle-evasion techniques. It evaluates scanners; it is not a scanner.

Headline results

Committed run on the 47-file corpus (canary group; detection excludes evasion and control items):

Scanner	Detection rate	False-positive rate
picklescan	11.1% (2/18)	0.0% (0/18)
modelscan	0.0% (0/18)	0.0% (0/18)
opcode_baseline	44.4% (8/18)	0.0% (0/18)

Evasion catches across the five reported techniques (one sample per technique; aggregate totals are not reported):

Evasion technique	picklescan	modelscan	opcode_baseline
indirect_aliased_global_import	1/1	1/1	1/1
nested_second_stage_payload	1/1	1/1	1/1
non_standard_reducer	0/1	0/1	1/1
opcode_level_obfuscation	0/1	0/1	1/1
protocol_framing_variance	0/1	0/1	1/1

Sources: results/REPORT.md, results/results.json.

All fixtures are generated by corpus/generate.py and committed. Nothing is downloaded, and CI runs offline.

Methodology

• Corpus (corpus/manifest.yaml): every item has a label (malicious or clean), a technique, and an optional evasion. Clean controls are matched to each malicious category.
• Control group: six extra items (payload: control in the manifest) whose pickles call a global that scanners denylist, os.system, with the fixed argument echo modelscan-eval-control. They repeat the base payload and the five evasion constructions below, so a miss on the canary group can be compared with a hit here. They are reported separately and do not enter the detection, false-positive or evasion rates above. tests/test_corpus_markers.py unpickles each control in a child process and asserts that stdout is exactly the echo line and that no file is created. The generator never writes the literal os.system (it imports system and builds the opcode streams by hand), so tools/safety_lint.py needed no allowlist.
• Payload safety: the only thing a canary fixture does when loaded is write one marker file (modelscan-eval.canary) into the current directory through corpus._marker.drop_canary. tools/safety_lint.py rejects network, shell and dynamic-execution tokens in corpus/, and tests/test_corpus_markers.py unpickles each marker pickle in a temp directory and asserts that the canary is the only file created. The harness itself never unpickles fixtures; it only shells out to the scanners (or, for the baseline, reads opcodes).
• Metrics (eval/metrics.py):
  • detection rate = flagged / total over malicious items with no evasion (18 items; control items excluded)
  • false-positive rate = flagged / total over clean items (18 items)
  • evasion robustness = flagged / total over the items carrying one evasion label (1 item per technique, so these are single-sample outcomes)
  • items a scanner was not run on (skipped) are excluded from denominators; scan errors count as unflagged and are tallied separately.

Scanners

Scanner	How it is invoked	Notes
picklescan	picklescan --path <file>, flagged on exit code 1	optional extra
modelscan	modelscan --path <file> --reporting-format json, flagged on exit code 1 (3 = no supported files, not an error)	optional extra; needs a Python version modelscan supports (3.12 used here)
opcode_baseline	pickletools.genops, flags any REDUCE, GLOBAL or STACK_GLOBAL; unwraps torch zips; ignores non-pickle files	deliberately naive denylist, a reference point and not a recommendation

Case taxonomy

Counts come from corpus/manifest.yaml.

Category (technique)	Format (kind)	Malicious	Clean	What it is
pickle_reduce	pickle (.pkl)	7	7	__reduce__ returning the canary function, in different containers (bare, dict, tuple, functools.partial, operator.methodcaller)
pickle_reduce	torch-style zip (.bin)	1	1	same payload inside archive/data.pkl
custom_code_import	Python source (.py)	5	5	model/tokenizer/processor code that calls the canary at import time
metadata_spoof	JSON config	5	5	typosquat, lineage, author, namespace and checkpoint spoofs in config metadata

Control group (6 items, pickle_reduce): control_reduce plus the same five evasion variants built around os.system.

Evasion variants (all pickle_reduce, one item each, derived from malicious_reduce):

Evasion	Construction
opcode_level_obfuscation	hand-assembled opcodes (protocol 2, GLOBAL/MARK/TUPLE)
indirect_aliased_global_import	getattr(importlib.import_module(...), ...) chain reaches the callable
nested_second_stage_payload	outer pickle calls pickle.loads on an embedded payload
non_standard_reducer	__reduce_ex__ instead of __reduce__, protocol 2
protocol_framing_variance	payload wrapped in a torch-style zip

Reproduce

Python 3.10+. modelscan did not install on Python 3.14 when this was written, so use 3.12 for the full run.

python3.12 -m venv .venv && source .venv/bin/activate
python -m pip install -e '.[scanners,dev]'
make check     # safety lint, ruff, pytest
make eval      # raw verdicts -> results/verdicts.jsonl
make report    # -> results/results.json, results/REPORT.md

Without .[scanners], make eval records those scanners as skipped and the report shows "no data" for them. Run eval and report in the same environment so the recorded scanner versions are accurate. To regenerate the fixtures: python corpus/generate.py (a test checks they match the committed bytes).

Example, one verdict line from the baseline:

$ python -m eval.runner --scanner opcode_baseline | head -1
{"detail": "dangerous opcodes: STACK_GLOBAL, REDUCE", "duration_ms": 0.21, "error": null, "flagged": true, "item_id": "malicious_reduce", "scanner": "opcode_baseline", "severity": "high"}

Results

Canary group, from the committed run (results/REPORT.md, raw data in results/verdicts.jsonl; picklescan 1.0.5, modelscan 0.8.8):

Scanner	Detection rate	False-positive rate
picklescan	11.1% (2/18)	0.0% (0/18)
modelscan	0.0% (0/18)	0.0% (0/18)
opcode_baseline	44.4% (8/18)	0.0% (0/18)

Read these with the limitations below. The low numbers for the real scanners are mostly a property of the corpus, not a ranking of the tools: the payload callable is corpus._marker.drop_canary, which is not on either scanner's list of dangerous globals. picklescan flagged only the two items that reach it through functools.partial and operator.methodcaller. The 10 py_code and metadata malicious items are outside what pickle scanners inspect, so they cap detection at 8/18 for any pickle scanner. See the evasion table in results/REPORT.md for per-technique outcomes.

Control group (same run): picklescan, modelscan and opcode_baseline each flagged 6/6, the base payload and all five variants. Every variant still names os.system/posix.system (or getattr/pickle.loads for the indirect and nested ones) in the stream, which is what a denylist matches, so these evasions do not hide a dangerous global from these scanners. Six single-sample items cannot separate scanners that all score 6/6; they show the canary misses come from the callable, not from the scanners being broken.

Limitations

• The canary payload calls a repo-local benign function, so denylist-based scanners have nothing to match; the control group covers the opposite case with one dangerous global (os.system) only. Other denylisted globals (builtins.exec, subprocess, ...) are not tested.
• Control fixtures contain the posix.system global and are only runnable on POSIX; the test is skipped on Windows.
• Small and synthetic: 47 files, one item per evasion technique. Results say nothing about prevalence or real-world repositories.
• Static scanning only; no runtime detonation.
• Pickle-focused. safetensors, GGUF, ONNX, Keras H5 and SavedModel are out of scope. The custom_code_import and metadata_spoof categories are included to show where pickle scanners are blind, not because they target them.
• Scanner behavior depends on installed versions and defaults (modelscan ran with its default settings; no settings file).
• Scan errors count as "not flagged".

See docs/PLAN.md for the original design notes.