Seeds: 16 · samples/circuit: 512 · each cell is the mean AUROC of recovering the known ground-truth circuit (1.0 = perfect, 0.5 = chance). No API keys, no downloads.
| method | family | clean (control) | confounded | suppressor |
|---|---|---|---|---|
| random | baseline | 0.496 | 0.496 | 0.496 |
| correlation | proxy | 1.000 | 0.750 | 1.000 |
| gradient | proxy | 1.000 | 1.000 | 0.750 |
| integrated_gradients | causal | 1.000 | 1.000 | 1.000 |
| patching | causal | 1.000 | 1.000 | 1.000 |
On confounded, a non-causal feature is correlated with the output. Correlation-based attribution falls from 1.000 (clean control) to 0.750, while activation patching stays at 1.000 — it reads the finite causal effect, which the decoy does not have.
On suppressor, a non-causal feature has a large local gradient but ~zero net effect (two near-cancelling steep gates). Gradient saliency falls from 1.000 (clean control) to 0.750, while activation patching stays at 1.000. Integrated gradients — the path-integrated correction to plain gradients — recovers too.
Shuffle the ground-truth labels and every method collapses to chance, so the AUROC above is not an artefact of the metric.
| method | null AUROC |
|---|---|
| random | 0.495 |
| correlation | 0.490 |
| gradient | 0.486 |
| integrated_gradients | 0.505 |
| patching | 0.505 |
Reproduce:
python -m evals.harness(writes this file),python -m evals.gate(asserts the dissociation).