Inspect what a trained neural network contains — its tensors, its structure, and (separately, below) what it will say if you ask it.
Two projects live in this repository and they have different evidentiary standards. Keeping them apart is the point of this README.
universal_decoder/— a model-file inspection library for GGUF, Ollama and PyTorch state dictionaries. It reads tensors, metadata and structure. The "books" it emits are structural and parameter statistics. One committed sample says so explicitly: it does not prove full model knowledge extraction.genuine_decoder/— the research pipeline behind the headline numbers. It is a closed-world elicitation and injection pipeline, and this README describes exactly what it does, including the control that bounds what its headline number means.
Being blunt about the mechanism, because the difference matters:
The facts are not discovered; they are checked against a registry. A human
authored relation_registry_v1.json: 8 relations × 16 subjects = 128 candidate
facts, each with a gold answer. Models are then prompted, not read. Each model
gets 5 templates per fact, its generated text is checked against the registry,
and rows that agree across models survive. The 17 facts that reached the
injection stage are 6 country capitals, 10 atomic numbers and 1 element symbol.
The "read out of the weights" claim holds for the activation-line work, not for
this chain. The causal battery in Why this is hard — patching, dose-response,
position selectivity — is real and is the strongest thing here. It is what
produces the concept-level pages. The 99.3% number does not come from that
battery; it comes from the registry-checked elicitation path described above.
Injection is ordinary fine-tuning. The 17 facts expand into 68 template
sentences and distilgpt2 (82M parameters) is trained on them with AdamW for 8
epochs. This does train a model, and this repository contains the code that does
it (genuine_decoder/brain_injection.py).
The causal tier is real but currently blocked. It passes 8 of 8 eligible rows, but that relation has 8 eligible rows against a sampling floor of 10. The repo's own status document is right that the published union rests on the agreement tier.
Most "model introspection" stops at correlation. You find a neuron that fires on Python syntax, note that it fires a lot, and call it a concept. That is a guess.
This project refuses to publish a guess. Every candidate concept has to survive a causal battery:
| Stage | Question |
|---|---|
| Unsupervised discovery | Does a structure exist at all, before anyone names it? |
| Competing hypotheses | Which of several meanings does it actually encode? |
| Hard negatives | Does it predict anything a trivial confound also predicts? |
| Causal battery | Does patching it move the model's output, monotonically, by dose? |
| Position selectivity | Is the effect about content, or merely about position? |
| Fresh replication | Does it survive in a new process with a different seed? |
| Sealed held-out | Does it hold on data consumed exactly once, at the end? |
Cross-model agreement is used as a second, independent trust signal. Relations come from cross-intervention; procedures come from causal ordering across layers.
On TinyLlama, the frozen patch layer is a dead zone. Subject-token patches flip the answer at layers 0–16 and do nothing at 17–21. This matches earlier causal tracing work on factual recall, so it is confirmation rather than a new discovery — but it was found by running the battery rather than by assuming, and it is kept.
Numbers below are recomputed from the committed artifacts in this repository. Where something did not work, it is recorded as a failure rather than quietly dropped.
Three open models (SmolLM2-360M, TinyLlama-1.1B, Qwen2.5-1.5B) were run through a frozen fact protocol — the thresholds were committed to git before any evaluation existed:
3 source models -> frozen fact protocol -> 70 validated beliefs
-> cross-model union with voting -> 17 corroborated facts
-> expansion into training text -> fine-tuning of a base model
-> 99.3% closed-book accuracy on phrasings never trained on
- Gold precision 1.000 on every published belief. The extractor under-reads its
sources rather than over-claiming: the frozen boundary rule accepts answers only as
a leading prefix, so
city of Parisscores as a miss. Every funnel number here is a lower bound on what the source model knows. - No two models ever contradicted each other across 384 elicitations and 51 believed facts. Agreement is a usable trust signal.
- Rehearsal is a Pareto win. Mixing general text into the injection corpus improved knowledge and reduced forgetting at all six grid points.
- Hyperparameters were pre-registered, and the selected configuration was re-validated on seeds it was not chosen on.
The 99.3% is the mean of four seeds (97.1%, then 100% three times) on 34 eval
prompts. It was published without a control: nobody trained distilgpt2 on the
registry's gold answers directly. That control has now been run
(genuine_decoder/gold_facts_control.py, result in
genuine_decoder/results/gold_control_v1/control_report.json).
Parity was enforced before the control was scored — same base model and pinned revision, same pre-registered optimizer settings, same four seeds, same 34 eval prompts read from the committed receipt, same forgetting probe — and the training text was verified byte-identical to the corpus behind the published number:
| Published arm | Registry-gold control | |
|---|---|---|
| Closed-book accuracy, mean of 4 seeds | 0.993 | 1.000 |
| Forgetting, mean | +1.928 nats | +1.893 nats |
| Training text | 68 sentences from the union | the same 68 sentences, byte for byte |
The control scores at least as high as the published arm. The 99.3% measures the 17 gold answers that were already sitting in the human-authored registry. The model-extraction pipeline contributes nothing measurable to that number. Given the byte-identical corpus this is the only outcome available — the control cannot differ — but it is worth stating plainly rather than leaving the number to imply more than it carries.
Full write-up: docs/gold_facts_control.md.
A second, weaker probe was added at the same time: 17 prompts from phrasings that appear in no corpus in this repository, authored after the fact and therefore not pre-registered. The injected model scores 11.8% on them, against a 0.0% baseline. Generalization to genuinely new phrasings is near chance; the original eval was measuring rewording of the training templates, as its own caveat said.
What the control does not show, stated in the report itself:
- It does not show the extraction pipeline is useless. The pipeline still decides which 17 of 128 registry facts get injected, and a hand-picked control cannot audit that choice.
- It does not test knowledge beyond 17 easy facts. Registry v2 (288 facts) and the causal tier remain blocked on registry size, not on method.
- It does not clear the 34-prompt eval. Only a genuinely held-out eval would.
These are kept, not hidden:
py_block_colon_duefails dict-key, slice and complete-statement hard negatives on every model. The feature encodes "is Python code", not "a colon is due".sentence_period_duefails the open-quotation stratum. The feature marks "a clause is complete", which is also true inside an unterminated quotation.arith_answer_dueshows non-monotone dose response on one model — necessity fails — despite a sealed held-out AUC of 1.000.
The remaining publication blocker is registry size, not method: the causal tier passes 8 of 8 eligible rows, but the relation has 8 eligible rows against a sampling floor of 10. That floor is a threshold the protocol-amendment path correctly refuses to touch, so those rows stay diagnostic and unpublished. Widening the registry is authoring work, not science.
Reported here rather than papered over:
fact_union_v1self-hashes do not verify. All 8 relation tables carry arelation_union_sha256that does not match a recomputation over the committed bytes, andunion_manifest.json'sfact_union_sha256does not match either. The union artifacts were written by an earlier code revision whose dict layout differed at hash time. The stored content is consistent and the eval set derived from it reproduces exactly, but the hashes are not verifiable receipts.injection_corpus_v1cannot be rebuilt to the same digest, because it bindsfact_union_sha256(above) and the union binds absolute source paths insource_tables[*].path. The 34 eval prompts and the 68 training sentences do regenerate identically; only the enclosing digest is not portable.train.jsonlandeval.jsonlare not committed, because.gitignoreexcludes*.jsonl. The corpus itself is regenerable from the committed extraction artifacts, and the control commits byte-identical copies undergenuine_decoder/results/gold_control_v1/, but the original pair is not in the tree.universal_decoder/__init__.pyreports__version__ = "0.1.2"whilepyproject.tomldeclares1.0.0.
Requires Python 3.11+. The default install gets you the library, the CLI, and
structural GGUF/Ollama/PyTorch inspection. numpy is the only hard dependency.
pip install -e .The activation-level attribution backends, everything under genuine_decoder/,
and the tests that train a real model need the internal extra:
pip install -e ".[internal]" # torch, transformers, safetensors, huggingface_hub, ...
pip install -e ".[test]" # pytest + torch + transformers, for the full suite
pip install -e ".[all]" # everythingPoint the decoder at your models with an environment variable — no path is hardcoded:
export UD_MODEL_ROOT=/path/to/your/models# what can this machine actually read?
universal-decoder list-local-sources
# decode an installed Ollama model into verified book chapters
universal-decoder run-installed-gguf --output universal_decoder/results/installed_models_v1
# verify a published Brain Pack's checksums
universal-decoder verify-brain-pack --input <pack-dir>The package is also importable:
from universal_decoder.v1.library_api import detect_model, get_capabilities, plan_extraction
info = detect_model("/path/to/model")
plan = plan_extraction("/path/to/model")Stated plainly, because the boundary is the point:
- No downloads, no
ollama pull, no network access to model sources. - The structural library runs no inference.
universal_decoder/v1reads tensors and metadata; it never runs a forward pass to answer a question. - The research pipeline does run inference, and it does train a model.
genuine_decoder/prompts source models to elicit facts, andgenuine_decoder/brain_injection.pyfine-tunesdistilgpt2on the decoded facts. No trained weights are committed, and the injection weights are not persisted — the run is reproducible from its manifest. Earlier versions of this README claimed "no inference, no chat, no generation" and "no model training" for the whole repository; that was true ofuniversal_decoder/v1and false ofgenuine_decoder/, and the blanket claim has been withdrawn. - Raw activation captures are not committed. Note this is narrower than it sounds:
~330
.npzand.npyarrays are tracked, including SAE directions and statistics. What is excluded iscapture_*.npzactivation dumps; each run'sartifact_manifest.jsoncarries their SHA-256. - The pipeline is generation-based. The README once said the fact chain "reads parameters directly, with no generation". It does not: it prompts models and checks their text against a registry. That correction is the same one that makes the control above necessary.
universal_decoder/ the package
v1/ the shipping library and CLI (structural inspection)
internal_attribution/ activation-level extraction backends
internal_lab/ the research platform
research_v2/ the atlas90 experimental line
results/ committed evidence from real runs
genuine_decoder/ fact protocol, cross-model union, injection, and the
gold-facts-only control
sandbox_universal_decoder/ standalone parameter probes
tests/ the library test suite
docs/ design notes and measurement reports
universal_decoder/results/ holds ~5,800 committed artifacts across 851
directories, and many carry the absolute paths of the machine that produced them
(C:/ud-home/...). The evidence is kept, but expect noise when navigating a fresh
clone.
A few properties this codebase holds to, because they are what make the rest trustworthy:
register_amended_fact_protocolrefuses threshold weakening. It permits changes to declared measurement methods only. Tests prove it rejects ten kinds of threshold relaxation.- Coverage is not inflation. Semantic knowledge coverage is reported as
not_measurablerather than as a percentage, wherever it genuinely is not measurable. - Held-out data is consumed once. The sealed split is read exactly once, at the end, and re-reading it is a test failure.
- Results directories are pruned from releases. Model files, databases, secrets and raw tensor dumps are excluded from packaged artifacts.
- Controls are run even when they are unflattering. The gold-facts-only control above was the most informative experiment in the repository and it does not support the number it was run to protect.
python -m pytest tests genuine_decoder -qTwo suites with different dependency floors:
tests/— the library. Runs on a bare install.genuine_decoder/— the research pipeline. Needs torch.
A test that cannot run on this machine skips; it does not fail. Guards live in
universal_decoder/testing.py:
from universal_decoder.testing import requires_torch, requires_cached_model- Missing torch is a skip, never a collection error. Fifteen test modules import
activation-level backends at module scope — all 11 in
genuine_decoder/plus 4 undertests/— and produced fifteen collection errors on a clone without torch beforepytest.importorskip("torch")was added above their imports. - Missing model files are a skip. Real-model tests are guarded on the local
HuggingFace cache and on
UD_MODEL_ROOT; none of them touches the network. - No test hardcodes a machine's home directory. One test (
test_iterative_discovery.py) had a path relative togenuine_decoder/that only resolved when pytest was invoked from inside that directory; it is now anchored to the test file.
Verified from a clean environment:
| Environment | tests |
genuine_decoder |
|---|---|---|
| Full (torch + cached models) | 231 passed, 12 skipped | 152 passed |
| torch present, empty model cache | 231 passed, 12 skipped | 78 passed, 74 skipped |
| No torch | 195 passed, 9 skipped | all 11 modules skipped |
Counts are for the state of the tree at the time of writing and will drift; the
claim worth checking is the shape: no failures and no collection errors in any
of the three. pytest tests genuine_decoder -q also works as one command.
A number of tests verify committed historical evidence, which is why some result artifacts are large — they are the receipts. Those receipts have the defects listed under Known evidence-chain defects; the control exists partly because the original corpus files are not in the tree.
MIT — see LICENSE.