Benchmark + autoresearch harness for the flattening stage

[mvdoc]

What this adds

A reproducible, provenance-tracked benchmark + autoresearch harness for optimizing the
pyflatten flattening stage, plus the first optimization result. All code lives under
benchmark/ (the installed autoflatten package is untouched); see benchmark/PLAN.md for
the full approved design and .claude/skills/autoflatten-autoresearch/ for session recovery.

Headline result: replacing the FreeSurfer-style projection + initial negative-area-removal
init with a guaranteed flip-free Tutte embedding reaches the same flattening quality at
~31% less runtime on the first hemisphere (sub-022 lh: 15.25% vs 15.20% mean distortion,
24 vs 25 flipped triangles, 454 s vs 658 s). The Tutte init starts at 0/385122 flipped
triangles (Tutte's theorem) vs the projection's 141690, so the ~4-min initial NAR phase is
unnecessary. Broader (n=4) validation and phase-removal experiments are running.

Components

• paths.py / ledger.py — output-location policy + append-only JSONL provenance ledger
  (commit SHA, env, seeds, metrics, artifacts, repro command, decision trace).
• metrics.py / harness.py — uniform per-patch metrics + method-agnostic
  evaluate(flatten_fn, ...) with on-disk k-ring caching.
• build_dataset.py — manifest from the public Narratives dataset (OpenNeuro ds002345),
  reusing cached projection patches (no FreeSurfer, no fetching).
• run_baseline.py — FreeSurfer-clone baseline + determinism check (verified bit-identical).
• probe_tutte_init.py — the flip-free (Tutte/LSCM) init probe.
• experiment.py — general autoresearch runner (init method + refinement-phase toggles).
• plot.py — fast flatmap renderer for visual verification.
• report.py — renders the ledger into a human-readable NOTEBOOK.md.
• Unit tests (benchmark/tests/), bench extra in pyproject.toml.

Output policy

Code + markdown docs live in the repo; all loop-generated artifacts (ledger, manifest,
flat patches, caches, figures, NOTEBOOK.md) are written to /data2/projects/autoflatten/,
not the repo. Each ledger record links its artifacts back to the producing commit SHA.

Status / verification

• Phase A (harness + baseline + ledger) verified end-to-end on CPU; determinism confirmed.
• Tutte probe verified flip-free and visually sound (clean single-blob flatmaps, no tangling).
• 12 unit tests pass; ruff clean.
• Compute is CPU-only (the box's GPUs are blocked by an old driver, 440/CUDA 10.2).

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces a comprehensive, reproducible, and provenance-tracked benchmark and autoresearch harness for the pyflatten stage of AutoFlatten. It includes dataset building from the public Narratives dataset, a method-agnostic evaluation harness, an append-only JSONL ledger for tracking experiments, quality metrics, plotting utilities, and a Tutte/LSCM flip-free initialization probe. The reviewer's feedback is highly constructive, pointing out several robustness improvements when parsing the JSON ledger (such as using .get() and handling explicit null values to avoid KeyError and TypeError exceptions) and highlighting a potential side-effect where mutating flattener.config in-place could unintentionally affect shared configuration objects across evaluations.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

To prevent a potential KeyError if the baseline record is missing the "metrics" key, use .get("metrics") instead of direct bracket access.

Suggested change

	base = Ledger().latest(kind="baseline")
	base_m = base["metrics"] if base else None
	base = Ledger().latest(kind="baseline")
	base_m = base.get("metrics") if base else None

[gemini-code-assist]

Mutating flattener.config directly modifies the shared config object passed into evaluate(). If the same config instance is reused across multiple evaluations or baseline runs, this side-effect will unexpectedly disable negative area removal for those runs as well. Consider copying the config or avoiding in-place mutation of shared arguments.

Suggested change

	flattener.config.negative_area_removal.enabled = False
	flattener.initial_projection = lambda: init # injected into run()
	import copy
	flattener.config = copy.deepcopy(flattener.config)
	flattener.config.negative_area_removal.enabled = False
	flattener.initial_projection = lambda: init # injected into run()

[gemini-code-assist]

To prevent a potential KeyError if the baseline record is missing the "metrics" key, use .get("metrics") instead of direct bracket access.

Suggested change

	def _baseline_reference():
	"""Most recent baseline record's aggregate metrics, for head-to-head printing."""
	rec = Ledger().latest(kind="baseline")
	return rec["metrics"] if rec else None
	def _baseline_reference():
	"""Most recent baseline record's aggregate metrics, for head-to-head printing."""
	rec = Ledger().latest(kind="baseline")
	return rec.get("metrics") if rec else None

[gemini-code-assist]

Use .get("experiment_id") instead of r["experiment_id"] to avoid raising a KeyError if any record in the ledger is missing the "experiment_id" key.

Suggested change

	records = Ledger().read()
	record = next((r for r in records if r["experiment_id"] == experiment_id), None)
	records = Ledger().read()
	record = next((r for r in records if r.get("experiment_id") == experiment_id), None)

[gemini-code-assist]

If "artifacts" is explicitly set to null in the JSON ledger, record.get("artifacts", []) will return None, causing a TypeError during iteration. Using record.get("artifacts") or [] ensures it always defaults to an empty list.

Suggested change

	artifacts = [
	a for a in record.get("artifacts", []) if a.get("kind") == "flat_patch"
	]
	artifacts = [
	a for a in (record.get("artifacts") or []) if a.get("kind") == "flat_patch"
	]

[gemini-code-assist]

If "git" is explicitly set to null in the JSON ledger, r.get("git", {}) will return None, causing an AttributeError when calling .get("commit"). Using r.get("git") or {} ensures it always defaults to a dictionary.

Suggested change

	commit = (r.get("git", {}).get("commit") or "")[:8]
	dirty = "*" if r.get("git", {}).get("dirty") else ""
	git = r.get("git") or {}
	commit = (git.get("commit") or "")[:8]
	dirty = "*" if git.get("dirty") else ""

[gemini-code-assist]

If "metrics" is explicitly set to null in the JSON ledger, r.get("metrics", {}) will return None, causing an AttributeError when calling .get(). Using r.get("metrics") or {} ensures it always defaults to a dictionary.

Suggested change

	m = r.get("metrics", {})
	m = r.get("metrics") or {}

[gemini-code-assist]

If "git" is explicitly set to null in the JSON ledger, r.get("git", {}) will return None, causing an AttributeError when calling .get("commit"). Using r.get("git") or {} ensures it always defaults to a dictionary.

Suggested change

	git = r.get("git", {})
	lines.append(
	f"- **commit**: `{git.get('commit')}` ({git.get('branch')})"
	+ (" ⚠️ dirty tree" if git.get("dirty") else "")
	)
	git = r.get("git") or {}
	lines.append(
	f"- **commit**: `{git.get('commit')}` ({git.get('branch')})"
	+ (" ⚠️ dirty tree" if git.get("dirty") else "")
	)

[gemini-code-assist]

If "environment" is explicitly set to null in the JSON ledger, r.get("environment", {}) will return None, causing an AttributeError when calling .get(). Using r.get("environment") or {} ensures it always defaults to a dictionary.

Suggested change

	env = r.get("environment", {})
	lines.append(
	f"- **env**: jax {env.get('jax_version')} [{env.get('jax_backend')}], "
	f"python {env.get('python')}, host {env.get('hostname')}"
	)
	env = r.get("environment") or {}
	lines.append(
	f"- **env**: jax {env.get('jax_version')} [{env.get('jax_backend')}], "
	f"python {env.get('python')}, host {env.get('hostname')}"
	)

[gemini-code-assist]

If "decision" is explicitly set to null in the JSON ledger, r.get("decision", {}) will return None, causing an AttributeError when calling .get(). Using r.get("decision") or {} ensures it always defaults to a dictionary.

Suggested change

	decision = r.get("decision", {})
	if decision.get("hypothesis"):
	decision = r.get("decision") or {}
	if decision.get("hypothesis"):

[mvdoc]

Autoresearch results (CPU, public Narratives benchmark)

Tutte flip-free init — verified across 4 hemispheres / 2 subjects (sub-022, sub-026):

	mean distortion	total flips	mean runtime
baseline (FS-clone)	14.73%	121	702 s
Tutte init	14.75%	121	442 s (−37%)

Equal quality (distortion +0.03pp, flips identical), ~37% faster. The Tutte map starts flip-free
(0 vs ~141k for the projection), so the initial negative-area-removal phase is unnecessary. Maps are
visually clean on both subjects.

Ablations — the final NAR and spring smoothing are NOT removable (a "numbers lie" result):
skipping the final NAR posts the lowest distortion in the study (13.21%) but leaves ~21000
flipped triangles — a folded, invalid map. Only the initial NAR is removable. LSCM init ≈ Tutte.

Full provenance for all 9 experiments is in the ledger (NOTEBOOK.md / experiments.jsonl under
the data root). Curated summary added in benchmark/FINDINGS.md.

[mvdoc]

Optimizer-algorithm experiments (answering "can we swap the optimizer?")

Tested replacing the FreeSurfer-style line-search GD with off-the-shelf optimizers (scipy
L-BFGS-B / CG) on the same metric+area energy, from the flip-free Tutte init.

• L-BFGS folds: reaches 10.31% distortion in 83 s (~8× faster) but 58025 flipped triangles.
• Pareto-worse than baseline: sweeping the area weight can't get low distortion and low flips
  (to match ~24 flips it needs ~25–29% distortion vs baseline 15%).
• Epoch weight schedule doesn't help: still folds (71269 flips at the distance-dominant stage).
• One-sided flip barrier: keeps injectivity (~0 flips) but L-BFGS then stalls at the init
  (33.7%) — local distance-reducing moves all push toward folding.

Conclusion: the FreeSurfer optimizer's value is the multi-level gradient smoothing (coherent
coarse-to-fine moves that avoid folding) + NAR, not the line-search algorithm. A drop-in optimizer
swap is a dead end on this energy; the productive directions are multiscale/hierarchical
(geometric-multigrid) optimization or a proper SLIM local-global solver. Details in
benchmark/FINDINGS.md; all runs in the ledger (10 experiments).