KuraFormer

Kuramoto oscillatory dynamics as parameter-efficient adapters for pretrained Transformers.

KuraFormer injects Kuramoto oscillator modules between Transformer blocks, enabling iterative refinement of hidden representations at inference time. We evaluated on GSM8K and MBPP with Mistral-7B and LLaMA-3-8B.

Important finding: A static adapter with the same bottleneck architecture but no Kuramoto dynamics matches KuraFormer's accuracy (54.1% vs 53.9% on GSM8K). The oscillatory dynamics do not contribute to accuracy --- the performance comes entirely from the down-projection/up-projection bottleneck. See the paper (v2.0.0) for full ablation and analysis.

Results

Static Adapter Ablation

The critical experiment: removing Kuramoto dynamics entirely and keeping only the bottleneck.

Adapter	Params	GSM8K Accuracy
Static (no Kuramoto)	33.56M	54.1%
KuraFormer v2	33.57M	53.9%
LoRA (r=48, reference)	40.9M	57.4%

The static adapter matches KuraFormer with ~10K fewer parameters. Kuramoto dynamics add complexity without improving accuracy.

Test-Time Scaling (v1 vs v2)

Before the ablation invalidated the core thesis, we discovered an interesting phenomenon in the dynamics:

v1 (random init): Both models exhibit a convergence window --- accuracy improves with more Kuramoto steps up to a model-specific optimum, then degrades.

Steps	Mistral-7B	LLaMA-3-8B
4	23.3%	46.3%
optimal	38.1% (16)	52.2% (8)
32	33.6%	50.9%

v2 (warm-start + schedules): Warm-start initialization with step-size and damping schedules eliminates the convergence window, producing flat curves. However, this stabilization effectively makes the dynamics a near no-op --- which the static adapter ablation confirmed.

Steps	Mistral-7B	LLaMA-3-8B
4	37.2%	53.7%
64	36.9%	53.9%

OOD Detection via Kuramoto Energy

We investigated synchronization energy as an out-of-distribution detector. Four controls revealed it is largely a proxy for base-model perplexity:

Method	AUROC
Kuramoto energy	0.90
Base model perplexity	0.85
Static adapter L2 output	0.81

The +0.05 gap over perplexity does not generalize across domains (AUROC drops to 0.40 on cross-domain code detection).

Architecture

KuraFormer inserts adapter modules between Transformer blocks. Each adapter:

1. Projects hidden states into an oscillator space (down-projection)
2. Runs Kuramoto synchronization dynamics on the unit sphere
3. Projects back and adds a gated residual connection (up-projection)

Transformer Block N -> KuraAdapter -> Transformer Block N+1
                       |-- Down-project to oscillator space
                       |-- Run T steps of Kuramoto dynamics
                       |-- Compute synchronization energy
                       +-- Up-project + gated residual

The base model stays completely frozen. Only adapter parameters are trained (~0.7-0.9% of total).

A StaticAdapter variant (no Kuramoto dynamics, same bottleneck) is included for ablation.

Quick Start

from transformers import AutoModelForCausalLM
from kuraformer import inject_kura_adapters

# Load any HuggingFace model
model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1")

# Inject KuraFormer adapters (base model stays frozen)
model = inject_kura_adapters(
    model,
    num_oscillators=64,
    num_steps=4,
    inject_every=4,
    warm_start=True,
    dt_schedule=True,
    damping_schedule=True,
)

# Or use a static adapter (no Kuramoto dynamics, same accuracy)
model = inject_kura_adapters(
    model,
    num_oscillators=64,
    inject_every=4,
    adapter_type="static",
)

See examples/quick_start.py for a complete runnable example.

Training

# Install with training dependencies
pip install -e ".[train]"

# Train KuraFormer on GSM8K
python scripts/train.py --config configs/llama3_8b_v2.yaml

# Train static adapter (ablation baseline)
python scripts/train.py --config configs/llama3_8b_v2.yaml --adapter_type static

# Evaluate
python scripts/evaluate_gsm8k.py \
    --model NousResearch/Meta-Llama-3-8B \
    --adapter_path outputs/llama3_8b_gsm8k_v2/kuraformer_final.pt \
    --step_values 4 8 16 32 64 \
    --warm_start --dt_schedule --damping_schedule

Installation

pip install git+https://github.com/seetrex-ai/kuraformer.git

Or from source:

git clone https://github.com/seetrex-ai/kuraformer.git
cd kuraformer
pip install -e .

Paper

KuraFormer: Oscillatory Dynamics as Parameter-Efficient Adapters for Pretrained Transformers Jesus Tabares Montilla, 2026. Zenodo (DOI: 10.5281/zenodo.19007695)

v2.0.0 includes the static adapter ablation and OOD control experiments documenting the negative results.

Citation

@article{tabares2026kuraformer,
    title={KuraFormer: Oscillatory Dynamics as Parameter-Efficient Adapters for Pretrained Transformers},
    author={Tabares Montilla, Jes{\'u}s},
    year={2026},
    doi={10.5281/zenodo.19007695},
    url={https://zenodo.org/records/19007695}
}

Contributing

See CONTRIBUTING.md for guidelines.

License

MIT. See LICENSE.

References

• AKOrN: Miyato et al., "Attentive Kuramoto Oscillatory Recurrent Networks", ICLR 2025
• CTM: Sakana AI, "Continuous Thought Machines", NeurIPS 2025
• Original Kuramoto model: Kuramoto (1975)