Text is not the native language of intelligence. It is the native language of keyboards.
A truly capable agent must think in the same modalities as the world: vision for appearance, audio for events, text for abstraction, proprioception for body state, and time for sequence. Multi-modality is not data fusion. It is learning a single representational geometry that makes sense across all senses.
The standard approach to multi-modal AI is to train separate encoders for each modality and fuse their outputs in a shared transformer. Vision tokens, audio tokens, and text tokens are concatenated and fed to a large model. This works but treats modalities as foreign languages that must be translated into text-like embeddings.
The SuperInstance approach is different. It begins with the observation that every modality can be reduced to a ternary signal field: local measurements that are either negative, neutral, or positive relative to some baseline.
- Vision: edges, motion, and salience maps are naturally signed.
- Audio: spectrograms can be represented as onset (+1), sustain (0), and offset (-1).
- Text: semantic embeddings can be binarized or ternarized without large accuracy loss.
- Proprioception: joint torques and IMU readings are signed by physics.
If every modality is a ternary signal field, then multi-modal fusion becomes a single operation: overlay the fields and look for correlated structure. This is the ternary multi-modal tensor:
M[x, y, t, c] ∈ {-1, 0, +1}
where x, y are spatial coordinates, t is time, and c is the channel (modality). Correlations across channels indicate cross-modal events: a flash of light and a loud sound at the same (x, y, t) suggest an explosion, not two independent events.
In a ternary multi-modal transformer, attention weights are also ternary:
+1: strong cross-modal alignment.0: no relationship.-1: strong misalignment or inhibition.
This makes attention interpretable. You can query "which visual regions attended to this word?" by looking for +1 entries in the vision-text attention slice. You can query "which sounds suppressed this visual detection?" by looking for -1 entries.
Different modalities have different compute requirements:
- Vision: high spatial resolution, moderate temporal resolution, tensor-core-friendly convolutions.
- Audio: high temporal resolution, low spatial resolution, recurrent or streaming attention.
- Text: low per-token compute but long context, memory-bandwidth-bound attention.
- Proprioception: tiny tensors but hard real-time deadlines.
The SuperInstance fleet handles this heterogeneity naturally. oxide-fleet assigns each modality's compute graph to agents with the right capabilities. A vision encoder runs on an agent with tensor cores. A proprioception loop runs on an ESP32 microcontroller. The fused representation is computed where the modalities meet.
- Ternary vision embeddings: Encode 10,000 ImageNet images into ternary feature vectors and measure k-NN classification accuracy versus FP32 baseline. Expected: 5-15% accuracy drop, 10-20× compression.
- Cross-modal retrieval: Given an audio clip, retrieve the matching video clip from a ternary multi-modal index. Expected: ternary indexing achieves >80% recall@10 with 16× smaller index.
- Real-time fusion latency: Run vision at 30Hz, audio at 100Hz, and proprioception at 1kHz. Measure end-to-end fusion latency. Expected: bottleneck is the slowest modality (vision) plus one frame of buffering.
- Attention interpretability: Visualize ternary attention maps for vision-language tasks and verify that
+1attention regions correspond to human-judged relevant image regions. Expected: >70% overlap.
- Robotic perception: Fuse camera, lidar, microphone, and tactile data into a single scene representation.
- Assistive technology: Agents that understand both spoken commands and visual context for users with disabilities.
- Video understanding: Long-form video analysis where audio cues, visual motion, and subtitle text must be aligned.
- Industrial inspection: Detect defects by fusing visual, thermal, and acoustic signatures.
- Immersive interfaces: Virtual companions that perceive gesture, gaze, voice, and emotion simultaneously.
- Modality imbalance: Vision tensors are enormous compared to text tokens. How do we prevent vision from dominating compute and attention?
- Temporal alignment: Audio and video have different sampling rates. What is the right time representation for fusion?
- Missing modalities: How should an agent behave when one modality is unavailable — should it impute, ignore, or explicitly report uncertainty?
- Ternary fidelity: Do ternary representations sacrifice too much fine-grained information for high-level cross-modal tasks?
- GPU-AS-MOTOR-CORTEX.md — The compute substrate for vision and audio encoders.
- EMBODIED-AI.md — Why multi-modal perception requires a body.
- THE-PACKED-FORMAT.md — Storing multi-modal ternary tensors compactly.
- TERNARY-QUANTIZATION.md — Quantizing encoders to {-1, 0, +1}.
- SPARSITY-IN-TERNARY-SYSTEMS.md — Multi-modal tensors are naturally sparse.
oxide-fleet— Assigns modality-specific work to capable agents.oxide-constructs— Stores multi-modal encoder constructs.
Pick two modalities you have access to — for example, text and images, or audio and accelerometer data. Encode 100 samples from each modality into ternary vectors of the same length using simple sign-thresholding. Build a nearest-neighbor index where cross-modal similarity is the dot product of ternary vectors. Test three queries: retrieve text given an image, retrieve image given text, and detect mismatched pairs. Measure precision@5 and index size. If precision is below 60%, experiment with learned projection layers before ternarization.