You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
pie 0.4.0: release pipeline, portability fix, and 0.4.0 docs
Per-compute-capability CUDA release matrix plus portable Vulkan / macOS Metal /
Windows binaries: node24 CI, manylinux_2_28 (glibc 2.28), self-hosted build
removed, standardized artifact names. Rustls/OpenSSL portability fix so the
release binaries no longer dynamically link libssl.so.1.1. Per-CC
auto-detecting installer (driver + compute capability).
Full 0.4.0 website/docs refresh: multimodal guide + a Forward Pass > Media page,
drivers (TensorRT-LLM surfaced in sidebar + card; the removed `dev` driver
dropped), models (GLM-5.1 / per-CC CUDA), multimodal in the SDK reference,
concise CHANGELOG, and the v0.5.0 roadmap.
Validated end-to-end on an L40S fleet (cuda12.8 and cuda13.0): documented
install, inferlet generation, and perf vs vLLM.
- GGUF portable driver build failure; first-shard validation in
116
-
`open_sharded`.
117
-
- Serialized forked-branch generation; CUDA quantized generation paths.
118
-
- GPU-local CPU-affinity parsing in the benchmarks (NIC columns in the
119
-
`nvidia-smi` topology), now also applied to `pie_bench`.
9
+
-**Multimodal**: vision/video/audio input and audio output via a model-agnostic media API (`append_image`/`append_audio`/`append_video`; TTS via `model.speak`).
description: Splicing image, video, and audio spans into a forward pass
4
+
sidebar_position: 3
5
+
---
6
+
7
+
# Media (multimodal)
8
+
9
+
A forward pass accepts **encoded media spans** (images, video frames, audio clips) alongside ordinary tokens. The inferlet hands the host raw encoded bytes; the driver decodes them, runs the model's vision/audio encoder, and **scatters the projected embedding rows into the hidden state** at the positions you choose. Surrounding text attends to the span through the normal causal mask, so generation after a media span is ordinary text decoding. The API is **model-agnostic**: no model constants leak into inferlet code.
10
+
11
+
Two layers sit over this, mirroring tokens vs. `Forward`:
12
+
13
+
-**`Context::append_*`**: the high-level wrapper that splices a span at the context's current position and advances the cursor for you. Most inferlets use this; see [Multimodal generation](../model/multimodal) for the full guide and the supported-model matrix.
14
+
-**`Forward::input_*`**: the per-pass builder, covered here. You pick the anchor position. Reach for it when you build passes directly: custom masks, scoring, or a hand-written decoder.
15
+
16
+
## Encoding media
17
+
18
+
Fetch the bytes and build a media handle. Decoding is synchronous and exposes layout metadata *before* any forward pass:
image.token_count(); // soft tokens the span occupies in the sequence
27
+
image.grid(); // (t, h, w) merged-token layout
28
+
image.position_span(); // sequence positions to advance past the span
29
+
```
30
+
31
+
`Audio::from_bytes(&model, &bytes)` and `Video::from_bytes(&model, &bytes, max_frames)` follow the same shape. For video, `max_frames` is your KV-budget knob: the host demuxes and uniformly samples up to that many frames, preprocessing each as an image span.
32
+
33
+
## Splicing into a forward pass
34
+
35
+
`Forward::input_image` / `input_audio` attach an encoded span at sequence position `anchor`. At `execute()`, the driver runs the encoder and overwrites `hidden[anchor .. anchor + token_count]` with the projected rows. Advance your position cursor by `position_span()` for whatever follows.
36
+
37
+
| Method | What it does |
38
+
|---|---|
39
+
|`fwd.input_image(&image, anchor)`| Splice a visual span (image or video frame) at `anchor`. |
40
+
|`fwd.input_audio(&audio, anchor)`| Splice an audio clip at `anchor`. |
41
+
42
+
```rust
43
+
letmutfwd=ctx.forward();
44
+
letanchor=fwd.start_position(); // first free sequence position
45
+
fwd.input_image(&image, anchor); // driver encodes + scatters rows at execute()
46
+
letout=fwd.execute().await?;
47
+
```
48
+
49
+
Spans are attached by reference and emitted in declaration order, so several can be spliced into one pass. The positions a span occupies are reserved like any other input; `Context::append_image` (below) does that bookkeeping for you, and at the raw level you manage pages through `ctx.inner()`.
50
+
51
+
## The high-level wrapper
52
+
53
+
`Context::append_image` is exactly `input_image` at the context's own position, plus the bookkeeping to reserve the span's pages and advance the cursor:
Use these unless you specifically need to control the anchor.
62
+
63
+
## Audio output
64
+
65
+
Audio **output** (text-to-speech) is a separate generation path via `model.speak(text).speaker(id).generate()`, not a forward-pass input. See [Multimodal generation](../model/multimodal#audio-output-text-to-speech).
0 commit comments