Skip to content

fix: map mmapped weights through Metal buffers instead of CPU buffers - #2037

Merged
leejet merged 2 commits into
leejet:masterfrom
nanguoyu:fix/metal-mmap-device-buffers
Sep 23, 2026
Merged

leejet merged 2 commits into
leejet:masterfrom
nanguoyu:fix/metal-mmap-device-buffers

Conversation

@nanguoyu

@nanguoyu nanguoyu commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Problem

With --mmap on Metal, every weight fails at graph execution and the output is noise:

ggml_metal_buffer_get_id: error: tensor 'text_encoders.llm.model.layers.0.self_attn.q_proj.weight' buffer is nil

can_mmap_storage() allows mmap when the compute backend reports buffer_from_host_ptr, which Metal does (unified memory), so params that live on the Metal device are mapped straight from the model file. But process_model_files() always wraps the mapping in a CPU buffer (ggml_backend_cpu_buffer_from_ptr). Metal cannot address a CPU buffer, and because the params backend is the compute backend, nothing stages these tensors either.

Change

  • ModelManager::mmap_params() groups the candidates by the device that computes on them: tensors a GPU computes on in place are mapped through that GPU's device, all others keep the CPU mapping.
  • ModelLoader::mmap_tensors() takes that device and maps the file through ggml_backend_dev_buffer_from_host_ptr(), created on first use and cached per file. If the device cannot wrap the file, its tensors are loaded as before.
  • load_tensors() skips tensors mapped through a device buffer, as it already does for CPU-mapped ones.

Only Metal changes behavior. Of the backends that report buffer_from_host_ptr, the others are CPU-side (BLAS wraps host memory in a CPU buffer anyway), and backends that cannot wrap host memory never mmap params they compute on.

Verification

Apple M1 Pro (32 GB), macOS 26, -DSD_METAL=ON -DGGML_METAL_EMBED_LIBRARY=ON, on master 2dc7f54:

sd-cli --diffusion-model qwen-image-2.1-Q4_K_M.gguf --llm Qwen3-VL-8B-Instruct-UD-Q4_K_XL.gguf \
    --vae qwen_image_2.1_vae_bf16.safetensors -p "a lovely cat holding a sign that says 'sd.cpp'" \
    --cfg-scale 6.0 --sampling-method euler --steps 20 -W 512 -H 512 --diffusion-fa -s 42 -v --mmap

(Qwen Image 2.1 Q4_K_M and Qwen3-VL-8B-Instruct UD-Q4_K_XL from unsloth, bf16 VAE from Comfy-Org.)

buffer is nil errors output peak memory footprint (/usr/bin/time -l)
master, without --mmap 0 reference 10.2 GB
master, --mmap 1846 (2-step run) noise
this PR, --mmap 0 pixel-identical to the reference 1.4 GB

With this PR the log shows mmap: mapped '...' for MTL0 for each GGUF file (the bf16 VAE is converted on load, so it is read as before). The mapped weights stay in the page cache as clean, file-backed pages instead of process memory.

Left to right: master without --mmap, master with --mmap, this PR with --mmap.

pr1-images

nanguoyu and others added 2 commits September 23, 2026 17:35
With mmap enabled on Metal, params that live on the Metal device are mapped
from the model file (the device can use host memory in place), but the
mapping is always wrapped in a CPU buffer. Metal cannot address a CPU buffer
and nothing stages these tensors, so graph execution logs
"ggml_metal_buffer_get_id: error: tensor ... buffer is nil" for every weight
and the output is garbage.

Map such tensors through a buffer the compute device creates from the mapped
memory (ggml_backend_dev_buffer_from_host_ptr), created on first use and
cached per file, and skip reading them in load_tensors like CPU-mapped
tensors. Writable mappings, used when LoRAs are merged in place, are not
wrapped; those params are loaded as before. CPU-side backends keep the CPU
mapping; backends that cannot wrap host memory never mmap params they compute
on, so only Metal changes.
@leejet
leejet merged commit 500ef5f into leejet:master Sep 23, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants