ramalama does not pass through GPU when run in a toolbox(1), though llama.cpp does

[ibotty]

Issue Description

With

ramalama --debug --engine podman serve hf://bartowski/Qwen_Qwen3.6-27B-GGUF:Q6_K_L --image=quay.io/ramalama/rocm --runtime-args="--spec-default" --privileged  --backend=vulkan --device /dev/dri --device /dev/kfd

a podman exec -itl ./llama-server --list-devices shows no available devices.

On the host (in a toolbox) I can see devices though.

> llama-server --list-devices
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon 780M Graphics, gfx1103 (0x1103), VMM: no, Wave Size: 32
Available devices:
  ROCm0: AMD Radeon 780M Graphics (31038 MiB, 31038 MiB free)

Steps to reproduce the issue

s.a.

Describe the results you received

s.a.

Describe the results you expected

s.a.

ramalama info output

{
    "Accelerator": "hip",
    "Config": {
        "benchmarks": {},
        "container": true,
        "engine": "podman",
        "provider": {
            "openai": {}
        },
        "runtimes": {
            "llama_cpp": {},
            "mlx": {}
        },
        "settings": {
            "config_files": [
                "/usr/share/ramalama/ramalama.conf",
                "/var/home/tob/.config/ramalama/ramalama.conf"
            ]
        },
        "user": {}
    },
    "Engine": {
        "Info": {
            "host": {
                "arch": "amd64",
                "buildahVersion": "1.43.1",
                "cgroupControllers": [
                    "cpu",
                    "io",
                    "memory",
                    "pids"
                ],
                "cgroupManager": "systemd",
                "cgroupVersion": "v2",
                "conmon": {
                    "package": "conmon-2.2.1-2.fc44.x86_64",
                    "path": "/usr/bin/conmon",
                    "version": "conmon version 2.2.1, commit: "
                },
                "cpuUtilization": {
                    "idlePercent": 95.08,
                    "systemPercent": 1.33,
                    "userPercent": 3.58
                },
                "cpus": 16,
                "databaseBackend": "sqlite",
                "distribution": {
                    "distribution": "fedora",
                    "variant": "sway-atomic",
                    "version": "44"
                },
                "emulatedArchitectures": [
                    "linux/arm64",
                    "linux/arm64be"
                ],
                "eventLogger": "journald",
                "freeLocks": 2044,
                "hostname": "framework",
                "idMappings": {
                    "gidmap": [
                        {
                            "container_id": 0,
                            "host_id": 1000,
                            "size": 1
                        },
                        {
                            "container_id": 1,
                            "host_id": 100000,
                            "size": 65536
                        }
                    ],
                    "uidmap": [
                        {
                            "container_id": 0,
                            "host_id": 1000,
                            "size": 1
                        },
                        {
                            "container_id": 1,
                            "host_id": 100000,
                            "size": 65536
                        }
                    ]
                },
                "kernel": "7.0.12-200.fc44.x86_64",
                "linkmode": "dynamic",
                "logDriver": "journald",
                "memFree": 13176299520,
                "memTotal": 65092734976,
                "networkBackend": "netavark",
                "networkBackendInfo": {
                    "backend": "netavark",
                    "defaultNetwork": "podman",
                    "dns": {
                        "package": "aardvark-dns-1.17.1-1.fc44.x86_64",
                        "path": "/usr/libexec/podman/aardvark-dns",
                        "version": "aardvark-dns 1.17.1"
                    },
                    "package": "netavark-1.17.2-1.fc44.x86_64",
                    "path": "/usr/libexec/podman/netavark",
                    "version": "netavark 1.17.2"
                },
                "ociRuntime": {
                    "name": "crun",
                    "package": "crun-1.28-1.fc44.x86_64",
                    "path": "/usr/bin/crun",
                    "version": "crun version 1.28\ncommit: 54f16ffbefcd022bf032af768b5c5ce075c18bfc\nrundir: /run/user/1000/crun\nspec: 1.0.0\n+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +JSON_C"
                },
                "os": "linux",
                "pasta": {
                    "executable": "/usr/bin/pasta",
                    "package": "passt-0^20260526.g038c51e-1.fc44.x86_64",
                    "version": "pasta 0^20260526.g038c51e-1.fc44.x86_64\nCopyright Red Hat\nGNU General Public License, version 2 or later\n  <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>\nThis is free software: you are free to change and redistribute it.\nThere is NO WARRANTY, to the extent permitted by law.\n"
                },
                "remoteSocket": {
                    "exists": true,
                    "path": "/run/user/1000/podman/podman.sock"
                },
                "rootlessNetworkCmd": "pasta",
                "security": {
                    "apparmorEnabled": false,
                    "capabilities": "CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT",
                    "rootless": true,
                    "seccompEnabled": true,
                    "seccompProfilePath": "/usr/share/containers/seccomp.json",
                    "selinuxEnabled": true
                },
                "serviceIsRemote": false,
                "slirp4netns": {
                    "executable": "/usr/bin/slirp4netns",
                    "package": "slirp4netns-1.3.1-4.fc44.x86_64",
                    "version": "slirp4netns version 1.3.1\ncommit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236\nlibslirp: 4.9.1\nSLIRP_CONFIG_VERSION_MAX: 6\nlibseccomp: 2.6.0"
                },
                "swapFree": 2578432000,
                "swapTotal": 8589930496,
                "uptime": "76h 55m 7.00s (Approximately 3.17 days)",
                "variant": ""
            },
            "plugins": {
                "authorization": null,
                "log": [
                    "k8s-file",
                    "none",
                    "passthrough",
                    "journald"
                ],
                "network": [
                    "bridge",
                    "macvlan",
                    "ipvlan"
                ],
                "volume": [
                    "local"
                ]
            },
            "registries": {
                "search": [
                    "registry.fedoraproject.org",
                    "registry.access.redhat.com",
                    "docker.io"
                ]
            },
            "store": {
                "configFile": "/var/home/tob/.config/containers/storage.conf",
                "containerStore": {
                    "number": 2,
                    "paused": 0,
                    "running": 2,
                    "stopped": 0
                },
                "graphDriverName": "overlay",
                "graphOptions": {},
                "graphRoot": "/var/home/tob/.local/share/containers/storage",
                "graphRootAllocated": 998483427328,
                "graphRootUsed": 705889763328,
                "graphStatus": {
                    "Backing Filesystem": "btrfs",
                    "Native Overlay Diff": "true",
                    "Supports d_type": "true",
                    "Supports shifting": "false",
                    "Supports volatile": "true",
                    "Using metacopy": "false"
                },
                "imageCopyTmpDir": "/var/tmp",
                "imageStore": {
                    "number": 4
                },
                "runRoot": "/run/user/1000/containers",
                "transientStore": false,
                "volumePath": "/var/home/tob/.local/share/containers/storage/volumes"
            },
            "version": {
                "APIVersion": "5.8.2",
                "BuildOrigin": "Fedora Project",
                "Built": 1776124800,
                "BuiltTime": "Tue Apr 14 02:00:00 2026",
                "GitCommit": "5b263b5f5b48004a87caac44e67349a8266d9ef4",
                "GoVersion": "go1.26.1-X:nodwarf5",
                "Os": "linux",
                "OsArch": "linux/amd64",
                "Version": "5.8.2"
            }
        },
        "Name": "podman"
    },
    "Image": "quay.io/ramalama/ramalama:0.21",
    "RagImage": "quay.io/ramalama/ramalama-rag:0.21",
    "Runtimes": {
        "Available": [
            "llama.cpp",
            "mlx",
            "vllm"
        ],
        "Default": "llama.cpp"
    },
    "Selinux": false,
    "Shortnames": {
        "Files": [
            "/usr/share/ramalama/shortnames.conf"
        ],
        "Names": {
            "cerebrum": "hf://froggeric/Cerebrum-1.0-7b-GGUF",
            "deepseek": "hf://bartowski/DeepSeek-R1-Distill-Qwen-7B-GGUF",
            "dragon": "hf://llmware/dragon-mistral-7b-v0",
            "gemma3": "hf://ggml-org/gemma-3-4b-it-GGUF",
            "gemma3:12b": "hf://ggml-org/gemma-3-12b-it-GGUF",
            "gemma3:1b": "hf://ggml-org/gemma-3-1b-it-GGUF",
            "gemma3:27b": "hf://ggml-org/gemma-3-27b-it-GGUF",
            "gemma3:4b": "hf://ggml-org/gemma-3-4b-it-GGUF",
            "gemma3n": "hf://ggml-org/gemma-3n-E4B-it-GGUF",
            "gemma3n:e2b": "hf://ggml-org/gemma-3n-E2B-it-GGUF",
            "gemma3n:e2b-it-f16": "hf://ggml-org/gemma-3n-E2B-it-GGUF:F16",
            "gemma3n:e2b-it-q8_0": "hf://ggml-org/gemma-3n-E2B-it-GGUF:Q8_0",
            "gemma3n:e4b": "hf://ggml-org/gemma-3n-E4B-it-GGUF",
            "gemma3n:e4b-it-f16": "hf://ggml-org/gemma-3n-E4B-it-GGUF:F16",
            "gemma3n:e4b-it-q8_0": "hf://ggml-org/gemma-3n-E4B-it-GGUF:Q8_0",
            "gpt-5.1": "openai://gpt-5.1-2025-11-13",
            "gpt-oss": "hf://ggml-org/gpt-oss-20b-GGUF",
            "gpt-oss:120b": "hf://ggml-org/gpt-oss-120b-GGUF",
            "gpt-oss:20b": "hf://ggml-org/gpt-oss-20b-GGUF",
            "granite": "hf://ibm-granite/granite-3.3-8b-instruct-GGUF",
            "granite-be-3.0:1b": "hf://taronaeo/Granite-3.0-1B-A400M-Instruct-BE-GGUF",
            "granite-be-3.3:2b": "hf://taronaeo/Granite-3.3-2B-Instruct-BE-GGUF",
            "granite-lab-7b": "hf://instructlab/granite-7b-lab-GGUF",
            "granite-lab-8b": "hf://ibm-granite/granite-3.3-8b-instruct-GGUF",
            "granite-lab:7b": "hf://instructlab/granite-7b-lab-GGUF",
            "granite:2b": "hf://bartowski/ibm-granite_granite-3.3-2b-instruct-GGUF",
            "granite:7b": "hf://instructlab/granite-7b-lab-GGUF",
            "granite:8b": "hf://ibm-granite/granite-3.3-8b-instruct-GGUF",
            "hermes": "hf://NousResearch/Hermes-2-Pro-Mistral-7B-GGUF",
            "ibm/granite": "hf://ibm-granite/granite-3.3-8b-instruct-GGUF",
            "ibm/granite:2b": "hf://bartowski/ibm-granite_granite-3.3-2b-instruct-GGUF",
            "ibm/granite:7b": "hf://instructlab/granite-7b-lab-GGUF",
            "ibm/granite:8b": "hf://ibm-granite/granite-3.3-8b-instruct-GGUF",
            "merlinite": "hf://instructlab/merlinite-7b-lab-GGUF",
            "merlinite-lab-7b": "hf://instructlab/merlinite-7b-lab-GGUF",
            "merlinite-lab:7b": "hf://instructlab/merlinite-7b-lab-GGUF",
            "merlinite:7b": "hf://instructlab/merlinite-7b-lab-GGUF",
            "mistral": "hf://lmstudio-community/Mistral-7B-Instruct-v0.3-GGUF",
            "mistral-small3.1": "hf://bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF",
            "mistral-small3.1:24b": "hf://bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF",
            "mistral:7b": "hf://lmstudio-community/Mistral-7B-Instruct-v0.3-GGUF",
            "mistral:7b-v1": "hf://TheBloke/Mistral-7B-Instruct-v0.1-GGUF",
            "mistral:7b-v2": "hf://TheBloke/Mistral-7B-Instruct-v0.2-GGUF",
            "mistral:7b-v3": "hf://lmstudio-community/Mistral-7B-Instruct-v0.3-GGUF",
            "mistral_code_16k": "hf://TheBloke/Mistral-7B-Code-16K-qlora-GGUF",
            "mistral_codealpaca": "hf://TheBloke/Mistral-7B-codealpaca-lora-GGUF",
            "mixtao": "hf://MaziyarPanahi/MixTAO-7Bx2-MoE-Instruct-v7.0-GGUF",
            "openchat": "hf://TheBloke/openchat-3.5-0106-GGUF",
            "openorca": "hf://TheBloke/Mistral-7B-OpenOrca-GGUF",
            "phi2": "hf://MaziyarPanahi/phi-2-GGUF",
            "qwen2.5vl": "hf://ggml-org/Qwen2.5-VL-32B-Instruct-GGUF",
            "qwen2.5vl:2b": "hf://ggml-org/Qwen2.5-VL-2B-Instruct-GGUF",
            "qwen2.5vl:32b": "hf://ggml-org/Qwen2.5-VL-32B-Instruct-GGUF",
            "qwen2.5vl:3b": "hf://ggml-org/Qwen2.5-VL-3B-Instruct-GGUF",
            "qwen2.5vl:7b": "hf://ggml-org/Qwen2.5-VL-7B-Instruct-GGUF",
            "qwen3.5:0.8b": "hf://bartowski/Qwen_Qwen3.5-0.8B-GGUF",
            "qwen3.5:27b": "hf://bartowski/Qwen_Qwen3.5-27B-GGUF",
            "qwen3.5:2b": "hf://bartowski/Qwen_Qwen3.5-2B-GGUF",
            "qwen3.5:4b": "hf://bartowski/Qwen_Qwen3.5-4B-GGUF",
            "qwen3.5:9b": "hf://bartowski/Qwen_Qwen3.5-9B-GGUF",
            "qwen3:0.6b": "hf://Qwen/Qwen3-0.6B-GGUF",
            "qwen3:1.7b": "hf://Qwen/Qwen3-1.7B-GGUF",
            "qwen3:32b": "hf://Qwen/Qwen3-32B-GGUF",
            "qwen3:4b": "hf://Qwen/Qwen3-4B-GGUF",
            "qwen3:8b": "hf://Qwen/Qwen3-8B-GGUF",
            "smollm:135m": "hf://HuggingFaceTB/smollm-135M-instruct-v0.2-Q8_0-GGUF",
            "smolvlm": "hf://ggml-org/SmolVLM-500M-Instruct-GGUF",
            "smolvlm:256m": "hf://ggml-org/SmolVLM-256M-Instruct-GGUF",
            "smolvlm:2b": "hf://ggml-org/SmolVLM-Instruct-GGUF",
            "smolvlm:500m": "hf://ggml-org/SmolVLM-500M-Instruct-GGUF",
            "stories-be:260k": "hf://taronaeo/tinyllamas-BE/stories260K-be.gguf",
            "tiny": "hf://TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF",
            "tinyllama": "hf://TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF"
        },
        "Sources": [
            "/usr/share/ramalama/shortnames.conf"
        ]
    },
    "Store": "/var/home/tob/.local/share/ramalama",
    "ToolsImage": "quay.io/ramalama/rocm-tools:0.21",
    "UseContainer": true,
    "Version": "0.21.0"
}

Upstream Latest Release

No

Additional environment details

No response

Additional information

No response

[olliewalsh]

@ibotty you mention running in a toolbox, is this an immutable OS e.g silverblue?

[ibotty]

Yes this is on Sway Atomic, similar to Silverblue.

The container gets started "on the host", I assume with the following podman wrapper (because podman in the container does not work in my setup).

flatpak-spawn --host /usr/bin/podman "$@"

I'll include podman inspect -l below.

[
     {
          "Id": "85a0570a2a9dba17604b2326af52682be1024deff047248f4e95a68b8f11b379",
          "Created": "2026-06-19T08:58:05.095513126+02:00",
          "Path": "/run/podman-init",
          "Args": [
               "--",
               "/app/tools.sh",
               "--server",
               "--host",
               "::",
               "--port",
               "8080",
               "--model",
               "/mnt/models/Qwen_Qwen3.6-27B-Q6_K_L.gguf",
               "--mmproj",
               "/mnt/models/mmproj-Qwen_Qwen3.6-27B-bf16.gguf",
               "--chat-template-file",
               "/mnt/models/chat_template_extracted",
               "--no-warmup",
               "--alias",
               "bartowski/Qwen_Qwen3.6-27B-GGUF",
               "--temp",
               "0.8",
               "--threads",
               "8",
               "--log-colors",
               "on",
               "--spec-default"
          ],
          "State": {
               "OciVersion": "1.2.1",
               "Status": "running",
               "Running": true,
               "Paused": false,
               "Restarting": false,
               "OOMKilled": false,
               "Dead": false,
               "Pid": 62853,
               "ConmonPid": 62851,
               "ExitCode": 0,
               "Error": "",
               "StartedAt": "2026-06-19T08:58:05.173340122+02:00",
               "FinishedAt": "0001-01-01T00:00:00Z",
               "CgroupPath": "/user.slice/user-1000.slice/user@1000.service/user.slice/libpod-85a0570a2a9dba17604b2326af52682be1024deff047248f4e95a68b8f11b379.scope",
               "CheckpointedAt": "0001-01-01T00:00:00Z",
               "RestoredAt": "0001-01-01T00:00:00Z"
          },
          "Image": "4173f6384071c80c009d47b5ebeb1ac522484ed155f3c2eab1c042e499b39387",
          "ImageDigest": "sha256:ce4bd1b653551e400028ecc270c0e975996db3b319b4ee00cac700306c214206",
          "ImageName": "ghcr.io/ggml-org/llama.cpp:full-vulkan",
          "Rootfs": "",
          "Pod": "",
          "ResolvConfPath": "/run/user/1000/containers/overlay-containers/85a0570a2a9dba17604b2326af52682be1024deff047248f4e95a68b8f11b379/userdata/resolv.conf",
          "HostnamePath": "/run/user/1000/containers/overlay-containers/85a0570a2a9dba17604b2326af52682be1024deff047248f4e95a68b8f11b379/userdata/hostname",
          "HostsPath": "/run/user/1000/containers/overlay-containers/85a0570a2a9dba17604b2326af52682be1024deff047248f4e95a68b8f11b379/userdata/hosts",
          "StaticDir": "/var/home/tob/.local/share/containers/storage/overlay-containers/85a0570a2a9dba17604b2326af52682be1024deff047248f4e95a68b8f11b379/userdata",
          "OCIConfigPath": "/var/home/tob/.local/share/containers/storage/overlay-containers/85a0570a2a9dba17604b2326af52682be1024deff047248f4e95a68b8f11b379/userdata/config.json",
          "OCIRuntime": "crun",
          "ConmonPidFile": "/run/user/1000/containers/overlay-containers/85a0570a2a9dba17604b2326af52682be1024deff047248f4e95a68b8f11b379/userdata/conmon.pid",
          "PidFile": "/run/user/1000/containers/overlay-containers/85a0570a2a9dba17604b2326af52682be1024deff047248f4e95a68b8f11b379/userdata/pidfile",
          "Name": "ramalama-x6yqjYBx19",
          "RestartCount": 0,
          "Driver": "overlay",
          "MountLabel": "system_u:object_r:container_file_t:s0:c1022,c1023",
          "ProcessLabel": "",
          "AppArmorProfile": "",
          "EffectiveCaps": [
               "CAP_AUDIT_CONTROL",
               "CAP_AUDIT_READ",
               "CAP_AUDIT_WRITE",
               "CAP_BLOCK_SUSPEND",
               "CAP_BPF",
               "CAP_CHECKPOINT_RESTORE",
               "CAP_CHOWN",
               "CAP_DAC_OVERRIDE",
               "CAP_DAC_READ_SEARCH",
               "CAP_FOWNER",
               "CAP_FSETID",
               "CAP_IPC_LOCK",
               "CAP_IPC_OWNER",
               "CAP_KILL",
               "CAP_LEASE",
               "CAP_LINUX_IMMUTABLE",
               "CAP_MAC_ADMIN",
               "CAP_MAC_OVERRIDE",
               "CAP_MKNOD",
               "CAP_NET_ADMIN",
               "CAP_NET_BIND_SERVICE",
               "CAP_NET_BROADCAST",
               "CAP_NET_RAW",
               "CAP_PERFMON",
               "CAP_SETFCAP",
               "CAP_SETGID",
               "CAP_SETPCAP",
               "CAP_SETUID",
               "CAP_SYSLOG",
               "CAP_SYS_ADMIN",
               "CAP_SYS_BOOT",
               "CAP_SYS_CHROOT",
               "CAP_SYS_MODULE",
               "CAP_SYS_NICE",
               "CAP_SYS_PACCT",
               "CAP_SYS_PTRACE",
               "CAP_SYS_RAWIO",
               "CAP_SYS_RESOURCE",
               "CAP_SYS_TIME",
               "CAP_SYS_TTY_CONFIG",
               "CAP_WAKE_ALARM"
          ],
          "BoundingCaps": [
               "CAP_AUDIT_CONTROL",
               "CAP_AUDIT_READ",
               "CAP_AUDIT_WRITE",
               "CAP_BLOCK_SUSPEND",
               "CAP_BPF",
               "CAP_CHECKPOINT_RESTORE",
               "CAP_CHOWN",
               "CAP_DAC_OVERRIDE",
               "CAP_DAC_READ_SEARCH",
               "CAP_FOWNER",
               "CAP_FSETID",
               "CAP_IPC_LOCK",
               "CAP_IPC_OWNER",
               "CAP_KILL",
               "CAP_LEASE",
               "CAP_LINUX_IMMUTABLE",
               "CAP_MAC_ADMIN",
               "CAP_MAC_OVERRIDE",
               "CAP_MKNOD",
               "CAP_NET_ADMIN",
               "CAP_NET_BIND_SERVICE",
               "CAP_NET_BROADCAST",
               "CAP_NET_RAW",
               "CAP_PERFMON",
               "CAP_SETFCAP",
               "CAP_SETGID",
               "CAP_SETPCAP",
               "CAP_SETUID",
               "CAP_SYSLOG",
               "CAP_SYS_ADMIN",
               "CAP_SYS_BOOT",
               "CAP_SYS_CHROOT",
               "CAP_SYS_MODULE",
               "CAP_SYS_NICE",
               "CAP_SYS_PACCT",
               "CAP_SYS_PTRACE",
               "CAP_SYS_RAWIO",
               "CAP_SYS_RESOURCE",
               "CAP_SYS_TIME",
               "CAP_SYS_TTY_CONFIG",
               "CAP_WAKE_ALARM"
          ],
          "ExecIDs": [],
          "GraphDriver": {
               "Name": "overlay",
               "Data": {
                    "LowerDir": "/var/home/tob/.local/share/containers/storage/overlay/13fa534d7aca787271a473abbf9d03b3215ea1ce882e9ab8eb6d416609d7cedb/diff:/var/home/tob/.local/share/containers/storage/overlay/c6db7a7aa4e823949be44b634d6e188d6f7b61641350aec12f47052ef7b9b464/diff:/var/home/tob/.local/share/containers/storage/overlay/2e22a894a70723c54adade95aae6af10daa536a635f63a4b93e53928db925876/diff:/var/home/tob/.local/share/containers/storage/overlay/49ee3473076782e5cfa58992666af9a3eea32ebf5cdf5daf8511525b04e96af7/diff:/var/home/tob/.local/share/containers/storage/overlay/d3394df0b492a37acdc6516ba68146e86d04cd3178eb594232310285d19b53eb/diff:/var/home/tob/.local/share/containers/storage/overlay/d1c61b2f77d0296deefdf1967b620fa09d2fe193daf75df0fb40a2d43cefd01b/diff:/var/home/tob/.local/share/containers/storage/overlay/0c3db79307ab91dad11fad2b136a2b56df6efeeb567c4c99e6e316b63885c9f6/diff",
                    "MergedDir": "/var/home/tob/.local/share/containers/storage/overlay/04f23a862d9d36ab56686e7ae1c220b1b04954875b787000777015d271dde4dc/merged",
                    "UpperDir": "/var/home/tob/.local/share/containers/storage/overlay/04f23a862d9d36ab56686e7ae1c220b1b04954875b787000777015d271dde4dc/diff",
                    "WorkDir": "/var/home/tob/.local/share/containers/storage/overlay/04f23a862d9d36ab56686e7ae1c220b1b04954875b787000777015d271dde4dc/work"
               }
          },
          "Mounts": [
               {
                    "Type": "bind",
                    "Source": "/var/home/tob/.local/share/ramalama/store/huggingface/bartowski/Qwen_Qwen3.6-27B-GGUF/blobs/sha256-c5c8c41da6d155a61edd21b7e3aa50b6ef77f122d502d36afe4a4d5c3e494d4f",
                    "Destination": "/mnt/models/mmproj-Qwen_Qwen3.6-27B-bf16.gguf",
                    "Driver": "",
                    "Mode": "",
                    "Options": [
                         "rbind"
                    ],
                    "RW": false,
                    "Propagation": "rprivate"
               },
               {
                    "Type": "bind",
                    "Source": "/var/home/tob/.local/share/ramalama/store/huggingface/bartowski/Qwen_Qwen3.6-27B-GGUF/blobs/sha256-e84f32a23fdda27689f868aa4a1a5621f41133e51a48d7f3efcbea2839574259",
                    "Destination": "/mnt/models/chat_template_extracted",
                    "Driver": "",
                    "Mode": "",
                    "Options": [
                         "rbind"
                    ],
                    "RW": false,
                    "Propagation": "rprivate"
               },
               {
                    "Type": "bind",
                    "Source": "/var/home/tob/.local/share/ramalama/store/huggingface/bartowski/Qwen_Qwen3.6-27B-GGUF/blobs/sha256-95c61f9a30b0e0ada59169177fde2234e6bfd0e477a45cbff52097763286f9f6",
                    "Destination": "/mnt/models/Qwen_Qwen3.6-27B-Q6_K_L.gguf",
                    "Driver": "",
                    "Mode": "",
                    "Options": [
                         "rbind"
                    ],
                    "RW": false,
                    "Propagation": "rprivate"
               }
          ],
          "Dependencies": [],
          "NetworkSettings": {
               "EndpointID": "",
               "Gateway": "",
               "IPAddress": "",
               "IPPrefixLen": 0,
               "IPv6Gateway": "",
               "GlobalIPv6Address": "",
               "GlobalIPv6PrefixLen": 0,
               "MacAddress": "",
               "Bridge": "",
               "SandboxID": "",
               "HairpinMode": false,
               "LinkLocalIPv6Address": "",
               "LinkLocalIPv6PrefixLen": 0,
               "Ports": {
                    "8080/tcp": [
                         {
                              "HostIp": "0.0.0.0",
                              "HostPort": "8080"
                         }
                    ]
               },
               "SandboxKey": "/run/user/1000/netns/netns-f17590b4-52ac-35a0-2645-d56310097e5c"
          },
          "Namespace": "",
          "IsInfra": false,
          "IsService": false,
          "KubeExitCodePropagation": "invalid",
          "lockNumber": 1,
          "Config": {
               "Hostname": "85a0570a2a9d",
               "Domainname": "",
               "User": "",
               "AttachStdin": false,
               "AttachStdout": false,
               "AttachStderr": false,
               "Tty": true,
               "OpenStdin": true,
               "StdinOnce": false,
               "Env": [
                    "PATH=/root/.venv/bin:/root/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                    "container=podman",
                    "TERM=xterm",
                    "HIP_VISIBLE_DEVICES=0",
                    "HOME=/tmp",
                    "HOSTNAME=85a0570a2a9d"
               ],
               "Cmd": [
                    "--server",
                    "--host",
                    "::",
                    "--port",
                    "8080",
                    "--model",
                    "/mnt/models/Qwen_Qwen3.6-27B-Q6_K_L.gguf",
                    "--mmproj",
                    "/mnt/models/mmproj-Qwen_Qwen3.6-27B-bf16.gguf",
                    "--chat-template-file",
                    "/mnt/models/chat_template_extracted",
                    "--no-warmup",
                    "--alias",
                    "bartowski/Qwen_Qwen3.6-27B-GGUF",
                    "--temp",
                    "0.8",
                    "--threads",
                    "8",
                    "--log-colors",
                    "on",
                    "--spec-default"
               ],
               "Image": "ghcr.io/ggml-org/llama.cpp:full-vulkan",
               "Volumes": null,
               "WorkingDir": "/app",
               "Entrypoint": [
                    "/app/tools.sh"
               ],
               "OnBuild": null,
               "Labels": {
                    "ai.ramalama": "",
                    "ai.ramalama.command": "run",
                    "ai.ramalama.engine": "podman",
                    "ai.ramalama.model": "hf://bartowski/Qwen_Qwen3.6-27B-GGUF:Q6_K_L",
                    "ai.ramalama.port": "8080",
                    "ai.ramalama.runtime": "llama.cpp",
                    "org.opencontainers.image.created": "2026-06-18T05:50:06Z",
                    "org.opencontainers.image.description": "LLM inference in C/C++",
                    "org.opencontainers.image.revision": "f3e182816421c648188b5eab269853bf1531d950",
                    "org.opencontainers.image.source": "https://github.com/ggml-org/llama.cpp",
                    "org.opencontainers.image.title": "llama.cpp",
                    "org.opencontainers.image.url": "https://github.com/ggml-org/llama.cpp",
                    "org.opencontainers.image.version": "b9692"
               },
               "Annotations": {
                    "io.container.manager": "libpod",
                    "io.podman.annotations.init": "TRUE",
                    "io.podman.annotations.privileged": "TRUE",
                    "org.opencontainers.image.stopSignal": "15",
                    "org.systemd.property.KillSignal": "15",
                    "org.systemd.property.TimeoutStopUSec": "uint64 10000000"
               },
               "StopSignal": "SIGTERM",
               "HealthcheckOnFailureAction": "none",
               "HealthLogDestination": "local",
               "HealthcheckMaxLogCount": 5,
               "HealthcheckMaxLogSize": 500,
               "CreateCommand": [
                    "podman",
                    "run",
                    "--label",
                    "ai.ramalama.model=hf://bartowski/Qwen_Qwen3.6-27B-GGUF:Q6_K_L",
                    "--label",
                    "ai.ramalama.engine=podman",
                    "--label",
                    "ai.ramalama.runtime=llama.cpp",
                    "--label",
                    "ai.ramalama.port=8080",
                    "--label",
                    "ai.ramalama.command=run",
                    "--privileged",
                    "--pull",
                    "newer",
                    "-d",
                    "--device",
                    "/dev/dri",
                    "--device",
                    "/dev/kfd",
                    "--device",
                    "/dev/dri",
                    "--device",
                    "/dev/kfd",
                    "--device",
                    "/dev/accel",
                    "-e",
                    "HIP_VISIBLE_DEVICES=0",
                    "-p",
                    "8080:8080",
                    "-t",
                    "-i",
                    "--label",
                    "ai.ramalama",
                    "--name",
                    "ramalama-x6yqjYBx19",
                    "--env=HOME=/tmp",
                    "--init",
                    "--mount=type=bind,src=/var/home/tob/.local/share/ramalama/store/huggingface/bartowski/Qwen_Qwen3.6-27B-GGUF/blobs/sha256-95c61f9a30b0e0ada59169177fde2234e6bfd0e477a45cbff52097763286f9f6,destination=/mnt/models/Qwen_Qwen3.6-27B-Q6_K_L.gguf,ro",
                    "--mount=type=bind,src=/var/home/tob/.local/share/ramalama/store/huggingface/bartowski/Qwen_Qwen3.6-27B-GGUF/blobs/sha256-c5c8c41da6d155a61edd21b7e3aa50b6ef77f122d502d36afe4a4d5c3e494d4f,destination=/mnt/models/mmproj-Qwen_Qwen3.6-27B-bf16.gguf,ro",
                    "--mount=type=bind,src=/var/home/tob/.local/share/ramalama/store/huggingface/bartowski/Qwen_Qwen3.6-27B-GGUF/blobs/sha256-e84f32a23fdda27689f868aa4a1a5621f41133e51a48d7f3efcbea2839574259,destination=/mnt/models/chat_template_extracted,ro",
                    "ghcr.io/ggml-org/llama.cpp:full-vulkan",
                    "--server",
                    "--host",
                    "::",
                    "--port",
                    "8080",
                    "--model",
                    "/mnt/models/Qwen_Qwen3.6-27B-Q6_K_L.gguf",
                    "--mmproj",
                    "/mnt/models/mmproj-Qwen_Qwen3.6-27B-bf16.gguf",
                    "--chat-template-file",
                    "/mnt/models/chat_template_extracted",
                    "--no-warmup",
                    "--alias",
                    "bartowski/Qwen_Qwen3.6-27B-GGUF",
                    "--temp",
                    "0.8",
                    "--threads",
                    "8",
                    "--log-colors",
                    "on",
                    "--spec-default"
               ],
               "Umask": "0022",
               "Timeout": 0,
               "StopTimeout": 10,
               "Passwd": true,
               "sdNotifyMode": "container",
               "ExposedPorts": {
                    "8080/tcp": {}
               }
          },
          "HostConfig": {
               "Binds": [
                    "/var/home/tob/.local/share/ramalama/store/huggingface/bartowski/Qwen_Qwen3.6-27B-GGUF/blobs/sha256-c5c8c41da6d155a61edd21b7e3aa50b6ef77f122d502d36afe4a4d5c3e494d4f:/mnt/models/mmproj-Qwen_Qwen3.6-27B-bf16.gguf:ro,rprivate,rbind",
                    "/var/home/tob/.local/share/ramalama/store/huggingface/bartowski/Qwen_Qwen3.6-27B-GGUF/blobs/sha256-e84f32a23fdda27689f868aa4a1a5621f41133e51a48d7f3efcbea2839574259:/mnt/models/chat_template_extracted:ro,rprivate,rbind",
                    "/var/home/tob/.local/share/ramalama/store/huggingface/bartowski/Qwen_Qwen3.6-27B-GGUF/blobs/sha256-95c61f9a30b0e0ada59169177fde2234e6bfd0e477a45cbff52097763286f9f6:/mnt/models/Qwen_Qwen3.6-27B-Q6_K_L.gguf:ro,rprivate,rbind"
               ],
               "CgroupManager": "systemd",
               "CgroupMode": "private",
               "ContainerIDFile": "",
               "LogConfig": {
                    "Type": "journald",
                    "Config": null,
                    "Path": "",
                    "Tag": "",
                    "Size": "-1B"
               },
               "NetworkMode": "pasta",
               "PortBindings": {
                    "8080/tcp": [
                         {
                              "HostIp": "0.0.0.0",
                              "HostPort": "8080"
                         }
                    ]
               },
               "RestartPolicy": {
                    "Name": "no",
                    "MaximumRetryCount": 0
               },
               "AutoRemove": false,
               "AutoRemoveImage": false,
               "Annotations": {
                    "io.container.manager": "libpod",
                    "io.podman.annotations.init": "TRUE",
                    "io.podman.annotations.privileged": "TRUE",
                    "org.opencontainers.image.stopSignal": "15",
                    "org.systemd.property.KillSignal": "15",
                    "org.systemd.property.TimeoutStopUSec": "uint64 10000000"
               },
               "VolumeDriver": "",
               "VolumesFrom": null,
               "CapAdd": [],
               "CapDrop": [],
               "Dns": [],
               "DnsOptions": [],
               "DnsSearch": [],
               "ExtraHosts": [],
               "HostsFile": "",
               "GroupAdd": [],
               "IpcMode": "shareable",
               "Cgroup": "",
               "Cgroups": "default",
               "Links": null,
               "OomScoreAdj": 0,
               "PidMode": "private",
               "Privileged": true,
               "PublishAllPorts": false,
               "ReadonlyRootfs": false,
               "SecurityOpt": [
                    "unmask=all"
               ],
               "Tmpfs": {},
               "UTSMode": "private",
               "UsernsMode": "",
               "ShmSize": 65536000,
               "Runtime": "oci",
               "ConsoleSize": [
                    0,
                    0
               ],
               "Isolation": "",
               "CpuShares": 0,
               "Memory": 0,
               "NanoCpus": 0,
               "CgroupParent": "user.slice",
               "BlkioWeight": 0,
               "BlkioWeightDevice": null,
               "BlkioDeviceReadBps": null,
               "BlkioDeviceWriteBps": null,
               "BlkioDeviceReadIOps": null,
               "BlkioDeviceWriteIOps": null,
               "CpuPeriod": 0,
               "CpuQuota": 0,
               "CpuRealtimePeriod": 0,
               "CpuRealtimeRuntime": 0,
               "CpusetCpus": "",
               "CpusetMems": "",
               "Devices": [],
               "DiskQuota": 0,
               "KernelMemory": 0,
               "MemoryReservation": 0,
               "MemorySwap": 0,
               "MemorySwappiness": 0,
               "OomKillDisable": false,
               "Init": true,
               "PidsLimit": 2048,
               "Ulimits": [
                    {
                         "Name": "RLIMIT_NOFILE",
                         "Soft": 524288,
                         "Hard": 524288
                    },
                    {
                         "Name": "RLIMIT_NPROC",
                         "Soft": 247346,
                         "Hard": 247346
                    }
               ],
               "CpuCount": 0,
               "CpuPercent": 0,
               "IOMaximumIOps": 0,
               "IOMaximumBandwidth": 0,
               "CgroupConf": null
          },
          "UseImageHosts": false,
          "UseImageHostname": false
     }
]

[bmahabirbu]

ramalama shouldnt need to run in toolbox because toolbox is already creating a container for you so gpu passthrough gets wonky

is it possible just to run it on host?

[ibotty]

That's possible but pollutes the host OS (which is not as nice with immutable OSs). I am not running ramalama in a toolbox because of security sandboxing.

Also note that llama-server --list-devices finds the device in the toolbox.

Testing further, it looks like it works when using ramalama serve --generate quadlet, so I have at least a workaround for running a model now.

[olliewalsh]

Using podman-remote worked for me on Fedora Silverblue the last time I looked at this (#2086)

[ibotty]

I have a local podman script that basically calls flatpak-spawn --host podman. That's pretty much what's recommended in the official documentation afaict.

[olliewalsh]

I have a local podman script that basically calls flatpak-spawn --host podman. That's pretty much what's recommended in the official documentation afaict.

@rhatdan those docs were added in #2477, so did this work for you?

[rhatdan]

Sure, although it would be nice if this worked right out of the box.