Entries matching “vulkan”

Hey everyone. I'm still wrapping my head around running local models and all the technical details involved. So the below is 99.9% Claude, as are the tests, harness, and conclusions. I'm just trying to make running local models on a strix halo better however I can. I don't like being a meat proxy, but here it is: "ROCm beats Vulkan at prompt processing on Strix Halo" is repeated a lot. After ten boots and five models, I think a large part of it is the IOMMU. model Vulkan/ROCm prefill @ iommu=pt @ amd_iommu=off gemma-4-26B-A4B q4_0 0.99 1.02 gpt-oss-120b mxfp4 0.99 1.05 gemma-4-26B-A4B Q8_0 0.86 1.00 muse-glimmer-30B Q4_K_M (dense) 0.71 0.91 Qwen3.8-27B Q8_0 (dense) 0.76 0.91 With the IOMMU on, Vulkan gives up as much as 29% of ROCm's prefill. Turn it off and that drops to ~10% at worst, and parity on the MoEs. Vulkan's gain tracks exactly how far behind it was. The prefill gains themselves: model GB read/forward Vulkan ROCm gemma-4-26B-A4B q4_0 2.0 +5.4% +2.6% gpt-oss-120b mxfp4 2.6 +8.0% +1.8% gemma-4-26B-A4B Q8_0 4.0 +20.0% +3.2% muse-glimmer-30B Q4_K_M 14.0 +31.6% +3.7% Qwen3.8-27B Q8_0 27.0 +26.2% +6.0% Method: A/B/A/B across ten boots, interleave

Picked up China version of the Mi50 (Radeon VII) 16GB VRAM GPU for about $135. Ran it using llama.cpp Ubuntu Vulkan prebuilt binary build: b29c606e2 (10964). Used a Power Limit or 220/190 watts on the GPUs. Dual Radeon 32GB Vram and 64GB DDR4 System Dual Radeon RX 7900 GRE and Radeon VII 32gb VRAM GGUF Models: • Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf • Accio-Lab_occamy-1.0-Q5_K_S.gguf • Laguna-XS-2.1-APEX-I-Balanced.gguf • NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Q5_K_M.gguf • Gemma-4-31B-it-Q6_K.gguf • Ateron_Gemma-4-MoonGem-31B-Q5_K_M.gguf • Qwen3-Coder-30B-A3B-Instruct-UD-Q6_K_XL.gguf • Qwen3-VL-30B-A3B-Thinking-UD-Q6_K_XL.gguf • Qwen3-Coder-30B-A3B-Instruct-UD-Q5_K_XL.gguf • North-Mini-Code-1.0-MXFP4_MOE.gguf • GLM-4.7-Flash-Uncen-Hrt-NEO-CODE-MAX-imat-D_AU-Q6_K.gguf • Muse-Glimmer-30B-UD-Q6_K_XL.gguf • Huihui-Qwen3.8-27B-abliterated-UD-Q6_K_XL.gguf • Qwen3.8-27B-Q6_K.gguf • Qwen3.8-27B-OBLITERATED-Q5_K_M.gguf • Medgemma-27b-it-UD-Q6_K_XL.gguf • Gemma-4-26B-A4B-it-UD-Q6_K_XL.gguf • Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf • GPT-OSS-20b-abliterated.i1-Q6_K.gguf Sorted by params then size model size params pp512 tg128 qwen35moe 35B.A3B Q5_K - Medium 24.76 GiB 3

Eleven matched on/off pairs across Gemma 4 and Qwen3.6, holding model, quant, card, corpus and concurrency fixed inside each pair. Speed: 1.65x to 2.54x, every pair. Accuracy: nothing the paired intervals could separate from ordinary run-to-run movement. Muse Glimmer is the one that lost. Meta's matching DFlash drafter made the same 7900 XTX 9% slower, keeping 24.55% of drafted tokens against roughly four in five for the Gemma and Qwen heads. Acceptance fell across the run instead of warming up. Meta's model card reports 3.1x on an RTX 5090, and there are open llama.cpp issues for DFlash on AMD and under Vulkan, so I read it as the backend rather than the model. Acceptance turned out to be a poor predictor of speed. It moved under four points across five models while the multiple nearly doubled. What tracks the multiple is how bandwidth-bound the target is: a heavier quant gains more, and the two mixture-of-experts pairs gained least. Worth knowing before you benchmark anything: -md mtp-head.gguf silently disables speculation. Use -hf REPO:QUANT -hfd REPO, then read speculative from /slots and confirm it is true. Per-pair table, intervals, acceptance counters and the raw predic