Reddit posts

Hey everyone. I'm still wrapping my head around running local models and all the technical details involved. So the below is 99.9% Claude, as are the tests, harness, and conclusions. I'm just trying to make running local models on a strix halo better however I can. I don't like being a meat proxy, but here it is: "ROCm beats Vulkan at prompt processing on Strix Halo" is repeated a lot. After ten boots and five models, I think a large part of it is the IOMMU. model Vulkan/ROCm prefill @ iommu=pt @ amd_iommu=off gemma-4-26B-A4B q4_0 0.99 1.02 gpt-oss-120b mxfp4 0.99 1.05 gemma-4-26B-A4B Q8_0 0.86 1.00 muse-glimmer-30B Q4_K_M (dense) 0.71 0.91 Qwen3.8-27B Q8_0 (dense) 0.76 0.91 With the IOMMU on, Vulkan gives up as much as 29% of ROCm's prefill. Turn it off and that drops to ~10% at worst, and parity on the MoEs. Vulkan's gain tracks exactly how far behind it was. The prefill gains themselves: model GB read/forward Vulkan ROCm gemma-4-26B-A4B q4_0 2.0 +5.4% +2.6% gpt-oss-120b mxfp4 2.6 +8.0% +1.8% gemma-4-26B-A4B Q8_0 4.0 +20.0% +3.2% muse-glimmer-30B Q4_K_M 14.0 +31.6% +3.7% Qwen3.8-27B Q8_0 27.0 +26.2% +6.0% Method: A/B/A/B across ten boots, interleave

The vLLM recipe page for Muse Glimmer has this for speculative decoding: --speculative-config '{"method": "dflash", "model": "meta-models/Muse-Glimmer-30B-assistant", "num_speculative_tokens": 15}' This errors out on the current vllm/vllm-openai:muse-glimmer image, and each fix reveals the next error. Six separate issues in total, all in the DFlash path. The base model runs fine without the spec config. The source for the image isn't public yet (the recipe says "code will be released soon"), so I pulled the image layers through the registry API and read the code to figure out what was going on. Also checked tensor names by range-requesting the safetensors headers off HF instead of downloading the weights. What I found: • The drafter's config declares MuseGlimmerAssistantModel, which is in vLLM's registry. But the dflash code renames it to DFlashMuseGlimmerAssistantModel before the registry lookup, and that name isn't registered. Dies in config validation. • vLLM maps the drafter's config to Qwen3Config (there's a comment calling it "Qwen3-shaped"). The muse JSON omits vocab_size and use_sliding_window, so Qwen3Config fills in its own defaults: vocab becomes 151936 (the model i

Disclaimer. This is the first time I've used Muse or VSCode as a harness. The reason I am using VSCode as a harness is this is a research project for my job, and we only have VSCode, Codex, and Claude Code approved for harnesses. I ran it in a folder with only one HTML file (800 lines) that is a Roblox-style COD game. I just gave it a prompt "Can you fix the bugs in the file". It read the file 3 times, found one bug, started to fix it, then got stuck reading the same 10 lines over and over. I imagine it's one of these three issues. • It's a prompt error, being way too vague/open ended for the capabilities of a smaller model. I tried again, with a specific prompt to fix a specific bug, and it still just ends up so confused, trying to grep/find the file despite already having read it, and trying to find the code inside of the file. • It's a limitation of small models running with a large harness/having way too much going on. I tried running it with Pi with its default prompt, and it just got stuck doing tool calls and never actually read the file. Tried running this just directly in the Unsloth Desktop UI with no harness but it failed to parse the file I inputted and tried to gen

Ran Muse Glimmer on a single RTX 3090 and found a max_tokens setting that made it look dumb; shares numbers at filled context and notes better-than-expected non-English handling.