Local & open models

Unsloth's docs page for running Meta's Muse Glimmer 30B locally with its GGUF quants.

Muse Glimmer Cookbook Clone it, run it on the GPU you already own, and ship a working agent — one that plans, calls tools, and self-corrects — in a single sitting, fully offline. Muse Glimmer is an open-weight model built for local agentic work on a single GPU. This cookbook is how you go from weights to a running agent. Previously shared Liquid AI's Cookbook.

KGP Talkie benchmarks generation speed, VRAM use and long-context behavior for both models on one RTX 5090.

@mpfaffenberger

@mpfaffenberger

A reproducible single-GPU deployment of Muse Glimmer 30B in BF16 with DFlash speculative decoding on a 96GB RTX PRO 6000 Blackwell, served via vLLM with pinned overlays and smoke tests.

@n3xtgentechitalia

@n3xtgentechitalia

A llama.cpp container that serves Muse Glimmer 30B with vision and DFlash on an RTX 5090, reporting 98.8 tok/s single-stream and about 250 tok/s aggregate with 131k context.

Fine-tune of Muse Glimmer 30B for Hermes Agent and agentic tool work that teaches the model to call one or two tools and stop.

This started as a failure. I had cut 6.34% of Meta's Muse-Glimmer-30B (the FFN sublayers of four layers, the next FFN after each cut retrained against the parent) and the healed model passed my fidelity bar at Q8_0. At Q4_K it failed by 0.006 KLD, and the arithmetic said why: the surgery's cost plus the ordinary Q4_K cost adds up to just over the bar, and three months of levers on the surgery side could not close a gap that small. So I attacked the other term. In a fixed GGUF the integer codes are frozen, but every quantised block still carries one or two fp16 scales, and the decoded weight is linear in them. That means the scales can be trained end to end against the parent's next-token distribution on the student's own forward pass, without touching the codec, the format, the byte length or the offsets. On the surgical model it worked: 0.05615 fail to 0.04949 pass on 45,056 held-out positions, and the preregistered control (the same recovery on the uncut parent at Q4_K) showed the two costs are not additive once the scales are trained; recovery took back part of the surgery error too. That file and the whole study are on my Hugging Face page. Then the obvious question: does it

Lightweight LoRA that makes Muse Glimmer 30B output a target element's bounding box directly from a screenshot and instruction, for computer-use agents.

@mitchallen

@mitchallen

A Makefile that downloads the three Muse Glimmer 30B GGUFs and builds and serves llama.cpp on Apple Silicon, automating a scriptable.com walkthrough.

Standalone packaging of the vision tower and projector extracted from Muse Glimmer 30B.

A Muse Glimmer fine-tune that writes, tunes and translates detection rules (Sigma to KQL and SPL, YARA, Wazuh) and runs 4-bit on a single 24 GB card.

Ran the model with quants (Q4) by Unsloth with latest (build from master) llama.cpp server. It takes ~20GB ram running on M5 Pro with 48GB at about 17t/s. Didn't do any reasoning loops/overthinking. Overall, sits below Qwen3.6 27B, wasn't able to get good code (frontend and backend) results. On the positive side, it didn't fail any tool calls. Your opinions/findings? Watch more: https://www.youtube.com/watch?v=_5wKhkUT438

I noticed on the same hardware that I can get 24 x 128k contexts with muse glimmer (30b q8_0 + mmproj+dflash) only gets me 3x 256k or 6x 128k with qwen. But a straight forward analysis of the architecture suggests to me that qwen's state per token is somewhat smaller than glimmers. So it seems llama.cpp is particularly memory inefficient for the qwen arch. I presume there is an existing issue for this, but I couldn't find one. What's the deal? The extra concurrency makes a big difference in batched performance.

Sumanth

@Sumanth_077

Run and fine-tune Meta's Muse Glimmer locally! Meta released Muse Glimmer, a 30B dense vision model designed for local agentic and coding workflows. The first open model from Meta Superintelligence Labs, released under Apache 2.0. The model runs locally at different memory

davidlee

@davidlee

Muse Glimmer powering any harness including Codex and Claude Code using @ollama

muse glimmer dropped yesterday and mlx-lm couldn't load it yet, so i wrote the text model port and opened a PR. i checked it against meta's own transformers reference before posting, 5 out of 5 next token matches and 0.9965 logit cosine, so it's not just coherent it actually matches the reference. if you want to run glimmer on apple silicon right now the model file is in the PR. https://github.com/ml-explore/mlx-lm/pull/1710

Got Muse Glimmer 30B running locally using the UD-Q2-K-XL quant paired with DFlash speculative decoding, and the results on modest hardware are pretty impressive. Hardware Setup Host: Ryzen 5 4600G with 96GB DDR4 RAM running headless Debian Trixie. Guest VM: QEMU/KVM assigned 4 cores and 32GB RAM, running Debian Sid with ROCm 7.2. GPU: AMD Radeon RX 7600 XT 16GB passed through to the VM, built llama.cpp fresh from master targeting gfx1102 and gfx1201 via HIP. Context Size: Set to 62144 tokens. Processed 14685 total tokens at roughly 308 tokens per second prompt evaluation and 20 tokens per second generation speed. Speculative Decoding: Using the dflash-kquant draft model with spec-draft-n-max set to 2. Fed it a clean context slate consisting of eight JavaScript files and one HTML file alongside the problem description. On the first turn, it identified and output the necessary diff snippets. A quick follow-up prompt telling it to stop being lazy and output the complete updated files yielded functional code that dropped straight in and worked on the first try.

Ryan | Intelligence Engineer

@RyanIntEngineer

I just built and validated a complete Muse Glimmer deployment for one DGX Spark. I didn’t make another quant—I integrated the available pieces into a pinned, verified, DFlash-accelerated, tool-capable, reboot-durable stack that actually works in OWUI.

HolaClaw's tutorial for running Muse Glimmer 30B behind OpenClaw on a Mac, with hardware requirements and llama.cpp and Ollama setup.

The vLLM recipe page for Muse Glimmer has this for speculative decoding: --speculative-config '{"method": "dflash", "model": "meta-models/Muse-Glimmer-30B-assistant", "num_speculative_tokens": 15}' This errors out on the current vllm/vllm-openai:muse-glimmer image, and each fix reveals the next error. Six separate issues in total, all in the DFlash path. The base model runs fine without the spec config. The source for the image isn't public yet (the recipe says "code will be released soon"), so I pulled the image layers through the registry API and read the code to figure out what was going on. Also checked tensor names by range-requesting the safetensors headers off HF instead of downloading the weights. What I found: • The drafter's config declares MuseGlimmerAssistantModel, which is in vLLM's registry. But the dflash code renames it to DFlashMuseGlimmerAssistantModel before the registry lookup, and that name isn't registered. Dies in config validation. • vLLM maps the drafter's config to Qwen3Config (there's a comment calling it "Qwen3-shaped"). The muse JSON omits vocab_size and use_sliding_window, so Qwen3Config fills in its own defaults: vocab becomes 151936 (the model i

Solo-built GGUF line of Muse Glimmer 30B with custom calibration, per-tensor allocations and an eval harness behind every reported number.

Hey Folks, I've been making quants for a while - recently I took a short break to get into hardcore research (submitted my first EMNLP paper during it!). Along the way, I built up a little arsenal of quant-optim techniques: everything from novel, paper-pending tricks to some genuinely sick tensor-mapping algos. I threw some of the secret sauce into the newly released Muse Glimmer 30B (META IS BACK!) and compared it to several OGs. I'm honestly shocked by how it never loses to any quant out there in every single VRAM class! One of the coolest ones is my Q8 quant, it is smaller than UD-Q8_K_XL and 21% closer to BF16. Full methodology is on the card - eval setup, CIs, held-out slices, the lot. Happy to answer questions in the comments. Model: https://huggingface.co/AaryanK/Muse-Glimmer-30B-GGUF I still had headroom left but ran out of compute credits :( Being a solo undergrad sophomore, I can't exactly spend H100 money that often, which is why the "hopefully" in the title :) I'm looking for internships in AI agent orchestration and model inference. If this work looks relevant to your team: linkedin.com/in/theaaryankapoor I plan on doing a write-up soon to describe some of the

@vcruz305

@vcruz305

Tested SGLang and vLLM launch recipes for an NVFP4 export of a Hermes-agentic fine-tune of Muse-Glimmer-30B on NVIDIA DGX Spark (GB10).

GGUF quantizations of the Inco AI DFlash2 drafter for Muse Glimmer 30B, for use with the BeeLlama.cpp llama.cpp fork.

IMPORTANT This post is meant to provide info regarding the best local models to run on CONSUMER HARDWARE. I am on an RTX 4060 with 8GB VRAM, 16GB of RAM and I am benchmarking models that can run on my computer. If you have sunk several thousands into graphics cards you won't find these statistics much useful. This post is for all the people who can't just install Qwen3.8 27B and call it a day. Additionally, I am not an LLM benchmarking expert. I am a hobbyist and occasional LLM user trying to extract useful information for both me and people on similar hardware. Context For the past few weeks I have been doing some benchmarks of some LLMs that can run on my laptop which only has 8GB VRAM and 16GB RAM. I was mostly toying around while trying to get some useful data about what the best model is for local inference on consumer hardware. This week I decided to make a "final" benchmark that would be way better with more questions, more question categories, newer models (a lot of people complained about the models I had benchmarked before being old but I didn't find most suggested models to be any good) and a better speed benchmark, this time using TTC (Time To Completion) as a pose to

ROCmFP4 and ROCmFP8 builds of Muse Glimmer 30B and its drafter, targeted and tested on AMD Strix Halo (gfx1151).

@47thtechcorner

@47thtechcorner

A YouTube tutorial companion repo where Muse Glimmer 30B (a 2-bit GGUF, run offline) looks at screenshots of badly designed web pages and generates and self-repairs modern replacement code.

LoRA adapter that makes Muse Glimmer 30B reliably commit to tool calls when it already knows the correct function.

Been building this for a few months, mostly for myself, and it just got a proper release so figured I'd post it. It's a native GGUF inference runtime with OpenAI/Anthropic-compatible APIs and a chat UI. The whole point is one consumer NVIDIA card + lots of RAM: MoE models that don't fit in VRAM run their experts on the CPU, or split with a hot set cached on the card. It figures out what fits at startup instead of you guessing offload layer counts. Runs Qwen 3.x dense and MoE (incl. Qwen3.8-Flash-Next), DeepSeek-V4-Flash, Ling 3.0, K2-Horizon, Gemma 4, Laguna, Muse Glimmer. Image input via mmproj on the Qwen models. Also does Z-Image-Turbo image gen next to a chat model on the same card. Numbers from my laptop (5070 Ti 12 GB, 60 GB RAM): - Qwen3.8-Flash-Next IQ1_S: ~35 tok/s decode, ~475 tok/s prefill - Qwen3.8-27B IQ2_XXS: ~40 tok/s - DeepSeek-V4-Flash: 6-7 tok/s (that's basically the DRAM bandwidth limit) - Z-Image 1024x1024 in ~15 s with a 35B loaded beside it Stuff I think is neat: - Kernels are compiled at runtime by NVRTC, so no CUDA toolkit in the wheel and no nvcc. Same kernel source compiles as plain C++ for a CPU-only backend. - KV cache in f16 / q8_0 / TurboQu

@PipeNetwork

@PipeNetwork

muse-glimmer-mlx is an MLX port of Muse Glimmer 30B for Apple Silicon that supplies the missing runtime so the many unloadable MLX conversions published on Hugging Face can actually be run.

@network-tocoder

@network-tocoder

A video companion repo with the commands and configs for running Muse Glimmer 30B locally on an RTX 3090 and comparing the Prime Agent and Hermes harnesses under the same conditions.

An unofficial parody MLX-VLM QLoRA adapter for Muse Glimmer 30B on Apple Silicon.

I've had Qwen3.6:27b (and Qwen 3 coder next before it) running along side gpt-oss:20b for a while now as my two main models (qwen for coding, gpt-oss for agentic stuff). Qwen is pretty self-explanatory, while I had been using gpt-oss because of how good it was at producing json and instruction following, despite the size and age. https://preview.redd.it/lr61tb02lejh1.png?width=1920&format=png&auto=webp&s=cef1f0bbfb43c8462162ec675d52c5896d174118 The "upgrade" to 3.8 is pretty evident, especially because of the SWE bench score improvement, but I was more reticent with Muse-Glimmer as I had some trouble when I tried Gemma4, which was far too opinionated when given a task, but Muse-Glimmer seems great - low memory footprint at 128K context, fairly fast and seems to follow instructions well. What models are you using locally and for what? did you have a similar experience with these latest models?

Benchmarked Muse Glimmer 30B on my RTX 5090 (32GB), 262k context, UD-Q5_K_M + dflash-kquant + mmproj. Workload Stock master + DFlash ngram-simple PR #26842 + DFlash Code patch 78 t/s 57 t/s 220-253 t/s Mixed agent turn 77 t/s 68 t/s 188-213 t/s Tool-call JSON 71 t/s 75 t/s 155-181 t/s Heavy reasoning 52 t/s 58 t/s 120-130 t/s PR #26842 moves the DFlash draft argmax from CPU to GPU, which was the bottleneck. I cherry-picked it onto master (it branched before the Muse merge, one conflict to resolve manually) and it builds clean. Code generation now matches Meta's published 233 t/s, which I could not reproduce on stock master. Notes: • ngram-simple loses to DFlash on every coding workload. • Server caps context at the model's metadata context_length, use --override-kv for 262k. • The reasoning budget flags do not work with this template. This is verified: with the budget set to 64, the model still burned 2000+ chars thinking and the budget message never appeared. Leave max_tokens headroom for the reasoning block. Flags: llama-server ^ --model Muse-Glimmer-30B-UD-Q5_K_M.gguf ^ --mmproj mmproj-kquant.gguf ^ -c 262144 --parallel 1 ^ --override-kv "muse-glimmer.context_le

Hey everyone. I'm still wrapping my head around running local models and all the technical details involved. So the below is 99.9% Claude, as are the tests, harness, and conclusions. I'm just trying to make running local models on a strix halo better however I can. I don't like being a meat proxy, but here it is: "ROCm beats Vulkan at prompt processing on Strix Halo" is repeated a lot. After ten boots and five models, I think a large part of it is the IOMMU. model Vulkan/ROCm prefill @ iommu=pt @ amd_iommu=off gemma-4-26B-A4B q4_0 0.99 1.02 gpt-oss-120b mxfp4 0.99 1.05 gemma-4-26B-A4B Q8_0 0.86 1.00 muse-glimmer-30B Q4_K_M (dense) 0.71 0.91 Qwen3.8-27B Q8_0 (dense) 0.76 0.91 With the IOMMU on, Vulkan gives up as much as 29% of ROCm's prefill. Turn it off and that drops to ~10% at worst, and parity on the MoEs. Vulkan's gain tracks exactly how far behind it was. The prefill gains themselves: model GB read/forward Vulkan ROCm gemma-4-26B-A4B q4_0 2.0 +5.4% +2.6% gpt-oss-120b mxfp4 2.6 +8.0% +1.8% gemma-4-26B-A4B Q8_0 4.0 +20.0% +3.2% muse-glimmer-30B Q4_K_M 14.0 +31.6% +3.7% Qwen3.8-27B Q8_0 27.0 +26.2% +6.0% Method: A/B/A/B across ten boots, interleave

Ran Muse Glimmer on a single RTX 3090 and found a max_tokens setting that made it look dumb; shares numbers at filled context and notes better-than-expected non-English handling.