Local & open models

PursuitOfDataScience

PursuitOfDataScience

LoRA adapter that makes Muse Glimmer 30B reliably commit to tool calls when it already knows the correct function.

godqhr1421

godqhr1421

A static Space showing a Hermes-agentic fine-tune of Muse Glimmer 30B proposing one or two tool calls or answering directly; the model card reports 16/20 on a simulated loop gate.

ValiantLabs

ValiantLabs

Valiant Labs' Esper 4 agentic coding fine-tune of Muse Glimmer 30B.

The vLLM recipe page for Muse Glimmer has this for speculative decoding: --speculative-config '{"method": "dflash", "model": "meta-models/Muse-Glimmer-30B-assistant", "num_speculative_tokens": 15}' This errors out on the current vllm/vllm-openai:muse-glimmer image, and each fix reveals the next error. Six separate issues in total, all in the DFlash path. The base model runs fine without the spec config. The source for the image isn't public yet (the recipe says "code will be released soon"), so I pulled the image layers through the registry API and read the code to figure out what was going on. Also checked tensor names by range-requesting the safetensors headers off HF instead of downloading the weights. What I found: • The drafter's config declares MuseGlimmerAssistantModel, which is in vLLM's registry. But the dflash code renames it to DFlashMuseGlimmerAssistantModel before the registry lookup, and that name isn't registered. Dies in config validation. • vLLM maps the drafter's config to Qwen3Config (there's a comment calling it "Qwen3-shaped"). The muse JSON omits vocab_size and use_sliding_window, so Qwen3Config fills in its own defaults: vocab becomes 151936 (the model i

Eleven matched on/off pairs across Gemma 4 and Qwen3.6, holding model, quant, card, corpus and concurrency fixed inside each pair. Speed: 1.65x to 2.54x, every pair. Accuracy: nothing the paired intervals could separate from ordinary run-to-run movement. Muse Glimmer is the one that lost. Meta's matching DFlash drafter made the same 7900 XTX 9% slower, keeping 24.55% of drafted tokens against roughly four in five for the Gemma and Qwen heads. Acceptance fell across the run instead of warming up. Meta's model card reports 3.1x on an RTX 5090, and there are open llama.cpp issues for DFlash on AMD and under Vulkan, so I read it as the backend rather than the model. Acceptance turned out to be a poor predictor of speed. It moved under four points across five models while the multiple nearly doubled. What tracks the multiple is how bandwidth-bound the target is: a heavier quant gains more, and the two mixture-of-experts pairs gained least. Worth knowing before you benchmark anything: -md mtp-head.gguf silently disables speculation. Use -hf REPO:QUANT -hfd REPO, then read speculative from /slots and confirm it is true. Per-pair table, intervals, acceptance counters and the raw predic

merve

merve

QLoRA adapter that teaches Muse Glimmer 30B to return click points on web screenshots from instructions, trained on the MolmoWeb dataset.

unsloth.ai

unsloth.ai

Unsloth's docs page for running Meta's Muse Glimmer 30B locally with its GGUF quants.

davidlee

@davidlee

Muse Glimmer powering any harness including Codex and Claude Code using @ollama

Muse Glimmer Cookbook Clone it, run it on the GPU you already own, and ship a working agent — one that plans, calls tools, and self-corrects — in a single sitting, fully offline. Muse Glimmer is an open-weight model built for local agentic work on a single GPU. This cookbook is how you go from weights to a running agent. Previously shared Liquid AI's Cookbook.

Quick update on the RTX 5060 Ti local LLM repo. It has changed quite a bit since my previous posts. The project started as a collection of practical notes and benchmark results. That was useful, but as the dataset grew it became harder to answer the question most people actually had: What configuration should I run? I have rebuilt the repo around tested, copyable presets rather than treating every successful benchmark request as a front-page result. What changed? The project now separates three things: • Presets: exact configurations intended for people to copy and run. • Evidence bundles: reviewed proof of context fit, retrieval, sustained generation and performance. • Raw receipts: retries, failed experiments and diagnostic runs that are kept separate as engineering material without automatically becoming recommendations. The website now leads with the published preset catalogue. The larger results explorer is still there for comparisons and historical data, but it is no longer the first thing visitors have to decipher. There are currently seven published presets across the 1× and 2× RTX 5060 Ti lanes: 1× RTX 5060 Ti 16GB • Qwen3.8 27B IQ3_XXS at 64K with q8 K

EryriLabs

EryriLabs

A Muse Glimmer fine-tune that writes, tunes and translates detection rules (Sigma to KQL and SPL, YARA, Wazuh) and runs 4-bit on a single 24 GB card.

Ran Muse Glimmer on a single RTX 3090 and found a max_tokens setting that made it look dumb; shares numbers at filled context and notes better-than-expected non-English handling.

SedimentLabs

SedimentLabs

Sediment's fine-tune of Muse Glimmer 30B that answers factual questions when confident and says "I don't know" otherwise, calibrated for the AA-Omniscience setting.

I've had Qwen3.6:27b (and Qwen 3 coder next before it) running along side gpt-oss:20b for a while now as my two main models (qwen for coding, gpt-oss for agentic stuff). Qwen is pretty self-explanatory, while I had been using gpt-oss because of how good it was at producing json and instruction following, despite the size and age. https://preview.redd.it/lr61tb02lejh1.png?width=1920&format=png&auto=webp&s=cef1f0bbfb43c8462162ec675d52c5896d174118 The "upgrade" to 3.8 is pretty evident, especially because of the SWE bench score improvement, but I was more reticent with Muse-Glimmer as I had some trouble when I tried Gemma4, which was far too opinionated when given a task, but Muse-Glimmer seems great - low memory footprint at 128K context, fairly fast and seems to follow instructions well. What models are you using locally and for what? did you have a similar experience with these latest models?

datacamp.com

datacamp.com

Abid Ali Awan sets up Muse Glimmer 30B on an RTX 5090 with llama.cpp, dynamic quantization and DFlash speculative decoding, serves it locally and wires it into OpenCode to build a medical research web app.

merve

@mervenoyann

this is super easy to run install llama binary: curl -LsSf llama.app/install.sh | sh run: llama serve -hf meta-models/muse-glimmer-30b --spec-type draft-dflash -fa on --jinja please spread the word

Been building this for a few months, mostly for myself, and it just got a proper release so figured I'd post it. It's a native GGUF inference runtime with OpenAI/Anthropic-compatible APIs and a chat UI. The whole point is one consumer NVIDIA card + lots of RAM: MoE models that don't fit in VRAM run their experts on the CPU, or split with a hot set cached on the card. It figures out what fits at startup instead of you guessing offload layer counts. Runs Qwen 3.x dense and MoE (incl. Qwen3.8-Flash-Next), DeepSeek-V4-Flash, Ling 3.0, K2-Horizon, Gemma 4, Laguna, Muse Glimmer. Image input via mmproj on the Qwen models. Also does Z-Image-Turbo image gen next to a chat model on the same card. Numbers from my laptop (5070 Ti 12 GB, 60 GB RAM): - Qwen3.8-Flash-Next IQ1_S: ~35 tok/s decode, ~475 tok/s prefill - Qwen3.8-27B IQ2_XXS: ~40 tok/s - DeepSeek-V4-Flash: 6-7 tok/s (that's basically the DRAM bandwidth limit) - Z-Image 1024x1024 in ~15 s with a 35B loaded beside it Stuff I think is neat: - Kernels are compiled at runtime by NVRTC, so no CUDA toolkit in the wheel and no nvcc. Same kernel source compiles as plain C++ for a CPU-only backend. - KV cache in f16 / q8_0 / TurboQu

Hey everyone. I'm still wrapping my head around running local models and all the technical details involved. So the below is 99.9% Claude, as are the tests, harness, and conclusions. I'm just trying to make running local models on a strix halo better however I can. I don't like being a meat proxy, but here it is: "ROCm beats Vulkan at prompt processing on Strix Halo" is repeated a lot. After ten boots and five models, I think a large part of it is the IOMMU. model Vulkan/ROCm prefill @ iommu=pt @ amd_iommu=off gemma-4-26B-A4B q4_0 0.99 1.02 gpt-oss-120b mxfp4 0.99 1.05 gemma-4-26B-A4B Q8_0 0.86 1.00 muse-glimmer-30B Q4_K_M (dense) 0.71 0.91 Qwen3.8-27B Q8_0 (dense) 0.76 0.91 With the IOMMU on, Vulkan gives up as much as 29% of ROCm's prefill. Turn it off and that drops to ~10% at worst, and parity on the MoEs. Vulkan's gain tracks exactly how far behind it was. The prefill gains themselves: model GB read/forward Vulkan ROCm gemma-4-26B-A4B q4_0 2.0 +5.4% +2.6% gpt-oss-120b mxfp4 2.6 +8.0% +1.8% gemma-4-26B-A4B Q8_0 4.0 +20.0% +3.2% muse-glimmer-30B Q4_K_M 14.0 +31.6% +3.7% Qwen3.8-27B Q8_0 27.0 +26.2% +6.0% Method: A/B/A/B across ten boots, interleave

Heeeey all! I just completed some fun tests with Muse Glimmer, I thought I'd let you know. In fact, the summary below was written by Muse itself! I ran a 2× DGX Spark cluster and got Meta's day-old Muse Glimmer 30B running the day after release — then pushed its context from the trained 131K all the way to 1M with YaRN, verifying retrieval at every rung. Sharing config + results since the "131,072+" hint in the model card turned out to be very real. Setup • Hardware: 2× NVIDIA DGX Spark (GB10, 128 GB unified each, ~273 GB/s), ConnectX-7 direct link between them • Engine: llama.cpp master (day-1 muse_glimmer support), built from source with CUDA sm_121 + GGML_RPC • Model: official Muse-Glimmer-30B-GGUF K-Quant-Dynamic (~18.3 GiB) + official mmproj (vision) + official DFlash drafter • Spec decode: --spec-type draft-dflash --spec-draft-n-max 15 (block-diffusion drafter) • Context extension: --rope-scaling yarn --rope-scale <2/4/8> --yarn-orig-ctx 131072 plus --override-kv muse-glimmer.context_length=int:<N> (llama.cpp caps at trained length otherwise) • Yes, we also ran it split across both Sparks with llama.cpp RPC — no reason beyond liking to cluster things for fun. Our daily dri

Hey. Just tried it on my old ass gpus 😄 Surprisingly Tensor Split is working on 2 gpus almost doubling PP (wonder how it will work with 4 gpus) Q6 — 1 GPU llama-server \ --model <MODEL_DIR>/Muse-Glimmer-30B-GGUF/Muse-Glimmer-30B-UD-Q6_K_XL.gguf \ --mmproj <MODEL_DIR>/Muse-Glimmer-30B-GGUF/mmproj-kquant.gguf \ --spec-draft-model <MODEL_DIR>/Muse-Glimmer-30B-GGUF/dflash-kquant.gguf \ --spec-type draft-dflash \ --spec-draft-ngl 999 \ --spec-draft-n-max 3 \ --spec-draft-type-k f16 \ --spec-draft-type-v f16 \ --ctx-size 65536 \ --override-kv muse-glimmer.context_length=int:65536,dflash.context_length=int:65536 \ --n-gpu-layers 999 \ --device ROCm0 \ --device-draft ROCm0 \ --split-mode layer \ --flash-attn on \ --fit off \ --parallel 1 \ --kv-unified \ --batch-size 2048 \ --ubatch-size 512 \ --threads 32 \ --threads-batch 32 \ --cache-type-k f16 \ --cache-type-v f16 \ --image-min-tokens 1024 \ --image-max-tokens 4096 \ --reasoning-preserve \ --temp 0.7 \ --top-p 0.95 \ --top-k 64 \ --min-p 0.0 \ --jinja Q8 — 2 GPUs with tensor split bash llama-server \ --model <MODEL_DIR>/Muse-Glimmer-30B-GGUF/Muse-Glimmer-30B-UD-Q8_K_XL.gguf \ --mmproj <MODEL_DIR>/Muse-Glimmer-30B-GGUF/mmproj-kquan

Ryan | Intelligence Engineer

@RyanIntEngineer

I just built and validated a complete Muse Glimmer deployment for one DGX Spark. I didn’t make another quant—I integrated the available pieces into a pinned, verified, DFlash-accelerated, tool-capable, reboot-durable stack that actually works in OWUI.

@SamuelAlexander

@SamuelAlexander

Samuel Alexander ran Muse Glimmer 30B entirely on a Qualcomm Dragonwing IQ-9075 board for zero-shot PCB defect inspection and tool calling, measuring 21.6 GB resident with full 131K context and 2.84 tokens/s generation.

kgptalkie.com

kgptalkie.com

KGP Talkie benchmarks generation speed, VRAM use and long-context behavior for both models on one RTX 5090.

Satgoy152

Satgoy152

DFlash2 drafter for Muse Glimmer 30B fine-tuned on long-horizon, on-policy software-engineering traces to speed up coding workloads.

Got Muse Glimmer 30B running locally using the UD-Q2-K-XL quant paired with DFlash speculative decoding, and the results on modest hardware are pretty impressive. Hardware Setup Host: Ryzen 5 4600G with 96GB DDR4 RAM running headless Debian Trixie. Guest VM: QEMU/KVM assigned 4 cores and 32GB RAM, running Debian Sid with ROCm 7.2. GPU: AMD Radeon RX 7600 XT 16GB passed through to the VM, built llama.cpp fresh from master targeting gfx1102 and gfx1201 via HIP. Context Size: Set to 62144 tokens. Processed 14685 total tokens at roughly 308 tokens per second prompt evaluation and 20 tokens per second generation speed. Speculative Decoding: Using the dflash-kquant draft model with spec-draft-n-max set to 2. Fed it a clean context slate consisting of eight JavaScript files and one HTML file alongside the problem description. On the first turn, it identified and output the necessary diff snippets. A quick follow-up prompt telling it to stop being lazy and output the complete updated files yielded functional code that dropped straight in and worked on the first try.

@nicedreamzapp

@nicedreamzapp

An architecture port adding the muse_glimmer model class (vision tower, language model, projector and image processor) to mlx-vlm, so any Muse Glimmer checkpoint runs multimodally on Apple Silicon.

I noticed on the same hardware that I can get 24 x 128k contexts with muse glimmer (30b q8_0 + mmproj+dflash) only gets me 3x 256k or 6x 128k with qwen. But a straight forward analysis of the architecture suggests to me that qwen's state per token is somewhat smaller than glimmers. So it seems llama.cpp is particularly memory inefficient for the qwen arch. I presume there is an existing issue for this, but I couldn't find one. What's the deal? The extra concurrency makes a big difference in batched performance.

huggingface.co

huggingface.co

A coding-specialised speculative-decoding drafter for Muse Glimmer 30B, fine-tuned on software-engineering traces. It raises mean acceptance length to about 6.9 versus 4.0 for the community DSpark, for about 3.8x over no speculation.

kaitchup

kaitchup

Muse Glimmer 30B quantized with Intel AutoRound at a 3.5-bit target and packed with llm-compressor, tested on vLLM.

These numbers were captured during a real feature implementation task in Next.js and Nest.js (adding a theme switching system across components). The structural predictability of UI/state refactoring is likely why DFlash hit such a high draft acceptance rate (~97%). Here is a quick log analysis and performance summary running Muse-Glimmer-30B (UD- Q6_K_XL) paired with DFlash (Speculative Decoding) via llama.cpp (llama-server + single RTX 5090). -ngl 99 -c 200000 --host 0.0.0.0 --port 8080 --timeout 600 --cache-reuse 256 --parallel 1 --flash-attn on --spec-type draft-dflash --spec-draft-n-max 16 --spec-draft-p-min 0.7 --spec-draft-ngl 99 --cache-type-k q8_0 --cache-type-v q8_0 --no-webui --load-mode none --cache-ram 12192 --temperature 0.8 --top-k 30 --top-p 0.95 --min-p 0.05 --repeat-penalty 1.1 --repeat-last-n 64 --reasoning on --chat-template-kwargs {"enable_thinking":true} Compared to Qwen 3.6 27B: No Chinese language-mixing bugs, no overthinking loops, and concise responses. Its lighter memory footprint at Q6 also freed up more VRAM/RAM for a much larger context size. Metric Measured Value Notes Generation Speed (Peak) 100 – 287 tokens/sec Average ~173 t/s across all ta

brenden7158

brenden7158

An unofficial parody MLX-VLM QLoRA adapter for Muse Glimmer 30B on Apple Silicon.

holaclaw.ai

holaclaw.ai

HolaClaw's tutorial for running Muse Glimmer 30B behind OpenClaw on a Mac, with hardware requirements and llama.cpp and Ollama setup.

Ran the model with quants (Q4) by Unsloth with latest (build from master) llama.cpp server. It takes ~20GB ram running on M5 Pro with 48GB at about 17t/s. Didn't do any reasoning loops/overthinking. Overall, sits below Qwen3.6 27B, wasn't able to get good code (frontend and backend) results. On the positive side, it didn't fail any tool calls. Your opinions/findings? Watch more: https://www.youtube.com/watch?v=_5wKhkUT438

muse glimmer dropped yesterday and mlx-lm couldn't load it yet, so i wrote the text model port and opened a PR. i checked it against meta's own transformers reference before posting, 5 out of 5 next token matches and 0.9965 logit cosine, so it's not just coherent it actually matches the reference. if you want to run glimmer on apple silicon right now the model file is in the PR. https://github.com/ml-explore/mlx-lm/pull/1710

SGLang

@sgl_project

SGLang is honored to provide day-0 support for @AIatMeta's Muse Glimmer. ~230 tok/s on a single RTX 5090 with NVFP4 + DFlash on, and it runs out of the box on @NVIDIAAI RTX PRO 6000, DGX Spark, and Apple Silicon via MLX. Huge thanks to the NVIDIA and Meta teams for the

holaclaw.ai

holaclaw.ai

HolaClaw tested Muse Glimmer on base M3 and M4 Macs: it fits in 24GB of RAM but generates at 4.3 tokens per second.