muse-glimmer-bench

English | 日本語

A benchmark for deciding, on our own GPU, whether Meta's Muse Glimmer 30B is
usable as an OpenCode backend — and whether it changes the answer that
deepseek-v4-flash-bench
arrived at.

The tasks and evaluation axes come unmodified from
letusfly85/coding-agent-bench
(cab), by way of deepseek-v4-flash-bench. bench/tasks/ is byte-identical to
both, so the numbers line up directly against the existing Qwen3.6-27B,
Qwen3-Coder-Next 80B and DeepSeek-V4-Flash results.

Status: first measurements are in (2026-08-11, RTX 5090 32GB,
llama.cpp b10365) —
docs/reports/2026-08-11-first-measurements.md.
Headline: 18/18 agent episodes with zero malformed tool calls, the full
128K window serving four slots wide on one 32GB card, and DFlash worth +88%.
That run also found the reasoning-strength knob was not connected — numbers
produced by this harness before 2026-08-11 were pinned near high regardless
of the configured strength. See docs/SETUP.md §0.

Still unmeasured: the vLLM column, the 17gb quant, and the OpenCode axis.

The question

deepseek-v4-flash-bench asked whether 192 GiB of GPU buys anything over one
RTX 5090 running a 27B, and answered: not decisively. DeepSeek-V4-Flash
(284B MoE) beat Qwen3.6-27B on Rust and on repairing its own compile errors,
lost slightly on Python, and was statistically indistinguishable on the agent
loop — 18/18 episodes completed for both, zero malformed tool calls for both.
The 10x hardware cost did not show up on the axis that matters for agent use.

That result has an obvious follow-up. Qwen3.6-27B is a general model that
happens to code. Muse Glimmer is a 30B built for agentic work, on the same
single-consumer-GPU footprint. So:

Does the small-model side of that comparison get materially stronger when the
small model is purpose-built for agents?

If Glimmer also completes 18/18, the "cheap local box is enough" conclusion
hardens considerably. If it beats DeepSeek on the agent axis, that is a
different and more interesting story. Either way the answer is worth the GPU
hours.

Subject

Model	meta-models/Muse-Glimmer-30B