Local LLM benchmark: Muse-Glimmer-30B vs Qwen3.6-27B vs Qwen3.8-27B

A small, auditable test bed for comparing locally served models under identical conditions: same questions, same prompts, same sampling, same token budget, with every prompt, reasoning trace, answer and verdict saved to disk.

Report site: https://nd-dac-dome.github.io/local-llm-benchmark/ — the findings, navigable, with a question explorer. The same content as text: results.md.

Findings in short

• Given enough room to finish thinking (32k tokens per answer), the three models are equivalent: MMLU-Pro 82 / 82 / 80% (Muse / Qwen3.6 / Qwen3.8), GPQA Diamond 80 vs 80% (Muse vs Qwen3.8).
• With a tighter budget (12k) Muse is ahead (MMLU-Pro 82% vs 73–77%), because the Qwens run out of tokens before answering, not because they reason worse. On LiveCodeBench Qwen3.8 does not finish 38% of the problems even at 32k (Muse 79%, Qwen3.8 60%).
• The Qwens use 2–3× the tokens per answer to reach the same score. Which model is "better" depends on how long an answer may take in the intended use.
• Quantization, speculative decoding (MTP / DFlash) and the machine (A6000 vs DGX Spark) do not change the scores; they change speed. Same model and recipe: 50.8 tok/s on the A6000, 20.3 on the Spark.

Datasets and grading: benchmarks.md. Vendor-published SWE-bench numbers (not reproduced here): swe-bench-cards.md.

Layout

File	Purpose
bench.py	The harness. Talks to any OpenAI-compatible endpoint (llama.cpp, vLLM, Ollama). Tasks: gsm8k, mmlu_pro, humaneval, gpqa_diamond, livecodebench, speed (decode tok/s, TTFT, prefill tok/s). Resumable; errors logged; aborts after 5 consecutive request errors.
compare.py	Prints a markdown table across every run under results/, with margins of error.
rescore_humaneval.py, rescore_livecodebench.py	Re-grade saved answers with the current scorer, without regenerating (used after scorer fixes; the LiveCodeBench one also grades under the judge environment).
run_all.sh	Run 1 end to end: serve Muse then Qwen3.6 (GGUF, llama.cpp) and run all tasks.
speed_only.sh	Re-measure only speed/prefill for an existing llama.cpp run, keeping its scores.
serve_vllm_nvfp4.sh	Serve an NVFP4 checkpoint with vLLM on an A6000; parameterized by env vars (mod