Local LLM benchmark: Glimmer vs Qwen

# Local LLM benchmark: Muse-Glimmer-30B vs Qwen3.6-27B vs Qwen3.8-27B

A small, auditable test bed for comparing locally served models under identical conditions: same questions, same prompts, same sampling, same token budget, with every prompt, reasoning trace, answer and verdict saved to disk.

**Report site:** https://nd-dac-dome.github.io/local-llm-benchmark/ — the findings, navigable, with a question explorer. The same content as text: [`results.md`](results.md).

## Findings in short

- Given enough room to finish thinking (32k tokens per answer), the three models are equivalent: MMLU-Pro 82 / 82 / 80% (Muse / Qwen3.6 / Qwen3.8), GPQA Diamond 80 vs 80% (Muse vs Qwen3.8).
- With a tighter budget (12k) Muse is ahead (MMLU-Pro 82% vs 73–77%), because the Qwens run out of tokens before answering, not because they reason worse. On LiveCodeBench Qwen3.8 does not finish 38% of the problems even at 32k (Muse 79%, Qwen3.8 60%).
- The Qwens use 2–3× the tokens per answer to reach the same score. Which model is "better" depends on how long an answer may take in the intended use.
- Quantization, speculative decoding (MTP / DFlash) and the machine (A6000 vs DGX Spark) do not change the scores; they change speed. Same model and recipe: 50.8 tok/s on the A6000, 20.3 on the Spark.

Datasets and grading: [`benchmarks.md`](benchmarks.md). Vendor-published SWE-bench numbers (not reproduced here): [`swe-bench-cards.md`](swe-bench-cards.md).

## Layout

| File | Purpose |
|---|---|
| `bench.py` | The harness. Talks to any OpenAI-compatible endpoint (llama.cpp, vLLM, Ollama). Tasks: `gsm8k`, `mmlu_pro`, `humaneval`, `gpqa_diamond`, `livecodebench`, `speed` (decode tok/s, TTFT, prefill tok/s). Resumable; errors logged; aborts after 5 consecutive request errors. |
| `compare.py` | Prints a markdown table across every run under `results/`, with margins of error. |
| `rescore_humaneval.py`, `rescore_livecodebench.py` | Re-grade saved answers with the current scorer, without regenerating (used after scorer fixes; the LiveCodeBench one also grades under the judge environment). |
| `run_all.sh` | Run 1 end to end: serve Muse then Qwen3.6 (GGUF, llama.cpp) and run all tasks. |
| `speed_only.sh` | Re-measure only speed/prefill for an existing llama.cpp run, keeping its scores. |
| `serve_vllm_nvfp4.sh` | Serve an NVFP4 checkpoint with vLLM on an A6000; parameterized by env vars (mod