Muse Metal

An experimental Rust + Metal inference runtime for Muse models on Apple Silicon, currently supporting Muse Glimmer. The near-term goal is a text-only path that borrows Lily's specialization philosophy while preserving each model's architecture and GGUF quantization layout.

The package and CLI are muse-metal; the Rust crate is muse_metal. Glimmer remains the name of the supported model architecture.

This is a learning project, not a production server. llama.cpp is the correctness and performance oracle while the custom path grows one component at a time.

The latest paired measurements with FP16 prefill and F32 decode/KV ran both
backend orders under the decorrelated schedule on 2026-09-06. Native decode
reached 93.7% / 90.3% / 88.7% of standard llama.cpp after 128 / 1024 /
2048-token prompts in the steady-state order, and 84.6% / 83.2% / 80.9% in
the other, so the 90% decode target is not met: no length clears it in both
orders and 2,048 fails in both. Token sequences, captured logits and response
scores are identical between the engines, so the deficit is throughput only.
See the acceptance results.
F32 prefill remains the default; FP16 prefill is opt-in.

Current state

• The 9.58 GB IQ2_XS model is downloaded to the external SSD and SHA-256 verified.
• A pinned Apple-Clang/Metal llama.cpp build runs the complete 52-layer text model.
• Baseline on this 16 GB M4: ~48 tok/s prefill and ~6.17 tok/s decode.
• Milestone 1 is complete: a strict Rust GGUF loader validates and memory-maps all 731 tensors without copying.
• Scalar IQ2_XS, Q2_K, Q4_K, Q5_K, and Q6_K dequantizers match the pinned llama.cpp oracle bit-for-bit across five original blocks plus 5,120 varied blocks. This is not end-to-end response parity.
• The M4 Metal lab now has F32/BF16 vector and RMSNorm kernels plus direct packed GEMV for every quant format in the checkpoint, validated against real model weights.
• Scale-aware GPU tests cover 480 combinations; a bounded streaming check covers all 202,048 output-head rows at about 41 MiB peak process RSS.
• Tiny F32 end-to-end parity now passes against pinned llama.cpp: every layer,
  logits, bounded local/global KV, reset/chunking, and a 16-token greedy continuation.
  The fixture is 5.52 MiB; the recorded sweep reported 78.28 MiB peak process RSS.
• A