Muse Glimmer 30B NVFP4 for DGX Spark and RTX 5090

# Meta Muse Glimmer 30B NVFP4 for DGX Spark / RTX 5090 / RTX 6000 PRO

<p align="center">
  <sub>by <a href="https://x.com/MiaAI_lab">Mia'a AI Lab</a></sub>
  <br><br>
  <a href="https://ko-fi.com/Z8Z3SPLOD" target="_blank" rel="noopener noreferrer" style="display:inline-block;margin:0 8px;vertical-align:middle;"><img src="https://storage.ko-fi.com/cdn/kofi6.png?v=6" alt="Buy Me a Coffee at ko-fi.com" height="28" style="height:28px;width:auto;vertical-align:middle;border:0;" /></a>
  <a href="https://x.com/MiaAI_lab" target="_blank" rel="noopener noreferrer" style="display:inline-block;margin:0 8px;vertical-align:middle;"><img src="https://img.shields.io/badge/Follow%20me%20on%20X-000000?style=for-the-badge&logo=x&logoColor=white" alt="Follow Mia on X" height="28" style="height:28px;width:auto;vertical-align:middle;border:0;" /></a>
</p>

Serve **Meta Muse Glimmer 30B** — quantized to **NVFP4**, with the **perception (vision) encoder intact** — on an **NVIDIA DGX Spark (GB10)**, **RTX 5090** or **RTX PRO 6000 Blackwell (sm_120, amd64)** using vLLM, accelerated by **DFlash speculative decoding** and running at up to **256K context**.

A single `start.sh` handles everything: it builds a local venv, installs the vLLM fork with Muse Glimmer support (precompiled kernels, no CUDA build), downloads the weights + DFlash drafter head, and boots an OpenAI-compatible server on port `8888` under the served name `muse-glimmer-30b`. `stop.sh` tears it down cleanly (including orphaned engine processes).

---

## Highlights

- **~19 GB NVFP4 checkpoint** (down from ~56 GB BF16) — the W4A4 build from the llm-compressor / Red Hat team ([`RedHatAI/Muse-Glimmer-30B-NVFP4`](https://huggingface.co/RedHatAI/Muse-Glimmer-30B-NVFP4)): 4-bit float weights **and** activations, `compressed-tensors` `nvfp4-pack-quantized`.
- **Vision intact** — the ViT-G/14 perception tower stays in BF16 and ships in the checkpoint, so image input still works.
- **DFlash speculative decoding** with the official `meta-models/Muse-Glimmer-30B-assistant` drafter head (block size 16, `num_speculative_tokens=16`).
- **256K context** (`--max-model-len 262144`) via RoPE cache sizing patches (details below; native trained context is 131072, beyond that is extrapolation — use with awareness).
- **FP8 KV cache** (`--kv-cache-dtype fp8_e4m3`) — halves KV memory vs BF16; `GPU_MEM_UTIL=0.42` still yields **9.2× conc