muse-glimmer-dspark

Meta's Muse-Glimmer-30B with a DSpark speculative drafter instead of
the shipped DFlash drafter.

This is a fork of deepseek-ai/DeepSpec
(MIT) that adds a MuseGlimmerDSparkModel target-model family, trained against
meta-models/Muse-Glimmer-30B using DeepSeek's DSpark recipe (see
deepseek-ai/DeepSeek-V4-Flash-DSpark).

The DSpark module shipped inside DeepSeek-V4-Flash-DSpark cannot be lifted
onto Muse-Glimmer — its weights are coupled to DeepSeek-V4's vocab, hidden
size, and layer count. This repo trains a fresh DSpark drafter against
Muse-Glimmer instead. See DESIGN.md for details.

Status (early validation)

End-to-end pipeline works on GPU: data → regen → target cache → train →
checkpoint → speculative eval. Highlights from the first run (4591-sample
slice, ~400 steps; see RESULTS.md):

• Drafter trains cleanly; loss 13.2 → 2.39 over 400 steps
• Acceptance length ~1.15 and first-token accept rate ~13% — above the
  no-speculation baseline (1.0), but far from production (1 partial epoch)
• Confidence head is learning: acceptance-prediction AUC 0.75–0.85 across
  gsm8k / math500 / aime25 / humaneval

Artifacts (checkpoints, regen data, eval results) are mirrored on Hugging Face:
say4n/muse-glimmer-dspark-10k.

Setup

On a fresh GPU box (Linux/CUDA):

bash scripts/setup.sh     # installs uv + rust, syncs main + sglang envs

This handles the two gotchas: sglang must come from git main (PyPI predates
the muse_glimmer backend) and needs a Rust toolchain to build its custom ops.

Training requires a CUDA build of torch (the default PyPI wheel works on Linux;
adjust if your box needs a different wheel). flex_attention (triton) is used
for drafter training.

Pipeline

# 0. Log in to HF (Muse-Glimmer is a gated repo)
huggingface-cli login

# 1. Download + split the dataset (CPU-only)
uv run python scripts/data/download_and_split.py \
  --dataset-name mlabonne/open-perfectblend \
  --train-output-path train_datasets/perfectblend_train.jsonl \
  --test-output-dir eval_datasets \
  --skip-existing

# 2. Start sglang servers (one per GPU; keep this running in another terminal)
NUM_WORKERS=1 uv r