smol-muse-glimmer

A 51M-parameter Muse Glimmer architecture study for TinyStories

Gated GQA · 3 Local : 1 Global · Local RoPE · Global NoPE · SwiGLU · Softcapped logits

Quick start · Architecture · Dashboard · Verification

---

smol-muse-glimmer is a training-oriented, text-only miniature of the
language backbone released in Meta Superintelligence Lab's
Muse Glimmer 30B model card.
It scales the defining language-model structure down to 51,155,328 trainable
parameters and provides a complete TinyStories trainer for one local CUDA
device.

The included run was stopped intentionally at step 5,000. It reached a
validation cross-entropy of 1.8127; best.pt, latest.pt, the full JSONL
history, loss curve, final sample, and dashboard state are retained under
checkpoints/muse-glimmer-50m/. No result is invented or borrowed from another
model.

[!IMPORTANT]
python3 train.py --dry-run validates the complete run plan and exits before
CUDA or model allocation. The normal command starts training. This project
never launches the training job on its own.

What was found in the upstream release

The model card and the
official Transformers implementation
agree on the language backbone:

• a dense causal Transformer with 52 decoder layers;
• a repeating local, local, local, global attention pattern;
• a 2,048-token sliding window on local layers and full causal attention on
  global layers;
• RoPE with theta 500,000 on local layers and NoPE on global layers;