smol-muse: 51M Muse Glimmer architecture study

<div align="center">

# smol-muse-glimmer

### A 51M-parameter Muse Glimmer architecture study for TinyStories

[![Parameters](https://img.shields.io/badge/parameters-51.16M-e7bd58?style=flat-square)](#model-card)
[![Attention](https://img.shields.io/badge/attention-18%20local%20%2B%206%20global-65c4b7?style=flat-square)](#architecture)
[![PyTorch](https://img.shields.io/badge/PyTorch-BF16-ee4c2c?style=flat-square&logo=pytorch&logoColor=white)](https://pytorch.org/)
[![Dataset](https://img.shields.io/badge/dataset-TinyStories-557f76?style=flat-square)](https://huggingface.co/datasets/roneneldan/TinyStories)
[![Tests](https://img.shields.io/badge/tests-8%20passing-65c4b7?style=flat-square)](#verification)

**Gated GQA · 3 Local : 1 Global · Local RoPE · Global NoPE · SwiGLU · Softcapped logits**

[Quick start](#quick-start) · [Architecture](#architecture) · [Dashboard](#live-training-dashboard) · [Verification](#verification)

</div>

---

`smol-muse-glimmer` is a training-oriented, text-only miniature of the
language backbone released in Meta Superintelligence Lab's
[Muse Glimmer 30B model card](https://huggingface.co/meta-models/Muse-Glimmer-30B).
It scales the defining language-model structure down to **51,155,328 trainable
parameters** and provides a complete TinyStories trainer for one local CUDA
device.

The included run was stopped intentionally at **step 5,000**. It reached a
validation cross-entropy of **1.8127**; `best.pt`, `latest.pt`, the full JSONL
history, loss curve, final sample, and dashboard state are retained under
`checkpoints/muse-glimmer-50m/`. No result is invented or borrowed from another
model.

> [!IMPORTANT]
> `python3 train.py --dry-run` validates the complete run plan and exits before
> CUDA or model allocation. The normal command starts training. This project
> never launches the training job on its own.

## What was found in the upstream release

The model card and the
[official Transformers implementation](https://github.com/huggingface/transformers/blob/main/src/transformers/models/muse_glimmer/modular_muse_glimmer.py)
agree on the language backbone:

- a dense causal Transformer with 52 decoder layers;
- a repeating `local, local, local, global` attention pattern;
- a 2,048-token sliding window on local layers and full causal attention on
  global layers;
- RoPE with theta 500,000 on local layers and NoPE on global layers;
-