№ 0412GitHub
mlx-vlm support for Muse Glimmer
An architecture port adding the muse_glimmer model class (vision tower, language model, projector and image processor) to mlx-vlm, so any Muse Glimmer checkpoint runs multimodally on Apple Silicon.
# mlx-vlm-muse-glimmer
MLX / [mlx-vlm](https://github.com/Blaizzy/mlx-vlm) model support for **Muse-Glimmer**,
a multimodal (vision + language) model. This is a pure architecture port — it loads and
runs **any** Muse-Glimmer checkpoint on Apple Silicon, including Meta's official release
and any fine-tuned or derived weights.
It ships the model class mlx-vlm needs (`muse_glimmer`): the Qwen2.5-VL-style vision
tower, the language model, the projector, and the image processor.
## Demo
One local model — recognition, fine-grained ID, landmarks, in-the-wild OCR, and chart
reading. Every answer below is a real, unedited output running on-device via MLX:
[](https://www.youtube.com/watch?v=5fs_FfkCaDA)
▶ **[Watch the demo on YouTube](https://www.youtube.com/watch?v=5fs_FfkCaDA)** (click the poster above)
- **Golden retriever** — recognition
- **Two men + a red Jaguar** — scene understanding
- **Badge blurred → still "Jaguar F-Type"** — fine-grained ID from the duck-tail spoiler alone
- **Taj Mahal** — landmark + geography (Agra, India)
- **"1501 BROADWAY" neon** — reads text in the wild
- **Bar chart** — reads data (Q4 highest at $73M, trend up)
## What's here
```
muse_glimmer/
config.py # VisionConfig / TextConfig / ModelConfig
vision.py # 50-layer ViT: 2D-RoPE, window/full attention,
# interpolated learned pos-emb, 2x2 pixel-shuffle merge
language.py # gated attention, qk-norm, centered RMSNorm,
# logit soft-cap, sliding/full attention
muse_glimmer.py # top-level: encode image -> adapter -> projection ->
# splice at the image token
processing_muse_glimmer.py # image processor + chat processor (smart-resize,
# temporal patchify, <|patch|> expansion)
__init__.py
```
## Install
Drop the package into your mlx-vlm install so `mlx_vlm.load` can find it:
```bash
pip install mlx-vlm
python - <<'PY'
import os, shutil, mlx_vlm
dst = os.path.join(os.path.dirname(mlx_vlm.__file__), "models", "muse_glimmer")
shutil.rmtree(dst, ignore_errors=True)
shutil.copytree("muse_glimmer", dst)
print("installed ->", dst)
PY
```
## Usage
```python
from mlx_vlm import load, generate
from mlx_vlm.prompt


ChatForm
Tgmlabs