mlx-vlm-muse-glimmer

MLX / mlx-vlm model support for Muse-Glimmer,
a multimodal (vision + language) model. This is a pure architecture port — it loads and
runs any Muse-Glimmer checkpoint on Apple Silicon, including Meta's official release
and any fine-tuned or derived weights.

It ships the model class mlx-vlm needs (muse_glimmer): the Qwen2.5-VL-style vision
tower, the language model, the projector, and the image processor.

Demo

One local model — recognition, fine-grained ID, landmarks, in-the-wild OCR, and chart
reading. Every answer below is a real, unedited output running on-device via MLX:

▶ Watch the demo on YouTube (click the poster above)

• Golden retriever — recognition
• Two men + a red Jaguar — scene understanding
• Badge blurred → still "Jaguar F-Type" — fine-grained ID from the duck-tail spoiler alone
• Taj Mahal — landmark + geography (Agra, India)
• "1501 BROADWAY" neon — reads text in the wild
• Bar chart — reads data (Q4 highest at $73M, trend up)

What's here

muse_glimmer/
  config.py                    # VisionConfig / TextConfig / ModelConfig
  vision.py                    # 50-layer ViT: 2D-RoPE, window/full attention,
                               #   interpolated learned pos-emb, 2x2 pixel-shuffle merge
  language.py                  # gated attention, qk-norm, centered RMSNorm,
                               #   logit soft-cap, sliding/full attention
  muse_glimmer.py              # top-level: encode image -> adapter -> projection ->
                               #   splice at the image token
  processing_muse_glimmer.py   # image processor + chat processor (smart-resize,
                               #   temporal patchify, <|patch|> expansion)
  __init__.py

Install

Drop the package into your mlx-vlm install so mlx_vlm.load can find it:

pip install mlx-vlm
python - <<'PY'
import os, shutil, mlx_vlm
dst = os.path.join(os.path.dirname(mlx_vlm.__file__), "models", "muse_glimmer")
shutil.rmtree(dst, ignore_errors=True)
shutil.copytree("muse_glimmer", dst)
print("installed ->", dst)
PY

Usage

from mlx_vlm import load, generate
from mlx_vlm.prompt