№ 0074GitHub
Muse Glimmer 30B on a single AMD MI300X
A reproducible setup serving Muse Glimmer 30B on one AMD MI300X with a custom 11 GB vLLM 0.28.0 ROCm image, exposed as an authenticated OpenAI-compatible endpoint and driven from the pi CLI.
Muse-Glimmer-30B on AMD MI300X with vLLM 0.28.0
Reproducible setup for serving meta-models/Muse-Glimmer-30B on a single AMD
Instinct MI300X, exposed as an OpenAI compatible HTTPS endpoint with API key
auth, and driven from the pi CLI on a local machine.
Everything used here is free and open source.
Why this repo builds its own vLLM image
AMD's prebuilt rocm/vllm images lag upstream. The newest published tag ships
vLLM 0.23.0, which does not know MuseGlimmerForConditionalGeneration and
has no muse_glimmer parsers, so it cannot load this model at all.
vLLM publishes exactly one ROCm wheel, vllm-0.28.0+rocm723-cp312, which is
Python 3.12 and ROCm 7.2.3 only. This repo therefore builds a small image on
a matching Python 3.12 / ROCm 7.2.3 base. The result is 11 GB, versus 68.9 GB
for AMD's image.
Verified environment
Built and tested end to end on:
| Component | Version |
|---|---|
| OS | Ubuntu 26.04 LTS |
| Kernel | 7.0.0-27-generic |
| GPU | AMD Instinct MI300X VF, gfx942, 191.7 GiB |
| Docker | 29.7.2 |
| Base image | rocm/dev-ubuntu-24.04:7.2.3 |
| Built image | vllm-rocm:0.28.0-gfx942 (11 GB) |
| vLLM | 0.28.0+rocm723 |
| torch | 2.12.0+git6bbd260 (ROCm) |
| transformers | 5.16.1 |
| Caddy | 2.11.4 |
| pi | 0.84.3 |
Disk needed: about 130 GB (11 GB image, 56 GB weights, plus build cache).
RAM: 64 GB or more. Weights stream through page cache on load.
What the model is
MuseGlimmerForConditionalGeneration, model_type: muse_glimmer, Apache 2.0.
- Dense, 52 layers, hidden 6656, 32 attention heads, 2 KV heads (GQA).
- Hybrid sliding attention: 39 sliding_attention plus 13 full_attention
layers. This makes KV cache very cheap. 37.47 GiB of cache holds
2,125,299 tokens. - Multimodal, accepts images.
- 131072 token context, vocab 202048.
- Tool calls use the ATEM protocol:
<atem:invoke name="...">inside<atem:function_calls>, with<|start|>/<|message|>/<|eot|>channel
tokens and a separate reasoning channel. This is not Hermes JSON and not
Qwen XML. The parser must bemuse_glimmerfor both tools and reasoning.
Quick start
git clone git@github.com:alpharomercoma/muse-glimmer-30b-mi300x.git
cd muse-glimmer-30b-mi300x
sudo ./scripts/00-preflight.sh # check the machine
sudo ./scripts/01-gpu-firmware.sh # only if preflight fails on the GPU, then
