Meta Muse Glimmer 30B Runpod Serverless worker

Public reference implementation for serving Meta Muse Glimmer 30B through a
Runpod Serverless load-balancing endpoint. This repository contains worker
software and deployment documentation only; it does not contain model
weights, Runpod credentials, registry credentials, or private lab data.

This bundle is a CUDA worker for Meta's official Muse Glimmer GGUF release. It
runs llama-server directly and supports both Runpod Serverless
load-balancing workers and manually managed Pods. The exposed port is a real
OpenAI-compatible HTTP API rather than a queue wrapper around a custom JSON
handler.

What is pinned

• A CUDA-enabled llama-server runtime. The published base image is pinned by
  digest in Dockerfile; rebuilders can replace BASE_IMAGE with a compatible
  CUDA/llama.cpp image. The repository does not assume one GPU vendor SKU.
• Hugging Face repo: meta-models/Muse-Glimmer-30B-GGUF.
• Main model: Muse-Glimmer-30B-KQuant-Dynamic-Q4_K_XL.gguf.
• Vision projector: mmproj-Muse-Glimmer-30B-Q4_K_M.gguf.
• DFlash drafter: dflash-Muse-Glimmer-30B-Q4_K_M.gguf.

The startup script downloads all three files to
/runpod-volume/models/muse-glimmer-30b and reuses them on later worker
starts. A file lock prevents concurrent first-downloads from racing on the
same volume. The worker fails closed if the network volume is absent unless
ALLOW_EPHEMERAL_MODEL_CACHE=1 is explicitly set for a disposable test.

API behavior

llama-server provides:

• GET /health
• GET /v1/models
• POST /v1/chat/completions, including SSE when stream=true
• OpenAI-style tools, tool_choice, and parsed tool_calls
• multimodal message content using OpenAI image blocks
• GET /metrics for Prometheus-compatible metrics

The server is started with --jinja, which is required for the model's
embedded chat template and tool-call parsing. Reasoning is returned separately
as reasoning_content when the model/template emits it. Muse Glimmer's
reasoning channel is not disabled by this configuration; REASONING_BUDGET
and REASONING_STRENGTH control its cost.

The worker does not enable llama.cpp's built-in she