Entries matching “gguf”

Unsloth AI

@UnslothAI

2-bit Muse Glimmer GGUF managed to call 100+ tools on just 14GB RAM. 🔥 Muse Glimmer did a complete repo bug hunt for 5 mins nonstop with: evidence, repro, fix, tests and a PR writeup. Run and train it in Unsloth. GitHub repo: github.com/unslothai/unsl…

Meta's own GGUF conversions of Muse Glimmer 30B for llama.cpp.

A suite of 27 GGUF quantizations of an abliterated Muse Glimmer 30B produced with Heretic v1.4.0.

Unsloth Dynamic 2.0 GGUF quants of Muse Glimmer 30B with a companion run guide and thinking toggles.

GGUF conversions of Inco AI's DFlash 2 draft model for Muse Glimmer 30B, for speculative decoding in llama.cpp.

GGUF builds of Muse Glimmer 30B all cut from the same BF16 source weights, including full-precision files.

Simon Willison

@simonw

Muse Glimmer, the new 30B model, is available on Hugging Face right now - here's the GGUF version: huggingface.co/meta-models/Mu…

Unsloth AI

@UnslothAI

Meta releases Muse Glimmer, a new 30B open model that runs on 18GB RAM. Muse Glimmer is Apache 2.0 licensed, supports vision and is the strongest agentic model for its size. Run and train the model via Unsloth. GGUF: huggingface.co/unsloth/Muse-G… Guide: unsloth.ai/docs/models/mu…

Abliterated Muse Glimmer 30B GGUF quant ladder, updated with a 1.63 GB abliterated DFlash drafter and a 1.40 GB multimodal projector.

Hey Folks, I've been making quants for a while - recently I took a short break to get into hardcore research (submitted my first EMNLP paper during it!). Along the way, I built up a little arsenal of quant-optim techniques: everything from novel, paper-pending tricks to some genuinely sick tensor-mapping algos. I threw some of the secret sauce into the newly released Muse Glimmer 30B (META IS BACK!) and compared it to several OGs. I'm honestly shocked by how it never loses to any quant out there in every single VRAM class! One of the coolest ones is my Q8 quant, it is smaller than UD-Q8_K_XL and 21% closer to BF16. Full methodology is on the card - eval setup, CIs, held-out slices, the lot. Happy to answer questions in the comments. Model: https://huggingface.co/AaryanK/Muse-Glimmer-30B-GGUF I still had headroom left but ran out of compute credits :( Being a solo undergrad sophomore, I can't exactly spend H100 money that often, which is why the "hopefully" in the title :) I'm looking for internships in AI agent orchestration and model inference. If this work looks relevant to your team: linkedin.com/in/theaaryankapoor I plan on doing a write-up soon to describe some of the

A quantization-aware-trained Q4_0 GGUF of Muse Glimmer 30B for llama.cpp. On held-out tokens it measures closer to BF16 than Meta's official Q4_K_M: 0.0213 vs 0.0228 KL and 95.9% vs 95.6% top-token agreement.

Unsloth's docs page for running Meta's Muse Glimmer 30B locally with its GGUF quants.

Kingy AI explains Muse Glimmer's benchmarks, 24-64 GB hardware needs, GGUF setup, pricing, runtimes and risks.

@mitchallen

@mitchallen

A Makefile that downloads the three Muse Glimmer 30B GGUFs and builds and serves llama.cpp on Apple Silicon, automating a scriptable.com walkthrough.

@cezaronx

@cezaronx

A reference CUDA worker that serves Meta's official Muse Glimmer 30B GGUF through llama-server on Runpod Serverless load-balancing endpoints or manual Pods, exposing a real OpenAI-compatible API.

@lobanov

@lobanov

An effort to make Muse Glimmer 30B actually use a 512k-token context (4x native) as a ~17GB GGUF in 32GB VRAM, trained on DGX Spark and evaluated with RULER-style retrieval tests.

@johnhalloran321

@johnhalloran321

Scripts that run Muse-Glimmer-30B with vision and tool calling on a 32GB M2 MacBook Pro via llama.cpp and Metal, without admin or sudo access, pulling Meta's official GGUFs.

@Abhishektenneti

@Abhishektenneti

A reproducible record of running the 17GB Muse Glimmer 30B GGUF with llama.cpp and Metal on a 24GB M4 Pro MacBook Pro, with notes on mistakes, fixes and how local inference works.

This started as a failure. I had cut 6.34% of Meta's Muse-Glimmer-30B (the FFN sublayers of four layers, the next FFN after each cut retrained against the parent) and the healed model passed my fidelity bar at Q8_0. At Q4_K it failed by 0.006 KLD, and the arithmetic said why: the surgery's cost plus the ordinary Q4_K cost adds up to just over the bar, and three months of levers on the surgery side could not close a gap that small. So I attacked the other term. In a fixed GGUF the integer codes are frozen, but every quantised block still carries one or two fp16 scales, and the decoded weight is linear in them. That means the scales can be trained end to end against the parent's next-token distribution on the student's own forward pass, without touching the codec, the format, the byte length or the offsets. On the surgical model it worked: 0.05615 fail to 0.04949 pass on 45,056 held-out positions, and the preregistered control (the same recovery on the uncut parent at Q4_K) showed the two costs are not additive once the scales are trained; recovery took back part of the surgery error too. That file and the whole study are on my Hugging Face page. Then the obvious question: does it

atomic.chat

@atomic_chat_hq

Run Meta's new Muse Glimmer 30B♾locally with 16GB VRAM! We ship our own GGUF quants. AD-IQ3_XXS does 62 tokens/s on a single RTX 4080 with vision and DFlash, and picks the same next token as the BF16 original 90% of the time! Run the model via Atomic Chat

Agent Workflow Lab runs the Q8 GGUF of Muse Glimmer 30B through llama.cpp on an RTX 4090 plus 3x RTX 3090, measures DFlash speedups and a 120K-token retrieval probe, then has it build a Three.js browser FPS with no human edits.

@TrevorS

@TrevorS

Refusal-direction ablation on Muse-Glimmer-30B that cut refusals from 128/150 to 3/150, adding an agentic-safety evaluation and publishing bf16 and GGUF uncensored weights.

LM Studio community highlight GGUF quants of Muse Glimmer 30B.

Fahd Mirza installs Muse Glimmer in GGUF format and tests it with DFlash speculative decoding for faster local inference.

GGUF quantizations of the Inco AI DFlash2 drafter for Muse Glimmer 30B, for use with the BeeLlama.cpp llama.cpp fork.

Solo-built GGUF line of Muse Glimmer 30B with custom calibration, per-tensor allocations and an eval harness behind every reported number.

ROCmFP4 GGUF of Muse Glimmer 30B with DFlash for AMD Strix Halo, requiring a ROCmFPX llama.cpp fork that adds the muse-glimmer architecture.

kwindla

@kwindla

Exciting to see Meta releasing new open weights this week. Meta trained the new 30B dense model Muse Glimmer with "agentic" use cases in mind. This generally means task-oriented, multi-turn, and heavy use of tool calling. I ran Muse Glimmer in the GGUF quant through a bunch of

A 4-byte GGUF header edit that raises Muse Glimmer 30B's 131,072 context limit to 262K. The author shows YaRN scaling only adds prefill cost on this architecture, and verified 262K with 8 slots on one RTX 4090.

llama.cpp imatrix quantizations of Muse Glimmer 30B with image support via mmproj and MTP/DFlash notes.

ROCmFP4 and ROCmFP8 builds of Muse Glimmer 30B and its drafter, targeted and tested on AMD Strix Halo (gfx1151).

Xuan-Son Nguyen

@ngxson

We are happy to announce that Muse Glimmer is day-0 supported on llama.cpp. Meta also provides an official GGUF quant:

@47thtechcorner

@47thtechcorner

A YouTube tutorial companion repo where Muse Glimmer 30B (a 2-bit GGUF, run offline) looks at screenshots of badly designed web pages and generates and self-repairs modern replacement code.

松xR

@matsu_vr

muse glimmer 30Bを公式の17GBに収まるよバージョンのggufでLM Studioで動かしてみました。写真の批評もバッチリしてくれる。これは僕が撮った写真なのでネットにないのでちゃんと写真を見ているはず。コンテクスト長64000にして、VRAM20GB以内に収まっているので、かなり実用的かもしれない！

@dangkhoa2016

@dangkhoa2016

A self-hosted llama.cpp serving stack that runs Muse-Glimmer-30B GGUF with DFlash2 speculative decoding on Kaggle's NVIDIA T4 x2, exposed through an authenticated OpenAI-compatible gateway.

@sxuff

@sxuff

A pinned llama.cpp recipe for Muse Glimmer 30B on one NVIDIA GB10 that verifies Meta's official GGUFs by SHA-256 and reproduces text, tool-call, coding, vision and throughput checks.

Been building this for a few months, mostly for myself, and it just got a proper release so figured I'd post it. It's a native GGUF inference runtime with OpenAI/Anthropic-compatible APIs and a chat UI. The whole point is one consumer NVIDIA card + lots of RAM: MoE models that don't fit in VRAM run their experts on the CPU, or split with a hot set cached on the card. It figures out what fits at startup instead of you guessing offload layer counts. Runs Qwen 3.x dense and MoE (incl. Qwen3.8-Flash-Next), DeepSeek-V4-Flash, Ling 3.0, K2-Horizon, Gemma 4, Laguna, Muse Glimmer. Image input via mmproj on the Qwen models. Also does Z-Image-Turbo image gen next to a chat model on the same card. Numbers from my laptop (5070 Ti 12 GB, 60 GB RAM): - Qwen3.8-Flash-Next IQ1_S: ~35 tok/s decode, ~475 tok/s prefill - Qwen3.8-27B IQ2_XXS: ~40 tok/s - DeepSeek-V4-Flash: 6-7 tok/s (that's basically the DRAM bandwidth limit) - Z-Image 1024x1024 in ~15 s with a 35B loaded beside it Stuff I think is neat: - Kernels are compiled at runtime by NVRTC, so no CUDA toolkit in the wheel and no nvcc. Same kernel source compiles as plain C++ for a CPU-only backend. - KV cache in f16 / q8_0 / TurboQu

Setup the latest (master) version of llama.cpp server with the guide and the official GGUF weights by Meta AI: https://huggingface.co/meta-models/Muse-Glimmer-30B-GGUF and connected the Hermes Agent to the llama.cpp endpoint. Getting about 22t/s (+3-4t/s) on M5 Pro, using ~24GB including the drafter (provided by Meta). The model did correct tool calls and actually did some useful work inside the Hermes Agent. Moreover, the resulting coding task/project works, which was not the case when running the model with OpenCode. Watch more: https://www.youtube.com/watch?v=cmENEolUtM4

Picked up China version of the Mi50 (Radeon VII) 16GB VRAM GPU for about $135. Ran it using llama.cpp Ubuntu Vulkan prebuilt binary build: b29c606e2 (10964). Used a Power Limit or 220/190 watts on the GPUs. Dual Radeon 32GB Vram and 64GB DDR4 System Dual Radeon RX 7900 GRE and Radeon VII 32gb VRAM GGUF Models: • Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf • Accio-Lab_occamy-1.0-Q5_K_S.gguf • Laguna-XS-2.1-APEX-I-Balanced.gguf • NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Q5_K_M.gguf • Gemma-4-31B-it-Q6_K.gguf • Ateron_Gemma-4-MoonGem-31B-Q5_K_M.gguf • Qwen3-Coder-30B-A3B-Instruct-UD-Q6_K_XL.gguf • Qwen3-VL-30B-A3B-Thinking-UD-Q6_K_XL.gguf • Qwen3-Coder-30B-A3B-Instruct-UD-Q5_K_XL.gguf • North-Mini-Code-1.0-MXFP4_MOE.gguf • GLM-4.7-Flash-Uncen-Hrt-NEO-CODE-MAX-imat-D_AU-Q6_K.gguf • Muse-Glimmer-30B-UD-Q6_K_XL.gguf • Huihui-Qwen3.8-27B-abliterated-UD-Q6_K_XL.gguf • Qwen3.8-27B-Q6_K.gguf • Qwen3.8-27B-OBLITERATED-Q5_K_M.gguf • Medgemma-27b-it-UD-Q6_K_XL.gguf • Gemma-4-26B-A4B-it-UD-Q6_K_XL.gguf • Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf • GPT-OSS-20b-abliterated.i1-Q6_K.gguf Sorted by params then size model size params pp512 tg128 qwen35moe 35B.A3B Q5_K - Medium 24.76 GiB 3

Heeeey all! I just completed some fun tests with Muse Glimmer, I thought I'd let you know. In fact, the summary below was written by Muse itself! I ran a 2× DGX Spark cluster and got Meta's day-old Muse Glimmer 30B running the day after release — then pushed its context from the trained 131K all the way to 1M with YaRN, verifying retrieval at every rung. Sharing config + results since the "131,072+" hint in the model card turned out to be very real. Setup • Hardware: 2× NVIDIA DGX Spark (GB10, 128 GB unified each, ~273 GB/s), ConnectX-7 direct link between them • Engine: llama.cpp master (day-1 muse_glimmer support), built from source with CUDA sm_121 + GGML_RPC • Model: official Muse-Glimmer-30B-GGUF K-Quant-Dynamic (~18.3 GiB) + official mmproj (vision) + official DFlash drafter • Spec decode: --spec-type draft-dflash --spec-draft-n-max 15 (block-diffusion drafter) • Context extension: --rope-scaling yarn --rope-scale <2/4/8> --yarn-orig-ctx 131072 plus --override-kv muse-glimmer.context_length=int:<N> (llama.cpp caps at trained length otherwise) • Yes, we also ran it split across both Sparks with llama.cpp RPC — no reason beyond liking to cluster things for fun. Our daily dri