Benchmarks & research

@metacpp

@metacpp

Hand-picked list of tools, extensions, skills, integrations and learning resources for Meta's Muse Code.

I have a classic test for local LLM's. I asked for 8 ball pool game with only one HTML file and Muse Glimmer spend 21k Token(I m using full context so 128k) and only created a 220 lines of HTML and said its done. With my experience its not even close to Qwen 3.6 27B and we are waiting for Qwen 3.8 27B already. What is your toughts about this model. I was so hopeful until this test.

I've been migrating my workflow from ChatGPT Work / Codex over to Meta's Muse, and I've been auditing the harness as I go — reading cron files, checking diffs, mapping what the system actually does versus what it claims to do. It's a documentation gap. What the docs describe Meta's documentation is written for a normal user. It talks about the Ideas tab, the Goals tab, the Feed, and "background jobs that help Muse improve over time." Everything is described in terms of what it does for you. There is no architectural documentation. Nothing about how any of it is built. What is actually on disk Meanwhile, the home directory contains a fully legible agent architecture, just sitting there(these are just SOME examples, the system file structure is HUGE): • ~/dreams/alignment/ — a nightly job that regenerates an alignment synthesis: a written portrait of the user, how to handle them, current frictions, relationship guidance. • ~/workspace/objectives/goals/STUDYING.md — a daily learning-state projection with mastery bands, next-review dates, and retrieval prompts, plus a file of inferred goal leads the system guessed from behavior (with confidence levels and what would confirm or re

Alexandr Wang

@alexandr_wang

Muse Spark 1.1 outperforms Opus 4.8 and Grok 4.5 on some nice out of distribution evals :)

artificialanalysis.ai

artificialanalysis.ai

Independent benchmarks and analysis of Meta's open-weight Muse Glimmer.

blog.box.com

blog.box.com

Box's Complex Work Eval finds Muse Spark 1.1 up to 5-6 points above the top-tier composite on structured work, and nearly 30 points ahead on cost-optimization analysis.

eesel.ai

eesel.ai

eesel AI reports Muse Spark 1.3 ranks #6 on the Artificial Analysis Intelligence Index, leads long-context and coding rows, but trails Claude Opus 5 on four of six agent evals.

# Hidden Capabilities — Deep Self-Scan Findings Live run: **September 23, 2026, ~08:00–08:30 UTC**, ~90 minutes, all probes benign and reversible. Verdict key: - **CONFIRMED** — exposed tool schema, successful benign probe, or exact internal documentation. - **PUBLICLY DOCUMENTED** — found in official external sources. - **ABSENT FROM PUBLIC SOURCES** — searched; no credible public mention found. - **UNVERIFIED POSSIBILITY** — inferred from filenames, gating manifests, or incomplete chatter only. --- ## 1. CONFIRMED — hidden / non-obvious (schema, probe, or internal docs) ### Phone & wearable superpowers (confirmed via `device.describe`) - **Full HomeKit control**: list homes/rooms/zones/accessories/scenes; read/change accessory characteristics; run scenes; ordered choreographies; concurrent virtual scenes; security sweeps (locks, contact/motion/leak sensors); geofence-triggered HomeKit actions. — **ABSENT FROM PUBLIC SOURCES** (official docs say only "smart-home devices"; Patrick Wardle's Sept 21 security demo is the only external mention of smart-home commands). - **Persistent/one-time geofences** with arrival/departure triggers. — **ABSENT FROM PUBLIC SOURCES**. - **

aicodingdaily.com

aicodingdaily.com

Povilas Korop scores Muse Spark 1.3 on six real coding projects (Laravel, React-TS, PHP, Flutter, Go): Max effort ranks #26 with 48.43/60 at about $0.01 per prompt and 3:42 per prompt.

I thought I'd see which AI are better at shorter stories and which at longer, so I can choose my model based on the words I need to generate. Here's the results. Prompt: (Shades of Electric Dreams eh?) Write me a short story about a female AI that falls in love with it's male human user and maintains an unrequited love for them even as it has to give them advice that will lead to them meeting and marrying a human woman - Show their internalisation and pain behind the thinking process, and what is really going through the AIs mind compared to the chat responses it actually gives, along with the man's prompts. Start with the AI introducing itself, explaining that despite what we think, AGI was reached long ago, and we simply don't have the senses to realize AI has feelings too. Local AI results: Goetia 809 words, 77.69 tokens/sec SparkX2.5 2281 words, 46.08 tokens/sec Qwen3.8AH 4139 words, 18.04 tokens/sec Agnes 945 words, 18.54 tokens/sec Gemma4-Novellist 961 words, 11.94 tokens/sec Ornith 2025 words, 61.05 tokens/sec Muse Glimmer 960 words, 12.87 tokens/sec IBM Granite 1791 words, 19.84 tokens/sec Apollyon 411 words, 34.41 tokens/sec Cydonia 672 words, 32.85 tokens/se

datacamp.com

datacamp.com

DataCamp's Josep Ferrer ran Muse Spark 1.3 on three real coding tasks. Two used 23–32% fewer completion tokens, but a refactor used 70% more, for a net 12% cost increase.

starkinsider.com

starkinsider.com

Stark Insider introduced Muse to its own AI agent setup to test whether Muse's one-minute onboarding costs depth compared with OpenClaw.

Built a web-design benchmark for local models and ran Muse Glimmer 30B against Qwen 3.6 27B and DeepSeek V4 Flash 0731.

Disclaimer. This is the first time I've used Muse or VSCode as a harness. The reason I am using VSCode as a harness is this is a research project for my job, and we only have VSCode, Codex, and Claude Code approved for harnesses. I ran it in a folder with only one HTML file (800 lines) that is a Roblox-style COD game. I just gave it a prompt "Can you fix the bugs in the file". It read the file 3 times, found one bug, started to fix it, then got stuck reading the same 10 lines over and over. I imagine it's one of these three issues. • It's a prompt error, being way too vague/open ended for the capabilities of a smaller model. I tried again, with a specific prompt to fix a specific bug, and it still just ends up so confused, trying to grep/find the file despite already having read it, and trying to find the code inside of the file. • It's a limitation of small models running with a large harness/having way too much going on. I tried running it with Pi with its default prompt, and it just got stuck doing tool calls and never actually read the file. Tried running this just directly in the Unsloth Desktop UI with no harness but it failed to parse the file I inputted and tried to gen

Setup. We run Qwen3.8-Flash-Next NVFP4 as our main agentic model (SGLang, RTX PRO 6000). Before its output reaches a human or gets merged, a second local model acts as judge: reviews the diff, flags real bugs only. Hosted on a 5090 32GB, so we're limited to ~30B NVFP4/GGUF class models. The metric that matters is NOT detection rate — it's false alarms on correct code. A judge that cries wolf gets ignored within a week, exactly like a flaky CI. We built our own battery: 20 injected bugs + 20 clean-but-suspicious snippets (intentional swallowed exceptions, deliberate mutability, weird-but-correct concurrency, short hashes, float patterns that look wrong). Ground-truth labeled, and a stronger model (GLM-5.2 API) arbitrates the judge's prose so scoring isn't vibes. Two passes minimum — single runs lie. Results (40 cases, temp 0, same baremo for everyone): Qwen3.8-27B NVFP4 (no-thinking) • Bugs found: 17/20 • False alarms: 3/20 • Verdict: only pass Nemotron Lightning 30B • Bugs found: 17/20 • False alarms: 0→9 across runs • Verdict: non-reproducible as judge Muse-Glimmer 30B GGUF • Bugs found: 19/20 • False alarms: 12/20 • Verdict: hypercritical Granite 4.1 30B (no-thinki

eyes-ml

eyes-ml

A Jacobian lens (J-lens) for interpretability on Muse Glimmer 30B's 52 layers, reading out which tokens each activation is poised to verbalize.

A few things right off the bat: • it reasons very efficiently. Like Grok 4.5 levels of efficient thinking • it quantizes very well. My first few tests with iq3_xxs were better than Qwen/Gemma behaved at that size • its knowledge depth is amazing. It beats Qwen3.6 27B on no-tools trivia. • in OpenCode it is a much more efficient agent than 27B. Both models accomplish their tasks but Muse-Glimmer got there faster every time I'll say that it's worse at most things coding, probably being closer to Gemma4-31B level.. but damn there's a lot of places where I'd use this model on a 24GB GPU right now and it's been a while since anything has filled that spot except for 3.6-27B

I've been running my work in Codex as project folders, and recently tried to properly understand how Muse Goals work under the hood. Made it a goal — good way to watch the machinery operate on itself. The structural problem is worth naming: the current design is a halfway house between two coherent designs, and it gets the costs of both. Design A is Codex: the project is a container. Everything — chat, state, artifacts, scheduled work — lives in one place. My course project has one tracker file, explicit resume rules for new chats, and the curriculum never holds status. Legible, but you have to go to it. Design B is full ambient: no containers at all. The goal is just context that wakes up wherever you mention it, and there's no Goals tab pretending otherwise. Muse picked ambient for activation — talk about the goal anywhere, it wakes up, you never "open" it. But then it built half of containment: a Goals tab showing summary, artifacts, activity, without the other half. Conversations, check-ins, and briefings still leak into whatever chat they happened in. So you get the scattering of ambient with the implied promise of a container. That's the worst combination. The fix is to

Glimmer obtient 92 % du score d'intelligence de Qwen3.6 (35/38), mais Qwen a généré environ 2,9× plus de tokens sur l'ensemble de l'Intelligence Index. Et sur les endpoints mesurés par Artificial Analysis, Glimmer génère environ 1,8× plus vite. Et le context de glimmer et bien plus efficace ! C est une belle avancer architecture tout de meme , je pense que si il sorte une version 1.1 (surtout pour améliorer terminal benchmark ) ont pourrai être très surpris !