How to evaluate a Muse agent build: a practical testing guide

To evaluate a Muse agent, collect 20 to 30 real tasks from your own workload, write a pass/fail check for each that doesn't rely on the agent's own summary, run every task several times at the effort level you'll actually pay for, and record pass rate, cost per pass and failure types. Stub any tool that spends money or sends messages. Public benchmarks tell you which models to try. Only your own task set tells you whether your build works.

Why public benchmarks aren't enough

Meta's numbers for Muse Spark 1.3 and Muse Glimmer are strong. But DataCamp found Spark 1.3 cost 12% more across three real tasks despite a claimed 25% token saving, and MindStudio's open-ended game test came out far worse than the leaderboard implied. Ryan Gu's Glimmer deployment scored top marks on vision and SRE-ops but failed five deployment gates. A benchmark measures its own tasks. More in Muse Spark benchmarks.

Step 1: Build a task set from real work

Wavect's Glimmer guide recommends a 20 to 30 task pilot before production. That's a good size: large enough to see patterns, small enough to check by hand.

Good tasks:

• come from your actual backlog or logs, not invented examples
• include a few you expect to fail
• cover the long tail: the odd PDF, the site with a CAPTCHA, the vendor with no API
• have an unambiguous "done" state

Paweł Huryn's approach is a clean model for coding agents: he planted 105 bugs across two real repos and counted fixes. You know the right answer because you put it there.

Step 2: Write checks the agent can't talk its way past

Agents are good at reporting success. Your checks should look at the world, not the transcript.

Task type	Check
Code change	Tests pass; diff touches only expected files
Form or booking	Confirmation record exists in the connector or inbox
Data extraction	Field-by-field match against a hand-labeled answer
Research	Claims have sources that say what's claimed
Money task	Final amount is within the authorized ceiling

The delegate-to-muse skill builds this in: review the diff and re-run tests "rather than trusting the summary." Maestro adds proof-of-done checks to Muse Code. ocodex has a separate supervisor audit every worker claim.

Step 3: Stub dangerous tools

Never test a purchasing or messaging agent against live accounts first. Replace tools that spend or send with stubs that record the call and return a realistic response.

glimmer-cli is a good reference: a local CLI for Glimmer and Spark 1.2 with stubbed tools and a reproducible tool-use eval harness. For Muse connectors, stand up a staging copy of your connector with fake data. Then graduate to read-only access on real accounts, and only then to writes behind approval.

Step 4: Run each task more than once

Agents are nondeterministic. Philippe Dourassov's CVE test shows why repeats matter: Spark 1.3 rediscovered 19 of 32 CVEs on average at pass@1, but 24 of 32 when three runs were pooled. Report both numbers:

• pass@1: how often a single run succeeds, which is what users feel
• pass@k: whether it can succeed at all, which tells you if retries help

Three to five runs per task is usually enough to see the spread.

Step 5: Test across effort levels and harnesses

Morgan Linton ran Spark 1.3 through VulcanBench v4 at every effort level and found serious issues at the lower ones. If you plan to run at medium to save money, evaluate at medium.

Harness matters too. NetworkCoder gave two agent harnesses the same Glimmer model, endpoint, project and prompt to isolate the harness effect. Venelin Valkov tested whether a better harness fixes Glimmer's mixed early reviews. Hold everything constant except the thing you're testing.

Step 6: Record cost and time per pass

For each run, log tokens in and out, cached tokens, wall time, tool calls and result. Then compute cost per successful task. A cheaper model that fails half the time can cost more per pass than a pricier one. One Reddit tester found Glimmer finished an eval in 3 to 4 hours versus about 30 for Qwen 3.8 at xhigh, a gap that matters as much as accuracy.

Step 7: Categorize failures

Pass rate alone won't tell you what to fix. Tag each failure:

• Wrong plan: misread the task
• Tool error: bad arguments, wrong tool, gave up after one error
• Illegal action: did something disallowed (DungeonBench counts illegal moves separately for this reason)
• Blocked: CAPTCHA, verification code, site block
• False success: claimed done, wasn't

False success is the most dangerous category. Track it on its own line.

Step 8: Re-run on every change

Keep the task set in version control and re-run it when you change the prompt, model version, connector or effort level. Spark went from 1.1 to 1.3 in under two months; your results may shift with each release.

Evaluating the personal agent

You can't stub tools inside the consumer Muse app, but you can still be systematic:

1. Start with connectors in read-only mode.
2. Give tasks with a checkable output, like "list every subscription charge in the last six months with amount and date," then compare to your statement.
3. Watch it work with something like TerMuse.
4. For money tasks, set a ceiling and keep approvals on. See agent guardrails.

Frequently asked questions

How many test cases do I need to evaluate an agent?

Start with 20 to 30 real tasks, as Wavect recommends for Glimmer pilots. Run each several times. Grow the set as you find new failure types.

How do I test an agent that makes purchases without spending money?

Replace the purchase tool with a stub that records the request and returns a realistic confirmation. Only move to real payments with approvals on and a hard spending ceiling.

What metrics should I track for an AI agent?

Pass@1, pass@k, cost per successful task, wall time, tool calls per task, and a count of false successes where the agent claimed a task was done when it wasn't.

Should I trust Meta's benchmarks for Muse Spark?

Use them to shortlist, not to decide. Independent tests have both confirmed some claims, like strong bug fixing, and contradicted others, like token savings.