DeepSearchQA Skill Eval Template

A harness for evaluating chat models on google/deepsearchqa: pi sessions run You.com
MCP tools, and answers are graded with the official DeepSearchQA rater.

The harness is intentionally thin. The agent is a stock pi session loading the upstream
@youdotcom-oss/pi extension; its bundled
you-web skill is passed directly into the system prompt, and read is withheld so the
model cannot open the answer key in data/prompts.jsonl.

What we found

Three scaffold generations were evaluated against a fixed local-weight root
(meta/muse-glimmer-30b via OpenRouter) on the same 50-task K=1 samples (and two full
900-task × 3-trial runs):

configuration	F1 (raw)	FC pass	notes
RLM v5 custom extension	0.8054 / 0.6910 (900×3)	64% / 52.8%	best recorded; hand-built distillation
Raw MCP port (8a7588d)	0.6647	60%	no extension
@youdotcom-oss/pi, all tools	0.5421	44%	read on (unsafe config)
@youdotcom-oss/pi, skill injected	0.5000	46%	read off
@youdotcom-oss/pi, you-search + you-contents	0.4788	40%	two tools only
@hicaru/pi-rlm scaffold	0.0	0%	failed experiment
RLM v6 / v7 / v8	0.7390 / 0.7054 / 0.2626		deeper sub-model autonomy degrades

Full history, configs, and paired analysis: docs/experiment-log.md.

Limits identified

The remaining constraints are structural, not tuning knobs:

• The model, not the tools, is the ceiling. Adding tools, skills, budget, or deeper
  scaffold autonomy moved the 50-task score within run-to-run noise (±0.05) or worse. The
  root over-searches (500–650 calls on the two-tool run) and under-synthesises.
• Skills are weak leverage here. pi surfaces a skill only when a file-reading tool is
  active — and that tool is an eval-integrity hole (it can read the expected answers).
  Passing the skill body directly into the prompt is the only safe mechanism, and it did
  not move the score.
• MCP content is not the bottleneck. Raw, truncated, highlighted, distilled, and
  full-page reads all land in the same band once synthesis is the limiting step.
• More autonomy hurts this root. The monotonic v6→v8 slide and the pi-rlm failures show
  that handing a local-weight model more recursiv