docs/tools/ — Tool profiles

One profile per setup under test. Read the individual files for mechanism detail; this index is for comparison at a glance.

Setups & upstream

All tools run on the same base executor model: claude-opus-4-7, base-model effort medium, context window 200k. The only thing that varies between rows is the setup (plugin, skill pack, hook kit, or bare-CLI configuration) wrapping Claude Code. Versions are pinned in versions.lock.json tools.*.

Per-task weighted-mean ranking

Pulled from results/<task>/final-report.md (weighted mean of per-judge means under the pre-registered 3 / 2 / 1 / 1 / 1 panel; equal-weight comparator alongside).

Cohort weighted-mean per task: feature 139.89 · bugfix 167.99 · refactor 172.07. No setup is top-2 on all three tasks. A single cross-task summary is sensitive to weighting; each cell is now 5 trials per cell, 75 judgments per cell (5 trials × 5 judges × 3 rounds) — read the three per-task final-report.md files together rather than collapsing to one leaderboard (README caveat 7).

Mechanism taxonomy

Grouping the 8 setups by the primary enhancement mechanism:

“Orchestrator” here means the setup adds its own agent/subagent layer on top of Claude Code, not just prompt + tool choices. The orchestrator row is not itself a ranking of orchestration quality — compound ranks 7 on feature but 5 on bugfix and refactor; bmad ranks 3 on refactor but 6 on bugfix. Architecture is under-determinative of outcome in this corpus.

Invocation & planning profile

Per-task winners

pure (no addons) is the only setup top-3 on every task and rank-1 on refactor. The null hypothesis “tools add no value over the bare CLI” is not rejected on refactor (top-5 within ≈ 5.8 weighted pts); on bugfix, claudekit (+9.40 pts over pure) is below the bugfix MDE (22.17), so the strict null is not rejected there (the earlier ~5-pt-tie-envelope claim is retracted — see PAPER §3 / §4).

Session audit (transcript-mined behavior)

Each per-tool profile now includes a session-audit table with the headline behavioral metrics — wall-clock, main/sidechain turns, sub-agent dispatches, file reads (split into tool-config vs target-repo), files edited, and cache hit ratio — computed across 3 trials per task from the raw JSONL transcripts (session-audit run-time subsystem is n=3; the per-task score tables are the full n=5 cohort).

Source: scripts/audit-sessions.py walks results/<tool>/t<N>/session-logs/*.jsonl (feature) and results/{bugfix,refactor}/<tool>/t<N>/session-logs/*.jsonl for the other tasks, plus the subagents/*.jsonl files underneath. Per-trial JSON: results/<...>/t<N>/session-audit.json. Cohort summary: results/_audits/session-audit.md.

Headline cross-tool findings from the audit:

• Sub-agent dispatch is not what most docs claimed. bmad dispatches 0 sub-agents on bugfix across all 3 trials; compound dispatches reviewer-personas in only 1 of 9 trials (compound/t2 feature). The “multi-reviewer gate” architecture documented upstream materialises as sequential attributionSkill changes inside a single thread, not as actual Agent tool calls.
• superpower’s skill activation is task-dependent. Sub-agent fan-out is 18.3 dispatches/trial on feature but ~0.5 on bugfix/refactor — the subagent-driven-development skill triggers on feature work only.
• omc is the only setup with material tool-config read overhead (~17 reads per feature trial vs ~0 for every other tool) — the “setup tax” of a self-maintaining orchestrator.
• pure is the only setup with zero attributionSkill rows — every assistant turn is base-model, confirming the no-addon mechanism.
• Cache hit ratios are 0.97–0.98 for most tools; omc is lowest at 0.93 (worker fan-out forces more cache creation).

Observed failure modes

Cross-cutting patterns from reading the transcripts:

• Over-orchestration on small tasks — compound, omc. Multi-agent setups paying a high fixed setup cost on a 30-minute task suffer when the task doesn’t use their multi-phase capacity.
• --auto gate-suppression — claudekit. /ck:cook --auto removes the human-review gates that the setup relies on; the scripted workflow runs end-to-end but the “stop on test failure” behavior doesn’t kick in.
• Skill activation depends on entry-point wording — superpower. A registry-style skill library whose triggers are generic vocabulary (“bug”, “error”, “root cause”) is not guaranteed to activate on a terse prompt that does not pattern-match. The bugfix harness names the slash-commands explicitly to isolate skill content from trigger-phrase sensitivity.
• Eng-review gate ≠ code-quality gate — gstack. /ship enforces process cleanliness but doesn’t catch all code-quality issues; rank-8 on feature and bugfix.

How to read these profiles

Each profile follows roughly:

1. Upstream & identity — repo, version, license, maintainer.
2. Performance — per-task weighted mean, within_σ, between_σ, rank.
3. Mechanism — what actually runs: skills, hooks, sub-agents, MCP, permission layer.
4. How this benchmark invoked it — exact prompt, plan-mode flag, setup turn.
5. What the transcripts show — per-task: tool-call mix, sub-agent use, commit shape.
6. Why it ranked where it did — grounded in transcript evidence.
7. Strengths & failure modes — per-task observations.
8. References — links back to upstream and to the benchmark artifacts.

The intent is: a reader should be able to close the profile with an accurate mental model of how the tool operates in practice on this corpus, not a re-phrasing of the upstream README.