docs/tools/ — Tool profiles

One profile per setup under test. Read the individual files for mechanism detail; this index is for comparison at a glance.

Top-4 CIs overlap on every task — do not read rank-ordering within the top cluster as statistically significant. The mindful and omc tools are author-adjacent (declared in each profile’s disclosure block).

Leaderboard & upstream

All tools run on the same base executor model: claude-opus-4-6. The only thing that varies between rows is the setup (plugin, skill pack, hook kit, or bare-CLI configuration) wrapping Claude Code.

Cross-task z-scores

Exact CIs in results/cross-task-stats.json. Top-4 ordering inside the overlap band is sampler-noise, not signal.

Mechanism taxonomy

Grouping the 9 setups by the primary enhancement mechanism:

“Orchestrator” here means the setup adds its own agent/subagent layer on top of Claude Code, not just prompt + tool choices. The compound/ecc/bmad/omc row is not a ranking of orchestration quality — compound ranks 7, bmad ranks 1. Architecture is under-determinative of outcome in this corpus.

Invocation & planning profile

Why pure and mindful get --permission-mode plan: they have no native planning layer. Every other setup brings its own plan step; stacking Claude’s plan mode on top of a setup’s plan would double-plan. gstack is excluded from plan-mode specifically because its /ship skill runs an eng-review gate at the tail — see ../methodology/pipeline.md §1.

Per-task winners

No tool wins on all three tasks. pure is the only tool to make top-3 on two tasks (bugfix, refactor). Rank-1 on refactor is omc, but on a task where judge Spearman ρ CIs straddle zero — treat rank-1 there as less meaningful than rank-1 on feature.

Observed failure modes

Cross-cutting patterns from reading the transcripts (see ../analysis/skill-and-hook-patterns.md for the deep dive):

• Over-orchestration on small tasks — compound, omc. Multi-agent setups paying a high fixed setup cost on a 30-minute task suffer when the task doesn’t use their multi-phase capacity.
• Setup-turn tax — mindful, omc. A dedicated /setup turn before the task costs one round of prompt + context. Hurts on greenfield-feature where planning context is tight.
• --auto gate-suppression — claudekit. /ck:cook --auto removes the human-review gates that the setup relies on; the scripted workflow runs end-to-end but the “stop on test failure” behavior doesn’t kick in, resulting in 15 failing tests in a green commit.
• Skill activation depends on entry-point wording — superpower. A registry-style skill library whose triggers are generic vocabulary (“bug”, “error”, “root cause”) is not guaranteed to activate on a terse prompt that does not pattern-match. The bugfix harness names the slash-commands explicitly (/superpowers:systematic-debugging, /superpowers:verification-before-completion) to isolate skill content from trigger-phrase sensitivity; see superpower.md and ../analysis/skill-content-effectiveness.md.
• Eng-review gate ≠ code-quality gate — gstack. /ship enforces process cleanliness but doesn’t catch all code-quality issues; average performance on bugfix/refactor despite tier-1 ranking on feature.

How to read these profiles

Each profile follows roughly:

1. Upstream & identity — repo, version, license, maintainer.
2. Performance — per-task z, CI, tier; disclosure of author-adjacency where applicable.
3. Mechanism — what actually runs: skills, hooks, sub-agents, MCP, permission layer.
4. How this benchmark invoked it — exact prompt, plan-mode flag, setup turn.
5. What the transcripts show — per-task: tool-call mix, sub-agent use, commit shape.
6. Why it ranked where it did — grounded in transcript evidence.
7. Strengths & failure modes — per-task observations.
8. References — links back to upstream and to the benchmark artifacts.

The intent is: a reader should be able to close the profile with an accurate mental model of how the tool operates in practice on this corpus, not a re-phrasing of the upstream README.