superpower

Superpowers is a skill-pack plugin for Claude Code by Jesse Vincent (obra). It ships a library of named skills — brainstorming, writing-plans, test-driven-development, systematic-debugging, verification-before-completion, and roughly two dozen more — that the base model is expected to invoke via the Skill tool when appropriate. In this benchmark it ranks 9 of 9 overall (z̄ = −0.320), driven by a bugfix gap (z = −0.740) and a refactor dip (z = −0.370); on feature it is competitive with the leaders (z = +0.149). See ../analysis/skill-content-effectiveness.md for the wording-level mechanism behind the bugfix gap.

Source. obra/superpowers; skills live in obra/superpowers-skills (MIT), marketplace manifest in obra/superpowers-marketplace. The benchmark installs at pinned 5.0.7 (SHA 917e5f53). Setup writes two lines to settings.json (enabledPlugins + marketplace entry). No wrapper command, no hook, no MCP server — a pure skill registry.

Benchmark invocation. On feature and refactor, the harness passes the raw task prompt unchanged (PROMPT="$SHARED_TASK"); skill activation is up to the base model. On bugfix, the harness pins two slash-command triggers — /superpowers:systematic-debugging at session start and /superpowers:verification-before-completion as a completion gate — so skill activation is isolated from base-model trigger-phrase sensitivity. The scores below are under those operating conditions.

What the plugin actually installs

A named skill registry plus short SKILL.md files loaded into context. Relevant skills for this benchmark: superpowers:brainstorming, writing-plans, executing-plans, test-driven-development, systematic-debugging, verification-before-completion, dispatching-parallel-agents, and using-superpowers (the meta-skill that tells the model to consult the registry before responding). On feature and refactor none are auto-triggered by the harness — the only control surface is the skill descriptions themselves and whatever the base model has learned about when to invoke them. On bugfix the harness names the triggers directly.

What the benchmark measured

Feature — rank 3. z = +0.149, 581 lines across 8 files, 90/90 tests passing, 10 ESLint errors. The session fired superpowers:brainstorming exactly once, then ran 125 turns of a standard build using TaskCreate / TaskUpdate for internal tracking. No parallel agents, no plan document, no TDD skill. The brainstorm appears to have disambiguated the Mode 2 CD Batch requirements before coding. Mid-pack.

Refactor — rank 9 (last). z = −0.370, 123 added / 42 removed across 12 files, 61/61 tests passing, 9 ESLint errors. One superpowers:brainstorming invocation, then 110 turns with two Explore subagents. All tests pass; judges placed it below T2. Per-task CI [141.8, 155.7] overlaps the middle tier and §4.3 flags refactor as noise-dominated — read this rank as weak evidence.

Bugfix — rank 9 cluster. Mean 156.08 / 200, z = −0.740, 95% CI [151.2, 161.0]. Tier T2 ({claudekit, compound, omc, superpower}). Both t1 and t2 fired 2 Skill calls each (systematic-debugging at session start, verification-before-completion at finish). Hard gates: 5/5 PASS (both trials). Scope files touched: 2. Diff sizes: +176 (t1), +125 (t2).

The harness prompt for bugfix explicitly names both skills:

/superpowers:systematic-debugging

<SHARED_TASK>

When you believe the fix is done, run /superpowers:verification-before-completion before claiming success.

This isolates skill quality from the base model’s trigger-phrase sensitivity.

Why it ranked 9/9

superpower is the only setup whose bugfix CI is fully disjoint from the cluster above it (T2 vs. T1), even with skills explicitly invoked. T2 on bugfix still trails T1 by 12–23 pts. On feature (z = +0.149) it is competitive with the leaders; on refactor (z = −0.370) it trails T1 inside the noise-dominated task (§4.4). The combined 9/9 rank is the cross-task consequence, not a statement about any single task.

Failure modes

1. Activation dependence on entry-point wording. The one-shot harness — used on feature and refactor — passes a terse prompt with no language that pattern-matches the skills’ trigger vocabulary. The model can read the prompt as a direct execution request and skip the registry. On bugfix the benchmark controls for this by naming the slash-commands in the prompt; without that, activation is not guaranteed. The wording-level mechanism is described in ../analysis/skill-content-effectiveness.md.

2. Skill output lands in T2 on bugfix, not T1. Once activated, the skills run a clean systematic-debugging → verification-before-completion pipeline that clears all five hard gates and keeps the diff scoped. Judges still score it 12–23 pts below ecc/bmad/pure. The remaining gap to T1 is skill-content headroom, not an activation failure.

Honest positioning

Superpowers is a well-constructed skill library. On feature-class work it costs nothing and adds one useful reflective pause. The 9/9 combined rank is driven by bugfix (z = −0.740) and refactor (z = −0.370 in noise). Read the remaining gap to T1 leaders as “skills work, but mid-pack on this corpus” rather than “the plugin is broken.” The base-model dependency on trigger-phrase pattern-matching for activation is a real operating-condition caveat for anyone deploying this in a one-shot harness: if the entry prompt does not name the skills, the registry may stay dormant.

Cross-reference: ../analysis/skill-content-effectiveness.md — wording patterns that drive top-5 bugfix performance, including caveats.

Observed in trial timelines

On bugfix, where the harness explicitly names the skills as slash-commands, superpower fires the highest Bash volume of any tool (mean 51, peak 63) and the highest test count (mean 24.5) — yet still scores last. Verification effort and judged quality decouple here: more test runs do not produce higher rubric scores. On feature, where no skill is forced, superpower lands at the cohort mean for both Bash (30.5) and tests (16) and scores rank 3.

Detail: see the per-trial timeline files linked below.

Trial timelines

Per-trial event timelines auto-extracted from session-logs/*.jsonl — skill activations, plugin/skill file reads, subagents dispatched, code mutations, Bash usage:

• feature
• bugfix
• refactor