bmad — BMad-Method

A structured Agile-style workflow harness that runs a Plan → Code → Review loop through a single /bmad-quick-dev slash command.

Upstream

• Repo: https://github.com/bmad-code-org/BMAD-METHOD
• Version used: 6.3.0 (pinned via npx bmad-method@6.3.0 install in scripts/setup-tool-config.sh; also recorded in plugin_versions of the bugfix and refactor auto-metrics.json).
• Author / maintainer: bmad-code-org (GitHub organisation). Individual maintainer unknown — not verified in this pass.
• License: unknown — not verified in this pass.
• Primary doc: https://github.com/bmad-code-org/BMAD-METHOD (README). Installer also writes _bmad/ and .claude/skills/bmad-* into the repo with its own docs; upstream doc URL beyond the README unknown — not verified in this pass.

Performance in this benchmark

Rank 1 / 9 overall (combined z̄ = +0.292). bmad is in the top statistical tier on feature (tier: bmad, gstack, superpower, ecc, pure) and on bugfix (tier: ecc, bmad, pure, mindful, gstack), and in the top tier but near the bottom of it on refactor. The ordering inside the top-4 overall (bmad, ecc, pure, gstack) is not statistically distinguishable.

Mechanism — what actually runs

• Install surface (from scripts/setup-tool-config.sh lines 145–157): npx bmad-method@6.3.0 install --directory . --modules bmm --tools claude-code --yes is run inside the cloned repo. This writes _bmad/ (config + agent prompts, gitignored by the bench safety rules), .claude/skills/bmad-* (skill files exposed to Claude Code), and _bmad-output/ (tracked, used for phase artifacts). No plugin is added to the CLI’s plugin registry — the config/bmad-t1/plugins/ directory contains only the default claude-plugins-official marketplace. The tool’s settings.json is just { "skipDangerousModePermissionPrompt": true }; all bmad behaviour comes from the in-repo skill files, not from CLI-level config.
• Entry point (from scripts/manual-bench.sh line 146): the harness prepends an intro that tells bmad which path to take, then sends PROMPT="/bmad-quick-dev $BMAD_INTRO\n\n$SHARED_TASK". The intro always says “Pick the Plan-Code-Review path” and adds task-shape hints (“non-trivial feature”, “scoped bugfix — reproduce first”, “scoped refactor — no behavior change”). Observable behaviour: the command parses the intro, then proceeds through plan → code → review phases inline. In the transcripts the loaded skill adds workflow instructions to the session (substring bmad-quick-dev appears 11/24/22 times across feature/bugfix/refactor; Plan-Code-Review appears 4/10/11 times; workflow 18/16/17 times; checkpoint 6/9/9 times; elicitation 1/4/4 times — all from within the transcript).
• Skills / sub-agents / hooks activated: one Claude Code Skill invocation at session start (feature log). During execution bmad dispatches Agent tool calls to sub-agents (4 in bugfix, 1 in refactor, 0 in feature). Roles observed in sub-agent prompts: an Explore investigator, a blind adversarial code reviewer, an edge-case hunter, and an acceptance auditor. No external hooks, no MCP servers — only the installed skill files and Claude Code’s built-in Task/Agent tool.
• Core mental model: BMad stages a cut-down Agile cycle — scope → plan → implement → multi-angle review — into a single slash command, with a fact-forcing preamble and explicit phase checkpoints before file edits.

How this benchmark invoked it

Exact PROMPT (from manual-bench.sh, with per-task intro):

/bmad-quick-dev Pick the Plan-Code-Review path — this is a <non-trivial feature | scoped bugfix | scoped refactor> in an existing <brownfield> codebase. <task-shape hint>

<SHARED_TASK>

Base model: claude-opus-4-6 (same for all nine tools).

What actually happened in the transcripts

• feature (Mode 2 CD Batch, z = +0.215): 270-turn session, no sub-agent dispatch. The agent skipped elicitation (“I have sufficient context. Mode 1 is library-only — planning implementation now.”) and went straight into extending ITDCDModeStrategy, wiring validateInventory for per-batch scheme context, adding a new td-cd-mode2.strategy.ts + utilities, and writing specs. Result: 12 files changed (+508 / −6), 82/82 tests passed, 0 tsc errors, 13 eslint errors / 6 warnings (results/bmad/t1/auto-metrics.json).
• bugfix (z = +0.640, the strongest trial): 273-turn session, 4 Agent dispatches. First an Explore sub-agent mapped savings-cd batch eligibility; then three independent reviewers (blind-adversarial diff review, edge-case hunter, acceptance auditor) ran against the proposed patch before commit. 3 files changed (+222 / −4), 90 tests passed / 15 failed (core: 27 pass / 15 fail; savings-cd: 63/63 green), 0 tsc errors, 10 eslint errors / 15 warnings.
• refactor (z = +0.020): 301-turn session, 1 Agent dispatch (scoped Explore mapping the two design seams being cleaned). The agent carried out 11 edits (+129 / −75) across libs/core, kept a fact-forcing preamble before each edit, and landed all 61 tests green — but the resulting diff scored at cohort mean. 8 eslint errors / 7 warnings.

Why it ranked 1 (noting the top-4 tie)

• The bugfix trial is the single biggest contributor (z = +0.640, the largest per-task lift among bmad’s three tasks). Judges rewarded the triple-reviewer gate before commit and a reproduced-test-first patch; every judge — opus (184.0), codex (160.5), qwen (192.0) — placed bmad at or near the top for this task.
• On feature, bmad sat in the top tier but near the top of the field on length of work rather than on elegance: the opus judge gave it the single highest feature score (124.85) while codex put it only mid-field (95.85). A meaningful opus/codex divergence.
• The refactor trial was flat (z ≈ 0); bmad’s rank on refactor is 6, so its overall rank-1 finish is entirely driven by the other two tasks and the top-4 tie is statistically indistinguishable in CI terms.

Strengths & failure modes

Strengths (transcript-grounded):

• Multi-reviewer gate: bugfix’s three post-diff reviewers (adversarial, edge-case, acceptance) show up in the sub-agent prompts, not in the shared task brief — they come from bmad’s skill files.
• Fact-forcing preamble before edits and explicit “checkpoint” markers keep the agent from drifting on long sessions (9 checkpoint references each in bugfix and refactor).
• Matches task shape via the intro: the harness-provided hint flips the workflow between feature / bugfix / refactor without changing the slash command.

Failure modes (transcript-grounded):

• Lint is not gated: 13 / 10 / 8 eslint errors at commit across feature / bugfix / refactor; the workflow does not round-trip eslint before declaring done.
• Pre-maturity elicitation is skipped when the agent judges context “sufficient” (feature turn 0: “I have sufficient context … planning implementation now”), so the plan step can collapse into a monologue.
• Refactor run used only one sub-agent and produced a mean-level result — the multi-reviewer pattern that lifted bugfix did not fire here, suggesting the “Plan-Code-Review” loop is strongest on tasks with a named defect to review against.
• Overhead shows in length: 270–301 turns per trial is among the longer sessions in the benchmark.

References

• Install surface: scripts/setup-tool-config.sh (case bmad), lines 145–157)
• Prompt construction: scripts/manual-bench.sh (case bmad), lines 131–149)
• Config snapshot: config/bmad-t1/settings.json, config/bmad-t1/plugins/
• Transcripts: results/bmad/t1/session-logs/03edbe91-3a0d-4351-a8f8-2d6956ab36e8.jsonl, results/bugfix/bmad/t1/session-logs/16b2a47a-78f9-4a3e-86b0-c8aef7432577.jsonl, results/refactor/bmad/t1/session-logs/809fef16-9f6f-45ad-8622-517fba071d30.jsonl
• Metrics: results/bmad/t1/auto-metrics.json, results/bugfix/bmad/t1/auto-metrics.json, results/refactor/bmad/t1/auto-metrics.json, results/cross-task-stats.json
• Upstream repo: https://github.com/bmad-code-org/BMAD-METHOD

Observed in trial timelines

bmad is the only tool whose skill content shows up as explicit Read events in the session (mean 3.5 unique skill files on feature, 6.5 on bugfix, 4.5 on refactor; range 2–8). Every other tool injects skill content via slash command into the system prompt, so no Read events fire — bmad’s step-by-step step-01…step-05 files are loaded at runtime as the workflow advances. This is why bmad’s transcripts read like a script: the skill content is materialised in-band.

Detail: see the per-trial timeline files linked below.

Trial timelines

Per-trial event timelines auto-extracted from session-logs/*.jsonl — skill activations, plugin/skill file reads, subagents dispatched, code mutations, Bash usage:

• feature
• bugfix
• refactor