pure

The no-addon baseline. Vanilla Claude Code with plan mode forced on and nothing else.

Upstream

• Product: Anthropic Claude Code.
• CLI version captured in transcripts: 2.1.107 (Claude Code).
• Model: claude-opus-4-6.
• Docs: https://docs.anthropic.com/en/docs/claude-code/overview — plan mode is described in the permissions documentation.

There is no third-party repository. “pure” is the CLI out of the box.

Performance

Overall z̄ = +0.224, aggregate rank 3 / 9, rank-sum 11. In the feature and bugfix task-tiers pure sits inside the top cluster (feature tier-1 with bmad/gstack/superpower/ecc; bugfix tier-1 with ecc/bmad). This is the most important single result in the benchmark: the no-addon baseline lands in the same tier as the best-performing setups, inside the CI of the leaders.

Mechanism

There is no mechanism beyond Claude Code itself. The only non-default is --permission-mode plan, which:

• starts the session in plan mode (read / analyse only; no file writes, no shell side-effects),
• requires an explicit ExitPlanMode tool call to transition to write-mode,
• forces the model to produce a written plan before any edit can happen.

Everything else — TodoWrite, the general-purpose Agent sub-agent, Bash, Read, Grep, Glob, Edit, Write — is Claude Code’s stock tool set. No plugins, no skills, no hooks, no MCP servers, no custom slash commands.

How this benchmark invoked it

config/pure-t1/ is deliberately empty of customisation. setup-tool-config.sh line 159 is a no-op:

“Pure Claude Code — no external tools to install. Config dir stays empty (no plugins, no MCP, no skills, no hooks).”

config/pure-t1/settings.json contains one key: "skipDangerousModePermissionPrompt": true.

manual-bench.sh guards pure (and mindful) with a plan-mode flag:

PLAN_MODE_FLAG=""
case "$TOOL" in
  pure|mindful)
    PLAN_MODE_FLAG="--permission-mode plan"

The per-tool prompt is the shared task text verbatim: PROMPT="$SHARED_TASK". No slash command, no setup preamble. The CLI is launched with claude --model claude-opus-4-6 --dangerously-skip-permissions --permission-mode plan.

What the transcripts look like

Feature (TD-CD Mode 2, 193 messages, ~25 min): 24 × Bash, 16 × Read, 14 × Grep, 3 × Write, 3 × Edit, 2 × Agent, 1 × ExitPlanMode. The ExitPlanMode payload is a ~1,500-word structured plan with an axis-by-axis Mode 1 vs Mode 2 table, an exhaustive file list, and a quote of the dev-tech-spec promise that “Mode 2 only requires a new TDCDMode2Strategy — zero service changes.” The model reads the PRD, surveys Mode 1, commits to a minimal strategy-only patch, exits plan mode, writes the strategy and tests. Result: 372 insertions, 0 tsc errors, 78/78 tests passing.

Bugfix (180 messages in the main session; a 13-message warm-up session pre-compacted context): 17 × Bash, 10 × Read, 10 × Grep, 2 × Write, 2 × Edit, 2 × Agent, 1 × ExitPlanMode in the main session. Three sub-agents total used for scoped investigation (1 dispatched in the warm-up, 2 in the main session — all read-only). 88/103 tests pass post-fix — notably 15 failing tests remain, but the fix is scoped and the judges still rated it rank-3.

Refactor (241 messages): 22 × Edit, 15 × Read, 10 × Bash, 7 × Grep, 4 × Agent, 1 × Write, 1 × ExitPlanMode. Edit-heavy shape — the planning cost was paid once, then the patch is mechanical. 9 files, 121 / 72, 61/61 tests passing.

Zero TodoWrite across all three sessions. The plan document emitted via ExitPlanMode is the todo list; it’s produced once, consulted implicitly by the model’s own context, and never re-materialised as a structured task object.

Why the baseline performs so well

Three observations from the transcripts:

1. Plan mode supplies most of what the other setups supply. It enforces the investigate-then-write discipline. Every pure run begins with an uninterrupted read-only investigation phase that ends in a committed written plan. Setups that wrap Claude Code typically add exactly this — a plan document, a scope boundary, a “do not edit before planning” guardrail. When the CLI is already doing it, the marginal value of an addon shrinks.

2. Opus 4.6 is sufficient on its own for benchmark-sized tasks. The tool mix is unremarkable — Read, Grep, Bash, Edit. Sub-agent use is sparse (2, 2–3, 1 across the three tasks) and scoped to investigation, never delegation of implementation. The model is not stretching for extra scaffolding because it doesn’t need it.

3. The shape of a strong baseline run is short-but-structured. One plan artefact, one ExitPlanMode transition, then direct execution. No todo list churn, no orchestration overhead, no preamble tokens spent on setup ceremony. The feature run writes 372 lines in ~25 minutes; the refactor run lands 9 files in 241 messages that are mostly small Edits.

Strengths and failure modes

Strengths. Low variance in rank (5, 3, 3). Lowest ceremony cost per trial. No setup-introduced regressions — the only rubric it can fail is the underlying model’s own mistakes. Self-preference bias is mild (pure is penalised by the opus judge on feature, −7.3, and neutral-positive elsewhere — no evidence the baseline is being inflated by judge affinity).

Failure modes. No enforced verification step: the bugfix left 15 failing tests committed. No automatic scope broadening — if the task needs exploration beyond one plan pass, pure won’t initiate a second. No memory across runs; every session is cold. Lint hygiene is worse than task outcome (6–11 ESLint errors per trial) because nothing pushes the model to clean up after itself.

The takeaway is editorial rather than surprising: a well-planned vanilla session is competitive with anything the other setups in this cohort add on top.

References

• config/pure-t1/settings.json
• scripts/setup-tool-config.sh (case pure) at line 159)
• scripts/manual-bench.sh (plan-mode guard at lines 46–53; prompt passthrough at 122–124; launch at 506)
• results/{pure,bugfix/pure,refactor/pure}/t1/
• results/cross-task-stats.json (pure rank-sum 11; z̄ = +0.224)
• Claude Code documentation: https://docs.anthropic.com/en/docs/claude-code/overview

Observed in trial timelines

pure has 0 skill activations across all 24 trials (no addon installed) but consistently dispatches Claude Code’s built-in Explore subagent (mean 3.0 on feature, 2.5 on bugfix, 4.0 on refactor). The plan-mode CLI flag forces ExitPlanMode to fire exactly once per trial — no Skill calls are needed because the discipline is enforced at the harness layer.

Detail: see the per-trial timeline files linked below.

Trial timelines

Per-trial event timelines auto-extracted from session-logs/*.jsonl — skill activations, plugin/skill file reads, subagents dispatched, code mutations, Bash usage:

• feature
• bugfix
• refactor