omc (oh-my-claudecode)

Disclosure. omc is maintained by the benchmark author (Yeachan-Heo/oh-my-claudecode); see README caveat 12. The ranking below should be read in that light — the tool was evaluated on a benchmark its own author designed, and it still lands near the bottom. That is the expected direction for an honest self-report, but it does not cancel the conflict of interest. Every rubric item, task brief, and judge prompt was chosen by the same author; an external replication is the only thing that would remove the concern.

Identity

• Repo: github.com/Yeachan-Heo/oh-my-claudecode
• Plugin version at run: 4.13.0 (bugfix trial), 4.13.1 (refactor trial); feature trial predates the version-stamp hook
• Package: oh-my-claude-sisyphus on npm
• License: MIT, Copyright 2025 Yeachan Heo
• Install mechanism: Claude Code plugin marketplace (claude plugin install oh-my-claudecode@omc)
• Invocation in this benchmark: two-message flow — /oh-my-claudecode:omc-setup, then /oh-my-claudecode:autopilot <task>

Mechanism

omc is an orchestration layer. It ships a large surface area — agents, hooks, skills, a status line, a .omc/ state directory, and an experimental agent-teams flag (CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1) — and routes work through specialized subagents via a top-level “autopilot” skill. The stated model is delegate-first: a planner decomposes the task, executor subagents write code, and reviewer/verifier subagents close the loop before a final commit. Session logs for the bench runs confirm heavy Task-tool usage: the feature and refactor trials each spawned three persisted subagent logs (omc-feature-t1: executor + Explore + verifier; omc-refactor-t1: three subagents); the bugfix trial spawned one (a single Explore dispatch). See docs/analysis/trial-timelines/{feature,bugfix,refactor}/omc.md for the per-trial breakdown. A parent transcript of 1.3 MB on the feature run and 1.0 MB on refactor is not unusual for this tool — it spends tokens coordinating.

Results

Rank-sum: 17 (tied with compound).

The refactor result is not a defense. Inter-judge Spearman ρ CIs straddle zero on the refactor task (PAPER §4.3, README caveat 3), so rank positions there are noise-dominated. Refactor self-preference for omc is +11.875, second-highest in the cohort — but PAPER §4.5 is explicit that this metric is not identified in a cohort where every executor runs on the same Anthropic base model. The honest read of the four rows is: omc is rank 8 overall, driven by a below-cohort bugfix trial (−0.599) and a below-cohort feature trial (−0.108). The refactor cell is consistent with that picture; it does not contradict it.

Automated artifacts line up with the judge scores. Feature: 8 files changed, +547 / −2 lines on a brief that did not require that much surface; 10 ESLint errors survived. Bugfix: only 2 files touched / +44 lines, but 15 core-test failures on commit — the orchestration did not prevent a shallow fix that broke the existing suite. Refactor: 11 files / +109 / −44, 8 ESLint errors, tests green.

What the mechanism did here

Two things stand out when the artifacts are read next to the session logs.

The ceremony entry point costs a full message. /oh-my-claudecode:omc-setup is a setup step — it bootstraps CLAUDE.md injection, hooks, and the .omc/ state directory — and no other tool in the cohort requires it. The bench runner has a dedicated OMC_TWO_MSG=1 branch precisely to accommodate this. In a single-session evaluation the setup message contributes no code; it is pure prelude. That alone does not explain an eight-point gap, but it sets the tone for the rest of the trace.

Orchestration did not substitute for scope discipline. The bugfix result is the most telling artifact in the set. The tool’s planner/executor/verifier loop produced a 2-file, 44-line patch that left 15 tests red. A simpler tool could have failed the same way, but the premise of an autopilot with multiple reviewer passes is that it catches exactly this class of mistake. On this trial it did not. On feature, the opposite failure mode shows: +547 lines for a PRD that the top-of-cohort tools closed in 100–300 lines of net change. Heavier does not correlate with higher rubric scores in this cohort — pure, the unadorned baseline, sits in the top-4 tie.

Honest read

omc is a plausible design. Its per-round σ is low (0.6 on bugfix, 0.6 on refactor, 1.8 on feature), meaning the tool is stable — re-running the judges would not move its mean much. The problem is not noise. It is that the orchestration surface, on these three tasks, did not produce code that judges preferred over a plain Claude session. The bench author’s own tool underperforms a zero-ceremony baseline, and the fix is not to reweight the judges; it is to take the artifacts at face value and investigate why a multi-agent loop produced a shallow bugfix and an oversized feature.

Reproducing

TASK=feature ./scripts/create-clones.sh 1
TASK=feature ./scripts/manual-bench.sh omc 1
# Paste message 1 (/oh-my-claudecode:omc-setup), wait, then paste message 2
# (/oh-my-claudecode:autopilot <task>). Exit when the tool has committed.

Artifacts live under results/omc/t1/, results/bugfix/omc/t1/, results/refactor/omc/t1/. Session logs and subagent transcripts are retained verbatim; the .omc/ state directory is gitignored by the benchmark safety rules and does not leave the clone.

Observed in trial timelines

omc is the only tool that consistently produces multi-file session logs (feature t1: 6 separate *.jsonl files; refactor t1–t2: 2 each), reflecting the autopilot pipeline’s process boundaries. Feature t1 spans roughly 13 hours of wall-clock time — by far the longest-running trial in the cohort.

Detail: see the per-trial timeline files linked below.

Trial timelines

Per-trial event timelines auto-extracted from session-logs/*.jsonl — skill activations, plugin/skill file reads, subagents dispatched, code mutations, Bash usage:

• feature
• bugfix
• refactor