Real-Agent Campaign Findings (2026-06-11)

# Real-Agent Campaign Findings (2026-06-11)

First full instruction-level campaign with provider-reported tokens. Raw JSONL

+ per-run transcripts: `results/agent-campaign/` (local, never committed).

**Setup.** `agent_workflow.py`, `bugfix_test_loop`, agents claude (CLI

2.1.173) + codex (0.139.0), subjects git + oak_installed (oak 0.96), minimal

agent environment, randomized agent/subject order, n=5 per cell per level.

Cursor (2026.06.11 build) verified separately (1 run: pass, provider tokens,

turn metrics). Never aggregate across levels; never compare codex's

token_metrics to claude's raw (codex input counts include cache reads).

## Results (medians, n=5 per cell)

| Level | Agent | Subject | Pass | Reported tokens | Turns | Cumulative input | Help calls (Σ) | Unknown cmd (Σ) | Wall s |

| --- | --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |

| zero-shot | claude | git | 5/5 | 4,627 | 10 | 183k | 0 | 0 | 32 |

| zero-shot | claude | oak | 5/5 | 7,438 | 19 | 371k | 16 | 2 | 61 |

| zero-shot | codex | git | 5/5 | 152,790 | 10 | 151k | 0 | 0 | 56 |

| zero-shot | codex | oak | 5/5 | 159,639 | 10 | 157k | 20 | 0 | 59 |

| cheat-sheet | claude | git | 5/5 | 2,209 | 9 | 163k | 0 | 0 | 33 |

| cheat-sheet | claude | oak | 5/5 | 3,456 | 14 | 262k | 10 | 10 | 49 |

| cheat-sheet | codex | git | 5/5 | 124,417 | 9 | 123k | 0 | 0 | 50 |

| cheat-sheet | codex | oak | 5/5 | 138,325 | 8 | 136k | 0 | 0 | 52 |

## Findings

1. **The familiarity tax is cost, not failure.** 40/40 tasks passed,

   including zero-shot oak. On this simple bugfix shape, models do not fail

   on oak — they pay extra context to learn it.

2. **Zero-shot oak ≈ 2× context for claude.** 19 vs 10 turns, 371k vs 183k

   cumulative input, 61s vs 32s. Every extra turn re-pays the transcript;

   this is the per-task dollar form of "git is in pretraining, oak is not."

3. **A 3-line cheat-sheet closes the gap for codex completely** (help 20→0,

   turns 8 vs git's 9 — parity) **and most of it for claude** (turns 19→14,

   cumulative input 371k→262k, help 16→10). The AGENTS.md story is

   quantitatively real.

4. **Cheat-sheet content gap found: `oak commit -m`.** At cheat-sheet level

   claude logged 10 unknown-command failures (vs 2 zero-shot): transcripts

   show `oak commit --no-verify -m "..."` — git habit appends `-m`, oak has

   no message flag (messages live on `oak desc`), agent recovers via

   `oak commit --help`. One added cheat-sheet line ("commits carry no

   message; use `oak desc`") should eliminate the whole class. Changing the

   generated cheat-sheet text is an instruction-material change: bump the

   level's version metadata and never mix runs across the change (ADR-0005

   discipline applies to prompts too).

5. **Provider accounting asymmetry (method note).** Codex's reported totals

   (~125–160k) include cache reads; claude's (~2–7k) are billed

   output-direction totals. Cross-AGENT token comparisons are invalid raw —

   use turns and per-provider deltas. Cross-SUBJECT deltas within one agent

   are valid and are the published numbers.

6. **n=5 per cell.** Per `docs/publish-checklist.md`, success-rate claims

   want n≥10 per level; the cost deltas here are large (2× turns) but a

   publish-grade drop should double the reps.