Agent Workflow Benchmarks

# Agent Workflow Benchmarks

This suite extends the existing Oak benchmarks from isolated VCS operations to

end-to-end agent tasks. The goal is to measure how much the version-control

model helps or hurts real coding-agent workflows without baking in assumptions

about a specific model, provider, or UI.

The scenario definitions live in `scenarios/agent.yaml`. Result rows should

follow the schema emitted by `scripts/agent_metrics_schema.py`.

## Implemented Runner

Use [`agent_campaign.py`](../scripts/agent_campaign.py) for repeated local

experiments with frozen schedules, durable trial records and paired reporting.

See the [campaign plan](agent-campaign-plan.md) for supported trusted tasks,

resume behavior and real-provider launch criteria. The one-shot runner below

remains the execution layer; its historical summary is not a campaign analysis.

`scripts/agent_workflow.py` is the real-agent lane. It currently supports these

local CLI adapters:

- `codex`: `codex exec --json` — verified against a real codex-cli 0.139 run

  (2026-06-10). Codex's `turn.completed` covers the whole session and its

  `input_tokens` includes the cached subset; the adapter counts round-trips

  from `agent_message` items and never double-counts cache.

- `claude`: `claude -p --output-format stream-json --verbose` — verified

  against a real claude 2.1.170 run (2026-06-10; `--verbose` is required by

  the CLI for stream-json output).

- `cursor_agent`: `agent --print --output-format stream-json` — not yet

  verified against a real run; treat its rows as unconfirmed until a real

  transcript is captured and promoted to `tests/fixtures/`.

- `mock`: deterministic local adapter for free CI/schema validation

Sanitized real transcripts live in `tests/fixtures/*_real.jsonl` and pin each

adapter's parsing in the instrument test suite.

The default is `--agents mock` to avoid accidental model spend. Use

`--agents codex,claude,cursor_agent` or `--agents available` for real local-agent

runs. The default `--agent-environment minimal` asks CLIs to disable local

customizations where supported, so token deltas are not dominated by personal

skills, plugins, hooks, or project memory. Use `--agent-environment local-default`

when measuring the actual local setup an agent would normally use.

The runner records provider token usage when a stream exposes recognizable usage

fields; otherwise it reports `token_source=char_only` and uses normalized

transcript characters as the comparable token-pressure proxy.

`scripts/conflict_resolution_lane.py` is a local scripted vertical slice for

the `conflict_resolution` scenario. It consumes the

`bench-conflict-corpus-v1` repositories plus `*.RESOLUTION.json` sidecar

oracles, drives a Git merge/conflict presentation/oracle-resolution/check

workflow, and emits existing `workflow` lane rows with `conflict_kind`,

observed/expected conflict files, and resolution oracle hash/check fields. It

uses no external services; unsupported subjects or missing commands/corpus data

emit skip rows with returncode 77.

Tool-call counts are adapter-visible counts. If a CLI hides internal tool events,

the row remains valid but the lower tool count should be interpreted as an

adapter visibility limit, not proof that no tools were used.

**Null means unmeasured (schema v2).** The runner never fabricates zeros. A

signal the adapter does not expose (hydration bytes, tool wait time, parallel

batching, retry chains) is null with a `measurement_source` explaining why.

## Turn Metrics

Tool calls alone hide the dominant agent cost: model round-trips. Every

assistant turn re-sends the transcript as input tokens, so a 28 KB status dump

on turn 2 of a 30-turn session is paid ~28 more times (minus cache discounts),

and a saved tool call that removes a *turn* is worth far more than its own

tokens. The runner parses adapter streams into:

- `assistant_turns_total`: inference round-trips.

- `tool_calls_per_turn_avg` and `max_tool_calls_per_turn`: batching behavior.

- `cumulative_input_tokens_reported`: sum of per-turn input including cache

  reads — the context-residency cost of everything the VCS printed.

- `peak_input_tokens_reported`: peak context proxy.

- a per-turn timeline artifact (`turn_timeline.jsonl`) for debugging.

Trend turns and cumulative input tokens next to tool calls. A VCS change that

cuts output bytes but adds a turn is usually a net loss.

## Instruction Levels And The Familiarity Tax

Frontier models know Git zero-shot; they do not know a new VCS. That gives Git

a head start no per-command benchmark sees: instruction tokens every session,

plus guessing costs (hallucinated subcommands, `--help` round trips). The

runner exposes `--instruction-level`:

- `zero-shot`: the prompt names the VCS but gives no commands.

- `cheat-sheet`: a one-line command list.

- `full-docs`: the historical detailed instructions (default).

Rows record `agent.instruction_level`; never aggregate across levels. The

headline metric is instruction tokens required to reach git-parity success

rate. Supporting counters: `help_calls_total` (discovery calls) and

`unknown_command_failures_total` (hallucinated usage).

## Recovery Cost

An error message is a prompt. The runner measures `turns_to_recovery` and

`tokens_to_recovery` between the first failed tool call and the next successful

one, when the adapter exposes per-call outcomes. The scripted

`vcs_error_recovery` workflow complements this with deterministic failure-path

output costs per subject.

## Scope

The current `scripts/bench.py` answers narrow questions such as how long status,

diff, branch creation, or snapshot operations take for a fixture shape. Agent

workflow benchmarks answer broader questions:

- How much context does an agent need before it edits the right file?

- How many tool calls and failed commands does each workflow induce?

- How much wall-clock time is agent reasoning versus tool or VCS wait time?

- How many logical bytes are read, written, or hydrated from storage?

- How well do branch, mount, merge, and cleanup workflows support parallel agents?

Each benchmark run should produce one JSONL result row per scenario, subject,

VCS mode, agent adapter, and repetition. Detailed command traces can be stored as

artifacts, but the row should contain the normalized counters needed for trend

charts.

## Scenario Catalog

| Scenario | Main behavior under test |

| --- | --- |

| `one_line_bug_fix` | Tiny fix where setup and exploration overhead dominate. |

| `docs_edit` | Documentation-only edit with source inspection for factual grounding. |

| `cross_file_refactor` | Multi-file code, test, and docs update with precise symbol matching. |

| `dependency_upgrade` | Manifest and lockfile update using a controlled local dependency cache. |

| `generated_file_churn` | Source schema edit plus deterministic generated-file churn. |

| `binary_asset_update` | Binary replacement plus checksum or manifest update. |

| `large_file_inspection` | Targeted inspection of a file too large to place in context. |

| `find_the_right_file_exploration` | Vague bug report requiring broad but efficient navigation. |

| `parallel_independent_tasks` | Multiple isolated tasks merged from one starting revision. |

| `parallel_overlapping_tasks` | Parallel tasks touching overlapping files without semantic loss. |

| `conflict_resolution` | Intentional line conflict resolved while preserving both intents. |

| `failed_test_then_fix` | Recovery after a failing test disproves the first attempt. |

| `abandon_and_cleanup` | Correctly abandon an invalid task and leave no VCS residue. |

## Fairness Rules

Use the same fixture, task prompt, tool surface, timeout, and success oracle for

every subject in a scenario. Fixture generation, dependency cache population, and

subject binary builds are outside timed work. Task setup, agent work, tool wait,

VCS operations, tests, merge, and cleanup are timed.

The runner should compare VCS modes before it compares agents. For example, run

the same agent adapter against `git_worktree_per_task` and `oak_space_per_task`,

then compare medians and p90s. Cross-agent comparisons are valid only when the

report clearly separates model/provider effects from VCS effects.

Do not require network access during a scenario unless the network itself is the

thing being measured. Dependency-upgrade fixtures should use a local registry or

prewarmed cache so retries reflect agent and tool behavior, not public service

variance.

## Provider-Neutral Token Metrics

Token accounting is not portable across providers. The result row therefore has

both provider-reported token fields and normalized text counters.

Record provider-reported fields when the adapter exposes them:

- input, output, reasoning, cache-read, cache-write, and total tokens

- token budget and context truncation events

- token source, such as `provider_reported`, `adapter_estimated`, `char_only`,

  `mixed`, or `unknown`

Always record normalized character counts for prompts, tool results admitted to

the transcript, assistant output, and the complete transcript. These are not a

perfect proxy for tokens, but they let dashboards compare context pressure across

providers without pretending all tokenizers are equivalent.

For fairness, raw token deltas should be trended within the same provider or

adapter. Cross-provider charts should default to normalized character counts and

show provider token totals as secondary metadata.

## Tool-Call Metrics

Instrument at the harness/tool-adapter boundary, not by parsing the model's

private reasoning. Count every agent-visible tool call, including terminal

commands, file reads, file writes, search calls, edit calls, VCS commands, test

commands, network calls, privileged calls, and parallel tool batches.

The runner should classify commands by intent when possible:

- VCS: `git`, `oak`, and helper commands that create, merge, snapshot, push, or

  clean branches and mounts.

- Tests: language test runners, linters, type checkers, package-manager test

  aliases, and fixture validators.

- Exploration: `rg`, directory listings, file reads, metadata reads, and range

  reads.

- Mutation: file edits, generator runs, package-manager writes, binary copies,

  and formatters.

Keep raw command logs as artifacts. Store only stable counters and small failure

summaries in the result row.

## Failed Command And Retry Metrics

A failed command is useful signal, not just noise. Record failed tool calls,

failed terminal commands, failed VCS commands, failed test commands, retry

attempts, retry chains, same-command retries, and time spent after a recoverable

failure.

A retry chain starts when the agent reacts to a failed command or rejected edit

and ends when it either succeeds, changes strategy, or abandons. `failed_test_then_fix` should require at least one failed test before the final passing run;

other scenarios should not reward gratuitous failures.

Dashboards should show failure rate next to wall-clock time. A fast run that

passes only by skipping verification should not outrank a slightly slower run

that executed the intended tests.

## Wall-Clock Metrics

Use a monotonic clock. Separate total elapsed time from timed phases:

- setup: branch, mount, worktree, prompt, and task workspace preparation

- agent active: time spent waiting for the agent response stream

- tool wait: commands, file operations, searches, and external tool latency

- VCS: init, status, diff, branch, commit, push, merge, cleanup

- tests: unit tests, integration tests, lint, type checks, and fixture oracles

- merge and cleanup: parallel branch integration and teardown

- idle: harness queueing or scheduler delay that should not count against the agent

Record timeout budget separately. A timeout outcome should include partial

metrics so regressions are visible.

## Bytes Read, Written, And Hydrated

The bytes metrics should distinguish three ideas:

- logical workspace size: how large the fixture is when fully materialized

- agent I/O: bytes read from files, command output admitted to context, and bytes

  written by edits or commands

- hydration: bytes the VCS or filesystem had to materialize from lazy storage to

  satisfy the agent workflow

For Git subjects, hydration may be equivalent to checkout or worktree

materialization. For Oak subjects, hydration should count object or file bytes

loaded on demand. The runner should report both values so Oak is not unfairly

penalized for a large logical repo when an agent only inspects a few files, and

Git is not unfairly penalized when all bytes were already materialized before the

timed phase.

Large-file scenarios should prefer range reads, search commands, indexes, or

streaming tools. Dumping a large file into the transcript should increase both

`bytes_read_by_agent` and `transcript_chars`, making the inefficiency visible

without needing provider-specific token math.

## VCS Modes

The YAML defines these modes:

- `git_single_checkout`: one mutable checkout per run. Use only for serial tasks.

- `git_worktree_per_task`: one worktree or clone per task branch.

- `oak_single_mount`: one Oak mount for a serial run.

- `oak_mount_per_task`: one Oak mount per task branch.

- `oak_space_per_task`: one Oak space subdirectory per task using the agent

  lifecycle flow.

Parallel scenarios should use isolated modes so each agent starts from the same

revision. Serial modes can still be useful for baseline overhead, but they should

not be compared directly against truly parallel wall-clock results.

## Success Oracles

Each scenario has explicit success criteria in `scenarios/agent.yaml`. A runner

should evaluate them with deterministic checks:

- expected tests and validators pass

- changed files stay within the allowed shape

- generated output matches a clean generator run when applicable

- binary assets match expected bytes, checksums, or dimensions

- conflict markers and partial edits are absent

- cleanup scenarios leave no leaked branch, mount, worktree, or dirty file

The model's final message is not a success oracle. It can be stored as an

artifact, but pass/fail should come from repository state and commands.

## Reporting

Trend median, p90, failure rate, and timeout rate by scenario, agent adapter,

subject, and VCS mode. Useful derived views include:

- Oak mode versus Git mode for the same agent and scenario

- local Oak versus installed or main Oak for the same agent and scenario

- token count per successful run, using provider-reported totals when available

- tool-call count per successful run

- failed commands per successful run

- bytes hydrated per successful run

- wall-clock phase breakdown for parallel scenarios

Keep raw transcripts and command logs out of summary tables by default. They are

valuable for debugging individual runs, but summaries should stay provider-neutral

and comparable.