End-to-end agent VCS campaigns: implementation and qualification plan

Status: staged engineering plan, not completed scientific evidence. A working campaign runner and mock trials demonstrate instrumentation, not agent or human performance. No real-provider results, independent Oak durability proof, power analysis or human study is established by this document.

Questions and estimands

The primary question is whether an agent reaches the requested correct, persisted repository state within a declared resource policy. Report operational success on all started trials, including infrastructure interruptions as their own category. Analyze VCS-attributable failures separately using a predefined classification; do not remove inconvenient failures after viewing results.

For confirmatory comparisons, predeclare a practically meaningful reliability noninferiority margin before interpreting efficiency gains. A faster task that loses work is not an efficiency win. Primary outcomes are exact verified task completion and data preservation. Secondary outcomes include agent-process and whole-task elapsed time, provider-reported tokens, turns, observed tool calls, help/error/recovery interactions and actual incurred resource use.

Success-conditioned latency answers a different question from resource cost over all attempts. The report names that conditioning explicitly. Its observed resource total divided by verified successes is an empirical campaign ratio, not a prediction of expected future cost to solve a new task. Zero successes leave this ratio undefined. Missing usage leaves complete totals undefined; observed partial sums and coverage remain visible. Cost-weighted tokens are not dollars. Cross-provider token totals with different cache/billing semantics are not directly comparable.

Stage 1 — frozen campaign and inspectable execution

Run the implemented local campaign

From an isolated benchmark checkout with Git and Oak available:

python3 scripts/agent_campaign.py plan \
  --manifest config/agent-campaign-smoke.json \
  --campaign-dir results/campaign-smoke
python3 scripts/agent_campaign.py run --campaign-dir results/campaign-smoke --limit 1
python3 scripts/agent_campaign.py status --campaign-dir results/campaign-smoke
python3 scripts/agent_campaign.py resume --campaign-dir results/campaign-smoke
python3 scripts/agent_campaign.py report --campaign-dir results/campaign-smoke --format markdown

The example schedules four mock config-refactor trials: two repetitions each for Git and Oak. The intentional one-trial run exits 3 because work remains. Run/resume exits 0 for complete all-pass evaluation, 1 for complete evaluation with failures, 2 for invalid input or preflight rejection, and 3 for pending or interrupted trials. A report command succeeding does not certify its evidence.

To change the matrix, copy the manifest outside source control and edit it before plan. Configuration paths are relative to the manifest file; absolute paths are also accepted. Supported workflow IDs are wide_config_refactor and large_asset_manifest. Instruction levels are separate conditions. Plans cap materialized schedules at 10,000 trials and reject a schedule larger than its declared max_trials. --limit bounds new trials in one invocation.

plan.json is immutable input evidence; events.jsonl is the authoritative durable trial journal; rows.jsonl is a derived convenience view. Trial artifacts include prompts, oracle logs and transcripts. A report reads the journal even if a crash prevented refreshing the derived rows. Do not discard a torn journal record: preserve it for recovery investigation. Artifact hashes describe files observed after a trial, not authenticated remote storage or a hostile-host seal.

Real execution additionally requires --enable-real, an explicit requested model and the manifest's isolation attestation. These controls do not enforce a dollar budget or provision OS containment. Apply the Stage 5 launch criteria before using real credentials.

Contract

Implemented in this increment: a versioned campaign plan, deterministic paired trial schedule, started/finished journal and diagnostic report interface. The concrete CLI/manifest syntax lives with oakbench.agent_campaign; the report consumes schedule, plan_sha256, and journal event/trial_id/row envelopes. Do not infer study completion from the existence of these files.

The manifest must freeze intended workflows, subjects, explicit requested model, CLI/config identity, instruction levels, environment, seed, repetition count, timeouts and campaign admission limits. The materialized plan retains provenance and the exact generated schedule. Stable trial IDs bind each attempt to its workflow, subject, model, instruction level and block; a common block identifies matching subject trials. The seed controls scheduling. Current fixed fixture repetitions are not randomly sampled tasks or independently seeded variants.

Acceptance:

• Identical plan input produces identical schedule and IDs; changed semantics produce a new plan hash and cannot resume the old journal.
• Subject order is counterbalanced within declared conditions and replayable. Provider/model, instruction and environment conditions are never pooled.
• Every measured trial uses an isolated task checkout and fresh agent session. Shared measurement locks exclude competing measurement runs; queue time is not counted as agent work.
• Journal starts precede execution; terminal rows are persisted incrementally. Duplicate/unknown events, changed plan hashes and contradictory outcomes fail closed. A started trial with no terminal event remains interrupted.
• Resume cannot silently overwrite or retry an interrupted/failed trial. Any later replacement is a new recorded attempt linked to its predecessor.
• Mock execution, unavailable metrics, skipped trials and incomplete schedules are explicit. A zero-work/all-skipped campaign is not a successful study.

Budget boundaries: trial_timeout_seconds constrains the agent subprocess, not setup, oracle execution and teardown together. max_wall_seconds is an admission budget. A new trial requires at least its entire frozen subprocess timeout remaining; otherwise the runner stops with insufficient_budget_for_frozen_trial_timeout and exit 3. The runner never shortens that timeout. Setup/oracle/teardown overhead can still exceed the admission allowance. Reports flag legacy trials with mismatching recorded timeouts as budget-truncated interruptions and exclude them from complete pairs. Reserving budget for an interrupted trial is accounting policy, not measured elapsed time. Neither establishes a hard end-to-end time limit. No hard dollar cap is claimed without trustworthy provider pricing/usage and enforceable request admission. Model token/turn limits must be either actually enforced and recorded by an adapter or explicitly unsupported, never fabricated.

Stage 2 — complete diagnostic reporting

Implemented: oakbench.campaign_report.report_campaign(plan, events) and render_markdown(summary) retain scheduled/started/finished, success, failure, timeout, infrastructure failure, interrupted, skipped and unstarted counts. Missing measurements stay null. Recorded zero is a measured zero.

Reports show verified successes divided by started trials. This is a partial verified fraction while interrupted trials remain unresolved, not an estimated final success probability. Wilson intervals are omitted for unresolved started trials. Unstarted/skipped trials remain in scheduled denominators. Descriptive paired differences use complete matching terminal blocks only; incomplete pairs are counted, and the conditioning is explicit. Single complete pairs get no bootstrap interval. Candidate minus reference direction is printed.

Paired agent-active-time and reported-token differences are computed only on jointly successful pairs with both metrics observed under the same recorded measurement convention. Missing/unknown/incompatible-source pairs remain counted. These mean differences use the actual matching blocks, not a ratio of independently summarized subject medians. Lower is better for these resource deltas; higher is better for success-rate deltas. Joint-success conditioning can select easier tasks and must not be interpreted as unconditional efficiency.

The bootstrap resamples recorded paired blocks, preserving covariance. It does not establish independent sessions or task generalization, correct for multiple comparisons, or implement sequential stopping inference. Reports always state claim_eligible: false. This avoids applying the existing zero-failure efficiency-scorecard policy to a reliability experiment in which well-observed failures are meaningful evidence.

Acceptance tests include all outcome categories, zero successes, missing versus zero tokens, interruptions, duplicate journals, mismatched model/metadata, paired covariance, seeded resampling and visible failure-only conditions. Independent reconstruction must reproduce every displayed count from events.

Pending: end-to-end enforcement needed for a restricted completion-time estimand. Once implemented, every unsuccessful attempt—including early abandonment, wrong final state, interruption and timeout—contributes the common limit to that estimand. Actual consumed time is retained separately. Until then, never convert admission reservations into observed latency.

Stage 3 — exact task, tool-use and durability qualification

Current trusted config/asset payload validators and Git committed-tree checks are a starting point, not a universal task oracle. Every admitted workflow must declare an expected final tree, graph relationships, preserved unrelated work, and independent evidence of persistence. Read-only archaeology needs an answer artifact with exact facts; recovery needs explicit restored/preserved state; merge tasks need parent, unresolved-path and final-content checks.

Required negative controls: empty/partial/extra payloads, same-size corruption, wrong modes/link targets, renamed paths, validator edits/import shadowing, misleading successful exits, unchanged snapshots, partial commits, untracked loss, conflict-marker leftovers and wrong history. Each must be rejected by both agent and scripted integration paths.

Oak file inspect and a subject-produced export do not alone provide independent complete-tree verification. Qualification remains blocked until a trusted object reader or independently verified fresh-reader contract enumerates and checks the entire persisted tree and relevant history. Sampled fingerprints or the subject's own success message cannot upgrade this evidence.

Actual VCS participation also needs evidence: a task solved by changing files while bypassing the requested VCS is not automatically a VCS success. PATH shims observe only invocations routed through them; absolute paths and alternate APIs can bypass them. Record coverage/limitations and independently verify the end state. An editing-only control is useful but has a different task contract and must be reported separately.

Tool surfaces must match the declared comparison: same editor, shell, allowed help, network access and agent tools, with subject-specific command syntax and equivalent documentation access. First-use, trained-use and warmed context are separate conditions. Core-equivalent and agent-default contracts remain distinct.

Stage 4 — faults, remote workflows and acquisition

Add capability-matched remote publication, divergence recovery, merge/handoff, worktree/mount setup and fresh-recipient continuation. Git local-file remotes cannot stand in for Oak network operations in comparative timing. Match server, network shaping, fixture state and acquisition scope, or label the asymmetry.

Predeclare fault locations: before request, during transfer, after remote apply but before acknowledgment, local interruption, disk-full/permission failure and repeated recovery. Verify both ends independently after restart. Exactly-once intent, idempotent retry and preservation of unpublished work are separate invariants. Transport interruption is not a power-loss/fsync durability test.

Acceptance: a fresh process/reader observes the intended persisted state after each claimed recovery, without relying solely on warmed caches or a subject's status output. Unsupported cases stay explicit coverage gaps, not skips hidden from the campaign denominator.

Stage 5 — controlled real-provider pilot and calibration

Run a small explicitly authorized real-provider pilot only after mock and local oracle qualification. Requested model aliases are not resolved model snapshots. Capture provider-returned model identity when available; otherwise label it unresolved and disallow model-specific confirmation claims. Record CLI version, reasoning/sampling settings, provider token conventions, cache accounting, prompt/documentation hashes and enforced budgets. Unsupported knobs stay null.

Real-launch go/no-go also requires a disposable OS VM/container/runner with restricted mounts and only the provider credentials needed for the trial. Current agent CLI modes may execute unrestricted commands. An isolated directory or manifest assertion is not OS containment: an agent with host access could modify the supposedly trusted oracle or read unrelated credentials/files. Verify containment with attempted out-of-scope reads/writes before admitting a real run. Directory-only trials are explicitly limited pilots and cannot support confirmatory broad correctness claims; independent task-directory placement is only a meaningful trust boundary when actual containment enforces it.

Use pinned hosts/configuration and record calendar/session/provider outages, background load and cache state. Randomize/counterbalance within sessions and replicate across sessions to reveal drift. Preserve warm/cold setup and familiarization conditions. Keep provider failures and infrastructure cost visible.

Cross-check reported usage against raw provider events. Calibrate estimates against the actual output styles and tokenizer; do not promote char-count or weighted-token proxies into billing claims. Collect actual USD only with versioned applicable prices and complete measured billable usage. Partial timeout/output-limit transcripts cannot supply complete token, turn or tool totals. Preserve the partial artifact and its coverage reason; report full usage unmeasured unless the adapter independently attests complete accounting.

Launch acceptance: provenance and event completeness independently audited; every selected task has a falsified trusted oracle; scope is capability-matched; no silent model/config drift; measured resource coverage stated; operator can stop and resume without losing attempts; all pilot reports remain exploratory.

Stage 6 — confirmatory design, power and generalization

Before confirmation, freeze primary estimands, reliability margin, practical efficiency threshold, task distribution, comparison family, exclusions, retry policy, sample/stopping rule and analysis code. Determine sample size using pilot variance, failure prevalence and participant/task/session structure. An arbitrary n=30 floor or noise-ratio heuristic is not a power calculation.

Source tasks independently of the optimization loop and hold out disjoint task families/fixture variants. Repetitions of one config edit do not count as many independent tasks. Use session/task-cluster or hierarchical analyses suited to the design; retain paired subject comparisons within blocks. Simulations must test interval coverage and false-positive behavior under skew, ties, rare failures, drift, clustering, missingness and stopping rules.

Multiplicity control requires a defined hypothesis family and valid test statistics, not merely calling a BH helper. Prespecify family-wise or FDR policy; retain adjusted and raw results. Confirmation data must be disjoint from the pilot/development data used to choose targets. Replicate borderline findings under a predefined independent confirmation rule. Show uncertainty, effect sizes and incomplete-data sensitivity, not a wall of unqualified GREEN cells.

Publication acceptance: complete attempts/outcomes accounted for; compatible resolved identities and oracles; independent durable evidence; required precision and powered reliability comparison; validated analysis and multiplicity control; archived raw evidence hashes; independent reproduction; no generalization beyond the sampled task/model/environment distribution. Failures may be publishable; missing or misleading evidence is not.

Stage 7 — humans are a separate evidence stream

Use docs/human-vcs-study-protocol.md: actual participants, experience strata, counterbalanced tasks/tool order, declared training and help, workload measures, participant-cluster uncertainty and independent task oracles. Human actions are not simulated by scripted commands or coding agents. Report human and model results separately even when they share fixtures and success contracts.

Journal recovery and derived evidence

status and report read the intact prefix of a journal with a torn final record without modifying it. They include an unacknowledged recovery marker. An existing unfinished start stays interrupted. When no start survived, the next scheduled pending trial is conservatively reserved as an uncertain start; this is accounting uncertainty, not evidence that an agent ran or elapsed time was observed. It is never retried or counted as a complete comparison.

Ordinary resume refuses a torn journal with exit 2. After inspecting it, use resume --acknowledge-torn-journal to archive the exact original journal bytes, record their SHA256, and atomically replace the active journal with its intact prefix plus the explicit recovery records. The archive remains in the campaign directory. Recovery only admits remaining trials. Started records are durable before trial-directory creation, so a crash at that boundary cannot leave a pending trial whose directory is silently overwritten.

Frozen provenance includes executable/configuration hashes, Python harness sources, non-hidden configuration and scenario inputs, and the canonical measurement lock path. General project documentation and unrelated dotfiles are not prompt inputs and are excluded. Finished events preserve schema-rejected rows under rejected_row (or their representation if not valid JSON), alongside the infrastructure error; derived rows.jsonl contains validated rows only. It is rebuilt from the authoritative journal on resume and updated incrementally.