ADR-0010: Agent campaigns preserve assignment, attempts and evidence limits

• Status: accepted
• Date: 2026-09-10

Context

An agent benchmark is an experiment over task outcomes, not just successful command timings. A process killed before summary generation must not erase its attempts. Failed tasks, unsupported measurement and provider accounting differences cannot become cheaper successes. Existing fixed-fixture repetitions also cannot establish generalization to unseen tasks or human users.

Decision

A frozen versioned plan contains the complete schedule and provenance. Each trial has a stable ID and matching-subject block ID. Started/finished journal events reference the plan hash and trial ID; the full agent row stays inside the terminal envelope. Journal persistence is incremental and duplicates or foreign identities are invalid, not silently overwritten.

The report API is report_campaign(plan, events) with a pure JSON result and render_markdown(summary) for human inspection. Its schedule entries identify workflow, instruction level, agent/model and subject. Groups never pool those conditions. Runtime population metadata and paired comparison identities must also agree. A finished skip may precede execution; all other terminal events require a start. A start without completion is interrupted; an absent start remains unstarted, except an ambiguous torn start is explicitly reserved as an uncertain interruption during journal recovery. Runner exceptions are infrastructure failures; missing required oracle instruments are unmeasured skips, not measured task failures.

Count every scheduled state. Operational verified-success fractions use started trials; interrupted trials are not verified successes. Do not treat the partial fraction as a final success estimate. Omit Wilson intervals while started trials remain unresolved or required instrument measurements were skipped. All incomplete/unstarted/skipped counts stay visible. Recorded trial timeouts that differ from the frozen timeout are excluded from complete paired comparisons.

Resource summaries expose measured coverage and partial observed sums. A complete total requires observed values for every started attempt. Null is not zero. Successful-trial means/medians are explicitly conditional; all-attempt resource totals include failed attempts. Cost-weighted tokens are a proxy, never USD.

Paired success differences resample complete recorded blocks with a reproducible seed, report excluded/incomplete pair counts, and remain descriptive. Singletons receive no bootstrap interval. Multiplicity is not adjusted in this initial instrument, and repeated fixed tasks are not independent sampled task families.

All initial campaign reports carry evidence_class: diagnostic_agent_campaign and claim_eligible: false, including apparently complete real-provider runs. This is an honest boundary until independent durability, task distribution, resolved model identity, validated experimental inference and publication gates are qualified. Mock agents prove instrumentation only. The existing zero-failure efficiency scorecard is not reused as an agent-reliability gate: properly measured failures can be scientific evidence.

Consequences

Plans, scenario/config definitions and analysis code may be committed; raw transcripts, result journals and task workspaces remain external/ignored. Results are inspectable and resumable without pretending interruption never happened. Later attempts need new identities and cannot replace originals.

The agent subprocess timeout and campaign admission budget are not a hard end-to-end deadline. Budget reservations for lost attempts are accounting, not observed timing. Admission requires the entire frozen subprocess timeout; insufficient remaining allowance stops the campaign without shortening a trial. There is no promised hard USD cap without actual measured pricing and enforceable admission. Human studies retain their own protocol.

The staged implementation and qualification acceptance criteria are in docs/agent-campaign-plan.md; having this ADR is not evidence that the pending scientific and durable-state gates have passed.