Measurement-integrity corrections after the landed campaign review
Baseline: main e9b767a51662. This is a corrective branch, not a revert.
Journal durability, recovery reservations, cache binding and measurement locking
remain required. A green test suite alone does not certify a public study.
Review scope
| Finding | Required invariant and coverage |
|---|---|
| Verified task lost to capture cap; setup blamed on agent | Explicit outcome attribution and launch state. Setup produces an agent.setup skip. Capture/instrument loss is unmeasured even when the task oracle observed completion. Producer-to-publication and summary tests retain that distinction. |
| Git/Oak correctness asymmetry | Both lanes dispatch all nine workflows to shared validation. Scoped Git reads bind immutable blob identities; text decoding and committed file modes use the same contract. Eight mutating workflows use the same committed-content requirements on both subjects. History answer correctness is explicitly unavailable for both until an answer contract exists. |
| Infrastructure failures inside paired success estimates | Resource accounting retains all started attempts; infrastructure failures cannot form complete success-rate pairs or Wilson intervals. |
| Independent Oak evidence impossible | An optional separately configured reader decodes and verifies local SQLite v1 committed objects without running Oak. Subject exports remain diagnostic. See reader qualification. |
| Patch oracle changes the measured sequence/cache | Full-tree diff/patch diagnostic probes follow the main timed sequence. Diff timing uses a separate reconstructed dirty repository and explicitly records hash-warmed cache state. Probe elapsed time is not command-performance evidence. |
| Failed mutating diff becomes skip | Before/after tree comparison runs regardless of command exit status. A mutation is a measured correctness defect and stops subsequent commands against that repository. Main-chain prerequisite failures stop later measurements; determinism probes retain first-attempt failures. |
| Truncated token totals invert | Incomplete captures cannot fall back to character-derived totals; transcript/tool totals are nullable and shared accessors respect completeness. |
| Workflow publication bypasses core-only hygiene | Workflow publication requires recorded order, environment consistency, distinct trial identities and a minimum sample floor, alongside output correctness and outcome attribution. The floor is not a power analysis. |
| Incompatible conditions disappear into singleton groups | Compare subject condition sets before splitting populations; incompatible evidence is refused explicitly. Measured task failures cannot be removed merely because HEAD validation was never reached. |
| Provider HOME hashed into journal | Per-trial HOME is outside the retained artifact subtree. Credential/cache filenames are not enumerated by artifact hashing. |
| Frozen budget admits no trial | Manifest validation rejects a total admission budget smaller than one frozen timeout. Later budget exhaustion still stops admission without shortening a trial. |
| Truncated recall/precision | Path metrics derived from incomplete output remain null. |
| Old semantics config raises opaque KeyError | Missing output-format entries produce a contextual configuration error. |
Evidence boundaries
The independent reader is optional; the base harness remains stdlib-only. Unsupported database formats, active journals, incomplete object storage and missing reader dependencies produce unavailable evidence, never verified persistence. Unsupported symlink modes can make the independent Oak reader unavailable where the Git reader can inspect that object; measurement availability is not claimed equal. Such rows block publication and cannot form complete paired success estimates. Object hash validation establishes local stored content, not remote publication, fsync durability, crash recovery, or protection from a hostile process rewriting the trusted runner itself.
History timings remain diagnostic. There is no agent answer artifact or scoring
key for history_archaeology; unchanged files are insufficient to declare a
correct answer. The task is therefore explicitly unmeasured on both subjects.
The core timing contract changes with command semantics 2026-09-18.1;
workflow validation is versioned separately. Older baselines cannot silently
pool the changed cache/measurement and correctness policies.
Validation
Final commands, counts and exact-head independent review receipts are retained outside the checkout in its worker directory. Raw campaign transcripts and benchmark outputs are not source artifacts. Both failed reproductions and final results must be preserved; a repeated passing timing run does not erase a noisy or failing earlier run.
Author verification on macOS with pinned Oak 0.103.0
(f63a888ab042ea65edb27be573f4f2b0973cebd2d6c68926325c0544b3986b7f):
- Full instrument suite: 1,156 tests, 25 skips, PASS (174.455 seconds).
- Optional independent reader: 18 real-storage/adversarial tests, PASS.
- Configured real Git/Oak workflow parity: 6 tests, PASS; configured producer publication/report qualification: 6 tests, PASS.
- Identical-binary core devloop: two repetitions plus two A/A repetitions, PASS; effective latency threshold 42.9%. Known Oak patch failure remains pre-existing.
These are local correctness and smoke checks, not a qualified performance study.
Linux/Windows runtime and paid-provider campaigns were not run. Author logs are
in /Users/mrmrs/o/worker-mrmrs-20260918T202621-bench-integrity-fixes/
(full-suite-final.log, reader-final.log, workflow-parity-pinned-head.log,
workflow-qualified-pinned-head.log, devloop.log). Exact-head independent QA
is a separate post-publication step; its receipt must identify the reviewed head.