Workflow correctness readers

Both lanes use oakbench.workflow_oracles for committed task content. The two large config/asset tasks validate exact path sets and bytes. Six other mutating workflows validate the same scoped committed payload markers for both VCSs; these are narrower checks, not whole-repository or behavioral proofs. The agent bugfix task additionally retains its test suite. sync_pull_dirty requires the local dirty edit in the worktree and upstream content in HEAD; it does not require committing the deliberately uncommitted edit.

history_archaeology currently has no answer artifact or answer-scoring key. Its command timings remain diagnostic, but the task-success oracle returns unmeasured for both subjects. An untouched tree alone does not prove a correct historical answer. A future answer contract needs a scenario/oracle version and negative controls for absent and incorrect answers.

Configure the readers

The default Git reader is git on PATH; set OAK_BENCH_GIT_INSTRUMENT to an absolute, operator-qualified Git reader when measuring another Git build. The reader's version and SHA-256 are recorded separately from the subject. Identical binary bytes are labeled subject_binary_committed_tree (or _payload), not independent. A copied binary or wrapper does not establish implementation independence; executable hashes record identity, not certification.

The default Oak reader uses Oak export and is explicitly subject-produced. It can detect a correct worktree with an incorrect HEAD but does not qualify as independent evidence. To read Oak's local SQLite v1 objects without invoking the Oak executable, configure a Python interpreter containing blake3 and zstandard:

python3 -m venv /absolute/worker/reader-env
/absolute/worker/reader-env/bin/python -m pip install blake3==1.0.9 zstandard==0.25.0
export OAK_BENCH_OAK_READER_PYTHON=/absolute/path/to/reader-env/bin/python
export OAK_BENCH_GIT_INSTRUMENT=/absolute/path/to/qualified/git

The harness invokes its pinned oak_sqlite_reader.py with Python -I and a new temporary destination. The reader checks object hash bindings and materializes bounded content from a stable private copy of a quiescent SQLite database. An active WAL or journal makes the read unavailable. The same task validator then checks the materialized content. Rows record reader code/version, Python binary hash, dependency versions and the observed HEAD. Unsupported storage formats or missing instruments remain unmeasured; the harness never silently falls back to subject export when an independent reader was requested. The reader supports local complete SQLite objects; chunk-only storage and other unqualified formats remain unavailable. This does not certify remote durability, crash recovery or all Oak storage variants.

Publication qualification

Successful compared subjects must have matching correctness evidence classes. A measured agent failure before committed-tree validation remains failure evidence; it is not required to invent successful persistence proof. Harness failures, unlaunched tasks and missing measurements cannot count as measured agent failures. Comparison identity includes correctness class; paired analysis must handle pre-HEAD failures explicitly rather than discarding their denominator.

Subjects must cover matching experimental-condition sets. Different models, instructions, tracks, environments or transports cannot disappear into singleton groups and escape the check. Workflow/agent publication additionally requires consistent environment identity, observed randomized subject order, distinct trial identities and at least 30 recorded trials per cell. This mechanical minimum is not a power analysis, proof of independent task sampling, or permission to claim a speedup. Exact-output evidence checks also apply outside core profiles. A campaign remains diagnostic until the study's separate qualification criteria are met.