Workflow correctness readers
Both lanes use oakbench.workflow_oracles for committed task content. The two
large config/asset tasks validate exact path sets and bytes. Six other mutating
workflows validate the same scoped committed payload markers for both VCSs;
these are narrower checks, not whole-repository or behavioral proofs. The agent
bugfix task additionally retains its test suite. sync_pull_dirty requires the
local dirty edit in the worktree and upstream content in HEAD; it does not require
committing the deliberately uncommitted edit.
history_archaeology currently has no answer artifact or answer-scoring key.
Its command timings remain diagnostic, but the task-success oracle returns
unmeasured for both subjects. An untouched tree alone does not prove a correct
historical answer. A future answer contract needs a scenario/oracle version and
negative controls for absent and incorrect answers.
Configure the readers
The default Git reader is git on PATH; set OAK_BENCH_GIT_INSTRUMENT to an
absolute, operator-qualified Git reader when measuring another Git build. The
reader's version and SHA-256 are recorded separately from the subject. Identical
binary bytes are labeled subject_binary_committed_tree (or _payload), not
independent. A copied binary or wrapper does not establish implementation
independence; executable hashes record identity, not certification.
The default Oak reader is the independent oak_sqlite_reader.py: it reads
Oak's local SQLite v1 objects without invoking the Oak executable and is
stdlib-only. BLAKE3 (oakbench/pureblake3.py) and zstd (oakbench/purezstd.py,
RFC 8878 decoder with XXH64 checksum) are this repository's own pure-Python
implementations, so verification shares no code with the Rust blake3 crate
or libzstd that Oak links. Oak export (subject-produced evidence) is used only
when explicitly requested with OAK_BENCH_OAK_COMMITTED_READER=export.
OAK_BENCH_OAK_READER_PYTHON still selects the interpreter; optional
OAK_BENCH_READER_BACKEND=native uses the blake3/zstandard packages for
speed (they wrap the same libraries Oak uses; rows record oak_reader_backend
and oak_reader_codec_sha256).
export OAK_BENCH_GIT_INSTRUMENT=/absolute/path/to/qualified/git
The harness invokes its pinned oak_sqlite_reader.py with Python -I, a new
temporary destination and --scope set to exactly the committed paths the Git
check reads (workflow_oracles.committed_scope): out-of-scope entries are never
decoded, so repo-wide conditions a task does not own cannot fail one reader and
not the other. In-scope non-regular entries are reported (nonregular_paths)
and fail the payload the way Git's mode check does. Reader resource and time
budgets (object/total/entry bounds, depth, deadline) are harness limits and map
to unmeasured (exit 77), never to an Oak correctness failure. Subject-controlled
state is a measured failure, as on the Git arm: a missing or linked
.oak/oak.db, or a repository with no committed HEAD. In-scope blobs over the
task's size limit (--max-blob-bytes) and colliding paths are reported and fail
with Git's reasons before any reader bound. Newer schema migrations (than
0014_change_captures) or a blob-hash convention that no sampled blob satisfies
are unrecognized formats (unmeasured). With the format pinned, an Oak-written
HEAD naming a missing commit row (a dangling HEAD) is an Oak failure. The format pin is
the required table/column set plus verification of the HEAD commit preimage and
root tree serialization; a mismatch there is an unrecognized format
(unmeasured), while deeper hash mismatches after a verified root are corrupt. The reader checks object hash bindings and materializes
bounded content from a stable private copy of a quiescent SQLite database. An active (non-empty) WAL or journal makes the read unavailable; a zero-byte one holds no frames and is ignored. A hash mismatch on a recognized format (any other commit, tree or blob verifies) is Invalid; only a store in which nothing verifies is an unrecognized format. Oracles never fall back to the worktree when the reader is unavailable: the check is unmeasured. The same task validator
then checks the materialized content. Rows record reader code/version, Python
binary hash, dependency versions and the observed HEAD. Unsupported storage
formats or missing instruments remain unmeasured; the harness never silently
falls back to subject export when an independent reader was requested. The
reader re-hashes every decoded object: blobs, trees (canonical
kind\tname\thash\tmode lines with Oak's name rules, strict order and
kind/mode pairs), the HEAD commit (v1 7-field preimage, the legacy 6-field
preimage and the server's microsecond-truncated timestamps, exactly as Oak
verifies them), and chunk-only blobs reassembled from blob_chunks/chunks
(contiguous offsets, per-chunk and whole-blob hashes). --deep also re-hashes
every ancestor commit (absent parents are reported as a shallow boundary) and
cross-checks the chunk map of complete blobs; --complete makes absent content
Invalid rather than unavailable. Unscoped reads materialize symlinks after all
regular files (no write can pass through a created link); --manifest / -
emits an exact path/mode/size/SHA-256/link-target manifest without touching the
filesystem. Other unqualified formats remain unavailable. This does not certify remote durability,
crash recovery or all Oak storage variants.
Publication qualification
Successful compared subjects must have matching correctness evidence classes. A measured agent failure before committed-tree validation remains failure evidence; it is not required to invent successful persistence proof. Harness failures, unlaunched tasks and missing measurements cannot count as measured agent failures. Comparison identity includes correctness class; paired analysis must handle pre-HEAD failures explicitly rather than discarding their denominator.
Subjects must cover matching experimental-condition sets. Different models, instructions, tracks, environments or transports cannot disappear into singleton groups and escape the check. Workflow/agent publication additionally requires consistent environment identity, observed randomized subject order, distinct trial identities and at least 30 recorded trials per cell. This mechanical minimum is not a power analysis, proof of independent task sampling, or permission to claim a speedup. Exact-output evidence checks also apply outside core profiles. A campaign remains diagnostic until the study's separate qualification criteria are met.