Commit 3354584a · oak/benchmarks

· Browse files at this commit

3354584a5d91

Merged from mrmrs-250abb

mrmrs · 1 month ago · 37 files +12 added ~25 modified

Description

P2 independent correctness: stdlib Oak reader wired into every Oak oracle, roundtrip conformance matrix (61 behaviours, git + oak), gated in devloop and publish_gate

Program lane P2 (oak/benchmarks). Branched from main b00752e4, synced onto main 5dd811a1 (P3 merge suite; docs/prior-art.md keeps both sections), then onto main 39bc21f7 (P4 robustness; no overlap). Worker: /Users/mrmrs/o/worker-mrmrs-20260928T133136-p2-correctness/benchmarks. Not merged.

1. Independent reader (audit + hardening)

Audit of main's oak_sqlite_reader.py (reader 2026-09-27.3):

It was NOT stdlib: it needed the blake3 + zstandard packages. On the stock macOS Python 3.9 it was always "unavailable", and every reader test was skipped. In practice every Oak oracle fell back to oak export, which is evidence produced by the subject itself.
What it did do: re-hash blobs, the root and nested trees, and the HEAD commit, with format pins.
What it could not handle: symlinks (unavailable), chunk-only blobs (unavailable), commit ancestry, and server commits with truncated timestamps.
False positive: it rejected legal backslash file names as "unsafe".
Its name rules did not match Oak's (Oak rejects control characters).

New:

oakbench/pureblake3.py: stdlib BLAKE3, written from the spec. It matches all 25 reference vectors.
oakbench/purezstd.py: stdlib RFC 8878 zstd decoder with the XXH64 checksum.
- It byte-matches all 400+ blobs and 199 trees of a real Oak DB, and the zstd CLI at levels 1 to 22 including --long.
- No stray exception in 24k random corruptions.
- Neither codec shares code with the Rust blake3 crate or libzstd that Oak links.
Reader 2026-09-28.2:
- Defaults to the pure backend; OAK_BENCH_READER_BACKEND=native opts into the packages.
- Re-hashes every object.
- Commits: v1 7-field, the legacy 6-field form, and the server's microsecond-truncated timestamps (the same rules Oak uses).
- Oak's exact name rules; strict ordering and kind/mode checks.
- Reassembles chunk-only blobs, with per-chunk and whole-blob hashes and contiguous offsets.
- --deep: re-hashes every ancestor commit (a shallow boundary is reported, not failed) and cross-checks chunk maps.
- --complete: absent content counts as Invalid.
- Symlinks are materialized last, so no write can pass through a link.
- --manifest / - gives an exact manifest without touching the filesystem.
- Format pin (QA round 1): the format counts as recognized when the schema is pinned and ANY stored commit, tree or blob verifies. A HEAD commit, root tree or blob hash mismatch on a recognized store is Invalid. Only a store where nothing verifies is "unrecognized format" (77). QA's own tamper and oracle harnesses now report Invalid for all three former false-PASS cases.
- A zero-byte -wal/-journal is ignored. task_loop's stat_cache snapshot opens the DB read-only (mode=ro), so it no longer leaves one behind.
- The deadline is 60 s plus 1.5 s per MiB actually verified. The workflow-oracle subprocess budget is 120 s plus 3 s per MiB of oak.db.
- zstd matches may reach only the current frame and window, as in libzstd; verified against the zstd CLI.

2. Oracles wired to the reader (no Oak binary in the check)

When the reader is unavailable, every oracle reports UNMEASURED and never falls back to the worktree:

integrity: content_integrity_check_passed is null and content_integrity_unmeasured_reason is set; bench.py and workflow_ab treat null as unmeasured, not failed.
contention: returncode 77 with a skip_reason.
task_loop: oracle null, or returncode 77.
workflow_oracles.task_evidence: the default Oak arm is now the independent reader (sys.executable -I). oak export runs only on the explicit opt-in OAK_BENCH_OAK_COMMITTED_READER=export. Oracle version 2026-09-28.2.
workflow_ab: non-trusted workflows had no committed read for Oak at all. They now get the same scoped committed-payload check as the Git arm.
integrity.content_integrity_fields: Oak snapshot content was read from the worktree, so it could never fail. It now reads committed HEAD through the reader. oak_independent_reader_payload_sha256 is a strong, public-trust source.
parallel_contention:
- Lost-update payloads come from committed HEAD.
- A deep reader verify replaces "oak has no fsck equivalent".
task_loop: verifier clones check committed payload and HEAD through the reader; oak hash must agree with it.
Legacy conformance.py: Oak identity (status) must equal the reader's HEAD, branch and root tree. Oracle renamed exact_tree_and_logical_identity_v2.
Git differential: every matrix Oak row records git_differential, comparing Oak's independently read tree with Git's committed tree for the same fixture.

3. Conformance matrix (`oakbench/conformance_matrix.py`, docs/vcs-conformance.md)

Scale: 56 new behaviours plus the 5 legacy ones = 61 IDs.

Flow for each behaviour:

Author the fixture.
Commit and push after every step (Git: bare repo; Oak: private loopback oak serve).
Fresh clone.
For multi-step behaviours, also pull into an early clone.

Legs:

commit_read@step (independent reader)
source_preserved@step
clone_fs, clone_read, clone_status_clean
pull_fs, pull_read, pull_status_clean

Coverage:

Paths: tab, newline, control characters, leading dash, metacharacters, backslash, NFC, NFD, NFC+NFD pair, non-UTF-8, 255-byte components, long and deep paths, reserved names, case-only rename, case-distinct names.
Exec bit add and remove.
Symlinks: relative, absolute, dangling and loops, to a directory, edge targets, retarget.
Type transitions: symlink to file, file to symlink, directory to symlink, symlink to directory, file to directory, directory to file.
Empty files and empty-directory semantics (observed, never failed).
Delete-all, all byte values.
Line endings: LF, CRLF, CR, mixed, no final newline, LF-to-CRLF, whitespace-only edits, BOMs.
Binary detection, same-size edit with the mtime restored (racy stat), 10 MiB chunked file, 1 GiB sparse file (gated).
Delete then re-add, rename, a blob shared across modes, 1,200 small files.

Harness details:

Git runs byte-exact config and the real Git, not the xcrun shim.
Capability skips are explicit.
Prior art: re-implemented from the behaviour of Git tests t0010/t0020/t0021/t0027/t0050/t2003/t2007/t2021/t3700/t3902/t3910/t4011/t4012/t4102/t4114/t4129/t7001, with IDs verified against v2.50.1. The dulwich compat pattern is attributed in docs/prior-art.md; no text was copied.

4. Gate (schema 2, after QA round 1)

config/conformance_expected_failures.json entries each have:
- a filed ticket;
- an outcome: bug, or refused for a typed policy refusal (fb-531);
- per failing leg, an error pattern checked with re.fullmatch against that leg's detail;
- optionally, fs_profiles.
Coverage: coverage.<fs_profile>.expected_skips. The current profile is darwin-ci-ni, APFS case-insensitive and normalization-insensitive.
oakbench/conformance_gate.py verdicts:
- Regression: a different leg set, OR the same legs with a different error. Examples: EEXIST becoming extra=['../../etc/evil'], a refusal becoming a panic, the legacy single leg hiding a new failure.
- FIXED: a pass, but only where the entry applies. A profile-scoped entry passing on Linux is a plain pass.
- UNMEASURED: coverage loss (pass or xfail turned into a skip) on a recorded profile, mixed profiles, a harness error, a reader-unmeasured leg, or a missing row.
Skips can never add up to a PASS (QA round 2):
- A row without fs_profile is UNMEASURED.
- Zero measured Oak behaviours is always UNMEASURED.
- On an unrecorded profile, only behaviours needing a known filesystem-dependent capability (case sensitivity, Unicode normalization, non-UTF-8 names) or gated behaviours may skip; anything else (e.g. a broken symlink probe) is UNMEASURED.
- publish_gate also requires the host profile to have a recorded coverage entry.
outcome: intermittent covers timing-dependent bugs: a pass is a pass (never FIXED), the pinned failure is INTERMITTENT, and any other failure is a regression. Tickets are fb-N; pending:<slug> is allowed only until the ticket is filed, and every gate summary lists unfiled tickets.
The gated 1 GiB behaviour with HUGE=1 is a recorded gated skip: filesystem legs are measured, reader legs are over the bound by design. It is no longer an unconditional UNMEASURED.
The real failure details from oak 0.105.0 are pinned in tests/data/conformance_oak_0105_failures.json.
devloop runs a conformance lane by default.
publish_gate check 14:
- Every Oak row needs binary identity (binary_sha256 or subject_details).
- The gate is evaluated per published Oak binary on that binary's rows only, so other builds' rows can neither satisfy nor mask it.
- It fails when a binary does not PASS or when the expected-failures list is stale.
public_readiness picks up results/conformance/latest.jsonl; vcs_conformance.py --gate exits 0/1/2.
The loopback oak serve is stopped by process group (killpg), so a server behind a wrapper script can't leak.

5. Results: oak 0.105.0 vs git 2.50.1 (Apple Git-155), macOS APFS (case- and normalization-insensitive)

Oak (final run, combined tree synced to 39bc21f7): 47 pass, 6 expected-fail (bugs), 4 policy-refused (fb-531), 0 intermittent-fail, 4 skip (3 need filesystem capabilities, 1 gated). Gate (schema 2): PASS. large_file_chunked_v1 passed this run; it failed intermittently in 2 of 7 runs (see below). Every failure is reproduced standalone. Git: 56 pass, 1 fail, 4 skip. The Git failure is its known behaviour on this filesystem: git add -A with ignorecase does not stage a case-only rename. Every Oak behaviour that committed has git_differential = identical, except where noted below. Gated 1 GiB sparse file:

Git: passes every leg.
Oak: all filesystem legs pass. Both reader legs are unmeasured because the file is over the reader's 32 MiB bound, so the row is recorded as a gated skip.

Oak failures (repro scripts in /Users/mrmrs/o/worker-mrmrs-20260928T133136-p2-correctness/repros/):

Behaviour	Failing legs	Evidence	Repro
path_tab_v1, path_roundtrip_v1	commit (refused)	fb-531 (policy): `cannot track "tab\tname"`	r1
path_newline_v1, path_control_chars_v1	commit (refused)	fb-531 (policy): newline, CR, DEL and other control characters rejected	r1
path_backslash_v1 (fb-543)	commit_push	`back\slash` is committed as `back/slash` and source status stays clean. With `trailing\`, `oak push` rejects the commit Oak just made	r2
path_long_component_v1 (fb-544)	clone_fs	a 255-byte name commits and pushes, then clone fails with "File name too long (os error 63)"	r3
path_case_only_rename_v1 (fb-545)	pull_fs, pull_status_clean	pulling File.txt -> file.txt deletes the file from the worktree (a fresh clone is fine)	r4
symlink_to_dir_real_v1 (fb-542, SECURITY)	pull_fs, pull_status_clean	pull writes p/inner.txt through the old symlink into its target and drops p. With an absolute target it writes outside the checkout	r5, r5b
file_to_dir_v1 (fb-546)	pull_fs	pull fails "File exists (os error 17)"	r6
file_to_deep_dir_v1 (fb-546)	pull_fs	pull fails "Not a directory (os error 20)"	r6b

New intermittent Oak bug found in QA round 2 (fb-547)

large_file_chunked_v1 failed 2 of 7 full matrix runs (and 1 of 6 isolated runs):

oak push exited 6 with "Server error: Failed to upload chunk: HTTP 500 Internal Server Error: Database error: database is locked" from the loopback oak serve.
The client does not retry.
Standalone repro: repros/r7-serve-chunk-upload-database-locked.sh. It failed 0 of 30 sequential attempts, but 3 of 3 with three copies running in parallel (each with its own server).
Hypothesis, not confirmed: serve opens a new SQLite connection for each upload request, runs PRAGMA journal_mode=WAL and its open path, and has a 5 s busy timeout. Concurrent chunk uploads on a contended disk hit SQLITE_BUSY, which surfaces as a 500.
Gated as outcome: intermittent with ticket fb-547.

Tests

Full suite python3 -m unittest discover -s tests on the combined tree (synced to main 39bc21f7, bare stdlib Python 3.9.6, oak 0.105.0, real Git): 1,361 OK, 7 skipped. On 5dd811a1 after QA round 2: 1,321 OK. Round 1 on the combined tree: 1,315 OK; before the sync: 1,276 OK. Main at P1 land was 1,209 OK, 39 skipped.

New test files:

tests/test_pure_codecs.py: vectors, per-byte corruption, live zstd CLI differential.
tests/test_oak_sqlite_reader.py: rewritten. A synthetic Oak-format DB means every tamper case runs without the Oak binary:
- blob bytes, compressed frame, and size
- nested tree bytes and missing tree
- ancestor commit
- chunk content, chunk gap, and chunk map vs blob
- absent content
- structural rules
- the symlink-escape guard
tests/test_conformance_matrix.py: every behaviour must pass a faultless copy-VCS and fail each injected fault (commit, source, clone, pull, status) at the named leg; plus catalog and citation checks and real Git/Oak runs.
tests/test_conformance_gate.py: gate semantics and devloop/publish_gate wiring.
tests/test_independent_oracle_wiring.py: covers reader-unavailable becoming unmeasured in integrity, contention and task_loop.
QA round 1 additions:
- pinned-format tamper tests (synthetic and real store): root tree, HEAD commit, all blobs corrupt; plus nothing-verifies returning 77;
- zero-byte versus non-empty WAL;
- no journal left after a stat_cache snapshot;
- zstd cross-frame and window-overrun rejection;
- killpg of a server behind a wrapper script;
- QA round 2: all-skip on an unrecorded profile, missing fs_profile, zero measured, and unrecorded-profile skip floor are UNMEASURED; publish_gate requires a recorded profile;
- gate v2 semantics: changed error, changed legs, profile scoping, coverage loss, mixed profiles, per-binary publish_gate, binary identity required.

Updated tests: parity matrix (the export arm is now opt-in), workflow oracles, publish_gate, public_readiness.

Not done / caveats

The bench.py remote.clone.cold row still has no tree check.
The agent lane still uses the subject's own status and log for the final_vcs_clean and snapshot_created checks.
The pure BLAKE3 runs at about 1.7 MB/s. Reader bounds (32 MiB per object, 128 MiB total) make larger reads unmeasured. The deadline now scales with bytes verified; there is no sampled-verify mode.
coverage is recorded only for darwin-ci-ni. Other profiles can run devloop (known filesystem-dependent skips only) but cannot publish until someone records their coverage.
Wall times in conformance rows are diagnostic only. The host was I/O-contended by other lanes: git add used 0.6 s of CPU but took 14 s of wall time.
Behaviours that need a case-sensitive or normalization-sensitive filesystem, or non-UTF-8 names, were not exercised on this APFS host.

Files 37

config

A conformance_expected_failures.json

docs

M independent-workflow-readers.md M prior-art.md M vcs-conformance.md

scripts

oakbench

M agent_campaign.py M conformance.py A conformance_gate.py A conformance_matrix.py M integrity.py M oak_sqlite_reader.py A pureblake3.py A purezstd.py M workflow_oracles.py

M agent_metrics_schema.py M bench.py M devloop.py M parallel_contention.py M public_readiness.py M publish_gate.py M repro_bundle.py M repro_verify.py M task_loop.py M vcs_conformance.py M workflow_ab.py

tests

data

A blake3_vectors.json A conformance_oak_0105_failures.json A zstd_vectors.json

A test_conformance_gate.py A test_conformance_matrix.py A test_independent_oracle_wiring.py M test_oak_sqlite_reader.py M test_oracle_parity_matrix.py M test_public_readiness.py M test_publish_gate.py A test_pure_codecs.py M test_workflow_oracles.py

M AGENTS.md

Details

Commit3354584a5d91

Parent39bc21f7f67e

Merge parentd234051be762

M AGENTS.md

@@ -97,6 +97,12 @@

9797 - **Run with your own results dir when working in parallel**

9898 (`--results results/<your-task-slug>`), and leave `results/latest.*` to

9999 serialized verdict runs.

100+- **Correctness is gated, not only speed.** devloop's `conformance` lane runs

101+ the roundtrip conformance matrix (`scripts/vcs_conformance.py`, see

102+ `docs/vcs-conformance.md`) against your Oak build. Oak state is checked by the

103+ independent stdlib reader (`oakbench/oak_sqlite_reader.py`), never by the

104+ binary under test. A behaviour that newly fails is REGRESSED; if you fix a

105+ listed failure, remove its entry from `config/conformance_expected_failures.json`.

100106 - **Skips are work items, not noise.** A returncode-77 row names exactly

101107 what infrastructure would unlock a measurement (see the "Unmeasured"

102108 section of opportunities.py). Unlocking coverage is as valuable as