/oak
Log in
benchmarks
docs/gap-closure-2026-09-10.md 135 lines · 7.9 KB

Benchmark gap closure — 2026-09-10 UTC

This document records the initial follow-up to the quality review. Its historical integration branch was mrmrs-benchmark-gaps-20260910, in /Users/mrmrs/o/worker-mrmrs-20260910T005814-benchmark-gaps/bench. That path is an evidence locator, not a recommended live checkout. Use the portable workstation guide for a new run.

Later QA found additional instrument and reporting defects, including agent-native probe failures, unavailable-oracle classification, patch evidence requirements, and campaign budget/recovery behavior. The historical test and smoke counts below do not validate the revised implementation. Current combined verification must be recorded separately after the integrated code is frozen.

Trusted task correctness

The config and asset workflows now use a shared validator in scripts/oakbench/workflow_oracles.py, outside the editable task checkout. Agent evaluation invokes it through an absolute path with Python isolation; neither workspace imports nor an edited fixture validator can replace it.

Config validation checks all 600 exact paths and canonical final bytes, including both the 200 intended timeout edits and the 400 unchanged configs. Asset validation checks all four identities, exact sizes/content hashes, manifest shape, unique entries/keys and reviewed flags. File type/mode and preserved fixture validators are checked. Empty inventories, partial inventories, extra paths, duplicate entries, same-size corruption and validator replacement cannot satisfy the task oracle. Both scripted and real-agent runners consume the same trusted rules, with versioned evaluation semantics.

For Git, the final task check additionally reads the complete scoped committed tree using a configured known-good Git instrument. It enumerates paths/modes and reads bounded blob batches by object ID, bypassing checkout filters and replacement objects. A correct worktree with an incomplete commit fails. Oak still has trusted worktree checks plus its existing metadata evidence; this change does not certify Oak's independently reconstructed committed tree or crash durability.

Exact output and patch evidence

Information recall and precision use exact decoded path sets from supported metadata formats. A substring in another filename or a mention inside a diff hunk is not a recovered path. Output-oracle versions distinguish these scores from historical permissive matching; do not join old and new scores into one trend population.

Full-diff probes apply the actual emitted patch bytes to a disposable pristine fixture with an external Git instrument, then compare the exact expected tree. This work is outside subject timing. Malformed/truncated patches, missing file sections and a successful apply that produces the wrong tree cannot pass. The historical pipe_compatible_unified flag remains a structural observation, not sufficient applicability evidence. Binary notices alone do not reconstruct binary content. Bounded or unavailable instrument checks remain unmeasured.

The publication gate rejects unversioned/legacy recall and precision evidence. An applicability claim requires a successful, versioned exact-tree patch application from the same output. The current producer marks that requirement on text diff.full probes; diff.dirty structural observations alone do not require patch-application proof. False, skipped or absent patch evidence is not permission to describe the output as applicable. Legacy and capability-limited rows remain available for diagnosis; publication inputs must support the claims they carry.

Saved baselines and claim eligibility

Baseline Book schema 2 retains comparison identity and complete outcome/sample accounting. Candidate versus saved baseline checks reject mismatched semantics, fixture/oracle versions, runner class, cache/load state, transport/server and applicable agent/calibration identity. Different VCS binary identities remain intentional comparator differences; mixtures within one subject still fail.

The scorecard CLI defaults to claim mode. Missing identity, unknown calibration, insufficient distinct trials, duplicate trials or incomplete outcomes prevent publicly decisive use. Legacy books require explicit diagnostic mode or a new measured campaign; the builder does not infer historical evidence from the machine doing the summarization. Diagnostic results remain explicitly claim_eligible: false. See ADR-0009 for the contract and migration rules.

Confidence levels are preserved in schema 2 books and scorecard results. Claim mode requires both baseline and candidate confidence of at least 95% and below 100%; missing or weaker baseline intervals cannot masquerade as 95% evidence. Invalid bootstrap/confidence settings are rejected. This is an evidence floor, not demonstrated interval coverage or a statistical-power guarantee.

The bootstrap report now retains failure-only and unmeasured cells and prints outcome denominators before success-conditioned statistics. Low successful latency cannot conceal the number of failed or skipped attempts.

Remaining work

  • Independent full committed-content verification for Oak, graph correctness, crash/disk-failure recovery and transport-matched fresh-reader validation.
  • Trusted task-specific answer/state oracles beyond config/assets; the current work does not certify history answers or every recovery scenario.
  • Controlled real-agent campaigns and resolved model/budget identity. The later campaign runner implements frozen schedules and measurement serialization, but those mechanisms alone establish no real-agent performance evidence.
  • A human event recorder and participant pilot under the proposed study protocol; no human performance is inferred from scripted or mock-agent runs.
  • Paired/cluster uncertainty, empirical coverage validation, multiplicity and independent confirmation. Distinct trial IDs and a sample floor are not proof of independence, statistical power or adequate tail precision.
  • Broader filesystem/path grammars, binary patch contracts, large fixtures and platform coverage. The new checks do not imply complete VCS conformance.

No speedup or industry-leadership claim follows from passing instrument tests.

Historical verification of the initial gap-closure snapshot

These are the results recorded for the initial branch above, before subsequent QA corrections. Relative artifact paths refer to that historical worker and may no longer be available. They are not the final QA results for the current code.

  • Initial combined instrument suite: 973 tests, OK, three skipped, in 77.856 seconds. Log retained outside source at ../final-tests.log.
  • Scripted and mock-agent config/assets checks passed for real Git and Oak; adversarial tests reject empty/missing/extra/corrupt payloads, validator tampering and correct-worktree/incomplete-Git-commit states.
  • Integrated micro core smoke against Git and the copied Oak 0.103.0 binary: 39 rows, 38 successful and one retained Oak diff.full.inforecall failure. Both subjects scored exact recall and precision 1.0. Git's patch applied; Oak's did not because its output omitted the no-final-newline marker. The CLI exited 1 rather than hiding the failed correctness probe. Raw rows are ignored at results/gap-smoke/20260910T011855Z.jsonl; this one-repetition smoke is diagnostic, not a performance comparison.
  • Independent reviews challenged trusted oracle isolation, saved comparison identity, confidence metadata, output parsing, patch fixture preparation and publication evidence. Regression tests cover the discovered marker/delimiter mistakes, instrument-cap null handling, malformed flags and omission of a previously measured target patch row.

These historical results exercised that snapshot and recorded an observed subject-output defect. They do not close later QA findings, certify the current harness or Oak durability, or substitute for human/real-agent campaigns.