Benchmark quality review — 2026-09-09
Reviewed base: a9e8fa39d0a097e03fba83c6fcd0029865e2387373e19b57c8f23da03ec173cd.
Integration worker:
/Users/mrmrs/o/worker-mrmrs-20260909T080918-benchmark-quality/bench.
Branch: mrmrs-0038ef. No changes were made to canonical checkouts.
The suite is useful for VCS development, but does not yet establish broad human usability or industry-leading measurement accuracy. Its strongest assets are explicit measurement contracts, diverse lanes, local integration fixtures, and publication safeguards. The largest risks are false-positive correctness oracles, incomplete comparison identity, and statistical promises that exceed what production consumers enforce. This review prioritizes those risks before increasing the volume of timing data.
Evidence and review map
- Methodology audit: accuracy, precision, population identity, sampling, uncertainty, reporting and experiment design.
- Oracle and coverage audit: concrete false-pass paths, lane/test inventory, durable-state checks and missing VCS behaviors.
- Upstream conformance map: inspected primary sources, pinned references, independently authored ports and deferred behavioral areas.
- Workstation guide: prepared binary/config paths, isolated execution, validation and requirements for larger campaigns.
- Human study protocol: proposed participant design, task matrix, observation contract and analysis/publication criteria.
The specialist reviews cite source lines at the reviewed base. They are a source/test audit, not an assertion that every remote or platform cell was executed. Local instrument tests and diagnostic subject runs are separate evidence from a calibrated comparative campaign.
Current testing strategies
| Layer | Existing strategy | What it establishes; remaining limit |
|---|---|---|
| Shared instruments | Stdlib unittest tests for rows, nulls, skips, timing/accounting, environments, fixtures, semantics, locks and schemas | Local contract behavior; passing tests do not validate experimental design |
| Statistics and reporting | Synthetic boundary cases, bootstrap/ratio helpers, scorecard guards, golden Python/Node reporting fixtures | Reproducible calculations; no broad empirical interval-coverage or false-positive validation |
| Core VCS | Deterministic fixture/profile matrix, tuned Git modes, output/recall probes, A/A controls and smoke integration | Scripted command behavior; exact durable content and output parsing need stronger adversarial checks |
| Scripted workflows | Bugfix/refactor/assets/history/recovery recipes, payload checks, local integration | Repeatable task recipes; they do not observe human actions or model decisions |
| Real agents | Mock adapter, recorded stream fixtures, local CLI adapters, task validators and end-state checks | Adapter and oracle mechanics; provider drift, missing resolved model identity and no human participants remain |
| Concurrency and failure recovery | Worker modes, fake providers, local HTTP fixtures, interruption/race tests and explicit skips | Bounded local failure models; broad crash durability and transport-matched hosted evidence remain incomplete |
| Conflict, mount and network | Corpus/fixture generators, Git conflict execution, mount probes, Linux netshape and remote lanes | Useful infrastructure; capability-limited, Git-only or unmeasured cells cannot support cross-subject wins |
| Publication | Provenance, transport, sample/skip checks, reproducibility bundles and baseline/target books | Necessary gates; baseline compatibility and complete campaign-level statistical enforcement remain open |
The original unrestricted baseline ran 903 tests successfully, with three skips. The first restricted run had 18 loopback-socket permission errors; rerunning with loopback access resolved all 18 without code changes. Generated logs are retained outside the tracked repository in the worker directory.
Changes delivered in this review
- Agent oracle checks require observed commit growth for snapshot workflows, fail closed on unsuccessful/unrecognized status observations, and track executable bits and symlink targets. Read-only history work has an explicit exception. Oracle versioning keeps changed evaluation semantics visible.
- Shared population guards prevent the bootstrap report and scorecards from pooling conflicting observed identities within a cell. This does not yet certify compatibility with a saved baseline or enforce every consumer.
- Five independently authored conformance scenarios exercise exact path, byte, mode/symlink, scoped-restore and failed-restore behavior against local Git and Oak. Negative controls challenge the oracles. These are diagnostic correctness records, not performance-lane rows or public trust attestations.
- Documentation now distinguishes adaptive/FDR helpers from implemented campaign protections and identifies required-n as a planning heuristic.
- A prepared local workstation, evidence-backed audits, and a proposed human experiment protocol make the next work concrete and reviewable.
Major gaps and development acceptance gates
Severity here prioritizes risk of an incorrect engineering conclusion, not product security severity. The detailed audits preserve exact evidence.
| Priority | Gap | Acceptance gate for subsequent work |
|---|---|---|
| P0 | Complete durable-state oracle | Independently verify exact committed paths, bytes, types/modes and history; reject no-op snapshots, deletion, extras and partial payloads; Oak worktree checks alone remain insufficient |
| P0 | Structured output and usable patches | Parse exact path records, test escaping/NUL formats, measure false positives as well as recall, and actually apply emitted patches in a fresh fixture |
| P1 | Baseline and consumer identity | Version comparison fingerprints through raw rows, books, targets, reports and dashboard; reject mismatched or insufficient public provenance |
| P1 | Public certainty from too little evidence | Separate diagnostic verdicts from claim eligibility; missing noise, singleton runs and inadequate independent replication cannot yield public GREEN/RED |
| P1 | Agent validity and generalization | Record resolved model/config/budgets and task order; serialize measured trials; hold out tasks/seeds from repeated optimization; count all failed attempts |
| P1 | Human usability | Build event recorder and independent oracle integration, then run the counterbalanced pilot and preregistered study; no proxy human scores |
| P1 | Failure/durability matrix | Add controlled interruption points before/during/after publication and recovery; verify both sides and idempotent retry; test ENOSPC/permissions/locks/corruption without conflating process kill with power loss |
| P1 | Cross-subject semantic coverage | Add conflict resolution, rename/delete, partial snapshot, task switch, history rewrite/undo, ignored/untracked preservation and recovery under explicit equivalent contracts |
| P2 | Statistical validation | Simulate false positives/interval coverage for skew, ties, dependence and drift; implement paired/cluster analysis, declared multiplicity and independent confirmation |
| P2 | Failures, tails and resource cost | Report every attempted outcome and censoring; separate success latency from full attempt cost; require tail interval precision rather than only a sample-count floor |
| P2 | Representative scale and platforms | Materialize and hash real/XL fixtures; replicate across filesystem/platform classes, warmed/cold states, tuned baselines, same-pipe servers and concurrency levels |
| P2 | Reproducible external evidence | Archive raw rows/provenance outside repo; verify bundle reconstruction and analysis independently; distinguish mock, local-loopback and hosted evidence |
Development sequence
First make correctness fail closed and propagate comparison identity. Then validate the instrument's error rates with A/A and simulations, and require complete evidence for public verdicts. Expand behavior using the upstream map and shared independent oracles. Finally run frozen agent and human studies with held-out variants and replicated sessions. Increasing repetitions reduces some sampling error; it cannot repair the wrong oracle, unfair semantics or an unrepresentative task distribution.
The delivered patches address a bounded first set of the gaps above. Remaining items are explicitly open; no performance superiority, human study result or complete VCS conformance is claimed by this review.
Final local verification
python3 -m unittest discover -s tests: 921 tests, OK, three skipped after the final load-tier compatibility fix (60.990 seconds). Log:../final-tests.login the integration worker; no raw log is committed.- Core workstation smoke: 117 rows, all returncode zero, across Git and copied
Oak 0.103.0 with one repetition and remotes disabled. Ignored rows:
results/workstation-smoke/20260909T181900Z.jsonl. These timings are diagnostic, not a calibrated comparison or public performance evidence. - New conformance: Git five passes; Oak four passes and the expected observed
failure at tab-filename commit. The runner exits 1 and retains the failure.
Raw diagnostic records:
../conformance-final.jsonloutside the checkout. A stopping failure does not certify later paths in that same scenario. - Current Oak structured probes: an isolated two-commit fixture observed count 1 to 2 and clean/dirty/clean status 0 to 1 to 0. This challenged the human-output parser assumptions discovered during independent review.
- Independent cross-reviews challenged each implementation; relative-binary resolution, current Oak JSON probes and load-tier identity were corrected. Human-protocol review clarified unsuccessful-attempt and attrition analysis.
The passing instrument suite does not erase the subject's conformance failure or turn the open roadmap into implemented coverage. Hosted, paid-agent, human, crash-durability and calibrated comparative campaigns were not run here.