mrmrs-0038ef

Closed into main

1 commit · 18 files +11 added ~7 modified

Description

Review benchmark validity and add upstream-inspired VCS conformance

Harden agent snapshot/status and filesystem-change oracles with current Oak JSON probes, and reject incompatible measurement populations in reports and scorecards. Add five independently authored local VCS conformance scenarios with negative controls and pinned Git/Jujutsu/Mercurial provenance. Document current testing strategies, prioritized gaps, workstation setup and a proposed human study protocol; clarify statistical helpers that are not production protections.

Validation: 921 unittest tests pass with 3 skips; core smoke has 117 successful rows. Git conformance passes 5/5; Oak 0.103.0 passes 4/5 and rejects tab filenames at commit, retained as a diagnostic failure. No calibrated performance, paid-agent or human claim is made. Baseline compatibility, full durable oracles and campaign statistical controls remain documented followups.

Commits 1

c8b43cc46596

mrmrs 1 month ago

Files 18

docs

A benchmark-workstation.md A human-vcs-study-protocol.md A review-2026-09-09-methodology.md A review-2026-09-09-oracles.md A review-2026-09-09.md M statistical-methodology.md A vcs-conformance.md

scripts

oakbench

A conformance.py A populations.py M scorecard.py

M agent_metrics_schema.py M agent_workflow.py M benchmark_stats.py A vcs_conformance.py

tests

A test_population_reporting.py M test_schema_and_oracle.py A test_vcs_conformance.py

M README.md

@@ -127,6 +127,13 @@

127127 `docs/benchmark-coverage.md`, `docs/publish-checklist.md`, and

128128 `docs/statistical-methodology.md`.

129129

130+The [September 2026 quality review](docs/review-2026-09-09.md) records current

131+testing strategies, major evidence gaps, and independently authored behavioral

132+ports from upstream VCS test suites. Start local development with the

133+[workstation guide](docs/benchmark-workstation.md). Human usability remains

134+unmeasured; the [proposed participant protocol](docs/human-vcs-study-protocol.md)

135+defines the separate study needed to evaluate it.

136+

130137 ## In Development And Roadmap

131138

132139 The benchmark suite is also a product map for Oak. Current gaps are intentional