Oracle and coverage audit — 2026-09-09

# Oracle and coverage audit — 2026-09-09

Scope: source review of benchmark lanes, instrument tests, shared correctness

checks, and current coverage documentation at base

`a9e8fa39d0a097e03fba83c6fcd0029865e2387373e19b57c8f23da03ec173cd`.

This is a source audit, not a new performance campaign. References below are

repository-relative; agent-workflow line numbers refer to this review's fixes.

## Assessment

The suite has substantial measurement discipline: shared row contracts,

null-honesty, subject provenance, tuned Git baselines, runner identity,

measurement locking, calibration, A/A controls, and public-claim gates. Its

strongest evidence is deterministic CLI cost and instrument behavior. It does

not yet establish industry-leading VCS correctness, real-agent generality, or

human usability. Those are separate claims requiring additional experiments.

## Current testing strategies

| Strategy | Existing evidence | Limit |

| --- | --- | --- |

| Unit and contract tests | `tests/test_rows.py`, `test_lane_contract.py`, `test_tokens.py`, `test_command_semantics.py`, `test_schema_and_oracle.py` | A valid row does not prove the subject performed the requested operation. |

| Golden and replay fixtures | `tests/fixtures/*stream*.jsonl`, `test_stream_adapters.py`, `test_reporting_golden.py` | Adapter compatibility is bounded by captured CLI formats; old traces do not validate a newly released CLI. |

| End-to-end Git and mock-agent checks | `tests/test_integration_lanes.py:39`, `:133`, `:165` | Real Git core/contention plus deterministic mock agent; not paid-agent statistical evidence. |

| Corruption/false-success checks | `test_integrity_attestation.py`, `test_review_fixes.py`, task-loop content and publication tests | Integrity strength differs by subject and lane. |

| Statistics and claim gates | `test_stats.py`, `test_stats_upgrades.py`, `test_public_claim_gate.py`, `test_publish_gate.py`, `test_devloop.py` | Correct math requires an appropriate experimental population and sufficiently strong success oracle. |

| Isolation and instrumentation | `test_environment.py`, `test_runlock.py`, `test_byteproxy.py`, `test_netshape.py`, `test_cachectl.py` | Capability-dependent tests can skip; loopback instrumentation is not hosted-network evidence. |

| Conflict resolution corpus | `test_conflict_resolution_lane.py` | `scripts/conflict_resolution_lane.py:742` explicitly skips non-Git subjects. |

| Real-agent campaign support | `scripts/agent_workflow.py`; instruction-level and adapter metrics | Repeat campaigns remain necessary for each model/CLI/instruction condition. Mock rows cannot substitute. |

The audit found over 900 `test_*` method definitions. Test count is an inventory

indicator, not semantic coverage or a claim that all tests passed.

## Proven oracle flaws fixed in this branch

1. **Status failure looked clean.** `scripts/agent_workflow.py` (`vcs_status`) previously

   ignored exit status; empty failed output returned zero dirty paths. Oak also

   accepted empty successful output as clean. Failed or unrecognized output now

   produces `null` dirtiness and fails the clean-final-state check. The schema

   explicitly permits the unmeasured value.

2. **Snapshot evidence was redundant.** `run_oracle` accepted

   `commit_delta > 0 or dirty_count == 0`. Once cleanliness passed, the snapshot

   check could not fail. Mutating tasks now require a positive observed commit

   delta; the read-only archaeology task is exempt. This remains a count-based

   oracle, not proof of exact committed payload or amendment support.

3. **File metadata escaped change-shape checks.** `workspace_manifest` hashed

   regular-file bytes only and followed links. It now records file kind,

   executable state, and literal symlink targets, including dangling links.

   This detects mode-only changes and retargeting between equal-content files.

Oak status and history observations now use validated `--json` outputs; the

old text log parser did not recognize Oak 0.103.0 compact history. A local

0.103.0 probe verified count 1 → 2 and clean → dirty → clean across two

commits. The targeted schema/agent/integration suite passed 23 tests. A full

sandboxed run executed 907 tests but had 18 local-socket permission errors and

three skips; it is not a passing full-suite result. Integration verification

must run where loopback binding is permitted.

Negative tests reproduced six false-pass assertions before correction. Agent

`scenario_version` now includes `ORACLE_VERSION`, keeping changed success

semantics distinguishable from old pass rates without changing fixture bytes.

## Remaining major gaps and acceptance tests

### P0 — Complete task and durable-content oracles

- `scripts/workflow_ab.py:1143` checks surviving config files and one sample.

  Deleting most configs can evade this payload check. The generated validator

  also enumerates surviving files (`make_wide_config_fixture`). Require the

  exact expected path set and expected configuration values; test deletion of

  a targeted config, deletion of an untargeted config, and changes to retries.

- `scripts/workflow_ab.py:1155` accepts an empty asset list through `all([])`.

  `make_large_asset_fixture`'s validator also loops only over submitted entries.

  The agent may change `manifest.json`, so an empty assets list can satisfy its

  current task validator. Require all four exact asset identities, sizes, and

  reviewed values; reject absent, duplicate, extra, reordered-if-order-matters,

  and malformed entries. Validate independently of agent-editable code.

- `scripts/agent_workflow.py` (`test_command_for`) supplies an unconditional successful command

  for tasks other than bugfix/config/assets. Archaeology cannot prove the agent

  identified the requested history fact merely by remaining read-only; recovery

  cannot prove the intended restored state from a README-only change shape.

  Add task-specific answer artifacts and trusted external validators.

- `scripts/oakbench/integrity.py:18` samples at most three payloads, and metadata

  fingerprint changes prove only some metadata changed. Oak's worktree fallback

  does not establish durable committed content. The explicit public-trust allow

  list at `:20` correctly admits only known-good Git HEAD content today. Add

  independent fresh-checkout or object-reader verification of full path/type/

  mode/content manifests; test an unrelated metadata write, dropped fourth

  payload, partial commit, and successful-looking no-op subject.

- Commit counting can miss an amend and trusts subject log output. Define

  snapshot semantics explicitly and move to before/after durable revision

  identity plus exact payload verification; preserve unknown observations as

  null instead of fabricated zero counts.

### P0 — Output usefulness is weaker than downstream compatibility

- `scripts/oakbench/output_semantics.py:30` counts a ground-truth path substring

  anywhere in output. A longer path or a diff-body mention can inflate recall.

  Use format-aware path decoding and exact identities; add shared suffixes,

  spaces, tabs, newline names, quoting, Unicode, and rename pairs.

- `scripts/oakbench/output_semantics.py:135` recognizes headers/hunks or a binary

  notice without applying the patch. This is a structural signal, not proof

  `patch`/`git apply` reconstructs the expected tree. Add a separate versioned

  round-trip operation on a pristine fixture and verify exact result manifests.

  Negative tests should include malformed hunk counts, truncated patches,

  missing file sections, and binary notices without an applicable patch.

### P1 — Correctness breadth and fault recovery

Core snapshots/status/diff are useful but do not cover the semantic state space

of rename/delete conflicts, type changes, executable bits, unusual names,

ignore rules, staged versus unstaged changes, timestamp/racy-stat behavior,

partial publication, lock contention, and failed workspace transitions.

Some isolated fixtures cover portions of these concerns; no cross-subject,

state-machine conformance matrix was found. Build versioned, deterministic

operation sequences with filesystem/revision invariants and independent

expected trees. Extend failure injection to interruption before/during/after

persistence, disk-full and permission failures, reconnect/retry idempotence,

and fresh-process recovery. Do not label injected transport interruption as

an fsync/power-loss durability test.

### P1 — Humans and agents need separate experiments

No participant-oriented human usability harness, assignment/counterbalancing,

training policy, or task correctness scorecard was found. Scripted

`human_action_count` fields in `task_loop.py` are not observed human behavior.

Add a human task protocol measuring successful completion, elapsed active

work, recovery, mistaken actions, and task comprehension, stratified by VCS

experience. Reuse the same external correctness oracles for humans and agents.

For agents, expand task diversity beyond the small published workflow catalog,

pin model and CLI versions, retain instruction-level separation, and report

success uncertainty and unsuccessful-run costs. Repeated mock execution does

not estimate agent reliability. Hold back some task variants from development

to detect tuning exclusively to visible fixtures.

### P1 — Measured coverage must remain separate from runnable scaffolding

`docs/benchmark-coverage.md` and README already distinguish many limitations:

XL/monorepo fixtures are registered but not active core profiles; mount and

hosted rows need infrastructure; Oak/Git remote transports differ; conflict

resolution is Git-only; Jujutsu is pilot configuration. Preserve these honest

skips and prioritize transport-matched acquisition, Oak conflict tasks, real

hosted integration, and pinned standard/large campaigns. A source change that

unlocks a lane is not evidence that the lane performs well.

## Recommended order

First close false-success paths and verify durable exact content. Then add

portable upstream-derived conformance cases and explicit fault invariants.

Next establish repeatable real-agent and human protocols with held-out tasks.

Only then widen public performance claims with pinned runner campaigns and

archived raw evidence. Each new scenario needs immutable identity, its oracle,

its required capabilities, and a documented fair comparator.