Reviews

2 open branches

Merged: 0 24h 16 total

Open

Open mrmrs-1b7902 Correct benchmark outcome attribution, workflow parity and measurement integrity

mrmrs 3d ago

Open oakmark-throughput-benchmark-transplant Add oakmark: per-repo push/clone throughput benchmark for oak (ForgeMark analogue)

mrmrs 2mo ago

Merged

Merged e9b767a5 Preserve measured agent failures and validate campaign copies before admission

mrmrs 9d ago

Merged b297e8a2 Version saved baseline comparisons and enforce claim evidence eligibility; retain explicit diagnostic scoring and outcome denominators

mrmrs 11d ago

Merged f86d1109 Review benchmark validity and add upstream-inspired VCS conformance

mrmrs 11d ago

Merged a9e8fa39 Add a controlled actual-Serve benchmark lane for the legacy selected-branch clone path. Pin fixture/server/client provenance; compare head, manifest, history, worktree bytes/modes, local artifacts, receipt and normalized network work; expose clone plus first status/no-op commit costs and stat-cache validity. Require explicit control identity and harness-controls markers, isolate credentials, bound processes, and state that A/A plus quiet-host evidence remain necessary before any timing claim.

mrmrs 13d ago

Merged 13de5e54 Measure bounded end-to-end agent task costs without production mutations.

mrmrs 13d ago

Merged 88e86b3f Rebuild benchmarks as a clean single-root repository from oak/benchmarks main@009dead264ca; source tree verified by SHA-256 manifest 1cc497d666f1a1c8e27b040edfab9e7730dd48c3b47567e616943e471570a5c8.

mrmrs 2mo ago

Closed

closed without merging

Closed mrmrs-d7b59e Add reproducible end-to-end agent campaigns

mrmrs 12d ago

Closed mrmrs-f0d825 Add frozen local agent campaign plans and resumable durable journals. Validate exact manifests, safe names and paths, deterministic counterbalanced schedules, pinned configuration/harness/binaries, mandatory measurement locking, explicit real-run attestation, bounded admission and agent timeout, started/finished fsync checkpoints, artifact hashes and terminal-row identity. Never silently rerun interrupted episodes. Rebuild derived rows on no-op resume after a journal-to-rows crash. Fourteen contract tests pass; earlier full985 sandbox run had only18 known loopback permission errors and3 skips; combined unrestricted QA runs in integration worker. Carries reviewed foundation; report and subprocess modules integrate from sibling branches. No paid runs.

mrmrs 12d ago

Closed mrmrs-999547 Bound real-agent subprocess execution and capture; isolate per-trial HOME, sanitize Git plumbing, and mark incomplete usage honestly. Includes reviewed benchmark quality foundation.

mrmrs 12d ago

Closed mrmrs-1ed996 Add diagnostic agent campaign accounting, paired outcome and efficiency reports, and staged qualification plan

mrmrs 12d ago

Closed mrmrs-benchmark-gaps-20260910 Close benchmark correctness and saved-comparison gaps

mrmrs 12d ago

Closed mrmrs-473c2f Require exact output path identities and verify full patches against pristine trees; record oracle versions and fail closed on lost application proof

mrmrs 12d ago

Closed mrmrs-8b790b Close config and asset task false passes with shared trusted external oracles. Verify exact 600 config outputs, four binary contents and manifest identities, fixture validator preservation, and complete Git committed task trees. Run real-agent validation with isolated Python outside editable fixtures; version correctness semantics. Negative partial-commit and bypass cases plus Git/Oak workflow and mock-agent positives pass. Carries reviewed benchmark foundation; full sandbox suite930 has only18 known loopback permission errors and3 skips pending unrestricted integration.

mrmrs 12d ago

Closed mrmrs-9d1d51 Harden agent correctness observations: fail closed on failed or unsupported status, parse current Oak structured history and status, require new snapshots for mutating tasks, and detect mode/type/symlink changes. Version oracle semantics, retain null dirtiness, add negative tests and a source-based coverage audit. Targeted 23 tests and actual Oak 0.103.0 tiny commit/status probe pass; full sandbox run has socket permission errors pending unrestricted integration validation.

mrmrs 12d ago

Closed mrmrs-9df99c Reject incompatible measurement populations in statistics and scorecards; document methodology audit and acceptance criteria

mrmrs 12d ago

Closed mrmrs-dfa321 Add exact VCS preservation diagnostics and pinned upstream test review

mrmrs 12d ago

docs/command-semantics.md 52 lines · 2.1 KB

Command Semantics

Public benchmark claims must say which comparison track they use.

Tracks

agent-default measures what a coding agent would naturally call today using current documented/default commands. This is the right track for CLI UX, transcript pressure, and agent ergonomics.

core-equivalent measures the closest available equivalent semantic output level across VCSs. This is the right track for VCS-mechanics claims, but only when the command mapping has an equivalent semantic contract.

The machine-readable source of truth is config/command_semantics.json.

Critical Rule

Do not use agent-default output to claim a core VCS engine win. For example, git diff --stat versus oak diff primarily measures CLI output semantics. That is useful, but it is a UX/token benchmark, not a pure diff-engine benchmark.

Current Compact Mappings

Oak now has compact equivalents for the previously diagnostic agent rows:

status.short: Git uses status --porcelain=v1; Oak uses status --porcelain.
diff.stat: Git uses diff --stat; Oak uses diff --stat.
diff.name_only: Git uses diff --name-only; Oak uses diff --name-only.
commit.quiet: Git uses add . plus commit --quiet; Oak uses commit --json --quiet --no-verify.

These mappings are recorded in config/command_semantics.json; changing any of them requires a version bump and should be reflected in every row's command_semantics_version.

Correctness Expectations

Every operation needs a success oracle:

status operations: dirty/clean state and changed path set match the fixture manifest
diff summary operations: changed path set and line-count summaries match the fixture manifest
full diff operations: text hunks apply cleanly and binary files are summarized without payload dumps
commit operations: a new snapshot exists and the working tree is clean
branch operations: the task branch exists and points at the expected base

For binary scenarios, a command that emits raw binary content should fail the semantic correctness check even if its return code is zero.

# Command Semantics

Public benchmark claims must say which comparison track they use.

## Tracks

`agent-default` measures what a coding agent would naturally call today using
current documented/default commands. This is the right track for CLI UX,
transcript pressure, and agent ergonomics.

`core-equivalent` measures the closest available equivalent semantic output
level across VCSs. This is the right track for VCS-mechanics claims, but only
when the command mapping has an equivalent semantic contract.

The machine-readable source of truth is `config/command_semantics.json`.

## Critical Rule

Do not use `agent-default` output to claim a core VCS engine win. For example,
`git diff --stat` versus `oak diff` primarily measures CLI output semantics.
That is useful, but it is a UX/token benchmark, not a pure diff-engine
benchmark.

## Current Compact Mappings

Oak now has compact equivalents for the previously diagnostic agent rows:

- `status.short`: Git uses `status --porcelain=v1`; Oak uses `status --porcelain`.
- `diff.stat`: Git uses `diff --stat`; Oak uses `diff --stat`.
- `diff.name_only`: Git uses `diff --name-only`; Oak uses `diff --name-only`.
- `commit.quiet`: Git uses `add .` plus `commit --quiet`; Oak uses
  `commit --json --quiet --no-verify`.

These mappings are recorded in `config/command_semantics.json`; changing any of
them requires a version bump and should be reflected in every row's
`command_semantics_version`.

## Correctness Expectations

Every operation needs a success oracle:

- status operations: dirty/clean state and changed path set match the fixture
  manifest
- diff summary operations: changed path set and line-count summaries match the
  fixture manifest
- full diff operations: text hunks apply cleanly and binary files are summarized
  without payload dumps
- commit operations: a new snapshot exists and the working tree is clean
- branch operations: the task branch exists and points at the expected base

For binary scenarios, a command that emits raw binary content should fail the
semantic correctness check even if its return code is zero.

1	`# Command Semantics`
2
3	`Public benchmark claims must say which comparison track they use.`
4
5	`## Tracks`
6
7	`agent-default` measures what a coding agent would naturally call today using
8	`current documented/default commands. This is the right track for CLI UX,`
9	`transcript pressure, and agent ergonomics.`
10
11	`core-equivalent` measures the closest available equivalent semantic output
12	`level across VCSs. This is the right track for VCS-mechanics claims, but only`
13	`when the command mapping has an equivalent semantic contract.`
14
15	The machine-readable source of truth is `config/command_semantics.json`.
16
17	`## Critical Rule`
18
19	Do not use `agent-default` output to claim a core VCS engine win. For example,
20	`git diff --stat` versus `oak diff` primarily measures CLI output semantics.
21	`That is useful, but it is a UX/token benchmark, not a pure diff-engine`
22	`benchmark.`
23
24	`## Current Compact Mappings`
25
26	`Oak now has compact equivalents for the previously diagnostic agent rows:`
27
28	- `status.short`: Git uses `status --porcelain=v1`; Oak uses `status --porcelain`.
29	- `diff.stat`: Git uses `diff --stat`; Oak uses `diff --stat`.
30	- `diff.name_only`: Git uses `diff --name-only`; Oak uses `diff --name-only`.
31	- `commit.quiet`: Git uses `add .` plus `commit --quiet`; Oak uses
32	`commit --json --quiet --no-verify`.
33
34	These mappings are recorded in `config/command_semantics.json`; changing any of
35	`them requires a version bump and should be reflected in every row's`
36	`command_semantics_version`.
37
38	`## Correctness Expectations`
39
40	`Every operation needs a success oracle:`
41
42	`- status operations: dirty/clean state and changed path set match the fixture`
43	`manifest`
44	`- diff summary operations: changed path set and line-count summaries match the`
45	`fixture manifest`
46	`- full diff operations: text hunks apply cleanly and binary files are summarized`
47	`without payload dumps`
48	`- commit operations: a new snapshot exists and the working tree is clean`
49	`- branch operations: the task branch exists and points at the expected base`
50
51	`For binary scenarios, a command that emits raw binary content should fail the`
52	`semantic correctness check even if its return code is zero.`