Log in
benchmarksPublic

Commit e70815b9

· Browse files at this commit

e70815b9044a

Merged on mrmrs-15c4c5
mrmrs · 1 month ago · 9 files ~9 modified
M suites/merge/README.md
@@ -96,9 +96,9 @@
9696 |---|---|
9797 | `equal-to-git` | Same clean/conflicted status and the same conflicted path set. Every non-conflicted path has identical content and mode. |
9898 | `differs-but-valid` | Differs from Git, but no side's change is lost or fabricated at any non-conflicted path. `reasons` say how, for example `oak_conflicts_where_git_clean` (a false conflict, typically because Oak has no rename detection), `oak_clean_where_git_conflicts` (Oak's path-level merge kept both sides where merge-ort reports a rename conflict), `tree_differs(...)` (Git relocated a file through directory-rename detection while Oak left it in place), or `text_merge_algorithm_differs`. |
99-| `oak-wrong` | A non-conflicted path lost a side's change (`lost_change`), lost a side's mode or kind change (`lost_mode_change`), was committed clean with conflict markers no input had (`committed_conflict_markers`), or holds content nothing explains (`fabricated_content`). Also: a scenario's stated expectation was violated (`expectation_violated`), or Oak errored on a valid input (`oak_error`). A failed pull that moved HEAD while leaving a pre-merge worktree adds `failed_pull_moved_head_worktree_stale`. |
99+| `oak-wrong` | A non-conflicted path lost a side's change (`lost_change`), lost a side's mode or kind change (`lost_mode_change`), was committed clean with conflict markers no input had (`committed_conflict_markers`), or holds content nothing explains (`fabricated_content`). Also: a scenario's stated expectation was violated (`expectation_violated`), or Oak errored on a valid input (`oak_error`). A failed pull must leave the worktree matching HEAD; one that moved HEAD over a pre-merge worktree adds `failed_pull_moved_head_worktree_stale`, and one that kept HEAD but half-applied the merge to a previously clean worktree adds `failed_pull_left_worktree_dirty`. |
100100 | `oak-unsupported` | Decided before Oak runs. The input cannot be expressed through Oak's surface: gitlinks (the git import drops them), several git merge bases, or unrelated histories. |
101-| `harness-error` | The instrument failed: fixture build, oracle, or shim. It is never an Oak verdict. |
101+| `harness-error` | The instrument failed (fixture build, oracle, shim), or a scripted setup step before the measured merge failed (`setup_failed:`). It is never an Oak verdict. |
102102
103103 Nothing passes silently. Unmeasured law values are `null` (ADR-0002). An
104104 example is the no-lost-change law where no unique merge base exists.
@@ -125,11 +125,33 @@
125125
126126 ### Known bugs
127127
128-`known_bugs.json` maps oak-wrong rows to filed Oak bugs (first matching rule
129-wins; fb-537..fb-541). Each row carries `known_bug`. The summary reports rows
130-per bug, oak-wrong rows no rule maps (`oak_wrong_unmapped`), and any bug that
131-no longer reproduces (`NOT reproduced (fixed?)`), so a fix flips visibly.
132-Unknown class values are rejected (`classify.validate_class`).
128+`known_bugs.json` maps oak-wrong rows to filed Oak bugs (fb-537..fb-541,
129+filed by o-4d on 2026-09-28 from QA's standalone repros). Each row carries
130+`known_bug`. The rules are strict so that a new bug cannot hide behind a
131+known one:
132+
133+- A row maps to a bug only if **every** reason on the row matches that bug's
134+ `reason_patterns` (regex). One unexplained extra reason leaves the row
135+ unmapped.
136+- fb-540 matches only its four exact error signatures (errno 17 and 20,
137+ "cannot build tree with both file…", "refusing to replace non-empty
138+ directory…") plus the two failed-pull invariant reasons. Any other Oak
139+ error, such as a panic or a server 500, stays unmapped.
140+- fb-537 requires `committed_conflict_markers_no_final_newline`. Markers in
141+ inputs that end with a newline give the plain `committed_conflict_markers`
142+ reason, which no bug maps.
143+- fb-541 requires the labelled expectation
144+ `expectation_violated[criss_cross_divergent_resolution]`.
145+- Failures in scripted setup steps (import, push, staged pull) are
146+ `harness-error` with `setup_failed:`, never an Oak verdict or a known bug.
147+- `expected_rows` pins each bug's rows from the oak 0.105.0 run. The summary
148+ reports `new_rows` and `vanished_rows` per bug (status `reproduced as
149+ pinned`, `changed`, or `NOT reproduced (fixed?)`), plus
150+ `oak_wrong_unmapped_rows`.
151+
152+`--resummarize RUN.jsonl` recomputes the summary and bug mapping from an
153+existing run without running Oak. Unknown class values are rejected
154+(`classify.validate_class`).
133155
134156 ## Scenario sources and licenses
135157