Merge correctness & quality suite (P3)

# Merge correctness & quality suite (P3)

A differential merge suite. Every scenario is built as a real Git history.

`git merge-ort` (via `git merge-tree --write-tree`) supplies the reference

tree and conflict set. The same inputs then go through **Oak's merge engine**,

and the two results are compared. Each directional case gets exactly one

class, and merge laws are checked on the side.

This suite produces diagnostic correctness evidence

(`evidence_class: diagnostic_merge_correctness`). It is not timing data.

Do not ingest its rows into performance reports, devloop baselines, or

public comparison tables.

## Run

```sh

# from the repo root, in an isolated worker (writes only under --workdir)

python3 suites/merge/run_merge_suite.py --workdir runs/merge --jobs 6 \

    --summary runs/merge-summary.json > runs/merge.jsonl

python3 suites/merge/run_merge_suite.py --workdir runs/merge --list          # scenario ids

python3 suites/merge/run_merge_suite.py --workdir runs/m1 --only 'incident/*' # one group or case

python3 -m unittest tests.test_merge_suite                                     # instrument tests

```

| Option | Meaning |

|---|---|

| `--groups` | Comma list from `gix-tree,gix-cartesian,gix-text,t64xx,native,generated`. The default is all of them. |

| `--oak-bin` | The Oak binary under test (default `oak`). |

| `--git-bin` | The oracle Git. By default the macOS xcrun shim is resolved to the real Git through `oakbench.subjects`. |

| `--keep-equal` | Keep case directories for equal-to-git cases. By default they are deleted, and every other case directory is kept as evidence. |

The exit code is 1 when any case is `oak-wrong` or `harness-error`, and 0

otherwise. Output is JSONL on stdout (one row per directional case), with a

summary table on stderr. Everything runs locally: a private `oak serve` on

loopback and an isolated `HOME`. Nothing contacts oak.space.

Requirements are Python 3.9+ stdlib (ADR-0001), a real `git`, `bash`, and

the `oak` binary. The Oak and Git SHA-256 and version are recorded on every

row.

## Which Oak path each result exercises (phase 1)

`path_exercised: oak-pull-parent-sync` is the Oak **client** merge engine.

It is run as a user would run it:

1. `oak clone <scenario.git>` imports main's lineage (Oak's git import) and

   `oak push` bootstraps `main` on a local `oak serve`.

2. A second `oak clone` imports the fork point. `oak commit` then checks in

   the scripted branch tree states, and `oak push` publishes the branch.

3. `oak fetch` runs, then `oak merge --dry-run --json`, which records the

   local merge prediction (`laws.prediction_agrees`).

4. `oak pull --json` runs the parent sync. This is the measured merge: merge

   base resolution over the local and hydrated graph, then

   `three_way_merge_manifests`, then diffy text merge and conflict markers.

   It is the same engine `oak merge` uses for non-main parents.

5. On a clean merge, a second `oak pull` is the idempotence check. On a

   conflict, `oak conflict show --json` supplies the conflict set. Each

   conflicted path is then overwritten with a sentinel and `oak pull

   --continue` commits, so the **committed** merge result is compared. The

   conflicted worktree is only a working view: Oak leaves paths the merge

   deletes in place until `--continue` (`conflict_worktree_stale_paths`).

Two pieces of test plumbing are involved. Neither contains merge logic.

- **Raw shim.** `oak serve` 0.105 lacks `GET /api/:o/:r/raw/:commit/:path`,

  which the client uses to hydrate parent blobs. `RawShim` fronts the real

  server and answers that one route from the publishing clone via

  `oak file inspect --at <commit> --output`. Oak hash-verifies those bytes.

  Every other request is forwarded unchanged.

- **Main staging.** For multi-step scenarios, the shim can report an earlier

  main commit as `main`'s head. This is how `main` "moves" between client

  pulls (for example, sync with M1, then main reverts in M2) without a

  hosted merge endpoint.

Some shapes are outside Oak's public surface. Ours-side histories are always

fork + checkpoints + CLI syncs, because Oak cannot import an arbitrary

ours-side DAG. Git DAGs with several merge bases, or with unrelated

histories, are therefore `oak-unsupported` under the fork+squash

construction. The criss-cross cases that matter are instead built from

Oak-native steps under `incident/criss-cross-*`. `git-t6416-criss-cross-modify-delete`

is a hybrid: the branch side is Oak-native, but main's merge commit is an

imported git merge, because `oak serve` has no merge endpoint.

**Phase 2 (not built):** the hosted squash-merge onto `main` (oakspace

`POST /branches/:b/merge`). `oak_adapter.HostedMergeAdapter` is the plug

point, and it returns the same result shape. `sources.PHASE2_SPECS` lists

specs that need it: concurrent merges (M0a), push during merge (M0b), the

server `/sync` that loses lineage, client-vs-server criss-cross tie-break

agreement, server clock-skew, and dirty tree across a moved branch (fb-530).

The summary names them under `phase2_specs_not_executed`. They are never

counted as passes.

## Classes

| Class | Meaning |

|---|---|

| `equal-to-git` | Same clean/conflicted status and the same conflicted path set. Every non-conflicted path has identical content and mode. |

| `differs-but-valid` | Differs from Git, but no side's change is lost or fabricated at any non-conflicted path. `reasons` say how, for example `oak_conflicts_where_git_clean` (a false conflict, typically because Oak has no rename detection), `oak_clean_where_git_conflicts` (Oak's path-level merge kept both sides where merge-ort reports a rename conflict), `tree_differs(...)` (Git relocated a file through directory-rename detection while Oak left it in place), or `text_merge_algorithm_differs`. |

| `oak-wrong` | A non-conflicted path lost a side's change (`lost_change`), lost a side's mode or kind change (`lost_mode_change`), was committed clean with conflict markers no input had (`committed_conflict_markers`), or holds content nothing explains (`fabricated_content`). Also: a scenario's stated expectation was violated (`expectation_violated`), or Oak errored on a valid input (`oak_error`). A failed pull must leave the worktree matching HEAD; one that moved HEAD over a pre-merge worktree adds `failed_pull_moved_head_worktree_stale`, and one that kept HEAD but half-applied the merge to a previously clean worktree adds `failed_pull_left_worktree_dirty`. |

| `oak-unsupported` | Decided before Oak runs. The input cannot be expressed through Oak's surface: gitlinks (the git import drops them), several git merge bases, or unrelated histories. |

| `harness-error` | The instrument failed (fixture build, oracle, shim), or a scripted setup step before the measured merge failed (`setup_failed:`). It is never an Oak verdict. |

Nothing passes silently. Unmeasured law values are `null` (ADR-0002). An

example is the no-lost-change law where no unique merge base exists.

### Merge laws (per row under `laws`)

- **no_lost_change**: path-level. A change made by one side only must be

  present. An identical change on both sides must be present. A divergent

  change on both sides must be text-merged, with every block either side

  inserted still present, no new markers, and the 3-way-merged mode/kind

  (a chmod on one side survives a content edit on the other). Conflicted paths are exempt.

  The text check is a heuristic: deletions are not verified, because lines

  repeat.

- **symmetric**: merge(A,B) ≅ merge(B,A). The clean flag, the conflicted

  path set, and non-conflicted content must all match. Conflicted file bytes

  are excluded, because side labels differ.

- **idempotent**: after a clean merge, `oak pull` again is a no-op. Head and

  tree are unchanged.

- **false_conflict / missed_conflict**: Oak conflicts where Git is clean, or

  the reverse. The summary reports rates over measured rows.

- **prediction_agrees**: the `oak merge --dry-run --json` prediction matches

  the executed merge (clean flag and conflict set). When the real pull fails,

  an available prediction counts as a disagreement.

### Known bugs

`known_bugs.json` maps oak-wrong rows to filed Oak bugs (fb-537..fb-541,

filed by o-4d on 2026-09-28 from QA's standalone repros). Each row carries

`known_bug`. The rules are strict so that a new bug cannot hide behind a

known one:

- A row maps to a bug only if **every** reason on the row matches that bug's

  `reason_patterns` (regex). One unexplained extra reason leaves the row

  unmapped.

- fb-540 matches only its four exact error signatures (errno 17 and 20,

  "cannot build tree with both file…", "refusing to replace non-empty

  directory…") plus the two failed-pull invariant reasons. Any other Oak

  error, such as a panic or a server 500, stays unmapped.

- fb-537 requires `committed_conflict_markers_no_final_newline`. Markers in

  inputs that end with a newline give the plain `committed_conflict_markers`

  reason, which no bug maps.

- fb-541 requires the labelled expectation

  `expectation_violated[criss_cross_divergent_resolution]`.

- Failures in scripted setup steps (import, push, staged pull) are

  `harness-error` with `setup_failed:`, never an Oak verdict or a known bug.

- `expected_rows` pins each bug's rows from the oak 0.105.0 run. The summary

  reports `new_rows` and `vanished_rows` per bug (status `reproduced as

  pinned`, `changed`, or `NOT reproduced (fixed?)`), plus

  `oak_wrong_unmapped_rows`.

`--resummarize RUN.jsonl` recomputes the summary and bug mapping from an

existing run without running Oak. Unknown class values are rejected

(`classify.validate_class`).

## Scenario sources and licenses

| Origin (`origin`) | What | License / port mode |

|---|---|---|

| `gitoxide-tree-baseline` | Named tree scenarios from `vendor/gitoxide/tree-baseline.sh`, merge-style only. The diff3 duplicates differ only in marker text, and Oak has one marker style. Both directions are run. | **MIT OR Apache-2.0.** Vendored verbatim. |

| `gitoxide-cartesian` | The 14×14 atomic-operation product (`matrix`) plus its gitlink twin (`matrix-submodule`, which is `oak-unsupported`). | Same. |

| `gitoxide-text-baseline` | Blob triples from `text-baseline.sh` (default style), each as a one-file tree merge. | Same. |

| `git-t64xx-reimplemented` | Git t6423 directory-rename cases 1a 1b 1c 1f 2a 3a 4a 5a 6a 6c 6d 7a 8c 9a 9b 12d 12i, and t6422 break-detection-1/2/3 and rename/rename(1to2) binary. The IDs are in `scenario`. | Git is **GPL-2.0-only**. Only the O/A/B shapes from the testcase comment prose were re-expressed, in `sources.T64XX`. Paths and contents are our own. No Git test text was copied. A unit test only guards against obvious leakage (Git test-harness keywords); it is not a full similarity scan, so review is the real control. Read at Git v2.50.1. |

| `native-incident` | This week's incident shapes, each with its correct expected outcome in `expect`. Seeds: `merge_base_fixtures.rs` (5d2ee0fb) and the repro-server-mergebase worker. | Suite-authored. |

| `generated-law` | 40 seeded random histories (edits in disjoint or overlapping regions, delete, add, chmod, rename, identical changes) for the laws. | Suite-authored. |

`vendor/gitoxide/` holds `tree-baseline.sh`, `tree-cartesian-baseline.sh`

and `text-baseline.sh`, byte-identical to upstream commit

`acf906f8fcdb8e8d143570737786543dc5a204a0`. Their SHA-256 digests are pinned

in `UPSTREAM.json`, and a unit test checks them. The directory also carries

`LICENSE-MIT`, `LICENSE-APACHE` and a `NOTICE` with the attribution. The

upstream scripts have no per-file headers, so none were added and the files

stay byte-identical. The suite only runs these scripts. Spork cases

(optional stretch) were not added.

This vendoring is a deliberate exception to the "no vendored external

benchmark code" rule. It is recorded, with rationale, license and pin, in

`docs/prior-art.md` ("Recorded Exception: gitoxide merge baselines"). GPL

sources stay clean-room.

### Incident scenarios

| id | Shape | Correct outcome |

|---|---|---|

| `incident/stale-base-squash-revert` | `R-C-X-M1..M9-H`, M5 reverts X, `H.merge_parent=Z1` (forked from C), branch `a1` on X. The ours clone holds only R-C-X locally (a partial graph), and the rest is hydrated by pull. | The base is X, so X stays reverted. |

| `incident/merge-base-via-merge-parent` | The fork is Z1, reachable only through H's merge parent. | The base is Z1, so edits to `z.txt` merge cleanly. |

| `incident/revert-after-client-sync` | The branch syncs main M1 (adds X) with `oak pull`, then main M2 reverts X. | The revert survives, because the sync records `merge_parent`. |

| `incident/revert-after-lineage-less-sync` | The branch copies X by hand, which is the server `/sync` shape. | Git and Oak both resurrect X. `intent_note` explains that lineage loss is unrecoverable. The hosted `/sync` itself is phase 2. |

| `incident/criss-cross-tie-break-{side-newer,side-older}` | Two best common ancestors {F, Mk}, then main reverts K. | K stays reverted, whatever the tie-break or timestamps. |

| `incident/clock-skew-merge-base-{skewed,monotone-control}` | The best common ancestor Y is 24 h older than X. | The base is Y, and Y stays reverted. |

| `git-t6416-criss-cross-modify-delete` | Two histories resolve the same modify/delete differently: the branch deletes (an Oak pull-sync with a scripted resolution), main keeps. Main's merge D is an imported git merge commit, not an Oak server squash-merge, so this shape is only partly Oak-native. | A conflict, not a silent pick. |

## Row schema (`merge_suite_schema_version: 1`)

Each row carries `scenario`, `scenario_version`, `direction`, `origin`,

`upstream{id,license,commit}`, `oak{bin,sha256,version,path_exercised}`,

`git{bin,sha256,version,role}`, `git_result{clean,conflicts{path:[types]},tree_digest,conflict_messages}`,

`oak_result{clean,conflicts,tree_digest,pull_receipt,prediction,idempotence,result_source,...}`,

`class`, `reasons`, `laws{...}`, `evidence`, `repro`, `case_dir`, `elapsed_ms`.

Rows that are not equal-to-git also carry `oak_steps` (argv, return code, ms,

stderr tail). Scenario ids are immutable identity (ADR-0005). Changed

semantics need a new id or a `scenario_version` bump. Raw rows are results,

and results are never committed (ADR-0003).

## First run

See `RESULTS-2026-09-28.md` for the first run against oak 0.105.0: counts

per class, law rates, and every oak-wrong case with a minimal repro.