Log in
benchmarksPublic

Viewing files at commit 369435870869Commit detailsView latest

Browse files
suites/merge/RESULTS-2026-09-28.md 187 lines · 9.9 KB

First run: oak 0.105.0 (2026-09-28, revised after QA rounds 1 and 2)

This page summarizes the evidence. Raw JSONL is not committed (ADR-0003). It is kept in the author's worker at worker-mrmrs-20260927T090659-p3-merge/runs/full4.jsonl, with the summary at runs/full4-summary.json. runs/full1.* holds the pre-QA run, and runs/full2.* and runs/full3.* are intermediate reruns. full4 has the same class for every case as full3; only reasons and bug mapping were refined.

  • Subject. oak 0.105.0, sha256 4dc220e1f089845ca1ada8da4025b18b37f7823e266989e757302b8fa8bc6647. The path exercised is oak-pull-parent-sync, the client merge engine (see README).
  • Oracle. git version 2.50.1 (Apple Git-155), the real Git (the xcrun shim was resolved), sha256 10f9c1df…d2a9.
  • Command. python3 suites/merge/run_merge_suite.py --workdir runs/full3 --jobs 8 --summary runs/full4-summary.json
  • Size. 387 scenarios and 762 directional cases. Self-merges (for example gitoxide no-change main/main) now run in one direction only; the first run counted them twice.
  • Determinism. Classes are identical across reruns. Conflict-marker bytes embed Oak's randomly named personal branch, so tree digests of files with markers differ between runs. Those bytes are never compared, except to detect markers.

Counts per class

originequal-to-gitdiffers-but-validoak-wrongoak-unsupportedharness-error
generated-law678500
git-t64xx-reimplemented1230000
gitoxide-cartesian14650142100
gitoxide-text-baseline5121100
gitoxide-tree-baseline444439200
native-incident80100
total328134702300

The 210 cartesian cases marked oak-unsupported are the gitlink matrix, because Oak's git import drops gitlinks. The 20 tree-baseline cases in that class are gitlink cases or cases with several git merge bases or no merge base.

Merge laws (true / measured)

LawResult
no_lost_change (content, mode and kind; no new markers)460/491
idempotent (second pull is a no-op after a clean merge)337/337
symmetric428/468
false_conflict (Oak conflicts where git is clean)42/494, 8.5%
missed_conflict (git conflicts where Oak is clean)136/494, 27.5%
prediction_agrees (oak merge --dry-run --json vs the executed pull)494/532

Missed conflicts.

  • By class: 110 are differs-but-valid and 26 are oak-wrong.
  • By git conflict kind: 104 rename or directory-rename (Oak has no rename detection), 23 content, 4 distinct-modes, 3 modify/delete, 2 binary/content. The content and binary ones are mostly fb-537.

Predictions. All 38 prediction disagreements are the pulls that failed (fb-540). For those, oak merge --dry-run --json predicted a clean merge in 19 cases and a conflict in 19. It never predicted the failure.

Incident shapes

All incident shapes pass on the client path except F6:

  • stale base C vs X, with a partial local graph hydrated on pull
  • the base reached via a merge parent
  • revert after a CLI sync
  • criss-cross tie-break, both timestamp orders
  • clock-skew best common ancestor, plus its monotone control

The lineage-less sync shape resurrects the reverted change in both Git and Oak. That is inherent once lineage is lost, and it is recorded as intent_note. The hosted-merge shapes (M0a, M0b, the server /sync, client-vs-server tie-break, server clock-skew, fb-530) are phase-2 specs and were not executed.

Oak-wrong findings (70 cases). Every row maps to a filed bug (known_bugs.json, 0 unmapped)

Oak is not fixed here. fb-537..fb-541 were filed by o-4d on 2026-09-28 from QA's standalone repros (bench-program/repro-p3). Mapping is strict: every reason on a row must be explained by the same bug. Each bug's rows are pinned in expected_rows, and all five reproduce exactly as pinned (0 new, 0 vanished). Each family gives its bug ID, case count, a minimal repro (--only runs just that case), and the observed behavior. The repro prefix is always:

python3 suites/merge/run_merge_suite.py --workdir /tmp/mr --groups <group> --only '<id>'

The failing case directory, with both clones, stays under <workdir>/cases/<id>/<direction>/.

  1. fb-540 (F5): file↔directory transitions make oak pull fail, and some failures move HEAD (38 cases). Blocker.
    • The minimal case is gix-cartesian/matrix/keep--file-to-dir (A-B): main replaces file source with source/child, and the branch did not touch it.
    • The failure messages are IO error: File exists (os error 17), Not a directory (os error 20), cannot build tree with both file "e" and child path "e/e", and refusing to replace non-empty directory … with a tracked file. Git merges most of these cleanly.
    • 24 of the 38 failed pulls break the invariant "a failed sync leaves the worktree matching HEAD", as found by the suite's post-failure check of HEAD vs worktree vs retry:
      • In 11, HEAD moved to the merge commit while the worktree stayed pre-merge (failed_pull_moved_head_worktree_stale). A retry then reports up_to_date, so the next oak commit would revert main's changes.
      • In 13, HEAD did not move but the merge was half-applied to a worktree that was clean before the pull (failed_pull_left_worktree_dirty). Examples: gix-tree/tree-to-non-tree (A-B), simple/side-1-2-various-conflicts, and matrix/modify-source--file-to-dir (B-A).
      • The other 14 failed cleanly. QA also observed that oak switch across a file→dir change leaves a half-switched worktree. The suite does not measure that.
    • Repro: --groups gix-cartesian --only 'gix-cartesian/matrix/add-free-a--file-to-dir' (the HEAD-moved variant) or 'gix-cartesian/matrix/keep--file-to-dir'.
  2. fb-537 (F1 and F3): conflict markers committed as a clean merge when a side has no trailing newline (13 cases). Blocker (invalid merge).
    • The root cause is a missing final newline. Diffy glues a marker onto the last line (line3x||||||| original, line3=======, B>>>>>>> main). Marker detection misses the glued marker, and oak pull reports merged and commits the markers.
    • CRLF is not the cause. The marker-newline-handling-crlf fixture fails only because it lacks a final newline, and CRLF content with a final newline conflicts correctly.
    • Cases: gix-text/complex/{conflict-without-LF,missing-LF-at-EOF,marker-newline-handling-lf,lf2,crlf} and gix-text/unique-merge-base-with-insertion.
    • Binary (the former "F3") has the same cause. gix-tree/both-modify-binary is a small NUL-containing file with no trailing newline. It is valid UTF-8, so Oak text-merges it and commits glued markers into it. With a trailing newline, Oak reports the conflict correctly (QA verified). Separately, Oak text-merges binary content whenever it decodes as UTF-8.
    • Repro: --groups gix-text --only 'gix-text/complex/conflict-without-LF'
  3. fb-538 (F2): symlinks (or symlink vs file) changed on both sides are text-merged, and the link target is committed with glued markers (8 cases). High.
    • This is the same missing-newline mechanism, because symlink targets never end in a newline. The deeper problem is that Oak text-merges symlink targets at all.
    • Cases: gix-tree/{symlink-addition,symlink-modification,type-change-to-symlink}, rename-add-symlink (B-A), and unrelated-renames-to-same-path-with-type-mismatch (B-A).
    • Repro: --groups gix-tree --only 'gix-tree/symlink-addition/A-B'
  4. fb-539 (F4): a mode change is silently lost (10 cases). High.
    • It happens in two situations: a chmod on one side against a content edit on the other, or both sides disagreeing on mode for the same content.
    • Core three_way_merge_manifests marks chmod-vs-edit as a conflict. The CLI resolver then text-merges cleanly and takes the branch entry's mode (mode: be.mode), so main's chmod is dropped and no conflict is reported. The result is asymmetric.
    • Cases: gix-tree/remove-executable-mode (B-A), same-rename-different-mode, identical-renames-to-same-destination-with-mode-change, and generated 004, 014, 021, 022 and 026. Generated 022 had been under-classified before QA.
    • Repro: --groups gix-tree --only 'gix-tree/remove-executable-mode/A-B'
  5. fb-541 (F6): criss-cross with divergent resolutions merges silently (1 case). Medium.
    • git-t6416-criss-cross-modify-delete: the branch resolved a modify/delete by deleting; main's merge D kept the file.
    • With two best common ancestors {B, C}, Oak takes C as the base and silently re-adds the file. Git reports a conflict.
    • The shape is a hybrid: main's merge D is an imported git merge commit.
    • Repro: --groups native --only 'git-t6416-*'

Observation (not counted as wrong)

While a sync is conflicted, the worktree still contains paths the merge deletes. Examples are git-t6423-8c/9b (B-A): z/b and z/c stay until oak pull --continue. The committed result is correct, and user edits made to those files during resolution are kept as uncommitted changes. The suite therefore compares the committed result (result_source) and reports the stale paths as conflict_worktree_stale_paths.

Differs-but-valid (134)

These cases are dominated by Oak having no rename or directory-rename detection. The t6423 directory-rename cases leave new files in the old directory, and rename/rename or rename/delete cases resolve cleanly by keeping both paths. Rename-vs-modify becomes a modify/delete conflict, which is a false conflict. None of these loses content or mode, fabricates content, or commits markers. Whether Oak should detect renames is a product decision, not a correctness bug.