Viewing files at commit 46ad54da0178Commit detailsView latest

Browse files
suites/merge/RESULTS-2026-09-28.md 140 lines · 7.5 KB

First run: oak 0.105.0 (2026-09-28)

This page summarizes the evidence. Raw JSONL is not committed (ADR-0003). It is kept in the author's worker at worker-mrmrs-20260927T090659-p3-merge/runs/full1.jsonl, with the summary at runs/full1-summary.json.

  • Subject. oak 0.105.0, sha256 4dc220e1f089845ca1ada8da4025b18b37f7823e266989e757302b8fa8bc6647. The path exercised is oak-pull-parent-sync, the client merge engine (see README).
  • Oracle. git version 2.50.1 (Apple Git-155) at /Applications/Xcode.app/Contents/Developer/usr/bin/git (the real Git; the xcrun shim was resolved), sha256 10f9c1df…d2a9.
  • Command. python3 suites/merge/run_merge_suite.py --workdir runs/full1 --jobs 8 --summary runs/full1-summary.json
  • Size. 387 scenarios and 765 directional cases.

Counts per class

originequal-to-gitdiffers-but-validoak-wrongoak-unsupportedharness-error
generated-law679400
git-t64xx-reimplemented1230000
gitoxide-cartesian14650142100
gitoxide-text-baseline5121100
gitoxide-tree-baseline474439200
native-incident80100
total331135692300

The 210 cartesian cases marked oak-unsupported are the gitlink matrix, because Oak's git import drops gitlinks. The 20 tree-baseline cases in that class are gitlink cases or cases with several git merge bases or no merge base.

Merge laws (true / measured)

LawResult
no_lost_change464/494
idempotent (second pull is a no-op after a clean merge)340/340
symmetric434/474
false_conflict (Oak conflicts where git is clean)42/497 cases, 8.5%
missed_conflict (git conflicts where Oak is clean)136/497 cases, 27%. Mostly rename and directory-rename cases, where Oak's path-level merge keeps both sides.
prediction_agrees (oak merge --dry-run --json vs the executed pull)497/497

Incident shapes

All incident shapes pass on the client path except F6:

  • stale base C vs X, with a partial local graph hydrated on pull
  • the base reached via a merge parent
  • revert after a CLI sync
  • criss-cross tie-break, both timestamp orders
  • clock-skew best common ancestor, plus its monotone control

The lineage-less sync shape resurrects the reverted change in both Git and Oak. That is inherent once lineage is lost, and it is recorded as intent_note. The hosted-merge shapes (M0a, M0b, the server /sync, client-vs-server tie-break, server clock-skew, fb-530) are phase-2 specs and were not executed.

Oak-wrong findings (69 cases, 6 families)

Oak is not fixed here. Each family gives the case count, a minimal repro (--only runs just that case), and the observed behavior. The repro prefix is always:

python3 suites/merge/run_merge_suite.py --workdir /tmp/mr --groups <group> --only '<id>'

The failing case directory, with both clones, stays under <workdir>/cases/<id>/<direction>/.

  1. F5: file↔directory transitions make oak pull fail (38 cases). High. The minimal case is gix-cartesian/matrix/keep--file-to-dir (A-B): main replaces file source with source/child, and the branch did not touch it. Parent sync fails with IO error: File exists (os error 17). Other shapes fail with Not a directory (os error 20), cannot build tree with both file "e" and child path "e/e", or refusing to replace non-empty directory … with a tracked file. Git merges most of these cleanly. Any branch whose parent converts a file to a directory, or the reverse, cannot sync. The likely owner is worktree materialization in the sync and conflict paths; for example, write_manifest_to_workdir uses DeleteScope::Nothing. This has not been confirmed by instrumentation. Repro: --groups gix-cartesian --only 'gix-cartesian/matrix/keep--file-to-dir'
  2. F1: conflict markers committed as a clean merge (11 cases). Blocker (invalid merge). This happens when a side lacks a trailing newline, or with CRLF/LF marker handling. The diffy output glues markers mid-line (line3x||||||| original, line3=======), the marker detector misses them, and oak pull reports merged and commits the markers. Cases: gix-text/complex/conflict-without-LF, complex/missing-LF-at-EOF, complex/marker-newline-handling-{lf,lf2,crlf}, and unique-merge-base-with-insertion. Git reports a conflict in all of them. Repro: --groups gix-text --only 'gix-text/complex/conflict-without-LF'
  3. F2: symlinks changed on both sides get text-merged into marker garbage and committed clean (7 cases). High. Cases: gix-tree/symlink-addition, symlink-modification, and type-change-to-symlink. The link target becomes <<<<<<< branch\na||||||| original\n=======\nb>>>>>>> main. rename-add-symlink (B-A) needs further triage: the symlink side is dropped under a reported conflict on another path. Repro: --groups gix-tree --only 'gix-tree/symlink-addition/A-B'
  4. F4: a mode change is silently lost when the other side changes content, or both sides disagree on mode (10 cases). High. Core three_way_merge_manifests marks chmod-vs-edit as a conflict. The CLI resolver then text-merges cleanly and takes the branch entry's mode (mode: be.mode), so main's chmod is dropped and no conflict is reported. The result is asymmetric (only the direction where main changed the mode loses it). Cases: gix-tree/remove-executable-mode (B-A), same-rename-different-mode, identical-renames-to-same-destination-with-mode-change, and generated cases 004, 014, 021 and 026 (overlapping edit vs chmod). Repro: --groups gix-tree --only 'gix-tree/remove-executable-mode/A-B'
  5. F3: when both sides modify a binary file, pull reports clean and keeps the branch version (2 cases). High. gix-tree/both-modify-binary: oak pull exits 0 with merged, and main's binary change is silently lost. The source's binary-conflict path keeps the branch version; whether and where the sync drops the conflict flag was not instrumented. Git reports a conflict. Repro: --groups gix-tree --only 'gix-tree/both-modify-binary/A-B'
  6. F6: criss-cross with divergent resolutions merges silently (1 case). Medium. git-t6416-criss-cross-modify-delete: the branch resolved a modify/delete by deleting, while main (merge D) resolved the same one by keeping. With two best common ancestors {B, C}, Oak takes C as the base and silently re-adds the file. Git reports a conflict. Repro: --groups native --only 'git-t6416-*'

Observation (not counted as wrong)

While a sync is conflicted, the worktree still contains paths the merge deletes. Examples are git-t6423-8c/9b (B-A): z/b and z/c stay until oak pull --continue. The committed result is correct, and user edits made to those files during resolution are kept as uncommitted changes. The suite therefore compares the committed result (result_source) and reports the stale paths as conflict_worktree_stale_paths.

Differs-but-valid (135)

These cases are dominated by Oak having no rename or directory-rename detection. The t6423 directory-rename cases leave new files in the old directory, and rename/rename or rename/delete cases resolve cleanly by keeping both paths. Rename-vs-modify becomes a modify/delete conflict, which is a false conflict. None of these loses or fabricates content. Whether Oak should detect renames is a product decision, not a correctness bug.