VCS behavioral conformance and upstream review

# VCS behavioral conformance and upstream review

## Roundtrip conformance matrix (gated)

`scripts/vcs_conformance.py` runs two suites (`--suite legacy|matrix|all`,

default `all`): the five v1 scenarios below and the **roundtrip matrix**

(`scripts/oakbench/conformance_matrix.py`, `MATRIX_VERSION 2026-09-28.2`).

`--list` prints the catalog. Every matrix behaviour has an immutable ID

(ADR-0005) and runs against Git and Oak:

1. The harness authors a fixture (one or more steps) in a fresh repository;

   the expected state is the filesystem it wrote (path set, bytes, executable

   bit, symlink targets), never a VCS answer.

2. After every step it commits and pushes (Git: a local bare repository; Oak:

   a private loopback `oak serve` started and stopped by the run).

3. Legs, each pass/fail/unmeasured:

   - `commit_read[@step]`: the authoring repository's committed tree, read by an

     independent instrument. Oak: `oak_sqlite_reader` (deep + complete: every

     blob, tree, chunk and ancestor commit re-hashed with pure-Python

     BLAKE3/zstd, no Oak code or binary). Git: `ls-tree -r -z` + `cat-file --batch`.

   - `source_preserved`: commit/push did not rewrite the authoring worktree.

   - `clone_fs`, `clone_read`, `clone_status_clean`: a fresh clone of the pushed

     branch - filesystem bytes, independently read committed state, and the

     subject's own claim that the clone is clean.

   - `pull_fs`, `pull_read`, `pull_status_clean` (multi-step behaviours): a clone

     made after step 1, updated with `pull`.

4. Oak rows carry `git_differential`: Oak's independently read committed tree

   against Git's committed tree for the same fixture (`identical` / `differs: ...`).

Git runs with `core.autocrlf=false`, `core.precomposeunicode=false`,

`core.filemode=true`, `core.symlinks=true` (byte-exact contract) and is the real

Git, never the macOS xcrun shim (ADR-0011). Behaviours needing a filesystem

capability (case-sensitive, normalization-sensitive, non-UTF-8 names, sparse

files) skip with `filesystem capability unavailable: ...`. `huge_file_sparse_v1`

(1 GiB + 5 bytes) is gated: `--include-gated` or `OAK_BENCH_CONFORMANCE_HUGE=1`,

and needs >= 8 GiB free; its reader legs are unmeasured (reader bounds) while its

filesystem legs still compare streamed SHA-256.

| Group | Behaviours |

| --- | --- |

| paths | `path_space_v1`, `path_tab_v1`, `path_newline_v1`, `path_control_chars_v1`, `path_leading_dash_v1`, `path_shell_metachars_v1`, `path_backslash_v1`, `path_unicode_nfc_v1`, `path_unicode_nfd_v1`, `path_unicode_nfc_nfd_pair_v1`, `path_non_utf8_v1`, `path_long_component_v1`, `path_long_path_v1`, `path_deep_tree_v1`, `path_reserved_names_v1`, `path_case_only_rename_v1`, `path_case_distinct_v1` |

| modes | `mode_exec_new_v1`, `mode_exec_add_v1`, `mode_exec_remove_v1`, `mode_exec_and_content_v1` |

| symlinks | `symlink_relative_v1`, `symlink_absolute_v1`, `symlink_dangling_v1`, `symlink_to_dir_v1`, `symlink_target_edge_v1`, `symlink_retarget_v1`, `symlink_to_file_v1`, `file_to_symlink_v1`, `dir_to_symlink_v1`, `symlink_to_dir_real_v1` |

| file_dir | `file_to_dir_v1`, `dir_to_file_v1`, `file_to_deep_dir_v1` |

| empty | `empty_file_v1`, `empty_dir_semantics_v1` (directories recorded as observations, never failed), `dir_emptied_v1`, `delete_all_v1` |

| content | `bytes_all_values_v1`, `eol_lf_v1`, `eol_crlf_v1`, `eol_cr_only_v1`, `eol_mixed_v1`, `eol_no_final_newline_v1`, `eol_lf_to_crlf_v1`, `whitespace_only_change_v1`, `utf8_bom_v1`, `binary_detection_v1`, `binary_text_transition_v1`, `modify_same_size_same_mtime_v1`, `large_file_chunked_v1` (10 MiB + 64 KiB, Oak's chunked path), `huge_file_sparse_v1` |

| history | `delete_readd_v1`, `rename_move_v1`, `shared_blob_across_modes_v1`, `many_small_files_v1` |

Merge behaviour (symlink/mode/binary/file-directory merges) belongs to the P3

merge suite; this matrix is single-branch roundtrip only.

### Gate

`config/conformance_expected_failures.json` (schema 2) lists every known Oak

failure with its ticket (`fb-N`; `pending:<slug>` only until filed, and every

gate summary lists unfiled tickets), an `outcome` (`bug`; `refused` for a

typed policy refusal that needs a product decision, e.g. fb-531 control

characters in names; `intermittent` for a timing-dependent bug, where a pass is

a pass and never FIXED but any other failure is still a regression), and **per failing leg an error pattern**

(`re.fullmatch` against that leg's detail; legacy v1 scenarios use the

pseudo-leg `behaviour` against the row's `failure`). `oakbench/conformance_gate.py`

applies strict-xfail semantics to the Oak rows:

- expected failure: identical failing-leg set AND every detail matches its

  pattern. A new leg, or the same leg with a different error (EEXIST turning

  into `extra=['../../etc/evil']`, a refusal turning into a panic) is a REGRESSION;

- FIXED: a listed behaviour passes on a profile the entry applies to (remove

  the entry; publish_gate refuses a stale list);

- filesystem profiles: every row carries `fs_profile`

  (`<platform>-<ci|cs>-<ni|ns>`, e.g. `darwin-ci-ni` for default APFS). Entries

  may be scoped with `fs_profiles`; elsewhere they do not apply (no false FIXED;

  the same failure there is a regression). Rows mixing profiles are UNMEASURED;

- coverage: `coverage.<profile>.expected_skips` lists the skips a recorded

  profile may have. Any other skip (a pass or xfail that turned into a skip) is a

  coverage loss (UNMEASURED). On an unrecorded profile only behaviours needing a

  known filesystem-dependent capability (case sensitivity, Unicode

  normalization, non-UTF-8 names) or gated behaviours may skip; any other skip

  (e.g. a broken symlink/exec probe) is UNMEASURED. A row without `fs_profile`

  is UNMEASURED, and a run with zero measured Oak behaviours is always

  UNMEASURED. Record a profile once its first full run is reviewed;

- harness errors, reader-unmeasured legs and missing behaviours are UNMEASURED.

  Git rows are the reference arm and never gate.

`huge_file_sparse_v1` (gated): with `--include-gated` / `OAK_BENCH_CONFORMANCE_HUGE=1`

its filesystem legs are measured and must pass; its reader legs are over the

reader's 32 MiB object bound by design, so a clean run is recorded as a gated

skip (`gated: filesystem legs passed; reader legs over reader bound ...`), never

UNMEASURED and never a pass. A filesystem-leg failure is a regression.

- `python3 scripts/vcs_conformance.py --workdir W --gate` exits 0/1/2.

- `devloop.py` runs the `conformance` lane by default against the local Oak

  binary; regressions make the verdict REGRESSED, unmeasured UNMEASURED, and

  FIXED entries are printed as notes.

- `publish_gate.py` check 14: every published Oak row must carry binary identity

  (`binary_sha256`, or `subject_details[subject].binary_sha256`); the host's

  `fs_profile` must have a recorded `coverage` entry; for each

  published Oak binary the gate is evaluated on that binary's conformance rows

  only (rows of another build can neither satisfy nor fail it) and must be PASS

  with no FIXED entries.

### Reader budgets

The reader's deadline is 60 s plus 1.5 s per MiB actually verified (pure BLAKE3

~1.7 MB/s, zstd ~30 MB/s on an M4: about 2.5x margin), so large and chunk-only

payloads scale instead of timing out; the workflow oracle's subprocess budget is

120 s + 3 s per MiB of `oak.db`. Object/total bounds (32 MiB / 128 MiB) still

make larger reads unmeasured.

## Legacy v1 scenarios

The legacy suite runs five independently authored correctness

diagnostics against local Git and Oak binaries. It does not measure human or

agent success, tokens, latency, or throughput. Its records deliberately use

`conformance_schema_version`, not the benchmark row schema, and

`evidence_class: diagnostic_conformance`. **Do not ingest these records into

performance reports, devloop baselines, or public comparison tables.** They are

development prerequisites and expose unsupported behavior without concealing it.

Run from an isolated worker checkout:

```sh

python3 scripts/vcs_conformance.py --workdir workdirs/conformance

python3 scripts/vcs_conformance.py --workdir workdirs/conformance --subjects git

python3 -m unittest discover -s tests -p test_vcs_conformance.py

```

Optional `--git-bin`, `--oak-bin`, and comma-separated `--scenarios` select exact

binaries and scenarios. stdout is diagnostic JSONL; archive it outside source

control. Records include binary path, SHA-256, reported version, command argv,

exit codes, and captured output. Each scenario has a fresh disposable repository

and isolated home. No remote commands run. The CLI exits 1 on any failed

scenario and 0 otherwise; individual missing-capability records use code 77

and null integrity. All-skipped output is not a successful conformance result.

Do not run these subprocess-heavy checks during latency campaigns.

## Implemented scenarios and their exact contracts

| Scenario | Required behavior | Independent oracle |

| --- | --- | --- |

| `path_roundtrip_v1` | Checkpoint and restore space, tab, newline, quote, leading-dash, Unicode, and nested paths. | Delete every fixture file before restore; compare exact path set, SHA-256, size, type, executable bit, HEAD and branch. |

| `byte_roundtrip_v1` | Preserve empty files, all byte values, mixed CR/LF bytes and absent final LF. | Same destructive roundtrip oracle; no whitespace normalization. |

| `mode_symlink_roundtrip_v1` | Preserve executable bit, regular file, relative link and dangling link. | `lstat` and link target inspection; no symlink traversal; full roundtrip and identity check. |

| `scoped_restore_preservation_v1` | Restore one tracked path while leaving unrelated dirty and untracked payloads intact. | Full-tree expected-state comparison plus unchanged logical repository identity. |

| `failed_restore_preservation_v1` | Invalid source must fail normally and preserve all user files and logical identity. | Positive nonzero exit required; success, signal termination, any changed/extra/missing path, or identity change fails. |

All scenario and fixture identities are version 1. Git identity includes HEAD,

branch and exact index entries (NUL-delimited path/mode/blob/stage projection).

Oak identity includes HEAD and branch from JSON status, which must agree with

HEAD, branch and root tree read independently by `oak_sqlite_reader`

(oracle `exact_tree_and_logical_identity_v2`). This is logical-state

preservation, not proof of byte-for-byte metadata atomicity, transaction-log

durability, full-history preservation, or crash recovery. The tree oracle

excludes root `.git`/`.oak`, empty directories, timestamps, ACLs/xattrs and

non-executable mode bits. Those require separate contracts. The manifest

records symlinks themselves, including broken links, without following them.

The existing shared environment helper is reused, with inherited Git plumbing

variables removed to prevent commands being redirected outside their fixture.

Git uses explicit HEAD worktree restore; Oak uses its documented default HEAD

source and `--force` to avoid a confirmation prompt. Installed Oak v0.103.0

does not accept literal `--source HEAD`; that spelling is not assumed equivalent.

Missing binaries and unavailable filesystem symlink/executable support are

explicit skips. Product rejection of a valid fixture or command remains a

failure, not a capability skip.

Negative controls inject a no-op restore, lost untracked file, changed bytes,

renamed/extra paths, changed mode/link target/type, successful invalid-source

response, signal termination, and unexpected HEAD movement. An instrument

that accepts any of these must fail its own test suite.

## Reviewed upstream sources and port mapping

Reviewed 2026-09-09. These are deliberately pinned historical releases, not

claims about the latest suite. Source inspection used official Git/Jujutsu

GitHub repositories and Mercurial's official repository. No external test

script, helper, fixture, expected output, or generated repository was copied or

translated. This follows [prior-art policy](prior-art.md). Local fixture bytes,

paths, Python implementations and oracles are independently authored.

Git v2.50.1 resolves to commit

`d82adb61ba2fd11d8f2587fca1b6bd7925ce4044`; Jujutsu v0.31.0 to

`312c610c592b41ec81f0661e124fd660cb510ab0`; Mercurial 6.9 to

`b267c5764cc6b804c619a42067405f27e8705beb`.

| Inspected primary source | Behavior learned | Local use or next port |

| --- | --- | --- |

| [Git t3902 quoted paths](https://github.com/git/git/blob/d82adb61ba2fd11d8f2587fca1b6bd7925ce4044/t/t3902-quoted.sh) | Path names with whitespace, line separators, quotes and non-ASCII content must remain unambiguous across command output. | `path_roundtrip_v1` covers storage/restoration; exact JSON/NUL output parsing remains a separate gap. |

| [Git t2070 restore](https://github.com/git/git/blob/d82adb61ba2fd11d8f2587fca1b6bd7925ce4044/t/t2070-restore.sh) | Restoring a chosen file/source must respect worktree/index scope; invalid option combinations must fail. | Scoped restore and invalid-source preservation probes; byte roundtrip extends the exact-file comparison principle. Index-specific restore/merge-conflict behavior deferred. |

| [Git t2003 checkout-index](https://github.com/git/git/blob/d82adb61ba2fd11d8f2587fca1b6bd7925ce4044/t/t2003-checkout-cache-mkdir.sh) | Symlinks, file/directory collisions and output prefixes require file-type checks, not content alone. | Mode/link roundtrip is the prerequisite; replacement of symlinked parent directories and no-write-outside-checkout checks remain deferred. |

| [Git t7519 fsmonitor](https://github.com/git/git/blob/d82adb61ba2fd11d8f2587fca1b6bd7925ce4044/t/t7519-status-fsmonitor.sh) | Tuned status must preserve full-index answers; sparse cases may expand the index. | P1: compare exact dirty-path sets against untuned independently generated manifests across rename/delete/untracked changes, watcher reset and sparse boundaries. |

| [Jujutsu restore tests](https://github.com/jj-vcs/jj/blob/312c610c592b41ec81f0661e124fd660cb510ab0/cli/tests/test_restore_command.rs) | `test_restore` checks source/destination behavior; interactive path selection must not restore unselected edits. | Scoped preservation is a common behavioral contract; no Jujutsu subject adapter is claimed. |

| [Jujutsu undo tests](https://github.com/jj-vcs/jj/blob/312c610c592b41ec81f0661e124fd660cb510ab0/cli/tests/test_undo.rs) | `test_undo_rewrite_with_child`, `test_git_push_undo`, `test_git_push_undo_with_import` and repeated-undo cases expose graph and remote-tracking consequences. | P1: local-history recovery with descendant and branch-head oracles. P2: undo/restore after publish must distinguish local state from external remote state; never assume local undo retracts a push. |

| [Mercurial change/delete conflicts](https://repo.mercurial-scm.org/hg/file/b267c5764cc6b804c619a42067405f27e8705beb/tests/test-merge-changedelete.t) | Noninteractive conflicts preserve unresolved entries; malformed/insufficient input stays recoverable; transitions among resolution policies retain consistent content and merge state. | P1: independently generated edit/delete conflicts in both directions, retry/abort and re-resolution matrix. Exact file state and unresolved-path state must be checked separately. Not implemented by the simple invalid-source scenario. |

The repository clean-room restriction applies regardless of upstream license.

Git/Mercurial copyleft suites and Jujutsu's Apache-2.0 headers are provenance

context, not permission to vendor tests under this policy. There are no imported

upstream files to relicense. Preserve these links when evolving the scenarios;

new semantics require new scenario identities.

## Local diagnostic observation

The implementation run on macOS with installed Oak v0.103.0 passed byte,

mode/symlink, scoped-restore and failed-restore checks. The path roundtrip failed

at checkpoint because Oak rejects tab-containing filenames. Git passed all five.

This is a concrete path-support gap against the declared fixture, not evidence

of data corruption, slower performance, or a general reliability ranking. Later

path entries are not independently certified by a scenario that stopped early.

The runner intentionally retains the failure. Generated raw records are not

committed; rerun with pinned binaries before using the observation in a decision.

## Remaining priorities

1. Add merge edit/delete, rename/edit and rename/rename cases with exact parent,

   unresolved-path, result-tree and abort-preservation oracles; adapt these to

   the existing conflict corpus lane instead of duplicating that runner.

2. Add metamorphic tuned-status/sparse/full-tree parity and safe symlink-parent

   replacement. Distinguish absent tracked entries from absent hydrated data.

3. Add graph-aware recovery after rewrite and push, with fresh second-reader

   remote validation and failure injection at request/acknowledgment boundaries.

4. Apply proven oracles to varied agent tasks and controlled human studies.

   Upstream command tests do not establish discoverability, human task time,

   agent success rate, prompt familiarity, or statistical power.

This bounded port does not represent exhaustive upstream coverage. Submodules,

partial-clone object absence, filesystem normalization/case collisions, hostile

paths, fsync/crash durability, broad graph topologies and large-repo performance

need independently specified fixtures and supported comparator semantics.