Prior Art And External Instruments

# Prior Art And External Instruments

This page records prior-art influence without importing another project's

benchmark implementation. The suite may learn from behavior categories,

reporting vocabulary, and measurement controls, but it must not vendor external

benchmark code or translate external shell scripts into local Python, except

for the recorded exception below.

## Recorded Exception: gitoxide merge baselines (2026-09-28)

`suites/merge/vendor/gitoxide/` vendors three gix-merge fixture scripts

(`tree-baseline.sh`, `tree-cartesian-baseline.sh`, `text-baseline.sh`)

verbatim from gitoxide commit `acf906f8fcdb8e8d143570737786543dc5a204a0`.

- License: MIT OR Apache-2.0 (permissive). Both license texts and a `NOTICE`

  are carried next to the scripts. The SHA-256 of each script is pinned in

  `UPSTREAM.json` and checked by `tests/test_merge_suite.py`.

- Rationale: the scripts are a curated, upstream-maintained catalogue of

  merge-ort edge cases, including an exhaustive 14x14 operation product.

  Re-deriving them would lose that coverage and add transcription risk. They

  are only executed with real Git to build fixture repositories. Git's own

  merge-ort computes the reference result, and no gitoxide code runs in the

  measured path.

- Scope: this exception covers only these files. GPL sources, including

  Git's t64xx merge tests, stay clean-room: shapes are re-expressed from

  testcase prose and cited by ID, and no text is copied.

## Clean-Room Rule

- Treat external suites as behavioral indexes, not source material.

- Copy no GPL shell, helper functions, fixtures, prompts, datasets, or

  generated repository states into this repository.

- Re-express every benchmark as an Oak benchmark scenario with this repo's row

  contract, fixture policy, command-semantics contract, and null-honesty rules.

- Keep external tools out of the timed path unless the scenario explicitly

  measures runner interference, host contention, or instrument cross-checking.

- Record tool versions, command lines, host metadata, and whether the tool was

  installed by the runner or preexisting on the host.

## Git `t/perf`

Git's upstream `t/perf` suite is the most relevant prior-art index for

single-VCS implementation pressure. The Phase 0 scan uses test names and test

titles as a behavioral map only. It does not port scripts, shell functions,

fixtures, or expected outputs.

The coverage cross-reference lives in `docs/benchmark-coverage.md`. It maps

Git perf-test areas to existing Oak benchmark evidence, planned scenario

coverage, or explicit exclusions.

## External-Instrument Policy

| Instrument | Use In This Repo | Policy |

| --- | --- | --- |

| [`hyperfine`](https://github.com/sharkdp/hyperfine) | Optional timing cross-check for selected commands. | Allowed as an installed binary invoked by `scripts/crosscheck_hyperfine.py`. Missing binaries emit returncode-77 skip rows on request. Hyperfine output is instrument-validation evidence, not a replacement for lane JSONL. |

| [`stress-ng`](https://github.com/ColinIanKing/stress-ng) | Optional host-load injector for future contention or robustness experiments. | Allowed only as an external installed binary. Rows must mark the stress profile and never mix stressed and unstressed samples in one aggregate. |

| [GitGoodBench](https://github.com/JetBrains-Research/git-good-bench) | Agentic VCS-task prior art: scenario taxonomy, dataset/reporting ideas, published Apache-2.0 dataset rows, and paper citation. | Import local dataset exports through `scripts/import_gitgoodbench.py`. Do not copy prompts, repository samples, baseline tool code, notebooks, or scenario implementations into Oak benchmarks. |

| [Phoronix Test Suite](https://www.phoronix-test-suite.com/) | Optional independent host characterization or broad system benchmark context. | Allowed only as an external runner whose outputs are linked as environmental context, not as Oak-vs-Git evidence. Do not vendor GPL test profiles, result parsers, or PTS orchestration code. |

## Reporting Policy

External-instrument output can support a claim only when it is clearly labeled:

- `environmental`: host load, CPU governor, storage health, or thermal context.

- `diagnostic`: developer-only timing used to decide whether a harness run is

  worth paying for.

- `comparative`: publishable Oak-vs-Git evidence, which must come from this

  repository's JSONL row contract rather than a foreign runner.

If an external tool affects the machine while Oak benchmark rows are collected,

the row or run metadata must name the tool, version, command line, duration,

and intended stress profile.

## GitGoodBench Import Status

The Lite split importer is implemented in `scripts/import_gitgoodbench.py`. It

consumes a local export of the dataset; rows reference upstream repositories,

so repos must be mirrored under a local mirror root before a scenario goes

`ready`. Anything unmaterializable becomes an honest skip entry with a

machine-readable reason, never an error. Both oracle modes are recorded per

scenario — strict exact-match (`strict-em`) and `normalized`

(`whitespace_and_blank_lines`) — with evaluation deferred to the agent lane.

The full ~900-scenario import is deferred to Phase 4.

Verified facts from the source (paper + repository + dataset card):

- The published schema is `id`, `name` (owner/repository), `default_branch`,

  `license`, `stargazers`, `created_at`, `topics`, `programming_language`,

  `scenario` (embedded per-scenario data; raw Hugging Face CSV rows currently

  encode it as a Python-literal dict string), `sample_type`, `project_size`,

  `difficulty`. `sample_type` has exactly two values: `merge` and

  `file_commit_chain`.

- A `file_commit_chain` sample (two commits bounding a chain in which one

  file changed in every intermediate commit) backs TWO scenario types:

  interactive rebase (clean up the local tree) and iterative commit creation

  from staged changes. Our importer expands each such sample into both.

- Datasets are distributed via Hugging Face (`JetBrains/git_good_bench*`,

  Apache-2.0): 900-sample evaluation suite, 120-sample Lite split, and a

  17,469-sample training corpus mined from 816 permissively licensed

  Python/Java/Kotlin repositories.

- Their released harness is explicitly NOT runnable (proprietary code

  removed), so there is nothing to vendor even if we wanted to — scenarios

  run in OUR agent lane against our row contract. Their baseline (GPT-4o

  with custom Git tools) solves 21.11% of the Lite split, reported as

  success-rate only.

- What we add that they do not measure: tokens, turns, wall clock, and

  thrash per scenario — resource-priced numbers on a published VCS agent

  benchmark — plus oak-vs-git on mined-real (not synthetic) scenarios. What

  we deliberately skip: their prompts, tool implementations, notebooks, and

  any scenario code (Apache-2.0 permits more, but the clean-room boundary

  keeps provenance unambiguous).

Cite when publishing results derived from these scenarios:

```bibtex

@inproceedings{lindenbauer-etal-2025-gitgoodbench,

    title = "{G}it{G}ood{B}ench: A Novel Benchmark For Evaluating Agentic Performance On Git",

    author = "Lindenbauer, Tobias and Bogomolov, Egor and Zharov, Yaroslav",

    booktitle = "Proceedings of the 1st Workshop for Research on Agent Language Models (REALM 2025)",

    month = jul,

    year = "2025",

    address = "Vienna, Austria",

    publisher = "Association for Computational Linguistics",

    url = "https://aclanthology.org/2025.realm-1.19/",

    doi = "10.18653/v1/2025.realm-1.19",

    pages = "272--288",

    ISBN = "979-8-89176-264-0",

}

```

## Git `t/perf` External Runner Status

The external cross-check runner is implemented in `scripts/tperf_runner.py`.

It drives an existing git source tree's `t/perf/run` at arm's length and

aggregates the raw `t/test-results/*.result` numbers (GPLv2 shell is never

vendored or translated). Rows are labeled `git_tperf_external`; t/perf's

min-of-N vs our median is a sanity band, not an equality check. Full

cross-check campaigns are a perf-runner activity, not part of CI.

## git-sizer (MIT) — scale-lane fixture statistics

`suites/scale/scalesuite/sizer.py` re-implements the metric set of

[git-sizer](https://github.com/github/git-sizer) (MIT, Copyright (c) 2018

GitHub; commit 88eaa80d read 2026-09-28): the metric symbols and definitions

from `sizes/output.go` (overall repository size, biggest objects, history

structure, biggest checkouts) and its "level of concern" reference values.

No git-sizer code is copied; the implementation walks objects with git

plumbing. Attribution and the unmodified license text:

`suites/scale/third_party/git-sizer/{NOTICE,LICENSE-MIT}`. Every scale-lane

row carries a compact subset of these statistics as fixture provenance.

## Git `t/perf` op list and repository tiers — scale lane

The scale lane (`suites/scale`) re-implements two ideas from Git's

`t/perf` (GPL-2.0-only), without copying text or code:

- the **repository-tier concept** of `perf-lib.sh`'s fresh/default/large

  repository helpers, realized as deterministic synthetic tiers (fresh,

  default, large, XL axes) instead of copies of git.git or linux;

- the **operation list**: each scale op cites the upstream test file names it

  corresponds to (`tperf_refs` in `scenarios/scale.yaml`), e.g.

  `p0001-rev-list.sh` and `p4205-log-pretty-formats.sh` for `log.full`,

  `p4211-line-log.sh` for `log.path`, `p8020-last-modified.sh` for `blame`,

  `p1006-cat-file.sh` for `file.show`, `p6300-for-each-ref.sh` for

  `branch.list`, `p6010-merge-base.sh` for the merge ops,

  `p0006-read-tree-checkout.sh` for switch, `p5304-prune.sh` /

  `p5313-pack-objects.sh` for `gc`, `p5311-pack-bitmaps-fetch.sh` for pull,

  `p5600-partial-clone.sh` / `p5601-clone-reference.sh` for clone.

Only file names are cited; test bodies, helpers and expected output were not

read into or reproduced in this repository.

## Roundtrip conformance matrix (P2) sources

The conformance matrix (`scripts/oakbench/conformance_matrix.py`,

`docs/vcs-conformance.md`) follows the clean-room rule above. Nothing was

copied or translated; every fixture, oracle and Python line is independently

authored. Behaviours cite the upstream tests whose *described behaviour* they

re-express (Git v2.50.1, `d82adb61ba2fd11d8f2587fca1b6bd7925ce4044`; test IDs

verified against that tree):

| Upstream (GPL-2.0, behaviour only) | Re-expressed as |

| --- | --- |

| git t3902-quoted | odd path names: space, tab, newline, control characters, quotes, backslash, leading dash, reserved names |

| git t0050-filesystem, t3910-mac-os-precompose | case-only renames, case-distinct names, NFC/NFD names and pairs |

| git t3700-add, t4129-apply-samemode | executable bit added/removed/combined with content changes |

| git t4011-diff-symlink, t4114-apply-typechange, t2007-checkout-symlink, t2003-checkout-cache-mkdir | symlink targets as content, file/symlink/directory type changes, links to directories |

| git t2021-checkout-overwrite | file <-> directory replacement |

| git t0020-crlf, t0021-conversion, t0027-auto-crlf | byte-exact LF/CRLF/CR/mixed/no-final-newline content, whitespace-only and EOL-only edits |

| git t4012-diff-binary | binary-looking payloads (NUL, late NUL, high bytes, PNG header) |

| git t0010-racy-git | same-size edit with the mtime restored |

| git t7001-mv, t4102-apply-rename | renames and moves |

- **dulwich `tests/compat`** (Apache-2.0 OR GPL-2.0-or-later,

  https://github.com/jelmer/dulwich): the *pattern* of running every operation

  against real Git and comparing results is adopted as the matrix's Git

  differential (`git_differential` on Oak rows). No dulwich code, fixtures or

  test text were copied, so no NOTICE file is required; this entry is the

  attribution.

- **Specifications implemented for the independent reader**: BLAKE3 paper

  (Aumasson, Neves, O'Connor, Wilcox-O'Hearn, 2020; CC0/Apache-2.0 reference

  material, used as a specification only), RFC 8878 (Zstandard) and the XXH64

  algorithm description. `tests/data/blake3_vectors.json` holds the official

  BLAKE3 test-vector hashes (input byte i = i % 251, lengths at block/chunk/tree

  boundaries; they equal the upstream `test_vectors.json` default-hash values,

  re-derived locally with the reference `blake3` package). `zstd_vectors.json`

  holds frames produced locally by the `zstd` CLI. Both are data, not code.