Runners Log in
benchmarksPublic

oak/benchmarks

docs/prior-art.md 207 lines · 12.2 KB

Prior Art And External Instruments

This page records prior-art influence without importing another project's benchmark implementation. The suite may learn from behavior categories, reporting vocabulary, and measurement controls, but it must not vendor external benchmark code or translate external shell scripts into local Python, except for the recorded exception below.

Recorded Exception: gitoxide merge baselines (2026-09-28)

suites/merge/vendor/gitoxide/ vendors three gix-merge fixture scripts (tree-baseline.sh, tree-cartesian-baseline.sh, text-baseline.sh) verbatim from gitoxide commit acf906f8fcdb8e8d143570737786543dc5a204a0.

  • License: MIT OR Apache-2.0 (permissive). Both license texts and a NOTICE are carried next to the scripts. The SHA-256 of each script is pinned in UPSTREAM.json and checked by tests/test_merge_suite.py.
  • Rationale: the scripts are a curated, upstream-maintained catalogue of merge-ort edge cases, including an exhaustive 14x14 operation product. Re-deriving them would lose that coverage and add transcription risk. They are only executed with real Git to build fixture repositories. Git's own merge-ort computes the reference result, and no gitoxide code runs in the measured path.
  • Scope: this exception covers only these files. GPL sources, including Git's t64xx merge tests, stay clean-room: shapes are re-expressed from testcase prose and cited by ID, and no text is copied.

Clean-Room Rule

  • Treat external suites as behavioral indexes, not source material.
  • Copy no GPL shell, helper functions, fixtures, prompts, datasets, or generated repository states into this repository.
  • Re-express every benchmark as an Oak benchmark scenario with this repo's row contract, fixture policy, command-semantics contract, and null-honesty rules.
  • Keep external tools out of the timed path unless the scenario explicitly measures runner interference, host contention, or instrument cross-checking.
  • Record tool versions, command lines, host metadata, and whether the tool was installed by the runner or preexisting on the host.

Git t/perf

Git's upstream t/perf suite is the most relevant prior-art index for single-VCS implementation pressure. The Phase 0 scan uses test names and test titles as a behavioral map only. It does not port scripts, shell functions, fixtures, or expected outputs.

The coverage cross-reference lives in docs/benchmark-coverage.md. It maps Git perf-test areas to existing Oak benchmark evidence, planned scenario coverage, or explicit exclusions.

External-Instrument Policy

InstrumentUse In This RepoPolicy
hyperfineOptional timing cross-check for selected commands.Allowed as an installed binary invoked by scripts/crosscheck_hyperfine.py. Missing binaries emit returncode-77 skip rows on request. Hyperfine output is instrument-validation evidence, not a replacement for lane JSONL.
stress-ngOptional host-load injector for future contention or robustness experiments.Allowed only as an external installed binary. Rows must mark the stress profile and never mix stressed and unstressed samples in one aggregate.
GitGoodBenchAgentic VCS-task prior art: scenario taxonomy, dataset/reporting ideas, published Apache-2.0 dataset rows, and paper citation.Import local dataset exports through scripts/import_gitgoodbench.py. Do not copy prompts, repository samples, baseline tool code, notebooks, or scenario implementations into Oak benchmarks.
Phoronix Test SuiteOptional independent host characterization or broad system benchmark context.Allowed only as an external runner whose outputs are linked as environmental context, not as Oak-vs-Git evidence. Do not vendor GPL test profiles, result parsers, or PTS orchestration code.

Reporting Policy

External-instrument output can support a claim only when it is clearly labeled:

  • environmental: host load, CPU governor, storage health, or thermal context.
  • diagnostic: developer-only timing used to decide whether a harness run is worth paying for.
  • comparative: publishable Oak-vs-Git evidence, which must come from this repository's JSONL row contract rather than a foreign runner.

If an external tool affects the machine while Oak benchmark rows are collected, the row or run metadata must name the tool, version, command line, duration, and intended stress profile.

GitGoodBench Import Status

The Lite split importer is implemented in scripts/import_gitgoodbench.py. It consumes a local export of the dataset; rows reference upstream repositories, so repos must be mirrored under a local mirror root before a scenario goes ready. Anything unmaterializable becomes an honest skip entry with a machine-readable reason, never an error. Both oracle modes are recorded per scenario — strict exact-match (strict-em) and normalized (whitespace_and_blank_lines) — with evaluation deferred to the agent lane. The full ~900-scenario import is deferred to Phase 4.

Verified facts from the source (paper + repository + dataset card):

  • The published schema is id, name (owner/repository), default_branch, license, stargazers, created_at, topics, programming_language, scenario (embedded per-scenario data; raw Hugging Face CSV rows currently encode it as a Python-literal dict string), sample_type, project_size, difficulty. sample_type has exactly two values: merge and file_commit_chain.
  • A file_commit_chain sample (two commits bounding a chain in which one file changed in every intermediate commit) backs TWO scenario types: interactive rebase (clean up the local tree) and iterative commit creation from staged changes. Our importer expands each such sample into both.
  • Datasets are distributed via Hugging Face (JetBrains/git_good_bench*, Apache-2.0): 900-sample evaluation suite, 120-sample Lite split, and a 17,469-sample training corpus mined from 816 permissively licensed Python/Java/Kotlin repositories.
  • Their released harness is explicitly NOT runnable (proprietary code removed), so there is nothing to vendor even if we wanted to — scenarios run in OUR agent lane against our row contract. Their baseline (GPT-4o with custom Git tools) solves 21.11% of the Lite split, reported as success-rate only.
  • What we add that they do not measure: tokens, turns, wall clock, and thrash per scenario — resource-priced numbers on a published VCS agent benchmark — plus oak-vs-git on mined-real (not synthetic) scenarios. What we deliberately skip: their prompts, tool implementations, notebooks, and any scenario code (Apache-2.0 permits more, but the clean-room boundary keeps provenance unambiguous).

Cite when publishing results derived from these scenarios:

@inproceedings{lindenbauer-etal-2025-gitgoodbench,
    title = "{G}it{G}ood{B}ench: A Novel Benchmark For Evaluating Agentic Performance On Git",
    author = "Lindenbauer, Tobias and Bogomolov, Egor and Zharov, Yaroslav",
    booktitle = "Proceedings of the 1st Workshop for Research on Agent Language Models (REALM 2025)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.realm-1.19/",
    doi = "10.18653/v1/2025.realm-1.19",
    pages = "272--288",
    ISBN = "979-8-89176-264-0",
}

Git t/perf External Runner Status

The external cross-check runner is implemented in scripts/tperf_runner.py. It drives an existing git source tree's t/perf/run at arm's length and aggregates the raw t/test-results/*.result numbers (GPLv2 shell is never vendored or translated). Rows are labeled git_tperf_external; t/perf's min-of-N vs our median is a sanity band, not an equality check. Full cross-check campaigns are a perf-runner activity, not part of CI.

git-sizer (MIT) — scale-lane fixture statistics

suites/scale/scalesuite/sizer.py re-implements the metric set of git-sizer (MIT, Copyright (c) 2018 GitHub; commit 88eaa80d read 2026-09-28): the metric symbols and definitions from sizes/output.go (overall repository size, biggest objects, history structure, biggest checkouts) and its "level of concern" reference values. No git-sizer code is copied; the implementation walks objects with git plumbing. Attribution and the unmodified license text: suites/scale/third_party/git-sizer/{NOTICE,LICENSE-MIT}. Every scale-lane row carries a compact subset of these statistics as fixture provenance.

Git t/perf op list and repository tiers — scale lane

The scale lane (suites/scale) re-implements two ideas from Git's t/perf (GPL-2.0-only), without copying text or code:

  • the repository-tier concept of perf-lib.sh's fresh/default/large repository helpers, realized as deterministic synthetic tiers (fresh, default, large, XL axes) instead of copies of git.git or linux;
  • the operation list: each scale op cites the upstream test file names it corresponds to (tperf_refs in scenarios/scale.yaml), e.g. p0001-rev-list.sh and p4205-log-pretty-formats.sh for log.full, p4211-line-log.sh for log.path, p8020-last-modified.sh for blame, p1006-cat-file.sh for file.show, p6300-for-each-ref.sh for branch.list, p6010-merge-base.sh for the merge ops, p0006-read-tree-checkout.sh for switch, p5304-prune.sh / p5313-pack-objects.sh for gc, p5311-pack-bitmaps-fetch.sh for pull, p5600-partial-clone.sh / p5601-clone-reference.sh for clone.

Only file names are cited; test bodies, helpers and expected output were not read into or reproduced in this repository.

Roundtrip conformance matrix (P2) sources

The conformance matrix (scripts/oakbench/conformance_matrix.py, docs/vcs-conformance.md) follows the clean-room rule above. Nothing was copied or translated; every fixture, oracle and Python line is independently authored. Behaviours cite the upstream tests whose described behaviour they re-express (Git v2.50.1, d82adb61ba2fd11d8f2587fca1b6bd7925ce4044; test IDs verified against that tree):

Upstream (GPL-2.0, behaviour only)Re-expressed as
git t3902-quotedodd path names: space, tab, newline, control characters, quotes, backslash, leading dash, reserved names
git t0050-filesystem, t3910-mac-os-precomposecase-only renames, case-distinct names, NFC/NFD names and pairs
git t3700-add, t4129-apply-samemodeexecutable bit added/removed/combined with content changes
git t4011-diff-symlink, t4114-apply-typechange, t2007-checkout-symlink, t2003-checkout-cache-mkdirsymlink targets as content, file/symlink/directory type changes, links to directories
git t2021-checkout-overwritefile <-> directory replacement
git t0020-crlf, t0021-conversion, t0027-auto-crlfbyte-exact LF/CRLF/CR/mixed/no-final-newline content, whitespace-only and EOL-only edits
git t4012-diff-binarybinary-looking payloads (NUL, late NUL, high bytes, PNG header)
git t0010-racy-gitsame-size edit with the mtime restored
git t7001-mv, t4102-apply-renamerenames and moves
  • dulwich tests/compat (Apache-2.0 OR GPL-2.0-or-later, https://github.com/jelmer/dulwich): the pattern of running every operation against real Git and comparing results is adopted as the matrix's Git differential (git_differential on Oak rows). No dulwich code, fixtures or test text were copied, so no NOTICE file is required; this entry is the attribution.
  • Specifications implemented for the independent reader: BLAKE3 paper (Aumasson, Neves, O'Connor, Wilcox-O'Hearn, 2020; CC0/Apache-2.0 reference material, used as a specification only), RFC 8878 (Zstandard) and the XXH64 algorithm description. tests/data/blake3_vectors.json holds the official BLAKE3 test-vector hashes (input byte i = i % 251, lengths at block/chunk/tree boundaries; they equal the upstream test_vectors.json default-hash values, re-derived locally with the reference blake3 package). zstd_vectors.json holds frames produced locally by the zstd CLI. Both are data, not code.