/oak
Log in
benchmarks
docs/prior-art.md 118 lines · 6.8 KB

Prior Art And External Instruments

This page records prior-art influence without importing another project's benchmark implementation. The suite may learn from behavior categories, reporting vocabulary, and measurement controls, but it must not vendor external benchmark code or translate external shell scripts into local Python.

Clean-Room Rule

  • Treat external suites as behavioral indexes, not source material.
  • Copy no GPL shell, helper functions, fixtures, prompts, datasets, or generated repository states into this repository.
  • Re-express every benchmark as an Oak benchmark scenario with this repo's row contract, fixture policy, command-semantics contract, and null-honesty rules.
  • Keep external tools out of the timed path unless the scenario explicitly measures runner interference, host contention, or instrument cross-checking.
  • Record tool versions, command lines, host metadata, and whether the tool was installed by the runner or preexisting on the host.

Git t/perf

Git's upstream t/perf suite is the most relevant prior-art index for single-VCS implementation pressure. The Phase 0 scan uses test names and test titles as a behavioral map only. It does not port scripts, shell functions, fixtures, or expected outputs.

The coverage cross-reference lives in docs/benchmark-coverage.md. It maps Git perf-test areas to existing Oak benchmark evidence, planned scenario coverage, or explicit exclusions.

External-Instrument Policy

InstrumentUse In This RepoPolicy
hyperfineOptional timing cross-check for selected commands.Allowed as an installed binary invoked by scripts/crosscheck_hyperfine.py. Missing binaries emit returncode-77 skip rows on request. Hyperfine output is instrument-validation evidence, not a replacement for lane JSONL.
stress-ngOptional host-load injector for future contention or robustness experiments.Allowed only as an external installed binary. Rows must mark the stress profile and never mix stressed and unstressed samples in one aggregate.
GitGoodBenchAgentic VCS-task prior art: scenario taxonomy, dataset/reporting ideas, published Apache-2.0 dataset rows, and paper citation.Import local dataset exports through scripts/import_gitgoodbench.py. Do not copy prompts, repository samples, baseline tool code, notebooks, or scenario implementations into Oak benchmarks.
Phoronix Test SuiteOptional independent host characterization or broad system benchmark context.Allowed only as an external runner whose outputs are linked as environmental context, not as Oak-vs-Git evidence. Do not vendor GPL test profiles, result parsers, or PTS orchestration code.

Reporting Policy

External-instrument output can support a claim only when it is clearly labeled:

  • environmental: host load, CPU governor, storage health, or thermal context.
  • diagnostic: developer-only timing used to decide whether a harness run is worth paying for.
  • comparative: publishable Oak-vs-Git evidence, which must come from this repository's JSONL row contract rather than a foreign runner.

If an external tool affects the machine while Oak benchmark rows are collected, the row or run metadata must name the tool, version, command line, duration, and intended stress profile.

GitGoodBench Import Status

The Lite split importer is implemented in scripts/import_gitgoodbench.py. It consumes a local export of the dataset; rows reference upstream repositories, so repos must be mirrored under a local mirror root before a scenario goes ready. Anything unmaterializable becomes an honest skip entry with a machine-readable reason, never an error. Both oracle modes are recorded per scenario — strict exact-match (strict-em) and normalized (whitespace_and_blank_lines) — with evaluation deferred to the agent lane. The full ~900-scenario import is deferred to Phase 4.

Verified facts from the source (paper + repository + dataset card):

  • The published schema is id, name (owner/repository), default_branch, license, stargazers, created_at, topics, programming_language, scenario (embedded per-scenario data; raw Hugging Face CSV rows currently encode it as a Python-literal dict string), sample_type, project_size, difficulty. sample_type has exactly two values: merge and file_commit_chain.
  • A file_commit_chain sample (two commits bounding a chain in which one file changed in every intermediate commit) backs TWO scenario types: interactive rebase (clean up the local tree) and iterative commit creation from staged changes. Our importer expands each such sample into both.
  • Datasets are distributed via Hugging Face (JetBrains/git_good_bench*, Apache-2.0): 900-sample evaluation suite, 120-sample Lite split, and a 17,469-sample training corpus mined from 816 permissively licensed Python/Java/Kotlin repositories.
  • Their released harness is explicitly NOT runnable (proprietary code removed), so there is nothing to vendor even if we wanted to — scenarios run in OUR agent lane against our row contract. Their baseline (GPT-4o with custom Git tools) solves 21.11% of the Lite split, reported as success-rate only.
  • What we add that they do not measure: tokens, turns, wall clock, and thrash per scenario — resource-priced numbers on a published VCS agent benchmark — plus oak-vs-git on mined-real (not synthetic) scenarios. What we deliberately skip: their prompts, tool implementations, notebooks, and any scenario code (Apache-2.0 permits more, but the clean-room boundary keeps provenance unambiguous).

Cite when publishing results derived from these scenarios:

@inproceedings{lindenbauer-etal-2025-gitgoodbench,
    title = "{G}it{G}ood{B}ench: A Novel Benchmark For Evaluating Agentic Performance On Git",
    author = "Lindenbauer, Tobias and Bogomolov, Egor and Zharov, Yaroslav",
    booktitle = "Proceedings of the 1st Workshop for Research on Agent Language Models (REALM 2025)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.realm-1.19/",
    doi = "10.18653/v1/2025.realm-1.19",
    pages = "272--288",
    ISBN = "979-8-89176-264-0",
}

Git t/perf External Runner Status

The external cross-check runner is implemented in scripts/tperf_runner.py. It drives an existing git source tree's t/perf/run at arm's length and aggregates the raw t/test-results/*.result numbers (GPLv2 shell is never vendored or translated). Rows are labeled git_tperf_external; t/perf's min-of-N vs our median is a sanity band, not an equality check. Full cross-check campaigns are a perf-runner activity, not part of CI.