Runners Log in

oak/benchmarks

docs/tuned-git-baselines.md 82 lines · 4.9 KB

Tuned Git Baselines

Stock Git is not enough for public claims. The benchmark suite should compare Oak against credible Git modes where those modes apply.

The machine-readable source of truth is config/git_modes.json.

Modes

ModePurposeApplies ToOak Comparison
git_stockDefault installed Git (the real binary; the macOS xcrun shim is resolved or refused).All scenarios.oak_default, oak_virtual_branch.
git_scalarScalar-recommended local configuration: core.fsmonitor, core.untrackedCache, core.preloadIndex, index.threads, index.version=4, core.multiPackIndex, core.commitGraph, status.aheadBehind=false, gc.auto=0; fsmonitor daemon started and cache warmed untimed.Large-tree scenarios (many_small_files, wide_dirty_tree, many_small_files_50k).oak_default.
git_untracked_cacheSpeed up status in large trees with many untracked paths.Large-tree, ignored-heavy, wide-dirty-tree scenarios.oak_default.
git_split_indexReduce index write cost in large working trees.Large-tree and wide-dirty-tree scenarios.oak_default.
git_fsmonitorAvoid full filesystem scans when platform support exists.Large working-tree scenarios.oak_default.
git_sparse_checkoutMaterialize only task-relevant paths.Sparse task and large-tree scenarios.oak_space_mount, oak_mount_per_task.
git_worktreeIsolate per-task branches.Parallel-agent and branch-per-task scenarios.oak_space_mount, oak_virtual_branch.
git_lfsAvoid storing large binary payloads directly in Git objects.Binary-heavy scenarios when LFS is installed.oak_default, oak_space_mount.

The macOS xcrun shim

/usr/bin/git on macOS is a trampoline that re-resolves the developer directory on every invocation before exec'ing the real Git. Measuring it adds a fixed per-call cost and hashes the stub rather than the binary that ran. oakbench.subjects resolves it (xcrun --find git, the selected developer directory, then Command Line Tools) and refuses to run when it cannot, unless --allow-git-shim / OAK_BENCH_ALLOW_GIT_SHIM=1 is set. Rows record subject_bin, subject_version, binary_sha256, git_requested_bin, git_shim_measured and git_config_profile (ADR-0011).

Scalar-tuned Git

git_scalar applies the subset of Scalar's set_recommended_config that affects local operations, as mode.setup.scalar_* rows after repo.init. Rows are labelled git_config_profile: scalar_recommended; stock Git rows say stock. The fsmonitor daemon is started and the untracked cache warmed by recorded mode.setup.scalar_fsmonitor_start / mode.setup.scalar_warm_status rows before any timed operation (the same for git_fsmonitor). On small fixtures fsmonitor IPC makes this mode slower than stock Git (measured 2026-09-27: status.clean 8.96 vs 5.32 ms on 24 files), so it is NOT a stronger baseline there and does not apply there. The bench summary's "Oak vs the strongest Git baseline per cell" table compares Oak with the fastest Git subject that measured each cell; never quote Oak against a weaker Git mode. Scalar's background maintenance (commit-graph and multi-pack-index file writes, prefetch) is not run: core-lane histories have 1-3 commits, where those files do not apply. The fsmonitor daemon is stopped after each run.

Fairness Rules

  • Report tuned modes separately from stock Git.
  • Mark unsupported modes explicitly; do not silently drop them.
  • Do not use sparse checkout as the full-tree baseline.
  • Do not compare Git LFS to normal Oak storage unless the scenario is explicitly about external binary-storage workflows.
  • Include setup costs only when the scenario is about task startup or isolation.

Implementation Notes

Each mode needs:

  • setup commands
  • teardown commands if any
  • compatibility checks
  • fixture applicability
  • exact command mapping for timed operations
  • correctness checks proving the mode saw the intended paths

The public claim should be “Oak beats stock Git” or “Oak beats tuned Git mode X” depending on which rows actually ran.

scripts/bench.py --git-modes lfs is wired for the binary fixture scenarios named in git_lfs.applies_to (this file's table is prose; the JSON list is enforced). The mode runs git lfs install --local and git lfs track "*.bin" after repo.init and before snapshot.initial, recorded as mode.setup.lfs_install and mode.setup.lfs_track operation rows (mirroring mode.setup.fsmonitor), so the generated binary files are tracked through LFS before the first commit and the setup cost stays visible. Scenarios outside applies_to and hosts without Git LFS emit scenario.mode_skipped rows, and a fixture file the track patterns miss fails setup closed with a mode.setup.lfs_track_coverage skip row (lfs_track_incomplete:<ext>) — an untracked comparator is an unfair comparator. All of these are unmeasured, not failures or hidden stock Git measurements.