/oak
Log in
benchmarks
docs/platform-benchmarks.md 114 lines · 5.9 KB

Platform Benchmarks (platform lane)

Purpose

The platform lane measures the hosted side of the task loop: GitHub-the-platform vs oak hosted workflows, from "task branch exists locally" to "change merged in main, present in local checkout" — including the async waits agents actually experience. GitHub computes PR mergeability asynchronously; on this lane that settling is a first-class measured operation, not noise to be averaged away.

Entry point: scripts/platform_lifecycle.py. Scenarios and their immutable operation vocabulary live in scenarios/platform.yaml (ADR-0005). MVP scope is pr_single_anatomy and poll_until_merged_i10s on the protection-none and check-instant variants, plus platform_capability_probe.

Two drivers

  • driver="cli" shells out to gh/git/oak through run_timed, the same measurement primitive as every CLI lane: full token and output accounting from captured output, because this is what an agent's terminal actually sees.
  • driver="harness-api" speaks stdlib-urllib REST to the GitHub API (oakbench/platform_api.py). It records exact api_metrics per operation: request_count, request_bytes, response_bytes, last-seen rate_limit_remaining, and GitHub request ids (capped at 50). Its token fields are null: REST JSON is not an agent transcript, so any token number would be fiction — and null means unmeasured, never zero (ADR-0002).
  • driver="fake-provider" is only for deterministic branch-triage oracle coverage. It runs no terminal commands and is not comparable to hosted platform timings; token/output byte fields are null because no agent-visible command transcript exists.

RateLimitBudgeter paces API runs under an ops-per-hour budget; injected waits are returned and recorded (race.pacing.injected), never silently folded into operation durations.

Clock discipline and poll quantization

All headline durations come from a single local monotonic clock (oakbench/platform_clock.py). Async settling is observed by polling, so every settle row carries poll_interval_s and poll_quantization_ms (= interval × 1000): a settle measured at 10s cadence has 10s resolution, and the harness prints that bound the way devloop prints detection limits. poll_until_merged_i10s encodes its 10-second cadence in the scenario name; a different cadence is a different scenario, never a silent change.

Timeout policy: a timed-out settle did not complete, so settled: false and settle_elapsed_ms/elapsed-to-settled is null (the settled state was never observed); the wall time spent watching is recorded as observed_wait_ms. Server-clock values (e.g. Date headers) may be recorded but only wrapped by bounded_cross_clock, marked {"cross_clock": true, "descriptive_only": true}.

Fairness contract

config/platform_semantics.json (versioned, semantics_version: platform-v1) declares the semantics classes both platforms must satisfy before comparison — MVP runs integration-default (publish required, reviewable integration unit, platform-default merge, merged result syncable locally). Every row carries platform, driver, platform_semantics_version, platform_semantics_class, protection_variant, platform_comparison_key, and semantic_contract_match (true when the run executed a declared class under an implemented protection variant; cross-platform comparisons join on platform_comparison_key, which includes both class and protection variant). All six protection variants are declared now to reserve their names (ADR-0005); protection-none and check-instant are implemented — requesting another yields skip rows.

Where a platform has no equivalent operation (oak's merge is synchronous, so integration.mergeable.settle has nothing to wait for), the row is structural: poll_iterations: 0 is a measured zero (zero commands ran), the settle value is null ("no async mergeability concept") — null means no equivalent, per ADR-0002.

Comparability policy

  • Always comparable across platforms (same semantics class + variant): round trips (request_count, tool_call_count), bytes, poll iterations, tokens (cli driver only).
  • Wall clock: only with per-platform RTT context attached and an explicit caveat, and never at finer resolution than the poll quantization. Rows carry runner identity (runner_id/runner_class, ADR-0007) because vantage point is part of the measurement.

Skip honesty and CI

Without credentials/repos/binaries the lane emits per-operation skip rows with explicit reasons (github_token_missing:<env>, gh_cli_missing, oak_cli_missing, oak_remote_missing, github_repo_not_configured, protection_variant_not_implemented:<v>) and exits 3 when everything was skipped — coverage gaps are recorded, never silent. This is the only path CI exercises; tests use fake transports and fake clocks and never touch GitHub.

Branch Triage

branch_triage_n4 seeds four branch/PR review cases: clean, stale, duplicate, and conflicting. The fake-provider path classifies each branch, emits an action plan (merge, refresh_then_merge, close_duplicate, keep_open_manual_resolution), applies it, then checks both the expected plan and the final survivors. Rows include branches_reviewed, branches_closed, merge_order_correct, stale_detected_count, duplicate_detected_count, and conflict_detected_count.

Real GitHub/Oak execution is intentionally skip-only in this minimal lane until a concrete provider is wired. The Oak implementation should use the existing checkout-free branch review probes (oak branch review --json <branch> and oak branch diff --json <branch>) declared in config/command_semantics.json; until then, real drivers emit branch_triage_real_provider_not_implemented:<platform>:<driver> rather than pretending coverage exists.

Excluded from devloop

This lane talks to real hosted services. It is deliberately not part of devloop's lanes: external variance (GitHub queue depth, network weather) must never poison a changeset verdict.