Token Calibration Factors (2026-06-11)

# Token Calibration Factors (2026-06-11)

Measured with `scripts/token_calibration.py` against the exact `o200k_base`

tokenizer (tiktoken), on per-style command outputs captured from one identical

fixture (40 files, 12 commits, 9 dirty + 1 new file) snapshotted by both

subjects. Factor = exact tokens / (chars ÷ 4); **factor > 1.0 means char/4

underestimates real tokens for that style**.

| Output style | Command | Factor | Sample bytes |

| --- | --- | ---: | ---: |

| `git_status` | `git status` | 1.216 | 589 |

| `git_status_short` | `git status --short` | 1.605 | 172 |

| `git_diff` | `git diff` | 1.727 | 1611 |

| `git_diff_stat` | `git diff --stat` | 1.643 | 223 |

| `git_log` | `git log --oneline -12` | 2.465 | 171 |

| `git_log_full` | `git log -5` | 1.863 | 697 |

| `oak_status` | `oak status` | 1.645 | 123 |

| `oak_status_json` | `oak status --json` | 1.176 | 704 |

| `oak_diff` | `oak diff --print` | 1.656 | 1382 |

| `oak_diff_stat` | `oak diff --stat --print` | 1.969 | 257 |

| `oak_log` | `oak log -n 12` | 2.107 | 705 |

| `oak_log_full` | `oak log -n 5 -v` | 1.962 | 418 |

## What this means for published deltas

- **Same-family agent-default pairs are close enough to compare raw** where

  both factors agree within ~5% (status: git `--short` 1.605 vs oak 1.645;

  diff: 1.727 vs 1.656). The char/4 *delta* direction holds there.

- **Log and stat families are not safe raw**: `git log --oneline` (2.465) vs

  `oak log` (2.107) is a 15-point bias gap favoring git in char/4 terms;

  `diff --stat` flips it (git 1.643 vs oak 1.969, ~20 points favoring oak).

  Apply the factors (or report exact-tokenizer counts) before publishing any

  cross-subject token claim involving log or stat output.

- Hash density drives the bias (hex tokenizes at ~2.4 chars/token), so any

  output style change in either tool invalidates its row here — recapture

  rather than reuse.

## Caveats

- n=1 sample per style on a small fixture (123–1611 bytes). Before a public

  drop, recapture from real result artifacts at scale (`--per-file` over

  workflow outputs) — these factors are working values, not final.

- Tool-call envelope constants in `oakbench/tokens.py` were updated the same

  day from the same tokenizer: per-call emitted 45→58, ingested 27→44

  (anthropic templates 61/45, openai 56/43). Historical rows used the old

  constants; `token_estimate.cost_weights`/`ENVELOPE_NOTE` on each row

  records which constants priced it.

Regenerate: capture outputs per style, then

`python3 scripts/token_calibration.py <files> --label <style> --compact`.