akadmin/headroom
1
0
Fork 0
mirror of https://github.com/headroomlabs-ai/headroom.git synced 2026-08-27 14:17:10 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions

fix(metrics): attribute tool-schema savings per model, not just compression (#3155)

Browse source 
## Description

Reported against 0.36.0 (VS Code + Copilot + Claude Code): the per-model
breakdown disagreed with the headline printed four lines above it.

```
Tokens saved: 625,277
  · messages       36,071
  · tool schemas  589,206
Per-Model Breakdown
  <a>: 35,907 tokens saved
  <b>:      0 tokens saved
  <c>:    164 tokens saved
  <d>:      0 tokens saved
```

The rows sum to **36,071** — the *messages* line exactly. All 589,206
tokens of tool-schema deferral, 94% of the headline, had no row to land
in, so every tool-heavy model reported "0 tokens saved" while real
dollars were credited to it.

Deferral is disjoint from message compression by construction: deferred
schemas never enter the message token counts, so they move neither
`tokens_saved` nor `tokens_sent`. The headline, the PERF line, and the
savings ledger (#2795) all already fold the two together. Three
per-model surfaces did not.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **`perf/analyzer.py`** — the per-model loop summed `tokens_saved`
while its own headline summed `tokens_saved + tool_saved`. Now uses the
same all-layers construction (`headline_before = before + tool_saved`),
and prints a `· messages / · tool schemas` split line only when there is
a split to show.
- **`proxy/savings_tracker.py`** — added a `tool_tokens_saved` bucket to
`_empty_by_model_entry()`, normalization, and
`_record_by_model_locked()`; `record_request()` gained a
`tool_search_saved` parameter. `_by_model_snapshot_locked()` ranks and
computes `savings_percent` off the combined figure and exposes
`headline_tokens_saved`.
- **`proxy/prometheus_metrics.py`** — **the seam.** `record_request`
already accepted `tool_search_saved` and already folded it into the
per-model *dollars*, but never passed it to
`savings_tracker.record_request`. Tokens and money therefore disagreed
on the same row.
- **`proxy/cost.py`** (feeds the dashboard's "Per-Model Token Savings"
table) — added `_tool_saved_by_model`, a `tool_schema_saved` kwarg, and
`compression_tokens_saved` / `tool_tokens_saved` alongside a combined
`tokens_saved`. The `stats()` loop now iterates the **union** of both
dicts: keying off compression alone dropped a deferral-only model from
the table entirely rather than merely under-reporting it.
- **`proxy/outcome.py`** — forwards the figure it already computed for
`metrics.record_request` to `cost_tracker.record_tokens`.
- **`dashboard.html`** — the "Tokens Saved" cell gains a `title` showing
the compression/deferral split.

Design notes:
- Components stay separately addressable rather than widening an
existing field's meaning in place, so persisted state remains readable
by older readers.
- Percentages use the all-layers numerator over `saved + sent` —
deferred schemas were never in `sent`, so that is still the pre-Headroom
volume.
- `CostTracker.stats()["savings_usd"]` is deliberately **not** widened:
deferral is already priced by `SavingsTracker`, and this tracker's
dollars feed budget enforcement, where counting it twice would
double-book the saving.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ pytest tests/test_per_model_tool_savings.py -q
11 passed in 0.94s

# Same file against pre-fix code (git stash), proving the tests bite:
5 failed, 1 passed
  FAILED test_per_model_rows_reconcile_with_the_headline
  FAILED test_a_tool_only_model_no_longer_reads_zero
  FAILED test_tracker_attributes_deferral_to_the_model
  FAILED test_tracker_default_is_unchanged_without_deferral
  FAILED test_state_written_before_this_field_existed_still_loads
(the one that passes pre-fix is the "compression-only model is unchanged" guard)

$ pytest tests/ -q          # this branch
3 failed, 11374 passed, 587 skipped in 343.55s

$ pytest tests/ -q          # clean origin/main, same machine
3 failed, 11364 passed, 587 skipped in 352.64s

Identical 3 failures on both — pre-existing and environmental, not regressions:
  test_learn/test_integration.py::TestCodexIntegration::test_full_pipeline
  test_release_workflows.py::test_no_native_tls_in_wheel_build_tree   (FileNotFoundError: 'cargo')
  test_graceful_shutdown.py::test_run_server_installs_cancelled_error_filter
    (whole-suite ordering flake; tests/test_graceful_shutdown.py passes 11/11 in isolation on this branch)

$ ruff check headroom/
All checks passed!

$ mypy headroom/proxy/cost.py headroom/proxy/savings_tracker.py \
       headroom/proxy/prometheus_metrics.py headroom/proxy/outcome.py \
       headroom/perf/analyzer.py
Success: no issues found in 5 source files
```

## Real Behavior Proof

- Environment: macOS, Python 3.12.13, this branch rebased on
`origin/main` @ `1f96dabc`.
- Exact command / steps: reproduced the reported shape as a unit test —
three models with 35,907 / 0 / 164 message savings and 400,000 / 189,206
/ 0 deferral, then rendered `format_report`.
- Observed result: headline `Tokens saved: 625,277` unchanged; rows now
read `435,907` / `189,206` / `164` and sum to the headline. The seam
test drives the real `PrometheusMetrics.record_request` and asserts the
tracker's `by_model` entry ends up at `tokens_saved=400,
tool_tokens_saved=54,000, headline_tokens_saved=54,400`.
- Not tested: no live proxy run against a real Copilot/Claude Code
session; the arithmetic is pinned at the four code seams instead. The
dashboard `title` tooltip is markup-only and not covered by a rendering
test.

## Runtime Rollout Safety

- Rollout-managed feature(s): none — this is reporting arithmetic, not a
request-path behavior.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: yes, displayed per-model token
savings and percentages increase to include tool-schema deferral. No
request is treated differently.
- Kill switch / disable path: n/a. Components remain separately readable
(`compression_tokens_saved` / `tool_tokens_saved`) if a consumer wants
the old message-only figure.
- Unsafe override required: none.
- Qualification impact: none — `savings_usd` and budget enforcement are
unchanged by design.
- Rollback path: revert the commit; `tool_tokens_saved` in persisted
state is then simply ignored by the older reader.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines

Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Tejas Chopra 2026-08-20 11:54:02 -07:00 • committed by GitHub
parent bf651c3dc1
commit 81fe9d5345
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
8 changed files with 359 additions and 9 deletions
Download patch file Download diff file Expand all files Collapse all files

2
headroom/dashboard/templates/dashboard.html
View file

@ -850,7 +850,7 @@
<span class="px-2 py-0.5 bg-border rounded text-xs" x-text="truncateModel(model)"></span>
</td>
<td class="px-4 py-3 text-right font-mono tabular-nums" x-text="info.requests"></td>
<td class="px-4 py-3 text-right font-mono tabular-nums text-accent" x-text="formatNumber(info.tokens_saved)"></td>
<td class="px-4 py-3 text-right font-mono tabular-nums text-accent" x-text="formatNumber(info.tokens_saved)" :title="formatNumber(info.compression_tokens_saved || 0) + ' from compression · ' + formatNumber(info.tool_tokens_saved || 0) + ' from deferred tool schemas'"></td>
<td class="px-4 py-3 text-right font-mono tabular-nums" x-text="formatNumber(info.tokens_sent)"></td>
<td class="px-4 py-3 text-right">
<span class="text-accent font-mono tabular-nums" x-text="info.reduction_pct.toFixed(1) + '%'"></span>

24
headroom/perf/analyzer.py
View file

@ -579,19 +579,37 @@ def format_report(report: PerfReport) -> str:
lines.append("Per-Model Breakdown")
lines.append("-" * 40)
for model, model_recs in sorted(by_model.items()):
# Same all-layers construction as the headline above. This loop used to
# sum ``tokens_saved`` alone, so every row reported message compression
# only while the headline it sat under counted deferral too. The rows
# then failed to add up to the total printed inches above them — in the
# report that prompted this, four rows summing to 36,071 under a
# headline of 625,277, because 589,206 tokens of tool-schema deferral
# had no row to land in. A tool-heavy model read "0 tokens saved".
m_saved = sum(r.tokens_saved for r in model_recs)
m_tool_saved = sum(r.tool_saved for r in model_recs)
m_headline_saved = m_saved + m_tool_saved
m_before = sum(r.tokens_before for r in model_recs)
m_pct = (m_saved / m_before * 100) if m_before > 0 else 0
m_headline_before = m_before + m_tool_saved
m_pct = (m_headline_saved / m_headline_before * 100) if m_headline_before > 0 else 0
list_price = _get_list_price(model)
price_str = f"${list_price:.2f}/MTok" if list_price else "unknown"
est_str = (
f" ~${m_saved * list_price / 1_000_000:.2f} at list price" if list_price else ""
f" ~${m_headline_saved * list_price / 1_000_000:.2f} at list price"
if list_price
else ""
)
lines.append(
f" {model}: {len(model_recs)} reqs, "
f"{m_saved:,} tokens saved ({m_pct:.0f}%), "
f"{m_headline_saved:,} tokens saved ({m_pct:.0f}%), "
f"list price {price_str}{est_str}"
)
# Only split the row when there is a split to show; a compression-only
# model keeps the single-line shape it has always had.
if m_tool_saved > 0:
lines.append(
f" · messages {max(0, m_saved):,} · tool schemas {m_tool_saved:,}"
)
lines.append(" * Actual bill savings depend on provider caching behavior")
lines.append("")

45
headroom/proxy/cost.py
View file

@ -706,6 +706,11 @@ class CostTracker:
# Token savings per model (exact, no dollar estimation)
self._tokens_saved_by_model: dict[str, int] = {}
# Tool-schema deferral per model, DISJOINT from _tokens_saved_by_model
# (deferred schemas are never in the message counts). Tracked separately
# so the compression-only figure stays available; `stats()` reports both
# the split and the sum.
self._tool_saved_by_model: dict[str, int] = {}
self._tokens_sent_by_model: dict[str, int] = {}
self._requests_by_model: dict[str, int] = {}
@ -721,6 +726,7 @@ class CostTracker:
self._costs.clear()
self._last_prune_time = datetime.now()
self._tokens_saved_by_model.clear()
self._tool_saved_by_model.clear()
self._tokens_sent_by_model.clear()
self._requests_by_model.clear()
self._api_cache_read_by_model.clear()
@ -810,6 +816,7 @@ class CostTracker:
uncached_tokens: int = 0,
output_tokens: int = 0,
cache_inferred: bool = False,
tool_schema_saved: int = 0,
):
"""Record token counts per model and accumulate request cost for budget enforcement.
@ -827,6 +834,13 @@ class CostTracker:
``uncached_tokens``, so it is excluded from the billed prompt
total and from the write premium. Defaults False, which preserves
behaviour for providers that report disjoint buckets.
tool_schema_saved: Tokens withheld by tool-schema deferral for this
request. Disjoint from ``tokens_saved`` — deferred schemas never
enter the message token counts, so they moved neither
``tokens_saved`` nor ``tokens_sent`` and had nowhere to be
attributed. The dashboard's per-model "Tokens Saved" column
therefore showed compression only, while the headline above it
counted both.
"""
# Post-guard invariant (all providers): Headroom never forwards a request
# larger than the original (handlers revert any inflation before sending),
@ -844,6 +858,9 @@ class CostTracker:
self._tokens_saved_by_model[model] = (
self._tokens_saved_by_model.get(model, 0) + tokens_saved
)
self._tool_saved_by_model[model] = self._tool_saved_by_model.get(model, 0) + max(
0, tool_schema_saved
)
self._tokens_sent_by_model[model] = self._tokens_sent_by_model.get(model, 0) + tokens_sent
self._requests_by_model[model] = self._requests_by_model.get(model, 0) + 1
self._api_cache_read_by_model[model] = (
@ -1094,17 +1111,34 @@ class CostTracker:
"""Get token statistics per model."""
per_model = {}
total_saved = 0
for model in sorted(self._tokens_saved_by_model.keys()):
saved = self._tokens_saved_by_model[model]
total_compression_saved = 0
total_tool_saved = 0
# A model may have tool savings and no compression savings at all (every
# turn deferral-only), so iterate the union — keying off
# ``_tokens_saved_by_model`` alone would drop such a model from the table
# entirely rather than merely under-report it.
for model in sorted(set(self._tokens_saved_by_model) | set(self._tool_saved_by_model)):
compression_saved = self._tokens_saved_by_model.get(model, 0)
tool_saved = self._tool_saved_by_model.get(model, 0)
# What the "Tokens Saved" column means: everything Headroom kept off
# the wire for this model. The two components stay addressable beside
# it so a caller can show the split.
saved = compression_saved + tool_saved
sent = self._tokens_sent_by_model.get(model, 0)
reqs = self._requests_by_model.get(model, 0)
total_saved += saved
total_compression_saved += compression_saved
total_tool_saved += tool_saved
per_model[model] = {
"requests": reqs,
"tokens_saved": saved,
"compression_tokens_saved": compression_saved,
"tool_tokens_saved": tool_saved,
"tokens_sent": sent,
"cache_write_5m_tokens": self._api_cache_write_5m_by_model.get(model, 0),
"cache_write_1h_tokens": self._api_cache_write_1h_by_model.get(model, 0),
# Deferred schemas were never in ``sent``, so ``saved + sent`` is
# still the pre-Headroom volume with the wider numerator.
"reduction_pct": round(saved / (saved + sent) * 100, 1)
if (saved + sent) > 0
else 0,
@ -1153,7 +1187,14 @@ class CostTracker:
savings_usd += saved * uncached_price
return {
# Sum of the per-model rows above, so the payload reconciles with
# itself. ``savings_usd`` below is deliberately NOT widened: tool
# deferral is already priced by SavingsTracker, and this tracker's
# dollars feed budget enforcement — counting it in both places would
# double-book the saving against a budget.
"total_tokens_saved": total_saved,
"total_compression_tokens_saved": total_compression_saved,
"total_tool_tokens_saved": total_tool_saved,
"total_input_tokens": total_input_tokens,
"total_input_cost_usd": round(cost_with_headroom, 4),
"cache_write_5m_tokens": sum(self._api_cache_write_5m_by_model.values()),

4
headroom/proxy/outcome.py
View file

@ -537,6 +537,10 @@ async def emit_request_outcome(handler: Any, outcome: RequestOutcome) -> None:
uncached_tokens=outcome.uncached_input_tokens,
cache_inferred=outcome.cache_inferred,
output_tokens=outcome.output_tokens,
# Same figure already handed to metrics.record_request above. The
# cost tracker feeds the dashboard's per-model table, which read
# compression only while its own headline counted both layers.
tool_schema_saved=tool_search_saved,
)
# 3. Per-request log (optional). The ``client`` outcome field is

8
headroom/proxy/prometheus_metrics.py
View file

@ -935,6 +935,14 @@ class PrometheusMetrics:
model=model,
input_tokens=input_tokens,
tokens_saved=tokens_saved,
# ``tokens_saved`` is message-only; deferral rides separately and
# the two are disjoint. This argument was the missing link: the
# value arrives at this method (see the parameter above) and is
# already folded into ``savings_usd`` below, but it stopped here,
# so per-model tokens under-reported by exactly the deferral while
# per-model dollars did not — a tool-heavy model showed real money
# saved next to "0 tokens saved".
tool_search_saved=tool_search_saved,
provider=provider,
project=project,
cache_read_tokens=cache_read_tokens,

38
headroom/proxy/savings_tracker.py
View file

@ -500,7 +500,16 @@ def _empty_display_session() -> dict[str, Any]:
def _empty_by_model_entry() -> dict[str, Any]:
return {
"requests": 0,
# Message-level compression only. Kept as its own bucket rather than
# widened in place, so the persisted meaning of an existing field never
# changes under a reader that predates this.
"tokens_saved": 0,
# Tool-schema deferral, attributed to the model that benefited. This was
# dropped on the floor: `record_request` was handed it and passed only
# the message figure down, so a tool-heavy model read "0 tokens saved"
# while the headline counted its deferral. Deferral is routinely the
# larger half -- 589,206 of 625,277 in the report that prompted this.
"tool_tokens_saved": 0,
"compression_savings_usd": 0.0,
"total_input_tokens": 0,
"total_input_cost_usd": 0.0,
@ -562,6 +571,8 @@ def _normalize_by_model(raw: Any) -> dict[str, dict[str, Any]]:
normalized = _empty_by_model_entry()
normalized["requests"] = _coerce_int(entry.get("requests"))
normalized["tokens_saved"] = _coerce_int(entry.get("tokens_saved"))
# Absent in state files written before this field existed -> 0.
normalized["tool_tokens_saved"] = _coerce_int(entry.get("tool_tokens_saved"))
normalized["compression_savings_usd"] = round(
_coerce_float(entry.get("compression_savings_usd")), 6
)
@ -741,6 +752,12 @@ class SavingsTracker:
model: str,
input_tokens: int,
tokens_saved: int,
# Tool-schema deferral for this request, DISJOINT from ``tokens_saved``
# (which is the bare message figure). The caller has always had this
# number and already folds it into the priced dollars below; it simply
# had no parameter to arrive through, so per-model tokens stayed
# message-only while per-model dollars did not.
tool_search_saved: int = 0,
output_tokens_saved: int = 0,
provider: str | None = None,
project: str | None = None,
@ -764,6 +781,7 @@ class SavingsTracker:
timestamp_dt = _utc_now()
delta_tokens_saved = _coerce_int(tokens_saved)
delta_tool_tokens_saved = max(_coerce_int(tool_search_saved), 0)
delta_input_tokens = _coerce_int(input_tokens)
delta_output_tokens_saved = max(_coerce_int(output_tokens_saved), 0)
delta_cache_read_tokens = _coerce_int(cache_read_tokens)
@ -902,6 +920,7 @@ class SavingsTracker:
model,
requests_delta=1,
tokens_saved_delta=delta_tokens_saved,
tool_tokens_saved_delta=delta_tool_tokens_saved,
savings_usd_delta=delta_savings_usd,
input_tokens_delta=delta_input_tokens,
input_cost_usd_delta=delta_input_cost_usd,
@ -1060,6 +1079,7 @@ class SavingsTracker:
*,
requests_delta: int = 0,
tokens_saved_delta: int = 0,
tool_tokens_saved_delta: int = 0,
savings_usd_delta: float = 0.0,
input_tokens_delta: int = 0,
input_cost_usd_delta: float = 0.0,
@ -1074,6 +1094,7 @@ class SavingsTracker:
entry = by_model.setdefault(key, _empty_by_model_entry())
entry["requests"] += max(requests_delta, 0)
entry["tokens_saved"] += max(tokens_saved_delta, 0)
entry["tool_tokens_saved"] += max(tool_tokens_saved_delta, 0)
entry["compression_savings_usd"] = round(
entry["compression_savings_usd"] + max(savings_usd_delta, 0.0), 6
)
@ -1106,15 +1127,26 @@ class SavingsTracker:
by_model = self._state.get("by_model", {})
ranked = sorted(
by_model.items(),
key=lambda item: item[1]["tokens_saved"],
key=lambda item: item[1]["tokens_saved"] + item[1].get("tool_tokens_saved", 0),
reverse=True,
)
result: dict[str, dict[str, Any]] = {}
for model, entry in ranked:
view = dict(entry)
total_before = entry["tokens_saved"] + entry["total_input_tokens"]
tool_saved = _coerce_int(entry.get("tool_tokens_saved"))
# Deferred tool schemas never reached the model, so they were never in
# ``total_input_tokens`` — the pre-Headroom denominator is the input we
# sent plus what we withheld. Same construction the PERF headline and
# perf/analyzer use, so a row and the total printed above it share one
# definition of "saved".
headline_saved = entry["tokens_saved"] + tool_saved
total_before = headline_saved + entry["total_input_tokens"]
# Named "headline" rather than "total" because this module already uses
# ``total_tokens_saved`` for the cumulative lifetime figure on history
# points; reusing it here would mean two different things in one file.
view["headline_tokens_saved"] = headline_saved
view["savings_percent"] = round(
(entry["tokens_saved"] / total_before * 100) if total_before > 0 else 0.0,
(headline_saved / total_before * 100) if total_before > 0 else 0.0,
2,
)
result[model] = view

242
tests/test_per_model_tool_savings.py Normal file
View file

@ -0,0 +1,242 @@
"""Per-model attribution must count tool-schema deferral, not just compression.
Reported against 0.36.0 with a per-model breakdown reading:
Tokens saved: 625,277
· messages 36,071
· tool schemas 589,206
Per-Model Breakdown
<model-a>: ... 35,907 tokens saved
<model-b>: ... 0 tokens saved
<model-c>: ... 164 tokens saved
<model-d>: ... 0 tokens saved
The rows sum to 36,071 — the *messages* line exactly. All 589,206 tokens of
tool-schema deferral, 94% of the headline, had no row to land in, so the
breakdown contradicted the total printed four lines above it and every
tool-heavy model reported "0 tokens saved".
Both surfaces had the same shape of bug and both are pinned here:
* ``perf/analyzer.py`` summed ``tokens_saved`` per model while its own headline
summed ``tokens_saved + tool_saved``.
* ``savings_tracker`` had no per-model field for deferral at all, and
``prometheus_metrics.record_request`` received the figure but did not pass it
down — while already folding it into the per-model *dollars*, so money and
tokens disagreed on the same row.
"""
from __future__ import annotations
from headroom.perf.analyzer import PerfRecord, PerfReport, format_report
from headroom.proxy.savings_tracker import SavingsTracker, _normalize_by_model
def _record(model: str, *, before: int, saved: int, tool_saved: int) -> PerfRecord:
return PerfRecord(
timestamp="2026-08-16T00:00:00Z",
request_id=f"req-{model}-{saved}-{tool_saved}",
model=model,
tokens_before=before,
tokens_after=before - saved,
tokens_saved=saved,
tool_saved=tool_saved,
)
# --------------------------------------------------------------------------- #
# CLI report (`headroom perf`)
# --------------------------------------------------------------------------- #
def test_per_model_rows_reconcile_with_the_headline() -> None:
"""The reported symptom: rows that do not add up to the total above them."""
report = PerfReport(
perf_records=[
_record("model-a", before=100_000, saved=35_907, tool_saved=400_000),
_record("model-b", before=50_000, saved=0, tool_saved=189_206),
_record("model-c", before=20_000, saved=164, tool_saved=0),
]
)
text = format_report(report)
# Headline is unchanged: 36,071 messages + 589,206 tool schemas.
assert "Tokens saved: 625,277" in text
# Every model's own tool savings now appear on its row.
assert "model-a: 1 reqs, 435,907 tokens saved" in text
assert "model-b: 1 reqs, 189,206 tokens saved" in text
assert "model-c: 1 reqs, 164 tokens saved" in text
def test_a_tool_only_model_no_longer_reads_zero() -> None:
"""A model whose entire win is deferral used to render as saving nothing."""
report = PerfReport(
perf_records=[_record("tool-heavy", before=8_000, saved=0, tool_saved=120_000)]
)
text = format_report(report)
assert "tool-heavy: 1 reqs, 120,000 tokens saved" in text
# Denominator includes what was withheld — deferred schemas were never in
# tokens_before — so the percent is 120,000/128,000, not 120,000/8,000.
assert "(94%)" in text
assert "· messages 0 · tool schemas 120,000" in text
def test_a_compression_only_model_keeps_its_single_line_shape() -> None:
report = PerfReport(perf_records=[_record("plain", before=10_000, saved=2_500, tool_saved=0)])
text = format_report(report)
assert "plain: 1 reqs, 2,500 tokens saved (25%)" in text
assert "tool schemas" not in text.split("Per-Model Breakdown")[1]
# --------------------------------------------------------------------------- #
# Dashboard / API (`savings_tracker`)
# --------------------------------------------------------------------------- #
def test_tracker_attributes_deferral_to_the_model(tmp_path) -> None:
tracker = SavingsTracker(path=str(tmp_path / "savings.json"))
tracker.record_request(
model="gpt-5-codex",
input_tokens=10_000,
tokens_saved=1_000,
tool_search_saved=90_000,
)
entry = tracker.snapshot()["by_model"]["gpt-5-codex"]
# The two layers stay separately addressable...
assert entry["tokens_saved"] == 1_000
assert entry["tool_tokens_saved"] == 90_000
# ...and the combined figure is what the percent is computed from.
assert entry["headline_tokens_saved"] == 91_000
# 91,000 / (91,000 + 10,000)
assert entry["savings_percent"] == 90.1
def test_tracker_default_is_unchanged_without_deferral(tmp_path) -> None:
"""Callers that pass no deferral must see exactly the old numbers."""
tracker = SavingsTracker(path=str(tmp_path / "savings.json"))
tracker.record_request(model="claude-sonnet-4-6", input_tokens=9_000, tokens_saved=1_000)
entry = tracker.snapshot()["by_model"]["claude-sonnet-4-6"]
assert entry["tool_tokens_saved"] == 0
assert entry["headline_tokens_saved"] == 1_000
assert entry["savings_percent"] == 10.0
def test_state_written_before_this_field_existed_still_loads() -> None:
"""Backward compatibility: the key is simply absent in older state files."""
normalized = _normalize_by_model(
{
"legacy-model": {
"requests": 3,
"tokens_saved": 500,
"compression_savings_usd": 0.25,
"total_input_tokens": 4_500,
"total_input_cost_usd": 1.5,
}
}
)
assert normalized["legacy-model"]["tool_tokens_saved"] == 0
assert normalized["legacy-model"]["tokens_saved"] == 500
# --------------------------------------------------------------------------- #
# Dashboard "Per-Model Token Savings" table (`cost.py`)
# --------------------------------------------------------------------------- #
def test_cost_tracker_per_model_counts_both_layers() -> None:
from headroom.proxy.cost import CostTracker
tracker = CostTracker()
tracker.record_tokens("gpt-5-codex", 1_000, 9_000, tool_schema_saved=40_000)
row = tracker.stats()["per_model"]["gpt-5-codex"]
assert row["compression_tokens_saved"] == 1_000
assert row["tool_tokens_saved"] == 40_000
assert row["tokens_saved"] == 41_000
# 41,000 / (41,000 + 9,000)
assert row["reduction_pct"] == 82.0
def test_cost_tracker_shows_a_deferral_only_model_at_all() -> None:
"""Keying the loop off compression alone dropped such a model entirely."""
from headroom.proxy.cost import CostTracker
tracker = CostTracker()
tracker.record_tokens("tool-only", 0, 2_000, tool_schema_saved=18_000)
stats = tracker.stats()
assert "tool-only" in stats["per_model"]
assert stats["per_model"]["tool-only"]["tokens_saved"] == 18_000
def test_cost_tracker_totals_reconcile_with_the_rows() -> None:
from headroom.proxy.cost import CostTracker
tracker = CostTracker()
tracker.record_tokens("model-a", 1_000, 5_000, tool_schema_saved=40_000)
tracker.record_tokens("model-b", 500, 5_000, tool_schema_saved=0)
stats = tracker.stats()
assert stats["total_tokens_saved"] == sum(
row["tokens_saved"] for row in stats["per_model"].values()
)
assert stats["total_compression_tokens_saved"] == 1_500
assert stats["total_tool_tokens_saved"] == 40_000
def test_cost_tracker_default_call_is_unchanged() -> None:
"""Existing callers that pass no deferral keep the old numbers exactly."""
from headroom.proxy.cost import CostTracker
tracker = CostTracker()
tracker.record_tokens("claude-sonnet-4-6", 2_500, 7_500)
row = tracker.stats()["per_model"]["claude-sonnet-4-6"]
assert row["tokens_saved"] == 2_500
assert row["tool_tokens_saved"] == 0
assert row["reduction_pct"] == 25.0
# --------------------------------------------------------------------------- #
# The seam itself
# --------------------------------------------------------------------------- #
def test_metrics_forwards_deferral_to_the_tracker(tmp_path) -> None:
"""`record_request` always received the figure; it just never passed it on.
Pinning this at the seam rather than only at the destination: the tracker
could be correct in isolation and the dashboard still read zero, which is
exactly the state that shipped.
"""
import asyncio
from headroom.proxy.prometheus_metrics import PrometheusMetrics
tracker = SavingsTracker(path=str(tmp_path / "savings.json"))
metrics = PrometheusMetrics(savings_tracker=tracker, stateless=True)
asyncio.run(
metrics.record_request(
provider="openai",
model="gpt-5-codex",
input_tokens=6_000,
output_tokens=100,
tokens_saved=400,
latency_ms=12.0,
tool_search_saved=54_000,
)
)
entry = tracker.snapshot()["by_model"]["gpt-5-codex"]
assert entry["tokens_saved"] == 400
assert entry["tool_tokens_saved"] == 54_000
assert entry["headline_tokens_saved"] == 54_400

5
tests/test_request_outcome.py
View file

@ -302,6 +302,11 @@ async def test_funnel_passes_canonical_record_tokens_shape() -> None:
# as uncached_tokens, so counting it in the billed prompt total would
# double it. Defaults False for providers with disjoint buckets.
"cache_inferred": False,
# Tool-schema deferral, disjoint from the positional ``tokens_saved``.
# The funnel already computed it for metrics.record_request; forwarding
# it here is what lets the dashboard's per-model table count the layer
# its own headline counts. Zero for this outcome (no deferral tags).
"tool_schema_saved": 0,
}