From 81fe9d534579d4dcac197ba901f65d6f19986d32 Mon Sep 17 00:00:00 2001 From: Tejas Chopra Date: Thu, 20 Aug 2026 11:54:02 -0700 Subject: [PATCH] fix(metrics): attribute tool-schema savings per model, not just compression (#3155) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Description Reported against 0.36.0 (VS Code + Copilot + Claude Code): the per-model breakdown disagreed with the headline printed four lines above it. ``` Tokens saved: 625,277 · messages 36,071 · tool schemas 589,206 Per-Model Breakdown : 35,907 tokens saved : 0 tokens saved : 164 tokens saved : 0 tokens saved ``` The rows sum to **36,071** — the *messages* line exactly. All 589,206 tokens of tool-schema deferral, 94% of the headline, had no row to land in, so every tool-heavy model reported "0 tokens saved" while real dollars were credited to it. Deferral is disjoint from message compression by construction: deferred schemas never enter the message token counts, so they move neither `tokens_saved` nor `tokens_sent`. The headline, the PERF line, and the savings ledger (#2795) all already fold the two together. Three per-model surfaces did not. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **`perf/analyzer.py`** — the per-model loop summed `tokens_saved` while its own headline summed `tokens_saved + tool_saved`. Now uses the same all-layers construction (`headline_before = before + tool_saved`), and prints a `· messages / · tool schemas` split line only when there is a split to show. - **`proxy/savings_tracker.py`** — added a `tool_tokens_saved` bucket to `_empty_by_model_entry()`, normalization, and `_record_by_model_locked()`; `record_request()` gained a `tool_search_saved` parameter. `_by_model_snapshot_locked()` ranks and computes `savings_percent` off the combined figure and exposes `headline_tokens_saved`. - **`proxy/prometheus_metrics.py`** — **the seam.** `record_request` already accepted `tool_search_saved` and already folded it into the per-model *dollars*, but never passed it to `savings_tracker.record_request`. Tokens and money therefore disagreed on the same row. - **`proxy/cost.py`** (feeds the dashboard's "Per-Model Token Savings" table) — added `_tool_saved_by_model`, a `tool_schema_saved` kwarg, and `compression_tokens_saved` / `tool_tokens_saved` alongside a combined `tokens_saved`. The `stats()` loop now iterates the **union** of both dicts: keying off compression alone dropped a deferral-only model from the table entirely rather than merely under-reporting it. - **`proxy/outcome.py`** — forwards the figure it already computed for `metrics.record_request` to `cost_tracker.record_tokens`. - **`dashboard.html`** — the "Tokens Saved" cell gains a `title` showing the compression/deferral split. Design notes: - Components stay separately addressable rather than widening an existing field's meaning in place, so persisted state remains readable by older readers. - Percentages use the all-layers numerator over `saved + sent` — deferred schemas were never in `sent`, so that is still the pre-Headroom volume. - `CostTracker.stats()["savings_usd"]` is deliberately **not** widened: deferral is already priced by `SavingsTracker`, and this tracker's dollars feed budget enforcement, where counting it twice would double-book the saving. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ pytest tests/test_per_model_tool_savings.py -q 11 passed in 0.94s # Same file against pre-fix code (git stash), proving the tests bite: 5 failed, 1 passed FAILED test_per_model_rows_reconcile_with_the_headline FAILED test_a_tool_only_model_no_longer_reads_zero FAILED test_tracker_attributes_deferral_to_the_model FAILED test_tracker_default_is_unchanged_without_deferral FAILED test_state_written_before_this_field_existed_still_loads (the one that passes pre-fix is the "compression-only model is unchanged" guard) $ pytest tests/ -q # this branch 3 failed, 11374 passed, 587 skipped in 343.55s $ pytest tests/ -q # clean origin/main, same machine 3 failed, 11364 passed, 587 skipped in 352.64s Identical 3 failures on both — pre-existing and environmental, not regressions: test_learn/test_integration.py::TestCodexIntegration::test_full_pipeline test_release_workflows.py::test_no_native_tls_in_wheel_build_tree (FileNotFoundError: 'cargo') test_graceful_shutdown.py::test_run_server_installs_cancelled_error_filter (whole-suite ordering flake; tests/test_graceful_shutdown.py passes 11/11 in isolation on this branch) $ ruff check headroom/ All checks passed! $ mypy headroom/proxy/cost.py headroom/proxy/savings_tracker.py \ headroom/proxy/prometheus_metrics.py headroom/proxy/outcome.py \ headroom/perf/analyzer.py Success: no issues found in 5 source files ``` ## Real Behavior Proof - Environment: macOS, Python 3.12.13, this branch rebased on `origin/main` @ `1f96dabc`. - Exact command / steps: reproduced the reported shape as a unit test — three models with 35,907 / 0 / 164 message savings and 400,000 / 189,206 / 0 deferral, then rendered `format_report`. - Observed result: headline `Tokens saved: 625,277` unchanged; rows now read `435,907` / `189,206` / `164` and sum to the headline. The seam test drives the real `PrometheusMetrics.record_request` and asserts the tracker's `by_model` entry ends up at `tokens_saved=400, tool_tokens_saved=54,000, headline_tokens_saved=54,400`. - Not tested: no live proxy run against a real Copilot/Claude Code session; the arithmetic is pinned at the four code seams instead. The dashboard `title` tooltip is markup-only and not covered by a rendering test. ## Runtime Rollout Safety - Rollout-managed feature(s): none — this is reporting arithmetic, not a request-path behavior. - Minimum rollout channel: n/a. - Stable/default behavior changed: yes, displayed per-model token savings and percentages increase to include tool-schema deferral. No request is treated differently. - Kill switch / disable path: n/a. Components remain separately readable (`compression_tokens_saved` / `tool_tokens_saved`) if a consumer wants the old message-only figure. - Unsafe override required: none. - Qualification impact: none — `savings_usd` and budget enforcement are unchanged by design. - Rollback path: revert the commit; `tool_tokens_saved` in persisted state is then simply ignored by the older reader. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines Co-authored-by: Tejas Chopra Co-authored-by: Claude Opus 5 --- headroom/dashboard/templates/dashboard.html | 2 +- headroom/perf/analyzer.py | 24 +- headroom/proxy/cost.py | 45 +++- headroom/proxy/outcome.py | 4 + headroom/proxy/prometheus_metrics.py | 8 + headroom/proxy/savings_tracker.py | 38 ++- tests/test_per_model_tool_savings.py | 242 ++++++++++++++++++++ tests/test_request_outcome.py | 5 + 8 files changed, 359 insertions(+), 9 deletions(-) create mode 100644 tests/test_per_model_tool_savings.py diff --git a/headroom/dashboard/templates/dashboard.html b/headroom/dashboard/templates/dashboard.html index 4fa809431..53e943bc8 100644 --- a/headroom/dashboard/templates/dashboard.html +++ b/headroom/dashboard/templates/dashboard.html @@ -850,7 +850,7 @@ - + diff --git a/headroom/perf/analyzer.py b/headroom/perf/analyzer.py index 01a81f072..43d15b25d 100644 --- a/headroom/perf/analyzer.py +++ b/headroom/perf/analyzer.py @@ -579,19 +579,37 @@ def format_report(report: PerfReport) -> str: lines.append("Per-Model Breakdown") lines.append("-" * 40) for model, model_recs in sorted(by_model.items()): + # Same all-layers construction as the headline above. This loop used to + # sum ``tokens_saved`` alone, so every row reported message compression + # only while the headline it sat under counted deferral too. The rows + # then failed to add up to the total printed inches above them — in the + # report that prompted this, four rows summing to 36,071 under a + # headline of 625,277, because 589,206 tokens of tool-schema deferral + # had no row to land in. A tool-heavy model read "0 tokens saved". m_saved = sum(r.tokens_saved for r in model_recs) + m_tool_saved = sum(r.tool_saved for r in model_recs) + m_headline_saved = m_saved + m_tool_saved m_before = sum(r.tokens_before for r in model_recs) - m_pct = (m_saved / m_before * 100) if m_before > 0 else 0 + m_headline_before = m_before + m_tool_saved + m_pct = (m_headline_saved / m_headline_before * 100) if m_headline_before > 0 else 0 list_price = _get_list_price(model) price_str = f"${list_price:.2f}/MTok" if list_price else "unknown" est_str = ( - f" ~${m_saved * list_price / 1_000_000:.2f} at list price" if list_price else "" + f" ~${m_headline_saved * list_price / 1_000_000:.2f} at list price" + if list_price + else "" ) lines.append( f" {model}: {len(model_recs)} reqs, " - f"{m_saved:,} tokens saved ({m_pct:.0f}%), " + f"{m_headline_saved:,} tokens saved ({m_pct:.0f}%), " f"list price {price_str}{est_str}" ) + # Only split the row when there is a split to show; a compression-only + # model keeps the single-line shape it has always had. + if m_tool_saved > 0: + lines.append( + f" · messages {max(0, m_saved):,} · tool schemas {m_tool_saved:,}" + ) lines.append(" * Actual bill savings depend on provider caching behavior") lines.append("") diff --git a/headroom/proxy/cost.py b/headroom/proxy/cost.py index b42ac18b0..5f7dd839e 100644 --- a/headroom/proxy/cost.py +++ b/headroom/proxy/cost.py @@ -706,6 +706,11 @@ class CostTracker: # Token savings per model (exact, no dollar estimation) self._tokens_saved_by_model: dict[str, int] = {} + # Tool-schema deferral per model, DISJOINT from _tokens_saved_by_model + # (deferred schemas are never in the message counts). Tracked separately + # so the compression-only figure stays available; `stats()` reports both + # the split and the sum. + self._tool_saved_by_model: dict[str, int] = {} self._tokens_sent_by_model: dict[str, int] = {} self._requests_by_model: dict[str, int] = {} @@ -721,6 +726,7 @@ class CostTracker: self._costs.clear() self._last_prune_time = datetime.now() self._tokens_saved_by_model.clear() + self._tool_saved_by_model.clear() self._tokens_sent_by_model.clear() self._requests_by_model.clear() self._api_cache_read_by_model.clear() @@ -810,6 +816,7 @@ class CostTracker: uncached_tokens: int = 0, output_tokens: int = 0, cache_inferred: bool = False, + tool_schema_saved: int = 0, ): """Record token counts per model and accumulate request cost for budget enforcement. @@ -827,6 +834,13 @@ class CostTracker: ``uncached_tokens``, so it is excluded from the billed prompt total and from the write premium. Defaults False, which preserves behaviour for providers that report disjoint buckets. + tool_schema_saved: Tokens withheld by tool-schema deferral for this + request. Disjoint from ``tokens_saved`` — deferred schemas never + enter the message token counts, so they moved neither + ``tokens_saved`` nor ``tokens_sent`` and had nowhere to be + attributed. The dashboard's per-model "Tokens Saved" column + therefore showed compression only, while the headline above it + counted both. """ # Post-guard invariant (all providers): Headroom never forwards a request # larger than the original (handlers revert any inflation before sending), @@ -844,6 +858,9 @@ class CostTracker: self._tokens_saved_by_model[model] = ( self._tokens_saved_by_model.get(model, 0) + tokens_saved ) + self._tool_saved_by_model[model] = self._tool_saved_by_model.get(model, 0) + max( + 0, tool_schema_saved + ) self._tokens_sent_by_model[model] = self._tokens_sent_by_model.get(model, 0) + tokens_sent self._requests_by_model[model] = self._requests_by_model.get(model, 0) + 1 self._api_cache_read_by_model[model] = ( @@ -1094,17 +1111,34 @@ class CostTracker: """Get token statistics per model.""" per_model = {} total_saved = 0 - for model in sorted(self._tokens_saved_by_model.keys()): - saved = self._tokens_saved_by_model[model] + total_compression_saved = 0 + total_tool_saved = 0 + # A model may have tool savings and no compression savings at all (every + # turn deferral-only), so iterate the union — keying off + # ``_tokens_saved_by_model`` alone would drop such a model from the table + # entirely rather than merely under-report it. + for model in sorted(set(self._tokens_saved_by_model) | set(self._tool_saved_by_model)): + compression_saved = self._tokens_saved_by_model.get(model, 0) + tool_saved = self._tool_saved_by_model.get(model, 0) + # What the "Tokens Saved" column means: everything Headroom kept off + # the wire for this model. The two components stay addressable beside + # it so a caller can show the split. + saved = compression_saved + tool_saved sent = self._tokens_sent_by_model.get(model, 0) reqs = self._requests_by_model.get(model, 0) total_saved += saved + total_compression_saved += compression_saved + total_tool_saved += tool_saved per_model[model] = { "requests": reqs, "tokens_saved": saved, + "compression_tokens_saved": compression_saved, + "tool_tokens_saved": tool_saved, "tokens_sent": sent, "cache_write_5m_tokens": self._api_cache_write_5m_by_model.get(model, 0), "cache_write_1h_tokens": self._api_cache_write_1h_by_model.get(model, 0), + # Deferred schemas were never in ``sent``, so ``saved + sent`` is + # still the pre-Headroom volume with the wider numerator. "reduction_pct": round(saved / (saved + sent) * 100, 1) if (saved + sent) > 0 else 0, @@ -1153,7 +1187,14 @@ class CostTracker: savings_usd += saved * uncached_price return { + # Sum of the per-model rows above, so the payload reconciles with + # itself. ``savings_usd`` below is deliberately NOT widened: tool + # deferral is already priced by SavingsTracker, and this tracker's + # dollars feed budget enforcement — counting it in both places would + # double-book the saving against a budget. "total_tokens_saved": total_saved, + "total_compression_tokens_saved": total_compression_saved, + "total_tool_tokens_saved": total_tool_saved, "total_input_tokens": total_input_tokens, "total_input_cost_usd": round(cost_with_headroom, 4), "cache_write_5m_tokens": sum(self._api_cache_write_5m_by_model.values()), diff --git a/headroom/proxy/outcome.py b/headroom/proxy/outcome.py index bc594f81e..c2b78b1dc 100644 --- a/headroom/proxy/outcome.py +++ b/headroom/proxy/outcome.py @@ -537,6 +537,10 @@ async def emit_request_outcome(handler: Any, outcome: RequestOutcome) -> None: uncached_tokens=outcome.uncached_input_tokens, cache_inferred=outcome.cache_inferred, output_tokens=outcome.output_tokens, + # Same figure already handed to metrics.record_request above. The + # cost tracker feeds the dashboard's per-model table, which read + # compression only while its own headline counted both layers. + tool_schema_saved=tool_search_saved, ) # 3. Per-request log (optional). The ``client`` outcome field is diff --git a/headroom/proxy/prometheus_metrics.py b/headroom/proxy/prometheus_metrics.py index 503a7ffe2..6bc838b67 100644 --- a/headroom/proxy/prometheus_metrics.py +++ b/headroom/proxy/prometheus_metrics.py @@ -935,6 +935,14 @@ class PrometheusMetrics: model=model, input_tokens=input_tokens, tokens_saved=tokens_saved, + # ``tokens_saved`` is message-only; deferral rides separately and + # the two are disjoint. This argument was the missing link: the + # value arrives at this method (see the parameter above) and is + # already folded into ``savings_usd`` below, but it stopped here, + # so per-model tokens under-reported by exactly the deferral while + # per-model dollars did not — a tool-heavy model showed real money + # saved next to "0 tokens saved". + tool_search_saved=tool_search_saved, provider=provider, project=project, cache_read_tokens=cache_read_tokens, diff --git a/headroom/proxy/savings_tracker.py b/headroom/proxy/savings_tracker.py index 57d55e747..53102d81c 100644 --- a/headroom/proxy/savings_tracker.py +++ b/headroom/proxy/savings_tracker.py @@ -500,7 +500,16 @@ def _empty_display_session() -> dict[str, Any]: def _empty_by_model_entry() -> dict[str, Any]: return { "requests": 0, + # Message-level compression only. Kept as its own bucket rather than + # widened in place, so the persisted meaning of an existing field never + # changes under a reader that predates this. "tokens_saved": 0, + # Tool-schema deferral, attributed to the model that benefited. This was + # dropped on the floor: `record_request` was handed it and passed only + # the message figure down, so a tool-heavy model read "0 tokens saved" + # while the headline counted its deferral. Deferral is routinely the + # larger half -- 589,206 of 625,277 in the report that prompted this. + "tool_tokens_saved": 0, "compression_savings_usd": 0.0, "total_input_tokens": 0, "total_input_cost_usd": 0.0, @@ -562,6 +571,8 @@ def _normalize_by_model(raw: Any) -> dict[str, dict[str, Any]]: normalized = _empty_by_model_entry() normalized["requests"] = _coerce_int(entry.get("requests")) normalized["tokens_saved"] = _coerce_int(entry.get("tokens_saved")) + # Absent in state files written before this field existed -> 0. + normalized["tool_tokens_saved"] = _coerce_int(entry.get("tool_tokens_saved")) normalized["compression_savings_usd"] = round( _coerce_float(entry.get("compression_savings_usd")), 6 ) @@ -741,6 +752,12 @@ class SavingsTracker: model: str, input_tokens: int, tokens_saved: int, + # Tool-schema deferral for this request, DISJOINT from ``tokens_saved`` + # (which is the bare message figure). The caller has always had this + # number and already folds it into the priced dollars below; it simply + # had no parameter to arrive through, so per-model tokens stayed + # message-only while per-model dollars did not. + tool_search_saved: int = 0, output_tokens_saved: int = 0, provider: str | None = None, project: str | None = None, @@ -764,6 +781,7 @@ class SavingsTracker: timestamp_dt = _utc_now() delta_tokens_saved = _coerce_int(tokens_saved) + delta_tool_tokens_saved = max(_coerce_int(tool_search_saved), 0) delta_input_tokens = _coerce_int(input_tokens) delta_output_tokens_saved = max(_coerce_int(output_tokens_saved), 0) delta_cache_read_tokens = _coerce_int(cache_read_tokens) @@ -902,6 +920,7 @@ class SavingsTracker: model, requests_delta=1, tokens_saved_delta=delta_tokens_saved, + tool_tokens_saved_delta=delta_tool_tokens_saved, savings_usd_delta=delta_savings_usd, input_tokens_delta=delta_input_tokens, input_cost_usd_delta=delta_input_cost_usd, @@ -1060,6 +1079,7 @@ class SavingsTracker: *, requests_delta: int = 0, tokens_saved_delta: int = 0, + tool_tokens_saved_delta: int = 0, savings_usd_delta: float = 0.0, input_tokens_delta: int = 0, input_cost_usd_delta: float = 0.0, @@ -1074,6 +1094,7 @@ class SavingsTracker: entry = by_model.setdefault(key, _empty_by_model_entry()) entry["requests"] += max(requests_delta, 0) entry["tokens_saved"] += max(tokens_saved_delta, 0) + entry["tool_tokens_saved"] += max(tool_tokens_saved_delta, 0) entry["compression_savings_usd"] = round( entry["compression_savings_usd"] + max(savings_usd_delta, 0.0), 6 ) @@ -1106,15 +1127,26 @@ class SavingsTracker: by_model = self._state.get("by_model", {}) ranked = sorted( by_model.items(), - key=lambda item: item[1]["tokens_saved"], + key=lambda item: item[1]["tokens_saved"] + item[1].get("tool_tokens_saved", 0), reverse=True, ) result: dict[str, dict[str, Any]] = {} for model, entry in ranked: view = dict(entry) - total_before = entry["tokens_saved"] + entry["total_input_tokens"] + tool_saved = _coerce_int(entry.get("tool_tokens_saved")) + # Deferred tool schemas never reached the model, so they were never in + # ``total_input_tokens`` — the pre-Headroom denominator is the input we + # sent plus what we withheld. Same construction the PERF headline and + # perf/analyzer use, so a row and the total printed above it share one + # definition of "saved". + headline_saved = entry["tokens_saved"] + tool_saved + total_before = headline_saved + entry["total_input_tokens"] + # Named "headline" rather than "total" because this module already uses + # ``total_tokens_saved`` for the cumulative lifetime figure on history + # points; reusing it here would mean two different things in one file. + view["headline_tokens_saved"] = headline_saved view["savings_percent"] = round( - (entry["tokens_saved"] / total_before * 100) if total_before > 0 else 0.0, + (headline_saved / total_before * 100) if total_before > 0 else 0.0, 2, ) result[model] = view diff --git a/tests/test_per_model_tool_savings.py b/tests/test_per_model_tool_savings.py new file mode 100644 index 000000000..63aa99b4b --- /dev/null +++ b/tests/test_per_model_tool_savings.py @@ -0,0 +1,242 @@ +"""Per-model attribution must count tool-schema deferral, not just compression. + +Reported against 0.36.0 with a per-model breakdown reading: + + Tokens saved: 625,277 + · messages 36,071 + · tool schemas 589,206 + Per-Model Breakdown + : ... 35,907 tokens saved + : ... 0 tokens saved + : ... 164 tokens saved + : ... 0 tokens saved + +The rows sum to 36,071 — the *messages* line exactly. All 589,206 tokens of +tool-schema deferral, 94% of the headline, had no row to land in, so the +breakdown contradicted the total printed four lines above it and every +tool-heavy model reported "0 tokens saved". + +Both surfaces had the same shape of bug and both are pinned here: + +* ``perf/analyzer.py`` summed ``tokens_saved`` per model while its own headline + summed ``tokens_saved + tool_saved``. +* ``savings_tracker`` had no per-model field for deferral at all, and + ``prometheus_metrics.record_request`` received the figure but did not pass it + down — while already folding it into the per-model *dollars*, so money and + tokens disagreed on the same row. +""" + +from __future__ import annotations + +from headroom.perf.analyzer import PerfRecord, PerfReport, format_report +from headroom.proxy.savings_tracker import SavingsTracker, _normalize_by_model + + +def _record(model: str, *, before: int, saved: int, tool_saved: int) -> PerfRecord: + return PerfRecord( + timestamp="2026-08-16T00:00:00Z", + request_id=f"req-{model}-{saved}-{tool_saved}", + model=model, + tokens_before=before, + tokens_after=before - saved, + tokens_saved=saved, + tool_saved=tool_saved, + ) + + +# --------------------------------------------------------------------------- # +# CLI report (`headroom perf`) +# --------------------------------------------------------------------------- # +def test_per_model_rows_reconcile_with_the_headline() -> None: + """The reported symptom: rows that do not add up to the total above them.""" + report = PerfReport( + perf_records=[ + _record("model-a", before=100_000, saved=35_907, tool_saved=400_000), + _record("model-b", before=50_000, saved=0, tool_saved=189_206), + _record("model-c", before=20_000, saved=164, tool_saved=0), + ] + ) + + text = format_report(report) + + # Headline is unchanged: 36,071 messages + 589,206 tool schemas. + assert "Tokens saved: 625,277" in text + + # Every model's own tool savings now appear on its row. + assert "model-a: 1 reqs, 435,907 tokens saved" in text + assert "model-b: 1 reqs, 189,206 tokens saved" in text + assert "model-c: 1 reqs, 164 tokens saved" in text + + +def test_a_tool_only_model_no_longer_reads_zero() -> None: + """A model whose entire win is deferral used to render as saving nothing.""" + report = PerfReport( + perf_records=[_record("tool-heavy", before=8_000, saved=0, tool_saved=120_000)] + ) + + text = format_report(report) + + assert "tool-heavy: 1 reqs, 120,000 tokens saved" in text + # Denominator includes what was withheld — deferred schemas were never in + # tokens_before — so the percent is 120,000/128,000, not 120,000/8,000. + assert "(94%)" in text + assert "· messages 0 · tool schemas 120,000" in text + + +def test_a_compression_only_model_keeps_its_single_line_shape() -> None: + report = PerfReport(perf_records=[_record("plain", before=10_000, saved=2_500, tool_saved=0)]) + + text = format_report(report) + + assert "plain: 1 reqs, 2,500 tokens saved (25%)" in text + assert "tool schemas" not in text.split("Per-Model Breakdown")[1] + + +# --------------------------------------------------------------------------- # +# Dashboard / API (`savings_tracker`) +# --------------------------------------------------------------------------- # +def test_tracker_attributes_deferral_to_the_model(tmp_path) -> None: + tracker = SavingsTracker(path=str(tmp_path / "savings.json")) + tracker.record_request( + model="gpt-5-codex", + input_tokens=10_000, + tokens_saved=1_000, + tool_search_saved=90_000, + ) + + entry = tracker.snapshot()["by_model"]["gpt-5-codex"] + + # The two layers stay separately addressable... + assert entry["tokens_saved"] == 1_000 + assert entry["tool_tokens_saved"] == 90_000 + # ...and the combined figure is what the percent is computed from. + assert entry["headline_tokens_saved"] == 91_000 + # 91,000 / (91,000 + 10,000) + assert entry["savings_percent"] == 90.1 + + +def test_tracker_default_is_unchanged_without_deferral(tmp_path) -> None: + """Callers that pass no deferral must see exactly the old numbers.""" + tracker = SavingsTracker(path=str(tmp_path / "savings.json")) + tracker.record_request(model="claude-sonnet-4-6", input_tokens=9_000, tokens_saved=1_000) + + entry = tracker.snapshot()["by_model"]["claude-sonnet-4-6"] + + assert entry["tool_tokens_saved"] == 0 + assert entry["headline_tokens_saved"] == 1_000 + assert entry["savings_percent"] == 10.0 + + +def test_state_written_before_this_field_existed_still_loads() -> None: + """Backward compatibility: the key is simply absent in older state files.""" + normalized = _normalize_by_model( + { + "legacy-model": { + "requests": 3, + "tokens_saved": 500, + "compression_savings_usd": 0.25, + "total_input_tokens": 4_500, + "total_input_cost_usd": 1.5, + } + } + ) + + assert normalized["legacy-model"]["tool_tokens_saved"] == 0 + assert normalized["legacy-model"]["tokens_saved"] == 500 + + +# --------------------------------------------------------------------------- # +# Dashboard "Per-Model Token Savings" table (`cost.py`) +# --------------------------------------------------------------------------- # +def test_cost_tracker_per_model_counts_both_layers() -> None: + from headroom.proxy.cost import CostTracker + + tracker = CostTracker() + tracker.record_tokens("gpt-5-codex", 1_000, 9_000, tool_schema_saved=40_000) + + row = tracker.stats()["per_model"]["gpt-5-codex"] + + assert row["compression_tokens_saved"] == 1_000 + assert row["tool_tokens_saved"] == 40_000 + assert row["tokens_saved"] == 41_000 + # 41,000 / (41,000 + 9,000) + assert row["reduction_pct"] == 82.0 + + +def test_cost_tracker_shows_a_deferral_only_model_at_all() -> None: + """Keying the loop off compression alone dropped such a model entirely.""" + from headroom.proxy.cost import CostTracker + + tracker = CostTracker() + tracker.record_tokens("tool-only", 0, 2_000, tool_schema_saved=18_000) + + stats = tracker.stats() + + assert "tool-only" in stats["per_model"] + assert stats["per_model"]["tool-only"]["tokens_saved"] == 18_000 + + +def test_cost_tracker_totals_reconcile_with_the_rows() -> None: + from headroom.proxy.cost import CostTracker + + tracker = CostTracker() + tracker.record_tokens("model-a", 1_000, 5_000, tool_schema_saved=40_000) + tracker.record_tokens("model-b", 500, 5_000, tool_schema_saved=0) + + stats = tracker.stats() + + assert stats["total_tokens_saved"] == sum( + row["tokens_saved"] for row in stats["per_model"].values() + ) + assert stats["total_compression_tokens_saved"] == 1_500 + assert stats["total_tool_tokens_saved"] == 40_000 + + +def test_cost_tracker_default_call_is_unchanged() -> None: + """Existing callers that pass no deferral keep the old numbers exactly.""" + from headroom.proxy.cost import CostTracker + + tracker = CostTracker() + tracker.record_tokens("claude-sonnet-4-6", 2_500, 7_500) + + row = tracker.stats()["per_model"]["claude-sonnet-4-6"] + + assert row["tokens_saved"] == 2_500 + assert row["tool_tokens_saved"] == 0 + assert row["reduction_pct"] == 25.0 + + +# --------------------------------------------------------------------------- # +# The seam itself +# --------------------------------------------------------------------------- # +def test_metrics_forwards_deferral_to_the_tracker(tmp_path) -> None: + """`record_request` always received the figure; it just never passed it on. + + Pinning this at the seam rather than only at the destination: the tracker + could be correct in isolation and the dashboard still read zero, which is + exactly the state that shipped. + """ + import asyncio + + from headroom.proxy.prometheus_metrics import PrometheusMetrics + + tracker = SavingsTracker(path=str(tmp_path / "savings.json")) + metrics = PrometheusMetrics(savings_tracker=tracker, stateless=True) + + asyncio.run( + metrics.record_request( + provider="openai", + model="gpt-5-codex", + input_tokens=6_000, + output_tokens=100, + tokens_saved=400, + latency_ms=12.0, + tool_search_saved=54_000, + ) + ) + + entry = tracker.snapshot()["by_model"]["gpt-5-codex"] + + assert entry["tokens_saved"] == 400 + assert entry["tool_tokens_saved"] == 54_000 + assert entry["headline_tokens_saved"] == 54_400 diff --git a/tests/test_request_outcome.py b/tests/test_request_outcome.py index d2295a5ea..5fda51fba 100644 --- a/tests/test_request_outcome.py +++ b/tests/test_request_outcome.py @@ -302,6 +302,11 @@ async def test_funnel_passes_canonical_record_tokens_shape() -> None: # as uncached_tokens, so counting it in the billed prompt total would # double it. Defaults False for providers with disjoint buckets. "cache_inferred": False, + # Tool-schema deferral, disjoint from the positional ``tokens_saved``. + # The funnel already computed it for metrics.record_request; forwarding + # it here is what lets the dashboard's per-model table count the layer + # its own headline counts. Zero for this outcome (no deferral tags). + "tool_schema_saved": 0, }