mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
9f1ffefe83
## Description Companion to #2545 (the "sources" double-count fix) — this fixes the "sinks" half found in the same savings audit: **tool-schema / deferral savings were never aggregated into `Metrics`**. They lived only in per-request log tags, so every sink that reads `metrics.*` silently dropped them, and one CLI mode disagreed with another. Confirmed sinks that under-reported: - **Session-summary printout** — `Tokens saved:` is message-only; a 24K-tool-deferral turn printed `0`. - **`cost.py` session summary** (feeds `/stats.summary`) — `total_tokens_saved_with_rtk` etc. were message+CLI only. - **`/stats` `all_layers_tokens_saved`** — the advertised "total" excluded the `tool_search` layer it enumerates in `by_layer`. - **`headroom perf --format json/csv`** — omitted `tool_saved` while the **text** output of the same command showed it. Closes # ## Type of Change - [x] Bug fix (non-breaking) / observability correctness ## Changes Made - `PrometheusMetrics.tool_search_saved_total` — new counter, accumulated in `record_request` from a new `tool_search_saved` arg; `emit_request_outcome` fills it from the `tool_search_deferred_tokens` + `turn_hook_tools_saved_tokens` tags. **One source of truth.** - Fed into: session summary (`Tool schemas deferred:` line), `cost.py` summary (new `tool_schema_tokens_saved` + `total_tokens_saved_all_layers`; existing fields unchanged for back-compat), `/stats` `all_layers` total, and `build_perf_summary` (`tool_saved`). - Kept **distinct** from `tokens_saved_total` (message compression) — tool bytes never move `tok_before/after`, so it's a separate layer, not a merge (no double-count). ## Testing - [x] `ruff` + `ruff format --check` + `mypy` clean - [x] Regression tests + existing suites pass ### Test Output ```text pytest tests/test_savings_tool_search_aggregation.py tests/test_cli_perf_format.py -q → 18 passed pytest tests/test_cli_perf_format.py test_proxy_savings_history.py test_dashboard_token_savings.py test_bundled_tools_savings.py test_openai_chat_turn_hooks.py → 68 passed, 2 skipped mypy (metrics/outcome/cost/analyzer) → clean ``` ## Real Behavior Proof - Standalone: `record_request(tool_search_saved=1500)` then `(…=800)` → `metrics.tool_search_saved_total == 2300`, `tokens_saved_total == 200` (message stays separate); `build_perf_summary` over records with `tool_saved` 5000+3000 → `tool_saved == 8000`. ## Checklist - [x] Self-reviewed; no new warnings; tests pass; did **not** edit `CHANGELOG.md` ## Additional Notes Together, #2545 (record once) + this (surface every layer) make savings correct **and** complete end-to-end across `/stats`, the dashboard, `headroom perf`, the session summary, and cost/budget. The `/stats` `by_layer.tool_search` and dashboard card already showed the layer (windowed, from the log scan); this makes the lifetime/metrics-based sinks agree.
67 lines
2.1 KiB
Python
67 lines
2.1 KiB
Python
"""Tool-search / deferral savings must aggregate into Metrics and surface in the
|
|
reporting sinks — not live only in per-request tags (which every sink reading
|
|
metrics.* structurally missed: session summary, cost summary, all-layers total,
|
|
`headroom perf --json`)."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
|
|
from headroom.perf.analyzer import PerfRecord, PerfReport, build_perf_summary
|
|
from headroom.proxy.prometheus_metrics import PrometheusMetrics
|
|
|
|
|
|
def test_metrics_accumulates_tool_search_saved_apart_from_message() -> None:
|
|
m = PrometheusMetrics()
|
|
|
|
async def go() -> None:
|
|
await m.record_request(
|
|
provider="anthropic",
|
|
model="claude-x",
|
|
input_tokens=100,
|
|
output_tokens=10,
|
|
tokens_saved=0,
|
|
latency_ms=1.0,
|
|
tool_search_saved=1500,
|
|
)
|
|
await m.record_request(
|
|
provider="anthropic",
|
|
model="claude-x",
|
|
input_tokens=100,
|
|
output_tokens=10,
|
|
tokens_saved=200,
|
|
latency_ms=1.0,
|
|
tool_search_saved=800,
|
|
)
|
|
|
|
asyncio.run(go())
|
|
assert m.tokens_saved_total == 200 # message compression only
|
|
assert m.tool_search_saved_total == 2300 # tool-schema layer, aggregated
|
|
|
|
|
|
def test_build_perf_summary_includes_tool_saved() -> None:
|
|
report = PerfReport(
|
|
perf_records=[
|
|
PerfRecord(
|
|
timestamp="t",
|
|
request_id="r1",
|
|
model="m",
|
|
tokens_before=1000,
|
|
tokens_after=900,
|
|
tokens_saved=100,
|
|
tool_saved=5000,
|
|
),
|
|
PerfRecord(
|
|
timestamp="t",
|
|
request_id="r2",
|
|
model="m",
|
|
tokens_before=500,
|
|
tokens_after=500,
|
|
tokens_saved=0,
|
|
tool_saved=3000,
|
|
),
|
|
]
|
|
)
|
|
summary = build_perf_summary(report)
|
|
assert summary["tokens_saved"] == 100 # message
|
|
assert summary["tool_saved"] == 8000 # tool-schema surfaced in json/csv sink
|