1 commit

Author	SHA1	Message	Date
Tejas Chopra	250ede2f7f	fix(reporting): show net vs gross savings, real skip thresholds, and the effective profile (#3123) ... ## Description Six reporting/config defects found while investigating a user reporting ~1% savings on Claude Code. **None of these changes how much Headroom compresses** — all of them change whether an operator can tell what it did. Every one was found by reading that user's own 227,777 lines of proxy logs against the code. ## Changes Made - **`perf/analyzer`: parse and render `tok_inflated`.** Every PERF line carried it; nothing downstream read it. The report could print `321,239,562 -> 313,274,727` directly above `8,455,763 saved` — two figures that differ by exactly the 490,928 tokens of inflation it omitted. - **`content_router`: report the real skip thresholds.** The routing summary hardcoded `skipped (<50 words)` regardless of what was in force. Wrong number (the message gate is `min_tokens`, 10–250 by profile), wrong unit (tokens and characters, never words), and it merged two different gates under one label. - **`perf/analyzer`: disclose that Transform Effectiveness is partial.** It is built only from `pipeline.py`'s `Transform NAME:` lines. `compression_units.py` / `compression_batches.py` contain zero logging calls, so the table read `content_router: 189,783 saved` against a PERF total 44x larger. Reports the divergence rather than a coverage ratio — the two are different populations and neither contains the other (those lines carry no request_id, fire per stage, and are emitted before the forwarder decides). - **`perf/analyzer`: disclose the routing denominator.** Percentages were taken over 4 of the router's 17 outcome buckets, silently dropping buckets larger than several it displayed. - **`savings_tracker`: stop dropping tool-schema dollars.** `estimate_request_savings_usd` prices four buckets; `record_request` read three. `tool_schema` was computed and discarded, so a quarter of the token headline never reached "Cost saved". The two inputs are disjoint (verified at the call site), so this is additive, not double-counting. - **`agent_savings`: an unknown profile no longer degrades to `balanced`.** `balanced` is a different product posture from the default `coding`: cache→token mode, dedup off, tool-search off, user messages uncompressed, message floor 25x higher, block floor 20x higher. A typo in `HEADROOM_SAVINGS_PROFILE` silently reconfigured the whole proxy. Now degrades to `DEFAULT_PROFILE` and names the resolved profile in the warning. - **`agent_savings`: give `min_chars_for_block` a config-object path.** Every other router pipeline kwarg travels on the config object; this one alone was env-only, so an unseeded proxy applied every sibling `coding` knob while this floor stayed at 500 instead of 25. - **`server`: log the resolved compression posture at startup**, reading cross-turn dedup off the constructed router rather than the environment (the router resolves it as `config OR env`, so reading env alone would be a guess). ## Testing - [x] Unit tests pass, [x] ruff, [x] mypy, [x] new tests added ```text uv run pytest tests/ -k "content_router or agent_savings or perf or analyzer or savings or proxy_server or cli_perf or prometheus" 620 passed, 25 skipped uv run mypy headroom # Success uv run ruff check . && ruff format --check . # clean ``` ## Real behavior proof - **Setup:** macOS arm64, Python 3.12, this branch. Input: 60 MB / 227,777 lines of real proxy logs from the reporting user (6 rotated files, 2,792 PERF lines, 2026-08-17 → 2026-08-19). - **Steps:** pointed `headroom.perf.analyzer.LOG_DIR` at that directory and rendered the report before and after the patch. - **After-fix output (real data, unmodified):** ```text Requests: 2792 Tokens: 321,288,161 -> 313,323,326 (2.6% messages) Tokens saved: 11,158,901 (3.4% reduction) · inflated 490,928 (net message reduction 7,964,835) · messages 8,455,763 · tool schemas 2,703,138 ! stage-level total 190,641 != PERF message total 8,455,763 — this table sees only engines that emit a Transform line, counts per stage, and does not check whether the mutation shipped Skipped: 44641 (77%) — below size floor (shares are of these 4 buckets only, n=58319; see `[router] route_counts=` for the full outcome space) ``` The arithmetic now closes on the page: `8,455,763 - 490,928 = 7,964,835`, matching the token delta exactly. Before the patch none of the three annotated lines existed and the `Skipped` line claimed `<50 words`. - **Profile resolution verified by execution**, not inspection — subprocesses with controlled env: ```text vanilla (nothing set) mode=cache dedupe=1 tool_search=1 min_tokens=10 min_chars=25 HEADROOM_SAVINGS_PROFILE=coding mode=cache dedupe=1 tool_search=1 min_tokens=10 min_chars=25 unknown profile name (before) mode=token dedupe=0 tool_search=0 min_tokens=250 min_chars=500 unknown profile name (after) -> resolves to `coding`, warning names it coding, seeding never runs min_chars=25 (was 500 before this patch) ``` - **Not tested:** live paid Anthropic traffic. These are reporting/config surfaces; the wire path is untouched by this PR. ## Review readiness - [x] Self-reviewed. Three overclaims in my own first draft were corrected before this PR: a false subset claim in the Transform Effectiveness note, a comment asserting `min_chars_for_block` was the *only* env-only field (it is the only env-only *router pipeline kwarg*; `cross_turn_dedup`, `tool_search`, `protect_reads`, `code_aware`, `effort_router`, `lossless` remain env-only via a different mechanism and are **not** fixed here), and a money-path expression that relied on `a + b if c else d` grouping. ## Known remaining (deliberately out of scope) - `Requests: N` still overcounts: the Codex WS forwarder reuses one `request_id` across every turn (one observed 156x), plus ~18 duplicate PERF emissions. - `compression_units.py` / `compression_batches.py` remain unlogged — this PR *discloses* the blind spot rather than closing it. - The headline stays **gross**. True net is `11,158,901 - 490,928 = 10,667,973` (3.3%, not 3.4%). Making net the headline lowers every user's reported savings ~4.4%; that is a product call, not mine, so the inflation is surfaced beside it instead. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>	2026-08-19 00:16:37 -07:00