fix(reporting): show net vs gross savings, real skip thresholds, and the effective profile (#3123)

## Description

Six reporting/config defects found while investigating a user reporting
~1% savings on Claude Code. **None of these changes how much Headroom
compresses** — all of them change whether an operator can tell what it
did. Every one was found by reading that user's own 227,777 lines of
proxy logs against the code.

## Changes Made

- **`perf/analyzer`: parse and render `tok_inflated`.** Every PERF line
carried it; nothing downstream read it. The report could print
`321,239,562 -> 313,274,727` directly above `8,455,763 saved` — two
figures that differ by exactly the 490,928 tokens of inflation it
omitted.
- **`content_router`: report the real skip thresholds.** The routing
summary hardcoded `skipped (<50 words)` regardless of what was in force.
Wrong number (the message gate is `min_tokens`, 10–250 by profile),
wrong unit (tokens and characters, never words), and it merged two
different gates under one label.
- **`perf/analyzer`: disclose that Transform Effectiveness is partial.**
It is built only from `pipeline.py`'s `Transform NAME:` lines.
`compression_units.py` / `compression_batches.py` contain zero logging
calls, so the table read `content_router: 189,783 saved` against a PERF
total 44x larger. Reports the divergence rather than a coverage ratio —
the two are different populations and neither contains the other (those
lines carry no request_id, fire per stage, and are emitted before the
forwarder decides).
- **`perf/analyzer`: disclose the routing denominator.** Percentages
were taken over 4 of the router's 17 outcome buckets, silently dropping
buckets larger than several it displayed.
- **`savings_tracker`: stop dropping tool-schema dollars.**
`estimate_request_savings_usd` prices four buckets; `record_request`
read three. `tool_schema` was computed and discarded, so a quarter of
the token headline never reached "Cost saved". The two inputs are
disjoint (verified at the call site), so this is additive, not
double-counting.
- **`agent_savings`: an unknown profile no longer degrades to
`balanced`.** `balanced` is a different product posture from the default
`coding`: cache→token mode, dedup off, tool-search off, user messages
uncompressed, message floor 25x higher, block floor 20x higher. A typo
in `HEADROOM_SAVINGS_PROFILE` silently reconfigured the whole proxy. Now
degrades to `DEFAULT_PROFILE` and names the resolved profile in the
warning.
- **`agent_savings`: give `min_chars_for_block` a config-object path.**
Every other router pipeline kwarg travels on the config object; this one
alone was env-only, so an unseeded proxy applied every sibling `coding`
knob while this floor stayed at 500 instead of 25.
- **`server`: log the resolved compression posture at startup**, reading
cross-turn dedup off the constructed router rather than the environment
(the router resolves it as `config OR env`, so reading env alone would
be a guess).

## Testing

- [x] Unit tests pass, [x] ruff, [x] mypy, [x] new tests added

```text
uv run pytest tests/ -k "content_router or agent_savings or perf or analyzer or savings or proxy_server or cli_perf or prometheus"
620 passed, 25 skipped
uv run mypy headroom  # Success
uv run ruff check . && ruff format --check .  # clean
```

## Real behavior proof

- **Setup:** macOS arm64, Python 3.12, this branch. Input: 60 MB /
227,777 lines of real proxy logs from the reporting user (6 rotated
files, 2,792 PERF lines, 2026-08-17 → 2026-08-19).
- **Steps:** pointed `headroom.perf.analyzer.LOG_DIR` at that directory
and rendered the report before and after the patch.
- **After-fix output (real data, unmodified):**

```text
Requests:     2792
Tokens:       321,288,161 -> 313,323,326 (2.6% messages)
Tokens saved: 11,158,901 (3.4% reduction)
  · inflated      490,928 (net message reduction 7,964,835)
  · messages       8,455,763
  · tool schemas   2,703,138
  ! stage-level total 190,641 != PERF message total 8,455,763 — this table sees only
    engines that emit a Transform line, counts per stage, and does not check whether
    the mutation shipped
  Skipped:     44641 (77%) — below size floor
  (shares are of these 4 buckets only, n=58319; see `[router] route_counts=` for the
   full outcome space)
```

The arithmetic now closes on the page: `8,455,763 - 490,928 =
7,964,835`, matching the token delta exactly. Before the patch none of
the three annotated lines existed and the `Skipped` line claimed `<50
words`.

- **Profile resolution verified by execution**, not inspection —
subprocesses with controlled env:

```text
vanilla (nothing set)          mode=cache dedupe=1 tool_search=1 min_tokens=10  min_chars=25
HEADROOM_SAVINGS_PROFILE=coding  mode=cache dedupe=1 tool_search=1 min_tokens=10  min_chars=25
unknown profile name (before)  mode=token dedupe=0 tool_search=0 min_tokens=250 min_chars=500
unknown profile name (after)   -> resolves to `coding`, warning names it
coding, seeding never runs     min_chars=25 (was 500 before this patch)
```

- **Not tested:** live paid Anthropic traffic. These are
reporting/config surfaces; the wire path is untouched by this PR.

## Review readiness

- [x] Self-reviewed. Three overclaims in my own first draft were
corrected before this PR: a false subset claim in the Transform
Effectiveness note, a comment asserting `min_chars_for_block` was the
*only* env-only field (it is the only env-only *router pipeline kwarg*;
`cross_turn_dedup`, `tool_search`, `protect_reads`, `code_aware`,
`effort_router`, `lossless` remain env-only via a different mechanism
and are **not** fixed here), and a money-path expression that relied on
`a + b if c else d` grouping.

## Known remaining (deliberately out of scope)

- `Requests: N` still overcounts: the Codex WS forwarder reuses one
`request_id` across every turn (one observed 156x), plus ~18 duplicate
PERF emissions.
- `compression_units.py` / `compression_batches.py` remain unlogged —
this PR *discloses* the blind spot rather than closing it.
- The headline stays **gross**. True net is `11,158,901 - 490,928 =
10,667,973` (3.3%, not 3.4%). Making net the headline lowers every
user's reported savings ~4.4%; that is a product call, not mine, so the
inflation is surfaced beside it instead.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Tejas Chopra 2026-08-19 00:16:37 -07:00 committed by GitHub
parent 05f5ef47cb
commit 250ede2f7f
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
7 changed files with 458 additions and 21 deletions

View file

@ -0,0 +1,223 @@
# Headroom Metrics — Dashboard Guide
What each metric shows, so you can build panels against it.
**Two endpoints.** Both are on the proxy (default `:8787`).
| Surface | How to get it | Use it for |
|---|---|---|
| **Prometheus**`GET /metrics` | Always on, no config | Everything below. Start here. |
| **OpenTelemetry** — OTLP/HTTP | `HEADROOM_OTEL_METRICS_ENABLED=1` + `pip install "headroom-ai[proxy,otel]"` | Same data, dotted names, plus per-tenant labels |
Names differ between them: Prometheus uses `headroom_tokens_saved_total` (**milliseconds** for timings), OTel uses `headroom.proxy.tokens.saved` (**seconds**). Both are listed below.
---
## The savings panel — start here
**`headroom.proxy.tokens.saved`** is the headline number. It already combines compression + tool-schema deferral — no need to add anything to it.
| Metric | What it shows |
|---|---|
| **`headroom.proxy.tokens.saved`** *(OTel)* | **Total input tokens Headroom kept out of the request.** Compression + tool savings, combined. This is your hero number. |
| `headroom.proxy.savings.usd{source}` *(OTel)* | **Dollars saved**, split by layer: `compression`, `tool_schema`, `output_shaping`, `provider_cache`. Sum for the total. |
| `headroom_persistent_savings_tokens_saved_total` | Same tokens-saved number, but **survives proxy restarts**. Use for "lifetime saved" tiles. |
| `headroom_persistent_savings_compression_savings_usd_total` | **Lifetime dollars saved**, durable across restarts. |
| `headroom_tokens_input_total` | Input tokens actually sent upstream (post-compression). The denominator for a reduction %. |
| `headroom_tokens_output_total` | Output tokens returned by the provider. |
```promql
# Hero tile: tokens saved per second
rate(headroom_tokens_saved_total[5m])
+ sum(rate(headroom_savings_attributed_tokens_total{source="tool_search",realized="true"}[5m]))
# Context reduction %
100 * rate(headroom_tokens_saved_total[5m])
/ clamp_min(rate(headroom_tokens_input_total[5m]) + rate(headroom_tokens_saved_total[5m]), 1)
# Lifetime tiles (survive restart)
headroom_persistent_savings_tokens_saved_total
headroom_persistent_savings_compression_savings_usd_total
```
> **One catch on the Prometheus side.** `headroom_tokens_saved_total` is compression **only** — it leaves out tool-schema deferral. The OTel `headroom.proxy.tokens.saved` includes both. That's why the query above adds the `tool_search` term back in. On tool-heavy workloads the gap is large.
---
## Latency panel
All Prometheus timings are in **milliseconds**, exposed as `_sum` / `_count` / `_min` / `_max`. Build means with `rate(sum)/rate(count)`.
| Metric | What it shows |
|---|---|
| **`headroom_overhead_ms_*`** | **Latency Headroom itself adds.** Handler entry → end of compression. Excludes the LLM call. This is the "what does this cost us" number. |
| `headroom_latency_ms_*` | Total request duration, including the provider. |
| `headroom_ttfb_ms_*` | Time to first byte from upstream. Streaming requests only. |
| `headroom_stage_timing_ms_*{path,stage}` | Where time went inside the handler — `compression_first_stage`, `upstream_connect`, `memory_context`, etc. |
| `headroom_transform_timing_ms_*{transform}` | Time per compression transform. Use to find a slow transform. |
```promql
# Headroom's added overhead, mean ms
rate(headroom_overhead_ms_sum[5m]) / rate(headroom_overhead_ms_count[5m])
# End-to-end, mean ms
rate(headroom_latency_ms_sum[5m]) / rate(headroom_latency_ms_count[5m])
# Slowest stages
topk(5, rate(headroom_stage_timing_ms_sum[5m]) / rate(headroom_stage_timing_ms_count[5m]))
```
> **No percentiles are available.** There are no histogram buckets on `/metrics`, and the OTel histograms ship with default buckets that put every request into one bucket, so `histogram_quantile()` returns nonsense. **Means work fine.** For real p95/p99 today, use the `headroom perf` CLI.
>
> Also: divide each `_sum` by **its own** `_count`. Overhead and TTFB are only sampled when > 0, so their counts are smaller than the latency count.
---
## Cache panel
| Metric | What it shows |
|---|---|
| `headroom_provider_cache_hit_requests_total{provider}` | Requests that read from the provider's prompt cache. |
| `headroom_provider_cache_requests_total{provider}` | Requests with any cache activity. **The correct denominator for hit rate.** |
| `headroom_cache_read_tokens_total{provider}` | Tokens served from cache (the discounted ones). |
| `headroom_cache_write_tokens_total{provider}` | Tokens written into cache (these carry a premium). |
| `headroom_cache_write_ttl_tokens_total{provider,ttl}` | Cache writes split by TTL — `5m` vs `1h`. |
| `headroom_uncached_input_tokens_total{provider}` | Input tokens that missed cache entirely. |
| `headroom_cache_bust_total` | Requests where compression broke a cached prefix. **Should stay near zero.** |
| `headroom_cache_miss_attribution_total{provider,reason}` | Why a cached prefix missed — `ttl_expiry`, `prefix_change`, `unknown`. |
```promql
# Cache hit rate by provider
sum by (provider) (rate(headroom_provider_cache_hit_requests_total[5m]))
/ sum by (provider) (rate(headroom_provider_cache_requests_total[5m]))
# Compression breaking cache — alert if this rises
rate(headroom_cache_bust_total[5m])
```
> **Don't use `headroom_requests_cached_total` as a hit rate.** It mixes the provider's prompt cache with Headroom's own response cache into one boolean, so it measures neither.
---
## Traffic & health panel
| Metric | What it shows |
|---|---|
| `headroom_requests_total` | Requests handled. Unlabelled. |
| `headroom_requests_by_provider{provider}` | Traffic split by provider — `anthropic`, `openai`, `gemini`, `bedrock`… |
| `headroom_requests_by_model{model}` | Traffic split by model. Capped at 1024 distinct; overflow lands in `model="other"`. |
| `headroom_requests_failed_total` | Upstream 5xx errors. |
| `headroom_requests_rate_limited_total` | Requests **Headroom** rejected via its own rate limiter (not upstream 429s). |
| `headroom_compression_failed_total{reason}` | Compression failures — `timeout` or `error`. Fails open, so traffic keeps flowing but savings quietly stop. **Worth an alert.** |
| `headroom_compression_quarantine_total{event}` | Compression disabled after repeated timeouts — `activated`, `skipped`, `released`. |
| `headroom_inbound_requests_active` | In-flight requests, gauge. Counts all HTTP including `/metrics`. |
| `headroom_active_ws_sessions` | Live Codex WebSocket sessions, gauge. |
```promql
# Failure rate
rate(headroom_requests_failed_total[5m])
/ clamp_min(rate(headroom_requests_total[5m]) + rate(headroom_requests_failed_total[5m]), 1)
# Savings silently stopped
sum by (reason) (rate(headroom_compression_failed_total[5m]))
# Traffic mix
sum by (provider) (rate(headroom_requests_by_provider[5m]))
```
---
## Anthropic subscription panel
Only if you're on an Anthropic OAuth/subscription plan. OTel only, gauges, no labels.
| Metric | What it shows |
|---|---|
| `headroom.subscription.5h_utilization_pct` | How much of the 5-hour rate-limit window is used (0100). |
| `headroom.subscription.7d_utilization_pct` | Same for the 7-day window. |
| `headroom.subscription.5h_seconds_to_reset` | Seconds until the 5-hour window resets. |
| `headroom.subscription.7d_seconds_to_reset` | Seconds until the 7-day window resets. |
| `headroom.subscription.overage_usd` | Extra-usage credits consumed, in dollars. |
---
## Attribution — where savings came from
| Metric | What it shows |
|---|---|
| `headroom_savings_attributed_tokens_total{source,realized}` | Tokens saved, broken out by named source. `source="tool_search"` is tool-schema deferral. |
| `headroom_savings_attributed_usd_total{source,realized}` | Dollars saved by source. **Gauge, can go negative** — don't `rate()` it. |
| `headroom_savings_attribution_events_total{source,realized}` | How often each source contributed. |
| `headroom_waste_signal_tokens_total{signal}` | Wasteful patterns *detected* in the input — `json_bloat`, `base64`, `repetition`, `reread`… This is diagnosis, **not savings**. |
These rows *explain* the headline total — they are never added to it.
---
## Compression internals
| Metric | What it shows |
|---|---|
| `headroom.compression.tokens.input` *(OTel)* | Tokens going into the compression pipeline. |
| `headroom.compression.tokens.output` *(OTel)* | Tokens coming out. |
| `headroom.compression.tokens.saved` *(OTel)* | The difference. Pipeline-level view of compression only. |
| `headroom.compression.runs` *(OTel)* | Pipeline executions. Note: **per pipeline run, not per request.** |
| `headroom.compression.pipeline.duration` *(OTel, seconds)* | How long the pipeline took. |
| `headroom.compression.transforms{transform}` *(OTel)* | Which transforms fired. **High cardinality — drop or aggregate at the collector.** |
---
## Five things that will break a dashboard
1. **Only savings counters survive a restart.** 55 of 60 Prometheus families reset to zero when the proxy restarts. Only `headroom_persistent_savings_*` is durable, and it needs `HEADROOM_WORKSPACE_DIR` on a persistent volume — otherwise it resets on every deploy.
2. **No percentiles anywhere.** Use means. See the latency section.
3. **`headroom_latency_ms` measures differently for streaming.** On streaming requests the timer starts *after* compression, so end-to-end is `latency + overhead`. On non-streaming it's just `latency`. Don't mix both in one panel.
4. **A 5xx erases its own savings.** Requests that fail upstream are dropped from every savings and token counter. During a provider incident, savings rates look artificially clean while throughput falls.
5. **`/metrics` needs auth if you set a proxy token.** With `HEADROOM_PROXY_TOKEN` set, any non-loopback scraper must send `Authorization: Bearer <token>`. Loopback is always exempt.
---
## Metrics the docs mention that don't exist
If panels came back empty, this is probably why. These names appear in the published docs but not in the code:
`headroom_compression_ratio` · `headroom_latency_seconds` (and `_bucket`) · `headroom_cache_hits_total` · `headroom_cache_misses_total` · `headroom_cost_usd_total` · the `mode="optimize"` label on `headroom_requests_total`
The shipped `examples/grafana/headroom-dashboard.json` also filters every panel on `pool` and `hook` labels that no metric emits — the dropdowns will be permanently empty. Its metric names are otherwise correct.
---
## Setup reference
```bash
# Prometheus — nothing to do, GET /metrics is always on
# OpenTelemetry
pip install "headroom-ai[proxy,otel]"
export HEADROOM_OTEL_METRICS_ENABLED=1
export HEADROOM_OTEL_METRICS_ENDPOINT=https://otel.corp.example/v1/metrics
export HEADROOM_OTEL_METRICS_HEADERS="authorization=Bearer XXX"
export HEADROOM_OTEL_RESOURCE_ATTRIBUTES="service.instance.id=$HOSTNAME"
```
| Variable | Default | Notes |
|---|---|---|
| `HEADROOM_OTEL_METRICS_ENABLED` | `0` | Master switch |
| `HEADROOM_OTEL_METRICS_EXPORTER` | `otlp_http` | Or `console`. No gRPC exporter exists. |
| `HEADROOM_OTEL_METRICS_ENDPOINT` | unset | Passed verbatim — `/v1/metrics` is **not** appended |
| `HEADROOM_OTEL_METRICS_HEADERS` | unset | `k=v,k2=v2` |
| `HEADROOM_OTEL_METRICS_EXPORT_INTERVAL_MS` | `10000` | |
| `HEADROOM_OTEL_SERVICE_NAME` | `headroom-proxy` | |
| `HEADROOM_OTEL_RESOURCE_ATTRIBUTES` | unset | **Set `service.instance.id` here** — Headroom doesn't, and replicas will collide |
Verify with `curl -s localhost:8787/stats | jq .otel`.
**Multi-tenant labels:** `register_otel_metric_attribute_provider()` adds request-scoped attributes (tenant, team, cost centre) to every OTel datapoint. Max 16 attributes, 256 chars each.
**Air-gapped deployments:** `HEADROOM_OFFLINE=1` disables all outbound traffic — the anonymous usage beacon (which is **on by default**), the update check, and model downloads.
---

View file

@ -207,27 +207,43 @@ _PROFILES: dict[str, AgentSavingsProfile] = {
def get_agent_savings_profile(name: str | None = None) -> AgentSavingsProfile: def get_agent_savings_profile(name: str | None = None) -> AgentSavingsProfile:
"""Return a named agent savings profile. """Return a named agent savings profile.
An unrecognized name falls back to the ``balanced`` profile with a warning An unrecognized name degrades with a warning instead of raising. The savings
instead of raising. The savings profile is a soft config knob, but it is profile is a soft config knob, but it is resolved during proxy startup
resolved during proxy startup (``proxy_pipeline_kwargs`` -> ``create_app``), (``proxy_pipeline_kwargs`` -> ``create_app``), so raising here takes the
so raising here takes the whole proxy down before it can open its port. That whole proxy down before it can open its port. That happens on desktop/runtime
happens on desktop/runtime version skew: a newer client requests a profile version skew: a newer client requests a profile (e.g. ``coding``) that an
(e.g. ``coding``) that an older pinned or fallback runtime predates. Degrade older pinned or fallback runtime predates. Degrade rather than leaving the
to ``balanced`` rather than leaving the user with no proxy at all. user with no proxy at all.
**Where it degrades to matters.** This used to land on ``balanced``
unconditionally, which is a drastically different posture from the
out-of-box default: cache->token mode, cross-turn dedup off, tool-search
off, user messages uncompressed, the message floor 25x higher (250 vs 10)
and the block floor 20x higher (500 vs 25). A single typo in
``HEADROOM_SAVINGS_PROFILE`` therefore silently reconfigured the whole
proxy, and the only trace was one WARNING at startup that operators read
past. Prefer :data:`DEFAULT_PROFILE` the documented out-of-box posture and
the same thing an unset variable resolves to, so a typo now costs nothing.
``balanced`` remains the last resort for the genuine version-skew case,
where an older runtime has no ``DEFAULT_PROFILE`` entry to fall back to.
""" """
key = (name or DEFAULT_PROFILE).strip().lower() key = (name or DEFAULT_PROFILE).strip().lower()
profile = _PROFILES.get(key) profile = _PROFILES.get(key)
if profile is not None: if profile is not None:
return profile return profile
fallback_name = DEFAULT_PROFILE if DEFAULT_PROFILE in _PROFILES else FALLBACK_PROFILE
valid = ", ".join(sorted(_PROFILES)) valid = ", ".join(sorted(_PROFILES))
logger.warning( logger.warning(
"unknown savings profile %r; falling back to %r (known: %s)", "unknown savings profile %r; falling back to %r (known: %s). "
"Set HEADROOM_SAVINGS_PROFILE to one of the known names, or unset it to "
"get %r explicitly.",
name, name,
FALLBACK_PROFILE, fallback_name,
valid, valid,
DEFAULT_PROFILE,
) )
return _PROFILES[FALLBACK_PROFILE] return _PROFILES[fallback_name]
def apply_agent_savings_env_defaults( def apply_agent_savings_env_defaults(
@ -300,6 +316,28 @@ def proxy_pipeline_kwargs(config: object) -> dict[str, object]:
# unset → Kompress decides / ambient default applies). # unset → Kompress decides / ambient default applies).
if profile.target_ratio is not None: if profile.target_ratio is not None:
kwargs["target_ratio"] = profile.target_ratio kwargs["target_ratio"] = profile.target_ratio
# Block-compression char floor. Every OTHER router pipeline kwarg in this
# function travels on the config object; this one alone was populated
# only from ``HEADROOM_MIN_CHARS_FOR_BLOCK`` (read below), so a proxy
# whose config carries ``savings_profile="coding"`` but whose process env
# was never seeded applied every sibling coding knob while this floor
# silently stayed at ``ContentRouterConfig.min_chars_for_block_compression``
# (500) instead of the profile's 25 — a 20x gap on the gate that governs
# tool_result blocks, the dominant content type in agent traffic.
#
# NOTE: this does not make the profile fully config-deliverable. The
# profile's ``cross_turn_dedup`` / ``tool_search`` / ``lossless_then_lossy``
# / ``protect_reads`` / ``code_aware`` / ``effort_router`` / ``lossless``
# fields are still env-only, but by a different mechanism: their consumers
# read ``os.environ`` directly (ContentRouter.__init__ for HEADROOM_DEDUPE,
# the Anthropic handler for HEADROOM_TOOL_SEARCH) and never pass through
# this function at all. Those remain seed-dependent and are the reason a
# profile can still be half-applied; fixing them means threading each
# consumer, which is a larger change than this one.
#
# The env read below still wins, since it is an explicit operator override.
if profile.min_chars_for_block is not None:
kwargs["min_chars_for_block_compression"] = profile.min_chars_for_block
if getattr(config, "compress_user_messages", False): if getattr(config, "compress_user_messages", False):
kwargs["compress_user_messages"] = True kwargs["compress_user_messages"] = True

View file

@ -157,6 +157,12 @@ class PerfRecord:
tokens_before: int = 0 tokens_before: int = 0
tokens_after: int = 0 tokens_after: int = 0
tokens_saved: int = 0 tokens_saved: int = 0
# Tokens the forwarded request GREW by (PERF ``tok_inflated``). Both
# endpoints are clamped — ``tok_saved`` at zero and ``tok_inflated`` at zero
# — so a turn that left the proxy bigger reports ``tok_saved=0`` and hides
# its growth in a field nothing downstream read. Carrying it here is what
# lets the report state net alongside gross instead of implying they agree.
tokens_inflated: int = 0
tool_saved: int = 0 tool_saved: int = 0
cache_read: int = 0 cache_read: int = 0
cache_write: int = 0 cache_write: int = 0
@ -397,6 +403,7 @@ def parse_log_files(last_n_hours: float = 168.0) -> PerfReport:
tokens_before=int(kv.get("tok_before", 0)), tokens_before=int(kv.get("tok_before", 0)),
tokens_after=int(kv.get("tok_after", 0)), tokens_after=int(kv.get("tok_after", 0)),
tokens_saved=int(kv.get("tok_saved", 0)), tokens_saved=int(kv.get("tok_saved", 0)),
tokens_inflated=int(kv.get("tok_inflated", 0)),
tool_saved=int(kv.get("tool_saved", 0)), tool_saved=int(kv.get("tool_saved", 0)),
savings_breakdown=_decode_perf_savings(kv.get("savings", "none")), savings_breakdown=_decode_perf_savings(kv.get("savings", "none")),
cache_read=int(kv.get("cache_read", 0)), cache_read=int(kv.get("cache_read", 0)),
@ -546,6 +553,19 @@ def format_report(report: PerfReport) -> str:
# include tool bytes), so it used to render as a rival "Tool saved" line — which # include tool bytes), so it used to render as a rival "Tool saved" line — which
# read as a side metric and hid the win on tool-heavy turns where tok_saved=0. # read as a side metric and hid the win on tool-heavy turns where tok_saved=0.
lines.append(f"Tokens saved: {total_headline_saved:,} ({headline_pct:.1f}% reduction)") lines.append(f"Tokens saved: {total_headline_saved:,} ({headline_pct:.1f}% reduction)")
# Gross vs net. ``tok_saved`` is clamped at zero per request, so turns
# where Headroom made the body BIGGER (CCR proactive expansion, memory
# injection) contribute nothing negative to the headline — their growth
# lands in ``tok_inflated`` instead, which nothing here used to read.
# Printing "321,239,562 -> 313,274,727" directly above "8,455,763 saved"
# implies the two reconcile; they differ by exactly the inflation. Show
# it whenever it is non-zero so the arithmetic closes on the page.
total_inflated = sum(r.tokens_inflated for r in records)
if total_inflated > 0:
lines.append(
f" · inflated {total_inflated:,} "
f"(net message reduction {total_before - total_after:,})"
)
if total_tool_saved > 0: if total_tool_saved > 0:
lines.append(f" · messages {max(0, total_saved):,}") lines.append(f" · messages {max(0, total_saved):,}")
lines.append(f" · tool schemas {total_tool_saved:,}") lines.append(f" · tool schemas {total_tool_saved:,}")
@ -698,6 +718,31 @@ def format_report(report: PerfReport) -> str:
lines.append( lines.append(
f" {name}: {avg_pct:.1f}% avg reduction, {len(recs)} uses, {total_s:,} saved" f" {name}: {avg_pct:.1f}% avg reduction, {len(recs)} uses, {total_s:,} saved"
) )
# This table is built ONLY from "Transform NAME: B -> A tokens (saved N)"
# lines, which just one engine emits (transforms/pipeline.py). The
# OpenAI-Responses engine (transforms/compression_units.py +
# compression_batches.py) applies the same strategies and contains no
# logging calls at all, so none of its work appears above. On real
# traffic that hid ~7M of ~8.5M message-token savings — the table read
# "content_router: 189,783 saved" against a PERF total 44x larger, which
# invites exactly the wrong conclusion about which compressors work.
#
# State the divergence, NOT a coverage ratio. The two totals are
# different populations and neither strictly contains the other: the
# Transform lines carry no request_id, fire once per pipeline STAGE (so
# several can describe one request), and are emitted before the forwarder
# decides anything — a mutation later discarded by the signed-thinking
# byte-lock still logs its "saved" here while the request's PERF line
# correctly reports 0. So "table covers X of Y" would be a false subset
# claim in both directions; report the two sums and let the reader judge.
table_total = sum(r.tokens_saved for r in report.transform_records)
perf_total = sum(r.tokens_saved for r in report.perf_records)
if table_total != perf_total:
lines.append(
f" ! stage-level total {table_total:,} != PERF message total {perf_total:,} "
"— this table sees only engines that emit a Transform line, counts "
"per stage, and does not check whether the mutation shipped"
)
lines.append("") lines.append("")
# Router routing breakdown # Router routing breakdown
@ -717,11 +762,24 @@ def format_report(report: PerfReport) -> str:
f" Excluded: {total_excluded} ({total_excluded / total_all * 100:.0f}%) — Read/Glob outputs" f" Excluded: {total_excluded} ({total_excluded / total_all * 100:.0f}%) — Read/Glob outputs"
) )
lines.append( lines.append(
f" Skipped: {total_skipped} ({total_skipped / total_all * 100:.0f}%) — <50 words" f" Skipped: {total_skipped} ({total_skipped / total_all * 100:.0f}%) — below size floor"
) )
lines.append( lines.append(
f" Unchanged: {total_unchanged} ({total_unchanged / total_all * 100:.0f}%) — ratio too high" f" Unchanged: {total_unchanged} ({total_unchanged / total_all * 100:.0f}%) — ratio too high"
) )
# These four buckets are NOT the router's full outcome space — the
# `[router] route_counts=` line carries 17 keys, and the ones omitted
# here (cache_hit, system_msg, error_protected, already_compressed,
# …) are individually larger than "Excluded". Percentages taken over
# this subset therefore overstate every share: on real traffic the
# "skipped" bucket read 77% here against 49.5% of actual terminal
# fates, which reads as a mis-set threshold rather than a narrow
# denominator. Say what the denominator is instead of implying it is
# everything.
lines.append(
f" (shares are of these 4 buckets only, n={total_all}; "
"see `[router] route_counts=` for the full outcome space)"
)
if total_excluded > total_compressed * 3: if total_excluded > total_compressed * 3:
lines.append(" ! Excluded tools dominate — consider compressing stale Read outputs") lines.append(" ! Excluded tools dominate — consider compressing stale Read outputs")
lines.append("") lines.append("")
@ -803,6 +861,7 @@ PERF_RECORD_FIELDS = [
# Appended last so every existing CSV column keeps its position; a reader # Appended last so every existing CSV column keeps its position; a reader
# that indexes by name is unaffected either way. # that indexes by name is unaffected either way.
"from_response_cache", "from_response_cache",
"tokens_inflated",
] ]

View file

@ -768,11 +768,31 @@ class SavingsTracker:
delta_output_tokens_saved = max(_coerce_int(output_tokens_saved), 0) delta_output_tokens_saved = max(_coerce_int(output_tokens_saved), 0)
delta_cache_read_tokens = _coerce_int(cache_read_tokens) delta_cache_read_tokens = _coerce_int(cache_read_tokens)
priced = estimated_savings_usd priced = estimated_savings_usd
delta_savings_usd = ( # ``estimate_request_savings_usd`` prices FOUR buckets, but this method
max(_coerce_float(priced.get("compression")), 0.0) # only ever read three — ``tool_schema`` was computed and dropped on the
if priced is not None # floor. Tool-schema deferral is a quarter of the token headline on real
else _estimate_compression_savings_usd(model, delta_tokens_saved) # traffic (2.7M of 11.2M), and ``tokens_saved`` here is the bare
) # message-level figure (the caller folds deferral in separately for the
# ledger, see prometheus_metrics.record_request), so the dollars were
# simply missing rather than counted elsewhere. That is why "Cost saved"
# read materially below the token-savings percent on the same traffic.
# Add it to the compression bucket, matching how the PERF headline and
# perf/analyzer fold deferral into one number.
if priced is not None:
# Two DISJOINT buckets: the caller passes bare message savings as
# ``compression_tokens_saved`` and deferral separately as
# ``tool_schema_tokens_saved`` (see prometheus_metrics.record_request),
# so summing them is additive, not double counting. Written as a
# statement rather than folded into the ternary below — a money path
# should not depend on the reader knowing that ``a + b if c else d``
# groups as ``(a + b) if c else d``.
delta_savings_usd = max(_coerce_float(priced.get("compression")), 0.0) + max(
_coerce_float(priced.get("tool_schema")), 0.0
)
else:
# No priced breakdown available: only message savings are known here,
# so this path stays message-only exactly as before.
delta_savings_usd = _estimate_compression_savings_usd(model, delta_tokens_saved)
delta_output_savings_usd = ( delta_output_savings_usd = (
max(_coerce_float(priced.get("output_shaping")), 0.0) max(_coerce_float(priced.get("output_shaping")), 0.0)
if priced is not None if priced is not None

View file

@ -66,7 +66,7 @@ except ImportError:
sys.path.insert(0, str(Path(__file__).parent.parent.parent)) sys.path.insert(0, str(Path(__file__).parent.parent.parent))
from headroom._version import __version__ from headroom._version import __version__
from headroom.agent_savings import proxy_pipeline_kwargs from headroom.agent_savings import DEFAULT_PROFILE, proxy_pipeline_kwargs
from headroom.cache.compression_feedback import get_compression_feedback from headroom.cache.compression_feedback import get_compression_feedback
from headroom.cache.compression_store import format_retrieval_miss_detail, get_compression_store from headroom.cache.compression_store import format_retrieval_miss_detail, get_compression_store
from headroom.ccr import ( from headroom.ccr import (
@ -1737,6 +1737,38 @@ class HeadroomProxy(
if self.config.mode == PROXY_MODE_CACHE: if self.config.mode == PROXY_MODE_CACHE:
logger.info(" Prefix freeze: strict (all prior turns immutable)") logger.info(" Prefix freeze: strict (all prior turns immutable)")
logger.info(" Mutations: latest turn only") logger.info(" Mutations: latest turn only")
# Effective compression posture, resolved (not merely requested). The
# savings profile reaches the router through two paths — the config
# object and the seeded process env — and `setdefault` semantics mean a
# stale HEADROOM_* value silently overrides the profile that names it.
# A deployment can therefore log `savings_profile=coding` while actually
# running balanced's thresholds, and the only prior evidence was a
# single easily-missed WARNING. Print what is actually in force so
# "which profile am I really running?" is answerable from the banner.
try:
_eff = proxy_pipeline_kwargs(self.config)
# Read cross-turn dedup off the CONSTRUCTED router, not off the env.
# ContentRouter resolves it as `config.enable_cross_turn_dedup OR
# $HEADROOM_DEDUPE`, so reporting the env alone would be a guess that
# happens to be right only while nothing sets the config field. A
# banner line exists to be trusted; it must read what was resolved.
_dedupe: object = "unknown"
for _t in getattr(self.anthropic_pipeline, "transforms", []):
if isinstance(_t, ContentRouter):
_dedupe = bool(getattr(_t, "_cross_turn_dedup_enabled", False))
break
logger.info(
"Savings profile: %s (effective: min_tokens=%s min_chars_block=%s "
"compress_user=%s dedupe=%s tool_search=%s)",
self.config.savings_profile or DEFAULT_PROFILE,
_eff.get("min_tokens_to_compress", "default"),
_eff.get("min_chars_for_block_compression", "default(500)"),
_eff.get("compress_user_messages", False),
_dedupe,
os.environ.get("HEADROOM_TOOL_SEARCH", "1") in ("1", "true", "yes", "on", "auto"),
)
except Exception: # never let a banner line block startup
logger.debug("effective savings-profile banner skipped", exc_info=True)
logger.info(f"Caching: {'ENABLED' if self.config.cache_enabled else 'DISABLED'}") logger.info(f"Caching: {'ENABLED' if self.config.cache_enabled else 'DISABLED'}")
logger.info(f"Rate Limiting: {'ENABLED' if self.config.rate_limit_enabled else 'DISABLED'}") logger.info(f"Rate Limiting: {'ENABLED' if self.config.rate_limit_enabled else 'DISABLED'}")
logger.info( logger.info(

View file

@ -5552,7 +5552,16 @@ class ContentRouter(Transform):
if route_counts["user_msg"]: if route_counts["user_msg"]:
parts.append(f"{route_counts['user_msg']} skipped (user)") parts.append(f"{route_counts['user_msg']} skipped (user)")
if route_counts["small"]: if route_counts["small"]:
parts.append(f"{route_counts['small']} skipped (<50 words)") # Report the thresholds actually in force, not a literal. This line
# used to read "skipped (<50 words)" unconditionally: wrong number
# (the message gate is `min_tokens`, which profiles set anywhere from
# 10 to 250), wrong unit (tokens and characters, never words), and it
# merged two different gates under one label. Operators read it as
# evidence of a mis-set threshold and tuned the wrong knob.
parts.append(
f"{route_counts['small']} skipped "
f"(<{min_tokens} tok msg / <{min_chars_for_block_compression} chars block)"
)
if route_counts["recent_code"]: if route_counts["recent_code"]:
parts.append(f"{route_counts['recent_code']} protected (recent code)") parts.append(f"{route_counts['recent_code']} protected (recent code)")
if route_counts["analysis_ctx"]: if route_counts["analysis_ctx"]:

View file

@ -10,6 +10,7 @@ from click.testing import CliRunner
from headroom.agent_savings import ( from headroom.agent_savings import (
AGENT_90_PROFILE, AGENT_90_PROFILE,
DEFAULT_PROFILE,
apply_agent_savings_env_defaults, apply_agent_savings_env_defaults,
apply_agent_savings_profile, apply_agent_savings_profile,
get_agent_savings_profile, get_agent_savings_profile,
@ -170,17 +171,30 @@ def test_agent_savings_env_defaults_preserve_user_overrides() -> None:
assert env["HEADROOM_SMART_CRUSHER_COMPACTION"] == "0" assert env["HEADROOM_SMART_CRUSHER_COMPACTION"] == "0"
def test_unknown_agent_savings_profile_falls_back_to_balanced( def test_unknown_agent_savings_profile_falls_back_to_default(
caplog: pytest.LogCaptureFixture, caplog: pytest.LogCaptureFixture,
) -> None: ) -> None:
# An unknown profile must NOT raise: it's resolved during proxy startup, so # An unknown profile must NOT raise: it's resolved during proxy startup, so
# raising takes the whole proxy down before it opens its port (desktop asked # raising takes the whole proxy down before it opens its port (desktop asked
# for a profile a fallback runtime predates). Degrade to "balanced" instead. # for a profile a fallback runtime predates).
#
# It must degrade to the DEFAULT profile, not to "balanced". The two are not
# interchangeable: balanced flips cache->token mode, turns cross-turn dedup
# and tool-search off, stops compressing user messages, and raises the
# message floor 25x (250 vs 10) and the block floor 20x (500 vs 25). A typo
# in HEADROOM_SAVINGS_PROFILE used to silently reconfigure the entire proxy
# into that posture, which is strictly worse than behaving as if the
# variable were unset.
with caplog.at_level(logging.WARNING): with caplog.at_level(logging.WARNING):
profile = get_agent_savings_profile("missing") profile = get_agent_savings_profile("missing")
assert profile is get_agent_savings_profile("balanced") assert profile is get_agent_savings_profile(None)
assert profile.name == DEFAULT_PROFILE
assert profile is not get_agent_savings_profile("balanced")
assert "unknown savings profile" in caplog.text assert "unknown savings profile" in caplog.text
assert "missing" in caplog.text assert "missing" in caplog.text
# The warning has to name the resolved profile, so an operator reading it
# knows what they actually got rather than only what they asked for.
assert DEFAULT_PROFILE in caplog.text
def test_with_target_savings_recomputes_target_ratio() -> None: def test_with_target_savings_recomputes_target_ratio() -> None:
@ -784,3 +798,45 @@ def test_agent_savings_smoke_fixture_passes_real_gate(tmp_path) -> None:
assert "codex: 91.0% savings meets 90.0%" in gate_result.output assert "codex: 91.0% savings meets 90.0%" in gate_result.output
assert "cursor: 93.0% savings meets 90.0%" in gate_result.output assert "cursor: 93.0% savings meets 90.0%" in gate_result.output
assert "100.0% accuracy meets 90.0%" in gate_result.output assert "100.0% accuracy meets 90.0%" in gate_result.output
def test_coding_profile_min_chars_block_reaches_router_without_env_seeding(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""The block-char floor must travel on the config object, not env only.
Every other router pipeline kwarg this function builds travels on the config
object; ``min_chars_for_block`` alone was populated only from
``HEADROOM_MIN_CHARS_FOR_BLOCK`` (emitted by ``proxy_env()``). A proxy whose
config carried ``savings_profile="coding"`` but whose process env was never
seeded applied every sibling coding knob while this floor silently stayed at
``ContentRouterConfig.min_chars_for_block_compression`` (500) instead of the
profile's 25 — a 20x gap on the gate that governs tool_result blocks.
Note this does not make the profile fully config-deliverable: fields whose
consumers read ``os.environ`` directly (``cross_turn_dedup`` via
ContentRouter, ``tool_search`` via the Anthropic handler) never pass through
this function and remain seed-dependent.
"""
monkeypatch.delenv("HEADROOM_MIN_CHARS_FOR_BLOCK", raising=False)
class _Config:
savings_profile = "coding"
min_tokens_to_crush = 500
kwargs = proxy_pipeline_kwargs(_Config())
assert kwargs["min_chars_for_block_compression"] == 25
assert kwargs["min_tokens_to_compress"] == 10
def test_explicit_min_chars_block_env_overrides_the_profile(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""An explicit operator override still wins over the profile value."""
monkeypatch.setenv("HEADROOM_MIN_CHARS_FOR_BLOCK", "120")
class _Config:
savings_profile = "coding"
min_tokens_to_crush = 500
assert proxy_pipeline_kwargs(_Config())["min_chars_for_block_compression"] == 120