mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
fix(reporting): show net vs gross savings, real skip thresholds, and the effective profile (#3123)
## Description
Six reporting/config defects found while investigating a user reporting
~1% savings on Claude Code. **None of these changes how much Headroom
compresses** — all of them change whether an operator can tell what it
did. Every one was found by reading that user's own 227,777 lines of
proxy logs against the code.
## Changes Made
- **`perf/analyzer`: parse and render `tok_inflated`.** Every PERF line
carried it; nothing downstream read it. The report could print
`321,239,562 -> 313,274,727` directly above `8,455,763 saved` — two
figures that differ by exactly the 490,928 tokens of inflation it
omitted.
- **`content_router`: report the real skip thresholds.** The routing
summary hardcoded `skipped (<50 words)` regardless of what was in force.
Wrong number (the message gate is `min_tokens`, 10–250 by profile),
wrong unit (tokens and characters, never words), and it merged two
different gates under one label.
- **`perf/analyzer`: disclose that Transform Effectiveness is partial.**
It is built only from `pipeline.py`'s `Transform NAME:` lines.
`compression_units.py` / `compression_batches.py` contain zero logging
calls, so the table read `content_router: 189,783 saved` against a PERF
total 44x larger. Reports the divergence rather than a coverage ratio —
the two are different populations and neither contains the other (those
lines carry no request_id, fire per stage, and are emitted before the
forwarder decides).
- **`perf/analyzer`: disclose the routing denominator.** Percentages
were taken over 4 of the router's 17 outcome buckets, silently dropping
buckets larger than several it displayed.
- **`savings_tracker`: stop dropping tool-schema dollars.**
`estimate_request_savings_usd` prices four buckets; `record_request`
read three. `tool_schema` was computed and discarded, so a quarter of
the token headline never reached "Cost saved". The two inputs are
disjoint (verified at the call site), so this is additive, not
double-counting.
- **`agent_savings`: an unknown profile no longer degrades to
`balanced`.** `balanced` is a different product posture from the default
`coding`: cache→token mode, dedup off, tool-search off, user messages
uncompressed, message floor 25x higher, block floor 20x higher. A typo
in `HEADROOM_SAVINGS_PROFILE` silently reconfigured the whole proxy. Now
degrades to `DEFAULT_PROFILE` and names the resolved profile in the
warning.
- **`agent_savings`: give `min_chars_for_block` a config-object path.**
Every other router pipeline kwarg travels on the config object; this one
alone was env-only, so an unseeded proxy applied every sibling `coding`
knob while this floor stayed at 500 instead of 25.
- **`server`: log the resolved compression posture at startup**, reading
cross-turn dedup off the constructed router rather than the environment
(the router resolves it as `config OR env`, so reading env alone would
be a guess).
## Testing
- [x] Unit tests pass, [x] ruff, [x] mypy, [x] new tests added
```text
uv run pytest tests/ -k "content_router or agent_savings or perf or analyzer or savings or proxy_server or cli_perf or prometheus"
620 passed, 25 skipped
uv run mypy headroom # Success
uv run ruff check . && ruff format --check . # clean
```
## Real behavior proof
- **Setup:** macOS arm64, Python 3.12, this branch. Input: 60 MB /
227,777 lines of real proxy logs from the reporting user (6 rotated
files, 2,792 PERF lines, 2026-08-17 → 2026-08-19).
- **Steps:** pointed `headroom.perf.analyzer.LOG_DIR` at that directory
and rendered the report before and after the patch.
- **After-fix output (real data, unmodified):**
```text
Requests: 2792
Tokens: 321,288,161 -> 313,323,326 (2.6% messages)
Tokens saved: 11,158,901 (3.4% reduction)
· inflated 490,928 (net message reduction 7,964,835)
· messages 8,455,763
· tool schemas 2,703,138
! stage-level total 190,641 != PERF message total 8,455,763 — this table sees only
engines that emit a Transform line, counts per stage, and does not check whether
the mutation shipped
Skipped: 44641 (77%) — below size floor
(shares are of these 4 buckets only, n=58319; see `[router] route_counts=` for the
full outcome space)
```
The arithmetic now closes on the page: `8,455,763 - 490,928 =
7,964,835`, matching the token delta exactly. Before the patch none of
the three annotated lines existed and the `Skipped` line claimed `<50
words`.
- **Profile resolution verified by execution**, not inspection —
subprocesses with controlled env:
```text
vanilla (nothing set) mode=cache dedupe=1 tool_search=1 min_tokens=10 min_chars=25
HEADROOM_SAVINGS_PROFILE=coding mode=cache dedupe=1 tool_search=1 min_tokens=10 min_chars=25
unknown profile name (before) mode=token dedupe=0 tool_search=0 min_tokens=250 min_chars=500
unknown profile name (after) -> resolves to `coding`, warning names it
coding, seeding never runs min_chars=25 (was 500 before this patch)
```
- **Not tested:** live paid Anthropic traffic. These are
reporting/config surfaces; the wire path is untouched by this PR.
## Review readiness
- [x] Self-reviewed. Three overclaims in my own first draft were
corrected before this PR: a false subset claim in the Transform
Effectiveness note, a comment asserting `min_chars_for_block` was the
*only* env-only field (it is the only env-only *router pipeline kwarg*;
`cross_turn_dedup`, `tool_search`, `protect_reads`, `code_aware`,
`effort_router`, `lossless` remain env-only via a different mechanism
and are **not** fixed here), and a money-path expression that relied on
`a + b if c else d` grouping.
## Known remaining (deliberately out of scope)
- `Requests: N` still overcounts: the Codex WS forwarder reuses one
`request_id` across every turn (one observed 156x), plus ~18 duplicate
PERF emissions.
- `compression_units.py` / `compression_batches.py` remain unlogged —
this PR *discloses* the blind spot rather than closing it.
- The headline stays **gross**. True net is `11,158,901 - 490,928 =
10,667,973` (3.3%, not 3.4%). Making net the headline lowers every
user's reported savings ~4.4%; that is a product call, not mine, so the
inflation is surfaced beside it instead.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
05f5ef47cb
commit
250ede2f7f
7 changed files with 458 additions and 21 deletions
223
docs/metrics-technical-guide.md
Normal file
223
docs/metrics-technical-guide.md
Normal file
|
|
@ -0,0 +1,223 @@
|
||||||
|
# Headroom Metrics — Dashboard Guide
|
||||||
|
|
||||||
|
What each metric shows, so you can build panels against it.
|
||||||
|
|
||||||
|
**Two endpoints.** Both are on the proxy (default `:8787`).
|
||||||
|
|
||||||
|
| Surface | How to get it | Use it for |
|
||||||
|
|---|---|---|
|
||||||
|
| **Prometheus** — `GET /metrics` | Always on, no config | Everything below. Start here. |
|
||||||
|
| **OpenTelemetry** — OTLP/HTTP | `HEADROOM_OTEL_METRICS_ENABLED=1` + `pip install "headroom-ai[proxy,otel]"` | Same data, dotted names, plus per-tenant labels |
|
||||||
|
|
||||||
|
Names differ between them: Prometheus uses `headroom_tokens_saved_total` (**milliseconds** for timings), OTel uses `headroom.proxy.tokens.saved` (**seconds**). Both are listed below.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The savings panel — start here
|
||||||
|
|
||||||
|
**`headroom.proxy.tokens.saved`** is the headline number. It already combines compression + tool-schema deferral — no need to add anything to it.
|
||||||
|
|
||||||
|
| Metric | What it shows |
|
||||||
|
|---|---|
|
||||||
|
| **`headroom.proxy.tokens.saved`** *(OTel)* | **Total input tokens Headroom kept out of the request.** Compression + tool savings, combined. This is your hero number. |
|
||||||
|
| `headroom.proxy.savings.usd{source}` *(OTel)* | **Dollars saved**, split by layer: `compression`, `tool_schema`, `output_shaping`, `provider_cache`. Sum for the total. |
|
||||||
|
| `headroom_persistent_savings_tokens_saved_total` | Same tokens-saved number, but **survives proxy restarts**. Use for "lifetime saved" tiles. |
|
||||||
|
| `headroom_persistent_savings_compression_savings_usd_total` | **Lifetime dollars saved**, durable across restarts. |
|
||||||
|
| `headroom_tokens_input_total` | Input tokens actually sent upstream (post-compression). The denominator for a reduction %. |
|
||||||
|
| `headroom_tokens_output_total` | Output tokens returned by the provider. |
|
||||||
|
|
||||||
|
```promql
|
||||||
|
# Hero tile: tokens saved per second
|
||||||
|
rate(headroom_tokens_saved_total[5m])
|
||||||
|
+ sum(rate(headroom_savings_attributed_tokens_total{source="tool_search",realized="true"}[5m]))
|
||||||
|
|
||||||
|
# Context reduction %
|
||||||
|
100 * rate(headroom_tokens_saved_total[5m])
|
||||||
|
/ clamp_min(rate(headroom_tokens_input_total[5m]) + rate(headroom_tokens_saved_total[5m]), 1)
|
||||||
|
|
||||||
|
# Lifetime tiles (survive restart)
|
||||||
|
headroom_persistent_savings_tokens_saved_total
|
||||||
|
headroom_persistent_savings_compression_savings_usd_total
|
||||||
|
```
|
||||||
|
|
||||||
|
> **One catch on the Prometheus side.** `headroom_tokens_saved_total` is compression **only** — it leaves out tool-schema deferral. The OTel `headroom.proxy.tokens.saved` includes both. That's why the query above adds the `tool_search` term back in. On tool-heavy workloads the gap is large.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Latency panel
|
||||||
|
|
||||||
|
All Prometheus timings are in **milliseconds**, exposed as `_sum` / `_count` / `_min` / `_max`. Build means with `rate(sum)/rate(count)`.
|
||||||
|
|
||||||
|
| Metric | What it shows |
|
||||||
|
|---|---|
|
||||||
|
| **`headroom_overhead_ms_*`** | **Latency Headroom itself adds.** Handler entry → end of compression. Excludes the LLM call. This is the "what does this cost us" number. |
|
||||||
|
| `headroom_latency_ms_*` | Total request duration, including the provider. |
|
||||||
|
| `headroom_ttfb_ms_*` | Time to first byte from upstream. Streaming requests only. |
|
||||||
|
| `headroom_stage_timing_ms_*{path,stage}` | Where time went inside the handler — `compression_first_stage`, `upstream_connect`, `memory_context`, etc. |
|
||||||
|
| `headroom_transform_timing_ms_*{transform}` | Time per compression transform. Use to find a slow transform. |
|
||||||
|
|
||||||
|
```promql
|
||||||
|
# Headroom's added overhead, mean ms
|
||||||
|
rate(headroom_overhead_ms_sum[5m]) / rate(headroom_overhead_ms_count[5m])
|
||||||
|
|
||||||
|
# End-to-end, mean ms
|
||||||
|
rate(headroom_latency_ms_sum[5m]) / rate(headroom_latency_ms_count[5m])
|
||||||
|
|
||||||
|
# Slowest stages
|
||||||
|
topk(5, rate(headroom_stage_timing_ms_sum[5m]) / rate(headroom_stage_timing_ms_count[5m]))
|
||||||
|
```
|
||||||
|
|
||||||
|
> **No percentiles are available.** There are no histogram buckets on `/metrics`, and the OTel histograms ship with default buckets that put every request into one bucket, so `histogram_quantile()` returns nonsense. **Means work fine.** For real p95/p99 today, use the `headroom perf` CLI.
|
||||||
|
>
|
||||||
|
> Also: divide each `_sum` by **its own** `_count`. Overhead and TTFB are only sampled when > 0, so their counts are smaller than the latency count.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Cache panel
|
||||||
|
|
||||||
|
| Metric | What it shows |
|
||||||
|
|---|---|
|
||||||
|
| `headroom_provider_cache_hit_requests_total{provider}` | Requests that read from the provider's prompt cache. |
|
||||||
|
| `headroom_provider_cache_requests_total{provider}` | Requests with any cache activity. **The correct denominator for hit rate.** |
|
||||||
|
| `headroom_cache_read_tokens_total{provider}` | Tokens served from cache (the discounted ones). |
|
||||||
|
| `headroom_cache_write_tokens_total{provider}` | Tokens written into cache (these carry a premium). |
|
||||||
|
| `headroom_cache_write_ttl_tokens_total{provider,ttl}` | Cache writes split by TTL — `5m` vs `1h`. |
|
||||||
|
| `headroom_uncached_input_tokens_total{provider}` | Input tokens that missed cache entirely. |
|
||||||
|
| `headroom_cache_bust_total` | Requests where compression broke a cached prefix. **Should stay near zero.** |
|
||||||
|
| `headroom_cache_miss_attribution_total{provider,reason}` | Why a cached prefix missed — `ttl_expiry`, `prefix_change`, `unknown`. |
|
||||||
|
|
||||||
|
```promql
|
||||||
|
# Cache hit rate by provider
|
||||||
|
sum by (provider) (rate(headroom_provider_cache_hit_requests_total[5m]))
|
||||||
|
/ sum by (provider) (rate(headroom_provider_cache_requests_total[5m]))
|
||||||
|
|
||||||
|
# Compression breaking cache — alert if this rises
|
||||||
|
rate(headroom_cache_bust_total[5m])
|
||||||
|
```
|
||||||
|
|
||||||
|
> **Don't use `headroom_requests_cached_total` as a hit rate.** It mixes the provider's prompt cache with Headroom's own response cache into one boolean, so it measures neither.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Traffic & health panel
|
||||||
|
|
||||||
|
| Metric | What it shows |
|
||||||
|
|---|---|
|
||||||
|
| `headroom_requests_total` | Requests handled. Unlabelled. |
|
||||||
|
| `headroom_requests_by_provider{provider}` | Traffic split by provider — `anthropic`, `openai`, `gemini`, `bedrock`… |
|
||||||
|
| `headroom_requests_by_model{model}` | Traffic split by model. Capped at 1024 distinct; overflow lands in `model="other"`. |
|
||||||
|
| `headroom_requests_failed_total` | Upstream 5xx errors. |
|
||||||
|
| `headroom_requests_rate_limited_total` | Requests **Headroom** rejected via its own rate limiter (not upstream 429s). |
|
||||||
|
| `headroom_compression_failed_total{reason}` | Compression failures — `timeout` or `error`. Fails open, so traffic keeps flowing but savings quietly stop. **Worth an alert.** |
|
||||||
|
| `headroom_compression_quarantine_total{event}` | Compression disabled after repeated timeouts — `activated`, `skipped`, `released`. |
|
||||||
|
| `headroom_inbound_requests_active` | In-flight requests, gauge. Counts all HTTP including `/metrics`. |
|
||||||
|
| `headroom_active_ws_sessions` | Live Codex WebSocket sessions, gauge. |
|
||||||
|
|
||||||
|
```promql
|
||||||
|
# Failure rate
|
||||||
|
rate(headroom_requests_failed_total[5m])
|
||||||
|
/ clamp_min(rate(headroom_requests_total[5m]) + rate(headroom_requests_failed_total[5m]), 1)
|
||||||
|
|
||||||
|
# Savings silently stopped
|
||||||
|
sum by (reason) (rate(headroom_compression_failed_total[5m]))
|
||||||
|
|
||||||
|
# Traffic mix
|
||||||
|
sum by (provider) (rate(headroom_requests_by_provider[5m]))
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Anthropic subscription panel
|
||||||
|
|
||||||
|
Only if you're on an Anthropic OAuth/subscription plan. OTel only, gauges, no labels.
|
||||||
|
|
||||||
|
| Metric | What it shows |
|
||||||
|
|---|---|
|
||||||
|
| `headroom.subscription.5h_utilization_pct` | How much of the 5-hour rate-limit window is used (0–100). |
|
||||||
|
| `headroom.subscription.7d_utilization_pct` | Same for the 7-day window. |
|
||||||
|
| `headroom.subscription.5h_seconds_to_reset` | Seconds until the 5-hour window resets. |
|
||||||
|
| `headroom.subscription.7d_seconds_to_reset` | Seconds until the 7-day window resets. |
|
||||||
|
| `headroom.subscription.overage_usd` | Extra-usage credits consumed, in dollars. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Attribution — where savings came from
|
||||||
|
|
||||||
|
| Metric | What it shows |
|
||||||
|
|---|---|
|
||||||
|
| `headroom_savings_attributed_tokens_total{source,realized}` | Tokens saved, broken out by named source. `source="tool_search"` is tool-schema deferral. |
|
||||||
|
| `headroom_savings_attributed_usd_total{source,realized}` | Dollars saved by source. **Gauge, can go negative** — don't `rate()` it. |
|
||||||
|
| `headroom_savings_attribution_events_total{source,realized}` | How often each source contributed. |
|
||||||
|
| `headroom_waste_signal_tokens_total{signal}` | Wasteful patterns *detected* in the input — `json_bloat`, `base64`, `repetition`, `reread`… This is diagnosis, **not savings**. |
|
||||||
|
|
||||||
|
These rows *explain* the headline total — they are never added to it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Compression internals
|
||||||
|
|
||||||
|
| Metric | What it shows |
|
||||||
|
|---|---|
|
||||||
|
| `headroom.compression.tokens.input` *(OTel)* | Tokens going into the compression pipeline. |
|
||||||
|
| `headroom.compression.tokens.output` *(OTel)* | Tokens coming out. |
|
||||||
|
| `headroom.compression.tokens.saved` *(OTel)* | The difference. Pipeline-level view of compression only. |
|
||||||
|
| `headroom.compression.runs` *(OTel)* | Pipeline executions. Note: **per pipeline run, not per request.** |
|
||||||
|
| `headroom.compression.pipeline.duration` *(OTel, seconds)* | How long the pipeline took. |
|
||||||
|
| `headroom.compression.transforms{transform}` *(OTel)* | Which transforms fired. **High cardinality — drop or aggregate at the collector.** |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Five things that will break a dashboard
|
||||||
|
|
||||||
|
1. **Only savings counters survive a restart.** 55 of 60 Prometheus families reset to zero when the proxy restarts. Only `headroom_persistent_savings_*` is durable, and it needs `HEADROOM_WORKSPACE_DIR` on a persistent volume — otherwise it resets on every deploy.
|
||||||
|
|
||||||
|
2. **No percentiles anywhere.** Use means. See the latency section.
|
||||||
|
|
||||||
|
3. **`headroom_latency_ms` measures differently for streaming.** On streaming requests the timer starts *after* compression, so end-to-end is `latency + overhead`. On non-streaming it's just `latency`. Don't mix both in one panel.
|
||||||
|
|
||||||
|
4. **A 5xx erases its own savings.** Requests that fail upstream are dropped from every savings and token counter. During a provider incident, savings rates look artificially clean while throughput falls.
|
||||||
|
|
||||||
|
5. **`/metrics` needs auth if you set a proxy token.** With `HEADROOM_PROXY_TOKEN` set, any non-loopback scraper must send `Authorization: Bearer <token>`. Loopback is always exempt.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Metrics the docs mention that don't exist
|
||||||
|
|
||||||
|
If panels came back empty, this is probably why. These names appear in the published docs but not in the code:
|
||||||
|
|
||||||
|
`headroom_compression_ratio` · `headroom_latency_seconds` (and `_bucket`) · `headroom_cache_hits_total` · `headroom_cache_misses_total` · `headroom_cost_usd_total` · the `mode="optimize"` label on `headroom_requests_total`
|
||||||
|
|
||||||
|
The shipped `examples/grafana/headroom-dashboard.json` also filters every panel on `pool` and `hook` labels that no metric emits — the dropdowns will be permanently empty. Its metric names are otherwise correct.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Setup reference
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Prometheus — nothing to do, GET /metrics is always on
|
||||||
|
|
||||||
|
# OpenTelemetry
|
||||||
|
pip install "headroom-ai[proxy,otel]"
|
||||||
|
export HEADROOM_OTEL_METRICS_ENABLED=1
|
||||||
|
export HEADROOM_OTEL_METRICS_ENDPOINT=https://otel.corp.example/v1/metrics
|
||||||
|
export HEADROOM_OTEL_METRICS_HEADERS="authorization=Bearer XXX"
|
||||||
|
export HEADROOM_OTEL_RESOURCE_ATTRIBUTES="service.instance.id=$HOSTNAME"
|
||||||
|
```
|
||||||
|
|
||||||
|
| Variable | Default | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| `HEADROOM_OTEL_METRICS_ENABLED` | `0` | Master switch |
|
||||||
|
| `HEADROOM_OTEL_METRICS_EXPORTER` | `otlp_http` | Or `console`. No gRPC exporter exists. |
|
||||||
|
| `HEADROOM_OTEL_METRICS_ENDPOINT` | unset | Passed verbatim — `/v1/metrics` is **not** appended |
|
||||||
|
| `HEADROOM_OTEL_METRICS_HEADERS` | unset | `k=v,k2=v2` |
|
||||||
|
| `HEADROOM_OTEL_METRICS_EXPORT_INTERVAL_MS` | `10000` | |
|
||||||
|
| `HEADROOM_OTEL_SERVICE_NAME` | `headroom-proxy` | |
|
||||||
|
| `HEADROOM_OTEL_RESOURCE_ATTRIBUTES` | unset | **Set `service.instance.id` here** — Headroom doesn't, and replicas will collide |
|
||||||
|
|
||||||
|
Verify with `curl -s localhost:8787/stats | jq .otel`.
|
||||||
|
|
||||||
|
**Multi-tenant labels:** `register_otel_metric_attribute_provider()` adds request-scoped attributes (tenant, team, cost centre) to every OTel datapoint. Max 16 attributes, 256 chars each.
|
||||||
|
|
||||||
|
**Air-gapped deployments:** `HEADROOM_OFFLINE=1` disables all outbound traffic — the anonymous usage beacon (which is **on by default**), the update check, and model downloads.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
@ -207,27 +207,43 @@ _PROFILES: dict[str, AgentSavingsProfile] = {
|
||||||
def get_agent_savings_profile(name: str | None = None) -> AgentSavingsProfile:
|
def get_agent_savings_profile(name: str | None = None) -> AgentSavingsProfile:
|
||||||
"""Return a named agent savings profile.
|
"""Return a named agent savings profile.
|
||||||
|
|
||||||
An unrecognized name falls back to the ``balanced`` profile with a warning
|
An unrecognized name degrades with a warning instead of raising. The savings
|
||||||
instead of raising. The savings profile is a soft config knob, but it is
|
profile is a soft config knob, but it is resolved during proxy startup
|
||||||
resolved during proxy startup (``proxy_pipeline_kwargs`` -> ``create_app``),
|
(``proxy_pipeline_kwargs`` -> ``create_app``), so raising here takes the
|
||||||
so raising here takes the whole proxy down before it can open its port. That
|
whole proxy down before it can open its port. That happens on desktop/runtime
|
||||||
happens on desktop/runtime version skew: a newer client requests a profile
|
version skew: a newer client requests a profile (e.g. ``coding``) that an
|
||||||
(e.g. ``coding``) that an older pinned or fallback runtime predates. Degrade
|
older pinned or fallback runtime predates. Degrade rather than leaving the
|
||||||
to ``balanced`` rather than leaving the user with no proxy at all.
|
user with no proxy at all.
|
||||||
|
|
||||||
|
**Where it degrades to matters.** This used to land on ``balanced``
|
||||||
|
unconditionally, which is a drastically different posture from the
|
||||||
|
out-of-box default: cache->token mode, cross-turn dedup off, tool-search
|
||||||
|
off, user messages uncompressed, the message floor 25x higher (250 vs 10)
|
||||||
|
and the block floor 20x higher (500 vs 25). A single typo in
|
||||||
|
``HEADROOM_SAVINGS_PROFILE`` therefore silently reconfigured the whole
|
||||||
|
proxy, and the only trace was one WARNING at startup that operators read
|
||||||
|
past. Prefer :data:`DEFAULT_PROFILE` — the documented out-of-box posture and
|
||||||
|
the same thing an unset variable resolves to, so a typo now costs nothing.
|
||||||
|
``balanced`` remains the last resort for the genuine version-skew case,
|
||||||
|
where an older runtime has no ``DEFAULT_PROFILE`` entry to fall back to.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
key = (name or DEFAULT_PROFILE).strip().lower()
|
key = (name or DEFAULT_PROFILE).strip().lower()
|
||||||
profile = _PROFILES.get(key)
|
profile = _PROFILES.get(key)
|
||||||
if profile is not None:
|
if profile is not None:
|
||||||
return profile
|
return profile
|
||||||
|
fallback_name = DEFAULT_PROFILE if DEFAULT_PROFILE in _PROFILES else FALLBACK_PROFILE
|
||||||
valid = ", ".join(sorted(_PROFILES))
|
valid = ", ".join(sorted(_PROFILES))
|
||||||
logger.warning(
|
logger.warning(
|
||||||
"unknown savings profile %r; falling back to %r (known: %s)",
|
"unknown savings profile %r; falling back to %r (known: %s). "
|
||||||
|
"Set HEADROOM_SAVINGS_PROFILE to one of the known names, or unset it to "
|
||||||
|
"get %r explicitly.",
|
||||||
name,
|
name,
|
||||||
FALLBACK_PROFILE,
|
fallback_name,
|
||||||
valid,
|
valid,
|
||||||
|
DEFAULT_PROFILE,
|
||||||
)
|
)
|
||||||
return _PROFILES[FALLBACK_PROFILE]
|
return _PROFILES[fallback_name]
|
||||||
|
|
||||||
|
|
||||||
def apply_agent_savings_env_defaults(
|
def apply_agent_savings_env_defaults(
|
||||||
|
|
@ -300,6 +316,28 @@ def proxy_pipeline_kwargs(config: object) -> dict[str, object]:
|
||||||
# unset → Kompress decides / ambient default applies).
|
# unset → Kompress decides / ambient default applies).
|
||||||
if profile.target_ratio is not None:
|
if profile.target_ratio is not None:
|
||||||
kwargs["target_ratio"] = profile.target_ratio
|
kwargs["target_ratio"] = profile.target_ratio
|
||||||
|
# Block-compression char floor. Every OTHER router pipeline kwarg in this
|
||||||
|
# function travels on the config object; this one alone was populated
|
||||||
|
# only from ``HEADROOM_MIN_CHARS_FOR_BLOCK`` (read below), so a proxy
|
||||||
|
# whose config carries ``savings_profile="coding"`` but whose process env
|
||||||
|
# was never seeded applied every sibling coding knob while this floor
|
||||||
|
# silently stayed at ``ContentRouterConfig.min_chars_for_block_compression``
|
||||||
|
# (500) instead of the profile's 25 — a 20x gap on the gate that governs
|
||||||
|
# tool_result blocks, the dominant content type in agent traffic.
|
||||||
|
#
|
||||||
|
# NOTE: this does not make the profile fully config-deliverable. The
|
||||||
|
# profile's ``cross_turn_dedup`` / ``tool_search`` / ``lossless_then_lossy``
|
||||||
|
# / ``protect_reads`` / ``code_aware`` / ``effort_router`` / ``lossless``
|
||||||
|
# fields are still env-only, but by a different mechanism: their consumers
|
||||||
|
# read ``os.environ`` directly (ContentRouter.__init__ for HEADROOM_DEDUPE,
|
||||||
|
# the Anthropic handler for HEADROOM_TOOL_SEARCH) and never pass through
|
||||||
|
# this function at all. Those remain seed-dependent and are the reason a
|
||||||
|
# profile can still be half-applied; fixing them means threading each
|
||||||
|
# consumer, which is a larger change than this one.
|
||||||
|
#
|
||||||
|
# The env read below still wins, since it is an explicit operator override.
|
||||||
|
if profile.min_chars_for_block is not None:
|
||||||
|
kwargs["min_chars_for_block_compression"] = profile.min_chars_for_block
|
||||||
|
|
||||||
if getattr(config, "compress_user_messages", False):
|
if getattr(config, "compress_user_messages", False):
|
||||||
kwargs["compress_user_messages"] = True
|
kwargs["compress_user_messages"] = True
|
||||||
|
|
|
||||||
|
|
@ -157,6 +157,12 @@ class PerfRecord:
|
||||||
tokens_before: int = 0
|
tokens_before: int = 0
|
||||||
tokens_after: int = 0
|
tokens_after: int = 0
|
||||||
tokens_saved: int = 0
|
tokens_saved: int = 0
|
||||||
|
# Tokens the forwarded request GREW by (PERF ``tok_inflated``). Both
|
||||||
|
# endpoints are clamped — ``tok_saved`` at zero and ``tok_inflated`` at zero
|
||||||
|
# — so a turn that left the proxy bigger reports ``tok_saved=0`` and hides
|
||||||
|
# its growth in a field nothing downstream read. Carrying it here is what
|
||||||
|
# lets the report state net alongside gross instead of implying they agree.
|
||||||
|
tokens_inflated: int = 0
|
||||||
tool_saved: int = 0
|
tool_saved: int = 0
|
||||||
cache_read: int = 0
|
cache_read: int = 0
|
||||||
cache_write: int = 0
|
cache_write: int = 0
|
||||||
|
|
@ -397,6 +403,7 @@ def parse_log_files(last_n_hours: float = 168.0) -> PerfReport:
|
||||||
tokens_before=int(kv.get("tok_before", 0)),
|
tokens_before=int(kv.get("tok_before", 0)),
|
||||||
tokens_after=int(kv.get("tok_after", 0)),
|
tokens_after=int(kv.get("tok_after", 0)),
|
||||||
tokens_saved=int(kv.get("tok_saved", 0)),
|
tokens_saved=int(kv.get("tok_saved", 0)),
|
||||||
|
tokens_inflated=int(kv.get("tok_inflated", 0)),
|
||||||
tool_saved=int(kv.get("tool_saved", 0)),
|
tool_saved=int(kv.get("tool_saved", 0)),
|
||||||
savings_breakdown=_decode_perf_savings(kv.get("savings", "none")),
|
savings_breakdown=_decode_perf_savings(kv.get("savings", "none")),
|
||||||
cache_read=int(kv.get("cache_read", 0)),
|
cache_read=int(kv.get("cache_read", 0)),
|
||||||
|
|
@ -546,6 +553,19 @@ def format_report(report: PerfReport) -> str:
|
||||||
# include tool bytes), so it used to render as a rival "Tool saved" line — which
|
# include tool bytes), so it used to render as a rival "Tool saved" line — which
|
||||||
# read as a side metric and hid the win on tool-heavy turns where tok_saved=0.
|
# read as a side metric and hid the win on tool-heavy turns where tok_saved=0.
|
||||||
lines.append(f"Tokens saved: {total_headline_saved:,} ({headline_pct:.1f}% reduction)")
|
lines.append(f"Tokens saved: {total_headline_saved:,} ({headline_pct:.1f}% reduction)")
|
||||||
|
# Gross vs net. ``tok_saved`` is clamped at zero per request, so turns
|
||||||
|
# where Headroom made the body BIGGER (CCR proactive expansion, memory
|
||||||
|
# injection) contribute nothing negative to the headline — their growth
|
||||||
|
# lands in ``tok_inflated`` instead, which nothing here used to read.
|
||||||
|
# Printing "321,239,562 -> 313,274,727" directly above "8,455,763 saved"
|
||||||
|
# implies the two reconcile; they differ by exactly the inflation. Show
|
||||||
|
# it whenever it is non-zero so the arithmetic closes on the page.
|
||||||
|
total_inflated = sum(r.tokens_inflated for r in records)
|
||||||
|
if total_inflated > 0:
|
||||||
|
lines.append(
|
||||||
|
f" · inflated {total_inflated:,} "
|
||||||
|
f"(net message reduction {total_before - total_after:,})"
|
||||||
|
)
|
||||||
if total_tool_saved > 0:
|
if total_tool_saved > 0:
|
||||||
lines.append(f" · messages {max(0, total_saved):,}")
|
lines.append(f" · messages {max(0, total_saved):,}")
|
||||||
lines.append(f" · tool schemas {total_tool_saved:,}")
|
lines.append(f" · tool schemas {total_tool_saved:,}")
|
||||||
|
|
@ -698,6 +718,31 @@ def format_report(report: PerfReport) -> str:
|
||||||
lines.append(
|
lines.append(
|
||||||
f" {name}: {avg_pct:.1f}% avg reduction, {len(recs)} uses, {total_s:,} saved"
|
f" {name}: {avg_pct:.1f}% avg reduction, {len(recs)} uses, {total_s:,} saved"
|
||||||
)
|
)
|
||||||
|
# This table is built ONLY from "Transform NAME: B -> A tokens (saved N)"
|
||||||
|
# lines, which just one engine emits (transforms/pipeline.py). The
|
||||||
|
# OpenAI-Responses engine (transforms/compression_units.py +
|
||||||
|
# compression_batches.py) applies the same strategies and contains no
|
||||||
|
# logging calls at all, so none of its work appears above. On real
|
||||||
|
# traffic that hid ~7M of ~8.5M message-token savings — the table read
|
||||||
|
# "content_router: 189,783 saved" against a PERF total 44x larger, which
|
||||||
|
# invites exactly the wrong conclusion about which compressors work.
|
||||||
|
#
|
||||||
|
# State the divergence, NOT a coverage ratio. The two totals are
|
||||||
|
# different populations and neither strictly contains the other: the
|
||||||
|
# Transform lines carry no request_id, fire once per pipeline STAGE (so
|
||||||
|
# several can describe one request), and are emitted before the forwarder
|
||||||
|
# decides anything — a mutation later discarded by the signed-thinking
|
||||||
|
# byte-lock still logs its "saved" here while the request's PERF line
|
||||||
|
# correctly reports 0. So "table covers X of Y" would be a false subset
|
||||||
|
# claim in both directions; report the two sums and let the reader judge.
|
||||||
|
table_total = sum(r.tokens_saved for r in report.transform_records)
|
||||||
|
perf_total = sum(r.tokens_saved for r in report.perf_records)
|
||||||
|
if table_total != perf_total:
|
||||||
|
lines.append(
|
||||||
|
f" ! stage-level total {table_total:,} != PERF message total {perf_total:,} "
|
||||||
|
"— this table sees only engines that emit a Transform line, counts "
|
||||||
|
"per stage, and does not check whether the mutation shipped"
|
||||||
|
)
|
||||||
lines.append("")
|
lines.append("")
|
||||||
|
|
||||||
# Router routing breakdown
|
# Router routing breakdown
|
||||||
|
|
@ -717,11 +762,24 @@ def format_report(report: PerfReport) -> str:
|
||||||
f" Excluded: {total_excluded} ({total_excluded / total_all * 100:.0f}%) — Read/Glob outputs"
|
f" Excluded: {total_excluded} ({total_excluded / total_all * 100:.0f}%) — Read/Glob outputs"
|
||||||
)
|
)
|
||||||
lines.append(
|
lines.append(
|
||||||
f" Skipped: {total_skipped} ({total_skipped / total_all * 100:.0f}%) — <50 words"
|
f" Skipped: {total_skipped} ({total_skipped / total_all * 100:.0f}%) — below size floor"
|
||||||
)
|
)
|
||||||
lines.append(
|
lines.append(
|
||||||
f" Unchanged: {total_unchanged} ({total_unchanged / total_all * 100:.0f}%) — ratio too high"
|
f" Unchanged: {total_unchanged} ({total_unchanged / total_all * 100:.0f}%) — ratio too high"
|
||||||
)
|
)
|
||||||
|
# These four buckets are NOT the router's full outcome space — the
|
||||||
|
# `[router] route_counts=` line carries 17 keys, and the ones omitted
|
||||||
|
# here (cache_hit, system_msg, error_protected, already_compressed,
|
||||||
|
# …) are individually larger than "Excluded". Percentages taken over
|
||||||
|
# this subset therefore overstate every share: on real traffic the
|
||||||
|
# "skipped" bucket read 77% here against 49.5% of actual terminal
|
||||||
|
# fates, which reads as a mis-set threshold rather than a narrow
|
||||||
|
# denominator. Say what the denominator is instead of implying it is
|
||||||
|
# everything.
|
||||||
|
lines.append(
|
||||||
|
f" (shares are of these 4 buckets only, n={total_all}; "
|
||||||
|
"see `[router] route_counts=` for the full outcome space)"
|
||||||
|
)
|
||||||
if total_excluded > total_compressed * 3:
|
if total_excluded > total_compressed * 3:
|
||||||
lines.append(" ! Excluded tools dominate — consider compressing stale Read outputs")
|
lines.append(" ! Excluded tools dominate — consider compressing stale Read outputs")
|
||||||
lines.append("")
|
lines.append("")
|
||||||
|
|
@ -803,6 +861,7 @@ PERF_RECORD_FIELDS = [
|
||||||
# Appended last so every existing CSV column keeps its position; a reader
|
# Appended last so every existing CSV column keeps its position; a reader
|
||||||
# that indexes by name is unaffected either way.
|
# that indexes by name is unaffected either way.
|
||||||
"from_response_cache",
|
"from_response_cache",
|
||||||
|
"tokens_inflated",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -768,11 +768,31 @@ class SavingsTracker:
|
||||||
delta_output_tokens_saved = max(_coerce_int(output_tokens_saved), 0)
|
delta_output_tokens_saved = max(_coerce_int(output_tokens_saved), 0)
|
||||||
delta_cache_read_tokens = _coerce_int(cache_read_tokens)
|
delta_cache_read_tokens = _coerce_int(cache_read_tokens)
|
||||||
priced = estimated_savings_usd
|
priced = estimated_savings_usd
|
||||||
delta_savings_usd = (
|
# ``estimate_request_savings_usd`` prices FOUR buckets, but this method
|
||||||
max(_coerce_float(priced.get("compression")), 0.0)
|
# only ever read three — ``tool_schema`` was computed and dropped on the
|
||||||
if priced is not None
|
# floor. Tool-schema deferral is a quarter of the token headline on real
|
||||||
else _estimate_compression_savings_usd(model, delta_tokens_saved)
|
# traffic (2.7M of 11.2M), and ``tokens_saved`` here is the bare
|
||||||
)
|
# message-level figure (the caller folds deferral in separately for the
|
||||||
|
# ledger, see prometheus_metrics.record_request), so the dollars were
|
||||||
|
# simply missing rather than counted elsewhere. That is why "Cost saved"
|
||||||
|
# read materially below the token-savings percent on the same traffic.
|
||||||
|
# Add it to the compression bucket, matching how the PERF headline and
|
||||||
|
# perf/analyzer fold deferral into one number.
|
||||||
|
if priced is not None:
|
||||||
|
# Two DISJOINT buckets: the caller passes bare message savings as
|
||||||
|
# ``compression_tokens_saved`` and deferral separately as
|
||||||
|
# ``tool_schema_tokens_saved`` (see prometheus_metrics.record_request),
|
||||||
|
# so summing them is additive, not double counting. Written as a
|
||||||
|
# statement rather than folded into the ternary below — a money path
|
||||||
|
# should not depend on the reader knowing that ``a + b if c else d``
|
||||||
|
# groups as ``(a + b) if c else d``.
|
||||||
|
delta_savings_usd = max(_coerce_float(priced.get("compression")), 0.0) + max(
|
||||||
|
_coerce_float(priced.get("tool_schema")), 0.0
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
# No priced breakdown available: only message savings are known here,
|
||||||
|
# so this path stays message-only exactly as before.
|
||||||
|
delta_savings_usd = _estimate_compression_savings_usd(model, delta_tokens_saved)
|
||||||
delta_output_savings_usd = (
|
delta_output_savings_usd = (
|
||||||
max(_coerce_float(priced.get("output_shaping")), 0.0)
|
max(_coerce_float(priced.get("output_shaping")), 0.0)
|
||||||
if priced is not None
|
if priced is not None
|
||||||
|
|
|
||||||
|
|
@ -66,7 +66,7 @@ except ImportError:
|
||||||
sys.path.insert(0, str(Path(__file__).parent.parent.parent))
|
sys.path.insert(0, str(Path(__file__).parent.parent.parent))
|
||||||
|
|
||||||
from headroom._version import __version__
|
from headroom._version import __version__
|
||||||
from headroom.agent_savings import proxy_pipeline_kwargs
|
from headroom.agent_savings import DEFAULT_PROFILE, proxy_pipeline_kwargs
|
||||||
from headroom.cache.compression_feedback import get_compression_feedback
|
from headroom.cache.compression_feedback import get_compression_feedback
|
||||||
from headroom.cache.compression_store import format_retrieval_miss_detail, get_compression_store
|
from headroom.cache.compression_store import format_retrieval_miss_detail, get_compression_store
|
||||||
from headroom.ccr import (
|
from headroom.ccr import (
|
||||||
|
|
@ -1737,6 +1737,38 @@ class HeadroomProxy(
|
||||||
if self.config.mode == PROXY_MODE_CACHE:
|
if self.config.mode == PROXY_MODE_CACHE:
|
||||||
logger.info(" Prefix freeze: strict (all prior turns immutable)")
|
logger.info(" Prefix freeze: strict (all prior turns immutable)")
|
||||||
logger.info(" Mutations: latest turn only")
|
logger.info(" Mutations: latest turn only")
|
||||||
|
# Effective compression posture, resolved (not merely requested). The
|
||||||
|
# savings profile reaches the router through two paths — the config
|
||||||
|
# object and the seeded process env — and `setdefault` semantics mean a
|
||||||
|
# stale HEADROOM_* value silently overrides the profile that names it.
|
||||||
|
# A deployment can therefore log `savings_profile=coding` while actually
|
||||||
|
# running balanced's thresholds, and the only prior evidence was a
|
||||||
|
# single easily-missed WARNING. Print what is actually in force so
|
||||||
|
# "which profile am I really running?" is answerable from the banner.
|
||||||
|
try:
|
||||||
|
_eff = proxy_pipeline_kwargs(self.config)
|
||||||
|
# Read cross-turn dedup off the CONSTRUCTED router, not off the env.
|
||||||
|
# ContentRouter resolves it as `config.enable_cross_turn_dedup OR
|
||||||
|
# $HEADROOM_DEDUPE`, so reporting the env alone would be a guess that
|
||||||
|
# happens to be right only while nothing sets the config field. A
|
||||||
|
# banner line exists to be trusted; it must read what was resolved.
|
||||||
|
_dedupe: object = "unknown"
|
||||||
|
for _t in getattr(self.anthropic_pipeline, "transforms", []):
|
||||||
|
if isinstance(_t, ContentRouter):
|
||||||
|
_dedupe = bool(getattr(_t, "_cross_turn_dedup_enabled", False))
|
||||||
|
break
|
||||||
|
logger.info(
|
||||||
|
"Savings profile: %s (effective: min_tokens=%s min_chars_block=%s "
|
||||||
|
"compress_user=%s dedupe=%s tool_search=%s)",
|
||||||
|
self.config.savings_profile or DEFAULT_PROFILE,
|
||||||
|
_eff.get("min_tokens_to_compress", "default"),
|
||||||
|
_eff.get("min_chars_for_block_compression", "default(500)"),
|
||||||
|
_eff.get("compress_user_messages", False),
|
||||||
|
_dedupe,
|
||||||
|
os.environ.get("HEADROOM_TOOL_SEARCH", "1") in ("1", "true", "yes", "on", "auto"),
|
||||||
|
)
|
||||||
|
except Exception: # never let a banner line block startup
|
||||||
|
logger.debug("effective savings-profile banner skipped", exc_info=True)
|
||||||
logger.info(f"Caching: {'ENABLED' if self.config.cache_enabled else 'DISABLED'}")
|
logger.info(f"Caching: {'ENABLED' if self.config.cache_enabled else 'DISABLED'}")
|
||||||
logger.info(f"Rate Limiting: {'ENABLED' if self.config.rate_limit_enabled else 'DISABLED'}")
|
logger.info(f"Rate Limiting: {'ENABLED' if self.config.rate_limit_enabled else 'DISABLED'}")
|
||||||
logger.info(
|
logger.info(
|
||||||
|
|
|
||||||
|
|
@ -5552,7 +5552,16 @@ class ContentRouter(Transform):
|
||||||
if route_counts["user_msg"]:
|
if route_counts["user_msg"]:
|
||||||
parts.append(f"{route_counts['user_msg']} skipped (user)")
|
parts.append(f"{route_counts['user_msg']} skipped (user)")
|
||||||
if route_counts["small"]:
|
if route_counts["small"]:
|
||||||
parts.append(f"{route_counts['small']} skipped (<50 words)")
|
# Report the thresholds actually in force, not a literal. This line
|
||||||
|
# used to read "skipped (<50 words)" unconditionally: wrong number
|
||||||
|
# (the message gate is `min_tokens`, which profiles set anywhere from
|
||||||
|
# 10 to 250), wrong unit (tokens and characters, never words), and it
|
||||||
|
# merged two different gates under one label. Operators read it as
|
||||||
|
# evidence of a mis-set threshold and tuned the wrong knob.
|
||||||
|
parts.append(
|
||||||
|
f"{route_counts['small']} skipped "
|
||||||
|
f"(<{min_tokens} tok msg / <{min_chars_for_block_compression} chars block)"
|
||||||
|
)
|
||||||
if route_counts["recent_code"]:
|
if route_counts["recent_code"]:
|
||||||
parts.append(f"{route_counts['recent_code']} protected (recent code)")
|
parts.append(f"{route_counts['recent_code']} protected (recent code)")
|
||||||
if route_counts["analysis_ctx"]:
|
if route_counts["analysis_ctx"]:
|
||||||
|
|
|
||||||
|
|
@ -10,6 +10,7 @@ from click.testing import CliRunner
|
||||||
|
|
||||||
from headroom.agent_savings import (
|
from headroom.agent_savings import (
|
||||||
AGENT_90_PROFILE,
|
AGENT_90_PROFILE,
|
||||||
|
DEFAULT_PROFILE,
|
||||||
apply_agent_savings_env_defaults,
|
apply_agent_savings_env_defaults,
|
||||||
apply_agent_savings_profile,
|
apply_agent_savings_profile,
|
||||||
get_agent_savings_profile,
|
get_agent_savings_profile,
|
||||||
|
|
@ -170,17 +171,30 @@ def test_agent_savings_env_defaults_preserve_user_overrides() -> None:
|
||||||
assert env["HEADROOM_SMART_CRUSHER_COMPACTION"] == "0"
|
assert env["HEADROOM_SMART_CRUSHER_COMPACTION"] == "0"
|
||||||
|
|
||||||
|
|
||||||
def test_unknown_agent_savings_profile_falls_back_to_balanced(
|
def test_unknown_agent_savings_profile_falls_back_to_default(
|
||||||
caplog: pytest.LogCaptureFixture,
|
caplog: pytest.LogCaptureFixture,
|
||||||
) -> None:
|
) -> None:
|
||||||
# An unknown profile must NOT raise: it's resolved during proxy startup, so
|
# An unknown profile must NOT raise: it's resolved during proxy startup, so
|
||||||
# raising takes the whole proxy down before it opens its port (desktop asked
|
# raising takes the whole proxy down before it opens its port (desktop asked
|
||||||
# for a profile a fallback runtime predates). Degrade to "balanced" instead.
|
# for a profile a fallback runtime predates).
|
||||||
|
#
|
||||||
|
# It must degrade to the DEFAULT profile, not to "balanced". The two are not
|
||||||
|
# interchangeable: balanced flips cache->token mode, turns cross-turn dedup
|
||||||
|
# and tool-search off, stops compressing user messages, and raises the
|
||||||
|
# message floor 25x (250 vs 10) and the block floor 20x (500 vs 25). A typo
|
||||||
|
# in HEADROOM_SAVINGS_PROFILE used to silently reconfigure the entire proxy
|
||||||
|
# into that posture, which is strictly worse than behaving as if the
|
||||||
|
# variable were unset.
|
||||||
with caplog.at_level(logging.WARNING):
|
with caplog.at_level(logging.WARNING):
|
||||||
profile = get_agent_savings_profile("missing")
|
profile = get_agent_savings_profile("missing")
|
||||||
assert profile is get_agent_savings_profile("balanced")
|
assert profile is get_agent_savings_profile(None)
|
||||||
|
assert profile.name == DEFAULT_PROFILE
|
||||||
|
assert profile is not get_agent_savings_profile("balanced")
|
||||||
assert "unknown savings profile" in caplog.text
|
assert "unknown savings profile" in caplog.text
|
||||||
assert "missing" in caplog.text
|
assert "missing" in caplog.text
|
||||||
|
# The warning has to name the resolved profile, so an operator reading it
|
||||||
|
# knows what they actually got rather than only what they asked for.
|
||||||
|
assert DEFAULT_PROFILE in caplog.text
|
||||||
|
|
||||||
|
|
||||||
def test_with_target_savings_recomputes_target_ratio() -> None:
|
def test_with_target_savings_recomputes_target_ratio() -> None:
|
||||||
|
|
@ -784,3 +798,45 @@ def test_agent_savings_smoke_fixture_passes_real_gate(tmp_path) -> None:
|
||||||
assert "codex: 91.0% savings meets 90.0%" in gate_result.output
|
assert "codex: 91.0% savings meets 90.0%" in gate_result.output
|
||||||
assert "cursor: 93.0% savings meets 90.0%" in gate_result.output
|
assert "cursor: 93.0% savings meets 90.0%" in gate_result.output
|
||||||
assert "100.0% accuracy meets 90.0%" in gate_result.output
|
assert "100.0% accuracy meets 90.0%" in gate_result.output
|
||||||
|
|
||||||
|
|
||||||
|
def test_coding_profile_min_chars_block_reaches_router_without_env_seeding(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
"""The block-char floor must travel on the config object, not env only.
|
||||||
|
|
||||||
|
Every other router pipeline kwarg this function builds travels on the config
|
||||||
|
object; ``min_chars_for_block`` alone was populated only from
|
||||||
|
``HEADROOM_MIN_CHARS_FOR_BLOCK`` (emitted by ``proxy_env()``). A proxy whose
|
||||||
|
config carried ``savings_profile="coding"`` but whose process env was never
|
||||||
|
seeded applied every sibling coding knob while this floor silently stayed at
|
||||||
|
``ContentRouterConfig.min_chars_for_block_compression`` (500) instead of the
|
||||||
|
profile's 25 — a 20x gap on the gate that governs tool_result blocks.
|
||||||
|
|
||||||
|
Note this does not make the profile fully config-deliverable: fields whose
|
||||||
|
consumers read ``os.environ`` directly (``cross_turn_dedup`` via
|
||||||
|
ContentRouter, ``tool_search`` via the Anthropic handler) never pass through
|
||||||
|
this function and remain seed-dependent.
|
||||||
|
"""
|
||||||
|
monkeypatch.delenv("HEADROOM_MIN_CHARS_FOR_BLOCK", raising=False)
|
||||||
|
|
||||||
|
class _Config:
|
||||||
|
savings_profile = "coding"
|
||||||
|
min_tokens_to_crush = 500
|
||||||
|
|
||||||
|
kwargs = proxy_pipeline_kwargs(_Config())
|
||||||
|
assert kwargs["min_chars_for_block_compression"] == 25
|
||||||
|
assert kwargs["min_tokens_to_compress"] == 10
|
||||||
|
|
||||||
|
|
||||||
|
def test_explicit_min_chars_block_env_overrides_the_profile(
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
) -> None:
|
||||||
|
"""An explicit operator override still wins over the profile value."""
|
||||||
|
monkeypatch.setenv("HEADROOM_MIN_CHARS_FOR_BLOCK", "120")
|
||||||
|
|
||||||
|
class _Config:
|
||||||
|
savings_profile = "coding"
|
||||||
|
min_tokens_to_crush = 500
|
||||||
|
|
||||||
|
assert proxy_pipeline_kwargs(_Config())["min_chars_for_block_compression"] == 120
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue