mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
feat(stats): per-bucket output-shaping savings in /stats-history (#1819)
## Description Adds per-bucket **output-shaping savings** to `/stats-history`. Today output-shaping savings exist only as a single global aggregate (`savings.by_layer.output_shaping`), so downstream consumers can't chart them over time. This threads a per-request output-savings estimate into the existing rollup so every `series` bucket carries `output_tokens_saved_delta` + `output_savings_usd_delta`, symmetric with the existing `compression_savings_usd_delta`. Motivation: on Claude Code subscription traffic, input is ~99% cache-discounted (the compressible live zone is a fraction of a percent), while output shaping is a ~36% reduction on full-price output tokens — so it's the dominant, honestly-attributable saving, and currently the only one a dashboard can't render per day. Closes #1816 ## Type of Change - [x] New feature (non-breaking change that adds functionality) ## Changes Made - `output_savings.py`: new read-only `SavingsRecorder.estimate_request_savings(labels, output_tokens)` → per-request synthetic-control estimate `max(0, baseline_mean(stratum) - output_tokens)` for treatment requests; 0 for control / unknown stratum / no label. Does **not** mutate the ledger, so it composes with `record_from_labels` without double-counting. `record_from_labels`'s `bool` contract is unchanged. - `outcome.py`: in the funnel, capture that estimate and pass it to `record_request(output_tokens_saved=...)`. - `savings_tracker.py`: `record_request` gains `output_tokens_saved`; accumulates lifetime cumulative `output_tokens_saved` / `output_savings_usd` (priced via new `_estimate_output_savings_usd`, output-rate), writes them into each checkpoint, and now checkpoints when **either** compression **or** output savings occurred (so output-only requests aren't dropped). `_build_rollup` diffs the cumulative into `output_tokens_saved_delta` / `output_savings_usd_delta` per bucket; `_normalize_history_entry` and the CSV export carry the fields. - Additive + backward-compatible: checkpoints predating the feature default the new fields to 0. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --extra dev pytest tests/test_output_shaping_rollup.py tests/test_output_savings.py \ tests/test_output_savings_cli.py tests/test_proxy_savings_history.py tests/test_request_outcome.py -q ... 103 passed $ uv run --extra dev ruff check headroom/proxy/savings_tracker.py headroom/proxy/output_savings.py \ headroom/proxy/prometheus_metrics.py headroom/proxy/outcome.py tests/test_output_shaping_rollup.py All checks passed! $ uv run --extra dev mypy headroom/proxy/savings_tracker.py headroom/proxy/output_savings.py Success: no issues found in 2 source files ``` New tests (`tests/test_output_shaping_rollup.py`): output savings bucket into the daily series; an output-only request (no compression) still checkpoints; pre-feature requests default to 0; `estimate_request_savings` returns the baseline-relative saving for treatment and 0 for control / unknown / over-baseline. ## Real Behavior Proof - Environment: macOS, CPython 3.10.18, this branch (rebased on latest `main`), litellm pricing available. - Exact command / steps: seed a baseline (as `learn --verbosity` would), then drive 3 requests through the real, unmocked chain `SavingsRecorder.estimate_request_savings` → `SavingsTracker.record_request` → `history_response()`, and print `series.daily`. Full script + raw output: ```text $ uv run python proof.py # seeds baseline ~1000 out-tok; 3 treatment requests (out=600/550/700), one with no compression [ { "timestamp": "2026-07-05T00:00:00Z", "tokens_saved": 120, "compression_savings_usd_delta": 0.0006, "output_tokens_saved_delta": 850, "output_savings_usd_delta": 0.02125 }, { "timestamp": "2026-07-06T00:00:00Z", "tokens_saved": 80, "compression_savings_usd_delta": 0.0004, "output_tokens_saved_delta": 300, "output_savings_usd_delta": 0.0075 } ] ``` - Observed result: output-shaping savings appear per day and independent of the compression axis. 2026-07-05 = 850 (400+450 saved by two treatment requests vs the ~1000-token baseline, including one request with zero compression — proving the output-only checkpoint path), 2026-07-06 = 300, each priced at the model's output rate. Matches expectations. - Not tested: the full live proxy over HTTP with a real learned baseline and organic traffic — I exercised the same code path minus the HTTP/streaming layer. The measured-vs-estimated `method` gating is unchanged by this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — backend-only change (no UI surface in this repo). The runtime effect is the `/stats-history` `series.daily` JSON with the new `output_tokens_saved_delta` / `output_savings_usd_delta` fields, shown under **Real Behavior Proof** above. The downstream chart that renders them lives in the separate Headroom desktop app. ## Additional Notes - Per CONTRIBUTING's issue-first policy for features, I opened #1816 first with the spec; happy to adjust the API surface (field names / gating) to whatever you prefer. A downstream consumer (Headroom desktop chart) is already implemented against this exact contract and stacks the segment only when `output_reduction.method == "measured"`. - Docs checkbox left unchecked: I didn't find a `/stats-history` schema doc to update; point me at one if it exists. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
gglucass
GitHub
parent
dec60de976
commit
12a9710665
7 changed files with 190 additions and 4 deletions
|
|
@ -111,6 +111,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
* **transforms:** first-class C# support in `CodeAwareCompressor` via the tree-sitter `csharp` grammar already shipped in the pinned `tree-sitter-language-pack` — no new dependencies ([#1664](https://github.com/headroomlabs-ai/headroom/issues/1664)). Parity with Java/C++/Rust: signatures preserved verbatim, method/constructor/destructor/operator/local-function bodies compressed; block-scoped and file-scoped namespaces, records, structs, interfaces, and enums handled; C#-distinctive auto-detection. Preprocessor conditionals (`#if`…`#endif`) are preserved verbatim as opaque regions (blocks wrapping only `using` directives stay with the imports), `#region` markers no longer swallow the following line during class-member extraction, and top-of-file license banners / `#region License` headers stay on top instead of being relocated below the code. Real-repo runs: 16.1% tokens saved on Newtonsoft.Json (945 files), 37.8% on Polly (797 files), output syntax-valid for 1742/1742 files.
|
||||
* **proxy:** add provider-only HTTP proxy routing via `--http-proxy` and `HEADROOM_HTTP_PROXY`. Upstream LLM provider calls can now use an HTTP proxy without setting process-wide `HTTP_PROXY`/`HTTPS_PROXY` variables that are inherited by tool executions; proxied provider clients use HTTP/1.1 so HTTPS provider APIs can tunnel through CONNECT.
|
||||
* **proxy:** add output shaping for OpenAI Responses traffic on `/v1/responses` HTTP requests and Codex WebSocket `response.create` frames, with stable output-savings holdout keys and counted WS token strata for the experiment.
|
||||
* **stats:** per-bucket output-shaping savings in `/stats-history`. Each `series` bucket (hourly/daily/weekly/monthly) now carries `output_tokens_saved_delta` and `output_savings_usd_delta` alongside the existing compression deltas, sourced from a per-request synthetic-control estimate (`SavingsRecorder.estimate_request_savings`) threaded through `record_request` into the rollup. Lets dashboards chart output-shaping savings over time as a distinct series — previously it existed only as a single global aggregate. Additive and backward-compatible: pre-feature checkpoints default the new fields to 0 ([#1816](https://github.com/headroomlabs-ai/headroom/issues/1816)).
|
||||
* **observability:** the `headroom.compression.pipeline` span now also carries the OpenTelemetry GenAI semantic-convention attribute `gen_ai.request.model` alongside the existing `headroom.*` attributes, so Headroom's traces group and filter by the standard `gen_ai.*` schema in any OTel-native backend (Grafana, Datadog, etc.). Purely additive; no existing attribute changed. `gen_ai.operation.name`, `gen_ai.provider.name`, and `gen_ai.usage.*` are deliberately deferred (they need per-caller operation threading, reliable upstream-provider resolution, and response-path usage respectively).
|
||||
* **wrap:** `headroom wrap claude --1m` preserves the 1M context window. Behind a custom `ANTHROPIC_BASE_URL` (the proxy) Claude Code drops the `context-1m` beta header and caps the window at 200k for entitled subscription users; the opt-in flag sets `ANTHROPIC_MODEL=<opus>[1m]` on the launched process so the 1M window activates through Headroom. A model already selected via `ANTHROPIC_MODEL` is preserved (only the `[1m]` suffix is appended) ([#1158](https://github.com/chopratejas/headroom/issues/1158)).
|
||||
* **learn:** weight loops in `headroom learn`. A new loop detector (`headroom/learn/loops.py`) recognizes repeated tool-call patterns — including RTK re-fetch loops, where RTK's output truncation makes the agent re-run larger-limit variants of a *successful* command — collapses output-limit variants to one signature, measures the wasted tokens, surfaces loops as a highest-priority digest section, and weights loop guardrails above one-off rules by their measured waste. Previously loops had no special weight and a no-failure re-fetch loop was skipped entirely. Adds an RTK-loop eval (`benchmarks/rtk_loop_learn_eval.py`) that reproduces a loop, runs it through Learn, and asserts the generated guardrail ranks first and prevents re-triggering.
|
||||
|
|
|
|||
|
|
@ -357,11 +357,16 @@ async def emit_request_outcome(handler: Any, outcome: RequestOutcome) -> None:
|
|||
# tags each request's (arm, stratum) onto ``transforms_applied``; feed the
|
||||
# observed output tokens to the recorder so it can produce an honest
|
||||
# reduction estimate. Best-effort: never let bookkeeping break a response.
|
||||
output_tokens_saved_est = 0
|
||||
if any(str(t).startswith("output_shaper:") for t in outcome.transforms_applied):
|
||||
try:
|
||||
from headroom.proxy.output_savings import get_recorder
|
||||
|
||||
get_recorder().record_from_labels(outcome.transforms_applied, outcome.output_tokens)
|
||||
_rec = get_recorder()
|
||||
_rec.record_from_labels(outcome.transforms_applied, outcome.output_tokens)
|
||||
output_tokens_saved_est = _rec.estimate_request_savings(
|
||||
outcome.transforms_applied, outcome.output_tokens
|
||||
)
|
||||
except Exception: # pragma: no cover - defensive
|
||||
pass
|
||||
|
||||
|
|
@ -388,6 +393,7 @@ async def emit_request_outcome(handler: Any, outcome: RequestOutcome) -> None:
|
|||
cache_write_1h_tokens=outcome.cache_write_1h_tokens,
|
||||
uncached_input_tokens=outcome.uncached_input_tokens,
|
||||
attempted_input_tokens=outcome.attempted_input_tokens,
|
||||
output_tokens_saved=output_tokens_saved_est,
|
||||
project=project,
|
||||
client=outcome.client,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -387,6 +387,26 @@ class SavingsRecorder:
|
|||
return True
|
||||
return False
|
||||
|
||||
def estimate_request_savings(self, labels: Any, output_tokens: int) -> int:
|
||||
"""Per-request output tokens saved, for the savings rollup.
|
||||
|
||||
For a treatment request, the synthetic-control estimate
|
||||
``max(0, baseline_mean(stratum) - output_tokens)``; 0 for control,
|
||||
unknown strata, or when no shaping label is present. Read-only:
|
||||
unlike ``record_from_labels`` it does not mutate the ledger, so the
|
||||
two compose without double-counting."""
|
||||
for label in labels or ():
|
||||
parsed = parse_stratum_label(str(label))
|
||||
if parsed is None:
|
||||
continue
|
||||
arm, key = parsed
|
||||
if arm != "treatment":
|
||||
return 0
|
||||
with self._lock:
|
||||
mean, _var, n = self._ledger.baseline.lookup(key)
|
||||
return max(0, int(round(mean - output_tokens))) if n > 0 else 0
|
||||
return 0
|
||||
|
||||
def _reload_baseline_locked(self) -> None:
|
||||
"""Adopt the on-disk baseline written by ``learn --verbosity --apply``.
|
||||
|
||||
|
|
|
|||
|
|
@ -637,6 +637,7 @@ class PrometheusMetrics:
|
|||
cache_write_1h_tokens: int = 0,
|
||||
uncached_input_tokens: int = 0,
|
||||
attempted_input_tokens: int = 0,
|
||||
output_tokens_saved: int = 0,
|
||||
project: str | None = None,
|
||||
client: str | None = None,
|
||||
):
|
||||
|
|
@ -763,6 +764,7 @@ class PrometheusMetrics:
|
|||
uncached_input_tokens=uncached_input_tokens,
|
||||
total_input_tokens=total_input_tokens,
|
||||
total_input_cost_usd=total_input_cost_usd,
|
||||
output_tokens_saved=output_tokens_saved,
|
||||
)
|
||||
|
||||
# Also append to the durable, multi-process savings ledger so
|
||||
|
|
|
|||
|
|
@ -39,6 +39,8 @@ DEFAULT_MAX_HISTORY_AGE_DAYS = 365
|
|||
DEFAULT_MAX_RESPONSE_HISTORY_POINTS = 500
|
||||
DEFAULT_DISPLAY_SESSION_INACTIVITY_MINUTES = 60
|
||||
DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN = 3.0 / 1_000_000
|
||||
# Blended output price used only when litellm cannot price the model.
|
||||
DEFAULT_FALLBACK_OUTPUT_COST_PER_TOKEN = 15.0 / 1_000_000
|
||||
|
||||
LITELLM_AVAILABLE = importlib.util.find_spec("litellm") is not None
|
||||
litellm: Any | None = None
|
||||
|
|
@ -221,6 +223,28 @@ def _estimate_compression_savings_usd(model: str, tokens_saved: int) -> float:
|
|||
return float(tokens_saved) * float(DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN)
|
||||
|
||||
|
||||
def _estimate_output_savings_usd(model: str, tokens_saved: int) -> float:
|
||||
"""Estimate output-shaping savings in USD from saved *output* tokens.
|
||||
|
||||
Mirrors ``_estimate_compression_savings_usd`` but prices at the model's
|
||||
output rate, since the shaper reduces generated (output) tokens, not input.
|
||||
"""
|
||||
litellm = _get_litellm_module()
|
||||
if tokens_saved <= 0:
|
||||
return 0.0
|
||||
if litellm is None:
|
||||
return float(tokens_saved) * float(DEFAULT_FALLBACK_OUTPUT_COST_PER_TOKEN)
|
||||
try:
|
||||
resolved = _resolve_litellm_model(model)
|
||||
info = litellm.model_cost.get(resolved, {})
|
||||
output_cost_per_token = info.get("output_cost_per_token")
|
||||
if not output_cost_per_token:
|
||||
raise RuntimeError("output cost unavailable")
|
||||
return float(tokens_saved) * float(output_cost_per_token)
|
||||
except Exception:
|
||||
return float(tokens_saved) * float(DEFAULT_FALLBACK_OUTPUT_COST_PER_TOKEN)
|
||||
|
||||
|
||||
def _estimate_cache_savings_usd(model: str, cache_read_tokens: int) -> float:
|
||||
"""Estimate cache-read savings in USD — the discount delta vs list price.
|
||||
|
||||
|
|
@ -329,6 +353,8 @@ def _normalize_history_entry(entry: Any) -> dict[str, Any] | None:
|
|||
cache_savings_usd = 0.0
|
||||
total_input_tokens = 0
|
||||
total_input_cost_usd = 0.0
|
||||
output_tokens_saved = 0
|
||||
output_savings_usd = 0.0
|
||||
provider = PROVIDER_UNKNOWN
|
||||
model = MODEL_UNKNOWN
|
||||
|
||||
|
|
@ -343,6 +369,8 @@ def _normalize_history_entry(entry: Any) -> dict[str, Any] | None:
|
|||
cache_savings_usd = _coerce_float(entry.get("cache_savings_usd"))
|
||||
total_input_tokens = _coerce_int(entry.get("total_input_tokens"))
|
||||
total_input_cost_usd = _coerce_float(entry.get("total_input_cost_usd"))
|
||||
output_tokens_saved = _coerce_int(entry.get("output_tokens_saved"))
|
||||
output_savings_usd = _coerce_float(entry.get("output_savings_usd"))
|
||||
provider = _normalize_provider(entry.get("provider"))
|
||||
model = _normalize_model(entry.get("model"))
|
||||
elif isinstance(entry, list | tuple) and len(entry) >= 2:
|
||||
|
|
@ -370,6 +398,8 @@ def _normalize_history_entry(entry: Any) -> dict[str, Any] | None:
|
|||
"cache_savings_usd": round(cache_savings_usd, 6),
|
||||
"total_input_tokens": total_input_tokens,
|
||||
"total_input_cost_usd": round(total_input_cost_usd, 6),
|
||||
"output_tokens_saved": output_tokens_saved,
|
||||
"output_savings_usd": round(output_savings_usd, 6),
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -634,6 +664,7 @@ class SavingsTracker:
|
|||
model: str,
|
||||
input_tokens: int,
|
||||
tokens_saved: int,
|
||||
output_tokens_saved: int = 0,
|
||||
provider: str | None = None,
|
||||
project: str | None = None,
|
||||
cache_read_tokens: int = 0,
|
||||
|
|
@ -657,6 +688,8 @@ class SavingsTracker:
|
|||
delta_tokens_saved = _coerce_int(tokens_saved)
|
||||
delta_input_tokens = _coerce_int(input_tokens)
|
||||
delta_savings_usd = _estimate_compression_savings_usd(model, delta_tokens_saved)
|
||||
delta_output_tokens_saved = max(_coerce_int(output_tokens_saved), 0)
|
||||
delta_output_savings_usd = _estimate_output_savings_usd(model, delta_output_tokens_saved)
|
||||
delta_cache_read_tokens = _coerce_int(cache_read_tokens)
|
||||
delta_cache_savings_usd = _estimate_cache_savings_usd(model, delta_cache_read_tokens)
|
||||
delta_input_cost_usd = _estimate_input_cost_usd(
|
||||
|
|
@ -711,6 +744,13 @@ class SavingsTracker:
|
|||
)
|
||||
lifetime["total_input_tokens"] = next_total_input_tokens
|
||||
lifetime["total_input_cost_usd"] = next_total_input_cost_usd
|
||||
lifetime["output_tokens_saved"] = (
|
||||
lifetime.get("output_tokens_saved", 0) + delta_output_tokens_saved
|
||||
)
|
||||
lifetime["output_savings_usd"] = round(
|
||||
lifetime.get("output_savings_usd", 0.0) + delta_output_savings_usd,
|
||||
6,
|
||||
)
|
||||
|
||||
session = self._state["display_session"]
|
||||
last_activity = _parse_timestamp(session.get("last_activity_at"))
|
||||
|
|
@ -771,8 +811,12 @@ class SavingsTracker:
|
|||
# not lossy-compressed, to keep Bedrock's prompt cache warm. Gating
|
||||
# on tokens_saved alone silently dropped every history point on
|
||||
# those requests even though real cache-read savings occurred.
|
||||
# Append whenever either mechanism produced a saving.
|
||||
if delta_tokens_saved > 0 or delta_cache_read_tokens > 0:
|
||||
# Append whenever any savings mechanism produced a saving.
|
||||
if (
|
||||
delta_tokens_saved > 0
|
||||
or delta_cache_read_tokens > 0
|
||||
or delta_output_tokens_saved > 0
|
||||
):
|
||||
self._state["history"].append(
|
||||
{
|
||||
"timestamp": _to_utc_iso(timestamp_dt),
|
||||
|
|
@ -784,6 +828,8 @@ class SavingsTracker:
|
|||
"cache_savings_usd": lifetime["cache_savings_usd"],
|
||||
"total_input_tokens": lifetime["total_input_tokens"],
|
||||
"total_input_cost_usd": lifetime["total_input_cost_usd"],
|
||||
"output_tokens_saved": lifetime.get("output_tokens_saved", 0),
|
||||
"output_savings_usd": lifetime.get("output_savings_usd", 0.0),
|
||||
}
|
||||
)
|
||||
self._trim_history_locked(reference_time=timestamp_dt)
|
||||
|
|
@ -1059,6 +1105,8 @@ class SavingsTracker:
|
|||
"total_input_tokens",
|
||||
"total_input_cost_usd_delta",
|
||||
"total_input_cost_usd",
|
||||
"output_tokens_saved_delta",
|
||||
"output_savings_usd_delta",
|
||||
]
|
||||
|
||||
buffer = StringIO()
|
||||
|
|
@ -1478,6 +1526,8 @@ class SavingsTracker:
|
|||
prev_total_usd = 0.0
|
||||
prev_total_input_tokens = 0
|
||||
prev_total_input_cost_usd = 0.0
|
||||
prev_output_tokens = 0
|
||||
prev_output_usd = 0.0
|
||||
|
||||
for point in history:
|
||||
timestamp = _parse_timestamp(point["timestamp"])
|
||||
|
|
@ -1491,6 +1541,8 @@ class SavingsTracker:
|
|||
total_usd = _coerce_float(point.get("compression_savings_usd"))
|
||||
total_input_tokens = _coerce_int(point.get("total_input_tokens"))
|
||||
total_input_cost_usd = _coerce_float(point.get("total_input_cost_usd"))
|
||||
total_output_tokens = _coerce_int(point.get("output_tokens_saved"))
|
||||
total_output_usd = _coerce_float(point.get("output_savings_usd"))
|
||||
delta_tokens = max(total_tokens_saved - prev_total_tokens, 0)
|
||||
delta_usd = max(total_usd - prev_total_usd, 0.0)
|
||||
delta_input_tokens = max(total_input_tokens - prev_total_input_tokens, 0)
|
||||
|
|
@ -1499,10 +1551,15 @@ class SavingsTracker:
|
|||
0.0,
|
||||
)
|
||||
|
||||
delta_output_tokens = max(total_output_tokens - prev_output_tokens, 0)
|
||||
delta_output_usd = max(total_output_usd - prev_output_usd, 0.0)
|
||||
|
||||
prev_total_tokens = total_tokens_saved
|
||||
prev_total_usd = total_usd
|
||||
prev_total_input_tokens = total_input_tokens
|
||||
prev_total_input_cost_usd = total_input_cost_usd
|
||||
prev_output_tokens = total_output_tokens
|
||||
prev_output_usd = total_output_usd
|
||||
|
||||
entry = aggregated.setdefault(
|
||||
bucket_key,
|
||||
|
|
@ -1516,6 +1573,8 @@ class SavingsTracker:
|
|||
"total_input_tokens": total_input_tokens,
|
||||
"total_input_cost_usd_delta": 0.0,
|
||||
"total_input_cost_usd": total_input_cost_usd,
|
||||
"output_tokens_saved_delta": 0,
|
||||
"output_savings_usd_delta": 0.0,
|
||||
"by_provider": {},
|
||||
"by_model": {},
|
||||
},
|
||||
|
|
@ -1534,6 +1593,11 @@ class SavingsTracker:
|
|||
entry["compression_savings_usd"] = round(total_usd, 6)
|
||||
entry["total_input_tokens"] = total_input_tokens
|
||||
entry["total_input_cost_usd"] = round(total_input_cost_usd, 6)
|
||||
entry["output_tokens_saved_delta"] += delta_output_tokens
|
||||
entry["output_savings_usd_delta"] = round(
|
||||
entry["output_savings_usd_delta"] + delta_output_usd,
|
||||
6,
|
||||
)
|
||||
|
||||
# Attribute this checkpoint's delta to the provider that produced
|
||||
# it. Each checkpoint comes from a single request, so its delta is
|
||||
|
|
|
|||
88
tests/test_output_shaping_rollup.py
Normal file
88
tests/test_output_shaping_rollup.py
Normal file
|
|
@ -0,0 +1,88 @@
|
|||
"""Per-bucket output-shaping savings in the /stats-history rollup.
|
||||
|
||||
Covers the feature that lets a downstream dashboard stack output-shaping
|
||||
savings as a distinct daily segment: SavingsTracker.record_request accepts a
|
||||
per-request output_tokens_saved, accumulates it into each time bucket as
|
||||
output_tokens_saved_delta / output_savings_usd_delta, and the read-only
|
||||
SavingsRecorder.estimate_request_savings supplies that per-request number.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from headroom.proxy.output_savings import (
|
||||
SavingsRecorder,
|
||||
stratum_key,
|
||||
stratum_label,
|
||||
)
|
||||
from headroom.proxy.savings_tracker import SavingsTracker
|
||||
|
||||
|
||||
def test_record_request_buckets_output_shaping_savings(tmp_path):
|
||||
tracker = SavingsTracker(path=str(tmp_path / "s.json"))
|
||||
|
||||
# Request with both compression and output-shaping savings.
|
||||
tracker.record_request(
|
||||
model="claude-opus-4-8",
|
||||
input_tokens=1000,
|
||||
tokens_saved=100,
|
||||
output_tokens_saved=5000,
|
||||
timestamp="2026-03-27T09:00:00Z",
|
||||
)
|
||||
# Output-shaping-ONLY request (no compression) must still checkpoint, else
|
||||
# its output savings would be dropped from the rollup.
|
||||
tracker.record_request(
|
||||
model="claude-opus-4-8",
|
||||
input_tokens=1000,
|
||||
tokens_saved=0,
|
||||
output_tokens_saved=3000,
|
||||
timestamp="2026-03-27T09:30:00Z",
|
||||
)
|
||||
|
||||
daily = tracker.history_response()["series"]["daily"]
|
||||
assert len(daily) == 1
|
||||
assert daily[0]["output_tokens_saved_delta"] == 8000
|
||||
assert daily[0]["output_savings_usd_delta"] > 0.0
|
||||
# Compression axis stays independent.
|
||||
assert daily[0]["tokens_saved"] == 100
|
||||
|
||||
|
||||
def test_record_request_without_output_savings_is_backward_compatible(tmp_path):
|
||||
tracker = SavingsTracker(path=str(tmp_path / "s.json"))
|
||||
tracker.record_request(
|
||||
model="gpt-4o",
|
||||
input_tokens=8192,
|
||||
tokens_saved=4096,
|
||||
timestamp="2026-03-27T09:00:00Z",
|
||||
)
|
||||
daily = tracker.history_response()["series"]["daily"]
|
||||
assert daily[0]["output_tokens_saved_delta"] == 0
|
||||
assert daily[0]["output_savings_usd_delta"] == 0.0
|
||||
|
||||
|
||||
def _key() -> str:
|
||||
return stratum_key(turn_kind="code", input_tokens=8000, model="claude-opus-4-8", has_tools=True)
|
||||
|
||||
|
||||
def test_estimate_request_savings_treatment_uses_baseline(tmp_path):
|
||||
rec = SavingsRecorder(str(tmp_path / "o.json"), flush_every=1)
|
||||
key = _key()
|
||||
for _ in range(5):
|
||||
rec._ledger.baseline.observe(key, 1000) # baseline mean ~1000
|
||||
|
||||
# Treatment request that emitted 600 -> saved ~400 vs the baseline.
|
||||
saved = rec.estimate_request_savings([stratum_label("treatment", key)], 600)
|
||||
assert saved == 400
|
||||
|
||||
|
||||
def test_estimate_request_savings_zero_for_control_and_unknown(tmp_path):
|
||||
rec = SavingsRecorder(str(tmp_path / "o.json"), flush_every=1)
|
||||
key = _key()
|
||||
for _ in range(5):
|
||||
rec._ledger.baseline.observe(key, 1000)
|
||||
|
||||
# Control arm is unshaped -> no attributable saving.
|
||||
assert rec.estimate_request_savings([stratum_label("control", key)], 600) == 0
|
||||
# No shaping label at all.
|
||||
assert rec.estimate_request_savings(["something-else"], 600) == 0
|
||||
# Treatment but output exceeded the baseline -> clamped to 0, never negative.
|
||||
assert rec.estimate_request_savings([stratum_label("treatment", key)], 5000) == 0
|
||||
|
|
@ -82,6 +82,8 @@ def test_savings_tracker_helpers_normalize_inputs_and_paths(tmp_path, monkeypatc
|
|||
"cache_savings_usd": 0.0,
|
||||
"total_input_tokens": 0,
|
||||
"total_input_cost_usd": 0.0,
|
||||
"output_tokens_saved": 0,
|
||||
"output_savings_usd": 0.0,
|
||||
}
|
||||
assert savings_tracker_module._normalize_history_entry({"timestamp": "bad"}) is None
|
||||
assert savings_tracker_module._normalize_history_entry(object()) is None
|
||||
|
|
@ -145,6 +147,8 @@ def test_savings_tracker_sanitizes_legacy_state_and_applies_retention(tmp_path):
|
|||
"cache_savings_usd": 0.0,
|
||||
"total_input_tokens": 0,
|
||||
"total_input_cost_usd": 0.0,
|
||||
"output_tokens_saved": 0,
|
||||
"output_savings_usd": 0.0,
|
||||
}
|
||||
]
|
||||
assert snapshot["retention"] == {
|
||||
|
|
@ -1288,7 +1292,8 @@ def test_stats_history_csv_export_is_frontend_friendly(tmp_path, monkeypatch):
|
|||
assert lines[0] == (
|
||||
"timestamp,tokens_saved,compression_savings_usd_delta,total_tokens_saved,"
|
||||
"compression_savings_usd,total_input_tokens_delta,total_input_tokens,"
|
||||
"total_input_cost_usd_delta,total_input_cost_usd"
|
||||
"total_input_cost_usd_delta,total_input_cost_usd,"
|
||||
"output_tokens_saved_delta,output_savings_usd_delta"
|
||||
)
|
||||
assert len(lines) >= 2
|
||||
assert "total_tokens_saved" in lines[0]
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue