mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
6 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Abhay Singh
|
b4f807f21a
|
fix(proxy/cost): price cache savings by most-used model, not first-seen (#2023)
## Description
`build_prefix_cache_stats` (`headroom/proxy/cost.py`) values each
provider's cache-read savings
using a single "base input price per token". It derives that price by
scanning
`cost_tracker._tokens_sent_by_model` and **breaking on the first**
provider-matching model that
has a price — even though the comment says "most-used model":
```python
# Get the base input price per token for the most-used model on this provider
input_price_per_token = None
if cost_tracker:
for model_name in cost_tracker._tokens_sent_by_model: # insertion order, NOT usage order
...
if is_match:
price_per_1m = cost_tracker._get_list_price(model_name)
if price_per_1m:
input_price_per_token = price_per_1m / 1_000_000
break # first match wins
```
`_tokens_sent_by_model` is insertion-ordered, so the price used depends
on which model was
*recorded first*, not on usage volume. A Claude Code session sends both
Sonnet (main loop) and
Haiku (titles/subagents). If Haiku ($0.80/M) was seen before Sonnet
($3/M), **all** of the
provider's cache-read savings are priced at Haiku's rate — understating
the dashboard's cache
savings by ~3.75×. Reverse the order and it overstates.
Closes: no issue filed — found while auditing the cache-savings pricing.
## Fix
Pick the provider-matching, priced model with the **highest token
volume** instead of breaking
on the first match:
```python
best_tokens = -1
for model_name, tokens_sent in cost_tracker._tokens_sent_by_model.items():
if is_match and tokens_sent > best_tokens:
price_per_1m = cost_tracker._get_list_price(model_name)
if price_per_1m:
input_price_per_token = price_per_1m / 1_000_000
best_tokens = tokens_sent
```
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/proxy/cost.py`: select the highest-volume provider-matching
model (with a known price) rather than the first-recorded one.
- `tests/test_proxy_cache_ttl_metrics.py`: add
`test_prefix_cache_stats_prices_by_most_used_model` using real distinct
per-model prices. (The existing cache-stats tests monkeypatch
`_get_list_price` to a constant `100.0`, which masked the
model-selection logic — hence the bug slipped through.)
## Testing
- [x] New regression test added
(`tests/test_proxy_cache_ttl_metrics.py`)
- [x] Linting/formatting clean — run with the CI-pinned `ruff==0.15.17`
- [ ] Full `pytest` deferred to CI (local-OOM reason below).
```text
$ uvx ruff@0.15.17 check headroom/proxy/cost.py tests/test_proxy_cache_ttl_metrics.py
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.10, headroom from this branch.
Importing `headroom` pulls in the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I verified the selection logic
with a dependency-free script and left the full pytest to CI.
- Exact command / steps: ran a `{haiku: 500, sonnet: 50000}` token map
(Haiku recorded first, Sonnet the higher volume) through both the old
first-match and new highest-volume selection with real prices.
- Observed result: the old logic picks Haiku's $0.80/M (first-inserted);
the new logic picks Sonnet's $3/M (highest volume) and is
insertion-order independent:
```text
OLD picks Haiku price: 0.80/M (first-inserted)
NEW picks Sonnet price: 3.00/M (highest volume)
-> old understates the input price by 3.75x (3.75x)
NEW is insertion-order independent
COST MOST-USED-MODEL FIX VERIFIED
```
- Not tested: rendering the live dashboard (needs the running app). The
fix is confined to the price-selection loop and the new test drives
`build_prefix_cache_stats` directly. Full local `pytest` deferred to CI
(OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — ran
lint + a standalone logic check; full pytest deferred to CI (local OOM,
disclosed above)
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- No new dependencies; a single-loop change plus a test with realistic
prices.
- @JerrettDavis tagging you — this skews the dashboard's per-provider
cache-savings dollar figure by the ratio between a provider's models
(≈3.75× for Sonnet/Haiku), so it seemed worth surfacing. Thanks!
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Rod Boev
|
53a465b121
|
fix(proxy): subtract cache write premiums from net savings (#1800)
## Description Cache stats already calculate both prompt-cache read savings and cache-write premium cost, but the exported `net_savings_usd` field used gross read savings alone. That made cache-heavy token-mode workloads look profitable even when extra cache writes offset or exceeded the read discount. This updates existing cache cost accounting so provider and total `net_savings_usd` subtract write premiums while keeping gross savings and write premium fields visible. Refs #327. The scope follows doublefx's controlled measurement in https://github.com/headroomlabs-ai/headroom/issues/327#issuecomment-4683604089, which showed token-mode compression increasing cache write volume and billed cost while dashboard token savings looked positive. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Subtract cache write premiums from provider-level cache `net_savings_usd`. - Subtract aggregate cache write premiums from total cache `net_savings_usd`. - Keep gross `savings_usd` and `write_premium_usd` visible for dashboard and telemetry consumers. - Add focused regressions for provider net, total net, and zero-write-premium preservation. - Update the dashboard cache TTL fixture to match the corrected net value. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_proxy_cache_ttl_metrics.py tests/test_dashboard_cache_ttl_playwright.py tests/test_proxy_dashboard_stats_cache.py -q`) - [x] Linting passes (`uv run ruff check headroom/proxy/cost.py tests/test_proxy_cache_ttl_metrics.py tests/test_dashboard_cache_ttl_playwright.py tests/test_proxy_dashboard_stats_cache.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text uv run pytest tests/test_proxy_cache_ttl_metrics.py tests/test_dashboard_cache_ttl_playwright.py tests/test_proxy_dashboard_stats_cache.py -q 28 passed, 2 skipped, 1 warning in 32.75s uv run pytest tests/test_proxy_cache_ttl_metrics.py -q -k keeps_net_equal_without_write_premium 1 passed, 16 deselected in 0.15s uv run ruff check headroom/proxy/cost.py tests/test_proxy_cache_ttl_metrics.py tests/test_dashboard_cache_ttl_playwright.py tests/test_proxy_dashboard_stats_cache.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows, Python through the project `uv` environment. - Exact command / steps: run the cache net-savings regressions against base and head. - Observed result: base reports provider net as `0.0036` instead of `0.0021` and total net as `0.0046` instead of `0.0031`; head passes the focused cache metrics suite and preserves `net_savings_usd == savings_usd` when there is no write premium. - Not tested: broader cache-hit-rate tuning, prompt-cache policy changes, and live provider billing. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes No changelog entry is needed because this corrects existing stats fields rather than adding a new command or control. Type checking was not part of the focused local validation for this Python-only fix. Dashboard Playwright coverage is CI-owned locally; the import-gated file was included in the focused pytest command and skipped because Playwright is not installed in this environment. |
||
|
Vinay Gupta
|
5fe4e7b195
|
fix(proxy): expose persistent savings metrics (#1647)
## Description Closes #1616 Expose the proxy's durable `persistent_savings.lifetime` totals through `/metrics` so Prometheus/Grafana scrapes can read the same lifetime savings counters already visible in `/stats` and `/stats-history`. The existing runtime counters remain process-local: `headroom_tokens_saved_total` still resets with the proxy process. New `headroom_persistent_savings_*` counters are sourced from the `SavingsTracker` lifetime block. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Export durable lifetime savings counters from `PrometheusMetrics.export()`: - `headroom_persistent_savings_requests_total` - `headroom_persistent_savings_tokens_saved_total` - `headroom_persistent_savings_input_tokens_total` - `headroom_persistent_savings_input_cost_usd_total` - `headroom_persistent_savings_compression_savings_usd_total` - Add a restart regression proving runtime counters reset while persistent savings counters remain available from the same savings file. - Extend the existing `/stats-history` restart test with `/metrics` endpoint assertions. - Update metrics docs to distinguish runtime `headroom_tokens_saved_total` from lifetime `headroom_persistent_savings_tokens_saved_total`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text Local focused checks: $ rtk /usr/bin/env HEADROOM_REQUIRE_RUST_CORE=false PYTHONPATH=. /tmp/headroom-1616-testenv/bin/python -m pytest tests/test_proxy_cache_ttl_metrics.py::test_prometheus_metrics_export_includes_extended_fields tests/test_proxy_cache_ttl_metrics.py::test_prometheus_export_includes_persistent_savings_after_restart 2 passed, 1 warning in 0.19s $ rtk /tmp/headroom-1616-testenv/bin/python -m ruff check headroom/proxy/prometheus_metrics.py tests/test_proxy_cache_ttl_metrics.py tests/test_proxy_savings_history.py All checks passed! $ rtk /tmp/headroom-1616-testenv/bin/python -m ruff format --check headroom/proxy/prometheus_metrics.py tests/test_proxy_cache_ttl_metrics.py tests/test_proxy_savings_history.py 3 files already formatted $ rtk git diff --check # no output GitHub Actions: All non-skipped checks passed on PR #1647, including lint, build, build-wheel, test (1-4), test-agno, test-extras, test-dashboard-ui, docker-native-e2e, docker-init-e2e, docker-wrap-e2e, security checks, merge-conflicts, and PR governance. ``` ## Real Behavior Proof - Environment: local macOS worktree, throwaway Python env at `/tmp/headroom-1616-testenv`, `PYTHONPATH=.`. - Exact command / steps: recorded a compressed request through `PrometheusMetrics.record_request()`, re-created `PrometheusMetrics` with the same `SavingsTracker` path, then exported `/metrics` text. - Observed result: runtime counters are zero after re-creating the metrics object, while `headroom_persistent_savings_tokens_saved_total` and related persistent counters still expose the durable lifetime values. - Not tested: full server-level pytest locally, because the local build is blocked by the known native `headroom._core`/`esaxx-rs` build issue (`fatal error: 'cstdint' file not found`). The app-level `/metrics` assertions passed in GitHub Actions. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes This intentionally does not rename or hydrate the existing runtime `headroom_tokens_saved_total` counter. That preserves the current process-local semantics and gives external dashboards a dedicated lifetime series that maps directly to `/stats.persistent_savings`. `mypy headroom` was not run as a standalone local command. CHANGELOG is N/A for this narrow proxy metrics fix unless maintainers prefer an entry. |
||
|
Lakshya Sharma
|
4658721ea0
|
feat(cache): attribute prompt-cache misses to TTL lapse vs prefix change (#1313) (#1343)
## Description A low prompt-cache hit rate is hard to act on without knowing *why* turns miss. Two very different causes need very different responses: - **TTL lapse** — the session went idle longer than the provider's cache lifetime, so the entry expired. The fix is a longer TTL (e.g. Anthropic's 1h breakpoint instead of the 5m default). - **Prefix change** — the cacheable message prefix shifted, so the new request couldn't match the cached key. A longer TTL won't help here at all. Right now those look identical from the dashboard (just "cache_read was 0"). This adds the attribution so a user can actually decide 5m vs 1h. Closes #1313 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made `PrefixCacheTracker` already kept the previous turn's forwarded messages and a per-turn activity timestamp, so the signal was already there — it just wasn't being read. - **`prefix_tracker.py`** — `classify_cache_miss()`: when a turn expected a cached prefix (non-zero cached tokens last turn) but read 0 this turn, returns `ttl_expiry` if the idle gap exceeded the provider cache TTL, else `prefix_change` if the forwarded prefix differs from last turn's, else `unknown`. **TTL wins ties** — once the entry lapsed, a coincident content change is moot, and the 5m-vs-1h decision is exactly what the TTL signal answers. A 1h-breakpoint session can widen the window via `PrefixFreezeConfig.cache_ttl_seconds`. Cold starts and hits return `is_miss=False`. - **Anthropic handlers (streaming + non-streaming)** — classify BEFORE `update_from_response` overwrites the last-turn state the classifier reads, then record the reason. - **`prometheus_metrics.py`** — a per-provider/per-reason counter, `record_cache_miss_attribution()`, reset handling, and a `headroom_cache_miss_attribution_total{provider,reason}` export series. - **`cost.py`** — `build_prefix_cache_stats()` aggregates a `miss_attribution` block (per-provider + totals, with the ttl/prefix split as a % of *attributed* misses, so `unknown` doesn't dilute the headline). - **dashboard** — a "Cache Miss Attribution" panel (TTL expiry / prefix change / unknown / total) with a "mostly TTL lapse" vs "mostly prefix change" headline. Scoped to Anthropic for this first cut (where the tracker is fully wired); OpenAI/Gemini can follow once the shape is proven. ## Testing - [x] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_cache/test_prefix_tracker.py -q 38 passed # 29 existing + 9 new classifier tests (TestClassifyCacheMiss). $ python -m pytest tests/test_proxy_cache_ttl_metrics.py -k "miss_attribution or reset_runtime_clears" -q 5 passed, 8 deselected # new: counter bucketing, stats aggregation, empty case, /metrics export, reset. ``` The full `test_proxy_cache_ttl_metrics.py` / `test_proxy_dashboard_stats_cache.py` files have some failures in this sandbox (`test_stats_endpoint_*`, streaming-parser, reset-counters) — those spin up the proxy server / Rust `_core` extension, which isn't built here. I confirmed via `git stash` that they fail identically on `main` without my changes, so they're pre-existing and unrelated. My additions to the stats dict are purely additive and don't break any passing assertion. ## Real Behavior Proof - Environment: Windows 11, Python 3.10. The Rust `_core` extension and a live proxy aren't available in this checkout. - Exact command / steps: drove `classify_cache_miss()` through every branch with a faithful warm-then-miss sequence; drove `record_cache_miss_attribution()` → `build_prefix_cache_stats()` → `export()` end to end. - Observed result: classifier returns `cold_start`/`hit`/`ttl_expiry`/`prefix_change`/`unknown` correctly, TTL wins the tie when both signals fire, a growing (append-only) prefix is treated as stable, and the 1h override widens the window. The stats builder produces `miss_attribution.totals` (`ttl_expiry`/`prefix_change`/`unknown`/`total` + `ttl_expiry_pct`/`prefix_change_pct` over attributed misses) and `by_provider`; `/metrics` emits `headroom_cache_miss_attribution_total{provider="anthropic",reason="ttl_expiry"}`. - Not tested: a live Anthropic session through the running proxy with a real idle-then-resume to confirm the handler wiring fires end-to-end. I verified the handler integration by reading scope/order (classify before `update_from_response`, `provider_name`/`self.metrics` in scope) and unit-tested every layer it calls, but didn't exercise the actual server loop. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - The classifier is intentionally pure (takes the cache-read result + current forwarded messages + an optional idle override) so it's order-independent and unit-testable without a live tracker clock. - No README/docs change yet — this surfaces in the dashboard and `/metrics`, which are self-describing; happy to add a docs page if you'd like one. - CHANGELOG.md isn't touched — release-please generates it from the `feat(cache):` commit subject. - Follow-ups if useful: extend to OpenAI/Gemini handlers, and add a per-provider breakdown row in the dashboard panel (the stats already carry `by_provider`). |
||
|
JerrettDavis
|
be6aa14110 |
feat: expose proxy OTEL metrics and Langfuse status
Wire the proxy's operational metrics facade into the new observability layer, expand built-in Prometheus export, surface OTEL and Langfuse status in /stats, and document the split between anonymous telemetry, OTEL metrics, and Langfuse traces. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
JerrettDavis
|
a6787556be | Add observed cache TTL metrics and dashboard coverage |