mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
4658721ea0
## Description A low prompt-cache hit rate is hard to act on without knowing *why* turns miss. Two very different causes need very different responses: - **TTL lapse** — the session went idle longer than the provider's cache lifetime, so the entry expired. The fix is a longer TTL (e.g. Anthropic's 1h breakpoint instead of the 5m default). - **Prefix change** — the cacheable message prefix shifted, so the new request couldn't match the cached key. A longer TTL won't help here at all. Right now those look identical from the dashboard (just "cache_read was 0"). This adds the attribution so a user can actually decide 5m vs 1h. Closes #1313 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made `PrefixCacheTracker` already kept the previous turn's forwarded messages and a per-turn activity timestamp, so the signal was already there — it just wasn't being read. - **`prefix_tracker.py`** — `classify_cache_miss()`: when a turn expected a cached prefix (non-zero cached tokens last turn) but read 0 this turn, returns `ttl_expiry` if the idle gap exceeded the provider cache TTL, else `prefix_change` if the forwarded prefix differs from last turn's, else `unknown`. **TTL wins ties** — once the entry lapsed, a coincident content change is moot, and the 5m-vs-1h decision is exactly what the TTL signal answers. A 1h-breakpoint session can widen the window via `PrefixFreezeConfig.cache_ttl_seconds`. Cold starts and hits return `is_miss=False`. - **Anthropic handlers (streaming + non-streaming)** — classify BEFORE `update_from_response` overwrites the last-turn state the classifier reads, then record the reason. - **`prometheus_metrics.py`** — a per-provider/per-reason counter, `record_cache_miss_attribution()`, reset handling, and a `headroom_cache_miss_attribution_total{provider,reason}` export series. - **`cost.py`** — `build_prefix_cache_stats()` aggregates a `miss_attribution` block (per-provider + totals, with the ttl/prefix split as a % of *attributed* misses, so `unknown` doesn't dilute the headline). - **dashboard** — a "Cache Miss Attribution" panel (TTL expiry / prefix change / unknown / total) with a "mostly TTL lapse" vs "mostly prefix change" headline. Scoped to Anthropic for this first cut (where the tracker is fully wired); OpenAI/Gemini can follow once the shape is proven. ## Testing - [x] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_cache/test_prefix_tracker.py -q 38 passed # 29 existing + 9 new classifier tests (TestClassifyCacheMiss). $ python -m pytest tests/test_proxy_cache_ttl_metrics.py -k "miss_attribution or reset_runtime_clears" -q 5 passed, 8 deselected # new: counter bucketing, stats aggregation, empty case, /metrics export, reset. ``` The full `test_proxy_cache_ttl_metrics.py` / `test_proxy_dashboard_stats_cache.py` files have some failures in this sandbox (`test_stats_endpoint_*`, streaming-parser, reset-counters) — those spin up the proxy server / Rust `_core` extension, which isn't built here. I confirmed via `git stash` that they fail identically on `main` without my changes, so they're pre-existing and unrelated. My additions to the stats dict are purely additive and don't break any passing assertion. ## Real Behavior Proof - Environment: Windows 11, Python 3.10. The Rust `_core` extension and a live proxy aren't available in this checkout. - Exact command / steps: drove `classify_cache_miss()` through every branch with a faithful warm-then-miss sequence; drove `record_cache_miss_attribution()` → `build_prefix_cache_stats()` → `export()` end to end. - Observed result: classifier returns `cold_start`/`hit`/`ttl_expiry`/`prefix_change`/`unknown` correctly, TTL wins the tie when both signals fire, a growing (append-only) prefix is treated as stable, and the 1h override widens the window. The stats builder produces `miss_attribution.totals` (`ttl_expiry`/`prefix_change`/`unknown`/`total` + `ttl_expiry_pct`/`prefix_change_pct` over attributed misses) and `by_provider`; `/metrics` emits `headroom_cache_miss_attribution_total{provider="anthropic",reason="ttl_expiry"}`. - Not tested: a live Anthropic session through the running proxy with a real idle-then-resume to confirm the handler wiring fires end-to-end. I verified the handler integration by reading scope/order (classify before `update_from_response`, `provider_name`/`self.metrics` in scope) and unit-tested every layer it calls, but didn't exercise the actual server loop. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - The classifier is intentionally pure (takes the cache-read result + current forwarded messages + an optional idle override) so it's order-independent and unit-testable without a live tracker clock. - No README/docs change yet — this surfaces in the dashboard and `/metrics`, which are self-describing; happy to add a docs page if you'd like one. - CHANGELOG.md isn't touched — release-please generates it from the `feat(cache):` commit subject. - Follow-ups if useful: extend to OpenAI/Gemini handlers, and add a per-provider breakdown row in the dashboard panel (the stats already carry `by_provider`). |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| test_anthropic.py | ||
| test_backends.py | ||
| test_base.py | ||
| test_client_integration.py | ||
| test_dynamic_detector.py | ||
| test_google.py | ||
| test_openai.py | ||
| test_prefix_tracker.py | ||
| test_registry.py | ||
| test_semantic.py | ||