mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
6 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Andrei Boldyrev
|
7bfb1d7f38
|
fix(cache): stable session identity and per-conversation prefix trackers under agentic clients (#2193)
## Description Running headroom as the proxy for Claude Code destroys Anthropic prompt-cache reuse (#2085: ~4.4x cache-creation inflation, 2.5–3x net cost). Tracing live Claude Code traffic through the proxy shows **two independent session-identity defects**, both of which orphan or thrash the frozen-prefix state; this PR fixes both. ### Defect 1: `<system-reminder>` turns rotate the fallback session id mid-conversation Claude Code interleaves reminder turns into the history as actual `role:"system"` messages (hook output, skills lists, file-truncation notices). `compute_session_id` hashed **every** system message, so the id rotated each time a reminder landed. Live trace (subagent reading two 80KB files; sid changes exactly when the truncation reminder appears, and the tracker restarts at turn 0): ``` REQ#2 sid=68d4ee666990 nmsg=3 [0]SYSTEM<<top-level system>> [1]user [2]SYSTEM<<skills reminder>> REQ#3 sid=6944948c9fb2 nmsg=6 ... [5]SYSTEM<<Truncated: PARTIAL view ...>> <- id rotated ``` Everything keyed on the session id is orphaned at that moment: the prefix tracker (freeze never survives past a reminder-bearing turn), beta-header stickiness, the CCR and memory-tool registries, and the compression cache. **Fix:** hash only the **leading run** of system messages (everything before the first non-system turn) — the top-level system prompt on the Anthropic path (folded in as the synthetic first message), the conventional leading system message(s) on the OpenAI path. Stable for the life of a conversation; mid-history system turns are content, not identity. ### Defect 2: conversations sharing a (now stable) id thrash one tracker With ids stable, the fallback tuple `model + system prompt` is identical across every same-type parallel subagent (and any sessions reusing one system prompt) — all of them collapse onto one `PrefixCacheTracker`, and their interleaved histories cross-contaminate the freeze state: the forwarded prefix is byte-unstable on nearly every turn and the provider cache is re-written instead of read. Reproduced against the real code paths (script below): ``` 1) fallback session ids: A=3dc639aaf4f48fa1 B=3dc639aaf4f48fa1 -> COLLIDE=True 2) single conversation, legacy : stable prefix on 4/4 later turns, trackers=1 2) interleaved (subagents), legacy : stable prefix on 0/9 later turns, trackers=1 2) interleaved, lineage resolution : stable prefix on 8/8 later turns, trackers=2 ``` **Fix:** `SessionTrackerStore.resolve_tracker` — within a session id, reuse the tracker whose previous request messages are a prefix of the incoming history (client histories are append-only, so a conversation's next request always extends its previous one); a diverging or rewritten history (client-side compaction) starts a fresh lineage. Matching uses the repo's existing canonical cross-turn equivalence (`_canonicalize_for_prefix_compare`, the same one the cache-stable delta path uses) on the **original client bytes**, so moved cache breakpoints, string<->block sugar, transport annotations, or a tail-mutating `pre_compress` hook never read as a rewrite. Byte-identical histories (templated fan-outs before they diverge) intentionally share a tracker — their provider cache line is identical too. ### Both fixes together, on live Claude Code traffic (sonnet, 2 parallel Explore agents) ``` main conversation: sid=5b7e245a... one tracker, turns 0->4, id stable across reminders agents (collide): sid=2bdffc9e... -> lineage bare (alpha) turns 0->1->2 -> lineage "~1" (beta) turns 0->1->2 ``` Before: the agents' ids rotated per reminder (every tracker stuck at turn 0), and whenever they did share an id they thrashed one tracker (`0/9` stable prefixes in the repro). ### Why not key the session id on conversation content? Draft #1912 folds the first user turn into the fallback id; this change composes with it, but identity-level keying alone can't close #2085: identical first turns (templated fan-outs) still collide, and everything keyed on the session id rotates with it when the client rewrites history. The "session" (client/workspace grouping) and the "conversation" (positional cache lineage) are different identities; only the tracker holds positional per-turn state that thrashes under collision — beta stickiness is a monotone union and the compression cache is content-addressed — so lineage resolution lives one level below the session id and leaves the id semantics (and every other consumer) untouched. ## Changes Made - `headroom/cache/prefix_tracker.py`: - `compute_session_id`: harvest only the leading system run (defect 1). - `SessionTrackerStore.resolve_tracker`: conversation-lineage resolution (defect 2). First lineage lives under the bare session id — single-conversation sessions behave byte-identically to before; degrades to `get_or_create` when messages are absent or prefix freeze is disabled. - Lineages are capped per session id (`PrefixFreezeConfig.max_lineages_per_session`, default 32). **Over-cap conversations share one overflow tracker instead of evicting an established lineage** — any eviction policy degrades every conversation once the working set exceeds the cap (under round-robin the victim is always the conversation about to arrive), while overflow sharing degrades only the over-cap tail, to exactly the pre-lineage shared behavior; `0` disables lineage splitting. Chains are stored as structural snapshots that normalize `NaN` (`json.loads` accepts bare NaN, and `NaN != NaN` would read a byte-identical resend as a rewrite). Synthetic lineage keys use a `\x00` separator, which cannot appear in an HTTP header value, so they can never collide with a client-supplied `x-headroom-session-id`. - `headroom/proxy/handlers/anthropic.py`, `openai.py`: the session id and the lineage both derive from the **same original client bytes** (a turn-dependent hook rewrite can no longer rotate one without the other); anthropic folds in its synthetic system message so explicit-header clients with different system prompts stay separate. Plus a docstring correction in `streaming.py` that falsely claimed its coarse mid-turn key "mirrors" `compute_session_id`. - `tests/test_cache/test_prefix_tracker.py`: 24 new test cases — reminder-rotation regression; interleaved isolation + per-conversation turn state; identical-first-turn share-then-split; cache_control movement (3 cases); representation churn (string<->block sugar / streaming `index` / Bedrock cachePoint); rewritten history → fresh lineage (compacted / middle-edited / truncated); legacy no-messages / freeze-disabled / empty-canonical fallbacks; NaN-in-tool-payload stability; overflow sharing, established-lineages-survive-cap, and a cap+1 round-robin no-cliff guard; TTL cleanup; session-id-not-rotated-by-lineage guard. One existing test renamed (`uses_all_system_messages` → `distinguishes_leading_system_run`) to match the new contract. - Three SimpleNamespace stub stores in existing tests gained a `resolve_tracker` field (handlers call it unconditionally — a silent `hasattr` fallback would degrade to the pre-fix behavior with no signal). One of them is the cold-start fast-pass suite (#2073), which landed while this branch was in review. - `CHANGELOG.md` entry. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Testing - [x] Unit tests pass (`pytest`) — 11 failed, 8652 passed, 528 skipped in 4:37 (the 11 are pre-existing on unmodified `main` — verified by rerunning the same node ids on a clean checkout: gh-CLI/onnx/PID-reuse/deadline flakes and order-dependent cases, none touching session/cache/proxy paths) - [x] Linting passes (`ruff check .`) — All checks passed (ruff 0.15.17, CI-pinned; `ruff format --check .` clean) - [x] Type checking passes (`mypy headroom`) — Success: no issues found in 471 source files - [x] New tests added for new functionality — 24 test cases; the rotation/isolation/no-cliff ones fail on `main` - [x] Manual testing performed — live Claude Code end-to-end, below ### Test Output ```text $ python -m pytest tests/ -q 11 failed, 8652 passed, 528 skipped, 5857 warnings in 276.68s (0:04:36) # same 11 fail on unmodified main (env/order-dependent: test_wrap_claude_base_url pid-reuse, # copilot_auth gh-cli fallback, image_compression onnx, content_router deadline, rtk/output-shaper/dedup order flakes) $ python -m pytest tests/test_cache/test_prefix_tracker.py -q 63 passed $ uvx ruff@0.15.17 check . && uvx ruff@0.15.17 format --check . All checks passed! / 1208 files already formatted $ mypy headroom Success: no issues found in 471 source files $ python repro_2085.py 1) fallback session ids: A=3dc639aaf4f48fa1 B=3dc639aaf4f48fa1 -> COLLIDE=True 2) single conversation, legacy : stable prefix on 4/4 later turns, trackers=1 2) interleaved (subagents), legacy : stable prefix on 0/9 later turns, trackers=1 2) interleaved, lineage resolution : stable prefix on 8/8 later turns, trackers=2 ``` ## Real Behavior Proof - Environment: macOS arm64, Python 3.13, `uv sync --extra dev --extra proxy`; real Claude Code CLI pointed at the proxy via `ANTHROPIC_BASE_URL=http://127.0.0.1:8790`, real Anthropic backend. - Exact command / steps: ran Claude Code sessions that launch 2–3 parallel Explore subagents (each reading multi-KB JSON files, several tool-loop turns each), with an observability wrapper printing each request's resolved session id, tracker identity, and turn counter inside the proxy. - Observed result: on `main`, subagent session ids rotate on reminder-bearing turns (trackers permanently stuck at turn 0); when conversations do share an id they share one tracker whose turn counter interleaves all of them. On this branch: ids stable for the life of each conversation; colliding subagents resolve to separate lineages (`bare`, `~1`) with clean per-conversation turn progressions (trace above). Unit-level repro shows forwarded-prefix stability going 0/9 → 8/8 for the interleaved shape. - Not tested: reporter-scale cache-economics (his 4.4x needs his long-session workload against a paid backend); happy to coordinate with @RomanAlexanderW on a before/after — the number to watch is the cache-read ratio in Claude Code transcripts recovering toward ~96%. <details> <summary>repro_2085.py</summary> ```python """Repro for #2085: concurrent conversations sharing a fallback session id (same model + system prompt — e.g. a Claude Code session and its parallel subagents) collapse onto one PrefixCacheTracker and thrash its frozen-prefix state -> byte-unstable forwarded prefixes -> the provider prompt cache is re-written on nearly every call. Uses headroom's real code paths. Run from the repo root: python ../repro_2085.py """ from headroom.cache.prefix_tracker import PrefixFreezeConfig, SessionTrackerStore MODEL = "claude-sonnet-5" # Claude Code system prompt: long, static, identical across the main session # and every parallel subagent of the same type. SYSTEM = ("You are Claude Code, Anthropic's official CLI for Claude. " * 40)[:2000] def convo(name: str, turns: int) -> list[dict]: msgs = [{"role": "system", "content": SYSTEM}] for t in range(turns): msgs.append({"role": "user", "content": f"[{name}] user turn {t}: " + ("x" * 800)}) msgs.append( {"role": "assistant", "content": f"[{name}] tool_result {t}: " + ('{"data": 1}' * 200)} ) return msgs class _Req: # request stub: no x-headroom-session-id header headers: dict = {} # --- Part 1: identity collision (real derivation) ---------------------------- store = SessionTrackerStore(PrefixFreezeConfig()) id_a = store.compute_session_id(_Req(), MODEL, convo("A", 3)) id_b = store.compute_session_id(_Req(), MODEL, convo("B", 5)) print(f"1) fallback session ids: A={id_a} B={id_b} -> COLLIDE={id_a == id_b}") # --- Part 2: interleaved conversations thrash the freeze state --------------- def run(interleave: bool, lineage_resolution: bool) -> tuple[int, int, int]: store = SessionTrackerStore(PrefixFreezeConfig()) stable_turns = 0 later_turns = 0 seq = [] for t in range(1, 6): seq.append(("A", convo("A", t))) if interleave: seq.append(("B", convo("B", t))) for _name, msgs in seq: sid = store.compute_session_id(_Req(), MODEL, msgs) if lineage_resolution: tracker = store.resolve_tracker(sid, "anthropic", messages=msgs) else: tracker = store.get_or_create(sid, "anthropic") if tracker._turn_number > 0: later_turns += 1 if tracker._forwarded_prefix_stable(msgs): stable_turns += 1 tracker.update_from_response( cache_read_tokens=5000 * len(msgs), cache_write_tokens=2000, messages=msgs, ) return stable_turns, later_turns, store.active_sessions for label, interleave, fixed in ( ("single conversation, legacy ", False, False), ("interleaved (subagents), legacy ", True, False), ("interleaved, lineage resolution ", True, True), ): stable, later, sessions = run(interleave, fixed) print(f"2) {label}: stable prefix on {stable}/{later} later turns, trackers={sessions}") ``` </details> ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (CHANGELOG only — no docs describe the tracker store) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - Addresses the session-identity mechanisms of #2085; intentionally does not `Closes` it — the reporter should confirm the cache-read ratio recovers on live traffic first. - Composes with draft #1912 (first-user-turn fallback id). - Known bounded tradeoffs (all strictly milder than the per-turn thrash this fixes): a fork-style branch that resends a parent's full history adopts the parent's lineage, costing the parent one cold restart at its next turn; a request that aborts before the response and is retried with different bytes starts a fresh lineage; history truncation/tail-edit starts a fresh lineage even though the shorter provider prefix may still be warm. - Hot-path cost, measured on a 199-message/2.1MB agentic history: canonical projection 0.21ms + structural snapshot 0.92ms + match loop 0.06ms with 32 candidate lineages (2.27ms absolute worst case) ≈ **1.3ms per request** — same order as the handler's existing request deepcopy (0.80ms) and below one `json.dumps` of the body (2.9ms). Chain memory is structure-only (~180-330KB per lineage; message strings are shared with state the tracker already retains). - Known semantic shift to flag: hashing only the leading system run means conversations distinguished ONLY by mid-list system messages (e.g. clients injecting a per-conversation system context late in the list) now share a fallback id. The tracker is protected by lineage resolution; the residual sharing concentrates in the CCR sticky-tool registry and the monotone beta union — the same pre-existing class as same-system-prompt conversations today. Happy to file the CCR-stickiness scoping as a follow-up. - Out of scope, observed while tracing: `SessionCcrTracker.has_done_ccr` mildly cross-contaminates conversations sharing an id (monotone, no thrash) — can file separately if useful. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Abhay Singh
|
ec6e60ea3e
|
fix(proxy/anthropic): scope session id by top-level system prompt (#2070)
## Description
`SessionTrackerStore.compute_session_id`
(`headroom/cache/prefix_tracker.py`) computes a fallback
session id (when no `x-headroom-session-id` header is present) from
`model` + system-prompt text.
But it harvests system text **only** from `messages` entries with `role
== "system"`:
```python
for msg in messages:
if msg.get("role") == "system":
... # collect system text
system_content = json.dumps(system_parts, ...)
key = f"{model}:{system_content}"
```
Anthropic's `/v1/messages` carries the system prompt as a **top-level**
`body["system"]` field —
it never sends `role:"system"` entries inside `messages`. And
`x-headroom-session-id` is a
Headroom-internal header no client sends. So for every genuine Anthropic
request `system_parts`
is empty and the id collapses to `md5(f"{model}:[]")` — **every
conversation on the same model
shares one session id**, and therefore one `PrefixCacheTracker` and all
session-sticky state.
The colliding state cross-contaminates across conversations
(`anthropic.py:1052`):
- sticky `headroom_retrieve` / memory tools keyed purely on `session_id`
(no content guard) get
injected into another conversation's tool list — busting its tools cache
and adding tools its
client never requested;
- sticky `anthropic-beta` header tokens leak across conversations;
- `frozen_message_count` and the per-session compression cache
cross-contaminate.
(The sibling `StreamingMixin._get_session_key` already reads
`body.get("system")` and its docstring
claims to mirror `compute_session_id` — which it did not.)
Closes: no issue filed — found while auditing the session/prefix
tracker.
## Fix
Add an optional `system` parameter to `compute_session_id` and fold its
text (a plain string or a
list of `{"type":"text"}` blocks) into the hash. The Anthropic handler
passes `body.get("system")`.
OpenAI callers don't pass it (defaults to `None`), so their behavior is
unchanged.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/cache/prefix_tracker.py`: `compute_session_id` accepts an
optional `system` and folds it into the id.
- `headroom/proxy/handlers/anthropic.py`: pass
`system=body.get("system")` when computing the session id.
- `tests/test_cache/test_prefix_tracker.py`: add
`test_compute_session_id_distinguishes_top_level_system` (distinct
systems → distinct ids; list-form == string-form; `system=None`
unchanged).
## Testing
- [x] New regression test added
(`tests/test_cache/test_prefix_tracker.py`)
- [x] Linting/formatting clean — run with the CI-pinned `ruff==0.15.17`
- [ ] Full `pytest` deferred to CI (local-OOM reason below).
```text
$ uvx ruff@0.15.17 check headroom/cache/prefix_tracker.py headroom/proxy/handlers/anthropic.py tests/test_cache/test_prefix_tracker.py
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.10, headroom from this branch.
Importing `headroom` pulls in the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I verified the hash logic with
a dependency-free script and left the full pytest to CI.
- Exact command / steps: computed ids for two conversations with the
same model and messages but different top-level `system` prompts,
through the old (never-folds-system) and new logic.
- Observed result: the old logic collapses both to one id (the leak);
the new logic separates them, folds list-form system the same as
string-form, and leaves the `system=None` (OpenAI) path unchanged:
```text
OLD: A=97d8857ba27010bb B=97d8857ba27010bb same=True
NEW: A=1e838c0f6e3980a6 B=18ec49bfa8240852 same=False
SESSION-ID SYSTEM FIX VERIFIED (old collapses Anthropic convos; new separates them)
```
- Not tested: a full two-conversation proxy run asserting no sticky-tool
leakage (needs the heavy stack). The fix is confined to
`compute_session_id` + the one handler call site, and the new test
drives the method directly. Full local `pytest` deferred to CI (OOM, per
above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — ran
lint + a standalone logic check; full pytest deferred to CI (local OOM,
disclosed above)
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- Backward-compatible: the new `system` parameter defaults to `None`, so
the OpenAI call sites (`openai.py`) need no change and their session ids
are identical.
- @JerrettDavis tagging you — this one lets one Anthropic conversation's
sticky tools/headers leak into another on the same model, so it seemed
worth surfacing. Thanks!
|
||
|
Rod Boev
|
0f606b6281
|
fix(cache): avoid fallback session collisions (#1827)
## Description Cache-mode session tracking currently collapses unrelated conversations when they share a large static first system prompt. The fallback session-id hash ignores later system messages entirely, so dynamic per-conversation context can get cut out of the key and two different sessions reuse the same `PrefixCacheTracker`. This hashes the full ordered system-text payload instead, while leaving explicit `x-headroom-session-id` overrides untouched. Refs #1808. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Collected all system-text content when building the fallback cache session id. - Stopped truncating fallback session-id input to the first 500 characters of the first system message. - Added a regression that proves two conversations with different later system context no longer collide. - Added a preservation test that appending only non-system turns keeps the same fallback session id. - Applied the pinned Ruff formatter to three pre-existing files on the current base so the repo-wide lint job passes unchanged semantics. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_cache/test_prefix_tracker.py -q`) - [x] Linting passes (`uv run ruff check headroom/cache/prefix_tracker.py tests/test_cache/test_prefix_tracker.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text uv run pytest tests/test_cache/test_prefix_tracker.py -q 40 passed, 1 warning in 0.15s uv run ruff check headroom/cache/prefix_tracker.py tests/test_cache/test_prefix_tracker.py All checks passed! uv run ruff check . All checks passed! uv run ruff format --check . 1046 files already formatted ``` ## Real Behavior Proof - Environment: Windows, project `uv` environment, focused cache-tracker regression. - Exact command / steps: run `tests/test_cache/test_prefix_tracker.py` on `origin/main` with the new collision regression present, then rerun the same file on this branch. - Observed result: base returns the same session id for two conversations that differ only in a later system message and fails `assert id_a != id_b`; head passes the focused file and keeps the fallback session id stable when only non-system turns are appended. - Not tested: live proxy traffic through a real agentic client. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes This is only the session-collision half of #1808. The duplicate-response-header fix stays separate so this PR can reference the issue without claiming the whole bug report is resolved. The extra formatting-only diff comes from the current base failing the pinned full-repo Ruff format check. |
||
|
Lakshya Sharma
|
4658721ea0
|
feat(cache): attribute prompt-cache misses to TTL lapse vs prefix change (#1313) (#1343)
## Description A low prompt-cache hit rate is hard to act on without knowing *why* turns miss. Two very different causes need very different responses: - **TTL lapse** — the session went idle longer than the provider's cache lifetime, so the entry expired. The fix is a longer TTL (e.g. Anthropic's 1h breakpoint instead of the 5m default). - **Prefix change** — the cacheable message prefix shifted, so the new request couldn't match the cached key. A longer TTL won't help here at all. Right now those look identical from the dashboard (just "cache_read was 0"). This adds the attribution so a user can actually decide 5m vs 1h. Closes #1313 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made `PrefixCacheTracker` already kept the previous turn's forwarded messages and a per-turn activity timestamp, so the signal was already there — it just wasn't being read. - **`prefix_tracker.py`** — `classify_cache_miss()`: when a turn expected a cached prefix (non-zero cached tokens last turn) but read 0 this turn, returns `ttl_expiry` if the idle gap exceeded the provider cache TTL, else `prefix_change` if the forwarded prefix differs from last turn's, else `unknown`. **TTL wins ties** — once the entry lapsed, a coincident content change is moot, and the 5m-vs-1h decision is exactly what the TTL signal answers. A 1h-breakpoint session can widen the window via `PrefixFreezeConfig.cache_ttl_seconds`. Cold starts and hits return `is_miss=False`. - **Anthropic handlers (streaming + non-streaming)** — classify BEFORE `update_from_response` overwrites the last-turn state the classifier reads, then record the reason. - **`prometheus_metrics.py`** — a per-provider/per-reason counter, `record_cache_miss_attribution()`, reset handling, and a `headroom_cache_miss_attribution_total{provider,reason}` export series. - **`cost.py`** — `build_prefix_cache_stats()` aggregates a `miss_attribution` block (per-provider + totals, with the ttl/prefix split as a % of *attributed* misses, so `unknown` doesn't dilute the headline). - **dashboard** — a "Cache Miss Attribution" panel (TTL expiry / prefix change / unknown / total) with a "mostly TTL lapse" vs "mostly prefix change" headline. Scoped to Anthropic for this first cut (where the tracker is fully wired); OpenAI/Gemini can follow once the shape is proven. ## Testing - [x] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_cache/test_prefix_tracker.py -q 38 passed # 29 existing + 9 new classifier tests (TestClassifyCacheMiss). $ python -m pytest tests/test_proxy_cache_ttl_metrics.py -k "miss_attribution or reset_runtime_clears" -q 5 passed, 8 deselected # new: counter bucketing, stats aggregation, empty case, /metrics export, reset. ``` The full `test_proxy_cache_ttl_metrics.py` / `test_proxy_dashboard_stats_cache.py` files have some failures in this sandbox (`test_stats_endpoint_*`, streaming-parser, reset-counters) — those spin up the proxy server / Rust `_core` extension, which isn't built here. I confirmed via `git stash` that they fail identically on `main` without my changes, so they're pre-existing and unrelated. My additions to the stats dict are purely additive and don't break any passing assertion. ## Real Behavior Proof - Environment: Windows 11, Python 3.10. The Rust `_core` extension and a live proxy aren't available in this checkout. - Exact command / steps: drove `classify_cache_miss()` through every branch with a faithful warm-then-miss sequence; drove `record_cache_miss_attribution()` → `build_prefix_cache_stats()` → `export()` end to end. - Observed result: classifier returns `cold_start`/`hit`/`ttl_expiry`/`prefix_change`/`unknown` correctly, TTL wins the tie when both signals fire, a growing (append-only) prefix is treated as stable, and the 1h override widens the window. The stats builder produces `miss_attribution.totals` (`ttl_expiry`/`prefix_change`/`unknown`/`total` + `ttl_expiry_pct`/`prefix_change_pct` over attributed misses) and `by_provider`; `/metrics` emits `headroom_cache_miss_attribution_total{provider="anthropic",reason="ttl_expiry"}`. - Not tested: a live Anthropic session through the running proxy with a real idle-then-resume to confirm the handler wiring fires end-to-end. I verified the handler integration by reading scope/order (classify before `update_from_response`, `provider_name`/`self.metrics` in scope) and unit-tested every layer it calls, but didn't exercise the actual server loop. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - The classifier is intentionally pure (takes the cache-read result + current forwarded messages + an optional idle override) so it's order-independent and unit-testable without a live tracker clock. - No README/docs change yet — this surfaces in the dashboard and `/metrics`, which are self-describing; happy to add a docs page if you'd like one. - CHANGELOG.md isn't touched — release-please generates it from the `feat(cache):` commit subject. - Follow-ups if useful: extend to OpenAI/Gemini handlers, and add a per-provider breakdown row in the dashboard panel (the stats already carry `by_provider`). |
||
|
Tejas Chopra
|
22dad133e2 |
fix: eliminate prefix cache busts from frozen count underestimation
Root cause: CompressionCache.compute_frozen_count() stopped at the first tool_result not in its cache, capping frozen_message_count at 2. Tool results excluded by content_router (Read/Glob) or skipped (ratio too high) never entered the cache, so every subsequent message was eligible for recompression — causing 192 cache busts per session. Four fixes: 1. Add _stable_hashes set to CompressionCache so excluded/skipped tool_results don't block the frozen count walk 2. Fix _estimate_message_tokens to count tool_result content and tool_use input fields (were counted as 0 tokens in Anthropic format) 3. Fix streaming handler to include assistant response and original_messages in prefix tracker updates (parity with non-streaming) 4. TTL-aware batch recompression: defer first-time compressions within the 5-min cache TTL window, batching them at the boundary to trade many small busts for one |
||
|
chopratejas
|
8f438b0674 |
Introducing headroom wrap
- headroom wrap claude is the simplest way to start up claude - It will also install rtk-ai locally - rtk-ai is a cli wrapper that can save ~90% tokens for CLI calls made by Claude Code |