2 commits

Author	SHA1	Message	Date
Tejas Chopra	d12ea50122	feat(proxy): unify proxy and sidecar compression on one session engine (#3271) ... > Replaces #3263 (same changeset, squashed to one conventional commit — after the base PRs squash-merged, the stacked branch's commit history could not pass the commitlint gate against main, and force-pushing the original branch was not permitted). #3261 and #3270 (which replaced #3262) are merged; this is the last piece of the stack. ## Goal One brain. The cache-management tier — freeze computation, Zone-1 byte swap, cached-prefix overlay — previously existed twice: inline in the proxy request handlers, and (as of #3270) in the `/v1/compress` sidecar path. This PR extracts it into **`headroom/proxy/session_engine.py`**, invoked by BOTH. Every future cache-management fix lands in both modes by construction. ## Design **`prepare_turn(...)` → `TurnPrep`** — freeze + `mark_stable_from_messages` + `apply_cached`, with two *deliberately different, documented* freeze policies: - `FREEZE_POLICY_CONFIRMED_CLAMP`: `min(tracker_frozen, cache_count)` — never freeze past provider-confirmed (the #327 posture). The Anthropic proxy passes its already-composed tracker/strict-override value, reproducing the previous `min()` byte-for-byte. - `FREEZE_POLICY_REPLAYABLE`: `max(cache_count, explicit)` — freeze everything locally replayable, because whatever was previously returned *is* the provider's cache contract; recompressing it (even "better") busts. **`finalize_turn(...)` → `TurnFinal`** — the byte-identical cached-prefix replay (`overlay_cached_prefix`) + conditional token recount hook. Run as a **strictly behavior-preserving extraction**: the bar was every pre-existing test passing *unmodified*, and it held. ## What migrated | Path | Status | |---|---| | `/v1/compress` sidecar turn | ✅ engine (REPLAYABLE); lock, executor offload, savings accounting, record_returned unchanged | | `anthropic.py` token-mode pre-block + overlay | ✅ engine (CONFIRMED_CLAMP); background compression, cold-start fast pass, `_cold_recompact_active` skip preserved | | `openai.py` proxy token-mode pre-block + overlay | ✅ engine (REPLAYABLE — formula-identical to the old bare `compute_frozen_count`); the added `mark_stable` call means the freeze now survives entry-level LRU eviction (test-pinned); the router's `_frozen_verdicts` remains the boundary-message protection | | `openai.py` cache-mode branch | ⏸ keeps bare `apply_cached` — cache mode keeps the latest observation mutable by design | Also fixed for BOTH handlers: overlay replay now runs under backpressure (shedding it busted every gated session's prompt cache exactly at peak load), and the inflation guard exempts replayed prefixes. ## Hardening (max-effort review, all applied) `/v1/usage` applies on the executor under the per-session turn lock with a timed acquire (503 `session_busy`); registry eviction skips sessions mid-turn; `peek()` is expiry-aware; silent fallbacks log warnings; RequestOutcome recorded on session 503s. ## Testing - `tests/test_session_engine.py`: 13 direct unit tests — both policies, explicit-pin precedence, REPLAYABLE-without-pin ≡ bare `compute_frozen_count`, overlay fires/doesn't, recount only on replay, freeze-survives-entry-eviction. - Parity bar: full pre-existing suites pass unmodified — cache-stability (Anthropic + OpenAI), overlay, backpressure (incl. replay-under-saturation regression), cold-start fast pass, cache-mode, session-mode byte-stability, compress-API, org-scale, registry. Full local suite: 11k+ green. - ruff check/format clean (CI's ruff 0.16.3). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01EWKCmcH47hvvoQ35wftXhE Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-08-26 15:57:31 +05:30
Tejas Chopra	826b600c9b	feat(proxy): self-limiting session state for the compression-cache registry (#3261) ... ## Problem The per-session `CompressionCache` registry (the map that replays previously-compressed messages byte-identically so the provider prefix cache stays warm) had no lifetime management: - Idle/dead sessions lived forever until the hardcoded 500-session cap was hit. - At capacity, eviction dropped the oldest-**created** quarter — which could wipe the busiest long-lived session (busting every one of its prefixes at once) while dead sessions survived. - Neither the cap nor any TTL was tunable, which blocks gateway deployments (e.g. Kong sidecar/pool) fanning many concurrent sessions into one process. ## Changes - **Idle-TTL sweep**: sessions idle longer than `HEADROOM_COMPRESSION_CACHE_TTL_SECONDS` (default 3900s) are evicted by a lazy sweep, at most once per 60s, piggybacked on `_get_compression_cache` — same pattern as `PrefixCacheTrackerRegistry._maybe_cleanup`, no background task. `last_seen` refreshes on **every** access, so an active session never expires. - **LRU capacity eviction**: the registry is now an access-ordered `OrderedDict`; capacity pressure sheds the *idlest* quarter, never a busy session. - **Tunable cap**: `HEADROOM_COMPRESSION_CACHE_MAX_SESSIONS` (default 500, floor 1). ## Why 3900s Eviction is bust-free only once the provider's own prompt cache has lapsed. Providers don't expose their cache TTLs, and the risk is one-sided (late eviction costs a few MB; early eviction *causes* the bust this state exists to prevent), so the default is the upper bound of documented lifetimes across providers — Anthropic's 1h extended breakpoint, OpenAI's "up to an hour off-peak", Gemini's 60-min default — plus 5m grace. A parse-time floor of 600s keeps the TTL from ever dropping below the prefix tracker's session TTL: after the tracker expires, the byte-identical swap is the only remaining protection for a still-live provider prefix. Read-hit signals are untouched: they govern the freeze boundary, never eviction — `read_hits == 0` usually means cold start or TTL lapse, where the map was just (re)written into the provider cache and deleting it would guarantee a second bust. ## Behavior impact - Steady state (any session active within the TTL): zero change — same instances, same bytes, same freeze behavior. - A session returning after >65 min idle now finds its map evicted — but every provider had already forgotten its prefix by then, so that turn was paying the cache-write price regardless (fail-open, no failed requests). - Capacity eviction now protects busy sessions instead of punishing them. ## Testing - New `tests/test_compression_cache_registry.py`: LRU-not-FIFO capacity eviction, small-cap edge case, TTL sweep eviction, access-refreshes-clock, sweep rate limiting. - 386 tests pass across compression-cache, cache-stability (Anthropic + OpenAI), prefix-overlay, cold-start, cache-mode, and Bedrock-tracker suites; ruff check/format clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01EWKCmcH47hvvoQ35wftXhE --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-08-26 15:04:43 +05:30