mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
826b600c9b
## Problem The per-session `CompressionCache` registry (the map that replays previously-compressed messages byte-identically so the provider prefix cache stays warm) had no lifetime management: - Idle/dead sessions lived forever until the hardcoded 500-session cap was hit. - At capacity, eviction dropped the oldest-**created** quarter — which could wipe the busiest long-lived session (busting every one of its prefixes at once) while dead sessions survived. - Neither the cap nor any TTL was tunable, which blocks gateway deployments (e.g. Kong sidecar/pool) fanning many concurrent sessions into one process. ## Changes - **Idle-TTL sweep**: sessions idle longer than `HEADROOM_COMPRESSION_CACHE_TTL_SECONDS` (default 3900s) are evicted by a lazy sweep, at most once per 60s, piggybacked on `_get_compression_cache` — same pattern as `PrefixCacheTrackerRegistry._maybe_cleanup`, no background task. `last_seen` refreshes on **every** access, so an active session never expires. - **LRU capacity eviction**: the registry is now an access-ordered `OrderedDict`; capacity pressure sheds the *idlest* quarter, never a busy session. - **Tunable cap**: `HEADROOM_COMPRESSION_CACHE_MAX_SESSIONS` (default 500, floor 1). ## Why 3900s Eviction is bust-free only once the provider's own prompt cache has lapsed. Providers don't expose their cache TTLs, and the risk is one-sided (late eviction costs a few MB; early eviction *causes* the bust this state exists to prevent), so the default is the upper bound of documented lifetimes across providers — Anthropic's 1h extended breakpoint, OpenAI's "up to an hour off-peak", Gemini's 60-min default — plus 5m grace. A parse-time floor of 600s keeps the TTL from ever dropping below the prefix tracker's session TTL: after the tracker expires, the byte-identical swap is the only remaining protection for a still-live provider prefix. Read-hit signals are untouched: they govern the freeze boundary, never eviction — `read_hits == 0` usually means cold start or TTL lapse, where the map was just (re)written into the provider cache and deleting it would guarantee a second bust. ## Behavior impact - Steady state (any session active within the TTL): zero change — same instances, same bytes, same freeze behavior. - A session returning after >65 min idle now finds its map evicted — but every provider had already forgotten its prefix by then, so that turn was paying the cache-write price regardless (fail-open, no failed requests). - Capacity eviction now protects busy sessions instead of punishing them. ## Testing - New `tests/test_compression_cache_registry.py`: LRU-not-FIFO capacity eviction, small-cap edge case, TTL sweep eviction, access-refreshes-clock, sweep rate limiting. - 386 tests pass across compression-cache, cache-stability (Anthropic + OpenAI), prefix-overlay, cold-start, cache-mode, and Bedrock-tracker suites; ruff check/format clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01EWKCmcH47hvvoQ35wftXhE --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| test_anthropic_buffered_timeout.py | ||
| test_anthropic_ccr_deferred_injection.py | ||
| test_anthropic_ccr_raise.py | ||
| test_anthropic_no_optimize_history_passthrough.py | ||
| test_anthropic_recount_and_reparse_safety.py | ||
| test_anthropic_streaming_ccr_retrieve.py | ||
| test_anthropic_upstream_header.py | ||
| test_background_compression.py | ||
| test_bedrock_passthrough.py | ||
| test_bedrock_sse_ping.py | ||
| test_cc_switch_reconciler.py | ||
| test_ccr_frozen_prefix_coupling.py | ||
| test_compression_failure_action.py | ||
| test_compression_timeout_config.py | ||
| test_compute_turn_id.py | ||
| test_gemini_savings_profile.py | ||
| test_header_safe_transforms.py | ||
| test_mcp_stats_aggregation.py | ||
| test_model_router.py | ||
| test_model_router_wiring.py | ||
| test_openai_backend_path.py | ||
| test_openai_chat_ccr_injection.py | ||
| test_openai_chat_savings_profile.py | ||
| test_openai_responses_ccr.py | ||
| test_openai_stream_usage_option.py | ||
| test_openai_transport_path_prefix.py | ||
| test_openai_upstream_header.py | ||
| test_phase3_byte_identity.py | ||
| test_request_logger.py | ||
| test_settings_fresh_process_precedence.py | ||
| test_settings_store.py | ||
| test_tool_search_repair_after_turn_hooks.py | ||
| test_transformations_feed.py | ||