1 commit

Author	SHA1	Message	Date
Tejas Chopra	48201345be	fix(proxy): keep cache_control bounded + stable so the freeze overlay stops busting (#1852) ... Follow-up to #1850. Two residual cache-bust sources, both `cache_control`-related: 1. **Guard too strict.** `overlay_cached_prefix` decided "is this turn an append-only extension?" by comparing whole message dicts — including `cache_control`. Clients (Claude Code, litellm) move the cache breakpoint to the newest message every call, so a marker landing in the frozen prefix made the guard fail, the overlay skip its replay, and the raw freeze forward ORIGINAL bytes over the cached COMPRESSED prefix → partial bust (the ~42% residual on the a10 run, `prefix_change=0`). Fix: run the append-only guard on **content only** (strip `cache_control` before comparing) — content is what the provider's cache keys on. 2. **Marker accumulation.** The overlay replays the markers that rode on each turn's then-newest message, so `cache_control` blocks pile up ~1/turn; Anthropic hard-errors at >4 total. Fix: `normalize_message_cache_control` strips every message-level marker and re-places a single ephemeral breakpoint on the last block (one breakpoint caches the whole prefix; cache is content-keyed so re-placing never busts). Wired into the Anthropic handler after the overlay. **Per-provider (deliberately scoped):** - **Anthropic**: `cache_control` markers → both fixes apply. - **OpenAI**: AUTOMATIC prefix caching, no markers → overlay (byte-identity) only; normalize is NOT applied (Anthropic markers on an OpenAI request would be wrong). - **Bedrock**: serves Claude via the pipeline but has no cachePoint/freeze-replay path → not affected; a cachePoint analog would be needed if caching is expanded. - **Gemini**: explicit Cache API (`cachedContent`), no inline markers/freeze → N/A. > Stacked on #1850 — review that first; the diff against `main` includes its overlay + `has_new_ccr_markers` work. ## Description Keeps the freeze overlay's cache-safety intact against real clients that relocate the `cache_control` breakpoint each turn, and prevents `cache_control` blocks from accumulating past Anthropic's 4-marker limit. See the two fixes above. Closes #<!-- none --> — follow-up to #1850 (no separate issue). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cache/prefix_tracker.py`: append-only guard in `overlay_cached_prefix` now compares **content only** (ignores `cache_control`); new `normalize_message_cache_control()` collapses message-level markers to a single ephemeral breakpoint on the last block. - `headroom/proxy/handlers/anthropic.py`: apply `normalize_message_cache_control` after the overlay (Anthropic only). - `tests/test_cache_control_move_bust.py`: reproduces the moved-marker bust + proves both fixes. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed (local, see below) ### Test Output ```text $ pytest tests/test_cache_control_move_bust.py -q ....... [100%] 7 passed in 0.19s # broader cache-safety suite (overlay + cross-turn + CCR deferred + openai/anthropic cache-stability + helpers) $ pytest tests/test_cache_control_move_bust.py tests/test_cache_prefix_overlay.py \ tests/test_cross_turn_cache_safety.py tests/test_proxy/test_anthropic_ccr_deferred_injection.py \ tests/test_proxy_handler_helpers.py tests/test_proxy_openai_cache_stability.py \ tests/test_proxy_anthropic_cache_stability.py -q 91 passed, 2 warnings in 29.98s $ ruff check . # ruff 0.15.17 (CI-pinned) All checks passed! $ ruff format --check . # ruff 0.15.17 1057 files already formatted $ mypy headroom --ignore-missing-imports Success: no issues found (changed modules: prefix_tracker, anthropic, openai, helpers) ``` ## Real Behavior Proof - **Environment:** local (`.venv`, Python 3.12), ruff 0.15.17 / mypy pinned to CI versions. - **Exact command / steps:** `tests/test_cache_control_move_bust.py` drives the REAL tracker + freeze + `overlay_cached_prefix` + `normalize_message_cache_control` across multiple append-only turns where the client moves the `cache_control` breakpoint each turn. - **Observed result:** with a moved marker in the frozen prefix, the content-only guard keeps the overlay replaying (forwarded prefix stays byte-identical → no bust); `cache_control` blocks stay ≤4 across many turns and content is never altered. The reproduction test fails without the fix and passes with it. - **Not tested (this PR):** the end-to-end a10 SWE-bench run is the field observation motivating fix #1 (~42% residual, `prefix_change=0`); not re-run here. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes Stacked on #1850; land that first. Docs/CHANGELOG untouched (behavioral cache-safety fix; no user-facing surface change). N/A: no screenshots (no UI).	2026-07-06 17:05:34 -07:00