akadmin/headroom
1
0
Fork 0
mirror of https://github.com/headroomlabs-ai/headroom.git synced 2026-08-27 14:17:10 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
2630 commits 410 branches 169 tags 142 MiB
Commit graph

4 commits

Author SHA1 Message Date
Rod Boev
c16be9bbbe
fix(proxy/anthropic): don't replay recorded prefix over live history (#3026) (#3052) 
## Description

A Claude Code session that reads a large tool result through `headroom
proxy` can fail on turn 2 with Anthropic's `400 prompt_too_long`. The
reporter's controlled comparison completed eight turns through 0.33.0
with 187,986 input tokens, while 0.35.0 failed after five requests with
753,077 input tokens. The local regression uses an actual prior
optimized request to populate tracker state, then a decision-false
bypass turn with Claude-shaped tool-result content. The old
unconditional replay path substitutes the compressed prefix; the
eligibility gate preserves the client's outbound body without claiming a
live provider reproduction.

The Anthropic `/v1/messages` route computes whether a request should be
compressed, but cached-prefix replay currently runs outside that
decision. The replay helper also derives its prefix length from the
original message list and applies that index to the optimized list
without proving the two lists still align. A stale forwarded prefix can
therefore be grafted onto the wrong positions and enlarge later
requests.

This change limits replay to requests whose existing compression
decision permits it and whose pre-upstream backpressure path is
inactive. It also makes `overlay_cached_prefix()` decline misaligned or
inflating candidates while preserving normal append-only replay.

Reported by @itsumonotakumi, whose controlled comparison isolated the
failure from compression, headers, one-request serialization, memory,
code graph, and CCR.

Closes #3026

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Gate Anthropic cached-prefix replay on the existing
`CompressionDecision.should_compress` result and the existing
pre-upstream backpressure state.
- Require positional alignment between optimized and original message
arrays before replay.
- Reject replay candidates that would serialize larger than the current
optimized messages.
- Add focused handler coverage for the decision-false tool-result
regression, bypass and backpressure paths, and outbound optimize-on
preservation.
- Add direct unit coverage for positional mismatch, no-inflation, and
JSON sizing-failure bailouts.
- Update the moved-cache-control and pure-block-append regression
fixtures to keep the no-inflation contract explicit.
- Run the unchanged OpenAI cache-stability preservation proof; no OpenAI
production code was edited.

## Testing

- [x] Unit tests pass (153 focused proxy, helper, cache-control,
block-append, cross-turn, byte-faithful, Anthropic, OpenAI, and
backpressure tests)
- [x] Linting passes (Ruff check and format validation on the seven
changed repository files)
- [ ] Type checking passes (`uv run mypy headroom`)
- [x] New tests added for new functionality when applicable
- [x] Manual testing performed with the in-process proxy and local stub
upstream

### Test Output

```text
python -m pytest tests/test_proxy/test_anthropic_no_optimize_history_passthrough.py tests/test_cache_prefix_overlay.py tests/test_cross_turn_cache_safety.py tests/test_cache_control_move_bust.py tests/test_proxy_byte_faithful_forwarding.py tests/test_proxy_anthropic_cache_stability.py tests/test_anthropic_pre_upstream_backpressure.py -q
python -m pytest tests/test_proxy_openai_cache_stability.py -q
python -m pytest tests/test_issue_2671_block_growth_cache.py::test_pure_append_replays_forwarded_blocks_and_advances_breakpoint -q
153 passed across focused invocations, exit code 0
optimize_off turn2_message_count=3 marker_count=1 outbound_compact_utf8_bytes=2293 client_compact_utf8_bytes=2293
optimize_on turn2_message_count=3 client_message_count=3 marker_count=0 outbound_compact_utf8_bytes=171 client_compact_utf8_bytes=182
python -m ruff check headroom/proxy/handlers/anthropic.py headroom/cache/prefix_tracker.py tests/test_proxy/test_anthropic_no_optimize_history_passthrough.py tests/test_cache_prefix_overlay.py tests/test_cache_control_move_bust.py tests/test_issue_2671_block_growth_cache.py tests/test_proxy_openai_cache_stability.py
All checks passed!, exit code 0

python -m ruff format headroom/proxy/handlers/anthropic.py headroom/cache/prefix_tracker.py tests/test_proxy/test_anthropic_no_optimize_history_passthrough.py tests/test_cache_prefix_overlay.py tests/test_cache_control_move_bust.py tests/test_issue_2671_block_growth_cache.py tests/test_proxy_openai_cache_stability.py --check
7 files already formatted, exit code 0

git diff --check
clean, exit code 0
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12 via `uv`, real Headroom proxy app
with a local stub Anthropic upstream
- Exact command / steps: send an actual optimize-on first request
through the in-process proxy with a deterministic production-pipeline
seam, then send a decision-false bypass turn containing a large
Claude-shaped `tool_result` with moved `cache_control`; separately send
an aligned optimize-on turn with a new suffix
- Observed result: the exact base checkout fails with `AssertionError:
assert 'compressed-tool-result' == 'large-tool-result-marker ...'`; the
guarded path passes with the client marker present once and outbound
compact JSON no larger than the client body. The optimize-on
preservation run records `optimize_on turn2_message_count=3
client_message_count=3 marker_count=0 outbound_compact_utf8_bytes=171
client_compact_utf8_bytes=182`, proving the actual compressed prefix is
outbound before the new suffix without turn-2 growth.
- Not tested: live Claude Code session against api.anthropic.com on this
host

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: cached-prefix replay now follows the
existing compression and backpressure decision and rejects misaligned or
inflating candidates.
- Kill switch / disable path: no new switch; the existing optimize and
bypass controls remain available.
- Unsafe override required: none.
- Qualification impact: none.
- Rollback path: revert the implementation commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes

## Additional Notes

`CHANGELOG.md` is not modified because Headroom's release automation
generates it from conventional commits.

This change does not add a context-limit guard or alter compression,
streaming tracker provenance, outbound-body selection, OpenAI behavior,
or provider limits. Local tests prove request-body ownership and replay
bounds. The reporter's live Claude Code completion and Anthropic token
acceptance remain external to this local proof.
 2026-08-17 15:02:18 -07:00
gglucass
def3d76e5a
fix(cache): mirror client cache_control positions instead of single-marker consolidation 
Preserve client cache-control breakpoint positions.
 2026-08-11 18:15:44 -07:00
Zhenjia ZHOU
8906d3a676
fix(cache): preserve client cache_control ttl when consolidating breakpoints (#2382) 
## Description

`normalize_message_cache_control()` consolidates message-level
`cache_control` breakpoints (strip all, re-place exactly one) to stay
under Anthropic's 4-block limit. The re-placed marker was hardcoded to
`{"type": "ephemeral"}`, so a client using 1-hour caching
(`cache_control: {"type": "ephemeral", "ttl": "1h"}`) was silently
downgraded to the 5-minute default on every consolidated turn — no
error, no signal, just quietly worse cache economics.

Fix: track the newest client marker while stripping, and re-place **that
marker verbatim** (a copy). Headroom keeps owning *where* the breakpoint
goes; the client keeps owning *what it says*. Older replayed markers
don't win — if the client's newest marker has no `ttl`, we don't
resurrect a stale `1h` (covered by a dedicated regression test).

Fixes #2375.

## Type of Change

- [x] Bug fix (silent 1h→5m cache downgrade)

## Changes Made

- `headroom/cache/prefix_tracker.py`:
`normalize_message_cache_control()` records the last marker dict seen in
message order and re-places a copy of it instead of a hardcoded
`{"type": "ephemeral"}`; docstring documents the ownership split.
- `tests/test_cache_control_move_bust.py`: 3 new tests — ttl preserved,
newest-marker-wins over stale ttls, ttl survives an 8-turn conversation
loop.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff` + `mypy`, CI-pinned settings)
- [x] Reproduced the bug first (2 new tests failed on the old code),
then verified the fix

### Test Output

```text
$ .venv/bin/python -m pytest tests/test_cache_control_move_bust.py -q
10 passed

# Before the fix, the two new ttl tests fail exactly as #2375 describes:
#   FAILED ...::test_normalize_preserves_ttl_of_newest_marker
#   FAILED ...::test_normalize_ttl_survives_many_turns

$ ruff check headroom/cache/prefix_tracker.py tests/test_cache_control_move_bust.py   # All checks passed!
$ ruff format --check <both files>                                                     # already formatted
$ mypy headroom/cache/prefix_tracker.py --ignore-missing-imports                       # Success: no issues
```

## Real Behavior Proof

- Environment: macOS (Darwin), Python in a uv venv, branch
`fix/cache-control-ttl-preserve` off `main` (`56c7d4a5`).
- Exact command / steps: drove `normalize_message_cache_control`
directly with a 2-message conversation whose marker carries `ttl: "1h"`,
printed the re-placed marker before/after the fix, and ran the new
regression tests against the unfixed code first.
- Observed result: before — output marker `{'type': 'ephemeral'}` (ttl
silently dropped); after — output marker `{'type': 'ephemeral', 'ttl':
'1h'}` with marker count still exactly 1 (the ≤4-block guarantee is
untouched).
- Not tested: a live Anthropic round-trip asserting
`cache_creation.ephemeral_1h_input_tokens` (needs a billed API call);
the marker dict forwarded on the wire is what the assertion pins.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — N/A
(docstring updated)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md — N/A

## Additional Notes

- The `test_normalize_newest_marker_wins_over_stale_ttl` test also
guards against over-fixing (e.g. "any 1h seen anywhere wins"), which
would pin users to 1h pricing after they switch back to the default.
 2026-07-18 09:52:22 -07:00
Tejas Chopra
48201345be
fix(proxy): keep cache_control bounded + stable so the freeze overlay stops busting (#1852) 
Follow-up to #1850. Two residual cache-bust sources, both
`cache_control`-related:

1. **Guard too strict.** `overlay_cached_prefix` decided "is this turn
an append-only extension?" by comparing whole message dicts — including
`cache_control`. Clients (Claude Code, litellm) move the cache
breakpoint to the newest message every call, so a marker landing in the
frozen prefix made the guard fail, the overlay skip its replay, and the
raw freeze forward ORIGINAL bytes over the cached COMPRESSED prefix →
partial bust (the ~42% residual on the a10 run, `prefix_change=0`). Fix:
run the append-only guard on **content only** (strip `cache_control`
before comparing) — content is what the provider's cache keys on.

2. **Marker accumulation.** The overlay replays the markers that rode on
each turn's then-newest message, so `cache_control` blocks pile up
~1/turn; Anthropic hard-errors at >4 total. Fix:
`normalize_message_cache_control` strips every message-level marker and
re-places a single ephemeral breakpoint on the last block (one
breakpoint caches the whole prefix; cache is content-keyed so re-placing
never busts). Wired into the Anthropic handler after the overlay.

**Per-provider (deliberately scoped):**
- **Anthropic**: `cache_control` markers → both fixes apply.
- **OpenAI**: AUTOMATIC prefix caching, no markers → overlay
(byte-identity) only; normalize is NOT applied (Anthropic markers on an
OpenAI request would be wrong).
- **Bedrock**: serves Claude via the pipeline but has no
cachePoint/freeze-replay path → not affected; a cachePoint analog would
be needed if caching is expanded.
- **Gemini**: explicit Cache API (`cachedContent`), no inline
markers/freeze → N/A.

> Stacked on #1850 — review that first; the diff against `main` includes
its overlay + `has_new_ccr_markers` work.

## Description

Keeps the freeze overlay's cache-safety intact against real clients that
relocate the `cache_control` breakpoint each turn, and prevents
`cache_control` blocks from accumulating past Anthropic's 4-marker
limit. See the two fixes above.

Closes #<!-- none --> — follow-up to #1850 (no separate issue).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cache/prefix_tracker.py`: append-only guard in
`overlay_cached_prefix` now compares **content only** (ignores
`cache_control`); new `normalize_message_cache_control()` collapses
message-level markers to a single ephemeral breakpoint on the last
block.
- `headroom/proxy/handlers/anthropic.py`: apply
`normalize_message_cache_control` after the overlay (Anthropic only).
- `tests/test_cache_control_move_bust.py`: reproduces the moved-marker
bust + proves both fixes.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed (local, see below)

### Test Output

```text
$ pytest tests/test_cache_control_move_bust.py -q
.......                                                                   [100%]
7 passed in 0.19s

# broader cache-safety suite (overlay + cross-turn + CCR deferred + openai/anthropic cache-stability + helpers)
$ pytest tests/test_cache_control_move_bust.py tests/test_cache_prefix_overlay.py \
    tests/test_cross_turn_cache_safety.py tests/test_proxy/test_anthropic_ccr_deferred_injection.py \
    tests/test_proxy_handler_helpers.py tests/test_proxy_openai_cache_stability.py \
    tests/test_proxy_anthropic_cache_stability.py -q
91 passed, 2 warnings in 29.98s

$ ruff check .          # ruff 0.15.17 (CI-pinned)
All checks passed!
$ ruff format --check . # ruff 0.15.17
1057 files already formatted
$ mypy headroom --ignore-missing-imports
Success: no issues found (changed modules: prefix_tracker, anthropic, openai, helpers)
```

## Real Behavior Proof

- **Environment:** local (`.venv`, Python 3.12), ruff 0.15.17 / mypy
pinned to CI versions.
- **Exact command / steps:** `tests/test_cache_control_move_bust.py`
drives the REAL tracker + freeze + `overlay_cached_prefix` +
`normalize_message_cache_control` across multiple append-only turns
where the client moves the `cache_control` breakpoint each turn.
- **Observed result:** with a moved marker in the frozen prefix, the
content-only guard keeps the overlay replaying (forwarded prefix stays
byte-identical → no bust); `cache_control` blocks stay ≤4 across many
turns and content is never altered. The reproduction test fails without
the fix and passes with it.
- **Not tested (this PR):** the end-to-end a10 SWE-bench run is the
field observation motivating fix #1 (~42% residual, `prefix_change=0`);
not re-run here.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

Stacked on #1850; land that first. Docs/CHANGELOG untouched (behavioral
cache-safety fix; no user-facing surface change). N/A: no screenshots
(no UI).
 2026-07-06 17:05:34 -07:00