mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
56 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Tejas Chopra
|
4f2e70a75c
|
fix(proxy/anthropic): authenticate and attribute buffered Copilot turns (#3277)
## Description Follow-up to #3258. That PR points the Anthropic target at the Copilot host so Claude models stop 401'ing. This PR fixes two things on the Anthropic path that were only ever correct on the **streaming** arm, and which #3258 makes reachable for real Copilot traffic. Copilot serves Claude models from its Anthropic surface (`/v1/messages`) on the same host as its OpenAI surface, so the resolved Anthropic target can be a Copilot host with no per-request `upstream_base_url` involved. That is the case both arms below get wrong. **1. The buffered arm sent no Copilot credential.** `apply_copilot_api_auth` is keyed on the upstream URL and was applied only by `_stream_response` (`handlers/streaming.py:1205`). The buffered/non-stream arm sends through `_retry_request` (`proxy/server.py:2132`), which forwards headers untouched — so the request carried whatever the client happened to send and none of Headroom's own credential handling: no minted or refreshed token (the one `wrap vscode` explicitly hands the proxy), no `Copilot-Integration-Id` default. A client token that went stale mid-session 401'd here while the streaming path recovered. That arm is not an edge case — it is the CCR `stream:true → buffered stream:false` flip, and Claude Code's non-stream retry. **2. Copilot turns were attributed to "anthropic".** `build_copilot_upstream_url` is the only place `mark_request_routed_to_copilot` fires (`copilot_auth.py:1288`), and `emit_request_outcome` relabels the provider off that flag (`proxy/outcome.py:419`). The buffered arm built its URL by f-string, skipping the chokepoint, so those turns showed as `anthropic` on the dashboard. The URL produced is byte-identical either way — this is attribution only, not routing. `proxy/cost.py` has no Copilot-specific branch, so pricing is unaffected. Both changes are inert off the Copilot path: `apply_copilot_api_auth` returns the headers unchanged for a non-Copilot URL, and `build_copilot_upstream_url` only joins base + path there. Independent of #3258 and based on `main` — the gaps are reachable today by setting `ANTHROPIC_TARGET_API_URL` to a Copilot host. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `handlers/anthropic.py`: build the default-target URL through `build_copilot_upstream_url` instead of an f-string, so the routed-to-Copilot flag is set for attribution. - `handlers/anthropic.py`: apply `apply_copilot_api_auth` on the buffered arm before the upstream send. Mutated in place, matching the accept-header handling directly above — the closures below capture `headers`, and the CCR continuation rebuilds its own header set from it, so the continuation inherits the auth too. - New test pinning both at the `_retry_request` seam: URL built, headers as they go on the wire, and the flag as it stands at send time. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`, CI-pinned 0.16.3) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality ### Test Output Both new assertions fail on `main` with exactly the symptoms described, and pass with the fix: ```text $ git stash && pytest tests/test_proxy/test_anthropic_copilot_upstream_auth.py tests/.../test_buffered_turn_to_copilot_is_authenticated E KeyError: 'authorization' tests/.../test_buffered_turn_to_copilot_is_flagged_for_attribution E assert False is True ==================== 2 failed, 2 passed, 1 warning in 3.38s ==================== $ git stash pop && pytest tests/test_proxy/test_anthropic_copilot_upstream_auth.py ========================= 4 passed, 1 warning in 2.88s ========================= ``` The two that pass on `main` are the invariants this must not break (path `/v1` preserved per #2409, non-Copilot target untouched). Regression run over the affected surface: ```text $ pytest tests/ -k "copilot or anthropic or outcome or provider_registry or proxy_routes or upstream" = 3 failed, 1111 passed, 33 skipped, 11112 deselected in 152.98s = ``` The 3 failures are `tests/test_proxy/test_openai_transport_path_prefix.py` and are **pre-existing on `main`** (verified by running that file on a clean checkout — same 3 fail). Untouched by this PR, which is Anthropic-path only. ```text $ uvx ruff@0.16.3 check headroom/proxy/handlers/anthropic.py tests/test_proxy/test_anthropic_copilot_upstream_auth.py All checks passed! $ mypy headroom/proxy/handlers/anthropic.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - **Environment:** macOS arm64, Python 3.12.13, `main` @ 0.36.5. - **Exact command / steps:** drive `POST /v1/messages` through the real app (`create_app` + `TestClient`, non-stream body) with the Anthropic target set to `https://api.githubcopilot.com`, intercepting `_retry_request` to capture what was about to go on the wire. Copilot token minting stubbed to a fixed value. - **Observed result:** before — no `Authorization` header at all on the buffered arm, and `request_routed_to_copilot()` is `False` at send time. After — `Authorization: Bearer <minted>` plus `Copilot-Integration-Id` and `Editor-Version`, flag `True`, URL unchanged at `https://api.githubcopilot.com/v1/messages`. With a non-Copilot target, no credential is invented and the flag stays `False`. - **Not tested:** against live `api.githubcopilot.com` — no Copilot subscription in this environment. Token minting is stubbed, so the refresh path itself is exercised only to the provider boundary. Anthropic **batch** endpoints (`/v1/messages/batches`, `handlers/anthropic.py:5066+`) still build against `self.ANTHROPIC_API_URL` and will point at Copilot, which does not serve them — pre-existing and out of scope here — filed as #3278. ## Runtime Rollout Safety - **Rollout-managed feature(s):** none — no flag or channel involved. - **Minimum rollout channel:** n/a. - **Stable/default behavior changed:** no, for every non-Copilot upstream: the URL is byte-identical and `apply_copilot_api_auth` early-returns for non-Copilot URLs. Behavior changes only when the Anthropic target is a Copilot host, which is the broken case. - **Kill switch / disable path:** set `ANTHROPIC_TARGET_API_URL` to a non-Copilot host; both paths go inert. - **Unsafe override required:** none. - **Qualification impact:** none. - **Rollback path:** revert this commit — it is self-contained to one file plus a new test. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tejas Chopra
|
826b600c9b
|
feat(proxy): self-limiting session state for the compression-cache registry (#3261)
## Problem The per-session `CompressionCache` registry (the map that replays previously-compressed messages byte-identically so the provider prefix cache stays warm) had no lifetime management: - Idle/dead sessions lived forever until the hardcoded 500-session cap was hit. - At capacity, eviction dropped the oldest-**created** quarter — which could wipe the busiest long-lived session (busting every one of its prefixes at once) while dead sessions survived. - Neither the cap nor any TTL was tunable, which blocks gateway deployments (e.g. Kong sidecar/pool) fanning many concurrent sessions into one process. ## Changes - **Idle-TTL sweep**: sessions idle longer than `HEADROOM_COMPRESSION_CACHE_TTL_SECONDS` (default 3900s) are evicted by a lazy sweep, at most once per 60s, piggybacked on `_get_compression_cache` — same pattern as `PrefixCacheTrackerRegistry._maybe_cleanup`, no background task. `last_seen` refreshes on **every** access, so an active session never expires. - **LRU capacity eviction**: the registry is now an access-ordered `OrderedDict`; capacity pressure sheds the *idlest* quarter, never a busy session. - **Tunable cap**: `HEADROOM_COMPRESSION_CACHE_MAX_SESSIONS` (default 500, floor 1). ## Why 3900s Eviction is bust-free only once the provider's own prompt cache has lapsed. Providers don't expose their cache TTLs, and the risk is one-sided (late eviction costs a few MB; early eviction *causes* the bust this state exists to prevent), so the default is the upper bound of documented lifetimes across providers — Anthropic's 1h extended breakpoint, OpenAI's "up to an hour off-peak", Gemini's 60-min default — plus 5m grace. A parse-time floor of 600s keeps the TTL from ever dropping below the prefix tracker's session TTL: after the tracker expires, the byte-identical swap is the only remaining protection for a still-live provider prefix. Read-hit signals are untouched: they govern the freeze boundary, never eviction — `read_hits == 0` usually means cold start or TTL lapse, where the map was just (re)written into the provider cache and deleting it would guarantee a second bust. ## Behavior impact - Steady state (any session active within the TTL): zero change — same instances, same bytes, same freeze behavior. - A session returning after >65 min idle now finds its map evicted — but every provider had already forgotten its prefix by then, so that turn was paying the cache-write price regardless (fail-open, no failed requests). - Capacity eviction now protects busy sessions instead of punishing them. ## Testing - New `tests/test_compression_cache_registry.py`: LRU-not-FIFO capacity eviction, small-cap edge case, TTL sweep eviction, access-refreshes-clock, sweep rate limiting. - 386 tests pass across compression-cache, cache-stability (Anthropic + OpenAI), prefix-overlay, cold-start, cache-mode, and Bedrock-tracker suites; ruff check/format clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01EWKCmcH47hvvoQ35wftXhE --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Tejas Chopra
|
1f96dabc19
|
fix(security): address u9up assessment findings (WEB-01–07) (#2207)
Hardens client-selected upstreams, memory identity resolution, downloaded binary integrity, telemetry import, Docker defaults, Neo4j credentials, and archive extraction. Refreshes the branch against current main and preserves newer same-origin and loopback protections. |
||
|
Rod Boev
|
c16be9bbbe
|
fix(proxy/anthropic): don't replay recorded prefix over live history (#3026) (#3052)
## Description A Claude Code session that reads a large tool result through `headroom proxy` can fail on turn 2 with Anthropic's `400 prompt_too_long`. The reporter's controlled comparison completed eight turns through 0.33.0 with 187,986 input tokens, while 0.35.0 failed after five requests with 753,077 input tokens. The local regression uses an actual prior optimized request to populate tracker state, then a decision-false bypass turn with Claude-shaped tool-result content. The old unconditional replay path substitutes the compressed prefix; the eligibility gate preserves the client's outbound body without claiming a live provider reproduction. The Anthropic `/v1/messages` route computes whether a request should be compressed, but cached-prefix replay currently runs outside that decision. The replay helper also derives its prefix length from the original message list and applies that index to the optimized list without proving the two lists still align. A stale forwarded prefix can therefore be grafted onto the wrong positions and enlarge later requests. This change limits replay to requests whose existing compression decision permits it and whose pre-upstream backpressure path is inactive. It also makes `overlay_cached_prefix()` decline misaligned or inflating candidates while preserving normal append-only replay. Reported by @itsumonotakumi, whose controlled comparison isolated the failure from compression, headers, one-request serialization, memory, code graph, and CCR. Closes #3026 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Gate Anthropic cached-prefix replay on the existing `CompressionDecision.should_compress` result and the existing pre-upstream backpressure state. - Require positional alignment between optimized and original message arrays before replay. - Reject replay candidates that would serialize larger than the current optimized messages. - Add focused handler coverage for the decision-false tool-result regression, bypass and backpressure paths, and outbound optimize-on preservation. - Add direct unit coverage for positional mismatch, no-inflation, and JSON sizing-failure bailouts. - Update the moved-cache-control and pure-block-append regression fixtures to keep the no-inflation contract explicit. - Run the unchanged OpenAI cache-stability preservation proof; no OpenAI production code was edited. ## Testing - [x] Unit tests pass (153 focused proxy, helper, cache-control, block-append, cross-turn, byte-faithful, Anthropic, OpenAI, and backpressure tests) - [x] Linting passes (Ruff check and format validation on the seven changed repository files) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [x] Manual testing performed with the in-process proxy and local stub upstream ### Test Output ```text python -m pytest tests/test_proxy/test_anthropic_no_optimize_history_passthrough.py tests/test_cache_prefix_overlay.py tests/test_cross_turn_cache_safety.py tests/test_cache_control_move_bust.py tests/test_proxy_byte_faithful_forwarding.py tests/test_proxy_anthropic_cache_stability.py tests/test_anthropic_pre_upstream_backpressure.py -q python -m pytest tests/test_proxy_openai_cache_stability.py -q python -m pytest tests/test_issue_2671_block_growth_cache.py::test_pure_append_replays_forwarded_blocks_and_advances_breakpoint -q 153 passed across focused invocations, exit code 0 optimize_off turn2_message_count=3 marker_count=1 outbound_compact_utf8_bytes=2293 client_compact_utf8_bytes=2293 optimize_on turn2_message_count=3 client_message_count=3 marker_count=0 outbound_compact_utf8_bytes=171 client_compact_utf8_bytes=182 python -m ruff check headroom/proxy/handlers/anthropic.py headroom/cache/prefix_tracker.py tests/test_proxy/test_anthropic_no_optimize_history_passthrough.py tests/test_cache_prefix_overlay.py tests/test_cache_control_move_bust.py tests/test_issue_2671_block_growth_cache.py tests/test_proxy_openai_cache_stability.py All checks passed!, exit code 0 python -m ruff format headroom/proxy/handlers/anthropic.py headroom/cache/prefix_tracker.py tests/test_proxy/test_anthropic_no_optimize_history_passthrough.py tests/test_cache_prefix_overlay.py tests/test_cache_control_move_bust.py tests/test_issue_2671_block_growth_cache.py tests/test_proxy_openai_cache_stability.py --check 7 files already formatted, exit code 0 git diff --check clean, exit code 0 ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12 via `uv`, real Headroom proxy app with a local stub Anthropic upstream - Exact command / steps: send an actual optimize-on first request through the in-process proxy with a deterministic production-pipeline seam, then send a decision-false bypass turn containing a large Claude-shaped `tool_result` with moved `cache_control`; separately send an aligned optimize-on turn with a new suffix - Observed result: the exact base checkout fails with `AssertionError: assert 'compressed-tool-result' == 'large-tool-result-marker ...'`; the guarded path passes with the client marker present once and outbound compact JSON no larger than the client body. The optimize-on preservation run records `optimize_on turn2_message_count=3 client_message_count=3 marker_count=0 outbound_compact_utf8_bytes=171 client_compact_utf8_bytes=182`, proving the actual compressed prefix is outbound before the new suffix without turn-2 growth. - Not tested: live Claude Code session against api.anthropic.com on this host ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: N/A. - Stable/default behavior changed: cached-prefix replay now follows the existing compression and backpressure decision and rejects misaligned or inflating candidates. - Kill switch / disable path: no new switch; the existing optimize and bypass controls remain available. - Unsafe override required: none. - Qualification impact: none. - Rollback path: revert the implementation commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes ## Additional Notes `CHANGELOG.md` is not modified because Headroom's release automation generates it from conventional commits. This change does not add a context-limit guard or alter compression, streaming tracker provenance, outbound-body selection, OpenAI behavior, or provider limits. Local tests prove request-body ownership and replay bounds. The reporter's live Claude Code completion and Anthropic token acceptance remain external to this local proof. |
||
|
Tejas Chopra
|
c502087db7
|
fix(ccr): only buffer a stream when a marker is actually redeemable (#3092)
## Description Closes #3071 `headroom_retrieve` is injected once and kept resident for the session so the tools array stays byte-stable and the prompt cache survives. The buffered-CCR path keyed on that tool merely being **present**, so once a session went sticky, *every* later streaming turn was silently converted to `stream: false`, buffered whole, and resynthesized as SSE: ``` CCR: stream:true request has headroom_retrieve available; using buffered stream:false upstream request ``` Buffering leaves time-to-last-byte roughly unchanged but makes **time-to-first-byte the entire generation**. The reporter measured 8s average and up to 100s across 234 requests in one day — turns that would have streamed a first token in ~1s instead delivered nothing until done. Retrieval can only expand a `<<ccr:...>>` marker present in the outgoing body, so a turn carrying none cannot benefit from the buffered path at all. Gate on that instead of on the tool. This is also the root cause #3082 traced independently from the OpenCode side — its plugin registers `headroom_retrieve` unconditionally, so *every* turn buffered and neither `--no-ccr` nor `HEADROOM_NO_CCR` stopped it. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - New `_outgoing_body_has_redeemable_marker()` scans the body **about to go on the wire** and verifies ownership against the compression store — the same `exists()` check the retrieve endpoint performs, so a same-shaped marker from another context tool is not adopted (#2836). Unexpected shapes answer `True`, keeping the long-standing behavior. - The buffered-stream decision site gates on it, and logs at INFO when it skips buffering. - The correctness detail worth reviewing: the check reads `body`, **not** the earlier `scan_for_markers(optimized_messages)` result. `optimized_messages` is reassigned five times after that scan (memory hooks, pre-send extensions, tool-search repair, CCR repair), so reusing it would have been stale. - Two existing test files encoded the very coupling this removes and had to be repaired — see Testing. Scope: this narrows *when* buffering happens; it does not make buffered turns stream. A turn that genuinely carries a marker still loses incremental delivery — restoring streaming there means wiring `StreamingCCRHandler`, which is #3069's scope. It does not fix #3088 either, whose requests do carry markers. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] New tests added for new functionality - [x] Manual testing performed Two existing files built a request with `headroom_retrieve` and **no** marker, relying on the tool alone to trigger buffering: - `tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py` (11 tests) fell through to the live streaming path, where only `_retry_request` is stubbed — so the requests reached the network and the file **hung indefinitely** rather than failing. Seeded real markers; now passes in ~5s. - `tests/test_proxy_response_cache_replay.py::test_buffered_ccr_turn_does_not_write_the_response_cache` asserts its own premise (*"the conversion really happened — otherwise this test proves nothing"*), so it failed loudly instead of passing vacuously. Seeded a marker. New `test_buffering_is_gated_on_a_redeemable_marker` pins all three directions: owned marker → buffered, no marker → streaming, foreign marker → streaming. ### Test Output ```text $ pytest tests/test_ccr_buffered_stream_signed_thinking.py -q 8 passed, 1 warning in 4.45s $ pytest tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py -q 11 passed, 1 warning in 4.94s # was: hung indefinitely $ pytest tests/test_proxy_response_cache_replay.py -q 9 passed, 1 warning in 1.69s $ pytest tests/ -q 3 failed, 11151 passed, 581 skipped in 398.28s (0:06:38) Same 3 failures as a clean-main baseline run on this machine: tests/test_graceful_shutdown.py::test_run_server_installs_cancelled_error_filter tests/test_learn/test_integration.py::TestCodexIntegration::test_full_pipeline tests/test_release_workflows.py::test_no_native_tls_in_wheel_build_tree $ ruff check . && ruff format --check . All checks passed! ``` ## Real Behavior Proof - Environment: this branch driven through the real FastAPI app with the outbound HTTP client captured; macOS arm64, Python 3.12. - Exact command / steps: posted a `stream: true` `/v1/messages` request carrying a resident `headroom_retrieve` tool in three variants — no marker, a marker seeded into the compression store, and a correctly-shaped marker the store does not own — recording whether `_retry_request` saw a `stream: false` body. - Observed result: no marker → **streams**, `_retry_request` never sees a flipped body; owned marker → **buffers**, exactly as before; foreign marker → streams, honoring #2836 rather than adopting another tool's hash. - Not tested: the latency improvement against live client traffic. The mechanism is verified (the buffered conversion no longer occurs), but the reported 8s → ~1s TTFB needs the reporter's traffic to confirm. ## Runtime Rollout Safety - Rollout-managed feature(s): none — this narrows an existing code path and is not behind a rollout channel. - Minimum rollout channel: n/a (ships to stable with the fix). - Stable/default behavior changed: yes. A streaming turn whose body carries no redeemable marker now stays streaming instead of being buffered. Turns carrying a marker are unchanged. - Kill switch / disable path: no new switch. Existing CCR controls still apply — disabling the CCR response handler bypasses this decision site entirely, and the helper fails open (returns `True`, i.e. the old behavior) on any unexpected message shape. - Unsafe override required: no. - Qualification impact: none — no qualification-gated surface is touched. - Rollback path: revert this commit. Note it also carries two test repairs; reverting the production change alone would leave those tests passing but vacuous. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Documentation update is marked N/A: no user-facing flag or endpoint changes. Type checking (`mypy headroom`) was not run separately; `ruff` is the gate this repo's CI enforces. Related: #2836 (marker ownership), #3069 (streaming CCR handler), #3082, #3088. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
JD Davis
|
8a1d38bc5d
|
fix(proxy): complete stateless Responses and buffered CCR lifecycle (#2997)
## Description
Consolidates the related OpenAI Responses ZDR/stateless continuation and
buffered CCR response-lifecycle corrections on current main. It
preserves client storage policy, makes Headroom-owned continuations
stateless across HTTP and WebSocket, and prevents buffered streaming
paths from committing a false HTTP 200 before the real upstream outcome
is known.
Closes #2675
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Preserves explicit and omitted Responses `store` policy instead of
forcing provider storage or disabling memory tools.
- Replays normalized input, replayable outputs, encrypted reasoning
content, and Headroom function outputs without `previous_response_id`.
- Applies the same stateless continuation policy to HTTP and WebSocket.
- Prevents transparent memory execution after client-visible WebSocket
output.
- Delays buffered CCR ASGI status/headers until the operation resolves
for Anthropic Messages and OpenAI Responses.
- Preserves real 429/5xx status and retry headers.
- Converts malformed non-JSON/non-SSE upstream 200 replies to a
sanitized 502 protocol error.
- Preserves valid JSON-to-SSE synthesis and existing SSE adaptation.
- Removes unreachable task cleanup left behind after replacing the old
keepalive polling loop with a direct awaited operation.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
118 passed across the changed HTTP/WS ZDR, lifecycle, and both-provider CCR suites
11526 tests collected with no collection errors
ruff check .: All checks passed
ruff format --check .: 1411 files already formatted
mypy headroom/proxy/handlers/anthropic.py headroom/proxy/handlers/openai.py:
Success: no issues found in 2 source files
```
Exact-head CI is entirely green on
`
|
||
|
Abhay Singh
|
7de35739c6
|
fix(proxy/anthropic): repair headroom_retrieve history references the tools array cannot support (#2876)
## Description #2805 / #2807 established the mechanism: Claude Code replays one transcript across requests that carry different `tools` arrays, and Anthropic validates every history reference against the array of the request at hand. #2807 fixed it for tool-search blocks by repairing history (`strip_unsupported_tool_search_blocks`) rather than trying to predict the client's tool set. The same mechanism applies to CCR's `headroom_retrieve`, and it is tool-agnostic. A passthrough side-request (the prompt-type Stop hook evaluator, `/compact`) that the proxy forwards without declaring `headroom_retrieve` still carries a historical `tool_use` naming it, and Anthropic 400s on the dangling reference. The injection-side fixes (#2766 / #2533) decide *when to re-declare the tool*; this makes the 400 *structurally impossible* where the tool is intentionally absent. It is belt-and-braces with them, not a replacement. The fix adds the symmetric repair next to #2807's. When the outbound `tools` array does not declare `headroom_retrieve`, it replaces each `headroom_retrieve` `tool_use` and its paired `tool_result` with a text block, so no dangling reference survives. It **neutralizes** (replaces in place) rather than **drops**, which is the one deliberate difference from #2807: CCR's `tool_use` lives in an assistant turn and its `tool_result` in the next user turn, i.e. two different messages. Dropping a whole message could leave two same-role messages adjacent and break Anthropic's strict user/assistant alternation, turning one 400 into another. Replacing blocks in place keeps every message and role intact, and preserves the retrieved text the model already saw. #2807's server-tool blocks both live in the same assistant turn, so dropping was safe there. It runs after CCR tool injection, so on the main loop -- where the tool IS injected (a present marker) -- it neutralizes nothing and the prompt-cache prefix is untouched, mirroring #2807's placement and sequencing. Fixes #2814 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/helpers.py`: added `strip_unsupported_ccr_retrieve_blocks(messages, tools)` (and a small `_ccr_result_as_text` helper). No-ops (returning the original object by identity) when `headroom_retrieve` is declared or no such history exists; otherwise neutralizes the `tool_use` and its paired `tool_result` to text. - `headroom/proxy/handlers/anthropic.py`: call the repair right after the tool-search history repair (which is after CCR tool injection), guarded on it actually changing anything, tagged `router:ccr_retrieve_repair:Nblocks`. - `tests/test_ccr_retrieve_history_repair.py`: 5 unit tests (no-op when declared, no-op without retrieve history, neutralize + preserve result text + keep alternation, leave foreign tool_use untouched, placeholder when the result has no text). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text tests/test_ccr_retrieve_history_repair.py 5 passed # Broader CCR / tool-search / handler suites (unchanged behavior): tests/test_ccr_retrieve_history_repair.py tests/test_proxy/test_anthropic_ccr_deferred_injection.py tests/test_issue_746_tool_search.py 71 passed # uvx ruff@0.15.17 check -> All checks passed! # uvx mypy@1.20.2 headroom/proxy/helpers.py -> Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1, ruff 0.15.17 and mypy 1.20.2 via uvx. - Exact command / steps: confirmed the injection point (`apply_session_sticky_ccr_tool`) and the tool-search repair placement in `handlers/anthropic.py`, confirmed `body["tools"]` reflects the CCR injection before the repair call site (`body["tools"] = tools` is written well upstream and the adjacent tool-search repair already relies on it), then drove the helper over a transcript with a `headroom_retrieve` tool_use + paired tool_result: with the tool declared it returns the original object unchanged; with the tool absent it neutralizes both blocks, preserves the result text, and keeps the message roles/count identical. - Observed result: a forwarded request that would 400 with "Tool reference 'headroom_retrieve' not found in available tools" now carries text blocks in place of the retrieve `tool_use`/`tool_result`, so there is no reference for Anthropic to reject, and user/assistant alternation is preserved. The main loop (tool present) is a no-op. - Not tested: a live multi-turn Claude Code session hitting a Stop-hook/`/compact` side-request against a real provider (no live provider here). The repair is a pure function verified directly over the exact block shapes Anthropic validates, and it mirrors the already-merged tool-search repair's mechanism and wiring. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md`: it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes The issue reporter noted their own logs show the tool-search variant of this 400 (61 across 11 days) but zero `headroom_retrieve` occurrences, because they run `HEADROOM_LOSSLESS=1` which disables CCR entirely. This PR fixes the CCR variant of the same, proven, tool-agnostic mechanism rather than a fresh CCR repro. The neutralize-vs-drop choice is the one place I departed from #2807, for the alternation reason above; if you would rather it drop (accepting the alternation handling that implies), I am happy to switch it. --------- Co-authored-by: Jerrett Davis <mxjerrett@gmail.com> |
||
|
Ashish Patel
|
41dab2d099
|
fix(ccr): verify a scanned marker's hash before advertising it (#2908)
## Description
`CCRToolInjector.scan_for_markers()` decides whether a compression
marker is Headroom's own by *shape* alone — any bracket marker carrying
a 24-hex hash counts, per the generic fallback pattern
(`\[.*?compressed.*?hash=([a-f0-9]{24})\]`). Other context tools emit
exactly that shape. Once a foreign hash is scanned,
`has_compressed_content` flips true and the retrieve tool + "Available
hashes" system instruction get injected for a hash this proxy never
stored — the model calls `headroom_retrieve`, gets a guaranteed miss,
and re-does work it already had. Two wasted turns per adopted foreign
hash.
Closes #2836
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/ccr/tool_injection.py`: added
`CCRToolInjector.verify_ownership()` — filters `detected_hashes` down to
hashes the compression store actually recognizes, via the same
`store.exists()` check the retrieve endpoint itself performs. Added a
small `_HashOwnershipStore` Protocol (structural typing, not a hard
dependency on the concrete `CompressionStore` class) and a
`compression_store` constructor field for dependency injection/testing.
`scan_for_markers()` itself is untouched — kept store-independent (pure
regex) rather than baking the check into the scan loop, since that
approach broke 24 existing tests that correctly test "does this shape
match" in isolation.
- `headroom/proxy/handlers/anthropic.py`,
`headroom/proxy/handlers/openai.py`: call `injector.verify_ownership()`
right after `scan_for_markers()` — the two real per-request call sites.
- `verify_ownership()` is also called inside `process_request()` (the
convenience wrapper `batch.py`'s Google path uses), so that path is
covered without a separate call site edit.
- `tests/test_ccr_tool_injection.py`: new `TestVerifyOwnership` class (6
tests) — the exact issue repro, a real-hash-survives case, mixed
own/foreign hashes, explicit store override, store-exception safety
(must not raise), and no-op-on-empty-hashes.
- `tests/test_proxy_anthropic_cache_stability.py`: 3 pre-existing tests
needed updating for the new (correct) behavior — two `_FakeInjector`
test doubles needed a `verify_ownership()` stub added, and one real
end-to-end test needed a genuine store entry seeded (via
`explicit_hash`) for the hash its hand-typed marker references, instead
of asserting on an unverified shape-only match.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) — not run locally, will
confirm via CI
- [x] New tests added for new functionality
- [x] Manual testing performed (see Real Behavior Proof)
### Test Output
```text
$ .venv/Scripts/python -m pytest tests/test_ccr_marker_policy.py tests/test_ccr_tool_always_on.py tests/test_ccr_tool_injection.py tests/test_proxy/test_ccr_frozen_prefix_coupling.py tests/test_proxy_anthropic_cache_stability.py tests/test_proxy_handlers_batch.py tests/test_proxy_handler_helpers.py -q
151 passed in 15.87s
$ .venv/Scripts/python -m pytest tests/test_proxy_ccr.py tests/test_proxy_openai_responses_stream_ccr.py tests/test_anthropic_ccr_workspace_unbound.py tests/test_compression_store.py tests/test_no_ccr_lossy.py tests/test_proxy_handlers_batch.py -q
121 passed in 20.73s
$ .venv/Scripts/ruff check . && .venv/Scripts/ruff format --check . # touched files only
All checks passed / already formatted
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.14.5, local venv
- Exact command / steps: ran the issue's exact 3-line repro
(`CCRToolInjector.scan_for_markers()` on the foreign marker text, then
`verify_ownership()`) before and after the fix; separately verified a
genuinely-Headroom-stored hash (via `store.store(...,
explicit_hash=...)`) still survives verification and still drives
injection
- Observed result: before the fix (scan only, no verify step exists yet)
`has_compressed_content` is `True` for the foreign marker — matches the
bug report exactly. After adding `verify_ownership()`: foreign marker →
`detected_hashes == []`, `has_compressed_content is False`; real stored
hash → `detected_hashes == [real_hash]`, `has_compressed_content is
True`.
- Not tested: have not driven this through a live two-context-tool proxy
session (e.g. Headroom alongside another CCR-shaped tool in the same
conversation) — verified at the unit/integration level (the exact repro
plus the real proxy handler call sites via
`test_proxy_anthropic_cache_stability.py`'s end-to-end `TestClient`
tests), not via a live multi-tool session.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A —
internal CCR safety behavior, no user-facing docs reference the
marker-adoption mechanism)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable (release-please
generates this automatically from commit messages)
## Additional Notes
Design note on why `verify_ownership()` is a separate step rather than
baked into `scan_for_markers()`: my first attempt did exactly that and
broke 24 tests across `test_ccr_tool_injection.py`,
`test_ccr_marker_policy.py`, `test_proxy_anthropic_cache_stability.py`,
and `test_proxy_handler_helpers.py` — all of them legitimately testing
"does the regex detect this marker shape" independent of any store
state. Keeping the scan pure and adding an explicit, separately-testable
verification step kept that test surface intact while still closing the
real gap at the three places that actually decide whether to advertise
the retrieve tool.
|
||
|
Abhay Singh
|
2b5ee7cde8
|
fix(proxy/anthropic): None-guard usage token counts on the direct buffered path (#2434)
## Description
The direct (non-backend) Anthropic buffered `/v1/messages` path reads
token counts from the response usage to record metrics and update the
prefix tracker:
```python
usage = resp_json.get("usage", {})
output_tokens = usage.get("output_tokens", 0)
cr_tokens = usage.get("cache_read_input_tokens", 0)
cw_tokens = usage.get("cache_creation_input_tokens", 0)
...
uncached_input_tokens = usage.get("input_tokens", 0)
```
`.get(key, default)` only falls back when the key is **absent**. When a
key is present with a **null** value, `.get` returns `None`. The direct
Anthropic API always sends integer usage, but this same handler serves
any Anthropic-compatible upstream reached through a custom
`ANTHROPIC_TARGET_API_URL` gateway (the scenario `install apply` now
supports), and such a gateway can emit null counts on a stopped or empty
turn.
Those `None`s then reach `max(0, expected_cached - cr_tokens)` in the
cache-bust block and the int-typed `RequestOutcome` / metrics recorder,
so a single such response raises an uncaught `TypeError` and 502s the
request. This is the same class as the Gemini crash fixed in #2347 and
the OpenAI chat path.
## Fix
Coerce the four counts with `int(... or 0)` at the direct-path
usage-extraction site, matching `_extract_anthropic_cache_ttl_metrics`
(which already guards its TTL buckets this way) and the Gemini fix. A
normal integer usage is unchanged; only a null (or absent) value now
becomes 0.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/handlers/anthropic.py`: `int(... or 0)`-guard
`output_tokens` / `cache_read_input_tokens` /
`cache_creation_input_tokens` / `input_tokens` at the direct
buffered-path usage-extraction site.
- `tests/test_proxy/test_anthropic_buffered_timeout.py`: regression
driving a buffered `/v1/messages` request whose upstream usage reports
null counts, asserting a 200 instead of a 502.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_proxy/test_anthropic_buffered_timeout.py -q
# all pass
# with the fix reverted, the new test fails (the null-usage response 502s):
$ git stash push -- headroom/proxy/handlers/anthropic.py
$ python -m pytest "tests/test_proxy/test_anthropic_buffered_timeout.py::test_anthropic_messages_buffered_survives_null_usage_counts" -q
1 failed (TypeError: unsupported operand type(s) for +: 'NoneType' and 'NoneType')
$ uvx ruff@0.15.17 check headroom/proxy/handlers/anthropic.py tests/test_proxy/test_anthropic_buffered_timeout.py
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, project venv (`uv sync --extra
proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`, pytest in the venv.
- Exact command / steps: ran the new FastAPI `TestClient` regression,
which drives the real direct buffered `/v1/messages` handler with
`proxy._retry_request` returning a 200 whose `usage` has null
`input_tokens` / `output_tokens` / `cache_read_input_tokens` /
`cache_creation_input_tokens`; then reverted only `anthropic.py` and
re-ran.
- Observed result: with the fix the request returns 200; with the fix
reverted the same request 502s with `TypeError: unsupported operand
type(s) for +: 'NoneType' and 'NoneType'`. Ran against the actual
handler via the app.
- Not tested: a live third-party Anthropic-compatible gateway emitting
null usage.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
Tejas Chopra
|
d0c1f5b8ad
|
fix(ccr): avoid injecting tool on chat streaming
Avoid unsupported CCR tool injection on OpenAI chat streaming (#2924). |
||
|
AxelRay
|
de9e0523da
|
fix(settings): accept documented HEADROOM_* env names as settings keys (#2833)
## Description Settings validation only accepted short JSON/API keys, so documented HEADROOM_* env names were rejected as unknown. Users following the docs (for example HEADROOM_LOSSLESS) hit SettingsValidationError / PUT /settings 400 even though those names are already on each registry field. This normalizes known env aliases to their short keys before validate/save, keeps existing short-key behavior, and rejects conflicting env+key pairs for the same field. Closes #2812 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added _BY_ENV and _normalize_values() in settings_store to map documented env names to short keys - Call normalization at the start of validate() and save() so clear/retain paths also accept env aliases - Reject payloads that supply both an env alias and its short key with different values - Add unit coverage for accept/clear/conflict/same-value paths and update registry monkeypatches to rebuild _BY_ENV ## Testing - [x] Unit tests pass (pytest) - [x] Linting passes (ruff check .) - [ ] Type checking passes (mypy headroom) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text python -m pytest tests/test_proxy/test_settings_store.py -q -k "env_alias or validate_accepts or save_rejects or same_env or conflicting or save_accepts or env_alias_clear or anthropic_extra_headers_retain or TestValidation" 23 passed, 11 deselected ruff check headroom/settings_store.py tests/test_proxy/test_settings_store.py tests/test_proxy_settings_endpoints.py All checks passed! ruff format --check headroom/settings_store.py tests/test_proxy/test_settings_store.py tests/test_proxy_settings_endpoints.py 3 files already formatted ``` ## Real Behavior Proof - Environment: Linux x86_64, Python 3.14.5 via contributor venv, worktree of headroom main at |
||
|
gglucass
|
c6f99482e1
|
fix(proxy/anthropic): run tool-search history repair after turn hooks
## Description
`strip_unsupported_tool_search_blocks` (#2807) validates every replayed
`tool_reference` in the transcript against the request's `tools` array.
It ran *before* the turn-hooks block in `handlers/anthropic.py`, and a
registered turn hook may rewrite that array — the hook surface is
documented as "a registered hook may inspect or rewrite the outbound
tools/messages before we send upstream".
So a hook that drops a tool named by a replayed reference leaves the
repair having validated against a stale view, and upstream rejects the
request:
```text
400 Tool reference 'X' not found in available tools
```
The repair's correctness argument is that it validates against exactly
the `tools` array upstream will see. That was true at the old call site
and stopped being true one block later.
### Fix
Move the repair to after the turn-hooks block, so it is the last stage
that can invalidate a reference:
- It still runs **after** the deferral injection, so the tool just
injected counts as present — the main loop strips nothing and the frozen
prefix stays byte-identical.
- Nothing past the new call site mutates `body["tools"]` on the outbound
path. (The two later `continuation_body["tools"]` assignments build a
*derived* body from the already-repaired `body`, so they inherit the
repair.)
- It still runs before the consistency token re-count, so `tok_after`
continues to reflect the repaired messages.
- It remains unconditional (not gated on `HEADROOM_TOOL_SEARCH`, not
gated on `_bypass`), so transcripts poisoned before the flag was turned
off still recover.
`strip_unsupported_tool_search_blocks` is copy-on-write and returns the
original `messages` object by identity when nothing is removed, so
relocating the call does not change the no-op path.
### Severity
Latent. No turn hook ships in-tree, so this cannot fire on a default
install — it is reachable only through a third-party registered hook
that shrinks the tools array. Filing the fix now so the ordering
constraint is enforced by a test rather than rediscovered.
Closes #2888
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/handlers/anthropic.py`: the tool-search history repair
block moves from just after the deferral injection to just after the
turn-hooks block. The comment now states the ordering constraint in both
directions (after injection, after hooks) so the next person to add a
stage knows where the boundary is. No logic change.
- `tests/test_proxy/test_tool_search_repair_after_turn_hooks.py` (new):
two handler-level regressions. Ordering is the whole property under
test, so a unit test of the helper cannot see it — these drive the real
handler through `TestClient` and assert on the forwarded body.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed — no in-tree turn hook exists to exercise
this against a live API key; the handler-level test below is the
substitute, see Not tested.
### Test Output
```text
$ uv run --extra dev pytest tests/test_proxy/ -q
======================= 241 passed, 1 warning in 35.82s ========================
$ uvx ruff check headroom tests
All checks passed!
$ uv run --extra dev mypy headroom
Success: no issues found in 515 source files
```
## Real Behavior Proof
- Environment: macOS 15 (darwin 24.6.0), Python 3.10.18, pytest 9.0.3,
in a worktree off `upstream/main` at
|
||
|
Tejas Chopra
|
f624d3a00a
|
perf(proxy): bound upstream calls and hot-path costs (#2852)
Seven commits from one week of load testing: one hang, two request-path correctness fixes, and four hot-path costs that only show up in production. ## Reliability **Bound every upstream call.** The litellm backend had no timeout at all, so a request the upstream never answered blocked its caller forever. Observed under load on 2026-08-07: four agent workers on ESTABLISHED connections for 36+ minutes while `/readyz` answered in 0.11s. No error, no retry, no log line — indistinguishable from slow work, which is the worst shape a failure can take. A float rather than an `httpx.Timeout`, deliberately: litellm expands a float across all four httpx phases, so on a streaming call it becomes the maximum gap *between chunks*, not a cap on total generation. A long answer streaming steadily is never cut off; a stalled one dies. Default 600s via `HEADROOM_UPSTREAM_TIMEOUT`; 0, negative, and junk fall back to the default rather than meaning "no timeout". **Keep the consistency re-count off the event loop.** It ran `tokenizer.count_messages` twice directly on the loop. Since Claude counting moved to a real BPE that is CPU-bound work stalling every other in-flight request — ~1s on a 2.3 MB body, with `/healthz` gaps tracking body size. Offloaded via `asyncio.to_thread` on the same tokenizer instance, so reported values are unchanged. (#2810) **Survive a re-parse MemoryError.** `MemoryError` is not a `ValueError`, so on 1M-context payloads the byte-faithful forwarder's verification re-parse escaped the handler and aborted an otherwise-fine request — 14 aborts across 8 days of reporter logs. (#2768) ## Performance All four are measured, not guessed. Each degrades with something a short benchmark does not vary: uptime, content shape, or process age. | fix | before | after | |---|---|---| | Cost-record walk per request (at 100k records) | 13.6 ms | bounded by model count | | JSON-block scan, JS-style object logs (1200 lines) | 4643 ms | 183 ms | | JSON-block scan, truncated JSONL | 3737 ms | 116 ms | | Lazy imports inside user requests | multi-second | paid at startup | | `count_text` (80% of local CPU) | — | memoised | Two worth calling out: - **The cost walk degrades with proxy *uptime*, not load.** A freshly started proxy pays ~0.01 ms; a month-old one pays 4–13 ms on every request, on the event loop, holding the metrics lock. Deliberately not a TTL cache over `stats()`: those values feed `check_budget()` when `--budget` is set, and a stale reading under-enforces the budget. The fix is to stop computing what the caller discards. - **The JSON-block memo is built only *after* a scan fails to balance.** That ordering is load-bearing, not an optimisation — caching from the start made pretty-printed JSON ~2x slower, since content that balances on the first scan has nothing to reuse and just pays the per-line dict traffic. Still a constant-factor fix, not an asymptotic one. ## Tests +1202 lines, 20 files. Each fix is pinned by a test that fails on the unmodified code: the re-count test asserts no `count_messages` pass runs with a live event loop in its thread; the re-parse test drives a `MemoryError` through the real request path and expects a 200; `totals()` equality with `stats()` is asserted across model counts, request volumes, and both pricing branches. The timeout test is structural rather than a mock — the failure mode is a dispatch path someone adds later without a guard, which mocking the existing four cannot catch. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Abhay Singh
|
3808f60ca6
|
fix(proxy/anthropic): inject headroom_retrieve whenever a CCR marker is present, not only for new markers (#2848)
## Description
On a frozen-prefix turn that replays an existing `<<ccr:hash>>` marker,
the proxy did not inject the `headroom_retrieve` tool, so the agent held
a marker it could not redeem. When it tried, the Anthropic API rejected
the whole request:
```text
API Error: 400 Tool reference 'headroom_retrieve' not found in available tools
```
This was a frequent, user-visible failure in Claude Code.
### Root cause
The sticky tool-injection gate in `handlers/anthropic.py` was driven by
`has_new_ccr_markers(...)` -- markers created THIS turn only:
```python
has_new_compressed_content = has_new_ccr_markers(
current_detected_hashes=injector.detected_hashes,
previous_forwarded_messages=prefix_tracker.get_last_forwarded_messages(),
provider="anthropic",
)
tools, ccr_tool_injected = apply_session_sticky_ccr_tool(
...,
has_compressed_content_this_turn=has_new_compressed_content,
)
```
`apply_session_sticky_ccr_tool` returns early with `decision="skip"` for
a session it considers fresh when `not
has_compressed_content_this_turn`. A marker replayed from the frozen
prefix is "historical" (already in `previous_forwarded_messages`), so
`has_new_ccr_markers` returns `False`, and on a fresh session the tool
is skipped even though the request carries a redeemable marker. The
`SessionCcrTracker` is per-process, so every proxy restart makes live
sessions look fresh again and re-arms the failure mid-conversation.
Anything that instructs the model to retrieve later (a project
instruction saying "call `headroom_retrieve` with the hash before
asserting an exact value") lands on this path by construction.
### Fix
Drive the gate from `injector.has_compressed_content` -- whether the
forwarded request carries ANY CCR marker, new or replayed -- instead of
new-markers-only. `#1850` narrowed the first-time gate to new markers to
avoid arming a session that never compressed, but a present marker means
the session HAS compressed, and a replayed marker is exactly as
unredeemable as a fresh one. Since a new marker is also a present
marker, `has_new_compressed_content or injector.has_compressed_content`
collapses to `injector.has_compressed_content`, so the now-redundant
`has_new_ccr_markers` call is removed.
The cache argument cuts in favor of this: toggling the tool in and out
of the tools array between turns is what busts the tools cache segment.
Injecting consistently whenever markers exist is the cache-stable
option, and it removes a hard 400 in exchange for at most one cache
miss. The frozen message prefix is still replayed byte-identical, so the
prompt-cache prefix is unaffected; only the tools array gains a stable
entry.
Fixes #2766
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/handlers/anthropic.py`: the sticky CCR tool-injection
gate now passes
`has_compressed_content_this_turn=injector.has_compressed_content` (any
marker present) instead of the new-markers-only signal, and the
now-redundant `has_new_ccr_markers` computation/import is dropped.
- `tests/test_proxy/test_anthropic_ccr_deferred_injection.py`: the two
tests that encoded the superseded `#1850` behavior (a replayed
historical marker forwarded WITHOUT the tool) now assert the tool IS
injected, with updated rationale. One was renamed from
`..._when_tool_injection_is_deferred` to
`..._and_injects_retrieve_tool`. The byte-identical message-prefix
replay assertions are unchanged.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
# Fail-before (source fix stashed, updated tests kept):
tests/test_proxy/test_anthropic_ccr_deferred_injection.py
::test_cache_mode_compresses_delta_but_replays_cached_prefix_when_markers_are_historical FAILED
::test_cache_mode_exact_prefix_replay_forwards_cached_compressed_prefix_and_injects_retrieve_tool FAILED
assert [tool["name"] for tool in forwarded["tools"]] == ["headroom_retrieve"]
KeyError: 'tools'
# Pass-after (fix applied):
tests/test_proxy/test_anthropic_ccr_deferred_injection.py 15 passed
# Broader CCR suites:
tests/test_proxy/test_ccr_frozen_prefix_coupling.py tests/test_ccr_tool_always_on.py
tests/test_ccr_session_tracker.py tests/test_ccr_tool_injection.py 61 passed
tests/test_ccr_marker_policy.py tests/test_anthropic_ccr_workspace_unbound.py
tests/test_ccr_tool_calls.py tests/test_corrupt_golden_bytes_recovery.py 21 passed
# uvx ruff@0.15.17 check -> All checks passed!
# uvx mypy@1.20.2 headroom/proxy/handlers/anthropic.py -> Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.15.17 and mypy 1.20.2 via uvx.
- Exact command / steps: traced the gate (`has_new_ccr_markers` ->
`apply_session_sticky_ccr_tool` fresh-session `skip`) and confirmed
`injector.has_compressed_content` reflects any marker present in the
forwarded messages (`len(_detected_hashes) > 0` after
`scan_for_markers`). Reproduced the exact bug in the handler harness: a
cache-mode frozen replay where `fake_tracker._last_forwarded_messages`
already holds the marker (so `has_new` is `False`) on a session the
reset tracker considers fresh, with the marker forwarded to upstream.
Fail-before with `git stash push headroom/proxy/handlers/anthropic.py`
and rerunning the two replay tests (the forwarded body has no `tools`),
pass-after with `git stash pop` (the body carries `headroom_retrieve`).
- Observed result: on a replayed-marker turn the forwarded request now
includes `"tools": [{"name": "headroom_retrieve", ...}]`, so the agent
can redeem the hash and Anthropic no longer 400s. The frozen message
prefix is still replayed byte-identical (`forwarded["messages"]`
unchanged). Sessions that never compressed still get no tool (no marker
-> `has_compressed_content` is `False`).
- Not tested: a live multi-turn Claude Code session across a real proxy
restart (no live provider here). The gate is exercised end-to-end
through the handler via the TestClient harness, reproducing the
historical-marker-on-fresh-session desync the issue describes.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`: it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
This deliberately reworks the `#1850` deferral for historical markers,
so it changes two tests that encoded "tool absent on frozen replay."
That behavior was the source of the 400: a marker in the prompt with no
tool to redeem it is a hard failure, whereas a re-injected tool is a
stable, cheap entry in the tools array. The reporter validated the same
change locally (33 requests, 0 errors, 0 `skip`). Scope is the Anthropic
interactive path where the bug was reported; the stateless batch path (a
separate `CCRToolInjector.process_request` gated on `tokens_saved > 0`)
is unchanged.
|
||
|
Abhay Singh
|
b97c7c6e99
|
fix(proxy/gemini): keep streaming-parity baseline so eligible_pct can't exceed 100 (#2824)
## Description The non-streaming Gemini `generateContent` finalizer builds its `RequestOutcome` with `optimized_tokens` set to Gemini's own `promptTokenCount` (the provider's tokenizer scale, which correctly feeds billing and the dashboard), while `original_tokens` stays a local estimator count. Those two are on different rulers. Every delta the beacon derives from the pair is a same-ruler difference: `tokens_saved`, `tokens_inflated`, `attempted_input_tokens`, and the beacon's `eligible_pct` / `yield_pct`. When Gemini counts the forwarded prompt higher than our local estimator does, `attempted_input_tokens` (which is `optimized_tokens + tokens_saved`) exceeds the local `original_tokens`, and the request ships a structurally-impossible `eligible_pct > 100` plus a phantom `tokens_inflated`. This is the exact class of bug #2756 removed, on a path #2756 did not touch: it fixed the non-streaming OpenAI handler, and the streaming finalizer (`_finalize_stream_response`) already guards against it by lifting the baseline onto the provider scale. The non-streaming Gemini path had neither treatment. The fix mirrors the streaming finalizer's already-tested handling: when a provider count is present, lift the baseline to `max(original_tokens, promptTokenCount + tokens_saved)` so `attempted_input_tokens <= original_tokens` holds and `tokens_inflated` collapses to 0. It is guarded on a present count, so a null or absent `promptTokenCount` leaves the local baseline untouched and the existing zero-usage preservation test still holds. `optimized_tokens` still carries the provider count, so billing and the dashboard are unchanged. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/handlers/gemini.py` (`handle_gemini_request`, non-streaming `generateContent` branch): compute `effective_original_tokens = max(original_tokens, total_input_tokens + tokens_saved)` when `total_input_tokens > 0` (else keep `original_tokens`), and pass it as the outcome's `original_tokens`. Mirrors the streaming finalizer's provider-usage handling. - `tests/test_proxy/test_gemini_savings_profile.py`: added `test_gemini_provider_count_above_local_estimate_does_not_inflate_eligible`, which drives a request where Gemini's `promptTokenCount` (150) exceeds the local post-compression count (80), and asserts `attempted_input_tokens <= original_tokens`, `tokens_inflated == 0`, the provider count is still carried in `optimized_tokens`, and the baseline is lifted to 170. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # Fail-before (source fix stashed, new test kept): tests/test_proxy/test_gemini_savings_profile.py::test_gemini_provider_count_above_local_estimate_does_not_inflate_eligible FAILED assert outcome.attempted_input_tokens <= outcome.original_tokens AssertionError: assert 170 <= 100 # Pass-after (fix applied): tests/test_proxy/test_gemini_savings_profile.py::test_gemini_provider_count_above_local_estimate_does_not_inflate_eligible PASSED # Full file + related outcome suites: tests/test_proxy/test_gemini_savings_profile.py tests/test_proxy_gemini_native_integration.py tests/test_request_outcome.py tests/test_outcome_token_scale.py 47 passed, 18 skipped # uvx ruff@0.15.17 check -> All checks passed! # uvx mypy@1.20.2 headroom/proxy/handlers/gemini.py -> Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12.11, project venv (litellm installed), pytest 9.1.1 with pytest-asyncio 1.4.0 (asyncio_mode=auto per pyproject), ruff 0.15.17 and mypy 1.20.2 via uvx. - Exact command / steps: confirmed the streaming sibling already lifts the baseline (`_finalize_stream_response` in `headroom/proxy/handlers/streaming.py` sets `effective_original_tokens = max(original_tokens, provider_input_tokens + tokens_saved)` for openai/gemini), then fail-before with `git stash push headroom/proxy/handlers/gemini.py` and `python -m pytest tests/test_proxy/test_gemini_savings_profile.py -k inflate_eligible` (the assertion fails with `170 <= 100`, i.e. eligible_pct 170%), then pass-after with `git stash pop` and rerunning (passes), then the full file plus the outcome suites (47 passed, 18 skipped). - Observed result: with Gemini reporting `promptTokenCount=150` against a local post-compression count of 80 (saved 20), the outcome now reports `original_tokens=170`, `attempted_input_tokens=170` (so `eligible_pct <= 100`) and `tokens_inflated=0`, while `optimized_tokens` stays 150 so billing and the dashboard are unchanged. Before the fix the same request reported `original_tokens=100`, `attempted_input_tokens=170` (eligible_pct 170%) and `tokens_inflated=50`. - Not tested: a live streamed call to real Gemini/Vertex (no provider credentials in this environment). The provider-count-above-local case is reproduced with a mock response mirroring Gemini's `usageMetadata` shape, and the baseline-lift it mirrors is existing, tested code on the streaming path. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md`: it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Docs and manual testing are N/A: this aligns the non-streaming Gemini finalizer with the already-correct streaming finalizer, no API surface change. The baseline lift is guarded on a present provider count, so the existing zero-usage preservation test (`test_gemini_zero_usage_prompt_count_is_preserved`) is unaffected: a null or zero `promptTokenCount` keeps the local baseline and leaves `optimized_tokens` at 0. |
||
|
Focused Instability
|
4dab254d52
|
fix: emit SSE ping before message_start on Bedrock streaming path (issue #902) (#1080)
## Description Closes #902 Mid-turn user interjections (steering) silently dropped through the Bedrock streaming path. The _stream_response_bedrock code path reconstructs Anthropic SSE events from parsed StreamEvent objects instead of passing raw bytes through, so SSE-level ping keepalives are never forwarded to Claude Code. Claude Code relies on ping events to arm its mid-turn steering / interruptible state; without them, queued interjections are discarded instead of sent. Root cause (confirmed): - Standard direct-Anthropic path does a raw yield-chunk passthrough — pings flow unchanged. - Bedrock path (_stream_response_bedrock.generate()) reconstructs events from litellm/anyllm stream_message() output, which only yields semantic events (message_start, content_block_*, message_delta, message_stop, error). No pings, ever. Fix: emit a synthetic 'event: ping / data: {}' at stream start (before the first message_start) so downstream clients see the same ping-then-content cadence as a real Anthropic stream. Note: periodic pings for very long responses (>~25s) may be needed if steering disarms on a timer. This commit arms it at turn start; follow-up if reporters confirm steering still drops on long turns. The causal link (ping → steering) is the reporter's hypothesis from hands-on debugging. The observable defect (zero pings in stream) is confirmed and fixed. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - headroom/proxy/handlers/streaming.py: yield ping event before the event loop in _stream_response_bedrock.generate() - tests/test_proxy/test_bedrock_sse_ping.py: 3 new tests asserting ping appears before message_start ## Testing - [x] Unit tests pass (pytest) - [x] Linting passes (ruff check .) - [x] New tests added for new functionality ### Test Output ``` tests/test_proxy/test_bedrock_sse_ping.py::test_bedrock_stream_emits_ping_before_message_start PASSED tests/test_proxy/test_bedrock_sse_ping.py::test_bedrock_stream_ping_has_empty_data PASSED tests/test_proxy/test_bedrock_sse_ping.py::test_bedrock_stream_contains_message_stop PASSED tests/test_backend_streaming_cache_metrics.py (4 tests) PASSED 7 passed in 4.41s ``` ## Real Behavior Proof - Environment: macOS, Python 3.11, headroom unit tests - Exact command / steps: pytest tests/test_proxy/test_bedrock_sse_ping.py -v - Observed result: 3 new tests pass; ping appears before message_start in Bedrock stream - Not tested: end-to-end against live Bedrock + Claude Code (no Bedrock credentials available) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] Code follows project style guidelines - [x] Code is commented where non-obvious - [x] No new warnings - [x] Tests added and passing ## Additional Notes The Rust proxy files mentioned in the issue (sse/framing.rs, sse/anthropic.rs) are NOT part of this fix. Those drops are in a telemetry-only tee task that never affects the client byte path — the Rust proxy does a raw bytes passthrough for all responses. The defect is Python-only, confined to the Bedrock path. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
nangsontay
|
08fce29b47
|
fix(proxy): stop toggling headroom_retrieve in the Anthropic tools array (#2672)
## Description `should_inject_ccr_tool` deferred CCR tool injection whenever `frozen_message_count > 0`. Because `tools` is the head of Anthropic's cache key, that dropped a tool which was already inside the provider-cached prefix and invalidated the whole prefix — in both directions (`0 → >0` removes it; `>0 → 0` on proxy restart, `/model` switch, lineage eviction or TTL lapse adds it back). On three days of local proxy logs the turns that flipped injection state carried **44.7% of all cache-write tokens at a 52.0% hit rate**, against 98.1% for non-flipping turns. The log signature is `cache_read` alternating between two values exactly 172 tokens apart — the 464-byte tool definition. This deletes the gate and calls `apply_session_sticky_ccr_tool` directly, which is **what `openai.py` already does** — the two handlers now have the same shape. Net −61 production lines, no new state, no new config flag. Fixes defect 1 of #2671. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Performance improvement ## Changes Made - `headroom/proxy/ccr_marker_policy.py` — deleted the `should_inject_ccr_tool` gate; `apply_session_sticky_ccr_tool` is now the single decision point. - `headroom/proxy/handlers/anthropic.py` — calls `apply_session_sticky_ccr_tool` directly, matching `openai.py`. - `headroom/proxy/helpers.py` — dropped the now-unused gate plumbing. - `tests/test_proxy_anthropic_cache_stability.py` — new test asserting the forwarded `tools` array is byte-identical across a `frozen 0 → >0` transition. - `tests/test_ccr_marker_policy.py` — removed the three unit tests that pinned the deleted decision (they encoded the defect). - `tests/test_proxy/test_ccr_frozen_prefix_coupling.py` — same unredeemable-marker intent, re-pinned at the sticky helper. - `tests/test_proxy/test_anthropic_ccr_deferred_injection.py` — autouse reset fixture for the process-global `SessionCcrTracker` (separate commit). - Formatting-only follow-up commit applying `ruff format` (pinned 0.15.17) to the two test files above. ### Why deleting the gate is safe `apply_session_sticky_ccr_tool` already holds the correct rule. Its four branches, in order: | # | condition | action | |---|---|---| | 1 | tool already in the incoming tool list (client/MCP pre-registered) | skip; the client's bytes win | | 2 | `session_id is None` (WS / pre-session) | per-turn flag drives it verbatim | | 3 | session has done CCR | always inject the recorded golden bytes | | 4 | fresh session, no compression this turn | **skip** | Branch 4 is the safety property: a session that has never compressed still gets no tool, so removing the gate cannot start injecting into non-CCR conversations. Branch 3 is what the gate was starving. `has_new_ccr_markers` still gates first-time injection, so markers replayed from the previously-forwarded prefix cannot trigger one. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed The three deleted unit tests encoded the defect. Coverage moves to the property that actually matters and was previously untested: **the forwarded `tools` array must be byte-identical across a `frozen 0 → >0` transition.** That test asserts on the forwarded request body rather than on a policy function's return value; unit-testing the old policy in isolation is exactly what let a wrong-but-self-consistent decision pass. Verified failing on `upstream/main` with an assertion on the missing tool (not an `ImportError`, so it fails for the right reason). Full suite: same pre-existing unrelated failures as `upstream/main`, **zero new** (verified by running the whole suite on both revisions and diffing the failure sets). ### Test Output ```text $ uv run pytest tests/test_ccr_marker_policy.py \ tests/test_proxy/test_anthropic_ccr_deferred_injection.py \ tests/test_proxy/test_ccr_frozen_prefix_coupling.py \ tests/test_proxy_anthropic_cache_stability.py -q collected 48 items tests/test_ccr_marker_policy.py ..... [ 10%] tests/test_proxy/test_anthropic_ccr_deferred_injection.py .............. [ 39%] . [ 41%] tests/test_proxy/test_ccr_frozen_prefix_coupling.py .. [ 45%] tests/test_proxy_anthropic_cache_stability.py .......................... [100%] ======================= 48 passed, 2 warnings in 13.59s ======================== $ ruff check . All checks passed! $ ruff format --check . 1349 files already formatted ``` ## Real Behavior Proof - Environment: local macOS proxy serving live Claude Code traffic to the Anthropic API; baseline = 3 days of proxy logs on `upstream/main`, after = 5.5 hours with this change live. - Exact command / steps: ran the proxy with this branch built in, drove normal Claude Code sessions through it (including `/model` switches and proxy restarts, the two events that used to flip injection state), then parsed 235 real turns from the proxy logs with the same parser used for the baseline in #2671. - Observed result: flip turns fell from 177 (44.7% of all cache write) to 2 (1.7%); steady-state write share 1.192% → 0.867%; aggregate hit rate 86.75% → 89.21%; main conversation warm hit rate 98.1% → 97.70% (n=149). The 2 remaining "flips" have `cache_read == 0` — cold starts that the bucketing counts as a state change, not real flips. | metric | baseline | after | |---|---|---| | flip turns | 177, carrying 44.7% of all cache write | **2**, carrying **1.7%** | | main conv, warm | 98.1% | **97.70%** (n=149) | | steady-state write share | 1.192% | **0.867%** | | aggregate | 86.75% | **89.21%** | - Not tested: `mypy headroom` was not run locally for this body; the OpenAI handler path (unchanged by this PR); tracker state loss mid-session (see note below); and defect 2 of #2671 (the sub-call breakpoint), which is untouched and is now 54.9% of remaining cache write — that is why aggregate stays just under 90%. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes **Pre-existing and unchanged here:** if the tracker loses state mid-session while the transcript still carries markers, branch 4 returns no tool and those markers are unredeemable. `upstream/main` has no recovery for that; this PR neither creates nor fixes it. See my comment on #2500, which adds a recovery path for the related dangling-reference case. **N/A checklist items:** no documentation changes — this removes an internal policy function with no user-facing surface. `mypy headroom` left unchecked because it was not run for this body; CI covers it. **Merge-order conflict with #2500 (please read before landing either):** this PR *deletes* `should_inject_ccr_tool`, which is the exact function #2500 extends with `transcript_requires_tool`. Whichever lands second needs a semantic rebase, not just a textual one — git will not flag it. If this PR lands first, #2500's recovery path should re-target `apply_session_sticky_ccr_tool` (the sticky helper now owns the decision alone) or the handler call site in `handlers/anthropic.py`. If #2500 lands first, the gate deletion here still applies but the `transcript_requires_tool` override needs to move with it. Happy to do the rebase either way — say which order you prefer. |
||
|
Rod Boev
|
a2e42fb877
|
fix(proxy): keep buffered CCR streams alive (#2479)
## Description Buffered CCR streaming currently waits for the full upstream response before sending any bytes back to the client. On the Anthropic path this shows up as `API Error: Stream idle timeout - no chunks received`, and the same buffer-then-synthesize mechanism still exists on the `/v1/responses` CCR path. This adds a narrow buffered-stream heartbeat layer so the client sees early stream activity while Headroom preserves the existing server-side retrieval round trip and final synthesized provider events. Closes #2465 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - open buffered CCR streams early and emit client-visible `event: ping` heartbeats while the buffered upstream call is still in flight - preserve the existing terminal Anthropic and Responses synthesis helpers instead of replacing their event-building logic - preserve early non-streaming failure semantics before the first heartbeat, including normal 429 passthrough and normal JSON 502 failures - log late buffered-task exceptions server-side and record one failed provider metric on that post-keepalive branch, while keeping the client-facing SSE error sanitized - add focused delayed-upstream regression coverage for both buffered provider paths, their early-failure branches, and their late-failure branches ## Testing - [x] Unit tests pass (`uv run pytest tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py tests/test_proxy/test_openai_responses_ccr.py -q`) - [x] Linting passes (`uv run ruff check headroom/proxy/handlers/anthropic.py headroom/proxy/handlers/openai.py tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py tests/test_proxy/test_openai_responses_ccr.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text uv run pytest tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py tests/test_proxy/test_openai_responses_ccr.py -q ======================= 17 passed, 1 warning in 42.47s ======================== uv run ruff check headroom/proxy/handlers/anthropic.py headroom/proxy/handlers/openai.py tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py tests/test_proxy/test_openai_responses_ccr.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows, Python via `uv`, proxy handler tests with gated buffered upstream fixtures - Exact command / steps: run the focused Anthropic and Responses CCR suites above; delayed-upstream tests consume the first client-visible SSE event before releasing the upstream, then consume the synthesized final events - Observed result: both buffered paths emitted `event: ping` before upstream release; pre-keepalive 429 responses preserved their real status and headers, pre-keepalive exceptions returned the normal JSON 502 shape, late transport failures recorded one failed provider metric and one server error log before emitting one sanitized SSE error, Anthropic preserved `done`, and Responses preserved `Resolved!` - Not tested: live slow upstream run with Claude Code or a real Responses client ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes `CHANGELOG.md` stays untouched because Headroom generates release notes from conventional commits. The PR should only claim the local buffered-stream contract and focused regression coverage; live client proof remains an owner check on a slow real upstream. |
||
|
Abhay Singh
|
313c290df9
|
fix(proxy/openai): None-guard usage token counts on the chat path (#2431)
## Description
`handle_openai_chat` reads token counts from the response usage to
record metrics and update the prefix tracker:
```python
usage = backend_response.body.get("usage", {})
output_tokens = usage.get("completion_tokens", 0)
total_input_tokens = usage.get("prompt_tokens", optimized_tokens)
```
`.get(key, default)` only falls back when the key is **absent**. When an
OpenAI-compatible backend emits a key with a **null** value (providers
do this on a stopped or empty turn, the same shape that caused the
Gemini crash in #2347), `.get` returns `None`. That `None` then flows
into:
- `_infer_openai_cache_write_tokens(total_input_tokens,
cache_read_tokens)` → `max(input_tokens - cache_read_tokens, 0)` (a
`None - int` → `TypeError`),
- `uncached_input_tokens = max(0, total_input_tokens - cache_read_tokens
- cache_write_tokens)`, and
- `RequestOutcome(output_tokens=..., optimized_tokens=...)`, whose
fields are `int` and which the metrics recorder increments.
Both chat usage-extraction sites are affected. On the direct-provider
branch the arithmetic runs **outside** the surrounding `try`, so a
single such response raises an uncaught `TypeError` and 500s the
request; on the backend branch it corrupts outcome recording.
## Fix
Coerce the three counts with the existing module-level `_usage_int`
guard (`max(int(value), 0)`, 0 on failure) at both sites, matching the
streaming path, the already-guarded cache keys in the same block
(`usage.get("cache_read_input_tokens", 0) or 0`), and the Gemini fix in
#2347. A normal integer usage is unchanged; only a null (or absent)
value now becomes the fallback/0. `prompt_tokens` keeps its
`optimized_tokens` fallback so our own input estimate is used when the
count is missing.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/handlers/openai.py`: `_usage_int`-guard
`completion_tokens` / `prompt_tokens` / `cached_tokens` at both
non-streaming usage-extraction sites in `handle_openai_chat`.
- `tests/test_proxy/test_openai_chat_savings_profile.py`: regression
driving a `/v1/chat/completions` request whose backend usage reports
null `prompt_tokens` / `completion_tokens`, asserting a 200 instead of a
crash.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_proxy/test_openai_chat_savings_profile.py -q
2 passed
# with the fix reverted, the new test fails (the null-usage response 500s):
$ git stash push -- headroom/proxy/handlers/openai.py
$ python -m pytest tests/test_proxy/test_openai_chat_savings_profile.py::test_chat_completions_survives_null_usage_token_counts -q
1 failed
$ uvx ruff@0.15.17 check headroom/proxy/handlers/openai.py tests/test_proxy/test_openai_chat_savings_profile.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/proxy/handlers/openai.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, project venv (`uv sync --extra
proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`, pytest in the venv.
- Exact command / steps: ran the new FastAPI `TestClient` regression,
which drives the real `handle_openai_chat` through a mock backend
returning `usage: {prompt_tokens: null, completion_tokens: null,
total_tokens: null}`; then reverted only `openai.py` and re-ran the same
test.
- Observed result: with the fix the request returns 200; with the fix
reverted the same request fails (the null count reaches the `max(...)`
arithmetic and outcome recording). Ran against the actual handler via
the app.
- Not tested: a live third-party OpenAI-compatible gateway emitting null
usage.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
|
||
|
Rod Boev
|
d6a1af40d5
|
fix(proxy): skip max_tokens rename for backend-routed openai chat (#2401)
## Description OpenAI-format `POST /v1/chat/completions` requests routed through `--backend litellm-vertex` fail when the client includes `max_tokens`. The proxy currently runs its direct-OpenAI compatibility shim before backend dispatch, renames `max_tokens` to `max_completion_tokens`, then the LiteLLM path no longer recognizes that field as standard and sweeps it into `extra_body`. Vertex rejects the resulting request with `extra_body: Extra inputs are not permitted`. This change scopes the rename shim to the direct OpenAI path only. Backend-routed chat requests now keep `max_tokens`, which LiteLLM already forwards correctly for the Vertex Anthropic path. Direct GPT-5 and o-series compatibility stays unchanged. Closes #2392. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Thread a backend-owned translation flag into `_normalize_openai_max_tokens`. - Skip the legacy-to-completion-token rename on backend-routed OpenAI chat requests. - Keep the direct OpenAI compatibility path covered with a backend-owned translation no-op test. - Add buffered and streaming handler-level regressions for the exact `litellm-vertex` request shape, proving the request survives the `/v1/chat/completions` normalization boundary with vendor fields intact. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py -q`) - [x] Linting passes (`uv run ruff check headroom/proxy/handlers/openai.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py -q ......sss............ [100%] 20 passed, 3 skipped, 1 warning in 42.13s $ uv run ruff check headroom/proxy/handlers/openai.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py All checks passed! $ uv run ruff format headroom/proxy/handlers/openai.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py --check 5 files already formatted ``` ## Real Behavior Proof - Environment: Windows, synced Headroom development environment, mocked LiteLLM provider boundary, no paid GCP credentials required - Exact command / steps: run `uv run pytest tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py -q`, using the issue payload shape `{"model":"claude-sonnet-4-6","max_tokens":32,"messages":[{"role":"user","content":"hi"}],"chat_template_kwargs":{"enable_thinking":false}}` through `POST /v1/chat/completions` - Observed result: buffered and streaming `litellm-vertex` requests keep `max_tokens` as a named backend kwarg, preserve `chat_template_kwargs` in `extra_body`, omit `max_completion_tokens` from `extra_body`, and return success through the handler boundary. Direct-path normalization still renames legacy `max_tokens`. - Not tested: live Vertex AI request ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [ ] My code follows the project's style guidelines - [ ] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - `CHANGELOG.md`: N/A, the release pipeline generates it from the conventional-commit subject. - Scope is intentionally narrow: this fixes the exact backend-routed `max_tokens` failure and does not broaden `extra_body` hardening for unrelated OpenAI fields. |
||
|
Abhay Singh
|
f64aac9733
|
fix(proxy/gemini): None-guard token counts from usageMetadata (#2347)
## Description
The non-streaming Gemini/Vertex handler reads token counts straight from
the response's `usageMetadata`:
```python
try:
usage = resp_json.get("usageMetadata", {})
total_input_tokens = usage.get("promptTokenCount", optimized_tokens)
output_tokens = usage.get("candidatesTokenCount", 0)
cache_read_tokens = usage.get("cachedContentTokenCount", 0)
except (...):
...
uncached_input_tokens = max(0, total_input_tokens - cache_read_tokens) # OUTSIDE the try
```
`.get(key, default)` only falls back when the key is **absent**. When
`usageMetadata` carries a key with a **null** value — which Gemini can
do on a safety-blocked turn that produced no candidates — `.get` returns
`None`. That `None` then reaches:
- `max(0, total_input_tokens - cache_read_tokens)` (a `None - int` →
`TypeError`), and
- `RequestOutcome(output_tokens=...)`, whose field is `int` and which
the metrics recorder increments (`tokens_output_total += output_tokens`
→ `TypeError`).
Both run on the success (non-`except`) path, so a single such response
crashes the request and its outcome recording. The Gemini streaming path
already guards these with a `_usage_int` helper; the non-streaming path
(two sites) did not.
## Fix
Coerce the three counts with `int(... or fallback)`, matching the
streaming `_usage_int` guard and the LiteLLM usage mappings:
```python
total_input_tokens = int(usage.get("promptTokenCount", optimized_tokens) or optimized_tokens)
output_tokens = int(usage.get("candidatesTokenCount", 0) or 0)
cache_read_tokens = int(usage.get("cachedContentTokenCount", 0) or 0)
```
No change for a normal integer usage; only a `None` (or absent) value
now becomes the fallback/0. Applied to both non-streaming
usage-extraction sites in `handlers/gemini.py`.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/handlers/gemini.py`: `int(... or fallback)`-guard
`promptTokenCount` / `candidatesTokenCount` / `cachedContentTokenCount`
at both non-streaming usage sites.
- `tests/test_proxy/test_gemini_savings_profile.py`: add a regression
driving a `generateContent` request whose
`usageMetadata.candidatesTokenCount` is `null`, asserting a 200, an
`int` `output_tokens == 0`, and `uncached_input_tokens == 20` (the
`max(0, …)` no longer raises).
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/proxy/handlers/gemini.py tests/test_proxy/test_gemini_savings_profile.py
All checks passed!
$ uvx ruff@0.15.17 format --check headroom/proxy/handlers/gemini.py tests/test_proxy/test_gemini_savings_profile.py
2 files already formatted
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/proxy/handlers/gemini.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17` / `uvx
mypy@1.20.2`. A full `pytest` OOMs this box (ML-stack import), so I
reproduced the extraction with a dependency-free script and left the
full pytest to CI.
- Exact command / steps: modeled the OLD (bare `.get`) and NEW (`int(...
or fallback)`) derivations for a blocked response
(`candidatesTokenCount: null`, valid prompt count), a null
`promptTokenCount`, a normal response, and an absent-usage response.
- Observed result: OLD raised `TypeError` at `max(0, None - …)` for a
null prompt count and left `output_tokens = None` (which crashes the
int-typed outcome/metrics recorder) for a null candidate count; NEW
produced `(20, 0)` for the blocked case, `(15, 0)` for the null-prompt
case (the `optimized_tokens` fallback), `(60, 30)` for a normal
response, and the fallbacks for absent usage. The added
`create_app`/`TestClient` test drives the handler end to end and asserts
a 200 with `int` outcome counts.
- Not tested: a live Gemini safety-blocked response; the added test uses
a mocked `_retry_request` returning a `usageMetadata` with a null count,
matching the existing Gemini test harness in this file.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" box is unchecked because a local pytest
run imports the ML stack and OOMs this box; the added test reuses the
existing `create_app`/`TestClient` + mocked-`_retry_request` harness in
`test_gemini_savings_profile.py` and runs under the normal CI pytest
job, and the behavior is corroborated by the standalone proof above.
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
Rod Boev
|
26b43f64d6
|
fix(proxy): keep anthropic ccr compression active across deferred injection (#2291) (#2297)
## Description Large native Claude Code requests on the Anthropic path can still forward with zero request compression after CCR tool injection is deferred on a frozen prefix. The stale skip branch treats deferred injection as a reason to bypass request compression entirely, even though the later sticky CCR path already knows when new markers actually require the tool. This removes that stale bypass so compression still runs while the reversible CCR path stays intact. Closes #2291. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Remove the stale `should_skip_ccr_request_compression` branch from `headroom/proxy/handlers/anthropic.py`, so deferred CCR tool injection no longer bypasses request compression in token, non-cache, or cache mode. - Keep the existing sticky CCR injection path as the only place that decides whether historical markers need the retrieval tool reintroduced. - Update `tests/test_proxy/test_anthropic_ccr_deferred_injection.py` to cover the two broken zero-compression cases and preserve the already-reversible frozen-prefix path. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_proxy/test_anthropic_ccr_deferred_injection.py::test_cache_mode_compresses_delta_but_replays_cached_prefix_when_markers_are_historical tests/test_proxy/test_anthropic_ccr_deferred_injection.py::test_non_token_non_cache_mode_keeps_compression_and_injects_tool_for_new_markers tests/test_proxy/test_anthropic_ccr_deferred_injection.py::test_existing_retrieve_tool_keeps_reversible_ccr_path_when_prefix_is_frozen tests/test_openai_tool_search_deferral.py tests/test_openai_responses_compression_units.py -q`) - [x] Linting passes (`uv run ruff check .`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text ======================= 53 passed, 1 warning in 12.71s ======================== All checks passed! 1310 files already formatted ``` ## Real Behavior Proof - Environment: synced branch and base worktrees on a local Windows proxy test host - Exact command / steps: run the updated Anthropic deferred-injection regressions directly against the base package tree and the branch package tree, then run the focused branch pytest suite above - Observed result: the base package tree fails the two updated zero-compression regressions (`FAIL test_cache_mode_compresses_delta_but_replays_cached_prefix_when_markers_are_historical`, `FAIL test_non_token_non_cache_mode_keeps_compression_and_injects_tool_for_new_markers`) while preserving the already-reversible path; the branch package tree prints three `PASS` lines for the same trio and keeps the neighboring OpenAI suites green in the 53-test focused run - Not tested: a live upstream Claude Code request with the reporter's exact provider/model credentials ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes `CHANGELOG.md` is not applicable here because Headroom's release pipeline derives it from conventional commits. Scope is limited to the Anthropic CCR request-compression seam that current #2291 evidence exercises. OpenAI tool-search deferral is untouched because the current live issue is a native Claude Code path and the concrete stale skip branch on `origin/main` is in `anthropic.py`. Co-authored-by: Tejas Chopra <chopratejas@gmail.com> |
||
|
Krishna Chaitanya
|
57e8dcb425
|
feat(proxy): add opt-in cost-aware model router (#1706) (#2205)
## Description Adds an optional, configuration driven model router (closes #1706). With `HEADROOM_MODEL_ROUTER_ENABLED` set, ordered rules in `HEADROOM_MODEL_ROUTES` rewrite the upstream model by estimated input size and tool presence, complementary to content compression, for example sending small, tool-free requests to a cheaper model. First matching rule wins and every decision is logged with a reason. Off by default so behavior is unchanged, skipped under `x-headroom-bypass`/passthrough, and wired on the Anthropic `/v1/messages` path. Malformed rules fail open, so a bad rule is skipped rather than silently widened. Closes #1706 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/model_router.py`: new `ModelRouter` component (ordered rules, first-match decision with reason, fail-open env parsing, tokenizer-free input estimate). - `headroom/proxy/models.py` + `headroom/proxy/server.py`: `ProxyConfig.model_router` field, env loader (`HEADROOM_MODEL_ROUTER_ENABLED` / `HEADROOM_MODEL_ROUTES`), and proxy wiring. - `headroom/proxy/handlers/anthropic.py`: apply routing on `/v1/messages` after the bypass gate, tracked as a body mutation. - Tests, docs (`configuration.mdx`), and a CHANGELOG entry. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest -q tests/test_proxy/test_model_router.py tests/test_proxy/test_model_router_wiring.py 36 passed, 1 warning $ ruff check . All checks passed! $ mypy headroom --ignore-missing-imports Success: no issues found in 477 source files ``` ## Real Behavior Proof - Environment: local, macOS, Python 3.12, headroom `.venv`, upstream mocked (no live provider call). - Exact command / steps: enable the router via `ProxyConfig(model_router=...)`, POST `/v1/messages` through `TestClient` with a rule routing low-risk requests to a cheaper model; repeat with header `x-headroom-bypass: true`. - Observed result: the forwarded upstream body model is rewritten from `claude-sonnet-4-6` to `claude-haiku-4-5` when the router is enabled, and is left unchanged under bypass (see `tests/test_proxy/test_model_router_wiring.py`). - Not tested: the OpenAI and Gemini handler paths (this PR wires the Anthropic path only); no live provider request (upstream is mocked). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes Happy to adjust the interface or scope (for example OpenAI and Gemini parity) if you'd prefer a different shape. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: reneleonhardt <65483435+reneleonhardt@users.noreply.github.com> |
||
|
nangsontay
|
e5b3a634df
|
fix(proxy): dedupe Codex WS request logging for accurate mixed-provider dashboards (#2189)
## Description Running Claude Code (Anthropic) and Codex (OpenAI) against the **same** Headroom proxy instance on one port produced incorrect, unstable dashboard data. The proxy core is provider-isolated and multi-provider-safe by design; the defect was in the observability layer. The Codex `/v1/responses` **WebSocket** handler was the only path in the proxy that wrote to the request logger by hand instead of through the unified `emit_request_outcome` funnel, and it did so twice per session close: the per-turn funnel record plus an unconditional cumulative session-summary `RequestLog`. This PR removes the duplicate summary log so Codex WS emits exactly one request log per turn, matching the HTTP provider paths. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Dropped the duplicate cumulative session-summary `RequestLog` in the Codex WS handler while preserving the per-turn `emit_request_outcome` path. - Preserved gated `request_messages` and `turn_id` on residual outcomes so dashboard telemetry keeps the useful attribution without double-counting tokens. - Ensured explicit `--anyllm-provider` wins over a leaked `HEADROOM_ANYLLM_PROVIDER` environment variable. - Registered retry delay settings that had drifted out of the settings registry. - Hardened tests against developer-shell `HEADROOM_*` / `ANTHROPIC_CUSTOM_HEADERS` leakage and stabilized several focused proxy/wrap test fixtures. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ .venv/bin/pytest tests/ -q -p no:cacheprovider 8529 passed, 537 skipped, 5831 warnings in 276.25s (0:04:36) $ .venv/bin/ruff check <touched files> All checks passed! ``` ## Real Behavior Proof - Environment: macOS (Darwin 25.4.0), Python 3.13.14, pytest 9.0.3, ruff via project venv, branch `fix/multi-provider-runtime`. - Exact command / steps: Ran the full test suite without pytest cache provider and Ruff on all touched files; used `git stash` to confirm the stale fake-config failures pre-existed this change. - Observed result: Full suite passed with no failures; Ruff passed; Codex WS now routes end-of-session logging through `emit_request_outcome`, emitting one request log per turn with the same accounting model as Anthropic HTTP turns. - Not tested: Live simultaneous Claude + Codex dashboard run. `mypy headroom` was not run to completion; a scoped run reported one pre-existing `settings_store.py:470` coercion error outside this diff. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A - server-side observability fix; no UI markup changed. ## Additional Notes - The proxy's multi-provider routing, header/auth isolation, and per-model cache keying are already correct and unchanged here; only the WS observability write path was double-counting. - Architectural assessment: `plans/reports/research-260714-0004-multi-provider-upstream-compression-report.md`; root-cause + resolution trail: `plans/reports/debug-assessment-260714-0011-dashboard-instability-mixed-claude-codex-report.md`. - No live simultaneous Claude + Codex dashboard run was performed; validation is from test coverage and code review of the WS logging path. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Tejas Chopra
|
4951cf80a2
|
fix(proxy): don't 502 Anthropic streaming on a legal mixed CCR + client-tool turn (#2089) (#2117)
## Description
Anthropic **buffered-streaming** returned a **502** on a legal turn:
when the model emits `headroom_retrieve` alongside a non-CCR client
tool, `CCRResponseHandler` intentionally skips CCR resolution (#839) and
hands both tool_use blocks back for the client to resolve. The
non-streaming path returns that as 200; the streaming path wrongly
failed closed with "Unable to safely complete streamed CCR retrieval."
Fix: add a provider-generic `CCRResponseHandler.residual_ccr_status()` →
`resolved` / `skipped_mixed_tools` / `error`. The streaming path now
only 502s on a genuine `error`; on the intentional mixed-tool skip it
falls through to the existing SSE resynthesis (200) preserving **both**
tool_use blocks — matching the non-streaming path. The misleading
"handled successfully" log no longer fires on skip.
Closes #2089
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/ccr/response_handler.py`: shared, provider-generic
`residual_ccr_status()`.
- `headroom/proxy/handlers/anthropic.py`: streaming path fails closed
only on a real residual-CCR error; passes through the legal skip as 200
SSE.
- `tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py`:
mixed-tool case now returns 200 SSE preserving both blocks.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] New/updated tests for the mixed-tool pass-through branch
### Test Output
```text
GitHub CI on current head:
- lint: pass
- build/build-wheel/build-wheel-windows: pass
- test matrix, test-agno, test-extras, test-dashboard-ui: pass
- docker-native-e2e: pass
Review spot-check:
uv run --extra proxy --extra dev python -m pytest tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py tests/test_ccr_response_handler_extra.py -q
15 passed, 1 warning
```
## Real Behavior Proof
- Environment: GitHub Actions on PR head `
|
||
|
Tejas Chopra
|
2976d49f18
|
fix(proxy): preserve sub-path in X-Headroom-Base-Url custom upstream (#2037) (#2127)
## Description `_resolve_openai_upstream_base` ran the `X-Headroom-Base-Url` value through `_normalize_origin`, which strips the path. A custom OpenAI-compatible upstream served from a sub-path, such as `https://host/api/v1`, was routed to the bare origin and returned `proxy_error` (#2037). This re-attaches the path after origin normalization. This is a clean extraction of the path fix from #2047, which bundled it with an unrelated `supports_websockets = true` to `false` default change across init/wrap/codex. #2047 can be closed in favor of this narrower fix. Closes #2037 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/handlers/openai.py`: re-attach the request header path component in `_resolve_openai_upstream_base` after origin normalization. - `tests/test_proxy/test_openai_upstream_header.py`: assert sub-paths are preserved and trailing slashes are normalized. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_proxy/test_openai_upstream_header.py 5 passed $ ruff check headroom/proxy/handlers/openai.py tests/test_proxy/test_openai_upstream_header.py All checks passed! ``` ## Real Behavior Proof - Environment: local proxy header resolution path, custom OpenAI-compatible upstream configured via `X-Headroom-Base-Url`. - Exact command / steps: resolve `X-Headroom-Base-Url: https://gateway.example/api/v1` through `_resolve_openai_upstream_base` / `_resolve_openai_upstream`. - Observed result: before the fix, the upstream resolved to `https://gateway.example` and lost `/api/v1`, causing the proxy to route to the wrong endpoint. After the fix, it resolves to `https://gateway.example/api/v1`; a trailing slash is normalized away. - Not tested: end-to-end request against a live third-party OpenAI-compatible gateway. The regression is covered at the proxy routing helper layer where the path was dropped. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A. ## Additional Notes - Documentation is not updated because this fixes the existing header behavior rather than changing a documented user-facing contract. - Changelog is not updated in this PR; the change is scoped to the regression and test. - `mypy headroom` was not run in the author's workflow. |
||
|
Abhay Singh
|
19201e842f
|
fix(proxy/openai): respect explicit stream_options.include_usage (#2026)
## Description
On the direct OpenAI `/v1/chat/completions` streaming path, the handler
injects
`stream_options.include_usage = True` so it can count tokens from the
trailing usage chunk —
but it does so **unconditionally**, including flipping an explicit
client `include_usage: false`
to `true` (`headroom/proxy/handlers/openai.py`):
```python
if "stream_options" not in body:
body["stream_options"] = {"include_usage": True}
elif isinstance(body.get("stream_options"), dict):
body["stream_options"]["include_usage"] = True # overrides an explicit `false`
```
When the client passed `stream_options: {"include_usage": false}` (or a
dict that set some
other key), the upstream is nevertheless asked for usage and appends a
terminal usage-only
frame:
```
data: {"id":...,"choices":[],"usage":{...}}
data: [DONE]
```
The extremely common client pattern `for chunk in stream:
chunk.choices[0].delta.content`
then raises `IndexError` on that empty-`choices` frame — for a usage
chunk the client
explicitly opted out of.
Closes: no issue filed — found while auditing the streaming
request-shaping.
## Fix
Only fill in `include_usage` when the client left the choice open — no
`stream_options` at all,
or a `stream_options` dict that doesn't mention `include_usage`. An
explicit `true`/`false` is
respected. Extracted into a small `_apply_stream_usage_option(body)`
helper (mirroring the
existing `_normalize_openai_max_tokens`) for a clean unit-test seam:
```python
stream_options = body.get("stream_options")
if stream_options is None:
body["stream_options"] = {"include_usage": True}
elif isinstance(stream_options, dict) and "include_usage" not in stream_options:
stream_options["include_usage"] = True
```
Scope note: this respects an explicit client choice, which is the
unambiguous defect. The
separate question of whether to strip the synthetic usage chunk when
Headroom injected the
option itself (the no-`stream_options` default, kept for token-counting)
touches the raw SSE
byte stream and is intentionally left out of this change.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/proxy/handlers/openai.py`: add
`_apply_stream_usage_option(body)` and call it from the streaming chat
path; it no longer overrides an explicit client `include_usage`.
- `tests/test_proxy/test_openai_stream_usage_option.py`: cover explicit
`false` (respected), explicit `true` (preserved), absent (injected), and
dict-without-key (filled in).
## Testing
- [x] New regression tests added
(`tests/test_proxy/test_openai_stream_usage_option.py`)
- [x] Linting/formatting clean — run with the CI-pinned `ruff==0.15.17`
- [ ] Full `pytest` deferred to CI (local-OOM reason below).
```text
$ uvx ruff@0.15.17 check headroom/proxy/handlers/openai.py tests/test_proxy/test_openai_stream_usage_option.py
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.10, headroom from this branch.
Importing `headroom` pulls in the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I verified the decision logic
with a dependency-free script and left the full pytest to CI.
- Exact command / steps: ran a client body with `stream_options:
{include_usage: false}` (plus the explicit-true, absent, and
dict-without-key cases) through the old unconditional injection and the
new helper.
- Observed result: the old logic flips the client's `false` to `true`;
the new logic respects it:
```text
explicit false: OLD -> {'include_usage': True} NEW -> {'include_usage': False}
INCLUDE_USAGE RESPECT-CLIENT FIX VERIFIED (old flips false->true; new respects false)
```
- Not tested: a full streaming round-trip through a live OpenAI upstream
(needs the heavy stack + a key). The fix is confined to the
request-shaping helper and the new tests drive it directly. Full local
`pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — ran
lint + a standalone logic check; full pytest deferred to CI (local OOM,
disclosed above)
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- Small, contained change plus a helper and tests; no new dependencies.
The backend-path injection (`test_backend_anyllm` /
`test_backend_streaming_cache_metrics`) is untouched — those pass an
explicit `include_usage: true`, which is preserved.
- @JerrettDavis tagging you — this one makes a client that sent
`include_usage: false` hit an `IndexError` on the usage chunk, so it
seemed worth surfacing. Thanks!
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
Abhay Singh
|
38306a331c
|
fix(proxy/gemini): thread savings-profile kwargs into apply() (#1994)
## Description
The native Gemini / Vertex-google compression handlers build the
pipeline call like this
(`headroom/proxy/handlers/gemini.py`, three sites —
`handle_gemini_generate_content` ~L487,
`handle_google_cloudcode_stream` ~L843, `handle_gemini_count_tokens`
~L1104):
```python
result = await self._run_compression_in_executor(
lambda: self.openai_pipeline.apply(
messages=messages,
model=model,
model_limit=context_limit,
context=extract_user_query(messages),
waste_messages=waste_messages,
), # <-- no **proxy_pipeline_kwargs(self.config)
...
)
```
Every other live compression path threads
`proxy_pipeline_kwargs(self.config)` into `apply()`
— `handlers/openai.py` (chat + responses), `handlers/anthropic.py`, and
the `/v1/compress`
endpoint. The Gemini handler never imports or calls it, so the savings
profile and the
ProxyConfig compression knobs never reach the pipeline for Gemini/Vertex
requests.
The proxy pipeline is built with only `transforms` + `provider`
(`server.py`), and
`ContentRouter` reads the accuracy-sensitive knobs per-call from
`**kwargs`. With the kwargs
missing, Gemini falls back to router defaults instead of the
profile/config values:
- `min_tokens_to_compress` → hardcoded **50** instead of the
coding-profile **25** / `config.min_tokens_to_crush`
- `protect_recent` → router default instead of the profile / configured
value
- `target_ratio` → **None** instead of the CLI default **0.4** used
everywhere else
- `max_items_after_crush`, `smart_crusher_with_compaction`,
`force_kompress` → router defaults
So Gemini/Vertex requests compress with a materially different (and
inconsistent) posture than
Claude/Codex/Cursor, and any user-tuned `HEADROOM_SAVINGS_PROFILE` /
`HEADROOM_TARGET_RATIO` / `HEADROOM_MIN_TOKENS` /
`HEADROOM_PROTECT_RECENT` is ignored on this
path.
This is the exact bug **#1534** fixed for the OpenAI chat path.
Closes: no issue filed — found while auditing profile/config threading
across the provider handlers.
## Fix
Import `proxy_pipeline_kwargs` (as `openai.py`/`anthropic.py` do) and
add
`**proxy_pipeline_kwargs(self.config)` to all three Gemini
`openai_pipeline.apply(...)` calls.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/proxy/handlers/gemini.py`: import `proxy_pipeline_kwargs`;
thread `**proxy_pipeline_kwargs(self.config)` into the three
`openai_pipeline.apply(...)` call sites (generateContent, Cloud Code
stream, countTokens).
- `tests/test_proxy/test_gemini_savings_profile.py`: drive the native
`/v1beta/models/{model}:generateContent` route with
`savings_profile="agent-90"` and assert the profile knobs
(`compress_user_messages`, `target_ratio`, `min_tokens_to_compress`,
`compress_system_messages`) reach `apply()`.
## Testing
- [x] New regression test added
(`tests/test_proxy/test_gemini_savings_profile.py`), mirroring the #1534
chat-path test
- [x] Linting passes (`ruff check`) and formatting is clean (`ruff
format --check`) — run with the CI-pinned `ruff==0.15.17`
- [ ] Full `pytest` deferred to CI (local-OOM reason below).
```text
$ uvx ruff@0.15.17 check headroom/proxy/handlers/gemini.py tests/test_proxy/test_gemini_savings_profile.py
All checks passed!
$ uvx ruff@0.15.17 format --check headroom/proxy/handlers/gemini.py tests/test_proxy/test_gemini_savings_profile.py
2 files already formatted
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.10, headroom from this branch.
Importing `headroom` pulls in the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I verified the kwargs threading
with a dependency-free script and left the full pytest to CI.
- Exact command / steps: mechanically reproduced the call site —
`apply(**base)` (old) vs `apply(**base,
**proxy_pipeline_kwargs(config))` (new) — and captured the kwargs each
produced.
- Observed result: the old call omits the profile knobs entirely; the
new call threads them:
```text
OLD gemini apply() kwargs: ['context', 'messages', 'model', 'model_limit', 'waste_messages']
-> profile knobs DROPPED (savings profile / config ignored)
NEW gemini apply() kwargs: ['compress_system_messages', 'compress_user_messages', 'context',
'max_items_after_crush', 'messages', 'min_tokens_to_compress', 'model', 'model_limit',
'protect_recent', 'target_ratio', 'waste_messages']
-> profile knobs THREADED: target_ratio=0.10, min_tokens_to_compress=120, compress_user/system=True
GEMINI KWARGS THREADING VERIFIED
```
- Not tested: a full request through a live Gemini/Vertex upstream
(needs the heavy stack + a key). The new test drives the native route
with a mocked upstream and asserts the kwargs reach `apply()`. Full
local `pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — ran
lint + a standalone logic check; full pytest deferred to CI (local OOM,
disclosed above)
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- No new dependencies; one import plus three call-site kwargs and a
test.
- @JerrettDavis tagging you — this is the Gemini sibling of the #1534
chat-path fix; the profile/config knobs are currently ignored for the
whole Gemini/Vertex surface, so it may be worth a look when you have a
moment.
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
Adryan Eka Vandra
|
bb2acf700a
|
fix(proxy): honor x-headroom-base-url on /v1/messages route (#1763)
## Description The Anthropic Messages route (`POST /v1/messages`) ignored the `x-headroom-base-url` per-request upstream override and unconditionally forwarded to `api.anthropic.com`. `handle_anthropic_messages` already accepts `upstream_base_url` (it builds the upstream URL via `build_copilot_upstream_url`), but the route never passed it. Clients that speak the Anthropic Messages wire format while authenticating against a non-Anthropic gateway (e.g. OpenCode Zen's "Go" tier) were forwarded to the real Anthropic API, which rejected the gateway key with `401 invalid x-api-key`. The route now reads and trims `x-headroom-base-url` and passes it through as `upstream_base_url`, mirroring the OpenAI-compatible routes and the generic passthrough route (`proxy_routes.py:996`). Closes #1760 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/providers/proxy_routes.py`: the `/v1/messages` route reads `x-headroom-base-url`; when present it strips whitespace and a trailing slash and passes the value as `upstream_base_url` to `handle_anthropic_messages`. Absent or whitespace-only headers keep the previous default (`api.anthropic.com`). - `tests/test_proxy/test_anthropic_upstream_header.py`: new test module pinning the route contract (header present, absent, empty, whitespace-only, trimming + trailing-slash stripping). - `docs/content/docs/configuration.mdx`: new "Proxy upstream override (`x-headroom-base-url`)" subsection under Per-Request Overrides documenting the header across the OpenAI, Anthropic Messages, and passthrough routes. - `CHANGELOG.md`: `Unreleased > Fixed` entry for the `/v1/messages` override. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_proxy/ -k "anthropic or passthrough or bedrock" collected 140 items / 91 deselected / 49 selected tests/test_proxy/test_anthropic_upstream_header.py .... [ 65%] ... 49 passed, 91 deselected, 1 warning in 79.68s $ ruff check headroom/providers/proxy_routes.py tests/test_proxy/test_anthropic_upstream_header.py All checks passed! $ mypy headroom/providers/proxy_routes.py Success: no issues found in 1 source file ``` ## Real Behavior Proof Ran the actual `headroom proxy` against a local mock upstream (a tiny HTTP server on `127.0.0.1:9911` that logs the path it receives) to reproduce the issue's before/after. - Environment: local, macOS, Python 3.12; ran `headroom proxy --port 8799` against a local mock upstream (a tiny HTTP server on `127.0.0.1:9911` that logs the path it receives). - Exact command / steps: started the proxy and the mock upstream, then sent one `POST /v1/messages` **with** the override header and one **without** it (negative control), using these two `curl` commands. ```bash # WITH the override header — expect routing to the mock at 127.0.0.1:9911 curl http://127.0.0.1:8799/v1/messages \ -H "content-type: application/json" -H "anthropic-version: 2023-06-01" \ -H "x-headroom-base-url: http://127.0.0.1:9911" \ -H "x-api-key: zen-test-key" \ -d '{"model":"glm-5.2","max_tokens":16,"messages":[{"role":"user","content":"hi"}]}' # WITHOUT the override header — expect routing to the real api.anthropic.com curl http://127.0.0.1:8799/v1/messages \ -H "content-type: application/json" -H "anthropic-version: 2023-06-01" \ -H "x-api-key: sk-ant-fake" \ -d '{"model":"claude-3-5-sonnet-20241022","max_tokens":16,"messages":[{"role":"user","content":"hi"}]}' ``` - Observed result: with the header, the mock upstream logged `HIT path=/v1/messages x-api-key=zen-test-key` and the proxy returned `HTTP 200`, confirming the request was routed to `<x-headroom-base-url>/v1/messages` carrying the gateway key. Without the header, the request went to the real `api.anthropic.com` (returned `HTTP 401` with a genuine `request_id` and `{"type":"authentication_error","message":"invalid x-api-key"}`) and the mock received no additional hit — matching the pre-fix behavior in the issue. Also verified by TDD: the two override unit cases failed before the route change (`assert None == 'https://opencode.ai/zen/go'`) and passed after it; all 4 new cases and 49 related proxy tests are green. - Not tested: a request against the real OpenCode Zen gateway (no credentials); the gateway path is verified with a local mock upstream instead. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - Manual testing against the real OpenCode Zen gateway is N/A (no credentials); a local mock upstream is used instead to prove the routing (see Real Behavior Proof). - Scope is limited to `/v1/messages`. The related `/v1/messages/count_tokens` route uses a fixed passthrough target and is out of scope for this issue. |
||
|
Rod Boev
|
62cd3072a2
|
feat(ccr): wire retrieve-tool interception into OpenAI Responses handler (#1898)
## Description Refs #1877. `handle_openai_responses` (the `/v1/responses` HTTP handler) had zero CCR / `headroom_retrieve` wiring, so a retrieve `function_call` in a Responses API reply passed straight through to the client instead of being resolved server-side, unlike the parallel chat-completions backend path (`handle_openai_chat`, ~2775-2848), which already intercepts `headroom_retrieve` tool calls via `ccr_response_handler.has_ccr_tool_calls()` / `handle_response()`. This PR is scoped to the core interception gap only. The issue's proposals A (egress scrubber) and B/C (event-level SSE parsing/splicing for true mid-stream interception) are out of scope here; the streaming case is instead handled by forcing a buffered (non-streaming) upstream call when `headroom_retrieve` is offered, matching the existing buffered-CCR pattern in the Anthropic handler. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/ccr/response_handler.py`: added an `"openai_responses"` provider branch to `CCRResponseHandler`; `_extract_tool_calls` reads flat `function_call` items from the top-level `output[]` array, tool-call IDs key off `call_id`, and `_extract_assistant_message` / `_create_tool_result_message` return sentinel-keyed item lists that `handle_response()` extends into the running item history. - `headroom/ccr/tool_injection.py`: added a `parse_tool_call` branch for `"openai_responses"` where name and arguments are flat on the item. - `headroom/proxy/handlers/openai.py`: detects non-streaming `headroom_retrieve` function calls and runs `ccr_response_handler.handle_response()` with a stateless continuation that resends the full `input[]` item history. - `headroom/proxy/handlers/openai.py`: forces `stream:true` requests with `headroom_retrieve` available through a buffered `stream:false` upstream call, resolves retrieval server-side, then reconstructs a minimal Responses SSE stream for the client. - `headroom/proxy/handlers/openai.py`: treats `ccr_response_handler` as optional on `OpenAIHandlerMixin` consumers, so handlers without CCR support keep the existing Codex routing, streaming, header stripping, memory timeout, and compression fail-open behavior. - Non-CCR streaming requests are unaffected; they still go through `_stream_response()` as before. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_ccr_response_handler_openai_responses.py -q`) - [x] Integration tests pass (`uv run pytest tests/test_proxy/test_openai_responses_ccr.py -q`) - [x] Regression tests pass (`uv run pytest tests/test_openai_codex_routing.py -q`) - [x] Linting passes (`uv run ruff check headroom/proxy/handlers/openai.py tests/test_openai_codex_routing.py tests/test_proxy/test_openai_responses_ccr.py tests/test_ccr_response_handler_openai_responses.py`) - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_openai_codex_routing.py -q tests\test_openai_codex_routing.py .................... [100%] 20 passed in 0.51s $ uv run pytest tests/test_proxy/test_openai_responses_ccr.py tests/test_ccr_response_handler_openai_responses.py -q tests\test_proxy\test_openai_responses_ccr.py .... [ 25%] tests\test_ccr_response_handler_openai_responses.py ............ [100%] 16 passed, 1 warning in 27.51s $ uv run ruff check headroom/proxy/handlers/openai.py tests/test_openai_codex_routing.py tests/test_proxy/test_openai_responses_ccr.py tests/test_ccr_response_handler_openai_responses.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows, Python 3.12 via uv-managed venv, local PR worktree on branch `pr/1877-ccr-responses-interception`, no live LLM provider needed because the tests stub upstream HTTP and CCR continuation behavior. - Exact command / steps: Ran `uv run pytest tests/test_openai_codex_routing.py -q` to reproduce the CI-failing Codex routing surface after the optional-handler fix; ran `uv run pytest tests/test_proxy/test_openai_responses_ccr.py tests/test_ccr_response_handler_openai_responses.py -q` to cover the positive Responses CCR interception path; ran targeted Ruff on the touched handler and related tests. - Observed result: Codex routing tests that previously failed with `AttributeError: '_DummyOpenAIHandler' object has no attribute 'ccr_response_handler'` now pass; Responses CCR still detects and resolves `headroom_retrieve` when a real proxy installs `ccr_response_handler`; non-CCR streaming requests still route through `_stream_response()`. - Not tested: true event-level mid-stream Responses SSE splicing and client-bound egress marker scrubbing are out of scope for this PR and remain future work from issue #1877's broader proposals. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes No documentation or changelog update was made because this is internal proxy CCR behavior, not a user-facing command or configuration change. |
||
|
Tejas Chopra
|
7c2f0ea079
|
feat(cache): provider-agnostic cache-mode delta + cc-agnostic prefix comparison (#1868)
## Description <!-- Briefly explain the change and why it is needed. --> Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # Paste relevant command output or artifact links here ``` ## Real Behavior Proof - Environment: - Exact command / steps: - Observed result: - Not tested: ## Review Readiness - [ ] I have performed a self-review - [ ] This PR is ready for human review ## Checklist - [ ] My code follows the project's style guidelines - [ ] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) Add screenshots to help explain your changes. ## Additional Notes <!-- Mention any N/A checklist items, tradeoffs, follow-ups, or maintainer context. --> |
||
|
Vinay Gupta
|
f18c6bd896
|
fix(codex): OpenCode Zen telemetry attribution (#1648)
## Description Fixes #1602. OpenCode Zen custom-base requests can reach Headroom through the generic passthrough path, but that route was not supplying endpoint/provider metadata for Zen chat completions. This made forwarded Zen traffic invisible in dashboard provider, usage, and token telemetry. Closes #1602 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added a narrow OpenCode Zen custom-base classifier for `POST /zen/v1/chat/completions` on `opencode.ai` and `www.opencode.ai`. - Passed `endpoint_name="chat/completions"` and `provider="zen"` into catch-all passthrough telemetry for matching Zen traffic. - Attributed normalized OpenCode transport traffic (`/v1/chat/completions` with `x-headroom-original-path: /zen/v1/chat/completions`) to `zen` for request outcomes while keeping the OpenAI parser path unchanged. - Added coverage for direct catch-all routing, normalized original-path routing, token usage outcome recording, and false-positive paths like `/mcp/v1/chat/completions`, `/npm/v1/chat/completions`, and `/context7/v1/chat/completions`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ rtk pytest tests/test_custom_base_passthrough_telemetry.py -q Pytest: 4 passed $ rtk uvx --from ruff==0.15.17 ruff check headroom/proxy/handlers/openai.py headroom/providers/proxy_routes.py tests/test_custom_base_passthrough_telemetry.py tests/test_provider_proxy_routes.py tests/test_proxy/test_openai_transport_path_prefix.py All checks passed! $ rtk uvx --from ruff==0.15.17 ruff format --check headroom/proxy/handlers/openai.py headroom/providers/proxy_routes.py tests/test_custom_base_passthrough_telemetry.py tests/test_provider_proxy_routes.py tests/test_proxy/test_openai_transport_path_prefix.py 5 files already formatted $ rtk /Library/Frameworks/Python.framework/Versions/3.13/bin/python3 -m py_compile headroom/proxy/handlers/openai.py headroom/providers/proxy_routes.py tests/test_custom_base_passthrough_telemetry.py tests/test_provider_proxy_routes.py tests/test_proxy/test_openai_transport_path_prefix.py # passed $ rtk git diff --check # passed ``` GitHub Actions also passed after the final push, including CI, Docker native/wrap/init E2E, security, lint, and PR governance. ## Real Behavior Proof - Environment: local worktree on macOS plus GitHub Actions for PR #1648. - Exact command / steps: ran focused pytest coverage for Zen passthrough telemetry, Ruff check/format validation on touched files, Python compile validation, `git diff --check`, and waited for the full GitHub Actions rollup. - Observed result: Zen custom-base chat completions now record request outcomes as provider `zen` with endpoint `chat/completions`; false-positive OpenCode paths remain unattributed to Zen; GitHub checks are green. - Not tested: full local test suite did not collect in this worktree because the native `headroom._core` extension is not installed. `rtk npm --prefix plugins/opencode test` is also blocked locally because `vitest` is not installed in `plugins/opencode/node_modules`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes The documentation and CHANGELOG checklist items are not applicable for this narrow telemetry bug fix. No new comments were added because the code path is covered by narrowly named helper/test cases. |
||
|
Abhay Singh
|
7ff842da17
|
fix(proxy/openai): thread savings-profile kwargs into chat completions (#1606)
## Description OpenAI-compatible `/v1/chat/completions` requests didn't receive the same proxy savings/profile kwargs as the other compression paths. The live chat handler (`handle_openai_chat` in `headroom/proxy/handlers/openai.py`) called `openai_pipeline.apply()` with only `model_limit` / `context` / `frozen_message_count` / `biases` / `compression_policy` — it never passed `proxy_pipeline_kwargs(self.config)`. So when the proxy runs with `HEADROOM_SAVINGS_PROFILE=agent-90`, the effective config reports user/system-message compression and `target_ratio=0.10`, but the real chat path silently dropped all of it. OpenAI-compatible clients such as OpenCode kept protecting user messages and missed the configured profile. For contrast, `handlers/anthropic.py` passes `**proxy_pipeline_kwargs(self.config)` to every `apply()` call, and so does the dedicated OpenAI compress endpoint in this same module — only the two chat-completions `apply()` sites were missing it. Closes #1534 ## Fix Add `**proxy_pipeline_kwargs(self.config)` to both chat-path `apply()` calls (the token-mode branch and the non-token branch): ```python lambda: self.openai_pipeline.apply( messages=messages, model=model, model_limit=context_limit, context=extract_user_query(messages), frozen_message_count=openai_frozen_count, biases=_hook_biases, compression_policy=compression_policy, **proxy_pipeline_kwargs(self.config), # ← added ) ``` `proxy_pipeline_kwargs` is already imported in the module and is the exact helper the Anthropic handler and the OpenAI compress endpoint use, so the chat path now matches them. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/proxy/handlers/openai.py`: pass `**proxy_pipeline_kwargs(self.config)` on both `apply()` call sites in `handle_openai_chat` (token-mode and non-token branches). - `tests/test_proxy/test_openai_chat_savings_profile.py`: new regression test driving the chat handler with `savings_profile="agent-90"` and asserting the profile knobs reach `apply()`. - `CHANGELOG.md`: Bug Fixes entry under Unreleased. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output The new test drives the real chat handler through the `create_app` + `TestClient` harness with a recording `apply()` stub. Before the fix it captures exactly the five kwargs the issue describes (no profile knobs); after the fix the profile knobs are present: ```text # before the fix (openai.py reverted, test kept) E AssertionError: assert None is True E + where None = {...}.get('compress_user_messages') # captured kwargs were: biases, compression_policy, messages, model, # model_limit, context, frozen_message_count — no profile knobs FAILED tests/test_proxy/test_openai_chat_savings_profile.py::test_chat_completions_threads_savings_profile_kwargs_into_apply # after the fix tests\test_proxy\test_openai_chat_savings_profile.py . ======================== 1 passed, 1 warning in 39.44s ======================== ``` No regression in the existing chat backend-path suite: ```text $ uv run pytest tests/test_proxy/test_openai_backend_path.py ======================== 5 passed, 1 warning in 15.78s ======================== $ uv run ruff check headroom/proxy/handlers/openai.py tests/test_proxy/test_openai_chat_savings_profile.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12.11, headroom built from this branch (`uv sync --extra dev`), proxy config `savings_profile="agent-90"`, `optimize=True`, `backend="anyllm"` with a mocked OpenAI upstream. - Exact command / steps: started the app with `create_app(config)`, replaced `proxy.openai_pipeline.apply` with a recording stub, and POSTed a real `/v1/chat/completions` request with a large user message so the compression decision fires. Inspected the kwargs the handler actually passed to `apply()`. - Observed result: before the fix the recorded `apply()` kwargs were `{biases, compression_policy, messages, model, model_limit, context, frozen_message_count}` — no profile knobs. After the fix the same call also carries `compress_user_messages=True`, `compress_system_messages=True`, `target_ratio=0.10`, `min_tokens_to_compress=120` (the agent-90 profile), matching the issue's "Expected". - Not tested: did not stand up a real OpenAI/OpenCode upstream end-to-end (no live key in this environment); the upstream is mocked and the assertion is on the kwargs the proxy threads into the compression pipeline, which is exactly what the bug was about. Did not run the full `mypy headroom` pass (two-line kwarg addition, no new types). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - Two-line change plus comments; no new dependencies. Reuses the existing `proxy_pipeline_kwargs` helper, so behavior is consistent across Anthropic, the OpenAI compress endpoint, and now the OpenAI chat path. - @chopratejas flagging you for review — this aligns the OpenAI chat path with the savings-profile handling the other providers already had. Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
Tejas Chopra
|
248ae0f3e0
|
fix(proxy): freeze must forward cached (compressed) prefix byte-identical — stop token-mode cache busting (#1850)
The freeze path (both providers) emits the agent's ORIGINAL bytes for a frozen message, but the provider cached whatever we FORWARDED last turn (the compressed form). Forwarding original then mismatches the cached prefix and busts it from that point — re-creating the whole suffix. Measured on a real SWE-bench run: 100% of attributed misses were prefix_change, ~56% of ALL cache-writes were bust-induced (2.8M tokens), driving cache_create +150% and cost +41% vs baseline. Cache mode already avoided this via _extract_cache_stable_delta (replay the previously-forwarded prefix, compress only the delta). Token mode called apply(frozen_count) directly, which forwards original for the frozen region. Fix: add a shared, provider-agnostic overlay_cached_prefix() that replays the previously-forwarded (cached, compressed) prefix byte-identical, append-only guarded and idempotent, and apply it in BOTH the Anthropic and OpenAI handlers right before forwarding. This makes freezing byte-identical in every mode, so the only remaining difference between "token" and "cache" mode is how large a mutable (still-compressible) tail each leaves — not whether the frozen prefix busts the cache. Tests: - test_cache_prefix_overlay.py: the helper (replay, append-only guard, idempotence). - test_cross_turn_cache_safety.py: the invariant that was missing — drive the REAL tracker + freeze + overlay over multiple append-only turns against a simulated provider prefix cache and assert the forwarded prefix stays byte-identical turn-over-turn. Load-bearing: it fails (detects the bust) without the overlay. ## Description <!-- Briefly explain the change and why it is needed. --> Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # Paste relevant command output or artifact links here ``` ## Real Behavior Proof - Environment: - Exact command / steps: - Observed result: - Not tested: ## Review Readiness - [ ] I have performed a self-review - [ ] This PR is ready for human review ## Checklist - [ ] My code follows the project's style guidelines - [ ] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) Add screenshots to help explain your changes. ## Additional Notes <!-- Mention any N/A checklist items, tradeoffs, follow-ups, or maintainer context. --> |
||
|
Kirill
|
6c48ac81f2
|
fix(proxy): honor x-headroom-base-url in dedicated OpenAI handlers (#1502)
## Description The dedicated OpenAI handlers (`/v1/chat/completions`, `/v1/responses`) ignore the `x-headroom-base-url` request header that the opencode/CLI transports already send on every routed request (`plugins/opencode/src/transport.ts`) and that the generic passthrough route already honors (`providers/proxy_routes.py:953`). As a result, OpenAI-compatible gateways (LiteLLM, CPA, self-hosted vLLM, Azure OpenAI) route correctly for passthrough traffic, but the dedicated chat/responses handlers fall back to the default `OPENAI_API_URL` and send the request — and the user's provider key — to the wrong upstream. This forces OpenCode users behind a custom gateway to run a hand-rolled plugin that re-spawns the proxy with `OPENAI_TARGET_API_URL` instead of the supported `HeadroomPlugin`. Refs #1503 (feature-request issue with full spec — API surface, failure modes, security considerations). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) Non-breaking: when the header is absent (the common case), behavior is identical to before — `_resolve_openai_upstream` falls back to `self.OPENAI_API_URL`. ## Changes Made - Added `OpenAIHandlerMixin._resolve_openai_upstream(request)` — returns `request.headers.get("x-headroom-base-url") or self.OPENAI_API_URL`. Prefers the header, falls back to the configured URL. - Used it at the two direct-path HTTP upstream sites: - `handle_openai_chat` → `build_copilot_upstream_url(self._resolve_openai_upstream(request), "/v1/chat/completions")` - `handle_openai_responses` → `build_copilot_upstream_url(self._resolve_openai_upstream(request), "/v1/responses")` - This makes the dedicated handlers behave identically to the catch-all passthrough and the Azure path (`_select_passthrough_base_url`, `providers/proxy_routes.py:66,:953`), which already read the same header. - The header is already stripped before forwarding by `helpers._strip_internal_headers`, so no upstream leakage / fingerprinting is introduced. - CHANGELOG entry under `### Bug Fixes`. ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) — not run locally (maturin native build not available in my env; covered by CI) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output New `tests/test_proxy/test_openai_upstream_header.py` pins the resolution contract (3 cases): ```text $ pytest tests/test_proxy/test_openai_upstream_header.py -q ... collected 3 items tests/test_proxy/test_openai_upstream_header.py ... [100%] ========================= 3 passed, 1 warning in 0.25s ========================= ``` Fail-before confirmed (unpatched handler raises `AttributeError: _resolve_openai_upstream`): ```text FAILED tests/test_proxy/test_openai_upstream_header.py::test_header_overrides_configured_url FAILED tests/test_proxy/test_openai_upstream_header.py::test_missing_header_falls_back_to_configured_url FAILED tests/test_proxy/test_openai_upstream_header.py::test_empty_header_falls_back_to_configured_url ========================= 3 failed, 1 warning in 0.29s ========================= ``` Lint/format: ```text $ ruff check headroom/proxy/handlers/openai.py tests/test_proxy/test_openai_upstream_header.py Ruff: No issues found $ ruff format --check headroom/proxy/handlers/openai.py tests/test_proxy/test_openai_upstream_header.py 2 files already formatted ``` ## Real Behavior Proof - Environment: macOS (darwin), Python 3.12 (pipx install of `headroom-ai`), Headroom proxy `headroom proxy --port 8787` with `OPENAI_TARGET_API_URL=https://cpa.funxyz.fun` (an OpenAI-compatible gateway — "CLI Proxy API"). OpenCode with a custom `cpa` provider (`@ai-sdk/openai-compatible`, `baseURL: https://cpa.funxyz.fun/v1`) using the official `HeadroomPlugin`. - Exact command / steps: traced the bug in the installed package source — confirmed `handle_openai_chat` builds its upstream URL from `self.OPENAI_API_URL` only (`proxy/handlers/openai.py:2487`), never reading `x-headroom-base-url`, while `providers/proxy_routes.py:953` reads it for passthrough. Then applied this patch and re-imported the handler from the repo source via `PYTHONPATH`. - Observed result: before the patch, `/v1/chat/completions` requests ignored the `x-headroom-base-url: https://cpa.funxyz.fun` header (set by the opencode transport) and routed to the default upstream, failing against a non-OpenAI gateway — requiring a custom respawn-plugin workaround. After the patch, `_resolve_openai_upstream` returns the header value and the request forwards to the configured gateway; the official `HeadroomPlugin` works without the env-var workaround. Unit tests pass (3/3) and fail on the unpatched handler (3/3). - Not tested: full `uv sync` CI matrix (native `headroom._core` maturin build unavailable locally, so `headroom.proxy.server` import chain that pulls `transforms/content_router` can't be exercised here — the edited handler module imports fine and the focused unit tests exercise the new method directly). WebSocket/Codex paths (`handle_openai_responses_ws`, `_ws_http_fallback`) — intentionally out of scope (see Additional Notes). `mypy headroom` — deferred to CI. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation — no public API/docs surface; the header is already documented as an internal control flag in `helpers.py:1489-1495` - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes **Scope boundary — WebSocket paths intentionally unchanged.** The two WS sites (`handle_openai_responses_ws`, `_ws_http_fallback`) are Codex-specific and left as-is: 1. They short-circuit to `chatgpt.com` under ChatGPT-session auth (not arbitrary gateways). 2. The WS path strips `x-headroom-base-url` from `upstream_headers` (`_strip_internal`, ~line 3756) before the upstream URL is built, and `_ws_http_fallback` receives already-stripped headers as a parameter. Honoring the header there would require threading it through the WS internals and changing a signature, for a path a custom OpenAI-compatible WebSocket gateway is unlikely to use. The HTTP paths cover the realistic gateway case. Happy to do it as a follow-up if maintainers want it. **Issue-first.** This is a behaviour change, so per CONTRIBUTING a feature-request issue (#1503) is open for triage with the full spec (API surface, user stories, failure modes, security). This PR implements it; holding for maintainer 👍 before treating as ready to merge. --------- Co-authored-by: ShutovKS <shutovks@example.local> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Vinay Gupta
|
d337e3b828
|
fix(proxy): handle streaming CCR retrieval (#1451)
## Description Fixes Anthropic-compatible streaming requests that can emit the internal `headroom_retrieve` CCR tool. When a `stream: true` request includes the CCR retrieve tool and response handling is enabled, Headroom now buffers the upstream call as `stream: false`, lets the existing CCR response handler retrieve and continue, and returns the final result as Anthropic SSE so streaming clients do not see the internal tool call. Closes #1450 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Detect direct Anthropic-compatible `stream: true` requests where `headroom_retrieve` is available and CCR response handling is enabled. - Route those requests through the existing buffered/non-stream CCR response handler, then convert the final response back to `text/event-stream`. - Fail closed with a 502 SSE error if a buffered response still contains `headroom_retrieve` after CCR handling, instead of leaking the internal tool to the client. - Preserve Anthropic `thinking`, `redacted_thinking`, signatures, and citations when converting response JSON back to SSE. - Add regression coverage for handled CCR retrieval, unused CCR tool availability, normal streaming passthrough, mixed client/CCR tool fail-closed behavior, and SSE conversion preservation. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ rtk gh pr checks 1451 --repo headroomlabs-ai/headroom CI Checks Summary: [ok] Passed: 20 [FAIL] Failed: 0 Relevant CI commands from .github/workflows/ci.yml: - ruff check . - ruff format --check . - mypy headroom --ignore-missing-imports - pytest tests scripts/tests $ rtk python3 -m py_compile headroom/proxy/handlers/anthropic.py headroom/proxy/handlers/streaming.py tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py tests/test_sse_thinking_blocks.py # passed, no output $ rtk pytest tests/test_sse_thinking_blocks.py -q Pytest: 6 passed $ MACOSX_DEPLOYMENT_TARGET=15.0 rtk uv run --python 3.13 pytest tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py -q Failed before test collection while building the local editable package: esaxx-rs build failed with fatal error: 'cstdint' file not found. ``` ## Real Behavior Proof - Environment: GitHub Actions CI on PR #1451 plus local macOS worktree `fix/1450-ccr-streaming-retrieve`. - Exact command / steps: CI ran lint, type checking, build, unit-test shards, dashboard tests, extras tests, and e2e jobs; locally ran syntax checks and the SSE conversion regression tests. - Observed result: CI passed 20 checks with 0 failures; local syntax checks passed; `tests/test_sse_thinking_blocks.py` passed with 6 tests. - Not tested: the new proxy-level regression test was not run locally because the local native extension build fails in `esaxx-rs` before proxy tests can collect; it is included in the CI-tested suite. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes - Scope: this handles the direct Anthropic-compatible HTTP `/v1/messages` path. The configured Bedrock/backend streaming path does not share this CCR continuation machinery in this PR. - Documentation, CHANGELOG, code-comment, and local-full-test checklist items are N/A for this narrow bug fix or not true locally. |
||
|
Rod Boev
|
c19347c310
|
fix(opencode): preserve custom OpenAI gateway paths (#1596)
## Description Custom OpenAI-compatible gateways mounted under provider-specific prefixes could miss Headroom's dedicated OpenAI compression routes when used through the OpenCode transport. A request such as `https://open.bigmodel.cn/api/coding/paas/v4/chat/completions` was replayed to the proxy at `/api/coding/paas/v4/chat/completions`, so the proxy selected catch-all passthrough instead of `/v1/chat/completions`. This change keeps the proxy-facing entrypoints stable on `/v1/chat/completions` and `/v1/responses` for OpenAI-compatible suffixes, while preserving the original upstream path in an internal header so the dedicated OpenAI handlers can reconstruct the real provider URL. Closes #1582 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Normalize opencode-routed OpenAI-compatible `/chat/completions` and `/responses` requests onto the proxy's stable `/v1/*` routes. - Preserve the original upstream pathname in an internal `x-headroom-original-path` signal for dedicated OpenAI handler reconstruction. - Reconstruct dedicated OpenAI upstream URLs from `x-headroom-base-url` plus the preserved path prefix, while preserving request query strings and rejecting non-HTTP base hints. - Keep nearby non-OpenAI paths such as `/base/v1/messages` on existing passthrough behavior. - Add focused transport and proxy regression coverage for prefixed gateway paths, invalid fallback cases, and internal-header stripping. ## Testing - [x] Transport regression tests pass (`npm --prefix plugins/opencode test -- src/transport.test.ts`) - [x] Proxy regression tests pass (`uv run pytest tests/test_proxy/test_openai_transport_path_prefix.py -q`) - [x] Linting passes (`uv run ruff check headroom/proxy/handlers/openai.py tests/test_proxy/test_openai_transport_path_prefix.py`) - [x] Type checking passes (`npm --prefix plugins/opencode run typecheck`) - [x] New tests added for the bugfix - [ ] Manual testing performed ### Test Output ```text npm --prefix plugins/opencode test -- src/transport.test.ts PASS, 11 tests passed. npm --prefix plugins/opencode run typecheck PASS uv run pytest tests/test_proxy/test_openai_transport_path_prefix.py -q PASS, 7 tests passed. uv run ruff check headroom/proxy/handlers/openai.py tests/test_proxy/test_openai_transport_path_prefix.py PASS, all checks passed. ``` ## Real Behavior Proof - Environment: Windows, Python 3.12 via `uv`, Node 18+, focused OpenCode transport and proxy handler tests. - Exact command / steps: on `origin/main`, copy the updated `plugins/opencode/src/transport.test.ts` into a base worktree and run `npm --prefix plugins/opencode test -- src/transport.test.ts`; on this branch, rerun that transport test plus `npm --prefix plugins/opencode run typecheck` and `uv run pytest tests/test_proxy/test_openai_transport_path_prefix.py -q`. - Observed result: the base worktree fails because prefixed `/chat/completions` and `/responses` requests still enter the proxy at their provider path, while this branch passes with `/v1/chat/completions` and `/v1/responses`, preserves `x-headroom-original-path`, reconstructs the provider-prefixed upstream URL and query string, falls back safely on invalid hints, and keeps nearby `/base/v1/messages` traffic on passthrough. - Not tested: full CI suite, live BigModel traffic, and generic catch-all passthrough compression. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes This completes the transport contract introduced in https://github.com/headroomlabs-ai/headroom/pull/1573 by keeping prefixed OpenAI-compatible traffic on Headroom's stable `/v1/*` surface while preserving the real upstream path for dedicated-handler reconstruction. https://github.com/headroomlabs-ai/headroom/pull/1367 is adjacent global proxy configuration work for direct deployments; this PR is the per-request OpenCode transport fix for custom upstream path prefixes. `CHANGELOG.md` is intentionally unchanged because this repo's release pipeline generates changelog entries from conventional commits. This stays scoped to `/chat/completions` and `/responses` suffixes. Generic catch-all passthrough compression remains separate from this bugfix slice. |
||
|
Omar Garcia
|
8e0dadfe02
|
fix: restore token-mode compression on frozen prefixes (#1489)
## Description Fixes token-mode compression for continued Claude Code turns with a frozen prefix when the client has not already supplied `headroom_retrieve`. The previous guard returned before request-side compression could run in token mode. This keeps the non-token safety behavior, but lets token mode use the existing marker-triggered CCR tool injection override so emitted markers stay redeemable. Closes #1487. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Let Anthropic token mode run request-side compression even when the client did not pre-register `headroom_retrieve`. - Kept the deferred-injection skip for cache-mode coverage. - Added a regression for the frozen-prefix token-mode path. - Updated `CHANGELOG.md` for the user-facing behavior change. ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ rtk uv run pytest -q tests/test_proxy/test_anthropic_ccr_deferred_injection.py 15 passed, 1 warning in 2.73s $ rtk uv run ruff check headroom/proxy/handlers/anthropic.py tests/test_proxy/test_anthropic_ccr_deferred_injection.py All checks passed! $ rtk uv run ruff format --check headroom/proxy/handlers/anthropic.py tests/test_proxy/test_anthropic_ccr_deferred_injection.py 2 files already formatted ``` ## Real Behavior Proof - Environment: macOS, Python 3.12.9, local FastAPI `TestClient`, Anthropic proxy path, `mode=token`, `ccr_inject_tool=True`, frozen prefix count = 1, no client-supplied `headroom_retrieve`. - Exact command / steps: ran a local `rtk uv run python` repro that builds `create_app(ProxyConfig(...))`, forces compression on the Anthropic path, simulates a frozen prefix, and posts `/v1/messages`. - Observed result: local `TestClient` request returned `STATUS=200`; token-mode frozen-prefix compression ran once with `FROZEN_MESSAGE_COUNT=1`; the forwarded message was the CCR marker; forwarded tools included `headroom_retrieve`. ```text STATUS= 200 FROZEN_MESSAGE_COUNT= 1 COMPRESSION_CALLS= 1 FORWARDED_MESSAGES= [{'role': 'user', 'content': '[100 items compressed to 10. Retrieve more: hash=abc123def456abc123def456]'}] FORWARDED_TOOLS= ['headroom_retrieve'] ``` - Not tested: live Claude Code session against a real Anthropic upstream, full repo-wide `uv run pytest`, and `mypy headroom`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas (N/A: no new hard-to-follow block needed) - [x] I have made corresponding changes to the documentation (N/A: changelog update covers this user-facing bug fix) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A; proxy behavior only. ## Additional Notes The pytest run still emits the existing Starlette/httpx deprecation warning from `fastapi.testclient`; this PR does not touch that dependency path. |
||
|
Lucas Santos
|
43494ff526
|
fix(proxy): stop re-compressing headroom_retrieve output and emitting unredeemable markers (#1323)
## Description Two related CCR problems that both end in unreadable content. The first one (#1077) is an infinite loop. Any tool output over ~500 bytes gets replaced with a `<<ccr:hash>>` marker, and you call `headroom_retrieve` to get the original back. But the proxy then compresses the *retrieve response too*, so what comes back is a brand new marker. Retrieve that one and you get another marker. The second one (#1006), the proxy makes two independent decisions per request: SmartCrusher compresses, and the `headroom_retrieve` tool gets injected. The injection is deferred when there's a frozen message prefix (`frozen_message_count > 0`), but compression keeps running anyway. So the agent receives `[... compressed to N. Retrieve more: hash=...]` markers with no `headroom_retrieve` tool to redeem them. For #1077, SmartCrusher now skips `headroom_retrieve` results. Before crushing a tool message (OpenAI `role=tool`) or tool-result block (Anthropic `type=tool_result`), it checks whether that tool id maps to the CCR tool, and if so leaves it alone. Retrieved content stays readable. For #1006, compression and injection are no longer decided in isolation. The injection decision is extracted into `should_inject_ccr_tool`, which the Anthropic handler calls: when injection was deferred because of a frozen prefix but compression just emitted new markers, it injects the tool anyway, so a marker is never handed to an agent that can't act on it. The existing session-sticky dedup means sessions that already have the tool don't get it re-injected and don't lose their cache. Closes #1077 Closes #1006 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/smart_crusher.py`: exempt `headroom_retrieve` results from compression on both the OpenAI `role=tool` and Anthropic `type=tool_result` paths. - `headroom/proxy/helpers.py`: add `should_inject_ccr_tool`, the deferral-plus-override decision the handler used to inline, so the #1006 behaviour is testable at the decision point. - `headroom/proxy/handlers/anthropic.py`: call `should_inject_ccr_tool` to couple injection with compression; rename the misleading `frozen_prefix=` log key to `frozen_message_count=`. - `tests/test_transforms/test_smart_crusher_ccr_retrieve_exemption.py` and `tests/test_proxy/test_ccr_frozen_prefix_coupling.py`: new tests; the frozen-prefix test now drives `should_inject_ccr_tool` so it would fail if the override were removed. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run --extra dev python -m pytest tests/test_proxy/test_ccr_frozen_prefix_coupling.py tests/test_transforms/test_smart_crusher_ccr_retrieve_exemption.py -q 5 passed, 1 skipped ruff: All checks passed! mypy: Success: no issues found ``` The SmartCrusher test skips locally because the Rust extension `.so` is built for a different OS, the same skip the existing SmartCrusher tests take locally. It runs in CI where the extension is built. ## Real Behavior Proof - Environment: macOS, Python 3.13, this branch. - Exact command / steps: `uv run --extra dev python -m pytest tests/test_proxy/test_ccr_frozen_prefix_coupling.py tests/test_transforms/test_smart_crusher_ccr_retrieve_exemption.py -q`. The frozen-prefix test calls `should_inject_ccr_tool` (the function the Anthropic handler now uses) with a frozen prefix and freshly emitted markers, then drives `apply_session_sticky_ccr_tool` end to end and asserts `headroom_retrieve` lands in the outbound tools. The exemption test runs a `headroom_retrieve` tool result through SmartCrusher on both the OpenAI and Anthropic shapes. - Observed result: 5 passed, 1 skipped. The retrieve tool is injected even under a frozen prefix once markers exist, and is not injected when no markers were emitted. Removing the handler override flips `should_inject_ccr_tool` and fails the test. - Not tested: a full live proxy session. The behaviours are covered at the decision, transform, and handler-call level by the new tests. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes This one touches compression gating, so it's worth a careful read on the injection coupling, that's the part where a wrong call would re-introduce data loss. 1. Tool results with no id mapping still compress, marked with `# ponytail:` comments. Only ids we can positively identify as the CCR tool are exempted. 2. The injection coupling keys off `injector.has_compressed_content`, so the tool only shows up when there's actually something to retrieve. --------- Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
Rod Boev
|
3be2526b76
|
fix(proxy): add an Anthropic buffered read-timeout override (#1331)
## Description Buffered Anthropic `/v1/messages` requests still use Headroom's generic 300-second read timeout, which can produce proxy-generated `502 ReadTimeout` errors on long turns. This adds a dedicated buffered Anthropic timeout, keeps it applied across CCR and memory continuations plus batch paths, and makes the direct server entrypoint enforce the same positive-integer contract as the Click CLI. Closes #1261. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Added `anthropic_buffered_request_timeout_seconds` for buffered Anthropic reads. - Routed `/v1/messages`, CCR continuation, memory continuation, batch create, batch passthrough, and batch results through that timeout. - Enforced the same positive-integer validation for `HEADROOM_ANTHROPIC_BUFFERED_REQUEST_TIMEOUT_SECONDS` and `--anthropic-buffered-request-timeout-seconds` in both startup paths. - Added focused regressions and updated `CHANGELOG.md`. ## Testing - [x] `uv run pytest tests/test_proxy/test_anthropic_buffered_timeout.py tests/test_cli_proxy_improvements.py::TestNewEnvVarWiring` - [x] `uv run ruff check .` - [x] `uv run ruff format . --check` ### Test Output ```text $ uv run pytest tests/test_proxy/test_anthropic_buffered_timeout.py tests/test_cli_proxy_improvements.py::TestNewEnvVarWiring 17 passed in 3.42s $ uv run ruff check . All checks passed! $ uv run ruff format . --check 966 files already formatted ``` ## Real Behavior Proof - Environment: local FastAPI `TestClient` with stubbed retry and HTTP client seams - Exact command / steps: run `uv run pytest tests/test_proxy/test_anthropic_buffered_timeout.py tests/test_cli_proxy_improvements.py::TestNewEnvVarWiring`; the tests build `ProxyConfig(request_timeout_seconds=7, connect_timeout_seconds=3, anthropic_buffered_request_timeout_seconds=19)`, drive `/v1/messages`, `/v1/messages/batches`, `/v1/messages/batches/{batch_id}/results`, a CCR continuation, and a memory continuation through `TestClient`, then verify `HEADROOM_ANTHROPIC_BUFFERED_REQUEST_TIMEOUT_SECONDS=0` falls back to `600`, `--anthropic-buffered-request-timeout-seconds 0` is rejected, and default proxy timeouts stay `read=300` and `write=300` - Observed result: buffered Anthropic paths use `httpx.Timeout(connect=3, read=19, write=7, pool=3)`, continuation requests stay on that same budget, invalid zero-valued startup config is rejected or ignored back to the default, and unrelated proxy timeout defaults stay unchanged - Not tested: live upstream Anthropic latency beyond the focused stubbed-timeout regression ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have added tests that prove the fix - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable |
||
|
Rod Boev
|
2cae13dd79
|
fix(ccr): skip Anthropic marker emission when tool injection is deferred (#1273)
## Description Anthropic request-side CCR can still compress a turn into retrieval markers after the frozen-prefix cache guard suppresses `headroom_retrieve` registration. That leaves the model with marker-only context it cannot redeem, so the proxy silently drops recoverable data on exactly the turns where cache preservation deferred tool injection. This change couples the Anthropic request-side CCR path to tool availability so a turn never emits retrieve-only markers without the retrieval tool, even when token mode or cache-mode prefix replay could otherwise reuse already-compressed marker text. Closes #1006 After a collaborator merged current `main` into this branch, CI also picked up unrelated offline-memory failures from the merged base. Those follow-up changes are test-only: they keep the offline Hugging Face cache lanes skipping cleanly instead of failing in memory tests that are outside the CCR runtime path. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Couple Anthropic request-side CCR compression to the same frozen-prefix guard that already defers `headroom_retrieve` registration. - Keep the existing cache-preservation behavior: frozen-prefix turns stop emitting CCR retrieval markers instead of forcing tool injection into the cached prefix. - Make the skip decision use the effective frozen prefix after token-mode reclamping, so turns that genuinely reclamp to zero still keep normal reversible CCR behavior. - Bypass cached marker reuse in both token mode and cache-mode prefix replay when tool injection is deferred. - Add focused regressions for the Anthropic request-path seams under this bug: - frozen-prefix turns do not emit marker-only payloads - unfrozen turns still keep normal reversible CCR behavior - token-mode reclamp back to zero still compresses normally - existing `headroom_retrieve` tools keep reversible CCR on frozen turns - cache-mode delta reuse and exact-prefix replay both forward original content when retrieval is unavailable - Add a `CHANGELOG.md` entry because the proxy's user-visible Anthropic CCR behavior changes. - Add a shared test skip helper for offline Hugging Face cache misses and apply it to the merged `main` memory tests that were failing only in the offline CI shards after the branch picked up current `main`. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_proxy/test_anthropic_ccr_deferred_injection.py`) - [x] Linting passes (`uv run ruff check . && uv run ruff format . --check`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text uv run pytest tests/test_proxy/test_anthropic_ccr_deferred_injection.py 14 passed, 1 warning in 34.19s uv run pytest tests/test_memory/test_skip_helpers.py tests/test_memory/test_embedder_mps_serialization.py::test_cpu_uses_shared_executor tests/test_memory/test_hierarchical.py::TestLocalEmbedder::test_embed_single tests/test_memory_bridge.py::TestMemoryBridgeImport::test_import_claude_code_memory tests/test_memory_handler_concurrent_init.py::test_real_localbackend_initializes_via_public_entrypoint tests/test_memory_system.py::TestLocalBackend::test_save_memory_basic -q 4 passed, 5 skipped, 11 warnings in 7.65s uv run ruff check . All checks passed! uv run ruff format . --check 968 files already formatted ``` ## Real Behavior Proof - Environment: local FastAPI `TestClient` for the Anthropic request path, plus Windows Python 3.12 offline-memory repros with `TRANSFORMERS_OFFLINE=1` - Exact command / steps: Run the focused CCR regression command and the offline-memory repro subset below on the merged branch state. - `uv run pytest tests/test_proxy/test_anthropic_ccr_deferred_injection.py` - `uv run pytest tests/test_memory/test_skip_helpers.py tests/test_memory/test_embedder_mps_serialization.py::test_cpu_uses_shared_executor tests/test_memory/test_hierarchical.py::TestLocalEmbedder::test_embed_single tests/test_memory_bridge.py::TestMemoryBridgeImport::test_import_claude_code_memory tests/test_memory_handler_concurrent_init.py::test_real_localbackend_initializes_via_public_entrypoint tests/test_memory_system.py::TestLocalBackend::test_save_memory_basic -q` - Scenario coverage from the CCR pytest command: - frozen prefix with deferred tool injection and cached marker text available in token mode - unfrozen turn with normal CCR marker emission - token mode where the tracked frozen prefix reclamps back to zero - frozen prefix where the client already supplied `headroom_retrieve` - cache-mode append-only delta reuse with a previously forwarded compressed prefix - cache-mode exact-prefix replay where the previous forwarded prefix already contained a marker - Scenario coverage from the offline-memory repro command: - direct local embedder startup on CPU with no cached HF model - hierarchical memory embedder startup with offline model cache missing - bridge import through `LocalBackend` - `MemoryHandler` public init warmup path - `LocalBackend` save path under the offline lane - Observed result: The CCR regression keeps marker-free forwarding on frozen turns without tool availability, and the merged-`main` offline-memory lanes now skip cleanly instead of failing unrelated CI shards. - frozen-prefix Anthropic turns without tool availability forward the original long transcript across both token-mode and cache-mode reuse paths, while unfrozen turns, reclamped token-mode turns, and frozen turns that already advertise `headroom_retrieve` keep the reversible CCR marker path - the merged-`main` offline-memory regressions now skip cleanly when the Hugging Face cache is unavailable instead of failing unrelated CI shards - Not tested: live Zed session cache-hit behavior, provider latency under real Anthropic upstreams, and online Hugging Face download lanes ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - Runtime scope is still intentionally narrow to Anthropic request-side CCR. OpenAI, Gemini, streaming, and response-side CCR behavior are unchanged. - The only non-CCR diff is the test-only offline-memory follow-up required after current `main` was merged into the branch. - The focused proxy pytest run still emits the Windows-local `StarletteDeprecationWarning` from `fastapi.testclient`'s `httpx` bridge. The offline-memory repro command also emits existing datetime deprecation warnings and a pytest teardown warning around skipped offline lanes; none of those warnings were introduced by the CCR runtime change. |
||
|
Zhenjia ZHOU
|
6c68ff4e9f
|
perf(compression): take large cold-start contexts off the synchronous kompress path (#1171) (#1298)
## Description On a cold-start large context, kompress (ModernBERT ONNX) runs **synchronously on the request thread** — ~200–300s for ~1M tokens. It blows the 30s compression budget, leaks a non-preemptible worker, and cascades (executor saturation → queue timeouts on healthy requests); on timeout the request is forwarded **uncompressed** after eating 30s. This adds four layered, **default-off, fail-open** mitigations so the request path is never blocked on ML compression. Closes #1171 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Phase 0 — size gate** (`HEADROOM_KOMPRESS_MAX_TOKENS`, default 50000): route oversized text away from ModernBERT (→ LogCompressor / TextCrusher / passthrough) at the single `_try_ml_compressor` boundary. - **Phase 1 — cooperative deadline** (`HEADROOM_COMPRESSION_DEADLINE_MS`, default 20000): any kompress run self-terminates at the next chunk boundary past the budget, keeping the unprocessed tail verbatim. - **Phase 2 — TextCrusher** (`HEADROOM_TEXT_CRUSHER`): a new **native Rust** extractive prose compressor in `crates/headroom-core/src/transforms/text_crusher/`, exposed via PyO3 as `headroom._core.TextCrusher` with a thin Python wrapper. It **reuses the shared `crate::relevance::BM25Scorer`** rather than reimplementing BM25, and ships record/replay parity fixtures (mirroring the SmartCrusher Rust-core + Python-shim pattern). - **Phase 3 — off-path compression** (`HEADROOM_BACKGROUND_COMPRESSION`): forward uncompressed immediately and compress in a per-process background drain; a byte-identical cache hit on a later turn means the request never blocks on ML. - Benchmark (`benchmarks/text_crusher_quality_eval.py`), CHANGELOG entry, and docstrings documenting the fail-open limits. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`, new modules) - [x] New tests added for new functionality - [ ] Manual testing performed (Phase 0/1 gate-fire + deadline observed on real traffic in earlier iterations; Phase 3 off-path is unit- + byte-identity-tested, not yet live-validated) ### Test Output ```text $ pytest tests/test_transforms/ tests/test_cache/ \ tests/test_proxy/test_background_compression.py tests/test_proxy/test_phase3_byte_identity.py -q 501 passed, 37 skipped in 40.33s $ cargo test -p headroom-core --lib text_crusher test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 834 filtered out $ ruff check <changed files> All checks passed! $ mypy headroom/proxy/background_compression.py headroom/transforms/text_crusher.py Success: no issues found in 2 source files ``` New coverage: size-gate incl. the strategy-dispatch funnel (KOMPRESS + TEXT); partial-run deadline (chunk-0 compressed + chunk-1 verbatim tail); BackgroundCompressor (dedup / queue-full / fail-open); Phase 3 byte-identity round-trip; TextCrusher unit + parity. ## Real Behavior Proof - Environment: macOS, local dev — `uv` venv, Rust `_core` built via `uv pip install -e .`. - Exact command / steps: the `pytest` / `cargo test` / `ruff` / `mypy` commands shown under Test Output; quality eval `python benchmarks/text_crusher_quality_eval.py /tmp/squad_dev.json`. - Observed result: 501 Python + 3 Rust tests pass; ruff + mypy clean on changed/new modules. Quality eval: TextCrusher keeps ~94% of buried SQuAD answers at 30% size vs ~36% truncate/random; self-contained speed run ~333k words in ~76ms (one O(n) pass) — sub-second where ModernBERT takes minutes (fast-vs-slow contrast, not a same-input run). - Not tested: Phase 3 off-path on live traffic; multi-worker (per-process by design — see Additional Notes). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - **All four features are off by default and fail-open** — with the env flags unset the paths are no-ops for realistic inputs; on any error the request is forwarded (compressed if possible, else verbatim), never dropped. A full background queue / duplicate key surfaces as `deferred:dropped`. - **Known limits (documented in `background_compression.py`):** Phase 3 is per-process, in-memory, and token-mode-only — these are **lost-savings, never lost-correctness**, and consistent with the project's existing per-process compression cache + sticky-session multi-worker model. The startup multi-worker warning now names off-path background compression. - Phase 2 reuses the existing BM25 scorer; reuse did not improve answer-retention over a Python prototype (query-awareness dominates) — its value is the Rust speed + repo-conventional Rust-core/Python-shim shape. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Tejas Chopra
|
bd55a426bc
|
fix(proxy): scope CORS to loopback + gate operator/content endpoints (#1226)
## Description Locks down the proxy's browser- and network-facing attack surface, which matters most under a `--host 0.0.0.0` bind (the Docker default). The wildcard CORS policy (`allow_origins=["*"]` + `allow_credentials=True`) let any web page the user had open read the proxy's content endpoints — `/v1/retrieve` returns raw, uncompressed tool outputs (source, secrets) — via a cross-origin fetch to `127.0.0.1` (CWE-346). Several operator endpoints additionally leaked sensitive data or allowed unauthenticated state mutation to any network-reachable client. This PR scopes CORS to loopback origins and extends the project's existing `require_loopback` trust boundary (already used for `/admin/*` and `/debug/*`) to the remaining exposed endpoints. Closes #863. Supersedes #864 and #758 — see "Additional Notes". ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **CORS**: replaced `allow_origins=["*"]` + `allow_credentials=True` with a port-agnostic loopback origin regex (`https?://(localhost|127\.0\.0\.1|\[::1\])(:\d+)?`), `allow_credentials=False`, and methods/headers narrowed to `GET/POST` + `Content-Type/Authorization`. `HEADROOM_CORS_ORIGINS` (comma-separated) pins an explicit allowlist for Docker/remote dashboards; `*` opts back into the old wildcard. - **`/transformations/feed`** and **`/cache/clear`** gated behind `require_loopback` → 404 for non-loopback callers. The feed returns full prompt/completion bodies when `log_full_messages` is on; `/cache/clear` is unauthenticated state mutation (cache-eviction DoS / cost amplification). - **`/health`**: the `config` block (upstream API URLs, savings profile) is now served only to loopback callers; network callers get the `/readyz`-shape body (status/checks). `/livez` and `/readyz` remain unauthenticated probes for orchestration. - **`/stats`**: `recent_requests` / `request_logs` (per-request ids, providers, models, errors) and `config` are served only to loopback callers; aggregate counters stay public for remote monitoring. - Added `_request_is_loopback()` helper mirroring `require_loopback`'s two-gate check (loopback peer IP + loopback `Host` header, the DNS-rebinding defence) but degrading the payload instead of returning 404, so monitors keep the non-sensitive fields. - Tests: new `tests/test_proxy_cors.py` and `tests/test_proxy_loopback_gating.py`; updated 4 existing tests that assert the now-loopback-only data to use loopback clients. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check headroom/proxy/server.py tests/test_proxy_cors.py \ tests/test_proxy_loopback_gating.py tests/test_proxy_healthchecks.py \ tests/test_proxy_stats_recent_requests.py tests/test_proxy/test_transformations_feed.py All checks passed! $ mypy headroom --ignore-missing-imports Success: no issues found in 380 source files $ pytest tests/test_proxy_cors.py tests/test_proxy_loopback_gating.py \ tests/test_proxy/test_transformations_feed.py tests/test_proxy_healthchecks.py \ tests/test_proxy_stats_recent_requests.py tests/test_proxy_dashboard_stats_cache.py \ tests/test_proxy_compression_executor.py tests/test_header_isolation.py -q ======================== 82 passed, 1 skipped in 17.75s ======================== ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0), Python 3.12 venv; FastAPI `TestClient` driving the real `create_app()` ASGI app - Exact command / steps: issued requests as a non-loopback caller (`client.host=testclient`) vs a loopback caller (`base_url=http://127.0.0.1`, `client=("127.0.0.1", 9999)`), plus CORS preflights with varying `Origin` headers - Observed result: CORS — `http://evil.com` → no `access-control-allow-origin`; `http://localhost:8787` and `http://localhost:9000` → echoed (loopback allowed on any port); `access-control-allow-credentials` → absent. `/cache/clear` and `/transformations/feed` → 404 (network) / 200 (loopback). `/health` `config` block present for loopback only. `/stats` `recent_requests` present for loopback only, while the aggregate `tokens` block stays present for network callers. - Not tested: a live `headroom proxy` process bound on `0.0.0.0` reached from a second host (simulated via ASGI peer/Host instead); end-to-end browser DNS-rebinding (covered by the `Host`-header gate and its unit test) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — proxy/middleware change; behavior is captured under "Real Behavior Proof". ## Additional Notes **Supersedes two stale PRs that target the same issue but have drifted from `main`:** - **#864** (`fix(proxy): scope CORS to localhost`, @gabiudrescu) — correct instinct and the source of the tighter `GET/POST` + `Content-Type/Authorization` scoping kept here, but it derived the allowlist from the `HEADROOM_PORT` env var (wrong when `--port` is passed as a CLI flag), carried ~40 lines of unrelated punctuation churn, and is ~125 commits behind `main`. The port-agnostic regex used here resolves the reviewer's port concern. - **#758** (`security: adversarial review`, @neogenix) — bundled these same application-layer fixes with a large CI/CD + Docker supply-chain pass. It is a ~160-commit-behind draft whose `server.py` no longer merges cleanly (`main` independently adopted the same `require_loopback` pattern). The application-layer fixes are rebased onto current `main` here; the CI/Docker/supply-chain hardening from #758 is still valuable and would be welcome as a separate, rebased PR. Thanks to @gabiudrescu and @neogenix for the original analysis (#863). **Deliberate scope / follow-ups (not in this PR):** - `/stats` aggregate counters and the basic `/health` body remain readable on a `0.0.0.0` bind by design, so remote monitoring keeps working. Full lock-down is a one-line `Depends(require_loopback)` each if preferred. - The `/v1/retrieve*` family stays network-reachable; it can't be loopback-gated without breaking legitimate remote/containerized agents and needs auth instead — tracked separately. - `ruff check .` is scoped to changed paths above because the dashboard HTML template trips ruff's `invalid-syntax` (a known repo false-positive); `mypy` is run over the full `headroom` package. |
||
|
Zhenjia ZHOU
|
e8fc8a0d18
|
feat(proxy): cc-switch reconciler — keep Headroom in the request path alongside cc-switch (#1030)
## Description [cc-switch](https://github.com/farion1231/cc-switch) is a desktop provider manager for Claude Code and other coding agents; when a Claude Code provider is selected, it writes that provider's endpoint and token into `~/.claude/settings.json`. This PR adds an opt-in reconciler so Headroom can stay in Claude Code's request path when cc-switch rewrites that file during provider switches. The reconciler captures third-party Anthropic-compatible upstream URLs, points Claude back at the local Headroom proxy, and leaves official/empty OAuth settings direct unless explicitly opted in. This update also hardens the watcher so rapid settings rewrites that share the same float-second mtime are still detected. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added an opt-in `HEADROOM_CC_SWITCH_RECONCILE=1` watcher for cc-switch direct-injection mode. - Added loopback-only `GET/PUT /admin/upstream` runtime upstream inspection and override endpoints. - Preserved token/model settings while rewriting only `env.ANTHROPIC_BASE_URL` back to the local Headroom proxy. - Switched reconciler change detection from float-second `st_mtime` to nanosecond `st_mtime_ns` so rapid provider switches are not missed. - Added pytest coverage for capture/rewrite behavior, official-provider defaults, route-official opt-in, loop safety, enabled flags, and the same-float-mtime provider-switch case. ## Testing - [ ] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text python -m pytest tests/test_proxy/test_cc_switch_reconciler.py 12 passed in 0.17s python -m ruff check . All checks passed! python -m mypy headroom Success: no issues found in 359 source files python -m pytest 7 failed, 5996 passed, 492 skipped, 5814 warnings in 396.41s ``` ## Real Behavior Proof - Environment: macOS, branch `feat/cc-switch-reconciler`, Python 3.13.3. - Exact command / steps: Ran the focused reconciler pytest file, full repository ruff check, full `mypy headroom`, and full pytest from the local PR branch. - Observed result: All 12 reconciler tests passed, including the rapid provider-switch case where two writes share the same float mtime but differ by nanoseconds. Full `ruff check .` and `mypy headroom` passed. Full pytest completed with 7 failures outside the cc-switch reconciler test file. - Not tested: Live cc-switch plus Claude Code end-to-end switch. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A. ## Additional Notes Documentation and CHANGELOG updates are not included in this PR. The reconciler remains opt-in and off by default. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Eyal Mizrachi
|
addebdb29c
|
feat(proxy): make COMPRESSION_TIMEOUT_SECONDS configurable via env (#946) (#991)
## Description The compression-pipeline timeout was hard-coded at 30s, so slow CPUs and long Claude Code conversations had no recourse. #946 asks to wire `HEADROOM_COMPRESSION_TIMEOUT_SECONDS` through. Refs #946. ## Type of Change - [x] New feature (non-breaking change that adds functionality) ## Changes Made - Read `HEADROOM_COMPRESSION_TIMEOUT_SECONDS` from the environment (float), falling back to 30 on an unparseable value. ## Testing - [x] Unit tests pass (`pytest`) - [x] New tests added for new functionality ### Test Output ```text $ pytest tests/test_proxy/test_compression_timeout_config.py -q 4 passed in 0.09s ``` ## Real Behavior Proof - Environment: Linux, Python 3.13, editable build of this branch - Exact command / steps: `HEADROOM_COMPRESSION_TIMEOUT_SECONDS=88 python -c "import headroom.proxy.helpers as h; print(h.COMPRESSION_TIMEOUT_SECONDS)"` - Observed result: prints `88.0` (default `30.0`; an unparseable value falls back to `30.0`) - Not tested: a live compression actually exceeding the configured timeout under real load ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
rongabbay
|
7edb27ab24
|
feat(proxy): compress AWS Bedrock InvokeModel requests via configurable upstream (#720)
## Description
Clients that speak **Bedrock to a local gateway** can't get proxy-level
compression. Claude Code launched with `CLAUDE_CODE_USE_BEDROCK=1` (and
any AWS SDK pointed at a custom endpoint) POSTs
`/model/{id}/invoke[-with-response-stream]` to
`AWS_ENDPOINT_URL_BEDROCK_RUNTIME`, never `/v1/messages`. Those requests
fell through the catch-all and were forwarded **verbatim — no
compression**.
`--backend bedrock` is the opposite direction: it accepts Anthropic
input and re-signs to AWS. It can't accept Bedrock-format input or
forward to a custom upstream. So the "client speaks Bedrock → local
re-signing gateway → AWS" topology (internal gateways, LiteLLM,
LocalStack; see #510) got nothing.
This adds a Bedrock InvokeModel passthrough that compresses the request
body with the **same** `anthropic_pipeline` used for `/v1/messages` —
the Bedrock InvokeModel body for Anthropic models *is* the Anthropic
Messages shape (`{anthropic_version, system, messages, max_tokens, …}`,
model in the URL), so there's no translation and no new compression
logic. The routes register **only** when `--bedrock-api-url` is set, so
default behavior is completely unchanged.
**Limitation (important):** rewriting the body invalidates the caller's
**SigV4** signature (it covers a hash of the body). Point
`--bedrock-api-url` at a gateway that re-signs or doesn't verify the
inbound signature (an internal gateway, LiteLLM, LocalStack, a corporate
Bedrock proxy) — **never raw AWS**, which would 403. For direct-to-AWS
compression, use `--backend bedrock` (which re-signs). The two are
complementary. This is documented in the flag help, the handler
docstring, the proxy docs, and the CHANGELOG.
Closes #734. Refs #510 (the Bedrock slice of the provider-agnostic
umbrella).
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- New `--bedrock-api-url` flag (env: `BEDROCK_TARGET_API_URL`). When
set, registers `POST /model/{id}/invoke` and `POST
/model/{id}/invoke-with-response-stream`.
- `BedrockHandlerMixin` compresses the request body via the existing
`anthropic_pipeline`, then forwards to the configured upstream,
preserving path/query.
- Responses forwarded byte-faithfully (non-streaming JSON and the
streaming AWS event-stream alike — neither is parsed or mutated, since
all compression is request-side).
- `{model_id:path}` captures inference-profile ids with
dots/colons/slashes (e.g.
`us.anthropic.claude-sonnet-4-5-20250929-v1:0`).
- Fail-open: a malformed body or compression error forwards verbatim
rather than erroring.
- Routes register only when the flag is set — default behavior
unchanged.
- Files: `headroom/proxy/handlers/bedrock.py` (new —
`BedrockHandlerMixin`); `headroom/providers/proxy_routes.py` (gated
route registration); `headroom/cli/proxy.py`,
`headroom/proxy/server.py`, `headroom/proxy/models.py` (flag + config
wiring); `docs/content/docs/proxy.mdx`, `CHANGELOG.md` (docs).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_proxy/test_bedrock_passthrough.py -q
.............. [100%]
14 passed in 12.46s
```
`tests/test_proxy/test_bedrock_passthrough.py` (14 tests) covers: route
gating (absent unless configured), body compression, non-message fields
preserved, inference-profile id capture + re-encoding, byte-faithful
streaming, fail-open on malformed body and on pipeline exceptions,
bypass when `optimize=False` and via the `x-headroom-bypass` header,
upstream connect failure surfacing as a 502, the content-length
regression, outcome recorded with `provider="bedrock"`, and
`BEDROCK_TARGET_API_URL` env wiring. `ruff check`/`format` clean.
## Real Behavior Proof
- Environment: macOS, Python 3.12; forked proxy on `:8788` with
`--bedrock-api-url` pointed at a local re-signing Bedrock gateway;
provider Anthropic Claude on Bedrock.
- Exact command / steps: `headroom proxy --port 8788 --bedrock-api-url
http://127.0.0.1:<gateway>`, then `curl -X POST
http://127.0.0.1:8788/model/claude-haiku-4-5/invoke --data
@bedrock_invoke.json` (a ~52k-token conversation with a large assistant
turn).
- Observed result: valid Claude response returned and the gateway
received the compressed body — proxy `/stats` reports `52,095 → 3,979
tokens` (92.4%, 48,116 removed), and the gateway's reported
`input_tokens: 3709` confirms the compressed body reached the model.
- Not tested: raw direct-to-AWS (out of scope by design — SigV4; use
`--backend bedrock`); non-Anthropic Bedrock model bodies (e.g.
Titan/Llama) — only the Anthropic Messages-shaped invoke body is
handled.
<details><summary>Proxy <code>/stats</code> output + content-length bug
note</summary>
```json
"compression": {
"requests_compressed": 1,
"avg_compression_pct": 92.4,
"best_detail": "52,095 → 3,979 tokens",
"total_tokens_removed": 48116
}
```
The first iteration of this proof surfaced a real bug — a shrunk body
still carried the inbound `Content-Length`, so httpx raised `Too little
data for declared Content-Length`. Fixed by dropping
`content-length`/`content-encoding` on the rewritten path so httpx
recomputes them; covered by a regression test.
</details>
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
`mypy headroom` is left unchecked above — type checking runs in the CI
matrix rather than locally on my side; the new code carries type hints
on all public functions. Design spec / feature request: #734.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
gglucass
|
2269e40bde
|
feat(proxy): log compressed messages alongside original request (#261)
## Description Expose the post-compression message list that was actually sent upstream as a new `compressed_messages` field on `RequestLog`, paired with the existing (now consistently pre-compression) `request_messages`. Consumers of `/transformations/feed` — dashboards and any downstream observability — can now diff the two sides of a compression to see exactly what the pipeline stripped, replaced, or kept. Turns an abstract "saved N tokens" into a legible before/after. Gated by the same `log_full_messages` flag as `request_messages` so the two sides stay in sync; it's pointless to store one without the other. Also fixes a latent correctness bug: today's `request_messages` field is inconsistent across the four `RequestLog` construction sites — sometimes it's the pre-compression snapshot, sometimes it's the mutated `body["messages"]` (which is the compressed list, because the proxy mutates `body` in place before the log call). After this change, `request_messages` always means pre-compression and `compressed_messages` always means what went upstream. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) Note on "breaking": strictly speaking this is a semantic correction of an inconsistently-populated field, not a schema break. The field name `request_messages` is unchanged and the JSON shape is unchanged; what changes is that the field now consistently holds the pre-compression list. Consumers that treated it as "whatever messages we have" continue to work. Consumers that depended on the accidental post-compression value (if any existed) would shift to `compressed_messages`. ## Changes Made - **`headroom/proxy/models.py`**: `RequestLog` gains `compressed_messages: list[dict] | None = None`. Doc comment explains it's paired with `request_messages` and gated by the same `log_full_messages` flag. - **`headroom/proxy/handlers/anthropic.py`** (2 sites — Bedrock non-streaming and main non-streaming): `request_messages` now consistently sources from `original_messages` (the pre-compression snapshot at line 724), `compressed_messages` sources from `body["messages"]` (the compressed list after in-place mutation at line 1189). Both gated symmetrically. - **`headroom/proxy/handlers/streaming.py`** (2 sites — main streaming in `_finalize_stream_response`, Bedrock streaming in `_stream_response_bedrock`): same treatment. `_stream_response_bedrock` gains a new `original_messages: list[dict] | None = None` parameter so it has access to the pre-compression snapshot; the sole caller in `anthropic.py` now threads it through. - **`headroom/proxy/server.py`**: `/transformations/feed` adds `compressed_messages` to the JSON payload alongside the existing `request_messages` / `response_content`. *Split into a separate preceding commit is a one-time EOL normalization to LF — the file blob in history carries CRLF but `.gitattributes` declares `*.py text eol=lf`, so any contributor editing `server.py` triggers the same whole-file renormalization. Separating the two commits keeps this feature commit's diff at a single line.* - **`headroom/proxy/request_logger.py`**: `compressed_messages` is stripped from the JSONL file log and from `get_recent()` alongside the existing `request_messages` / `response_content` stripping. `get_memory_stats()` also counts it toward the deque's byte budget. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .` and `ruff format --check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed (via the Headroom Desktop client that consumes `/transformations/feed` — confirmed both fields arrive and render) Test coverage added/extended: - `tests/test_proxy/test_request_logger.py` (new file): round-trip unit tests for `RequestLogger`. Confirms `get_recent` strips both sides (pre + post), `get_recent_with_messages` exposes both, and the JSONL file log drops both when `log_full_messages=False`. - `tests/test_proxy/test_transformations_feed.py`: extended to assert `compressed_messages` appears in the endpoint payload alongside `request_messages` / `response_content`. - `tests/test_proxy_streaming_request_logger.py`: existing include/omit tests updated to assert both sides populate when the flag is on and both are `None` when it's off. ## Test Output ``` $ uv run ruff check headroom tests All checks passed! $ uv run ruff format --check headroom tests 614 files already formatted $ uv run pytest tests/test_proxy/test_request_logger.py tests/test_proxy_streaming_request_logger.py tests/test_proxy/test_transformations_feed.py -v ... tests/test_proxy/test_request_logger.py::test_get_recent_strips_compressed_messages_alongside_request_and_response PASSED tests/test_proxy/test_request_logger.py::test_get_recent_with_messages_returns_compressed_messages PASSED tests/test_proxy/test_request_logger.py::test_jsonl_file_strips_both_sides_when_log_full_messages_disabled PASSED tests/test_proxy_streaming_request_logger.py::test_finalize_stream_response_logs_original_and_compressed_messages PASSED tests/test_proxy_streaming_request_logger.py::test_finalize_stream_response_omits_messages_when_log_full_messages_disabled PASSED tests/test_proxy/test_transformations_feed.py::test_transformations_feed_returns_messages PASSED ... ============================== 11 passed in 5.36s ============================== ``` ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas (the two-sided gating at each log site, the `_stream_response_bedrock` parameter addition, and the `get_memory_stats` accounting) - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes ### Non-Anthropic backends `handlers/openai.py` and `handlers/gemini.py` do not currently emit `RequestLog` entries at all — only Anthropic and the shared streaming paths do. This PR therefore only populates `compressed_messages` on Anthropic traffic (which is what `/transformations/feed` shows today). Wiring OpenAI and Gemini into `RequestLogger` end-to-end is a separate, larger gap worth its own PR. ### `server.py` EOL normalization The feature change in `server.py` is a single line. To keep the diff readable, the preceding commit is a whitespace-only `chore(proxy): normalize server.py to LF per .gitattributes` — the file blob was stored with CRLF terminators but `.gitattributes` declares `*.py text eol=lf`. Any contributor touching `server.py` triggers this renormalization; isolating it here keeps the feature commit reviewable. Happy to rebase / drop / reshape as preferred. ### Downstream desktop compatibility The Headroom Desktop client I work on now consumes `compressed_messages` and renders the pre/post pair side-by-side on the "Recent large compression" card. The desktop was updated to handle both shapes: proxies without the field render the legacy single "Request" block; proxies with the field render "Request (original, N tokens)" + "Request (compressed, M tokens)" where N/M come from `input_tokens_original` / `input_tokens_optimized`. No changes needed downstream if this PR lands. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
gglucass
|
8f374263d3
|
feat(transforms): attribute read_lifecycle + smart_crush tags (#249)
## What Enrich the `transforms_applied` tags emitted by `ReadLifecycleManager` and `SmartCrusher` so each tag carries the specific target it acted on, instead of being an opaque counter: - `read_lifecycle:<state>` -> `read_lifecycle:<state>:<file_path>` - `smart_crush:<n>` -> `smart_crush:<n>:<tool1,tool2,...>` (tool names resolved from the assistant's `tool_calls` / `tool_use` metadata; falls back to `smart_crush:<n>` when no name resolves) Downstream UIs can then show *what* a compression acted on (which file was a stale read, which tools had their output crushed), not just that it happened. ## Note on the rebase The original revision targeted `ToolCrusher` / `tool_crush:<n>`. That transform has since been retired and replaced by the Rust-backed `SmartCrusher` (which emits `smart_crush:<n>`), so the tool-name attribution moved to `smart_crusher.py`. The `read_lifecycle` half is unchanged. ## Response-header compatibility `x-headroom-transforms` is built as `",".join(transforms_applied)`. A tag containing a comma (tool-name lists; file paths) would make that header ambiguous to split back into tags. To keep the header backward compatible, `header_safe_transforms` (`headroom/proxy/cost.py`) collapses the enriched tags back to their legacy counter shape **for the header only** -- the full enriched detail still flows through the structured `transforms_applied` list (dashboards, request logs, activity feed). Applied at all three header sites (openai / anthropic / gemini handlers). Paths containing `:` survive in `transforms_applied` because consumers bound their split to 3 parts. ## Tests - `tests/test_transforms/test_read_lifecycle.py` -- OpenAI + Anthropic tag shape, colon-in-path preservation - `tests/test_transforms/test_smart_crusher_attribution.py` -- OpenAI + Anthropic tool-name resolution, dedup, no-name fallback, id/name-missing skips - `tests/test_proxy/test_header_safe_transforms.py` -- header normalization keeps the joined header unambiguous (incl. comma-in-path) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Ashish
|
8db5efc6f9
|
fix(anthropic): CCR exception must re-raise, not silently swallow (#838)
## Summary
- When `ccr_response_handler.handle_response` throws on the Anthropic
path, the old code logged a `WARNING` and continued — silently returning
the raw `headroom_retrieve` tool-call block to the client (compressed
content never retrieved, client sees an unknown internal tool)
- The OpenAI handler already does `logger.error + raise` (commit
`
|
||
|
ashishpatel26
|
e8ecd08829 | fix(codex): fail open for proxy compression timeout |