mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
40 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
nangsontay
|
08fce29b47
|
fix(proxy): stop toggling headroom_retrieve in the Anthropic tools array (#2672)
## Description `should_inject_ccr_tool` deferred CCR tool injection whenever `frozen_message_count > 0`. Because `tools` is the head of Anthropic's cache key, that dropped a tool which was already inside the provider-cached prefix and invalidated the whole prefix — in both directions (`0 → >0` removes it; `>0 → 0` on proxy restart, `/model` switch, lineage eviction or TTL lapse adds it back). On three days of local proxy logs the turns that flipped injection state carried **44.7% of all cache-write tokens at a 52.0% hit rate**, against 98.1% for non-flipping turns. The log signature is `cache_read` alternating between two values exactly 172 tokens apart — the 464-byte tool definition. This deletes the gate and calls `apply_session_sticky_ccr_tool` directly, which is **what `openai.py` already does** — the two handlers now have the same shape. Net −61 production lines, no new state, no new config flag. Fixes defect 1 of #2671. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Performance improvement ## Changes Made - `headroom/proxy/ccr_marker_policy.py` — deleted the `should_inject_ccr_tool` gate; `apply_session_sticky_ccr_tool` is now the single decision point. - `headroom/proxy/handlers/anthropic.py` — calls `apply_session_sticky_ccr_tool` directly, matching `openai.py`. - `headroom/proxy/helpers.py` — dropped the now-unused gate plumbing. - `tests/test_proxy_anthropic_cache_stability.py` — new test asserting the forwarded `tools` array is byte-identical across a `frozen 0 → >0` transition. - `tests/test_ccr_marker_policy.py` — removed the three unit tests that pinned the deleted decision (they encoded the defect). - `tests/test_proxy/test_ccr_frozen_prefix_coupling.py` — same unredeemable-marker intent, re-pinned at the sticky helper. - `tests/test_proxy/test_anthropic_ccr_deferred_injection.py` — autouse reset fixture for the process-global `SessionCcrTracker` (separate commit). - Formatting-only follow-up commit applying `ruff format` (pinned 0.15.17) to the two test files above. ### Why deleting the gate is safe `apply_session_sticky_ccr_tool` already holds the correct rule. Its four branches, in order: | # | condition | action | |---|---|---| | 1 | tool already in the incoming tool list (client/MCP pre-registered) | skip; the client's bytes win | | 2 | `session_id is None` (WS / pre-session) | per-turn flag drives it verbatim | | 3 | session has done CCR | always inject the recorded golden bytes | | 4 | fresh session, no compression this turn | **skip** | Branch 4 is the safety property: a session that has never compressed still gets no tool, so removing the gate cannot start injecting into non-CCR conversations. Branch 3 is what the gate was starving. `has_new_ccr_markers` still gates first-time injection, so markers replayed from the previously-forwarded prefix cannot trigger one. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed The three deleted unit tests encoded the defect. Coverage moves to the property that actually matters and was previously untested: **the forwarded `tools` array must be byte-identical across a `frozen 0 → >0` transition.** That test asserts on the forwarded request body rather than on a policy function's return value; unit-testing the old policy in isolation is exactly what let a wrong-but-self-consistent decision pass. Verified failing on `upstream/main` with an assertion on the missing tool (not an `ImportError`, so it fails for the right reason). Full suite: same pre-existing unrelated failures as `upstream/main`, **zero new** (verified by running the whole suite on both revisions and diffing the failure sets). ### Test Output ```text $ uv run pytest tests/test_ccr_marker_policy.py \ tests/test_proxy/test_anthropic_ccr_deferred_injection.py \ tests/test_proxy/test_ccr_frozen_prefix_coupling.py \ tests/test_proxy_anthropic_cache_stability.py -q collected 48 items tests/test_ccr_marker_policy.py ..... [ 10%] tests/test_proxy/test_anthropic_ccr_deferred_injection.py .............. [ 39%] . [ 41%] tests/test_proxy/test_ccr_frozen_prefix_coupling.py .. [ 45%] tests/test_proxy_anthropic_cache_stability.py .......................... [100%] ======================= 48 passed, 2 warnings in 13.59s ======================== $ ruff check . All checks passed! $ ruff format --check . 1349 files already formatted ``` ## Real Behavior Proof - Environment: local macOS proxy serving live Claude Code traffic to the Anthropic API; baseline = 3 days of proxy logs on `upstream/main`, after = 5.5 hours with this change live. - Exact command / steps: ran the proxy with this branch built in, drove normal Claude Code sessions through it (including `/model` switches and proxy restarts, the two events that used to flip injection state), then parsed 235 real turns from the proxy logs with the same parser used for the baseline in #2671. - Observed result: flip turns fell from 177 (44.7% of all cache write) to 2 (1.7%); steady-state write share 1.192% → 0.867%; aggregate hit rate 86.75% → 89.21%; main conversation warm hit rate 98.1% → 97.70% (n=149). The 2 remaining "flips" have `cache_read == 0` — cold starts that the bucketing counts as a state change, not real flips. | metric | baseline | after | |---|---|---| | flip turns | 177, carrying 44.7% of all cache write | **2**, carrying **1.7%** | | main conv, warm | 98.1% | **97.70%** (n=149) | | steady-state write share | 1.192% | **0.867%** | | aggregate | 86.75% | **89.21%** | - Not tested: `mypy headroom` was not run locally for this body; the OpenAI handler path (unchanged by this PR); tracker state loss mid-session (see note below); and defect 2 of #2671 (the sub-call breakpoint), which is untouched and is now 54.9% of remaining cache write — that is why aggregate stays just under 90%. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes **Pre-existing and unchanged here:** if the tracker loses state mid-session while the transcript still carries markers, branch 4 returns no tool and those markers are unredeemable. `upstream/main` has no recovery for that; this PR neither creates nor fixes it. See my comment on #2500, which adds a recovery path for the related dangling-reference case. **N/A checklist items:** no documentation changes — this removes an internal policy function with no user-facing surface. `mypy headroom` left unchecked because it was not run for this body; CI covers it. **Merge-order conflict with #2500 (please read before landing either):** this PR *deletes* `should_inject_ccr_tool`, which is the exact function #2500 extends with `transcript_requires_tool`. Whichever lands second needs a semantic rebase, not just a textual one — git will not flag it. If this PR lands first, #2500's recovery path should re-target `apply_session_sticky_ccr_tool` (the sticky helper now owns the decision alone) or the handler call site in `handlers/anthropic.py`. If #2500 lands first, the gate deletion here still applies but the `transcript_requires_tool` override needs to move with it. Happy to do the rebase either way — say which order you prefer. |
||
|
Rod Boev
|
a2e42fb877
|
fix(proxy): keep buffered CCR streams alive (#2479)
## Description Buffered CCR streaming currently waits for the full upstream response before sending any bytes back to the client. On the Anthropic path this shows up as `API Error: Stream idle timeout - no chunks received`, and the same buffer-then-synthesize mechanism still exists on the `/v1/responses` CCR path. This adds a narrow buffered-stream heartbeat layer so the client sees early stream activity while Headroom preserves the existing server-side retrieval round trip and final synthesized provider events. Closes #2465 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - open buffered CCR streams early and emit client-visible `event: ping` heartbeats while the buffered upstream call is still in flight - preserve the existing terminal Anthropic and Responses synthesis helpers instead of replacing their event-building logic - preserve early non-streaming failure semantics before the first heartbeat, including normal 429 passthrough and normal JSON 502 failures - log late buffered-task exceptions server-side and record one failed provider metric on that post-keepalive branch, while keeping the client-facing SSE error sanitized - add focused delayed-upstream regression coverage for both buffered provider paths, their early-failure branches, and their late-failure branches ## Testing - [x] Unit tests pass (`uv run pytest tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py tests/test_proxy/test_openai_responses_ccr.py -q`) - [x] Linting passes (`uv run ruff check headroom/proxy/handlers/anthropic.py headroom/proxy/handlers/openai.py tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py tests/test_proxy/test_openai_responses_ccr.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text uv run pytest tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py tests/test_proxy/test_openai_responses_ccr.py -q ======================= 17 passed, 1 warning in 42.47s ======================== uv run ruff check headroom/proxy/handlers/anthropic.py headroom/proxy/handlers/openai.py tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py tests/test_proxy/test_openai_responses_ccr.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows, Python via `uv`, proxy handler tests with gated buffered upstream fixtures - Exact command / steps: run the focused Anthropic and Responses CCR suites above; delayed-upstream tests consume the first client-visible SSE event before releasing the upstream, then consume the synthesized final events - Observed result: both buffered paths emitted `event: ping` before upstream release; pre-keepalive 429 responses preserved their real status and headers, pre-keepalive exceptions returned the normal JSON 502 shape, late transport failures recorded one failed provider metric and one server error log before emitting one sanitized SSE error, Anthropic preserved `done`, and Responses preserved `Resolved!` - Not tested: live slow upstream run with Claude Code or a real Responses client ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes `CHANGELOG.md` stays untouched because Headroom generates release notes from conventional commits. The PR should only claim the local buffered-stream contract and focused regression coverage; live client proof remains an owner check on a slow real upstream. |
||
|
Abhay Singh
|
313c290df9
|
fix(proxy/openai): None-guard usage token counts on the chat path (#2431)
## Description
`handle_openai_chat` reads token counts from the response usage to
record metrics and update the prefix tracker:
```python
usage = backend_response.body.get("usage", {})
output_tokens = usage.get("completion_tokens", 0)
total_input_tokens = usage.get("prompt_tokens", optimized_tokens)
```
`.get(key, default)` only falls back when the key is **absent**. When an
OpenAI-compatible backend emits a key with a **null** value (providers
do this on a stopped or empty turn, the same shape that caused the
Gemini crash in #2347), `.get` returns `None`. That `None` then flows
into:
- `_infer_openai_cache_write_tokens(total_input_tokens,
cache_read_tokens)` → `max(input_tokens - cache_read_tokens, 0)` (a
`None - int` → `TypeError`),
- `uncached_input_tokens = max(0, total_input_tokens - cache_read_tokens
- cache_write_tokens)`, and
- `RequestOutcome(output_tokens=..., optimized_tokens=...)`, whose
fields are `int` and which the metrics recorder increments.
Both chat usage-extraction sites are affected. On the direct-provider
branch the arithmetic runs **outside** the surrounding `try`, so a
single such response raises an uncaught `TypeError` and 500s the
request; on the backend branch it corrupts outcome recording.
## Fix
Coerce the three counts with the existing module-level `_usage_int`
guard (`max(int(value), 0)`, 0 on failure) at both sites, matching the
streaming path, the already-guarded cache keys in the same block
(`usage.get("cache_read_input_tokens", 0) or 0`), and the Gemini fix in
#2347. A normal integer usage is unchanged; only a null (or absent)
value now becomes the fallback/0. `prompt_tokens` keeps its
`optimized_tokens` fallback so our own input estimate is used when the
count is missing.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/handlers/openai.py`: `_usage_int`-guard
`completion_tokens` / `prompt_tokens` / `cached_tokens` at both
non-streaming usage-extraction sites in `handle_openai_chat`.
- `tests/test_proxy/test_openai_chat_savings_profile.py`: regression
driving a `/v1/chat/completions` request whose backend usage reports
null `prompt_tokens` / `completion_tokens`, asserting a 200 instead of a
crash.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_proxy/test_openai_chat_savings_profile.py -q
2 passed
# with the fix reverted, the new test fails (the null-usage response 500s):
$ git stash push -- headroom/proxy/handlers/openai.py
$ python -m pytest tests/test_proxy/test_openai_chat_savings_profile.py::test_chat_completions_survives_null_usage_token_counts -q
1 failed
$ uvx ruff@0.15.17 check headroom/proxy/handlers/openai.py tests/test_proxy/test_openai_chat_savings_profile.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/proxy/handlers/openai.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, project venv (`uv sync --extra
proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`, pytest in the venv.
- Exact command / steps: ran the new FastAPI `TestClient` regression,
which drives the real `handle_openai_chat` through a mock backend
returning `usage: {prompt_tokens: null, completion_tokens: null,
total_tokens: null}`; then reverted only `openai.py` and re-ran the same
test.
- Observed result: with the fix the request returns 200; with the fix
reverted the same request fails (the null count reaches the `max(...)`
arithmetic and outcome recording). Ran against the actual handler via
the app.
- Not tested: a live third-party OpenAI-compatible gateway emitting null
usage.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
|
||
|
Rod Boev
|
d6a1af40d5
|
fix(proxy): skip max_tokens rename for backend-routed openai chat (#2401)
## Description OpenAI-format `POST /v1/chat/completions` requests routed through `--backend litellm-vertex` fail when the client includes `max_tokens`. The proxy currently runs its direct-OpenAI compatibility shim before backend dispatch, renames `max_tokens` to `max_completion_tokens`, then the LiteLLM path no longer recognizes that field as standard and sweeps it into `extra_body`. Vertex rejects the resulting request with `extra_body: Extra inputs are not permitted`. This change scopes the rename shim to the direct OpenAI path only. Backend-routed chat requests now keep `max_tokens`, which LiteLLM already forwards correctly for the Vertex Anthropic path. Direct GPT-5 and o-series compatibility stays unchanged. Closes #2392. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Thread a backend-owned translation flag into `_normalize_openai_max_tokens`. - Skip the legacy-to-completion-token rename on backend-routed OpenAI chat requests. - Keep the direct OpenAI compatibility path covered with a backend-owned translation no-op test. - Add buffered and streaming handler-level regressions for the exact `litellm-vertex` request shape, proving the request survives the `/v1/chat/completions` normalization boundary with vendor fields intact. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py -q`) - [x] Linting passes (`uv run ruff check headroom/proxy/handlers/openai.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py -q ......sss............ [100%] 20 passed, 3 skipped, 1 warning in 42.13s $ uv run ruff check headroom/proxy/handlers/openai.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py All checks passed! $ uv run ruff format headroom/proxy/handlers/openai.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py --check 5 files already formatted ``` ## Real Behavior Proof - Environment: Windows, synced Headroom development environment, mocked LiteLLM provider boundary, no paid GCP credentials required - Exact command / steps: run `uv run pytest tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py -q`, using the issue payload shape `{"model":"claude-sonnet-4-6","max_tokens":32,"messages":[{"role":"user","content":"hi"}],"chat_template_kwargs":{"enable_thinking":false}}` through `POST /v1/chat/completions` - Observed result: buffered and streaming `litellm-vertex` requests keep `max_tokens` as a named backend kwarg, preserve `chat_template_kwargs` in `extra_body`, omit `max_completion_tokens` from `extra_body`, and return success through the handler boundary. Direct-path normalization still renames legacy `max_tokens`. - Not tested: live Vertex AI request ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [ ] My code follows the project's style guidelines - [ ] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - `CHANGELOG.md`: N/A, the release pipeline generates it from the conventional-commit subject. - Scope is intentionally narrow: this fixes the exact backend-routed `max_tokens` failure and does not broaden `extra_body` hardening for unrelated OpenAI fields. |
||
|
Abhay Singh
|
f64aac9733
|
fix(proxy/gemini): None-guard token counts from usageMetadata (#2347)
## Description
The non-streaming Gemini/Vertex handler reads token counts straight from
the response's `usageMetadata`:
```python
try:
usage = resp_json.get("usageMetadata", {})
total_input_tokens = usage.get("promptTokenCount", optimized_tokens)
output_tokens = usage.get("candidatesTokenCount", 0)
cache_read_tokens = usage.get("cachedContentTokenCount", 0)
except (...):
...
uncached_input_tokens = max(0, total_input_tokens - cache_read_tokens) # OUTSIDE the try
```
`.get(key, default)` only falls back when the key is **absent**. When
`usageMetadata` carries a key with a **null** value — which Gemini can
do on a safety-blocked turn that produced no candidates — `.get` returns
`None`. That `None` then reaches:
- `max(0, total_input_tokens - cache_read_tokens)` (a `None - int` →
`TypeError`), and
- `RequestOutcome(output_tokens=...)`, whose field is `int` and which
the metrics recorder increments (`tokens_output_total += output_tokens`
→ `TypeError`).
Both run on the success (non-`except`) path, so a single such response
crashes the request and its outcome recording. The Gemini streaming path
already guards these with a `_usage_int` helper; the non-streaming path
(two sites) did not.
## Fix
Coerce the three counts with `int(... or fallback)`, matching the
streaming `_usage_int` guard and the LiteLLM usage mappings:
```python
total_input_tokens = int(usage.get("promptTokenCount", optimized_tokens) or optimized_tokens)
output_tokens = int(usage.get("candidatesTokenCount", 0) or 0)
cache_read_tokens = int(usage.get("cachedContentTokenCount", 0) or 0)
```
No change for a normal integer usage; only a `None` (or absent) value
now becomes the fallback/0. Applied to both non-streaming
usage-extraction sites in `handlers/gemini.py`.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/handlers/gemini.py`: `int(... or fallback)`-guard
`promptTokenCount` / `candidatesTokenCount` / `cachedContentTokenCount`
at both non-streaming usage sites.
- `tests/test_proxy/test_gemini_savings_profile.py`: add a regression
driving a `generateContent` request whose
`usageMetadata.candidatesTokenCount` is `null`, asserting a 200, an
`int` `output_tokens == 0`, and `uncached_input_tokens == 20` (the
`max(0, …)` no longer raises).
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/proxy/handlers/gemini.py tests/test_proxy/test_gemini_savings_profile.py
All checks passed!
$ uvx ruff@0.15.17 format --check headroom/proxy/handlers/gemini.py tests/test_proxy/test_gemini_savings_profile.py
2 files already formatted
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/proxy/handlers/gemini.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17` / `uvx
mypy@1.20.2`. A full `pytest` OOMs this box (ML-stack import), so I
reproduced the extraction with a dependency-free script and left the
full pytest to CI.
- Exact command / steps: modeled the OLD (bare `.get`) and NEW (`int(...
or fallback)`) derivations for a blocked response
(`candidatesTokenCount: null`, valid prompt count), a null
`promptTokenCount`, a normal response, and an absent-usage response.
- Observed result: OLD raised `TypeError` at `max(0, None - …)` for a
null prompt count and left `output_tokens = None` (which crashes the
int-typed outcome/metrics recorder) for a null candidate count; NEW
produced `(20, 0)` for the blocked case, `(15, 0)` for the null-prompt
case (the `optimized_tokens` fallback), `(60, 30)` for a normal
response, and the fallbacks for absent usage. The added
`create_app`/`TestClient` test drives the handler end to end and asserts
a 200 with `int` outcome counts.
- Not tested: a live Gemini safety-blocked response; the added test uses
a mocked `_retry_request` returning a `usageMetadata` with a null count,
matching the existing Gemini test harness in this file.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" box is unchecked because a local pytest
run imports the ML stack and OOMs this box; the added test reuses the
existing `create_app`/`TestClient` + mocked-`_retry_request` harness in
`test_gemini_savings_profile.py` and runs under the normal CI pytest
job, and the behavior is corroborated by the standalone proof above.
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
Rod Boev
|
26b43f64d6
|
fix(proxy): keep anthropic ccr compression active across deferred injection (#2291) (#2297)
## Description Large native Claude Code requests on the Anthropic path can still forward with zero request compression after CCR tool injection is deferred on a frozen prefix. The stale skip branch treats deferred injection as a reason to bypass request compression entirely, even though the later sticky CCR path already knows when new markers actually require the tool. This removes that stale bypass so compression still runs while the reversible CCR path stays intact. Closes #2291. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Remove the stale `should_skip_ccr_request_compression` branch from `headroom/proxy/handlers/anthropic.py`, so deferred CCR tool injection no longer bypasses request compression in token, non-cache, or cache mode. - Keep the existing sticky CCR injection path as the only place that decides whether historical markers need the retrieval tool reintroduced. - Update `tests/test_proxy/test_anthropic_ccr_deferred_injection.py` to cover the two broken zero-compression cases and preserve the already-reversible frozen-prefix path. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_proxy/test_anthropic_ccr_deferred_injection.py::test_cache_mode_compresses_delta_but_replays_cached_prefix_when_markers_are_historical tests/test_proxy/test_anthropic_ccr_deferred_injection.py::test_non_token_non_cache_mode_keeps_compression_and_injects_tool_for_new_markers tests/test_proxy/test_anthropic_ccr_deferred_injection.py::test_existing_retrieve_tool_keeps_reversible_ccr_path_when_prefix_is_frozen tests/test_openai_tool_search_deferral.py tests/test_openai_responses_compression_units.py -q`) - [x] Linting passes (`uv run ruff check .`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text ======================= 53 passed, 1 warning in 12.71s ======================== All checks passed! 1310 files already formatted ``` ## Real Behavior Proof - Environment: synced branch and base worktrees on a local Windows proxy test host - Exact command / steps: run the updated Anthropic deferred-injection regressions directly against the base package tree and the branch package tree, then run the focused branch pytest suite above - Observed result: the base package tree fails the two updated zero-compression regressions (`FAIL test_cache_mode_compresses_delta_but_replays_cached_prefix_when_markers_are_historical`, `FAIL test_non_token_non_cache_mode_keeps_compression_and_injects_tool_for_new_markers`) while preserving the already-reversible path; the branch package tree prints three `PASS` lines for the same trio and keeps the neighboring OpenAI suites green in the 53-test focused run - Not tested: a live upstream Claude Code request with the reporter's exact provider/model credentials ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes `CHANGELOG.md` is not applicable here because Headroom's release pipeline derives it from conventional commits. Scope is limited to the Anthropic CCR request-compression seam that current #2291 evidence exercises. OpenAI tool-search deferral is untouched because the current live issue is a native Claude Code path and the concrete stale skip branch on `origin/main` is in `anthropic.py`. Co-authored-by: Tejas Chopra <chopratejas@gmail.com> |
||
|
Krishna Chaitanya
|
57e8dcb425
|
feat(proxy): add opt-in cost-aware model router (#1706) (#2205)
## Description Adds an optional, configuration driven model router (closes #1706). With `HEADROOM_MODEL_ROUTER_ENABLED` set, ordered rules in `HEADROOM_MODEL_ROUTES` rewrite the upstream model by estimated input size and tool presence, complementary to content compression, for example sending small, tool-free requests to a cheaper model. First matching rule wins and every decision is logged with a reason. Off by default so behavior is unchanged, skipped under `x-headroom-bypass`/passthrough, and wired on the Anthropic `/v1/messages` path. Malformed rules fail open, so a bad rule is skipped rather than silently widened. Closes #1706 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/model_router.py`: new `ModelRouter` component (ordered rules, first-match decision with reason, fail-open env parsing, tokenizer-free input estimate). - `headroom/proxy/models.py` + `headroom/proxy/server.py`: `ProxyConfig.model_router` field, env loader (`HEADROOM_MODEL_ROUTER_ENABLED` / `HEADROOM_MODEL_ROUTES`), and proxy wiring. - `headroom/proxy/handlers/anthropic.py`: apply routing on `/v1/messages` after the bypass gate, tracked as a body mutation. - Tests, docs (`configuration.mdx`), and a CHANGELOG entry. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest -q tests/test_proxy/test_model_router.py tests/test_proxy/test_model_router_wiring.py 36 passed, 1 warning $ ruff check . All checks passed! $ mypy headroom --ignore-missing-imports Success: no issues found in 477 source files ``` ## Real Behavior Proof - Environment: local, macOS, Python 3.12, headroom `.venv`, upstream mocked (no live provider call). - Exact command / steps: enable the router via `ProxyConfig(model_router=...)`, POST `/v1/messages` through `TestClient` with a rule routing low-risk requests to a cheaper model; repeat with header `x-headroom-bypass: true`. - Observed result: the forwarded upstream body model is rewritten from `claude-sonnet-4-6` to `claude-haiku-4-5` when the router is enabled, and is left unchanged under bypass (see `tests/test_proxy/test_model_router_wiring.py`). - Not tested: the OpenAI and Gemini handler paths (this PR wires the Anthropic path only); no live provider request (upstream is mocked). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes Happy to adjust the interface or scope (for example OpenAI and Gemini parity) if you'd prefer a different shape. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: reneleonhardt <65483435+reneleonhardt@users.noreply.github.com> |
||
|
nangsontay
|
e5b3a634df
|
fix(proxy): dedupe Codex WS request logging for accurate mixed-provider dashboards (#2189)
## Description Running Claude Code (Anthropic) and Codex (OpenAI) against the **same** Headroom proxy instance on one port produced incorrect, unstable dashboard data. The proxy core is provider-isolated and multi-provider-safe by design; the defect was in the observability layer. The Codex `/v1/responses` **WebSocket** handler was the only path in the proxy that wrote to the request logger by hand instead of through the unified `emit_request_outcome` funnel, and it did so twice per session close: the per-turn funnel record plus an unconditional cumulative session-summary `RequestLog`. This PR removes the duplicate summary log so Codex WS emits exactly one request log per turn, matching the HTTP provider paths. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Dropped the duplicate cumulative session-summary `RequestLog` in the Codex WS handler while preserving the per-turn `emit_request_outcome` path. - Preserved gated `request_messages` and `turn_id` on residual outcomes so dashboard telemetry keeps the useful attribution without double-counting tokens. - Ensured explicit `--anyllm-provider` wins over a leaked `HEADROOM_ANYLLM_PROVIDER` environment variable. - Registered retry delay settings that had drifted out of the settings registry. - Hardened tests against developer-shell `HEADROOM_*` / `ANTHROPIC_CUSTOM_HEADERS` leakage and stabilized several focused proxy/wrap test fixtures. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ .venv/bin/pytest tests/ -q -p no:cacheprovider 8529 passed, 537 skipped, 5831 warnings in 276.25s (0:04:36) $ .venv/bin/ruff check <touched files> All checks passed! ``` ## Real Behavior Proof - Environment: macOS (Darwin 25.4.0), Python 3.13.14, pytest 9.0.3, ruff via project venv, branch `fix/multi-provider-runtime`. - Exact command / steps: Ran the full test suite without pytest cache provider and Ruff on all touched files; used `git stash` to confirm the stale fake-config failures pre-existed this change. - Observed result: Full suite passed with no failures; Ruff passed; Codex WS now routes end-of-session logging through `emit_request_outcome`, emitting one request log per turn with the same accounting model as Anthropic HTTP turns. - Not tested: Live simultaneous Claude + Codex dashboard run. `mypy headroom` was not run to completion; a scoped run reported one pre-existing `settings_store.py:470` coercion error outside this diff. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A - server-side observability fix; no UI markup changed. ## Additional Notes - The proxy's multi-provider routing, header/auth isolation, and per-model cache keying are already correct and unchanged here; only the WS observability write path was double-counting. - Architectural assessment: `plans/reports/research-260714-0004-multi-provider-upstream-compression-report.md`; root-cause + resolution trail: `plans/reports/debug-assessment-260714-0011-dashboard-instability-mixed-claude-codex-report.md`. - No live simultaneous Claude + Codex dashboard run was performed; validation is from test coverage and code review of the WS logging path. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Tejas Chopra
|
4951cf80a2
|
fix(proxy): don't 502 Anthropic streaming on a legal mixed CCR + client-tool turn (#2089) (#2117)
## Description
Anthropic **buffered-streaming** returned a **502** on a legal turn:
when the model emits `headroom_retrieve` alongside a non-CCR client
tool, `CCRResponseHandler` intentionally skips CCR resolution (#839) and
hands both tool_use blocks back for the client to resolve. The
non-streaming path returns that as 200; the streaming path wrongly
failed closed with "Unable to safely complete streamed CCR retrieval."
Fix: add a provider-generic `CCRResponseHandler.residual_ccr_status()` →
`resolved` / `skipped_mixed_tools` / `error`. The streaming path now
only 502s on a genuine `error`; on the intentional mixed-tool skip it
falls through to the existing SSE resynthesis (200) preserving **both**
tool_use blocks — matching the non-streaming path. The misleading
"handled successfully" log no longer fires on skip.
Closes #2089
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/ccr/response_handler.py`: shared, provider-generic
`residual_ccr_status()`.
- `headroom/proxy/handlers/anthropic.py`: streaming path fails closed
only on a real residual-CCR error; passes through the legal skip as 200
SSE.
- `tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py`:
mixed-tool case now returns 200 SSE preserving both blocks.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] New/updated tests for the mixed-tool pass-through branch
### Test Output
```text
GitHub CI on current head:
- lint: pass
- build/build-wheel/build-wheel-windows: pass
- test matrix, test-agno, test-extras, test-dashboard-ui: pass
- docker-native-e2e: pass
Review spot-check:
uv run --extra proxy --extra dev python -m pytest tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py tests/test_ccr_response_handler_extra.py -q
15 passed, 1 warning
```
## Real Behavior Proof
- Environment: GitHub Actions on PR head `
|
||
|
Tejas Chopra
|
2976d49f18
|
fix(proxy): preserve sub-path in X-Headroom-Base-Url custom upstream (#2037) (#2127)
## Description `_resolve_openai_upstream_base` ran the `X-Headroom-Base-Url` value through `_normalize_origin`, which strips the path. A custom OpenAI-compatible upstream served from a sub-path, such as `https://host/api/v1`, was routed to the bare origin and returned `proxy_error` (#2037). This re-attaches the path after origin normalization. This is a clean extraction of the path fix from #2047, which bundled it with an unrelated `supports_websockets = true` to `false` default change across init/wrap/codex. #2047 can be closed in favor of this narrower fix. Closes #2037 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/handlers/openai.py`: re-attach the request header path component in `_resolve_openai_upstream_base` after origin normalization. - `tests/test_proxy/test_openai_upstream_header.py`: assert sub-paths are preserved and trailing slashes are normalized. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_proxy/test_openai_upstream_header.py 5 passed $ ruff check headroom/proxy/handlers/openai.py tests/test_proxy/test_openai_upstream_header.py All checks passed! ``` ## Real Behavior Proof - Environment: local proxy header resolution path, custom OpenAI-compatible upstream configured via `X-Headroom-Base-Url`. - Exact command / steps: resolve `X-Headroom-Base-Url: https://gateway.example/api/v1` through `_resolve_openai_upstream_base` / `_resolve_openai_upstream`. - Observed result: before the fix, the upstream resolved to `https://gateway.example` and lost `/api/v1`, causing the proxy to route to the wrong endpoint. After the fix, it resolves to `https://gateway.example/api/v1`; a trailing slash is normalized away. - Not tested: end-to-end request against a live third-party OpenAI-compatible gateway. The regression is covered at the proxy routing helper layer where the path was dropped. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A. ## Additional Notes - Documentation is not updated because this fixes the existing header behavior rather than changing a documented user-facing contract. - Changelog is not updated in this PR; the change is scoped to the regression and test. - `mypy headroom` was not run in the author's workflow. |
||
|
Abhay Singh
|
19201e842f
|
fix(proxy/openai): respect explicit stream_options.include_usage (#2026)
## Description
On the direct OpenAI `/v1/chat/completions` streaming path, the handler
injects
`stream_options.include_usage = True` so it can count tokens from the
trailing usage chunk —
but it does so **unconditionally**, including flipping an explicit
client `include_usage: false`
to `true` (`headroom/proxy/handlers/openai.py`):
```python
if "stream_options" not in body:
body["stream_options"] = {"include_usage": True}
elif isinstance(body.get("stream_options"), dict):
body["stream_options"]["include_usage"] = True # overrides an explicit `false`
```
When the client passed `stream_options: {"include_usage": false}` (or a
dict that set some
other key), the upstream is nevertheless asked for usage and appends a
terminal usage-only
frame:
```
data: {"id":...,"choices":[],"usage":{...}}
data: [DONE]
```
The extremely common client pattern `for chunk in stream:
chunk.choices[0].delta.content`
then raises `IndexError` on that empty-`choices` frame — for a usage
chunk the client
explicitly opted out of.
Closes: no issue filed — found while auditing the streaming
request-shaping.
## Fix
Only fill in `include_usage` when the client left the choice open — no
`stream_options` at all,
or a `stream_options` dict that doesn't mention `include_usage`. An
explicit `true`/`false` is
respected. Extracted into a small `_apply_stream_usage_option(body)`
helper (mirroring the
existing `_normalize_openai_max_tokens`) for a clean unit-test seam:
```python
stream_options = body.get("stream_options")
if stream_options is None:
body["stream_options"] = {"include_usage": True}
elif isinstance(stream_options, dict) and "include_usage" not in stream_options:
stream_options["include_usage"] = True
```
Scope note: this respects an explicit client choice, which is the
unambiguous defect. The
separate question of whether to strip the synthetic usage chunk when
Headroom injected the
option itself (the no-`stream_options` default, kept for token-counting)
touches the raw SSE
byte stream and is intentionally left out of this change.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/proxy/handlers/openai.py`: add
`_apply_stream_usage_option(body)` and call it from the streaming chat
path; it no longer overrides an explicit client `include_usage`.
- `tests/test_proxy/test_openai_stream_usage_option.py`: cover explicit
`false` (respected), explicit `true` (preserved), absent (injected), and
dict-without-key (filled in).
## Testing
- [x] New regression tests added
(`tests/test_proxy/test_openai_stream_usage_option.py`)
- [x] Linting/formatting clean — run with the CI-pinned `ruff==0.15.17`
- [ ] Full `pytest` deferred to CI (local-OOM reason below).
```text
$ uvx ruff@0.15.17 check headroom/proxy/handlers/openai.py tests/test_proxy/test_openai_stream_usage_option.py
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.10, headroom from this branch.
Importing `headroom` pulls in the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I verified the decision logic
with a dependency-free script and left the full pytest to CI.
- Exact command / steps: ran a client body with `stream_options:
{include_usage: false}` (plus the explicit-true, absent, and
dict-without-key cases) through the old unconditional injection and the
new helper.
- Observed result: the old logic flips the client's `false` to `true`;
the new logic respects it:
```text
explicit false: OLD -> {'include_usage': True} NEW -> {'include_usage': False}
INCLUDE_USAGE RESPECT-CLIENT FIX VERIFIED (old flips false->true; new respects false)
```
- Not tested: a full streaming round-trip through a live OpenAI upstream
(needs the heavy stack + a key). The fix is confined to the
request-shaping helper and the new tests drive it directly. Full local
`pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — ran
lint + a standalone logic check; full pytest deferred to CI (local OOM,
disclosed above)
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- Small, contained change plus a helper and tests; no new dependencies.
The backend-path injection (`test_backend_anyllm` /
`test_backend_streaming_cache_metrics`) is untouched — those pass an
explicit `include_usage: true`, which is preserved.
- @JerrettDavis tagging you — this one makes a client that sent
`include_usage: false` hit an `IndexError` on the usage chunk, so it
seemed worth surfacing. Thanks!
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
Abhay Singh
|
38306a331c
|
fix(proxy/gemini): thread savings-profile kwargs into apply() (#1994)
## Description
The native Gemini / Vertex-google compression handlers build the
pipeline call like this
(`headroom/proxy/handlers/gemini.py`, three sites —
`handle_gemini_generate_content` ~L487,
`handle_google_cloudcode_stream` ~L843, `handle_gemini_count_tokens`
~L1104):
```python
result = await self._run_compression_in_executor(
lambda: self.openai_pipeline.apply(
messages=messages,
model=model,
model_limit=context_limit,
context=extract_user_query(messages),
waste_messages=waste_messages,
), # <-- no **proxy_pipeline_kwargs(self.config)
...
)
```
Every other live compression path threads
`proxy_pipeline_kwargs(self.config)` into `apply()`
— `handlers/openai.py` (chat + responses), `handlers/anthropic.py`, and
the `/v1/compress`
endpoint. The Gemini handler never imports or calls it, so the savings
profile and the
ProxyConfig compression knobs never reach the pipeline for Gemini/Vertex
requests.
The proxy pipeline is built with only `transforms` + `provider`
(`server.py`), and
`ContentRouter` reads the accuracy-sensitive knobs per-call from
`**kwargs`. With the kwargs
missing, Gemini falls back to router defaults instead of the
profile/config values:
- `min_tokens_to_compress` → hardcoded **50** instead of the
coding-profile **25** / `config.min_tokens_to_crush`
- `protect_recent` → router default instead of the profile / configured
value
- `target_ratio` → **None** instead of the CLI default **0.4** used
everywhere else
- `max_items_after_crush`, `smart_crusher_with_compaction`,
`force_kompress` → router defaults
So Gemini/Vertex requests compress with a materially different (and
inconsistent) posture than
Claude/Codex/Cursor, and any user-tuned `HEADROOM_SAVINGS_PROFILE` /
`HEADROOM_TARGET_RATIO` / `HEADROOM_MIN_TOKENS` /
`HEADROOM_PROTECT_RECENT` is ignored on this
path.
This is the exact bug **#1534** fixed for the OpenAI chat path.
Closes: no issue filed — found while auditing profile/config threading
across the provider handlers.
## Fix
Import `proxy_pipeline_kwargs` (as `openai.py`/`anthropic.py` do) and
add
`**proxy_pipeline_kwargs(self.config)` to all three Gemini
`openai_pipeline.apply(...)` calls.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/proxy/handlers/gemini.py`: import `proxy_pipeline_kwargs`;
thread `**proxy_pipeline_kwargs(self.config)` into the three
`openai_pipeline.apply(...)` call sites (generateContent, Cloud Code
stream, countTokens).
- `tests/test_proxy/test_gemini_savings_profile.py`: drive the native
`/v1beta/models/{model}:generateContent` route with
`savings_profile="agent-90"` and assert the profile knobs
(`compress_user_messages`, `target_ratio`, `min_tokens_to_compress`,
`compress_system_messages`) reach `apply()`.
## Testing
- [x] New regression test added
(`tests/test_proxy/test_gemini_savings_profile.py`), mirroring the #1534
chat-path test
- [x] Linting passes (`ruff check`) and formatting is clean (`ruff
format --check`) — run with the CI-pinned `ruff==0.15.17`
- [ ] Full `pytest` deferred to CI (local-OOM reason below).
```text
$ uvx ruff@0.15.17 check headroom/proxy/handlers/gemini.py tests/test_proxy/test_gemini_savings_profile.py
All checks passed!
$ uvx ruff@0.15.17 format --check headroom/proxy/handlers/gemini.py tests/test_proxy/test_gemini_savings_profile.py
2 files already formatted
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.10, headroom from this branch.
Importing `headroom` pulls in the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I verified the kwargs threading
with a dependency-free script and left the full pytest to CI.
- Exact command / steps: mechanically reproduced the call site —
`apply(**base)` (old) vs `apply(**base,
**proxy_pipeline_kwargs(config))` (new) — and captured the kwargs each
produced.
- Observed result: the old call omits the profile knobs entirely; the
new call threads them:
```text
OLD gemini apply() kwargs: ['context', 'messages', 'model', 'model_limit', 'waste_messages']
-> profile knobs DROPPED (savings profile / config ignored)
NEW gemini apply() kwargs: ['compress_system_messages', 'compress_user_messages', 'context',
'max_items_after_crush', 'messages', 'min_tokens_to_compress', 'model', 'model_limit',
'protect_recent', 'target_ratio', 'waste_messages']
-> profile knobs THREADED: target_ratio=0.10, min_tokens_to_compress=120, compress_user/system=True
GEMINI KWARGS THREADING VERIFIED
```
- Not tested: a full request through a live Gemini/Vertex upstream
(needs the heavy stack + a key). The new test drives the native route
with a mocked upstream and asserts the kwargs reach `apply()`. Full
local `pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — ran
lint + a standalone logic check; full pytest deferred to CI (local OOM,
disclosed above)
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- No new dependencies; one import plus three call-site kwargs and a
test.
- @JerrettDavis tagging you — this is the Gemini sibling of the #1534
chat-path fix; the profile/config knobs are currently ignored for the
whole Gemini/Vertex surface, so it may be worth a look when you have a
moment.
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
Adryan Eka Vandra
|
bb2acf700a
|
fix(proxy): honor x-headroom-base-url on /v1/messages route (#1763)
## Description The Anthropic Messages route (`POST /v1/messages`) ignored the `x-headroom-base-url` per-request upstream override and unconditionally forwarded to `api.anthropic.com`. `handle_anthropic_messages` already accepts `upstream_base_url` (it builds the upstream URL via `build_copilot_upstream_url`), but the route never passed it. Clients that speak the Anthropic Messages wire format while authenticating against a non-Anthropic gateway (e.g. OpenCode Zen's "Go" tier) were forwarded to the real Anthropic API, which rejected the gateway key with `401 invalid x-api-key`. The route now reads and trims `x-headroom-base-url` and passes it through as `upstream_base_url`, mirroring the OpenAI-compatible routes and the generic passthrough route (`proxy_routes.py:996`). Closes #1760 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/providers/proxy_routes.py`: the `/v1/messages` route reads `x-headroom-base-url`; when present it strips whitespace and a trailing slash and passes the value as `upstream_base_url` to `handle_anthropic_messages`. Absent or whitespace-only headers keep the previous default (`api.anthropic.com`). - `tests/test_proxy/test_anthropic_upstream_header.py`: new test module pinning the route contract (header present, absent, empty, whitespace-only, trimming + trailing-slash stripping). - `docs/content/docs/configuration.mdx`: new "Proxy upstream override (`x-headroom-base-url`)" subsection under Per-Request Overrides documenting the header across the OpenAI, Anthropic Messages, and passthrough routes. - `CHANGELOG.md`: `Unreleased > Fixed` entry for the `/v1/messages` override. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_proxy/ -k "anthropic or passthrough or bedrock" collected 140 items / 91 deselected / 49 selected tests/test_proxy/test_anthropic_upstream_header.py .... [ 65%] ... 49 passed, 91 deselected, 1 warning in 79.68s $ ruff check headroom/providers/proxy_routes.py tests/test_proxy/test_anthropic_upstream_header.py All checks passed! $ mypy headroom/providers/proxy_routes.py Success: no issues found in 1 source file ``` ## Real Behavior Proof Ran the actual `headroom proxy` against a local mock upstream (a tiny HTTP server on `127.0.0.1:9911` that logs the path it receives) to reproduce the issue's before/after. - Environment: local, macOS, Python 3.12; ran `headroom proxy --port 8799` against a local mock upstream (a tiny HTTP server on `127.0.0.1:9911` that logs the path it receives). - Exact command / steps: started the proxy and the mock upstream, then sent one `POST /v1/messages` **with** the override header and one **without** it (negative control), using these two `curl` commands. ```bash # WITH the override header — expect routing to the mock at 127.0.0.1:9911 curl http://127.0.0.1:8799/v1/messages \ -H "content-type: application/json" -H "anthropic-version: 2023-06-01" \ -H "x-headroom-base-url: http://127.0.0.1:9911" \ -H "x-api-key: zen-test-key" \ -d '{"model":"glm-5.2","max_tokens":16,"messages":[{"role":"user","content":"hi"}]}' # WITHOUT the override header — expect routing to the real api.anthropic.com curl http://127.0.0.1:8799/v1/messages \ -H "content-type: application/json" -H "anthropic-version: 2023-06-01" \ -H "x-api-key: sk-ant-fake" \ -d '{"model":"claude-3-5-sonnet-20241022","max_tokens":16,"messages":[{"role":"user","content":"hi"}]}' ``` - Observed result: with the header, the mock upstream logged `HIT path=/v1/messages x-api-key=zen-test-key` and the proxy returned `HTTP 200`, confirming the request was routed to `<x-headroom-base-url>/v1/messages` carrying the gateway key. Without the header, the request went to the real `api.anthropic.com` (returned `HTTP 401` with a genuine `request_id` and `{"type":"authentication_error","message":"invalid x-api-key"}`) and the mock received no additional hit — matching the pre-fix behavior in the issue. Also verified by TDD: the two override unit cases failed before the route change (`assert None == 'https://opencode.ai/zen/go'`) and passed after it; all 4 new cases and 49 related proxy tests are green. - Not tested: a request against the real OpenCode Zen gateway (no credentials); the gateway path is verified with a local mock upstream instead. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - Manual testing against the real OpenCode Zen gateway is N/A (no credentials); a local mock upstream is used instead to prove the routing (see Real Behavior Proof). - Scope is limited to `/v1/messages`. The related `/v1/messages/count_tokens` route uses a fixed passthrough target and is out of scope for this issue. |
||
|
Rod Boev
|
62cd3072a2
|
feat(ccr): wire retrieve-tool interception into OpenAI Responses handler (#1898)
## Description Refs #1877. `handle_openai_responses` (the `/v1/responses` HTTP handler) had zero CCR / `headroom_retrieve` wiring, so a retrieve `function_call` in a Responses API reply passed straight through to the client instead of being resolved server-side, unlike the parallel chat-completions backend path (`handle_openai_chat`, ~2775-2848), which already intercepts `headroom_retrieve` tool calls via `ccr_response_handler.has_ccr_tool_calls()` / `handle_response()`. This PR is scoped to the core interception gap only. The issue's proposals A (egress scrubber) and B/C (event-level SSE parsing/splicing for true mid-stream interception) are out of scope here; the streaming case is instead handled by forcing a buffered (non-streaming) upstream call when `headroom_retrieve` is offered, matching the existing buffered-CCR pattern in the Anthropic handler. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/ccr/response_handler.py`: added an `"openai_responses"` provider branch to `CCRResponseHandler`; `_extract_tool_calls` reads flat `function_call` items from the top-level `output[]` array, tool-call IDs key off `call_id`, and `_extract_assistant_message` / `_create_tool_result_message` return sentinel-keyed item lists that `handle_response()` extends into the running item history. - `headroom/ccr/tool_injection.py`: added a `parse_tool_call` branch for `"openai_responses"` where name and arguments are flat on the item. - `headroom/proxy/handlers/openai.py`: detects non-streaming `headroom_retrieve` function calls and runs `ccr_response_handler.handle_response()` with a stateless continuation that resends the full `input[]` item history. - `headroom/proxy/handlers/openai.py`: forces `stream:true` requests with `headroom_retrieve` available through a buffered `stream:false` upstream call, resolves retrieval server-side, then reconstructs a minimal Responses SSE stream for the client. - `headroom/proxy/handlers/openai.py`: treats `ccr_response_handler` as optional on `OpenAIHandlerMixin` consumers, so handlers without CCR support keep the existing Codex routing, streaming, header stripping, memory timeout, and compression fail-open behavior. - Non-CCR streaming requests are unaffected; they still go through `_stream_response()` as before. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_ccr_response_handler_openai_responses.py -q`) - [x] Integration tests pass (`uv run pytest tests/test_proxy/test_openai_responses_ccr.py -q`) - [x] Regression tests pass (`uv run pytest tests/test_openai_codex_routing.py -q`) - [x] Linting passes (`uv run ruff check headroom/proxy/handlers/openai.py tests/test_openai_codex_routing.py tests/test_proxy/test_openai_responses_ccr.py tests/test_ccr_response_handler_openai_responses.py`) - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_openai_codex_routing.py -q tests\test_openai_codex_routing.py .................... [100%] 20 passed in 0.51s $ uv run pytest tests/test_proxy/test_openai_responses_ccr.py tests/test_ccr_response_handler_openai_responses.py -q tests\test_proxy\test_openai_responses_ccr.py .... [ 25%] tests\test_ccr_response_handler_openai_responses.py ............ [100%] 16 passed, 1 warning in 27.51s $ uv run ruff check headroom/proxy/handlers/openai.py tests/test_openai_codex_routing.py tests/test_proxy/test_openai_responses_ccr.py tests/test_ccr_response_handler_openai_responses.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows, Python 3.12 via uv-managed venv, local PR worktree on branch `pr/1877-ccr-responses-interception`, no live LLM provider needed because the tests stub upstream HTTP and CCR continuation behavior. - Exact command / steps: Ran `uv run pytest tests/test_openai_codex_routing.py -q` to reproduce the CI-failing Codex routing surface after the optional-handler fix; ran `uv run pytest tests/test_proxy/test_openai_responses_ccr.py tests/test_ccr_response_handler_openai_responses.py -q` to cover the positive Responses CCR interception path; ran targeted Ruff on the touched handler and related tests. - Observed result: Codex routing tests that previously failed with `AttributeError: '_DummyOpenAIHandler' object has no attribute 'ccr_response_handler'` now pass; Responses CCR still detects and resolves `headroom_retrieve` when a real proxy installs `ccr_response_handler`; non-CCR streaming requests still route through `_stream_response()`. - Not tested: true event-level mid-stream Responses SSE splicing and client-bound egress marker scrubbing are out of scope for this PR and remain future work from issue #1877's broader proposals. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes No documentation or changelog update was made because this is internal proxy CCR behavior, not a user-facing command or configuration change. |
||
|
Tejas Chopra
|
7c2f0ea079
|
feat(cache): provider-agnostic cache-mode delta + cc-agnostic prefix comparison (#1868)
## Description <!-- Briefly explain the change and why it is needed. --> Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # Paste relevant command output or artifact links here ``` ## Real Behavior Proof - Environment: - Exact command / steps: - Observed result: - Not tested: ## Review Readiness - [ ] I have performed a self-review - [ ] This PR is ready for human review ## Checklist - [ ] My code follows the project's style guidelines - [ ] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) Add screenshots to help explain your changes. ## Additional Notes <!-- Mention any N/A checklist items, tradeoffs, follow-ups, or maintainer context. --> |
||
|
Vinay Gupta
|
f18c6bd896
|
fix(codex): OpenCode Zen telemetry attribution (#1648)
## Description Fixes #1602. OpenCode Zen custom-base requests can reach Headroom through the generic passthrough path, but that route was not supplying endpoint/provider metadata for Zen chat completions. This made forwarded Zen traffic invisible in dashboard provider, usage, and token telemetry. Closes #1602 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added a narrow OpenCode Zen custom-base classifier for `POST /zen/v1/chat/completions` on `opencode.ai` and `www.opencode.ai`. - Passed `endpoint_name="chat/completions"` and `provider="zen"` into catch-all passthrough telemetry for matching Zen traffic. - Attributed normalized OpenCode transport traffic (`/v1/chat/completions` with `x-headroom-original-path: /zen/v1/chat/completions`) to `zen` for request outcomes while keeping the OpenAI parser path unchanged. - Added coverage for direct catch-all routing, normalized original-path routing, token usage outcome recording, and false-positive paths like `/mcp/v1/chat/completions`, `/npm/v1/chat/completions`, and `/context7/v1/chat/completions`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ rtk pytest tests/test_custom_base_passthrough_telemetry.py -q Pytest: 4 passed $ rtk uvx --from ruff==0.15.17 ruff check headroom/proxy/handlers/openai.py headroom/providers/proxy_routes.py tests/test_custom_base_passthrough_telemetry.py tests/test_provider_proxy_routes.py tests/test_proxy/test_openai_transport_path_prefix.py All checks passed! $ rtk uvx --from ruff==0.15.17 ruff format --check headroom/proxy/handlers/openai.py headroom/providers/proxy_routes.py tests/test_custom_base_passthrough_telemetry.py tests/test_provider_proxy_routes.py tests/test_proxy/test_openai_transport_path_prefix.py 5 files already formatted $ rtk /Library/Frameworks/Python.framework/Versions/3.13/bin/python3 -m py_compile headroom/proxy/handlers/openai.py headroom/providers/proxy_routes.py tests/test_custom_base_passthrough_telemetry.py tests/test_provider_proxy_routes.py tests/test_proxy/test_openai_transport_path_prefix.py # passed $ rtk git diff --check # passed ``` GitHub Actions also passed after the final push, including CI, Docker native/wrap/init E2E, security, lint, and PR governance. ## Real Behavior Proof - Environment: local worktree on macOS plus GitHub Actions for PR #1648. - Exact command / steps: ran focused pytest coverage for Zen passthrough telemetry, Ruff check/format validation on touched files, Python compile validation, `git diff --check`, and waited for the full GitHub Actions rollup. - Observed result: Zen custom-base chat completions now record request outcomes as provider `zen` with endpoint `chat/completions`; false-positive OpenCode paths remain unattributed to Zen; GitHub checks are green. - Not tested: full local test suite did not collect in this worktree because the native `headroom._core` extension is not installed. `rtk npm --prefix plugins/opencode test` is also blocked locally because `vitest` is not installed in `plugins/opencode/node_modules`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes The documentation and CHANGELOG checklist items are not applicable for this narrow telemetry bug fix. No new comments were added because the code path is covered by narrowly named helper/test cases. |
||
|
Abhay Singh
|
7ff842da17
|
fix(proxy/openai): thread savings-profile kwargs into chat completions (#1606)
## Description OpenAI-compatible `/v1/chat/completions` requests didn't receive the same proxy savings/profile kwargs as the other compression paths. The live chat handler (`handle_openai_chat` in `headroom/proxy/handlers/openai.py`) called `openai_pipeline.apply()` with only `model_limit` / `context` / `frozen_message_count` / `biases` / `compression_policy` — it never passed `proxy_pipeline_kwargs(self.config)`. So when the proxy runs with `HEADROOM_SAVINGS_PROFILE=agent-90`, the effective config reports user/system-message compression and `target_ratio=0.10`, but the real chat path silently dropped all of it. OpenAI-compatible clients such as OpenCode kept protecting user messages and missed the configured profile. For contrast, `handlers/anthropic.py` passes `**proxy_pipeline_kwargs(self.config)` to every `apply()` call, and so does the dedicated OpenAI compress endpoint in this same module — only the two chat-completions `apply()` sites were missing it. Closes #1534 ## Fix Add `**proxy_pipeline_kwargs(self.config)` to both chat-path `apply()` calls (the token-mode branch and the non-token branch): ```python lambda: self.openai_pipeline.apply( messages=messages, model=model, model_limit=context_limit, context=extract_user_query(messages), frozen_message_count=openai_frozen_count, biases=_hook_biases, compression_policy=compression_policy, **proxy_pipeline_kwargs(self.config), # ← added ) ``` `proxy_pipeline_kwargs` is already imported in the module and is the exact helper the Anthropic handler and the OpenAI compress endpoint use, so the chat path now matches them. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/proxy/handlers/openai.py`: pass `**proxy_pipeline_kwargs(self.config)` on both `apply()` call sites in `handle_openai_chat` (token-mode and non-token branches). - `tests/test_proxy/test_openai_chat_savings_profile.py`: new regression test driving the chat handler with `savings_profile="agent-90"` and asserting the profile knobs reach `apply()`. - `CHANGELOG.md`: Bug Fixes entry under Unreleased. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output The new test drives the real chat handler through the `create_app` + `TestClient` harness with a recording `apply()` stub. Before the fix it captures exactly the five kwargs the issue describes (no profile knobs); after the fix the profile knobs are present: ```text # before the fix (openai.py reverted, test kept) E AssertionError: assert None is True E + where None = {...}.get('compress_user_messages') # captured kwargs were: biases, compression_policy, messages, model, # model_limit, context, frozen_message_count — no profile knobs FAILED tests/test_proxy/test_openai_chat_savings_profile.py::test_chat_completions_threads_savings_profile_kwargs_into_apply # after the fix tests\test_proxy\test_openai_chat_savings_profile.py . ======================== 1 passed, 1 warning in 39.44s ======================== ``` No regression in the existing chat backend-path suite: ```text $ uv run pytest tests/test_proxy/test_openai_backend_path.py ======================== 5 passed, 1 warning in 15.78s ======================== $ uv run ruff check headroom/proxy/handlers/openai.py tests/test_proxy/test_openai_chat_savings_profile.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12.11, headroom built from this branch (`uv sync --extra dev`), proxy config `savings_profile="agent-90"`, `optimize=True`, `backend="anyllm"` with a mocked OpenAI upstream. - Exact command / steps: started the app with `create_app(config)`, replaced `proxy.openai_pipeline.apply` with a recording stub, and POSTed a real `/v1/chat/completions` request with a large user message so the compression decision fires. Inspected the kwargs the handler actually passed to `apply()`. - Observed result: before the fix the recorded `apply()` kwargs were `{biases, compression_policy, messages, model, model_limit, context, frozen_message_count}` — no profile knobs. After the fix the same call also carries `compress_user_messages=True`, `compress_system_messages=True`, `target_ratio=0.10`, `min_tokens_to_compress=120` (the agent-90 profile), matching the issue's "Expected". - Not tested: did not stand up a real OpenAI/OpenCode upstream end-to-end (no live key in this environment); the upstream is mocked and the assertion is on the kwargs the proxy threads into the compression pipeline, which is exactly what the bug was about. Did not run the full `mypy headroom` pass (two-line kwarg addition, no new types). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - Two-line change plus comments; no new dependencies. Reuses the existing `proxy_pipeline_kwargs` helper, so behavior is consistent across Anthropic, the OpenAI compress endpoint, and now the OpenAI chat path. - @chopratejas flagging you for review — this aligns the OpenAI chat path with the savings-profile handling the other providers already had. Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
Tejas Chopra
|
248ae0f3e0
|
fix(proxy): freeze must forward cached (compressed) prefix byte-identical — stop token-mode cache busting (#1850)
The freeze path (both providers) emits the agent's ORIGINAL bytes for a frozen message, but the provider cached whatever we FORWARDED last turn (the compressed form). Forwarding original then mismatches the cached prefix and busts it from that point — re-creating the whole suffix. Measured on a real SWE-bench run: 100% of attributed misses were prefix_change, ~56% of ALL cache-writes were bust-induced (2.8M tokens), driving cache_create +150% and cost +41% vs baseline. Cache mode already avoided this via _extract_cache_stable_delta (replay the previously-forwarded prefix, compress only the delta). Token mode called apply(frozen_count) directly, which forwards original for the frozen region. Fix: add a shared, provider-agnostic overlay_cached_prefix() that replays the previously-forwarded (cached, compressed) prefix byte-identical, append-only guarded and idempotent, and apply it in BOTH the Anthropic and OpenAI handlers right before forwarding. This makes freezing byte-identical in every mode, so the only remaining difference between "token" and "cache" mode is how large a mutable (still-compressible) tail each leaves — not whether the frozen prefix busts the cache. Tests: - test_cache_prefix_overlay.py: the helper (replay, append-only guard, idempotence). - test_cross_turn_cache_safety.py: the invariant that was missing — drive the REAL tracker + freeze + overlay over multiple append-only turns against a simulated provider prefix cache and assert the forwarded prefix stays byte-identical turn-over-turn. Load-bearing: it fails (detects the bust) without the overlay. ## Description <!-- Briefly explain the change and why it is needed. --> Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # Paste relevant command output or artifact links here ``` ## Real Behavior Proof - Environment: - Exact command / steps: - Observed result: - Not tested: ## Review Readiness - [ ] I have performed a self-review - [ ] This PR is ready for human review ## Checklist - [ ] My code follows the project's style guidelines - [ ] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) Add screenshots to help explain your changes. ## Additional Notes <!-- Mention any N/A checklist items, tradeoffs, follow-ups, or maintainer context. --> |
||
|
Kirill
|
6c48ac81f2
|
fix(proxy): honor x-headroom-base-url in dedicated OpenAI handlers (#1502)
## Description The dedicated OpenAI handlers (`/v1/chat/completions`, `/v1/responses`) ignore the `x-headroom-base-url` request header that the opencode/CLI transports already send on every routed request (`plugins/opencode/src/transport.ts`) and that the generic passthrough route already honors (`providers/proxy_routes.py:953`). As a result, OpenAI-compatible gateways (LiteLLM, CPA, self-hosted vLLM, Azure OpenAI) route correctly for passthrough traffic, but the dedicated chat/responses handlers fall back to the default `OPENAI_API_URL` and send the request — and the user's provider key — to the wrong upstream. This forces OpenCode users behind a custom gateway to run a hand-rolled plugin that re-spawns the proxy with `OPENAI_TARGET_API_URL` instead of the supported `HeadroomPlugin`. Refs #1503 (feature-request issue with full spec — API surface, failure modes, security considerations). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) Non-breaking: when the header is absent (the common case), behavior is identical to before — `_resolve_openai_upstream` falls back to `self.OPENAI_API_URL`. ## Changes Made - Added `OpenAIHandlerMixin._resolve_openai_upstream(request)` — returns `request.headers.get("x-headroom-base-url") or self.OPENAI_API_URL`. Prefers the header, falls back to the configured URL. - Used it at the two direct-path HTTP upstream sites: - `handle_openai_chat` → `build_copilot_upstream_url(self._resolve_openai_upstream(request), "/v1/chat/completions")` - `handle_openai_responses` → `build_copilot_upstream_url(self._resolve_openai_upstream(request), "/v1/responses")` - This makes the dedicated handlers behave identically to the catch-all passthrough and the Azure path (`_select_passthrough_base_url`, `providers/proxy_routes.py:66,:953`), which already read the same header. - The header is already stripped before forwarding by `helpers._strip_internal_headers`, so no upstream leakage / fingerprinting is introduced. - CHANGELOG entry under `### Bug Fixes`. ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) — not run locally (maturin native build not available in my env; covered by CI) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output New `tests/test_proxy/test_openai_upstream_header.py` pins the resolution contract (3 cases): ```text $ pytest tests/test_proxy/test_openai_upstream_header.py -q ... collected 3 items tests/test_proxy/test_openai_upstream_header.py ... [100%] ========================= 3 passed, 1 warning in 0.25s ========================= ``` Fail-before confirmed (unpatched handler raises `AttributeError: _resolve_openai_upstream`): ```text FAILED tests/test_proxy/test_openai_upstream_header.py::test_header_overrides_configured_url FAILED tests/test_proxy/test_openai_upstream_header.py::test_missing_header_falls_back_to_configured_url FAILED tests/test_proxy/test_openai_upstream_header.py::test_empty_header_falls_back_to_configured_url ========================= 3 failed, 1 warning in 0.29s ========================= ``` Lint/format: ```text $ ruff check headroom/proxy/handlers/openai.py tests/test_proxy/test_openai_upstream_header.py Ruff: No issues found $ ruff format --check headroom/proxy/handlers/openai.py tests/test_proxy/test_openai_upstream_header.py 2 files already formatted ``` ## Real Behavior Proof - Environment: macOS (darwin), Python 3.12 (pipx install of `headroom-ai`), Headroom proxy `headroom proxy --port 8787` with `OPENAI_TARGET_API_URL=https://cpa.funxyz.fun` (an OpenAI-compatible gateway — "CLI Proxy API"). OpenCode with a custom `cpa` provider (`@ai-sdk/openai-compatible`, `baseURL: https://cpa.funxyz.fun/v1`) using the official `HeadroomPlugin`. - Exact command / steps: traced the bug in the installed package source — confirmed `handle_openai_chat` builds its upstream URL from `self.OPENAI_API_URL` only (`proxy/handlers/openai.py:2487`), never reading `x-headroom-base-url`, while `providers/proxy_routes.py:953` reads it for passthrough. Then applied this patch and re-imported the handler from the repo source via `PYTHONPATH`. - Observed result: before the patch, `/v1/chat/completions` requests ignored the `x-headroom-base-url: https://cpa.funxyz.fun` header (set by the opencode transport) and routed to the default upstream, failing against a non-OpenAI gateway — requiring a custom respawn-plugin workaround. After the patch, `_resolve_openai_upstream` returns the header value and the request forwards to the configured gateway; the official `HeadroomPlugin` works without the env-var workaround. Unit tests pass (3/3) and fail on the unpatched handler (3/3). - Not tested: full `uv sync` CI matrix (native `headroom._core` maturin build unavailable locally, so `headroom.proxy.server` import chain that pulls `transforms/content_router` can't be exercised here — the edited handler module imports fine and the focused unit tests exercise the new method directly). WebSocket/Codex paths (`handle_openai_responses_ws`, `_ws_http_fallback`) — intentionally out of scope (see Additional Notes). `mypy headroom` — deferred to CI. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation — no public API/docs surface; the header is already documented as an internal control flag in `helpers.py:1489-1495` - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes **Scope boundary — WebSocket paths intentionally unchanged.** The two WS sites (`handle_openai_responses_ws`, `_ws_http_fallback`) are Codex-specific and left as-is: 1. They short-circuit to `chatgpt.com` under ChatGPT-session auth (not arbitrary gateways). 2. The WS path strips `x-headroom-base-url` from `upstream_headers` (`_strip_internal`, ~line 3756) before the upstream URL is built, and `_ws_http_fallback` receives already-stripped headers as a parameter. Honoring the header there would require threading it through the WS internals and changing a signature, for a path a custom OpenAI-compatible WebSocket gateway is unlikely to use. The HTTP paths cover the realistic gateway case. Happy to do it as a follow-up if maintainers want it. **Issue-first.** This is a behaviour change, so per CONTRIBUTING a feature-request issue (#1503) is open for triage with the full spec (API surface, user stories, failure modes, security). This PR implements it; holding for maintainer 👍 before treating as ready to merge. --------- Co-authored-by: ShutovKS <shutovks@example.local> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Vinay Gupta
|
d337e3b828
|
fix(proxy): handle streaming CCR retrieval (#1451)
## Description Fixes Anthropic-compatible streaming requests that can emit the internal `headroom_retrieve` CCR tool. When a `stream: true` request includes the CCR retrieve tool and response handling is enabled, Headroom now buffers the upstream call as `stream: false`, lets the existing CCR response handler retrieve and continue, and returns the final result as Anthropic SSE so streaming clients do not see the internal tool call. Closes #1450 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Detect direct Anthropic-compatible `stream: true` requests where `headroom_retrieve` is available and CCR response handling is enabled. - Route those requests through the existing buffered/non-stream CCR response handler, then convert the final response back to `text/event-stream`. - Fail closed with a 502 SSE error if a buffered response still contains `headroom_retrieve` after CCR handling, instead of leaking the internal tool to the client. - Preserve Anthropic `thinking`, `redacted_thinking`, signatures, and citations when converting response JSON back to SSE. - Add regression coverage for handled CCR retrieval, unused CCR tool availability, normal streaming passthrough, mixed client/CCR tool fail-closed behavior, and SSE conversion preservation. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ rtk gh pr checks 1451 --repo headroomlabs-ai/headroom CI Checks Summary: [ok] Passed: 20 [FAIL] Failed: 0 Relevant CI commands from .github/workflows/ci.yml: - ruff check . - ruff format --check . - mypy headroom --ignore-missing-imports - pytest tests scripts/tests $ rtk python3 -m py_compile headroom/proxy/handlers/anthropic.py headroom/proxy/handlers/streaming.py tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py tests/test_sse_thinking_blocks.py # passed, no output $ rtk pytest tests/test_sse_thinking_blocks.py -q Pytest: 6 passed $ MACOSX_DEPLOYMENT_TARGET=15.0 rtk uv run --python 3.13 pytest tests/test_proxy/test_anthropic_streaming_ccr_retrieve.py -q Failed before test collection while building the local editable package: esaxx-rs build failed with fatal error: 'cstdint' file not found. ``` ## Real Behavior Proof - Environment: GitHub Actions CI on PR #1451 plus local macOS worktree `fix/1450-ccr-streaming-retrieve`. - Exact command / steps: CI ran lint, type checking, build, unit-test shards, dashboard tests, extras tests, and e2e jobs; locally ran syntax checks and the SSE conversion regression tests. - Observed result: CI passed 20 checks with 0 failures; local syntax checks passed; `tests/test_sse_thinking_blocks.py` passed with 6 tests. - Not tested: the new proxy-level regression test was not run locally because the local native extension build fails in `esaxx-rs` before proxy tests can collect; it is included in the CI-tested suite. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes - Scope: this handles the direct Anthropic-compatible HTTP `/v1/messages` path. The configured Bedrock/backend streaming path does not share this CCR continuation machinery in this PR. - Documentation, CHANGELOG, code-comment, and local-full-test checklist items are N/A for this narrow bug fix or not true locally. |
||
|
Rod Boev
|
c19347c310
|
fix(opencode): preserve custom OpenAI gateway paths (#1596)
## Description Custom OpenAI-compatible gateways mounted under provider-specific prefixes could miss Headroom's dedicated OpenAI compression routes when used through the OpenCode transport. A request such as `https://open.bigmodel.cn/api/coding/paas/v4/chat/completions` was replayed to the proxy at `/api/coding/paas/v4/chat/completions`, so the proxy selected catch-all passthrough instead of `/v1/chat/completions`. This change keeps the proxy-facing entrypoints stable on `/v1/chat/completions` and `/v1/responses` for OpenAI-compatible suffixes, while preserving the original upstream path in an internal header so the dedicated OpenAI handlers can reconstruct the real provider URL. Closes #1582 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Normalize opencode-routed OpenAI-compatible `/chat/completions` and `/responses` requests onto the proxy's stable `/v1/*` routes. - Preserve the original upstream pathname in an internal `x-headroom-original-path` signal for dedicated OpenAI handler reconstruction. - Reconstruct dedicated OpenAI upstream URLs from `x-headroom-base-url` plus the preserved path prefix, while preserving request query strings and rejecting non-HTTP base hints. - Keep nearby non-OpenAI paths such as `/base/v1/messages` on existing passthrough behavior. - Add focused transport and proxy regression coverage for prefixed gateway paths, invalid fallback cases, and internal-header stripping. ## Testing - [x] Transport regression tests pass (`npm --prefix plugins/opencode test -- src/transport.test.ts`) - [x] Proxy regression tests pass (`uv run pytest tests/test_proxy/test_openai_transport_path_prefix.py -q`) - [x] Linting passes (`uv run ruff check headroom/proxy/handlers/openai.py tests/test_proxy/test_openai_transport_path_prefix.py`) - [x] Type checking passes (`npm --prefix plugins/opencode run typecheck`) - [x] New tests added for the bugfix - [ ] Manual testing performed ### Test Output ```text npm --prefix plugins/opencode test -- src/transport.test.ts PASS, 11 tests passed. npm --prefix plugins/opencode run typecheck PASS uv run pytest tests/test_proxy/test_openai_transport_path_prefix.py -q PASS, 7 tests passed. uv run ruff check headroom/proxy/handlers/openai.py tests/test_proxy/test_openai_transport_path_prefix.py PASS, all checks passed. ``` ## Real Behavior Proof - Environment: Windows, Python 3.12 via `uv`, Node 18+, focused OpenCode transport and proxy handler tests. - Exact command / steps: on `origin/main`, copy the updated `plugins/opencode/src/transport.test.ts` into a base worktree and run `npm --prefix plugins/opencode test -- src/transport.test.ts`; on this branch, rerun that transport test plus `npm --prefix plugins/opencode run typecheck` and `uv run pytest tests/test_proxy/test_openai_transport_path_prefix.py -q`. - Observed result: the base worktree fails because prefixed `/chat/completions` and `/responses` requests still enter the proxy at their provider path, while this branch passes with `/v1/chat/completions` and `/v1/responses`, preserves `x-headroom-original-path`, reconstructs the provider-prefixed upstream URL and query string, falls back safely on invalid hints, and keeps nearby `/base/v1/messages` traffic on passthrough. - Not tested: full CI suite, live BigModel traffic, and generic catch-all passthrough compression. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes This completes the transport contract introduced in https://github.com/headroomlabs-ai/headroom/pull/1573 by keeping prefixed OpenAI-compatible traffic on Headroom's stable `/v1/*` surface while preserving the real upstream path for dedicated-handler reconstruction. https://github.com/headroomlabs-ai/headroom/pull/1367 is adjacent global proxy configuration work for direct deployments; this PR is the per-request OpenCode transport fix for custom upstream path prefixes. `CHANGELOG.md` is intentionally unchanged because this repo's release pipeline generates changelog entries from conventional commits. This stays scoped to `/chat/completions` and `/responses` suffixes. Generic catch-all passthrough compression remains separate from this bugfix slice. |
||
|
Omar Garcia
|
8e0dadfe02
|
fix: restore token-mode compression on frozen prefixes (#1489)
## Description Fixes token-mode compression for continued Claude Code turns with a frozen prefix when the client has not already supplied `headroom_retrieve`. The previous guard returned before request-side compression could run in token mode. This keeps the non-token safety behavior, but lets token mode use the existing marker-triggered CCR tool injection override so emitted markers stay redeemable. Closes #1487. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Let Anthropic token mode run request-side compression even when the client did not pre-register `headroom_retrieve`. - Kept the deferred-injection skip for cache-mode coverage. - Added a regression for the frozen-prefix token-mode path. - Updated `CHANGELOG.md` for the user-facing behavior change. ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ rtk uv run pytest -q tests/test_proxy/test_anthropic_ccr_deferred_injection.py 15 passed, 1 warning in 2.73s $ rtk uv run ruff check headroom/proxy/handlers/anthropic.py tests/test_proxy/test_anthropic_ccr_deferred_injection.py All checks passed! $ rtk uv run ruff format --check headroom/proxy/handlers/anthropic.py tests/test_proxy/test_anthropic_ccr_deferred_injection.py 2 files already formatted ``` ## Real Behavior Proof - Environment: macOS, Python 3.12.9, local FastAPI `TestClient`, Anthropic proxy path, `mode=token`, `ccr_inject_tool=True`, frozen prefix count = 1, no client-supplied `headroom_retrieve`. - Exact command / steps: ran a local `rtk uv run python` repro that builds `create_app(ProxyConfig(...))`, forces compression on the Anthropic path, simulates a frozen prefix, and posts `/v1/messages`. - Observed result: local `TestClient` request returned `STATUS=200`; token-mode frozen-prefix compression ran once with `FROZEN_MESSAGE_COUNT=1`; the forwarded message was the CCR marker; forwarded tools included `headroom_retrieve`. ```text STATUS= 200 FROZEN_MESSAGE_COUNT= 1 COMPRESSION_CALLS= 1 FORWARDED_MESSAGES= [{'role': 'user', 'content': '[100 items compressed to 10. Retrieve more: hash=abc123def456abc123def456]'}] FORWARDED_TOOLS= ['headroom_retrieve'] ``` - Not tested: live Claude Code session against a real Anthropic upstream, full repo-wide `uv run pytest`, and `mypy headroom`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas (N/A: no new hard-to-follow block needed) - [x] I have made corresponding changes to the documentation (N/A: changelog update covers this user-facing bug fix) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A; proxy behavior only. ## Additional Notes The pytest run still emits the existing Starlette/httpx deprecation warning from `fastapi.testclient`; this PR does not touch that dependency path. |
||
|
Lucas Santos
|
43494ff526
|
fix(proxy): stop re-compressing headroom_retrieve output and emitting unredeemable markers (#1323)
## Description Two related CCR problems that both end in unreadable content. The first one (#1077) is an infinite loop. Any tool output over ~500 bytes gets replaced with a `<<ccr:hash>>` marker, and you call `headroom_retrieve` to get the original back. But the proxy then compresses the *retrieve response too*, so what comes back is a brand new marker. Retrieve that one and you get another marker. The second one (#1006), the proxy makes two independent decisions per request: SmartCrusher compresses, and the `headroom_retrieve` tool gets injected. The injection is deferred when there's a frozen message prefix (`frozen_message_count > 0`), but compression keeps running anyway. So the agent receives `[... compressed to N. Retrieve more: hash=...]` markers with no `headroom_retrieve` tool to redeem them. For #1077, SmartCrusher now skips `headroom_retrieve` results. Before crushing a tool message (OpenAI `role=tool`) or tool-result block (Anthropic `type=tool_result`), it checks whether that tool id maps to the CCR tool, and if so leaves it alone. Retrieved content stays readable. For #1006, compression and injection are no longer decided in isolation. The injection decision is extracted into `should_inject_ccr_tool`, which the Anthropic handler calls: when injection was deferred because of a frozen prefix but compression just emitted new markers, it injects the tool anyway, so a marker is never handed to an agent that can't act on it. The existing session-sticky dedup means sessions that already have the tool don't get it re-injected and don't lose their cache. Closes #1077 Closes #1006 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/smart_crusher.py`: exempt `headroom_retrieve` results from compression on both the OpenAI `role=tool` and Anthropic `type=tool_result` paths. - `headroom/proxy/helpers.py`: add `should_inject_ccr_tool`, the deferral-plus-override decision the handler used to inline, so the #1006 behaviour is testable at the decision point. - `headroom/proxy/handlers/anthropic.py`: call `should_inject_ccr_tool` to couple injection with compression; rename the misleading `frozen_prefix=` log key to `frozen_message_count=`. - `tests/test_transforms/test_smart_crusher_ccr_retrieve_exemption.py` and `tests/test_proxy/test_ccr_frozen_prefix_coupling.py`: new tests; the frozen-prefix test now drives `should_inject_ccr_tool` so it would fail if the override were removed. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run --extra dev python -m pytest tests/test_proxy/test_ccr_frozen_prefix_coupling.py tests/test_transforms/test_smart_crusher_ccr_retrieve_exemption.py -q 5 passed, 1 skipped ruff: All checks passed! mypy: Success: no issues found ``` The SmartCrusher test skips locally because the Rust extension `.so` is built for a different OS, the same skip the existing SmartCrusher tests take locally. It runs in CI where the extension is built. ## Real Behavior Proof - Environment: macOS, Python 3.13, this branch. - Exact command / steps: `uv run --extra dev python -m pytest tests/test_proxy/test_ccr_frozen_prefix_coupling.py tests/test_transforms/test_smart_crusher_ccr_retrieve_exemption.py -q`. The frozen-prefix test calls `should_inject_ccr_tool` (the function the Anthropic handler now uses) with a frozen prefix and freshly emitted markers, then drives `apply_session_sticky_ccr_tool` end to end and asserts `headroom_retrieve` lands in the outbound tools. The exemption test runs a `headroom_retrieve` tool result through SmartCrusher on both the OpenAI and Anthropic shapes. - Observed result: 5 passed, 1 skipped. The retrieve tool is injected even under a frozen prefix once markers exist, and is not injected when no markers were emitted. Removing the handler override flips `should_inject_ccr_tool` and fails the test. - Not tested: a full live proxy session. The behaviours are covered at the decision, transform, and handler-call level by the new tests. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes This one touches compression gating, so it's worth a careful read on the injection coupling, that's the part where a wrong call would re-introduce data loss. 1. Tool results with no id mapping still compress, marked with `# ponytail:` comments. Only ids we can positively identify as the CCR tool are exempted. 2. The injection coupling keys off `injector.has_compressed_content`, so the tool only shows up when there's actually something to retrieve. --------- Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
Rod Boev
|
3be2526b76
|
fix(proxy): add an Anthropic buffered read-timeout override (#1331)
## Description Buffered Anthropic `/v1/messages` requests still use Headroom's generic 300-second read timeout, which can produce proxy-generated `502 ReadTimeout` errors on long turns. This adds a dedicated buffered Anthropic timeout, keeps it applied across CCR and memory continuations plus batch paths, and makes the direct server entrypoint enforce the same positive-integer contract as the Click CLI. Closes #1261. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Added `anthropic_buffered_request_timeout_seconds` for buffered Anthropic reads. - Routed `/v1/messages`, CCR continuation, memory continuation, batch create, batch passthrough, and batch results through that timeout. - Enforced the same positive-integer validation for `HEADROOM_ANTHROPIC_BUFFERED_REQUEST_TIMEOUT_SECONDS` and `--anthropic-buffered-request-timeout-seconds` in both startup paths. - Added focused regressions and updated `CHANGELOG.md`. ## Testing - [x] `uv run pytest tests/test_proxy/test_anthropic_buffered_timeout.py tests/test_cli_proxy_improvements.py::TestNewEnvVarWiring` - [x] `uv run ruff check .` - [x] `uv run ruff format . --check` ### Test Output ```text $ uv run pytest tests/test_proxy/test_anthropic_buffered_timeout.py tests/test_cli_proxy_improvements.py::TestNewEnvVarWiring 17 passed in 3.42s $ uv run ruff check . All checks passed! $ uv run ruff format . --check 966 files already formatted ``` ## Real Behavior Proof - Environment: local FastAPI `TestClient` with stubbed retry and HTTP client seams - Exact command / steps: run `uv run pytest tests/test_proxy/test_anthropic_buffered_timeout.py tests/test_cli_proxy_improvements.py::TestNewEnvVarWiring`; the tests build `ProxyConfig(request_timeout_seconds=7, connect_timeout_seconds=3, anthropic_buffered_request_timeout_seconds=19)`, drive `/v1/messages`, `/v1/messages/batches`, `/v1/messages/batches/{batch_id}/results`, a CCR continuation, and a memory continuation through `TestClient`, then verify `HEADROOM_ANTHROPIC_BUFFERED_REQUEST_TIMEOUT_SECONDS=0` falls back to `600`, `--anthropic-buffered-request-timeout-seconds 0` is rejected, and default proxy timeouts stay `read=300` and `write=300` - Observed result: buffered Anthropic paths use `httpx.Timeout(connect=3, read=19, write=7, pool=3)`, continuation requests stay on that same budget, invalid zero-valued startup config is rejected or ignored back to the default, and unrelated proxy timeout defaults stay unchanged - Not tested: live upstream Anthropic latency beyond the focused stubbed-timeout regression ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have added tests that prove the fix - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable |
||
|
Rod Boev
|
2cae13dd79
|
fix(ccr): skip Anthropic marker emission when tool injection is deferred (#1273)
## Description Anthropic request-side CCR can still compress a turn into retrieval markers after the frozen-prefix cache guard suppresses `headroom_retrieve` registration. That leaves the model with marker-only context it cannot redeem, so the proxy silently drops recoverable data on exactly the turns where cache preservation deferred tool injection. This change couples the Anthropic request-side CCR path to tool availability so a turn never emits retrieve-only markers without the retrieval tool, even when token mode or cache-mode prefix replay could otherwise reuse already-compressed marker text. Closes #1006 After a collaborator merged current `main` into this branch, CI also picked up unrelated offline-memory failures from the merged base. Those follow-up changes are test-only: they keep the offline Hugging Face cache lanes skipping cleanly instead of failing in memory tests that are outside the CCR runtime path. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Couple Anthropic request-side CCR compression to the same frozen-prefix guard that already defers `headroom_retrieve` registration. - Keep the existing cache-preservation behavior: frozen-prefix turns stop emitting CCR retrieval markers instead of forcing tool injection into the cached prefix. - Make the skip decision use the effective frozen prefix after token-mode reclamping, so turns that genuinely reclamp to zero still keep normal reversible CCR behavior. - Bypass cached marker reuse in both token mode and cache-mode prefix replay when tool injection is deferred. - Add focused regressions for the Anthropic request-path seams under this bug: - frozen-prefix turns do not emit marker-only payloads - unfrozen turns still keep normal reversible CCR behavior - token-mode reclamp back to zero still compresses normally - existing `headroom_retrieve` tools keep reversible CCR on frozen turns - cache-mode delta reuse and exact-prefix replay both forward original content when retrieval is unavailable - Add a `CHANGELOG.md` entry because the proxy's user-visible Anthropic CCR behavior changes. - Add a shared test skip helper for offline Hugging Face cache misses and apply it to the merged `main` memory tests that were failing only in the offline CI shards after the branch picked up current `main`. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_proxy/test_anthropic_ccr_deferred_injection.py`) - [x] Linting passes (`uv run ruff check . && uv run ruff format . --check`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text uv run pytest tests/test_proxy/test_anthropic_ccr_deferred_injection.py 14 passed, 1 warning in 34.19s uv run pytest tests/test_memory/test_skip_helpers.py tests/test_memory/test_embedder_mps_serialization.py::test_cpu_uses_shared_executor tests/test_memory/test_hierarchical.py::TestLocalEmbedder::test_embed_single tests/test_memory_bridge.py::TestMemoryBridgeImport::test_import_claude_code_memory tests/test_memory_handler_concurrent_init.py::test_real_localbackend_initializes_via_public_entrypoint tests/test_memory_system.py::TestLocalBackend::test_save_memory_basic -q 4 passed, 5 skipped, 11 warnings in 7.65s uv run ruff check . All checks passed! uv run ruff format . --check 968 files already formatted ``` ## Real Behavior Proof - Environment: local FastAPI `TestClient` for the Anthropic request path, plus Windows Python 3.12 offline-memory repros with `TRANSFORMERS_OFFLINE=1` - Exact command / steps: Run the focused CCR regression command and the offline-memory repro subset below on the merged branch state. - `uv run pytest tests/test_proxy/test_anthropic_ccr_deferred_injection.py` - `uv run pytest tests/test_memory/test_skip_helpers.py tests/test_memory/test_embedder_mps_serialization.py::test_cpu_uses_shared_executor tests/test_memory/test_hierarchical.py::TestLocalEmbedder::test_embed_single tests/test_memory_bridge.py::TestMemoryBridgeImport::test_import_claude_code_memory tests/test_memory_handler_concurrent_init.py::test_real_localbackend_initializes_via_public_entrypoint tests/test_memory_system.py::TestLocalBackend::test_save_memory_basic -q` - Scenario coverage from the CCR pytest command: - frozen prefix with deferred tool injection and cached marker text available in token mode - unfrozen turn with normal CCR marker emission - token mode where the tracked frozen prefix reclamps back to zero - frozen prefix where the client already supplied `headroom_retrieve` - cache-mode append-only delta reuse with a previously forwarded compressed prefix - cache-mode exact-prefix replay where the previous forwarded prefix already contained a marker - Scenario coverage from the offline-memory repro command: - direct local embedder startup on CPU with no cached HF model - hierarchical memory embedder startup with offline model cache missing - bridge import through `LocalBackend` - `MemoryHandler` public init warmup path - `LocalBackend` save path under the offline lane - Observed result: The CCR regression keeps marker-free forwarding on frozen turns without tool availability, and the merged-`main` offline-memory lanes now skip cleanly instead of failing unrelated CI shards. - frozen-prefix Anthropic turns without tool availability forward the original long transcript across both token-mode and cache-mode reuse paths, while unfrozen turns, reclamped token-mode turns, and frozen turns that already advertise `headroom_retrieve` keep the reversible CCR marker path - the merged-`main` offline-memory regressions now skip cleanly when the Hugging Face cache is unavailable instead of failing unrelated CI shards - Not tested: live Zed session cache-hit behavior, provider latency under real Anthropic upstreams, and online Hugging Face download lanes ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - Runtime scope is still intentionally narrow to Anthropic request-side CCR. OpenAI, Gemini, streaming, and response-side CCR behavior are unchanged. - The only non-CCR diff is the test-only offline-memory follow-up required after current `main` was merged into the branch. - The focused proxy pytest run still emits the Windows-local `StarletteDeprecationWarning` from `fastapi.testclient`'s `httpx` bridge. The offline-memory repro command also emits existing datetime deprecation warnings and a pytest teardown warning around skipped offline lanes; none of those warnings were introduced by the CCR runtime change. |
||
|
Zhenjia ZHOU
|
6c68ff4e9f
|
perf(compression): take large cold-start contexts off the synchronous kompress path (#1171) (#1298)
## Description On a cold-start large context, kompress (ModernBERT ONNX) runs **synchronously on the request thread** — ~200–300s for ~1M tokens. It blows the 30s compression budget, leaks a non-preemptible worker, and cascades (executor saturation → queue timeouts on healthy requests); on timeout the request is forwarded **uncompressed** after eating 30s. This adds four layered, **default-off, fail-open** mitigations so the request path is never blocked on ML compression. Closes #1171 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Phase 0 — size gate** (`HEADROOM_KOMPRESS_MAX_TOKENS`, default 50000): route oversized text away from ModernBERT (→ LogCompressor / TextCrusher / passthrough) at the single `_try_ml_compressor` boundary. - **Phase 1 — cooperative deadline** (`HEADROOM_COMPRESSION_DEADLINE_MS`, default 20000): any kompress run self-terminates at the next chunk boundary past the budget, keeping the unprocessed tail verbatim. - **Phase 2 — TextCrusher** (`HEADROOM_TEXT_CRUSHER`): a new **native Rust** extractive prose compressor in `crates/headroom-core/src/transforms/text_crusher/`, exposed via PyO3 as `headroom._core.TextCrusher` with a thin Python wrapper. It **reuses the shared `crate::relevance::BM25Scorer`** rather than reimplementing BM25, and ships record/replay parity fixtures (mirroring the SmartCrusher Rust-core + Python-shim pattern). - **Phase 3 — off-path compression** (`HEADROOM_BACKGROUND_COMPRESSION`): forward uncompressed immediately and compress in a per-process background drain; a byte-identical cache hit on a later turn means the request never blocks on ML. - Benchmark (`benchmarks/text_crusher_quality_eval.py`), CHANGELOG entry, and docstrings documenting the fail-open limits. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`, new modules) - [x] New tests added for new functionality - [ ] Manual testing performed (Phase 0/1 gate-fire + deadline observed on real traffic in earlier iterations; Phase 3 off-path is unit- + byte-identity-tested, not yet live-validated) ### Test Output ```text $ pytest tests/test_transforms/ tests/test_cache/ \ tests/test_proxy/test_background_compression.py tests/test_proxy/test_phase3_byte_identity.py -q 501 passed, 37 skipped in 40.33s $ cargo test -p headroom-core --lib text_crusher test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 834 filtered out $ ruff check <changed files> All checks passed! $ mypy headroom/proxy/background_compression.py headroom/transforms/text_crusher.py Success: no issues found in 2 source files ``` New coverage: size-gate incl. the strategy-dispatch funnel (KOMPRESS + TEXT); partial-run deadline (chunk-0 compressed + chunk-1 verbatim tail); BackgroundCompressor (dedup / queue-full / fail-open); Phase 3 byte-identity round-trip; TextCrusher unit + parity. ## Real Behavior Proof - Environment: macOS, local dev — `uv` venv, Rust `_core` built via `uv pip install -e .`. - Exact command / steps: the `pytest` / `cargo test` / `ruff` / `mypy` commands shown under Test Output; quality eval `python benchmarks/text_crusher_quality_eval.py /tmp/squad_dev.json`. - Observed result: 501 Python + 3 Rust tests pass; ruff + mypy clean on changed/new modules. Quality eval: TextCrusher keeps ~94% of buried SQuAD answers at 30% size vs ~36% truncate/random; self-contained speed run ~333k words in ~76ms (one O(n) pass) — sub-second where ModernBERT takes minutes (fast-vs-slow contrast, not a same-input run). - Not tested: Phase 3 off-path on live traffic; multi-worker (per-process by design — see Additional Notes). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - **All four features are off by default and fail-open** — with the env flags unset the paths are no-ops for realistic inputs; on any error the request is forwarded (compressed if possible, else verbatim), never dropped. A full background queue / duplicate key surfaces as `deferred:dropped`. - **Known limits (documented in `background_compression.py`):** Phase 3 is per-process, in-memory, and token-mode-only — these are **lost-savings, never lost-correctness**, and consistent with the project's existing per-process compression cache + sticky-session multi-worker model. The startup multi-worker warning now names off-path background compression. - Phase 2 reuses the existing BM25 scorer; reuse did not improve answer-retention over a Python prototype (query-awareness dominates) — its value is the Rust speed + repo-conventional Rust-core/Python-shim shape. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Tejas Chopra
|
bd55a426bc
|
fix(proxy): scope CORS to loopback + gate operator/content endpoints (#1226)
## Description Locks down the proxy's browser- and network-facing attack surface, which matters most under a `--host 0.0.0.0` bind (the Docker default). The wildcard CORS policy (`allow_origins=["*"]` + `allow_credentials=True`) let any web page the user had open read the proxy's content endpoints — `/v1/retrieve` returns raw, uncompressed tool outputs (source, secrets) — via a cross-origin fetch to `127.0.0.1` (CWE-346). Several operator endpoints additionally leaked sensitive data or allowed unauthenticated state mutation to any network-reachable client. This PR scopes CORS to loopback origins and extends the project's existing `require_loopback` trust boundary (already used for `/admin/*` and `/debug/*`) to the remaining exposed endpoints. Closes #863. Supersedes #864 and #758 — see "Additional Notes". ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **CORS**: replaced `allow_origins=["*"]` + `allow_credentials=True` with a port-agnostic loopback origin regex (`https?://(localhost|127\.0\.0\.1|\[::1\])(:\d+)?`), `allow_credentials=False`, and methods/headers narrowed to `GET/POST` + `Content-Type/Authorization`. `HEADROOM_CORS_ORIGINS` (comma-separated) pins an explicit allowlist for Docker/remote dashboards; `*` opts back into the old wildcard. - **`/transformations/feed`** and **`/cache/clear`** gated behind `require_loopback` → 404 for non-loopback callers. The feed returns full prompt/completion bodies when `log_full_messages` is on; `/cache/clear` is unauthenticated state mutation (cache-eviction DoS / cost amplification). - **`/health`**: the `config` block (upstream API URLs, savings profile) is now served only to loopback callers; network callers get the `/readyz`-shape body (status/checks). `/livez` and `/readyz` remain unauthenticated probes for orchestration. - **`/stats`**: `recent_requests` / `request_logs` (per-request ids, providers, models, errors) and `config` are served only to loopback callers; aggregate counters stay public for remote monitoring. - Added `_request_is_loopback()` helper mirroring `require_loopback`'s two-gate check (loopback peer IP + loopback `Host` header, the DNS-rebinding defence) but degrading the payload instead of returning 404, so monitors keep the non-sensitive fields. - Tests: new `tests/test_proxy_cors.py` and `tests/test_proxy_loopback_gating.py`; updated 4 existing tests that assert the now-loopback-only data to use loopback clients. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check headroom/proxy/server.py tests/test_proxy_cors.py \ tests/test_proxy_loopback_gating.py tests/test_proxy_healthchecks.py \ tests/test_proxy_stats_recent_requests.py tests/test_proxy/test_transformations_feed.py All checks passed! $ mypy headroom --ignore-missing-imports Success: no issues found in 380 source files $ pytest tests/test_proxy_cors.py tests/test_proxy_loopback_gating.py \ tests/test_proxy/test_transformations_feed.py tests/test_proxy_healthchecks.py \ tests/test_proxy_stats_recent_requests.py tests/test_proxy_dashboard_stats_cache.py \ tests/test_proxy_compression_executor.py tests/test_header_isolation.py -q ======================== 82 passed, 1 skipped in 17.75s ======================== ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0), Python 3.12 venv; FastAPI `TestClient` driving the real `create_app()` ASGI app - Exact command / steps: issued requests as a non-loopback caller (`client.host=testclient`) vs a loopback caller (`base_url=http://127.0.0.1`, `client=("127.0.0.1", 9999)`), plus CORS preflights with varying `Origin` headers - Observed result: CORS — `http://evil.com` → no `access-control-allow-origin`; `http://localhost:8787` and `http://localhost:9000` → echoed (loopback allowed on any port); `access-control-allow-credentials` → absent. `/cache/clear` and `/transformations/feed` → 404 (network) / 200 (loopback). `/health` `config` block present for loopback only. `/stats` `recent_requests` present for loopback only, while the aggregate `tokens` block stays present for network callers. - Not tested: a live `headroom proxy` process bound on `0.0.0.0` reached from a second host (simulated via ASGI peer/Host instead); end-to-end browser DNS-rebinding (covered by the `Host`-header gate and its unit test) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — proxy/middleware change; behavior is captured under "Real Behavior Proof". ## Additional Notes **Supersedes two stale PRs that target the same issue but have drifted from `main`:** - **#864** (`fix(proxy): scope CORS to localhost`, @gabiudrescu) — correct instinct and the source of the tighter `GET/POST` + `Content-Type/Authorization` scoping kept here, but it derived the allowlist from the `HEADROOM_PORT` env var (wrong when `--port` is passed as a CLI flag), carried ~40 lines of unrelated punctuation churn, and is ~125 commits behind `main`. The port-agnostic regex used here resolves the reviewer's port concern. - **#758** (`security: adversarial review`, @neogenix) — bundled these same application-layer fixes with a large CI/CD + Docker supply-chain pass. It is a ~160-commit-behind draft whose `server.py` no longer merges cleanly (`main` independently adopted the same `require_loopback` pattern). The application-layer fixes are rebased onto current `main` here; the CI/Docker/supply-chain hardening from #758 is still valuable and would be welcome as a separate, rebased PR. Thanks to @gabiudrescu and @neogenix for the original analysis (#863). **Deliberate scope / follow-ups (not in this PR):** - `/stats` aggregate counters and the basic `/health` body remain readable on a `0.0.0.0` bind by design, so remote monitoring keeps working. Full lock-down is a one-line `Depends(require_loopback)` each if preferred. - The `/v1/retrieve*` family stays network-reachable; it can't be loopback-gated without breaking legitimate remote/containerized agents and needs auth instead — tracked separately. - `ruff check .` is scoped to changed paths above because the dashboard HTML template trips ruff's `invalid-syntax` (a known repo false-positive); `mypy` is run over the full `headroom` package. |
||
|
Zhenjia ZHOU
|
e8fc8a0d18
|
feat(proxy): cc-switch reconciler — keep Headroom in the request path alongside cc-switch (#1030)
## Description [cc-switch](https://github.com/farion1231/cc-switch) is a desktop provider manager for Claude Code and other coding agents; when a Claude Code provider is selected, it writes that provider's endpoint and token into `~/.claude/settings.json`. This PR adds an opt-in reconciler so Headroom can stay in Claude Code's request path when cc-switch rewrites that file during provider switches. The reconciler captures third-party Anthropic-compatible upstream URLs, points Claude back at the local Headroom proxy, and leaves official/empty OAuth settings direct unless explicitly opted in. This update also hardens the watcher so rapid settings rewrites that share the same float-second mtime are still detected. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added an opt-in `HEADROOM_CC_SWITCH_RECONCILE=1` watcher for cc-switch direct-injection mode. - Added loopback-only `GET/PUT /admin/upstream` runtime upstream inspection and override endpoints. - Preserved token/model settings while rewriting only `env.ANTHROPIC_BASE_URL` back to the local Headroom proxy. - Switched reconciler change detection from float-second `st_mtime` to nanosecond `st_mtime_ns` so rapid provider switches are not missed. - Added pytest coverage for capture/rewrite behavior, official-provider defaults, route-official opt-in, loop safety, enabled flags, and the same-float-mtime provider-switch case. ## Testing - [ ] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text python -m pytest tests/test_proxy/test_cc_switch_reconciler.py 12 passed in 0.17s python -m ruff check . All checks passed! python -m mypy headroom Success: no issues found in 359 source files python -m pytest 7 failed, 5996 passed, 492 skipped, 5814 warnings in 396.41s ``` ## Real Behavior Proof - Environment: macOS, branch `feat/cc-switch-reconciler`, Python 3.13.3. - Exact command / steps: Ran the focused reconciler pytest file, full repository ruff check, full `mypy headroom`, and full pytest from the local PR branch. - Observed result: All 12 reconciler tests passed, including the rapid provider-switch case where two writes share the same float mtime but differ by nanoseconds. Full `ruff check .` and `mypy headroom` passed. Full pytest completed with 7 failures outside the cc-switch reconciler test file. - Not tested: Live cc-switch plus Claude Code end-to-end switch. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A. ## Additional Notes Documentation and CHANGELOG updates are not included in this PR. The reconciler remains opt-in and off by default. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Eyal Mizrachi
|
addebdb29c
|
feat(proxy): make COMPRESSION_TIMEOUT_SECONDS configurable via env (#946) (#991)
## Description The compression-pipeline timeout was hard-coded at 30s, so slow CPUs and long Claude Code conversations had no recourse. #946 asks to wire `HEADROOM_COMPRESSION_TIMEOUT_SECONDS` through. Refs #946. ## Type of Change - [x] New feature (non-breaking change that adds functionality) ## Changes Made - Read `HEADROOM_COMPRESSION_TIMEOUT_SECONDS` from the environment (float), falling back to 30 on an unparseable value. ## Testing - [x] Unit tests pass (`pytest`) - [x] New tests added for new functionality ### Test Output ```text $ pytest tests/test_proxy/test_compression_timeout_config.py -q 4 passed in 0.09s ``` ## Real Behavior Proof - Environment: Linux, Python 3.13, editable build of this branch - Exact command / steps: `HEADROOM_COMPRESSION_TIMEOUT_SECONDS=88 python -c "import headroom.proxy.helpers as h; print(h.COMPRESSION_TIMEOUT_SECONDS)"` - Observed result: prints `88.0` (default `30.0`; an unparseable value falls back to `30.0`) - Not tested: a live compression actually exceeding the configured timeout under real load ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
rongabbay
|
7edb27ab24
|
feat(proxy): compress AWS Bedrock InvokeModel requests via configurable upstream (#720)
## Description
Clients that speak **Bedrock to a local gateway** can't get proxy-level
compression. Claude Code launched with `CLAUDE_CODE_USE_BEDROCK=1` (and
any AWS SDK pointed at a custom endpoint) POSTs
`/model/{id}/invoke[-with-response-stream]` to
`AWS_ENDPOINT_URL_BEDROCK_RUNTIME`, never `/v1/messages`. Those requests
fell through the catch-all and were forwarded **verbatim — no
compression**.
`--backend bedrock` is the opposite direction: it accepts Anthropic
input and re-signs to AWS. It can't accept Bedrock-format input or
forward to a custom upstream. So the "client speaks Bedrock → local
re-signing gateway → AWS" topology (internal gateways, LiteLLM,
LocalStack; see #510) got nothing.
This adds a Bedrock InvokeModel passthrough that compresses the request
body with the **same** `anthropic_pipeline` used for `/v1/messages` —
the Bedrock InvokeModel body for Anthropic models *is* the Anthropic
Messages shape (`{anthropic_version, system, messages, max_tokens, …}`,
model in the URL), so there's no translation and no new compression
logic. The routes register **only** when `--bedrock-api-url` is set, so
default behavior is completely unchanged.
**Limitation (important):** rewriting the body invalidates the caller's
**SigV4** signature (it covers a hash of the body). Point
`--bedrock-api-url` at a gateway that re-signs or doesn't verify the
inbound signature (an internal gateway, LiteLLM, LocalStack, a corporate
Bedrock proxy) — **never raw AWS**, which would 403. For direct-to-AWS
compression, use `--backend bedrock` (which re-signs). The two are
complementary. This is documented in the flag help, the handler
docstring, the proxy docs, and the CHANGELOG.
Closes #734. Refs #510 (the Bedrock slice of the provider-agnostic
umbrella).
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- New `--bedrock-api-url` flag (env: `BEDROCK_TARGET_API_URL`). When
set, registers `POST /model/{id}/invoke` and `POST
/model/{id}/invoke-with-response-stream`.
- `BedrockHandlerMixin` compresses the request body via the existing
`anthropic_pipeline`, then forwards to the configured upstream,
preserving path/query.
- Responses forwarded byte-faithfully (non-streaming JSON and the
streaming AWS event-stream alike — neither is parsed or mutated, since
all compression is request-side).
- `{model_id:path}` captures inference-profile ids with
dots/colons/slashes (e.g.
`us.anthropic.claude-sonnet-4-5-20250929-v1:0`).
- Fail-open: a malformed body or compression error forwards verbatim
rather than erroring.
- Routes register only when the flag is set — default behavior
unchanged.
- Files: `headroom/proxy/handlers/bedrock.py` (new —
`BedrockHandlerMixin`); `headroom/providers/proxy_routes.py` (gated
route registration); `headroom/cli/proxy.py`,
`headroom/proxy/server.py`, `headroom/proxy/models.py` (flag + config
wiring); `docs/content/docs/proxy.mdx`, `CHANGELOG.md` (docs).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_proxy/test_bedrock_passthrough.py -q
.............. [100%]
14 passed in 12.46s
```
`tests/test_proxy/test_bedrock_passthrough.py` (14 tests) covers: route
gating (absent unless configured), body compression, non-message fields
preserved, inference-profile id capture + re-encoding, byte-faithful
streaming, fail-open on malformed body and on pipeline exceptions,
bypass when `optimize=False` and via the `x-headroom-bypass` header,
upstream connect failure surfacing as a 502, the content-length
regression, outcome recorded with `provider="bedrock"`, and
`BEDROCK_TARGET_API_URL` env wiring. `ruff check`/`format` clean.
## Real Behavior Proof
- Environment: macOS, Python 3.12; forked proxy on `:8788` with
`--bedrock-api-url` pointed at a local re-signing Bedrock gateway;
provider Anthropic Claude on Bedrock.
- Exact command / steps: `headroom proxy --port 8788 --bedrock-api-url
http://127.0.0.1:<gateway>`, then `curl -X POST
http://127.0.0.1:8788/model/claude-haiku-4-5/invoke --data
@bedrock_invoke.json` (a ~52k-token conversation with a large assistant
turn).
- Observed result: valid Claude response returned and the gateway
received the compressed body — proxy `/stats` reports `52,095 → 3,979
tokens` (92.4%, 48,116 removed), and the gateway's reported
`input_tokens: 3709` confirms the compressed body reached the model.
- Not tested: raw direct-to-AWS (out of scope by design — SigV4; use
`--backend bedrock`); non-Anthropic Bedrock model bodies (e.g.
Titan/Llama) — only the Anthropic Messages-shaped invoke body is
handled.
<details><summary>Proxy <code>/stats</code> output + content-length bug
note</summary>
```json
"compression": {
"requests_compressed": 1,
"avg_compression_pct": 92.4,
"best_detail": "52,095 → 3,979 tokens",
"total_tokens_removed": 48116
}
```
The first iteration of this proof surfaced a real bug — a shrunk body
still carried the inbound `Content-Length`, so httpx raised `Too little
data for declared Content-Length`. Fixed by dropping
`content-length`/`content-encoding` on the rewritten path so httpx
recomputes them; covered by a regression test.
</details>
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
`mypy headroom` is left unchecked above — type checking runs in the CI
matrix rather than locally on my side; the new code carries type hints
on all public functions. Design spec / feature request: #734.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
gglucass
|
2269e40bde
|
feat(proxy): log compressed messages alongside original request (#261)
## Description Expose the post-compression message list that was actually sent upstream as a new `compressed_messages` field on `RequestLog`, paired with the existing (now consistently pre-compression) `request_messages`. Consumers of `/transformations/feed` — dashboards and any downstream observability — can now diff the two sides of a compression to see exactly what the pipeline stripped, replaced, or kept. Turns an abstract "saved N tokens" into a legible before/after. Gated by the same `log_full_messages` flag as `request_messages` so the two sides stay in sync; it's pointless to store one without the other. Also fixes a latent correctness bug: today's `request_messages` field is inconsistent across the four `RequestLog` construction sites — sometimes it's the pre-compression snapshot, sometimes it's the mutated `body["messages"]` (which is the compressed list, because the proxy mutates `body` in place before the log call). After this change, `request_messages` always means pre-compression and `compressed_messages` always means what went upstream. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) Note on "breaking": strictly speaking this is a semantic correction of an inconsistently-populated field, not a schema break. The field name `request_messages` is unchanged and the JSON shape is unchanged; what changes is that the field now consistently holds the pre-compression list. Consumers that treated it as "whatever messages we have" continue to work. Consumers that depended on the accidental post-compression value (if any existed) would shift to `compressed_messages`. ## Changes Made - **`headroom/proxy/models.py`**: `RequestLog` gains `compressed_messages: list[dict] | None = None`. Doc comment explains it's paired with `request_messages` and gated by the same `log_full_messages` flag. - **`headroom/proxy/handlers/anthropic.py`** (2 sites — Bedrock non-streaming and main non-streaming): `request_messages` now consistently sources from `original_messages` (the pre-compression snapshot at line 724), `compressed_messages` sources from `body["messages"]` (the compressed list after in-place mutation at line 1189). Both gated symmetrically. - **`headroom/proxy/handlers/streaming.py`** (2 sites — main streaming in `_finalize_stream_response`, Bedrock streaming in `_stream_response_bedrock`): same treatment. `_stream_response_bedrock` gains a new `original_messages: list[dict] | None = None` parameter so it has access to the pre-compression snapshot; the sole caller in `anthropic.py` now threads it through. - **`headroom/proxy/server.py`**: `/transformations/feed` adds `compressed_messages` to the JSON payload alongside the existing `request_messages` / `response_content`. *Split into a separate preceding commit is a one-time EOL normalization to LF — the file blob in history carries CRLF but `.gitattributes` declares `*.py text eol=lf`, so any contributor editing `server.py` triggers the same whole-file renormalization. Separating the two commits keeps this feature commit's diff at a single line.* - **`headroom/proxy/request_logger.py`**: `compressed_messages` is stripped from the JSONL file log and from `get_recent()` alongside the existing `request_messages` / `response_content` stripping. `get_memory_stats()` also counts it toward the deque's byte budget. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .` and `ruff format --check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed (via the Headroom Desktop client that consumes `/transformations/feed` — confirmed both fields arrive and render) Test coverage added/extended: - `tests/test_proxy/test_request_logger.py` (new file): round-trip unit tests for `RequestLogger`. Confirms `get_recent` strips both sides (pre + post), `get_recent_with_messages` exposes both, and the JSONL file log drops both when `log_full_messages=False`. - `tests/test_proxy/test_transformations_feed.py`: extended to assert `compressed_messages` appears in the endpoint payload alongside `request_messages` / `response_content`. - `tests/test_proxy_streaming_request_logger.py`: existing include/omit tests updated to assert both sides populate when the flag is on and both are `None` when it's off. ## Test Output ``` $ uv run ruff check headroom tests All checks passed! $ uv run ruff format --check headroom tests 614 files already formatted $ uv run pytest tests/test_proxy/test_request_logger.py tests/test_proxy_streaming_request_logger.py tests/test_proxy/test_transformations_feed.py -v ... tests/test_proxy/test_request_logger.py::test_get_recent_strips_compressed_messages_alongside_request_and_response PASSED tests/test_proxy/test_request_logger.py::test_get_recent_with_messages_returns_compressed_messages PASSED tests/test_proxy/test_request_logger.py::test_jsonl_file_strips_both_sides_when_log_full_messages_disabled PASSED tests/test_proxy_streaming_request_logger.py::test_finalize_stream_response_logs_original_and_compressed_messages PASSED tests/test_proxy_streaming_request_logger.py::test_finalize_stream_response_omits_messages_when_log_full_messages_disabled PASSED tests/test_proxy/test_transformations_feed.py::test_transformations_feed_returns_messages PASSED ... ============================== 11 passed in 5.36s ============================== ``` ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas (the two-sided gating at each log site, the `_stream_response_bedrock` parameter addition, and the `get_memory_stats` accounting) - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes ### Non-Anthropic backends `handlers/openai.py` and `handlers/gemini.py` do not currently emit `RequestLog` entries at all — only Anthropic and the shared streaming paths do. This PR therefore only populates `compressed_messages` on Anthropic traffic (which is what `/transformations/feed` shows today). Wiring OpenAI and Gemini into `RequestLogger` end-to-end is a separate, larger gap worth its own PR. ### `server.py` EOL normalization The feature change in `server.py` is a single line. To keep the diff readable, the preceding commit is a whitespace-only `chore(proxy): normalize server.py to LF per .gitattributes` — the file blob was stored with CRLF terminators but `.gitattributes` declares `*.py text eol=lf`. Any contributor touching `server.py` triggers this renormalization; isolating it here keeps the feature commit reviewable. Happy to rebase / drop / reshape as preferred. ### Downstream desktop compatibility The Headroom Desktop client I work on now consumes `compressed_messages` and renders the pre/post pair side-by-side on the "Recent large compression" card. The desktop was updated to handle both shapes: proxies without the field render the legacy single "Request" block; proxies with the field render "Request (original, N tokens)" + "Request (compressed, M tokens)" where N/M come from `input_tokens_original` / `input_tokens_optimized`. No changes needed downstream if this PR lands. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
gglucass
|
8f374263d3
|
feat(transforms): attribute read_lifecycle + smart_crush tags (#249)
## What Enrich the `transforms_applied` tags emitted by `ReadLifecycleManager` and `SmartCrusher` so each tag carries the specific target it acted on, instead of being an opaque counter: - `read_lifecycle:<state>` -> `read_lifecycle:<state>:<file_path>` - `smart_crush:<n>` -> `smart_crush:<n>:<tool1,tool2,...>` (tool names resolved from the assistant's `tool_calls` / `tool_use` metadata; falls back to `smart_crush:<n>` when no name resolves) Downstream UIs can then show *what* a compression acted on (which file was a stale read, which tools had their output crushed), not just that it happened. ## Note on the rebase The original revision targeted `ToolCrusher` / `tool_crush:<n>`. That transform has since been retired and replaced by the Rust-backed `SmartCrusher` (which emits `smart_crush:<n>`), so the tool-name attribution moved to `smart_crusher.py`. The `read_lifecycle` half is unchanged. ## Response-header compatibility `x-headroom-transforms` is built as `",".join(transforms_applied)`. A tag containing a comma (tool-name lists; file paths) would make that header ambiguous to split back into tags. To keep the header backward compatible, `header_safe_transforms` (`headroom/proxy/cost.py`) collapses the enriched tags back to their legacy counter shape **for the header only** -- the full enriched detail still flows through the structured `transforms_applied` list (dashboards, request logs, activity feed). Applied at all three header sites (openai / anthropic / gemini handlers). Paths containing `:` survive in `transforms_applied` because consumers bound their split to 3 parts. ## Tests - `tests/test_transforms/test_read_lifecycle.py` -- OpenAI + Anthropic tag shape, colon-in-path preservation - `tests/test_transforms/test_smart_crusher_attribution.py` -- OpenAI + Anthropic tool-name resolution, dedup, no-name fallback, id/name-missing skips - `tests/test_proxy/test_header_safe_transforms.py` -- header normalization keeps the joined header unambiguous (incl. comma-in-path) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Ashish
|
8db5efc6f9
|
fix(anthropic): CCR exception must re-raise, not silently swallow (#838)
## Summary
- When `ccr_response_handler.handle_response` throws on the Anthropic
path, the old code logged a `WARNING` and continued — silently returning
the raw `headroom_retrieve` tool-call block to the client (compressed
content never retrieved, client sees an unknown internal tool)
- The OpenAI handler already does `logger.error + raise` (commit
`
|
||
|
ashishpatel26
|
e8ecd08829 | fix(codex): fail open for proxy compression timeout | ||
|
chopratejas
|
20dc1f28f3 |
fix(proxy): Strands MCP bundle + backend path fixes + Codex fail-closed protection
Three logically-related sets of proxy changes ship in this branch:
1. Strands integration on the Bedrock path (HeadroomBundle + 4 OpenAI
handler fixes + LiteLLM cache stats + dep pin)
2. /stats MCP aggregation (cross-process events log → proxy summary)
3. Codex compression-failure fail-closed (WS + HTTP /v1/responses)
== 1. Strands integration on the Bedrock path ==
* HeadroomBundle (headroom/integrations/strands/bundle.py): single-helper
MCP wiring for a Strands Agent — Headroom MCP server (headroom_compress
/ headroom_retrieve / headroom_stats) plus optional Serena MCP and
optional in-process compression hook. Constructor builds unstarted
MCPClient instances per server; Strands' Agent owns the subprocess
lifecycle. Default config: MCP enabled, Serena enabled, hook OFF
(proxy is the single source of truth for compression). User-side
integration is two lines in any Strands app.
* headroom/proxy/handlers/openai.py — backend path now:
- calls PrefixCacheTracker.update_from_response (was direct-OpenAI only)
- intercepts CCR headroom_retrieve tool_calls server-side, mirroring
the Anthropic handler pattern; NO silent fallback, re-raises on
CCR errors (per feedback_no_silent_fallbacks)
- works for both non-streaming and streaming paths
* headroom/proxy/handlers/streaming.py: _stream_openai_via_backend now
accepts prefix_tracker + optimized_messages, parses cache stats from
the SSE final-usage frame (cache_creation_input_tokens added to the
state machine), records CCR retrieve feedback via a new
_record_ccr_feedback_from_openai_sse helper. Streaming CCR intercept
is intentionally out of scope (mirrors Anthropic streaming behaviour).
* headroom/backends/litellm.py: send_openai_message response usage block
now carries cache_read_input_tokens / cache_creation_input_tokens
(Anthropic/Bedrock dialect) and prompt_tokens_details.cached_tokens
(OpenAI dialect). Backwards-compatible — cold-start callers see the
same 3-key shape; cache keys appear only when the underlying provider
returns them. Pinned by test_no_cache_fields_means_no_cache_keys.
* headroom/proxy/auth_mode.py: ("strands-agents/", "strands") added to
CLIENT_UA_MAP. Production callers should also set X-Client: strands
since the default openai-python UA carries no Strands signal.
* pyproject.toml: huggingface-hub>=1.5.0,<2.0 pinned in [ml] so a sibling
install (e.g. strands-agents) can't drag the version below the floor
transformers 5.x requires (otherwise Kompress silently goes
"unavailable").
== 2. /stats MCP aggregation ==
* headroom/proxy/cost.py: _aggregate_mcp_events() reads the cross-process
shared events file the Headroom MCP server already writes to and
surfaces summary.mcp with three new keys:
- compressions (count of headroom_compress invocations)
- tokens_removed (sum of input - output across those)
- retrievals (count of headroom_retrieve — the load-bearing
over-compression alarm; if it grows linearly
with turn count, lossy compressors are
dropping info the model actually needs)
Defensive on every axis — missing MCP SDK, missing file, malformed
events, read errors — never blocks /stats.
* examples/strands_bundle_demo.py: stats panel prints the new fields so
the demo shows the full proxy-HTTP + MCP-tool story in one view.
== 3. Codex compression-failure fail-closed protection ==
Reported by Camille (2026-05-21): Codex threads were locking with
"ran out of room in the model's context window" after Headroom's
compression timed out on an oversized response.create frame and
forwarded the original ~1.7 MB frame to the upstream, which then
rejected it. Codex's auto-compact heuristic gates on the upstream-
reported total_usage_tokens (which Headroom had been shrinking on
earlier turns), so its compaction never fired and the thread locked.
Validated against open Codex issues (CLI + Desktop share codex-rs/core):
* #16068 — confirms compaction gates on total_usage_tokens,
estimated_token_count is computed but only logged
* #19806 — confirms image token estimator unbounded, contributes to
the same ContextManager.get_total_token_usage → auto-compaction chain
* headroom/proxy/helpers.py: decide_compression_failure_action() with a
unit-tested decision matrix:
- asyncio.TimeoutError → refuse, always
- non-timeout failure + frame > 256 KiB (configurable) → refuse
- non-timeout failure + small frame → forward (legacy)
Operator escape hatches:
- HEADROOM_WS_FAIL_OPEN_ON_COMPRESSION_FAILURE=1 restores legacy
- HEADROOM_WS_COMPRESSION_FAIL_THRESHOLD_BYTES tunes the threshold
* headroom/proxy/handlers/openai.py (WS /v1/responses): consults the
helper after compression failure. On refuse: close client websocket
code 1009 with "headroom: compression <reason> — please compact
context and retry" reason; set termination_cause for the outer
lifecycle finally; return.
* headroom/proxy/handlers/openai.py (HTTP /v1/responses): same helper.
On refuse: raise HTTPException(413) with a structured error body so
FastAPI's HTTPException handler emits a clean 413. The existing
`except HTTPException: raise` guard in this handler already ensures
the 413 propagates without being swallowed by the 502 catch-all.
Anthropic /v1/messages NOT changed in this branch: no equivalent bug
report on Anthropic-protocol clients, Claude Code (Anthropic-owned)
handles context overflow via its own cache_control/ephemeral
primitives, and Cursor/Aider don't maintain the local-Y estimate the
Codex bug requires. Deferred until a real report lands; the patch is
a one-liner reusing the same helper.
== Tests + verification ==
* tests/test_backends/test_litellm_cache_stats.py — 3 tests pinning
cache-stat surfacing across Anthropic/OpenAI dialects + backwards-
compat for no-cache responses.
* tests/test_proxy/test_openai_backend_path.py — 5 tests (Bedrock cache
fields, OpenAI fallback shape, CCR intercept with provider="openai",
CCR re-raise on exception, streaming signature contract).
* tests/test_proxy/test_mcp_stats_aggregation.py — 5 tests pinning the
aggregator across compress+retrieve mixes, empty events, unknown event
types, missing token fields, and read failures.
* tests/test_proxy/test_compression_failure_action.py — 12 tests pinning
the fail-closed decision matrix (timeout always refuses, small
transient passes through, oversize refuses, env override variants,
custom threshold, invalid threshold falls back, 0/negative ignored).
* examples/strands_bedrock_demo.py — model_id bumped from deprecated
Claude 3 Haiku to Sonnet 4.5 (the deprecated model now errors on
account access).
* examples/strands_via_proxy_demo.py — proxy + Bedrock cache + streaming
smoke test.
* examples/strands_mcp_dispatch_test.py — pure MCP round-trip probe.
* examples/strands_bundle_demo.py — full Strands + HeadroomBundle E2E
demo (this is the shape a real Strands user copies into their app).
Full pytest: 5327 passed, 178 skipped. The previously-failing
test_core_operations.py::TestAddBatch::test_add_batch_basic passes now
that the huggingface-hub pin in pyproject.toml unblocks transformers
imports.
E2E verified live against AWS Bedrock (Sonnet 4.5):
* cache_write=10,438 on turn A → cache_read=10,438 on turn B
* streaming SSE final usage frame carries cache_read_input_tokens
* 78.7% reduction on a 50 KB JSON tool_result via SmartCrusher (
dispatched per-content-type by ContentRouter)
* Strands Agent + HeadroomBundle: model autonomously called
headroom_compress + headroom_retrieve via MCP; CompressionStore
round-trip succeeded; final answer correct.
|
||
|
Garm
|
084678df7c |
fix(proxy): strip cache_control before hashing turn_id
compute_turn_id hashed the raw message dicts, which meant the same user-text message produced a different hash on each call of one agent loop because clients (notably Claude Code) move the cache_control breakpoint to the newest message per call. The user-text block carries cache_control on call 1 and not on call 2, so the serialized prefix differs and the turn_id rolls over. Effect downstream: every API call becomes its own "turn" and any prompt-level aggregation (e.g. the Headroom desktop app's prompt all-time record) collapses to the largest single call, not the sum across the prompt. Add a small recursive normalization pass that strips cache_control from the hashed prefix and from list-shaped system prompts before hashing. Two new tests cover cache_control moving between calls on both the messages array and the system prompt. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Garm
|
58282bbc5e |
test(proxy): cover turn_id branches flagged by codecov
Patch coverage on helpers.py was 87% — 5 lines of compute_turn_id were untested. Add cases for: non-dict / non-user messages in the reverse scan, empty-string user content (should keep scanning), mixed text+tool_result content (agent-loop continuation, not a turn boundary), and system=None (hashes without the system segment). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Garm
|
e8835affb8 |
docs(changelog): record turn_id feature; fix import order in new test
Follow-up to
|
||
|
Garm
|
c4464c066f |
feat(proxy): emit turn_id linking agent-loop API calls from one user prompt
Adds compute_turn_id() helper that hashes (model, system, messages prefix up to the last user text message). An agent loop sends the same user-text prefix across every iteration plus a growing tool chain, so this id is stable across the turn but rolls over when the user sends a new prompt. Stamps the id onto RequestLog at all three call sites (anthropic handler bedrock + direct branches, and the streaming handler) and surfaces it as turn_id in /transformations/feed so downstream consumers can aggregate savings per user prompt rather than per API call. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
JerrettDavis
|
0aae886f77 |
feat: add live transformations feed to dashboard
- New /transformations/feed endpoint returning message diffs - Alpine.js drawer UI with virtual scrolling and auto-stream pause - Live Feed button hidden when log_full_messages=false - Added --log-messages CLI flag to enable full message logging - Backend stores request/response messages when enabled Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |