akadmin/headroom
1
0
Fork 0
mirror of https://github.com/headroomlabs-ai/headroom.git synced 2026-08-27 14:17:10 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
2630 commits 410 branches 169 tags 142 MiB
Commit graph

10 commits

Author SHA1 Message Date
JD Davis
b3f443636d
fix(proxy): align signed-thinking wire accounting (#3015) 
## Description

Signed-thinking histories force byte-faithful passthrough because
re-serializing signed Anthropic blocks can invalidate their signatures.
Headroom correctly forwarded the original client bytes, but continued
reporting mutations, transforms, savings, response headers, and prefix
state from a different body that never reached the provider. Separately,
the final Anthropic guard hoisted every `role: system` message into the
top-level prompt, including valid mid-conversation system sections,
changing their semantics and destroying the cached prefix if that
mutation ever shipped.

This coupled fix makes downstream accounting use the actual wire body
whenever the signed-thinking lock discards edits, and narrows system
relocation to the current Anthropic model and placement contract.

Closes #2990
Closes #2991

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- Detects signed thinking in the original request as well as the mutated
body, so a transform cannot remove the block and accidentally bypass the
byte lock.
- Keeps the original-body signature probe best-effort under malformed,
recursive, and `MemoryError` conditions.
- Carries discarded mutation reasons through the streaming forwarder and
emits the existing structured warning on HTTP streaming paths too.
- When signed passthrough wins, resets message savings, tool-schema
savings, attribution ledgers, transform labels, response headers, and
prefix tracking to the original client wire body.
- Adds bounded public diagnostic tags naming/counting discarded mutation
reasons without exposing body content.
- Preserves valid mid-conversation system sections on currently
supported Claude models and official Anthropic, Bedrock, and parsed
`*.googleapis.com` routes; hostname-boundary validation rejects
lookalike and userinfo URLs.
- Preserves consecutive system sections and enforces documented
predecessor/successor placement rules.
- Continues relocating initial, invalidly placed, unsupported-model, and
conservative third-party-gateway system messages to avoid upstream 400s.
- Includes current `main`, including #2996, #2997, #2971, #3009, #3012,
and the MCP dependency cap.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
uv run pytest -q <wire/cache/savings/system focused suite>
379 passed

uv run pytest -q tests/test_proxy/test_anthropic_recount_and_reparse_safety.py tests/test_proxy_byte_faithful_forwarding.py tests/test_proxy_handler_helpers.py
99 passed

pytest tests scripts/tests --splits 4 --group N --tb=short -q
All four CI-shaped fresh-process groups passed locally after correcting the MemoryError regression; each completed in roughly 75-83 seconds.

Post-CodeQL correction: 91 focused tests passed; all four exact-head CI-shaped shards passed in roughly 82-95 seconds.

uv run ruff format --check .
1411 files already formatted
uv run ruff check .
All checks passed
uv run mypy headroom
Success: no issues found in 520 source files
```

## Real Behavior Proof

- Environment: macOS arm64, Python 3.13, branch rebased onto current
`main`.
- Exact command / steps: sent a signed-thinking request whose tool
schema is measurably compacted inside the handler, captured the exact
upstream bytes, wrapped the real outcome funnel, and inspected response
headers, aggregate metrics, attribution tags, transforms, and
prefix-tracker state. Exercised valid, consecutive, invalid, initial,
supported-model, and unsupported-model system placements.
- Observed result: upstream bytes remain byte-identical to the client;
discarded edits contribute zero tokens, zero tool savings, no transform
header, and no attribution while the prefix tracker stores the actual
wire messages. Valid mid-conversation system sections remain in place;
only out-of-contract sections relocate.
- Not tested: live paid Anthropic traffic with production credentials.
The placement/model contract was verified against the current official
documentation and wire behavior is covered with a byte-capturing
transport.

## Runtime Rollout Safety

- Rollout-managed feature(s): signed-thinking wire-truth accounting and
Anthropic mid-conversation system preservation.
- Minimum rollout channel: normal patch release after exact-head CI is
entirely green.
- Stable/default behavior changed: discarded mutations no longer inflate
savings; supported valid system sections are no longer hoisted into the
top-level prompt.
- Kill switch / disable path: no unsafe runtime override; human revert
restores the previous conservative relocation/accounting behavior.
- Unsafe override required: none.
- Qualification impact: all Python shards, byte-forwarding,
cache-prefix, outcome/savings, signed-thinking, Anthropic handler,
static, Docker, and security checks must remain green.
- Rollback path: fix forward through a human-reviewed corrective PR; no
persisted data or configuration migration is involved.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation — inline
wire-contract documentation; no separate guide is required
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable; proxy wire behavior and accounting only.

## Additional Notes

Human review only. No merge or auto-merge is configured. Current
provider contract reference:
https://platform.claude.com/docs/en/build-with-claude/mid-conversation-system-messages
 2026-08-16 20:44:38 -07:00
Parideboy
f1c34d336c
fix(proxy/anthropic): don't buffer a CCR stream when passthrough discards the stream flip (#2953) 
## Description

Fixes #2952. Since `dc163bcd` (#2254), a Claude Code session with
extended thinking on dies from the turn where the first signed
`thinking` block enters history:

```
API Error: API returned an empty or malformed response (HTTP 200) — check for a proxy or gateway intercepting the request
```

`select_outbound_body` forwards the client's original bytes whenever the
body carries a signed `thinking` / `redacted_thinking` block, and it
decides that **before** it looks at `body_mutated` — so every edit the
handler made is discarded. The buffered-CCR path depends on exactly such
an edit: it sets `body["stream"] = False` (`anthropic.py:3073-3078`) so
the reply arrives as one JSON document it can scan for
`headroom_retrieve` calls. With the flip discarded, upstream streams,
`response.json()` fails, SSE resynthesis is skipped, and the client is
handed a 200 it cannot read.

From the reporter's `proxy.log`, the turn that breaks — note
`body_bytes` equals the inbound `content_length` byte for byte, and
`source=passthrough` despite two recorded mutations:

```
CCR: stream:true request has headroom_retrieve available; using buffered stream:false upstream request
event=outbound_request forwarder=anthropic_messages body_bytes=132017 body_mutated=true
  mutation_reasons=structural_diff_vs_original,ccr_streaming_retrieve_buffered_non_stream source=passthrough
PERF ... msgs=6 tok_saved=3575 tool_saved=16064 tok_out=0 total_ms=9623
```

This is a different failure from #2251 (a 400 from Anthropic). Signed
thinking blocks still leave as original bytes here, so that fix is
untouched.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

Primary fix, `headroom/proxy/handlers/anthropic.py`:

- Gate `buffered_stream_ccr` on the outbound body actually being ours to
change, via a new `outbound_body_is_client_bytes()` predicate that
mirrors the passthrough branch. Those turns take the plain streaming
path instead, which is the coherent outcome: the injected retrieve tool
is itself a discarded mutation there, so the model was never going to
see it. One INFO line records the choice.

Three follow-on defenses, each of which independently kept the failure
alive or invisible:

- A non-JSON 200 on the buffered path now logs at WARNING (it was DEBUG,
which is why nothing in `proxy.log` looked wrong) and the upstream SSE
is relayed to the client verbatim, instead of falling through to a plain
`Response` that `_BufferedCCRResponse` can only turn into a bare `event:
error` once its 1 s keepalive has committed headers.
- The semantic cache no longer stores a body that did not parse as JSON,
and drops the stored `content-type` on the hit path. The cache key has
no `stream` component, so a cached SSE body was replayed to buffered
callers for the full 3600 s TTL — that replay is the request the
reporter actually saw the error on (a 2 ms `PERF ... transforms=none`
cache hit).
- `select_outbound_body` now reports the mutations passthrough discarded
(`dropped_mutations` / `dropped_mutation_reasons`), and
`log_outbound_request` logs them as
`event=outbound_body_mutations_dropped` at WARNING. Without it, `PERF`
reports savings and tool injections that never reached the wire and
nothing contradicts it.

`prepare_outbound_body_bytes` keeps its two-value shape, so existing
callers are unchanged.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

New `tests/test_ccr_buffered_stream_signed_thinking.py` covers the gate
(signed-thinking history takes the streaming path and still leaves as
`stream: true`; the same request without thinking blocks still takes the
buffered path with `stream: false`), the SSE relay, and the cache
guards. The relay case is parametrized on upstream latency because the
failure only reaches its worst form past the 1 s keepalive, where a
plain `Response` has no `body_iterator` left to forward.

Verified red before green — with the source changes stashed and the
tests in place:

```text
FAILED tests/test_ccr_buffered_stream_signed_thinking.py::test_signed_thinking_history_skips_the_buffered_ccr_path[True-True]
FAILED tests/test_ccr_buffered_stream_signed_thinking.py::test_buffered_ccr_relays_an_unexpected_sse_reply_and_does_not_cache_it[prompt]
FAILED tests/test_ccr_buffered_stream_signed_thinking.py::test_buffered_ccr_relays_an_unexpected_sse_reply_and_does_not_cache_it[past-keepalive]
FAILED tests/test_ccr_buffered_stream_signed_thinking.py::test_cache_hit_never_replays_a_foreign_content_type
========================= 4 failed, 1 passed in 8.88s =========================
```

With the fix applied:

```text
$ python -m pytest tests/test_proxy_byte_faithful_forwarding.py tests/test_ccr_buffered_stream_signed_thinking.py -q
55 passed in 11.40s

$ python -m pytest tests/test_compression_cache.py tests/test_ccr_inline_resolve_handlers.py \
    tests/test_ccr_sqlite_backend.py tests/test_anthropic_stage_timings.py \
    tests/test_backend_nonstreaming_cache_metrics.py tests/test_cache_mode_cold_start.py \
    tests/test_cache_breakpoint_diagnostics.py -q
87 passed, 1 skipped in 17.57s

$ python -m ruff check .
All checks passed!

$ python -m mypy headroom --ignore-missing-imports --python-version 3.13
Found 12 errors in 3 files (checked 517 source files)
# all 12 pre-existing in headroom/ccr/mcp_server.py and headroom/memory/mcp_server.py
# (local mcp package version); none in the four files this PR touches
```

## Real Behavior Proof

- Environment: headroom 0.35.0-dev source checkout, Python 3.13.11,
Windows 11, Claude Code CLI 2.1.228 against api.anthropic.com
(claude-opus-5), proxy run as `headroom proxy --port 8787 --memory
--code-aware --mode token`
- Exact command / steps: reproduced from the reporter's
`~/.headroom/logs/proxy.log` — request `hr_1786551079_000005` shows
`source=passthrough` with `body_bytes` identical to the inbound
`content_length` while `mutation_reasons` contains
`ccr_streaming_retrieve_buffered_non_stream`, then `tok_out=0`; request
`hr_1786551089_000006` is the 2 ms `transforms=none` semantic-cache hit
that returned the poisoned SSE body to a caller asking for JSON. The
preceding turn (`...000004`, no thinking block yet) was
`source=canonical` with `tok_out=341`. Then: pytest suites above, with
the red/green stash comparison
- Observed result: with the gate in place the thinking-bearing turn goes
down `_stream_response` and the forwarded body still says `"stream":
true`, matching the bytes passthrough will send; a buffered turn whose
upstream answers with SSE reaches the client as a stream and writes
nothing to the semantic cache; a cache entry can no longer hand a caller
a content-type from a differently-shaped request
- Not tested: a live end-to-end Claude Code session against Anthropic
with the patched proxy (the reproduction here is the reporter's proxy
log plus handler-level tests); `/v1/responses`, which has the same
`stream`-flip pattern (`openai.py:5479-5487`) but carries `input` rather
than `messages`, so `has_signed_thinking_blocks` never fires there and I
left it alone

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

Deliberately out of scope, worth a separate issue: option (b) of #2251 —
a canonical re-serialization that preserves signed blocks byte-for-byte
so compression and tool injection survive a thinking-bearing history
rather than being silently dropped. #2251 reports a 400 even on a no-op
re-encode whose only transform was `tool_search_deferral`, which hints
the signature covers `tools` too, so getting it wrong would re-break
every multi-turn thinking session. That needs validating against the
live API, not guessing. Until then the new WARNING at least makes the
dropped work visible.

Also unfixed by design: the savings accounting itself. A passthrough
turn still books `tok_saved` / `tool_saved` in `PERF` for bytes that
never shipped; correcting the numbers is a wider change than this bug
needs.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
 2026-08-13 11:46:55 -05:00
AxelRay
dc163bcd1c
fix(proxy): preserve signed Anthropic thinking blocks on outbound re-serialize (#2254) 
## Description

When multi-turn Anthropic requests include signed `thinking` or
`redacted_thinking` blocks in conversation history, the proxy
re-serializes the body through `serialize_body_canonical` whenever
`body_mutated` is true. That re-encode changes the byte representation
of signed blocks and upstream rejects the turn with 400 "blocks cannot
be modified".

This detects those content blocks and, when original request bytes are
available, forwards them byte-for-byte instead of re-encoding. That
matches the preferred option from the issue and mirrors the existing
Agno skip for thinking-bearing histories.

Closes #2251

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Add `has_signed_thinking_blocks()` in
`headroom/proxy/body_forwarding.py`
- Prefer original-byte passthrough in `select_outbound_body` when signed
thinking blocks are present and original bytes exist
- Unit tests for thinking and redacted_thinking passthrough,
missing-original canonical fallback, legacy override, and unchanged
non-thinking behavior

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
uv run pytest tests/test_proxy_byte_faithful_forwarding.py -q --tb=short
# 43 passed, 1 skipped

uv run ruff check / format on touched files
# passed
```

## Real Behavior Proof

- Environment: unit-level body forwarding with multi-turn
Anthropic-shaped payloads containing signed `thinking` /
`redacted_thinking` blocks
- Exact command / steps: focused pytest suite above
- Observed result: with `body_mutated=True` and original bytes present,
outbound source is `passthrough` and content equals original bytes;
without original bytes, behavior remains canonical; non-thinking mutated
bodies still use canonical
- Not tested: full `headroom wrap claude` multi-turn session against
Anthropic / Claude Code (no live Claude credentials here)

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A

## Additional Notes

- Requests without thinking blocks keep existing
passthrough/canonical/legacy selection
- When original bytes are unavailable, signed-thinking requests still
re-serialize

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
 2026-08-11 23:48:56 -05:00
Krishna Chaitanya
57e8dcb425
feat(proxy): add opt-in cost-aware model router (#1706) (#2205) 
## Description

Adds an optional, configuration driven model router (closes #1706). With
`HEADROOM_MODEL_ROUTER_ENABLED` set, ordered rules in
`HEADROOM_MODEL_ROUTES` rewrite the upstream model by estimated input
size and tool presence, complementary to content compression, for
example sending small, tool-free requests to a cheaper model. First
matching rule wins and every decision is logged with a reason. Off by
default so behavior is unchanged, skipped under
`x-headroom-bypass`/passthrough, and wired on the Anthropic
`/v1/messages` path. Malformed rules fail open, so a bad rule is skipped
rather than silently widened.

Closes #1706

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/model_router.py`: new `ModelRouter` component (ordered
rules, first-match decision with reason, fail-open env parsing,
tokenizer-free input estimate).
- `headroom/proxy/models.py` + `headroom/proxy/server.py`:
`ProxyConfig.model_router` field, env loader
(`HEADROOM_MODEL_ROUTER_ENABLED` / `HEADROOM_MODEL_ROUTES`), and proxy
wiring.
- `headroom/proxy/handlers/anthropic.py`: apply routing on
`/v1/messages` after the bypass gate, tracked as a body mutation.
- Tests, docs (`configuration.mdx`), and a CHANGELOG entry.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q tests/test_proxy/test_model_router.py tests/test_proxy/test_model_router_wiring.py
36 passed, 1 warning

$ ruff check .
All checks passed!

$ mypy headroom --ignore-missing-imports
Success: no issues found in 477 source files
```

## Real Behavior Proof

- Environment: local, macOS, Python 3.12, headroom `.venv`, upstream
mocked (no live provider call).
- Exact command / steps: enable the router via
`ProxyConfig(model_router=...)`, POST `/v1/messages` through
`TestClient` with a rule routing low-risk requests to a cheaper model;
repeat with header `x-headroom-bypass: true`.
- Observed result: the forwarded upstream body model is rewritten from
`claude-sonnet-4-6` to `claude-haiku-4-5` when the router is enabled,
and is left unchanged under bypass (see
`tests/test_proxy/test_model_router_wiring.py`).
- Not tested: the OpenAI and Gemini handler paths (this PR wires the
Anthropic path only); no live provider request (upstream is mocked).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

Happy to adjust the interface or scope (for example OpenAI and Gemini
parity) if you'd prefer a different shape.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: reneleonhardt <65483435+reneleonhardt@users.noreply.github.com>
 2026-07-15 19:58:17 +00:00
JD Davis
1f3696a3d0
refactor(proxy): isolate body forwarding policy (#1935) 
## Description
Extracts the byte-faithful Python forwarder policy out of the broad
proxy helpers module into a dedicated `headroom.proxy.body_forwarding`
domain. The new module owns the outbound body algebra: passthrough
original bytes, canonical JSON bytes for mutated bodies, and explicit
legacy JSON rollback mode.

Closes #

## Type of Change
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made
- Added `headroom.proxy.body_forwarding` with `OutboundBody`,
`OutboundBodySource`, `BodyMutationTracker`, mode resolution, canonical
serialization, and body selection helpers.
- Kept `headroom.proxy.helpers` compatibility exports for existing
callers.
- Updated Python forwarder call sites to import body-forwarding policy
from the dedicated module.
- Added tests for the new value object and compatibility exports.

## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output
```text
python -m pytest tests/test_proxy_byte_faithful_forwarding.py -q
============================= 40 passed in 3.61s =============================

python -m ruff check headroom/proxy/body_forwarding.py headroom/proxy/helpers.py headroom/proxy/server.py headroom/proxy/handlers/streaming.py headroom/proxy/handlers/openai.py headroom/proxy/handlers/anthropic.py headroom/proxy/handlers/batch.py tests/test_proxy_byte_faithful_forwarding.py
All checks passed!

python -m mypy headroom/proxy/body_forwarding.py
Success: no issues found in 1 source file

python -m compileall -q headroom\proxy\body_forwarding.py headroom\proxy\helpers.py headroom\proxy\server.py headroom\proxy\handlers\streaming.py headroom\proxy\handlers\openai.py headroom\proxy\handlers\anthropic.py headroom\proxy\handlers\batch.py
# no output; exited 0

git commit -m "refactor(proxy): isolate body forwarding policy"
Sync plugin versions.....................................................Passed
check for merge conflicts................................................Passed
ruff.....................................................................Passed
ruff-format..............................................................Passed
mypy.....................................................................Passed
```

## Real Behavior Proof
- Environment: Windows PowerShell, Python 3.13.13, branch
`jd/architecture-slice-2` based on `headroomlabs/main`.
- Exact command / steps: Ran the focused byte-faithful forwarding suite,
focused ruff command, targeted mypy, compileall over touched modules,
and commit hooks.
- Observed result: Forwarding behavior stayed byte-faithful;
compatibility exports remain intact; lint, formatting, and mypy passed.
- Not tested: Full pytest suite, live upstream proxy traffic, and manual
end-to-end clients.

## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)
N/A

## Additional Notes
Documentation and CHANGELOG updates are N/A for this internal refactor.
The full pytest suite was not run; validation is focused on the
body-forwarding domain and existing byte-faithful forwarding coverage.
 2026-07-10 17:27:26 -05:00
Rod Boev
0f553a8ebb
fix(proxy): preserve streaming passthrough beta headers (#1783) 
## Description

Anthropic-compatible custom upstreams can reject streaming passthrough
requests when Headroom expands the client's `anthropic-beta` header with
sticky session tokens. The request body is still forwarded
byte-faithfully, but the header no longer matches the direct request
that succeeds against the same upstream. This change keeps sticky beta
learning intact while preserving the direct client beta header for the
custom-upstream streaming passthrough path that owns the 503.

Closes #1724

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Preserves client `anthropic-beta` headers on Vertex
`:streamRawPredict` and custom Anthropic API URL streaming passthrough
requests.
- Keeps sticky beta tracking and adjacent sticky-header behavior for
non-hazard paths.
- Adds focused regression coverage that captures outgoing streaming
headers and preserves existing byte-faithful body checks.

## Testing

- [x] Unit tests pass (`uv run pytest
tests/test_proxy_byte_faithful_forwarding.py
tests/test_anthropic_beta_session_sticky.py -q`)
- [x] Linting passes (`uvx ruff==0.15.17 check .` and `uvx ruff==0.15.17
format --check .`)
- [ ] Type checking passes (`uv run mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
pytest base: exit_code=1, stdout excerpt: AssertionError: assert 'sticky-beta-2024-01-01,claude-code-20250219' == 'claude-code-20250219'
pytest head: exit_code=0, stdout excerpt: 64 passed, 1 warning in 3.57s
ruff: exit_code=0, stdout excerpt: All checks passed! / 1044 files already formatted
```

## Real Behavior Proof

- Environment: Windows, focused local proxy tests through the headless
runner.
- Exact command / steps: Pre-seed sticky beta state, send streaming
Vertex `:streamRawPredict` and `/v1/messages` requests through custom
upstream routing with `anthropic-beta: claude-code-20250219`, and
capture the outgoing request headers. Run the same focused pytest
command on the base checkout, then on the fixed checkout. Run pinned
Ruff 0.15.17 check and format validation against the final branch.
- Observed result: The base checkout expands the streaming
custom-upstream beta header, and the fixed checkout preserves the direct
client beta header for both streaming routes while adjacent
non-streaming custom-upstream requests still carry the sticky union.
- Not tested: The reporter's live MaaS upstream and the full test suite.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

Documentation is left unchecked because the fix preserves the existing
passthrough contract rather than adding a new user-facing option. The
changelog box is left unchecked because Headroom generates changelog
entries from conventional commits.
 2026-07-07 23:16:59 -05:00
Vinay Gupta
2a34a822f2
fix(proxy): preserve Responses passthrough bytes (#1598) 
## Description

Fixes the Python `/v1/responses` forwarding path for encoded Codex
Desktop requests.

When Headroom receives a compressed Responses request, the request body
is decoded before JSON parsing. The handler then forwarded a rewritten
JSON body while preserving the inbound `Content-Encoding` header, so
upstream could receive plain JSON bytes that were still labeled as
`zstd`/`gzip`. This change keeps the decoded original bytes for true
passthrough requests, strips stale entity headers, and marks Responses
body mutations so memory/compression paths still use canonical
serialization.

Closes #1542

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Switched `/v1/responses` parsing to keep the decoded original request
bytes.
- Stripped stale `content-encoding` and `transfer-encoding` headers
before forwarding decoded JSON bodies.
- Wired Responses streaming and non-streaming forwarding through the
existing byte-faithful passthrough controls.
- Marked Responses memory and compression body mutations so mutated
requests continue to serialize canonically.
- Added regression tests for gzip and zstd encoded Responses passthrough
bodies.

## Testing

- [x] Unit tests pass (`pytest`) — GitHub CI test shards passed
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`) — GitHub CI ran `mypy
headroom --ignore-missing-imports`
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ /tmp/headroom-1542-testenv/bin/python -m ruff check .
All checks passed!

$ /tmp/headroom-1542-testenv/bin/python -m ruff format --check .
1014 files already formatted!

$ HEADROOM_REQUIRE_RUST_CORE=false PYTHONPATH=/Users/vinaygupta/Desktop/git/headroom-fix-1542-zstd-passthrough /tmp/headroom-1542-testenv/bin/python - <<'PY'
# Injected an in-memory headroom._core import stub for this local checkout,
# then ran pytest.main(["tests/test_openai_codex_routing.py", "-q"])
PY
19 passed in 0.55s

$ HEADROOM_REQUIRE_RUST_CORE=false PYTHONPATH=/Users/vinaygupta/Desktop/git/headroom-fix-1542-zstd-passthrough /tmp/headroom-1542-testenv/bin/python - <<'PY'
# Injected an in-memory headroom._core import stub for this local checkout,
# then ran pytest.main(["tests/test_proxy_byte_faithful_forwarding.py", "-q"])
PY
35 passed, 1 warning in 1.33s

$ HEADROOM_REQUIRE_RUST_CORE=false PYTHONPATH=/Users/vinaygupta/Desktop/git/headroom-fix-1542-zstd-passthrough /tmp/headroom-1542-testenv/bin/python - <<'PY'
# Injected the same in-memory headroom._core import stub,
# then ran pytest.main(["tests/test_proxy_compression_headers.py", "-q"])
PY
10 passed in 0.05s

GitHub CI on `66507d98d4`:
- `lint`: SUCCESS, including `ruff check .`, `ruff format --check .`, and `mypy headroom --ignore-missing-imports` (`Success: no issues found in 404 source files`).
- `test (1)` through `test (4)`: SUCCESS.
- `test-agno`, `test-extras`, `test-dashboard-ui`, `docker-init-e2e`, `docker-wrap-e2e`, and `docker-native-e2e`: SUCCESS.
- PR Governance and Security checks: SUCCESS.

```

## Real Behavior Proof

- Environment: macOS local checkout, Python 3.13 throwaway test env.
- Exact command / steps: sent encoded `/v1/responses` test requests
through `TestClient` with a fake upstream transport capturing outbound
bytes and headers.
- Observed result: upstream received decoded JSON bytes, no stale
`content-encoding`, recomputed `content-length`, and logs reported
`body_mutated=false source=passthrough`.
- Not tested: full `pytest` and `mypy headroom` were not run locally. A
normal `uv run pytest ...` attempt was blocked by the local native build
error in `esaxx-rs` (`fatal error: 'cstdint' file not found`).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A

## Additional Notes

README/docs and CHANGELOG updates are not applicable for this narrow
proxy bug fix. GitHub CI is green on the rebased branch. Full local
pytest and local mypy were not run because the local native extension
build was blocked, so the full-suite signal comes from GitHub Actions.
 2026-06-30 14:37:47 -05:00
Vinay Gupta
a9322477e3
fix: preserve anthropic passthrough tool order (#1427) 
## Description

Preserves Anthropic `tools` order when Headroom is forwarding a
passthrough/no-optimize request. This fixes a Claude Code style
`tool_result` continuation failure against stricter Anthropic-compatible
upstreams that treat the client's original tool ordering as part of the
conversation state.

Closes #1417

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Preserve client-provided Anthropic `tools` order when `optimize=False`
or the request is explicitly in Headroom passthrough/bypass mode.
- Keep deterministic tool sorting for optimized requests where Headroom
may rewrite the body for cache stability.
- Avoid sorting batch-request tools before the no-optimize passthrough
branch.
- Add regression coverage for the Anthropic HTTP path to prove
no-optimize forwarding keeps `Read`, then `Bash` tool order.
- Update existing cache-stability and byte-faithful forwarding tests so
no-optimize/passthrough expects preserved client order while optimized
mode still proves deterministic sorting.

## Testing

- [x] Focused unit tests pass (`pytest` on touched proxy test files)
- [x] Linting passes (`ruff check` and `ruff format --check` on touched
files)
- [x] Type checking passes (`mypy headroom`)
- [x] New regression tests added
- [x] Manual testing performed

### Test Output

```text
$ uv run --no-project --python /opt/homebrew/opt/python@3.13/bin/python3.13 --with pytest --with pytest-asyncio --with anyio --with 'httpx[http2]' --with fastapi --with pydantic --with tiktoken --with click --with rich --with opentelemetry-api --with opentelemetry-sdk --with zstandard --with openai --with mcp --with uvicorn pytest tests/test_proxy_handler_helpers.py tests/test_anthropic_stage_timings.py tests/test_proxy_anthropic_cache_stability.py tests/test_proxy_byte_faithful_forwarding.py -q
============================= test session starts ==============================
platform darwin -- Python 3.13.11, pytest-9.1.1, pluggy-1.6.0
rootdir: /Users/vinaygupta/Desktop/git/headroom-fix-anthropic-tool-order
configfile: pyproject.toml
plugins: anyio-4.14.1, asyncio-1.4.0
asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 87 items

tests/test_proxy_handler_helpers.py ..........................           [ 29%]
tests/test_anthropic_stage_timings.py ....                               [ 34%]
tests/test_proxy_anthropic_cache_stability.py .........................  [ 63%]
tests/test_proxy_byte_faithful_forwarding.py ........................... [ 94%]
.....                                                                    [100%]

=============================== warnings summary ===============================
.../fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.

======================== 87 passed, 1 warning in 5.13s =========================

$ uv run --no-project --python /opt/homebrew/opt/python@3.13/bin/python3.13 --with ruff==0.15.17 ruff format --check headroom/proxy/handlers/anthropic.py tests/test_proxy_handler_helpers.py tests/test_anthropic_stage_timings.py tests/test_proxy_anthropic_cache_stability.py tests/test_proxy_byte_faithful_forwarding.py
5 files already formatted

$ uv run --no-project --python /opt/homebrew/opt/python@3.13/bin/python3.13 --with ruff==0.15.17 ruff check headroom/proxy/handlers/anthropic.py tests/test_proxy_handler_helpers.py tests/test_anthropic_stage_timings.py tests/test_proxy_anthropic_cache_stability.py tests/test_proxy_byte_faithful_forwarding.py
All checks passed!
```

## Real Behavior Proof

- Environment: macOS, Python 3.13.11, local fake Anthropic-compatible
upstream, local Headroom proxy launched with `--no-optimize --no-cache
--no-rate-limit --stateless`.
- Exact command / steps: ran a local reproduction harness that starts a
fake `/v1/messages` upstream and Headroom proxy, then sends a Claude
Code style two-turn flow: first assistant `Bash` `tool_use`, then user
`tool_result`.
- Observed result: after this patch, both direct and proxied flows
returned `200` for `first_tool_use` and `second_tool_result`. The fake
upstream log showed the proxied `tools` array remained `["Read",
"Bash"]` on both turns.

```text
DIRECT
  first_tool_use: 200
  second_tool_result: 200

PROXIED
  first_tool_use: 200
  second_tool_result: 200

UPSTREAM REQUEST LOG
  proxied first turn tools: ["Read", "Bash"]
  proxied tool_result turn tools: ["Read", "Bash"]
```

- Not tested: full `pytest`, full-repo `ruff check .`, `mypy headroom`,
or a live third-party Anthropic-compatible provider.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A

## Additional Notes

- This PR intentionally does not add documentation because it fixes
passthrough behavior rather than introducing a new user-facing option.
- The code-comment checklist item is left unchecked because the change
is covered by a small helper docstring and regression tests; no extra
inline comments seemed necessary.
- `CHANGELOG.md` is left unchanged because this is a narrowly scoped bug
fix.
- Local pytest collection for these proxy tests required a local
`headroom._core` extension symlink, which was removed before committing.
 2026-06-30 08:38:51 -05:00
Rod Boev
1f18d59809
fix(proxy): preserve byte-faithful Anthropic tool forwarding (#1222) 
## Description

Anthropic tool sorting was rewriting `tools` lists even when canonical
order was already present, which forced a mutation path that bypassed
byte-faithful forwarding and reduced prefix-cache hit stability on
repeated `/v1/messages` turns.

Closes #1042

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- changed Anthropic tool canonicalization so already-canonical tool
arrays are not rewritten in both single-request and batch paths
- preserved byte-faithful forwarding for no-op single-request canonical
tool-order requests by avoiding unnecessary `body["tools"]` reassignment
- kept canonicalization behavior intact for out-of-order tool arrays
- added one true regression proof for the PRE_SEND empty-tools path,
plus forward-coverage tests for canonical no-op and real-sort-mutation
request bodies in `test_proxy_byte_faithful_forwarding.py`
- updated `CHANGELOG.md` to document the prefix-cache fix

## Testing

- [x] Unit tests pass (`uv run pytest
tests/test_proxy_handler_helpers.py
tests/test_proxy_byte_faithful_forwarding.py -x -v`)
- [x] Linting passes (`uv run ruff check
headroom/proxy/handlers/anthropic.py tests/test_proxy_handler_helpers.py
tests/test_proxy_byte_faithful_forwarding.py`)
- [ ] Type checking passes (`uv run mypy headroom`) or explain N/A
truthfully
- [x] New tests added for new functionality when applicable
- [ ] Manual testing performed

### Test Output

```text
uv run pytest tests/test_proxy_handler_helpers.py tests/test_proxy_byte_faithful_forwarding.py -x -v
56 passed in 12.42s
uv run ruff check headroom/proxy/handlers/anthropic.py tests/test_proxy_handler_helpers.py tests/test_proxy_byte_faithful_forwarding.py
All checks passed!
uv run ruff format headroom/proxy/handlers/anthropic.py tests/test_proxy_handler_helpers.py tests/test_proxy_byte_faithful_forwarding.py --check
3 files already formatted
```

## Real Behavior Proof

- Environment: Windows, local proxy test environment, Anthropic
request-forwarding path
- Exact command / steps: post `/v1/messages` requests with canonical
tool order and then intentionally unsorted tool order through the
`TestClient` path with the no-optimize app variant
- Observed result: the PRE_SEND empty-tools runtime path now keeps
`body_mutated` false where base would mark the request mutated,
canonical-order tool payloads still preserve exact inbound bytes
end-to-end, and unsorted tool-order payloads are still canonicalized as
expected.
- Batch coverage: no dedicated runtime batch regression test was added;
batch no-op/mutation correctness is addressed through the same
compare-and-assign pattern on both batch canonical-sort call sites.
- Not tested: full-suite behavior outside the touched Anthropic
forwarding regression surface

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

Not applicable.

## Additional Notes

The important boundary is not whether tool arrays can be sorted. The
real contract is whether a no-op canonicalization should count as a
mutation. This change keeps canonicalization for real reorder cases and
restores byte-faithful forwarding for already-canonical requests.
 2026-06-21 10:07:29 -07:00
chopratejas
f0dcc02775 fix: A3 — byte-faithful Python forwarders; serialize canonical only when mutated 
Eliminates P0-2 universally. Every Python forwarder (server.py
`_retry_request`, handlers/streaming.py `_stream_response`,
handlers/openai.py `_ws_http_fallback`, handlers/batch.py `_batch_passthrough`
+ batch-create + Google batch passthrough, handlers/anthropic.py CCR
continuation + batch endpoint) now switches from `httpx ... json=body` to
`httpx ... content=raw_bytes`. The default httpx JSON encoder was
re-serializing every request with `, `/`: ` separators and `\\uXXXX` ASCII
escapes — collapsing Anthropic prompt-cache hit-rate.

Forwarder strategy:
  - unmutated body → forward `await request.body()` verbatim;
  - mutated body  → re-serialize once via the new
    `serialize_body_canonical(body) -> bytes` helper (compact separators,
    `ensure_ascii=False`, dict insertion order preserved).

`HEADROOM_PROXY_PYTHON_FORWARDER_MODE` env var configures the mode:
  - `byte_faithful` (default) — the new behavior;
  - `legacy_json_kwarg` — explicit operator opt-in for emergency rollback.
Documented in `docs/content/docs/configuration.mdx`. NOT a fallback —
unknown values raise loudly per build constraint #4.

`BodyMutationTracker` accompanies each request through the handler so
transform sites mark the tracker (`memory_injection`,
`image_compression`, `compression_*`, `batch_compression`,
`ccr_continuation`, etc.). At forwarder dispatch we additionally compare
the final body dict against the parsed original bytes as a structural
safety net — any silent mutation we missed still triggers canonical
re-serialization.

A2 follow-up: `handlers/openai.py:534-540` (Chat Completions memory
injection) was prepending a system message; replaced with
`append_text_to_latest_user_chat_message`, the OpenAI Chat Completions
analog of `_append_context_to_latest_non_frozen_user_turn`. The cache
hot zone (system messages) is now sacrosanct on /v1/chat/completions
too. Honors `HEADROOM_MEMORY_INJECTION_MODE=disabled`.

Structured logging: every forwarder emits an `event=outbound_request`
log line with `forwarder`, `path`, `body_bytes`, `body_mutated`,
`mutation_reasons`, `source` (passthrough|canonical|legacy),
`request_id`. Never logs Authorization or full body.

`_read_request_json` factored to share `_read_request_body_bytes` with
new `read_request_json_with_bytes` so the anthropic handler can capture
both the parsed dict and the original (decompressed) bytes.

Tests:
  - `tests/test_proxy_byte_faithful_forwarding.py` (28 tests):
    SHA-256 byte-equality on /v1/messages and streaming, unicode
    preservation, numeric precision, mutation-tracker invariants,
    canonical-serializer properties, legacy-mode rollback, OpenAI
    Chat memory routing.
  - Existing test mocks updated to accept the new `**kwargs` on
    `_retry_request` (no behavior change).
  - `tests/test_proxy_handlers_batch.py` updated to read the captured
    `content=` bytes (formerly `json=`).
  - One A2 test corrected (`test_anthropic_tool_sort_and_context_append_helpers`)
    to match the live-zone-tail semantics introduced by A2.

Constraints satisfied: configurable env var; no new regex / hardcodes;
no silent fallback (`legacy_json_kwarg` is operator opt-in);
performant (`prepare_outbound_body_bytes` is O(1) for passthrough);
elegant single-responsibility helpers; structured tracing logs.
 2026-05-02 09:02:10 -07:00