headroom

akadmin/headroom

Fork 0

mirror of https://github.com/headroomlabs-ai/headroom.git synced 2026-08-27 14:17:10 -04:00

Commit graph

Author	SHA1	Message	Date
Ben Younes	9fde127534	fix(proxy): relocate stray system-role messages to the top-level system param (#765 ) (#1357 ) ## Description On requests large enough to trigger compression, the proxy emitted an upstream Anthropic request whose `messages[0]` had `role: "system"`. Anthropic's Messages API rejects any `system` role inside `messages[]`: ``` 400 invalid_request_error: "messages.0: use the top-level 'system' parameter for the initial system prompt" ``` The original request correctly carries its system prompt in the top-level `system` parameter; a compression/transform/pipeline step relocates the harness system block into `messages[0]`, so the request fails outright (intermittent only because it requires a context large enough to compress). This adds a wire-contract guard in the Anthropic forwarder: as the last step before sending upstream (after every transform, memory injection, tool sort, and pipeline extension, covering both the Bedrock and direct paths), any stray `role="system"` message is relocated out of `messages[]` and merged back into the top-level `system` parameter. Content order is preserved (existing system first, relocated content after) and block-level `cache_control` survives. The guard is a no-op on the common path (no system-role entry → inputs pass through unchanged). Closes #765 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/helpers.py`: new pure helper `relocate_system_messages_to_top_level(messages, system) -> (clean_messages, new_system, changed)` plus `_system_message_to_blocks`. Handles `system` being `None`/`str`/`list`, never drops content, preserves order and content blocks. - `headroom/proxy/handlers/anthropic.py`: invoke the guard just before the byte-faithful forward block; on relocation, update `body["messages"]`/`body["system"]`, mark the body mutated (`system_role_relocated`) so the byte-faithful forwarder re-serializes, and log a warning. - `tests/test_proxy_handler_helpers.py`: 3 unit tests (relocate stray system into top-level, append-to-existing-system order, no-op without a system entry). - `CHANGELOG.md`: Bug Fixes entry. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --extra dev python -m pytest tests/test_proxy_handler_helpers.py -q 29 passed in 4.95s # Regression on the forward path (byte-faithful forwarding, system-prompt immutability, cache stability): $ uv run --extra dev python -m pytest tests/test_proxy_handler_helpers.py tests/test_proxy_byte_faithful_forwarding.py tests/test_proxy_system_prompt_immutable.py tests/test_proxy_anthropic_cache_stability.py -q 90 passed, 15 warnings in 29.72s $ uv run ruff check headroom/proxy/helpers.py headroom/proxy/handlers/anthropic.py tests/test_proxy_handler_helpers.py All checks passed! $ uv run ruff format --check ... # 3 files already formatted $ uv run mypy headroom/proxy/helpers.py headroom/proxy/handlers/anthropic.py Success: no issues found in 2 source files ``` ## Test verification (RED → GREEN) The new tests exercise the guard directly and import the new helper at module top, so reverting the production fix makes them fail at collection. RED — production fix reverted (helper removed): ```text ImportError while importing test module 'tests/test_proxy_handler_helpers.py'. E ImportError: cannot import name 'relocate_system_messages_to_top_level' from 'headroom.proxy.helpers' =========================== 1 error in 0.41s =============================== ``` GREEN — production fix applied: ```text tests/test_proxy_handler_helpers.py ... [100%] ======================= 3 passed, 26 deselected in 1.50s ======================= ``` ## Real Behavior Proof - Environment: Python 3.13, `uv run` in this repo, branch `fix/issue-765`. - Exact command / steps: ran the guard on a body in the exact #765 failure shape — `system: None` and a `role="system"` harness block at `messages[0]`: - Observed result: ```text BEFORE: messages[0].role = system (Anthropic 400 trigger) changed = True AFTER roles = ['user', 'assistant'] system param = [{"type": "text", "text": "You are Claude Code. <system-reminder>...</system-reminder>"}] OK: no role=system in messages[]; system content preserved in top-level param ``` The illegal `role="system"` entry is removed from `messages[]` and its content lands in the top-level `system` parameter — exactly the body Anthropic accepts. - Not tested: a full live 250k+-token Claude Code session against the real Anthropic API (needs a large live context + API key); the fix is validated at the request-shaping boundary the 400 is raised on. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes The guard intentionally fires at the forwarder boundary rather than in any single transform: the issue's captures show the relocation can originate from the compression path, and pipeline extensions / hooks can also mutate `messages` late. Enforcing Anthropic's wire contract once, at the point the body is serialized upstream, fixes the 400 regardless of which step introduced the stray entry and matches the architecture invariant "never produce a `system`-role entry within `messages[]`". --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>	2026-08-13 11:52:09 -05:00
chopratejas	95fd6d8688	Add multi-provider batch API support with CCR post-processing This commit adds comprehensive batch API support for all three major LLM providers (Anthropic, OpenAI, Google/Gemini) with integrated CCR (Compress-Cache-Retrieve) functionality for asynchronous batch processing. ## Batch CCR Post-Processing Architecture When batch APIs are used, responses are processed asynchronously. If the model calls the CCR retrieval tool (`headroom_retrieve`) within a batch response, the system now handles this automatically: 1. Batch Submit: Request context (messages, tools, model) is stored in BatchContextStore keyed by batch_id 2. Batch Results: When results are retrieved, CCR tool calls are detected in the responses 3. Continuation: For each CCR tool call, the system executes local retrieval and makes a continuation API call to complete the response 4. Result Update: The batch result is updated with the complete response, transparent to the caller ## New Components - `headroom/ccr/batch_store.py`: TTL-based context storage for batch requests, enabling CCR retrieval during result processing - `headroom/ccr/batch_processor.py`: Processes batch results, detects CCR tool calls across all provider formats, executes continuations ## Provider Support ### Anthropic - POST /v1/messages/batches (create with compression) - GET /v1/messages/batches (list) - GET /v1/messages/batches/{id} (status) - GET /v1/messages/batches/{id}/results (with CCR post-processing) ### OpenAI - POST /v1/batches (create) - GET /v1/batches (list) - GET /v1/batches/{id} (status) - Batch file upload/download support ### Google/Gemini - Native API support: /v1beta/models/{model}:generateContent - Batch API: /v1beta/models/{model}:batchGenerateContent - Token counting: /v1beta/models/{model}:countTokens - OpenAI-compatible endpoint support ## CCR Enhancements - Added Google/Gemini format support to response_handler.py - Extended tool_injection.py with multiple marker patterns for different compressors (SmartCrusher, TextCompressor, LogCompressor, etc.) - Added Google functionCall/functionResponse handling ## Proxy Server Updates - Added Gemini native API handlers alongside OpenAI-compatible endpoints - Integrated batch context storage on submission - Added batch result processing with CCR continuation - Rate limiting and metrics tracking for all providers ## Test Coverage Added comprehensive integration tests (all skip gracefully without API keys): - test_proxy_batch_integration.py: Anthropic and OpenAI batch APIs - test_proxy_gemini_integration.py: Gemini via OpenAI-compatible endpoint - test_proxy_gemini_native_integration.py: Gemini native API - test_proxy_count_tokens_integration.py: Token counting endpoint - test_proxy_openai_responses_integration.py: OpenAI responses API - test_proxy_passthrough_integration.py: Passthrough endpoints ## Compression Results (Real API Testing) - Token savings: 83-98% on tool result content - CCR tool injection: Working across all providers - Model behavior: OpenAI gpt-4o-mini successfully called headroom_retrieve when presented with compressed data, proving end-to-end CCR functionality	2026-01-24 11:41:18 -08:00

Author

SHA1

Message

Date

Ben Younes

9fde127534

fix(proxy): relocate stray system-role messages to the top-level system param (#765 ) (#1357 )

## Description

On requests large enough to trigger compression, the proxy emitted an
upstream Anthropic request whose `messages[0]` had `role: "system"`.
Anthropic's Messages API rejects any `system` role inside `messages[]`:

```
400 invalid_request_error: "messages.0: use the top-level 'system' parameter for the initial system prompt"
```

The original request correctly carries its system prompt in the
top-level `system` parameter; a compression/transform/pipeline step
relocates the harness system block into `messages[0]`, so the request
fails outright (intermittent only because it requires a context large
enough to compress).

This adds a wire-contract guard in the Anthropic forwarder: as the
**last** step before sending upstream (after every transform, memory
injection, tool sort, and pipeline extension, covering both the Bedrock
and direct paths), any stray `role="system"` message is relocated out of
`messages[]` and merged back into the top-level `system` parameter.
Content order is preserved (existing system first, relocated content
after) and block-level `cache_control` survives. The guard is a no-op on
the common path (no system-role entry → inputs pass through unchanged).

Closes #765

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/helpers.py`: new pure helper
`relocate_system_messages_to_top_level(messages, system) ->
(clean_messages, new_system, changed)` plus `_system_message_to_blocks`.
Handles `system` being `None`/`str`/`list`, never drops content,
preserves order and content blocks.
- `headroom/proxy/handlers/anthropic.py`: invoke the guard just before
the byte-faithful forward block; on relocation, update
`body["messages"]`/`body["system"]`, mark the body mutated
(`system_role_relocated`) so the byte-faithful forwarder re-serializes,
and log a warning.
- `tests/test_proxy_handler_helpers.py`: 3 unit tests (relocate stray
system into top-level, append-to-existing-system order, no-op without a
system entry).
- `CHANGELOG.md`: Bug Fixes entry.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --extra dev python -m pytest tests/test_proxy_handler_helpers.py -q
29 passed in 4.95s

# Regression on the forward path (byte-faithful forwarding, system-prompt immutability, cache stability):
$ uv run --extra dev python -m pytest tests/test_proxy_handler_helpers.py tests/test_proxy_byte_faithful_forwarding.py tests/test_proxy_system_prompt_immutable.py tests/test_proxy_anthropic_cache_stability.py -q
90 passed, 15 warnings in 29.72s

$ uv run ruff check headroom/proxy/helpers.py headroom/proxy/handlers/anthropic.py tests/test_proxy_handler_helpers.py
All checks passed!

$ uv run ruff format --check ...   # 3 files already formatted
$ uv run mypy headroom/proxy/helpers.py headroom/proxy/handlers/anthropic.py
Success: no issues found in 2 source files
```

## Test verification (RED → GREEN)

The new tests exercise the guard directly and import the new helper at
module top, so reverting the production fix makes them fail at
collection.

**RED — production fix reverted (helper removed):**
```text
ImportError while importing test module 'tests/test_proxy_handler_helpers.py'.
E   ImportError: cannot import name 'relocate_system_messages_to_top_level' from 'headroom.proxy.helpers'
=========================== 1 error in 0.41s ===============================
```

**GREEN — production fix applied:**
```text
tests/test_proxy_handler_helpers.py ...                                  [100%]
======================= 3 passed, 26 deselected in 1.50s =======================
```

## Real Behavior Proof

- Environment: Python 3.13, `uv run` in this repo, branch
`fix/issue-765`.
- Exact command / steps: ran the guard on a body in the exact #765
failure shape — `system: None` and a `role="system"` harness block at
`messages[0]`:
- Observed result:
  ```text
  BEFORE: messages[0].role = system (Anthropic 400 trigger)
  changed       = True
  AFTER roles   = ['user', 'assistant']
system param = [{"type": "text", "text": "You are Claude Code.
<system-reminder>...</system-reminder>"}]
OK: no role=system in messages[]; system content preserved in top-level
param
  ```
The illegal `role="system"` entry is removed from `messages[]` and its
content lands in the top-level `system` parameter — exactly the body
Anthropic accepts.
- Not tested: a full live 250k+-token Claude Code session against the
real Anthropic API (needs a large live context + API key); the fix is
validated at the request-shaping boundary the 400 is raised on.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

The guard intentionally fires at the forwarder boundary rather than in
any single transform: the issue's captures show the relocation can
originate from the compression path, and pipeline extensions / hooks can
also mutate `messages` late. Enforcing Anthropic's wire contract once,
at the point the body is serialized upstream, fixes the 400 regardless
of which step introduced the stray entry and matches the architecture
invariant "never produce a `system`-role entry within `messages[]`".

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>

2026-08-13 11:52:09 -05:00

chopratejas

95fd6d8688

Add multi-provider batch API support with CCR post-processing

This commit adds comprehensive batch API support for all three major LLM
providers (Anthropic, OpenAI, Google/Gemini) with integrated CCR
(Compress-Cache-Retrieve) functionality for asynchronous batch processing.

## Batch CCR Post-Processing Architecture

When batch APIs are used, responses are processed asynchronously. If the
model calls the CCR retrieval tool (`headroom_retrieve`) within a batch
response, the system now handles this automatically:

1. **Batch Submit**: Request context (messages, tools, model) is stored
   in BatchContextStore keyed by batch_id
2. **Batch Results**: When results are retrieved, CCR tool calls are
   detected in the responses
3. **Continuation**: For each CCR tool call, the system executes local
   retrieval and makes a continuation API call to complete the response
4. **Result Update**: The batch result is updated with the complete
   response, transparent to the caller

## New Components

- `headroom/ccr/batch_store.py`: TTL-based context storage for batch
  requests, enabling CCR retrieval during result processing
- `headroom/ccr/batch_processor.py`: Processes batch results, detects
  CCR tool calls across all provider formats, executes continuations

## Provider Support

### Anthropic
- POST /v1/messages/batches (create with compression)
- GET /v1/messages/batches (list)
- GET /v1/messages/batches/{id} (status)
- GET /v1/messages/batches/{id}/results (with CCR post-processing)

### OpenAI
- POST /v1/batches (create)
- GET /v1/batches (list)
- GET /v1/batches/{id} (status)
- Batch file upload/download support

### Google/Gemini
- Native API support: /v1beta/models/{model}:generateContent
- Batch API: /v1beta/models/{model}:batchGenerateContent
- Token counting: /v1beta/models/{model}:countTokens
- OpenAI-compatible endpoint support

## CCR Enhancements

- Added Google/Gemini format support to response_handler.py
- Extended tool_injection.py with multiple marker patterns for different
  compressors (SmartCrusher, TextCompressor, LogCompressor, etc.)
- Added Google functionCall/functionResponse handling

## Proxy Server Updates

- Added Gemini native API handlers alongside OpenAI-compatible endpoints
- Integrated batch context storage on submission
- Added batch result processing with CCR continuation
- Rate limiting and metrics tracking for all providers

## Test Coverage

Added comprehensive integration tests (all skip gracefully without API keys):
- test_proxy_batch_integration.py: Anthropic and OpenAI batch APIs
- test_proxy_gemini_integration.py: Gemini via OpenAI-compatible endpoint
- test_proxy_gemini_native_integration.py: Gemini native API
- test_proxy_count_tokens_integration.py: Token counting endpoint
- test_proxy_openai_responses_integration.py: OpenAI responses API
- test_proxy_passthrough_integration.py: Passthrough endpoints

## Compression Results (Real API Testing)

- Token savings: 83-98% on tool result content
- CCR tool injection: Working across all providers
- Model behavior: OpenAI gpt-4o-mini successfully called headroom_retrieve
  when presented with compressed data, proving end-to-end CCR functionality

2026-01-24 11:41:18 -08:00

2 commits