mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
7edb27ab24
## Description
Clients that speak **Bedrock to a local gateway** can't get proxy-level
compression. Claude Code launched with `CLAUDE_CODE_USE_BEDROCK=1` (and
any AWS SDK pointed at a custom endpoint) POSTs
`/model/{id}/invoke[-with-response-stream]` to
`AWS_ENDPOINT_URL_BEDROCK_RUNTIME`, never `/v1/messages`. Those requests
fell through the catch-all and were forwarded **verbatim — no
compression**.
`--backend bedrock` is the opposite direction: it accepts Anthropic
input and re-signs to AWS. It can't accept Bedrock-format input or
forward to a custom upstream. So the "client speaks Bedrock → local
re-signing gateway → AWS" topology (internal gateways, LiteLLM,
LocalStack; see #510) got nothing.
This adds a Bedrock InvokeModel passthrough that compresses the request
body with the **same** `anthropic_pipeline` used for `/v1/messages` —
the Bedrock InvokeModel body for Anthropic models *is* the Anthropic
Messages shape (`{anthropic_version, system, messages, max_tokens, …}`,
model in the URL), so there's no translation and no new compression
logic. The routes register **only** when `--bedrock-api-url` is set, so
default behavior is completely unchanged.
**Limitation (important):** rewriting the body invalidates the caller's
**SigV4** signature (it covers a hash of the body). Point
`--bedrock-api-url` at a gateway that re-signs or doesn't verify the
inbound signature (an internal gateway, LiteLLM, LocalStack, a corporate
Bedrock proxy) — **never raw AWS**, which would 403. For direct-to-AWS
compression, use `--backend bedrock` (which re-signs). The two are
complementary. This is documented in the flag help, the handler
docstring, the proxy docs, and the CHANGELOG.
Closes #734. Refs #510 (the Bedrock slice of the provider-agnostic
umbrella).
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- New `--bedrock-api-url` flag (env: `BEDROCK_TARGET_API_URL`). When
set, registers `POST /model/{id}/invoke` and `POST
/model/{id}/invoke-with-response-stream`.
- `BedrockHandlerMixin` compresses the request body via the existing
`anthropic_pipeline`, then forwards to the configured upstream,
preserving path/query.
- Responses forwarded byte-faithfully (non-streaming JSON and the
streaming AWS event-stream alike — neither is parsed or mutated, since
all compression is request-side).
- `{model_id:path}` captures inference-profile ids with
dots/colons/slashes (e.g.
`us.anthropic.claude-sonnet-4-5-20250929-v1:0`).
- Fail-open: a malformed body or compression error forwards verbatim
rather than erroring.
- Routes register only when the flag is set — default behavior
unchanged.
- Files: `headroom/proxy/handlers/bedrock.py` (new —
`BedrockHandlerMixin`); `headroom/providers/proxy_routes.py` (gated
route registration); `headroom/cli/proxy.py`,
`headroom/proxy/server.py`, `headroom/proxy/models.py` (flag + config
wiring); `docs/content/docs/proxy.mdx`, `CHANGELOG.md` (docs).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_proxy/test_bedrock_passthrough.py -q
.............. [100%]
14 passed in 12.46s
```
`tests/test_proxy/test_bedrock_passthrough.py` (14 tests) covers: route
gating (absent unless configured), body compression, non-message fields
preserved, inference-profile id capture + re-encoding, byte-faithful
streaming, fail-open on malformed body and on pipeline exceptions,
bypass when `optimize=False` and via the `x-headroom-bypass` header,
upstream connect failure surfacing as a 502, the content-length
regression, outcome recorded with `provider="bedrock"`, and
`BEDROCK_TARGET_API_URL` env wiring. `ruff check`/`format` clean.
## Real Behavior Proof
- Environment: macOS, Python 3.12; forked proxy on `:8788` with
`--bedrock-api-url` pointed at a local re-signing Bedrock gateway;
provider Anthropic Claude on Bedrock.
- Exact command / steps: `headroom proxy --port 8788 --bedrock-api-url
http://127.0.0.1:<gateway>`, then `curl -X POST
http://127.0.0.1:8788/model/claude-haiku-4-5/invoke --data
@bedrock_invoke.json` (a ~52k-token conversation with a large assistant
turn).
- Observed result: valid Claude response returned and the gateway
received the compressed body — proxy `/stats` reports `52,095 → 3,979
tokens` (92.4%, 48,116 removed), and the gateway's reported
`input_tokens: 3709` confirms the compressed body reached the model.
- Not tested: raw direct-to-AWS (out of scope by design — SigV4; use
`--backend bedrock`); non-Anthropic Bedrock model bodies (e.g.
Titan/Llama) — only the Anthropic Messages-shaped invoke body is
handled.
<details><summary>Proxy <code>/stats</code> output + content-length bug
note</summary>
```json
"compression": {
"requests_compressed": 1,
"avg_compression_pct": 92.4,
"best_detail": "52,095 → 3,979 tokens",
"total_tokens_removed": 48116
}
```
The first iteration of this proof surfaced a real bug — a shrunk body
still carried the inbound `Content-Length`, so httpx raised `Too little
data for declared Content-Length`. Fixed by dropping
`content-length`/`content-encoding` on the rewritten path so httpx
recomputes them; covered by a regression test.
</details>
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
`mypy headroom` is left unchecked above — type checking runs in the CI
matrix rather than locally on my side; the new code carries type hints
on all public functions. Design spec / feature request: #734.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|---|---|---|
| .. | ||
| app | ||
| components | ||
| content/docs | ||
| overrides | ||
| screenshots | ||
| spec | ||
| .gitignore | ||
| auth-modes.md | ||
| bedrock.md | ||
| bun.lock | ||
| next.config.mjs | ||
| observability.md | ||
| package-lock.json | ||
| package.json | ||
| postcss.config.mjs | ||
| proxy.ts | ||
| README.md | ||
| rtk-architecture.md | ||
| source.config.ts | ||
| tsconfig.json | ||
docs
This is a Next.js application generated with Create Fumadocs.
Run development server:
npm run dev
# or
pnpm dev
# or
yarn dev
Open http://localhost:3000 with your browser to see the result.
Explore
In the project, you can see:
lib/source.ts: Code for content source adapter,loader()provides the interface to access your content.lib/layout.shared.tsx: Shared options for layouts, optional but preferred to keep.
| Route | Description |
|---|---|
app/(home) |
The route group for your landing page and other pages. |
app/docs |
The documentation layout and pages. |
app/api/search/route.ts |
The Route Handler for search. |
Fumadocs MDX
A source.config.ts config file has been included, you can customise different options like frontmatter schema.
Read the Introduction for further details.
Learn More
To learn more about Next.js and Fumadocs, take a look at the following resources:
- Next.js Documentation - learn about Next.js features and API.
- Learn Next.js - an interactive Next.js tutorial.
- Fumadocs - learn about Fumadocs