mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
4 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Abhay Singh
|
1843346283
|
fix(proxy/vertex): route google-publisher requests to the request region (#2069)
## Description
The Vertex `publisher=google` routes forward to a **fixed** upstream
host, ignoring the
request's region. In `headroom/providers/proxy_routes.py`,
`vertex_generate_content`,
`vertex_stream_generate_content`, and `vertex_count_tokens` all do:
```python
del api_version, project, location # <-- location discarded
if publisher == "google":
return await proxy.handle_gemini_generate_content(
request, model,
_api_target(proxy, "vertex"), # <-- single fixed host (default us-central1)
"vertex:google",
)
```
The sibling Anthropic `rawPredict` route already does this correctly —
it keeps `location` and
passes `_vertex_target_for_location(proxy, location)`, which derives the
regional host from the
path.
So a request to
`.../locations/europe-west1/publishers/google/models/gemini-2.0-flash:generateContent`
(with the proxy left at the default Vertex URL) is forwarded to
`https://us-central1-aiplatform.googleapis.com/...europe-west1...` — a
`us-central1` host serving a
`europe-west1` path. Vertex requires the host region to match the path
location, so it rejects the
request. `_vertex_target_for_location` and the region-aware Anthropic
routing landed together in
`
|
||
|
Matt Haitana
|
7d87aa2f1c
|
fix(bedrock): route ARNs via converse, named AWS profiles, and au. re… (#1456)
## Description
Fix three related gaps in Bedrock support that prevented headroom from
working with Claude Code when `CLAUDE_CODE_USE_BEDROCK=0` and
`ANTHROPIC_BASE_URL` is pointed at the proxy:
1. **ARN passthrough used the wrong LiteLLM route** — application
inference profile ARNs (e.g.
`arn:aws:bedrock:ap-southeast-2:<account>:application-inference-profile/<id>`)
were forwarded as `bedrock/<arn>`, which LiteLLM rejects with HTTP 400
"Try calling via converse route". Fixed to `bedrock/converse/<arn>`.
2. **Named AWS profile not forwarded to completion calls** —
`--bedrock-profile` was wired through the CLI → config →
`LiteLLMBackend.__init__` and used to fetch the model map at startup,
but never stored on `self`. All four `acompletion()` call sites
(`send_message`, `stream_message`, `send_openai_message`,
`stream_openai_message`) passed only `aws_region_name` — the
actual Bedrock calls used ambient credentials regardless of the flag.
Fixed by storing `self.profile_name` and passing `aws_profile_name=` to
every `acompletion()` call.
3. **`ap-southeast-2` used the wrong region prefix** — Australia should
use `au.` for cross-region inference profile IDs, not `apac.`. Added
`ap-southeast-2 → "au"` to `_BEDROCK_REGION_PREFIXES` and `"au."` to the
strip list in `_normalize_bedrock_profile_id`.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `backends/litellm.py`: route `arn:aws:` model IDs via
`bedrock/converse/<arn>` in `map_model_id`
- `backends/litellm.py`: store `profile_name` as `self.profile_name` in
`LiteLLMBackend.__init__`; pass `aws_profile_name=` to `acompletion()`
in all four call sites; use
`boto3.Session(profile_name=...)` for startup discovery; cache key is
`region:profile_name` to prevent cross-profile collisions
- `backends/litellm.py`: add `ap-southeast-2 → "au"` to
`_BEDROCK_REGION_PREFIXES`; add `"au."` to prefix strip list in
`_normalize_bedrock_profile_id`
- `providers/registry.py`: pass `profile_name=bedrock_profile` to
`LiteLLMBackend`
- `proxy/server.py`: pass `config.bedrock_profile` to
`create_proxy_backend`
- `docs/claude-code-bedrock-headroom.md`: remove false claim that ARNs
in `ANTHROPIC_DEFAULT_*_MODEL` bypass the proxy; fix troubleshooting
table
- `tests/test_bedrock_region.py`: update `test_arn_passthrough` to
expect `bedrock/converse/<arn>`; update cache key format; add
`test_profile_cache_isolation`,
`test_ap_southeast_2_uses_au_prefix`, and
`TestBedrockProfileForwardedToCompletion` (3 async tests asserting
`aws_profile_name` appears in `acompletion()` kwargs for named profiles
and is
absent for the no-profile case)
- `tests/test_provider_registry*.py`,
`test_vertex_claude_compression.py`: update `litellm_backend_cls` stubs
to accept `profile_name=None`
## Testing
- [x] Unit tests pass (`pytest`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_bedrock_region.py tests/test_provider_registry.py tests/test_provider_registry_extended.py \
-k "not test_fallback_when_boto3_import_fails and not test_fallback_when_api_call_fails and not test_successful_fetch" -q
collected 51 items / 3 deselected / 48 selected
tests/test_bedrock_region.py ...........................
tests/test_provider_registry.py ...........
tests/test_provider_registry_extended.py .......
48 passed, 3 deselected in 2.00s
```
Note: 3 deselected tests use patch("builtins.__import__") which hangs
under Python 3.13 — pre-existing issue unrelated to these changes.
## Real Behavior Proof
- Environment: macOS, Python 3.13, Claude Code with
`CLAUDE_CODE_USE_BEDROCK=0`, `ANTHROPIC_BASE_URL=http://127.0.0.1:8787`,
AWS ap-southeast-2, application inference profile ARNs in
`ANTHROPIC_DEFAULT_*_MODEL`
- Exact command / steps: `headroom proxy --port 8787 --backend bedrock
--region ap-southeast-2 --bedrock-profile "my-sso-profile"`
- Observed result: Requests routed correctly to
`bedrock/converse/arn:aws:bedrock:ap-southeast-2:...:application-inference-profile/<id>`
as confirmed in LiteLLM logs
- Not tested: EU/APAC region ARN passthrough (logic is identical);
non-SSO credential flows
```text
15:29:44 - LiteLLM:INFO: utils.py:4090 -
LiteLLM completion() model= converse/arn:aws:bedrock:ap-southeast-2:<account>:application-inference-profile/<id>; provider = bedrock
2026-06-26 15:29:44,322 - LiteLLM - INFO -
LiteLLM completion() model= converse/arn:aws:bedrock:ap-southeast-2:<account>:application-inference-profile/<id>; provider = bedrock
15:31:09 - LiteLLM:INFO: utils.py:4090 -
LiteLLM completion() model= converse/arn:aws:bedrock:ap-southeast-2:<account>:application-inference-profile/<id>; provider = bedrock
2026-06-26 15:31:09,928 - LiteLLM - INFO -
LiteLLM completion() model= converse/arn:aws:bedrock:ap-southeast-2:<account>:application-inference-profile/<id>; provider = bedrock
15:34:26 - LiteLLM:INFO: utils.py:4090 -
LiteLLM completion() model= converse/arn:aws:bedrock:ap-southeast-2:<account>:application-inference-profile/<id>; provider = bedrock
2026-06-26 15:34:26,811 - LiteLLM - INFO -
LiteLLM completion() model= converse/arn:aws:bedrock:ap-southeast-2:<account>:application-inference-profile/<id>; provider = bedrock
```
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
## Additional Notes
The 3 skipped tests (`test_fallback_when_boto3_import_fails`,
`test_fallback_when_api_call_fails`, `test_successful_fetch`) pre-exist
in the repo and use `patch("builtins.__import__")` which hangs under
Python 3.13. Not affected by these changes.
---------
Co-authored-by: Matt Haitana <mhaitana@costar.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
vladgrish
|
cff7247efd
|
fix: Vertex AI support for Claude Code with ANTHROPIC_VERTEX_BASE_URL (#1393)
## Description Fixes two bugs that prevent headroom from working with Claude Code in Vertex AI mode (`CLAUDE_CODE_USE_VERTEX=1` + `ANTHROPIC_VERTEX_BASE_URL`). Closes #1392 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Add `vertex_raw_predict_no_version` route for `/projects/{project}/locations/{location}/publishers/{publisher}/models/{model}:rawPredict` — Claude Code omits the `/v1` API version prefix when using `ANTHROPIC_VERTEX_BASE_URL`, causing all requests to fall through to the catch-all handler which forwards to OpenAI (404). The new handler prepends `/v1` to `request.scope["path"]` before calling `handle_anthropic_messages`. - Add `vertex_stream_raw_predict_no_version` route for `:streamRawPredict` — same fix for streaming. - In `_start_proxy` (`headroom/cli/wrap.py`): auto-set `HEADROOM_HTTP2=false` in the proxy subprocess env when `CLAUDE_CODE_USE_VERTEX` or `ANTHROPIC_VERTEX_PROJECT_ID` is detected. Vertex AI RST_STREAMs HTTP/2 connections (`StreamReset error_code:2`); HTTP/1.1 works correctly. ## Testing - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # Direct curl to patched proxy — versionless paths now routed correctly $ curl -s -w "\nHTTP:%{http_code}" -X POST \ "http://127.0.0.1:8787/projects/$GCP_PROJECT/locations/$CLOUD_ML_REGION/publishers/anthropic/models/claude-haiku-4-5@20251001:streamRawPredict" \ -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \ -H "anthropic-version: 2023-06-01" \ -d '{"anthropic_version":"vertex-2023-10-16","messages":[{"role":"user","content":"hi"}],"max_tokens":5,"stream":true}' event: message_start ... event: message_stop HTTP:200 $ curl -s -w "\nHTTP:%{http_code}" -X POST \ "http://127.0.0.1:8787/projects/$GCP_PROJECT/locations/$CLOUD_ML_REGION/publishers/anthropic/models/claude-sonnet-4-5@20250929:rawPredict" \ ... HTTP:200 # Before fix: both returned HTTP:404 (falling through to catch-all → OpenAI) # Before HTTP/2 fix: streamRawPredict returned StreamReset error_code:2 ``` ## Real Behavior Proof - Environment: macOS Apple Silicon, Python 3.14.3, headroom-ai 0.27.0 (patched locally), Claude Code 2.1.176, `CLAUDE_CODE_USE_VERTEX=1`, `CLOUD_ML_REGION=<region>`, `ANTHROPIC_VERTEX_PROJECT_ID=<project-id>` - Exact command / steps: `headroom wrap claude -- --model haiku -p "test"` and `headroom wrap claude -- --model sonnet -p "test"` - Observed result: Before fix — all models fail with "There's an issue with the selected model" (404 from catch-all routing to OpenAI). After fix — Claude Code connects and responds successfully via proxy (HTTP 200 from Vertex confirmed via curl). - Not tested: automated unit/integration tests (require live GCP credentials), non-Vertex backends (code paths untouched) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes The versionless route fix is the critical one — without it, 100% of Claude Code Vertex requests fail. The HTTP/2 fix is defense-in-depth; users can also set `HEADROOM_HTTP2=false` manually. Both fixes are non-breaking: existing `/v1/projects/...` routes are untouched, and the HTTP/2 change only applies when a Vertex env var is present. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Tejas Chopra
|
0e0591506c
|
feat(vertex): turnkey Claude Code + Vertex compression (+ fixes from the Vertex review) (#1113)
## Description
Makes **Claude Code on Google Vertex AI** actually receive Headroom's
prompt compression, and fixes the issues found in a deep review of the
Vertex path. The headline is a turnkey path: `headroom wrap claude`
(with the user's existing Vertex env) compresses each request and
forwards to Vertex using the client's own GCP ADC token — Headroom holds
no credentials.
_No linked issue — this addresses the internal Vertex code review
(`docs/proposals/vertex-claude-compression-review.md`)._
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `cli/wrap.py`: `wrap claude` detects `CLAUDE_CODE_USE_VERTEX=1` and
points Claude Code's Vertex endpoint at the proxy via
`ANTHROPIC_VERTEX_BASE_URL` (Claude Code ignores `ANTHROPIC_BASE_URL` in
Vertex mode). Client keeps its own GCP ADC auth. Adds
`--backend`/`--region` flags (parity with `wrap aider`).
- `providers/registry.py`: alias `litellm-vertex` → provider
`vertex_ai`. Previously it resolved to `"vertex"` (not in the registry)
→ generic pass-through with the wrong model prefix, dropped region, and
mishandled auth, even though all help text advertises `litellm-vertex`.
- `providers/proxy_routes.py`: derive the Vertex upstream host
per-request from the path's `locations/{location}` (handles `global`)
instead of pinning the configured fixed-region host; explicit
`--vertex-api-url` overrides still win.
- `docs/content/docs/claude-code-vertex.mdx` (+ nav): simple user guide
for running Claude Code on Vertex through Headroom.
- `docs/proposals/vertex-claude-compression-review.md`: the deep-review
findings these fixes address.
- `tests/test_vertex_claude_compression.py`: new tests.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_vertex_claude_compression.py -q
8 passed
$ python -m pytest tests/test_provider_proxy_routes.py tests/test_provider_registry.py tests/test_cli_proxy_env.py tests/test_backend_bugs.py -q
108 passed
$ ruff check headroom/providers/registry.py headroom/providers/proxy_routes.py headroom/cli/wrap.py tests/test_vertex_claude_compression.py
All checks passed!
$ mypy headroom/providers/registry.py headroom/providers/proxy_routes.py headroom/cli/wrap.py
Success: no issues found in 3 source files
```
## Real Behavior Proof
- Environment: local macOS, Python 3.12 `.venv`, branch
`feat/vertex-claude-compression`.
- Exact command / steps: ran the test suite above; verified in code that
the native `:rawPredict` route (publisher=anthropic) delegates to
`handle_anthropic_messages` with the region-derived host, that
`create_proxy_backend("litellm-vertex")` resolves to provider
`vertex_ai`, and that `wrap claude` sets `ANTHROPIC_VERTEX_BASE_URL`
when `CLAUDE_CODE_USE_VERTEX` is set.
- Observed result: 8 new tests + 108 existing tests pass; ruff + mypy
clean; the alias, region derivation (incl. `global` and explicit
override), and rawPredict→compression-handler delegation all behave as
asserted.
- Not tested: a live end-to-end run of Claude Code against a real Google
Vertex project (no GCP credentials available in this environment).
Recommend one smoke test against a live Vertex project before announcing
GA. The Rust `headroom-proxy` Vertex path is intentionally out of scope
(separate, unwired binary).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
- CHANGELOG.md left unchanged — releases are managed by release-please
from conventional commits.
- Follow-ups (not in this PR): wire or formally retire the Rust
`headroom-proxy` Vertex implementation; add a live-Vertex smoke test
once CI has GCP credentials.
|