mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
c07da992dd
## The gap
Headroom picks its egress backend **once**, at startup:
`create_proxy_backend` returns a single `Backend` (or `None` for the
direct Anthropic path) and every request goes through it. That is the
right shape for *"run this whole proxy against Bedrock instead of
Anthropic"* and the wrong shape for *"this request is cheaper on a
different provider than the last one."*
`ModelRouter` already lets an extension change `body["model"]` per
request — but only within the protocol the request arrived in, because a
model id alone cannot move a request to another provider.
So an extension can currently **decide** something Headroom has no way
to **carry out**. This adds the missing half.
## The seam
An extension publishes a decision on the request state:
```python
request.state.headroom_route = SimpleNamespace(
model="moonshot/kimi-k2", # required
provider="moonshot", # optional; inferred from the model id if absent
reason="cheaper at this prefix length",
)
```
Headroom resolves a `LiteLLMBackend` for that provider — which is where
translation already lives — and serves **that one request** from it.
Nothing in core names any particular extension; the field is duck-typed,
so an extension does not import Headroom to talk to Headroom.
## Absent means unchanged
This is the property the tests are built around, and the reason this
should be safe to merge.
With nothing published, every path is what it was before. Advice that is
**absent, malformed, names an unknown provider, names a native provider,
or fails to build** all resolve to `self.anthropic_backend` — including
when that is `None`, which is the direct-API path and must survive. A
routing preference can never take traffic down.
## Coverage
| path | |
|---|---|
| `/v1/messages` | non-streaming + streaming |
| `/v1/chat/completions` | non-streaming + streaming |
| Responses API | untouched — does not use the backend abstraction |
Streaming is the one that matters. The resolver rewrites
`body["model"]`, so had `_stream_response_bedrock` kept reading
`self.anthropic_backend`, every streamed routed request would have sent
a foreign model id to Anthropic. Both streaming helpers now take an
optional `backend`, defaulting to the configured one.
## Details worth review
- **Validate the provider name before building.** `LiteLLMBackend`
accepts *any* provider string — the registry falls through to a generic
pass-through config — so a typo silently builds a backend that only
fails later, at request time, with an error pointing nowhere near the
typo. `_known_provider()` checks against `litellm.provider_list` first.
- **Cache per provider, and cache the failures too**, or a broken
provider name costs a construction attempt on every request. (Bedrock
construction calls out to AWS to enumerate inference profiles — it is
not free.)
- **`backend_owns_translation` now asks the per-request backend.** It
decides whether Headroom or the backend owns the `max_tokens` /
`max_completion_tokens` spelling; asking `self.anthropic_backend` would
answer "Headroom does" for a request about to be served by a translating
backend that does.
- **`_route_resolver` lives in `route_advice.py`, not on a handler
mixin.** Two mixins need it, and reaching across sibling mixins only
works by accident of how `HeadroomProxy` composes them.
## Tests
`tests/test_route_advice.py` — 20 tests, most of them asserting the
absent-means-unchanged property from a different angle.
Local runs: 20/20 on the new file; **1102 passed, 1 failed** on `-k
"openai or chat_completions or ccr"`, and **414 passed, 0 failed** on
`-k "stream or bedrock or route_advice"`. The single failure is
`test_realignment_live_multi_turn::test_ccr_marker_round_trip_live`,
which fails identically on this branch's merge-base — verified by
checking out `59314cff~1` and re-running it.
Note for anyone reproducing: `pytest-asyncio` is a declared dev
dependency but was missing from my venv, which made every `async def
test_` in the repo fail. Worth checking before diagnosing a large
failure count.
## Docs
`docs/content/docs/pipeline-extensions.mdx` gains a section on the
contract, next to the existing `x-headroom-base-url` one.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|---|---|---|
| .. | ||
| agent-orchestration.mdx | ||
| agno.mdx | ||
| anthropic-sdk.mdx | ||
| api-reference.mdx | ||
| architecture.mdx | ||
| autogen.mdx | ||
| benchmarks.mdx | ||
| cache-optimization.mdx | ||
| ccr.mdx | ||
| ci-cd-flows.mdx | ||
| claude-code-azure-foundry.mdx | ||
| claude-code-vertex.mdx | ||
| code-compression.mdx | ||
| codex-recovery.mdx | ||
| configuration.mdx | ||
| context-management.mdx | ||
| crewai.mdx | ||
| docker-install.mdx | ||
| errors.mdx | ||
| failure-learning.mdx | ||
| filesystem-contract.mdx | ||
| grok-build.mdx | ||
| how-compression-works.mdx | ||
| image-compression.mdx | ||
| index.mdx | ||
| installation.mdx | ||
| langchain.mdx | ||
| limitations.mdx | ||
| litellm.mdx | ||
| local-llm-prefill.mdx | ||
| mcp.mdx | ||
| memory.mdx | ||
| meta.json | ||
| metrics.mdx | ||
| openai-sdk.mdx | ||
| opencode.mdx | ||
| persistent-installs.mdx | ||
| pipeline-extensions.mdx | ||
| proxy.mdx | ||
| quickstart.mdx | ||
| releases.mdx | ||
| savings.mdx | ||
| shared-context.mdx | ||
| simulation.mdx | ||
| smart-crusher.mdx | ||
| strands.mdx | ||
| text-and-logs.mdx | ||
| troubleshooting.mdx | ||
| vercel-ai-sdk.mdx | ||
| vscode-claude-code.mdx | ||
| vscode-copilot.mdx | ||