mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
feat(proxy): add provider-only HTTP proxy (#1807)
## Description Adds provider-only HTTP proxy configuration for upstream LLM calls without setting process-wide proxy environment variables. `--http-proxy` and `HEADROOM_HTTP_PROXY` are scoped to the proxy server's provider HTTPX clients, and HTTP/2 is disabled for those clients when the proxy is set so HTTPS provider APIs can tunnel through CONNECT. Using process env vars such as `HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`, or `NO_PROXY` would also affect HTTPX, but those vars are inherited by tool executions, so this keeps proxy routing out of the global environment. Closes: N/A ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added `--http-proxy` with `HEADROOM_HTTP_PROXY` fallback. - Passed the proxy URL only into provider HTTPX clients. - Disabled provider HTTP/2 when the proxy is configured. - Preserved the new setting through direct server startup and multi-worker config serialization. - Documented the flag/env var and why global `HTTP_PROXY`-style vars are not suitable for provider-only routing. - Added an Unreleased changelog entry. - Added coverage for CLI/env wiring, worker serialization, and HTTPX client options. ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --frozen pytest tests/test_cli_proxy_improvements.py tests/test_proxy_scalability.py ============================== 72 passed in 5.95s ============================== $ uv run --frozen ruff check headroom/cli/proxy.py headroom/proxy/models.py headroom/proxy/server.py tests/test_cli_proxy_improvements.py tests/test_proxy_scalability.py All checks passed! $ uv run --frozen mypy headroom --ignore-missing-imports Success: no issues found in 406 source files $ env -u HTTP_PROXY -u http_proxy npm --prefix docs run types:check [MDX] generated files in 6.351916000000074ms Generating route types... [MDX] generated files in 5.813166999999794ms ✓ Types generated successfully $ git diff --check # no output ``` ## Real Behavior Proof - Environment: local provider setup that requires outbound LLM traffic through an HTTP proxy - Exact command / steps: ran focused pytest, Ruff, mypy, docs `types:check`, and `git diff --check` after rebasing the branch onto `origin/main`; reviewed the docs and changelog diffs; actively used the new proxy setting locally for a provider that requires proxied egress - Observed result: CLI/env/config tests passed; static checks passed; docs type generation passed; local provider traffic can be routed through the provider-only proxy setting without exporting global proxy variables to tool executions - Not tested: broad provider matrix across every supported upstream ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A. CLI/backend/docs update only. ## Additional Notes The branch keeps implementation, docs, changelog, and formatting changes in separate commits.
This commit is contained in:
Andrew McFague
GitHub
parent
838c5234a8
commit
ebe0a3bd7b
8 changed files with 127 additions and 21 deletions
|
|
@ -68,6 +68,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
|
||||
### Features
|
||||
|
||||
* **proxy:** add provider-only HTTP proxy routing via `--http-proxy` and
|
||||
`HEADROOM_HTTP_PROXY`. Upstream LLM provider calls can now use an HTTP proxy
|
||||
without setting process-wide `HTTP_PROXY`/`HTTPS_PROXY` variables that are
|
||||
inherited by tool executions; proxied provider clients use HTTP/1.1 so HTTPS
|
||||
provider APIs can tunnel through CONNECT.
|
||||
* **proxy:** add output shaping for OpenAI Responses traffic on `/v1/responses` HTTP requests and Codex WebSocket `response.create` frames, with stable output-savings holdout keys and counted WS token strata for the experiment.
|
||||
* **wrap:** `headroom wrap claude --1m` preserves the 1M context window. Behind a custom `ANTHROPIC_BASE_URL` (the proxy) Claude Code drops the `context-1m` beta header and caps the window at 200k for entitled subscription users; the opt-in flag sets `ANTHROPIC_MODEL=<opus>[1m]` on the launched process so the 1M window activates through Headroom. A model already selected via `ANTHROPIC_MODEL` is preserved (only the `[1m]` suffix is appended) ([#1158](https://github.com/chopratejas/headroom/issues/1158)).
|
||||
* **learn:** weight loops in `headroom learn`. A new loop detector (`headroom/learn/loops.py`) recognizes repeated tool-call patterns — including RTK re-fetch loops, where RTK's output truncation makes the agent re-run larger-limit variants of a *successful* command — collapses output-limit variants to one signature, measures the wasted tokens, surfaces loops as a highest-priority digest section, and weights loop guardrails above one-off rules by their measured waste. Previously loops had no special weight and a no-failure re-fetch loop was skipped entirely. Adds an RTK-loop eval (`benchmarks/rtk_loop_learn_eval.py`) that reproduces a loop, runs it through Learn, and asserts the generated guardrail ranks first and prevents re-triggering.
|
||||
|
|
|
|||
|
|
@ -228,6 +228,7 @@ headroom proxy --learn --min-evidence 3
|
|||
| `HEADROOM_MAX_CONNECTIONS` | Maximum upstream HTTP connections | `500` |
|
||||
| `HEADROOM_MAX_KEEPALIVE` | Maximum upstream keep-alive connections | `100` |
|
||||
| `HEADROOM_KEEPALIVE_EXPIRY` | Seconds an idle upstream keep-alive connection is kept open | `90` |
|
||||
| `HEADROOM_HTTP_PROXY` | HTTP proxy URL for upstream provider requests only; HTTPS provider APIs use CONNECT | -- |
|
||||
| `HEADROOM_BUDGET` | Daily budget limit in USD | -- |
|
||||
| `HEADROOM_TELEMETRY` | Set to `on` to opt in to anonymous telemetry | `off` (opt-in) |
|
||||
| `HEADROOM_STATELESS` | Set to `true` to disable filesystem writes | `false` |
|
||||
|
|
@ -252,6 +253,8 @@ headroom proxy --learn --min-evidence 3
|
|||
| `HEADROOM_BETA_HEADER_STICKY` | Controls per-session `anthropic-beta` / `OpenAI-Beta` re-echo. `enabled` (default): the proxy unions beta tokens across turns within a session — if the client sends a token in turn N and omits it in turn N+1, the proxy re-injects it to preserve prefix-cache stability. `disabled`: the client's value is forwarded verbatim with no accumulation. Any other value raises at request time. See [Session Beta Header Tracking](/docs/configuration#session-beta-header-tracking). | `enabled` |
|
||||
| `HEADROOM_BETA_TRACKER_MAX_SESSIONS` | LRU capacity of the in-memory session beta tracker. Once full, the oldest session entry is evicted. | `1000` |
|
||||
|
||||
For provider-only proxying, prefer `HEADROOM_HTTP_PROXY` over process-wide variables such as `HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`, or `NO_PROXY`. HTTPX reads those global variables, but Headroom also passes them through to tool executions.
|
||||
|
||||
### Session Beta Header Tracking
|
||||
|
||||
When running as a proxy, Headroom maintains a per-session union of `anthropic-beta` (and `OpenAI-Beta`) tokens via `SessionBetaTracker`. The session key is derived from the `x-headroom-session-id` header if present, otherwise from `md5(model + system_prompt[:500])[:16]` — stable across turns of the same conversation.
|
||||
|
|
|
|||
|
|
@ -34,6 +34,7 @@ Telemetry is **off by default** (opt-in). Opt in with `HEADROOM_TELEMETRY=on` or
|
|||
| `--limit-concurrency` | `1000` | Maximum concurrent connections before Uvicorn returns 503 |
|
||||
| `--max-connections` | `500` | Maximum upstream HTTP connections |
|
||||
| `--max-keepalive` | `100` | Maximum upstream keep-alive connections |
|
||||
| `--http-proxy` | None | HTTP proxy URL for upstream provider requests only; HTTPS provider APIs use CONNECT |
|
||||
| `--mode` | `token` | Optimization mode: `token` prioritizes compression, `cache` preserves provider prefix-cache stability |
|
||||
| `--no-optimize` | `false` | Disable optimization (passthrough mode) |
|
||||
| `--no-cache` | `false` | Disable semantic caching |
|
||||
|
|
@ -50,6 +51,14 @@ Telemetry is **off by default** (opt-in). Opt in with `HEADROOM_TELEMETRY=on` or
|
|||
| `--no-telemetry` | `false` | Force anonymous telemetry off (already the default) |
|
||||
| `--stateless` | `false` | Disable filesystem writes and keep runtime state in memory |
|
||||
|
||||
Use `--http-proxy` or `HEADROOM_HTTP_PROXY` when only provider API traffic should go through a proxy:
|
||||
|
||||
```bash
|
||||
headroom proxy --http-proxy http://proxy.internal:8080
|
||||
```
|
||||
|
||||
Avoid setting process-wide variables such as `HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`, or `NO_PROXY` for this use case. HTTPX reads those variables too, but Headroom also inherits them into tool executions, so they can proxy unrelated tool traffic.
|
||||
|
||||
### Context management
|
||||
|
||||
| Option | Default | Description |
|
||||
|
|
|
|||
|
|
@ -198,6 +198,15 @@ def dashboard(port: int, no_open: bool) -> None:
|
|||
"(SSLV3_ALERT_BAD_RECORD_MAC) when many concurrent streams are cancelled."
|
||||
),
|
||||
)
|
||||
@click.option(
|
||||
"--http-proxy",
|
||||
default=None,
|
||||
envvar="HEADROOM_HTTP_PROXY",
|
||||
help=(
|
||||
"HTTP proxy URL for upstream provider requests only "
|
||||
"(HTTPS uses CONNECT; env: HEADROOM_HTTP_PROXY)."
|
||||
),
|
||||
)
|
||||
@click.option(
|
||||
"--keepalive-expiry",
|
||||
"keepalive_expiry",
|
||||
|
|
@ -845,6 +854,7 @@ def proxy(
|
|||
max_keepalive_connections: int,
|
||||
keepalive_expiry: float,
|
||||
http2: bool,
|
||||
http_proxy: str | None,
|
||||
intercept_tool_results: bool,
|
||||
no_optimize: bool,
|
||||
no_cache: bool,
|
||||
|
|
@ -1126,6 +1136,7 @@ def proxy(
|
|||
max_keepalive_connections=max_keepalive_connections,
|
||||
keepalive_expiry=keepalive_expiry,
|
||||
http2=http2,
|
||||
http_proxy=http_proxy,
|
||||
log_file=None if is_stateless else log_file,
|
||||
log_full_messages=log_messages
|
||||
or os.environ.get("HEADROOM_LOG_MESSAGES", "").lower() in ("true", "1", "yes", "on"),
|
||||
|
|
|
|||
|
|
@ -287,6 +287,7 @@ class ProxyConfig:
|
|||
max_keepalive_connections: int = 100
|
||||
keepalive_expiry: float = 90.0
|
||||
http2: bool = True
|
||||
http_proxy: str | None = None
|
||||
|
||||
# Memory System
|
||||
memory_enabled: bool = False
|
||||
|
|
|
|||
|
|
@ -597,6 +597,29 @@ def _apply_stateless_persistence(config: ProxyConfig) -> None:
|
|||
get_toin(TOINConfig(storage_path=""))
|
||||
|
||||
|
||||
def _provider_httpx_client_options(
|
||||
config: ProxyConfig,
|
||||
verify: Any,
|
||||
) -> tuple[bool, dict[str, Any]]:
|
||||
client_kwargs: dict[str, Any] = {
|
||||
"timeout": httpx.Timeout(
|
||||
connect=config.connect_timeout_seconds,
|
||||
read=config.request_timeout_seconds,
|
||||
write=config.request_timeout_seconds,
|
||||
pool=config.connect_timeout_seconds,
|
||||
),
|
||||
"limits": httpx.Limits(
|
||||
max_connections=config.max_connections,
|
||||
max_keepalive_connections=config.max_keepalive_connections,
|
||||
keepalive_expiry=config.keepalive_expiry,
|
||||
),
|
||||
"verify": verify,
|
||||
}
|
||||
if config.http_proxy:
|
||||
client_kwargs["proxy"] = config.http_proxy
|
||||
return config.http2 and not config.http_proxy, client_kwargs
|
||||
|
||||
|
||||
class HeadroomProxy(
|
||||
StreamingMixin,
|
||||
AnthropicHandlerMixin,
|
||||
|
|
@ -1372,27 +1395,12 @@ class HeadroomProxy(
|
|||
# is configured, else a strict-relaxed default context when
|
||||
# HEADROOM_TLS_STRICT=0, else httpx's default strict verification.
|
||||
_verify = build_httpx_verify()
|
||||
_client_kwargs: dict[str, Any] = {
|
||||
"timeout": httpx.Timeout(
|
||||
connect=self.config.connect_timeout_seconds,
|
||||
read=self.config.request_timeout_seconds,
|
||||
write=self.config.request_timeout_seconds,
|
||||
pool=self.config.connect_timeout_seconds,
|
||||
),
|
||||
"limits": httpx.Limits(
|
||||
max_connections=self.config.max_connections,
|
||||
max_keepalive_connections=self.config.max_keepalive_connections,
|
||||
keepalive_expiry=self.config.keepalive_expiry,
|
||||
),
|
||||
"verify": _verify,
|
||||
}
|
||||
self.http_client = httpx.AsyncClient(http2=self.config.http2, **_client_kwargs)
|
||||
_http2, _client_kwargs = _provider_httpx_client_options(self.config, _verify)
|
||||
self.http_client = httpx.AsyncClient(http2=_http2, **_client_kwargs)
|
||||
# Reuse the primary client when HTTP/2 is already off; otherwise keep a
|
||||
# dedicated HTTP/1.1 client for ChatGPT passthrough.
|
||||
self.http_client_h1 = (
|
||||
self.http_client
|
||||
if not self.config.http2
|
||||
else httpx.AsyncClient(http2=False, **_client_kwargs)
|
||||
self.http_client if not _http2 else httpx.AsyncClient(http2=False, **_client_kwargs)
|
||||
)
|
||||
logger.info("Headroom Proxy started")
|
||||
logger.info(f"Optimization: {'ENABLED' if self.config.optimize else 'DISABLED'}")
|
||||
|
|
@ -1411,7 +1419,7 @@ class HeadroomProxy(
|
|||
logger.info(
|
||||
f"Connection Pool: max_connections={self.config.max_connections}, "
|
||||
f"max_keepalive={self.config.max_keepalive_connections}, "
|
||||
f"http2={'ENABLED' if self.config.http2 else 'DISABLED'}"
|
||||
f"http2={'ENABLED' if _http2 else 'DISABLED'}"
|
||||
)
|
||||
|
||||
# Unit 4 pre-upstream concurrency announcement. Report the resolved
|
||||
|
|
@ -4189,6 +4197,7 @@ def _proxy_config_from_env() -> ProxyConfig:
|
|||
max_keepalive_connections=_get_env_int("HEADROOM_MAX_KEEPALIVE", 100),
|
||||
keepalive_expiry=_get_env_float("HEADROOM_KEEPALIVE_EXPIRY", 90.0),
|
||||
http2=_get_env_bool("HEADROOM_HTTP2", True),
|
||||
http_proxy=os.environ.get("HEADROOM_HTTP_PROXY") or None,
|
||||
periodic_toin_stats_enabled=_get_env_bool("HEADROOM_PERIODIC_TOIN_STATS", True),
|
||||
proxy_token=os.environ.get("HEADROOM_PROXY_TOKEN") or None,
|
||||
offline=_get_env_bool("HEADROOM_OFFLINE", False),
|
||||
|
|
@ -4246,7 +4255,7 @@ def run_server(
|
|||
|
||||
# Format connection pool info
|
||||
pool_info = f"max={config.max_connections}, keepalive={config.max_keepalive_connections}"
|
||||
http2_status = "ENABLED" if config.http2 else "DISABLED"
|
||||
http2_status = "ENABLED" if (config.http2 and not config.http_proxy) else "DISABLED"
|
||||
|
||||
backend_status = format_backend_status(
|
||||
backend=config.backend,
|
||||
|
|
@ -4580,6 +4589,10 @@ if __name__ == "__main__":
|
|||
action="store_true",
|
||||
help="Disable HTTP/2 (enabled by default for better throughput)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--http-proxy",
|
||||
help=("HTTP proxy URL for upstream provider requests only (env: HEADROOM_HTTP_PROXY)"),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--workers",
|
||||
type=int,
|
||||
|
|
@ -4832,6 +4845,7 @@ if __name__ == "__main__":
|
|||
max_keepalive_connections=_get_env_int("HEADROOM_MAX_KEEPALIVE", args.max_keepalive),
|
||||
keepalive_expiry=_get_env_float("HEADROOM_KEEPALIVE_EXPIRY", args.keepalive_expiry),
|
||||
http2=not args.no_http2 and _get_env_bool("HEADROOM_HTTP2", True),
|
||||
http_proxy=_get_env_str("HEADROOM_HTTP_PROXY", args.http_proxy or "") or None,
|
||||
read_maturation=_get_env_bool("HEADROOM_READ_MATURATION", False),
|
||||
read_maturation_quiesce_turns=_get_env_int("HEADROOM_READ_MATURATION_QUIESCE_TURNS", 5),
|
||||
read_maturation_max_hold_turns=_get_env_int("HEADROOM_READ_MATURATION_MAX_HOLD_TURNS", 25),
|
||||
|
|
|
|||
|
|
@ -89,6 +89,38 @@ class TestLearnNoLearnConflict:
|
|||
assert cfg.traffic_learning_enabled is False
|
||||
|
||||
|
||||
class TestHttpProxyOption:
|
||||
"""--http-proxy should configure only the provider HTTPX clients."""
|
||||
|
||||
def test_http_proxy_cli_flag(self, runner: CliRunner, mock_run_server: dict) -> None:
|
||||
result = runner.invoke(
|
||||
main,
|
||||
["proxy", "--http-proxy", "http://proxy.local:8080"],
|
||||
catch_exceptions=False,
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
assert mock_run_server["config"].http_proxy == "http://proxy.local:8080"
|
||||
|
||||
def test_http_proxy_env_var(self, runner: CliRunner, mock_run_server: dict) -> None:
|
||||
result = runner.invoke(
|
||||
main,
|
||||
["proxy"],
|
||||
env={"HEADROOM_HTTP_PROXY": "http://proxy.local:8080"},
|
||||
catch_exceptions=False,
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
assert mock_run_server["config"].http_proxy == "http://proxy.local:8080"
|
||||
|
||||
def test_direct_server_env_http_proxy(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
import headroom.proxy.server as server_mod
|
||||
|
||||
monkeypatch.delenv(server_mod._MULTI_WORKER_CONFIG_ENV, raising=False)
|
||||
monkeypatch.setenv("HEADROOM_HTTP_PROXY", "http://proxy.local:8080")
|
||||
|
||||
config = server_mod._proxy_config_from_env()
|
||||
assert config.http_proxy == "http://proxy.local:8080"
|
||||
|
||||
|
||||
class TestSubscriptionPollIntervalValidation:
|
||||
"""--subscription-poll-interval should reject values outside 1-3600."""
|
||||
|
||||
|
|
|
|||
|
|
@ -250,7 +250,12 @@ class TestWorkerConfiguration:
|
|||
from headroom.proxy.server import _MULTI_WORKER_CONFIG_ENV, run_server
|
||||
|
||||
captured = {}
|
||||
config = ProxyConfig(host="0.0.0.0", port=8787, max_connections=200)
|
||||
config = ProxyConfig(
|
||||
host="0.0.0.0",
|
||||
port=8787,
|
||||
max_connections=200,
|
||||
http_proxy="http://proxy.local:8080",
|
||||
)
|
||||
|
||||
def fake_run(app, **kwargs):
|
||||
captured["app"] = app
|
||||
|
|
@ -270,6 +275,7 @@ class TestWorkerConfiguration:
|
|||
assert payload["host"] == "0.0.0.0"
|
||||
assert payload["port"] == 8787
|
||||
assert payload["max_connections"] == 200
|
||||
assert payload["http_proxy"] == "http://proxy.local:8080"
|
||||
finally:
|
||||
# run_server sets this via raw os.environ. Pop it directly rather
|
||||
# than via monkeypatch.delenv: delenv records the current (JSON)
|
||||
|
|
@ -312,3 +318,28 @@ class TestWorkerConfiguration:
|
|||
server_mod.run_server(ProxyConfig(), print_banner=False)
|
||||
|
||||
assert "loop" not in captured["kwargs"]
|
||||
|
||||
|
||||
class TestProviderHttpClientOptions:
|
||||
"""Provider HTTPX options should keep proxy settings scoped to provider clients."""
|
||||
|
||||
def test_default_http2_preserved_without_proxy(self):
|
||||
from headroom.proxy.models import ProxyConfig
|
||||
from headroom.proxy.server import _provider_httpx_client_options
|
||||
|
||||
http2, kwargs = _provider_httpx_client_options(ProxyConfig(http2=True), verify=True)
|
||||
|
||||
assert http2 is True
|
||||
assert "proxy" not in kwargs
|
||||
|
||||
def test_http_proxy_sets_proxy_and_forces_http1(self):
|
||||
from headroom.proxy.models import ProxyConfig
|
||||
from headroom.proxy.server import _provider_httpx_client_options
|
||||
|
||||
http2, kwargs = _provider_httpx_client_options(
|
||||
ProxyConfig(http2=True, http_proxy="http://proxy.local:8080"),
|
||||
verify=True,
|
||||
)
|
||||
|
||||
assert http2 is False
|
||||
assert kwargs["proxy"] == "http://proxy.local:8080"
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue