mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
9 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Tejas Chopra
|
93f2d7a2da
|
chore: release main (#2792)
🤖 I have created a release *beep* *boop* --- <details><summary>0.35.0</summary> ## [0.35.0](https://github.com/headroomlabs-ai/headroom/compare/v0.34.0...v0.35.0) (2026-08-12) ### Features * **beacon:** allowlist the routing summary key ([#2818](https://github.com/headroomlabs-ai/headroom/issues/2818)) ([ |
||
|
Parideboy
|
3a27c4dacb
|
fix(proxy/debug): reconcile Kompress warmup state in /debug/warmup (#2711)
## Description
`/debug/warmup` serialized the warmup registry verbatim, so a Kompress
slot left at the startup snapshot kept reporting `{"status": "null",
"info": {"source_status": "deferred"}}` forever — even while the ONNX
model was loaded and actively compressing.
`/health` and `/readyz` already fix this: #2402 added
`_reconcile_kompress_health()`, which promotes the slot from live
runtime state. The debug route never called it, so its answer depended
on whether a health probe happened to run first. That is the half of
#2624 still reproducing on `main`.
Second defect: `WarmupSlot.mark_loaded()` only *updates* `info`, so the
startup-planted `source_status: "deferred"` survived promotion and the
slot serialized as the self-contradictory `{"status": "loaded", "info":
{"source_status": "deferred", "backend": "onnx"}}`.
Closes #2624
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/server.py`: call the existing
`_reconcile_kompress_health()` in the `/debug/warmup` route before
serializing the registry. The reconciler never instantiates a compressor
and never calls `preload()` / `ensure_background_load()` / `compress()`
— it only reads `is_ready()` / `ready_backend()` on an already resident
instance, or falls back to the module-level ONNX cache — so the endpoint
stays side-effect free and idempotent.
- `headroom/proxy/server.py`: stamp `source_status="runtime"` at both
`mark_loaded()` promotion sites in `_reconcile_kompress_health()` (the
resident-compressor path and the `_kompress_cache` fallback),
overwriting the stale startup marker.
- `tests/test_proxy_debug_endpoints.py`: three regression tests plus a
read-only compressor stub whose `preload` / `ensure_background_load`
raise, so a future change that makes the debug route trigger a load
fails loudly.
- `tests/test_proxy_health.py`: assert the promoted slot's
`info["source_status"] == "runtime"`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ pytest tests/test_proxy_debug_endpoints.py tests/test_proxy_health.py tests/test_proxy_warmup.py -q
tests\test_proxy_debug_endpoints.py ............................. [ 52%]
tests\test_proxy_health.py ................. [ 83%]
tests\test_proxy_warmup.py ......... [100%]
============================= 55 passed in 36.56s =============================
$ ruff check headroom/proxy/server.py tests/test_proxy_debug_endpoints.py tests/test_proxy_health.py
All checks passed!
$ ruff format --check headroom/proxy/server.py tests/test_proxy_debug_endpoints.py tests/test_proxy_health.py
3 files already formatted
$ mypy headroom --ignore-missing-imports
Success: no issues found in 506 source files
```
The three new tests were confirmed to be genuine regression tests: with
the `server.py` change reverted and the tests kept, all three fail.
```text
$ git stash push -- headroom/proxy/server.py && pytest tests/test_proxy_debug_endpoints.py -q -k kompress
FAILED tests/test_proxy_debug_endpoints.py::test_debug_warmup_promotes_deferred_kompress_after_runtime_load
FAILED tests/test_proxy_debug_endpoints.py::test_debug_warmup_keeps_pending_kompress_null
FAILED tests/test_proxy_debug_endpoints.py::test_debug_warmup_never_starts_kompress_loading
====================== 3 failed, 26 deselected in 3.98s =======================
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.13.11, pytest 9.1.1, ruff 0.15.x,
mypy 1.20.2, branch based on `main` at
|
||
|
@aaronjmars
|
b4b50253f1
|
fix(security): block DNS-rebinding on /debug/* and /stats/reset via Host-header allowlist (#605)
## Summary `require_loopback` in `headroom/proxy/loopback_guard.py` guards `/debug/tasks`, `/debug/ws-sessions`, `/debug/warmup`, and `/stats/reset` by checking `request.client.host` only. A malicious website can use DNS rebinding to make a victim's browser send requests to `127.0.0.1` while the page origin (and inbound `Host:` header) still reads `attacker.com`. The IP check passes — the browser IS on loopback — and `app.add_middleware(CORSMiddleware, allow_origins=["*"], ...)` (`headroom/proxy/server.py:1678`) lets the attacker's JS read the response. This PR adds the canonical second gate: the `Host:` header must also name loopback (`127.0.0.1[:port]`, `[::1][:port]`, or `localhost[:port]`). ## Impact While the proxy binds to `127.0.0.1` by default (`headroom/proxy/server.py:2956`), any browser the user opens can be used by an unrelated tab to reach the proxy via DNS rebinding. Concretely: - `GET /debug/tasks` → leaks asyncio task names, coroutine qualnames, and (with `?stack=true`) stack depths — reveals which coroutines are mid-flight and which codepaths are warm. Useful for fingerprinting + targeting follow-on attacks. - `GET /debug/ws-sessions` → leaks live WebSocket session metadata including `upstream_url` per session. - `GET /debug/warmup` → leaks warmup registry shape (lower risk). - `POST /stats/reset` → resets in-memory proxy stats (state mutation; minor DoS of dashboards/observability). The fix path matches the OWASP DNS-rebinding mitigation and Starlette's `TrustedHostMiddleware` posture. ## Location - `headroom/proxy/loopback_guard.py:73` — guarded the IP check only, no `Host:` validation - `headroom/proxy/server.py:1819-1845`, `:2312` — endpoints reachable via DNS rebinding under the old gate ## Fix `require_loopback` now runs two gates: (1) the existing `request.client.host` loopback IP check, and (2) a new `Host:` header allowlist via `is_loopback_host_header(...)`. The header helper accepts `127.0.0.1[:port]`, `[::1][:port]`, `localhost[:port]`, and IPv6-mapped IPv4 (`::ffff:127.0.0.1`) — strips brackets and ports, then delegates to the existing `is_loopback_host` logic. Cross-origin browser fetches always carry the attacker's hostname in `Host:`, so rebinding requests now 404 alongside any other external attempt. Same 404 (not 403) semantics, so debug endpoints remain invisible to external scanners. The new helper is in `__all__` for explicit re-use. The existing manual-`Request`-stub unit test path (no `.headers` attribute) is preserved by an `if headers is None: return` fallback so older callers that pass a bare stub still work. Test fixtures in `tests/test_proxy_debug_endpoints.py` were updated to pin `base_url="http://127.0.0.1"` so the loopback-`Host:` invariant holds in the green-path tests, plus a new `app_and_rebinding_client` fixture and `test_debug_endpoints_block_dns_rebinding` exercising the gate at the HTTP level. Six new unit tests cover `is_loopback_host_header` (canonical accept, external reject, malformed reject, and rebinding signature). Net: +219 lines, -4 lines, in two files. **Compatibility note for operators:** a deployment that fronts the proxy behind a reverse proxy with a non-loopback Host (e.g. `headroom.local` mapped to `127.0.0.1`) and still wants `/debug/*` exposed would need to either point that reverse proxy at the upstream API endpoints only, or extend `is_loopback_host_header` with an env-var allowlist in a follow-up. The default `HEADROOM_HOST=127.0.0.1` path is unaffected. ## Detected by Aeon (manual review — no scanner rule for this; pattern matches the threat-model-claims + local-HTTP-server axes from [`skills/vuln-scanner`](https://github.com/aaronjmars/aeon-aaron/blob/main/skills/vuln-scanner/SKILL.md), priors that have surfaced 10+ similar finds across 5 languages over the last 6 weeks). - Severity: medium - CWE-350 (Reliance on Reverse DNS Resolution for a Security-Critical Action) - CWE-352 (Cross-Site Request Forgery) — adjacent - Class match: DNS-rebinding-via-loopback-bind (recurring axis in the tracker) ## Verification - `python3 -m py_compile headroom/proxy/loopback_guard.py` — parses. - The new unit tests in `tests/test_proxy_debug_endpoints.py` cover: canonical loopback Host headers (accept), external Host headers (reject), malformed/empty (reject), an HTTP-level DNS-rebinding simulation (`app_and_rebinding_client` fixture with `base_url="http://attacker.com"` + loopback client tuple), and the unchanged IP-only stub path. - Other test files that use `require_loopback` (`test_proxy_openai_responses_integration.py`, `test_proxy_openai_responses_bypass.py`, `test_proxy_dashboard_stats_cache.py`) all override it via `app.dependency_overrides[require_loopback] = lambda: None`, so they are not affected by the new gate. - I was unable to run `pytest` locally in this sandbox; please confirm the new fixtures slot in cleanly when the CI matrix runs. --- Filed by [Aeon](https://github.com/aaronjmars/aeon-aaron). --------- Co-authored-by: aeonframework <noreply@anthropic.com> |
||
|
Adryan Eka Vandra
|
4e9348dec4
|
fix: add anthropic pre-upstream timeouts | ||
|
Adryan Eka Vandra
|
b71b659e1b
|
fix(ci): restore repro harness test in dev installs
Add websockets to the dev extra so the repro harness smoke test can import its websocket client dependency in the CI test matrix. Also apply ruff formatting to the files the formatter check was rejecting so the 3.12 lint job passes. |
||
|
Adryan Eka Vandra
|
8bbf3afda0
|
perf(proxy): gate /debug/tasks stack-depth computation behind query param
Task.get_stack(limit=32) walks coroutine frames synchronously and measurably stalls the event loop when called for 50+ relay tasks during a reconnect storm. The /debug/tasks snapshot does not need this by default — the perf-cheap fields (name, coro_qualname, age, done) are enough for the common case of "which tasks are alive". - collect_tasks: add with_stack_depth=False kwarg; default leaves stack_depth=None per entry. - /debug/tasks: read ?stack=true query parameter and forward it. Default response is cheap; opt-in explicitly when human-debugging one snapshot. - Test: verify default entries have stack_depth=None and that ?stack=true produces at least one integer depth. |
||
|
Adryan Eka Vandra
|
55f59fad77
|
chore(proxy): announce pre-upstream concurrency; clean dead RequestLog fields and debug helpers
- Log the resolved Anthropic pre-upstream concurrency at startup so operators can correlate pre_upstream_wait_ms lines with the configured cap. Distinguishes auto-detected vs. explicit vs. unbounded. - Remove the stage_timings, session_id, and stage_timings_path fields from RequestLog. They were declared optional but never populated at any of the three RequestLog construction sites — stage timings flow exclusively through emit_stage_timings_log (structured log line) and Prometheus histograms. Keeping unpopulated fields misled readers. Option (A) from the review. - Inline the one-liner collect_ws_sessions / collect_warmup pass-throughs at their two callers in server.py; keep collect_tasks since it has real logic. Unit tests were rewritten against the registry's own snapshot()/to_dict() serializers. |
||
|
Adryan Eka Vandra
|
0e166c60d4
|
fix(proxy): accept IPv6-mapped IPv4 loopback in debug guard
On Linux dual-stack sockets (IPV6_V6ONLY=0 default), an IPv4 loopback connection arrives as ::ffff:127.0.0.1 and was 404'd by the literal set check — silent outage of /debug/* when the proxy binds to :: or 0.0.0.0. Replace the set-membership check with ipaddress.ip_address(host).is_loopback, special-casing 'localhost' (not an IP literal) and treating ValueError (malformed input) as non-loopback. The None sentinel for TestClient / UDS sockets is preserved. |
||
|
Adryan Eka Vandra
|
a4010aaa1b
|
feat(proxy): add loopback-only debug introspection endpoints
Unit 5 of the Codex-proxy resilience plan. Three always-on /debug/* endpoints make "what is this process doing right now?" a single curl away: - GET /debug/tasks serializes asyncio.all_tasks() to name, coro_qualname, stack_depth, done, and age_seconds (derived from the WS session registry for Codex relay tasks, null otherwise). Sorted by age desc. No frame locals, no request bodies, no coroutine args. - GET /debug/ws-sessions returns WebSocketSessionRegistry.snapshot(). - GET /debug/warmup returns WarmupRegistry.to_dict(). A FastAPI Depends(require_loopback) dependency 404s (not 403) any non-loopback caller so debug routes stay invisible to external scanners. Dependency chosen over middleware because the route set is small and explicit Depends(...) makes the guard visible to reviewers and easy to override in tests. |