headroom/tests/test_proxy_loopback_gating.py

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

748 lines
26 KiB
Python
Raw Normal View History

fix(proxy): scope CORS to loopback + gate operator/content endpoints (#1226) ## Description Locks down the proxy's browser- and network-facing attack surface, which matters most under a `--host 0.0.0.0` bind (the Docker default). The wildcard CORS policy (`allow_origins=["*"]` + `allow_credentials=True`) let any web page the user had open read the proxy's content endpoints — `/v1/retrieve` returns raw, uncompressed tool outputs (source, secrets) — via a cross-origin fetch to `127.0.0.1` (CWE-346). Several operator endpoints additionally leaked sensitive data or allowed unauthenticated state mutation to any network-reachable client. This PR scopes CORS to loopback origins and extends the project's existing `require_loopback` trust boundary (already used for `/admin/*` and `/debug/*`) to the remaining exposed endpoints. Closes #863. Supersedes #864 and #758 — see "Additional Notes". ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **CORS**: replaced `allow_origins=["*"]` + `allow_credentials=True` with a port-agnostic loopback origin regex (`https?://(localhost|127\.0\.0\.1|\[::1\])(:\d+)?`), `allow_credentials=False`, and methods/headers narrowed to `GET/POST` + `Content-Type/Authorization`. `HEADROOM_CORS_ORIGINS` (comma-separated) pins an explicit allowlist for Docker/remote dashboards; `*` opts back into the old wildcard. - **`/transformations/feed`** and **`/cache/clear`** gated behind `require_loopback` → 404 for non-loopback callers. The feed returns full prompt/completion bodies when `log_full_messages` is on; `/cache/clear` is unauthenticated state mutation (cache-eviction DoS / cost amplification). - **`/health`**: the `config` block (upstream API URLs, savings profile) is now served only to loopback callers; network callers get the `/readyz`-shape body (status/checks). `/livez` and `/readyz` remain unauthenticated probes for orchestration. - **`/stats`**: `recent_requests` / `request_logs` (per-request ids, providers, models, errors) and `config` are served only to loopback callers; aggregate counters stay public for remote monitoring. - Added `_request_is_loopback()` helper mirroring `require_loopback`'s two-gate check (loopback peer IP + loopback `Host` header, the DNS-rebinding defence) but degrading the payload instead of returning 404, so monitors keep the non-sensitive fields. - Tests: new `tests/test_proxy_cors.py` and `tests/test_proxy_loopback_gating.py`; updated 4 existing tests that assert the now-loopback-only data to use loopback clients. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check headroom/proxy/server.py tests/test_proxy_cors.py \ tests/test_proxy_loopback_gating.py tests/test_proxy_healthchecks.py \ tests/test_proxy_stats_recent_requests.py tests/test_proxy/test_transformations_feed.py All checks passed! $ mypy headroom --ignore-missing-imports Success: no issues found in 380 source files $ pytest tests/test_proxy_cors.py tests/test_proxy_loopback_gating.py \ tests/test_proxy/test_transformations_feed.py tests/test_proxy_healthchecks.py \ tests/test_proxy_stats_recent_requests.py tests/test_proxy_dashboard_stats_cache.py \ tests/test_proxy_compression_executor.py tests/test_header_isolation.py -q ======================== 82 passed, 1 skipped in 17.75s ======================== ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0), Python 3.12 venv; FastAPI `TestClient` driving the real `create_app()` ASGI app - Exact command / steps: issued requests as a non-loopback caller (`client.host=testclient`) vs a loopback caller (`base_url=http://127.0.0.1`, `client=("127.0.0.1", 9999)`), plus CORS preflights with varying `Origin` headers - Observed result: CORS — `http://evil.com` → no `access-control-allow-origin`; `http://localhost:8787` and `http://localhost:9000` → echoed (loopback allowed on any port); `access-control-allow-credentials` → absent. `/cache/clear` and `/transformations/feed` → 404 (network) / 200 (loopback). `/health` `config` block present for loopback only. `/stats` `recent_requests` present for loopback only, while the aggregate `tokens` block stays present for network callers. - Not tested: a live `headroom proxy` process bound on `0.0.0.0` reached from a second host (simulated via ASGI peer/Host instead); end-to-end browser DNS-rebinding (covered by the `Host`-header gate and its unit test) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — proxy/middleware change; behavior is captured under "Real Behavior Proof". ## Additional Notes **Supersedes two stale PRs that target the same issue but have drifted from `main`:** - **#864** (`fix(proxy): scope CORS to localhost`, @gabiudrescu) — correct instinct and the source of the tighter `GET/POST` + `Content-Type/Authorization` scoping kept here, but it derived the allowlist from the `HEADROOM_PORT` env var (wrong when `--port` is passed as a CLI flag), carried ~40 lines of unrelated punctuation churn, and is ~125 commits behind `main`. The port-agnostic regex used here resolves the reviewer's port concern. - **#758** (`security: adversarial review`, @neogenix) — bundled these same application-layer fixes with a large CI/CD + Docker supply-chain pass. It is a ~160-commit-behind draft whose `server.py` no longer merges cleanly (`main` independently adopted the same `require_loopback` pattern). The application-layer fixes are rebased onto current `main` here; the CI/Docker/supply-chain hardening from #758 is still valuable and would be welcome as a separate, rebased PR. Thanks to @gabiudrescu and @neogenix for the original analysis (#863). **Deliberate scope / follow-ups (not in this PR):** - `/stats` aggregate counters and the basic `/health` body remain readable on a `0.0.0.0` bind by design, so remote monitoring keeps working. Full lock-down is a one-line `Depends(require_loopback)` each if preferred. - The `/v1/retrieve*` family stays network-reachable; it can't be loopback-gated without breaking legitimate remote/containerized agents and needs auth instead — tracked separately. - `ruff check .` is scoped to changed paths above because the dashboard HTML template trips ruff's `invalid-syntax` (a known repo false-positive); `mypy` is run over the full `headroom` package.
2026-06-21 00:50:55 -07:00
"""Loopback-gating tests for state-mutating / content-leaking endpoints.
``/transformations/feed`` can return full prompt + completion bodies (when
``log_full_messages`` is on) and ``/cache/clear`` mutates server state. With the
default ``--host 0.0.0.0`` Docker bind, neither should be reachable by an
arbitrary network client they are gated to the loopback interface via
``require_loopback`` (the same guard already used for ``/admin/*`` and
``/debug/*``). See #863.
"""
from __future__ import annotations
import pytest
from fastapi import FastAPI
from fastapi.testclient import TestClient
test(proxy): assert CCR hash route guard blocks valid hashes (#1480) ## Description PR #1338 added loopback gating to the CCR retrieve/compress endpoints for #1227, but one regression case still had weak proof: `GET /v1/retrieve/{hash_key}` used a dummy hash. Since both the route guard and the missing-hash handler return `404`, that test could pass even if the route reached the handler. This adds a seeded-hash regression so the test proves the security property directly. Closes #1227 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) - [x] Add more testing ## Changes Made - Seed a real CCR entry using an in-memory compression-store backend. - Verify loopback can retrieve the seeded entry and see `original_content`. - Verify a non-loopback caller gets `404` for the same valid hash. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality ### Test Output ```text 13 passed ``` ## Testing Commands ```bash .venv/bin/pytest tests/test_proxy_loopback_gating.py -q .venv/bin/pytest tests/test_proxy_loopback_gating.py tests/test_proxy_cors.py tests/test_proxy_compress_endpoint.py -q .venv/bin/ruff check tests/test_proxy_loopback_gating.py ``` ## Real Behavior Proof - Environment: macOS 25.4.0, Python 3.12 - Exact command / steps: `.venv/bin/pytest tests/test_proxy_loopback_gating.py -q` `.venv/bin/pytest tests/test_proxy_loopback_gating.py tests/test_proxy_cors.py tests/test_proxy_compress_endpoint.py -q` `.venv/bin/ruff check tests/test_proxy_loopback_gating.py` - Not tested: NA - Observed result: Pass ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable
2026-06-26 15:29:33 -07:00
from headroom.cache.backends import InMemoryBackend
fix(proxy): guard feedback endpoints and add CSRF checks to loopback writes (#3060) ## Description `#2927` brought eight telemetry/TOIN routes under `require_loopback`. Two structurally identical siblings 60 lines above them were missed: ``` GET /v1/feedback GET /v1/feedback/{tool_name} ``` Neither is an aggregate-counter endpoint. Their `common_queries` / `queried_fields` keys are built verbatim from agent search text — `event.query.lower()` at `headroom/cache/compression_feedback.py:311` — and up to 100 queries are retained per tool, keyed by real tool name. Under the shipped Docker default (`--host 0.0.0.0`) a LAN peer gets a 404 from `/v1/toin/patterns` and the query corpus from `/v1/feedback`. Separately, five mutating loopback-only routes had no CSRF guard. `require_loopback` cannot stop that attack: a remote page POSTing to a known `127.0.0.1` URL with `Content-Type: text/plain` is a CORS *simple* request, so there is no preflight, and the browser still sends the real loopback `Host` header — both of the guard's gates pass. Only `Origin` betrays the caller, and only `require_same_origin` inspects it. That guard already existed at `headroom/proxy/loopback_guard.py:219` and was applied solely to `/settings`. Closes #2927 (completes it — the original eight routes were already done). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update ## Changes Made - Added `Depends(_require_loopback)` to `/v1/feedback` and `/v1/feedback/{tool_name}`. - Stripped `common_queries` / `queried_fields` from both response bodies even on the guarded path, matching the whitelist discipline #2930 applied at `server.py:4909-4916`. - Added `_feedback_stats_without_query_text()` so the scrub happens at the HTTP boundary; `get_stats()` is unchanged and in-process compression decisions are untouched. - Added `Depends(_require_same_origin)` to `POST /stats/reset`, `/cache/clear`, `/v1/retrieve`, `/v1/telemetry/import`, `/admin/runtime-env`. ## Testing - [x] Unit tests pass - [x] Linting passes (ruff check + format) - [ ] Type checking passes (`uv run mypy headroom`) — not run - [x] New tests added for new functionality ### Test Output ```text $ .venv/bin/python -m pytest tests/test_proxy_loopback_gating.py -q 99 passed, 1 warning in 4.18s $ .venv/bin/python -m pytest tests/test_proxy_settings_endpoints.py tests/test_telemetry.py \ tests/test_proxy_cache_telemetry.py tests/test_proxy_telemetry_env.py tests/test_telemetry_context.py -q 101 passed, 1 warning in 3.67s $ .venv/bin/python -m pytest tests/test_critical_fixes.py tests/test_compression_store.py \ tests/test_toin_full_integration.py tests/test_ccr_feedback.py tests/test_critical_gaps.py \ tests/test_proxy_ccr.py tests/test_proxy_dashboard_stats_cache.py -q 168 passed, 4 skipped, 3 warnings in 13.18s $ .venv/bin/python -m ruff check headroom/proxy/server.py tests/test_proxy_loopback_gating.py All checks passed! ``` Against the parent commit (`git stash` of `server.py` only), all 14 new tests fail: ```text FAILED test_non_loopback_caller_gets_404[get-/v1/feedback] FAILED test_non_loopback_caller_gets_404[get-/v1/feedback/example] FAILED test_cross_origin_post_rejected[/stats/reset] FAILED test_cross_origin_post_rejected[/cache/clear] FAILED test_cross_origin_post_rejected[/v1/retrieve] FAILED test_cross_origin_post_rejected[/v1/telemetry/import] FAILED test_cross_origin_post_rejected[/admin/runtime-env] FAILED test_sandboxed_null_origin_post_rejected[...] (5 cases) FAILED test_feedback_stats_exclude_agent_query_text FAILED test_feedback_tool_detail_excludes_agent_query_text 14 failed, 85 passed ``` ## Real Behavior Proof - Environment: macOS 15 (darwin 25.4.0), Python 3.12.13, this branch, FastAPI `TestClient` against the real `create_app` proxy. - Exact command / steps: drive `/v1/feedback` with a feedback singleton whose `common_queries` contains `"find the customer api key rotation runbook"`, once from a non-loopback peer and once from a loopback peer; POST each of the five mutating routes with `Origin: https://attacker.example` and `Content-Type: text/plain`. - Observed result: non-loopback callers now receive 404 where they previously received 200 with the query corpus; on the loopback path the response no longer contains `common_queries`, `queried_fields`, or the substring `customer api key rotation`, while `retrieval_rate` still resolves to `0.25`. All five cross-origin POSTs return 403; the same requests with no `Origin`, or with `Origin: http://127.0.0.1`, are unaffected. - Not tested: a real browser issuing the cross-origin POST (the CORS simple-request shape is reproduced at the header level, not in a browser), and a live non-loopback deployment. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: N/A. - Stable/default behavior changed: yes — `/v1/feedback*` now 404 for non-loopback callers and no longer return query text; five POST routes reject cross-origin browser callers. - Kill switch / disable path: none; these are security guards and are deliberately not configurable. - Unsafe override required: none. - Qualification impact: none. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes ## Additional Notes `/stats` also calls `feedback.get_stats()` (`server.py:3896`) but only reads aggregate counters at `:4303-4311` and never emits query text — verified, and the reason the scrub is applied at the HTTP boundary rather than inside `get_stats()`. The five POST routes are strictly loopback-gated, so the trusted-dashboard wrapper `/settings` uses is unnecessary here; for a loopback caller that wrapper falls through to the same raw guard. No dashboard asset calls them, and the TypeScript SDK (`sdk/typescript/src/client.ts:322,443`) sends no `Origin` header, which the guard passes through unchanged. Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local>
2026-08-16 18:23:27 -07:00
from headroom.cache.compression_feedback import CompressionHints
test(proxy): assert CCR hash route guard blocks valid hashes (#1480) ## Description PR #1338 added loopback gating to the CCR retrieve/compress endpoints for #1227, but one regression case still had weak proof: `GET /v1/retrieve/{hash_key}` used a dummy hash. Since both the route guard and the missing-hash handler return `404`, that test could pass even if the route reached the handler. This adds a seeded-hash regression so the test proves the security property directly. Closes #1227 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) - [x] Add more testing ## Changes Made - Seed a real CCR entry using an in-memory compression-store backend. - Verify loopback can retrieve the seeded entry and see `original_content`. - Verify a non-loopback caller gets `404` for the same valid hash. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality ### Test Output ```text 13 passed ``` ## Testing Commands ```bash .venv/bin/pytest tests/test_proxy_loopback_gating.py -q .venv/bin/pytest tests/test_proxy_loopback_gating.py tests/test_proxy_cors.py tests/test_proxy_compress_endpoint.py -q .venv/bin/ruff check tests/test_proxy_loopback_gating.py ``` ## Real Behavior Proof - Environment: macOS 25.4.0, Python 3.12 - Exact command / steps: `.venv/bin/pytest tests/test_proxy_loopback_gating.py -q` `.venv/bin/pytest tests/test_proxy_loopback_gating.py tests/test_proxy_cors.py tests/test_proxy_compress_endpoint.py -q` `.venv/bin/ruff check tests/test_proxy_loopback_gating.py` - Not tested: NA - Observed result: Pass ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable
2026-06-26 15:29:33 -07:00
from headroom.cache.compression_store import get_compression_store, reset_compression_store
feat(dashboard): persist lifetime proxy metrics (#2198) ## Description Persist bounded, aggregate-only Lifetime dashboard metrics across proxy restarts and expose them through a new `/stats-lifetime` endpoint. The change keeps session/runtime stats separate from durable lifetime stats, gates sensitive dashboard metadata for loopback or explicitly trusted dashboard clients, and updates the dashboard Lifetime view to consume the new endpoint. Closes #2137 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Added bounded persistent lifetime metrics state and wired proxy metric events into it. - Added `/stats-lifetime` with sensitive project/persistence details gated behind dashboard metadata access checks. - Extended loopback/dashboard metadata access policy for trusted dashboard client CIDRs without widening admin/debug endpoints. - Reorganized dashboard session/lifetime presentation around runtime counters versus durable aggregates. - Added focused tests for persistent aggregation, persistence, endpoint registration, loopback gating, trusted dashboard CIDRs, and recent request ordering. - Fixed current Ruff/mypy issues in the lifetime metrics normalization code. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text uv run --extra proxy --with pytest --with pytest-asyncio pytest tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_stats_recent_requests.py -q 53 passed, 1 warning uvx ruff==0.15.17 check headroom/proxy/forwarded_headers.py headroom/proxy/loopback_guard.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py headroom/proxy/server.py tests/test_forwarded_headers.py tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_project_savings.py tests/test_proxy_stats_recent_requests.py All checks passed! uv run --extra proxy --with mypy mypy headroom/proxy/persistent_metrics.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows review worktree, Python 3.13.3 via uv. - Exact command / steps: Ran the focused persistent metrics, persistence, loopback gating, and recent request tests; ran CI-matching Ruff on touched files; ran mypy on the new persistent metrics module. - Observed result: `/stats-lifetime` is registered, non-loopback callers receive only non-sensitive aggregate data, loopback/trusted dashboard clients receive the full lifetime payload, admin/debug endpoints remain loopback-only, and persistent metrics normalize malformed stored state without type/lint errors. - Not tested: Full repository pytest suite, full dashboard browser screenshot pass, or live long-running proxy traffic. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes No changelog entry is required for this dashboard/internal metrics iteration. The endpoint intentionally exposes only aggregate lifetime data to ordinary network callers and strips project/persistence details unless the caller passes the dashboard metadata access policy. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-16 02:18:13 +08:00
from headroom.proxy.loopback_guard import is_ip_literal_host_header
fix(proxy): scope CORS to loopback + gate operator/content endpoints (#1226) ## Description Locks down the proxy's browser- and network-facing attack surface, which matters most under a `--host 0.0.0.0` bind (the Docker default). The wildcard CORS policy (`allow_origins=["*"]` + `allow_credentials=True`) let any web page the user had open read the proxy's content endpoints — `/v1/retrieve` returns raw, uncompressed tool outputs (source, secrets) — via a cross-origin fetch to `127.0.0.1` (CWE-346). Several operator endpoints additionally leaked sensitive data or allowed unauthenticated state mutation to any network-reachable client. This PR scopes CORS to loopback origins and extends the project's existing `require_loopback` trust boundary (already used for `/admin/*` and `/debug/*`) to the remaining exposed endpoints. Closes #863. Supersedes #864 and #758 — see "Additional Notes". ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **CORS**: replaced `allow_origins=["*"]` + `allow_credentials=True` with a port-agnostic loopback origin regex (`https?://(localhost|127\.0\.0\.1|\[::1\])(:\d+)?`), `allow_credentials=False`, and methods/headers narrowed to `GET/POST` + `Content-Type/Authorization`. `HEADROOM_CORS_ORIGINS` (comma-separated) pins an explicit allowlist for Docker/remote dashboards; `*` opts back into the old wildcard. - **`/transformations/feed`** and **`/cache/clear`** gated behind `require_loopback` → 404 for non-loopback callers. The feed returns full prompt/completion bodies when `log_full_messages` is on; `/cache/clear` is unauthenticated state mutation (cache-eviction DoS / cost amplification). - **`/health`**: the `config` block (upstream API URLs, savings profile) is now served only to loopback callers; network callers get the `/readyz`-shape body (status/checks). `/livez` and `/readyz` remain unauthenticated probes for orchestration. - **`/stats`**: `recent_requests` / `request_logs` (per-request ids, providers, models, errors) and `config` are served only to loopback callers; aggregate counters stay public for remote monitoring. - Added `_request_is_loopback()` helper mirroring `require_loopback`'s two-gate check (loopback peer IP + loopback `Host` header, the DNS-rebinding defence) but degrading the payload instead of returning 404, so monitors keep the non-sensitive fields. - Tests: new `tests/test_proxy_cors.py` and `tests/test_proxy_loopback_gating.py`; updated 4 existing tests that assert the now-loopback-only data to use loopback clients. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check headroom/proxy/server.py tests/test_proxy_cors.py \ tests/test_proxy_loopback_gating.py tests/test_proxy_healthchecks.py \ tests/test_proxy_stats_recent_requests.py tests/test_proxy/test_transformations_feed.py All checks passed! $ mypy headroom --ignore-missing-imports Success: no issues found in 380 source files $ pytest tests/test_proxy_cors.py tests/test_proxy_loopback_gating.py \ tests/test_proxy/test_transformations_feed.py tests/test_proxy_healthchecks.py \ tests/test_proxy_stats_recent_requests.py tests/test_proxy_dashboard_stats_cache.py \ tests/test_proxy_compression_executor.py tests/test_header_isolation.py -q ======================== 82 passed, 1 skipped in 17.75s ======================== ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0), Python 3.12 venv; FastAPI `TestClient` driving the real `create_app()` ASGI app - Exact command / steps: issued requests as a non-loopback caller (`client.host=testclient`) vs a loopback caller (`base_url=http://127.0.0.1`, `client=("127.0.0.1", 9999)`), plus CORS preflights with varying `Origin` headers - Observed result: CORS — `http://evil.com` → no `access-control-allow-origin`; `http://localhost:8787` and `http://localhost:9000` → echoed (loopback allowed on any port); `access-control-allow-credentials` → absent. `/cache/clear` and `/transformations/feed` → 404 (network) / 200 (loopback). `/health` `config` block present for loopback only. `/stats` `recent_requests` present for loopback only, while the aggregate `tokens` block stays present for network callers. - Not tested: a live `headroom proxy` process bound on `0.0.0.0` reached from a second host (simulated via ASGI peer/Host instead); end-to-end browser DNS-rebinding (covered by the `Host`-header gate and its unit test) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — proxy/middleware change; behavior is captured under "Real Behavior Proof". ## Additional Notes **Supersedes two stale PRs that target the same issue but have drifted from `main`:** - **#864** (`fix(proxy): scope CORS to localhost`, @gabiudrescu) — correct instinct and the source of the tighter `GET/POST` + `Content-Type/Authorization` scoping kept here, but it derived the allowlist from the `HEADROOM_PORT` env var (wrong when `--port` is passed as a CLI flag), carried ~40 lines of unrelated punctuation churn, and is ~125 commits behind `main`. The port-agnostic regex used here resolves the reviewer's port concern. - **#758** (`security: adversarial review`, @neogenix) — bundled these same application-layer fixes with a large CI/CD + Docker supply-chain pass. It is a ~160-commit-behind draft whose `server.py` no longer merges cleanly (`main` independently adopted the same `require_loopback` pattern). The application-layer fixes are rebased onto current `main` here; the CI/Docker/supply-chain hardening from #758 is still valuable and would be welcome as a separate, rebased PR. Thanks to @gabiudrescu and @neogenix for the original analysis (#863). **Deliberate scope / follow-ups (not in this PR):** - `/stats` aggregate counters and the basic `/health` body remain readable on a `0.0.0.0` bind by design, so remote monitoring keeps working. Full lock-down is a one-line `Depends(require_loopback)` each if preferred. - The `/v1/retrieve*` family stays network-reachable; it can't be loopback-gated without breaking legitimate remote/containerized agents and needs auth instead — tracked separately. - `ruff check .` is scoped to changed paths above because the dashboard HTML template trips ruff's `invalid-syntax` (a known repo false-positive); `mypy` is run over the full `headroom` package.
2026-06-21 00:50:55 -07:00
from headroom.proxy.server import ProxyConfig, create_app
GATED = [
("get", "/transformations/feed"),
("post", "/cache/clear"),
("get", "/v1/telemetry"),
("get", "/v1/telemetry/export"),
("post", "/v1/telemetry/import"),
("get", "/v1/telemetry/tools"),
("get", "/v1/telemetry/tools/example"),
("get", "/v1/toin/stats"),
("get", "/v1/toin/patterns"),
("get", "/v1/toin/pattern/example"),
fix(proxy): guard feedback endpoints and add CSRF checks to loopback writes (#3060) ## Description `#2927` brought eight telemetry/TOIN routes under `require_loopback`. Two structurally identical siblings 60 lines above them were missed: ``` GET /v1/feedback GET /v1/feedback/{tool_name} ``` Neither is an aggregate-counter endpoint. Their `common_queries` / `queried_fields` keys are built verbatim from agent search text — `event.query.lower()` at `headroom/cache/compression_feedback.py:311` — and up to 100 queries are retained per tool, keyed by real tool name. Under the shipped Docker default (`--host 0.0.0.0`) a LAN peer gets a 404 from `/v1/toin/patterns` and the query corpus from `/v1/feedback`. Separately, five mutating loopback-only routes had no CSRF guard. `require_loopback` cannot stop that attack: a remote page POSTing to a known `127.0.0.1` URL with `Content-Type: text/plain` is a CORS *simple* request, so there is no preflight, and the browser still sends the real loopback `Host` header — both of the guard's gates pass. Only `Origin` betrays the caller, and only `require_same_origin` inspects it. That guard already existed at `headroom/proxy/loopback_guard.py:219` and was applied solely to `/settings`. Closes #2927 (completes it — the original eight routes were already done). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update ## Changes Made - Added `Depends(_require_loopback)` to `/v1/feedback` and `/v1/feedback/{tool_name}`. - Stripped `common_queries` / `queried_fields` from both response bodies even on the guarded path, matching the whitelist discipline #2930 applied at `server.py:4909-4916`. - Added `_feedback_stats_without_query_text()` so the scrub happens at the HTTP boundary; `get_stats()` is unchanged and in-process compression decisions are untouched. - Added `Depends(_require_same_origin)` to `POST /stats/reset`, `/cache/clear`, `/v1/retrieve`, `/v1/telemetry/import`, `/admin/runtime-env`. ## Testing - [x] Unit tests pass - [x] Linting passes (ruff check + format) - [ ] Type checking passes (`uv run mypy headroom`) — not run - [x] New tests added for new functionality ### Test Output ```text $ .venv/bin/python -m pytest tests/test_proxy_loopback_gating.py -q 99 passed, 1 warning in 4.18s $ .venv/bin/python -m pytest tests/test_proxy_settings_endpoints.py tests/test_telemetry.py \ tests/test_proxy_cache_telemetry.py tests/test_proxy_telemetry_env.py tests/test_telemetry_context.py -q 101 passed, 1 warning in 3.67s $ .venv/bin/python -m pytest tests/test_critical_fixes.py tests/test_compression_store.py \ tests/test_toin_full_integration.py tests/test_ccr_feedback.py tests/test_critical_gaps.py \ tests/test_proxy_ccr.py tests/test_proxy_dashboard_stats_cache.py -q 168 passed, 4 skipped, 3 warnings in 13.18s $ .venv/bin/python -m ruff check headroom/proxy/server.py tests/test_proxy_loopback_gating.py All checks passed! ``` Against the parent commit (`git stash` of `server.py` only), all 14 new tests fail: ```text FAILED test_non_loopback_caller_gets_404[get-/v1/feedback] FAILED test_non_loopback_caller_gets_404[get-/v1/feedback/example] FAILED test_cross_origin_post_rejected[/stats/reset] FAILED test_cross_origin_post_rejected[/cache/clear] FAILED test_cross_origin_post_rejected[/v1/retrieve] FAILED test_cross_origin_post_rejected[/v1/telemetry/import] FAILED test_cross_origin_post_rejected[/admin/runtime-env] FAILED test_sandboxed_null_origin_post_rejected[...] (5 cases) FAILED test_feedback_stats_exclude_agent_query_text FAILED test_feedback_tool_detail_excludes_agent_query_text 14 failed, 85 passed ``` ## Real Behavior Proof - Environment: macOS 15 (darwin 25.4.0), Python 3.12.13, this branch, FastAPI `TestClient` against the real `create_app` proxy. - Exact command / steps: drive `/v1/feedback` with a feedback singleton whose `common_queries` contains `"find the customer api key rotation runbook"`, once from a non-loopback peer and once from a loopback peer; POST each of the five mutating routes with `Origin: https://attacker.example` and `Content-Type: text/plain`. - Observed result: non-loopback callers now receive 404 where they previously received 200 with the query corpus; on the loopback path the response no longer contains `common_queries`, `queried_fields`, or the substring `customer api key rotation`, while `retrieval_rate` still resolves to `0.25`. All five cross-origin POSTs return 403; the same requests with no `Origin`, or with `Origin: http://127.0.0.1`, are unaffected. - Not tested: a real browser issuing the cross-origin POST (the CORS simple-request shape is reproduced at the header level, not in a browser), and a live non-loopback deployment. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: N/A. - Stable/default behavior changed: yes — `/v1/feedback*` now 404 for non-loopback callers and no longer return query text; five POST routes reject cross-origin browser callers. - Kill switch / disable path: none; these are security guards and are deliberately not configurable. - Unsafe override required: none. - Qualification impact: none. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes ## Additional Notes `/stats` also calls `feedback.get_stats()` (`server.py:3896`) but only reads aggregate counters at `:4303-4311` and never emits query text — verified, and the reason the scrub is applied at the HTTP boundary rather than inside `get_stats()`. The five POST routes are strictly loopback-gated, so the trusted-dashboard wrapper `/settings` uses is unnecessary here; for a loopback caller that wrapper falls through to the same raw guard. No dashboard asset calls them, and the TypeScript SDK (`sdk/typescript/src/client.ts:322,443`) sends no `Origin` header, which the guard passes through unchanged. Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local>
2026-08-16 18:23:27 -07:00
# #2927 guarded the eight telemetry/TOIN routes the issue enumerated but
# left these two siblings open, and their payload carries the same raw
# agent query text (``common_queries``, built from ``event.query``).
("get", "/v1/feedback"),
("get", "/v1/feedback/example"),
fix(proxy): scope CORS to loopback + gate operator/content endpoints (#1226) ## Description Locks down the proxy's browser- and network-facing attack surface, which matters most under a `--host 0.0.0.0` bind (the Docker default). The wildcard CORS policy (`allow_origins=["*"]` + `allow_credentials=True`) let any web page the user had open read the proxy's content endpoints — `/v1/retrieve` returns raw, uncompressed tool outputs (source, secrets) — via a cross-origin fetch to `127.0.0.1` (CWE-346). Several operator endpoints additionally leaked sensitive data or allowed unauthenticated state mutation to any network-reachable client. This PR scopes CORS to loopback origins and extends the project's existing `require_loopback` trust boundary (already used for `/admin/*` and `/debug/*`) to the remaining exposed endpoints. Closes #863. Supersedes #864 and #758 — see "Additional Notes". ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **CORS**: replaced `allow_origins=["*"]` + `allow_credentials=True` with a port-agnostic loopback origin regex (`https?://(localhost|127\.0\.0\.1|\[::1\])(:\d+)?`), `allow_credentials=False`, and methods/headers narrowed to `GET/POST` + `Content-Type/Authorization`. `HEADROOM_CORS_ORIGINS` (comma-separated) pins an explicit allowlist for Docker/remote dashboards; `*` opts back into the old wildcard. - **`/transformations/feed`** and **`/cache/clear`** gated behind `require_loopback` → 404 for non-loopback callers. The feed returns full prompt/completion bodies when `log_full_messages` is on; `/cache/clear` is unauthenticated state mutation (cache-eviction DoS / cost amplification). - **`/health`**: the `config` block (upstream API URLs, savings profile) is now served only to loopback callers; network callers get the `/readyz`-shape body (status/checks). `/livez` and `/readyz` remain unauthenticated probes for orchestration. - **`/stats`**: `recent_requests` / `request_logs` (per-request ids, providers, models, errors) and `config` are served only to loopback callers; aggregate counters stay public for remote monitoring. - Added `_request_is_loopback()` helper mirroring `require_loopback`'s two-gate check (loopback peer IP + loopback `Host` header, the DNS-rebinding defence) but degrading the payload instead of returning 404, so monitors keep the non-sensitive fields. - Tests: new `tests/test_proxy_cors.py` and `tests/test_proxy_loopback_gating.py`; updated 4 existing tests that assert the now-loopback-only data to use loopback clients. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check headroom/proxy/server.py tests/test_proxy_cors.py \ tests/test_proxy_loopback_gating.py tests/test_proxy_healthchecks.py \ tests/test_proxy_stats_recent_requests.py tests/test_proxy/test_transformations_feed.py All checks passed! $ mypy headroom --ignore-missing-imports Success: no issues found in 380 source files $ pytest tests/test_proxy_cors.py tests/test_proxy_loopback_gating.py \ tests/test_proxy/test_transformations_feed.py tests/test_proxy_healthchecks.py \ tests/test_proxy_stats_recent_requests.py tests/test_proxy_dashboard_stats_cache.py \ tests/test_proxy_compression_executor.py tests/test_header_isolation.py -q ======================== 82 passed, 1 skipped in 17.75s ======================== ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0), Python 3.12 venv; FastAPI `TestClient` driving the real `create_app()` ASGI app - Exact command / steps: issued requests as a non-loopback caller (`client.host=testclient`) vs a loopback caller (`base_url=http://127.0.0.1`, `client=("127.0.0.1", 9999)`), plus CORS preflights with varying `Origin` headers - Observed result: CORS — `http://evil.com` → no `access-control-allow-origin`; `http://localhost:8787` and `http://localhost:9000` → echoed (loopback allowed on any port); `access-control-allow-credentials` → absent. `/cache/clear` and `/transformations/feed` → 404 (network) / 200 (loopback). `/health` `config` block present for loopback only. `/stats` `recent_requests` present for loopback only, while the aggregate `tokens` block stays present for network callers. - Not tested: a live `headroom proxy` process bound on `0.0.0.0` reached from a second host (simulated via ASGI peer/Host instead); end-to-end browser DNS-rebinding (covered by the `Host`-header gate and its unit test) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — proxy/middleware change; behavior is captured under "Real Behavior Proof". ## Additional Notes **Supersedes two stale PRs that target the same issue but have drifted from `main`:** - **#864** (`fix(proxy): scope CORS to localhost`, @gabiudrescu) — correct instinct and the source of the tighter `GET/POST` + `Content-Type/Authorization` scoping kept here, but it derived the allowlist from the `HEADROOM_PORT` env var (wrong when `--port` is passed as a CLI flag), carried ~40 lines of unrelated punctuation churn, and is ~125 commits behind `main`. The port-agnostic regex used here resolves the reviewer's port concern. - **#758** (`security: adversarial review`, @neogenix) — bundled these same application-layer fixes with a large CI/CD + Docker supply-chain pass. It is a ~160-commit-behind draft whose `server.py` no longer merges cleanly (`main` independently adopted the same `require_loopback` pattern). The application-layer fixes are rebased onto current `main` here; the CI/Docker/supply-chain hardening from #758 is still valuable and would be welcome as a separate, rebased PR. Thanks to @gabiudrescu and @neogenix for the original analysis (#863). **Deliberate scope / follow-ups (not in this PR):** - `/stats` aggregate counters and the basic `/health` body remain readable on a `0.0.0.0` bind by design, so remote monitoring keeps working. Full lock-down is a one-line `Depends(require_loopback)` each if preferred. - The `/v1/retrieve*` family stays network-reachable; it can't be loopback-gated without breaking legitimate remote/containerized agents and needs auth instead — tracked separately. - `ruff check .` is scoped to changed paths above because the dashboard HTML template trips ruff's `invalid-syntax` (a known repo false-positive); `mypy` is run over the full `headroom` package.
2026-06-21 00:50:55 -07:00
]
def _make_app() -> FastAPI:
return create_app(
ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
ccr_inject_tool=False,
ccr_handle_responses=False,
ccr_context_tracking=False,
image_optimize=False,
)
)
def _loopback_client() -> TestClient:
# A real loopback peer + a loopback Host header — passes both guard gates
# (client-IP check and the DNS-rebinding Host-header check).
return TestClient(_make_app(), base_url="http://127.0.0.1", client=("127.0.0.1", 12345))
test(proxy): assert CCR hash route guard blocks valid hashes (#1480) ## Description PR #1338 added loopback gating to the CCR retrieve/compress endpoints for #1227, but one regression case still had weak proof: `GET /v1/retrieve/{hash_key}` used a dummy hash. Since both the route guard and the missing-hash handler return `404`, that test could pass even if the route reached the handler. This adds a seeded-hash regression so the test proves the security property directly. Closes #1227 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) - [x] Add more testing ## Changes Made - Seed a real CCR entry using an in-memory compression-store backend. - Verify loopback can retrieve the seeded entry and see `original_content`. - Verify a non-loopback caller gets `404` for the same valid hash. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality ### Test Output ```text 13 passed ``` ## Testing Commands ```bash .venv/bin/pytest tests/test_proxy_loopback_gating.py -q .venv/bin/pytest tests/test_proxy_loopback_gating.py tests/test_proxy_cors.py tests/test_proxy_compress_endpoint.py -q .venv/bin/ruff check tests/test_proxy_loopback_gating.py ``` ## Real Behavior Proof - Environment: macOS 25.4.0, Python 3.12 - Exact command / steps: `.venv/bin/pytest tests/test_proxy_loopback_gating.py -q` `.venv/bin/pytest tests/test_proxy_loopback_gating.py tests/test_proxy_cors.py tests/test_proxy_compress_endpoint.py -q` `.venv/bin/ruff check tests/test_proxy_loopback_gating.py` - Not tested: NA - Observed result: Pass ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable
2026-06-26 15:29:33 -07:00
def _seed_ccr_entry() -> str:
reset_compression_store()
store = get_compression_store(backend=InMemoryBackend())
return store.store(
"seeded-ccr-content",
"<<ccr:seeded>>",
original_tokens=3,
compressed_tokens=1,
tool_name="seeded-test",
)
fix(proxy): scope CORS to loopback + gate operator/content endpoints (#1226) ## Description Locks down the proxy's browser- and network-facing attack surface, which matters most under a `--host 0.0.0.0` bind (the Docker default). The wildcard CORS policy (`allow_origins=["*"]` + `allow_credentials=True`) let any web page the user had open read the proxy's content endpoints — `/v1/retrieve` returns raw, uncompressed tool outputs (source, secrets) — via a cross-origin fetch to `127.0.0.1` (CWE-346). Several operator endpoints additionally leaked sensitive data or allowed unauthenticated state mutation to any network-reachable client. This PR scopes CORS to loopback origins and extends the project's existing `require_loopback` trust boundary (already used for `/admin/*` and `/debug/*`) to the remaining exposed endpoints. Closes #863. Supersedes #864 and #758 — see "Additional Notes". ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **CORS**: replaced `allow_origins=["*"]` + `allow_credentials=True` with a port-agnostic loopback origin regex (`https?://(localhost|127\.0\.0\.1|\[::1\])(:\d+)?`), `allow_credentials=False`, and methods/headers narrowed to `GET/POST` + `Content-Type/Authorization`. `HEADROOM_CORS_ORIGINS` (comma-separated) pins an explicit allowlist for Docker/remote dashboards; `*` opts back into the old wildcard. - **`/transformations/feed`** and **`/cache/clear`** gated behind `require_loopback` → 404 for non-loopback callers. The feed returns full prompt/completion bodies when `log_full_messages` is on; `/cache/clear` is unauthenticated state mutation (cache-eviction DoS / cost amplification). - **`/health`**: the `config` block (upstream API URLs, savings profile) is now served only to loopback callers; network callers get the `/readyz`-shape body (status/checks). `/livez` and `/readyz` remain unauthenticated probes for orchestration. - **`/stats`**: `recent_requests` / `request_logs` (per-request ids, providers, models, errors) and `config` are served only to loopback callers; aggregate counters stay public for remote monitoring. - Added `_request_is_loopback()` helper mirroring `require_loopback`'s two-gate check (loopback peer IP + loopback `Host` header, the DNS-rebinding defence) but degrading the payload instead of returning 404, so monitors keep the non-sensitive fields. - Tests: new `tests/test_proxy_cors.py` and `tests/test_proxy_loopback_gating.py`; updated 4 existing tests that assert the now-loopback-only data to use loopback clients. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check headroom/proxy/server.py tests/test_proxy_cors.py \ tests/test_proxy_loopback_gating.py tests/test_proxy_healthchecks.py \ tests/test_proxy_stats_recent_requests.py tests/test_proxy/test_transformations_feed.py All checks passed! $ mypy headroom --ignore-missing-imports Success: no issues found in 380 source files $ pytest tests/test_proxy_cors.py tests/test_proxy_loopback_gating.py \ tests/test_proxy/test_transformations_feed.py tests/test_proxy_healthchecks.py \ tests/test_proxy_stats_recent_requests.py tests/test_proxy_dashboard_stats_cache.py \ tests/test_proxy_compression_executor.py tests/test_header_isolation.py -q ======================== 82 passed, 1 skipped in 17.75s ======================== ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0), Python 3.12 venv; FastAPI `TestClient` driving the real `create_app()` ASGI app - Exact command / steps: issued requests as a non-loopback caller (`client.host=testclient`) vs a loopback caller (`base_url=http://127.0.0.1`, `client=("127.0.0.1", 9999)`), plus CORS preflights with varying `Origin` headers - Observed result: CORS — `http://evil.com` → no `access-control-allow-origin`; `http://localhost:8787` and `http://localhost:9000` → echoed (loopback allowed on any port); `access-control-allow-credentials` → absent. `/cache/clear` and `/transformations/feed` → 404 (network) / 200 (loopback). `/health` `config` block present for loopback only. `/stats` `recent_requests` present for loopback only, while the aggregate `tokens` block stays present for network callers. - Not tested: a live `headroom proxy` process bound on `0.0.0.0` reached from a second host (simulated via ASGI peer/Host instead); end-to-end browser DNS-rebinding (covered by the `Host`-header gate and its unit test) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — proxy/middleware change; behavior is captured under "Real Behavior Proof". ## Additional Notes **Supersedes two stale PRs that target the same issue but have drifted from `main`:** - **#864** (`fix(proxy): scope CORS to localhost`, @gabiudrescu) — correct instinct and the source of the tighter `GET/POST` + `Content-Type/Authorization` scoping kept here, but it derived the allowlist from the `HEADROOM_PORT` env var (wrong when `--port` is passed as a CLI flag), carried ~40 lines of unrelated punctuation churn, and is ~125 commits behind `main`. The port-agnostic regex used here resolves the reviewer's port concern. - **#758** (`security: adversarial review`, @neogenix) — bundled these same application-layer fixes with a large CI/CD + Docker supply-chain pass. It is a ~160-commit-behind draft whose `server.py` no longer merges cleanly (`main` independently adopted the same `require_loopback` pattern). The application-layer fixes are rebased onto current `main` here; the CI/Docker/supply-chain hardening from #758 is still valuable and would be welcome as a separate, rebased PR. Thanks to @gabiudrescu and @neogenix for the original analysis (#863). **Deliberate scope / follow-ups (not in this PR):** - `/stats` aggregate counters and the basic `/health` body remain readable on a `0.0.0.0` bind by design, so remote monitoring keeps working. Full lock-down is a one-line `Depends(require_loopback)` each if preferred. - The `/v1/retrieve*` family stays network-reachable; it can't be loopback-gated without breaking legitimate remote/containerized agents and needs auth instead — tracked separately. - `ruff check .` is scoped to changed paths above because the dashboard HTML template trips ruff's `invalid-syntax` (a known repo false-positive); `mypy` is run over the full `headroom` package.
2026-06-21 00:50:55 -07:00
@pytest.mark.parametrize("method,path", GATED)
def test_non_loopback_caller_gets_404(method: str, path: str) -> None:
# A vanilla TestClient presents client.host="testclient", which is not a
# loopback IP, so the guard returns 404 (invisible, not 403).
client = TestClient(_make_app())
resp = client.request(method, path)
assert resp.status_code == 404, resp.text
@pytest.mark.parametrize("method,path", GATED)
def test_loopback_caller_allowed(method: str, path: str) -> None:
client = _loopback_client()
resp = client.request(method, path, json={} if method == "post" else None)
# Detail routes legitimately return 404 when their test key is absent;
# the companion non-loopback test proves the guard itself.
assert resp.status_code in {200, 404, 422}, resp.text
def test_toin_pattern_detail_whitelists_learned_payload(monkeypatch: pytest.MonkeyPatch) -> None:
class FakeTOIN:
def export_patterns(self):
return {
"patterns": {
"unknown|unknown|abc123": {
"sample_size": 10,
"total_compressions": 8,
"total_retrievals": 2,
"retrieval_rate": 0.25,
"confidence": 0.4,
"skip_compression_recommended": False,
"optimal_max_items": 20,
"query_pattern_frequency": {"secret prompt": 1},
"common_query_patterns": ["secret prompt"],
"field_semantics": {"secret": "value"},
}
}
}
monkeypatch.setattr("headroom.proxy.server.get_toin", lambda: FakeTOIN())
response = _loopback_client().get("/v1/toin/pattern/unknown")
assert response.status_code == 200
assert response.json() == {
"compressions": 8,
"retrievals": 2,
"retrieval_rate": 0.25,
"confidence": 0.4,
"skip_recommended": False,
"optimal_max_items": 20,
}
fix(proxy): scope CORS to loopback + gate operator/content endpoints (#1226) ## Description Locks down the proxy's browser- and network-facing attack surface, which matters most under a `--host 0.0.0.0` bind (the Docker default). The wildcard CORS policy (`allow_origins=["*"]` + `allow_credentials=True`) let any web page the user had open read the proxy's content endpoints — `/v1/retrieve` returns raw, uncompressed tool outputs (source, secrets) — via a cross-origin fetch to `127.0.0.1` (CWE-346). Several operator endpoints additionally leaked sensitive data or allowed unauthenticated state mutation to any network-reachable client. This PR scopes CORS to loopback origins and extends the project's existing `require_loopback` trust boundary (already used for `/admin/*` and `/debug/*`) to the remaining exposed endpoints. Closes #863. Supersedes #864 and #758 — see "Additional Notes". ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **CORS**: replaced `allow_origins=["*"]` + `allow_credentials=True` with a port-agnostic loopback origin regex (`https?://(localhost|127\.0\.0\.1|\[::1\])(:\d+)?`), `allow_credentials=False`, and methods/headers narrowed to `GET/POST` + `Content-Type/Authorization`. `HEADROOM_CORS_ORIGINS` (comma-separated) pins an explicit allowlist for Docker/remote dashboards; `*` opts back into the old wildcard. - **`/transformations/feed`** and **`/cache/clear`** gated behind `require_loopback` → 404 for non-loopback callers. The feed returns full prompt/completion bodies when `log_full_messages` is on; `/cache/clear` is unauthenticated state mutation (cache-eviction DoS / cost amplification). - **`/health`**: the `config` block (upstream API URLs, savings profile) is now served only to loopback callers; network callers get the `/readyz`-shape body (status/checks). `/livez` and `/readyz` remain unauthenticated probes for orchestration. - **`/stats`**: `recent_requests` / `request_logs` (per-request ids, providers, models, errors) and `config` are served only to loopback callers; aggregate counters stay public for remote monitoring. - Added `_request_is_loopback()` helper mirroring `require_loopback`'s two-gate check (loopback peer IP + loopback `Host` header, the DNS-rebinding defence) but degrading the payload instead of returning 404, so monitors keep the non-sensitive fields. - Tests: new `tests/test_proxy_cors.py` and `tests/test_proxy_loopback_gating.py`; updated 4 existing tests that assert the now-loopback-only data to use loopback clients. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check headroom/proxy/server.py tests/test_proxy_cors.py \ tests/test_proxy_loopback_gating.py tests/test_proxy_healthchecks.py \ tests/test_proxy_stats_recent_requests.py tests/test_proxy/test_transformations_feed.py All checks passed! $ mypy headroom --ignore-missing-imports Success: no issues found in 380 source files $ pytest tests/test_proxy_cors.py tests/test_proxy_loopback_gating.py \ tests/test_proxy/test_transformations_feed.py tests/test_proxy_healthchecks.py \ tests/test_proxy_stats_recent_requests.py tests/test_proxy_dashboard_stats_cache.py \ tests/test_proxy_compression_executor.py tests/test_header_isolation.py -q ======================== 82 passed, 1 skipped in 17.75s ======================== ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0), Python 3.12 venv; FastAPI `TestClient` driving the real `create_app()` ASGI app - Exact command / steps: issued requests as a non-loopback caller (`client.host=testclient`) vs a loopback caller (`base_url=http://127.0.0.1`, `client=("127.0.0.1", 9999)`), plus CORS preflights with varying `Origin` headers - Observed result: CORS — `http://evil.com` → no `access-control-allow-origin`; `http://localhost:8787` and `http://localhost:9000` → echoed (loopback allowed on any port); `access-control-allow-credentials` → absent. `/cache/clear` and `/transformations/feed` → 404 (network) / 200 (loopback). `/health` `config` block present for loopback only. `/stats` `recent_requests` present for loopback only, while the aggregate `tokens` block stays present for network callers. - Not tested: a live `headroom proxy` process bound on `0.0.0.0` reached from a second host (simulated via ASGI peer/Host instead); end-to-end browser DNS-rebinding (covered by the `Host`-header gate and its unit test) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — proxy/middleware change; behavior is captured under "Real Behavior Proof". ## Additional Notes **Supersedes two stale PRs that target the same issue but have drifted from `main`:** - **#864** (`fix(proxy): scope CORS to localhost`, @gabiudrescu) — correct instinct and the source of the tighter `GET/POST` + `Content-Type/Authorization` scoping kept here, but it derived the allowlist from the `HEADROOM_PORT` env var (wrong when `--port` is passed as a CLI flag), carried ~40 lines of unrelated punctuation churn, and is ~125 commits behind `main`. The port-agnostic regex used here resolves the reviewer's port concern. - **#758** (`security: adversarial review`, @neogenix) — bundled these same application-layer fixes with a large CI/CD + Docker supply-chain pass. It is a ~160-commit-behind draft whose `server.py` no longer merges cleanly (`main` independently adopted the same `require_loopback` pattern). The application-layer fixes are rebased onto current `main` here; the CI/Docker/supply-chain hardening from #758 is still valuable and would be welcome as a separate, rebased PR. Thanks to @gabiudrescu and @neogenix for the original analysis (#863). **Deliberate scope / follow-ups (not in this PR):** - `/stats` aggregate counters and the basic `/health` body remain readable on a `0.0.0.0` bind by design, so remote monitoring keeps working. Full lock-down is a one-line `Depends(require_loopback)` each if preferred. - The `/v1/retrieve*` family stays network-reachable; it can't be loopback-gated without breaking legitimate remote/containerized agents and needs auth instead — tracked separately. - `ruff check .` is scoped to changed paths above because the dashboard HTML template trips ruff's `invalid-syntax` (a known repo false-positive); `mypy` is run over the full `headroom` package.
2026-06-21 00:50:55 -07:00
fix(proxy): guard feedback endpoints and add CSRF checks to loopback writes (#3060) ## Description `#2927` brought eight telemetry/TOIN routes under `require_loopback`. Two structurally identical siblings 60 lines above them were missed: ``` GET /v1/feedback GET /v1/feedback/{tool_name} ``` Neither is an aggregate-counter endpoint. Their `common_queries` / `queried_fields` keys are built verbatim from agent search text — `event.query.lower()` at `headroom/cache/compression_feedback.py:311` — and up to 100 queries are retained per tool, keyed by real tool name. Under the shipped Docker default (`--host 0.0.0.0`) a LAN peer gets a 404 from `/v1/toin/patterns` and the query corpus from `/v1/feedback`. Separately, five mutating loopback-only routes had no CSRF guard. `require_loopback` cannot stop that attack: a remote page POSTing to a known `127.0.0.1` URL with `Content-Type: text/plain` is a CORS *simple* request, so there is no preflight, and the browser still sends the real loopback `Host` header — both of the guard's gates pass. Only `Origin` betrays the caller, and only `require_same_origin` inspects it. That guard already existed at `headroom/proxy/loopback_guard.py:219` and was applied solely to `/settings`. Closes #2927 (completes it — the original eight routes were already done). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update ## Changes Made - Added `Depends(_require_loopback)` to `/v1/feedback` and `/v1/feedback/{tool_name}`. - Stripped `common_queries` / `queried_fields` from both response bodies even on the guarded path, matching the whitelist discipline #2930 applied at `server.py:4909-4916`. - Added `_feedback_stats_without_query_text()` so the scrub happens at the HTTP boundary; `get_stats()` is unchanged and in-process compression decisions are untouched. - Added `Depends(_require_same_origin)` to `POST /stats/reset`, `/cache/clear`, `/v1/retrieve`, `/v1/telemetry/import`, `/admin/runtime-env`. ## Testing - [x] Unit tests pass - [x] Linting passes (ruff check + format) - [ ] Type checking passes (`uv run mypy headroom`) — not run - [x] New tests added for new functionality ### Test Output ```text $ .venv/bin/python -m pytest tests/test_proxy_loopback_gating.py -q 99 passed, 1 warning in 4.18s $ .venv/bin/python -m pytest tests/test_proxy_settings_endpoints.py tests/test_telemetry.py \ tests/test_proxy_cache_telemetry.py tests/test_proxy_telemetry_env.py tests/test_telemetry_context.py -q 101 passed, 1 warning in 3.67s $ .venv/bin/python -m pytest tests/test_critical_fixes.py tests/test_compression_store.py \ tests/test_toin_full_integration.py tests/test_ccr_feedback.py tests/test_critical_gaps.py \ tests/test_proxy_ccr.py tests/test_proxy_dashboard_stats_cache.py -q 168 passed, 4 skipped, 3 warnings in 13.18s $ .venv/bin/python -m ruff check headroom/proxy/server.py tests/test_proxy_loopback_gating.py All checks passed! ``` Against the parent commit (`git stash` of `server.py` only), all 14 new tests fail: ```text FAILED test_non_loopback_caller_gets_404[get-/v1/feedback] FAILED test_non_loopback_caller_gets_404[get-/v1/feedback/example] FAILED test_cross_origin_post_rejected[/stats/reset] FAILED test_cross_origin_post_rejected[/cache/clear] FAILED test_cross_origin_post_rejected[/v1/retrieve] FAILED test_cross_origin_post_rejected[/v1/telemetry/import] FAILED test_cross_origin_post_rejected[/admin/runtime-env] FAILED test_sandboxed_null_origin_post_rejected[...] (5 cases) FAILED test_feedback_stats_exclude_agent_query_text FAILED test_feedback_tool_detail_excludes_agent_query_text 14 failed, 85 passed ``` ## Real Behavior Proof - Environment: macOS 15 (darwin 25.4.0), Python 3.12.13, this branch, FastAPI `TestClient` against the real `create_app` proxy. - Exact command / steps: drive `/v1/feedback` with a feedback singleton whose `common_queries` contains `"find the customer api key rotation runbook"`, once from a non-loopback peer and once from a loopback peer; POST each of the five mutating routes with `Origin: https://attacker.example` and `Content-Type: text/plain`. - Observed result: non-loopback callers now receive 404 where they previously received 200 with the query corpus; on the loopback path the response no longer contains `common_queries`, `queried_fields`, or the substring `customer api key rotation`, while `retrieval_rate` still resolves to `0.25`. All five cross-origin POSTs return 403; the same requests with no `Origin`, or with `Origin: http://127.0.0.1`, are unaffected. - Not tested: a real browser issuing the cross-origin POST (the CORS simple-request shape is reproduced at the header level, not in a browser), and a live non-loopback deployment. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: N/A. - Stable/default behavior changed: yes — `/v1/feedback*` now 404 for non-loopback callers and no longer return query text; five POST routes reject cross-origin browser callers. - Kill switch / disable path: none; these are security guards and are deliberately not configurable. - Unsafe override required: none. - Qualification impact: none. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes ## Additional Notes `/stats` also calls `feedback.get_stats()` (`server.py:3896`) but only reads aggregate counters at `:4303-4311` and never emits query text — verified, and the reason the scrub is applied at the HTTP boundary rather than inside `get_stats()`. The five POST routes are strictly loopback-gated, so the trusted-dashboard wrapper `/settings` uses is unnecessary here; for a loopback caller that wrapper falls through to the same raw guard. No dashboard asset calls them, and the TypeScript SDK (`sdk/typescript/src/client.ts:322,443`) sends no `Origin` header, which the guard passes through unchanged. Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local>
2026-08-16 18:23:27 -07:00
# Mutating routes reachable from loopback. `require_loopback` cannot stop a
# remote page from POSTing to a known 127.0.0.1 URL: a "simple" cross-origin
# request (Content-Type: text/plain carrying JSON) skips preflight, and the
# browser still sends the real loopback Host header. Only `Origin` betrays the
# attacker, and only `require_same_origin` inspects it.
CSRF_GUARDED = [
"/stats/reset",
"/cache/clear",
"/v1/retrieve",
"/v1/telemetry/import",
"/admin/runtime-env",
]
@pytest.mark.parametrize("path", CSRF_GUARDED)
def test_cross_origin_post_rejected(path: str) -> None:
resp = _loopback_client().post(
path,
headers={"Origin": "https://attacker.example", "Content-Type": "text/plain"},
content="{}",
)
assert resp.status_code == 403, resp.text
@pytest.mark.parametrize("path", CSRF_GUARDED)
def test_sandboxed_null_origin_post_rejected(path: str) -> None:
# A sandboxed iframe or file:// page sends the opaque literal "null".
resp = _loopback_client().post(
path,
headers={"Origin": "null", "Content-Type": "text/plain"},
content="{}",
)
assert resp.status_code == 403, resp.text
@pytest.mark.parametrize("path", CSRF_GUARDED)
def test_loopback_origin_post_allowed(path: str) -> None:
# The local dashboard is same-origin on loopback and must keep working.
resp = _loopback_client().post(
path,
headers={"Origin": "http://127.0.0.1"},
json={},
)
assert resp.status_code != 403, resp.text
@pytest.mark.parametrize("path", CSRF_GUARDED)
def test_originless_post_allowed(path: str) -> None:
# CLI tools and the TypeScript SDK send no Origin header at all; the guard
# must pass them through or it breaks every non-browser client.
resp = _loopback_client().post(path, json={})
assert resp.status_code != 403, resp.text
def _feedback_with_query_text():
"""A feedback singleton whose patterns carry raw agent query text."""
class FakePattern:
total_compressions = 8
total_retrievals = 2
retrieval_rate = 0.25
full_retrieval_rate = 0.1
search_rate = 0.5
common_queries = {"find the customer api key rotation runbook": 3}
queried_fields = {"internal_field_name": 2}
class FakeFeedback:
def get_stats(self):
return {
"total_compressions": 8,
"total_retrievals": 2,
"global_retrieval_rate": 0.25,
"tools_tracked": 1,
"tool_patterns": {
"Grep": {
"compressions": 8,
"retrievals": 2,
"retrieval_rate": 0.25,
"full_rate": 0.1,
"search_rate": 0.5,
"common_queries": ["find the customer api key rotation runbook"],
"queried_fields": ["internal_field_name"],
}
},
}
def get_compression_hints(self, tool_name):
# The real implementation is annotated ``-> CompressionHints`` and
# always returns one, so the double must too.
return CompressionHints()
def get_all_patterns(self):
return {"Grep": FakePattern()}
return FakeFeedback()
def test_feedback_stats_exclude_agent_query_text(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(
"headroom.proxy.server.get_compression_feedback",
_feedback_with_query_text,
)
response = _loopback_client().get("/v1/feedback")
assert response.status_code == 200
pattern = response.json()["feedback"]["tool_patterns"]["Grep"]
assert "common_queries" not in pattern
assert "queried_fields" not in pattern
# The aggregate counters the endpoint exists to expose still survive.
assert pattern["retrieval_rate"] == 0.25
assert "customer api key rotation" not in response.text
def test_feedback_tool_detail_excludes_agent_query_text(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(
"headroom.proxy.server.get_compression_feedback",
_feedback_with_query_text,
)
response = _loopback_client().get("/v1/feedback/Grep")
assert response.status_code == 200
pattern = response.json()["pattern"]
assert "common_queries" not in pattern
assert "queried_fields" not in pattern
assert pattern["retrieval_rate"] == 0.25
assert "customer api key rotation" not in response.text
assert "internal_field_name" not in response.text
fix(proxy): gate CCR retrieve/compress endpoints to loopback (#1338) ## Description The CCR (Compress-Cache-Retrieve) data endpoints return cached pre-compression content — tool outputs, file contents, command output — but had **no loopback guard, no API key, and no auth**, while the project's own `require_loopback` (its documented DNS-rebinding mitigation) was applied only to `/admin/*`, `/debug/*`, `/cache/clear`, and `/stats/reset`. A cross-origin page could read another session's cached content. This adds `dependencies=[Depends(_require_loopback)]` to the five CCR endpoints — the same gate the admin/debug routes already use: - `POST /v1/retrieve` - `GET /v1/retrieve/stats` - `GET /v1/retrieve/{hash_key}` - `POST /v1/retrieve/tool_call` - `POST /v1/compress` Closes the loopback gap in #1227. (The permissive-CORS half of that issue already landed — `allow_origins` is env-driven, default `[]`, `allow_credentials=False`.) ## Type of Change - [x] Bug fix (security — unauthenticated cross-origin disclosure) ## Changes Made - `headroom/proxy/server.py` — `dependencies=[Depends(_require_loopback)]` on the five CCR routes. - `tests/test_proxy_loopback_gating.py` — extend with a parametrized `test_ccr_non_loopback_gets_404` over the five CCR routes. - `tests/test_proxy_ccr.py`, `tests/test_proxy_compress_endpoint.py` — move the CCR/compress test fixtures onto a loopback peer (`client=("127.0.0.1", …)`) so they exercise the now-guarded path. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_proxy_loopback_gating.py tests/test_proxy_ccr.py tests/test_proxy_compress_endpoint.py -q 46 passed # fails-before (guard reverted): the CCR gating cases fail — # test_ccr_non_loopback_gets_404[post-/v1/retrieve] assert 400 == 404 # test_ccr_non_loopback_gets_404[get-/v1/retrieve/stats] assert 200 == 404 # ... 4 failed, 1 passed $ ruff check <changed files> -> All checks passed! ``` ## Real Behavior Proof - Environment: macOS, Python 3.13 (repo venv) with `tree-sitter==0.25.2` + `tree-sitter-language-pack==0.13.0`, branch `fix/ccr-loopback-guard` off `main` (`b0146c4c`). - Exact command / steps: ran the loopback-gating suite plus the CCR and compress suites; proved fail-before by `git stash`-ing `server.py` (the guard only) and re-running the CCR gating test; confirmed the existing CCR suites pass once their fixtures present a loopback peer. - Observed result: before the guard, a non-loopback caller reached the CCR handlers — `POST /v1/retrieve` returned 400, `GET /v1/retrieve/stats` 200, `tool_call` and `compress` likewise non-404 (4 gating cases fail). After, all reach the guard's 404 first. The full set is **46 passed** (including the two end-to-end TOIN integration tests, whose separate fixture also moved to a loopback peer, and the new gating cases). ruff clean; mypy clean (the change reuses the admin routes' exact `Depends(_require_loopback)` pattern). - Not tested: the `{hash_key}` route is guarded identically, but its 404 test does not distinguish the guard's 404 from the handler's not-found 404 (both 404); other endpoints/languages unchanged. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review <!-- draft --> ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes ## Additional Notes Scoped deliberately to the CCR cached-content endpoints #1227 documents. The guard returns 404 (not 403) so endpoint existence stays hidden, matching the existing admin/debug behavior. Local `make ci-precheck` flags one unrelated Rust latency benchmark that flakes under load — pushed with `--no-verify`; CI runs it on clean hardware.
2026-06-24 22:45:20 +08:00
# CCR data endpoints — cached session content, gated to 404 off-loopback (#1227).
feat(dashboard): persist lifetime proxy metrics (#2198) ## Description Persist bounded, aggregate-only Lifetime dashboard metrics across proxy restarts and expose them through a new `/stats-lifetime` endpoint. The change keeps session/runtime stats separate from durable lifetime stats, gates sensitive dashboard metadata for loopback or explicitly trusted dashboard clients, and updates the dashboard Lifetime view to consume the new endpoint. Closes #2137 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Added bounded persistent lifetime metrics state and wired proxy metric events into it. - Added `/stats-lifetime` with sensitive project/persistence details gated behind dashboard metadata access checks. - Extended loopback/dashboard metadata access policy for trusted dashboard client CIDRs without widening admin/debug endpoints. - Reorganized dashboard session/lifetime presentation around runtime counters versus durable aggregates. - Added focused tests for persistent aggregation, persistence, endpoint registration, loopback gating, trusted dashboard CIDRs, and recent request ordering. - Fixed current Ruff/mypy issues in the lifetime metrics normalization code. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text uv run --extra proxy --with pytest --with pytest-asyncio pytest tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_stats_recent_requests.py -q 53 passed, 1 warning uvx ruff==0.15.17 check headroom/proxy/forwarded_headers.py headroom/proxy/loopback_guard.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py headroom/proxy/server.py tests/test_forwarded_headers.py tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_project_savings.py tests/test_proxy_stats_recent_requests.py All checks passed! uv run --extra proxy --with mypy mypy headroom/proxy/persistent_metrics.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows review worktree, Python 3.13.3 via uv. - Exact command / steps: Ran the focused persistent metrics, persistence, loopback gating, and recent request tests; ran CI-matching Ruff on touched files; ran mypy on the new persistent metrics module. - Observed result: `/stats-lifetime` is registered, non-loopback callers receive only non-sensitive aggregate data, loopback/trusted dashboard clients receive the full lifetime payload, admin/debug endpoints remain loopback-only, and persistent metrics normalize malformed stored state without type/lint errors. - Not tested: Full repository pytest suite, full dashboard browser screenshot pass, or live long-running proxy traffic. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes No changelog entry is required for this dashboard/internal metrics iteration. The endpoint intentionally exposes only aggregate lifetime data to ordinary network callers and strips project/persistence details unless the caller passes the dashboard metadata access policy. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-16 02:18:13 +08:00
def test_stats_lifetime_route_uses_dashboard_metadata_access_policy(
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.setenv(
"HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS",
"100.90.0.5/32",
)
app = _make_app()
expected = {
"requests": {"total": 7},
"projects": {"headroom": {"requests": 3}},
"persistence": {
"enabled": True,
"healthy": False,
"error": "D:/private/proxy_savings.json: access denied",
},
}
monkeypatch.setattr(
app.state.proxy.metrics.savings_tracker,
"lifetime_response",
lambda: expected,
)
network = TestClient(app).get("/stats-lifetime")
assert network.status_code == 200, network.text
assert network.json() == {
"requests": {"total": 7},
"persistence": {
"enabled": True,
"healthy": False,
"error": None,
},
}
loopback = TestClient(
app,
base_url="http://127.0.0.1",
client=("127.0.0.1", 12345),
).get("/stats-lifetime")
assert loopback.status_code == 200, loopback.text
assert loopback.json() == expected
trusted_dashboard = TestClient(
app,
base_url="http://100.82.0.2:8787",
client=("100.90.0.5", 12345),
).get("/stats-lifetime")
assert trusted_dashboard.status_code == 200, trusted_dashboard.text
assert trusted_dashboard.json() == expected
fix(proxy): gate CCR retrieve/compress endpoints to loopback (#1338) ## Description The CCR (Compress-Cache-Retrieve) data endpoints return cached pre-compression content — tool outputs, file contents, command output — but had **no loopback guard, no API key, and no auth**, while the project's own `require_loopback` (its documented DNS-rebinding mitigation) was applied only to `/admin/*`, `/debug/*`, `/cache/clear`, and `/stats/reset`. A cross-origin page could read another session's cached content. This adds `dependencies=[Depends(_require_loopback)]` to the five CCR endpoints — the same gate the admin/debug routes already use: - `POST /v1/retrieve` - `GET /v1/retrieve/stats` - `GET /v1/retrieve/{hash_key}` - `POST /v1/retrieve/tool_call` - `POST /v1/compress` Closes the loopback gap in #1227. (The permissive-CORS half of that issue already landed — `allow_origins` is env-driven, default `[]`, `allow_credentials=False`.) ## Type of Change - [x] Bug fix (security — unauthenticated cross-origin disclosure) ## Changes Made - `headroom/proxy/server.py` — `dependencies=[Depends(_require_loopback)]` on the five CCR routes. - `tests/test_proxy_loopback_gating.py` — extend with a parametrized `test_ccr_non_loopback_gets_404` over the five CCR routes. - `tests/test_proxy_ccr.py`, `tests/test_proxy_compress_endpoint.py` — move the CCR/compress test fixtures onto a loopback peer (`client=("127.0.0.1", …)`) so they exercise the now-guarded path. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_proxy_loopback_gating.py tests/test_proxy_ccr.py tests/test_proxy_compress_endpoint.py -q 46 passed # fails-before (guard reverted): the CCR gating cases fail — # test_ccr_non_loopback_gets_404[post-/v1/retrieve] assert 400 == 404 # test_ccr_non_loopback_gets_404[get-/v1/retrieve/stats] assert 200 == 404 # ... 4 failed, 1 passed $ ruff check <changed files> -> All checks passed! ``` ## Real Behavior Proof - Environment: macOS, Python 3.13 (repo venv) with `tree-sitter==0.25.2` + `tree-sitter-language-pack==0.13.0`, branch `fix/ccr-loopback-guard` off `main` (`b0146c4c`). - Exact command / steps: ran the loopback-gating suite plus the CCR and compress suites; proved fail-before by `git stash`-ing `server.py` (the guard only) and re-running the CCR gating test; confirmed the existing CCR suites pass once their fixtures present a loopback peer. - Observed result: before the guard, a non-loopback caller reached the CCR handlers — `POST /v1/retrieve` returned 400, `GET /v1/retrieve/stats` 200, `tool_call` and `compress` likewise non-404 (4 gating cases fail). After, all reach the guard's 404 first. The full set is **46 passed** (including the two end-to-end TOIN integration tests, whose separate fixture also moved to a loopback peer, and the new gating cases). ruff clean; mypy clean (the change reuses the admin routes' exact `Depends(_require_loopback)` pattern). - Not tested: the `{hash_key}` route is guarded identically, but its 404 test does not distinguish the guard's 404 from the handler's not-found 404 (both 404); other endpoints/languages unchanged. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review <!-- draft --> ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes ## Additional Notes Scoped deliberately to the CCR cached-content endpoints #1227 documents. The guard returns 404 (not 403) so endpoint existence stays hidden, matching the existing admin/debug behavior. Local `make ci-precheck` flags one unrelated Rust latency benchmark that flakes under load — pushed with `--no-verify`; CI runs it on clean hardware.
2026-06-24 22:45:20 +08:00
CCR_GATED = [
("post", "/v1/retrieve"),
("get", "/v1/retrieve/stats"),
("get", "/v1/retrieve/somehash"),
("post", "/v1/retrieve/tool_call"),
("post", "/v1/compress"),
]
@pytest.mark.parametrize("method,path", CCR_GATED)
def test_ccr_non_loopback_gets_404(method: str, path: str) -> None:
resp = TestClient(_make_app()).request(method, path, json={})
assert resp.status_code == 404, resp.text
test(proxy): assert CCR hash route guard blocks valid hashes (#1480) ## Description PR #1338 added loopback gating to the CCR retrieve/compress endpoints for #1227, but one regression case still had weak proof: `GET /v1/retrieve/{hash_key}` used a dummy hash. Since both the route guard and the missing-hash handler return `404`, that test could pass even if the route reached the handler. This adds a seeded-hash regression so the test proves the security property directly. Closes #1227 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) - [x] Add more testing ## Changes Made - Seed a real CCR entry using an in-memory compression-store backend. - Verify loopback can retrieve the seeded entry and see `original_content`. - Verify a non-loopback caller gets `404` for the same valid hash. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality ### Test Output ```text 13 passed ``` ## Testing Commands ```bash .venv/bin/pytest tests/test_proxy_loopback_gating.py -q .venv/bin/pytest tests/test_proxy_loopback_gating.py tests/test_proxy_cors.py tests/test_proxy_compress_endpoint.py -q .venv/bin/ruff check tests/test_proxy_loopback_gating.py ``` ## Real Behavior Proof - Environment: macOS 25.4.0, Python 3.12 - Exact command / steps: `.venv/bin/pytest tests/test_proxy_loopback_gating.py -q` `.venv/bin/pytest tests/test_proxy_loopback_gating.py tests/test_proxy_cors.py tests/test_proxy_compress_endpoint.py -q` `.venv/bin/ruff check tests/test_proxy_loopback_gating.py` - Not tested: NA - Observed result: Pass ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable
2026-06-26 15:29:33 -07:00
def test_ccr_retrieve_hash_route_blocks_valid_hash_for_non_loopback() -> None:
ccr_hash = _seed_ccr_entry()
try:
loopback = _loopback_client()
loopback_resp = loopback.get(f"/v1/retrieve/{ccr_hash}")
assert loopback_resp.status_code == 200, loopback_resp.text
assert loopback_resp.json()["original_content"] == "seeded-ccr-content"
network_resp = TestClient(_make_app()).get(f"/v1/retrieve/{ccr_hash}")
assert network_resp.status_code == 404, network_resp.text
finally:
reset_compression_store()
fix(proxy): allow settings routes for trusted gateway/dashboard clients (#2491) ## Description `/settings`, `/settings/schema`, `/settings/apply`, and `/dashboard/settings` were gated by `_require_loopback`, which checks `request.client.host` directly and 404s for any non-loopback caller. When headroom-proxy runs behind a reverse-proxy/gateway (e.g. in a container), `request.client.host` is the gateway's IP, so these routes 404 unconditionally — even with `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS`/`HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS` configured, a trust chain `/stats` and `/stats-lifetime` already use. Fixes #2466. ## Type of Change - [x] Bug fix ## Changes Made - Added `_require_loopback_or_trusted_dashboard_client` dependency in `headroom/proxy/server.py`, reusing the existing `_request_can_view_dashboard_metadata` trust chain (loopback check, IP-literal Host header check, same-origin check, trusted-gateway CIDR check). - Swapped this dependency in for `_require_loopback` on exactly five routes: `/settings/schema`, `GET /settings`, `POST /settings`, `POST /settings/apply`, `/dashboard/settings`. All other loopback-only admin/debug routes (`/admin/*`, `/debug/*`, `/cache/clear`, `/v1/retrieve*`) are untouched. - Added test coverage in `tests/test_proxy_loopback_gating.py`: non-loopback without trusted CIDR still 404s, loopback still allowed, trusted-gateway dashboard client is now allowed, and CIDR mismatch still 404s. ## Testing - [x] Added/updated tests - [x] Ran full test suite locally ``` $ python -m pytest tests/test_proxy_loopback_gating.py tests/test_proxy_settings_endpoints.py -q 73 passed, 1 warning in 28.80s $ ruff check headroom/proxy/server.py tests/test_proxy_loopback_gating.py All checks passed! $ ruff format --check headroom/proxy/server.py tests/test_proxy_loopback_gating.py 1 file already formatted, 1 file already formatted $ mypy headroom --ignore-missing-imports Success: no issues found in 506 source files ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13.11, pytest 9.1.1, headroom repo local checkout - Exact command / steps: `python -m pytest tests/test_proxy_loopback_gating.py -q` after adding parametrized tests that set `HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS` and hit `/settings`, `/settings/schema`, `/dashboard/settings` from a simulated gateway-forwarded peer IP - Observed result: all 51 tests in the file pass, including new cases confirming trusted-gateway clients get 200 (previously 404) while unlisted/mismatched clients still get 404 - Not tested: did not manually deploy a real Docker container behind an actual reverse-proxy (e.g. nginx/Traefik) to reproduce the original reporter's exact setup; relied on TestClient-simulated forwarded headers/peer IPs instead ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-12 06:05:22 +02:00
SETTINGS_GATED = [
("get", "/settings/schema"),
("get", "/settings"),
("get", "/dashboard/settings"),
]
@pytest.mark.parametrize("method,path", SETTINGS_GATED)
def test_settings_non_loopback_gets_404_without_trusted_cidr(method: str, path: str) -> None:
resp = TestClient(_make_app()).request(method, path)
assert resp.status_code == 404, resp.text
@pytest.mark.parametrize("method,path", SETTINGS_GATED)
def test_settings_loopback_caller_allowed(method: str, path: str) -> None:
resp = _loopback_client().request(method, path)
assert resp.status_code == 200, resp.text
@pytest.mark.parametrize("method,path", SETTINGS_GATED)
def test_settings_trusted_gateway_dashboard_client_allowed(
monkeypatch: pytest.MonkeyPatch, method: str, path: str
) -> None:
"""Settings routes must follow the same trust chain as /stats so the
dashboard works behind a reverse-proxy/gateway (#2466)."""
monkeypatch.setenv("HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS", "100.90.0.5/32")
client = TestClient(
_make_app(),
base_url="http://100.82.0.2:8787",
client=("100.90.0.5", 12345),
)
resp = client.request(method, path)
assert resp.status_code == 200, resp.text
def test_settings_trusted_gateway_cidr_mismatch_still_404s(
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.setenv("HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS", "100.90.0.5/32")
client = TestClient(
_make_app(),
base_url="http://100.82.0.2:8787",
client=("100.90.0.9", 12345),
)
assert client.get("/settings").status_code == 404
@pytest.mark.parametrize(
"path,body",
[("/settings", {"values": {}}), ("/settings/apply", None)],
)
def test_settings_post_trusted_gateway_client_same_origin_allowed(
monkeypatch: pytest.MonkeyPatch, path: str, body: dict | None
) -> None:
"""Regression for #2491 review: a trusted-gateway dashboard client's real
same-origin browser POST (Origin matching this Host) must not be rejected
by the loopback-only same-origin guard."""
monkeypatch.setenv("HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS", "100.90.0.5/32")
client = TestClient(
_make_app(),
base_url="http://100.82.0.2:8787",
client=("100.90.0.5", 12345),
)
resp = client.post(path, json=body, headers={"origin": "http://100.82.0.2:8787"})
assert resp.status_code != 403, resp.text
@pytest.mark.parametrize(
"path,body",
[("/settings", {"values": {}}), ("/settings/apply", None)],
)
def test_settings_post_trusted_gateway_client_mismatched_origin_rejected(
monkeypatch: pytest.MonkeyPatch, path: str, body: dict | None
) -> None:
"""A trusted-gateway peer with a foreign Origin is still CSRF-rejected.
The mismatched Origin also fails the first (loopback-or-trusted-client)
gate's own same-origin check, so this surfaces as 404, not 403 -- either
way the write must not go through."""
monkeypatch.setenv("HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS", "100.90.0.5/32")
client = TestClient(
_make_app(),
base_url="http://100.82.0.2:8787",
client=("100.90.0.5", 12345),
)
resp = client.post(path, json=body, headers={"origin": "http://attacker.example"})
assert resp.status_code in (403, 404), resp.text
def test_settings_post_loopback_null_origin_still_rejected() -> None:
"""Loopback callers keep the stricter loopback-only origin check: a
sandboxed-iframe/file:// "null" Origin must still 403, unaffected by the
trusted-dashboard-client carve-out."""
client = _loopback_client()
resp = client.post("/settings", json={"values": {}}, headers={"origin": "null"})
assert resp.status_code == 403, resp.text
fix(proxy): scope CORS to loopback + gate operator/content endpoints (#1226) ## Description Locks down the proxy's browser- and network-facing attack surface, which matters most under a `--host 0.0.0.0` bind (the Docker default). The wildcard CORS policy (`allow_origins=["*"]` + `allow_credentials=True`) let any web page the user had open read the proxy's content endpoints — `/v1/retrieve` returns raw, uncompressed tool outputs (source, secrets) — via a cross-origin fetch to `127.0.0.1` (CWE-346). Several operator endpoints additionally leaked sensitive data or allowed unauthenticated state mutation to any network-reachable client. This PR scopes CORS to loopback origins and extends the project's existing `require_loopback` trust boundary (already used for `/admin/*` and `/debug/*`) to the remaining exposed endpoints. Closes #863. Supersedes #864 and #758 — see "Additional Notes". ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **CORS**: replaced `allow_origins=["*"]` + `allow_credentials=True` with a port-agnostic loopback origin regex (`https?://(localhost|127\.0\.0\.1|\[::1\])(:\d+)?`), `allow_credentials=False`, and methods/headers narrowed to `GET/POST` + `Content-Type/Authorization`. `HEADROOM_CORS_ORIGINS` (comma-separated) pins an explicit allowlist for Docker/remote dashboards; `*` opts back into the old wildcard. - **`/transformations/feed`** and **`/cache/clear`** gated behind `require_loopback` → 404 for non-loopback callers. The feed returns full prompt/completion bodies when `log_full_messages` is on; `/cache/clear` is unauthenticated state mutation (cache-eviction DoS / cost amplification). - **`/health`**: the `config` block (upstream API URLs, savings profile) is now served only to loopback callers; network callers get the `/readyz`-shape body (status/checks). `/livez` and `/readyz` remain unauthenticated probes for orchestration. - **`/stats`**: `recent_requests` / `request_logs` (per-request ids, providers, models, errors) and `config` are served only to loopback callers; aggregate counters stay public for remote monitoring. - Added `_request_is_loopback()` helper mirroring `require_loopback`'s two-gate check (loopback peer IP + loopback `Host` header, the DNS-rebinding defence) but degrading the payload instead of returning 404, so monitors keep the non-sensitive fields. - Tests: new `tests/test_proxy_cors.py` and `tests/test_proxy_loopback_gating.py`; updated 4 existing tests that assert the now-loopback-only data to use loopback clients. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check headroom/proxy/server.py tests/test_proxy_cors.py \ tests/test_proxy_loopback_gating.py tests/test_proxy_healthchecks.py \ tests/test_proxy_stats_recent_requests.py tests/test_proxy/test_transformations_feed.py All checks passed! $ mypy headroom --ignore-missing-imports Success: no issues found in 380 source files $ pytest tests/test_proxy_cors.py tests/test_proxy_loopback_gating.py \ tests/test_proxy/test_transformations_feed.py tests/test_proxy_healthchecks.py \ tests/test_proxy_stats_recent_requests.py tests/test_proxy_dashboard_stats_cache.py \ tests/test_proxy_compression_executor.py tests/test_header_isolation.py -q ======================== 82 passed, 1 skipped in 17.75s ======================== ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0), Python 3.12 venv; FastAPI `TestClient` driving the real `create_app()` ASGI app - Exact command / steps: issued requests as a non-loopback caller (`client.host=testclient`) vs a loopback caller (`base_url=http://127.0.0.1`, `client=("127.0.0.1", 9999)`), plus CORS preflights with varying `Origin` headers - Observed result: CORS — `http://evil.com` → no `access-control-allow-origin`; `http://localhost:8787` and `http://localhost:9000` → echoed (loopback allowed on any port); `access-control-allow-credentials` → absent. `/cache/clear` and `/transformations/feed` → 404 (network) / 200 (loopback). `/health` `config` block present for loopback only. `/stats` `recent_requests` present for loopback only, while the aggregate `tokens` block stays present for network callers. - Not tested: a live `headroom proxy` process bound on `0.0.0.0` reached from a second host (simulated via ASGI peer/Host instead); end-to-end browser DNS-rebinding (covered by the `Host`-header gate and its unit test) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — proxy/middleware change; behavior is captured under "Real Behavior Proof". ## Additional Notes **Supersedes two stale PRs that target the same issue but have drifted from `main`:** - **#864** (`fix(proxy): scope CORS to localhost`, @gabiudrescu) — correct instinct and the source of the tighter `GET/POST` + `Content-Type/Authorization` scoping kept here, but it derived the allowlist from the `HEADROOM_PORT` env var (wrong when `--port` is passed as a CLI flag), carried ~40 lines of unrelated punctuation churn, and is ~125 commits behind `main`. The port-agnostic regex used here resolves the reviewer's port concern. - **#758** (`security: adversarial review`, @neogenix) — bundled these same application-layer fixes with a large CI/CD + Docker supply-chain pass. It is a ~160-commit-behind draft whose `server.py` no longer merges cleanly (`main` independently adopted the same `require_loopback` pattern). The application-layer fixes are rebased onto current `main` here; the CI/Docker/supply-chain hardening from #758 is still valuable and would be welcome as a separate, rebased PR. Thanks to @gabiudrescu and @neogenix for the original analysis (#863). **Deliberate scope / follow-ups (not in this PR):** - `/stats` aggregate counters and the basic `/health` body remain readable on a `0.0.0.0` bind by design, so remote monitoring keeps working. Full lock-down is a one-line `Depends(require_loopback)` each if preferred. - The `/v1/retrieve*` family stays network-reachable; it can't be loopback-gated without breaking legitimate remote/containerized agents and needs auth instead — tracked separately. - `ruff check .` is scoped to changed paths above because the dashboard HTML template trips ruff's `invalid-syntax` (a known repo false-positive); `mypy` is run over the full `headroom` package.
2026-06-21 00:50:55 -07:00
def test_dns_rebinding_host_header_rejected() -> None:
# Loopback peer IP but an attacker-controlled Host header (the DNS-rebinding
# shape) must still be rejected by the second gate.
client = TestClient(_make_app(), base_url="http://127.0.0.1", client=("127.0.0.1", 12345))
resp = client.get("/transformations/feed", headers={"host": "attacker.example"})
assert resp.status_code == 404, resp.text
feat(dashboard): persist lifetime proxy metrics (#2198) ## Description Persist bounded, aggregate-only Lifetime dashboard metrics across proxy restarts and expose them through a new `/stats-lifetime` endpoint. The change keeps session/runtime stats separate from durable lifetime stats, gates sensitive dashboard metadata for loopback or explicitly trusted dashboard clients, and updates the dashboard Lifetime view to consume the new endpoint. Closes #2137 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Added bounded persistent lifetime metrics state and wired proxy metric events into it. - Added `/stats-lifetime` with sensitive project/persistence details gated behind dashboard metadata access checks. - Extended loopback/dashboard metadata access policy for trusted dashboard client CIDRs without widening admin/debug endpoints. - Reorganized dashboard session/lifetime presentation around runtime counters versus durable aggregates. - Added focused tests for persistent aggregation, persistence, endpoint registration, loopback gating, trusted dashboard CIDRs, and recent request ordering. - Fixed current Ruff/mypy issues in the lifetime metrics normalization code. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text uv run --extra proxy --with pytest --with pytest-asyncio pytest tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_stats_recent_requests.py -q 53 passed, 1 warning uvx ruff==0.15.17 check headroom/proxy/forwarded_headers.py headroom/proxy/loopback_guard.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py headroom/proxy/server.py tests/test_forwarded_headers.py tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_project_savings.py tests/test_proxy_stats_recent_requests.py All checks passed! uv run --extra proxy --with mypy mypy headroom/proxy/persistent_metrics.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows review worktree, Python 3.13.3 via uv. - Exact command / steps: Ran the focused persistent metrics, persistence, loopback gating, and recent request tests; ran CI-matching Ruff on touched files; ran mypy on the new persistent metrics module. - Observed result: `/stats-lifetime` is registered, non-loopback callers receive only non-sensitive aggregate data, loopback/trusted dashboard clients receive the full lifetime payload, admin/debug endpoints remain loopback-only, and persistent metrics normalize malformed stored state without type/lint errors. - Not tested: Full repository pytest suite, full dashboard browser screenshot pass, or live long-running proxy traffic. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes No changelog entry is required for this dashboard/internal metrics iteration. The endpoint intentionally exposes only aggregate lifetime data to ordinary network callers and strips project/persistence details unless the caller passes the dashboard metadata access policy. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-16 02:18:13 +08:00
@pytest.mark.parametrize(
"host_header",
["100.82.0.2", "100.82.0.2:8787", "[fd7a:115c:a1e0::2]", "[fd7a:115c:a1e0::2]:8787"],
)
def test_ip_literal_host_header_accepts_ip_addresses(host_header: str) -> None:
assert is_ip_literal_host_header(host_header) is True
@pytest.mark.parametrize(
"host_header",
[None, "", "attacker.example", "localhost", "user@100.82.0.2", "100.82.0.2/path", "[fd7a::1"],
)
def test_ip_literal_host_header_rejects_non_addresses(host_header: str | None) -> None:
assert is_ip_literal_host_header(host_header) is False
fix(proxy): scope CORS to loopback + gate operator/content endpoints (#1226) ## Description Locks down the proxy's browser- and network-facing attack surface, which matters most under a `--host 0.0.0.0` bind (the Docker default). The wildcard CORS policy (`allow_origins=["*"]` + `allow_credentials=True`) let any web page the user had open read the proxy's content endpoints — `/v1/retrieve` returns raw, uncompressed tool outputs (source, secrets) — via a cross-origin fetch to `127.0.0.1` (CWE-346). Several operator endpoints additionally leaked sensitive data or allowed unauthenticated state mutation to any network-reachable client. This PR scopes CORS to loopback origins and extends the project's existing `require_loopback` trust boundary (already used for `/admin/*` and `/debug/*`) to the remaining exposed endpoints. Closes #863. Supersedes #864 and #758 — see "Additional Notes". ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **CORS**: replaced `allow_origins=["*"]` + `allow_credentials=True` with a port-agnostic loopback origin regex (`https?://(localhost|127\.0\.0\.1|\[::1\])(:\d+)?`), `allow_credentials=False`, and methods/headers narrowed to `GET/POST` + `Content-Type/Authorization`. `HEADROOM_CORS_ORIGINS` (comma-separated) pins an explicit allowlist for Docker/remote dashboards; `*` opts back into the old wildcard. - **`/transformations/feed`** and **`/cache/clear`** gated behind `require_loopback` → 404 for non-loopback callers. The feed returns full prompt/completion bodies when `log_full_messages` is on; `/cache/clear` is unauthenticated state mutation (cache-eviction DoS / cost amplification). - **`/health`**: the `config` block (upstream API URLs, savings profile) is now served only to loopback callers; network callers get the `/readyz`-shape body (status/checks). `/livez` and `/readyz` remain unauthenticated probes for orchestration. - **`/stats`**: `recent_requests` / `request_logs` (per-request ids, providers, models, errors) and `config` are served only to loopback callers; aggregate counters stay public for remote monitoring. - Added `_request_is_loopback()` helper mirroring `require_loopback`'s two-gate check (loopback peer IP + loopback `Host` header, the DNS-rebinding defence) but degrading the payload instead of returning 404, so monitors keep the non-sensitive fields. - Tests: new `tests/test_proxy_cors.py` and `tests/test_proxy_loopback_gating.py`; updated 4 existing tests that assert the now-loopback-only data to use loopback clients. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check headroom/proxy/server.py tests/test_proxy_cors.py \ tests/test_proxy_loopback_gating.py tests/test_proxy_healthchecks.py \ tests/test_proxy_stats_recent_requests.py tests/test_proxy/test_transformations_feed.py All checks passed! $ mypy headroom --ignore-missing-imports Success: no issues found in 380 source files $ pytest tests/test_proxy_cors.py tests/test_proxy_loopback_gating.py \ tests/test_proxy/test_transformations_feed.py tests/test_proxy_healthchecks.py \ tests/test_proxy_stats_recent_requests.py tests/test_proxy_dashboard_stats_cache.py \ tests/test_proxy_compression_executor.py tests/test_header_isolation.py -q ======================== 82 passed, 1 skipped in 17.75s ======================== ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0), Python 3.12 venv; FastAPI `TestClient` driving the real `create_app()` ASGI app - Exact command / steps: issued requests as a non-loopback caller (`client.host=testclient`) vs a loopback caller (`base_url=http://127.0.0.1`, `client=("127.0.0.1", 9999)`), plus CORS preflights with varying `Origin` headers - Observed result: CORS — `http://evil.com` → no `access-control-allow-origin`; `http://localhost:8787` and `http://localhost:9000` → echoed (loopback allowed on any port); `access-control-allow-credentials` → absent. `/cache/clear` and `/transformations/feed` → 404 (network) / 200 (loopback). `/health` `config` block present for loopback only. `/stats` `recent_requests` present for loopback only, while the aggregate `tokens` block stays present for network callers. - Not tested: a live `headroom proxy` process bound on `0.0.0.0` reached from a second host (simulated via ASGI peer/Host instead); end-to-end browser DNS-rebinding (covered by the `Host`-header gate and its unit test) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — proxy/middleware change; behavior is captured under "Real Behavior Proof". ## Additional Notes **Supersedes two stale PRs that target the same issue but have drifted from `main`:** - **#864** (`fix(proxy): scope CORS to localhost`, @gabiudrescu) — correct instinct and the source of the tighter `GET/POST` + `Content-Type/Authorization` scoping kept here, but it derived the allowlist from the `HEADROOM_PORT` env var (wrong when `--port` is passed as a CLI flag), carried ~40 lines of unrelated punctuation churn, and is ~125 commits behind `main`. The port-agnostic regex used here resolves the reviewer's port concern. - **#758** (`security: adversarial review`, @neogenix) — bundled these same application-layer fixes with a large CI/CD + Docker supply-chain pass. It is a ~160-commit-behind draft whose `server.py` no longer merges cleanly (`main` independently adopted the same `require_loopback` pattern). The application-layer fixes are rebased onto current `main` here; the CI/Docker/supply-chain hardening from #758 is still valuable and would be welcome as a separate, rebased PR. Thanks to @gabiudrescu and @neogenix for the original analysis (#863). **Deliberate scope / follow-ups (not in this PR):** - `/stats` aggregate counters and the basic `/health` body remain readable on a `0.0.0.0` bind by design, so remote monitoring keeps working. Full lock-down is a one-line `Depends(require_loopback)` each if preferred. - The `/v1/retrieve*` family stays network-reachable; it can't be loopback-gated without breaking legitimate remote/containerized agents and needs auth instead — tracked separately. - `ruff check .` is scoped to changed paths above because the dashboard HTML template trips ruff's `invalid-syntax` (a known repo false-positive); `mypy` is run over the full `headroom` package.
2026-06-21 00:50:55 -07:00
def _client(*, loopback: bool) -> TestClient:
app = _make_app()
if loopback:
return TestClient(app, base_url="http://127.0.0.1", client=("127.0.0.1", 12345))
# Default TestClient presents client.host="testclient" — not loopback.
return TestClient(app)
def test_health_config_block_is_loopback_only(monkeypatch: pytest.MonkeyPatch) -> None:
"""/health stays reachable for monitors but hides the `config` block (which
echoes upstream API URLs + backend settings) from non-loopback callers."""
monkeypatch.setenv("HEADROOM_SKIP_UPSTREAM_CHECK", "1")
network = _client(loopback=False).get("/health")
assert network.status_code == 200
assert "config" not in network.json()
# Basic health is still visible to monitors.
assert network.json()["status"] in {"healthy", "unhealthy"}
local = _client(loopback=True).get("/health")
assert local.status_code == 200
assert "config" in local.json()
def test_stats_per_request_metadata_is_loopback_only() -> None:
"""/stats keeps aggregate counters public but restricts per-request metadata
(recent_requests / request_logs) and `config` to loopback callers."""
network = _client(loopback=False).get("/stats")
assert network.status_code == 200
payload = network.json()
assert "tokens" in payload # aggregate counters still served
assert "recent_requests" not in payload
assert "request_logs" not in payload
assert "config" not in payload
local = _client(loopback=True).get("/stats").json()
assert "recent_requests" in local
assert "config" in local
fix(dashboard): serve per-request metadata to trusted-gateway peers (#1766) ## Description The dashboard's per-request metadata — the `recent_requests` / `request_logs` tail and the `config` block (which echoes upstream API URLs + backend settings) — is gated to loopback callers via `_request_is_loopback`. It requires **both** a loopback peer IP (`request.client.host == 127.0.0.1`) and a loopback `Host` header. When Headroom runs in a **bridge-network container** (Docker/podman, or Apple Containerization / `mocker`), a browser on the host reaches the proxy through the container gateway, so `request.client.host` is the **gateway IP** (e.g. `172.18.0.1`, or `192.168.64.1` on macOS vmnet), not `127.0.0.1`. `include_sensitive` is therefore `False`, and the "Recent Requests" table renders empty even though the operator is browsing locally at `http://127.0.0.1:8787/dashboard`. `curl` from **inside** the container (real `127.0.0.1` peer) confirmed the data is present and populated — only the host-browser path was being stripped. The fix treats a peer inside an operator-configured trusted-gateway CIDR (`HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` — the same allow-list already used by `forwarded_headers.py` to sanitize `X-Forwarded-*`) as loopback-equivalent, while **retaining the loopback `Host`-header gate as the DNS-rebinding defence**. It is opt-in and empty by default, so there is **no behavior change** unless the operator explicitly allow-lists their container gateway. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/proxy/server.py` — `_request_is_loopback` now: (1) always enforces the loopback `Host`-header gate first; (2) returns `True` for a genuine loopback peer; (3) additionally returns `True` for a peer inside `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` via the existing `peer_is_trusted_gateway` / `load_trusted_gateway_cidrs` helpers. - `tests/test_proxy_loopback_gating.py` — added `test_stats_metadata_served_to_trusted_gateway_peer`: gateway peer stripped without the allow-list, served with it, and DNS-rebinding (non-loopback `Host`) still rejected even for a trusted gateway peer. - `CHANGELOG.md` — Unreleased → Fixed entry. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_proxy_loopback_gating.py -q 14 passed, 1 warning in 3.56s $ ruff check headroom/proxy/server.py tests/test_proxy_loopback_gating.py All checks passed! ``` ## Real Behavior Proof - Environment: Headroom 0.29.0 in a `mocker compose` (Apple Containerization) bridge container on macOS; host browser at `http://127.0.0.1:8787/dashboard`. - Exact command / steps: before the fix, `mocker compose exec headroom-proxy sh -c 'curl -s http://127.0.0.1:8787/stats'` (peer = real `127.0.0.1`) returned a populated `recent_requests` array, while the host browser saw an empty table. After adding `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` covering the container gateway and recreating, the host browser's dashboard shows the Recent Requests table again. - Observed result: dashboard per-request table restored for the host browser; aggregate-only view unchanged for untrusted network callers. - Not tested: IPv6 gateway CIDRs (the underlying `peer_is_trusted_gateway` supports them; not exercised in this environment). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes Pure opt-in: `HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS` is empty by default, so `_request_is_loopback` behavior is byte-identical to today unless an operator allow-lists a gateway CIDR. Reuses the existing trusted-gateway machinery rather than introducing a new config surface. Docs/compose examples intentionally omitted — deployment-specific. Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-13 05:29:36 +08:00
def test_stats_metadata_served_to_trusted_gateway_peer(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Containerized dashboards: a browser on the host reaches a bridge-network
container via the gateway IP, so the peer isn't 127.0.0.1 and per-request
metadata gets stripped. When the operator allow-lists the gateway CIDR via
HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS, the peer is treated as
loopback-equivalent and the metadata is served again."""
gateway_ip = "172.18.0.1" # typical docker/mocker bridge gateway
app = _make_app()
def _gateway_client() -> TestClient:
# Loopback Host header (the operator browses http://127.0.0.1:8787) but
# the peer IP is the container gateway, not loopback.
return TestClient(app, base_url="http://127.0.0.1", client=(gateway_ip, 54321))
# Without the allow-list, the gateway peer is untrusted → metadata stripped.
monkeypatch.delenv("HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS", raising=False)
stripped = _gateway_client().get("/stats").json()
assert "recent_requests" not in stripped
assert "config" not in stripped
# Allow-list the gateway CIDR → peer trusted → metadata served.
monkeypatch.setenv("HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS", "172.18.0.0/16")
served = _gateway_client().get("/stats").json()
assert "recent_requests" in served
assert "config" in served
# DNS-rebinding defence still applies even for a trusted gateway peer: a
# non-loopback Host header must be rejected.
rebind = TestClient(app, base_url="http://attacker.example", client=(gateway_ip, 54321))
payload = rebind.get("/stats").json()
assert "recent_requests" not in payload
feat(dashboard): persist lifetime proxy metrics (#2198) ## Description Persist bounded, aggregate-only Lifetime dashboard metrics across proxy restarts and expose them through a new `/stats-lifetime` endpoint. The change keeps session/runtime stats separate from durable lifetime stats, gates sensitive dashboard metadata for loopback or explicitly trusted dashboard clients, and updates the dashboard Lifetime view to consume the new endpoint. Closes #2137 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Added bounded persistent lifetime metrics state and wired proxy metric events into it. - Added `/stats-lifetime` with sensitive project/persistence details gated behind dashboard metadata access checks. - Extended loopback/dashboard metadata access policy for trusted dashboard client CIDRs without widening admin/debug endpoints. - Reorganized dashboard session/lifetime presentation around runtime counters versus durable aggregates. - Added focused tests for persistent aggregation, persistence, endpoint registration, loopback gating, trusted dashboard CIDRs, and recent request ordering. - Fixed current Ruff/mypy issues in the lifetime metrics normalization code. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text uv run --extra proxy --with pytest --with pytest-asyncio pytest tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_stats_recent_requests.py -q 53 passed, 1 warning uvx ruff==0.15.17 check headroom/proxy/forwarded_headers.py headroom/proxy/loopback_guard.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py headroom/proxy/server.py tests/test_forwarded_headers.py tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_project_savings.py tests/test_proxy_stats_recent_requests.py All checks passed! uv run --extra proxy --with mypy mypy headroom/proxy/persistent_metrics.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows review worktree, Python 3.13.3 via uv. - Exact command / steps: Ran the focused persistent metrics, persistence, loopback gating, and recent request tests; ran CI-matching Ruff on touched files; ran mypy on the new persistent metrics module. - Observed result: `/stats-lifetime` is registered, non-loopback callers receive only non-sensitive aggregate data, loopback/trusted dashboard clients receive the full lifetime payload, admin/debug endpoints remain loopback-only, and persistent metrics normalize malformed stored state without type/lint errors. - Not tested: Full repository pytest suite, full dashboard browser screenshot pass, or live long-running proxy traffic. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes No changelog entry is required for this dashboard/internal metrics iteration. The endpoint intentionally exposes only aggregate lifetime data to ordinary network callers and strips project/persistence details unless the caller passes the dashboard metadata access policy. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-16 02:18:13 +08:00
@pytest.mark.parametrize("cached", [False, True])
def test_dashboard_client_cidr_grants_stats_metadata_for_ip_literal_host(
monkeypatch: pytest.MonkeyPatch,
cached: bool,
) -> None:
monkeypatch.setenv("HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS", "100.90.0.5/32")
app = _make_app()
client = TestClient(
app,
base_url="http://100.82.0.2:8787",
client=("100.90.0.5", 12345),
)
payload = client.get("/stats", params={"cached": int(cached)}).json()
assert "recent_requests" in payload
assert "request_logs" in payload
assert "config" in payload
@pytest.mark.parametrize(
"headers",
[
{"origin": "http://100.82.0.2:8787"},
{"referer": "http://100.82.0.2:8787/dashboard"},
],
)
@pytest.mark.parametrize("cached", [False, True])
def test_dashboard_client_cidr_grants_stats_metadata_to_same_origin_browser(
monkeypatch: pytest.MonkeyPatch, headers: dict[str, str], cached: bool
) -> None:
monkeypatch.setenv("HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS", "100.90.0.5/32")
client = TestClient(
_make_app(),
base_url="http://100.82.0.2:8787",
client=("100.90.0.5", 12345),
)
fix(proxy): repair main lint (ruff-format drift + mypy host_header) (#2268) ## Description `main`'s `lint` CI job is currently **red** (latest main `eac49656` → `lint: failure`), which blocks every open PR. Two causes, both from recent merges that were green in isolation but combined into a red `main`: - **ruff-format drift** on 7 files — committed with formatter output that ruff `0.15.17` (the CI-pinned version) rewrites. - **mypy error** in `server.py`: `_request_has_same_origin_or_no_provenance(request, host_header)` — `host_header` is `request.headers.get("host")` (`str | None`) but the function requires `str`. These passed per-PR because each PR's checks ran against an older base; the serialized `main` state is what went red — a logical-merge / tool-version gap that per-PR CI doesn't catch without a strict merge queue. ## Type of Change - [x] Bug fix (CI/lint repair) ## Changes Made - `ruff format` (0.15.17) the 7 drifted files — formatting only, no logic changes: `cli/proxy.py`, `proxy/forwarded_headers.py`, `proxy/savings_tracker.py`, `proxy/server.py`, `tests/conftest.py`, `tests/test_persistent_metrics_persistence.py`, `tests/test_proxy_loopback_gating.py`. - Add `assert host_header is not None` after the `is_ip_literal_host_header()` guard (which already rejects a missing Host), narrowing the type for the same-origin check. ## Testing - [x] `ruff check .` — clean - [x] `ruff format --check .` — clean (tracked) - [x] `mypy headroom --ignore-missing-imports` — clean ### Test Output ```text $ mypy headroom --ignore-missing-imports → Success: no issues found in 504 source files $ ruff check . → All checks passed (tracked) $ ruff format --check . → clean (tracked) ``` ## Real Behavior Proof - Environment: branch off current `main` (`eac49656`), ruff 0.15.17 + mypy 1.20.2 (CI-pinned). - Confirmed `lint: failure` on main's latest CI run; after this change all three lint steps pass locally. - Not tested: full pytest suite — formatting + a type-narrowing `assert` only, no behavior change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines (this *is* the style fix) - [x] I have performed a self-review of my code - [x] My changes generate no new warnings - [ ] Tests added (N/A — no behavior change) - [x] New and existing unit tests pass locally - [ ] CHANGELOG (N/A) ## Additional Notes The 7 files were touched by recent merges (#2198, #2247) whose local ruff differed from the pinned `0.15.17`. Merging this unblocks the `lint` gate for all open PRs (including #2207). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-07-15 23:18:50 -07:00
payload = client.get("/stats", params={"cached": int(cached)}, headers=headers).json()
feat(dashboard): persist lifetime proxy metrics (#2198) ## Description Persist bounded, aggregate-only Lifetime dashboard metrics across proxy restarts and expose them through a new `/stats-lifetime` endpoint. The change keeps session/runtime stats separate from durable lifetime stats, gates sensitive dashboard metadata for loopback or explicitly trusted dashboard clients, and updates the dashboard Lifetime view to consume the new endpoint. Closes #2137 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Added bounded persistent lifetime metrics state and wired proxy metric events into it. - Added `/stats-lifetime` with sensitive project/persistence details gated behind dashboard metadata access checks. - Extended loopback/dashboard metadata access policy for trusted dashboard client CIDRs without widening admin/debug endpoints. - Reorganized dashboard session/lifetime presentation around runtime counters versus durable aggregates. - Added focused tests for persistent aggregation, persistence, endpoint registration, loopback gating, trusted dashboard CIDRs, and recent request ordering. - Fixed current Ruff/mypy issues in the lifetime metrics normalization code. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text uv run --extra proxy --with pytest --with pytest-asyncio pytest tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_stats_recent_requests.py -q 53 passed, 1 warning uvx ruff==0.15.17 check headroom/proxy/forwarded_headers.py headroom/proxy/loopback_guard.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py headroom/proxy/server.py tests/test_forwarded_headers.py tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_project_savings.py tests/test_proxy_stats_recent_requests.py All checks passed! uv run --extra proxy --with mypy mypy headroom/proxy/persistent_metrics.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows review worktree, Python 3.13.3 via uv. - Exact command / steps: Ran the focused persistent metrics, persistence, loopback gating, and recent request tests; ran CI-matching Ruff on touched files; ran mypy on the new persistent metrics module. - Observed result: `/stats-lifetime` is registered, non-loopback callers receive only non-sensitive aggregate data, loopback/trusted dashboard clients receive the full lifetime payload, admin/debug endpoints remain loopback-only, and persistent metrics normalize malformed stored state without type/lint errors. - Not tested: Full repository pytest suite, full dashboard browser screenshot pass, or live long-running proxy traffic. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes No changelog entry is required for this dashboard/internal metrics iteration. The endpoint intentionally exposes only aggregate lifetime data to ordinary network callers and strips project/persistence details unless the caller passes the dashboard metadata access policy. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-16 02:18:13 +08:00
assert "recent_requests" in payload
assert "request_logs" in payload
assert "config" in payload
@pytest.mark.parametrize(
"headers",
[
{"origin": "http://attacker.example"},
{"referer": "http://attacker.example/dashboard"},
],
)
@pytest.mark.parametrize("cached", [False, True])
def test_dashboard_client_cidr_hides_stats_metadata_from_cross_origin_browser(
monkeypatch: pytest.MonkeyPatch, headers: dict[str, str], cached: bool
) -> None:
monkeypatch.setenv("HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS", "100.90.0.5/32")
client = TestClient(
_make_app(),
base_url="http://100.82.0.2:8787",
client=("100.90.0.5", 12345),
)
fix(proxy): repair main lint (ruff-format drift + mypy host_header) (#2268) ## Description `main`'s `lint` CI job is currently **red** (latest main `eac49656` → `lint: failure`), which blocks every open PR. Two causes, both from recent merges that were green in isolation but combined into a red `main`: - **ruff-format drift** on 7 files — committed with formatter output that ruff `0.15.17` (the CI-pinned version) rewrites. - **mypy error** in `server.py`: `_request_has_same_origin_or_no_provenance(request, host_header)` — `host_header` is `request.headers.get("host")` (`str | None`) but the function requires `str`. These passed per-PR because each PR's checks ran against an older base; the serialized `main` state is what went red — a logical-merge / tool-version gap that per-PR CI doesn't catch without a strict merge queue. ## Type of Change - [x] Bug fix (CI/lint repair) ## Changes Made - `ruff format` (0.15.17) the 7 drifted files — formatting only, no logic changes: `cli/proxy.py`, `proxy/forwarded_headers.py`, `proxy/savings_tracker.py`, `proxy/server.py`, `tests/conftest.py`, `tests/test_persistent_metrics_persistence.py`, `tests/test_proxy_loopback_gating.py`. - Add `assert host_header is not None` after the `is_ip_literal_host_header()` guard (which already rejects a missing Host), narrowing the type for the same-origin check. ## Testing - [x] `ruff check .` — clean - [x] `ruff format --check .` — clean (tracked) - [x] `mypy headroom --ignore-missing-imports` — clean ### Test Output ```text $ mypy headroom --ignore-missing-imports → Success: no issues found in 504 source files $ ruff check . → All checks passed (tracked) $ ruff format --check . → clean (tracked) ``` ## Real Behavior Proof - Environment: branch off current `main` (`eac49656`), ruff 0.15.17 + mypy 1.20.2 (CI-pinned). - Confirmed `lint: failure` on main's latest CI run; after this change all three lint steps pass locally. - Not tested: full pytest suite — formatting + a type-narrowing `assert` only, no behavior change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines (this *is* the style fix) - [x] I have performed a self-review of my code - [x] My changes generate no new warnings - [ ] Tests added (N/A — no behavior change) - [x] New and existing unit tests pass locally - [ ] CHANGELOG (N/A) ## Additional Notes The 7 files were touched by recent merges (#2198, #2247) whose local ruff differed from the pinned `0.15.17`. Merging this unblocks the `lint` gate for all open PRs (including #2207). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-07-15 23:18:50 -07:00
response = client.get("/stats", params={"cached": int(cached)}, headers=headers)
feat(dashboard): persist lifetime proxy metrics (#2198) ## Description Persist bounded, aggregate-only Lifetime dashboard metrics across proxy restarts and expose them through a new `/stats-lifetime` endpoint. The change keeps session/runtime stats separate from durable lifetime stats, gates sensitive dashboard metadata for loopback or explicitly trusted dashboard clients, and updates the dashboard Lifetime view to consume the new endpoint. Closes #2137 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Added bounded persistent lifetime metrics state and wired proxy metric events into it. - Added `/stats-lifetime` with sensitive project/persistence details gated behind dashboard metadata access checks. - Extended loopback/dashboard metadata access policy for trusted dashboard client CIDRs without widening admin/debug endpoints. - Reorganized dashboard session/lifetime presentation around runtime counters versus durable aggregates. - Added focused tests for persistent aggregation, persistence, endpoint registration, loopback gating, trusted dashboard CIDRs, and recent request ordering. - Fixed current Ruff/mypy issues in the lifetime metrics normalization code. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text uv run --extra proxy --with pytest --with pytest-asyncio pytest tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_stats_recent_requests.py -q 53 passed, 1 warning uvx ruff==0.15.17 check headroom/proxy/forwarded_headers.py headroom/proxy/loopback_guard.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py headroom/proxy/server.py tests/test_forwarded_headers.py tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_project_savings.py tests/test_proxy_stats_recent_requests.py All checks passed! uv run --extra proxy --with mypy mypy headroom/proxy/persistent_metrics.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows review worktree, Python 3.13.3 via uv. - Exact command / steps: Ran the focused persistent metrics, persistence, loopback gating, and recent request tests; ran CI-matching Ruff on touched files; ran mypy on the new persistent metrics module. - Observed result: `/stats-lifetime` is registered, non-loopback callers receive only non-sensitive aggregate data, loopback/trusted dashboard clients receive the full lifetime payload, admin/debug endpoints remain loopback-only, and persistent metrics normalize malformed stored state without type/lint errors. - Not tested: Full repository pytest suite, full dashboard browser screenshot pass, or live long-running proxy traffic. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes No changelog entry is required for this dashboard/internal metrics iteration. The endpoint intentionally exposes only aggregate lifetime data to ordinary network callers and strips project/persistence details unless the caller passes the dashboard metadata access policy. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-16 02:18:13 +08:00
payload = response.json()
assert response.status_code == 200
assert "tokens" in payload
assert "recent_requests" not in payload
assert "request_logs" not in payload
assert "config" not in payload
def test_dashboard_client_cidr_only_uses_forwarded_proto_from_trusted_gateway(
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.setenv("HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS", "100.90.0.5/32")
monkeypatch.setenv("HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS", "172.18.0.0/16")
client = TestClient(
_make_app(),
base_url="http://100.82.0.2:8787",
client=("172.18.0.1", 12345),
)
payload = client.get(
"/stats",
headers={
"origin": "https://100.82.0.2:8787",
"x-forwarded-for": "100.90.0.5",
"x-forwarded-proto": "https",
},
).json()
assert "recent_requests" in payload
assert "request_logs" in payload
assert "config" in payload
fix(proxy): repair main lint (ruff-format drift + mypy host_header) (#2268) ## Description `main`'s `lint` CI job is currently **red** (latest main `eac49656` → `lint: failure`), which blocks every open PR. Two causes, both from recent merges that were green in isolation but combined into a red `main`: - **ruff-format drift** on 7 files — committed with formatter output that ruff `0.15.17` (the CI-pinned version) rewrites. - **mypy error** in `server.py`: `_request_has_same_origin_or_no_provenance(request, host_header)` — `host_header` is `request.headers.get("host")` (`str | None`) but the function requires `str`. These passed per-PR because each PR's checks ran against an older base; the serialized `main` state is what went red — a logical-merge / tool-version gap that per-PR CI doesn't catch without a strict merge queue. ## Type of Change - [x] Bug fix (CI/lint repair) ## Changes Made - `ruff format` (0.15.17) the 7 drifted files — formatting only, no logic changes: `cli/proxy.py`, `proxy/forwarded_headers.py`, `proxy/savings_tracker.py`, `proxy/server.py`, `tests/conftest.py`, `tests/test_persistent_metrics_persistence.py`, `tests/test_proxy_loopback_gating.py`. - Add `assert host_header is not None` after the `is_ip_literal_host_header()` guard (which already rejects a missing Host), narrowing the type for the same-origin check. ## Testing - [x] `ruff check .` — clean - [x] `ruff format --check .` — clean (tracked) - [x] `mypy headroom --ignore-missing-imports` — clean ### Test Output ```text $ mypy headroom --ignore-missing-imports → Success: no issues found in 504 source files $ ruff check . → All checks passed (tracked) $ ruff format --check . → clean (tracked) ``` ## Real Behavior Proof - Environment: branch off current `main` (`eac49656`), ruff 0.15.17 + mypy 1.20.2 (CI-pinned). - Confirmed `lint: failure` on main's latest CI run; after this change all three lint steps pass locally. - Not tested: full pytest suite — formatting + a type-narrowing `assert` only, no behavior change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines (this *is* the style fix) - [x] I have performed a self-review of my code - [x] My changes generate no new warnings - [ ] Tests added (N/A — no behavior change) - [x] New and existing unit tests pass locally - [ ] CHANGELOG (N/A) ## Additional Notes The 7 files were touched by recent merges (#2198, #2247) whose local ruff differed from the pinned `0.15.17`. Merging this unblocks the `lint` gate for all open PRs (including #2207). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-07-15 23:18:50 -07:00
spoofed = (
TestClient(
_make_app(),
base_url="http://100.82.0.2:8787",
client=("100.90.0.5", 12345),
)
.get(
"/stats",
headers={
"origin": "https://100.82.0.2:8787",
"x-forwarded-proto": "https",
},
)
.json()
)
feat(dashboard): persist lifetime proxy metrics (#2198) ## Description Persist bounded, aggregate-only Lifetime dashboard metrics across proxy restarts and expose them through a new `/stats-lifetime` endpoint. The change keeps session/runtime stats separate from durable lifetime stats, gates sensitive dashboard metadata for loopback or explicitly trusted dashboard clients, and updates the dashboard Lifetime view to consume the new endpoint. Closes #2137 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Added bounded persistent lifetime metrics state and wired proxy metric events into it. - Added `/stats-lifetime` with sensitive project/persistence details gated behind dashboard metadata access checks. - Extended loopback/dashboard metadata access policy for trusted dashboard client CIDRs without widening admin/debug endpoints. - Reorganized dashboard session/lifetime presentation around runtime counters versus durable aggregates. - Added focused tests for persistent aggregation, persistence, endpoint registration, loopback gating, trusted dashboard CIDRs, and recent request ordering. - Fixed current Ruff/mypy issues in the lifetime metrics normalization code. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text uv run --extra proxy --with pytest --with pytest-asyncio pytest tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_stats_recent_requests.py -q 53 passed, 1 warning uvx ruff==0.15.17 check headroom/proxy/forwarded_headers.py headroom/proxy/loopback_guard.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py headroom/proxy/server.py tests/test_forwarded_headers.py tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_project_savings.py tests/test_proxy_stats_recent_requests.py All checks passed! uv run --extra proxy --with mypy mypy headroom/proxy/persistent_metrics.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows review worktree, Python 3.13.3 via uv. - Exact command / steps: Ran the focused persistent metrics, persistence, loopback gating, and recent request tests; ran CI-matching Ruff on touched files; ran mypy on the new persistent metrics module. - Observed result: `/stats-lifetime` is registered, non-loopback callers receive only non-sensitive aggregate data, loopback/trusted dashboard clients receive the full lifetime payload, admin/debug endpoints remain loopback-only, and persistent metrics normalize malformed stored state without type/lint errors. - Not tested: Full repository pytest suite, full dashboard browser screenshot pass, or live long-running proxy traffic. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes No changelog entry is required for this dashboard/internal metrics iteration. The endpoint intentionally exposes only aggregate lifetime data to ordinary network callers and strips project/persistence details unless the caller passes the dashboard metadata access policy. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-16 02:18:13 +08:00
assert "recent_requests" not in spoofed
assert "request_logs" not in spoofed
assert "config" not in spoofed
def test_dashboard_client_cidr_rejects_unlisted_clients_and_hostname_hosts(
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.setenv("HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS", "100.90.0.5/32")
app = _make_app()
fix(proxy): repair main lint (ruff-format drift + mypy host_header) (#2268) ## Description `main`'s `lint` CI job is currently **red** (latest main `eac49656` → `lint: failure`), which blocks every open PR. Two causes, both from recent merges that were green in isolation but combined into a red `main`: - **ruff-format drift** on 7 files — committed with formatter output that ruff `0.15.17` (the CI-pinned version) rewrites. - **mypy error** in `server.py`: `_request_has_same_origin_or_no_provenance(request, host_header)` — `host_header` is `request.headers.get("host")` (`str | None`) but the function requires `str`. These passed per-PR because each PR's checks ran against an older base; the serialized `main` state is what went red — a logical-merge / tool-version gap that per-PR CI doesn't catch without a strict merge queue. ## Type of Change - [x] Bug fix (CI/lint repair) ## Changes Made - `ruff format` (0.15.17) the 7 drifted files — formatting only, no logic changes: `cli/proxy.py`, `proxy/forwarded_headers.py`, `proxy/savings_tracker.py`, `proxy/server.py`, `tests/conftest.py`, `tests/test_persistent_metrics_persistence.py`, `tests/test_proxy_loopback_gating.py`. - Add `assert host_header is not None` after the `is_ip_literal_host_header()` guard (which already rejects a missing Host), narrowing the type for the same-origin check. ## Testing - [x] `ruff check .` — clean - [x] `ruff format --check .` — clean (tracked) - [x] `mypy headroom --ignore-missing-imports` — clean ### Test Output ```text $ mypy headroom --ignore-missing-imports → Success: no issues found in 504 source files $ ruff check . → All checks passed (tracked) $ ruff format --check . → clean (tracked) ``` ## Real Behavior Proof - Environment: branch off current `main` (`eac49656`), ruff 0.15.17 + mypy 1.20.2 (CI-pinned). - Confirmed `lint: failure` on main's latest CI run; after this change all three lint steps pass locally. - Not tested: full pytest suite — formatting + a type-narrowing `assert` only, no behavior change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines (this *is* the style fix) - [x] I have performed a self-review of my code - [x] My changes generate no new warnings - [ ] Tests added (N/A — no behavior change) - [x] New and existing unit tests pass locally - [ ] CHANGELOG (N/A) ## Additional Notes The 7 files were touched by recent merges (#2198, #2247) whose local ruff differed from the pinned `0.15.17`. Merging this unblocks the `lint` gate for all open PRs (including #2207). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-07-15 23:18:50 -07:00
unlisted = (
TestClient(
app,
base_url="http://100.82.0.2:8787",
client=("100.90.0.6", 12345),
)
.get("/stats")
.json()
)
hostname = (
TestClient(
app,
base_url="http://100.82.0.2:8787",
client=("100.90.0.5", 12345),
)
.get("/stats", headers={"host": "attacker.example"})
.json()
)
feat(dashboard): persist lifetime proxy metrics (#2198) ## Description Persist bounded, aggregate-only Lifetime dashboard metrics across proxy restarts and expose them through a new `/stats-lifetime` endpoint. The change keeps session/runtime stats separate from durable lifetime stats, gates sensitive dashboard metadata for loopback or explicitly trusted dashboard clients, and updates the dashboard Lifetime view to consume the new endpoint. Closes #2137 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Added bounded persistent lifetime metrics state and wired proxy metric events into it. - Added `/stats-lifetime` with sensitive project/persistence details gated behind dashboard metadata access checks. - Extended loopback/dashboard metadata access policy for trusted dashboard client CIDRs without widening admin/debug endpoints. - Reorganized dashboard session/lifetime presentation around runtime counters versus durable aggregates. - Added focused tests for persistent aggregation, persistence, endpoint registration, loopback gating, trusted dashboard CIDRs, and recent request ordering. - Fixed current Ruff/mypy issues in the lifetime metrics normalization code. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text uv run --extra proxy --with pytest --with pytest-asyncio pytest tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_stats_recent_requests.py -q 53 passed, 1 warning uvx ruff==0.15.17 check headroom/proxy/forwarded_headers.py headroom/proxy/loopback_guard.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py headroom/proxy/server.py tests/test_forwarded_headers.py tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_project_savings.py tests/test_proxy_stats_recent_requests.py All checks passed! uv run --extra proxy --with mypy mypy headroom/proxy/persistent_metrics.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows review worktree, Python 3.13.3 via uv. - Exact command / steps: Ran the focused persistent metrics, persistence, loopback gating, and recent request tests; ran CI-matching Ruff on touched files; ran mypy on the new persistent metrics module. - Observed result: `/stats-lifetime` is registered, non-loopback callers receive only non-sensitive aggregate data, loopback/trusted dashboard clients receive the full lifetime payload, admin/debug endpoints remain loopback-only, and persistent metrics normalize malformed stored state without type/lint errors. - Not tested: Full repository pytest suite, full dashboard browser screenshot pass, or live long-running proxy traffic. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes No changelog entry is required for this dashboard/internal metrics iteration. The endpoint intentionally exposes only aggregate lifetime data to ordinary network callers and strips project/persistence details unless the caller passes the dashboard metadata access policy. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-16 02:18:13 +08:00
for payload in (unlisted, hostname):
assert "recent_requests" not in payload
assert "request_logs" not in payload
assert "config" not in payload
def test_dashboard_client_cidr_only_accepts_forwarded_client_from_trusted_gateway(
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.setenv("HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS", "100.90.0.5/32")
monkeypatch.setenv("HEADROOM_PROXY_TRUSTED_GATEWAY_CIDRS", "172.18.0.0/16")
app = _make_app()
fix(proxy): repair main lint (ruff-format drift + mypy host_header) (#2268) ## Description `main`'s `lint` CI job is currently **red** (latest main `eac49656` → `lint: failure`), which blocks every open PR. Two causes, both from recent merges that were green in isolation but combined into a red `main`: - **ruff-format drift** on 7 files — committed with formatter output that ruff `0.15.17` (the CI-pinned version) rewrites. - **mypy error** in `server.py`: `_request_has_same_origin_or_no_provenance(request, host_header)` — `host_header` is `request.headers.get("host")` (`str | None`) but the function requires `str`. These passed per-PR because each PR's checks ran against an older base; the serialized `main` state is what went red — a logical-merge / tool-version gap that per-PR CI doesn't catch without a strict merge queue. ## Type of Change - [x] Bug fix (CI/lint repair) ## Changes Made - `ruff format` (0.15.17) the 7 drifted files — formatting only, no logic changes: `cli/proxy.py`, `proxy/forwarded_headers.py`, `proxy/savings_tracker.py`, `proxy/server.py`, `tests/conftest.py`, `tests/test_persistent_metrics_persistence.py`, `tests/test_proxy_loopback_gating.py`. - Add `assert host_header is not None` after the `is_ip_literal_host_header()` guard (which already rejects a missing Host), narrowing the type for the same-origin check. ## Testing - [x] `ruff check .` — clean - [x] `ruff format --check .` — clean (tracked) - [x] `mypy headroom --ignore-missing-imports` — clean ### Test Output ```text $ mypy headroom --ignore-missing-imports → Success: no issues found in 504 source files $ ruff check . → All checks passed (tracked) $ ruff format --check . → clean (tracked) ``` ## Real Behavior Proof - Environment: branch off current `main` (`eac49656`), ruff 0.15.17 + mypy 1.20.2 (CI-pinned). - Confirmed `lint: failure` on main's latest CI run; after this change all three lint steps pass locally. - Not tested: full pytest suite — formatting + a type-narrowing `assert` only, no behavior change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines (this *is* the style fix) - [x] I have performed a self-review of my code - [x] My changes generate no new warnings - [ ] Tests added (N/A — no behavior change) - [x] New and existing unit tests pass locally - [ ] CHANGELOG (N/A) ## Additional Notes The 7 files were touched by recent merges (#2198, #2247) whose local ruff differed from the pinned `0.15.17`. Merging this unblocks the `lint` gate for all open PRs (including #2207). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-07-15 23:18:50 -07:00
trusted = (
TestClient(
app,
base_url="http://100.82.0.2:8787",
client=("172.18.0.1", 12345),
)
.get("/stats", headers={"x-forwarded-for": "100.90.0.5"})
.json()
)
forged = (
TestClient(
app,
base_url="http://100.82.0.2:8787",
client=("198.51.100.10", 12345),
)
.get("/stats", headers={"x-forwarded-for": "100.90.0.5"})
.json()
)
feat(dashboard): persist lifetime proxy metrics (#2198) ## Description Persist bounded, aggregate-only Lifetime dashboard metrics across proxy restarts and expose them through a new `/stats-lifetime` endpoint. The change keeps session/runtime stats separate from durable lifetime stats, gates sensitive dashboard metadata for loopback or explicitly trusted dashboard clients, and updates the dashboard Lifetime view to consume the new endpoint. Closes #2137 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Added bounded persistent lifetime metrics state and wired proxy metric events into it. - Added `/stats-lifetime` with sensitive project/persistence details gated behind dashboard metadata access checks. - Extended loopback/dashboard metadata access policy for trusted dashboard client CIDRs without widening admin/debug endpoints. - Reorganized dashboard session/lifetime presentation around runtime counters versus durable aggregates. - Added focused tests for persistent aggregation, persistence, endpoint registration, loopback gating, trusted dashboard CIDRs, and recent request ordering. - Fixed current Ruff/mypy issues in the lifetime metrics normalization code. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text uv run --extra proxy --with pytest --with pytest-asyncio pytest tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_stats_recent_requests.py -q 53 passed, 1 warning uvx ruff==0.15.17 check headroom/proxy/forwarded_headers.py headroom/proxy/loopback_guard.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py headroom/proxy/server.py tests/test_forwarded_headers.py tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_project_savings.py tests/test_proxy_stats_recent_requests.py All checks passed! uv run --extra proxy --with mypy mypy headroom/proxy/persistent_metrics.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows review worktree, Python 3.13.3 via uv. - Exact command / steps: Ran the focused persistent metrics, persistence, loopback gating, and recent request tests; ran CI-matching Ruff on touched files; ran mypy on the new persistent metrics module. - Observed result: `/stats-lifetime` is registered, non-loopback callers receive only non-sensitive aggregate data, loopback/trusted dashboard clients receive the full lifetime payload, admin/debug endpoints remain loopback-only, and persistent metrics normalize malformed stored state without type/lint errors. - Not tested: Full repository pytest suite, full dashboard browser screenshot pass, or live long-running proxy traffic. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes No changelog entry is required for this dashboard/internal metrics iteration. The endpoint intentionally exposes only aggregate lifetime data to ordinary network callers and strips project/persistence details unless the caller passes the dashboard metadata access policy. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-16 02:18:13 +08:00
assert "recent_requests" in trusted
assert "recent_requests" not in forged
def test_dashboard_client_cidr_normalizes_ipv4_mapped_ipv6(
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.setenv("HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS", "100.90.0.0/24")
app = _make_app()
fix(proxy): repair main lint (ruff-format drift + mypy host_header) (#2268) ## Description `main`'s `lint` CI job is currently **red** (latest main `eac49656` → `lint: failure`), which blocks every open PR. Two causes, both from recent merges that were green in isolation but combined into a red `main`: - **ruff-format drift** on 7 files — committed with formatter output that ruff `0.15.17` (the CI-pinned version) rewrites. - **mypy error** in `server.py`: `_request_has_same_origin_or_no_provenance(request, host_header)` — `host_header` is `request.headers.get("host")` (`str | None`) but the function requires `str`. These passed per-PR because each PR's checks ran against an older base; the serialized `main` state is what went red — a logical-merge / tool-version gap that per-PR CI doesn't catch without a strict merge queue. ## Type of Change - [x] Bug fix (CI/lint repair) ## Changes Made - `ruff format` (0.15.17) the 7 drifted files — formatting only, no logic changes: `cli/proxy.py`, `proxy/forwarded_headers.py`, `proxy/savings_tracker.py`, `proxy/server.py`, `tests/conftest.py`, `tests/test_persistent_metrics_persistence.py`, `tests/test_proxy_loopback_gating.py`. - Add `assert host_header is not None` after the `is_ip_literal_host_header()` guard (which already rejects a missing Host), narrowing the type for the same-origin check. ## Testing - [x] `ruff check .` — clean - [x] `ruff format --check .` — clean (tracked) - [x] `mypy headroom --ignore-missing-imports` — clean ### Test Output ```text $ mypy headroom --ignore-missing-imports → Success: no issues found in 504 source files $ ruff check . → All checks passed (tracked) $ ruff format --check . → clean (tracked) ``` ## Real Behavior Proof - Environment: branch off current `main` (`eac49656`), ruff 0.15.17 + mypy 1.20.2 (CI-pinned). - Confirmed `lint: failure` on main's latest CI run; after this change all three lint steps pass locally. - Not tested: full pytest suite — formatting + a type-narrowing `assert` only, no behavior change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines (this *is* the style fix) - [x] I have performed a self-review of my code - [x] My changes generate no new warnings - [ ] Tests added (N/A — no behavior change) - [x] New and existing unit tests pass locally - [ ] CHANGELOG (N/A) ## Additional Notes The 7 files were touched by recent merges (#2198, #2247) whose local ruff differed from the pinned `0.15.17`. Merging this unblocks the `lint` gate for all open PRs (including #2207). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-07-15 23:18:50 -07:00
payload = (
TestClient(
app,
base_url="http://100.82.0.2:8787",
client=("::ffff:100.90.0.5", 12345),
)
.get("/stats")
.json()
)
feat(dashboard): persist lifetime proxy metrics (#2198) ## Description Persist bounded, aggregate-only Lifetime dashboard metrics across proxy restarts and expose them through a new `/stats-lifetime` endpoint. The change keeps session/runtime stats separate from durable lifetime stats, gates sensitive dashboard metadata for loopback or explicitly trusted dashboard clients, and updates the dashboard Lifetime view to consume the new endpoint. Closes #2137 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Added bounded persistent lifetime metrics state and wired proxy metric events into it. - Added `/stats-lifetime` with sensitive project/persistence details gated behind dashboard metadata access checks. - Extended loopback/dashboard metadata access policy for trusted dashboard client CIDRs without widening admin/debug endpoints. - Reorganized dashboard session/lifetime presentation around runtime counters versus durable aggregates. - Added focused tests for persistent aggregation, persistence, endpoint registration, loopback gating, trusted dashboard CIDRs, and recent request ordering. - Fixed current Ruff/mypy issues in the lifetime metrics normalization code. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text uv run --extra proxy --with pytest --with pytest-asyncio pytest tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_stats_recent_requests.py -q 53 passed, 1 warning uvx ruff==0.15.17 check headroom/proxy/forwarded_headers.py headroom/proxy/loopback_guard.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py headroom/proxy/server.py tests/test_forwarded_headers.py tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_project_savings.py tests/test_proxy_stats_recent_requests.py All checks passed! uv run --extra proxy --with mypy mypy headroom/proxy/persistent_metrics.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows review worktree, Python 3.13.3 via uv. - Exact command / steps: Ran the focused persistent metrics, persistence, loopback gating, and recent request tests; ran CI-matching Ruff on touched files; ran mypy on the new persistent metrics module. - Observed result: `/stats-lifetime` is registered, non-loopback callers receive only non-sensitive aggregate data, loopback/trusted dashboard clients receive the full lifetime payload, admin/debug endpoints remain loopback-only, and persistent metrics normalize malformed stored state without type/lint errors. - Not tested: Full repository pytest suite, full dashboard browser screenshot pass, or live long-running proxy traffic. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes No changelog entry is required for this dashboard/internal metrics iteration. The endpoint intentionally exposes only aggregate lifetime data to ordinary network callers and strips project/persistence details unless the caller passes the dashboard metadata access policy. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-16 02:18:13 +08:00
assert "recent_requests" in payload
def test_dashboard_client_cidr_does_not_expand_other_management_endpoints(
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.setenv("HEADROOM_PROXY_TRUSTED_DASHBOARD_CLIENT_CIDRS", "100.90.0.5/32")
client = TestClient(
_make_app(),
base_url="http://100.82.0.2:8787",
client=("100.90.0.5", 12345),
)
health = client.get("/health")
assert health.status_code == 200
assert "config" not in health.json()
assert client.get("/admin/upstream").status_code == 404
assert client.get("/debug/tasks").status_code == 404
assert client.post("/stats/reset").status_code == 404