headroom/tests/test_proxy_hardening.py
Tejas Chopra 546ab553dc
feat(proxy): pilot hardening — inbound auth, security headers, audit log, air-gap switch (#1537)
## Description

Tier-2 pilot security hardening from the engineering plan (Tier-1 landed
in #1515). Four operator-facing controls, each verified open on `main`
and grounded in a real exposure or enterprise requirement rather than a
form answer:

- **Optional inbound auth token (`HEADROOM_PROXY_TOKEN`).** When set,
non-loopback callers to the data plane must present it (`Authorization:
Bearer <token>` or `X-Headroom-Proxy-Token`); loopback callers and
health probes are exempt. Constant-time (bytes) comparison. Closes the
gap where the Docker image binds `0.0.0.0:8787` and exposes
unauthenticated `/v1/*` routes to the pod network. A loud startup
warning fires when binding a non-loopback host with no token set.
- **Response security headers** (`X-Content-Type-Options: nosniff`,
`X-Frame-Options: DENY`, `Referrer-Policy: no-referrer`, HSTS) on every
response, including 401s.
- **Audit log for state-mutating admin endpoints.** A structured
`headroom.audit` JSON event (source IP, method, path, status) for
`/admin/*`, `/cache/clear`, `/stats/reset`; `/admin/runtime-env`
additionally records the changed key names (values omitted so secrets
are never logged). Logger-only — safe under `HEADROOM_STATELESS` (no new
file writes).
- **Air-gap master switch (`HEADROOM_OFFLINE=1`).** Hard-disables all
outbound egress in one flag — telemetry beacon, update check,
license/usage reporter, and HuggingFace model downloads (forces
`HF_HUB_OFFLINE`/`TRANSFORMERS_OFFLINE`) — and logs an offline banner.

The first three live in one outermost security middleware that wraps
every inbound request; the offline switch is centralized in a new
top-level `headroom/offline.py` predicate the egress paths consult.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)

## Changes Made

- `headroom/offline.py` (new): `is_offline()` predicate +
`apply_offline_env()`; consulted by `beacon.is_telemetry_enabled`,
`update_check.is_update_check_enabled`, and the license-reporter gate.
- `headroom/proxy/audit.py` (new): `headroom.audit` structured logger +
`record_admin_action` / `is_auditable_path`.
- `headroom/proxy/server.py`: outermost `_security_gate` middleware
(token enforcement + security headers + admin audit), offline activation
+ banner in `create_app`, non-loopback-no-token startup warning,
`runtime-env` change auditing, env wiring in `_proxy_config_from_env`.
- `headroom/proxy/models.py`: `ProxyConfig.proxy_token` and
`ProxyConfig.offline`.
- `headroom/cli/proxy.py`: env wiring + a Security banner line (flags
the open-bind case).
- `headroom/telemetry/beacon.py`, `headroom/update_check.py`: offline
short-circuit.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ ruff check <changed files>
All checks passed!

$ mypy headroom/proxy/server.py headroom/offline.py headroom/proxy/audit.py headroom/proxy/models.py \
       headroom/telemetry/beacon.py headroom/update_check.py
Success: no issues found

$ pytest tests/test_proxy_hardening.py -q
15 passed

$ pytest tests/ -q  (full suite, model/eval-dependent dirs ignored)
32 failed, 7131 passed, 126 skipped in 646s
```

The 32 failures are pre-existing/environmental, not introduced by this
change — verified by running the same tests on `main` (they fail
identically there). They are all `...Real` / `...live` / `real_api`
integration tests that make live backend calls: AWS Bedrock returns
"model is Legacy, access denied" on this host's credentials, plus a
local tree-sitter version that requires `bytes`. On CI (no AWS/API
creds) these tests skip. None touch the hardening code paths.

## Real Behavior Proof

- Environment: macOS, Python 3.12, repo `.venv`; tests via FastAPI
`TestClient` against `create_app`.
- Exact command / steps: configure `ProxyConfig(proxy_token="...")`,
then issue requests from a non-loopback client (`client=("203.0.113.5",
...)`) and a loopback client (`client=("127.0.0.1", ...)`).
- Observed result: non-loopback request with no/!wrong token → 401; with
correct `Authorization: Bearer` or `X-Headroom-Proxy-Token` → not 401;
loopback and `/livez`/`/readyz` → never challenged. Every response
(incl. the 401) carries `X-Content-Type-Options: nosniff` /
`X-Frame-Options: DENY`. `POST /cache/clear` emits a `headroom.audit`
JSON line with the source IP, path, and status. `HEADROOM_OFFLINE=1`
makes `is_telemetry_enabled()` and `is_update_check_enabled()` return
False and sets `HF_HUB_OFFLINE`/`TRANSFORMERS_OFFLINE`.
- Not tested: WebSocket routes (see limitations); live upstream proxying
of `/v1/*` (covered by existing integration tests / CI).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

Known limitations (by design / scope, documented in code):
- WebSocket routes are not covered by the HTTP token middleware
(`@app.middleware("http")` does not run for WS). The HTTP data plane is
the main surface; the open-bind warning still applies. Follow-up.
- The token keys off the direct peer IP — behind a same-host reverse
proxy all requests appear loopback, so enforce auth at the reverse proxy
in that topology (same property as the existing loopback guard).
- OTEL metrics export is intentionally left on under offline mode — it
targets the customer's own sink, not a Headroom phone-home.

CHANGELOG not updated (handled by release tooling).
2026-06-28 12:09:00 -07:00

182 lines
7.5 KiB
Python

"""Tests for the Tier-2 pilot hardening features:
- 2.1 optional inbound auth token (HEADROOM_PROXY_TOKEN) on the data plane
- 3.1 response security headers
- 2.4 admin/state-mutating audit log
- 2.2 air-gap master switch (HEADROOM_OFFLINE)
"""
from __future__ import annotations
import logging
import pytest
pytest.importorskip("fastapi")
from fastapi.testclient import TestClient
from headroom.cache.compression_store import reset_compression_store
from headroom.offline import apply_offline_env, is_offline
from headroom.proxy.audit import is_auditable_path
from headroom.proxy.server import ProxyConfig, create_app
NONLOOPBACK = ("203.0.113.5", 44444) # TEST-NET-3, never loopback
LOOPBACK = ("127.0.0.1", 12345)
def _make_app(**overrides):
reset_compression_store()
config = ProxyConfig(
optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
**overrides,
)
return create_app(config)
# ───────────────────────────── 2.1 inbound auth token ─────────────────────
class TestInboundAuthToken:
def test_no_token_configured_leaves_data_plane_open(self):
"""Default (no token): non-loopback callers are not challenged."""
app = _make_app()
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
assert c.get("/livez").status_code == 200
def test_token_set_rejects_nonloopback_without_credential(self):
app = _make_app(proxy_token="s3cr3t-token")
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
resp = c.get("/stats")
assert resp.status_code == 401
def test_token_set_accepts_correct_bearer(self):
app = _make_app(proxy_token="s3cr3t-token")
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
resp = c.get("/stats", headers={"Authorization": "Bearer s3cr3t-token"})
assert resp.status_code != 401
def test_token_set_accepts_custom_header(self):
app = _make_app(proxy_token="s3cr3t-token")
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
resp = c.get("/stats", headers={"X-Headroom-Proxy-Token": "s3cr3t-token"})
assert resp.status_code != 401
def test_token_set_rejects_wrong_token(self):
app = _make_app(proxy_token="s3cr3t-token")
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
resp = c.get("/stats", headers={"Authorization": "Bearer wrong"})
assert resp.status_code == 401
def test_loopback_is_exempt_from_token(self):
"""Loopback callers (same trust boundary as admin routes) skip the token."""
app = _make_app(proxy_token="s3cr3t-token")
with TestClient(app, base_url="http://127.0.0.1", client=LOOPBACK) as c:
assert c.get("/stats").status_code != 401
def test_health_endpoints_exempt_even_nonloopback(self):
"""Orchestrator health probes must work without the token."""
app = _make_app(proxy_token="s3cr3t-token")
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
assert c.get("/livez").status_code == 200
assert c.get("/readyz").status_code in (200, 503) # ready/not-ready, never 401
# ───────────────────────────── 3.1 security headers ───────────────────────
class TestSecurityHeaders:
def test_headers_present_on_responses(self):
app = _make_app()
with TestClient(app, base_url="http://127.0.0.1", client=LOOPBACK) as c:
h = c.get("/livez").headers
assert h.get("X-Content-Type-Options") == "nosniff"
assert h.get("X-Frame-Options") == "DENY"
assert h.get("Referrer-Policy") == "no-referrer"
assert "max-age=" in h.get("Strict-Transport-Security", "")
def test_headers_present_on_401(self):
app = _make_app(proxy_token="s3cr3t-token")
with TestClient(app, base_url="http://testserver", client=NONLOOPBACK) as c:
resp = c.get("/stats")
assert resp.status_code == 401
assert resp.headers.get("X-Content-Type-Options") == "nosniff"
# ───────────────────────────── 2.4 admin audit log ────────────────────────
class TestAdminAuditLog:
def test_auditable_path_classification(self):
assert is_auditable_path("/admin/runtime-env")
assert is_auditable_path("/cache/clear")
assert is_auditable_path("/stats/reset")
assert not is_auditable_path("/v1/messages")
assert not is_auditable_path("/livez")
def test_cache_clear_emits_audit_event(self):
# Capture the dedicated audit logger directly (the proxy's logging setup
# configures propagation, so attach to the logger rather than rely on
# caplog's root handler).
messages: list[str] = []
class _Capture(logging.Handler):
def emit(self, record: logging.LogRecord) -> None:
messages.append(record.getMessage())
handler = _Capture()
audit_logger = logging.getLogger("headroom.audit")
audit_logger.setLevel(logging.INFO)
audit_logger.addHandler(handler)
try:
app = _make_app()
with TestClient(app, base_url="http://127.0.0.1", client=LOOPBACK) as c:
assert c.post("/cache/clear").status_code == 200
finally:
audit_logger.removeHandler(handler)
assert messages, "expected an audit record for /cache/clear"
assert any("/cache/clear" in m for m in messages)
assert any("headroom_admin_audit" in m for m in messages)
assert any('"source_ip": "127.0.0.1"' in m for m in messages)
# ───────────────────────────── 2.2 air-gap switch ─────────────────────────
class TestOfflineSwitch:
def test_is_offline_reads_env(self, monkeypatch):
monkeypatch.delenv("HEADROOM_OFFLINE", raising=False)
assert is_offline() is False
monkeypatch.setenv("HEADROOM_OFFLINE", "1")
assert is_offline() is True
monkeypatch.setenv("HEADROOM_OFFLINE", "off")
assert is_offline() is False
def test_offline_disables_telemetry(self, monkeypatch):
from headroom.telemetry.beacon import is_telemetry_enabled
monkeypatch.setenv("HEADROOM_TELEMETRY", "on")
monkeypatch.setenv("HEADROOM_OFFLINE", "1")
assert is_telemetry_enabled() is False # offline overrides the opt-in
def test_offline_disables_update_check(self, monkeypatch):
from headroom.update_check import is_update_check_enabled
monkeypatch.delenv("CI", raising=False)
monkeypatch.delenv("HEADROOM_STATELESS", raising=False)
monkeypatch.setenv("HEADROOM_OFFLINE", "1")
assert is_update_check_enabled() is False
def test_apply_offline_env_sets_hf_offline(self, monkeypatch):
monkeypatch.delenv("HF_HUB_OFFLINE", raising=False)
monkeypatch.delenv("TRANSFORMERS_OFFLINE", raising=False)
monkeypatch.setenv("HEADROOM_OFFLINE", "1")
apply_offline_env()
import os
assert os.environ.get("HF_HUB_OFFLINE") == "1"
assert os.environ.get("TRANSFORMERS_OFFLINE") == "1"