fix: A0 — fail-loud rust core deployment smoke test

Production incident (Finding #2 of HEADROOM_PROXY_LOG_FINDINGS_2026_05_03.md):
on this customer's deployment the Rust extension `headroom._core` was
never installed into the runtime Docker image. Diff compression failed
54 times in a single day; "Optimization failed: ModuleNotFoundError" hit
379 times. The failure rate climbed every day and reached ~223/day on
2026-05-03 — effectively 100% of requests on the Rust path. Every Rust
PR we'd merged (MessageScorer, ICM, DiffCompressor, etc.) was providing
zero customer value because the module wasn't loadable at all.

Root cause: the Dockerfile builder stage installed Python deps and the
in-tree `headroom-ai` package but never ran `maturin build` for the
`headroom-py` crate, so the runtime image shipped without `_core.so`.
The Python proxy continued to start because the extension's absence is
caught and routed through Python-only fallbacks that either silently
no-op or raise per-request.

This change makes that mode impossible by default:

* `headroom.proxy.server._check_rust_core()` runs as the first step of
  the FastAPI lifespan. If the import fails it prints a structured
  diagnostic, logs `event=rust_core_missing`, and calls `sys.exit(78)`
  (sysexits.h `EX_CONFIG`). Process supervisors (systemd / k8s /
  docker) treat this as a deliberate config error and stop restart
  loops.
* `HEADROOM_REQUIRE_RUST_CORE=false` is the explicit opt-out for
  Python-only `pip install -e .` developer flows; lifespan logs
  `event=rust_core_disabled` and continues. Any other value (including
  unset) keeps the fail-loud default.
* `/health` now surfaces `rust_core: "loaded" | "disabled" | "missing"`
  (plus `rust_core_error` when non-loaded) so operators can alert on
  the degraded state rather than discovering it via a customer ticket.
* `scripts/build_rust_extension.sh` is the single dev-time path: build
  → install → import-verify with the same `hello()` marker the lifespan
  checks. Failures are loud at every step.
* `Makefile` exposes the script as `make verify-rust-core`.
* `Dockerfile` now installs `rustup` + `maturin`, builds the wheel from
  `crates/headroom-py`, force-installs it into site-packages, and runs
  the same `hello()` import-verify in the build image so a broken build
  fails the docker-build, not the next runtime restart.

Tests:
* `tests/test_rust_core_smoke.py` pins all four contracts:
  - `_core.hello()` returns `"headroom-core"`
  - missing extension + default env → `SystemExit(78)`
  - missing extension + opt-out env → lifespan starts, `/health`
    returns `rust_core: "disabled"` with the underlying error
  - present extension + default env → `("loaded", None)`

Per-finding-#2: ~/Desktop/HEADROOM_PROXY_LOG_FINDINGS_2026_05_03.md.
This commit is contained in:
chopratejas 2026-05-02 17:51:24 -07:00
parent dcbc921d63
commit 00ab1ea74d
5 changed files with 406 additions and 19 deletions

View file

@ -12,14 +12,32 @@ FROM python:${PYTHON_VERSION}-slim@${PYTHON_DIGEST} AS builder
ARG UV_VERSION ARG UV_VERSION
# build-essential / g++ for any C extension wheels uv may need to build
# from source. curl + ca-certificates are required by the rustup
# bootstrap below. Hotfix-A0 (Finding #2) added the rust toolchain so the
# image actually carries `headroom._core`; previously the runtime image
# shipped without the Rust extension and every compressed request fell
# back to a Python-only path or no-op.
RUN apt-get update && \ RUN apt-get update && \
apt-get install -y --no-install-recommends \ apt-get install -y --no-install-recommends \
build-essential \ build-essential \
g++ \ g++ \
curl \
ca-certificates \
&& rm -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/*
RUN python -m pip install --no-cache-dir uv==${UV_VERSION} RUN python -m pip install --no-cache-dir uv==${UV_VERSION}
# Rust toolchain for the headroom._core extension build. Pinned via
# rust-toolchain.toml at the repo root so this matches what local devs
# build with. Installed as root before WORKDIR change so the env
# additions stick for every subsequent RUN.
ENV CARGO_HOME=/usr/local/cargo \
RUSTUP_HOME=/usr/local/rustup \
PATH=/usr/local/cargo/bin:${PATH}
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs \
| sh -s -- -y --no-modify-path --profile minimal --default-toolchain stable
WORKDIR /build WORKDIR /build
# Layer 1: install deps only (cached unless pyproject.toml/uv.lock change) # Layer 1: install deps only (cached unless pyproject.toml/uv.lock change)
@ -35,6 +53,28 @@ COPY headroom/ headroom/
RUN --mount=type=cache,target=/root/.cache/uv \ RUN --mount=type=cache,target=/root/.cache/uv \
uv pip install --system --no-deps --reinstall-package headroom-ai . uv pip install --system --no-deps --reinstall-package headroom-ai .
# Layer 3 (Hotfix-A0): build and install the Rust extension wheel. uv
# already installed `maturin` as a transitive of the [proxy]/[code]
# extras; if it didn't, install it explicitly here so the build never
# silently skips.
COPY crates/ crates/
COPY Cargo.toml Cargo.lock rust-toolchain.toml ./
RUN --mount=type=cache,target=/root/.cache/uv \
--mount=type=cache,target=/root/.cargo/registry \
--mount=type=cache,target=/build/target \
uv pip install --system maturin \
&& maturin build --release -m crates/headroom-py/Cargo.toml --out /build/wheels \
&& uv pip install --system --force-reinstall --no-deps /build/wheels/headroom_core_py-*.whl
# Layer 4 (Hotfix-A0): verify the extension actually loads end-to-end
# inside the build image. If this fails, the runtime image would fail
# its lifespan smoke test on every restart — better to break the build
# loudly here than ship a broken image.
RUN python -c "from headroom._core import hello; \
marker = hello(); \
assert marker == 'headroom-core', f'expected headroom-core, got {marker!r}'; \
print(f'build-stage rust core verify OK: {marker}')"
# ---- Runtime stage (python-slim): supports root/nonroot via build arg ---- # ---- Runtime stage (python-slim): supports root/nonroot via build arg ----
FROM python:${PYTHON_VERSION}-slim@${PYTHON_DIGEST} AS runtime-slim-base FROM python:${PYTHON_VERSION}-slim@${PYTHON_DIGEST} AS runtime-slim-base

View file

@ -7,7 +7,7 @@ MATURIN ?= maturin
PYTHON ?= python3 PYTHON ?= python3
FIXTURES ?= tests/parity/fixtures FIXTURES ?= tests/parity/fixtures
.PHONY: help test test-parity bench build-proxy build-wheel fmt fmt-check lint clippy clean ci-precheck ci-precheck-rust ci-precheck-python ci-precheck-commitlint install-git-hooks .PHONY: help test test-parity bench build-proxy build-wheel fmt fmt-check lint clippy clean ci-precheck ci-precheck-rust ci-precheck-python ci-precheck-commitlint install-git-hooks verify-rust-core
help: help:
@echo "Headroom Rust targets:" @echo "Headroom Rust targets:"
@ -16,6 +16,7 @@ help:
@echo " make bench - cargo bench --workspace" @echo " make bench - cargo bench --workspace"
@echo " make build-proxy - release build + strip headroom-proxy, print size" @echo " make build-proxy - release build + strip headroom-proxy, print size"
@echo " make build-wheel - release wheel for headroom-py" @echo " make build-wheel - release wheel for headroom-py"
@echo " make verify-rust-core - build + install + import-verify headroom._core"
@echo " make fmt - cargo fmt --all" @echo " make fmt - cargo fmt --all"
@echo " make fmt-check - cargo fmt --all -- --check" @echo " make fmt-check - cargo fmt --all -- --check"
@echo " make lint - cargo clippy --workspace -- -D warnings" @echo " make lint - cargo clippy --workspace -- -D warnings"
@ -52,6 +53,18 @@ build-proxy:
build-wheel: build-wheel:
$(MATURIN) build --release -m crates/headroom-py/Cargo.toml $(MATURIN) build --release -m crates/headroom-py/Cargo.toml
# Hotfix-A0: maturin-develop + symlink + import-verify in one shot. Run this
# any time you suspect the proxy is silently falling back to Python-only
# mode (Finding #2 in HEADROOM_PROXY_LOG_FINDINGS_2026_05_03.md). The
# proxy itself runs the same check at lifespan startup; this target
# exposes it as a developer-facing one-liner.
verify-rust-core:
@if [ -z "$$VIRTUAL_ENV" ]; then \
echo "error: activate a venv first (e.g. source .venv/bin/activate)"; \
exit 1; \
fi
bash scripts/build_rust_extension.sh
fmt: fmt:
$(CARGO) fmt --all $(CARGO) fmt --all

View file

@ -186,6 +186,93 @@ logger = logging.getLogger("headroom.proxy")
_MULTI_WORKER_CONFIG_ENV = "HEADROOM_PROXY_CONFIG_JSON" _MULTI_WORKER_CONFIG_ENV = "HEADROOM_PROXY_CONFIG_JSON"
# Env var that opts out of the Rust core deployment smoke test (Hotfix-A0).
# Default behavior: hard-fail at startup if `headroom._core` is unimportable
# (Finding #2 in HEADROOM_PROXY_LOG_FINDINGS_2026_05_03.md — production
# deployment was silently running without the Rust extension and degrading
# every compressed request to a Python-only path or a no-op).
#
# Set to the literal string "false" to start the proxy in degraded
# Python-only mode. Any other value (including unset) keeps the
# fail-loud behavior.
_RUST_CORE_REQUIRED_ENV = "HEADROOM_REQUIRE_RUST_CORE"
# sysexits.h(3) — EX_CONFIG. Process supervisors (systemd, k8s, docker)
# treat this as a deliberate configuration failure rather than a crash, so
# they won't restart-loop on a broken deployment.
_EXIT_CONFIG = 78
def _check_rust_core() -> tuple[str, str | None]:
"""Verify the Rust extension `headroom._core` is loadable at startup.
Returns a `(status, error)` tuple:
- ``("loaded", None)`` `headroom._core.hello()` returned the
expected sentinel.
- ``("disabled", reason)`` opt-out env var was set; proxy starts
in Python-only degraded mode. `reason` carries the underlying
import error (or ``None`` if the import actually succeeded).
- ``("missing", reason)`` never returned: this branch calls
``sys.exit(78)`` so the proxy refuses to start. The branch exists
only as a typed sentinel for callers that want to reason about
all three states (e.g. health endpoints).
Behavior is gated by the ``HEADROOM_REQUIRE_RUST_CORE`` env var:
any value other than ``"false"`` (case-insensitive) keeps the
fail-loud default.
"""
require = os.environ.get(_RUST_CORE_REQUIRED_ENV, "true").strip().lower() != "false"
try:
from headroom._core import hello as _rust_hello
marker = _rust_hello()
except Exception as exc: # ImportError, but also any init-time PyO3 failure
reason = f"{type(exc).__name__}: {exc}"
if not require:
logger.warning(
"event=rust_core_disabled reason=%r opt_out_env=%s=false mode=python_only_degraded",
reason,
_RUST_CORE_REQUIRED_ENV,
)
return ("disabled", reason)
# Fail loud. Print to stderr in addition to logging so operators
# see it even if the logging handler is mis-configured.
msg = (
f"FATAL: Rust extension `headroom._core` not loadable.\n"
f" error: {reason}\n"
f" fix: `make build-wheel && pip install --force-reinstall "
f"target/wheels/headroom_*.whl`\n"
f" opt-out: set {_RUST_CORE_REQUIRED_ENV}=false to start in "
f"degraded Python-only mode\n"
)
logger.error("event=rust_core_missing reason=%r action=exit_78", reason)
print(msg, file=sys.stderr, flush=True)
sys.exit(_EXIT_CONFIG)
# Import succeeded; sanity-check the marker so we catch a stale or
# mis-linked .so where the symbol name resolves but returns garbage.
if marker != "headroom-core":
reason = f"unexpected marker {marker!r}"
if not require:
logger.warning(
"event=rust_core_disabled reason=%r opt_out_env=%s=false",
reason,
_RUST_CORE_REQUIRED_ENV,
)
return ("disabled", reason)
msg = (
f"FATAL: Rust extension `headroom._core` is loaded but the "
f"marker function returned {marker!r}; expected 'headroom-core'.\n"
f" fix: rebuild: `make build-wheel && pip install "
f"--force-reinstall target/wheels/headroom_*.whl`\n"
)
logger.error("event=rust_core_marker_mismatch marker=%r action=exit_78", marker)
print(msg, file=sys.stderr, flush=True)
sys.exit(_EXIT_CONFIG)
logger.info("event=rust_core_loaded marker=%r", marker)
return ("loaded", None)
# Compression pipeline timeout in seconds # Compression pipeline timeout in seconds
@ -1258,6 +1345,17 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
@asynccontextmanager @asynccontextmanager
async def lifespan(app: FastAPI): # type: ignore[no-untyped-def] async def lifespan(app: FastAPI): # type: ignore[no-untyped-def]
# Hotfix-A0: Rust core deployment smoke test. Refuse to accept
# traffic if the Rust extension is missing unless the operator
# explicitly opted out with HEADROOM_REQUIRE_RUST_CORE=false. See
# Finding #2 in HEADROOM_PROXY_LOG_FINDINGS_2026_05_03.md.
# `_check_rust_core` either returns ("loaded"|"disabled", _) or
# calls `sys.exit(78)` — execution past this line implies the
# rust_core_status is recorded.
_rust_core_status, _rust_core_error = _check_rust_core()
app.state.rust_core_status = _rust_core_status
app.state.rust_core_error = _rust_core_error
configure_otel_metrics(OTelMetricsConfig.from_env(default_service_name="headroom-proxy")) configure_otel_metrics(OTelMetricsConfig.from_env(default_service_name="headroom-proxy"))
configure_langfuse_tracing( configure_langfuse_tracing(
LangfuseTracingConfig.from_env(default_service_name="headroom-proxy") LangfuseTracingConfig.from_env(default_service_name="headroom-proxy")
@ -1315,6 +1413,12 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
app.state.started_at = None app.state.started_at = None
app.state.ready = False app.state.ready = False
app.state.startup_error = None app.state.startup_error = None
# Set by the lifespan startup smoke test (`_check_rust_core`). Default
# "missing" means lifespan hasn't run yet — anything reading /health
# before startup completes (rare; lifespan runs before the first
# request) sees an honest "missing" rather than a stale "loaded".
app.state.rust_core_status = "missing"
app.state.rust_core_error = None
def _iso_utc_now() -> str: def _iso_utc_now() -> str:
return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z") return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
@ -1432,7 +1536,13 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
"uptime_seconds": _uptime_seconds(), "uptime_seconds": _uptime_seconds(),
"checks": checks, "checks": checks,
"runtime": _runtime_payload(), "runtime": _runtime_payload(),
# Hotfix-A0: surface rust core load state so operators can alert
# on `rust_core != "loaded"` (Finding #2).
"rust_core": getattr(app.state, "rust_core_status", "missing"),
} }
rust_core_error = getattr(app.state, "rust_core_error", None)
if rust_core_error:
payload["rust_core_error"] = rust_core_error
deployment_profile = os.environ.get("HEADROOM_DEPLOYMENT_PROFILE") deployment_profile = os.environ.get("HEADROOM_DEPLOYMENT_PROFILE")
if deployment_profile: if deployment_profile:
payload["deployment"] = { payload["deployment"] = {

View file

@ -9,37 +9,82 @@
# the maturin overlay, so `import headroom._core` fails. Symlinking the # the maturin overlay, so `import headroom._core` fails. Symlinking the
# built `.so` into `headroom/` fixes the lookup with zero copies. # built `.so` into `headroom/` fixes the lookup with zero copies.
# #
# Hotfix-A0 (2026-05-02): the script now also runs an end-to-end import
# verification with the `hello()` marker so a partial / stale build is
# caught before the proxy is started. This mirrors the lifespan smoke
# test in `headroom.proxy.server._check_rust_core` so dev-time and
# deploy-time both catch the same class of failure.
#
# Idempotent. Safe to run repeatedly. Requires `maturin` in PATH (i.e. # Idempotent. Safe to run repeatedly. Requires `maturin` in PATH (i.e.
# inside the project venv). # inside the project venv).
set -euo pipefail set -euo pipefail
# Move to repo root so all relative paths below are stable.
cd "$(dirname "$0")/.." cd "$(dirname "$0")/.."
log() {
printf '[build_rust_extension] %s\n' "$*" >&2
}
fail() {
printf '[build_rust_extension] error: %s\n' "$*" >&2
exit 1
}
# Step 1: pre-flight. Maturin must be on PATH and a venv must be active —
# `maturin develop` writes into site-packages, and we want that write to
# land in the same env the proxy will run in.
if ! command -v maturin >/dev/null 2>&1; then if ! command -v maturin >/dev/null 2>&1; then
echo "error: maturin not found. Activate the venv first:" >&2 fail "maturin not found on PATH. Activate the venv first: source .venv/bin/activate"
echo " source .venv/bin/activate" >&2
exit 1
fi fi
# Build the wheel + install into the venv site-packages. if [[ -z "${VIRTUAL_ENV:-}" ]]; then
maturin develop -m crates/headroom-py/Cargo.toml log "warning: VIRTUAL_ENV is unset; maturin will install into the system Python."
log " If that is not what you want, abort and 'source .venv/bin/activate' first."
fi
# Locate the built `.so`. `maturin develop` writes it under # Step 2: build + install via `maturin develop` (in-place editable install
# `crates/headroom-py/python/headroom/_core.cpython-<ver>-<platform>.so`. # with C extensions). This produces a `.so` under
# crates/headroom-py/python/headroom/.
log "step 1/3: maturin develop"
maturin develop -m crates/headroom-py/Cargo.toml \
|| fail "maturin develop failed (see output above)"
# Step 3: locate the built artifact. `maturin develop` writes
# `_core.cpython-<ver>-<platform>.{so,dylib,pyd}` into the package dir.
SO_FILE=$(find crates/headroom-py/python/headroom -maxdepth 1 \ SO_FILE=$(find crates/headroom-py/python/headroom -maxdepth 1 \
-name "_core.cpython-*.so" -o -name "_core.cpython-*.dylib" -o -name "_core.pyd" \ \( -name "_core.cpython-*.so" -o -name "_core.cpython-*.dylib" -o -name "_core.pyd" \) \
2>/dev/null | head -1) 2>/dev/null | head -1 || true)
if [[ -z "$SO_FILE" ]]; then if [[ -z "${SO_FILE}" ]]; then
echo "error: maturin develop succeeded but produced no _core.* binary." >&2 fail "maturin develop succeeded but produced no _core.* binary in crates/headroom-py/python/headroom/"
exit 1
fi fi
# Symlink into the in-tree package dir. # Step 4: symlink into the in-tree package dir so the in-tree
LINK_NAME="headroom/$(basename "$SO_FILE")" # `headroom/__init__.py` resolves the `_core` submodule. Only the symlink
ln -sf "$(pwd)/$SO_FILE" "$LINK_NAME" # style is supported; copy semantics drift on every rebuild.
echo "linked: $LINK_NAME -> $SO_FILE" LINK_NAME="headroom/$(basename "${SO_FILE}")"
ln -sf "$(pwd)/${SO_FILE}" "${LINK_NAME}" \
|| fail "failed to symlink ${SO_FILE} into ${LINK_NAME}"
log "step 2/3: linked ${LINK_NAME} -> ${SO_FILE}"
# Smoke-test the import to fail loudly if anything is misconfigured. # Step 5: end-to-end import verification. This is the same check the
python -c "from headroom._core import DiffCompressor; print('headroom._core OK:', DiffCompressor)" # proxy lifespan runs at startup. Failing here means the build produced
# something that can't be loaded — fix the build, don't fix the proxy.
log "step 3/3: verifying \`from headroom._core import hello\`"
python -c '
import sys
try:
from headroom._core import hello, DiffCompressor
except Exception as exc:
print(f"verify FAILED: {type(exc).__name__}: {exc}", file=sys.stderr)
sys.exit(1)
marker = hello()
if marker != "headroom-core":
print(f"verify FAILED: hello() returned {marker!r}, expected \"headroom-core\"", file=sys.stderr)
sys.exit(1)
print(f"verify OK: hello()={marker!r}, DiffCompressor={DiffCompressor!r}")
' || fail "import verification failed (see above)"
log "headroom._core build + install + verify: OK"

View file

@ -0,0 +1,179 @@
"""Hotfix-A0 smoke tests: deployment-stage Rust core verification.
Background Finding #2 of HEADROOM_PROXY_LOG_FINDINGS_2026_05_03.md.
A customer's production proxy was silently running without the
`headroom._core` PyO3 extension because the Docker image never built it
into the runtime layer. Diff compression failed 54 times in one day;
optimization failed 379 times. Once the failure rate hit ~100%, every
Rust port we'd shipped was providing zero customer value.
These tests pin the contract for the fix:
1. The PyO3 module exposes a `hello()` marker function returning
``"headroom-core"`` so the deployment smoke test has something stable
to latch onto. Reusing an existing function instead of inventing a
new one means we don't drift the Rust API surface for diagnostic
purposes.
2. The proxy lifespan refuses to start when ``headroom._core`` is
unimportable, exiting with `sysexits.h` ``EX_CONFIG`` (78) so process
supervisors recognize this as a deliberate configuration failure
rather than a crash they should retry forever.
3. An explicit opt-out ``HEADROOM_REQUIRE_RUST_CORE=false`` keeps
the dev-time `pip install -e .` workflow alive without forcing every
contributor to run maturin.
The opt-out test also verifies that `/health` surfaces the rust core
state so operators can alert on `rust_core != "loaded"` in production.
"""
from __future__ import annotations
import pytest
# ──────────────────────────────────────────────────────────────────────────
# 1. The Rust extension's `hello()` marker is stable and importable.
# ──────────────────────────────────────────────────────────────────────────
def test_rust_core_imports() -> None:
"""`headroom._core.hello()` returns the documented sentinel.
The deployment smoke test (`headroom.proxy.server._check_rust_core`)
asserts on the exact return value so a stale or mis-linked .so is
caught not just a complete `ImportError`. If you change the marker
string, you also need to update the lifespan check to match.
"""
from headroom._core import hello
assert hello() == "headroom-core"
# ──────────────────────────────────────────────────────────────────────────
# 2. Default behavior: missing extension blocks startup with exit 78.
# ──────────────────────────────────────────────────────────────────────────
def test_proxy_refuses_to_start_when_rust_core_missing(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""When `headroom._core` raises ImportError on import and the opt-out
env var is not set, the lifespan smoke test must call `sys.exit(78)`.
We invoke the helper directly (rather than spinning up FastAPI)
because the helper is the single source of truth for the policy and
`sys.exit` propagates through the lifespan context manager
transparently. Hitting it directly keeps the test tight and avoids
the lifespan's 30+ side effects (OTel, Langfuse, beacon, ...).
"""
from headroom.proxy import server
# Ensure the env var is absent so the default fail-loud path runs.
monkeypatch.delenv("HEADROOM_REQUIRE_RUST_CORE", raising=False)
# Force the import to raise. Patching the symbol on the module is
# not enough because `_check_rust_core` runs `from headroom._core
# import hello` inside its body — we need the import machinery to
# raise. `sys.modules` is the cleanest hook for that.
import sys
# Pre-purge any cached binding so the import statement re-runs.
sys.modules.pop("headroom._core", None)
# Sentinel that pretends to be the module but raises on attribute
# access. `from X import Y` first looks up `X` in sys.modules; if
# present it skips re-import and uses it directly. Setting
# sys.modules["headroom._core"] to None forces a real ImportError.
sys.modules["headroom._core"] = None # type: ignore[assignment]
try:
with pytest.raises(SystemExit) as exc_info:
server._check_rust_core()
assert exc_info.value.code == 78, (
f"expected exit code 78 (EX_CONFIG), got {exc_info.value.code!r}"
)
finally:
# Restore the real module so subsequent tests can import it.
sys.modules.pop("headroom._core", None)
# ──────────────────────────────────────────────────────────────────────────
# 3. Opt-out path: HEADROOM_REQUIRE_RUST_CORE=false → degraded mode + /health.
# ──────────────────────────────────────────────────────────────────────────
def test_proxy_starts_in_degraded_mode_when_opt_out_set(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""With the opt-out env var set, the lifespan must start successfully
even when `headroom._core` is missing, and `/health` must surface
`rust_core: "disabled"` so operators can detect the degraded mode.
We spin up a real FastAPI app via `TestClient` because the spec
requires the health endpoint to reflect the lifespan state the
helper alone doesn't tell us the wiring is right.
"""
from fastapi.testclient import TestClient
from headroom.proxy.server import ProxyConfig, create_app
monkeypatch.setenv("HEADROOM_REQUIRE_RUST_CORE", "false")
# Force the import to fail at lifespan time. Same trick as above:
# sys.modules["headroom._core"] = None makes Python treat the module
# as known-unimportable, raising ImportError on the next `from ...
# import` without re-trying the loader.
import sys
sys.modules.pop("headroom._core", None)
sys.modules["headroom._core"] = None # type: ignore[assignment]
config = ProxyConfig(
optimize=False,
image_optimize=False,
cache_enabled=False,
rate_limit_enabled=False,
cost_tracking_enabled=False,
log_requests=False,
ccr_inject_tool=False,
ccr_handle_responses=False,
ccr_context_tracking=False,
)
app = create_app(config)
try:
with TestClient(app) as client:
response = client.get("/health")
assert response.status_code == 200
payload = response.json()
assert payload["rust_core"] == "disabled", (
f"expected rust_core=disabled, got {payload.get('rust_core')!r}; "
f"full payload keys: {sorted(payload.keys())}"
)
# The error reason should be carried through so operators can
# see *why* the extension isn't loaded.
assert "rust_core_error" in payload
assert (
"ModuleNotFoundError" in payload["rust_core_error"]
or "ImportError" in payload["rust_core_error"]
)
finally:
sys.modules.pop("headroom._core", None)
# ──────────────────────────────────────────────────────────────────────────
# 4. Happy path through the helper: real extension present → status=loaded.
# ──────────────────────────────────────────────────────────────────────────
def test_check_rust_core_returns_loaded_when_extension_present(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""When `headroom._core` is loadable and `hello()` returns the
expected sentinel, `_check_rust_core` returns ``("loaded", None)``.
This covers the production-happy path so a future change that
breaks the marker check (e.g. tightening to require a JSON dict
return) will fail this test rather than degrading silently.
"""
from headroom.proxy import server
# Make sure the env var doesn't tip us into the disabled branch.
monkeypatch.delenv("HEADROOM_REQUIRE_RUST_CORE", raising=False)
status, error = server._check_rust_core()
assert status == "loaded"
assert error is None