Eliminates the dual-package architecture that was the root cause of #355.
`pip install headroom-ai` now produces ONE wheel containing both the Python
source (headroom/*.py) and the compiled Rust extension (headroom/_core.so).
No more separate `headroom-core-py` package, no more chicken-and-egg with
PyPI publication, no more wheelhouse / PIP_FIND_LINKS / composite-action
plumbing in CI.
This is the canonical pattern used by cryptography, polars, ruff,
pydantic-core, and other Rust-as-core Python packages. Honors the
"Rust as core engine" direction.
## What changed
- pyproject.toml: `[build-system]` swapped from hatchling to maturin.
`[tool.hatch.*]` deleted; `[tool.maturin]` added pointing at
`crates/headroom-py/Cargo.toml` for the cdylib. `python-source = "."`
picks up the root `headroom/` package directly (dashboard HTML
templates and other non-Python files included automatically).
- crates/headroom-py/pyproject.toml: deleted. The crate is no longer a
separate published package; its Cargo.toml stays as the cdylib build
target invoked via `[tool.maturin] manifest-path`.
- crates/headroom-py/python/: deleted (placeholder layout for the old
separate package).
## CI updates
- ci.yml: `test` / `test-extras` / `test-agno` jobs simplified — Rust
toolchain set up before `pip install -e .` (which now invokes maturin
via build-system). Removed the "build wheel + symlink .so" dance.
`build` job swapped from `python -m build` (hatch) to
`maturin build` + `maturin sdist`.
- release.yml: collapsed dual-package matrix into one. New `build-wheels`
matrix produces cross-platform wheels for cp310/11/12/13 ×
{linux x86_64, linux aarch64, macos x86_64, macos aarch64}. New
`collect-dist` aggregator merges artifacts. publish-pypi consumes the
merged dist.
- init-native-e2e.yml: dropped windows-latest from the matrix —
upstream `esaxx-rs` (/MT) and `ort-sys` (/MD) link with conflicting
MSVC C runtime libraries, so the Rust extension cannot build for
win_amd64 today. Tracked as a follow-up; not a blocker for Linux+macOS.
- headroom-e2e-setup: composite action now sets up Rust toolchain +
Swatinem/rust-cache before `pip install -e .[proxy]`.
- eval.yml, publish.yml, rust.yml: same pattern — rust toolchain before
install. rust.yml's wheels job builds from root pyproject.toml (no
more `-m crates/headroom-py/Cargo.toml`).
- e2e/init/Dockerfile, e2e/wrap/Dockerfile: install rust + maturin in
the build stage; copy `crates/` + workspace `Cargo.toml/lock` so the
install can build the extension. Dropped `HEADROOM_REQUIRE_RUST_CORE=false`
from wrap-e2e — the image now ships the full Rust core.
- Dockerfile (main): simplified — no more Layer 2/3 dance with
`headroom-core-py` install + symlink. Single `uv pip install` builds
+ installs everything.
- .devcontainer/Dockerfile: rust toolchain + libssl-dev + maturin
added so `uv sync` builds the extension inside the devcontainer.
## Lockfile + script
- uv.lock: regenerated. No `headroom-core-py` entries remain.
- scripts/build_rust_extension.sh: simplified from a symlink-into-tree
workaround to a thin wrapper around `pip install -e .`. The maturin
build-backend handles placement automatically.
## Local validation (all green on macOS aarch64)
1. Clean venv `pip install -e .` → `from headroom._core import …` works.
2. `maturin build --release` → 13.8 MB wheel, 336 files including
`headroom/_core.cpython-311-darwin.so` (32 MB cdylib) and
`headroom/dashboard/templates/dashboard.html`.
3. `pip install <wheel>` in fresh venv → import works.
4. Wheel contents verified via `unzip -l`.
5. `pytest tests/test_transforms/test_diff_compressor.py` — 29 passed.
6. `pytest tests/test_relevance.py` — 30 passed.
7. `cargo build --workspace` + `cargo test --workspace` — all green.
8. `make ci-precheck` — 176 Python tests + Rust + commitlint green.
## Migration notes
Users on `pip install headroom-ai` get the Rust core automatically
(linux + macos wheels). sdist installs require rust toolchain available
locally — pip will build via maturin.
Closes#355
Supersedes #357 (workarounds-based fix abandoned in favor of
architectural fix)
The Release workflow's multi-arch publish-docker job failed after
78 minutes of QEMU-emulated arm64 cargo compilation. Maturin's
wheel-link repair step needs `patchelf` to bundle external
shared libraries (libssl.so.3, libcrypto.so.3, libzstd.so.1)
into the wheel and rewrite their RPATH:
🔗 External shared libraries to be copied into the wheel:
libssl.so.3 => /usr/lib/aarch64-linux-gnu/libssl.so.3
libzstd.so.1 => /usr/lib/aarch64-linux-gnu/libzstd.so.1.5.7
libcrypto.so.3 => /usr/lib/aarch64-linux-gnu/libcrypto.so.3
💥 maturin failed
Caused by: Failed to execute 'patchelf', did you install it?
Compounding chain:
1. PR #350 added pkg-config + libssl-dev to unblock the cargo build
(openssl-sys couldn't find OpenSSL headers).
2. That made Cargo dynamically link to libssl.
3. Maturin then needs patchelf to rewrite the wheel's RPATH so the
bundled .so references resolve at runtime.
4. patchelf was never installed → fail.
Why this didn't surface in PR CI: docker-native-e2e builds only
the host platform (amd64). The Release workflow's docker-bake
builds linux/amd64 + linux/arm64 via setup-qemu-action, and the
arm64 emulation chain hits the patchelf path (different bundling
heuristic from amd64).
Follow-up that's NOT in this hotfix:
The 78-minute QEMU compile is the bigger structural issue. Switching
the Release workflow to native arm64 runners (`runs-on:
ubuntu-24.04-arm`) would cut that to ~5 min. Filing separately.
Run that failed: 25268839539
The build-stage verify kept failing in PR #350 CI with
"ModuleNotFoundError: No module named 'headroom._core'" even after
the install order was correct. Diagnostic dump (commit 28a4883)
proved why:
headroom.__file__ = /build/headroom/__init__.py
headroom.__path__ = ['/build/headroom']
WORKDIR /build puts cwd at the front of sys.path for python -c.
Python resolves `import headroom` to /build/headroom/ — the source
tree COPYd in by Layer 3 — instead of
/usr/local/lib/python3.11/site-packages/headroom/ where the wheel
installed _core.so. The source tree has no _core.so, so the import
falsely fails.
Build-time-only quirk: production startup runs the proxy from a
different cwd where site-packages wins. The customer's box that
motivated A0 was hitting a different failure mode entirely (no
_core.so in the venv at all).
Fix: cd /tmp && python -c ... — /tmp has no headroom/ directory, so
import resolution falls through to site-packages, matching production
order. Removed the diagnostic preamble; it served its purpose.
Diagnostic step in the Dockerfile builder: list site-packages/headroom/
contents, run pip show -f on both headroom-core-py and headroom-ai,
print sys.path and headroom.__path__ before the import-verify. Lets us
see exactly what's on disk when A0's build-time verify keeps failing
in PR #350 CI. Will be removed once the wheel install order issue is
diagnosed.
PR #350 CI: docker-native-e2e's wheel install succeeded but the
build-stage verify (`from headroom._core import hello`) failed with
`ModuleNotFoundError: No module named 'headroom._core'`. Same failure
mode the customer hit in production (Finding #2) — but in CI we have
the full layer trace.
Root cause: the headroom-core-py wheel claims ownership of both
`headroom/__init__.py` (stub from maturin's python-source layout)
AND `headroom/_core.cpython-*.so`. The previous Dockerfile installed
headroom-ai FIRST (which laid down the real `headroom/` tree), then
the wheel SECOND with `--force-reinstall`. pip's --force-reinstall
uninstalls the wheel's previously installed files before reinstalling
— but the wheel's stub `__init__.py` had already overwritten
headroom-ai's at first install. Net result: pip deleted
`headroom/__init__.py` and `headroom/_core.so` ownership records
got into a state where the .so wasn't present after the install.
Fix: swap the order. Install the wheel first (lays down stub
`__init__.py` + `_core.so`), then install headroom-ai (overwrites the
stub with the real `__init__.py` and adds the rest of the
`headroom/` tree). `_core.so` survives because headroom-ai doesn't
claim ownership of it. Drop `--force-reinstall` from the wheel step
since nothing is installing the wheel before it.
This is the exact failure A0 was designed to catch — a deployment
that ships without `_core` working. CI is now serving as a
regression gate for the production install path.
The remaining 3 PR check failures (validate × 3 / Dev Containers)
are environmental: the runner's PyPI mirror (`pypi.netflix.net`)
times out fetching `cuda-bindings==12.9.4` /
`nvidia-cuda-cupti-cu12==12.8.90` / `safetensors==0.7.0`. These come
from `headroom-ai[dev]` → `sentence-transformers` → `torch` → CUDA
deps. Not caused by the realignment branch; the post-create script
needs a `--extra dev-light` profile or the mirror needs the packages
cached. Tracking separately.
Two CI failures introduced by Hotfix-A0's deployment-stage smoke test:
1. docker-native-e2e: the new maturin step in the builder stage failed
with "Could not find openssl via pkg-config". The workspace
transitively depends on `openssl-sys` (via reqwest's native-tls
path in some dep chain). The previous Dockerfile only installed
`build-essential`/`g++`/`curl`/`ca-certificates` — enough for the
proxy binary build because cached target/ artefacts already had
openssl-sys compiled, but the fresh maturin invocation hits a cold
build and needs the dev headers. Add `pkg-config` + `libssl-dev`.
2. docker-wrap-e2e: this image is a `node:22-bookworm` base that
installs headroom in editable mode for CLI-routing-only tests
(aider, codex, openclaw via the wrap subcommand). It deliberately
does NOT build the Rust extension. After A0, the proxy
`lifespan` startup refuses to start when `headroom._core` can't
import — so the wrap-e2e proxy port never opens, the harness's
/health check times out, and the test fails. The wrap-e2e scope
doesn't cover compression behaviour, so set
`HEADROOM_REQUIRE_RUST_CORE=false` to start in degraded
Python-only mode. Compression is exercised end-to-end by the
smoke-test and docker-native-e2e jobs which build via the main
Dockerfile.
The remaining 3 PR check failures (validate * 3) were transient
PyPI download failures (`nvidia-cuda-cupti-cu12==12.8.90`,
`safetensors==0.7.0`) — unrelated to the realignment branch; they
need a re-run, not a code change.
Production incident (Finding #2 of HEADROOM_PROXY_LOG_FINDINGS_2026_05_03.md):
on this customer's deployment the Rust extension `headroom._core` was
never installed into the runtime Docker image. Diff compression failed
54 times in a single day; "Optimization failed: ModuleNotFoundError" hit
379 times. The failure rate climbed every day and reached ~223/day on
2026-05-03 — effectively 100% of requests on the Rust path. Every Rust
PR we'd merged (MessageScorer, ICM, DiffCompressor, etc.) was providing
zero customer value because the module wasn't loadable at all.
Root cause: the Dockerfile builder stage installed Python deps and the
in-tree `headroom-ai` package but never ran `maturin build` for the
`headroom-py` crate, so the runtime image shipped without `_core.so`.
The Python proxy continued to start because the extension's absence is
caught and routed through Python-only fallbacks that either silently
no-op or raise per-request.
This change makes that mode impossible by default:
* `headroom.proxy.server._check_rust_core()` runs as the first step of
the FastAPI lifespan. If the import fails it prints a structured
diagnostic, logs `event=rust_core_missing`, and calls `sys.exit(78)`
(sysexits.h `EX_CONFIG`). Process supervisors (systemd / k8s /
docker) treat this as a deliberate config error and stop restart
loops.
* `HEADROOM_REQUIRE_RUST_CORE=false` is the explicit opt-out for
Python-only `pip install -e .` developer flows; lifespan logs
`event=rust_core_disabled` and continues. Any other value (including
unset) keeps the fail-loud default.
* `/health` now surfaces `rust_core: "loaded" | "disabled" | "missing"`
(plus `rust_core_error` when non-loaded) so operators can alert on
the degraded state rather than discovering it via a customer ticket.
* `scripts/build_rust_extension.sh` is the single dev-time path: build
→ install → import-verify with the same `hello()` marker the lifespan
checks. Failures are loud at every step.
* `Makefile` exposes the script as `make verify-rust-core`.
* `Dockerfile` now installs `rustup` + `maturin`, builds the wheel from
`crates/headroom-py`, force-installs it into site-packages, and runs
the same `hello()` import-verify in the build image so a broken build
fails the docker-build, not the next runtime restart.
Tests:
* `tests/test_rust_core_smoke.py` pins all four contracts:
- `_core.hello()` returns `"headroom-core"`
- missing extension + default env → `SystemExit(78)`
- missing extension + opt-out env → lifespan starts, `/health`
returns `rust_core: "disabled"` with the underlying error
- present extension + default env → `("loaded", None)`
Per-finding-#2: ~/Desktop/HEADROOM_PROXY_LOG_FINDINGS_2026_05_03.md.
Add a default devcontainer and a compose-backed memory-stack profile, validate them in CI, and document the contributor workflow.
Also lock the memory-stack dependencies, pin related container tooling, and sync the latest healthcheck shutdown fix for stubbed memory handlers.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add /livez and /readyz, keep /health backward-compatible, and wire readiness into Docker artifacts and docs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add .dockerignore to exclude unnecessary files from Docker builds
- Add Dockerfile to containerize Python application with health checks
- Add docker-compose.yml for easy local development and deployment of headroom proxy service
