refactor: single-wheel maturin build backend (fixes #355)

Eliminates the dual-package architecture that was the root cause of #355.
`pip install headroom-ai` now produces ONE wheel containing both the Python
source (headroom/*.py) and the compiled Rust extension (headroom/_core.so).
No more separate `headroom-core-py` package, no more chicken-and-egg with
PyPI publication, no more wheelhouse / PIP_FIND_LINKS / composite-action
plumbing in CI.

This is the canonical pattern used by cryptography, polars, ruff,
pydantic-core, and other Rust-as-core Python packages. Honors the
"Rust as core engine" direction.

## What changed

- pyproject.toml: `[build-system]` swapped from hatchling to maturin.
  `[tool.hatch.*]` deleted; `[tool.maturin]` added pointing at
  `crates/headroom-py/Cargo.toml` for the cdylib. `python-source = "."`
  picks up the root `headroom/` package directly (dashboard HTML
  templates and other non-Python files included automatically).
- crates/headroom-py/pyproject.toml: deleted. The crate is no longer a
  separate published package; its Cargo.toml stays as the cdylib build
  target invoked via `[tool.maturin] manifest-path`.
- crates/headroom-py/python/: deleted (placeholder layout for the old
  separate package).

## CI updates

- ci.yml: `test` / `test-extras` / `test-agno` jobs simplified — Rust
  toolchain set up before `pip install -e .` (which now invokes maturin
  via build-system). Removed the "build wheel + symlink .so" dance.
  `build` job swapped from `python -m build` (hatch) to
  `maturin build` + `maturin sdist`.
- release.yml: collapsed dual-package matrix into one. New `build-wheels`
  matrix produces cross-platform wheels for cp310/11/12/13 ×
  {linux x86_64, linux aarch64, macos x86_64, macos aarch64}. New
  `collect-dist` aggregator merges artifacts. publish-pypi consumes the
  merged dist.
- init-native-e2e.yml: dropped windows-latest from the matrix —
  upstream `esaxx-rs` (/MT) and `ort-sys` (/MD) link with conflicting
  MSVC C runtime libraries, so the Rust extension cannot build for
  win_amd64 today. Tracked as a follow-up; not a blocker for Linux+macOS.
- headroom-e2e-setup: composite action now sets up Rust toolchain +
  Swatinem/rust-cache before `pip install -e .[proxy]`.
- eval.yml, publish.yml, rust.yml: same pattern — rust toolchain before
  install. rust.yml's wheels job builds from root pyproject.toml (no
  more `-m crates/headroom-py/Cargo.toml`).
- e2e/init/Dockerfile, e2e/wrap/Dockerfile: install rust + maturin in
  the build stage; copy `crates/` + workspace `Cargo.toml/lock` so the
  install can build the extension. Dropped `HEADROOM_REQUIRE_RUST_CORE=false`
  from wrap-e2e — the image now ships the full Rust core.
- Dockerfile (main): simplified — no more Layer 2/3 dance with
  `headroom-core-py` install + symlink. Single `uv pip install` builds
  + installs everything.
- .devcontainer/Dockerfile: rust toolchain + libssl-dev + maturin
  added so `uv sync` builds the extension inside the devcontainer.

## Lockfile + script

- uv.lock: regenerated. No `headroom-core-py` entries remain.
- scripts/build_rust_extension.sh: simplified from a symlink-into-tree
  workaround to a thin wrapper around `pip install -e .`. The maturin
  build-backend handles placement automatically.

## Local validation (all green on macOS aarch64)

1. Clean venv `pip install -e .` → `from headroom._core import …` works.
2. `maturin build --release` → 13.8 MB wheel, 336 files including
   `headroom/_core.cpython-311-darwin.so` (32 MB cdylib) and
   `headroom/dashboard/templates/dashboard.html`.
3. `pip install <wheel>` in fresh venv → import works.
4. Wheel contents verified via `unzip -l`.
5. `pytest tests/test_transforms/test_diff_compressor.py` — 29 passed.
6. `pytest tests/test_relevance.py` — 30 passed.
7. `cargo build --workspace` + `cargo test --workspace` — all green.
8. `make ci-precheck` — 176 Python tests + Rust + commitlint green.

## Migration notes

Users on `pip install headroom-ai` get the Rust core automatically
(linux + macos wheels). sdist installs require rust toolchain available
locally — pip will build via maturin.

Closes #355
Supersedes #357 (workarounds-based fix abandoned in favor of
architectural fix)
This commit is contained in:
chopratejas 2026-05-03 13:16:41 -07:00
parent 4bf559d5b5
commit 2a91cbb4b4
20 changed files with 4908 additions and 4872 deletions

View file

@ -14,15 +14,10 @@ ARG UV_VERSION
# build-essential / g++ for any C extension wheels uv may need to build
# from source. curl + ca-certificates are required by the rustup
# bootstrap below. Hotfix-A0 (Finding #2) added the rust toolchain so the
# image actually carries `headroom._core`; previously the runtime image
# shipped without the Rust extension and every compressed request fell
# back to a Python-only path or no-op.
#
# `pkg-config` + `libssl-dev` are required because the workspace
# transitively pulls `openssl-sys` (via reqwest/native-tls in some
# dependency chain). Without them, `cargo` fails the maturin build with
# "Could not find openssl via pkg-config" — observed in PR #350 CI.
# bootstrap below. pkg-config + libssl-dev for `openssl-sys` (transitive
# from fastembed/hf-hub/ureq → native-tls — the workspace `rustls-tls`
# pin loses to cargo feature unification). patchelf for maturin's
# wheel-link repair on linux.
RUN apt-get update && \
apt-get install -y --no-install-recommends \
build-essential \
@ -36,10 +31,10 @@ RUN apt-get update && \
RUN python -m pip install --no-cache-dir uv==${UV_VERSION}
# Rust toolchain for the headroom._core extension build. Pinned via
# rust-toolchain.toml at the repo root so this matches what local devs
# build with. Installed as root before WORKDIR change so the env
# additions stick for every subsequent RUN.
# Rust toolchain for the headroom._core extension. With single-wheel
# architecture (post-#355), `pip install -e .` invokes maturin via
# pyproject.toml's [build-system], which calls cargo. No more separate
# headroom-core-py package.
ENV CARGO_HOME=/usr/local/cargo \
RUSTUP_HOME=/usr/local/rustup \
PATH=/usr/local/cargo/bin:${PATH}
@ -48,73 +43,28 @@ RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs \
WORKDIR /build
# Layer 1: install deps only (cached unless pyproject.toml/uv.lock change)
# Copy the full set of files maturin needs to build the wheel: the root
# pyproject.toml + Cargo workspace + Rust crates + Python source. The
# uv install builds + installs the wheel in one shot.
COPY pyproject.toml uv.lock README.md ./
# Stub package so uv can resolve the local extras without full source
RUN mkdir -p headroom && touch headroom/__init__.py
ARG HEADROOM_EXTRAS=proxy,code
RUN --mount=type=cache,target=/root/.cache/uv \
uv pip install --system ".[${HEADROOM_EXTRAS}]"
# Layer 2 (Hotfix-A0): build and install the Rust extension wheel
# BEFORE installing headroom-ai source. Why this order:
#
# * The headroom-core-py wheel includes a stub `headroom/__init__.py`
# plus `headroom/_core.cpython-*.so` (maturin's `python-source`
# layout — see `crates/headroom-py/pyproject.toml`).
# * The headroom-ai install also writes files under `headroom/`.
# * If headroom-ai is installed FIRST and the wheel goes second with
# `--force-reinstall`, pip uninstalls the wheel's previously
# installed files, deleting `headroom/__init__.py` (which the wheel
# also claims). headroom-ai's __init__.py was already overwritten
# by the wheel's empty stub at install-time, so the deletion leaves
# no `__init__.py` at all — `from headroom._core import hello`
# then fails with `ModuleNotFoundError: No module named
# 'headroom._core'`. Observed in PR #350 CI before this reorder.
# * Installing the wheel FIRST means: wheel lays down stub
# `__init__.py` + `_core.so`. Then headroom-ai install OVERWRITES
# `__init__.py` with the real one and adds the rest of the
# `headroom/` tree. `_core.so` survives because headroom-ai
# doesn't claim ownership of it.
#
# uv already installed `maturin` as a transitive of the [proxy]/[code]
# extras; if it didn't, install it explicitly here so the build never
# silently skips.
COPY crates/ crates/
COPY Cargo.toml Cargo.lock rust-toolchain.toml ./
COPY crates/ crates/
COPY headroom/ headroom/
ARG HEADROOM_EXTRAS=proxy,code
RUN --mount=type=cache,target=/root/.cache/uv \
--mount=type=cache,target=/root/.cargo/registry \
--mount=type=cache,target=/build/target \
uv pip install --system maturin \
&& maturin build --release -m crates/headroom-py/Cargo.toml --out /build/wheels \
&& uv pip install --system --no-deps /build/wheels/headroom_core_py-*.whl
uv pip install --system ".[${HEADROOM_EXTRAS}]"
# Layer 3: copy real source, install headroom-ai (no deps). This
# overwrites the wheel's stub `headroom/__init__.py` with the real one
# and adds the full `headroom/` tree alongside the surviving
# `_core.so` from Layer 2.
COPY headroom/ headroom/
RUN --mount=type=cache,target=/root/.cache/uv \
uv pip install --system --no-deps --reinstall-package headroom-ai .
# Layer 4 (Hotfix-A0): verify the extension actually loads end-to-end
# inside the build image. If this fails, the runtime image would fail
# its lifespan smoke test on every restart — better to break the build
# loudly here than ship a broken image.
#
# IMPORTANT: run from `/tmp`, not from `/build`. `WORKDIR /build` puts
# `''` (cwd) at the front of `sys.path`, which makes `import headroom`
# resolve to `/build/headroom/` (the source tree we just COPY'd in)
# instead of `/usr/local/lib/python3.11/site-packages/headroom/` (where
# the wheel installed `_core.so`). The source tree has no `_core.so`,
# so the verify falsely fails. Production startup runs from a different
# cwd (the proxy's working directory or `/`), so this is a build-time-
# only quirk caused by `WORKDIR /build`. Anchoring the verify in `/tmp`
# matches the production import order: site-packages wins.
RUN cd /tmp && python -c "from headroom._core import hello; \
marker = hello(); \
assert marker == 'headroom-core', f'expected headroom-core, got {marker!r}'; \
print(f'build-stage rust core verify OK: {marker}')"
# Build-stage smoke check: verify the extension loads end-to-end inside
# the build image before we copy site-packages into the runtime image.
# If this fails, the runtime image would fail Phase A0's fail-loud
# startup check on every restart. Run from /tmp so cwd doesn't shadow
# site-packages with /build/headroom/ (which has no _core.so since
# maturin installed the .so into site-packages).
RUN cd /tmp && python -c "from headroom._core import DiffCompressor, SmartCrusher; \
print(f'build-stage rust core verify OK: {DiffCompressor.__name__}, {SmartCrusher.__name__}')"
# ---- Runtime stage (python-slim): supports root/nonroot via build arg ----
FROM python:${PYTHON_VERSION}-slim@${PYTHON_DIGEST} AS runtime-slim-base