mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
Eliminates the dual-package architecture that was the root cause of #355. `pip install headroom-ai` now produces ONE wheel containing both the Python source (headroom/*.py) and the compiled Rust extension (headroom/_core.so). No more separate `headroom-core-py` package, no more chicken-and-egg with PyPI publication, no more wheelhouse / PIP_FIND_LINKS / composite-action plumbing in CI. This is the canonical pattern used by cryptography, polars, ruff, pydantic-core, and other Rust-as-core Python packages. Honors the "Rust as core engine" direction. ## What changed - pyproject.toml: `[build-system]` swapped from hatchling to maturin. `[tool.hatch.*]` deleted; `[tool.maturin]` added pointing at `crates/headroom-py/Cargo.toml` for the cdylib. `python-source = "."` picks up the root `headroom/` package directly (dashboard HTML templates and other non-Python files included automatically). - crates/headroom-py/pyproject.toml: deleted. The crate is no longer a separate published package; its Cargo.toml stays as the cdylib build target invoked via `[tool.maturin] manifest-path`. - crates/headroom-py/python/: deleted (placeholder layout for the old separate package). ## CI updates - ci.yml: `test` / `test-extras` / `test-agno` jobs simplified — Rust toolchain set up before `pip install -e .` (which now invokes maturin via build-system). Removed the "build wheel + symlink .so" dance. `build` job swapped from `python -m build` (hatch) to `maturin build` + `maturin sdist`. - release.yml: collapsed dual-package matrix into one. New `build-wheels` matrix produces cross-platform wheels for cp310/11/12/13 × {linux x86_64, linux aarch64, macos x86_64, macos aarch64}. New `collect-dist` aggregator merges artifacts. publish-pypi consumes the merged dist. - init-native-e2e.yml: dropped windows-latest from the matrix — upstream `esaxx-rs` (/MT) and `ort-sys` (/MD) link with conflicting MSVC C runtime libraries, so the Rust extension cannot build for win_amd64 today. Tracked as a follow-up; not a blocker for Linux+macOS. - headroom-e2e-setup: composite action now sets up Rust toolchain + Swatinem/rust-cache before `pip install -e .[proxy]`. - eval.yml, publish.yml, rust.yml: same pattern — rust toolchain before install. rust.yml's wheels job builds from root pyproject.toml (no more `-m crates/headroom-py/Cargo.toml`). - e2e/init/Dockerfile, e2e/wrap/Dockerfile: install rust + maturin in the build stage; copy `crates/` + workspace `Cargo.toml/lock` so the install can build the extension. Dropped `HEADROOM_REQUIRE_RUST_CORE=false` from wrap-e2e — the image now ships the full Rust core. - Dockerfile (main): simplified — no more Layer 2/3 dance with `headroom-core-py` install + symlink. Single `uv pip install` builds + installs everything. - .devcontainer/Dockerfile: rust toolchain + libssl-dev + maturin added so `uv sync` builds the extension inside the devcontainer. ## Lockfile + script - uv.lock: regenerated. No `headroom-core-py` entries remain. - scripts/build_rust_extension.sh: simplified from a symlink-into-tree workaround to a thin wrapper around `pip install -e .`. The maturin build-backend handles placement automatically. ## Local validation (all green on macOS aarch64) 1. Clean venv `pip install -e .` → `from headroom._core import …` works. 2. `maturin build --release` → 13.8 MB wheel, 336 files including `headroom/_core.cpython-311-darwin.so` (32 MB cdylib) and `headroom/dashboard/templates/dashboard.html`. 3. `pip install <wheel>` in fresh venv → import works. 4. Wheel contents verified via `unzip -l`. 5. `pytest tests/test_transforms/test_diff_compressor.py` — 29 passed. 6. `pytest tests/test_relevance.py` — 30 passed. 7. `cargo build --workspace` + `cargo test --workspace` — all green. 8. `make ci-precheck` — 176 Python tests + Rust + commitlint green. ## Migration notes Users on `pip install headroom-ai` get the Rust core automatically (linux + macos wheels). sdist installs require rust toolchain available locally — pip will build via maturin. Closes #355 Supersedes #357 (workarounds-based fix abandoned in favor of architectural fix)
131 lines
4.8 KiB
Docker
131 lines
4.8 KiB
Docker
ARG PYTHON_VERSION=3.11
|
|
ARG UV_VERSION=0.6.17
|
|
# Pinned 2026-04-15. Update via Dependabot or: docker pull python:3.11-slim
|
|
ARG PYTHON_DIGEST=sha256:233de06753d30d120b1a3ce359d8d3be8bda78524cd8f520c99883bfe33964cf
|
|
# Pinned 2026-04-15. Update via Dependabot or: docker pull gcr.io/distroless/python3-debian13
|
|
ARG DISTROLESS_DIGEST=sha256:ed3a4beb46f8f8baac068743ba1b1f95ea3f793422129cf6dd23967f779b6018
|
|
ARG DISTROLESS_IMAGE=gcr.io/distroless/python3-debian13
|
|
ARG PYTHON_SITE_PACKAGES=/usr/local/lib/python${PYTHON_VERSION}/site-packages
|
|
|
|
# ---- Build stage: compile native extensions, build wheel ----
|
|
FROM python:${PYTHON_VERSION}-slim@${PYTHON_DIGEST} AS builder
|
|
|
|
ARG UV_VERSION
|
|
|
|
# build-essential / g++ for any C extension wheels uv may need to build
|
|
# from source. curl + ca-certificates are required by the rustup
|
|
# bootstrap below. pkg-config + libssl-dev for `openssl-sys` (transitive
|
|
# from fastembed/hf-hub/ureq → native-tls — the workspace `rustls-tls`
|
|
# pin loses to cargo feature unification). patchelf for maturin's
|
|
# wheel-link repair on linux.
|
|
RUN apt-get update && \
|
|
apt-get install -y --no-install-recommends \
|
|
build-essential \
|
|
g++ \
|
|
curl \
|
|
ca-certificates \
|
|
pkg-config \
|
|
libssl-dev \
|
|
patchelf \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
RUN python -m pip install --no-cache-dir uv==${UV_VERSION}
|
|
|
|
# Rust toolchain for the headroom._core extension. With single-wheel
|
|
# architecture (post-#355), `pip install -e .` invokes maturin via
|
|
# pyproject.toml's [build-system], which calls cargo. No more separate
|
|
# headroom-core-py package.
|
|
ENV CARGO_HOME=/usr/local/cargo \
|
|
RUSTUP_HOME=/usr/local/rustup \
|
|
PATH=/usr/local/cargo/bin:${PATH}
|
|
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs \
|
|
| sh -s -- -y --no-modify-path --profile minimal --default-toolchain stable
|
|
|
|
WORKDIR /build
|
|
|
|
# Copy the full set of files maturin needs to build the wheel: the root
|
|
# pyproject.toml + Cargo workspace + Rust crates + Python source. The
|
|
# uv install builds + installs the wheel in one shot.
|
|
COPY pyproject.toml uv.lock README.md ./
|
|
COPY Cargo.toml Cargo.lock rust-toolchain.toml ./
|
|
COPY crates/ crates/
|
|
COPY headroom/ headroom/
|
|
|
|
ARG HEADROOM_EXTRAS=proxy,code
|
|
RUN --mount=type=cache,target=/root/.cache/uv \
|
|
--mount=type=cache,target=/root/.cargo/registry \
|
|
--mount=type=cache,target=/build/target \
|
|
uv pip install --system ".[${HEADROOM_EXTRAS}]"
|
|
|
|
# Build-stage smoke check: verify the extension loads end-to-end inside
|
|
# the build image before we copy site-packages into the runtime image.
|
|
# If this fails, the runtime image would fail Phase A0's fail-loud
|
|
# startup check on every restart. Run from /tmp so cwd doesn't shadow
|
|
# site-packages with /build/headroom/ (which has no _core.so since
|
|
# maturin installed the .so into site-packages).
|
|
RUN cd /tmp && python -c "from headroom._core import DiffCompressor, SmartCrusher; \
|
|
print(f'build-stage rust core verify OK: {DiffCompressor.__name__}, {SmartCrusher.__name__}')"
|
|
|
|
# ---- Runtime stage (python-slim): supports root/nonroot via build arg ----
|
|
FROM python:${PYTHON_VERSION}-slim@${PYTHON_DIGEST} AS runtime-slim-base
|
|
|
|
ARG RUNTIME_USER=nonroot
|
|
ARG PYTHON_SITE_PACKAGES
|
|
|
|
RUN apt-get update && \
|
|
apt-get install -y --no-install-recommends curl && \
|
|
rm -rf /var/lib/apt/lists/*
|
|
|
|
COPY --from=builder ${PYTHON_SITE_PACKAGES} ${PYTHON_SITE_PACKAGES}
|
|
COPY --from=builder /usr/local/bin/headroom /usr/local/bin/headroom
|
|
|
|
RUN mkdir -p /home/nonroot /data && \
|
|
if [ "$RUNTIME_USER" = "nonroot" ]; then \
|
|
groupadd --gid 1000 nonroot && \
|
|
useradd --uid 1000 --gid nonroot --create-home nonroot && \
|
|
mkdir -p /home/nonroot/.headroom && \
|
|
chown -R nonroot:nonroot /data /home/nonroot; \
|
|
else \
|
|
mkdir -p /root/.headroom; \
|
|
fi
|
|
|
|
USER ${RUNTIME_USER}
|
|
WORKDIR /home/nonroot
|
|
|
|
ENV HEADROOM_HOST=0.0.0.0 \
|
|
PYTHONUNBUFFERED=1 \
|
|
PYTHONDONTWRITEBYTECODE=1
|
|
|
|
EXPOSE 8787
|
|
|
|
HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \
|
|
CMD ["curl", "--fail", "--silent", "http://127.0.0.1:8787/readyz"]
|
|
|
|
ENTRYPOINT ["headroom", "proxy"]
|
|
CMD ["--host", "0.0.0.0", "--port", "8787"]
|
|
|
|
FROM ${DISTROLESS_IMAGE}@${DISTROLESS_DIGEST} AS runtime-slim
|
|
|
|
ARG RUNTIME_USER=nonroot
|
|
ARG PYTHON_SITE_PACKAGES
|
|
|
|
COPY --from=builder ${PYTHON_SITE_PACKAGES} ${PYTHON_SITE_PACKAGES}
|
|
|
|
USER ${RUNTIME_USER}
|
|
WORKDIR /app
|
|
|
|
ENV HEADROOM_HOST=0.0.0.0 \
|
|
PYTHONUNBUFFERED=1 \
|
|
PYTHONDONTWRITEBYTECODE=1 \
|
|
PYTHONPATH=${PYTHON_SITE_PACKAGES}
|
|
|
|
EXPOSE 8787
|
|
|
|
HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \
|
|
CMD ["python3", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8787/readyz', timeout=5)"]
|
|
|
|
ENTRYPOINT ["python3", "-m", "headroom.cli", "proxy"]
|
|
CMD ["--host", "0.0.0.0", "--port", "8787"]
|
|
|
|
# Default published image remains python-slim runtime
|
|
FROM runtime-slim-base AS runtime
|