mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
Eliminates the dual-package architecture that was the root cause of #355. `pip install headroom-ai` now produces ONE wheel containing both the Python source (headroom/*.py) and the compiled Rust extension (headroom/_core.so). No more separate `headroom-core-py` package, no more chicken-and-egg with PyPI publication, no more wheelhouse / PIP_FIND_LINKS / composite-action plumbing in CI. This is the canonical pattern used by cryptography, polars, ruff, pydantic-core, and other Rust-as-core Python packages. Honors the "Rust as core engine" direction. ## What changed - pyproject.toml: `[build-system]` swapped from hatchling to maturin. `[tool.hatch.*]` deleted; `[tool.maturin]` added pointing at `crates/headroom-py/Cargo.toml` for the cdylib. `python-source = "."` picks up the root `headroom/` package directly (dashboard HTML templates and other non-Python files included automatically). - crates/headroom-py/pyproject.toml: deleted. The crate is no longer a separate published package; its Cargo.toml stays as the cdylib build target invoked via `[tool.maturin] manifest-path`. - crates/headroom-py/python/: deleted (placeholder layout for the old separate package). ## CI updates - ci.yml: `test` / `test-extras` / `test-agno` jobs simplified — Rust toolchain set up before `pip install -e .` (which now invokes maturin via build-system). Removed the "build wheel + symlink .so" dance. `build` job swapped from `python -m build` (hatch) to `maturin build` + `maturin sdist`. - release.yml: collapsed dual-package matrix into one. New `build-wheels` matrix produces cross-platform wheels for cp310/11/12/13 × {linux x86_64, linux aarch64, macos x86_64, macos aarch64}. New `collect-dist` aggregator merges artifacts. publish-pypi consumes the merged dist. - init-native-e2e.yml: dropped windows-latest from the matrix — upstream `esaxx-rs` (/MT) and `ort-sys` (/MD) link with conflicting MSVC C runtime libraries, so the Rust extension cannot build for win_amd64 today. Tracked as a follow-up; not a blocker for Linux+macOS. - headroom-e2e-setup: composite action now sets up Rust toolchain + Swatinem/rust-cache before `pip install -e .[proxy]`. - eval.yml, publish.yml, rust.yml: same pattern — rust toolchain before install. rust.yml's wheels job builds from root pyproject.toml (no more `-m crates/headroom-py/Cargo.toml`). - e2e/init/Dockerfile, e2e/wrap/Dockerfile: install rust + maturin in the build stage; copy `crates/` + workspace `Cargo.toml/lock` so the install can build the extension. Dropped `HEADROOM_REQUIRE_RUST_CORE=false` from wrap-e2e — the image now ships the full Rust core. - Dockerfile (main): simplified — no more Layer 2/3 dance with `headroom-core-py` install + symlink. Single `uv pip install` builds + installs everything. - .devcontainer/Dockerfile: rust toolchain + libssl-dev + maturin added so `uv sync` builds the extension inside the devcontainer. ## Lockfile + script - uv.lock: regenerated. No `headroom-core-py` entries remain. - scripts/build_rust_extension.sh: simplified from a symlink-into-tree workaround to a thin wrapper around `pip install -e .`. The maturin build-backend handles placement automatically. ## Local validation (all green on macOS aarch64) 1. Clean venv `pip install -e .` → `from headroom._core import …` works. 2. `maturin build --release` → 13.8 MB wheel, 336 files including `headroom/_core.cpython-311-darwin.so` (32 MB cdylib) and `headroom/dashboard/templates/dashboard.html`. 3. `pip install <wheel>` in fresh venv → import works. 4. Wheel contents verified via `unzip -l`. 5. `pytest tests/test_transforms/test_diff_compressor.py` — 29 passed. 6. `pytest tests/test_relevance.py` — 30 passed. 7. `cargo build --workspace` + `cargo test --workspace` — all green. 8. `make ci-precheck` — 176 Python tests + Rust + commitlint green. ## Migration notes Users on `pip install headroom-ai` get the Rust core automatically (linux + macos wheels). sdist installs require rust toolchain available locally — pip will build via maturin. Closes #355 Supersedes #357 (workarounds-based fix abandoned in favor of architectural fix)
149 lines
4.8 KiB
YAML
149 lines
4.8 KiB
YAML
name: rust
|
|
|
|
on:
|
|
push:
|
|
branches: [ main, rust-rewrite ]
|
|
paths:
|
|
- 'crates/**'
|
|
- 'Cargo.toml'
|
|
- 'Cargo.lock'
|
|
- 'rust-toolchain.toml'
|
|
- 'tests/parity/**'
|
|
- 'Makefile'
|
|
- '.github/workflows/rust.yml'
|
|
pull_request:
|
|
paths:
|
|
- 'crates/**'
|
|
- 'Cargo.toml'
|
|
- 'Cargo.lock'
|
|
- 'rust-toolchain.toml'
|
|
- 'tests/parity/**'
|
|
- 'Makefile'
|
|
- '.github/workflows/rust.yml'
|
|
schedule:
|
|
# Nightly parity run at 07:17 UTC (weekdays only). Phase 0 allows failure.
|
|
- cron: '17 7 * * 1-5'
|
|
|
|
concurrency:
|
|
group: rust-${{ github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
# Default permissions: read-only. Individual jobs override only what they need.
|
|
# Mitigates CodeQL/CWE-275 (missing-workflow-permissions): the GITHUB_TOKEN
|
|
# defaults to whatever the repo policy is, which can be read-write. Pinning
|
|
# this here means even if the repo default changes, this workflow stays safe.
|
|
permissions:
|
|
contents: read
|
|
|
|
jobs:
|
|
test:
|
|
name: test (ubuntu)
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- name: Install stable toolchain
|
|
# Pin action code to @stable (latest fixes), toolchain version
|
|
# via input. The @1.95.0 ref shipped action code that errors on
|
|
# ubuntu-latest with `detected conflict: 'bin/cargo-clippy'`
|
|
# because the pre-installed runner Rust collides with the
|
|
# clippy-preview component install.
|
|
uses: dtolnay/rust-toolchain@stable
|
|
with:
|
|
toolchain: 1.95.0
|
|
components: rustfmt, clippy
|
|
- name: Cache cargo registry + build
|
|
uses: Swatinem/rust-cache@v2
|
|
- name: cargo fmt --check
|
|
run: cargo fmt --all -- --check
|
|
- name: cargo clippy
|
|
run: cargo clippy --workspace -- -D warnings
|
|
- name: cargo test
|
|
run: cargo test --workspace
|
|
|
|
wheels:
|
|
name: wheels (${{ matrix.target }})
|
|
runs-on: ${{ matrix.os }}
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
include:
|
|
- os: ubuntu-latest
|
|
target: x86_64-unknown-linux-gnu
|
|
maturin-target: x86_64
|
|
- os: macos-14
|
|
target: aarch64-apple-darwin
|
|
maturin-target: aarch64-apple-darwin
|
|
# macOS x86_64 (Intel) is NOT in this matrix.
|
|
# `fastembed` → `ort` → `ort-sys` does not publish prebuilt ONNX
|
|
# Runtime binaries for `x86_64-apple-darwin`; building from source
|
|
# in CI is a multi-hour cmake job. Apple Silicon has been the
|
|
# default macOS target since 2020 and is sufficient for the wheels
|
|
# we ship. If a customer needs Intel macOS, build from source
|
|
# locally (the toolchain works; only prebuilt distribution skips
|
|
# this target).
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
python-version: '3.11'
|
|
- name: Build wheel (single-wheel architecture: builds headroom-ai)
|
|
uses: PyO3/maturin-action@v1
|
|
# Maturin reads `[tool.maturin]` from the root `pyproject.toml`
|
|
# which points at `crates/headroom-py/Cargo.toml` for the cdylib.
|
|
# Output is `headroom_ai-<ver>-<py>-<py>-<platform>.whl` containing
|
|
# both Python source and the compiled `headroom/_core.so`.
|
|
with:
|
|
command: build
|
|
args: --release --out dist
|
|
target: ${{ matrix.maturin-target }}
|
|
- name: Upload wheel artifact
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: wheels-${{ matrix.target }}
|
|
path: dist/*.whl
|
|
|
|
audit:
|
|
name: audit
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- uses: dtolnay/rust-toolchain@stable
|
|
with:
|
|
toolchain: 1.95.0
|
|
- uses: Swatinem/rust-cache@v2
|
|
- name: Install cargo-audit + cargo-deny
|
|
run: |
|
|
cargo install --locked cargo-audit || true
|
|
cargo install --locked cargo-deny || true
|
|
- name: cargo audit (soft-fail)
|
|
continue-on-error: true
|
|
run: cargo audit
|
|
- name: cargo deny check licenses
|
|
continue-on-error: true
|
|
run: cargo deny check licenses
|
|
|
|
parity-nightly:
|
|
name: parity (nightly, allowed to fail during Phase 0)
|
|
if: github.event_name == 'schedule'
|
|
runs-on: ubuntu-latest
|
|
continue-on-error: true
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- uses: dtolnay/rust-toolchain@stable
|
|
with:
|
|
toolchain: 1.95.0
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
python-version: '3.11'
|
|
- uses: Swatinem/rust-cache@v2
|
|
- name: Install deps
|
|
run: |
|
|
python -m venv .venv
|
|
source .venv/bin/activate
|
|
pip install --upgrade pip
|
|
pip install maturin
|
|
pip install -e .
|
|
- name: Run parity harness
|
|
run: |
|
|
source .venv/bin/activate
|
|
make test-parity
|