# Root cause of the wheel-build cascade
We have shipped 5 release-pipeline hot-fixes in 12 hours, each
addressing a different symptom of the same architectural problem:
1. PR #363 — npm artifact downloads + tried `yum openssl-devel`
2. PR #367 — vendored OpenSSL in `headroom-proxy` + dropped Intel mac
3. PR #369 — Debian-cross perl install (`perl` not `libipc-cmd-perl`)
4. PR #370 — moved `openssl/vendored` from headroom-proxy to headroom-py
5. (this PR) — ELIMINATE OpenSSL entirely
Each fix exposed a different missing system package or feature flag in
a different build surface (manylinux x86_64 vs aarch64-cross-Debian vs
macOS Intel vs e2e/wrap Dockerfile vs e2e/init Dockerfile vs main
Dockerfile vs devcontainer). We were playing whack-a-mole because every
Cargo dep change to the OpenSSL surface required matching system-package
updates in 6+ different Dockerfiles and workflows, and the PR-level CI
didn't exercise all of them.
# Why this PR is the structural fix
`fastembed` exposes clean rustls feature flags:
- `hf-hub-rustls-tls` (replaces default `hf-hub-native-tls`)
- `ort-download-binaries-rustls-tls` (replaces default `…native-tls`)
By disabling fastembed's default features and enabling the rustls
variants explicitly, we remove `native-tls` (and therefore `openssl-sys`,
`openssl`, `openssl-src`, perl modules, OpenSSL build-time deps,
vendored OpenSSL ~30s build cost) from the entire workspace dep tree.
Verified locally:
$ cargo tree -p headroom-py -i openssl-sys
error: package ID specification `openssl-sys` did not match any packages
$ cargo tree -p headroom-py -i native-tls
error: package ID specification `native-tls` did not match any packages
$ cargo build --release -p headroom-py
Finished `release` profile [optimized] target(s) in 25.57s
(Down from 1m+ with vendored OpenSSL.)
# Cleanups enabled by this change
- crates/headroom-py/Cargo.toml — dropped the `openssl/vendored`
workaround from PR #370.
- crates/headroom-proxy/Cargo.toml — same dep removed.
- e2e/wrap/Dockerfile — dropped `yum install openssl-devel pkgconfig
perl-IPC-Cmd`. Comment retained explaining why.
- e2e/init/Dockerfile — same.
- Dockerfile (main) — dropped `pkg-config libssl-dev` from apt-get.
- .devcontainer/Dockerfile — dropped `pkg-config libssl-dev`.
- .github/workflows/release.yml — removed the entire before-script-linux
block (perl install probe + multi-package-manager dispatch + fail-loud
assertion). No longer needed.
# Regression gate
Three new structural tests in tests/test_release_workflows.py:
- test_no_openssl_sys_in_wheel_build_tree — runs `cargo tree -p <crate>
-i openssl-sys` for headroom-py / headroom-proxy / headroom-core. If
openssl-sys reappears (a future native-tls enabler creeping in via a
new dep), this fails AT PR TIME with an actionable message.
- test_no_native_tls_in_wheel_build_tree — same shape, native-tls is
the proximate cause.
- test_fastembed_uses_rustls_features — checks the Cargo.toml so a
future "let me bump fastembed and forget the features" doesn't
silently re-introduce OpenSSL.
Plus two cleanup gates:
- test_dockerfiles_no_longer_install_openssl_devel
- test_release_yml_does_not_install_openssl_or_perl_for_wheels
All 13 release-workflow tests pass. `make ci-precheck` PASSED.
# What this teaches us about rollouts (per user's ultrathink ask)
The 5-fix cascade exposed three meta-problems:
1. PR checks don't block merges. PR #370 had docker-init-e2e,
docker-wrap-e2e, docker-native-e2e all FAILED yet got merged.
Branch protection should require these checks. Operator action
needed (cannot fix in code).
2. Local validation is misleading. `cargo build -p headroom-py` from
the workspace root used the workspace lockfile and looked green;
CI did fresh resolution against headroom-py's manifest alone where
the feature wasn't enabled. Lesson: verify structural invariants
with `cargo tree -e features` before trusting that a build "works."
3. 6+ build surfaces with independent system-dep state. Every Cargo
change required matching updates in 6 places. The structural answer
(this PR) is to NOT depend on system OpenSSL at all. Where structural
fixes are not possible, the answer is a single shared
scripts/install-rust-build-deps.sh — but with this PR there's
nothing left to install.
`seq_claude_local` e2e assertion in e2e/init/run.py expects
`claude plugin marketplace add /workspace` but the actual command
was `claude plugin marketplace add chopratejas/headroom`.
Root cause: `_marketplace_source()` in headroom/cli/init.py walks
`Path(__file__).resolve().parents[2]` to find `.claude-plugin/
marketplace.json`. Before the single-wheel refactor, that path was
`/workspace/headroom/cli/init.py` -> parents[2] = `/workspace`,
where `.claude-plugin/marketplace.json` exists (COPY'd into the
e2e image). After the refactor, `headroom` is installed from a
wheel into site-packages, so `__file__` is now under
`/opt/headroom-venv/.../site-packages/headroom/cli/init.py` ->
parents[2] is the site-packages dir, which has no plugin manifest.
The function then falls back to the remote `chopratejas/headroom`.
Fix: set `HEADROOM_MARKETPLACE_SOURCE=/workspace` in the e2e/init
runtime ENV. The function honors this override before doing the
filesystem walk. The local `.claude-plugin/marketplace.json` is
already COPY'd into `/workspace/.claude-plugin/`, so the override
points at a valid source.
PR #360's previous attempt (multi-stage manylinux_2_28 build) still
failed with the same `__isoc23_strtoll` undefined-symbol ImportError.
Local repro showed the wheel built inside manylinux_2_28 has THREE
glibc 2.38+ C23 symbol references (`__isoc23_strtol`, `__isoc23_strtoll`,
`__isoc23_strtoull`) embedded by one of our transitive C/C++ deps
during cc-rs compilation — most likely libstdc++'s `<cstdlib>` resolving
`std::strtoll` to the C23 variant when the manylinux toolchain has
newer-glibc-aware headers. We can't easily fix the source of that
emission downstream.
Path of least resistance: switch the e2e runtime stage from a
glibc-2.36 base to one with glibc 2.38+. Verified on Mac (linux/arm64
native): the same wheel that fails on `node:22-bookworm` (glibc 2.36)
imports cleanly on `python:3.11-slim` (now trixie, glibc 2.41).
## Changes
- e2e/init/Dockerfile: stage 2 base `node:22-trixie` →
`python:3.11-slim`. The init harness only needs Python; no Node 22.
Drops apt-get install of python3/python3-pip/python3-venv (already in
the base image) and the `ln -sf` python alias.
- e2e/wrap/Dockerfile: stage 2 base `node:22-bookworm` →
`python:3.11-slim`. The wrap harness needs both Python 3.11
(aider-chat==0.86.2 requires Python <3.12) AND Node 22 (codex,
openclaw). Trixie's default python3 is 3.13 — too new for aider —
so we build on top of `python:3.11-slim` (trixie + py 3.11) and
install Node 22 from NodeSource.
- Both: stage 1 `--interpreter` reverted from python3.13 to python3.11
to match the runtime.
## Verification (local, linux/arm64)
docker buildx build -f e2e/wrap/Dockerfile.aarch64-test \
--platform linux/arm64 -t headroom-wrap-test .
→ stage 1 manylinux build green
→ stage 2 `from headroom._core import DiffCompressor` → OK
→ stage 2 aider-chat install in progress (separate venv)
## Production-side note (out of scope for this PR)
`pip install headroom-ai` from PyPI on a glibc-2.36 host (e.g. Debian
12, Ubuntu 22.04) will hit the same ImportError once the wheel matrix
publishes. python:3.X-slim is now trixie (glibc 2.41) for ALL of
3.10/3.11/3.12/3.13, so users on those base images are unaffected.
Tracking the underlying cc-rs symbol-emission bug as a separate issue.
## Two distinct failures on PR #360
### docker-init-e2e + docker-wrap-e2e + docker-native-e2e
Building headroom-ai from source inside `node:22-bookworm` produced a
`_core.so` that referenced `__isoc23_strtoll` (a glibc 2.38+ symbol).
The same image's runtime libc.so.6 (whatever it actually ships) can't
resolve it at import time:
ImportError: /workspace/headroom/_core.cpython-311-x86_64-linux-gnu.so:
undefined symbol: __isoc23_strtoll
Most likely cause: cc-rs invoking the bookworm gcc against headers that
have C23 wrappers exposed (libc6-dev backport, gcc 13 default mode, or
something similar), generating object code that references a symbol the
runtime libc.so doesn't actually have.
Fix: multi-stage docker build. Stage 1 builds the wheel inside
`quay.io/pypa/manylinux_2_28_x86_64` (AlmaLinux 8, glibc 2.28 baseline).
Stage 2 (node:22-bookworm) just installs the prebuilt wheel — no rust
toolchain needed at runtime, no build inside the runtime image. Same
pattern release.yml already uses for cross-platform wheel matrix.
Removed `COPY headroom/` and `COPY pyproject.toml` from the runtime
stage to prevent the source-only `headroom/` from shadowing the
installed wheel via cwd (Python would import the .py-only package and
miss `_core.so`).
### test (3.10/3.11/3.12/3.13)
The release-workflows test asserts the literal `needs:` list of the
create-release job. The single-wheel maturin refactor added
`build-wheels` and `collect-dist` jobs between `build` and the publish
jobs; create-release now waits for those too. Updated the assertion +
added explicit checks for the new `needs.<job>.result == 'success'`
guards.
rust-toolchain.toml at the repo root requests
`components = ["rustfmt", "clippy"]`. When `pip install -e .` invokes
maturin → cargo from inside `/workspace`, rustup auto-detects the
toolchain file and tries to add the missing components on top of the
`--profile minimal` install we did earlier. The install fails with:
info: downloading component clippy
info: rolling back changes
error: failed to install component: 'rustfmt-preview-x86_64-unknown-linux-gnu',
detected conflict: 'bin/cargo-fmt'
— rustup's auto-component install hits a `bin/cargo-fmt` conflict
inside the toolchain it just installed. The fix is to install the
required components up-front via `-c rustfmt -c clippy`, so the
toolchain matches what rust-toolchain.toml expects on first cargo run
and rustup never needs to mutate it.
Applied to: Dockerfile (main), e2e/init/Dockerfile, e2e/wrap/Dockerfile,
.devcontainer/Dockerfile. Also pinned the main Dockerfile's toolchain
from `stable` to `1.95.0` so all four images now match the lockfile
(prevents drift if rust-toolchain.toml is bumped later).
Eliminates the dual-package architecture that was the root cause of #355.
`pip install headroom-ai` now produces ONE wheel containing both the Python
source (headroom/*.py) and the compiled Rust extension (headroom/_core.so).
No more separate `headroom-core-py` package, no more chicken-and-egg with
PyPI publication, no more wheelhouse / PIP_FIND_LINKS / composite-action
plumbing in CI.
This is the canonical pattern used by cryptography, polars, ruff,
pydantic-core, and other Rust-as-core Python packages. Honors the
"Rust as core engine" direction.
## What changed
- pyproject.toml: `[build-system]` swapped from hatchling to maturin.
`[tool.hatch.*]` deleted; `[tool.maturin]` added pointing at
`crates/headroom-py/Cargo.toml` for the cdylib. `python-source = "."`
picks up the root `headroom/` package directly (dashboard HTML
templates and other non-Python files included automatically).
- crates/headroom-py/pyproject.toml: deleted. The crate is no longer a
separate published package; its Cargo.toml stays as the cdylib build
target invoked via `[tool.maturin] manifest-path`.
- crates/headroom-py/python/: deleted (placeholder layout for the old
separate package).
## CI updates
- ci.yml: `test` / `test-extras` / `test-agno` jobs simplified — Rust
toolchain set up before `pip install -e .` (which now invokes maturin
via build-system). Removed the "build wheel + symlink .so" dance.
`build` job swapped from `python -m build` (hatch) to
`maturin build` + `maturin sdist`.
- release.yml: collapsed dual-package matrix into one. New `build-wheels`
matrix produces cross-platform wheels for cp310/11/12/13 ×
{linux x86_64, linux aarch64, macos x86_64, macos aarch64}. New
`collect-dist` aggregator merges artifacts. publish-pypi consumes the
merged dist.
- init-native-e2e.yml: dropped windows-latest from the matrix —
upstream `esaxx-rs` (/MT) and `ort-sys` (/MD) link with conflicting
MSVC C runtime libraries, so the Rust extension cannot build for
win_amd64 today. Tracked as a follow-up; not a blocker for Linux+macOS.
- headroom-e2e-setup: composite action now sets up Rust toolchain +
Swatinem/rust-cache before `pip install -e .[proxy]`.
- eval.yml, publish.yml, rust.yml: same pattern — rust toolchain before
install. rust.yml's wheels job builds from root pyproject.toml (no
more `-m crates/headroom-py/Cargo.toml`).
- e2e/init/Dockerfile, e2e/wrap/Dockerfile: install rust + maturin in
the build stage; copy `crates/` + workspace `Cargo.toml/lock` so the
install can build the extension. Dropped `HEADROOM_REQUIRE_RUST_CORE=false`
from wrap-e2e — the image now ships the full Rust core.
- Dockerfile (main): simplified — no more Layer 2/3 dance with
`headroom-core-py` install + symlink. Single `uv pip install` builds
+ installs everything.
- .devcontainer/Dockerfile: rust toolchain + libssl-dev + maturin
added so `uv sync` builds the extension inside the devcontainer.
## Lockfile + script
- uv.lock: regenerated. No `headroom-core-py` entries remain.
- scripts/build_rust_extension.sh: simplified from a symlink-into-tree
workaround to a thin wrapper around `pip install -e .`. The maturin
build-backend handles placement automatically.
## Local validation (all green on macOS aarch64)
1. Clean venv `pip install -e .` → `from headroom._core import …` works.
2. `maturin build --release` → 13.8 MB wheel, 336 files including
`headroom/_core.cpython-311-darwin.so` (32 MB cdylib) and
`headroom/dashboard/templates/dashboard.html`.
3. `pip install <wheel>` in fresh venv → import works.
4. Wheel contents verified via `unzip -l`.
5. `pytest tests/test_transforms/test_diff_compressor.py` — 29 passed.
6. `pytest tests/test_relevance.py` — 30 passed.
7. `cargo build --workspace` + `cargo test --workspace` — all green.
8. `make ci-precheck` — 176 Python tests + Rust + commitlint green.
## Migration notes
Users on `pip install headroom-ai` get the Rust core automatically
(linux + macos wheels). sdist installs require rust toolchain available
locally — pip will build via maturin.
Closes#355
Supersedes #357 (workarounds-based fix abandoned in favor of
architectural fix)
Port e2e/init/run.py onto the shared harness and extend coverage so
issue #245 (bare ``headroom init -g`` with no agents) is locked in:
* ``seq_claude_local`` / ``seq_copilot_global`` / ``seq_codex_local`` —
the original scenario, now expressed as a sequence of Cases sharing
one scratch so the manifest-merge behavior (claude + codex targets)
is still exercised end-to-end
* ``bare_init_g_no_shims`` — regression guard for issue #245: asserts
the new guided error mentions every probed target and the concrete
``headroom init -g <agent>`` example
* ``bare_init_g_with_all_shims`` — complementary happy path with all
four shims present; asserts all three configurable agents report
``Configured ... (user scope)`` on stdout
* ``init_g_{claude,codex,copilot}_explicit`` — one case per
subcommand, each with only its own shim on PATH, asserting exit 0
and the correct per-agent settings file is written
* ``init_g_openclaw_missing`` — negative path for openclaw when its
binary isn't installed (delegates to ``headroom wrap openclaw`` which
can't be shimmed cheaply)
* ``init_verbose_no_shims`` — smoke test for ``headroom init -v``
ensuring ``detect_init_targets``, ``global_scope=True``, and every
agent name appear on stderr
Dockerfile is updated to COPY e2e/__init__.py and e2e/_lib/ so the
harness is importable inside the container. A new e2e/__init__.py
marks the tree as a package.
One small harness fix rides along: ``_resolve_headroom_bin`` captures
the absolute path to headroom before ``with_clean_path`` narrows PATH.
This is required for any case run inside a venv-scoped image - the
real ``headroom`` lives outside the shim dir and would otherwise be
hidden by the scrubbed PATH. Same bug would have bitten every future
command suite, so the fix belongs in the harness rather than run.py.
Verified locally inside the Docker image: all 10 cases pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
