## Description
Fix `pip install headroom-ai` on Intel Mac (`x86_64-apple-darwin`).
Source installs failed because `ort-sys 2.0.0-rc.12` (transitive via
`fastembed`) does not ship prebuilt ONNX Runtime binaries for that
target, causing maturin/cargo to exit during the wheel build.
This PR mirrors the existing Windows fix: build the Rust core with
`ort-load-dynamic`, pin `ORT_DYLIB_PATH` to the pip `onnxruntime` native
library at import time, and publish Intel macOS wheels from CI.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `crates/headroom-core/Cargo.toml`: use `ort-load-dynamic` for
`x86_64-apple-darwin` instead of `ort-download-binaries-rustls-tls`.
- `headroom/_ort.py`: extend the ORT dylib pin hook to Intel macOS
(`darwin` + `x86_64`), resolving `libonnxruntime*.dylib` from the pip
`onnxruntime` package.
- `.github/workflows/release.yml` and `.github/workflows/rust.yml`: add
`macos-15-intel` / `x86_64-apple-darwin` wheel matrix entries.
- `tests/test_release_workflows.py` and
`tests/test_transforms/test_ort_dylib.py`: update/add coverage for the
new target and dylib pin behavior.
- `README.md`: note that prebuilt wheels are published for Intel macOS.
## Testing
- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_transforms/test_ort_dylib.py \
tests/test_release_workflows.py::test_fastembed_uses_dynamic_ort_on_windows \
tests/test_release_workflows.py::test_build_wheels_matrix_includes_intel_macos_with_dynamic_ort -q
.......................... [100%]
10 passed in 0.18s
$ maturin build --release -o /tmp/headroom-dist
📦 Built wheel for abi3 Python ≥ 3.10 to /tmp/headroom-dist/headroom_ai-0.27.0-cp310-abi3-macosx_10_12_x86_64.whl
$ python3.11 -m venv /tmp/hr-venv && /tmp/hr-venv/bin/pip install .
Successfully built headroom-ai
Successfully installed headroom-ai-0.27.0
$ cd /tmp && /tmp/hr-venv/bin/python -c "import headroom; import headroom._core; print('ok')"
version 0.27.0
_core ok
```
## Real Behavior Proof
- Environment: macOS `x86_64-apple-darwin`, Python 3.11.5, Rust 1.95.0
- Exact command / steps: Reproduced the reported failure with `pip
install headroom-ai` (sdist build dies in `ort-sys` for
`x86_64-apple-darwin`); after this patch ran `maturin build --release`,
then `pip install .` in a clean venv, then `python -c "import
headroom._core"`.
- Observed result: Before fix, cargo/maturin exit 101 on missing ORT
prebuilts; after fix, wheel build succeeds and `headroom._core` imports
cleanly (`version 0.27.0`, `_core ok`).
- Not tested: `macos-15-intel` GitHub Actions wheel matrix row (will be
validated by CI after merge).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
- ML features (magika detection, fastembed embeddings) still require
`onnxruntime` at runtime on Intel Mac. Users should install
`headroom-ai[proxy]` or `pip install onnxruntime`; `_ort.py` auto-pins
`ORT_DYLIB_PATH` when that package is present.
- Apple Silicon (`aarch64-apple-darwin`) behavior is unchanged: it
continues to bundle ORT via `ort-download-binaries-rustls-tls`.
- Lint/mypy not re-run locally in this pass; targeted pytest +
maturin/pip install proof covers the changed surface.
---------
Co-authored-by: Bor <you@example.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
## Description
Supply-chain hardening: takes the **shipped** dependency surface from
**26 known CVEs to 0**. `pip install headroom-ai[all]` now resolves with
no known vulnerabilities (verified with Anchore syft + grype). Also
publishes a checked-in SBOM package (`sbom/`) so any user — especially
pilots running their own security review — can verify what's inside and
that we track it.
This addresses the Dependabot alerts on `main` (9 high / 4 moderate / 7
low at time of writing).
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
**Rust**
- `pyo3` 0.24 → 0.29 (GHSA-36hh-v3qg-5jq4 High, GHSA-chgr-c6px-7xpp
Med). Migrated `Python::allow_threads` → `Python::detach` (10 sites) and
added `from_py_object` to the `Clone`-deriving `#[pyclass]` types (both
required by the 0.25+ API).
- `pyo3-log` 0.12 → 0.13; `lru` 0.12 → 0.18 (GHSA-rhfx-m35p-ff5j).
**Python**
- `torch` → 2.12.1, `mem0ai` → 2.x.
- Floor-pinned transitive CVE deps via `[tool.uv]
constraint-dependencies`: `pygments>=2.20.0`,
`pydantic-settings>=2.14.2`, `gitpython>=3.1.50`, `langsmith>=0.9.0`.
- **Removed `benchmark` from the `[all]` aggregate** so the default
install is CVE-free. `lm-eval` is invoked as an external subprocess
(`python -m lm_eval`) and never imported, so it is not a true runtime
dep — it remains available via the opt-in `[benchmark]` extra. See
[Accepted Risks](#additional-notes).
**npm (build/test tooling — never shipped in the
wheel/container/published SDK)**
- `esbuild` override `>=0.28.1` in `sdk/typescript` + `plugins/openclaw`
(GHSA-g7r4-m6w7-qqqr).
- `docs/`: `@anthropic-ai/sdk` → `^0.106.0` (GHSA-p7fg-763f-g4gf),
`postcss` override to force Next.js's bundled copy ≥8.5.10
(GHSA-qx2v-qp2m-jg93); regenerated a stale `bun.lock` that carried a
**Critical** vitest/vite.
**CI**
- Pinned `pypa/gh-action-pypi-publish` `@release/v1` → `@v1.13.0`
(GHSA-vxmw-7h4f-hqxh) in `release.yml` + `publish.yml`.
**SBOM**
- New `sbom/` directory: CycloneDX 1.7 + SPDX 2.3 SBOMs, grype scan
evidence, 330-package license inventory, and a regeneration guide.
## Testing
- [ ] Unit tests pass (`pytest`) — N/A, no Python source changed
(deps/config only)
- [x] Linting passes — `cargo fmt --check` + `cargo clippy` clean on the
changed crate; 0 `.py` files changed so `ruff`/`mypy` scope is
unaffected
- [x] Type checking passes — `cargo check --workspace` (0 errors)
- [ ] New tests added — N/A (dependency bumps; covered by existing
suites)
- [x] Manual testing performed — see Real Behavior Proof
### Test Output
```text
# headroom-ai[all] product surface — the number that matters
$ grype sbom:sbom/headroom-sbom-all-extra.cdx.json
No vulnerabilities found
# full repo scan (universal lock incl. opt-in [benchmark] + dev)
$ grype sbom:sbom/headroom-sbom.cdx.json
NAME INSTALLED TYPE VULNERABILITY SEVERITY
sqlitedict 2.1.0 python GHSA-g4r7-86gm-pgqc High # [benchmark]-only, unpatchable, accepted
nltk 3.9.4 python GHSA-p4gq-832x-fm9v High # [benchmark]-only, unpatchable, accepted
# pyo3 0.29 migration — extension builds + imports + runs
$ cargo check --workspace
Finished `dev` profile [unoptimized + debuginfo] target(s)
$ maturin develop && python -c "from headroom._core import DiffCompressor, SmartCrusher; ..."
extension OK — detach + from_py_object paths exercised
# lru 0.18 — eviction path
$ cargo test -p headroom-proxy --lib drift
14 passed, 213 filtered out
# per-ecosystem npm audits
$ (cd sdk/typescript && npm audit) -> found 0 vulnerabilities
$ (cd plugins/openclaw && npm audit) -> found 0 vulnerabilities
$ (cd docs && npm audit && bun audit) -> found 0 vulnerabilities / No vulnerabilities found
```
## Real Behavior Proof
- Environment: macOS (darwin 25.4.0, arm64), Python 3.12 `.venv`, Rust
1.95 toolchain, syft 1.46.0, grype 0.115.0, bun 1.3.14, maturin 1.13.3.
- Exact command / steps: (1) `uv export --extra all --no-dev
--no-emit-project | syft → grype` for the product surface; (2) `cargo
check --workspace` + `maturin develop` + extension import/compress smoke
test; (3) `cargo test -p headroom-proxy --lib drift`; (4) `cargo fmt
--check` + `cargo clippy -p headroom-py`; (5) `npm audit` in
sdk/openclaw/docs + `bun audit` in docs.
- Observed result: `headroom-ai[all]` resolution scans clean — "No
vulnerabilities found" (179 pkgs); full/prod SBOM shows only the 2
documented accepted CVEs; pyo3 0.29 extension imports and runs (detach +
from_py_object paths exercised); drift tests 14/14 pass; cargo fmt +
clippy clean; all npm/bun audits report 0.
- Not tested: full `pytest` suite (no Python source changed);
release-profile wheel build (used dev-profile `maturin develop` for the
import proof — the extension is semantically identical).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
(`sbom/README.md`)
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective — N/A
(dependency bumps; existing suites + scans cover it)
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md — N/A (Release Please
auto-generates from the conventional commit)
## Additional Notes
**Accepted risks (the 2 residual CVEs).** Both originate solely from the
EleutherAI `lm-evaluation-harness` under the **opt-in `[benchmark]`
extra**, which Headroom invokes as a subprocess (never imports):
- `sqlitedict` CVE-2024-35515 (High) — pickle deserialization; package
abandoned (last release 2021), **no upstream fix exists**.
- `nltk` CVE-2026-54293 (High) — path traversal in `nltk.data.load()`;
affects ≤3.9.4 (current latest), **no patched release**.
Neither is in `[all]`, the published wheel, or the container. They are
documented in `sbom/README.md` and will be picked up automatically once
upstream ships fixes.
**Release/CHANGELOG:** N/A items above are because this is a
dependency/security PR with no Python source changes; CHANGELOG is
Release-Please-managed via the conventional commit message.
## Description
Add a clear dashboard view for per-agent token usage so end users can
see Cursor, Claude, Codex, and other detected clients with before/after
token counts, tokens saved, and savings percentages. The stats API now
exposes a stable `agent_usage` object that the dashboard renders near
the top of the session view.
Fixes #
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
### New Files
**Tests:**
- `tests/test_dashboard_agent_usage.py` — Covers agent classification,
exact per-request aggregation, and aggregate fallback behavior.
### Modified Files
- `headroom/proxy/server.py` — Adds per-agent usage aggregation to
`/stats` with before tokens, after tokens, output tokens, saved tokens,
savings percentage, source, providers, and models.
- `headroom/dashboard/templates/dashboard.html` — Adds a prominent Agent
Usage panel with totals, coverage status, per-agent token-flow bars,
request counts, before/after tokens, saved tokens, and share of savings.
## Testing
- [x] Unit tests pass: `.venv312/bin/pytest
tests/test_dashboard_agent_usage.py`
- [x] Linting passes: `.venv312/bin/ruff check headroom/proxy/server.py
tests/test_dashboard_agent_usage.py`
- [x] Diff whitespace check passes: `git diff --check
origin/main...HEAD`
- [x] Dashboard smoke render: local proxy on `127.0.0.1:8790`, captured
Chrome headless screenshot of `/dashboard`
- [x] New tests added for new functionality
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing relevant unit tests pass locally with my changes
- [ ] I have made corresponding changes to the documentation
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
The agent usage panel uses exact request-log data when available. If
detailed request logs are empty, it falls back to aggregate
provider/model request counts and labels the coverage as aggregate
fallback so users are not misled.
* ci: speed up GitHub Actions - path filters, caching, timeouts, version upgrades
Performance improvements:
- init-e2e.yml, wrap-e2e.yml: add path filters so e2e Docker builds only run when
e2e-related files change (saves ~10 min per irrelevant PR push)
- init-e2e.yml, wrap-e2e.yml: add concurrency groups to cancel superseded PR runs
- ci.yml: add pip caching to lint and build jobs
- ci.yml: cache actionlint + act binaries in workflow-validation (skip curl on hits)
- eval.yml: add pip caching to smoke-test and weekly-suite jobs
- docs.yml: add pip caching for mkdocs-material install
- rust.yml: replace cargo install --locked cargo-audit/deny with taiki-e/install-action
(prebuilt binaries; saves 2-5 min per audit run)
Bug fixes:
- docker.yml: fix actions/checkout@v6 -> @v4 (v6 does not exist; would break all
Docker builds on every release/PR touching docker paths)
Version upgrades:
- wagoid/commitlint-github-action: @v5 -> @v6
- devcontainers.yml: docker/setup-buildx-action@v3 -> @v4 (align with docker.yml)
Safety improvements:
- ci.yml: add timeout-minutes to all 13 jobs (changes, lint, build-wheel,
prefetch-model, test x4, test-extras, test-agno, commitlint, build,
workflow-validation, docker-native-e2e, windows-native-wrapper, macos-native-wrapper)
- docker.yml: add timeout-minutes to docker-build (75m), docker-manifest (20m),
promote-latest (10m)
- eval.yml: add timeout-minutes to smoke-test (30m); bump weekly-suite 60->90m
- rust.yml: add timeout-minutes to test (30m), wheels (45m), audit (20m)
Observed wall-clock impact on recent PRs:
- Init E2E and Wrap E2E were running on every single PR push regardless of content
- CI workflow was taking 12-17 min; path filters reduce unnecessary e2e runs to 0
* fix(ci): bust actionlint+act cache when workflow file changes
Static cache key 'ci-tools-actionlint-act-v1' never invalidated on
tool version updates. Switched to hashFiles('.github/workflows/ci.yml')
so the cache busts automatically whenever the download scripts are
updated to point at a newer release.
Flagged by adversarial review (Architecture + Testing/Reliability personas).
* fix(ci): add missing Dockerfile COPY paths to e2e path filters
e2e/init/Dockerfile and e2e/wrap/Dockerfile COPY files not covered
by the initial path filter set:
init-e2e: Cargo.toml, Cargo.lock, rust-toolchain.toml, uv.lock,
.claude-plugin, .github/plugin/**, plugins/headroom-agent-hooks/**
wrap-e2e: Cargo.toml, Cargo.lock, rust-toolchain.toml, uv.lock,
sdk/typescript/**, plugins/openclaw/**
Without these, a Rust toolchain bump or SDK change on a PR would
skip the e2e gate entirely, only catching it on the merge to main.
Flagged by adversarial review (Domain/Correctness persona).
* fix(devcontainer): upgrade uv to >=0.7.0 to parse uv.lock revision=3
* fix(devcontainer): set UV_SKIP_WHEEL_FILENAME_CHECK=1 in post-create.sh for gitpython wheel
* ci: bump actions/checkout and actions/setup-node to v5 (Node.js 20 EOL Jun 16)
* fix(devcontainer): export UV_SKIP_WHEEL_FILENAME_CHECK so uv run also skips wheel check
* ci: bump all GitHub Actions to latest versions (Node.js 24)
* fix(test): accept release-please-action v4 or v5 in workflow assertion
* fix(format): ruff format test_release_workflows.py
release-please's default for manifest configs is
`include-component-in-tag: true`, which prepends the package name
to every tag: `headroom-ai-v0.22.4`. The repo's existing tags are
plain `vX.Y.Z` (v0.22.0, v0.22.1, v0.22.2, v0.22.3 — all created
by the prior release_version.py + create-release flow).
Without this override, the first release-please run on main
opened PR #496 trying to ship `headroom-ai-v0.23.0` and computed
its changelog from genesis because it couldn't find any tag
matching its expected prefix.
Set `include-component-in-tag: false` so the bot uses `vX.Y.Z`
and finds the existing baseline (v0.22.3) cleanly. Add a
regression test pinning this setting.
`tomllib` is a Python 3.11+ stdlib module. The project supports
3.10+ (per pyproject.toml `requires-python = ">=3.10"`) and the
test (3.10) matrix job correctly caught this:
ModuleNotFoundError: No module named 'tomllib'
at tests/test_release_workflows.py:1036
Apply the same try/except fallback pattern used in
headroom/release_version.py: tomllib on 3.11+, tomli as the
backport on 3.10.
Replace "every push to main = release" with release-please's
release-PR pattern: the bot watches main and maintains a single
"chore: release vX.Y.Z" PR aggregating conventional commits; merging
that PR creates the tag + GitHub Release, which fires the
release:published event that release.yml now triggers on.
Why
---
Per-merge releases burned PyPI's 10 GiB per-project storage quota
(one fresh wheel matrix ~= 200 MB per merged fix/feat PR).
publish-pypi has failed on every main merge since PR #482 with
"400 Project size too large". Consolidating many fixes into one
release cuts upload frequency ~5x.
What changed
------------
- .github/workflows/release-please.yml: bot watching main
- .release-please-config.json: python release-type + extra-files
for sdk/typescript and plugins/openclaw package.json
- .release-please-manifest.json: tracks current 0.9.1
- .github/workflows/release.yml:
* trigger: push to main -> release: published
* detect-version: reads tag from github.event.release.tag_name
(strips leading "v") so release_version.py does not re-bump
past the bot's tag
* create-release: when release already exists (typical
release-please path), do not pass --notes-file -- that would
clobber the bot's auto-generated changelog body
Tests
-----
Five new regression tests in test_release_workflows.py prevent
silent reversion to per-push triggering and assert the bot
workflow + config invariants.
Note
----
This commit does NOT fix the existing quota breach. Request a
PyPI quota increase, yank old releases, or shrink the wheel
matrix to free immediate space. This PR ensures the future
release cadence stops growing the problem.
The structural-invariant test in test_release_workflows.py pinned
literal strings from the previous single-LICENSE shape:
- 'include = [{ path = "LICENSE", format = "sdist" }]' (single line)
- 'name: Verify sdist includes top-level LICENSE'
- 'license_path = f"{root}/LICENSE"' (verifier line)
The previous commit broke all three intentionally — sdist include is
now multi-line and lists NOTICE alongside LICENSE; the workflow step
was renamed to "Verify sdist license-file metadata matches tarball
contents"; the verifier no longer hardcodes a single license_path,
it parses every License-File entry from PKG-INFO.
Update the test to:
- Pin substrings (not full lines) for both LICENSE and NOTICE
entries so future formatting changes won't break the test
- Pin the new step name
- Pin two distinctive verifier signatures (the License-File parse
line and the missing-files error message) so a refactor that
silently drops the cross-check fails loudly
- Add a docstring explaining the regression history so a future
"let me clean up this test" pass can't accidentally re-loosen
the invariant
The X1 smoke-import gate (PR #387) catches runtime symbol mismatches
on the wheel before publish, but only at release time. Recent break
patterns were upstream of that:
- #379 (docker bake `name=` regression in PR #376)
- #382 (sdist `os: ubuntu-latest` → `ubuntu-24.04` rename)
- #384 / #385 / #386 (glibc shim alias / link-order iterations)
- #387's own heredoc-indent regression that broke main on the FIRST
release run after merge — the heredoc was inside `bash -ec '...'`,
no PR-time check exercised it
X2 adds a `pull_request:` trigger to release.yml with a NARROW path
filter so the dry-run runs at PR time for changes that affect wheel
layout / release pipeline, but skips for source-only PRs to
`crates/headroom-core` / `crates/headroom-proxy`.
Path filter: release.yml, docker.yml, crates/headroom-py/**,
pyproject.toml, root Cargo.toml, Cargo.lock.
publish-pypi / publish-npm / publish-github-packages / publish-docker
/ create-release all gate on `github.event_name != 'pull_request'`,
so a PR run never publishes — the dry-run is build + collect-dist +
smoke-import only.
concurrency rules:
- PR runs: namespaced by PR number (`pr-N`), cancel-in-progress=true.
- main runs: namespaced by ref_name, cancel-in-progress=false (a
tag-push release that's mid-flight must not be cancelled).
Test pin covers all four invariants (trigger, path filter, publish
gates, concurrency split). 21 tests in test_release_workflows.py,
all green locally.
Issue #355 plus the three follow-on hotfixes (#384/#385/#386) all
share a pattern: the wheel is technically valid (clippy passes,
tests pass, auditwheel is happy, the static-symbol audit added in
#384 is happy) but FAILS at runtime on a customer's box because of
a dynamic-link symbol mismatch. None of our pre-publish gates
actually `import headroom._core` on a representative customer
environment. They only build it.
What X1 adds
------------
A `smoke-import-wheels` job that runs after `build-wheels` and
before `publish-pypi` / `publish-docker` / `create-release`.
Matrix (6 jobs in parallel, ~3 min wall-clock):
- `manylinux_2_28_x86_64` + Python 3.11 (the floor we promise)
- `ubuntu:22.04` (glibc 2.35) + Python 3.12 (issue #355's env)
- `ubuntu:20.04` (glibc 2.31) + Python 3.10 (older LTS)
- `manylinux_2_28_aarch64` + Python 3.11 (aarch64 floor)
- `ubuntu:22.04` arm64 + Python 3.12 (aarch64 customer env)
- `macos-14` host + Python 3.13 (Apple Silicon)
Each job downloads its arch's wheel artifact, installs the wheel
matching its Python version inside the container, and runs the
exact command the proxy's `_check_rust_core` runs at startup:
from headroom._core import hello as _rust_hello
If any matrix entry fails, `publish-pypi` / `publish-docker` /
`create-release` are blocked. The matrix tells us exactly which
customer environment combination breaks.
Regression test in tests/test_release_workflows.py:
`test_release_workflow_has_smoke_import_wheel_gate` pins the job's
existence, the required matrix entries, and — critically — the
gating wires (publish-pypi / publish-docker / create-release all
need-and-require-success on the smoke job). A future "this slow
CI step always passes anyway, drop it" refactor fails at PR time.
Companion tests `test_glibc_compat_shim_present_in_headroom_py`
and `test_release_workflow_audits_wheel_glibc_symbols` (added in
#384) cover the static-symbol gate; this PR is the dynamic-link
gate. Both are needed.
Issue #355: published headroom_ai-*-manylinux_2_28_*.whl fails to import on Ubuntu 22.04, Debian 11/12, Conda envs with libc < 2.38: 'ImportError: undefined symbol: __isoc23_strtoll'.
Root cause: ORT prebuilt artifacts (downloaded via fastembed's ort-download-binaries-rustls-tls feature) are compiled with gcc-14.2.1 on a glibc-2.38+ host and reference __isoc23_strtoll. Our manylinux build host has glibc 2.38 so the link succeeds; end users with older glibc don't.
Two-part fix: (1) crates/headroom-py/glibc_compat.c provides weak-alias definitions for __isoc23_strtol/strtoll/strtoul/strtoull delegating to the older strtol* family, compiled by build.rs on Linux/glibc only. The dynamic linker prefers glibc's strong symbol when present (>= 2.38) and falls back to ours when not (< 2.38). (2) scripts/audit_wheel_glibc_symbols.py is a release.yml gate that runs objdump -T on every Linux wheel and rejects any UND symbol whose required glibc version exceeds the wheel's manylinux floor.
Validated: the audit correctly rejects the actually-broken v0.20.26 wheel with a precise diagnostic. The shim itself is a tiny static link with zero runtime cost.
Regression tests in tests/test_release_workflows.py pin the shim's load-bearing pieces (.c file, build.rs trigger, [build-dependencies] cc dep) and the audit invocation in release.yml. Future drift fails at PR time, not in the next release.
This is the same bug class as PR #371 (rustls-everywhere). Each instance gets fixed; the audit gate now catches the *class* — any future static linkage that introduces a post-floor symbol gets blocked before publish.
PR #376 pinned the wheel matrix's `os:` from `ubuntu-latest` to
`ubuntu-24.04` (explicit pinning, no semantic change). It silently
disabled the sdist build, whose conditional was
if: matrix.os == 'ubuntu-latest' && matrix.target == 'x86_64-unknown-linux-gnu'
The literal `'ubuntu-latest'` no longer matched, sdist never built,
`release-assets/*.tar.gz` was empty, and `create-release` failed at
gh release upload release-assets/*.tar.gz --clobber
with "no matches found". v0.20.22's release didn't get its sdist on
the GitHub Release.
Fix: key the conditional on `matrix.target` only. sdist is
platform-independent so any single matrix row is fine; using `target`
decouples the sdist build from any future host-runner rename.
Regression test in tests/test_release_workflows.py:
`test_sdist_build_conditional_keyed_on_target_not_os` pins the
target-only form. A future "let's add `os` back for clarity" refactor
will fail at PR time, not 8 minutes into a release.
This is a follow-up to PR #379 (docker bake `name=`) — same root
cause class: PR #376's matrix-shape changes silently broke a
downstream conditional whose literal value was tied to the OLD
matrix shape. Both fixes are now in place + pinned by tests.
PR #376's per-arch fan-out correctly removed `bake-file-tags` from
the docker-build step (tags belong on the multi-arch manifest, not
on per-arch images). But that left bake without ANY reference for
the push target — no tags AND no explicit `name=` in the output
spec. Every release-time docker-build job failed with the
misleading message:
ERROR: tag is needed when pushing to registry
Buildx's actual constraint is "no tags AND no `name=` in the
output = no push target." Since push-by-digest discards tags
anyway, the fix is to specify `name=<registry>/<image>` directly
in the `*.output` spec. Tags are still applied later, only on the
multi-arch manifest by `docker-manifest`.
Regression test in tests/test_release_workflows.py:
`test_docker_per_arch_build_specifies_image_name_in_output`
pins the `name=` substring so a future "the labels block already
has the registry, surely buildx can figure it out" refactor will
fail at PR time rather than 2 minutes into release.
GitHub-hosted Linux arm64 runners (`ubuntu-24.04-arm`) went GA in Aug
2025 and are free for public repositories. Switching the aarch64
wheel + the multi-arch docker matrix off `ubuntu-latest`+QEMU onto
the native runner cuts wall-clock on both surfaces.
release.yml — build-wheels matrix
* `aarch64-unknown-linux-gnu`: `ubuntu-latest` → `ubuntu-24.04-arm`.
maturin-action still runs inside `quay.io/pypa/manylinux_2_28_aarch64`,
but the container now executes natively on an aarch64 kernel
instead of through QEMU emulation. Aarch64 wheel build drops
from ~50–60 min to ~10 min.
* `x86_64-unknown-linux-gnu`: `ubuntu-latest` → `ubuntu-24.04`
(pin the moving alias for reproducibility; no semantic change).
docker.yml — fan-out + manifest merge
* Pre-#377: one `docker-variant-tags` matrix job per variant on
`ubuntu-latest`, using bake's `platforms = [amd64, arm64]` with
QEMU for the arm64 leg. ~1h per variant, 8 variants.
* Post-#377: split into `docker-build` (variant × arch = 16
parallel jobs, each on its native runner, single-platform
push-by-digest) and `docker-manifest` (per variant, merges the
two arch digests into a multi-arch tagged manifest with
`docker buildx imagetools create`, signs the index manifest
with cosign). Wall-clock drops from ~1h per variant to ~10 min.
* `docker/setup-qemu-action` removed — there's no QEMU left.
* Per-(variant, arch) GHA cache scopes so the two arches don't
collide on cache keys.
* `promote-latest` rewired to depend on `docker-manifest`.
Behavior change: cosign now signs only the multi-arch index digest
per variant, not each per-platform image. `cosign verify <repo>:tag`
(the typical flow) is unchanged because cosign resolves the tag to
the index digest. Verifiers pinning a specific per-arch digest will
need to verify the index digest instead.
Regression tests in tests/test_release_workflows.py:
* `test_aarch64_wheel_uses_native_arm64_runner` — pins the
aarch64 row to `ubuntu-24.04-arm` (and the amd64 row to
`ubuntu-24.04`, not `-latest`), so a future "let me unify on
ubuntu-latest" refactor surfaces the QEMU regression at PR time.
* `test_docker_workflow_builds_on_native_arch_runners` — pins the
fan-out matrix's arch entries, asserts push-by-digest, asserts
`setup-qemu-action` is absent from non-comment lines, asserts
the manifest-merge job exists.
Verified:
* Both workflow files parse as valid YAML with the expected job
graph (`docker-build` → `docker-manifest` → `promote-latest`,
16 fan-out jobs, 8 manifest jobs).
* `docker buildx imagetools inspect <tag> --format '{{ json . }}'`
exposes the index digest at `.manifest.digest` (confirmed via
Docker's official reference).
* `ubuntu-24.04-arm` is the correct GitHub-hosted runner label
(GA 2025-08-07, free for public repos).
* `make ci-precheck-rust` and `make ci-precheck-python` both pass
locally; `tests/test_release_workflows.py` is 15/15 green
(13 existing + 2 new).
# Root cause of the wheel-build cascade
We have shipped 5 release-pipeline hot-fixes in 12 hours, each
addressing a different symptom of the same architectural problem:
1. PR #363 — npm artifact downloads + tried `yum openssl-devel`
2. PR #367 — vendored OpenSSL in `headroom-proxy` + dropped Intel mac
3. PR #369 — Debian-cross perl install (`perl` not `libipc-cmd-perl`)
4. PR #370 — moved `openssl/vendored` from headroom-proxy to headroom-py
5. (this PR) — ELIMINATE OpenSSL entirely
Each fix exposed a different missing system package or feature flag in
a different build surface (manylinux x86_64 vs aarch64-cross-Debian vs
macOS Intel vs e2e/wrap Dockerfile vs e2e/init Dockerfile vs main
Dockerfile vs devcontainer). We were playing whack-a-mole because every
Cargo dep change to the OpenSSL surface required matching system-package
updates in 6+ different Dockerfiles and workflows, and the PR-level CI
didn't exercise all of them.
# Why this PR is the structural fix
`fastembed` exposes clean rustls feature flags:
- `hf-hub-rustls-tls` (replaces default `hf-hub-native-tls`)
- `ort-download-binaries-rustls-tls` (replaces default `…native-tls`)
By disabling fastembed's default features and enabling the rustls
variants explicitly, we remove `native-tls` (and therefore `openssl-sys`,
`openssl`, `openssl-src`, perl modules, OpenSSL build-time deps,
vendored OpenSSL ~30s build cost) from the entire workspace dep tree.
Verified locally:
$ cargo tree -p headroom-py -i openssl-sys
error: package ID specification `openssl-sys` did not match any packages
$ cargo tree -p headroom-py -i native-tls
error: package ID specification `native-tls` did not match any packages
$ cargo build --release -p headroom-py
Finished `release` profile [optimized] target(s) in 25.57s
(Down from 1m+ with vendored OpenSSL.)
# Cleanups enabled by this change
- crates/headroom-py/Cargo.toml — dropped the `openssl/vendored`
workaround from PR #370.
- crates/headroom-proxy/Cargo.toml — same dep removed.
- e2e/wrap/Dockerfile — dropped `yum install openssl-devel pkgconfig
perl-IPC-Cmd`. Comment retained explaining why.
- e2e/init/Dockerfile — same.
- Dockerfile (main) — dropped `pkg-config libssl-dev` from apt-get.
- .devcontainer/Dockerfile — dropped `pkg-config libssl-dev`.
- .github/workflows/release.yml — removed the entire before-script-linux
block (perl install probe + multi-package-manager dispatch + fail-loud
assertion). No longer needed.
# Regression gate
Three new structural tests in tests/test_release_workflows.py:
- test_no_openssl_sys_in_wheel_build_tree — runs `cargo tree -p <crate>
-i openssl-sys` for headroom-py / headroom-proxy / headroom-core. If
openssl-sys reappears (a future native-tls enabler creeping in via a
new dep), this fails AT PR TIME with an actionable message.
- test_no_native_tls_in_wheel_build_tree — same shape, native-tls is
the proximate cause.
- test_fastembed_uses_rustls_features — checks the Cargo.toml so a
future "let me bump fastembed and forget the features" doesn't
silently re-introduce OpenSSL.
Plus two cleanup gates:
- test_dockerfiles_no_longer_install_openssl_devel
- test_release_yml_does_not_install_openssl_or_perl_for_wheels
All 13 release-workflow tests pass. `make ci-precheck` PASSED.
# What this teaches us about rollouts (per user's ultrathink ask)
The 5-fix cascade exposed three meta-problems:
1. PR checks don't block merges. PR #370 had docker-init-e2e,
docker-wrap-e2e, docker-native-e2e all FAILED yet got merged.
Branch protection should require these checks. Operator action
needed (cannot fix in code).
2. Local validation is misleading. `cargo build -p headroom-py` from
the workspace root used the workspace lockfile and looked green;
CI did fresh resolution against headroom-py's manifest alone where
the feature wasn't enabled. Lesson: verify structural invariants
with `cargo tree -e features` before trusting that a build "works."
3. 6+ build surfaces with independent system-dep state. Every Cargo
change required matching updates in 6 places. The structural answer
(this PR) is to NOT depend on system OpenSSL at all. Where structural
fixes are not possible, the answer is a single shared
scripts/install-rust-build-deps.sh — but with this PR there's
nothing left to install.
PR #367 added `openssl = { features = ["vendored"] }` to
`crates/headroom-proxy/Cargo.toml`, expecting Cargo's feature
unification to propagate the vendored feature throughout the
workspace. PR #369 unblocked aarch64 by fixing the perl install.
The next release run on `1323830f` (the #369 merge) showed aarch64
+ macOS wheels build successfully but the **x86_64 Linux wheel still
fails** with:
The system library `openssl` required by crate `openssl-sys`
was not found.
# Why my previous reasoning was wrong
Cargo's feature unification only applies to the resolution graph
that's actually built. `maturin build --manifest-path
crates/headroom-py/Cargo.toml` resolves headroom-py's deps only —
headroom-py does NOT depend on headroom-proxy, so the
`openssl/vendored` enable in headroom-proxy never propagates to the
wheel build. `cargo tree -p headroom-py -e features` confirms:
openssl-sys is built with `default` only, no `vendored`.
(I missed this when I tested locally because `cargo build -p
headroom-py` from the workspace root reads the workspace's Cargo.lock,
which was already populated with `openssl-src` from when I added
the dep to headroom-proxy. The CI job runs `cargo rustc` with a
fresh resolution against headroom-py's manifest, where the vendored
feature isn't enabled.)
# Fix
Move the `openssl = { version = "0.10", features = ["vendored"] }`
dep from `headroom-proxy/Cargo.toml` to `headroom-py/Cargo.toml`.
The wheel-producing crate now declares the feature directly.
`cargo tree -p headroom-py -e features` after this change:
openssl-sys feature "vendored"
└── openssl-sys feature "openssl-src"
└── openssl-src feature "default"
└── openssl-src v300.6.0+3.6.2
Local `cargo build --release -p headroom-py` confirms openssl-src
is now compiled (35s wheel build).
# Tests
`test_headroom_py_vendors_openssl` (renamed from
`test_headroom_proxy_vendors_openssl`) gates the dep on the right
crate. Docstring explains WHY it must live in headroom-py — so a
future refactor doesn't move it back to headroom-proxy and silently
break the wheel build.
All 11 release-workflow tests pass. `make ci-precheck` PASSED.
Previous wheel hot-fix (#367) introduced a NEW failure mode that the
F1-merge release run surfaced:
E: Unable to locate package libipc-cmd-perl
The aarch64-unknown-linux-gnu maturin-action target does NOT use the
AlmaLinux 8 manylinux_2_28 image — it uses a Debian/Ubuntu-based
cross-compile container. The previous hot-fix's apt branch installed
`libipc-cmd-perl` which is a deprecated alias and is no longer in the
default Debian/Ubuntu sources. The build failed before openssl-src
could even start.
# What the script actually needs to do
`IPC::Cmd` is a Perl core module since 5.10, so any working `perl`
install provides it. The fix:
1. **Probe first.** `perl -MIPC::Cmd -e 1` exits cleanly when the
module is already importable — skip the install entirely. Some
manylinux images already ship it; others don't.
2. **Cover every package manager.** dnf (modern RHEL family) → yum
(older RHEL) → apt-get (Debian/Ubuntu) → apk (Alpine/musllinux).
The maturin-action uses different containers per (target, manylinux)
combo and we don't get to pick.
3. **Use `perl` not `libipc-cmd-perl` on Debian.** The plain `perl`
meta-package pulls `perl-modules-*` which contains IPC::Cmd. Works
on every Debian/Ubuntu version we'll see; `libipc-cmd-perl` is
gone from default sources.
4. **Fail loud after install.** `perl -MIPC::Cmd -e 'print "loaded
OK"'` runs unconditionally at the end. If somehow the module is
STILL missing, we fail here — not 5 minutes later in the
openssl-src compile step where the error message is harder to
debug. Matches the project's "no silent fallback" rule.
# Tests
`test_build_wheels_installs_perl_ipc_cmd_for_vendored_openssl`
updated to gate the new shape:
- Asserts `perl -MIPC::Cmd -e 1` pre-probe is present
- Asserts dnf/yum/apt-get/apk branches all exist
- Asserts apt branch installs `perl` not `libipc-cmd-perl`
- Asserts `libipc-cmd-perl` does not appear on any non-comment line
- Asserts the final `perl -MIPC::Cmd` fail-loud assertion is present
All 11 release-workflow tests pass. `make ci-precheck` PASSED.
The previous hot-fix (#363) addressed npm artifact downloads and added
openssl-devel installs in the manylinux container, but the wheel build
still fails on three of four matrix entries with three distinct errors:
1. ubuntu-x86_64 with `manylinux: auto` resolved to manylinux2014
(CentOS 7 / OpenSSL 1.0.2k). `openssl-sys 0.9` requires OpenSSL
1.1.0+ — "different version of OpenSSL was found".
2. ubuntu-aarch64 cross-compiles via `aarch64-unknown-linux-gnu-gcc`
from an x86_64 manylinux container. The `yum install openssl-devel`
we added installs x86_64 headers; `/usr/aarch64-unknown-linux-gnu/
include/` has no OpenSSL — "openssl/opensslv.h: No such file or
directory".
3. macos-15-intel fails on `ort-sys` (transitive via the ML compression
backend), which has no prebuilt ONNX Runtime binaries for
`x86_64-apple-darwin`. Unrelated to OpenSSL; an upstream limitation.
Why the workspace pulls openssl-sys at all: `hf-hub` (transitive via
`fastembed`) hard-codes `native-tls` as a default feature. Cargo's
feature unification then enables openssl-sys for the whole workspace
despite our `reqwest`/`tokio-tungstenite`/`tokio-rustls` preferences.
# Fix 1: vendored OpenSSL
Add `openssl = { version = "0.10", features = ["vendored"] }` to
`crates/headroom-proxy/Cargo.toml`. The `vendored` feature compiles
OpenSSL from source as part of the cargo build — works on every target
uniformly. Local build verified: cargo now pulls
`openssl-src v300.6.0+3.6.2` and compiles it. ~30s extra one-time
build cost.
The `openssl/vendored` feature DEFEATS `OPENSSL_DIR`. We therefore
remove the previous hot-fix's "Install OpenSSL (macOS)" step that
exported `OPENSSL_DIR` — leaving it would silently regress to the
system-OpenSSL path that broke originally.
# Fix 2: pin manylinux floor to 2_28
Change x86_64-unknown-linux-gnu from `manylinux: auto` to
`manylinux: 2_28` (matching aarch64 + the e2e Dockerfiles). This
isn't strictly required with vendored OpenSSL — the floor is now
glibc 2.28 / AlmaLinux 8 which has modern toolchain — but it removes
the CentOS-7 surface entirely and matches our runtime container
target.
# Fix 3: drop x86_64-apple-darwin from the matrix
`ort-sys 2.0.0-rc.12` has no prebuilt ONNX Runtime binaries for that
target. Building ORT from source would add CMake + ~5 minutes per
build. Apple Silicon macOS (`aarch64-apple-darwin`) is fully covered;
Intel-mac users install from the platform-independent sdist this
matrix also produces.
Tracked as a follow-up: switch the ML backend to `ort-tract` or
upstream a request for x86_64 macOS prebuilts.
# before-script-linux: keep perl-IPC-Cmd, drop openssl-devel
OpenSSL's vendored `Configure` script needs `IPC::Cmd` (without it
the build fails with "Can't locate IPC/Cmd.pm"). System
openssl-devel is no longer needed.
# Tests
4 new regression tests gate this:
- `test_headroom_proxy_vendors_openssl`
- `test_build_wheels_installs_perl_ipc_cmd_for_vendored_openssl`
- `test_build_wheels_does_not_set_openssl_dir`
- `test_build_wheels_matrix_excludes_intel_macos`
Plus the previous 7. All 11 release-workflow tests pass.
`make ci-precheck` PASSED. Local `cargo build --release -p headroom-py`
green.
Three independent failures on the post-merge release run for PR #360,
all introduced by the single-wheel maturin refactor:
1. Linux wheel matrix (`ubuntu-x86_64`, `aarch64-unknown-linux-gnu`)
failed inside the manylinux container with:
Could not find openssl via pkg-config
The system library `openssl` required by crate `openssl-sys`
was not found.
`openssl-sys` is transitive via `fastembed` → `hf-hub` → `ureq`
→ `native-tls`. The e2e Dockerfiles (`e2e/wrap`, `e2e/init`) we
wrote for #360 install `openssl-devel` upfront, but the
`build-wheels` matrix uses `PyO3/maturin-action@v1` which spins
up its OWN manylinux container that does not inherit those
installs. Fix: add a `before-script-linux:` to the action with a
yum/apt-get conditional so it works on RHEL-family (manylinux2014,
manylinux_2_28) and Debian-family musllinux variants.
2. macOS x86_64 wheel build failed with `maturin` exit 1 from the
same `openssl-sys` lookup. The aarch64 macos-14 runner happens
to have `/opt/homebrew/opt/openssl@3` on `openssl-sys`'s default
discovery path; the Intel macos-15-intel runner uses
`/usr/local/Cellar` which is NOT on that path. Fix: add a
pre-maturin step that runs `brew install openssl@3` and exports
`OPENSSL_DIR` / `OPENSSL_LIB_DIR` / `OPENSSL_INCLUDE_DIR` /
`PKG_CONFIG_PATH`. The aarch64 runner happily picks up the
explicit env vars too — no regression.
3. `publish-npm` and `publish-github-packages` both fail with
"Artifact not found for name: dist". Both jobs `npm pack` + `npm
publish` directly from the checked-out source tree — they never
consume the Python `dist` artifact. The `Download dist artifact`
step was vestigial dead code carried over from a prior workflow
shape; the only reason it didn't fail before #360 is that the
pre-refactor `build` job DID upload a `dist` artifact. Post-#360,
`dist` is produced by `collect-dist` and neither publish job is
gated on it (by design — npm vs PyPI ecosystems publish
independently). Fix: remove the dead download step from both
jobs. Loose coupling is preserved; `create-release` still gates
the GitHub Release tag on all of build / build-wheels /
collect-dist / publish-* succeeding.
Why the PR-level CI didn't catch any of this: `release.yml` only
runs on `push: branches: [main]`. PR #360's CI exercised `ci.yml`
which has a separate `ci-build-wheels-on-pr` matrix that uses a
different setup. The release surface only fires post-merge.
Tests added (regression gates):
- `test_build_wheels_installs_openssl_devel_on_linux_via_before_script`
- `test_build_wheels_resolves_openssl_dir_explicitly_on_macos`
- `test_npm_publish_jobs_do_not_download_dist_artifact`
All 9 release-workflow tests pass. `make ci-precheck` PASSED.
## Two distinct failures on PR #360
### docker-init-e2e + docker-wrap-e2e + docker-native-e2e
Building headroom-ai from source inside `node:22-bookworm` produced a
`_core.so` that referenced `__isoc23_strtoll` (a glibc 2.38+ symbol).
The same image's runtime libc.so.6 (whatever it actually ships) can't
resolve it at import time:
ImportError: /workspace/headroom/_core.cpython-311-x86_64-linux-gnu.so:
undefined symbol: __isoc23_strtoll
Most likely cause: cc-rs invoking the bookworm gcc against headers that
have C23 wrappers exposed (libc6-dev backport, gcc 13 default mode, or
something similar), generating object code that references a symbol the
runtime libc.so doesn't actually have.
Fix: multi-stage docker build. Stage 1 builds the wheel inside
`quay.io/pypa/manylinux_2_28_x86_64` (AlmaLinux 8, glibc 2.28 baseline).
Stage 2 (node:22-bookworm) just installs the prebuilt wheel — no rust
toolchain needed at runtime, no build inside the runtime image. Same
pattern release.yml already uses for cross-platform wheel matrix.
Removed `COPY headroom/` and `COPY pyproject.toml` from the runtime
stage to prevent the source-only `headroom/` from shadowing the
installed wheel via cwd (Python would import the .py-only package and
miss `_core.so`).
### test (3.10/3.11/3.12/3.13)
The release-workflows test asserts the literal `needs:` list of the
create-release job. The single-wheel maturin refactor added
`build-wheels` and `collect-dist` jobs between `build` and the publish
jobs; create-release now waits for those too. Updated the assertion +
added explicit checks for the new `needs.<job>.result == 'success'`
guards.
`.gitattributes` declares `*.py text eol=lf` and `*.sh text eol=lf`, but
74 files (73 .py, 1 .sh) are stored in the index with CRLF line endings,
violating that contract. Every macOS/Linux clone reports these files as
"modified" on fresh checkout because git's diff engine sees the stored
bytes don't match the attribute contract, even though the working tree
and index match byte-for-byte.
Running `git add --renormalize .` rewrites each affected blob so the
stored form matches the attribute declaration. No semantic changes —
every affected file's diff is "N insertions, N deletions" with inserts
and deletes being the same lines modulo line endings.
Follow-up commit adds `.git-blame-ignore-revs` so `git blame` / GitHub
blame skip this mechanical commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drive-by: main is currently failing `ruff format --check .` because of
two missing blank lines between two top-level functions in this file
(introduced in 8bf11d2). Fixing it here so this PR's CI can go green —
no other way to unblock the format check without landing a separate PR
first.
