## Description
`headroom-opencode` is documented as an npm package, but the release
workflow never published it, so installs failed with a registry 404 even
though the plugin source already lived under `plugins/opencode`. This
wires the existing package into the npm release path, keeps its version
synced with root releases, and adds release guards for the new package.
Closes#76.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- added `headroom-opencode` to the npm release workflow, including
release-version stamping and `headroom-ai` dependency rewrite before
publish
- added `plugins/opencode/package.json` to release-please and local
version-sync guards
- synced the source opencode package version to the current release line
and documented the new npm package in the release docs
- added focused release workflow and version-sync tests for the opencode
package
- aligned the two failing dashboard Playwright tests with the current
Session/Lifetime split and `/stats-lifetime` fixture contract
## Testing
- [x] Unit tests pass (`uv run pytest scripts/tests/test_version_sync.py
-q`, `uv run pytest tests/test_release_workflows.py -q -k
'publish_npm_rewrites_opencode_dependency_after_version_and_before_publish
or opencode_source_dependency_matches_lockfile_registry_range or
release_please_manifest_config_consistency'`)
- [x] Unit tests pass (`uv run pytest
tests/test_dashboard_cache_lifetime_playwright.py
tests/test_dashboard_cache_ttl_playwright.py -q`)
- [x] Linting passes (`uv run ruff check scripts/verify-versions.py
scripts/version-sync.py scripts/tests/test_version_sync.py
tests/test_release_workflows.py`)
- [ ] Type checking passes (`uv run mypy headroom`)
- [x] New tests added for new functionality when applicable
- [x] Manual testing performed
### Test Output
```text
$ uv run pytest scripts/tests/test_version_sync.py -q
8 passed, 1 warning in 0.51s
$ uv run pytest tests/test_release_workflows.py -q -k 'publish_npm_rewrites_opencode_dependency_after_version_and_before_publish or opencode_source_dependency_matches_lockfile_registry_range or release_please_manifest_config_consistency'
2 passed, 38 deselected, 1 warning in 0.07s
$ uv run pytest tests/test_dashboard_cache_lifetime_playwright.py tests/test_dashboard_cache_ttl_playwright.py -q
4 passed, 1 warning in 4.04s
$ uv run ruff check scripts/verify-versions.py scripts/version-sync.py scripts/tests/test_version_sync.py tests/test_release_workflows.py
All checks passed!
$ npm ci && npm run build (plugins/opencode)
Build success; dist/index.js, dist/entry.opencode.js, and DTS outputs emitted
```
## Real Behavior Proof
- Environment: Windows, Python 3.11.15, Node v24.15.0, npm 11.16.0
- Exact command / steps: inspected `.github/workflows/release.yml`,
updated the npm publish path for `plugins/opencode`, aligned the two
failing dashboard Playwright tests with the current Session/Lifetime
split, then ran the focused pytest commands above plus `npm ci && npm
run build` in `plugins/opencode`
- Observed result: the release workflow now versions and publishes
`headroom-opencode`, release-please and version-sync track
`plugins/opencode/package.json`, the dashboard tests now fetch durable
cache and setup-url data from the Lifetime view, and the opencode
package still builds locally from source
- Not tested: GitHub Package Registry publish
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
`CHANGELOG.md` is unchanged because release-please owns changelog
generation here.
---------
Co-authored-by: JD Davis <mxjerrett@gmail.com>
Automated by Headroom + Kimi (Fireworks) in a Modal Sandbox.
Request:
Make ONLY this one change, nothing else: in
.github/workflows/release.yml, in the publish-pypi job's step that uses
pypa/gh-action-pypi-publish, add exactly one line 'attestations: false'
immediately after the 'skip-existing: true' line in its with: block
(same indentation). Do NOT edit any other file, do NOT investigate the
codebase, do NOT make any other change. Then finish.
Co-authored-by: Headroom Kimi <kimi@headroom.dev>
## Description
Harden the release artifact workflow so maintainers can reproduce npm
and Python release checks locally and so OpenClaw packaging does not
depend on a not-yet-published SDK version. This follow-up also keeps the
OpenClaw loader export contract explicit and updates the PR after the
branch was merged with current `headroomlabs/main`.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [x] Documentation update
## Changes Made
- Added reusable local release smoke scripts for npm assets, Python
wheel/sdist artifacts, and the combined release gate.
- Switched the release workflow npm asset build to the reusable npm
builder.
- Made OpenClaw release asset building install against the just-built
local SDK tarball before rewriting packed metadata to the release
dependency range.
- Kept the OpenClaw source dependency registry-installable at
`headroom-ai@^0.22.3` while the packed artifact still ships
`^<release-version>`.
- Added `registerHeadroomPlugin` as a named export while preserving the
default `{ register }` OpenClaw loader contract.
- Added Windows development bootstrap docs/script and regression tests
for version sync, npm asset ordering, OpenClaw source installability,
and Python wheel smoke import isolation.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
uv run --with pytest python -m pytest tests/test_release_workflows.py scripts/tests/test_version_sync.py -q
44 passed, 1 warning
node --check scripts/build_npm_release_assets.mjs
node --check scripts/verify_npm_release_assets.mjs
python -m py_compile scripts/build_python_release_smoke.py scripts/release_smoke_all.py scripts/version-sync.py
python scripts/verify-versions.py
All versions aligned at 0.31.0
npm ci (plugins/openclaw)
added 88 packages, audited 89 packages, found 0 vulnerabilities
python scripts/release_smoke_all.py --out release-assets-local/all-pr1824-postmerge-fixed-20260710-104739
Verified npm release assets for 0.31.0
wheel metadata OK: headroom_ai-0.31.0-cp310-abi3-win_amd64.whl contains headroom/_core.pyd
sdist License-File metadata OK: ['LICENSE', 'NOTICE']
smoke-import OK: version=0.31.0 hello=headroom-core
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.14.3, Node v24.14.0, npm 11.9.0, uv
0.11.15, Rust/Cargo already installed in the local development
environment.
- Exact command / steps: Ran `python scripts/release_smoke_all.py --out
release-assets-local/all-pr1824-postmerge-fixed-20260710-104739` after
syncing this branch with current `headroomlabs/main`.
- Observed result: The command built `headroom-ai-0.31.0.tgz`,
`headroom-openclaw-0.31.0.tgz`,
`headroom_ai-0.31.0-cp310-abi3-win_amd64.whl`, and
`headroom_ai-0.31.0.tar.gz`; npm verifier passed; the wheel installed
into a fresh venv and imported `headroom._core`.
- Not tested: Full GitHub Actions release matrix and publish jobs with
real PyPI/npm/GitHub Packages credentials.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
N/A.
## Additional Notes
The post-merge full smoke initially failed because the OpenClaw source
package depended on `headroom-ai@^0.31.0`, which is not available on
public npm yet. The builder now installs OpenClaw against the just-built
local SDK tarball before build/pack, and then rewrites packed metadata
to `^0.31.0`.
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
## Summary
- add a uv constraint floor for `setuptools>=83.0.0` to address
`PYSEC-2026-3447`
- refresh `uv.lock` so the production audit export resolves with
`setuptools 83.0.0`
- harden the Release wheel smoke-import gate by retrying
Ubuntu-container `apt-get` operations and using `--fix-missing`
- keep the generated `requirements-prod.txt` uncommitted; it is produced
by the security workflow
## Why
This clears the new dependency audit alert that made PRs red:
- `setuptools 80.10.2`
- `PYSEC-2026-3447`
- fixed in `83.0.0`
While validating the queue, the same PR class also hit a Release
smoke-import failure in the Ubuntu 22.04 ARM container due apt mirror
skew:
`E: Failed to fetch ... python3-httplib2_0.20.2-2ubuntu0.1_all.deb 404
Not Found`
The smoke gate should still fail for broken wheels, but transient apt
mirror skew should not make unrelated PRs red.
## Lockfile impact
- `setuptools 80.10.2 -> 83.0.0`
- `torch 2.12.1 -> 2.13.0`, required for pip resolver compatibility with
`setuptools 83.0.0` in the exported audit set
- `cuda-toolkit 13.0.2 -> 13.0.3.0`, pulled by the torch lock refresh
- uv also refreshed the existing project metadata for the sandbox extra
so `uv lock --check` passes
## Validation
- `uv lock --check`
- `uv export --frozen --no-dev --no-emit-project --no-hashes --extra all
--format requirements-txt > requirements-prod.txt`
- confirmed generated `requirements-prod.txt` contains
`setuptools==83.0.0`, `torch==2.13.0`, `cuda-toolkit==13.0.3.0`
- `uvx pip-audit -r requirements-prod.txt` -> No known vulnerabilities
found
- `python -m pytest tests/test_release_workflows.py -q` -> 32 passed
- `uvx ruff@0.15.17 check tests/test_release_workflows.py` -> All checks
passed
- `git diff --check`
## Description
Fix `pip install headroom-ai` on Intel Mac (`x86_64-apple-darwin`).
Source installs failed because `ort-sys 2.0.0-rc.12` (transitive via
`fastembed`) does not ship prebuilt ONNX Runtime binaries for that
target, causing maturin/cargo to exit during the wheel build.
This PR mirrors the existing Windows fix: build the Rust core with
`ort-load-dynamic`, pin `ORT_DYLIB_PATH` to the pip `onnxruntime` native
library at import time, and publish Intel macOS wheels from CI.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `crates/headroom-core/Cargo.toml`: use `ort-load-dynamic` for
`x86_64-apple-darwin` instead of `ort-download-binaries-rustls-tls`.
- `headroom/_ort.py`: extend the ORT dylib pin hook to Intel macOS
(`darwin` + `x86_64`), resolving `libonnxruntime*.dylib` from the pip
`onnxruntime` package.
- `.github/workflows/release.yml` and `.github/workflows/rust.yml`: add
`macos-15-intel` / `x86_64-apple-darwin` wheel matrix entries.
- `tests/test_release_workflows.py` and
`tests/test_transforms/test_ort_dylib.py`: update/add coverage for the
new target and dylib pin behavior.
- `README.md`: note that prebuilt wheels are published for Intel macOS.
## Testing
- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_transforms/test_ort_dylib.py \
tests/test_release_workflows.py::test_fastembed_uses_dynamic_ort_on_windows \
tests/test_release_workflows.py::test_build_wheels_matrix_includes_intel_macos_with_dynamic_ort -q
.......................... [100%]
10 passed in 0.18s
$ maturin build --release -o /tmp/headroom-dist
📦 Built wheel for abi3 Python ≥ 3.10 to /tmp/headroom-dist/headroom_ai-0.27.0-cp310-abi3-macosx_10_12_x86_64.whl
$ python3.11 -m venv /tmp/hr-venv && /tmp/hr-venv/bin/pip install .
Successfully built headroom-ai
Successfully installed headroom-ai-0.27.0
$ cd /tmp && /tmp/hr-venv/bin/python -c "import headroom; import headroom._core; print('ok')"
version 0.27.0
_core ok
```
## Real Behavior Proof
- Environment: macOS `x86_64-apple-darwin`, Python 3.11.5, Rust 1.95.0
- Exact command / steps: Reproduced the reported failure with `pip
install headroom-ai` (sdist build dies in `ort-sys` for
`x86_64-apple-darwin`); after this patch ran `maturin build --release`,
then `pip install .` in a clean venv, then `python -c "import
headroom._core"`.
- Observed result: Before fix, cargo/maturin exit 101 on missing ORT
prebuilts; after fix, wheel build succeeds and `headroom._core` imports
cleanly (`version 0.27.0`, `_core ok`).
- Not tested: `macos-15-intel` GitHub Actions wheel matrix row (will be
validated by CI after merge).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
- ML features (magika detection, fastembed embeddings) still require
`onnxruntime` at runtime on Intel Mac. Users should install
`headroom-ai[proxy]` or `pip install onnxruntime`; `_ort.py` auto-pins
`ORT_DYLIB_PATH` when that package is present.
- Apple Silicon (`aarch64-apple-darwin`) behavior is unchanged: it
continues to bundle ORT via `ort-download-binaries-rustls-tls`.
- Lint/mypy not re-run locally in this pass; targeted pytest +
maturin/pip install proof covers the changed surface.
---------
Co-authored-by: Bor <you@example.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
## Description
Supply-chain hardening: takes the **shipped** dependency surface from
**26 known CVEs to 0**. `pip install headroom-ai[all]` now resolves with
no known vulnerabilities (verified with Anchore syft + grype). Also
publishes a checked-in SBOM package (`sbom/`) so any user — especially
pilots running their own security review — can verify what's inside and
that we track it.
This addresses the Dependabot alerts on `main` (9 high / 4 moderate / 7
low at time of writing).
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
**Rust**
- `pyo3` 0.24 → 0.29 (GHSA-36hh-v3qg-5jq4 High, GHSA-chgr-c6px-7xpp
Med). Migrated `Python::allow_threads` → `Python::detach` (10 sites) and
added `from_py_object` to the `Clone`-deriving `#[pyclass]` types (both
required by the 0.25+ API).
- `pyo3-log` 0.12 → 0.13; `lru` 0.12 → 0.18 (GHSA-rhfx-m35p-ff5j).
**Python**
- `torch` → 2.12.1, `mem0ai` → 2.x.
- Floor-pinned transitive CVE deps via `[tool.uv]
constraint-dependencies`: `pygments>=2.20.0`,
`pydantic-settings>=2.14.2`, `gitpython>=3.1.50`, `langsmith>=0.9.0`.
- **Removed `benchmark` from the `[all]` aggregate** so the default
install is CVE-free. `lm-eval` is invoked as an external subprocess
(`python -m lm_eval`) and never imported, so it is not a true runtime
dep — it remains available via the opt-in `[benchmark]` extra. See
[Accepted Risks](#additional-notes).
**npm (build/test tooling — never shipped in the
wheel/container/published SDK)**
- `esbuild` override `>=0.28.1` in `sdk/typescript` + `plugins/openclaw`
(GHSA-g7r4-m6w7-qqqr).
- `docs/`: `@anthropic-ai/sdk` → `^0.106.0` (GHSA-p7fg-763f-g4gf),
`postcss` override to force Next.js's bundled copy ≥8.5.10
(GHSA-qx2v-qp2m-jg93); regenerated a stale `bun.lock` that carried a
**Critical** vitest/vite.
**CI**
- Pinned `pypa/gh-action-pypi-publish` `@release/v1` → `@v1.13.0`
(GHSA-vxmw-7h4f-hqxh) in `release.yml` + `publish.yml`.
**SBOM**
- New `sbom/` directory: CycloneDX 1.7 + SPDX 2.3 SBOMs, grype scan
evidence, 330-package license inventory, and a regeneration guide.
## Testing
- [ ] Unit tests pass (`pytest`) — N/A, no Python source changed
(deps/config only)
- [x] Linting passes — `cargo fmt --check` + `cargo clippy` clean on the
changed crate; 0 `.py` files changed so `ruff`/`mypy` scope is
unaffected
- [x] Type checking passes — `cargo check --workspace` (0 errors)
- [ ] New tests added — N/A (dependency bumps; covered by existing
suites)
- [x] Manual testing performed — see Real Behavior Proof
### Test Output
```text
# headroom-ai[all] product surface — the number that matters
$ grype sbom:sbom/headroom-sbom-all-extra.cdx.json
No vulnerabilities found
# full repo scan (universal lock incl. opt-in [benchmark] + dev)
$ grype sbom:sbom/headroom-sbom.cdx.json
NAME INSTALLED TYPE VULNERABILITY SEVERITY
sqlitedict 2.1.0 python GHSA-g4r7-86gm-pgqc High # [benchmark]-only, unpatchable, accepted
nltk 3.9.4 python GHSA-p4gq-832x-fm9v High # [benchmark]-only, unpatchable, accepted
# pyo3 0.29 migration — extension builds + imports + runs
$ cargo check --workspace
Finished `dev` profile [unoptimized + debuginfo] target(s)
$ maturin develop && python -c "from headroom._core import DiffCompressor, SmartCrusher; ..."
extension OK — detach + from_py_object paths exercised
# lru 0.18 — eviction path
$ cargo test -p headroom-proxy --lib drift
14 passed, 213 filtered out
# per-ecosystem npm audits
$ (cd sdk/typescript && npm audit) -> found 0 vulnerabilities
$ (cd plugins/openclaw && npm audit) -> found 0 vulnerabilities
$ (cd docs && npm audit && bun audit) -> found 0 vulnerabilities / No vulnerabilities found
```
## Real Behavior Proof
- Environment: macOS (darwin 25.4.0, arm64), Python 3.12 `.venv`, Rust
1.95 toolchain, syft 1.46.0, grype 0.115.0, bun 1.3.14, maturin 1.13.3.
- Exact command / steps: (1) `uv export --extra all --no-dev
--no-emit-project | syft → grype` for the product surface; (2) `cargo
check --workspace` + `maturin develop` + extension import/compress smoke
test; (3) `cargo test -p headroom-proxy --lib drift`; (4) `cargo fmt
--check` + `cargo clippy -p headroom-py`; (5) `npm audit` in
sdk/openclaw/docs + `bun audit` in docs.
- Observed result: `headroom-ai[all]` resolution scans clean — "No
vulnerabilities found" (179 pkgs); full/prod SBOM shows only the 2
documented accepted CVEs; pyo3 0.29 extension imports and runs (detach +
from_py_object paths exercised); drift tests 14/14 pass; cargo fmt +
clippy clean; all npm/bun audits report 0.
- Not tested: full `pytest` suite (no Python source changed);
release-profile wheel build (used dev-profile `maturin develop` for the
import proof — the extension is semantically identical).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
(`sbom/README.md`)
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective — N/A
(dependency bumps; existing suites + scans cover it)
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md — N/A (Release Please
auto-generates from the conventional commit)
## Additional Notes
**Accepted risks (the 2 residual CVEs).** Both originate solely from the
EleutherAI `lm-evaluation-harness` under the **opt-in `[benchmark]`
extra**, which Headroom invokes as a subprocess (never imports):
- `sqlitedict` CVE-2024-35515 (High) — pickle deserialization; package
abandoned (last release 2021), **no upstream fix exists**.
- `nltk` CVE-2026-54293 (High) — path traversal in `nltk.data.load()`;
affects ≤3.9.4 (current latest), **no patched release**.
Neither is in `[all]`, the published wheel, or the container. They are
documented in `sbom/README.md` and will be picked up automatically once
upstream ships fixes.
**Release/CHANGELOG:** N/A items above are because this is a
dependency/security PR with no Python source changes; CHANGELOG is
Release-Please-managed via the conventional commit message.
## Description
We ship wheels for macOS arm64 and manylinux x86_64/aarch64, but there's
no `win_amd64` wheel on PyPI for any Python version. So on Windows,
pip/uv can't find a binary and try to build from the sdist with maturin,
which pulls the Rust toolchain from static.rust-lang.org and crates from
crates.io. On locked-down machines (corporate proxies, CI runners, the
GitHub Copilot CLI sandbox, anything air-gapped) those hosts aren't
reachable and the install just dies:
```
error: could not download file from 'https://static.rust-lang.org/dist/channel-rust-stable.toml.sha256'
error: failed to get pyo3-macros as a dependency of package pyo3 v0.24.2
[28] Timeout was reached (Failed to connect to index.crates.io port 443)
```
This adds the Windows wheel to the release matrix so `pip install
headroom-ai` works on Windows without a local Rust install.
Closes#1328
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Added a `windows-latest` / `x86_64-pc-windows-msvc` row to the
`build-wheels` matrix. The runner already has MSVC and maturin-action
sets up Rust, so it produces `headroom_ai-*-win_amd64.whl` on every
release. I checked `crates/headroom-core/Cargo.toml` first — the Windows
ONNX path is already on `ort-load-dynamic` under `cfg(windows)`, so the
wheel loads ORT at runtime instead of linking the DirectML SDK libs.
Nothing else was needed on the Rust side.
- Added a matching `windows-latest` row to `smoke-import-wheels` so a
broken Windows wheel blocks publish like the other platforms do. Windows
needed its own step: the venv puts Python under `Scripts\` not `bin/`,
and the runner defaults to pwsh. I also pinned the shared script-staging
step to `shell: bash` since it uses a heredoc that pwsh can't run (Git
Bash is on the runner), and added a `setup-python` step to get the right
minor version.
- Updated the README install section so the "install Rust first"
workaround is clearly only for the sdist fallback (e.g. Intel macOS) now
that Windows/Linux/macOS-arm64 all have prebuilt wheels.
## Testing
- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
This is a CI workflow + docs change, no Python runtime code. I leaned on
the existing `tests/test_release_workflows.py` structural gates plus a
YAML parse and matrix-shape sanity check.
### Test Output
```text
$ python -m pytest tests/test_release_workflows.py -q
28 passed, 1 skipped, 1 failed
# The one failure, test_no_native_tls_in_wheel_build_tree, shells out to cargo, which
# isn't installed here. I confirmed with `git stash` that it fails the same way on main
# without my changes, so it's pre-existing and unrelated.
$ python -c "import yaml; d=yaml.safe_load(open('.github/workflows/release.yml',encoding='utf-8')); \
j=d['jobs']; print('build-wheels rows:', len(j['build-wheels']['strategy']['matrix']['include'])); \
print('smoke rows:', len(j['smoke-import-wheels']['strategy']['matrix']['include']))"
build-wheels rows: 4
smoke rows: 6
```
## Real Behavior Proof
- Environment: Windows 11 local clone; CI runs on GitHub-hosted
`windows-latest`.
- Exact command / steps: edited the build-wheels and smoke-import-wheels
matrices in `.github/workflows/release.yml` and the README, then ran the
release-workflow tests and the YAML/matrix-shape check above.
- Observed result: tests pass, YAML parses, build matrix is now 4 rows
(Linux x64, Linux arm64, macOS arm64, Windows x64) and the smoke matrix
is 6 rows including the new native Windows row.
- Not tested: the actual win_amd64 build + PyPI publish. Those jobs only
run in the release workflow on a tag or workflow_dispatch, not on a
feature PR. The PR-time release dry-run will exercise the new rows once
a maintainer approves the workflow run. I couldn't run `maturin build
--target x86_64-pc-windows-msvc` end to end here.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
- No new test file: the existing structural gates in
`tests/test_release_workflows.py`
(`test_build_wheels_matrix_excludes_intel_macos`,
`test_aarch64_wheel_uses_native_arm64_runner`, the smoke-import gate
test) already assert the matrix contract and still pass with the Windows
row added.
- I didn't touch CHANGELOG.md — release-please generates it from the
Conventional Commit subject, so the `ci(release):` commit gets picked up
automatically.
- The win_amd64 wheel actually shows up on PyPI on the next tagged
release.
## Description
Sets pyo3 params to support python above 3.13
Fixes #(408
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Updated Cargo.toml
## Testing
Describe the tests you ran to verify your changes:
- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed
## Test Output
```
Compiles
```
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
## What
Loosen over-pinned Python dependency constraints and add missing upper
bounds in `pyproject.toml`. Also bump the neo4j Docker image and uv
builder version.
## Why
Several dependencies had constraints that either blocked security
patches or allowed silent major-version jumps:
- `litellm==1.82.3` was an exact pin — every security patch release
requires a manual lockfile bump
- `transformers`, `sentence-transformers` had no upper bound and have
already crossed major version boundaries without a constraint gate
- `neo4j>=5.20.0` had no upper cap; the driver has already reached 6.x
in the wild
- `mem0ai>=0.1.100` had a pre-1.0 floor while the locked version is
already 1.0.11
- `langchain-core`, `langchain-openai`, `qdrant-client`, `uvicorn` had
no upper bound on a range with active major-version churn
- `docker-compose.yml` pinned neo4j at `5.15.0`, which is 11 patch
releases behind the current 5.x LTS
- `Dockerfile` pinned uv at `0.11.16`; latest stable is `0.11.18`
## How
Constraint changes only — no code changes, no `uv lock --upgrade`. The
existing locked versions all satisfy the new bounds (we added caps, not
floors). `uv` re-resolved the lockfile to format revision 3 (adds
`upload-time` metadata fields) and cleaned up the defunct `llmlingua`
extra entries.
| Dependency | Before | After |
|---|---|---|
| `litellm` | `==1.82.3` | `>=1.82.3,<2.0` |
| `transformers` | `>=4.30.0` | `>=4.30.0,<6.0` |
| `sentence-transformers` | `>=2.2.0` | `>=2.2.0,<6.0` |
| `neo4j` | `>=5.20.0` | `>=5.20.0,<7.0` |
| `mem0ai` | `>=0.1.100` | `>=1.0.0,<2.0` |
| `langchain-core` | `>=0.2.0` | `>=0.2.0,<4.0` |
| `langchain-openai` | `>=0.1.0` | `>=0.1.0,<2.0` |
| `qdrant-client` | `>=1.9.0` | `>=1.9.0,<2.0` |
| `uvicorn` | `>=0.23.0` | `>=0.23.0,<1.0` |
| neo4j Docker image | `5.15.0` | `5.26` |
| uv (Dockerfile ARG) | `0.11.16` | `0.11.18` |
## Breaking changes
None. All currently installed versions fall within the new ranges.
Installers that previously resolved `litellm` to an older exact pin may
now resolve newer patch releases — which is the desired behavior.
---------
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
* ci: speed up GitHub Actions - path filters, caching, timeouts, version upgrades
Performance improvements:
- init-e2e.yml, wrap-e2e.yml: add path filters so e2e Docker builds only run when
e2e-related files change (saves ~10 min per irrelevant PR push)
- init-e2e.yml, wrap-e2e.yml: add concurrency groups to cancel superseded PR runs
- ci.yml: add pip caching to lint and build jobs
- ci.yml: cache actionlint + act binaries in workflow-validation (skip curl on hits)
- eval.yml: add pip caching to smoke-test and weekly-suite jobs
- docs.yml: add pip caching for mkdocs-material install
- rust.yml: replace cargo install --locked cargo-audit/deny with taiki-e/install-action
(prebuilt binaries; saves 2-5 min per audit run)
Bug fixes:
- docker.yml: fix actions/checkout@v6 -> @v4 (v6 does not exist; would break all
Docker builds on every release/PR touching docker paths)
Version upgrades:
- wagoid/commitlint-github-action: @v5 -> @v6
- devcontainers.yml: docker/setup-buildx-action@v3 -> @v4 (align with docker.yml)
Safety improvements:
- ci.yml: add timeout-minutes to all 13 jobs (changes, lint, build-wheel,
prefetch-model, test x4, test-extras, test-agno, commitlint, build,
workflow-validation, docker-native-e2e, windows-native-wrapper, macos-native-wrapper)
- docker.yml: add timeout-minutes to docker-build (75m), docker-manifest (20m),
promote-latest (10m)
- eval.yml: add timeout-minutes to smoke-test (30m); bump weekly-suite 60->90m
- rust.yml: add timeout-minutes to test (30m), wheels (45m), audit (20m)
Observed wall-clock impact on recent PRs:
- Init E2E and Wrap E2E were running on every single PR push regardless of content
- CI workflow was taking 12-17 min; path filters reduce unnecessary e2e runs to 0
* fix(ci): bust actionlint+act cache when workflow file changes
Static cache key 'ci-tools-actionlint-act-v1' never invalidated on
tool version updates. Switched to hashFiles('.github/workflows/ci.yml')
so the cache busts automatically whenever the download scripts are
updated to point at a newer release.
Flagged by adversarial review (Architecture + Testing/Reliability personas).
* fix(ci): add missing Dockerfile COPY paths to e2e path filters
e2e/init/Dockerfile and e2e/wrap/Dockerfile COPY files not covered
by the initial path filter set:
init-e2e: Cargo.toml, Cargo.lock, rust-toolchain.toml, uv.lock,
.claude-plugin, .github/plugin/**, plugins/headroom-agent-hooks/**
wrap-e2e: Cargo.toml, Cargo.lock, rust-toolchain.toml, uv.lock,
sdk/typescript/**, plugins/openclaw/**
Without these, a Rust toolchain bump or SDK change on a PR would
skip the e2e gate entirely, only catching it on the merge to main.
Flagged by adversarial review (Domain/Correctness persona).
* fix(devcontainer): upgrade uv to >=0.7.0 to parse uv.lock revision=3
* fix(devcontainer): set UV_SKIP_WHEEL_FILENAME_CHECK=1 in post-create.sh for gitpython wheel
* ci: bump actions/checkout and actions/setup-node to v5 (Node.js 20 EOL Jun 16)
* fix(devcontainer): export UV_SKIP_WHEEL_FILENAME_CHECK so uv run also skips wheel check
* ci: bump all GitHub Actions to latest versions (Node.js 24)
* fix(test): accept release-please-action v4 or v5 in workflow assertion
* fix(format): ruff format test_release_workflows.py
Replace "every push to main = release" with release-please's
release-PR pattern: the bot watches main and maintains a single
"chore: release vX.Y.Z" PR aggregating conventional commits; merging
that PR creates the tag + GitHub Release, which fires the
release:published event that release.yml now triggers on.
Why
---
Per-merge releases burned PyPI's 10 GiB per-project storage quota
(one fresh wheel matrix ~= 200 MB per merged fix/feat PR).
publish-pypi has failed on every main merge since PR #482 with
"400 Project size too large". Consolidating many fixes into one
release cuts upload frequency ~5x.
What changed
------------
- .github/workflows/release-please.yml: bot watching main
- .release-please-config.json: python release-type + extra-files
for sdk/typescript and plugins/openclaw package.json
- .release-please-manifest.json: tracks current 0.9.1
- .github/workflows/release.yml:
* trigger: push to main -> release: published
* detect-version: reads tag from github.event.release.tag_name
(strips leading "v") so release_version.py does not re-bump
past the bot's tag
* create-release: when release already exists (typical
release-please path), do not pass --notes-file -- that would
clobber the bot's auto-generated changelog body
Tests
-----
Five new regression tests in test_release_workflows.py prevent
silent reversion to per-push triggering and assert the bot
workflow + config invariants.
Note
----
This commit does NOT fix the existing quota breach. Request a
PyPI quota increase, yank old releases, or shrink the wheel
matrix to free immediate space. This PR ensures the future
release cadence stops growing the problem.
Every release since v0.20.16 has uploaded 12 wheels but no sdist. The
underlying failure is a 400 from PyPI:
400 License-File NOTICE does not exist in distribution file
headroom_ai-X.Y.Z.tar.gz at headroom_ai-X.Y.Z/NOTICE
Two-part regression:
1. The hatch -> maturin migration in 2a91cbb (single-wheel maturin build
backend, May 4) replaced `[tool.hatch.build.targets.sdist].include`,
which listed both `LICENSE` and `NOTICE`, with maturin's own include
directive that only carried `LICENSE` over. Maturin's PEP 639 license
auto-discovery still emits `License-File: NOTICE` into the sdist's
PKG-INFO (because NOTICE exists at the project root and matches the
default glob), so the sdist tarball declares a license file it
doesn't physically contain. PyPI's PEP 639 validator rejects with
400. Wheels were unaffected because maturin auto-injects both files
into `*.dist-info/licenses/`.
2. CI showed "publish-pypi" green for ~22 releases despite this break
because twine was bailing earlier with `400 File already exists` on
the wheels (the version detector kept computing the same v0.21.5).
PR #412 added `skip-existing: true` (May 6) to make wheel re-uploads
idempotent. With wheels now silently skipping, twine proceeded to
upload the sdist for the first time in three weeks - and the
dormant License-File error surfaced as a hard 400.
Fix:
- Add `NOTICE` alongside `LICENSE` in `[tool.maturin].include` for the
`sdist` format. Both files now ship in the tarball, matching what
PEP 639 already declares in PKG-INFO.
- Replace the existing "verify sdist contains LICENSE" check with a
generic "every License-File entry in PKG-INFO resolves to a real
tarball member" check. This catches the same bug class for any
future addition (COPYING, AUTHORS, etc.) without another bespoke
literal.
Verified locally:
$ maturin sdist --out dist
Including license file `LICENSE`
Including license file `NOTICE`
Including files matching "LICENSE"
Including files matching "NOTICE"
Built source distribution to dist/headroom_ai-0.9.1.tar.gz
$ tar -tzf dist/headroom_ai-0.9.1.tar.gz | grep -E '(LICENSE|NOTICE)$'
headroom_ai-0.9.1/LICENSE
headroom_ai-0.9.1/NOTICE
$ twine check dist/headroom_ai-0.9.1.tar.gz
Checking dist/headroom_ai-0.9.1.tar.gz: PASSED
Every push to main since v0.21.5 was first published has failed the
publish-pypi job with `400 File already exists`. The workflow's
detect-version step has been computing v0.21.5 repeatedly (the
canonical+commit-height algorithm hasn't bumped past it for the
recent fix-only commits), so each run rebuilds the same wheels with
the same version and twine rejects the duplicates.
Failed runs:
- 25443521479 (PR #406 merge, 15:04 UTC)
- 25452026402 (PR #409 merge, 17:55 UTC)
- 25452038283 (next push, 17:56 UTC)
PyPA's recommended pattern for this scenario is `skip-existing: true`
on the publish action — duplicate uploads become no-ops, fresh
versions still publish normally. Idempotent.
This unblocks main without touching the version-detection algorithm.
A follow-up audit of `headroom/release_version.py` is the right
deeper fix (so a series of `fix:` commits between releases produces
a sequence of patch bumps), but that's a deeper investigation; this
patch just stops the publish job from going red on every push.
Effect after this lands:
- Push to main → wheels rebuilt with whatever version detect-version
computes
- If that version's wheels are already on PyPI → twine skips them,
exit 0, downstream jobs (publish-npm, publish-docker, create-release)
run normally
- If detect-version computes a NEW version not on PyPI → wheels
publish as before, no behaviour change
PR #396's first dry-run on its own changes failed three smoke matrix
entries — two fixed by PR #397's __libc_single_threaded shim (now on
main, this PR is rebased on top), one orthogonal: ubuntu:20.04 + 3.10.
Failure mode on ubuntu:20.04 + 3.10: deadsnakes PPA install path
stopped reliably provisioning python3.10-venv on focal once Ubuntu
20.04 hit End of Standard Support in May 2025. The error is from
apt-get, NOT from `import headroom._core` — the wheel never gets a
chance to load:
E: Unable to locate package python3.10-venv
Continuing to promise wheel-runtime correctness on glibc 2.31 in CI
would require either pulling ESM-tier ubuntu:focal images (paid /
auth-gated) or pinning a specific deadsnakes snapshot URL — neither
of which we want owning long-term.
Floor coverage we keep:
- manylinux_2_28_x86_64 (glibc 2.28 — the floor we promise)
- manylinux_2_28_aarch64 (glibc 2.28 — same)
- ubuntu:22.04 + 3.12 x86_64 (glibc 2.35 — issue #355's reporter env)
- ubuntu:22.04 + 3.12 aarch64 (glibc 2.35 — arm equivalent)
- macos-14 + 3.13 (Apple Silicon native)
Test pin in tests/test_release_workflows.py
(test_release_workflow_has_smoke_import_wheel_gate) only requires the
manylinux floors, ubuntu:22.04, and macos-14, so this drop doesn't
break the structural invariant.
The X1 smoke-import gate (PR #387) catches runtime symbol mismatches
on the wheel before publish, but only at release time. Recent break
patterns were upstream of that:
- #379 (docker bake `name=` regression in PR #376)
- #382 (sdist `os: ubuntu-latest` → `ubuntu-24.04` rename)
- #384 / #385 / #386 (glibc shim alias / link-order iterations)
- #387's own heredoc-indent regression that broke main on the FIRST
release run after merge — the heredoc was inside `bash -ec '...'`,
no PR-time check exercised it
X2 adds a `pull_request:` trigger to release.yml with a NARROW path
filter so the dry-run runs at PR time for changes that affect wheel
layout / release pipeline, but skips for source-only PRs to
`crates/headroom-core` / `crates/headroom-proxy`.
Path filter: release.yml, docker.yml, crates/headroom-py/**,
pyproject.toml, root Cargo.toml, Cargo.lock.
publish-pypi / publish-npm / publish-github-packages / publish-docker
/ create-release all gate on `github.event_name != 'pull_request'`,
so a PR run never publishes — the dry-run is build + collect-dist +
smoke-import only.
concurrency rules:
- PR runs: namespaced by PR number (`pr-N`), cancel-in-progress=true.
- main runs: namespaced by ref_name, cancel-in-progress=false (a
tag-push release that's mid-flight must not be cancelled).
Test pin covers all four invariants (trigger, path filter, publish
gates, concurrency split). 21 tests in test_release_workflows.py,
all green locally.
The X1 smoke-import job (PR #387) embedded the smoke check as a
`<<PY ... PY` heredoc inside `bash -ec '...'`. The outer bash
single-quote preserves whitespace, so the heredoc body and the
closing `PY` retained their YAML indentation (column 14 inside
`bash -ec`). Bash never found a column-0 `PY` and read past EOF:
bash: line 71: warning: here-document at line 65 delimited by
end-of-file (wanted `PY')
IndentationError: unexpected indent
Process completed with exit code 1
Caught immediately on the post-merge release run on main,
manylinux_2_28_x86_64 / Python 3.11 (glibc 2.28-floor):
https://github.com/chopratejas/headroom/actions/runs/25361396712/job/74362427755
`python -c "<f-string>"` was the obvious next try but reintroduces
single-quote nesting (Python f-strings need quote chars; outer
`bash -ec '...'` cannot contain unescaped single quotes).
Fix: write the smoke script to `${RUNNER_TEMP}/smoke_import.py`
in a new "Stage smoke-import script" step (one heredoc at YAML
`run: |` level — uniform indent strip works fine). Linux job
mounts it via `-v ${RUNNER_TEMP}/smoke_import.py:/smoke_import.py:ro`
and runs `python /smoke_import.py`. macOS host runs the same
file directly. No quoting drift between paths.
Locally validated:
- actionlint clean
- 20/20 tests in test_release_workflows.py pass
- hand-execution of the heredoc + script roundtrip works
This is the second X1 follow-up after PR #387's shellcheck fix.
The original X1 design's gap: no PR-time release dry-run that
would have caught the heredoc on PR #387 itself. X2 (PR-time
dry-run) is the structural fix.
actionlint+shellcheck flagged the bash heredoc on the macOS host
step (release.yml:486) for SC2012 (use find instead of ls) and
SC2086 (quote variables to prevent globbing/word splitting). The
Linux container step had the same pattern but escaped detection
because actionlint can't reach into `docker run bash -ec '...'`.
Replace `ls .../headroom_ai-*-${py_tag}-${py_tag}-${arch_tag}.whl
| head -1` with `find ... -name "..." -print -quit` so the glob
expansion happens in find (handles non-alphanumeric filenames
correctly) and the variables sit inside a quoted -name argument
(no SC2086 trigger). Also replace the diagnostic `ls -la` calls
with portable find variants (-printf for GNU find on Linux,
-exec basename for BSD find on macOS).
Net behaviour identical: still picks one matching wheel (only one
exists per py_tag+arch_tag), still prints all wheels for diagnosis
on miss.
Issue #355 plus the three follow-on hotfixes (#384/#385/#386) all
share a pattern: the wheel is technically valid (clippy passes,
tests pass, auditwheel is happy, the static-symbol audit added in
#384 is happy) but FAILS at runtime on a customer's box because of
a dynamic-link symbol mismatch. None of our pre-publish gates
actually `import headroom._core` on a representative customer
environment. They only build it.
What X1 adds
------------
A `smoke-import-wheels` job that runs after `build-wheels` and
before `publish-pypi` / `publish-docker` / `create-release`.
Matrix (6 jobs in parallel, ~3 min wall-clock):
- `manylinux_2_28_x86_64` + Python 3.11 (the floor we promise)
- `ubuntu:22.04` (glibc 2.35) + Python 3.12 (issue #355's env)
- `ubuntu:20.04` (glibc 2.31) + Python 3.10 (older LTS)
- `manylinux_2_28_aarch64` + Python 3.11 (aarch64 floor)
- `ubuntu:22.04` arm64 + Python 3.12 (aarch64 customer env)
- `macos-14` host + Python 3.13 (Apple Silicon)
Each job downloads its arch's wheel artifact, installs the wheel
matching its Python version inside the container, and runs the
exact command the proxy's `_check_rust_core` runs at startup:
from headroom._core import hello as _rust_hello
If any matrix entry fails, `publish-pypi` / `publish-docker` /
`create-release` are blocked. The matrix tells us exactly which
customer environment combination breaks.
Regression test in tests/test_release_workflows.py:
`test_release_workflow_has_smoke_import_wheel_gate` pins the job's
existence, the required matrix entries, and — critically — the
gating wires (publish-pypi / publish-docker / create-release all
need-and-require-success on the smoke job). A future "this slow
CI step always passes anyway, drop it" refactor fails at PR time.
Companion tests `test_glibc_compat_shim_present_in_headroom_py`
and `test_release_workflow_audits_wheel_glibc_symbols` (added in
#384) cover the static-symbol gate; this PR is the dynamic-link
gate. Both are needed.
Issue #355: published headroom_ai-*-manylinux_2_28_*.whl fails to import on Ubuntu 22.04, Debian 11/12, Conda envs with libc < 2.38: 'ImportError: undefined symbol: __isoc23_strtoll'.
Root cause: ORT prebuilt artifacts (downloaded via fastembed's ort-download-binaries-rustls-tls feature) are compiled with gcc-14.2.1 on a glibc-2.38+ host and reference __isoc23_strtoll. Our manylinux build host has glibc 2.38 so the link succeeds; end users with older glibc don't.
Two-part fix: (1) crates/headroom-py/glibc_compat.c provides weak-alias definitions for __isoc23_strtol/strtoll/strtoul/strtoull delegating to the older strtol* family, compiled by build.rs on Linux/glibc only. The dynamic linker prefers glibc's strong symbol when present (>= 2.38) and falls back to ours when not (< 2.38). (2) scripts/audit_wheel_glibc_symbols.py is a release.yml gate that runs objdump -T on every Linux wheel and rejects any UND symbol whose required glibc version exceeds the wheel's manylinux floor.
Validated: the audit correctly rejects the actually-broken v0.20.26 wheel with a precise diagnostic. The shim itself is a tiny static link with zero runtime cost.
Regression tests in tests/test_release_workflows.py pin the shim's load-bearing pieces (.c file, build.rs trigger, [build-dependencies] cc dep) and the audit invocation in release.yml. Future drift fails at PR time, not in the next release.
This is the same bug class as PR #371 (rustls-everywhere). Each instance gets fixed; the audit gate now catches the *class* — any future static linkage that introduces a post-floor symbol gets blocked before publish.
PR #376 pinned the wheel matrix's `os:` from `ubuntu-latest` to
`ubuntu-24.04` (explicit pinning, no semantic change). It silently
disabled the sdist build, whose conditional was
if: matrix.os == 'ubuntu-latest' && matrix.target == 'x86_64-unknown-linux-gnu'
The literal `'ubuntu-latest'` no longer matched, sdist never built,
`release-assets/*.tar.gz` was empty, and `create-release` failed at
gh release upload release-assets/*.tar.gz --clobber
with "no matches found". v0.20.22's release didn't get its sdist on
the GitHub Release.
Fix: key the conditional on `matrix.target` only. sdist is
platform-independent so any single matrix row is fine; using `target`
decouples the sdist build from any future host-runner rename.
Regression test in tests/test_release_workflows.py:
`test_sdist_build_conditional_keyed_on_target_not_os` pins the
target-only form. A future "let's add `os` back for clarity" refactor
will fail at PR time, not 8 minutes into a release.
This is a follow-up to PR #379 (docker bake `name=`) — same root
cause class: PR #376's matrix-shape changes silently broke a
downstream conditional whose literal value was tied to the OLD
matrix shape. Both fixes are now in place + pinned by tests.
GitHub-hosted Linux arm64 runners (`ubuntu-24.04-arm`) went GA in Aug
2025 and are free for public repositories. Switching the aarch64
wheel + the multi-arch docker matrix off `ubuntu-latest`+QEMU onto
the native runner cuts wall-clock on both surfaces.
release.yml — build-wheels matrix
* `aarch64-unknown-linux-gnu`: `ubuntu-latest` → `ubuntu-24.04-arm`.
maturin-action still runs inside `quay.io/pypa/manylinux_2_28_aarch64`,
but the container now executes natively on an aarch64 kernel
instead of through QEMU emulation. Aarch64 wheel build drops
from ~50–60 min to ~10 min.
* `x86_64-unknown-linux-gnu`: `ubuntu-latest` → `ubuntu-24.04`
(pin the moving alias for reproducibility; no semantic change).
docker.yml — fan-out + manifest merge
* Pre-#377: one `docker-variant-tags` matrix job per variant on
`ubuntu-latest`, using bake's `platforms = [amd64, arm64]` with
QEMU for the arm64 leg. ~1h per variant, 8 variants.
* Post-#377: split into `docker-build` (variant × arch = 16
parallel jobs, each on its native runner, single-platform
push-by-digest) and `docker-manifest` (per variant, merges the
two arch digests into a multi-arch tagged manifest with
`docker buildx imagetools create`, signs the index manifest
with cosign). Wall-clock drops from ~1h per variant to ~10 min.
* `docker/setup-qemu-action` removed — there's no QEMU left.
* Per-(variant, arch) GHA cache scopes so the two arches don't
collide on cache keys.
* `promote-latest` rewired to depend on `docker-manifest`.
Behavior change: cosign now signs only the multi-arch index digest
per variant, not each per-platform image. `cosign verify <repo>:tag`
(the typical flow) is unchanged because cosign resolves the tag to
the index digest. Verifiers pinning a specific per-arch digest will
need to verify the index digest instead.
Regression tests in tests/test_release_workflows.py:
* `test_aarch64_wheel_uses_native_arm64_runner` — pins the
aarch64 row to `ubuntu-24.04-arm` (and the amd64 row to
`ubuntu-24.04`, not `-latest`), so a future "let me unify on
ubuntu-latest" refactor surfaces the QEMU regression at PR time.
* `test_docker_workflow_builds_on_native_arch_runners` — pins the
fan-out matrix's arch entries, asserts push-by-digest, asserts
`setup-qemu-action` is absent from non-comment lines, asserts
the manifest-merge job exists.
Verified:
* Both workflow files parse as valid YAML with the expected job
graph (`docker-build` → `docker-manifest` → `promote-latest`,
16 fan-out jobs, 8 manifest jobs).
* `docker buildx imagetools inspect <tag> --format '{{ json . }}'`
exposes the index digest at `.manifest.digest` (confirmed via
Docker's official reference).
* `ubuntu-24.04-arm` is the correct GitHub-hosted runner label
(GA 2025-08-07, free for public repos).
* `make ci-precheck-rust` and `make ci-precheck-python` both pass
locally; `tests/test_release_workflows.py` is 15/15 green
(13 existing + 2 new).
# Root cause of the wheel-build cascade
We have shipped 5 release-pipeline hot-fixes in 12 hours, each
addressing a different symptom of the same architectural problem:
1. PR #363 — npm artifact downloads + tried `yum openssl-devel`
2. PR #367 — vendored OpenSSL in `headroom-proxy` + dropped Intel mac
3. PR #369 — Debian-cross perl install (`perl` not `libipc-cmd-perl`)
4. PR #370 — moved `openssl/vendored` from headroom-proxy to headroom-py
5. (this PR) — ELIMINATE OpenSSL entirely
Each fix exposed a different missing system package or feature flag in
a different build surface (manylinux x86_64 vs aarch64-cross-Debian vs
macOS Intel vs e2e/wrap Dockerfile vs e2e/init Dockerfile vs main
Dockerfile vs devcontainer). We were playing whack-a-mole because every
Cargo dep change to the OpenSSL surface required matching system-package
updates in 6+ different Dockerfiles and workflows, and the PR-level CI
didn't exercise all of them.
# Why this PR is the structural fix
`fastembed` exposes clean rustls feature flags:
- `hf-hub-rustls-tls` (replaces default `hf-hub-native-tls`)
- `ort-download-binaries-rustls-tls` (replaces default `…native-tls`)
By disabling fastembed's default features and enabling the rustls
variants explicitly, we remove `native-tls` (and therefore `openssl-sys`,
`openssl`, `openssl-src`, perl modules, OpenSSL build-time deps,
vendored OpenSSL ~30s build cost) from the entire workspace dep tree.
Verified locally:
$ cargo tree -p headroom-py -i openssl-sys
error: package ID specification `openssl-sys` did not match any packages
$ cargo tree -p headroom-py -i native-tls
error: package ID specification `native-tls` did not match any packages
$ cargo build --release -p headroom-py
Finished `release` profile [optimized] target(s) in 25.57s
(Down from 1m+ with vendored OpenSSL.)
# Cleanups enabled by this change
- crates/headroom-py/Cargo.toml — dropped the `openssl/vendored`
workaround from PR #370.
- crates/headroom-proxy/Cargo.toml — same dep removed.
- e2e/wrap/Dockerfile — dropped `yum install openssl-devel pkgconfig
perl-IPC-Cmd`. Comment retained explaining why.
- e2e/init/Dockerfile — same.
- Dockerfile (main) — dropped `pkg-config libssl-dev` from apt-get.
- .devcontainer/Dockerfile — dropped `pkg-config libssl-dev`.
- .github/workflows/release.yml — removed the entire before-script-linux
block (perl install probe + multi-package-manager dispatch + fail-loud
assertion). No longer needed.
# Regression gate
Three new structural tests in tests/test_release_workflows.py:
- test_no_openssl_sys_in_wheel_build_tree — runs `cargo tree -p <crate>
-i openssl-sys` for headroom-py / headroom-proxy / headroom-core. If
openssl-sys reappears (a future native-tls enabler creeping in via a
new dep), this fails AT PR TIME with an actionable message.
- test_no_native_tls_in_wheel_build_tree — same shape, native-tls is
the proximate cause.
- test_fastembed_uses_rustls_features — checks the Cargo.toml so a
future "let me bump fastembed and forget the features" doesn't
silently re-introduce OpenSSL.
Plus two cleanup gates:
- test_dockerfiles_no_longer_install_openssl_devel
- test_release_yml_does_not_install_openssl_or_perl_for_wheels
All 13 release-workflow tests pass. `make ci-precheck` PASSED.
# What this teaches us about rollouts (per user's ultrathink ask)
The 5-fix cascade exposed three meta-problems:
1. PR checks don't block merges. PR #370 had docker-init-e2e,
docker-wrap-e2e, docker-native-e2e all FAILED yet got merged.
Branch protection should require these checks. Operator action
needed (cannot fix in code).
2. Local validation is misleading. `cargo build -p headroom-py` from
the workspace root used the workspace lockfile and looked green;
CI did fresh resolution against headroom-py's manifest alone where
the feature wasn't enabled. Lesson: verify structural invariants
with `cargo tree -e features` before trusting that a build "works."
3. 6+ build surfaces with independent system-dep state. Every Cargo
change required matching updates in 6 places. The structural answer
(this PR) is to NOT depend on system OpenSSL at all. Where structural
fixes are not possible, the answer is a single shared
scripts/install-rust-build-deps.sh — but with this PR there's
nothing left to install.
Previous wheel hot-fix (#367) introduced a NEW failure mode that the
F1-merge release run surfaced:
E: Unable to locate package libipc-cmd-perl
The aarch64-unknown-linux-gnu maturin-action target does NOT use the
AlmaLinux 8 manylinux_2_28 image — it uses a Debian/Ubuntu-based
cross-compile container. The previous hot-fix's apt branch installed
`libipc-cmd-perl` which is a deprecated alias and is no longer in the
default Debian/Ubuntu sources. The build failed before openssl-src
could even start.
# What the script actually needs to do
`IPC::Cmd` is a Perl core module since 5.10, so any working `perl`
install provides it. The fix:
1. **Probe first.** `perl -MIPC::Cmd -e 1` exits cleanly when the
module is already importable — skip the install entirely. Some
manylinux images already ship it; others don't.
2. **Cover every package manager.** dnf (modern RHEL family) → yum
(older RHEL) → apt-get (Debian/Ubuntu) → apk (Alpine/musllinux).
The maturin-action uses different containers per (target, manylinux)
combo and we don't get to pick.
3. **Use `perl` not `libipc-cmd-perl` on Debian.** The plain `perl`
meta-package pulls `perl-modules-*` which contains IPC::Cmd. Works
on every Debian/Ubuntu version we'll see; `libipc-cmd-perl` is
gone from default sources.
4. **Fail loud after install.** `perl -MIPC::Cmd -e 'print "loaded
OK"'` runs unconditionally at the end. If somehow the module is
STILL missing, we fail here — not 5 minutes later in the
openssl-src compile step where the error message is harder to
debug. Matches the project's "no silent fallback" rule.
# Tests
`test_build_wheels_installs_perl_ipc_cmd_for_vendored_openssl`
updated to gate the new shape:
- Asserts `perl -MIPC::Cmd -e 1` pre-probe is present
- Asserts dnf/yum/apt-get/apk branches all exist
- Asserts apt branch installs `perl` not `libipc-cmd-perl`
- Asserts `libipc-cmd-perl` does not appear on any non-comment line
- Asserts the final `perl -MIPC::Cmd` fail-loud assertion is present
All 11 release-workflow tests pass. `make ci-precheck` PASSED.
The previous hot-fix (#363) addressed npm artifact downloads and added
openssl-devel installs in the manylinux container, but the wheel build
still fails on three of four matrix entries with three distinct errors:
1. ubuntu-x86_64 with `manylinux: auto` resolved to manylinux2014
(CentOS 7 / OpenSSL 1.0.2k). `openssl-sys 0.9` requires OpenSSL
1.1.0+ — "different version of OpenSSL was found".
2. ubuntu-aarch64 cross-compiles via `aarch64-unknown-linux-gnu-gcc`
from an x86_64 manylinux container. The `yum install openssl-devel`
we added installs x86_64 headers; `/usr/aarch64-unknown-linux-gnu/
include/` has no OpenSSL — "openssl/opensslv.h: No such file or
directory".
3. macos-15-intel fails on `ort-sys` (transitive via the ML compression
backend), which has no prebuilt ONNX Runtime binaries for
`x86_64-apple-darwin`. Unrelated to OpenSSL; an upstream limitation.
Why the workspace pulls openssl-sys at all: `hf-hub` (transitive via
`fastembed`) hard-codes `native-tls` as a default feature. Cargo's
feature unification then enables openssl-sys for the whole workspace
despite our `reqwest`/`tokio-tungstenite`/`tokio-rustls` preferences.
# Fix 1: vendored OpenSSL
Add `openssl = { version = "0.10", features = ["vendored"] }` to
`crates/headroom-proxy/Cargo.toml`. The `vendored` feature compiles
OpenSSL from source as part of the cargo build — works on every target
uniformly. Local build verified: cargo now pulls
`openssl-src v300.6.0+3.6.2` and compiles it. ~30s extra one-time
build cost.
The `openssl/vendored` feature DEFEATS `OPENSSL_DIR`. We therefore
remove the previous hot-fix's "Install OpenSSL (macOS)" step that
exported `OPENSSL_DIR` — leaving it would silently regress to the
system-OpenSSL path that broke originally.
# Fix 2: pin manylinux floor to 2_28
Change x86_64-unknown-linux-gnu from `manylinux: auto` to
`manylinux: 2_28` (matching aarch64 + the e2e Dockerfiles). This
isn't strictly required with vendored OpenSSL — the floor is now
glibc 2.28 / AlmaLinux 8 which has modern toolchain — but it removes
the CentOS-7 surface entirely and matches our runtime container
target.
# Fix 3: drop x86_64-apple-darwin from the matrix
`ort-sys 2.0.0-rc.12` has no prebuilt ONNX Runtime binaries for that
target. Building ORT from source would add CMake + ~5 minutes per
build. Apple Silicon macOS (`aarch64-apple-darwin`) is fully covered;
Intel-mac users install from the platform-independent sdist this
matrix also produces.
Tracked as a follow-up: switch the ML backend to `ort-tract` or
upstream a request for x86_64 macOS prebuilts.
# before-script-linux: keep perl-IPC-Cmd, drop openssl-devel
OpenSSL's vendored `Configure` script needs `IPC::Cmd` (without it
the build fails with "Can't locate IPC/Cmd.pm"). System
openssl-devel is no longer needed.
# Tests
4 new regression tests gate this:
- `test_headroom_proxy_vendors_openssl`
- `test_build_wheels_installs_perl_ipc_cmd_for_vendored_openssl`
- `test_build_wheels_does_not_set_openssl_dir`
- `test_build_wheels_matrix_excludes_intel_macos`
Plus the previous 7. All 11 release-workflow tests pass.
`make ci-precheck` PASSED. Local `cargo build --release -p headroom-py`
green.
Three independent failures on the post-merge release run for PR #360,
all introduced by the single-wheel maturin refactor:
1. Linux wheel matrix (`ubuntu-x86_64`, `aarch64-unknown-linux-gnu`)
failed inside the manylinux container with:
Could not find openssl via pkg-config
The system library `openssl` required by crate `openssl-sys`
was not found.
`openssl-sys` is transitive via `fastembed` → `hf-hub` → `ureq`
→ `native-tls`. The e2e Dockerfiles (`e2e/wrap`, `e2e/init`) we
wrote for #360 install `openssl-devel` upfront, but the
`build-wheels` matrix uses `PyO3/maturin-action@v1` which spins
up its OWN manylinux container that does not inherit those
installs. Fix: add a `before-script-linux:` to the action with a
yum/apt-get conditional so it works on RHEL-family (manylinux2014,
manylinux_2_28) and Debian-family musllinux variants.
2. macOS x86_64 wheel build failed with `maturin` exit 1 from the
same `openssl-sys` lookup. The aarch64 macos-14 runner happens
to have `/opt/homebrew/opt/openssl@3` on `openssl-sys`'s default
discovery path; the Intel macos-15-intel runner uses
`/usr/local/Cellar` which is NOT on that path. Fix: add a
pre-maturin step that runs `brew install openssl@3` and exports
`OPENSSL_DIR` / `OPENSSL_LIB_DIR` / `OPENSSL_INCLUDE_DIR` /
`PKG_CONFIG_PATH`. The aarch64 runner happily picks up the
explicit env vars too — no regression.
3. `publish-npm` and `publish-github-packages` both fail with
"Artifact not found for name: dist". Both jobs `npm pack` + `npm
publish` directly from the checked-out source tree — they never
consume the Python `dist` artifact. The `Download dist artifact`
step was vestigial dead code carried over from a prior workflow
shape; the only reason it didn't fail before #360 is that the
pre-refactor `build` job DID upload a `dist` artifact. Post-#360,
`dist` is produced by `collect-dist` and neither publish job is
gated on it (by design — npm vs PyPI ecosystems publish
independently). Fix: remove the dead download step from both
jobs. Loose coupling is preserved; `create-release` still gates
the GitHub Release tag on all of build / build-wheels /
collect-dist / publish-* succeeding.
Why the PR-level CI didn't catch any of this: `release.yml` only
runs on `push: branches: [main]`. PR #360's CI exercised `ci.yml`
which has a separate `ci-build-wheels-on-pr` matrix that uses a
different setup. The release surface only fires post-merge.
Tests added (regression gates):
- `test_build_wheels_installs_openssl_devel_on_linux_via_before_script`
- `test_build_wheels_resolves_openssl_dir_explicitly_on_macos`
- `test_npm_publish_jobs_do_not_download_dist_artifact`
All 9 release-workflow tests pass. `make ci-precheck` PASSED.
Code-scanning alert #65 (CodeQL actions/missing-workflow-permissions,
CWE-275) flagged the new build-wheels job for not declaring an explicit
permissions block. While at it, audit the rest of release.yml — same
gap exists on detect-version, build, collect-dist, and publish-npm.
Each job gets `contents: read` (the minimal default) since none of them
push, write packages, or mutate releases through GITHUB_TOKEN. Existing
write-bearing jobs (publish-pypi: id-token, publish-github-packages:
packages, create-release: contents) keep their narrower scopes.
Eliminates the dual-package architecture that was the root cause of #355.
`pip install headroom-ai` now produces ONE wheel containing both the Python
source (headroom/*.py) and the compiled Rust extension (headroom/_core.so).
No more separate `headroom-core-py` package, no more chicken-and-egg with
PyPI publication, no more wheelhouse / PIP_FIND_LINKS / composite-action
plumbing in CI.
This is the canonical pattern used by cryptography, polars, ruff,
pydantic-core, and other Rust-as-core Python packages. Honors the
"Rust as core engine" direction.
## What changed
- pyproject.toml: `[build-system]` swapped from hatchling to maturin.
`[tool.hatch.*]` deleted; `[tool.maturin]` added pointing at
`crates/headroom-py/Cargo.toml` for the cdylib. `python-source = "."`
picks up the root `headroom/` package directly (dashboard HTML
templates and other non-Python files included automatically).
- crates/headroom-py/pyproject.toml: deleted. The crate is no longer a
separate published package; its Cargo.toml stays as the cdylib build
target invoked via `[tool.maturin] manifest-path`.
- crates/headroom-py/python/: deleted (placeholder layout for the old
separate package).
## CI updates
- ci.yml: `test` / `test-extras` / `test-agno` jobs simplified — Rust
toolchain set up before `pip install -e .` (which now invokes maturin
via build-system). Removed the "build wheel + symlink .so" dance.
`build` job swapped from `python -m build` (hatch) to
`maturin build` + `maturin sdist`.
- release.yml: collapsed dual-package matrix into one. New `build-wheels`
matrix produces cross-platform wheels for cp310/11/12/13 ×
{linux x86_64, linux aarch64, macos x86_64, macos aarch64}. New
`collect-dist` aggregator merges artifacts. publish-pypi consumes the
merged dist.
- init-native-e2e.yml: dropped windows-latest from the matrix —
upstream `esaxx-rs` (/MT) and `ort-sys` (/MD) link with conflicting
MSVC C runtime libraries, so the Rust extension cannot build for
win_amd64 today. Tracked as a follow-up; not a blocker for Linux+macOS.
- headroom-e2e-setup: composite action now sets up Rust toolchain +
Swatinem/rust-cache before `pip install -e .[proxy]`.
- eval.yml, publish.yml, rust.yml: same pattern — rust toolchain before
install. rust.yml's wheels job builds from root pyproject.toml (no
more `-m crates/headroom-py/Cargo.toml`).
- e2e/init/Dockerfile, e2e/wrap/Dockerfile: install rust + maturin in
the build stage; copy `crates/` + workspace `Cargo.toml/lock` so the
install can build the extension. Dropped `HEADROOM_REQUIRE_RUST_CORE=false`
from wrap-e2e — the image now ships the full Rust core.
- Dockerfile (main): simplified — no more Layer 2/3 dance with
`headroom-core-py` install + symlink. Single `uv pip install` builds
+ installs everything.
- .devcontainer/Dockerfile: rust toolchain + libssl-dev + maturin
added so `uv sync` builds the extension inside the devcontainer.
## Lockfile + script
- uv.lock: regenerated. No `headroom-core-py` entries remain.
- scripts/build_rust_extension.sh: simplified from a symlink-into-tree
workaround to a thin wrapper around `pip install -e .`. The maturin
build-backend handles placement automatically.
## Local validation (all green on macOS aarch64)
1. Clean venv `pip install -e .` → `from headroom._core import …` works.
2. `maturin build --release` → 13.8 MB wheel, 336 files including
`headroom/_core.cpython-311-darwin.so` (32 MB cdylib) and
`headroom/dashboard/templates/dashboard.html`.
3. `pip install <wheel>` in fresh venv → import works.
4. Wheel contents verified via `unzip -l`.
5. `pytest tests/test_transforms/test_diff_compressor.py` — 29 passed.
6. `pytest tests/test_relevance.py` — 30 passed.
7. `cargo build --workspace` + `cargo test --workspace` — all green.
8. `make ci-precheck` — 176 Python tests + Rust + commitlint green.
## Migration notes
Users on `pip install headroom-ai` get the Rust core automatically
(linux + macos wheels). sdist installs require rust toolchain available
locally — pip will build via maturin.
Closes#355
Supersedes #357 (workarounds-based fix abandoned in favor of
architectural fix)
The openclaw plugin depends on headroom-ai, but during the release build
the version-sync script updates the dependency to the new version (e.g.
^0.6.7) which hasn't been published to npm yet. Fix by installing the
locally-packed SDK tarball first so the dependency is already satisfied
when npm install runs for remaining packages.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The version-sync.py script already sets the target version in package.json
before npm version runs. When npm version receives the same version that's
already in package.json, it exits with "Version not changed" (exit code 1),
breaking the build job. Adding --allow-same-version makes npm version a
no-op when the version matches, fixing the release pipeline.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fix workflow validation failures by wiring detect-version outputs into all
release publish jobs, renaming the GitHub Packages skip variable to a
valid Actions variable name, and adjusting the macOS PATH export for
actionlint.
Also make min_tokens_to_compress use token counting instead of whitespace
splits so compact JSON tool outputs still compress after merging the
latest main branch changes, and add a regression test for that path.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Call the Docker workflow from the release pipeline so Docker publishes in
the same run, build npm tarballs alongside Python distributions, and
attach those artifacts to the GitHub release page.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Determine the release bump from all unreleased commits since the previous
release tag and apply the highest required semantic version increment.
This keeps feat commits at a minor bump unless a breaking change requires
major, even when later patch-level commits are present.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace the inline release version math with a tested helper that normalizes legacy four-part tags and computes a single semantic version for packages and GitHub releases.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Grant the release job contents write permission so GitHub releases can be created, and add the missing docs/overrides directory required by MkDocs deployment.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- publish-pypi: add permissions: id-token: write for OIDC trusted publishing
- publish-npm: add npm run build before npm publish for both packages
- publish-github-packages: add npm run build, use --registry for GPR
- version-sync: add update_openclaw_package_json to sync headroom-ai dep range
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The release workflow now uses a loop-free algorithm:
- pyproject.toml is the canonical source of truth (never committed by workflow)
- Git tags use v{canonical}.{height} format (e.g. v0.5.25.3)
- npm publishes use 3-part semver bumped from canonical
- No commit step eliminates infinite release loops
- paths-ignore reduces unnecessary workflow triggers
Also:
- Add .releaseetadata to .gitignore
- Separate npm_version output for semver-compatible npm publishing
- create-release no longer blocks on publish jobs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
