* ci: speed up GitHub Actions - path filters, caching, timeouts, version upgrades
Performance improvements:
- init-e2e.yml, wrap-e2e.yml: add path filters so e2e Docker builds only run when
e2e-related files change (saves ~10 min per irrelevant PR push)
- init-e2e.yml, wrap-e2e.yml: add concurrency groups to cancel superseded PR runs
- ci.yml: add pip caching to lint and build jobs
- ci.yml: cache actionlint + act binaries in workflow-validation (skip curl on hits)
- eval.yml: add pip caching to smoke-test and weekly-suite jobs
- docs.yml: add pip caching for mkdocs-material install
- rust.yml: replace cargo install --locked cargo-audit/deny with taiki-e/install-action
(prebuilt binaries; saves 2-5 min per audit run)
Bug fixes:
- docker.yml: fix actions/checkout@v6 -> @v4 (v6 does not exist; would break all
Docker builds on every release/PR touching docker paths)
Version upgrades:
- wagoid/commitlint-github-action: @v5 -> @v6
- devcontainers.yml: docker/setup-buildx-action@v3 -> @v4 (align with docker.yml)
Safety improvements:
- ci.yml: add timeout-minutes to all 13 jobs (changes, lint, build-wheel,
prefetch-model, test x4, test-extras, test-agno, commitlint, build,
workflow-validation, docker-native-e2e, windows-native-wrapper, macos-native-wrapper)
- docker.yml: add timeout-minutes to docker-build (75m), docker-manifest (20m),
promote-latest (10m)
- eval.yml: add timeout-minutes to smoke-test (30m); bump weekly-suite 60->90m
- rust.yml: add timeout-minutes to test (30m), wheels (45m), audit (20m)
Observed wall-clock impact on recent PRs:
- Init E2E and Wrap E2E were running on every single PR push regardless of content
- CI workflow was taking 12-17 min; path filters reduce unnecessary e2e runs to 0
* fix(ci): bust actionlint+act cache when workflow file changes
Static cache key 'ci-tools-actionlint-act-v1' never invalidated on
tool version updates. Switched to hashFiles('.github/workflows/ci.yml')
so the cache busts automatically whenever the download scripts are
updated to point at a newer release.
Flagged by adversarial review (Architecture + Testing/Reliability personas).
* fix(ci): add missing Dockerfile COPY paths to e2e path filters
e2e/init/Dockerfile and e2e/wrap/Dockerfile COPY files not covered
by the initial path filter set:
init-e2e: Cargo.toml, Cargo.lock, rust-toolchain.toml, uv.lock,
.claude-plugin, .github/plugin/**, plugins/headroom-agent-hooks/**
wrap-e2e: Cargo.toml, Cargo.lock, rust-toolchain.toml, uv.lock,
sdk/typescript/**, plugins/openclaw/**
Without these, a Rust toolchain bump or SDK change on a PR would
skip the e2e gate entirely, only catching it on the merge to main.
Flagged by adversarial review (Domain/Correctness persona).
* fix(devcontainer): upgrade uv to >=0.7.0 to parse uv.lock revision=3
* fix(devcontainer): set UV_SKIP_WHEEL_FILENAME_CHECK=1 in post-create.sh for gitpython wheel
* ci: bump actions/checkout and actions/setup-node to v5 (Node.js 20 EOL Jun 16)
* fix(devcontainer): export UV_SKIP_WHEEL_FILENAME_CHECK so uv run also skips wheel check
* ci: bump all GitHub Actions to latest versions (Node.js 24)
* fix(test): accept release-please-action v4 or v5 in workflow assertion
* fix(format): ruff format test_release_workflows.py
# Root cause of the wheel-build cascade
We have shipped 5 release-pipeline hot-fixes in 12 hours, each
addressing a different symptom of the same architectural problem:
1. PR #363 — npm artifact downloads + tried `yum openssl-devel`
2. PR #367 — vendored OpenSSL in `headroom-proxy` + dropped Intel mac
3. PR #369 — Debian-cross perl install (`perl` not `libipc-cmd-perl`)
4. PR #370 — moved `openssl/vendored` from headroom-proxy to headroom-py
5. (this PR) — ELIMINATE OpenSSL entirely
Each fix exposed a different missing system package or feature flag in
a different build surface (manylinux x86_64 vs aarch64-cross-Debian vs
macOS Intel vs e2e/wrap Dockerfile vs e2e/init Dockerfile vs main
Dockerfile vs devcontainer). We were playing whack-a-mole because every
Cargo dep change to the OpenSSL surface required matching system-package
updates in 6+ different Dockerfiles and workflows, and the PR-level CI
didn't exercise all of them.
# Why this PR is the structural fix
`fastembed` exposes clean rustls feature flags:
- `hf-hub-rustls-tls` (replaces default `hf-hub-native-tls`)
- `ort-download-binaries-rustls-tls` (replaces default `…native-tls`)
By disabling fastembed's default features and enabling the rustls
variants explicitly, we remove `native-tls` (and therefore `openssl-sys`,
`openssl`, `openssl-src`, perl modules, OpenSSL build-time deps,
vendored OpenSSL ~30s build cost) from the entire workspace dep tree.
Verified locally:
$ cargo tree -p headroom-py -i openssl-sys
error: package ID specification `openssl-sys` did not match any packages
$ cargo tree -p headroom-py -i native-tls
error: package ID specification `native-tls` did not match any packages
$ cargo build --release -p headroom-py
Finished `release` profile [optimized] target(s) in 25.57s
(Down from 1m+ with vendored OpenSSL.)
# Cleanups enabled by this change
- crates/headroom-py/Cargo.toml — dropped the `openssl/vendored`
workaround from PR #370.
- crates/headroom-proxy/Cargo.toml — same dep removed.
- e2e/wrap/Dockerfile — dropped `yum install openssl-devel pkgconfig
perl-IPC-Cmd`. Comment retained explaining why.
- e2e/init/Dockerfile — same.
- Dockerfile (main) — dropped `pkg-config libssl-dev` from apt-get.
- .devcontainer/Dockerfile — dropped `pkg-config libssl-dev`.
- .github/workflows/release.yml — removed the entire before-script-linux
block (perl install probe + multi-package-manager dispatch + fail-loud
assertion). No longer needed.
# Regression gate
Three new structural tests in tests/test_release_workflows.py:
- test_no_openssl_sys_in_wheel_build_tree — runs `cargo tree -p <crate>
-i openssl-sys` for headroom-py / headroom-proxy / headroom-core. If
openssl-sys reappears (a future native-tls enabler creeping in via a
new dep), this fails AT PR TIME with an actionable message.
- test_no_native_tls_in_wheel_build_tree — same shape, native-tls is
the proximate cause.
- test_fastembed_uses_rustls_features — checks the Cargo.toml so a
future "let me bump fastembed and forget the features" doesn't
silently re-introduce OpenSSL.
Plus two cleanup gates:
- test_dockerfiles_no_longer_install_openssl_devel
- test_release_yml_does_not_install_openssl_or_perl_for_wheels
All 13 release-workflow tests pass. `make ci-precheck` PASSED.
# What this teaches us about rollouts (per user's ultrathink ask)
The 5-fix cascade exposed three meta-problems:
1. PR checks don't block merges. PR #370 had docker-init-e2e,
docker-wrap-e2e, docker-native-e2e all FAILED yet got merged.
Branch protection should require these checks. Operator action
needed (cannot fix in code).
2. Local validation is misleading. `cargo build -p headroom-py` from
the workspace root used the workspace lockfile and looked green;
CI did fresh resolution against headroom-py's manifest alone where
the feature wasn't enabled. Lesson: verify structural invariants
with `cargo tree -e features` before trusting that a build "works."
3. 6+ build surfaces with independent system-dep state. Every Cargo
change required matching updates in 6 places. The structural answer
(this PR) is to NOT depend on system OpenSSL at all. Where structural
fixes are not possible, the answer is a single shared
scripts/install-rust-build-deps.sh — but with this PR there's
nothing left to install.
PR #360's first run after the rustup-component fix exposed a deeper
issue: the devcontainer's manual rustup install in .devcontainer/Dockerfile
runs as root and leaves /usr/local/cargo root-owned. When post-create.sh
later runs `uv sync` as the `vscode` user, maturin → cargo can't write
to the registry cache:
warning: failed to write cache, path:
/usr/local/cargo/registry/index/.../aho-corasick,
error: Permission denied (os error 13)
The official ghcr.io/devcontainers/features/rust:1 feature solves this
by chowning the toolchain dirs to the runtime user (and adding them to
a `rustlang` group). It also handles component installation cleanly so
we don't need the manual `-c rustfmt -c clippy` workaround in the
container build.
Changes:
- .devcontainer/devcontainer.json: add rust feature with version=1.95.0,
profile=minimal, components=rustfmt,clippy
- .devcontainer/memory-stack/devcontainer.json: same
- .devcontainer/Dockerfile: drop manual rustup install (feature replaces
it). Keep the apt-get pkg-config + libssl-dev for openssl-sys, and
the maturin pip install.
Note: e2e/{init,wrap}/Dockerfile keep their manual rustup install
because they're stand-alone runtime images, not devcontainers.
rust-toolchain.toml at the repo root requests
`components = ["rustfmt", "clippy"]`. When `pip install -e .` invokes
maturin → cargo from inside `/workspace`, rustup auto-detects the
toolchain file and tries to add the missing components on top of the
`--profile minimal` install we did earlier. The install fails with:
info: downloading component clippy
info: rolling back changes
error: failed to install component: 'rustfmt-preview-x86_64-unknown-linux-gnu',
detected conflict: 'bin/cargo-fmt'
— rustup's auto-component install hits a `bin/cargo-fmt` conflict
inside the toolchain it just installed. The fix is to install the
required components up-front via `-c rustfmt -c clippy`, so the
toolchain matches what rust-toolchain.toml expects on first cargo run
and rustup never needs to mutate it.
Applied to: Dockerfile (main), e2e/init/Dockerfile, e2e/wrap/Dockerfile,
.devcontainer/Dockerfile. Also pinned the main Dockerfile's toolchain
from `stable` to `1.95.0` so all four images now match the lockfile
(prevents drift if rust-toolchain.toml is bumped later).
Two early failures on PR #360's first CI run:
1. validate (default/memory-stack) + validate-worktree failed at the
apt-get update step in .devcontainer/Dockerfile. The base image
mcr.microsoft.com/devcontainers/python:1-3.12-bookworm ships with
dl.yarnpkg.com configured as an apt source whose GPG key has expired:
Err:4 https://dl.yarnpkg.com/debian stable InRelease
The following signatures couldn't be verified because the public
key is not available: NO_PUBKEY 62D54FD4003F6525
apt-get returns exit 100, the whole RUN aborts before pkg-config /
libssl-dev install. The maturin refactor doesn't need yarn, so drop
/etc/apt/sources.list.d/yarn.list before apt-get update. The debian
main repo updates fine on its own.
2. workflow-validation (actionlint) failed parsing rust.yml step name
"Build wheel (single-wheel architecture: builds headroom-ai)" —
actionlint's YAML parser saw the unquoted colon inside the name as
a mapping. Quote the string.
Eliminates the dual-package architecture that was the root cause of #355.
`pip install headroom-ai` now produces ONE wheel containing both the Python
source (headroom/*.py) and the compiled Rust extension (headroom/_core.so).
No more separate `headroom-core-py` package, no more chicken-and-egg with
PyPI publication, no more wheelhouse / PIP_FIND_LINKS / composite-action
plumbing in CI.
This is the canonical pattern used by cryptography, polars, ruff,
pydantic-core, and other Rust-as-core Python packages. Honors the
"Rust as core engine" direction.
## What changed
- pyproject.toml: `[build-system]` swapped from hatchling to maturin.
`[tool.hatch.*]` deleted; `[tool.maturin]` added pointing at
`crates/headroom-py/Cargo.toml` for the cdylib. `python-source = "."`
picks up the root `headroom/` package directly (dashboard HTML
templates and other non-Python files included automatically).
- crates/headroom-py/pyproject.toml: deleted. The crate is no longer a
separate published package; its Cargo.toml stays as the cdylib build
target invoked via `[tool.maturin] manifest-path`.
- crates/headroom-py/python/: deleted (placeholder layout for the old
separate package).
## CI updates
- ci.yml: `test` / `test-extras` / `test-agno` jobs simplified — Rust
toolchain set up before `pip install -e .` (which now invokes maturin
via build-system). Removed the "build wheel + symlink .so" dance.
`build` job swapped from `python -m build` (hatch) to
`maturin build` + `maturin sdist`.
- release.yml: collapsed dual-package matrix into one. New `build-wheels`
matrix produces cross-platform wheels for cp310/11/12/13 ×
{linux x86_64, linux aarch64, macos x86_64, macos aarch64}. New
`collect-dist` aggregator merges artifacts. publish-pypi consumes the
merged dist.
- init-native-e2e.yml: dropped windows-latest from the matrix —
upstream `esaxx-rs` (/MT) and `ort-sys` (/MD) link with conflicting
MSVC C runtime libraries, so the Rust extension cannot build for
win_amd64 today. Tracked as a follow-up; not a blocker for Linux+macOS.
- headroom-e2e-setup: composite action now sets up Rust toolchain +
Swatinem/rust-cache before `pip install -e .[proxy]`.
- eval.yml, publish.yml, rust.yml: same pattern — rust toolchain before
install. rust.yml's wheels job builds from root pyproject.toml (no
more `-m crates/headroom-py/Cargo.toml`).
- e2e/init/Dockerfile, e2e/wrap/Dockerfile: install rust + maturin in
the build stage; copy `crates/` + workspace `Cargo.toml/lock` so the
install can build the extension. Dropped `HEADROOM_REQUIRE_RUST_CORE=false`
from wrap-e2e — the image now ships the full Rust core.
- Dockerfile (main): simplified — no more Layer 2/3 dance with
`headroom-core-py` install + symlink. Single `uv pip install` builds
+ installs everything.
- .devcontainer/Dockerfile: rust toolchain + libssl-dev + maturin
added so `uv sync` builds the extension inside the devcontainer.
## Lockfile + script
- uv.lock: regenerated. No `headroom-core-py` entries remain.
- scripts/build_rust_extension.sh: simplified from a symlink-into-tree
workaround to a thin wrapper around `pip install -e .`. The maturin
build-backend handles placement automatically.
## Local validation (all green on macOS aarch64)
1. Clean venv `pip install -e .` → `from headroom._core import …` works.
2. `maturin build --release` → 13.8 MB wheel, 336 files including
`headroom/_core.cpython-311-darwin.so` (32 MB cdylib) and
`headroom/dashboard/templates/dashboard.html`.
3. `pip install <wheel>` in fresh venv → import works.
4. Wheel contents verified via `unzip -l`.
5. `pytest tests/test_transforms/test_diff_compressor.py` — 29 passed.
6. `pytest tests/test_relevance.py` — 30 passed.
7. `cargo build --workspace` + `cargo test --workspace` — all green.
8. `make ci-precheck` — 176 Python tests + Rust + commitlint green.
## Migration notes
Users on `pip install headroom-ai` get the Rust core automatically
(linux + macos wheels). sdist installs require rust toolchain available
locally — pip will build via maturin.
Closes#355
Supersedes #357 (workarounds-based fix abandoned in favor of
architectural fix)
Add a default devcontainer and a compose-backed memory-stack profile, validate them in CI, and document the contributor workflow.
Also lock the memory-stack dependencies, pin related container tooling, and sync the latest healthcheck shutdown fix for stubbed memory handlers.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
