mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
18 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Tejas Chopra
|
a30305bc4c
|
ci: require ONNX Runtime >= 1.24 and fail fast when it is missing or too old (#2591)
CI installed 'onnxruntime>=1.16.0' for the Rust ort runtime. That floor is 8 minor versions too low, and the failure mode below it is a silent hang. Why 1.24: ort-sys computes ORT_API_VERSION = 17 + one per enabled api-N feature, and Cargo features are additive across the graph. fastembed 5.17.3 enables api-24, so the constant resolves to 24 and ort rejects any lower runtime. Why it hangs instead of failing: on rejection ort calls Error::new() from inside load_dylib_from_path, which already runs inside the Once that setup_api() is initialising. Building the error re-enters that Once, and std::sync::Once blocks forever on re-entry. Reproduced in isolation with a bare Session::builder() and onnxruntime 1.21.1 - killed after >1h at 0% CPU, no output; ORT_DYLIB_PATH makes no difference. With 1.24.4 the same call returns in 1.2s and the kompress parity fixtures pass 21/21. Per @RubenAAA this is not limited to old runtimes: a box that resolves no libonnxruntime at all hangs identically (0.0% CPU, threads in futex_wait_queue, nothing onnx-shaped in /proc/<pid>/maps). Any failure inside load_dylib_from_path re-enters the Once, so a pin alone cannot close it. So this adds a pre-flight to the dylib step asserting, before any test runs, that onnxruntime imports, that its minor is >= 24, and that a libonnxruntime object exists under capi/. Each failure exits 1 with an ::error:: annotation naming the cause, instead of burning the 30-minute timeout with an empty log. Verified all three branches locally (absent -> rc=1, 1.21.1 -> rc=1, 1.24.4 -> rc=0) and in CI, where it resolved onnxruntime 1.28.0 and exported the .so path. pyproject.toml is deliberately untouched: bumping the floor there makes headroom-ai[all] unsatisfiable via a pillow chain (onnxruntime>=1.24 forces pillow>=10.3.0,<12.0 while [all] requires pillow>=12.3.0). The user-facing hazard via headroom/_ort.py remains open and needs its own change. |
||
|
Tejas Chopra
|
e562d007d8
|
ci(rust): gate jobs with if: instead of a workflow-level paths filter (#2580)
Prerequisite for making `parity` a required status check on main. A workflow skipped by a top-level `paths:` filter never creates its check runs at all, so a required check sourced from it sits pending forever on any PR that misses those paths and the PR can never merge. A job skipped by `if:` still creates a check run, reports skipped, and GitHub counts skipped as success. Moves the seven path patterns verbatim from the `on:` block into a new `rust-changes` job (dorny/paths-filter) and gates all five existing jobs on `needs.rust-changes.outputs.rust == 'true'`. Named rust-changes, not changes, to avoid colliding with ci.yml's existing check. Job `name:` fields unchanged, so no check names move. schedule/workflow_dispatch force rust=true, preserving the nightly full-suite behaviour. CI spend unchanged: same jobs on the same PRs, plus a ~15s gate job on PRs that previously skipped the workflow outright. Verified by the PR itself — it edits rust.yml, which is in the path list, so it exercises the rust=true branch: rust-changes and parity both SUCCESS. Follow-up before adding parity to required checks: confirm a non-Rust PR reports parity as skipped rather than absent. ci.yml has the mirror-image problem (paths-ignore on docs) and is deliberately not addressed here. |
||
|
Tejas Chopra
|
c15e557da1
|
ci(parity): make the parity harness a real per-PR gate (#2567)
Three hardening steps on the Rust-vs-Python parity harness: - Drop the dead maturin/venv step. headroom-parity has no pyo3 dependency, so the venv requirement, the `maturin develop` rebuild, and the CI job's Python toolchain were all overhead. Verified no-op: identical report, exit 0. - Register a text_crusher comparator. 6 recorded fixtures were invisible because the transform was missing from builtin_comparators(); parity-run only walks directories it has a comparator for. All 6 match on the first run. - Promote parity to a blocking per-PR gate. Safe to harden now because parity-run exits non-zero only on a Diff, so the 65 still-stubbed fixtures report Skipped and cannot turn it red. Harness: total=176 matched=111 skipped=65 diffed=0, exit 0. Deliberately not widening the path filter to Python paths: the fixtures are frozen recordings of Python output and the harness never invokes Python, so it measures Rust-vs-snapshot and a Python edit cannot move the result. |
||
|
Parideboy
|
c46cd8f950
|
fix(core): load ONNX Runtime dynamically so headroom._core imports on non-AVX2 x86-64 (#1715)
## Description
`import headroom._core` dies with SIGILL (`Illegal instruction`) on
x86-64 CPUs without AVX2 (Pentium N4200, Celeron N4500, AMD FX 8350 —
all reported on the issue). The repo sets no `RUSTFLAGS`/`target-cpu`
anywhere, so first-party Rust code is baseline x86-64; the AVX2 code
comes from Microsoft's prebuilt ONNX Runtime, statically linked into the
extension by fastembed's `ort-download-binaries-rustls-tls` feature on
non-Windows targets. Because it is statically linked, its code is mapped
and initialized when the extension module loads — **before** the runtime
AVX2 guard from #1162 can run, which is why that fix helped Magika init
but not the import-time crash.
Fix, mirroring what Windows already does for its own reasons (DirectML
link libs): build with `ort-load-dynamic` on every platform, so ONNX
Runtime is only `dlopen`'d at first use, where the #1162 AVX2 guard
falls back to the non-ONNX detection tiers on unsupported CPUs. Since
both target blocks became identical, they are collapsed into one
platform-independent `fastembed` dependency.
To keep Magika/fastembed working out of the box on Linux/macOS, the
existing `ORT_DYLIB_PATH` auto-pin (`headroom/_ort.py`, previously
Windows-only) now resolves the pip `onnxruntime` package's shared
library on all platforms (`onnxruntime.dll` / `libonnxruntime.so*` /
`libonnxruntime*.dylib`). The pip `onnxruntime` CPU wheels use runtime
CPU dispatch, so they also work on pre-AVX2 machines — non-AVX2 users
get working ML detection instead of a crash. Without the `onnxruntime`
package, ML detection degrades gracefully to the non-ONNX tiers exactly
as it already does on Windows.
Fixes #1278
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `crates/headroom-core/Cargo.toml`: replaced the per-target `fastembed`
blocks (`ort-download-binaries-rustls-tls` on non-Windows,
`ort-load-dynamic` on Windows) with a single platform-independent
dependency on `ort-load-dynamic`, with a comment documenting both the
DirectML and the AVX2/#1278 rationale.
- `Cargo.lock`: regenerated — `ort-sys` drops its static-download
dependencies (`hmac-sha256`, `lzma-rust2`, `ureq`); no version bumps.
- `headroom/_ort.py`: `ORT_DYLIB_PATH` auto-pin extended from
Windows-only to all platforms via a small `_find_dylib` helper that
resolves the platform's shared-library name inside the pip `onnxruntime`
package.
- `tests/test_transforms/test_ort_dylib.py`: replaced the obsolete
`test_noop_on_non_windows` with Linux (versioned `.so`) and macOS
(`.dylib`) pin tests; module docstring updated.
- `docs/content/docs/configuration.mdx`: `ORT_DYLIB_PATH` row updated
from Windows-only wording to the cross-platform behavior.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ cargo fmt --all -- --check && cargo clippy --workspace -- -D warnings && cargo test -p headroom-core --lib
clean
test result: 844 passed; 0 failed; 1 ignored
$ python -m pytest tests/test_transforms/test_ort_dylib.py -q
8 passed
$ ruff check headroom/_ort.py tests/test_transforms/test_ort_dylib.py
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11 (AVX2-capable — the SIGILL itself is not
reproducible on this machine), Python 3.13, Rust 1.95.0, local checkout
branched from `upstream/main` (
|
||
|
JD Davis
|
2c9eb7c5f1
|
feat(simulators): add provider simulator service (#2014)
## Description
Adds a Rust-only `headroom-simulators` workspace crate: a deterministic
local upstream simulator service for Headroom proxy and pipeline
validation. It supplies configurable stubs plus bottled provider-shaped
responses for supported provider/path surfaces without calling real
LLMs.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Added `crates/headroom-simulators` Rust crate with library and
`headroom-simulators` binary.
- Added clean domain classification for supported surfaces: Anthropic
`/v1/messages`, OpenAI chat/responses/conversations, Bedrock
invoke/stream routes, Vertex raw/stream predict, health, and generic
fallback.
- Added JSON-configured stub matching by method, path, body substring,
and JSON pointer.
- Added bottled provider-shaped JSON, SSE, and Bedrock EventStream
responses for unconfigured requests.
- Added a container `Dockerfile` and README for local/GitHub Actions
usage.
- Added unit and HTTP integration tests for defaults, configured stubs,
SSE, Vertex, and Bedrock EventStream behavior.
- Added proxy-level simulator-backed E2E tests that run Headroom against
the simulator across Anthropic, OpenAI Chat, OpenAI Responses, OpenAI
Conversations, Bedrock invoke/converse/streaming, Vertex raw/stream
predict, and upstream health.
- Added simulator-backed provider error-path E2E coverage for OpenAI
429, Anthropic 529, Bedrock 502, and Vertex 503 responses flowing
through Headroom unchanged.
- Added Headroom-owned preflight error E2E coverage proving Bedrock
missing credentials and invalid Vertex envelopes stop inside the proxy
instead of silently falling through to the simulator/provider.
- Fixed direct Rust `headroom-core` binaries/tests on Windows so Magika
initializes ONNX Runtime via `ort::init_from` from an explicit pip
`onnxruntime` library path, with fail-fast fallback only when no safe
runtime is discoverable.
- Added a Rust CI `simulator-e2e` matrix for `ubuntu-latest`,
`macos-latest`, and `windows-latest` that runs `cargo test -p
headroom-proxy --test e2e_simulators`.
- Gated dynamic Magika `Path`/`PathBuf` imports to Windows and x86_64
macOS so Linux clippy does not see unused dynamic-ORT-only imports.
## Testing
- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
cargo fmt --all -- --check
# passed
cargo clippy --workspace -- -D warnings
# passed
$env:ORT_DYLIB_PATH=$null
cargo test -p headroom-core transforms::magika_detector::tests:: --lib
# 17 passed, 0 failed; Magika initialized from discovered pip
onnxruntime DLL
$env:ORT_DYLIB_PATH=$null
cargo test --workspace
# passed
gitleaks protect --staged --no-banner --redact
# no leaks found
gitleaks git --log-opts="headroomlabs/main..HEAD" --no-banner --redact
# 5 commits scanned; no leaks found
## Real Behavior Proof
- **Environment:** Windows PowerShell, Rust toolchain `1.95.0`, clean
worktree from `headroomlabs/main` at `
|
||
|
dependabot[bot]
|
bfe4b4d21f
|
ci: bump actions/checkout from 4 to 7 (#1414)
Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 7. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/actions/checkout/releases">actions/checkout's releases</a>.</em></p> <blockquote> <h2>v7.0.0</h2> <h2>What's Changed</h2> <ul> <li>block checking out fork pr for pull_request_target and workflow_run by <a href="https://github.com/aiqiaoy"><code>@aiqiaoy</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2454">actions/checkout#2454</a></li> <li>Bump actions/publish-immutable-action from 0.0.3 to 0.0.4 in the minor-actions-dependencies group across 1 directory by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/actions/checkout/pull/2458">actions/checkout#2458</a></li> <li>Bump flatted from 3.3.1 to 3.4.2 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/actions/checkout/pull/2460">actions/checkout#2460</a></li> <li>Bump js-yaml from 4.1.0 to 4.2.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/actions/checkout/pull/2461">actions/checkout#2461</a></li> <li>Bump <code>@actions/core</code> and <code>@actions/tool-cache</code> and Remove uuid by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/actions/checkout/pull/2459">actions/checkout#2459</a></li> <li>upgrade module to esm and update dependencies by <a href="https://github.com/aiqiaoy"><code>@aiqiaoy</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2463">actions/checkout#2463</a></li> <li>Bump the minor-npm-dependencies group across 1 directory with 3 updates by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/actions/checkout/pull/2462">actions/checkout#2462</a></li> <li>getting ready for checkout v7 release by <a href="https://github.com/aiqiaoy"><code>@aiqiaoy</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2464">actions/checkout#2464</a></li> <li>update error wording by <a href="https://github.com/aiqiaoy"><code>@aiqiaoy</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2467">actions/checkout#2467</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/aiqiaoy"><code>@aiqiaoy</code></a> made their first contribution in <a href="https://redirect.github.com/actions/checkout/pull/2454">actions/checkout#2454</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/actions/checkout/compare/v6.0.3...v7.0.0">https://github.com/actions/checkout/compare/v6.0.3...v7.0.0</a></p> <h2>v6.0.3</h2> <h2>What's Changed</h2> <ul> <li>Update changelog by <a href="https://github.com/ericsciple"><code>@ericsciple</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2357">actions/checkout#2357</a></li> <li>fix: expand merge commit SHA regex and add SHA-256 test cases by <a href="https://github.com/yaananth"><code>@yaananth</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2414">actions/checkout#2414</a></li> <li>Fix checkout init for SHA-256 repositories by <a href="https://github.com/yaananth"><code>@yaananth</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2439">actions/checkout#2439</a></li> <li>Update changelog for v6.0.3 by <a href="https://github.com/yaananth"><code>@yaananth</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2446">actions/checkout#2446</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/yaananth"><code>@yaananth</code></a> made their first contribution in <a href="https://redirect.github.com/actions/checkout/pull/2414">actions/checkout#2414</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/actions/checkout/compare/v6...v6.0.3">https://github.com/actions/checkout/compare/v6...v6.0.3</a></p> <h2>v6.0.2</h2> <h2>What's Changed</h2> <ul> <li>Add orchestration_id to git user-agent when ACTIONS_ORCHESTRATION_ID is set by <a href="https://github.com/TingluoHuang"><code>@TingluoHuang</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2355">actions/checkout#2355</a></li> <li>Fix tag handling: preserve annotations and explicit fetch-tags by <a href="https://github.com/ericsciple"><code>@ericsciple</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2356">actions/checkout#2356</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/actions/checkout/compare/v6.0.1...v6.0.2">https://github.com/actions/checkout/compare/v6.0.1...v6.0.2</a></p> <h2>v6.0.1</h2> <h2>What's Changed</h2> <ul> <li>Update all references from v5 and v4 to v6 by <a href="https://github.com/ericsciple"><code>@ericsciple</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2314">actions/checkout#2314</a></li> <li>Add worktree support for persist-credentials includeIf by <a href="https://github.com/ericsciple"><code>@ericsciple</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2327">actions/checkout#2327</a></li> <li>Clarify v6 README by <a href="https://github.com/ericsciple"><code>@ericsciple</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2328">actions/checkout#2328</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/actions/checkout/compare/v6...v6.0.1">https://github.com/actions/checkout/compare/v6...v6.0.1</a></p> <h2>v6.0.0</h2> <h2>What's Changed</h2> <ul> <li>Update README to include Node.js 24 support details and requirements by <a href="https://github.com/salmanmkc"><code>@salmanmkc</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2248">actions/checkout#2248</a></li> <li>Persist creds to a separate file by <a href="https://github.com/ericsciple"><code>@ericsciple</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2286">actions/checkout#2286</a></li> <li>v6-beta by <a href="https://github.com/ericsciple"><code>@ericsciple</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2298">actions/checkout#2298</a></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/actions/checkout/blob/main/CHANGELOG.md">actions/checkout's changelog</a>.</em></p> <blockquote> <h1>Changelog</h1> <h2>v7.0.0</h2> <ul> <li>Block checking out fork PR for pull_request_target and workflow_run by <a href="https://github.com/aiqiaoy"><code>@aiqiaoy</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2454">actions/checkout#2454</a></li> <li>Bump actions/publish-immutable-action from 0.0.3 to 0.0.4 in the minor-actions-dependencies group across 1 directory by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/actions/checkout/pull/2458">actions/checkout#2458</a></li> <li>Bump flatted from 3.3.1 to 3.4.2 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/actions/checkout/pull/2460">actions/checkout#2460</a></li> <li>Bump js-yaml from 4.1.0 to 4.2.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/actions/checkout/pull/2461">actions/checkout#2461</a></li> <li>Bump <code>@actions/core</code> and <code>@actions/tool-cache</code> and Remove uuid by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/actions/checkout/pull/2459">actions/checkout#2459</a></li> <li>upgrade module to esm and update dependencies by <a href="https://github.com/aiqiaoy"><code>@aiqiaoy</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2463">actions/checkout#2463</a></li> <li>Bump the minor-npm-dependencies group across 1 directory with 3 updates by <a href="https://github.com/dependabot"><code>@dependabot</code></a>[bot] in <a href="https://redirect.github.com/actions/checkout/pull/2462">actions/checkout#2462</a></li> </ul> <h2>v6.0.3</h2> <ul> <li>Fix checkout init for SHA-256 repositories by <a href="https://github.com/yaananth"><code>@yaananth</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2439">actions/checkout#2439</a></li> <li>fix: expand merge commit SHA regex and add SHA-256 test cases by <a href="https://github.com/yaananth"><code>@yaananth</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2414">actions/checkout#2414</a></li> </ul> <h2>v6.0.2</h2> <ul> <li>Fix tag handling: preserve annotations and explicit fetch-tags by <a href="https://github.com/ericsciple"><code>@ericsciple</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2356">actions/checkout#2356</a></li> </ul> <h2>v6.0.1</h2> <ul> <li>Add worktree support for persist-credentials includeIf by <a href="https://github.com/ericsciple"><code>@ericsciple</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2327">actions/checkout#2327</a></li> </ul> <h2>v6.0.0</h2> <ul> <li>Persist creds to a separate file by <a href="https://github.com/ericsciple"><code>@ericsciple</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2286">actions/checkout#2286</a></li> <li>Update README to include Node.js 24 support details and requirements by <a href="https://github.com/salmanmkc"><code>@salmanmkc</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2248">actions/checkout#2248</a></li> </ul> <h2>v5.0.1</h2> <ul> <li>Port v6 cleanup to v5 by <a href="https://github.com/ericsciple"><code>@ericsciple</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2301">actions/checkout#2301</a></li> </ul> <h2>v5.0.0</h2> <ul> <li>Update actions checkout to use node 24 by <a href="https://github.com/salmanmkc"><code>@salmanmkc</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2226">actions/checkout#2226</a></li> </ul> <h2>v4.3.1</h2> <ul> <li>Port v6 cleanup to v4 by <a href="https://github.com/ericsciple"><code>@ericsciple</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2305">actions/checkout#2305</a></li> </ul> <h2>v4.3.0</h2> <ul> <li>docs: update README.md by <a href="https://github.com/motss"><code>@motss</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/1971">actions/checkout#1971</a></li> <li>Add internal repos for checking out multiple repositories by <a href="https://github.com/mouismail"><code>@mouismail</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/1977">actions/checkout#1977</a></li> <li>Documentation update - add recommended permissions to Readme by <a href="https://github.com/benwells"><code>@benwells</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2043">actions/checkout#2043</a></li> <li>Adjust positioning of user email note and permissions heading by <a href="https://github.com/joshmgross"><code>@joshmgross</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2044">actions/checkout#2044</a></li> <li>Update README.md by <a href="https://github.com/nebuk89"><code>@nebuk89</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2194">actions/checkout#2194</a></li> <li>Update CODEOWNERS for actions by <a href="https://github.com/TingluoHuang"><code>@TingluoHuang</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2224">actions/checkout#2224</a></li> <li>Update package dependencies by <a href="https://github.com/salmanmkc"><code>@salmanmkc</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/2236">actions/checkout#2236</a></li> </ul> <h2>v4.2.2</h2> <ul> <li><code>url-helper.ts</code> now leverages well-known environment variables by <a href="https://github.com/jww3"><code>@jww3</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/1941">actions/checkout#1941</a></li> <li>Expand unit test coverage for <code>isGhes</code> by <a href="https://github.com/jww3"><code>@jww3</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/1946">actions/checkout#1946</a></li> </ul> <h2>v4.2.1</h2> <ul> <li>Check out other refs/* by commit if provided, fall back to ref by <a href="https://github.com/orhantoy"><code>@orhantoy</code></a> in <a href="https://redirect.github.com/actions/checkout/pull/1924">actions/checkout#1924</a></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
Rob Francis
|
32ce99e4b4
|
fix(build): enable Intel macOS pip installs via ort-load-dynamic (#1538)
## Description
Fix `pip install headroom-ai` on Intel Mac (`x86_64-apple-darwin`).
Source installs failed because `ort-sys 2.0.0-rc.12` (transitive via
`fastembed`) does not ship prebuilt ONNX Runtime binaries for that
target, causing maturin/cargo to exit during the wheel build.
This PR mirrors the existing Windows fix: build the Rust core with
`ort-load-dynamic`, pin `ORT_DYLIB_PATH` to the pip `onnxruntime` native
library at import time, and publish Intel macOS wheels from CI.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `crates/headroom-core/Cargo.toml`: use `ort-load-dynamic` for
`x86_64-apple-darwin` instead of `ort-download-binaries-rustls-tls`.
- `headroom/_ort.py`: extend the ORT dylib pin hook to Intel macOS
(`darwin` + `x86_64`), resolving `libonnxruntime*.dylib` from the pip
`onnxruntime` package.
- `.github/workflows/release.yml` and `.github/workflows/rust.yml`: add
`macos-15-intel` / `x86_64-apple-darwin` wheel matrix entries.
- `tests/test_release_workflows.py` and
`tests/test_transforms/test_ort_dylib.py`: update/add coverage for the
new target and dylib pin behavior.
- `README.md`: note that prebuilt wheels are published for Intel macOS.
## Testing
- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_transforms/test_ort_dylib.py \
tests/test_release_workflows.py::test_fastembed_uses_dynamic_ort_on_windows \
tests/test_release_workflows.py::test_build_wheels_matrix_includes_intel_macos_with_dynamic_ort -q
.......................... [100%]
10 passed in 0.18s
$ maturin build --release -o /tmp/headroom-dist
📦 Built wheel for abi3 Python ≥ 3.10 to /tmp/headroom-dist/headroom_ai-0.27.0-cp310-abi3-macosx_10_12_x86_64.whl
$ python3.11 -m venv /tmp/hr-venv && /tmp/hr-venv/bin/pip install .
Successfully built headroom-ai
Successfully installed headroom-ai-0.27.0
$ cd /tmp && /tmp/hr-venv/bin/python -c "import headroom; import headroom._core; print('ok')"
version 0.27.0
_core ok
```
## Real Behavior Proof
- Environment: macOS `x86_64-apple-darwin`, Python 3.11.5, Rust 1.95.0
- Exact command / steps: Reproduced the reported failure with `pip
install headroom-ai` (sdist build dies in `ort-sys` for
`x86_64-apple-darwin`); after this patch ran `maturin build --release`,
then `pip install .` in a clean venv, then `python -c "import
headroom._core"`.
- Observed result: Before fix, cargo/maturin exit 101 on missing ORT
prebuilts; after fix, wheel build succeeds and `headroom._core` imports
cleanly (`version 0.27.0`, `_core ok`).
- Not tested: `macos-15-intel` GitHub Actions wheel matrix row (will be
validated by CI after merge).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
- ML features (magika detection, fastembed embeddings) still require
`onnxruntime` at runtime on Intel Mac. Users should install
`headroom-ai[proxy]` or `pip install onnxruntime`; `_ort.py` auto-pins
`ORT_DYLIB_PATH` when that package is present.
- Apple Silicon (`aarch64-apple-darwin`) behavior is unchanged: it
continues to bundle ORT via `ort-download-binaries-rustls-tls`.
- Lint/mypy not re-run locally in this pass; targeted pytest +
maturin/pip install proof covers the changed surface.
---------
Co-authored-by: Bor <you@example.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
|
||
|
Patrick A
|
53d2342291
|
ci: speed up GitHub Actions — path filters, caching, timeouts, version upgrades (#620)
* ci: speed up GitHub Actions - path filters, caching, timeouts, version upgrades
Performance improvements:
- init-e2e.yml, wrap-e2e.yml: add path filters so e2e Docker builds only run when
e2e-related files change (saves ~10 min per irrelevant PR push)
- init-e2e.yml, wrap-e2e.yml: add concurrency groups to cancel superseded PR runs
- ci.yml: add pip caching to lint and build jobs
- ci.yml: cache actionlint + act binaries in workflow-validation (skip curl on hits)
- eval.yml: add pip caching to smoke-test and weekly-suite jobs
- docs.yml: add pip caching for mkdocs-material install
- rust.yml: replace cargo install --locked cargo-audit/deny with taiki-e/install-action
(prebuilt binaries; saves 2-5 min per audit run)
Bug fixes:
- docker.yml: fix actions/checkout@v6 -> @v4 (v6 does not exist; would break all
Docker builds on every release/PR touching docker paths)
Version upgrades:
- wagoid/commitlint-github-action: @v5 -> @v6
- devcontainers.yml: docker/setup-buildx-action@v3 -> @v4 (align with docker.yml)
Safety improvements:
- ci.yml: add timeout-minutes to all 13 jobs (changes, lint, build-wheel,
prefetch-model, test x4, test-extras, test-agno, commitlint, build,
workflow-validation, docker-native-e2e, windows-native-wrapper, macos-native-wrapper)
- docker.yml: add timeout-minutes to docker-build (75m), docker-manifest (20m),
promote-latest (10m)
- eval.yml: add timeout-minutes to smoke-test (30m); bump weekly-suite 60->90m
- rust.yml: add timeout-minutes to test (30m), wheels (45m), audit (20m)
Observed wall-clock impact on recent PRs:
- Init E2E and Wrap E2E were running on every single PR push regardless of content
- CI workflow was taking 12-17 min; path filters reduce unnecessary e2e runs to 0
* fix(ci): bust actionlint+act cache when workflow file changes
Static cache key 'ci-tools-actionlint-act-v1' never invalidated on
tool version updates. Switched to hashFiles('.github/workflows/ci.yml')
so the cache busts automatically whenever the download scripts are
updated to point at a newer release.
Flagged by adversarial review (Architecture + Testing/Reliability personas).
* fix(ci): add missing Dockerfile COPY paths to e2e path filters
e2e/init/Dockerfile and e2e/wrap/Dockerfile COPY files not covered
by the initial path filter set:
init-e2e: Cargo.toml, Cargo.lock, rust-toolchain.toml, uv.lock,
.claude-plugin, .github/plugin/**, plugins/headroom-agent-hooks/**
wrap-e2e: Cargo.toml, Cargo.lock, rust-toolchain.toml, uv.lock,
sdk/typescript/**, plugins/openclaw/**
Without these, a Rust toolchain bump or SDK change on a PR would
skip the e2e gate entirely, only catching it on the merge to main.
Flagged by adversarial review (Domain/Correctness persona).
* fix(devcontainer): upgrade uv to >=0.7.0 to parse uv.lock revision=3
* fix(devcontainer): set UV_SKIP_WHEEL_FILENAME_CHECK=1 in post-create.sh for gitpython wheel
* ci: bump actions/checkout and actions/setup-node to v5 (Node.js 20 EOL Jun 16)
* fix(devcontainer): export UV_SKIP_WHEEL_FILENAME_CHECK so uv run also skips wheel check
* ci: bump all GitHub Actions to latest versions (Node.js 24)
* fix(test): accept release-please-action v4 or v5 in workflow assertion
* fix(format): ruff format test_release_workflows.py
|
||
|
chopratejas
|
86177da871 |
fix(ci): yarnpkg GPG + YAML colon syntax in refactor
Two early failures on PR #360's first CI run: 1. validate (default/memory-stack) + validate-worktree failed at the apt-get update step in .devcontainer/Dockerfile. The base image mcr.microsoft.com/devcontainers/python:1-3.12-bookworm ships with dl.yarnpkg.com configured as an apt source whose GPG key has expired: Err:4 https://dl.yarnpkg.com/debian stable InRelease The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 62D54FD4003F6525 apt-get returns exit 100, the whole RUN aborts before pkg-config / libssl-dev install. The maturin refactor doesn't need yarn, so drop /etc/apt/sources.list.d/yarn.list before apt-get update. The debian main repo updates fine on its own. 2. workflow-validation (actionlint) failed parsing rust.yml step name "Build wheel (single-wheel architecture: builds headroom-ai)" — actionlint's YAML parser saw the unquoted colon inside the name as a mapping. Quote the string. |
||
|
chopratejas
|
2a91cbb4b4 |
refactor: single-wheel maturin build backend (fixes #355)
Eliminates the dual-package architecture that was the root cause of #355. `pip install headroom-ai` now produces ONE wheel containing both the Python source (headroom/*.py) and the compiled Rust extension (headroom/_core.so). No more separate `headroom-core-py` package, no more chicken-and-egg with PyPI publication, no more wheelhouse / PIP_FIND_LINKS / composite-action plumbing in CI. This is the canonical pattern used by cryptography, polars, ruff, pydantic-core, and other Rust-as-core Python packages. Honors the "Rust as core engine" direction. ## What changed - pyproject.toml: `[build-system]` swapped from hatchling to maturin. `[tool.hatch.*]` deleted; `[tool.maturin]` added pointing at `crates/headroom-py/Cargo.toml` for the cdylib. `python-source = "."` picks up the root `headroom/` package directly (dashboard HTML templates and other non-Python files included automatically). - crates/headroom-py/pyproject.toml: deleted. The crate is no longer a separate published package; its Cargo.toml stays as the cdylib build target invoked via `[tool.maturin] manifest-path`. - crates/headroom-py/python/: deleted (placeholder layout for the old separate package). ## CI updates - ci.yml: `test` / `test-extras` / `test-agno` jobs simplified — Rust toolchain set up before `pip install -e .` (which now invokes maturin via build-system). Removed the "build wheel + symlink .so" dance. `build` job swapped from `python -m build` (hatch) to `maturin build` + `maturin sdist`. - release.yml: collapsed dual-package matrix into one. New `build-wheels` matrix produces cross-platform wheels for cp310/11/12/13 × {linux x86_64, linux aarch64, macos x86_64, macos aarch64}. New `collect-dist` aggregator merges artifacts. publish-pypi consumes the merged dist. - init-native-e2e.yml: dropped windows-latest from the matrix — upstream `esaxx-rs` (/MT) and `ort-sys` (/MD) link with conflicting MSVC C runtime libraries, so the Rust extension cannot build for win_amd64 today. Tracked as a follow-up; not a blocker for Linux+macOS. - headroom-e2e-setup: composite action now sets up Rust toolchain + Swatinem/rust-cache before `pip install -e .[proxy]`. - eval.yml, publish.yml, rust.yml: same pattern — rust toolchain before install. rust.yml's wheels job builds from root pyproject.toml (no more `-m crates/headroom-py/Cargo.toml`). - e2e/init/Dockerfile, e2e/wrap/Dockerfile: install rust + maturin in the build stage; copy `crates/` + workspace `Cargo.toml/lock` so the install can build the extension. Dropped `HEADROOM_REQUIRE_RUST_CORE=false` from wrap-e2e — the image now ships the full Rust core. - Dockerfile (main): simplified — no more Layer 2/3 dance with `headroom-core-py` install + symlink. Single `uv pip install` builds + installs everything. - .devcontainer/Dockerfile: rust toolchain + libssl-dev + maturin added so `uv sync` builds the extension inside the devcontainer. ## Lockfile + script - uv.lock: regenerated. No `headroom-core-py` entries remain. - scripts/build_rust_extension.sh: simplified from a symlink-into-tree workaround to a thin wrapper around `pip install -e .`. The maturin build-backend handles placement automatically. ## Local validation (all green on macOS aarch64) 1. Clean venv `pip install -e .` → `from headroom._core import …` works. 2. `maturin build --release` → 13.8 MB wheel, 336 files including `headroom/_core.cpython-311-darwin.so` (32 MB cdylib) and `headroom/dashboard/templates/dashboard.html`. 3. `pip install <wheel>` in fresh venv → import works. 4. Wheel contents verified via `unzip -l`. 5. `pytest tests/test_transforms/test_diff_compressor.py` — 29 passed. 6. `pytest tests/test_relevance.py` — 30 passed. 7. `cargo build --workspace` + `cargo test --workspace` — all green. 8. `make ci-precheck` — 176 Python tests + Rust + commitlint green. ## Migration notes Users on `pip install headroom-ai` get the Rust core automatically (linux + macos wheels). sdist installs require rust toolchain available locally — pip will build via maturin. Closes #355 Supersedes #357 (workarounds-based fix abandoned in favor of architectural fix) |
||
|
chopratejas
|
bcef763720 |
ci: pin dtolnay/rust-toolchain@stable, set toolchain via input
The @1.95.0 git ref of dtolnay/rust-toolchain shipped action code that errors on ubuntu-latest with: failed to install component: 'clippy-preview-x86_64-unknown-linux-gnu', detected conflict: 'bin/cargo-clippy' The runner's pre-installed Rust ships cargo-clippy at $HOME/.cargo/bin, and the older action code's rustup invocation hits a path conflict when adding the clippy-preview component for 1.95.0. The @stable ref of the action has the fix; pass toolchain: 1.95.0 as input so the version stays pinned. rust-toolchain.toml continues to be the source of truth for the version (used by cargo's auto-detection); this keeps the action's install in sync. |
||
|
chopratejas
|
beec0789ed |
fix(ci): pin dtolnay/rust-toolchain to 1.95.0 to match rust-toolchain.toml
The action was set to @stable, which installs whatever the latest stable is (1.95.0 right now). Then maturin invokes cargo, which reads rust-toolchain.toml and re-resolves to "1.95.0 + clippy + rustfmt". rustup treats stable and 1.95.0 as distinct toolchain identities and refuses the second install with: failed to install component 'clippy-preview-x86_64-unknown-linux-gnu', detected conflict: 'bin/cargo-clippy' This was intermittent across the matrix (only test (3.10) tripped on the most recent run; others got lucky on cache state). Pinning the action ref to 1.95.0 makes both sides ask for the exact same toolchain identity, so the second install is a no-op and the conflict can't fire. Bump procedure stays the same: when rust-toolchain.toml's channel changes, update these refs in lock-step. Plugin manifests auto-bumped 0.11.0 -> 0.13.2 by sync-plugin-versions hook (unrelated to the workflow fix). |
||
|
chopratejas
|
d6a00ee89c |
ci: fix smart_crusher branch CI failures + add make ci-precheck pre-push gate
Five gates broke on the 2026-04-27 push of the smart_crusher branch.
Each is fixed below; the second half adds a `make ci-precheck` target
(plus an installable git pre-push hook) so the same dance never happens
again.
Failures fixed:
1. cargo fmt — 22 files had formatting drift introduced over the
stage 3c.1 work. `cargo fmt --all` reformatted them; no semantic
changes. `cargo test --workspace` still green (388 + supporting).
2. wheels job (macOS x86_64) — `fastembed -> ort -> ort-sys` does not
publish prebuilt ONNX Runtime binaries for `x86_64-apple-darwin`.
Removed that target from `.github/workflows/rust.yml`'s wheels
matrix. Apple Silicon (`aarch64-apple-darwin`) covers macOS
distribution; Intel macOS users can build from source. The matrix
now has 2 targets: linux x86_64 + macOS aarch64.
3. test-extras (relevance.py) — `tests/test_relevance.py::TestSmartCrusherIntegration`
constructs a `SmartCrusher`, which hard-imports `headroom._core`
since the python implementation was retired in stage 3c.1b. The
test-extras job didn't build the rust extension. Added the same
`maturin build + symlink` block the main `test` job uses.
4. smoke-test (eval.yml) — same root cause:
`compression_only.evaluate_ccr_lossless` instantiates a SmartCrusher.
Same fix: build the rust extension before the smoke test runs.
5. commitlint — three rules tripped:
- `subject-case` rejects PascalCase identifiers in subjects, but
the project deliberately names classes (SmartCrusher, HfTokenizer,
ContentRouter, DiffCompressor) in commit subjects. Disabled.
- `footer-leading-blank` is a warning that the wagoid action turns
into a CI failure; lines like `Module: foo.rs` in our bodies
match the conventional footer pattern and trip it. Disabled.
- `type-enum` doesn't include `parity`, but the project ships
parity-test infrastructure as its own concern (separate from
`test:`); added `parity` to the allowed types.
Pre-push verification — the prevention half:
`make ci-precheck` runs all of the above CI gates locally:
- `ci-precheck-rust`: cargo fmt --check + clippy + test --workspace.
- `ci-precheck-python`: builds the rust extension via maturin, then
runs the smart_crusher-affected python test files (185 tests across
test_transforms/, test_relevance*, test_ccr, test_acceptance,
test_critical_fixes, test_quality_retention).
- `ci-precheck-commitlint`: `npx commitlint --from origin/main --to
HEAD` against the same config CI uses. Skipped silently if npx is
not on PATH (install Node 18+ to enable).
`make install-git-hooks` (or `scripts/install-git-hooks.sh`) installs
a git pre-push hook that runs `make ci-precheck` automatically.
Bypass with `--no-verify` only when truly needed.
When new CI gates land in `.github/workflows/`, mirror them into a
`make ci-precheck-*` target. The Makefile is the local mirror of the
CI configuration; keeping them in sync is a load-bearing invariant.
Verification: `make ci-precheck` runs green on this commit.
|
||
|
chopratejas
|
3447dd6378 |
ci: replace deprecated macos-13 runner with macos-15-intel
GitHub Actions deprecated the macos-13 runner label. The validate-workflows actionlint step in CI fails because macos-13 is no longer in the available labels list. macos-15-intel is the current x86_64 macOS runner. (Bumped from macos-14 to macos-15 for arm64 was unnecessary; macos-14 is still valid and we keep it for cache-warmth.) |
||
|
chopratejas
|
0144cfba51 |
ci: fix cargo fmt + maturin action invocation
cargo fmt --check failed in CI: import order in proxy.rs (cfg(test) attributes before/after non-attr imports) and a few line-wrapping nits in e2e_real.rs. Ran cargo fmt --all to fix. maturin-action@v1 does not have a 'manifest-path' input — the action warned 'Unexpected input(s) manifest-path' and proceeded to invoke maturin from the repo root, which sees the workspace Cargo.toml with no [package] section and bails. Move -m crates/headroom-py/Cargo.toml back inside the 'args' string. |
||
|
chopratejas
|
15877fb63f |
ci: workflow permissions + comprehensive e2e tests (phase-1)
CodeQL alert #61 (CWE-275, actions/missing-workflow-permissions): add explicit `permissions: contents: read` to the rust workflow root. Defaults the GITHUB_TOKEN to read-only across all jobs, so even if the repo policy changes, this workflow stays at least-privilege. No job in this workflow needs write — wheels/audit/parity all read-only. Add real end-to-end test suite at tests/e2e_real.rs gated behind HEADROOM_E2E=1. Spawns the actual Python Headroom proxy as a subprocess, runs the Rust proxy in-process in front of it, and exercises: - health endpoints across the full chain - Anthropic non-streaming (real API call) - Anthropic streaming SSE (real API call) with chunk-level validation - OpenAI non-streaming (real API call) - X-Request-Id generation and pass-through Adds tokio-process feature for Command/Child usage. Loads .env at the repo root for API keys (does not log values). Tests skip cleanly when HEADROOM_E2E is unset, so cargo test stays fast. |
||
|
chopratejas
|
1bbbf96700 |
fix(ci): maturin manifest-path is Cargo.toml not pyproject.toml
maturin>=1.5 requires -m to point to Cargo.toml, not pyproject.toml. Fixes wheel build job failure in CI (all three matrix targets). Also switches to manifest-path: action param for cleaner workflow syntax. Applies same fix to Makefile build-wheel and develop targets. |
||
|
chopratejas
|
0414cb70e4 |
feat(rust): scaffold workspace + parity harness (phase-0)
Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |