mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
4 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
JD Davis
|
a708c0571e
|
fix(ci): prevent native detector from hanging test shards (#2996)
## Description
CI shard 4 was not merely slow: after thousands of fast tests it parked
indefinitely inside `headroom._core.detect_content_type` at 0% CPU. The
router watchdogged only the first native call and then permanently
trusted direct calls via `_detect_native_verified`. Earlier suite
activity can change ORT/native state after that first success, making a
later call deadlock until GitHub cancels the job.
This keeps every native call bounded by the existing watchdog, activates
the process-wide pure-Python circuit breaker after a timeout, restores
the test-job ceiling to 30 minutes, and removes a separate wall-clock
scheduler assertion that generated false shard-1 failures despite the
structural regression guards passing.
No issue is auto-closed by this infrastructure repair.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Removed the unsafe process-lifetime `_detect_native_verified` fast
path.
- Kept every native detection call behind the existing bounded watchdog.
- Preserved the process-wide fallback circuit breaker so only the first
wedged call consumes the watchdog budget.
- Added a success-then-hang regression test.
- Isolated native circuit-breaker state in fallback exception tests.
- Restored the CI test timeout from the temporary 90-minute diagnostic
ceiling to 30 minutes.
- Replaced the Codex scheduler's noise-sensitive p99/p50 assertion with
its meaningful absolute regression ceiling while retaining source-level
guards against the removed semaphore and nested executor.
- Corrected import order and formatting defects inherited from current
main so the synthetic merge commit passes repository-wide lint.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
Exact local shard-4 command with coverage:
2723 passed, 172 skipped, 8661 deselected in 108.40s
Focused detector/router suite:
62 passed
Codex scheduler suite:
3 passed, 1 skipped
ruff check .
All checks passed!
ruff format --check .
1411 files already formatted
mypy headroom/transforms/content_router.py
Success: no issues found in 1 source file
```
Exact-head GitHub CI on `
|
||
|
JD Davis
|
a3fe5cb65b
|
fix(onnx): enforce Rust API-24 runtime compatibility (#2979)
## Description Rust fastembed enables ORT C API 24, but the Python dependency allowed ONNX Runtime 1.23.2. Entering ort's initializer with that library deadlocks permanently instead of returning an error. Align dependency resolution where compatible wheels exist and preflight native detection where they do not. Closes #2960 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Require ONNX Runtime 1.24+ for Python 3.11+ in the proxy and voice extras. - Keep the available pre-1.24 runtime on Python 3.10 for Python ONNX consumers. - Refuse to auto-pin an incompatible runtime into the Rust extension. - Bypass native detection immediately when API 24 is unavailable, preserving Python fallback without a five-second watchdog delay or stuck native thread. - Add dependency, pinning, override, and router regression coverage. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_transforms/test_ort_dylib.py tests/test_onnx_dependency_contract.py tests/test_onnx_runtime.py tests/test_transforms/test_content_router.py 88 passed in 9.31s $ uv run ruff check headroom/_ort.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py tests/test_onnx_dependency_contract.py All checks passed! ``` ## Real Behavior Proof - Environment: macOS arm64; Python 3.13.14 and uv-managed Python 3.10.20. - Exact command / steps: run the issue's direct `headroom._core.detect_content_type` call in a subprocess with a 12-second timeout on Python 3.13; run `_detect_content` on Python 3.10 after resolving the proxy extra. - Observed result: Python 3.13 resolves ORT 1.26.0 and native detection returns `json_array`; Python 3.10 resolves ORT 1.23.2, leaves `ORT_DYLIB_PATH` unset, reports compatibility false, and immediately returns the Python `json_array` fallback. - Not tested: Linux-specific shared-object execution locally; CI's existing Linux Rust job already preflights ORT 1.24+ and exercises native tests. ## Runtime Rollout Safety - Rollout-managed feature(s): Native Rust content detection. - Minimum rollout channel: Stable/default; this is a deadlock prevention guard. - Stable/default behavior changed: Python 3.11+ installs a compatible ORT; Python 3.10 skips incompatible native detection. - Kill switch / disable path: `HEADROOM_DETECT_BACKEND=python` remains available; an explicit `ORT_DYLIB_PATH` remains an operator override. - Unsafe override required: No. - Qualification impact: Native detection stays enabled only with API-24-compatible ORT. - Rollback path: Revert this PR, which restores the old watchdog-only degradation. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable. ## Additional Notes The large lockfile diff is dependency resolution: Python 3.10 keeps ORT 1.23.2 while 3.11+ resolves 1.26.0. The functional Python change is intentionally small and keeps explicit `ORT_DYLIB_PATH` overrides working. |
||
|
Tejas Chopra
|
5771a8020e
|
fix(deps): remediate dependency CVEs and publish SBOM (#1509)
## Description
Supply-chain hardening: takes the **shipped** dependency surface from
**26 known CVEs to 0**. `pip install headroom-ai[all]` now resolves with
no known vulnerabilities (verified with Anchore syft + grype). Also
publishes a checked-in SBOM package (`sbom/`) so any user — especially
pilots running their own security review — can verify what's inside and
that we track it.
This addresses the Dependabot alerts on `main` (9 high / 4 moderate / 7
low at time of writing).
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
**Rust**
- `pyo3` 0.24 → 0.29 (GHSA-36hh-v3qg-5jq4 High, GHSA-chgr-c6px-7xpp
Med). Migrated `Python::allow_threads` → `Python::detach` (10 sites) and
added `from_py_object` to the `Clone`-deriving `#[pyclass]` types (both
required by the 0.25+ API).
- `pyo3-log` 0.12 → 0.13; `lru` 0.12 → 0.18 (GHSA-rhfx-m35p-ff5j).
**Python**
- `torch` → 2.12.1, `mem0ai` → 2.x.
- Floor-pinned transitive CVE deps via `[tool.uv]
constraint-dependencies`: `pygments>=2.20.0`,
`pydantic-settings>=2.14.2`, `gitpython>=3.1.50`, `langsmith>=0.9.0`.
- **Removed `benchmark` from the `[all]` aggregate** so the default
install is CVE-free. `lm-eval` is invoked as an external subprocess
(`python -m lm_eval`) and never imported, so it is not a true runtime
dep — it remains available via the opt-in `[benchmark]` extra. See
[Accepted Risks](#additional-notes).
**npm (build/test tooling — never shipped in the
wheel/container/published SDK)**
- `esbuild` override `>=0.28.1` in `sdk/typescript` + `plugins/openclaw`
(GHSA-g7r4-m6w7-qqqr).
- `docs/`: `@anthropic-ai/sdk` → `^0.106.0` (GHSA-p7fg-763f-g4gf),
`postcss` override to force Next.js's bundled copy ≥8.5.10
(GHSA-qx2v-qp2m-jg93); regenerated a stale `bun.lock` that carried a
**Critical** vitest/vite.
**CI**
- Pinned `pypa/gh-action-pypi-publish` `@release/v1` → `@v1.13.0`
(GHSA-vxmw-7h4f-hqxh) in `release.yml` + `publish.yml`.
**SBOM**
- New `sbom/` directory: CycloneDX 1.7 + SPDX 2.3 SBOMs, grype scan
evidence, 330-package license inventory, and a regeneration guide.
## Testing
- [ ] Unit tests pass (`pytest`) — N/A, no Python source changed
(deps/config only)
- [x] Linting passes — `cargo fmt --check` + `cargo clippy` clean on the
changed crate; 0 `.py` files changed so `ruff`/`mypy` scope is
unaffected
- [x] Type checking passes — `cargo check --workspace` (0 errors)
- [ ] New tests added — N/A (dependency bumps; covered by existing
suites)
- [x] Manual testing performed — see Real Behavior Proof
### Test Output
```text
# headroom-ai[all] product surface — the number that matters
$ grype sbom:sbom/headroom-sbom-all-extra.cdx.json
No vulnerabilities found
# full repo scan (universal lock incl. opt-in [benchmark] + dev)
$ grype sbom:sbom/headroom-sbom.cdx.json
NAME INSTALLED TYPE VULNERABILITY SEVERITY
sqlitedict 2.1.0 python GHSA-g4r7-86gm-pgqc High # [benchmark]-only, unpatchable, accepted
nltk 3.9.4 python GHSA-p4gq-832x-fm9v High # [benchmark]-only, unpatchable, accepted
# pyo3 0.29 migration — extension builds + imports + runs
$ cargo check --workspace
Finished `dev` profile [unoptimized + debuginfo] target(s)
$ maturin develop && python -c "from headroom._core import DiffCompressor, SmartCrusher; ..."
extension OK — detach + from_py_object paths exercised
# lru 0.18 — eviction path
$ cargo test -p headroom-proxy --lib drift
14 passed, 213 filtered out
# per-ecosystem npm audits
$ (cd sdk/typescript && npm audit) -> found 0 vulnerabilities
$ (cd plugins/openclaw && npm audit) -> found 0 vulnerabilities
$ (cd docs && npm audit && bun audit) -> found 0 vulnerabilities / No vulnerabilities found
```
## Real Behavior Proof
- Environment: macOS (darwin 25.4.0, arm64), Python 3.12 `.venv`, Rust
1.95 toolchain, syft 1.46.0, grype 0.115.0, bun 1.3.14, maturin 1.13.3.
- Exact command / steps: (1) `uv export --extra all --no-dev
--no-emit-project | syft → grype` for the product surface; (2) `cargo
check --workspace` + `maturin develop` + extension import/compress smoke
test; (3) `cargo test -p headroom-proxy --lib drift`; (4) `cargo fmt
--check` + `cargo clippy -p headroom-py`; (5) `npm audit` in
sdk/openclaw/docs + `bun audit` in docs.
- Observed result: `headroom-ai[all]` resolution scans clean — "No
vulnerabilities found" (179 pkgs); full/prod SBOM shows only the 2
documented accepted CVEs; pyo3 0.29 extension imports and runs (detach +
from_py_object paths exercised); drift tests 14/14 pass; cargo fmt +
clippy clean; all npm/bun audits report 0.
- Not tested: full `pytest` suite (no Python source changed);
release-profile wheel build (used dev-profile `maturin develop` for the
import proof — the extension is semantically identical).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
(`sbom/README.md`)
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective — N/A
(dependency bumps; existing suites + scans cover it)
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md — N/A (Release Please
auto-generates from the conventional commit)
## Additional Notes
**Accepted risks (the 2 residual CVEs).** Both originate solely from the
EleutherAI `lm-evaluation-harness` under the **opt-in `[benchmark]`
extra**, which Headroom invokes as a subprocess (never imports):
- `sqlitedict` CVE-2024-35515 (High) — pickle deserialization; package
abandoned (last release 2021), **no upstream fix exists**.
- `nltk` CVE-2026-54293 (High) — path traversal in `nltk.data.load()`;
affects ≤3.9.4 (current latest), **no patched release**.
Neither is in `[all]`, the published wheel, or the container. They are
documented in `sbom/README.md` and will be picked up automatically once
upstream ships fixes.
**Release/CHANGELOG:** N/A items above are because this is a
dependency/security PR with no Python source changes; CHANGELOG is
Release-Please-managed via the conventional commit message.
|
||
|
Parideboy
|
a00fb6761e
|
fix(router): degrade to pure-Python detection on native panic (#1123) (#1260)
## Description When the native (Rust) content detector panicked, the pyo3 `PanicException` (a `BaseException`, not `Exception`) escaped `_detect_content` and surfaced as an HTTP 500 instead of degrading. This catches `BaseException` (excluding control-flow exceptions) around the native call and falls back to the pure-Python regex detector, logging a single warning. Closes #1123 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/transforms/content_router.py`: wrapped the native detect call in `_detect_content` so any `BaseException` (except `KeyboardInterrupt`/`SystemExit`/`GeneratorExit`) degrades to `_regex_detect_content_type`, warning once via a module-level `_detect_panic_warned` flag. - `tests/test_transforms/test_detect_fallback_1123.py`: new regression tests for RuntimeError fallback, BaseException-panic fallback, and KeyboardInterrupt propagation. ## Testing - [x] Unit tests pass (`pytest`) - [x] New tests added for new functionality ### Test Output ```text $ pytest tests/test_transforms/test_detect_fallback_1123.py tests/test_transforms/test_content_router.py -q 54 passed ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, pytest-asyncio 1.4.0 - Exact command / steps: Monkeypatched the native detector to raise RuntimeError, a BaseException-derived fake panic, and KeyboardInterrupt, then called `_detect_content`. - Observed result: RuntimeError and the BaseException panic both degrade to a valid regex detection result; KeyboardInterrupt still propagates. 54 tests pass. - Not tested: Could not reproduce a real pyo3 panic in this build (`pyo3_runtime` is not importable here), so the fallback is exercised via simulated exceptions. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |