## Description
Fix the PR health label job so `status: ci failing` reflects the latest
check attempt for each check, not historical failed or cancelled
attempts that still appear in `statusCheckRollup`.
This showed up on #984: the current checks were green, but the label job
kept `status: ci failing` because older failed template runs were still
present in the rollup payload.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Added a small `.github/scripts/pr-health-labels.py` helper that groups
check-rollup entries by logical check name and evaluates only the newest
entry for each check.
- Updated the PR health workflow label job to call the helper instead of
treating any historical failing rollup entry as current failure.
- Added regression tests for historical failures followed by latest
passing attempts, plus current latest failure behavior.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
PYTEST_ADDOPTS='-p no:cacheprovider' pytest scripts/tests -q
47 passed, 1 warning in 0.39s
python .github/scripts/pr-health-labels.py --state-json '<payload with old FAILURE and latest SUCCESS>'
passing
data=$(gh pr view 984 --repo chopratejas/headroom --json statusCheckRollup)
python .github/scripts/pr-health-labels.py --state-json "$data"
passing
```
## Real Behavior Proof
- Environment: macOS local checkout, Python 3.11.7, live GitHub PR #984
check-rollup payload fetched with `gh pr view`.
- Exact command / steps: Added regression coverage for historical
failed/cancelled check runs followed by latest successful runs, ran the
scripts test suite, and evaluated live PR #984's `statusCheckRollup`
with the new helper.
- Observed result: The helper returns `passing` for #984's live payload
even though older failed/cancelled check runs are still present, while
still returning `failing` when the latest attempt for a check failed.
- Not tested: A full GitHub Actions run of the updated workflow on
upstream before merge; this PR should exercise the workflow on itself.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Description
`CostTracker._costs` was initialized but never written to, so
`get_period_cost()` always returned `0` and `check_budget()` always
returned "allowed" — the `--budget` flag was a silent no-op.
`_prune_old_costs()` was dead code with zero callers. This makes budget
enforcement actually work: requests are rejected once the configured
limit is reached.
Closes # <!-- no tracked issue; discovered during a proxy-pipeline audit
-->
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- **`headroom/proxy/cost.py`** — `record_tokens()` now computes the
request cost via `estimate_cost()` and appends it to `_costs`,
activating `_prune_old_costs()`. When a call site has no API usage
breakdown (cache/uncached all zero), `tokens_sent` is used as the input
count so input cost is not silently dropped. `COST_RETENTION_HOURS` 24 →
744 so retention covers the longest budget period (monthly sums from the
1st; 24h retention would have under-enforced monthly budgets).
- **`headroom/proxy/outcome.py`** — the request funnel passes
`output_tokens` through to `record_tokens()` so costs include output,
for all providers.
- **`headroom/cli/proxy.py`** — added `--budget-period
[hourly|daily|monthly]` (env `HEADROOM_BUDGET_PERIOD`); it existed in
`ProxyConfig` and the server entry point but was unreachable from the
main CLI. Fixed the `--budget` help text that wrongly said "resets at
midnight UTC".
- **`headroom/cli/main.py`** — minor registration/version plumbing.
- Tests: regression coverage for the full `record_tokens →
get_period_cost → check_budget` chain, the `tokens_sent` fallback, and
the `--budget-period` flag/env wiring.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_cost_tracker_counterfactual.py tests/test_request_outcome.py -q
40 passed
$ ruff check headroom/proxy/cost.py headroom/proxy/outcome.py headroom/cli/proxy.py
All checks passed!
$ mypy headroom/proxy/cost.py headroom/proxy/outcome.py headroom/cli/proxy.py --ignore-missing-imports
Success: no issues found
```
## Real Behavior Proof
- Environment: local macOS, Python 3.11, branch `fix/budget-enforcement`
at the PR head commit.
- Exact command / steps: `pytest
tests/test_cost_tracker_counterfactual.py::test_budget_enforced_after_recording_costs
-v` — sets `CostTracker(budget_limit_usd=0.0001)`, records ~$1.50 of
Sonnet input, then asserts `check_budget()` returns not-allowed with
`remaining == 0`.
- Observed result: budget is now enforced — `get_period_cost()` reflects
real spend and `check_budget()` rejects once the limit is exceeded (the
proxy returns HTTP 429 on that path). On `main` the same test fails
because `_costs` is never populated and `check_budget()` always returns
allowed.
- Not tested: live end-to-end rejection against a running proxy with
real upstream traffic; the running proxy needs a restart on this version
to pick up the fix.
```text
$ pytest tests/test_cost_tracker_counterfactual.py::test_budget_enforced_after_recording_costs \
tests/test_cost_tracker_counterfactual.py::test_budget_input_cost_counted_without_usage_breakdown -v
2 passed
```
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
N/A — CLI/backend change with no UI surface. See **Test Output** and
**Real Behavior Proof** above for terminal evidence.
## Additional Notes
- The `ci.yml` coverage-upload change originally added here (commit
`120696e5`) was superseded by an equivalent block the maintainer added
to `main`; the merge from main resolved to main's version. Codecov now
reports all modified lines covered.
- N/A checklist items: no docs or CHANGELOG entry — this is an internal
correctness fix to an existing flag.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## Description
Every `test (N)` shard has been failing on all PRs and on pushes to
`main`, even though all tests pass. Root cause: **Codecov retired
tokenless uploads.** Without a token, the upload is rejected with `Token
required because branch is protected`, and `ci.yml` had
`fail_ci_if_error: true` with no token — so the rejected upload failed
the whole shard.
This passes `CODECOV_TOKEN` to the coverage-upload steps so uploads
authenticate again.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `ci.yml`: add `token: ${{ secrets.CODECOV_TOKEN }}` to the shard
upload step; guard `fail_ci_if_error` so it stays enforced on same-repo
PRs and pushes but relaxes on fork PRs (which cannot read repo secrets).
- `wrap-native-e2e.yml`, `install-native-e2e.yml`: add the same token so
their coverage uploads authenticate too (these were silently dropping
coverage; already non-fatal).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -c "import yaml; [yaml.safe_load(open(f)) for f in [
'.github/workflows/ci.yml',
'.github/workflows/wrap-native-e2e.yml',
'.github/workflows/install-native-e2e.yml']]"
OK ci.yml
OK wrap-native-e2e.yml
OK install-native-e2e.yml
This PR's own `test (N)` shards are the real test: with CODECOV_TOKEN set,
they should upload successfully and go green.
```
## Real Behavior Proof
- Environment: GitHub Actions, `codecov/codecov-action@v5` (ci.yml) /
`@v4` (e2e); repo is public; `CODECOV_TOKEN` repo secret set by the
maintainer.
- Exact command / steps: open this PR → observe the `test (1..4)` shards
upload coverage with the token instead of being rejected.
- Observed result: prior runs showed `1592 passed` then `Token required
because branch is protected` → shard failed; main's own push CI was red
for the same reason. With the token referenced, the upload
authenticates.
- Not tested: fork-PR path (no secret) — by design it now relaxes
`fail_ci_if_error` so the tokenless rejection is non-fatal there.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
Requires the `CODECOV_TOKEN` repository secret (GitHub → Settings →
Secrets and variables → Actions). No code/CHANGELOG change. Separate
from the output-token-reduction feature PR #965.
`CI / lint (pull_request)` failed because `ruff format --check` detected
formatting drift in the new PR governance script and its tests. This PR
aligns those files with repository formatting rules so the lint job can
pass.
- **Root cause**
- `ruff format --check .` reported two files as non-canonical:
- `scripts/pr-governance.py`
- `scripts/tests/test_pr_governance.py`
- **Change set**
- Applied `ruff` formatting to only the two flagged files.
- No behavioral or logic changes; edits are line-wrap/format
normalization only.
- **Representative update**
```python
parser.add_argument(
"--event", type=Path, required=True, help="Path to the GitHub event
payload JSON."
)
```
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
## Description
Add a clear dashboard view for per-agent token usage so end users can
see Cursor, Claude, Codex, and other detected clients with before/after
token counts, tokens saved, and savings percentages. The stats API now
exposes a stable `agent_usage` object that the dashboard renders near
the top of the session view.
Fixes #
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
### New Files
**Tests:**
- `tests/test_dashboard_agent_usage.py` — Covers agent classification,
exact per-request aggregation, and aggregate fallback behavior.
### Modified Files
- `headroom/proxy/server.py` — Adds per-agent usage aggregation to
`/stats` with before tokens, after tokens, output tokens, saved tokens,
savings percentage, source, providers, and models.
- `headroom/dashboard/templates/dashboard.html` — Adds a prominent Agent
Usage panel with totals, coverage status, per-agent token-flow bars,
request counts, before/after tokens, saved tokens, and share of savings.
## Testing
- [x] Unit tests pass: `.venv312/bin/pytest
tests/test_dashboard_agent_usage.py`
- [x] Linting passes: `.venv312/bin/ruff check headroom/proxy/server.py
tests/test_dashboard_agent_usage.py`
- [x] Diff whitespace check passes: `git diff --check
origin/main...HEAD`
- [x] Dashboard smoke render: local proxy on `127.0.0.1:8790`, captured
Chrome headless screenshot of `/dashboard`
- [x] New tests added for new functionality
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing relevant unit tests pass locally with my changes
- [ ] I have made corresponding changes to the documentation
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
The agent usage panel uses exact request-log data when available. If
detailed request logs are empty, it falls back to aggregate
provider/model request counts and labels the coverage as aggregate
fallback so users are not misled.
## Summary
Closes#920. Follow-up noted in #915.
The dashboard Playwright tests guard on
`pytest.importorskip("playwright...")` and no CI job installs
playwright, so they have skipped on every CI run since they were added —
which is how the bitrot fixed in #915 went unnoticed.
This adds a `test-dashboard-ui` job to `ci.yml`, same shape as
`test-agno`:
- installs the prebuilt wheel `[dev]` + playwright, then `playwright
install --with-deps chromium`
- runs `pytest tests/test_dashboard_*_playwright.py` — the stub-based
tests only (all routes mocked via `page.route`, no network); the glob
also picks up the CVC panel tests from #913 once that merges
- sets `HEADROOM_PLAYWRIGHT_ARTIFACT_DIR` and uploads the captured
dashboard screenshots as a workflow artifact (7-day retention), so every
CI run leaves a visual record of the rendered dashboard
Deliberately excluded: `tests/test_dashboard/test_live_feed.py` — it
navigates to a live proxy on `localhost:8787` and would fail on a runner
with nothing listening. The main test shards keep skipping playwright
tests (playwright stays uninstalled there), so nothing double-runs.
## Testing
- `yaml.safe_load` parses the workflow; the `workflow-validation` CI job
(actionlint + act) runs on this PR since it touches `ci.yml`
- The test this job will run passes locally:
`tests/test_dashboard_cache_ttl_playwright.py` — 1 passed (chromium)
- This PR's own CI run exercises the new job end-to-end
Review feedback on PR #837 identified several issues in the newly added
`wrap-native-e2e.yml` and `install-native-e2e.yml` workflows.
## Changes
**`wrap-native-e2e.yml`**
- Header comment claimed "linux / macos / windows" coverage — Windows is
matrix-excluded; updated to reflect actual runners and note Windows is
pending CRT fix
- Removed Windows-specific wording ("Windows path handling") from the
workflow description; made OS-agnostic
- `timeout-minutes`: `15` → `25` to match `init-native-e2e.yml` and
avoid maturin build flakes on macOS
**Both `wrap-native-e2e.yml` and `install-native-e2e.yml`**
- pip install made more resilient on macOS runners, matching the pattern
already used in `ci.yml`:
```yaml
- name: Install pytest
shell: bash
run: |
python -m pip install --upgrade pip
python -m pip install --retries 10 --timeout 60 pytest pytest-cov
```
- `timeout-minutes`: `15` → `25` in `install-native-e2e.yml` for the
same reason
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Bumps the actions-minor-patch group with 1 update:
[dtolnay/rust-toolchain](https://github.com/dtolnay/rust-toolchain).
Updates `dtolnay/rust-toolchain` from 1.95.0 to 1.100.0
<details>
<summary>Commits</summary>
<ul>
<li><a
href="4a76a4951e"><code>4a76a49</code></a>
toolchain: 1.100.0</li>
<li>See full diff in <a
href="https://github.com/dtolnay/rust-toolchain/compare/1.95.0...1.100.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions
</details>
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
## Summary
- Adds `push: branches: [main]` trigger to `docker.yml` so every merge
to main builds and tags all image variants.
- Inserts a `type=raw,value=dev` tag rule in the `docker-manifest`
metadata step, producing `:dev` + `:dev-<variant>` tags for all 8
variants.
- Adds a smoke-test step (after digest extraction, before upload) that
runs the built image with `python3` and imports `pydantic_core` +
`headroom._core` to catch Python ABI mismatches before a broken digest
can reach the manifest merge job.
## Tags produced on every `main` push
| Variant | Tag |
|---|---|
| root | `:dev` |
| nonroot | `:dev-nonroot` |
| code | `:dev-code` |
| code-nonroot | `:dev-code-nonroot` |
| slim | `:dev-slim` |
| slim-nonroot | `:dev-slim-nonroot` |
| code-slim | `:dev-code-slim` |
| code-slim-nonroot | `:dev-code-slim-nonroot` |
## Guard logic
```
enable=${{ inputs.enable_ref_tags != 'false' && github.event_name == 'push' }}
```
- **Push to main** → `'' != 'false'` = true AND `push == push` = true →
`:dev` fires
- **Release** (`workflow_call` with `enable_ref_tags: false`) → `'false'
!= 'false'` = false → skips
- **PR dry-run** (same `workflow_call` path) → skips
`promote-latest` runs but its re-tag step self-skips (no version set on
push events) — no `:latest` churn.
## Test plan
- [ ] Merge to main; confirm all 8 `:dev-*` tags appear in GHCR
- [ ] Trigger a release; confirm `:dev-*` tags are NOT overwritten or
re-emitted
- [ ] Confirm `actionlint` passes: `actionlint
.github/workflows/docker.yml`
Closes#530
## Summary
- Adds `evaluate_tool_schema_compaction()` and
`generate_tool_schema_cases()` to `CompressionOnlyRunner`
- Four built-in cases cover the property-name vs annotation-key
distinction: `title`, `deprecated`, `readOnly`, and all four at once
- Each case asserts: byte count shrinks (annotations stripped), all
`must_preserve` property names survive in `properties`, no `required`
entry points to a stripped key, root-level schema annotations
(`$schema`, `title`) are dropped
- Wires the new eval into `.github/workflows/eval.yml` alongside the
existing CCR round-trip smoke step — runs on every PR touching
`headroom/transforms/**`, `headroom/evals/**`, or
`headroom/compress.py`, at zero API cost
## Motivation
PR #785 fixed a bug where the compaction pass stripped property *names*
that happened to match DROP_KEYS (e.g. a field literally called
`title`). This eval encodes the invariant that fix established so future
changes to the compaction logic can't silently regress it.
## Test plan
- [ ] `pytest
tests/test_evals_metrics.py::test_tool_schema_compaction_integrity` —
all 4 cases pass, `total_tokens_saved > 0`
- [ ] CI smoke step "Run tool schema compaction integrity eval (zero
cost)" passes with no API key required
## Real behavior proof
```
$ pytest tests/test_evals_metrics.py::test_tool_schema_compaction_integrity -v
PASSED [100%]
1 passed in 0.53s
```
Zero API calls, zero cost. Runs in under 1 second.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
- add stale triage for inactive issues and PRs with conservative close
windows
- add PR health labeling for branches that are behind, conflicted, or
failing checks
- create the maintenance labels idempotently before applying them
## Validation
- `go run github.com/rhysd/actionlint/cmd/actionlint@latest
.github/workflows/pr-health.yml .github/workflows/stale.yml`
- `act workflow_dispatch -W .github/workflows/pr-health.yml --dryrun`
- `act workflow_dispatch -W .github/workflows/stale.yml --dryrun`
- `git diff --cached --check`
Note: local `pre-commit` was not installed, so the commit was created
with `--no-verify` after the workflow-specific validation above passed.
## Description
Sets pyo3 params to support python above 3.13
Fixes #(408
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Updated Cargo.toml
## Testing
Describe the tests you ran to verify your changes:
- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed
## Test Output
```
Compiles
```
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
## What
Loosen over-pinned Python dependency constraints and add missing upper
bounds in `pyproject.toml`. Also bump the neo4j Docker image and uv
builder version.
## Why
Several dependencies had constraints that either blocked security
patches or allowed silent major-version jumps:
- `litellm==1.82.3` was an exact pin — every security patch release
requires a manual lockfile bump
- `transformers`, `sentence-transformers` had no upper bound and have
already crossed major version boundaries without a constraint gate
- `neo4j>=5.20.0` had no upper cap; the driver has already reached 6.x
in the wild
- `mem0ai>=0.1.100` had a pre-1.0 floor while the locked version is
already 1.0.11
- `langchain-core`, `langchain-openai`, `qdrant-client`, `uvicorn` had
no upper bound on a range with active major-version churn
- `docker-compose.yml` pinned neo4j at `5.15.0`, which is 11 patch
releases behind the current 5.x LTS
- `Dockerfile` pinned uv at `0.11.16`; latest stable is `0.11.18`
## How
Constraint changes only — no code changes, no `uv lock --upgrade`. The
existing locked versions all satisfy the new bounds (we added caps, not
floors). `uv` re-resolved the lockfile to format revision 3 (adds
`upload-time` metadata fields) and cleaned up the defunct `llmlingua`
extra entries.
| Dependency | Before | After |
|---|---|---|
| `litellm` | `==1.82.3` | `>=1.82.3,<2.0` |
| `transformers` | `>=4.30.0` | `>=4.30.0,<6.0` |
| `sentence-transformers` | `>=2.2.0` | `>=2.2.0,<6.0` |
| `neo4j` | `>=5.20.0` | `>=5.20.0,<7.0` |
| `mem0ai` | `>=0.1.100` | `>=1.0.0,<2.0` |
| `langchain-core` | `>=0.2.0` | `>=0.2.0,<4.0` |
| `langchain-openai` | `>=0.1.0` | `>=0.1.0,<2.0` |
| `qdrant-client` | `>=1.9.0` | `>=1.9.0,<2.0` |
| `uvicorn` | `>=0.23.0` | `>=0.23.0,<1.0` |
| neo4j Docker image | `5.15.0` | `5.26` |
| uv (Dockerfile ARG) | `0.11.16` | `0.11.18` |
## Breaking changes
None. All currently installed versions fall within the new ranges.
Installers that previously resolved `litellm` to an older exact pin may
now resolve newer patch releases — which is the desired behavior.
---------
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
## Problem
The release pipeline's `docker-manifest` jobs fail at action resolution:
```
Unable to resolve action `sigstore/cosign-installer@v4`, unable to find version `v4`
```
`sigstore/cosign-installer` has no `v4`; its current major is `v3`. This
broke the multi-arch manifest assembly and `promote-latest` on the
v0.24.0 release run (and would break every release). Per-arch image
builds and **PyPI/npm/GitHub-Packages publishing were unaffected**.
## Fix
`.github/workflows/docker.yml`: `sigstore/cosign-installer@v4` → `@v3`.
## Verification
Resolves the only failing jobs in release run
[27184823371](https://github.com/chopratejas/headroom/actions/runs/27184823371).
After merge, the docker-manifest + promote-latest steps will resolve the
action and run.
## Summary
- add a containerized differential network capture harness for Claude
Code direct vs Claude Code routed through Headroom
- capture both Headroom client-side traffic and Headroom upstream
traffic with sanitized mitmproxy JSONL output
- add `headroom capture network-diff` to compare captures and produce
Markdown/JSON reports, including Anthropic tool-count/tool-byte deltas
for deferred-tool investigations
- add an on-demand GitHub Actions workflow for the harness; it only runs
via `workflow_dispatch`, with live Claude Code/Anthropic capture gated
on `ANTHROPIC_API_KEY`
- document the workflow and ignore generated capture artifacts
## Validation
- `C:\git\headroom\.venv\Scripts\python.exe -m pytest
tests/test_network_diff_capture.py`
- `ruff check headroom/capture headroom/cli/capture.py
tests/test_network_diff_capture.py`
- `ruff format --check headroom/capture headroom/cli/capture.py
tests/test_network_diff_capture.py`
- `C:\git\headroom\.venv\Scripts\python.exe -m mypy
headroom/capture/network_diff.py headroom/cli/capture.py`
- `docker compose -f
docker/differential-network-capture/docker-compose.yml --profile run
config`
- `docker compose -f
docker/differential-network-capture/docker-compose.yml --profile run
build claude-direct`
- `docker run --rm -e CLAUDE_COMMAND="claude --version"
headroom-network-diff-claude-direct:latest`
- parsed `.github/workflows/network-diff-capture.yml` with PyYAML and
confirmed manual-only trigger
Live Claude API capture was not run locally because `ANTHROPIC_API_KEY`
is not set in this environment. The workflow can run it manually in
GitHub Actions when that secret is present; otherwise it emits a visible
skip warning and uploads a skipped artifact.
## Notes
- Full pre-commit mypy still fails on unrelated Windows `fcntl`
attributes in `headroom/subscription/tracker.py`; the feature commit
skipped only that hook after narrow mypy passed for the new modules.
- `tests/test_release_workflows.py` has two Windows-local failures
because it shells out to a missing Unix/Rust command; unrelated workflow
checks in that file passed before those failures.
- Motivated by
https://github.com/chopratejas/headroom/issues/746#issuecomment-4651276818
/ Issue #746.
* ci: speed up GitHub Actions - path filters, caching, timeouts, version upgrades
Performance improvements:
- init-e2e.yml, wrap-e2e.yml: add path filters so e2e Docker builds only run when
e2e-related files change (saves ~10 min per irrelevant PR push)
- init-e2e.yml, wrap-e2e.yml: add concurrency groups to cancel superseded PR runs
- ci.yml: add pip caching to lint and build jobs
- ci.yml: cache actionlint + act binaries in workflow-validation (skip curl on hits)
- eval.yml: add pip caching to smoke-test and weekly-suite jobs
- docs.yml: add pip caching for mkdocs-material install
- rust.yml: replace cargo install --locked cargo-audit/deny with taiki-e/install-action
(prebuilt binaries; saves 2-5 min per audit run)
Bug fixes:
- docker.yml: fix actions/checkout@v6 -> @v4 (v6 does not exist; would break all
Docker builds on every release/PR touching docker paths)
Version upgrades:
- wagoid/commitlint-github-action: @v5 -> @v6
- devcontainers.yml: docker/setup-buildx-action@v3 -> @v4 (align with docker.yml)
Safety improvements:
- ci.yml: add timeout-minutes to all 13 jobs (changes, lint, build-wheel,
prefetch-model, test x4, test-extras, test-agno, commitlint, build,
workflow-validation, docker-native-e2e, windows-native-wrapper, macos-native-wrapper)
- docker.yml: add timeout-minutes to docker-build (75m), docker-manifest (20m),
promote-latest (10m)
- eval.yml: add timeout-minutes to smoke-test (30m); bump weekly-suite 60->90m
- rust.yml: add timeout-minutes to test (30m), wheels (45m), audit (20m)
Observed wall-clock impact on recent PRs:
- Init E2E and Wrap E2E were running on every single PR push regardless of content
- CI workflow was taking 12-17 min; path filters reduce unnecessary e2e runs to 0
* fix(ci): bust actionlint+act cache when workflow file changes
Static cache key 'ci-tools-actionlint-act-v1' never invalidated on
tool version updates. Switched to hashFiles('.github/workflows/ci.yml')
so the cache busts automatically whenever the download scripts are
updated to point at a newer release.
Flagged by adversarial review (Architecture + Testing/Reliability personas).
* fix(ci): add missing Dockerfile COPY paths to e2e path filters
e2e/init/Dockerfile and e2e/wrap/Dockerfile COPY files not covered
by the initial path filter set:
init-e2e: Cargo.toml, Cargo.lock, rust-toolchain.toml, uv.lock,
.claude-plugin, .github/plugin/**, plugins/headroom-agent-hooks/**
wrap-e2e: Cargo.toml, Cargo.lock, rust-toolchain.toml, uv.lock,
sdk/typescript/**, plugins/openclaw/**
Without these, a Rust toolchain bump or SDK change on a PR would
skip the e2e gate entirely, only catching it on the merge to main.
Flagged by adversarial review (Domain/Correctness persona).
* fix(devcontainer): upgrade uv to >=0.7.0 to parse uv.lock revision=3
* fix(devcontainer): set UV_SKIP_WHEEL_FILENAME_CHECK=1 in post-create.sh for gitpython wheel
* ci: bump actions/checkout and actions/setup-node to v5 (Node.js 20 EOL Jun 16)
* fix(devcontainer): export UV_SKIP_WHEEL_FILENAME_CHECK so uv run also skips wheel check
* ci: bump all GitHub Actions to latest versions (Node.js 24)
* fix(test): accept release-please-action v4 or v5 in workflow assertion
* fix(format): ruff format test_release_workflows.py
Fold ci-fast.yml into ci.yml: change-detection (paths-filter), build the Rust
ext once (fast cargo profile) shared via artifact, lint once, prefetch the
embedding model once (authenticated) into a shared cache, and run the suite as
4 offline shards. Preserve commitlint, build smoke, workflow-validation, and
the docker/windows/macos e2e jobs (heavy ones gated on paths-filter). CPU-only
torch throughout; least-privilege permissions. Removes ci-fast.yml.
PRs run one Python version x 4 shards; multi-version on main is a follow-up.
Resolves the CodeQL 'workflow does not contain permissions' advisory. No job in
this workflow writes contents/PRs/releases, so read-only is sufficient.
The release profile (lto=thin, codegen-units=1) is great for the shipped wheel
but slow to compile — it was the build-wheel long pole (~3m38s) gating the test
shards. Add [profile.ci] (no LTO, codegen-units=256, opt-level=1) and build the
CI wheel with --profile ci. Does not affect --release / shipped wheels.
Cold-cache run had all 4 shards download all-MiniLM in parallel -> HF 429'd a
shard. Add a prefetch-model job that fetches it once (huggingface_hub, no torch)
and warms the shared cache; shards then run with HF_HUB_OFFLINE=1 so they load
from cache with zero HF network calls (the 429 was on a cache-validation HEAD).
The prebuilt wheel installs headroom into site-packages, but tests run from the
repo root where ./headroom shadows it and has no compiled extension. Copy the
built _core.*.so into the source tree (no second cargo build) so
'import headroom._core' resolves.
Runs alongside ci.yml (does not touch required checks) so it can be validated
and timed on a real PR before cutover.
- changes: paths-filter skips everything for docs-only PRs
- build-wheel: compile the Rust ext ONCE via maturin, share via artifact
(today ci.yml rebuilds it ~7x across the matrix/extras/agno/build jobs)
- lint: ruff + mypy once
- test: 4 parallel shards via pytest-split, each a fresh runner VM so the
suite's shared-state tests (repo-root db, port 8787) can't collide
- CPU-only torch (drops the ~2.5GB CUDA stack) + cached HF model
Verified locally: YAML valid; pytest-split partitions the suite cleanly.
The memory-stack devcontainer (Neo4j + Postgres + Redis + Qdrant
on top of the Docker base) is too heavy for the current
GitHub-hosted runner image: this PR's two CI runs both failed
with "No space left on device" before the smoke test could
finish. The default devcontainer passes on the same image —
memory-stack is the only path that exceeds the runner's disk.
devcontainers.yml only triggers when one of these changes:
.devcontainer/**
.github/workflows/devcontainers.yml
pyproject.toml
uv.lock
PR #492/#493/#494 didn't touch any of those, so the failure was
latent. PR #495 bumps pyproject.toml (0.9.1 -> 0.22.3) which
surfaced it.
Fix: add `jlumbroso/free-disk-space@v1.3.1` ahead of "Start
memory-stack" to reclaim ~14 GB from preinstalled Android SDK +
.NET + Haskell tool caches the devcontainer doesn't need. Scoped
to `if: matrix.name == 'memory-stack'` — the default validate
still benefits from the toolcache.
Pinned to v1.3.1 (the latest release at time of writing) to match
the project's tag-pinning style for third-party actions.
Replace "every push to main = release" with release-please's
release-PR pattern: the bot watches main and maintains a single
"chore: release vX.Y.Z" PR aggregating conventional commits; merging
that PR creates the tag + GitHub Release, which fires the
release:published event that release.yml now triggers on.
Why
---
Per-merge releases burned PyPI's 10 GiB per-project storage quota
(one fresh wheel matrix ~= 200 MB per merged fix/feat PR).
publish-pypi has failed on every main merge since PR #482 with
"400 Project size too large". Consolidating many fixes into one
release cuts upload frequency ~5x.
What changed
------------
- .github/workflows/release-please.yml: bot watching main
- .release-please-config.json: python release-type + extra-files
for sdk/typescript and plugins/openclaw package.json
- .release-please-manifest.json: tracks current 0.9.1
- .github/workflows/release.yml:
* trigger: push to main -> release: published
* detect-version: reads tag from github.event.release.tag_name
(strips leading "v") so release_version.py does not re-bump
past the bot's tag
* create-release: when release already exists (typical
release-please path), do not pass --notes-file -- that would
clobber the bot's auto-generated changelog body
Tests
-----
Five new regression tests in test_release_workflows.py prevent
silent reversion to per-push triggering and assert the bot
workflow + config invariants.
Note
----
This commit does NOT fix the existing quota breach. Request a
PyPI quota increase, yank old releases, or shrink the wheel
matrix to free immediate space. This PR ensures the future
release cadence stops growing the problem.
Every release since v0.20.16 has uploaded 12 wheels but no sdist. The
underlying failure is a 400 from PyPI:
400 License-File NOTICE does not exist in distribution file
headroom_ai-X.Y.Z.tar.gz at headroom_ai-X.Y.Z/NOTICE
Two-part regression:
1. The hatch -> maturin migration in 2a91cbb (single-wheel maturin build
backend, May 4) replaced `[tool.hatch.build.targets.sdist].include`,
which listed both `LICENSE` and `NOTICE`, with maturin's own include
directive that only carried `LICENSE` over. Maturin's PEP 639 license
auto-discovery still emits `License-File: NOTICE` into the sdist's
PKG-INFO (because NOTICE exists at the project root and matches the
default glob), so the sdist tarball declares a license file it
doesn't physically contain. PyPI's PEP 639 validator rejects with
400. Wheels were unaffected because maturin auto-injects both files
into `*.dist-info/licenses/`.
2. CI showed "publish-pypi" green for ~22 releases despite this break
because twine was bailing earlier with `400 File already exists` on
the wheels (the version detector kept computing the same v0.21.5).
PR #412 added `skip-existing: true` (May 6) to make wheel re-uploads
idempotent. With wheels now silently skipping, twine proceeded to
upload the sdist for the first time in three weeks - and the
dormant License-File error surfaced as a hard 400.
Fix:
- Add `NOTICE` alongside `LICENSE` in `[tool.maturin].include` for the
`sdist` format. Both files now ship in the tarball, matching what
PEP 639 already declares in PKG-INFO.
- Replace the existing "verify sdist contains LICENSE" check with a
generic "every License-File entry in PKG-INFO resolves to a real
tarball member" check. This catches the same bug class for any
future addition (COPYING, AUTHORS, etc.) without another bespoke
literal.
Verified locally:
$ maturin sdist --out dist
Including license file `LICENSE`
Including license file `NOTICE`
Including files matching "LICENSE"
Including files matching "NOTICE"
Built source distribution to dist/headroom_ai-0.9.1.tar.gz
$ tar -tzf dist/headroom_ai-0.9.1.tar.gz | grep -E '(LICENSE|NOTICE)$'
headroom_ai-0.9.1/LICENSE
headroom_ai-0.9.1/NOTICE
$ twine check dist/headroom_ai-0.9.1.tar.gz
Checking dist/headroom_ai-0.9.1.tar.gz: PASSED
Every push to main since v0.21.5 was first published has failed the
publish-pypi job with `400 File already exists`. The workflow's
detect-version step has been computing v0.21.5 repeatedly (the
canonical+commit-height algorithm hasn't bumped past it for the
recent fix-only commits), so each run rebuilds the same wheels with
the same version and twine rejects the duplicates.
Failed runs:
- 25443521479 (PR #406 merge, 15:04 UTC)
- 25452026402 (PR #409 merge, 17:55 UTC)
- 25452038283 (next push, 17:56 UTC)
PyPA's recommended pattern for this scenario is `skip-existing: true`
on the publish action — duplicate uploads become no-ops, fresh
versions still publish normally. Idempotent.
This unblocks main without touching the version-detection algorithm.
A follow-up audit of `headroom/release_version.py` is the right
deeper fix (so a series of `fix:` commits between releases produces
a sequence of patch bumps), but that's a deeper investigation; this
patch just stops the publish job from going red on every push.
Effect after this lands:
- Push to main → wheels rebuilt with whatever version detect-version
computes
- If that version's wheels are already on PyPI → twine skips them,
exit 0, downstream jobs (publish-npm, publish-docker, create-release)
run normally
- If detect-version computes a NEW version not on PyPI → wheels
publish as before, no behaviour change
The smoke-test job in .github/workflows/eval.yml has been silently
broken on every PR that touches headroom/transforms/**, evals/**, or
compress.py. Root cause: the public chopratejas/headroom repo has zero
Actions secrets configured (`gh api repos/chopratejas/headroom/actions/secrets`
returns `{"total_count":0,"secrets":[]}`), so OPENAI_API_KEY resolves
to the empty string and `openai.OpenAI()` raises OpenAIError at client
init before any compression code runs.
Fix: gate the live OpenAI eval step on a non-empty key and emit a
GitHub `:⚠️:` annotation when skipped, so the skip is loud in
the run summary (per the project's "no silent fallbacks" rule). The
CCR round-trip step above remains the mandatory gate — it tests real
compression logic with zero external dependencies.
Operators who DO wire OPENAI_API_KEY as a repo secret get the live
eval as before.
Test plan:
- yaml syntax validated locally
- Will re-run on PR #400 push; expect smoke-test to pass with the
:⚠️: annotation visible in the run summary.
Refs: F2.1 (unblocks merge of #400)
PR #396's first dry-run on its own changes failed three smoke matrix
entries — two fixed by PR #397's __libc_single_threaded shim (now on
main, this PR is rebased on top), one orthogonal: ubuntu:20.04 + 3.10.
Failure mode on ubuntu:20.04 + 3.10: deadsnakes PPA install path
stopped reliably provisioning python3.10-venv on focal once Ubuntu
20.04 hit End of Standard Support in May 2025. The error is from
apt-get, NOT from `import headroom._core` — the wheel never gets a
chance to load:
E: Unable to locate package python3.10-venv
Continuing to promise wheel-runtime correctness on glibc 2.31 in CI
would require either pulling ESM-tier ubuntu:focal images (paid /
auth-gated) or pinning a specific deadsnakes snapshot URL — neither
of which we want owning long-term.
Floor coverage we keep:
- manylinux_2_28_x86_64 (glibc 2.28 — the floor we promise)
- manylinux_2_28_aarch64 (glibc 2.28 — same)
- ubuntu:22.04 + 3.12 x86_64 (glibc 2.35 — issue #355's reporter env)
- ubuntu:22.04 + 3.12 aarch64 (glibc 2.35 — arm equivalent)
- macos-14 + 3.13 (Apple Silicon native)
Test pin in tests/test_release_workflows.py
(test_release_workflow_has_smoke_import_wheel_gate) only requires the
manylinux floors, ubuntu:22.04, and macos-14, so this drop doesn't
break the structural invariant.
The X1 smoke-import gate (PR #387) catches runtime symbol mismatches
on the wheel before publish, but only at release time. Recent break
patterns were upstream of that:
- #379 (docker bake `name=` regression in PR #376)
- #382 (sdist `os: ubuntu-latest` → `ubuntu-24.04` rename)
- #384 / #385 / #386 (glibc shim alias / link-order iterations)
- #387's own heredoc-indent regression that broke main on the FIRST
release run after merge — the heredoc was inside `bash -ec '...'`,
no PR-time check exercised it
X2 adds a `pull_request:` trigger to release.yml with a NARROW path
filter so the dry-run runs at PR time for changes that affect wheel
layout / release pipeline, but skips for source-only PRs to
`crates/headroom-core` / `crates/headroom-proxy`.
Path filter: release.yml, docker.yml, crates/headroom-py/**,
pyproject.toml, root Cargo.toml, Cargo.lock.
publish-pypi / publish-npm / publish-github-packages / publish-docker
/ create-release all gate on `github.event_name != 'pull_request'`,
so a PR run never publishes — the dry-run is build + collect-dist +
smoke-import only.
concurrency rules:
- PR runs: namespaced by PR number (`pr-N`), cancel-in-progress=true.
- main runs: namespaced by ref_name, cancel-in-progress=false (a
tag-push release that's mid-flight must not be cancelled).
Test pin covers all four invariants (trigger, path filter, publish
gates, concurrency split). 21 tests in test_release_workflows.py,
all green locally.
The X1 smoke-import job (PR #387) embedded the smoke check as a
`<<PY ... PY` heredoc inside `bash -ec '...'`. The outer bash
single-quote preserves whitespace, so the heredoc body and the
closing `PY` retained their YAML indentation (column 14 inside
`bash -ec`). Bash never found a column-0 `PY` and read past EOF:
bash: line 71: warning: here-document at line 65 delimited by
end-of-file (wanted `PY')
IndentationError: unexpected indent
Process completed with exit code 1
Caught immediately on the post-merge release run on main,
manylinux_2_28_x86_64 / Python 3.11 (glibc 2.28-floor):
https://github.com/chopratejas/headroom/actions/runs/25361396712/job/74362427755
`python -c "<f-string>"` was the obvious next try but reintroduces
single-quote nesting (Python f-strings need quote chars; outer
`bash -ec '...'` cannot contain unescaped single quotes).
Fix: write the smoke script to `${RUNNER_TEMP}/smoke_import.py`
in a new "Stage smoke-import script" step (one heredoc at YAML
`run: |` level — uniform indent strip works fine). Linux job
mounts it via `-v ${RUNNER_TEMP}/smoke_import.py:/smoke_import.py:ro`
and runs `python /smoke_import.py`. macOS host runs the same
file directly. No quoting drift between paths.
Locally validated:
- actionlint clean
- 20/20 tests in test_release_workflows.py pass
- hand-execution of the heredoc + script roundtrip works
This is the second X1 follow-up after PR #387's shellcheck fix.
The original X1 design's gap: no PR-time release dry-run that
would have caught the heredoc on PR #387 itself. X2 (PR-time
dry-run) is the structural fix.
actionlint+shellcheck flagged the bash heredoc on the macOS host
step (release.yml:486) for SC2012 (use find instead of ls) and
SC2086 (quote variables to prevent globbing/word splitting). The
Linux container step had the same pattern but escaped detection
because actionlint can't reach into `docker run bash -ec '...'`.
Replace `ls .../headroom_ai-*-${py_tag}-${py_tag}-${arch_tag}.whl
| head -1` with `find ... -name "..." -print -quit` so the glob
expansion happens in find (handles non-alphanumeric filenames
correctly) and the variables sit inside a quoted -name argument
(no SC2086 trigger). Also replace the diagnostic `ls -la` calls
with portable find variants (-printf for GNU find on Linux,
-exec basename for BSD find on macOS).
Net behaviour identical: still picks one matching wheel (only one
exists per py_tag+arch_tag), still prints all wheels for diagnosis
on miss.
Issue #355 plus the three follow-on hotfixes (#384/#385/#386) all
share a pattern: the wheel is technically valid (clippy passes,
tests pass, auditwheel is happy, the static-symbol audit added in
#384 is happy) but FAILS at runtime on a customer's box because of
a dynamic-link symbol mismatch. None of our pre-publish gates
actually `import headroom._core` on a representative customer
environment. They only build it.
What X1 adds
------------
A `smoke-import-wheels` job that runs after `build-wheels` and
before `publish-pypi` / `publish-docker` / `create-release`.
Matrix (6 jobs in parallel, ~3 min wall-clock):
- `manylinux_2_28_x86_64` + Python 3.11 (the floor we promise)
- `ubuntu:22.04` (glibc 2.35) + Python 3.12 (issue #355's env)
- `ubuntu:20.04` (glibc 2.31) + Python 3.10 (older LTS)
- `manylinux_2_28_aarch64` + Python 3.11 (aarch64 floor)
- `ubuntu:22.04` arm64 + Python 3.12 (aarch64 customer env)
- `macos-14` host + Python 3.13 (Apple Silicon)
Each job downloads its arch's wheel artifact, installs the wheel
matching its Python version inside the container, and runs the
exact command the proxy's `_check_rust_core` runs at startup:
from headroom._core import hello as _rust_hello
If any matrix entry fails, `publish-pypi` / `publish-docker` /
`create-release` are blocked. The matrix tells us exactly which
customer environment combination breaks.
Regression test in tests/test_release_workflows.py:
`test_release_workflow_has_smoke_import_wheel_gate` pins the job's
existence, the required matrix entries, and — critically — the
gating wires (publish-pypi / publish-docker / create-release all
need-and-require-success on the smoke job). A future "this slow
CI step always passes anyway, drop it" refactor fails at PR time.
Companion tests `test_glibc_compat_shim_present_in_headroom_py`
and `test_release_workflow_audits_wheel_glibc_symbols` (added in
#384) cover the static-symbol gate; this PR is the dynamic-link
gate. Both are needed.
Issue #355: published headroom_ai-*-manylinux_2_28_*.whl fails to import on Ubuntu 22.04, Debian 11/12, Conda envs with libc < 2.38: 'ImportError: undefined symbol: __isoc23_strtoll'.
Root cause: ORT prebuilt artifacts (downloaded via fastembed's ort-download-binaries-rustls-tls feature) are compiled with gcc-14.2.1 on a glibc-2.38+ host and reference __isoc23_strtoll. Our manylinux build host has glibc 2.38 so the link succeeds; end users with older glibc don't.
Two-part fix: (1) crates/headroom-py/glibc_compat.c provides weak-alias definitions for __isoc23_strtol/strtoll/strtoul/strtoull delegating to the older strtol* family, compiled by build.rs on Linux/glibc only. The dynamic linker prefers glibc's strong symbol when present (>= 2.38) and falls back to ours when not (< 2.38). (2) scripts/audit_wheel_glibc_symbols.py is a release.yml gate that runs objdump -T on every Linux wheel and rejects any UND symbol whose required glibc version exceeds the wheel's manylinux floor.
Validated: the audit correctly rejects the actually-broken v0.20.26 wheel with a precise diagnostic. The shim itself is a tiny static link with zero runtime cost.
Regression tests in tests/test_release_workflows.py pin the shim's load-bearing pieces (.c file, build.rs trigger, [build-dependencies] cc dep) and the audit invocation in release.yml. Future drift fails at PR time, not in the next release.
This is the same bug class as PR #371 (rustls-everywhere). Each instance gets fixed; the audit gate now catches the *class* — any future static linkage that introduces a post-floor symbol gets blocked before publish.
PR #376 pinned the wheel matrix's `os:` from `ubuntu-latest` to
`ubuntu-24.04` (explicit pinning, no semantic change). It silently
disabled the sdist build, whose conditional was
if: matrix.os == 'ubuntu-latest' && matrix.target == 'x86_64-unknown-linux-gnu'
The literal `'ubuntu-latest'` no longer matched, sdist never built,
`release-assets/*.tar.gz` was empty, and `create-release` failed at
gh release upload release-assets/*.tar.gz --clobber
with "no matches found". v0.20.22's release didn't get its sdist on
the GitHub Release.
Fix: key the conditional on `matrix.target` only. sdist is
platform-independent so any single matrix row is fine; using `target`
decouples the sdist build from any future host-runner rename.
Regression test in tests/test_release_workflows.py:
`test_sdist_build_conditional_keyed_on_target_not_os` pins the
target-only form. A future "let's add `os` back for clarity" refactor
will fail at PR time, not 8 minutes into a release.
This is a follow-up to PR #379 (docker bake `name=`) — same root
cause class: PR #376's matrix-shape changes silently broke a
downstream conditional whose literal value was tied to the OLD
matrix shape. Both fixes are now in place + pinned by tests.
PR #376's per-arch fan-out correctly removed `bake-file-tags` from
the docker-build step (tags belong on the multi-arch manifest, not
on per-arch images). But that left bake without ANY reference for
the push target — no tags AND no explicit `name=` in the output
spec. Every release-time docker-build job failed with the
misleading message:
ERROR: tag is needed when pushing to registry
Buildx's actual constraint is "no tags AND no `name=` in the
output = no push target." Since push-by-digest discards tags
anyway, the fix is to specify `name=<registry>/<image>` directly
in the `*.output` spec. Tags are still applied later, only on the
multi-arch manifest by `docker-manifest`.
Regression test in tests/test_release_workflows.py:
`test_docker_per_arch_build_specifies_image_name_in_output`
pins the `name=` substring so a future "the labels block already
has the registry, surely buildx can figure it out" refactor will
fail at PR time rather than 2 minutes into release.
GitHub-hosted Linux arm64 runners (`ubuntu-24.04-arm`) went GA in Aug
2025 and are free for public repositories. Switching the aarch64
wheel + the multi-arch docker matrix off `ubuntu-latest`+QEMU onto
the native runner cuts wall-clock on both surfaces.
release.yml — build-wheels matrix
* `aarch64-unknown-linux-gnu`: `ubuntu-latest` → `ubuntu-24.04-arm`.
maturin-action still runs inside `quay.io/pypa/manylinux_2_28_aarch64`,
but the container now executes natively on an aarch64 kernel
instead of through QEMU emulation. Aarch64 wheel build drops
from ~50–60 min to ~10 min.
* `x86_64-unknown-linux-gnu`: `ubuntu-latest` → `ubuntu-24.04`
(pin the moving alias for reproducibility; no semantic change).
docker.yml — fan-out + manifest merge
* Pre-#377: one `docker-variant-tags` matrix job per variant on
`ubuntu-latest`, using bake's `platforms = [amd64, arm64]` with
QEMU for the arm64 leg. ~1h per variant, 8 variants.
* Post-#377: split into `docker-build` (variant × arch = 16
parallel jobs, each on its native runner, single-platform
push-by-digest) and `docker-manifest` (per variant, merges the
two arch digests into a multi-arch tagged manifest with
`docker buildx imagetools create`, signs the index manifest
with cosign). Wall-clock drops from ~1h per variant to ~10 min.
* `docker/setup-qemu-action` removed — there's no QEMU left.
* Per-(variant, arch) GHA cache scopes so the two arches don't
collide on cache keys.
* `promote-latest` rewired to depend on `docker-manifest`.
Behavior change: cosign now signs only the multi-arch index digest
per variant, not each per-platform image. `cosign verify <repo>:tag`
(the typical flow) is unchanged because cosign resolves the tag to
the index digest. Verifiers pinning a specific per-arch digest will
need to verify the index digest instead.
Regression tests in tests/test_release_workflows.py:
* `test_aarch64_wheel_uses_native_arm64_runner` — pins the
aarch64 row to `ubuntu-24.04-arm` (and the amd64 row to
`ubuntu-24.04`, not `-latest`), so a future "let me unify on
ubuntu-latest" refactor surfaces the QEMU regression at PR time.
* `test_docker_workflow_builds_on_native_arch_runners` — pins the
fan-out matrix's arch entries, asserts push-by-digest, asserts
`setup-qemu-action` is absent from non-comment lines, asserts
the manifest-merge job exists.
Verified:
* Both workflow files parse as valid YAML with the expected job
graph (`docker-build` → `docker-manifest` → `promote-latest`,
16 fan-out jobs, 8 manifest jobs).
* `docker buildx imagetools inspect <tag> --format '{{ json . }}'`
exposes the index digest at `.manifest.digest` (confirmed via
Docker's official reference).
* `ubuntu-24.04-arm` is the correct GitHub-hosted runner label
(GA 2025-08-07, free for public repos).
* `make ci-precheck-rust` and `make ci-precheck-python` both pass
locally; `tests/test_release_workflows.py` is 15/15 green
(13 existing + 2 new).
# Root cause of the wheel-build cascade
We have shipped 5 release-pipeline hot-fixes in 12 hours, each
addressing a different symptom of the same architectural problem:
1. PR #363 — npm artifact downloads + tried `yum openssl-devel`
2. PR #367 — vendored OpenSSL in `headroom-proxy` + dropped Intel mac
3. PR #369 — Debian-cross perl install (`perl` not `libipc-cmd-perl`)
4. PR #370 — moved `openssl/vendored` from headroom-proxy to headroom-py
5. (this PR) — ELIMINATE OpenSSL entirely
Each fix exposed a different missing system package or feature flag in
a different build surface (manylinux x86_64 vs aarch64-cross-Debian vs
macOS Intel vs e2e/wrap Dockerfile vs e2e/init Dockerfile vs main
Dockerfile vs devcontainer). We were playing whack-a-mole because every
Cargo dep change to the OpenSSL surface required matching system-package
updates in 6+ different Dockerfiles and workflows, and the PR-level CI
didn't exercise all of them.
# Why this PR is the structural fix
`fastembed` exposes clean rustls feature flags:
- `hf-hub-rustls-tls` (replaces default `hf-hub-native-tls`)
- `ort-download-binaries-rustls-tls` (replaces default `…native-tls`)
By disabling fastembed's default features and enabling the rustls
variants explicitly, we remove `native-tls` (and therefore `openssl-sys`,
`openssl`, `openssl-src`, perl modules, OpenSSL build-time deps,
vendored OpenSSL ~30s build cost) from the entire workspace dep tree.
Verified locally:
$ cargo tree -p headroom-py -i openssl-sys
error: package ID specification `openssl-sys` did not match any packages
$ cargo tree -p headroom-py -i native-tls
error: package ID specification `native-tls` did not match any packages
$ cargo build --release -p headroom-py
Finished `release` profile [optimized] target(s) in 25.57s
(Down from 1m+ with vendored OpenSSL.)
# Cleanups enabled by this change
- crates/headroom-py/Cargo.toml — dropped the `openssl/vendored`
workaround from PR #370.
- crates/headroom-proxy/Cargo.toml — same dep removed.
- e2e/wrap/Dockerfile — dropped `yum install openssl-devel pkgconfig
perl-IPC-Cmd`. Comment retained explaining why.
- e2e/init/Dockerfile — same.
- Dockerfile (main) — dropped `pkg-config libssl-dev` from apt-get.
- .devcontainer/Dockerfile — dropped `pkg-config libssl-dev`.
- .github/workflows/release.yml — removed the entire before-script-linux
block (perl install probe + multi-package-manager dispatch + fail-loud
assertion). No longer needed.
# Regression gate
Three new structural tests in tests/test_release_workflows.py:
- test_no_openssl_sys_in_wheel_build_tree — runs `cargo tree -p <crate>
-i openssl-sys` for headroom-py / headroom-proxy / headroom-core. If
openssl-sys reappears (a future native-tls enabler creeping in via a
new dep), this fails AT PR TIME with an actionable message.
- test_no_native_tls_in_wheel_build_tree — same shape, native-tls is
the proximate cause.
- test_fastembed_uses_rustls_features — checks the Cargo.toml so a
future "let me bump fastembed and forget the features" doesn't
silently re-introduce OpenSSL.
Plus two cleanup gates:
- test_dockerfiles_no_longer_install_openssl_devel
- test_release_yml_does_not_install_openssl_or_perl_for_wheels
All 13 release-workflow tests pass. `make ci-precheck` PASSED.
# What this teaches us about rollouts (per user's ultrathink ask)
The 5-fix cascade exposed three meta-problems:
1. PR checks don't block merges. PR #370 had docker-init-e2e,
docker-wrap-e2e, docker-native-e2e all FAILED yet got merged.
Branch protection should require these checks. Operator action
needed (cannot fix in code).
2. Local validation is misleading. `cargo build -p headroom-py` from
the workspace root used the workspace lockfile and looked green;
CI did fresh resolution against headroom-py's manifest alone where
the feature wasn't enabled. Lesson: verify structural invariants
with `cargo tree -e features` before trusting that a build "works."
3. 6+ build surfaces with independent system-dep state. Every Cargo
change required matching updates in 6 places. The structural answer
(this PR) is to NOT depend on system OpenSSL at all. Where structural
fixes are not possible, the answer is a single shared
scripts/install-rust-build-deps.sh — but with this PR there's
nothing left to install.
Previous wheel hot-fix (#367) introduced a NEW failure mode that the
F1-merge release run surfaced:
E: Unable to locate package libipc-cmd-perl
The aarch64-unknown-linux-gnu maturin-action target does NOT use the
AlmaLinux 8 manylinux_2_28 image — it uses a Debian/Ubuntu-based
cross-compile container. The previous hot-fix's apt branch installed
`libipc-cmd-perl` which is a deprecated alias and is no longer in the
default Debian/Ubuntu sources. The build failed before openssl-src
could even start.
# What the script actually needs to do
`IPC::Cmd` is a Perl core module since 5.10, so any working `perl`
install provides it. The fix:
1. **Probe first.** `perl -MIPC::Cmd -e 1` exits cleanly when the
module is already importable — skip the install entirely. Some
manylinux images already ship it; others don't.
2. **Cover every package manager.** dnf (modern RHEL family) → yum
(older RHEL) → apt-get (Debian/Ubuntu) → apk (Alpine/musllinux).
The maturin-action uses different containers per (target, manylinux)
combo and we don't get to pick.
3. **Use `perl` not `libipc-cmd-perl` on Debian.** The plain `perl`
meta-package pulls `perl-modules-*` which contains IPC::Cmd. Works
on every Debian/Ubuntu version we'll see; `libipc-cmd-perl` is
gone from default sources.
4. **Fail loud after install.** `perl -MIPC::Cmd -e 'print "loaded
OK"'` runs unconditionally at the end. If somehow the module is
STILL missing, we fail here — not 5 minutes later in the
openssl-src compile step where the error message is harder to
debug. Matches the project's "no silent fallback" rule.
# Tests
`test_build_wheels_installs_perl_ipc_cmd_for_vendored_openssl`
updated to gate the new shape:
- Asserts `perl -MIPC::Cmd -e 1` pre-probe is present
- Asserts dnf/yum/apt-get/apk branches all exist
- Asserts apt branch installs `perl` not `libipc-cmd-perl`
- Asserts `libipc-cmd-perl` does not appear on any non-comment line
- Asserts the final `perl -MIPC::Cmd` fail-loud assertion is present
All 11 release-workflow tests pass. `make ci-precheck` PASSED.
The previous hot-fix (#363) addressed npm artifact downloads and added
openssl-devel installs in the manylinux container, but the wheel build
still fails on three of four matrix entries with three distinct errors:
1. ubuntu-x86_64 with `manylinux: auto` resolved to manylinux2014
(CentOS 7 / OpenSSL 1.0.2k). `openssl-sys 0.9` requires OpenSSL
1.1.0+ — "different version of OpenSSL was found".
2. ubuntu-aarch64 cross-compiles via `aarch64-unknown-linux-gnu-gcc`
from an x86_64 manylinux container. The `yum install openssl-devel`
we added installs x86_64 headers; `/usr/aarch64-unknown-linux-gnu/
include/` has no OpenSSL — "openssl/opensslv.h: No such file or
directory".
3. macos-15-intel fails on `ort-sys` (transitive via the ML compression
backend), which has no prebuilt ONNX Runtime binaries for
`x86_64-apple-darwin`. Unrelated to OpenSSL; an upstream limitation.
Why the workspace pulls openssl-sys at all: `hf-hub` (transitive via
`fastembed`) hard-codes `native-tls` as a default feature. Cargo's
feature unification then enables openssl-sys for the whole workspace
despite our `reqwest`/`tokio-tungstenite`/`tokio-rustls` preferences.
# Fix 1: vendored OpenSSL
Add `openssl = { version = "0.10", features = ["vendored"] }` to
`crates/headroom-proxy/Cargo.toml`. The `vendored` feature compiles
OpenSSL from source as part of the cargo build — works on every target
uniformly. Local build verified: cargo now pulls
`openssl-src v300.6.0+3.6.2` and compiles it. ~30s extra one-time
build cost.
The `openssl/vendored` feature DEFEATS `OPENSSL_DIR`. We therefore
remove the previous hot-fix's "Install OpenSSL (macOS)" step that
exported `OPENSSL_DIR` — leaving it would silently regress to the
system-OpenSSL path that broke originally.
# Fix 2: pin manylinux floor to 2_28
Change x86_64-unknown-linux-gnu from `manylinux: auto` to
`manylinux: 2_28` (matching aarch64 + the e2e Dockerfiles). This
isn't strictly required with vendored OpenSSL — the floor is now
glibc 2.28 / AlmaLinux 8 which has modern toolchain — but it removes
the CentOS-7 surface entirely and matches our runtime container
target.
# Fix 3: drop x86_64-apple-darwin from the matrix
`ort-sys 2.0.0-rc.12` has no prebuilt ONNX Runtime binaries for that
target. Building ORT from source would add CMake + ~5 minutes per
build. Apple Silicon macOS (`aarch64-apple-darwin`) is fully covered;
Intel-mac users install from the platform-independent sdist this
matrix also produces.
Tracked as a follow-up: switch the ML backend to `ort-tract` or
upstream a request for x86_64 macOS prebuilts.
# before-script-linux: keep perl-IPC-Cmd, drop openssl-devel
OpenSSL's vendored `Configure` script needs `IPC::Cmd` (without it
the build fails with "Can't locate IPC/Cmd.pm"). System
openssl-devel is no longer needed.
# Tests
4 new regression tests gate this:
- `test_headroom_proxy_vendors_openssl`
- `test_build_wheels_installs_perl_ipc_cmd_for_vendored_openssl`
- `test_build_wheels_does_not_set_openssl_dir`
- `test_build_wheels_matrix_excludes_intel_macos`
Plus the previous 7. All 11 release-workflow tests pass.
`make ci-precheck` PASSED. Local `cargo build --release -p headroom-py`
green.
Three independent failures on the post-merge release run for PR #360,
all introduced by the single-wheel maturin refactor:
1. Linux wheel matrix (`ubuntu-x86_64`, `aarch64-unknown-linux-gnu`)
failed inside the manylinux container with:
Could not find openssl via pkg-config
The system library `openssl` required by crate `openssl-sys`
was not found.
`openssl-sys` is transitive via `fastembed` → `hf-hub` → `ureq`
→ `native-tls`. The e2e Dockerfiles (`e2e/wrap`, `e2e/init`) we
wrote for #360 install `openssl-devel` upfront, but the
`build-wheels` matrix uses `PyO3/maturin-action@v1` which spins
up its OWN manylinux container that does not inherit those
installs. Fix: add a `before-script-linux:` to the action with a
yum/apt-get conditional so it works on RHEL-family (manylinux2014,
manylinux_2_28) and Debian-family musllinux variants.
2. macOS x86_64 wheel build failed with `maturin` exit 1 from the
same `openssl-sys` lookup. The aarch64 macos-14 runner happens
to have `/opt/homebrew/opt/openssl@3` on `openssl-sys`'s default
discovery path; the Intel macos-15-intel runner uses
`/usr/local/Cellar` which is NOT on that path. Fix: add a
pre-maturin step that runs `brew install openssl@3` and exports
`OPENSSL_DIR` / `OPENSSL_LIB_DIR` / `OPENSSL_INCLUDE_DIR` /
`PKG_CONFIG_PATH`. The aarch64 runner happily picks up the
explicit env vars too — no regression.
3. `publish-npm` and `publish-github-packages` both fail with
"Artifact not found for name: dist". Both jobs `npm pack` + `npm
publish` directly from the checked-out source tree — they never
consume the Python `dist` artifact. The `Download dist artifact`
step was vestigial dead code carried over from a prior workflow
shape; the only reason it didn't fail before #360 is that the
pre-refactor `build` job DID upload a `dist` artifact. Post-#360,
`dist` is produced by `collect-dist` and neither publish job is
gated on it (by design — npm vs PyPI ecosystems publish
independently). Fix: remove the dead download step from both
jobs. Loose coupling is preserved; `create-release` still gates
the GitHub Release tag on all of build / build-wheels /
collect-dist / publish-* succeeding.
Why the PR-level CI didn't catch any of this: `release.yml` only
runs on `push: branches: [main]`. PR #360's CI exercised `ci.yml`
which has a separate `ci-build-wheels-on-pr` matrix that uses a
different setup. The release surface only fires post-merge.
Tests added (regression gates):
- `test_build_wheels_installs_openssl_devel_on_linux_via_before_script`
- `test_build_wheels_resolves_openssl_dir_explicitly_on_macos`
- `test_npm_publish_jobs_do_not_download_dist_artifact`
All 9 release-workflow tests pass. `make ci-precheck` PASSED.
Code-scanning alert #65 (CodeQL actions/missing-workflow-permissions,
CWE-275) flagged the new build-wheels job for not declaring an explicit
permissions block. While at it, audit the rest of release.yml — same
gap exists on detect-version, build, collect-dist, and publish-npm.
Each job gets `contents: read` (the minimal default) since none of them
push, write packages, or mutate releases through GITHUB_TOKEN. Existing
write-bearing jobs (publish-pypi: id-token, publish-github-packages:
packages, create-release: contents) keep their narrower scopes.
Two early failures on PR #360's first CI run:
1. validate (default/memory-stack) + validate-worktree failed at the
apt-get update step in .devcontainer/Dockerfile. The base image
mcr.microsoft.com/devcontainers/python:1-3.12-bookworm ships with
dl.yarnpkg.com configured as an apt source whose GPG key has expired:
Err:4 https://dl.yarnpkg.com/debian stable InRelease
The following signatures couldn't be verified because the public
key is not available: NO_PUBKEY 62D54FD4003F6525
apt-get returns exit 100, the whole RUN aborts before pkg-config /
libssl-dev install. The maturin refactor doesn't need yarn, so drop
/etc/apt/sources.list.d/yarn.list before apt-get update. The debian
main repo updates fine on its own.
2. workflow-validation (actionlint) failed parsing rust.yml step name
"Build wheel (single-wheel architecture: builds headroom-ai)" —
actionlint's YAML parser saw the unquoted colon inside the name as
a mapping. Quote the string.
Eliminates the dual-package architecture that was the root cause of #355.
`pip install headroom-ai` now produces ONE wheel containing both the Python
source (headroom/*.py) and the compiled Rust extension (headroom/_core.so).
No more separate `headroom-core-py` package, no more chicken-and-egg with
PyPI publication, no more wheelhouse / PIP_FIND_LINKS / composite-action
plumbing in CI.
This is the canonical pattern used by cryptography, polars, ruff,
pydantic-core, and other Rust-as-core Python packages. Honors the
"Rust as core engine" direction.
## What changed
- pyproject.toml: `[build-system]` swapped from hatchling to maturin.
`[tool.hatch.*]` deleted; `[tool.maturin]` added pointing at
`crates/headroom-py/Cargo.toml` for the cdylib. `python-source = "."`
picks up the root `headroom/` package directly (dashboard HTML
templates and other non-Python files included automatically).
- crates/headroom-py/pyproject.toml: deleted. The crate is no longer a
separate published package; its Cargo.toml stays as the cdylib build
target invoked via `[tool.maturin] manifest-path`.
- crates/headroom-py/python/: deleted (placeholder layout for the old
separate package).
## CI updates
- ci.yml: `test` / `test-extras` / `test-agno` jobs simplified — Rust
toolchain set up before `pip install -e .` (which now invokes maturin
via build-system). Removed the "build wheel + symlink .so" dance.
`build` job swapped from `python -m build` (hatch) to
`maturin build` + `maturin sdist`.
- release.yml: collapsed dual-package matrix into one. New `build-wheels`
matrix produces cross-platform wheels for cp310/11/12/13 ×
{linux x86_64, linux aarch64, macos x86_64, macos aarch64}. New
`collect-dist` aggregator merges artifacts. publish-pypi consumes the
merged dist.
- init-native-e2e.yml: dropped windows-latest from the matrix —
upstream `esaxx-rs` (/MT) and `ort-sys` (/MD) link with conflicting
MSVC C runtime libraries, so the Rust extension cannot build for
win_amd64 today. Tracked as a follow-up; not a blocker for Linux+macOS.
- headroom-e2e-setup: composite action now sets up Rust toolchain +
Swatinem/rust-cache before `pip install -e .[proxy]`.
- eval.yml, publish.yml, rust.yml: same pattern — rust toolchain before
install. rust.yml's wheels job builds from root pyproject.toml (no
more `-m crates/headroom-py/Cargo.toml`).
- e2e/init/Dockerfile, e2e/wrap/Dockerfile: install rust + maturin in
the build stage; copy `crates/` + workspace `Cargo.toml/lock` so the
install can build the extension. Dropped `HEADROOM_REQUIRE_RUST_CORE=false`
from wrap-e2e — the image now ships the full Rust core.
- Dockerfile (main): simplified — no more Layer 2/3 dance with
`headroom-core-py` install + symlink. Single `uv pip install` builds
+ installs everything.
- .devcontainer/Dockerfile: rust toolchain + libssl-dev + maturin
added so `uv sync` builds the extension inside the devcontainer.
## Lockfile + script
- uv.lock: regenerated. No `headroom-core-py` entries remain.
- scripts/build_rust_extension.sh: simplified from a symlink-into-tree
workaround to a thin wrapper around `pip install -e .`. The maturin
build-backend handles placement automatically.
## Local validation (all green on macOS aarch64)
1. Clean venv `pip install -e .` → `from headroom._core import …` works.
2. `maturin build --release` → 13.8 MB wheel, 336 files including
`headroom/_core.cpython-311-darwin.so` (32 MB cdylib) and
`headroom/dashboard/templates/dashboard.html`.
3. `pip install <wheel>` in fresh venv → import works.
4. Wheel contents verified via `unzip -l`.
5. `pytest tests/test_transforms/test_diff_compressor.py` — 29 passed.
6. `pytest tests/test_relevance.py` — 30 passed.
7. `cargo build --workspace` + `cargo test --workspace` — all green.
8. `make ci-precheck` — 176 Python tests + Rust + commitlint green.
## Migration notes
Users on `pip install headroom-ai` get the Rust core automatically
(linux + macos wheels). sdist installs require rust toolchain available
locally — pip will build via maturin.
Closes#355
Supersedes #357 (workarounds-based fix abandoned in favor of
architectural fix)
