## Description
The CI lint job (`ruff check .` → `ruff format --check .` → `mypy
headroom`) was red on `main`
and therefore on every open PR, for two unrelated reasons that the early
ruff failure was masking:
1. **ruff**: the lint job installs `ruff` unpinned, and ruff 0.15.17
began enforcing import-block
sorting (`I001`) and formatting that older ruff accepted → `ruff check
.` / `ruff format --check .`
fail on files nobody touched.
2. **mypy**: `headroom/providers/opencode/config.py` had two `return
json.loads(...)` statements in
a function declared `-> dict[str, Any]`; `json.loads` is typed `Any`, so
`mypy headroom` fails
with `no-any-return` (reproduced on mypy 1.20.2 — not a version-specific
quirk).
This restores a green lint baseline and pins both linters so a future
release can't silently break
CI again.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `.github/workflows/ci.yml`: pin `ruff==0.15.17` and `mypy==1.20.2` in
the lint job.
- Applied `ruff check --fix .` (6 `I001` import-sort fixes) and `ruff
format .` (10 files) across
the repo — import ordering and whitespace only, no behavior change.
- `headroom/providers/opencode/config.py`: narrow both
`_parse_json_loose` return sites with an
`isinstance(parsed, dict)` guard, so the `dict[str, Any]` annotation is
true at runtime
(non-dict JSON falls back to `{}`) and mypy's `no-any-return` is
resolved.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`, `ruff format --check .`, `mypy
headroom --ignore-missing-imports`)
### Test Output
```text
$ python -m ruff check .
All checks passed!
$ python -m ruff format --check .
913 files already formatted
$ python -m mypy headroom/providers/opencode/config.py --ignore-missing-imports
Success: no issues found in 1 source file
$ python -m pytest tests/test_providers_opencode_config.py -q
37 passed
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.13, ruff 0.15.17, mypy 1.20.2,
branch ci/fix-ruff-lint off
headroomlabs-ai/main
- Exact command / steps: reproduced the red lint (latest ruff: 6 `I001`
+ 10 unformatted files;
the mypy failure was read from the #1295 CI lint log —
`config.py:125,133 no-any-return`, and
re-confirmed locally on mypy 1.20.2). Applied the ruff auto-fix/format,
added the dict guard,
pinned both linters, and re-ran each lint step.
- Observed result: `ruff check .` → "All checks passed!"; `ruff format
--check .` → "913 files
already formatted"; `mypy` on the fixed file → "Success: no issues
found"; full `mypy headroom`
reports only Unix `fcntl` attributes that don't exist on this Windows
box (present on the Linux
CI runner, where the prior run showed exactly the two now-fixed errors).
37 opencode-config
tests pass.
- Not tested: did not run the full OS/Python test matrix — the change is
formatting + two CI
dependency pins + a two-line type-narrowing guard, with no runtime
behavior change for dict JSON.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
This PR implements transparent `headroom wrap opencode` support without
asking users to edit OpenCode provider URLs, choose an extra CLI flag,
or maintain a static provider list.
The wrapper now lives at the runtime transport boundary: OpenCode keeps
its user/provider config, while Headroom intercepts outbound provider
traffic in-process and routes it through the local Headroom proxy.
## What changed
### Transparent OpenCode wrapping
- `headroom wrap opencode` injects the `headroom-opencode` plugin
through `OPENCODE_CONFIG_CONTENT`.
- Existing OpenCode provider URLs are preserved. We do not rewrite user
config URLs to point at Headroom.
- Existing `OPENAI_BASE_URL` and `ANTHROPIC_BASE_URL` env vars are
preserved.
- Local OpenCode traffic, localhost traffic, and Headroom proxy traffic
bypass the shim to avoid loops.
### Runtime transport interception
- Added an OpenCode plugin transport shim that wraps:
- `globalThis.fetch`
- `http.request` / `http.get`
- `https.request` / `https.get`
- External provider calls are routed to the local Headroom proxy.
- The original upstream origin is passed through `x-headroom-base-url`,
so the proxy can forward to the real provider without changing OpenCode
config.
- External `http2.connect` is blocked loudly instead of allowing direct
provider traffic to leak outside Headroom.
### Live provider additions
Provider coverage is no longer based on a static config scan. Because
routing happens at outbound request time, providers added mid-session
are routed through Headroom automatically as long as they use the
covered Node transport paths.
### Subagent and child-process coverage
- The parent OpenCode plugin sets a packaged Node preload shim through
`NODE_OPTIONS=--import=.../hook-shim/handler.js`.
- The transport shim patches `child_process.spawn`, `exec`, `execFile`,
and `fork` so child Node processes receive the Headroom preload even
when OpenCode passes a custom `env`.
- The child-process shim fails closed if it loads without
`HEADROOM_OPENCODE_TRANSPORT_PROXY_URL`.
- This closes the subagent leak path where a child Node process could
otherwise start without Headroom transport interception.
## Why this goes beyond PR #1089
PR #1089 improves OpenCode provider registration, but it still focuses
on provider config shape. This PR moves the enforcement boundary to
runtime transport interception.
This PR goes further because:
- No provider URL rewriting is required.
- New providers added mid-session are covered automatically.
- Subagents and child Node processes inherit the Headroom transport
shim.
- Direct external HTTP/2 paths fail loudly instead of leaking.
- The wrap remains transparent to the user's OpenCode provider config.
- The wrapper is fail-closed for unsupported child-process preload
state.
## Additional robustness fixes
While validating the change in Docker, the full Python suite exposed
unrelated Linux/container robustness issues. These are fixed in this PR
so the suite is green:
- Binary cache handling now treats cache paths under a non-writable
existing parent as unavailable, including when tests run as root in
Docker.
- `release_version.py` honors `MANUAL_VER` before git calls so direct
script execution works outside a `.git` checkout.
- Test logger isolation now resets relevant Headroom child loggers so
proxy logging setup cannot poison later `caplog` tests.
- The scanner missing-path test now uses a guaranteed missing `tmp_path`
child instead of relying on `/nonexistent/path`.
## Validation
All implementation validation was run inside Docker.
- Full Python suite from a fresh Docker copy: `6605 passed, 523
skipped`.
- Ruff on changed Python/OpenCode paths: passed.
- OpenCode plugin typecheck: passed.
- OpenCode plugin tests: `9 passed`.
- OpenCode plugin build: passed.
- Hook shim preload smoke test: passed.
## Notes
This PR intentionally does not add a CLI option. `headroom wrap
opencode` means full wrap. Either Headroom wraps OpenCode transparently,
or the path fails loudly instead of silently leaking provider traffic.
---------
Co-authored-by: Rudimar Ronsoni <6081613+rudironsoni@users.noreply.github.com>
## Description
`main` is red on the **Wrap E2E** workflow and on CI's
**`docker-native-e2e`** job. Both run `e2e/wrap/run.py` and fail on the
same assertion:
```
e2e/wrap/run.py:553 assert_true(project_agents.exists(), "Codex wrap should create project AGENTS.md")
AssertionError: Codex wrap should create project AGENTS.md
```
PR #1240 (`fix(wrap): keep Codex RTK guidance global`) intentionally
moved Codex RTK guidance to the global `~/.codex/AGENTS.md` and stopped
writing a project-level `AGENTS.md` (a project `AGENTS.md` is now
created only when `wrap codex --memory` is used, for memory guidance).
#1240 updated its unit test (`tests/test_cli/test_wrap_codex.py`) but
not the wrap **e2e** harness, so `verify_codex_wrap` still asserted the
old project-level behavior. This corrects the e2e harness to match the
shipped behavior — it is a stale-test fix, not a behavior change.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `e2e/wrap/run.py` `verify_codex_wrap`: removed the two now-false
project-level assertions (`project_agents.exists()` and the project
RTK-marker check) and the unused `project_agents` variable.
- Kept the global assertions (`~/.codex/AGENTS.md` exists + contains the
RTK marker) — these already match the shipped behavior.
- Added a comment documenting that Codex RTK guidance is global-only
(#1240) and a project `AGENTS.md` appears only with `--memory`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ ruff check e2e/wrap/run.py
All checks passed!
$ python -m pytest tests/test_cli/test_wrap_codex.py -q
============================== 57 passed in 6.13s ==============================
# includes test_wrap_codex_injects_rtk_globally_without_changing_project_agents,
# which asserts the RTK marker lands in ~/.codex/AGENTS.md and the project
# AGENTS.md is left byte-for-byte unchanged — the contract this e2e now matches.
```
## Real Behavior Proof
- Environment: macOS (darwin 25.4.0), Python 3.12 venv; root-caused from
the failing CI logs and verified the behavior contract via the unit
suite (the Docker wrap-e2e itself runs in CI)
- Exact command / steps: read the failing step logs for CI run
`27912260743` and Wrap E2E run `27912260746` (both fail at
`e2e/wrap/run.py:553`); confirmed via `headroom/cli/wrap.py:3679` that
RTK injects only into `~/.codex/AGENTS.md`; ran `pytest
tests/test_cli/test_wrap_codex.py` and `ruff check e2e/wrap/run.py`
- Observed result: 57/57 codex-wrap unit tests pass;
`test_wrap_codex_injects_rtk_globally_without_changing_project_agents`
confirms the RTK marker is written to `~/.codex/AGENTS.md` while the
project `AGENTS.md` is left unchanged — exactly what the corrected e2e
asserts. ruff clean.
- Not tested: the full Docker `Wrap E2E` / `docker-native-e2e` jobs
locally (require Docker + a wheel build); they run on this PR's CI to
confirm the fix turns both jobs green.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
N/A — e2e harness fix; evidence is under "Real Behavior Proof".
## Additional Notes
- `mypy` / "new tests added" are unchecked: this is a test-only
correction to an existing e2e assertion, no production code or new test
surface.
- Root-cause detail: a project-level `AGENTS.md` is created by `wrap
codex` only inside the `if memory:` branch
(`headroom/cli/wrap.py:3704`/`3715`); the e2e runs `wrap codex --
--help` without `--memory`, so no project file is created — the
assertion could never pass after #1240.
- `ruff check .` scoped to the changed file here (the dashboard HTML
template trips ruff's `invalid-syntax`, a known repo false-positive).
## Summary
Adds a **per-project savings breakdown** to the proxy dashboard,
covering **all wrap-supported agents: Claude Code, Codex, aider,
Copilot, and Cursor**. Spec and rationale in #802 (feature-request issue
— happy to adjust scope per maintainer feedback).
How it works — two attribution channels, by client capability:
**Header channel** (clients that can send custom headers):
- `headroom wrap claude` appends `X-Headroom-Project: <basename(cwd)>`
to `ANTHROPIC_CUSTOM_HEADERS` (a user-supplied x-headroom-project header
always wins; other user headers preserved).
- `headroom wrap codex` extends the injected
`[model_providers.headroom]` block with `env_http_headers = {
"X-Headroom-Project" = "HEADROOM_PROJECT" }` and sets `HEADROOM_PROJECT`
per launch — Codex only sends the header when the env var is set, so the
static config stays inert outside wrap.
**Base-URL prefix channel** (clients that cannot send custom headers):
- The proxy accepts `/p/<url-encoded-name>/...`; middleware strips the
prefix before routing (before Starlette caches the URL) and binds the
project. The explicit header wins over the prefix.
- `headroom wrap aider` points `OPENAI_API_BASE` / `ANTHROPIC_BASE_URL`
at the prefixed URL.
- `headroom wrap copilot` points `COPILOT_PROVIDER_BASE_URL` at the
prefixed URL (BYOK anthropic/openai provider types and the
GitHub-subscription path).
- `headroom wrap cursor` prints the prefixed Override Base URL in its
setup instructions, with a note explaining the attribution.
**Shared plumbing:**
- New `headroom/proxy/project_context.py`: header classification, `/p/`
prefix split/strip, URL-prefix builder, and a contextvar bound per
request (HTTP middleware + WS accept for the Codex responses bridge);
the outcome funnel resolves it (explicit `RequestOutcome.project` wins),
stamps `RequestLog.tags["project"]`, and forwards it through
`PrometheusMetrics.record_request` into the `SavingsTracker`.
- `SavingsTracker`: persisted state gains a `projects` map (requests,
tokens saved, savings USD, input tokens/cost, last activity). Schema v2
→ v3 with transparent forward migration (v2 files load cleanly,
`projects` starts empty). Names sanitized (printable-only, 128-char
cap); map capped at 50 projects, evicting the smallest bucket.
- `/stats` exposes `savings.per_project` (and
`projects`/`projects_limit` inside `persistent_savings`);
`/stats-history` exposes `projects`.
- Dashboard gains a "Per-Project Savings" table mirroring the per-model
table (Alpine `x-text` only — names are user-supplied, no HTML
injection).
**Behavior changes:** none for unattributed traffic — no header and no
prefix means no bucket, aggregate totals exactly as before
(regression-tested: legacy `/stats` and `/stats-history` shapes are
pinned by tests).
## Real behavior proof
**Setup tested on:** macOS (Darwin 25.5.0), Python 3.11.9, `pip install
-e ".[dev]"`, proxy on spare ports with isolated
`HEADROOM_SAVINGS_PATH`; real Claude Code CLI (subscription auth) and
real Codex CLI (ChatGPT auth) as clients.
**Header channel — exact steps:**
```
HEADROOM_SAVINGS_PATH=/tmp/headroom-proof-savings.json .venv/bin/python -m headroom.cli proxy --port 9123 # background
cd /tmp/proof-alpha && headroom wrap claude --port 9123 --no-rtk --no-mcp --no-serena -- -p "Reply with exactly: OK"
cd /tmp/proof-beta && headroom wrap claude --port 9123 --no-rtk --no-mcp --no-serena -- -p "Reply with exactly: OK"
cd /tmp/proof-beta && headroom wrap codex --port 9123 --no-rtk --no-mcp --no-serena -- exec --skip-git-repo-check "Reply with exactly: OK"
```
**Observed** (copied live `/stats` output; all runs replied `OK` through
the proxy):
```json
{
"proof-beta": {
"requests": 3, "tokens_saved": 140, "compression_savings_usd": 0.0007,
"total_input_tokens": 38581, "total_input_cost_usd": 0.360433,
"last_activity_at": "2026-06-10T09:19:17Z", "savings_percent": 0.36
},
"proof-alpha": {
"requests": 1, "tokens_saved": 0, "compression_savings_usd": 0.0,
"total_input_tokens": 9050, "total_input_cost_usd": 0.32245,
"last_activity_at": "2026-06-10T09:18:09Z", "savings_percent": 0.0
}
}
```
`proof-beta` aggregates one Claude Code turn + one Codex `exec` run
(Codex attribution flows through `env_http_headers`).
**Prefix channel — exact steps** (the same mechanism the
aider/copilot/cursor wraps emit, driven by a real client):
```
.venv/bin/python -m headroom.cli proxy --port 9124 # background, isolated savings path
ANTHROPIC_BASE_URL="http://127.0.0.1:9124/p/aider-style-project" claude -p "Reply with exactly: OK"
```
**Observed:**
```json
{
"aider-style-project": {
"requests": 1, "tokens_saved": 0, "compression_savings_usd": 0.0,
"total_input_tokens": 8571, "total_input_cost_usd": 1.018575,
"last_activity_at": "2026-06-10T10:10:13Z", "savings_percent": 0.0
}
}
```
`/stats-history` returned `schema_version: 3` with the same `projects`
keys; `GET /dashboard` HTML contains the new "Per-Project Savings"
table; persisted state survived a tracker reload; a hand-written v2
savings file loaded cleanly with an empty `projects` map.
**What I did NOT test live:** the actual aider/Copilot/Cursor binaries
end-to-end (their wraps emit exactly the prefixed URLs exercised above —
unit tests pin the emitted env/URLs); Codex subscription-mode routing
via the built-in `openai` provider (no provider headers there — such
traffic simply stays unattributed); multi-worker uvicorn; Windows.
## Tests
- `tests/test_proxy_project_savings.py` (17 tests): sanitization, header
classification, `/p/` prefix split + URL-builder round-trip, tracker
aggregation/persistence/migration/cardinality-cap/state-sanitization,
funnel→`/stats` end-to-end, middleware binding for header + prefix +
precedence, plus regression tests pinning legacy
`/stats`/`/stats-history` shape and unattributed-traffic totals.
- Wrap/provider tests extended: `test_cli/test_wrap_helpers.py`,
`test_cli/test_wrap_codex.py`, `test_provider_aider.py`,
`test_provider_cursor.py`, `test_provider_copilot_wrap.py` (prefixed
env/URLs, user-override wins, no duplicate header, TOML block contents,
block strip, setup-line note).
- Full `pytest` suite run locally; `ruff check` + `ruff format` clean on
all touched files.
- `CHANGELOG.md` updated.
## Dependencies
None added or bumped.
Closes#802 (pending maintainer 👍 per CONTRIBUTING — raised there first
with the full spec).
---------
Co-authored-by: Ash Rhodes <ashley.rhodes@king.com>
Adds four new `headroom wrap <agent>` subcommands so the proxy can
front-end Cline (VS Code), Continue (VS Code/JetBrains), Goose (Block CLI),
and OpenHands (CLI), extending the existing claude/codex/aider/copilot/
cursor pattern. Phase G PR-G1 of the realignment work.
Architectural decision: extended the existing `headroom/cli/wrap.py`
module in-place rather than splitting it into a `headroom/cli/wrap/`
package. The spec at REALIGNMENT/09-phase-G-rtk-observability.md
mentions per-agent files under a package, but the existing five wrap
subcommands all live in the single module and the extension is small
relative to the file. Keeping the file together preserves the simple
import surface used by tests (`from headroom.cli import wrap as wrap_mod`).
Per-agent wiring:
- cline → injects RTK block into `.clinerules` at project root
(Cline is a VS Code extension; API base URL is configured in the UI,
so the command prints config instructions and blocks on the proxy).
- continue → injects RTK guidance into the `systemMessage` field of
`.continue/config.json` (idempotent; refuses malformed JSON or
non-object roots; supports `--config` for custom paths).
- goose → injects RTK block into `.goosehints` at project root and
launches the goose CLI with OPENAI_BASE_URL / OPENAI_API_BASE /
ANTHROPIC_BASE_URL env vars pointed at the proxy.
- openhands → injects RTK guidance via the `OPENHANDS_INSTRUCTIONS`
env var at launch (no on-disk artifact) and sets OPENAI_BASE_URL /
ANTHROPIC_BASE_URL / LLM_BASE_URL. Preserves any pre-existing
OPENHANDS_INSTRUCTIONS content.
Tests:
- tests/test_cli/test_wrap_cline.py: 4 tests covering prepare-only
injection, idempotence, --no-context-tool, and existing content
preservation.
- tests/test_cli/test_wrap_continue.py: 8 tests covering the new
`_inject_continue_rtk_systemmessage` helper (new-file, existing
keys, idempotence, malformed JSON, non-object roots) and the click
command surface (default path, custom --config).
- tests/test_cli/test_wrap_goose.py: 5 tests covering env-var wiring,
`.goosehints` injection, idempotence, missing-binary error, and
--no-context-tool.
- tests/test_cli/test_wrap_openhands.py: 6 tests covering env-var
wiring, OPENHANDS_INSTRUCTIONS injection, preservation of existing
instructions, idempotence, missing-binary error, and
--no-context-tool.
E2E: extended `e2e/wrap/run.py` with `--prepare-only` smoke tests for
all four new wrappers (full launches require agent CLIs not present in
the e2e image; unit tests cover the env-var wiring).
The wrap-e2e harness passed `--startup-timeout-ms 5000` to `headroom
wrap openclaw`, leaving zero slack for the openclaw plugin's auto-start
launcher to bring up the headroom proxy before the 5s health-check
deadline. On a busy CI runner, cold Python import of `headroom.cli` plus
pyo3 dlopen plus FastAPI app boot routinely lands in the 4–8s range, so
this was always a coin-flip.
Evidence: run 25897154424 failed on main with the exact code that
passed pre-merge on PR #474's docker-wrap-e2e check (run 25897085244).
Both logs show identical openclaw "Config warnings" output — that's
normal noise, not the cause. The differentiating line is
`[plugins] Headroom proxy started and reachable` (pass) vs
`[plugins] Headroom proxy unavailable: health check failed` (fail).
30s matches what other wrap-e2e callers already use as a working margin
for the headroom proxy boot path; the runtime default for `headroom
wrap openclaw --startup-timeout-ms` is 20s.
Three files modified in the previous commit (4071d57) needed ruff
format reformatting per CI's `ruff format --check .` step.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Restore build_provider_section() to headroom/providers/codex/install.py
without requires_openai_auth (was removed entirely; pre-existing test
test_provider_codex_install.py imports it and would fail to collect)
- Flip test_codex_provider_section_preserves_openai_oauth to assert
requires_openai_auth is ABSENT, not present (old behavior was wrong)
- Fix test_provider_codex_runtime.py:337 same way — init config must
NOT contain requires_openai_auth
- Fix Ruff B023 lint error in test_providers.py:492 — capture loop
variable config_path in lambda default arg (_p=config_path)
- Fix e2e/init/run.py _verify_codex_local and _verify_codex_global to
assert requires_openai_auth is absent, not present
- Fix e2e/wrap/run.py verify_codex_wrap same way
All unit tests pass locally (82 affected tests green).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Preserve Codex OAuth-safe provider config across init, wrap, and
persistent install paths, and strengthen coverage so Codex requests
are proven to reach Headroom and the mock upstream.
The wrap e2e now sends a real chat-completions probe and checks
Headroom /stats. Runtime tests cover temporary launch env, install
env, init config, provider-scope config delivery, and the Python
3.11 ws bootstrap path.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
# Root cause of the wheel-build cascade
We have shipped 5 release-pipeline hot-fixes in 12 hours, each
addressing a different symptom of the same architectural problem:
1. PR #363 — npm artifact downloads + tried `yum openssl-devel`
2. PR #367 — vendored OpenSSL in `headroom-proxy` + dropped Intel mac
3. PR #369 — Debian-cross perl install (`perl` not `libipc-cmd-perl`)
4. PR #370 — moved `openssl/vendored` from headroom-proxy to headroom-py
5. (this PR) — ELIMINATE OpenSSL entirely
Each fix exposed a different missing system package or feature flag in
a different build surface (manylinux x86_64 vs aarch64-cross-Debian vs
macOS Intel vs e2e/wrap Dockerfile vs e2e/init Dockerfile vs main
Dockerfile vs devcontainer). We were playing whack-a-mole because every
Cargo dep change to the OpenSSL surface required matching system-package
updates in 6+ different Dockerfiles and workflows, and the PR-level CI
didn't exercise all of them.
# Why this PR is the structural fix
`fastembed` exposes clean rustls feature flags:
- `hf-hub-rustls-tls` (replaces default `hf-hub-native-tls`)
- `ort-download-binaries-rustls-tls` (replaces default `…native-tls`)
By disabling fastembed's default features and enabling the rustls
variants explicitly, we remove `native-tls` (and therefore `openssl-sys`,
`openssl`, `openssl-src`, perl modules, OpenSSL build-time deps,
vendored OpenSSL ~30s build cost) from the entire workspace dep tree.
Verified locally:
$ cargo tree -p headroom-py -i openssl-sys
error: package ID specification `openssl-sys` did not match any packages
$ cargo tree -p headroom-py -i native-tls
error: package ID specification `native-tls` did not match any packages
$ cargo build --release -p headroom-py
Finished `release` profile [optimized] target(s) in 25.57s
(Down from 1m+ with vendored OpenSSL.)
# Cleanups enabled by this change
- crates/headroom-py/Cargo.toml — dropped the `openssl/vendored`
workaround from PR #370.
- crates/headroom-proxy/Cargo.toml — same dep removed.
- e2e/wrap/Dockerfile — dropped `yum install openssl-devel pkgconfig
perl-IPC-Cmd`. Comment retained explaining why.
- e2e/init/Dockerfile — same.
- Dockerfile (main) — dropped `pkg-config libssl-dev` from apt-get.
- .devcontainer/Dockerfile — dropped `pkg-config libssl-dev`.
- .github/workflows/release.yml — removed the entire before-script-linux
block (perl install probe + multi-package-manager dispatch + fail-loud
assertion). No longer needed.
# Regression gate
Three new structural tests in tests/test_release_workflows.py:
- test_no_openssl_sys_in_wheel_build_tree — runs `cargo tree -p <crate>
-i openssl-sys` for headroom-py / headroom-proxy / headroom-core. If
openssl-sys reappears (a future native-tls enabler creeping in via a
new dep), this fails AT PR TIME with an actionable message.
- test_no_native_tls_in_wheel_build_tree — same shape, native-tls is
the proximate cause.
- test_fastembed_uses_rustls_features — checks the Cargo.toml so a
future "let me bump fastembed and forget the features" doesn't
silently re-introduce OpenSSL.
Plus two cleanup gates:
- test_dockerfiles_no_longer_install_openssl_devel
- test_release_yml_does_not_install_openssl_or_perl_for_wheels
All 13 release-workflow tests pass. `make ci-precheck` PASSED.
# What this teaches us about rollouts (per user's ultrathink ask)
The 5-fix cascade exposed three meta-problems:
1. PR checks don't block merges. PR #370 had docker-init-e2e,
docker-wrap-e2e, docker-native-e2e all FAILED yet got merged.
Branch protection should require these checks. Operator action
needed (cannot fix in code).
2. Local validation is misleading. `cargo build -p headroom-py` from
the workspace root used the workspace lockfile and looked green;
CI did fresh resolution against headroom-py's manifest alone where
the feature wasn't enabled. Lesson: verify structural invariants
with `cargo tree -e features` before trusting that a build "works."
3. 6+ build surfaces with independent system-dep state. Every Cargo
change required matching updates in 6 places. The structural answer
(this PR) is to NOT depend on system OpenSSL at all. Where structural
fixes are not possible, the answer is a single shared
scripts/install-rust-build-deps.sh — but with this PR there's
nothing left to install.
PR #360's previous attempt (multi-stage manylinux_2_28 build) still
failed with the same `__isoc23_strtoll` undefined-symbol ImportError.
Local repro showed the wheel built inside manylinux_2_28 has THREE
glibc 2.38+ C23 symbol references (`__isoc23_strtol`, `__isoc23_strtoll`,
`__isoc23_strtoull`) embedded by one of our transitive C/C++ deps
during cc-rs compilation — most likely libstdc++'s `<cstdlib>` resolving
`std::strtoll` to the C23 variant when the manylinux toolchain has
newer-glibc-aware headers. We can't easily fix the source of that
emission downstream.
Path of least resistance: switch the e2e runtime stage from a
glibc-2.36 base to one with glibc 2.38+. Verified on Mac (linux/arm64
native): the same wheel that fails on `node:22-bookworm` (glibc 2.36)
imports cleanly on `python:3.11-slim` (now trixie, glibc 2.41).
## Changes
- e2e/init/Dockerfile: stage 2 base `node:22-trixie` →
`python:3.11-slim`. The init harness only needs Python; no Node 22.
Drops apt-get install of python3/python3-pip/python3-venv (already in
the base image) and the `ln -sf` python alias.
- e2e/wrap/Dockerfile: stage 2 base `node:22-bookworm` →
`python:3.11-slim`. The wrap harness needs both Python 3.11
(aider-chat==0.86.2 requires Python <3.12) AND Node 22 (codex,
openclaw). Trixie's default python3 is 3.13 — too new for aider —
so we build on top of `python:3.11-slim` (trixie + py 3.11) and
install Node 22 from NodeSource.
- Both: stage 1 `--interpreter` reverted from python3.13 to python3.11
to match the runtime.
## Verification (local, linux/arm64)
docker buildx build -f e2e/wrap/Dockerfile.aarch64-test \
--platform linux/arm64 -t headroom-wrap-test .
→ stage 1 manylinux build green
→ stage 2 `from headroom._core import DiffCompressor` → OK
→ stage 2 aider-chat install in progress (separate venv)
## Production-side note (out of scope for this PR)
`pip install headroom-ai` from PyPI on a glibc-2.36 host (e.g. Debian
12, Ubuntu 22.04) will hit the same ImportError once the wheel matrix
publishes. python:3.X-slim is now trixie (glibc 2.41) for ALL of
3.10/3.11/3.12/3.13, so users on those base images are unaffected.
Tracking the underlying cc-rs symbol-emission bug as a separate issue.
## Two distinct failures on PR #360
### docker-init-e2e + docker-wrap-e2e + docker-native-e2e
Building headroom-ai from source inside `node:22-bookworm` produced a
`_core.so` that referenced `__isoc23_strtoll` (a glibc 2.38+ symbol).
The same image's runtime libc.so.6 (whatever it actually ships) can't
resolve it at import time:
ImportError: /workspace/headroom/_core.cpython-311-x86_64-linux-gnu.so:
undefined symbol: __isoc23_strtoll
Most likely cause: cc-rs invoking the bookworm gcc against headers that
have C23 wrappers exposed (libc6-dev backport, gcc 13 default mode, or
something similar), generating object code that references a symbol the
runtime libc.so doesn't actually have.
Fix: multi-stage docker build. Stage 1 builds the wheel inside
`quay.io/pypa/manylinux_2_28_x86_64` (AlmaLinux 8, glibc 2.28 baseline).
Stage 2 (node:22-bookworm) just installs the prebuilt wheel — no rust
toolchain needed at runtime, no build inside the runtime image. Same
pattern release.yml already uses for cross-platform wheel matrix.
Removed `COPY headroom/` and `COPY pyproject.toml` from the runtime
stage to prevent the source-only `headroom/` from shadowing the
installed wheel via cwd (Python would import the .py-only package and
miss `_core.so`).
### test (3.10/3.11/3.12/3.13)
The release-workflows test asserts the literal `needs:` list of the
create-release job. The single-wheel maturin refactor added
`build-wheels` and `collect-dist` jobs between `build` and the publish
jobs; create-release now waits for those too. Updated the assertion +
added explicit checks for the new `needs.<job>.result == 'success'`
guards.
rust-toolchain.toml at the repo root requests
`components = ["rustfmt", "clippy"]`. When `pip install -e .` invokes
maturin → cargo from inside `/workspace`, rustup auto-detects the
toolchain file and tries to add the missing components on top of the
`--profile minimal` install we did earlier. The install fails with:
info: downloading component clippy
info: rolling back changes
error: failed to install component: 'rustfmt-preview-x86_64-unknown-linux-gnu',
detected conflict: 'bin/cargo-fmt'
— rustup's auto-component install hits a `bin/cargo-fmt` conflict
inside the toolchain it just installed. The fix is to install the
required components up-front via `-c rustfmt -c clippy`, so the
toolchain matches what rust-toolchain.toml expects on first cargo run
and rustup never needs to mutate it.
Applied to: Dockerfile (main), e2e/init/Dockerfile, e2e/wrap/Dockerfile,
.devcontainer/Dockerfile. Also pinned the main Dockerfile's toolchain
from `stable` to `1.95.0` so all four images now match the lockfile
(prevents drift if rust-toolchain.toml is bumped later).
Eliminates the dual-package architecture that was the root cause of #355.
`pip install headroom-ai` now produces ONE wheel containing both the Python
source (headroom/*.py) and the compiled Rust extension (headroom/_core.so).
No more separate `headroom-core-py` package, no more chicken-and-egg with
PyPI publication, no more wheelhouse / PIP_FIND_LINKS / composite-action
plumbing in CI.
This is the canonical pattern used by cryptography, polars, ruff,
pydantic-core, and other Rust-as-core Python packages. Honors the
"Rust as core engine" direction.
## What changed
- pyproject.toml: `[build-system]` swapped from hatchling to maturin.
`[tool.hatch.*]` deleted; `[tool.maturin]` added pointing at
`crates/headroom-py/Cargo.toml` for the cdylib. `python-source = "."`
picks up the root `headroom/` package directly (dashboard HTML
templates and other non-Python files included automatically).
- crates/headroom-py/pyproject.toml: deleted. The crate is no longer a
separate published package; its Cargo.toml stays as the cdylib build
target invoked via `[tool.maturin] manifest-path`.
- crates/headroom-py/python/: deleted (placeholder layout for the old
separate package).
## CI updates
- ci.yml: `test` / `test-extras` / `test-agno` jobs simplified — Rust
toolchain set up before `pip install -e .` (which now invokes maturin
via build-system). Removed the "build wheel + symlink .so" dance.
`build` job swapped from `python -m build` (hatch) to
`maturin build` + `maturin sdist`.
- release.yml: collapsed dual-package matrix into one. New `build-wheels`
matrix produces cross-platform wheels for cp310/11/12/13 ×
{linux x86_64, linux aarch64, macos x86_64, macos aarch64}. New
`collect-dist` aggregator merges artifacts. publish-pypi consumes the
merged dist.
- init-native-e2e.yml: dropped windows-latest from the matrix —
upstream `esaxx-rs` (/MT) and `ort-sys` (/MD) link with conflicting
MSVC C runtime libraries, so the Rust extension cannot build for
win_amd64 today. Tracked as a follow-up; not a blocker for Linux+macOS.
- headroom-e2e-setup: composite action now sets up Rust toolchain +
Swatinem/rust-cache before `pip install -e .[proxy]`.
- eval.yml, publish.yml, rust.yml: same pattern — rust toolchain before
install. rust.yml's wheels job builds from root pyproject.toml (no
more `-m crates/headroom-py/Cargo.toml`).
- e2e/init/Dockerfile, e2e/wrap/Dockerfile: install rust + maturin in
the build stage; copy `crates/` + workspace `Cargo.toml/lock` so the
install can build the extension. Dropped `HEADROOM_REQUIRE_RUST_CORE=false`
from wrap-e2e — the image now ships the full Rust core.
- Dockerfile (main): simplified — no more Layer 2/3 dance with
`headroom-core-py` install + symlink. Single `uv pip install` builds
+ installs everything.
- .devcontainer/Dockerfile: rust toolchain + libssl-dev + maturin
added so `uv sync` builds the extension inside the devcontainer.
## Lockfile + script
- uv.lock: regenerated. No `headroom-core-py` entries remain.
- scripts/build_rust_extension.sh: simplified from a symlink-into-tree
workaround to a thin wrapper around `pip install -e .`. The maturin
build-backend handles placement automatically.
## Local validation (all green on macOS aarch64)
1. Clean venv `pip install -e .` → `from headroom._core import …` works.
2. `maturin build --release` → 13.8 MB wheel, 336 files including
`headroom/_core.cpython-311-darwin.so` (32 MB cdylib) and
`headroom/dashboard/templates/dashboard.html`.
3. `pip install <wheel>` in fresh venv → import works.
4. Wheel contents verified via `unzip -l`.
5. `pytest tests/test_transforms/test_diff_compressor.py` — 29 passed.
6. `pytest tests/test_relevance.py` — 30 passed.
7. `cargo build --workspace` + `cargo test --workspace` — all green.
8. `make ci-precheck` — 176 Python tests + Rust + commitlint green.
## Migration notes
Users on `pip install headroom-ai` get the Rust core automatically
(linux + macos wheels). sdist installs require rust toolchain available
locally — pip will build via maturin.
Closes#355
Supersedes #357 (workarounds-based fix abandoned in favor of
architectural fix)
Two CI failures introduced by Hotfix-A0's deployment-stage smoke test:
1. docker-native-e2e: the new maturin step in the builder stage failed
with "Could not find openssl via pkg-config". The workspace
transitively depends on `openssl-sys` (via reqwest's native-tls
path in some dep chain). The previous Dockerfile only installed
`build-essential`/`g++`/`curl`/`ca-certificates` — enough for the
proxy binary build because cached target/ artefacts already had
openssl-sys compiled, but the fresh maturin invocation hits a cold
build and needs the dev headers. Add `pkg-config` + `libssl-dev`.
2. docker-wrap-e2e: this image is a `node:22-bookworm` base that
installs headroom in editable mode for CLI-routing-only tests
(aider, codex, openclaw via the wrap subcommand). It deliberately
does NOT build the Rust extension. After A0, the proxy
`lifespan` startup refuses to start when `headroom._core` can't
import — so the wrap-e2e proxy port never opens, the harness's
/health check times out, and the test fails. The wrap-e2e scope
doesn't cover compression behaviour, so set
`HEADROOM_REQUIRE_RUST_CORE=false` to start in degraded
Python-only mode. Compression is exercised end-to-end by the
smoke-test and docker-native-e2e jobs which build via the main
Dockerfile.
The remaining 3 PR check failures (validate * 3) were transient
PyPI download failures (`nvidia-cuda-cupti-cu12==12.8.90`,
`safetensors==0.7.0`) — unrelated to the realignment branch; they
need a re-run, not a code change.
Align Docker-native wrapper help and runtime behavior with the Python install contract, including persistent deployment metadata, baked install-image defaults, and explicit unsupported wrap targets.
Harden the Python persistent-install path with profile validation, safer provider-scope handling, Windows environment restoration, runtime parity improvements, and rollback-safe apply/update behavior.
Update README, Docker install docs, CI, and focused regressions to cover the Windows BOM failure, wrapper parity, compose coverage, and Docker-native wrap behavior.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
TemporaryDirectory cleanup fails with OSError when OpenClaw gateway
leaves behind lock/session files. ignore_cleanup_errors=True lets
the test pass while the OS cleans up /tmp on reboot.
Make the Docker wrap e2e harness validate live proxy env wiring for Codex and Aider, start a real OpenClaw gateway in-container, and clear the repo-wide Ruff issues that were keeping the Python 3.12 CI job red.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a Docker-based end-to-end harness that validates Headroom's Codex, Aider, Cursor, and OpenClaw wrap flows without calling real model providers.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
