akadmin/headroom
1
0
Fork 0
mirror of https://github.com/headroomlabs-ai/headroom.git synced 2026-08-27 14:17:10 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
2630 commits 410 branches 169 tags 142 MiB
Commit graph

8 commits

Author SHA1 Message Date
Rudimar Ronsoni
b4571cc346
feat: headroom wrap opencode / unwrap opencode CLI (#1105) 
## Summary

This PR implements transparent `headroom wrap opencode` support without
asking users to edit OpenCode provider URLs, choose an extra CLI flag,
or maintain a static provider list.

The wrapper now lives at the runtime transport boundary: OpenCode keeps
its user/provider config, while Headroom intercepts outbound provider
traffic in-process and routes it through the local Headroom proxy.

## What changed

### Transparent OpenCode wrapping

- `headroom wrap opencode` injects the `headroom-opencode` plugin
through `OPENCODE_CONFIG_CONTENT`.
- Existing OpenCode provider URLs are preserved. We do not rewrite user
config URLs to point at Headroom.
- Existing `OPENAI_BASE_URL` and `ANTHROPIC_BASE_URL` env vars are
preserved.
- Local OpenCode traffic, localhost traffic, and Headroom proxy traffic
bypass the shim to avoid loops.

### Runtime transport interception

- Added an OpenCode plugin transport shim that wraps:
  - `globalThis.fetch`
  - `http.request` / `http.get`
  - `https.request` / `https.get`
- External provider calls are routed to the local Headroom proxy.
- The original upstream origin is passed through `x-headroom-base-url`,
so the proxy can forward to the real provider without changing OpenCode
config.
- External `http2.connect` is blocked loudly instead of allowing direct
provider traffic to leak outside Headroom.

### Live provider additions

Provider coverage is no longer based on a static config scan. Because
routing happens at outbound request time, providers added mid-session
are routed through Headroom automatically as long as they use the
covered Node transport paths.

### Subagent and child-process coverage

- The parent OpenCode plugin sets a packaged Node preload shim through
`NODE_OPTIONS=--import=.../hook-shim/handler.js`.
- The transport shim patches `child_process.spawn`, `exec`, `execFile`,
and `fork` so child Node processes receive the Headroom preload even
when OpenCode passes a custom `env`.
- The child-process shim fails closed if it loads without
`HEADROOM_OPENCODE_TRANSPORT_PROXY_URL`.
- This closes the subagent leak path where a child Node process could
otherwise start without Headroom transport interception.

## Why this goes beyond PR #1089

PR #1089 improves OpenCode provider registration, but it still focuses
on provider config shape. This PR moves the enforcement boundary to
runtime transport interception.

This PR goes further because:

- No provider URL rewriting is required.
- New providers added mid-session are covered automatically.
- Subagents and child Node processes inherit the Headroom transport
shim.
- Direct external HTTP/2 paths fail loudly instead of leaking.
- The wrap remains transparent to the user's OpenCode provider config.
- The wrapper is fail-closed for unsupported child-process preload
state.

## Additional robustness fixes

While validating the change in Docker, the full Python suite exposed
unrelated Linux/container robustness issues. These are fixed in this PR
so the suite is green:

- Binary cache handling now treats cache paths under a non-writable
existing parent as unavailable, including when tests run as root in
Docker.
- `release_version.py` honors `MANUAL_VER` before git calls so direct
script execution works outside a `.git` checkout.
- Test logger isolation now resets relevant Headroom child loggers so
proxy logging setup cannot poison later `caplog` tests.
- The scanner missing-path test now uses a guaranteed missing `tmp_path`
child instead of relying on `/nonexistent/path`.

## Validation

All implementation validation was run inside Docker.

- Full Python suite from a fresh Docker copy: `6605 passed, 523
skipped`.
- Ruff on changed Python/OpenCode paths: passed.
- OpenCode plugin typecheck: passed.
- OpenCode plugin tests: `9 passed`.
- OpenCode plugin build: passed.
- Hook shim preload smoke test: passed.

## Notes

This PR intentionally does not add a CLI option. `headroom wrap
opencode` means full wrap. Either Headroom wraps OpenCode transparently,
or the path fails loudly instead of silently leaking provider traffic.

---------

Co-authored-by: Rudimar Ronsoni <6081613+rudironsoni@users.noreply.github.com>
 2026-06-22 11:07:12 -05:00
chopratejas
6f2c0a8400 fix(ci): rustls-everywhere — eliminate openssl-sys from build tree 
# Root cause of the wheel-build cascade

We have shipped 5 release-pipeline hot-fixes in 12 hours, each
addressing a different symptom of the same architectural problem:

1. PR #363 — npm artifact downloads + tried `yum openssl-devel`
2. PR #367 — vendored OpenSSL in `headroom-proxy` + dropped Intel mac
3. PR #369 — Debian-cross perl install (`perl` not `libipc-cmd-perl`)
4. PR #370 — moved `openssl/vendored` from headroom-proxy to headroom-py
5. (this PR) — ELIMINATE OpenSSL entirely

Each fix exposed a different missing system package or feature flag in
a different build surface (manylinux x86_64 vs aarch64-cross-Debian vs
macOS Intel vs e2e/wrap Dockerfile vs e2e/init Dockerfile vs main
Dockerfile vs devcontainer). We were playing whack-a-mole because every
Cargo dep change to the OpenSSL surface required matching system-package
updates in 6+ different Dockerfiles and workflows, and the PR-level CI
didn't exercise all of them.

# Why this PR is the structural fix

`fastembed` exposes clean rustls feature flags:
- `hf-hub-rustls-tls`               (replaces default `hf-hub-native-tls`)
- `ort-download-binaries-rustls-tls` (replaces default `…native-tls`)

By disabling fastembed's default features and enabling the rustls
variants explicitly, we remove `native-tls` (and therefore `openssl-sys`,
`openssl`, `openssl-src`, perl modules, OpenSSL build-time deps,
vendored OpenSSL ~30s build cost) from the entire workspace dep tree.

Verified locally:

    $ cargo tree -p headroom-py -i openssl-sys
    error: package ID specification `openssl-sys` did not match any packages

    $ cargo tree -p headroom-py -i native-tls
    error: package ID specification `native-tls` did not match any packages

    $ cargo build --release -p headroom-py
    Finished `release` profile [optimized] target(s) in 25.57s

(Down from 1m+ with vendored OpenSSL.)

# Cleanups enabled by this change

- crates/headroom-py/Cargo.toml — dropped the `openssl/vendored`
  workaround from PR #370.
- crates/headroom-proxy/Cargo.toml — same dep removed.
- e2e/wrap/Dockerfile — dropped `yum install openssl-devel pkgconfig
  perl-IPC-Cmd`. Comment retained explaining why.
- e2e/init/Dockerfile — same.
- Dockerfile (main) — dropped `pkg-config libssl-dev` from apt-get.
- .devcontainer/Dockerfile — dropped `pkg-config libssl-dev`.
- .github/workflows/release.yml — removed the entire before-script-linux
  block (perl install probe + multi-package-manager dispatch + fail-loud
  assertion). No longer needed.

# Regression gate

Three new structural tests in tests/test_release_workflows.py:

- test_no_openssl_sys_in_wheel_build_tree — runs `cargo tree -p <crate>
  -i openssl-sys` for headroom-py / headroom-proxy / headroom-core. If
  openssl-sys reappears (a future native-tls enabler creeping in via a
  new dep), this fails AT PR TIME with an actionable message.
- test_no_native_tls_in_wheel_build_tree — same shape, native-tls is
  the proximate cause.
- test_fastembed_uses_rustls_features — checks the Cargo.toml so a
  future "let me bump fastembed and forget the features" doesn't
  silently re-introduce OpenSSL.

Plus two cleanup gates:
- test_dockerfiles_no_longer_install_openssl_devel
- test_release_yml_does_not_install_openssl_or_perl_for_wheels

All 13 release-workflow tests pass. `make ci-precheck` PASSED.

# What this teaches us about rollouts (per user's ultrathink ask)

The 5-fix cascade exposed three meta-problems:

1. PR checks don't block merges. PR #370 had docker-init-e2e,
   docker-wrap-e2e, docker-native-e2e all FAILED yet got merged.
   Branch protection should require these checks. Operator action
   needed (cannot fix in code).

2. Local validation is misleading. `cargo build -p headroom-py` from
   the workspace root used the workspace lockfile and looked green;
   CI did fresh resolution against headroom-py's manifest alone where
   the feature wasn't enabled. Lesson: verify structural invariants
   with `cargo tree -e features` before trusting that a build "works."

3. 6+ build surfaces with independent system-dep state. Every Cargo
   change required matching updates in 6 places. The structural answer
   (this PR) is to NOT depend on system OpenSSL at all. Where structural
   fixes are not possible, the answer is a single shared
   scripts/install-rust-build-deps.sh — but with this PR there's
   nothing left to install.
 2026-05-03 23:26:04 -07:00
chopratejas
73a4782917 fix(ci): switch e2e runtime to python:3.11-slim (trixie, glibc 2.41) 
PR #360's previous attempt (multi-stage manylinux_2_28 build) still
failed with the same `__isoc23_strtoll` undefined-symbol ImportError.
Local repro showed the wheel built inside manylinux_2_28 has THREE
glibc 2.38+ C23 symbol references (`__isoc23_strtol`, `__isoc23_strtoll`,
`__isoc23_strtoull`) embedded by one of our transitive C/C++ deps
during cc-rs compilation — most likely libstdc++'s `<cstdlib>` resolving
`std::strtoll` to the C23 variant when the manylinux toolchain has
newer-glibc-aware headers. We can't easily fix the source of that
emission downstream.

Path of least resistance: switch the e2e runtime stage from a
glibc-2.36 base to one with glibc 2.38+. Verified on Mac (linux/arm64
native): the same wheel that fails on `node:22-bookworm` (glibc 2.36)
imports cleanly on `python:3.11-slim` (now trixie, glibc 2.41).

## Changes

- e2e/init/Dockerfile: stage 2 base `node:22-trixie` →
  `python:3.11-slim`. The init harness only needs Python; no Node 22.
  Drops apt-get install of python3/python3-pip/python3-venv (already in
  the base image) and the `ln -sf` python alias.
- e2e/wrap/Dockerfile: stage 2 base `node:22-bookworm` →
  `python:3.11-slim`. The wrap harness needs both Python 3.11
  (aider-chat==0.86.2 requires Python <3.12) AND Node 22 (codex,
  openclaw). Trixie's default python3 is 3.13 — too new for aider —
  so we build on top of `python:3.11-slim` (trixie + py 3.11) and
  install Node 22 from NodeSource.
- Both: stage 1 `--interpreter` reverted from python3.13 to python3.11
  to match the runtime.

## Verification (local, linux/arm64)

  docker buildx build -f e2e/wrap/Dockerfile.aarch64-test \
    --platform linux/arm64 -t headroom-wrap-test .

→ stage 1 manylinux build green
→ stage 2 `from headroom._core import DiffCompressor` → OK
→ stage 2 aider-chat install in progress (separate venv)

## Production-side note (out of scope for this PR)

`pip install headroom-ai` from PyPI on a glibc-2.36 host (e.g. Debian
12, Ubuntu 22.04) will hit the same ImportError once the wheel matrix
publishes. python:3.X-slim is now trixie (glibc 2.41) for ALL of
3.10/3.11/3.12/3.13, so users on those base images are unaffected.
Tracking the underlying cc-rs symbol-emission bug as a separate issue.
 2026-05-03 14:30:46 -07:00
chopratejas
b31a34b4ac fix(ci): multi-stage manylinux build for e2e dockerfiles + release workflow test 
## Two distinct failures on PR #360

### docker-init-e2e + docker-wrap-e2e + docker-native-e2e

Building headroom-ai from source inside `node:22-bookworm` produced a
`_core.so` that referenced `__isoc23_strtoll` (a glibc 2.38+ symbol).
The same image's runtime libc.so.6 (whatever it actually ships) can't
resolve it at import time:

    ImportError: /workspace/headroom/_core.cpython-311-x86_64-linux-gnu.so:
      undefined symbol: __isoc23_strtoll

Most likely cause: cc-rs invoking the bookworm gcc against headers that
have C23 wrappers exposed (libc6-dev backport, gcc 13 default mode, or
something similar), generating object code that references a symbol the
runtime libc.so doesn't actually have.

Fix: multi-stage docker build. Stage 1 builds the wheel inside
`quay.io/pypa/manylinux_2_28_x86_64` (AlmaLinux 8, glibc 2.28 baseline).
Stage 2 (node:22-bookworm) just installs the prebuilt wheel — no rust
toolchain needed at runtime, no build inside the runtime image. Same
pattern release.yml already uses for cross-platform wheel matrix.

Removed `COPY headroom/` and `COPY pyproject.toml` from the runtime
stage to prevent the source-only `headroom/` from shadowing the
installed wheel via cwd (Python would import the .py-only package and
miss `_core.so`).

### test (3.10/3.11/3.12/3.13)

The release-workflows test asserts the literal `needs:` list of the
create-release job. The single-wheel maturin refactor added
`build-wheels` and `collect-dist` jobs between `build` and the publish
jobs; create-release now waits for those too. Updated the assertion +
added explicit checks for the new `needs.<job>.result == 'success'`
guards.
 2026-05-03 14:08:25 -07:00
chopratejas
2ae57725e7 fix(ci): pre-install rustfmt+clippy components in all Dockerfiles 
rust-toolchain.toml at the repo root requests
`components = ["rustfmt", "clippy"]`. When `pip install -e .` invokes
maturin → cargo from inside `/workspace`, rustup auto-detects the
toolchain file and tries to add the missing components on top of the
`--profile minimal` install we did earlier. The install fails with:

    info: downloading component clippy
    info: rolling back changes
    error: failed to install component: 'rustfmt-preview-x86_64-unknown-linux-gnu',
           detected conflict: 'bin/cargo-fmt'

— rustup's auto-component install hits a `bin/cargo-fmt` conflict
inside the toolchain it just installed. The fix is to install the
required components up-front via `-c rustfmt -c clippy`, so the
toolchain matches what rust-toolchain.toml expects on first cargo run
and rustup never needs to mutate it.

Applied to: Dockerfile (main), e2e/init/Dockerfile, e2e/wrap/Dockerfile,
.devcontainer/Dockerfile. Also pinned the main Dockerfile's toolchain
from `stable` to `1.95.0` so all four images now match the lockfile
(prevents drift if rust-toolchain.toml is bumped later).
 2026-05-03 13:35:01 -07:00
chopratejas
2a91cbb4b4 refactor: single-wheel maturin build backend (fixes #355) 
Eliminates the dual-package architecture that was the root cause of #355.
`pip install headroom-ai` now produces ONE wheel containing both the Python
source (headroom/*.py) and the compiled Rust extension (headroom/_core.so).
No more separate `headroom-core-py` package, no more chicken-and-egg with
PyPI publication, no more wheelhouse / PIP_FIND_LINKS / composite-action
plumbing in CI.

This is the canonical pattern used by cryptography, polars, ruff,
pydantic-core, and other Rust-as-core Python packages. Honors the
"Rust as core engine" direction.

## What changed

- pyproject.toml: `[build-system]` swapped from hatchling to maturin.
  `[tool.hatch.*]` deleted; `[tool.maturin]` added pointing at
  `crates/headroom-py/Cargo.toml` for the cdylib. `python-source = "."`
  picks up the root `headroom/` package directly (dashboard HTML
  templates and other non-Python files included automatically).
- crates/headroom-py/pyproject.toml: deleted. The crate is no longer a
  separate published package; its Cargo.toml stays as the cdylib build
  target invoked via `[tool.maturin] manifest-path`.
- crates/headroom-py/python/: deleted (placeholder layout for the old
  separate package).

## CI updates

- ci.yml: `test` / `test-extras` / `test-agno` jobs simplified — Rust
  toolchain set up before `pip install -e .` (which now invokes maturin
  via build-system). Removed the "build wheel + symlink .so" dance.
  `build` job swapped from `python -m build` (hatch) to
  `maturin build` + `maturin sdist`.
- release.yml: collapsed dual-package matrix into one. New `build-wheels`
  matrix produces cross-platform wheels for cp310/11/12/13 ×
  {linux x86_64, linux aarch64, macos x86_64, macos aarch64}. New
  `collect-dist` aggregator merges artifacts. publish-pypi consumes the
  merged dist.
- init-native-e2e.yml: dropped windows-latest from the matrix —
  upstream `esaxx-rs` (/MT) and `ort-sys` (/MD) link with conflicting
  MSVC C runtime libraries, so the Rust extension cannot build for
  win_amd64 today. Tracked as a follow-up; not a blocker for Linux+macOS.
- headroom-e2e-setup: composite action now sets up Rust toolchain +
  Swatinem/rust-cache before `pip install -e .[proxy]`.
- eval.yml, publish.yml, rust.yml: same pattern — rust toolchain before
  install. rust.yml's wheels job builds from root pyproject.toml (no
  more `-m crates/headroom-py/Cargo.toml`).
- e2e/init/Dockerfile, e2e/wrap/Dockerfile: install rust + maturin in
  the build stage; copy `crates/` + workspace `Cargo.toml/lock` so the
  install can build the extension. Dropped `HEADROOM_REQUIRE_RUST_CORE=false`
  from wrap-e2e — the image now ships the full Rust core.
- Dockerfile (main): simplified — no more Layer 2/3 dance with
  `headroom-core-py` install + symlink. Single `uv pip install` builds
  + installs everything.
- .devcontainer/Dockerfile: rust toolchain + libssl-dev + maturin
  added so `uv sync` builds the extension inside the devcontainer.

## Lockfile + script

- uv.lock: regenerated. No `headroom-core-py` entries remain.
- scripts/build_rust_extension.sh: simplified from a symlink-into-tree
  workaround to a thin wrapper around `pip install -e .`. The maturin
  build-backend handles placement automatically.

## Local validation (all green on macOS aarch64)

1. Clean venv `pip install -e .` → `from headroom._core import …` works.
2. `maturin build --release` → 13.8 MB wheel, 336 files including
   `headroom/_core.cpython-311-darwin.so` (32 MB cdylib) and
   `headroom/dashboard/templates/dashboard.html`.
3. `pip install <wheel>` in fresh venv → import works.
4. Wheel contents verified via `unzip -l`.
5. `pytest tests/test_transforms/test_diff_compressor.py` — 29 passed.
6. `pytest tests/test_relevance.py` — 30 passed.
7. `cargo build --workspace` + `cargo test --workspace` — all green.
8. `make ci-precheck` — 176 Python tests + Rust + commitlint green.

## Migration notes

Users on `pip install headroom-ai` get the Rust core automatically
(linux + macos wheels). sdist installs require rust toolchain available
locally — pip will build via maturin.

Closes #355
Supersedes #357 (workarounds-based fix abandoned in favor of
architectural fix)
 2026-05-03 13:16:41 -07:00
chopratejas
55dfc19e1d fix(ci): unblock A0 Docker e2e — install pkg-config + opt out wrap-e2e 
Two CI failures introduced by Hotfix-A0's deployment-stage smoke test:

1. docker-native-e2e: the new maturin step in the builder stage failed
   with "Could not find openssl via pkg-config". The workspace
   transitively depends on `openssl-sys` (via reqwest's native-tls
   path in some dep chain). The previous Dockerfile only installed
   `build-essential`/`g++`/`curl`/`ca-certificates` — enough for the
   proxy binary build because cached target/ artefacts already had
   openssl-sys compiled, but the fresh maturin invocation hits a cold
   build and needs the dev headers. Add `pkg-config` + `libssl-dev`.

2. docker-wrap-e2e: this image is a `node:22-bookworm` base that
   installs headroom in editable mode for CLI-routing-only tests
   (aider, codex, openclaw via the wrap subcommand). It deliberately
   does NOT build the Rust extension. After A0, the proxy
   `lifespan` startup refuses to start when `headroom._core` can't
   import — so the wrap-e2e proxy port never opens, the harness's
   /health check times out, and the test fails. The wrap-e2e scope
   doesn't cover compression behaviour, so set
   `HEADROOM_REQUIRE_RUST_CORE=false` to start in degraded
   Python-only mode. Compression is exercised end-to-end by the
   smoke-test and docker-native-e2e jobs which build via the main
   Dockerfile.

The remaining 3 PR check failures (validate * 3) were transient
PyPI download failures (`nvidia-cuda-cupti-cu12==12.8.90`,
`safetensors==0.7.0`) — unrelated to the realignment branch; they
need a re-run, not a code change.
 2026-05-02 18:43:09 -07:00
JerrettDavis
1967859ef6 feat(ci): add docker wrap e2e workflow 
Add a Docker-based end-to-end harness that validates Headroom's Codex, Aider, Cursor, and OpenClaw wrap flows without calling real model providers.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-07 22:36:43 -05:00