The /subscription-window endpoint cached the last `tracker.state` snapshot
verbatim. The tracker polls Anthropic's /api/oauth/usage every 5 minutes
(aggressive polling risks 429s / OAuth-token flagging). When the user's
5-hour window rolls over between two polls, the cached `utilization_pct`
described the OLD window — the dashboard rendered e.g. 44% while Claude
Code itself showed 0%. (Issue #281)
Fix: display-time synthesis. We already have all the data needed locally:
- snapshot.five_hour.resets_at — the API-reported reset boundary.
- session_tracking.compute_window_tokens — transcript-derived token
counts we can scope to [resets_at, now] to estimate usage in the new
window.
When `now >= resets_at`, render_state() synthesizes:
- used = local transcript tokens since resets_at (capped at limit; we
undercount tokens spent on Claude Code outside this proxy and must
never report >100%).
- utilization_pct = used / limit * 100.
- resets_at = next_reset (advanced by window_duration; marked
`resets_at_estimated=True`).
- synthesized=True.
When `now < resets_at`, the cached snapshot is returned verbatim with
`synthesized=False`. Backward compatible: every existing tracker.state
key is preserved; only new keys (synthesized, resets_at_estimated,
optional render_warning) are added per window dict.
Also adds maybe_poll_on_demand(): a 60s-floored singleton poll triggered
on dashboard load. Bounded across users (well within Anthropic
tolerance), wrapped in asyncio.wait_for(2s) so a slow upstream never
blocks the request handler. Exceptions are swallowed and logged.
All synthesis decisions emit structured logs:
event=subscription_window_synthesized window=... used=... limit=...
event=subscription_render_synthesis_failed (warn fallback)
event=subscription_on_demand_poll_triggered/skipped_floor/timeout/failed
Tests (12 new, all green):
- render within window returns cached pct
- render after reset synthesizes from local tokens
- render after reset with zero local tokens => 0%
- render capped at 100%
- render handles missing resets_at gracefully
- render preserves existing state keys (backward compat)
- render with no snapshot returns base state
- render synthesis fallback path logs and returns cached
- synthesize helper handles None window
- synthesize helper advances reset multiple windows when dashboard
loaded long after reset (e.g. machine asleep)
- maybe_poll_on_demand singleton 60s floor (mock-counted)
- maybe_poll_on_demand swallows API failures
Closes#281
# Root cause of the wheel-build cascade
We have shipped 5 release-pipeline hot-fixes in 12 hours, each
addressing a different symptom of the same architectural problem:
1. PR #363 — npm artifact downloads + tried `yum openssl-devel`
2. PR #367 — vendored OpenSSL in `headroom-proxy` + dropped Intel mac
3. PR #369 — Debian-cross perl install (`perl` not `libipc-cmd-perl`)
4. PR #370 — moved `openssl/vendored` from headroom-proxy to headroom-py
5. (this PR) — ELIMINATE OpenSSL entirely
Each fix exposed a different missing system package or feature flag in
a different build surface (manylinux x86_64 vs aarch64-cross-Debian vs
macOS Intel vs e2e/wrap Dockerfile vs e2e/init Dockerfile vs main
Dockerfile vs devcontainer). We were playing whack-a-mole because every
Cargo dep change to the OpenSSL surface required matching system-package
updates in 6+ different Dockerfiles and workflows, and the PR-level CI
didn't exercise all of them.
# Why this PR is the structural fix
`fastembed` exposes clean rustls feature flags:
- `hf-hub-rustls-tls` (replaces default `hf-hub-native-tls`)
- `ort-download-binaries-rustls-tls` (replaces default `…native-tls`)
By disabling fastembed's default features and enabling the rustls
variants explicitly, we remove `native-tls` (and therefore `openssl-sys`,
`openssl`, `openssl-src`, perl modules, OpenSSL build-time deps,
vendored OpenSSL ~30s build cost) from the entire workspace dep tree.
Verified locally:
$ cargo tree -p headroom-py -i openssl-sys
error: package ID specification `openssl-sys` did not match any packages
$ cargo tree -p headroom-py -i native-tls
error: package ID specification `native-tls` did not match any packages
$ cargo build --release -p headroom-py
Finished `release` profile [optimized] target(s) in 25.57s
(Down from 1m+ with vendored OpenSSL.)
# Cleanups enabled by this change
- crates/headroom-py/Cargo.toml — dropped the `openssl/vendored`
workaround from PR #370.
- crates/headroom-proxy/Cargo.toml — same dep removed.
- e2e/wrap/Dockerfile — dropped `yum install openssl-devel pkgconfig
perl-IPC-Cmd`. Comment retained explaining why.
- e2e/init/Dockerfile — same.
- Dockerfile (main) — dropped `pkg-config libssl-dev` from apt-get.
- .devcontainer/Dockerfile — dropped `pkg-config libssl-dev`.
- .github/workflows/release.yml — removed the entire before-script-linux
block (perl install probe + multi-package-manager dispatch + fail-loud
assertion). No longer needed.
# Regression gate
Three new structural tests in tests/test_release_workflows.py:
- test_no_openssl_sys_in_wheel_build_tree — runs `cargo tree -p <crate>
-i openssl-sys` for headroom-py / headroom-proxy / headroom-core. If
openssl-sys reappears (a future native-tls enabler creeping in via a
new dep), this fails AT PR TIME with an actionable message.
- test_no_native_tls_in_wheel_build_tree — same shape, native-tls is
the proximate cause.
- test_fastembed_uses_rustls_features — checks the Cargo.toml so a
future "let me bump fastembed and forget the features" doesn't
silently re-introduce OpenSSL.
Plus two cleanup gates:
- test_dockerfiles_no_longer_install_openssl_devel
- test_release_yml_does_not_install_openssl_or_perl_for_wheels
All 13 release-workflow tests pass. `make ci-precheck` PASSED.
# What this teaches us about rollouts (per user's ultrathink ask)
The 5-fix cascade exposed three meta-problems:
1. PR checks don't block merges. PR #370 had docker-init-e2e,
docker-wrap-e2e, docker-native-e2e all FAILED yet got merged.
Branch protection should require these checks. Operator action
needed (cannot fix in code).
2. Local validation is misleading. `cargo build -p headroom-py` from
the workspace root used the workspace lockfile and looked green;
CI did fresh resolution against headroom-py's manifest alone where
the feature wasn't enabled. Lesson: verify structural invariants
with `cargo tree -e features` before trusting that a build "works."
3. 6+ build surfaces with independent system-dep state. Every Cargo
change required matching updates in 6 places. The structural answer
(this PR) is to NOT depend on system OpenSSL at all. Where structural
fixes are not possible, the answer is a single shared
scripts/install-rust-build-deps.sh — but with this PR there's
nothing left to install.
PR #367 added `openssl = { features = ["vendored"] }` to
`crates/headroom-proxy/Cargo.toml`, expecting Cargo's feature
unification to propagate the vendored feature throughout the
workspace. PR #369 unblocked aarch64 by fixing the perl install.
The next release run on `1323830f` (the #369 merge) showed aarch64
+ macOS wheels build successfully but the **x86_64 Linux wheel still
fails** with:
The system library `openssl` required by crate `openssl-sys`
was not found.
# Why my previous reasoning was wrong
Cargo's feature unification only applies to the resolution graph
that's actually built. `maturin build --manifest-path
crates/headroom-py/Cargo.toml` resolves headroom-py's deps only —
headroom-py does NOT depend on headroom-proxy, so the
`openssl/vendored` enable in headroom-proxy never propagates to the
wheel build. `cargo tree -p headroom-py -e features` confirms:
openssl-sys is built with `default` only, no `vendored`.
(I missed this when I tested locally because `cargo build -p
headroom-py` from the workspace root reads the workspace's Cargo.lock,
which was already populated with `openssl-src` from when I added
the dep to headroom-proxy. The CI job runs `cargo rustc` with a
fresh resolution against headroom-py's manifest, where the vendored
feature isn't enabled.)
# Fix
Move the `openssl = { version = "0.10", features = ["vendored"] }`
dep from `headroom-proxy/Cargo.toml` to `headroom-py/Cargo.toml`.
The wheel-producing crate now declares the feature directly.
`cargo tree -p headroom-py -e features` after this change:
openssl-sys feature "vendored"
└── openssl-sys feature "openssl-src"
└── openssl-src feature "default"
└── openssl-src v300.6.0+3.6.2
Local `cargo build --release -p headroom-py` confirms openssl-src
is now compiled (35s wheel build).
# Tests
`test_headroom_py_vendors_openssl` (renamed from
`test_headroom_proxy_vendors_openssl`) gates the dep on the right
crate. Docstring explains WHY it must live in headroom-py — so a
future refactor doesn't move it back to headroom-proxy and silently
break the wheel build.
All 11 release-workflow tests pass. `make ci-precheck` PASSED.
Previous wheel hot-fix (#367) introduced a NEW failure mode that the
F1-merge release run surfaced:
E: Unable to locate package libipc-cmd-perl
The aarch64-unknown-linux-gnu maturin-action target does NOT use the
AlmaLinux 8 manylinux_2_28 image — it uses a Debian/Ubuntu-based
cross-compile container. The previous hot-fix's apt branch installed
`libipc-cmd-perl` which is a deprecated alias and is no longer in the
default Debian/Ubuntu sources. The build failed before openssl-src
could even start.
# What the script actually needs to do
`IPC::Cmd` is a Perl core module since 5.10, so any working `perl`
install provides it. The fix:
1. **Probe first.** `perl -MIPC::Cmd -e 1` exits cleanly when the
module is already importable — skip the install entirely. Some
manylinux images already ship it; others don't.
2. **Cover every package manager.** dnf (modern RHEL family) → yum
(older RHEL) → apt-get (Debian/Ubuntu) → apk (Alpine/musllinux).
The maturin-action uses different containers per (target, manylinux)
combo and we don't get to pick.
3. **Use `perl` not `libipc-cmd-perl` on Debian.** The plain `perl`
meta-package pulls `perl-modules-*` which contains IPC::Cmd. Works
on every Debian/Ubuntu version we'll see; `libipc-cmd-perl` is
gone from default sources.
4. **Fail loud after install.** `perl -MIPC::Cmd -e 'print "loaded
OK"'` runs unconditionally at the end. If somehow the module is
STILL missing, we fail here — not 5 minutes later in the
openssl-src compile step where the error message is harder to
debug. Matches the project's "no silent fallback" rule.
# Tests
`test_build_wheels_installs_perl_ipc_cmd_for_vendored_openssl`
updated to gate the new shape:
- Asserts `perl -MIPC::Cmd -e 1` pre-probe is present
- Asserts dnf/yum/apt-get/apk branches all exist
- Asserts apt branch installs `perl` not `libipc-cmd-perl`
- Asserts `libipc-cmd-perl` does not appear on any non-comment line
- Asserts the final `perl -MIPC::Cmd` fail-loud assertion is present
All 11 release-workflow tests pass. `make ci-precheck` PASSED.
The previous hot-fix (#363) addressed npm artifact downloads and added
openssl-devel installs in the manylinux container, but the wheel build
still fails on three of four matrix entries with three distinct errors:
1. ubuntu-x86_64 with `manylinux: auto` resolved to manylinux2014
(CentOS 7 / OpenSSL 1.0.2k). `openssl-sys 0.9` requires OpenSSL
1.1.0+ — "different version of OpenSSL was found".
2. ubuntu-aarch64 cross-compiles via `aarch64-unknown-linux-gnu-gcc`
from an x86_64 manylinux container. The `yum install openssl-devel`
we added installs x86_64 headers; `/usr/aarch64-unknown-linux-gnu/
include/` has no OpenSSL — "openssl/opensslv.h: No such file or
directory".
3. macos-15-intel fails on `ort-sys` (transitive via the ML compression
backend), which has no prebuilt ONNX Runtime binaries for
`x86_64-apple-darwin`. Unrelated to OpenSSL; an upstream limitation.
Why the workspace pulls openssl-sys at all: `hf-hub` (transitive via
`fastembed`) hard-codes `native-tls` as a default feature. Cargo's
feature unification then enables openssl-sys for the whole workspace
despite our `reqwest`/`tokio-tungstenite`/`tokio-rustls` preferences.
# Fix 1: vendored OpenSSL
Add `openssl = { version = "0.10", features = ["vendored"] }` to
`crates/headroom-proxy/Cargo.toml`. The `vendored` feature compiles
OpenSSL from source as part of the cargo build — works on every target
uniformly. Local build verified: cargo now pulls
`openssl-src v300.6.0+3.6.2` and compiles it. ~30s extra one-time
build cost.
The `openssl/vendored` feature DEFEATS `OPENSSL_DIR`. We therefore
remove the previous hot-fix's "Install OpenSSL (macOS)" step that
exported `OPENSSL_DIR` — leaving it would silently regress to the
system-OpenSSL path that broke originally.
# Fix 2: pin manylinux floor to 2_28
Change x86_64-unknown-linux-gnu from `manylinux: auto` to
`manylinux: 2_28` (matching aarch64 + the e2e Dockerfiles). This
isn't strictly required with vendored OpenSSL — the floor is now
glibc 2.28 / AlmaLinux 8 which has modern toolchain — but it removes
the CentOS-7 surface entirely and matches our runtime container
target.
# Fix 3: drop x86_64-apple-darwin from the matrix
`ort-sys 2.0.0-rc.12` has no prebuilt ONNX Runtime binaries for that
target. Building ORT from source would add CMake + ~5 minutes per
build. Apple Silicon macOS (`aarch64-apple-darwin`) is fully covered;
Intel-mac users install from the platform-independent sdist this
matrix also produces.
Tracked as a follow-up: switch the ML backend to `ort-tract` or
upstream a request for x86_64 macOS prebuilts.
# before-script-linux: keep perl-IPC-Cmd, drop openssl-devel
OpenSSL's vendored `Configure` script needs `IPC::Cmd` (without it
the build fails with "Can't locate IPC/Cmd.pm"). System
openssl-devel is no longer needed.
# Tests
4 new regression tests gate this:
- `test_headroom_proxy_vendors_openssl`
- `test_build_wheels_installs_perl_ipc_cmd_for_vendored_openssl`
- `test_build_wheels_does_not_set_openssl_dir`
- `test_build_wheels_matrix_excludes_intel_macos`
Plus the previous 7. All 11 release-workflow tests pass.
`make ci-precheck` PASSED. Local `cargo build --release -p headroom-py`
green.
Three independent failures on the post-merge release run for PR #360,
all introduced by the single-wheel maturin refactor:
1. Linux wheel matrix (`ubuntu-x86_64`, `aarch64-unknown-linux-gnu`)
failed inside the manylinux container with:
Could not find openssl via pkg-config
The system library `openssl` required by crate `openssl-sys`
was not found.
`openssl-sys` is transitive via `fastembed` → `hf-hub` → `ureq`
→ `native-tls`. The e2e Dockerfiles (`e2e/wrap`, `e2e/init`) we
wrote for #360 install `openssl-devel` upfront, but the
`build-wheels` matrix uses `PyO3/maturin-action@v1` which spins
up its OWN manylinux container that does not inherit those
installs. Fix: add a `before-script-linux:` to the action with a
yum/apt-get conditional so it works on RHEL-family (manylinux2014,
manylinux_2_28) and Debian-family musllinux variants.
2. macOS x86_64 wheel build failed with `maturin` exit 1 from the
same `openssl-sys` lookup. The aarch64 macos-14 runner happens
to have `/opt/homebrew/opt/openssl@3` on `openssl-sys`'s default
discovery path; the Intel macos-15-intel runner uses
`/usr/local/Cellar` which is NOT on that path. Fix: add a
pre-maturin step that runs `brew install openssl@3` and exports
`OPENSSL_DIR` / `OPENSSL_LIB_DIR` / `OPENSSL_INCLUDE_DIR` /
`PKG_CONFIG_PATH`. The aarch64 runner happily picks up the
explicit env vars too — no regression.
3. `publish-npm` and `publish-github-packages` both fail with
"Artifact not found for name: dist". Both jobs `npm pack` + `npm
publish` directly from the checked-out source tree — they never
consume the Python `dist` artifact. The `Download dist artifact`
step was vestigial dead code carried over from a prior workflow
shape; the only reason it didn't fail before #360 is that the
pre-refactor `build` job DID upload a `dist` artifact. Post-#360,
`dist` is produced by `collect-dist` and neither publish job is
gated on it (by design — npm vs PyPI ecosystems publish
independently). Fix: remove the dead download step from both
jobs. Loose coupling is preserved; `create-release` still gates
the GitHub Release tag on all of build / build-wheels /
collect-dist / publish-* succeeding.
Why the PR-level CI didn't catch any of this: `release.yml` only
runs on `push: branches: [main]`. PR #360's CI exercised `ci.yml`
which has a separate `ci-build-wheels-on-pr` matrix that uses a
different setup. The release surface only fires post-merge.
Tests added (regression gates):
- `test_build_wheels_installs_openssl_devel_on_linux_via_before_script`
- `test_build_wheels_resolves_openssl_dir_explicitly_on_macos`
- `test_npm_publish_jobs_do_not_download_dist_artifact`
All 9 release-workflow tests pass. `make ci-precheck` PASSED.
Phase D PR-D1 lands the first native Rust path for AWS Bedrock,
replacing the lossy LiteLLM Python converter for Anthropic-on-Bedrock
non-streaming requests. Eliminates part of P4-37 and P4-39.
What landed
-----------
- New crates/headroom-proxy/src/bedrock/ module:
- envelope.rs: parses the {"anthropic_version": "...", ...}
Bedrock body shape; re-emits with anthropic_version preserved
as the first key (relies on serde_json preserve_order).
- sigv4.rs: AWS SigV4 signer wrapping the aws-sigv4 crate.
Forces PayloadChecksumKind::XAmzSha256 so x-amz-content-sha256
is in the canonical request, hashed over the post-compression
body bytes (the bytes that actually hit Bedrock). No silent
fallback: signing failures return 5xx with
event=bedrock_sigv4_failed.
- invoke.rs: POST handler for /model/{model_id}/invoke
(and /converse - same wire shape for anthropic.claude-*).
Detects Anthropic vendor via literal starts_with("anthropic.")
(no regex per project rule), routes Anthropic-shape bodies
through the existing compress_anthropic_request live-zone
dispatcher, then signs and forwards to the configured Bedrock
endpoint.
- Modified:
- proxy.rs: routes /model/:model_id/invoke and
/model/:model_id/converse when enable_bedrock_native is on
(default). Adds bedrock_credentials: Option<Arc<Credentials>>
to AppState.
- config.rs: new flags --bedrock-region (default us-east-1,
env HEADROOM_PROXY_BEDROCK_REGION), --bedrock-endpoint
(operator override for FIPS/VPC/test setups),
--enable-bedrock-native (default true), --aws-profile.
- main.rs: resolves AWS credentials at startup via
aws_config::defaults(BehaviorVersion::latest()). Failure logs
event=bedrock_credentials_unavailable at WARN; the handler
refuses to forward unsigned (event=bedrock_credentials_missing).
- Cargo.toml: workspace deps aws-sigv4, aws-config,
aws-credential-types, aws-smithy-runtime-api.
Tests
-----
8 integration tests under
crates/headroom-proxy/tests/integration_bedrock_invoke.rs:
1. native_envelope_round_trip_byte_equal
2. sigv4_signed_correctly_after_compression - confirms
authorization is SigV4-shape and x-amz-content-sha256
matches sha256(body received by upstream).
3. thinking_block_preserved_through_bedrock
4. redacted_thinking_preserved
5. document_block_preserved
6. tool_result_array_with_image_preserved
7. stop_sequence_null_only_when_present - pins that the proxy
does NOT inject stop_sequence: null (P4-37 hardcode).
8. tool_use_input_byte_equal_preserves_key_order
All eight pass. Full workspace test run is green; clippy + fmt
clean. make ci-precheck (rust + python + commitlint) passes
locally.
Build constraints honoured
--------------------------
- No silent fallbacks: missing creds / signing failures return
5xx with structured event=... log; no path ever forwards
unsigned.
- No hardcodes: region, endpoint, profile, enable-flag all
configurable via CLI + env.
- No regexes: vendor detection is str::starts_with.
- Comprehensive structured logs: event=bedrock_invoke_received,
bedrock_envelope_parsed, bedrock_compression_skipped,
bedrock_credentials_missing, sigv4_signed,
bedrock_invoke_forwarded, etc.
- Performant: body buffered once, passed by &[u8] to signer
(zero-copy), Bytes::clone only for ownership transfer to
reqwest. Sign exactly once per request.
- Elegant: 4 small focused modules mirror handlers/ + sse/.
- Tests use realistic Anthropic block content (real thinking,
redacted_thinking, document, base64 image fixtures).
Streaming (PR-D2) and observability (PR-D3) follow.
Phase C realignment final step. The Anthropic↔OpenAI Responses↔Chat
Completions converter (`headroom/proxy/responses_converter.py`) was a
fragile shim that mishandled Codex `phase`, multi-text-part rebuild,
and unknown item types. It existed only because the Python compression
pipeline operates on Chat Completions messages and Responses items
needed to be coerced. After PR-C3 (Rust HTTP) and PR-C4 (Rust
streaming + Conversations awareness) the Rust handler at
`crates/headroom-proxy/src/handlers/responses.rs` processes Responses
items natively without converting between shapes, so the converter has
no remaining caller and is retired.
Changes:
- Delete `headroom/proxy/responses_converter.py` (336 lines).
- Delete `tests/test_responses_converter.py` (408 lines) and
`tests/test_proxy_responses_phase_preservation.py` (148 lines).
Coverage moves to `crates/headroom-proxy/tests/integration_responses*`.
- `handlers/openai.py::handle_openai_responses` (HTTP): drop the
converter import, the list-input → messages conversion, the
full compression dispatch (the `original_items is not None`
guard was its only caller), the inflation-revert block, and the
back-conversion. Memory injection is preserved via the existing
`append_text_to_latest_user_input_item` helpers which operate on
`body["input"]` directly. Telemetry vars stay zeroed.
- `handle_openai_responses_ws` (WebSocket): retire the first-frame
compression block plus the list-input branch in memory search.
WS sessions now pass through unmodified; WS-side compression is a
follow-up via Rust if ever needed.
- Drop the now-unused `previous_response_id` local and the inner
imports of `COMPRESSION_TIMEOUT_SECONDS`, `get_tokenizer`,
`extract_user_query` from the WS handler.
- Rename and update `test_handle_openai_responses_stream_keeps_compression`
to assert `apply.call_count == 0` — the new contract is that Python
compression on /v1/responses is retired.
Acceptance criteria:
- `git grep -n "responses_converter" -- headroom/ tests/` returns nothing.
- `pytest -x` green (4543 passed, 410 skipped).
- `cargo test --workspace` green (0 failures).
- Live validation against OpenAI through the proxy: /v1/chat/completions,
/v1/responses string + list input, /v1/responses SSE stream, and
ws:// /v1/responses all forward correctly without invoking compression.
Net diff in handlers/openai.py: +31 / -179.
Refs REALIGNMENT/05-phase-C-rust-proxy.md PR-C5.
`seq_claude_local` e2e assertion in e2e/init/run.py expects
`claude plugin marketplace add /workspace` but the actual command
was `claude plugin marketplace add chopratejas/headroom`.
Root cause: `_marketplace_source()` in headroom/cli/init.py walks
`Path(__file__).resolve().parents[2]` to find `.claude-plugin/
marketplace.json`. Before the single-wheel refactor, that path was
`/workspace/headroom/cli/init.py` -> parents[2] = `/workspace`,
where `.claude-plugin/marketplace.json` exists (COPY'd into the
e2e image). After the refactor, `headroom` is installed from a
wheel into site-packages, so `__file__` is now under
`/opt/headroom-venv/.../site-packages/headroom/cli/init.py` ->
parents[2] is the site-packages dir, which has no plugin manifest.
The function then falls back to the remote `chopratejas/headroom`.
Fix: set `HEADROOM_MARKETPLACE_SOURCE=/workspace` in the e2e/init
runtime ENV. The function honors this override before doing the
filesystem walk. The local `.claude-plugin/marketplace.json` is
already COPY'd into `/workspace/.claude-plugin/`, so the override
points at a valid source.
PR #360's previous attempt (multi-stage manylinux_2_28 build) still
failed with the same `__isoc23_strtoll` undefined-symbol ImportError.
Local repro showed the wheel built inside manylinux_2_28 has THREE
glibc 2.38+ C23 symbol references (`__isoc23_strtol`, `__isoc23_strtoll`,
`__isoc23_strtoull`) embedded by one of our transitive C/C++ deps
during cc-rs compilation — most likely libstdc++'s `<cstdlib>` resolving
`std::strtoll` to the C23 variant when the manylinux toolchain has
newer-glibc-aware headers. We can't easily fix the source of that
emission downstream.
Path of least resistance: switch the e2e runtime stage from a
glibc-2.36 base to one with glibc 2.38+. Verified on Mac (linux/arm64
native): the same wheel that fails on `node:22-bookworm` (glibc 2.36)
imports cleanly on `python:3.11-slim` (now trixie, glibc 2.41).
## Changes
- e2e/init/Dockerfile: stage 2 base `node:22-trixie` →
`python:3.11-slim`. The init harness only needs Python; no Node 22.
Drops apt-get install of python3/python3-pip/python3-venv (already in
the base image) and the `ln -sf` python alias.
- e2e/wrap/Dockerfile: stage 2 base `node:22-bookworm` →
`python:3.11-slim`. The wrap harness needs both Python 3.11
(aider-chat==0.86.2 requires Python <3.12) AND Node 22 (codex,
openclaw). Trixie's default python3 is 3.13 — too new for aider —
so we build on top of `python:3.11-slim` (trixie + py 3.11) and
install Node 22 from NodeSource.
- Both: stage 1 `--interpreter` reverted from python3.13 to python3.11
to match the runtime.
## Verification (local, linux/arm64)
docker buildx build -f e2e/wrap/Dockerfile.aarch64-test \
--platform linux/arm64 -t headroom-wrap-test .
→ stage 1 manylinux build green
→ stage 2 `from headroom._core import DiffCompressor` → OK
→ stage 2 aider-chat install in progress (separate venv)
## Production-side note (out of scope for this PR)
`pip install headroom-ai` from PyPI on a glibc-2.36 host (e.g. Debian
12, Ubuntu 22.04) will hit the same ImportError once the wheel matrix
publishes. python:3.X-slim is now trixie (glibc 2.41) for ALL of
3.10/3.11/3.12/3.13, so users on those base images are unaffected.
Tracking the underlying cc-rs symbol-emission bug as a separate issue.
## Two distinct failures on PR #360
### docker-init-e2e + docker-wrap-e2e + docker-native-e2e
Building headroom-ai from source inside `node:22-bookworm` produced a
`_core.so` that referenced `__isoc23_strtoll` (a glibc 2.38+ symbol).
The same image's runtime libc.so.6 (whatever it actually ships) can't
resolve it at import time:
ImportError: /workspace/headroom/_core.cpython-311-x86_64-linux-gnu.so:
undefined symbol: __isoc23_strtoll
Most likely cause: cc-rs invoking the bookworm gcc against headers that
have C23 wrappers exposed (libc6-dev backport, gcc 13 default mode, or
something similar), generating object code that references a symbol the
runtime libc.so doesn't actually have.
Fix: multi-stage docker build. Stage 1 builds the wheel inside
`quay.io/pypa/manylinux_2_28_x86_64` (AlmaLinux 8, glibc 2.28 baseline).
Stage 2 (node:22-bookworm) just installs the prebuilt wheel — no rust
toolchain needed at runtime, no build inside the runtime image. Same
pattern release.yml already uses for cross-platform wheel matrix.
Removed `COPY headroom/` and `COPY pyproject.toml` from the runtime
stage to prevent the source-only `headroom/` from shadowing the
installed wheel via cwd (Python would import the .py-only package and
miss `_core.so`).
### test (3.10/3.11/3.12/3.13)
The release-workflows test asserts the literal `needs:` list of the
create-release job. The single-wheel maturin refactor added
`build-wheels` and `collect-dist` jobs between `build` and the publish
jobs; create-release now waits for those too. Updated the assertion +
added explicit checks for the new `needs.<job>.result == 'success'`
guards.
Devcontainer validate jobs were failing on PR #360 with:
× Failed to fetch:
https://pypi.netflix.net/packages/.../nvidia_nvshmem_cu12-3.4.5-...whl
├─▶ Request failed after 3 retries
╰─▶ operation timed out
`pypi.netflix.net` is Netflix's internal PyPI mirror. It got into the
lockfile because my local `~/.config/uv/uv.toml` had:
index-url = "https://pypi.netflix.net/simple"
Running `uv lock` from that machine baked Netflix-internal URLs into
uv.lock for every package. Public CI runners (and any external
contributor) can't resolve them.
Two fixes:
1. Add `[[tool.uv.index]]` block to pyproject.toml pinning public PyPI
as the project's default index. uv now ignores user-level config when
resolving for this project, regardless of who runs `uv lock`. This
prevents the same contamination from any developer in the future.
2. Regenerate uv.lock against public PyPI. All package URLs now point
at `https://files.pythonhosted.org/...` and `https://pypi.org/simple/`.
Zero references to `pypi.netflix.net` remain in the lockfile.
Verified: `grep -c "pypi.netflix" uv.lock` returns 0.
PR #360's first run after the rustup-component fix exposed a deeper
issue: the devcontainer's manual rustup install in .devcontainer/Dockerfile
runs as root and leaves /usr/local/cargo root-owned. When post-create.sh
later runs `uv sync` as the `vscode` user, maturin → cargo can't write
to the registry cache:
warning: failed to write cache, path:
/usr/local/cargo/registry/index/.../aho-corasick,
error: Permission denied (os error 13)
The official ghcr.io/devcontainers/features/rust:1 feature solves this
by chowning the toolchain dirs to the runtime user (and adding them to
a `rustlang` group). It also handles component installation cleanly so
we don't need the manual `-c rustfmt -c clippy` workaround in the
container build.
Changes:
- .devcontainer/devcontainer.json: add rust feature with version=1.95.0,
profile=minimal, components=rustfmt,clippy
- .devcontainer/memory-stack/devcontainer.json: same
- .devcontainer/Dockerfile: drop manual rustup install (feature replaces
it). Keep the apt-get pkg-config + libssl-dev for openssl-sys, and
the maturin pip install.
Note: e2e/{init,wrap}/Dockerfile keep their manual rustup install
because they're stand-alone runtime images, not devcontainers.
rust-toolchain.toml at the repo root requests
`components = ["rustfmt", "clippy"]`. When `pip install -e .` invokes
maturin → cargo from inside `/workspace`, rustup auto-detects the
toolchain file and tries to add the missing components on top of the
`--profile minimal` install we did earlier. The install fails with:
info: downloading component clippy
info: rolling back changes
error: failed to install component: 'rustfmt-preview-x86_64-unknown-linux-gnu',
detected conflict: 'bin/cargo-fmt'
— rustup's auto-component install hits a `bin/cargo-fmt` conflict
inside the toolchain it just installed. The fix is to install the
required components up-front via `-c rustfmt -c clippy`, so the
toolchain matches what rust-toolchain.toml expects on first cargo run
and rustup never needs to mutate it.
Applied to: Dockerfile (main), e2e/init/Dockerfile, e2e/wrap/Dockerfile,
.devcontainer/Dockerfile. Also pinned the main Dockerfile's toolchain
from `stable` to `1.95.0` so all four images now match the lockfile
(prevents drift if rust-toolchain.toml is bumped later).
Code-scanning alert #65 (CodeQL actions/missing-workflow-permissions,
CWE-275) flagged the new build-wheels job for not declaring an explicit
permissions block. While at it, audit the rest of release.yml — same
gap exists on detect-version, build, collect-dist, and publish-npm.
Each job gets `contents: read` (the minimal default) since none of them
push, write packages, or mutate releases through GITHUB_TOKEN. Existing
write-bearing jobs (publish-pypi: id-token, publish-github-packages:
packages, create-release: contents) keep their narrower scopes.
Two early failures on PR #360's first CI run:
1. validate (default/memory-stack) + validate-worktree failed at the
apt-get update step in .devcontainer/Dockerfile. The base image
mcr.microsoft.com/devcontainers/python:1-3.12-bookworm ships with
dl.yarnpkg.com configured as an apt source whose GPG key has expired:
Err:4 https://dl.yarnpkg.com/debian stable InRelease
The following signatures couldn't be verified because the public
key is not available: NO_PUBKEY 62D54FD4003F6525
apt-get returns exit 100, the whole RUN aborts before pkg-config /
libssl-dev install. The maturin refactor doesn't need yarn, so drop
/etc/apt/sources.list.d/yarn.list before apt-get update. The debian
main repo updates fine on its own.
2. workflow-validation (actionlint) failed parsing rust.yml step name
"Build wheel (single-wheel architecture: builds headroom-ai)" —
actionlint's YAML parser saw the unquoted colon inside the name as
a mapping. Quote the string.
Eliminates the dual-package architecture that was the root cause of #355.
`pip install headroom-ai` now produces ONE wheel containing both the Python
source (headroom/*.py) and the compiled Rust extension (headroom/_core.so).
No more separate `headroom-core-py` package, no more chicken-and-egg with
PyPI publication, no more wheelhouse / PIP_FIND_LINKS / composite-action
plumbing in CI.
This is the canonical pattern used by cryptography, polars, ruff,
pydantic-core, and other Rust-as-core Python packages. Honors the
"Rust as core engine" direction.
## What changed
- pyproject.toml: `[build-system]` swapped from hatchling to maturin.
`[tool.hatch.*]` deleted; `[tool.maturin]` added pointing at
`crates/headroom-py/Cargo.toml` for the cdylib. `python-source = "."`
picks up the root `headroom/` package directly (dashboard HTML
templates and other non-Python files included automatically).
- crates/headroom-py/pyproject.toml: deleted. The crate is no longer a
separate published package; its Cargo.toml stays as the cdylib build
target invoked via `[tool.maturin] manifest-path`.
- crates/headroom-py/python/: deleted (placeholder layout for the old
separate package).
## CI updates
- ci.yml: `test` / `test-extras` / `test-agno` jobs simplified — Rust
toolchain set up before `pip install -e .` (which now invokes maturin
via build-system). Removed the "build wheel + symlink .so" dance.
`build` job swapped from `python -m build` (hatch) to
`maturin build` + `maturin sdist`.
- release.yml: collapsed dual-package matrix into one. New `build-wheels`
matrix produces cross-platform wheels for cp310/11/12/13 ×
{linux x86_64, linux aarch64, macos x86_64, macos aarch64}. New
`collect-dist` aggregator merges artifacts. publish-pypi consumes the
merged dist.
- init-native-e2e.yml: dropped windows-latest from the matrix —
upstream `esaxx-rs` (/MT) and `ort-sys` (/MD) link with conflicting
MSVC C runtime libraries, so the Rust extension cannot build for
win_amd64 today. Tracked as a follow-up; not a blocker for Linux+macOS.
- headroom-e2e-setup: composite action now sets up Rust toolchain +
Swatinem/rust-cache before `pip install -e .[proxy]`.
- eval.yml, publish.yml, rust.yml: same pattern — rust toolchain before
install. rust.yml's wheels job builds from root pyproject.toml (no
more `-m crates/headroom-py/Cargo.toml`).
- e2e/init/Dockerfile, e2e/wrap/Dockerfile: install rust + maturin in
the build stage; copy `crates/` + workspace `Cargo.toml/lock` so the
install can build the extension. Dropped `HEADROOM_REQUIRE_RUST_CORE=false`
from wrap-e2e — the image now ships the full Rust core.
- Dockerfile (main): simplified — no more Layer 2/3 dance with
`headroom-core-py` install + symlink. Single `uv pip install` builds
+ installs everything.
- .devcontainer/Dockerfile: rust toolchain + libssl-dev + maturin
added so `uv sync` builds the extension inside the devcontainer.
## Lockfile + script
- uv.lock: regenerated. No `headroom-core-py` entries remain.
- scripts/build_rust_extension.sh: simplified from a symlink-into-tree
workaround to a thin wrapper around `pip install -e .`. The maturin
build-backend handles placement automatically.
## Local validation (all green on macOS aarch64)
1. Clean venv `pip install -e .` → `from headroom._core import …` works.
2. `maturin build --release` → 13.8 MB wheel, 336 files including
`headroom/_core.cpython-311-darwin.so` (32 MB cdylib) and
`headroom/dashboard/templates/dashboard.html`.
3. `pip install <wheel>` in fresh venv → import works.
4. Wheel contents verified via `unzip -l`.
5. `pytest tests/test_transforms/test_diff_compressor.py` — 29 passed.
6. `pytest tests/test_relevance.py` — 30 passed.
7. `cargo build --workspace` + `cargo test --workspace` — all green.
8. `make ci-precheck` — 176 Python tests + Rust + commitlint green.
## Migration notes
Users on `pip install headroom-ai` get the Rust core automatically
(linux + macos wheels). sdist installs require rust toolchain available
locally — pip will build via maturin.
Closes#355
Supersedes #357 (workarounds-based fix abandoned in favor of
architectural fix)
Phase C PR-C4 of the Realignment. Completes the Responses surface
in the Rust proxy and lays the Conversations passthrough rails.
# /v1/responses streaming pipeline
C3 left an INFO-WARN breadcrumb (`responses_streaming_passthrough_until_c4`)
on every SSE-Accept request because the byte-level SSE framer +
ResponseState machine wired by C1 was not formally tied into the
endpoint's contract. C4 closes the loop:
- Replaces the C3 fallback warn with a structured-log INFO event
`event = "responses_streaming_pipeline_active"`. The bytes flow
unchanged (compression on streamed response output is OUT of scope
per live-zone-only contract); telemetry runs in the parallel task.
- New emergency-rollback toggle `--enable-responses-streaming`
(env: `HEADROOM_PROXY_ENABLE_RESPONSES_STREAMING`, default `true`).
When `false`, the OpenAI-Responses SSE state machine is skipped
and `event = "responses_streaming_state_machine_skipped"` is
emitted; bytes still pass through. Other providers' state
machines are unaffected.
- Request-side compression (the C3 live-zone dispatcher) continues
to run on streaming requests — `should_intercept` does not gate on
Accept, so SSE clients get the same body compression as
non-streaming clients.
# Conversations API surface (`/v1/conversations*`)
New `crates/headroom-proxy/src/handlers/conversations.rs` mounts
explicit axum routes (no regex per build constraints) for OpenAI's
stateful thread surface:
- `POST /v1/conversations`
- `GET /v1/conversations/{id}`
- `POST /v1/conversations/{id}` (metadata update)
- `DELETE /v1/conversations/{id}`
- `POST /v1/conversations/{id}/items`
- `GET /v1/conversations/{id}/items`
- `GET /v1/conversations/{id}/items/{item_id}`
- `DELETE /v1/conversations/{id}/items/{item_id}`
For PR-C4 each handler is passthrough-with-instrumentation: forward
upstream byte-equal via `forward_http`, emit
`event = "conversations_passthrough_pr_c4"` with route name +
extracted IDs. Compression of stored conversation items is C5+/B-phase
territory and explicitly NOT done here. Bodies are streamed (no
buffering) — `is_compressible_path` does not match
`/v1/conversations*` so the compression gate's else-branch streams
the body via `reqwest::Body::wrap_stream`.
New toggle `--enable-conversations-passthrough` (env:
`HEADROOM_PROXY_ENABLE_CONVERSATIONS_PASSTHROUGH`, default `true`).
When `false`, the per-route handlers are NOT mounted — requests
still reach upstream via the catch-all (no per-route logs); a
single WARN at app-build time confirms the rollback.
# Tests added
- `tests/integration_responses_streaming.rs` (4 tests):
request bytes byte-equal upstream on streaming; client receives
bytes that round-trip through the same SseFramer + ResponseState
the proxy spawns; rollback path still passes bytes;
below-threshold streaming request round-trips byte-equal.
- `tests/integration_conversations.rs` (10 tests):
every CRUD endpoint passthrough byte-equal through wiremock;
4xx upstream errors surface verbatim (no swallowing);
passthrough disabled still falls through to catch-all byte-equal.
- `tests/sse_openai_responses.rs` (+2 tests):
`chunk_boundary_invariance_pr_c4` — every single-byte split point
produces the same final state (cache-safety streaming property);
`minimal_upstream_response_pr_c4` — empty `[DONE]` upstream never
panics the state machine.
# Structured-log events introduced
- `responses_streaming_pipeline_active` (INFO) — replaces the C3 WARN
- `responses_streaming_pipeline_disabled` (WARN, only when toggle off)
- `responses_streaming_state_machine_skipped` (INFO, in proxy.rs)
- `conversations_passthrough_pr_c4` (INFO, per request)
- `conversations_passthrough_disabled` (WARN, app-build time)
# Config keys introduced
- `enable_responses_streaming: bool` (default `true`)
- `enable_conversations_passthrough: bool` (default `true`)
# Verification
- `cargo build --workspace --all-features`: green
- `cargo test --workspace --all-features`: 793 + 14 + ... all green
- `cargo clippy --workspace --all-features -- -D warnings`: zero
- `cargo fmt --all --check`: green
- `make ci-precheck-rust`: green
- `make ci-precheck-python` (against main repo .venv): 176 passed
No Python files modified; Conversations compression deferred to C5+
and follow-up B-phase work that retires
`headroom/proxy/responses_converter.py`.
Ports the OpenAI Responses API request path to Rust with first-class
per-item-type handling, replacing the fragile
`headroom/proxy/responses_converter.py` shim that flattens
Responses-shape items into Chat-Completions-shape (and silently
breaks every time OpenAI lands a new item type).
What lands:
- New `compress_openai_responses_live_zone` dispatcher in
`headroom-core` (sibling of the Anthropic / Chat Completions ones)
that walks the `input` array (with `messages` accepted as legacy
alias) and identifies the latest of each compressible kind:
`function_call_output`, `local_shell_call_output`,
`apply_patch_call_output`, plus the latest user-role `message`.
Earlier *_output items are FROZEN (cached prefix). All other item
types pass through verbatim via byte-range surgery.
- New 2 KiB output-item floor for `*_output` items (per spec PR-C3
line 167) on top of the existing per-content-type byte thresholds.
- `crates/headroom-proxy/src/responses_items.rs` exposes a typed
`ResponseItem<'a>` enum for telemetry / decision-making, paired
with a `ClassifiedItem` two-pass classifier that keeps the
original `&RawValue` slice so byte fidelity is preserved
independently of the typed view. `Cow<'a, str>` on string fields
handles both borrowed (no-escape) and owned (escape-bearing) JSON
values without allocation on the common path.
- `crates/headroom-proxy/src/compression/live_zone_responses.rs`
proxy-side dispatcher mirrors the Chat Completions shape (same
`Outcome` / `Passthrough` arms, same structured logs, same
manifest aggregation). Logs unknown `type` values at warn
(`event = responses_unknown_item_type`) and never strips them.
- `crates/headroom-proxy/src/handlers/responses.rs` POST handler
buffers the body and re-injects via `forward_http`. Detects
`Accept: text/event-stream` and emits
`event = responses_streaming_passthrough_until_c4` so we can
measure the volume before C4 wires the streaming state machine.
- Image-generation log redaction: `image_generation_call` items are
logged with byte size only (no `image_data` in the log path).
The upstream-bound bytes are NOT mutated — redaction is
log-channel only, per spec.
- `CompressibleEndpoint::OpenAiResponses` variant; route
`POST /v1/responses` wired in `proxy.rs`.
Tests added (~40 across the workspace):
- 9 core dispatcher unit tests in `live_zone.rs`
- 4 proxy dispatcher unit tests + 5 handler unit tests + 5 typed-enum
unit tests
- 16 integration tests in `tests/integration_responses.rs` covering:
V4A patch byte-equality, argv-array preservation, Codex
`phase=commentary`/`final_answer`, compaction passthrough,
reasoning passthrough, function_call.arguments string preservation,
call_id-vs-id distinction, 2 KiB output-item floor (below + above),
local_shell output compression, MCP / computer-use / image
generation passthrough, unknown-type warn-and-preserve, and a
representative round-trip with reasoning + function_call +
local_shell + apply_patch + custom items.
Per-PR-C3 plan: REALIGNMENT/05-phase-C-rust-proxy.md.
Adds a POST handler for /v1/chat/completions and a sibling live-zone
dispatcher for the OpenAI Chat Completions request shape. Same
compressor backend as Anthropic (SmartCrusher / LogCompressor /
SearchCompressor / DiffCompressor), same per-content-type byte
thresholds, same tokenizer-validated rejection gate, same byte-range
surgery for cache-stable rewrite.
Live zone for Chat Completions: the latest role=tool message's
content AND the latest role=user message's text content. Earlier
tool/user messages are part of the cache hot zone; never touched.
tools[] and tool_choice are never read or rewritten — they
round-trip byte-equal as a side effect of byte-range surgery.
Behaviours:
- n > 1 → passthrough (multiple completions imply non-determinism;
the proxy gate skips dispatch and forwards original bytes).
- stream: true → pass through to forward_http's existing C1 SSE
parser tee (ChunkState).
- tool_choice change → never mutated.
- mode == Off → passthrough with structured 'mode_off' log.
- Body not JSON / no messages → passthrough; the dispatcher logs
the decision and forwards original bytes.
The handler is wired as an explicit POST route on /v1/chat/completions,
buffers the body into Bytes, and re-injects it into the shared
forward_http function. forward_http's compression gate now classifies
the path (AnthropicMessages vs OpenAiChatCompletions) and dispatches
to the right module (compress_anthropic_request /
compress_openai_chat_request). Single forwarding code path keeps
SSE telemetry, header stripping, and request-id plumbing
single-source.
Files added:
- crates/headroom-proxy/src/handlers/chat_completions.rs
- crates/headroom-proxy/src/handlers/mod.rs
- crates/headroom-proxy/src/compression/live_zone_openai.rs
- crates/headroom-proxy/tests/integration_chat_completions.rs
Files modified:
- crates/headroom-core/src/transforms/live_zone.rs
(+compress_openai_chat_live_zone, +helpers)
- crates/headroom-core/src/transforms/mod.rs (re-export)
- crates/headroom-proxy/src/compression/mod.rs (+CompressibleEndpoint
classification, expose live_zone_openai)
- crates/headroom-proxy/src/lib.rs (expose handlers module)
- crates/headroom-proxy/src/proxy.rs (route + dispatch)
Tests:
- 7 integration tests in tests/integration_chat_completions.rs
covering passthrough byte-equality, tool message compression
(≥40% reduction on 1500-row JSON-array fodder), n>1 passthrough,
stream_options round-trip, tool_choice non-mutation, refusal
delta handling via ChunkState, and tool_call argument
accumulation across three streaming chunks.
- Unit tests on compress_openai_chat_live_zone (6) and
compress_openai_chat_request (7) cover the dispatcher and proxy
shim independently.
Workspace test count: 953 (after C1) → 975. fmt clean. clippy
--all-targets --all-features -D warnings clean. make ci-precheck
PASSED.
Plugin marketplace versions auto-bumped by the sync-plugin-versions
pre-commit hook.
Per-PR-C2 plan: REALIGNMENT/05-phase-C-rust-proxy.md.
The Release workflow's multi-arch publish-docker job failed after
78 minutes of QEMU-emulated arm64 cargo compilation. Maturin's
wheel-link repair step needs `patchelf` to bundle external
shared libraries (libssl.so.3, libcrypto.so.3, libzstd.so.1)
into the wheel and rewrite their RPATH:
🔗 External shared libraries to be copied into the wheel:
libssl.so.3 => /usr/lib/aarch64-linux-gnu/libssl.so.3
libzstd.so.1 => /usr/lib/aarch64-linux-gnu/libzstd.so.1.5.7
libcrypto.so.3 => /usr/lib/aarch64-linux-gnu/libcrypto.so.3
💥 maturin failed
Caused by: Failed to execute 'patchelf', did you install it?
Compounding chain:
1. PR #350 added pkg-config + libssl-dev to unblock the cargo build
(openssl-sys couldn't find OpenSSL headers).
2. That made Cargo dynamically link to libssl.
3. Maturin then needs patchelf to rewrite the wheel's RPATH so the
bundled .so references resolve at runtime.
4. patchelf was never installed → fail.
Why this didn't surface in PR CI: docker-native-e2e builds only
the host platform (amd64). The Release workflow's docker-bake
builds linux/amd64 + linux/arm64 via setup-qemu-action, and the
arm64 emulation chain hits the patchelf path (different bundling
heuristic from amd64).
Follow-up that's NOT in this hotfix:
The 78-minute QEMU compile is the bigger structural issue. Switching
the Release workflow to native arm64 runners (`runs-on:
ubuntu-24.04-arm`) would cut that to ~5 min. Filing separately.
Run that failed: 25268839539
Foundation of Phase C. Delivers:
* Byte-level SSE framing (bytes::Bytes / BytesMut) with UTF-8
decoded only at \n\n event boundaries — no per-chunk decode,
no errors=ignore data loss across TCP reads.
* Three provider state machines:
- Anthropic: blocks keyed by index, all delta types
(text/thinking/input_json/citations/signature) preserved
byte-equal.
- OpenAI Chat: ToolCallState concatenation, refusal field,
include_usage final chunk handling.
- OpenAI Responses: items keyed by id (not position) for
out-of-order completion; full event coverage.
* State machine runs in parallel with byte-passthrough via a
tokio::spawn task fed by a bounded mpsc — clients see raw
bytes immediately; telemetry populates without blocking.
Retires P1-8, P1-9, P1-14, P1-15, P1-17, P4-48 in the Rust path
(Python A8 hotfix preserved as fallback until Phase H).
Per-PR-C1 plan: REALIGNMENT/05-phase-C-rust-proxy.md.
The build-stage verify kept failing in PR #350 CI with
"ModuleNotFoundError: No module named 'headroom._core'" even after
the install order was correct. Diagnostic dump (commit 28a4883)
proved why:
headroom.__file__ = /build/headroom/__init__.py
headroom.__path__ = ['/build/headroom']
WORKDIR /build puts cwd at the front of sys.path for python -c.
Python resolves `import headroom` to /build/headroom/ — the source
tree COPYd in by Layer 3 — instead of
/usr/local/lib/python3.11/site-packages/headroom/ where the wheel
installed _core.so. The source tree has no _core.so, so the import
falsely fails.
Build-time-only quirk: production startup runs the proxy from a
different cwd where site-packages wins. The customer's box that
motivated A0 was hitting a different failure mode entirely (no
_core.so in the venv at all).
Fix: cd /tmp && python -c ... — /tmp has no headroom/ directory, so
import resolution falls through to site-packages, matching production
order. Removed the diagnostic preamble; it served its purpose.
Diagnostic step in the Dockerfile builder: list site-packages/headroom/
contents, run pip show -f on both headroom-core-py and headroom-ai,
print sys.path and headroom.__path__ before the import-verify. Lets us
see exactly what's on disk when A0's build-time verify keeps failing
in PR #350 CI. Will be removed once the wheel install order issue is
diagnosed.
The validate × 3 devcontainer CI failures were NOT environmental —
they were caused by this branch.
Root cause: commit 967b0db (PR-B1 big delete) was made on a Netflix
machine where uv was configured to use the internal mirror. The
subagent ran `uv lock` to regenerate after deleting deps, capturing
`pypi.netflix.net/simple` as the registry for every package and
`pypi.netflix.net/packages/<id>/<file>.whl` as the URL for every
wheel and sdist. main's lock points at public `pypi.org/simple` and
`files.pythonhosted.org/packages/...`.
When CI ran on GitHub Actions runners (no Netflix network access),
uv tried to fetch from `pypi.netflix.net` and timed out — surfacing
as "Failed to download cuda-bindings==12.9.4 / safetensors==0.7.0
/ nvidia-cuda-cupti-cu12==12.8.90 — request failed after 3 retries".
Devs running the same devcontainer locally on a Netflix machine
saw it work because their box could reach the internal mirror.
Fix: restore main's uv.lock and regenerate against public PyPI:
UV_INDEX_URL=https://pypi.org/simple \
UV_DEFAULT_INDEX=https://pypi.org/simple \
uv lock
The regenerated lock has 311 pypi.org URLs and 0 pypi.netflix.net
URLs. The pytest `live` marker added in Wave 3 was the only real
pyproject.toml change in the branch — no dep deltas — so the lock's
package set matches what main resolves modulo a handful of
transitive bumps (loguru, mmh3, py-rust-stemmers, win32-setctime,
pillow 11.3.0).
This is the correct lock for upstream CI. Anyone working on a
Netflix box should rely on uv's index-URL override at install time
(or pin via UV_INDEX_URL in their shell), NOT bake the internal
mirror into the canonical lockfile that ships in the repo.
PR #350 CI: docker-native-e2e's wheel install succeeded but the
build-stage verify (`from headroom._core import hello`) failed with
`ModuleNotFoundError: No module named 'headroom._core'`. Same failure
mode the customer hit in production (Finding #2) — but in CI we have
the full layer trace.
Root cause: the headroom-core-py wheel claims ownership of both
`headroom/__init__.py` (stub from maturin's python-source layout)
AND `headroom/_core.cpython-*.so`. The previous Dockerfile installed
headroom-ai FIRST (which laid down the real `headroom/` tree), then
the wheel SECOND with `--force-reinstall`. pip's --force-reinstall
uninstalls the wheel's previously installed files before reinstalling
— but the wheel's stub `__init__.py` had already overwritten
headroom-ai's at first install. Net result: pip deleted
`headroom/__init__.py` and `headroom/_core.so` ownership records
got into a state where the .so wasn't present after the install.
Fix: swap the order. Install the wheel first (lays down stub
`__init__.py` + `_core.so`), then install headroom-ai (overwrites the
stub with the real `__init__.py` and adds the rest of the
`headroom/` tree). `_core.so` survives because headroom-ai doesn't
claim ownership of it. Drop `--force-reinstall` from the wheel step
since nothing is installing the wheel before it.
This is the exact failure A0 was designed to catch — a deployment
that ships without `_core` working. CI is now serving as a
regression gate for the production install path.
The remaining 3 PR check failures (validate × 3 / Dev Containers)
are environmental: the runner's PyPI mirror (`pypi.netflix.net`)
times out fetching `cuda-bindings==12.9.4` /
`nvidia-cuda-cupti-cu12==12.8.90` / `safetensors==0.7.0`. These come
from `headroom-ai[dev]` → `sentence-transformers` → `torch` → CUDA
deps. Not caused by the realignment branch; the post-create script
needs a `--extra dev-light` profile or the mirror needs the packages
cached. Tracking separately.
Two CI failures introduced by Hotfix-A0's deployment-stage smoke test:
1. docker-native-e2e: the new maturin step in the builder stage failed
with "Could not find openssl via pkg-config". The workspace
transitively depends on `openssl-sys` (via reqwest's native-tls
path in some dep chain). The previous Dockerfile only installed
`build-essential`/`g++`/`curl`/`ca-certificates` — enough for the
proxy binary build because cached target/ artefacts already had
openssl-sys compiled, but the fresh maturin invocation hits a cold
build and needs the dev headers. Add `pkg-config` + `libssl-dev`.
2. docker-wrap-e2e: this image is a `node:22-bookworm` base that
installs headroom in editable mode for CLI-routing-only tests
(aider, codex, openclaw via the wrap subcommand). It deliberately
does NOT build the Rust extension. After A0, the proxy
`lifespan` startup refuses to start when `headroom._core` can't
import — so the wrap-e2e proxy port never opens, the harness's
/health check times out, and the test fails. The wrap-e2e scope
doesn't cover compression behaviour, so set
`HEADROOM_REQUIRE_RUST_CORE=false` to start in degraded
Python-only mode. Compression is exercised end-to-end by the
smoke-test and docker-native-e2e jobs which build via the main
Dockerfile.
The remaining 3 PR check failures (validate * 3) were transient
PyPI download failures (`nvidia-cuda-cupti-cu12==12.8.90`,
`safetensors==0.7.0`) — unrelated to the realignment branch; they
need a re-run, not a code change.
GitGuardian flagged two strings on PR #350 as leaked secrets. Both are
synthetic fixtures, NOT real credentials:
1. tests/test_cache_aligner_detector_only.py:215 — the canonical
`eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiIxIn0.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c`
JWT (header `{"alg":"HS256"}`, payload `{"sub":"1"}`) used to verify
our `detect_volatile_content` recognises JWT-shaped strings.
2. tests/test_realignment_live_multi_turn.py:1091 — Anthropic-shaped
tokens whose payloads literally contain "fixture"
(`sk-ant-api03-payg-fixture`, `sk-ant-oat01-oauth-fixture`,
`sk-ant-api03-payg-bearer-fixture`). Used to assert the auth-mode
classifier routes PAYG / OAuth headers correctly. No live API call
is ever made with these tokens — the test only inspects header
shape.
Two-layer remediation:
* `.gitguardian.yaml` (new) — explicit allowlist with the literal
match strings, each tagged with the file it lives in and the
rationale. Anything else GG flags should be treated as a real
incident; this file is the audit trail.
* Inline `# ggignore` + `# noqa: S105` comments on each fixture line
so a reviewer reading the test in isolation sees the intent without
having to cross-reference the config.
Per-feedback memory: secrets are routed via `.env`; the user's keys
were never in chat or version control. These rows document the
classifier-sweep false positive without weakening the detection rule.
When a placeholder is lost during compression, restore_tags now
discards the wrap rather than appending the original tag at the
trailing edge of the output. The old "append" fallback emitted
malformed XML — an opening tag with no body and no closing tag —
on ~350 production requests over 9 days. Per the proxy log
findings, the corruption pattern was `compressed-stuff <tag>`,
which downstream models interpret as a truncated message.
Concrete changes:
* `crates/headroom-core/src/transforms/tag_protector.rs`:
- `restore_tags` no longer accumulates `tail_appends`. Lost
placeholders are silently dropped from the output bytes.
- New `restore_tags_with_request_id` entry point threads an
optional request id into the structured ERROR log so the
proxy layer can wire request context end-to-end. PyO3 binding
keeps the existing 2-arg signature (no Python caller has a
request id today).
- `tag_lost_warn` is replaced by `tag_lost_error`. Severity
moves from WARN to ERROR with structured fields
(`event=tag_protector_placeholder_lost`, `tag_preview`,
`compressed_length`, `action=discarded_wrap`, optional
`request_id`) so operators can alert on the corruption rather
than have it disappear into a WARN line.
- `parse_tag_at` gained a bounds check after consuming a
leading '/' — proptest discovered an OOB on input `</`.
- The old `restore_lost_placeholder_appended` test (which
pinned the broken behavior) is replaced with three positive
tests: wrap-discard, idempotence on full loss, and
partial-loss-keeps-present-drops-lost.
- New proptest suite enforces three invariants over arbitrary
inputs: no introduced asymmetry, idempotence on full
placeholder loss, and no orphan-byte injection.
* `headroom/transforms/tag_protector.py`: docstring updated
to document the discard-wrap semantics — the prior text
("appended on the trailing edge") is now incorrect.
* `tests/test_tag_protector_invariant.py` (new): Python-side
invariant suite that exercises the same three properties
end-to-end through the public Python API. Uses a deterministic
seeded random walk (no `hypothesis` dependency) so CI is stable
and reproducible.
* `tests/test_transforms/test_tag_protector.py`: replaces the
broken-behavior test with the new wrap-discard semantics.
Per-finding-#3: ~/Desktop/HEADROOM_PROXY_LOG_FINDINGS_2026_05_03.md.
Production incident (Finding #2 of HEADROOM_PROXY_LOG_FINDINGS_2026_05_03.md):
on this customer's deployment the Rust extension `headroom._core` was
never installed into the runtime Docker image. Diff compression failed
54 times in a single day; "Optimization failed: ModuleNotFoundError" hit
379 times. The failure rate climbed every day and reached ~223/day on
2026-05-03 — effectively 100% of requests on the Rust path. Every Rust
PR we'd merged (MessageScorer, ICM, DiffCompressor, etc.) was providing
zero customer value because the module wasn't loadable at all.
Root cause: the Dockerfile builder stage installed Python deps and the
in-tree `headroom-ai` package but never ran `maturin build` for the
`headroom-py` crate, so the runtime image shipped without `_core.so`.
The Python proxy continued to start because the extension's absence is
caught and routed through Python-only fallbacks that either silently
no-op or raise per-request.
This change makes that mode impossible by default:
* `headroom.proxy.server._check_rust_core()` runs as the first step of
the FastAPI lifespan. If the import fails it prints a structured
diagnostic, logs `event=rust_core_missing`, and calls `sys.exit(78)`
(sysexits.h `EX_CONFIG`). Process supervisors (systemd / k8s /
docker) treat this as a deliberate config error and stop restart
loops.
* `HEADROOM_REQUIRE_RUST_CORE=false` is the explicit opt-out for
Python-only `pip install -e .` developer flows; lifespan logs
`event=rust_core_disabled` and continues. Any other value (including
unset) keeps the fail-loud default.
* `/health` now surfaces `rust_core: "loaded" | "disabled" | "missing"`
(plus `rust_core_error` when non-loaded) so operators can alert on
the degraded state rather than discovering it via a customer ticket.
* `scripts/build_rust_extension.sh` is the single dev-time path: build
→ install → import-verify with the same `hello()` marker the lifespan
checks. Failures are loud at every step.
* `Makefile` exposes the script as `make verify-rust-core`.
* `Dockerfile` now installs `rustup` + `maturin`, builds the wheel from
`crates/headroom-py`, force-installs it into site-packages, and runs
the same `hello()` import-verify in the build image so a broken build
fails the docker-build, not the next runtime restart.
Tests:
* `tests/test_rust_core_smoke.py` pins all four contracts:
- `_core.hello()` returns `"headroom-core"`
- missing extension + default env → `SystemExit(78)`
- missing extension + opt-out env → lifespan starts, `/health`
returns `rust_core: "disabled"` with the underlying error
- present extension + default env → `("loaded", None)`
Per-finding-#2: ~/Desktop/HEADROOM_PROXY_LOG_FINDINGS_2026_05_03.md.
Adds tests/test_realignment_live_multi_turn.py with 9 OPT-IN live tests
that validate the load-bearing claims of the Phase A+B megamerge against
real upstream APIs (Anthropic, OpenAI, Gemini). Each test maps to one or
more realignment PRs:
1. test_anthropic_cache_hit_across_two_turns — A2/A6/E
Identical cache_control'd system+messages on two turns must
eventually produce cache_read_input_tokens > 0. Guards the cache
hot zone invariant (I2): proxy must not mutate frozen prefix bytes.
Uses a bounded retry loop (max 4 attempts) to absorb Anthropic's
eventually-consistent prompt-cache write latency without masking
a real "proxy broke cache stability" regression.
2. test_anthropic_cache_stable_when_live_zone_compresses — B2/B3
Turn 2 mutates only the LATEST user content (8KB+ JSON tail);
cache_read on turn 2 must still be > 0 AND the proxy must emit
compression headers — proving the live-zone block dispatcher
ran on the new tail without disturbing the cached prefix.
3. test_anthropic_cache_control_passthrough_byte_faithful — A3/A4
Wraps proxy._retry_request to snapshot the upstream-bound body
and assert cache_control on system blocks survives verbatim,
and user content is not flattened from list to string form.
4. test_openai_chat_completions_multi_turn_through_proxy — A8/B
Three-turn conversation through /v1/chat/completions; each
turn returns valid content, prior assistant turns survive in
the messages list (proxy doesn't drop them).
5. test_openai_streaming_sse_chunks_arrive_in_order — A8 (SSE wire)
Streams /v1/chat/completions; asserts each event is
'data: ...\\n\\n', terminator is 'data: [DONE]\\n\\n',
reassembled content non-empty, no malformed events.
6. test_gemini_multi_turn_through_proxy — Gemini reach
Two-turn conversation through native
/v1beta/models/{model}:generateContent. Proves Gemini handler
wiring stayed intact through the megamerge.
7. test_ccr_marker_round_trip_live — B7 (CCR)
Pre-populates compression_store with a fixture entry, embeds
a CCR marker on a tool_result, verifies (a) headroom_retrieve
tool is injected into the upstream tools array (PR-B7
always-on), and (b) /v1/retrieve returns the original bytes
by hash with all rows intact. Pre-populating the Python store
(vs. driving SmartCrusher's internal Rust store) matches the
established pattern in tests/test_proxy_ccr.py and exercises
the surface served by /v1/retrieve.
8. test_memory_tail_injection_does_not_modify_system_prompt_live — B6/A2
Spins up a memory-enabled proxy with MemoryMode.AUTO_TAIL,
seeds LocalBackend, captures upstream-bound body. Asserts:
(a) system prompt byte-identical to input; (b) memory text
lands on latest user message tail; (c) earlier messages
untouched. Guards the live-zone-only injection contract.
9. test_classify_auth_mode_routes_payg_vs_oauth — Phase F-prep / B5
NOT a live API call. Sends three header shapes through the
proxy (x-api-key=..., Bearer sk-ant-oat01-..., Bearer
sk-ant-api03-...), captures dispatcher headers via a wrap on
_retry_request, and asserts the canonical auth-mode classifier
maps each correctly. Codifies the Phase F contract.
Conventions:
* file-level pytestmark = pytest.mark.live → excluded by default
via 'pytest -m "not live"'. Adds a 'live' marker registration in
pyproject.toml's [tool.pytest.ini_options].markers.
* each test skipif's on the relevant API key — no silent fallbacks,
no real-API runs against fake keys.
* uses tests/_dotenv.py helpers (load_env_overrides + autouse_apply_env)
rather than re-implementing env loading.
* model IDs and thresholds live in a top-of-file LIVE_CONFIG dict
(no hardcodes); Anthropic primary/fallback resolves at runtime per
key entitlement.
* assertions are direction-only (cache_read > 0, tokens_after <=
tokens_before) — never tied to upstream pricing/tokenizer drift.
* shared module-scoped TestClient fixture for performance; CCR and
memory tests build dedicated proxies for their config-specific paths.
Verification:
* pytest tests/test_realignment_live_multi_turn.py -v
→ 9 passed, 0 skipped, 0 failed in ~25s (with all keys set)
* pytest -m "not live" --tb=short -q
→ 4694 passed, 265 skipped, 9 deselected — same baseline as today
* make ci-precheck → green (rust + python + commitlint)
Per-realignment-plan: REALIGNMENT/04-phase-B-live-zone.md.
Two follow-ups surfaced when B6 and B7 were merged onto the megamerge
branch and the full suite ran:
1. tests/test_proxy_anthropic_cache_stability.py
PR-B7 added `injector.scan_for_markers(optimized_messages)` to the
Anthropic handler so the always-on tool-registration logic can see
detected hashes for the current request. The two pre-existing
`_FakeInjector` mocks (`test_ccr_system_instruction_injection_disabled_*`
and `test_ccr_tool_injection_disabled_*`) didn't implement that method.
Added a no-op `scan_for_markers` returning [] to both mocks — matches
the real injector's contract for the not-yet-compressed request shape
these tests exercise.
2. tests/test_memory_tool_mode.py::test_tool_mode_skip_emits_structured_log
The B6 caplog assertion passed in isolation but failed in the full
suite. Root cause: when an earlier test triggers proxy startup,
`_setup_file_logging` flips `headroom.propagate=False` and attaches a
RotatingFileHandler to the headroom logger. caplog captures via
propagation to root, so log records stop reaching it. The conftest
autouse fixture that resets `propagate=True` before every test gets
shadowed by fixture-ordering edge cases.
Principled fix: attach `caplog.handler` directly to
`headroom.proxy.memory_handler` for the duration of the test so the
capture is independent of propagation state. Restore the original
level + remove the handler in `finally` to keep the test hermetic.
Both B6 and B7 cherry-picks themselves are unmodified. This commit only
adjusts test harness code so the pre-existing mocks/capture stay
consistent with the new code paths.
