GitHub Actions deprecated the macos-13 runner label. The validate-workflows
actionlint step in CI fails because macos-13 is no longer in the available
labels list. macos-15-intel is the current x86_64 macOS runner.
(Bumped from macos-14 to macos-15 for arm64 was unnecessary; macos-14 is
still valid and we keep it for cache-warmth.)
cargo fmt --check failed in CI: import order in proxy.rs (cfg(test)
attributes before/after non-attr imports) and a few line-wrapping
nits in e2e_real.rs. Ran cargo fmt --all to fix.
maturin-action@v1 does not have a 'manifest-path' input — the action
warned 'Unexpected input(s) manifest-path' and proceeded to invoke
maturin from the repo root, which sees the workspace Cargo.toml with
no [package] section and bails. Move -m crates/headroom-py/Cargo.toml
back inside the 'args' string.
Cargo.lock: pick up tokio-util added in the WS half-close fix.
RUST_DEV.md: document how to run headroom-proxy in passthrough mode
(listen + upstream flags, e2e test gate, env vars).
Previously, any package registered under the headroom.proxy_extension
entry-point group auto-loaded at proxy startup. A user pip-installing a
plugin (or pulling one in transitively) would get its middleware running
in front of all their LLM traffic with zero opt-in or visibility — the
same mechanism that masked the Shield Enterprise streaming bug.
Change: install_all() now takes an explicit enabled set (or reads
HEADROOM_PROXY_EXTENSIONS). Discovery still runs to enumerate what's
available, but only names the operator opted into actually install.
The literal '*' is a wildcard for trusted environments.
CLI: headroom proxy --proxy-extension shield_enterprise
headroom proxy --proxy-extension shield_enterprise,mypkg
headroom proxy --proxy-extension '*'
Env: HEADROOM_PROXY_EXTENSIONS=shield_enterprise
The startup banner now shows discovered + enabled extensions:
Extensions: discovered=shield_enterprise (opt-in: --proxy-extension ...)
Extensions: ENABLED shield_enterprise (available: shield_enterprise)
Extensions: ENABLED (wildcard) shield_enterprise
Names that were requested but not found are logged as warnings.
Adds proxy_extensions: list[str] | None to ProxyConfig. Plumbs it
through CLI -> ProxyConfig -> install_all(enabled=...).
This is a behavior change for users who relied on auto-loading.
Existing Shield/extension users must add --proxy-extension or set
HEADROOM_PROXY_EXTENSIONS to keep their middleware running.
CodeQL alert #61 (CWE-275, actions/missing-workflow-permissions):
add explicit `permissions: contents: read` to the rust workflow root.
Defaults the GITHUB_TOKEN to read-only across all jobs, so even if the
repo policy changes, this workflow stays at least-privilege. No job in
this workflow needs write — wheels/audit/parity all read-only.
Add real end-to-end test suite at tests/e2e_real.rs gated behind
HEADROOM_E2E=1. Spawns the actual Python Headroom proxy as a subprocess,
runs the Rust proxy in-process in front of it, and exercises:
- health endpoints across the full chain
- Anthropic non-streaming (real API call)
- Anthropic streaming SSE (real API call) with chunk-level validation
- OpenAI non-streaming (real API call)
- X-Request-Id generation and pass-through
Adds tokio-process feature for Command/Child usage. Loads .env at the
repo root for API keys (does not log values). Tests skip cleanly when
HEADROOM_E2E is unset, so cargo test stays fast.
Bug 1 (HIGH) health.rs: Url::join('healthz') used relative resolution,
stripping non-trailing-slash base paths. Fixed with set_path('/healthz').
Bug 2 (HIGH) main.rs: graceful_shutdown_timeout was configured and logged
but never enforced. Now sleeps for the configured duration after signal
before axum exits, giving in-flight LLM streams time to drain.
Bug 3 (MEDIUM) websocket.rs: WS pump half-close could hang forever if
close() on one side failed. Replaced tokio::join! on async blocks with
spawned tasks + CancellationToken so either direction cancels the other.
Bug 4 (MEDIUM) proxy.rs/websocket.rs: URL path-join logic was copy-pasted
verbatim in two places. Extracted to join_upstream_path() helper; websocket
now calls it instead of duplicating the 15-line block.
Bug 5 (MEDIUM) proxy.rs: mid-stream upstream errors were silently swallowed
by Body::from_stream. Added a .map() wrapper that logs before re-raising.
Bug 6 (LOW) websocket.rs: WS session log was missing the request path,
making it hard to correlate logs with client sessions. Added path field.
Bug 7 (LOW) websocket.rs: scheme match arm 'ws'|'wss' borrowed joined
immutably while set_scheme needed a mutable borrow. Fixed by using literal
'ws' (set_scheme on an already-ws URL is a no-op for the ws case).
maturin>=1.5 requires -m to point to Cargo.toml, not pyproject.toml.
Fixes wheel build job failure in CI (all three matrix targets).
Also switches to manifest-path: action param for cleaner workflow syntax.
Applies same fix to Makefile build-wheel and develop targets.
15 integration tests across five suites that spin up the proxy on an
ephemeral port pointed at a per-test mock upstream:
- integration_http: all 7 methods round-trip with body, status passthrough
for 404/500/502, query strings preserved, 1MB POST streams through.
- integration_sse: a 10-event in-process hyper SSE upstream emits at 50ms
cadence; chunks reach the client with max gap < 500ms (loose CI bound)
and a client disconnect propagates to the upstream within 2s.
- integration_ws: 5 text + 5 binary messages echo through a tungstenite
upstream byte-equal; client-initiated close propagates.
- integration_headers: hop-by-hop strip both directions, X-Forwarded-*
injection, X-Forwarded-For appends to existing value, multi-valued
response headers preserved.
- integration_body: 5MB POST round-trips byte-equal; streaming response
yields first byte before the upstream finishes sending.
- integration_health: own /healthz always 200; /healthz/upstream is 200
when upstream healthy and 503 when down.
The Sec-WebSocket-Protocol forwarding is exercised implicitly by the WS
tests via tungstenite handshake. The harness lives at tests/common/mod.rs
and is shared by every integration suite.
/healthz returns 200 unconditionally (own health). /healthz/upstream
proxies a GET to the upstream's /healthz and returns 200 when reachable
+ 2xx, 503 otherwise. Both endpoints are intercepted in axum and never
forwarded; documented in RUST_DEV.md as reserved paths.
When the catch-all sees an Upgrade: websocket request, hand it to the ws
module: axum upgrades the client side, tokio-tungstenite connects to the
upstream (rewriting http->ws / https->wss while preserving path + query),
and two pumps shovel messages until either side closes. Forwarded headers
exclude what tungstenite manages (Host, Upgrade, Connection, Sec-*) but
preserve Authorization, Sec-WebSocket-Protocol, etc. Supports text,
binary, ping, pong, and close frames in both directions.
Implements RFC 7230 6.1 hop-by-hop filtering on both request and response
sides (Connection, Keep-Alive, Proxy-Authenticate, Proxy-Authorization,
TE, Trailers, Transfer-Encoding, Upgrade), plus the additional headers
listed inside any incoming Connection: header. Injects X-Forwarded-For
(appending to existing value if any), X-Forwarded-Proto, X-Forwarded-Host,
and X-Request-Id. The proxy module wires these in for both HTTP and WS.
Builds out crates/headroom-proxy from a /healthz stub into a transparent
reverse proxy: catch-all router that forwards every method/path/query to
--upstream verbatim, streaming both request and response bodies through
reqwest without buffering. Adds clap-based config (CLI + env), thiserror
error type with sane upstream-status mapping, JSON tracing-subscriber
logging, and graceful shutdown. The library surface (build_app, AppState,
Config) is reused by the integration tests.
Lists the line-ending renormalization commit so `git blame` and GitHub's
blame UI skip it. Contributors can opt in locally with:
git config blame.ignoreRevsFile .git-blame-ignore-revs
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`.gitattributes` declares `*.py text eol=lf` and `*.sh text eol=lf`, but
74 files (73 .py, 1 .sh) are stored in the index with CRLF line endings,
violating that contract. Every macOS/Linux clone reports these files as
"modified" on fresh checkout because git's diff engine sees the stored
bytes don't match the attribute contract, even though the working tree
and index match byte-for-byte.
Running `git add --renormalize .` rewrites each affected blob so the
stored form matches the attribute declaration. No semantic changes —
every affected file's diff is "N insertions, N deletions" with inserts
and deletes being the same lines modulo line endings.
Follow-up commit adds `.git-blame-ignore-revs` so `git blame` / GitHub
blame skip this mechanical commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Learned: error recovery section was bloating with stale, near-duplicate,
and contradictory entries because the dedup key was the literal rendered
bullet text and there was no TTL or re-validation.
- Normalize the hash key for error_recovery patterns. Read recoveries key
on (basename(error_path), basename(success_path)); Bash recoveries strip
volatile suffixes (| tail -N, 2>&1, etc.) and hash only the primary
command before the first | or &&. Non-error-recovery categories keep
literal-content hashing.
- Stamp first_seen_at / last_seen_at on every pattern; bump both in
_bump_persisted_evidence via json_set. Stored in metadata JSON — no
schema change.
- Refine at render time (error_recovery only): drop rows not re-observed
in 21 days, re-validate Read success paths against the filesystem,
collapse same-error_path-with-multiple-targets into one "use Glob/Grep
first" bullet, rank by evidence_count * 0.5 ** (days/5), cap at 15
bullets.
15 new tests (TestNormalizedHash, TestRefineErrorRecovery). Full suite:
526 passed, 1 skipped. Ruff + mypy clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Make the ONNX + sqlite-vec memory path truly batched.
Batch ONNX embed_batch calls, batch sqlite-vec index/remove work under a single cached connection, and update MCP warm-up to use batch embed/save/index flows.
Add focused regression tests for ONNX batching, sqlite-vec single-connection batch behavior, and MCP warm-up batching.
Skip the MCP-specific test when optional MCP dependencies are not installed.
Refs #240
Two fixes for the init-native-e2e matrix surfaced on PR #256:
1. Composite action installed `headroom` without extras, but
`headroom/cli/__init__.py` eagerly imports `proxy.server` (via
`cli/proxy.py`), which requires `fastapi`. All 6 POSIX jobs hit
`ModuleNotFoundError: No module named 'fastapi'` before `init` ran.
Fix: install `-e .[proxy]` to match the Docker e2e image.
2. On Windows, shims are `.cmd` files and Git Bash's `which` cannot
resolve them (exact-match only). Python's `shutil.which` (used by
`headroom init`) honors PATHEXT and finds the shim fine, but the
pre-flight `which` step failed first. Fix: use `Get-Command` via
`pwsh` for the Windows verification step.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The test added in bb91cfe used ``CliRunner(mix_stderr=False)`` to keep
stderr separate from stdout for assertion purposes. That parameter was
removed in Click 8.2. The repo's pyproject.toml pins ``click>=8.1.0``,
so either Click 8.1 (needs mix_stderr) or Click 8.2+ (must omit it)
could appear in CI.
Switch to reading ``result.stderr`` when the attribute is populated,
falling back to ``result.output`` (combined stream) otherwise. This
covers every Click 8.x variant without branching on the installed
version.
Verified in the Docker e2e image (Click 8.3.3): all 45 tests in
tests/test_cli/test_init_cli.py pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Existing Docker init-e2e runs on ubuntu only. Platform-specific bugs
(Windows path separators in written hook commands, PowerShell-vs-bash
matcher strings, macOS keychain prompts, shutil.which PATHEXT quirks)
slip past it. Add a matrix workflow that drops a noop shim for each
target agent and runs ``headroom init -g <target>`` on each of the
three supported OSes, then asserts the settings file was written to
the platform-correct location.
Matrix: [ubuntu-latest, macos-latest, windows-latest] x [claude,
codex, copilot]. ``openclaw`` is excluded because it delegates to
``headroom wrap openclaw`` which needs a real OpenClaw CLI and can't
be stubbed with a noop shim; the Docker suite already covers its
negative path.
Common setup (Python install, editable headroom install, shim drop,
PATH wiring) is factored into a composite action at
.github/actions/headroom-e2e-setup so follow-up per-command workflows
(install-native-e2e, wrap-native-e2e) can be near-copies that only
supply their matrix and assertion blocks. The composite action uses
the cross-platform shim scripts from e2e/_lib/make_shim.{sh,ps1} that
landed with the harness refactor.
Scoped trigger: pull_request touching init code OR the harness, plus
pushes to main and manual dispatch. This avoids burning CI minutes on
every push to unrelated feature branches while still gating every PR
that could regress init behavior.
Not verified locally: Windows runner behavior. Reviewer should watch
the first matrix run on PR.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Port e2e/init/run.py onto the shared harness and extend coverage so
issue #245 (bare ``headroom init -g`` with no agents) is locked in:
* ``seq_claude_local`` / ``seq_copilot_global`` / ``seq_codex_local`` —
the original scenario, now expressed as a sequence of Cases sharing
one scratch so the manifest-merge behavior (claude + codex targets)
is still exercised end-to-end
* ``bare_init_g_no_shims`` — regression guard for issue #245: asserts
the new guided error mentions every probed target and the concrete
``headroom init -g <agent>`` example
* ``bare_init_g_with_all_shims`` — complementary happy path with all
four shims present; asserts all three configurable agents report
``Configured ... (user scope)`` on stdout
* ``init_g_{claude,codex,copilot}_explicit`` — one case per
subcommand, each with only its own shim on PATH, asserting exit 0
and the correct per-agent settings file is written
* ``init_g_openclaw_missing`` — negative path for openclaw when its
binary isn't installed (delegates to ``headroom wrap openclaw`` which
can't be shimmed cheaply)
* ``init_verbose_no_shims`` — smoke test for ``headroom init -v``
ensuring ``detect_init_targets``, ``global_scope=True``, and every
agent name appear on stderr
Dockerfile is updated to COPY e2e/__init__.py and e2e/_lib/ so the
harness is importable inside the container. A new e2e/__init__.py
marks the tree as a package.
One small harness fix rides along: ``_resolve_headroom_bin`` captures
the absolute path to headroom before ``with_clean_path`` narrows PATH.
This is required for any case run inside a venv-scoped image - the
real ``headroom`` lives outside the shim dir and would otherwise be
hidden by the scrubbed PATH. Same bug would have bitten every future
command suite, so the fix belongs in the harness rather than run.py.
Verified locally inside the Docker image: all 10 cases pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The sync-plugin-versions pre-commit hook recomputes plugin semver from
git history + conventional-commits bump rules. Adding the feat(init)
-v/--verbose commit triggers a minor bump (0.11.4 -> 0.12.0). Land
that bump as its own chore so subsequent test/ci commits on this
branch aren't flagged as drift by the hook.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When users hit an init regression it's opaque why: no visible state
about which agents were probed, which paths were written, which
subprocesses ran. Add a top-level flag to ``headroom init`` that routes
debug-level logging from the ``headroom.cli.init`` logger to stderr.
Instrumented decision points:
* detect_init_targets / _probe_init_targets — scope + per-target
shutil.which result
* _write_json, _ensure_claude_hooks, _ensure_copilot_hooks,
_ensure_codex_hooks, _ensure_codex_provider — file paths being written
* _apply_user_env — chosen scope (windows vs unix) and env-var keys
* _run_checked — each subprocess command + exit code + truncated
stdout/stderr (useful when ``claude plugin install`` fails)
* _run_init_targets — target dispatch order and resolved profile
* top-level init callback — all flag values and invoked_subcommand
Log output goes to stderr so stdout stays clean for pipes. The handler
attached by ``_enable_verbose_logging`` is idempotent - nested
subcommand invocations don't duplicate output. The logger does not
propagate to the root logger, so enabling ``headroom init -v`` does not
affect the rest of the process.
The flag is declared on the parent Click group. Subcommands (claude,
codex, copilot, openclaw) inherit the enabled logger automatically
because the group callback runs before dispatch.
Added tests cover:
* ``init -v`` emits the expected markers to stderr, including
``detect_init_targets``, ``global_scope=True``, and each agent name
* ``_enable_verbose_logging`` is safe to call repeatedly (handler
remains singular)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fixes#245.
Running ``headroom init -g`` with no supported agents on PATH previously
produced a single-line ClickException that read like the -g flag had
been removed:
Error: No supported user init targets were auto-detected. Specify one explicitly.
This left reporter #245 concluding the feature was gone. Replace that
message with a structured diagnostic that:
* states which scope (user / local) was tried
* lists every target probed (claude, codex, copilot, openclaw) and the
shutil.which() result for each
* explicitly confirms that -g / --global is still a supported flag
* shows the concrete per-target invocation for each agent
(``headroom init -g claude``, ...) so the user knows the escape hatch
The implementation factors ``detect_init_targets`` into a ``_probe_init_targets``
helper that returns ``[(name, which_result)]``. ``detect_init_targets``
keeps its existing signature so the test suite and external imports
aren't broken; the new helper backs both the auto-detection path and
the diagnostic error formatter.
Unit tests in tests/test_cli/test_init_cli.py cover:
* the end-to-end message shape (structural markers + every target name +
the example invocation)
* the local-scope variant omitting global-only agents (copilot / openclaw)
* that found binaries are surfaced with their absolute path so users can
debug cases where shutil.which returns an unexpected result
No behavior change when at least one target is detected.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Centralize Docker / CI e2e test helpers so per-command suites can be
declarative and future commands (install, wrap, ...) can reuse the same
shim/PATH/assertion primitives without duplicating infrastructure.
The harness provides:
* Case dataclass describing one test as argv + shims + expected exit /
stdout / stderr / files / custom callbacks
* make_shim() factory producing cross-platform executable shims (.sh on
POSIX, .cmd on Windows) with noop / fail / record-args behaviors
* with_clean_path() context manager that isolates PATH to a minimal
known-good value plus any extras supplied by the case
* agent_settings_path() locator mirroring headroom.cli.init so tests can
assert the right file was written without touching private init state
* run_cases() for independent cases and run_case_sequence() for cases
that must share scratch state (e.g. manifest-merge scenarios)
Shell / PowerShell shim-creation scripts are also shipped for CI steps
that need to drop a shim without spinning up Python first.
No behavior change in this commit - pure infrastructure. The init suite
and new subcommand suites consume the harness in follow-up commits.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Running the repo's sync-plugin-versions pre-commit hook updates
.claude-plugin/marketplace.json, .github/plugin/marketplace.json, and
the two headroom-agent-hooks plugin.json manifests to the release
semver computed from git tags (0.11.4 at time of branch). Landing this
first keeps subsequent commits on this branch from tripping the
hook's auto-fix path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`headroom wrap codex` injects a `model_provider = "headroom"` block
plus a `[model_providers.headroom]` table into `~/.codex/config.toml`
so Codex routes both HTTP and WebSocket traffic through the proxy. The
matching `unwrap codex` subcommand did not exist, so the injected
block stayed in `config.toml` forever — the moment the proxy stopped,
Codex (CLI and macOS app) started erroring with
`Missing environment variable: OPENAI_API_KEY`, and users had to hand-
edit the file to recover.
Fix:
* `_inject_codex_provider_config` now snapshots the pre-wrap file to
`~/.codex/config.toml.headroom-backup` before the first modification
and leaves that snapshot untouched on subsequent wrap runs. The
injection is also rewritten to use two self-contained marker-
delimited blocks (top-level key and provider table) so stripping
them never consumes user content that sits between them.
* `_inject_memory_mcp_config` takes the same snapshot, so
`wrap codex --memory` without a full provider injection is still
fully reversible.
* New `_restore_codex_provider_config` helper and `unwrap codex`
click command:
* backup present → restore byte-for-byte and delete the backup;
* backup absent but Headroom block present → strip the block and
keep surrounding user content;
* config contained only Headroom content → remove the file so
Codex falls back to defaults;
* nothing to undo → safe no-op.
Codex is the only wrap target that modifies a persistent user config
file: claude/aider/cursor/copilot all go through env vars or project-
scoped files only, so this bug was unique to Codex.
Tests:
* `tests/test_cli/test_wrap_codex.py` adds 20 new cases covering the
strip/snapshot helpers directly, round-trip idempotency of
wrap → wrap → unwrap, handling of malformed prior configs, and
end-to-end CliRunner invocations of `headroom wrap codex
--prepare-only` / `headroom unwrap codex` against a temp `$HOME`.
* All 153 existing `tests/test_cli/` tests continue to pass.
Plugin manifest versions were re-synced from `pyproject.toml` (0.11.2)
by the `sync-plugin-versions` pre-commit hook; the previous values
(0.10.3) had drifted.
Reported by @raenaryl in Discord on 0.6.3; confirmed still broken on
current `main` (0.11.x).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI runs `ruff format --check`, which flagged the two multi-line
expressions added in the previous commit. Pure whitespace reflow —
no behavior change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
compute_turn_id hashed the raw message dicts, which meant the same
user-text message produced a different hash on each call of one agent
loop because clients (notably Claude Code) move the cache_control
breakpoint to the newest message per call. The user-text block carries
cache_control on call 1 and not on call 2, so the serialized prefix
differs and the turn_id rolls over. Effect downstream: every API call
becomes its own "turn" and any prompt-level aggregation (e.g. the
Headroom desktop app's prompt all-time record) collapses to the
largest single call, not the sum across the prompt.
Add a small recursive normalization pass that strips cache_control from
the hashed prefix and from list-shaped system prompts before hashing.
Two new tests cover cache_control moving between calls on both the
messages array and the system prompt.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
