mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
2606 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Tejas Chopra
|
24fc6a8d43 |
fix(docker): publish compose ports on loopback only
`docker compose up -d` published every service on 0.0.0.0, and none of the three authenticates an inbound caller by default: 8787 proxy — /v1/* data plane is open unless HEADROOM_PROXY_TOKEN is set 6333 Qdrant — no API key at all; holds embeddings derived from prompts 7687 Neo4j — NEO4J_AUTH falls back to neo4j/devpassword, published here So the shipped default handed any peer on the surrounding network a relay through the proxy plus direct read/write on the vector and graph stores built from the operator's own prompt content. The proxy already warns loudly about this shape at server.py:3289 — the compose file just never took the advice. Publishing is now pinned to 127.0.0.1. The proxy container still listens on 0.0.0.0 internally, so service-to-service traffic on the compose network is unaffected, and `http://localhost:8787` from the host still works exactly as the quick-start describes. Only cross-machine reachability changes, which is the behavior that was unsafe. The header comment now documents how to expose the proxy deliberately, pairing the port override with HEADROOM_PROXY_TOKEN rather than leaving that implicit. Adds a regression test asserting every published port names a loopback host IP; it fails on the parent commit. |
||
|
Tejas Chopra
|
a6ab359a5d
|
fix(proxy): guard feedback endpoints and add CSRF checks to loopback writes (#3060)
## Description
`#2927` brought eight telemetry/TOIN routes under `require_loopback`.
Two structurally identical siblings 60 lines above them were missed:
```
GET /v1/feedback
GET /v1/feedback/{tool_name}
```
Neither is an aggregate-counter endpoint. Their `common_queries` /
`queried_fields` keys are built verbatim from agent search text —
`event.query.lower()` at `headroom/cache/compression_feedback.py:311` —
and up to 100 queries are retained per tool, keyed by real tool name.
Under the shipped Docker default (`--host 0.0.0.0`) a LAN peer gets a
404 from `/v1/toin/patterns` and the query corpus from `/v1/feedback`.
Separately, five mutating loopback-only routes had no CSRF guard.
`require_loopback` cannot stop that attack: a remote page POSTing to a
known `127.0.0.1` URL with `Content-Type: text/plain` is a CORS *simple*
request, so there is no preflight, and the browser still sends the real
loopback `Host` header — both of the guard's gates pass. Only `Origin`
betrays the caller, and only `require_same_origin` inspects it. That
guard already existed at `headroom/proxy/loopback_guard.py:219` and was
applied solely to `/settings`.
Closes #2927 (completes it — the original eight routes were already
done).
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
## Changes Made
- Added `Depends(_require_loopback)` to `/v1/feedback` and
`/v1/feedback/{tool_name}`.
- Stripped `common_queries` / `queried_fields` from both response bodies
even on the guarded path, matching the whitelist discipline #2930
applied at `server.py:4909-4916`.
- Added `_feedback_stats_without_query_text()` so the scrub happens at
the HTTP boundary; `get_stats()` is unchanged and in-process compression
decisions are untouched.
- Added `Depends(_require_same_origin)` to `POST /stats/reset`,
`/cache/clear`, `/v1/retrieve`, `/v1/telemetry/import`,
`/admin/runtime-env`.
## Testing
- [x] Unit tests pass
- [x] Linting passes (ruff check + format)
- [ ] Type checking passes (`uv run mypy headroom`) — not run
- [x] New tests added for new functionality
### Test Output
```text
$ .venv/bin/python -m pytest tests/test_proxy_loopback_gating.py -q
99 passed, 1 warning in 4.18s
$ .venv/bin/python -m pytest tests/test_proxy_settings_endpoints.py tests/test_telemetry.py \
tests/test_proxy_cache_telemetry.py tests/test_proxy_telemetry_env.py tests/test_telemetry_context.py -q
101 passed, 1 warning in 3.67s
$ .venv/bin/python -m pytest tests/test_critical_fixes.py tests/test_compression_store.py \
tests/test_toin_full_integration.py tests/test_ccr_feedback.py tests/test_critical_gaps.py \
tests/test_proxy_ccr.py tests/test_proxy_dashboard_stats_cache.py -q
168 passed, 4 skipped, 3 warnings in 13.18s
$ .venv/bin/python -m ruff check headroom/proxy/server.py tests/test_proxy_loopback_gating.py
All checks passed!
```
Against the parent commit (`git stash` of `server.py` only), all 14 new
tests fail:
```text
FAILED test_non_loopback_caller_gets_404[get-/v1/feedback]
FAILED test_non_loopback_caller_gets_404[get-/v1/feedback/example]
FAILED test_cross_origin_post_rejected[/stats/reset]
FAILED test_cross_origin_post_rejected[/cache/clear]
FAILED test_cross_origin_post_rejected[/v1/retrieve]
FAILED test_cross_origin_post_rejected[/v1/telemetry/import]
FAILED test_cross_origin_post_rejected[/admin/runtime-env]
FAILED test_sandboxed_null_origin_post_rejected[...] (5 cases)
FAILED test_feedback_stats_exclude_agent_query_text
FAILED test_feedback_tool_detail_excludes_agent_query_text
14 failed, 85 passed
```
## Real Behavior Proof
- Environment: macOS 15 (darwin 25.4.0), Python 3.12.13, this branch,
FastAPI `TestClient` against the real `create_app` proxy.
- Exact command / steps: drive `/v1/feedback` with a feedback singleton
whose `common_queries` contains `"find the customer api key rotation
runbook"`, once from a non-loopback peer and once from a loopback peer;
POST each of the five mutating routes with `Origin:
https://attacker.example` and `Content-Type: text/plain`.
- Observed result: non-loopback callers now receive 404 where they
previously received 200 with the query corpus; on the loopback path the
response no longer contains `common_queries`, `queried_fields`, or the
substring `customer api key rotation`, while `retrieval_rate` still
resolves to `0.25`. All five cross-origin POSTs return 403; the same
requests with no `Origin`, or with `Origin: http://127.0.0.1`, are
unaffected.
- Not tested: a real browser issuing the cross-origin POST (the CORS
simple-request shape is reproduced at the header level, not in a
browser), and a live non-loopback deployment.
## Runtime Rollout Safety
- Rollout-managed feature(s): none.
- Minimum rollout channel: N/A.
- Stable/default behavior changed: yes — `/v1/feedback*` now 404 for
non-loopback callers and no longer return query text; five POST routes
reject cross-origin browser callers.
- Kill switch / disable path: none; these are security guards and are
deliberately not configurable.
- Unsafe override required: none.
- Qualification impact: none.
- Rollback path: revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
## Additional Notes
`/stats` also calls `feedback.get_stats()` (`server.py:3896`) but only
reads aggregate counters at `:4303-4311` and never emits query text —
verified, and the reason the scrub is applied at the HTTP boundary
rather than inside `get_stats()`.
The five POST routes are strictly loopback-gated, so the
trusted-dashboard wrapper `/settings` uses is unnecessary here; for a
loopback caller that wrapper falls through to the same raw guard. No
dashboard asset calls them, and the TypeScript SDK
(`sdk/typescript/src/client.ts:322,443`) sends no `Origin` header, which
the guard passes through unchanged.
Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local>
|
||
|
Tejas Chopra
|
96c25f5181
|
fix(cli): stop the macOS malloc re-exec replacing an embedder's process (#3064)
## Description **`main` cannot currently run its own test suite on macOS.** `pytest tests/` dies at roughly 2% with exit code 2 — no traceback, no summary, no failing test named. The pytest process is simply gone. Two independent defects, both landed today, both invisible to CI. ### 1. The macOS malloc re-exec replaces the calling process `headroom proxy` re-execs itself once on Darwin to apply two libmalloc knobs that libmalloc only reads before `main()` (#2820, PR #2879): ```python os.execv(sys.executable, [sys.executable, "-m", "headroom.cli", *sys.argv[1:]]) ``` That reconstruction is only faithful when the process really *is* the Headroom CLI. Ten-plus test files invoke the `proxy` command in-process through Click's `CliRunner`. There, `os.execv` replaces **pytest** with a Headroom process holding pytest's argv. Run with `-s`, the mechanism is visible: ``` tests/test_agent_savings.py Usage: python -m headroom.cli [OPTIONS] COMMAND [ARGS]... Error: No such command 'tests/test_agent_savings.py::test_proxy_cli_reads_agent_90_profile_env'. ``` Everything after the first such test — roughly 98% of the suite — never runs. The same hazard applies to any application embedding the CLI. **The documented kill switch does not help.** `tests/conftest.py:41` scrubs every `HEADROOM_*` variable for hermeticity, so `HEADROOM_MALLOC_TUNING` is deleted before the guard reads it. Only the private `_HEADROOM_MALLOC_TUNED` survives, because it starts with an underscore. **CI could not have caught this.** The tuning is Darwin-only, and while the repo *does* have macOS jobs (`macos-native-wrapper`, `wrap-native (macos-latest)`), neither runs the Python test suite — the `test` shards are `ubuntu-latest` only. So `sys.platform != "darwin"` returns first everywhere pytest actually runs. #2879 merged with 37 green checks. ### 2. A semantic merge conflict between two green PRs #3051 added `bind_scope(tags, request.scope)` at `gemini.py:325` and updated the three Gemini fakes it knew about. #3035 branched earlier and added a fourth `_FakeRequest` without `.scope`. Each was green against its own base; together they fail: ``` AttributeError: '_FakeRequest' object has no attribute 'scope' ``` Git merged both cleanly. Only running the suite on merged `main` surfaces it. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update ## Changes Made - Added `_process_is_headroom_cli_entrypoint()`: the re-exec now verifies its own precondition — `argv[0]` must be the `headroom` console script or `headroom/cli/__main__.py`. - The embedded path returns **before** stamping `_HEADROOM_MALLOC_TUNED`, so a genuine CLI child inheriting the environment can still apply the tuning. - Gave the Gemini `_FakeRequest` the `.scope` every real Starlette `Request` carries. - `test_reexec_skips_when_operator_already_set_vars` now sets a realistic `argv[0]`, matching its sibling exec test. - New `tests/test_cli_proxy_malloc_reexec_guard.py` asserting the guard's logic on **every** platform, since no CI runner is macOS. ## Testing - [x] Unit tests pass - [x] Linting passes (ruff check + format) - [ ] Type checking passes (`uv run mypy headroom`) — not run - [x] New tests added for new functionality ### Test Output Before, on `main`: ```text $ .venv/bin/python -m pytest tests/ -q collected 11622 items / 8 skipped ... tests/test_agent_savings.py ............................ $ echo $? 2 ``` No summary line — the run does not end, it is replaced. After, on this branch: ```text $ .venv/bin/python -m pytest tests/ -q 3 failed, 11055 passed, 581 skipped, 6034 warnings in 303.69s (0:05:03) ``` All three remaining failures reproduce at ` |
||
|
AxelRay
|
ef7e07e0f5
|
fix(policy): price net-cost mutations with the 1h cache-write tier (#2780)
## Description This fixes the net-cost mutation gate for requests using Anthropic's 1-hour prompt-cache TTL. The gate previously hardcoded the 5-minute cache-write multiplier of 1.25x. A 1-hour cache write costs 2.0x, so the old calculation understated the true write penalty and could incorrectly recommend mutation for 1-hour clients. Closes #2773 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added TTL-aware cache-write multiplier selection for 5-minute and 1-hour tiers. - Threaded the resolved TTL through the content router and compression policy helpers. - Preserved the existing 5-minute behavior as the default. - Added Python and Rust regression coverage for the 1-hour tier. - Retuned the netcost gate fixtures so the 1-hour write tier flips the decision in the full ContentRouter path. - Did not edit CHANGELOG.md. ## Testing - [x] Unit tests pass (pytest) - [x] Linting passes (ruff check .) - [ ] Type checking passes (mypy headroom) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text pytest tests/test_compression_policy.py -q 20 passed cargo test -p headroom-core --lib compression_policy -- --nocapture 14 passed pytest tests/test_netcost_gate.py -q 27 passed Ruff checks and formatting passed. git diff --check passed. ``` ## Real Behavior Proof - Environment: Linux x86_64 contributor checkout with Python and Rust test environments. - Exact command / steps: - Ran the Python compression policy test suite. - Ran the Rust compression policy unit tests. - Ran the netcost gate suite, including the 1-hour env and request-marker cases. - Exercised the new 1-hour TTL golden case alongside the existing 5-minute cases. - Observed result: The 1-hour case uses the 2.0x write multiplier and skips the same candidate that still mutates under 5-minute pricing. Existing 5-minute behavior remains covered and passing. - Not tested: A live Anthropic request through the proxy and production traffic. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit CHANGELOG.md - it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable for this backend policy fix. ## Additional Notes Ready for review. CI is green on the current tip. |
||
|
Abhay Singh
|
2a8472525d
|
feat(wrap/claude): make the --1m fallback model configurable via HEADROOM_1M_MODEL (#2983)
## Description
The model `headroom wrap claude --1m` falls back to (when no model is
otherwise selected) was a hardcoded constant `claude-opus-4-8`, with no
env var or config key to override it. So it goes stale with every new
Opus release, and the only workaround is pinning `ANTHROPIC_MODEL`
globally -- which also changes every non-`--1m` session and overrides
Claude Code's own `/model` picker. The knob the user actually wants
("what should `--1m` default to") did not exist (#2937).
## Fix
Add a `HEADROOM_1M_MODEL` env override that `_resolve_1m_model` consults
for its fallback default, and bump the built-in default to
`claude-opus-5` (Opus 5 has shipped):
```python
_1M_MODEL_ENV = "HEADROOM_1M_MODEL"
_DEFAULT_1M_MODEL = "claude-opus-5"
def _resolve_1m_model(current: str | None) -> str:
fallback = (os.environ.get(_1M_MODEL_ENV) or "").strip() or _DEFAULT_1M_MODEL
base = (current or "").strip() or fallback
return base if base.endswith(_CONTEXT_1M_SUFFIX) else f"{base}{_CONTEXT_1M_SUFFIX}"
```
Precedence is unchanged: an explicit `ANTHROPIC_MODEL` (or a
pass-through `--model`, via the existing `_apply_1m_to_claude_args`)
still wins. `HEADROOM_1M_MODEL` only supplies the fallback when nothing
else is selected. The `[1m]` suffixing and idempotency are unchanged.
Fixes #2937
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `docs/content/docs/configuration.mdx`: document `HEADROOM_1M_MODEL`
(new "Claude 1M context window" subsection covering `--1m` resolution
order and `[1m]` acceptance) and register it in the Environment
Variables catalog with its current default.
- `tests/test_cli/test_wrap_helpers.py`: assert the knob stays
documented and the documented default tracks `_DEFAULT_1M_MODEL`, so it
cannot silently drift.
- `headroom/cli/wrap.py`: add `HEADROOM_1M_MODEL` env override in
`_resolve_1m_model`; bump `_DEFAULT_1M_MODEL` to `claude-opus-5`.
- `tests/test_cli/test_wrap_helpers.py`: env override wins the fallback;
an explicit current model still wins over the env; blank env falls back
to the built-in; env value is idempotent for an already-`[1m]` value.
Updated the existing "falls back to default" test to assert against the
constant (robust to future bumps) and to clear the env var.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added
### Test Output
```text
tests/test_cli/test_wrap_helpers.py -k "resolve_1m or apply_1m" 11 passed
tests/test_cli/test_wrap_claude_vertex_proxy_env.py -k 1m 4 passed
# uvx ruff@0.15.22 check -> All checks passed!
# uvx mypy@1.20.2 headroom/cli/wrap.py -> Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.15.22 and mypy 1.20.2 via uvx.
- Exact command / steps: exercised `_resolve_1m_model` directly with the
env var set/unset. With `HEADROOM_1M_MODEL=claude-opus-9` and no
`ANTHROPIC_MODEL`, `--1m` resolves to `claude-opus-9[1m]`; with the env
var unset it resolves to `claude-opus-5[1m]`; a set `ANTHROPIC_MODEL`
(e.g. `claude-sonnet-5`) still wins as `claude-sonnet-5[1m]`.
- Observed result: operators can point `--1m` at the current Opus
without a code change and without pinning `ANTHROPIC_MODEL` globally,
and a fresh install no longer silently opts `--1m` into the previous
generation.
- Not tested: a live Claude Code 1M session (no entitled account here).
The resolution is verified at the helper the launch path uses.
## Runtime Rollout Safety
- Rollout-managed feature(s): none. `wrap claude --1m` model resolution
is a launch-time CLI helper, not a rollout-channel-gated runtime
feature.
- Minimum rollout channel: N/A (no rollout-managed behavior).
- Stable/default behavior changed: yes, narrowly. The built-in `--1m`
fallback default moves from `claude-opus-4-8` to `claude-opus-5` only
when neither `HEADROOM_1M_MODEL` nor `ANTHROPIC_MODEL` is set; any
explicit selection is unaffected.
- Kill switch / disable path: set `HEADROOM_1M_MODEL` (or
`ANTHROPIC_MODEL`) to pin any model; both override the default.
- Unsafe override required: no.
- Qualification impact: none. No proxy request path, routing, or token
accounting is touched.
- Rollback path: revert this PR, or set
`HEADROOM_1M_MODEL=claude-opus-4-8` to restore the prior default without
a code change.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`: it is generated by
release-please from my Conventional Commit PR title
## Additional Notes
The default bump (`claude-opus-4-8` -> `claude-opus-5`) is the second
half of the issue's request. If you would rather keep the constant and
ship only the env override, I can drop that one line; the override alone
already lets operators avoid the stale default.
---------
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
Abhay Singh
|
ddd9f76729
|
fix(install): stop the PowerShell installer leaking temp dirs into the real user PATH (#2985)
## Description
`scripts/install.ps1` persists the install directory to the user's PATH
through `Ensure-PathEntry`, which calls
`[Environment]::SetEnvironmentVariable('Path', ..., 'User')`. That value
lives in the `HKCU\Environment` registry key, so it is **not** scoped by
a `HOME` / `USERPROFILE` override.
`tests/test_install/test_native_installers.py::test_powershell_native_installer_supports_persistent_docker_lifecycle`
runs that real installer against a `tmp_path` fake home. Every run
therefore prepended the test's throwaway shim directory to the
developer's actual, persistent user PATH -- and it stayed there after
the test finished. The entries accumulate one per run, ahead of the real
install dir; and since the installer also drops
`headroom.ps1`/`headroom.cmd` into that dir, `headroom` in a fresh shell
could then resolve to a leftover wrapper from a deleted temp directory
(#2970).
## Fix
Make the persistence scope configurable via
`HEADROOM_INSTALL_PATH_SCOPE`, defaulting to `'User'` so production
behavior is unchanged:
```powershell
$scope = if ($env:HEADROOM_INSTALL_PATH_SCOPE) { $env:HEADROOM_INSTALL_PATH_SCOPE } else { 'User' }
$currentPath = [Environment]::GetEnvironmentVariable('Path', $scope)
...
[Environment]::SetEnvironmentVariable('Path', ($newPath -join ';'), $scope)
```
The installer tests (`_build_env`) set
`HEADROOM_INSTALL_PATH_SCOPE=Process`, so the PATH update stays in the
spawned PowerShell process (discarded when it exits) instead of writing
to the registry.
Fixes #2970
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `scripts/install.ps1` (`Ensure-PathEntry`): read/write the PATH via
`$env:HEADROOM_INSTALL_PATH_SCOPE` (default `'User'`).
- `tests/test_install/test_native_installers.py`: `_build_env` sets
`HEADROOM_INSTALL_PATH_SCOPE=Process` for every installer invocation;
add a Windows-only
`test_powershell_installer_does_not_leak_into_user_path` asserting the
real User PATH entry count is unchanged across an installer run.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] New test added
### Test Output
```text
tests/test_install/test_native_installers.py -k does_not_leak_into_user_path 1 passed
# uvx ruff@0.15.22 check tests/test_install/test_native_installers.py -> All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11, Windows PowerShell 5.1, Python 3.12.11,
project venv, pytest 9.1.1, ruff 0.15.22 via uvx.
- Exact command / steps: recorded the real user PATH entry count
(`([Environment]::GetEnvironmentVariable('Path','User') -split
';').Count` = 27), ran the PowerShell installer test with the fix, then
re-read the count: still 27 -- no leak. The new
`test_powershell_installer_does_not_leak_into_user_path` formalizes this
(before == after).
- Observed result: running the installer test suite no longer mutates
the developer's persistent user PATH; production installs still persist
to `'User'` as before.
- Not tested: the sibling
`test_powershell_native_installer_supports_persistent_docker_lifecycle`
fails on my Windows host on an unrelated `trusted_cidrs`
dashboard-gateway assertion (it fails identically on `main` without this
change, and the whole PowerShell suite is skipped on the Linux CI
runners). This PR does not touch that path.
## Runtime Rollout Safety
- Rollout-managed feature(s): none. This is the native PowerShell
installer script, not a rollout-channel-gated runtime feature.
- Minimum rollout channel: N/A (no rollout-managed behavior).
- Stable/default behavior changed: no. Production installs still persist
PATH to the `User` scope exactly as before; the new
`HEADROOM_INSTALL_PATH_SCOPE` override defaults to `User` and is used
only by the test suite to avoid mutating the developer's persistent
PATH.
- Kill switch / disable path: leave `HEADROOM_INSTALL_PATH_SCOPE` unset
(the default) for the normal `User` behavior.
- Unsafe override required: no.
- Qualification impact: none. Installer-only; no proxy runtime path is
touched.
- Rollback path: revert this PR; the installer returns to writing the
`User` PATH unconditionally.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`: it is generated by
release-please from my Conventional Commit PR title
## Additional Notes
The scope override defaults to `'User'`, so nothing changes for real
installs. It doubles as an escape hatch for any environment (CI images,
ephemeral containers) that must not touch the persistent user PATH.
---------
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
Nestor G Pestelos Jr
|
c8310819a4
|
fix(wrap): set xAI upstream for grok-build proxy (#2772)
## Description `headroom wrap grok-build` injected the client hop into `~/.grok/config.toml` but started the local proxy **without** setting the OpenAI-compatible upstream to xAI. The proxy defaulted to `api.openai.com`, so Grok session auth returned **401** on every chat completion even though compression still ran. `wrap grok` already passes `openai_api_url` → xAI. This PR aligns `wrap grok-build` and the Grok-only persistent `install` path on the shared `DEFAULT_API_URL` (`https://api.x.ai`). Closes # (none — discovered in live Grok Build pilot) ## Type of Change - [x] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Pass `openai_api_url=_GROK_DEFAULT_API_URL` into `_run_proxy_only_watcher` from `wrap grok-build` - Use shared `DEFAULT_API_URL` from `wrap grok` (no hard-coded string drift) - Print proxy upstream in Grok Build setup lines - Persistent install: when targets are Grok-only, set `OPENAI_TARGET_API_URL` + `--openai-api-url` (skip when Codex/Copilot/Aider/OpenCode share the proxy; explicit env still wins) - Regression tests for wrap kwargs, setup lines, and install planner ## Testing - [x] Unit tests pass (`pytest` targeted suite) - [ ] Linting passes (`ruff check .`) — not run in this environment (no native editable build) - [ ] Type checking passes (`mypy headroom`) — not run - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ PYTHONPATH=$PWD python -m pytest \ tests/test_cli/test_wrap_bridge.py::test_wrap_grok_build_passes_xai_openai_api_url \ tests/test_cli/test_wrap_bridge.py::test_wrap_grok_build_uses_actual_proxy_port \ tests/test_install/test_planner.py::test_build_manifest_grok_build_only_sets_xai_upstream \ tests/test_install/test_planner.py::test_build_manifest_grok_with_codex_does_not_force_xai \ tests/test_install/test_planner.py::test_build_manifest_extra_env_wins_over_grok_xai_default \ tests/test_provider_grok_build.py::test_grok_build_setup_lines_include_proxy_url -q ...... 6 passed in 0.33s ``` ## Real Behavior Proof - Environment: macOS (darwin), Headroom 0.33.0 via `uv tool install "headroom-ai[proxy,mcp,code]==0.33.0"`, Grok Build CLI, models `grok-build` and `grok-4.5`, proxy on `127.0.0.1:8787`, upstream must be xAI - Exact command / steps: (1) Before: stock `headroom wrap grok-build` then `grok -m grok-build` one-shot prompt. (2) After: same wrap path with this branch (`openai_api_url=DEFAULT_API_URL` into `_run_proxy_only_watcher`) then `grok -m grok-build -p '…HEADROOM_XAI_OK…'`. Also exercised `grok-4.5` via `[model."grok-4.5"] base_url` → same proxy. - Observed result: Before — proxy log outbound `api.openai.com` → HTTP 401; client failed while local compression still ran. After — setup line prints Proxy upstream `https://api.x.ai`; proxy log `POST https://api.x.ai/v1/chat/completions` (and `/v1/responses` for grok-4.5) → status=200; dashboard shows 0 failed requests and accumulating token savings on live traffic. - Not tested: full `uv run` editable/maturin native build on this host; multi-tool install matrix beyond planner unit tests; Windows; ruff/mypy full tree ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I commented my code, particularly in hard-to-understand areas - [ ] I made corresponding changes to the documentation (CLI help text / setup lines only) - [x] My changes generate no new warnings - [x] I added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — generated by release-please from Conventional Commit PR title (a CI guard enforces this) ## Additional Notes - Intentional non-goal: changing default model, savings %, or Grok Build context-tool defaults - Mixed-target install (e.g. `grok_build` + `codex`) does **not** force xAI — operator must set upstream explicitly if they share one proxy - Related live routing: manual `[model."grok-4.5"] base_url` through the same proxy works once upstream is xAI (`/v1/responses`) --------- Co-authored-by: Grok 4.5 <noreply@x.ai> Co-authored-by: Nestor G Pestelos Jr <ngpestelos@me.com> Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net> |
||
|
Abhay Singh
|
6d87825f62
|
fix(proxy): tune macOS libmalloc and trim allocator pages so long-lived RSS stays bounded (#2879)
## Summary Fixes #2820. Prevents long-lived macOS proxies from retaining every largest transient request-body allocation in libmalloc. The reporter’s production A/B isolated the allocator behavior and verified the two pre-main libmalloc knobs; this PR applies them through a one-time Darwin-only re-exec and adds periodic per-worker pressure relief. - `MallocAggressiveMadvise=1` returns freed pages eagerly. - `MallocLargeCache=0` disables the large-allocation death-row cache. - Operator-set allocator variables are preserved; `HEADROOM_MALLOC_TUNING=0` is the kill switch. - Periodic trim defaults on only for macOS, runs off the event loop, performs no forced Python GC, validates its interval, and is retained/cancelled through the app lifecycle. - Non-Darwin behavior remains unchanged unless explicitly enabled. - Semantically rebased onto current `main`, retaining startup dependency validation, MCP SDK v1 compatibility, and all newer proxy behavior. ## Verification - 147 proxy CLI/config/malloc/MCP-contract tests pass; 1 platform skip. - Ruff check and formatting clean; `git diff --check` clean. - The reporter’s macOS A/B reduced dirty empty malloc regions to zero and lowered steady/startup RSS; the control flow and shutdown lifecycle are covered locally. ## Safety The re-exec is Darwin-only, PID-preserving, loop-guarded, and opt-out. The trim task is per worker because allocator state is per process, and shutdown cancels it explicitly. |
||
|
Abhay Singh
|
be5b26d807
|
fix(doctor): surface that Claude Desktop agent sessions bypass the proxy (#2987)
## Description `headroom doctor` reports the `claude` check as a pass whenever `~/.claude/settings.json` carries an `ANTHROPIC_BASE_URL` pointing at the proxy. That is correct for the terminal Claude Code CLI. But Claude Desktop (`com.anthropic.claudefordesktop`) unconditionally overwrites that variable when spawning agent sessions (#869), so on a Desktop-primary machine `doctor` asserts routing that is in fact discarded, and nothing in the output hints that Desktop sessions are unrouted (#2925). ## Fix Add a per-surface `claude desktop` check that warns about the bypass when Claude Desktop's config directory is detected, pointing at #869. Following the issue's suggestion, it models per-surface reporting like the existing `wrap_marker` / `shell env` rows: it is a separate row emitted only when Desktop is present, so it never contradicts a genuinely routed CLI, and the existing `claude` check is left unchanged. Detection uses Claude Desktop's per-user config directory (distinct from the CLI's `~/.claude`): - macOS: `~/Library/Application Support/Claude` - Windows: `%APPDATA%\Claude` - Linux: `$XDG_CONFIG_HOME/Claude` (or `~/.config/Claude`) Fixes #2925 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cli/doctor.py`: add `claude_desktop_config_dir()` (cross-platform) and `check_claude_desktop()` (WARN when the dir exists, `None` otherwise); append it to the `doctor()` check list when present. - `tests/test_cli_doctor.py`: `TestClaudeDesktop` -- no row when absent; WARN naming the bypass and #869 when present; the `doctor --json` entrypoint appends the row only when Desktop is detected. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] New tests added ### Test Output ```text tests/test_cli_doctor.py 78 passed # uvx ruff@0.15.22 check -> All checks passed! # uvx mypy@1.20.2 headroom/cli/doctor.py -> Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1, ruff 0.15.22 and mypy 1.20.2 via uvx. - Exact command / steps: `uvx ruff@0.15.22 check headroom/cli/doctor.py tests/test_cli_doctor.py`; `uvx mypy@1.20.2 headroom/cli/doctor.py`; `python -m pytest tests/test_cli_doctor.py -q`; then drove the check directly and through the `doctor --json` entrypoint with `claude_desktop_config_dir` pointed at a tmp dir (created the dir, ran `doctor --json`, then removed it and reran). - Observed result: with the dir present, a `claude desktop` row appears with status `warn` and a `#869` hint; with the dir absent, no such row is emitted and the rest of the report is unchanged. A Desktop-primary machine now gets an explicit warning that Desktop agent sessions bypass the proxy, instead of a bare `claude: pass` that reads as though all Claude routing is live. - Not tested: a live Claude Desktop install (detection is directory-existence, exercised against a tmp dir). ## Runtime Rollout Safety - Rollout-managed feature(s): none. This adds a read-only diagnostic row to `headroom doctor`; it is not behind any rollout channel or feature flag. - Minimum rollout channel: N/A (no rollout-managed behavior). - Stable/default behavior changed: no. The existing `claude` check and all other rows are unchanged; the new `claude desktop` row is additive and only appears when Claude Desktop's config directory is detected. - Kill switch / disable path: N/A. The row self-suppresses (returns `None`) on any machine without the Desktop config directory. - Unsafe override required: no. - Qualification impact: none. No proxy request path, routing, or token accounting is touched; the change is confined to the doctor diagnostic surface. - Rollback path: revert this PR; the doctor output returns to its prior set of rows with no state or migration to undo. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md`: it is generated by release-please from my Conventional Commit PR title ## Additional Notes Scope: this warns whenever Claude Desktop is present, which is accurate (Desktop agent sessions always bypass per #869) and matches the precedent for doctor-accuracy fixes (#2618/#2614 Codex, #2566 ollama). The issue's stronger refinement -- suppress the warning when a `client=claude-code` request has recently reached the proxy -- would need per-client traffic observation the doctor does not have today; I left that as a follow-up rather than build new traffic-tracking infra into this fix. Happy to add it if you'd prefer the conditional form. Rebased onto current `main` to resolve an overlap with the newly merged `check_claude_auth_conflict` in `doctor.py`; both checks now coexist. Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
Abhay Singh
|
536c949a69
|
fix(proxy/openai): propagate provider usage on the Responses WS->HTTP fallback (#2988)
## Description
When Codex uses the OpenAI Responses WebSocket endpoint through Headroom
and the upstream WebSocket is rejected, Headroom falls back to HTTPS
POST/SSE. On that fallback the dashboard reported zero or tiny input
tokens for a large request, and invalid savings:
```json
{ "input_tokens_original": 3, "input_tokens_optimized": 0,
"output_tokens": 246, "tokens_saved": 31052, "savings_percent": 33233.33 }
```
## Root cause
`_ws_http_fallback` (openai.py) relays the SSE `data:` events to the
client but never parses the terminal `response.completed` event for
usage. The non-fallback WS path accumulates
`_extract_responses_usage(event)` into the session totals on every
`response.completed` frame (openai.py ~8182); the fallback path did not.
So `ws_input_tokens_total` stayed at the small local count, and the
session-end RequestLog computed `optimized_tokens =
residual_input_tokens = 0`, leaving `tokens_saved >
input_tokens_original` and `savings_percent` far above 100%.
## Fix
`_ws_http_fallback` now parses each relayed `response.completed` line
with the existing `_extract_responses_usage` and returns the accumulated
`(input, output, cache_read, cache_write, uncached)` provider usage. The
caller folds it into the WS session totals, so the session-end outcome
uses the authoritative provider wire-token count -- bringing the
fallback to parity with the non-fallback WS path. SSE relay behaviour is
otherwise unchanged.
Fixes #2957
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/handlers/openai.py` (`_ws_http_fallback`): accumulate
usage from `response.completed` SSE lines (both the main relay loop and
the buffer flush) and return the `(input, output, cache_read,
cache_write, uncached)` tuple from every exit path; the WS handler
caller adds it to `ws_input_tokens_total` / `ws_output_tokens_total` /
cache / uncached totals before the session-end RequestLog.
- `tests/test_ws_http_fallback.py`: the fallback returns the provider
usage from a `response.completed` event
(input/output/cache_read/uncached), and returns all-zeros when no
completed event arrives.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added
### Test Output
```text
tests/test_ws_http_fallback.py 13 passed (11 existing + 2 new)
# uvx ruff@0.15.22 check -> All checks passed!
# uvx mypy@1.20.2 headroom/proxy/handlers/openai.py -> Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.15.22 and mypy 1.20.2 via uvx.
- Exact command / steps: drove `_ws_http_fallback` with the existing
WS/stream mocks, feeding an SSE `response.completed` carrying
`usage.input_tokens=31055`, `output_tokens=246`,
`input_tokens_details.cached_tokens=20000`. The method now returns
`(31055, 246, 20000, ..., 11055)`; a stream with no completed event
returns all zeros. The existing 11 relay/routing/retry tests are
unchanged (they ignore the new return value).
- Observed result: the fallback surfaces the provider's real input
usage, so the WS session-end outcome records the actual input tokens
instead of 0, and savings percentages stay within a meaningful range.
- Not tested: a live Codex WS session that triggers the upstream-WS
rejection and HTTP fallback end to end (needs a real upstream refusing
the WS). The usage-propagation contract is verified at the fallback
boundary with the same mocks the existing fallback tests use.
## Runtime Rollout Safety
- Rollout-managed feature(s): none. The OpenAI Responses WS-to-HTTP
fallback is always-on transport behavior, not rollout-channel-gated.
- Minimum rollout channel: N/A (no rollout-managed behavior).
- Stable/default behavior changed: yes, as a bug fix. On the WS-to-HTTP
fallback the session-end outcome now records the provider's real
input/output/cache usage from `response.completed` instead of leaving
`ws_input_tokens_total` at 0 (which produced >100% savings). SSE relay
to the client is unchanged.
- Kill switch / disable path: N/A. This corrects accounting only; there
is no behavioral toggle and no user-facing surface beyond the recorded
outcome numbers.
- Unsafe override required: no.
- Qualification impact: fallback-path token accounting now matches the
non-fallback WS path and the HTTP Responses path (all three use
`_extract_responses_usage`); savings percentages return to a valid
range.
- Rollback path: revert this PR; the fallback returns to reporting zero
input usage on this path.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`: it is generated by
release-please from my Conventional Commit PR title
## Additional Notes
The fix reuses the already-present `_extract_responses_usage` (same
parser the non-fallback WS path and HTTP Responses path use), so
cache-read/write and uncached accounting stay consistent across all
three transports.
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
Gil Korzen
|
a06a51eca6
|
fix(proxy): preserve Codex WebSocket model attribution (#3029)
## Description
Codex can switch models during a multi-turn Responses WebSocket
conversation. Headroom was not consistently attributing each completed
turn to the model that handled it, which made per-model usage and
savings reporting inaccurate.
Closes #3027
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Attribute each completed WebSocket response to its reported model.
- Keep session-end metrics consistent with the response that completed.
- Add a regression test covering two different models on one WebSocket
session.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added new functionality
- [x] Manual testing performed
### Test Output
```text
uv run pytest -q tests/test_openai_codex_ws_lifecycle.py -k session_metrics_track_model_per_response_create
1 passed, 51 deselected in 2.09s
Full Codex WebSocket lifecycle module: 52 passed
Adjacent Codex WebSocket suites: 77 passed, 1 skipped
uv run ruff check .
All checks passed
uv run ruff format --check .
1411 files already formatted
uv run mypy headroom
Success: no issues found in 520 source files
```
## Real Behavior Proof
- Environment: Windows, Python 3.13.3, OpenAI Codex Responses WebSocket.
- Exact command / steps: From the repository root, run `uv sync --extra
dev --extra proxy`, then run `uv run headroom wrap codex`; in one live
Codex conversation complete one turn with model A, switch to model B,
complete a second turn, and inspect the proxy dashboard or
`http://localhost:8787/stats` recent requests.
- Observed result: Both completed turns appeared under the models that
handled them, in order.
- Not tested: Production deployment and non-Codex transports.
## Runtime Rollout Safety
- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable/default.
- Stable/default behavior changed: Corrects telemetry attribution only;
no public API or routing changes.
- Kill switch / disable path: Revert the change or use the previous
release.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert commit `
|
||
|
Abhay Singh
|
a01897c791
|
fix(proxy/gemini): guard CCR continuation usage against present-null counts (#3035)
## Description
On the Gemini native `generateContent` path, a successful (200) response
that triggers a CCR retrieval continuation is masked as a synthetic 502
when the continuation response carries a present-null usage count.
`handle_gemini_generate_content` reads `usageMetadata` at three sites.
The initial-response site and the non-CCR site both guard against Gemini
returning a present-null count (a key present with a JSON `null`, which
`.get(key, default)` returns as `None` rather than the default). The
CCR-continuation site read the continuation's `usageMetadata` with a
bare `.get(key, prior)`:
```python
total_input_tokens = usage.get("promptTokenCount", total_input_tokens)
output_tokens = usage.get("candidatesTokenCount", output_tokens)
cache_read_tokens = usage.get("cachedContentTokenCount", cache_read_tokens)
```
When the continuation turn reports `"promptTokenCount": null`,
`total_input_tokens` becomes `None`, and the following
`uncached_input_tokens = max(0, total_input_tokens - cache_read_tokens)`
and the `total_input_tokens > 0` baseline guard raise `TypeError`. The
method's outer `except Exception` then returns a 502 JSONResponse and
records a provider failure, so a genuinely successful upstream turn is
reported to the client as a 502.
## Fix
Read the continuation usage through the same `_usage_int` guard the two
sibling sites use, keeping the pre-continuation count as the fallback
(`_usage_int(value, default)` returns `default` when `value is None`).
Behavior is otherwise unchanged: a present, valid count is still used,
and an absent count still falls back to the pre-continuation value.
Fixes #3034
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/handlers/gemini.py` (`handle_gemini_generate_content`,
CCR-continuation branch): read `promptTokenCount` /
`candidatesTokenCount` / `cachedContentTokenCount` through
`_usage_int(..., prior)` instead of a bare `.get(key, prior)`.
- `tests/test_gemini_ccr_continuation_usage.py`: drive the handler
through a CCR continuation whose `usageMetadata` counts are
present-null; assert the client gets 200 (not 502), no provider failure
is recorded, and the pre-continuation count survives as the fallback.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added
### Test Output
```text
tests/test_gemini_ccr_continuation_usage.py 1 passed
tests/test_gemini_nonjson_status.py tests/test_gemini_compression_offload.py tests/test_proxy_gemini_native_integration.py (all pass; platform-skipped cases skipped)
# uvx ruff@0.15.22 check -> All checks passed!
# uvx mypy@1.20.2 headroom/proxy/handlers/gemini.py -> Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.15.22 and mypy 1.20.2 via uvx.
- Exact command / steps: ran `python -m pytest
tests/test_gemini_ccr_continuation_usage.py -q` (pass-after); proved
fail-before by `git stash`-ing only the `gemini.py` change and
re-running (the test failed with `assert 502 == 200` and the captured
log `TypeError: unsupported operand type(s) for -: 'NoneType' and
'NoneType'` at `gemini.py`), then restored the fix and re-ran green; ran
the surrounding Gemini suite (`test_gemini_nonjson_status.py`,
`test_gemini_compression_offload.py`,
`test_proxy_gemini_native_integration.py`); then `uvx ruff@0.15.22
check` and `uvx mypy@1.20.2 headroom/proxy/handlers/gemini.py`.
- Observed result: with the fix a CCR continuation carrying a
present-null `promptTokenCount` returns 200 to the client and records
the outcome with the pre-continuation count (100) instead of raising
`TypeError` and returning a synthetic 502.
- Not tested: a live Gemini session that both triggers a CCR retrieval
continuation and receives a present-null continuation usage payload
(needs a real safety-blocked continuation). The contract is verified at
the handler with the same stub pattern the existing
`test_gemini_nonjson_status.py` uses.
## Runtime Rollout Safety
- Rollout-managed feature(s): none. This is the always-on Gemini native
`generateContent` request path, not a rollout-channel-gated feature.
- Minimum rollout channel: N/A (no rollout-managed behavior).
- Stable/default behavior changed: yes, as a bug fix. A CCR continuation
with a present-null usage count now returns the real 200 instead of a
synthetic 502; all other cases (present valid count, absent count) are
unchanged.
- Kill switch / disable path: N/A. There is no behavioral toggle; the
change only makes the existing continuation path null-safe.
- Unsafe override required: no.
- Qualification impact: brings the CCR-continuation usage extraction to
parity with the two sibling sites that already guard present-null
counts; no routing, compression, or pricing change.
- Rollback path: revert this PR; the continuation site returns to the
bare `.get(key, prior)` read.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`: it is generated by
release-please from my Conventional Commit PR title
## Additional Notes
The unguarded site was introduced in #2253 (native CCR retrieval); the
present-null guard on the sibling sites landed separately and did not
extend to it. The fix reuses the existing `_usage_int` helper so all
three Gemini usage-extraction sites now handle present-null identically.
|
||
|
Parideboy
|
9d370592b0
|
fix(proxy): stop cached responses replaying the producing turn's wire framing (#3024)
## Description
Closes #3019
A response-cache hit could hand the client an HTTP 200 that the client
could not read, and nothing in the logs marked the turn as anything
other than normal.
Two separate problems combine to produce the reported failure.
**The unreadable 200.** A cache entry stores the producing upstream's
response headers verbatim. When the entry is replayed, the Anthropic
handler removed only `content-encoding`, `content-length` and
`content-type` before handing those headers to a brand-new `Response`.
Anything else describing how that *other* connection framed its body
rode along — most damagingly `transfer-encoding: chunked`. RFC 9112 §6.1
makes `Transfer-Encoding` override `Content-Length`, so the client is
told to parse a plain JSON body as chunked frames, finds no valid
chunk-size line, and reads an empty body out of a 200. Every other
response-forwarding site in the Python proxy already strips that header;
the two cache-hit sites were the only ones that did not.
**How a CCR turn could put a foreign response in the cache.** On the
Anthropic path, `cache.get` is gated on `not stream` but `cache.set` was
not, and the cache key has no `stream` component. A CCR buffered-stream
conversion takes a request the client sent with `stream: true`, forces
`stream: false` upstream, and — unlike every other streaming turn, which
returns via `_stream_response` and never touches the cache — falls
through to the store site. The stored reply was shaped by that forced
flip plus CCR tool injection, and the key cannot distinguish it from an
ordinary non-streaming reply, so a later non-streaming caller could be
served a response built for a request it never made. This is why the
reporters saw the failures pair with CCR activity and stop under
`--lossless` / `--no-ccr`.
**Why it was invisible.** The cache-hit block emitted no log line at
all, and the `PERF` line rendered no field for
`RequestOutcome.from_response_cache`. A cache-served turn contacts no
upstream, so it has no `outbound_request` line, no upstream stage
timings, and all-zero token counters — byte-for-byte what a turn that
died would look like. That is why `headroom doctor` reported zero
failures while turns were dying.
### Scope note
The header fix also lands on the OpenAI cache-hit site, which
additionally never received the `content-type` fix from #2952. The `not
stream` gate is added to the OpenAI store site too, where it is
currently redundant — a streaming chat request returns via
`_stream_response` long before that point — purely to state the
invariant, since the Anthropic handler had exactly that shape until a
buffered-CCR branch began falling through to it.
Because the strip list now lives in one shared helper, the OpenAI
handler's other five forwarding sites strip the three added headers as
well. That is a widening, so it is worth being explicit about: each of
those sites builds a fresh fixed-length `Response` (or, at
`openai.py:6122`, synthesises SSE) from `response.content`, so replaying
the upstream's framing there was the same latent bug, just without a
cache to make it outlive the request that produced it. The precedent is
already in the file — `openai.py:9865` passes `"transfer-encoding",
"connection"` as extra names by hand, which is exactly the gap this PR
closes centrally. That call site keeps its now-redundant arguments;
removing them is a cleanup for another PR.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- Added `sanitize_forwarded_response_headers` to
`headroom/proxy/helpers.py`, promoting the private helper that already
lived in `headroom/proxy/handlers/openai.py` and extending it with the
remaining wire-framing headers (`transfer-encoding`, `connection`,
`keep-alive`). Matching is now case-insensitive; surviving headers keep
their original casing. `openai.py`'s
`_sanitize_forwarded_response_headers` is now a thin alias so its six
call sites and the Anthropic handler strip an identical set.
- `headroom/proxy/handlers/anthropic.py`: the response-cache hit now
sanitises through that helper (passing `content-type` as an extra name,
preserving #2952) instead of three hand-rolled `pop` calls.
- `headroom/proxy/handlers/openai.py`: the response-cache hit sanitises
the same way, gains the `content-type` handling it was missing, and sets
`media_type="application/json"` explicitly.
- `headroom/proxy/handlers/anthropic.py`: `cache.set` is now gated on
`not stream`, mirroring the read gate. `stream` still holds the client's
original flag at that point — the buffered-CCR conversion flips
`body["stream"]`, never the local variable.
- `headroom/proxy/handlers/openai.py`: the same `not stream` gate on its
store site, as an invariant guard.
- Both cache-hit sites now log `RESPONSE-CACHE-HIT: model=… bytes=…
age_s=… hits=…`, following the existing `CACHE-MISS-ATTRIBUTION` line
style.
- `headroom/proxy/outcome.py`: the `PERF` line appends `cached=1` on a
response-cache hit. It is appended only on a hit, so every other PERF
line is byte-identical to before and existing parsers are unaffected.
- `headroom/perf/analyzer.py`: `PerfRecord.from_response_cache` reads
that field, so `headroom perf` can tell a cache-served turn from a dead
one. It defaults to `False`, so older logs still parse.
`PERF_RECORD_FIELDS` gains the name at the end of the list, which is
what `headroom perf --format csv --raw` uses as its column set;
appending keeps every existing column at its current position. `--format
json --raw` gains the key too.
- `tests/test_anthropic_pre_upstream_backpressure.py`: its cache-hit
double was a partial hand-rolled stand-in for `CacheEntry` carrying only
a body and headers, so it broke once the hit path started reading the
entry's age and hit count. It now constructs a real `CacheEntry`, which
is what the cache actually returns.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
### Test Output
```text
$ python -m pytest tests/test_proxy_response_cache_replay.py -q
tests\test_proxy_response_cache_replay.py ......... [100%]
============================== 9 passed in 4.22s ==============================
# Everything that mentions PERF, the sanitiser, cache.set, PerfRecord or
# response_headers, plus the whole proxy suite.
$ python -m pytest tests/test_proxy/ tests/test_proxy_compression_headers.py \
tests/test_agent_savings.py tests/test_anthropic_pre_upstream_backpressure.py \
tests/test_backend_nonstreaming_cache_metrics.py tests/test_backend_streaming_cache_metrics.py \
tests/test_ccr_buffered_stream_signed_thinking.py tests/test_cli_perf_format.py \
tests/test_codex_ws_compression_scheduler.py tests/test_handler_outcome_tag_invariant.py \
tests/test_openai_codex_ws_lifecycle.py tests/test_provider_codex_images.py \
tests/test_proxy_handlers_batch.py tests/test_proxy_passthrough_transient_retry.py \
tests/test_proxy_response_cache_replay.py tests/test_proxy_semantic_cache_key.py \
tests/test_proxy_streaming_request_logger.py tests/test_request_outcome.py \
tests/test_savings_tool_search_aggregation.py -q
================== 555 passed, 1 skipped in 88.60s (0:01:28) ==================
# Full suite, 16 workers. See "Real Behavior Proof" below for how every
# failure here was traced to a pre-existing failure or a parallelism flake.
$ python -m pytest tests scripts/tests -n 16 -q -p no:randomly --timeout=300
83 failed, 10493 passed, 657 skipped, 80 errors in 437.00s (0:07:17)
$ ruff check .
All checks passed!
$ ruff format --check <the 7 changed files>
7 files already formatted
$ python -m mypy headroom --ignore-missing-imports --python-version 3.13
Found 12 errors in 3 files (checked 520 source files)
# All 12 are pre-existing MCP-SDK/tomllib drift in release_version.py,
# ccr/mcp_server.py and memory/mcp_server.py; identical count before and
# after this change, none in the files it touches.
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.13.11, pytest 9.1.1, ruff 0.16.2,
branch based on `upstream/main` at `
|
||
|
Tejas Chopra
|
f9807fd69e
|
feat(proxy): let extensions report cost savings and their own latency (#3051)
## What
Two changes that let a proxy extension report **what it saved** and
**what it cost**, so both show up under `/stats`, the dashboard, and
Prometheus.
`record_scope_savings` already existed and already accepted `usd` — the
one channel in the proxy that can express savings *without* tokens. Two
things stopped it working end to end.
### 1. Savings were silently dropped on Gemini traffic (bug)
`bind_scope` shares one attribution ledger between ASGI middleware and
the request handler. Anthropic and OpenAI call it; **Gemini never did**,
so anything an extension recorded into the request scope was discarded
for Gemini traffic only — silently, because an empty ledger and an
unbound one are indistinguishable at the outcome funnel. Now bound at
all four Gemini tag sites.
### 2. An extension's own latency was invisible (gap)
`overhead_ms` is measured *inside* the handler, and an ASGI extension
**wraps** that handler — so every millisecond it spends reaches the
client while every timing surface stays flat. An extension that halves
the bill and adds 200 ms per request is a trade the operator has to see
both halves of, and only one half was reaching the dashboard.
`record_scope_timing(scope, stage, ms)` is the symmetric counterpart to
`record_scope_savings`, carried on the same bound ledger and merged into
`RequestOutcome.pipeline_timing` at the outcome funnel — one place, so
every provider picks it up at once.
## API surface
```python
from headroom.proxy.savings_attribution import record_scope_savings, record_scope_timing
record_scope_savings(scope, "my_extension", tokens=0, usd=0.004) # money without tokens
record_scope_timing(scope, "my_extension", elapsed_ms)
```
Both take the ASGI `scope`, because middleware has no other way in.
Documented in `extensions.py` — the module extension authors actually
read, and the stability contract for this interface.
- Savings → `/stats` `savings.by_source`, dashboard card,
`headroom_savings_attributed_usd_total{source=...}`
- Timing → `/stats` `pipeline_timing`, dashboard Performance panel,
`headroom_transform_timing_ms_*`
**Attribution only.** These rows explain the headline total; they are
never added to it.
## Changes to existing behavior
- `public_tags` now strips `_headroom_stage_timing` as well as
`_headroom_savings_attribution`. Both ride on `tags` because that is the
one dict reaching the outcome funnel from every handler, and a list and
a dict must not land in a string-keyed label store.
- `pipeline_timing` passed to `metrics.record_request` is merged rather
than passed through **only when an extension contributed timings**; with
no extension the handler's own dict is passed through unchanged
(asserted by identity in the tests).
- Stage names are extension-supplied, so they are capped at 16 and
namespaced `ext:` — `deep_copy` reported by a plugin must never
accumulate into the same series as `deep_copy` measured by the pipeline.
A handler's own timing wins a collision (unreachable while the prefix
stands; the safe way round if it ever goes).
## Failure modes
Both calls are bounded (32 sources, 16 stages), never raise, and never
change a response — telemetry from a plugin must not be able to break
the request it is describing. Non-positive and non-numeric durations are
ignored: a zero is a clock artifact, not an observation, and averaging
it in would drag the mean down exactly where the stage is cheapest to
skip. `timings_from_tags` tolerates junk on the tag.
## Test-double fix
Three Gemini test fakes (`FakeRequest`, `_FakeRequest`,
`_VertexGeminiImageRequest`) had no `.scope`, which every real Starlette
`Request` has. They now do. This is a double that had drifted from the
type it stands in for; the alternative was weakening the handler to
tolerate a request shape that cannot occur in production.
---
## Real behavior proof
**Setup:** macOS 15.4 (darwin 25.4.0), Python 3.12.13, this branch at
`
|
||
|
Joseph Benno
|
2f4d001c9f
|
fix(proxy): keep prefixed core tools resident (#3046)
## Description Headroom's Tool Search deferral lowercased core tool names but did not account for client namespace prefixes. Oh My Pi sends built-ins such as `_read`, `_edit`, `_write`, and `_bash`, so those core tools were incorrectly marked `defer_loading=True`. This change centralizes resident-name normalization for both the Anthropic and OpenAI paths. It lowercases names and removes only leading underscores, preserving internal separators such as `mcp__server__read` so unrelated tools do not become resident. Closes #3031 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added a shared resident-tool name normalizer in `headroom/proxy/helpers.py`. - Applied the same normalization to Anthropic and OpenAI Tool Search deferral. - Added a regression test for Oh My Pi's exact 12-tool surface at the deferral threshold. - Added OpenAI coverage for prefixed resident tools and negative namespace cases. ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --no-sync pytest --noconftest -q tests/test_openai_tool_search_deferral.py tests/test_issue_746_tool_search.py -k 'not normalize_tool_search_mode and not configure_' 72 passed, 23 deselected in 0.25s $ uv run --no-sync ruff check . All checks passed! $ uv run --no-sync ruff format --check . 1499 files already formatted $ UV_CACHE_DIR=/tmp/headroom-uv-cache uv run --no-sync mypy headroom Success: no issues found in 520 source files ``` ## Real Behavior Proof - Environment: Linux x86_64 sandbox; Python 3.12.13; uv 0.11.33; no provider credentials. - Exact command / steps: Exercised the exact 12-tool Oh My Pi fixture through the Anthropic deferral helper and prefixed resident plus negative names through the OpenAI helper. - Observed result: Anthropic kept `_edit`, `_task`, `_read`, `_bash`, `_glob`, `_grep`, `_write`, `computer`, and `web_search` resident while deferring `_hub`, `_todo`, and `_eval`. OpenAI kept prefixed core tools resident while `mcp__server__read` and `terminal_helper` remained deferred. - Not tested: Live Oh My Pi traffic against Anthropic, provider E2E tests, and the full native-backed pytest suite. ## Runtime Rollout Safety - Rollout-managed feature(s): Existing server-side Tool Search deferral for Anthropic and OpenAI. - Minimum rollout channel: N/A; targeted bug fix to existing behavior. - Stable/default behavior changed: Yes. Leading-underscore names that normalize to known resident names now remain resident. - Kill switch / disable path: Set `HEADROOM_TOOL_SEARCH=0`. - Unsafe override required: No. - Qualification impact: Prefixed core tools remain immediately available; non-core and MCP namespace behavior is unchanged. - Rollback path: Revert this commit or disable Tool Search with `HEADROOM_TOOL_SEARCH=0`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes |
||
|
dependabot[bot]
|
322425c43b
|
deps: bump sha2 from 0.10.9 to 0.11.0 (#2288)
Bumps [sha2](https://github.com/RustCrypto/hashes) from 0.10.9 to 0.11.0. <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
5731be7e68
|
deps: bump axum from 0.7.9 to 0.8.9 (#2966)
Bumps [axum](https://github.com/tokio-rs/axum) from 0.7.9 to 0.8.9. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/tokio-rs/axum/releases">axum's releases</a>.</em></p> <blockquote> <h2>axum-v0.8.9</h2> <ul> <li><strong>added:</strong> <code>WebSocketUpgrade::{requested_protocols, set_selected_protocol}</code> for more flexible subprotocol selection (<a href="https://redirect.github.com/tokio-rs/axum/issues/3597">#3597</a>)</li> <li><strong>changed:</strong> Update minimum rust version to 1.80 (<a href="https://redirect.github.com/tokio-rs/axum/issues/3620">#3620</a>)</li> <li><strong>fixed:</strong> Set connect endpoint on correct field in MethodRouter (<a href="https://redirect.github.com/tokio-rs/axum/issues/3656">#3656</a>)</li> <li><strong>fixed:</strong> Return specific error message when multipart body limit is exceeded (<a href="https://redirect.github.com/tokio-rs/axum/issues/3611">#3611</a>)</li> </ul> <p><a href="https://redirect.github.com/tokio-rs/axum/issues/3597">#3597</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3597">tokio-rs/axum#3597</a> <a href="https://redirect.github.com/tokio-rs/axum/issues/3620">#3620</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3620">tokio-rs/axum#3620</a> <a href="https://redirect.github.com/tokio-rs/axum/issues/3656">#3656</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3656">tokio-rs/axum#3656</a> <a href="https://redirect.github.com/tokio-rs/axum/issues/3611">#3611</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3611">tokio-rs/axum#3611</a></p> <h2>axum v0.8.8</h2> <ul> <li>Clarify documentation for <code>Router::route_layer</code> (<a href="https://redirect.github.com/tokio-rs/axum/issues/3567">#3567</a>)</li> </ul> <p><a href="https://redirect.github.com/tokio-rs/axum/issues/3567">#3567</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3567">tokio-rs/axum#3567</a></p> <h2>axum v0.8.7</h2> <ul> <li>Relax implicit <code>Send</code> / <code>Sync</code> bounds on <code>RouterAsService</code>, <code>RouterIntoService</code> (<a href="https://redirect.github.com/tokio-rs/axum/issues/3555">#3555</a>)</li> <li>Make it easier to visually scan for default features (<a href="https://redirect.github.com/tokio-rs/axum/issues/3550">#3550</a>)</li> <li>Fix some documentation typos</li> </ul> <p><a href="https://redirect.github.com/tokio-rs/axum/issues/3550">#3550</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3550">tokio-rs/axum#3550</a> <a href="https://redirect.github.com/tokio-rs/axum/issues/3555">#3555</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3555">tokio-rs/axum#3555</a></p> <h2>axum v0.8.5</h2> <ul> <li><strong>fixed:</strong> Reject JSON request bodies with trailing characters after the JSON document (<a href="https://redirect.github.com/tokio-rs/axum/issues/3453">#3453</a>)</li> <li><strong>added:</strong> Implement <code>OptionalFromRequest</code> for <code>Multipart</code> (<a href="https://redirect.github.com/tokio-rs/axum/issues/3220">#3220</a>)</li> <li><strong>added:</strong> Getter methods <code>Location::{status_code, location}</code></li> <li><strong>added:</strong> Support for writing arbitrary binary data into server-sent events (<a href="https://redirect.github.com/tokio-rs/axum/issues/3425">#3425</a>)]</li> <li><strong>added:</strong> <code>middleware::ResponseAxumBodyLayer</code> for mapping response body to <code>axum::body::Body</code> (<a href="https://redirect.github.com/tokio-rs/axum/issues/3469">#3469</a>)</li> <li><strong>added:</strong> <code>impl FusedStream for WebSocket</code> (<a href="https://redirect.github.com/tokio-rs/axum/issues/3443">#3443</a>)</li> <li><strong>changed:</strong> The <code>sse</code> module and <code>Sse</code> type no longer depend on the <code>tokio</code> feature (<a href="https://redirect.github.com/tokio-rs/axum/issues/3154">#3154</a>)</li> <li><strong>changed:</strong> If the location given to one of <code>Redirect</code>s constructors is not a valid header value, instead of panicking on construction, the <code>IntoResponse</code> impl now returns an HTTP 500, just like <code>Json</code> does when serialization fails (<a href="https://redirect.github.com/tokio-rs/axum/issues/3377">#3377</a>)</li> <li><strong>changed:</strong> Update minimum rust version to 1.78 (<a href="https://redirect.github.com/tokio-rs/axum/issues/3412">#3412</a>)</li> </ul> <p><a href="https://redirect.github.com/tokio-rs/axum/issues/3154">#3154</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3154">tokio-rs/axum#3154</a> <a href="https://redirect.github.com/tokio-rs/axum/issues/3220">#3220</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3220">tokio-rs/axum#3220</a> <a href="https://redirect.github.com/tokio-rs/axum/issues/3377">#3377</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3377">tokio-rs/axum#3377</a> <a href="https://redirect.github.com/tokio-rs/axum/issues/3412">#3412</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3412">tokio-rs/axum#3412</a> <a href="https://redirect.github.com/tokio-rs/axum/issues/3425">#3425</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3425">tokio-rs/axum#3425</a> <a href="https://redirect.github.com/tokio-rs/axum/issues/3443">#3443</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3443">tokio-rs/axum#3443</a> <a href="https://redirect.github.com/tokio-rs/axum/issues/3453">#3453</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3453">tokio-rs/axum#3453</a> <a href="https://redirect.github.com/tokio-rs/axum/issues/3469">#3469</a>: <a href="https://redirect.github.com/tokio-rs/axum/pull/3469">tokio-rs/axum#3469</a></p> <h2>axum v0.8.4</h2> <ul> <li><strong>added:</strong> <code>Router::reset_fallback</code> (<a href="https://redirect.github.com/tokio-rs/axum/issues/3320">#3320</a>)</li> <li><strong>added:</strong> <code>WebSocketUpgrade::selected_protocol</code> (<a href="https://redirect.github.com/tokio-rs/axum/issues/3248">#3248</a>)</li> <li><strong>fixed:</strong> Panic location for overlapping method routes (<a href="https://redirect.github.com/tokio-rs/axum/issues/3319">#3319</a>)</li> <li><strong>fixed:</strong> Don't leak a tokio task when using <code>serve</code> without graceful shutdown (<a href="https://redirect.github.com/tokio-rs/axum/issues/3129">#3129</a>)</li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
ff17961cd7
|
deps: bump ruff from 0.15.22 to 0.16.2 in the pip-minor-patch group across 1 directory (#2962)
Bumps the pip-minor-patch group with 1 update in the / directory: [ruff](https://github.com/astral-sh/ruff). Updates `ruff` from 0.15.22 to 0.16.2 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/astral-sh/ruff/releases">ruff's releases</a>.</em></p> <blockquote> <h2>0.16.2</h2> <h2>Release Notes</h2> <p>Released on 2026-08-06.</p> <h3>Bug fixes</h3> <ul> <li>[<code>flake8-pyi</code>] Avoid false positives on <code>singledispatch</code> functions (<code>PYI041</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/27335">#27335</a>)</li> </ul> <h3>Server</h3> <ul> <li>Register formatting capabilities dynamically to exclude TOML files (<a href="https://redirect.github.com/astral-sh/ruff/pull/27332">#27332</a>)</li> </ul> <h3>Contributors</h3> <ul> <li><a href="https://github.com/MeGaGiGaGon"><code>@MeGaGiGaGon</code></a></li> <li><a href="https://github.com/charliermarsh"><code>@charliermarsh</code></a></li> <li><a href="https://github.com/epage"><code>@epage</code></a></li> <li><a href="https://github.com/sharkdp"><code>@sharkdp</code></a></li> <li><a href="https://github.com/ntBre"><code>@ntBre</code></a></li> </ul> <h2>Install ruff 0.16.2</h2> <h3>Install prebuilt binaries via shell script</h3> <pre lang="sh"><code>curl --proto '=https' --tlsv1.2 -LsSf https://releases.astral.sh/github/ruff/releases/download/0.16.2/ruff-installer.sh | sh </code></pre> <h3>Install prebuilt binaries via powershell script</h3> <pre lang="sh"><code>powershell -ExecutionPolicy Bypass -c "irm https://releases.astral.sh/github/ruff/releases/download/0.16.2/ruff-installer.ps1 | iex" </code></pre> <h2>Download ruff 0.16.2</h2> <table> <thead> <tr> <th>File</th> <th>Platform</th> <th>Checksum</th> </tr> </thead> <tbody> <tr> <td><a href="https://releases.astral.sh/github/ruff/releases/download/0.16.2/ruff-aarch64-apple-darwin.tar.gz">ruff-aarch64-apple-darwin.tar.gz</a></td> <td>Apple Silicon macOS</td> <td><a href="https://releases.astral.sh/github/ruff/releases/download/0.16.2/ruff-aarch64-apple-darwin.tar.gz.sha256">checksum</a></td> </tr> <tr> <td><a href="https://releases.astral.sh/github/ruff/releases/download/0.16.2/ruff-x86_64-apple-darwin.tar.gz">ruff-x86_64-apple-darwin.tar.gz</a></td> <td>Intel macOS</td> <td><a href="https://releases.astral.sh/github/ruff/releases/download/0.16.2/ruff-x86_64-apple-darwin.tar.gz.sha256">checksum</a></td> </tr> <tr> <td><a href="https://releases.astral.sh/github/ruff/releases/download/0.16.2/ruff-aarch64-pc-windows-msvc.zip">ruff-aarch64-pc-windows-msvc.zip</a></td> <td>ARM64 Windows</td> <td><a href="https://releases.astral.sh/github/ruff/releases/download/0.16.2/ruff-aarch64-pc-windows-msvc.zip.sha256">checksum</a></td> </tr> <tr> <td><a href="https://releases.astral.sh/github/ruff/releases/download/0.16.2/ruff-i686-pc-windows-msvc.zip">ruff-i686-pc-windows-msvc.zip</a></td> <td>x86 Windows</td> <td><a href="https://releases.astral.sh/github/ruff/releases/download/0.16.2/ruff-i686-pc-windows-msvc.zip.sha256">checksum</a></td> </tr> <tr> <td><a href="https://releases.astral.sh/github/ruff/releases/download/0.16.2/ruff-x86_64-pc-windows-msvc.zip">ruff-x86_64-pc-windows-msvc.zip</a></td> <td>x64 Windows</td> <td><a href="https://releases.astral.sh/github/ruff/releases/download/0.16.2/ruff-x86_64-pc-windows-msvc.zip.sha256">checksum</a></td> </tr> <tr> <td><a href="https://releases.astral.sh/github/ruff/releases/download/0.16.2/ruff-aarch64-unknown-linux-gnu.tar.gz">ruff-aarch64-unknown-linux-gnu.tar.gz</a></td> <td>ARM64 Linux</td> <td><a href="https://releases.astral.sh/github/ruff/releases/download/0.16.2/ruff-aarch64-unknown-linux-gnu.tar.gz.sha256">checksum</a></td> </tr> <tr> <td><a href="https://releases.astral.sh/github/ruff/releases/download/0.16.2/ruff-i686-unknown-linux-gnu.tar.gz">ruff-i686-unknown-linux-gnu.tar.gz</a></td> <td>x86 Linux</td> <td><a href="https://releases.astral.sh/github/ruff/releases/download/0.16.2/ruff-i686-unknown-linux-gnu.tar.gz.sha256">checksum</a></td> </tr> <tr> <td><a href="https://releases.astral.sh/github/ruff/releases/download/0.16.2/ruff-powerpc64-unknown-linux-gnu.tar.gz">ruff-powerpc64-unknown-linux-gnu.tar.gz</a></td> <td>PPC64 Linux</td> <td><a href="https://releases.astral.sh/github/ruff/releases/download/0.16.2/ruff-powerpc64-unknown-linux-gnu.tar.gz.sha256">checksum</a></td> </tr> <tr> <td><a href="https://releases.astral.sh/github/ruff/releases/download/0.16.2/ruff-powerpc64le-unknown-linux-gnu.tar.gz">ruff-powerpc64le-unknown-linux-gnu.tar.gz</a></td> <td>PPC64LE Linux</td> <td><a href="https://releases.astral.sh/github/ruff/releases/download/0.16.2/ruff-powerpc64le-unknown-linux-gnu.tar.gz.sha256">checksum</a></td> </tr> <tr> <td><a href="https://releases.astral.sh/github/ruff/releases/download/0.16.2/ruff-riscv64gc-unknown-linux-gnu.tar.gz">ruff-riscv64gc-unknown-linux-gnu.tar.gz</a></td> <td>RISCV Linux</td> <td><a href="https://releases.astral.sh/github/ruff/releases/download/0.16.2/ruff-riscv64gc-unknown-linux-gnu.tar.gz.sha256">checksum</a></td> </tr> <tr> <td><a href="https://releases.astral.sh/github/ruff/releases/download/0.16.2/ruff-s390x-unknown-linux-gnu.tar.gz">ruff-s390x-unknown-linux-gnu.tar.gz</a></td> <td>S390x Linux</td> <td><a href="https://releases.astral.sh/github/ruff/releases/download/0.16.2/ruff-s390x-unknown-linux-gnu.tar.gz.sha256">checksum</a></td> </tr> </tbody> </table> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md">ruff's changelog</a>.</em></p> <blockquote> <h2>0.16.2</h2> <p>Released on 2026-08-06.</p> <h3>Bug fixes</h3> <ul> <li>[<code>flake8-pyi</code>] Avoid false positives on <code>singledispatch</code> functions (<code>PYI041</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/27335">#27335</a>)</li> </ul> <h3>Server</h3> <ul> <li>Register formatting capabilities dynamically to exclude TOML files (<a href="https://redirect.github.com/astral-sh/ruff/pull/27332">#27332</a>)</li> </ul> <h3>Contributors</h3> <ul> <li><a href="https://github.com/MeGaGiGaGon"><code>@MeGaGiGaGon</code></a></li> <li><a href="https://github.com/charliermarsh"><code>@charliermarsh</code></a></li> <li><a href="https://github.com/epage"><code>@epage</code></a></li> <li><a href="https://github.com/sharkdp"><code>@sharkdp</code></a></li> <li><a href="https://github.com/ntBre"><code>@ntBre</code></a></li> </ul> <h2>0.16.1</h2> <p>Released on 2026-07-30.</p> <h3>Preview features</h3> <ul> <li>Add an option to opt out of human-readable names (<a href="https://redirect.github.com/astral-sh/ruff/pull/27160">#27160</a>)</li> <li>[<code>flake8-pytest-style</code>] Make fixes safe by default and unsafe only when comments are present (<code>PT018</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/27201">#27201</a>)</li> <li>[<code>pyupgrade</code>] Skip fix when a defaulted <code>TypeVar</code> precedes a non-defaulted one (<code>UP040</code>, <code>UP046</code>, <code>UP047</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/27133">#27133</a>)</li> <li>[<code>ruff</code>] Fix false positive with unpacked arguments (<code>RUF065</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/26959">#26959</a>)</li> </ul> <h3>Bug fixes</h3> <ul> <li>Bump <code>gen-lsp-types</code> to gracefully handle unknown enumeration values in LSP messages (<a href="https://redirect.github.com/astral-sh/ruff/pull/27230">#27230</a>)</li> <li>[<code>flake8-bugbear</code>] Mark <code>range</code> as immutable (<code>B008</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/27247">#27247</a>)</li> <li>[<code>flake8-comprehensions</code>] NFKC-normalize keyword names in <code>C408</code> fix (<a href="https://redirect.github.com/astral-sh/ruff/pull/26813">#26813</a>)</li> <li>[<code>flake8-return</code>] Fix false positive when variable is read in <code>finally</code> clause (<code>RET504</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/25441">#25441</a>)</li> <li>[<code>pydocstyle</code>] Skip section detection inside RST directive bodies (<code>D214</code>, <code>D405</code>, <code>D413</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/23635">#23635</a>)</li> <li>[<code>refurb</code>] Parenthesize <code>yield</code> arguments in the <code>FURB192</code> fix (<a href="https://redirect.github.com/astral-sh/ruff/pull/27192">#27192</a>)</li> </ul> <h3>Rule changes</h3> <ul> <li>[<code>flake8-pytest-style</code>] Mark <code>PT022</code> fixes as unsafe (<a href="https://redirect.github.com/astral-sh/ruff/pull/26440">#26440</a>)</li> <li>[<code>refurb</code>] Mark fixes that remove unknown separators as unsafe (<code>FURB105</code>) (<a href="https://redirect.github.com/astral-sh/ruff/pull/27200">#27200</a>)</li> </ul> <h3>Server</h3> <ul> <li>Fix indexing of excluded nested Ruff workspaces (<a href="https://redirect.github.com/astral-sh/ruff/pull/27303">#27303</a>)</li> <li>Lint TOML files in the LSP (<a href="https://redirect.github.com/astral-sh/ruff/pull/26862">#26862</a>)</li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
bbe901319d
|
deps: bump tokio-tungstenite from 0.24.0 to 0.30.0 (#2967)
Bumps [tokio-tungstenite](https://github.com/snapview/tokio-tungstenite) from 0.24.0 to 0.30.0. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/snapview/tokio-tungstenite/blob/master/CHANGELOG.md">tokio-tungstenite's changelog</a>.</em></p> <blockquote> <h1>0.30.0</h1> <ul> <li>Update <code>tungstenite</code> to <code>0.30.0</code>. See <a href="https://github.com/snapview/tungstenite-rs/blob/master/CHANGELOG.md"><code>tungstenite</code> release</a>.</li> </ul> <h1>0.29.0</h1> <ul> <li>Update <code>tungstenite</code> to <code>0.29.0</code>. See <a href="https://github.com/snapview/tungstenite-rs/blob/master/CHANGELOG.md"><code>tungstenite</code> release</a>.</li> </ul> <h1>0.28.0</h1> <ul> <li>Update <code>tungstenite</code> to <code>0.28.0</code>. See <a href="https://github.com/snapview/tungstenite-rs/blob/master/CHANGELOG.md"><code>tungstenite</code> release</a>.</li> </ul> <h1>0.27.0</h1> <ul> <li>See <a href="https://github.com/snapview/tungstenite-rs/blob/master/CHANGELOG.md#0270">performance updates in <code>tungstenite-rs</code></a>.</li> </ul> <h1>0.26.2</h1> <ul> <li>Update <code>tungstenite</code>, see <a href="https://github.com/snapview/tungstenite-rs/blob/master/CHANGELOG.md#0262">changes here</a>.</li> </ul> <h1>0.26.1</h1> <ul> <li>Update <code>tungstenite</code> to address an issue that might cause UB in certain cases.</li> </ul> <h1>0.26.0</h1> <ul> <li>Update <code>tungstenite</code> to <code>0.26.0</code> (<a href="https://github.com/snapview/tungstenite-rs/blob/master/CHANGELOG.md#0260">breaking changes</a>).</li> </ul> <h1>0.25.0</h1> <ul> <li>Update <code>tungstenite</code> to <code>0.25.0</code> (<a href="https://github.com/snapview/tungstenite-rs/blob/master/CHANGELOG.md#0250">important updates!</a>).</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
888a9f4e14
|
deps: bump the cargo-minor-patch group across 1 directory with 4 updates (#2964)
Bumps the cargo-minor-patch group with 4 updates in the / directory: [aws-config](https://github.com/smithy-lang/smithy-rs), [rusqlite](https://github.com/rusqlite/rusqlite), [async-trait](https://github.com/dtolnay/async-trait) and [cc](https://github.com/rust-lang/cc-rs). Updates `aws-config` from 1.10.0 to 1.10.1 <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/smithy-lang/smithy-rs/commits">compare view</a></li> </ul> </details> <br /> Updates `rusqlite` from 0.40.1 to 0.40.2 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/rusqlite/rusqlite/releases">rusqlite's releases</a>.</em></p> <blockquote> <h2>0.40.2</h2> <h2>What's Changed</h2> <ul> <li>Lower MSRV to 1.88.0</li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/rusqlite/rusqlite/compare/v0.40.1...v0.40.2">https://github.com/rusqlite/rusqlite/compare/v0.40.1...v0.40.2</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
JD Davis
|
2d88e31a40
|
fix(claude): reject conflicting auth before proxy startup (#2993)
## Description Fixes #1443. Claude Code rejects an effective configuration containing both ANTHROPIC_API_KEY and ANTHROPIC_AUTH_TOKEN before any request reaches Headroom. The existing wrapper started the proxy and mutated project settings before Claude surfaced its generic Invalid API key message, leaving users to guess which credential came from their shell, global settings, or project settings. Headroom does not own either credential, and both represent legitimate but different auth/billing modes, so automatically deleting one would be destructive. This PR detects the contradiction before any proxy/config mutation and tells the user which source contains each key without exposing credential values. ## Changes Made - Add a pure Claude auth-conflict classifier with explicit settings-layer precedence. - Cover user settings, project .claude/settings.json, project .claude/settings.local.json, and shell environment. - Treat higher-precedence empty values as clearing inherited credentials. - Abort wrap claude before proxy registration/startup when both keys remain effective. - Add a headroom doctor failure with the same source-aware, value-redacted remediation. - Preserve both user credentials and require an explicit choice between API-key billing and token/gateway auth. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text 151 Claude runtime, wrap, doctor, Remote Control, and MCP dependency-contract tests passed ruff check and format checks passed git diff --check passed ``` Branch contains current main, including the MCP v1 cap and the five just-merged blocker PRs. ## Real Behavior Proof - Environment: isolated local worktree on current `main` with Claude wrapper and doctor fixtures. - Exact command / steps: exercised conflicting and non-conflicting shell, user, project, and local-project credential layers through the focused wrap and doctor test suites. - Observed result: conflicting effective credentials fail before proxy startup or settings mutation, report only credential sources, and never expose values. - Not tested: a live Claude Code login with production credentials; credential precedence and side-effect boundaries are covered by fixtures. ## Runtime Rollout Safety - Rollout-managed feature(s): Claude authentication-conflict preflight. - Minimum rollout channel: normal patch release. - Stable/default behavior changed: only configurations with both effective credentials now stop early with actionable diagnostics. - Kill switch / disable path: remove or clear either conflicting credential in its reported source. - Unsafe override required: none; Headroom deliberately does not choose or delete a user credential. - Qualification impact: Claude wrap, doctor, Remote Control, and MCP dependency-contract tests must remain green. - Rollback path: human revert restores the previous late Claude Code rejection; no persisted migration is involved. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Safety No credential value is returned by the classifier, printed by wrap, or emitted in doctor JSON. The preflight runs before _register_proxy_client, proxy startup, MCP registration, or settings writes. |
||
|
JD Davis
|
aa811fa91f
|
ci: allow generated dependency commit bodies (#3012)
## Description Disable commitlint's per-line body length limit because Dependabot generates grouped-update commit bodies with dependency/link lines whose length varies with group contents. PR #2964 currently fails only because one generated line is 274 characters long. Closes # N/A ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Disabled `body-max-line-length` in `.commitlintrc.json`. - Kept Conventional Commit type, subject, and all other configured validation rules enforced. ## Testing - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text PR #2964 Dependabot commit (49 lines; maximum line length 274) exit code: 0 bogus: should fail type must be one of [build, chore, ci, docs, deps, feat, fix, parity, perf, refactor, revert, style, test] [type-enum] exit code: 1 fix: subject may not be empty [subject-empty] exit code: 1 git diff --check exit code: 0 ``` ## Real Behavior Proof - Environment: Windows PowerShell; Node.js 22; `@commitlint/cli` and `@commitlint/config-conventional` 19.8.1. - Exact command / steps: Fetched the current PR #2964 commit message through the GitHub API and piped the complete message into commitlint using this branch's `.commitlintrc.json`; then ran negative type and subject cases. - Observed result: The exact grouped Dependabot commit passed; an unapproved type and empty subject remained rejected. - Not tested: Python/Rust unit tests and runtime behavior; this change only modifies commit-message validation configuration. ## Runtime Rollout Safety - Rollout-managed feature(s): N/A; CI configuration only. - Minimum rollout channel: N/A. - Stable/default behavior changed: Commit bodies may contain lines of any length; all other commitlint rules remain active. - Kill switch / disable path: Revert this commit or restore a numeric `body-max-line-length` limit. - Unsafe override required: No. - Qualification impact: Generated Dependabot group descriptions no longer fail CI due solely to a long dependency/link line. - Rollback path: Revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A; this change has no user interface. ## Additional Notes - The comment and documentation checklist items are not applicable to this one-line commitlint configuration change. - No automated test file was added; the exact positive and negative commitlint cases were run manually as shown above. - Full application tests were not run because no application code or runtime behavior changed. - Keep this PR unmerged pending maintainer review. |
||
|
JD Davis
|
7e3128057c
|
ci: allow Dependabot deps commits (#3009)
## Description Allow the `deps:` Conventional Commit type emitted by Dependabot. Dependabot PRs currently fail the CI `commitlint` job because `deps` is not included in the repository's configured `type-enum`. Closes # N/A ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added `deps` to the allowed commit types in `.commitlintrc.json`. - Existing and future Dependabot commits using `deps: ...` can pass the commit-message policy. ## Testing - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text deps: bump ruff from 0.15.22 to 0.16.2 exit code: 0 bogus: should fail type must be one of [build, chore, ci, docs, deps, feat, fix, parity, perf, refactor, revert, style, test] [type-enum] exit code: 1 git diff --check exit code: 0 ``` ## Real Behavior Proof - Environment: Windows PowerShell; Node.js 22; `@commitlint/cli` and `@commitlint/config-conventional` 19.8.1. - Exact command / steps: Ran commitlint with `.commitlintrc.json` against a real failing Dependabot subject, then against an unapproved `bogus:` type. - Observed result: The `deps:` subject passed; the unapproved type remained rejected by `type-enum`. - Not tested: Python/Rust unit tests and runtime behavior; this change only modifies commit-message validation configuration. ## Runtime Rollout Safety - Rollout-managed feature(s): N/A; CI configuration only. - Minimum rollout channel: N/A. - Stable/default behavior changed: Commitlint now accepts the `deps` type. - Kill switch / disable path: Revert this commit or remove `deps` from `type-enum`. - Unsafe override required: No. - Qualification impact: Dependabot PR commit messages no longer fail solely because their type is `deps`. - Rollback path: Revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A; this change has no user interface. ## Additional Notes - The comment and documentation checklist items are not applicable to this one-line commitlint configuration change. - No automated test file was added; the exact positive and negative commitlint cases were run manually as shown above. - Full application tests were not run because no application code or runtime behavior changed. - Keep this PR unmerged pending maintainer review. |
||
|
Copilot
|
8ea87e7804
|
fix: tool_search_tool_regex deferred and falsely resolved on direct-Anthropic path (#2971)
## Description Direct Anthropic users could receive `400 Tool reference 'tool_search_tool_regex' not found in available tools` when Claude Code sent a typeless `tool_search_tool_regex` entry. Headroom treated it as an ordinary deferrable tool, injected a typed search tool with the same name, and later mistook that typed search mechanism for a valid target of the stale `tool_reference`. This change prevents the duplicate injection and repairs already-poisoned transcripts without stripping valid references to ordinary deferred tools. It addresses the first-party Anthropic regression reported in [PR #2539's follow-up](https://github.com/headroomlabs-ai/headroom/pull/2539#issuecomment-5280259642) and complements the history repair from #2805. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - recognize typeless, case-insensitive `tool_search_tool_*` names as an existing client tool-search surface and skip Headroom's duplicate injection - exclude typed Anthropic search mechanisms from the set of valid `tool_reference` targets - preserve valid regular deferred-tool references and the normal deferral path for similar non-reserved names - add a first-party Anthropic handler regression that proves the outbound tools remain unchanged and stale search bookkeeping is removed - rebase onto #2996, which prevents the native detector from hanging the full CI test shard ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text .venv/Scripts/python.exe -m pytest \ tests/test_issue_746_tool_search.py \ tests/test_anthropic_stage_timings.py \ tests/test_cache_control_ttl_order.py \ tests/test_cache_ttl_preserved.py \ tests/test_proxy/test_tool_search_repair_after_turn_hooks.py \ tests/test_transforms/test_detect_fallback_1123.py \ tests/test_transforms_content_detection.py \ tests/test_transforms_content_router.py \ -q --disable-warnings --maxfail=1 170 passed, 1 warning in 10.27s .venv/Scripts/ruff.exe check . All checks passed! .venv/Scripts/ruff.exe format --check . 1411 files already formatted pre-commit run mypy --all-files Success: no issues found in 519 source files git diff --check (no output) ``` ## Real Behavior Proof - Environment: Windows, Python 3.13.3, first-party Anthropic handler test with `HEADROOM_TOOL_SEARCH` at its default enabled setting - Exact command / steps: run the same helper-level payload against the pre-fix base and this branch, then run `test_anthropic_direct_path_repairs_typeless_tool_search_regression` through `handle_anthropic_messages()` with 20 ordinary tools, one typeless `tool_search_tool_regex`, and a stale self-reference - Observed result: before the fix, Headroom injected a second typed search tool, deferred the typeless client tool, and removed 0 stale blocks; on this branch, it skips duplicate injection, preserves the client tools array, and removes the paired `server_tool_use` and `tool_search_tool_result` blocks before forwarding - Not tested: a live request against a paid Anthropic account; the production handler's outbound body is captured before the network boundary instead ## Runtime Rollout Safety - Rollout-managed feature(s): Anthropic server-side tool-search deferral (`HEADROOM_TOOL_SEARCH`) - Minimum rollout channel: standard CI; narrow corrective change to an existing default-on path - Stable/default behavior changed: yes; reserved typeless client search tools now suppress duplicate injection, and typed search mechanisms no longer satisfy deferred-tool references - Kill switch / disable path: set `HEADROOM_TOOL_SEARCH=0` to disable new injection; history repair remains unconditional so existing poisoned sessions can recover - Unsafe override required: no - Qualification impact: no new rollout surface or configuration; focused handler, helper, cache-control, and hook-order regressions cover the affected path - Rollback path: revert this PR; operators can set `HEADROOM_TOOL_SEARCH=0` while rolling back ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — proxy request transformation only. ## Additional Notes - Documentation is not changed because this fixes internal request classification and transcript repair without adding a user-facing option or workflow. - Anthropic documents `tool_search_tool_regex` / `tool_search_tool_bm25` as server search mechanisms; deferred definitions, rather than the search mechanism itself, are the valid `tool_reference` targets. - Rebased onto #2996, which fixes the unrelated native-detector hang that timed out shard 4 on the prior merge commit. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: JerrettDavis <2610199+JerrettDavis@users.noreply.github.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
JD Davis
|
8a1d38bc5d
|
fix(proxy): complete stateless Responses and buffered CCR lifecycle (#2997)
## Description
Consolidates the related OpenAI Responses ZDR/stateless continuation and
buffered CCR response-lifecycle corrections on current main. It
preserves client storage policy, makes Headroom-owned continuations
stateless across HTTP and WebSocket, and prevents buffered streaming
paths from committing a false HTTP 200 before the real upstream outcome
is known.
Closes #2675
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Preserves explicit and omitted Responses `store` policy instead of
forcing provider storage or disabling memory tools.
- Replays normalized input, replayable outputs, encrypted reasoning
content, and Headroom function outputs without `previous_response_id`.
- Applies the same stateless continuation policy to HTTP and WebSocket.
- Prevents transparent memory execution after client-visible WebSocket
output.
- Delays buffered CCR ASGI status/headers until the operation resolves
for Anthropic Messages and OpenAI Responses.
- Preserves real 429/5xx status and retry headers.
- Converts malformed non-JSON/non-SSE upstream 200 replies to a
sanitized 502 protocol error.
- Preserves valid JSON-to-SSE synthesis and existing SSE adaptation.
- Removes unreachable task cleanup left behind after replacing the old
keepalive polling loop with a direct awaited operation.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
118 passed across the changed HTTP/WS ZDR, lifecycle, and both-provider CCR suites
11526 tests collected with no collection errors
ruff check .: All checks passed
ruff format --check .: 1411 files already formatted
mypy headroom/proxy/handlers/anthropic.py headroom/proxy/handlers/openai.py:
Success: no issues found in 2 source files
```
Exact-head CI is entirely green on
`
|
||
|
JD Davis
|
a708c0571e
|
fix(ci): prevent native detector from hanging test shards (#2996)
## Description
CI shard 4 was not merely slow: after thousands of fast tests it parked
indefinitely inside `headroom._core.detect_content_type` at 0% CPU. The
router watchdogged only the first native call and then permanently
trusted direct calls via `_detect_native_verified`. Earlier suite
activity can change ORT/native state after that first success, making a
later call deadlock until GitHub cancels the job.
This keeps every native call bounded by the existing watchdog, activates
the process-wide pure-Python circuit breaker after a timeout, restores
the test-job ceiling to 30 minutes, and removes a separate wall-clock
scheduler assertion that generated false shard-1 failures despite the
structural regression guards passing.
No issue is auto-closed by this infrastructure repair.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Removed the unsafe process-lifetime `_detect_native_verified` fast
path.
- Kept every native detection call behind the existing bounded watchdog.
- Preserved the process-wide fallback circuit breaker so only the first
wedged call consumes the watchdog budget.
- Added a success-then-hang regression test.
- Isolated native circuit-breaker state in fallback exception tests.
- Restored the CI test timeout from the temporary 90-minute diagnostic
ceiling to 30 minutes.
- Replaced the Codex scheduler's noise-sensitive p99/p50 assertion with
its meaningful absolute regression ceiling while retaining source-level
guards against the removed semaphore and nested executor.
- Corrected import order and formatting defects inherited from current
main so the synthetic merge commit passes repository-wide lint.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
Exact local shard-4 command with coverage:
2723 passed, 172 skipped, 8661 deselected in 108.40s
Focused detector/router suite:
62 passed
Codex scheduler suite:
3 passed, 1 skipped
ruff check .
All checks passed!
ruff format --check .
1411 files already formatted
mypy headroom/transforms/content_router.py
Success: no issues found in 1 source file
```
Exact-head GitHub CI on `
|
||
|
Tejas Chopra
|
3145242645
|
Unify savings attribution across stats, perf, metrics, and dashboard (#2976)
## Summary Adds a small provider-neutral savings attribution seam. Named sources can attach realized or projected token/USD deltas to a request without changing headline arithmetic or introducing private-package inventory into OSS. Also fixes the Anthropic buffered lifecycle so normal successful responses run response hooks, applies stream-safety filtering, includes tool savings in per-model perf totals, and surfaces the same breakdown in request logs, `/stats`, `headroom perf`, Prometheus, OTEL, and the dashboard. ## Why Request-local savings were split between canonical token deltas, process-global extension counters, and tool-only tags. This made correct headline totals possible while losing attribution in perf, recent requests, metrics, and the dashboard. Normal Anthropic responses also skipped response hooks unless CCR ran. ## Validation - 74 focused tests passed: turn hooks, OpenAI hook lifecycle, outcome funnel, perf formats, and tool-search repair - Ruff passes on all changed Python files - Existing compression-observability suite: 11 passed; 2 tokenizer-cache tests require network access to fetch the tiktoken vocabulary ## Compatibility No named private packages or private inventory are encoded in OSS. Existing hooks remain source-compatible because all new TurnContext fields are optional. --------- Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> |
||
|
JD Davis
|
1aa701adaa
|
fix(vscode): persist compatible Claude modes and route Copilot CAPI (#2986)
## Description Fixes #2492, #2028, and #2827. Claude daemon workers consume project settings rather than reliably inheriting wrapper environment state, while the Claude VS Code webview cannot render deferred-tool response blocks. Separately, recent Copilot Chat versions use the whole CAPI override for generation; the legacy proxy override alone only sends model discovery through Headroom. This PR carries both integrations through to the actual consumers instead of only changing their launch-time surface configuration. ## Type of Change - [x] Bug fix - [ ] New feature - [ ] Breaking change - [ ] Documentation update - [ ] Build / CI ## Changes Made - Persist the resolved Claude ENABLE_TOOL_SEARCH value into project settings for daemon workers and restore it transactionally after wrap exits. - Use compatibility-safe Foundry and Claude VS Code defaults while preserving explicit user choices. - Configure both Copilot overrideProxyUrl and overrideCapiUrl in the reversible managed VS Code settings block. - Route Copilot unprefixed POST /chat/completions and HTTP /responses requests through the real compression handlers. - Keep /responses out of the Codex WebSocket aliases because Copilot and Codex use different WebSocket wire protocols. - Extend wrap E2E assertions for both the Claude webview mode and Copilot CAPI routing. ## Testing - [x] 127 combined Claude, Copilot, route-integration, and MCP dependency-contract tests pass. - [x] Ruff check passes on all changed Python files. - [x] Ruff format check passes. - [x] Python compilation and git diff --check pass. ## Runtime Safety Standalone Claude CLI defaults remain unchanged. Explicit Claude tool-search values retain precedence, and project settings are restored through the existing cleanup path. Copilot model/session helper endpoints continue through generic passthrough, while only validated HTTP generation paths receive explicit compression routes. Existing Codex WebSocket behavior is unchanged. ## Review Readiness - [x] Current main and MCP v1 compatibility retained - [x] Worker-facing Claude persistence covered - [x] Reversible Copilot and Claude settings behavior covered - [x] Copilot generation routes covered at registration and proxy integration layers - [x] Ready for review |
||
|
JD Davis
|
eafdf11a2c
|
fix(docker): ship Bedrock auth and current registry (#2982)
## Description Fixes #1551 and #1692. Every published Headroom Docker image now installs the existing `bedrock` extra, so `--backend bedrock` can authenticate with temporary STS, SSO, and credential-process credentials instead of failing because `botocore` is absent. Public Docker instructions now consistently use `ghcr.io/headroomlabs-ai/headroom`. Several still pointed at the old personal package, which is frozen at 0.27.0 and caused users to report that no latest image existed. ## Type of Change - [x] Bug fix - [ ] New feature - [ ] Breaking change - [x] Documentation update - [x] Build / CI ## Changes Made - Add `bedrock` to the standalone Dockerfile default extras. - Add `bedrock` to all nine root/code/slim/nonroot bake targets. - Replace obsolete personal GHCR references in README, llms.txt, Compose guidance, testing guidance, and wiki docs. - Add release contract tests for Bedrock dependencies and the current organization registry. ## Testing - [x] Focused Docker release and Bedrock preflight tests pass. - [x] Full updater suites pass: 69 tests. - [x] `uv run ruff check tests/test_release_workflows.py` - [x] `docker buildx bake --print` - [x] `git diff --check` ## Real Behavior Proof Before this change, every published bake target installed only `proxy` or `proxy,code`, so `AWS_SESSION_TOKEN` selected an unavailable botocore path. Public copy-paste commands also referenced `ghcr.io/chopratejas/headroom`, which the existing migration code and changelog identify as frozen at 0.27.0. After this change, all nine parsed bake targets install `bedrock`; the regression resolves that package extra and confirms `boto3` plus `botocore`. Every public Docker instruction covered by the contract names `ghcr.io/headroomlabs-ai/headroom`. ## Runtime Rollout Safety This changes image contents and documentation only; proxy routing and non-Docker installs are unchanged. Static AWS credentials remain unaffected. Existing manifests using the deprecated image continue to be migrated by the established install-state logic. Rollback is a Docker/bake extras and documentation revert. ## Review Readiness - [x] Two related Docker blockers batched in one PR - [x] Regression coverage included - [x] No unrelated lockfile changes - [x] Ready for review |
||
|
JD Davis
|
ddd2a259ec
|
fix(install): consolidate Windows fallback and cleanup safety (#2980)
## Description Consolidates two fully reviewed installation-safety fixes whose original PRs can no longer merge under current branch protection: Windows persistent-service deployments need a supported Task Scheduler fallback, and legacy context-tool cleanup must never delete user-owned RTK/lean-ctx artifacts. Closes #2552 Closes #2817 Supersedes #2600 and #2828 while preserving their authors' commits and review-driven corrections. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Convert Windows `persistent-service` plans to the supported `persistent-task` supervisor and make the fallback explicit in CLI output. - Restrict context-tool cleanup to artifacts proven to live under Headroom's managed directory. - Recognize wrapped, relative, and platform-specific managed commands without accepting prefixed/path-boundary lookalikes. - Scope cleanup completion state correctly across projects and alternate agent homes. - Stamp cleanup complete only after all managed remnants are settled. - Preserve the original focused regression suites and behavior-proof artifact. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_install/test_planner.py tests/test_install/test_supervisors.py tests/test_cli/test_install_cli.py tests/test_context_tool_cleanup.py tests/test_cli/test_unwrap_claude.py 135 passed in 0.45s $ uv run ruff check <changed Python and test files> All checks passed! $ uv run ruff format --check <changed Python and test files> 8 files already formatted ``` ## Real Behavior Proof - Environment: macOS arm64 for consolidated current-main validation; the Windows fallback source PR was independently validated on Windows and includes its captured verification artifact. - Exact command / steps: run the planner, supervisor, install CLI, cleanup provenance, and unwrap suites on the rebased combined branch. - Observed result: 135/135 focused tests pass. Windows service requests resolve to `persistent-task`; cleanup rejects user-owned and path-prefix lookalikes while removing managed artifacts. - Not tested: a fresh privileged Windows host deployment in this local pass; #2600's accepted review contains the Windows-specific proof. ## Runtime Rollout Safety - Rollout-managed feature(s): Install supervisor selection and one-time legacy cleanup. - Minimum rollout channel: Stable/default; both prevent currently destructive or nonfunctional install paths. - Stable/default behavior changed: Windows service requests use Task Scheduler; cleanup requires managed provenance. - Kill switch / disable path: Select `persistent-task` explicitly; cleanup remains bounded by its completion stamp and provenance checks. - Unsafe override required: No. - Qualification impact: Windows native install and wrap/unwrap cleanup suites. - Rollback path: Revert this PR, restoring the two pre-fix behaviors. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) The Windows verification artifact from #2600 is retained at `.github/pr-images/issue-2552-windows-fallback-verification.png`. ## Additional Notes This is intentionally an installation-safety batch rather than two replacement PRs. Original commit authorship is preserved, and the combined diff was applied cleanly to current `main` after #2832 and #1628 landed. --------- Co-authored-by: Inference1 <68734681+Inference1@users.noreply.github.com> Co-authored-by: Dennis Alexis Valin Dittrich <dd+github@dr-dittrich.de> |
||
|
JD Davis
|
a3fe5cb65b
|
fix(onnx): enforce Rust API-24 runtime compatibility (#2979)
## Description Rust fastembed enables ORT C API 24, but the Python dependency allowed ONNX Runtime 1.23.2. Entering ort's initializer with that library deadlocks permanently instead of returning an error. Align dependency resolution where compatible wheels exist and preflight native detection where they do not. Closes #2960 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Require ONNX Runtime 1.24+ for Python 3.11+ in the proxy and voice extras. - Keep the available pre-1.24 runtime on Python 3.10 for Python ONNX consumers. - Refuse to auto-pin an incompatible runtime into the Rust extension. - Bypass native detection immediately when API 24 is unavailable, preserving Python fallback without a five-second watchdog delay or stuck native thread. - Add dependency, pinning, override, and router regression coverage. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_transforms/test_ort_dylib.py tests/test_onnx_dependency_contract.py tests/test_onnx_runtime.py tests/test_transforms/test_content_router.py 88 passed in 9.31s $ uv run ruff check headroom/_ort.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py tests/test_onnx_dependency_contract.py All checks passed! ``` ## Real Behavior Proof - Environment: macOS arm64; Python 3.13.14 and uv-managed Python 3.10.20. - Exact command / steps: run the issue's direct `headroom._core.detect_content_type` call in a subprocess with a 12-second timeout on Python 3.13; run `_detect_content` on Python 3.10 after resolving the proxy extra. - Observed result: Python 3.13 resolves ORT 1.26.0 and native detection returns `json_array`; Python 3.10 resolves ORT 1.23.2, leaves `ORT_DYLIB_PATH` unset, reports compatibility false, and immediately returns the Python `json_array` fallback. - Not tested: Linux-specific shared-object execution locally; CI's existing Linux Rust job already preflights ORT 1.24+ and exercises native tests. ## Runtime Rollout Safety - Rollout-managed feature(s): Native Rust content detection. - Minimum rollout channel: Stable/default; this is a deadlock prevention guard. - Stable/default behavior changed: Python 3.11+ installs a compatible ORT; Python 3.10 skips incompatible native detection. - Kill switch / disable path: `HEADROOM_DETECT_BACKEND=python` remains available; an explicit `ORT_DYLIB_PATH` remains an operator override. - Unsafe override required: No. - Qualification impact: Native detection stays enabled only with API-24-compatible ORT. - Rollback path: Revert this PR, which restores the old watchdog-only degradation. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable. ## Additional Notes The large lockfile diff is dependency resolution: Python 3.10 keeps ORT 1.23.2 while 3.11+ resolves 1.26.0. The functional Python change is intentionally small and keeps explicit `ORT_DYLIB_PATH` overrides working. |
||
|
Abhay Singh
|
7de35739c6
|
fix(proxy/anthropic): repair headroom_retrieve history references the tools array cannot support (#2876)
## Description #2805 / #2807 established the mechanism: Claude Code replays one transcript across requests that carry different `tools` arrays, and Anthropic validates every history reference against the array of the request at hand. #2807 fixed it for tool-search blocks by repairing history (`strip_unsupported_tool_search_blocks`) rather than trying to predict the client's tool set. The same mechanism applies to CCR's `headroom_retrieve`, and it is tool-agnostic. A passthrough side-request (the prompt-type Stop hook evaluator, `/compact`) that the proxy forwards without declaring `headroom_retrieve` still carries a historical `tool_use` naming it, and Anthropic 400s on the dangling reference. The injection-side fixes (#2766 / #2533) decide *when to re-declare the tool*; this makes the 400 *structurally impossible* where the tool is intentionally absent. It is belt-and-braces with them, not a replacement. The fix adds the symmetric repair next to #2807's. When the outbound `tools` array does not declare `headroom_retrieve`, it replaces each `headroom_retrieve` `tool_use` and its paired `tool_result` with a text block, so no dangling reference survives. It **neutralizes** (replaces in place) rather than **drops**, which is the one deliberate difference from #2807: CCR's `tool_use` lives in an assistant turn and its `tool_result` in the next user turn, i.e. two different messages. Dropping a whole message could leave two same-role messages adjacent and break Anthropic's strict user/assistant alternation, turning one 400 into another. Replacing blocks in place keeps every message and role intact, and preserves the retrieved text the model already saw. #2807's server-tool blocks both live in the same assistant turn, so dropping was safe there. It runs after CCR tool injection, so on the main loop -- where the tool IS injected (a present marker) -- it neutralizes nothing and the prompt-cache prefix is untouched, mirroring #2807's placement and sequencing. Fixes #2814 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/helpers.py`: added `strip_unsupported_ccr_retrieve_blocks(messages, tools)` (and a small `_ccr_result_as_text` helper). No-ops (returning the original object by identity) when `headroom_retrieve` is declared or no such history exists; otherwise neutralizes the `tool_use` and its paired `tool_result` to text. - `headroom/proxy/handlers/anthropic.py`: call the repair right after the tool-search history repair (which is after CCR tool injection), guarded on it actually changing anything, tagged `router:ccr_retrieve_repair:Nblocks`. - `tests/test_ccr_retrieve_history_repair.py`: 5 unit tests (no-op when declared, no-op without retrieve history, neutralize + preserve result text + keep alternation, leave foreign tool_use untouched, placeholder when the result has no text). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text tests/test_ccr_retrieve_history_repair.py 5 passed # Broader CCR / tool-search / handler suites (unchanged behavior): tests/test_ccr_retrieve_history_repair.py tests/test_proxy/test_anthropic_ccr_deferred_injection.py tests/test_issue_746_tool_search.py 71 passed # uvx ruff@0.15.17 check -> All checks passed! # uvx mypy@1.20.2 headroom/proxy/helpers.py -> Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1, ruff 0.15.17 and mypy 1.20.2 via uvx. - Exact command / steps: confirmed the injection point (`apply_session_sticky_ccr_tool`) and the tool-search repair placement in `handlers/anthropic.py`, confirmed `body["tools"]` reflects the CCR injection before the repair call site (`body["tools"] = tools` is written well upstream and the adjacent tool-search repair already relies on it), then drove the helper over a transcript with a `headroom_retrieve` tool_use + paired tool_result: with the tool declared it returns the original object unchanged; with the tool absent it neutralizes both blocks, preserves the result text, and keeps the message roles/count identical. - Observed result: a forwarded request that would 400 with "Tool reference 'headroom_retrieve' not found in available tools" now carries text blocks in place of the retrieve `tool_use`/`tool_result`, so there is no reference for Anthropic to reject, and user/assistant alternation is preserved. The main loop (tool present) is a no-op. - Not tested: a live multi-turn Claude Code session hitting a Stop-hook/`/compact` side-request against a real provider (no live provider here). The repair is a pure function verified directly over the exact block shapes Anthropic validates, and it mirrors the already-merged tool-search repair's mechanism and wiring. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md`: it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes The issue reporter noted their own logs show the tool-search variant of this 400 (61 across 11 days) but zero `headroom_retrieve` occurrences, because they run `HEADROOM_LOSSLESS=1` which disables CCR entirely. This PR fixes the CCR variant of the same, proven, tool-agnostic mechanism rather than a fresh CCR repro. The neutralize-vs-drop choice is the one place I departed from #2807, for the alternation reason above; if you would rather it drop (accepting the alternation handling that implies), I am happy to switch it. --------- Co-authored-by: Jerrett Davis <mxjerrett@gmail.com> |
||
|
JD Davis
|
6077e5a149
|
fix(mcp): restore SDK v1 compatibility cap (#2978)
## Description PR #2963 widened the MCP dependency to 2.x while Headroom's live MCP server still uses the v1 low-level `Server.list_tools()` and `Server.call_tool()` decorators. Fresh installs therefore crash before serving tools. Restore the v1 cap until the explicit SDK 2.x port in #2658 lands, and pin that compatibility contract with a regression test. Closes #2977 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Restore `mcp>=1.28.1,<2.0.0` in the `proxy` and `mcp` extras. - Regenerate `uv.lock`, resolving MCP 1.28.1 and removing the incompatible 2.x transitive set. - Add a dependency-contract test covering both shipping extras. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_mcp_dependency_contract.py tests/test_ccr_mcp_server.py tests/test_cli/test_mcp.py 41 passed in 0.56s $ uv run ruff check tests/test_mcp_dependency_contract.py All checks passed! $ uv run ruff format --check tests/test_mcp_dependency_contract.py 1 file already formatted ``` ## Real Behavior Proof - Environment: macOS arm64, Python 3.13.14, uv-managed project environment. - Exact command / steps: resolve the `mcp` extra, inspect the installed SDK version and v1 decorators, then instantiate `HeadroomMCPServer(check_proxy=False)`. - Observed result: `1.28.1 True True`; server construction returns a v1 `Server` successfully. - Not tested: full stdio exchange against every external MCP client; existing MCP unit and CLI suites cover server setup and handlers. ## Runtime Rollout Safety - Rollout-managed feature(s): None; dependency resolution guard. - Minimum rollout channel: Stable/default. - Stable/default behavior changed: Fresh installs stop resolving the incompatible MCP SDK 2.x release. - Kill switch / disable path: Revert the dependency cap after #2658 lands. - Unsafe override required: No. - Qualification impact: MCP extras and proxy installs remain on the maintained MCP 1.x line. - Rollback path: Revert this PR; not recommended until the v2 server port is merged and tested. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable. ## Additional Notes The documentation change is the inline dependency rationale next to the cap. The long-term migration remains #2658; this PR deliberately does not mix that breaking SDK port into the release-blocker rollback. |
||
|
石岳峰
|
b7f342c153
|
fix(wrap): verify proxy deps before mutating Codex config (#1628)
## Description \`headroom wrap codex\` now verifies that optional proxy dependencies (\`headroom-ai[proxy]\`) are installed before mutating Codex \`config.toml\`. If the check fails, the command exits with the same error message as \`headroom proxy\` and leaves Codex config untouched. Fixes #1614 (Bug 1: config mutated before proxy dependency check). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Extract \`ensure_proxy_dependencies()\` in \`headroom/cli/proxy.py\` (shared with \`headroom proxy\`) - Call it at the start of \`wrap codex\` when \`not no_proxy\`, before config snapshot/injection - Add regression tests for prepare-only abort, \`--no-proxy\` skip, and import failure messaging ## Testing - [x] Unit tests pass (\`pytest\`) - [x] Linting passes (\`ruff check .\`) - [ ] Type checking passes (\`mypy headroom\`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output \`\`\`bash pytest tests/test_cli/test_wrap_codex.py::test_wrap_codex_aborts_before_mutating_config_when_proxy_deps_missing \ tests/test_cli/test_wrap_codex.py::test_wrap_codex_skips_proxy_dependency_check_with_no_proxy \ tests/test_cli/test_wrap_codex.py::test_ensure_proxy_dependencies_exits_when_server_import_fails -q # 3 passed ruff check headroom/cli/wrap.py headroom/cli/proxy.py tests/test_cli/test_wrap_codex.py ruff format --check headroom/cli/wrap.py headroom/cli/proxy.py tests/test_cli/test_wrap_codex.py \`\`\` ## Real Behavior Proof Environment: Linux (Ubuntu), Python 3.12, local checkout with \`PYTHONPATH\` pointed at patched sources. Exact command / steps: 1. Created a temp \`~/.codex/config.toml\` with \`model_provider = "openai"\`. 2. Patched \`headroom.cli.wrap.ensure_proxy_dependencies\` to raise \`SystemExit(1)\` (simulating missing \`[proxy]\` extra). 3. Ran \`headroom wrap codex --prepare-only --no-serena --port 8787\`. Observed result: exit code 1; \`config.toml\` unchanged; no \`config.toml.headroom-backup\` created; no \`[mcp_servers.headroom]\` block written. Also verified: \`headroom wrap codex --prepare-only --no-proxy ...\` does not invoke the dependency check. Not tested: Windows-specific proxy selector behavior (covered separately in #1655). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did not edit CHANGELOG.md; release notes are generated automatically --------- Co-authored-by: syf2211 <syf2211@users.noreply.github.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net> |
||
|
Ben Younes
|
9fde127534
|
fix(proxy): relocate stray system-role messages to the top-level system param (#765) (#1357)
## Description On requests large enough to trigger compression, the proxy emitted an upstream Anthropic request whose `messages[0]` had `role: "system"`. Anthropic's Messages API rejects any `system` role inside `messages[]`: ``` 400 invalid_request_error: "messages.0: use the top-level 'system' parameter for the initial system prompt" ``` The original request correctly carries its system prompt in the top-level `system` parameter; a compression/transform/pipeline step relocates the harness system block into `messages[0]`, so the request fails outright (intermittent only because it requires a context large enough to compress). This adds a wire-contract guard in the Anthropic forwarder: as the **last** step before sending upstream (after every transform, memory injection, tool sort, and pipeline extension, covering both the Bedrock and direct paths), any stray `role="system"` message is relocated out of `messages[]` and merged back into the top-level `system` parameter. Content order is preserved (existing system first, relocated content after) and block-level `cache_control` survives. The guard is a no-op on the common path (no system-role entry → inputs pass through unchanged). Closes #765 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/helpers.py`: new pure helper `relocate_system_messages_to_top_level(messages, system) -> (clean_messages, new_system, changed)` plus `_system_message_to_blocks`. Handles `system` being `None`/`str`/`list`, never drops content, preserves order and content blocks. - `headroom/proxy/handlers/anthropic.py`: invoke the guard just before the byte-faithful forward block; on relocation, update `body["messages"]`/`body["system"]`, mark the body mutated (`system_role_relocated`) so the byte-faithful forwarder re-serializes, and log a warning. - `tests/test_proxy_handler_helpers.py`: 3 unit tests (relocate stray system into top-level, append-to-existing-system order, no-op without a system entry). - `CHANGELOG.md`: Bug Fixes entry. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --extra dev python -m pytest tests/test_proxy_handler_helpers.py -q 29 passed in 4.95s # Regression on the forward path (byte-faithful forwarding, system-prompt immutability, cache stability): $ uv run --extra dev python -m pytest tests/test_proxy_handler_helpers.py tests/test_proxy_byte_faithful_forwarding.py tests/test_proxy_system_prompt_immutable.py tests/test_proxy_anthropic_cache_stability.py -q 90 passed, 15 warnings in 29.72s $ uv run ruff check headroom/proxy/helpers.py headroom/proxy/handlers/anthropic.py tests/test_proxy_handler_helpers.py All checks passed! $ uv run ruff format --check ... # 3 files already formatted $ uv run mypy headroom/proxy/helpers.py headroom/proxy/handlers/anthropic.py Success: no issues found in 2 source files ``` ## Test verification (RED → GREEN) The new tests exercise the guard directly and import the new helper at module top, so reverting the production fix makes them fail at collection. **RED — production fix reverted (helper removed):** ```text ImportError while importing test module 'tests/test_proxy_handler_helpers.py'. E ImportError: cannot import name 'relocate_system_messages_to_top_level' from 'headroom.proxy.helpers' =========================== 1 error in 0.41s =============================== ``` **GREEN — production fix applied:** ```text tests/test_proxy_handler_helpers.py ... [100%] ======================= 3 passed, 26 deselected in 1.50s ======================= ``` ## Real Behavior Proof - Environment: Python 3.13, `uv run` in this repo, branch `fix/issue-765`. - Exact command / steps: ran the guard on a body in the exact #765 failure shape — `system: None` and a `role="system"` harness block at `messages[0]`: - Observed result: ```text BEFORE: messages[0].role = system (Anthropic 400 trigger) changed = True AFTER roles = ['user', 'assistant'] system param = [{"type": "text", "text": "You are Claude Code. <system-reminder>...</system-reminder>"}] OK: no role=system in messages[]; system content preserved in top-level param ``` The illegal `role="system"` entry is removed from `messages[]` and its content lands in the top-level `system` parameter — exactly the body Anthropic accepts. - Not tested: a full live 250k+-token Claude Code session against the real Anthropic API (needs a large live context + API key); the fix is validated at the request-shaping boundary the 400 is raised on. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes The guard intentionally fires at the forwarder boundary rather than in any single transform: the issue's captures show the relocation can originate from the compression path, and pipeline extensions / hooks can also mutate `messages` late. Enforcing Anthropic's wire contract once, at the point the body is serialized upstream, fixes the 400 regardless of which step introduced the stray entry and matches the architecture invariant "never produce a `system`-role entry within `messages[]`". --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Sergei Baikin
|
6576ef639c
|
fix(openclaw-plugin): circuit breaker + per-request timeout for proxy resilience (#639)
## Description This change adds bounded timeout and circuit-breaker behavior so OpenClaw can degrade safely when Headroom or the upstream stream stalls, while returning structured proxy errors instead of hanging. Closes #638 by improving OpenClaw/proxy resilience when the Headroom proxy stalls or Anthropic resets a stream. The PR adds proxy-side handling for `httpx.RemoteProtocolError`, returns structured 502 responses for otherwise unhandled proxy middleware errors, and adds OpenClaw plugin timeout/circuit-breaker fallback behavior. ## Type of Change - [x] Bug fix - [ ] New feature - [x] Documentation - [ ] Refactor - [x] Tests only ## Changes Made - Added OpenClaw plugin per-request compression timeout and circuit breaker fallback. - Cleared timeout timers after successful or failed compression so successful calls do not leave pending timers. - Added a focused Vitest regression for timeout cleanup. - Added `contracts.tools` for `headroom_retrieve` without whole-file manifest reformatting. - Added proxy handling for mid-stream `httpx.RemoteProtocolError` and structured 502 fallback behavior. - Documented the new OpenClaw resilience configuration fields. ## Testing - [x] Unit tests - [x] Integration-style proxy tests - [x] Typecheck/build - [ ] Manual testing ### Test Output ```text cd plugins/openclaw && npm test Test Files 6 passed (6), Tests 55 passed (55) cd plugins/openclaw && npm run typecheck passed cd plugins/openclaw && npm run build Build success UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --with pytest --with pytest-asyncio --with fastapi --with httpx --with uvicorn --with h2 python -m pytest tests/test_proxy_streaming_resilience.py -q 24 passed in 2.16s ``` ## Real Behavior Proof - Environment: Windows 11, Node/npm from local plugin worktree, Python 3.13.3, focused local worktree for PR #639. - Exact command / steps: Installed plugin dependencies, ran OpenClaw plugin tests/typecheck/build, and ran the proxy streaming resilience suite with required async/FastAPI/httpx extras. - Observed result: Plugin tests, typecheck, build, and proxy resilience tests all passed. - Not tested: Live OpenClaw gateway session in this pass; original reporter previously verified patched files in a container and OpenClaw degraded/recovered cleanly. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: Sergei Baikin <sergei.baikin@fotograf.de> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net> |
||
|
Abhay Singh
|
82526191a1
|
fix(cli/install): resolve the deployment profile instead of dead-ending on default (#2832)
## Description `headroom init` installs its persistent deployment under a non-`default` profile name (`init-user` for a global-scope install), but every `headroom install <lifecycle>` subcommand hardcodes `--profile default`. The docs show those commands without `--profile`, so on a machine set up by `headroom init` every documented lifecycle command fails while the real deployment is running fine: ```console $ headroom install status Error: No deployment profile named 'default' is installed. $ headroom install status --profile init-user Status: running Healthy: yes ``` The error named neither the installed profile nor the `--profile` flag, so there was nothing to lead the user to `init-user`, which exists only as an internal constant. When the requested profile is not installed, `_require_manifest` now resolves the real target instead of dead-ending on a name the user never chose: 1. an explicit `HEADROOM_DEPLOYMENT_PROFILE` (which the runtime already exports) wins; 2. otherwise, when `--profile` was left at its `default` default and exactly one deployment is installed, that one is used; 3. when it still cannot decide, the error lists the installed profiles and points at `--profile`. This changes only the not-found path. An installed `default` still loads exactly as before, and an explicit typo'd `--profile` still fails, now with a helpful list. Fixes #2811 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cli/install.py` (`_require_manifest`): on a manifest miss, resolve via `HEADROOM_DEPLOYMENT_PROFILE`, then a single installed deployment when the request is the bare `default`, and otherwise raise an error that lists installed profiles and points at `--profile`. Imported `list_manifests` (already present in `headroom.install.state`) for the enumeration. - `tests/test_cli/test_install_cli.py`: added `test_require_manifest_resolves_single_profile_when_default_missing`, `test_require_manifest_honors_env_profile`, and `test_require_manifest_lists_installed_profiles_when_ambiguous`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # Before/after on the exact reported scenario (one installed profile "init-user"): # ORIGINAL: _require_manifest("default") -> raises "No deployment profile named 'default' is installed." # FIXED: _require_manifest("default") -> resolves to "init-user" # Pass-after, install suites: tests/test_cli/test_install_cli.py 33 passed tests/test_install/ 174 passed, 1 skipped, 1 pre-existing failure # the 1 failure is tests/test_install/test_native_installers.py:: # test_powershell_native_installer_supports_persistent_docker_lifecycle, which # runs scripts/install.ps1 and fails identically on clean main with these changes # stashed (an environment-specific PowerShell exit, unrelated to this diff). # uvx ruff@0.15.17 check -> All checks passed! # uvx mypy@1.20.2 headroom/cli/install.py -> Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1, ruff 0.15.17 and mypy 1.20.2 via uvx. - Exact command / steps: read `detect`/`_require_manifest` and the lifecycle command options (`--profile default` at install.py:720/748/763/775/789/818/830) to confirm the mismatch with `init.py`'s `_GLOBAL_PROFILE = "init-user"`, then demonstrated before/after by monkeypatching `load_manifest`/`list_manifests`: on the original code `_require_manifest("default")` raises "No deployment profile named 'default' is installed."; with the fix it returns the single installed manifest (`init-user`). Fail-before via `git stash push headroom/cli/install.py` and a direct call; pass-after with `git stash pop` and the install suites (33 passed in the CLI file, 174 passed in test_install with one pre-existing environment failure). - Observed result: a bare lifecycle command on an init'd machine now targets the running deployment instead of failing, matching the `--profile init-user` command the issue reporter confirmed works. An explicit `HEADROOM_DEPLOYMENT_PROFILE` selects the target, and an ambiguous multi-profile machine gets an error naming the installed profiles and the `--profile` flag. - Not tested: a full end-to-end `headroom init` then `headroom install status` on a fresh host (that flow spawns a real deployment and supervisor). The resolution logic is a pure function verified directly, and the manifest loading it calls is existing, tested code. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md`: it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes The resolution deliberately only triggers on the not-found path and only auto-selects when a single deployment is installed or an explicit env profile names one, so it never silently picks the wrong deployment on a multi-profile host. The docs that show the bare commands (`docs/content/docs/persistent-installs.mdx`, `wiki/persistent-installs.md`, `wiki/cli.md`) become correct again without needing a `--profile` on every line. |
||
|
Parideboy
|
f1c34d336c
|
fix(proxy/anthropic): don't buffer a CCR stream when passthrough discards the stream flip (#2953)
## Description
Fixes #2952. Since `
|
||
|
Parideboy
|
2d1e96b85c
|
fix(memory): sanitize entity_refs to prevent dict-shaped entries crashing search (#2951)
## Description Closes #2947 `entity_refs` is annotated `list[str]` everywhere, but nothing enforced that at runtime. `LocalBackend.save_memory`'s `entities` argument is filled straight from LLM-supplied `memory_save` tool input (`headroom/memory/system.py:575` into `memory_handler.py:1242`), so a caller can pass the typed `{"entity": ..., "entity_type": ...}` shape, which is the format `extracted_entities` expects, into it by mistake. Those dicts were then persisted verbatim into `entity_refs`, both in the `memories` table and in the duplicated copy the vector index keeps for post-filtering. Every later `search_memories` call does `set().update(memory.entity_refs)` while collecting entities for graph expansion. Hashing a dict raises `TypeError: unhashable type: 'dict'`, and because that happens inside the vector-result loop rather than per-item, **one** poisoned row aborted the **entire** search. The proxy's memory handler catches the exception and returns no memories, so recall went quietly dark rather than failing loudly, and the bad row kept re-appearing in top-k for related queries, so it stayed dark. The issue reporter hit this in production: 4 bad rows disabled memory search for a whole project for a day, with nothing visible to the end user beyond a swallowed warning in `proxy.log`. The same root cause has two more crash modes, both confirmed below: `AttributeError: 'dict' object has no attribute 'lower'` during graph linking on the save path, and the same error in the `entities` search filter (`ref.lower()`). The fix adds one helper and applies it at both ends of the data flow. Dicts are **unwrapped to their `entity` name** rather than dropped, so rows that are already corrupted keep contributing to graph expansion instead of silently losing their entities. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **New helper `normalize_entity_refs()` in `headroom/memory/models.py`.** Coerces a raw entity-reference list into the `list[str]` it claims to be: strings pass through, dicts are unwrapped via their `entity` (or `name`) key, and anything with no recoverable name is dropped rather than stringified, since a ref like `"{'entity_type': 'project'}"` would only pollute the graph. Order is preserved and duplicate names are collapsed. - **Write path, to stop new corruption at the door.** `LocalBackend.save_memory` normalizes `entities` before it reaches `entity_refs` and graph linking. `LocalBackend.search_memories` normalizes the `entities` *filter* argument too, since it arrives from the same untrusted tool input (`memory_handler.py:1320`). - **Read path, to heal rows that were written before this fix.** Applied at the three deserialization boundaries, so no data migration is needed and corrupted rows normalize themselves the next time they are loaded: `Memory.from_dict` (`headroom/memory/models.py`), `SQLiteMemoryStore._row_to_memory` (`headroom/memory/adapters/sqlite.py`), and the vector indexes' own `entity_refs` copies used for post-filtering, `VectorMetadata.from_json` (`headroom/memory/adapters/sqlite_vector.py`) and `IndexedMemoryMetadata.from_dict` (`headroom/memory/adapters/hnsw.py`). - **Defensive normalization on emitted results.** `search_memories` and `text_search` normalize the refs they return as `related_entities`, so a backend that produces `Memory` objects by some path not covered above still cannot take a whole query down, and callers never receive a dict where they expect an entity name. **Note on scope versus the patch proposed in the issue.** The issue proposed normalizing in two places (`save_memory` plus the `set().update()` line). I widened it slightly because that pair leaves three related failures live: the `entities` filter still crashes on `ref.lower()`, `related_entities` still hands dicts back to the caller, and, most importantly, already-poisoned rows stay poisoned in storage. Normalizing at the deserialization boundaries fixes all three at once and is what makes existing corrupted databases recover on their own. **Behavior change worth flagging.** `entity_refs` is now de-duplicated (case-sensitively) on both save and load. Refs were already treated as a set for graph expansion, so this is semantically a no-op, but it is a visible difference if anything asserts on exact list contents. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed New file `tests/test_memory/test_entity_ref_sanitization.py` adds 10 tests covering the helper, both write paths, all three deserialization boundaries, and the three crash modes. ### Test Output ```text $ python -m pytest tests/test_memory/test_entity_ref_sanitization.py -q .......... [100%] 10 passed, 17 warnings in 0.34s ``` Full memory suite, plus a before/after comparison of the failure set to prove no regressions: ```text $ python -m pytest tests/test_memory/ -q 13 failed, 576 passed, 3 skipped, 1072 warnings, 25 errors in 44.74s # the same run with the source changes stashed (baseline on upstream/main @ |
||
|
Parideboy
|
6147883d5e
|
fix(wrap): stop the Serena pre-index stalling the launch path for 300s (#2945)
## Description
`headroom wrap <agent>` could sit silently for a full 300 seconds before
the agent launched, and leaked one orphaned process every time it did.
`_setup_serena_mcp` runs `serena project index` synchronously on the
launch path, with `capture_output=True`, an inherited stdin and
`timeout=300`. When a project has no `.serena/project.yml`, Serena
auto-creates one — and that auto-creation asks one `[y/N]` question per
additionally-detected language server. Three things then combine:
1. stdin was inherited, so Serena believed it could prompt.
2. stdout was captured, so the question never reached the terminal.
3. the call was synchronous, so the agent waited out the entire timeout.
The user saw no prompt, no progress and no error — only a wrapper that
appeared to hang. The pre-index could never succeed in that state, so
the 300 seconds bought nothing.
On top of that, `subprocess.run` kills only its direct child on timeout.
`uvx` is a launcher that execs the real `serena` executable as a
grandchild, which was never signalled: it reparented to PID 1 and
survived indefinitely. Same class of bug as #615 and #880.
Closes #2938
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `_serena_project_skip_reason` (`headroom/cli/wrap.py`) now returns a
skip reason when `.serena/project.yml` is absent, so the pre-index does
not run in the one state where it cannot succeed.
- `_index_serena_project` passes `stdin=subprocess.DEVNULL`, so a
subprocess that decides to prompt gets EOF and exits in about a second
instead of blocking behind a captured pipe. This is deliberately kept as
a second line of defence even though the skip above already avoids the
known prompt.
- `_index_serena_project` now spawns via `subprocess.Popen` in its own
process group (`start_new_session=True` on POSIX,
`CREATE_NEW_PROCESS_GROUP` on Windows) instead of `run(...)`, so the
whole tree can be signalled.
- New `_kill_serena_index_tree` helper kills that tree on timeout —
`killpg(..., SIGKILL)` on POSIX, `taskkill /F /T /PID` on Windows — then
reaps the child and closes the capture pipes. Best-effort throughout; it
never raises.
- Corrected two comments that asserted the opposite of the observed
behaviour ("a failure or timeout here never blocks the wrap", "neither
blocks the wrap"). Both were accurate about intent and wrong about
effect.
- Added `_SERENA_INDEX_TIMEOUT` (still 300) and a line announcing the
pre-index, so a legitimately long index no longer looks like a hang.
- Tests in `tests/test_cli/test_wrap_serena_boost.py` rewritten for the
`Popen` path and extended to cover the DEVNULL stdin, the process-group
flag, the timeout tree-kill, the new skip reason, and the
`_setup_serena_mcp` wiring on both a fresh project and one that already
has `project.yml`.
### Behaviour change worth a reviewer's attention
**On a project with no `.serena/project.yml`, the pre-index no longer
runs at all.** That is the first wrap of any project, so this is the
common case.
I went this way rather than fixing the prompt because there is no way to
fix it from Headroom's side without re-introducing something the project
deliberately removed. Serena's `project index` command has no
non-interactive switch: `ProjectCommands._create_project` calls
`ProjectConfig.autogenerate(..., interactive=True)` with `interactive`
hardcoded. The only path that skips the prompt is passing
`--ls/--language` explicitly, which means Headroom guessing the
project's languages again — exactly the hand-maintained
extension-to-language map that was removed in #2674, with a comment in
this same function explaining why Serena should own that job.
The cost of skipping is small and self-correcting. Serena's MCP server
(`serena start-mcp-server --project-from-cwd`) generates `project.yml`
itself, non-interactively, on first start, and indexes lazily on demand
— which is the fallback the existing docstring already relied on. So the
first wrap now launches immediately with lazy indexing, and every wrap
after that pre-indexes for real. Previously the first wrap cost 300
seconds *and* still produced no index, so nothing of value is lost.
Happy to switch to passing `--ls` instead if maintainers would rather
keep the pre-index on the first wrap and accept a language map; the
other two changes stand either way.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_cli/test_wrap_serena_boost.py tests/test_cli/test_serena_migrate.py -q
collected 29 items
tests\test_cli\test_wrap_serena_boost.py .............s........... [ 86%]
tests\test_cli\test_serena_migrate.py .... [100%]
======================== 28 passed, 1 skipped in 0.54s ========================
$ python -m ruff check headroom/cli/wrap.py tests/test_cli/test_wrap_serena_boost.py
All checks passed!
$ python -m ruff format --check headroom/cli/wrap.py tests/test_cli/test_wrap_serena_boost.py
2 files already formatted
$ python -m mypy headroom/cli/wrap.py --ignore-missing-imports --python-version 3.13 --follow-imports=silent
Success: no issues found in 1 source file
```
The single skip is `test_kill_tree_signals_the_group_on_posix`, which is
platform-gated; the Windows counterpart ran. I develop on Windows, so
the POSIX `killpg` branch is covered by unit test only — the end-to-end
tree-kill proof below is the Windows `taskkill` branch.
## Real Behavior Proof
- Environment: Windows 11 Pro 26200, Python 3.13.11, headroom checkout
at
|
||
|
Ashish Patel
|
41dab2d099
|
fix(ccr): verify a scanned marker's hash before advertising it (#2908)
## Description
`CCRToolInjector.scan_for_markers()` decides whether a compression
marker is Headroom's own by *shape* alone — any bracket marker carrying
a 24-hex hash counts, per the generic fallback pattern
(`\[.*?compressed.*?hash=([a-f0-9]{24})\]`). Other context tools emit
exactly that shape. Once a foreign hash is scanned,
`has_compressed_content` flips true and the retrieve tool + "Available
hashes" system instruction get injected for a hash this proxy never
stored — the model calls `headroom_retrieve`, gets a guaranteed miss,
and re-does work it already had. Two wasted turns per adopted foreign
hash.
Closes #2836
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/ccr/tool_injection.py`: added
`CCRToolInjector.verify_ownership()` — filters `detected_hashes` down to
hashes the compression store actually recognizes, via the same
`store.exists()` check the retrieve endpoint itself performs. Added a
small `_HashOwnershipStore` Protocol (structural typing, not a hard
dependency on the concrete `CompressionStore` class) and a
`compression_store` constructor field for dependency injection/testing.
`scan_for_markers()` itself is untouched — kept store-independent (pure
regex) rather than baking the check into the scan loop, since that
approach broke 24 existing tests that correctly test "does this shape
match" in isolation.
- `headroom/proxy/handlers/anthropic.py`,
`headroom/proxy/handlers/openai.py`: call `injector.verify_ownership()`
right after `scan_for_markers()` — the two real per-request call sites.
- `verify_ownership()` is also called inside `process_request()` (the
convenience wrapper `batch.py`'s Google path uses), so that path is
covered without a separate call site edit.
- `tests/test_ccr_tool_injection.py`: new `TestVerifyOwnership` class (6
tests) — the exact issue repro, a real-hash-survives case, mixed
own/foreign hashes, explicit store override, store-exception safety
(must not raise), and no-op-on-empty-hashes.
- `tests/test_proxy_anthropic_cache_stability.py`: 3 pre-existing tests
needed updating for the new (correct) behavior — two `_FakeInjector`
test doubles needed a `verify_ownership()` stub added, and one real
end-to-end test needed a genuine store entry seeded (via
`explicit_hash`) for the hash its hand-typed marker references, instead
of asserting on an unverified shape-only match.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) — not run locally, will
confirm via CI
- [x] New tests added for new functionality
- [x] Manual testing performed (see Real Behavior Proof)
### Test Output
```text
$ .venv/Scripts/python -m pytest tests/test_ccr_marker_policy.py tests/test_ccr_tool_always_on.py tests/test_ccr_tool_injection.py tests/test_proxy/test_ccr_frozen_prefix_coupling.py tests/test_proxy_anthropic_cache_stability.py tests/test_proxy_handlers_batch.py tests/test_proxy_handler_helpers.py -q
151 passed in 15.87s
$ .venv/Scripts/python -m pytest tests/test_proxy_ccr.py tests/test_proxy_openai_responses_stream_ccr.py tests/test_anthropic_ccr_workspace_unbound.py tests/test_compression_store.py tests/test_no_ccr_lossy.py tests/test_proxy_handlers_batch.py -q
121 passed in 20.73s
$ .venv/Scripts/ruff check . && .venv/Scripts/ruff format --check . # touched files only
All checks passed / already formatted
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.14.5, local venv
- Exact command / steps: ran the issue's exact 3-line repro
(`CCRToolInjector.scan_for_markers()` on the foreign marker text, then
`verify_ownership()`) before and after the fix; separately verified a
genuinely-Headroom-stored hash (via `store.store(...,
explicit_hash=...)`) still survives verification and still drives
injection
- Observed result: before the fix (scan only, no verify step exists yet)
`has_compressed_content` is `True` for the foreign marker — matches the
bug report exactly. After adding `verify_ownership()`: foreign marker →
`detected_hashes == []`, `has_compressed_content is False`; real stored
hash → `detected_hashes == [real_hash]`, `has_compressed_content is
True`.
- Not tested: have not driven this through a live two-context-tool proxy
session (e.g. Headroom alongside another CCR-shaped tool in the same
conversation) — verified at the unit/integration level (the exact repro
plus the real proxy handler call sites via
`test_proxy_anthropic_cache_stability.py`'s end-to-end `TestClient`
tests), not via a live multi-tool session.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A —
internal CCR safety behavior, no user-facing docs reference the
marker-adoption mechanism)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable (release-please
generates this automatically from commit messages)
## Additional Notes
Design note on why `verify_ownership()` is a separate step rather than
baked into `scan_for_markers()`: my first attempt did exactly that and
broke 24 tests across `test_ccr_tool_injection.py`,
`test_ccr_marker_policy.py`, `test_proxy_anthropic_cache_stability.py`,
and `test_proxy_handler_helpers.py` — all of them legitimately testing
"does the regex detect this marker shape" independent of any store
state. Keeping the scan pure and adding an explicit, separately-testable
verification step kept that test surface intact while still closing the
real gap at the three places that actually decide whether to advertise
the retrieve tool.
|
||
|
Pragadeesh
|
d76fce04a3
|
fix(proxy): adapt 200 SSE upstream replies on buffered /v1/responses instead of 502 (#2622)
## Description The buffered HTTP `/v1/responses` path (`_buffered_ccr_operation` in `headroom/proxy/handlers/openai.py`) assumed every upstream reply to a `stream: false` request is JSON. Some OpenAI-compatible upstreams answer with a valid `200 text/event-stream` body carrying Responses API events. `response.json()` raised `JSONDecodeError`, which is not in the narrow usage-extraction catch (`KeyError, TypeError, AttributeError`), so it escaped to the outer handler and the **successful** upstream reply was converted into a generic `502 proxy_error` — the client loses the response and typically retries, duplicating paid calls. The fix classifies the upstream reply at the ingestion boundary by its declared `Content-Type` (the SSE spec's own discriminator) instead of parsing by expectation: - **200 SSE with a terminal `response.completed` event** → the complete response object is reassembled from that event (`_openai_responses_from_sse`, the inverse of the existing `_openai_responses_to_sse`) and swapped in as a synthesized `application/json` response *before any parsing happens*. Everything downstream — usage extraction, CCR retrieval handling, memory-tool handling — runs unmodified. - **200 SSE without a recognizable terminal event** → the successful upstream body is forwarded to the client unchanged (sanitized headers) rather than fabricating a 502. Adapt only when the adaptation is provably faithful; otherwise pass through. - **Everything else** (normal JSON replies, non-200s) → byte-identical pre-existing behavior. Deliberately *not* done: widening the `except` clause (would leave `resp_json` unbound and break the downstream pipeline) and body sniffing (the declared media type is trusted; a mislabeled body keeps today's behavior). Closes #2613 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/handlers/openai.py`: new module-level helper `_openai_responses_from_sse()` (SSE-spec framing: blank-line event separation, multi-line `data:` joining, `\r` tolerance, at most one stripped space, `[DONE]` skipped; returns the terminal event's `response` object or `None`), placed next to its inverse `_openai_responses_to_sse()`. - `headroom/proxy/handlers/openai.py::_buffered_ccr_operation()`: content-type dispatch for 200 replies immediately after the upstream response (and after wire-debug capture, so debug logs keep the true upstream bytes) — adapt SSE→JSON when a terminal event exists, pass through unchanged when it doesn't. - `tests/test_openai_codex_routing.py`: two new handler-level tests (see below). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # Both new tests watched failing BEFORE the fix with the exact issue signature: # ERROR headroom.proxy:openai.py [req-1] OpenAI responses request failed: JSONDecodeError: Expecting value: line 1 column 1 (char 0) # assert 502 == 200 $ pytest tests/test_openai_codex_routing.py -q 24 passed in 2.08s $ pytest tests/test_openai_codex_routing.py tests/test_ccr_response_handler_openai_responses.py tests/test_codex_responses_passthrough_bytes.py -q 38 passed, 1 warning in 13.80s $ pytest tests/test_output_shaper_responses.py tests/test_codex_responses_waste_signals.py tests/test_codex_openai_contract_parity.py tests/test_openai_beta_session_sticky.py tests/test_openai_max_completion_tokens.py tests/test_openai_response_cache_key.py tests/test_litellm_openai_passthrough.py -q 61 passed, 1 warning $ ruff check headroom/proxy/handlers/openai.py tests/test_openai_codex_routing.py All checks passed! $ mypy headroom/proxy/handlers/openai.py Success: no issues found in 1 source file ``` New tests: - `test_handle_openai_responses_non_stream_adapts_sse_upstream` — 200 SSE with `response.completed` → client gets 200 `application/json` with the reassembled response. - `test_handle_openai_responses_non_stream_passes_through_unparseable_sse` — 200 SSE with no terminal event → client gets 200 with the body unchanged, never a 502. ## Real Behavior Proof - Environment: macOS 26.5 (arm64), Python 3.12 via `uv`, headroom from source (editable install). Local fake OpenAI-compatible upstream (`http.server`) that answers every `POST` with `200 text/event-stream` containing a `response.completed` event and `data: [DONE]` — the upstream behavior reported in the issue. Proxy started with `OPENAI_TARGET_API_URL=http://127.0.0.1:9302 headroom proxy --port <port>`. - Exact command / steps: same-session A/B against real proxy processes — identical upstream and identical request, only the checked-out revision changed: ```bash curl -s -w "\nHTTP_STATUS=%{http_code} CONTENT_TYPE=%{content_type}\n" \ -X POST http://127.0.0.1:<port>/v1/responses \ -H "content-type: application/json" -H "authorization: Bearer sk-test" \ -d '{"model":"gpt-5.4","stream":false,"input":"hello"}' ``` - Observed result: unpatched `main` converts the successful upstream reply into the issue's 502; this branch returns the complete response as JSON. Full captures: **Before (unpatched `main`, port 8794):** ``` {"error":{"message":"An error occurred while processing your request. Please try again.","type":"server_error","code":"proxy_error"}} HTTP_STATUS=502 ``` **After (this branch, port 8795):** ``` {"id": "resp_sse_repro", "object": "response", "status": "completed", "model": "gpt-5.4", "output": [{"type": "message", "id": "msg_1", "role": "assistant", "content": [{"type": "output_text", "text": "hello from sse upstream"}]}], "usage": {"input_tokens": 2, "output_tokens": 1}} HTTP_STATUS=200 CONTENT_TYPE=application/json ``` - Not tested: a wild third-party SSE-answering upstream (the repro uses a local stub shaped per the issue report); the buffered-stream-CCR variant of this path against a live upstream (unit-tested only); Windows. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes - Documentation checklist item is N/A — internal proxy behavior fix, no documented surface changes. - Known residual (pre-existing, out of this issue's scope): a **non-200** upstream reply with a non-JSON body (an SSE error stream, a gateway HTML error page) still follows the old `JSONDecodeError → 502` path, blurring a meaningful upstream error into a generic 502. This PR deliberately adapts only declared-SSE **200** replies, where reassembly from `response.completed` is provably faithful. Happy to file the non-200 case as a follow-up issue if maintainers want it tracked. |
||
|
Abhay Singh
|
d6d121e399
|
fix(ccr): re-inject headroom_retrieve when history references it on the sessionless path (#2440) (#2533)
## Description Fixes #2440. `apply_session_sticky_ccr_tool` bypasses the `SessionCcrTracker` when `session_id` is `None` (WS / pre-session paths) and drives injection purely off the per-turn `has_compressed_content_this_turn` flag: ```python if not session_id: if not has_compressed_content_this_turn: ... # skip: tool NOT re-declared return tools_out, False ... ``` If an earlier turn emitted a `headroom_retrieve` tool_use into history but the current turn produced no fresh compression marker, the tool definition is not re-declared in `tools`, while the forwarded history still references it. The provider then rejects the whole request: ``` API Error: 400 Tool reference 'headroom_retrieve' not found in available tools. ``` Without a session the tracker can't remember the earlier turn's CCR, so this is unique to the sessionless path. ## Fix Add `history_references_ccr_tool(messages)` which detects an existing `headroom_retrieve` call in the forwarded messages — both the Anthropic assistant `tool_use` content block and the OpenAI assistant `tool_calls[].function.name` shapes, fully null-guarded. On the sessionless path, injection now fires when `has_compressed_content_this_turn` **or** history already references the tool, so the definition is re-declared and the request validates. The decision is logged as a new `inject_history_reference` outcome. Both handlers pass the signal computed from `optimized_messages` (the bytes actually forwarded). Behavior with a real `session_id` (the sticky tracker path) is unchanged. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/helpers.py`: add `history_references_ccr_tool`; add a `history_has_ccr_reference` parameter to `apply_session_sticky_ccr_tool` and OR it into the sessionless injection decision. - `headroom/proxy/tool_injection_logging.py`: add the `inject_history_reference` decision literal. - `headroom/proxy/handlers/anthropic.py`, `headroom/proxy/handlers/openai.py`: pass `history_references_ccr_tool(optimized_messages)` into the sticky-tool call. - `tests/test_ccr_tool_always_on.py`: regressions for the detector (both provider shapes + malformed inputs) and for sessionless re-injection when history references the tool. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_ccr_tool_always_on.py -q 14 passed # with just the `or history_has_ccr_reference` condition reverted, the new # sessionless re-injection test fails (tool not injected -> would 400) $ uvx ruff@0.15.17 check headroom/proxy/helpers.py headroom/proxy/tool_injection_logging.py headroom/proxy/handlers/anthropic.py headroom/proxy/handlers/openai.py tests/test_ccr_tool_always_on.py All checks passed! $ uvx mypy@1.20.2 --ignore-missing-imports headroom/proxy/helpers.py headroom/proxy/tool_injection_logging.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, project venv (`uv sync --extra proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`, pytest in the venv. - Exact command / steps: called `history_references_ccr_tool` on Anthropic `tool_use` and OpenAI `tool_calls` histories (plus null/non-list shapes), and `apply_session_sticky_ccr_tool(session_id=None, has_compressed_content_this_turn=False, history_has_ccr_reference=True)`; then temporarily reverted only the `or history_has_ccr_reference` condition and re-ran the regression. - Observed result: the detector returns `True` for both provider shapes and `False`/no-crash for malformed input; with the fix the sessionless call injects the tool (`was_injected=True`, tool present) even with no fresh compression; with the condition reverted the same call returns `was_injected=False` (the tool is dropped — exactly the 400 path). Ran against the actual module via `tests/test_ccr_tool_always_on.py`. - Not tested: a live sessionless multi-turn WS request reproducing the upstream 400 end to end. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net> |
||
|
dependabot[bot]
|
b30f339d69
|
deps: bump criterion from 0.5.1 to 0.8.2 (#2965)
Bumps [criterion](https://github.com/criterion-rs/criterion.rs) from 0.5.1 to 0.8.2. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/criterion-rs/criterion.rs/releases">criterion's releases</a>.</em></p> <blockquote> <h2>criterion-plot-v0.8.2</h2> <h3>Other</h3> <ul> <li>Update Readme</li> </ul> <h2>criterion-v0.8.2</h2> <h3>Fixed</h3> <ul> <li>don't build alloca on unsupported targets</li> </ul> <h3>Other</h3> <ul> <li><em>(deps)</em> bump crate-ci/typos from 1.40.0 to 1.43.0</li> <li>Fix panic with uniform iteration durations in benchmarks</li> <li>Update Readme</li> <li>Exclude development scripts from published package</li> </ul> <h2>criterion-plot-v0.8.1</h2> <h3>Fixed</h3> <ul> <li>Typo</li> </ul> <h2>criterion-v0.8.1</h2> <h3>Fixed</h3> <ul> <li>Homepage link</li> </ul> <h3>Other</h3> <ul> <li><em>(deps)</em> bump crate-ci/typos from 1.23.5 to 1.40.0</li> <li><em>(deps)</em> bump jontze/action-mdbook from 3 to 4</li> <li><em>(deps)</em> bump actions/checkout from 4 to 6</li> </ul> <h2>criterion-plot-v0.8.0</h2> <p>No release notes provided.</p> <h2>criterion-v0.8.0</h2> <h3>BREAKING</h3> <ul> <li>Drop async-std support</li> </ul> <h3>Changed</h3> <ul> <li>Bump MSRV to 1.86, stable to 1.91.1</li> </ul> <h3>Added</h3> <ul> <li>Add ability to plot throughput on summary page.</li> <li>Add support for reporting throughput in elements and bytes - <code>Throughput::ElementsAndBytes</code> allows the text summary to report throughput in both units simultaneously.</li> <li>Add alloca-based memory layout randomisation to mitigate memory effects on measurements.</li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/criterion-rs/criterion.rs/blob/master/CHANGELOG.md">criterion's changelog</a>.</em></p> <blockquote> <h2><a href="https://github.com/criterion-rs/criterion.rs/compare/criterion-v0.8.1...criterion-v0.8.2">0.8.2</a> - 2026-02-04</h2> <h3>Fixed</h3> <ul> <li>don't build alloca on unsupported targets</li> </ul> <h3>Other</h3> <ul> <li><em>(deps)</em> bump crate-ci/typos from 1.40.0 to 1.43.0</li> <li>Fix panic with uniform iteration durations in benchmarks</li> <li>Update Readme</li> <li>Exclude development scripts from published package</li> </ul> <h2><a href="https://github.com/criterion-rs/criterion.rs/compare/criterion-v0.8.0...criterion-v0.8.1">0.8.1</a> - 2025-12-07</h2> <h3>Fixed</h3> <ul> <li>Homepage link</li> </ul> <h3>Other</h3> <ul> <li><em>(deps)</em> bump crate-ci/typos from 1.23.5 to 1.40.0</li> <li><em>(deps)</em> bump jontze/action-mdbook from 3 to 4</li> <li><em>(deps)</em> bump actions/checkout from 4 to 6</li> </ul> <h2><a href="https://github.com/criterion-rs/criterion.rs/compare/criterion-v0.7.0...criterion-v0.8.0">0.8.0</a> - 2025-11-29</h2> <h3>BREAKING</h3> <ul> <li>Drop async-std support</li> </ul> <h3>Changed</h3> <ul> <li>Bump MSRV to 1.86, stable to 1.91.1</li> </ul> <h3>Added</h3> <ul> <li>Add ability to plot throughput on summary page.</li> <li>Add support for reporting throughput in elements and bytes - <code>Throughput::ElementsAndBytes</code> allows the text summary to report throughput in both units simultaneously.</li> <li>Add alloca-based memory layout randomisation to mitigate memory effects on measurements.</li> <li>Add doc comment to benchmark runner in criterion_group macro (removes linter warnings)</li> </ul> <h3>Fixed</h3> <ul> <li>Fix plotting NaN bug</li> </ul> <h3>Other</h3> <ul> <li>Remove Master API Docs links temporarily while we restore the docs publishing.</li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
d6fb5365f6
|
deps: update mcp requirement from <2.0.0,>=1.28.1 to >=1.28.1,<3.0.0 (#2963)
Updates the requirements on [mcp](https://github.com/modelcontextprotocol/python-sdk) to permit the latest version. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/modelcontextprotocol/python-sdk/releases">mcp's releases</a>.</em></p> <blockquote> <h2>v2.0.0</h2> <h1>MCP Python SDK v2 Stable Release</h1> <p>This is v2.0.0, the stable v2 release of the MCP Python SDK. It supports the 2026-07-28 revision of the Model Context Protocol and serves every earlier revision from the same server. <code>pip install mcp</code> now installs 2.x.</p> <pre lang="bash"><code>pip install "mcp[cli]" # or uv add "mcp[cli]" </code></pre> <h3>Documentation Rewrite</h3> <p>The <a href="https://py.sdk.modelcontextprotocol.io/">documentation</a> has the full tutorial and API reference. Coming from v1? <a href="https://py.sdk.modelcontextprotocol.io/whats-new/">What's new in v2</a> is the tour of what changed and why, and the <a href="https://py.sdk.modelcontextprotocol.io/migration/">migration guide</a> lists every breaking change with before-and-after code.</p> <h3>V1 Maintenance mode</h3> <p><strong>v1.x is in maintenance mode and will only receive security fixes from now on</strong> The 1.x line lives on the <a href="https://github.com/modelcontextprotocol/python-sdk/tree/v1.x"><code>v1.x</code> branch</a>, continues to receive critical bug fixes and security patches, and is documented at <a href="https://py.sdk.modelcontextprotocol.io/v1/">https://py.sdk.modelcontextprotocol.io/v1/</a>. If your project is not ready to migrate, keep a <code><2</code> upper bound on your requirement (for example <code>mcp>=1.28,<2</code>).</p> <h2>Highlights</h2> <h3>One SDK, both protocol eras</h3> <p>v2 speaks the 2026-07-28 revision (stateless requests with no handshake, <code>server/discover</code>, <code>subscriptions/listen</code>, multi-round-trip requests) and still serves every 2025-era client from the same <code>MCPServer</code>, over Streamable HTTP and stdio, with nothing to configure. <code>Client(target)</code> negotiates the version automatically.</p> <h3><code>FastMCP</code> is now <code>MCPServer</code>, and there is a first-class <code>Client</code></h3> <p>The decorator API is unchanged; the low-level <code>Server</code> is rebuilt around a shared dispatcher engine, and one <code>Client</code> object replaces v1's transport-plus-<code>ClientSession</code>-plus-<code>initialize()</code> layering. It connects to a URL, a stdio subprocess, a custom transport, or straight to a server object in memory for tests.</p> <h3>Multi-round-trip requests and resolver dependency injection</h3> <p>At 2026-07-28 the server can no longer call the client, so tools return the question instead. A <code>Resolve(fn)</code> parameter is filled by your function invisibly to the model and can put a question to the user; one tool body serves both eras.</p> <h3>Extension APIs, OpenTelemetry, and a standalone types package</h3> <p>Servers and clients compose protocol extensions through pluggable extension APIs (MCP Apps built in); OpenTelemetry tracing ships on by default; every protocol type is its own package, <code>mcp-types</code> (imported as <code>mcp_types</code>), published in lock-step with <code>mcp</code>.</p> <h3>Hardened stdio and auth</h3> <p>stdio servers keep handler subprocesses and stray prints off the wire, and stdout is diverted to stderr while serving. OAuth adds RFC 9207 issuer validation, the SEP-990 identity-assertion flow, and the client-credentials extension.</p> <h2>Coming from a v2 pre-release</h2> <p>Since the last release candidate: the per-version wire packages are private (<code>mcp_types._v*</code>), <code>mcp.types</code> is a permanent alias for <code>mcp_types</code>, the auth registration request model is split from the registered-client record, cancelled requests are no longer answered, and log notifications are gated on the per-request log-level opt-in at 2026-07-28. Since the betas: <code>Client(cache=False)</code> is now <code>cache=None</code> with <code>CacheConfig()</code> the default; <code>Context.client_id</code>, <code>RFC7523OAuthClientProvider</code>, and <code>OAuthClientProvider(timeout=)</code> are removed; the client-credentials providers take <code>scope=</code>; <code>message_handler</code> receives notifications and exceptions only; <code>FileResource(is_binary=)</code> becomes <code>encoding</code>; <code>MCP_*</code> env vars are gone with <code>pydantic-settings</code>; Streamable HTTP servers reject bodies over 4 MiB with HTTP 413. The migration guide covers all of it.</p> <h2>Known gaps</h2> <p>The tasks extension (SEP-2663) is not part of this release. On the client, the DPoP proof binding (SEP-1932) and the workload-identity <code>jwt-bearer</code> grant are not implemented; both are additive and can land in 2.x.</p> <h2>Feedback</h2> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
JD Davis
|
e269afb935
|
fix(ci): unjam release and Docker publishing (#2958)
## Description
Fixes two release-automation defects exposed by the 0.35.0 release:
1. Release Please grouped the single package into a PR titled `chore:
release main`, which could not be matched back to the `headroom-ai`
component/version and therefore never emitted the release event that
starts PyPI publishing.
2. Docker manifest jobs downloaded digest artifacts with overlapping
variant globs. For example, `digests-code-*` also selected code-nonroot,
code-slim, and code-slim-nonroot artifacts, yielding eight markers where
exactly two were required.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Route the single root package through Release Please's normal
versioned-PR path.
- Preserve both `${component}` and `${version}` in generated release PR
titles.
- Download Docker amd64 and arm64 digest artifacts by exact name instead
of an overlapping variant glob.
- Add regression assertions for both Release Please title matching and
Docker artifact isolation.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
python -m pytest tests/test_release_workflows.py -q
44 passed in 2.29s
pre-commit hooks:
Sync plugin versions........................Passed
Verify Ruff version alignment...............Passed
check for merge conflicts...................Passed
ruff (legacy alias).........................Passed
ruff format.................................Passed
mypy........................................Passed
```
## Real Behavior Proof
- Environment: GitHub Actions release runs 31653073583 and 31664975515,
`main` at the 0.35.0 release merge (`93f2d7a2`).
- Exact command / steps: Inspected the Release Please rerun and each
failed Docker manifest job; enumerated the digest artifacts downloaded
by their configured patterns.
- Observed result: Release Please logged `There are untagged, merged
release PRs outstanding - aborting`. Docker's `slim`, `code`, and
`code-slim` manifest cells found 4, 8, and 4 digest markers respectively
instead of 2 because their prefix globs included related variants. The
new Docker workflow requests `digests-<variant>-amd64` and
`digests-<variant>-arm64` by exact name.
- Not tested: A synthetic release was not published because registry
versions/tags are irreversible. Both configuration invariants are
covered by regression tests, and GitHub's PR workflow validation runs
against this branch.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A — workflow configuration and regression-test changes only.
## Additional Notes
- Documentation is N/A: these are internal release workflow corrections
with no user-facing command or API changes.
- The 0.35.0 Python release recovery is proceeding separately through
the existing wheel smoke-test and PyPI publication gates.
- The already-started 0.35.0 Docker run used the old workflow from tag
commit `
|
||
|
JD Davis
|
3077ac81e8
|
feat: add deterministic runtime rollout controls (#1490)
## Description Establish one centrally resolved, observable, deterministic, versioned runtime rollout-control mechanism for Headroom. Runtime rollout controls which behaviors an already-built artifact may expose; it does not select or qualify a Headroom release/version. ## Type of Change - [x] New feature (non-breaking change that adds functionality) - [x] Bug fix (non-breaking change that fixes rollout enforcement regressions) - [x] Documentation update - [x] Code refactoring (no functional changes) ## Changes Made - Added `RolloutChannel`, `HEADROOM_ROLLOUT_CHANNEL`, `--rollout-channel`, and a versioned immutable `RolloutSnapshot` shared by Python configuration boundaries. - Added schema/policy versions, canonical registry and snapshot SHA-256 identities, per-feature decision reasons, disable precedence, unsafe qualification poisoning, strict CLI validation, and fail-closed environment handling. - Added `headroom rollout status --json`, Python `/stats.rollout`, and Rust `/rollout/status` runtime provenance. - Added equivalent Rust snapshot semantics and shared Python/Rust policy vectors while retaining language-specific feature registries. - Enforced rollout policy at alternate Python server composition roots so `HEADROOM_READ_MATURATION=1` cannot bypass its beta gate. - Preserved typed rollout snapshots across multi-worker serialization with schema, policy, registry, snapshot-digest, type, and feature-name validation. - Made loopback runtime output-shaper updates replace the immutable snapshot atomically for request readers, retain explicit request/disable provenance, preserve channel and kill-switch precedence, invalidate cached stats, and return the effective rollout decision. - Made `headroom learn --verbosity --apply` report a channel-blocked update instead of claiming the shaper is live. - Made explicit CLI feature flags fail loudly when their current channel blocks them. - Made persistent interceptor installation select canary automatically, or reject an explicitly insufficient channel unless the break-glass override is set. - Updated architecture, proxy, rollout, learn, and output-shaper documentation with required channels and hot-reload semantics. ## Testing - [x] Unit tests pass - [x] Linting passes (`ruff check .` and `ruff format --check .`) - [x] Type checking passes (`mypy headroom --ignore-missing-imports`) - [x] New regression tests added for every corrected behavior - [x] Rust tests and production-target Clippy pass - [x] Documentation build passes ### Test Output ```text Focused rollout coverage suite 57 passed; headroom.rollout + rollout CLI: 98% coverage Affected proxy/rollout/transform/governance suites 222 passed; 0 failed Final changed regression suites 100 passed; 0 failed Cross-module hot-reload isolation regression 6 passed; 0 failed cargo test -p headroom-core -p headroom-proxy --quiet headroom-core: 924 passed; 1 ignored headroom-proxy and integration suites: all passed cargo clippy -p headroom-core -p headroom-proxy --lib --bins -- -D warnings cargo fmt --all -- --check ruff check . ruff format --check . mypy headroom --ignore-missing-imports git diff --check All passed cd docs && npm run build Compiled successfully; 164 static pages generated ``` The unsharded Windows-only CI selection exposed unrelated baseline failures, principally the existing `sqlite:///C:\\...` URL parser producing an invalid `\\C:\\...` path. At commit ` |
||
|
Tejas Chopra
|
93f2d7a2da
|
chore: release main (#2792)
🤖 I have created a release *beep* *boop* --- <details><summary>0.35.0</summary> ## [0.35.0](https://github.com/headroomlabs-ai/headroom/compare/v0.34.0...v0.35.0) (2026-08-12) ### Features * **beacon:** allowlist the routing summary key ([#2818](https://github.com/headroomlabs-ai/headroom/issues/2818)) ([ |
||
|
dependabot[bot]
|
148d8605e2
|
deps: bump the cargo-minor-patch group across 1 directory with 22 updates (#2916)
Bumps the cargo-minor-patch group with 21 updates in the / directory: | Package | From | To | | --- | --- | --- | | [serde_json](https://github.com/serde-rs/json) | `1.0.150` | `1.0.151` | | [thiserror](https://github.com/dtolnay/thiserror) | `2.0.18` | `2.0.19` | | [anyhow](https://github.com/dtolnay/anyhow) | `1.0.103` | `1.0.104` | | [clap](https://github.com/clap-rs/clap) | `4.6.2` | `4.6.6` | | [tokio](https://github.com/tokio-rs/tokio) | `1.52.3` | `1.53.1` | | [pyo3](https://github.com/pyo3/pyo3) | `0.29.0` | `0.29.2` | | [aws-config](https://github.com/smithy-lang/smithy-rs) | `1.9.0` | `1.10.1` | | [aws-smithy-runtime-api](https://github.com/smithy-lang/smithy-rs) | `1.13.0` | `1.14.0` | | [unidiff](https://github.com/messense/unidiff-rs) | `0.4.0` | `0.4.1` | | [aho-corasick](https://github.com/BurntSushi/aho-corasick) | `1.1.4` | `1.1.5` | | [toml](https://github.com/toml-rs/toml) | `1.1.3+spec-1.1.0` | `1.1.4+spec-1.1.0` | | [blake3](https://github.com/BLAKE3-team/BLAKE3) | `1.8.5` | `1.8.6` | | [http](https://github.com/hyperium/http) | `1.4.2` | `1.5.0` | | [futures](https://github.com/rust-lang/futures-rs) | `0.3.32` | `0.3.33` | | [hyper](https://github.com/hyperium/hyper) | `1.10.1` | `1.11.0` | | [bytesize](https://github.com/bytesize-rs/bytesize) | `2.4.2` | `2.7.0` | | [tokio-util](https://github.com/tokio-rs/tokio) | `0.7.18` | `0.7.19` | | [lru](https://github.com/jeromefroe/lru-rs) | `0.18.1` | `0.18.2` | | [async-trait](https://github.com/dtolnay/async-trait) | `0.1.89` | `0.1.91` | | [tokio-stream](https://github.com/tokio-rs/tokio) | `0.1.18` | `0.1.19` | | [cc](https://github.com/rust-lang/cc-rs) | `1.2.67` | `1.4.1` | Updates `serde_json` from 1.0.150 to 1.0.151 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/serde-rs/json/releases">serde_json's releases</a>.</em></p> <blockquote> <h2>v1.0.151</h2> <ul> <li>Add RawValue::from_string_unchecked (<a href="https://redirect.github.com/serde-rs/json/issues/1331">#1331</a>, thanks <a href="https://github.com/WonderLawrence"><code>@WonderLawrence</code></a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
Parideboy
|
3752458022
|
fix(cache): enforce Anthropic's 1h-before-5m cache_control ordering before forwarding (#2941)
## Description
Anthropic evaluates prompt-cache breakpoints in **one pass over the
whole request** — `tools`, then `system`, then `messages` — and rejects
the request outright when a `ttl='1h'` breakpoint appears after a
5-minute one. A bare `{"type": "ephemeral"}` marker counts as 5 minutes,
so this is easy to trip without any `ttl` field being visibly wrong:
```
API Error: 400 messages.15.content.1.cache_control.ttl: a ttl='1h' cache_control block must not
come after a ttl='5m' cache_control block. Note that blocks are processed in the following order:
tools, system, messages.
```
Headroom rewrites `cache_control` markers in several independent places,
each looking at one section, and nothing checked the invariant that
spans them. The failure mode is a dead turn, not a silent cost
regression.
Two paths can leave the forwarded body illegal today:
1. **Replayed 1h marker in a 5m request.** Claude Code picks its TTL
lane per request, not per session: the main loop asks for 1h and sends
the `extended-cache-ttl` beta header, while a side question (`/btw` in
the report) goes out in the 5m lane with bare markers and no beta
header. Headroom replays part of the previous turn's forwarded bytes
into `messages` to keep the prefix stable, and those bytes still carry
`ttl: "1h"`. `tools`/`system` at 5m, `messages` at 1h — 400.
2. **Tools breakpoint downgraded.** `inject_tool_search_deferral`
re-places the *last* marker it stripped, so a bare marker on a later
deferred tool overwrites a `ttl='1h'` one. The tools prefix goes
upstream at 5m while message breakpoints are still 1h — 400. Reported
separately as #2767.
The rule spans `tools`/`system`/`messages`, so no individual transform
is in a position to check it. The fix is a guard at the last seam before
the body goes on the wire, plus the one Python/Rust divergence that
manufactures the violation upstream of it.
Closes #2939
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/helpers.py`: new `enforce_cache_control_ttl_order`,
plus `cache_control_ttl_lane` / `cache_control_ttl_lanes` /
`walk_cache_control`. The walk visits markers in Anthropic's documented
order and matches the traversal `count_cache_breakpoints` already
performs, so the two cannot disagree about what counts as a breakpoint.
TTL ranking is ported verbatim from the Rust `TtlOrderingWalk::observe`:
absent or `"5m"` is short, `"1h"` is long, any other value is left alone
rather than guessed at. Two repairs:
- **Lane containment** — when the client sent no 1h marker of its own,
strip `ttl` from any 1h marker that leaked in. That request never sent
the `extended-cache-ttl` beta header, so it could not have written a 1h
entry anyway; nothing is lost. Other marker fields (`scope`, …) are
preserved.
- **Ordering** — when the client did ask for 1h, promote every 5m marker
preceding the last 1h one. Demoting would also make the request legal
but would discard 1h caching the client is explicitly paying for, which
is the regression #2375 / #2382 / #2651 were filed to stop. A violation
seen on the way out means headroom downgraded or introduced a marker, so
promoting restores what the client's own (legal) request asked for at
that position.
- Copy-on-write: a legal body is returned by identity, so the hot path
pays only a walk over at most a handful of markers. Kill switch
`HEADROOM_CACHE_CONTROL_TTL_GUARD=0`, matching the
`HEADROOM_TOOL_SEARCH=0` convention.
- `headroom/proxy/handlers/anthropic.py`:
- Capture the client's TTL lane from the inbound snapshot, before any
transform runs. This cannot be inferred from the session or from config
— the lane is a per-request property of Claude Code, which is the whole
reason the `/btw` case exists.
- Call the guard immediately before `log_cache_breakpoints`, i.e. after
every transform, the tool sort, the deferral, CCR injection and the
pipeline extensions. Mark the body mutated when a repair fires, and log
a WARNING carrying the repair kind, the counts and the offending
sections — the diagnostic the next report of this class will need.
- `_sort_tools_deterministically` now skips the sort when any tool
carries `cache_control`, logging `event=tool_sort_skipped
reason=marker_present`. A breakpoint on a tool means "cache through
here", so reordering changes what is inside the cached prefix and can
move a 1h-marked tool behind a 5m-marked one. The Rust proxy already
refuses for exactly this reason (`any_tool_has_cache_control` in
`crates/headroom-proxy/src/compression/live_zone_anthropic.rs:651`); the
Python path never got the same guard. Putting the check in
`_sort_tools_deterministically` rather than `_tools_for_forwarding`
covers all call sites including the batch path.
- `tests/test_cache_control_ttl_order.py`: new, 24 cases.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
The new tests validate against an independent reimplementation of
Anthropic's rule rather than against the guard's own walk, so a bug in
the walk cannot make the assertions pass. Coverage: lane classification
(bare marker is 5m, unknown TTLs are `other`); containment of a replayed
1h marker including preservation of non-`ttl` fields; promotion across
`tools`→`messages`, `system`→`messages` and within `messages`; markers
nested in `tool_result` sub-blocks; only markers before the *last* 1h
one are rewritten, so a legal 1h-then-5m ordering is left alone; legal
bodies returned by `is` identity; unknown `ttl` untouched; kill switch;
the tool sort skipping on marked tools and still sorting unmarked ones,
with one test pinning that the sort *would* have created a violation
without the guard; and an end-to-end regression running
`inject_tool_search_deferral` then the guard on the #2767 shape.
### Test Output
```text
$ pytest tests/test_cache_control_ttl_order.py -q
24 passed in 0.77s
$ pytest tests/test_cache_ttl_preserved.py tests/test_cache_control_move_bust.py \
tests/test_cache_breakpoint_diagnostics.py tests/test_issue_746_tool_search.py -q
86 passed in 1.36s
$ pytest tests/test_cache/ tests/test_proxy/ -q
16 failed, 489 passed, 2 skipped in 91.41s
$ ruff check headroom/ tests/test_cache_control_ttl_order.py
All checks passed!
$ ruff format --check headroom/ tests/test_cache_control_ttl_order.py
1 file would be reformatted, 520 files already formatted
$ mypy --python-version 3.13 headroom --ignore-missing-imports
Found 12 errors in 3 files (checked 517 source files)
```
The three non-green results above are all pre-existing on a clean
`upstream/main` in this environment, verified by stashing the changes
and re-running:
- The 16 failures are all in
`tests/test_cache/test_client_integration.py` and are a Windows
temp-path problem in this sandbox (`OSError: [WinError 123] ...
'\\C:\\Users\\...\\Temp'`), not a code failure. They fail identically
with the branch stashed.
- `ruff format --check` flags `headroom/testing/README.md`, a docs code
block untouched by this PR.
- The mypy errors are in `headroom/memory/mcp_server.py` and
`headroom/release_version.py`; none are in the files this PR changes.
`--python-version 3.13` is needed locally because the pinned
`python_version = "3.10"` makes mypy reject the installed numpy stubs
before it checks anything.
## Real Behavior Proof
- Environment: Windows 11, Python 3.13.11, headroom at this branch's
head, run against the real `headroom.proxy` forwarding helpers. No
Anthropic API key is available in this environment, so Anthropic's
validator is reimplemented locally from its documented rule and its own
error string; the request bodies are produced by the real code path
(`_sort_tools_deterministically` then `inject_tool_search_deferral` then
the guard), not hand-written.
- Exact command / steps: build two request shapes — (A) a 5m-lane
request whose `messages` carries a replayed `ttl:"1h"` marker, the
`/btw` case; (B) 13 tools with markers on two deferred tools plus 1h
message breakpoints, the #2767 case — push each through the forwarding
helpers twice, once with `HEADROOM_CACHE_CONTROL_TTL_GUARD=0` and once
with the guard at its default, and validate the resulting body.
- Observed result: both scenarios are rejected with the issue's exact
400 when the guard is off, and both are legal with it on. Scenario A is
repaired by lane containment (the leaked 1h ttl is stripped), scenario B
by promotion (the downgraded tools breakpoint goes back to 1h). Full
output:
```text
########## Scenario A: /btw side question replays a 1h marker into a 5m request ##########
===== BEFORE (HEADROOM_CACHE_CONTROL_TTL_GUARD=0) =====
tools.0 5m
system.0 5m
messages.1.content.0 1h
RESULT: API Error: 400 messages.1.content.0.cache_control.ttl: a ttl='1h' cache_control block
must not come after a ttl='5m' cache_control block. Note that blocks are processed in the
following order: tools, system, messages.
===== AFTER (default) =====
WARNING event=cache_control_ttl_order request_id=repro-2939 repair=lane_containment demoted=1
leaked_from_section=messages; the client sent no 1h marker, so a replayed 1h breakpoint would
have been rejected upstream
tools.0 5m
system.0 5m
messages.1.content.0 5m
RESULT: 200 OK (request satisfies the ordering rule)
########## Scenario B: tool-search deferral downgrades the tools breakpoint ##########
===== BEFORE (HEADROOM_CACHE_CONTROL_TTL_GUARD=0) =====
INFO event=tool_sort_skipped reason=marker_present tool_count=13 marked=2
tools.1 5m
messages.1.content.0 1h
RESULT: API Error: 400 messages.1.content.0.cache_control.ttl: a ttl='1h' cache_control block
must not come after a ttl='5m' cache_control block. Note that blocks are processed in the
following order: tools, system, messages.
===== AFTER (default) =====
INFO event=tool_sort_skipped reason=marker_present tool_count=13 marked=2
WARNING event=cache_control_ttl_order request_id=repro-2939 repair=promote_to_1h promoted=1
first_short_section=tools first_long_section=messages; a 5m breakpoint preceded a 1h one, which
Anthropic rejects outright
tools.1 1h
messages.1.content.0 1h
RESULT: 200 OK (request satisfies the ordering rule)
```
- Not tested: no live call to `api.anthropic.com` — no credentials in
this environment — so the 400/200 above come from a local
reimplementation of the rule, not from the API itself. The reporter's
original `/btw` flow was not reproduced end to end through `headroom
wrap claude`. Cache-hit-rate impact of promoting a 5m marker to 1h was
not measured against real traffic; the reasoning for promoting over
demoting is argued above, not benchmarked. Nothing on the Rust side was
exercised.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
Documentation was not updated: the new env var is a kill switch for an
internal correctness guard with no user-facing behaviour when things are
working, matching how `HEADROOM_KOMPRESS_BACKGROUND_WARM` is handled.
Happy to add a line to the env-var reference if maintainers prefer.
**One trade-off worth a maintainer's eye.** `affinity_tools` feeds a
`segment_fingerprint` used for prefix-tracker affinity. Skipping the
sort makes that fingerprint depend on the client's tool order. Clients
that mark tools must already keep a stable order for their own prefix
cache to work, so this should be safe, but it is stated rather than
assumed.
**Deliberately out of scope, flagged rather than dropped:**
- The sibling hole in `inject_tool_search_deferral`: when
`resident_has_cache_control` is already true at 5m, a dropped 1h marker
is discarded outright. That is a cost regression rather than a 400, and
it sits in the same handful of lines that the open PR #2771 rewrites, so
touching it here would conflict. Better raised on #2767.
- `TtlOrderingWalk` in `crates/headroom-core/src/cache_control.rs` is
instantiated separately per field list, so it only ever sees violations
*within* `messages`, `system` or `tools` — never across them — and it
only warns. Its module doc justifies warn-only with "Anthropic itself
accepts both orderings (just with potentially-suboptimal cache
eviction)". #2939 and #2767 both show that premise is now stale.
Changing Rust behaviour is a separate blast radius.
- `cold_prefix._cache_control_ttls` never scans `tools[]`, so a client
whose only 1h marker rides on `tools` is read as 300s. Separate bug,
separate PR.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|