mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
2582 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
JD Davis
|
8a1d38bc5d
|
fix(proxy): complete stateless Responses and buffered CCR lifecycle (#2997)
## Description
Consolidates the related OpenAI Responses ZDR/stateless continuation and
buffered CCR response-lifecycle corrections on current main. It
preserves client storage policy, makes Headroom-owned continuations
stateless across HTTP and WebSocket, and prevents buffered streaming
paths from committing a false HTTP 200 before the real upstream outcome
is known.
Closes #2675
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Preserves explicit and omitted Responses `store` policy instead of
forcing provider storage or disabling memory tools.
- Replays normalized input, replayable outputs, encrypted reasoning
content, and Headroom function outputs without `previous_response_id`.
- Applies the same stateless continuation policy to HTTP and WebSocket.
- Prevents transparent memory execution after client-visible WebSocket
output.
- Delays buffered CCR ASGI status/headers until the operation resolves
for Anthropic Messages and OpenAI Responses.
- Preserves real 429/5xx status and retry headers.
- Converts malformed non-JSON/non-SSE upstream 200 replies to a
sanitized 502 protocol error.
- Preserves valid JSON-to-SSE synthesis and existing SSE adaptation.
- Removes unreachable task cleanup left behind after replacing the old
keepalive polling loop with a direct awaited operation.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
118 passed across the changed HTTP/WS ZDR, lifecycle, and both-provider CCR suites
11526 tests collected with no collection errors
ruff check .: All checks passed
ruff format --check .: 1411 files already formatted
mypy headroom/proxy/handlers/anthropic.py headroom/proxy/handlers/openai.py:
Success: no issues found in 2 source files
```
Exact-head CI is entirely green on
`
|
||
|
JD Davis
|
a708c0571e
|
fix(ci): prevent native detector from hanging test shards (#2996)
## Description
CI shard 4 was not merely slow: after thousands of fast tests it parked
indefinitely inside `headroom._core.detect_content_type` at 0% CPU. The
router watchdogged only the first native call and then permanently
trusted direct calls via `_detect_native_verified`. Earlier suite
activity can change ORT/native state after that first success, making a
later call deadlock until GitHub cancels the job.
This keeps every native call bounded by the existing watchdog, activates
the process-wide pure-Python circuit breaker after a timeout, restores
the test-job ceiling to 30 minutes, and removes a separate wall-clock
scheduler assertion that generated false shard-1 failures despite the
structural regression guards passing.
No issue is auto-closed by this infrastructure repair.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Removed the unsafe process-lifetime `_detect_native_verified` fast
path.
- Kept every native detection call behind the existing bounded watchdog.
- Preserved the process-wide fallback circuit breaker so only the first
wedged call consumes the watchdog budget.
- Added a success-then-hang regression test.
- Isolated native circuit-breaker state in fallback exception tests.
- Restored the CI test timeout from the temporary 90-minute diagnostic
ceiling to 30 minutes.
- Replaced the Codex scheduler's noise-sensitive p99/p50 assertion with
its meaningful absolute regression ceiling while retaining source-level
guards against the removed semaphore and nested executor.
- Corrected import order and formatting defects inherited from current
main so the synthetic merge commit passes repository-wide lint.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
Exact local shard-4 command with coverage:
2723 passed, 172 skipped, 8661 deselected in 108.40s
Focused detector/router suite:
62 passed
Codex scheduler suite:
3 passed, 1 skipped
ruff check .
All checks passed!
ruff format --check .
1411 files already formatted
mypy headroom/transforms/content_router.py
Success: no issues found in 1 source file
```
Exact-head GitHub CI on `
|
||
|
Tejas Chopra
|
3145242645
|
Unify savings attribution across stats, perf, metrics, and dashboard (#2976)
## Summary Adds a small provider-neutral savings attribution seam. Named sources can attach realized or projected token/USD deltas to a request without changing headline arithmetic or introducing private-package inventory into OSS. Also fixes the Anthropic buffered lifecycle so normal successful responses run response hooks, applies stream-safety filtering, includes tool savings in per-model perf totals, and surfaces the same breakdown in request logs, `/stats`, `headroom perf`, Prometheus, OTEL, and the dashboard. ## Why Request-local savings were split between canonical token deltas, process-global extension counters, and tool-only tags. This made correct headline totals possible while losing attribution in perf, recent requests, metrics, and the dashboard. Normal Anthropic responses also skipped response hooks unless CCR ran. ## Validation - 74 focused tests passed: turn hooks, OpenAI hook lifecycle, outcome funnel, perf formats, and tool-search repair - Ruff passes on all changed Python files - Existing compression-observability suite: 11 passed; 2 tokenizer-cache tests require network access to fetch the tiktoken vocabulary ## Compatibility No named private packages or private inventory are encoded in OSS. Existing hooks remain source-compatible because all new TurnContext fields are optional. --------- Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> |
||
|
JD Davis
|
1aa701adaa
|
fix(vscode): persist compatible Claude modes and route Copilot CAPI (#2986)
## Description Fixes #2492, #2028, and #2827. Claude daemon workers consume project settings rather than reliably inheriting wrapper environment state, while the Claude VS Code webview cannot render deferred-tool response blocks. Separately, recent Copilot Chat versions use the whole CAPI override for generation; the legacy proxy override alone only sends model discovery through Headroom. This PR carries both integrations through to the actual consumers instead of only changing their launch-time surface configuration. ## Type of Change - [x] Bug fix - [ ] New feature - [ ] Breaking change - [ ] Documentation update - [ ] Build / CI ## Changes Made - Persist the resolved Claude ENABLE_TOOL_SEARCH value into project settings for daemon workers and restore it transactionally after wrap exits. - Use compatibility-safe Foundry and Claude VS Code defaults while preserving explicit user choices. - Configure both Copilot overrideProxyUrl and overrideCapiUrl in the reversible managed VS Code settings block. - Route Copilot unprefixed POST /chat/completions and HTTP /responses requests through the real compression handlers. - Keep /responses out of the Codex WebSocket aliases because Copilot and Codex use different WebSocket wire protocols. - Extend wrap E2E assertions for both the Claude webview mode and Copilot CAPI routing. ## Testing - [x] 127 combined Claude, Copilot, route-integration, and MCP dependency-contract tests pass. - [x] Ruff check passes on all changed Python files. - [x] Ruff format check passes. - [x] Python compilation and git diff --check pass. ## Runtime Safety Standalone Claude CLI defaults remain unchanged. Explicit Claude tool-search values retain precedence, and project settings are restored through the existing cleanup path. Copilot model/session helper endpoints continue through generic passthrough, while only validated HTTP generation paths receive explicit compression routes. Existing Codex WebSocket behavior is unchanged. ## Review Readiness - [x] Current main and MCP v1 compatibility retained - [x] Worker-facing Claude persistence covered - [x] Reversible Copilot and Claude settings behavior covered - [x] Copilot generation routes covered at registration and proxy integration layers - [x] Ready for review |
||
|
JD Davis
|
eafdf11a2c
|
fix(docker): ship Bedrock auth and current registry (#2982)
## Description Fixes #1551 and #1692. Every published Headroom Docker image now installs the existing `bedrock` extra, so `--backend bedrock` can authenticate with temporary STS, SSO, and credential-process credentials instead of failing because `botocore` is absent. Public Docker instructions now consistently use `ghcr.io/headroomlabs-ai/headroom`. Several still pointed at the old personal package, which is frozen at 0.27.0 and caused users to report that no latest image existed. ## Type of Change - [x] Bug fix - [ ] New feature - [ ] Breaking change - [x] Documentation update - [x] Build / CI ## Changes Made - Add `bedrock` to the standalone Dockerfile default extras. - Add `bedrock` to all nine root/code/slim/nonroot bake targets. - Replace obsolete personal GHCR references in README, llms.txt, Compose guidance, testing guidance, and wiki docs. - Add release contract tests for Bedrock dependencies and the current organization registry. ## Testing - [x] Focused Docker release and Bedrock preflight tests pass. - [x] Full updater suites pass: 69 tests. - [x] `uv run ruff check tests/test_release_workflows.py` - [x] `docker buildx bake --print` - [x] `git diff --check` ## Real Behavior Proof Before this change, every published bake target installed only `proxy` or `proxy,code`, so `AWS_SESSION_TOKEN` selected an unavailable botocore path. Public copy-paste commands also referenced `ghcr.io/chopratejas/headroom`, which the existing migration code and changelog identify as frozen at 0.27.0. After this change, all nine parsed bake targets install `bedrock`; the regression resolves that package extra and confirms `boto3` plus `botocore`. Every public Docker instruction covered by the contract names `ghcr.io/headroomlabs-ai/headroom`. ## Runtime Rollout Safety This changes image contents and documentation only; proxy routing and non-Docker installs are unchanged. Static AWS credentials remain unaffected. Existing manifests using the deprecated image continue to be migrated by the established install-state logic. Rollback is a Docker/bake extras and documentation revert. ## Review Readiness - [x] Two related Docker blockers batched in one PR - [x] Regression coverage included - [x] No unrelated lockfile changes - [x] Ready for review |
||
|
JD Davis
|
ddd2a259ec
|
fix(install): consolidate Windows fallback and cleanup safety (#2980)
## Description Consolidates two fully reviewed installation-safety fixes whose original PRs can no longer merge under current branch protection: Windows persistent-service deployments need a supported Task Scheduler fallback, and legacy context-tool cleanup must never delete user-owned RTK/lean-ctx artifacts. Closes #2552 Closes #2817 Supersedes #2600 and #2828 while preserving their authors' commits and review-driven corrections. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Convert Windows `persistent-service` plans to the supported `persistent-task` supervisor and make the fallback explicit in CLI output. - Restrict context-tool cleanup to artifacts proven to live under Headroom's managed directory. - Recognize wrapped, relative, and platform-specific managed commands without accepting prefixed/path-boundary lookalikes. - Scope cleanup completion state correctly across projects and alternate agent homes. - Stamp cleanup complete only after all managed remnants are settled. - Preserve the original focused regression suites and behavior-proof artifact. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_install/test_planner.py tests/test_install/test_supervisors.py tests/test_cli/test_install_cli.py tests/test_context_tool_cleanup.py tests/test_cli/test_unwrap_claude.py 135 passed in 0.45s $ uv run ruff check <changed Python and test files> All checks passed! $ uv run ruff format --check <changed Python and test files> 8 files already formatted ``` ## Real Behavior Proof - Environment: macOS arm64 for consolidated current-main validation; the Windows fallback source PR was independently validated on Windows and includes its captured verification artifact. - Exact command / steps: run the planner, supervisor, install CLI, cleanup provenance, and unwrap suites on the rebased combined branch. - Observed result: 135/135 focused tests pass. Windows service requests resolve to `persistent-task`; cleanup rejects user-owned and path-prefix lookalikes while removing managed artifacts. - Not tested: a fresh privileged Windows host deployment in this local pass; #2600's accepted review contains the Windows-specific proof. ## Runtime Rollout Safety - Rollout-managed feature(s): Install supervisor selection and one-time legacy cleanup. - Minimum rollout channel: Stable/default; both prevent currently destructive or nonfunctional install paths. - Stable/default behavior changed: Windows service requests use Task Scheduler; cleanup requires managed provenance. - Kill switch / disable path: Select `persistent-task` explicitly; cleanup remains bounded by its completion stamp and provenance checks. - Unsafe override required: No. - Qualification impact: Windows native install and wrap/unwrap cleanup suites. - Rollback path: Revert this PR, restoring the two pre-fix behaviors. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) The Windows verification artifact from #2600 is retained at `.github/pr-images/issue-2552-windows-fallback-verification.png`. ## Additional Notes This is intentionally an installation-safety batch rather than two replacement PRs. Original commit authorship is preserved, and the combined diff was applied cleanly to current `main` after #2832 and #1628 landed. --------- Co-authored-by: Inference1 <68734681+Inference1@users.noreply.github.com> Co-authored-by: Dennis Alexis Valin Dittrich <dd+github@dr-dittrich.de> |
||
|
JD Davis
|
a3fe5cb65b
|
fix(onnx): enforce Rust API-24 runtime compatibility (#2979)
## Description Rust fastembed enables ORT C API 24, but the Python dependency allowed ONNX Runtime 1.23.2. Entering ort's initializer with that library deadlocks permanently instead of returning an error. Align dependency resolution where compatible wheels exist and preflight native detection where they do not. Closes #2960 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Require ONNX Runtime 1.24+ for Python 3.11+ in the proxy and voice extras. - Keep the available pre-1.24 runtime on Python 3.10 for Python ONNX consumers. - Refuse to auto-pin an incompatible runtime into the Rust extension. - Bypass native detection immediately when API 24 is unavailable, preserving Python fallback without a five-second watchdog delay or stuck native thread. - Add dependency, pinning, override, and router regression coverage. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_transforms/test_ort_dylib.py tests/test_onnx_dependency_contract.py tests/test_onnx_runtime.py tests/test_transforms/test_content_router.py 88 passed in 9.31s $ uv run ruff check headroom/_ort.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py tests/test_onnx_dependency_contract.py All checks passed! ``` ## Real Behavior Proof - Environment: macOS arm64; Python 3.13.14 and uv-managed Python 3.10.20. - Exact command / steps: run the issue's direct `headroom._core.detect_content_type` call in a subprocess with a 12-second timeout on Python 3.13; run `_detect_content` on Python 3.10 after resolving the proxy extra. - Observed result: Python 3.13 resolves ORT 1.26.0 and native detection returns `json_array`; Python 3.10 resolves ORT 1.23.2, leaves `ORT_DYLIB_PATH` unset, reports compatibility false, and immediately returns the Python `json_array` fallback. - Not tested: Linux-specific shared-object execution locally; CI's existing Linux Rust job already preflights ORT 1.24+ and exercises native tests. ## Runtime Rollout Safety - Rollout-managed feature(s): Native Rust content detection. - Minimum rollout channel: Stable/default; this is a deadlock prevention guard. - Stable/default behavior changed: Python 3.11+ installs a compatible ORT; Python 3.10 skips incompatible native detection. - Kill switch / disable path: `HEADROOM_DETECT_BACKEND=python` remains available; an explicit `ORT_DYLIB_PATH` remains an operator override. - Unsafe override required: No. - Qualification impact: Native detection stays enabled only with API-24-compatible ORT. - Rollback path: Revert this PR, which restores the old watchdog-only degradation. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable. ## Additional Notes The large lockfile diff is dependency resolution: Python 3.10 keeps ORT 1.23.2 while 3.11+ resolves 1.26.0. The functional Python change is intentionally small and keeps explicit `ORT_DYLIB_PATH` overrides working. |
||
|
Abhay Singh
|
7de35739c6
|
fix(proxy/anthropic): repair headroom_retrieve history references the tools array cannot support (#2876)
## Description #2805 / #2807 established the mechanism: Claude Code replays one transcript across requests that carry different `tools` arrays, and Anthropic validates every history reference against the array of the request at hand. #2807 fixed it for tool-search blocks by repairing history (`strip_unsupported_tool_search_blocks`) rather than trying to predict the client's tool set. The same mechanism applies to CCR's `headroom_retrieve`, and it is tool-agnostic. A passthrough side-request (the prompt-type Stop hook evaluator, `/compact`) that the proxy forwards without declaring `headroom_retrieve` still carries a historical `tool_use` naming it, and Anthropic 400s on the dangling reference. The injection-side fixes (#2766 / #2533) decide *when to re-declare the tool*; this makes the 400 *structurally impossible* where the tool is intentionally absent. It is belt-and-braces with them, not a replacement. The fix adds the symmetric repair next to #2807's. When the outbound `tools` array does not declare `headroom_retrieve`, it replaces each `headroom_retrieve` `tool_use` and its paired `tool_result` with a text block, so no dangling reference survives. It **neutralizes** (replaces in place) rather than **drops**, which is the one deliberate difference from #2807: CCR's `tool_use` lives in an assistant turn and its `tool_result` in the next user turn, i.e. two different messages. Dropping a whole message could leave two same-role messages adjacent and break Anthropic's strict user/assistant alternation, turning one 400 into another. Replacing blocks in place keeps every message and role intact, and preserves the retrieved text the model already saw. #2807's server-tool blocks both live in the same assistant turn, so dropping was safe there. It runs after CCR tool injection, so on the main loop -- where the tool IS injected (a present marker) -- it neutralizes nothing and the prompt-cache prefix is untouched, mirroring #2807's placement and sequencing. Fixes #2814 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/helpers.py`: added `strip_unsupported_ccr_retrieve_blocks(messages, tools)` (and a small `_ccr_result_as_text` helper). No-ops (returning the original object by identity) when `headroom_retrieve` is declared or no such history exists; otherwise neutralizes the `tool_use` and its paired `tool_result` to text. - `headroom/proxy/handlers/anthropic.py`: call the repair right after the tool-search history repair (which is after CCR tool injection), guarded on it actually changing anything, tagged `router:ccr_retrieve_repair:Nblocks`. - `tests/test_ccr_retrieve_history_repair.py`: 5 unit tests (no-op when declared, no-op without retrieve history, neutralize + preserve result text + keep alternation, leave foreign tool_use untouched, placeholder when the result has no text). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text tests/test_ccr_retrieve_history_repair.py 5 passed # Broader CCR / tool-search / handler suites (unchanged behavior): tests/test_ccr_retrieve_history_repair.py tests/test_proxy/test_anthropic_ccr_deferred_injection.py tests/test_issue_746_tool_search.py 71 passed # uvx ruff@0.15.17 check -> All checks passed! # uvx mypy@1.20.2 headroom/proxy/helpers.py -> Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1, ruff 0.15.17 and mypy 1.20.2 via uvx. - Exact command / steps: confirmed the injection point (`apply_session_sticky_ccr_tool`) and the tool-search repair placement in `handlers/anthropic.py`, confirmed `body["tools"]` reflects the CCR injection before the repair call site (`body["tools"] = tools` is written well upstream and the adjacent tool-search repair already relies on it), then drove the helper over a transcript with a `headroom_retrieve` tool_use + paired tool_result: with the tool declared it returns the original object unchanged; with the tool absent it neutralizes both blocks, preserves the result text, and keeps the message roles/count identical. - Observed result: a forwarded request that would 400 with "Tool reference 'headroom_retrieve' not found in available tools" now carries text blocks in place of the retrieve `tool_use`/`tool_result`, so there is no reference for Anthropic to reject, and user/assistant alternation is preserved. The main loop (tool present) is a no-op. - Not tested: a live multi-turn Claude Code session hitting a Stop-hook/`/compact` side-request against a real provider (no live provider here). The repair is a pure function verified directly over the exact block shapes Anthropic validates, and it mirrors the already-merged tool-search repair's mechanism and wiring. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md`: it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes The issue reporter noted their own logs show the tool-search variant of this 400 (61 across 11 days) but zero `headroom_retrieve` occurrences, because they run `HEADROOM_LOSSLESS=1` which disables CCR entirely. This PR fixes the CCR variant of the same, proven, tool-agnostic mechanism rather than a fresh CCR repro. The neutralize-vs-drop choice is the one place I departed from #2807, for the alternation reason above; if you would rather it drop (accepting the alternation handling that implies), I am happy to switch it. --------- Co-authored-by: Jerrett Davis <mxjerrett@gmail.com> |
||
|
JD Davis
|
6077e5a149
|
fix(mcp): restore SDK v1 compatibility cap (#2978)
## Description PR #2963 widened the MCP dependency to 2.x while Headroom's live MCP server still uses the v1 low-level `Server.list_tools()` and `Server.call_tool()` decorators. Fresh installs therefore crash before serving tools. Restore the v1 cap until the explicit SDK 2.x port in #2658 lands, and pin that compatibility contract with a regression test. Closes #2977 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Restore `mcp>=1.28.1,<2.0.0` in the `proxy` and `mcp` extras. - Regenerate `uv.lock`, resolving MCP 1.28.1 and removing the incompatible 2.x transitive set. - Add a dependency-contract test covering both shipping extras. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_mcp_dependency_contract.py tests/test_ccr_mcp_server.py tests/test_cli/test_mcp.py 41 passed in 0.56s $ uv run ruff check tests/test_mcp_dependency_contract.py All checks passed! $ uv run ruff format --check tests/test_mcp_dependency_contract.py 1 file already formatted ``` ## Real Behavior Proof - Environment: macOS arm64, Python 3.13.14, uv-managed project environment. - Exact command / steps: resolve the `mcp` extra, inspect the installed SDK version and v1 decorators, then instantiate `HeadroomMCPServer(check_proxy=False)`. - Observed result: `1.28.1 True True`; server construction returns a v1 `Server` successfully. - Not tested: full stdio exchange against every external MCP client; existing MCP unit and CLI suites cover server setup and handlers. ## Runtime Rollout Safety - Rollout-managed feature(s): None; dependency resolution guard. - Minimum rollout channel: Stable/default. - Stable/default behavior changed: Fresh installs stop resolving the incompatible MCP SDK 2.x release. - Kill switch / disable path: Revert the dependency cap after #2658 lands. - Unsafe override required: No. - Qualification impact: MCP extras and proxy installs remain on the maintained MCP 1.x line. - Rollback path: Revert this PR; not recommended until the v2 server port is merged and tested. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable. ## Additional Notes The documentation change is the inline dependency rationale next to the cap. The long-term migration remains #2658; this PR deliberately does not mix that breaking SDK port into the release-blocker rollback. |
||
|
石岳峰
|
b7f342c153
|
fix(wrap): verify proxy deps before mutating Codex config (#1628)
## Description \`headroom wrap codex\` now verifies that optional proxy dependencies (\`headroom-ai[proxy]\`) are installed before mutating Codex \`config.toml\`. If the check fails, the command exits with the same error message as \`headroom proxy\` and leaves Codex config untouched. Fixes #1614 (Bug 1: config mutated before proxy dependency check). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Extract \`ensure_proxy_dependencies()\` in \`headroom/cli/proxy.py\` (shared with \`headroom proxy\`) - Call it at the start of \`wrap codex\` when \`not no_proxy\`, before config snapshot/injection - Add regression tests for prepare-only abort, \`--no-proxy\` skip, and import failure messaging ## Testing - [x] Unit tests pass (\`pytest\`) - [x] Linting passes (\`ruff check .\`) - [ ] Type checking passes (\`mypy headroom\`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output \`\`\`bash pytest tests/test_cli/test_wrap_codex.py::test_wrap_codex_aborts_before_mutating_config_when_proxy_deps_missing \ tests/test_cli/test_wrap_codex.py::test_wrap_codex_skips_proxy_dependency_check_with_no_proxy \ tests/test_cli/test_wrap_codex.py::test_ensure_proxy_dependencies_exits_when_server_import_fails -q # 3 passed ruff check headroom/cli/wrap.py headroom/cli/proxy.py tests/test_cli/test_wrap_codex.py ruff format --check headroom/cli/wrap.py headroom/cli/proxy.py tests/test_cli/test_wrap_codex.py \`\`\` ## Real Behavior Proof Environment: Linux (Ubuntu), Python 3.12, local checkout with \`PYTHONPATH\` pointed at patched sources. Exact command / steps: 1. Created a temp \`~/.codex/config.toml\` with \`model_provider = "openai"\`. 2. Patched \`headroom.cli.wrap.ensure_proxy_dependencies\` to raise \`SystemExit(1)\` (simulating missing \`[proxy]\` extra). 3. Ran \`headroom wrap codex --prepare-only --no-serena --port 8787\`. Observed result: exit code 1; \`config.toml\` unchanged; no \`config.toml.headroom-backup\` created; no \`[mcp_servers.headroom]\` block written. Also verified: \`headroom wrap codex --prepare-only --no-proxy ...\` does not invoke the dependency check. Not tested: Windows-specific proxy selector behavior (covered separately in #1655). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did not edit CHANGELOG.md; release notes are generated automatically --------- Co-authored-by: syf2211 <syf2211@users.noreply.github.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net> |
||
|
Ben Younes
|
9fde127534
|
fix(proxy): relocate stray system-role messages to the top-level system param (#765) (#1357)
## Description On requests large enough to trigger compression, the proxy emitted an upstream Anthropic request whose `messages[0]` had `role: "system"`. Anthropic's Messages API rejects any `system` role inside `messages[]`: ``` 400 invalid_request_error: "messages.0: use the top-level 'system' parameter for the initial system prompt" ``` The original request correctly carries its system prompt in the top-level `system` parameter; a compression/transform/pipeline step relocates the harness system block into `messages[0]`, so the request fails outright (intermittent only because it requires a context large enough to compress). This adds a wire-contract guard in the Anthropic forwarder: as the **last** step before sending upstream (after every transform, memory injection, tool sort, and pipeline extension, covering both the Bedrock and direct paths), any stray `role="system"` message is relocated out of `messages[]` and merged back into the top-level `system` parameter. Content order is preserved (existing system first, relocated content after) and block-level `cache_control` survives. The guard is a no-op on the common path (no system-role entry → inputs pass through unchanged). Closes #765 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/helpers.py`: new pure helper `relocate_system_messages_to_top_level(messages, system) -> (clean_messages, new_system, changed)` plus `_system_message_to_blocks`. Handles `system` being `None`/`str`/`list`, never drops content, preserves order and content blocks. - `headroom/proxy/handlers/anthropic.py`: invoke the guard just before the byte-faithful forward block; on relocation, update `body["messages"]`/`body["system"]`, mark the body mutated (`system_role_relocated`) so the byte-faithful forwarder re-serializes, and log a warning. - `tests/test_proxy_handler_helpers.py`: 3 unit tests (relocate stray system into top-level, append-to-existing-system order, no-op without a system entry). - `CHANGELOG.md`: Bug Fixes entry. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --extra dev python -m pytest tests/test_proxy_handler_helpers.py -q 29 passed in 4.95s # Regression on the forward path (byte-faithful forwarding, system-prompt immutability, cache stability): $ uv run --extra dev python -m pytest tests/test_proxy_handler_helpers.py tests/test_proxy_byte_faithful_forwarding.py tests/test_proxy_system_prompt_immutable.py tests/test_proxy_anthropic_cache_stability.py -q 90 passed, 15 warnings in 29.72s $ uv run ruff check headroom/proxy/helpers.py headroom/proxy/handlers/anthropic.py tests/test_proxy_handler_helpers.py All checks passed! $ uv run ruff format --check ... # 3 files already formatted $ uv run mypy headroom/proxy/helpers.py headroom/proxy/handlers/anthropic.py Success: no issues found in 2 source files ``` ## Test verification (RED → GREEN) The new tests exercise the guard directly and import the new helper at module top, so reverting the production fix makes them fail at collection. **RED — production fix reverted (helper removed):** ```text ImportError while importing test module 'tests/test_proxy_handler_helpers.py'. E ImportError: cannot import name 'relocate_system_messages_to_top_level' from 'headroom.proxy.helpers' =========================== 1 error in 0.41s =============================== ``` **GREEN — production fix applied:** ```text tests/test_proxy_handler_helpers.py ... [100%] ======================= 3 passed, 26 deselected in 1.50s ======================= ``` ## Real Behavior Proof - Environment: Python 3.13, `uv run` in this repo, branch `fix/issue-765`. - Exact command / steps: ran the guard on a body in the exact #765 failure shape — `system: None` and a `role="system"` harness block at `messages[0]`: - Observed result: ```text BEFORE: messages[0].role = system (Anthropic 400 trigger) changed = True AFTER roles = ['user', 'assistant'] system param = [{"type": "text", "text": "You are Claude Code. <system-reminder>...</system-reminder>"}] OK: no role=system in messages[]; system content preserved in top-level param ``` The illegal `role="system"` entry is removed from `messages[]` and its content lands in the top-level `system` parameter — exactly the body Anthropic accepts. - Not tested: a full live 250k+-token Claude Code session against the real Anthropic API (needs a large live context + API key); the fix is validated at the request-shaping boundary the 400 is raised on. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes The guard intentionally fires at the forwarder boundary rather than in any single transform: the issue's captures show the relocation can originate from the compression path, and pipeline extensions / hooks can also mutate `messages` late. Enforcing Anthropic's wire contract once, at the point the body is serialized upstream, fixes the 400 regardless of which step introduced the stray entry and matches the architecture invariant "never produce a `system`-role entry within `messages[]`". --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Sergei Baikin
|
6576ef639c
|
fix(openclaw-plugin): circuit breaker + per-request timeout for proxy resilience (#639)
## Description This change adds bounded timeout and circuit-breaker behavior so OpenClaw can degrade safely when Headroom or the upstream stream stalls, while returning structured proxy errors instead of hanging. Closes #638 by improving OpenClaw/proxy resilience when the Headroom proxy stalls or Anthropic resets a stream. The PR adds proxy-side handling for `httpx.RemoteProtocolError`, returns structured 502 responses for otherwise unhandled proxy middleware errors, and adds OpenClaw plugin timeout/circuit-breaker fallback behavior. ## Type of Change - [x] Bug fix - [ ] New feature - [x] Documentation - [ ] Refactor - [x] Tests only ## Changes Made - Added OpenClaw plugin per-request compression timeout and circuit breaker fallback. - Cleared timeout timers after successful or failed compression so successful calls do not leave pending timers. - Added a focused Vitest regression for timeout cleanup. - Added `contracts.tools` for `headroom_retrieve` without whole-file manifest reformatting. - Added proxy handling for mid-stream `httpx.RemoteProtocolError` and structured 502 fallback behavior. - Documented the new OpenClaw resilience configuration fields. ## Testing - [x] Unit tests - [x] Integration-style proxy tests - [x] Typecheck/build - [ ] Manual testing ### Test Output ```text cd plugins/openclaw && npm test Test Files 6 passed (6), Tests 55 passed (55) cd plugins/openclaw && npm run typecheck passed cd plugins/openclaw && npm run build Build success UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --with pytest --with pytest-asyncio --with fastapi --with httpx --with uvicorn --with h2 python -m pytest tests/test_proxy_streaming_resilience.py -q 24 passed in 2.16s ``` ## Real Behavior Proof - Environment: Windows 11, Node/npm from local plugin worktree, Python 3.13.3, focused local worktree for PR #639. - Exact command / steps: Installed plugin dependencies, ran OpenClaw plugin tests/typecheck/build, and ran the proxy streaming resilience suite with required async/FastAPI/httpx extras. - Observed result: Plugin tests, typecheck, build, and proxy resilience tests all passed. - Not tested: Live OpenClaw gateway session in this pass; original reporter previously verified patched files in a container and OpenClaw degraded/recovered cleanly. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: Sergei Baikin <sergei.baikin@fotograf.de> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net> |
||
|
Abhay Singh
|
82526191a1
|
fix(cli/install): resolve the deployment profile instead of dead-ending on default (#2832)
## Description `headroom init` installs its persistent deployment under a non-`default` profile name (`init-user` for a global-scope install), but every `headroom install <lifecycle>` subcommand hardcodes `--profile default`. The docs show those commands without `--profile`, so on a machine set up by `headroom init` every documented lifecycle command fails while the real deployment is running fine: ```console $ headroom install status Error: No deployment profile named 'default' is installed. $ headroom install status --profile init-user Status: running Healthy: yes ``` The error named neither the installed profile nor the `--profile` flag, so there was nothing to lead the user to `init-user`, which exists only as an internal constant. When the requested profile is not installed, `_require_manifest` now resolves the real target instead of dead-ending on a name the user never chose: 1. an explicit `HEADROOM_DEPLOYMENT_PROFILE` (which the runtime already exports) wins; 2. otherwise, when `--profile` was left at its `default` default and exactly one deployment is installed, that one is used; 3. when it still cannot decide, the error lists the installed profiles and points at `--profile`. This changes only the not-found path. An installed `default` still loads exactly as before, and an explicit typo'd `--profile` still fails, now with a helpful list. Fixes #2811 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/cli/install.py` (`_require_manifest`): on a manifest miss, resolve via `HEADROOM_DEPLOYMENT_PROFILE`, then a single installed deployment when the request is the bare `default`, and otherwise raise an error that lists installed profiles and points at `--profile`. Imported `list_manifests` (already present in `headroom.install.state`) for the enumeration. - `tests/test_cli/test_install_cli.py`: added `test_require_manifest_resolves_single_profile_when_default_missing`, `test_require_manifest_honors_env_profile`, and `test_require_manifest_lists_installed_profiles_when_ambiguous`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # Before/after on the exact reported scenario (one installed profile "init-user"): # ORIGINAL: _require_manifest("default") -> raises "No deployment profile named 'default' is installed." # FIXED: _require_manifest("default") -> resolves to "init-user" # Pass-after, install suites: tests/test_cli/test_install_cli.py 33 passed tests/test_install/ 174 passed, 1 skipped, 1 pre-existing failure # the 1 failure is tests/test_install/test_native_installers.py:: # test_powershell_native_installer_supports_persistent_docker_lifecycle, which # runs scripts/install.ps1 and fails identically on clean main with these changes # stashed (an environment-specific PowerShell exit, unrelated to this diff). # uvx ruff@0.15.17 check -> All checks passed! # uvx mypy@1.20.2 headroom/cli/install.py -> Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1, ruff 0.15.17 and mypy 1.20.2 via uvx. - Exact command / steps: read `detect`/`_require_manifest` and the lifecycle command options (`--profile default` at install.py:720/748/763/775/789/818/830) to confirm the mismatch with `init.py`'s `_GLOBAL_PROFILE = "init-user"`, then demonstrated before/after by monkeypatching `load_manifest`/`list_manifests`: on the original code `_require_manifest("default")` raises "No deployment profile named 'default' is installed."; with the fix it returns the single installed manifest (`init-user`). Fail-before via `git stash push headroom/cli/install.py` and a direct call; pass-after with `git stash pop` and the install suites (33 passed in the CLI file, 174 passed in test_install with one pre-existing environment failure). - Observed result: a bare lifecycle command on an init'd machine now targets the running deployment instead of failing, matching the `--profile init-user` command the issue reporter confirmed works. An explicit `HEADROOM_DEPLOYMENT_PROFILE` selects the target, and an ambiguous multi-profile machine gets an error naming the installed profiles and the `--profile` flag. - Not tested: a full end-to-end `headroom init` then `headroom install status` on a fresh host (that flow spawns a real deployment and supervisor). The resolution logic is a pure function verified directly, and the manifest loading it calls is existing, tested code. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md`: it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes The resolution deliberately only triggers on the not-found path and only auto-selects when a single deployment is installed or an explicit env profile names one, so it never silently picks the wrong deployment on a multi-profile host. The docs that show the bare commands (`docs/content/docs/persistent-installs.mdx`, `wiki/persistent-installs.md`, `wiki/cli.md`) become correct again without needing a `--profile` on every line. |
||
|
Parideboy
|
f1c34d336c
|
fix(proxy/anthropic): don't buffer a CCR stream when passthrough discards the stream flip (#2953)
## Description
Fixes #2952. Since `
|
||
|
Parideboy
|
2d1e96b85c
|
fix(memory): sanitize entity_refs to prevent dict-shaped entries crashing search (#2951)
## Description Closes #2947 `entity_refs` is annotated `list[str]` everywhere, but nothing enforced that at runtime. `LocalBackend.save_memory`'s `entities` argument is filled straight from LLM-supplied `memory_save` tool input (`headroom/memory/system.py:575` into `memory_handler.py:1242`), so a caller can pass the typed `{"entity": ..., "entity_type": ...}` shape, which is the format `extracted_entities` expects, into it by mistake. Those dicts were then persisted verbatim into `entity_refs`, both in the `memories` table and in the duplicated copy the vector index keeps for post-filtering. Every later `search_memories` call does `set().update(memory.entity_refs)` while collecting entities for graph expansion. Hashing a dict raises `TypeError: unhashable type: 'dict'`, and because that happens inside the vector-result loop rather than per-item, **one** poisoned row aborted the **entire** search. The proxy's memory handler catches the exception and returns no memories, so recall went quietly dark rather than failing loudly, and the bad row kept re-appearing in top-k for related queries, so it stayed dark. The issue reporter hit this in production: 4 bad rows disabled memory search for a whole project for a day, with nothing visible to the end user beyond a swallowed warning in `proxy.log`. The same root cause has two more crash modes, both confirmed below: `AttributeError: 'dict' object has no attribute 'lower'` during graph linking on the save path, and the same error in the `entities` search filter (`ref.lower()`). The fix adds one helper and applies it at both ends of the data flow. Dicts are **unwrapped to their `entity` name** rather than dropped, so rows that are already corrupted keep contributing to graph expansion instead of silently losing their entities. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **New helper `normalize_entity_refs()` in `headroom/memory/models.py`.** Coerces a raw entity-reference list into the `list[str]` it claims to be: strings pass through, dicts are unwrapped via their `entity` (or `name`) key, and anything with no recoverable name is dropped rather than stringified, since a ref like `"{'entity_type': 'project'}"` would only pollute the graph. Order is preserved and duplicate names are collapsed. - **Write path, to stop new corruption at the door.** `LocalBackend.save_memory` normalizes `entities` before it reaches `entity_refs` and graph linking. `LocalBackend.search_memories` normalizes the `entities` *filter* argument too, since it arrives from the same untrusted tool input (`memory_handler.py:1320`). - **Read path, to heal rows that were written before this fix.** Applied at the three deserialization boundaries, so no data migration is needed and corrupted rows normalize themselves the next time they are loaded: `Memory.from_dict` (`headroom/memory/models.py`), `SQLiteMemoryStore._row_to_memory` (`headroom/memory/adapters/sqlite.py`), and the vector indexes' own `entity_refs` copies used for post-filtering, `VectorMetadata.from_json` (`headroom/memory/adapters/sqlite_vector.py`) and `IndexedMemoryMetadata.from_dict` (`headroom/memory/adapters/hnsw.py`). - **Defensive normalization on emitted results.** `search_memories` and `text_search` normalize the refs they return as `related_entities`, so a backend that produces `Memory` objects by some path not covered above still cannot take a whole query down, and callers never receive a dict where they expect an entity name. **Note on scope versus the patch proposed in the issue.** The issue proposed normalizing in two places (`save_memory` plus the `set().update()` line). I widened it slightly because that pair leaves three related failures live: the `entities` filter still crashes on `ref.lower()`, `related_entities` still hands dicts back to the caller, and, most importantly, already-poisoned rows stay poisoned in storage. Normalizing at the deserialization boundaries fixes all three at once and is what makes existing corrupted databases recover on their own. **Behavior change worth flagging.** `entity_refs` is now de-duplicated (case-sensitively) on both save and load. Refs were already treated as a set for graph expansion, so this is semantically a no-op, but it is a visible difference if anything asserts on exact list contents. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed New file `tests/test_memory/test_entity_ref_sanitization.py` adds 10 tests covering the helper, both write paths, all three deserialization boundaries, and the three crash modes. ### Test Output ```text $ python -m pytest tests/test_memory/test_entity_ref_sanitization.py -q .......... [100%] 10 passed, 17 warnings in 0.34s ``` Full memory suite, plus a before/after comparison of the failure set to prove no regressions: ```text $ python -m pytest tests/test_memory/ -q 13 failed, 576 passed, 3 skipped, 1072 warnings, 25 errors in 44.74s # the same run with the source changes stashed (baseline on upstream/main @ |
||
|
Parideboy
|
6147883d5e
|
fix(wrap): stop the Serena pre-index stalling the launch path for 300s (#2945)
## Description
`headroom wrap <agent>` could sit silently for a full 300 seconds before
the agent launched, and leaked one orphaned process every time it did.
`_setup_serena_mcp` runs `serena project index` synchronously on the
launch path, with `capture_output=True`, an inherited stdin and
`timeout=300`. When a project has no `.serena/project.yml`, Serena
auto-creates one — and that auto-creation asks one `[y/N]` question per
additionally-detected language server. Three things then combine:
1. stdin was inherited, so Serena believed it could prompt.
2. stdout was captured, so the question never reached the terminal.
3. the call was synchronous, so the agent waited out the entire timeout.
The user saw no prompt, no progress and no error — only a wrapper that
appeared to hang. The pre-index could never succeed in that state, so
the 300 seconds bought nothing.
On top of that, `subprocess.run` kills only its direct child on timeout.
`uvx` is a launcher that execs the real `serena` executable as a
grandchild, which was never signalled: it reparented to PID 1 and
survived indefinitely. Same class of bug as #615 and #880.
Closes #2938
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `_serena_project_skip_reason` (`headroom/cli/wrap.py`) now returns a
skip reason when `.serena/project.yml` is absent, so the pre-index does
not run in the one state where it cannot succeed.
- `_index_serena_project` passes `stdin=subprocess.DEVNULL`, so a
subprocess that decides to prompt gets EOF and exits in about a second
instead of blocking behind a captured pipe. This is deliberately kept as
a second line of defence even though the skip above already avoids the
known prompt.
- `_index_serena_project` now spawns via `subprocess.Popen` in its own
process group (`start_new_session=True` on POSIX,
`CREATE_NEW_PROCESS_GROUP` on Windows) instead of `run(...)`, so the
whole tree can be signalled.
- New `_kill_serena_index_tree` helper kills that tree on timeout —
`killpg(..., SIGKILL)` on POSIX, `taskkill /F /T /PID` on Windows — then
reaps the child and closes the capture pipes. Best-effort throughout; it
never raises.
- Corrected two comments that asserted the opposite of the observed
behaviour ("a failure or timeout here never blocks the wrap", "neither
blocks the wrap"). Both were accurate about intent and wrong about
effect.
- Added `_SERENA_INDEX_TIMEOUT` (still 300) and a line announcing the
pre-index, so a legitimately long index no longer looks like a hang.
- Tests in `tests/test_cli/test_wrap_serena_boost.py` rewritten for the
`Popen` path and extended to cover the DEVNULL stdin, the process-group
flag, the timeout tree-kill, the new skip reason, and the
`_setup_serena_mcp` wiring on both a fresh project and one that already
has `project.yml`.
### Behaviour change worth a reviewer's attention
**On a project with no `.serena/project.yml`, the pre-index no longer
runs at all.** That is the first wrap of any project, so this is the
common case.
I went this way rather than fixing the prompt because there is no way to
fix it from Headroom's side without re-introducing something the project
deliberately removed. Serena's `project index` command has no
non-interactive switch: `ProjectCommands._create_project` calls
`ProjectConfig.autogenerate(..., interactive=True)` with `interactive`
hardcoded. The only path that skips the prompt is passing
`--ls/--language` explicitly, which means Headroom guessing the
project's languages again — exactly the hand-maintained
extension-to-language map that was removed in #2674, with a comment in
this same function explaining why Serena should own that job.
The cost of skipping is small and self-correcting. Serena's MCP server
(`serena start-mcp-server --project-from-cwd`) generates `project.yml`
itself, non-interactively, on first start, and indexes lazily on demand
— which is the fallback the existing docstring already relied on. So the
first wrap now launches immediately with lazy indexing, and every wrap
after that pre-indexes for real. Previously the first wrap cost 300
seconds *and* still produced no index, so nothing of value is lost.
Happy to switch to passing `--ls` instead if maintainers would rather
keep the pre-index on the first wrap and accept a language map; the
other two changes stand either way.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_cli/test_wrap_serena_boost.py tests/test_cli/test_serena_migrate.py -q
collected 29 items
tests\test_cli\test_wrap_serena_boost.py .............s........... [ 86%]
tests\test_cli\test_serena_migrate.py .... [100%]
======================== 28 passed, 1 skipped in 0.54s ========================
$ python -m ruff check headroom/cli/wrap.py tests/test_cli/test_wrap_serena_boost.py
All checks passed!
$ python -m ruff format --check headroom/cli/wrap.py tests/test_cli/test_wrap_serena_boost.py
2 files already formatted
$ python -m mypy headroom/cli/wrap.py --ignore-missing-imports --python-version 3.13 --follow-imports=silent
Success: no issues found in 1 source file
```
The single skip is `test_kill_tree_signals_the_group_on_posix`, which is
platform-gated; the Windows counterpart ran. I develop on Windows, so
the POSIX `killpg` branch is covered by unit test only — the end-to-end
tree-kill proof below is the Windows `taskkill` branch.
## Real Behavior Proof
- Environment: Windows 11 Pro 26200, Python 3.13.11, headroom checkout
at
|
||
|
Ashish Patel
|
41dab2d099
|
fix(ccr): verify a scanned marker's hash before advertising it (#2908)
## Description
`CCRToolInjector.scan_for_markers()` decides whether a compression
marker is Headroom's own by *shape* alone — any bracket marker carrying
a 24-hex hash counts, per the generic fallback pattern
(`\[.*?compressed.*?hash=([a-f0-9]{24})\]`). Other context tools emit
exactly that shape. Once a foreign hash is scanned,
`has_compressed_content` flips true and the retrieve tool + "Available
hashes" system instruction get injected for a hash this proxy never
stored — the model calls `headroom_retrieve`, gets a guaranteed miss,
and re-does work it already had. Two wasted turns per adopted foreign
hash.
Closes #2836
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/ccr/tool_injection.py`: added
`CCRToolInjector.verify_ownership()` — filters `detected_hashes` down to
hashes the compression store actually recognizes, via the same
`store.exists()` check the retrieve endpoint itself performs. Added a
small `_HashOwnershipStore` Protocol (structural typing, not a hard
dependency on the concrete `CompressionStore` class) and a
`compression_store` constructor field for dependency injection/testing.
`scan_for_markers()` itself is untouched — kept store-independent (pure
regex) rather than baking the check into the scan loop, since that
approach broke 24 existing tests that correctly test "does this shape
match" in isolation.
- `headroom/proxy/handlers/anthropic.py`,
`headroom/proxy/handlers/openai.py`: call `injector.verify_ownership()`
right after `scan_for_markers()` — the two real per-request call sites.
- `verify_ownership()` is also called inside `process_request()` (the
convenience wrapper `batch.py`'s Google path uses), so that path is
covered without a separate call site edit.
- `tests/test_ccr_tool_injection.py`: new `TestVerifyOwnership` class (6
tests) — the exact issue repro, a real-hash-survives case, mixed
own/foreign hashes, explicit store override, store-exception safety
(must not raise), and no-op-on-empty-hashes.
- `tests/test_proxy_anthropic_cache_stability.py`: 3 pre-existing tests
needed updating for the new (correct) behavior — two `_FakeInjector`
test doubles needed a `verify_ownership()` stub added, and one real
end-to-end test needed a genuine store entry seeded (via
`explicit_hash`) for the hash its hand-typed marker references, instead
of asserting on an unverified shape-only match.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) — not run locally, will
confirm via CI
- [x] New tests added for new functionality
- [x] Manual testing performed (see Real Behavior Proof)
### Test Output
```text
$ .venv/Scripts/python -m pytest tests/test_ccr_marker_policy.py tests/test_ccr_tool_always_on.py tests/test_ccr_tool_injection.py tests/test_proxy/test_ccr_frozen_prefix_coupling.py tests/test_proxy_anthropic_cache_stability.py tests/test_proxy_handlers_batch.py tests/test_proxy_handler_helpers.py -q
151 passed in 15.87s
$ .venv/Scripts/python -m pytest tests/test_proxy_ccr.py tests/test_proxy_openai_responses_stream_ccr.py tests/test_anthropic_ccr_workspace_unbound.py tests/test_compression_store.py tests/test_no_ccr_lossy.py tests/test_proxy_handlers_batch.py -q
121 passed in 20.73s
$ .venv/Scripts/ruff check . && .venv/Scripts/ruff format --check . # touched files only
All checks passed / already formatted
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.14.5, local venv
- Exact command / steps: ran the issue's exact 3-line repro
(`CCRToolInjector.scan_for_markers()` on the foreign marker text, then
`verify_ownership()`) before and after the fix; separately verified a
genuinely-Headroom-stored hash (via `store.store(...,
explicit_hash=...)`) still survives verification and still drives
injection
- Observed result: before the fix (scan only, no verify step exists yet)
`has_compressed_content` is `True` for the foreign marker — matches the
bug report exactly. After adding `verify_ownership()`: foreign marker →
`detected_hashes == []`, `has_compressed_content is False`; real stored
hash → `detected_hashes == [real_hash]`, `has_compressed_content is
True`.
- Not tested: have not driven this through a live two-context-tool proxy
session (e.g. Headroom alongside another CCR-shaped tool in the same
conversation) — verified at the unit/integration level (the exact repro
plus the real proxy handler call sites via
`test_proxy_anthropic_cache_stability.py`'s end-to-end `TestClient`
tests), not via a live multi-tool session.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A —
internal CCR safety behavior, no user-facing docs reference the
marker-adoption mechanism)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable (release-please
generates this automatically from commit messages)
## Additional Notes
Design note on why `verify_ownership()` is a separate step rather than
baked into `scan_for_markers()`: my first attempt did exactly that and
broke 24 tests across `test_ccr_tool_injection.py`,
`test_ccr_marker_policy.py`, `test_proxy_anthropic_cache_stability.py`,
and `test_proxy_handler_helpers.py` — all of them legitimately testing
"does the regex detect this marker shape" independent of any store
state. Keeping the scan pure and adding an explicit, separately-testable
verification step kept that test surface intact while still closing the
real gap at the three places that actually decide whether to advertise
the retrieve tool.
|
||
|
Pragadeesh
|
d76fce04a3
|
fix(proxy): adapt 200 SSE upstream replies on buffered /v1/responses instead of 502 (#2622)
## Description The buffered HTTP `/v1/responses` path (`_buffered_ccr_operation` in `headroom/proxy/handlers/openai.py`) assumed every upstream reply to a `stream: false` request is JSON. Some OpenAI-compatible upstreams answer with a valid `200 text/event-stream` body carrying Responses API events. `response.json()` raised `JSONDecodeError`, which is not in the narrow usage-extraction catch (`KeyError, TypeError, AttributeError`), so it escaped to the outer handler and the **successful** upstream reply was converted into a generic `502 proxy_error` — the client loses the response and typically retries, duplicating paid calls. The fix classifies the upstream reply at the ingestion boundary by its declared `Content-Type` (the SSE spec's own discriminator) instead of parsing by expectation: - **200 SSE with a terminal `response.completed` event** → the complete response object is reassembled from that event (`_openai_responses_from_sse`, the inverse of the existing `_openai_responses_to_sse`) and swapped in as a synthesized `application/json` response *before any parsing happens*. Everything downstream — usage extraction, CCR retrieval handling, memory-tool handling — runs unmodified. - **200 SSE without a recognizable terminal event** → the successful upstream body is forwarded to the client unchanged (sanitized headers) rather than fabricating a 502. Adapt only when the adaptation is provably faithful; otherwise pass through. - **Everything else** (normal JSON replies, non-200s) → byte-identical pre-existing behavior. Deliberately *not* done: widening the `except` clause (would leave `resp_json` unbound and break the downstream pipeline) and body sniffing (the declared media type is trusted; a mislabeled body keeps today's behavior). Closes #2613 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/handlers/openai.py`: new module-level helper `_openai_responses_from_sse()` (SSE-spec framing: blank-line event separation, multi-line `data:` joining, `\r` tolerance, at most one stripped space, `[DONE]` skipped; returns the terminal event's `response` object or `None`), placed next to its inverse `_openai_responses_to_sse()`. - `headroom/proxy/handlers/openai.py::_buffered_ccr_operation()`: content-type dispatch for 200 replies immediately after the upstream response (and after wire-debug capture, so debug logs keep the true upstream bytes) — adapt SSE→JSON when a terminal event exists, pass through unchanged when it doesn't. - `tests/test_openai_codex_routing.py`: two new handler-level tests (see below). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # Both new tests watched failing BEFORE the fix with the exact issue signature: # ERROR headroom.proxy:openai.py [req-1] OpenAI responses request failed: JSONDecodeError: Expecting value: line 1 column 1 (char 0) # assert 502 == 200 $ pytest tests/test_openai_codex_routing.py -q 24 passed in 2.08s $ pytest tests/test_openai_codex_routing.py tests/test_ccr_response_handler_openai_responses.py tests/test_codex_responses_passthrough_bytes.py -q 38 passed, 1 warning in 13.80s $ pytest tests/test_output_shaper_responses.py tests/test_codex_responses_waste_signals.py tests/test_codex_openai_contract_parity.py tests/test_openai_beta_session_sticky.py tests/test_openai_max_completion_tokens.py tests/test_openai_response_cache_key.py tests/test_litellm_openai_passthrough.py -q 61 passed, 1 warning $ ruff check headroom/proxy/handlers/openai.py tests/test_openai_codex_routing.py All checks passed! $ mypy headroom/proxy/handlers/openai.py Success: no issues found in 1 source file ``` New tests: - `test_handle_openai_responses_non_stream_adapts_sse_upstream` — 200 SSE with `response.completed` → client gets 200 `application/json` with the reassembled response. - `test_handle_openai_responses_non_stream_passes_through_unparseable_sse` — 200 SSE with no terminal event → client gets 200 with the body unchanged, never a 502. ## Real Behavior Proof - Environment: macOS 26.5 (arm64), Python 3.12 via `uv`, headroom from source (editable install). Local fake OpenAI-compatible upstream (`http.server`) that answers every `POST` with `200 text/event-stream` containing a `response.completed` event and `data: [DONE]` — the upstream behavior reported in the issue. Proxy started with `OPENAI_TARGET_API_URL=http://127.0.0.1:9302 headroom proxy --port <port>`. - Exact command / steps: same-session A/B against real proxy processes — identical upstream and identical request, only the checked-out revision changed: ```bash curl -s -w "\nHTTP_STATUS=%{http_code} CONTENT_TYPE=%{content_type}\n" \ -X POST http://127.0.0.1:<port>/v1/responses \ -H "content-type: application/json" -H "authorization: Bearer sk-test" \ -d '{"model":"gpt-5.4","stream":false,"input":"hello"}' ``` - Observed result: unpatched `main` converts the successful upstream reply into the issue's 502; this branch returns the complete response as JSON. Full captures: **Before (unpatched `main`, port 8794):** ``` {"error":{"message":"An error occurred while processing your request. Please try again.","type":"server_error","code":"proxy_error"}} HTTP_STATUS=502 ``` **After (this branch, port 8795):** ``` {"id": "resp_sse_repro", "object": "response", "status": "completed", "model": "gpt-5.4", "output": [{"type": "message", "id": "msg_1", "role": "assistant", "content": [{"type": "output_text", "text": "hello from sse upstream"}]}], "usage": {"input_tokens": 2, "output_tokens": 1}} HTTP_STATUS=200 CONTENT_TYPE=application/json ``` - Not tested: a wild third-party SSE-answering upstream (the repro uses a local stub shaped per the issue report); the buffered-stream-CCR variant of this path against a live upstream (unit-tested only); Windows. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes - Documentation checklist item is N/A — internal proxy behavior fix, no documented surface changes. - Known residual (pre-existing, out of this issue's scope): a **non-200** upstream reply with a non-JSON body (an SSE error stream, a gateway HTML error page) still follows the old `JSONDecodeError → 502` path, blurring a meaningful upstream error into a generic 502. This PR deliberately adapts only declared-SSE **200** replies, where reassembly from `response.completed` is provably faithful. Happy to file the non-200 case as a follow-up issue if maintainers want it tracked. |
||
|
Abhay Singh
|
d6d121e399
|
fix(ccr): re-inject headroom_retrieve when history references it on the sessionless path (#2440) (#2533)
## Description Fixes #2440. `apply_session_sticky_ccr_tool` bypasses the `SessionCcrTracker` when `session_id` is `None` (WS / pre-session paths) and drives injection purely off the per-turn `has_compressed_content_this_turn` flag: ```python if not session_id: if not has_compressed_content_this_turn: ... # skip: tool NOT re-declared return tools_out, False ... ``` If an earlier turn emitted a `headroom_retrieve` tool_use into history but the current turn produced no fresh compression marker, the tool definition is not re-declared in `tools`, while the forwarded history still references it. The provider then rejects the whole request: ``` API Error: 400 Tool reference 'headroom_retrieve' not found in available tools. ``` Without a session the tracker can't remember the earlier turn's CCR, so this is unique to the sessionless path. ## Fix Add `history_references_ccr_tool(messages)` which detects an existing `headroom_retrieve` call in the forwarded messages — both the Anthropic assistant `tool_use` content block and the OpenAI assistant `tool_calls[].function.name` shapes, fully null-guarded. On the sessionless path, injection now fires when `has_compressed_content_this_turn` **or** history already references the tool, so the definition is re-declared and the request validates. The decision is logged as a new `inject_history_reference` outcome. Both handlers pass the signal computed from `optimized_messages` (the bytes actually forwarded). Behavior with a real `session_id` (the sticky tracker path) is unchanged. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/helpers.py`: add `history_references_ccr_tool`; add a `history_has_ccr_reference` parameter to `apply_session_sticky_ccr_tool` and OR it into the sessionless injection decision. - `headroom/proxy/tool_injection_logging.py`: add the `inject_history_reference` decision literal. - `headroom/proxy/handlers/anthropic.py`, `headroom/proxy/handlers/openai.py`: pass `history_references_ccr_tool(optimized_messages)` into the sticky-tool call. - `tests/test_ccr_tool_always_on.py`: regressions for the detector (both provider shapes + malformed inputs) and for sessionless re-injection when history references the tool. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_ccr_tool_always_on.py -q 14 passed # with just the `or history_has_ccr_reference` condition reverted, the new # sessionless re-injection test fails (tool not injected -> would 400) $ uvx ruff@0.15.17 check headroom/proxy/helpers.py headroom/proxy/tool_injection_logging.py headroom/proxy/handlers/anthropic.py headroom/proxy/handlers/openai.py tests/test_ccr_tool_always_on.py All checks passed! $ uvx mypy@1.20.2 --ignore-missing-imports headroom/proxy/helpers.py headroom/proxy/tool_injection_logging.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, project venv (`uv sync --extra proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`, pytest in the venv. - Exact command / steps: called `history_references_ccr_tool` on Anthropic `tool_use` and OpenAI `tool_calls` histories (plus null/non-list shapes), and `apply_session_sticky_ccr_tool(session_id=None, has_compressed_content_this_turn=False, history_has_ccr_reference=True)`; then temporarily reverted only the `or history_has_ccr_reference` condition and re-ran the regression. - Observed result: the detector returns `True` for both provider shapes and `False`/no-crash for malformed input; with the fix the sessionless call injects the tool (`was_injected=True`, tool present) even with no fresh compression; with the condition reverted the same call returns `was_injected=False` (the tool is dropped — exactly the 400 path). Ran against the actual module via `tests/test_ccr_tool_always_on.py`. - Not tested: a live sessionless multi-turn WS request reproducing the upstream 400 end to end. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net> |
||
|
dependabot[bot]
|
b30f339d69
|
deps: bump criterion from 0.5.1 to 0.8.2 (#2965)
Bumps [criterion](https://github.com/criterion-rs/criterion.rs) from 0.5.1 to 0.8.2. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/criterion-rs/criterion.rs/releases">criterion's releases</a>.</em></p> <blockquote> <h2>criterion-plot-v0.8.2</h2> <h3>Other</h3> <ul> <li>Update Readme</li> </ul> <h2>criterion-v0.8.2</h2> <h3>Fixed</h3> <ul> <li>don't build alloca on unsupported targets</li> </ul> <h3>Other</h3> <ul> <li><em>(deps)</em> bump crate-ci/typos from 1.40.0 to 1.43.0</li> <li>Fix panic with uniform iteration durations in benchmarks</li> <li>Update Readme</li> <li>Exclude development scripts from published package</li> </ul> <h2>criterion-plot-v0.8.1</h2> <h3>Fixed</h3> <ul> <li>Typo</li> </ul> <h2>criterion-v0.8.1</h2> <h3>Fixed</h3> <ul> <li>Homepage link</li> </ul> <h3>Other</h3> <ul> <li><em>(deps)</em> bump crate-ci/typos from 1.23.5 to 1.40.0</li> <li><em>(deps)</em> bump jontze/action-mdbook from 3 to 4</li> <li><em>(deps)</em> bump actions/checkout from 4 to 6</li> </ul> <h2>criterion-plot-v0.8.0</h2> <p>No release notes provided.</p> <h2>criterion-v0.8.0</h2> <h3>BREAKING</h3> <ul> <li>Drop async-std support</li> </ul> <h3>Changed</h3> <ul> <li>Bump MSRV to 1.86, stable to 1.91.1</li> </ul> <h3>Added</h3> <ul> <li>Add ability to plot throughput on summary page.</li> <li>Add support for reporting throughput in elements and bytes - <code>Throughput::ElementsAndBytes</code> allows the text summary to report throughput in both units simultaneously.</li> <li>Add alloca-based memory layout randomisation to mitigate memory effects on measurements.</li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/criterion-rs/criterion.rs/blob/master/CHANGELOG.md">criterion's changelog</a>.</em></p> <blockquote> <h2><a href="https://github.com/criterion-rs/criterion.rs/compare/criterion-v0.8.1...criterion-v0.8.2">0.8.2</a> - 2026-02-04</h2> <h3>Fixed</h3> <ul> <li>don't build alloca on unsupported targets</li> </ul> <h3>Other</h3> <ul> <li><em>(deps)</em> bump crate-ci/typos from 1.40.0 to 1.43.0</li> <li>Fix panic with uniform iteration durations in benchmarks</li> <li>Update Readme</li> <li>Exclude development scripts from published package</li> </ul> <h2><a href="https://github.com/criterion-rs/criterion.rs/compare/criterion-v0.8.0...criterion-v0.8.1">0.8.1</a> - 2025-12-07</h2> <h3>Fixed</h3> <ul> <li>Homepage link</li> </ul> <h3>Other</h3> <ul> <li><em>(deps)</em> bump crate-ci/typos from 1.23.5 to 1.40.0</li> <li><em>(deps)</em> bump jontze/action-mdbook from 3 to 4</li> <li><em>(deps)</em> bump actions/checkout from 4 to 6</li> </ul> <h2><a href="https://github.com/criterion-rs/criterion.rs/compare/criterion-v0.7.0...criterion-v0.8.0">0.8.0</a> - 2025-11-29</h2> <h3>BREAKING</h3> <ul> <li>Drop async-std support</li> </ul> <h3>Changed</h3> <ul> <li>Bump MSRV to 1.86, stable to 1.91.1</li> </ul> <h3>Added</h3> <ul> <li>Add ability to plot throughput on summary page.</li> <li>Add support for reporting throughput in elements and bytes - <code>Throughput::ElementsAndBytes</code> allows the text summary to report throughput in both units simultaneously.</li> <li>Add alloca-based memory layout randomisation to mitigate memory effects on measurements.</li> <li>Add doc comment to benchmark runner in criterion_group macro (removes linter warnings)</li> </ul> <h3>Fixed</h3> <ul> <li>Fix plotting NaN bug</li> </ul> <h3>Other</h3> <ul> <li>Remove Master API Docs links temporarily while we restore the docs publishing.</li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
d6fb5365f6
|
deps: update mcp requirement from <2.0.0,>=1.28.1 to >=1.28.1,<3.0.0 (#2963)
Updates the requirements on [mcp](https://github.com/modelcontextprotocol/python-sdk) to permit the latest version. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/modelcontextprotocol/python-sdk/releases">mcp's releases</a>.</em></p> <blockquote> <h2>v2.0.0</h2> <h1>MCP Python SDK v2 Stable Release</h1> <p>This is v2.0.0, the stable v2 release of the MCP Python SDK. It supports the 2026-07-28 revision of the Model Context Protocol and serves every earlier revision from the same server. <code>pip install mcp</code> now installs 2.x.</p> <pre lang="bash"><code>pip install "mcp[cli]" # or uv add "mcp[cli]" </code></pre> <h3>Documentation Rewrite</h3> <p>The <a href="https://py.sdk.modelcontextprotocol.io/">documentation</a> has the full tutorial and API reference. Coming from v1? <a href="https://py.sdk.modelcontextprotocol.io/whats-new/">What's new in v2</a> is the tour of what changed and why, and the <a href="https://py.sdk.modelcontextprotocol.io/migration/">migration guide</a> lists every breaking change with before-and-after code.</p> <h3>V1 Maintenance mode</h3> <p><strong>v1.x is in maintenance mode and will only receive security fixes from now on</strong> The 1.x line lives on the <a href="https://github.com/modelcontextprotocol/python-sdk/tree/v1.x"><code>v1.x</code> branch</a>, continues to receive critical bug fixes and security patches, and is documented at <a href="https://py.sdk.modelcontextprotocol.io/v1/">https://py.sdk.modelcontextprotocol.io/v1/</a>. If your project is not ready to migrate, keep a <code><2</code> upper bound on your requirement (for example <code>mcp>=1.28,<2</code>).</p> <h2>Highlights</h2> <h3>One SDK, both protocol eras</h3> <p>v2 speaks the 2026-07-28 revision (stateless requests with no handshake, <code>server/discover</code>, <code>subscriptions/listen</code>, multi-round-trip requests) and still serves every 2025-era client from the same <code>MCPServer</code>, over Streamable HTTP and stdio, with nothing to configure. <code>Client(target)</code> negotiates the version automatically.</p> <h3><code>FastMCP</code> is now <code>MCPServer</code>, and there is a first-class <code>Client</code></h3> <p>The decorator API is unchanged; the low-level <code>Server</code> is rebuilt around a shared dispatcher engine, and one <code>Client</code> object replaces v1's transport-plus-<code>ClientSession</code>-plus-<code>initialize()</code> layering. It connects to a URL, a stdio subprocess, a custom transport, or straight to a server object in memory for tests.</p> <h3>Multi-round-trip requests and resolver dependency injection</h3> <p>At 2026-07-28 the server can no longer call the client, so tools return the question instead. A <code>Resolve(fn)</code> parameter is filled by your function invisibly to the model and can put a question to the user; one tool body serves both eras.</p> <h3>Extension APIs, OpenTelemetry, and a standalone types package</h3> <p>Servers and clients compose protocol extensions through pluggable extension APIs (MCP Apps built in); OpenTelemetry tracing ships on by default; every protocol type is its own package, <code>mcp-types</code> (imported as <code>mcp_types</code>), published in lock-step with <code>mcp</code>.</p> <h3>Hardened stdio and auth</h3> <p>stdio servers keep handler subprocesses and stray prints off the wire, and stdout is diverted to stderr while serving. OAuth adds RFC 9207 issuer validation, the SEP-990 identity-assertion flow, and the client-credentials extension.</p> <h2>Coming from a v2 pre-release</h2> <p>Since the last release candidate: the per-version wire packages are private (<code>mcp_types._v*</code>), <code>mcp.types</code> is a permanent alias for <code>mcp_types</code>, the auth registration request model is split from the registered-client record, cancelled requests are no longer answered, and log notifications are gated on the per-request log-level opt-in at 2026-07-28. Since the betas: <code>Client(cache=False)</code> is now <code>cache=None</code> with <code>CacheConfig()</code> the default; <code>Context.client_id</code>, <code>RFC7523OAuthClientProvider</code>, and <code>OAuthClientProvider(timeout=)</code> are removed; the client-credentials providers take <code>scope=</code>; <code>message_handler</code> receives notifications and exceptions only; <code>FileResource(is_binary=)</code> becomes <code>encoding</code>; <code>MCP_*</code> env vars are gone with <code>pydantic-settings</code>; Streamable HTTP servers reject bodies over 4 MiB with HTTP 413. The migration guide covers all of it.</p> <h2>Known gaps</h2> <p>The tasks extension (SEP-2663) is not part of this release. On the client, the DPoP proof binding (SEP-1932) and the workload-identity <code>jwt-bearer</code> grant are not implemented; both are additive and can land in 2.x.</p> <h2>Feedback</h2> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
JD Davis
|
e269afb935
|
fix(ci): unjam release and Docker publishing (#2958)
## Description
Fixes two release-automation defects exposed by the 0.35.0 release:
1. Release Please grouped the single package into a PR titled `chore:
release main`, which could not be matched back to the `headroom-ai`
component/version and therefore never emitted the release event that
starts PyPI publishing.
2. Docker manifest jobs downloaded digest artifacts with overlapping
variant globs. For example, `digests-code-*` also selected code-nonroot,
code-slim, and code-slim-nonroot artifacts, yielding eight markers where
exactly two were required.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Route the single root package through Release Please's normal
versioned-PR path.
- Preserve both `${component}` and `${version}` in generated release PR
titles.
- Download Docker amd64 and arm64 digest artifacts by exact name instead
of an overlapping variant glob.
- Add regression assertions for both Release Please title matching and
Docker artifact isolation.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
python -m pytest tests/test_release_workflows.py -q
44 passed in 2.29s
pre-commit hooks:
Sync plugin versions........................Passed
Verify Ruff version alignment...............Passed
check for merge conflicts...................Passed
ruff (legacy alias).........................Passed
ruff format.................................Passed
mypy........................................Passed
```
## Real Behavior Proof
- Environment: GitHub Actions release runs 31653073583 and 31664975515,
`main` at the 0.35.0 release merge (`93f2d7a2`).
- Exact command / steps: Inspected the Release Please rerun and each
failed Docker manifest job; enumerated the digest artifacts downloaded
by their configured patterns.
- Observed result: Release Please logged `There are untagged, merged
release PRs outstanding - aborting`. Docker's `slim`, `code`, and
`code-slim` manifest cells found 4, 8, and 4 digest markers respectively
instead of 2 because their prefix globs included related variants. The
new Docker workflow requests `digests-<variant>-amd64` and
`digests-<variant>-arm64` by exact name.
- Not tested: A synthetic release was not published because registry
versions/tags are irreversible. Both configuration invariants are
covered by regression tests, and GitHub's PR workflow validation runs
against this branch.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A — workflow configuration and regression-test changes only.
## Additional Notes
- Documentation is N/A: these are internal release workflow corrections
with no user-facing command or API changes.
- The 0.35.0 Python release recovery is proceeding separately through
the existing wheel smoke-test and PyPI publication gates.
- The already-started 0.35.0 Docker run used the old workflow from tag
commit `
|
||
|
JD Davis
|
3077ac81e8
|
feat: add deterministic runtime rollout controls (#1490)
## Description Establish one centrally resolved, observable, deterministic, versioned runtime rollout-control mechanism for Headroom. Runtime rollout controls which behaviors an already-built artifact may expose; it does not select or qualify a Headroom release/version. ## Type of Change - [x] New feature (non-breaking change that adds functionality) - [x] Bug fix (non-breaking change that fixes rollout enforcement regressions) - [x] Documentation update - [x] Code refactoring (no functional changes) ## Changes Made - Added `RolloutChannel`, `HEADROOM_ROLLOUT_CHANNEL`, `--rollout-channel`, and a versioned immutable `RolloutSnapshot` shared by Python configuration boundaries. - Added schema/policy versions, canonical registry and snapshot SHA-256 identities, per-feature decision reasons, disable precedence, unsafe qualification poisoning, strict CLI validation, and fail-closed environment handling. - Added `headroom rollout status --json`, Python `/stats.rollout`, and Rust `/rollout/status` runtime provenance. - Added equivalent Rust snapshot semantics and shared Python/Rust policy vectors while retaining language-specific feature registries. - Enforced rollout policy at alternate Python server composition roots so `HEADROOM_READ_MATURATION=1` cannot bypass its beta gate. - Preserved typed rollout snapshots across multi-worker serialization with schema, policy, registry, snapshot-digest, type, and feature-name validation. - Made loopback runtime output-shaper updates replace the immutable snapshot atomically for request readers, retain explicit request/disable provenance, preserve channel and kill-switch precedence, invalidate cached stats, and return the effective rollout decision. - Made `headroom learn --verbosity --apply` report a channel-blocked update instead of claiming the shaper is live. - Made explicit CLI feature flags fail loudly when their current channel blocks them. - Made persistent interceptor installation select canary automatically, or reject an explicitly insufficient channel unless the break-glass override is set. - Updated architecture, proxy, rollout, learn, and output-shaper documentation with required channels and hot-reload semantics. ## Testing - [x] Unit tests pass - [x] Linting passes (`ruff check .` and `ruff format --check .`) - [x] Type checking passes (`mypy headroom --ignore-missing-imports`) - [x] New regression tests added for every corrected behavior - [x] Rust tests and production-target Clippy pass - [x] Documentation build passes ### Test Output ```text Focused rollout coverage suite 57 passed; headroom.rollout + rollout CLI: 98% coverage Affected proxy/rollout/transform/governance suites 222 passed; 0 failed Final changed regression suites 100 passed; 0 failed Cross-module hot-reload isolation regression 6 passed; 0 failed cargo test -p headroom-core -p headroom-proxy --quiet headroom-core: 924 passed; 1 ignored headroom-proxy and integration suites: all passed cargo clippy -p headroom-core -p headroom-proxy --lib --bins -- -D warnings cargo fmt --all -- --check ruff check . ruff format --check . mypy headroom --ignore-missing-imports git diff --check All passed cd docs && npm run build Compiled successfully; 164 static pages generated ``` The unsharded Windows-only CI selection exposed unrelated baseline failures, principally the existing `sqlite:///C:\\...` URL parser producing an invalid `\\C:\\...` path. At commit ` |
||
|
Tejas Chopra
|
93f2d7a2da
|
chore: release main (#2792)
🤖 I have created a release *beep* *boop* --- <details><summary>0.35.0</summary> ## [0.35.0](https://github.com/headroomlabs-ai/headroom/compare/v0.34.0...v0.35.0) (2026-08-12) ### Features * **beacon:** allowlist the routing summary key ([#2818](https://github.com/headroomlabs-ai/headroom/issues/2818)) ([ |
||
|
dependabot[bot]
|
148d8605e2
|
deps: bump the cargo-minor-patch group across 1 directory with 22 updates (#2916)
Bumps the cargo-minor-patch group with 21 updates in the / directory: | Package | From | To | | --- | --- | --- | | [serde_json](https://github.com/serde-rs/json) | `1.0.150` | `1.0.151` | | [thiserror](https://github.com/dtolnay/thiserror) | `2.0.18` | `2.0.19` | | [anyhow](https://github.com/dtolnay/anyhow) | `1.0.103` | `1.0.104` | | [clap](https://github.com/clap-rs/clap) | `4.6.2` | `4.6.6` | | [tokio](https://github.com/tokio-rs/tokio) | `1.52.3` | `1.53.1` | | [pyo3](https://github.com/pyo3/pyo3) | `0.29.0` | `0.29.2` | | [aws-config](https://github.com/smithy-lang/smithy-rs) | `1.9.0` | `1.10.1` | | [aws-smithy-runtime-api](https://github.com/smithy-lang/smithy-rs) | `1.13.0` | `1.14.0` | | [unidiff](https://github.com/messense/unidiff-rs) | `0.4.0` | `0.4.1` | | [aho-corasick](https://github.com/BurntSushi/aho-corasick) | `1.1.4` | `1.1.5` | | [toml](https://github.com/toml-rs/toml) | `1.1.3+spec-1.1.0` | `1.1.4+spec-1.1.0` | | [blake3](https://github.com/BLAKE3-team/BLAKE3) | `1.8.5` | `1.8.6` | | [http](https://github.com/hyperium/http) | `1.4.2` | `1.5.0` | | [futures](https://github.com/rust-lang/futures-rs) | `0.3.32` | `0.3.33` | | [hyper](https://github.com/hyperium/hyper) | `1.10.1` | `1.11.0` | | [bytesize](https://github.com/bytesize-rs/bytesize) | `2.4.2` | `2.7.0` | | [tokio-util](https://github.com/tokio-rs/tokio) | `0.7.18` | `0.7.19` | | [lru](https://github.com/jeromefroe/lru-rs) | `0.18.1` | `0.18.2` | | [async-trait](https://github.com/dtolnay/async-trait) | `0.1.89` | `0.1.91` | | [tokio-stream](https://github.com/tokio-rs/tokio) | `0.1.18` | `0.1.19` | | [cc](https://github.com/rust-lang/cc-rs) | `1.2.67` | `1.4.1` | Updates `serde_json` from 1.0.150 to 1.0.151 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/serde-rs/json/releases">serde_json's releases</a>.</em></p> <blockquote> <h2>v1.0.151</h2> <ul> <li>Add RawValue::from_string_unchecked (<a href="https://redirect.github.com/serde-rs/json/issues/1331">#1331</a>, thanks <a href="https://github.com/WonderLawrence"><code>@WonderLawrence</code></a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
Parideboy
|
3752458022
|
fix(cache): enforce Anthropic's 1h-before-5m cache_control ordering before forwarding (#2941)
## Description
Anthropic evaluates prompt-cache breakpoints in **one pass over the
whole request** — `tools`, then `system`, then `messages` — and rejects
the request outright when a `ttl='1h'` breakpoint appears after a
5-minute one. A bare `{"type": "ephemeral"}` marker counts as 5 minutes,
so this is easy to trip without any `ttl` field being visibly wrong:
```
API Error: 400 messages.15.content.1.cache_control.ttl: a ttl='1h' cache_control block must not
come after a ttl='5m' cache_control block. Note that blocks are processed in the following order:
tools, system, messages.
```
Headroom rewrites `cache_control` markers in several independent places,
each looking at one section, and nothing checked the invariant that
spans them. The failure mode is a dead turn, not a silent cost
regression.
Two paths can leave the forwarded body illegal today:
1. **Replayed 1h marker in a 5m request.** Claude Code picks its TTL
lane per request, not per session: the main loop asks for 1h and sends
the `extended-cache-ttl` beta header, while a side question (`/btw` in
the report) goes out in the 5m lane with bare markers and no beta
header. Headroom replays part of the previous turn's forwarded bytes
into `messages` to keep the prefix stable, and those bytes still carry
`ttl: "1h"`. `tools`/`system` at 5m, `messages` at 1h — 400.
2. **Tools breakpoint downgraded.** `inject_tool_search_deferral`
re-places the *last* marker it stripped, so a bare marker on a later
deferred tool overwrites a `ttl='1h'` one. The tools prefix goes
upstream at 5m while message breakpoints are still 1h — 400. Reported
separately as #2767.
The rule spans `tools`/`system`/`messages`, so no individual transform
is in a position to check it. The fix is a guard at the last seam before
the body goes on the wire, plus the one Python/Rust divergence that
manufactures the violation upstream of it.
Closes #2939
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/helpers.py`: new `enforce_cache_control_ttl_order`,
plus `cache_control_ttl_lane` / `cache_control_ttl_lanes` /
`walk_cache_control`. The walk visits markers in Anthropic's documented
order and matches the traversal `count_cache_breakpoints` already
performs, so the two cannot disagree about what counts as a breakpoint.
TTL ranking is ported verbatim from the Rust `TtlOrderingWalk::observe`:
absent or `"5m"` is short, `"1h"` is long, any other value is left alone
rather than guessed at. Two repairs:
- **Lane containment** — when the client sent no 1h marker of its own,
strip `ttl` from any 1h marker that leaked in. That request never sent
the `extended-cache-ttl` beta header, so it could not have written a 1h
entry anyway; nothing is lost. Other marker fields (`scope`, …) are
preserved.
- **Ordering** — when the client did ask for 1h, promote every 5m marker
preceding the last 1h one. Demoting would also make the request legal
but would discard 1h caching the client is explicitly paying for, which
is the regression #2375 / #2382 / #2651 were filed to stop. A violation
seen on the way out means headroom downgraded or introduced a marker, so
promoting restores what the client's own (legal) request asked for at
that position.
- Copy-on-write: a legal body is returned by identity, so the hot path
pays only a walk over at most a handful of markers. Kill switch
`HEADROOM_CACHE_CONTROL_TTL_GUARD=0`, matching the
`HEADROOM_TOOL_SEARCH=0` convention.
- `headroom/proxy/handlers/anthropic.py`:
- Capture the client's TTL lane from the inbound snapshot, before any
transform runs. This cannot be inferred from the session or from config
— the lane is a per-request property of Claude Code, which is the whole
reason the `/btw` case exists.
- Call the guard immediately before `log_cache_breakpoints`, i.e. after
every transform, the tool sort, the deferral, CCR injection and the
pipeline extensions. Mark the body mutated when a repair fires, and log
a WARNING carrying the repair kind, the counts and the offending
sections — the diagnostic the next report of this class will need.
- `_sort_tools_deterministically` now skips the sort when any tool
carries `cache_control`, logging `event=tool_sort_skipped
reason=marker_present`. A breakpoint on a tool means "cache through
here", so reordering changes what is inside the cached prefix and can
move a 1h-marked tool behind a 5m-marked one. The Rust proxy already
refuses for exactly this reason (`any_tool_has_cache_control` in
`crates/headroom-proxy/src/compression/live_zone_anthropic.rs:651`); the
Python path never got the same guard. Putting the check in
`_sort_tools_deterministically` rather than `_tools_for_forwarding`
covers all call sites including the batch path.
- `tests/test_cache_control_ttl_order.py`: new, 24 cases.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
The new tests validate against an independent reimplementation of
Anthropic's rule rather than against the guard's own walk, so a bug in
the walk cannot make the assertions pass. Coverage: lane classification
(bare marker is 5m, unknown TTLs are `other`); containment of a replayed
1h marker including preservation of non-`ttl` fields; promotion across
`tools`→`messages`, `system`→`messages` and within `messages`; markers
nested in `tool_result` sub-blocks; only markers before the *last* 1h
one are rewritten, so a legal 1h-then-5m ordering is left alone; legal
bodies returned by `is` identity; unknown `ttl` untouched; kill switch;
the tool sort skipping on marked tools and still sorting unmarked ones,
with one test pinning that the sort *would* have created a violation
without the guard; and an end-to-end regression running
`inject_tool_search_deferral` then the guard on the #2767 shape.
### Test Output
```text
$ pytest tests/test_cache_control_ttl_order.py -q
24 passed in 0.77s
$ pytest tests/test_cache_ttl_preserved.py tests/test_cache_control_move_bust.py \
tests/test_cache_breakpoint_diagnostics.py tests/test_issue_746_tool_search.py -q
86 passed in 1.36s
$ pytest tests/test_cache/ tests/test_proxy/ -q
16 failed, 489 passed, 2 skipped in 91.41s
$ ruff check headroom/ tests/test_cache_control_ttl_order.py
All checks passed!
$ ruff format --check headroom/ tests/test_cache_control_ttl_order.py
1 file would be reformatted, 520 files already formatted
$ mypy --python-version 3.13 headroom --ignore-missing-imports
Found 12 errors in 3 files (checked 517 source files)
```
The three non-green results above are all pre-existing on a clean
`upstream/main` in this environment, verified by stashing the changes
and re-running:
- The 16 failures are all in
`tests/test_cache/test_client_integration.py` and are a Windows
temp-path problem in this sandbox (`OSError: [WinError 123] ...
'\\C:\\Users\\...\\Temp'`), not a code failure. They fail identically
with the branch stashed.
- `ruff format --check` flags `headroom/testing/README.md`, a docs code
block untouched by this PR.
- The mypy errors are in `headroom/memory/mcp_server.py` and
`headroom/release_version.py`; none are in the files this PR changes.
`--python-version 3.13` is needed locally because the pinned
`python_version = "3.10"` makes mypy reject the installed numpy stubs
before it checks anything.
## Real Behavior Proof
- Environment: Windows 11, Python 3.13.11, headroom at this branch's
head, run against the real `headroom.proxy` forwarding helpers. No
Anthropic API key is available in this environment, so Anthropic's
validator is reimplemented locally from its documented rule and its own
error string; the request bodies are produced by the real code path
(`_sort_tools_deterministically` then `inject_tool_search_deferral` then
the guard), not hand-written.
- Exact command / steps: build two request shapes — (A) a 5m-lane
request whose `messages` carries a replayed `ttl:"1h"` marker, the
`/btw` case; (B) 13 tools with markers on two deferred tools plus 1h
message breakpoints, the #2767 case — push each through the forwarding
helpers twice, once with `HEADROOM_CACHE_CONTROL_TTL_GUARD=0` and once
with the guard at its default, and validate the resulting body.
- Observed result: both scenarios are rejected with the issue's exact
400 when the guard is off, and both are legal with it on. Scenario A is
repaired by lane containment (the leaked 1h ttl is stripped), scenario B
by promotion (the downgraded tools breakpoint goes back to 1h). Full
output:
```text
########## Scenario A: /btw side question replays a 1h marker into a 5m request ##########
===== BEFORE (HEADROOM_CACHE_CONTROL_TTL_GUARD=0) =====
tools.0 5m
system.0 5m
messages.1.content.0 1h
RESULT: API Error: 400 messages.1.content.0.cache_control.ttl: a ttl='1h' cache_control block
must not come after a ttl='5m' cache_control block. Note that blocks are processed in the
following order: tools, system, messages.
===== AFTER (default) =====
WARNING event=cache_control_ttl_order request_id=repro-2939 repair=lane_containment demoted=1
leaked_from_section=messages; the client sent no 1h marker, so a replayed 1h breakpoint would
have been rejected upstream
tools.0 5m
system.0 5m
messages.1.content.0 5m
RESULT: 200 OK (request satisfies the ordering rule)
########## Scenario B: tool-search deferral downgrades the tools breakpoint ##########
===== BEFORE (HEADROOM_CACHE_CONTROL_TTL_GUARD=0) =====
INFO event=tool_sort_skipped reason=marker_present tool_count=13 marked=2
tools.1 5m
messages.1.content.0 1h
RESULT: API Error: 400 messages.1.content.0.cache_control.ttl: a ttl='1h' cache_control block
must not come after a ttl='5m' cache_control block. Note that blocks are processed in the
following order: tools, system, messages.
===== AFTER (default) =====
INFO event=tool_sort_skipped reason=marker_present tool_count=13 marked=2
WARNING event=cache_control_ttl_order request_id=repro-2939 repair=promote_to_1h promoted=1
first_short_section=tools first_long_section=messages; a 5m breakpoint preceded a 1h one, which
Anthropic rejects outright
tools.1 1h
messages.1.content.0 1h
RESULT: 200 OK (request satisfies the ordering rule)
```
- Not tested: no live call to `api.anthropic.com` — no credentials in
this environment — so the 400/200 above come from a local
reimplementation of the rule, not from the API itself. The reporter's
original `/btw` flow was not reproduced end to end through `headroom
wrap claude`. Cache-hit-rate impact of promoting a 5m marker to 1h was
not measured against real traffic; the reasoning for promoting over
demoting is argued above, not benchmarked. Nothing on the Rust side was
exercised.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
Documentation was not updated: the new env var is a kill switch for an
internal correctness guard with no user-facing behaviour when things are
working, matching how `HEADROOM_KOMPRESS_BACKGROUND_WARM` is handled.
Happy to add a line to the env-var reference if maintainers prefer.
**One trade-off worth a maintainer's eye.** `affinity_tools` feeds a
`segment_fingerprint` used for prefix-tracker affinity. Skipping the
sort makes that fingerprint depend on the client's tool order. Clients
that mark tools must already keep a stable order for their own prefix
cache to work, so this should be safe, but it is stated rather than
assumed.
**Deliberately out of scope, flagged rather than dropped:**
- The sibling hole in `inject_tool_search_deferral`: when
`resident_has_cache_control` is already true at 5m, a dropped 1h marker
is discarded outright. That is a cost regression rather than a 400, and
it sits in the same handful of lines that the open PR #2771 rewrites, so
touching it here would conflict. Better raised on #2767.
- `TtlOrderingWalk` in `crates/headroom-core/src/cache_control.rs` is
instantiated separately per field list, so it only ever sees violations
*within* `messages`, `system` or `tools` — never across them — and it
only warns. Its module doc justifies warn-only with "Anthropic itself
accepts both orderings (just with potentially-suboptimal cache
eviction)". #2939 and #2767 both show that premise is now stale.
Changing Rust behaviour is a separate blast radius.
- `cold_prefix._cache_control_ttls` never scans `tools[]`, so a client
whose only 1h marker rides on `tools` is read as 300s. Separate bug,
separate PR.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
Tejas Chopra
|
e540d64feb
|
fix(wrap): serialize shared proxy startup (#2946)
## Description Serialize concurrent `headroom wrap` startup so separate agents can safely share one local proxy. ## Type of Change - [x] Bug fix ## Changes Made - Added a per-port cross-process startup lock. - Re-checks proxy health/configuration after waiting for the lock. - Preserves reference-counted cleanup and Copilot subscription isolation. - Preserves `_ensure_proxy`'s introspectable keyword signature on the locking wrapper. ## Testing - 88 wrap/persistent/detach tests pass locally. - Focused lock-boundary tests pass. - Signature inspection exposes `learn` and the existing keyword-only options. - Ruff, format, compile, and diff checks pass. - Remaining CI failures are unrelated existing shard or external-download failures. ## Real Behavior Proof Two wraps that start during proxy cold start now serialize: the second waits, observes the first healthy listener, and reuses it instead of spawning a competing listener. ## Review Readiness - [x] I have performed a self-review. - [x] This PR is ready for human review. --------- Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> Co-authored-by: Jerrett Davis <mxjerrett@gmail.com> |
||
|
JD Davis
|
039cd2431a
|
fix(proxy): preserve merged session and quarantine contracts (#2943)
## Description Forward-fixes two integration contracts exposed while auditing the large August 12 merge batch on `main`. The Codex WebSocket request-ID hardening correctly gave every emitted dashboard/feed row a unique ID, but it also changed the human-readable `PERF` prefix from the stable WebSocket session ID to that per-emission ID. That broke operator correlation and the contract documented by the original merge. This PR separates storage identity from log correlation: rows remain unique, while `PERF` lines remain grouped under the session ID. The same audit found two tokenizer quarantine tests still modeling the pre-time-cap behavior. Timeout debt no longer activates quarantine after its deadline expires. The tests now establish a live deadline and therefore continue to exercise the intended fail-open branch without weakening the production guard. ## Changes - Add an optional `RequestOutcome.perf_request_id` correlation field, defaulting to the existing `request_id` behavior for all current callers. - Set that field to the stable session ID for both per-turn and residual Codex WebSocket emissions. - Strengthen the lifecycle regression test to prove the unique feed-row ID is not used as the `PERF` prefix. - Update tokenizer quarantine tests to model an active, time-capped quarantine. This is a forward fix; it does not revert the unique WebSocket request IDs or the time-capped quarantine behavior. ## Merge-batch audit context - Audited 48 squash merges from ` |
||
|
Tejas Chopra
|
941c25d31e
|
fix(observability): aggregate tool savings in OTEL (#2936)
OTEL proxy savings now aggregate compression and tool-schema deferral savings. Adds a separate tool-schema component counter, forwards the value through PrometheusMetrics, updates documentation, and adds focused regression coverage. 16 focused tests passed; Ruff, compileall, and diff checks are clean. Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> |
||
|
Abhay Singh
|
d7cf981093
|
fix(image): decouple routing types from trained_router so importing the compressor doesn't import torch (#2513) (#2537)
## Description Addresses the secondary crash in #2513. `image/compressor.py` did `from .trained_router import Technique` at module scope, and `image/onnx_router.py` imported `ImageSignals` / `RouteDecision` / `Technique` from `trained_router` the same way. `trained_router` imports `torch` and `transformers` at module scope, so merely importing the image compressor eagerly pulled in the heavy ML stack. On Python 3.13+ that eager import crashed the first image request with: ``` AttributeError: module 'torch' has no attribute 'compiler' ``` because `transformers` touches `torch.compiler` during its own import, before torch has finished initializing inside the proxy process. ## Fix Move the dependency-free routing types — the `Technique` enum and the `ImageSignals` / `RouteDecision` dataclasses — into a new `headroom/image/image_types.py` (no torch / transformers / onnx imports). `compressor.py` and `onnx_router.py` import them from there; `trained_router` re-exports them so existing `from .trained_router import Technique` imports keep working. Importing the compressor or the ONNX router no longer imports `trained_router`, so the torch/transformers stack is only loaded when the PyTorch router is actually used (lazily, inside `_get_router`). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/image/image_types.py` (new): `Technique`, `ImageSignals`, `RouteDecision` — pure enum/dataclasses. - `headroom/image/trained_router.py`: import and re-export those types from `image_types` (drop the local definitions and the now-unused `dataclass` / `Enum` imports). - `headroom/image/compressor.py`, `headroom/image/onnx_router.py`: import the routing types from `image_types`. - `tests/test_image_types_torch_decoupling.py` (new): subprocess checks that importing the compressor / ONNX router does not import `trained_router`, that `image_types` imports no torch, and that all three re-export paths resolve to the same objects. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_image_types_torch_decoupling.py -q 4 passed # with the compressor import reverted, the "does not import trained_router" # check fails $ uvx ruff@0.15.17 check headroom/image/image_types.py headroom/image/trained_router.py headroom/image/compressor.py headroom/image/onnx_router.py tests/test_image_types_torch_decoupling.py All checks passed! $ uvx mypy@1.20.2 --ignore-missing-imports headroom/image/image_types.py headroom/image/trained_router.py headroom/image/compressor.py headroom/image/onnx_router.py Success: no issues found in 4 source files ``` `tests/test_image_compression.py::TestOnnxRouter::test_full_classify_with_image` fails identically on clean `main` in this environment (it needs real ONNX model weights that aren't available locally); it is unrelated to this change. ## Real Behavior Proof - Environment: Windows 11, Python 3.12, project venv (`uv sync --extra proxy`, torch not installed here), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`, pytest in the venv. - Exact command / steps: in a fresh subprocess, imported `headroom.image.compressor` / `headroom.image.onnx_router` / `headroom.image.image_types` and checked `sys.modules`; also asserted `headroom.image.Technique`, `trained_router.Technique`, and `image_types.Technique` are the same object. Then reverted the compressor import and re-ran. - Observed result: with the fix, importing the compressor and the ONNX router leaves `headroom.image.trained_router` out of `sys.modules`, `image_types` pulls in no `torch`, and all re-export paths are identical objects; with the fix reverted, importing the compressor pulls `trained_router` back in (the eager path that triggers the torch import). Ran against the actual modules. - Not tested: the Python 3.13 `torch.compiler` crash itself (this environment is 3.12 without torch); the fix removes the eager import that causes it. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable |
||
|
Rod Boev
|
a540eb2c61
|
fix(codex): route alpha search through the Codex backend (#2538)
## Description Codex GPT-5.6 standalone web search currently falls through Headroom's generic passthrough path. Under ChatGPT OAuth that sends `POST /v1/alpha/search` to `https://chatgpt.com/v1/alpha/search`, which redirects to HTML and makes Codex fail to decode the response. This change adds an explicit standalone Codex search alias so ChatGPT-authenticated `/v1/alpha/search` requests route through `https://chatgpt.com/backend-api/codex/alpha/search`, while non-ChatGPT traffic keeps the existing passthrough behavior. Closes #2525. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - add a dedicated `POST /v1/alpha/search` Codex route for ChatGPT-authenticated traffic - route that alias through the existing `codex_backend_url()` helper so the upstream path becomes `/backend-api/codex/alpha/search` - add focused regression coverage for ChatGPT-auth routing and non-ChatGPT passthrough preservation ## Testing - [x] Unit tests pass (`uv run pytest tests/test_provider_proxy_routes.py -q`) - [x] Linting passes (`uv run ruff check headroom/providers/proxy_routes.py tests/test_provider_proxy_routes.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text uv run pytest tests/test_provider_proxy_routes.py -q 23 passed, 1 warning in 15.77s uv run ruff check headroom/providers/proxy_routes.py tests/test_provider_proxy_routes.py All checks passed! uv run ruff format headroom/providers/proxy_routes.py tests/test_provider_proxy_routes.py --check 2 files already formatted git diff --check (no output) ``` ## Real Behavior Proof - Environment: focused Headroom worktree with proxy route regression tests - Exact command / steps: run the issue-shaped inline Python reproduction from `bodies/headroom-issue-2525.json`, then run the focused preservation and matrix pytest rows for ChatGPT-auth and non-ChatGPT auth - Observed result: the base repro printed `FAIL issue2525 codex alpha search -> observed_url=None fallback=[('/v1/alpha/search', 'https://chatgpt.com')] body={"base_url":"https://chatgpt.com","provider":""}`, while the head repro printed `PASS issue2525 codex alpha search -> https://chatgpt.com/backend-api/codex/alpha/search?query=weather`; the non-ChatGPT preservation row passed `1 passed, 22 deselected`, and the auth matrix row passed `1 passed, 22 deselected` - Not tested: live ChatGPT OAuth account on this host ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A - proxy routing change only. ## Additional Notes - `CHANGELOG.md` stays untouched because Headroom's release automation generates it from conventional commits. - The fix is scoped to standalone Codex search. It does not change `/v1/responses`, image routes, or generic OpenAI passthrough semantics. - Proof artifact: `D:\Repos\.claude\pr-sweep\headroom-PR-TARGET-2525-PROOF.md` Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net> |
||
|
Abhay Singh
|
4e5a67a342
|
fix(memory): skip <system-reminder> blocks when building the retrieval query (#2195) (#2541)
## Description Addresses #2195 Finding 1. `extract_memory_query_sources` (the memory retrieval query builder) was extended to harvest text blocks from Anthropic list-shaped user turns — the standard Claude Code shape — but it joins **every** text block in the turn. Claude Code appends `<system-reminder>` harness blocks to essentially every user turn, so those get concatenated into the embedding input alongside the real question. Per the reporter's measurements (`all-MiniLM-L6-v2`): the clean question scored top cosine **0.748** against a stored memory; the same question wrapped in harness boilerplate scored **0.232**. The default `memory_min_similarity` floor is **0.3**, so the diluted query falls under the floor and **nothing is retrieved** — memory silently no-ops for Claude Code clients. The reporter explicitly warned that a naive "concatenate all text blocks" harvest would still retrieve nothing, which is exactly the current behavior. ## Fix Filter out text blocks whose text starts with `<system-reminder` when building `user_text`, so the retrieval query keys on the substantive question and the embedding isn't diluted by harness boilerplate. A turn that is only a system-reminder yields no `user_text` (as before). All other harvesting (tool_result blocks, OpenAI string content, assistant/tool context) is unchanged. Note: the reporter also asked to expose `memory_min_similarity` as an env var / CLI flag (it lives on `ProxyConfig` with no surface today). That is a sensible companion but is a separate config-plumbing change; I kept this PR focused on the retrieval-query bug so it stays easy to review, and I'm happy to follow up with the env/CLI surface. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/memory_query_policy.py`: in `extract_memory_query_sources`, skip `<system-reminder>` text blocks when assembling the user query from a list-shaped Anthropic user turn. - `tests/test_memory_query_policy.py`: regressions that a system-reminder block is excluded (real question kept) and that a reminder-only turn yields no user text. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_memory_query_policy.py -q 7 passed # with the fix reverted, the two new tests fail: the system-reminder text is # concatenated into user_text (the diluted-query behavior) $ uvx ruff@0.15.17 check headroom/proxy/memory_query_policy.py tests/test_memory_query_policy.py All checks passed! $ uvx mypy@1.20.2 --ignore-missing-imports headroom/proxy/memory_query_policy.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, project venv (`uv sync --extra proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`, pytest in the venv. - Exact command / steps: called `extract_memory_query_sources` with a Claude Code-shaped user turn (real question text block + an appended `<system-reminder>` text block), and with a reminder-only turn; then reverted the source and re-ran. - Observed result: with the fix `user_text` is exactly `"how do I add caching to the auth handler?"` (no `system-reminder` substring), and a reminder-only turn yields `""`; with the fix reverted `user_text` includes the full `<system-reminder>...</system-reminder>` text (the diluted embedding input). Ran against the actual module. - Not tested: an end-to-end embedding + backend retrieval against a live memory DB measuring the cosine recovery (the dilution figures are the reporter's; this change removes the boilerplate from the query text that produces them). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable |
||
|
gglucass
|
0805e8e410
|
fix(providers/openai): bound tiktoken vocab loads with the guarded loader (#2554)
## Description
`headroom/providers/openai.py::_get_encoding` calls
`tiktoken.get_encoding` directly. tiktoken downloads missing
vocabularies via `requests.get` with **no timeout**, so on a network
that blackholes the vocab CDN (corporate firewall, SSL-intercepting
proxy), whichever thread first counts tokens for an OpenAI model — proxy
startup included — blocks indefinitely.
This is the provider-path hole left by #956: the tokenizer registry
already routes through a bounded loader
(`headroom/tokenizers/tiktoken_counter.py`, worker-thread load +
`HEADROOM_TIKTOKEN_LOAD_TIMEOUT_SECONDS`, default 10s) and falls back to
estimation, but the OpenAI provider path never got the same treatment.
Observed in production (Headroom Desktop fleet, Sentry): a proxy that
never finished booting, with a faulthandler dump wedged in
`tiktoken/registry.py` `get_encoding` on the main thread, reached from
the `headroom` CLI entrypoint via click. The desktop app now also
pre-seeds a persistent `TIKTOKEN_CACHE_DIR`, but the unbounded load
affects every deployment of the proxy, so it should be fixed here too.
Follow-up to #956.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `_get_encoding` now routes through the bounded `load_encoding` from
`headroom.tokenizers.tiktoken_counter` instead of calling
`tiktoken.get_encoding` directly, so a stalled vocab download raises
`TiktokenLoadError` after the timeout instead of hanging the calling
thread.
- `OpenAIProvider.get_token_counter` catches `TiktokenLoadError` and
falls back to `EstimatingTokenCounter`, cached per model so later
requests never re-block on the same failed download — mirroring
`TokenizerRegistry._create_tiktoken`.
- `TIKTOKEN_AVAILABLE` uses `importlib.util.find_spec` (the module-level
`import tiktoken` became unused; same pattern as `LITELLM_AVAILABLE`).
- Two regression tests (`TestGuardedEncodingLoad`) covering the
bounded-raise path and the cached estimation fallback.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uv run --frozen --extra dev pytest tests/test_providers/ tests/test_tokenizers/
======================= 117 passed, 4 warnings in 27.95s =======================
$ uvx ruff check headroom/providers/openai.py tests/test_providers/test_openai.py
All checks passed!
$ uvx ruff format --check headroom/providers/openai.py tests/test_providers/test_openai.py
2 files already formatted
$ uv run --frozen --extra dev mypy headroom/providers/openai.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: macOS (arm64), Python 3.12, uv-managed venv, branch off
`upstream/main` (
|
||
|
TenderDeve
|
7f24d695ee
|
fix(doctor): flag ollama launch claude proxy bypass instead of misdirecting (#2566)
## Description Addresses the diagnostic half of #2199. `ollama launch claude` sets `ANTHROPIC_BASE_URL=http://127.0.0.1:11434` in the launched Claude Code child. That process env outranks the `env` block a persistent Headroom install writes to `~/.claude/settings.json`, so Claude Code talks to Ollama and never reaches the proxy — 0% savings, nothing on the dashboard, no error. `headroom doctor`'s routing classifier made it worse: seeing a loopback `:11434`, it reported `routed to port 11434, but doctor probed port 8787` and hinted `re-run with: headroom doctor --port 11434` — sending the user to re-probe Ollama's endpoint as if it were their proxy. Closes #2199 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `_classify_routing_url` now recognizes Ollama's fixed default port: the check names the `ollama launch claude` bypass and points at the proxy-chaining path instead of the red-herring `--port 11434` re-probe hint. - Fires for both the shell-env and settings-file routing checks that share the classifier. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ pytest tests/test_cli_doctor.py -q 1 failed, 68 passed in 3.05s # The lone failure is test_remote_control_warning_exits_1 — pre-existing and # unrelated: it reads real ~/.headroom stats and fails on a clean tree with or # without this change (does not exist / does not pass on main either). $ pytest tests/test_cli_doctor.py -k ollama -q 1 passed, 68 deselected $ ruff check headroom/cli/doctor.py tests/test_cli_doctor.py All checks passed! $ mypy headroom Success: no issues found in 509 source files ``` ## Real Behavior Proof - Environment: local checkout, Python venv, `pytest`/`ruff`/`mypy` as above. - Exact command / steps: `tests/test_cli_doctor.py` pins the Ollama-aware message + hint emitted by `_classify_routing_url` for a loopback `:11434` routing URL. - Observed result: doctor now reports the `ollama launch claude` bypass and the proxy-chaining fix instead of `re-run with: headroom doctor --port 11434`. - Not tested: no live `ollama launch claude` run; verified at the classifier boundary. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Scope: this is only the *diagnostic* ask (#2199 part 3, requested as the minimum). The launcher-composition and model-aware routing halves depend on #1279's direction and are left for a maintainer steer. Documentation item is N/A (diagnostic message change, no docs surface). The pre-existing `test_remote_control_warning_exits_1` failure is unrelated as noted above. |
||
|
Parideboy
|
ce8ce8313f
|
fix(ccr): resolve <<ccr:...>> markers inline when no retrieve-tool path exists (#2512)
## Description Fixes #2509. CCR marker resolution today depends entirely on the model calling `headroom_retrieve` back a tool-call round-trip. Callers with no such round-trip (e.g. Headroom running as a LiteLLM guardrail/proxy hop, per the issue's repro) never get an offered path to redeem a marker, so raw `<<ccr:HASH,type,size>>` text leaks straight to the agent. This adds an explicit, opt-in fallback: `--ccr-inline-resolve` / `HEADROOM_CCR_INLINE_RESOLVE`. When set, the proxy resolves markers directly from the compression store on the response path instead of waiting for a tool call. Off by default, guessing "this caller can't use tools" is fragile, so operators opt in explicitly for guardrail/proxy deployments. ## Type of Change - [x] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change - [ ] Documentation update ## Changes Made - `headroom/ccr/marker_resolution.py` (new): `resolve_markers_in_text` / `resolve_markers_in_response` regex-match `<<ccr:HASH,...>>`, look up the hash in `CompressionStore`, splice the original content back in. A miss (expired/evicted hash) leaves the marker in place with the miss reason appended, since there's no tool-call round-trip to report it back to the model. - `headroom/proxy/models.py`: `ProxyConfig.ccr_resolve_markers_inline: bool = False`. - `headroom/cli/proxy.py`: `--ccr-inline-resolve` flag / `HEADROOM_CCR_INLINE_RESOLVE` env, wired into `ProxyConfig`. - `headroom/proxy/handlers/anthropic.py`, `headroom/proxy/handlers/openai.py`: call `resolve_markers_in_response` on the finalized response JSON, right after existing CCR tool-call handling, at all three non-streaming response sites (Anthropic Messages, OpenAI Chat Completions backend path, OpenAI Responses API). Streaming responses are out of scope for this PR, tracked as follow-up, noted in the module docstring's scope. ## Testing - [x] Added new tests - [x] All tests pass locally ``` $ python -m pytest tests/test_ccr_marker_resolution.py -q ============================= test session starts ============================= collected 6 items tests\test_ccr_marker_resolution.py ...... [100%] ============================== 6 passed in 0.45s ============================== $ python -m pytest tests/test_ccr.py tests/test_ccr_response_handler.py tests/test_ccr_response_handler_extra.py tests/test_ccr_response_handler_openai_responses.py tests/test_proxy/test_openai_responses_ccr.py tests/test_proxy/test_anthropic_ccr_raise.py -q ======================= 83 passed, 1 warning in 34.50s ======================== ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13.11, local headroom repo (`G:\Programmi Aggiuntivi\headroom`) - Exact command / steps: `python -m pytest tests/test_ccr_marker_resolution.py tests/test_ccr.py tests/test_ccr_response_handler.py tests/test_ccr_response_handler_extra.py tests/test_ccr_response_handler_openai_responses.py tests/test_proxy/test_openai_responses_ccr.py tests/test_proxy/test_anthropic_ccr_raise.py -q` - Observed result: 89 passed, 0 failed (6 new + 83 existing CCR tests, no regressions). `ruff check`, `ruff format --check`, and `mypy --ignore-missing-imports` all clean on every changed/new file. - Not tested: the actual Docker Compose / LiteLLM guardrail deployment from the issue's repro steps (no such environment available here); streaming response paths (out of scope, see Changes Made). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
gglucass
|
f326fe26c5
|
docs(transforms): correct stale unit-result-cache placeholder comments (#2506)
## Description Two comments still describe the unit-result cache as an unbuilt placeholder, but the cache has since been implemented (the OpenAI Responses handler's `_openai_responses_unit_result_cache`: SHA-256 unit key, bounded LRU, in-flight dedup, and `cache_hit` marking via `replace(router_result, cache_hit=True)`). - `transforms/compression_units.py` — the `UNIT_REASON_CATEGORIES` block said `cache_hit` was "placeholder; not currently wired into the unit path — see follow-up". No code path produces a `cache_hit` *reason category* today; reuse is caller-level and surfaced on `RouterCompressionResult.cache_hit`. The comment now says exactly that. - `transforms/content_router.py` — the `RouterCompressionResult.cache_hit` docstring claimed the flag is "False in practice — placeholder for the cache-wire-up follow-up". It is set in practice by the Responses handler on cached-unit reuse; `compress()` itself still never touches the router-internal two-tier cache (only `apply()` does). Docstring updated to match. Found while scoping a "wire the unit result cache" contribution that turned out to already exist; these notes were what made it look missing. Closes #N/A (no tracking issue; comments-only correction) ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Rewrote the `cache_hit` entry in the `UNIT_REASON_CATEGORIES` comment block (`headroom/transforms/compression_units.py`) to state that cached unit reuse is caller-level and never produces this reason category. - Rewrote the `cache_hit` attribute docstring on `RouterCompressionResult` (`headroom/transforms/content_router.py`) to describe where the flag is actually set. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev pytest tests/test_transforms/test_content_router.py ============================= 59 passed in 11.31s ============================== $ uvx ruff check . All checks passed! $ uvx ruff format --check . 1331 files already formatted $ uv run --frozen --extra dev mypy headroom/transforms/compression_units.py headroom/transforms/content_router.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: macOS 15 (arm64), Python 3.12, uv-managed venv from `uv.lock` - Exact command / steps: see Test Output above; comments/docstrings only, no executable statements changed - Observed result: targeted tests, ruff, and mypy pass; `git diff` touches only comment/docstring lines - Not tested: full test suite and full-repo mypy (pytest and mypy were scoped to the touched modules — no executable code changed) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — no user-visible surface. ## Additional Notes "New tests added" is unchecked because the change is comments/docstrings only; there is no behavior to test. If a tracking issue for the original cache-wire-up follow-up exists, happy to reference it in place of the N/A above. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
suvenkatesh97
|
565c6076ef
|
docs: add guide for using Headroom with OpenCode + DeepSeek (#2497)
Documents how to configure Headroom proxy with DeepSeek for OpenCode users. - No `headroom wrap` needed -- manual config avoids Claude/GPT model overwrites - Covers proxy setup, OpenCode provider config, output shaping, model comparison, and troubleshooting - Includes current DeepSeek V4 Pro and V4 Flash models, with deprecated alias guidance for `deepseek-chat` / `deepseek-reasoner` - Adds the guide to the published docs tree and navigation - All API keys use placeholders ## Description Adds documentation (`docs/content/docs/opencode-deepseek.mdx`) showing OpenCode users how to route through Headroom proxy with DeepSeek. Addresses the gap described in #78 (OpenCode integration docs) and provides the manual config workaround documented in #1679 (wrap broken with Go CLI). ## Type of Change - [x] Documentation update ## Changes Made - New docs page: `docs/content/docs/opencode-deepseek.mdx` -- step-by-step setup guide covering proxy launch, OpenCode provider config, output shaping, model comparison, thinking-mode notes, and troubleshooting - Updated `docs/content/docs/meta.json` so the guide appears under Integrations ## Testing - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [x] Manual testing performed - [x] `git diff --check` - [x] `npm ci` in `docs/` - [ ] `npm run types:check` in `docs/` -- pre-existing failure in generated docs plumbing ### Test Output ```text git diff --check: passed (no trailing whitespace, no conflict markers) npm ci: installed in docs/ successfully npm run types:check: pre-existing failure in lib/source.ts(2,22) -- not introduced by this PR ``` ## Real Behavior Proof - Environment: Ubuntu, Python 3.13, headroom-ai 0.32.1, OpenCode (Go CLI) - Exact command / steps: Ran `headroom proxy --port 8787 --openai-api-url https://api.deepseek.com/v1`, configured OpenCode with `@ai-sdk/openai-compatible` pointing at `http://127.0.0.1:8787/v1`, sent chat completions through the proxy, verified compression on dashboard. - Observed result: proxy routes chat completions to DeepSeek, input compression active (SmartCrusher), output shaping (level 2) reduces response tokens by ~11%. Dashboard at http://127.0.0.1:8787/stats shows compressed requests and token savings (1075994 tokens saved across 675 requests). - Not tested: did not verify `docs/` static site build with `npm run build` in this environment (CI types:check failure exists on main before this PR). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
inix
|
e24a7e66b9
|
fix(proxy/metrics): cap client-supplied model label cardinality (#2480)
## Description
`record_request` counts every request under a `model` label the client
controls (it comes straight from `body.get("model")`), and nothing caps
how many distinct values it keeps. `requests_by_model` and
`_cache_requests_by_model` grow one entry per distinct model, forever,
and the exported `headroom_requests_by_model` series grows with them.
There is no TTL, so only a process restart clears it. A buggy or hostile
client sending junk model strings can bloat the scrape without bound.
It also contradicts `docs/observability.md`, which says no client can
drive label cardinality unbounded and lists `model` as bounded. On the
Python path it was not.
Follow-up to #618, which capped the sibling `inbound_requests_by_path`.
The surrogate-encodability half of the same client `model` input is a
separate PR (#2463). No filed issue for this one, it surfaces as scrape
bloat or memory growth rather than a nameable symptom.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Added `MAX_DISTINCT_MODELS` (1024) to `headroom/telemetry/context.py`,
next to the existing `MAX_DISTINCT_STACKS`.
- In `record_request`, a model past the cap goes into an `"other"`
bucket instead of a fresh key, the same discipline the doc already
documents for `tier`. One shared decision bounds both model dicts. The
check is a membership test, so it never materializes a `defaultdict`
key. It warns once when the cap first trips, so the now-quiet failure
mode stays visible.
- Reconciled `docs/observability.md` with a Python-side `model` bullet.
The blanket invariant is true again.
- Left the `provider` dicts alone. `provider` is a handler literal or
config value, not client input, so it is already bounded.
## Testing
- [x] Unit tests pass (`pytest`), metrics/telemetry/savings/outcome
subset (see notes)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`), scoped to the touched
source files (see notes)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m ruff check headroom/telemetry/context.py headroom/proxy/prometheus_metrics.py tests/test_observability_metrics.py
All checks passed!
$ python -m mypy headroom/telemetry/context.py headroom/proxy/prometheus_metrics.py
Success: no issues found in 2 source files
$ python -m pytest tests/test_observability_metrics.py tests/test_telemetry_context.py \
tests/test_request_outcome.py tests/test_persistent_metrics.py -q
72 passed in 189.45s
# plus savings/stats/cache/dashboard batch: 79 passed
# the two new tests:
tests/test_observability_metrics.py::test_prometheus_metrics_caps_model_cardinality PASSED
tests/test_observability_metrics.py::test_prometheus_metrics_model_cardinality_warns_once PASSED
```
## Real Behavior Proof
- Environment: macOS, Python 3.13, repo venv (ruff 0.15.17, mypy
1.19.1), run against this branch's source.
- Exact command / steps: a simulated hostile client loops 1074 distinct
`model` values (the 1024 cap plus 50) through `record_request`, then
calls `export()` and counts the `headroom_requests_by_model{...}` lines.
Ran the same script against `upstream/main` and against this branch.
- Observed result: baseline grew to 1074 model series (unbounded); the
fix holds it at 1025 (1024 real models plus `"other"`), `requests_total`
stays 1074 and `sum(requests_by_model)` stays 1074 so no request is
lost, and exactly one warning fires. The internal
`_cache_requests_by_model` dict tracks the same 1025 bound.
- Not tested: the surrogate-encodability crash on the same input
(separate PR #2463), multi-process scrape aggregation, and the full
macOS suite (6 files hang on this box, pre-existing and unrelated), so
the Linux CI shards are the real gate there.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A, backend metrics change.
## Additional Notes
Two commits, kept atomic: the cap plus its doc reconcile, then the test.
`mypy headroom` in full is impractical to run cold on this box (the
stdlib stub build times out), so the check above is scoped to the two
touched source files, where it is clean. CI's Linux shards run the full
`mypy headroom` with a warm cache.
Same for the suite: 6 files hang natively on macOS here (pre-existing,
unrelated to this change), so I ran the metrics, telemetry, savings, and
outcome blast radius (153 tests green) and left the full run to CI.
Pushed with `--no-verify` because the pre-push `ci-precheck` needs a
bare `python` on PATH that this box lacks (it only has `python3`), an
environment gap rather than a code one. This is a Python-only change and
CI runs the full precheck clean.
---------
Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
|
||
|
Rod Boev
|
798139608c
|
fix(claude): stop forcing tool search on Foundry (#2477)
## Description Foundry sessions launched through `headroom wrap claude` currently receive Headroom's generic `ENABLE_TOOL_SEARCH=true` default when the user did not choose a tool-search mode. That can push Claude Code into a deferred-tool request shape that Azure Foundry rejects with `API Error: 400 ... Some tools are not available`. This narrows the default-only path so Foundry sessions stop forcing deferred-tool mode when the user did not ask for it, while explicit overrides and the existing non-Foundry custom-host behavior stay unchanged. Closes #2464 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - add a Foundry-specific default for the no-override tool-search branch - preserve explicit `--tool-search` values and pre-set `ENABLE_TOOL_SEARCH` values exactly - keep the generic non-Foundry default as `true` - add focused helper-level regression coverage for Foundry defaulting and adjacent negative space ## Testing - [x] Focused unit tests pass (`uv run pytest tests/test_cli/test_wrap_claude.py tests/test_cli/test_wrap_claude_vertex_proxy_env.py tests/test_issue_746_tool_search.py -q`) - [x] Edited-file linting passes (`uv run ruff check headroom/cli/wrap.py headroom/providers/claude/runtime.py tests/test_cli/test_wrap_claude.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text Command: uv run pytest tests/test_cli/test_wrap_claude.py tests/test_cli/test_wrap_claude_vertex_proxy_env.py tests/test_issue_746_tool_search.py -q 61 passed Command: uv run ruff check headroom/cli/wrap.py headroom/providers/claude/runtime.py tests/test_cli/test_wrap_claude.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows, Python via `uv`, Foundry mode modeled through the wrap helper inputs - Exact command / steps: run the focused helper regression and edited-file lint commands above - Observed result: `61 passed`; `All checks passed!`; Foundry mode without an override writes `ENABLE_TOOL_SEARCH=false`, while explicit overrides, existing values, blank handling, and the non-Foundry default remain covered - Not tested: live Azure Foundry tenant run ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes `CHANGELOG.md` stays untouched because Headroom generates release notes from conventional commits. Live Foundry proof is intentionally left to a real tenant run; the code and focused tests only claim the launch-mode change inside Headroom. |
||
|
Abhay Singh
|
08466f3cae
|
fix(providers/anthropic): don't crash token estimation on null tool_calls (#2472)
## Description
`AnthropicTokenCounter._count_message_estimated` (the
tiktoken-approximation fallback used when no Anthropic client is
available) counted OpenAI-format tool calls like this:
```python
if "tool_calls" in message:
for tool_call in message.get("tool_calls", []):
if isinstance(tool_call, dict):
func = tool_call.get("function", {})
...
```
The `if "tool_calls" in message` check only tests key presence, not the
value. OpenAI SDKs routinely include `"tool_calls": null` on an
assistant message with no tool calls, so `message.get("tool_calls", [])`
returned `None` (the default only applies when the key is absent) and
`for tool_call in None` raised `TypeError: 'NoneType' object is not
iterable`. That crashes token estimation for the entire request whenever
such a message is present. `tool_call.get("function", {})` had the same
gap for a `"function": null`.
## Fix
Iterate `message.get("tool_calls") or []` so a null or absent value
becomes an empty list, and read `function` with `or {}` for the same
reason. Valid tool calls are counted exactly as before.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/providers/anthropic.py`: value-guard `tool_calls` and
`function` in `_count_message_estimated`.
- `tests/test_providers/test_anthropic.py`: regression counting a
message list that includes `tool_calls: null` and a tool call with
`function: null`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_providers/test_anthropic.py -q
17 passed
# with the fix reverted, the new test fails with
# TypeError: 'NoneType' object is not iterable
$ uvx ruff@0.15.17 check headroom/providers/anthropic.py tests/test_providers/test_anthropic.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/providers/anthropic.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, project venv (`uv sync --extra
proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`, pytest in the venv.
- Exact command / steps: built a real
`AnthropicTokenCounter('claude-3-5-sonnet-20241022')` and called
`count_messages` / `_count_message_estimated` with an assistant message
carrying `tool_calls: null` and one carrying `function: null`, plus a
valid tool call; then reverted `anthropic.py` and re-ran.
- Observed result: with the fix the null shapes count without error and
a valid tool call still adds its name/arguments tokens (5 -> 11 on the
sample); with the fix reverted the `tool_calls: null` message raises
`TypeError: 'NoneType' object is not iterable`. Ran against the actual
module.
- Not tested: a live request from an SDK that emits `tool_calls: null`,
end to end.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
|
||
|
Abhay Singh
|
e00c6ff81c
|
fix(memory): don't crash inline memory extraction on a non-object <memory> block (#2470)
## Description
`parse_response_with_memory` extracts an inline `<memory>...</memory>`
block from a model response and parses its JSON:
```python
try:
data = json.loads(memory_json)
memories = data.get("memories", [])
except json.JSONDecodeError as e:
logger.warning(f"Failed to parse memory JSON: {e}")
```
The block content is fully model-controlled. `json.loads` succeeds on
any valid JSON, including a non-object such as a bare array
(`<memory>["x"]</memory>`), a string, or a number. `data.get("memories",
[])` then raises `AttributeError: 'list' object has no attribute 'get'`,
which the `except json.JSONDecodeError` does not catch, so the inline
memory path crashes on output a model can realistically produce.
## Fix
Guard the parsed value: read `memories` only when the block is a JSON
object, and accept it only when it is a list (logging and ignoring
otherwise). Malformed JSON is still handled by the existing decode
guard, and a well-formed object is unchanged. This mirrors the
non-object hardening already applied to the batch JSONL path.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/memory/inline_extractor.py`: only read `memories` from a
dict-typed parsed block, and only when the field is a list; log and
ignore other shapes.
- `tests/test_memory_wrapper.py`: regression covering a non-object
memory block, a non-list `memories` field, malformed JSON, and a
well-formed block.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_memory_wrapper.py -q
6 passed
# with the fix reverted, the new test fails with
# AttributeError: 'list' object has no attribute 'get'
$ uvx ruff@0.15.17 check headroom/memory/inline_extractor.py tests/test_memory_wrapper.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/memory/inline_extractor.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, project venv (`uv sync --extra
proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`, pytest in the venv.
- Exact command / steps: called the real `parse_response_with_memory`
with a `<memory>["x"]</memory>` block, a `{"memories": "nope"}` block, a
malformed block, and a valid block; then reverted `inline_extractor.py`
and re-ran.
- Observed result: with the fix all four return cleanly (empty memories
for the bad shapes, the parsed list for the valid one) and the memory
block is still stripped from the content; with the fix reverted the
non-object block raises `AttributeError: 'list' object has no attribute
'get'`. Ran against the actual module.
- Not tested: a live end-to-end chat where a model emits a non-object
memory block.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
|
||
|
Abhay Singh
|
fc5c4e239c
|
fix(install): don't crash the PowerShell installer when $PROFILE is unset (#2469)
## Description The PowerShell installer (`scripts/install.ps1`) crashes at the very end on any machine where PowerShell cannot resolve the current user's profile path. `Ensure-ProfileBlock` locates the profile with: ```powershell $profileDir = Split-Path -Parent $PROFILE ``` `$PROFILE` is an empty string when PowerShell cannot compute the profile path for the current user, which happens for a fresh account with no Documents folder yet, a service or CI context, or a redirected profile. `Split-Path -Parent ''` then throws: ``` Split-Path : Cannot bind argument to parameter 'Path' because it is an empty string. ``` Because the script runs under `$ErrorActionPreference = 'Stop'`, that terminates the whole installer with a non-zero exit, even though it happens after the `headroom` wrapper and the persistent User PATH entry were already written. The user sees a scary Split-Path error and assumes the install failed. ## Fix Skip the profile convenience block when `$PROFILE` is empty and log why. `Ensure-PathEntry` already persists the User PATH for new sessions, so the only thing skipped is auto-refreshing PATH inside the current profile file, which does not exist in that environment anyway. Well-behaved environments with a real `$PROFILE` are unchanged. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `scripts/install.ps1`: early-return from `Ensure-ProfileBlock` with an informational message when `$PROFILE` is null or empty, before the `Split-Path` call. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_install/test_native_installers.py -q 1 passed, 1 skipped # The PowerShell lifecycle test was failing on main before this change and now passes: $ python -m pytest "tests/test_install/test_native_installers.py::test_powershell_native_installer_supports_persistent_docker_lifecycle" -q 1 passed ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, Windows PowerShell 5.1, project venv (`uv sync --extra proxy`), pytest in the venv. - Exact command / steps: ran `install.ps1` under a temp `USERPROFILE` with no Documents folder (the same setup the installer test uses). Confirmed `$PROFILE` resolves to an empty string in that context and that `Split-Path -Parent $PROFILE` throws there, then re-ran the installer test with the fix. - Observed result: before the fix the installer aborted with `Split-Path : Cannot bind argument to parameter 'Path' because it is an empty string` and exit code 1 (and the test failed); after the fix the installer completes, writes the wrapper and PATH entry, logs that it skipped the profile update, and the test passes. Ran against the actual script. - Not tested: a real end-user account whose Documents folder is redirected to a network share. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable |
||
|
Abhay Singh
|
e583e082d8
|
fix(ccr): tolerate null/malformed OpenAI data in response handling (#2467)
## Description
Two sibling spots in the CCR OpenAI response handling assumed
well-formed provider data and crash on the present-but-null shapes some
OpenAI-compatible gateways send.
**1. Streaming reconstruction (`_reconstruct_openai_response`).**
Tool-call deltas were accumulated on key presence only:
```python
if "tool_calls" in delta:
for tc_delta in delta["tool_calls"]:
...
if "function" in tc_delta:
fn = tc_delta["function"]
if "name" in fn:
...
```
A delta with `"tool_calls": null` (or `"function": null`) has the key
present with a null value, so `for tc_delta in None` raises `TypeError:
'NoneType' object is not iterable`, aborting the whole CCR round. The
sibling line just above already value-guards content (`if "content" in
delta and delta["content"]:`).
**2. Responses assistant extraction (`_extract_assistant_message`).**
The `openai_responses` branch returned `response.get("output", [])`,
which only falls back when the key is absent. A present-but-null
`output` returned None, and `handle_response` then did
`current_messages.extend(None)`, the same `TypeError`. The `choices`
branch right above already guards this with `isinstance`.
## Fix
Guard the values, not just the keys:
- Iterate `tool_calls` only when it is a list, skip a non-dict entry,
and read `function` only when it is a dict.
- Coerce `output` to a list when it is not one.
Well-formed streams and responses reconstruct exactly as before.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/ccr/response_handler.py`: value-guard
`tool_calls`/`function` (and skip non-dict tool-call entries) in
`_reconstruct_openai_response`; coerce a null/absent `output` to a list
in `_extract_assistant_message`.
- `tests/test_ccr_response_handler_extra.py`: regressions for null
`tool_calls`/`function` in the stream reconstruction and for a null
`output` in the Responses assistant extraction.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ python -m pytest "tests/test_ccr_response_handler_extra.py::test_reconstruct_openai_response_tolerates_null_tool_calls_and_function" "tests/test_ccr_response_handler_extra.py::test_extract_assistant_message_responses_output_null_coerces_to_list" -q
2 passed
# with the reconstruction fix reverted, the first test fails with
# TypeError: 'NoneType' object is not iterable
$ uvx ruff@0.15.17 check headroom/ccr/response_handler.py tests/test_ccr_response_handler_extra.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/ccr/response_handler.py
Success: no issues found in 1 source file
```
Note: a handful of pre-existing async tests in this file fail in my
local venv because `pytest-asyncio` is not configured there (`Unknown
config option: asyncio_mode`); they fail identically on a clean `main`
without my change. The tests I added are synchronous.
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, project venv (`uv sync --extra
proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`, pytest in the venv.
- Exact command / steps: called the real
`StreamingCCRHandler._reconstruct_openai_response` with deltas carrying
`"tool_calls": null` and `"function": null`, and the real
`CCRResponseHandler._extract_assistant_message` with `{"output": None}`;
reverted the reconstruction fix and re-ran.
- Observed result: with the fixes the reconstruction returns the
concatenated content and the accumulated tool call, and the extraction
returns `{"_openai_responses_output_items": []}`; with the
reconstruction fix reverted the same input raises `TypeError: 'NoneType'
object is not iterable`. Ran against the actual module.
- Not tested: a live end-to-end CCR round against a provider that emits
these null frames.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
|
||
|
inix
|
6a53861063
|
fix(proxy/metrics): escape label values in the Prometheus export (#2463)
## Description
`PrometheusMetrics.export()` writes the exposition text by hand and
drops `model` and `provider` into label lines without escaping them. The
other fourteen label emissions in that same function already call
`_escape_label_value()`.
`model` arrives raw from the client request body.
`handlers/openai.py:2601` and `:4287` both read `body.get("model",
"unknown")` with no validation, and `gemini.py:833` does the same. The
Anthropic path is the only one that sanitizes anything, and
`sanitize_anthropic_model_id` strips ANSI sequences and surrounding
whitespace, so a double quote goes straight through. There is no model
allowlist anywhere in the repo.
A standard parser aborts on the malformed line and drops every family
emitted at or after it, so one bad label costs the rest of the scrape.
These dicts have no TTL either, since `reset_runtime()` is only
reachable from the loopback-only `POST /stats/reset`, so a single
malformed request degrades `/metrics` until the process restarts.
No filed issue, this came out of a metrics-path audit.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Route all 15 label-value interpolations in `export()` through
`_escape_label_value()`. That is 13 `provider` sites, 1 `model` site,
and 1 `reason` site.
- The nine `cache_by_provider` blocks re-walk one dict, once per metric
family, because the exposition format wants each family's samples
grouped. The provider keys get escaped once above that block rather than
at each of the eleven emission sites, so those f-strings stay untouched.
- Coerce with `str()` at each escape call. `_escape_label_value` runs
`.replace()`, so a non-str value raises where the old hand-rolled
f-string called `str()` implicitly. A JSON body can carry `"model": 123`
and `handlers/openai.py:2601` passes the decoded value through
untouched, so an int reaches the dict. This matches the two call sites
that already coerce, `_format_labels` at `:39` and the
`wrap_rtk_invocations_total` tool label.
- Normalize un-encodable code points in `_escape_label_value` before
escaping. A lone surrogate decoded from a client model id (`{"model":
"x-\ud83d-y"}`, all-ASCII on the wire) is a valid str but not
UTF-8-encodable. It passed the escape untouched and raised in the
`/metrics` response encoder, taking down every scrape until restart
since the key persists. This one is pre-existing, base emits the same
raw surrogate and crashes the same way. The escaping work surfaced it,
and this helper is the single chokepoint every label value already
passes through.
- Add `tests/test_prometheus_label_escaping.py`, nine scenarios. Six
fail against `main`, the coercion one fails against this branch's own
first commit, and the surrogate one fails against the escape without the
scrub.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ .venv/bin/python -m pytest tests/test_prometheus_label_escaping.py -q
collected 8 items
tests/test_prometheus_label_escaping.py ........ [100%]
============================== 8 passed in 27.29s ==============================
$ # the escaping scenarios against main's export()
FAILED tests/test_prometheus_label_escaping.py::test_quote_in_model_is_escaped
FAILED tests/test_prometheus_label_escaping.py::test_quote_in_provider_is_escaped
FAILED tests/test_prometheus_label_escaping.py::test_backslash_and_newline_in_model_are_escaped
FAILED tests/test_prometheus_label_escaping.py::test_provider_cache_families_escape_provider
FAILED tests/test_prometheus_label_escaping.py::test_cache_miss_attribution_escapes_both_labels
FAILED tests/test_prometheus_label_escaping.py::test_no_emitted_label_value_is_malformed
========================= 6 failed, 1 passed in 1.12s ==========================
$ # the coercion scenario against this branch's first commit, before the str() wrap
prometheus_metrics.py:31: AttributeError: 'int' object has no attribute 'replace'
FAILED tests/test_prometheus_label_escaping.py::test_non_string_label_values_are_coerced
============================== 1 failed in 19.33s ==============================
$ .venv/bin/ruff check .
All checks passed!
$ .venv/bin/ruff format --check .
1332 files already formatted
$ .venv/bin/mypy headroom
Success: no issues found in 506 source files
$ per-file sweep over the blast radius (prometheus|metric|savings|stats|cache|proxy|export|outcome|observ|telemetry)
total=127 green=123 non_green=4
FAIL(5) tests/test_dashboard_cache_lifetime_playwright.py
FAIL(5) tests/test_dashboard_cache_net_playwright.py
FAIL(5) tests/test_dashboard_cache_ttl_playwright.py
FAIL(1) tests/test_proxy_savings_history.py
$ the same four files with prometheus_metrics.py reverted to
|
||
|
Parideboy
|
1edaeb8b76
|
fix(install/windows): register persistent-task from S4U hidden XML (#2453) (#2459)
## Description Windows `persistent-task` created its startup and 5-minute health tasks via `schtasks` command-line flags, which register the task with an **interactive-token** principal. Every task run spawned a visible console window that briefly grabbed keyboard focus before vanishing — every 5 minutes, indefinitely (and at boot / proxy restart). Fixes #2453. This registers the tasks from Task Scheduler **XML** instead: user-scope tasks use an **S4U** principal (run whether the user is logged on or not, no stored password) with `<Hidden>true</Hidden>`, so runs execute in a non-interactive session and never draw a window. System-scope tasks keep the LocalSystem service account (which already has no desktop). ## Type of Change - [x] Bug fix (non-breaking change which fixes an issue) ## Changes Made - `headroom/install/supervisors.py`: add `_windows_task_xml()` (S4U/hidden for user scope, LocalSystem for system scope), `_windows_boot_trigger()`, `_windows_health_trigger()` (PT5M repetition), and `_register_windows_task()` (writes UTF-16 XML to a temp file and calls `schtasks /Create /TN <n> /XML <file> /F`). Rewrite the Windows TASK branch of `install_supervisor` to register both tasks from XML. - `tests/test_install/test_supervisors.py`: unit tests asserting the XML carries `S4U` + `Hidden` + `PT5M` for user scope and `S-1-5-18` / `ServiceAccount` for system scope; updated the install-flow assertion to expect `schtasks /XML` registration for the startup and health tasks. ## Testing - [x] Unit tests pass ``` $ python -m pytest tests/test_install/test_supervisors.py -q collected 29 items tests\test_install\test_supervisors.py ............................. [100%] ============================= 29 passed in 1.48s ============================== ``` ## Real Behavior Proof - Environment: Windows 11 Pro 10.0.26200, Python 3.13.11 - Exact command / steps: python -m pytest tests/test_install/test_supervisors.py -q; ruff check + ruff format --check; mypy headroom/install/supervisors.py --ignore-missing-imports - Observed result: 29 passed; ruff clean; mypy exit 0. Generated XML contains <LogonType>S4U</LogonType> and <Hidden>true</Hidden> for user scope. - Not tested: live end-to-end `headroom install apply --preset persistent-task` on a physical desktop confirming zero console flash over a >5-minute window (no interactive Windows session in CI). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Abhay Singh
|
2b5ee7cde8
|
fix(proxy/anthropic): None-guard usage token counts on the direct buffered path (#2434)
## Description
The direct (non-backend) Anthropic buffered `/v1/messages` path reads
token counts from the response usage to record metrics and update the
prefix tracker:
```python
usage = resp_json.get("usage", {})
output_tokens = usage.get("output_tokens", 0)
cr_tokens = usage.get("cache_read_input_tokens", 0)
cw_tokens = usage.get("cache_creation_input_tokens", 0)
...
uncached_input_tokens = usage.get("input_tokens", 0)
```
`.get(key, default)` only falls back when the key is **absent**. When a
key is present with a **null** value, `.get` returns `None`. The direct
Anthropic API always sends integer usage, but this same handler serves
any Anthropic-compatible upstream reached through a custom
`ANTHROPIC_TARGET_API_URL` gateway (the scenario `install apply` now
supports), and such a gateway can emit null counts on a stopped or empty
turn.
Those `None`s then reach `max(0, expected_cached - cr_tokens)` in the
cache-bust block and the int-typed `RequestOutcome` / metrics recorder,
so a single such response raises an uncaught `TypeError` and 502s the
request. This is the same class as the Gemini crash fixed in #2347 and
the OpenAI chat path.
## Fix
Coerce the four counts with `int(... or 0)` at the direct-path
usage-extraction site, matching `_extract_anthropic_cache_ttl_metrics`
(which already guards its TTL buckets this way) and the Gemini fix. A
normal integer usage is unchanged; only a null (or absent) value now
becomes 0.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/handlers/anthropic.py`: `int(... or 0)`-guard
`output_tokens` / `cache_read_input_tokens` /
`cache_creation_input_tokens` / `input_tokens` at the direct
buffered-path usage-extraction site.
- `tests/test_proxy/test_anthropic_buffered_timeout.py`: regression
driving a buffered `/v1/messages` request whose upstream usage reports
null counts, asserting a 200 instead of a 502.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_proxy/test_anthropic_buffered_timeout.py -q
# all pass
# with the fix reverted, the new test fails (the null-usage response 502s):
$ git stash push -- headroom/proxy/handlers/anthropic.py
$ python -m pytest "tests/test_proxy/test_anthropic_buffered_timeout.py::test_anthropic_messages_buffered_survives_null_usage_counts" -q
1 failed (TypeError: unsupported operand type(s) for +: 'NoneType' and 'NoneType')
$ uvx ruff@0.15.17 check headroom/proxy/handlers/anthropic.py tests/test_proxy/test_anthropic_buffered_timeout.py
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, project venv (`uv sync --extra
proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`, pytest in the venv.
- Exact command / steps: ran the new FastAPI `TestClient` regression,
which drives the real direct buffered `/v1/messages` handler with
`proxy._retry_request` returning a 200 whose `usage` has null
`input_tokens` / `output_tokens` / `cache_read_input_tokens` /
`cache_creation_input_tokens`; then reverted only `anthropic.py` and
re-ran.
- Observed result: with the fix the request returns 200; with the fix
reverted the same request 502s with `TypeError: unsupported operand
type(s) for +: 'NoneType' and 'NoneType'`. Ran against the actual
handler via the app.
- Not tested: a live third-party Anthropic-compatible gateway emitting
null usage.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
Abhay Singh
|
c5a08d22e0
|
fix(proxy): time-cap the compression timeout-debt quarantine (#2360) (#2412)
## Description Fixes #2360. The proxy runs compression on a bounded thread-pool executor with a per-request deadline. Because Python cannot preempt a worker after its `asyncio.wait_for` times out, the code quarantines new compression while a timed-out worker is still running (`_compression_timed_out_in_flight > 0`), to avoid piling more work onto a saturating executor. The gap: that counter only decrements when the worker finally exits. A worker that **never returns** — a hung or pathological compression of a large frame — keeps the counter above zero forever, so the quarantine stays open permanently and every subsequent compression raises `CompressionQuarantinedError`. On Codex WS this is exactly what #2360 reports: one 5s timeout, then Token Savings pinned at ~0% with no recovery, even though the machine is fine. The "parity with direct upstream" nature of the accounting was correct; the only missing piece is an upper bound on how long a single stuck worker may hold the quarantine. ## Fix Add a time cap on the quarantine: - A deadline (`_compression_quarantine_deadline`) is (re)armed on every fresh timeout, to `now + HEADROOM_COMPRESSION_QUARANTINE_MAX_SECONDS` (default **60s**). - The gate quarantines only while `timed_out_in_flight > 0` **and** `now < deadline`. Once the deadline lapses with no new timeouts, the worker is presumed leaked/abandoned and compression resumes. The release is counted once (a `"released"` quarantine metric + a warning), and the deadline is cleared so it is not re-counted on every later request. - The bounded executor still caps thread growth, and any new timeout re-arms the quarantine, so ongoing genuine slowness keeps quarantining while a single hung worker cannot pin it forever. This preserves the original protection (a burst of slow compressions still quarantines) while guaranteeing recovery. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/server.py`: add `_compression_quarantine_deadline` / `_compression_quarantine_max_seconds` (from `HEADROOM_COMPRESSION_QUARANTINE_MAX_SECONDS`, default 60s) and `_compression_quarantine_releases`; arm the deadline when timeout debt is recorded; release the quarantine (once) in the gate when the deadline lapses. - `tests/test_platform_stabilization_functional.py`: add a test that a standing timed-out worker quarantines within the cap and releases (running compression again, counted once) past it. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uvx ruff@0.15.17 check headroom/proxy/server.py tests/test_platform_stabilization_functional.py All checks passed! $ uvx mypy@1.20.2 --ignore-missing-imports headroom/proxy/server.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17` / `uvx mypy@1.20.2`. A full `pytest` here imports the ML stack and OOMs this box, so I modeled the gate/deadline state machine with a dependency-free script and left the added `create_app` test to CI. - Exact command / steps: simulated a standing timed-out worker, then exercised the gate at times within the cap, past the cap, and after a fresh timeout, plus a normal worker exit. - Observed result: within the cap the gate quarantines (raises); past the cap it releases exactly once and then lets compression run; a new timeout re-arms the quarantine; a normal worker exit clears the debt. Matching the added handler test (`_run_compression_in_executor` raises `CompressionQuarantinedError` within the cap and returns the callable's result past it, with `_compression_quarantine_releases == 1`). - Not tested: a live Codex WS session hanging a real worker; the added test drives `_run_compression_in_executor` directly with the quarantine state set. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes The default cap (60s) is deliberately well above a normal slow-but-completing compression so the original saturation protection is unchanged in practice; it only ever fires for a worker that has run far past its deadline. Tunable via `HEADROOM_COMPRESSION_QUARANTINE_MAX_SECONDS`. The `"released"` quarantine metric and a one-time warning make the recovery observable. The "unit tests pass locally" box is unchecked because the added `create_app` test imports the ML stack (OOM on this box); it runs under the normal CI job, and the state machine is verified by the standalone proof above. |
||
|
Abhay Singh
|
74275b7c3e
|
fix(subscription): dedup transcript usage by message id (#2340 token inflation) (#2408)
## Description Addresses the usage-inflation part of #2340. `compute_window_tokens` (`headroom/subscription/session_tracking.py`) sums `message.usage` for every transcript line whose timestamp falls in the window: ```python for line in _read_transcript_lines(path): ... usage = msg.get("usage") if not usage: continue _add_usage_to_tokens(totals, usage) ``` But Claude Code can store a single assistant response across **multiple transcript lines** (e.g. one entry per content block), and each of those lines carries the **same request-level `message.usage`**. Summing per line therefore multiplies that one response's tokens by its block count. #2340 observed a single 420,609-input-token response counted **19 times** (~8M attributed input tokens from one record), which is most of the reported window-total inflation. ## Fix Count each response's usage once, keyed by the Anthropic `message.id` (unique per response): ```python seen_message_ids: set[str] = set() ... msg_id = msg.get("id") if isinstance(msg_id, str) and msg_id: if msg_id in seen_message_ids: continue seen_message_ids.add(msg_id) _add_usage_to_tokens(totals, usage) ``` Entries without a `message.id` keep the previous per-line behavior, so this only ever removes true duplicates: a response is de-duplicated only when the exact same (unique) message id appears more than once, and distinct responses are unaffected. Scope: this fixes the token-accounting inflation only. The separate retry-amplification / `tool_search_tool_result` SSE-502 behavior described in the same issue is a different code path and is not touched here. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/subscription/session_tracking.py`: dedup usage accumulation by `message.id` in `compute_window_tokens`. - `tests/test_subscription_session_tracking.py`: add a test where one response is stored across three lines (plus a distinct response and an id-less line) and assert its usage is counted once. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uvx ruff@0.15.17 check headroom/subscription/session_tracking.py tests/test_subscription_session_tracking.py All checks passed! $ uvx mypy@1.20.2 --ignore-missing-imports headroom/subscription/session_tracking.py Success: no issues found in 1 source file # session_tracking is import-light, so I ran the exact logic against the real # module in the project venv (uv sync): a response stored on 3 lines yields # input=106 (100 once + 5 + 1 id-less), not 306. ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, project venv (`uv sync --extra proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`. - Exact command / steps: wrote a transcript with `msg_dup` repeated on three lines (same usage, 100/10), one distinct `msg_other` (5/2), and one id-less line (1/1); called the real `compute_window_tokens` over the window. - Observed result: `input == 106` and `output == 13` (the duplicated response counted once, the id-less line still counted); the pre-fix code would report `input == 306`. Because `session_tracking` has no heavy imports, this ran against the actual module. - Not tested: a live Claude Code transcript with real multi-block responses; the added unit test reproduces the multi-line-per-response shape. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes `session_tracking` is a light module, so I verified against the real code in the venv (output above) in addition to the added test. This is deliberately scoped to the usage-double-count sub-part of #2340; the retry-amplification/SSE side is separate and untouched. Keyed on `message.id` so it is safe by construction: no id or a unique id behaves exactly as before. |
||
|
Matt Van Horn
|
1db6d88ab4
|
fix(wrap): honor Copilot OAuth wire-api override and model default (#2387)
## Description `headroom wrap copilot -- --model gpt-5.4` fails with `400 The requested model is not available for integrator "copilot-language-server"`, even though the same GitHub account uses GPT-5.4 fine in the native `copilot` CLI. On the hosted Copilot OAuth path (authenticated via `headroom copilot-auth`, no `--subscription`), `headroom/cli/wrap.py` forced the `completions` wire API for every non-`--subscription` launch and ignored a caller-supplied `COPILOT_PROVIDER_WIRE_API`. GPT-5.x / o-series reasoning models need the `responses` wire API. The OAuth path now honors a valid inherited `COPILOT_PROVIDER_WIRE_API` (`completions` or `responses`) and otherwise uses the model-aware default via `_copilot_default_wire_api_for_model(selected_model)`, so GPT-5.x routes to `responses` while GPT-4.1 stays on `completions`. The `--subscription` path is unchanged. This supersedes the earlier closed #2243, which mixed the fix with unrelated proxy/CI changes and hit merge conflicts. This PR ships only the `headroom/cli/wrap.py` wire-api selection plus regression tests, rebased clean on `main`. Closes #2222 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Hosted Copilot OAuth launch resolves the wire API from an explicit `COPILOT_PROVIDER_WIRE_API` env value when it is `completions`/`responses`, else from `_copilot_default_wire_api_for_model(selected_model)` instead of a hardcoded `completions`. The `subscription`-only gating on the model-aware default is removed so OAuth and subscription paths pick the same model-correct wire API. ## Testing - [x] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ .venv/bin/python -m pytest tests/test_cli/test_wrap_copilot.py tests/test_provider_copilot_wrap.py -q 70 passed in 0.28s ``` New tests cover: the OAuth path honoring an inherited `COPILOT_PROVIDER_WIRE_API`, the OAuth path resolving GPT-5.4 to `responses`, and `default_wire_api_for_model("gpt-5.4")` returning `responses`. ## Real Behavior Proof - Environment: local checkout, Python 3.14, target tests only. - Exact command / steps: `.venv/bin/python -m pytest tests/test_cli/test_wrap_copilot.py tests/test_provider_copilot_wrap.py -q` - Observed result: 70 passed; the OAuth-path tests assert `env["COPILOT_PROVIDER_WIRE_API"] == "responses"` for GPT-5.x and honor an inherited override. - Not tested: the end-to-end live Copilot `400` reproduction, which needs a real Copilot OAuth session plus a GPT-5.x request. The wire-api selection that caused the `400` is covered by the unit tests above. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — CLI behavior change covered by the unit tests above. ## Additional Notes The change is scoped to the wire-api selection line; the `--subscription` path and the existing explicit `--wire-api` CLI flag are untouched. AI was used for assistance. --------- Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> |
||
|
Andrei Boldyrev
|
f6398a6476
|
fix(proxy): port session-sticky beta headers to the Rust proxy (#2381)
## Description The Python proxy protects prompt caches with `SessionBetaTracker` (PR-A6, `headroom/proxy/helpers.py`): interactive clients (Claude Code, Codex CLI) may drop an `anthropic-beta` / `openai-beta` token between turn N and turn N+1 of the same conversation, and since beta headers are part of the bytes that determine the upstream prefix-cache key, the drop rotates the key and the provider re-writes the whole prefix at the customer's cost. The tracker unions the client's tokens with everything previously seen for that `(provider, session)` and forwards the union — a documented operator contract (`docs/configuration.mdx`, "Session Beta Header Tracking"). The Rust proxy has no equivalent, and Phase H (#2258) deletes the tracker together with `helpers.py` and its test file (`tests/test_anthropic_beta_session_sticky.py`). None of the Phase A–G plans port it (Phase F consumes beta headers for auth-mode classification only), so the protection would silently not survive the migration — and the Phase-H gate "Cache-hit-rate parity with direct upstream confirmed" can't catch the loss, because re-injection makes proxied traffic *beat* direct upstream on cache hits; when the mechanism disappears, proxied traffic degrades *to* direct-upstream levels, which that comparison reads as parity. This PR ports the tracker semantics into the Rust proxy so the protection lives in the codebase Phase H keeps. Closes #2380 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) (New Rust functionality, but a parity port of already-shipped, already-documented Python behavior — the PR title uses `fix:` per `REALIGNMENT/INDEX.md`: "Commit prefix: `fix:` for Rust-migration phase commits".) ## Changes Made - **`cache_stabilization/beta_sticky.rs`** — the tracker: bounded LRU (1000 sessions, same sizing rationale and `# Panics` contract as the drift detector's capacity) keyed by `(provider, session)`, storing the per-session ordered token list. Union preserves first-seen order; dedup is case-insensitive with first-seen casing winning; lookups touch recency; overflow evicts the oldest — mirroring the Python tracker. The header-plumbing lives in the module too (`apply_sticky_betas`), so the merge is unit-testable without booting a proxy. - **`proxy.rs` wiring** — on the intercepted POST routes (`/v1/messages`, `/v1/chat/completions`, `/v1/responses`), right after the drift-detector observation, reusing the drift detector's `derive_session_key` output so both cache-stability subsystems agree on conversation identity. - **`config.rs`** — `--beta-header-sticky` / `HEADROOM_PROXY_BETA_HEADER_STICKY` (`enabled` default; `disabled` forwards the client value verbatim and keeps no state), mirroring the `StripInternalHeaders` flag pattern and the existing `HEADROOM_*` → `HEADROOM_PROXY_*` Python→Rust env pairing. Since the merge runs inside the compression interceptor, startup logs a warning when the flag is `enabled` while `--compression` is off, and both the CLI doc and the docs row state the dependency. - **`tests/integration_beta_header_sticky.rs`** — 9 end-to-end tests against a wiremock upstream asserting the headers/bytes the upstream actually receives; 21 unit tests port the behavioral contract from `tests/test_anthropic_beta_session_sticky.py` and cover the header-map plumbing. - **`docs/content/docs/configuration.mdx`** — one row for `HEADROOM_PROXY_BETA_HEADER_STICKY` next to the existing Python/Rust flag pairs. ## Testing - [x] Unit tests pass (`cargo test -p headroom-proxy`; Python side via `make ci-precheck-python` — `pytest` subset, 174 passed) - [x] Linting passes (`cargo clippy --all-targets` — 0 warnings; `cargo fmt --check` clean; Rust-only change, so `ruff`/`mypy` are covered by the untouched-Python `ci-precheck-python` build) - [ ] Type checking passes (`mypy headroom`) — N/A, no Python files touched - [x] New tests added for new functionality - [x] Manual testing performed (RED/GREEN before-and-after runs below) ### Test Output ```text $ cargo test -p headroom-proxy --lib beta_sticky test result: ok. 21 passed; 0 failed; 0 ignored; 0 measured; 248 filtered out; finished in 0.03s $ cargo test -p headroom-proxy --test integration_beta_header_sticky test result: ok. 9 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.07s $ cargo test -p headroom-proxy # full crate: 37 suites, all ok $ cargo clippy -p headroom-proxy --all-targets # 0 warnings $ make ci-precheck-rust ci-precheck-python ci-precheck-commitlint # green ``` ## Real Behavior Proof - Environment: macOS arm64 (Darwin 24.6), `rustc 1.95.0`, real Rust proxy booted on an ephemeral port in front of a wiremock upstream (`tests/common::start_proxy_with`, `compression = true`). - Exact command / steps: two-turn conversation through the proxy — turn 1 `POST /v1/messages` with `anthropic-beta: context-management-2025-06-27,interleaved-thinking-2025-05-14`; turn 2, same conversation, client drops the second token. The wiremock responder captures the headers the upstream actually receives (`cargo test -p headroom-proxy --test integration_beta_header_sticky`). - Observed result: **before** the port (test written first, run against the unmodified proxy) the upstream sees the shrunken token set and the prefix-cache key rotates — ```text assertion `left == right` failed: turn 2 must re-inject the dropped token so the upstream prefix-cache key stays byte-stable left: Some("context-management-2025-06-27") right: Some("context-management-2025-06-27,interleaved-thinking-2025-05-14") ``` **After** the port the same scenario passes: the upstream receives the full union on turn 2, the internal `x-headroom-session-id` never crosses the upstream boundary, and the forwarded body is SHA-256-identical to what the client sent (asserted by `body_bytes_stay_byte_equal_while_header_is_rewritten`). - Not tested: live traffic against a real provider upstream (wiremock only); the WebSocket path and Bedrock/Vertex routes (out of scope — see Additional Notes). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A (proxy behavior; see Real Behavior Proof). ## Additional Notes Design decisions, and where I'd like reviewer judgment: 1. **Applies to all auth modes, like the Python handler.** The Phase-E module doctrine gates *body*-mutating normalizers on PAYG; this mechanism mutates headers only, and the Python source of truth applies it unconditionally — an auth-mode gate here would create a behavioral delta exactly where the PR's purpose is behavior preservation. It's also stealth-consistent by construction: the union only ever contains tokens this client itself sent (Headroom-added tokens are never recorded), `auth_mode.rs`'s own docs name "beta-header drift voids them" as the OAuth cache hazard (stickiness is the anti-drift), and F2's `CompressionPolicy` has no beta field — no gate is structurally expected. I've extended the `cache_stabilization/mod.rs` taxonomy with a third category ("re-echo client-sent state") to keep the module doctrine honest. Flagging explicitly since invariant #10 ("no beta drift") is subscription-critical: if you read it as "forward beta verbatim on Subscription", say so and I'll add the gate. 2. **One deliberate divergence from Python: sessions are keyed per conversation, not per `(model, system)` bucket.** The Python tracker keys on the store session id — explicit header, else a hash of model + leading system prompt — so a Claude Code session and every one of its subagents share one token union and cross-inherit tokens; two *different users* behind an org proxy with the same (model, system) do too. This port keys on the drift detector's conversation-aware key (#2301), so each conversation keeps its own union (pinned by `separate_conversations_do_not_leak_tokens`). That's the same conflation defect #2085/#2193/#2301 chased out of the other session-sticky subsystems, and it makes "the union only contains tokens this client sent" actually true — under the Python fallback key it isn't (cross-user union). Cost: Python's accidental cross-conversation repair is gone, and an OAuth access-token refresh mid-conversation re-keys the session (one turn forwards verbatim, then re-learns — fails safe). 3. **Repeated header lines are joined per RFC 9110 list semantics before recording.** A client sending two `anthropic-beta` lines gets both recorded; a later rewrite collapses to one line carrying the full set. (Reading only the first line — or Python's actual behavior, which keeps only the *last* line via its `dict(headers)` collapse — can shrink the upstream token set mid-conversation when a rewrite fires.) 4. **Scope: the three intercepted HTTP routes.** With the compression interceptor off the proxy is a strict byte-pipe (Phase-A invariant) — no header mutation, hence the startup warning. WebSocket keeps its behavior (Python's WS site keys on a per-connection UUID, so cross-turn accumulation is a near-no-op there; the Rust WS tunnel doesn't touch beta headers). Bedrock/Vertex are skipped by the same match that skips the drift detector (betas travel in the body as `anthropic_beta` on Bedrock). 5. **Log discipline**: `event=beta_header_merge` carries token *counts* only (beta tokens can carry experiment IDs; same privacy contract as Python's `log_beta_header_merge`, plus the drift detector's hashed session-key prefix instead of Python's raw session id). One deviation from Python's unconditional info: the no-op case logs at debug, matching the drift detector's silent-on-stable precedent — an info-level `beta_header_merge` always marks an actual cache-affecting rewrite. 6. **Capacity is a const (1000), not a flag** — following the drift-detector precedent rather than Python's `HEADROOM_BETA_TRACKER_MAX_SESSIONS` env var. Happy to make it configurable if you'd rather keep that operator knob. 7. **Fail-open everywhere**: non-ASCII client values are forwarded verbatim with nothing recorded; a poisoned tracker lock forwards the client value verbatim; an unencodable union (unreachable — every token came from a parsed header value) logs and forwards verbatim. The protection never delays or drops a request. |