mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
2448 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Tejas Chopra
|
01f86665e5 |
fix(opencode): don't preload a missing transport shim into child processes
The wrap transport plugin appended `NODE_OPTIONS=--import=<plugin dir>/../hook-shim/handler.js` to its own env and to every child it spawns. That path only resolves in a repo checkout (`plugins/opencode/dist/` has a `hook-shim/` sibling). Wheel installs load the standalone bundle from `headroom/providers/opencode/_dist/`, where no shim exists — `hook-shim/` lives under `plugins/` and maturin only ships files under `headroom/`. Every Node child then aborted with ERR_MODULE_NOT_FOUND before running, including OpenCode's stdio MCP servers, which surfaced as `<server> MCP error -32000: Connection closed` for third-party servers (codegraph, firecrawl) while Headroom's own Python MCP server stayed up. Resolve the shim only when it is present on disk, and skip the NODE_OPTIONS mutation otherwise: children go direct instead of dying. Checkout builds keep child-process transport hooking unchanged. |
||
|
JD Davis
|
64e203931b
|
fix(deps): enforce audited transitive dependency floors (#2791)
## Description Enforces patched minimum versions for the vulnerable transitive `aiohttp` and `cryptography` dependencies so future lockfile refreshes cannot reintroduce the pip-audit failures affecting open pull requests. Related to the shared Security / pip-audit failures across open PRs. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Enforces `aiohttp>=3.14.3` for PYSEC-2026-3545/3546/3547. - Enforces `cryptography>=50.0.0` for PYSEC-2026-3552/3553/3554. - Synchronizes the project version recorded in `uv.lock` with `pyproject.toml`. ## Testing - [x] Dependency audit passes (`pip-audit`) - [x] Lockfile validation passes (`uv lock --check`) - [ ] Unit tests pass (`pytest`) - [ ] Type checking passes (`mypy headroom`) - [x] Manual verification performed ### Test Output ```text $ uv lock --check Resolved 269 packages $ uv export --frozen --no-dev --no-emit-project --no-hashes --extra all --format requirements-txt | uvx --python 3.12 pip-audit -r /dev/stdin No known vulnerabilities found ``` ## Real Behavior Proof - Environment: Local macOS worktree using CPython 3.12.13 and the frozen production dependency export. - Exact command / steps: Validated the lockfile, exported every production dependency with the `all` extra, and audited that exact export with pip-audit. - Observed result: The lockfile resolved successfully and pip-audit reported no known vulnerabilities. - Not tested: Publishing or deployment; the refreshed GitHub CI suite covers builds, wheels, containers, security scans, and platform tests. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project style guidelines - [x] I have performed a self-review of my changes - [x] No explanatory code comments are required beyond the PYSEC constraint annotations - [x] Documentation changes are not required for transitive security floors - [x] My changes generate no new local warnings - [x] The dependency audit proves the security fix is effective - [ ] Full repository tests are delegated to GitHub CI - [x] I did not edit `CHANGELOG.md`; release-please owns it ## Screenshots (if applicable) N/A — dependency metadata only. ## Additional Notes The earlier Docker-native failure was a transient Docker Hub HTTP 502 while resolving `python:3.13-slim`; the build did not reach project code. A fresh CI suite is running on the current head. Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net> |
||
|
JD Davis
|
f236ef2e31
|
test(ccr): cross SQLite max lifetime boundary (#2794)
## Description
Fixes the failing Rust test on `main` after #2669 made SQLite CCR
entries valid at the exact TTL boundary. The integration test waited
only 3.3 seconds for a three-second ceiling; unix-second truncation can
represent that as exactly three seconds, so the entry is correctly still
valid. The test now crosses a guaranteed four-second elapsed boundary.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Extend the max-lifetime test's access loop from four to five 700 ms
gaps.
- Document why four gaps can land on the valid equality boundary and why
five are deterministic.
- Leave production SQLite TTL behavior and defaults unchanged.
## Testing
- [x] Unit tests pass (`cargo test -p headroom-core --test
ccr_backends`)
- [x] Formatting passes (`cargo fmt --all -- --check`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
cargo test -p headroom-core --test ccr_backends
test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
5 consecutive repetitions of the previously failing test:
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 11 filtered out
```
## Real Behavior Proof
- Environment: macOS, Rust workspace at
`
|
||
|
Tejas Chopra
|
e9a24f3ec1
|
fix(beacon): report all-layers savings, not context-compression only (#2796)
## Description
`rates.saved_pct` and `rates.yield_pct` in the beacon payload divide
`tokens_saved` by `original` / `attempted`. Tool-schema deferral never
lands in either denominator — `outcome.py` says so explicitly, and
`tokens.tool_saved` exists precisely because of it — so every beacon
rate silently reports context compression only.
On a tool-heavy fleet that is not a rounding difference. Across the
first 516 sessions in the corpus the beacon reads **2.80%** where the
dashboard headline for the same traffic reads **12.82%**: 157.6M context
tokens vs 803.9M all-layers, with 646.3M of tool-schema deferral missing
from the ratio.
`headroom/proxy/server.py` already resolved this for the dashboard in
#2737 — `savings_percent` is `all_layers_saved / (input +
all_layers_saved)` and `active_savings_percent` puts tool savings on
both sides of the ratio. The beacon was never brought along. This does
that.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/telemetry/session.py`: add `rates.all_layers_saved_pct` and
`rates.all_layers_yield_pct`, computed the way `server.py` builds
`savings_percent` / `active_savings_percent` — tool savings added to
**both** sides, since deferred schemas were attempted work that
succeeded whole.
- `headroom/telemetry/session.py`: extend the `demo()` self-test to
assert both new rates against the existing tool-heavy fixture.
- `deploy/beacon/query.sh`: add an `all_layers_pct` column to the fleet
summary, so the reader stops showing the understated number too.
**Kept alongside `saved_pct` rather than folded into it.** Every row
already in the corpus means context-only under that name; redefining it
would make old and new rows non-comparable with no field to tell them
apart.
**`SCHEMA_VERSION` deliberately stays at 1.** The change is purely
additive, nothing reads the field, and the query uses `union_by_name =
true`, so old and new rows mix cleanly. Happy to bump it if maintainers
want the marker.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m headroom.telemetry.session
ok
$ python -m pytest tests/test_savings_tool_search_aggregation.py tests/test_outcome_dual_ruler_funnel.py -q
FAILED tests/test_outcome_dual_ruler_funnel.py::test_ledger_delta_stays_on_the_local_ruler
FAILED tests/test_outcome_dual_ruler_funnel.py::test_ledger_falls_back_to_billed_when_local_omitted
2 failed, 5 passed, 3 warnings in 4.42s
$ ruff check headroom/
All checks passed!
$ mypy --python-version 3.12 headroom/telemetry/session.py
Success: no issues found in 1 source file
```
The two `test_outcome_dual_ruler_funnel.py` failures are **pre-existing
on `main`, not caused by this PR** — verified by `git stash`-ing the
change and re-running:
```text
$ git stash && python -m pytest tests/test_outcome_dual_ruler_funnel.py -q
FAILED tests/test_outcome_dual_ruler_funnel.py::test_ledger_delta_stays_on_the_local_ruler
FAILED tests/test_outcome_dual_ruler_funnel.py::test_ledger_falls_back_to_billed_when_local_omitted
2 failed, 3 passed, 3 warnings in 3.71s
```
## Real Behavior Proof
- **Environment:** macOS 25.4.0 arm64, Python 3.12.6, repo `.venv`,
branch rebased on `upstream/main` @ `
|
||
|
Parideboy
|
b6f9877c78
|
fix(tokenizer): coerce non-string tool_call fields before counting (#2801)
## Description
`/v1/compress` returned HTTP 503 with an unhandled `TypeError` when a
message carried a `tool_calls[].function.arguments` value that was not a
string. `arguments` is a JSON *string* per the OpenAI spec, but
OpenAI-compatible upstreams do emit `None` or a raw object there, and
every token counter passed the value straight to `tiktoken.encode()`.
Because the malformed message persists in conversation history, the
failure was sticky: every later request replaying that history failed
too, regardless of destination provider.
Reported in #2782. The exact repro in that issue (`arguments: null`) no
longer raises — `count_text` grew a falsy guard since 0.33.0 — but the
root cause is still live for any *truthy* non-string, which I reproduced
against all four counters on `main` before the fix.
## Type of Change
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Documentation update
- [ ] Refactor / internal cleanup
## Changes Made
- `headroom/tokenizers/base.py`: new `coerce_countable_text()`. Strings
pass through untouched, `None` counts as nothing, dict/list/tuple are
JSON-serialized, anything else falls back to `str()`. The serialized
form is capped at 200K chars so a malformed upstream can't turn a token
*estimate* into a multi-megabyte encode.
- Applied at the tool-call field sites (`function.name`,
`function.arguments`, `id`, and the legacy `function_call`) in
`tokenizers/base.py`, `tokenizers/tiktoken_counter.py`,
`providers/openai.py`, `providers/openai_compatible.py`,
`providers/anthropic.py`.
- Guarded `{"function": null}` / `{"id": null}`, which reach the same
encode path.
- New test file `tests/test_tool_call_arguments_not_a_string.py` (11
cases).
Serializing dicts rather than the one-liner suggested in the issue
(`str(func.get("arguments") or "")`) is deliberate: `str()` on a dict
yields Python repr with single quotes, which is not what the upstream
would have billed, and it is unbounded.
## Testing
- [x] Existing tests pass
- [x] New tests added for the fix
- [ ] Manual testing performed
New tests:
```
$ python -m pytest tests/test_tool_call_arguments_not_a_string.py -q
tests\test_tool_call_arguments_not_a_string.py ........... [100%]
============================= 11 passed in 0.40s ==============================
```
Surrounding tokenizer/provider suites, unchanged:
```
$ python -m pytest tests/test_tool_call_arguments_not_a_string.py tests/test_tokenizer.py \
tests/test_tokenizers.py tests/test_tokenizers \
tests/test_provider_counter_content_blocks.py tests/test_provider_tokenizer_one_ruler.py -q
tests\test_provider_counter_content_blocks.py ............ [ 91%]
tests\test_provider_tokenizer_one_ruler.py ......... [100%]
======================= 91 passed, 14 skipped in 1.50s ========================
```
Lint/format on the touched files:
```
$ ruff check <touched files> && ruff format --check <touched files>
All checks passed!
6 files already formatted
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.13.11, pytest 9.1.1, repo at
`upstream/main` (
|
||
|
Agistaris
|
d0a86d409f
|
fix(ccr): preserve exact SQLite TTL boundary (#2669)
## Description SQLite CCR timestamps have whole-second resolution. Expiring a row when `last_accessed + ttl == now` or `created_at + max_lifetime == now` can shorten the configured lifetime by almost one second. This change keeps entries valid at the exact boundary and expires them one second later. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Use strict expiration predicates for idle TTL and maximum lifetime. - Keep lookup predicates valid at the exact boundary. - Add deterministic fixed-time tests for both boundaries. ## Testing - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text running 3 tests test ccr::backends::sqlite::tests::exact_max_lifetime_boundary_is_still_valid ... ok test ccr::backends::sqlite::tests::exact_idle_ttl_boundary_is_still_valid ... ok test transforms::code_compressor::tests::english_exact_token_match_unchanged ... ok test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 911 filtered out ``` ## Real Behavior Proof - Environment: Linux x86_64, repository Rust toolchain. - Exact command / steps: `cargo test -p headroom-core exact_` - Observed result: Both SQLite boundary tests returned the stored payload at the exact configured boundary and removed it one second later. The focused command passed 3/3 selected tests, including one unrelated existing exact-token test. - Not tested: Live provider or model traffic; the change is isolated to the deterministic Rust SQLite backend. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project style guidelines - [x] I have performed a self-review of my code - [x] I have commented the boundary behavior - [ ] I have made corresponding documentation changes (not applicable; behavior and tests are local to the backend) - [x] My changes generate no new warnings - [x] I have added tests that prove the fix is effective - [x] New and existing relevant tests pass locally with my changes - [x] I did not edit `CHANGELOG.md` ## Screenshots (if applicable) Not applicable. ## Additional Notes Rust formatting and `headroom-core` Clippy pass. A broader package run passed 994 tests with three ignored; two unrelated ONNX parity tests were excluded after reproducing their pre-existing futex stall. |
||
|
pgjh
|
a97b82413b
|
fix(proxy): unwrap Hermes tool_call bridge in tool name map (#2717)
## Description
Hermes Agent (NousResearch/hermes-agent) loads on-demand ("deferred")
tools via a `tool_search` → `tool_describe` → `tool_call` indirection.
On the wire, the emitted tool call is named **`tool_call`**, and the
REAL tool name lives inside the arguments payload:
```json
{
"id": "call_abc123",
"type": "function",
"function": {
"name": "tool_call",
"arguments": "{\"name\": \"read_file\", \"arguments\": {\"path\": \"/etc/hostname\"}}"
}
}
```
`ContentRouter._build_tool_name_map` only reads
`tool_calls[].function.name`, so it maps `tool_call` → `"tool_call"`
instead of the real tool name.
**Consequence**: `HEADROOM_EXCLUDE_TOOLS` /
`HEADROOM_PROTECT_TOOL_RESULTS` silently no-op for **ALL** deferred
tools (`read_file`, `write_file`, `search_files`, `mcp__*`,
`headroom_retrieve`, etc.). Their outputs get lossy-compressed even when
explicitly whitelisted, and `headroom_retrieve` falls into an endless
re-compression loop (`<<ccr:hash>>` → retrieve → re-compress →
`original_tokens: 0`).
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
Add an `unwrap_tool_call_name(name, arguments)` helper in `config.py`
(beside the existing `_tool_name_aliases`) and apply it at the **3
sites** where the tool_call_id → tool_name map is built:
1. **OpenAI chat path** — `content_router.py` `_build_tool_name_map`
(`tool_calls[].function`)
2. **Anthropic path** — `content_router.py` `_build_tool_name_map`
(`tool_use` blocks)
3. **Responses API path** — `openai.py`
`_compress_openai_responses_live_text_units_with_router`
(`function_call` items)
The helper:
- Passes non-wrapper names through unchanged
- Parses the arguments payload (JSON string or dict) and extracts the
inner `name`
- Fails open (returns the wrapper name) on malformed/unparseable
payloads — safe for all other clients
After unwrap, `is_tool_excluded()`/protect-list matching sees the real
tool name, so whitelists work for deferred tools exactly as they already
did for classic tools.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] New tests added for new functionality
- [x] Manual testing performed
`tests/test_hermes_tool_call_unwrap.py` — **14 tests**, all pass:
- Helper unit tests: passthrough, None/bad-JSON/missing-name fail-open,
unwrap (web_search, read_file, mcp__*, dict-args form)
- Whitelist activation: unwrapped name + `is_tool_excluded` against
`DEFAULT_EXCLUDE_TOOLS`
- Integration: `_build_tool_name_map` with OpenAI-format `tool_call`
wrapper and Anthropic-format `tool_use` wrapper
- Regression guard: documents that `tool_call` itself is NOT in
`DEFAULT_EXCLUDE_TOOLS` (the pre-fix failure mode)
### Test Output
```text
$ uv run pytest tests/test_hermes_tool_call_unwrap.py tests/test_content_router_exclude_tools.py tests/test_config.py -q
56 passed in 1.24s
$ uv run ruff check .
All checks passed!
$ uv run ruff format --check .
All checks passed!
```
## Real Behavior Proof
- Environment: Ubuntu 24.04 (kernel 7.0.0-28), Python 3.11.15, headroom
built from source at `v0.33.0-5-g6d5516dc` via `uv sync` (maturin), Rust
1.95.0 toolchain. Headroom proxy 0.34.0-dev running as a systemd user
service, `HEADROOM_PROTECT_TOOL_RESULTS=read_file,headroom_retrieve`,
mode=token. Upstream: local new-api-compatible gateway (deepseek-v4-pro
/ GLM-5.2).
- Exact command / steps: Client = Hermes Agent (fresh session via
`hermes chat -q --provider newapi2`, traffic routed through the headroom
proxy at 127.0.0.1:8788). Trigger a deferred `read_file` tool call and
observe `/stats` → `recent_requests`.
- Observed result: After fix, live traffic evidence shows request
`hr_1785683282_000016` (GLM-5.2 session) with `transforms_applied:
["router:excluded:tool", "openai:chat:tool_schema_compaction"]` — the
deferred `read_file` tool was recognized and excluded (whitelist hit),
21918 → 17218 input tokens. Before the fix this request showed no
`router:excluded:tool` for deferred tools — they were compressed. E2E
unit-level proof (`test_build_tool_name_map_exclusion_after_unwrap` +
standalone pipeline run): a `tool_call`-wrapped `read_file` output
(~9KB, 100 lines) passed through `ContentRouter.apply()` **verbatim**
(byte-identical), while a non-whitelisted tool (Bash) in the same
pipeline was compressed — proving the whitelist now works and the
pipeline still compresses normally.
- Not tested: Anthropic native clients (Claude Code) and Responses-API
clients (Codex) end-to-end — the patch sites for those paths are covered
by unit tests only. No new dependencies; no public API changes.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
No documentation changes needed (no public API change; behavior is
internal to the proxy tool-name map). Follow-up: end-to-end verification
with Anthropic/Responses-API clients once maintainers can run the proxy
CI on those paths.
Co-authored-by: pgjh <pgjh@users.noreply.github.com>
|
||
|
dependabot[bot]
|
267c2bdcb5
|
deps: bump postcss from 8.5.19 to 8.5.25 in /sdk/typescript (#2747)
Bumps [postcss](https://github.com/postcss/postcss) from 8.5.19 to 8.5.25. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/postcss/postcss/releases">postcss's releases</a>.</em></p> <blockquote> <h2>8.5.25</h2> <ul> <li>Fixed 8.5.17 visitor regression.</li> <li>Fixed <code>list.split()</code> for non-string values (by <a href="https://github.com/amir-rezaei"><code>@amir-rezaei</code></a>).</li> </ul> <h2>8.5.24</h2> <ul> <li>Preserve the BOM after the processing (by <a href="https://github.com/hdimer"><code>@hdimer</code></a>).</li> </ul> <h2>8.5.23</h2> <ul> <li>Do not load source map without <code>opts.from</code> for security reasons.</li> </ul> <h2>8.5.22</h2> <ul> <li>Fixed custom property losing semicolon before a comment (by <a href="https://github.com/sarathfrancis90"><code>@sarathfrancis90</code></a>).</li> </ul> <h2>8.5.21</h2> <ul> <li>Fixed childless at-rule losing semicolon before comment (by <a href="https://github.com/sarathfrancis90"><code>@sarathfrancis90</code></a>).</li> <li>Fixed docs (by <a href="https://github.com/isker"><code>@isker</code></a>).</li> </ul> <h2>8.5.20</h2> <ul> <li>Fixed missing space if <code>AtRule#params</code> is set after (by <a href="https://github.com/sarathfrancis90"><code>@sarathfrancis90</code></a>).</li> <li>Fixed mixing AST error on warnings (by <a href="https://github.com/MahinAnowar"><code>@MahinAnowar</code></a>).</li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/postcss/postcss/blob/main/CHANGELOG.md">postcss's changelog</a>.</em></p> <blockquote> <h2>8.5.25</h2> <ul> <li>Fixed 8.5.17 visitor regression.</li> <li>Fixed <code>list.split()</code> for non-string values (by <a href="https://github.com/amir-rezaei"><code>@amir-rezaei</code></a>).</li> </ul> <h2>8.5.24</h2> <ul> <li>Preserve the BOM after the processing (by <a href="https://github.com/hdimer"><code>@hdimer</code></a>).</li> </ul> <h2>8.5.23</h2> <ul> <li>Do not load source map without <code>opts.from</code> for security reasons.</li> </ul> <h2>8.5.22</h2> <ul> <li>Fixed custom property losing semicolon before a comment (by <a href="https://github.com/sarathfrancis90"><code>@sarathfrancis90</code></a>).</li> </ul> <h2>8.5.21</h2> <ul> <li>Fixed childless at-rule losing semicolon before comment (by <a href="https://github.com/sarathfrancis90"><code>@sarathfrancis90</code></a>).</li> <li>Fixed docs (by <a href="https://github.com/isker"><code>@isker</code></a>).</li> </ul> <h2>8.5.20</h2> <ul> <li>Fixed missing space if <code>AtRule#params</code> is set after (by <a href="https://github.com/sarathfrancis90"><code>@sarathfrancis90</code></a>).</li> <li>Fixed mixing AST error on warnings (by <a href="https://github.com/MahinAnowar"><code>@MahinAnowar</code></a>).</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
ff4e0167bb
|
deps: bump postcss from 8.5.19 to 8.5.25 in /plugins/opencode (#2748)
Bumps [postcss](https://github.com/postcss/postcss) from 8.5.19 to 8.5.25. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/postcss/postcss/releases">postcss's releases</a>.</em></p> <blockquote> <h2>8.5.25</h2> <ul> <li>Fixed 8.5.17 visitor regression.</li> <li>Fixed <code>list.split()</code> for non-string values (by <a href="https://github.com/amir-rezaei"><code>@amir-rezaei</code></a>).</li> </ul> <h2>8.5.24</h2> <ul> <li>Preserve the BOM after the processing (by <a href="https://github.com/hdimer"><code>@hdimer</code></a>).</li> </ul> <h2>8.5.23</h2> <ul> <li>Do not load source map without <code>opts.from</code> for security reasons.</li> </ul> <h2>8.5.22</h2> <ul> <li>Fixed custom property losing semicolon before a comment (by <a href="https://github.com/sarathfrancis90"><code>@sarathfrancis90</code></a>).</li> </ul> <h2>8.5.21</h2> <ul> <li>Fixed childless at-rule losing semicolon before comment (by <a href="https://github.com/sarathfrancis90"><code>@sarathfrancis90</code></a>).</li> <li>Fixed docs (by <a href="https://github.com/isker"><code>@isker</code></a>).</li> </ul> <h2>8.5.20</h2> <ul> <li>Fixed missing space if <code>AtRule#params</code> is set after (by <a href="https://github.com/sarathfrancis90"><code>@sarathfrancis90</code></a>).</li> <li>Fixed mixing AST error on warnings (by <a href="https://github.com/MahinAnowar"><code>@MahinAnowar</code></a>).</li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/postcss/postcss/blob/main/CHANGELOG.md">postcss's changelog</a>.</em></p> <blockquote> <h2>8.5.25</h2> <ul> <li>Fixed 8.5.17 visitor regression.</li> <li>Fixed <code>list.split()</code> for non-string values (by <a href="https://github.com/amir-rezaei"><code>@amir-rezaei</code></a>).</li> </ul> <h2>8.5.24</h2> <ul> <li>Preserve the BOM after the processing (by <a href="https://github.com/hdimer"><code>@hdimer</code></a>).</li> </ul> <h2>8.5.23</h2> <ul> <li>Do not load source map without <code>opts.from</code> for security reasons.</li> </ul> <h2>8.5.22</h2> <ul> <li>Fixed custom property losing semicolon before a comment (by <a href="https://github.com/sarathfrancis90"><code>@sarathfrancis90</code></a>).</li> </ul> <h2>8.5.21</h2> <ul> <li>Fixed childless at-rule losing semicolon before comment (by <a href="https://github.com/sarathfrancis90"><code>@sarathfrancis90</code></a>).</li> <li>Fixed docs (by <a href="https://github.com/isker"><code>@isker</code></a>).</li> </ul> <h2>8.5.20</h2> <ul> <li>Fixed missing space if <code>AtRule#params</code> is set after (by <a href="https://github.com/sarathfrancis90"><code>@sarathfrancis90</code></a>).</li> <li>Fixed mixing AST error on warnings (by <a href="https://github.com/MahinAnowar"><code>@MahinAnowar</code></a>).</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
cd60ee9ae8
|
deps: bump postcss from 8.5.19 to 8.5.25 in /plugins/openclaw (#2749)
Bumps [postcss](https://github.com/postcss/postcss) from 8.5.19 to 8.5.25. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/postcss/postcss/releases">postcss's releases</a>.</em></p> <blockquote> <h2>8.5.25</h2> <ul> <li>Fixed 8.5.17 visitor regression.</li> <li>Fixed <code>list.split()</code> for non-string values (by <a href="https://github.com/amir-rezaei"><code>@amir-rezaei</code></a>).</li> </ul> <h2>8.5.24</h2> <ul> <li>Preserve the BOM after the processing (by <a href="https://github.com/hdimer"><code>@hdimer</code></a>).</li> </ul> <h2>8.5.23</h2> <ul> <li>Do not load source map without <code>opts.from</code> for security reasons.</li> </ul> <h2>8.5.22</h2> <ul> <li>Fixed custom property losing semicolon before a comment (by <a href="https://github.com/sarathfrancis90"><code>@sarathfrancis90</code></a>).</li> </ul> <h2>8.5.21</h2> <ul> <li>Fixed childless at-rule losing semicolon before comment (by <a href="https://github.com/sarathfrancis90"><code>@sarathfrancis90</code></a>).</li> <li>Fixed docs (by <a href="https://github.com/isker"><code>@isker</code></a>).</li> </ul> <h2>8.5.20</h2> <ul> <li>Fixed missing space if <code>AtRule#params</code> is set after (by <a href="https://github.com/sarathfrancis90"><code>@sarathfrancis90</code></a>).</li> <li>Fixed mixing AST error on warnings (by <a href="https://github.com/MahinAnowar"><code>@MahinAnowar</code></a>).</li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/postcss/postcss/blob/main/CHANGELOG.md">postcss's changelog</a>.</em></p> <blockquote> <h2>8.5.25</h2> <ul> <li>Fixed 8.5.17 visitor regression.</li> <li>Fixed <code>list.split()</code> for non-string values (by <a href="https://github.com/amir-rezaei"><code>@amir-rezaei</code></a>).</li> </ul> <h2>8.5.24</h2> <ul> <li>Preserve the BOM after the processing (by <a href="https://github.com/hdimer"><code>@hdimer</code></a>).</li> </ul> <h2>8.5.23</h2> <ul> <li>Do not load source map without <code>opts.from</code> for security reasons.</li> </ul> <h2>8.5.22</h2> <ul> <li>Fixed custom property losing semicolon before a comment (by <a href="https://github.com/sarathfrancis90"><code>@sarathfrancis90</code></a>).</li> </ul> <h2>8.5.21</h2> <ul> <li>Fixed childless at-rule losing semicolon before comment (by <a href="https://github.com/sarathfrancis90"><code>@sarathfrancis90</code></a>).</li> <li>Fixed docs (by <a href="https://github.com/isker"><code>@isker</code></a>).</li> </ul> <h2>8.5.20</h2> <ul> <li>Fixed missing space if <code>AtRule#params</code> is set after (by <a href="https://github.com/sarathfrancis90"><code>@sarathfrancis90</code></a>).</li> <li>Fixed mixing AST error on warnings (by <a href="https://github.com/MahinAnowar"><code>@MahinAnowar</code></a>).</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
0fd0b996a4
|
deps: bump next from 16.2.10 to 16.3.0 in /docs (#2750)
Bumps [next](https://github.com/vercel/next.js) from 16.2.10 to 16.3.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/vercel/next.js/releases">next's releases</a>.</em></p> <blockquote> <h2>v16.3.0</h2> <h3>Core Changes</h3> <ul> <li>Update vendored lodash to 4.17.23 to fix CVE-2025-13465: <a href="https://redirect.github.com/vercel/next.js/issues/91558">#91558</a></li> <li>Fix invalid HTML response for route-level RSC requests in deployment adapter: <a href="https://redirect.github.com/vercel/next.js/issues/91541">#91541</a></li> <li>Normalize encoded dynamic placeholders in app routes: <a href="https://redirect.github.com/vercel/next.js/issues/91603">#91603</a></li> <li>Fix(pages-router): restore Content-Length and ETag for /_next/data/ JSON responses: <a href="https://redirect.github.com/vercel/next.js/issues/90304">#90304</a></li> <li>Update tokio from 1.43.0 to 1.47.3: <a href="https://redirect.github.com/vercel/next.js/issues/90945">#90945</a></li> <li>[turbopack] Simplify snapshotting logic: <a href="https://redirect.github.com/vercel/next.js/issues/91178">#91178</a></li> <li>Turbopack: enable server HMR for app route handlers: <a href="https://redirect.github.com/vercel/next.js/issues/91466">#91466</a></li> <li>turbo-tasks-backend: batch find_and_schedule_dirty using for_each_task_meta: <a href="https://redirect.github.com/vercel/next.js/issues/91497">#91497</a></li> <li>[turbopack] Use bail! instead of panic! for duplicate module ident error: <a href="https://redirect.github.com/vercel/next.js/issues/91636">#91636</a></li> <li>Skip loadBindings() Lightning CSS check during next start: <a href="https://redirect.github.com/vercel/next.js/issues/91538">#91538</a></li> <li>turbo-tasks-backend: batch schedule dirty tasks in aggregation_update: <a href="https://redirect.github.com/vercel/next.js/issues/91461">#91461</a></li> <li>Turbopack: Add importModule() support to webpack loaders: <a href="https://redirect.github.com/vercel/next.js/issues/89630">#89630</a></li> <li>turbo-persistence: fix mmap page alignment and improve error context in MetaFile::open_internal: <a href="https://redirect.github.com/vercel/next.js/issues/91640">#91640</a></li> <li>turbopack-css: demote recoverable CSS parse warnings to Warning severity: <a href="https://redirect.github.com/vercel/next.js/issues/91524">#91524</a></li> <li>feat(node-streams): add config flag, define-env, and env precedence test: <a href="https://redirect.github.com/vercel/next.js/issues/90427">#90427</a></li> <li>Rename /_next/webpack-hmr to /_next/hmr: <a href="https://redirect.github.com/vercel/next.js/issues/91415">#91415</a></li> <li>Add per-slot error attribution for instant validation using slot markers and config depth preference: <a href="https://redirect.github.com/vercel/next.js/issues/91610">#91610</a></li> <li>Handle encoded params further: <a href="https://redirect.github.com/vercel/next.js/issues/91627">#91627</a></li> <li>[turbopack] Respect <code>{eval:true}</code> in worker_threads constructors: <a href="https://redirect.github.com/vercel/next.js/issues/91666">#91666</a></li> <li>Fix missing route in otel spans without base-server: <a href="https://redirect.github.com/vercel/next.js/issues/91665">#91665</a></li> <li>[turbopack] Optimize compaction cpu usage: <a href="https://redirect.github.com/vercel/next.js/issues/91468">#91468</a></li> <li>Fix layout segment optimization: move app-page imports to server-utility transition: <a href="https://redirect.github.com/vercel/next.js/issues/91701">#91701</a></li> <li>Fix server actions in standalone mode with <code>cacheComponents</code>: <a href="https://redirect.github.com/vercel/next.js/issues/91711">#91711</a></li> <li>turbo-persistence: remove Unmergeable mmap advice: <a href="https://redirect.github.com/vercel/next.js/issues/91713">#91713</a></li> <li>turbopack: move "compact database" tracing span to backend layer: <a href="https://redirect.github.com/vercel/next.js/issues/91693">#91693</a></li> <li>Turbopack: lazy require metadata and handle TLA: <a href="https://redirect.github.com/vercel/next.js/issues/91705">#91705</a></li> <li>Fix adapter outputs for dynamic metadata routes: <a href="https://redirect.github.com/vercel/next.js/issues/91680">#91680</a></li> <li>Turbopack: fix webpack loader runner layer: <a href="https://redirect.github.com/vercel/next.js/issues/91727">#91727</a></li> <li>[turbopack] Remove incorrect debug_assert in try_read_task_cell: <a href="https://redirect.github.com/vercel/next.js/issues/91699">#91699</a></li> <li>Add module count field to module graph tracing spans: <a href="https://redirect.github.com/vercel/next.js/issues/91697">#91697</a></li> <li>turbopack-cli: add --persistent-caching flag for filesystem-backed cache: <a href="https://redirect.github.com/vercel/next.js/issues/91657">#91657</a></li> <li>Turbopack: pull in updated vercel/nft tests: <a href="https://redirect.github.com/vercel/next.js/issues/91651">#91651</a></li> <li>[turbopack] Improve regressed build speed on cross-compiled MUSL: <a href="https://redirect.github.com/vercel/next.js/issues/91477">#91477</a></li> <li>[Segment Bundling] [Scaffolding] Ensure inlining hint correctness: <a href="https://redirect.github.com/vercel/next.js/issues/91320">#91320</a></li> <li>[Segment Bundling] [Scaffolding] Track which segments can be omitted from prefetch: <a href="https://redirect.github.com/vercel/next.js/issues/91438">#91438</a></li> <li>Avoid deprecated TS node10 moduleResolution defaults: <a href="https://redirect.github.com/vercel/next.js/issues/91847">#91847</a></li> <li>[turbopack] Rebuild the docker build scripts: <a href="https://redirect.github.com/vercel/next.js/issues/91799">#91799</a></li> <li>Fix TS6 baseUrl deprecation for extended tsconfig: <a href="https://redirect.github.com/vercel/next.js/issues/91855">#91855</a></li> <li>Add <code>next internal post-build</code> CLI command for Turbopack database compaction: <a href="https://redirect.github.com/vercel/next.js/issues/91336">#91336</a></li> <li>Turbopack: Define <code>Effect</code> as a trait instead of a closure: <a href="https://redirect.github.com/vercel/next.js/issues/89080">#89080</a></li> <li>Turbopack: Implement TraceRawVcs and NonLocalValue correctly for Effects: <a href="https://redirect.github.com/vercel/next.js/issues/89133">#89133</a></li> <li>turbo-tasks-backend: improve print_cache_item_size instrumentation: <a href="https://redirect.github.com/vercel/next.js/issues/91742">#91742</a></li> <li>Turbopack: switch from base40 to base38 hash encoding (remove ~ and . from charset): <a href="https://redirect.github.com/vercel/next.js/issues/91832">#91832</a></li> <li>Use charCodeAt for normalizePathTrailingSlash: <a href="https://redirect.github.com/vercel/next.js/issues/91380">#91380</a></li> <li>Turbopack: Only patch lockfile when bindings fails to load: <a href="https://redirect.github.com/vercel/next.js/issues/91379">#91379</a></li> <li>[create-next-app] Skip interactive prompts when CLI flags are provided: <a href="https://redirect.github.com/vercel/next.js/issues/91840">#91840</a></li> <li>[devtools] Make instant navs panel draggable: <a href="https://redirect.github.com/vercel/next.js/issues/91914">#91914</a></li> <li>[Segment Bundling] Bundle static prefetches based on size: <a href="https://redirect.github.com/vercel/next.js/issues/91439">#91439</a></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
dependabot[bot]
|
56ee57be98
|
deps: bump brace-expansion from 5.0.7 to 5.0.9 in /docs (#2751)
Bumps [brace-expansion](https://github.com/juliangruber/brace-expansion) from 5.0.7 to 5.0.9. <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
Lucas Santos
|
3107994aed
|
fix(litellm): add async_post_call_success_hook to HeadroomCallback (#1322)
## Description Pointing a litellm proxy at the Headroom callback blows up on the post-call success path: ``` type object 'HeadroomCallback' has no attribute 'async_post_call_success_hook' ``` litellm's logging contract calls `async_post_call_success_hook` after a successful response, and `HeadroomCallback` simply doesn't have it. We implement `async_pre_call_hook`, `async_success_handler` and `async_failure_handler`, but not this one, so litellm hits an `AttributeError` instead of a no-op and the whole request fails. This adds the missing `async_post_call_success_hook(self, data, user_api_key_dict, response)` matching litellm's signature. It returns `response` unchanged, the token accounting already lives in `async_success_handler` so there's nothing to do here except not crash. A few notes: 1. I did not make `HeadroomCallback` inherit litellm's `CustomLogger`, on purpose. The class keeps litellm as an optional dependency, so it stays a plain class and just provides the hooks litellm looks up by name. 2. It's a pass-through, so it's safe regardless of what the response contains. Closes #1114 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/integrations/litellm_callback.py`: add `async_post_call_success_hook` to `HeadroomCallback`, returning the response unchanged; update the class docstring to list the full set of litellm hooks. - `tests/test_integrations/test_litellm_callback.py`: new tests that the method exists, is a coroutine, and returns the response untouched; build the module path with `pathlib` instead of a fragile `__file__.replace`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run --extra dev python -m pytest tests/test_integrations/test_litellm_callback.py -q 3 passed ruff: All checks passed! mypy: Success: no issues found ``` ## Real Behavior Proof - Environment: macOS, Python 3.13, this branch. - Exact command / steps: `uv run --extra dev python -m pytest tests/test_integrations/test_litellm_callback.py -q`. The tests import the callback module directly and resolve `async_post_call_success_hook` by name, the same way litellm does, then await it with a sentinel response. - Observed result: 3 passed. The hook exists, is a coroutine, and returns the exact response object it was given. Before the fix, resolving the attribute raised `AttributeError`. - Not tested: I did not stand up a full litellm proxy end to end. The fix is the missing hook method, which the unit tests cover. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes --------- Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
Focused Instability
|
4dab254d52
|
fix: emit SSE ping before message_start on Bedrock streaming path (issue #902) (#1080)
## Description Closes #902 Mid-turn user interjections (steering) silently dropped through the Bedrock streaming path. The _stream_response_bedrock code path reconstructs Anthropic SSE events from parsed StreamEvent objects instead of passing raw bytes through, so SSE-level ping keepalives are never forwarded to Claude Code. Claude Code relies on ping events to arm its mid-turn steering / interruptible state; without them, queued interjections are discarded instead of sent. Root cause (confirmed): - Standard direct-Anthropic path does a raw yield-chunk passthrough — pings flow unchanged. - Bedrock path (_stream_response_bedrock.generate()) reconstructs events from litellm/anyllm stream_message() output, which only yields semantic events (message_start, content_block_*, message_delta, message_stop, error). No pings, ever. Fix: emit a synthetic 'event: ping / data: {}' at stream start (before the first message_start) so downstream clients see the same ping-then-content cadence as a real Anthropic stream. Note: periodic pings for very long responses (>~25s) may be needed if steering disarms on a timer. This commit arms it at turn start; follow-up if reporters confirm steering still drops on long turns. The causal link (ping → steering) is the reporter's hypothesis from hands-on debugging. The observable defect (zero pings in stream) is confirmed and fixed. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - headroom/proxy/handlers/streaming.py: yield ping event before the event loop in _stream_response_bedrock.generate() - tests/test_proxy/test_bedrock_sse_ping.py: 3 new tests asserting ping appears before message_start ## Testing - [x] Unit tests pass (pytest) - [x] Linting passes (ruff check .) - [x] New tests added for new functionality ### Test Output ``` tests/test_proxy/test_bedrock_sse_ping.py::test_bedrock_stream_emits_ping_before_message_start PASSED tests/test_proxy/test_bedrock_sse_ping.py::test_bedrock_stream_ping_has_empty_data PASSED tests/test_proxy/test_bedrock_sse_ping.py::test_bedrock_stream_contains_message_stop PASSED tests/test_backend_streaming_cache_metrics.py (4 tests) PASSED 7 passed in 4.41s ``` ## Real Behavior Proof - Environment: macOS, Python 3.11, headroom unit tests - Exact command / steps: pytest tests/test_proxy/test_bedrock_sse_ping.py -v - Observed result: 3 new tests pass; ping appears before message_start in Bedrock stream - Not tested: end-to-end against live Bedrock + Claude Code (no Bedrock credentials available) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] Code follows project style guidelines - [x] Code is commented where non-obvious - [x] No new warnings - [x] Tests added and passing ## Additional Notes The Rust proxy files mentioned in the issue (sse/framing.rs, sse/anthropic.rs) are NOT part of this fix. Those drops are in a telemetry-only tee task that never affects the client byte path — the Rust proxy does a raw bytes passthrough for all responses. The defect is Python-only, confined to the Bedrock path. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Tejas Chopra
|
9fd5ae3d53
|
chore: release main (#2679)
🤖 I have created a release *beep* *boop* --- <details><summary>0.34.0</summary> ## [0.34.0](https://github.com/headroomlabs-ai/headroom/compare/v0.33.0...v0.34.0) (2026-08-05) ### Features * **claude:** support Claude Code in VS Code ([#2752](https://github.com/headroomlabs-ai/headroom/issues/2752)) ([ |
||
|
Tejas Chopra
|
702a076fc1
|
docs(readme): surface the Serena opt-out in the wrap quickstart (#2790)
## Description
The README quickstart announces that `headroom wrap` installs Serena,
then stops — it offers no way to decline. The opt-out exists
(`--code-memory none`) but was documented only on the docs site at
`docs/content/docs/proxy.mdx:78`, which a README reader never reaches.
In #2783 a user followed the quickstart verbatim, hit a Serena startup
failure, and found the flag via `headroom wrap claude --help` only
afterwards, while recovering the machine. Their words: *"a first-time
user following the documented `headroom wrap claude` command has no
signal that this failure mode exists or how to avoid it."*
One line at the point where the install is announced closes that gap.
Closes #2788
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `README.md` — one sentence after the existing wrap paragraph:
> Serena is registered at **user scope** (for Claude Code, in
`~/.claude.json`), so it stays available in your other projects until
you run `headroom unwrap`. To skip it entirely, wrap with `--code-memory
none`.
Two facts, both of which the #2783 reporter needed and could not find:
the registration is user-scoped (so Serena shows up in projects that
were never wrapped, until `unwrap`), and there is a flag to skip it.
`docs/content/docs/quickstart.mdx` was checked for the same gap and has
no equivalent wrap section — its only `wrap` mention is `vscode-claude`
— so there is nothing to mirror. `proxy.mdx` already covers the flag and
is unchanged.
## Testing
- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed
A two-line Markdown addition to `README.md`. No Python, no config, no
build inputs — pytest / ruff / mypy have nothing to cover. What was
worth checking is that the sentence is *true*, so I verified both claims
against the CLI on `main` rather than against the issue text.
### Test Output
```text
$ grep -n '"--code-memory"' -A 2 headroom/cli/wrap.py
924: "--code-memory",
925- type=click.Choice([_CODE_MEMORY_SERENA, _CODE_MEMORY_NONE]),
926- default=None,
$ headroom wrap claude --help | grep -A3 code-memory
headroom wrap claude --code-memory none # No code-memory MCP
...
--code-memory [serena|none] Code-memory MCP to register: 'serena' (default)
or 'none'. Also set by HEADROOM_CODE_MEMORY.
Replaces --serena/--no-serena.
```
## Real Behavior Proof
- **Environment:** macOS (darwin 25.4.0), repo `.venv`, branch cut from
`upstream/main` @ `
|
||
|
Tejas Chopra
|
6b63b623e0
|
docs(metrics): document OTLP metric export and Dynatrace ingest (#2785)
## Description
The proxy can already push its counters to any OTLP/HTTP endpoint via
`HEADROOM_OTEL_METRICS_*`, but the docs site only surfaced this as a
single row in the proxy env table (`proxy.mdx:287`). The endpoint,
header, service-name, and resource-attribute variables were documented
only in `wiki/metrics.md` — so an operator reading the Vercel docs had
no way to wire Headroom into their existing observability stack.
This adds that section, plus a Dynatrace subsection, because Dynatrace
has a silent failure mode that costs an afternoon to diagnose.
Closes #
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `docs/content/docs/metrics.mdx` — new `### OpenTelemetry (OTLP)
Export` section after the Prometheus section: the
`headroom-ai[proxy,otel]` install, all seven `HEADROOM_OTEL_*` variables
in a table, the exported counter names (`headroom.proxy.tokens.saved` et
al.), the `curl /stats | jq .otel` verification, and the note that an
app-managed global meter provider is recorded into automatically.
- `docs/content/docs/metrics.mdx` — new `### Dynatrace` subsection:
copy-paste env block, `metrics.ingest` token scope, a `warn` Callout on
the delta-temporality requirement, the ActiveGate URL variant, the
Collector + `cumulativetodelta` alternative, and one paragraph
explaining that trace export needs `opentelemetry-instrument`
(Headroom's self-configured tracing targets Langfuse only).
- `docs/content/docs/proxy.mdx` — the `HEADROOM_OTEL_METRICS_ENABLED`
row now links to `/docs/metrics#opentelemetry-otlp-export`.
No code, config, or nav changes — the Observability nav slot already
points at `metrics.mdx`.
## Testing
- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed
Docs-only change: no Python touched, so pytest/ruff/mypy have nothing to
cover here. `next build` was **not** run — `docs/node_modules` is absent
in this checkout, which would require a full `npm install`; Vercel's
preview build is the real gate. In its place I verified the MDX cannot
break the build by parsing for the two things that actually fail MDX v3
— unbalanced JSX and bare `<`/`{` in prose.
### Test Output
```text
$ python - <<'PY' # strip fenced + inline code, then scan prose for MDX hazards
...
PY
hazards: [(80, '<Tabs groupId="lang" items={[\'TypeScript\', \'Python\']}>'),
(125, '<Tabs groupId="lang" items={[\'Python\', \'Proxy\']}>')]
Callout balance: 1 open / 1 close
```
Both flagged lines are pre-existing `<Tabs>` JSX expressions, untouched
by this PR. The added prose introduces no bare `<` or `{` (every
`<env-id>` / `<activegate>` placeholder sits inside a code fence or
inline backticks). `type="warn"` is already used on three other pages,
and the anchor `#opentelemetry-otlp-export` matches the GitHub-slugger
form of the new heading.
## Real Behavior Proof
- **Environment:** macOS (darwin 25.4.0), repo `.venv`,
`opentelemetry-sdk` 1.44.0, `opentelemetry-exporter-otlp-proto-http`,
headroom @
|
||
|
Tejas Chopra
|
3c10e8ff00
|
chore(docs): one documentation site, not two (#2784)
## Description The repo published **two** documentation sites from two source trees: ``` docs/ -> Next.js/Fumadocs -> headroom-docs.vercel.app <- canonical wiki/ -> MkDocs -> gh-pages branch -> github.io/headroom <- orphan ``` The Vercel site is what the README badge and **every** README deep link point at, and what `pyproject.toml` names as both `Homepage` and `Documentation`. The Pages site is referenced from **nowhere** in the repo — not README, not `pyproject`, not `CLAUDE.md`, not any docs page. I grepped for `github.io` and `gh-pages` across all of them and got zero hits. So it was costing work and causing breakage while nobody was reading it: - **Every documented change had to be written twice.** This session I wrote the same configuration content into `docs/content/docs/configuration.mdx` *and* `wiki/configuration.md`. That's the tax, and it compounds silently — the two drift and no one notices which is stale. - **It broke the Vercel deployment.** Each Pages deploy runs `mkdocs gh-deploy --force`, force-pushing `gh-pages`. Vercel's Git integration then tries to build that branch with Root Directory `docs`, which fails: *"The specified Root Directory `docs` does not exist"* — because `gh-pages` holds only the rendered site (`.nojekyll`, `404.html`, …). Timing was exact: ```text 23:06:44 main |
||
|
Tejas Chopra
|
a9a2fbd74f
|
ci(docs): deploy Pages on wiki changes, drop the never-working Vercel job (#2746)
## Description
Two independent bugs in `.github/workflows/docs.yml`.
**1. Pages went stale because the push filter watched the wrong
directory.**
`mkdocs.yml` sets `docs_dir: wiki`, but the push filter listed `docs/**`
and not `wiki/**`. So a merge touching only `wiki/` never triggered the
workflow and the published Pages site silently went stale, while a
`docs/**`-only change triggered a Pages rebuild whose sources mkdocs
doesn't even read.
Push now filters on `wiki/**` + `mkdocs.yml`. **Pull requests keep
`docs/**`**, so `validate-nextjs` still catches a broken MDX change
before merge.
**2. `deploy-vercel` never worked — it wasn't a working path that went
stale.**
It ran `npx vercel deploy --prod --token=${{ secrets.VERCEL_TOKEN }}`,
but those secrets don't exist:
```text
$ gh api repos/headroomlabs-ai/headroom/actions/secrets --jq '.secrets[].name' | grep -i vercel
(nothing)
$ gh api orgs/headroomlabs-ai/actions/secrets --jq '.secrets[].name' | grep -i vercel
(nothing)
```
So it invoked the CLI with an empty `--token=` and exited 1, every time:
```text
30 failed runs on main, 2026-07-14 .. 2026-08-04
most recent:
30874173333
|
||
|
Tejas Chopra
|
04e1517ede
|
fix(telemetry): stop mixing tokenizer scales in RequestOutcome, and fix the overhead framing (#2756)
## Description
Two defects found by reading real beacon payloads, not by inspection.
### 1. `eligible_pct: 120`
A gpt-4o-mini session shipped this:
```json
"tokens": {"original": 10, "attempted": 12, "input": 12, "saved": 0},
"rates": {"eligible_pct": 120}
```
120% is structurally impossible — you cannot attempt to compress more
than arrived. And nothing had grown.
`original_tokens` is our **local tokenizer** count. `optimized_tokens`
on the OpenAI path carried the **provider's** `usage.prompt_tokens`. Our
estimator undercounted gpt-4o-mini by 2 tokens on a 10-token request,
and every quantity derived from that pair inherited the mismatch:
- `attempted_input_tokens = optimized + saved` → 12, exceeding
`original` → `eligible_pct` 120, `yield_pct` contaminated
- `tokens_inflated` (added in #2708) → reported **2 tokens of phantom
growth**
`optimized_tokens` was dual-purpose by design — *"post-compression bytes
actually forwarded, for `input_tokens` and `tok_after`"*. Billing wants
the provider's count; deltas need the same ruler as `original_tokens`.
Those are different jobs sharing one field.
This is the same class of bug as #2743, on the request path instead of
`/v1/compress`, and it is the exact false positive I flagged as
theoretical when reviewing #2708 — where I measured the margin on a real
722-request log as **exactly zero**, so any provider counting above our
estimator would flip it. gpt-4o-mini does, and it is now in production
telemetry.
### 2. `overhead_pct` was documented as wall-clock, and is not
A 1393-turn session shipped `latency_ms_total: 16396565.8` against
`duration_s: 8904` — **4.55h of latency inside a 2.47h session, 1.84×
over wall clock.**
Both terms are sums over turns, and turns run concurrently (parallel
tool calls, subagents, several clients per proxy), so each sum
over-counts elapsed time. `overhead_pct` is still a meaningful
latency-weighted per-request share, but the comment claiming *"what
fraction of wall-clock did Headroom itself add?"* was wrong, and
shipping `latency_ms_total` beside `duration_s` invites a comparison
that yields nonsense.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
**Scale split (`outcome.py`, `handlers/openai.py`)**
- `optimized_tokens` is now always the **local** count — same tokenizer
as `original_tokens`, so every delta built from the pair is coherent.
- New optional `provider_input_tokens` carries the provider's own count.
Defaults to `0`, so the other emit sites need no change.
- Cost and volume totals read `provider_input_tokens or
optimized_tokens`, so **billing is unchanged** wherever a provider
reports usage, and falls back exactly as before where it doesn't.
- Removes a band-aid: one of the three OpenAI sites already computed
`effective_original_tokens = max(original_tokens, optimized + saved)`,
inflating `original` upward so `attempted` could not exceed it. That hid
the symptom at one site while the other two shipped the impossible
ratio.
**Overhead framing (`telemetry/session.py`)**
- Corrects the wall-clock comment on `overhead_pct` and states what it
actually is.
- Documents that `latency_ms_total` is a sum of per-request durations,
not elapsed time, and why it can exceed `session.duration_s`.
- Adds `overhead_ms_per_turn` and `latency_ms_per_turn` — unambiguous
under concurrency and comparable across sessions.
- Field names kept for schema-v1 consumers; **additive only**, no schema
bump.
## Testing
- [x] Unit tests pass (new file)
- [x] Linting passes (`ruff check` + `format --check`, pinned 0.15.17)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/proxy/outcome.py headroom/proxy/handlers/openai.py headroom/telemetry/session.py
All checks passed!
$ uvx ruff@0.15.17 format --check <same three>
3 files already formatted
$ pytest tests/test_outcome_token_scale.py -q
5 passed in 0.24s
```
## Real Behavior Proof
- **Environment:** macOS 26.4 arm64, isolated git worktree off
`upstream/main`, throwaway venv (see caveat).
Replayed both reported payloads through the corrected arithmetic:
```text
ENTRY 1 (gpt-4o-mini, the eligible_pct:120 case)
BEFORE: original=10 (local) optimized=12 (PROVIDER) saved=0
attempted = 12+0 = 12 eligible_pct = 120.0 <- impossible
tok_inflated = max(0, 12-10) = 2 <- phantom
AFTER : original=10 (local) optimized=10 (local) provider_input=12
attempted = 10+0 = 10 eligible_pct = 100.0
tok_inflated = 0
billed_input = 12 -> cost/cache math unchanged
ENTRY 2 (1393 turns, the overhead case)
latency_ms_total = 16397s vs duration 8904s -> 1.84x wall clock
NEW overhead_ms_per_turn = 333.5 latency_ms_per_turn = 11770.7
wall clock per turn = 6392.0 -> per-turn latency exceeds it => concurrency
```
Genuine post-compression growth still surfaces: `55,161 → 57,845`
reports `tokens_inflated = 2,684` (pinned as a test), so the fix doesn't
mute what #2708 exists to show.
- **Not tested locally beyond the new file.** The repo venv currently
has no `pytest`, no `ruff`, and no compiled `headroom._core`, so I used
a throwaway venv. The outcome/telemetry suites fail there for missing
deps — `click` (12/12), then `headroom._core` (10/10) — with **zero
assertion failures**, and an **identical failure set on this branch and
on clean `upstream/main`** in the same env. So they are
environment-only, not regressions. CI on this PR is the authoritative
signal for the full suite.
- Unrelated: `Wrap E2E / docker-wrap-e2e` is currently red on `main`
from a quay.io CDN `tls: internal error` pulling the manylinux base
image — infrastructure, transient, and green on the three prior runs.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] I did **not** edit `CHANGELOG.md`
## Follow-ups not in this PR
- **`cache_write` and `uncached` double-count** on inferred-cache
providers. The same beacon entry shows `input: 12, cache_read: 0,
cache_write: 12, uncached: 12` — both fields describe the identical 12
tokens, because `_infer_openai_cache_write_tokens` and
`uncached_input_tokens` are computed the same way (`input −
cache_read`). Any consumer summing them gets 2×. The payload also
carries no `cache_inferred` flag, so a reader can't distinguish an
inferred write from an Anthropic-reported one.
- **No skip reason for compression itself.** That entry records
`memory_skip:no_handler` but nothing explains why compression didn't
fire; "below the size floor" and "compressor failed" are
indistinguishable — the same ambiguity #2708 just removed for inflation.
- **`eligible_pct` can still legitimately exceed 100** when a request
genuinely grows after compression (memory injection, proactive
expansion), since `attempted = optimized + saved` uses the forwarded
size. Left alone deliberately: clamping would hide real inflation, and
`tokens_inflated` now expresses it properly.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
Tejas Chopra
|
0e1d6bfa79
|
refactor(pricing): make LiteLLM the source of truth, not the hardcoded table (#2779)
## Description > **Stacked on #2777.** That PR corrects the built-in table's *values*; this one stops the table being *authoritative*. Both are wanted — the fallback should be right **and** not in charge. Merge #2777 first. You asked whether the token/savings code could be simpler and whether the hardcoding could go. This is the hardcoding half, and the encouraging finding is that **almost none of it needed writing** — the infrastructure already existed and was simply unused. `headroom/pricing/litellm_pricing.py` (300 lines, LiteLLM-backed, with an `ImportError` fallback and gateway-prefix handling) has been in the tree the whole time. `ModelInfo`'s own docstring says: > *"Pricing is fetched dynamically from LiteLLM's database. Use `ModelRegistry.estimate_cost()` to get current pricing."* Yet **zero of the four providers called it** (`grep -c litellm_pricing` → openai 0, anthropic 0, google 0, cohere 0). Each kept a parallel hardcoded table. `_get_pricing` had no LiteLLM lookup at all, unlike `get_context_limit` — which is precisely how it went ~18 months stale and priced `gpt-4.1-nano` **300× over**. ## Changes Made **1. Resolution order now mirrors `get_context_limit`**, so limits and prices can't disagree: ``` explicit user config -> LiteLLM -> built-in table -> family -> unknown default ``` Config beats LiteLLM because a configured price is a decision, not a guess. The table stays because it must: the `litellm` dependency is gated `python_version < '3.14'`, and LiteLLM doesn't know every model. It just isn't in charge, so its drift only reaches installs with no LiteLLM. **2. Gateway-routed names now resolve at all.** `litellm.model_cost` keys the *unwrapped* form, so `bedrock/anthropic.claude-...` missed every candidate and silently took the $2.50/$10.00 unknown default: | model | before | after | |---|---|---| | `bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0` | $2.50 / $10.00 | **$3.00 / $15.00** | | `bedrock/us.anthropic.claude-3-5-sonnet-...-v2:0` | $2.50 / $10.00 | **$3.00 / $15.00** | | `vertex_ai/claude-sonnet-4-5` | $2.50 / $10.00 | **$3.00 / $15.00** | | `groq/llama-3.3-70b-versatile` | $2.50 / $10.00 | **$0.59 / $0.79** | | `gemini-2.5-flash` | $2.50 / $10.00 | **$0.30 / $2.50** | | `deepseek-chat` | $2.50 / $10.00 | **$0.28 / $0.42** | `pricing_lookup_candidates` only ever *prepended* provider prefixes. It now also tries progressively unwrapped forms, derived by splitting on `/` — deliberately **not** another hardcoded gateway-prefix list. A wrong guess costs nothing: each candidate is an exact dict lookup, so it just misses. **3. The staleness warning became meaningful.** It fires only when the fallback table is actually used. Before, it was unconditional — and with `_PRICING_LAST_UPDATED = 2025-01-14` against a 60-day window it had been firing for ~18 months, which trains people to ignore it. **4. `pricing_per_1m` rounds to 6dp.** LiteLLM stores cost *per token*, so `× 1e6` leaves float noise ($0.4/1M arrives as `0.39999999999999997`). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Code refactoring ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check` + `ruff format`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality ### Test Output ```text $ pytest tests/test_pricing_from_litellm.py -q 10 passed ``` Full pricing / cost / provider / models / savings / reporting / tokenizer set: ```text $ pytest tests/test_*{pricing,cost,provider,models,savings,utils,reporting,token}*.py -q 3 failed, 1026 passed, 38 skipped pre-existing on main (all three in my recorded baseline): test_bundled_tools_savings.py::test_compressed_payload_preserves_answer_anthropic test_compress_route_tokenizer_by_model.py::...[deepseek/deepseek-v4] (needs transformers) test_compress_route_tokenizer_by_model.py::...[deepseek/deepseek-v4] (needs transformers) ``` Deferring the full sharded run to CI — no maturin/Rust core locally. ### One test of mine changed, and why Three cases in #2777's `test_openai_pricing_resolution.py` failed on exact equality once prices started coming from LiteLLM — `gpt-4.1-mini` arrived as `0.39999999999999997` rather than `0.4`. The values were right; binary floating point isn't exact. Switched those to `pytest.approx(..., abs=0.001)` — money compared to the cent — which passes whether the number comes from LiteLLM or the literal table. ## Deliberately NOT in this PR - **Encodings stay hardcoded.** LiteLLM carries no tiktoken encoding data, and `_lookup_encoding_name`'s `None` return is load-bearing (it's the "not an OpenAI model" signal from #2761). Encodings also track tokenizer generations, not monthly price changes — they aren't the drift problem. - **Anthropic / Cohere / Google providers.** Same shape, same fix, but Anthropic's pricing is a `{input, output, cached_input}` dict rather than a tuple, and its matcher is worse (`if model in known_model or known_model in model` — bidirectional substring). Worth its own PR rather than tripling this diff. - **Context limits.** Already LiteLLM-first; the layering there was correct all along. - `accounts/fireworks/models/kimi-k2` still falls back — LiteLLM genuinely has no entry. The file already shows the pattern for filling such gaps (`_register_minimax_pricing`, `_inject_deepseek_pricing`) if we want it. |
||
|
Tejas Chopra
|
f03cc6d88b
|
fix(router): stop counting an image's base64 payload as suffix tokens (#2778)
## Description
`_netcost_message_tokens` walked block-list content itself and fell back
to `str(block)` for anything that wasn't `text` or `tool_result` — on
the stated assumption that such blocks *"rarely dominate a suffix"*. An
`image` block is the exception that breaks it: `str()` embeds the whole
base64 payload.
```text
counted real over
512x512 PNG 20,034 349 57x
1092x1092 screenshot 100,034 1,589 63x
1568x1568 233,367 1,600 146x
```
**Why this changes behaviour, not just a number.** S is the cache-bust
cost — the tokens re-written if message *j* is mutated. `apply()` builds
it as a running suffix sum:
```python
for j in range(num_messages - 1, -1, -1):
netcost_suffix_tokens[j] = netcost_suffix_tokens[j + 1] + _netcost_message_tokens(...)
```
So one image inflates S for **every message before it**, and the
break-even gate then declines to compress any of them. A single
screenshot could switch off net-cost-gated compression for the whole
earlier conversation — and screenshots are routine in agent sessions.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
Delegate block-list content to `tokenizers.base.count_content_blocks`,
deleting the local walk. That counter already guards exactly this case —
its comment reads *"1MB image = ~330K fake tokens without this"* — so
this walk simply predated it.
Beyond the raw fix, this removes a **second pricing rule**: the gate now
values images the same way the tokenizer that computes
`tokens_before`/`tokens_after` does (a flat 1600, "max after
auto-resize"). Pricing images one way for the gate and another for the
savings math is the same class of problem as #2761.
Verified byte-identical on the shapes the old walk handled correctly:
```text
old walk canonical
text only 101 101
tool_result str 81 81
tool_result list 61 61
image only 100,034 1,600
mixed 100,036 1,602
```
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check` + `ruff format`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for new functionality
### Test Output
```text
$ pytest tests/test_netcost_suffix_image_tokens.py -q
8 passed
$ git stash push headroom/ && pytest tests/test_netcost_suffix_image_tokens.py -q
4 failed, 4 passed
# the 4 failures are the payload-scaling assertions; the 4 passes are the
# text/tool_result/string shapes, included to prove delegation is behaviour-preserving
```
All netcost + content-router suites:
```text
$ pytest tests/test_netcost_gate.py tests/test_content_router_*.py \
tests/test_transforms_content_router.py tests/test_netcost_suffix_image_tokens.py -q
126 passed
```
```text
$ ruff check headroom/transforms/content_router.py tests/... All checks passed!
$ mypy headroom/transforms/content_router.py no new errors
```
Deferring the full suite to CI — no maturin/Rust core in this
environment.
## One existing test rewritten — please look at this bit
`test_netcost_gate.py::TestNetCostHelpers::test_message_tokens_block_list_beats_repr`
fails under the fix, and I want to be explicit that I changed a test
rather than bury it.
It built its image block as `{"type": "image", "source": {"data": "x" *
500}}`. A 500-char stub is **cheaper than a single image's real token
cost**, so `str()` over it looked harmless (~130 tokens) and its
assertion `abs(helper - text_only) < text_only * 0.5` held. That
unrepresentative fixture is precisely why the payload-scaling bug
survived — the test named "beats repr" was passing on the one payload
size where repr happens not to be catastrophic.
Rewritten to use a realistic 200KB payload and to assert what actually
matters:
```python
assert helper >= text_only # text still counted in full
assert helper - text_only <= 2000 # image cost is bounded, not payload-scaled
assert helper < count_text(str(content)) / 10 # ...and far below repr
```
I checked this both ways, so it is a real test and not a rubber stamp:
```text
old test + fixed code -> FAILS (it was pinning the defect)
new test + main -> FAILS (it catches the real bug)
new test + fixed code -> passes
```
## Known limitation
The canonical estimate is a flat 1600 per image regardless of
dimensions, so a small icon is now over-charged (~1600 vs ~13 real)
where repr would have charged ~200. I kept the flat constant
deliberately: it is the value every other counter in the codebase uses,
and introducing a third rule here to shave small-icon cost would
recreate the inconsistency this PR removes. The error is bounded at 1600
tokens and biases the gate conservative, versus an unbounded 100K+ error
before.
|
||
|
Tejas Chopra
|
fc4680b37a
|
fix(tokenizers): resolve gpt-5 and mixed-case model names to the right encoding (#2776)
## Description Two defects in `get_encoding_for_model`, both reachable through the normal `get_tokenizer()` path. **1. `gpt-5` had no prefix entry.** It fell through to `DEFAULT_ENCODING` (`cl100k_base`) instead of `o200k_base`. Same class as the `o4` gap already patched in that tuple. cl100k emits ~33% more tokens than o200k on CJK, so every gpt-5 count was inflated there: ```text CJK sample (30x repeated sentence) o200k_base (correct) 450 tokens cl100k_base (actual) 600 tokens +33.3% ``` Note #2758 taught the *registry* that `gpt-5` → the tiktoken backend; this is the next hop, where that backend picks its *encoding*. So gpt-5 got the right tokenizer family and the wrong encoding inside it. **2. Resolution was case-sensitive.** `TokenizerRegistry.get` lowercases only its **cache key**, then constructs the counter from the caller's original string (`_create_tokenizer(model, backend)`). An uppercase deployment name — routine on Azure, where the deployment name is user-chosen — arrived verbatim, matched no prefix, and took the default encoding. The cache makes this one genuinely unpleasant: key lowercased, construction not, so **the encoding a model receives depends on the casing of whichever request warmed the cache first**, and can differ across restarts. ```text cold cache, uppercase resolved first: GPT-4o -> cl100k_base CJK=600 WRONG GPT-4.1 -> cl100k_base CJK=600 WRONG Gpt-4O-Mini -> cl100k_base CJK=600 WRONG gpt-4o -> o200k_base CJK=450 ok ``` I nearly filed this as "not reachable" — my first check ran the lowercase spelling first, which populated the shared lowercased cache key and masked it completely. The tests call `clear_cache()` so the uppercase spelling resolves cold, which is the failing order. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Added `("gpt-5", "o200k_base")` to the ordered prefix tuple. - `get_encoding_for_model` now lowercases its input. The lowercasing is deliberately scoped to this function rather than the registry: every `MODEL_TO_ENCODING` key is already lowercase (asserted), so it is safe here — whereas lowercasing in `TokenizerRegistry` would break HuggingFace repo ids, which *are* case-sensitive (`Qwen/Qwen3-Coder`). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check` + `ruff format`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality ### Test Output ```text $ pytest tests/test_tokenizer_encoding_resolution.py -q 21 passed $ git stash push headroom/ && pytest tests/test_tokenizer_encoding_resolution.py -q # gpt-5 family, every uppercase case, and both cache-order tests fail ``` Targeted run across the tokenizer/pricing/provider suites, including `test_evals_cjk_tokenization.py` since CJK is the affected content type: ```text $ pytest tests/test_utils.py tests/test_reporting.py tests/test_cost_pricing_warning_dedup.py \ tests/test_pricing.py tests/test_pricing_litellm.py tests/test_provider_model_fallback.py \ tests/test_models.py tests/test_savings_ledger.py tests/test_tokenizers.py \ tests/test_tokenizer.py tests/test_tokenizer_selection_coverage.py \ tests/test_provider_tokenizer_one_ruler.py tests/test_openai_model_table_resolution.py \ tests/test_evals_cjk_tokenization.py -q 233 passed, 16 skipped ``` ```text $ ruff check <changed files> All checks passed! $ mypy headroom/tokenizers/tiktoken_counter.py # only pre-existing release_version.py tomllib redef, present on main ``` Deferring the full suite to CI — this environment has no maturin/Rust core, so the native-dependent shards can't run locally. ## Real Behavior Proof - **Environment:** macOS, Python 3.13.7, isolated worktree at `upstream/main` (`0cb72f45`). - **Exact command / steps:** `TokenizerRegistry.clear_cache()`, then resolve each spelling cold and count a CJK sample. - **Observed result:** ```text before after gpt-5 cl100k_base CJK=600 o200k_base CJK=450 gpt-5-mini cl100k_base CJK=600 o200k_base CJK=450 GPT-4o cl100k_base CJK=600 o200k_base CJK=450 GPT-4.1 cl100k_base CJK=600 o200k_base CJK=450 Gpt-4O-Mini cl100k_base CJK=600 o200k_base CJK=450 GPT-4 cl100k_base CJK=600 cl100k_base CJK=600 (unchanged, correct) gpt-4o o200k_base CJK=450 o200k_base CJK=450 (unchanged) gpt-3.5-turbo cl100k_base CJK=600 cl100k_base CJK=600 (unchanged) ``` `GPT-4` was previously "correct" only by accident — it missed every prefix and landed on `DEFAULT_ENCODING`, which happens to be `cl100k_base`. It is now correct by resolution. |
||
|
Tejas Chopra
|
0cb72f45b2
|
fix(providers): stop a shorter model family shadowing a longer one (#2762)
## Description > **Stacked on #2761** — that PR splits `_lookup_encoding_name` out of `_get_encoding_name_for_model`, which this one builds on. Please merge #2761 first; the diff here will shrink to just this commit afterwards. `_MODEL_ENCODINGS` and `_CONTEXT_LIMITS` are matched by prefix, iterating in **plain dict order** — so the first *inserted* prefix wins rather than the most specific one. `gpt-4.1` matched the `gpt-4` entry: | model | resolved | actual | | |---|---|---|---| | `gpt-4.1` | 8192 | 1,047,576 | **128× under** | | `gpt-4.1-mini` | 8192 | 1,047,576 | **128× under** | | `gpt-4.1-nano` | 8192 | 1,047,576 | **128× under** | | `gpt-4-32k-0613` | 8192 | 32,768 | 4× under | | `gpt-5` / `-mini` / `-nano` | 128,000 | 400,000 | fell to unknown-model default | | `o4-mini` | 128,000 | 200,000 | fell to unknown-model default | A 128× under-estimate matters because the context limit is what tells the proxy how much headroom is left: it treats a 1M-context model as nearly full and compresses accordingly. The same shadowing picked the **wrong encoding** — `gpt-4.1` got `cl100k_base` instead of `o200k_base`. Measured cost of that: ```text cl100k o200k error python code 420 420 +0.0% json blob 555 555 +0.0% logs 580 580 +0.0% english 201 201 +0.0% CJK 600 450 +33.3% ``` So the encoding half is narrow but real — it only bites CJK content, which the repo already treats as a case worth testing (`tests/test_evals_cjk_tokenization.py`). **Scope honestly:** `get_context_limit` consults LiteLLM *before* this table, so the limit half only surfaces where LiteLLM is absent or does not know the model. That is not hypothetical — the `litellm` dependency carries a `python_version < '3.14'` marker (`pyproject.toml:56`), so **any install on Python 3.14+ has no LiteLLM** and this table is load-bearing. The encoding half never had a LiteLLM fallback and was always wrong. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Both prefix loops now iterate `sorted(..., key=len, reverse=True)` — longest prefix wins. This is the root-cause fix: it also protects the *next* model added to these tables. - Added the missing families: `gpt-4.1` (+`-mini`/`-nano`), `gpt-5` (+`-mini`/`-nano`), `o4-mini` to both tables. - Left `supports_model`'s prefix loop alone — it only returns a bool, so order cannot change its answer. Not touched: `_PRICING`. `gpt-4.1`/`gpt-5` also fall through to the GPT-4o pricing tier, which skews cost reporting, but that is a separate concern with its own verification burden (published rates, staleness window) and does not belong in a tokenizer-correctness fix. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check` + `ruff format`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality ### Test Output 12 of 24 new cases fail without the fix; the 12 that pass are the "must not regress" rows (`gpt-4`, `gpt-4-turbo`, `gpt-4o`, `o3`, `gpt-3.5-turbo`) — included precisely so the longest-prefix change can't quietly move them: ```text $ git stash push headroom/providers/openai.py && pytest tests/test_openai_model_table_resolution.py -q FAILED ...::test_context_limit_prefers_the_most_specific_prefix[gpt-4.1-mini-1047576] FAILED ...::test_context_limit_prefers_the_most_specific_prefix[gpt-4.1-nano-1047576] FAILED ...::test_context_limit_prefers_the_most_specific_prefix[gpt-4.1-2025-04-14-1047576] FAILED ...::test_context_limit_prefers_the_most_specific_prefix[gpt-4-32k-0613-32768] FAILED ...::test_context_limit_prefers_the_most_specific_prefix[gpt-5-400000] FAILED ...::test_context_limit_prefers_the_most_specific_prefix[gpt-5-mini-400000] FAILED ...::test_context_limit_prefers_the_most_specific_prefix[o4-mini-200000] FAILED ...::test_encoding_prefers_the_most_specific_prefix[gpt-4.1-o200k_base] FAILED ...::test_encoding_prefers_the_most_specific_prefix[gpt-4.1-mini-o200k_base] FAILED ...::test_encoding_prefers_the_most_specific_prefix[gpt-4.1-2025-04-14-o200k_base] FAILED ...::test_cjk_is_not_over_counted_for_gpt_41 12 failed, 12 passed in 0.70s $ git stash pop && pytest tests/test_openai_model_table_resolution.py -q 24 passed in 0.43s ``` Regression check — 119 suites touching openai / cost / savings / token / compress / outcome / budget, this branch vs clean `main` in the same environment, comparing failure *sets*: ```text branch : 5 failed, 1486 passed, 86 skipped in 111.01s main : 5 failed, 1453 passed, 86 skipped in 132.07s NEW failures introduced: (none) pre-existing on both: test_bundled_tools_savings.py::test_compressed_payload_preserves_answer_anthropic test_compress_route_tokenizer_by_model.py::...[deepseek/deepseek-v4] (needs `transformers`) test_compress_route_tokenizer_by_model.py::...[deepseek/deepseek-v4] (needs `transformers`) test_image_compressor_singleton_reuse.py::test_onnx_router_is_built_once_and_cached test_openai_streaming_backend.py::...test_litellm_vertex_streaming_preserves_max_tokens_and_vendor_fields ``` ```text $ ruff check headroom/providers/openai.py tests/test_openai_model_table_resolution.py All checks passed! $ mypy headroom/providers/openai.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - **Environment:** macOS, Python 3.13.7, isolated worktree at `upstream/main` (`ad56dd38`), no `litellm` installed (matching a Python 3.14+ install, where the dep marker excludes it). - **Exact command / steps:** resolve context limit + encoding for each model against published OpenAI values, before and after. - **Observed result:** ```text before after model limit enc model limit enc gpt-4.1 8192 cl100k gpt-4.1 1047576 o200k_base gpt-4.1-mini 8192 cl100k gpt-4.1-mini 1047576 o200k_base gpt-4.1-nano 8192 cl100k gpt-4.1-nano 1047576 o200k_base gpt-4.1-2025-04-14 8192 cl100k gpt-4.1-2025-04-14 1047576 o200k_base gpt-4-32k-0613 8192 cl100k gpt-4-32k-0613 32768 cl100k_base gpt-5 128000 o200k gpt-5 400000 o200k_base o4-mini 128000 o200k o4-mini 200000 o200k_base unchanged: gpt-4=8192/cl100k, gpt-4-turbo=128000/cl100k, gpt-4o=128000/o200k, o3=200000, gpt-3.5-turbo=16385/cl100k ``` |
||
|
Tejas Chopra
|
cd92ed52ff
|
fix(providers): give every model exactly one tokenizer (#2761)
## Description `/v1/chat/completions` is a multi-provider passthrough, but `OpenAIProvider` handed **any** unrecognized model a guessed `o200k_base` encoding. Kimi through Fireworks — a documented Headroom configuration — counted **~19% low**. This is not just an accuracy nit, because **two resolvers race on the same request**: - handlers count via the tokenizer registry (`count_tokens_offloaded` → `get_tokenizer(model)`) - `TransformPipeline` counts via `provider.get_token_counter(model)`, because the proxy builds its pipelines with `provider=self.openai_provider` (`server.py:953-958`) `tokens_saved = original_tokens - optimized_tokens`, and in token mode those two operands come from *different* resolvers (`handlers/openai.py:3150-3155` keeps the handler's `original_tokens` and takes the pipeline's `optimized_tokens`). When the rulers disagree the subtraction is noise — it can invent savings on an untouched request, or trip the `optimization inflated tokens` revert guard at `handlers/openai.py:3219` and throw away real compression. Measured on `main` before this change, same 2-message payload: | model | registry (handler) | provider (pipeline) | gap | |---|---|---|---| | `moonshotai/kimi-k2` | 686 | 554 | **19.2%** | | `accounts/fireworks/models/kimi-k2-instruct` | 686 | 554 | **19.2%** | | `gemini-2.5-pro` | 534 | 554 | 3.7% | | `command-r-plus` | 534 | 554 | 3.7% | | `mistral-large-latest` | 563 | 554 | 1.6% | | `gpt-4o` / `claude-sonnet-4-6` | 552 | 554 | 0.4% | The provider returned **554 for every model** — it was model-blind. This follows the precedent already documented in `tests/test_compress_route_tokenizer_by_model.py`: pinning one provider's counter for a multi-model route is the bug, and the registry is the canonical resolver (every registry tokenizer derives from `BaseTokenizer`, whose `_count_content_parts` ends in a serialize-and-count catch-all). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `_get_encoding_name_for_model` split into `_lookup_encoding_name` (returns `None` when nothing claims the model) plus the original fallback wrapper, so callers can distinguish "OpenAI model" from "guessed". - `OpenAIProvider.get_token_counter` defers to `get_tokenizer(model)` when no real tiktoken encoding claims the model. - Per-message overhead `4` → `3`. OpenAI's counting guide uses `tokens_per_message = 3` for every model since `gpt-3.5-turbo-0613`; only the retired `gpt-3.5-turbo-0301` used 4. Staying on 4 over-counted every message by one token *and* disagreed with the registry, so a 100-message conversation drifted by 100 tokens depending on who counted it. - `_token_counters` annotation widened to `dict[str, TokenCounter]`. Preserved deliberately: explicit `model -> encoding` mappings (custom config / `HEADROOM_MODEL_LIMITS`) still win, and genuine OpenAI models still use `OpenAITokenCounter`. Both are pinned by tests. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check` + `ruff format`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality ### Test Output New tests fail on `main` and pass here — 7 of 9 fail without the fix (the 2 that pass are the invariants the fix must not break): ```text $ git stash push headroom/providers/openai.py && pytest tests/test_provider_tokenizer_one_ruler.py -q FAILED ...::test_non_openai_models_resolve_to_the_registry_tokenizer[moonshotai/kimi-k2] FAILED ...::test_non_openai_models_resolve_to_the_registry_tokenizer[accounts/fireworks/models/kimi-k2-instruct] FAILED ...::test_non_openai_models_resolve_to_the_registry_tokenizer[gemini-2.5-pro] FAILED ...::test_non_openai_models_resolve_to_the_registry_tokenizer[command-r-plus] FAILED ...::test_non_openai_models_resolve_to_the_registry_tokenizer[claude-sonnet-4-6] FAILED ...::test_kimi_is_not_counted_with_an_openai_encoding FAILED ...::test_per_message_overhead_matches_openai_and_the_registry 7 failed, 2 passed in 0.49s $ git stash pop && pytest tests/test_provider_tokenizer_one_ruler.py -q 9 passed in 0.51s ``` Regression check — 119 suites touching openai / cost / savings / token / compress / outcome / budget, run on this branch and on clean `main` **in the same environment**, comparing failure *sets*: ```text branch : 5 failed, 1453 passed, 86 skipped in 97.60s main : 5 failed, 1453 passed, 86 skipped in 132.07s NEW failures introduced by fix: (none) pre-existing on both: test_bundled_tools_savings.py::test_compressed_payload_preserves_answer_anthropic test_compress_route_tokenizer_by_model.py::...[deepseek/deepseek-v4] (needs `transformers`) test_compress_route_tokenizer_by_model.py::...[deepseek/deepseek-v4] (needs `transformers`) test_image_compressor_singleton_reuse.py::test_onnx_router_is_built_once_and_cached test_openai_streaming_backend.py::...test_litellm_vertex_streaming_preserves_max_tokens_and_vendor_fields ``` ```text $ ruff check headroom/providers/openai.py tests/test_provider_tokenizer_one_ruler.py All checks passed! $ mypy headroom/providers/openai.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - **Environment:** macOS, Python 3.13.7, isolated worktree at `upstream/main` (`ad56dd38`), in-place `headroom/_core.abi3.so` copied in. - **Exact command / steps:** resolve both tokenizers for the same 2-message payload and compare, before and after. - **Observed result:** ```text before (main) gpt-4o registry=552 provider=554 DIVERGES 2 moonshotai/kimi-k2 registry=686 provider=554 DIVERGES 132 (19.2%) gemini-2.5-pro registry=534 provider=554 DIVERGES 20 (3.7%) command-r-plus registry=534 provider=554 DIVERGES 20 (3.7%) after (this branch) gpt-4o registry=552 provider=552 AGREE gpt-5 registry=552 provider=552 AGREE o4-mini registry=552 provider=552 AGREE claude-sonnet-4-6 registry=552 provider=552 AGREE moonshotai/kimi-k2 registry=686 provider=686 AGREE gemini-2.5-pro registry=534 provider=534 AGREE command-r-plus registry=534 provider=534 AGREE ``` ## Known remaining gap (deliberately not in this PR) Plain and `name`-bearing messages now agree exactly, but **tool-call accounting still differs** on genuine OpenAI models: ```text tool msg (tool_call_id) registry=11 provider=13 delta +2 assistant tool_calls registry=14 provider=21 delta +7 ``` `OpenAITokenCounter` adds flat guesses (`+10` per tool call, `+2` per `tool_call_id`); the registry serializes the real structure and counts it. I believe the registry is closer to what the model actually sees, but I could not ground-truth it — there are no recorded `usage.prompt_tokens` fixtures in `tests/parity/`, and I did not want to shift everyone's tool-heavy numbers on a hunch. Tool-heavy agent traffic is the dominant Headroom workload, so this deserves its own PR with a real API capture to compare against. Filing separately. |
||
|
Tejas Chopra
|
ad56dd382b
|
fix(router): compare token quantities in one unit (#2759)
## Description Two places compared a token quantity against something measured in a **different unit**. Both changed compression **behaviour**, not just reporting — which is the worse class. ### 1. The CONFIG branch put a word count in a token ratio `compressed_tokens = len(compressed.split())` was divided by `original_tokens`, which comes from `_estimate_tokens(content)`. Words run ~2.8× fewer than estimator tokens on config text, so a compressor that returned its input **byte-identically** scored ~0.36. `min_ratio` is 1.0 — accept any real shrink — so the router **accepted the no-op**: cached the result, pinned a frozen "compress" verdict, emitted a `router:config_compressor` label into `transforms_applied`, and recorded a fabricated saving to TOIN. ```text mkdocs.yml, compressor returns its input unchanged denominator (_estimate_tokens) = 936 OLD numerator len(split()) = 334 -> ratio 0.357 claims 64% saved ACCEPTED NEW numerator (_estimate_tokens) = 936 -> ratio 1.000 correctly rejected ``` The sibling TABULAR branch already used `_estimate_tokens`; CONFIG was the outlier. ### 2. The Kompress size gate tested a token cap in chars/4 `len(text_to_compress) > self._kompress_max_tokens * 4` under-counts anything denser than 4 chars/token, and compact JSON runs ~3.2. Against the 50,000-token default there is a band where an oversized payload passes: ```text records chars old_gate(len/4) new_gate(tokens) 2700 119,281 False False 4000 177,781 False True <- 44,445 vs 55,557 tokens, 11% over 4400 195,781 False True <- 48,945 vs 61,182 tokens, 22% over 5000 222,781 True True ``` Those payloads entered ONNX inference — exactly the >30s non-preemptible worker stall the gate exists to prevent (#1171). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `content_router.py` CONFIG branch — `compressed_tokens` now from `_estimate_tokens(compressed)`, matching its denominator and every sibling branch. - `content_router.py` Kompress gate — compared with `_estimate_tokens`, the unit the cap is actually expressed in. The extra O(n) char scan is negligible against the inference it guards. ## Testing - [x] Unit tests pass - [x] Linting passes (`ruff check` + `format --check`, pinned 0.15.17) - [x] New tests added - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_content_router_token_units.py -q 4 passed in 0.33s $ uvx ruff@0.15.17 check headroom/transforms/content_router.py tests/test_content_router_token_units.py All checks passed! ``` **Regression check against clean `upstream/main` in the same environment:** ```text tests/test_transforms/ + test_transforms_content_router.py + kompress suites upstream/main : 2 failed, 468 passed, 78 skipped this branch : 2 failed, 468 passed, 78 skipped failure sets : identical ``` The 2 failures are `test_kompress_failsafe`'s artifact-selection tests, which need a real `onnxruntime` this throwaway env lacks. Unrelated to this change. The 4 new tests pin the unit contract *and* the bounds of the disagreement band — including the cases where both formulations agree, so the band is demonstrated rather than assumed. ## Real Behavior Proof - **Environment:** macOS 26.4 arm64, isolated worktree off `upstream/main`. `content_router` needs the compiled `headroom._core`, which isn't in a fresh worktree (gitignored, built in-place); I copied the built `.so` in to run these, then removed it before committing. - **Observed:** both tables above are from running the real `_estimate_tokens` against the real thresholds, not reconstructed arithmetic. - **Not tested:** no live ONNX inference — the >30s stall the gate prevents is cited from #1171, not reproduced. The CONFIG no-op was demonstrated at the ratio level rather than by driving a stubbed compressor through `apply()`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] I did **not** edit `CHANGELOG.md` ## Related Third of three PRs from one tokenizer-consistency audit — see #2757 (litellm total-prompt / `--budget`) and #2758 (HuggingFace chat templates, `gpt-5`, gateway-wrapped names). Separate subsystems, separate risk. Known remaining from the same audit, not in any of the three: `_netcost_message_tokens` pricing an image by Python `repr` (34× over-count, flag-gated), three transforms reporting via `count_text(str(content))` where the pipeline uses `count_messages` (19% apart in one log file), `frozen_message_count` walking a chars/3.5 estimate against provider-reported cached tokens, and `target_ratio` honoured in words while documented as tokens. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Tejas Chopra
|
a033ac4176
|
fix(cost): send litellm the total prompt so --budget stops seeing $0 (#2757)
## Description **`--budget` has been silently inert on any cache-warm request** — which is the normal case in an agent session. This is a disabled control, not a metrics bug. `record_tokens` passed only the **uncached slice** as litellm's `prompt_tokens`. Measured, `litellm.cost_per_token` charges: ``` (prompt_tokens - cache_read - cache_creation) * input_rate + cache_read * read_rate + cache_creation * write_rate ``` So `prompt_tokens` is the **whole** prompt and litellm removes the cached parts itself. Handing it the uncached slice drives the input term **negative** as soon as anything is cached. `estimate_cost` ends with `float(total) if total > 0 else None`, so it returned `None`, no `CostEntry` was appended, and `check_budget()` saw **$0**. | model | 100k prompt, 80k cached — old call | booked | | --- | --- | --- | | `gpt-5` | **-$0.065000** | None | | `gpt-4o-mini` | **-$0.003000** | None | | `claude-sonnet-4-5` | **-$0.156000** | None | Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made Fixing the total alone would **over-charge OpenAI**, because two bugs are entangled here. OpenAI exposes no cache-write counter, so `_infer_openai_cache_write_tokens` uses the uncached portion as a write proxy. At two of the four inference sites (`openai.py:4304`, `openai.py:5413`) `uncached_input_tokens` is derived by subtracting **only** `cache_read`, so `cache_write_tokens` and `uncached_tokens` are **the same tokens**. Summing all three would double-count the prompt and charge a write premium OpenAI does not have. (The other two sites — `openai.py:3969` and `streaming.py:2024` — subtract both, so their buckets are genuinely disjoint; those are left alone.) - `cost.py` — `record_tokens` now passes `uncached + cache_read + cache_write` as the prompt total. - `cost.py` — new `cache_inferred: bool = False` parameter. When set, the inferred write is excluded from **both** the prompt total and the write premium. The default preserves behaviour for every provider that reports disjoint buckets. - `outcome.py` — plumbs `outcome.cache_inferred` through. The field already existed on `RequestOutcome` for the dashboard; it just never reached cost. - `handlers/openai.py` — sets `cache_inferred=True` at the two outcome sites whose buckets genuinely duplicate. ## Testing - [x] Unit tests pass (new file) - [x] Linting passes (`ruff check` + `format --check`, pinned 0.15.17) - [x] New tests added - [x] Manual testing performed ### Test Output ```text $ uvx ruff@0.15.17 check headroom/proxy/cost.py headroom/proxy/outcome.py headroom/proxy/handlers/openai.py All checks passed! $ uvx ruff@0.15.17 format --check <same three> 3 files already formatted $ pytest tests/test_cost_budget_total_prompt.py -q 5 passed in 0.26s ``` The 5 new tests assert on the **arguments handed to `estimate_cost`** rather than on dollar values, so they pin the contract that broke without depending on litellm's pricing tables — or on litellm being installed at all. ## Real Behavior Proof - **Environment:** macOS 26.4 arm64, isolated git worktree off `upstream/main`. **litellm's actual formula**, established by probe rather than assumed: ```text rates (claude-sonnet-4-5): input=3e-06 read=3e-07 write=3.75e-06 total=100k, 80k read, 5k write -> $0.087750 hypothesis (p-r)*ir + r*rr + w*wr = $0.102750 x hypothesis (p-r-w)*ir + r*rr + w*wr = $0.087750 <- matches ``` **Before → after:** ```text Anthropic, disjoint: uncached=900 read=48000 write=1500 OLD prompt=900 raw=-0.125775 booked=None <-- BUDGET BLIND NEW prompt=50400 raw= 0.022725 booked=0.022725 OpenAI, inferred write == uncached: uncached=20000 read=80000 write=20000 OLD prompt=20000 raw=-0.090000 booked=None <-- BUDGET BLIND NEW prompt=100000, write excluded raw= 0.035000 booked=0.035 truth=0.035000 ``` The OpenAI "after" equals the hand-computed truth `20,000*input + 80,000*read_rate` exactly. - **Not fully tested locally.** The cost/outcome suites show an **identical 10-failure set** on this branch and on clean `upstream/main` in the same throwaway env — all `ModuleNotFoundError: headroom._core`, the compiled Rust extension this machine cannot currently build. So they are environment-only, not regressions. One of them, `test_funnel_passes_canonical_record_tokens_shape`, covers the `record_tokens` call shape this PR changes, so **CI is the authoritative check for that one**. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] I did **not** edit `CHANGELOG.md` ## Context Found during a tokenizer-consistency audit that also turned up: HuggingFace-routed models counting a 6,000-char message as **2 tokens**, `gpt-5`/`o4-mini` falling to a char estimator, and the Kompress size gate missing its own cap by 24%. Those are separate PRs — different subsystems, different risk. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Tejas Chopra
|
06add9e9d8
|
fix(providers): stop pricing modern content blocks at zero (#2760)
## Description Each token counter in `headroom/providers/` had grown its own shortened content-block walker, handling only the shapes its provider was expected to send. Everything else fell through and contributed **nothing**. Measured on one 6,800-char block, via `count_messages` of a single-block message — so 7–8 is message overhead alone: | block type | OpenAI ctr | Anthropic ctr | |---|---|---| | `text` (control) | 3409 | 3748 | | `tool_result` | **8** | 3748 | | `thinking` | **8** | **7** | | `document` | **8** | **7** | | `mcp_tool_result` | **8** | **7** | | `output_text` | **8** | **7** | | `refusal` | **8** | **7** | Two things make this worse than a coverage gap: 1. **Each counter zeroed blocks from its own provider.** `output_text` and `refusal` are OpenAI Responses shapes; `thinking` and `document` are Anthropic's. 2. **These are the counters the live pipelines use.** `proxy/server.py` builds them with `AnthropicProvider` / `OpenAIProvider`, so this is the main request path — not an edge case. #2743 fixed this for `/v1/compress` only, by routing that route to the registry tokenizers, whose `BaseTokenizer` walker is complete. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made Rather than add a **fifth** partial walker, the counters now delegate to the audited one: - `tokenizers/base.py` — new `count_content_blocks(parts, count_text_fn)` plus a thin `_DelegatingBlockCounter` adapter, since the provider counters are not `BaseTokenizer` subclasses. `BaseTokenizer` itself is untouched. - `providers/openai.py`, `providers/anthropic.py`, `providers/openai_compatible.py` — list-content branches delegate. **Why delegate instead of adding a `count_text(str(block))` catch-all:** that would serialize a base64 blob and price it as text. `tiktoken_counter.py` already documents the failure — a 1MB image becomes ~330K phantom tokens. The shared walker gives media a pixel/byte-based estimate. **Scope:** the three counters that accumulate token counts. `google.py` and `cohere.py` extract a *text string* first and count that, so the same defect there needs a differently-shaped fix — left as a follow-up. ## Testing - [x] Unit tests pass - [x] Linting passes (`ruff check` + `format --check`, pinned 0.15.17) - [x] New tests added - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_provider_counter_content_blocks.py -q 12 passed in 0.52s $ uvx ruff@0.15.17 check headroom/ tests/... --exclude headroom/dashboard/templates All checks passed! ``` **After the fix**, every shape lands within ~1% of the equivalent plain text, and media stays bounded: ```text block OpenAI Anthropic text (control) 3409 3748 tool_result 3409 3748 thinking 3419 3759 document 3423 3763 mcp_tool_result 3422 3762 output_text 3420 3760 refusal 3421 3761 image b64 200KB 1608 1607 <- pixel estimate, not ~50K as text ``` ## Real Behavior Proof — including a regression I caught **This change flipped an existing test**, and I only found it because every suite was run against clean `upstream/main` in the same environment with the failure sets diffed: ```text before the test rewrite: upstream/main : 1 failed, 104 passed this branch : 2 failed, 103 passed <- regression diff : + test_openai_compatible_token_counter_ignores_unhandled_content_shapes ``` That test asserted `content: [{"type": "image"}, 123] == 8` — i.e. it **pinned the defect**, that unhandled shapes contribute nothing. Rewritten as `..._prices_declared_media`: a declared image is now priced (1608) while a bare int is still correctly ignored (8), with the rationale in the docstring. ```text after the rewrite: upstream/main : 1 failed, 104 passed, 10 skipped, 25 errors this branch : 1 failed, 104 passed, 10 skipped, 25 errors failure sets : IDENTICAL ``` - **Pre-existing, not from this change:** the 1 failure and all 25 errors. The errors are all in `test_compress_route_tokenizer_by_model.py`, whose loopback `TestClient` fixture this throwaway env cannot satisfy. - **Environment note:** `content_router` and several suites need the compiled `headroom._core`, which isn't in a fresh worktree (gitignored, built in-place). I copied the built `.so` in to run these and removed it before committing. - **Not tested:** no live provider call, so the *absolute* accuracy of the 1600 image estimate against a real Anthropic/OpenAI bill is unverified — it is the value `BaseTokenizer` already used, and this PR only changes which blocks reach it. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] I did **not** edit `CHANGELOG.md` ## Related Fourth PR from one tokenizer-consistency audit: #2757 (litellm total prompt / `--budget`), #2758 (HuggingFace chat templates, `gpt-5`, gateway-wrapped names), #2759 (router token units). Plus #2756, which splits the local/provider token scales in `RequestOutcome`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Tejas Chopra
|
0ed306b22b
|
fix(tokenizers): count HuggingFace chat templates, and resolve gpt-5 / gateway-wrapped names (#2758)
## Description Three tokenizer-selection defects, all measured against real counters on identical text. ### 1. HuggingFace-routed models counted a whole conversation as **2 tokens** `transformers >= 5` defaults `apply_chat_template(tokenize=True)` to `return_dict=True` and returns a `BatchEncoding`, so `len(formatted)` counted **dict keys** — `input_ids`, `attention_mask` — instead of tokens. ```text Qwen2.5-72B, one 6,000-char message before: count_messages = 2 count_message = -1 after : count_messages = 1020 count_message = 1017 true : ~1003 ``` `count_message` goes negative because `BaseTokenizer` subtracts a 3-token reply overhead from it. A **~99.8% undercount** on every HF-routed family whose resolved tokenizer carries a chat template — llama, qwen, deepseek, phi, yi, falcon, starcoder. `pyproject.toml` pins `transformers>=5.5.0,<6.0`, so the affected version is the only installable one, and nothing covered `count_messages`. It hid behind a second bug while I reproduced it: `DeepSeek-V3` mis-resolves to `deepseek-llm-7b-base` (a 2023 model with **no** chat template), which falls back to the estimator and looks fine. That mis-resolution is left for a follow-up. ### 2. The current OpenAI flagships had no pattern `MODEL_PATTERNS` stopped at `^gpt-4` / `^o1` / `^o3`: ```text gpt-5, gpt-5.1, gpt-5-mini, gpt-5.1-codex, o4-mini -> EstimatingTokenCounter ``` Deviation vs the correct `o200k` encoding: **+20% English, -33% JSON, -44% logs.** ### 3. Every pattern is `^`-anchored, so gateway-wrapped ids matched nothing ```text bedrock/anthropic.claude-3-5-sonnet -> EstimatingTokenCounter vertex_ai/claude-sonnet-4-6 -> EstimatingTokenCounter openrouter/anthropic/claude-sonnet-4-6 -> EstimatingTokenCounter us.anthropic.claude-sonnet-4-6-v1:0 -> EstimatingTokenCounter azure/gpt-4o -> EstimatingTokenCounter ``` Deviation: **+15% English, -33% JSON, -38% logs.** Not hypothetical — `handlers/openai.py` already documents that LiteLLM's `headroom` guardrail passes exactly these forms. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `tokenizers/huggingface.py` — pass `return_dict=False` to `apply_chat_template`. - `tokenizers/registry.py` — add `^gpt-5` and `^o4` to `MODEL_PATTERNS`. - `tokenizers/registry.py` — new `_name_candidates()`; `_detect_backend` now tries progressively-unwrapped forms: path segments stripped left-to-right, then Bedrock's dotted `[region.]vendor.model`. **Why candidates rather than rewriting the name:** the full name is candidate 0, so no currently-correct resolution can move, and an unknown alias still falls back to estimation rather than matching by accident. The estimator is a legitimate *fallback*; the bug was reaching it when a real tokenizer for that family exists. ## Testing - [x] Unit tests pass - [x] Linting passes (`ruff check` + `format --check`, pinned 0.15.17) - [x] New tests added - [x] Manual testing performed ### Test Output ```text $ uvx ruff@0.15.17 check headroom/ tests/test_tokenizer_selection_coverage.py --exclude headroom/dashboard/templates All checks passed! $ pytest tests/test_tokenizer_selection_coverage.py -q 20 passed in 0.60s $ pytest tests/test_huggingface_tokenizer_timeout.py tests/test_tokenizers/ -q this branch: 12 passed clean upstream/main: 12 passed <- no regression ``` 20 new tests cover all three defects **plus** the no-regression cases: bare names unchanged, unknown aliases still estimated, wrapped Gemini matching its bare form exactly, and candidate ordering/dedup. ## Real Behavior Proof - **Environment:** macOS 26.4 arm64, isolated worktree off `upstream/main`. The HF measurement used a real `transformers 5.14.1` with `Qwen/Qwen2.5-72B` from the local HF cache. **After the fix, resolution across every form a gateway realistically sends:** ```text gpt-4o TiktokenCounter gpt-5 TiktokenCounter <- was Estimating gpt-5.1 TiktokenCounter <- was Estimating o3-mini TiktokenCounter o4-mini TiktokenCounter <- was Estimating claude-sonnet-4-6 TiktokenCounter bedrock/anthropic.claude-3-5-sonnet TiktokenCounter <- was Estimating anthropic.claude-3-5-sonnet-20241022-v2:0 TiktokenCounter <- was Estimating us.anthropic.claude-sonnet-4-6-v1:0 TiktokenCounter <- was Estimating vertex_ai/claude-sonnet-4-6 TiktokenCounter <- was Estimating openrouter/anthropic/claude-sonnet-4-6 TiktokenCounter <- was Estimating azure/gpt-4o TiktokenCounter <- was Estimating vertex_ai/gemini-2.5-pro EstimatingTokenCounter (google backend, correct) groq/llama-3.3-70b-versatile HuggingFaceTokenizer <- was Estimating my-gateway/big-model EstimatingTokenCounter (correct fallback) ``` `vertex_ai/gemini-2.5-pro` and `gemini-2.5-pro` return **identical** counts (600 on the same input), confirming the prefix strip reaches the google backend rather than the generic fallback. - **Not fully tested locally:** `tests/test_evals_cjk_tokenization.py` cannot collect in this env — `ModuleNotFoundError: headroom._core`, the compiled Rust extension this machine can't currently build. Identical on baseline, so CI is the check there. It is CJK-related and this PR changes encoding selection for `gpt-5`/`o4`/wrapped names, so it's the suite most worth watching. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] I did **not** edit `CHANGELOG.md` ## Known follow-ups, deliberately not here - `DeepSeek-V3` → `deepseek-llm-7b-base`, `Qwen/Qwen2.5-72B` → `Qwen/Qwen-7B`: `get_tokenizer_name` prefix-matches against the whole string including the org segment, and has no version boundary. - `get_encoding_for_model` is case-sensitive while `_detect_backend` lowercases, so `GPT-4O` gets `cl100k` (+38.9% on CJK). - `providers/openai.py` has a second, divergent encoding resolver — it disagrees with `tokenizers/` on `gpt-4.1`, `gpt-5`, `text-embedding-3-large`, `davinci`. - Provider counters price most modern content blocks at literally zero (`thinking`, `document`, `mcp_tool_result`, and OpenAI's own `output_text`/`refusal`). 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
nangsontay
|
184146b688
|
fix(savings): surface request growth the tok_saved clamp swallows (#2708)
## Description `tokens_saved` is clamped at zero, so a request the proxy forwards **larger** than it arrived is indistinguishable in the PERF line from one it simply could not compress. Both read `tok_saved=0`. That ambiguity hides real regressions. Anything that appends to the body after compression — proactive context expansion, memory injection — can outweigh the compression it sits on top of and still look like a neutral turn. On the session that prompted this, a request went from 55,161 tokens in to 57,845 out and reported `tok_saved=0`, for 19 consecutive turns, with nothing in the logs distinguishing it from a turn with nothing left to compress. This reports the swallowed amount as `tok_inflated`. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `RequestOutcome.tokens_inflated`: `max(0, optimized_tokens - original_tokens)`, derived from two counts the outcome already carries — **no new plumbing at any of the emit sites**. - Added `tok_inflated=` to the PERF log line, next to `tok_saved=`. Diagnostic only, deliberately. It does **not** feed `tokens_saved` or `attempted_input_tokens`, for two reasons: 1. `attempted_input_tokens = optimized_tokens + tokens_saved` is a *size*, not a signed delta. Letting the second term go negative makes it smaller than the bytes actually forwarded, corrupting the active-savings denominator. 2. Injection paths already book their own cost through the retrieval-drawback channel. A negative landing in `tokens_saved` as well would count the same loss twice. So the clamp stays and the hidden number surfaces beside it. Worth noting there is already a revert-on-inflation guard *before* compression's own inflation can escape (`anthropic.py`, "Optimization inflated tokens … reverting to original messages") — it is only growth added *after* that point which the clamp was silently absorbing. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_request_outcome.py tests/test_cli_perf_format.py -q ============================== 54 passed in 1.56s ============================== $ pytest tests/ -q -k "outcome or perf or savings or stats" ======= 510 passed, 33 skipped, 9674 deselected, 542 warnings in 40.37s ======== $ ruff check headroom/proxy/outcome.py tests/test_request_outcome.py All checks passed! $ ruff format --check headroom/proxy/outcome.py tests/test_request_outcome.py 2 files already formatted $ mypy headroom/proxy/outcome.py --ignore-missing-imports Success: no issues found in 1 source file ``` Four new tests pin the distinction that was missing: shrank (0), no-op compression (0, and `tok_saved` also 0 — the two cases that used to look identical), grew (reports 2684 while `tok_saved` stays 0), and that `attempted_input_tokens` / `savings_pct` keep their unsigned semantics. `tests/test_cli_perf_format.py` parses hand-written PERF fixtures by field name, so adding a field does not disturb it — verified green above. ## Real Behavior Proof ### The field catching a real inflating request - Environment: macOS 15 (arm64), Python 3.13.14. A proxy booted from this branch: `headroom proxy --mode token --backend anthropic --anthropic-api-url http://127.0.0.1:<stub>`, isolated `HOME` so the run could not touch a developer's live logs/store, `HF_HOME` pointed at cached kompress weights so the lossy+CCR-marker path is exercised and proactive expansion can actually arm. - Exact command / steps: three requests over one conversation through the real HTTP path (`x-headroom-cwd: /tmp/proof`, `user-agent: claude-code/1.4.2`): a user turn carrying the real `~/.claude/rules/*.md` text (~8.2k tokens); then `assistant` + a short user turn so that block becomes compressible and gets tracked as a CCR entry; then a follow-up whose leading text block shares vocabulary with it, so proactive expansion fires and appends the original — which is how a request ends up leaving larger than it arrived. PERF lines read from the isolated `~/.headroom/logs/proxy.log`. - Observed result: real PERF output from that run — ```text msgs=1 tok_before=8170 tok_after=9553 tok_saved=0 tok_inflated=1383 ... transforms=router:text_block:mixed msgs=3 tok_before=8184 tok_after=9567 tok_saved=0 tok_inflated=1383 ... transforms=router:text_block:mixed msgs=5 tok_before=8245 tok_after=11880 tok_saved=0 tok_inflated=3635 ... transforms=router:text_block:mixed ``` Correlated from the same run: `CCR Tracker: Proactively expanded f0cf4efb42373ec225f57725 (1417 items)`, and the stub upstream confirms the block reached the wire (`has_expansion_block: true`, forwarded body 54,130 B on the third request). Every one of those turns reports `tok_saved=0`. Before this change that is all the log said, and it is the same thing it says when there was simply nothing left to compress. `tok_inflated=3635` is the number that was missing. For contrast, the same scenario run against a build where the request genuinely shrinks reported `tok_before=8245 tok_after=7359 tok_saved=886` — that build predates this field, so it does not print `tok_inflated`; the point is only that the inflating and shrinking cases are the two states the field has to separate, and on `main` today both render as `tok_saved=0` whenever the growth path is taken. The `tok_inflated=0` case on a shrinking request is covered by unit test. ### Scale of what was hidden From a live `--mode token` proxy on Claude Code traffic across four rotated logs: **305 of 3,743 requests (8%)** had `tok_after > tok_before` while every one reported `tok_saved=0` — 192,829 tokens of growth rendered as "nothing to compress". The worst single session held +2,643/turn for 19 consecutive turns. - Not tested: the `headroom perf` CLI was not run against a real log file containing the new field — it parses by field name and `tests/test_cli_perf_format.py` is green, but that is test-level rather than end-to-end evidence. Streaming responses were not exercised (the stub replies non-streaming), so the streaming emit path carries the new field on the strength of sharing `emit_request_outcome` rather than by observation. No dashboard or Prometheus consumer was re-run. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — one added field on an existing log line; no user-facing surface. ## Additional Notes - Independent of #2706 and #2707 — verified `conflicts=0` via `git merge-tree`; mergeable in any order. - Related but deliberately out of scope: `main` has no producer for retrieval-cost accounting (`record_savings_event` takes no `kind`/`tokens_retrieved`, and nothing writes `tokens_retrieved` anywhere), so proactive expansion's cost is not booked into net savings at all. Adding that channel is a cross-cutting accounting change and belongs in its own PR; this one only makes the growth visible in the log. |
||
|
Rod Boev
|
dcb674b5e4
|
fix(compression): honor qualified CCR names across integrations (#2698)
## Description Three compression consumers compare tool names against the bare literal `headroom_retrieve`, so the qualified forms MCP clients actually send (`mcp__Headroom__headroom_retrieve`, `mcp_Headroom_headroom_retrieve`) slip past the guard and get recompressed. `SmartCrusher.apply` has the bare comparison at both its OpenAI `role=tool` site and its Anthropic `tool_result` block site; the LangGraph compressor and the Strands hook have no tool-name check at all. Recompressing already-retrieved CCR content mints a new `<<ccr:hash>>` marker the agent cannot redeem. `headroom.config.is_tool_excluded` already owns alias resolution, including the MCP wrapper forms. This routes all three consumers through it instead of adding a second name matcher. Closes #2656. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `SmartCrusher.apply` routes both its `role=tool` and its Anthropic `tool_result` guards through `is_tool_excluded` - `_should_skip` in the LangGraph compressor takes the tool name and skips excluded tools; tool-call names are indexed by id so a `ToolMessage` without a copied `name` is still classifiable - `_should_skip_compression` in the Strands hook takes the tool name and skips excluded tools, recording `tool_excluded` - regressions for the qualified and bare names across all three consumers, the Anthropic block shape, the MCP wrapper entry point, and a near-match name that must still compress - a LangGraph regression for incomplete tool-call metadata that continues to a later qualified call ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output `pytest tests/test_smart_crusher.py tests/integrations/test_langgraph.py tests/integrations/test_strands tests/test_transforms/test_smart_crusher_ccr_retrieve_exemption.py -q` ```text tests\test_smart_crusher.py ............ [ 10%] tests\integrations\test_langgraph.py ..... [ 15%] tests\integrations\test_strands\test_ccr_exclusion.py ..... [ 19%] tests\integrations\test_strands\test_hooks.py sssssssss [ 27%] tests\integrations\test_strands\test_hooks_unit.py ssssssssssssssssssssssssssssssssss [ 57%] tests\integrations\test_strands\test_model.py ssssssssssssssss [ 71%] tests\integrations\test_strands\test_model_unit.py sssssssssssssssssssssssssss [ 95%] tests\test_transforms\test_smart_crusher_ccr_retrieve_exemption.py ..... [100%] 28 passed, 86 skipped in the focused invariant suite ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12.13, `headroom._core` built - Exact command / steps: `uv run pytest tests/test_smart_crusher.py tests/integrations/test_langgraph.py tests/integrations/test_strands -q`, and the same suite run against the pre-change implementation with the new tests in place - Observed result: before the change, five regressions fail. `SmartCrusher` returns non-byte-identical content for a `mcp__Headroom__headroom_retrieve` result, the LangGraph compressor replaces the message content, and the Strands hook returns `"compressed"` in place of the tool output. After the change all three preserve the content byte-for-byte, incomplete LangGraph tool-call metadata is ignored while the later qualified call remains indexed, the Strands hook records `tool_excluded` and never calls the crusher, and `HeadroomMCPCompressor.compress` returns the payload unchanged. `mcp__Headroom__headroom_retrieve_extra` still compresses in all three, and the Kompress and ContentRouter suites are unchanged. - Not tested: the optional Strands package, so the additions to `tests/integrations/test_strands/test_hooks_unit.py` skip locally ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes One deliberate divergence from the issue: the suggested snippet passes `DEFAULT_VERBATIM_EXCLUDE_TOOLS` to `is_tool_excluded`, but that constant holds only `WebSearch`, `WebFetch`, `web_search`, `web_fetch`. Applied literally it would drop `headroom_retrieve` from the comparison entirely and delete the #1077 guard these two SmartCrusher sites exist to enforce. This passes `(CCR_TOOL_NAME,)` so each guard keeps doing the one thing it documents. If you'd rather these paths also honor the verbatim-exclude set, the tuple can become `(CCR_TOOL_NAME, *DEFAULT_VERBATIM_EXCLUDE_TOOLS)` — the CCR name has to stay in it either way. Adjacent work: PR #2654 covers `ContentRouter` only. |
||
|
michaeltarleton
|
677e09735a
|
fix(transforms): stop ContentRouter recompressing headroom_retrieve results (#2654)
## Description `ContentRouter` (the transform actually registered in the default/proxy compression pipeline -- see `transforms/pipeline.py`) recompresses the output of its own `headroom_retrieve` tool. That tool's entire contract is returning already-retrieved, original content verbatim; recompressing it produces a new `<<ccr:hash>>` marker the caller can never redeem -- an unresolvable retrieval loop. `SmartCrusher` already has a guard against this exact failure mode (#1077), but only on its `apply()` entry point. `ContentRouter` calls the lower-level `SmartCrusher.crush()` directly, bypassing that guard entirely, since `crush()` takes a raw content string with no tool identity at all. Closes #1077 (reopens the same failure mode ContentRouter's own call path, which #1077's original fix did not cover). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `transforms/content_router.py`: adds an unconditional guard to all three of the places `ContentRouter` can hand a `headroom_retrieve` result to compression: the OpenAI-shape `role:"tool"`/legacy `role:"function"` string-content loop, the Anthropic-shape `tool_result` block loop, and a third, distinct shape -- top-level `{"type": "text"}` blocks under a `role:"tool"`/`"function"` message that never go through a `tool_result` wrapper (a real, already-tested wire shape in this codebase; see `test_tool_role_text_blocks_compressed_by_default`). All three use `is_tool_excluded()` (not a bare comparison) because MCP-served tools appear here under their qualified form, e.g. `mcp__headroom__headroom_retrieve`. Legacy `role:"function"` messages carry no call id in that shape, so the tool name is read directly off the message's `name` field instead of through the id-keyed `tool_name_map`. - Hoisted the per-iteration `is_tool_excluded(..., ("headroom_retrieve",))` calls into a single precomputed `ccr_retrieve_tool_ids` set, computed once alongside the existing `excluded_tool_ids` set, rather than recomputing aliases on every message/block. - `config.py`: adds `"headroom_retrieve"` to `DEFAULT_EXCLUDE_TOOLS` and `DEFAULT_VERBATIM_EXCLUDE_TOOLS` -- this also covers a third path (cross-turn message dedup, `_cross_turn_dedup_messages`) that consults the same frozensets and has no dedicated guard of its own. Also hardens `_tool_name_aliases()` against a non-string tool name (pre-existing fragility, not introduced by this PR, but shares the same call path) by returning no aliases instead of crashing on `.lower()`. - Documentation: updated `ContentRouterConfig.exclude_tools`'s field comment (was stale -- didn't mention this override is unconditional even when a caller explicitly empties `exclude_tools`), and added a comment on `DEFAULT_VERBATIM_EXCLUDE_TOOLS` noting all three real consumers. - Kept `"headroom_retrieve"` as a literal string (matching every other entry in those frozensets) rather than importing the existing `CCR_TOOL_NAME` constant from `ccr.tool_injection` into `content_router.py` -- that module is imported eagerly by `pipeline.py` (unlike `smart_crusher.py`, which imports the same constant lazily), so pulling in `headroom.ccr` there would add a new eager-import edge to a hot module for a one-line DRY win. Happy to change this if a maintainer prefers the constant. **Known, accepted tradeoff:** `is_tool_excluded()`'s alias matching strips any `mcp__<server>__` prefix before comparing, so a third-party MCP server exposing a tool literally named `headroom_retrieve` would also match. Narrowing this to headroom's own server specifically would need a bespoke check inconsistent with how every other excluded-tool entry is matched in this codebase; given how specific the name is, the collision risk is accepted rather than special-cased. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_transforms/ tests/test_transforms_content_router.py -q 1 failed, 420 passed, 62 skipped in 12.50s FAILED tests/test_transforms/test_kompress_compressor.py::...test_onnx_session_options_read_thread_caps (pre-existing, unrelated to this diff -- confirmed via `git stash` that it fails identically against unmodified upstream/main; an ONNX thread-cap assertion, not a compression-routing test) $ uv run ruff check headroom/config.py headroom/transforms/content_router.py \ tests/test_transforms/test_content_router_ccr_retrieve_exemption.py \ tests/test_transforms_content_router.py tests/test_transforms/test_content_router.py All checks passed! $ uv run ruff format --check <same files> 5 files already formatted $ uv run mypy headroom/config.py headroom/transforms/content_router.py Success: no issues found in 2 source files ``` - `tests/test_transforms/test_content_router_ccr_retrieve_exemption.py`: 10 tests -- MCP-qualified name (Anthropic + OpenAI shape), bare name, unconditional-even-with- `exclude_tools=frozenset()`, negative control (normal tools still compressed, asserted via the absence of the `router:excluded:ccr_retrieve` marker), the top-level-text-block shape, legacy `role:"function"`, litellm list-form content nested in a `tool_result` block, mixed retrieve+normal blocks in one turn, and a content well below the compression floor (proving the guard is size-independent). - `tests/test_transforms/test_content_router.py`: `test_anthropic_mcp_bare_tool_alias_exclude_tools` (#1822) updated to assert the new, stronger byte-verbatim guarantee for `headroom_retrieve` specifically; `test_anthropic_mcp_bare_tool_alias_exclude_tools_generic` added to keep the original #1822 general-mechanism coverage (bare-alias matching for an arbitrary, non-exempt tool). - `tests/test_transforms_content_router.py`: updated 10 pre-existing `_process_content_blocks()` unit tests for the new `ccr_retrieve_tool_ids` parameter (all pass empty sets -- none of those tests involve `headroom_retrieve`). - Verified the local installed package copy (a separate, drifted internal version) with a standalone repro script exercising the two new shapes directly against `ContentRouter.apply()` -- both correctly report `router:excluded:ccr_retrieve`. ## Real Behavior Proof - Environment: Windows 11, Python 3.13.7, `uv sync --extra dev` on this branch. - Exact command / steps: standalone repro building an assistant `tool_use` for `mcp__headroom__headroom_retrieve` paired with a large-JSON `tool_result`, through `ContentRouter().apply()`; repeated for the top-level-text-block and legacy-`function`-role shapes. - Observed result: unpatched (Anthropic `tool_result` shape, `git stash` to `upstream/main`), the retrieve output was rewritten 3680 -> 1861 bytes (mangled into a compact tabular form); patched (this branch), it is forwarded 3680 -> 3680 bytes byte-identical, no `<<ccr:` marker present. The two additional shapes fixed in this PR's second commit -- top-level text block under `role:"tool"`, and legacy OpenAI `role:"function"` -- both report `excluded=True` (protected) against this branch, where they reported `excluded=False` (recompressed) before the second commit. - Not tested: the actual `headroom mcp serve` + `headroom wrap` proxy end-to-end over a live Anthropic API call (would need API credentials); the OpenAI-chat-completions `CompressionUnit` path (out of scope, see #1176 below); the opt-in `ToolResultInterceptorTransform` path. ## Relationship to other issues/PRs - Issue #1077 (closed) is this exact bug; PR #1323 fixed it only for `SmartCrusher.apply()`'s own call path (the "legacy" pipeline path, per `smart_crusher.py`'s own comment), not `ContentRouter`, which is what the default/proxy pipeline actually uses. - Open PR #1176 addresses an adjacent, non-overlapping gap: the `CompressionUnit`-based OpenAI chat-completions path (`router.compress()` calls in `transforms/compression_units.py`/`compression_batches.py`), which has no tool-identity context at all and needs its own capture/restore mechanism. This PR does not touch that path. - Filed #2656 as a follow-up: code review on this PR found the same bug class still reachable through `SmartCrusher.apply()`'s own bare-name guard (not alias-aware, so it misses the MCP-qualified form) and through two unguarded direct `.crush()` calls in the LangGraph and Strands integrations. Both are pre-existing, narrower/separate call paths from `ContentRouter`'s primary proxy pipeline, so tracking them separately keeps this PR reviewable as one logical change. - Also not covered by this PR (flagging rather than silently omitting): `proxy/system_compaction.py`'s `router.compress(text, context="")` call, and the opt-in `ToolResultInterceptorTransform` (`HEADROOM_INTERCEPT_ENABLED=1`) -- neither was checked for CCR-awareness. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` -- it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A -- this is a backend compression-routing fix with no UI surface. ## Additional Notes This PR is two commits: the first commit added the initial two-loop guard; a second commit followed after code review found the guard was incomplete for two additional wire shapes (top-level text blocks, legacy `role:"function"`) and added the missing test coverage plus a few cleanup items (deduplicated guard logic, comment accuracy, a pre-existing non-string-tool-name fragility). See `Changes Made` above for the full list. Filed #2656 for the remaining out-of-scope gaps found during that same review. --------- Co-authored-by: Michael Tarleton <mtarleton@istation.com> |
||
|
JD Davis
|
13a310a00d
|
feat(claude): support Claude Code in VS Code (#2752)
## Description Add first-class Headroom support for the official Claude Code extension in VS Code. The new wrapper starts the local proxy, configures the Claude Code user settings consumed by the embedded extension process, preserves authentication and model selection, and provides a conflict-safe reversible unwrap lifecycle. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add `headroom wrap vscode-claude` and `headroom unwrap vscode-claude`. - Configure project-scoped `ANTHROPIC_BASE_URL` plus `ENABLE_TOOL_SEARCH=true` in Claude Code user settings while preserving existing values. - Respect `CLAUDE_CONFIG_DIR`, macOS/Linux home paths, Windows `USERPROFILE`, custom `--settings-file`, and `--no-configure`. - Add durable Headroom-owned restore state and refuse malformed settings or conflicting user edits. - Add unit, CLI, and Docker-harness e2e coverage for configuration, real proxy forwarding, and restoration. - Document setup, remote development, undo, and troubleshooting. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ UV_NO_SYNC=1 uv run pytest -q tests/test_provider_claude_vscode_config.py tests/test_cli/test_wrap_vscode_claude.py tests/test_cli/test_wrap_vscode.py tests/test_cli/test_wrap_claude_base_url.py tests/test_provider_copilot_vscode_config.py tests/test_copilot_auth.py 160 passed in 0.45s $ UV_NO_SYNC=1 uv run ruff check . All checks passed! $ UV_NO_SYNC=1 uv run mypy headroom Success: no issues found in 512 source files $ npm run build # from docs/ Compiled successfully; generated 155 static pages ``` ## Real Behavior Proof - Environment: macOS, Python 3.13 editable install, isolated temporary HOME and Claude settings, local mock Anthropic Messages upstream. - Exact command / steps: invoked the new `verify_vscode_claude_wrap` e2e function, which launched real `headroom wrap vscode-claude`, waited for proxy readiness, POSTed an Anthropic `/v1/messages` request through the generated project-scoped URL, stopped the wrapper, then ran `headroom unwrap vscode-claude`. - Observed result: HTTP 200 with the mock Claude response through Headroom; generated settings retained unrelated values and enabled tool deferral; unwrap restored the original Claude settings. - Not tested: real Anthropic account traffic or the full Docker image locally because Docker Desktop was unavailable. The same e2e function is wired into the existing Docker wrap CI job. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable; this adds CLI configuration and proxy routing without changing VS Code UI. ## Additional Notes The wrapper deliberately leaves the endpoint configured when stopped so requests fail closed instead of silently bypassing Headroom. `headroom unwrap vscode-claude` restores the exact prior managed values and preserves unrelated settings. --------- Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net> |
||
|
nangsontay
|
08fce29b47
|
fix(proxy): stop toggling headroom_retrieve in the Anthropic tools array (#2672)
## Description `should_inject_ccr_tool` deferred CCR tool injection whenever `frozen_message_count > 0`. Because `tools` is the head of Anthropic's cache key, that dropped a tool which was already inside the provider-cached prefix and invalidated the whole prefix — in both directions (`0 → >0` removes it; `>0 → 0` on proxy restart, `/model` switch, lineage eviction or TTL lapse adds it back). On three days of local proxy logs the turns that flipped injection state carried **44.7% of all cache-write tokens at a 52.0% hit rate**, against 98.1% for non-flipping turns. The log signature is `cache_read` alternating between two values exactly 172 tokens apart — the 464-byte tool definition. This deletes the gate and calls `apply_session_sticky_ccr_tool` directly, which is **what `openai.py` already does** — the two handlers now have the same shape. Net −61 production lines, no new state, no new config flag. Fixes defect 1 of #2671. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Performance improvement ## Changes Made - `headroom/proxy/ccr_marker_policy.py` — deleted the `should_inject_ccr_tool` gate; `apply_session_sticky_ccr_tool` is now the single decision point. - `headroom/proxy/handlers/anthropic.py` — calls `apply_session_sticky_ccr_tool` directly, matching `openai.py`. - `headroom/proxy/helpers.py` — dropped the now-unused gate plumbing. - `tests/test_proxy_anthropic_cache_stability.py` — new test asserting the forwarded `tools` array is byte-identical across a `frozen 0 → >0` transition. - `tests/test_ccr_marker_policy.py` — removed the three unit tests that pinned the deleted decision (they encoded the defect). - `tests/test_proxy/test_ccr_frozen_prefix_coupling.py` — same unredeemable-marker intent, re-pinned at the sticky helper. - `tests/test_proxy/test_anthropic_ccr_deferred_injection.py` — autouse reset fixture for the process-global `SessionCcrTracker` (separate commit). - Formatting-only follow-up commit applying `ruff format` (pinned 0.15.17) to the two test files above. ### Why deleting the gate is safe `apply_session_sticky_ccr_tool` already holds the correct rule. Its four branches, in order: | # | condition | action | |---|---|---| | 1 | tool already in the incoming tool list (client/MCP pre-registered) | skip; the client's bytes win | | 2 | `session_id is None` (WS / pre-session) | per-turn flag drives it verbatim | | 3 | session has done CCR | always inject the recorded golden bytes | | 4 | fresh session, no compression this turn | **skip** | Branch 4 is the safety property: a session that has never compressed still gets no tool, so removing the gate cannot start injecting into non-CCR conversations. Branch 3 is what the gate was starving. `has_new_ccr_markers` still gates first-time injection, so markers replayed from the previously-forwarded prefix cannot trigger one. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed The three deleted unit tests encoded the defect. Coverage moves to the property that actually matters and was previously untested: **the forwarded `tools` array must be byte-identical across a `frozen 0 → >0` transition.** That test asserts on the forwarded request body rather than on a policy function's return value; unit-testing the old policy in isolation is exactly what let a wrong-but-self-consistent decision pass. Verified failing on `upstream/main` with an assertion on the missing tool (not an `ImportError`, so it fails for the right reason). Full suite: same pre-existing unrelated failures as `upstream/main`, **zero new** (verified by running the whole suite on both revisions and diffing the failure sets). ### Test Output ```text $ uv run pytest tests/test_ccr_marker_policy.py \ tests/test_proxy/test_anthropic_ccr_deferred_injection.py \ tests/test_proxy/test_ccr_frozen_prefix_coupling.py \ tests/test_proxy_anthropic_cache_stability.py -q collected 48 items tests/test_ccr_marker_policy.py ..... [ 10%] tests/test_proxy/test_anthropic_ccr_deferred_injection.py .............. [ 39%] . [ 41%] tests/test_proxy/test_ccr_frozen_prefix_coupling.py .. [ 45%] tests/test_proxy_anthropic_cache_stability.py .......................... [100%] ======================= 48 passed, 2 warnings in 13.59s ======================== $ ruff check . All checks passed! $ ruff format --check . 1349 files already formatted ``` ## Real Behavior Proof - Environment: local macOS proxy serving live Claude Code traffic to the Anthropic API; baseline = 3 days of proxy logs on `upstream/main`, after = 5.5 hours with this change live. - Exact command / steps: ran the proxy with this branch built in, drove normal Claude Code sessions through it (including `/model` switches and proxy restarts, the two events that used to flip injection state), then parsed 235 real turns from the proxy logs with the same parser used for the baseline in #2671. - Observed result: flip turns fell from 177 (44.7% of all cache write) to 2 (1.7%); steady-state write share 1.192% → 0.867%; aggregate hit rate 86.75% → 89.21%; main conversation warm hit rate 98.1% → 97.70% (n=149). The 2 remaining "flips" have `cache_read == 0` — cold starts that the bucketing counts as a state change, not real flips. | metric | baseline | after | |---|---|---| | flip turns | 177, carrying 44.7% of all cache write | **2**, carrying **1.7%** | | main conv, warm | 98.1% | **97.70%** (n=149) | | steady-state write share | 1.192% | **0.867%** | | aggregate | 86.75% | **89.21%** | - Not tested: `mypy headroom` was not run locally for this body; the OpenAI handler path (unchanged by this PR); tracker state loss mid-session (see note below); and defect 2 of #2671 (the sub-call breakpoint), which is untouched and is now 54.9% of remaining cache write — that is why aggregate stays just under 90%. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes **Pre-existing and unchanged here:** if the tracker loses state mid-session while the transcript still carries markers, branch 4 returns no tool and those markers are unredeemable. `upstream/main` has no recovery for that; this PR neither creates nor fixes it. See my comment on #2500, which adds a recovery path for the related dangling-reference case. **N/A checklist items:** no documentation changes — this removes an internal policy function with no user-facing surface. `mypy headroom` left unchecked because it was not run for this body; CI covers it. **Merge-order conflict with #2500 (please read before landing either):** this PR *deletes* `should_inject_ccr_tool`, which is the exact function #2500 extends with `transcript_requires_tool`. Whichever lands second needs a semantic rebase, not just a textual one — git will not flag it. If this PR lands first, #2500's recovery path should re-target `apply_session_sticky_ccr_tool` (the sticky helper now owns the decision alone) or the handler call site in `handlers/anthropic.py`. If #2500 lands first, the gate deletion here still applies but the `transcript_requires_tool` override needs to move with it. Happy to do the rebase either way — say which order you prefer. |
||
|
Tejas Chopra
|
0221e7f240
|
fix(deps): bump aiohttp and cryptography to clear the CVEs blocking 0.34.0 (#2753)
## Description
`Dependency audit (pip-audit)` is the **only** failing check on the
0.34.0 release PR (#2679), so this blocks the release regardless of what
else lands in it.
`aiohttp 3.14.1` carries three advisories, all reachable through the
`--extra all` production set that CI audits (transitive via `litellm` /
`instructor` / `kubernetes` / `fsspec`):
| CVE | Impact | Fixed in |
|---|---|---|
| CVE-2026-69243 | Request smuggling via an edge case in the WebSocket
upgrade procedure (server-side component) | 3.14.2 |
| CVE-2026-69244 | Out-of-bounds heap read in the C response parser
building an error message for a malformed response — an
attacker-controlled server can DoS the client | **3.14.3** |
| CVE-2026-59881 | Decompresses frames with RSV1 set even when
`permessage-deflate` was not negotiated | 3.14.2 |
3.14.3 is the floor that clears all three.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- Lock-only bump via `uv lock --upgrade-package aiohttp`. **No
`pyproject.toml` constraint added** — every parent already permits
3.14.3, so a floor would be redundant surface to maintain.
- The diff also syncs `headroom-ai` `0.32.0` → `0.33.0` in the lock. `uv
lock` rewrites that from `pyproject.toml` (`version = "0.33.0"`); the
lock's record of the project's own version was stale. Same drift #2663
targets — happy to drop this PR if #2663 lands first and you'd rather
keep them separate.
Diff is exactly two version changes (plus their wheel-hash blocks).
## Testing
- [x] Manual testing performed
- [x] Linting passes — no Python source touched
### Test Output
```text
$ uv lock --upgrade-package aiohttp
Resolved 269 packages in 2.91s
Updated aiohttp v3.14.1 -> v3.14.3
Updated headroom-ai v0.32.0 -> v0.33.0
$ git diff --stat uv.lock
uv.lock | 456 +++++++++++++++++-------------------
1 file changed, 234 insertions(+), 222 deletions(-)
$ git diff uv.lock | grep -E '^[+-]version = '
-version = "3.14.1"
+version = "3.14.3"
-version = "0.32.0"
+version = "0.33.0"
```
## Real Behavior Proof
- **Environment:** macOS 26.4 arm64, `uv` 0.x from Homebrew, isolated
git worktree off `upstream/main` @ `
|
||
|
Raúl
|
3f2ca99fe1
|
fix(ci): restrict Codecov shard uploads (#2745)
## Description Closes #2744 Restrict each Codecov Action v5 matrix upload to its declared `coverage-${{ matrix.shard }}.xml` report. This prevents automatic discovery from uploading the unsharded `coverage.xml` alongside every shard. ## Type of Change - [x] Bug fix (non-breaking change fixes an issue) ## Changes Made - Set Codecov Action `disable_search: true` for Python shard uploads. - Add a CI workflow contract test that protects the explicit-report-only setup. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy headroom`) (not applicable: workflow/test-only change) - [x] New tests added new functionality - [x] Manual testing performed (not applicable: GitHub Actions will execute the workflow) ### Test Output ```text $ uv run --with ruff ruff format --check scripts/tests/test_ci_workflow.py 1 file already formatted $ uv run --with ruff ruff check scripts/tests/test_ci_workflow.py All checks passed! $ uv run --with pytest pytest scripts/tests/test_ci_workflow.py -q 2 passed ``` ## Real Behavior Proof - Environment: GitHub Actions Ubuntu runner using Python 3.12.13; Codecov Action v5. - Exact command / steps: Run the CI Python test matrix, which writes `coverage-${{ matrix.shard }}.xml`, then runs the Codecov Action upload step. Inspect the uploader's discovered/uploaded report list. - Observed result: Before this change, raw CI logs showed the Action explicitly uploading `coverage-2.xml` and additionally discovering/uploading `coverage.xml`. This PR configures `disable_search: true`; the workflow contract test confirms the explicit report setting and search disablement. Runtime upload evidence will be added from this draft PR's CI run. - Not tested: Codecov's final cross-shard patch calculation; that depends on Codecov processing the reports after CI completes. ## Review Readiness - [x] I have performed a self-review - [x] This PR ready for human review ## Checklist - [x] My code follows project's style guidelines - [x] I have performed a self-review - [x] I commented my code, particularly in hard-to-understand areas - [x] I made corresponding changes documentation (not applicable) - [x] My changes generate no new warnings - [x] I added tests prove my fix is effective or feature works - [x] New and existing unit tests pass locally changes - [x] I did **not** edit `CHANGELOG.md` — generated by release-please from Conventional Commit PR title (a CI guard enforces this) ## Additional Notes This is intentionally limited to the Codecov upload configuration and its workflow contract test. It does not include the unrelated Copilot Keychain fix. |
||
|
Tejas Chopra
|
6422a80a58
|
fix(compress): resolve the /v1/compress tokenizer per model, and document the real contract (#2743)
## Description
`/v1/compress` does no format conversion — callers send whichever wire
shape they already use — but the pipeline pinned **one provider's token
counter for the whole route**.
`OpenAITokenCounter.count_message` walks list content for `text` and
`image_url` only and has **no else branch**, so Anthropic content blocks
contributed literally zero. A 599-token `tool_result` scored 8. A
request that really removed 235 characters reported `tokens_saved: 0` —
so a caller gating on `tokens_saved > 0` concludes compression is broken
while it is working.
Prompted by a Kong integration question ("do you support the Anthropic
native format?"). The answer is that we already did — we just reported
zeros for it, and the docs said otherwise.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Documentation update
## Changes Made
### Tokenizer resolution (no hardcoded lists)
Build the derived pipelines with `provider=None` so `TransformPipeline`
resolves the tokenizer from the **per-model registry**. Every registry
tokenizer derives from `BaseTokenizer`, whose `_count_content_parts`
ends in a serialize-and-count catch-all, which means:
- No block type counts as zero, and there is **no per-provider
block-type list to keep in sync**. An enumerated set was the first thing
I tried and it already missed `mcp_tool_result`,
`web_search_tool_result`, `document`, and `thinking`.
- Gemini / Mistral / DeepSeek / Kimi stop defaulting to a tiktoken count
when the registry already has a calibrated counter for them.
- Gateway aliases matching no vendor pattern still count correctly.
`mode="ccr"` now runs a derived pipeline too, for the same reason —
sharing `openai_pipeline` pinned its provider. Costs that mode its own
cold compression cache; correct metrics win.
### Tokenizer selection stays separate from context-limit resolution
Deliberately not welded together. `model_limit` feeds `context_pressure
-> min_ratio`, so letting a tokenizer decision pick the limit table
changes compression aggressiveness: `gpt-4-32k` answered by the
Anthropic table is **8,192 instead of 32,768**, a 4× under-estimate.
`test_tokenizer_choice_does_not_move_the_context_limit` pins the
independence.
### Docs, rewritten from the code
- **`proxy.mdx`** — the loopback-only default and **404-not-403**
behavior, previously undocumented *anywhere* in `docs/` despite shipping
in #2458 explicitly for gateway sidecars;
`HEADROOM_COMPRESS_ALLOW_REMOTE`; all four request fields; the whole
`config` object including every `mode` value and `frozen_message_count`;
`transforms_summary`; the 400/401/404/503 contract; and the timeout
fail-open shape (`compression_skipped` / `skip_reason`).
- **Corrected "never calls an LLM"** — accurate about *generative*
provider requests, misleading for a sidecar operator. Kompress (a
ModernBERT **encoder**, classification not generation) and Magika run
**in-process**, and `HEADROOM_KOMPRESS_ENDPOINT` offloads inference over
HTTP — **real egress**. Now stated explicitly, with
`HEADROOM_DISABLE_KOMPRESS=1` as the structural-only option.
- **Both wire formats documented as accepted**, and removed
`anthropic-sdk.mdx`'s claim that OpenAI format is "the compression
engine's native format" — the exact misconception that prompted this
work. The SDK's conversion is now framed as an SDK choice, not an API
requirement.
- **`litellm.mdx`** had no mention of the endpoint at all, despite the
code naming LiteLLM's guardrail as its primary consumer. Added the HTTP
deployment path, the `HEADROOM_COMPRESS_ALLOW_REMOTE` requirement, and
why to leave `config.mode` unset.
- **`index.mdx`** printed `compressionRatio * 100` labelled "Saved …%",
so a 77% saving displayed as **23%**. `api-reference.mdx` already
defined it correctly, so the docs contradicted each other.
- `openai-sdk.mdx`, `wiki/proxy.md`, `wiki/typescript-sdk.md` — same
corrections; dropped "any HTTP client", "Cloud", and a CacheAligner
claim (it is detector-only).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ .venv/bin/ruff check headroom/ tests/ --exclude headroom/dashboard/templates
All checks passed!
$ .venv/bin/mypy headroom/
Success: no issues found in 511 source files
$ python -m pytest tests/test_compress_route_tokenizer_by_model.py \
tests/test_proxy_compress_endpoint.py tests/test_compress_api.py \
tests/test_platform_stabilization_functional.py tests/test_proxy_eager_preload_bind.py -q
99 passed, 2 warnings in 47.15s
```
Broader sweep (`-k "compress or litellm or gateway or guardrail"`):
**1625 passed, 4 failed** — all 4 pre-existing, verified by stashing
this diff and re-running on clean `main` (2 strands hook tests, 1 codex
WS semaphore-tail timing test, 1 unrelated local WIP test).
## Real Behavior Proof
- **Environment:** macOS 26.4 arm64, Python 3.12.6, repo `.venv`, branch
rebased on `upstream/main`.
**(1) Before → after, same request** (60-line grep payload in an
Anthropic `tool_result`):
| model | before | after |
| --- | --- | --- |
| `claude-sonnet-4-6` | `before=28 saved=0` | `before=1223 saved=58` |
| `bedrock/anthropic.claude-3-5-sonnet` | `saved=0` | `before=1037
saved=59` |
| `my-gateway/big-model` (alias) | `saved=0` | `before=1037 saved=59` |
| `gemini-2.5-pro` | `saved=0` | `before=1036 saved=59` |
| `gpt-4o` + OpenAI shape | `before=1225 saved=58` | `before=1225
saved=58` (unchanged) |
All three `config.mode` values verified for each. Response shape
preserved: `type=tool_result`, `tool_use_id` intact.
**(2) Counter-level root cause**, 6.8 KB body, `count_message()`:
```text
OpenAITokenCounter string-content -> 1406 tool_result block -> 5
registry (BaseTokenizer) claude tool_result=408 thinking=418 mcp_tool_result=421
web_search_tool_result=421 document=422
```
**(3) Every documented behavior asserted against the running app** — 13
checks, all PASS: 400s for missing `messages`/`model`, invalid
`config.mode`, and all four invalid `frozen_message_count` forms; 200
for valid ones; non-dict `config` ignored; bypass and empty-messages
omit `transforms_summary`; success returns exactly the 8 documented
keys.
- **Not tested:** the docs site was not built (`docs/node_modules`
absent) — MDX was checked for balanced `<Callout>` tags only, so a
reviewer with the site running should eyeball rendering. No live
gateway/Kong request; verification is via `TestClient` against the real
ASGI app.
- **Note:** `HEADROOM_DISABLE_KOMPRESS` is read into `ProxyConfig` at
`server.py:4919` and by the CLI, not by `create_app(ProxyConfig(...))`
directly — I confirmed `disable_kompress=True` does reach the derived
pipeline.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
Matt Van Horn
|
789a4f3060
|
fix: normalize /p/<project> prefix on WebSocket upgrades so the Responses WS route is not rejected with 403 (#2379)
## Description
A Responses WebSocket upgrade to a project-prefixed URL
(`ws://127.0.0.1:8787/p/<project>/v1/responses`) was rejected with `403
Forbidden`, so the client fell back to HTTP transport. The `/p/<name>`
base-URL prefix is stripped by
`strip_project_path_prefix(request.scope)` inside
`@app.middleware("http")`, but Starlette runs `@app.middleware("http")`
for `http` scopes only, never `websocket` scopes. So an HTTP `POST
/p/<project>/v1/responses` has its prefix stripped and matches
`/v1/responses`, while the WS upgrade keeps the prefix, matches no
registered WebSocket route (`OPENAI_RESPONSES_WEBSOCKET_PATHS` are all
unprefixed), and Starlette rejects the unmatched WebSocket with `403`.
This normalizes the prefix for WebSocket scopes before routing so the
upgrade reaches the existing Responses WS handler and stays attributed
to the project.
Closes #2355
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/server.py` — added a small pure-ASGI
`WebSocketProjectPrefixMiddleware` (registered in `create_app`) that,
for `websocket` scopes only, strips the `/p/<name>` prefix via the
existing `strip_project_path_prefix` and binds the project context,
mirroring the HTTP middleware. HTTP and lifespan scopes pass through
untouched (no double-strip).
- `headroom/proxy/handlers/openai.py` — `handle_openai_responses_ws`
previously called `set_current_project(classify_project(ws_headers))`
unconditionally, clearing the middleware-bound project for prefix-only
clients (no `X-Headroom-Project` header). It now falls back to the
already-bound path-prefix project (`classify_project(ws_headers) or
get_current_project()`), so prefix-only WebSocket clients (aider,
Copilot BYOK, Cursor and other `/p/<name>` base-URL wraps) stay
attributed, exactly as on the HTTP path.
- `tests/test_provider_proxy_routes.py` — added a regression test.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_provider_proxy_routes.py -q
21 passed, 1 warning in 23.95s
$ ruff check headroom/proxy/server.py headroom/proxy/handlers/openai.py
All checks passed!
$ mypy --python-version 3.13 headroom/proxy/server.py headroom/proxy/handlers/openai.py
Success: no issues found in 2 source files
```
## Real Behavior Proof
- Environment: local, `uv` venv, Python 3.14, `uv run pytest`.
- Exact command / steps: added
`test_project_prefixed_openai_response_websocket_delegates_to_openai_ws_handler`,
which connects a WebSocket to `/p/test-project/v1/responses`.
- Observed result: the connection is accepted (no 403), the handler is
reached with the canonical `/v1/responses` path, and the request is
attributed to project `test-project`.
- Not tested: live end-to-end against a real upstream Responses
WebSocket server (validated via the routing/attribution regression test
only).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A — backend routing change with no user-facing UI.
## Additional Notes
Documentation checklist item is N/A: this is an internal routing fix
with no configuration or public-API surface change. The fix mirrors the
existing HTTP prefix-strip behavior so project-prefixed WebSocket
clients behave identically to their HTTP counterparts.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
|
||
|
Tejas Chopra
|
f9db5b5060
|
fix(proxy/openai): run tool-description compaction on chat-completions (#2741)
## Description
`HEADROOM_TOOL_DESC_MAX_CHARS` was wired into the Anthropic handler and
the Responses (Codex) handler, but never into **chat-completions** — so
the env var was a silent no-op for every chat client: opencode, Cline,
Aider, Roo, anything routed through LiteLLM.
Tool descriptions live on the `tools` array, which the message pipeline
never inspects, so no other pass was covering them.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- Run the L2 tool-description pass on the chat-completions path,
mirroring the block the Anthropic and Responses handlers already had.
- `compact_tool_descriptions` already walks both wire shapes — nested
`{"function": {"description": ...}}` for chat, flat for Responses — so
this is wiring, not a new codec.
- Chains after the existing schema compaction, seeding the token
"before" count only when that pass didn't, so the two compose instead of
double-counting.
- Labelled `openai:chat:tool_desc_compaction`, distinct from the
Anthropic and Responses labels so `headroom perf --by-transform` can
attribute it.
- Still opt-in and off by default: an unset env var leaves the tools
array — and therefore its cache prefix — byte-identical.
### Scope note: two adjacent "gaps" that turned out not to be
While surveying handler parity I flagged three missing chat-completions
transforms. Only one was real; recording the other two so nobody
re-opens them:
- **`tool_search_deferral` — correctly absent.** `{"type":
"tool_search"}` and `defer_loading` are Responses-API constructs, and
`_model_supports_openai_tool_search` gates them to `gpt-5.4+`. Injecting
that shape into a chat-completions request would be invalid, not an
improvement.
- **`system_prompt_compaction` — not applicable.** Anthropic needs a
dedicated pass because `system` is an out-of-band top-level field the
message pipeline never sees. On chat-completions the system prompt *is*
`messages[0]`, so it already reaches ContentRouter and is governed by
the existing `compress_system_messages` / `skip_system` gate. Wiring a
second path there would change system-prefix cache behavior for no new
coverage.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ .venv/bin/ruff check headroom/ tests/test_openai_chat_tool_desc_compaction.py --exclude headroom/dashboard/templates
All checks passed!
$ .venv/bin/mypy headroom/
Success: no issues found in 508 source files
$ python -m pytest tests/test_openai_chat_tool_desc_compaction.py tests/test_tool_schema_compaction.py \
tests/test_proxy_openai_cache_stability.py tests/test_openai_responses_context_compaction.py -q
49 passed in 16.59s
```
## Real Behavior Proof
- **Environment:** macOS 26.4 arm64, Python 3.12.6, repo `.venv`.
- **Exact command / steps:** ran `compact_tool_descriptions` at
`HEADROOM_TOOL_DESC_MAX_CHARS=30` against both wire shapes with the same
tool (a `read` tool with an 86-char description and a described `path`
param).
- **Observed result:**
```text
chat-completions (nested) modified=True bytes 272->215
responses (flat) modified=True bytes 259->202
```
Chat previously reported `modified=False` from the handler because the
pass was never invoked at all.
- **Not tested:** no live chat-completions request against a real
provider — the handler block is a thin adapter over
`compact_tool_descriptions`, and the regression was a missing *call*,
which the wiring test catches at source level. A full end-to-end drive
would need an upstream endpoint.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
Tejas Chopra
|
224578e80b
|
fix(kompress): reject artifacts that fail at run, and prefetch model files at startup (#2740)
## Description Three cold-start / robustness gaps found while debugging a user report of **0.12% savings across 722 requests** (49.8M input tokens, 60,920 saved). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made ### 1. The artifact fallback was unreachable for run-time failures `_create_onnx_session` tries `int8-wo` → `fp32` → `int8`, and its docstring describes exactly this scenario — but it only skipped a candidate when `InferenceSession(...)` **construction** threw. The int8 weight-only artifact carries `MatMulNBits` with `bits=8`. ORT's CPU kernel only handles 8-bit through the prepacked MLAS path, so a build or ISA without an 8-bit `SQNBitGemm` kernel falls into `ComputeBUnpacked`, which hard-asserts `nbits_ == 4`. That raises on `session.run()` **after** construction succeeded — so the fp32 candidate was never reached and ML compression was dead for the process lifetime. The reported log has 207 consecutive failures over three days. A two-token `_smoke_run` inside the existing candidate loop makes the fallback fire. `onnxruntime>=1.16.0` is unpinned, so which side of this an install lands on is a lottery. ### 2. A broken model cost an inference on every request, forever The per-request handler logged a `WARNING` and passed through with no latch — 207 identical lines that read as noise rather than "ML compression is dead". Now latches to passthrough after **3 consecutive** failures (any success resets the count) with one actionable `ERROR` naming the artifact override. ### 3. The model download began on the first request, not at startup #2001 was right to move Kompress off the startup path — on RHEL/CentOS 7-family hosts, entering cached native init before the port binds segfaults in `libarrow`/jemalloc with no Python traceback (#1908), which no `try/except` can catch. **This PR does not touch that.** But #2001 left the ~4-minute *download* on the first request, with every request in that window silently uncompressed behind one "model not ready" warning. Downloading is separable from loading. `prefetch_kompress_artifacts` resolves the files over plain `huggingface_hub` HTTP and never constructs an `InferenceSession` or imports `transformers`, so startup can prefetch bytes without touching the boundary #1908 crashes on. Native load stays deferred, status stays `deferred`, and a test asserts no session is constructed during prefetch. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ .venv/bin/ruff check headroom/ tests/test_kompress_failsafe.py tests/test_kompress_preload_deferral.py --exclude headroom/dashboard/templates All checks passed! $ .venv/bin/mypy headroom/ Success: no issues found in 508 source files $ python -m pytest tests/test_kompress_failsafe.py tests/test_kompress_preload_deferral.py \ tests/test_kompress_request_nonblocking.py tests/test_force_kompress_all.py \ tests/test_kompress_must_keep.py tests/test_proxy_disable_kompress.py \ tests/test_proxy_per_provider_kompress.py tests/test_proxy_warmup.py \ tests/test_proxy_eager_preload_bind.py -q 95 passed in 10.51s ``` ## Real Behavior Proof - **Environment:** macOS 26.4 arm64, Python 3.12.6, onnxruntime 1.21.1, repo `.venv`. **(1) Fallback chain, against the real HF repo:** ```text WARNING ONNX artifact 'onnx/kompress-int8-wo.onnx' from chopratejas/kompress-v2-base is unusable (... nbits_ == 4 was false ...); trying next candidate SESSION OK -> ['input_ids', 'attention_mask'] SMOKE RUN OK on the selected artifact ``` Also confirmed the default artifact really is 8-bit, by loading the cached blob: `{'bits': [8], 'block_size': [128]}`. **(2) Files-only prefetch, with `InferenceSession` patched to raise:** ```text INFO Kompress: prefetching model artifacts for chopratejas/kompress-v2-base ... prefetch ok=True in 0.08s, no session constructed ``` - **Not tested / important caveat:** the user's exact failure **cannot be reproduced on this machine**. On ORT 1.21.1 arm64 the int8-wo artifact fails at *construction* (`matmul_nbits.cc:115`), which the pre-existing load-only fallback already caught. Their build fails at *execution* (`matmul_nbits.cc:442`, `ComputeBUnpacked`). So the run-time path is pinned with a fake ORT session that constructs fine and then rejects `run()` — a mechanism test, not a reproduction of their build. Confirming the fix on their host needs their `onnxruntime` version. - **Not tested:** no RHEL/CentOS 7 host available to re-verify #1908 non-regression; the argument is structural (prefetch never constructs a session) and asserted by `test_prefetch_never_constructs_a_session_or_imports_transformers`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Tejas Chopra
|
8262a4a321
|
fix(stats): report one "Tokens Saved" headline across every harness (#2737)
## Description
The "tokens saved" figure a user sees depended on which harness they
ran. Headroom saves tool-definition tokens in two accounting shapes,
both legitimate, but the rule was never written down — so two harnesses
silently dropped savings and three surfaces open-coded the sum
differently.
- **Compaction** rewrites the tool array, so both endpoints are
countable → handlers fold the delta into
`original_tokens`/`optimized_tokens`, keeping `tok_before - tok_after ==
tok_saved` coherent.
- **Deferral / hook shrink** removes schemas `count_messages` never sees
→ can only be recorded as a tag, additive to `tokens_saved`.
`tool_schema_savings_policy` now owns the sum via
`headline_tokens_saved()`, and every reporting surface routes through
it.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
Producer gaps (both Anthropic — i.e. Claude Code, the primary harness):
- `anthropic:tool_schema_compaction` / `anthropic:tool_desc_compaction`
computed their savings, debug-logged them, and **discarded them**. Now
folded at the final recount, mirroring the OpenAI chat handler. A
14-tool array drops 786 tokens that previously reported `tok_saved=0`.
- Anthropic never wrote `turn_hook_tools_saved_tokens` at all, so a
turn-hook extension that shrinks tools got zero credit there while
OpenAI credited it. Now tagged.
Reporting gaps:
- `headroom perf` printed `Total saved (messages)` and `Tool saved` as
rival lines — on a tool-heavy session the headline read `0` and the real
win looked like a footnote. Now one `Tokens saved:` headline with a
messages/tool-schemas breakdown.
- `active_savings_percent` divided a **compression-only numerator** by a
denominator that already included compacted tool schema, undercounting
every tool-heavy session. Numerator is now all-layers, with deferred
schemas added to both sides.
- The headline and its percent now share a numerator. Previously the
dashboard tile showed an all-layers total next to a compression-only
percent.
- Session summary and dashboard tile relabelled to `Tokens Saved`; the
tool-schema panel is labelled as a component (`Tokens Saved · Tool
Schemas`) rather than a rival metric.
- `outcome.py` had two drifted inline copies of the tag sum; both now
call the policy module that exists for it. `total_saved=` added to the
PERF line.
- JSON: added `total_tokens_saved` / `total_savings_pct`; existing
`tokens_saved` / `tool_saved` / `savings_pct` keys unchanged for
back-compat.
Not changed by design: the Codex per-component attribution sub-line
would need a 9th positional tuple element threaded through 4 unpack
sites, and its headline is already correct without it.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ .venv/bin/ruff check headroom/ tests/test_tool_schema_savings_policy.py --exclude headroom/dashboard/templates
All checks passed!
$ .venv/bin/ruff format --check headroom/ tests/... --exclude headroom/dashboard/templates
510 files already formatted
$ .venv/bin/mypy headroom/
Success: no issues found in 508 source files
$ python -m pytest tests/test_tool_schema_savings_policy.py tests/test_cli_perf_format.py \
tests/test_request_outcome.py tests/test_savings_tool_search_aggregation.py \
tests/test_dashboard_token_savings.py tests/test_anthropic_compaction_transforms.py -q
70 passed, 1 warning in 6.15s
$ python -m pytest tests/test_handler_outcome_tag_invariant.py tests/test_cold_start_fast_pass.py \
tests/test_anthropic_ccr_workspace_unbound.py tests/test_anthropic_pre_upstream_backpressure.py \
tests/test_vertex_claude_compression.py tests/test_provider_route_specs.py -q
50 passed in 10.11s
$ python -m pytest tests/test_agent_savings.py tests/test_bundled_tools_savings.py \
tests/test_codex_ws_savings_deferral.py tests/test_savings_ledger_before_forwarded.py \
tests/test_savings_ledger_offload.py tests/test_proxy_savings_history.py \
tests/test_proxy_dashboard_stats_cache.py tests/test_output_savings_cli.py -q
97 passed, 2 skipped in 18.60s
$ python -m pytest tests/test_tool_schema_compaction.py tests/test_openai_responses_context_compaction.py \
tests/test_proxy_openai_cache_stability.py tests/test_codex_ws_compression_scheduler.py \
tests/test_proxy_streaming_request_logger.py -q
66 passed, 1 skipped in 16.69s
```
## Real Behavior Proof
- **Environment:** macOS 26.4 arm64, Python 3.12.6, repo `.venv`.
Motivated by a real user proxy log (0.33.0, `client=opencode` →
nano-gpt, 722 requests) reporting 0.12% savings.
- **Exact command / steps (1) — the Anthropic fold, real compaction +
real provider tokenizer:**
```python
tok = AnthropicProvider().get_token_counter("claude-sonnet-4-6")
payload = {"tools": [ ...14 tools with $schema/title/examples... ]}
body, modified, bb, ba = compact_tools(payload)
```
**Observed:**
```text
modified=True bytes 4503->2539 TOKENS 1650->864 delta=786
tok_before=6650 tok_after=5864 tok_saved=786 coherent=True
pre-fix: Claude Code reported tok_saved=0 and discarded 786 tokens
```
Pinned as `test_tool_schema_compaction_saves_real_tokens_not_just_bytes`
— it asserts a positive **token** delta (not just bytes), which is the
premise of folding at all.
- **Exact command / steps (2) — the report, on the reported session's
shape** (tool schemas carry the win, message compression is 0 because
everything routed to `excluded_tool`):
**Observed after:**
```text
Requests: 2
Tokens: 45,760 -> 45,760 (0.0% messages)
Tokens saved: 811 (1.7% reduction)
· messages 0
· tool schemas 811
JSON: {'total_tokens_saved': 811, 'total_savings_pct': 1.7, 'tokens_saved': 0,
'tool_saved': 811, 'savings_pct': 0.0}
```
Before, the same input printed `Total saved: 0 tokens (messages)` as the
headline with `Tool saved: 811` beneath it.
- **Not tested:** no live proxy run against a real provider — the
Anthropic fold is proven at the accounting layer (real `compact_tools` +
real provider tokenizer) and via the existing handler suites, not by an
end-to-end Claude Code session. Dashboard changes are template-label
edits verified by reading `stats.tokens.saved` / `by_layer.tool_search`
shapes, not by a browser screenshot.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
Tejas Chopra
|
3d23d76248
|
fix(kompress): let orgs run Kompress on their own inference stack (#2736)
## What this enables An org pulls the Kompress weights from HuggingFace, serves them on their own infrastructure, and points Headroom at it: ```bash HEADROOM_KOMPRESS_ENDPOINT=https://ml.internal.acme.com ``` No credential needed, no local ML dependencies, and original content never leaves their network (the CCR store stays proxy-local, so `headroom_retrieve` keeps working). ## The one thing that was actually broken Almost all of this already worked. The blocker was a hardcoded path: ```python self._url = endpoint.rstrip("/") + "/compress" ``` Real inference servers don't serve at `/compress`: | Stack | Path | |---|---| | TorchServe | `/predictions/kompress` | | KServe / Seldon | `/v1/models/kompress:predict` | | SageMaker | `/invocations` | Appending `/compress` to those 404s. And because remote Kompress **fails open**, that 404 is invisible — compression silently stops instead of erroring. The only workaround was standing up a reverse proxy purely to rename a path. ## Two new env vars, both defaulting to current behaviour | Var | Default | Purpose | |---|---|---| | `HEADROOM_KOMPRESS_ENDPOINT_PATH` | `/compress` | Set empty to use the endpoint URL verbatim | | `HEADROOM_KOMPRESS_ENDPOINT_HEADERS` | *(none)* | `k=v,k2=v2`, merged last so it can replace `Authorization` | Headers are applied after the token deliberately, so a gateway wanting `x-api-key` or `X-Tenant-Id` needs no separate auth-scheme setting. ## No regression With only `HEADROOM_KOMPRESS_ENDPOINT` set, the request is **byte-identical** to before — `POST <endpoint>/compress` with an optional Bearer token. Existing Modal deployments need no change. `os.environ.get` with a default distinguishes "unset" (use `/compress`) from an explicit empty value (endpoint is a complete URL), so the escape hatch can't fire by accident. The regression cases are deliberately the *first* tests in the new file. Verified through the real router wiring: ``` modal (today's config) -> https://acme--kompress.modal.run/compress modal + token -> …/compress {'authorization': 'Bearer tok'} self-hosted KServe (full URL) -> https://ml.acme.com/v1/models/kompress:predict self-hosted TorchServe (path) -> https://ts.acme.com/predictions/kompress self-hosted, x-api-key, no token -> …/compress {'x-api-key': 'k', 'x-tenant-id': 'acme'} ``` ## Documents the HTTP contract The endpoint contract was only discoverable by reading the source. Now in the module docstring: ``` request {"content": "<text>", "target_ratio": 0.5 | null} response {"compressed": "<text>", # REQUIRED "original_tokens": int, # optional, derived if absent "compressed_tokens": int, # optional "compression_ratio": float, # optional "model_used": str} # optional ``` `compressed` is the only required field, so a shim in front of an existing inference server is a few lines. Also logs the **resolved** URL at startup — with fail-open, a mistyped path otherwise manifests as nothing happening at all. ## Notes - `parse_endpoint_headers` reimplements the `HEADROOM_OTEL_METRICS_HEADERS` format rather than importing it: `observability.metrics` imports opentelemetry at module scope, and remote Kompress exists precisely so a proxy can run without heavy optional deps. - 27 new tests. Pre-existing unrelated flake in `test_content_router_single_item_deadline.py` (fails 3/3 on clean main). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Tejas Chopra
|
7c9b046595
|
fix(dashboard): serve tailwind/htmx/alpine locally instead of from CDNs (#2734)
## Description The dashboard loaded all three of its front-end dependencies from third-party CDNs at page load: ```html <script src="https://cdn.tailwindcss.com"></script> <script src="https://unpkg.com/htmx.org@1.9.10"></script> <script src="https://unpkg.com/alpinejs@3.13.3/dist/cdn.min.js" defer></script> ``` Microsoft Edge's Tracking Prevention classifies `unpkg.com` as a tracker and blocks it by default on Windows; locked-down corporate proxies block both hosts. On those machines none of the three scripts executed — no Tailwind CSS, no htmx polling, no Alpine bindings, plus an uncaught `ReferenceError: tailwind is not defined` from the inline `tailwind.config` assignment at `dashboard.html:21`. The dashboard rendered blank. Reported from a Windows user's console: ```text Tracking Prevention blocked access to storage for https://unpkg.com/htmx.org@1.9.10. Tracking Prevention blocked access to storage for https://unpkg.com/alpinejs@3.13.3/dist/cdn.min.js. ``` This vendors the three files and serves them from the proxy, so the dashboard has no external network dependency at all. Note for anyone triaging the same report: the `cdn.tailwindcss.com should not be used in production` line in that console output is **not** related. It is an unconditional `console.warn` in the Tailwind Play CDN build (no hostname guard), so it fires on every load, localhost included, and it still fires now that the bundle is self-hosted. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Vendored `headroom/dashboard/static/{tailwind.min.js,htmx.min.js,alpine.min.js}` — Tailwind Play CDN 3.4.17, htmx 1.9.10, Alpine 3.13.3, byte-for-byte as published. - `headroom/dashboard/__init__.py`: added `STATIC_DIR`. - `headroom/proxy/server.py`: mounted `/dashboard/static`, registered **before** `register_provider_routes`' catch-all so the asset requests are not tunneled to the wrapped upstream provider (same ordering constraint as the `/favicon.ico` route, GH #1787). `check_dir=False` so a missing assets directory 404s the dashboard JS rather than aborting proxy startup. - `headroom/dashboard/templates/{dashboard,settings}.html`: script `src` → `/dashboard/static/…`. - `NOTICE`: MIT / 0BSD attribution for the three vendored bundles. - `tests/test_dashboard_static_assets.py`: new. No packaging change needed — `[tool.maturin]` includes everything under `headroom/`, so the wheel picks the assets up. Wheel grows ~498 KB (407 KB of that is the Tailwind Play bundle). ## Testing - [x] Unit tests pass (`pytest`) — targeted, see note under *Not tested* - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_dashboard_static_assets.py tests/test_proxy_settings_endpoints.py -q tests/test_dashboard_static_assets.py ...... [ 21%] tests/test_proxy_settings_endpoints.py ...................... [100%] ============================== 28 passed in 4.47s ============================== $ ruff check . All checks passed! $ ruff format --check headroom/proxy/server.py headroom/dashboard/__init__.py tests/test_dashboard_static_assets.py 3 files already formatted $ mypy headroom Success: no issues found in 509 source files ``` ## Real Behavior Proof - **Environment:** macOS 15 (Darwin 25.4.0), Python 3.12.6, headless Chromium via Playwright, proxy served in-process with `create_app(ProxyConfig(optimize=False, cache_enabled=False, log_full_messages=True))` on `:8787`. - **Exact command / steps:** loaded `/dashboard` and `/dashboard/settings` with `wait_until="networkidle"`, then asserted the globals exist, that Tailwind actually generated CSS (computed style of a `px-3` element), and recorded every non-localhost request plus all `pageerror`/`console.error` events. - **Observed result:** ```text /dashboard | alpine: True | tailwind css: True | external: none | errors: none /dashboard/settings | alpine: True | tailwind css: True | external: none | errors: none /dashboard 200 text/html; charset=utf-8 191549 /dashboard/static/tailwind.min.js 200 text/javascript; charset=utf-8 407279 /dashboard/static/htmx.min.js 200 text/javascript; charset=utf-8 47755 /dashboard/static/alpine.min.js 200 text/javascript; charset=utf-8 43441 feed-toggle visible: True alpine loaded: True htmx: True tailwind: True tailwind applied (px-3 padding): 12px external hosts: none console errors: none ``` Zero external requests on either page, so the Edge/firewall failure mode is structurally gone rather than worked around. - **Not tested:** - No Windows machine available — the fix is verified as "makes zero external requests", which is the property the Windows failure depended on, but it has not been confirmed against Edge with Tracking Prevention on. Worth a check by someone on Windows before release. - Full `pytest` suite not run (targeted runs only); CI covers it. - `tests/test_dashboard/test_live_feed.py` still has 2 failures, both pre-existing and unrelated: those tests need a manually started proxy on `:8787` with `--log-messages`, and `test_live_feed_button_exists` asserts `is_visible()` with no wait for the `/stats` poll that flips `log_full_messages`. The other 2 in that file pass against this change, which is itself end-to-end evidence that Alpine and htmx work from the vendored bundles. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes - No issue number: reported directly rather than filed, so `Closes #` is omitted. Closed #22 ("Dashboard is not working") and closed #533 (Windows cp949 `get_dashboard_html()`) are different failures. - **Docs checklist item is N/A** — nothing user-facing changes; the dashboard URL and behaviour are identical. - Deliberately **not** switching to a real Tailwind CLI build. It would cut 407 KB to ~20 KB and silence the production warning, but it puts Node in the release path and silently leaves any class added to the 2,713-line template unstyled with no CI guard. The Play bundle behaves exactly as it does today, just served locally. Worth revisiting if wheel size becomes a problem (note the PyPI project-size ceiling). - Upgrades are now manual: bumping these three means re-downloading the files. Pinned versions are recorded in `NOTICE`. |
||
|
gglucass
|
a70e5ff78d
|
fix(learn): run project discovery off the event loop (#2731)
## Description
`TrafficLearner.flush_to_file` is a coroutine, but it called
`plugin.discover_projects()` inline. That function walks the filesystem
to decode escaped project directory names — in
`learn/plugins/claude.py`, `_greedy_path_decode` recurses through
`iterdir()` at every level and tries each tokenization of each child,
backtracking on a miss — so on a large home tree it runs for minutes.
Doing that on the event loop freezes uvicorn for the whole window. The
port keeps accepting TCP, but `/readyz` never answers, so a supervisor
health-checking the proxy kills a process that is merely busy.
Field thread dumps show exactly that:
```
Current thread (most recent call first):
File "python3.12/pathlib.py", line 1056 in iterdir
File "headroom/learn/plugins/claude.py", line 454 in _greedy_path_decode
File "headroom/learn/plugins/claude.py", line 478 in _greedy_path_decode
File "headroom/learn/plugins/claude.py", line 478 in _greedy_path_decode
File "headroom/learn/plugins/claude.py", line 426 in _decode_project_path
File "headroom/learn/plugins/claude.py", line 71 in discover_projects
File "headroom/memory/traffic_learner.py", line 591 in flush_to_file
File "headroom/memory/traffic_learner.py", line 535 in _flush_worker
File "python3.12/asyncio/events.py", line 88 in _run
File "python3.12/asyncio/base_events.py", line 1999 in _run_once
File "python3.12/asyncio/base_events.py", line 645 in run_forever
File "uvicorn/server.py", line 75 in run
File "headroom/proxy/server.py", line 4992 in run_server
```
Accompanying signals from the same incidents: port accepts TCP,
`/readyz` times out, process CPU 2-13s across the window (I/O bound, not
spinning), proxy log silent 66-336s.
## Type of Change
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/memory/traffic_learner.py`: `flush_to_file` now awaits
`asyncio.to_thread(plugin.discover_projects)` instead of calling it
inline. `asyncio` was already imported. The result is cached per learner
(`_project_roots_cache`), so the steady-state flush path pays nothing
for the thread hop.
- `tests/test_memory/test_traffic_learner.py`: added
`test_discover_projects_does_not_block_the_event_loop`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uv run --frozen --extra dev pytest tests/test_memory/test_traffic_learner.py -q
.................................................. [100%]
============================= 152 passed in 2.93s ==============================
$ uvx ruff check headroom/memory/traffic_learner.py tests/test_memory/test_traffic_learner.py
All checks passed!
$ uvx ruff format --check headroom/memory/traffic_learner.py tests/test_memory/test_traffic_learner.py
2 files already formatted
$ uv run --frozen --extra dev mypy headroom/memory/traffic_learner.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: macOS 15.6 arm64, Python 3.10.18, pytest 9.0.3, branch
off `main` @ `
|
||
|
Tejas Chopra
|
9cfb00838a
|
fix(telemetry): anonymous compression stats — no prompts, no data (#2728)
## In one line
Headroom starts reporting **how well compression is working** — counters
and percentages only. **No prompts. No code. No file paths. Nothing
about what you're building.**
## Why
Right now nobody knows whether compression actually helps real users.
You can see your own numbers in `/stats`, but that's it — there's no way
to tell whether a given workload compresses well, or why it sometimes
doesn't. This closes that loop so we can make compression better for
everyone.
## Exactly what gets sent
One message per session, and every 5 minutes while you're active:
```json
{
"session": { "id": "random", "turns": 47, "duration_s": 4210, "seq": 3 },
"tokens": { "original": 890000, "attempted": 410000, "saved": 320000,
"tool_saved": 48000, "cache_read": 210000 },
"rates": { "saved_pct": 35.96, "eligible_pct": 46.07, "yield_pct": 78.05,
"cache_read_pct": 23.60, "overhead_pct": 1.96 },
"compression": { "transforms": {"crush": 47}, "passthrough_turns": 0 },
"skips": {},
"sources": { "proxy": 47 },
"providers": ["anthropic"],
"models": ["claude-sonnet-4-5-20250929"],
"failures": 2
}
```
Plus a random install ID, the Headroom version, and OS/architecture
(`darwin`, `arm64`).
That's the whole thing. A full example lives at
`deploy/beacon/sample-event.json`.
## What is never sent
- Your prompts or the model's responses
- Your code
- File paths, project names, repo names
- Tool names or MCP server names
- Hostname, username, or IP address
- Custom or fine-tuned model names (an id like `ft:gpt-4o:acme-corp:…`
contains a company name, so only models in a public registry are
reported)
**This is structural, not a pinky-swear.** Every value in the payload is
a number, a fixed word, or a random ID — there is no free-text field
anywhere for content to hide in. The receiver
(`deploy/beacon/worker.js`, in this repo so you can read it) drops
anything not on an explicit allowlist before storing.
## Turning it off
Any one of these:
```bash
HEADROOM_BEACON=off # or
DO_NOT_TRACK=1 # or
# offline mode
```
It's on by default, and Headroom says so at startup:
```
Telemetry: anonymous compression stats — never prompts, code, or file paths.
Helps us improve compression | Off: HEADROOM_BEACON=off
```
`HEADROOM_TELEMETRY` is a **separate** switch that still only affects
local stats. If you had turned that on, this change does not start
uploading anything — you answered a different question, and upgrading
should not change the answer.
## Why the percentages, not just "tokens saved"
"We saved 36%" hides the interesting part. In the example above only
**46% of tokens were eligible** for compression at all — the rest is
frozen cache prefix and system prompts we deliberately do not touch. Of
what we *could* touch, we removed **78%**.
Those are two separate problems. Raising eligibility is proxy work;
raising yield is compressor work. A single number cannot tell us which
to fix.
## Coverage
`emit_request_outcome` is a single chokepoint —
`handler.metrics.record_request` is called from exactly one place,
inside the funnel — so all 30 `RequestOutcome` construction sites are
covered: Anthropic, OpenAI, Gemini, Bedrock, batch, streaming, and the
long-lived Codex Responses-WS path.
The `headroom_compress` MCP path bypassed that funnel and is now wired
in separately. It has a different shape (no provider, no upstream
latency, and everything handed to the tool is eligible by construction),
so `sources` counts turns by origin — MCP turns always read
`eligible_pct: 100` and must not drag the proxy's real eligibility
ceiling upward.
**Subagents.** All subagent traffic through the proxy merges into one
session, which is correct for savings and retention but means `turns`
conflates fan-out with depth. Fan-out is still derivable —
`compression.latency_ms_total / session.duration_s` gives the
concurrency ratio (~1x serial, ~4x for four parallel agents), so no
extra field is needed. Verified no lost updates under 6-way concurrency
(1,200 turns).
**Known gap:** `--workers N` gives each process its own aggregator, so
one user session becomes up to N. Token totals and fleet rates stay
correct; session counts inflate. This matches the existing documented
limitation that TOIN state, CostTracker, and the prefix tracker are all
per-process.
## Notes for reviewers
- **Cumulative snapshots, not deltas.** Every report restates running
totals under one session ID, so the highest `seq` per `(install,
session)` is the complete session. Dedupe is a window function, and a
lost report costs nothing.
- **Never breaks the proxy.** Every path swallows its own exceptions;
uploads go out on a daemon thread so nothing blocks the request loop.
- **Explicit User-Agent is load-bearing.** urllib's default is blocked
by Cloudflare (error 1010). Combined with fire-and-forget error
handling, that would have failed every upload while looking perfectly
healthy.
- **The exit flush was broken and is fixed.** `atexit` handed the POST
to a daemon thread, and daemon threads are killed before they finish
during interpreter shutdown — so nothing was sent. That silently dropped
*every session shorter than the 5-minute heartbeat*, plus all
short-lived subagent MCP processes. The exit path now posts
synchronously with a 2s timeout.
- Receiver and query tooling are in `deploy/beacon/`.
## Testing
- `python -m headroom.telemetry.session` self-check: dedupe, cumulative
totals, dropped-report recovery, payload contains no model id or
prompt-derived string, allowlist coverage
- 175 telemetry/outcome tests pass; 6 new ones cover the opt-out notice
- Verified end to end against a live deployment: client → receiver →
storage → query
## Still to do before release
The default endpoint currently points at a temporary `workers.dev` URL.
It needs to move to a Headroom-owned hostname before this ships in a
tagged release — noted inline at `DEFAULT_ENDPOINT`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
JD Davis
|
007446c73a
|
feat(copilot): proxy VS Code models transparently (#2687)
## Description Make Headroom a transparent proxy for VS Code GitHub Copilot. Users keep using Copilot's normal model picker—GPT-4.1, Claude Sonnet, Claude Opus, and other models in their entitlement—while Headroom silently forwards the selected model instead of registering or requiring a separate "Headroom" model. This also fixes GitHub's device OAuth exchange by sending form-encoded request bodies, matching the endpoint contract. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add `headroom wrap vscode` to start a Copilot-seeded subscription proxy and safely configure VS Code's shipped Copilot proxy override. - Add `headroom unwrap vscode` for reversible cleanup. - Preserve VS Code's selected model by changing only the proxy URL/auth override; no custom model is registered and no model preference is written. - Support stable VS Code settings locations on macOS, Windows, and Linux, plus `--settings-file` for Insiders, portable, and other installations. - Edit JSONC settings with a marker-owned block while preserving unrelated bytes, comments, ordering, and trailing commas. - Refuse malformed markers, invalid JSONC, or unmanaged existing Copilot overrides instead of overwriting user configuration. - Fix SIGINT cleanup so the managed settings block is removed and normal shutdown exits successfully. - Fix Copilot device OAuth start/poll requests to use `application/x-www-form-urlencoded`. - Add a compatibility matrix, setup/removal flow, credential behavior, remote-development guidance, enterprise notes, troubleshooting, and verification documentation. ## Testing - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ .venv/bin/pytest -q tests/test_provider_copilot_vscode_config.py tests/test_cli/test_wrap_vscode.py tests/test_cli/test_wrap_helpers.py tests/test_copilot_auth.py tests/test_cli/test_copilot_auth.py tests/test_provider_copilot_wrap.py tests/test_cli/test_wrap_copilot.py tests/test_copilot_provider_label.py tests/test_copilot_subscription_smoke.py 244 passed in 0.59s $ .venv/bin/ruff check <changed Python files and tests> All checks passed! $ .venv/bin/mypy headroom/providers/copilot/vscode.py Success: no issues found in 1 source file $ cd docs && npm run types:check fumadocs-mdx && next typegen && tsc --noEmit # exited 0 $ git diff --check # exited 0 ``` The full 10,179-test suite was also sampled through approximately 83%, but was stopped because of its runtime. It exposed existing failures in `test_recover_codex.py`, `test_wrap_stale_marker.py`, and `test_proxy_health.py`; therefore the broad `pytest`, repository-wide Ruff, and repository-wide mypy boxes are intentionally not checked. ## Real Behavior Proof - Environment: macOS arm64, VS Code 1.131.0, built-in GitHub Copilot 0.59.0, Headroom 0.33.1-dev. - Exact command / steps: 1. Completed `headroom copilot login` with GitHub's device flow. 2. Ran `.venv/bin/headroom wrap vscode --port 8788`. 3. Confirmed VS Code retained its ordinary Copilot model catalog and made `GET /models` through Headroom with `GitHubCopilotChat/0.59.0` and `editor-version: vscode/1.131.0`. 4. Sent native Copilot `/p/headroom/chat/completions` requests through the same endpoint using `gpt-4.1`, `claude-sonnet-4.6`, and `claude-opus-4.7`. - Observed result: - All three completion requests returned HTTP 200. - GPT-4.1 resolved upstream to `gpt-4.1-2025-04-14`; Sonnet and Opus retained their exact selected IDs. - All returned the requested exact marker content. - VS Code's settings contained only the Headroom proxy URL and token auth override—no Headroom model or model-selection setting. - The proxy health endpoint remained ready with `openai_api_url` set to `https://api.githubcopilot.com`. - Not tested: - Physical Windows or Linux hosts (their path/config behavior is covered by unit tests). - WSL, dev containers, SSH remotes, VS Code Insiders, or enterprise Copilot deployments end-to-end. - Every model in the live Copilot catalog. - A fully submitted chat from VS Code's UI automation; the real extension's catalog request and native completion paths were verified separately. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing targeted unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable; this integration intentionally has no separate UI or model entry. ## Additional Notes The integration uses VS Code Copilot's shipped advanced/debug proxy endpoint seam. The managed settings block is deliberately narrow and reversible. Remote extension hosts may need their own reachable proxy/configuration as documented. --------- Co-authored-by: JerrettDavis <2610199+JerrettDavis@users.noreply.github.com> |
||
|
AxelRay
|
56b3e4c1b1
|
fix(proxy): skip OpenAI tool_search deferral for Codex client (#2729)
## Description
OpenAI Responses tool_search deferral injects `defer_loading` and a
`tool_search` tool for eligible gpt-5.4+ requests. When the model later
calls a deferred tool, the function_call item carries a `namespace`
field. Codex CLI round-trip structs drop unknown fields, so the next
request omits `namespace` and OpenAI returns 400, killing the session
mid-run. Proxy logs for these Codex turns show no tool savings, so the
injection breaks Codex without benefit.
This skips OpenAI tool_search deferral when the classified client is
Codex, leaving other clients unchanged.
Closes #2726
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Add `openai_tool_search_client_supported` and a Codex-only unsupported
client set
- Pass optional `client` into `inject_tool_search_deferral_openai` and
no-op for Codex
- Plumb `client` through Responses compression (HTTP, WebSocket,
passthrough) with legacy-signature retries
- Add regression tests for Codex skip and non-Codex still injects
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ .venv/bin/python -m pytest tests/test_openai_tool_search_deferral.py -q -o addopts=
34 passed, 1 warning in 0.27s
$ .venv/bin/python -m ruff check headroom/proxy/helpers.py headroom/proxy/handlers/openai.py tests/test_openai_tool_search_deferral.py
All checks passed!
$ .venv/bin/python -m ruff format --check headroom/proxy/helpers.py headroom/proxy/handlers/openai.py tests/test_openai_tool_search_deferral.py
3 files already formatted
```
## Real Behavior Proof
- Environment: Linux x86_64, Python 3.14.5 in repo .venv, shallow
checkout of headroomlabs-ai/headroom main at
|
||
|
Parideboy
|
01df245252
|
fix(proxy/cost): mark estimated-basis budget records and add an enforcement policy (#2713) (#2725)
## Description
`CostTracker.check_budget()` is a hard spend control — the Anthropic
handler refuses the request with a 429 once the period budget is gone.
The ledger that control reads could not tell a measured dollar from a
guessed one.
When a provider response carries no input-token breakdown,
`record_tokens()` substitutes Headroom's own `tokens_sent` estimate for
the input count so input cost isn't silently dropped from the budget.
That fallback is the right call, but the resulting record was
byte-identical to a provider-measured one: no field, no log line, no
separation in `/stats`. `RequestOutcome.uncached_input_tokens` defaults
to `0`, so any route whose response omits usage lands on this branch in
production. An estimate can drift in either direction, so a budget check
could pass after real spend had already gone over — with nothing saying
the decision rested on an estimate.
This keeps the fallback and makes it visible, then lets operators decide
what an estimate is allowed to do to a hard limit.
Closes #2713
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- New `headroom/proxy/budget_basis_policy.py` (pure policy module,
matching the existing `*_policy.py` convention): the
`measured`/`estimated` basis constants, the `count`/`ignore`/`block`
policy values, and `resolve_estimated_basis_policy()` (explicit value →
`HEADROOM_BUDGET_ESTIMATED_BASIS` → `count`; an unknown value warns once
and falls back rather than failing proxy startup).
- `headroom/proxy/cost.py`: ledger entries are now `CostEntry(timestamp,
cost_usd, basis)` instead of a bare tuple; `record_tokens()` marks the
fallback branch `estimated` and logs one WARNING per model (deduped the
same way pricing warnings are, per #2504 — an unguarded warning on this
path fires once per request for a provider that never reports usage);
new `period_cost_breakdown()` and an optional `basis` filter on
`get_period_cost()`; new `budget_denial_detail()` builds the 429 body
where the ledger lives; `check_budget()` honors the policy while keeping
its `(allowed, remaining)` signature.
- `stats()` gains `budget_estimated_basis` (the active policy) and
`budget_basis` (the period split: `total_usd`, `measured_usd`,
`estimated_usd`, `estimated_pct`, `records`, `estimated_records`).
`merge_cost_stats()` already spreads `**cost_stats`, so both reach
`/stats["cost"]` with no extra plumbing.
- Operator knob wired through every config layer:
`ProxyConfig.budget_estimated_basis` (`models.py`), the Click
`--budget-estimated-basis` option with `envvar=` (`cli/proxy.py`), the
argparse `--budget-estimated-basis` flag (`server.py`, `default=None` so
the env var stays reachable), and a `SettingField` in the `Budget` group
(`settings_store.py`).
- `headroom/proxy/handlers/anthropic.py`: the 429 body now comes from
`budget_denial_detail()`, which names how much of the period's spend was
booked from an estimate and distinguishes "you overspent" from "I refuse
to enforce a hard limit on a guess".
- `headroom/cli/doctor.py`: the budget check stays **PASS** and appends
the estimated share (and the policy, when it isn't the default). No new
WARN state — a provider that never reports usage would otherwise sit at
a permanent WARN. Every new read is `.get()` + type-guarded so `doctor`
still works against an older running proxy.
- `docs/content/docs/metrics.mdx`: a "Measured vs Estimated Spend"
subsection with the `/stats` shape and the three policy values.
- Tests: new `tests/test_cost_budget_basis.py` (20 tests) plus 4 new
`doctor` tests; `tests/test_anthropic_pre_upstream_backpressure.py`'s
cost-tracker double gained `budget_denial_detail()` to match the
handler's duck-typed contract.
### Policy values
| `HEADROOM_BUDGET_ESTIMATED_BASIS` | Effect on the hard limit |
|---|---|
| `count` (default) | Unchanged behavior — estimated spend consumes the
budget. |
| `ignore` | Booked and reported, but only measured spend enforces. |
| `block` | Fail closed — refuse rather than enforce a hard limit on a
guess. |
Default enforcement is unchanged. `CHANGELOG.md` is untouched.
## Testing
- [x] Unit tests pass (`pytest`) — every test covering the changed
modules; see `Not tested` for this machine's pre-existing environment
failures
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`) — clean on every file this
PR touches
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_cost_budget_basis.py tests/test_cost_tracker_counterfactual.py tests/test_cost_pricing_warning_dedup.py -q
31 passed
$ python -m pytest tests/test_cli_doctor.py -q
72 passed
$ python -m pytest tests/test_anthropic_pre_upstream_backpressure.py -q
25 passed
$ python -m pytest tests/test_proxy_settings_endpoints.py tests/test_proxy/test_settings_store.py -q
50 passed
# full suite (see "Not tested" below for the excluded modules and the pre-existing failures)
$ python -m pytest -q
...
tests\test_cost_budget_basis.py .................... [ 25%]
tests\test_cost_pricing_warning_dedup.py ... [ 25%]
tests\test_cost_tracker_counterfactual.py ........ [ 25%]
...
217 failed, 8807 passed, 657 skipped, 5318 warnings, 59 errors in 716.30s (0:11:56)
# same failing files re-run on clean upstream/main with the change stashed -> identical count
$ git stash push -u -- headroom tests docs
$ python -m pytest tests/test_log_compressor.py tests/test_cache/test_client_integration.py \
tests/test_fsutil.py tests/test_savings_ledger.py tests/test_ccr_mcp_http.py \
tests/test_router_registry_dispatch.py tests/test_proxy_savings_history.py \
tests/test_text_compressors.py tests/test_builtin_compressor_adapters.py \
tests/test_cli_proxy_env.py -q
73 failed, 182 passed in 34.82s # 40+16+2+2+1+1+1+4+3+3 = 73, matching the run above
$ python -m mypy headroom --ignore-missing-imports --python-version 3.13
# 12 errors, all in release_version.py / ccr/mcp_server.py / memory/mcp_server.py
# (stale local `mcp` stubs) — none in any file this PR touches
$ python -m ruff check headroom/proxy/budget_basis_policy.py headroom/proxy/cost.py headroom/proxy/server.py headroom/proxy/models.py headroom/proxy/handlers/anthropic.py headroom/cli/proxy.py headroom/cli/doctor.py headroom/settings_store.py tests/test_cost_budget_basis.py tests/test_cli_doctor.py tests/test_anthropic_pre_upstream_backpressure.py
All checks passed!
$ python -m ruff format --check <same 11 files>
11 files already formatted
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.13.11, branch
`fix/budget-estimated-basis-2713` off `upstream/main` @ `
|
||
|
Chester
|
3eb0122068
|
fix(learn): filter ambient user-role scaffolding (#2275)
## Description Fixes #2274. Headroom Learn currently trusts `role=user` as sufficient preference provenance. Agent harnesses can transport ambient UI and orchestration context in user-role messages, and OpenAI Responses normalization also promotes missing roles to `user`. Correction-like text in those inputs can therefore become durable user preferences. This change keeps preference learning fail-closed for known non-user sources while preserving genuine user corrections. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Refactoring only ## Changes Made - Preserve missing OpenAI Responses roles as `unknown` instead of promoting them to `user`. - Canonicalize user-role text before preference extraction. - Remove proxy-appended `## Relevant Memories` suffixes from preference evidence. - Reject strict ambient-only harness prefixes such as heartbeat, environment, workspace-instruction, delegation, and app-context envelopes. - Apply the same guard in `on_messages` and `_extract_preferences` for defense in depth. - Add regression coverage for system/developer/unknown roles, ambient-only user messages, memory-only messages, and mixed genuine-user-plus-memory input. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text 149 passed, 1 warning ruff check: passed ruff format --check: passed git diff --check: passed ``` Focused test files: ```text tests/test_memory/test_traffic_learner.py tests/test_openai_responses_traffic_learner.py ``` ## Real Behavior Proof - Environment: macOS; Python 3.13; current Headroom main; direct invocation of the real `TrafficLearner` class, with no proxy or database mocks - Exact command / steps: create `TrafficLearner(backend=None, min_evidence=1)`; feed system, developer, heartbeat user-role, and memory-only user-role messages; read `patterns_extracted`; feed a genuine user correction followed by a `## Relevant Memories` suffix; read `patterns_extracted` again - Observed result: `ambient_patterns=0`, `after_user_patterns=1` — the ambient batch produced no preference evidence; the genuine correction produced one pattern, while the appended memory content did not become evidence - Not tested: live provider traffic against a remote OpenAI endpoint; every possible third-party harness envelope; migration or cleanup of already-persisted noisy memories ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] No new dependency - [x] Fail-open proxy behavior is unchanged - [x] Regression tests added - [x] Public examples contain no real user data - [x] CHANGELOG update, if requested (not requested — N/A) ## Additional Notes This extends the source filtering introduced by #466 rather than replacing it. The prefix checks are deliberately strict and anchored at the start of a canonicalized message. The intended failure mode is a missed preference, not durable storage of non-user instructions. Note: the strict prefix set was discussed and confirmed in JerrettDavis's review approvals. |
||
|
Rod Boev
|
232fb49c73
|
fix(proxy): route Codex Live voice through a dedicated /v1/live transport (#2709)
## Description Codex Live traffic currently reaches an unrouted WebSocket path and receives HTTP 403 before the proxy can contact an upstream. Closes #2653 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring ## Changes Made - Add a dedicated `/v1/live` WebSocket route family and transparent transport. - Preserve subscription auth routing, account headers, origin policy, subprotocols, and text/binary frame bytes. - Propagate WebSocket close metadata and cancel relay tasks deterministically on every exit. - Keep Live outside the Responses parser, compression, memory injection, and Responses beta-header path. - Keep generic HTTP paths on the existing catch-all and document the Live aliases plus the derived-path override. - Add real-app route, relay, and loopback integration proof. - Add coverage for authorization fallback, defensive receive events, and cancellation cleanup in the Live relay. ## Testing The focused Live handshake, preservation suites, Ruff, format, and diff checks pass. The base comparison, Codex Desktop owner round trip, and ChatGPT backend acceptance of the derived `/backend-api/codex/live` path remain untested. - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy headroom --ignore-missing-imports`) - [x] New tests added for the reported failure - [x] Manual loopback testing performed ### Test Output ```text uv run pytest tests/test_codex_live.py -q: 6 passed, 7 warnings in 11.03s uv run pytest tests/test_provider_proxy_routes.py tests/test_proxy_codex_route_aliases.py tests/test_provider_codex_endpoints.py tests/test_openai_codex_routing.py -q: 53 passed in 90.54s (0:01:30) uv run ruff check headroom/providers/codex/live.py headroom/providers/proxy_routes.py headroom/proxy/handlers/openai.py headroom/proxy/ws_headers.py tests/test_codex_live.py: All checks passed uv run ruff format --check headroom/providers/codex/live.py headroom/providers/proxy_routes.py headroom/proxy/handlers/openai.py headroom/proxy/ws_headers.py tests/test_codex_live.py: 5 files already formatted uv run mypy headroom --ignore-missing-imports: Success: no issues found in 508 source files git diff --check: pass ``` ## Real Behavior Proof The local WebSocket integration floor uses real uvicorn, a real WebSocket client, and a real loopback WebSocket upstream. The base 403 comparison was not run. Head observes HTTP 101 on every Live alias and a byte-identical binary frame relay. - Environment: Windows, CPython 3.13, the Headroom proxy test environment. - Exact command / steps: run the focused Live test against the local uvicorn proxy and loopback WebSocket upstream, then run the preservation suite listed in `Test Output`. - Observed result: all four Live aliases return HTTP 101, negotiate `codex.live.v1`, preserve text and binary frames, and pass the preservation suite. - Not tested: Codex Desktop Live session; ChatGPT backend acceptance of `/backend-api/codex/live`; the base 403 comparison. ## Review Readiness - Live has a separate transport and does not enter Responses handling. - Existing Responses and generic passthrough suites remain preservation gates. - No `CHANGELOG.md` or install/crate changes are included. - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] Closes #2653 - [x] Real loopback handshake and binary-frame proof required - [x] No audio payload logging - [x] No unqualified end-to-end claim ## Screenshots Not applicable. ## Additional Notes The upstream Live path is derived from the repository’s Codex URL formula and remains explicitly unconfirmed until owner evidence is available. |