mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
2423 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Tejas Chopra
|
ad56dd382b
|
fix(router): compare token quantities in one unit (#2759)
## Description Two places compared a token quantity against something measured in a **different unit**. Both changed compression **behaviour**, not just reporting — which is the worse class. ### 1. The CONFIG branch put a word count in a token ratio `compressed_tokens = len(compressed.split())` was divided by `original_tokens`, which comes from `_estimate_tokens(content)`. Words run ~2.8× fewer than estimator tokens on config text, so a compressor that returned its input **byte-identically** scored ~0.36. `min_ratio` is 1.0 — accept any real shrink — so the router **accepted the no-op**: cached the result, pinned a frozen "compress" verdict, emitted a `router:config_compressor` label into `transforms_applied`, and recorded a fabricated saving to TOIN. ```text mkdocs.yml, compressor returns its input unchanged denominator (_estimate_tokens) = 936 OLD numerator len(split()) = 334 -> ratio 0.357 claims 64% saved ACCEPTED NEW numerator (_estimate_tokens) = 936 -> ratio 1.000 correctly rejected ``` The sibling TABULAR branch already used `_estimate_tokens`; CONFIG was the outlier. ### 2. The Kompress size gate tested a token cap in chars/4 `len(text_to_compress) > self._kompress_max_tokens * 4` under-counts anything denser than 4 chars/token, and compact JSON runs ~3.2. Against the 50,000-token default there is a band where an oversized payload passes: ```text records chars old_gate(len/4) new_gate(tokens) 2700 119,281 False False 4000 177,781 False True <- 44,445 vs 55,557 tokens, 11% over 4400 195,781 False True <- 48,945 vs 61,182 tokens, 22% over 5000 222,781 True True ``` Those payloads entered ONNX inference — exactly the >30s non-preemptible worker stall the gate exists to prevent (#1171). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `content_router.py` CONFIG branch — `compressed_tokens` now from `_estimate_tokens(compressed)`, matching its denominator and every sibling branch. - `content_router.py` Kompress gate — compared with `_estimate_tokens`, the unit the cap is actually expressed in. The extra O(n) char scan is negligible against the inference it guards. ## Testing - [x] Unit tests pass - [x] Linting passes (`ruff check` + `format --check`, pinned 0.15.17) - [x] New tests added - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_content_router_token_units.py -q 4 passed in 0.33s $ uvx ruff@0.15.17 check headroom/transforms/content_router.py tests/test_content_router_token_units.py All checks passed! ``` **Regression check against clean `upstream/main` in the same environment:** ```text tests/test_transforms/ + test_transforms_content_router.py + kompress suites upstream/main : 2 failed, 468 passed, 78 skipped this branch : 2 failed, 468 passed, 78 skipped failure sets : identical ``` The 2 failures are `test_kompress_failsafe`'s artifact-selection tests, which need a real `onnxruntime` this throwaway env lacks. Unrelated to this change. The 4 new tests pin the unit contract *and* the bounds of the disagreement band — including the cases where both formulations agree, so the band is demonstrated rather than assumed. ## Real Behavior Proof - **Environment:** macOS 26.4 arm64, isolated worktree off `upstream/main`. `content_router` needs the compiled `headroom._core`, which isn't in a fresh worktree (gitignored, built in-place); I copied the built `.so` in to run these, then removed it before committing. - **Observed:** both tables above are from running the real `_estimate_tokens` against the real thresholds, not reconstructed arithmetic. - **Not tested:** no live ONNX inference — the >30s stall the gate prevents is cited from #1171, not reproduced. The CONFIG no-op was demonstrated at the ratio level rather than by driving a stubbed compressor through `apply()`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] I did **not** edit `CHANGELOG.md` ## Related Third of three PRs from one tokenizer-consistency audit — see #2757 (litellm total-prompt / `--budget`) and #2758 (HuggingFace chat templates, `gpt-5`, gateway-wrapped names). Separate subsystems, separate risk. Known remaining from the same audit, not in any of the three: `_netcost_message_tokens` pricing an image by Python `repr` (34× over-count, flag-gated), three transforms reporting via `count_text(str(content))` where the pipeline uses `count_messages` (19% apart in one log file), `frozen_message_count` walking a chars/3.5 estimate against provider-reported cached tokens, and `target_ratio` honoured in words while documented as tokens. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Tejas Chopra
|
a033ac4176
|
fix(cost): send litellm the total prompt so --budget stops seeing $0 (#2757)
## Description **`--budget` has been silently inert on any cache-warm request** — which is the normal case in an agent session. This is a disabled control, not a metrics bug. `record_tokens` passed only the **uncached slice** as litellm's `prompt_tokens`. Measured, `litellm.cost_per_token` charges: ``` (prompt_tokens - cache_read - cache_creation) * input_rate + cache_read * read_rate + cache_creation * write_rate ``` So `prompt_tokens` is the **whole** prompt and litellm removes the cached parts itself. Handing it the uncached slice drives the input term **negative** as soon as anything is cached. `estimate_cost` ends with `float(total) if total > 0 else None`, so it returned `None`, no `CostEntry` was appended, and `check_budget()` saw **$0**. | model | 100k prompt, 80k cached — old call | booked | | --- | --- | --- | | `gpt-5` | **-$0.065000** | None | | `gpt-4o-mini` | **-$0.003000** | None | | `claude-sonnet-4-5` | **-$0.156000** | None | Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made Fixing the total alone would **over-charge OpenAI**, because two bugs are entangled here. OpenAI exposes no cache-write counter, so `_infer_openai_cache_write_tokens` uses the uncached portion as a write proxy. At two of the four inference sites (`openai.py:4304`, `openai.py:5413`) `uncached_input_tokens` is derived by subtracting **only** `cache_read`, so `cache_write_tokens` and `uncached_tokens` are **the same tokens**. Summing all three would double-count the prompt and charge a write premium OpenAI does not have. (The other two sites — `openai.py:3969` and `streaming.py:2024` — subtract both, so their buckets are genuinely disjoint; those are left alone.) - `cost.py` — `record_tokens` now passes `uncached + cache_read + cache_write` as the prompt total. - `cost.py` — new `cache_inferred: bool = False` parameter. When set, the inferred write is excluded from **both** the prompt total and the write premium. The default preserves behaviour for every provider that reports disjoint buckets. - `outcome.py` — plumbs `outcome.cache_inferred` through. The field already existed on `RequestOutcome` for the dashboard; it just never reached cost. - `handlers/openai.py` — sets `cache_inferred=True` at the two outcome sites whose buckets genuinely duplicate. ## Testing - [x] Unit tests pass (new file) - [x] Linting passes (`ruff check` + `format --check`, pinned 0.15.17) - [x] New tests added - [x] Manual testing performed ### Test Output ```text $ uvx ruff@0.15.17 check headroom/proxy/cost.py headroom/proxy/outcome.py headroom/proxy/handlers/openai.py All checks passed! $ uvx ruff@0.15.17 format --check <same three> 3 files already formatted $ pytest tests/test_cost_budget_total_prompt.py -q 5 passed in 0.26s ``` The 5 new tests assert on the **arguments handed to `estimate_cost`** rather than on dollar values, so they pin the contract that broke without depending on litellm's pricing tables — or on litellm being installed at all. ## Real Behavior Proof - **Environment:** macOS 26.4 arm64, isolated git worktree off `upstream/main`. **litellm's actual formula**, established by probe rather than assumed: ```text rates (claude-sonnet-4-5): input=3e-06 read=3e-07 write=3.75e-06 total=100k, 80k read, 5k write -> $0.087750 hypothesis (p-r)*ir + r*rr + w*wr = $0.102750 x hypothesis (p-r-w)*ir + r*rr + w*wr = $0.087750 <- matches ``` **Before → after:** ```text Anthropic, disjoint: uncached=900 read=48000 write=1500 OLD prompt=900 raw=-0.125775 booked=None <-- BUDGET BLIND NEW prompt=50400 raw= 0.022725 booked=0.022725 OpenAI, inferred write == uncached: uncached=20000 read=80000 write=20000 OLD prompt=20000 raw=-0.090000 booked=None <-- BUDGET BLIND NEW prompt=100000, write excluded raw= 0.035000 booked=0.035 truth=0.035000 ``` The OpenAI "after" equals the hand-computed truth `20,000*input + 80,000*read_rate` exactly. - **Not fully tested locally.** The cost/outcome suites show an **identical 10-failure set** on this branch and on clean `upstream/main` in the same throwaway env — all `ModuleNotFoundError: headroom._core`, the compiled Rust extension this machine cannot currently build. So they are environment-only, not regressions. One of them, `test_funnel_passes_canonical_record_tokens_shape`, covers the `record_tokens` call shape this PR changes, so **CI is the authoritative check for that one**. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] I did **not** edit `CHANGELOG.md` ## Context Found during a tokenizer-consistency audit that also turned up: HuggingFace-routed models counting a 6,000-char message as **2 tokens**, `gpt-5`/`o4-mini` falling to a char estimator, and the Kompress size gate missing its own cap by 24%. Those are separate PRs — different subsystems, different risk. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Tejas Chopra
|
06add9e9d8
|
fix(providers): stop pricing modern content blocks at zero (#2760)
## Description Each token counter in `headroom/providers/` had grown its own shortened content-block walker, handling only the shapes its provider was expected to send. Everything else fell through and contributed **nothing**. Measured on one 6,800-char block, via `count_messages` of a single-block message — so 7–8 is message overhead alone: | block type | OpenAI ctr | Anthropic ctr | |---|---|---| | `text` (control) | 3409 | 3748 | | `tool_result` | **8** | 3748 | | `thinking` | **8** | **7** | | `document` | **8** | **7** | | `mcp_tool_result` | **8** | **7** | | `output_text` | **8** | **7** | | `refusal` | **8** | **7** | Two things make this worse than a coverage gap: 1. **Each counter zeroed blocks from its own provider.** `output_text` and `refusal` are OpenAI Responses shapes; `thinking` and `document` are Anthropic's. 2. **These are the counters the live pipelines use.** `proxy/server.py` builds them with `AnthropicProvider` / `OpenAIProvider`, so this is the main request path — not an edge case. #2743 fixed this for `/v1/compress` only, by routing that route to the registry tokenizers, whose `BaseTokenizer` walker is complete. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made Rather than add a **fifth** partial walker, the counters now delegate to the audited one: - `tokenizers/base.py` — new `count_content_blocks(parts, count_text_fn)` plus a thin `_DelegatingBlockCounter` adapter, since the provider counters are not `BaseTokenizer` subclasses. `BaseTokenizer` itself is untouched. - `providers/openai.py`, `providers/anthropic.py`, `providers/openai_compatible.py` — list-content branches delegate. **Why delegate instead of adding a `count_text(str(block))` catch-all:** that would serialize a base64 blob and price it as text. `tiktoken_counter.py` already documents the failure — a 1MB image becomes ~330K phantom tokens. The shared walker gives media a pixel/byte-based estimate. **Scope:** the three counters that accumulate token counts. `google.py` and `cohere.py` extract a *text string* first and count that, so the same defect there needs a differently-shaped fix — left as a follow-up. ## Testing - [x] Unit tests pass - [x] Linting passes (`ruff check` + `format --check`, pinned 0.15.17) - [x] New tests added - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_provider_counter_content_blocks.py -q 12 passed in 0.52s $ uvx ruff@0.15.17 check headroom/ tests/... --exclude headroom/dashboard/templates All checks passed! ``` **After the fix**, every shape lands within ~1% of the equivalent plain text, and media stays bounded: ```text block OpenAI Anthropic text (control) 3409 3748 tool_result 3409 3748 thinking 3419 3759 document 3423 3763 mcp_tool_result 3422 3762 output_text 3420 3760 refusal 3421 3761 image b64 200KB 1608 1607 <- pixel estimate, not ~50K as text ``` ## Real Behavior Proof — including a regression I caught **This change flipped an existing test**, and I only found it because every suite was run against clean `upstream/main` in the same environment with the failure sets diffed: ```text before the test rewrite: upstream/main : 1 failed, 104 passed this branch : 2 failed, 103 passed <- regression diff : + test_openai_compatible_token_counter_ignores_unhandled_content_shapes ``` That test asserted `content: [{"type": "image"}, 123] == 8` — i.e. it **pinned the defect**, that unhandled shapes contribute nothing. Rewritten as `..._prices_declared_media`: a declared image is now priced (1608) while a bare int is still correctly ignored (8), with the rationale in the docstring. ```text after the rewrite: upstream/main : 1 failed, 104 passed, 10 skipped, 25 errors this branch : 1 failed, 104 passed, 10 skipped, 25 errors failure sets : IDENTICAL ``` - **Pre-existing, not from this change:** the 1 failure and all 25 errors. The errors are all in `test_compress_route_tokenizer_by_model.py`, whose loopback `TestClient` fixture this throwaway env cannot satisfy. - **Environment note:** `content_router` and several suites need the compiled `headroom._core`, which isn't in a fresh worktree (gitignored, built in-place). I copied the built `.so` in to run these and removed it before committing. - **Not tested:** no live provider call, so the *absolute* accuracy of the 1600 image estimate against a real Anthropic/OpenAI bill is unverified — it is the value `BaseTokenizer` already used, and this PR only changes which blocks reach it. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] I did **not** edit `CHANGELOG.md` ## Related Fourth PR from one tokenizer-consistency audit: #2757 (litellm total prompt / `--budget`), #2758 (HuggingFace chat templates, `gpt-5`, gateway-wrapped names), #2759 (router token units). Plus #2756, which splits the local/provider token scales in `RequestOutcome`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Tejas Chopra
|
0ed306b22b
|
fix(tokenizers): count HuggingFace chat templates, and resolve gpt-5 / gateway-wrapped names (#2758)
## Description Three tokenizer-selection defects, all measured against real counters on identical text. ### 1. HuggingFace-routed models counted a whole conversation as **2 tokens** `transformers >= 5` defaults `apply_chat_template(tokenize=True)` to `return_dict=True` and returns a `BatchEncoding`, so `len(formatted)` counted **dict keys** — `input_ids`, `attention_mask` — instead of tokens. ```text Qwen2.5-72B, one 6,000-char message before: count_messages = 2 count_message = -1 after : count_messages = 1020 count_message = 1017 true : ~1003 ``` `count_message` goes negative because `BaseTokenizer` subtracts a 3-token reply overhead from it. A **~99.8% undercount** on every HF-routed family whose resolved tokenizer carries a chat template — llama, qwen, deepseek, phi, yi, falcon, starcoder. `pyproject.toml` pins `transformers>=5.5.0,<6.0`, so the affected version is the only installable one, and nothing covered `count_messages`. It hid behind a second bug while I reproduced it: `DeepSeek-V3` mis-resolves to `deepseek-llm-7b-base` (a 2023 model with **no** chat template), which falls back to the estimator and looks fine. That mis-resolution is left for a follow-up. ### 2. The current OpenAI flagships had no pattern `MODEL_PATTERNS` stopped at `^gpt-4` / `^o1` / `^o3`: ```text gpt-5, gpt-5.1, gpt-5-mini, gpt-5.1-codex, o4-mini -> EstimatingTokenCounter ``` Deviation vs the correct `o200k` encoding: **+20% English, -33% JSON, -44% logs.** ### 3. Every pattern is `^`-anchored, so gateway-wrapped ids matched nothing ```text bedrock/anthropic.claude-3-5-sonnet -> EstimatingTokenCounter vertex_ai/claude-sonnet-4-6 -> EstimatingTokenCounter openrouter/anthropic/claude-sonnet-4-6 -> EstimatingTokenCounter us.anthropic.claude-sonnet-4-6-v1:0 -> EstimatingTokenCounter azure/gpt-4o -> EstimatingTokenCounter ``` Deviation: **+15% English, -33% JSON, -38% logs.** Not hypothetical — `handlers/openai.py` already documents that LiteLLM's `headroom` guardrail passes exactly these forms. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `tokenizers/huggingface.py` — pass `return_dict=False` to `apply_chat_template`. - `tokenizers/registry.py` — add `^gpt-5` and `^o4` to `MODEL_PATTERNS`. - `tokenizers/registry.py` — new `_name_candidates()`; `_detect_backend` now tries progressively-unwrapped forms: path segments stripped left-to-right, then Bedrock's dotted `[region.]vendor.model`. **Why candidates rather than rewriting the name:** the full name is candidate 0, so no currently-correct resolution can move, and an unknown alias still falls back to estimation rather than matching by accident. The estimator is a legitimate *fallback*; the bug was reaching it when a real tokenizer for that family exists. ## Testing - [x] Unit tests pass - [x] Linting passes (`ruff check` + `format --check`, pinned 0.15.17) - [x] New tests added - [x] Manual testing performed ### Test Output ```text $ uvx ruff@0.15.17 check headroom/ tests/test_tokenizer_selection_coverage.py --exclude headroom/dashboard/templates All checks passed! $ pytest tests/test_tokenizer_selection_coverage.py -q 20 passed in 0.60s $ pytest tests/test_huggingface_tokenizer_timeout.py tests/test_tokenizers/ -q this branch: 12 passed clean upstream/main: 12 passed <- no regression ``` 20 new tests cover all three defects **plus** the no-regression cases: bare names unchanged, unknown aliases still estimated, wrapped Gemini matching its bare form exactly, and candidate ordering/dedup. ## Real Behavior Proof - **Environment:** macOS 26.4 arm64, isolated worktree off `upstream/main`. The HF measurement used a real `transformers 5.14.1` with `Qwen/Qwen2.5-72B` from the local HF cache. **After the fix, resolution across every form a gateway realistically sends:** ```text gpt-4o TiktokenCounter gpt-5 TiktokenCounter <- was Estimating gpt-5.1 TiktokenCounter <- was Estimating o3-mini TiktokenCounter o4-mini TiktokenCounter <- was Estimating claude-sonnet-4-6 TiktokenCounter bedrock/anthropic.claude-3-5-sonnet TiktokenCounter <- was Estimating anthropic.claude-3-5-sonnet-20241022-v2:0 TiktokenCounter <- was Estimating us.anthropic.claude-sonnet-4-6-v1:0 TiktokenCounter <- was Estimating vertex_ai/claude-sonnet-4-6 TiktokenCounter <- was Estimating openrouter/anthropic/claude-sonnet-4-6 TiktokenCounter <- was Estimating azure/gpt-4o TiktokenCounter <- was Estimating vertex_ai/gemini-2.5-pro EstimatingTokenCounter (google backend, correct) groq/llama-3.3-70b-versatile HuggingFaceTokenizer <- was Estimating my-gateway/big-model EstimatingTokenCounter (correct fallback) ``` `vertex_ai/gemini-2.5-pro` and `gemini-2.5-pro` return **identical** counts (600 on the same input), confirming the prefix strip reaches the google backend rather than the generic fallback. - **Not fully tested locally:** `tests/test_evals_cjk_tokenization.py` cannot collect in this env — `ModuleNotFoundError: headroom._core`, the compiled Rust extension this machine can't currently build. Identical on baseline, so CI is the check there. It is CJK-related and this PR changes encoding selection for `gpt-5`/`o4`/wrapped names, so it's the suite most worth watching. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] I did **not** edit `CHANGELOG.md` ## Known follow-ups, deliberately not here - `DeepSeek-V3` → `deepseek-llm-7b-base`, `Qwen/Qwen2.5-72B` → `Qwen/Qwen-7B`: `get_tokenizer_name` prefix-matches against the whole string including the org segment, and has no version boundary. - `get_encoding_for_model` is case-sensitive while `_detect_backend` lowercases, so `GPT-4O` gets `cl100k` (+38.9% on CJK). - `providers/openai.py` has a second, divergent encoding resolver — it disagrees with `tokenizers/` on `gpt-4.1`, `gpt-5`, `text-embedding-3-large`, `davinci`. - Provider counters price most modern content blocks at literally zero (`thinking`, `document`, `mcp_tool_result`, and OpenAI's own `output_text`/`refusal`). 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
nangsontay
|
184146b688
|
fix(savings): surface request growth the tok_saved clamp swallows (#2708)
## Description `tokens_saved` is clamped at zero, so a request the proxy forwards **larger** than it arrived is indistinguishable in the PERF line from one it simply could not compress. Both read `tok_saved=0`. That ambiguity hides real regressions. Anything that appends to the body after compression — proactive context expansion, memory injection — can outweigh the compression it sits on top of and still look like a neutral turn. On the session that prompted this, a request went from 55,161 tokens in to 57,845 out and reported `tok_saved=0`, for 19 consecutive turns, with nothing in the logs distinguishing it from a turn with nothing left to compress. This reports the swallowed amount as `tok_inflated`. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `RequestOutcome.tokens_inflated`: `max(0, optimized_tokens - original_tokens)`, derived from two counts the outcome already carries — **no new plumbing at any of the emit sites**. - Added `tok_inflated=` to the PERF log line, next to `tok_saved=`. Diagnostic only, deliberately. It does **not** feed `tokens_saved` or `attempted_input_tokens`, for two reasons: 1. `attempted_input_tokens = optimized_tokens + tokens_saved` is a *size*, not a signed delta. Letting the second term go negative makes it smaller than the bytes actually forwarded, corrupting the active-savings denominator. 2. Injection paths already book their own cost through the retrieval-drawback channel. A negative landing in `tokens_saved` as well would count the same loss twice. So the clamp stays and the hidden number surfaces beside it. Worth noting there is already a revert-on-inflation guard *before* compression's own inflation can escape (`anthropic.py`, "Optimization inflated tokens … reverting to original messages") — it is only growth added *after* that point which the clamp was silently absorbing. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_request_outcome.py tests/test_cli_perf_format.py -q ============================== 54 passed in 1.56s ============================== $ pytest tests/ -q -k "outcome or perf or savings or stats" ======= 510 passed, 33 skipped, 9674 deselected, 542 warnings in 40.37s ======== $ ruff check headroom/proxy/outcome.py tests/test_request_outcome.py All checks passed! $ ruff format --check headroom/proxy/outcome.py tests/test_request_outcome.py 2 files already formatted $ mypy headroom/proxy/outcome.py --ignore-missing-imports Success: no issues found in 1 source file ``` Four new tests pin the distinction that was missing: shrank (0), no-op compression (0, and `tok_saved` also 0 — the two cases that used to look identical), grew (reports 2684 while `tok_saved` stays 0), and that `attempted_input_tokens` / `savings_pct` keep their unsigned semantics. `tests/test_cli_perf_format.py` parses hand-written PERF fixtures by field name, so adding a field does not disturb it — verified green above. ## Real Behavior Proof ### The field catching a real inflating request - Environment: macOS 15 (arm64), Python 3.13.14. A proxy booted from this branch: `headroom proxy --mode token --backend anthropic --anthropic-api-url http://127.0.0.1:<stub>`, isolated `HOME` so the run could not touch a developer's live logs/store, `HF_HOME` pointed at cached kompress weights so the lossy+CCR-marker path is exercised and proactive expansion can actually arm. - Exact command / steps: three requests over one conversation through the real HTTP path (`x-headroom-cwd: /tmp/proof`, `user-agent: claude-code/1.4.2`): a user turn carrying the real `~/.claude/rules/*.md` text (~8.2k tokens); then `assistant` + a short user turn so that block becomes compressible and gets tracked as a CCR entry; then a follow-up whose leading text block shares vocabulary with it, so proactive expansion fires and appends the original — which is how a request ends up leaving larger than it arrived. PERF lines read from the isolated `~/.headroom/logs/proxy.log`. - Observed result: real PERF output from that run — ```text msgs=1 tok_before=8170 tok_after=9553 tok_saved=0 tok_inflated=1383 ... transforms=router:text_block:mixed msgs=3 tok_before=8184 tok_after=9567 tok_saved=0 tok_inflated=1383 ... transforms=router:text_block:mixed msgs=5 tok_before=8245 tok_after=11880 tok_saved=0 tok_inflated=3635 ... transforms=router:text_block:mixed ``` Correlated from the same run: `CCR Tracker: Proactively expanded f0cf4efb42373ec225f57725 (1417 items)`, and the stub upstream confirms the block reached the wire (`has_expansion_block: true`, forwarded body 54,130 B on the third request). Every one of those turns reports `tok_saved=0`. Before this change that is all the log said, and it is the same thing it says when there was simply nothing left to compress. `tok_inflated=3635` is the number that was missing. For contrast, the same scenario run against a build where the request genuinely shrinks reported `tok_before=8245 tok_after=7359 tok_saved=886` — that build predates this field, so it does not print `tok_inflated`; the point is only that the inflating and shrinking cases are the two states the field has to separate, and on `main` today both render as `tok_saved=0` whenever the growth path is taken. The `tok_inflated=0` case on a shrinking request is covered by unit test. ### Scale of what was hidden From a live `--mode token` proxy on Claude Code traffic across four rotated logs: **305 of 3,743 requests (8%)** had `tok_after > tok_before` while every one reported `tok_saved=0` — 192,829 tokens of growth rendered as "nothing to compress". The worst single session held +2,643/turn for 19 consecutive turns. - Not tested: the `headroom perf` CLI was not run against a real log file containing the new field — it parses by field name and `tests/test_cli_perf_format.py` is green, but that is test-level rather than end-to-end evidence. Streaming responses were not exercised (the stub replies non-streaming), so the streaming emit path carries the new field on the strength of sharing `emit_request_outcome` rather than by observation. No dashboard or Prometheus consumer was re-run. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — one added field on an existing log line; no user-facing surface. ## Additional Notes - Independent of #2706 and #2707 — verified `conflicts=0` via `git merge-tree`; mergeable in any order. - Related but deliberately out of scope: `main` has no producer for retrieval-cost accounting (`record_savings_event` takes no `kind`/`tokens_retrieved`, and nothing writes `tokens_retrieved` anywhere), so proactive expansion's cost is not booked into net savings at all. Adding that channel is a cross-cutting accounting change and belongs in its own PR; this one only makes the growth visible in the log. |
||
|
Rod Boev
|
dcb674b5e4
|
fix(compression): honor qualified CCR names across integrations (#2698)
## Description Three compression consumers compare tool names against the bare literal `headroom_retrieve`, so the qualified forms MCP clients actually send (`mcp__Headroom__headroom_retrieve`, `mcp_Headroom_headroom_retrieve`) slip past the guard and get recompressed. `SmartCrusher.apply` has the bare comparison at both its OpenAI `role=tool` site and its Anthropic `tool_result` block site; the LangGraph compressor and the Strands hook have no tool-name check at all. Recompressing already-retrieved CCR content mints a new `<<ccr:hash>>` marker the agent cannot redeem. `headroom.config.is_tool_excluded` already owns alias resolution, including the MCP wrapper forms. This routes all three consumers through it instead of adding a second name matcher. Closes #2656. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `SmartCrusher.apply` routes both its `role=tool` and its Anthropic `tool_result` guards through `is_tool_excluded` - `_should_skip` in the LangGraph compressor takes the tool name and skips excluded tools; tool-call names are indexed by id so a `ToolMessage` without a copied `name` is still classifiable - `_should_skip_compression` in the Strands hook takes the tool name and skips excluded tools, recording `tool_excluded` - regressions for the qualified and bare names across all three consumers, the Anthropic block shape, the MCP wrapper entry point, and a near-match name that must still compress - a LangGraph regression for incomplete tool-call metadata that continues to a later qualified call ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output `pytest tests/test_smart_crusher.py tests/integrations/test_langgraph.py tests/integrations/test_strands tests/test_transforms/test_smart_crusher_ccr_retrieve_exemption.py -q` ```text tests\test_smart_crusher.py ............ [ 10%] tests\integrations\test_langgraph.py ..... [ 15%] tests\integrations\test_strands\test_ccr_exclusion.py ..... [ 19%] tests\integrations\test_strands\test_hooks.py sssssssss [ 27%] tests\integrations\test_strands\test_hooks_unit.py ssssssssssssssssssssssssssssssssss [ 57%] tests\integrations\test_strands\test_model.py ssssssssssssssss [ 71%] tests\integrations\test_strands\test_model_unit.py sssssssssssssssssssssssssss [ 95%] tests\test_transforms\test_smart_crusher_ccr_retrieve_exemption.py ..... [100%] 28 passed, 86 skipped in the focused invariant suite ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12.13, `headroom._core` built - Exact command / steps: `uv run pytest tests/test_smart_crusher.py tests/integrations/test_langgraph.py tests/integrations/test_strands -q`, and the same suite run against the pre-change implementation with the new tests in place - Observed result: before the change, five regressions fail. `SmartCrusher` returns non-byte-identical content for a `mcp__Headroom__headroom_retrieve` result, the LangGraph compressor replaces the message content, and the Strands hook returns `"compressed"` in place of the tool output. After the change all three preserve the content byte-for-byte, incomplete LangGraph tool-call metadata is ignored while the later qualified call remains indexed, the Strands hook records `tool_excluded` and never calls the crusher, and `HeadroomMCPCompressor.compress` returns the payload unchanged. `mcp__Headroom__headroom_retrieve_extra` still compresses in all three, and the Kompress and ContentRouter suites are unchanged. - Not tested: the optional Strands package, so the additions to `tests/integrations/test_strands/test_hooks_unit.py` skip locally ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes One deliberate divergence from the issue: the suggested snippet passes `DEFAULT_VERBATIM_EXCLUDE_TOOLS` to `is_tool_excluded`, but that constant holds only `WebSearch`, `WebFetch`, `web_search`, `web_fetch`. Applied literally it would drop `headroom_retrieve` from the comparison entirely and delete the #1077 guard these two SmartCrusher sites exist to enforce. This passes `(CCR_TOOL_NAME,)` so each guard keeps doing the one thing it documents. If you'd rather these paths also honor the verbatim-exclude set, the tuple can become `(CCR_TOOL_NAME, *DEFAULT_VERBATIM_EXCLUDE_TOOLS)` — the CCR name has to stay in it either way. Adjacent work: PR #2654 covers `ContentRouter` only. |
||
|
michaeltarleton
|
677e09735a
|
fix(transforms): stop ContentRouter recompressing headroom_retrieve results (#2654)
## Description `ContentRouter` (the transform actually registered in the default/proxy compression pipeline -- see `transforms/pipeline.py`) recompresses the output of its own `headroom_retrieve` tool. That tool's entire contract is returning already-retrieved, original content verbatim; recompressing it produces a new `<<ccr:hash>>` marker the caller can never redeem -- an unresolvable retrieval loop. `SmartCrusher` already has a guard against this exact failure mode (#1077), but only on its `apply()` entry point. `ContentRouter` calls the lower-level `SmartCrusher.crush()` directly, bypassing that guard entirely, since `crush()` takes a raw content string with no tool identity at all. Closes #1077 (reopens the same failure mode ContentRouter's own call path, which #1077's original fix did not cover). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `transforms/content_router.py`: adds an unconditional guard to all three of the places `ContentRouter` can hand a `headroom_retrieve` result to compression: the OpenAI-shape `role:"tool"`/legacy `role:"function"` string-content loop, the Anthropic-shape `tool_result` block loop, and a third, distinct shape -- top-level `{"type": "text"}` blocks under a `role:"tool"`/`"function"` message that never go through a `tool_result` wrapper (a real, already-tested wire shape in this codebase; see `test_tool_role_text_blocks_compressed_by_default`). All three use `is_tool_excluded()` (not a bare comparison) because MCP-served tools appear here under their qualified form, e.g. `mcp__headroom__headroom_retrieve`. Legacy `role:"function"` messages carry no call id in that shape, so the tool name is read directly off the message's `name` field instead of through the id-keyed `tool_name_map`. - Hoisted the per-iteration `is_tool_excluded(..., ("headroom_retrieve",))` calls into a single precomputed `ccr_retrieve_tool_ids` set, computed once alongside the existing `excluded_tool_ids` set, rather than recomputing aliases on every message/block. - `config.py`: adds `"headroom_retrieve"` to `DEFAULT_EXCLUDE_TOOLS` and `DEFAULT_VERBATIM_EXCLUDE_TOOLS` -- this also covers a third path (cross-turn message dedup, `_cross_turn_dedup_messages`) that consults the same frozensets and has no dedicated guard of its own. Also hardens `_tool_name_aliases()` against a non-string tool name (pre-existing fragility, not introduced by this PR, but shares the same call path) by returning no aliases instead of crashing on `.lower()`. - Documentation: updated `ContentRouterConfig.exclude_tools`'s field comment (was stale -- didn't mention this override is unconditional even when a caller explicitly empties `exclude_tools`), and added a comment on `DEFAULT_VERBATIM_EXCLUDE_TOOLS` noting all three real consumers. - Kept `"headroom_retrieve"` as a literal string (matching every other entry in those frozensets) rather than importing the existing `CCR_TOOL_NAME` constant from `ccr.tool_injection` into `content_router.py` -- that module is imported eagerly by `pipeline.py` (unlike `smart_crusher.py`, which imports the same constant lazily), so pulling in `headroom.ccr` there would add a new eager-import edge to a hot module for a one-line DRY win. Happy to change this if a maintainer prefers the constant. **Known, accepted tradeoff:** `is_tool_excluded()`'s alias matching strips any `mcp__<server>__` prefix before comparing, so a third-party MCP server exposing a tool literally named `headroom_retrieve` would also match. Narrowing this to headroom's own server specifically would need a bespoke check inconsistent with how every other excluded-tool entry is matched in this codebase; given how specific the name is, the collision risk is accepted rather than special-cased. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_transforms/ tests/test_transforms_content_router.py -q 1 failed, 420 passed, 62 skipped in 12.50s FAILED tests/test_transforms/test_kompress_compressor.py::...test_onnx_session_options_read_thread_caps (pre-existing, unrelated to this diff -- confirmed via `git stash` that it fails identically against unmodified upstream/main; an ONNX thread-cap assertion, not a compression-routing test) $ uv run ruff check headroom/config.py headroom/transforms/content_router.py \ tests/test_transforms/test_content_router_ccr_retrieve_exemption.py \ tests/test_transforms_content_router.py tests/test_transforms/test_content_router.py All checks passed! $ uv run ruff format --check <same files> 5 files already formatted $ uv run mypy headroom/config.py headroom/transforms/content_router.py Success: no issues found in 2 source files ``` - `tests/test_transforms/test_content_router_ccr_retrieve_exemption.py`: 10 tests -- MCP-qualified name (Anthropic + OpenAI shape), bare name, unconditional-even-with- `exclude_tools=frozenset()`, negative control (normal tools still compressed, asserted via the absence of the `router:excluded:ccr_retrieve` marker), the top-level-text-block shape, legacy `role:"function"`, litellm list-form content nested in a `tool_result` block, mixed retrieve+normal blocks in one turn, and a content well below the compression floor (proving the guard is size-independent). - `tests/test_transforms/test_content_router.py`: `test_anthropic_mcp_bare_tool_alias_exclude_tools` (#1822) updated to assert the new, stronger byte-verbatim guarantee for `headroom_retrieve` specifically; `test_anthropic_mcp_bare_tool_alias_exclude_tools_generic` added to keep the original #1822 general-mechanism coverage (bare-alias matching for an arbitrary, non-exempt tool). - `tests/test_transforms_content_router.py`: updated 10 pre-existing `_process_content_blocks()` unit tests for the new `ccr_retrieve_tool_ids` parameter (all pass empty sets -- none of those tests involve `headroom_retrieve`). - Verified the local installed package copy (a separate, drifted internal version) with a standalone repro script exercising the two new shapes directly against `ContentRouter.apply()` -- both correctly report `router:excluded:ccr_retrieve`. ## Real Behavior Proof - Environment: Windows 11, Python 3.13.7, `uv sync --extra dev` on this branch. - Exact command / steps: standalone repro building an assistant `tool_use` for `mcp__headroom__headroom_retrieve` paired with a large-JSON `tool_result`, through `ContentRouter().apply()`; repeated for the top-level-text-block and legacy-`function`-role shapes. - Observed result: unpatched (Anthropic `tool_result` shape, `git stash` to `upstream/main`), the retrieve output was rewritten 3680 -> 1861 bytes (mangled into a compact tabular form); patched (this branch), it is forwarded 3680 -> 3680 bytes byte-identical, no `<<ccr:` marker present. The two additional shapes fixed in this PR's second commit -- top-level text block under `role:"tool"`, and legacy OpenAI `role:"function"` -- both report `excluded=True` (protected) against this branch, where they reported `excluded=False` (recompressed) before the second commit. - Not tested: the actual `headroom mcp serve` + `headroom wrap` proxy end-to-end over a live Anthropic API call (would need API credentials); the OpenAI-chat-completions `CompressionUnit` path (out of scope, see #1176 below); the opt-in `ToolResultInterceptorTransform` path. ## Relationship to other issues/PRs - Issue #1077 (closed) is this exact bug; PR #1323 fixed it only for `SmartCrusher.apply()`'s own call path (the "legacy" pipeline path, per `smart_crusher.py`'s own comment), not `ContentRouter`, which is what the default/proxy pipeline actually uses. - Open PR #1176 addresses an adjacent, non-overlapping gap: the `CompressionUnit`-based OpenAI chat-completions path (`router.compress()` calls in `transforms/compression_units.py`/`compression_batches.py`), which has no tool-identity context at all and needs its own capture/restore mechanism. This PR does not touch that path. - Filed #2656 as a follow-up: code review on this PR found the same bug class still reachable through `SmartCrusher.apply()`'s own bare-name guard (not alias-aware, so it misses the MCP-qualified form) and through two unguarded direct `.crush()` calls in the LangGraph and Strands integrations. Both are pre-existing, narrower/separate call paths from `ContentRouter`'s primary proxy pipeline, so tracking them separately keeps this PR reviewable as one logical change. - Also not covered by this PR (flagging rather than silently omitting): `proxy/system_compaction.py`'s `router.compress(text, context="")` call, and the opt-in `ToolResultInterceptorTransform` (`HEADROOM_INTERCEPT_ENABLED=1`) -- neither was checked for CCR-awareness. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` -- it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A -- this is a backend compression-routing fix with no UI surface. ## Additional Notes This PR is two commits: the first commit added the initial two-loop guard; a second commit followed after code review found the guard was incomplete for two additional wire shapes (top-level text blocks, legacy `role:"function"`) and added the missing test coverage plus a few cleanup items (deduplicated guard logic, comment accuracy, a pre-existing non-string-tool-name fragility). See `Changes Made` above for the full list. Filed #2656 for the remaining out-of-scope gaps found during that same review. --------- Co-authored-by: Michael Tarleton <mtarleton@istation.com> |
||
|
JD Davis
|
13a310a00d
|
feat(claude): support Claude Code in VS Code (#2752)
## Description Add first-class Headroom support for the official Claude Code extension in VS Code. The new wrapper starts the local proxy, configures the Claude Code user settings consumed by the embedded extension process, preserves authentication and model selection, and provides a conflict-safe reversible unwrap lifecycle. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add `headroom wrap vscode-claude` and `headroom unwrap vscode-claude`. - Configure project-scoped `ANTHROPIC_BASE_URL` plus `ENABLE_TOOL_SEARCH=true` in Claude Code user settings while preserving existing values. - Respect `CLAUDE_CONFIG_DIR`, macOS/Linux home paths, Windows `USERPROFILE`, custom `--settings-file`, and `--no-configure`. - Add durable Headroom-owned restore state and refuse malformed settings or conflicting user edits. - Add unit, CLI, and Docker-harness e2e coverage for configuration, real proxy forwarding, and restoration. - Document setup, remote development, undo, and troubleshooting. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ UV_NO_SYNC=1 uv run pytest -q tests/test_provider_claude_vscode_config.py tests/test_cli/test_wrap_vscode_claude.py tests/test_cli/test_wrap_vscode.py tests/test_cli/test_wrap_claude_base_url.py tests/test_provider_copilot_vscode_config.py tests/test_copilot_auth.py 160 passed in 0.45s $ UV_NO_SYNC=1 uv run ruff check . All checks passed! $ UV_NO_SYNC=1 uv run mypy headroom Success: no issues found in 512 source files $ npm run build # from docs/ Compiled successfully; generated 155 static pages ``` ## Real Behavior Proof - Environment: macOS, Python 3.13 editable install, isolated temporary HOME and Claude settings, local mock Anthropic Messages upstream. - Exact command / steps: invoked the new `verify_vscode_claude_wrap` e2e function, which launched real `headroom wrap vscode-claude`, waited for proxy readiness, POSTed an Anthropic `/v1/messages` request through the generated project-scoped URL, stopped the wrapper, then ran `headroom unwrap vscode-claude`. - Observed result: HTTP 200 with the mock Claude response through Headroom; generated settings retained unrelated values and enabled tool deferral; unwrap restored the original Claude settings. - Not tested: real Anthropic account traffic or the full Docker image locally because Docker Desktop was unavailable. The same e2e function is wired into the existing Docker wrap CI job. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable; this adds CLI configuration and proxy routing without changing VS Code UI. ## Additional Notes The wrapper deliberately leaves the endpoint configured when stopped so requests fail closed instead of silently bypassing Headroom. `headroom unwrap vscode-claude` restores the exact prior managed values and preserves unrelated settings. --------- Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net> |
||
|
nangsontay
|
08fce29b47
|
fix(proxy): stop toggling headroom_retrieve in the Anthropic tools array (#2672)
## Description `should_inject_ccr_tool` deferred CCR tool injection whenever `frozen_message_count > 0`. Because `tools` is the head of Anthropic's cache key, that dropped a tool which was already inside the provider-cached prefix and invalidated the whole prefix — in both directions (`0 → >0` removes it; `>0 → 0` on proxy restart, `/model` switch, lineage eviction or TTL lapse adds it back). On three days of local proxy logs the turns that flipped injection state carried **44.7% of all cache-write tokens at a 52.0% hit rate**, against 98.1% for non-flipping turns. The log signature is `cache_read` alternating between two values exactly 172 tokens apart — the 464-byte tool definition. This deletes the gate and calls `apply_session_sticky_ccr_tool` directly, which is **what `openai.py` already does** — the two handlers now have the same shape. Net −61 production lines, no new state, no new config flag. Fixes defect 1 of #2671. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Performance improvement ## Changes Made - `headroom/proxy/ccr_marker_policy.py` — deleted the `should_inject_ccr_tool` gate; `apply_session_sticky_ccr_tool` is now the single decision point. - `headroom/proxy/handlers/anthropic.py` — calls `apply_session_sticky_ccr_tool` directly, matching `openai.py`. - `headroom/proxy/helpers.py` — dropped the now-unused gate plumbing. - `tests/test_proxy_anthropic_cache_stability.py` — new test asserting the forwarded `tools` array is byte-identical across a `frozen 0 → >0` transition. - `tests/test_ccr_marker_policy.py` — removed the three unit tests that pinned the deleted decision (they encoded the defect). - `tests/test_proxy/test_ccr_frozen_prefix_coupling.py` — same unredeemable-marker intent, re-pinned at the sticky helper. - `tests/test_proxy/test_anthropic_ccr_deferred_injection.py` — autouse reset fixture for the process-global `SessionCcrTracker` (separate commit). - Formatting-only follow-up commit applying `ruff format` (pinned 0.15.17) to the two test files above. ### Why deleting the gate is safe `apply_session_sticky_ccr_tool` already holds the correct rule. Its four branches, in order: | # | condition | action | |---|---|---| | 1 | tool already in the incoming tool list (client/MCP pre-registered) | skip; the client's bytes win | | 2 | `session_id is None` (WS / pre-session) | per-turn flag drives it verbatim | | 3 | session has done CCR | always inject the recorded golden bytes | | 4 | fresh session, no compression this turn | **skip** | Branch 4 is the safety property: a session that has never compressed still gets no tool, so removing the gate cannot start injecting into non-CCR conversations. Branch 3 is what the gate was starving. `has_new_ccr_markers` still gates first-time injection, so markers replayed from the previously-forwarded prefix cannot trigger one. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed The three deleted unit tests encoded the defect. Coverage moves to the property that actually matters and was previously untested: **the forwarded `tools` array must be byte-identical across a `frozen 0 → >0` transition.** That test asserts on the forwarded request body rather than on a policy function's return value; unit-testing the old policy in isolation is exactly what let a wrong-but-self-consistent decision pass. Verified failing on `upstream/main` with an assertion on the missing tool (not an `ImportError`, so it fails for the right reason). Full suite: same pre-existing unrelated failures as `upstream/main`, **zero new** (verified by running the whole suite on both revisions and diffing the failure sets). ### Test Output ```text $ uv run pytest tests/test_ccr_marker_policy.py \ tests/test_proxy/test_anthropic_ccr_deferred_injection.py \ tests/test_proxy/test_ccr_frozen_prefix_coupling.py \ tests/test_proxy_anthropic_cache_stability.py -q collected 48 items tests/test_ccr_marker_policy.py ..... [ 10%] tests/test_proxy/test_anthropic_ccr_deferred_injection.py .............. [ 39%] . [ 41%] tests/test_proxy/test_ccr_frozen_prefix_coupling.py .. [ 45%] tests/test_proxy_anthropic_cache_stability.py .......................... [100%] ======================= 48 passed, 2 warnings in 13.59s ======================== $ ruff check . All checks passed! $ ruff format --check . 1349 files already formatted ``` ## Real Behavior Proof - Environment: local macOS proxy serving live Claude Code traffic to the Anthropic API; baseline = 3 days of proxy logs on `upstream/main`, after = 5.5 hours with this change live. - Exact command / steps: ran the proxy with this branch built in, drove normal Claude Code sessions through it (including `/model` switches and proxy restarts, the two events that used to flip injection state), then parsed 235 real turns from the proxy logs with the same parser used for the baseline in #2671. - Observed result: flip turns fell from 177 (44.7% of all cache write) to 2 (1.7%); steady-state write share 1.192% → 0.867%; aggregate hit rate 86.75% → 89.21%; main conversation warm hit rate 98.1% → 97.70% (n=149). The 2 remaining "flips" have `cache_read == 0` — cold starts that the bucketing counts as a state change, not real flips. | metric | baseline | after | |---|---|---| | flip turns | 177, carrying 44.7% of all cache write | **2**, carrying **1.7%** | | main conv, warm | 98.1% | **97.70%** (n=149) | | steady-state write share | 1.192% | **0.867%** | | aggregate | 86.75% | **89.21%** | - Not tested: `mypy headroom` was not run locally for this body; the OpenAI handler path (unchanged by this PR); tracker state loss mid-session (see note below); and defect 2 of #2671 (the sub-call breakpoint), which is untouched and is now 54.9% of remaining cache write — that is why aggregate stays just under 90%. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes **Pre-existing and unchanged here:** if the tracker loses state mid-session while the transcript still carries markers, branch 4 returns no tool and those markers are unredeemable. `upstream/main` has no recovery for that; this PR neither creates nor fixes it. See my comment on #2500, which adds a recovery path for the related dangling-reference case. **N/A checklist items:** no documentation changes — this removes an internal policy function with no user-facing surface. `mypy headroom` left unchecked because it was not run for this body; CI covers it. **Merge-order conflict with #2500 (please read before landing either):** this PR *deletes* `should_inject_ccr_tool`, which is the exact function #2500 extends with `transcript_requires_tool`. Whichever lands second needs a semantic rebase, not just a textual one — git will not flag it. If this PR lands first, #2500's recovery path should re-target `apply_session_sticky_ccr_tool` (the sticky helper now owns the decision alone) or the handler call site in `handlers/anthropic.py`. If #2500 lands first, the gate deletion here still applies but the `transcript_requires_tool` override needs to move with it. Happy to do the rebase either way — say which order you prefer. |
||
|
Tejas Chopra
|
0221e7f240
|
fix(deps): bump aiohttp and cryptography to clear the CVEs blocking 0.34.0 (#2753)
## Description
`Dependency audit (pip-audit)` is the **only** failing check on the
0.34.0 release PR (#2679), so this blocks the release regardless of what
else lands in it.
`aiohttp 3.14.1` carries three advisories, all reachable through the
`--extra all` production set that CI audits (transitive via `litellm` /
`instructor` / `kubernetes` / `fsspec`):
| CVE | Impact | Fixed in |
|---|---|---|
| CVE-2026-69243 | Request smuggling via an edge case in the WebSocket
upgrade procedure (server-side component) | 3.14.2 |
| CVE-2026-69244 | Out-of-bounds heap read in the C response parser
building an error message for a malformed response — an
attacker-controlled server can DoS the client | **3.14.3** |
| CVE-2026-59881 | Decompresses frames with RSV1 set even when
`permessage-deflate` was not negotiated | 3.14.2 |
3.14.3 is the floor that clears all three.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- Lock-only bump via `uv lock --upgrade-package aiohttp`. **No
`pyproject.toml` constraint added** — every parent already permits
3.14.3, so a floor would be redundant surface to maintain.
- The diff also syncs `headroom-ai` `0.32.0` → `0.33.0` in the lock. `uv
lock` rewrites that from `pyproject.toml` (`version = "0.33.0"`); the
lock's record of the project's own version was stale. Same drift #2663
targets — happy to drop this PR if #2663 lands first and you'd rather
keep them separate.
Diff is exactly two version changes (plus their wheel-hash blocks).
## Testing
- [x] Manual testing performed
- [x] Linting passes — no Python source touched
### Test Output
```text
$ uv lock --upgrade-package aiohttp
Resolved 269 packages in 2.91s
Updated aiohttp v3.14.1 -> v3.14.3
Updated headroom-ai v0.32.0 -> v0.33.0
$ git diff --stat uv.lock
uv.lock | 456 +++++++++++++++++-------------------
1 file changed, 234 insertions(+), 222 deletions(-)
$ git diff uv.lock | grep -E '^[+-]version = '
-version = "3.14.1"
+version = "3.14.3"
-version = "0.32.0"
+version = "0.33.0"
```
## Real Behavior Proof
- **Environment:** macOS 26.4 arm64, `uv` 0.x from Homebrew, isolated
git worktree off `upstream/main` @ `
|
||
|
Raúl
|
3f2ca99fe1
|
fix(ci): restrict Codecov shard uploads (#2745)
## Description Closes #2744 Restrict each Codecov Action v5 matrix upload to its declared `coverage-${{ matrix.shard }}.xml` report. This prevents automatic discovery from uploading the unsharded `coverage.xml` alongside every shard. ## Type of Change - [x] Bug fix (non-breaking change fixes an issue) ## Changes Made - Set Codecov Action `disable_search: true` for Python shard uploads. - Add a CI workflow contract test that protects the explicit-report-only setup. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy headroom`) (not applicable: workflow/test-only change) - [x] New tests added new functionality - [x] Manual testing performed (not applicable: GitHub Actions will execute the workflow) ### Test Output ```text $ uv run --with ruff ruff format --check scripts/tests/test_ci_workflow.py 1 file already formatted $ uv run --with ruff ruff check scripts/tests/test_ci_workflow.py All checks passed! $ uv run --with pytest pytest scripts/tests/test_ci_workflow.py -q 2 passed ``` ## Real Behavior Proof - Environment: GitHub Actions Ubuntu runner using Python 3.12.13; Codecov Action v5. - Exact command / steps: Run the CI Python test matrix, which writes `coverage-${{ matrix.shard }}.xml`, then runs the Codecov Action upload step. Inspect the uploader's discovered/uploaded report list. - Observed result: Before this change, raw CI logs showed the Action explicitly uploading `coverage-2.xml` and additionally discovering/uploading `coverage.xml`. This PR configures `disable_search: true`; the workflow contract test confirms the explicit report setting and search disablement. Runtime upload evidence will be added from this draft PR's CI run. - Not tested: Codecov's final cross-shard patch calculation; that depends on Codecov processing the reports after CI completes. ## Review Readiness - [x] I have performed a self-review - [x] This PR ready for human review ## Checklist - [x] My code follows project's style guidelines - [x] I have performed a self-review - [x] I commented my code, particularly in hard-to-understand areas - [x] I made corresponding changes documentation (not applicable) - [x] My changes generate no new warnings - [x] I added tests prove my fix is effective or feature works - [x] New and existing unit tests pass locally changes - [x] I did **not** edit `CHANGELOG.md` — generated by release-please from Conventional Commit PR title (a CI guard enforces this) ## Additional Notes This is intentionally limited to the Codecov upload configuration and its workflow contract test. It does not include the unrelated Copilot Keychain fix. |
||
|
Tejas Chopra
|
6422a80a58
|
fix(compress): resolve the /v1/compress tokenizer per model, and document the real contract (#2743)
## Description
`/v1/compress` does no format conversion — callers send whichever wire
shape they already use — but the pipeline pinned **one provider's token
counter for the whole route**.
`OpenAITokenCounter.count_message` walks list content for `text` and
`image_url` only and has **no else branch**, so Anthropic content blocks
contributed literally zero. A 599-token `tool_result` scored 8. A
request that really removed 235 characters reported `tokens_saved: 0` —
so a caller gating on `tokens_saved > 0` concludes compression is broken
while it is working.
Prompted by a Kong integration question ("do you support the Anthropic
native format?"). The answer is that we already did — we just reported
zeros for it, and the docs said otherwise.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Documentation update
## Changes Made
### Tokenizer resolution (no hardcoded lists)
Build the derived pipelines with `provider=None` so `TransformPipeline`
resolves the tokenizer from the **per-model registry**. Every registry
tokenizer derives from `BaseTokenizer`, whose `_count_content_parts`
ends in a serialize-and-count catch-all, which means:
- No block type counts as zero, and there is **no per-provider
block-type list to keep in sync**. An enumerated set was the first thing
I tried and it already missed `mcp_tool_result`,
`web_search_tool_result`, `document`, and `thinking`.
- Gemini / Mistral / DeepSeek / Kimi stop defaulting to a tiktoken count
when the registry already has a calibrated counter for them.
- Gateway aliases matching no vendor pattern still count correctly.
`mode="ccr"` now runs a derived pipeline too, for the same reason —
sharing `openai_pipeline` pinned its provider. Costs that mode its own
cold compression cache; correct metrics win.
### Tokenizer selection stays separate from context-limit resolution
Deliberately not welded together. `model_limit` feeds `context_pressure
-> min_ratio`, so letting a tokenizer decision pick the limit table
changes compression aggressiveness: `gpt-4-32k` answered by the
Anthropic table is **8,192 instead of 32,768**, a 4× under-estimate.
`test_tokenizer_choice_does_not_move_the_context_limit` pins the
independence.
### Docs, rewritten from the code
- **`proxy.mdx`** — the loopback-only default and **404-not-403**
behavior, previously undocumented *anywhere* in `docs/` despite shipping
in #2458 explicitly for gateway sidecars;
`HEADROOM_COMPRESS_ALLOW_REMOTE`; all four request fields; the whole
`config` object including every `mode` value and `frozen_message_count`;
`transforms_summary`; the 400/401/404/503 contract; and the timeout
fail-open shape (`compression_skipped` / `skip_reason`).
- **Corrected "never calls an LLM"** — accurate about *generative*
provider requests, misleading for a sidecar operator. Kompress (a
ModernBERT **encoder**, classification not generation) and Magika run
**in-process**, and `HEADROOM_KOMPRESS_ENDPOINT` offloads inference over
HTTP — **real egress**. Now stated explicitly, with
`HEADROOM_DISABLE_KOMPRESS=1` as the structural-only option.
- **Both wire formats documented as accepted**, and removed
`anthropic-sdk.mdx`'s claim that OpenAI format is "the compression
engine's native format" — the exact misconception that prompted this
work. The SDK's conversion is now framed as an SDK choice, not an API
requirement.
- **`litellm.mdx`** had no mention of the endpoint at all, despite the
code naming LiteLLM's guardrail as its primary consumer. Added the HTTP
deployment path, the `HEADROOM_COMPRESS_ALLOW_REMOTE` requirement, and
why to leave `config.mode` unset.
- **`index.mdx`** printed `compressionRatio * 100` labelled "Saved …%",
so a 77% saving displayed as **23%**. `api-reference.mdx` already
defined it correctly, so the docs contradicted each other.
- `openai-sdk.mdx`, `wiki/proxy.md`, `wiki/typescript-sdk.md` — same
corrections; dropped "any HTTP client", "Cloud", and a CacheAligner
claim (it is detector-only).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ .venv/bin/ruff check headroom/ tests/ --exclude headroom/dashboard/templates
All checks passed!
$ .venv/bin/mypy headroom/
Success: no issues found in 511 source files
$ python -m pytest tests/test_compress_route_tokenizer_by_model.py \
tests/test_proxy_compress_endpoint.py tests/test_compress_api.py \
tests/test_platform_stabilization_functional.py tests/test_proxy_eager_preload_bind.py -q
99 passed, 2 warnings in 47.15s
```
Broader sweep (`-k "compress or litellm or gateway or guardrail"`):
**1625 passed, 4 failed** — all 4 pre-existing, verified by stashing
this diff and re-running on clean `main` (2 strands hook tests, 1 codex
WS semaphore-tail timing test, 1 unrelated local WIP test).
## Real Behavior Proof
- **Environment:** macOS 26.4 arm64, Python 3.12.6, repo `.venv`, branch
rebased on `upstream/main`.
**(1) Before → after, same request** (60-line grep payload in an
Anthropic `tool_result`):
| model | before | after |
| --- | --- | --- |
| `claude-sonnet-4-6` | `before=28 saved=0` | `before=1223 saved=58` |
| `bedrock/anthropic.claude-3-5-sonnet` | `saved=0` | `before=1037
saved=59` |
| `my-gateway/big-model` (alias) | `saved=0` | `before=1037 saved=59` |
| `gemini-2.5-pro` | `saved=0` | `before=1036 saved=59` |
| `gpt-4o` + OpenAI shape | `before=1225 saved=58` | `before=1225
saved=58` (unchanged) |
All three `config.mode` values verified for each. Response shape
preserved: `type=tool_result`, `tool_use_id` intact.
**(2) Counter-level root cause**, 6.8 KB body, `count_message()`:
```text
OpenAITokenCounter string-content -> 1406 tool_result block -> 5
registry (BaseTokenizer) claude tool_result=408 thinking=418 mcp_tool_result=421
web_search_tool_result=421 document=422
```
**(3) Every documented behavior asserted against the running app** — 13
checks, all PASS: 400s for missing `messages`/`model`, invalid
`config.mode`, and all four invalid `frozen_message_count` forms; 200
for valid ones; non-dict `config` ignored; bypass and empty-messages
omit `transforms_summary`; success returns exactly the 8 documented
keys.
- **Not tested:** the docs site was not built (`docs/node_modules`
absent) — MDX was checked for balanced `<Callout>` tags only, so a
reviewer with the site running should eyeball rendering. No live
gateway/Kong request; verification is via `TestClient` against the real
ASGI app.
- **Note:** `HEADROOM_DISABLE_KOMPRESS` is read into `ProxyConfig` at
`server.py:4919` and by the CLI, not by `create_app(ProxyConfig(...))`
directly — I confirmed `disable_kompress=True` does reach the derived
pipeline.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
Matt Van Horn
|
789a4f3060
|
fix: normalize /p/<project> prefix on WebSocket upgrades so the Responses WS route is not rejected with 403 (#2379)
## Description
A Responses WebSocket upgrade to a project-prefixed URL
(`ws://127.0.0.1:8787/p/<project>/v1/responses`) was rejected with `403
Forbidden`, so the client fell back to HTTP transport. The `/p/<name>`
base-URL prefix is stripped by
`strip_project_path_prefix(request.scope)` inside
`@app.middleware("http")`, but Starlette runs `@app.middleware("http")`
for `http` scopes only, never `websocket` scopes. So an HTTP `POST
/p/<project>/v1/responses` has its prefix stripped and matches
`/v1/responses`, while the WS upgrade keeps the prefix, matches no
registered WebSocket route (`OPENAI_RESPONSES_WEBSOCKET_PATHS` are all
unprefixed), and Starlette rejects the unmatched WebSocket with `403`.
This normalizes the prefix for WebSocket scopes before routing so the
upgrade reaches the existing Responses WS handler and stays attributed
to the project.
Closes #2355
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/server.py` — added a small pure-ASGI
`WebSocketProjectPrefixMiddleware` (registered in `create_app`) that,
for `websocket` scopes only, strips the `/p/<name>` prefix via the
existing `strip_project_path_prefix` and binds the project context,
mirroring the HTTP middleware. HTTP and lifespan scopes pass through
untouched (no double-strip).
- `headroom/proxy/handlers/openai.py` — `handle_openai_responses_ws`
previously called `set_current_project(classify_project(ws_headers))`
unconditionally, clearing the middleware-bound project for prefix-only
clients (no `X-Headroom-Project` header). It now falls back to the
already-bound path-prefix project (`classify_project(ws_headers) or
get_current_project()`), so prefix-only WebSocket clients (aider,
Copilot BYOK, Cursor and other `/p/<name>` base-URL wraps) stay
attributed, exactly as on the HTTP path.
- `tests/test_provider_proxy_routes.py` — added a regression test.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_provider_proxy_routes.py -q
21 passed, 1 warning in 23.95s
$ ruff check headroom/proxy/server.py headroom/proxy/handlers/openai.py
All checks passed!
$ mypy --python-version 3.13 headroom/proxy/server.py headroom/proxy/handlers/openai.py
Success: no issues found in 2 source files
```
## Real Behavior Proof
- Environment: local, `uv` venv, Python 3.14, `uv run pytest`.
- Exact command / steps: added
`test_project_prefixed_openai_response_websocket_delegates_to_openai_ws_handler`,
which connects a WebSocket to `/p/test-project/v1/responses`.
- Observed result: the connection is accepted (no 403), the handler is
reached with the canonical `/v1/responses` path, and the request is
attributed to project `test-project`.
- Not tested: live end-to-end against a real upstream Responses
WebSocket server (validated via the routing/attribution regression test
only).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A — backend routing change with no user-facing UI.
## Additional Notes
Documentation checklist item is N/A: this is an internal routing fix
with no configuration or public-API surface change. The fix mirrors the
existing HTTP prefix-strip behavior so project-prefixed WebSocket
clients behave identically to their HTTP counterparts.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
|
||
|
Tejas Chopra
|
f9db5b5060
|
fix(proxy/openai): run tool-description compaction on chat-completions (#2741)
## Description
`HEADROOM_TOOL_DESC_MAX_CHARS` was wired into the Anthropic handler and
the Responses (Codex) handler, but never into **chat-completions** — so
the env var was a silent no-op for every chat client: opencode, Cline,
Aider, Roo, anything routed through LiteLLM.
Tool descriptions live on the `tools` array, which the message pipeline
never inspects, so no other pass was covering them.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- Run the L2 tool-description pass on the chat-completions path,
mirroring the block the Anthropic and Responses handlers already had.
- `compact_tool_descriptions` already walks both wire shapes — nested
`{"function": {"description": ...}}` for chat, flat for Responses — so
this is wiring, not a new codec.
- Chains after the existing schema compaction, seeding the token
"before" count only when that pass didn't, so the two compose instead of
double-counting.
- Labelled `openai:chat:tool_desc_compaction`, distinct from the
Anthropic and Responses labels so `headroom perf --by-transform` can
attribute it.
- Still opt-in and off by default: an unset env var leaves the tools
array — and therefore its cache prefix — byte-identical.
### Scope note: two adjacent "gaps" that turned out not to be
While surveying handler parity I flagged three missing chat-completions
transforms. Only one was real; recording the other two so nobody
re-opens them:
- **`tool_search_deferral` — correctly absent.** `{"type":
"tool_search"}` and `defer_loading` are Responses-API constructs, and
`_model_supports_openai_tool_search` gates them to `gpt-5.4+`. Injecting
that shape into a chat-completions request would be invalid, not an
improvement.
- **`system_prompt_compaction` — not applicable.** Anthropic needs a
dedicated pass because `system` is an out-of-band top-level field the
message pipeline never sees. On chat-completions the system prompt *is*
`messages[0]`, so it already reaches ContentRouter and is governed by
the existing `compress_system_messages` / `skip_system` gate. Wiring a
second path there would change system-prefix cache behavior for no new
coverage.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ .venv/bin/ruff check headroom/ tests/test_openai_chat_tool_desc_compaction.py --exclude headroom/dashboard/templates
All checks passed!
$ .venv/bin/mypy headroom/
Success: no issues found in 508 source files
$ python -m pytest tests/test_openai_chat_tool_desc_compaction.py tests/test_tool_schema_compaction.py \
tests/test_proxy_openai_cache_stability.py tests/test_openai_responses_context_compaction.py -q
49 passed in 16.59s
```
## Real Behavior Proof
- **Environment:** macOS 26.4 arm64, Python 3.12.6, repo `.venv`.
- **Exact command / steps:** ran `compact_tool_descriptions` at
`HEADROOM_TOOL_DESC_MAX_CHARS=30` against both wire shapes with the same
tool (a `read` tool with an 86-char description and a described `path`
param).
- **Observed result:**
```text
chat-completions (nested) modified=True bytes 272->215
responses (flat) modified=True bytes 259->202
```
Chat previously reported `modified=False` from the handler because the
pass was never invoked at all.
- **Not tested:** no live chat-completions request against a real
provider — the handler block is a thin adapter over
`compact_tool_descriptions`, and the regression was a missing *call*,
which the wiring test catches at source level. A full end-to-end drive
would need an upstream endpoint.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
Tejas Chopra
|
224578e80b
|
fix(kompress): reject artifacts that fail at run, and prefetch model files at startup (#2740)
## Description Three cold-start / robustness gaps found while debugging a user report of **0.12% savings across 722 requests** (49.8M input tokens, 60,920 saved). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made ### 1. The artifact fallback was unreachable for run-time failures `_create_onnx_session` tries `int8-wo` → `fp32` → `int8`, and its docstring describes exactly this scenario — but it only skipped a candidate when `InferenceSession(...)` **construction** threw. The int8 weight-only artifact carries `MatMulNBits` with `bits=8`. ORT's CPU kernel only handles 8-bit through the prepacked MLAS path, so a build or ISA without an 8-bit `SQNBitGemm` kernel falls into `ComputeBUnpacked`, which hard-asserts `nbits_ == 4`. That raises on `session.run()` **after** construction succeeded — so the fp32 candidate was never reached and ML compression was dead for the process lifetime. The reported log has 207 consecutive failures over three days. A two-token `_smoke_run` inside the existing candidate loop makes the fallback fire. `onnxruntime>=1.16.0` is unpinned, so which side of this an install lands on is a lottery. ### 2. A broken model cost an inference on every request, forever The per-request handler logged a `WARNING` and passed through with no latch — 207 identical lines that read as noise rather than "ML compression is dead". Now latches to passthrough after **3 consecutive** failures (any success resets the count) with one actionable `ERROR` naming the artifact override. ### 3. The model download began on the first request, not at startup #2001 was right to move Kompress off the startup path — on RHEL/CentOS 7-family hosts, entering cached native init before the port binds segfaults in `libarrow`/jemalloc with no Python traceback (#1908), which no `try/except` can catch. **This PR does not touch that.** But #2001 left the ~4-minute *download* on the first request, with every request in that window silently uncompressed behind one "model not ready" warning. Downloading is separable from loading. `prefetch_kompress_artifacts` resolves the files over plain `huggingface_hub` HTTP and never constructs an `InferenceSession` or imports `transformers`, so startup can prefetch bytes without touching the boundary #1908 crashes on. Native load stays deferred, status stays `deferred`, and a test asserts no session is constructed during prefetch. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ .venv/bin/ruff check headroom/ tests/test_kompress_failsafe.py tests/test_kompress_preload_deferral.py --exclude headroom/dashboard/templates All checks passed! $ .venv/bin/mypy headroom/ Success: no issues found in 508 source files $ python -m pytest tests/test_kompress_failsafe.py tests/test_kompress_preload_deferral.py \ tests/test_kompress_request_nonblocking.py tests/test_force_kompress_all.py \ tests/test_kompress_must_keep.py tests/test_proxy_disable_kompress.py \ tests/test_proxy_per_provider_kompress.py tests/test_proxy_warmup.py \ tests/test_proxy_eager_preload_bind.py -q 95 passed in 10.51s ``` ## Real Behavior Proof - **Environment:** macOS 26.4 arm64, Python 3.12.6, onnxruntime 1.21.1, repo `.venv`. **(1) Fallback chain, against the real HF repo:** ```text WARNING ONNX artifact 'onnx/kompress-int8-wo.onnx' from chopratejas/kompress-v2-base is unusable (... nbits_ == 4 was false ...); trying next candidate SESSION OK -> ['input_ids', 'attention_mask'] SMOKE RUN OK on the selected artifact ``` Also confirmed the default artifact really is 8-bit, by loading the cached blob: `{'bits': [8], 'block_size': [128]}`. **(2) Files-only prefetch, with `InferenceSession` patched to raise:** ```text INFO Kompress: prefetching model artifacts for chopratejas/kompress-v2-base ... prefetch ok=True in 0.08s, no session constructed ``` - **Not tested / important caveat:** the user's exact failure **cannot be reproduced on this machine**. On ORT 1.21.1 arm64 the int8-wo artifact fails at *construction* (`matmul_nbits.cc:115`), which the pre-existing load-only fallback already caught. Their build fails at *execution* (`matmul_nbits.cc:442`, `ComputeBUnpacked`). So the run-time path is pinned with a fake ORT session that constructs fine and then rejects `run()` — a mechanism test, not a reproduction of their build. Confirming the fix on their host needs their `onnxruntime` version. - **Not tested:** no RHEL/CentOS 7 host available to re-verify #1908 non-regression; the argument is structural (prefetch never constructs a session) and asserted by `test_prefetch_never_constructs_a_session_or_imports_transformers`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Tejas Chopra
|
8262a4a321
|
fix(stats): report one "Tokens Saved" headline across every harness (#2737)
## Description
The "tokens saved" figure a user sees depended on which harness they
ran. Headroom saves tool-definition tokens in two accounting shapes,
both legitimate, but the rule was never written down — so two harnesses
silently dropped savings and three surfaces open-coded the sum
differently.
- **Compaction** rewrites the tool array, so both endpoints are
countable → handlers fold the delta into
`original_tokens`/`optimized_tokens`, keeping `tok_before - tok_after ==
tok_saved` coherent.
- **Deferral / hook shrink** removes schemas `count_messages` never sees
→ can only be recorded as a tag, additive to `tokens_saved`.
`tool_schema_savings_policy` now owns the sum via
`headline_tokens_saved()`, and every reporting surface routes through
it.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
Producer gaps (both Anthropic — i.e. Claude Code, the primary harness):
- `anthropic:tool_schema_compaction` / `anthropic:tool_desc_compaction`
computed their savings, debug-logged them, and **discarded them**. Now
folded at the final recount, mirroring the OpenAI chat handler. A
14-tool array drops 786 tokens that previously reported `tok_saved=0`.
- Anthropic never wrote `turn_hook_tools_saved_tokens` at all, so a
turn-hook extension that shrinks tools got zero credit there while
OpenAI credited it. Now tagged.
Reporting gaps:
- `headroom perf` printed `Total saved (messages)` and `Tool saved` as
rival lines — on a tool-heavy session the headline read `0` and the real
win looked like a footnote. Now one `Tokens saved:` headline with a
messages/tool-schemas breakdown.
- `active_savings_percent` divided a **compression-only numerator** by a
denominator that already included compacted tool schema, undercounting
every tool-heavy session. Numerator is now all-layers, with deferred
schemas added to both sides.
- The headline and its percent now share a numerator. Previously the
dashboard tile showed an all-layers total next to a compression-only
percent.
- Session summary and dashboard tile relabelled to `Tokens Saved`; the
tool-schema panel is labelled as a component (`Tokens Saved · Tool
Schemas`) rather than a rival metric.
- `outcome.py` had two drifted inline copies of the tag sum; both now
call the policy module that exists for it. `total_saved=` added to the
PERF line.
- JSON: added `total_tokens_saved` / `total_savings_pct`; existing
`tokens_saved` / `tool_saved` / `savings_pct` keys unchanged for
back-compat.
Not changed by design: the Codex per-component attribution sub-line
would need a 9th positional tuple element threaded through 4 unpack
sites, and its headline is already correct without it.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ .venv/bin/ruff check headroom/ tests/test_tool_schema_savings_policy.py --exclude headroom/dashboard/templates
All checks passed!
$ .venv/bin/ruff format --check headroom/ tests/... --exclude headroom/dashboard/templates
510 files already formatted
$ .venv/bin/mypy headroom/
Success: no issues found in 508 source files
$ python -m pytest tests/test_tool_schema_savings_policy.py tests/test_cli_perf_format.py \
tests/test_request_outcome.py tests/test_savings_tool_search_aggregation.py \
tests/test_dashboard_token_savings.py tests/test_anthropic_compaction_transforms.py -q
70 passed, 1 warning in 6.15s
$ python -m pytest tests/test_handler_outcome_tag_invariant.py tests/test_cold_start_fast_pass.py \
tests/test_anthropic_ccr_workspace_unbound.py tests/test_anthropic_pre_upstream_backpressure.py \
tests/test_vertex_claude_compression.py tests/test_provider_route_specs.py -q
50 passed in 10.11s
$ python -m pytest tests/test_agent_savings.py tests/test_bundled_tools_savings.py \
tests/test_codex_ws_savings_deferral.py tests/test_savings_ledger_before_forwarded.py \
tests/test_savings_ledger_offload.py tests/test_proxy_savings_history.py \
tests/test_proxy_dashboard_stats_cache.py tests/test_output_savings_cli.py -q
97 passed, 2 skipped in 18.60s
$ python -m pytest tests/test_tool_schema_compaction.py tests/test_openai_responses_context_compaction.py \
tests/test_proxy_openai_cache_stability.py tests/test_codex_ws_compression_scheduler.py \
tests/test_proxy_streaming_request_logger.py -q
66 passed, 1 skipped in 16.69s
```
## Real Behavior Proof
- **Environment:** macOS 26.4 arm64, Python 3.12.6, repo `.venv`.
Motivated by a real user proxy log (0.33.0, `client=opencode` →
nano-gpt, 722 requests) reporting 0.12% savings.
- **Exact command / steps (1) — the Anthropic fold, real compaction +
real provider tokenizer:**
```python
tok = AnthropicProvider().get_token_counter("claude-sonnet-4-6")
payload = {"tools": [ ...14 tools with $schema/title/examples... ]}
body, modified, bb, ba = compact_tools(payload)
```
**Observed:**
```text
modified=True bytes 4503->2539 TOKENS 1650->864 delta=786
tok_before=6650 tok_after=5864 tok_saved=786 coherent=True
pre-fix: Claude Code reported tok_saved=0 and discarded 786 tokens
```
Pinned as `test_tool_schema_compaction_saves_real_tokens_not_just_bytes`
— it asserts a positive **token** delta (not just bytes), which is the
premise of folding at all.
- **Exact command / steps (2) — the report, on the reported session's
shape** (tool schemas carry the win, message compression is 0 because
everything routed to `excluded_tool`):
**Observed after:**
```text
Requests: 2
Tokens: 45,760 -> 45,760 (0.0% messages)
Tokens saved: 811 (1.7% reduction)
· messages 0
· tool schemas 811
JSON: {'total_tokens_saved': 811, 'total_savings_pct': 1.7, 'tokens_saved': 0,
'tool_saved': 811, 'savings_pct': 0.0}
```
Before, the same input printed `Total saved: 0 tokens (messages)` as the
headline with `Tool saved: 811` beneath it.
- **Not tested:** no live proxy run against a real provider — the
Anthropic fold is proven at the accounting layer (real `compact_tools` +
real provider tokenizer) and via the existing handler suites, not by an
end-to-end Claude Code session. Dashboard changes are template-label
edits verified by reading `stats.tokens.saved` / `by_layer.tool_search`
shapes, not by a browser screenshot.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
Tejas Chopra
|
3d23d76248
|
fix(kompress): let orgs run Kompress on their own inference stack (#2736)
## What this enables An org pulls the Kompress weights from HuggingFace, serves them on their own infrastructure, and points Headroom at it: ```bash HEADROOM_KOMPRESS_ENDPOINT=https://ml.internal.acme.com ``` No credential needed, no local ML dependencies, and original content never leaves their network (the CCR store stays proxy-local, so `headroom_retrieve` keeps working). ## The one thing that was actually broken Almost all of this already worked. The blocker was a hardcoded path: ```python self._url = endpoint.rstrip("/") + "/compress" ``` Real inference servers don't serve at `/compress`: | Stack | Path | |---|---| | TorchServe | `/predictions/kompress` | | KServe / Seldon | `/v1/models/kompress:predict` | | SageMaker | `/invocations` | Appending `/compress` to those 404s. And because remote Kompress **fails open**, that 404 is invisible — compression silently stops instead of erroring. The only workaround was standing up a reverse proxy purely to rename a path. ## Two new env vars, both defaulting to current behaviour | Var | Default | Purpose | |---|---|---| | `HEADROOM_KOMPRESS_ENDPOINT_PATH` | `/compress` | Set empty to use the endpoint URL verbatim | | `HEADROOM_KOMPRESS_ENDPOINT_HEADERS` | *(none)* | `k=v,k2=v2`, merged last so it can replace `Authorization` | Headers are applied after the token deliberately, so a gateway wanting `x-api-key` or `X-Tenant-Id` needs no separate auth-scheme setting. ## No regression With only `HEADROOM_KOMPRESS_ENDPOINT` set, the request is **byte-identical** to before — `POST <endpoint>/compress` with an optional Bearer token. Existing Modal deployments need no change. `os.environ.get` with a default distinguishes "unset" (use `/compress`) from an explicit empty value (endpoint is a complete URL), so the escape hatch can't fire by accident. The regression cases are deliberately the *first* tests in the new file. Verified through the real router wiring: ``` modal (today's config) -> https://acme--kompress.modal.run/compress modal + token -> …/compress {'authorization': 'Bearer tok'} self-hosted KServe (full URL) -> https://ml.acme.com/v1/models/kompress:predict self-hosted TorchServe (path) -> https://ts.acme.com/predictions/kompress self-hosted, x-api-key, no token -> …/compress {'x-api-key': 'k', 'x-tenant-id': 'acme'} ``` ## Documents the HTTP contract The endpoint contract was only discoverable by reading the source. Now in the module docstring: ``` request {"content": "<text>", "target_ratio": 0.5 | null} response {"compressed": "<text>", # REQUIRED "original_tokens": int, # optional, derived if absent "compressed_tokens": int, # optional "compression_ratio": float, # optional "model_used": str} # optional ``` `compressed` is the only required field, so a shim in front of an existing inference server is a few lines. Also logs the **resolved** URL at startup — with fail-open, a mistyped path otherwise manifests as nothing happening at all. ## Notes - `parse_endpoint_headers` reimplements the `HEADROOM_OTEL_METRICS_HEADERS` format rather than importing it: `observability.metrics` imports opentelemetry at module scope, and remote Kompress exists precisely so a proxy can run without heavy optional deps. - 27 new tests. Pre-existing unrelated flake in `test_content_router_single_item_deadline.py` (fails 3/3 on clean main). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Tejas Chopra
|
7c9b046595
|
fix(dashboard): serve tailwind/htmx/alpine locally instead of from CDNs (#2734)
## Description The dashboard loaded all three of its front-end dependencies from third-party CDNs at page load: ```html <script src="https://cdn.tailwindcss.com"></script> <script src="https://unpkg.com/htmx.org@1.9.10"></script> <script src="https://unpkg.com/alpinejs@3.13.3/dist/cdn.min.js" defer></script> ``` Microsoft Edge's Tracking Prevention classifies `unpkg.com` as a tracker and blocks it by default on Windows; locked-down corporate proxies block both hosts. On those machines none of the three scripts executed — no Tailwind CSS, no htmx polling, no Alpine bindings, plus an uncaught `ReferenceError: tailwind is not defined` from the inline `tailwind.config` assignment at `dashboard.html:21`. The dashboard rendered blank. Reported from a Windows user's console: ```text Tracking Prevention blocked access to storage for https://unpkg.com/htmx.org@1.9.10. Tracking Prevention blocked access to storage for https://unpkg.com/alpinejs@3.13.3/dist/cdn.min.js. ``` This vendors the three files and serves them from the proxy, so the dashboard has no external network dependency at all. Note for anyone triaging the same report: the `cdn.tailwindcss.com should not be used in production` line in that console output is **not** related. It is an unconditional `console.warn` in the Tailwind Play CDN build (no hostname guard), so it fires on every load, localhost included, and it still fires now that the bundle is self-hosted. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Vendored `headroom/dashboard/static/{tailwind.min.js,htmx.min.js,alpine.min.js}` — Tailwind Play CDN 3.4.17, htmx 1.9.10, Alpine 3.13.3, byte-for-byte as published. - `headroom/dashboard/__init__.py`: added `STATIC_DIR`. - `headroom/proxy/server.py`: mounted `/dashboard/static`, registered **before** `register_provider_routes`' catch-all so the asset requests are not tunneled to the wrapped upstream provider (same ordering constraint as the `/favicon.ico` route, GH #1787). `check_dir=False` so a missing assets directory 404s the dashboard JS rather than aborting proxy startup. - `headroom/dashboard/templates/{dashboard,settings}.html`: script `src` → `/dashboard/static/…`. - `NOTICE`: MIT / 0BSD attribution for the three vendored bundles. - `tests/test_dashboard_static_assets.py`: new. No packaging change needed — `[tool.maturin]` includes everything under `headroom/`, so the wheel picks the assets up. Wheel grows ~498 KB (407 KB of that is the Tailwind Play bundle). ## Testing - [x] Unit tests pass (`pytest`) — targeted, see note under *Not tested* - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_dashboard_static_assets.py tests/test_proxy_settings_endpoints.py -q tests/test_dashboard_static_assets.py ...... [ 21%] tests/test_proxy_settings_endpoints.py ...................... [100%] ============================== 28 passed in 4.47s ============================== $ ruff check . All checks passed! $ ruff format --check headroom/proxy/server.py headroom/dashboard/__init__.py tests/test_dashboard_static_assets.py 3 files already formatted $ mypy headroom Success: no issues found in 509 source files ``` ## Real Behavior Proof - **Environment:** macOS 15 (Darwin 25.4.0), Python 3.12.6, headless Chromium via Playwright, proxy served in-process with `create_app(ProxyConfig(optimize=False, cache_enabled=False, log_full_messages=True))` on `:8787`. - **Exact command / steps:** loaded `/dashboard` and `/dashboard/settings` with `wait_until="networkidle"`, then asserted the globals exist, that Tailwind actually generated CSS (computed style of a `px-3` element), and recorded every non-localhost request plus all `pageerror`/`console.error` events. - **Observed result:** ```text /dashboard | alpine: True | tailwind css: True | external: none | errors: none /dashboard/settings | alpine: True | tailwind css: True | external: none | errors: none /dashboard 200 text/html; charset=utf-8 191549 /dashboard/static/tailwind.min.js 200 text/javascript; charset=utf-8 407279 /dashboard/static/htmx.min.js 200 text/javascript; charset=utf-8 47755 /dashboard/static/alpine.min.js 200 text/javascript; charset=utf-8 43441 feed-toggle visible: True alpine loaded: True htmx: True tailwind: True tailwind applied (px-3 padding): 12px external hosts: none console errors: none ``` Zero external requests on either page, so the Edge/firewall failure mode is structurally gone rather than worked around. - **Not tested:** - No Windows machine available — the fix is verified as "makes zero external requests", which is the property the Windows failure depended on, but it has not been confirmed against Edge with Tracking Prevention on. Worth a check by someone on Windows before release. - Full `pytest` suite not run (targeted runs only); CI covers it. - `tests/test_dashboard/test_live_feed.py` still has 2 failures, both pre-existing and unrelated: those tests need a manually started proxy on `:8787` with `--log-messages`, and `test_live_feed_button_exists` asserts `is_visible()` with no wait for the `/stats` poll that flips `log_full_messages`. The other 2 in that file pass against this change, which is itself end-to-end evidence that Alpine and htmx work from the vendored bundles. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes - No issue number: reported directly rather than filed, so `Closes #` is omitted. Closed #22 ("Dashboard is not working") and closed #533 (Windows cp949 `get_dashboard_html()`) are different failures. - **Docs checklist item is N/A** — nothing user-facing changes; the dashboard URL and behaviour are identical. - Deliberately **not** switching to a real Tailwind CLI build. It would cut 407 KB to ~20 KB and silence the production warning, but it puts Node in the release path and silently leaves any class added to the 2,713-line template unstyled with no CI guard. The Play bundle behaves exactly as it does today, just served locally. Worth revisiting if wheel size becomes a problem (note the PyPI project-size ceiling). - Upgrades are now manual: bumping these three means re-downloading the files. Pinned versions are recorded in `NOTICE`. |
||
|
gglucass
|
a70e5ff78d
|
fix(learn): run project discovery off the event loop (#2731)
## Description
`TrafficLearner.flush_to_file` is a coroutine, but it called
`plugin.discover_projects()` inline. That function walks the filesystem
to decode escaped project directory names — in
`learn/plugins/claude.py`, `_greedy_path_decode` recurses through
`iterdir()` at every level and tries each tokenization of each child,
backtracking on a miss — so on a large home tree it runs for minutes.
Doing that on the event loop freezes uvicorn for the whole window. The
port keeps accepting TCP, but `/readyz` never answers, so a supervisor
health-checking the proxy kills a process that is merely busy.
Field thread dumps show exactly that:
```
Current thread (most recent call first):
File "python3.12/pathlib.py", line 1056 in iterdir
File "headroom/learn/plugins/claude.py", line 454 in _greedy_path_decode
File "headroom/learn/plugins/claude.py", line 478 in _greedy_path_decode
File "headroom/learn/plugins/claude.py", line 478 in _greedy_path_decode
File "headroom/learn/plugins/claude.py", line 426 in _decode_project_path
File "headroom/learn/plugins/claude.py", line 71 in discover_projects
File "headroom/memory/traffic_learner.py", line 591 in flush_to_file
File "headroom/memory/traffic_learner.py", line 535 in _flush_worker
File "python3.12/asyncio/events.py", line 88 in _run
File "python3.12/asyncio/base_events.py", line 1999 in _run_once
File "python3.12/asyncio/base_events.py", line 645 in run_forever
File "uvicorn/server.py", line 75 in run
File "headroom/proxy/server.py", line 4992 in run_server
```
Accompanying signals from the same incidents: port accepts TCP,
`/readyz` times out, process CPU 2-13s across the window (I/O bound, not
spinning), proxy log silent 66-336s.
## Type of Change
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/memory/traffic_learner.py`: `flush_to_file` now awaits
`asyncio.to_thread(plugin.discover_projects)` instead of calling it
inline. `asyncio` was already imported. The result is cached per learner
(`_project_roots_cache`), so the steady-state flush path pays nothing
for the thread hop.
- `tests/test_memory/test_traffic_learner.py`: added
`test_discover_projects_does_not_block_the_event_loop`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uv run --frozen --extra dev pytest tests/test_memory/test_traffic_learner.py -q
.................................................. [100%]
============================= 152 passed in 2.93s ==============================
$ uvx ruff check headroom/memory/traffic_learner.py tests/test_memory/test_traffic_learner.py
All checks passed!
$ uvx ruff format --check headroom/memory/traffic_learner.py tests/test_memory/test_traffic_learner.py
2 files already formatted
$ uv run --frozen --extra dev mypy headroom/memory/traffic_learner.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: macOS 15.6 arm64, Python 3.10.18, pytest 9.0.3, branch
off `main` @ `
|
||
|
Tejas Chopra
|
9cfb00838a
|
fix(telemetry): anonymous compression stats — no prompts, no data (#2728)
## In one line
Headroom starts reporting **how well compression is working** — counters
and percentages only. **No prompts. No code. No file paths. Nothing
about what you're building.**
## Why
Right now nobody knows whether compression actually helps real users.
You can see your own numbers in `/stats`, but that's it — there's no way
to tell whether a given workload compresses well, or why it sometimes
doesn't. This closes that loop so we can make compression better for
everyone.
## Exactly what gets sent
One message per session, and every 5 minutes while you're active:
```json
{
"session": { "id": "random", "turns": 47, "duration_s": 4210, "seq": 3 },
"tokens": { "original": 890000, "attempted": 410000, "saved": 320000,
"tool_saved": 48000, "cache_read": 210000 },
"rates": { "saved_pct": 35.96, "eligible_pct": 46.07, "yield_pct": 78.05,
"cache_read_pct": 23.60, "overhead_pct": 1.96 },
"compression": { "transforms": {"crush": 47}, "passthrough_turns": 0 },
"skips": {},
"sources": { "proxy": 47 },
"providers": ["anthropic"],
"models": ["claude-sonnet-4-5-20250929"],
"failures": 2
}
```
Plus a random install ID, the Headroom version, and OS/architecture
(`darwin`, `arm64`).
That's the whole thing. A full example lives at
`deploy/beacon/sample-event.json`.
## What is never sent
- Your prompts or the model's responses
- Your code
- File paths, project names, repo names
- Tool names or MCP server names
- Hostname, username, or IP address
- Custom or fine-tuned model names (an id like `ft:gpt-4o:acme-corp:…`
contains a company name, so only models in a public registry are
reported)
**This is structural, not a pinky-swear.** Every value in the payload is
a number, a fixed word, or a random ID — there is no free-text field
anywhere for content to hide in. The receiver
(`deploy/beacon/worker.js`, in this repo so you can read it) drops
anything not on an explicit allowlist before storing.
## Turning it off
Any one of these:
```bash
HEADROOM_BEACON=off # or
DO_NOT_TRACK=1 # or
# offline mode
```
It's on by default, and Headroom says so at startup:
```
Telemetry: anonymous compression stats — never prompts, code, or file paths.
Helps us improve compression | Off: HEADROOM_BEACON=off
```
`HEADROOM_TELEMETRY` is a **separate** switch that still only affects
local stats. If you had turned that on, this change does not start
uploading anything — you answered a different question, and upgrading
should not change the answer.
## Why the percentages, not just "tokens saved"
"We saved 36%" hides the interesting part. In the example above only
**46% of tokens were eligible** for compression at all — the rest is
frozen cache prefix and system prompts we deliberately do not touch. Of
what we *could* touch, we removed **78%**.
Those are two separate problems. Raising eligibility is proxy work;
raising yield is compressor work. A single number cannot tell us which
to fix.
## Coverage
`emit_request_outcome` is a single chokepoint —
`handler.metrics.record_request` is called from exactly one place,
inside the funnel — so all 30 `RequestOutcome` construction sites are
covered: Anthropic, OpenAI, Gemini, Bedrock, batch, streaming, and the
long-lived Codex Responses-WS path.
The `headroom_compress` MCP path bypassed that funnel and is now wired
in separately. It has a different shape (no provider, no upstream
latency, and everything handed to the tool is eligible by construction),
so `sources` counts turns by origin — MCP turns always read
`eligible_pct: 100` and must not drag the proxy's real eligibility
ceiling upward.
**Subagents.** All subagent traffic through the proxy merges into one
session, which is correct for savings and retention but means `turns`
conflates fan-out with depth. Fan-out is still derivable —
`compression.latency_ms_total / session.duration_s` gives the
concurrency ratio (~1x serial, ~4x for four parallel agents), so no
extra field is needed. Verified no lost updates under 6-way concurrency
(1,200 turns).
**Known gap:** `--workers N` gives each process its own aggregator, so
one user session becomes up to N. Token totals and fleet rates stay
correct; session counts inflate. This matches the existing documented
limitation that TOIN state, CostTracker, and the prefix tracker are all
per-process.
## Notes for reviewers
- **Cumulative snapshots, not deltas.** Every report restates running
totals under one session ID, so the highest `seq` per `(install,
session)` is the complete session. Dedupe is a window function, and a
lost report costs nothing.
- **Never breaks the proxy.** Every path swallows its own exceptions;
uploads go out on a daemon thread so nothing blocks the request loop.
- **Explicit User-Agent is load-bearing.** urllib's default is blocked
by Cloudflare (error 1010). Combined with fire-and-forget error
handling, that would have failed every upload while looking perfectly
healthy.
- **The exit flush was broken and is fixed.** `atexit` handed the POST
to a daemon thread, and daemon threads are killed before they finish
during interpreter shutdown — so nothing was sent. That silently dropped
*every session shorter than the 5-minute heartbeat*, plus all
short-lived subagent MCP processes. The exit path now posts
synchronously with a 2s timeout.
- Receiver and query tooling are in `deploy/beacon/`.
## Testing
- `python -m headroom.telemetry.session` self-check: dedupe, cumulative
totals, dropped-report recovery, payload contains no model id or
prompt-derived string, allowlist coverage
- 175 telemetry/outcome tests pass; 6 new ones cover the opt-out notice
- Verified end to end against a live deployment: client → receiver →
storage → query
## Still to do before release
The default endpoint currently points at a temporary `workers.dev` URL.
It needs to move to a Headroom-owned hostname before this ships in a
tagged release — noted inline at `DEFAULT_ENDPOINT`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
JD Davis
|
007446c73a
|
feat(copilot): proxy VS Code models transparently (#2687)
## Description Make Headroom a transparent proxy for VS Code GitHub Copilot. Users keep using Copilot's normal model picker—GPT-4.1, Claude Sonnet, Claude Opus, and other models in their entitlement—while Headroom silently forwards the selected model instead of registering or requiring a separate "Headroom" model. This also fixes GitHub's device OAuth exchange by sending form-encoded request bodies, matching the endpoint contract. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add `headroom wrap vscode` to start a Copilot-seeded subscription proxy and safely configure VS Code's shipped Copilot proxy override. - Add `headroom unwrap vscode` for reversible cleanup. - Preserve VS Code's selected model by changing only the proxy URL/auth override; no custom model is registered and no model preference is written. - Support stable VS Code settings locations on macOS, Windows, and Linux, plus `--settings-file` for Insiders, portable, and other installations. - Edit JSONC settings with a marker-owned block while preserving unrelated bytes, comments, ordering, and trailing commas. - Refuse malformed markers, invalid JSONC, or unmanaged existing Copilot overrides instead of overwriting user configuration. - Fix SIGINT cleanup so the managed settings block is removed and normal shutdown exits successfully. - Fix Copilot device OAuth start/poll requests to use `application/x-www-form-urlencoded`. - Add a compatibility matrix, setup/removal flow, credential behavior, remote-development guidance, enterprise notes, troubleshooting, and verification documentation. ## Testing - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ .venv/bin/pytest -q tests/test_provider_copilot_vscode_config.py tests/test_cli/test_wrap_vscode.py tests/test_cli/test_wrap_helpers.py tests/test_copilot_auth.py tests/test_cli/test_copilot_auth.py tests/test_provider_copilot_wrap.py tests/test_cli/test_wrap_copilot.py tests/test_copilot_provider_label.py tests/test_copilot_subscription_smoke.py 244 passed in 0.59s $ .venv/bin/ruff check <changed Python files and tests> All checks passed! $ .venv/bin/mypy headroom/providers/copilot/vscode.py Success: no issues found in 1 source file $ cd docs && npm run types:check fumadocs-mdx && next typegen && tsc --noEmit # exited 0 $ git diff --check # exited 0 ``` The full 10,179-test suite was also sampled through approximately 83%, but was stopped because of its runtime. It exposed existing failures in `test_recover_codex.py`, `test_wrap_stale_marker.py`, and `test_proxy_health.py`; therefore the broad `pytest`, repository-wide Ruff, and repository-wide mypy boxes are intentionally not checked. ## Real Behavior Proof - Environment: macOS arm64, VS Code 1.131.0, built-in GitHub Copilot 0.59.0, Headroom 0.33.1-dev. - Exact command / steps: 1. Completed `headroom copilot login` with GitHub's device flow. 2. Ran `.venv/bin/headroom wrap vscode --port 8788`. 3. Confirmed VS Code retained its ordinary Copilot model catalog and made `GET /models` through Headroom with `GitHubCopilotChat/0.59.0` and `editor-version: vscode/1.131.0`. 4. Sent native Copilot `/p/headroom/chat/completions` requests through the same endpoint using `gpt-4.1`, `claude-sonnet-4.6`, and `claude-opus-4.7`. - Observed result: - All three completion requests returned HTTP 200. - GPT-4.1 resolved upstream to `gpt-4.1-2025-04-14`; Sonnet and Opus retained their exact selected IDs. - All returned the requested exact marker content. - VS Code's settings contained only the Headroom proxy URL and token auth override—no Headroom model or model-selection setting. - The proxy health endpoint remained ready with `openai_api_url` set to `https://api.githubcopilot.com`. - Not tested: - Physical Windows or Linux hosts (their path/config behavior is covered by unit tests). - WSL, dev containers, SSH remotes, VS Code Insiders, or enterprise Copilot deployments end-to-end. - Every model in the live Copilot catalog. - A fully submitted chat from VS Code's UI automation; the real extension's catalog request and native completion paths were verified separately. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing targeted unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable; this integration intentionally has no separate UI or model entry. ## Additional Notes The integration uses VS Code Copilot's shipped advanced/debug proxy endpoint seam. The managed settings block is deliberately narrow and reversible. Remote extension hosts may need their own reachable proxy/configuration as documented. --------- Co-authored-by: JerrettDavis <2610199+JerrettDavis@users.noreply.github.com> |
||
|
AxelRay
|
56b3e4c1b1
|
fix(proxy): skip OpenAI tool_search deferral for Codex client (#2729)
## Description
OpenAI Responses tool_search deferral injects `defer_loading` and a
`tool_search` tool for eligible gpt-5.4+ requests. When the model later
calls a deferred tool, the function_call item carries a `namespace`
field. Codex CLI round-trip structs drop unknown fields, so the next
request omits `namespace` and OpenAI returns 400, killing the session
mid-run. Proxy logs for these Codex turns show no tool savings, so the
injection breaks Codex without benefit.
This skips OpenAI tool_search deferral when the classified client is
Codex, leaving other clients unchanged.
Closes #2726
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Add `openai_tool_search_client_supported` and a Codex-only unsupported
client set
- Pass optional `client` into `inject_tool_search_deferral_openai` and
no-op for Codex
- Plumb `client` through Responses compression (HTTP, WebSocket,
passthrough) with legacy-signature retries
- Add regression tests for Codex skip and non-Codex still injects
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ .venv/bin/python -m pytest tests/test_openai_tool_search_deferral.py -q -o addopts=
34 passed, 1 warning in 0.27s
$ .venv/bin/python -m ruff check headroom/proxy/helpers.py headroom/proxy/handlers/openai.py tests/test_openai_tool_search_deferral.py
All checks passed!
$ .venv/bin/python -m ruff format --check headroom/proxy/helpers.py headroom/proxy/handlers/openai.py tests/test_openai_tool_search_deferral.py
3 files already formatted
```
## Real Behavior Proof
- Environment: Linux x86_64, Python 3.14.5 in repo .venv, shallow
checkout of headroomlabs-ai/headroom main at
|
||
|
Parideboy
|
01df245252
|
fix(proxy/cost): mark estimated-basis budget records and add an enforcement policy (#2713) (#2725)
## Description
`CostTracker.check_budget()` is a hard spend control — the Anthropic
handler refuses the request with a 429 once the period budget is gone.
The ledger that control reads could not tell a measured dollar from a
guessed one.
When a provider response carries no input-token breakdown,
`record_tokens()` substitutes Headroom's own `tokens_sent` estimate for
the input count so input cost isn't silently dropped from the budget.
That fallback is the right call, but the resulting record was
byte-identical to a provider-measured one: no field, no log line, no
separation in `/stats`. `RequestOutcome.uncached_input_tokens` defaults
to `0`, so any route whose response omits usage lands on this branch in
production. An estimate can drift in either direction, so a budget check
could pass after real spend had already gone over — with nothing saying
the decision rested on an estimate.
This keeps the fallback and makes it visible, then lets operators decide
what an estimate is allowed to do to a hard limit.
Closes #2713
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- New `headroom/proxy/budget_basis_policy.py` (pure policy module,
matching the existing `*_policy.py` convention): the
`measured`/`estimated` basis constants, the `count`/`ignore`/`block`
policy values, and `resolve_estimated_basis_policy()` (explicit value →
`HEADROOM_BUDGET_ESTIMATED_BASIS` → `count`; an unknown value warns once
and falls back rather than failing proxy startup).
- `headroom/proxy/cost.py`: ledger entries are now `CostEntry(timestamp,
cost_usd, basis)` instead of a bare tuple; `record_tokens()` marks the
fallback branch `estimated` and logs one WARNING per model (deduped the
same way pricing warnings are, per #2504 — an unguarded warning on this
path fires once per request for a provider that never reports usage);
new `period_cost_breakdown()` and an optional `basis` filter on
`get_period_cost()`; new `budget_denial_detail()` builds the 429 body
where the ledger lives; `check_budget()` honors the policy while keeping
its `(allowed, remaining)` signature.
- `stats()` gains `budget_estimated_basis` (the active policy) and
`budget_basis` (the period split: `total_usd`, `measured_usd`,
`estimated_usd`, `estimated_pct`, `records`, `estimated_records`).
`merge_cost_stats()` already spreads `**cost_stats`, so both reach
`/stats["cost"]` with no extra plumbing.
- Operator knob wired through every config layer:
`ProxyConfig.budget_estimated_basis` (`models.py`), the Click
`--budget-estimated-basis` option with `envvar=` (`cli/proxy.py`), the
argparse `--budget-estimated-basis` flag (`server.py`, `default=None` so
the env var stays reachable), and a `SettingField` in the `Budget` group
(`settings_store.py`).
- `headroom/proxy/handlers/anthropic.py`: the 429 body now comes from
`budget_denial_detail()`, which names how much of the period's spend was
booked from an estimate and distinguishes "you overspent" from "I refuse
to enforce a hard limit on a guess".
- `headroom/cli/doctor.py`: the budget check stays **PASS** and appends
the estimated share (and the policy, when it isn't the default). No new
WARN state — a provider that never reports usage would otherwise sit at
a permanent WARN. Every new read is `.get()` + type-guarded so `doctor`
still works against an older running proxy.
- `docs/content/docs/metrics.mdx`: a "Measured vs Estimated Spend"
subsection with the `/stats` shape and the three policy values.
- Tests: new `tests/test_cost_budget_basis.py` (20 tests) plus 4 new
`doctor` tests; `tests/test_anthropic_pre_upstream_backpressure.py`'s
cost-tracker double gained `budget_denial_detail()` to match the
handler's duck-typed contract.
### Policy values
| `HEADROOM_BUDGET_ESTIMATED_BASIS` | Effect on the hard limit |
|---|---|
| `count` (default) | Unchanged behavior — estimated spend consumes the
budget. |
| `ignore` | Booked and reported, but only measured spend enforces. |
| `block` | Fail closed — refuse rather than enforce a hard limit on a
guess. |
Default enforcement is unchanged. `CHANGELOG.md` is untouched.
## Testing
- [x] Unit tests pass (`pytest`) — every test covering the changed
modules; see `Not tested` for this machine's pre-existing environment
failures
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`) — clean on every file this
PR touches
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_cost_budget_basis.py tests/test_cost_tracker_counterfactual.py tests/test_cost_pricing_warning_dedup.py -q
31 passed
$ python -m pytest tests/test_cli_doctor.py -q
72 passed
$ python -m pytest tests/test_anthropic_pre_upstream_backpressure.py -q
25 passed
$ python -m pytest tests/test_proxy_settings_endpoints.py tests/test_proxy/test_settings_store.py -q
50 passed
# full suite (see "Not tested" below for the excluded modules and the pre-existing failures)
$ python -m pytest -q
...
tests\test_cost_budget_basis.py .................... [ 25%]
tests\test_cost_pricing_warning_dedup.py ... [ 25%]
tests\test_cost_tracker_counterfactual.py ........ [ 25%]
...
217 failed, 8807 passed, 657 skipped, 5318 warnings, 59 errors in 716.30s (0:11:56)
# same failing files re-run on clean upstream/main with the change stashed -> identical count
$ git stash push -u -- headroom tests docs
$ python -m pytest tests/test_log_compressor.py tests/test_cache/test_client_integration.py \
tests/test_fsutil.py tests/test_savings_ledger.py tests/test_ccr_mcp_http.py \
tests/test_router_registry_dispatch.py tests/test_proxy_savings_history.py \
tests/test_text_compressors.py tests/test_builtin_compressor_adapters.py \
tests/test_cli_proxy_env.py -q
73 failed, 182 passed in 34.82s # 40+16+2+2+1+1+1+4+3+3 = 73, matching the run above
$ python -m mypy headroom --ignore-missing-imports --python-version 3.13
# 12 errors, all in release_version.py / ccr/mcp_server.py / memory/mcp_server.py
# (stale local `mcp` stubs) — none in any file this PR touches
$ python -m ruff check headroom/proxy/budget_basis_policy.py headroom/proxy/cost.py headroom/proxy/server.py headroom/proxy/models.py headroom/proxy/handlers/anthropic.py headroom/cli/proxy.py headroom/cli/doctor.py headroom/settings_store.py tests/test_cost_budget_basis.py tests/test_cli_doctor.py tests/test_anthropic_pre_upstream_backpressure.py
All checks passed!
$ python -m ruff format --check <same 11 files>
11 files already formatted
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.13.11, branch
`fix/budget-estimated-basis-2713` off `upstream/main` @ `
|
||
|
Chester
|
3eb0122068
|
fix(learn): filter ambient user-role scaffolding (#2275)
## Description Fixes #2274. Headroom Learn currently trusts `role=user` as sufficient preference provenance. Agent harnesses can transport ambient UI and orchestration context in user-role messages, and OpenAI Responses normalization also promotes missing roles to `user`. Correction-like text in those inputs can therefore become durable user preferences. This change keeps preference learning fail-closed for known non-user sources while preserving genuine user corrections. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Refactoring only ## Changes Made - Preserve missing OpenAI Responses roles as `unknown` instead of promoting them to `user`. - Canonicalize user-role text before preference extraction. - Remove proxy-appended `## Relevant Memories` suffixes from preference evidence. - Reject strict ambient-only harness prefixes such as heartbeat, environment, workspace-instruction, delegation, and app-context envelopes. - Apply the same guard in `on_messages` and `_extract_preferences` for defense in depth. - Add regression coverage for system/developer/unknown roles, ambient-only user messages, memory-only messages, and mixed genuine-user-plus-memory input. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text 149 passed, 1 warning ruff check: passed ruff format --check: passed git diff --check: passed ``` Focused test files: ```text tests/test_memory/test_traffic_learner.py tests/test_openai_responses_traffic_learner.py ``` ## Real Behavior Proof - Environment: macOS; Python 3.13; current Headroom main; direct invocation of the real `TrafficLearner` class, with no proxy or database mocks - Exact command / steps: create `TrafficLearner(backend=None, min_evidence=1)`; feed system, developer, heartbeat user-role, and memory-only user-role messages; read `patterns_extracted`; feed a genuine user correction followed by a `## Relevant Memories` suffix; read `patterns_extracted` again - Observed result: `ambient_patterns=0`, `after_user_patterns=1` — the ambient batch produced no preference evidence; the genuine correction produced one pattern, while the appended memory content did not become evidence - Not tested: live provider traffic against a remote OpenAI endpoint; every possible third-party harness envelope; migration or cleanup of already-persisted noisy memories ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] No new dependency - [x] Fail-open proxy behavior is unchanged - [x] Regression tests added - [x] Public examples contain no real user data - [x] CHANGELOG update, if requested (not requested — N/A) ## Additional Notes This extends the source filtering introduced by #466 rather than replacing it. The prefix checks are deliberately strict and anchored at the start of a canonicalized message. The intended failure mode is a missed preference, not durable storage of non-user instructions. Note: the strict prefix set was discussed and confirmed in JerrettDavis's review approvals. |
||
|
Rod Boev
|
232fb49c73
|
fix(proxy): route Codex Live voice through a dedicated /v1/live transport (#2709)
## Description Codex Live traffic currently reaches an unrouted WebSocket path and receives HTTP 403 before the proxy can contact an upstream. Closes #2653 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring ## Changes Made - Add a dedicated `/v1/live` WebSocket route family and transparent transport. - Preserve subscription auth routing, account headers, origin policy, subprotocols, and text/binary frame bytes. - Propagate WebSocket close metadata and cancel relay tasks deterministically on every exit. - Keep Live outside the Responses parser, compression, memory injection, and Responses beta-header path. - Keep generic HTTP paths on the existing catch-all and document the Live aliases plus the derived-path override. - Add real-app route, relay, and loopback integration proof. - Add coverage for authorization fallback, defensive receive events, and cancellation cleanup in the Live relay. ## Testing The focused Live handshake, preservation suites, Ruff, format, and diff checks pass. The base comparison, Codex Desktop owner round trip, and ChatGPT backend acceptance of the derived `/backend-api/codex/live` path remain untested. - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy headroom --ignore-missing-imports`) - [x] New tests added for the reported failure - [x] Manual loopback testing performed ### Test Output ```text uv run pytest tests/test_codex_live.py -q: 6 passed, 7 warnings in 11.03s uv run pytest tests/test_provider_proxy_routes.py tests/test_proxy_codex_route_aliases.py tests/test_provider_codex_endpoints.py tests/test_openai_codex_routing.py -q: 53 passed in 90.54s (0:01:30) uv run ruff check headroom/providers/codex/live.py headroom/providers/proxy_routes.py headroom/proxy/handlers/openai.py headroom/proxy/ws_headers.py tests/test_codex_live.py: All checks passed uv run ruff format --check headroom/providers/codex/live.py headroom/providers/proxy_routes.py headroom/proxy/handlers/openai.py headroom/proxy/ws_headers.py tests/test_codex_live.py: 5 files already formatted uv run mypy headroom --ignore-missing-imports: Success: no issues found in 508 source files git diff --check: pass ``` ## Real Behavior Proof The local WebSocket integration floor uses real uvicorn, a real WebSocket client, and a real loopback WebSocket upstream. The base 403 comparison was not run. Head observes HTTP 101 on every Live alias and a byte-identical binary frame relay. - Environment: Windows, CPython 3.13, the Headroom proxy test environment. - Exact command / steps: run the focused Live test against the local uvicorn proxy and loopback WebSocket upstream, then run the preservation suite listed in `Test Output`. - Observed result: all four Live aliases return HTTP 101, negotiate `codex.live.v1`, preserve text and binary frames, and pass the preservation suite. - Not tested: Codex Desktop Live session; ChatGPT backend acceptance of `/backend-api/codex/live`; the base 403 comparison. ## Review Readiness - Live has a separate transport and does not enter Responses handling. - Existing Responses and generic passthrough suites remain preservation gates. - No `CHANGELOG.md` or install/crate changes are included. - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] Closes #2653 - [x] Real loopback handshake and binary-frame proof required - [x] No audio payload logging - [x] No unqualified end-to-end claim ## Screenshots Not applicable. ## Additional Notes The upstream Live path is derived from the repository’s Codex URL formula and remains explicitly unconfirmed until owner evidence is available. |
||
|
Parideboy
|
3a27c4dacb
|
fix(proxy/debug): reconcile Kompress warmup state in /debug/warmup (#2711)
## Description
`/debug/warmup` serialized the warmup registry verbatim, so a Kompress
slot left at the startup snapshot kept reporting `{"status": "null",
"info": {"source_status": "deferred"}}` forever — even while the ONNX
model was loaded and actively compressing.
`/health` and `/readyz` already fix this: #2402 added
`_reconcile_kompress_health()`, which promotes the slot from live
runtime state. The debug route never called it, so its answer depended
on whether a health probe happened to run first. That is the half of
#2624 still reproducing on `main`.
Second defect: `WarmupSlot.mark_loaded()` only *updates* `info`, so the
startup-planted `source_status: "deferred"` survived promotion and the
slot serialized as the self-contradictory `{"status": "loaded", "info":
{"source_status": "deferred", "backend": "onnx"}}`.
Closes #2624
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/server.py`: call the existing
`_reconcile_kompress_health()` in the `/debug/warmup` route before
serializing the registry. The reconciler never instantiates a compressor
and never calls `preload()` / `ensure_background_load()` / `compress()`
— it only reads `is_ready()` / `ready_backend()` on an already resident
instance, or falls back to the module-level ONNX cache — so the endpoint
stays side-effect free and idempotent.
- `headroom/proxy/server.py`: stamp `source_status="runtime"` at both
`mark_loaded()` promotion sites in `_reconcile_kompress_health()` (the
resident-compressor path and the `_kompress_cache` fallback),
overwriting the stale startup marker.
- `tests/test_proxy_debug_endpoints.py`: three regression tests plus a
read-only compressor stub whose `preload` / `ensure_background_load`
raise, so a future change that makes the debug route trigger a load
fails loudly.
- `tests/test_proxy_health.py`: assert the promoted slot's
`info["source_status"] == "runtime"`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ pytest tests/test_proxy_debug_endpoints.py tests/test_proxy_health.py tests/test_proxy_warmup.py -q
tests\test_proxy_debug_endpoints.py ............................. [ 52%]
tests\test_proxy_health.py ................. [ 83%]
tests\test_proxy_warmup.py ......... [100%]
============================= 55 passed in 36.56s =============================
$ ruff check headroom/proxy/server.py tests/test_proxy_debug_endpoints.py tests/test_proxy_health.py
All checks passed!
$ ruff format --check headroom/proxy/server.py tests/test_proxy_debug_endpoints.py tests/test_proxy_health.py
3 files already formatted
$ mypy headroom --ignore-missing-imports
Success: no issues found in 506 source files
```
The three new tests were confirmed to be genuine regression tests: with
the `server.py` change reverted and the tests kept, all three fail.
```text
$ git stash push -- headroom/proxy/server.py && pytest tests/test_proxy_debug_endpoints.py -q -k kompress
FAILED tests/test_proxy_debug_endpoints.py::test_debug_warmup_promotes_deferred_kompress_after_runtime_load
FAILED tests/test_proxy_debug_endpoints.py::test_debug_warmup_keeps_pending_kompress_null
FAILED tests/test_proxy_debug_endpoints.py::test_debug_warmup_never_starts_kompress_loading
====================== 3 failed, 26 deselected in 3.98s =======================
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.13.11, pytest 9.1.1, ruff 0.15.x,
mypy 1.20.2, branch based on `main` at
|
||
|
Raúl
|
9ce5af02b1
|
test(recover-codex): bind AF_UNIX socket via short relative path (#2396)
## Description `test_recovery_records_sockets_and_secures_both_backups` binds a Unix domain socket at its absolute path under pytest's `tmp_path`. On macOS the `AF_UNIX` `sun_path` limit (~104 bytes) is shorter than that path, so `bind()` raises `OSError: AF_UNIX path too long` and the test fails locally. It stays green on CI Linux only because `/tmp`-rooted temp paths there are short enough. Bind a short relative name from inside `source` instead; the socket is still created at `source/codex.sock` and the recovery scan behaves identically. Closes #2394 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `tests/test_cli/test_recover_codex.py`: in `test_recovery_records_sockets_and_secures_both_backups`, `monkeypatch.chdir` into `source` and `bind(socket_path.name)` (a short relative name) instead of `bind(str(socket_path))` (a long absolute path). Added the `monkeypatch` fixture to the signature and a one-line comment explaining the `sun_path` cap. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy`) - [x] Manual testing performed ### Test Output ```text # BEFORE (on this macOS box, at the branch base): $ uv run pytest -q \ "tests/test_cli/test_recover_codex.py::test_recovery_records_sockets_and_secures_both_backups" tests/test_cli/test_recover_codex.py:777: in test_recovery_records_sockets_and_secures_both_backups codex_socket.bind(str(socket_path)) E OSError: AF_UNIX path too long 1 failed in 0.70s # AFTER (whole file, no regressions): $ uv run pytest -q tests/test_cli/test_recover_codex.py 31 passed in 0.72s $ uv run ruff format --check tests/test_cli/test_recover_codex.py 1 file already formatted $ uv run ruff check tests/test_cli/test_recover_codex.py All checks passed! $ uv run mypy tests/test_cli/test_recover_codex.py Success: no issues found in 1 source file ``` <img width="1073" height="200" alt="image" src="https://github.com/user-attachments/assets/82155e76-3005-4c4f-93f4-4802ae5e7405" /> ## Real Behavior Proof - Environment: macOS 26.5 (darwin 25.5.0), Python 3.13.7, ruff 0.14.14, `tempfile.gettempdir()` = `/var/folders/.../T` (48 chars, before the `pytest-of-*/pytest-N/test_.../headroom-codex-home-broken/codex.sock` suffix, which pushes the absolute `sun_path` over the macOS ~104-byte cap). - Exact command / steps: run the focused test at the branch base (fails with `AF_UNIX path too long`), apply the one-line relative-bind change, re-run the whole file. - Observed result: before = 1 failed; after = 31 passed. `ruff`/`mypy` clean. - Not tested: Linux/Windows (the test is `skipif` on win32 / no `AF_UNIX`; on Linux it already passed pre-change because temp paths are short). No production code touched, so no proxy/runtime behavior was re-validated. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (N/A: test-only) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works (this IS the corrected test; it fails before and passes after) - [x] New and existing unit tests pass locally with my changes - [x] I did not edit `CHANGELOG.md` ## Additional Notes - Pure test-portability fix; no production behavior change. `test:` type keeps it out of the release-please changelog, which is correct for a test-only change. |
||
|
Tejas Chopra
|
3e348f327f
|
fix(ccr): stop persisting retrieval markers as original content (#2694) (#2703)
## Description CCR entries could end up holding a `<<ccr:...>>` marker — or nothing at all — where the original bytes belonged, so `headroom_retrieve(hash)` answered with the very placeholder the caller was trying to resolve. For a base64/credential field that is permanent, silent data loss: the inner marker's hash is the only handle on the real payload, and it disappears from anywhere the model can see. Four sites, one root cause — **a compressed intermediate (or nothing) was stored in place of the source**, the same defect class as #1209 (tag placeholders persisted as originals). Closes #2694 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **`compaction/walker.rs`** — `walk_array` compacted through the store-LESS `compact()`. Opaque cells inside a compacted table got a marker whose payload was **never written**, so retrieval 404'd forever. Now uses `compact_with_store` so the emitted hash resolves. - **`compaction/classifier.rs`** — nothing stopped an already-marked string from being offloaded a second time, which stashed the MARKER as the new entry's "original". Marker-bearing text is our own output, not source content, so it is never classified opaque. One guard at the choke point both the walker and the table compactor share. - **`smart_crusher/crusher.rs`** — on the prose-hook path the row-drop marker hashed and stored rows whose leaves were **already** rewritten (prose compressed, blobs marker-substituted), so retrieving dropped rows returned compressed output. Now hashes and stashes the pre-processing array via `crush_array_with_source`. - **`content_router.py`** — compression pinning matched only `Retrieve more: hash=` / `Retrieve original: hash=`, **not** `<<ccr:`, so opaque-blob output was readmitted to the compressor on a later turn — the path that feeds the corruption above. Consolidated into `_is_already_compressed()` and applied at all three pinning sites. - **`cache/compression_store.py`** — store-level guard: refuse to persist a *bare* marker as `original_content` and log at ERROR, so a future producer regression surfaces loudly instead of silently converting "retrievable" into "gone". Deliberately narrow — originals may legally *contain* markers (nested offloads); only a bare marker is rejected. - **Regression tests** — `test_nested_table_markers_resolve_to_source_bytes` (asserts payloads are verbatim-retrievable, not merely that a marker was emitted) and `test_already_marked_content_is_not_re_offloaded`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ cargo build -p headroom-core Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 37s $ cargo test -p headroom-core --lib smart_crusher test result: ok. 329 passed; 0 failed; 0 ignored; 0 measured; 583 filtered out; finished in 0.19s $ python -m pytest tests/test_transforms/test_smart_crusher_ccr_roundtrip.py -q 16 passed in 0.68s $ python -m pytest tests/test_ccr_row_drop_store_bridge.py tests/test_ccr_tool_injection.py -q 50 passed in 12.52s $ python -m pytest tests/test_compression_store.py tests/test_lossless_mode.py -q 100 passed in 13.14s $ ruff check headroom/transforms/content_router.py headroom/cache/compression_store.py \ tests/test_transforms/test_smart_crusher_ccr_roundtrip.py All checks passed! $ mypy headroom/transforms/content_router.py headroom/cache/compression_store.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - **Environment:** macOS (Darwin 25.4.0), Python 3.12.6, headroom-ai 0.33.0 editable, `HEADROOM_CCR_BACKEND=memory`, Rust extension rebuilt via `maturin develop --release`. - **Exact command / steps:** compact a nested document — 5 rows whose `detail` field is a stringified sub-array of 6 base64 blobs (1600 B each) — then, for every `<<ccr:HASH>>` marker in the output, call `ccr_get(HASH)` and check the payload is the verbatim source rather than a marker. ```python inner = [{"k": f"key{i}", "v": i, "tok": blob(1200)} for i in range(6)] doc = {"rows": [{"id": i, "detail": json.dumps(inner), "note": "x"} for i in range(5)]} out = SmartCrusher().compact_document_json(json.dumps(doc)) for h in re.findall(r"<<ccr:([0-9a-f]+)", out): payload = crusher.ccr_get(h) # must be real bytes, not a marker ``` - **Observed result — BEFORE (on `main`):** all six payloads collapsed into a single dead marker. The rendered sub-table was re-classified opaque (`html`, because `<<` reads as a tag), offloaded again, and its payload never stored — so the six inner hashes were erased from the visible text *and* the outer hash resolved to nothing. ```text {"rows":"[5]{detail:string,id:int,note:string} <<ccr:3fb1d44933da,html,289B>>,0,x <<ccr:3fb1d44933da,html,289B>>,1,x ..."} 3fb1d44933da -> RUST MISS # unrecoverable — 6 × 1600 B gone ``` - **Observed result — AFTER (this branch):** the sub-table stays inline, each blob keeps its own marker, and every marker resolves to verbatim source. ```text {"rows":"[5]{detail:string,id:int,note:string} \"[6]{k:string,tok:string,v:int} key0,\"\"<<ccr:955b1fed2ef7,base64,1.6KB>>\"\",0 ..."} 6ad5846997f4: resolves, len=1600, is-verbatim-source=True 78a0bd9364a7: resolves, len=1600, is-verbatim-source=True 955b1fed2ef7: resolves, len=1600, is-verbatim-source=True a0cef69da7f0: resolves, len=1600, is-verbatim-source=True dfcde5e940c0: resolves, len=1600, is-verbatim-source=True e57c4e0a3ce8: resolves, len=1600, is-verbatim-source=True RESULT: PASS — every marker resolves to real source bytes ``` ## Notes for reviewers - The issue also reports **function words dropped from retained prose** (`is`, `a`, `the`) and **interleaved log output corrupting `headroom doctor`'s table borders**. Those are separate defects on different paths (extractive prose compression and log-handler buffering respectively) and are **not** addressed here — this PR is scoped to the CCR store/retrieve corruption. They should be tracked separately; the prose one overlaps #2586. - The `crusher.rs` prose-hook fix is on the Rust pipeline (`json_offload`) rather than the Python proxy path, but it is the same store-the-intermediate bug and was cheap to close while in the file. |
||
|
Raúl
|
1a2688b57f
|
test(kompress): close patch-coverage gaps from #2716 (#2721)
## Description Codecov flagged 9 uncovered lines on #2716 after it merged: `hf_entry_known_absent`'s own body in `headroom/onnx_runtime.py` was only ever exercised indirectly (every existing test in `tests/test_transforms/test_kompress_compressor.py` monkeypatched it away rather than calling the real implementation), and `_load_pytorch_weights` / `_load_kompress_pytorch` in `headroom/transforms/kompress_compressor.py` had three untested branches: the double cache-miss under `allow_download=False` (merged.pt confirmed absent AND the plain fallback also not cached), a genuine non-404 download failure propagating instead of silently falling back, and the already-cached fast path in `_load_kompress_pytorch`. ## Type of Change - [ ] Bug fix - [ ] New feature - [x] Test coverage improvement, no production code change ## Changes Made - `tests/test_onnx_runtime.py`: added `_write_fake_hf_cache` (builds a minimal on-disk HF hub cache layout, including the `.no_exist/<hash>/<filename>` marker huggingface_hub writes after a real 404) and three direct tests of `hf_entry_known_absent` against the real `huggingface_hub.try_to_load_from_cache`, not a mock of it. - `tests/test_transforms/test_kompress_compressor.py`: added `test_cache_only_raises_when_confirmed_absent_but_plain_also_missing`, `test_genuine_download_failure_propagates_instead_of_falling_back`, and a new `TestLoadKompressPytorchCaching` class covering the already-cached fast path. ## Testing ```text $ .venv/bin/python3 -m pytest tests/test_onnx_runtime.py tests/test_transforms/test_kompress_compressor.py -q 51 passed $ .venv/bin/python3 -m pytest tests/ -k "kompress or onnx_runtime" -q --cov=headroom.transforms.kompress_compressor --cov=headroom.onnx_runtime --cov-report=term-missing # before: onnx_runtime.py Missing includes 132-136 (hf_entry_known_absent's entire body); # kompress_compressor.py Missing includes 805-806, 818, 836 # after: none of those lines appear in Missing anymore 191 passed, 7 skipped $ .venv/bin/python3 -m ruff format --check tests/test_onnx_runtime.py tests/test_transforms/test_kompress_compressor.py 2 files already formatted $ .venv/bin/python3 -m ruff check tests/test_onnx_runtime.py tests/test_transforms/test_kompress_compressor.py All checks passed! ``` ## Review Readiness - Test-only, additive diff (113 insertions, 0 deletions, 0 lines touched outside the two test files). No behavior change possible. ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] New and existing unit tests pass locally with my changes - [x] I did not edit `CHANGELOG.md` ## Additional Notes Not closing: the remaining branch-partial on the `device == "auto"` cuda/mps/cpu selection in `_load_kompress_pytorch` (would need mocking `torch.cuda.is_available()` / `torch.backends.mps.is_available()` for marginal benefit); left as-is. |
||
|
Tejas Chopra
|
2797099bec
|
feat(compress): accept config.frozen_message_count on /v1/compress (#2718)
## Problem
Callers that resend a growing conversation every turn — agent loops
generally, and an in-progress Strands plugin specifically — cannot keep
a stable prompt-cache prefix through `/v1/compress`. The router
compresses older messages more aggressively as the conversation grows,
so their bytes change and the provider's cache misses from the first
rewritten message onward. That trades a 90% read discount for nothing.
Measured before this change, compressing the same conversation at
increasing lengths and comparing the first 16 messages against the
4-turn baseline:
```
turns msgs first 16 still byte-identical?
4 16 16/16 OK
8 32 16/16 OK
16 64 12/16 DRIFT at [2, 6, 10, 14]
32 128 12/16 DRIFT at [2, 6, 10, 14]
64 256 12/16 DRIFT at [2, 6, 10, 14]
```
Indices 2/6/10/14 are the tool-result messages.
## Fix
`TransformPipeline` and `ContentRouter` already honour
`frozen_message_count` — `content_router.py` skips any message below the
index, and `pipeline.py` even logs *"freezing first N/M messages (prefix
cached by provider)"*. It was simply missing from this endpoint's
`config` parsing, so no HTTP caller could reach it.
This adds it alongside the existing `compress_user_messages` /
`target_ratio` / `protect_recent` / `protect_analysis_context` options.
Pinning still lets cross-message transforms such as dedup *read* the
prefix; it only forbids rewriting it. `protect_recent` guards the
opposite end of the list and cannot express this.
Invalid values return 400, matching the existing `config.mode`
validation. `bool` is rejected explicitly, since `isinstance(True, int)`
is `True` in Python and a JSON `true` silently becoming
`frozen_message_count=1` would be a nasty surprise.
## Verified against a live proxy
```
sent 64 messages, pinned first 32 -> returned unchanged: True
unpinned tail still compressed: True
simulated agent loop carrying the compressed prefix forward:
4 turns: prefix of 0 held 16 turns: prefix of 48 held
8 turns: prefix of 16 held 24 turns: prefix of 64 held
12 turns: prefix of 32 held 32 turns: prefix of 96 held
prefix never drifted across the whole run: True
```
## Tests
13 new tests in `TestCompressEndpointFrozenMessageCount`, including the
regression itself: compress a 6-turn and a 24-turn conversation with the
same pin and assert the prefix is identical. 48/48 pass in
`test_proxy_compress_endpoint.py` (35 pre-existing, unchanged). ruff and
ruff-format clean.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Raúl
|
46da91b2f1
|
fix(kompress): load merged.pt for the v2 checkpoint instead of the unmerged PEFT safetensors (#2716)
# PR draft: fix(kompress): load merged.pt for the v2 checkpoint instead of the unmerged PEFT safetensors Branch: `rnoz/fix-kompress-merged-checkpoint` (off `upstream/main`). Two commits. Issue: https://github.com/headroomlabs-ai/headroom/issues/2714 (filed, open). --- ## Description `_load_kompress_pytorch` in `headroom/transforms/kompress_compressor.py` downloaded `model.safetensors` from the default model repo `chopratejas/kompress-v2-base` and loaded it with `strict=False`, discarding the missing/unexpected key report. That file is the unmerged PEFT checkpoint (encoder keys prefixed `encoder.base_model.model...`), which never matches `HeadroomCompressorModel`'s plain `encoder.*` keys. The LoRA-adapted encoder weights were silently dropped while `token_head`/`span_conv` happened to match and loaded fine, so the model ran with a stock, non-adapted `answerdotai/ModernBERT-base` encoder feeding correctly trained decision heads, with no error and a healthy status reported everywhere. `scripts/export_kompress_v2_onnx.py` already documents this exact mismatch and loads the correct `merged.pt` sub-state-dicts for its own export path. This PR mirrors that same loading logic into the runtime PyTorch loader, with a fallback to the plain `model.safetensors` format for repos that never shipped a `merged.pt` (verified against the v1 `chopratejas/kompress-base` repo via the public HF API, which has no `merged.pt`). Both paths now check the missing/unexpected key report and raise instead of silently proceeding on a mismatch. A second commit fixes a gap an adversarial review caught in the first: the cache-only (`allow_download=False`, startup preload) path could not tell "this repo genuinely has no merged.pt" apart from "merged.pt exists but is not downloaded yet", so it would have fallen back to a stale `model.safetensors` left over from before this fix on exactly the upgrade path this PR is meant to close. It now uses `huggingface_hub`'s own `.no_exist` cache marker (via a new `hf_entry_known_absent()` helper in `headroom/onnx_runtime.py`) to make that distinction without a network call, and only falls back when absence is confirmed; otherwise it raises `KompressModelNotCached` so the caller defers instead of guessing. Closes #2714 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/transforms/kompress_compressor.py`: added `_load_merged_state_dict`, `_load_plain_state_dict`, and `_load_pytorch_weights`, replacing the inline `model.safetensors` download + `load_state_dict(strict=False)` call in `_load_kompress_pytorch`. `merged.pt` is tried first; the plain format is only used when its absence is confirmed. - `headroom/onnx_runtime.py`: added `hf_entry_known_absent()`, a thin wrapper around `huggingface_hub.try_to_load_from_cache()` that reads the on-disk `.no_exist` marker HF writes after a real 404, so cache-only code can distinguish "confirmed absent" from "never checked" without hitting the network. - `tests/test_transforms/test_kompress_compressor.py`: added `TestPytorchWeightLoading` (8 tests) covering the merged-checkpoint happy path, missing-section and key-mismatch failures, the plain-format fallback for repos without `merged.pt`, and the cache-only ambiguity fix (confirmed-absent vs unconfirmed). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] Manual testing performed (real download against the live `chopratejas/kompress-v2-base` repo, not just mocks) ### Test Output ```text $ .venv/bin/python3 -m pytest tests/ -k kompress -q 178 passed, 7 skipped in 22.39s $ .venv/bin/python3 -m ruff check headroom/onnx_runtime.py headroom/transforms/kompress_compressor.py tests/test_transforms/test_kompress_compressor.py All checks passed! $ .venv/bin/python3 -m ruff format --check headroom/onnx_runtime.py headroom/transforms/kompress_compressor.py tests/test_transforms/test_kompress_compressor.py 3 files already formatted $ .venv/bin/python3 -m mypy headroom/onnx_runtime.py headroom/transforms/kompress_compressor.py Success: no issues found in 2 source files ``` ## Real Behavior Proof Ran the actual fixed loader against the live `chopratejas/kompress-v2-base` HF repo (not a mock), before and after each fix: ```text # BEFORE (parsed the real cached model.safetensors header by hand, no safetensors lib needed): total tensors: 316 encoder.base_model.model.embeddings.norm.weight encoder.base_model.model.embeddings.tok_embeddings.weight ...(all 310 encoder tensors share this prefix)... span_conv.0.bias / span_conv.0.weight / span_conv.2.bias / span_conv.2.weight token_head.bias / token_head.weight exact prefix match count with plain "encoder.<rest>": 0 # This confirms the pre-fix code's model.load_state_dict(state_dict, strict=False) # silently dropped every encoder weight (0 keys match HeadroomCompressorModel.encoder), # while token_head/span_conv happened to match and loaded. # AFTER (commit 1, real merged.pt download + load): $ .venv/bin/python3 -c " import headroom.transforms.kompress_compressor as kmod model = kmod._get_model_class()() kmod._load_pytorch_weights(model, 'chopratejas/kompress-v2-base', allow_download=True) print('SUCCESS: 0 missing/unexpected keys across all three sections') " SUCCESS: 0 missing/unexpected keys across all three sections # AFTER (full pipeline, real end-to-end compression through the public API): $ .venv/bin/python3 -c " import headroom.transforms.kompress_compressor as kmod compressor = kmod.KompressCompressor() result = compressor.compress(sample_traceback_plus_boilerplate_text) print(result.original_tokens, result.compressed_tokens, result.tokens_saved) print('ValueError: bad input' in result.compressed) " 497 454 43 True # must-keep line (the actual error) survived compression # AFTER (commit 2, cache-only ambiguity): unit tests # test_cache_only_defers_instead_of_using_stale_plain_checkpoint: PASSED # test_cache_only_uses_plain_checkpoint_when_merged_pt_confirmed_absent: PASSED ``` ## Review Readiness - [x] I have performed a self-review - [x] An independent adversarial review pass was run on both commits before this PR was opened; its one finding (the cache-only ambiguity) is fixed in commit 2, verified with new regression tests - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (N/A: internal loader behavior, no public API or config surface changed) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective (regression tests for both the original silent-drop bug and the cache-only ambiguity found in review) - [x] New and existing unit tests pass locally with my changes - [x] I did not edit `CHANGELOG.md` ## Additional Notes - Both `merged.pt` and the plain `model.safetensors` fallback now raise loudly on any state-dict mismatch instead of proceeding with partially-loaded weights, closing the general silent-failure class this bug belonged to, not just this one instance of it. - No other call sites of `_load_kompress_pytorch` or its removed inline code exist; its public signature is unchanged. |
||
|
Parideboy
|
6d5516dcb8
|
feat(code): add PHP support to CodeAwareCompressor (#2423)
## Description Adds PHP to `CodeAwareCompressor`, fixing #201. PHP was already *detected* as code (Magika labels in `headroom/compression/detector.py` include `php`, and the Rust `magika_detector.rs` lists it too) but there was no PHP `LangConfig`, so PHP content silently passed through uncompressed. This wires PHP through the tree-sitter compression path following the C# pattern (the most recently added, fully functional language — deliberately not the quarantined Perl path). A secondary detection bug is fixed along the way: PHP's `$variables` match Perl's prefilter regex, and the existing Perl-dominance guard in `detect_language` returned `UNKNOWN` for PHP files. An explicit `<?php` open tag — which no Perl source contains — now drops Perl from the candidate set before that guard runs. ## Type of Change - [ ] Bug fix - [x] New feature - [ ] Documentation update - [ ] Refactor - [ ] Other ## Changes Made - `headroom/transforms/code_compressor.py`: `CodeLanguage.PHP` + `phtml`/`php5`/`php7`/`php8` aliases; PHP `LangConfig` built from the actual tree-sitter-php grammar (node names verified by parsing samples): `namespace_use_declaration` imports, `function_definition`/`method_declaration` functions, `class_declaration`/`interface_declaration`/`trait_declaration` classes, `enum_declaration` types, `declaration_list` class bodies, `compound_statement` function bodies. `namespace_definition` maps to `package_node` so statement-scoped `namespace App;` hoists ahead of the `use` imports (required PHP ordering); the rare block-scoped `namespace A { }` form takes the same path and is preserved verbatim — valid output, just no compression inside the block. PHP prefilter regexes added; supported-languages error message updated; `<?php`-tag Perl disambiguation in `detect_language`. - `headroom/transforms/content_detector.py`: `php` entry in `_CODE_PATTERNS` so raw PHP classifies as `SOURCE_CODE` and reaches the code-aware route. - `tests/test_transforms/test_code_compressor.py`: new `TestPhpSupport` mirroring `TestCSharpSupport` — signatures preserved / bodies elided, `<?php` → `namespace` → `use` → declarations ordering, auto-detection despite the Perl sigil overlap, alias coercion, malformed passthrough. - `tests/test_code_compressor_language_alias.py`: `php` in the canonical list, `phtml` in the alias table. - `docs/content/docs/code-compression.mdx`: PHP added to the Tier 2 supported-languages row. No new dependency: `tree-sitter-language-pack` (the existing `[code]` extra) already ships the PHP grammar. No Rust changes needed. ## Testing - [x] New unit tests added and passing - [x] Full affected test suites pass locally **Test Output** ``` $ python -m pytest tests/test_transforms/test_code_compressor.py tests/test_code_compressor_language_alias.py -q ============================= 120 passed in 7.81s ============================= $ python -m pytest tests/test_transforms/ -q 3 failed, 443 passed # the 3 failures (kompress ONNX thread caps, kompress size gate, # text_crusher unicode parity) reproduce identically on a clean # upstream/main checkout in this environment — pre-existing local # ONNX runtime quirks, unrelated to this change $ ruff check . (0.15.17, CI-pinned) → All checks passed! | ruff format --check → clean $ mypy headroom/transforms/code_compressor.py headroom/transforms/content_detector.py --ignore-missing-imports Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13, tree-sitter + tree-sitter-language-pack (<1.0) installed, branch `feat/201-php-code-compression` off `upstream/main`. - Exact command / steps: parsed PHP samples (namespaced class w/ methods, block-scoped namespace, mixed HTML+PHP) with `tree_sitter_language_pack.get_parser('php')` to verify every node name used in the config; then ran `CodeAwareCompressor().compress(php_code, language="php")` and `compress(php_code)` (auto-detection) on a 48-line realistic service class. - Observed result: explicit and auto-detected paths both return `language=CodeLanguage.PHP`, `compression_ratio=0.64`, `syntax_valid=True`; method bodies elided to `// [N lines omitted]` while `<?php`, `namespace`, `use` lines, class header, and all signatures are preserved verbatim in the original order. Before the detection fix, auto-detection returned `UNKNOWN` (Perl prefilter dominance) — reproduced and then verified fixed. - Not tested: exotic PHP shapes (heredoc-heavy code, attributes `#[...]` on methods, interleaved multi-`<?php ?>` HTML templates beyond the basic mixed case); these fall back to verbatim preservation via the uncaptured-node pass or malformed-passthrough, both of which are covered by tests for the simple cases. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Tejas Chopra
|
b7a79ac31a
|
refactor: remove the dead headroom/prediction module (#2692)
## Description Deletes `headroom/prediction/` — 2,614 LOC of LLM output-length prediction feature extraction that was never wired into anything and shipped in every platform wheel regardless. It was added on 2026-01-26 in ` |
||
|
Tejas Chopra
|
f2c48e26c6
|
feat(compress): reach the lossless provider seam on the general path and default /v1/compress to marker-free output (#2691)
## Description Two related changes to the compression seams, plus the review fixes for both. Supersedes #2661 and #2662, which are closed in favour of this branch — the fixes are inseparable from the code they fix, so reviewing them together is cheaper than landing two PRs and patching them afterwards. **1. A registered lossless provider now competes on the general path.** The `headroom.transforms.lossless_provider` seam was only ever consulted from `_lossless_compact_excluded`, gated on `DEFAULT_EXCLUDE_TOOLS` (`config.py:216` — `Read/Grep/Glob/Write/Edit/WebSearch/WebFetch`). Gateway traffic carries the caller's own tool names — LiteLLM's `headroom` guardrail (https://docs.litellm.ai/docs/proxy/headroom) posts requests containing tools like `search_docs` / `run_ci` / `fetch_rows` — so a registered provider was structurally unreachable for every gateway/sidecar deployment. The seam existed; nothing could get to it. **2. `POST /v1/compress` is marker-free by default.** A CCR marker is only useful to a caller that also injects the `headroom_retrieve` tool AND can reach `/v1/retrieve`. Neither holds here: tool injection lives in the provider request handlers (`handlers/anthropic.py:1894`), never in `handle_compress`; and every `/v1/retrieve*` route is `Depends(_require_loopback)` (`server.py:4422, 4470, 4749, 4781`) with no remote opt-in — `HEADROOM_COMPRESS_ALLOW_REMOTE` drops the loopback dependency on `/v1/compress` only. So a gateway forwards a `Retrieve more: hash=…` pointer the model cannot follow, and the proxy pays a CCR store write nobody reads. `config.mode="ccr"` opts back in. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [x] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made ### Seam — `content_router.py`, `lossless_provider.py` - `_lossless_first` (STAGE 0, every block on every path) consults `get_lossless_provider()` and keeps whichever output is smaller. **Strict no-op when no provider is registered**, which is the default; and because it is best-of rather than authoritative, a provider can never do worse than the built-in folds. - Malformed provider output can no longer escape. Every shape check runs inside the `try`: result must be `None`, or a 2-element tuple/list of two `str`. Anything else is ignored at debug level. (Previously the unpack sat outside the `try`, so a 3-tuple raised `ValueError` up through `TransformPipeline.apply`, which re-raises.) - Empty / whitespace-only candidates are rejected rather than silently replacing block content. - **Providers are never offered diff content.** Diff folding is subtractive with no inverse check and a reflowed hunk breaks `git apply` — the same reason the built-in `diff` fold is restricted at `content_router.py:2481`. - The third-party `kind` label is sanitised against `^[a-z0-9_]{1,32}$` before reaching `transforms_applied` and the per-strategy metric dicts, so a caller-controlled string cannot explode Prometheus label cardinality. `fullmatch`, not `match`: `$` also matches before a trailing newline, which would put a newline in a label. - `set_lossless_provider(provider, *, verifier=None)` — in lossless-only mode, where STAGE 0's output is final and there is no marker to recover from, a registered verifier must confirm the fold or the candidate is dropped. No verifier registered = today's behaviour. `provider=None` clears both. - The provider is invoked once per block, not twice (`_has_lossless_fold` probes `_lossless_first` and discards the result, then STAGE 0 recomputes). Bounded memo, wholesale clear on overflow, no lock — a race costs one redundant fold. The memo keys on the provider registration generation, so registering or clearing a provider after a block was already folded takes effect. - The seam docstring now records that the provider runs on the general path and inside the parallel compression pool, so it must be thread-safe as well as deterministic. ### Route — `handlers/openai.py`, `server.py` - `_derived_compress_pipeline(key, **overrides)` replaces the copy-pasted pipeline-derivation block; `_no_ccr_pipeline` (the new default) and `_lossy_inline_pipeline` both use it. - The default pipeline is **built at startup** and included in `_eager_preload_transforms`, so a fresh pod does not pay ContentRouter construction and compressor load on its first request, inside the compression-executor budget. - An unrecognised `config.mode` returns 400 naming the valid values instead of silently falling back to the default. - Claude-family model names resolve their context limit from the Anthropic provider. Real divergence: `bedrock/anthropic.claude-3-5-sonnet` is 200000 there and 128000 on the OpenAI provider. The tokenizer still comes from the OpenAI pipeline's provider — a separate, larger change, noted in a comment. - Documents why the derived router deliberately does **not** share the base router's compression cache: keys do not encode CCR-marker mode, so sharing would leak marker-laden entries into the marker-free path. ### Behavior change A `/v1/compress` caller that relied on default markers now gets none. The only in-tree caller that can resolve them is the TypeScript SDK (`sdk/typescript/src/client.ts:398 retrieve`, `:422 handleToolCall`); it needs `config: {"mode": "ccr"}` to keep today's behaviour, and landing that SDK default in the same release would leave only gateway callers — for whom markers were never resolvable — seeing a difference. No `HEADROOM_COMPRESS_DEFAULT_MODE` compat env deliberately: a flag nobody sets becomes permanent debt, and the wire-level `mode` already covers the one caller that needs it. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_lossless_first_dispatch.py tests/test_lossless_excluded_compaction.py \ tests/test_lossless_mode.py tests/test_lossless_then_lossy.py tests/test_lossless_diff_fold_guard.py \ tests/test_bash_search_lossless_fold.py tests/test_proxy_compress_endpoint.py \ tests/test_ccr_row_drop_store_bridge.py tests/test_gateway_sidecar_ports.py tests/test_compress_api.py \ tests/test_platform_stabilization_functional.py tests/test_proxy_eager_preload_bind.py \ tests/test_proxy_warmup.py tests/test_router_registry_smartcrusher.py -q 189 passed in 32.04s $ ruff check <all 9 changed files> All checks passed! $ ruff format --check <all 9 changed files> 9 files already formatted $ mypy headroom/transforms/content_router.py headroom/transforms/lossless_provider.py \ headroom/proxy/handlers/openai.py headroom/proxy/server.py Success: no issues found in 4 source files ``` New tests cover, one concern each: every malformed provider shape; empty and whitespace-only results; diff content never reaching a provider (call-recording); `kind` sanitisation including the trailing-newline case; the verifier accepting / rejecting / raising; clearing a provider clearing its verifier; single provider invocation per block; memo invalidation on re-registration; unknown and valid `mode` values; the default pipeline existing before any request; and Claude vs OpenAI context-limit resolution with `token_budget` precedence preserved. ## Real Behavior Proof - **Environment:** macOS arm64, Python 3.12.6, proxy built from `_proxy_config_from_env()` with the default `coding` savings profile; Kompress both disabled and offloaded to a remote `kompress-v2-base` endpoint; `HEADROOM_COMPRESSION_TIMEOUT_SECONDS=300`. - **Steps:** `POST /v1/compress` over `TestClient` with OpenAI-shaped payloads under non-excluded tool names (`run_ci`, `list_files`, `code_search`, `fetch_rows`) — a CI log with ANSI escapes and repeated lines, a 160-path listing, a 150-line grep dump, a 150-row JSON array; plus a second payload with a RAG user blob, a 200-row JSON tool result and a 300-line log. Ran with and without a provider registered via `set_lossless_provider`. - **Observed — seam reachability:** | Kompress | no provider registered | provider registered | |---|---|---| | off | 19,284 → 10,265 tokens (46.8%) | 19,284 → **7,879 (59.1%)** | | on (remote) | 19,284 → 9,366 tokens (51.4%) | 19,284 → **7,146 (62.9%)** | Before this change the right-hand column was identical to the left — the registered provider was never called on this payload. - **Observed — marker-free default costs nothing:** | Config | tokens | saved | |---|---|---| | markers on (previous default) | 37,791 → 24,415 | 35.4% | | markers off (new default) | 37,791 → 24,415 | **35.4% — identical** | | `mode="lossy_inline"` | 37,791 → 25,129 | 33.5% | | `--lossless` | 37,791 → 35,100 | 7.1% | - **Observed — memo staleness, before the fix:** registering a provider that folds a grep block to 5 bytes left the block at its 1496-byte built-in fold, and clearing a provider kept serving the provider's output. Both correct after keying on the registration generation. - **Observed — context limit:** `bedrock/anthropic.claude-3-5-sonnet` resolves 200000 via the Anthropic provider, 128000 via the OpenAI provider. - **Not tested:** `tests/test_transforms_content_router.py` was not run — it does not complete on this machine, wedging on its 5th test while that test passes in 5.7s alone. Verified pre-existing before this work: with the diff stashed, the clean tree stalled at the identical test, and it stalled the same way under `HF_HUB_OFFLINE=1 HEADROOM_OFFLINE=1`. The machine was also out of disk at the time, which may be the real cause rather than the suspected native-detector deadlock (#575) — worth a separate issue either way. CI should be the arbiter here. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Two pre-existing test files needed adjusting, both direct consequences rather than scope creep: - `test_platform_stabilization_functional.py::test_v1_compress_success_reports_actual_metrics` patches `openai_pipeline.apply`, which the default mode no longer routes through. **This test was already failing on the marker-free-default commit** before any of the fixes — my original test selection missed it. It now patches the pipeline the route actually uses. - `test_proxy_eager_preload_bind.py` substitutes fake pipelines to control exactly what the preload walks; the eager build injected the real derived router's statuses into an exact-equality assertion. Its shared helper now clears the derived cache, preserving each test's intent without weakening an assertion. Docs unchecked — follow-ups worth doing in the same release: document `config.mode` values in `docs/content/docs/litellm.mdx` and `wiki/proxy.md`; the TS SDK `mode:"ccr"` default; and an operational note that `COMPRESSION_TIMEOUT_SECONDS` defaults to 30 (`helpers.py:687`) while a remote ML endpoint makes one sequential call per unit — on a large payload it trips the executor timeout and the handler fails open, returning `compression_skipped: true` with `tokens_before: 0`, which reads as "nothing to save" rather than "we gave up". Those zeroed counters are misleading and worth a separate fix. |
||
|
Tejas Chopra
|
e0ce4b1d48
|
fix: remove rtk and lean-ctx CLI context tools (#2677)
## Description Removes both third-party CLI context tools — **rtk** and **lean-ctx** — and with them the context-tool selector itself. Headroom no longer downloads, installs or configures either one, and there is no replacement. The previous pass (#2344) gated only three entry points inside `headroom/cli/wrap.py`. That left the feature reachable in practice: | Gap | Effect | |---|---| | `scripts/install.sh:1544`, `install.ps1:1681` | Ran `rtk init --global --auto-patch` from bash/PowerShell, **bypassing the Python gate entirely** — `curl \| sh` still wrote a Claude Code `PreToolUse` hook regardless of `HEADROOM_RTK` | | `wrap.py` `_setup_context_tool_for_agent` | **`wrap openhands` was broken by default**: `rtk_required=True` met a gate returning `None` → `SystemExit(1)`. Invisible because all 8 openhands tests patched `_ensure_rtk_binary` to a fake path | | `proxy/helpers.py`, `subscription/tracker.py` | Proxy shelled out to `rtk gain` from `/stats`, the dashboard and `headroom perf`; the tracker polled it per contribution (`_RTK_WIRING_DEFAULT = "enabled"`) | | No cleanup path | Nothing removed artifacts an earlier default had installed, so a machine that once ran the old default kept rtk in the loop forever (#1669, #1955) | Also worth noting: the rtk binary download had **no SHA or signature verification** — only `rtk --version` as a smoke test. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [x] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made **Removed** — `headroom/rtk/` and `headroom/lean_ctx/` packages, `headroom/cli/wrap_rtk_metrics.py`, `_selected_context_tool` / `_setup_context_tool_for_agent` / `_VALID_CONTEXT_TOOLS`, the `--rtk` / `--no-rtk` / `--no-project-rtk` / `--keep-rtk` flags across all 18 wrap subcommands, `HEADROOM_RTK*`, the proxy-side `rtk gain` polling, the dashboard CLI-filtering panel (rows + all 8 `cliFiltering*` Alpine getters), `paths.rtk_path()` / `lean_ctx_path()`, the SDK path helpers, `benchmarks/rtk_loop_learn_eval.py`, and the `headroom/rtk/**` CI path filters. **Fails loudly, not silently** — `--context-tool` / `--no-context-tool` / `HEADROOM_CONTEXT_TOOL` are kept solely to error out. They live in shell profiles, aliases and CI jobs, and accepting them as a no-op would read as Headroom having quietly stopped working. The installers reject them too, which matters more than it looks: their arg parsers forward the first unknown flag **and everything after it** to the wrapped tool, so a leftover `--no-rtk` would have silently swallowed a following `--port` and then been ignored downstream. **New `headroom/context_tool_cleanup.py`** — deleting the code cannot help a machine that already ran the old default, since the hooks, binaries and injected guidance are durable on disk. `purge_context_tool_artifacts()` runs once per `wrap`/`unwrap` and removes the registered hook entries, the generated hook scripts, the Headroom-managed `~/.local/bin` symlinks, the vendored `~/.headroom/bin/{rtk,lean-ctx}` binaries, the `lean-ctx` MCP server entry and the marker-fenced instruction blocks. Deliberately conservative: idempotent, **skips** a malformed config rather than overwriting it, and only unlinks a symlink resolving inside Headroom's own bin dir so a user's own build is untouched. It reports on **stderr**, because `wrap/unwrap openclaw --prepare-only` emit machine-readable JSON on stdout as their entire contract. Skipped for `wrap selfheal` (runs from a SessionStart hook; must not race Claude Code's writer for `~/.claude.json`) and for `--help`, which must stay read-only. **Client-config hardening** (discovered while investigating a "corrupted Serena settings file" report) — `wrap.py` reset a settings file to `{}` when an existing file would not parse, then wrote that back. One hand-edited typo or a transient `EACCES`/`EINTR` on a valid file destroyed the user's `permissions`, `env` and `hooks`, on **every `headroom wrap claude`**. It now refuses to write. Separately, `fsutil.write_text` is now atomic (temp file + `fsync` + `os.replace`), fixing all 14 non-atomic client-config writes at once; it follows symlinks rather than replacing them (dotfile managers) and preserves an existing file's mode. **Deliberately kept** — `rtk` stays in the wrapper-peel list in `transforms/content_router.py`. It sits beside `sudo`/`env`/`timeout` as shell-command grammar, so `rtk cat f` is still classified as a file read for anyone running their own rtk install, which the purge intentionally leaves alone. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check headroom/ tests/ e2e/ --exclude headroom/dashboard/templates All checks passed! $ ruff format --check headroom/ tests/ e2e/ --exclude headroom/dashboard/templates 1255 files already formatted $ mypy headroom/ Success: no issues found in 508 source files $ pytest tests/test_context_tool_cleanup.py -q 11 passed $ pytest tests/test_fsutil.py -q 12 passed $ pytest tests/test_cli/test_wrap_codex.py -q # 89 tests 89 passed in 431.68s $ pytest tests/test_cli/test_wrap_opencode.py -q 39 passed in 257.46s $ pytest tests/test_cli/test_wrap_helpers.py -q 45 passed $ pytest tests/test_paths.py -q 75 passed $ pytest tests/test_cli/test_unwrap_claude.py -q 14 passed $ pytest tests/test_proxy_savings_history.py -q 39 passed $ pytest tests/test_cli/test_wrap_copilot.py -q 27 passed $ pytest tests/test_cli/test_wrap_zcode.py -q 20 passed $ pytest tests/test_subscription_tracker.py -q 9 passed $ pytest tests/test_proxy_dashboard_stats_cache.py -q 5 passed, 1 skipped ``` Repo-wide grep for 14 removed symbols (`headroom.rtk`, `headroom.lean_ctx`, `_ensure_rtk_binary`, `_selected_context_tool`, `_get_context_tool_stats`, `rtk_path`, `lean_ctx_path`, `wrap_rtk_metrics`, `HEADROOM_RTK`, `cli_tokens_avoided`, `tokens_saved_rtk`, …) across `*.py`, `*.ts`, `*.sh`, `*.ps1`, `*.yml`, `*.html`: **zero hits**. Notable test changes: `test_wrap_openhands.py` no longer patches `_ensure_rtk_binary` and asserts `wrap openhands --prepare-only` exits 0 unpatched — the regression that was previously masked. `test_wrap_continue.py` and `test_wrap_hintfile_agents.py` were removed (every test drove RTK instruction injection). A new `test_subscription_tracker.py::test_load_state_written_before_cli_context_tools_were_removed` proves a pre-removal `subscription_state.json` still loads. ## Real Behavior Proof - **Environment:** macOS 15.4 (darwin 25.4.0), Python 3.12.6, Headroom @ this branch, real `~/.headroom` and `~/.claude` on the dev machine. - **Exact command / steps and observed result:** ```text # 1. Retired flag fails loudly instead of silently no-op'ing $ headroom wrap codex --prepare-only --context-tool rtk Error: CLI context tools (rtk, lean-ctx) have been removed from Headroom: they rewrote shell commands through a third-party binary Headroom no longer manages. Drop --context-tool / --no-context-tool and unset HEADROOM_CONTEXT_TOOL; `headroom wrap` uninstalls what they left behind on first run. $ HEADROOM_CONTEXT_TOOL=lean-ctx headroom wrap codex --prepare-only Error: CLI context tools (rtk, lean-ctx) have been removed from Headroom: ... # 2. install.sh rejects the retired flags (extracted parse_wrap_args harness) ['--no-rtk', '--port', '9999'] rc=1 ERROR: CLI context tools ... Drop --no-rtk ['--context-tool=rtk'] rc=1 ERROR: CLI context tools ... Drop --context-tool $ bash -n scripts/install.sh # syntax OK # 3. Purge ran against the real machine, which had all the orphaned artifacts $ python -c "from headroom.context_tool_cleanup import purge_context_tool_artifacts; ..." removed ~/.headroom/bin/lean-ctx (51 MB) removed ~/.headroom/bin/rtk (7.7 MB) removed ~/.local/bin/rtk (symlink into ~/.headroom/bin) removed ~/.claude/hooks/rtk-rewrite.sh removed 8 lean-ctx-* hook scripts # ~/.claude.json afterwards: 90 top-level keys, 19 projects, mcpServers unchanged # → ~59 MB reclaimed, no unrelated key touched # 4. stdout stays machine-readable while the purge reports (planted a fake artifact) $ headroom wrap openclaw --prepare-only --gateway-provider-id codex >out 2>err $ cat out {"enabled":true,"config":{"proxyPort":8787,...}} # parses as JSON $ cat err Retired CLI context tool cleanup: removed /Users/tcms/.headroom/bin/rtk # 5. --help is inert (planted artifact survives), a real run purges $ headroom wrap codex --help → artifact survived: CORRECT $ headroom wrap openclaw --prepare-only → purged: CORRECT # 6. MCP purge dry-run against a copy of the real 82 KB ~/.claude.json top-level keys 90 -> 90; projects 19 -> 19; LOST keys: none all content outside mcpServers byte-identical: True ``` Dashboard rendered via the Playwright test after the panel removal: "Token Savings" shows only `Proxy 0 (0.0%)` / `Of total wire: 36.86%`, and "Token Usage" reads Before Compression → Proxy Removed → After Compression with no "Filtered (this session)" row. Nothing below the removed panel broke. - **Not tested:** Windows and Linux (macOS only) — `install.ps1` is verified by brace-balance and inspection, not executed, since no `pwsh` is available locally. The wrap e2e suite (`e2e/wrap/run.py`) was updated but not run; it needs the Docker e2e image. `serena project index` interaction is exercised in the stacked base PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes **Stacked on #2676** (`tejas/serena-config-bootstrap`) — please merge that first; this PR's base should then be retargeted to `main`, or it will read as containing that fix too. **Breaking-change migration for users:** - Drop `--rtk`, `--no-rtk`, `--no-project-rtk`, `--keep-rtk`, `--context-tool`, `--no-context-tool` from any alias, script or CI job, and unset `HEADROOM_RTK*` / `HEADROOM_CONTEXT_TOOL`. They now error rather than being ignored, so the failure is immediate and self-explaining. - Previously-installed artifacts are purged automatically on the next `wrap`/`unwrap`; no manual cleanup needed. - `headroom perf --json` no longer carries a `cli_filtering` key, and `/stats` no longer returns a `context_tool` section. **Docs:** `docs/rtk-architecture.md` deleted; RTK/lean-ctx removed from `README.md`, `docs/content/docs/{configuration,opencode,grok-build,docker-install,filesystem-contract}.mdx`, `docs/observability.md` and the matching `wiki/` pages. `REALIGNMENT/09-phase-G-rtk-observability.md` is marked SUPERSEDED rather than deleted, to keep the planning record. **Follow-ups not in scope:** `_emit_wrap_interrupted` was deleted as dead code — its only caller was the `except KeyboardInterrupt` guarding the binary download, so with no download there is nothing slow left to interrupt. |
||
|
Tejas Chopra
|
759209cff3
|
fix(wrap/serena): stop creating serena_config.yml, unbricking Serena on fresh installs (#2676)
## Description
`_ensure_serena_dashboard_disabled()` wrote a one-key bootstrap config
(`web_dashboard_open_on_launch: false`) into
`~/.serena/serena_config.yml` when the file was absent, assuming Serena
fills in any key it omits.
Verified against **Serena 1.6.2.dev0**
(`serena/config/serena_config.py`), that holds for every field except
one. Serena autogenerates its own complete config **only when the path
does not exist**:
```python
if not os.path.exists(config_file_path):
cls._generate_config_file(config_file_path)
```
Once any file is present it validates instead. Every other field falls
back to a dataclass default via `get_value_or_default`, but a missing
`projects` key is fatal (~line 1064):
```
SerenaConfigError: `projects` key not found in Serena configuration.
```
So Headroom's own bootstrap file killed Serena on **every machine
without a pre-existing Serena config**. The MCP server exited during
handshake — surfacing as `connection closed: initialize response` on
Codex and a bare `MCP error -32000: Connection closed` on OpenCode
(#2674) — and `serena project index` failed identically.
Headroom now leaves that file to Serena. That is immune to Serena adding
required keys later; guessing the schema is what caused the outage. The
popup never needed the file anyway: `build_serena_spec` passes
`--open-web-dashboard False`, which Serena applies *after* loading the
config (`serena/mcp.py:361` — `config.web_dashboard_open_on_launch =
open_web_dashboard`), so the flag wins regardless of what is on disk.
An **existing** config is still edited in place — dashboard key flipped,
`projects: []` backfilled to repair machines an affected version already
wrote — preserving a populated `projects` list, other keys and comments.
Closes #2674
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)
## Changes Made
- `_ensure_serena_dashboard_disabled()` never creates
`serena_config.yml`; it only edits an existing one, and backfills
`projects: []` there to repair already-broken machines.
- Dropped `_scope_serena_languages` + `_detect_repo_languages` +
`_EXT_TO_SERENA_LANGUAGE` (**−133 lines**). Dead weight: Serena
determines languages itself in `ProjectConfig.autogenerate`
(`_determine_project_language_servers`) and records them under
`language_servers` — `languages`, which Headroom wrote, is a legacy name
Serena migrates via `RENAMED_FIELDS`. Serena's generated file uses a
block-style list, so our single-line-flow regex never matched it: on any
Serena-generated `project.yml` the function was a **verified no-op**.
The only case where it acted was creating the file — the same
partial-config trap — which also skipped the `project.local.yml` sidecar
Serena writes alongside.
- **Test isolation:** the MCP install ledger defaults to
`~/.headroom/mcp_installs.json`, so any test registering a server wrote
into the developer's real ledger (observed adding a live `claude/serena`
entry during a local run). `conftest.py` now redirects it per-test.
- **Repo config:** `.serena/project.yml` carried a stale `project_name`
(`"feature-opencode-wrap"`) and listed only `typescript`, so Serena's
symbol index skipped 1331 Python and 194 Rust files for every
contributor.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_wrap_code_memory.py tests/test_cli/test_wrap_serena_boost.py \
tests/test_cli/test_serena_migrate.py tests/test_cli/test_serena_disable.py -q
35 passed, 1 skipped in 0.84s
$ SERENA_SRC=<serena checkout> pytest tests/test_wrap_code_memory.py -q
13 passed in 0.62s # the skipped test runs when a Serena source tree is available
$ ruff check headroom/ tests/ --exclude headroom/dashboard/templates
All checks passed!
$ mypy headroom/cli/wrap.py
Success: no issues found in 1 source file
```
New tests. The key one asserts the invariant rather than our own key
list, so it stays correct even if Serena adds a required key — a test
pinning `projects: []` would keep passing while users broke again:
- `test_serena_config_is_never_created_by_headroom` — Headroom must not
pre-empt Serena's bootstrap
- `test_serena_dashboard_disabled_repairs_config_missing_projects` —
heals a config an affected version wrote
- `test_serena_dashboard_disabled_preserves_registered_projects` — never
clobbers the real registry; comments kept, no duplicate key
- `test_serena_dashboard_disabled_is_idempotent`
- `test_serena_config_required_keys_match_serena_source` — reads
Serena's real source and pins the two facts this fix rests on
(bootstrap-only-when-absent, `projects` is the sole fatal omission).
Skipped unless `SERENA_SRC` is set; deliberately **not** named
`HEADROOM_*` because `conftest.py` scrubs that namespace, which would
make it silently always-skip.
## Real Behavior Proof
- **Environment:** macOS 15.4 (darwin 25.4.0), Python 3.12.6, Serena
1.6.2.dev0 via `uvx --from git+https://github.com/oraios/serena`, Codex
CLI 0.146.0.
- **Exact command / steps:** a probe doing a real JSON-RPC `initialize`
handshake against the exact command `headroom wrap` registers — i.e.
what Codex/OpenCode actually do — in a throwaway `HOME` per case. (A)
pre-seeded with the one-line config an affected version wrote; (B) no
config; (C) real `headroom wrap codex --prepare-only`, then handshake.
- **Observed result:**
```text
=== A. BROKEN: single-key config (Headroom 0.33.0) ===
MCP handshake: FAIL — no initialize response (exit=1). stderr tail:
File ".../serena/config/serena_config.py", line 1064, in from_config_file
raise SerenaConfigError("`projects` key not found in Serena configuration. ...")
serena.config.serena_config.SerenaConfigError: `projects` key not found ...
config after run: 1 lines, has 'projects': False
=== B. FIXED: no config, Serena bootstraps it ===
MCP handshake: PASS — initialize OK — serverInfo.name='Serena'
config after run: 213 lines, has 'projects': True
=== C. FULL FLOW: real `headroom wrap codex` then handshake ===
Serena: no serena_config.yml yet — letting Serena generate it
Serena MCP: registered (restart OpenAI Codex CLI if it was already running)
Serena: project pre-indexed (symbol cache warmed)
serena_config.yml: 213 lines, written by Serena (correct)
MCP handshake: PASS — initialize OK — serverInfo.name='Serena'
--- verdict ---
A (broken config) started: False <- expected False
B (fixed, no config) started: True <- expected True
C (after real wrap) started: True <- expected True
```
A second `wrap` in the same HOME flips the dashboard without damage:
`true` → `false`, `projects` intact, all 153 comment lines intact. The
writer was isolated against a pristine 213-line Serena config: **delta 0
newlines**.
- **Not tested:** Windows and Linux (macOS only); Serena versions other
than 1.6.2.dev0; the JetBrains language backend. The probe needs network
+ `uvx` (~2 min) so it is a manual verification tool, not wired into CI.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
- **Docs:** N/A — no user-facing docs described the `serena_config.yml`
bootstrap or the language scoping.
- Also fixes the OpenCode report (#2674). The Codex-side report of the
same root cause quotes the `SerenaConfigError` verbatim; OpenCode only
surfaces the generic `-32000`, which is why it read as two different
bugs.
- Users already broken by an affected version are repaired automatically
on their next `headroom wrap` — no manual `serena_config.yml` edit
needed.
- A stacked PR removing the rtk/lean-ctx CLI context tools is based on
this branch; this one is deliberately small so it can land first.
|
||
|
Tejas Chopra
|
28aa53dc7c
|
chore: release main (#2339)
🤖 I have created a release *beep* *boop* --- <details><summary>0.33.0</summary> ## [0.33.0](https://github.com/headroomlabs-ai/headroom/compare/v0.32.0...v0.33.0) (2026-07-29) ### Features * **lossless:** factor shared directory prefix in the grep search fold ([#2547](https://github.com/headroomlabs-ai/headroom/issues/2547)) ([ |
||
|
Tejas Chopra
|
5383c6bf2f
|
fix(release): sync generated version metadata on the release branch (#2659)
## Description
The 0.33.0 release PR (#2339) has sat in `changes-requested` since
2026-07-17. Root cause: **release-please only rewrites `pyproject.toml`
and its configured `extra-files`**, but other tracked files also carry
the version — and `server.json` is asserted byte-for-byte against
`render_server_json()`, which derives its version from `pyproject.toml`.
So the bump alone fails
`tests/test_mcp_registry/test_server_json.py::test_root_server_json_matches_builder`
(the `test (2)` shard) on every regenerated release PR.
Nothing in the repo regenerated `server.json` at all, so it fell behind
every release.
Unblocks #2339.
### Why the release *build* passes but the release PR does not
`release.yml` already runs `scripts/version-sync.py` immediately before
its own `verify-versions.py` gate (lines 145 and 278). That is why
`build` and `build-wheels` are green on #2339 despite the drift — it
syncs in the workspace, uncommitted. The regular CI test job does
**not** sync, so the fix has to be committed to the branch.
This also explains why reviewers kept seeing `verify-versions.py` fail
locally while CI's build jobs passed: the verifier is never run
un-synced inside `release.yml`.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- **`scripts/version-sync.py`**: also write `server.json`. It was the
one version-carrying file with no writer anywhere. Values are rewritten
in place so key order and formatting keep matching the builder's
byte-for-byte output (verified: the file is pure ASCII and round-trips
exactly through `json.dumps(..., indent=2) + "\n"`).
- **`.github/workflows/release-metadata-sync.yml`** (new): on a push to
`release-please--branches--**`, run version-sync → gate on
verify-versions → commit if changed.
- **Keyed off the branch push** because release-please force-regenerates
that branch on every merge to main. That is precisely what wiped the
hand-pushed metadata fixes on #2339 (`2a86c8ff`, `d5ea4dc5`) — a push
trigger re-heals after every regeneration instead of being lost.
- **Uses the same PAT as `release-please.yml`**: a `GITHUB_TOKEN` push
does not trigger workflows, so the release PR's checks would never
re-run against the synced commit and would stay red.
- **Idempotent**: the self-triggered rerun finds no diff and exits
before pushing, so the loop terminates after one no-op run.
- **Corrected pre-existing drift on `main`**: the agent-hooks plugin
manifests, both marketplace manifests, and `.releasemetadata` were
stranded at **0.31.0** — never bumped for 0.32.0 either.
`verify-versions.py` now passes on `main`.
### Why not more `extra-files` entries
That would need ~13 jsonpath entries restating what `version-sync.py`
already knows, and a jsonpath that fails to match **fails silently** —
the same class of failure this PR removes, discoverable only after a
real release PR regenerates. There is also no precedent for nested
jsonpath (`$.packages[0].version`, `$.metadata.version`) in the config
today; both existing entries are plain `$.version`. Running the script
keeps one source of truth, and files added to it later are covered with
no change here.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) — no `headroom/` sources
touched
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest scripts/tests/ tests/test_release_workflows.py tests/test_mcp_registry/ -q
207 passed in 2.69s
$ ruff check scripts/version-sync.py scripts/tests/test_version_sync.py tests/test_release_workflows.py
All checks passed!
$ ruff format --check <same>
3 files already formatted
$ actionlint .github/workflows/release-metadata-sync.yml
(clean)
```
New tests:
- `test_server_json_version_is_synchronized` — version-sync moves both
`server.json` version fields and preserves the other keys.
- `test_release_metadata_sync_runs_on_release_please_branch` — asserts
the trigger, the sync→verify→commit ordering, the no-op guard, and the
PAT.
- `test_version_sync_covers_every_file_the_verifier_gates` — guards
`version-sync.py` and `verify-versions.py` against drifting apart again,
which is the root cause here.
## Real Behavior Proof
- **Environment:** macOS (Darwin arm64), Python 3.12, repo venv.
- **Exact command / steps:** reproduced the CI failure locally by
simulating release-please's partial bump, then applying the fix.
**Reproducing the exact `test (2)` failure** — set `pyproject` to 0.33.0
while `server.json` stays at 0.32.0, as release-please leaves it:
```text
$ python -m pytest tests/test_mcp_registry/test_server_json.py -q
FAILED tests/test_mcp_registry/test_server_json.py::test_root_server_json_matches_builder
1 failed, 3 passed
```
**After `version-sync.py`:**
```text
$ python scripts/version-sync.py && python -m pytest tests/test_mcp_registry/test_server_json.py -q
4 passed
```
**Both gates green on a simulated 0.33.0 bump:**
```text
$ python scripts/version-sync.py --version 0.33.0
Version synchronized to 0.33.0
$ python scripts/verify-versions.py
All versions aligned at 0.33.0
$ python -m pytest tests/test_mcp_registry/test_server_json.py -q
4 passed
```
**Idempotency** (the property the workflow's loop-termination relies
on): re-running against an already-synced tree leaves `pyproject.toml`,
`server.json`, `openclaw`, and `sdk/typescript` untouched.
- **Not tested:** the workflow has not executed on a real release-please
branch regeneration — that can only be exercised once this is on `main`
and release-please next updates #2339. The PAT push path and the
self-trigger no-op are reasoned from `release-please.yml`'s existing
token comment and from local idempotency, not observed in CI.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
**Context on the v0.32.0 release failure, since it is easy to misread as
"images never build".** Every artifact built for v0.32.0 — all 5 wheel
platforms including Windows, all 16 Docker builds + 8 manifests +
`promote-latest`, npm, and GitHub Packages. Only `publish-pypi` failed
(PyPI attestations, already fixed by `
|
||
|
Robert Schorr
|
b3f016b866
|
fix(mcp): pin mcp dependency to <2.0.0 to prevent server startup crash (#2642)
## Description
The MCP Python SDK recently released version `2.0.0`, which introduced
breaking changes to the high-level server interface (removing
`.list_tools()` and `.call_tool()` decorators on `mcp.server.Server`).
Because `headroom-ai` specified `mcp>=1.28.1` without an upper bound,
installing or upgrading `headroom-ai` pulled in `mcp 2.0.0`. When
`headroom mcp serve` was started by an MCP client (such as OpenCode or
Claude Code), the server crashed immediately on startup with
`AttributeError: 'Server' object has no attribute 'list_tools'`,
resulting in the connection closing error (`headroom MCP error -32000:
Connection closed`).
This PR pins the `mcp` dependency to `<2.0.0` (`mcp>=1.28.1,<2.0.0`) in
`pyproject.toml` so compatible 1.x SDK releases (e.g. `1.29.0`) are
used.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Pinned `mcp` dependency to `"mcp>=1.28.1,<2.0.0"` under both `proxy`
dependencies and the `mcp` extra in `pyproject.toml`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ ruff check pyproject.toml headroom/ccr/mcp_server.py
All checks passed!
$ pytest tests/test_ccr_mcp_server.py tests/test_cli/test_mcp.py tests/test_cli/test_mcp_status.py
============================== 45 passed in 1.03s ==============================
```
## Real Behavior Proof
- Environment: macOS (Darwin arm64), Python 3.14.5, uv
- Exact command / steps: Executed uv sync --all-extras and sent stdio
JSON-RPC initialize and tools/list requests to .venv/bin/headroom mcp
serve.
- Observed result: Server initializes cleanly and returns JSON-RPC
response
{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2024-11-05","capabilities":{"experimental":{},"tools":{"listChanged":false}},"serverInfo":{"name":"headroom","version":"1.29.0"}}}
with no startup AttributeError or closed pipe errors.
- Not tested: N/A
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did not edit CHANGELOG.md — it is generated by release-please
from my Conventional Commit PR title (a CI guard enforces this)
## Screenshots (if applicable)
N/A
## Additional Notes
Capping mcp<2.0.0 ensures stability with current headroom releases while
a future update can adopt MCP SDK 2.x interface changes if desired.
|
||
|
JD Davis
|
2dc7e4ab27
|
test: add fluent Headroom harness (#2650)
## Description Adds `headroom.testing`, a fluent, contractual test harness for building Headroom scenarios and suites that can be simulated locally, orchestrated, deployed through the proxy, and handed off to `headroom-bench` / `agent-evals` with bench-native manifests. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add `headroom.testing.Headroom` fluent scenario builder with provider/platform/configuration facets such as `WithBedrock`, `OnAppleSilicon`, `Configure`, `WithCompression`, `WithCCR`, `WithCache`, `WithPrefixFreeze`, `WithReadMaturation`, and `WithMemory`. - Add contractual coverage over the current `HeadroomConfig` and `ProxyConfig` dataclass surfaces, including full JSON-ready proxy deployment payloads. - Add no-key local simulations, scenario/suite orchestration, guarantee evaluation, deployment plans, and a local proxy lifecycle context manager. - Add `headroom-bench` handoff artifacts, including `agent_evals.models.RunManifest`-compatible JSON without taking a runtime dependency on `agent-evals`. - Add demonstration tests for providers, feature facets, manifests, suites, guarantees, deployment payloads, and the no-key simulation path. - Fix unversioned OTEL meter lookup typing so `mypy headroom` remains green on current `main`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text python -m ruff check . All checks passed! python -m mypy headroom headroom\proxy\server.py:1680: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] headroom\proxy\server.py:1691: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] Success: no issues found in 512 source files python -m pytest tests/test_cli/test_subprocess_utf8_encoding.py tests/test_testing_harness.py -q 24 passed, 1 warning in 4.68s ``` ## Real Behavior Proof - Environment: Windows, Python 3.13.13, branch `feat/headroom-test-harness` rebased on `headroomlabs-ai/main`. - Exact command / steps: built a `Headroom.WithOpenAI().WithCompression(mode="cache", kompress=False).Build()` scenario and entered `scenario.deploy_local(port=19192, timeout_s=20)`. - Observed result: proxy launched, `/readyz` succeeded, handle returned `http://127.0.0.1:19192`, `OPENAI_BASE_URL=http://127.0.0.1:19192/v1`, and context-manager teardown completed. - Exact command / steps: emitted `scenario.agent_evals_manifest(...).to_dict()` and validated it with the current cloned `headroom-bench` `agent_evals.models.RunManifest` pydantic model. - Observed result: validation succeeded with arms `a0_direct`, `a1_passthrough`, and `b_headroom` for provider `openai`. - Not tested: upstream-provider API calls requiring real OpenAI/Anthropic/Bedrock keys; phase-1 validation intentionally stays no-key/local. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A. ## Additional Notes The pytest warning shown above is the existing OpenAI pricing-data staleness warning from cost estimation. The harness does not call upstream providers during local simulation. |
||
|
Tejas Chopra
|
e0d2cd0c5a
|
fix(cache): preserve cache_control ttl when re-anchoring a breakpoint (#2651)
## Description
`normalize_message_cache_control` deliberately reuses the client's
marker verbatim so an explicit `cache_control.ttl` (e.g. `"1h"`)
survives breakpoint consolidation instead of silently downgrading to the
5-minute default (#2375).
Two other sites also strip a breakpoint and re-place it, and both
hardcoded a bare `{"type": "ephemeral"}` — undoing that guarantee.
A downgrade is invisible: the request still succeeds, and the cost shows
up later as a full prefix re-write on every idle gap past 5 minutes.
Measured over 10,409 local Claude Code API requests, cache writes are
**6.1% of raw input tokens but 44.8% of the price-weighted input bill**
(5m write 1.25x vs read 0.1x), and **89% of those write tokens are
re-writes of content cached one request earlier**. Honoring a 1h TTL
when the client asks for it is the cheapest thing we can do about that.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/read_maturation.py` — `relocate_cache_breakpoint`
now carries the stripped marker forward when re-anchoring before the
held-Read region. This is the one that mattered most: it runs **after**
`normalize_message_cache_control` in the Anthropic handler
(`anthropic.py:1747` vs `:1642`), so it had the final say — a 1h client
with read maturation enabled was being downgraded to 5m.
- `headroom/proxy/helpers.py` — `inject_tool_search_deferral` keeps the
dropped marker when moving the tools-array breakpoint off a now-deferred
tool onto the last resident real tool.
- Both fall back to a bare ephemeral only when the client sent no ttl,
and neither invents a breakpoint where none existed.
- `headroom/transforms/compression_policy.py` — comment only. Notes that
`CACHE_WRITE_MULTIPLIER` is hardcoded to the 5m tier (1.25x), so a
client already on 1h caching (2.0x) has its mutations gated with a ~40%
under-stated write penalty. Harmless while the net-cost gate stays
default-off (`HEADROOM_NET_COST_POLICY`); names the plumbing needed if
it is ever enabled.
Both changed code paths sit behind off-by-default flags
(`HEADROOM_READ_MATURATION`, `HEADROOM_TOOL_SEARCH`), so this is a
latent-bug fix with **no default behavior change**.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_cache_ttl_preserved.py tests/test_read_maturation.py \
tests/test_read_maturation_handler_nobust.py tests/test_cache_control_move_bust.py -q
tests/test_cache_ttl_preserved.py ..... [ 12%]
tests/test_read_maturation.py ...................... [ 67%]
tests/test_read_maturation_handler_nobust.py ... [ 75%]
tests/test_cache_control_move_bust.py .......... [100%]
============================= 40 passed in 15.52s ==============================
$ ruff check headroom/ tests/ --exclude headroom/dashboard/templates
All checks passed!
$ mypy headroom
Success: no issues found in 509 source files
```
Broader regression sweep over every cache/breakpoint-adjacent suite:
```text
$ python -m pytest tests/ -q -k "read_maturation or tool_search or cache_control or prefix_tracker or ttl_preserved"
204 passed, 10145 deselected in 59.95s
```
## Real Behavior Proof
- **Environment:** macOS 25.4.0 (arm64), Python 3.12.6, pytest 9.0.2,
branched from `main` at
|
||
|
Zhenjia ZHOU
|
e825588bfb
|
fix(ccr): sliding idle-window TTL with max-lifetime ceiling in the Rust core backends (#2604) (#2631)
## Description
Rust-core counterpart of the CCR mid-session expiry fix. #2604 (and its
duplicate #2616) report that the 30-minute wall-clock TTL kills entries
in the middle of a normal multi-agent burst: the clock starts at
compression time and never refreshes, so an entry the session keeps
touching still dies.
#2607 fixes this on the Python side by turning the TTL into an idle
window that restarts on every successful retrieval, bounded by an
absolute max lifetime (8x the idle TTL) — but it explicitly notes the
caveat that the Rust core still measures TTL from insertion. This PR
closes that gap: all three Rust CCR backends (`InMemoryCcrStore`,
`SqliteCcrStore`, `RedisCcrStore`) now use the same sliding idle-window
+ max-lifetime-ceiling semantics as the Python `CompressionStore`.
Scoped to the Rust core only; it deliberately does not touch
`DEFAULT_TTL`'s value (1800), which #2607 bumps to 3600 — happy to
rebase in lockstep whichever lands first.
Refs #2604, #2616. Complements #2607.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `crates/headroom-core/src/ccr/mod.rs`:
`DEFAULT_MAX_LIFETIME_MULTIPLIER = 8` + `max_lifetime_for()` helper;
documents the idle-window semantics.
- `in_memory.rs`: entries track `last_accessed`; a hit refreshes it
under the shard write lock (`get_mut`), expiry checks idle window OR max
lifetime, and the existing `remove_if` TOCTOU protection now uses the
same predicate. New `with_capacity_and_ttls` constructor for independent
control of window and ceiling.
- `sqlite.rs`: new `last_accessed` column (legacy DBs migrated in place
via `ALTER TABLE`, backfilled from `created_at` so old rows keep their
original expiry baseline); lazy purge and the lookup honour both bounds;
a hit touches the row under the same connection mutex as the read. New
`open_with_ttls` constructor.
- `redis.rs`: a hit re-arms the key's expiry, capped by a companion
`{prefix}:{hash}:born` key whose remaining TTL marks the absolute
ceiling; entries written by pre-sliding builds (no born key) are
backfilled rather than dropped.
- `tests/ccr_backends.rs`: 6 new tests — sliding-window survival and
max-lifetime cap for in-memory and SQLite, legacy-schema migration, and
a gated Redis sliding test.
No public API is broken: existing constructors keep their signatures and
derive the ceiling as 8x the idle TTL.
## Testing
- [x] Unit tests pass (`cargo test -p headroom-core`)
- [x] Linting passes (`cargo clippy -p headroom-core --all-features`,
`cargo fmt --check`)
- [ ] Type checking passes (`mypy headroom`) — N/A, no Python files
touched
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ cargo test -p headroom-core --test ccr_backends
test result: ok. 12 passed; 0 failed; 0 ignored (8.12s)
$ cargo test -p headroom-core ccr # all ccr-named tests across suites
38 passed, 949 filtered out (13 suites)
$ cargo test -p headroom-core --test ccr_roundtrip --test live_zone_ccr
18 passed (2 suites)
$ cargo check -p headroom-core --features redis # cfg-gated backend compiles
Finished `dev` profile in 25.09s
$ cargo clippy -p headroom-core --all-features
No issues found
```
## Real Behavior Proof
- Environment: macOS (Darwin 25.5), local checkout at upstream `main`
(
|
||
|
Zhenjia ZHOU
|
e86c6390ce
|
fix(rust): port CJK-aware relevance-query matching to CodeCompressor (#2634)
## Description The Rust port of `CodeCompressor` (#1154, parity-only) did not carry over the CJK-aware relevance-query matching from `headroom/transforms/code_compressor.py` (`_CONTEXT_DELIMS` / `_CJK_CHARS` / `_query_context_tokens()` / `_symbol_in_context()`, lines 2353-2387, called from lines 987/1009): - Rust tokenized the context with an ASCII-only delimiter class `[\s,;:.()\[\]{}"']+`, so a CJK query (no spaces, CJK punctuation) collapses into a single blob and never isolates an ASCII symbol name. - The substring-fallback guard `chars().count() > 3` had no CJK relaxation, so a short ASCII name glued to CJK text (e.g. `run` in `修复run函数的报错`, `db` in `请保留db相关的逻辑`) could never receive the +3.0 context boost — while Python does boost it. Same `(code, context)` input, different `symbol_scores`. This PR ports the two Python helpers with identical semantics: - `query_context_tokens()` — delimiter class extended with the CJK/full-width punctuation and ideographic space from Python's `_CONTEXT_DELIMS`; returns `(words, lowered, has_cjk)` with CJK detection over U+3000-U+9FFF, U+AC00-U+D7AF, U+FF00-U+FFEF (Python's `_CJK_CHARS`). - `symbol_in_context()` — exact token match, plus the substring fallback gated by `> 3` **characters** (Python `len()`, not bytes), relaxed when the query contains CJK. The call site in `analyze_symbols` now uses these helpers; no other behavior changed. Pure-ASCII query behavior is identical to before (exact token match, `>3`-gated substring fallback), which the tests pin down. Closes #2630 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `crates/headroom-core/src/transforms/code_compressor.rs`: extract `query_context_tokens()` / `symbol_in_context()` free functions mirroring the Python helpers (CJK/full-width delimiter class, CJK detection, CJK-relaxed `>3`-character guard); replace the inline ASCII-only tokenization + guard in `analyze_symbols` with calls to them. - Unit tests mirroring `tests/test_transforms/test_code_compressor_cjk.py` case-for-case, plus a character-vs-byte guard test and an end-to-end `compress_with` test asserting `symbol_scores`. ## Testing - [x] Unit tests pass (`cargo test -p headroom-core` — Rust-only change; Python untouched) - [x] Linting passes (`cargo fmt --check`, `cargo clippy -p headroom-core --all-targets` — no new warnings) - [ ] Type checking passes (`mypy headroom`) — N/A, no Python changes - [x] New tests added for new functionality - [x] Manual testing performed New tests (all in `code_compressor.rs` `mod tests`): - `cjk_query_isolates_wrapped_ascii_symbol` — full-width parens isolate `parse_config` - `cjk_query_matches_short_ascii_name_glued_to_cjk` — `db` (len 2) glued to CJK matches via the relaxed guard - `english_short_name_substring_still_gated` — `db` vs "keep the database helper" must NOT match (ASCII guard unchanged) - `english_exact_token_match_unchanged`, `english_long_name_substring_fallback_unchanged`, `empty_context_matches_nothing` - `guard_counts_chars_not_bytes` — the guard is a character count, matching Python `len()` - `cjk_context_boosts_named_symbol_end_to_end` — full `compress_with` run asserting `symbol_scores` (red on main, green here — see proof) ### Test Output ```text $ cargo test -p headroom-core --lib -- code_compressor::tests test transforms::code_compressor::tests::empty_and_short_passthrough ... ok test transforms::code_compressor::tests::empty_context_matches_nothing ... ok test transforms::code_compressor::tests::estimate_tokens_uses_chars_div_4_min_1 ... ok test transforms::code_compressor::tests::py_round3_matches_cpython ... ok test transforms::code_compressor::tests::py_round_int_is_half_to_even ... ok test transforms::code_compressor::tests::cjk_query_isolates_wrapped_ascii_symbol ... ok test transforms::code_compressor::tests::english_short_name_substring_still_gated ... ok test transforms::code_compressor::tests::cjk_query_matches_short_ascii_name_glued_to_cjk ... ok test transforms::code_compressor::tests::english_exact_token_match_unchanged ... ok test transforms::code_compressor::tests::english_long_name_substring_fallback_unchanged ... ok test transforms::code_compressor::tests::guard_counts_chars_not_bytes ... ok test transforms::code_compressor::tests::cjk_context_boosts_named_symbol_end_to_end ... ok test transforms::code_compressor::tests::detect_language_basic ... ok test result: ok. 13 passed; 0 failed; 0 ignored; 0 measured; 899 filtered out; finished in 0.05s $ cargo test -p headroom-core # per-binary summaries lib .......................... ok. 911 passed; 0 failed; 1 ignored auth_mode .................... ok. 16 passed; 0 failed cache_control ................ ok. 14 passed; 0 failed ccr_backends ................. ok. 7 passed; 0 failed ccr_roundtrip ................ ok. 15 passed; 0 failed code_compressor_parity ....... ok. 1 passed; 0 failed (recorded byte-parity fixtures) live_zone_ccr ................ ok. 3 passed; 0 failed live_zone_dispatch ........... ok. 6 passed; 0 failed live_zone_thresholds ......... ok. 2 passed; 0 failed live_zone_token_validation ... ok. 3 passed; 0 failed recommendations_loader ....... ok. 4 passed; 0 failed tokenizer_proptest ........... ok. 5 passed; 0 failed doc-tests .................... ok. 1 passed; 0 failed; 2 ignored $ cargo fmt --check # clean $ cargo clippy -p headroom-core --all-targets # no new warnings ``` ## Real Behavior Proof - Environment: macOS (Darwin 25.5.0), repo-pinned Rust toolchain (`rust-toolchain.toml`), branch based on current `main`. - Exact command / steps: the end-to-end test was written first and run against unmodified `main` (red), then after the fix (green). Input: Python source with two signal-symmetric functions `run` and `keep`; context `修复run函数的报错`. The Python reference gives `run` the boost (`_symbol_in_context('run', ...) == True`, `_symbol_in_context('keep', ...) == False`, verified against the live Python implementation), so expected normalized scores are `run = 1.0`, `keep = 0.0`. - Observed result: on unmodified main the end-to-end test fails (`left: 0.5, right: 1.0` — the CJK query `修复run函数的报错` gives `run` no boost, both symbols collapse to 0.5, while Python scores `run=1.0, keep=0.0`); on this branch all 8 new tests pass and the same query boosts `run` to 1.0, matching Python. Full output: Before (unmodified `main` + new test only — Rust gives no boost, both symbols collapse to 0.5): ```text ---- transforms::code_compressor::tests::cjk_context_boosts_named_symbol_end_to_end stdout ---- thread '...cjk_context_boosts_named_symbol_end_to_end' panicked at crates/headroom-core/src/transforms/code_compressor.rs:1890:9: assertion `left == right` failed: run must get the context boost left: 0.5 right: 1.0 test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 904 filtered out ``` After (this branch): the same test passes, including its ASCII control case (`fix the runner` must NOT boost `run` — scores stay 0.5/0.5), proving pure-ASCII behavior is unchanged. The recorded byte-parity fixture suite (`code_compressor_parity`) also still passes. - Not tested: real proxy traffic end-to-end (change is confined to the symbol-scoring context boost inside the Rust compressor; the Python implementation is the behavioral reference and is untouched). `kompress_parity` was not run locally — it is model-gated and my sandbox blocks the model fetch; it is unrelated to this change and CI covers its skip path. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation — N/A (internal behavior fix) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes For background, the Python-side CJK handling comes from the merged CJK sweep work (#2220 and follow-ups); #1154 predates part of it, which is likely how the port missed it. Longer names wrapped in full-width punctuation happened to still match in Rust via the substring fallback, but the token set itself was wrong; this PR restores exact-token semantics for those too. |
||
|
Abhay Singh
|
22b707fd31
|
fix(proxy/cost): count Gemini thinking tokens in output usage (#2639)
## Description
The Gemini handlers take the response's output-token count straight from
`candidatesTokenCount`:
```python
output_tokens = _usage_int(usage.get("candidatesTokenCount"))
```
For Gemini 2.5 thinking models that undercounts. Gemini reports
`candidatesTokenCount` **sometimes inclusive** of the reasoning tokens
(`thoughtsTokenCount`) and **sometimes exclusive** of them. When it is
exclusive, the thinking tokens are a separate bucket that is still
billed at the output rate, so dropping them makes `output_tokens` (and
therefore the output cost that flows through `record_tokens` ->
`estimate_cost`) too low. The gap grows with reasoning effort.
litellm handles exactly this: it adds `thoughtsTokenCount` to completion
tokens unless `promptTokenCount + candidatesTokenCount ==
totalTokenCount` (its `is_candidate_token_count_inclusive` check). The
Headroom handlers had no equivalent.
## Fix
Add `gemini_output_tokens(usage_meta)` in
`headroom/proxy/token_counting.py`:
- No `thoughtsTokenCount` (the common non-2.5 case): return
`candidatesTokenCount` unchanged.
- `promptTokenCount + candidatesTokenCount == totalTokenCount`:
candidates already include thoughts, return `candidatesTokenCount`.
- Otherwise: return `candidatesTokenCount + thoughtsTokenCount`.
This mirrors litellm's rule and is robust to missing or null fields.
Wire it into the native Gemini handler (both the generate and count
paths), the streaming usage extractors, and the OpenAI-compatible
passthrough usage normalizer, so every Gemini usage path counts output
the same way.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring
## Changes Made
- `headroom/proxy/token_counting.py`: add `gemini_output_tokens()`.
- `headroom/proxy/handlers/gemini.py`: use it for `output_tokens` on
both response paths.
- `headroom/proxy/handlers/streaming.py`: use it in the two Gemini
streaming usage extractors.
- `headroom/proxy/handlers/openai.py`: use it in
`_passthrough_usage_from_json` (Gemini-shaped usage).
- `tests/test_proxy_handler_helpers.py`: unit test for
`gemini_output_tokens` (inclusive / exclusive / no-thinking / empty) and
a `_passthrough_usage_from_json` test that thinking tokens land in
`output_tokens`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check` / `ruff format --check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for the fix
- [ ] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_proxy_handler_helpers.py -k "gemini_output_tokens or thinking or vertex_usage_metadata" -q
3 passed
$ python -m pytest tests/test_proxy_gemini_native_integration.py tests/test_proxy/test_gemini_savings_profile.py tests/test_proxy_handler_helpers.py -q
38 passed, 18 skipped
# with the wiring reverted, the passthrough test fails (output_tokens is 200, not 700):
$ git stash push headroom/proxy/handlers/openai.py && \
python -m pytest tests/test_proxy_handler_helpers.py -k passthrough_usage_counts_gemini_thinking -q
1 failed
$ uvx ruff@0.15.17 check headroom/proxy/token_counting.py headroom/proxy/handlers/gemini.py headroom/proxy/handlers/streaming.py headroom/proxy/handlers/openai.py tests/test_proxy_handler_helpers.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/proxy/token_counting.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, project venv (`uv sync --extra
proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`, pytest in the venv.
- Exact command / steps: called `gemini_output_tokens` on an exclusive
usage (`prompt=1000, candidates=200, thoughts=500, total=1700`), an
inclusive usage (`candidates=700, total=1700`), a no-thinking usage, and
`{}`; drove `_passthrough_usage_from_json` with a thinking usage; then
reverted the handler wiring and re-ran the passthrough test.
- Observed result: exclusive returns 700 (200 visible plus 500
thinking), inclusive returns 700, no-thinking returns the candidates
count, empty returns 0; `_passthrough_usage_from_json` reports
`output_tokens=700`. With the wiring reverted it reports 200 (the
undercount). Verified against litellm's documented rule.
- Not tested: a live Gemini 2.5 request end to end (the accounting is
verified at the usage-extraction boundary against litellm's reference
logic).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
Fabien Culpo
|
1d29738818
|
fix(proxy): keep core tools and the client's ToolSearch resident for PascalCase clients (#2647)
## Description `_TOOL_SEARCH_CORE_TOOLS` is spelled in lowercase, but the membership test compared the raw tool name, so the core-tool exemption never fired for clients that send PascalCase names. For Claude Code (`Bash`, `Read`, `Edit`, `ToolSearch`) **every** tool in the request body was deferred. The damaging part is that Claude Code's own `ToolSearch` was deferred. It is the schema fetcher for tools the client keeps in its local registry and never sends in the body — `TaskCreate`, `TaskUpdate`, `TaskList`, `WebFetch`, `EnterPlanMode`, `Monitor`, `LSP`, `Cron*`, `SendMessage`. Hiding it makes all of them permanently uncallable: advertised to the model in a `<system-reminder>`, but no search can return their schemas, because the injected `tool_search_tool_regex` only indexes what is in the request body. Closes #2646 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Compare tool names against the core set case-insensitively in `inject_tool_search_deferral` (`helpers.py`). - Add `"toolsearch"` to `_TOOL_SEARCH_CORE_TOOLS` so a client's own schema-fetch tool is never deferred. - Apply the same case-insensitive comparison to `inject_tool_search_deferral_openai`, which had the identical exact-match bug (including against `_OPENAI_TOOL_SEARCH_RESIDENT_NAMES = {"terminal"}`). - Add 3 tests on the Anthropic path and 1 on the OpenAI path. Both source changes are required: case-folding alone does not help `ToolSearch` (it was not in the set), and adding it alone does not help `Bash`/`Read`/`Edit`. **The token saving is unchanged** — MCP tools are still deferred. This is not a request to disable the feature. Beyond the stranded tools, the old behaviour also meant (a) routine `Bash`/`Read`/`Edit` loops each paid a search round-trip, the exact cost the core set exists to avoid, and (b) zero resident *real* tools remained, silently violating the invariant documented on `inject_tool_search_deferral` — the injected search tool is typed and does not satisfy it — which risks an upstream 400. The existing assertion for that invariant passes today only because its fixture uses lowercase names. ## Testing - [x] Unit tests pass (`pytest`) — the two affected files; see scope note below - [ ] Linting passes (`ruff check .`) — see note - [ ] Type checking passes (`mypy headroom`) — could not run, see note - [x] New tests added for new functionality - [x] Manual testing performed `ruff check .` reports 4 findings repo-wide, **all pre-existing and unrelated** (`plugins/headroom-oauth2/`), confirmed identical on unmodified `main`. Zero findings in the three files this PR touches, and `ruff format --check` is clean on all three. Left unchecked because the repo-wide command does not exit 0. `mypy headroom` could not run in my environment (numpy stubs error out under the resolved Python version before checking begins). Not attempted further — CI should be the authority. ### Test Output ```text $ python -m pytest tests/test_issue_746_tool_search.py tests/test_openai_tool_search_deferral.py -q 65 passed, 1 warning in 0.70s # Baseline on those two files before this PR: 62 (36 + 26). # The 3 new Anthropic tests + 1 new OpenAI test bring it to 65. # Red before the source change (tests written first): tests/test_issue_746_tool_search.py::test_core_tools_match_case_insensitively FAILED AssertionError: Bash assert True is None where {'name': 'Bash', ..., 'defer_loading': True}.get('defer_loading') tests/test_issue_746_tool_search.py::test_client_tool_search_tool_is_never_deferred FAILED AssertionError: assert True is None where {'name': 'ToolSearch', ..., 'defer_loading': True}.get('defer_loading') tests/test_issue_746_tool_search.py::test_resident_real_tool_survives_pascal_case_surface FAILED assert any(not t.get("type") and not t.get("defer_loading") for t in out) assert False 3 failed, 36 deselected $ python -m ruff check headroom/proxy/helpers.py tests/test_issue_746_tool_search.py tests/test_openai_tool_search_deferral.py All checks passed! $ python -m ruff format --check <same three files> 3 files already formatted ``` ## Real Behavior Proof - Environment: headroom 0.32.1 installed / 0.32.0 source, Python 3.13, macOS 15 (Darwin 25.5.0), Claude Code 2.1.220 with `ENABLE_TOOL_SEARCH=true` and `ANTHROPIC_BASE_URL=http://localhost:8787`, first-party Anthropic upstream, `HEADROOM_TOOL_SEARCH` truthy - Exact command / steps: build a Claude Code tool surface and pass it through the injector — `names = ["Bash","Read","Write","Edit","Glob","Grep","ToolSearch"] + [f"mcp__srv__t{i}" for i in range(12)]`, `tools = [{"name": n, "description": n, "input_schema": {}} for n in names]`, then `inject_tool_search_deferral(tools)` and print which entries carry `defer_loading` - Observed result: before the fix `resident real tools: []` with `ToolSearch deferred: True` (every built-in deferred). After the fix `resident real tools: ['Bash','Edit','Glob','Grep','Read','ToolSearch','Write']` with all 12 `mcp__srv__t*` still deferred, so the saving is retained. This matches a live session: the proxy logged `router:tool_search_deferral:25tools:22182tok ... client=claude-code` and `tool_search_tool_regex` could resolve only `mcp__*` tools — `TaskCreate`/`WebFetch`/`EnterPlanMode` returned no match until `ToolSearch` was recovered by regex-searching for it and then calling `select:TaskCreate,...` - Not tested: the full pytest suite (164 modules fail collection with `ModuleNotFoundError: No module named 'headroom._core'` because my environment imports the package via `PYTHONPATH` without building the Rust extension; identical failure confirmed on unmodified `main`, so it is environmental). `mypy headroom` not runnable here. No end-to-end run against a live upstream through a rebuilt proxy — verification is at the function boundary plus the live-session log evidence above. The OpenAI Responses path is covered by unit test only, not exercised against a real gpt-5.4+ deployment. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes - **Documentation**: N/A — no user-facing surface changes; behaviour returns to what the existing comments and docstring already describe. - **"New and existing unit tests pass locally"**: left unchecked deliberately. The tests covering the changed symbols pass (65), but I cannot run the whole suite locally without the compiled `headroom._core`. Not claiming more than I verified. - **Scope**: the OpenAI-path fix rides along because it is the identical three-line comparison bug in the sibling function. Happy to split it into its own PR if you would rather keep this Anthropic-only. - **Deliberately not done**: I did not add a `client != "claude-code"` gate at `handlers/anthropic.py`, even though the feature's own comment block scopes it to non-Claude-Code clients and `client=claude-code` is already known there (it appears in the `transforms=` log line). Gating there would forfeit the ~22k tokens/request currently saved on Claude Code's eagerly-shipped MCP schemas; keeping the meta-tool resident preserves both the saving and reachability. Flagging in case you would prefer to gate as well. - **Adjacent blind spot, out of scope**: `claude_code_tool_search_inactive` already checks both the tools array *and* the `anthropic-beta` header, but the injector's early-return guard checks only the array. That is why a plain-function `ToolSearch` slips past it and the injection runs on a client that is already deferring. Co-authored-by: Fabien Culpo <fabien.culpo@dawex.com> |
||
|
Devanshi Vyas
|
1588f5e041
|
feat: expose configured OTEL meters to integrations (#2519)
## Description
Expose a small public observability API that lets optional integrations
create
OpenTelemetry instruments using Headroom's configured meter provider.
Without this API, an integration must either rely on observability
internals or
create a second provider and exporter. `get_otel_meter(name, version)`
keeps
configuration, export, and shutdown ownership inside Headroom while
allowing
integration-specific instruments to use their own instrumentation scope.
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Add `HeadroomOtelMetrics.get_meter(name, version)` to obtain a meter
from the
provider already owned by the Headroom metrics facade.
- Add and publicly export `headroom.observability.get_otel_meter(...)`.
- Preserve no-op-compatible OpenTelemetry behavior when Headroom-managed
metric
export is not configured.
- Add a focused test proving integration instruments are collected by
the same
configured provider.
- Add no dependencies and make no changes to existing metrics or
configuration.
## Testing
<!-- Check what you actually ran, then paste the real command output
below. -->
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ uv run pytest tests/test_observability_metrics.py -q
collected 6 items
tests/test_observability_metrics.py ...... [100%]
6 passed in 4.72s
$ uv run ruff check .
All checks passed!
$ uv run ruff format --check .
1331 files already formatted
```
## Real Behavior Proof
- Environment: macOS (Darwin), Python 3.12.13, Headroom `0.33.0-dev`,
OpenTelemetry SDK `1.39.1`, console metric exporter.
- Exact command / steps: Run the command below:
```bash
uv run python -c 'from headroom.observability import OTelMetricsConfig,
configure_otel_metrics, get_otel_meter, shutdown_otel_metrics;
configure_otel_metrics(OTelMetricsConfig(enabled=True,
exporter="console", service_name="headroom-integration-proof",
export_interval_millis=60000)); get_otel_meter("example.integration",
"1.0.0").create_counter("example.integration.events").add(3, {"source":
"extension-api"}); shutdown_otel_metrics()'
```
- Observed result: Headroom's console exporter emitted
`example.integration.events` with value `3`, attribute
`source="extension-api"`, instrumentation scope `example.integration`
version `1.0.0`, and resource service name `headroom-integration-proof`.
This demonstrates that the public accessor participates in Headroom's
configured provider and shutdown lifecycle.
- Not tested: network OTLP export, the complete repository test suite,
`mypy`,
or Python versions other than 3.12 in this final validation.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A — this change has no user-interface surface.
|
||
|
Tejas Chopra
|
57bf720d5c
|
feat(router): route embedded & nested JSON through the compressor dispatch (#2623)
## Description
`ContentRouter` only compressed JSON when the **whole** `tool_result`
block was a single JSON value. JSON embedded inside larger output (`gh
api` dumps, MCP tool results, `curl | jq` tails, log lines ending in a
JSON blob) was invisible to the JSON compressors — and in practice that
embedded shape is the large majority of JSON an agent actually sees.
This adds a structural routing step: find balanced JSON spans at **any
offset** in a block and route each one through the router's **existing,
unchanged** `_apply_strategy_to_content`, splicing the result back with
the surrounding bytes kept exact.
Because each span takes the same dispatch path a whole-block JSON
already takes, SmartCrusher/CodeCompressor register their `<<ccr:…>>`
retrieval markers exactly as before — CCR is hash-keyed, so it is
location-agnostic and unaffected by nesting.
Closes #
## Type of Change
- [x] New feature (non-breaking change that adds functionality)
## Changes Made
- New `headroom/transforms/recursive_json.py` — `route_embedded_json()`:
deterministic balanced-span scan + splice; skips spans already carrying
a `<<ccr:` marker (never re-compresses); token-gated.
- One guarded call at the top of `_apply_strategy_to_content` plus an
`_allow_embedded` **one-shot re-entrancy guard** (not a depth cap).
- **No size/min or depth thresholds** — the only gates are correctness
(round-trip) and benefit (token reduction). Strict no-op when a block
has no embedded JSON, so the 97%+ of non-JSON blocks are byte-identical
to today.
- Deterministic + per-block → prefix-cache- and CCR-store-stable.
- `tests/test_recursive_json.py`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_recursive_json.py tests/test_content_router_compact_json.py tests/test_content_router_tool_role_reversibility.py -q
19 passed in 5.90s
$ ruff check headroom/transforms/recursive_json.py headroom/transforms/content_router.py tests/test_recursive_json.py
All checks passed!
$ mypy headroom/transforms/recursive_json.py headroom/transforms/content_router.py
Success: no issues found in 2 source files
```
## Real Behavior Proof
- **Environment:** local,
`ContentRouter(ContentRouterConfig(lossless=False))`, pure-Python
content detector.
- **Exact steps:** `router.compress(block)` where `block` = prose with a
120-row JSON array embedded mid-text (`"I queried the ECS API
...\n[{...}]\nAll services healthy."`).
- **Observed result:** `strategy_used=MIXED`; block **11,986 → 3,798
chars**; leading/trailing prose preserved byte-exact; the embedded JSON
folded to a columnar table. Previously this block's embedded array was
not routed to the JSON compressor at all.
- **CCR:** a span already containing `<<ccr:` is passed through
untouched (unit-tested); folded spans go through the unchanged dispatch,
so markers register and resolve identically to whole-block JSON.
- **Not tested:** live proxy end-to-end with markers force-enabled
(covered by the unchanged dispatch path + unit tests); non-CC transcript
shapes beyond the local corpus.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
Scoped to the OSS structural-routing step only. The lossless *fold
kinds* it routes into (JSON columnar / log template) are maintained in
the `headroom-lossless-guard` extension. N/A: no docs/screenshots;
`Closes #` left blank (no tracking issue).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
Ruben A.
|
e530de5ad2
|
feat(rust): port CodeCompressor AST compressor to Rust (parity-only) (#1154)
Adds crates/headroom-core/src/transforms/code_compressor.rs (1,882 lines): the AST-aware CodeCompressor ported to Rust on tree-sitter, with grammars for Python, JavaScript, TypeScript, Go, Rust, Java, C and C++. Parity-only, like #1153. Nothing calls it: the only references outside the module are the pub mod / pub use declarations in transforms/mod.rs, and live_zone.rs still routes SourceCode to a no-op. The pyo3 bridge is untouched and no Python source changes, so the engine is unreachable from the shipped package. #1155 wires it into live-zone dispatch. Every grammar is pinned with '=' to the exact version of the corresponding Python tree-sitter-<lang> PyPI wheel. Same version on crates.io and PyPI means the same grammar.js, hence the same generated parser.c, hence node-for-node identical ASTs — the precondition for byte-parity. A canary over 9 samples x 8 languages confirmed identical node-type and line-span trees at these pins; bumping any pin requires re-running it and re-recording the fixtures. Ships 30 recorded parity fixtures, a CodeCompressorComparator in headroom-parity, and scripts/record_code_compressor_fixtures.py. Verified byte-identical to the recorded Python output: [code_aware_compressor] total=30 matched=30 skipped=0 diffed=0 Full harness on the merge result: 227 fixtures, 182 matched, 45 skipped (cache_aligner + ccr stubs), 0 diffed, exit 0 — with kompress at 21/21 under ONNX Runtime 1.24.4 (see #2591). Also verified cargo check -p headroom-core --no-default-features passes, so the static-musl path stays intact. |
||
|
Ruben A.
|
83e27e5036
|
feat(rust): port Kompress ML prose compressor to Rust (parity-only) (#1153)
Adds crates/headroom-core/src/transforms/kompress.rs (672 lines): the Kompress ML prose compressor ported to Rust, running the ModernBERT tokenizer plus the kompress-v2-base ONNX model through ort with a cache-only loader that never touches the network. Parity-only. Nothing calls it: the only references outside the module are the pub mod / pub use declarations in transforms/mod.rs. live_zone.rs still carries TODO(PR-B4), so PlainText dispatch remains a no-op and Python continues to serve prose compression. The pyo3 bridge is untouched and no Python source changes, so the new engine is unreachable from the shipped package. #1155 wires it up. Ships 21 recorded parity fixtures, a KompressComparator in headroom-parity, and scripts/record_kompress_fixtures.py. Verified byte-identical to the recorded Python output: [kompress] total=21 matched=21 skipped=0 diffed=0 That required ONNX Runtime >= 1.24 (see #2591) — below it ort deadlocks instead of erroring, which is why these fixtures had never been run. In CI the model is absent from the HF cache, so the comparator errors and the fixtures report Skipped rather than hanging. Also gates the module behind the ml feature, matching magika_detector: kompress.rs uses ort, which is optional = true, so an unconditional pub mod broke cargo check --no-default-features (the static-musl path). CI does not catch that class of break because cargo test --workspace only builds default features. |
||
|
Tejas Chopra
|
a30305bc4c
|
ci: require ONNX Runtime >= 1.24 and fail fast when it is missing or too old (#2591)
CI installed 'onnxruntime>=1.16.0' for the Rust ort runtime. That floor is 8 minor versions too low, and the failure mode below it is a silent hang. Why 1.24: ort-sys computes ORT_API_VERSION = 17 + one per enabled api-N feature, and Cargo features are additive across the graph. fastembed 5.17.3 enables api-24, so the constant resolves to 24 and ort rejects any lower runtime. Why it hangs instead of failing: on rejection ort calls Error::new() from inside load_dylib_from_path, which already runs inside the Once that setup_api() is initialising. Building the error re-enters that Once, and std::sync::Once blocks forever on re-entry. Reproduced in isolation with a bare Session::builder() and onnxruntime 1.21.1 - killed after >1h at 0% CPU, no output; ORT_DYLIB_PATH makes no difference. With 1.24.4 the same call returns in 1.2s and the kompress parity fixtures pass 21/21. Per @RubenAAA this is not limited to old runtimes: a box that resolves no libonnxruntime at all hangs identically (0.0% CPU, threads in futex_wait_queue, nothing onnx-shaped in /proc/<pid>/maps). Any failure inside load_dylib_from_path re-enters the Once, so a pin alone cannot close it. So this adds a pre-flight to the dylib step asserting, before any test runs, that onnxruntime imports, that its minor is >= 24, and that a libonnxruntime object exists under capi/. Each failure exits 1 with an ::error:: annotation naming the cause, instead of burning the 30-minute timeout with an empty log. Verified all three branches locally (absent -> rc=1, 1.21.1 -> rc=1, 1.24.4 -> rc=0) and in CI, where it resolved onnxruntime 1.28.0 and exported the .so path. pyproject.toml is deliberately untouched: bumping the floor there makes headroom-ai[all] unsatisfiable via a pillow chain (onnxruntime>=1.24 forces pillow>=10.3.0,<12.0 while [all] requires pillow>=12.3.0). The user-facing hazard via headroom/_ort.py remains open and needs its own change. |