mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
1553 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Lucas Santos
|
3107994aed
|
fix(litellm): add async_post_call_success_hook to HeadroomCallback (#1322)
## Description Pointing a litellm proxy at the Headroom callback blows up on the post-call success path: ``` type object 'HeadroomCallback' has no attribute 'async_post_call_success_hook' ``` litellm's logging contract calls `async_post_call_success_hook` after a successful response, and `HeadroomCallback` simply doesn't have it. We implement `async_pre_call_hook`, `async_success_handler` and `async_failure_handler`, but not this one, so litellm hits an `AttributeError` instead of a no-op and the whole request fails. This adds the missing `async_post_call_success_hook(self, data, user_api_key_dict, response)` matching litellm's signature. It returns `response` unchanged, the token accounting already lives in `async_success_handler` so there's nothing to do here except not crash. A few notes: 1. I did not make `HeadroomCallback` inherit litellm's `CustomLogger`, on purpose. The class keeps litellm as an optional dependency, so it stays a plain class and just provides the hooks litellm looks up by name. 2. It's a pass-through, so it's safe regardless of what the response contains. Closes #1114 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/integrations/litellm_callback.py`: add `async_post_call_success_hook` to `HeadroomCallback`, returning the response unchanged; update the class docstring to list the full set of litellm hooks. - `tests/test_integrations/test_litellm_callback.py`: new tests that the method exists, is a coroutine, and returns the response untouched; build the module path with `pathlib` instead of a fragile `__file__.replace`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run --extra dev python -m pytest tests/test_integrations/test_litellm_callback.py -q 3 passed ruff: All checks passed! mypy: Success: no issues found ``` ## Real Behavior Proof - Environment: macOS, Python 3.13, this branch. - Exact command / steps: `uv run --extra dev python -m pytest tests/test_integrations/test_litellm_callback.py -q`. The tests import the callback module directly and resolve `async_post_call_success_hook` by name, the same way litellm does, then await it with a sentinel response. - Observed result: 3 passed. The hook exists, is a coroutine, and returns the exact response object it was given. Before the fix, resolving the attribute raised `AttributeError`. - Not tested: I did not stand up a full litellm proxy end to end. The fix is the missing hook method, which the unit tests cover. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes --------- Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
Focused Instability
|
4dab254d52
|
fix: emit SSE ping before message_start on Bedrock streaming path (issue #902) (#1080)
## Description Closes #902 Mid-turn user interjections (steering) silently dropped through the Bedrock streaming path. The _stream_response_bedrock code path reconstructs Anthropic SSE events from parsed StreamEvent objects instead of passing raw bytes through, so SSE-level ping keepalives are never forwarded to Claude Code. Claude Code relies on ping events to arm its mid-turn steering / interruptible state; without them, queued interjections are discarded instead of sent. Root cause (confirmed): - Standard direct-Anthropic path does a raw yield-chunk passthrough — pings flow unchanged. - Bedrock path (_stream_response_bedrock.generate()) reconstructs events from litellm/anyllm stream_message() output, which only yields semantic events (message_start, content_block_*, message_delta, message_stop, error). No pings, ever. Fix: emit a synthetic 'event: ping / data: {}' at stream start (before the first message_start) so downstream clients see the same ping-then-content cadence as a real Anthropic stream. Note: periodic pings for very long responses (>~25s) may be needed if steering disarms on a timer. This commit arms it at turn start; follow-up if reporters confirm steering still drops on long turns. The causal link (ping → steering) is the reporter's hypothesis from hands-on debugging. The observable defect (zero pings in stream) is confirmed and fixed. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - headroom/proxy/handlers/streaming.py: yield ping event before the event loop in _stream_response_bedrock.generate() - tests/test_proxy/test_bedrock_sse_ping.py: 3 new tests asserting ping appears before message_start ## Testing - [x] Unit tests pass (pytest) - [x] Linting passes (ruff check .) - [x] New tests added for new functionality ### Test Output ``` tests/test_proxy/test_bedrock_sse_ping.py::test_bedrock_stream_emits_ping_before_message_start PASSED tests/test_proxy/test_bedrock_sse_ping.py::test_bedrock_stream_ping_has_empty_data PASSED tests/test_proxy/test_bedrock_sse_ping.py::test_bedrock_stream_contains_message_stop PASSED tests/test_backend_streaming_cache_metrics.py (4 tests) PASSED 7 passed in 4.41s ``` ## Real Behavior Proof - Environment: macOS, Python 3.11, headroom unit tests - Exact command / steps: pytest tests/test_proxy/test_bedrock_sse_ping.py -v - Observed result: 3 new tests pass; ping appears before message_start in Bedrock stream - Not tested: end-to-end against live Bedrock + Claude Code (no Bedrock credentials available) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] Code follows project style guidelines - [x] Code is commented where non-obvious - [x] No new warnings - [x] Tests added and passing ## Additional Notes The Rust proxy files mentioned in the issue (sse/framing.rs, sse/anthropic.rs) are NOT part of this fix. Those drops are in a telemetry-only tee task that never affects the client byte path — the Rust proxy does a raw bytes passthrough for all responses. The defect is Python-only, confined to the Bedrock path. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Tejas Chopra
|
04e1517ede
|
fix(telemetry): stop mixing tokenizer scales in RequestOutcome, and fix the overhead framing (#2756)
## Description
Two defects found by reading real beacon payloads, not by inspection.
### 1. `eligible_pct: 120`
A gpt-4o-mini session shipped this:
```json
"tokens": {"original": 10, "attempted": 12, "input": 12, "saved": 0},
"rates": {"eligible_pct": 120}
```
120% is structurally impossible — you cannot attempt to compress more
than arrived. And nothing had grown.
`original_tokens` is our **local tokenizer** count. `optimized_tokens`
on the OpenAI path carried the **provider's** `usage.prompt_tokens`. Our
estimator undercounted gpt-4o-mini by 2 tokens on a 10-token request,
and every quantity derived from that pair inherited the mismatch:
- `attempted_input_tokens = optimized + saved` → 12, exceeding
`original` → `eligible_pct` 120, `yield_pct` contaminated
- `tokens_inflated` (added in #2708) → reported **2 tokens of phantom
growth**
`optimized_tokens` was dual-purpose by design — *"post-compression bytes
actually forwarded, for `input_tokens` and `tok_after`"*. Billing wants
the provider's count; deltas need the same ruler as `original_tokens`.
Those are different jobs sharing one field.
This is the same class of bug as #2743, on the request path instead of
`/v1/compress`, and it is the exact false positive I flagged as
theoretical when reviewing #2708 — where I measured the margin on a real
722-request log as **exactly zero**, so any provider counting above our
estimator would flip it. gpt-4o-mini does, and it is now in production
telemetry.
### 2. `overhead_pct` was documented as wall-clock, and is not
A 1393-turn session shipped `latency_ms_total: 16396565.8` against
`duration_s: 8904` — **4.55h of latency inside a 2.47h session, 1.84×
over wall clock.**
Both terms are sums over turns, and turns run concurrently (parallel
tool calls, subagents, several clients per proxy), so each sum
over-counts elapsed time. `overhead_pct` is still a meaningful
latency-weighted per-request share, but the comment claiming *"what
fraction of wall-clock did Headroom itself add?"* was wrong, and
shipping `latency_ms_total` beside `duration_s` invites a comparison
that yields nonsense.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
**Scale split (`outcome.py`, `handlers/openai.py`)**
- `optimized_tokens` is now always the **local** count — same tokenizer
as `original_tokens`, so every delta built from the pair is coherent.
- New optional `provider_input_tokens` carries the provider's own count.
Defaults to `0`, so the other emit sites need no change.
- Cost and volume totals read `provider_input_tokens or
optimized_tokens`, so **billing is unchanged** wherever a provider
reports usage, and falls back exactly as before where it doesn't.
- Removes a band-aid: one of the three OpenAI sites already computed
`effective_original_tokens = max(original_tokens, optimized + saved)`,
inflating `original` upward so `attempted` could not exceed it. That hid
the symptom at one site while the other two shipped the impossible
ratio.
**Overhead framing (`telemetry/session.py`)**
- Corrects the wall-clock comment on `overhead_pct` and states what it
actually is.
- Documents that `latency_ms_total` is a sum of per-request durations,
not elapsed time, and why it can exceed `session.duration_s`.
- Adds `overhead_ms_per_turn` and `latency_ms_per_turn` — unambiguous
under concurrency and comparable across sessions.
- Field names kept for schema-v1 consumers; **additive only**, no schema
bump.
## Testing
- [x] Unit tests pass (new file)
- [x] Linting passes (`ruff check` + `format --check`, pinned 0.15.17)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/proxy/outcome.py headroom/proxy/handlers/openai.py headroom/telemetry/session.py
All checks passed!
$ uvx ruff@0.15.17 format --check <same three>
3 files already formatted
$ pytest tests/test_outcome_token_scale.py -q
5 passed in 0.24s
```
## Real Behavior Proof
- **Environment:** macOS 26.4 arm64, isolated git worktree off
`upstream/main`, throwaway venv (see caveat).
Replayed both reported payloads through the corrected arithmetic:
```text
ENTRY 1 (gpt-4o-mini, the eligible_pct:120 case)
BEFORE: original=10 (local) optimized=12 (PROVIDER) saved=0
attempted = 12+0 = 12 eligible_pct = 120.0 <- impossible
tok_inflated = max(0, 12-10) = 2 <- phantom
AFTER : original=10 (local) optimized=10 (local) provider_input=12
attempted = 10+0 = 10 eligible_pct = 100.0
tok_inflated = 0
billed_input = 12 -> cost/cache math unchanged
ENTRY 2 (1393 turns, the overhead case)
latency_ms_total = 16397s vs duration 8904s -> 1.84x wall clock
NEW overhead_ms_per_turn = 333.5 latency_ms_per_turn = 11770.7
wall clock per turn = 6392.0 -> per-turn latency exceeds it => concurrency
```
Genuine post-compression growth still surfaces: `55,161 → 57,845`
reports `tokens_inflated = 2,684` (pinned as a test), so the fix doesn't
mute what #2708 exists to show.
- **Not tested locally beyond the new file.** The repo venv currently
has no `pytest`, no `ruff`, and no compiled `headroom._core`, so I used
a throwaway venv. The outcome/telemetry suites fail there for missing
deps — `click` (12/12), then `headroom._core` (10/10) — with **zero
assertion failures**, and an **identical failure set on this branch and
on clean `upstream/main`** in the same env. So they are
environment-only, not regressions. CI on this PR is the authoritative
signal for the full suite.
- Unrelated: `Wrap E2E / docker-wrap-e2e` is currently red on `main`
from a quay.io CDN `tls: internal error` pulling the manylinux base
image — infrastructure, transient, and green on the three prior runs.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] I did **not** edit `CHANGELOG.md`
## Follow-ups not in this PR
- **`cache_write` and `uncached` double-count** on inferred-cache
providers. The same beacon entry shows `input: 12, cache_read: 0,
cache_write: 12, uncached: 12` — both fields describe the identical 12
tokens, because `_infer_openai_cache_write_tokens` and
`uncached_input_tokens` are computed the same way (`input −
cache_read`). Any consumer summing them gets 2×. The payload also
carries no `cache_inferred` flag, so a reader can't distinguish an
inferred write from an Anthropic-reported one.
- **No skip reason for compression itself.** That entry records
`memory_skip:no_handler` but nothing explains why compression didn't
fire; "below the size floor" and "compressor failed" are
indistinguishable — the same ambiguity #2708 just removed for inflation.
- **`eligible_pct` can still legitimately exceed 100** when a request
genuinely grows after compression (memory injection, proactive
expansion), since `attempted = optimized + saved` uses the forwarded
size. Left alone deliberately: clamping would hide real inflation, and
`tokens_inflated` now expresses it properly.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
Tejas Chopra
|
0e1d6bfa79
|
refactor(pricing): make LiteLLM the source of truth, not the hardcoded table (#2779)
## Description > **Stacked on #2777.** That PR corrects the built-in table's *values*; this one stops the table being *authoritative*. Both are wanted — the fallback should be right **and** not in charge. Merge #2777 first. You asked whether the token/savings code could be simpler and whether the hardcoding could go. This is the hardcoding half, and the encouraging finding is that **almost none of it needed writing** — the infrastructure already existed and was simply unused. `headroom/pricing/litellm_pricing.py` (300 lines, LiteLLM-backed, with an `ImportError` fallback and gateway-prefix handling) has been in the tree the whole time. `ModelInfo`'s own docstring says: > *"Pricing is fetched dynamically from LiteLLM's database. Use `ModelRegistry.estimate_cost()` to get current pricing."* Yet **zero of the four providers called it** (`grep -c litellm_pricing` → openai 0, anthropic 0, google 0, cohere 0). Each kept a parallel hardcoded table. `_get_pricing` had no LiteLLM lookup at all, unlike `get_context_limit` — which is precisely how it went ~18 months stale and priced `gpt-4.1-nano` **300× over**. ## Changes Made **1. Resolution order now mirrors `get_context_limit`**, so limits and prices can't disagree: ``` explicit user config -> LiteLLM -> built-in table -> family -> unknown default ``` Config beats LiteLLM because a configured price is a decision, not a guess. The table stays because it must: the `litellm` dependency is gated `python_version < '3.14'`, and LiteLLM doesn't know every model. It just isn't in charge, so its drift only reaches installs with no LiteLLM. **2. Gateway-routed names now resolve at all.** `litellm.model_cost` keys the *unwrapped* form, so `bedrock/anthropic.claude-...` missed every candidate and silently took the $2.50/$10.00 unknown default: | model | before | after | |---|---|---| | `bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0` | $2.50 / $10.00 | **$3.00 / $15.00** | | `bedrock/us.anthropic.claude-3-5-sonnet-...-v2:0` | $2.50 / $10.00 | **$3.00 / $15.00** | | `vertex_ai/claude-sonnet-4-5` | $2.50 / $10.00 | **$3.00 / $15.00** | | `groq/llama-3.3-70b-versatile` | $2.50 / $10.00 | **$0.59 / $0.79** | | `gemini-2.5-flash` | $2.50 / $10.00 | **$0.30 / $2.50** | | `deepseek-chat` | $2.50 / $10.00 | **$0.28 / $0.42** | `pricing_lookup_candidates` only ever *prepended* provider prefixes. It now also tries progressively unwrapped forms, derived by splitting on `/` — deliberately **not** another hardcoded gateway-prefix list. A wrong guess costs nothing: each candidate is an exact dict lookup, so it just misses. **3. The staleness warning became meaningful.** It fires only when the fallback table is actually used. Before, it was unconditional — and with `_PRICING_LAST_UPDATED = 2025-01-14` against a 60-day window it had been firing for ~18 months, which trains people to ignore it. **4. `pricing_per_1m` rounds to 6dp.** LiteLLM stores cost *per token*, so `× 1e6` leaves float noise ($0.4/1M arrives as `0.39999999999999997`). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Code refactoring ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check` + `ruff format`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality ### Test Output ```text $ pytest tests/test_pricing_from_litellm.py -q 10 passed ``` Full pricing / cost / provider / models / savings / reporting / tokenizer set: ```text $ pytest tests/test_*{pricing,cost,provider,models,savings,utils,reporting,token}*.py -q 3 failed, 1026 passed, 38 skipped pre-existing on main (all three in my recorded baseline): test_bundled_tools_savings.py::test_compressed_payload_preserves_answer_anthropic test_compress_route_tokenizer_by_model.py::...[deepseek/deepseek-v4] (needs transformers) test_compress_route_tokenizer_by_model.py::...[deepseek/deepseek-v4] (needs transformers) ``` Deferring the full sharded run to CI — no maturin/Rust core locally. ### One test of mine changed, and why Three cases in #2777's `test_openai_pricing_resolution.py` failed on exact equality once prices started coming from LiteLLM — `gpt-4.1-mini` arrived as `0.39999999999999997` rather than `0.4`. The values were right; binary floating point isn't exact. Switched those to `pytest.approx(..., abs=0.001)` — money compared to the cent — which passes whether the number comes from LiteLLM or the literal table. ## Deliberately NOT in this PR - **Encodings stay hardcoded.** LiteLLM carries no tiktoken encoding data, and `_lookup_encoding_name`'s `None` return is load-bearing (it's the "not an OpenAI model" signal from #2761). Encodings also track tokenizer generations, not monthly price changes — they aren't the drift problem. - **Anthropic / Cohere / Google providers.** Same shape, same fix, but Anthropic's pricing is a `{input, output, cached_input}` dict rather than a tuple, and its matcher is worse (`if model in known_model or known_model in model` — bidirectional substring). Worth its own PR rather than tripling this diff. - **Context limits.** Already LiteLLM-first; the layering there was correct all along. - `accounts/fireworks/models/kimi-k2` still falls back — LiteLLM genuinely has no entry. The file already shows the pattern for filling such gaps (`_register_minimax_pricing`, `_inject_deepseek_pricing`) if we want it. |
||
|
Tejas Chopra
|
f03cc6d88b
|
fix(router): stop counting an image's base64 payload as suffix tokens (#2778)
## Description
`_netcost_message_tokens` walked block-list content itself and fell back
to `str(block)` for anything that wasn't `text` or `tool_result` — on
the stated assumption that such blocks *"rarely dominate a suffix"*. An
`image` block is the exception that breaks it: `str()` embeds the whole
base64 payload.
```text
counted real over
512x512 PNG 20,034 349 57x
1092x1092 screenshot 100,034 1,589 63x
1568x1568 233,367 1,600 146x
```
**Why this changes behaviour, not just a number.** S is the cache-bust
cost — the tokens re-written if message *j* is mutated. `apply()` builds
it as a running suffix sum:
```python
for j in range(num_messages - 1, -1, -1):
netcost_suffix_tokens[j] = netcost_suffix_tokens[j + 1] + _netcost_message_tokens(...)
```
So one image inflates S for **every message before it**, and the
break-even gate then declines to compress any of them. A single
screenshot could switch off net-cost-gated compression for the whole
earlier conversation — and screenshots are routine in agent sessions.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
Delegate block-list content to `tokenizers.base.count_content_blocks`,
deleting the local walk. That counter already guards exactly this case —
its comment reads *"1MB image = ~330K fake tokens without this"* — so
this walk simply predated it.
Beyond the raw fix, this removes a **second pricing rule**: the gate now
values images the same way the tokenizer that computes
`tokens_before`/`tokens_after` does (a flat 1600, "max after
auto-resize"). Pricing images one way for the gate and another for the
savings math is the same class of problem as #2761.
Verified byte-identical on the shapes the old walk handled correctly:
```text
old walk canonical
text only 101 101
tool_result str 81 81
tool_result list 61 61
image only 100,034 1,600
mixed 100,036 1,602
```
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check` + `ruff format`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for new functionality
### Test Output
```text
$ pytest tests/test_netcost_suffix_image_tokens.py -q
8 passed
$ git stash push headroom/ && pytest tests/test_netcost_suffix_image_tokens.py -q
4 failed, 4 passed
# the 4 failures are the payload-scaling assertions; the 4 passes are the
# text/tool_result/string shapes, included to prove delegation is behaviour-preserving
```
All netcost + content-router suites:
```text
$ pytest tests/test_netcost_gate.py tests/test_content_router_*.py \
tests/test_transforms_content_router.py tests/test_netcost_suffix_image_tokens.py -q
126 passed
```
```text
$ ruff check headroom/transforms/content_router.py tests/... All checks passed!
$ mypy headroom/transforms/content_router.py no new errors
```
Deferring the full suite to CI — no maturin/Rust core in this
environment.
## One existing test rewritten — please look at this bit
`test_netcost_gate.py::TestNetCostHelpers::test_message_tokens_block_list_beats_repr`
fails under the fix, and I want to be explicit that I changed a test
rather than bury it.
It built its image block as `{"type": "image", "source": {"data": "x" *
500}}`. A 500-char stub is **cheaper than a single image's real token
cost**, so `str()` over it looked harmless (~130 tokens) and its
assertion `abs(helper - text_only) < text_only * 0.5` held. That
unrepresentative fixture is precisely why the payload-scaling bug
survived — the test named "beats repr" was passing on the one payload
size where repr happens not to be catastrophic.
Rewritten to use a realistic 200KB payload and to assert what actually
matters:
```python
assert helper >= text_only # text still counted in full
assert helper - text_only <= 2000 # image cost is bounded, not payload-scaled
assert helper < count_text(str(content)) / 10 # ...and far below repr
```
I checked this both ways, so it is a real test and not a rubber stamp:
```text
old test + fixed code -> FAILS (it was pinning the defect)
new test + main -> FAILS (it catches the real bug)
new test + fixed code -> passes
```
## Known limitation
The canonical estimate is a flat 1600 per image regardless of
dimensions, so a small icon is now over-charged (~1600 vs ~13 real)
where repr would have charged ~200. I kept the flat constant
deliberately: it is the value every other counter in the codebase uses,
and introducing a third rule here to shave small-icon cost would
recreate the inconsistency this PR removes. The error is bounded at 1600
tokens and biases the gate conservative, versus an unbounded 100K+ error
before.
|
||
|
Tejas Chopra
|
fc4680b37a
|
fix(tokenizers): resolve gpt-5 and mixed-case model names to the right encoding (#2776)
## Description Two defects in `get_encoding_for_model`, both reachable through the normal `get_tokenizer()` path. **1. `gpt-5` had no prefix entry.** It fell through to `DEFAULT_ENCODING` (`cl100k_base`) instead of `o200k_base`. Same class as the `o4` gap already patched in that tuple. cl100k emits ~33% more tokens than o200k on CJK, so every gpt-5 count was inflated there: ```text CJK sample (30x repeated sentence) o200k_base (correct) 450 tokens cl100k_base (actual) 600 tokens +33.3% ``` Note #2758 taught the *registry* that `gpt-5` → the tiktoken backend; this is the next hop, where that backend picks its *encoding*. So gpt-5 got the right tokenizer family and the wrong encoding inside it. **2. Resolution was case-sensitive.** `TokenizerRegistry.get` lowercases only its **cache key**, then constructs the counter from the caller's original string (`_create_tokenizer(model, backend)`). An uppercase deployment name — routine on Azure, where the deployment name is user-chosen — arrived verbatim, matched no prefix, and took the default encoding. The cache makes this one genuinely unpleasant: key lowercased, construction not, so **the encoding a model receives depends on the casing of whichever request warmed the cache first**, and can differ across restarts. ```text cold cache, uppercase resolved first: GPT-4o -> cl100k_base CJK=600 WRONG GPT-4.1 -> cl100k_base CJK=600 WRONG Gpt-4O-Mini -> cl100k_base CJK=600 WRONG gpt-4o -> o200k_base CJK=450 ok ``` I nearly filed this as "not reachable" — my first check ran the lowercase spelling first, which populated the shared lowercased cache key and masked it completely. The tests call `clear_cache()` so the uppercase spelling resolves cold, which is the failing order. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Added `("gpt-5", "o200k_base")` to the ordered prefix tuple. - `get_encoding_for_model` now lowercases its input. The lowercasing is deliberately scoped to this function rather than the registry: every `MODEL_TO_ENCODING` key is already lowercase (asserted), so it is safe here — whereas lowercasing in `TokenizerRegistry` would break HuggingFace repo ids, which *are* case-sensitive (`Qwen/Qwen3-Coder`). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check` + `ruff format`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality ### Test Output ```text $ pytest tests/test_tokenizer_encoding_resolution.py -q 21 passed $ git stash push headroom/ && pytest tests/test_tokenizer_encoding_resolution.py -q # gpt-5 family, every uppercase case, and both cache-order tests fail ``` Targeted run across the tokenizer/pricing/provider suites, including `test_evals_cjk_tokenization.py` since CJK is the affected content type: ```text $ pytest tests/test_utils.py tests/test_reporting.py tests/test_cost_pricing_warning_dedup.py \ tests/test_pricing.py tests/test_pricing_litellm.py tests/test_provider_model_fallback.py \ tests/test_models.py tests/test_savings_ledger.py tests/test_tokenizers.py \ tests/test_tokenizer.py tests/test_tokenizer_selection_coverage.py \ tests/test_provider_tokenizer_one_ruler.py tests/test_openai_model_table_resolution.py \ tests/test_evals_cjk_tokenization.py -q 233 passed, 16 skipped ``` ```text $ ruff check <changed files> All checks passed! $ mypy headroom/tokenizers/tiktoken_counter.py # only pre-existing release_version.py tomllib redef, present on main ``` Deferring the full suite to CI — this environment has no maturin/Rust core, so the native-dependent shards can't run locally. ## Real Behavior Proof - **Environment:** macOS, Python 3.13.7, isolated worktree at `upstream/main` (`0cb72f45`). - **Exact command / steps:** `TokenizerRegistry.clear_cache()`, then resolve each spelling cold and count a CJK sample. - **Observed result:** ```text before after gpt-5 cl100k_base CJK=600 o200k_base CJK=450 gpt-5-mini cl100k_base CJK=600 o200k_base CJK=450 GPT-4o cl100k_base CJK=600 o200k_base CJK=450 GPT-4.1 cl100k_base CJK=600 o200k_base CJK=450 Gpt-4O-Mini cl100k_base CJK=600 o200k_base CJK=450 GPT-4 cl100k_base CJK=600 cl100k_base CJK=600 (unchanged, correct) gpt-4o o200k_base CJK=450 o200k_base CJK=450 (unchanged) gpt-3.5-turbo cl100k_base CJK=600 cl100k_base CJK=600 (unchanged) ``` `GPT-4` was previously "correct" only by accident — it missed every prefix and landed on `DEFAULT_ENCODING`, which happens to be `cl100k_base`. It is now correct by resolution. |
||
|
Tejas Chopra
|
0cb72f45b2
|
fix(providers): stop a shorter model family shadowing a longer one (#2762)
## Description > **Stacked on #2761** — that PR splits `_lookup_encoding_name` out of `_get_encoding_name_for_model`, which this one builds on. Please merge #2761 first; the diff here will shrink to just this commit afterwards. `_MODEL_ENCODINGS` and `_CONTEXT_LIMITS` are matched by prefix, iterating in **plain dict order** — so the first *inserted* prefix wins rather than the most specific one. `gpt-4.1` matched the `gpt-4` entry: | model | resolved | actual | | |---|---|---|---| | `gpt-4.1` | 8192 | 1,047,576 | **128× under** | | `gpt-4.1-mini` | 8192 | 1,047,576 | **128× under** | | `gpt-4.1-nano` | 8192 | 1,047,576 | **128× under** | | `gpt-4-32k-0613` | 8192 | 32,768 | 4× under | | `gpt-5` / `-mini` / `-nano` | 128,000 | 400,000 | fell to unknown-model default | | `o4-mini` | 128,000 | 200,000 | fell to unknown-model default | A 128× under-estimate matters because the context limit is what tells the proxy how much headroom is left: it treats a 1M-context model as nearly full and compresses accordingly. The same shadowing picked the **wrong encoding** — `gpt-4.1` got `cl100k_base` instead of `o200k_base`. Measured cost of that: ```text cl100k o200k error python code 420 420 +0.0% json blob 555 555 +0.0% logs 580 580 +0.0% english 201 201 +0.0% CJK 600 450 +33.3% ``` So the encoding half is narrow but real — it only bites CJK content, which the repo already treats as a case worth testing (`tests/test_evals_cjk_tokenization.py`). **Scope honestly:** `get_context_limit` consults LiteLLM *before* this table, so the limit half only surfaces where LiteLLM is absent or does not know the model. That is not hypothetical — the `litellm` dependency carries a `python_version < '3.14'` marker (`pyproject.toml:56`), so **any install on Python 3.14+ has no LiteLLM** and this table is load-bearing. The encoding half never had a LiteLLM fallback and was always wrong. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Both prefix loops now iterate `sorted(..., key=len, reverse=True)` — longest prefix wins. This is the root-cause fix: it also protects the *next* model added to these tables. - Added the missing families: `gpt-4.1` (+`-mini`/`-nano`), `gpt-5` (+`-mini`/`-nano`), `o4-mini` to both tables. - Left `supports_model`'s prefix loop alone — it only returns a bool, so order cannot change its answer. Not touched: `_PRICING`. `gpt-4.1`/`gpt-5` also fall through to the GPT-4o pricing tier, which skews cost reporting, but that is a separate concern with its own verification burden (published rates, staleness window) and does not belong in a tokenizer-correctness fix. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check` + `ruff format`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality ### Test Output 12 of 24 new cases fail without the fix; the 12 that pass are the "must not regress" rows (`gpt-4`, `gpt-4-turbo`, `gpt-4o`, `o3`, `gpt-3.5-turbo`) — included precisely so the longest-prefix change can't quietly move them: ```text $ git stash push headroom/providers/openai.py && pytest tests/test_openai_model_table_resolution.py -q FAILED ...::test_context_limit_prefers_the_most_specific_prefix[gpt-4.1-mini-1047576] FAILED ...::test_context_limit_prefers_the_most_specific_prefix[gpt-4.1-nano-1047576] FAILED ...::test_context_limit_prefers_the_most_specific_prefix[gpt-4.1-2025-04-14-1047576] FAILED ...::test_context_limit_prefers_the_most_specific_prefix[gpt-4-32k-0613-32768] FAILED ...::test_context_limit_prefers_the_most_specific_prefix[gpt-5-400000] FAILED ...::test_context_limit_prefers_the_most_specific_prefix[gpt-5-mini-400000] FAILED ...::test_context_limit_prefers_the_most_specific_prefix[o4-mini-200000] FAILED ...::test_encoding_prefers_the_most_specific_prefix[gpt-4.1-o200k_base] FAILED ...::test_encoding_prefers_the_most_specific_prefix[gpt-4.1-mini-o200k_base] FAILED ...::test_encoding_prefers_the_most_specific_prefix[gpt-4.1-2025-04-14-o200k_base] FAILED ...::test_cjk_is_not_over_counted_for_gpt_41 12 failed, 12 passed in 0.70s $ git stash pop && pytest tests/test_openai_model_table_resolution.py -q 24 passed in 0.43s ``` Regression check — 119 suites touching openai / cost / savings / token / compress / outcome / budget, this branch vs clean `main` in the same environment, comparing failure *sets*: ```text branch : 5 failed, 1486 passed, 86 skipped in 111.01s main : 5 failed, 1453 passed, 86 skipped in 132.07s NEW failures introduced: (none) pre-existing on both: test_bundled_tools_savings.py::test_compressed_payload_preserves_answer_anthropic test_compress_route_tokenizer_by_model.py::...[deepseek/deepseek-v4] (needs `transformers`) test_compress_route_tokenizer_by_model.py::...[deepseek/deepseek-v4] (needs `transformers`) test_image_compressor_singleton_reuse.py::test_onnx_router_is_built_once_and_cached test_openai_streaming_backend.py::...test_litellm_vertex_streaming_preserves_max_tokens_and_vendor_fields ``` ```text $ ruff check headroom/providers/openai.py tests/test_openai_model_table_resolution.py All checks passed! $ mypy headroom/providers/openai.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - **Environment:** macOS, Python 3.13.7, isolated worktree at `upstream/main` (`ad56dd38`), no `litellm` installed (matching a Python 3.14+ install, where the dep marker excludes it). - **Exact command / steps:** resolve context limit + encoding for each model against published OpenAI values, before and after. - **Observed result:** ```text before after model limit enc model limit enc gpt-4.1 8192 cl100k gpt-4.1 1047576 o200k_base gpt-4.1-mini 8192 cl100k gpt-4.1-mini 1047576 o200k_base gpt-4.1-nano 8192 cl100k gpt-4.1-nano 1047576 o200k_base gpt-4.1-2025-04-14 8192 cl100k gpt-4.1-2025-04-14 1047576 o200k_base gpt-4-32k-0613 8192 cl100k gpt-4-32k-0613 32768 cl100k_base gpt-5 128000 o200k gpt-5 400000 o200k_base o4-mini 128000 o200k o4-mini 200000 o200k_base unchanged: gpt-4=8192/cl100k, gpt-4-turbo=128000/cl100k, gpt-4o=128000/o200k, o3=200000, gpt-3.5-turbo=16385/cl100k ``` |
||
|
Tejas Chopra
|
cd92ed52ff
|
fix(providers): give every model exactly one tokenizer (#2761)
## Description `/v1/chat/completions` is a multi-provider passthrough, but `OpenAIProvider` handed **any** unrecognized model a guessed `o200k_base` encoding. Kimi through Fireworks — a documented Headroom configuration — counted **~19% low**. This is not just an accuracy nit, because **two resolvers race on the same request**: - handlers count via the tokenizer registry (`count_tokens_offloaded` → `get_tokenizer(model)`) - `TransformPipeline` counts via `provider.get_token_counter(model)`, because the proxy builds its pipelines with `provider=self.openai_provider` (`server.py:953-958`) `tokens_saved = original_tokens - optimized_tokens`, and in token mode those two operands come from *different* resolvers (`handlers/openai.py:3150-3155` keeps the handler's `original_tokens` and takes the pipeline's `optimized_tokens`). When the rulers disagree the subtraction is noise — it can invent savings on an untouched request, or trip the `optimization inflated tokens` revert guard at `handlers/openai.py:3219` and throw away real compression. Measured on `main` before this change, same 2-message payload: | model | registry (handler) | provider (pipeline) | gap | |---|---|---|---| | `moonshotai/kimi-k2` | 686 | 554 | **19.2%** | | `accounts/fireworks/models/kimi-k2-instruct` | 686 | 554 | **19.2%** | | `gemini-2.5-pro` | 534 | 554 | 3.7% | | `command-r-plus` | 534 | 554 | 3.7% | | `mistral-large-latest` | 563 | 554 | 1.6% | | `gpt-4o` / `claude-sonnet-4-6` | 552 | 554 | 0.4% | The provider returned **554 for every model** — it was model-blind. This follows the precedent already documented in `tests/test_compress_route_tokenizer_by_model.py`: pinning one provider's counter for a multi-model route is the bug, and the registry is the canonical resolver (every registry tokenizer derives from `BaseTokenizer`, whose `_count_content_parts` ends in a serialize-and-count catch-all). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `_get_encoding_name_for_model` split into `_lookup_encoding_name` (returns `None` when nothing claims the model) plus the original fallback wrapper, so callers can distinguish "OpenAI model" from "guessed". - `OpenAIProvider.get_token_counter` defers to `get_tokenizer(model)` when no real tiktoken encoding claims the model. - Per-message overhead `4` → `3`. OpenAI's counting guide uses `tokens_per_message = 3` for every model since `gpt-3.5-turbo-0613`; only the retired `gpt-3.5-turbo-0301` used 4. Staying on 4 over-counted every message by one token *and* disagreed with the registry, so a 100-message conversation drifted by 100 tokens depending on who counted it. - `_token_counters` annotation widened to `dict[str, TokenCounter]`. Preserved deliberately: explicit `model -> encoding` mappings (custom config / `HEADROOM_MODEL_LIMITS`) still win, and genuine OpenAI models still use `OpenAITokenCounter`. Both are pinned by tests. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check` + `ruff format`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality ### Test Output New tests fail on `main` and pass here — 7 of 9 fail without the fix (the 2 that pass are the invariants the fix must not break): ```text $ git stash push headroom/providers/openai.py && pytest tests/test_provider_tokenizer_one_ruler.py -q FAILED ...::test_non_openai_models_resolve_to_the_registry_tokenizer[moonshotai/kimi-k2] FAILED ...::test_non_openai_models_resolve_to_the_registry_tokenizer[accounts/fireworks/models/kimi-k2-instruct] FAILED ...::test_non_openai_models_resolve_to_the_registry_tokenizer[gemini-2.5-pro] FAILED ...::test_non_openai_models_resolve_to_the_registry_tokenizer[command-r-plus] FAILED ...::test_non_openai_models_resolve_to_the_registry_tokenizer[claude-sonnet-4-6] FAILED ...::test_kimi_is_not_counted_with_an_openai_encoding FAILED ...::test_per_message_overhead_matches_openai_and_the_registry 7 failed, 2 passed in 0.49s $ git stash pop && pytest tests/test_provider_tokenizer_one_ruler.py -q 9 passed in 0.51s ``` Regression check — 119 suites touching openai / cost / savings / token / compress / outcome / budget, run on this branch and on clean `main` **in the same environment**, comparing failure *sets*: ```text branch : 5 failed, 1453 passed, 86 skipped in 97.60s main : 5 failed, 1453 passed, 86 skipped in 132.07s NEW failures introduced by fix: (none) pre-existing on both: test_bundled_tools_savings.py::test_compressed_payload_preserves_answer_anthropic test_compress_route_tokenizer_by_model.py::...[deepseek/deepseek-v4] (needs `transformers`) test_compress_route_tokenizer_by_model.py::...[deepseek/deepseek-v4] (needs `transformers`) test_image_compressor_singleton_reuse.py::test_onnx_router_is_built_once_and_cached test_openai_streaming_backend.py::...test_litellm_vertex_streaming_preserves_max_tokens_and_vendor_fields ``` ```text $ ruff check headroom/providers/openai.py tests/test_provider_tokenizer_one_ruler.py All checks passed! $ mypy headroom/providers/openai.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - **Environment:** macOS, Python 3.13.7, isolated worktree at `upstream/main` (`ad56dd38`), in-place `headroom/_core.abi3.so` copied in. - **Exact command / steps:** resolve both tokenizers for the same 2-message payload and compare, before and after. - **Observed result:** ```text before (main) gpt-4o registry=552 provider=554 DIVERGES 2 moonshotai/kimi-k2 registry=686 provider=554 DIVERGES 132 (19.2%) gemini-2.5-pro registry=534 provider=554 DIVERGES 20 (3.7%) command-r-plus registry=534 provider=554 DIVERGES 20 (3.7%) after (this branch) gpt-4o registry=552 provider=552 AGREE gpt-5 registry=552 provider=552 AGREE o4-mini registry=552 provider=552 AGREE claude-sonnet-4-6 registry=552 provider=552 AGREE moonshotai/kimi-k2 registry=686 provider=686 AGREE gemini-2.5-pro registry=534 provider=534 AGREE command-r-plus registry=534 provider=534 AGREE ``` ## Known remaining gap (deliberately not in this PR) Plain and `name`-bearing messages now agree exactly, but **tool-call accounting still differs** on genuine OpenAI models: ```text tool msg (tool_call_id) registry=11 provider=13 delta +2 assistant tool_calls registry=14 provider=21 delta +7 ``` `OpenAITokenCounter` adds flat guesses (`+10` per tool call, `+2` per `tool_call_id`); the registry serializes the real structure and counts it. I believe the registry is closer to what the model actually sees, but I could not ground-truth it — there are no recorded `usage.prompt_tokens` fixtures in `tests/parity/`, and I did not want to shift everyone's tool-heavy numbers on a hunch. Tool-heavy agent traffic is the dominant Headroom workload, so this deserves its own PR with a real API capture to compare against. Filing separately. |
||
|
Tejas Chopra
|
ad56dd382b
|
fix(router): compare token quantities in one unit (#2759)
## Description Two places compared a token quantity against something measured in a **different unit**. Both changed compression **behaviour**, not just reporting — which is the worse class. ### 1. The CONFIG branch put a word count in a token ratio `compressed_tokens = len(compressed.split())` was divided by `original_tokens`, which comes from `_estimate_tokens(content)`. Words run ~2.8× fewer than estimator tokens on config text, so a compressor that returned its input **byte-identically** scored ~0.36. `min_ratio` is 1.0 — accept any real shrink — so the router **accepted the no-op**: cached the result, pinned a frozen "compress" verdict, emitted a `router:config_compressor` label into `transforms_applied`, and recorded a fabricated saving to TOIN. ```text mkdocs.yml, compressor returns its input unchanged denominator (_estimate_tokens) = 936 OLD numerator len(split()) = 334 -> ratio 0.357 claims 64% saved ACCEPTED NEW numerator (_estimate_tokens) = 936 -> ratio 1.000 correctly rejected ``` The sibling TABULAR branch already used `_estimate_tokens`; CONFIG was the outlier. ### 2. The Kompress size gate tested a token cap in chars/4 `len(text_to_compress) > self._kompress_max_tokens * 4` under-counts anything denser than 4 chars/token, and compact JSON runs ~3.2. Against the 50,000-token default there is a band where an oversized payload passes: ```text records chars old_gate(len/4) new_gate(tokens) 2700 119,281 False False 4000 177,781 False True <- 44,445 vs 55,557 tokens, 11% over 4400 195,781 False True <- 48,945 vs 61,182 tokens, 22% over 5000 222,781 True True ``` Those payloads entered ONNX inference — exactly the >30s non-preemptible worker stall the gate exists to prevent (#1171). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `content_router.py` CONFIG branch — `compressed_tokens` now from `_estimate_tokens(compressed)`, matching its denominator and every sibling branch. - `content_router.py` Kompress gate — compared with `_estimate_tokens`, the unit the cap is actually expressed in. The extra O(n) char scan is negligible against the inference it guards. ## Testing - [x] Unit tests pass - [x] Linting passes (`ruff check` + `format --check`, pinned 0.15.17) - [x] New tests added - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_content_router_token_units.py -q 4 passed in 0.33s $ uvx ruff@0.15.17 check headroom/transforms/content_router.py tests/test_content_router_token_units.py All checks passed! ``` **Regression check against clean `upstream/main` in the same environment:** ```text tests/test_transforms/ + test_transforms_content_router.py + kompress suites upstream/main : 2 failed, 468 passed, 78 skipped this branch : 2 failed, 468 passed, 78 skipped failure sets : identical ``` The 2 failures are `test_kompress_failsafe`'s artifact-selection tests, which need a real `onnxruntime` this throwaway env lacks. Unrelated to this change. The 4 new tests pin the unit contract *and* the bounds of the disagreement band — including the cases where both formulations agree, so the band is demonstrated rather than assumed. ## Real Behavior Proof - **Environment:** macOS 26.4 arm64, isolated worktree off `upstream/main`. `content_router` needs the compiled `headroom._core`, which isn't in a fresh worktree (gitignored, built in-place); I copied the built `.so` in to run these, then removed it before committing. - **Observed:** both tables above are from running the real `_estimate_tokens` against the real thresholds, not reconstructed arithmetic. - **Not tested:** no live ONNX inference — the >30s stall the gate prevents is cited from #1171, not reproduced. The CONFIG no-op was demonstrated at the ratio level rather than by driving a stubbed compressor through `apply()`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] I did **not** edit `CHANGELOG.md` ## Related Third of three PRs from one tokenizer-consistency audit — see #2757 (litellm total-prompt / `--budget`) and #2758 (HuggingFace chat templates, `gpt-5`, gateway-wrapped names). Separate subsystems, separate risk. Known remaining from the same audit, not in any of the three: `_netcost_message_tokens` pricing an image by Python `repr` (34× over-count, flag-gated), three transforms reporting via `count_text(str(content))` where the pipeline uses `count_messages` (19% apart in one log file), `frozen_message_count` walking a chars/3.5 estimate against provider-reported cached tokens, and `target_ratio` honoured in words while documented as tokens. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Tejas Chopra
|
a033ac4176
|
fix(cost): send litellm the total prompt so --budget stops seeing $0 (#2757)
## Description **`--budget` has been silently inert on any cache-warm request** — which is the normal case in an agent session. This is a disabled control, not a metrics bug. `record_tokens` passed only the **uncached slice** as litellm's `prompt_tokens`. Measured, `litellm.cost_per_token` charges: ``` (prompt_tokens - cache_read - cache_creation) * input_rate + cache_read * read_rate + cache_creation * write_rate ``` So `prompt_tokens` is the **whole** prompt and litellm removes the cached parts itself. Handing it the uncached slice drives the input term **negative** as soon as anything is cached. `estimate_cost` ends with `float(total) if total > 0 else None`, so it returned `None`, no `CostEntry` was appended, and `check_budget()` saw **$0**. | model | 100k prompt, 80k cached — old call | booked | | --- | --- | --- | | `gpt-5` | **-$0.065000** | None | | `gpt-4o-mini` | **-$0.003000** | None | | `claude-sonnet-4-5` | **-$0.156000** | None | Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made Fixing the total alone would **over-charge OpenAI**, because two bugs are entangled here. OpenAI exposes no cache-write counter, so `_infer_openai_cache_write_tokens` uses the uncached portion as a write proxy. At two of the four inference sites (`openai.py:4304`, `openai.py:5413`) `uncached_input_tokens` is derived by subtracting **only** `cache_read`, so `cache_write_tokens` and `uncached_tokens` are **the same tokens**. Summing all three would double-count the prompt and charge a write premium OpenAI does not have. (The other two sites — `openai.py:3969` and `streaming.py:2024` — subtract both, so their buckets are genuinely disjoint; those are left alone.) - `cost.py` — `record_tokens` now passes `uncached + cache_read + cache_write` as the prompt total. - `cost.py` — new `cache_inferred: bool = False` parameter. When set, the inferred write is excluded from **both** the prompt total and the write premium. The default preserves behaviour for every provider that reports disjoint buckets. - `outcome.py` — plumbs `outcome.cache_inferred` through. The field already existed on `RequestOutcome` for the dashboard; it just never reached cost. - `handlers/openai.py` — sets `cache_inferred=True` at the two outcome sites whose buckets genuinely duplicate. ## Testing - [x] Unit tests pass (new file) - [x] Linting passes (`ruff check` + `format --check`, pinned 0.15.17) - [x] New tests added - [x] Manual testing performed ### Test Output ```text $ uvx ruff@0.15.17 check headroom/proxy/cost.py headroom/proxy/outcome.py headroom/proxy/handlers/openai.py All checks passed! $ uvx ruff@0.15.17 format --check <same three> 3 files already formatted $ pytest tests/test_cost_budget_total_prompt.py -q 5 passed in 0.26s ``` The 5 new tests assert on the **arguments handed to `estimate_cost`** rather than on dollar values, so they pin the contract that broke without depending on litellm's pricing tables — or on litellm being installed at all. ## Real Behavior Proof - **Environment:** macOS 26.4 arm64, isolated git worktree off `upstream/main`. **litellm's actual formula**, established by probe rather than assumed: ```text rates (claude-sonnet-4-5): input=3e-06 read=3e-07 write=3.75e-06 total=100k, 80k read, 5k write -> $0.087750 hypothesis (p-r)*ir + r*rr + w*wr = $0.102750 x hypothesis (p-r-w)*ir + r*rr + w*wr = $0.087750 <- matches ``` **Before → after:** ```text Anthropic, disjoint: uncached=900 read=48000 write=1500 OLD prompt=900 raw=-0.125775 booked=None <-- BUDGET BLIND NEW prompt=50400 raw= 0.022725 booked=0.022725 OpenAI, inferred write == uncached: uncached=20000 read=80000 write=20000 OLD prompt=20000 raw=-0.090000 booked=None <-- BUDGET BLIND NEW prompt=100000, write excluded raw= 0.035000 booked=0.035 truth=0.035000 ``` The OpenAI "after" equals the hand-computed truth `20,000*input + 80,000*read_rate` exactly. - **Not fully tested locally.** The cost/outcome suites show an **identical 10-failure set** on this branch and on clean `upstream/main` in the same throwaway env — all `ModuleNotFoundError: headroom._core`, the compiled Rust extension this machine cannot currently build. So they are environment-only, not regressions. One of them, `test_funnel_passes_canonical_record_tokens_shape`, covers the `record_tokens` call shape this PR changes, so **CI is the authoritative check for that one**. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] I did **not** edit `CHANGELOG.md` ## Context Found during a tokenizer-consistency audit that also turned up: HuggingFace-routed models counting a 6,000-char message as **2 tokens**, `gpt-5`/`o4-mini` falling to a char estimator, and the Kompress size gate missing its own cap by 24%. Those are separate PRs — different subsystems, different risk. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Tejas Chopra
|
06add9e9d8
|
fix(providers): stop pricing modern content blocks at zero (#2760)
## Description Each token counter in `headroom/providers/` had grown its own shortened content-block walker, handling only the shapes its provider was expected to send. Everything else fell through and contributed **nothing**. Measured on one 6,800-char block, via `count_messages` of a single-block message — so 7–8 is message overhead alone: | block type | OpenAI ctr | Anthropic ctr | |---|---|---| | `text` (control) | 3409 | 3748 | | `tool_result` | **8** | 3748 | | `thinking` | **8** | **7** | | `document` | **8** | **7** | | `mcp_tool_result` | **8** | **7** | | `output_text` | **8** | **7** | | `refusal` | **8** | **7** | Two things make this worse than a coverage gap: 1. **Each counter zeroed blocks from its own provider.** `output_text` and `refusal` are OpenAI Responses shapes; `thinking` and `document` are Anthropic's. 2. **These are the counters the live pipelines use.** `proxy/server.py` builds them with `AnthropicProvider` / `OpenAIProvider`, so this is the main request path — not an edge case. #2743 fixed this for `/v1/compress` only, by routing that route to the registry tokenizers, whose `BaseTokenizer` walker is complete. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made Rather than add a **fifth** partial walker, the counters now delegate to the audited one: - `tokenizers/base.py` — new `count_content_blocks(parts, count_text_fn)` plus a thin `_DelegatingBlockCounter` adapter, since the provider counters are not `BaseTokenizer` subclasses. `BaseTokenizer` itself is untouched. - `providers/openai.py`, `providers/anthropic.py`, `providers/openai_compatible.py` — list-content branches delegate. **Why delegate instead of adding a `count_text(str(block))` catch-all:** that would serialize a base64 blob and price it as text. `tiktoken_counter.py` already documents the failure — a 1MB image becomes ~330K phantom tokens. The shared walker gives media a pixel/byte-based estimate. **Scope:** the three counters that accumulate token counts. `google.py` and `cohere.py` extract a *text string* first and count that, so the same defect there needs a differently-shaped fix — left as a follow-up. ## Testing - [x] Unit tests pass - [x] Linting passes (`ruff check` + `format --check`, pinned 0.15.17) - [x] New tests added - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_provider_counter_content_blocks.py -q 12 passed in 0.52s $ uvx ruff@0.15.17 check headroom/ tests/... --exclude headroom/dashboard/templates All checks passed! ``` **After the fix**, every shape lands within ~1% of the equivalent plain text, and media stays bounded: ```text block OpenAI Anthropic text (control) 3409 3748 tool_result 3409 3748 thinking 3419 3759 document 3423 3763 mcp_tool_result 3422 3762 output_text 3420 3760 refusal 3421 3761 image b64 200KB 1608 1607 <- pixel estimate, not ~50K as text ``` ## Real Behavior Proof — including a regression I caught **This change flipped an existing test**, and I only found it because every suite was run against clean `upstream/main` in the same environment with the failure sets diffed: ```text before the test rewrite: upstream/main : 1 failed, 104 passed this branch : 2 failed, 103 passed <- regression diff : + test_openai_compatible_token_counter_ignores_unhandled_content_shapes ``` That test asserted `content: [{"type": "image"}, 123] == 8` — i.e. it **pinned the defect**, that unhandled shapes contribute nothing. Rewritten as `..._prices_declared_media`: a declared image is now priced (1608) while a bare int is still correctly ignored (8), with the rationale in the docstring. ```text after the rewrite: upstream/main : 1 failed, 104 passed, 10 skipped, 25 errors this branch : 1 failed, 104 passed, 10 skipped, 25 errors failure sets : IDENTICAL ``` - **Pre-existing, not from this change:** the 1 failure and all 25 errors. The errors are all in `test_compress_route_tokenizer_by_model.py`, whose loopback `TestClient` fixture this throwaway env cannot satisfy. - **Environment note:** `content_router` and several suites need the compiled `headroom._core`, which isn't in a fresh worktree (gitignored, built in-place). I copied the built `.so` in to run these and removed it before committing. - **Not tested:** no live provider call, so the *absolute* accuracy of the 1600 image estimate against a real Anthropic/OpenAI bill is unverified — it is the value `BaseTokenizer` already used, and this PR only changes which blocks reach it. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] I did **not** edit `CHANGELOG.md` ## Related Fourth PR from one tokenizer-consistency audit: #2757 (litellm total prompt / `--budget`), #2758 (HuggingFace chat templates, `gpt-5`, gateway-wrapped names), #2759 (router token units). Plus #2756, which splits the local/provider token scales in `RequestOutcome`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Tejas Chopra
|
0ed306b22b
|
fix(tokenizers): count HuggingFace chat templates, and resolve gpt-5 / gateway-wrapped names (#2758)
## Description Three tokenizer-selection defects, all measured against real counters on identical text. ### 1. HuggingFace-routed models counted a whole conversation as **2 tokens** `transformers >= 5` defaults `apply_chat_template(tokenize=True)` to `return_dict=True` and returns a `BatchEncoding`, so `len(formatted)` counted **dict keys** — `input_ids`, `attention_mask` — instead of tokens. ```text Qwen2.5-72B, one 6,000-char message before: count_messages = 2 count_message = -1 after : count_messages = 1020 count_message = 1017 true : ~1003 ``` `count_message` goes negative because `BaseTokenizer` subtracts a 3-token reply overhead from it. A **~99.8% undercount** on every HF-routed family whose resolved tokenizer carries a chat template — llama, qwen, deepseek, phi, yi, falcon, starcoder. `pyproject.toml` pins `transformers>=5.5.0,<6.0`, so the affected version is the only installable one, and nothing covered `count_messages`. It hid behind a second bug while I reproduced it: `DeepSeek-V3` mis-resolves to `deepseek-llm-7b-base` (a 2023 model with **no** chat template), which falls back to the estimator and looks fine. That mis-resolution is left for a follow-up. ### 2. The current OpenAI flagships had no pattern `MODEL_PATTERNS` stopped at `^gpt-4` / `^o1` / `^o3`: ```text gpt-5, gpt-5.1, gpt-5-mini, gpt-5.1-codex, o4-mini -> EstimatingTokenCounter ``` Deviation vs the correct `o200k` encoding: **+20% English, -33% JSON, -44% logs.** ### 3. Every pattern is `^`-anchored, so gateway-wrapped ids matched nothing ```text bedrock/anthropic.claude-3-5-sonnet -> EstimatingTokenCounter vertex_ai/claude-sonnet-4-6 -> EstimatingTokenCounter openrouter/anthropic/claude-sonnet-4-6 -> EstimatingTokenCounter us.anthropic.claude-sonnet-4-6-v1:0 -> EstimatingTokenCounter azure/gpt-4o -> EstimatingTokenCounter ``` Deviation: **+15% English, -33% JSON, -38% logs.** Not hypothetical — `handlers/openai.py` already documents that LiteLLM's `headroom` guardrail passes exactly these forms. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `tokenizers/huggingface.py` — pass `return_dict=False` to `apply_chat_template`. - `tokenizers/registry.py` — add `^gpt-5` and `^o4` to `MODEL_PATTERNS`. - `tokenizers/registry.py` — new `_name_candidates()`; `_detect_backend` now tries progressively-unwrapped forms: path segments stripped left-to-right, then Bedrock's dotted `[region.]vendor.model`. **Why candidates rather than rewriting the name:** the full name is candidate 0, so no currently-correct resolution can move, and an unknown alias still falls back to estimation rather than matching by accident. The estimator is a legitimate *fallback*; the bug was reaching it when a real tokenizer for that family exists. ## Testing - [x] Unit tests pass - [x] Linting passes (`ruff check` + `format --check`, pinned 0.15.17) - [x] New tests added - [x] Manual testing performed ### Test Output ```text $ uvx ruff@0.15.17 check headroom/ tests/test_tokenizer_selection_coverage.py --exclude headroom/dashboard/templates All checks passed! $ pytest tests/test_tokenizer_selection_coverage.py -q 20 passed in 0.60s $ pytest tests/test_huggingface_tokenizer_timeout.py tests/test_tokenizers/ -q this branch: 12 passed clean upstream/main: 12 passed <- no regression ``` 20 new tests cover all three defects **plus** the no-regression cases: bare names unchanged, unknown aliases still estimated, wrapped Gemini matching its bare form exactly, and candidate ordering/dedup. ## Real Behavior Proof - **Environment:** macOS 26.4 arm64, isolated worktree off `upstream/main`. The HF measurement used a real `transformers 5.14.1` with `Qwen/Qwen2.5-72B` from the local HF cache. **After the fix, resolution across every form a gateway realistically sends:** ```text gpt-4o TiktokenCounter gpt-5 TiktokenCounter <- was Estimating gpt-5.1 TiktokenCounter <- was Estimating o3-mini TiktokenCounter o4-mini TiktokenCounter <- was Estimating claude-sonnet-4-6 TiktokenCounter bedrock/anthropic.claude-3-5-sonnet TiktokenCounter <- was Estimating anthropic.claude-3-5-sonnet-20241022-v2:0 TiktokenCounter <- was Estimating us.anthropic.claude-sonnet-4-6-v1:0 TiktokenCounter <- was Estimating vertex_ai/claude-sonnet-4-6 TiktokenCounter <- was Estimating openrouter/anthropic/claude-sonnet-4-6 TiktokenCounter <- was Estimating azure/gpt-4o TiktokenCounter <- was Estimating vertex_ai/gemini-2.5-pro EstimatingTokenCounter (google backend, correct) groq/llama-3.3-70b-versatile HuggingFaceTokenizer <- was Estimating my-gateway/big-model EstimatingTokenCounter (correct fallback) ``` `vertex_ai/gemini-2.5-pro` and `gemini-2.5-pro` return **identical** counts (600 on the same input), confirming the prefix strip reaches the google backend rather than the generic fallback. - **Not fully tested locally:** `tests/test_evals_cjk_tokenization.py` cannot collect in this env — `ModuleNotFoundError: headroom._core`, the compiled Rust extension this machine can't currently build. Identical on baseline, so CI is the check there. It is CJK-related and this PR changes encoding selection for `gpt-5`/`o4`/wrapped names, so it's the suite most worth watching. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] I did **not** edit `CHANGELOG.md` ## Known follow-ups, deliberately not here - `DeepSeek-V3` → `deepseek-llm-7b-base`, `Qwen/Qwen2.5-72B` → `Qwen/Qwen-7B`: `get_tokenizer_name` prefix-matches against the whole string including the org segment, and has no version boundary. - `get_encoding_for_model` is case-sensitive while `_detect_backend` lowercases, so `GPT-4O` gets `cl100k` (+38.9% on CJK). - `providers/openai.py` has a second, divergent encoding resolver — it disagrees with `tokenizers/` on `gpt-4.1`, `gpt-5`, `text-embedding-3-large`, `davinci`. - Provider counters price most modern content blocks at literally zero (`thinking`, `document`, `mcp_tool_result`, and OpenAI's own `output_text`/`refusal`). 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
nangsontay
|
184146b688
|
fix(savings): surface request growth the tok_saved clamp swallows (#2708)
## Description `tokens_saved` is clamped at zero, so a request the proxy forwards **larger** than it arrived is indistinguishable in the PERF line from one it simply could not compress. Both read `tok_saved=0`. That ambiguity hides real regressions. Anything that appends to the body after compression — proactive context expansion, memory injection — can outweigh the compression it sits on top of and still look like a neutral turn. On the session that prompted this, a request went from 55,161 tokens in to 57,845 out and reported `tok_saved=0`, for 19 consecutive turns, with nothing in the logs distinguishing it from a turn with nothing left to compress. This reports the swallowed amount as `tok_inflated`. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `RequestOutcome.tokens_inflated`: `max(0, optimized_tokens - original_tokens)`, derived from two counts the outcome already carries — **no new plumbing at any of the emit sites**. - Added `tok_inflated=` to the PERF log line, next to `tok_saved=`. Diagnostic only, deliberately. It does **not** feed `tokens_saved` or `attempted_input_tokens`, for two reasons: 1. `attempted_input_tokens = optimized_tokens + tokens_saved` is a *size*, not a signed delta. Letting the second term go negative makes it smaller than the bytes actually forwarded, corrupting the active-savings denominator. 2. Injection paths already book their own cost through the retrieval-drawback channel. A negative landing in `tokens_saved` as well would count the same loss twice. So the clamp stays and the hidden number surfaces beside it. Worth noting there is already a revert-on-inflation guard *before* compression's own inflation can escape (`anthropic.py`, "Optimization inflated tokens … reverting to original messages") — it is only growth added *after* that point which the clamp was silently absorbing. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_request_outcome.py tests/test_cli_perf_format.py -q ============================== 54 passed in 1.56s ============================== $ pytest tests/ -q -k "outcome or perf or savings or stats" ======= 510 passed, 33 skipped, 9674 deselected, 542 warnings in 40.37s ======== $ ruff check headroom/proxy/outcome.py tests/test_request_outcome.py All checks passed! $ ruff format --check headroom/proxy/outcome.py tests/test_request_outcome.py 2 files already formatted $ mypy headroom/proxy/outcome.py --ignore-missing-imports Success: no issues found in 1 source file ``` Four new tests pin the distinction that was missing: shrank (0), no-op compression (0, and `tok_saved` also 0 — the two cases that used to look identical), grew (reports 2684 while `tok_saved` stays 0), and that `attempted_input_tokens` / `savings_pct` keep their unsigned semantics. `tests/test_cli_perf_format.py` parses hand-written PERF fixtures by field name, so adding a field does not disturb it — verified green above. ## Real Behavior Proof ### The field catching a real inflating request - Environment: macOS 15 (arm64), Python 3.13.14. A proxy booted from this branch: `headroom proxy --mode token --backend anthropic --anthropic-api-url http://127.0.0.1:<stub>`, isolated `HOME` so the run could not touch a developer's live logs/store, `HF_HOME` pointed at cached kompress weights so the lossy+CCR-marker path is exercised and proactive expansion can actually arm. - Exact command / steps: three requests over one conversation through the real HTTP path (`x-headroom-cwd: /tmp/proof`, `user-agent: claude-code/1.4.2`): a user turn carrying the real `~/.claude/rules/*.md` text (~8.2k tokens); then `assistant` + a short user turn so that block becomes compressible and gets tracked as a CCR entry; then a follow-up whose leading text block shares vocabulary with it, so proactive expansion fires and appends the original — which is how a request ends up leaving larger than it arrived. PERF lines read from the isolated `~/.headroom/logs/proxy.log`. - Observed result: real PERF output from that run — ```text msgs=1 tok_before=8170 tok_after=9553 tok_saved=0 tok_inflated=1383 ... transforms=router:text_block:mixed msgs=3 tok_before=8184 tok_after=9567 tok_saved=0 tok_inflated=1383 ... transforms=router:text_block:mixed msgs=5 tok_before=8245 tok_after=11880 tok_saved=0 tok_inflated=3635 ... transforms=router:text_block:mixed ``` Correlated from the same run: `CCR Tracker: Proactively expanded f0cf4efb42373ec225f57725 (1417 items)`, and the stub upstream confirms the block reached the wire (`has_expansion_block: true`, forwarded body 54,130 B on the third request). Every one of those turns reports `tok_saved=0`. Before this change that is all the log said, and it is the same thing it says when there was simply nothing left to compress. `tok_inflated=3635` is the number that was missing. For contrast, the same scenario run against a build where the request genuinely shrinks reported `tok_before=8245 tok_after=7359 tok_saved=886` — that build predates this field, so it does not print `tok_inflated`; the point is only that the inflating and shrinking cases are the two states the field has to separate, and on `main` today both render as `tok_saved=0` whenever the growth path is taken. The `tok_inflated=0` case on a shrinking request is covered by unit test. ### Scale of what was hidden From a live `--mode token` proxy on Claude Code traffic across four rotated logs: **305 of 3,743 requests (8%)** had `tok_after > tok_before` while every one reported `tok_saved=0` — 192,829 tokens of growth rendered as "nothing to compress". The worst single session held +2,643/turn for 19 consecutive turns. - Not tested: the `headroom perf` CLI was not run against a real log file containing the new field — it parses by field name and `tests/test_cli_perf_format.py` is green, but that is test-level rather than end-to-end evidence. Streaming responses were not exercised (the stub replies non-streaming), so the streaming emit path carries the new field on the strength of sharing `emit_request_outcome` rather than by observation. No dashboard or Prometheus consumer was re-run. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — one added field on an existing log line; no user-facing surface. ## Additional Notes - Independent of #2706 and #2707 — verified `conflicts=0` via `git merge-tree`; mergeable in any order. - Related but deliberately out of scope: `main` has no producer for retrieval-cost accounting (`record_savings_event` takes no `kind`/`tokens_retrieved`, and nothing writes `tokens_retrieved` anywhere), so proactive expansion's cost is not booked into net savings at all. Adding that channel is a cross-cutting accounting change and belongs in its own PR; this one only makes the growth visible in the log. |
||
|
Rod Boev
|
dcb674b5e4
|
fix(compression): honor qualified CCR names across integrations (#2698)
## Description Three compression consumers compare tool names against the bare literal `headroom_retrieve`, so the qualified forms MCP clients actually send (`mcp__Headroom__headroom_retrieve`, `mcp_Headroom_headroom_retrieve`) slip past the guard and get recompressed. `SmartCrusher.apply` has the bare comparison at both its OpenAI `role=tool` site and its Anthropic `tool_result` block site; the LangGraph compressor and the Strands hook have no tool-name check at all. Recompressing already-retrieved CCR content mints a new `<<ccr:hash>>` marker the agent cannot redeem. `headroom.config.is_tool_excluded` already owns alias resolution, including the MCP wrapper forms. This routes all three consumers through it instead of adding a second name matcher. Closes #2656. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `SmartCrusher.apply` routes both its `role=tool` and its Anthropic `tool_result` guards through `is_tool_excluded` - `_should_skip` in the LangGraph compressor takes the tool name and skips excluded tools; tool-call names are indexed by id so a `ToolMessage` without a copied `name` is still classifiable - `_should_skip_compression` in the Strands hook takes the tool name and skips excluded tools, recording `tool_excluded` - regressions for the qualified and bare names across all three consumers, the Anthropic block shape, the MCP wrapper entry point, and a near-match name that must still compress - a LangGraph regression for incomplete tool-call metadata that continues to a later qualified call ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output `pytest tests/test_smart_crusher.py tests/integrations/test_langgraph.py tests/integrations/test_strands tests/test_transforms/test_smart_crusher_ccr_retrieve_exemption.py -q` ```text tests\test_smart_crusher.py ............ [ 10%] tests\integrations\test_langgraph.py ..... [ 15%] tests\integrations\test_strands\test_ccr_exclusion.py ..... [ 19%] tests\integrations\test_strands\test_hooks.py sssssssss [ 27%] tests\integrations\test_strands\test_hooks_unit.py ssssssssssssssssssssssssssssssssss [ 57%] tests\integrations\test_strands\test_model.py ssssssssssssssss [ 71%] tests\integrations\test_strands\test_model_unit.py sssssssssssssssssssssssssss [ 95%] tests\test_transforms\test_smart_crusher_ccr_retrieve_exemption.py ..... [100%] 28 passed, 86 skipped in the focused invariant suite ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12.13, `headroom._core` built - Exact command / steps: `uv run pytest tests/test_smart_crusher.py tests/integrations/test_langgraph.py tests/integrations/test_strands -q`, and the same suite run against the pre-change implementation with the new tests in place - Observed result: before the change, five regressions fail. `SmartCrusher` returns non-byte-identical content for a `mcp__Headroom__headroom_retrieve` result, the LangGraph compressor replaces the message content, and the Strands hook returns `"compressed"` in place of the tool output. After the change all three preserve the content byte-for-byte, incomplete LangGraph tool-call metadata is ignored while the later qualified call remains indexed, the Strands hook records `tool_excluded` and never calls the crusher, and `HeadroomMCPCompressor.compress` returns the payload unchanged. `mcp__Headroom__headroom_retrieve_extra` still compresses in all three, and the Kompress and ContentRouter suites are unchanged. - Not tested: the optional Strands package, so the additions to `tests/integrations/test_strands/test_hooks_unit.py` skip locally ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes One deliberate divergence from the issue: the suggested snippet passes `DEFAULT_VERBATIM_EXCLUDE_TOOLS` to `is_tool_excluded`, but that constant holds only `WebSearch`, `WebFetch`, `web_search`, `web_fetch`. Applied literally it would drop `headroom_retrieve` from the comparison entirely and delete the #1077 guard these two SmartCrusher sites exist to enforce. This passes `(CCR_TOOL_NAME,)` so each guard keeps doing the one thing it documents. If you'd rather these paths also honor the verbatim-exclude set, the tuple can become `(CCR_TOOL_NAME, *DEFAULT_VERBATIM_EXCLUDE_TOOLS)` — the CCR name has to stay in it either way. Adjacent work: PR #2654 covers `ContentRouter` only. |
||
|
michaeltarleton
|
677e09735a
|
fix(transforms): stop ContentRouter recompressing headroom_retrieve results (#2654)
## Description `ContentRouter` (the transform actually registered in the default/proxy compression pipeline -- see `transforms/pipeline.py`) recompresses the output of its own `headroom_retrieve` tool. That tool's entire contract is returning already-retrieved, original content verbatim; recompressing it produces a new `<<ccr:hash>>` marker the caller can never redeem -- an unresolvable retrieval loop. `SmartCrusher` already has a guard against this exact failure mode (#1077), but only on its `apply()` entry point. `ContentRouter` calls the lower-level `SmartCrusher.crush()` directly, bypassing that guard entirely, since `crush()` takes a raw content string with no tool identity at all. Closes #1077 (reopens the same failure mode ContentRouter's own call path, which #1077's original fix did not cover). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `transforms/content_router.py`: adds an unconditional guard to all three of the places `ContentRouter` can hand a `headroom_retrieve` result to compression: the OpenAI-shape `role:"tool"`/legacy `role:"function"` string-content loop, the Anthropic-shape `tool_result` block loop, and a third, distinct shape -- top-level `{"type": "text"}` blocks under a `role:"tool"`/`"function"` message that never go through a `tool_result` wrapper (a real, already-tested wire shape in this codebase; see `test_tool_role_text_blocks_compressed_by_default`). All three use `is_tool_excluded()` (not a bare comparison) because MCP-served tools appear here under their qualified form, e.g. `mcp__headroom__headroom_retrieve`. Legacy `role:"function"` messages carry no call id in that shape, so the tool name is read directly off the message's `name` field instead of through the id-keyed `tool_name_map`. - Hoisted the per-iteration `is_tool_excluded(..., ("headroom_retrieve",))` calls into a single precomputed `ccr_retrieve_tool_ids` set, computed once alongside the existing `excluded_tool_ids` set, rather than recomputing aliases on every message/block. - `config.py`: adds `"headroom_retrieve"` to `DEFAULT_EXCLUDE_TOOLS` and `DEFAULT_VERBATIM_EXCLUDE_TOOLS` -- this also covers a third path (cross-turn message dedup, `_cross_turn_dedup_messages`) that consults the same frozensets and has no dedicated guard of its own. Also hardens `_tool_name_aliases()` against a non-string tool name (pre-existing fragility, not introduced by this PR, but shares the same call path) by returning no aliases instead of crashing on `.lower()`. - Documentation: updated `ContentRouterConfig.exclude_tools`'s field comment (was stale -- didn't mention this override is unconditional even when a caller explicitly empties `exclude_tools`), and added a comment on `DEFAULT_VERBATIM_EXCLUDE_TOOLS` noting all three real consumers. - Kept `"headroom_retrieve"` as a literal string (matching every other entry in those frozensets) rather than importing the existing `CCR_TOOL_NAME` constant from `ccr.tool_injection` into `content_router.py` -- that module is imported eagerly by `pipeline.py` (unlike `smart_crusher.py`, which imports the same constant lazily), so pulling in `headroom.ccr` there would add a new eager-import edge to a hot module for a one-line DRY win. Happy to change this if a maintainer prefers the constant. **Known, accepted tradeoff:** `is_tool_excluded()`'s alias matching strips any `mcp__<server>__` prefix before comparing, so a third-party MCP server exposing a tool literally named `headroom_retrieve` would also match. Narrowing this to headroom's own server specifically would need a bespoke check inconsistent with how every other excluded-tool entry is matched in this codebase; given how specific the name is, the collision risk is accepted rather than special-cased. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_transforms/ tests/test_transforms_content_router.py -q 1 failed, 420 passed, 62 skipped in 12.50s FAILED tests/test_transforms/test_kompress_compressor.py::...test_onnx_session_options_read_thread_caps (pre-existing, unrelated to this diff -- confirmed via `git stash` that it fails identically against unmodified upstream/main; an ONNX thread-cap assertion, not a compression-routing test) $ uv run ruff check headroom/config.py headroom/transforms/content_router.py \ tests/test_transforms/test_content_router_ccr_retrieve_exemption.py \ tests/test_transforms_content_router.py tests/test_transforms/test_content_router.py All checks passed! $ uv run ruff format --check <same files> 5 files already formatted $ uv run mypy headroom/config.py headroom/transforms/content_router.py Success: no issues found in 2 source files ``` - `tests/test_transforms/test_content_router_ccr_retrieve_exemption.py`: 10 tests -- MCP-qualified name (Anthropic + OpenAI shape), bare name, unconditional-even-with- `exclude_tools=frozenset()`, negative control (normal tools still compressed, asserted via the absence of the `router:excluded:ccr_retrieve` marker), the top-level-text-block shape, legacy `role:"function"`, litellm list-form content nested in a `tool_result` block, mixed retrieve+normal blocks in one turn, and a content well below the compression floor (proving the guard is size-independent). - `tests/test_transforms/test_content_router.py`: `test_anthropic_mcp_bare_tool_alias_exclude_tools` (#1822) updated to assert the new, stronger byte-verbatim guarantee for `headroom_retrieve` specifically; `test_anthropic_mcp_bare_tool_alias_exclude_tools_generic` added to keep the original #1822 general-mechanism coverage (bare-alias matching for an arbitrary, non-exempt tool). - `tests/test_transforms_content_router.py`: updated 10 pre-existing `_process_content_blocks()` unit tests for the new `ccr_retrieve_tool_ids` parameter (all pass empty sets -- none of those tests involve `headroom_retrieve`). - Verified the local installed package copy (a separate, drifted internal version) with a standalone repro script exercising the two new shapes directly against `ContentRouter.apply()` -- both correctly report `router:excluded:ccr_retrieve`. ## Real Behavior Proof - Environment: Windows 11, Python 3.13.7, `uv sync --extra dev` on this branch. - Exact command / steps: standalone repro building an assistant `tool_use` for `mcp__headroom__headroom_retrieve` paired with a large-JSON `tool_result`, through `ContentRouter().apply()`; repeated for the top-level-text-block and legacy-`function`-role shapes. - Observed result: unpatched (Anthropic `tool_result` shape, `git stash` to `upstream/main`), the retrieve output was rewritten 3680 -> 1861 bytes (mangled into a compact tabular form); patched (this branch), it is forwarded 3680 -> 3680 bytes byte-identical, no `<<ccr:` marker present. The two additional shapes fixed in this PR's second commit -- top-level text block under `role:"tool"`, and legacy OpenAI `role:"function"` -- both report `excluded=True` (protected) against this branch, where they reported `excluded=False` (recompressed) before the second commit. - Not tested: the actual `headroom mcp serve` + `headroom wrap` proxy end-to-end over a live Anthropic API call (would need API credentials); the OpenAI-chat-completions `CompressionUnit` path (out of scope, see #1176 below); the opt-in `ToolResultInterceptorTransform` path. ## Relationship to other issues/PRs - Issue #1077 (closed) is this exact bug; PR #1323 fixed it only for `SmartCrusher.apply()`'s own call path (the "legacy" pipeline path, per `smart_crusher.py`'s own comment), not `ContentRouter`, which is what the default/proxy pipeline actually uses. - Open PR #1176 addresses an adjacent, non-overlapping gap: the `CompressionUnit`-based OpenAI chat-completions path (`router.compress()` calls in `transforms/compression_units.py`/`compression_batches.py`), which has no tool-identity context at all and needs its own capture/restore mechanism. This PR does not touch that path. - Filed #2656 as a follow-up: code review on this PR found the same bug class still reachable through `SmartCrusher.apply()`'s own bare-name guard (not alias-aware, so it misses the MCP-qualified form) and through two unguarded direct `.crush()` calls in the LangGraph and Strands integrations. Both are pre-existing, narrower/separate call paths from `ContentRouter`'s primary proxy pipeline, so tracking them separately keeps this PR reviewable as one logical change. - Also not covered by this PR (flagging rather than silently omitting): `proxy/system_compaction.py`'s `router.compress(text, context="")` call, and the opt-in `ToolResultInterceptorTransform` (`HEADROOM_INTERCEPT_ENABLED=1`) -- neither was checked for CCR-awareness. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` -- it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A -- this is a backend compression-routing fix with no UI surface. ## Additional Notes This PR is two commits: the first commit added the initial two-loop guard; a second commit followed after code review found the guard was incomplete for two additional wire shapes (top-level text blocks, legacy `role:"function"`) and added the missing test coverage plus a few cleanup items (deduplicated guard logic, comment accuracy, a pre-existing non-string-tool-name fragility). See `Changes Made` above for the full list. Filed #2656 for the remaining out-of-scope gaps found during that same review. --------- Co-authored-by: Michael Tarleton <mtarleton@istation.com> |
||
|
JD Davis
|
13a310a00d
|
feat(claude): support Claude Code in VS Code (#2752)
## Description Add first-class Headroom support for the official Claude Code extension in VS Code. The new wrapper starts the local proxy, configures the Claude Code user settings consumed by the embedded extension process, preserves authentication and model selection, and provides a conflict-safe reversible unwrap lifecycle. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add `headroom wrap vscode-claude` and `headroom unwrap vscode-claude`. - Configure project-scoped `ANTHROPIC_BASE_URL` plus `ENABLE_TOOL_SEARCH=true` in Claude Code user settings while preserving existing values. - Respect `CLAUDE_CONFIG_DIR`, macOS/Linux home paths, Windows `USERPROFILE`, custom `--settings-file`, and `--no-configure`. - Add durable Headroom-owned restore state and refuse malformed settings or conflicting user edits. - Add unit, CLI, and Docker-harness e2e coverage for configuration, real proxy forwarding, and restoration. - Document setup, remote development, undo, and troubleshooting. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ UV_NO_SYNC=1 uv run pytest -q tests/test_provider_claude_vscode_config.py tests/test_cli/test_wrap_vscode_claude.py tests/test_cli/test_wrap_vscode.py tests/test_cli/test_wrap_claude_base_url.py tests/test_provider_copilot_vscode_config.py tests/test_copilot_auth.py 160 passed in 0.45s $ UV_NO_SYNC=1 uv run ruff check . All checks passed! $ UV_NO_SYNC=1 uv run mypy headroom Success: no issues found in 512 source files $ npm run build # from docs/ Compiled successfully; generated 155 static pages ``` ## Real Behavior Proof - Environment: macOS, Python 3.13 editable install, isolated temporary HOME and Claude settings, local mock Anthropic Messages upstream. - Exact command / steps: invoked the new `verify_vscode_claude_wrap` e2e function, which launched real `headroom wrap vscode-claude`, waited for proxy readiness, POSTed an Anthropic `/v1/messages` request through the generated project-scoped URL, stopped the wrapper, then ran `headroom unwrap vscode-claude`. - Observed result: HTTP 200 with the mock Claude response through Headroom; generated settings retained unrelated values and enabled tool deferral; unwrap restored the original Claude settings. - Not tested: real Anthropic account traffic or the full Docker image locally because Docker Desktop was unavailable. The same e2e function is wired into the existing Docker wrap CI job. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable; this adds CLI configuration and proxy routing without changing VS Code UI. ## Additional Notes The wrapper deliberately leaves the endpoint configured when stopped so requests fail closed instead of silently bypassing Headroom. `headroom unwrap vscode-claude` restores the exact prior managed values and preserves unrelated settings. --------- Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net> |
||
|
nangsontay
|
08fce29b47
|
fix(proxy): stop toggling headroom_retrieve in the Anthropic tools array (#2672)
## Description `should_inject_ccr_tool` deferred CCR tool injection whenever `frozen_message_count > 0`. Because `tools` is the head of Anthropic's cache key, that dropped a tool which was already inside the provider-cached prefix and invalidated the whole prefix — in both directions (`0 → >0` removes it; `>0 → 0` on proxy restart, `/model` switch, lineage eviction or TTL lapse adds it back). On three days of local proxy logs the turns that flipped injection state carried **44.7% of all cache-write tokens at a 52.0% hit rate**, against 98.1% for non-flipping turns. The log signature is `cache_read` alternating between two values exactly 172 tokens apart — the 464-byte tool definition. This deletes the gate and calls `apply_session_sticky_ccr_tool` directly, which is **what `openai.py` already does** — the two handlers now have the same shape. Net −61 production lines, no new state, no new config flag. Fixes defect 1 of #2671. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Performance improvement ## Changes Made - `headroom/proxy/ccr_marker_policy.py` — deleted the `should_inject_ccr_tool` gate; `apply_session_sticky_ccr_tool` is now the single decision point. - `headroom/proxy/handlers/anthropic.py` — calls `apply_session_sticky_ccr_tool` directly, matching `openai.py`. - `headroom/proxy/helpers.py` — dropped the now-unused gate plumbing. - `tests/test_proxy_anthropic_cache_stability.py` — new test asserting the forwarded `tools` array is byte-identical across a `frozen 0 → >0` transition. - `tests/test_ccr_marker_policy.py` — removed the three unit tests that pinned the deleted decision (they encoded the defect). - `tests/test_proxy/test_ccr_frozen_prefix_coupling.py` — same unredeemable-marker intent, re-pinned at the sticky helper. - `tests/test_proxy/test_anthropic_ccr_deferred_injection.py` — autouse reset fixture for the process-global `SessionCcrTracker` (separate commit). - Formatting-only follow-up commit applying `ruff format` (pinned 0.15.17) to the two test files above. ### Why deleting the gate is safe `apply_session_sticky_ccr_tool` already holds the correct rule. Its four branches, in order: | # | condition | action | |---|---|---| | 1 | tool already in the incoming tool list (client/MCP pre-registered) | skip; the client's bytes win | | 2 | `session_id is None` (WS / pre-session) | per-turn flag drives it verbatim | | 3 | session has done CCR | always inject the recorded golden bytes | | 4 | fresh session, no compression this turn | **skip** | Branch 4 is the safety property: a session that has never compressed still gets no tool, so removing the gate cannot start injecting into non-CCR conversations. Branch 3 is what the gate was starving. `has_new_ccr_markers` still gates first-time injection, so markers replayed from the previously-forwarded prefix cannot trigger one. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed The three deleted unit tests encoded the defect. Coverage moves to the property that actually matters and was previously untested: **the forwarded `tools` array must be byte-identical across a `frozen 0 → >0` transition.** That test asserts on the forwarded request body rather than on a policy function's return value; unit-testing the old policy in isolation is exactly what let a wrong-but-self-consistent decision pass. Verified failing on `upstream/main` with an assertion on the missing tool (not an `ImportError`, so it fails for the right reason). Full suite: same pre-existing unrelated failures as `upstream/main`, **zero new** (verified by running the whole suite on both revisions and diffing the failure sets). ### Test Output ```text $ uv run pytest tests/test_ccr_marker_policy.py \ tests/test_proxy/test_anthropic_ccr_deferred_injection.py \ tests/test_proxy/test_ccr_frozen_prefix_coupling.py \ tests/test_proxy_anthropic_cache_stability.py -q collected 48 items tests/test_ccr_marker_policy.py ..... [ 10%] tests/test_proxy/test_anthropic_ccr_deferred_injection.py .............. [ 39%] . [ 41%] tests/test_proxy/test_ccr_frozen_prefix_coupling.py .. [ 45%] tests/test_proxy_anthropic_cache_stability.py .......................... [100%] ======================= 48 passed, 2 warnings in 13.59s ======================== $ ruff check . All checks passed! $ ruff format --check . 1349 files already formatted ``` ## Real Behavior Proof - Environment: local macOS proxy serving live Claude Code traffic to the Anthropic API; baseline = 3 days of proxy logs on `upstream/main`, after = 5.5 hours with this change live. - Exact command / steps: ran the proxy with this branch built in, drove normal Claude Code sessions through it (including `/model` switches and proxy restarts, the two events that used to flip injection state), then parsed 235 real turns from the proxy logs with the same parser used for the baseline in #2671. - Observed result: flip turns fell from 177 (44.7% of all cache write) to 2 (1.7%); steady-state write share 1.192% → 0.867%; aggregate hit rate 86.75% → 89.21%; main conversation warm hit rate 98.1% → 97.70% (n=149). The 2 remaining "flips" have `cache_read == 0` — cold starts that the bucketing counts as a state change, not real flips. | metric | baseline | after | |---|---|---| | flip turns | 177, carrying 44.7% of all cache write | **2**, carrying **1.7%** | | main conv, warm | 98.1% | **97.70%** (n=149) | | steady-state write share | 1.192% | **0.867%** | | aggregate | 86.75% | **89.21%** | - Not tested: `mypy headroom` was not run locally for this body; the OpenAI handler path (unchanged by this PR); tracker state loss mid-session (see note below); and defect 2 of #2671 (the sub-call breakpoint), which is untouched and is now 54.9% of remaining cache write — that is why aggregate stays just under 90%. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes **Pre-existing and unchanged here:** if the tracker loses state mid-session while the transcript still carries markers, branch 4 returns no tool and those markers are unredeemable. `upstream/main` has no recovery for that; this PR neither creates nor fixes it. See my comment on #2500, which adds a recovery path for the related dangling-reference case. **N/A checklist items:** no documentation changes — this removes an internal policy function with no user-facing surface. `mypy headroom` left unchecked because it was not run for this body; CI covers it. **Merge-order conflict with #2500 (please read before landing either):** this PR *deletes* `should_inject_ccr_tool`, which is the exact function #2500 extends with `transcript_requires_tool`. Whichever lands second needs a semantic rebase, not just a textual one — git will not flag it. If this PR lands first, #2500's recovery path should re-target `apply_session_sticky_ccr_tool` (the sticky helper now owns the decision alone) or the handler call site in `handlers/anthropic.py`. If #2500 lands first, the gate deletion here still applies but the `transcript_requires_tool` override needs to move with it. Happy to do the rebase either way — say which order you prefer. |
||
|
Tejas Chopra
|
6422a80a58
|
fix(compress): resolve the /v1/compress tokenizer per model, and document the real contract (#2743)
## Description
`/v1/compress` does no format conversion — callers send whichever wire
shape they already use — but the pipeline pinned **one provider's token
counter for the whole route**.
`OpenAITokenCounter.count_message` walks list content for `text` and
`image_url` only and has **no else branch**, so Anthropic content blocks
contributed literally zero. A 599-token `tool_result` scored 8. A
request that really removed 235 characters reported `tokens_saved: 0` —
so a caller gating on `tokens_saved > 0` concludes compression is broken
while it is working.
Prompted by a Kong integration question ("do you support the Anthropic
native format?"). The answer is that we already did — we just reported
zeros for it, and the docs said otherwise.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Documentation update
## Changes Made
### Tokenizer resolution (no hardcoded lists)
Build the derived pipelines with `provider=None` so `TransformPipeline`
resolves the tokenizer from the **per-model registry**. Every registry
tokenizer derives from `BaseTokenizer`, whose `_count_content_parts`
ends in a serialize-and-count catch-all, which means:
- No block type counts as zero, and there is **no per-provider
block-type list to keep in sync**. An enumerated set was the first thing
I tried and it already missed `mcp_tool_result`,
`web_search_tool_result`, `document`, and `thinking`.
- Gemini / Mistral / DeepSeek / Kimi stop defaulting to a tiktoken count
when the registry already has a calibrated counter for them.
- Gateway aliases matching no vendor pattern still count correctly.
`mode="ccr"` now runs a derived pipeline too, for the same reason —
sharing `openai_pipeline` pinned its provider. Costs that mode its own
cold compression cache; correct metrics win.
### Tokenizer selection stays separate from context-limit resolution
Deliberately not welded together. `model_limit` feeds `context_pressure
-> min_ratio`, so letting a tokenizer decision pick the limit table
changes compression aggressiveness: `gpt-4-32k` answered by the
Anthropic table is **8,192 instead of 32,768**, a 4× under-estimate.
`test_tokenizer_choice_does_not_move_the_context_limit` pins the
independence.
### Docs, rewritten from the code
- **`proxy.mdx`** — the loopback-only default and **404-not-403**
behavior, previously undocumented *anywhere* in `docs/` despite shipping
in #2458 explicitly for gateway sidecars;
`HEADROOM_COMPRESS_ALLOW_REMOTE`; all four request fields; the whole
`config` object including every `mode` value and `frozen_message_count`;
`transforms_summary`; the 400/401/404/503 contract; and the timeout
fail-open shape (`compression_skipped` / `skip_reason`).
- **Corrected "never calls an LLM"** — accurate about *generative*
provider requests, misleading for a sidecar operator. Kompress (a
ModernBERT **encoder**, classification not generation) and Magika run
**in-process**, and `HEADROOM_KOMPRESS_ENDPOINT` offloads inference over
HTTP — **real egress**. Now stated explicitly, with
`HEADROOM_DISABLE_KOMPRESS=1` as the structural-only option.
- **Both wire formats documented as accepted**, and removed
`anthropic-sdk.mdx`'s claim that OpenAI format is "the compression
engine's native format" — the exact misconception that prompted this
work. The SDK's conversion is now framed as an SDK choice, not an API
requirement.
- **`litellm.mdx`** had no mention of the endpoint at all, despite the
code naming LiteLLM's guardrail as its primary consumer. Added the HTTP
deployment path, the `HEADROOM_COMPRESS_ALLOW_REMOTE` requirement, and
why to leave `config.mode` unset.
- **`index.mdx`** printed `compressionRatio * 100` labelled "Saved …%",
so a 77% saving displayed as **23%**. `api-reference.mdx` already
defined it correctly, so the docs contradicted each other.
- `openai-sdk.mdx`, `wiki/proxy.md`, `wiki/typescript-sdk.md` — same
corrections; dropped "any HTTP client", "Cloud", and a CacheAligner
claim (it is detector-only).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ .venv/bin/ruff check headroom/ tests/ --exclude headroom/dashboard/templates
All checks passed!
$ .venv/bin/mypy headroom/
Success: no issues found in 511 source files
$ python -m pytest tests/test_compress_route_tokenizer_by_model.py \
tests/test_proxy_compress_endpoint.py tests/test_compress_api.py \
tests/test_platform_stabilization_functional.py tests/test_proxy_eager_preload_bind.py -q
99 passed, 2 warnings in 47.15s
```
Broader sweep (`-k "compress or litellm or gateway or guardrail"`):
**1625 passed, 4 failed** — all 4 pre-existing, verified by stashing
this diff and re-running on clean `main` (2 strands hook tests, 1 codex
WS semaphore-tail timing test, 1 unrelated local WIP test).
## Real Behavior Proof
- **Environment:** macOS 26.4 arm64, Python 3.12.6, repo `.venv`, branch
rebased on `upstream/main`.
**(1) Before → after, same request** (60-line grep payload in an
Anthropic `tool_result`):
| model | before | after |
| --- | --- | --- |
| `claude-sonnet-4-6` | `before=28 saved=0` | `before=1223 saved=58` |
| `bedrock/anthropic.claude-3-5-sonnet` | `saved=0` | `before=1037
saved=59` |
| `my-gateway/big-model` (alias) | `saved=0` | `before=1037 saved=59` |
| `gemini-2.5-pro` | `saved=0` | `before=1036 saved=59` |
| `gpt-4o` + OpenAI shape | `before=1225 saved=58` | `before=1225
saved=58` (unchanged) |
All three `config.mode` values verified for each. Response shape
preserved: `type=tool_result`, `tool_use_id` intact.
**(2) Counter-level root cause**, 6.8 KB body, `count_message()`:
```text
OpenAITokenCounter string-content -> 1406 tool_result block -> 5
registry (BaseTokenizer) claude tool_result=408 thinking=418 mcp_tool_result=421
web_search_tool_result=421 document=422
```
**(3) Every documented behavior asserted against the running app** — 13
checks, all PASS: 400s for missing `messages`/`model`, invalid
`config.mode`, and all four invalid `frozen_message_count` forms; 200
for valid ones; non-dict `config` ignored; bypass and empty-messages
omit `transforms_summary`; success returns exactly the 8 documented
keys.
- **Not tested:** the docs site was not built (`docs/node_modules`
absent) — MDX was checked for balanced `<Callout>` tags only, so a
reviewer with the site running should eyeball rendering. No live
gateway/Kong request; verification is via `TestClient` against the real
ASGI app.
- **Note:** `HEADROOM_DISABLE_KOMPRESS` is read into `ProxyConfig` at
`server.py:4919` and by the CLI, not by `create_app(ProxyConfig(...))`
directly — I confirmed `disable_kompress=True` does reach the derived
pipeline.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
Matt Van Horn
|
789a4f3060
|
fix: normalize /p/<project> prefix on WebSocket upgrades so the Responses WS route is not rejected with 403 (#2379)
## Description
A Responses WebSocket upgrade to a project-prefixed URL
(`ws://127.0.0.1:8787/p/<project>/v1/responses`) was rejected with `403
Forbidden`, so the client fell back to HTTP transport. The `/p/<name>`
base-URL prefix is stripped by
`strip_project_path_prefix(request.scope)` inside
`@app.middleware("http")`, but Starlette runs `@app.middleware("http")`
for `http` scopes only, never `websocket` scopes. So an HTTP `POST
/p/<project>/v1/responses` has its prefix stripped and matches
`/v1/responses`, while the WS upgrade keeps the prefix, matches no
registered WebSocket route (`OPENAI_RESPONSES_WEBSOCKET_PATHS` are all
unprefixed), and Starlette rejects the unmatched WebSocket with `403`.
This normalizes the prefix for WebSocket scopes before routing so the
upgrade reaches the existing Responses WS handler and stays attributed
to the project.
Closes #2355
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/server.py` — added a small pure-ASGI
`WebSocketProjectPrefixMiddleware` (registered in `create_app`) that,
for `websocket` scopes only, strips the `/p/<name>` prefix via the
existing `strip_project_path_prefix` and binds the project context,
mirroring the HTTP middleware. HTTP and lifespan scopes pass through
untouched (no double-strip).
- `headroom/proxy/handlers/openai.py` — `handle_openai_responses_ws`
previously called `set_current_project(classify_project(ws_headers))`
unconditionally, clearing the middleware-bound project for prefix-only
clients (no `X-Headroom-Project` header). It now falls back to the
already-bound path-prefix project (`classify_project(ws_headers) or
get_current_project()`), so prefix-only WebSocket clients (aider,
Copilot BYOK, Cursor and other `/p/<name>` base-URL wraps) stay
attributed, exactly as on the HTTP path.
- `tests/test_provider_proxy_routes.py` — added a regression test.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_provider_proxy_routes.py -q
21 passed, 1 warning in 23.95s
$ ruff check headroom/proxy/server.py headroom/proxy/handlers/openai.py
All checks passed!
$ mypy --python-version 3.13 headroom/proxy/server.py headroom/proxy/handlers/openai.py
Success: no issues found in 2 source files
```
## Real Behavior Proof
- Environment: local, `uv` venv, Python 3.14, `uv run pytest`.
- Exact command / steps: added
`test_project_prefixed_openai_response_websocket_delegates_to_openai_ws_handler`,
which connects a WebSocket to `/p/test-project/v1/responses`.
- Observed result: the connection is accepted (no 403), the handler is
reached with the canonical `/v1/responses` path, and the request is
attributed to project `test-project`.
- Not tested: live end-to-end against a real upstream Responses
WebSocket server (validated via the routing/attribution regression test
only).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A — backend routing change with no user-facing UI.
## Additional Notes
Documentation checklist item is N/A: this is an internal routing fix
with no configuration or public-API surface change. The fix mirrors the
existing HTTP prefix-strip behavior so project-prefixed WebSocket
clients behave identically to their HTTP counterparts.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
|
||
|
Tejas Chopra
|
f9db5b5060
|
fix(proxy/openai): run tool-description compaction on chat-completions (#2741)
## Description
`HEADROOM_TOOL_DESC_MAX_CHARS` was wired into the Anthropic handler and
the Responses (Codex) handler, but never into **chat-completions** — so
the env var was a silent no-op for every chat client: opencode, Cline,
Aider, Roo, anything routed through LiteLLM.
Tool descriptions live on the `tools` array, which the message pipeline
never inspects, so no other pass was covering them.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- Run the L2 tool-description pass on the chat-completions path,
mirroring the block the Anthropic and Responses handlers already had.
- `compact_tool_descriptions` already walks both wire shapes — nested
`{"function": {"description": ...}}` for chat, flat for Responses — so
this is wiring, not a new codec.
- Chains after the existing schema compaction, seeding the token
"before" count only when that pass didn't, so the two compose instead of
double-counting.
- Labelled `openai:chat:tool_desc_compaction`, distinct from the
Anthropic and Responses labels so `headroom perf --by-transform` can
attribute it.
- Still opt-in and off by default: an unset env var leaves the tools
array — and therefore its cache prefix — byte-identical.
### Scope note: two adjacent "gaps" that turned out not to be
While surveying handler parity I flagged three missing chat-completions
transforms. Only one was real; recording the other two so nobody
re-opens them:
- **`tool_search_deferral` — correctly absent.** `{"type":
"tool_search"}` and `defer_loading` are Responses-API constructs, and
`_model_supports_openai_tool_search` gates them to `gpt-5.4+`. Injecting
that shape into a chat-completions request would be invalid, not an
improvement.
- **`system_prompt_compaction` — not applicable.** Anthropic needs a
dedicated pass because `system` is an out-of-band top-level field the
message pipeline never sees. On chat-completions the system prompt *is*
`messages[0]`, so it already reaches ContentRouter and is governed by
the existing `compress_system_messages` / `skip_system` gate. Wiring a
second path there would change system-prefix cache behavior for no new
coverage.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ .venv/bin/ruff check headroom/ tests/test_openai_chat_tool_desc_compaction.py --exclude headroom/dashboard/templates
All checks passed!
$ .venv/bin/mypy headroom/
Success: no issues found in 508 source files
$ python -m pytest tests/test_openai_chat_tool_desc_compaction.py tests/test_tool_schema_compaction.py \
tests/test_proxy_openai_cache_stability.py tests/test_openai_responses_context_compaction.py -q
49 passed in 16.59s
```
## Real Behavior Proof
- **Environment:** macOS 26.4 arm64, Python 3.12.6, repo `.venv`.
- **Exact command / steps:** ran `compact_tool_descriptions` at
`HEADROOM_TOOL_DESC_MAX_CHARS=30` against both wire shapes with the same
tool (a `read` tool with an 86-char description and a described `path`
param).
- **Observed result:**
```text
chat-completions (nested) modified=True bytes 272->215
responses (flat) modified=True bytes 259->202
```
Chat previously reported `modified=False` from the handler because the
pass was never invoked at all.
- **Not tested:** no live chat-completions request against a real
provider — the handler block is a thin adapter over
`compact_tool_descriptions`, and the regression was a missing *call*,
which the wiring test catches at source level. A full end-to-end drive
would need an upstream endpoint.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
Tejas Chopra
|
224578e80b
|
fix(kompress): reject artifacts that fail at run, and prefetch model files at startup (#2740)
## Description Three cold-start / robustness gaps found while debugging a user report of **0.12% savings across 722 requests** (49.8M input tokens, 60,920 saved). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made ### 1. The artifact fallback was unreachable for run-time failures `_create_onnx_session` tries `int8-wo` → `fp32` → `int8`, and its docstring describes exactly this scenario — but it only skipped a candidate when `InferenceSession(...)` **construction** threw. The int8 weight-only artifact carries `MatMulNBits` with `bits=8`. ORT's CPU kernel only handles 8-bit through the prepacked MLAS path, so a build or ISA without an 8-bit `SQNBitGemm` kernel falls into `ComputeBUnpacked`, which hard-asserts `nbits_ == 4`. That raises on `session.run()` **after** construction succeeded — so the fp32 candidate was never reached and ML compression was dead for the process lifetime. The reported log has 207 consecutive failures over three days. A two-token `_smoke_run` inside the existing candidate loop makes the fallback fire. `onnxruntime>=1.16.0` is unpinned, so which side of this an install lands on is a lottery. ### 2. A broken model cost an inference on every request, forever The per-request handler logged a `WARNING` and passed through with no latch — 207 identical lines that read as noise rather than "ML compression is dead". Now latches to passthrough after **3 consecutive** failures (any success resets the count) with one actionable `ERROR` naming the artifact override. ### 3. The model download began on the first request, not at startup #2001 was right to move Kompress off the startup path — on RHEL/CentOS 7-family hosts, entering cached native init before the port binds segfaults in `libarrow`/jemalloc with no Python traceback (#1908), which no `try/except` can catch. **This PR does not touch that.** But #2001 left the ~4-minute *download* on the first request, with every request in that window silently uncompressed behind one "model not ready" warning. Downloading is separable from loading. `prefetch_kompress_artifacts` resolves the files over plain `huggingface_hub` HTTP and never constructs an `InferenceSession` or imports `transformers`, so startup can prefetch bytes without touching the boundary #1908 crashes on. Native load stays deferred, status stays `deferred`, and a test asserts no session is constructed during prefetch. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ .venv/bin/ruff check headroom/ tests/test_kompress_failsafe.py tests/test_kompress_preload_deferral.py --exclude headroom/dashboard/templates All checks passed! $ .venv/bin/mypy headroom/ Success: no issues found in 508 source files $ python -m pytest tests/test_kompress_failsafe.py tests/test_kompress_preload_deferral.py \ tests/test_kompress_request_nonblocking.py tests/test_force_kompress_all.py \ tests/test_kompress_must_keep.py tests/test_proxy_disable_kompress.py \ tests/test_proxy_per_provider_kompress.py tests/test_proxy_warmup.py \ tests/test_proxy_eager_preload_bind.py -q 95 passed in 10.51s ``` ## Real Behavior Proof - **Environment:** macOS 26.4 arm64, Python 3.12.6, onnxruntime 1.21.1, repo `.venv`. **(1) Fallback chain, against the real HF repo:** ```text WARNING ONNX artifact 'onnx/kompress-int8-wo.onnx' from chopratejas/kompress-v2-base is unusable (... nbits_ == 4 was false ...); trying next candidate SESSION OK -> ['input_ids', 'attention_mask'] SMOKE RUN OK on the selected artifact ``` Also confirmed the default artifact really is 8-bit, by loading the cached blob: `{'bits': [8], 'block_size': [128]}`. **(2) Files-only prefetch, with `InferenceSession` patched to raise:** ```text INFO Kompress: prefetching model artifacts for chopratejas/kompress-v2-base ... prefetch ok=True in 0.08s, no session constructed ``` - **Not tested / important caveat:** the user's exact failure **cannot be reproduced on this machine**. On ORT 1.21.1 arm64 the int8-wo artifact fails at *construction* (`matmul_nbits.cc:115`), which the pre-existing load-only fallback already caught. Their build fails at *execution* (`matmul_nbits.cc:442`, `ComputeBUnpacked`). So the run-time path is pinned with a fake ORT session that constructs fine and then rejects `run()` — a mechanism test, not a reproduction of their build. Confirming the fix on their host needs their `onnxruntime` version. - **Not tested:** no RHEL/CentOS 7 host available to re-verify #1908 non-regression; the argument is structural (prefetch never constructs a session) and asserted by `test_prefetch_never_constructs_a_session_or_imports_transformers`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Tejas Chopra
|
8262a4a321
|
fix(stats): report one "Tokens Saved" headline across every harness (#2737)
## Description
The "tokens saved" figure a user sees depended on which harness they
ran. Headroom saves tool-definition tokens in two accounting shapes,
both legitimate, but the rule was never written down — so two harnesses
silently dropped savings and three surfaces open-coded the sum
differently.
- **Compaction** rewrites the tool array, so both endpoints are
countable → handlers fold the delta into
`original_tokens`/`optimized_tokens`, keeping `tok_before - tok_after ==
tok_saved` coherent.
- **Deferral / hook shrink** removes schemas `count_messages` never sees
→ can only be recorded as a tag, additive to `tokens_saved`.
`tool_schema_savings_policy` now owns the sum via
`headline_tokens_saved()`, and every reporting surface routes through
it.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
Producer gaps (both Anthropic — i.e. Claude Code, the primary harness):
- `anthropic:tool_schema_compaction` / `anthropic:tool_desc_compaction`
computed their savings, debug-logged them, and **discarded them**. Now
folded at the final recount, mirroring the OpenAI chat handler. A
14-tool array drops 786 tokens that previously reported `tok_saved=0`.
- Anthropic never wrote `turn_hook_tools_saved_tokens` at all, so a
turn-hook extension that shrinks tools got zero credit there while
OpenAI credited it. Now tagged.
Reporting gaps:
- `headroom perf` printed `Total saved (messages)` and `Tool saved` as
rival lines — on a tool-heavy session the headline read `0` and the real
win looked like a footnote. Now one `Tokens saved:` headline with a
messages/tool-schemas breakdown.
- `active_savings_percent` divided a **compression-only numerator** by a
denominator that already included compacted tool schema, undercounting
every tool-heavy session. Numerator is now all-layers, with deferred
schemas added to both sides.
- The headline and its percent now share a numerator. Previously the
dashboard tile showed an all-layers total next to a compression-only
percent.
- Session summary and dashboard tile relabelled to `Tokens Saved`; the
tool-schema panel is labelled as a component (`Tokens Saved · Tool
Schemas`) rather than a rival metric.
- `outcome.py` had two drifted inline copies of the tag sum; both now
call the policy module that exists for it. `total_saved=` added to the
PERF line.
- JSON: added `total_tokens_saved` / `total_savings_pct`; existing
`tokens_saved` / `tool_saved` / `savings_pct` keys unchanged for
back-compat.
Not changed by design: the Codex per-component attribution sub-line
would need a 9th positional tuple element threaded through 4 unpack
sites, and its headline is already correct without it.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ .venv/bin/ruff check headroom/ tests/test_tool_schema_savings_policy.py --exclude headroom/dashboard/templates
All checks passed!
$ .venv/bin/ruff format --check headroom/ tests/... --exclude headroom/dashboard/templates
510 files already formatted
$ .venv/bin/mypy headroom/
Success: no issues found in 508 source files
$ python -m pytest tests/test_tool_schema_savings_policy.py tests/test_cli_perf_format.py \
tests/test_request_outcome.py tests/test_savings_tool_search_aggregation.py \
tests/test_dashboard_token_savings.py tests/test_anthropic_compaction_transforms.py -q
70 passed, 1 warning in 6.15s
$ python -m pytest tests/test_handler_outcome_tag_invariant.py tests/test_cold_start_fast_pass.py \
tests/test_anthropic_ccr_workspace_unbound.py tests/test_anthropic_pre_upstream_backpressure.py \
tests/test_vertex_claude_compression.py tests/test_provider_route_specs.py -q
50 passed in 10.11s
$ python -m pytest tests/test_agent_savings.py tests/test_bundled_tools_savings.py \
tests/test_codex_ws_savings_deferral.py tests/test_savings_ledger_before_forwarded.py \
tests/test_savings_ledger_offload.py tests/test_proxy_savings_history.py \
tests/test_proxy_dashboard_stats_cache.py tests/test_output_savings_cli.py -q
97 passed, 2 skipped in 18.60s
$ python -m pytest tests/test_tool_schema_compaction.py tests/test_openai_responses_context_compaction.py \
tests/test_proxy_openai_cache_stability.py tests/test_codex_ws_compression_scheduler.py \
tests/test_proxy_streaming_request_logger.py -q
66 passed, 1 skipped in 16.69s
```
## Real Behavior Proof
- **Environment:** macOS 26.4 arm64, Python 3.12.6, repo `.venv`.
Motivated by a real user proxy log (0.33.0, `client=opencode` →
nano-gpt, 722 requests) reporting 0.12% savings.
- **Exact command / steps (1) — the Anthropic fold, real compaction +
real provider tokenizer:**
```python
tok = AnthropicProvider().get_token_counter("claude-sonnet-4-6")
payload = {"tools": [ ...14 tools with $schema/title/examples... ]}
body, modified, bb, ba = compact_tools(payload)
```
**Observed:**
```text
modified=True bytes 4503->2539 TOKENS 1650->864 delta=786
tok_before=6650 tok_after=5864 tok_saved=786 coherent=True
pre-fix: Claude Code reported tok_saved=0 and discarded 786 tokens
```
Pinned as `test_tool_schema_compaction_saves_real_tokens_not_just_bytes`
— it asserts a positive **token** delta (not just bytes), which is the
premise of folding at all.
- **Exact command / steps (2) — the report, on the reported session's
shape** (tool schemas carry the win, message compression is 0 because
everything routed to `excluded_tool`):
**Observed after:**
```text
Requests: 2
Tokens: 45,760 -> 45,760 (0.0% messages)
Tokens saved: 811 (1.7% reduction)
· messages 0
· tool schemas 811
JSON: {'total_tokens_saved': 811, 'total_savings_pct': 1.7, 'tokens_saved': 0,
'tool_saved': 811, 'savings_pct': 0.0}
```
Before, the same input printed `Total saved: 0 tokens (messages)` as the
headline with `Tool saved: 811` beneath it.
- **Not tested:** no live proxy run against a real provider — the
Anthropic fold is proven at the accounting layer (real `compact_tools` +
real provider tokenizer) and via the existing handler suites, not by an
end-to-end Claude Code session. Dashboard changes are template-label
edits verified by reading `stats.tokens.saved` / `by_layer.tool_search`
shapes, not by a browser screenshot.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
Tejas Chopra
|
3d23d76248
|
fix(kompress): let orgs run Kompress on their own inference stack (#2736)
## What this enables An org pulls the Kompress weights from HuggingFace, serves them on their own infrastructure, and points Headroom at it: ```bash HEADROOM_KOMPRESS_ENDPOINT=https://ml.internal.acme.com ``` No credential needed, no local ML dependencies, and original content never leaves their network (the CCR store stays proxy-local, so `headroom_retrieve` keeps working). ## The one thing that was actually broken Almost all of this already worked. The blocker was a hardcoded path: ```python self._url = endpoint.rstrip("/") + "/compress" ``` Real inference servers don't serve at `/compress`: | Stack | Path | |---|---| | TorchServe | `/predictions/kompress` | | KServe / Seldon | `/v1/models/kompress:predict` | | SageMaker | `/invocations` | Appending `/compress` to those 404s. And because remote Kompress **fails open**, that 404 is invisible — compression silently stops instead of erroring. The only workaround was standing up a reverse proxy purely to rename a path. ## Two new env vars, both defaulting to current behaviour | Var | Default | Purpose | |---|---|---| | `HEADROOM_KOMPRESS_ENDPOINT_PATH` | `/compress` | Set empty to use the endpoint URL verbatim | | `HEADROOM_KOMPRESS_ENDPOINT_HEADERS` | *(none)* | `k=v,k2=v2`, merged last so it can replace `Authorization` | Headers are applied after the token deliberately, so a gateway wanting `x-api-key` or `X-Tenant-Id` needs no separate auth-scheme setting. ## No regression With only `HEADROOM_KOMPRESS_ENDPOINT` set, the request is **byte-identical** to before — `POST <endpoint>/compress` with an optional Bearer token. Existing Modal deployments need no change. `os.environ.get` with a default distinguishes "unset" (use `/compress`) from an explicit empty value (endpoint is a complete URL), so the escape hatch can't fire by accident. The regression cases are deliberately the *first* tests in the new file. Verified through the real router wiring: ``` modal (today's config) -> https://acme--kompress.modal.run/compress modal + token -> …/compress {'authorization': 'Bearer tok'} self-hosted KServe (full URL) -> https://ml.acme.com/v1/models/kompress:predict self-hosted TorchServe (path) -> https://ts.acme.com/predictions/kompress self-hosted, x-api-key, no token -> …/compress {'x-api-key': 'k', 'x-tenant-id': 'acme'} ``` ## Documents the HTTP contract The endpoint contract was only discoverable by reading the source. Now in the module docstring: ``` request {"content": "<text>", "target_ratio": 0.5 | null} response {"compressed": "<text>", # REQUIRED "original_tokens": int, # optional, derived if absent "compressed_tokens": int, # optional "compression_ratio": float, # optional "model_used": str} # optional ``` `compressed` is the only required field, so a shim in front of an existing inference server is a few lines. Also logs the **resolved** URL at startup — with fail-open, a mistyped path otherwise manifests as nothing happening at all. ## Notes - `parse_endpoint_headers` reimplements the `HEADROOM_OTEL_METRICS_HEADERS` format rather than importing it: `observability.metrics` imports opentelemetry at module scope, and remote Kompress exists precisely so a proxy can run without heavy optional deps. - 27 new tests. Pre-existing unrelated flake in `test_content_router_single_item_deadline.py` (fails 3/3 on clean main). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Tejas Chopra
|
7c9b046595
|
fix(dashboard): serve tailwind/htmx/alpine locally instead of from CDNs (#2734)
## Description The dashboard loaded all three of its front-end dependencies from third-party CDNs at page load: ```html <script src="https://cdn.tailwindcss.com"></script> <script src="https://unpkg.com/htmx.org@1.9.10"></script> <script src="https://unpkg.com/alpinejs@3.13.3/dist/cdn.min.js" defer></script> ``` Microsoft Edge's Tracking Prevention classifies `unpkg.com` as a tracker and blocks it by default on Windows; locked-down corporate proxies block both hosts. On those machines none of the three scripts executed — no Tailwind CSS, no htmx polling, no Alpine bindings, plus an uncaught `ReferenceError: tailwind is not defined` from the inline `tailwind.config` assignment at `dashboard.html:21`. The dashboard rendered blank. Reported from a Windows user's console: ```text Tracking Prevention blocked access to storage for https://unpkg.com/htmx.org@1.9.10. Tracking Prevention blocked access to storage for https://unpkg.com/alpinejs@3.13.3/dist/cdn.min.js. ``` This vendors the three files and serves them from the proxy, so the dashboard has no external network dependency at all. Note for anyone triaging the same report: the `cdn.tailwindcss.com should not be used in production` line in that console output is **not** related. It is an unconditional `console.warn` in the Tailwind Play CDN build (no hostname guard), so it fires on every load, localhost included, and it still fires now that the bundle is self-hosted. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Vendored `headroom/dashboard/static/{tailwind.min.js,htmx.min.js,alpine.min.js}` — Tailwind Play CDN 3.4.17, htmx 1.9.10, Alpine 3.13.3, byte-for-byte as published. - `headroom/dashboard/__init__.py`: added `STATIC_DIR`. - `headroom/proxy/server.py`: mounted `/dashboard/static`, registered **before** `register_provider_routes`' catch-all so the asset requests are not tunneled to the wrapped upstream provider (same ordering constraint as the `/favicon.ico` route, GH #1787). `check_dir=False` so a missing assets directory 404s the dashboard JS rather than aborting proxy startup. - `headroom/dashboard/templates/{dashboard,settings}.html`: script `src` → `/dashboard/static/…`. - `NOTICE`: MIT / 0BSD attribution for the three vendored bundles. - `tests/test_dashboard_static_assets.py`: new. No packaging change needed — `[tool.maturin]` includes everything under `headroom/`, so the wheel picks the assets up. Wheel grows ~498 KB (407 KB of that is the Tailwind Play bundle). ## Testing - [x] Unit tests pass (`pytest`) — targeted, see note under *Not tested* - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_dashboard_static_assets.py tests/test_proxy_settings_endpoints.py -q tests/test_dashboard_static_assets.py ...... [ 21%] tests/test_proxy_settings_endpoints.py ...................... [100%] ============================== 28 passed in 4.47s ============================== $ ruff check . All checks passed! $ ruff format --check headroom/proxy/server.py headroom/dashboard/__init__.py tests/test_dashboard_static_assets.py 3 files already formatted $ mypy headroom Success: no issues found in 509 source files ``` ## Real Behavior Proof - **Environment:** macOS 15 (Darwin 25.4.0), Python 3.12.6, headless Chromium via Playwright, proxy served in-process with `create_app(ProxyConfig(optimize=False, cache_enabled=False, log_full_messages=True))` on `:8787`. - **Exact command / steps:** loaded `/dashboard` and `/dashboard/settings` with `wait_until="networkidle"`, then asserted the globals exist, that Tailwind actually generated CSS (computed style of a `px-3` element), and recorded every non-localhost request plus all `pageerror`/`console.error` events. - **Observed result:** ```text /dashboard | alpine: True | tailwind css: True | external: none | errors: none /dashboard/settings | alpine: True | tailwind css: True | external: none | errors: none /dashboard 200 text/html; charset=utf-8 191549 /dashboard/static/tailwind.min.js 200 text/javascript; charset=utf-8 407279 /dashboard/static/htmx.min.js 200 text/javascript; charset=utf-8 47755 /dashboard/static/alpine.min.js 200 text/javascript; charset=utf-8 43441 feed-toggle visible: True alpine loaded: True htmx: True tailwind: True tailwind applied (px-3 padding): 12px external hosts: none console errors: none ``` Zero external requests on either page, so the Edge/firewall failure mode is structurally gone rather than worked around. - **Not tested:** - No Windows machine available — the fix is verified as "makes zero external requests", which is the property the Windows failure depended on, but it has not been confirmed against Edge with Tracking Prevention on. Worth a check by someone on Windows before release. - Full `pytest` suite not run (targeted runs only); CI covers it. - `tests/test_dashboard/test_live_feed.py` still has 2 failures, both pre-existing and unrelated: those tests need a manually started proxy on `:8787` with `--log-messages`, and `test_live_feed_button_exists` asserts `is_visible()` with no wait for the `/stats` poll that flips `log_full_messages`. The other 2 in that file pass against this change, which is itself end-to-end evidence that Alpine and htmx work from the vendored bundles. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes - No issue number: reported directly rather than filed, so `Closes #` is omitted. Closed #22 ("Dashboard is not working") and closed #533 (Windows cp949 `get_dashboard_html()`) are different failures. - **Docs checklist item is N/A** — nothing user-facing changes; the dashboard URL and behaviour are identical. - Deliberately **not** switching to a real Tailwind CLI build. It would cut 407 KB to ~20 KB and silence the production warning, but it puts Node in the release path and silently leaves any class added to the 2,713-line template unstyled with no CI guard. The Play bundle behaves exactly as it does today, just served locally. Worth revisiting if wheel size becomes a problem (note the PyPI project-size ceiling). - Upgrades are now manual: bumping these three means re-downloading the files. Pinned versions are recorded in `NOTICE`. |
||
|
gglucass
|
a70e5ff78d
|
fix(learn): run project discovery off the event loop (#2731)
## Description
`TrafficLearner.flush_to_file` is a coroutine, but it called
`plugin.discover_projects()` inline. That function walks the filesystem
to decode escaped project directory names — in
`learn/plugins/claude.py`, `_greedy_path_decode` recurses through
`iterdir()` at every level and tries each tokenization of each child,
backtracking on a miss — so on a large home tree it runs for minutes.
Doing that on the event loop freezes uvicorn for the whole window. The
port keeps accepting TCP, but `/readyz` never answers, so a supervisor
health-checking the proxy kills a process that is merely busy.
Field thread dumps show exactly that:
```
Current thread (most recent call first):
File "python3.12/pathlib.py", line 1056 in iterdir
File "headroom/learn/plugins/claude.py", line 454 in _greedy_path_decode
File "headroom/learn/plugins/claude.py", line 478 in _greedy_path_decode
File "headroom/learn/plugins/claude.py", line 478 in _greedy_path_decode
File "headroom/learn/plugins/claude.py", line 426 in _decode_project_path
File "headroom/learn/plugins/claude.py", line 71 in discover_projects
File "headroom/memory/traffic_learner.py", line 591 in flush_to_file
File "headroom/memory/traffic_learner.py", line 535 in _flush_worker
File "python3.12/asyncio/events.py", line 88 in _run
File "python3.12/asyncio/base_events.py", line 1999 in _run_once
File "python3.12/asyncio/base_events.py", line 645 in run_forever
File "uvicorn/server.py", line 75 in run
File "headroom/proxy/server.py", line 4992 in run_server
```
Accompanying signals from the same incidents: port accepts TCP,
`/readyz` times out, process CPU 2-13s across the window (I/O bound, not
spinning), proxy log silent 66-336s.
## Type of Change
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/memory/traffic_learner.py`: `flush_to_file` now awaits
`asyncio.to_thread(plugin.discover_projects)` instead of calling it
inline. `asyncio` was already imported. The result is cached per learner
(`_project_roots_cache`), so the steady-state flush path pays nothing
for the thread hop.
- `tests/test_memory/test_traffic_learner.py`: added
`test_discover_projects_does_not_block_the_event_loop`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uv run --frozen --extra dev pytest tests/test_memory/test_traffic_learner.py -q
.................................................. [100%]
============================= 152 passed in 2.93s ==============================
$ uvx ruff check headroom/memory/traffic_learner.py tests/test_memory/test_traffic_learner.py
All checks passed!
$ uvx ruff format --check headroom/memory/traffic_learner.py tests/test_memory/test_traffic_learner.py
2 files already formatted
$ uv run --frozen --extra dev mypy headroom/memory/traffic_learner.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: macOS 15.6 arm64, Python 3.10.18, pytest 9.0.3, branch
off `main` @ `
|
||
|
Tejas Chopra
|
9cfb00838a
|
fix(telemetry): anonymous compression stats — no prompts, no data (#2728)
## In one line
Headroom starts reporting **how well compression is working** — counters
and percentages only. **No prompts. No code. No file paths. Nothing
about what you're building.**
## Why
Right now nobody knows whether compression actually helps real users.
You can see your own numbers in `/stats`, but that's it — there's no way
to tell whether a given workload compresses well, or why it sometimes
doesn't. This closes that loop so we can make compression better for
everyone.
## Exactly what gets sent
One message per session, and every 5 minutes while you're active:
```json
{
"session": { "id": "random", "turns": 47, "duration_s": 4210, "seq": 3 },
"tokens": { "original": 890000, "attempted": 410000, "saved": 320000,
"tool_saved": 48000, "cache_read": 210000 },
"rates": { "saved_pct": 35.96, "eligible_pct": 46.07, "yield_pct": 78.05,
"cache_read_pct": 23.60, "overhead_pct": 1.96 },
"compression": { "transforms": {"crush": 47}, "passthrough_turns": 0 },
"skips": {},
"sources": { "proxy": 47 },
"providers": ["anthropic"],
"models": ["claude-sonnet-4-5-20250929"],
"failures": 2
}
```
Plus a random install ID, the Headroom version, and OS/architecture
(`darwin`, `arm64`).
That's the whole thing. A full example lives at
`deploy/beacon/sample-event.json`.
## What is never sent
- Your prompts or the model's responses
- Your code
- File paths, project names, repo names
- Tool names or MCP server names
- Hostname, username, or IP address
- Custom or fine-tuned model names (an id like `ft:gpt-4o:acme-corp:…`
contains a company name, so only models in a public registry are
reported)
**This is structural, not a pinky-swear.** Every value in the payload is
a number, a fixed word, or a random ID — there is no free-text field
anywhere for content to hide in. The receiver
(`deploy/beacon/worker.js`, in this repo so you can read it) drops
anything not on an explicit allowlist before storing.
## Turning it off
Any one of these:
```bash
HEADROOM_BEACON=off # or
DO_NOT_TRACK=1 # or
# offline mode
```
It's on by default, and Headroom says so at startup:
```
Telemetry: anonymous compression stats — never prompts, code, or file paths.
Helps us improve compression | Off: HEADROOM_BEACON=off
```
`HEADROOM_TELEMETRY` is a **separate** switch that still only affects
local stats. If you had turned that on, this change does not start
uploading anything — you answered a different question, and upgrading
should not change the answer.
## Why the percentages, not just "tokens saved"
"We saved 36%" hides the interesting part. In the example above only
**46% of tokens were eligible** for compression at all — the rest is
frozen cache prefix and system prompts we deliberately do not touch. Of
what we *could* touch, we removed **78%**.
Those are two separate problems. Raising eligibility is proxy work;
raising yield is compressor work. A single number cannot tell us which
to fix.
## Coverage
`emit_request_outcome` is a single chokepoint —
`handler.metrics.record_request` is called from exactly one place,
inside the funnel — so all 30 `RequestOutcome` construction sites are
covered: Anthropic, OpenAI, Gemini, Bedrock, batch, streaming, and the
long-lived Codex Responses-WS path.
The `headroom_compress` MCP path bypassed that funnel and is now wired
in separately. It has a different shape (no provider, no upstream
latency, and everything handed to the tool is eligible by construction),
so `sources` counts turns by origin — MCP turns always read
`eligible_pct: 100` and must not drag the proxy's real eligibility
ceiling upward.
**Subagents.** All subagent traffic through the proxy merges into one
session, which is correct for savings and retention but means `turns`
conflates fan-out with depth. Fan-out is still derivable —
`compression.latency_ms_total / session.duration_s` gives the
concurrency ratio (~1x serial, ~4x for four parallel agents), so no
extra field is needed. Verified no lost updates under 6-way concurrency
(1,200 turns).
**Known gap:** `--workers N` gives each process its own aggregator, so
one user session becomes up to N. Token totals and fleet rates stay
correct; session counts inflate. This matches the existing documented
limitation that TOIN state, CostTracker, and the prefix tracker are all
per-process.
## Notes for reviewers
- **Cumulative snapshots, not deltas.** Every report restates running
totals under one session ID, so the highest `seq` per `(install,
session)` is the complete session. Dedupe is a window function, and a
lost report costs nothing.
- **Never breaks the proxy.** Every path swallows its own exceptions;
uploads go out on a daemon thread so nothing blocks the request loop.
- **Explicit User-Agent is load-bearing.** urllib's default is blocked
by Cloudflare (error 1010). Combined with fire-and-forget error
handling, that would have failed every upload while looking perfectly
healthy.
- **The exit flush was broken and is fixed.** `atexit` handed the POST
to a daemon thread, and daemon threads are killed before they finish
during interpreter shutdown — so nothing was sent. That silently dropped
*every session shorter than the 5-minute heartbeat*, plus all
short-lived subagent MCP processes. The exit path now posts
synchronously with a 2s timeout.
- Receiver and query tooling are in `deploy/beacon/`.
## Testing
- `python -m headroom.telemetry.session` self-check: dedupe, cumulative
totals, dropped-report recovery, payload contains no model id or
prompt-derived string, allowlist coverage
- 175 telemetry/outcome tests pass; 6 new ones cover the opt-out notice
- Verified end to end against a live deployment: client → receiver →
storage → query
## Still to do before release
The default endpoint currently points at a temporary `workers.dev` URL.
It needs to move to a Headroom-owned hostname before this ships in a
tagged release — noted inline at `DEFAULT_ENDPOINT`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
JD Davis
|
007446c73a
|
feat(copilot): proxy VS Code models transparently (#2687)
## Description Make Headroom a transparent proxy for VS Code GitHub Copilot. Users keep using Copilot's normal model picker—GPT-4.1, Claude Sonnet, Claude Opus, and other models in their entitlement—while Headroom silently forwards the selected model instead of registering or requiring a separate "Headroom" model. This also fixes GitHub's device OAuth exchange by sending form-encoded request bodies, matching the endpoint contract. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add `headroom wrap vscode` to start a Copilot-seeded subscription proxy and safely configure VS Code's shipped Copilot proxy override. - Add `headroom unwrap vscode` for reversible cleanup. - Preserve VS Code's selected model by changing only the proxy URL/auth override; no custom model is registered and no model preference is written. - Support stable VS Code settings locations on macOS, Windows, and Linux, plus `--settings-file` for Insiders, portable, and other installations. - Edit JSONC settings with a marker-owned block while preserving unrelated bytes, comments, ordering, and trailing commas. - Refuse malformed markers, invalid JSONC, or unmanaged existing Copilot overrides instead of overwriting user configuration. - Fix SIGINT cleanup so the managed settings block is removed and normal shutdown exits successfully. - Fix Copilot device OAuth start/poll requests to use `application/x-www-form-urlencoded`. - Add a compatibility matrix, setup/removal flow, credential behavior, remote-development guidance, enterprise notes, troubleshooting, and verification documentation. ## Testing - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ .venv/bin/pytest -q tests/test_provider_copilot_vscode_config.py tests/test_cli/test_wrap_vscode.py tests/test_cli/test_wrap_helpers.py tests/test_copilot_auth.py tests/test_cli/test_copilot_auth.py tests/test_provider_copilot_wrap.py tests/test_cli/test_wrap_copilot.py tests/test_copilot_provider_label.py tests/test_copilot_subscription_smoke.py 244 passed in 0.59s $ .venv/bin/ruff check <changed Python files and tests> All checks passed! $ .venv/bin/mypy headroom/providers/copilot/vscode.py Success: no issues found in 1 source file $ cd docs && npm run types:check fumadocs-mdx && next typegen && tsc --noEmit # exited 0 $ git diff --check # exited 0 ``` The full 10,179-test suite was also sampled through approximately 83%, but was stopped because of its runtime. It exposed existing failures in `test_recover_codex.py`, `test_wrap_stale_marker.py`, and `test_proxy_health.py`; therefore the broad `pytest`, repository-wide Ruff, and repository-wide mypy boxes are intentionally not checked. ## Real Behavior Proof - Environment: macOS arm64, VS Code 1.131.0, built-in GitHub Copilot 0.59.0, Headroom 0.33.1-dev. - Exact command / steps: 1. Completed `headroom copilot login` with GitHub's device flow. 2. Ran `.venv/bin/headroom wrap vscode --port 8788`. 3. Confirmed VS Code retained its ordinary Copilot model catalog and made `GET /models` through Headroom with `GitHubCopilotChat/0.59.0` and `editor-version: vscode/1.131.0`. 4. Sent native Copilot `/p/headroom/chat/completions` requests through the same endpoint using `gpt-4.1`, `claude-sonnet-4.6`, and `claude-opus-4.7`. - Observed result: - All three completion requests returned HTTP 200. - GPT-4.1 resolved upstream to `gpt-4.1-2025-04-14`; Sonnet and Opus retained their exact selected IDs. - All returned the requested exact marker content. - VS Code's settings contained only the Headroom proxy URL and token auth override—no Headroom model or model-selection setting. - The proxy health endpoint remained ready with `openai_api_url` set to `https://api.githubcopilot.com`. - Not tested: - Physical Windows or Linux hosts (their path/config behavior is covered by unit tests). - WSL, dev containers, SSH remotes, VS Code Insiders, or enterprise Copilot deployments end-to-end. - Every model in the live Copilot catalog. - A fully submitted chat from VS Code's UI automation; the real extension's catalog request and native completion paths were verified separately. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing targeted unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable; this integration intentionally has no separate UI or model entry. ## Additional Notes The integration uses VS Code Copilot's shipped advanced/debug proxy endpoint seam. The managed settings block is deliberately narrow and reversible. Remote extension hosts may need their own reachable proxy/configuration as documented. --------- Co-authored-by: JerrettDavis <2610199+JerrettDavis@users.noreply.github.com> |
||
|
AxelRay
|
56b3e4c1b1
|
fix(proxy): skip OpenAI tool_search deferral for Codex client (#2729)
## Description
OpenAI Responses tool_search deferral injects `defer_loading` and a
`tool_search` tool for eligible gpt-5.4+ requests. When the model later
calls a deferred tool, the function_call item carries a `namespace`
field. Codex CLI round-trip structs drop unknown fields, so the next
request omits `namespace` and OpenAI returns 400, killing the session
mid-run. Proxy logs for these Codex turns show no tool savings, so the
injection breaks Codex without benefit.
This skips OpenAI tool_search deferral when the classified client is
Codex, leaving other clients unchanged.
Closes #2726
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Add `openai_tool_search_client_supported` and a Codex-only unsupported
client set
- Pass optional `client` into `inject_tool_search_deferral_openai` and
no-op for Codex
- Plumb `client` through Responses compression (HTTP, WebSocket,
passthrough) with legacy-signature retries
- Add regression tests for Codex skip and non-Codex still injects
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ .venv/bin/python -m pytest tests/test_openai_tool_search_deferral.py -q -o addopts=
34 passed, 1 warning in 0.27s
$ .venv/bin/python -m ruff check headroom/proxy/helpers.py headroom/proxy/handlers/openai.py tests/test_openai_tool_search_deferral.py
All checks passed!
$ .venv/bin/python -m ruff format --check headroom/proxy/helpers.py headroom/proxy/handlers/openai.py tests/test_openai_tool_search_deferral.py
3 files already formatted
```
## Real Behavior Proof
- Environment: Linux x86_64, Python 3.14.5 in repo .venv, shallow
checkout of headroomlabs-ai/headroom main at
|
||
|
Parideboy
|
01df245252
|
fix(proxy/cost): mark estimated-basis budget records and add an enforcement policy (#2713) (#2725)
## Description
`CostTracker.check_budget()` is a hard spend control — the Anthropic
handler refuses the request with a 429 once the period budget is gone.
The ledger that control reads could not tell a measured dollar from a
guessed one.
When a provider response carries no input-token breakdown,
`record_tokens()` substitutes Headroom's own `tokens_sent` estimate for
the input count so input cost isn't silently dropped from the budget.
That fallback is the right call, but the resulting record was
byte-identical to a provider-measured one: no field, no log line, no
separation in `/stats`. `RequestOutcome.uncached_input_tokens` defaults
to `0`, so any route whose response omits usage lands on this branch in
production. An estimate can drift in either direction, so a budget check
could pass after real spend had already gone over — with nothing saying
the decision rested on an estimate.
This keeps the fallback and makes it visible, then lets operators decide
what an estimate is allowed to do to a hard limit.
Closes #2713
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- New `headroom/proxy/budget_basis_policy.py` (pure policy module,
matching the existing `*_policy.py` convention): the
`measured`/`estimated` basis constants, the `count`/`ignore`/`block`
policy values, and `resolve_estimated_basis_policy()` (explicit value →
`HEADROOM_BUDGET_ESTIMATED_BASIS` → `count`; an unknown value warns once
and falls back rather than failing proxy startup).
- `headroom/proxy/cost.py`: ledger entries are now `CostEntry(timestamp,
cost_usd, basis)` instead of a bare tuple; `record_tokens()` marks the
fallback branch `estimated` and logs one WARNING per model (deduped the
same way pricing warnings are, per #2504 — an unguarded warning on this
path fires once per request for a provider that never reports usage);
new `period_cost_breakdown()` and an optional `basis` filter on
`get_period_cost()`; new `budget_denial_detail()` builds the 429 body
where the ledger lives; `check_budget()` honors the policy while keeping
its `(allowed, remaining)` signature.
- `stats()` gains `budget_estimated_basis` (the active policy) and
`budget_basis` (the period split: `total_usd`, `measured_usd`,
`estimated_usd`, `estimated_pct`, `records`, `estimated_records`).
`merge_cost_stats()` already spreads `**cost_stats`, so both reach
`/stats["cost"]` with no extra plumbing.
- Operator knob wired through every config layer:
`ProxyConfig.budget_estimated_basis` (`models.py`), the Click
`--budget-estimated-basis` option with `envvar=` (`cli/proxy.py`), the
argparse `--budget-estimated-basis` flag (`server.py`, `default=None` so
the env var stays reachable), and a `SettingField` in the `Budget` group
(`settings_store.py`).
- `headroom/proxy/handlers/anthropic.py`: the 429 body now comes from
`budget_denial_detail()`, which names how much of the period's spend was
booked from an estimate and distinguishes "you overspent" from "I refuse
to enforce a hard limit on a guess".
- `headroom/cli/doctor.py`: the budget check stays **PASS** and appends
the estimated share (and the policy, when it isn't the default). No new
WARN state — a provider that never reports usage would otherwise sit at
a permanent WARN. Every new read is `.get()` + type-guarded so `doctor`
still works against an older running proxy.
- `docs/content/docs/metrics.mdx`: a "Measured vs Estimated Spend"
subsection with the `/stats` shape and the three policy values.
- Tests: new `tests/test_cost_budget_basis.py` (20 tests) plus 4 new
`doctor` tests; `tests/test_anthropic_pre_upstream_backpressure.py`'s
cost-tracker double gained `budget_denial_detail()` to match the
handler's duck-typed contract.
### Policy values
| `HEADROOM_BUDGET_ESTIMATED_BASIS` | Effect on the hard limit |
|---|---|
| `count` (default) | Unchanged behavior — estimated spend consumes the
budget. |
| `ignore` | Booked and reported, but only measured spend enforces. |
| `block` | Fail closed — refuse rather than enforce a hard limit on a
guess. |
Default enforcement is unchanged. `CHANGELOG.md` is untouched.
## Testing
- [x] Unit tests pass (`pytest`) — every test covering the changed
modules; see `Not tested` for this machine's pre-existing environment
failures
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`) — clean on every file this
PR touches
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_cost_budget_basis.py tests/test_cost_tracker_counterfactual.py tests/test_cost_pricing_warning_dedup.py -q
31 passed
$ python -m pytest tests/test_cli_doctor.py -q
72 passed
$ python -m pytest tests/test_anthropic_pre_upstream_backpressure.py -q
25 passed
$ python -m pytest tests/test_proxy_settings_endpoints.py tests/test_proxy/test_settings_store.py -q
50 passed
# full suite (see "Not tested" below for the excluded modules and the pre-existing failures)
$ python -m pytest -q
...
tests\test_cost_budget_basis.py .................... [ 25%]
tests\test_cost_pricing_warning_dedup.py ... [ 25%]
tests\test_cost_tracker_counterfactual.py ........ [ 25%]
...
217 failed, 8807 passed, 657 skipped, 5318 warnings, 59 errors in 716.30s (0:11:56)
# same failing files re-run on clean upstream/main with the change stashed -> identical count
$ git stash push -u -- headroom tests docs
$ python -m pytest tests/test_log_compressor.py tests/test_cache/test_client_integration.py \
tests/test_fsutil.py tests/test_savings_ledger.py tests/test_ccr_mcp_http.py \
tests/test_router_registry_dispatch.py tests/test_proxy_savings_history.py \
tests/test_text_compressors.py tests/test_builtin_compressor_adapters.py \
tests/test_cli_proxy_env.py -q
73 failed, 182 passed in 34.82s # 40+16+2+2+1+1+1+4+3+3 = 73, matching the run above
$ python -m mypy headroom --ignore-missing-imports --python-version 3.13
# 12 errors, all in release_version.py / ccr/mcp_server.py / memory/mcp_server.py
# (stale local `mcp` stubs) — none in any file this PR touches
$ python -m ruff check headroom/proxy/budget_basis_policy.py headroom/proxy/cost.py headroom/proxy/server.py headroom/proxy/models.py headroom/proxy/handlers/anthropic.py headroom/cli/proxy.py headroom/cli/doctor.py headroom/settings_store.py tests/test_cost_budget_basis.py tests/test_cli_doctor.py tests/test_anthropic_pre_upstream_backpressure.py
All checks passed!
$ python -m ruff format --check <same 11 files>
11 files already formatted
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.13.11, branch
`fix/budget-estimated-basis-2713` off `upstream/main` @ `
|
||
|
Chester
|
3eb0122068
|
fix(learn): filter ambient user-role scaffolding (#2275)
## Description Fixes #2274. Headroom Learn currently trusts `role=user` as sufficient preference provenance. Agent harnesses can transport ambient UI and orchestration context in user-role messages, and OpenAI Responses normalization also promotes missing roles to `user`. Correction-like text in those inputs can therefore become durable user preferences. This change keeps preference learning fail-closed for known non-user sources while preserving genuine user corrections. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Refactoring only ## Changes Made - Preserve missing OpenAI Responses roles as `unknown` instead of promoting them to `user`. - Canonicalize user-role text before preference extraction. - Remove proxy-appended `## Relevant Memories` suffixes from preference evidence. - Reject strict ambient-only harness prefixes such as heartbeat, environment, workspace-instruction, delegation, and app-context envelopes. - Apply the same guard in `on_messages` and `_extract_preferences` for defense in depth. - Add regression coverage for system/developer/unknown roles, ambient-only user messages, memory-only messages, and mixed genuine-user-plus-memory input. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text 149 passed, 1 warning ruff check: passed ruff format --check: passed git diff --check: passed ``` Focused test files: ```text tests/test_memory/test_traffic_learner.py tests/test_openai_responses_traffic_learner.py ``` ## Real Behavior Proof - Environment: macOS; Python 3.13; current Headroom main; direct invocation of the real `TrafficLearner` class, with no proxy or database mocks - Exact command / steps: create `TrafficLearner(backend=None, min_evidence=1)`; feed system, developer, heartbeat user-role, and memory-only user-role messages; read `patterns_extracted`; feed a genuine user correction followed by a `## Relevant Memories` suffix; read `patterns_extracted` again - Observed result: `ambient_patterns=0`, `after_user_patterns=1` — the ambient batch produced no preference evidence; the genuine correction produced one pattern, while the appended memory content did not become evidence - Not tested: live provider traffic against a remote OpenAI endpoint; every possible third-party harness envelope; migration or cleanup of already-persisted noisy memories ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] No new dependency - [x] Fail-open proxy behavior is unchanged - [x] Regression tests added - [x] Public examples contain no real user data - [x] CHANGELOG update, if requested (not requested — N/A) ## Additional Notes This extends the source filtering introduced by #466 rather than replacing it. The prefix checks are deliberately strict and anchored at the start of a canonicalized message. The intended failure mode is a missed preference, not durable storage of non-user instructions. Note: the strict prefix set was discussed and confirmed in JerrettDavis's review approvals. |
||
|
Rod Boev
|
232fb49c73
|
fix(proxy): route Codex Live voice through a dedicated /v1/live transport (#2709)
## Description Codex Live traffic currently reaches an unrouted WebSocket path and receives HTTP 403 before the proxy can contact an upstream. Closes #2653 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring ## Changes Made - Add a dedicated `/v1/live` WebSocket route family and transparent transport. - Preserve subscription auth routing, account headers, origin policy, subprotocols, and text/binary frame bytes. - Propagate WebSocket close metadata and cancel relay tasks deterministically on every exit. - Keep Live outside the Responses parser, compression, memory injection, and Responses beta-header path. - Keep generic HTTP paths on the existing catch-all and document the Live aliases plus the derived-path override. - Add real-app route, relay, and loopback integration proof. - Add coverage for authorization fallback, defensive receive events, and cancellation cleanup in the Live relay. ## Testing The focused Live handshake, preservation suites, Ruff, format, and diff checks pass. The base comparison, Codex Desktop owner round trip, and ChatGPT backend acceptance of the derived `/backend-api/codex/live` path remain untested. - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy headroom --ignore-missing-imports`) - [x] New tests added for the reported failure - [x] Manual loopback testing performed ### Test Output ```text uv run pytest tests/test_codex_live.py -q: 6 passed, 7 warnings in 11.03s uv run pytest tests/test_provider_proxy_routes.py tests/test_proxy_codex_route_aliases.py tests/test_provider_codex_endpoints.py tests/test_openai_codex_routing.py -q: 53 passed in 90.54s (0:01:30) uv run ruff check headroom/providers/codex/live.py headroom/providers/proxy_routes.py headroom/proxy/handlers/openai.py headroom/proxy/ws_headers.py tests/test_codex_live.py: All checks passed uv run ruff format --check headroom/providers/codex/live.py headroom/providers/proxy_routes.py headroom/proxy/handlers/openai.py headroom/proxy/ws_headers.py tests/test_codex_live.py: 5 files already formatted uv run mypy headroom --ignore-missing-imports: Success: no issues found in 508 source files git diff --check: pass ``` ## Real Behavior Proof The local WebSocket integration floor uses real uvicorn, a real WebSocket client, and a real loopback WebSocket upstream. The base 403 comparison was not run. Head observes HTTP 101 on every Live alias and a byte-identical binary frame relay. - Environment: Windows, CPython 3.13, the Headroom proxy test environment. - Exact command / steps: run the focused Live test against the local uvicorn proxy and loopback WebSocket upstream, then run the preservation suite listed in `Test Output`. - Observed result: all four Live aliases return HTTP 101, negotiate `codex.live.v1`, preserve text and binary frames, and pass the preservation suite. - Not tested: Codex Desktop Live session; ChatGPT backend acceptance of `/backend-api/codex/live`; the base 403 comparison. ## Review Readiness - Live has a separate transport and does not enter Responses handling. - Existing Responses and generic passthrough suites remain preservation gates. - No `CHANGELOG.md` or install/crate changes are included. - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] Closes #2653 - [x] Real loopback handshake and binary-frame proof required - [x] No audio payload logging - [x] No unqualified end-to-end claim ## Screenshots Not applicable. ## Additional Notes The upstream Live path is derived from the repository’s Codex URL formula and remains explicitly unconfirmed until owner evidence is available. |
||
|
Parideboy
|
3a27c4dacb
|
fix(proxy/debug): reconcile Kompress warmup state in /debug/warmup (#2711)
## Description
`/debug/warmup` serialized the warmup registry verbatim, so a Kompress
slot left at the startup snapshot kept reporting `{"status": "null",
"info": {"source_status": "deferred"}}` forever — even while the ONNX
model was loaded and actively compressing.
`/health` and `/readyz` already fix this: #2402 added
`_reconcile_kompress_health()`, which promotes the slot from live
runtime state. The debug route never called it, so its answer depended
on whether a health probe happened to run first. That is the half of
#2624 still reproducing on `main`.
Second defect: `WarmupSlot.mark_loaded()` only *updates* `info`, so the
startup-planted `source_status: "deferred"` survived promotion and the
slot serialized as the self-contradictory `{"status": "loaded", "info":
{"source_status": "deferred", "backend": "onnx"}}`.
Closes #2624
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/server.py`: call the existing
`_reconcile_kompress_health()` in the `/debug/warmup` route before
serializing the registry. The reconciler never instantiates a compressor
and never calls `preload()` / `ensure_background_load()` / `compress()`
— it only reads `is_ready()` / `ready_backend()` on an already resident
instance, or falls back to the module-level ONNX cache — so the endpoint
stays side-effect free and idempotent.
- `headroom/proxy/server.py`: stamp `source_status="runtime"` at both
`mark_loaded()` promotion sites in `_reconcile_kompress_health()` (the
resident-compressor path and the `_kompress_cache` fallback),
overwriting the stale startup marker.
- `tests/test_proxy_debug_endpoints.py`: three regression tests plus a
read-only compressor stub whose `preload` / `ensure_background_load`
raise, so a future change that makes the debug route trigger a load
fails loudly.
- `tests/test_proxy_health.py`: assert the promoted slot's
`info["source_status"] == "runtime"`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ pytest tests/test_proxy_debug_endpoints.py tests/test_proxy_health.py tests/test_proxy_warmup.py -q
tests\test_proxy_debug_endpoints.py ............................. [ 52%]
tests\test_proxy_health.py ................. [ 83%]
tests\test_proxy_warmup.py ......... [100%]
============================= 55 passed in 36.56s =============================
$ ruff check headroom/proxy/server.py tests/test_proxy_debug_endpoints.py tests/test_proxy_health.py
All checks passed!
$ ruff format --check headroom/proxy/server.py tests/test_proxy_debug_endpoints.py tests/test_proxy_health.py
3 files already formatted
$ mypy headroom --ignore-missing-imports
Success: no issues found in 506 source files
```
The three new tests were confirmed to be genuine regression tests: with
the `server.py` change reverted and the tests kept, all three fail.
```text
$ git stash push -- headroom/proxy/server.py && pytest tests/test_proxy_debug_endpoints.py -q -k kompress
FAILED tests/test_proxy_debug_endpoints.py::test_debug_warmup_promotes_deferred_kompress_after_runtime_load
FAILED tests/test_proxy_debug_endpoints.py::test_debug_warmup_keeps_pending_kompress_null
FAILED tests/test_proxy_debug_endpoints.py::test_debug_warmup_never_starts_kompress_loading
====================== 3 failed, 26 deselected in 3.98s =======================
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.13.11, pytest 9.1.1, ruff 0.15.x,
mypy 1.20.2, branch based on `main` at
|
||
|
Raúl
|
9ce5af02b1
|
test(recover-codex): bind AF_UNIX socket via short relative path (#2396)
## Description `test_recovery_records_sockets_and_secures_both_backups` binds a Unix domain socket at its absolute path under pytest's `tmp_path`. On macOS the `AF_UNIX` `sun_path` limit (~104 bytes) is shorter than that path, so `bind()` raises `OSError: AF_UNIX path too long` and the test fails locally. It stays green on CI Linux only because `/tmp`-rooted temp paths there are short enough. Bind a short relative name from inside `source` instead; the socket is still created at `source/codex.sock` and the recovery scan behaves identically. Closes #2394 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `tests/test_cli/test_recover_codex.py`: in `test_recovery_records_sockets_and_secures_both_backups`, `monkeypatch.chdir` into `source` and `bind(socket_path.name)` (a short relative name) instead of `bind(str(socket_path))` (a long absolute path). Added the `monkeypatch` fixture to the signature and a one-line comment explaining the `sun_path` cap. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy`) - [x] Manual testing performed ### Test Output ```text # BEFORE (on this macOS box, at the branch base): $ uv run pytest -q \ "tests/test_cli/test_recover_codex.py::test_recovery_records_sockets_and_secures_both_backups" tests/test_cli/test_recover_codex.py:777: in test_recovery_records_sockets_and_secures_both_backups codex_socket.bind(str(socket_path)) E OSError: AF_UNIX path too long 1 failed in 0.70s # AFTER (whole file, no regressions): $ uv run pytest -q tests/test_cli/test_recover_codex.py 31 passed in 0.72s $ uv run ruff format --check tests/test_cli/test_recover_codex.py 1 file already formatted $ uv run ruff check tests/test_cli/test_recover_codex.py All checks passed! $ uv run mypy tests/test_cli/test_recover_codex.py Success: no issues found in 1 source file ``` <img width="1073" height="200" alt="image" src="https://github.com/user-attachments/assets/82155e76-3005-4c4f-93f4-4802ae5e7405" /> ## Real Behavior Proof - Environment: macOS 26.5 (darwin 25.5.0), Python 3.13.7, ruff 0.14.14, `tempfile.gettempdir()` = `/var/folders/.../T` (48 chars, before the `pytest-of-*/pytest-N/test_.../headroom-codex-home-broken/codex.sock` suffix, which pushes the absolute `sun_path` over the macOS ~104-byte cap). - Exact command / steps: run the focused test at the branch base (fails with `AF_UNIX path too long`), apply the one-line relative-bind change, re-run the whole file. - Observed result: before = 1 failed; after = 31 passed. `ruff`/`mypy` clean. - Not tested: Linux/Windows (the test is `skipif` on win32 / no `AF_UNIX`; on Linux it already passed pre-change because temp paths are short). No production code touched, so no proxy/runtime behavior was re-validated. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (N/A: test-only) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works (this IS the corrected test; it fails before and passes after) - [x] New and existing unit tests pass locally with my changes - [x] I did not edit `CHANGELOG.md` ## Additional Notes - Pure test-portability fix; no production behavior change. `test:` type keeps it out of the release-please changelog, which is correct for a test-only change. |
||
|
Tejas Chopra
|
3e348f327f
|
fix(ccr): stop persisting retrieval markers as original content (#2694) (#2703)
## Description CCR entries could end up holding a `<<ccr:...>>` marker — or nothing at all — where the original bytes belonged, so `headroom_retrieve(hash)` answered with the very placeholder the caller was trying to resolve. For a base64/credential field that is permanent, silent data loss: the inner marker's hash is the only handle on the real payload, and it disappears from anywhere the model can see. Four sites, one root cause — **a compressed intermediate (or nothing) was stored in place of the source**, the same defect class as #1209 (tag placeholders persisted as originals). Closes #2694 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **`compaction/walker.rs`** — `walk_array` compacted through the store-LESS `compact()`. Opaque cells inside a compacted table got a marker whose payload was **never written**, so retrieval 404'd forever. Now uses `compact_with_store` so the emitted hash resolves. - **`compaction/classifier.rs`** — nothing stopped an already-marked string from being offloaded a second time, which stashed the MARKER as the new entry's "original". Marker-bearing text is our own output, not source content, so it is never classified opaque. One guard at the choke point both the walker and the table compactor share. - **`smart_crusher/crusher.rs`** — on the prose-hook path the row-drop marker hashed and stored rows whose leaves were **already** rewritten (prose compressed, blobs marker-substituted), so retrieving dropped rows returned compressed output. Now hashes and stashes the pre-processing array via `crush_array_with_source`. - **`content_router.py`** — compression pinning matched only `Retrieve more: hash=` / `Retrieve original: hash=`, **not** `<<ccr:`, so opaque-blob output was readmitted to the compressor on a later turn — the path that feeds the corruption above. Consolidated into `_is_already_compressed()` and applied at all three pinning sites. - **`cache/compression_store.py`** — store-level guard: refuse to persist a *bare* marker as `original_content` and log at ERROR, so a future producer regression surfaces loudly instead of silently converting "retrievable" into "gone". Deliberately narrow — originals may legally *contain* markers (nested offloads); only a bare marker is rejected. - **Regression tests** — `test_nested_table_markers_resolve_to_source_bytes` (asserts payloads are verbatim-retrievable, not merely that a marker was emitted) and `test_already_marked_content_is_not_re_offloaded`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ cargo build -p headroom-core Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 37s $ cargo test -p headroom-core --lib smart_crusher test result: ok. 329 passed; 0 failed; 0 ignored; 0 measured; 583 filtered out; finished in 0.19s $ python -m pytest tests/test_transforms/test_smart_crusher_ccr_roundtrip.py -q 16 passed in 0.68s $ python -m pytest tests/test_ccr_row_drop_store_bridge.py tests/test_ccr_tool_injection.py -q 50 passed in 12.52s $ python -m pytest tests/test_compression_store.py tests/test_lossless_mode.py -q 100 passed in 13.14s $ ruff check headroom/transforms/content_router.py headroom/cache/compression_store.py \ tests/test_transforms/test_smart_crusher_ccr_roundtrip.py All checks passed! $ mypy headroom/transforms/content_router.py headroom/cache/compression_store.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - **Environment:** macOS (Darwin 25.4.0), Python 3.12.6, headroom-ai 0.33.0 editable, `HEADROOM_CCR_BACKEND=memory`, Rust extension rebuilt via `maturin develop --release`. - **Exact command / steps:** compact a nested document — 5 rows whose `detail` field is a stringified sub-array of 6 base64 blobs (1600 B each) — then, for every `<<ccr:HASH>>` marker in the output, call `ccr_get(HASH)` and check the payload is the verbatim source rather than a marker. ```python inner = [{"k": f"key{i}", "v": i, "tok": blob(1200)} for i in range(6)] doc = {"rows": [{"id": i, "detail": json.dumps(inner), "note": "x"} for i in range(5)]} out = SmartCrusher().compact_document_json(json.dumps(doc)) for h in re.findall(r"<<ccr:([0-9a-f]+)", out): payload = crusher.ccr_get(h) # must be real bytes, not a marker ``` - **Observed result — BEFORE (on `main`):** all six payloads collapsed into a single dead marker. The rendered sub-table was re-classified opaque (`html`, because `<<` reads as a tag), offloaded again, and its payload never stored — so the six inner hashes were erased from the visible text *and* the outer hash resolved to nothing. ```text {"rows":"[5]{detail:string,id:int,note:string} <<ccr:3fb1d44933da,html,289B>>,0,x <<ccr:3fb1d44933da,html,289B>>,1,x ..."} 3fb1d44933da -> RUST MISS # unrecoverable — 6 × 1600 B gone ``` - **Observed result — AFTER (this branch):** the sub-table stays inline, each blob keeps its own marker, and every marker resolves to verbatim source. ```text {"rows":"[5]{detail:string,id:int,note:string} \"[6]{k:string,tok:string,v:int} key0,\"\"<<ccr:955b1fed2ef7,base64,1.6KB>>\"\",0 ..."} 6ad5846997f4: resolves, len=1600, is-verbatim-source=True 78a0bd9364a7: resolves, len=1600, is-verbatim-source=True 955b1fed2ef7: resolves, len=1600, is-verbatim-source=True a0cef69da7f0: resolves, len=1600, is-verbatim-source=True dfcde5e940c0: resolves, len=1600, is-verbatim-source=True e57c4e0a3ce8: resolves, len=1600, is-verbatim-source=True RESULT: PASS — every marker resolves to real source bytes ``` ## Notes for reviewers - The issue also reports **function words dropped from retained prose** (`is`, `a`, `the`) and **interleaved log output corrupting `headroom doctor`'s table borders**. Those are separate defects on different paths (extractive prose compression and log-handler buffering respectively) and are **not** addressed here — this PR is scoped to the CCR store/retrieve corruption. They should be tracked separately; the prose one overlaps #2586. - The `crusher.rs` prose-hook fix is on the Rust pipeline (`json_offload`) rather than the Python proxy path, but it is the same store-the-intermediate bug and was cheap to close while in the file. |
||
|
Raúl
|
1a2688b57f
|
test(kompress): close patch-coverage gaps from #2716 (#2721)
## Description Codecov flagged 9 uncovered lines on #2716 after it merged: `hf_entry_known_absent`'s own body in `headroom/onnx_runtime.py` was only ever exercised indirectly (every existing test in `tests/test_transforms/test_kompress_compressor.py` monkeypatched it away rather than calling the real implementation), and `_load_pytorch_weights` / `_load_kompress_pytorch` in `headroom/transforms/kompress_compressor.py` had three untested branches: the double cache-miss under `allow_download=False` (merged.pt confirmed absent AND the plain fallback also not cached), a genuine non-404 download failure propagating instead of silently falling back, and the already-cached fast path in `_load_kompress_pytorch`. ## Type of Change - [ ] Bug fix - [ ] New feature - [x] Test coverage improvement, no production code change ## Changes Made - `tests/test_onnx_runtime.py`: added `_write_fake_hf_cache` (builds a minimal on-disk HF hub cache layout, including the `.no_exist/<hash>/<filename>` marker huggingface_hub writes after a real 404) and three direct tests of `hf_entry_known_absent` against the real `huggingface_hub.try_to_load_from_cache`, not a mock of it. - `tests/test_transforms/test_kompress_compressor.py`: added `test_cache_only_raises_when_confirmed_absent_but_plain_also_missing`, `test_genuine_download_failure_propagates_instead_of_falling_back`, and a new `TestLoadKompressPytorchCaching` class covering the already-cached fast path. ## Testing ```text $ .venv/bin/python3 -m pytest tests/test_onnx_runtime.py tests/test_transforms/test_kompress_compressor.py -q 51 passed $ .venv/bin/python3 -m pytest tests/ -k "kompress or onnx_runtime" -q --cov=headroom.transforms.kompress_compressor --cov=headroom.onnx_runtime --cov-report=term-missing # before: onnx_runtime.py Missing includes 132-136 (hf_entry_known_absent's entire body); # kompress_compressor.py Missing includes 805-806, 818, 836 # after: none of those lines appear in Missing anymore 191 passed, 7 skipped $ .venv/bin/python3 -m ruff format --check tests/test_onnx_runtime.py tests/test_transforms/test_kompress_compressor.py 2 files already formatted $ .venv/bin/python3 -m ruff check tests/test_onnx_runtime.py tests/test_transforms/test_kompress_compressor.py All checks passed! ``` ## Review Readiness - Test-only, additive diff (113 insertions, 0 deletions, 0 lines touched outside the two test files). No behavior change possible. ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] New and existing unit tests pass locally with my changes - [x] I did not edit `CHANGELOG.md` ## Additional Notes Not closing: the remaining branch-partial on the `device == "auto"` cuda/mps/cpu selection in `_load_kompress_pytorch` (would need mocking `torch.cuda.is_available()` / `torch.backends.mps.is_available()` for marginal benefit); left as-is. |
||
|
Tejas Chopra
|
2797099bec
|
feat(compress): accept config.frozen_message_count on /v1/compress (#2718)
## Problem
Callers that resend a growing conversation every turn — agent loops
generally, and an in-progress Strands plugin specifically — cannot keep
a stable prompt-cache prefix through `/v1/compress`. The router
compresses older messages more aggressively as the conversation grows,
so their bytes change and the provider's cache misses from the first
rewritten message onward. That trades a 90% read discount for nothing.
Measured before this change, compressing the same conversation at
increasing lengths and comparing the first 16 messages against the
4-turn baseline:
```
turns msgs first 16 still byte-identical?
4 16 16/16 OK
8 32 16/16 OK
16 64 12/16 DRIFT at [2, 6, 10, 14]
32 128 12/16 DRIFT at [2, 6, 10, 14]
64 256 12/16 DRIFT at [2, 6, 10, 14]
```
Indices 2/6/10/14 are the tool-result messages.
## Fix
`TransformPipeline` and `ContentRouter` already honour
`frozen_message_count` — `content_router.py` skips any message below the
index, and `pipeline.py` even logs *"freezing first N/M messages (prefix
cached by provider)"*. It was simply missing from this endpoint's
`config` parsing, so no HTTP caller could reach it.
This adds it alongside the existing `compress_user_messages` /
`target_ratio` / `protect_recent` / `protect_analysis_context` options.
Pinning still lets cross-message transforms such as dedup *read* the
prefix; it only forbids rewriting it. `protect_recent` guards the
opposite end of the list and cannot express this.
Invalid values return 400, matching the existing `config.mode`
validation. `bool` is rejected explicitly, since `isinstance(True, int)`
is `True` in Python and a JSON `true` silently becoming
`frozen_message_count=1` would be a nasty surprise.
## Verified against a live proxy
```
sent 64 messages, pinned first 32 -> returned unchanged: True
unpinned tail still compressed: True
simulated agent loop carrying the compressed prefix forward:
4 turns: prefix of 0 held 16 turns: prefix of 48 held
8 turns: prefix of 16 held 24 turns: prefix of 64 held
12 turns: prefix of 32 held 32 turns: prefix of 96 held
prefix never drifted across the whole run: True
```
## Tests
13 new tests in `TestCompressEndpointFrozenMessageCount`, including the
regression itself: compress a 6-turn and a 24-turn conversation with the
same pin and assert the prefix is identical. 48/48 pass in
`test_proxy_compress_endpoint.py` (35 pre-existing, unchanged). ruff and
ruff-format clean.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Raúl
|
46da91b2f1
|
fix(kompress): load merged.pt for the v2 checkpoint instead of the unmerged PEFT safetensors (#2716)
# PR draft: fix(kompress): load merged.pt for the v2 checkpoint instead of the unmerged PEFT safetensors Branch: `rnoz/fix-kompress-merged-checkpoint` (off `upstream/main`). Two commits. Issue: https://github.com/headroomlabs-ai/headroom/issues/2714 (filed, open). --- ## Description `_load_kompress_pytorch` in `headroom/transforms/kompress_compressor.py` downloaded `model.safetensors` from the default model repo `chopratejas/kompress-v2-base` and loaded it with `strict=False`, discarding the missing/unexpected key report. That file is the unmerged PEFT checkpoint (encoder keys prefixed `encoder.base_model.model...`), which never matches `HeadroomCompressorModel`'s plain `encoder.*` keys. The LoRA-adapted encoder weights were silently dropped while `token_head`/`span_conv` happened to match and loaded fine, so the model ran with a stock, non-adapted `answerdotai/ModernBERT-base` encoder feeding correctly trained decision heads, with no error and a healthy status reported everywhere. `scripts/export_kompress_v2_onnx.py` already documents this exact mismatch and loads the correct `merged.pt` sub-state-dicts for its own export path. This PR mirrors that same loading logic into the runtime PyTorch loader, with a fallback to the plain `model.safetensors` format for repos that never shipped a `merged.pt` (verified against the v1 `chopratejas/kompress-base` repo via the public HF API, which has no `merged.pt`). Both paths now check the missing/unexpected key report and raise instead of silently proceeding on a mismatch. A second commit fixes a gap an adversarial review caught in the first: the cache-only (`allow_download=False`, startup preload) path could not tell "this repo genuinely has no merged.pt" apart from "merged.pt exists but is not downloaded yet", so it would have fallen back to a stale `model.safetensors` left over from before this fix on exactly the upgrade path this PR is meant to close. It now uses `huggingface_hub`'s own `.no_exist` cache marker (via a new `hf_entry_known_absent()` helper in `headroom/onnx_runtime.py`) to make that distinction without a network call, and only falls back when absence is confirmed; otherwise it raises `KompressModelNotCached` so the caller defers instead of guessing. Closes #2714 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/transforms/kompress_compressor.py`: added `_load_merged_state_dict`, `_load_plain_state_dict`, and `_load_pytorch_weights`, replacing the inline `model.safetensors` download + `load_state_dict(strict=False)` call in `_load_kompress_pytorch`. `merged.pt` is tried first; the plain format is only used when its absence is confirmed. - `headroom/onnx_runtime.py`: added `hf_entry_known_absent()`, a thin wrapper around `huggingface_hub.try_to_load_from_cache()` that reads the on-disk `.no_exist` marker HF writes after a real 404, so cache-only code can distinguish "confirmed absent" from "never checked" without hitting the network. - `tests/test_transforms/test_kompress_compressor.py`: added `TestPytorchWeightLoading` (8 tests) covering the merged-checkpoint happy path, missing-section and key-mismatch failures, the plain-format fallback for repos without `merged.pt`, and the cache-only ambiguity fix (confirmed-absent vs unconfirmed). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] Manual testing performed (real download against the live `chopratejas/kompress-v2-base` repo, not just mocks) ### Test Output ```text $ .venv/bin/python3 -m pytest tests/ -k kompress -q 178 passed, 7 skipped in 22.39s $ .venv/bin/python3 -m ruff check headroom/onnx_runtime.py headroom/transforms/kompress_compressor.py tests/test_transforms/test_kompress_compressor.py All checks passed! $ .venv/bin/python3 -m ruff format --check headroom/onnx_runtime.py headroom/transforms/kompress_compressor.py tests/test_transforms/test_kompress_compressor.py 3 files already formatted $ .venv/bin/python3 -m mypy headroom/onnx_runtime.py headroom/transforms/kompress_compressor.py Success: no issues found in 2 source files ``` ## Real Behavior Proof Ran the actual fixed loader against the live `chopratejas/kompress-v2-base` HF repo (not a mock), before and after each fix: ```text # BEFORE (parsed the real cached model.safetensors header by hand, no safetensors lib needed): total tensors: 316 encoder.base_model.model.embeddings.norm.weight encoder.base_model.model.embeddings.tok_embeddings.weight ...(all 310 encoder tensors share this prefix)... span_conv.0.bias / span_conv.0.weight / span_conv.2.bias / span_conv.2.weight token_head.bias / token_head.weight exact prefix match count with plain "encoder.<rest>": 0 # This confirms the pre-fix code's model.load_state_dict(state_dict, strict=False) # silently dropped every encoder weight (0 keys match HeadroomCompressorModel.encoder), # while token_head/span_conv happened to match and loaded. # AFTER (commit 1, real merged.pt download + load): $ .venv/bin/python3 -c " import headroom.transforms.kompress_compressor as kmod model = kmod._get_model_class()() kmod._load_pytorch_weights(model, 'chopratejas/kompress-v2-base', allow_download=True) print('SUCCESS: 0 missing/unexpected keys across all three sections') " SUCCESS: 0 missing/unexpected keys across all three sections # AFTER (full pipeline, real end-to-end compression through the public API): $ .venv/bin/python3 -c " import headroom.transforms.kompress_compressor as kmod compressor = kmod.KompressCompressor() result = compressor.compress(sample_traceback_plus_boilerplate_text) print(result.original_tokens, result.compressed_tokens, result.tokens_saved) print('ValueError: bad input' in result.compressed) " 497 454 43 True # must-keep line (the actual error) survived compression # AFTER (commit 2, cache-only ambiguity): unit tests # test_cache_only_defers_instead_of_using_stale_plain_checkpoint: PASSED # test_cache_only_uses_plain_checkpoint_when_merged_pt_confirmed_absent: PASSED ``` ## Review Readiness - [x] I have performed a self-review - [x] An independent adversarial review pass was run on both commits before this PR was opened; its one finding (the cache-only ambiguity) is fixed in commit 2, verified with new regression tests - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (N/A: internal loader behavior, no public API or config surface changed) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective (regression tests for both the original silent-drop bug and the cache-only ambiguity found in review) - [x] New and existing unit tests pass locally with my changes - [x] I did not edit `CHANGELOG.md` ## Additional Notes - Both `merged.pt` and the plain `model.safetensors` fallback now raise loudly on any state-dict mismatch instead of proceeding with partially-loaded weights, closing the general silent-failure class this bug belonged to, not just this one instance of it. - No other call sites of `_load_kompress_pytorch` or its removed inline code exist; its public signature is unchanged. |
||
|
Parideboy
|
6d5516dcb8
|
feat(code): add PHP support to CodeAwareCompressor (#2423)
## Description Adds PHP to `CodeAwareCompressor`, fixing #201. PHP was already *detected* as code (Magika labels in `headroom/compression/detector.py` include `php`, and the Rust `magika_detector.rs` lists it too) but there was no PHP `LangConfig`, so PHP content silently passed through uncompressed. This wires PHP through the tree-sitter compression path following the C# pattern (the most recently added, fully functional language — deliberately not the quarantined Perl path). A secondary detection bug is fixed along the way: PHP's `$variables` match Perl's prefilter regex, and the existing Perl-dominance guard in `detect_language` returned `UNKNOWN` for PHP files. An explicit `<?php` open tag — which no Perl source contains — now drops Perl from the candidate set before that guard runs. ## Type of Change - [ ] Bug fix - [x] New feature - [ ] Documentation update - [ ] Refactor - [ ] Other ## Changes Made - `headroom/transforms/code_compressor.py`: `CodeLanguage.PHP` + `phtml`/`php5`/`php7`/`php8` aliases; PHP `LangConfig` built from the actual tree-sitter-php grammar (node names verified by parsing samples): `namespace_use_declaration` imports, `function_definition`/`method_declaration` functions, `class_declaration`/`interface_declaration`/`trait_declaration` classes, `enum_declaration` types, `declaration_list` class bodies, `compound_statement` function bodies. `namespace_definition` maps to `package_node` so statement-scoped `namespace App;` hoists ahead of the `use` imports (required PHP ordering); the rare block-scoped `namespace A { }` form takes the same path and is preserved verbatim — valid output, just no compression inside the block. PHP prefilter regexes added; supported-languages error message updated; `<?php`-tag Perl disambiguation in `detect_language`. - `headroom/transforms/content_detector.py`: `php` entry in `_CODE_PATTERNS` so raw PHP classifies as `SOURCE_CODE` and reaches the code-aware route. - `tests/test_transforms/test_code_compressor.py`: new `TestPhpSupport` mirroring `TestCSharpSupport` — signatures preserved / bodies elided, `<?php` → `namespace` → `use` → declarations ordering, auto-detection despite the Perl sigil overlap, alias coercion, malformed passthrough. - `tests/test_code_compressor_language_alias.py`: `php` in the canonical list, `phtml` in the alias table. - `docs/content/docs/code-compression.mdx`: PHP added to the Tier 2 supported-languages row. No new dependency: `tree-sitter-language-pack` (the existing `[code]` extra) already ships the PHP grammar. No Rust changes needed. ## Testing - [x] New unit tests added and passing - [x] Full affected test suites pass locally **Test Output** ``` $ python -m pytest tests/test_transforms/test_code_compressor.py tests/test_code_compressor_language_alias.py -q ============================= 120 passed in 7.81s ============================= $ python -m pytest tests/test_transforms/ -q 3 failed, 443 passed # the 3 failures (kompress ONNX thread caps, kompress size gate, # text_crusher unicode parity) reproduce identically on a clean # upstream/main checkout in this environment — pre-existing local # ONNX runtime quirks, unrelated to this change $ ruff check . (0.15.17, CI-pinned) → All checks passed! | ruff format --check → clean $ mypy headroom/transforms/code_compressor.py headroom/transforms/content_detector.py --ignore-missing-imports Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13, tree-sitter + tree-sitter-language-pack (<1.0) installed, branch `feat/201-php-code-compression` off `upstream/main`. - Exact command / steps: parsed PHP samples (namespaced class w/ methods, block-scoped namespace, mixed HTML+PHP) with `tree_sitter_language_pack.get_parser('php')` to verify every node name used in the config; then ran `CodeAwareCompressor().compress(php_code, language="php")` and `compress(php_code)` (auto-detection) on a 48-line realistic service class. - Observed result: explicit and auto-detected paths both return `language=CodeLanguage.PHP`, `compression_ratio=0.64`, `syntax_valid=True`; method bodies elided to `// [N lines omitted]` while `<?php`, `namespace`, `use` lines, class header, and all signatures are preserved verbatim in the original order. Before the detection fix, auto-detection returned `UNKNOWN` (Perl prefilter dominance) — reproduced and then verified fixed. - Not tested: exotic PHP shapes (heredoc-heavy code, attributes `#[...]` on methods, interleaved multi-`<?php ?>` HTML templates beyond the basic mixed case); these fall back to verbatim preservation via the uncaptured-node pass or malformed-passthrough, both of which are covered by tests for the simple cases. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Tejas Chopra
|
f2c48e26c6
|
feat(compress): reach the lossless provider seam on the general path and default /v1/compress to marker-free output (#2691)
## Description Two related changes to the compression seams, plus the review fixes for both. Supersedes #2661 and #2662, which are closed in favour of this branch — the fixes are inseparable from the code they fix, so reviewing them together is cheaper than landing two PRs and patching them afterwards. **1. A registered lossless provider now competes on the general path.** The `headroom.transforms.lossless_provider` seam was only ever consulted from `_lossless_compact_excluded`, gated on `DEFAULT_EXCLUDE_TOOLS` (`config.py:216` — `Read/Grep/Glob/Write/Edit/WebSearch/WebFetch`). Gateway traffic carries the caller's own tool names — LiteLLM's `headroom` guardrail (https://docs.litellm.ai/docs/proxy/headroom) posts requests containing tools like `search_docs` / `run_ci` / `fetch_rows` — so a registered provider was structurally unreachable for every gateway/sidecar deployment. The seam existed; nothing could get to it. **2. `POST /v1/compress` is marker-free by default.** A CCR marker is only useful to a caller that also injects the `headroom_retrieve` tool AND can reach `/v1/retrieve`. Neither holds here: tool injection lives in the provider request handlers (`handlers/anthropic.py:1894`), never in `handle_compress`; and every `/v1/retrieve*` route is `Depends(_require_loopback)` (`server.py:4422, 4470, 4749, 4781`) with no remote opt-in — `HEADROOM_COMPRESS_ALLOW_REMOTE` drops the loopback dependency on `/v1/compress` only. So a gateway forwards a `Retrieve more: hash=…` pointer the model cannot follow, and the proxy pays a CCR store write nobody reads. `config.mode="ccr"` opts back in. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [x] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made ### Seam — `content_router.py`, `lossless_provider.py` - `_lossless_first` (STAGE 0, every block on every path) consults `get_lossless_provider()` and keeps whichever output is smaller. **Strict no-op when no provider is registered**, which is the default; and because it is best-of rather than authoritative, a provider can never do worse than the built-in folds. - Malformed provider output can no longer escape. Every shape check runs inside the `try`: result must be `None`, or a 2-element tuple/list of two `str`. Anything else is ignored at debug level. (Previously the unpack sat outside the `try`, so a 3-tuple raised `ValueError` up through `TransformPipeline.apply`, which re-raises.) - Empty / whitespace-only candidates are rejected rather than silently replacing block content. - **Providers are never offered diff content.** Diff folding is subtractive with no inverse check and a reflowed hunk breaks `git apply` — the same reason the built-in `diff` fold is restricted at `content_router.py:2481`. - The third-party `kind` label is sanitised against `^[a-z0-9_]{1,32}$` before reaching `transforms_applied` and the per-strategy metric dicts, so a caller-controlled string cannot explode Prometheus label cardinality. `fullmatch`, not `match`: `$` also matches before a trailing newline, which would put a newline in a label. - `set_lossless_provider(provider, *, verifier=None)` — in lossless-only mode, where STAGE 0's output is final and there is no marker to recover from, a registered verifier must confirm the fold or the candidate is dropped. No verifier registered = today's behaviour. `provider=None` clears both. - The provider is invoked once per block, not twice (`_has_lossless_fold` probes `_lossless_first` and discards the result, then STAGE 0 recomputes). Bounded memo, wholesale clear on overflow, no lock — a race costs one redundant fold. The memo keys on the provider registration generation, so registering or clearing a provider after a block was already folded takes effect. - The seam docstring now records that the provider runs on the general path and inside the parallel compression pool, so it must be thread-safe as well as deterministic. ### Route — `handlers/openai.py`, `server.py` - `_derived_compress_pipeline(key, **overrides)` replaces the copy-pasted pipeline-derivation block; `_no_ccr_pipeline` (the new default) and `_lossy_inline_pipeline` both use it. - The default pipeline is **built at startup** and included in `_eager_preload_transforms`, so a fresh pod does not pay ContentRouter construction and compressor load on its first request, inside the compression-executor budget. - An unrecognised `config.mode` returns 400 naming the valid values instead of silently falling back to the default. - Claude-family model names resolve their context limit from the Anthropic provider. Real divergence: `bedrock/anthropic.claude-3-5-sonnet` is 200000 there and 128000 on the OpenAI provider. The tokenizer still comes from the OpenAI pipeline's provider — a separate, larger change, noted in a comment. - Documents why the derived router deliberately does **not** share the base router's compression cache: keys do not encode CCR-marker mode, so sharing would leak marker-laden entries into the marker-free path. ### Behavior change A `/v1/compress` caller that relied on default markers now gets none. The only in-tree caller that can resolve them is the TypeScript SDK (`sdk/typescript/src/client.ts:398 retrieve`, `:422 handleToolCall`); it needs `config: {"mode": "ccr"}` to keep today's behaviour, and landing that SDK default in the same release would leave only gateway callers — for whom markers were never resolvable — seeing a difference. No `HEADROOM_COMPRESS_DEFAULT_MODE` compat env deliberately: a flag nobody sets becomes permanent debt, and the wire-level `mode` already covers the one caller that needs it. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_lossless_first_dispatch.py tests/test_lossless_excluded_compaction.py \ tests/test_lossless_mode.py tests/test_lossless_then_lossy.py tests/test_lossless_diff_fold_guard.py \ tests/test_bash_search_lossless_fold.py tests/test_proxy_compress_endpoint.py \ tests/test_ccr_row_drop_store_bridge.py tests/test_gateway_sidecar_ports.py tests/test_compress_api.py \ tests/test_platform_stabilization_functional.py tests/test_proxy_eager_preload_bind.py \ tests/test_proxy_warmup.py tests/test_router_registry_smartcrusher.py -q 189 passed in 32.04s $ ruff check <all 9 changed files> All checks passed! $ ruff format --check <all 9 changed files> 9 files already formatted $ mypy headroom/transforms/content_router.py headroom/transforms/lossless_provider.py \ headroom/proxy/handlers/openai.py headroom/proxy/server.py Success: no issues found in 4 source files ``` New tests cover, one concern each: every malformed provider shape; empty and whitespace-only results; diff content never reaching a provider (call-recording); `kind` sanitisation including the trailing-newline case; the verifier accepting / rejecting / raising; clearing a provider clearing its verifier; single provider invocation per block; memo invalidation on re-registration; unknown and valid `mode` values; the default pipeline existing before any request; and Claude vs OpenAI context-limit resolution with `token_budget` precedence preserved. ## Real Behavior Proof - **Environment:** macOS arm64, Python 3.12.6, proxy built from `_proxy_config_from_env()` with the default `coding` savings profile; Kompress both disabled and offloaded to a remote `kompress-v2-base` endpoint; `HEADROOM_COMPRESSION_TIMEOUT_SECONDS=300`. - **Steps:** `POST /v1/compress` over `TestClient` with OpenAI-shaped payloads under non-excluded tool names (`run_ci`, `list_files`, `code_search`, `fetch_rows`) — a CI log with ANSI escapes and repeated lines, a 160-path listing, a 150-line grep dump, a 150-row JSON array; plus a second payload with a RAG user blob, a 200-row JSON tool result and a 300-line log. Ran with and without a provider registered via `set_lossless_provider`. - **Observed — seam reachability:** | Kompress | no provider registered | provider registered | |---|---|---| | off | 19,284 → 10,265 tokens (46.8%) | 19,284 → **7,879 (59.1%)** | | on (remote) | 19,284 → 9,366 tokens (51.4%) | 19,284 → **7,146 (62.9%)** | Before this change the right-hand column was identical to the left — the registered provider was never called on this payload. - **Observed — marker-free default costs nothing:** | Config | tokens | saved | |---|---|---| | markers on (previous default) | 37,791 → 24,415 | 35.4% | | markers off (new default) | 37,791 → 24,415 | **35.4% — identical** | | `mode="lossy_inline"` | 37,791 → 25,129 | 33.5% | | `--lossless` | 37,791 → 35,100 | 7.1% | - **Observed — memo staleness, before the fix:** registering a provider that folds a grep block to 5 bytes left the block at its 1496-byte built-in fold, and clearing a provider kept serving the provider's output. Both correct after keying on the registration generation. - **Observed — context limit:** `bedrock/anthropic.claude-3-5-sonnet` resolves 200000 via the Anthropic provider, 128000 via the OpenAI provider. - **Not tested:** `tests/test_transforms_content_router.py` was not run — it does not complete on this machine, wedging on its 5th test while that test passes in 5.7s alone. Verified pre-existing before this work: with the diff stashed, the clean tree stalled at the identical test, and it stalled the same way under `HF_HUB_OFFLINE=1 HEADROOM_OFFLINE=1`. The machine was also out of disk at the time, which may be the real cause rather than the suspected native-detector deadlock (#575) — worth a separate issue either way. CI should be the arbiter here. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Two pre-existing test files needed adjusting, both direct consequences rather than scope creep: - `test_platform_stabilization_functional.py::test_v1_compress_success_reports_actual_metrics` patches `openai_pipeline.apply`, which the default mode no longer routes through. **This test was already failing on the marker-free-default commit** before any of the fixes — my original test selection missed it. It now patches the pipeline the route actually uses. - `test_proxy_eager_preload_bind.py` substitutes fake pipelines to control exactly what the preload walks; the eager build injected the real derived router's statuses into an exact-equality assertion. Its shared helper now clears the derived cache, preserving each test's intent without weakening an assertion. Docs unchecked — follow-ups worth doing in the same release: document `config.mode` values in `docs/content/docs/litellm.mdx` and `wiki/proxy.md`; the TS SDK `mode:"ccr"` default; and an operational note that `COMPRESSION_TIMEOUT_SECONDS` defaults to 30 (`helpers.py:687`) while a remote ML endpoint makes one sequential call per unit — on a large payload it trips the executor timeout and the handler fails open, returning `compression_skipped: true` with `tokens_before: 0`, which reads as "nothing to save" rather than "we gave up". Those zeroed counters are misleading and worth a separate fix. |
||
|
Tejas Chopra
|
e0ce4b1d48
|
fix: remove rtk and lean-ctx CLI context tools (#2677)
## Description Removes both third-party CLI context tools — **rtk** and **lean-ctx** — and with them the context-tool selector itself. Headroom no longer downloads, installs or configures either one, and there is no replacement. The previous pass (#2344) gated only three entry points inside `headroom/cli/wrap.py`. That left the feature reachable in practice: | Gap | Effect | |---|---| | `scripts/install.sh:1544`, `install.ps1:1681` | Ran `rtk init --global --auto-patch` from bash/PowerShell, **bypassing the Python gate entirely** — `curl \| sh` still wrote a Claude Code `PreToolUse` hook regardless of `HEADROOM_RTK` | | `wrap.py` `_setup_context_tool_for_agent` | **`wrap openhands` was broken by default**: `rtk_required=True` met a gate returning `None` → `SystemExit(1)`. Invisible because all 8 openhands tests patched `_ensure_rtk_binary` to a fake path | | `proxy/helpers.py`, `subscription/tracker.py` | Proxy shelled out to `rtk gain` from `/stats`, the dashboard and `headroom perf`; the tracker polled it per contribution (`_RTK_WIRING_DEFAULT = "enabled"`) | | No cleanup path | Nothing removed artifacts an earlier default had installed, so a machine that once ran the old default kept rtk in the loop forever (#1669, #1955) | Also worth noting: the rtk binary download had **no SHA or signature verification** — only `rtk --version` as a smoke test. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [x] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made **Removed** — `headroom/rtk/` and `headroom/lean_ctx/` packages, `headroom/cli/wrap_rtk_metrics.py`, `_selected_context_tool` / `_setup_context_tool_for_agent` / `_VALID_CONTEXT_TOOLS`, the `--rtk` / `--no-rtk` / `--no-project-rtk` / `--keep-rtk` flags across all 18 wrap subcommands, `HEADROOM_RTK*`, the proxy-side `rtk gain` polling, the dashboard CLI-filtering panel (rows + all 8 `cliFiltering*` Alpine getters), `paths.rtk_path()` / `lean_ctx_path()`, the SDK path helpers, `benchmarks/rtk_loop_learn_eval.py`, and the `headroom/rtk/**` CI path filters. **Fails loudly, not silently** — `--context-tool` / `--no-context-tool` / `HEADROOM_CONTEXT_TOOL` are kept solely to error out. They live in shell profiles, aliases and CI jobs, and accepting them as a no-op would read as Headroom having quietly stopped working. The installers reject them too, which matters more than it looks: their arg parsers forward the first unknown flag **and everything after it** to the wrapped tool, so a leftover `--no-rtk` would have silently swallowed a following `--port` and then been ignored downstream. **New `headroom/context_tool_cleanup.py`** — deleting the code cannot help a machine that already ran the old default, since the hooks, binaries and injected guidance are durable on disk. `purge_context_tool_artifacts()` runs once per `wrap`/`unwrap` and removes the registered hook entries, the generated hook scripts, the Headroom-managed `~/.local/bin` symlinks, the vendored `~/.headroom/bin/{rtk,lean-ctx}` binaries, the `lean-ctx` MCP server entry and the marker-fenced instruction blocks. Deliberately conservative: idempotent, **skips** a malformed config rather than overwriting it, and only unlinks a symlink resolving inside Headroom's own bin dir so a user's own build is untouched. It reports on **stderr**, because `wrap/unwrap openclaw --prepare-only` emit machine-readable JSON on stdout as their entire contract. Skipped for `wrap selfheal` (runs from a SessionStart hook; must not race Claude Code's writer for `~/.claude.json`) and for `--help`, which must stay read-only. **Client-config hardening** (discovered while investigating a "corrupted Serena settings file" report) — `wrap.py` reset a settings file to `{}` when an existing file would not parse, then wrote that back. One hand-edited typo or a transient `EACCES`/`EINTR` on a valid file destroyed the user's `permissions`, `env` and `hooks`, on **every `headroom wrap claude`**. It now refuses to write. Separately, `fsutil.write_text` is now atomic (temp file + `fsync` + `os.replace`), fixing all 14 non-atomic client-config writes at once; it follows symlinks rather than replacing them (dotfile managers) and preserves an existing file's mode. **Deliberately kept** — `rtk` stays in the wrapper-peel list in `transforms/content_router.py`. It sits beside `sudo`/`env`/`timeout` as shell-command grammar, so `rtk cat f` is still classified as a file read for anyone running their own rtk install, which the purge intentionally leaves alone. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check headroom/ tests/ e2e/ --exclude headroom/dashboard/templates All checks passed! $ ruff format --check headroom/ tests/ e2e/ --exclude headroom/dashboard/templates 1255 files already formatted $ mypy headroom/ Success: no issues found in 508 source files $ pytest tests/test_context_tool_cleanup.py -q 11 passed $ pytest tests/test_fsutil.py -q 12 passed $ pytest tests/test_cli/test_wrap_codex.py -q # 89 tests 89 passed in 431.68s $ pytest tests/test_cli/test_wrap_opencode.py -q 39 passed in 257.46s $ pytest tests/test_cli/test_wrap_helpers.py -q 45 passed $ pytest tests/test_paths.py -q 75 passed $ pytest tests/test_cli/test_unwrap_claude.py -q 14 passed $ pytest tests/test_proxy_savings_history.py -q 39 passed $ pytest tests/test_cli/test_wrap_copilot.py -q 27 passed $ pytest tests/test_cli/test_wrap_zcode.py -q 20 passed $ pytest tests/test_subscription_tracker.py -q 9 passed $ pytest tests/test_proxy_dashboard_stats_cache.py -q 5 passed, 1 skipped ``` Repo-wide grep for 14 removed symbols (`headroom.rtk`, `headroom.lean_ctx`, `_ensure_rtk_binary`, `_selected_context_tool`, `_get_context_tool_stats`, `rtk_path`, `lean_ctx_path`, `wrap_rtk_metrics`, `HEADROOM_RTK`, `cli_tokens_avoided`, `tokens_saved_rtk`, …) across `*.py`, `*.ts`, `*.sh`, `*.ps1`, `*.yml`, `*.html`: **zero hits**. Notable test changes: `test_wrap_openhands.py` no longer patches `_ensure_rtk_binary` and asserts `wrap openhands --prepare-only` exits 0 unpatched — the regression that was previously masked. `test_wrap_continue.py` and `test_wrap_hintfile_agents.py` were removed (every test drove RTK instruction injection). A new `test_subscription_tracker.py::test_load_state_written_before_cli_context_tools_were_removed` proves a pre-removal `subscription_state.json` still loads. ## Real Behavior Proof - **Environment:** macOS 15.4 (darwin 25.4.0), Python 3.12.6, Headroom @ this branch, real `~/.headroom` and `~/.claude` on the dev machine. - **Exact command / steps and observed result:** ```text # 1. Retired flag fails loudly instead of silently no-op'ing $ headroom wrap codex --prepare-only --context-tool rtk Error: CLI context tools (rtk, lean-ctx) have been removed from Headroom: they rewrote shell commands through a third-party binary Headroom no longer manages. Drop --context-tool / --no-context-tool and unset HEADROOM_CONTEXT_TOOL; `headroom wrap` uninstalls what they left behind on first run. $ HEADROOM_CONTEXT_TOOL=lean-ctx headroom wrap codex --prepare-only Error: CLI context tools (rtk, lean-ctx) have been removed from Headroom: ... # 2. install.sh rejects the retired flags (extracted parse_wrap_args harness) ['--no-rtk', '--port', '9999'] rc=1 ERROR: CLI context tools ... Drop --no-rtk ['--context-tool=rtk'] rc=1 ERROR: CLI context tools ... Drop --context-tool $ bash -n scripts/install.sh # syntax OK # 3. Purge ran against the real machine, which had all the orphaned artifacts $ python -c "from headroom.context_tool_cleanup import purge_context_tool_artifacts; ..." removed ~/.headroom/bin/lean-ctx (51 MB) removed ~/.headroom/bin/rtk (7.7 MB) removed ~/.local/bin/rtk (symlink into ~/.headroom/bin) removed ~/.claude/hooks/rtk-rewrite.sh removed 8 lean-ctx-* hook scripts # ~/.claude.json afterwards: 90 top-level keys, 19 projects, mcpServers unchanged # → ~59 MB reclaimed, no unrelated key touched # 4. stdout stays machine-readable while the purge reports (planted a fake artifact) $ headroom wrap openclaw --prepare-only --gateway-provider-id codex >out 2>err $ cat out {"enabled":true,"config":{"proxyPort":8787,...}} # parses as JSON $ cat err Retired CLI context tool cleanup: removed /Users/tcms/.headroom/bin/rtk # 5. --help is inert (planted artifact survives), a real run purges $ headroom wrap codex --help → artifact survived: CORRECT $ headroom wrap openclaw --prepare-only → purged: CORRECT # 6. MCP purge dry-run against a copy of the real 82 KB ~/.claude.json top-level keys 90 -> 90; projects 19 -> 19; LOST keys: none all content outside mcpServers byte-identical: True ``` Dashboard rendered via the Playwright test after the panel removal: "Token Savings" shows only `Proxy 0 (0.0%)` / `Of total wire: 36.86%`, and "Token Usage" reads Before Compression → Proxy Removed → After Compression with no "Filtered (this session)" row. Nothing below the removed panel broke. - **Not tested:** Windows and Linux (macOS only) — `install.ps1` is verified by brace-balance and inspection, not executed, since no `pwsh` is available locally. The wrap e2e suite (`e2e/wrap/run.py`) was updated but not run; it needs the Docker e2e image. `serena project index` interaction is exercised in the stacked base PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes **Stacked on #2676** (`tejas/serena-config-bootstrap`) — please merge that first; this PR's base should then be retargeted to `main`, or it will read as containing that fix too. **Breaking-change migration for users:** - Drop `--rtk`, `--no-rtk`, `--no-project-rtk`, `--keep-rtk`, `--context-tool`, `--no-context-tool` from any alias, script or CI job, and unset `HEADROOM_RTK*` / `HEADROOM_CONTEXT_TOOL`. They now error rather than being ignored, so the failure is immediate and self-explaining. - Previously-installed artifacts are purged automatically on the next `wrap`/`unwrap`; no manual cleanup needed. - `headroom perf --json` no longer carries a `cli_filtering` key, and `/stats` no longer returns a `context_tool` section. **Docs:** `docs/rtk-architecture.md` deleted; RTK/lean-ctx removed from `README.md`, `docs/content/docs/{configuration,opencode,grok-build,docker-install,filesystem-contract}.mdx`, `docs/observability.md` and the matching `wiki/` pages. `REALIGNMENT/09-phase-G-rtk-observability.md` is marked SUPERSEDED rather than deleted, to keep the planning record. **Follow-ups not in scope:** `_emit_wrap_interrupted` was deleted as dead code — its only caller was the `except KeyboardInterrupt` guarding the binary download, so with no download there is nothing slow left to interrupt. |
||
|
Tejas Chopra
|
759209cff3
|
fix(wrap/serena): stop creating serena_config.yml, unbricking Serena on fresh installs (#2676)
## Description
`_ensure_serena_dashboard_disabled()` wrote a one-key bootstrap config
(`web_dashboard_open_on_launch: false`) into
`~/.serena/serena_config.yml` when the file was absent, assuming Serena
fills in any key it omits.
Verified against **Serena 1.6.2.dev0**
(`serena/config/serena_config.py`), that holds for every field except
one. Serena autogenerates its own complete config **only when the path
does not exist**:
```python
if not os.path.exists(config_file_path):
cls._generate_config_file(config_file_path)
```
Once any file is present it validates instead. Every other field falls
back to a dataclass default via `get_value_or_default`, but a missing
`projects` key is fatal (~line 1064):
```
SerenaConfigError: `projects` key not found in Serena configuration.
```
So Headroom's own bootstrap file killed Serena on **every machine
without a pre-existing Serena config**. The MCP server exited during
handshake — surfacing as `connection closed: initialize response` on
Codex and a bare `MCP error -32000: Connection closed` on OpenCode
(#2674) — and `serena project index` failed identically.
Headroom now leaves that file to Serena. That is immune to Serena adding
required keys later; guessing the schema is what caused the outage. The
popup never needed the file anyway: `build_serena_spec` passes
`--open-web-dashboard False`, which Serena applies *after* loading the
config (`serena/mcp.py:361` — `config.web_dashboard_open_on_launch =
open_web_dashboard`), so the flag wins regardless of what is on disk.
An **existing** config is still edited in place — dashboard key flipped,
`projects: []` backfilled to repair machines an affected version already
wrote — preserving a populated `projects` list, other keys and comments.
Closes #2674
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)
## Changes Made
- `_ensure_serena_dashboard_disabled()` never creates
`serena_config.yml`; it only edits an existing one, and backfills
`projects: []` there to repair already-broken machines.
- Dropped `_scope_serena_languages` + `_detect_repo_languages` +
`_EXT_TO_SERENA_LANGUAGE` (**−133 lines**). Dead weight: Serena
determines languages itself in `ProjectConfig.autogenerate`
(`_determine_project_language_servers`) and records them under
`language_servers` — `languages`, which Headroom wrote, is a legacy name
Serena migrates via `RENAMED_FIELDS`. Serena's generated file uses a
block-style list, so our single-line-flow regex never matched it: on any
Serena-generated `project.yml` the function was a **verified no-op**.
The only case where it acted was creating the file — the same
partial-config trap — which also skipped the `project.local.yml` sidecar
Serena writes alongside.
- **Test isolation:** the MCP install ledger defaults to
`~/.headroom/mcp_installs.json`, so any test registering a server wrote
into the developer's real ledger (observed adding a live `claude/serena`
entry during a local run). `conftest.py` now redirects it per-test.
- **Repo config:** `.serena/project.yml` carried a stale `project_name`
(`"feature-opencode-wrap"`) and listed only `typescript`, so Serena's
symbol index skipped 1331 Python and 194 Rust files for every
contributor.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_wrap_code_memory.py tests/test_cli/test_wrap_serena_boost.py \
tests/test_cli/test_serena_migrate.py tests/test_cli/test_serena_disable.py -q
35 passed, 1 skipped in 0.84s
$ SERENA_SRC=<serena checkout> pytest tests/test_wrap_code_memory.py -q
13 passed in 0.62s # the skipped test runs when a Serena source tree is available
$ ruff check headroom/ tests/ --exclude headroom/dashboard/templates
All checks passed!
$ mypy headroom/cli/wrap.py
Success: no issues found in 1 source file
```
New tests. The key one asserts the invariant rather than our own key
list, so it stays correct even if Serena adds a required key — a test
pinning `projects: []` would keep passing while users broke again:
- `test_serena_config_is_never_created_by_headroom` — Headroom must not
pre-empt Serena's bootstrap
- `test_serena_dashboard_disabled_repairs_config_missing_projects` —
heals a config an affected version wrote
- `test_serena_dashboard_disabled_preserves_registered_projects` — never
clobbers the real registry; comments kept, no duplicate key
- `test_serena_dashboard_disabled_is_idempotent`
- `test_serena_config_required_keys_match_serena_source` — reads
Serena's real source and pins the two facts this fix rests on
(bootstrap-only-when-absent, `projects` is the sole fatal omission).
Skipped unless `SERENA_SRC` is set; deliberately **not** named
`HEADROOM_*` because `conftest.py` scrubs that namespace, which would
make it silently always-skip.
## Real Behavior Proof
- **Environment:** macOS 15.4 (darwin 25.4.0), Python 3.12.6, Serena
1.6.2.dev0 via `uvx --from git+https://github.com/oraios/serena`, Codex
CLI 0.146.0.
- **Exact command / steps:** a probe doing a real JSON-RPC `initialize`
handshake against the exact command `headroom wrap` registers — i.e.
what Codex/OpenCode actually do — in a throwaway `HOME` per case. (A)
pre-seeded with the one-line config an affected version wrote; (B) no
config; (C) real `headroom wrap codex --prepare-only`, then handshake.
- **Observed result:**
```text
=== A. BROKEN: single-key config (Headroom 0.33.0) ===
MCP handshake: FAIL — no initialize response (exit=1). stderr tail:
File ".../serena/config/serena_config.py", line 1064, in from_config_file
raise SerenaConfigError("`projects` key not found in Serena configuration. ...")
serena.config.serena_config.SerenaConfigError: `projects` key not found ...
config after run: 1 lines, has 'projects': False
=== B. FIXED: no config, Serena bootstraps it ===
MCP handshake: PASS — initialize OK — serverInfo.name='Serena'
config after run: 213 lines, has 'projects': True
=== C. FULL FLOW: real `headroom wrap codex` then handshake ===
Serena: no serena_config.yml yet — letting Serena generate it
Serena MCP: registered (restart OpenAI Codex CLI if it was already running)
Serena: project pre-indexed (symbol cache warmed)
serena_config.yml: 213 lines, written by Serena (correct)
MCP handshake: PASS — initialize OK — serverInfo.name='Serena'
--- verdict ---
A (broken config) started: False <- expected False
B (fixed, no config) started: True <- expected True
C (after real wrap) started: True <- expected True
```
A second `wrap` in the same HOME flips the dashboard without damage:
`true` → `false`, `projects` intact, all 153 comment lines intact. The
writer was isolated against a pristine 213-line Serena config: **delta 0
newlines**.
- **Not tested:** Windows and Linux (macOS only); Serena versions other
than 1.6.2.dev0; the JetBrains language backend. The probe needs network
+ `uvx` (~2 min) so it is a manual verification tool, not wired into CI.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
- **Docs:** N/A — no user-facing docs described the `serena_config.yml`
bootstrap or the language scoping.
- Also fixes the OpenCode report (#2674). The Codex-side report of the
same root cause quotes the `SerenaConfigError` verbatim; OpenCode only
surfaces the generic `-32000`, which is why it read as two different
bugs.
- Users already broken by an affected version are repaired automatically
on their next `headroom wrap` — no manual `serena_config.yml` edit
needed.
- A stacked PR removing the rtk/lean-ctx CLI context tools is based on
this branch; this one is deliberately small so it can land first.
|
||
|
Tejas Chopra
|
5383c6bf2f
|
fix(release): sync generated version metadata on the release branch (#2659)
## Description
The 0.33.0 release PR (#2339) has sat in `changes-requested` since
2026-07-17. Root cause: **release-please only rewrites `pyproject.toml`
and its configured `extra-files`**, but other tracked files also carry
the version — and `server.json` is asserted byte-for-byte against
`render_server_json()`, which derives its version from `pyproject.toml`.
So the bump alone fails
`tests/test_mcp_registry/test_server_json.py::test_root_server_json_matches_builder`
(the `test (2)` shard) on every regenerated release PR.
Nothing in the repo regenerated `server.json` at all, so it fell behind
every release.
Unblocks #2339.
### Why the release *build* passes but the release PR does not
`release.yml` already runs `scripts/version-sync.py` immediately before
its own `verify-versions.py` gate (lines 145 and 278). That is why
`build` and `build-wheels` are green on #2339 despite the drift — it
syncs in the workspace, uncommitted. The regular CI test job does
**not** sync, so the fix has to be committed to the branch.
This also explains why reviewers kept seeing `verify-versions.py` fail
locally while CI's build jobs passed: the verifier is never run
un-synced inside `release.yml`.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- **`scripts/version-sync.py`**: also write `server.json`. It was the
one version-carrying file with no writer anywhere. Values are rewritten
in place so key order and formatting keep matching the builder's
byte-for-byte output (verified: the file is pure ASCII and round-trips
exactly through `json.dumps(..., indent=2) + "\n"`).
- **`.github/workflows/release-metadata-sync.yml`** (new): on a push to
`release-please--branches--**`, run version-sync → gate on
verify-versions → commit if changed.
- **Keyed off the branch push** because release-please force-regenerates
that branch on every merge to main. That is precisely what wiped the
hand-pushed metadata fixes on #2339 (`2a86c8ff`, `d5ea4dc5`) — a push
trigger re-heals after every regeneration instead of being lost.
- **Uses the same PAT as `release-please.yml`**: a `GITHUB_TOKEN` push
does not trigger workflows, so the release PR's checks would never
re-run against the synced commit and would stay red.
- **Idempotent**: the self-triggered rerun finds no diff and exits
before pushing, so the loop terminates after one no-op run.
- **Corrected pre-existing drift on `main`**: the agent-hooks plugin
manifests, both marketplace manifests, and `.releasemetadata` were
stranded at **0.31.0** — never bumped for 0.32.0 either.
`verify-versions.py` now passes on `main`.
### Why not more `extra-files` entries
That would need ~13 jsonpath entries restating what `version-sync.py`
already knows, and a jsonpath that fails to match **fails silently** —
the same class of failure this PR removes, discoverable only after a
real release PR regenerates. There is also no precedent for nested
jsonpath (`$.packages[0].version`, `$.metadata.version`) in the config
today; both existing entries are plain `$.version`. Running the script
keeps one source of truth, and files added to it later are covered with
no change here.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) — no `headroom/` sources
touched
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest scripts/tests/ tests/test_release_workflows.py tests/test_mcp_registry/ -q
207 passed in 2.69s
$ ruff check scripts/version-sync.py scripts/tests/test_version_sync.py tests/test_release_workflows.py
All checks passed!
$ ruff format --check <same>
3 files already formatted
$ actionlint .github/workflows/release-metadata-sync.yml
(clean)
```
New tests:
- `test_server_json_version_is_synchronized` — version-sync moves both
`server.json` version fields and preserves the other keys.
- `test_release_metadata_sync_runs_on_release_please_branch` — asserts
the trigger, the sync→verify→commit ordering, the no-op guard, and the
PAT.
- `test_version_sync_covers_every_file_the_verifier_gates` — guards
`version-sync.py` and `verify-versions.py` against drifting apart again,
which is the root cause here.
## Real Behavior Proof
- **Environment:** macOS (Darwin arm64), Python 3.12, repo venv.
- **Exact command / steps:** reproduced the CI failure locally by
simulating release-please's partial bump, then applying the fix.
**Reproducing the exact `test (2)` failure** — set `pyproject` to 0.33.0
while `server.json` stays at 0.32.0, as release-please leaves it:
```text
$ python -m pytest tests/test_mcp_registry/test_server_json.py -q
FAILED tests/test_mcp_registry/test_server_json.py::test_root_server_json_matches_builder
1 failed, 3 passed
```
**After `version-sync.py`:**
```text
$ python scripts/version-sync.py && python -m pytest tests/test_mcp_registry/test_server_json.py -q
4 passed
```
**Both gates green on a simulated 0.33.0 bump:**
```text
$ python scripts/version-sync.py --version 0.33.0
Version synchronized to 0.33.0
$ python scripts/verify-versions.py
All versions aligned at 0.33.0
$ python -m pytest tests/test_mcp_registry/test_server_json.py -q
4 passed
```
**Idempotency** (the property the workflow's loop-termination relies
on): re-running against an already-synced tree leaves `pyproject.toml`,
`server.json`, `openclaw`, and `sdk/typescript` untouched.
- **Not tested:** the workflow has not executed on a real release-please
branch regeneration — that can only be exercised once this is on `main`
and release-please next updates #2339. The PAT push path and the
self-trigger no-op are reasoned from `release-please.yml`'s existing
token comment and from local idempotency, not observed in CI.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
**Context on the v0.32.0 release failure, since it is easy to misread as
"images never build".** Every artifact built for v0.32.0 — all 5 wheel
platforms including Windows, all 16 Docker builds + 8 manifests +
`promote-latest`, npm, and GitHub Packages. Only `publish-pypi` failed
(PyPI attestations, already fixed by `
|
||
|
JD Davis
|
2dc7e4ab27
|
test: add fluent Headroom harness (#2650)
## Description Adds `headroom.testing`, a fluent, contractual test harness for building Headroom scenarios and suites that can be simulated locally, orchestrated, deployed through the proxy, and handed off to `headroom-bench` / `agent-evals` with bench-native manifests. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add `headroom.testing.Headroom` fluent scenario builder with provider/platform/configuration facets such as `WithBedrock`, `OnAppleSilicon`, `Configure`, `WithCompression`, `WithCCR`, `WithCache`, `WithPrefixFreeze`, `WithReadMaturation`, and `WithMemory`. - Add contractual coverage over the current `HeadroomConfig` and `ProxyConfig` dataclass surfaces, including full JSON-ready proxy deployment payloads. - Add no-key local simulations, scenario/suite orchestration, guarantee evaluation, deployment plans, and a local proxy lifecycle context manager. - Add `headroom-bench` handoff artifacts, including `agent_evals.models.RunManifest`-compatible JSON without taking a runtime dependency on `agent-evals`. - Add demonstration tests for providers, feature facets, manifests, suites, guarantees, deployment payloads, and the no-key simulation path. - Fix unversioned OTEL meter lookup typing so `mypy headroom` remains green on current `main`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text python -m ruff check . All checks passed! python -m mypy headroom headroom\proxy\server.py:1680: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] headroom\proxy\server.py:1691: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] Success: no issues found in 512 source files python -m pytest tests/test_cli/test_subprocess_utf8_encoding.py tests/test_testing_harness.py -q 24 passed, 1 warning in 4.68s ``` ## Real Behavior Proof - Environment: Windows, Python 3.13.13, branch `feat/headroom-test-harness` rebased on `headroomlabs-ai/main`. - Exact command / steps: built a `Headroom.WithOpenAI().WithCompression(mode="cache", kompress=False).Build()` scenario and entered `scenario.deploy_local(port=19192, timeout_s=20)`. - Observed result: proxy launched, `/readyz` succeeded, handle returned `http://127.0.0.1:19192`, `OPENAI_BASE_URL=http://127.0.0.1:19192/v1`, and context-manager teardown completed. - Exact command / steps: emitted `scenario.agent_evals_manifest(...).to_dict()` and validated it with the current cloned `headroom-bench` `agent_evals.models.RunManifest` pydantic model. - Observed result: validation succeeded with arms `a0_direct`, `a1_passthrough`, and `b_headroom` for provider `openai`. - Not tested: upstream-provider API calls requiring real OpenAI/Anthropic/Bedrock keys; phase-1 validation intentionally stays no-key/local. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A. ## Additional Notes The pytest warning shown above is the existing OpenAI pricing-data staleness warning from cost estimation. The harness does not call upstream providers during local simulation. |
||
|
Tejas Chopra
|
e0d2cd0c5a
|
fix(cache): preserve cache_control ttl when re-anchoring a breakpoint (#2651)
## Description
`normalize_message_cache_control` deliberately reuses the client's
marker verbatim so an explicit `cache_control.ttl` (e.g. `"1h"`)
survives breakpoint consolidation instead of silently downgrading to the
5-minute default (#2375).
Two other sites also strip a breakpoint and re-place it, and both
hardcoded a bare `{"type": "ephemeral"}` — undoing that guarantee.
A downgrade is invisible: the request still succeeds, and the cost shows
up later as a full prefix re-write on every idle gap past 5 minutes.
Measured over 10,409 local Claude Code API requests, cache writes are
**6.1% of raw input tokens but 44.8% of the price-weighted input bill**
(5m write 1.25x vs read 0.1x), and **89% of those write tokens are
re-writes of content cached one request earlier**. Honoring a 1h TTL
when the client asks for it is the cheapest thing we can do about that.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/read_maturation.py` — `relocate_cache_breakpoint`
now carries the stripped marker forward when re-anchoring before the
held-Read region. This is the one that mattered most: it runs **after**
`normalize_message_cache_control` in the Anthropic handler
(`anthropic.py:1747` vs `:1642`), so it had the final say — a 1h client
with read maturation enabled was being downgraded to 5m.
- `headroom/proxy/helpers.py` — `inject_tool_search_deferral` keeps the
dropped marker when moving the tools-array breakpoint off a now-deferred
tool onto the last resident real tool.
- Both fall back to a bare ephemeral only when the client sent no ttl,
and neither invents a breakpoint where none existed.
- `headroom/transforms/compression_policy.py` — comment only. Notes that
`CACHE_WRITE_MULTIPLIER` is hardcoded to the 5m tier (1.25x), so a
client already on 1h caching (2.0x) has its mutations gated with a ~40%
under-stated write penalty. Harmless while the net-cost gate stays
default-off (`HEADROOM_NET_COST_POLICY`); names the plumbing needed if
it is ever enabled.
Both changed code paths sit behind off-by-default flags
(`HEADROOM_READ_MATURATION`, `HEADROOM_TOOL_SEARCH`), so this is a
latent-bug fix with **no default behavior change**.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_cache_ttl_preserved.py tests/test_read_maturation.py \
tests/test_read_maturation_handler_nobust.py tests/test_cache_control_move_bust.py -q
tests/test_cache_ttl_preserved.py ..... [ 12%]
tests/test_read_maturation.py ...................... [ 67%]
tests/test_read_maturation_handler_nobust.py ... [ 75%]
tests/test_cache_control_move_bust.py .......... [100%]
============================= 40 passed in 15.52s ==============================
$ ruff check headroom/ tests/ --exclude headroom/dashboard/templates
All checks passed!
$ mypy headroom
Success: no issues found in 509 source files
```
Broader regression sweep over every cache/breakpoint-adjacent suite:
```text
$ python -m pytest tests/ -q -k "read_maturation or tool_search or cache_control or prefix_tracker or ttl_preserved"
204 passed, 10145 deselected in 59.95s
```
## Real Behavior Proof
- **Environment:** macOS 25.4.0 (arm64), Python 3.12.6, pytest 9.0.2,
branched from `main` at
|
||
|
Abhay Singh
|
22b707fd31
|
fix(proxy/cost): count Gemini thinking tokens in output usage (#2639)
## Description
The Gemini handlers take the response's output-token count straight from
`candidatesTokenCount`:
```python
output_tokens = _usage_int(usage.get("candidatesTokenCount"))
```
For Gemini 2.5 thinking models that undercounts. Gemini reports
`candidatesTokenCount` **sometimes inclusive** of the reasoning tokens
(`thoughtsTokenCount`) and **sometimes exclusive** of them. When it is
exclusive, the thinking tokens are a separate bucket that is still
billed at the output rate, so dropping them makes `output_tokens` (and
therefore the output cost that flows through `record_tokens` ->
`estimate_cost`) too low. The gap grows with reasoning effort.
litellm handles exactly this: it adds `thoughtsTokenCount` to completion
tokens unless `promptTokenCount + candidatesTokenCount ==
totalTokenCount` (its `is_candidate_token_count_inclusive` check). The
Headroom handlers had no equivalent.
## Fix
Add `gemini_output_tokens(usage_meta)` in
`headroom/proxy/token_counting.py`:
- No `thoughtsTokenCount` (the common non-2.5 case): return
`candidatesTokenCount` unchanged.
- `promptTokenCount + candidatesTokenCount == totalTokenCount`:
candidates already include thoughts, return `candidatesTokenCount`.
- Otherwise: return `candidatesTokenCount + thoughtsTokenCount`.
This mirrors litellm's rule and is robust to missing or null fields.
Wire it into the native Gemini handler (both the generate and count
paths), the streaming usage extractors, and the OpenAI-compatible
passthrough usage normalizer, so every Gemini usage path counts output
the same way.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring
## Changes Made
- `headroom/proxy/token_counting.py`: add `gemini_output_tokens()`.
- `headroom/proxy/handlers/gemini.py`: use it for `output_tokens` on
both response paths.
- `headroom/proxy/handlers/streaming.py`: use it in the two Gemini
streaming usage extractors.
- `headroom/proxy/handlers/openai.py`: use it in
`_passthrough_usage_from_json` (Gemini-shaped usage).
- `tests/test_proxy_handler_helpers.py`: unit test for
`gemini_output_tokens` (inclusive / exclusive / no-thinking / empty) and
a `_passthrough_usage_from_json` test that thinking tokens land in
`output_tokens`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check` / `ruff format --check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for the fix
- [ ] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_proxy_handler_helpers.py -k "gemini_output_tokens or thinking or vertex_usage_metadata" -q
3 passed
$ python -m pytest tests/test_proxy_gemini_native_integration.py tests/test_proxy/test_gemini_savings_profile.py tests/test_proxy_handler_helpers.py -q
38 passed, 18 skipped
# with the wiring reverted, the passthrough test fails (output_tokens is 200, not 700):
$ git stash push headroom/proxy/handlers/openai.py && \
python -m pytest tests/test_proxy_handler_helpers.py -k passthrough_usage_counts_gemini_thinking -q
1 failed
$ uvx ruff@0.15.17 check headroom/proxy/token_counting.py headroom/proxy/handlers/gemini.py headroom/proxy/handlers/streaming.py headroom/proxy/handlers/openai.py tests/test_proxy_handler_helpers.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/proxy/token_counting.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, project venv (`uv sync --extra
proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`, pytest in the venv.
- Exact command / steps: called `gemini_output_tokens` on an exclusive
usage (`prompt=1000, candidates=200, thoughts=500, total=1700`), an
inclusive usage (`candidates=700, total=1700`), a no-thinking usage, and
`{}`; drove `_passthrough_usage_from_json` with a thinking usage; then
reverted the handler wiring and re-ran the passthrough test.
- Observed result: exclusive returns 700 (200 visible plus 500
thinking), inclusive returns 700, no-thinking returns the candidates
count, empty returns 0; `_passthrough_usage_from_json` reports
`output_tokens=700`. With the wiring reverted it reports 200 (the
undercount). Verified against litellm's documented rule.
- Not tested: a live Gemini 2.5 request end to end (the accounting is
verified at the usage-extraction boundary against litellm's reference
logic).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
Fabien Culpo
|
1d29738818
|
fix(proxy): keep core tools and the client's ToolSearch resident for PascalCase clients (#2647)
## Description `_TOOL_SEARCH_CORE_TOOLS` is spelled in lowercase, but the membership test compared the raw tool name, so the core-tool exemption never fired for clients that send PascalCase names. For Claude Code (`Bash`, `Read`, `Edit`, `ToolSearch`) **every** tool in the request body was deferred. The damaging part is that Claude Code's own `ToolSearch` was deferred. It is the schema fetcher for tools the client keeps in its local registry and never sends in the body — `TaskCreate`, `TaskUpdate`, `TaskList`, `WebFetch`, `EnterPlanMode`, `Monitor`, `LSP`, `Cron*`, `SendMessage`. Hiding it makes all of them permanently uncallable: advertised to the model in a `<system-reminder>`, but no search can return their schemas, because the injected `tool_search_tool_regex` only indexes what is in the request body. Closes #2646 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Compare tool names against the core set case-insensitively in `inject_tool_search_deferral` (`helpers.py`). - Add `"toolsearch"` to `_TOOL_SEARCH_CORE_TOOLS` so a client's own schema-fetch tool is never deferred. - Apply the same case-insensitive comparison to `inject_tool_search_deferral_openai`, which had the identical exact-match bug (including against `_OPENAI_TOOL_SEARCH_RESIDENT_NAMES = {"terminal"}`). - Add 3 tests on the Anthropic path and 1 on the OpenAI path. Both source changes are required: case-folding alone does not help `ToolSearch` (it was not in the set), and adding it alone does not help `Bash`/`Read`/`Edit`. **The token saving is unchanged** — MCP tools are still deferred. This is not a request to disable the feature. Beyond the stranded tools, the old behaviour also meant (a) routine `Bash`/`Read`/`Edit` loops each paid a search round-trip, the exact cost the core set exists to avoid, and (b) zero resident *real* tools remained, silently violating the invariant documented on `inject_tool_search_deferral` — the injected search tool is typed and does not satisfy it — which risks an upstream 400. The existing assertion for that invariant passes today only because its fixture uses lowercase names. ## Testing - [x] Unit tests pass (`pytest`) — the two affected files; see scope note below - [ ] Linting passes (`ruff check .`) — see note - [ ] Type checking passes (`mypy headroom`) — could not run, see note - [x] New tests added for new functionality - [x] Manual testing performed `ruff check .` reports 4 findings repo-wide, **all pre-existing and unrelated** (`plugins/headroom-oauth2/`), confirmed identical on unmodified `main`. Zero findings in the three files this PR touches, and `ruff format --check` is clean on all three. Left unchecked because the repo-wide command does not exit 0. `mypy headroom` could not run in my environment (numpy stubs error out under the resolved Python version before checking begins). Not attempted further — CI should be the authority. ### Test Output ```text $ python -m pytest tests/test_issue_746_tool_search.py tests/test_openai_tool_search_deferral.py -q 65 passed, 1 warning in 0.70s # Baseline on those two files before this PR: 62 (36 + 26). # The 3 new Anthropic tests + 1 new OpenAI test bring it to 65. # Red before the source change (tests written first): tests/test_issue_746_tool_search.py::test_core_tools_match_case_insensitively FAILED AssertionError: Bash assert True is None where {'name': 'Bash', ..., 'defer_loading': True}.get('defer_loading') tests/test_issue_746_tool_search.py::test_client_tool_search_tool_is_never_deferred FAILED AssertionError: assert True is None where {'name': 'ToolSearch', ..., 'defer_loading': True}.get('defer_loading') tests/test_issue_746_tool_search.py::test_resident_real_tool_survives_pascal_case_surface FAILED assert any(not t.get("type") and not t.get("defer_loading") for t in out) assert False 3 failed, 36 deselected $ python -m ruff check headroom/proxy/helpers.py tests/test_issue_746_tool_search.py tests/test_openai_tool_search_deferral.py All checks passed! $ python -m ruff format --check <same three files> 3 files already formatted ``` ## Real Behavior Proof - Environment: headroom 0.32.1 installed / 0.32.0 source, Python 3.13, macOS 15 (Darwin 25.5.0), Claude Code 2.1.220 with `ENABLE_TOOL_SEARCH=true` and `ANTHROPIC_BASE_URL=http://localhost:8787`, first-party Anthropic upstream, `HEADROOM_TOOL_SEARCH` truthy - Exact command / steps: build a Claude Code tool surface and pass it through the injector — `names = ["Bash","Read","Write","Edit","Glob","Grep","ToolSearch"] + [f"mcp__srv__t{i}" for i in range(12)]`, `tools = [{"name": n, "description": n, "input_schema": {}} for n in names]`, then `inject_tool_search_deferral(tools)` and print which entries carry `defer_loading` - Observed result: before the fix `resident real tools: []` with `ToolSearch deferred: True` (every built-in deferred). After the fix `resident real tools: ['Bash','Edit','Glob','Grep','Read','ToolSearch','Write']` with all 12 `mcp__srv__t*` still deferred, so the saving is retained. This matches a live session: the proxy logged `router:tool_search_deferral:25tools:22182tok ... client=claude-code` and `tool_search_tool_regex` could resolve only `mcp__*` tools — `TaskCreate`/`WebFetch`/`EnterPlanMode` returned no match until `ToolSearch` was recovered by regex-searching for it and then calling `select:TaskCreate,...` - Not tested: the full pytest suite (164 modules fail collection with `ModuleNotFoundError: No module named 'headroom._core'` because my environment imports the package via `PYTHONPATH` without building the Rust extension; identical failure confirmed on unmodified `main`, so it is environmental). `mypy headroom` not runnable here. No end-to-end run against a live upstream through a rebuilt proxy — verification is at the function boundary plus the live-session log evidence above. The OpenAI Responses path is covered by unit test only, not exercised against a real gpt-5.4+ deployment. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes - **Documentation**: N/A — no user-facing surface changes; behaviour returns to what the existing comments and docstring already describe. - **"New and existing unit tests pass locally"**: left unchecked deliberately. The tests covering the changed symbols pass (65), but I cannot run the whole suite locally without the compiled `headroom._core`. Not claiming more than I verified. - **Scope**: the OpenAI-path fix rides along because it is the identical three-line comparison bug in the sibling function. Happy to split it into its own PR if you would rather keep this Anthropic-only. - **Deliberately not done**: I did not add a `client != "claude-code"` gate at `handlers/anthropic.py`, even though the feature's own comment block scopes it to non-Claude-Code clients and `client=claude-code` is already known there (it appears in the `transforms=` log line). Gating there would forfeit the ~22k tokens/request currently saved on Claude Code's eagerly-shipped MCP schemas; keeping the meta-tool resident preserves both the saving and reachability. Flagging in case you would prefer to gate as well. - **Adjacent blind spot, out of scope**: `claude_code_tool_search_inactive` already checks both the tools array *and* the `anthropic-beta` header, but the injector's early-return guard checks only the array. That is why a plain-function `ToolSearch` slips past it and the injection runs on a client that is already deferring. Co-authored-by: Fabien Culpo <fabien.culpo@dawex.com> |
||
|
Devanshi Vyas
|
1588f5e041
|
feat: expose configured OTEL meters to integrations (#2519)
## Description
Expose a small public observability API that lets optional integrations
create
OpenTelemetry instruments using Headroom's configured meter provider.
Without this API, an integration must either rely on observability
internals or
create a second provider and exporter. `get_otel_meter(name, version)`
keeps
configuration, export, and shutdown ownership inside Headroom while
allowing
integration-specific instruments to use their own instrumentation scope.
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Add `HeadroomOtelMetrics.get_meter(name, version)` to obtain a meter
from the
provider already owned by the Headroom metrics facade.
- Add and publicly export `headroom.observability.get_otel_meter(...)`.
- Preserve no-op-compatible OpenTelemetry behavior when Headroom-managed
metric
export is not configured.
- Add a focused test proving integration instruments are collected by
the same
configured provider.
- Add no dependencies and make no changes to existing metrics or
configuration.
## Testing
<!-- Check what you actually ran, then paste the real command output
below. -->
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ uv run pytest tests/test_observability_metrics.py -q
collected 6 items
tests/test_observability_metrics.py ...... [100%]
6 passed in 4.72s
$ uv run ruff check .
All checks passed!
$ uv run ruff format --check .
1331 files already formatted
```
## Real Behavior Proof
- Environment: macOS (Darwin), Python 3.12.13, Headroom `0.33.0-dev`,
OpenTelemetry SDK `1.39.1`, console metric exporter.
- Exact command / steps: Run the command below:
```bash
uv run python -c 'from headroom.observability import OTelMetricsConfig,
configure_otel_metrics, get_otel_meter, shutdown_otel_metrics;
configure_otel_metrics(OTelMetricsConfig(enabled=True,
exporter="console", service_name="headroom-integration-proof",
export_interval_millis=60000)); get_otel_meter("example.integration",
"1.0.0").create_counter("example.integration.events").add(3, {"source":
"extension-api"}); shutdown_otel_metrics()'
```
- Observed result: Headroom's console exporter emitted
`example.integration.events` with value `3`, attribute
`source="extension-api"`, instrumentation scope `example.integration`
version `1.0.0`, and resource service name `headroom-integration-proof`.
This demonstrates that the public accessor participates in Headroom's
configured provider and shutdown lifecycle.
- Not tested: network OTLP export, the complete repository test suite,
`mypy`,
or Python versions other than 3.12 in this final validation.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A — this change has no user-interface surface.
|
||
|
Tejas Chopra
|
57bf720d5c
|
feat(router): route embedded & nested JSON through the compressor dispatch (#2623)
## Description
`ContentRouter` only compressed JSON when the **whole** `tool_result`
block was a single JSON value. JSON embedded inside larger output (`gh
api` dumps, MCP tool results, `curl | jq` tails, log lines ending in a
JSON blob) was invisible to the JSON compressors — and in practice that
embedded shape is the large majority of JSON an agent actually sees.
This adds a structural routing step: find balanced JSON spans at **any
offset** in a block and route each one through the router's **existing,
unchanged** `_apply_strategy_to_content`, splicing the result back with
the surrounding bytes kept exact.
Because each span takes the same dispatch path a whole-block JSON
already takes, SmartCrusher/CodeCompressor register their `<<ccr:…>>`
retrieval markers exactly as before — CCR is hash-keyed, so it is
location-agnostic and unaffected by nesting.
Closes #
## Type of Change
- [x] New feature (non-breaking change that adds functionality)
## Changes Made
- New `headroom/transforms/recursive_json.py` — `route_embedded_json()`:
deterministic balanced-span scan + splice; skips spans already carrying
a `<<ccr:` marker (never re-compresses); token-gated.
- One guarded call at the top of `_apply_strategy_to_content` plus an
`_allow_embedded` **one-shot re-entrancy guard** (not a depth cap).
- **No size/min or depth thresholds** — the only gates are correctness
(round-trip) and benefit (token reduction). Strict no-op when a block
has no embedded JSON, so the 97%+ of non-JSON blocks are byte-identical
to today.
- Deterministic + per-block → prefix-cache- and CCR-store-stable.
- `tests/test_recursive_json.py`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_recursive_json.py tests/test_content_router_compact_json.py tests/test_content_router_tool_role_reversibility.py -q
19 passed in 5.90s
$ ruff check headroom/transforms/recursive_json.py headroom/transforms/content_router.py tests/test_recursive_json.py
All checks passed!
$ mypy headroom/transforms/recursive_json.py headroom/transforms/content_router.py
Success: no issues found in 2 source files
```
## Real Behavior Proof
- **Environment:** local,
`ContentRouter(ContentRouterConfig(lossless=False))`, pure-Python
content detector.
- **Exact steps:** `router.compress(block)` where `block` = prose with a
120-row JSON array embedded mid-text (`"I queried the ECS API
...\n[{...}]\nAll services healthy."`).
- **Observed result:** `strategy_used=MIXED`; block **11,986 → 3,798
chars**; leading/trailing prose preserved byte-exact; the embedded JSON
folded to a columnar table. Previously this block's embedded array was
not routed to the JSON compressor at all.
- **CCR:** a span already containing `<<ccr:` is passed through
untouched (unit-tested); folded spans go through the unchanged dispatch,
so markers register and resolve identically to whole-block JSON.
- **Not tested:** live proxy end-to-end with markers force-enabled
(covered by the unchanged dispatch path + unit tests); non-CC transcript
shapes beyond the local corpus.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
Scoped to the OSS structural-routing step only. The lossless *fold
kinds* it routes into (JSON columnar / log template) are maintained in
the `headroom-lossless-guard` extension. N/A: no docs/screenshots;
`Closes #` left blank (no tracking issue).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
Ruben A.
|
e530de5ad2
|
feat(rust): port CodeCompressor AST compressor to Rust (parity-only) (#1154)
Adds crates/headroom-core/src/transforms/code_compressor.rs (1,882 lines): the AST-aware CodeCompressor ported to Rust on tree-sitter, with grammars for Python, JavaScript, TypeScript, Go, Rust, Java, C and C++. Parity-only, like #1153. Nothing calls it: the only references outside the module are the pub mod / pub use declarations in transforms/mod.rs, and live_zone.rs still routes SourceCode to a no-op. The pyo3 bridge is untouched and no Python source changes, so the engine is unreachable from the shipped package. #1155 wires it into live-zone dispatch. Every grammar is pinned with '=' to the exact version of the corresponding Python tree-sitter-<lang> PyPI wheel. Same version on crates.io and PyPI means the same grammar.js, hence the same generated parser.c, hence node-for-node identical ASTs — the precondition for byte-parity. A canary over 9 samples x 8 languages confirmed identical node-type and line-span trees at these pins; bumping any pin requires re-running it and re-recording the fixtures. Ships 30 recorded parity fixtures, a CodeCompressorComparator in headroom-parity, and scripts/record_code_compressor_fixtures.py. Verified byte-identical to the recorded Python output: [code_aware_compressor] total=30 matched=30 skipped=0 diffed=0 Full harness on the merge result: 227 fixtures, 182 matched, 45 skipped (cache_aligner + ccr stubs), 0 diffed, exit 0 — with kompress at 21/21 under ONNX Runtime 1.24.4 (see #2591). Also verified cargo check -p headroom-core --no-default-features passes, so the static-musl path stays intact. |
||
|
Ruben A.
|
83e27e5036
|
feat(rust): port Kompress ML prose compressor to Rust (parity-only) (#1153)
Adds crates/headroom-core/src/transforms/kompress.rs (672 lines): the Kompress ML prose compressor ported to Rust, running the ModernBERT tokenizer plus the kompress-v2-base ONNX model through ort with a cache-only loader that never touches the network. Parity-only. Nothing calls it: the only references outside the module are the pub mod / pub use declarations in transforms/mod.rs. live_zone.rs still carries TODO(PR-B4), so PlainText dispatch remains a no-op and Python continues to serve prose compression. The pyo3 bridge is untouched and no Python source changes, so the new engine is unreachable from the shipped package. #1155 wires it up. Ships 21 recorded parity fixtures, a KompressComparator in headroom-parity, and scripts/record_kompress_fixtures.py. Verified byte-identical to the recorded Python output: [kompress] total=21 matched=21 skipped=0 diffed=0 That required ONNX Runtime >= 1.24 (see #2591) — below it ort deadlocks instead of erroring, which is why these fixtures had never been run. In CI the model is absent from the HF cache, so the comparator errors and the fixtures report Skipped rather than hanging. Also gates the module behind the ml feature, matching magika_detector: kompress.rs uses ort, which is optional = true, so an unconditional pub mod broke cargo check --no-default-features (the static-musl path). CI does not catch that class of break because cargo test --workspace only builds default features. |