mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
5 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Tejas Chopra
|
ad56dd382b
|
fix(router): compare token quantities in one unit (#2759)
## Description Two places compared a token quantity against something measured in a **different unit**. Both changed compression **behaviour**, not just reporting — which is the worse class. ### 1. The CONFIG branch put a word count in a token ratio `compressed_tokens = len(compressed.split())` was divided by `original_tokens`, which comes from `_estimate_tokens(content)`. Words run ~2.8× fewer than estimator tokens on config text, so a compressor that returned its input **byte-identically** scored ~0.36. `min_ratio` is 1.0 — accept any real shrink — so the router **accepted the no-op**: cached the result, pinned a frozen "compress" verdict, emitted a `router:config_compressor` label into `transforms_applied`, and recorded a fabricated saving to TOIN. ```text mkdocs.yml, compressor returns its input unchanged denominator (_estimate_tokens) = 936 OLD numerator len(split()) = 334 -> ratio 0.357 claims 64% saved ACCEPTED NEW numerator (_estimate_tokens) = 936 -> ratio 1.000 correctly rejected ``` The sibling TABULAR branch already used `_estimate_tokens`; CONFIG was the outlier. ### 2. The Kompress size gate tested a token cap in chars/4 `len(text_to_compress) > self._kompress_max_tokens * 4` under-counts anything denser than 4 chars/token, and compact JSON runs ~3.2. Against the 50,000-token default there is a band where an oversized payload passes: ```text records chars old_gate(len/4) new_gate(tokens) 2700 119,281 False False 4000 177,781 False True <- 44,445 vs 55,557 tokens, 11% over 4400 195,781 False True <- 48,945 vs 61,182 tokens, 22% over 5000 222,781 True True ``` Those payloads entered ONNX inference — exactly the >30s non-preemptible worker stall the gate exists to prevent (#1171). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `content_router.py` CONFIG branch — `compressed_tokens` now from `_estimate_tokens(compressed)`, matching its denominator and every sibling branch. - `content_router.py` Kompress gate — compared with `_estimate_tokens`, the unit the cap is actually expressed in. The extra O(n) char scan is negligible against the inference it guards. ## Testing - [x] Unit tests pass - [x] Linting passes (`ruff check` + `format --check`, pinned 0.15.17) - [x] New tests added - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_content_router_token_units.py -q 4 passed in 0.33s $ uvx ruff@0.15.17 check headroom/transforms/content_router.py tests/test_content_router_token_units.py All checks passed! ``` **Regression check against clean `upstream/main` in the same environment:** ```text tests/test_transforms/ + test_transforms_content_router.py + kompress suites upstream/main : 2 failed, 468 passed, 78 skipped this branch : 2 failed, 468 passed, 78 skipped failure sets : identical ``` The 2 failures are `test_kompress_failsafe`'s artifact-selection tests, which need a real `onnxruntime` this throwaway env lacks. Unrelated to this change. The 4 new tests pin the unit contract *and* the bounds of the disagreement band — including the cases where both formulations agree, so the band is demonstrated rather than assumed. ## Real Behavior Proof - **Environment:** macOS 26.4 arm64, isolated worktree off `upstream/main`. `content_router` needs the compiled `headroom._core`, which isn't in a fresh worktree (gitignored, built in-place); I copied the built `.so` in to run these, then removed it before committing. - **Observed:** both tables above are from running the real `_estimate_tokens` against the real thresholds, not reconstructed arithmetic. - **Not tested:** no live ONNX inference — the >30s stall the gate prevents is cited from #1171, not reproduced. The CONFIG no-op was demonstrated at the ratio level rather than by driving a stubbed compressor through `apply()`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] I did **not** edit `CHANGELOG.md` ## Related Third of three PRs from one tokenizer-consistency audit — see #2757 (litellm total-prompt / `--budget`) and #2758 (HuggingFace chat templates, `gpt-5`, gateway-wrapped names). Separate subsystems, separate risk. Known remaining from the same audit, not in any of the three: `_netcost_message_tokens` pricing an image by Python `repr` (34× over-count, flag-gated), three transforms reporting via `count_text(str(content))` where the pipeline uses `count_messages` (19% apart in one log file), `frozen_message_count` walking a chars/3.5 estimate against provider-reported cached tokens, and `target_ratio` honoured in words while documented as tokens. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Tejas Chopra
|
446ec26003
|
feat(transforms): dispatch kompress/text via the compressor registry + forward question (#2411)
## What
Completes the if/elif → registry migration in the content router:
**KOMPRESS and TEXT** now dispatch through the `kompress` built-in
adapter (`_registry_compress`), like every other strategy. Also **fixes
a latent bug** in `_invoke_kompress` that dropped the QA-aware
`question` argument (hardcoded `None`) — `question` now rides
`CompressInput.config['question']` and is forwarded into
`_try_ml_compressor`, so QA-aware compression content is preserved.
## Intentionally NOT byte-identical (one approved change)
The sole behavior change is the KOMPRESS/TEXT **token metric**: reported
`compressed_tokens` is now `_estimate_tokens(output.content)` — the
router's calibrated estimate, consistent with `original_tokens` and
every other registry-dispatched strategy — instead of the Kompress
model's own tuple count. **Compressed content is preserved byte-for-byte
in all paths.**
## Decision-impact analysis (traced every reader of `compressed_tokens`)
No content, routing, keep/drop, fallback, or lossless-then-lossy
decision reads this metric for KOMPRESS/TEXT: they're not in
`fallback_eligible_strategy` nor `{SEARCH,LOG,HTML}`, and the
STAGE-0/general layering calls `_try_ml_compressor` directly (unchanged,
already forwards `question`). The only downstream value-reader is
`_record_to_toin`'s skip gate (`original_tokens <= compressed_tokens`) —
**telemetry/learning only**, never affects returned content or routing,
and arguably more correct now (both sides on the same `_estimate_tokens`
scale). Consciously accepted.
## Tests
Rewrote the PR-C2 deferral-pinning tests →
registry-dispatch-matches-direct (content matches the direct
`_try_ml_compressor(..., question)` call; token assertion switched
`==<model count>` → `==_estimate_tokens(output)`, the only assertion
change, solely due to the approved metric switch). Added a
QA-differential test (question changes content) + an adapter-level
`question`-forwarding test. Offline suite: 96 passed; ruff 0.15.17 +
mypy clean.
**Note:** the full content-router CI suite may require further test
updates for any test that exercises the real KOMPRESS/TEXT branch and
asserts the returned count equals the model's tuple `compressed_tokens`
— those should switch to `_estimate_tokens(output)`. (The broad
content_router/compression selection wasn't run locally — it needs
ONNX/HF.)
After this, the router's per-strategy dispatch is fully
registry-resolved.
|
||
|
Tejas Chopra
|
7c7bf43057
|
feat(transforms): dispatch smart_crusher via the compressor registry (defer kompress/text ML boundary) (#2404)
## What Final increment of the adapter phase (builds on #2391/#2399/#2400). Flips the **SMART_CRUSHER** primary `.crush()` invocation in `_apply_strategy_to_content` to registry-resolved dispatch, following the CODE_AWARE/HTML pattern. The shared SmartCrusher→Kompress→Log fallback block is unchanged. ## Byte-identical (SMART_CRUSHER) The `smart_crusher` adapter delegates to the same `_get_smart_crusher().crush(content, query=context, bias=bias)` (same cached getter, same method), so `output.content == result.compressed`; the branch recomputes the same `_estimate_tokens` metric; the `if crusher:` guard and the entire fallback chain / `strategy_chain` / `decision_reason` mutations are preserved verbatim. ## Deferred — KOMPRESS and TEXT (honest contract limitation) The `kompress` adapter can't reproduce the direct `_try_ml_compressor(content, context, question)` byte-for-byte, for two independent reasons: 1. **`question` is dropped** — the adapter hardcodes `None`, so QA-aware compression content would diverge. 2. **Token count differs** — the historical branch returns Kompress's own `compressed_tokens` (a word count taken *before* the CCR marker is appended), while the registry path recomputes `_estimate_tokens` over the marker-augmented output. Structurally different numbers whenever Kompress actually compresses. Flipping them would require evolving the adapter/`CompressOutput` contract (forward `question`; carry the compressor's own token count), which is a separate change and would touch the ML boundary — so they're left byte-for-byte here. ## Testing New `tests/test_router_registry_smartcrusher.py`: SMART_CRUSHER success (differential vs a real crush), query/bias forwarding, Kompress-fallback (`[smart_crusher, kompress]`) and Log-fallback (`[smart_crusher, kompress, log]`) chains; plus KOMPRESS/TEXT tests that *pin the deferral facts* (token mismatch + `question` forwarding). Offline suite: 94 passed; ruff (0.15.17) + mypy clean. Full content-router CI suite is the authoritative byte-identical gate. No new config/env. Reversibility gate, external dispatch (#2388), default behavior unchanged. |
||
|
Tejas Chopra
|
7ebda67ef6
|
feat(transforms): add compressed signal + dispatch code_aware/html/diff via registry (#2400)
## What Third increment of the adapter phase (builds on #2391/#2399). Adds a `compressed: bool` field to `CompressOutput` and uses it to flip the **fallback/passthrough** strategies — CODE_AWARE and HTML (and DIFF where clean) — to registry-resolved dispatch, byte-identically. ## The contract addition (the enabling piece) `CompressOutput.compressed: bool = True` — lets a compressor signal **passthrough** (did-not-compress, `content` is the original unchanged) vs a real result. This is what the router's `None`-driven fallback/passthrough branches needed to move to the registry without changing behavior. Default `True`, so existing and external compressors are unaffected. ## How (byte-identical) A new `_registry_compress` helper returns the `CompressOutput` (or `None` when the built-in is unavailable, preserving the `_get_*` guard's passthrough). The flipped branches map that back to their historical `compressed is None` semantics: - **CODE_AWARE:** a passthrough (`not output.compressed` / `None`) sets local `compressed = None`, so the existing `_try_ml_compressor` Kompress fallback + `lossless_then_lossy` no-shrink retry + `strategy`/`strategy_chain` mutations run **verbatim**. - **HTML:** a `None`/passthrough falls through to the bottom passthrough exactly as before (`strategy_chain == [html, passthrough]`). ## Deferred SMART_CRUSHER, KOMPRESS, TEXT, PASSTHROUGH — the SmartCrusher→Kompress→Log fallback chain + the ML boundary — are the next (final) increment, left byte-for-byte here. Reversibility gate, external dispatch (#2388), default behavior unchanged. No new config/env. ## Testing `tests/test_router_registry_dispatch.py` + `tests/test_builtin_compressor_adapters.py` extended: differential tests for CODE_AWARE (success AND None→Kompress-fallback with matching `strategy_chain`, ML mocked), HTML (success AND None→`[html, passthrough]`), and the adapter `compressed=False`-on-None mapping. Offline suite: 88 passed; ruff + mypy clean. The full content-router suite in CI is the authoritative byte-identical gate. |
||
|
Tejas Chopra
|
fc9c63f18c
|
refactor(transforms): dispatch simple built-in strategies via the compressor registry (#2399)
## What Second increment of the adapter phase (builds on #2391). Flips the content router's per-strategy dispatch in `_apply_strategy_to_content` from the hardcoded if/elif to **registry-resolved** — but only for the *clean, single-compressor* strategies: **SEARCH, LOG, TABULAR, CONFIG**. Each resolves its compressor by name from `compressor_registry` and runs it over the pure-data `CompressInput`/`CompressOutput` contract via a shared `_registry_compress_content` helper, then maps back to the branch's exact historical return shape. ## Byte-identical by construction - The built-in adapter delegates to the SAME `_get_<name>()` getter + method with the same args (`context`→query, `bias`→budget), so returned content is identical to the old direct call. - Each flipped branch **keeps its `enable_*` gate and `_get_*` availability guard** — so the built-in-unavailable → passthrough behavior is preserved and the adapter's `None`→content collapse is never reached. - Each branch **recomputes its token count with its own historical metric** (`_estimate_tokens` for search/log/tabular; `len(split())` for config). - `content_type` in `CompressInput` is inert (built-ins don't consume it), so it can't shift output. ## Deferred (left byte-for-byte as-is) — and why - **CODE_AWARE** — has a Kompress/ML fallback chain (`compressed is None` → `_try_ml_compressor`, plus a `lossless_then_lossy` no-shrink retry) that mutates `strategy`/`strategy_chain`. Not a clean single call. - **HTML** — uses `.extract().extracted` (different shape) and relies on `None` extraction falling through to bottom passthrough (`[html, passthrough]`); the adapter's `None`→content collapse would change the chain. Not byte-identical through the entry. - **SMART_CRUSHER** (fallback chain), **KOMPRESS/TEXT** (ML boundary), **PASSTHROUGH**, **DIFF** — untouched per plan. The reversibility gate, external-compressor dispatch (#2388), and default (nothing-selected) behavior are unchanged. No new config/env. ## Testing New `tests/test_router_registry_dispatch.py` (6 tests): differential test per flipped strategy asserting registry-dispatch output == old direct-dispatch output (content + branch token metric + `[strategy]` chain), plus assertions that deferred SMART_CRUSHER and KOMPRESS are unchanged. Offline suite: 78 passed; ruff + mypy clean. The broad content-router suite (HF-Hub/ONNX) is deferred to CI — **that full suite is the authoritative byte-identical gate for the flipped strategies.** |