2 commits

Author	SHA1	Message	Date
Tejas Chopra	c5493ea93b	fix(content-router): token-measure lossless folds at the acceptance gate (#1772) ... ## Description Unit-mismatch bug in the compression acceptance gate. `router.apply()` computes `compression_ratio` from `len(text.split())` (word count), but a **lossless** search/log fold (`compact_lossless`) saves **bytes** by collapsing a repeated path prefix into a single heading — word count stays flat or even *rises* (the heading adds a word). So the gate saw `ratio ≥ 1.0` and discarded every free, byte-recoverable win as `ratio_too_high`. (Raising the floor to 1.0 in #1771 did **not** fix this — the word-ratio was already ≥ 1.0.) Measure lossless results (those whose `strategy_chain` carries a `lossless_*` entry) by **byte ratio** at the gate and in the result cache — the real saving. Lossy strategies are unchanged (word count tracks their token savings), and the reversibility gate is untouched (`LOG`/`SEARCH`/`DIFF` aren't in `LOSSY_UNMARKED_STRATEGIES`). The excluded-tool and bash-search paths already bypass this gate via `continue`; this fixes the **main strategy dispatch** (the lossless-mode `LOG`/`SEARCH`/`DIFF` path). Follow-up to #1771. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - At the `apply()` acceptance gate: compute `accept_ratio` = byte ratio for lossless results (`strategy_chain` has `lossless_*`), else the existing word ratio. Gate + result-cache entry now use `accept_ratio`. - Added an end-to-end regression test that drives the full `router.apply()` path. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text tests/test_lossless_mode.py::test_router_apply_accepts_lossless_search_byte_measured PASSED tests/test_content_router_tool_role_reversibility.py .......... (10 passed) # broader (pre-move) sweep on the same change: tests/test_lossless_mode.py / test_transforms/test_content_router.py / test_lossless_excluded_compaction.py / test_bash_search_lossless_fold.py — 121 passed ruff check headroom/transforms/content_router.py -> All checks passed! mypy headroom/transforms/content_router.py -> Success: no issues found ``` ## Real Behavior Proof - Environment: local worktree, Python 3.12, `PYTHONPATH` pinned to the branch. - Exact command / steps: new regression test constructs a single-file grep result, runs it through `ContentRouter(lossless=True).apply(...)`, and asserts the tool output is byte-smaller and recovers exactly (`search_unheading(out) == original`). - Observed result: before this fix the fold was rejected (`out == original`, counted `ratio_too_high`); after, it's applied (`len(out) < len(original)`, marker-free, byte-exact recovery). The test also asserts the fold's word count is ≥ the original's, so the test is meaningless if "fixed" by word count. - Not tested: no live end-to-end proxy run; validated via the full `apply()` path in unit tests. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable (handled at release time) ## Additional Notes Why prior tests missed it: `compress()` and `_apply_strategy_to_content` return the folded result directly and never touch the `apply()` acceptance gate, so the existing lossless-mode unit tests (which call those) passed while the real proxy path silently discarded the fold. The new test exercises `apply()` end-to-end.	2026-07-03 15:04:07 -07:00
Tejas Chopra	c75ebdee6d	feat(proxy): add --lossless no-CCR mode with format-native compaction (#1721) ... ## Description A new `--lossless` / `HEADROOM_LOSSLESS` proxy mode for deployments **without an MCP retrieve tool** (e.g. bash-only coding agents), where a `<<ccr:…>>` retrieval marker is a dangling, unrecoverable reference. In this mode the ContentRouter compresses tool outputs but **never emits a retrieval marker**, so no MCP round-trip is needed. Routing and prefix caching are unchanged. The guarantee is **no-CCR**, not "everything lossless": the structural compressors get format-native *lossless* compaction, while the ML/prose paths keep their existing (lossy) compression — just made marker-free. Closes # ## Type of Change - [x] New feature (non-breaking; opt-in flag, default off) ## Changes Made - **Flag plumbing** (both proxy entry paths), mirroring `--force-kompress-all`: `ProxyConfig.lossless` (models.py), `--lossless` Click option + argparse arg + `HEADROOM_LOSSLESS` env (cli/proxy.py, server.py), `ContentRouterConfig.lossless`. When on: `smart_crusher_lossless_only=True`, `ccr_inject_marker=False`, and retrieve-tool injection off. - **`headroom/transforms/lossless_compaction.py`** (new, pure stdlib): format-native reversible transforms, each with an exact inverse + runtime round-trip self-check (returns original if it can't safely shrink; never raises): - LOG → `strip_ansi` + `collapse_runs`/`expand_runs` (syslog `repeated ×N`) - SEARCH → `search_heading`/`search_unheading` (ripgrep `--heading` fold) - DIFF → `diff_strip_index` (drop `index <sha>..<sha>`; diff still applies) - **Router disposition**: in lossless mode LOG/SEARCH/DIFF route through `compact_lossless` instead of the lossy Rust drop path; SmartCrusher is marker-free via `smart_crusher_lossless_only`. - **Kompress made marker-free**: `_get_kompress` now builds Kompress with `enable_ccr` tied to `ccr_inject_marker` (previously always `True`). In lossless mode Kompress still drops tokens (lossy, as intended) but no longer appends a `Retrieve more: hash=` marker or writes the CCR store — closing the one path that would otherwise leak an unredeemable marker in production. ## Testing - [x] Unit tests pass - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] New tests added ### Test Output ```text $ pytest tests/test_lossless_mode.py -q 25 passed in 4.88s $ pytest tests/test_transforms_content_router.py tests/test_transforms_content_detection.py -q 46 passed in 0.56s $ ruff check <changed files> -> All checks passed! $ ruff format --check <changed files> -> already formatted $ mypy headroom/transforms/content_router.py -> Success: no issues found ``` ## Real Behavior Proof - Environment: local, Python 3.12.6. - No-CCR invariant: `ContentRouter(ContentRouterConfig(lossless=True))` on repetitive log + grep + diff payloads produces output with **no `<<ccr:` and no `Retrieve ` substring** (chains `lossless_log` / `lossless_search` / `lossless_diff`). - Marker-free Kompress proven **without the model loaded** (the case tests previously couldn't cover): `test_lossless_mode_builds_kompress_marker_free` asserts the router builds Kompress with `enable_ccr=False` in lossless mode and `True` in normal mode. - Reversibility: `collapse_runs`/`expand_runs`, `search_heading`/`search_unheading` round-trip byte-exactly; `compact_lossless` reverts to the original on any round-trip mismatch or non-shrink. - Not tested: end-to-end proxy request replay; live Kompress model output. ## Review Readiness - [x] Self-reviewed - [x] Ready for human review ## Additional Notes - **Stage B (follow-up):** split the low-value KEEP/DROP tail and run Kompress on the *tail* inline (with identifiers registered as Kompress protected tokens), rather than only whole-block ML paths. Not in this PR. - Savings are content-dependent: high on repetitive logs and path-heavy grep, low on diffs and source reads.	2026-07-02 19:04:14 -07:00