## Description
Removes both third-party CLI context tools — **rtk** and **lean-ctx** —
and with them the context-tool selector itself. Headroom no longer
downloads, installs or configures either one, and there is no
replacement.
The previous pass (#2344) gated only three entry points inside
`headroom/cli/wrap.py`. That left the feature reachable in practice:
| Gap | Effect |
|---|---|
| `scripts/install.sh:1544`, `install.ps1:1681` | Ran `rtk init --global
--auto-patch` from bash/PowerShell, **bypassing the Python gate
entirely** — `curl \| sh` still wrote a Claude Code `PreToolUse` hook
regardless of `HEADROOM_RTK` |
| `wrap.py` `_setup_context_tool_for_agent` | **`wrap openhands` was
broken by default**: `rtk_required=True` met a gate returning `None` →
`SystemExit(1)`. Invisible because all 8 openhands tests patched
`_ensure_rtk_binary` to a fake path |
| `proxy/helpers.py`, `subscription/tracker.py` | Proxy shelled out to
`rtk gain` from `/stats`, the dashboard and `headroom perf`; the tracker
polled it per contribution (`_RTK_WIRING_DEFAULT = "enabled"`) |
| No cleanup path | Nothing removed artifacts an earlier default had
installed, so a machine that once ran the old default kept rtk in the
loop forever (#1669, #1955) |
Also worth noting: the rtk binary download had **no SHA or signature
verification** — only `rtk --version` as a smoke test.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)
## Changes Made
**Removed** — `headroom/rtk/` and `headroom/lean_ctx/` packages,
`headroom/cli/wrap_rtk_metrics.py`, `_selected_context_tool` /
`_setup_context_tool_for_agent` / `_VALID_CONTEXT_TOOLS`, the `--rtk` /
`--no-rtk` / `--no-project-rtk` / `--keep-rtk` flags across all 18 wrap
subcommands, `HEADROOM_RTK*`, the proxy-side `rtk gain` polling, the
dashboard CLI-filtering panel (rows + all 8 `cliFiltering*` Alpine
getters), `paths.rtk_path()` / `lean_ctx_path()`, the SDK path helpers,
`benchmarks/rtk_loop_learn_eval.py`, and the `headroom/rtk/**` CI path
filters.
**Fails loudly, not silently** — `--context-tool` / `--no-context-tool`
/ `HEADROOM_CONTEXT_TOOL` are kept solely to error out. They live in
shell profiles, aliases and CI jobs, and accepting them as a no-op would
read as Headroom having quietly stopped working. The installers reject
them too, which matters more than it looks: their arg parsers forward
the first unknown flag **and everything after it** to the wrapped tool,
so a leftover `--no-rtk` would have silently swallowed a following
`--port` and then been ignored downstream.
**New `headroom/context_tool_cleanup.py`** — deleting the code cannot
help a machine that already ran the old default, since the hooks,
binaries and injected guidance are durable on disk.
`purge_context_tool_artifacts()` runs once per `wrap`/`unwrap` and
removes the registered hook entries, the generated hook scripts, the
Headroom-managed `~/.local/bin` symlinks, the vendored
`~/.headroom/bin/{rtk,lean-ctx}` binaries, the `lean-ctx` MCP server
entry and the marker-fenced instruction blocks. Deliberately
conservative: idempotent, **skips** a malformed config rather than
overwriting it, and only unlinks a symlink resolving inside Headroom's
own bin dir so a user's own build is untouched. It reports on
**stderr**, because `wrap/unwrap openclaw --prepare-only` emit
machine-readable JSON on stdout as their entire contract. Skipped for
`wrap selfheal` (runs from a SessionStart hook; must not race Claude
Code's writer for `~/.claude.json`) and for `--help`, which must stay
read-only.
**Client-config hardening** (discovered while investigating a "corrupted
Serena settings file" report) — `wrap.py` reset a settings file to `{}`
when an existing file would not parse, then wrote that back. One
hand-edited typo or a transient `EACCES`/`EINTR` on a valid file
destroyed the user's `permissions`, `env` and `hooks`, on **every
`headroom wrap claude`**. It now refuses to write. Separately,
`fsutil.write_text` is now atomic (temp file + `fsync` + `os.replace`),
fixing all 14 non-atomic client-config writes at once; it follows
symlinks rather than replacing them (dotfile managers) and preserves an
existing file's mode.
**Deliberately kept** — `rtk` stays in the wrapper-peel list in
`transforms/content_router.py`. It sits beside `sudo`/`env`/`timeout` as
shell-command grammar, so `rtk cat f` is still classified as a file read
for anyone running their own rtk install, which the purge intentionally
leaves alone.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ ruff check headroom/ tests/ e2e/ --exclude headroom/dashboard/templates
All checks passed!
$ ruff format --check headroom/ tests/ e2e/ --exclude headroom/dashboard/templates
1255 files already formatted
$ mypy headroom/
Success: no issues found in 508 source files
$ pytest tests/test_context_tool_cleanup.py -q
11 passed
$ pytest tests/test_fsutil.py -q
12 passed
$ pytest tests/test_cli/test_wrap_codex.py -q # 89 tests
89 passed in 431.68s
$ pytest tests/test_cli/test_wrap_opencode.py -q
39 passed in 257.46s
$ pytest tests/test_cli/test_wrap_helpers.py -q
45 passed
$ pytest tests/test_paths.py -q
75 passed
$ pytest tests/test_cli/test_unwrap_claude.py -q
14 passed
$ pytest tests/test_proxy_savings_history.py -q
39 passed
$ pytest tests/test_cli/test_wrap_copilot.py -q
27 passed
$ pytest tests/test_cli/test_wrap_zcode.py -q
20 passed
$ pytest tests/test_subscription_tracker.py -q
9 passed
$ pytest tests/test_proxy_dashboard_stats_cache.py -q
5 passed, 1 skipped
```
Repo-wide grep for 14 removed symbols (`headroom.rtk`,
`headroom.lean_ctx`, `_ensure_rtk_binary`, `_selected_context_tool`,
`_get_context_tool_stats`, `rtk_path`, `lean_ctx_path`,
`wrap_rtk_metrics`, `HEADROOM_RTK`, `cli_tokens_avoided`,
`tokens_saved_rtk`, …) across `*.py`, `*.ts`, `*.sh`, `*.ps1`, `*.yml`,
`*.html`: **zero hits**.
Notable test changes: `test_wrap_openhands.py` no longer patches
`_ensure_rtk_binary` and asserts `wrap openhands --prepare-only` exits 0
unpatched — the regression that was previously masked.
`test_wrap_continue.py` and `test_wrap_hintfile_agents.py` were removed
(every test drove RTK instruction injection). A new
`test_subscription_tracker.py::test_load_state_written_before_cli_context_tools_were_removed`
proves a pre-removal `subscription_state.json` still loads.
## Real Behavior Proof
- **Environment:** macOS 15.4 (darwin 25.4.0), Python 3.12.6, Headroom @
this branch, real `~/.headroom` and `~/.claude` on the dev machine.
- **Exact command / steps and observed result:**
```text
# 1. Retired flag fails loudly instead of silently no-op'ing
$ headroom wrap codex --prepare-only --context-tool rtk
Error: CLI context tools (rtk, lean-ctx) have been removed from Headroom: they
rewrote shell commands through a third-party binary Headroom no longer manages.
Drop --context-tool / --no-context-tool and unset HEADROOM_CONTEXT_TOOL;
`headroom wrap` uninstalls what they left behind on first run.
$ HEADROOM_CONTEXT_TOOL=lean-ctx headroom wrap codex --prepare-only
Error: CLI context tools (rtk, lean-ctx) have been removed from Headroom: ...
# 2. install.sh rejects the retired flags (extracted parse_wrap_args harness)
['--no-rtk', '--port', '9999'] rc=1 ERROR: CLI context tools ... Drop --no-rtk
['--context-tool=rtk'] rc=1 ERROR: CLI context tools ... Drop --context-tool
$ bash -n scripts/install.sh # syntax OK
# 3. Purge ran against the real machine, which had all the orphaned artifacts
$ python -c "from headroom.context_tool_cleanup import purge_context_tool_artifacts; ..."
removed ~/.headroom/bin/lean-ctx (51 MB)
removed ~/.headroom/bin/rtk (7.7 MB)
removed ~/.local/bin/rtk (symlink into ~/.headroom/bin)
removed ~/.claude/hooks/rtk-rewrite.sh
removed 8 lean-ctx-* hook scripts
# ~/.claude.json afterwards: 90 top-level keys, 19 projects, mcpServers unchanged
# → ~59 MB reclaimed, no unrelated key touched
# 4. stdout stays machine-readable while the purge reports (planted a fake artifact)
$ headroom wrap openclaw --prepare-only --gateway-provider-id codex >out 2>err
$ cat out
{"enabled":true,"config":{"proxyPort":8787,...}} # parses as JSON
$ cat err
Retired CLI context tool cleanup: removed /Users/tcms/.headroom/bin/rtk
# 5. --help is inert (planted artifact survives), a real run purges
$ headroom wrap codex --help → artifact survived: CORRECT
$ headroom wrap openclaw --prepare-only → purged: CORRECT
# 6. MCP purge dry-run against a copy of the real 82 KB ~/.claude.json
top-level keys 90 -> 90; projects 19 -> 19; LOST keys: none
all content outside mcpServers byte-identical: True
```
Dashboard rendered via the Playwright test after the panel removal:
"Token Savings" shows only `Proxy 0 (0.0%)` / `Of total wire: 36.86%`,
and "Token Usage" reads Before Compression → Proxy Removed → After
Compression with no "Filtered (this session)" row. Nothing below the
removed panel broke.
- **Not tested:** Windows and Linux (macOS only) — `install.ps1` is
verified by brace-balance and inspection, not executed, since no `pwsh`
is available locally. The wrap e2e suite (`e2e/wrap/run.py`) was updated
but not run; it needs the Docker e2e image. `serena project index`
interaction is exercised in the stacked base PR.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
**Stacked on #2676** (`tejas/serena-config-bootstrap`) — please merge
that first; this PR's base should then be retargeted to `main`, or it
will read as containing that fix too.
**Breaking-change migration for users:**
- Drop `--rtk`, `--no-rtk`, `--no-project-rtk`, `--keep-rtk`,
`--context-tool`, `--no-context-tool` from any alias, script or CI job,
and unset `HEADROOM_RTK*` / `HEADROOM_CONTEXT_TOOL`. They now error
rather than being ignored, so the failure is immediate and
self-explaining.
- Previously-installed artifacts are purged automatically on the next
`wrap`/`unwrap`; no manual cleanup needed.
- `headroom perf --json` no longer carries a `cli_filtering` key, and
`/stats` no longer returns a `context_tool` section.
**Docs:** `docs/rtk-architecture.md` deleted; RTK/lean-ctx removed from
`README.md`,
`docs/content/docs/{configuration,opencode,grok-build,docker-install,filesystem-contract}.mdx`,
`docs/observability.md` and the matching `wiki/` pages.
`REALIGNMENT/09-phase-G-rtk-observability.md` is marked SUPERSEDED
rather than deleted, to keep the planning record.
**Follow-ups not in scope:** `_emit_wrap_interrupted` was deleted as
dead code — its only caller was the `except KeyboardInterrupt` guarding
the binary download, so with no download there is nothing slow left to
interrupt.
## Description
`TextCrusher` (the native extractive prose compressor added in #1171)
only handled ASCII: `split_segments` split on `.!?`+whitespace and
`tokens` split on whitespace/alphanumeric runs. CJK
(Chinese/Japanese/Korean) has neither spaces nor ASCII terminators, so a
whole CJK paragraph collapsed into **one segment / one token** — it
passed through at ~0% compression, and BM25 relevance + salience scored
zero terms.
This makes `TextCrusher` CJK-aware. CJK-bearing content takes an ICU
(`icu_segmenter`, UAX#29 sentence + dictionary word) segmentation path,
with a length fallback for terminator-sparse runs, a local BM25 over the
ICU word tokens, and ICU-token salience. Dispatch is on **content
only**, so pure-ASCII text is byte-identical to before — the shared
`BM25Scorer` and the ASCII path are untouched.
It also adds a committed, reproducible answer-retention eval
(`benchmarks/i18n_compression_eval.py`) with a deterministic zh/ja/ko CI
regression gate, so the improvement below is permanently verifiable
rather than a one-off measurement.
Extends #1171.
## Type of Change
- [x] Bug fix (CJK passed through near-uncompressed)
- [x] New feature (CJK segmentation / relevance support)
- [x] Performance improvement (CJK now compresses; ICU segmenters
cached, not rebuilt per call)
## Changes Made
- `is_cjk` predicate gates a CJK path (ideographs, kana, Hangul, CJK
punctuation, full/half-width forms).
- `split_segments` → ICU `SentenceSegmenter` for CJK + a mandatory
length fallback (whitespace / CJK punctuation / hard cap) for
terminator-sparse runs; ASCII path unchanged.
- `tokens` → ICU `WordSegmenter` (dictionary) for CJK; ASCII path
unchanged.
- `relevance_cjk`: a local BM25 over ICU word tokens — the shared ASCII
`BM25Scorer` scores zero terms for CJK and is parity-locked, so this is
an intentional separate scorer (documented in code).
- CJK salience uses ICU tokens (whitespace-split gave one giant "word" →
zero salience).
- `count_tokens`: CJK-aware so `compression_ratio` isn't nonsense for
space-free text.
- ICU segmenters resolved once in `static LazyLock` (compiled_data is
static) instead of rebuilt per call.
- New dep `icu_segmenter` 2.2, `compiled_data` only (see Dependency
below).
- `benchmarks/i18n_compression_eval.py` +
`tests/test_transforms/test_text_crusher_cjk_eval.py`: a zh/ja/ko
answer-retention eval — a deterministic needle CI gate (always-runs, no
external data), real-transcript fidelity with CJK-aware salient, and
optional `multi-wiki-qa` natural-data retention (loaded via the
`[evals]` `datasets` extra, skipped if absent; data never vendored —
CC-BY-NC-SA).
## Testing
- [x] Unit tests pass (`pytest` + `cargo test`)
- [x] Linting passes (`ruff check`/`format` on the new eval + test —
clean)
- [ ] Type checking passes (`mypy headroom`) — N/A, the only Python
added is a benchmark + test, not `headroom/` source
- [x] New tests added for new functionality
- [x] Manual testing performed (see Real Behavior Proof)
### Test Output
```text
$ cargo test -p headroom-core --lib text_crusher
running 12 tests
test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 841 filtered out
$ .venv/bin/python -m pytest tests/test_transforms/test_text_crusher*.py
15 passed
$ .venv/bin/python -m pytest tests/test_transforms/test_text_crusher_cjk_eval.py
6 passed # deterministic zh/ja/ko needle CI gate
$ cargo clippy -p headroom-core && ruff check benchmarks/i18n_compression_eval.py # both clean
```
## Real Behavior Proof
- Environment: macOS (Darwin 25.3.0), Python in a uv venv,
`headroom-core` built via `uv pip install -e .` (maturin), branch
`feat/cjk-text-compression`.
- Exact command / steps: built `_core`, then ran a mixed
Chinese+Japanese doc (no spaces, `。` terminators) through
`TextCrusher().compress(doc, "认证令牌缓存策略", 0.3)`; separately evaluated
answer-retention on the public CMRC2018 Chinese QA dev set (bury the
gold-answer paragraph among 25 distractors, query = the question,
compress to 30%, check the gold answer survives), and end-to-end through
`ContentRouter`.
- Observed result: a mixed Chinese+Japanese doc compressed 189 → 78
tokens (ratio 0.41, kept 3/8 segments) with the query-relevant sentence
surviving — before this change the same doc was a single segment → 100%
passthrough. On the public CMRC2018 Chinese QA dev set, answer-retention
under 30% compression rose 34% → 93% (multiple seeds). End-to-end
through `ContentRouter` on real CJK content, aggregate savings rose 16%
→ 40%. Pure-ASCII (English) output stayed byte-identical (the English
parity fixtures did not move). Demo terminal output:
```text
ORIGINAL tokens= 189 chars=189
COMPRESS tokens= 78 ratio=0.41 segments kept 3/8
QUERY-RELEVANT sentence survived: True
--- compressed output (verbatim kept CJK sentences) ---
认证令牌的缓存策略采用最近最少使用淘汰算法来管理过期条目。
请求重试使用指数退避并设置最大次数上限。
数据备份每天凌晨执行并保留最近三十天的快照。
```
The committed eval now demonstrates this across all three CJK languages.
The deterministic needle gate (in CI via
`tests/test_transforms/test_text_crusher_cjk_eval.py`, 6 passed) has
TextCrusher keep the query-relevant needle while truncate/random drop it
in zh, ja, and ko. On real `multi-wiki-qa` natural data (n=80/lang),
query-aware answer-retention is **zh 74% / ja 70% / ko 50%** vs
**25–41%** for the truncate/random baselines:
```text
=== Part A: multi-wiki-qa answer-retention (n=80/lang, target_ratio=0.3)
===
lang text_crusher truncate random
zh-cn 74% 25% 38%
ja 70% 31% 39%
ko 50% 26% 41%
```
Korean is measurably weaker (ICU has no Korean dictionary and falls back
to UAX#29 word-breaking) — still well above baselines, and scoped as a
follow-up.
- Not tested: the live proxy HTTP path (validated at the `ContentRouter`
/ `TextCrusher` layer, not via a running proxy); no-space Korean
(standard Korean is space-delimited and is covered); non-CJK SE-Asian
scripts (out of scope).
## Dependency (per CONTRIBUTING supply-chain policy)
`icu_segmenter` 2.2 (ICU4X), `features = ["compiled_data"]`:
- **Why this package (vs. ourselves / existing deps):** CJK needs
dictionary/UAX#29 segmentation. A hand-rolled char-bigram scored
slightly worse on real data (CMRC2018 answer-retention: 92.5% ICU vs 91%
bigram, 4 seeds); jieba/lindera are ZH-only or 13–207 MB dicts. ICU4X
covers zh/ja/ko in one crate. The existing `unicode-segmentation` does
UAX#29 only (no CJK dictionary), so it can't word-segment space-free
CJK.
- **Who maintains it:** the official `unicode-org` ICU4X project; active
release cadence (2.2 in 2025); no known CVEs.
- **Install surface:** ~13 new pure-Rust crates, no build scripts, no
native code, no build/runtime network. `compiled_data` bundles locale
data at compile time (hermetic). `auto`/`lstm` deliberately NOT enabled
— LSTM covers SE-Asian scripts (Thai/Lao), not CJK, and would pull in
`libm` for nothing.
- **Why this version:** 2.x is the stabilized ICU4X API (1.x used a
different data-provider model); floored at 2.2 (Cargo.lock pins the
patch) since segmenter boundaries are observable in output and bumps
should be deliberate.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation (CHANGELOG)
- [x] My changes generate no new warnings (clippy + fmt clean)
- [x] I have added tests that prove my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md
## Additional Notes
- **Parity:** the shared `BM25Scorer` (byte-exact parity-locked with
`headroom/relevance/bm25.py`) is untouched. `relevance_cjk` is a
separate local scorer because the shared one's tokenizer is ASCII-only.
The whole CJK path lives in Rust (`text_crusher.py` is a thin wrapper
over `_core`), so there is no Python mirror to keep in sync; the parity
fixtures stay green (only the CJK `unicode` fixture was re-recorded,
intentionally; English fixtures unchanged).
- **Known by-design gap (not a bug):** CJK content + a pure-ASCII query
yields no token overlap, so relevance falls back to recency + salience
(cross-script query matching is unsupported).
- The Python added is a benchmark
(`benchmarks/i18n_compression_eval.py`) plus its test, not `headroom/`
runtime source — both are `ruff`-clean; `mypy headroom` is unaffected.
- **License:** the optional Part A pulls `alexandrainst/multi-wiki-qa`
(CC-BY-NC-SA-4.0) at run time via the `[evals]` extra and is skipped if
absent — the dataset is never vendored into the repo, and the always-run
CI gate (Part C) uses only our own deterministic data.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Description
Sync the docs with the code after the live-zone realignment. The
`IntelligentContextManager` (ICM), `RollingWindow`, and scoring modules
were deleted in PR #350 (May 2026), but the README and benchmark
docstrings still advertised them as live, and an example still imported
the deleted module (broken on run). This fixes the README + benchmarks
and removes the dead example.
I validated the README against the code with three parallel
static-analysis sub-agents (features/architecture,
CLI/extras/wrap-matrix, public API/integrations). Most of the README
checked out accurate; only the items below were stale/wrong.
Closes #
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- README: removed the `IntelligentContext` bullet and
`IntelligentContext / RollingWindow` from the transforms list (both
deleted in PR #350).
- README: standardized `Kompress-base` -> `Kompress-v2-base` to match
the HF model id `chopratejas/kompress-v2-base` and the existing badges
(diagram re-aligned).
- README: corrected the CodeCompressor language list to match the
`CodeLanguage` enum (added TS, C, Perl).
- README: softened the unanchored "6 algorithms" tagline to
"content-aware compressors".
- README: Cortex Code is library-mode only — there is no `headroom wrap
cortex`, so the compatibility-matrix row no longer shows a wrap
checkmark.
- Deleted `examples/test_intelligent_context_toin_ccr.py` — it imported
the deleted `IntelligentContextManager` (ImportError on run) and is
unreferenced.
- Removed stale `RollingWindow` mentions from benchmark
docstrings/comments (`benchmarks/__init__.py`, `bench_transforms.py`,
`bench_latency.py`, `scenarios/conversations.py`); the accurate PR-B1
retirement comment is kept.
## Testing
- [ ] Unit tests pass (`pytest`) — N/A, docs/docstring + example
deletion only
- [x] Linting passes — `ruff check` clean on all changed benchmark files
- [ ] Type checking passes — N/A (no type-relevant changes)
- [ ] New tests added — N/A
- [x] Manual testing performed — see Real Behavior Proof
### Test Output
```text
$ ruff check benchmarks/__init__.py benchmarks/bench_transforms.py benchmarks/bench_latency.py benchmarks/scenarios/conversations.py
All checks passed!
# stale refs remaining in README/benchmarks (excluding accurate retirement notes):
$ grep -rn "IntelligentContext|RollingWindow|Kompress-base" README.md benchmarks/ | grep -v retire
(only benchmarks/bench_transforms.py:362 — the accurate PR-B1 retirement comment)
# deleted example is unreferenced anywhere:
$ grep -rn "test_intelligent_context_toin_ccr" --include=*.md --include=*.yml --include=*.py .
(no hits)
```
## Real Behavior Proof
- Environment: macOS (darwin, arm64), Python 3.12 `.venv`, ruff 0.14.x,
repo at branch `docs/sync-readme-with-code` off latest `main`.
- Exact command / steps: (1) three parallel sub-agents
grep/Read-validated README claims vs `headroom/`, `pyproject.toml`,
`sdk/typescript/`; (2) directly verified each flagged mismatch
(`CodeLanguage` enum, `HF_MODEL_ID`, absence of
`IntelligentContext`/`RollingWindow` classes); (3) confirmed the example
imports a deleted module and is unreferenced; (4) `ruff check` on
changed benchmark files; (5) re-grepped README + benchmarks for any
remaining stale refs.
- Observed result: README and benchmark docstrings now match the code;
the only surviving `RollingWindow` string is the accurate retirement
comment; the broken example is removed; ruff passes; the ASCII
architecture diagram still aligns after the `Kompress-v2-base` rename.
- Not tested: rendering of the README on GitHub/PyPI (text-only change);
the separate `docs/content/` and `wiki/` doc sets (see Additional Notes
— out of scope for this PR).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective — N/A
(docs/example cleanup)
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md — N/A (Release Please
auto-generates from the conventional commit)
## Additional Notes
**Larger related finding (NOT in this PR):** the published docs site
(`docs/content/docs/*.mdx`) and the `wiki/*.md` set still document
`IntelligentContextManager`, `RollingWindow`, `RollingWindowConfig`,
`IntelligentContextConfig`, and `ScoringWeights` as live API — with
`from headroom import RollingWindow` / `from headroom.transforms import
IntelligentContextManager` code examples that would `ImportError`. It is
half-migrated (a couple of `.mdx` files already note "removed in 0.9.x"
while neighbors still teach it as current). This is ~15 files and the
fixes require rewriting examples to the live-zone model, not just
deletions — recommended as a focused follow-up PR rather than bundling
it here.
The optional `query` parameter on headroom_retrieve routed retrieval
through CompressionStore.search(), which BM25-scored the items inside a
single cached blob and dropped everything below a 0.3 relevance floor.
On small per-blob corpora with conversational queries this returned an
empty result the large majority of the time, so the LLM saw "nothing
found" for content that was actually present — pushing users to turn
compression off entirely.
Retrieval is fundamentally a hash lookup (this already matches the Rust
proxy's CCR store, which is put/get only — "no BM25 search"). Remove the
query/search path end to end and always return the full original
content:
Core (Python proxy):
- tool schemas (anthropic/openai/google) drop the `query` property
- parse_tool_call returns the hash (str | None) instead of (hash, query)
- response handler, proxy POST/GET/tool-call handlers, the MCP retrieve
tool, and the streaming feedback recorders retrieve by hash only
- proactive context-tracker expansion always restores full content
- delete CompressionStore.search() and its BM25 machinery (the bm25
module stays — it is still used by relevance/)
- CCRToolCall.query, CCRToolResult.was_search, and
ExpansionRecommendation.expand_full/search_query are removed
Plugins (advertised a now-defunct query param to the LLM):
- hermes (Python), openclaw + opencode (TypeScript) retrieve tools drop
`query` from their schemas, signatures, request URLs, and tests
Benchmarks/docs:
- ccr_regression + adversarial benchmarks switch from store.search() to
full hash retrieval (search input-injection tests repurposed to the
hash, the only remaining input surface)
- wiki/ARCHITECTURE.md, wiki/ccr.md, docs/content/docs/ccr.mdx,
config.py and store docstrings updated to describe hash-only retrieval
Tests updated to assert full-content retrieval and guard the removed
surface; the full CCR/proxy/store/TOIN suite passes. ruff + mypy clean.
## Description
<!-- Briefly explain the change and why it is needed. -->
Closes #
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
-
## Testing
<!-- Check what you actually ran, then paste the real command output
below. -->
- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
# Paste relevant command output or artifact links here
```
## Real Behavior Proof
- Environment:
- Exact command / steps:
- Observed result:
- Not tested:
## Review Readiness
- [ ] I have performed a self-review
- [ ] This PR is ready for human review
## Checklist
- [ ] My code follows the project's style guidelines
- [ ] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
Add screenshots to help explain your changes.
## Additional Notes
<!-- Mention any N/A checklist items, tradeoffs, follow-ups, or
maintainer context. -->
## Description
On a cold-start large context, kompress (ModernBERT ONNX) runs
**synchronously on the request thread** — ~200–300s for ~1M tokens. It
blows the 30s compression budget, leaks a non-preemptible worker, and
cascades (executor saturation → queue timeouts on healthy requests); on
timeout the request is forwarded **uncompressed** after eating 30s. This
adds four layered, **default-off, fail-open** mitigations so the request
path is never blocked on ML compression.
Closes#1171
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- **Phase 0 — size gate** (`HEADROOM_KOMPRESS_MAX_TOKENS`, default
50000): route oversized text away from ModernBERT (→ LogCompressor /
TextCrusher / passthrough) at the single `_try_ml_compressor` boundary.
- **Phase 1 — cooperative deadline**
(`HEADROOM_COMPRESSION_DEADLINE_MS`, default 20000): any kompress run
self-terminates at the next chunk boundary past the budget, keeping the
unprocessed tail verbatim.
- **Phase 2 — TextCrusher** (`HEADROOM_TEXT_CRUSHER`): a new **native
Rust** extractive prose compressor in
`crates/headroom-core/src/transforms/text_crusher/`, exposed via PyO3 as
`headroom._core.TextCrusher` with a thin Python wrapper. It **reuses the
shared `crate::relevance::BM25Scorer`** rather than reimplementing BM25,
and ships record/replay parity fixtures (mirroring the SmartCrusher
Rust-core + Python-shim pattern).
- **Phase 3 — off-path compression**
(`HEADROOM_BACKGROUND_COMPRESSION`): forward uncompressed immediately
and compress in a per-process background drain; a byte-identical cache
hit on a later turn means the request never blocks on ML.
- Benchmark (`benchmarks/text_crusher_quality_eval.py`), CHANGELOG
entry, and docstrings documenting the fail-open limits.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`, new modules)
- [x] New tests added for new functionality
- [ ] Manual testing performed (Phase 0/1 gate-fire + deadline observed
on real traffic in earlier iterations; Phase 3 off-path is unit- +
byte-identity-tested, not yet live-validated)
### Test Output
```text
$ pytest tests/test_transforms/ tests/test_cache/ \
tests/test_proxy/test_background_compression.py tests/test_proxy/test_phase3_byte_identity.py -q
501 passed, 37 skipped in 40.33s
$ cargo test -p headroom-core --lib text_crusher
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 834 filtered out
$ ruff check <changed files>
All checks passed!
$ mypy headroom/proxy/background_compression.py headroom/transforms/text_crusher.py
Success: no issues found in 2 source files
```
New coverage: size-gate incl. the strategy-dispatch funnel (KOMPRESS +
TEXT); partial-run deadline (chunk-0 compressed + chunk-1 verbatim
tail); BackgroundCompressor (dedup / queue-full / fail-open); Phase 3
byte-identity round-trip; TextCrusher unit + parity.
## Real Behavior Proof
- Environment: macOS, local dev — `uv` venv, Rust `_core` built via `uv
pip install -e .`.
- Exact command / steps: the `pytest` / `cargo test` / `ruff` / `mypy`
commands shown under Test Output; quality eval `python
benchmarks/text_crusher_quality_eval.py /tmp/squad_dev.json`.
- Observed result: 501 Python + 3 Rust tests pass; ruff + mypy clean on
changed/new modules. Quality eval: TextCrusher keeps ~94% of buried
SQuAD answers at 30% size vs ~36% truncate/random; self-contained speed
run ~333k words in ~76ms (one O(n) pass) — sub-second where ModernBERT
takes minutes (fast-vs-slow contrast, not a same-input run).
- Not tested: Phase 3 off-path on live traffic; multi-worker
(per-process by design — see Additional Notes).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- **All four features are off by default and fail-open** — with the env
flags unset the paths are no-ops for realistic inputs; on any error the
request is forwarded (compressed if possible, else verbatim), never
dropped. A full background queue / duplicate key surfaces as
`deferred:dropped`.
- **Known limits (documented in `background_compression.py`):** Phase 3
is per-process, in-memory, and token-mode-only — these are
**lost-savings, never lost-correctness**, and consistent with the
project's existing per-process compression cache + sticky-session
multi-worker model. The startup multi-worker warning now names off-path
background compression.
- Phase 2 reuses the existing BM25 scorer; reuse did not improve
answer-retention over a Python prototype (query-awareness dominates) —
its value is the Rust speed + repo-conventional Rust-core/Python-shim
shape.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Description
`headroom learn` ranked recommendations by a single LLM-guessed
`estimated_tokens_saved` with a flat hardcoded `confidence`, and had
**no notion of a loop**. So (1) RTK re-fetch loops were invisible - RTK
truncates a command's output, the agent re-runs larger-limit variants,
those calls *succeed* (`is_error=False`), and `analyze()` even
early-returned when a session had no failures and no events - and (2)
even when surfaced, a loop ranked no higher than a one-off mistake. This
adds loop-aware weighting plus the eval that reproduces an RTK loop,
runs it through Learn, and checks the guardrail prevents re-triggering.
Closes#1159
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- New `headroom/learn/loops.py`: `detect_loops()` (canonical signature
collapses RTK pagination/limit variants; classifies error vs rtk-refetch
loops; **measured** wasted tokens), `format_loops_for_digest()`,
`apply_loop_weighting()`.
- `analyzer.py`: detect loops up front (fixes the no-failure
early-return), lead the digest with them, prioritize loops in the system
prompt, re-sort after weighting.
- `models.py`: `Recommendation.is_loop_guardrail` / `loop_occurrences`.
- `benchmarks/rtk_loop_learn_eval.py` + `headroom/learn/fixtures.py`:
the two-phase RTK-loop eval and its session fixtures.
- Tests, `docs/rtk-loop-weighting.md`, CHANGELOG entry.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) - not run (mypy not in my
minimal env; see Not tested)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_learn/ -q
190 passed, 3 skipped, 1 warning in 5.85s
$ ruff check <changed files>
All checks passed!
```
## Real Behavior Proof
- Environment: macOS (Darwin 25.0), Python 3.10.18, fresh venv (`pip
install -e` minus the optional `hnswlib`/proxy extras, which are
unrelated to `learn`); real LLM via the analyzer's claude CLI backend
(`HEADROOM_LEARN_CLI=claude`, claude-cli 2.1.158) — no API key used.
- Exact command / steps: `HEADROOM_LEARN_CLI=claude python -c "from
benchmarks.rtk_loop_learn_eval import run_eval;
c=run_eval(use_real_llm=True); print(c.render())"`
- Observed result: the analyzer shelled out to a real model and produced
the "Commands" guardrail quoted below, naming the looping command. The
digest reports the measured 5,005-token waste and asks the model to rank
loops first, so the model emitted that figure; in this run the guardrail
ranked **#1** and the scorecard was all-PASS (below). Caveat — real-mode
is run-dependent: the rule's wording, and whether the post-hoc
`apply_loop_weighting` fuzzy match fires, vary across runs (in one run
it did not tag the rule). The **deterministic CI eval** (stub LLM) is
the stable, reproducible artifact; this real run corroborates it.
- Not tested: the analyzer's API-key path (ANTHROPIC/OPENAI/GEMINI) —
exercised the equivalent claude CLI backend instead; `mypy`; a live
agent *obeying* the written rule end-to-end (Phase 2 is a non-recurrence
check, not a live agent — called out in the doc).
Real model output from this run, ranked #1 at the measured 5,005-token
weight:
> **Commands** — When grepping logs (or any large file), never loop with
increasing `| head -N` limits — tool output is capped at ~4 KB
regardless of N, so repeated attempts return identical bytes. Instead:
redirect to a temp file (`grep ... > /tmp/out.txt`) then read it, or use
`grep -c` first…
```text
[PASS] loop_detected (1 loop(s), ~5,005 tok wasted)
[PASS] guardrail_produced
[PASS] ranked_first
[PASS] names_command
[PASS] prescribes_fix
[PASS] weight_reflects_waste
[PASS] guardrail_holds
RESULT: PASS
```
(One real-mode run via the claude CLI backend. The deterministic
`pytest` eval above is the stable artifact; see the run-dependence
caveat under Observed result.)
The real run also caught an over-brittle check: an earlier
`names_command` required the literal "TimeoutError"; the real model
wrote a *more general* rule (grep + `head -N`) without it, so I fixed
the check to verify the looping **command** is named, not an incidental
literal.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- No new dependencies. No network, no user/assistant content dropped —
operates on already-captured session digests.
- Kept as one logical change. mypy not run locally (minimal env); happy
to address anything CI's mypy flags.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
Phase B step 2 of the live-zone-only realignment. Replaces PR-A1's
unconditional "passthrough" stub with a real dispatcher that
inspects the Anthropic /v1/messages body, identifies the live zone
(latest user message at index >= frozen_message_count), and routes
each block to a per-type compressor. PR-B2 wires every per-type
compressor to a no-op, so the dispatcher returns
LiveZoneOutcome::NoChange on every call — bytes-in == bytes-out.
PR-B3+ replaces the no-ops with SmartCrusher, Log, Search, Diff,
and Code compressors.
Adds:
- crates/headroom-core/src/transforms/live_zone.rs — public API:
- `compress_live_zone(body, frozen_message_count, AuthMode)`
- `LiveZoneOutcome::{NoChange, Modified}`
- `CompressionManifest` with per-block outcomes (message_index,
block_index, block_type, BlockAction).
- `BlockAction::{NoOpSkeleton, Excluded { reason }}`. The
HOT_ZONE_BLOCK_TYPES list (`tool_use`, `thinking`,
`redacted_thinking`, `compaction`) excludes blocks even when
they appear in the latest user message.
- `AuthMode::{Payg, OAuth, Subscription}` — accepted but unused
in B2; PR-F2 wires the auth-mode gate.
- 12 unit tests pin: empty messages, no messages field, invalid
JSON, latest user message selection, frozen_count respect,
hot-zone block exclusion, string-shaped content, no user msg
in live zone, AuthMode no-op, NoChange contract, manifest
counters, frozen-count clamping.
- crates/headroom-proxy/src/compression/live_zone_anthropic.rs —
new entry point. `compress_anthropic_request` parses the body,
resolves frozen_count via `resolve_frozen_count` (PR-A4 helper),
dispatches via `compress_live_zone`, and returns
`Outcome::NoCompression` on PR-B2 success / `Outcome::Passthrough
{ reason: NotJson | NoMessages | ModeOff }` on body-shape /
policy issues. Six unit tests pin: mode_off short-circuit, no
messages field, invalid JSON, valid body NoCompression,
empty body, cache_control disabled.
Modifies:
- compression/mod.rs — re-exports `compress_anthropic_request` from
`live_zone_anthropic` instead of `anthropic`. The old anthropic
module is reduced to the `resolve_frozen_count` helper only
(not deleted, because its CacheControlAutoFrozen-policy gate is
reused).
- proxy.rs — passes `state.config.cache_control_auto_frozen` into
the dispatcher. Drops the obsolete "live_zone reserved for
Phase B" warning that PR-A1 emitted on every request.
- compression/anthropic.rs — pruned to the resolve_frozen_count
helper plus its tests. The PR-A1 passthrough stub
`compress_anthropic_request` is gone (live_zone_anthropic owns
the name now).
- config.rs — `compression_mode` doc updated to reflect the wired
dispatcher (no longer "reserved for Phase B").
- tests/integration_compression.rs — `compression_decision_logged`
pins the new log contract (`decision="no_change"`,
`reason="no_op_skeleton_pr_b2"`, plus manifest fields
`frozen_message_count`, `messages_total`, `live_zone_blocks`).
Asserts the obsolete Phase A warning is NOT emitted.
- proxy.rs no longer imports CompressionMode (only used inside the
retired warning).
Benchmark cleanup (B1 leftovers that surfaced now):
- benchmarks/proxy_mode_benchmark.py + claude_session_mode_benchmark.py:
drop `intelligent_context=False` arg from ProxyConfig (the field
was retired in B1; tests/test_proxy_mode_benchmark.py and
tests/test_claude_session_mode_benchmark.py imported these
factories and started failing).
- benchmarks/bench_transforms.py: delete TestRollingWindowBenchmarks
class; rewire TestTransformPipelineBenchmarks fixture without
RollingWindow.
- benchmarks/conftest.py: drop rolling_window_config fixture.
- benchmarks/run_benchmarks.py: drop the `window` suite + table
rows referencing RollingWindow.
Cache-safety invariant:
- PR-B2 dispatcher never mutates body bytes (no-op skeleton). The
proxy forwards the original buffered bytes byte-equal. Phase A's
SHA-256 fixtures pin this.
- `passthrough_mode_live_zone_currently_passthrough_byte_equal_sha256`
retitled comment to reflect the dispatcher being live but
no-op.
Acceptance:
- cargo build --workspace + clippy + fmt: green.
- cargo test --workspace --exclude headroom-py: all green
(777 + 12 new live_zone + 6 new live_zone_anthropic tests).
- pytest: 4678 passed, 240 skipped, 0 failed.
- Anthropic decision log includes manifest fields per the
observability contract documented in
REALIGNMENT/02-architecture.md.
Per-PR-B2 plan: REALIGNMENT/04-phase-B-live-zone.md.
`headroom/transforms/text_compressor.py` was a regex-line-sampling
fallback that nothing in the runtime called. ContentRouter routes
`CompressionStrategy.TEXT` straight to the Kompress ML compressor at
`content_router.py:1046` — the comment there literally says 'Prefer
Kompress ML compressor for text'. The Python file was orphaned but
still imported by its own test class, making it look live in the 3e
queue.
Drops the 3e.3 port from the queue: there's nothing to port.
# Removed
* `headroom/transforms/text_compressor.py` (255 LOC, unused)
* `tests/test_text_compressors.py::TestTextCompressor` (3 tests)
* `text_compressor` mention in `error_detection.py` shim docstring
* `text_compressor` mention in `test_signals_keyword_parity.py` docstring
* `TextCompressor` mention in `bench_latency.py` scenario comment
# Kept (defensive)
The legacy marker regex in `ccr/tool_injection.py:213` stays — it
parses an even older TextCompressor output format (pre-2026), is
purely defensive, and removal buys nothing. Test references to that
format in `test_ccr_tool_injection.py` document the regex contract
and stay too.
# Test plan
* `make ci-precheck` clean
* `tests/test_text_compressors.py` 19 passes (was 22, dropped 3)
Sync the Anthropic cache stability test double with the prefix tracker contract used by the handler.
Format the benchmark scripts that were failing ruff format --check in CI.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
LLMLingua was the original ML text compressor (BERT-based). Kompress
(ModernBERT, trained on 330K structured tool outputs) replaced it with
better compression quality and simpler architecture.
Removed across 35 files:
- Deleted headroom/transforms/llmlingua_compressor.py
- Deleted tests/test_transforms/test_llmlingua_compressor.py
- Deleted tests/test_proxy_llmlingua.py
- Removed all enable_llmlingua config, _get_llmlingua methods,
LLMLingua fallback paths, LLMLINGUA strategy enum values
- Removed CLI flags, model configs, compression handler references
- Simplified ContentRouter: Kompress is primary and only text compressor
- headroom wrap claude is the simplest way to start up claude
- It will also install rtk-ai locally
- rtk-ai is a cli wrapper that can save ~90% tokens for CLI calls made by Claude Code
- Add quality_retention_eval.py for needle-in-haystack testing to verify
intelligent compression retains critical information (100% retention achieved)
- Add intelligent_context_integration_test.py for comprehensive pipeline testing
- Add test_progressive_summarizer.py with 36 tests for ProgressiveSummarizer
- Add HeadroomConfig parameter to HeadroomClient for direct config injection
- Update pipeline.py with IntelligentContextManager wiring and logging
- Fix all ruff linting issues and format for Python 3.12 compatibility
- Add comprehensive_eval.py benchmark for multi-scenario evaluation
- Add real_data_demo.py for production-scale volume testing
- Add reasoning agent test examples (groq, debug)
- Add detailed breakdowns by provider and model to /stats
- Include compression, telemetry, and feedback loop statistics
- Add latency.average_ms metric
- Add real-world agent benchmark with MCP tool patterns
- Add worst-case and adversarial benchmarks for edge cases
- Bump version to 0.2.12
