mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
4035c04187
## Description `TextCrusher` (the native extractive prose compressor added in #1171) only handled ASCII: `split_segments` split on `.!?`+whitespace and `tokens` split on whitespace/alphanumeric runs. CJK (Chinese/Japanese/Korean) has neither spaces nor ASCII terminators, so a whole CJK paragraph collapsed into **one segment / one token** — it passed through at ~0% compression, and BM25 relevance + salience scored zero terms. This makes `TextCrusher` CJK-aware. CJK-bearing content takes an ICU (`icu_segmenter`, UAX#29 sentence + dictionary word) segmentation path, with a length fallback for terminator-sparse runs, a local BM25 over the ICU word tokens, and ICU-token salience. Dispatch is on **content only**, so pure-ASCII text is byte-identical to before — the shared `BM25Scorer` and the ASCII path are untouched. It also adds a committed, reproducible answer-retention eval (`benchmarks/i18n_compression_eval.py`) with a deterministic zh/ja/ko CI regression gate, so the improvement below is permanently verifiable rather than a one-off measurement. Extends #1171. ## Type of Change - [x] Bug fix (CJK passed through near-uncompressed) - [x] New feature (CJK segmentation / relevance support) - [x] Performance improvement (CJK now compresses; ICU segmenters cached, not rebuilt per call) ## Changes Made - `is_cjk` predicate gates a CJK path (ideographs, kana, Hangul, CJK punctuation, full/half-width forms). - `split_segments` → ICU `SentenceSegmenter` for CJK + a mandatory length fallback (whitespace / CJK punctuation / hard cap) for terminator-sparse runs; ASCII path unchanged. - `tokens` → ICU `WordSegmenter` (dictionary) for CJK; ASCII path unchanged. - `relevance_cjk`: a local BM25 over ICU word tokens — the shared ASCII `BM25Scorer` scores zero terms for CJK and is parity-locked, so this is an intentional separate scorer (documented in code). - CJK salience uses ICU tokens (whitespace-split gave one giant "word" → zero salience). - `count_tokens`: CJK-aware so `compression_ratio` isn't nonsense for space-free text. - ICU segmenters resolved once in `static LazyLock` (compiled_data is static) instead of rebuilt per call. - New dep `icu_segmenter` 2.2, `compiled_data` only (see Dependency below). - `benchmarks/i18n_compression_eval.py` + `tests/test_transforms/test_text_crusher_cjk_eval.py`: a zh/ja/ko answer-retention eval — a deterministic needle CI gate (always-runs, no external data), real-transcript fidelity with CJK-aware salient, and optional `multi-wiki-qa` natural-data retention (loaded via the `[evals]` `datasets` extra, skipped if absent; data never vendored — CC-BY-NC-SA). ## Testing - [x] Unit tests pass (`pytest` + `cargo test`) - [x] Linting passes (`ruff check`/`format` on the new eval + test — clean) - [ ] Type checking passes (`mypy headroom`) — N/A, the only Python added is a benchmark + test, not `headroom/` source - [x] New tests added for new functionality - [x] Manual testing performed (see Real Behavior Proof) ### Test Output ```text $ cargo test -p headroom-core --lib text_crusher running 12 tests test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 841 filtered out $ .venv/bin/python -m pytest tests/test_transforms/test_text_crusher*.py 15 passed $ .venv/bin/python -m pytest tests/test_transforms/test_text_crusher_cjk_eval.py 6 passed # deterministic zh/ja/ko needle CI gate $ cargo clippy -p headroom-core && ruff check benchmarks/i18n_compression_eval.py # both clean ``` ## Real Behavior Proof - Environment: macOS (Darwin 25.3.0), Python in a uv venv, `headroom-core` built via `uv pip install -e .` (maturin), branch `feat/cjk-text-compression`. - Exact command / steps: built `_core`, then ran a mixed Chinese+Japanese doc (no spaces, `。` terminators) through `TextCrusher().compress(doc, "认证令牌缓存策略", 0.3)`; separately evaluated answer-retention on the public CMRC2018 Chinese QA dev set (bury the gold-answer paragraph among 25 distractors, query = the question, compress to 30%, check the gold answer survives), and end-to-end through `ContentRouter`. - Observed result: a mixed Chinese+Japanese doc compressed 189 → 78 tokens (ratio 0.41, kept 3/8 segments) with the query-relevant sentence surviving — before this change the same doc was a single segment → 100% passthrough. On the public CMRC2018 Chinese QA dev set, answer-retention under 30% compression rose 34% → 93% (multiple seeds). End-to-end through `ContentRouter` on real CJK content, aggregate savings rose 16% → 40%. Pure-ASCII (English) output stayed byte-identical (the English parity fixtures did not move). Demo terminal output: ```text ORIGINAL tokens= 189 chars=189 COMPRESS tokens= 78 ratio=0.41 segments kept 3/8 QUERY-RELEVANT sentence survived: True --- compressed output (verbatim kept CJK sentences) --- 认证令牌的缓存策略采用最近最少使用淘汰算法来管理过期条目。 请求重试使用指数退避并设置最大次数上限。 数据备份每天凌晨执行并保留最近三十天的快照。 ``` The committed eval now demonstrates this across all three CJK languages. The deterministic needle gate (in CI via `tests/test_transforms/test_text_crusher_cjk_eval.py`, 6 passed) has TextCrusher keep the query-relevant needle while truncate/random drop it in zh, ja, and ko. On real `multi-wiki-qa` natural data (n=80/lang), query-aware answer-retention is **zh 74% / ja 70% / ko 50%** vs **25–41%** for the truncate/random baselines: ```text === Part A: multi-wiki-qa answer-retention (n=80/lang, target_ratio=0.3) === lang text_crusher truncate random zh-cn 74% 25% 38% ja 70% 31% 39% ko 50% 26% 41% ``` Korean is measurably weaker (ICU has no Korean dictionary and falls back to UAX#29 word-breaking) — still well above baselines, and scoped as a follow-up. - Not tested: the live proxy HTTP path (validated at the `ContentRouter` / `TextCrusher` layer, not via a running proxy); no-space Korean (standard Korean is space-delimited and is covered); non-CJK SE-Asian scripts (out of scope). ## Dependency (per CONTRIBUTING supply-chain policy) `icu_segmenter` 2.2 (ICU4X), `features = ["compiled_data"]`: - **Why this package (vs. ourselves / existing deps):** CJK needs dictionary/UAX#29 segmentation. A hand-rolled char-bigram scored slightly worse on real data (CMRC2018 answer-retention: 92.5% ICU vs 91% bigram, 4 seeds); jieba/lindera are ZH-only or 13–207 MB dicts. ICU4X covers zh/ja/ko in one crate. The existing `unicode-segmentation` does UAX#29 only (no CJK dictionary), so it can't word-segment space-free CJK. - **Who maintains it:** the official `unicode-org` ICU4X project; active release cadence (2.2 in 2025); no known CVEs. - **Install surface:** ~13 new pure-Rust crates, no build scripts, no native code, no build/runtime network. `compiled_data` bundles locale data at compile time (hermetic). `auto`/`lstm` deliberately NOT enabled — LSTM covers SE-Asian scripts (Thai/Lao), not CJK, and would pull in `libm` for nothing. - **Why this version:** 2.x is the stabilized ICU4X API (1.x used a different data-provider model); floored at 2.2 (Cargo.lock pins the patch) since segmenter boundaries are observable in output and bumps should be deliberate. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (CHANGELOG) - [x] My changes generate no new warnings (clippy + fmt clean) - [x] I have added tests that prove my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md ## Additional Notes - **Parity:** the shared `BM25Scorer` (byte-exact parity-locked with `headroom/relevance/bm25.py`) is untouched. `relevance_cjk` is a separate local scorer because the shared one's tokenizer is ASCII-only. The whole CJK path lives in Rust (`text_crusher.py` is a thin wrapper over `_core`), so there is no Python mirror to keep in sync; the parity fixtures stay green (only the CJK `unicode` fixture was re-recorded, intentionally; English fixtures unchanged). - **Known by-design gap (not a bug):** CJK content + a pure-ASCII query yields no token overlap, so relevance falls back to recency + salience (cross-script query matching is unsupported). - The Python added is a benchmark (`benchmarks/i18n_compression_eval.py`) plus its test, not `headroom/` runtime source — both are `ruff`-clean; `mypy headroom` is unaffected. - **License:** the optional Part A pulls `alexandrainst/multi-wiki-qa` (CC-BY-NC-SA-4.0) at run time via the `[evals]` extra and is skipped if absent — the dataset is never vendored into the repo, and the always-run CI gate (Part C) uses only our own deterministic data. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| scenarios | ||
| __init__.py | ||
| adversarial_ccr_tests.py | ||
| agent_cost_benchmark.py | ||
| bench_latency.py | ||
| bench_relevance.py | ||
| bench_transforms.py | ||
| cache_bust_trace_report.py | ||
| cache_validation_bundle.py | ||
| ccr_regression_benchmark.py | ||
| claude_session_branch_compare.py | ||
| claude_session_mode_benchmark.py | ||
| comprehensive_eval.py | ||
| compression_benchmark.py | ||
| conftest.py | ||
| dynamic_detector_benchmark.py | ||
| headroom_adversarial_benchmark.py | ||
| headroom_worst_case_benchmark.py | ||
| i18n_compression_eval.py | ||
| prefix_cache_benchmark.py | ||
| proxy_mode_benchmark.py | ||
| real_world_agent_benchmark.py | ||
| rtk_loop_learn_eval.py | ||
| run_benchmarks.py | ||
| synthetic_long_cache_suite_report.py | ||
| synthetic_token_cache_bust_report.py | ||
| text_crusher_quality_eval.py | ||