mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
feat(text-crusher): CJK-aware segmentation + relevance via ICU (#1504)
## Description `TextCrusher` (the native extractive prose compressor added in #1171) only handled ASCII: `split_segments` split on `.!?`+whitespace and `tokens` split on whitespace/alphanumeric runs. CJK (Chinese/Japanese/Korean) has neither spaces nor ASCII terminators, so a whole CJK paragraph collapsed into **one segment / one token** — it passed through at ~0% compression, and BM25 relevance + salience scored zero terms. This makes `TextCrusher` CJK-aware. CJK-bearing content takes an ICU (`icu_segmenter`, UAX#29 sentence + dictionary word) segmentation path, with a length fallback for terminator-sparse runs, a local BM25 over the ICU word tokens, and ICU-token salience. Dispatch is on **content only**, so pure-ASCII text is byte-identical to before — the shared `BM25Scorer` and the ASCII path are untouched. It also adds a committed, reproducible answer-retention eval (`benchmarks/i18n_compression_eval.py`) with a deterministic zh/ja/ko CI regression gate, so the improvement below is permanently verifiable rather than a one-off measurement. Extends #1171. ## Type of Change - [x] Bug fix (CJK passed through near-uncompressed) - [x] New feature (CJK segmentation / relevance support) - [x] Performance improvement (CJK now compresses; ICU segmenters cached, not rebuilt per call) ## Changes Made - `is_cjk` predicate gates a CJK path (ideographs, kana, Hangul, CJK punctuation, full/half-width forms). - `split_segments` → ICU `SentenceSegmenter` for CJK + a mandatory length fallback (whitespace / CJK punctuation / hard cap) for terminator-sparse runs; ASCII path unchanged. - `tokens` → ICU `WordSegmenter` (dictionary) for CJK; ASCII path unchanged. - `relevance_cjk`: a local BM25 over ICU word tokens — the shared ASCII `BM25Scorer` scores zero terms for CJK and is parity-locked, so this is an intentional separate scorer (documented in code). - CJK salience uses ICU tokens (whitespace-split gave one giant "word" → zero salience). - `count_tokens`: CJK-aware so `compression_ratio` isn't nonsense for space-free text. - ICU segmenters resolved once in `static LazyLock` (compiled_data is static) instead of rebuilt per call. - New dep `icu_segmenter` 2.2, `compiled_data` only (see Dependency below). - `benchmarks/i18n_compression_eval.py` + `tests/test_transforms/test_text_crusher_cjk_eval.py`: a zh/ja/ko answer-retention eval — a deterministic needle CI gate (always-runs, no external data), real-transcript fidelity with CJK-aware salient, and optional `multi-wiki-qa` natural-data retention (loaded via the `[evals]` `datasets` extra, skipped if absent; data never vendored — CC-BY-NC-SA). ## Testing - [x] Unit tests pass (`pytest` + `cargo test`) - [x] Linting passes (`ruff check`/`format` on the new eval + test — clean) - [ ] Type checking passes (`mypy headroom`) — N/A, the only Python added is a benchmark + test, not `headroom/` source - [x] New tests added for new functionality - [x] Manual testing performed (see Real Behavior Proof) ### Test Output ```text $ cargo test -p headroom-core --lib text_crusher running 12 tests test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 841 filtered out $ .venv/bin/python -m pytest tests/test_transforms/test_text_crusher*.py 15 passed $ .venv/bin/python -m pytest tests/test_transforms/test_text_crusher_cjk_eval.py 6 passed # deterministic zh/ja/ko needle CI gate $ cargo clippy -p headroom-core && ruff check benchmarks/i18n_compression_eval.py # both clean ``` ## Real Behavior Proof - Environment: macOS (Darwin 25.3.0), Python in a uv venv, `headroom-core` built via `uv pip install -e .` (maturin), branch `feat/cjk-text-compression`. - Exact command / steps: built `_core`, then ran a mixed Chinese+Japanese doc (no spaces, `。` terminators) through `TextCrusher().compress(doc, "认证令牌缓存策略", 0.3)`; separately evaluated answer-retention on the public CMRC2018 Chinese QA dev set (bury the gold-answer paragraph among 25 distractors, query = the question, compress to 30%, check the gold answer survives), and end-to-end through `ContentRouter`. - Observed result: a mixed Chinese+Japanese doc compressed 189 → 78 tokens (ratio 0.41, kept 3/8 segments) with the query-relevant sentence surviving — before this change the same doc was a single segment → 100% passthrough. On the public CMRC2018 Chinese QA dev set, answer-retention under 30% compression rose 34% → 93% (multiple seeds). End-to-end through `ContentRouter` on real CJK content, aggregate savings rose 16% → 40%. Pure-ASCII (English) output stayed byte-identical (the English parity fixtures did not move). Demo terminal output: ```text ORIGINAL tokens= 189 chars=189 COMPRESS tokens= 78 ratio=0.41 segments kept 3/8 QUERY-RELEVANT sentence survived: True --- compressed output (verbatim kept CJK sentences) --- 认证令牌的缓存策略采用最近最少使用淘汰算法来管理过期条目。 请求重试使用指数退避并设置最大次数上限。 数据备份每天凌晨执行并保留最近三十天的快照。 ``` The committed eval now demonstrates this across all three CJK languages. The deterministic needle gate (in CI via `tests/test_transforms/test_text_crusher_cjk_eval.py`, 6 passed) has TextCrusher keep the query-relevant needle while truncate/random drop it in zh, ja, and ko. On real `multi-wiki-qa` natural data (n=80/lang), query-aware answer-retention is **zh 74% / ja 70% / ko 50%** vs **25–41%** for the truncate/random baselines: ```text === Part A: multi-wiki-qa answer-retention (n=80/lang, target_ratio=0.3) === lang text_crusher truncate random zh-cn 74% 25% 38% ja 70% 31% 39% ko 50% 26% 41% ``` Korean is measurably weaker (ICU has no Korean dictionary and falls back to UAX#29 word-breaking) — still well above baselines, and scoped as a follow-up. - Not tested: the live proxy HTTP path (validated at the `ContentRouter` / `TextCrusher` layer, not via a running proxy); no-space Korean (standard Korean is space-delimited and is covered); non-CJK SE-Asian scripts (out of scope). ## Dependency (per CONTRIBUTING supply-chain policy) `icu_segmenter` 2.2 (ICU4X), `features = ["compiled_data"]`: - **Why this package (vs. ourselves / existing deps):** CJK needs dictionary/UAX#29 segmentation. A hand-rolled char-bigram scored slightly worse on real data (CMRC2018 answer-retention: 92.5% ICU vs 91% bigram, 4 seeds); jieba/lindera are ZH-only or 13–207 MB dicts. ICU4X covers zh/ja/ko in one crate. The existing `unicode-segmentation` does UAX#29 only (no CJK dictionary), so it can't word-segment space-free CJK. - **Who maintains it:** the official `unicode-org` ICU4X project; active release cadence (2.2 in 2025); no known CVEs. - **Install surface:** ~13 new pure-Rust crates, no build scripts, no native code, no build/runtime network. `compiled_data` bundles locale data at compile time (hermetic). `auto`/`lstm` deliberately NOT enabled — LSTM covers SE-Asian scripts (Thai/Lao), not CJK, and would pull in `libm` for nothing. - **Why this version:** 2.x is the stabilized ICU4X API (1.x used a different data-provider model); floored at 2.2 (Cargo.lock pins the patch) since segmenter boundaries are observable in output and bumps should be deliberate. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (CHANGELOG) - [x] My changes generate no new warnings (clippy + fmt clean) - [x] I have added tests that prove my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md ## Additional Notes - **Parity:** the shared `BM25Scorer` (byte-exact parity-locked with `headroom/relevance/bm25.py`) is untouched. `relevance_cjk` is a separate local scorer because the shared one's tokenizer is ASCII-only. The whole CJK path lives in Rust (`text_crusher.py` is a thin wrapper over `_core`), so there is no Python mirror to keep in sync; the parity fixtures stay green (only the CJK `unicode` fixture was re-recorded, intentionally; English fixtures unchanged). - **Known by-design gap (not a bug):** CJK content + a pure-ASCII query yields no token overlap, so relevance falls back to recency + salience (cross-script query matching is unsupported). - The Python added is a benchmark (`benchmarks/i18n_compression_eval.py`) plus its test, not `headroom/` runtime source — both are `ruff`-clean; `mypy headroom` is unaffected. - **License:** the optional Part A pulls `alexandrainst/multi-wiki-qa` (CC-BY-NC-SA-4.0) at run time via the `[evals]` extra and is skipped if absent — the dataset is never vendored into the repo, and the always-run CI gate (Part C) uses only our own deterministic data. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Zhenjia ZHOU
GitHub
parent
3757a7cef3
commit
4035c04187
7 changed files with 742 additions and 21 deletions
|
|
@ -117,6 +117,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
* **learn:** weight loops in `headroom learn`. A new loop detector (`headroom/learn/loops.py`) recognizes repeated tool-call patterns — including RTK re-fetch loops, where RTK's output truncation makes the agent re-run larger-limit variants of a *successful* command — collapses output-limit variants to one signature, measures the wasted tokens, surfaces loops as a highest-priority digest section, and weights loop guardrails above one-off rules by their measured waste. Previously loops had no special weight and a no-failure re-fetch loop was skipped entirely. Adds an RTK-loop eval (`benchmarks/rtk_loop_learn_eval.py`) that reproduces a loop, runs it through Learn, and asserts the generated guardrail ranks first and prevents re-triggering.
|
||||
* **learn:** write per-project learnings to the personal, gitignored `CLAUDE.local.md` by default instead of the team-shared `CLAUDE.md`, matching Claude Code's memory convention so machine-specific paths and tool-discovery byproducts no longer pollute the shared file. Adds a `--target` flag to override the destination (e.g. `--target CLAUDE.md` to opt back into the shared file, or any custom path), and auto-migrates a stale learned-patterns block out of an existing `CLAUDE.md` into `CLAUDE.local.md` with a warning ([#1072](https://github.com/chopratejas/headroom/issues/1072)).
|
||||
* **proxy/transforms:** take large cold-start contexts off the synchronous kompress path — the root cause behind the `compression_first_stage` 30s-timeout + leaked-thread → executor-saturation cascade ([#1171](https://github.com/chopratejas/headroom/issues/1171)). A token size-gate inside the ML boundary routes oversized text away from ModernBERT (`HEADROOM_KOMPRESS_MAX_TOKENS`); a cooperative chunk-deadline bounds any kompress run that does proceed (`HEADROOM_COMPRESSION_DEADLINE_MS`); an opt-in off-path mode forwards uncompressed immediately and compresses in a single per-process background drain so the request never blocks on ML (`HEADROOM_BACKGROUND_COMPRESSION`); and a new native `TextCrusher` — a fast deterministic extractive prose compressor in `headroom._core` that reuses the shared BM25 relevance scorer — is the fast alternative to ModernBERT for large plain text (`HEADROOM_TEXT_CRUSHER`). All default off and fail-open. On a SQuAD answer-retention eval (requires the SQuAD dev set) TextCrusher keeps ~94% of buried answers at 30% size vs ~36% for truncate/random, and runs in one O(n) pass -- sub-second where ModernBERT takes minutes (self-contained speed benchmark in `benchmarks/text_crusher_quality_eval.py`).
|
||||
* **proxy/transforms:** `TextCrusher` now compresses CJK (Chinese/Japanese/Korean) text ([#1171](https://github.com/chopratejas/headroom/issues/1171)). CJK has no spaces or ASCII sentence terminators, so the prior ASCII splitter/tokenizer collapsed a whole CJK paragraph into one segment/one token and passed it through near-uncompressed. CJK-bearing input now takes an ICU (`icu_segmenter`, UAX#29 + dictionary) sentence/word segmentation path with a local BM25 relevance over the ICU tokens; pure-ASCII text is byte-identical to before, and the shared BM25 scorer is untouched. On real CMRC2018 Chinese QA, answer-retention under compression rises from 34% to ~91%; end-to-end aggregate savings on real CJK content rise from 16% to 40%.
|
||||
* **proxy:** measure and surface rolling and current token throughput metrics (active/wall-clock input, compression, effective forward, and streamed generation) in `headroom perf` CLI and the dashboard ([#959](https://github.com/chopratejas/headroom/issues/959)).
|
||||
* **vibe:** add Mistral Vibe CLI support with `headroom wrap vibe`.
|
||||
* **proxy:** per-project savings breakdown on the dashboard for all wrapped agents — Claude Code, Codex, aider, Copilot, and Cursor ([#802](https://github.com/chopratejas/headroom/issues/802)). `headroom wrap claude`/`codex` tag requests with an `X-Headroom-Project` header (launch-directory name); `wrap aider`/`copilot`/`cursor` — whose clients cannot send custom headers — use a `/p/<name>` base-URL prefix the proxy strips. Savings are aggregated per project (persisted, schema v3 with transparent v2 migration), exposed as `savings.per_project` in `/stats` and `projects` in `/stats-history`, and shown in a Per-Project Savings dashboard table.
|
||||
|
|
|
|||
67
Cargo.lock
generated
67
Cargo.lock
generated
|
|
@ -1113,6 +1113,15 @@ version = "0.8.7"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b"
|
||||
|
||||
[[package]]
|
||||
name = "core_maths"
|
||||
version = "0.1.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "77745e017f5edba1a9c1d854f6f3a52dac8a12dd5af5d2f54aecf61e43d80d30"
|
||||
dependencies = [
|
||||
"libm",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "cpufeatures"
|
||||
version = "0.2.17"
|
||||
|
|
@ -1866,6 +1875,7 @@ dependencies = [
|
|||
"flate2",
|
||||
"hf-hub 0.4.3",
|
||||
"http 1.4.2",
|
||||
"icu_segmenter",
|
||||
"magika",
|
||||
"md-5",
|
||||
"ort",
|
||||
|
|
@ -2225,6 +2235,21 @@ dependencies = [
|
|||
"zerovec",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "icu_locale"
|
||||
version = "2.2.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d5a396343c7208121dc86e35623d3dfe19814a7613cfd14964994cdc9c9a2e26"
|
||||
dependencies = [
|
||||
"icu_collections",
|
||||
"icu_locale_core",
|
||||
"icu_locale_data",
|
||||
"icu_provider",
|
||||
"potential_utf",
|
||||
"tinystr",
|
||||
"zerovec",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "icu_locale_core"
|
||||
version = "2.2.0"
|
||||
|
|
@ -2233,11 +2258,18 @@ checksum = "92219b62b3e2b4d88ac5119f8904c10f8f61bf7e95b640d25ba3075e6cac2c29"
|
|||
dependencies = [
|
||||
"displaydoc",
|
||||
"litemap",
|
||||
"serde",
|
||||
"tinystr",
|
||||
"writeable",
|
||||
"zerovec",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "icu_locale_data"
|
||||
version = "2.2.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d5fdcc9ac77c6d74ff5cf6e65ef3181d6af32003b16fce3a77fb451d2f695993"
|
||||
|
||||
[[package]]
|
||||
name = "icu_normalizer"
|
||||
version = "2.2.0"
|
||||
|
|
@ -2286,6 +2318,8 @@ checksum = "139c4cf31c8b5f33d7e199446eff9c1e02decfc2f0eec2c8d71f65befa45b421"
|
|||
dependencies = [
|
||||
"displaydoc",
|
||||
"icu_locale_core",
|
||||
"serde",
|
||||
"stable_deref_trait",
|
||||
"writeable",
|
||||
"yoke",
|
||||
"zerofrom",
|
||||
|
|
@ -2293,6 +2327,28 @@ dependencies = [
|
|||
"zerovec",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "icu_segmenter"
|
||||
version = "2.2.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "5c0794db0b1a86193ac9c48768d0e6c52c54448e0870ad87907d456ee0dac964"
|
||||
dependencies = [
|
||||
"core_maths",
|
||||
"icu_collections",
|
||||
"icu_locale",
|
||||
"icu_provider",
|
||||
"icu_segmenter_data",
|
||||
"potential_utf",
|
||||
"utf8_iter",
|
||||
"zerovec",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "icu_segmenter_data"
|
||||
version = "2.2.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e4a2c462a4d927d512f5f882a033ddd62f33a05bb9f230d98f736ac3dc85938f"
|
||||
|
||||
[[package]]
|
||||
name = "ident_case"
|
||||
version = "1.0.1"
|
||||
|
|
@ -2522,6 +2578,12 @@ dependencies = [
|
|||
"windows-link",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "libm"
|
||||
version = "0.2.16"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981"
|
||||
|
||||
[[package]]
|
||||
name = "libredox"
|
||||
version = "0.1.17"
|
||||
|
|
@ -3106,6 +3168,8 @@ version = "0.1.5"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "0103b1cef7ec0cf76490e969665504990193874ea05c85ff9bab8b911d0a0564"
|
||||
dependencies = [
|
||||
"serde_core",
|
||||
"writeable",
|
||||
"zerovec",
|
||||
]
|
||||
|
||||
|
|
@ -4215,6 +4279,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
|||
checksum = "c8323304221c2a851516f22236c5722a72eaa19749016521d6dff0824447d96d"
|
||||
dependencies = [
|
||||
"displaydoc",
|
||||
"serde_core",
|
||||
"zerovec",
|
||||
]
|
||||
|
||||
|
|
@ -5318,6 +5383,7 @@ dependencies = [
|
|||
"displaydoc",
|
||||
"yoke",
|
||||
"zerofrom",
|
||||
"zerovec",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
@ -5326,6 +5392,7 @@ version = "0.11.6"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "90f911cbc359ab6af17377d242225f4d75119aec87ea711a880987b18cd7b239"
|
||||
dependencies = [
|
||||
"serde",
|
||||
"yoke",
|
||||
"zerofrom",
|
||||
"zerovec-derive",
|
||||
|
|
|
|||
282
benchmarks/i18n_compression_eval.py
Normal file
282
benchmarks/i18n_compression_eval.py
Normal file
|
|
@ -0,0 +1,282 @@
|
|||
#!/usr/bin/env python3
|
||||
"""i18n compression-quality eval (zh/ja/ko): does extractive compression keep
|
||||
the answer-bearing content in CJK? No LLM/API calls -- fully local.
|
||||
|
||||
Part C -- our own DETERMINISTIC needle answer-retention (zh/ja/ko): the always-
|
||||
runs regression gate. A distinctive needle sentence is buried (in the middle) in
|
||||
language-matched distractor sentences; compress query-aware; assert the needle
|
||||
survives. No external data. TextCrusher (query-aware) vs truncate (keep-recent)
|
||||
vs random baselines.
|
||||
|
||||
Part B -- real-transcript fidelity with CJK-aware salient: optional, anonymized.
|
||||
|
||||
Part A -- natural-data answer-retention on alexandrainst/multi-wiki-qa
|
||||
(zh-cn/ja/ko): optional, via the [evals] datasets extra, skipped if absent.
|
||||
|
||||
Usage: python benchmarks/i18n_compression_eval.py [transcript.jsonl]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import glob
|
||||
import os
|
||||
import random
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
|
||||
from headroom.transforms.text_crusher import TextCrusher
|
||||
|
||||
_REDACT = [
|
||||
(re.compile(r"/Users/[^/\s]+"), "/Users/USER"),
|
||||
(re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.-]+\b"), "EMAIL"),
|
||||
(re.compile(r"\b(?:sk|pk|ghp|gho|xox[baprs])-[A-Za-z0-9_-]{10,}\b"), "TOKEN"),
|
||||
(re.compile(r"\b[A-Fa-f0-9]{40,}\b"), "HEX"),
|
||||
]
|
||||
# Split on ASCII and full-width CJK terminators so baselines segment CJK too.
|
||||
_SEG = re.compile(r"(?<=[.!?。！？])\s*|\n+")
|
||||
_CJK_RUN = re.compile(r"[㐀-鿿-ヿ가-]+")
|
||||
|
||||
|
||||
def anon(t: str) -> str:
|
||||
for rx, rep in _REDACT:
|
||||
t = rx.sub(rep, t)
|
||||
return t
|
||||
|
||||
|
||||
def norm(s: str) -> str:
|
||||
# CJK has no spaces; drop all whitespace so substring match is robust.
|
||||
return re.sub(r"\s+", "", s.lower())
|
||||
|
||||
|
||||
def _segs(text: str) -> list[str]:
|
||||
return [s for s in _SEG.split(text) if s.strip()]
|
||||
|
||||
|
||||
def truncate_keep_last(text: str, ratio: float) -> str:
|
||||
segs = _segs(text)
|
||||
budget = int(sum(len(s) for s in segs) * ratio)
|
||||
kept: list[str] = []
|
||||
c = 0
|
||||
for s in reversed(segs):
|
||||
if c >= budget:
|
||||
break
|
||||
kept.append(s)
|
||||
c += len(s)
|
||||
return "".join(reversed(kept))
|
||||
|
||||
|
||||
def random_keep(text: str, ratio: float, seed: int) -> str:
|
||||
segs = _segs(text)
|
||||
idx = list(range(len(segs)))
|
||||
random.Random(seed).shuffle(idx)
|
||||
budget = int(sum(len(s) for s in segs) * ratio)
|
||||
kept: set[int] = set()
|
||||
c = 0
|
||||
for i in idx:
|
||||
if c >= budget:
|
||||
break
|
||||
kept.add(i)
|
||||
c += len(segs[i])
|
||||
return "".join(segs[i] for i in sorted(kept))
|
||||
|
||||
|
||||
# --- Part C: deterministic needle retention (zh / ja / ko) ---------------------
|
||||
|
||||
# Each needle carries a distinctive verbatim KEY that must survive. Distractors
|
||||
# are generated (deterministic, distinct, topic-unrelated to the query) so the
|
||||
# haystack is large enough to FORCE real compression -- the needle only survives
|
||||
# under TextCrusher because it is query-relevant, not because of passthrough.
|
||||
_NEEDLES = {
|
||||
"zh": {
|
||||
"query": "认证令牌缓存淘汰策略",
|
||||
"key": "最近最少使用淘汰",
|
||||
"needle": "认证令牌的缓存采用最近最少使用淘汰算法来管理过期条目。",
|
||||
"distractor": lambda i: f"第{i}号监控服务器的日志显示子系统{i}今天运行平稳没有出现异常。",
|
||||
},
|
||||
"ja": {
|
||||
"query": "認証トークン キャッシュ 破棄 アルゴリズム",
|
||||
"key": "最長未使用",
|
||||
"needle": "認証トークンのキャッシュは最長未使用アルゴリズムで管理される。",
|
||||
"distractor": lambda i: (
|
||||
f"{i}番目の監視サーバーのログには{i}番のサブシステムが本日も正常に稼働したと記録されている。"
|
||||
),
|
||||
},
|
||||
"ko": {
|
||||
"query": "인증 토큰 캐시 제거 알고리즘",
|
||||
"key": "최근 최소 사용",
|
||||
"needle": "인증 토큰 캐시는 최근 최소 사용 알고리즘으로 관리된다.",
|
||||
"distractor": lambda i: (
|
||||
f"{i}번 모니터링 서버의 로그에는 {i}번 하위 시스템이 오늘도 정상 작동했다고 기록되어 있다."
|
||||
),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _haystack(spec: dict, n_distract: int = 24) -> str:
|
||||
half = n_distract // 2
|
||||
before = [spec["distractor"](i) for i in range(half)]
|
||||
after = [spec["distractor"](i) for i in range(half, n_distract)]
|
||||
# needle in the MIDDLE so keep-recent (truncate) reliably misses it.
|
||||
return "".join(before + [spec["needle"]] + after)
|
||||
|
||||
|
||||
def retention_synthetic(lang: str, ratio: float = 0.3, seed: int = 0) -> dict[str, bool]:
|
||||
spec = _NEEDLES[lang]
|
||||
hay = _haystack(spec)
|
||||
key = norm(spec["key"])
|
||||
tc = TextCrusher()
|
||||
out_tc = tc.compress(hay, spec["query"], ratio).compressed
|
||||
return {
|
||||
"text_crusher": key in norm(out_tc),
|
||||
"truncate": key in norm(truncate_keep_last(hay, ratio)),
|
||||
"random": key in norm(random_keep(hay, ratio, seed)),
|
||||
}
|
||||
|
||||
|
||||
def eval_synthetic(ratio: float = 0.3) -> None:
|
||||
print(f"\n=== Part C: synthetic needle retention (zh/ja/ko, target_ratio={ratio}) ===")
|
||||
print(f" {'lang':5} {'text_crusher':>13} {'truncate':>9} {'random':>7}")
|
||||
for lang in ("zh", "ja", "ko"):
|
||||
r = retention_synthetic(lang, ratio)
|
||||
print(
|
||||
f" {lang:5} {str(r['text_crusher']):>13} {str(r['truncate']):>9} {str(r['random']):>7}"
|
||||
)
|
||||
print(" (needle must survive under TextCrusher; baselines are the contrast)")
|
||||
|
||||
|
||||
# --- Part B: real CJK transcript fidelity (CJK-aware salient) ------------------
|
||||
|
||||
# ASCII salient (identifiers/numbers/errors) STILL matters in CJK coding context.
|
||||
_SALIENT_ASCII = re.compile(
|
||||
r"\b(?:error|exception|fail(?:ed|ure)?|warning|traceback|assert|todo|fixme)\b"
|
||||
r"|\b[A-Z]{2,}\b|\b[A-Za-z_][A-Za-z0-9_]*\.[A-Za-z_][A-Za-z0-9_]*\b|\b\d+\b"
|
||||
)
|
||||
|
||||
|
||||
def _cjk_hapax(text: str) -> set[str]:
|
||||
# distinctive CJK content = char-bigrams occurring exactly once (rare = must-keep)
|
||||
grams: dict[str, int] = {}
|
||||
for run in _CJK_RUN.findall(text):
|
||||
for i in range(len(run) - 1):
|
||||
g = run[i : i + 2]
|
||||
grams[g] = grams.get(g, 0) + 1
|
||||
return {g for g, c in grams.items() if c == 1}
|
||||
|
||||
|
||||
def salient_set(text: str) -> set[str]:
|
||||
return set(_SALIENT_ASCII.findall(text)) | _cjk_hapax(text)
|
||||
|
||||
|
||||
def _block_texts(jsonl_path: str, min_chars: int, limit: int) -> list[str]:
|
||||
import json
|
||||
|
||||
out: list[str] = []
|
||||
with open(jsonl_path, encoding="utf-8") as fh:
|
||||
for line in fh:
|
||||
try:
|
||||
o = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
c = (o.get("message") or {}).get("content")
|
||||
parts = (
|
||||
[c]
|
||||
if isinstance(c, str)
|
||||
else [
|
||||
p["text"] for p in c if isinstance(p, dict) and isinstance(p.get("text"), str)
|
||||
]
|
||||
if isinstance(c, list)
|
||||
else []
|
||||
)
|
||||
for t in parts:
|
||||
if len(t) >= min_chars and _CJK_RUN.search(t): # CJK-bearing only
|
||||
out.append(anon(t))
|
||||
if len(out) >= limit:
|
||||
break
|
||||
return out[:limit]
|
||||
|
||||
|
||||
def eval_transcript(
|
||||
jsonl_path: str, ratio: float = 0.4, min_chars: int = 600, limit: int = 40
|
||||
) -> None:
|
||||
blocks = _block_texts(jsonl_path, min_chars, limit)
|
||||
if not blocks:
|
||||
print(
|
||||
f"\n=== Part B: no CJK blocks >= {min_chars} chars in {os.path.basename(jsonl_path)} ==="
|
||||
)
|
||||
return
|
||||
tc = TextCrusher()
|
||||
ratios: list[float] = []
|
||||
times: list[float] = []
|
||||
retentions: list[float] = []
|
||||
for b in blocks:
|
||||
sal_before = salient_set(b)
|
||||
t0 = time.perf_counter()
|
||||
out = tc.compress(b, "", ratio).compressed
|
||||
times.append((time.perf_counter() - t0) * 1000)
|
||||
retentions.append(len(sal_before & salient_set(out)) / max(1, len(sal_before)))
|
||||
ratios.append(len(out) / max(1, len(b)))
|
||||
n = len(blocks)
|
||||
print(
|
||||
f"\n=== Part B: real CJK transcript fidelity (n={n}, anonymized, target_ratio={ratio}) ==="
|
||||
)
|
||||
print(f" mean char-ratio kept: {sum(ratios) / n:.2f}")
|
||||
print(f" mean speed: {sum(times) / n:.1f} ms/block")
|
||||
print(f" CJK-aware salient retention: {sum(retentions) / n:.1%}")
|
||||
|
||||
|
||||
# --- Part A: optional natural-data retention (multi-wiki-qa zh/ja/ko) ----------
|
||||
# Schema verified: row = {id, title, context, question, answers:{text:[...]}}.
|
||||
# Answers are guaranteed verbatim substrings of the (long) context; CC-BY-NC-SA.
|
||||
|
||||
|
||||
def eval_multiwiki(
|
||||
langs=("zh-cn", "ja", "ko"), n: int = 80, ratio: float = 0.3, seed: int = 0
|
||||
) -> None:
|
||||
try:
|
||||
from datasets import load_dataset
|
||||
except ImportError:
|
||||
print(
|
||||
"\n=== Part A: `datasets` not installed; skipping (pip install headroom-ai[evals]) ==="
|
||||
)
|
||||
return
|
||||
tc = TextCrusher()
|
||||
print(f"\n=== Part A: multi-wiki-qa answer-retention (n={n}/lang, target_ratio={ratio}) ===")
|
||||
print(f" {'lang':6} {'text_crusher':>13} {'truncate':>9} {'random':>7}")
|
||||
for lang in langs:
|
||||
try:
|
||||
ds = load_dataset("alexandrainst/multi-wiki-qa", lang, split=f"train[:{n * 2}]")
|
||||
except Exception as e: # noqa: BLE001 -- optional path, fail-open
|
||||
print(f" {lang}: load failed ({e}); skipping")
|
||||
continue
|
||||
ex = []
|
||||
for r in ds:
|
||||
ans = r.get("answers")
|
||||
a = ans["text"][0] if isinstance(ans, dict) and ans.get("text") else None
|
||||
if r.get("context") and r.get("question") and a:
|
||||
ex.append((r["context"], r["question"], a))
|
||||
random.Random(seed).shuffle(ex)
|
||||
ex = ex[:n]
|
||||
hit = {"text_crusher": 0, "truncate": 0, "random": 0}
|
||||
for ctx, q, ans in ex:
|
||||
a = norm(ans)
|
||||
hit["text_crusher"] += a in norm(tc.compress(ctx, q, ratio).compressed)
|
||||
hit["truncate"] += a in norm(truncate_keep_last(ctx, ratio))
|
||||
hit["random"] += a in norm(random_keep(ctx, ratio, seed))
|
||||
m = max(1, len(ex))
|
||||
print(
|
||||
f" {lang:6} {hit['text_crusher'] / m:>12.0%} {hit['truncate'] / m:>9.0%} {hit['random'] / m:>7.0%}"
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
eval_synthetic()
|
||||
tx = sys.argv[1] if len(sys.argv) > 1 else None
|
||||
if tx is None:
|
||||
found = glob.glob(os.path.expanduser("~/.claude/projects/*headroom*/*.jsonl"))
|
||||
tx = max(found, key=os.path.getsize) if found else None
|
||||
if tx and os.path.exists(tx):
|
||||
eval_transcript(tx)
|
||||
else:
|
||||
print("\nno transcript jsonl found; skipping Part B")
|
||||
eval_multiwiki()
|
||||
|
|
@ -37,6 +37,22 @@ dashmap = "6"
|
|||
# `regex` is already a transitive dep of tokenizers; depend on it directly so
|
||||
# our hunk-header parser and priority-pattern matcher have a stable surface.
|
||||
regex = "1"
|
||||
# CJK sentence + word segmentation for TextCrusher (#1171). CJK has no spaces or
|
||||
# ASCII terminators, so the default ASCII splitter/tokenizer collapses a whole
|
||||
# CJK paragraph into one segment/one token -> 0% compression. ICU4X's UAX#29
|
||||
# sentence + dictionary word segmenters fix this. Chosen over a hand-rolled
|
||||
# char-bigram (benchmarked on real CMRC2018 Chinese QA: 92.5% vs 91% answer-
|
||||
# retention) and over jieba/lindera (ZH-only / tens-to-hundreds of MB dicts).
|
||||
# - Why this version: 2.x is the stabilized ICU4X API (1.x used a different data
|
||||
# provider model); floored at 2.2 (Cargo.lock pins the exact patch) since
|
||||
# segmenter boundaries are observable in output -- bumps should be deliberate.
|
||||
# - Install surface: ~13 new crates, all pure Rust, no build scripts, no native
|
||||
# code, no build/runtime network. `compiled_data` bundles locale data at
|
||||
# compile time (hermetic). Maintained by the official unicode-org.
|
||||
# - `compiled_data` only (no `auto`/`lstm`): LSTM models cover SE-Asian scripts
|
||||
# (Thai/Lao), not CJK -- CJK uses the dictionary, so `auto` would pull in libm
|
||||
# for nothing. Required for CJK; pure-ASCII paths are unchanged.
|
||||
icu_segmenter = { version = "2.2", features = ["compiled_data"] }
|
||||
# `flate2` for `_validate_with_zlib` in `adaptive_sizer`. Python's adaptive
|
||||
# sizing pipeline uses `zlib.compress(..., level=1)` to validate the chosen
|
||||
# K against compression-ratio diversity. We use the default `miniz_oxide`
|
||||
|
|
|
|||
|
|
@ -6,15 +6,58 @@
|
|||
//! target ratio. Output is extractive: the kept sentences are verbatim words
|
||||
//! (each segment trimmed, re-joined with `\n`) -- no invented words, no rewrite.
|
||||
//!
|
||||
//! The relevance term REUSES the shared [`BM25Scorer`](crate::relevance) rather
|
||||
//! than reimplementing BM25 -- only the prose-specific splitting + selection
|
||||
//! lives here.
|
||||
//! The relevance term REUSES the shared [`BM25Scorer`](crate::relevance) for
|
||||
//! non-CJK text. CJK has no spaces/ASCII terminators, so CJK-bearing input takes
|
||||
//! an ICU (UAX#29 + dictionary) sentence/word segmentation path with a local
|
||||
//! BM25 over the ICU tokens; pure-ASCII text is byte-identical to before.
|
||||
|
||||
use std::cmp::Ordering;
|
||||
use std::collections::HashSet;
|
||||
use std::sync::LazyLock;
|
||||
|
||||
use super::config::TextCrusherConfig;
|
||||
use crate::relevance::{BM25Scorer, RelevanceScorer};
|
||||
use icu_segmenter::{
|
||||
SentenceSegmenter, SentenceSegmenterBorrowed, WordSegmenter, WordSegmenterBorrowed,
|
||||
};
|
||||
|
||||
// ICU segmenters resolved ONCE and reused (compiled_data is static, so the
|
||||
// borrowed view is 'static). A fresh segmenter per call would dominate this
|
||||
// compressor's request-path budget -- compress() tokenizes every segment.
|
||||
static SENTENCE_SEGMENTER: LazyLock<SentenceSegmenterBorrowed<'static>> =
|
||||
LazyLock::new(|| SentenceSegmenter::new(Default::default()));
|
||||
static WORD_SEGMENTER: LazyLock<WordSegmenterBorrowed<'static>> =
|
||||
LazyLock::new(|| WordSegmenter::new_dictionary(Default::default()));
|
||||
|
||||
/// True for CJK ideographs, kana, Hangul, plus CJK punctuation (。、「」) and
|
||||
/// half/full-width forms — scripts/marks without ASCII spaces or terminators,
|
||||
/// which the default ASCII splitter/tokenizer can't segment. CJK-bearing text
|
||||
/// takes the ICU path; pure-ASCII text is byte-identical to before.
|
||||
fn is_cjk(c: char) -> bool {
|
||||
matches!(
|
||||
c as u32,
|
||||
0x3000..=0x303F // CJK symbols & punctuation (。、「」【】)
|
||||
| 0x3040..=0x30FF // Hiragana + Katakana
|
||||
| 0x3400..=0x4DBF // CJK Ext A
|
||||
| 0x4E00..=0x9FFF // CJK Unified
|
||||
| 0xAC00..=0xD7AF // Hangul syllables
|
||||
| 0xF900..=0xFAFF // CJK Compatibility ideographs
|
||||
| 0xFF00..=0xFFEF // half/full-width forms (！？ ｶﾅ)
|
||||
| 0x20000..=0x2FA1F // CJK Ext B–F + Compat Supplement
|
||||
)
|
||||
}
|
||||
|
||||
/// Token count for the reported ratio: ASCII whitespace words for non-CJK
|
||||
/// (unchanged), CJK-aware tokens when CJK is present. Whitespace-splitting would
|
||||
/// count a space-free CJK string as ONE token, making compression_ratio nonsense
|
||||
/// (a newline-joined N-segment output looks like N tokens vs a 1-token input).
|
||||
fn count_tokens(s: &str) -> usize {
|
||||
if s.chars().any(is_cjk) {
|
||||
tokens(s).len()
|
||||
} else {
|
||||
s.split_whitespace().count()
|
||||
}
|
||||
}
|
||||
|
||||
const KEYWORDS: [&str; 10] = [
|
||||
"error",
|
||||
|
|
@ -59,7 +102,7 @@ impl TextCrusher {
|
|||
}
|
||||
|
||||
fn passthrough(content: &str, n_segments: usize) -> TextCrusherResult {
|
||||
let toks = content.split_whitespace().count();
|
||||
let toks = count_tokens(content);
|
||||
TextCrusherResult {
|
||||
compressed: content.to_string(),
|
||||
original_tokens: toks,
|
||||
|
|
@ -90,19 +133,37 @@ impl TextCrusher {
|
|||
// admit nothing and silently fall back to a 100% passthrough).
|
||||
let target_chars = ((total_chars as f64 * ratio) as usize).max(1);
|
||||
|
||||
// Relevance via the shared BM25 scorer (already [0, 1]).
|
||||
let seg_refs: Vec<&str> = segments.iter().map(|s| s.as_str()).collect();
|
||||
let relevance = self.scorer.score_batch(&seg_refs, context);
|
||||
|
||||
let seg_tokens: Vec<Vec<String>> = segments.iter().map(|s| tokens(s)).collect();
|
||||
|
||||
// CJK content: relevance via a local BM25 over the ICU word tokens (the
|
||||
// shared ASCII BM25Scorer scores zero terms for CJK). Dispatch on the
|
||||
// CONTENT only -- pure-ASCII content keeps the shared scorer even when
|
||||
// the query is CJK, so English output stays byte-identical.
|
||||
let relevance: Vec<f64> = if segments.iter().any(|s| s.chars().any(is_cjk)) {
|
||||
relevance_cjk(&seg_tokens, context)
|
||||
} else {
|
||||
let seg_refs: Vec<&str> = segments.iter().map(|s| s.as_str()).collect();
|
||||
self.scorer
|
||||
.score_batch(&seg_refs, context)
|
||||
.iter()
|
||||
.map(|r| r.score)
|
||||
.collect()
|
||||
};
|
||||
|
||||
let mut scores = vec![0.0f64; n];
|
||||
for i in 0..n {
|
||||
let recency = (i as f64 + 1.0) / n as f64;
|
||||
let rel = relevance.get(i).map(|r| r.score).unwrap_or(0.0);
|
||||
let words: Vec<&str> = segments[i].split_whitespace().collect();
|
||||
let salient = words.iter().filter(|w| is_salient(w)).count();
|
||||
let salience = salient as f64 / (words.len() as f64 + 1.0);
|
||||
let rel = relevance.get(i).copied().unwrap_or(0.0);
|
||||
// CJK segments have no spaces, so split_whitespace yields one giant
|
||||
// "word" and zero salience; use the already-computed ICU tokens.
|
||||
let (salient, word_count) = if segments[i].chars().any(is_cjk) {
|
||||
let s = seg_tokens[i].iter().filter(|w| is_salient(w)).count();
|
||||
(s, seg_tokens[i].len())
|
||||
} else {
|
||||
let words: Vec<&str> = segments[i].split_whitespace().collect();
|
||||
(words.iter().filter(|w| is_salient(w)).count(), words.len())
|
||||
};
|
||||
let salience = salient as f64 / (word_count as f64 + 1.0);
|
||||
let mut score =
|
||||
cfg.w_recency * recency + cfg.w_relevance * rel + cfg.w_salience * salience;
|
||||
if segments[i].len() < cfg.min_segment_chars {
|
||||
|
|
@ -153,8 +214,8 @@ impl TextCrusher {
|
|||
.map(|i| segments[i].as_str())
|
||||
.collect::<Vec<_>>()
|
||||
.join("\n");
|
||||
let orig_tok = content.split_whitespace().count();
|
||||
let comp_tok = compressed.split_whitespace().count();
|
||||
let orig_tok = count_tokens(content);
|
||||
let comp_tok = count_tokens(&compressed);
|
||||
TextCrusherResult {
|
||||
compression_ratio: if orig_tok > 0 {
|
||||
comp_tok as f64 / orig_tok as f64
|
||||
|
|
@ -170,9 +231,18 @@ impl TextCrusher {
|
|||
}
|
||||
}
|
||||
|
||||
/// Split into sentence/line segments: on newlines, and after `.`/`!`/`?`
|
||||
/// followed by whitespace. Byte-faithful (kept segments are joined verbatim).
|
||||
/// Sentence/line segmentation, dispatched on content: ICU for CJK-bearing text,
|
||||
/// the original ASCII splitter otherwise (byte-identical to before).
|
||||
fn split_segments(text: &str) -> Vec<String> {
|
||||
if text.chars().any(is_cjk) {
|
||||
split_segments_icu(text)
|
||||
} else {
|
||||
split_segments_ascii(text)
|
||||
}
|
||||
}
|
||||
|
||||
/// ASCII path (unchanged): on newlines, and after `.`/`!`/`?` + whitespace.
|
||||
fn split_segments_ascii(text: &str) -> Vec<String> {
|
||||
let mut segs = Vec::new();
|
||||
for line in text.split('\n') {
|
||||
let trimmed = line.trim();
|
||||
|
|
@ -202,7 +272,79 @@ fn split_segments(text: &str) -> Vec<String> {
|
|||
segs
|
||||
}
|
||||
|
||||
/// CJK path: ICU/UAX#29 sentence boundaries per line, then the length fallback.
|
||||
fn split_segments_icu(text: &str) -> Vec<String> {
|
||||
let seg = *SENTENCE_SEGMENTER;
|
||||
let mut out = Vec::new();
|
||||
for line in text.split('\n') {
|
||||
let trimmed = line.trim();
|
||||
if trimmed.is_empty() {
|
||||
continue;
|
||||
}
|
||||
let mut prev = 0usize;
|
||||
for b in seg.segment_str(trimmed) {
|
||||
if b > prev {
|
||||
let s = trimmed[prev..b].trim();
|
||||
if !s.is_empty() {
|
||||
out.push(s.to_string());
|
||||
}
|
||||
prev = b;
|
||||
}
|
||||
}
|
||||
if prev < trimmed.len() {
|
||||
let s = trimmed[prev..].trim();
|
||||
if !s.is_empty() {
|
||||
out.push(s.to_string());
|
||||
}
|
||||
}
|
||||
}
|
||||
apply_length_fallback(out)
|
||||
}
|
||||
|
||||
/// Mandatory terminator-sparse fallback: split any over-long CJK-bearing segment
|
||||
/// on whitespace / CJK secondary punctuation, then a hard char cap.
|
||||
fn apply_length_fallback(segs: Vec<String>) -> Vec<String> {
|
||||
let cap = 60usize;
|
||||
let hard = 40usize;
|
||||
let mut out = Vec::new();
|
||||
for s in segs {
|
||||
if s.chars().count() <= cap || !s.chars().any(is_cjk) {
|
||||
out.push(s);
|
||||
continue;
|
||||
}
|
||||
let mut piece = String::new();
|
||||
for c in s.chars() {
|
||||
piece.push(c);
|
||||
let n = piece.chars().count();
|
||||
let soft = c.is_whitespace() || matches!(c, '、' | '，' | '；' | '：' | '·' | '…');
|
||||
if (soft && n >= hard / 2) || n >= hard {
|
||||
let t = piece.trim();
|
||||
if !t.is_empty() {
|
||||
out.push(t.to_string());
|
||||
}
|
||||
piece.clear();
|
||||
}
|
||||
}
|
||||
let t = piece.trim();
|
||||
if !t.is_empty() {
|
||||
out.push(t.to_string());
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Word-unit tokenization for shingles/relevance, dispatched on content: ICU
|
||||
/// word segmentation for CJK, the original ASCII alnum-run tokenizer otherwise.
|
||||
fn tokens(text: &str) -> Vec<String> {
|
||||
if text.chars().any(is_cjk) {
|
||||
tokens_icu(text)
|
||||
} else {
|
||||
tokens_ascii(text)
|
||||
}
|
||||
}
|
||||
|
||||
/// ASCII path (unchanged): lowercased alphanumeric/underscore runs.
|
||||
fn tokens_ascii(text: &str) -> Vec<String> {
|
||||
let mut out = Vec::new();
|
||||
let mut cur = String::new();
|
||||
for c in text.chars() {
|
||||
|
|
@ -220,6 +362,77 @@ fn tokens(text: &str) -> Vec<String> {
|
|||
out
|
||||
}
|
||||
|
||||
/// CJK path: ICU WordSegmenter (dictionary) word units; alnum-bearing, lowercased.
|
||||
fn tokens_icu(text: &str) -> Vec<String> {
|
||||
let seg = *WORD_SEGMENTER;
|
||||
let mut out = Vec::new();
|
||||
let mut prev = 0usize;
|
||||
for b in seg.segment_str(text) {
|
||||
if b > prev {
|
||||
let w = text[prev..b].trim();
|
||||
if !w.is_empty() && w.chars().any(|c| c.is_alphanumeric()) {
|
||||
out.push(w.to_lowercase());
|
||||
}
|
||||
prev = b;
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// BM25 relevance over the segments' ICU word tokens. This is INTENTIONALLY a
|
||||
/// separate scorer from the shared [`BM25Scorer`](crate::relevance): that one is
|
||||
/// parity-locked to Python and tokenizes with an ASCII-only regex, so it scores
|
||||
/// zero terms for CJK and cannot be reused here. This variant takes pre-computed
|
||||
/// ICU word tokens and uses textbook BM25 (k1=1.2, b=0.75) -- deliberately NOT
|
||||
/// BM25Scorer's ASCII-tuned k1=1.5 + long-identifier bonus, which don't transfer
|
||||
/// to CJK words. The `+1` inside the idf log keeps it non-negative; output is
|
||||
/// max-normalized to [0, 1] to match the shared scorer's range in the weighting.
|
||||
fn relevance_cjk(seg_tokens: &[Vec<String>], context: &str) -> Vec<f64> {
|
||||
use std::collections::HashMap;
|
||||
let n = seg_tokens.len();
|
||||
let qtokens: HashSet<String> = tokens(context).into_iter().collect();
|
||||
if n == 0 || qtokens.is_empty() {
|
||||
return vec![0.0; n];
|
||||
}
|
||||
let mut df: HashMap<&str, usize> = HashMap::new();
|
||||
for toks in seg_tokens {
|
||||
let uniq: HashSet<&str> = toks.iter().map(|s| s.as_str()).collect();
|
||||
for t in uniq {
|
||||
*df.entry(t).or_insert(0) += 1;
|
||||
}
|
||||
}
|
||||
let nf = n as f64;
|
||||
let idf = |t: &str| -> f64 {
|
||||
let d = *df.get(t).unwrap_or(&0) as f64;
|
||||
(((nf - d + 0.5) / (d + 0.5)) + 1.0).ln()
|
||||
};
|
||||
let (k1, b) = (1.2_f64, 0.75_f64);
|
||||
let avgdl = (seg_tokens.iter().map(|t| t.len()).sum::<usize>() as f64 / nf).max(1.0);
|
||||
let mut out = vec![0.0f64; n];
|
||||
for (i, toks) in seg_tokens.iter().enumerate() {
|
||||
let mut tf: HashMap<&str, usize> = HashMap::new();
|
||||
for t in toks {
|
||||
*tf.entry(t.as_str()).or_insert(0) += 1;
|
||||
}
|
||||
let dl = toks.len() as f64;
|
||||
let mut score = 0.0;
|
||||
for q in &qtokens {
|
||||
if let Some(&f) = tf.get(q.as_str()) {
|
||||
let f = f as f64;
|
||||
score += idf(q) * (f * (k1 + 1.0)) / (f + k1 * (1.0 - b + b * dl / avgdl));
|
||||
}
|
||||
}
|
||||
out[i] = score;
|
||||
}
|
||||
let max = out.iter().cloned().fold(0.0f64, f64::max);
|
||||
if max > 0.0 {
|
||||
for s in &mut out {
|
||||
*s /= max;
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
fn shingles(words: &[String], k: usize) -> HashSet<String> {
|
||||
let mut set = HashSet::new();
|
||||
if words.is_empty() {
|
||||
|
|
@ -314,4 +527,118 @@ mod tests {
|
|||
let r = TextCrusher::default().compress("one. two. three.", "", None);
|
||||
assert_eq!(r.compression_ratio, 1.0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cjk_splits_on_full_width_terminators() {
|
||||
let zh = "今天天气很好。我们去公园散步。然后回家吃饭。下午还要开会。晚上看电影。";
|
||||
let segs = split_segments(zh);
|
||||
assert!(
|
||||
segs.len() >= 4,
|
||||
"expected multiple CJK sentences, got {segs:?}"
|
||||
);
|
||||
for s in &segs {
|
||||
assert!(zh.contains(s.as_str()), "segment not verbatim: {s}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cjk_terminator_sparse_length_fallback() {
|
||||
// a long flowing CJK run with NO terminators must STILL split (the fallback)
|
||||
let zh = "机器学习模型从数据中学习特征并识别模式进行预测的系统会不断地调整参数\
|
||||
以最小化误差从而提升准确率这是一段很长的没有任何标点的中文用来测试兜底\
|
||||
切分是否生效以及能否产生多个段落供后续打分与去重使用确保不会整段透传";
|
||||
assert!(
|
||||
split_segments(zh).len() >= 2,
|
||||
"terminator-sparse CJK must still split into multiple segments"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cjk_tokens_not_one_giant_token() {
|
||||
// the old ASCII tokenizer collapsed a whole Han run into ONE token
|
||||
assert!(
|
||||
tokens("数据库连接失败重试三次").len() >= 3,
|
||||
"CJK run must yield multiple word-ish tokens"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cjk_relevance_keeps_query_match() {
|
||||
let needle = "认证令牌的缓存策略采用最近最少使用淘汰算法来管理过期。";
|
||||
let filler = "今天天气很好。我们去公园散步。然后回家吃饭。下午还要开会。\
|
||||
晚上看电影。明天继续工作。周末去爬山。后天有个会议。";
|
||||
let doc = format!("{filler}{needle}{filler}");
|
||||
let r = TextCrusher::default().compress(&doc, "认证令牌缓存策略", Some(0.3));
|
||||
assert!(r.compressed_tokens < r.original_tokens, "should compress");
|
||||
assert!(
|
||||
r.compressed.contains("认证令牌"),
|
||||
"query-relevant CJK sentence must survive selection: {}",
|
||||
r.compressed
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mixed_cjk_latin_keeps_ascii_terms_and_compresses() {
|
||||
let content = "系统启动失败。认证模块超时。\nERROR: connection refused at host.\n\
|
||||
数据库连接池耗尽。重试机制触发。服务降级处理完成。请检查日志。";
|
||||
let r = TextCrusher::default().compress(content, "ERROR connection", Some(0.5));
|
||||
assert!(r.compression_ratio < 1.0, "mixed content must compress");
|
||||
assert!(r.original_tokens > 5, "must not collapse CJK to one token");
|
||||
assert!(
|
||||
r.compressed.contains("ERROR"),
|
||||
"ASCII term relevant to the query must survive: {}",
|
||||
r.compressed
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn korean_tokenizes_and_splits() {
|
||||
let ko = "인증 토큰의 캐시 전략은 최근 최소 사용 알고리즘으로 관리된다。\
|
||||
세션은 만료 시간에 따라 자동으로 정리된다。";
|
||||
assert!(
|
||||
tokens(ko).len() >= 4,
|
||||
"Korean must yield multiple tokens via ICU"
|
||||
);
|
||||
assert!(split_segments(ko).len() >= 2, "Korean sentences must split");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn japanese_no_space_tokenizes_via_dictionary() {
|
||||
// Japanese has no spaces; ICU dictionary segmentation must still split
|
||||
// this into multiple word tokens (whitespace-splitting would give one).
|
||||
let ja = "認証トークンのキャッシュ戦略は最近最少使用アルゴリズムで管理される。\
|
||||
セッションは有効期限に従って自動的に整理される。";
|
||||
assert!(
|
||||
tokens(ja).len() >= 5,
|
||||
"Japanese must split into multiple ICU tokens"
|
||||
);
|
||||
assert!(
|
||||
split_segments(ja).len() >= 2,
|
||||
"Japanese sentences must split on 。"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cjk_token_count_and_ratio_are_sane() {
|
||||
let zh = "系统架构遵循微服务模式。每个服务拥有自己的数据存储和接口。".repeat(20);
|
||||
let r = TextCrusher::default().compress(&zh, "", Some(0.4));
|
||||
assert!(
|
||||
r.original_tokens > 10,
|
||||
"CJK token count must not collapse to 1"
|
||||
);
|
||||
assert!(r.compressed_tokens < r.original_tokens);
|
||||
assert!(r.compression_ratio > 0.0 && r.compression_ratio <= 1.0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ascii_content_unchanged_even_with_cjk_query() {
|
||||
// C-fix: dispatch is on CONTENT, not query. A CJK query against pure-ASCII
|
||||
// content must take the unchanged ASCII path (shared BM25Scorer).
|
||||
let content = doc(40);
|
||||
let with_ascii_q = TextCrusher::default().compress(&content, "topic 7", Some(0.3));
|
||||
let with_cjk_q = TextCrusher::default().compress(&content, "主题 七", Some(0.3));
|
||||
// ASCII content compresses identically regardless of the query script
|
||||
assert_eq!(with_ascii_q.total_segments, with_cjk_q.total_segments);
|
||||
assert!(with_cjk_q.compressed_tokens < with_cjk_q.original_tokens);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -7,12 +7,12 @@
|
|||
"target_ratio": 0.4
|
||||
},
|
||||
"output": {
|
||||
"compressed": "句子 0 描述了系统在主题 0 上的行为细节。 句子 1 描述了系统在主题 1 上的行为细节。 句子 2 描述了系统在主题 2 上的行为细节。 句子 3 描述了系统在主题 3 上的行为细节。 句子 4 描述了系统在主题 4 上的行为细节。 句子 5 描述了系统在主题 5 上的行为细节。 句子 6 描述了系统在主题 6 上的行为细节。 句子 7 描述了系统在主题 7 上的行为细节。 句子 8 描述了系统在主题 8 上的行为细节。 句子 9 描述了系统在主题 9 上的行为细节。 句子 10 描述了系统在主题 10 上的行为细节。 句子 11 描述了系统在主题 11 上的行为细节。",
|
||||
"original_tokens": 60,
|
||||
"compressed": "句子 7 描述了系统在主题 7 上的行为细节。\n句子 8 描述了系统在主题 8 上的行为细节。\n句子 9 描述了系统在主题 9 上的行为细节。\n句子 10 描述了系统在主题 10 上的行为细节。\n句子 11 描述了系统在主题 11 上的行为细节。",
|
||||
"original_tokens": 144,
|
||||
"compressed_tokens": 60,
|
||||
"compression_ratio": 1.0,
|
||||
"kept_segments": 1,
|
||||
"total_segments": 1
|
||||
"compression_ratio": 0.4166666666666667,
|
||||
"kept_segments": 5,
|
||||
"total_segments": 12
|
||||
},
|
||||
"input_sha256": "d46b3fb761b16a7771144b388721042e322655450fb9f10e80459d118aabf600"
|
||||
}
|
||||
28
tests/test_transforms/test_text_crusher_cjk_eval.py
Normal file
28
tests/test_transforms/test_text_crusher_cjk_eval.py
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
"""CI regression gate for CJK (zh/ja/ko) compression answer-retention.
|
||||
|
||||
Deterministic: a query-relevant needle buried among distractors must survive
|
||||
query-aware compression (TextCrusher) and must do at least as well as the
|
||||
keep-recent / random baselines. Guards the #1171/#1504 CJK TextCrusher path.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
# benchmarks/ is not a package on the import path by default; add the repo root.
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
|
||||
|
||||
from benchmarks.i18n_compression_eval import retention_synthetic # noqa: E402
|
||||
|
||||
|
||||
@pytest.mark.parametrize("lang", ["zh", "ja", "ko"])
|
||||
def test_cjk_needle_survives_compression(lang: str) -> None:
|
||||
r = retention_synthetic(lang, ratio=0.3, seed=0)
|
||||
assert r["text_crusher"], f"{lang}: query-relevant needle dropped by TextCrusher"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("lang", ["zh", "ja", "ko"])
|
||||
def test_text_crusher_beats_or_ties_baselines(lang: str) -> None:
|
||||
r = retention_synthetic(lang, ratio=0.3, seed=0)
|
||||
assert r["text_crusher"] >= max(r["truncate"], r["random"])
|
||||
Loading…
Add table
Add a link
Reference in a new issue