mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
844d9caaa1
## Description Real CJK content routinely mixes full-width and half-width forms (`ＡＰＩ` vs `API`, `０` vs `0`, the ideographic space ` ` vs a normal space). After #1504's ICU tokenization, these width variants produced *different* token keys, so `ＡＰＩ` and `API` didn't dedup or match as the same term on the CJK relevance and near-duplicate paths. This folds full-width ASCII (U+FF01–U+FF5E, via the U+FEE0 offset) and the ideographic space (U+3000 → space) to their half-width forms **when building the internal token key** inside `tokens_icu`. Only the token key is normalized — the kept output stays byte-verbatim, so TextCrusher's extractive / byte-faithful contract is preserved. CJK-gated (`tokens_icu` is the CJK path); the ASCII path is untouched. ## Type of Change - [x] Bug fix / enhancement (non-breaking) ## Changes Made - `crates/headroom-core/src/transforms/text_crusher/crusher.rs`: a `width_fold(c)` helper applied when building token keys in `tokens_icu`. - Rust unit tests: full-width ASCII folds to half-width in token keys; CJK segments split on full-width terminators. ## Testing - [x] Unit tests pass (`cargo test`) - [x] Linting passes (`cargo clippy` / `cargo fmt`) - [x] New tests added ### Test Output ```text $ cargo test -p headroom-core --lib text_crusher test result: ok. 13 passed; 0 failed $ cargo clippy / fmt # clean ``` ## Real Behavior Proof - Environment: macOS (Darwin), Rust via cargo, branch `feat/text-crusher-fullwidth-fold` off `main` (rebased after #1504 merged). - Exact command / steps: `cargo test -p headroom-core --lib text_crusher`. - Observed result: `fullwidth_ascii_folds_to_halfwidth` confirms `ＡＰＩ` and `API` now produce the same token key (so they dedup/relevance-match as one term); `cjk_splits_on_full_width_terminators` confirms full-width `！`/`？` terminate segments. All 13 text_crusher tests pass; the kept output is byte-verbatim (only the internal key is folded). - Not tested: no Python side — TextCrusher is Rust-only, and output stays byte-verbatim, so no parity fixtures change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation — N/A - [x] My changes generate no new warnings - [x] I have added tests that prove my change is effective - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — happy to add an entry if preferred. ## Additional Notes - Follow-up to #1504 (CJK-aware TextCrusher); it only touches the CJK-gated `tokens_icu` path, so English tokenization is unchanged. |
||
|---|---|---|
| .. | ||
| headroom-core | ||
| headroom-parity | ||
| headroom-proxy | ||
| headroom-py | ||
| headroom-simulators | ||