## Description
Two functions in the `headroom/evals/` metric layer silently assumed
ASCII, so the eval framework produced wrong numbers for CJK
(Chinese/Japanese/Korean) text:
- `metrics.py::tokenize` used `re.findall(r"\b\w+\b", ...)`. A
space-free CJK string matches as **one** token (`"你好世界" → ["你好世界"]`), so
token-F1 (`compute_f1`, which builds on `tokenize`) is all-or-nothing on
whole CJK strings instead of token-level.
- `core.py::CompressionEvaluator._estimate_tokens` returned
`len(text)//4`. CJK is ~1–2 tokens/char, not 0.25, so CJK compression
savings were under-counted ~4–6×.
This fixes both: CJK runs are split into overlapping char bigrams (the
same idiom #1504 uses in TextCrusher) so F1/recall are token-level, and
token estimation counts CJK chars at ~1.5 tokens each. ASCII/digit
behavior is unchanged.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/evals/metrics.py::tokenize` — CJK-aware: split each `\w+`
token into maximal CJK / non-CJK runs; CJK runs become overlapping char
bigrams (unigram if length 1); ASCII/digit runs are kept whole.
- `headroom/evals/core.py::_estimate_tokens` — count CJK chars at ~1.5
tokens each, the rest at ~4 chars/token.
- `tests/test_evals_cjk_tokenization.py` — new tests for both, plus
ASCII-unchanged guards.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy` on the changed files)
- [x] New tests added for new functionality
- [x] Manual testing performed (see Real Behavior Proof)
### Test Output
```text
$ .venv/bin/python -m pytest tests/test_evals_cjk_tokenization.py
5 passed
$ .venv/bin/python -m pytest tests/test_evals_metrics.py tests/test_evals/
6 passed, 2 skipped # no regression in existing F1/metrics tests
$ ruff check headroom/evals/metrics.py headroom/evals/core.py # clean
```
## Real Behavior Proof
- Environment: macOS (Darwin 25.3.0), Python in a uv venv, branch
`feat/evals-cjk-tokenization` off `main`.
- Exact command / steps: imported `headroom.evals.metrics.tokenize` and
`headroom.evals.core.CompressionEvaluator._estimate_tokens` and called
them on CJK input before/after the change.
- Observed result: `tokenize("数据库连接失败")` went from `["数据库连接失败"]` (1
token) to `["数据","据库","库连","连接","接失","失败"]` (6 tokens);
`_estimate_tokens("数"*20)` went from `5` to `30` (was a ~6× undercount);
`compute_f1("数据库连接失败", "数据库连接成功")` went from `0.0` to a partial score in
`(0, 1)`. ASCII is unchanged: `tokenize("Hello, World 42") ==
["hello","world","42"]` and `_estimate_tokens("x"*40) == 10`. The
existing eval metrics tests stay green (6 passed, 2 skipped).
- Not tested: end-to-end framework runs against a live LLM (the fix is
at the metric layer; verified directly on the functions and via the
existing metric tests).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — N/A
(internal eval-tooling fix, not user-facing runtime)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md — N/A: `headroom/evals/` is
internal dev tooling, not user-facing runtime
## Additional Notes
- No new dependencies. The bigram idiom mirrors the CJK tokenization in
`TextCrusher` (#1504), keeping the two consistent.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
