mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
99a8540e65
## Description Two functions in the `headroom/evals/` metric layer silently assumed ASCII, so the eval framework produced wrong numbers for CJK (Chinese/Japanese/Korean) text: - `metrics.py::tokenize` used `re.findall(r"\b\w+\b", ...)`. A space-free CJK string matches as **one** token (`"你好世界" → ["你好世界"]`), so token-F1 (`compute_f1`, which builds on `tokenize`) is all-or-nothing on whole CJK strings instead of token-level. - `core.py::CompressionEvaluator._estimate_tokens` returned `len(text)//4`. CJK is ~1–2 tokens/char, not 0.25, so CJK compression savings were under-counted ~4–6×. This fixes both: CJK runs are split into overlapping char bigrams (the same idiom #1504 uses in TextCrusher) so F1/recall are token-level, and token estimation counts CJK chars at ~1.5 tokens each. ASCII/digit behavior is unchanged. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/evals/metrics.py::tokenize` — CJK-aware: split each `\w+` token into maximal CJK / non-CJK runs; CJK runs become overlapping char bigrams (unigram if length 1); ASCII/digit runs are kept whole. - `headroom/evals/core.py::_estimate_tokens` — count CJK chars at ~1.5 tokens each, the rest at ~4 chars/token. - `tests/test_evals_cjk_tokenization.py` — new tests for both, plus ASCII-unchanged guards. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy` on the changed files) - [x] New tests added for new functionality - [x] Manual testing performed (see Real Behavior Proof) ### Test Output ```text $ .venv/bin/python -m pytest tests/test_evals_cjk_tokenization.py 5 passed $ .venv/bin/python -m pytest tests/test_evals_metrics.py tests/test_evals/ 6 passed, 2 skipped # no regression in existing F1/metrics tests $ ruff check headroom/evals/metrics.py headroom/evals/core.py # clean ``` ## Real Behavior Proof - Environment: macOS (Darwin 25.3.0), Python in a uv venv, branch `feat/evals-cjk-tokenization` off `main`. - Exact command / steps: imported `headroom.evals.metrics.tokenize` and `headroom.evals.core.CompressionEvaluator._estimate_tokens` and called them on CJK input before/after the change. - Observed result: `tokenize("数据库连接失败")` went from `["数据库连接失败"]` (1 token) to `["数据","据库","库连","连接","接失","失败"]` (6 tokens); `_estimate_tokens("数"*20)` went from `5` to `30` (was a ~6× undercount); `compute_f1("数据库连接失败", "数据库连接成功")` went from `0.0` to a partial score in `(0, 1)`. ASCII is unchanged: `tokenize("Hello, World 42") == ["hello","world","42"]` and `_estimate_tokens("x"*40) == 10`. The existing eval metrics tests stay green (6 passed, 2 skipped). - Not tested: end-to-end framework runs against a live LLM (the fix is at the metric layer; verified directly on the functions and via the existing metric tests). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation — N/A (internal eval-tooling fix, not user-facing runtime) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A: `headroom/evals/` is internal dev tooling, not user-facing runtime ## Additional Notes - No new dependencies. The bigram idiom mirrors the CJK tokenization in `TextCrusher` (#1504), keeping the two consistent. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
34 lines
1.3 KiB
Python
34 lines
1.3 KiB
Python
"""CJK correctness for the evals metric layer.
|
|
|
|
The shared eval metrics assumed ASCII: `tokenize` used `\\b\\w+\\b` (a space-free
|
|
CJK string collapses to ONE token, so token-F1 is all-or-nothing), and
|
|
`_estimate_tokens` used `len(text)//4` (CJK is ~1-2 tokens/char, not 0.25), so
|
|
CJK compression savings were reported ~4-6x wrong.
|
|
"""
|
|
|
|
from headroom.evals.core import CompressionEvaluator
|
|
from headroom.evals.metrics import compute_f1, tokenize
|
|
|
|
|
|
def test_tokenize_splits_cjk_into_units():
|
|
toks = tokenize("数据库连接失败")
|
|
assert len(toks) >= 3, f"CJK must split into multiple units, got {toks}"
|
|
|
|
|
|
def test_tokenize_ascii_unchanged():
|
|
assert tokenize("Hello, World 42") == ["hello", "world", "42"]
|
|
|
|
|
|
def test_f1_partial_credit_on_overlapping_cjk():
|
|
# two CJK strings that share most characters must score strictly between 0 and 1
|
|
f1 = compute_f1("数据库连接失败", "数据库连接成功")
|
|
assert 0.0 < f1 < 1.0, f"overlapping CJK should be partial credit, got {f1}"
|
|
|
|
|
|
def test_estimate_tokens_cjk_not_underestimated():
|
|
# 20 CJK chars: len//4 gives 5; CJK-aware should be >= ~13
|
|
assert CompressionEvaluator._estimate_tokens(None, "数" * 20) >= 13
|
|
|
|
|
|
def test_estimate_tokens_ascii_unchanged():
|
|
assert CompressionEvaluator._estimate_tokens(None, "x" * 40) == 10
|