fix(evals): CJK-aware F1 tokenization + token estimation (#1527)

## Description

Two functions in the `headroom/evals/` metric layer silently assumed
ASCII, so the eval framework produced wrong numbers for CJK
(Chinese/Japanese/Korean) text:

- `metrics.py::tokenize` used `re.findall(r"\b\w+\b", ...)`. A
space-free CJK string matches as **one** token (`"你好世界" → ["你好世界"]`), so
token-F1 (`compute_f1`, which builds on `tokenize`) is all-or-nothing on
whole CJK strings instead of token-level.
- `core.py::CompressionEvaluator._estimate_tokens` returned
`len(text)//4`. CJK is ~1–2 tokens/char, not 0.25, so CJK compression
savings were under-counted ~4–6×.

This fixes both: CJK runs are split into overlapping char bigrams (the
same idiom #1504 uses in TextCrusher) so F1/recall are token-level, and
token estimation counts CJK chars at ~1.5 tokens each. ASCII/digit
behavior is unchanged.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/evals/metrics.py::tokenize` — CJK-aware: split each `\w+`
token into maximal CJK / non-CJK runs; CJK runs become overlapping char
bigrams (unigram if length 1); ASCII/digit runs are kept whole.
- `headroom/evals/core.py::_estimate_tokens` — count CJK chars at ~1.5
tokens each, the rest at ~4 chars/token.
- `tests/test_evals_cjk_tokenization.py` — new tests for both, plus
ASCII-unchanged guards.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy` on the changed files)
- [x] New tests added for new functionality
- [x] Manual testing performed (see Real Behavior Proof)

### Test Output

```text
$ .venv/bin/python -m pytest tests/test_evals_cjk_tokenization.py
5 passed

$ .venv/bin/python -m pytest tests/test_evals_metrics.py tests/test_evals/
6 passed, 2 skipped     # no regression in existing F1/metrics tests

$ ruff check headroom/evals/metrics.py headroom/evals/core.py   # clean
```

## Real Behavior Proof

- Environment: macOS (Darwin 25.3.0), Python in a uv venv, branch
`feat/evals-cjk-tokenization` off `main`.
- Exact command / steps: imported `headroom.evals.metrics.tokenize` and
`headroom.evals.core.CompressionEvaluator._estimate_tokens` and called
them on CJK input before/after the change.
- Observed result: `tokenize("数据库连接失败")` went from `["数据库连接失败"]` (1
token) to `["数据","据库","库连","连接","接失","失败"]` (6 tokens);
`_estimate_tokens("数"*20)` went from `5` to `30` (was a ~6× undercount);
`compute_f1("数据库连接失败", "数据库连接成功")` went from `0.0` to a partial score in
`(0, 1)`. ASCII is unchanged: `tokenize("Hello, World 42") ==
["hello","world","42"]` and `_estimate_tokens("x"*40) == 10`. The
existing eval metrics tests stay green (6 passed, 2 skipped).
- Not tested: end-to-end framework runs against a live LLM (the fix is
at the metric layer; verified directly on the functions and via the
existing metric tests).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — N/A
(internal eval-tooling fix, not user-facing runtime)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md — N/A: `headroom/evals/` is
internal dev tooling, not user-facing runtime

## Additional Notes

- No new dependencies. The bigram idiom mirrors the CJK tokenization in
`TextCrusher` (#1504), keeping the two consistent.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

headroom/evals/core.py

@@ -260,8 +260,10 @@ class CompressionEvaluator:
         return f"Based on the context, {' '.join(words[:20])}..."
 
     def _estimate_tokens(self, text: str) -> int:
-        """Estimate token count (roughly 4 chars per token)."""
-        return len(text) // 4
+        """Estimate token count: ~1.5 tokens per CJK char (CJK is dense, ~1-2
+        tokens/char), ~4 chars per token otherwise."""
+        cjk = sum(1 for c in text if "　" <= c <= "鿿" or "가" <= c <= "힯")
+        return int(cjk * 1.5) + (len(text) - cjk) // 4
 
     def evaluate_case(
         self,

headroom/evals/metrics.py

@@ -22,11 +22,26 @@ def normalize_text(text: str) -> str:
     return text
 
 
+# CJK has no word spaces, so a CJK run is split into overlapping char bigrams
+# (unigram if length 1) -- this makes token-level F1/recall meaningful instead of
+# all-or-nothing on a whole-string CJK token. ASCII/digit runs are kept whole.
+_CJK = re.compile(r"[㐀-鿿぀-ヿ가-힯]")
+_CJK_OR_OTHER = re.compile(r"[㐀-鿿぀-ヿ가-힯]+|[^㐀-鿿぀-ヿ가-힯]+")
+
+
 def tokenize(text: str) -> list[str]:
-    """Simple word tokenization."""
-    # Split on whitespace and punctuation
-    tokens = re.findall(r"\b\w+\b", text.lower())
-    return tokens
+    """Word tokenization (CJK-aware: CJK runs -> overlapping char bigrams)."""
+    out: list[str] = []
+    for tok in re.findall(r"\b\w+\b", text.lower()):
+        for run in _CJK_OR_OTHER.findall(tok):
+            if _CJK.match(run):
+                if len(run) > 1:
+                    out.extend(run[i : i + 2] for i in range(len(run) - 1))
+                else:
+                    out.append(run)
+            else:
+                out.append(run)
+    return out
 
 
 def compute_exact_match(response_a: str, response_b: str) -> bool:

tests/test_evals_cjk_tokenization.py

@@ -0,0 +1,34 @@
+"""CJK correctness for the evals metric layer.
+
+The shared eval metrics assumed ASCII: `tokenize` used `\\b\\w+\\b` (a space-free
+CJK string collapses to ONE token, so token-F1 is all-or-nothing), and
+`_estimate_tokens` used `len(text)//4` (CJK is ~1-2 tokens/char, not 0.25), so
+CJK compression savings were reported ~4-6x wrong.
+"""
+
+from headroom.evals.core import CompressionEvaluator
+from headroom.evals.metrics import compute_f1, tokenize
+
+
+def test_tokenize_splits_cjk_into_units():
+    toks = tokenize("数据库连接失败")
+    assert len(toks) >= 3, f"CJK must split into multiple units, got {toks}"
+
+
+def test_tokenize_ascii_unchanged():
+    assert tokenize("Hello, World 42") == ["hello", "world", "42"]
+
+
+def test_f1_partial_credit_on_overlapping_cjk():
+    # two CJK strings that share most characters must score strictly between 0 and 1
+    f1 = compute_f1("数据库连接失败", "数据库连接成功")
+    assert 0.0 < f1 < 1.0, f"overlapping CJK should be partial credit, got {f1}"
+
+
+def test_estimate_tokens_cjk_not_underestimated():
+    # 20 CJK chars: len//4 gives 5; CJK-aware should be >= ~13
+    assert CompressionEvaluator._estimate_tokens(None, "数" * 20) >= 13
+
+
+def test_estimate_tokens_ascii_unchanged():
+    assert CompressionEvaluator._estimate_tokens(None, "x" * 40) == 10