fix(code-compressor): CJK-aware relevance-query symbol matching (#1747)

## Description

`CodeAwareCompressor` gives a code symbol a relevance "context boost"
when the query names it. The query tokenizer in
`_analyze_symbol_importance` used an ASCII-only delimiter class, so a
CJK query (no spaces, CJK punctuation) collapsed into one blob and never
matched an ASCII symbol name; the substring fallback was also gated
behind `len(name) > 3`, dropping short ASCII names glued to CJK.

This extracts the query tokenization + matching into two pure helpers,
adds CJK/full-width punctuation as delimiters, and relaxes the `len>3`
guard only for CJK queries. ASCII/English behavior is byte-identical.
`code_compressor` is pure-Python (no Rust twin, no parity fixtures).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `headroom/transforms/code_compressor.py`: add `_query_context_tokens`
(CJK/full-width punctuation + ideographic space as delimiters) and
`_symbol_in_context` (substring `len>3` guard relaxed only for CJK
queries), used by `_analyze_symbol_importance`.
- `tests/test_transforms/test_code_compressor_cjk.py`: pure-function
tests (CJK isolation, short-name relaxation, English-unchanged, empty).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for new functionality
- [x] Manual testing performed (see Real Behavior Proof)

### Test Output

```text
$ .venv/bin/python -m pytest tests/test_transforms/test_code_compressor_cjk.py
6 passed

$ .venv/bin/python -m pytest tests/test_transforms/test_code_compressor.py
35 passed, 39 skipped   # no regression (skips need the [code] tree-sitter extra)

$ ruff check / mypy headroom/transforms/code_compressor.py   # clean
```

## Real Behavior Proof

- Environment: macOS (Darwin), Python in a uv venv, branch
`feat/code-compressor-cjk-relevance` off `main`.
- Exact command / steps: called the extracted helpers directly on CJK
and ASCII queries.
- Observed result: `_query_context_tokens("请重点保留（parse_config）的解析配置")`
isolates `parse_config` as its own token (before: the whole query was
one blob, so the exact-match boost never fired);
`_symbol_in_context("db", ...)` now matches a short ASCII name glued to
a CJK query (before: dropped by the `len>3` guard). English is unchanged
— for `"keep the database helper"`, `_symbol_in_context("db", ...)`
still returns `False` (no spurious short substring match). All 6 new
tests pass; the existing 35 `code_compressor` tests are unchanged.
- Not tested: end-to-end `compress()` (needs the `[code]` tree-sitter
extra); the fix is at the pure query-matching layer and is verified
there.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — N/A
(internal compressor behavior)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md — N/A: internal relevance-scoring
fix, no user-facing surface change

## Additional Notes

- Scope note: a pure-CJK query that names the function only by a Chinese
description (no ASCII token anywhere) still cannot match an ASCII symbol
name — cross-script query matching remains out of scope.

headroom/transforms/code_compressor.py

@@ -875,9 +875,7 @@ class CodeAwareCompressor(Transform):
             ref_counts[qname] = max(0, count - short_name_def_count.get(short, 1))
 
         # Raw importance signals per symbol
-        context_lower = context.lower() if context else ""
-        context_words = set(re.split(r"[\s,;:.()\[\]{}\"']+", context_lower)) if context else set()
-        context_words.discard("")
+        context_words, context_lower, context_has_cjk = _query_context_tokens(context)
 
         raw_signals: dict[str, float] = {}
         for qname in definitions:
@@ -898,13 +896,9 @@ class CodeAwareCompressor(Transform):
                 if short and short[0].isupper():
                     raw += 1.0
 
-            # Context boost
-            if context_words:
-                name_lower = short.lower()
-                if name_lower in context_words or (
-                    len(name_lower) > 3 and name_lower in context_lower
-                ):
-                    raw += 3.0
+            # Context boost: the relevance query named this symbol.
+            if _symbol_in_context(short.lower(), context_words, context_lower, context_has_cjk):
+                raw += 3.0
 
             raw_signals[qname] = raw
 
@@ -2056,6 +2050,43 @@ def _get_definition_name(node: Any) -> str | None:
     return None
 
 
+# Symbol names are ASCII identifiers; CJK relevance queries have no spaces and use
+# CJK/full-width punctuation, so the ASCII-only delimiter class would collapse the
+# whole query into one blob and never isolate an ASCII name the user asked to keep.
+_CONTEXT_DELIMS = re.compile(r"[\s,;:.()\[\]{}\"'，、；：。．！？（）【】「」『』《》〈〉·…—　]+")
+_CJK_CHARS = re.compile(r"[　-鿿가-힯＀-￯]")
+
+
+def _query_context_tokens(context: str) -> tuple[set[str], str, bool]:
+    """Tokenize a relevance query for symbol-name matching (CJK-aware).
+
+    Returns (word set, lowercased query, has_cjk). CJK/full-width punctuation and
+    the ideographic space are delimiters so an ASCII symbol name wrapped in CJK is
+    still isolated as its own token.
+    """
+    if not context:
+        return set(), "", False
+    lowered = context.lower()
+    words = set(_CONTEXT_DELIMS.split(lowered))
+    words.discard("")
+    return words, lowered, bool(_CJK_CHARS.search(lowered))
+
+
+def _symbol_in_context(name_lower: str, words: set[str], context_lower: str, has_cjk: bool) -> bool:
+    """Whether the relevance query names this symbol.
+
+    Exact token match, or a substring fallback gated by len>3 for ASCII queries
+    (avoids spurious short-name matches) but relaxed for CJK queries -- a short
+    ASCII name glued to CJK has no delimiter to isolate it, so exact-match can't
+    fire and the guard would wrongly drop it.
+    """
+    if not words or not name_lower:
+        return False
+    if name_lower in words:
+        return True
+    return name_lower in context_lower and (len(name_lower) > 3 or has_cjk)
+
+
 def _is_public_symbol(name: str, language: CodeLanguage) -> bool:
     """Heuristic for whether a symbol is public/exported."""
     if not name:

tests/test_transforms/test_code_compressor_cjk.py

@@ -0,0 +1,58 @@
+"""CJK-aware relevance-query matching in the code compressor.
+
+The symbol-importance context boost tokenized the query with an ASCII-only
+delimiter class, so a CJK query (no spaces, CJK punctuation) collapsed into one
+blob and never isolated/matched an ASCII symbol name the user asked to keep.
+These exercise the extracted pure helpers (no tree-sitter needed).
+"""
+
+from headroom.transforms.code_compressor import (
+    _query_context_tokens,
+    _symbol_in_context,
+)
+
+
+def test_cjk_query_isolates_wrapped_ascii_symbol():
+    # full-width parens around the name must still tokenize parse_config out
+    words, lowered, has_cjk = _query_context_tokens("请重点保留（parse_config）的解析配置")
+    assert has_cjk
+    assert "parse_config" in words
+    assert _symbol_in_context("parse_config", words, lowered, has_cjk)
+
+
+def test_cjk_query_matches_short_ascii_name_glued_to_cjk():
+    # 'db' (len 2) glued to CJK has no delimiter to isolate it; the len>3 guard is
+    # relaxed for CJK so the substring fallback still matches.
+    words, lowered, has_cjk = _query_context_tokens("请保留db相关的逻辑")
+    assert has_cjk
+    assert _symbol_in_context("db", words, lowered, has_cjk)
+
+
+def test_english_short_name_substring_still_gated():
+    # ASCII query unchanged: a short name that is only a substring (not a token)
+    # of an English query must NOT match (avoids spurious boosts).
+    words, lowered, has_cjk = _query_context_tokens("keep the database helper")
+    assert not has_cjk
+    assert not _symbol_in_context("db", words, lowered, has_cjk)
+
+
+def test_english_exact_token_match_unchanged():
+    words, lowered, has_cjk = _query_context_tokens("keep parse_config and helper")
+    assert not has_cjk
+    assert _symbol_in_context("parse_config", words, lowered, has_cjk)
+    assert _symbol_in_context("helper", words, lowered, has_cjk)
+
+
+def test_english_long_name_substring_fallback_unchanged():
+    # ASCII path, len>3 substring fallback: 'parse_config' is not a standalone
+    # token but is a substring of 'parse_configs' -> must still match (unchanged).
+    words, lowered, has_cjk = _query_context_tokens("parse_configs and related helpers")
+    assert not has_cjk
+    assert "parse_config" not in words
+    assert _symbol_in_context("parse_config", words, lowered, has_cjk)
+
+
+def test_empty_context_matches_nothing():
+    words, lowered, has_cjk = _query_context_tokens("")
+    assert words == set() and lowered == "" and has_cjk is False
+    assert not _symbol_in_context("foo", words, lowered, has_cjk)