mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
b38315cf72
## Description
`CodeAwareCompressor` gives a code symbol a relevance "context boost"
when the query names it. The query tokenizer in
`_analyze_symbol_importance` used an ASCII-only delimiter class, so a
CJK query (no spaces, CJK punctuation) collapsed into one blob and never
matched an ASCII symbol name; the substring fallback was also gated
behind `len(name) > 3`, dropping short ASCII names glued to CJK.
This extracts the query tokenization + matching into two pure helpers,
adds CJK/full-width punctuation as delimiters, and relaxes the `len>3`
guard only for CJK queries. ASCII/English behavior is byte-identical.
`code_compressor` is pure-Python (no Rust twin, no parity fixtures).
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/transforms/code_compressor.py`: add `_query_context_tokens`
(CJK/full-width punctuation + ideographic space as delimiters) and
`_symbol_in_context` (substring `len>3` guard relaxed only for CJK
queries), used by `_analyze_symbol_importance`.
- `tests/test_transforms/test_code_compressor_cjk.py`: pure-function
tests (CJK isolation, short-name relaxation, English-unchanged, empty).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for new functionality
- [x] Manual testing performed (see Real Behavior Proof)
### Test Output
```text
$ .venv/bin/python -m pytest tests/test_transforms/test_code_compressor_cjk.py
6 passed
$ .venv/bin/python -m pytest tests/test_transforms/test_code_compressor.py
35 passed, 39 skipped # no regression (skips need the [code] tree-sitter extra)
$ ruff check / mypy headroom/transforms/code_compressor.py # clean
```
## Real Behavior Proof
- Environment: macOS (Darwin), Python in a uv venv, branch
`feat/code-compressor-cjk-relevance` off `main`.
- Exact command / steps: called the extracted helpers directly on CJK
and ASCII queries.
- Observed result: `_query_context_tokens("请重点保留（parse_config）的解析配置")`
isolates `parse_config` as its own token (before: the whole query was
one blob, so the exact-match boost never fired);
`_symbol_in_context("db", ...)` now matches a short ASCII name glued to
a CJK query (before: dropped by the `len>3` guard). English is unchanged
— for `"keep the database helper"`, `_symbol_in_context("db", ...)`
still returns `False` (no spurious short substring match). All 6 new
tests pass; the existing 35 `code_compressor` tests are unchanged.
- Not tested: end-to-end `compress()` (needs the `[code]` tree-sitter
extra); the fix is at the pure query-matching layer and is verified
there.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — N/A
(internal compressor behavior)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md — N/A: internal relevance-scoring
fix, no user-facing surface change
## Additional Notes
- Scope note: a pure-CJK query that names the function only by a Chinese
description (no ASCII token anywhere) still cannot match an ASCII symbol
name — cross-script query matching remains out of scope.
58 lines
2.5 KiB
Python
58 lines
2.5 KiB
Python
"""CJK-aware relevance-query matching in the code compressor.
|
||
|
||
The symbol-importance context boost tokenized the query with an ASCII-only
|
||
delimiter class, so a CJK query (no spaces, CJK punctuation) collapsed into one
|
||
blob and never isolated/matched an ASCII symbol name the user asked to keep.
|
||
These exercise the extracted pure helpers (no tree-sitter needed).
|
||
"""
|
||
|
||
from headroom.transforms.code_compressor import (
|
||
_query_context_tokens,
|
||
_symbol_in_context,
|
||
)
|
||
|
||
|
||
def test_cjk_query_isolates_wrapped_ascii_symbol():
|
||
# full-width parens around the name must still tokenize parse_config out
|
||
words, lowered, has_cjk = _query_context_tokens("请重点保留（parse_config）的解析配置")
|
||
assert has_cjk
|
||
assert "parse_config" in words
|
||
assert _symbol_in_context("parse_config", words, lowered, has_cjk)
|
||
|
||
|
||
def test_cjk_query_matches_short_ascii_name_glued_to_cjk():
|
||
# 'db' (len 2) glued to CJK has no delimiter to isolate it; the len>3 guard is
|
||
# relaxed for CJK so the substring fallback still matches.
|
||
words, lowered, has_cjk = _query_context_tokens("请保留db相关的逻辑")
|
||
assert has_cjk
|
||
assert _symbol_in_context("db", words, lowered, has_cjk)
|
||
|
||
|
||
def test_english_short_name_substring_still_gated():
|
||
# ASCII query unchanged: a short name that is only a substring (not a token)
|
||
# of an English query must NOT match (avoids spurious boosts).
|
||
words, lowered, has_cjk = _query_context_tokens("keep the database helper")
|
||
assert not has_cjk
|
||
assert not _symbol_in_context("db", words, lowered, has_cjk)
|
||
|
||
|
||
def test_english_exact_token_match_unchanged():
|
||
words, lowered, has_cjk = _query_context_tokens("keep parse_config and helper")
|
||
assert not has_cjk
|
||
assert _symbol_in_context("parse_config", words, lowered, has_cjk)
|
||
assert _symbol_in_context("helper", words, lowered, has_cjk)
|
||
|
||
|
||
def test_english_long_name_substring_fallback_unchanged():
|
||
# ASCII path, len>3 substring fallback: 'parse_config' is not a standalone
|
||
# token but is a substring of 'parse_configs' -> must still match (unchanged).
|
||
words, lowered, has_cjk = _query_context_tokens("parse_configs and related helpers")
|
||
assert not has_cjk
|
||
assert "parse_config" not in words
|
||
assert _symbol_in_context("parse_config", words, lowered, has_cjk)
|
||
|
||
|
||
def test_empty_context_matches_nothing():
|
||
words, lowered, has_cjk = _query_context_tokens("")
|
||
assert words == set() and lowered == "" and has_cjk is False
|
||
assert not _symbol_in_context("foo", words, lowered, has_cjk)
|