mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
985621d60e
## Description
The search compressor's relevance scorer (`score_matches`, present in
both the Rust runtime path and the Python legacy mirror) split the query
on whitespace. A spaceless CJK query therefore matched a result line
only when the WHOLE query was a literal substring of that line — partial
overlaps never boosted relevant lines, so correct matches got dropped
when the result set was over budget.
This adds CJK character bigrams to the query match set, so a longer CJK
query boosts lines that share a substring. It also fixes two latent
Rust/Python parity divergences the ASCII-only fixtures had masked:
- **Length filter**: Rust counted word length in BYTES (`w.len()`),
Python in codepoints (`len(w)`), so a CJK word crossed the `> 2`
threshold differently. Rust now uses `chars().count()`.
- **Dedup**: Rust collected words into a `Vec` (no dedup), Python into a
`set`, so a repeated query word double-counted in Rust. Rust now uses a
`BTreeSet`.
Both scorers are byte-exact now; non-CJK output is unchanged (the 53
existing tests and the parity fixtures stay green).
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `crates/headroom-core/src/transforms/search_compressor.rs` +
`headroom/transforms/search_compressor.py`: add
`is_cjk_char`/`_is_cjk_char` and `cjk_bigrams`/`_cjk_bigrams` (identical
ranges + logic), union CJK bigrams into the query match set, and align
the Rust word set to Python (`chars().count()` length, `BTreeSet`
dedup).
- `tests/test_search_compressor_cjk.py` + a Rust unit test: CJK bigram
extraction (same input/expected in both languages) and a CJK query
boosting a partially-overlapping line.
- Corrected a stale `_score_matches` docstring that referenced a
non-existent parity assertion; it now states honestly how the two sides
are pinned (test-equal for word-overlap + CJK bigrams; a few error-boost
keywords still diverge, fixed only Rust-side).
## Testing
- [x] Unit tests pass (`cargo test` + `pytest`)
- [x] Linting passes (`cargo clippy` / `cargo fmt` / `ruff` / `mypy`)
- [x] New tests added for new functionality
- [x] Manual testing performed (see Real Behavior Proof)
### Test Output
```text
$ cargo test -p headroom-core --lib search_compressor
test result: ok. 16 passed; 0 failed
$ .venv/bin/python -m pytest tests/test_search_compressor_cjk.py \
tests/test_transforms_search_compressor.py tests/test_search_compressor.py
55 passed # 2 new CJK tests + 53 existing (no regression)
```
## Real Behavior Proof
- Environment: macOS (Darwin), Rust via cargo, Python in a uv venv
(`_core` rebuilt on this branch), branch `feat/search-compressor-cjk`
off `main`.
- Exact command / steps: scored a CJK content line against a longer CJK
query whose whole form is not a substring of the line.
- Observed result: for content `src/a.py:10:认证令牌已过期需要重新登录` and query
`认证令牌缓存淘汰策略` (the whole query is NOT a substring of the line, but its
bigrams are), the line now scores `> 0` (bigrams 认证 / 证令 / 令牌 match);
before, it scored `0`. An ASCII-only line still scores `0`. All 53
existing search-compressor tests are unchanged. `cjk_bigrams("认证令牌")`
returns `{认证, 证令, 令牌}` in **both** Rust and Python.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — N/A
(internal relevance scoring)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md — N/A: internal relevance-scoring
fix, no user-facing surface change
## Additional Notes
- The two parity divergences (byte-vs-codepoint length, `Vec`-vs-`set`
dedup) were pre-existing and only reachable with non-ASCII or
repeated-word queries — the all-ASCII fixtures never exercised them.
This PR brings both sides back to byte-exact for the word-overlap +
CJK-bigram scoring. The remaining error-boost keyword divergence is
pre-existing (fixed only Rust-side in the 3e.1 port) and is now
documented in the code rather than glossed over.
36 lines
1.5 KiB
Python
36 lines
1.5 KiB
Python
"""CJK-aware relevance scoring in the search compressor.
|
|
|
|
The relevance scorer tokenized the query on whitespace, so a spaceless CJK query
|
|
matched content only when the WHOLE query was a literal substring of a line. CJK
|
|
char bigrams now let a longer CJK query boost lines that share a substring. The
|
|
Rust<->Python parity was also hardened (dedup like Python's set; char-length
|
|
filter instead of bytes). These exercise the Python legacy scorer that mirrors
|
|
Rust.
|
|
"""
|
|
|
|
from headroom.transforms.search_compressor import (
|
|
SearchCompressor,
|
|
SearchCompressorConfig,
|
|
_cjk_bigrams,
|
|
)
|
|
|
|
|
|
def test_cjk_bigrams_from_runs():
|
|
assert _cjk_bigrams("认证令牌") == {"认证", "证令", "令牌"}
|
|
assert _cjk_bigrams("hello world") == set() # ASCII -> no CJK bigrams
|
|
assert _cjk_bigrams("a认b证") == set() # isolated CJK chars -> no adjacent pair
|
|
|
|
|
|
def test_score_matches_cjk_query_bigrams_boost():
|
|
compressor = SearchCompressor(SearchCompressorConfig(boost_errors=False, context_keywords=[]))
|
|
content = "\n".join(
|
|
[
|
|
"src/a.py:10:认证令牌已过期需要重新登录",
|
|
"src/b.py:2:plain ascii content here",
|
|
]
|
|
)
|
|
parsed = compressor._parse_search_results(content)
|
|
# the whole query is NOT a substring of the content line, but its bigrams are
|
|
compressor._score_matches(parsed, "认证令牌缓存淘汰策略")
|
|
assert parsed["src/a.py"].matches[0].score > 0 # 认证/证令/令牌 bigrams match
|
|
assert parsed["src/b.py"].matches[0].score == 0
|