mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
3 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Abhay Singh
|
8a90523209
|
fix(transforms/adaptive-sizer): honor max_k on small-input fast path (#2319)
## Description
`compute_optimal_k` in the adaptive sizer takes a `max_k` argument
documented as "Never return more than this (None = no cap)". Every tier
honors that contract except the small-input fast path.
```python
n = len(items)
effective_max = max_k if max_k is not None else n
# Tier 1: Fast path
if n <= 8:
return n
```
The near-total-redundancy branch returns `min(k, effective_max)`, the
standard tier ends with `k = max(min_k, min(k, effective_max))`, and the
zlib validator clamps to `max_k` too. Only the `n <= 8` fast path
returns the raw item count, ignoring the cap.
So a caller that passes a tight budget on a small list gets back more
items than it asked for. For example `compute_optimal_k(items_of_len_8,
max_k=5)` returns `8`, not `5`. The downstream compressor then keeps 8
items when it budgeted for 5, over-filling whatever search/log budget
the cap represented.
## Fix
Return `min(n, effective_max)` on the fast path, matching what the other
tiers already do:
```python
if n <= 8:
return min(n, effective_max)
```
When `max_k` is `None`, `effective_max` is `n`, so `min(n, n) == n` and
the existing "return n unchanged" behavior is preserved. Only the capped
case changes.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/adaptive_sizer.py`: clamp the `n <= 8` fast path
to `effective_max` so `max_k` is honored on small inputs.
- `tests/test_adaptive_sizer.py`: add `test_small_array_respects_max_k`
asserting a small array honors a tight `max_k` and is unchanged when the
cap is loose.
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/transforms/adaptive_sizer.py tests/test_adaptive_sizer.py
All checks passed!
$ uvx ruff@0.15.17 format --check headroom/transforms/adaptive_sizer.py tests/test_adaptive_sizer.py
2 files already formatted
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/transforms/adaptive_sizer.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17` / `uvx
mypy@1.20.2`. A full `pytest` OOMs this box (ML-stack import), so I
reproduced the tier-1 logic with a dependency-free script and left the
full pytest to CI.
- Exact command / steps: ran both the OLD (`return n`) and NEW (`return
min(n, effective_max)`) fast-path logic for `n=8` across `max_k` in `{3,
5, 20, None}` in a standalone script.
- Observed result: OLD returned `8` for every case (ignoring the cap);
NEW returned `3, 5, 8, 8` respectively, matching the documented contract
and leaving the uncapped case unchanged.
- Not tested: the end-to-end search/log compressor path that supplies
`max_k`; the added unit test exercises `compute_optimal_k` directly, and
the standalone proof pins the fast-path arithmetic.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" box is unchecked because this box's
ML-stack import OOMs a local pytest run; the added test is a pure
dataclass-free check that runs under the normal CI pytest job, and the
behavior is corroborated by the standalone proof above.
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
Zhenjia ZHOU
|
8879c50dbe
|
fix(adaptive-sizer): char bigrams for spaceless CJK items (#1748)
## Description `compute_unique_bigram_curve` — the adaptive sizer's coverage-curve builder, mirrored in Rust and Python — word-splits each item on whitespace to form word bigrams. A spaceless CJK item has no whitespace, so it collapsed into one `(whole_string, "")` pseudo-bigram: the coverage curve then grew ~1 per item, the kneedle knee detector found no knee, and CJK lists under-compressed. Spaceless CJK items now use character bigrams, producing a real coverage curve. Mirrored byte-exactly in Rust and Python (identical reference-test curve values). Non-CJK items — anything whitespace-bearing or spaceless-ASCII — are byte-identical to before, so the `smart_crusher` parity fixtures are unchanged. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `crates/headroom-core/src/transforms/adaptive_sizer.rs` + `headroom/transforms/adaptive_sizer.py`: add `is_cjk_char`/`_is_cjk_char` (identical code-point ranges) and a spaceless-CJK character-bigram branch in `compute_unique_bigram_curve`. - Rust unit tests + `tests/test_adaptive_sizer.py`: CJK curve, single-char CJK, ASCII-unchanged, empty-item — the Rust and Python reference values are identical. ## Testing - [x] Unit tests pass (`cargo test` + `pytest`) - [x] Linting passes (`cargo clippy` / `cargo fmt` / `ruff` / `mypy`) - [x] New tests added for new functionality - [x] Manual testing performed (see Real Behavior Proof) ### Test Output ```text $ cargo test -p headroom-core --lib adaptive_sizer test result: ok. 35 passed; 0 failed $ .venv/bin/python -m pytest tests/test_adaptive_sizer.py 20 passed $ .venv/bin/python -m pytest -k "smart_crusher and parity" 18 passed, 6 skipped # non-CJK fixtures unchanged ``` ## Real Behavior Proof - Environment: macOS (Darwin), Rust via cargo, Python in a uv venv, branch `feat/adaptive-sizer-cjk` off `main`. - Exact command / steps: called `compute_unique_bigram_curve` on a CJK list and on ASCII lists, in both implementations. - Observed result: `compute_unique_bigram_curve(["数据库连接失败", "数据库连接成功"])` returns `[6, 8]` in **both** Rust and Python (before: ~`[1, 2]` — one pseudo-bigram per item, no coverage signal). ASCII curves are unchanged: `["the cat", "the dog", "a fish"]` → `[1, 2, 3]`. The `smart_crusher` parity suite (all-ASCII fixtures) stays green, confirming non-CJK output is byte-identical. - Byte-exact parity: the Rust reference test (`vec![6, 8]`) and the Python test (`[6, 8]`) use the same inputs and the same expected values, so the two implementations are pinned to agree. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation — N/A (internal sizing heuristic) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A: internal sizing-heuristic fix, no user-facing surface change ## Additional Notes - This is a parity-locked function (Rust and Python must agree byte-for-byte). The fix is CJK-gated, so non-CJK output is byte-identical and the `smart_crusher` parity fixtures need no re-recording. |
||
|
chopratejas
|
f582c1932a |
Diversity-aware SmartCrusher: keep unique items, compress text within
Root fix: compute_optimal_k() now scales k with content diversity using the SimHash uniqueness ratio already computed in the function. diversity ~1.0 → keep 100% of items (all unique, dropping any loses info) diversity ~0.5 → keep ~65% diversity ~0.0 → keep ~30% (same as before for repetitive data) No hardcoded RAG detection. No field name heuristics. Pure statistics — works for any JSON array regardless of source (Pinecone, Chroma, Weaviate, LangChain, custom APIs). When all items are kept (high diversity), SmartCrusher tries to compress text WITHIN each item's long string fields using Kompress (if available). Falls back gracefully when Kompress is not installed. Before: 12 unique RAG chunks → kept 2, dropped 10 (0/6 key concepts) After: 12 unique RAG chunks → kept 12, compressed within (6/6 concepts) Also adds tests/test_adaptive_sizer.py (16 tests covering high/low/moderate diversity, knee interactions, bias, caps). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |