mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
4e30dde2ac
## Description
`ContentRouter` counts section tokens with `len(content.split())`. On
compact machine-generated JSON — the default output of
`json.dumps(separators=(",", ":"))`, `JSON.stringify`, and boto3 — there
are no spaces, so a large payload counts as ~1 "token". Every section
compression ratio then computes as ~1.0 and the `min_ratio` acceptance
gate silently rejects the compressor's real output: the router logs
`router:noop` while SmartCrusher separately logs `was_modified=true`.
Compression effectively no-ops on the most common agent payload type
(tool results returning JSON), on every provider.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Add `_estimate_tokens(text)` — a size-proportional estimate
(`len(text) // 4`, floored at 1), monotone in content size for any
format.
- Replace the decision-relevant `len(...split())` counts in
`ContentRouter` (section original/compressed token counts feeding the
ratio gates, plus the debug estimates) with `_estimate_tokens(...)`.
- Add `tests/test_content_router_compact_json.py`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_content_router_compact_json.py -q
2 passed, 1 warning
$ ruff check headroom/transforms/content_router.py tests/test_content_router_compact_json.py
All checks passed!
$ mypy headroom/transforms/content_router.py --ignore-missing-imports
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: `ContentRouter` invoked directly on a 150-item
ECS-service JSON tool_result, Python 3.13, estimator tokenizer.
- Exact command / steps: run the same payload two ways — compact
(`json.dumps(..., separators=(",", ":"))`) and the identical data with
spaces (`separators=(", ", ": ")`) — through
`ContentRouter(ContentRouterConfig(skip_user_messages=False))`.
- Observed result: before this change, compact JSON saved 0.0%
(`router:noop`) while the identical data with spaces saved 43.3%
(`router:tool_result:smart_crusher`) — same data, same compressor, only
whitespace differed. After this change, compact JSON compresses
equivalently to the spaced form.
- Not tested: no behavior change expected for content that already
tokenizes with whitespace (prose, code); those counts move from
word-count to chars/4 but the ratio comparison is self-consistent (both
sides use the same estimator).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
Scope kept deliberately narrow: only the counts that feed
compression-acceptance decisions are changed. Non-decision `.split()`
uses elsewhere are left alone. Happy to add a CHANGELOG entry if you'd
like one.
---------
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
41 lines
1.5 KiB
Python
41 lines
1.5 KiB
Python
"""Phase 2 (#1171): the kompress size-gate routes oversized text to TextCrusher
|
|
when HEADROOM_TEXT_CRUSHER is enabled (real prose savings), instead of the
|
|
LogCompressor (which yields ~0 on prose) or ModernBERT (slow)."""
|
|
|
|
from __future__ import annotations
|
|
|
|
from headroom.transforms.content_router import ContentRouter, _estimate_tokens
|
|
|
|
|
|
def _prose() -> str:
|
|
return " ".join(
|
|
f"Sentence {i} about distributed systems and authentication tokens expiring soon."
|
|
for i in range(300)
|
|
)
|
|
|
|
|
|
def test_gate_routes_to_text_crusher_when_enabled(monkeypatch):
|
|
monkeypatch.setenv("HEADROOM_TEXT_CRUSHER", "1")
|
|
router = ContentRouter()
|
|
router._kompress_max_tokens = 50 # tiny ceiling so the gate fires
|
|
|
|
def _boom():
|
|
raise AssertionError("kompress must not run for gated input")
|
|
|
|
monkeypatch.setattr(router, "_get_kompress", _boom)
|
|
|
|
prose = _prose()
|
|
out, ntok = router._try_ml_compressor(prose, "authentication tokens")
|
|
|
|
assert router._kompress_gate_fires == 1
|
|
# TextCrusher actually compressed (LogCompressor ~0). Compare like-for-like
|
|
# against the same token estimator the compressor reports in, not a raw
|
|
# word count, so the assertion tracks real token reduction.
|
|
assert ntok < _estimate_tokens(prose)
|
|
assert set(out.split()) <= set(prose.split()) # extractive: no invented words
|
|
|
|
|
|
def test_text_crusher_disabled_by_default(monkeypatch):
|
|
monkeypatch.delenv("HEADROOM_TEXT_CRUSHER", raising=False)
|
|
router = ContentRouter()
|
|
assert router._get_text_crusher() is None
|