## Description
CodeAwareCompressor was slicing Python strings with tree-sitter
`start_byte` / `end_byte` offsets directly. That works for ASCII-only
files, but it corrupts slices after non-ASCII source text such as CJK
characters or emoji because tree-sitter offsets are UTF-8 byte offsets
while Python string indexes are character offsets.
This caused code-aware compression to produce invalid intermediate
Python and then safely fall back to the original file, resulting in 0%
compression on affected files.
Closes#1319
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Added `_slice_code_bytes()` in
`headroom/transforms/code_compressor.py` to slice source text using
UTF-8 byte offsets.
- Updated `_get_node_text()` to use byte-safe slicing.
- Routed the other direct tree-sitter byte-offset slices through the
same helper.
- Added regression tests in
`tests/test_transforms/test_code_compressor.py`:
- `test_get_node_text_uses_utf8_byte_offsets`
- `test_ast_compresses_python_after_non_ascii_source`
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ .venv/bin/python -m pytest tests/test_transforms/test_code_compressor.py
68 passed, 1 warning
$ .venv/bin/python -m ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py
All checks passed!
$ .venv/bin/python -m ruff format --check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py
2 files already formatted
$ git diff --check
# no output
$ /tmp/headroom-1319-venv/bin/python -m mypy headroom
headroom/proxy/server.py:1186: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked]
headroom/proxy/server.py:1257: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked]
headroom/proxy/server.py:1261: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked]
pyproject.toml: note: unused section(s): module = ['mlx.*']
Success: no issues found in 394 source files
```
## Real Behavior Proof
- Environment: macOS arm64, Python 3.14.5, tree-sitter 0.25.2,
tree-sitter-language-pack 0.13.0
- Exact command / steps: On `main`, ran a local reproducer with a Python
source string containing a CJK docstring before a second function;
called `_get_node_text()` on the second tree-sitter function node; ran a
full `CodeAwareCompressor.compress(...)` repro with non-ASCII module
text before an import and a compressible function; re-ran both repros on
this branch.
- Observed result: Before fix, `_get_node_text()` returned the wrong
slice (`'nd():\n return 2\n'` instead of `'def second():\n return 2'`)
and full compression fell back to the original file with
`compression_ratio: 1.0`; after fix, `_get_node_text()` returns the full
expected function slice and full compression succeeds with
`compression_ratio < 1.0`, `syntax_valid: True`, and does not return the
original.
- Not tested: Full repository test suite; live proxy/provider
integrations; Windows/Linux platform-specific behavior.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
N/A
## Additional Notes
- Documentation was not updated because this is an internal bug fix with
no user-facing API or behavior change beyond restoring intended
compression.
- `CHANGELOG.md` was not updated because the fix is narrow and
issue-scoped; maintainers can advise if they want a changelog entry.
- The fix is intentionally small and targeted: it only changes how
tree-sitter byte offsets are converted back into Python source text,
without changing compression heuristics or language behavior.
## Description
CodeAwareCompressor used the same `body_node_types` config to find both
executable function bodies and class/impl member containers. That works
when those AST nodes happen to match, but it misses member containers
such as Java `class_body`, C++ `field_declaration_list`, and Rust
`declaration_list`, so class methods were returned essentially
uncompressed.
This adds an optional `class_body_node_types` override for class/impl
member containers and uses it only in class compression. It also skips
anonymous punctuation tokens while reconstructing class bodies and keeps
same-line C++ class semicolons attached to the compressed class
declaration.
Closes#1318
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Added `LangConfig.class_body_node_types` for languages whose
class/impl member container differs from executable method-body nodes.
- Configured class member containers for JavaScript, TypeScript, Java,
C++, and Rust.
- Updated `_compress_class_ast` to use class-member containers, skip
anonymous punctuation children, and preserve C++ `};` output without
creating stray top-level semicolons.
- Added regression coverage proving class/impl methods compress for
JavaScript, TypeScript, Java, C++, and Rust.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ PYTHONPATH=. /tmp/headroom-1319-venv/bin/python -m pytest tests/test_transforms/test_code_compressor.py -q
collected 71 items
tests/test_transforms/test_code_compressor.py .......................... [ 36%]
............................................. [100%]
71 passed, 1 warning in 0.36s
$ /tmp/headroom-1319-venv/bin/python -m ruff check .
All checks passed!
$ /tmp/headroom-1319-venv/bin/python -m ruff format --check .
965 files already formatted
$ PYTHONPATH=. /tmp/headroom-1319-venv/bin/python -m mypy headroom
headroom/proxy/server.py:1186: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked]
headroom/proxy/server.py:1257: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked]
headroom/proxy/server.py:1261: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked]
pyproject.toml: note: unused section(s): module = ['mlx.*']
Success: no issues found in 394 source files
```
## Real Behavior Proof
- Environment: macOS arm64, Python 3.14.5, branch
`fix-code-compressor-class-members`, tree-sitter grammar pack installed
in `/tmp/headroom-1319-venv`, repo imported with `PYTHONPATH=.`.
- Exact command / steps: Reproduced class-method compression with
`CodeAwareCompressor(CodeCompressorConfig(enable_ccr=False,
min_tokens_for_compression=1, max_body_lines=1))` for Java/C++/Rust
before the fix, then reran the pytest/ruff/mypy commands listed above
after the patch.
- Observed result: Java/C++/Rust class methods now compress below 1.0
while `syntax_valid` remains true; C++ output preserves `};`; regression
coverage also verifies JavaScript/TypeScript class member containers.
- Not tested: Full repository pytest suite; local `uv run` editable
builds are blocked on this machine by native C++ header failures in
optional/native dependencies (`hnswlib` / Rust `esaxx-rs`), so
validation used a lightweight venv with `PYTHONPATH=.`.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
N/A
## Additional Notes
Documentation and CHANGELOG updates are not applicable for this narrow
bug fix. The pytest warning shown above is from running without
`pytest-asyncio` in the lightweight verification venv (`asyncio_mode`
config is unknown there); it is unrelated to this change.
## Description
Fix a Python code-compression validity gap from #1233 where tree-sitter
parsing could mark compressed output as syntactically valid even when
Python compile-time syntax rules reject it.
This keeps `from __future__ import ...` statements in the
import-preservation bucket so they stay before executable definitions,
and adds Python `compile(..., "exec")` verification after `ast.parse`.
It also keeps the earlier conservative class-method decorator
indentation hardening from this branch.
Refs #1233.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Treat Python `future_import_statement` nodes as preserved imports.
- Verify Python compressed output with both `ast.parse` and
`compile(..., "exec")`.
- Preserve original source-line indentation for decorators attached to
class methods.
- Add a regression fixture covering `from __future__ import
annotations`, class decorators, property decorators, async methods, and
`match` statements.
- Add a direct regression assertion that future imports stay before
executable definitions.
- Document the user-visible fix in `CHANGELOG.md`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ /tmp/headroom-issue-1233-venv/bin/python -m pytest tests/test_transforms/test_code_compressor.py::TestTreeSitterIntegration::test_python_future_import_stays_at_module_start -q
1 passed, 1 warning
$ /tmp/headroom-issue-1233-venv/bin/python -m pytest tests/test_transforms/test_code_compressor.py -q
61 passed, 1 warning
$ uv run --extra dev ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py
All checks passed!
$ uv run --extra dev ruff format --check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py
2 files already formatted
$ git diff --check
# no output
```
## Real Behavior Proof
- Environment: macOS, Python 3.11.14, local checkout with `[code]`
dependencies installed in `/tmp/headroom-issue-1233-venv`.
- Exact command / steps: added
`test_python_future_import_stays_at_module_start`, ran it before the fix
to confirm the compressed output failure, then reran the focused test
and full `tests/test_transforms/test_code_compressor.py` after the
patch.
- Observed result: before this patch, the regression fixture produced
compressed Python with `from __future__ import annotations` after
class/function definitions. `result.syntax_valid` was `True`, but
`compile(result.compressed, "<test>", "exec")` failed with `SyntaxError:
from __future__ imports must occur at the beginning of the file`. After
this patch, the focused regression and full code-compressor test file
pass locally, and the regression now directly asserts that the future
import appears before executable definitions.
- Not tested: full repository pytest, `mypy headroom`, and a broad
corpus run over third-party source files.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
N/A
## Additional Notes
This PR is now scoped to the stable compile-time failure path in #1233.
The broader syntax-failure rate from the issue may still need
corpus-level follow-up.
Co-authored-by: JD Davis <mxjerrett@gmail.com>
## Description
`is_tree_sitter_available()` / `_check_tree_sitter_available()` in
`headroom/transforms/code_compressor.py` return `True` based on
importing `tree_sitter_language_pack` alone, without ever constructing a
parser or attempting a parse. When the installed pack/parser combination
is ABI-incompatible, `get_parser`/`parse` raises at runtime; the caller
catches it and silently falls back to the lossy text compressor, while
the availability flag and startup banner still report code-aware as on.
This is the defensive half that the `<1.0` pin in #1234 does not cover:
if that cap is ever lifted, the availability signal silently lies again.
Follow-up to #1231.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Make `_check_tree_sitter_available()` construct a parser and parse a
tiny snippet, returning `True` only if it yields a real `module` AST
instead of trusting an import.
- Add `_tree_sitter_importable()` for the cheap import-only probe, and
use it to guard parser construction so the real-parse check cannot
recurse.
- Add tests asserting the check is `False` when parsing raises and
`True` on a real parse, plus that AST compression runs for python/rust
without falling back.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
# pytest tests/test_transforms/test_code_compressor.py -> passed locally (tree-sitter-language-pack 0.13.0)
# ruff check . and ruff format --check . pass locally on the rebased branch.
# Full pytest suite / mypy not run locally; left to CI.
```
## Real Behavior Proof
- Environment: local repo on tree-sitter-language-pack 0.13.0,
tree-sitter 0.25.2, Python 3.12, Linux
- Exact command / steps: call `is_tree_sitter_available()`, then run
`pytest tests/test_transforms/test_code_compressor.py`
- Observed result: with a working pack the probe parses and returns
`True` (code-aware runs, strategy `CODE_AWARE` rather than the kompress
fallback); the new
`test_check_tree_sitter_available_false_when_parse_broken` confirms that
when parsing raises the check now returns `False` instead of the old
import-only `True`, so the lossy fallback is no longer entered silently.
- Not tested: reproducing the specific ABI-incompatible 1.x pack combo
against a live install (covered instead by a mocked broken parse in the
test)
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
LLMLingua was the original ML text compressor (BERT-based). Kompress
(ModernBERT, trained on 330K structured tool outputs) replaced it with
better compression quality and simpler architecture.
Removed across 35 files:
- Deleted headroom/transforms/llmlingua_compressor.py
- Deleted tests/test_transforms/test_llmlingua_compressor.py
- Deleted tests/test_proxy_llmlingua.py
- Removed all enable_llmlingua config, _get_llmlingua methods,
LLMLingua fallback paths, LLMLINGUA strategy enum values
- Removed CLI flags, model configs, compression handler references
- Simplified ContentRouter: Kompress is primary and only text compressor
CodeAwareCompressor now analyzes intra-file symbol relationships before
compression, using tree-sitter AST walks to count references, map call
graphs, and detect public/private visibility. This replaces uniform
"keep first N body lines" compression with budget-based allocation driven
by the existing target_compression_rate config.
Key design decisions:
- Distribution-based scoring (min-max normalized within each file) so it
adapts to any file structure: utility libs, test files, orchestrators
- Budget allocation: target_compression_rate determines total body line
budget, distributed proportionally to importance × body size
- max_body_lines respected as a hard cap over budget allocation
- Context-aware: the existing `context` parameter now boosts symbols
matching the user's task (word-boundary matching, not substring)
- Qualified names (ClassName.method) internally to avoid collisions
between identically-named methods in different classes
- Omitted comments include call graph info from AST analysis
- Zero new dependencies — uses tree-sitter already in headroom[code]
- semantic_analysis=True by default, fully backward-compatible when False
CodeAwareCompressor:
- Tree-sitter based AST parsing for Python, JS, TS, Go, Rust, Java, C, C++
- Preserves imports, signatures, type annotations, error handlers
- Guarantees syntactically valid output
- Uses tree-sitter-language-pack for broad language support
ContentRouter:
- Intelligent compression orchestrator
- Auto-routes content to optimal compressor based on type detection
- Source hint support for high-confidence routing
Custom Model Configuration:
- HEADROOM_MODEL_LIMITS env var and ~/.headroom/models.json support
- Pattern-based inference for unknown models (opus/sonnet/haiku tiers)
- Support for Claude 4.5, Claude 4, o3, o3-mini
- Graceful fallback - never crashes on unknown models
