mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
6 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Shengbo_Wang
|
1de35e775f
|
fix(code): parse-probe tree-sitter availability in code_handler (#1231) (#1300)
## Description `_check_tree_sitter()` in `headroom/compression/handlers/code_handler.py` only verified that `tree_sitter_language_pack` could be imported. When tree-sitter core and the language pack are built against different ABIs, the import succeeds but `parser.language = get_language(...)` raises at request time, silently falling back to the generic text compressor — with no warning, while the banner still reports code-aware as enabled. #1299 fixed the same class of bug in `transforms/code_compressor.py`. This PR is the defensive follow-up tracked by #1231: it applies the same parse probe to the compression **structure handler** so both code-aware paths are consistent. Closes #1231 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Replace the import-only probe in `code_handler._check_tree_sitter()` with a real parse probe: construct a `Parser`, assign a Python language, and parse `b"x = 1\n"` — if any step fails, mark unavailable - Log a WARNING when import succeeds but parsing fails, so the downgrade is visible - Add `TestAvailabilityProbe` covering the simulated ABI mismatch (-> False) and healthy install (-> True) cases > Note: an earlier revision of this PR also touched `transforms/code_compressor.py`, but that fix landed independently via #1299. After rebasing onto current `main`, this PR is scoped to the remaining `code_handler.py` gap only. ## Testing - [x] Unit tests pass (`pytest tests/test_compression/test_code_handler.py` -> 22 passed, 8 skipped) - [x] Linting / formatting pass (`ruff check .`, `ruff format --check .`) - [x] New tests added for new functionality ### Real Behavior Proof - Setup: Windows 11 (GBK locale), Python 3.10, tree-sitter NOT installed - Before fix: `_check_tree_sitter()` returns `True` on a partial/ABI-mismatched install (import succeeds), then silently degrades to the text compressor at request time with no warning - After fix: the probe parses a trivial snippet; an ABI mismatch is caught at probe time, `_check_tree_sitter()` returns `False`, and a WARNING is logged. `TestAvailabilityProbe::test_abi_mismatch_returns_false` reproduces this with a fake Parser whose `language` setter raises. Signed-off-by: RTCartist <wangshengb@buaa.edu.cn> |
||
|
Veesh Goldman
|
f39858c233
|
feat(code): add Perl support to code-aware compressor (#1125)
## Description Adds Perl as a supported language for `CodeAwareCompressor` / `CodeStructureHandler`. Function bodies are compressed while `use`/`require` imports, `sub`/`method` signatures, and `package`/`class`/`role` declarations are preserved — bringing Perl up to parity with the other Tier-2 languages. No new dependencies: the Perl grammar already ships in `tree-sitter-language-pack` (already a Headroom dependency), so this is pure configuration. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `code_handler.py`: Perl entries in the four per-language tables — `_STRUCTURAL_NODE_TYPES`, `_SIGNATURE_PATTERNS` (regex fallback), `_LANGUAGE_MARKERS` (detection), `_IMPORT_PATTERNS`. The existing `_CONTAINER_BODY_TYPES` already covers Perl's `block` body node, so no change was needed there. - `code_compressor.py`: `CodeLanguage.PERL` enum value, a data-driven `LangConfig`, a `_LANGUAGE_PREFILTER` entry, and the supported-language string in the parser error message. - Node-type names (`subroutine_declaration_statement`, `package_statement`, `signature`, `block`, …) are from the `tree-sitter-perl/tree-sitter-perl` grammar (MIT). - Tests: 1 detection test + 2 regex-path signature/import-preservation tests. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_compression/test_code_handler.py -q collected 28 items tests/test_compression/test_code_handler.py ............................ [100%] ======================== 28 passed, 1 warning in 2.53s ========================= $ ruff check headroom/compression/handlers/code_handler.py headroom/transforms/code_compressor.py tests/test_compression/test_code_handler.py All checks passed! $ mypy headroom/compression/handlers/code_handler.py headroom/transforms/code_compressor.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: Python 3.12, `pip install -e ".[code]"` (tree-sitter-language-pack installed, `is_tree_sitter_available() == True`). - Exact command / steps: ran `CodeStructureHandler().get_mask(code, language="perl")` on a real Perl module (package + two subs with bodies). - Observed result: detected as `perl`, parsed via the `tree-sitter` path (not regex), and the preserved span was exactly the imports + package + sub signatures, with both sub bodies marked compressible: ```text tree-sitter available: True parser: tree-sitter | detected: perl --- PRESERVED (signatures/imports/structure) --- use strict;use warnings;package Greeter;sub new sub greet ``` - Not tested: the full proxy/MCP server end-to-end path (out of scope — this PR only touches the code compressor's language tables). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - Docs/CHANGELOG left unchecked — happy to add a Perl line to either if you'd like; I wasn't sure of your preferred location. - *Disclosure: I maintain the upstream `tree-sitter-perl` grammar this relies on. It's already a transitive dependency of Headroom via `tree-sitter-language-pack` — this PR only adds config to use it, with no dependency changes.* Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ashish
|
6cdb846200
|
fix(compression): use thread-local tree-sitter parsers in code handler (#893)
## Description `CodeStructureHandler` cached tree-sitter parsers in a process-global dict; the lock only guarded creation, while `parse()` ran unlocked on any thread. tree-sitter `Parser` objects are pyo3 `unsendable` — using one from a non-creator thread panics. The proxy invokes handlers from executor pool threads, so a shared parser is an eventual crash. Same class already fixed in `transforms/code_compressor.py` (#604). Stacked on #892. Closes # <!-- compression-handler review --> ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/compression/handlers/code_handler.py`: one parser per (thread, language) via `threading.local()`, porting the pattern from `transforms/code_compressor.py`. - `tests/test_compression/test_code_handler.py`: regression test parsing from a 4-worker thread pool, asserting every call stays on the tree-sitter path. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_compression/ -q 94 passed, 8 skipped ``` ## Real Behavior Proof - Environment: local macOS, Python 3.11, tree-sitter-language-pack 1.8.1, branch `fix/code-threadlocal-parsers` (stacked on #892). - Exact command / steps: `pytest tests/test_compression/ -q`. - Observed result: 16 parses across a 4-worker pool all stay on the tree-sitter path with no pyo3 panic; previously a shared parser would be touched cross-thread. - Not tested: Reproducing the original panic under production concurrency (covered structurally by the thread-pool test). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — library change. See Test Output. ## Additional Notes Stacked on #892 — review the top commit until that merges. PR 5 of 7. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
Ashish
|
615e1ed6f5
|
test(compression): fill code handler coverage gaps (#895)
## Description `CodeStructureHandler` had zero dedicated tests before this series — which is exactly why the P0 bugs in #890/#892/#893 went unnoticed. This fills the remaining coverage gaps beyond the per-fix regression tests. Stacked on #893. Closes # <!-- compression-handler review --> ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - `tests/test_compression/test_code_handler.py`: language detection (python/go/rust + default fallback); regex-path signature/import preservation across go/rust/typescript/javascript; regex confidence value; empty/whitespace content; unknown language; mask-length invariant. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_compression/test_code_handler.py -q 25 passed ``` ## Real Behavior Proof - Environment: local macOS, Python 3.11, tree-sitter-language-pack 1.8.1, branch `test/code-handler-coverage` (stacked on #893). - Exact command / steps: `pytest tests/test_compression/test_code_handler.py -q`. - Observed result: 25 tests pass; tree-sitter classes skip cleanly when the pack is absent, regex-path tests always run. - Not tested: N/A — this PR is tests only. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — tests only. See Test Output. ## Additional Notes Stacked on #893 — review the top commit until that merges. PR 6 of 7. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Ashish
|
b1f700fc27
|
fix(compression): convert tree-sitter byte offsets to char offsets (#892)
## Description tree-sitter reports node positions as byte offsets into the UTF-8 encoding, but `CodeStructureHandler` builds a character-indexed mask. Any multi-byte character (accents, emoji, CJK in docstrings/comments/strings) shifted every subsequent span, preserving the wrong characters and leaking signature bytes into bodies. Stacked on #890. Closes # <!-- compression-handler review --> ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/compression/handlers/code_handler.py`: remap spans through a byte->char table before masking; pure-ASCII content (byte == char) skips the conversion. - `tests/test_compression/test_code_handler.py`: regression test with `café münü 🎉` in a comment, asserting the following signature and body are correctly aligned. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_compression/ -q 93 passed, 8 skipped ``` ## Real Behavior Proof - Environment: local macOS, Python 3.11, tree-sitter-language-pack 1.8.1, branch `fix/code-byte-char-offsets` (stacked on #890). - Exact command / steps: `pytest tests/test_compression/ -q`. - Observed result: With 9 extra UTF-8 bytes ahead of it, a function signature is exactly preserved and its body stays compressible; before, the offsets were shifted. - Not tested: End-to-end through the live proxy pipeline. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — library change. See Test Output. ## Additional Notes Stacked on #890 — review the top commit until that merges. PR 4 of 7. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Ashish
|
16ed73bca6
|
fix(compression): keep container bodies compressible in code handler (#890)
## Description Two bugs in `CodeStructureHandler`'s tree-sitter path. (1) Container nodes (class/impl/trait/decorated definitions) were marked structural over their full span and `_spans_to_mask` never un-marks, so every method body inside a class was preserved and compression silently no-opped at confidence 0.95. (2) Discovered while testing: `tree-sitter-language-pack >= 1.0` switched to a Rust binding (methods, not attributes; `parse(str)`), so the handler raised `TypeError` on every call and silently fell back to regex. Closes # <!-- compression-handler review --> ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/compression/handlers/code_handler.py`: containers emit a signature-only span (start to body start); recursion gives nested functions their own signature/body split; decorated definitions emit no whole-node span. - `headroom/compression/handlers/code_handler.py`: small compat shim supporting both the classic attribute API and the new Rust-binding method API. - `tests/test_compression/test_code_handler.py`: new file (the handler had zero dedicated tests) covering class/decorated/impl body compressibility, regex fallback, and a preservation-ratio bound. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_compression/ -q 92 passed, 8 skipped ``` ## Real Behavior Proof - Environment: local macOS, Python 3.11, tree-sitter-language-pack 1.8.1 installed, branch `fix/code-container-bodies`. - Exact command / steps: `pytest tests/test_compression/ -q`. - Observed result: Class method bodies are now compressible (preservation ratio drops from ~1.0 to roughly the signature fraction); the tree-sitter path runs instead of falling back to regex. - Not tested: End-to-end through the live proxy pipeline. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — library change. See Test Output. ## Additional Notes PR 3 of 7; branched fresh from main (independent of #887/#889). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |