From f11a271229975fd3d2b97372bf346f434add1abc Mon Sep 17 00:00:00 2001 From: Tufan ALIN Date: Sun, 21 Jun 2026 20:06:59 +0300 Subject: [PATCH] fix(code): pin tree-sitter-language-pack <1.0 so code compression works (#1234) ## Description The `[code]` extra requires `tree-sitter-language-pack>=0.10.0` with no upper bound, so it now resolves to the 1.x line. tree-sitter-language-pack 1.0 (2026-03-21) is a breaking rewrite whose `get_language()` / `get_parser()` return the pack's own binding types instead of standalone `tree_sitter.Language` / `tree_sitter.Parser`. As a result `headroom/transforms/code_compressor.py::_get_parser()` raises, the exception is caught upstream, and AST code compression silently falls back to passthrough (0% reduction, no error surfaced) on a fresh `pip install headroom-ai[code]`. This caps the dependency below the breaking rewrite and pins the matching tree-sitter range, which is the line the existing code is written against. Closes #1232 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Pin `tree-sitter-language-pack>=0.10.0,<1.0` in the `[code]` extra (was `>=0.10.0`). - Add an explicit `tree-sitter>=0.25.2,<0.26` pin to document the supported range (0.13.0 already requires `tree-sitter>=0.25.2`). - Add an inline comment explaining why the `<1.0` cap is required, to prevent a future re-bump. ## Testing - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [x] Manual testing performed I did not run the full pytest / ruff / mypy suite for this change (it is a dependency-constraint pin); I verified the actual runtime behavior the pin restores. See Real Behavior Proof. ### Test Output ```text # BEFORE (resolved tree-sitter-language-pack 1.9.1): code compression no-ops CodeAwareCompressor().compress() -> compression_ratio = 1.0 (0% on every file sampled) # AFTER (tree-sitter-language-pack 0.13.0 + tree-sitter 0.25.2), headroom code unchanged: is_tree_sitter_available(): True # 60 varied real Python files (headroom, litellm, pydantic, openai), default CodeCompressorConfig: compressed OK (valid + reduced): 31 (52%) rejected for invalid syntax: 17 (28%) -> returns original, never serves broken code no reduction / too small: 12 (20%) reduction when it worked: min 4.4% median 37.2% max 88.8% # All compressed outputs re-parsed clean with ast.parse(). ``` ## Real Behavior Proof - Environment: Python 3.12, headroom-ai 0.26.0. Before: tree-sitter-language-pack 1.9.1 (what `[code]` resolves today). After: tree-sitter-language-pack 0.13.0 + tree-sitter 0.25.2 (what this pin resolves). - Exact command / steps: `pip install "headroom-ai[code]"`; then run `CodeAwareCompressor(CodeCompressorConfig()).compress(src)` over a sample of real `.py` files and re-tokenize before/after with tiktoken (cl100k_base), re-parsing each output with `ast.parse`. - Observed result: with the unpinned (1.x) resolution, every sampled file returned `compression_ratio == 1.0` (0%, silent passthrough). With the pinned (0.x) resolution and no code changes, `is_tree_sitter_available()` is True and 31/60 files compressed validly at a ~37% median (up to ~89%); all compressed outputs re-parsed clean. - Not tested: the full pytest / ruff / mypy suite; per-language rates for JS/TS/Go/Rust/Java/C/C++ (they share the same `_get_parser()` path, so the fix applies, but I measured Python specifically); the ~28% invalid-syntax rejections are a separate pre-existing robustness issue tracked in #1233, not addressed here. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A (dependency-constraint change). ## Additional Notes - This is the minimal fix to restore functionality. The proper longer-term fix is to migrate `_get_parser()` and the AST walker to the tree-sitter-language-pack 1.x API, after which the `<1.0` cap can be lifted; happy to follow up with that if preferred. - Unchecked checklist items, with rationale: no docs change needed (constraint-only); no new tests added (a corpus-based compress-and-reparse regression test would be valuable but belongs with the robustness work in #1233); I did not run the full local unit-test suite for a dependency pin; CHANGELOG appears to be release-please managed, so I left it untouched. - I am not a maintainer; this came out of an independent evaluation of the `[code]` path. Pinning `<1.0` parks the project on the now-superseded 0.x pack, which is the tradeoff for a one-line fix today. Co-authored-by: mitralone <5514599+mitralone@users.noreply.github.com> --- pyproject.toml | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/pyproject.toml b/pyproject.toml index 30b520806..56b9dc6cd 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -86,8 +86,14 @@ proxy-prod = [ "gunicorn>=21.0.0; sys_platform != 'win32'", ] # AST-based code compression (tree-sitter) +# NOTE: cap below 1.0. tree-sitter-language-pack 1.x is a breaking rewrite whose +# get_language()/get_parser() return the pack's own binding types instead of +# standalone tree_sitter.Language/Parser, so _get_parser() in +# transforms/code_compressor.py fails and code compression silently no-ops. +# The 0.x line (>=0.10,<1.0) returns standalone tree_sitter objects as expected. code = [ - "tree-sitter-language-pack>=0.10.0", + "tree-sitter-language-pack>=0.10.0,<1.0", + "tree-sitter>=0.25.2,<0.26", ] # ML-based compression with Kompress (ModernBERT). # (The legacy [llmlingua] extra was removed in 0.9.x — no live code path used it.