mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-10 14:27:00 -04:00
14 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Parideboy
|
6d5516dcb8
|
feat(code): add PHP support to CodeAwareCompressor (#2423)
## Description Adds PHP to `CodeAwareCompressor`, fixing #201. PHP was already *detected* as code (Magika labels in `headroom/compression/detector.py` include `php`, and the Rust `magika_detector.rs` lists it too) but there was no PHP `LangConfig`, so PHP content silently passed through uncompressed. This wires PHP through the tree-sitter compression path following the C# pattern (the most recently added, fully functional language — deliberately not the quarantined Perl path). A secondary detection bug is fixed along the way: PHP's `$variables` match Perl's prefilter regex, and the existing Perl-dominance guard in `detect_language` returned `UNKNOWN` for PHP files. An explicit `<?php` open tag — which no Perl source contains — now drops Perl from the candidate set before that guard runs. ## Type of Change - [ ] Bug fix - [x] New feature - [ ] Documentation update - [ ] Refactor - [ ] Other ## Changes Made - `headroom/transforms/code_compressor.py`: `CodeLanguage.PHP` + `phtml`/`php5`/`php7`/`php8` aliases; PHP `LangConfig` built from the actual tree-sitter-php grammar (node names verified by parsing samples): `namespace_use_declaration` imports, `function_definition`/`method_declaration` functions, `class_declaration`/`interface_declaration`/`trait_declaration` classes, `enum_declaration` types, `declaration_list` class bodies, `compound_statement` function bodies. `namespace_definition` maps to `package_node` so statement-scoped `namespace App;` hoists ahead of the `use` imports (required PHP ordering); the rare block-scoped `namespace A { }` form takes the same path and is preserved verbatim — valid output, just no compression inside the block. PHP prefilter regexes added; supported-languages error message updated; `<?php`-tag Perl disambiguation in `detect_language`. - `headroom/transforms/content_detector.py`: `php` entry in `_CODE_PATTERNS` so raw PHP classifies as `SOURCE_CODE` and reaches the code-aware route. - `tests/test_transforms/test_code_compressor.py`: new `TestPhpSupport` mirroring `TestCSharpSupport` — signatures preserved / bodies elided, `<?php` → `namespace` → `use` → declarations ordering, auto-detection despite the Perl sigil overlap, alias coercion, malformed passthrough. - `tests/test_code_compressor_language_alias.py`: `php` in the canonical list, `phtml` in the alias table. - `docs/content/docs/code-compression.mdx`: PHP added to the Tier 2 supported-languages row. No new dependency: `tree-sitter-language-pack` (the existing `[code]` extra) already ships the PHP grammar. No Rust changes needed. ## Testing - [x] New unit tests added and passing - [x] Full affected test suites pass locally **Test Output** ``` $ python -m pytest tests/test_transforms/test_code_compressor.py tests/test_code_compressor_language_alias.py -q ============================= 120 passed in 7.81s ============================= $ python -m pytest tests/test_transforms/ -q 3 failed, 443 passed # the 3 failures (kompress ONNX thread caps, kompress size gate, # text_crusher unicode parity) reproduce identically on a clean # upstream/main checkout in this environment — pre-existing local # ONNX runtime quirks, unrelated to this change $ ruff check . (0.15.17, CI-pinned) → All checks passed! | ruff format --check → clean $ mypy headroom/transforms/code_compressor.py headroom/transforms/content_detector.py --ignore-missing-imports Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13, tree-sitter + tree-sitter-language-pack (<1.0) installed, branch `feat/201-php-code-compression` off `upstream/main`. - Exact command / steps: parsed PHP samples (namespaced class w/ methods, block-scoped namespace, mixed HTML+PHP) with `tree_sitter_language_pack.get_parser('php')` to verify every node name used in the config; then ran `CodeAwareCompressor().compress(php_code, language="php")` and `compress(php_code)` (auto-detection) on a 48-line realistic service class. - Observed result: explicit and auto-detected paths both return `language=CodeLanguage.PHP`, `compression_ratio=0.64`, `syntax_valid=True`; method bodies elided to `// [N lines omitted]` while `<?php`, `namespace`, `use` lines, class header, and all signatures are preserved verbatim in the original order. Before the detection fix, auto-detection returned `UNKNOWN` (Perl prefilter dominance) — reproduced and then verified fixed. - Not tested: exotic PHP shapes (heredoc-heavy code, attributes `#[...]` on methods, interleaved multi-`<?php ?>` HTML templates beyond the basic mixed case); these fall back to verbatim preservation via the uncaptured-node pass or malformed-passthrough, both of which are covered by tests for the simple cases. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Rod Boev
|
dbbef4bd41
|
fix(code-compressor): recover valid Python rewrites after local syntax rejection (#2202)
## Description A compile-invalid Python definition rewrite currently makes `CodeAwareCompressor` discard every otherwise valid rewrite in the file and return the original source at 0 percent reduction. The existing whole-file safety guard stays in place, while a Python-only recovery replay now preserves the rejected definition and keeps independent valid compression. The recovery reuses the current Python validation authority in `ast.parse()` plus `compile()`, runs only after the first assembled module already fails `_verify_syntax()`, and stays out of non-Python paths. Closes #1233 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added a Python-only recovery replay after the first assembled module fails syntax validation. - Preserved only the invalid function or class rewrite while allowing independent valid definitions to remain compressed. - Kept the existing whole-file syntax guard and original-source fallback as the terminal safety check. - Added focused invalid-node, valid-modern-syntax, and fail-safe coverage. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_python_invalid_node_falls_back_locally tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_python_recovery_does_not_block_valid_modern_syntax tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_python_recovery_still_returns_original_when_all_candidates_invalid tests/test_transforms/test_code_compressor.py::TestEdgeCases::test_syntax_errors_in_input tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_ast_preserves_structure_for_python -v`) - [x] Linting passes (`uv run ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text uv run pytest tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_python_invalid_node_falls_back_locally tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_python_recovery_does_not_block_valid_modern_syntax tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_python_recovery_still_returns_original_when_all_candidates_invalid tests/test_transforms/test_code_compressor.py::TestEdgeCases::test_syntax_errors_in_input tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_ast_preserves_structure_for_python -v 5 passed in 0.34s uv run ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py All checks passed! uv run ruff format headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py --check 2 files already formatted ``` ## Real Behavior Proof - Environment: Windows, synced `uv` environment with `dev` and `code` extras installed - Exact command / steps: run the focused invalid-node regression through public `compress(..., language="python")` - Observed result: `1 passed in 0.19s`; the invalid candidate stays original, the neighboring valid candidate remains compressed, and `headroom-PR-TARGET-1233-PROOF.md` records the base `ratio=1.0` whole-file rollback against the fixed head behavior. - Not tested: the stale future-import mismatch discussed in the old issue comment, already covered on current main ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - The stale future-import comment on #1233 is not the live slice here; current main already validates Python with `compile()` and already covers that ordering case. - This fix keeps the existing whole-file fail-safe and does not broaden into cross-language recovery or new syntax models. - `CHANGELOG.md` remains unchanged because Headroom generates release notes from conventional commits. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Yevhen Koval
|
ec55ddcfb3
|
feat(transforms): first-class C# support in CodeAwareCompressor (#1926)
Refs #1664 ## Description First-class C# support in `CodeAwareCompressor` via the tree-sitter `csharp` grammar, at parity with Java/C++/Rust: `using` directives, namespace headers, and type/member signatures preserved verbatim; method/constructor/destructor/operator/local-function bodies compressed; malformed input passes through unchanged. **No new dependencies** — the grammar ships inside the already-pinned `tree-sitter-language-pack==0.13.0` (resolves only as `"csharp"`; `c_sharp`/`cs` raise `LookupError`). Spec and maintainer go-ahead in the issue. Closes #1664 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `CodeLanguage.CSHARP` + full `_LANG_CONFIGS` entry; `_LANGUAGE_PREFILTER` and `content_detector` patterns chosen to be C#-distinctive (so Java doesn't mis-tag). - New data-driven `LangConfig` fields (pattern of #1334's `class_body_node_types`): `container_node_types` — block-scoped `namespace { }` routed through class compression so members compress without the wrapper being re-emitted verbatim; `opaque_node_types` — `#if`…`#endif` wrappers preserved verbatim without recursion (recursing + wrapper re-emit duplicated whole files, up to ~1.9x input on real repos); `#if` blocks wrapping only usings are emitted with the imports so they stay ahead of type declarations. - Shared-path fixes surfaced by real C# repos, each guarded and covered by a fail-before test: keep an Allman `{` on its own line in class reconstruction (K&R path byte-for-byte unchanged; Allman Java now compresses instead of falling back); line-based child extraction no longer swallows the following line for nodes ending at column 0 (C# `#region`/`#endregion` span their trailing newline — the over-slice duplicated the next member's signature or the closing brace); uncaptured top-level nodes preceding the first captured node (license banners, `#region License`) are emitted first instead of relocated below the code (tree-sitter-c-sharp rejects top-level `#region` after a type declaration, so relocation forfeited compression for the whole file). - `TestCSharpSupport` (8 tests) + a C# case in the parametrized member-container test; CHANGELOG entry. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_transforms/test_code_compressor.py -q 2 failed, 78 passed, 1 warning, 4 errors # the 2 failures / 4 errors reproduce # identically on main in the same env # (network-dependent tokenizer setup) Fail-before: with both changed sources reverted to main, the new C#-scoped selection reports "10 failed, 5 passed" (the 5 other languages keep passing); on the branch: "15 passed". $ ruff check headroom/transforms/code_compressor.py headroom/transforms/content_detector.py tests/test_transforms/test_code_compressor.py All checks passed! $ ruff format --check <same files> 3 files already formatted ``` ## Real Behavior Proof - Environment: Linux 6.8.0 aarch64, Python 3.10.12; `uv run --no-project --with "tree-sitter-language-pack==0.13.0" --with "tree-sitter>=0.25.2,<0.26" --with "pydantic>=2.0.0"`; real `CodeAwareCompressor` (`CodeCompressorConfig(enable_ccr=False)`, otherwise defaults), no mocks. - Exact command / steps: cloned two real .NET repos at depth 1 (`github.com/JamesNK/Newtonsoft.Json @ 4f73e74`, `github.com/App-vNext/Polly @ 7a1d10f`), ran `python proof_csharp.py <repo>` over every `.cs` file (chars/4 token estimate; tiktoken BPE download unavailable in my sandbox). Script in the collapsed section below. - Observed result: 16.1% tokens saved on Newtonsoft.Json (945/945 syntax-valid), 37.8% on Polly (797/797 syntax-valid), zero content duplication; full output: ```text repo: Newtonsoft.Json (945 .cs files) tokens before: 1,777,691 after: 1,490,629 saved: 287,062 (16.1%) files compressed: 479 pass-through: 466 inflated(>before): 19 syntax_valid: 945/945 latency ms P50: 0.7 P95: 18.7 P99: 44.1 max: 255.0 mean: 3.5 repo: Polly (797 .cs files) tokens before: 1,100,523 after: 684,303 saved: 416,220 (37.8%) files compressed: 693 pass-through: 104 inflated(>before): 15 syntax_valid: 797/797 latency ms P50: 0.8 P95: 11.6 P99: 28.9 max: 74.1 mean: 2.4 ``` After rebasing onto current `main` (which touched the same transform files via #1906/#1747/#1668) I re-ran the Polly proof on the rebased tree: 37.8% saved, 797/797 syntax-valid, P99 28.5ms — unchanged. Signatures/properties verbatim, bodies elided with call summaries, `using` order and preproc balance intact; residual "inflated" files are +2…+209 chars of assembly blank lines, not duplicated content. Newtonsoft is the adversarial case (multi-targeting: heavy `#if`, `#region`, Allman) — its conditional regions stay verbatim by design. Latency at parity with Java (<50ms P99; max is the pre-existing symbol-analysis cost on ~1800+-line files, shared with other languages). - Not tested: proxy end-to-end path with C# through `ContentRouter` (tested the `CodeAwareCompressor` API directly); CCR retrieval round-trips (`enable_ccr=False` in proof runs); exact tiktoken counts (chars/4 estimate — relative ratios are tokenizer-independent); Windows/macOS; full native `uv run pytest` with the Rust extension (ran the complete `test_code_compressor.py` in a lightweight venv; its 2 failures/4 errors reproduce identically on `main`); `mypy`. <details> <summary>proof_csharp.py (reproducible)</summary> ```python """Real behavior proof: run the real CodeAwareCompressor over a .NET repo.""" import pathlib import statistics import sys import time from headroom.transforms.code_compressor import ( CodeAwareCompressor, CodeCompressorConfig, ) try: import tiktoken ENC = tiktoken.get_encoding("cl100k_base") def toks(s: str) -> int: return len(ENC.encode(s, disallowed_special=())) except Exception: def toks(s: str) -> int: return len(s) // 4 target = pathlib.Path(sys.argv[1]) comp = CodeAwareCompressor(CodeCompressorConfig(enable_ccr=False)) tot_before = tot_after = 0 n_files = n_compressed = n_valid = n_passthrough = n_inflated = 0 times_ms: list[float] = [] for f in sorted(target.rglob("*.cs")): try: code = f.read_text(encoding="utf-8-sig", errors="replace") except OSError: continue t0 = time.perf_counter() r = comp.compress(code, language="csharp") times_ms.append((time.perf_counter() - t0) * 1000) n_files += 1 b, a = toks(code), toks(r.compressed) tot_before += b tot_after += a if r.compressed == code: n_passthrough += 1 else: n_compressed += 1 if r.syntax_valid: n_valid += 1 if a > b: n_inflated += 1 times_ms.sort() p = lambda q: times_ms[min(int(len(times_ms) * q), len(times_ms) - 1)] print(f"repo: {target.name} ({n_files} .cs files)") print(f" tokens before: {tot_before:,} after: {tot_after:,} saved: {tot_before - tot_after:,} ({(1 - tot_after / tot_before) * 100:.1f}%)") print(f" files compressed: {n_compressed} pass-through: {n_passthrough} inflated(>before): {n_inflated}") print(f" syntax_valid: {n_valid}/{n_files}") print(f" latency ms P50: {p(0.50):.1f} P95: {p(0.95):.1f} P99: {p(0.99):.1f} max: {times_ms[-1]:.1f} mean: {statistics.mean(times_ms):.1f}") ``` </details> ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — terminal evidence above. ## Additional Notes - Dependency justification: none added, none bumped; the `csharp` grammar is inside the already-pinned `tree-sitter-language-pack==0.13.0` wheel; `uv.lock` untouched. - Architecture: malformed input passes through byte-identical; every risky construct prefers the false negative (verbatim) over corruption; invalid reassembly falls back to the original via the existing validation gate (observed live); no new imports at module load; P99 <50ms on both proof repos. - Known v1 limitations (deliberate false negatives, possible follow-ups): expression-bodied members and property accessor bodies stay verbatim; declarations inside `#if` regions stay verbatim. - Related pre-existing finding, out of scope: C/C++ exhibit the same `#if`-wrapper duplication on `main` (an `#if`-wrapped C++ class is emitted twice, ratio 1.62). Happy to file separately. - `mypy` unchecked above because I did not run it in my environment. |
||
|
Hafiz Ismail
|
adf8fed9bd
|
fix(code): stop TS export duplication + comment displacement (#1906)
## Description `CodeAwareCompressor` (AST-based code compression, `headroom/transforms/code_compressor.py`) had two bugs in its structure-reassembly path, found while investigating a reported Go brace-duplication issue (the Go bug itself — `statement_list` row-range swallowing a block's closing brace — was already fixed on `main` in #1668; this PR fixes what was *actually* still broken): 1. **TS/JS `export` keyword duplication.** `export function foo() {}` / `export class Foo {}` compressed to `export export function foo() {}` — invalid syntax, silently discarded by `_verify_syntax`'s fallback (the caller never sees an error, compression just quietly no-ops). Root cause: `_compress_function_ast` / `_compress_class_ast` slice a node's source by **line**, not by byte offset, deliberately — to preserve leading indentation for definitions nested inside classes. But when a node shares its *first* line with a preceding sibling (the `export` keyword is a sibling of the function inside tree-sitter's `export_statement` node, not part of the function node itself), that line-based slice pulled the sibling's text in too. The `export_statement` handler then re-prepended the same `export` text on top, producing the duplicate. 2. **Doc-comment displacement (all languages).** A `/** ... */` or `//` doc comment directly above a top-level function/class/type got detached from its declaration during AST extraction and re-emitted in one cluster at the very end of the compressed output, instead of staying attached to what it documents. Root cause: doc comments are top-level *siblings* of the declaration they document, not children of it — the extractor didn't attach them to anything, so they fell through to a "leftover top-level code" bucket that gets flushed as a single block after all functions. Also tightens `test_actual_go_compression`, which — per its own comment — was written to *tolerate* the Go bug (`compression_ratio may be 1.0 if compression produces invalid syntax`) rather than catch it. Since the underlying Go bug is already fixed on `main`, this now asserts real compression (`compression_ratio < 1.0`), matching its JS/Python siblings. Closes #1905 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made Two commits: the fix itself, then the tests that prove it — bisectable independently, both pass the full suite on their own. **Commit 1 — `fix(code):`** - `headroom/transforms/code_compressor.py`: add `_get_node_lines()` — line-based node slicing that still preserves indentation, but trims a preceding sibling's text from the first line when that prefix isn't pure whitespace (i.e. an `export` keyword sharing the line), so callers that re-add the sibling text themselves don't get a duplicate; used by `_compress_function_ast` and `_compress_class_ast`. - `headroom/transforms/code_compressor.py`: add `_get_leading_comment_text()` — walks a node's `prev_sibling` chain to collect contiguous doc-comment nodes immediately above it (no blank line in between) and returns them for the caller to prepend, also marking their byte ranges as captured so they aren't independently swept into the leftover top-level-code bucket; wired into every capture branch in `_extract_structure` (package, import, export statement, decorator, function, class, type). - `CHANGELOG.md`: added an entry under `### Fixed`. **Commit 2 — `test(code):`** - `tests/test_transforms/test_code_compressor.py`: `test_actual_go_compression` now asserts `compression_ratio < 1.0` instead of tolerating a 1.0 fallback. - `tests/test_code_aware_brace_comment_regressions.py` (new): 4 regression tests — TS `export` not duplicated + valid syntax, TS doc comments stay attached, Go doc comments stay attached, and a real-TS-compression parity test matching the existing JS/Python/Go "actual compression" tests. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py tests/test_code_aware_brace_comment_regressions.py All checks passed! $ ruff format --check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py tests/test_code_aware_brace_comment_regressions.py 3 files already formatted $ pytest tests/test_transforms/test_code_compressor.py tests/test_code_aware_regressions.py tests/test_code_aware_brace_comment_regressions.py -q 83 passed in 6.23s $ pytest -q # full suite 7912 passed, 5 failed, 442 skipped in 417.43s (0:06:57) # The 5 failures are pre-existing and unrelated: confirmed to fail identically # with this PR's changes stashed out (clean upstream/main checkout). # - test_wrap_marker_is_stale_when_pid_reused (PID-reuse detection, env-specific) # - test_read_cached_oauth_token_falls_back_to_gh_cli (leaks real local `gh` credentials) # - test_rtk_reader_returns_none_on_nonzero_exit / test_lean_ctx_reader_returns_none_on_failure_and_logs # (pass in isolation; fail only in full-suite order — pre-existing test-pollution, unrelated to code_compressor.py) # - test_parser_usable_in_thread_pool (test itself passes a str to parser.parse(), # which tree-sitter's binding has always required as bytes — a pre-existing test # bug unrelated to this change; separate fix in progress on another branch) $ mypy headroom Success: no issues found in 408 source files ``` ## Real Behavior Proof - Environment: macOS (Darwin 24.6.0), Python 3.14.5, headroom-ai dev checkout built via `uv sync --extra dev` + `maturin develop -m crates/headroom-py/Cargo.toml` (real `headroom._core` build, not mocked), `tree-sitter==0.25.2` / `tree-sitter-language-pack` per the pinned `[code]` extra. - Exact command / steps: ran `CodeAwareCompressor(CodeCompressorConfig(min_tokens_for_compression=10, enable_ccr=False)).compress(open("sdk/typescript/src/client.ts").read(), language="typescript")` identically against `git stash`-ed (pre-fix) and current (post-fix) trees; full snippet and additional samples below. - Observed result: `client.ts` (real 20KB SDK file in this repo) went from `compression_ratio=1.0` with a silent fallback (`export export class HeadroomClient` in the raw AST attempt, invalid syntax) to `compression_ratio=0.942`, `syntax_valid=True` — real compression, no duplication; full before/after table below. - Not tested: real-world repos beyond this repo's own SDK sample and the bundled benchmark fixture — broader corpus testing may follow as a comment on this PR. **Exact command, full snippet:** ```python from headroom.transforms.code_compressor import CodeAwareCompressor, CodeCompressorConfig compressor = CodeAwareCompressor(CodeCompressorConfig(min_tokens_for_compression=10, enable_ccr=False)) with open("sdk/typescript/src/client.ts") as f: code = f.read() result = compressor.compress(code, language="typescript") ``` **Observed result, before vs. after, real code:** | Sample | Before (main) | After (this fix) | |---|---|---| | `sdk/typescript/src/client.ts` (real 20KB SDK file, this repo) | `compression_ratio=1.0`, silent fallback — `export export class HeadroomClient` in the raw AST attempt, invalid syntax | `compression_ratio=0.942`, `syntax_valid=True` — real compression, no duplication | | TS fixture exercising both bugs (exported fn/class + doc comments) | `compression_ratio=1.0`, silent fallback | `compression_ratio=0.993`, `syntax_valid=True` | | `middleware/ratelimit.go` (bundled benchmark sample) | `compression_ratio=0.862`, `syntax_valid=True` — unaffected (Go bug already fixed on `main` by #1668) | `compression_ratio=0.862`, `syntax_valid=True` — unchanged, confirms no regression | | `generate_go_code(3)` (existing test fixture) | `compression_ratio=0.498` | `compression_ratio=0.498` — unchanged, confirms no regression | On code shaped to actually exercise elision (function bodies long enough to exceed `max_body_lines=5`), TypeScript compresses in line with other languages once the correctness bug stops blocking it entirely: | Language | Compression savings (synthetic fixture, ~10-line function bodies) | |---|---| | Python | 64.4% | | Go | 52.3% | | TypeScript | 49.0% | | JavaScript | 42.8% | (`client.ts`'s real-world 5.8% savings is lower than the synthetic TypeScript number above because most of its methods are ≤5 lines — under the elision threshold regardless of language — not because of a language-specific limitation.) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes The Go brace-duplication bug that motivated this investigation was already fixed on `main` (#1668, merged before this branch was based) — confirmed via the minimal repro and `ratelimit.go`, both compress cleanly with no duplicated braces. This PR fixes what was still actually broken: the TS/JS `export`-duplication bug and the doc-comment displacement bug (both present across languages), found empirically while verifying the original bug report against the current `main`. --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
Vinay Gupta
|
82384022bd
|
fix(code): slice tree-sitter byte offsets as UTF-8 (#1332)
## Description CodeAwareCompressor was slicing Python strings with tree-sitter `start_byte` / `end_byte` offsets directly. That works for ASCII-only files, but it corrupts slices after non-ASCII source text such as CJK characters or emoji because tree-sitter offsets are UTF-8 byte offsets while Python string indexes are character offsets. This caused code-aware compression to produce invalid intermediate Python and then safely fall back to the original file, resulting in 0% compression on affected files. Closes #1319 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added `_slice_code_bytes()` in `headroom/transforms/code_compressor.py` to slice source text using UTF-8 byte offsets. - Updated `_get_node_text()` to use byte-safe slicing. - Routed the other direct tree-sitter byte-offset slices through the same helper. - Added regression tests in `tests/test_transforms/test_code_compressor.py`: - `test_get_node_text_uses_utf8_byte_offsets` - `test_ast_compresses_python_after_non_ascii_source` ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ .venv/bin/python -m pytest tests/test_transforms/test_code_compressor.py 68 passed, 1 warning $ .venv/bin/python -m ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py All checks passed! $ .venv/bin/python -m ruff format --check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py 2 files already formatted $ git diff --check # no output $ /tmp/headroom-1319-venv/bin/python -m mypy headroom headroom/proxy/server.py:1186: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] headroom/proxy/server.py:1257: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] headroom/proxy/server.py:1261: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] pyproject.toml: note: unused section(s): module = ['mlx.*'] Success: no issues found in 394 source files ``` ## Real Behavior Proof - Environment: macOS arm64, Python 3.14.5, tree-sitter 0.25.2, tree-sitter-language-pack 0.13.0 - Exact command / steps: On `main`, ran a local reproducer with a Python source string containing a CJK docstring before a second function; called `_get_node_text()` on the second tree-sitter function node; ran a full `CodeAwareCompressor.compress(...)` repro with non-ASCII module text before an import and a compressible function; re-ran both repros on this branch. - Observed result: Before fix, `_get_node_text()` returned the wrong slice (`'nd():\n return 2\n'` instead of `'def second():\n return 2'`) and full compression fell back to the original file with `compression_ratio: 1.0`; after fix, `_get_node_text()` returns the full expected function slice and full compression succeeds with `compression_ratio < 1.0`, `syntax_valid: True`, and does not return the original. - Not tested: Full repository test suite; live proxy/provider integrations; Windows/Linux platform-specific behavior. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes - Documentation was not updated because this is an internal bug fix with no user-facing API or behavior change beyond restoring intended compression. - `CHANGELOG.md` was not updated because the fix is narrow and issue-scoped; maintainers can advise if they want a changelog entry. - The fix is intentionally small and targeted: it only changes how tree-sitter byte offsets are converted back into Python source text, without changing compression heuristics or language behavior. |
||
|
Vinay Gupta
|
c35af858ea
|
fix(code): compress class member containers (#1334)
## Description CodeAwareCompressor used the same `body_node_types` config to find both executable function bodies and class/impl member containers. That works when those AST nodes happen to match, but it misses member containers such as Java `class_body`, C++ `field_declaration_list`, and Rust `declaration_list`, so class methods were returned essentially uncompressed. This adds an optional `class_body_node_types` override for class/impl member containers and uses it only in class compression. It also skips anonymous punctuation tokens while reconstructing class bodies and keeps same-line C++ class semicolons attached to the compressed class declaration. Closes #1318 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added `LangConfig.class_body_node_types` for languages whose class/impl member container differs from executable method-body nodes. - Configured class member containers for JavaScript, TypeScript, Java, C++, and Rust. - Updated `_compress_class_ast` to use class-member containers, skip anonymous punctuation children, and preserve C++ `};` output without creating stray top-level semicolons. - Added regression coverage proving class/impl methods compress for JavaScript, TypeScript, Java, C++, and Rust. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ PYTHONPATH=. /tmp/headroom-1319-venv/bin/python -m pytest tests/test_transforms/test_code_compressor.py -q collected 71 items tests/test_transforms/test_code_compressor.py .......................... [ 36%] ............................................. [100%] 71 passed, 1 warning in 0.36s $ /tmp/headroom-1319-venv/bin/python -m ruff check . All checks passed! $ /tmp/headroom-1319-venv/bin/python -m ruff format --check . 965 files already formatted $ PYTHONPATH=. /tmp/headroom-1319-venv/bin/python -m mypy headroom headroom/proxy/server.py:1186: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] headroom/proxy/server.py:1257: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] headroom/proxy/server.py:1261: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] pyproject.toml: note: unused section(s): module = ['mlx.*'] Success: no issues found in 394 source files ``` ## Real Behavior Proof - Environment: macOS arm64, Python 3.14.5, branch `fix-code-compressor-class-members`, tree-sitter grammar pack installed in `/tmp/headroom-1319-venv`, repo imported with `PYTHONPATH=.`. - Exact command / steps: Reproduced class-method compression with `CodeAwareCompressor(CodeCompressorConfig(enable_ccr=False, min_tokens_for_compression=1, max_body_lines=1))` for Java/C++/Rust before the fix, then reran the pytest/ruff/mypy commands listed above after the patch. - Observed result: Java/C++/Rust class methods now compress below 1.0 while `syntax_valid` remains true; C++ output preserves `};`; regression coverage also verifies JavaScript/TypeScript class member containers. - Not tested: Full repository pytest suite; local `uv run` editable builds are blocked on this machine by native C++ header failures in optional/native dependencies (`hnswlib` / Rust `esaxx-rs`), so validation used a lightweight venv with `PYTHONPATH=.`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes Documentation and CHANGELOG updates are not applicable for this narrow bug fix. The pytest warning shown above is from running without `pytest-asyncio` in the lightweight verification venv (`asyncio_mode` config is unknown there); it is unrelated to this change. |
||
|
Parafee41
|
cbd361de2a
|
fix(code): validate Python compressed syntax (#1302)
## Description Fix a Python code-compression validity gap from #1233 where tree-sitter parsing could mark compressed output as syntactically valid even when Python compile-time syntax rules reject it. This keeps `from __future__ import ...` statements in the import-preservation bucket so they stay before executable definitions, and adds Python `compile(..., "exec")` verification after `ast.parse`. It also keeps the earlier conservative class-method decorator indentation hardening from this branch. Refs #1233. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Treat Python `future_import_statement` nodes as preserved imports. - Verify Python compressed output with both `ast.parse` and `compile(..., "exec")`. - Preserve original source-line indentation for decorators attached to class methods. - Add a regression fixture covering `from __future__ import annotations`, class decorators, property decorators, async methods, and `match` statements. - Add a direct regression assertion that future imports stay before executable definitions. - Document the user-visible fix in `CHANGELOG.md`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ /tmp/headroom-issue-1233-venv/bin/python -m pytest tests/test_transforms/test_code_compressor.py::TestTreeSitterIntegration::test_python_future_import_stays_at_module_start -q 1 passed, 1 warning $ /tmp/headroom-issue-1233-venv/bin/python -m pytest tests/test_transforms/test_code_compressor.py -q 61 passed, 1 warning $ uv run --extra dev ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py All checks passed! $ uv run --extra dev ruff format --check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py 2 files already formatted $ git diff --check # no output ``` ## Real Behavior Proof - Environment: macOS, Python 3.11.14, local checkout with `[code]` dependencies installed in `/tmp/headroom-issue-1233-venv`. - Exact command / steps: added `test_python_future_import_stays_at_module_start`, ran it before the fix to confirm the compressed output failure, then reran the focused test and full `tests/test_transforms/test_code_compressor.py` after the patch. - Observed result: before this patch, the regression fixture produced compressed Python with `from __future__ import annotations` after class/function definitions. `result.syntax_valid` was `True`, but `compile(result.compressed, "<test>", "exec")` failed with `SyntaxError: from __future__ imports must occur at the beginning of the file`. After this patch, the focused regression and full code-compressor test file pass locally, and the regression now directly asserts that the future import appears before executable definitions. - Not tested: full repository pytest, `mypy headroom`, and a broad corpus run over third-party source files. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes This PR is now scoped to the stable compile-time failure path in #1233. The broader syntax-failure rate from the issue may still need corpus-level follow-up. Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
Rocker Zhang
|
5e0bb69725
|
fix(code): verify a real parse in tree-sitter availability check (#1231) (#1299)
## Description `is_tree_sitter_available()` / `_check_tree_sitter_available()` in `headroom/transforms/code_compressor.py` return `True` based on importing `tree_sitter_language_pack` alone, without ever constructing a parser or attempting a parse. When the installed pack/parser combination is ABI-incompatible, `get_parser`/`parse` raises at runtime; the caller catches it and silently falls back to the lossy text compressor, while the availability flag and startup banner still report code-aware as on. This is the defensive half that the `<1.0` pin in #1234 does not cover: if that cap is ever lifted, the availability signal silently lies again. Follow-up to #1231. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Make `_check_tree_sitter_available()` construct a parser and parse a tiny snippet, returning `True` only if it yields a real `module` AST instead of trusting an import. - Add `_tree_sitter_importable()` for the cheap import-only probe, and use it to guard parser construction so the real-parse check cannot recurse. - Add tests asserting the check is `False` when parsing raises and `True` on a real parse, plus that AST compression runs for python/rust without falling back. ## Testing - [ ] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # pytest tests/test_transforms/test_code_compressor.py -> passed locally (tree-sitter-language-pack 0.13.0) # ruff check . and ruff format --check . pass locally on the rebased branch. # Full pytest suite / mypy not run locally; left to CI. ``` ## Real Behavior Proof - Environment: local repo on tree-sitter-language-pack 0.13.0, tree-sitter 0.25.2, Python 3.12, Linux - Exact command / steps: call `is_tree_sitter_available()`, then run `pytest tests/test_transforms/test_code_compressor.py` - Observed result: with a working pack the probe parses and returns `True` (code-aware runs, strategy `CODE_AWARE` rather than the kompress fallback); the new `test_check_tree_sitter_available_false_when_parse_broken` confirms that when parsing raises the check now returns `False` instead of the old import-only `True`, so the lossy fallback is no longer entered silently. - Not tested: reproducing the specific ABI-incompatible 1.x pack combo against a live install (covered instead by a mocked broken parse in the test) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable |
||
|
chopratejas
|
72eebd4fe6 |
Fix CI test, bump to 0.5.7
- test_nested_functions: guard syntax_valid assert behind is_tree_sitter_available() (CI doesn't have tree-sitter) - Bump version to 0.5.7 |
||
|
chopratejas
|
3290a3d582 |
Remove LLMLingua: Kompress is the sole text compressor
LLMLingua was the original ML text compressor (BERT-based). Kompress (ModernBERT, trained on 330K structured tool outputs) replaced it with better compression quality and simpler architecture. Removed across 35 files: - Deleted headroom/transforms/llmlingua_compressor.py - Deleted tests/test_transforms/test_llmlingua_compressor.py - Deleted tests/test_proxy_llmlingua.py - Removed all enable_llmlingua config, _get_llmlingua methods, LLMLingua fallback paths, LLMLINGUA strategy enum values - Removed CLI flags, model configs, compression handler references - Simplified ContentRouter: Kompress is primary and only text compressor |
||
|
Tejas Chopra
|
4a655fbc6e |
feat: provider-aware prefix cache tracking and combined savings dashboard
Add per-provider prefix cache metrics (Anthropic/OpenAI/Google/Bedrock) with correct economics (read discounts, write premiums, bust detection). Model-aware bust detection excludes cold starts when switching models. Dashboard hero metric shows combined savings (compression + cache) with per-provider breakdown table, efficiency bar, and hit rate tracking. - Add _CACHE_ECONOMICS dict and _build_prefix_cache_stats() helper - Track cache_by_provider with per-model cold start awareness - Add _merge_cost_stats() to combine compression + cache savings - Dashboard: "Prefix Cache Impact" section with provider breakdown - Dashboard: hero "Total Savings" shows compress + cache breakdown - Fix ruff (unused var, quoted annotations) and mypy type errors - Refactor code_compressor to data-driven language config Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
chopratejas
|
9c31b22bff |
feat(code): add semantic symbol importance to CodeAwareCompressor
CodeAwareCompressor now analyzes intra-file symbol relationships before compression, using tree-sitter AST walks to count references, map call graphs, and detect public/private visibility. This replaces uniform "keep first N body lines" compression with budget-based allocation driven by the existing target_compression_rate config. Key design decisions: - Distribution-based scoring (min-max normalized within each file) so it adapts to any file structure: utility libs, test files, orchestrators - Budget allocation: target_compression_rate determines total body line budget, distributed proportionally to importance × body size - max_body_lines respected as a hard cap over budget allocation - Context-aware: the existing `context` parameter now boosts symbols matching the user's task (word-boundary matching, not substring) - Qualified names (ClassName.method) internally to avoid collisions between identically-named methods in different classes - Omitted comments include call graph info from AST analysis - Zero new dependencies — uses tree-sitter already in headroom[code] - semantic_analysis=True by default, fully backward-compatible when False |
||
|
chopratejas
|
2ce26438a0 |
Integrate DynamicContentDetector into CacheAligner (Phase 1)
- Add DynamicContentDetector integration for comprehensive dynamic content detection (20+ patterns vs previous 4 date patterns) - New detection: UUIDs, API keys, JWT tokens, Unix timestamps, request/trace IDs, hex hashes (MD5/SHA1/SHA256), version numbers, high-entropy strings - Add CacheAlignerConfig options: use_dynamic_detector, detection_tiers, extra_dynamic_labels, entropy_threshold - Maintain backward compatibility with legacy date-only mode - Add 25 new comprehensive tests for Phase 1 functionality - Fix code compressor fallback test to properly mock LLMLingua availability Expected cache hit improvement: 30-50% by extracting more dynamic content |
||
|
chopratejas
|
905c229251 |
Add AST-based code compression and custom model configuration
CodeAwareCompressor: - Tree-sitter based AST parsing for Python, JS, TS, Go, Rust, Java, C, C++ - Preserves imports, signatures, type annotations, error handlers - Guarantees syntactically valid output - Uses tree-sitter-language-pack for broad language support ContentRouter: - Intelligent compression orchestrator - Auto-routes content to optimal compressor based on type detection - Source hint support for high-confidence routing Custom Model Configuration: - HEADROOM_MODEL_LIMITS env var and ~/.headroom/models.json support - Pattern-based inference for unknown models (opus/sonnet/haiku tiers) - Support for Claude 4.5, Claude 4, o3, o3-mini - Graceful fallback - never crashes on unknown models |