mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
ec55ddcfb3
Refs #1664 ## Description First-class C# support in `CodeAwareCompressor` via the tree-sitter `csharp` grammar, at parity with Java/C++/Rust: `using` directives, namespace headers, and type/member signatures preserved verbatim; method/constructor/destructor/operator/local-function bodies compressed; malformed input passes through unchanged. **No new dependencies** — the grammar ships inside the already-pinned `tree-sitter-language-pack==0.13.0` (resolves only as `"csharp"`; `c_sharp`/`cs` raise `LookupError`). Spec and maintainer go-ahead in the issue. Closes #1664 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `CodeLanguage.CSHARP` + full `_LANG_CONFIGS` entry; `_LANGUAGE_PREFILTER` and `content_detector` patterns chosen to be C#-distinctive (so Java doesn't mis-tag). - New data-driven `LangConfig` fields (pattern of #1334's `class_body_node_types`): `container_node_types` — block-scoped `namespace { }` routed through class compression so members compress without the wrapper being re-emitted verbatim; `opaque_node_types` — `#if`…`#endif` wrappers preserved verbatim without recursion (recursing + wrapper re-emit duplicated whole files, up to ~1.9x input on real repos); `#if` blocks wrapping only usings are emitted with the imports so they stay ahead of type declarations. - Shared-path fixes surfaced by real C# repos, each guarded and covered by a fail-before test: keep an Allman `{` on its own line in class reconstruction (K&R path byte-for-byte unchanged; Allman Java now compresses instead of falling back); line-based child extraction no longer swallows the following line for nodes ending at column 0 (C# `#region`/`#endregion` span their trailing newline — the over-slice duplicated the next member's signature or the closing brace); uncaptured top-level nodes preceding the first captured node (license banners, `#region License`) are emitted first instead of relocated below the code (tree-sitter-c-sharp rejects top-level `#region` after a type declaration, so relocation forfeited compression for the whole file). - `TestCSharpSupport` (8 tests) + a C# case in the parametrized member-container test; CHANGELOG entry. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_transforms/test_code_compressor.py -q 2 failed, 78 passed, 1 warning, 4 errors # the 2 failures / 4 errors reproduce # identically on main in the same env # (network-dependent tokenizer setup) Fail-before: with both changed sources reverted to main, the new C#-scoped selection reports "10 failed, 5 passed" (the 5 other languages keep passing); on the branch: "15 passed". $ ruff check headroom/transforms/code_compressor.py headroom/transforms/content_detector.py tests/test_transforms/test_code_compressor.py All checks passed! $ ruff format --check <same files> 3 files already formatted ``` ## Real Behavior Proof - Environment: Linux 6.8.0 aarch64, Python 3.10.12; `uv run --no-project --with "tree-sitter-language-pack==0.13.0" --with "tree-sitter>=0.25.2,<0.26" --with "pydantic>=2.0.0"`; real `CodeAwareCompressor` (`CodeCompressorConfig(enable_ccr=False)`, otherwise defaults), no mocks. - Exact command / steps: cloned two real .NET repos at depth 1 (`github.com/JamesNK/Newtonsoft.Json @ 4f73e74`, `github.com/App-vNext/Polly @ 7a1d10f`), ran `python proof_csharp.py <repo>` over every `.cs` file (chars/4 token estimate; tiktoken BPE download unavailable in my sandbox). Script in the collapsed section below. - Observed result: 16.1% tokens saved on Newtonsoft.Json (945/945 syntax-valid), 37.8% on Polly (797/797 syntax-valid), zero content duplication; full output: ```text repo: Newtonsoft.Json (945 .cs files) tokens before: 1,777,691 after: 1,490,629 saved: 287,062 (16.1%) files compressed: 479 pass-through: 466 inflated(>before): 19 syntax_valid: 945/945 latency ms P50: 0.7 P95: 18.7 P99: 44.1 max: 255.0 mean: 3.5 repo: Polly (797 .cs files) tokens before: 1,100,523 after: 684,303 saved: 416,220 (37.8%) files compressed: 693 pass-through: 104 inflated(>before): 15 syntax_valid: 797/797 latency ms P50: 0.8 P95: 11.6 P99: 28.9 max: 74.1 mean: 2.4 ``` After rebasing onto current `main` (which touched the same transform files via #1906/#1747/#1668) I re-ran the Polly proof on the rebased tree: 37.8% saved, 797/797 syntax-valid, P99 28.5ms — unchanged. Signatures/properties verbatim, bodies elided with call summaries, `using` order and preproc balance intact; residual "inflated" files are +2…+209 chars of assembly blank lines, not duplicated content. Newtonsoft is the adversarial case (multi-targeting: heavy `#if`, `#region`, Allman) — its conditional regions stay verbatim by design. Latency at parity with Java (<50ms P99; max is the pre-existing symbol-analysis cost on ~1800+-line files, shared with other languages). - Not tested: proxy end-to-end path with C# through `ContentRouter` (tested the `CodeAwareCompressor` API directly); CCR retrieval round-trips (`enable_ccr=False` in proof runs); exact tiktoken counts (chars/4 estimate — relative ratios are tokenizer-independent); Windows/macOS; full native `uv run pytest` with the Rust extension (ran the complete `test_code_compressor.py` in a lightweight venv; its 2 failures/4 errors reproduce identically on `main`); `mypy`. <details> <summary>proof_csharp.py (reproducible)</summary> ```python """Real behavior proof: run the real CodeAwareCompressor over a .NET repo.""" import pathlib import statistics import sys import time from headroom.transforms.code_compressor import ( CodeAwareCompressor, CodeCompressorConfig, ) try: import tiktoken ENC = tiktoken.get_encoding("cl100k_base") def toks(s: str) -> int: return len(ENC.encode(s, disallowed_special=())) except Exception: def toks(s: str) -> int: return len(s) // 4 target = pathlib.Path(sys.argv[1]) comp = CodeAwareCompressor(CodeCompressorConfig(enable_ccr=False)) tot_before = tot_after = 0 n_files = n_compressed = n_valid = n_passthrough = n_inflated = 0 times_ms: list[float] = [] for f in sorted(target.rglob("*.cs")): try: code = f.read_text(encoding="utf-8-sig", errors="replace") except OSError: continue t0 = time.perf_counter() r = comp.compress(code, language="csharp") times_ms.append((time.perf_counter() - t0) * 1000) n_files += 1 b, a = toks(code), toks(r.compressed) tot_before += b tot_after += a if r.compressed == code: n_passthrough += 1 else: n_compressed += 1 if r.syntax_valid: n_valid += 1 if a > b: n_inflated += 1 times_ms.sort() p = lambda q: times_ms[min(int(len(times_ms) * q), len(times_ms) - 1)] print(f"repo: {target.name} ({n_files} .cs files)") print(f" tokens before: {tot_before:,} after: {tot_after:,} saved: {tot_before - tot_after:,} ({(1 - tot_after / tot_before) * 100:.1f}%)") print(f" files compressed: {n_compressed} pass-through: {n_passthrough} inflated(>before): {n_inflated}") print(f" syntax_valid: {n_valid}/{n_files}") print(f" latency ms P50: {p(0.50):.1f} P95: {p(0.95):.1f} P99: {p(0.99):.1f} max: {times_ms[-1]:.1f} mean: {statistics.mean(times_ms):.1f}") ``` </details> ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — terminal evidence above. ## Additional Notes - Dependency justification: none added, none bumped; the `csharp` grammar is inside the already-pinned `tree-sitter-language-pack==0.13.0` wheel; `uv.lock` untouched. - Architecture: malformed input passes through byte-identical; every risky construct prefers the false negative (verbatim) over corruption; invalid reassembly falls back to the original via the existing validation gate (observed live); no new imports at module load; P99 <50ms on both proof repos. - Known v1 limitations (deliberate false negatives, possible follow-ups): expression-bodied members and property accessor bodies stay verbatim; declarations inside `#if` regions stay verbatim. - Related pre-existing finding, out of scope: C/C++ exhibit the same `#if`-wrapper duplication on `main` (an `#if`-wrapped C++ class is emitted twice, ratio 1.62). Happy to file separately. - `mypy` unchecked above because I did not run it in my environment. |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| test_code_compressor.py | ||
| test_code_compressor_cjk.py | ||
| test_content_router.py | ||
| test_detect_fallback_1123.py | ||
| test_diff_compressor.py | ||
| test_diff_compressor_rust_parity.py | ||
| test_html_extractor.py | ||
| test_kompress_compressor.py | ||
| test_kompress_deadline.py | ||
| test_kompress_size_gate.py | ||
| test_ort_dylib.py | ||
| test_pipeline_waste_signal_limit.py | ||
| test_read_lifecycle.py | ||
| test_smart_crusher_attribution.py | ||
| test_smart_crusher_audit_safe.py | ||
| test_smart_crusher_bugs.py | ||
| test_smart_crusher_ccr_retrieve_exemption.py | ||
| test_smart_crusher_ccr_roundtrip.py | ||
| test_smart_crusher_lossless_default.py | ||
| test_smart_crusher_rust_parity.py | ||
| test_tag_protector.py | ||
| test_text_crusher.py | ||
| test_text_crusher_parity.py | ||
| test_text_crusher_routing.py | ||
| test_tree_sitter_thread_safety.py | ||