2 commits

Author	SHA1	Message	Date
Parideboy	6d5516dcb8	feat(code): add PHP support to CodeAwareCompressor (#2423) ... ## Description Adds PHP to `CodeAwareCompressor`, fixing #201. PHP was already *detected* as code (Magika labels in `headroom/compression/detector.py` include `php`, and the Rust `magika_detector.rs` lists it too) but there was no PHP `LangConfig`, so PHP content silently passed through uncompressed. This wires PHP through the tree-sitter compression path following the C# pattern (the most recently added, fully functional language — deliberately not the quarantined Perl path). A secondary detection bug is fixed along the way: PHP's `$variables` match Perl's prefilter regex, and the existing Perl-dominance guard in `detect_language` returned `UNKNOWN` for PHP files. An explicit `<?php` open tag — which no Perl source contains — now drops Perl from the candidate set before that guard runs. ## Type of Change - [ ] Bug fix - [x] New feature - [ ] Documentation update - [ ] Refactor - [ ] Other ## Changes Made - `headroom/transforms/code_compressor.py`: `CodeLanguage.PHP` + `phtml`/`php5`/`php7`/`php8` aliases; PHP `LangConfig` built from the actual tree-sitter-php grammar (node names verified by parsing samples): `namespace_use_declaration` imports, `function_definition`/`method_declaration` functions, `class_declaration`/`interface_declaration`/`trait_declaration` classes, `enum_declaration` types, `declaration_list` class bodies, `compound_statement` function bodies. `namespace_definition` maps to `package_node` so statement-scoped `namespace App;` hoists ahead of the `use` imports (required PHP ordering); the rare block-scoped `namespace A { }` form takes the same path and is preserved verbatim — valid output, just no compression inside the block. PHP prefilter regexes added; supported-languages error message updated; `<?php`-tag Perl disambiguation in `detect_language`. - `headroom/transforms/content_detector.py`: `php` entry in `_CODE_PATTERNS` so raw PHP classifies as `SOURCE_CODE` and reaches the code-aware route. - `tests/test_transforms/test_code_compressor.py`: new `TestPhpSupport` mirroring `TestCSharpSupport` — signatures preserved / bodies elided, `<?php` → `namespace` → `use` → declarations ordering, auto-detection despite the Perl sigil overlap, alias coercion, malformed passthrough. - `tests/test_code_compressor_language_alias.py`: `php` in the canonical list, `phtml` in the alias table. - `docs/content/docs/code-compression.mdx`: PHP added to the Tier 2 supported-languages row. No new dependency: `tree-sitter-language-pack` (the existing `[code]` extra) already ships the PHP grammar. No Rust changes needed. ## Testing - [x] New unit tests added and passing - [x] Full affected test suites pass locally **Test Output** ``` $ python -m pytest tests/test_transforms/test_code_compressor.py tests/test_code_compressor_language_alias.py -q ============================= 120 passed in 7.81s ============================= $ python -m pytest tests/test_transforms/ -q 3 failed, 443 passed # the 3 failures (kompress ONNX thread caps, kompress size gate, # text_crusher unicode parity) reproduce identically on a clean # upstream/main checkout in this environment — pre-existing local # ONNX runtime quirks, unrelated to this change $ ruff check . (0.15.17, CI-pinned) → All checks passed! | ruff format --check → clean $ mypy headroom/transforms/code_compressor.py headroom/transforms/content_detector.py --ignore-missing-imports Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13, tree-sitter + tree-sitter-language-pack (<1.0) installed, branch `feat/201-php-code-compression` off `upstream/main`. - Exact command / steps: parsed PHP samples (namespaced class w/ methods, block-scoped namespace, mixed HTML+PHP) with `tree_sitter_language_pack.get_parser('php')` to verify every node name used in the config; then ran `CodeAwareCompressor().compress(php_code, language="php")` and `compress(php_code)` (auto-detection) on a 48-line realistic service class. - Observed result: explicit and auto-detected paths both return `language=CodeLanguage.PHP`, `compression_ratio=0.64`, `syntax_valid=True`; method bodies elided to `// [N lines omitted]` while `<?php`, `namespace`, `use` lines, class header, and all signatures are preserved verbatim in the original order. Before the detection fix, auto-detection returned `UNKNOWN` (Perl prefilter dominance) — reproduced and then verified fixed. - Not tested: exotic PHP shapes (heredoc-heavy code, attributes `#[...]` on methods, interleaved multi-`<?php ?>` HTML templates beyond the basic mixed case); these fall back to verbatim preservation via the uncaptured-node pass or malformed-passthrough, both of which are covered by tests for the simple cases. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>	2026-07-31 15:54:13 -07:00
Abhay Singh	27ddde1f5e	fix(transforms/code): coerce language aliases instead of raising (#1975) ... ## Description `CodeAwareCompressor.compress()` picks the language for AST-based compression like this (`headroom/transforms/code_compressor.py`): ```python if language: detected_lang = CodeLanguage(language.lower()) # <-- raises on anything not an exact enum value confidence = 1.0 elif self.config.language_hint: detected_lang = CodeLanguage(self.config.language_hint.lower()) confidence = 1.0 else: detected_lang, confidence = detect_language(code) ``` `CodeLanguage` only accepts `python`/`javascript`/`typescript`/`go`/`rust`/`java`/`c`/`cpp`/`perl`. The very common markdown fence tags and hints — `js`, `ts`, `py`, `jsx`, `tsx`, `node`, `rs`, `c++` — are **not** enum values, so `CodeLanguage("js")` raises `ValueError`. That construction is *above* the method's own `try/except`, so: - **Direct callers** — `CodeAwareCompressor().compress(code, language="js")` and the module-level `compress_code(code, language="js")` — crash with an uncaught `ValueError`. - **In the router (mixed content):** `split_into_sections` extracts the raw fence tag (`_CODE_FENCE_PATTERN` captures `\w*`, e.g. `js`) into `ContentSection.language`, and that string is passed straight into `compress(...)`. The `ValueError` is swallowed by the outer `try/except` in the strategy dispatch, so a ` ```js ` / ` ```ts ` / ` ```py ` block silently **skips code-aware compression** even when `enable_code_aware=True`, falling back to the generic path. So the three most common web/scripting languages, written with their usual fence tags, never get the structure-aware compressor. Closes: no issue filed — found while auditing the code-compression language path. ## Fix Add a `coerce_language()` helper that maps common aliases/fence tags to the canonical `CodeLanguage` and returns `CodeLanguage.UNKNOWN` (never raises) for anything unrecognized. `compress()` now coerces the hint and, when the result is `UNKNOWN`, falls back to content-based `detect_language(code)` instead of constructing the enum directly: ```python if language: detected_lang = coerce_language(language) if detected_lang == CodeLanguage.UNKNOWN: detected_lang, confidence = detect_language(code) else: confidence = 1.0 ``` ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/transforms/code_compressor.py`: add `_LANGUAGE_ALIASES` and `coerce_language()`; use them in `compress()` for both the `language` argument and `config.language_hint`, with a content-detection fallback on `UNKNOWN`. - `tests/test_code_compressor_language_alias.py`: cover alias mapping, canonical passthrough, case/whitespace handling, unknown-returns-UNKNOWN (no `ValueError`), and that `compress(language="js")` no longer raises. ## Testing - [x] New regression tests added (`tests/test_code_compressor_language_alias.py`) - [x] Linting passes (`ruff check`) and formatting is clean (`ruff format --check`) - [ ] Full `pytest` deferred to CI (local-OOM reason below). ```text $ uv run ruff check headroom/transforms/code_compressor.py tests/test_code_compressor_language_alias.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.10, headroom from this branch. Importing `headroom` pulls in the torch/transformers stack and a full `pytest` gets OOM-killed on this box, so I verified the coercion logic with a dependency-free script (replicating the enum + helper) and left the full pytest to CI. - Exact command / steps: ran the common aliases and the canonical values through both the old `CodeLanguage(value.lower())` construction and the new `coerce_language()`. - Observed result: the old construction raises `ValueError` on every alias (the crash / silent-skip); the new helper maps them and never raises: ```text OK alias 'js': old raised ValueError -> new maps to javascript OK alias 'ts': old raised ValueError -> new maps to typescript OK alias 'py': old raised ValueError -> new maps to python OK alias 'jsx': old raised ValueError -> new maps to javascript OK alias 'node': old raised ValueError -> new maps to javascript OK canonical values pass through OK case-insensitive + trimmed OK unknown -> UNKNOWN (no ValueError) LANGUAGE COERCION VERIFIED ``` - Not tested: running a full mixed-content document with ` ```js ` fences through a booted compression pipeline (needs the heavy stack). The unit tests exercise the coercion directly and the `compress(language="js")` entry point. Full local `pytest` deferred to CI (OOM, per above). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes — ran lint + a standalone logic check; full pytest deferred to CI (local OOM, disclosed above) - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - No new dependencies; a small lookup table plus a helper and a call-site change. - @JerrettDavis tagging you — this one silently disables code-aware compression for the most common fence tags (`js`/`ts`/`py`), so it may be worth a look when you have a moment.	2026-07-10 23:57:33 -05:00