fix(transforms/code): coerce language aliases instead of raising (#1975)
## Description
`CodeAwareCompressor.compress()` picks the language for AST-based
compression like this
(`headroom/transforms/code_compressor.py`):
```python
if language:
detected_lang = CodeLanguage(language.lower()) # <-- raises on anything not an exact enum value
confidence = 1.0
elif self.config.language_hint:
detected_lang = CodeLanguage(self.config.language_hint.lower())
confidence = 1.0
else:
detected_lang, confidence = detect_language(code)
```
`CodeLanguage` only accepts
`python`/`javascript`/`typescript`/`go`/`rust`/`java`/`c`/`cpp`/`perl`.
The very common markdown fence tags and hints — `js`, `ts`, `py`, `jsx`,
`tsx`, `node`, `rs`,
`c++` — are **not** enum values, so `CodeLanguage("js")` raises
`ValueError`. That construction
is *above* the method's own `try/except`, so:
- **Direct callers** — `CodeAwareCompressor().compress(code,
language="js")` and the module-level
`compress_code(code, language="js")` — crash with an uncaught
`ValueError`.
- **In the router (mixed content):** `split_into_sections` extracts the
raw fence tag
(`_CODE_FENCE_PATTERN` captures `\w*`, e.g. `js`) into
`ContentSection.language`, and that string
is passed straight into `compress(...)`. The `ValueError` is swallowed
by the outer `try/except`
in the strategy dispatch, so a ` ```js ` / ` ```ts ` / ` ```py ` block
silently **skips
code-aware compression** even when `enable_code_aware=True`, falling
back to the generic path.
So the three most common web/scripting languages, written with their
usual fence tags, never get
the structure-aware compressor.
Closes: no issue filed — found while auditing the code-compression
language path.
## Fix
Add a `coerce_language()` helper that maps common aliases/fence tags to
the canonical
`CodeLanguage` and returns `CodeLanguage.UNKNOWN` (never raises) for
anything unrecognized.
`compress()` now coerces the hint and, when the result is `UNKNOWN`,
falls back to
content-based `detect_language(code)` instead of constructing the enum
directly:
```python
if language:
detected_lang = coerce_language(language)
if detected_lang == CodeLanguage.UNKNOWN:
detected_lang, confidence = detect_language(code)
else:
confidence = 1.0
```
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/transforms/code_compressor.py`: add `_LANGUAGE_ALIASES` and
`coerce_language()`; use them in `compress()` for both the `language`
argument and `config.language_hint`, with a content-detection fallback
on `UNKNOWN`.
- `tests/test_code_compressor_language_alias.py`: cover alias mapping,
canonical passthrough, case/whitespace handling, unknown-returns-UNKNOWN
(no `ValueError`), and that `compress(language="js")` no longer raises.
## Testing
- [x] New regression tests added
(`tests/test_code_compressor_language_alias.py`)
- [x] Linting passes (`ruff check`) and formatting is clean (`ruff
format --check`)
- [ ] Full `pytest` deferred to CI (local-OOM reason below).
```text
$ uv run ruff check headroom/transforms/code_compressor.py tests/test_code_compressor_language_alias.py
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.10, headroom from this branch.
Importing `headroom` pulls in the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I verified the coercion logic
with a dependency-free script (replicating the enum + helper) and left
the full pytest to CI.
- Exact command / steps: ran the common aliases and the canonical values
through both the old `CodeLanguage(value.lower())` construction and the
new `coerce_language()`.
- Observed result: the old construction raises `ValueError` on every
alias (the crash / silent-skip); the new helper maps them and never
raises:
```text
OK alias 'js': old raised ValueError -> new maps to javascript
OK alias 'ts': old raised ValueError -> new maps to typescript
OK alias 'py': old raised ValueError -> new maps to python
OK alias 'jsx': old raised ValueError -> new maps to javascript
OK alias 'node': old raised ValueError -> new maps to javascript
OK canonical values pass through
OK case-insensitive + trimmed
OK unknown -> UNKNOWN (no ValueError)
LANGUAGE COERCION VERIFIED
```
- Not tested: running a full mixed-content document with ` ```js `
fences through a booted compression pipeline (needs the heavy stack).
The unit tests exercise the coercion directly and the
`compress(language="js")` entry point. Full local `pytest` deferred to
CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — ran
lint + a standalone logic check; full pytest deferred to CI (local OOM,
disclosed above)
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- No new dependencies; a small lookup table plus a helper and a
call-site change.
- @JerrettDavis tagging you — this one silently disables code-aware
compression for the most common fence tags (`js`/`ts`/`py`), so it may
be worth a look when you have a moment.
2026-07-11 10:27:33 +05:30
|
|
|
"""Regression tests for language-hint / fence-tag coercion in code_compressor.
|
|
|
|
|
|
|
|
|
|
`CodeAwareCompressor.compress(code, language=...)` used to build the language
|
|
|
|
|
with `CodeLanguage(language.lower())`, which raises `ValueError` for anything
|
|
|
|
|
that is not an exact enum value. Common markdown fence tags and hints — `js`,
|
|
|
|
|
`ts`, `py` — are not enum values, so:
|
|
|
|
|
|
|
|
|
|
* direct callers (`compress(code, language="js")`) crashed, and
|
|
|
|
|
* inside the router the ValueError was swallowed, so ` ```js ` / ` ```ts ` /
|
|
|
|
|
` ```py ` fenced blocks silently skipped code-aware compression.
|
|
|
|
|
|
|
|
|
|
`coerce_language` maps aliases to the canonical language and returns UNKNOWN
|
|
|
|
|
(never raises) for unrecognized tags, letting the caller fall back to
|
|
|
|
|
content-based detection.
|
|
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
import pytest
|
|
|
|
|
|
|
|
|
|
from headroom.transforms.code_compressor import CodeLanguage, coerce_language
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
|
|
|
"alias,expected",
|
|
|
|
|
[
|
|
|
|
|
("js", CodeLanguage.JAVASCRIPT),
|
|
|
|
|
("jsx", CodeLanguage.JAVASCRIPT),
|
|
|
|
|
("node", CodeLanguage.JAVASCRIPT),
|
|
|
|
|
("ts", CodeLanguage.TYPESCRIPT),
|
|
|
|
|
("tsx", CodeLanguage.TYPESCRIPT),
|
|
|
|
|
("py", CodeLanguage.PYTHON),
|
|
|
|
|
("python3", CodeLanguage.PYTHON),
|
|
|
|
|
("golang", CodeLanguage.GO),
|
|
|
|
|
("rs", CodeLanguage.RUST),
|
|
|
|
|
("c++", CodeLanguage.CPP),
|
feat(code): add PHP support to CodeAwareCompressor (#2423)
## Description
Adds PHP to `CodeAwareCompressor`, fixing #201. PHP was already
*detected* as code (Magika labels in `headroom/compression/detector.py`
include `php`, and the Rust `magika_detector.rs` lists it too) but there
was no PHP `LangConfig`, so PHP content silently passed through
uncompressed. This wires PHP through the tree-sitter compression path
following the C# pattern (the most recently added, fully functional
language — deliberately not the quarantined Perl path).
A secondary detection bug is fixed along the way: PHP's `$variables`
match Perl's prefilter regex, and the existing Perl-dominance guard in
`detect_language` returned `UNKNOWN` for PHP files. An explicit `<?php`
open tag — which no Perl source contains — now drops Perl from the
candidate set before that guard runs.
## Type of Change
- [ ] Bug fix
- [x] New feature
- [ ] Documentation update
- [ ] Refactor
- [ ] Other
## Changes Made
- `headroom/transforms/code_compressor.py`: `CodeLanguage.PHP` +
`phtml`/`php5`/`php7`/`php8` aliases; PHP `LangConfig` built from the
actual tree-sitter-php grammar (node names verified by parsing samples):
`namespace_use_declaration` imports,
`function_definition`/`method_declaration` functions,
`class_declaration`/`interface_declaration`/`trait_declaration` classes,
`enum_declaration` types, `declaration_list` class bodies,
`compound_statement` function bodies. `namespace_definition` maps to
`package_node` so statement-scoped `namespace App;` hoists ahead of the
`use` imports (required PHP ordering); the rare block-scoped `namespace
A { }` form takes the same path and is preserved verbatim — valid
output, just no compression inside the block. PHP prefilter regexes
added; supported-languages error message updated; `<?php`-tag Perl
disambiguation in `detect_language`.
- `headroom/transforms/content_detector.py`: `php` entry in
`_CODE_PATTERNS` so raw PHP classifies as `SOURCE_CODE` and reaches the
code-aware route.
- `tests/test_transforms/test_code_compressor.py`: new `TestPhpSupport`
mirroring `TestCSharpSupport` — signatures preserved / bodies elided,
`<?php` → `namespace` → `use` → declarations ordering, auto-detection
despite the Perl sigil overlap, alias coercion, malformed passthrough.
- `tests/test_code_compressor_language_alias.py`: `php` in the canonical
list, `phtml` in the alias table.
- `docs/content/docs/code-compression.mdx`: PHP added to the Tier 2
supported-languages row.
No new dependency: `tree-sitter-language-pack` (the existing `[code]`
extra) already ships the PHP grammar. No Rust changes needed.
## Testing
- [x] New unit tests added and passing
- [x] Full affected test suites pass locally
**Test Output**
```
$ python -m pytest tests/test_transforms/test_code_compressor.py tests/test_code_compressor_language_alias.py -q
============================= 120 passed in 7.81s =============================
$ python -m pytest tests/test_transforms/ -q
3 failed, 443 passed # the 3 failures (kompress ONNX thread caps, kompress size gate,
# text_crusher unicode parity) reproduce identically on a clean
# upstream/main checkout in this environment — pre-existing local
# ONNX runtime quirks, unrelated to this change
$ ruff check . (0.15.17, CI-pinned) → All checks passed! | ruff format --check → clean
$ mypy headroom/transforms/code_compressor.py headroom/transforms/content_detector.py --ignore-missing-imports
Success: no issues found in 2 source files
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.13, tree-sitter +
tree-sitter-language-pack (<1.0) installed, branch
`feat/201-php-code-compression` off `upstream/main`.
- Exact command / steps: parsed PHP samples (namespaced class w/
methods, block-scoped namespace, mixed HTML+PHP) with
`tree_sitter_language_pack.get_parser('php')` to verify every node name
used in the config; then ran `CodeAwareCompressor().compress(php_code,
language="php")` and `compress(php_code)` (auto-detection) on a 48-line
realistic service class.
- Observed result: explicit and auto-detected paths both return
`language=CodeLanguage.PHP`, `compression_ratio=0.64`,
`syntax_valid=True`; method bodies elided to `// [N lines omitted]`
while `<?php`, `namespace`, `use` lines, class header, and all
signatures are preserved verbatim in the original order. Before the
detection fix, auto-detection returned `UNKNOWN` (Perl prefilter
dominance) — reproduced and then verified fixed.
- Not tested: exotic PHP shapes (heredoc-heavy code, attributes `#[...]`
on methods, interleaved multi-`<?php ?>` HTML templates beyond the basic
mixed case); these fall back to verbatim preservation via the
uncaptured-node pass or malformed-passthrough, both of which are covered
by tests for the simple cases.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-08-01 00:54:13 +02:00
|
|
|
("phtml", CodeLanguage.PHP),
|
fix(transforms/code): coerce language aliases instead of raising (#1975)
## Description
`CodeAwareCompressor.compress()` picks the language for AST-based
compression like this
(`headroom/transforms/code_compressor.py`):
```python
if language:
detected_lang = CodeLanguage(language.lower()) # <-- raises on anything not an exact enum value
confidence = 1.0
elif self.config.language_hint:
detected_lang = CodeLanguage(self.config.language_hint.lower())
confidence = 1.0
else:
detected_lang, confidence = detect_language(code)
```
`CodeLanguage` only accepts
`python`/`javascript`/`typescript`/`go`/`rust`/`java`/`c`/`cpp`/`perl`.
The very common markdown fence tags and hints — `js`, `ts`, `py`, `jsx`,
`tsx`, `node`, `rs`,
`c++` — are **not** enum values, so `CodeLanguage("js")` raises
`ValueError`. That construction
is *above* the method's own `try/except`, so:
- **Direct callers** — `CodeAwareCompressor().compress(code,
language="js")` and the module-level
`compress_code(code, language="js")` — crash with an uncaught
`ValueError`.
- **In the router (mixed content):** `split_into_sections` extracts the
raw fence tag
(`_CODE_FENCE_PATTERN` captures `\w*`, e.g. `js`) into
`ContentSection.language`, and that string
is passed straight into `compress(...)`. The `ValueError` is swallowed
by the outer `try/except`
in the strategy dispatch, so a ` ```js ` / ` ```ts ` / ` ```py ` block
silently **skips
code-aware compression** even when `enable_code_aware=True`, falling
back to the generic path.
So the three most common web/scripting languages, written with their
usual fence tags, never get
the structure-aware compressor.
Closes: no issue filed — found while auditing the code-compression
language path.
## Fix
Add a `coerce_language()` helper that maps common aliases/fence tags to
the canonical
`CodeLanguage` and returns `CodeLanguage.UNKNOWN` (never raises) for
anything unrecognized.
`compress()` now coerces the hint and, when the result is `UNKNOWN`,
falls back to
content-based `detect_language(code)` instead of constructing the enum
directly:
```python
if language:
detected_lang = coerce_language(language)
if detected_lang == CodeLanguage.UNKNOWN:
detected_lang, confidence = detect_language(code)
else:
confidence = 1.0
```
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/transforms/code_compressor.py`: add `_LANGUAGE_ALIASES` and
`coerce_language()`; use them in `compress()` for both the `language`
argument and `config.language_hint`, with a content-detection fallback
on `UNKNOWN`.
- `tests/test_code_compressor_language_alias.py`: cover alias mapping,
canonical passthrough, case/whitespace handling, unknown-returns-UNKNOWN
(no `ValueError`), and that `compress(language="js")` no longer raises.
## Testing
- [x] New regression tests added
(`tests/test_code_compressor_language_alias.py`)
- [x] Linting passes (`ruff check`) and formatting is clean (`ruff
format --check`)
- [ ] Full `pytest` deferred to CI (local-OOM reason below).
```text
$ uv run ruff check headroom/transforms/code_compressor.py tests/test_code_compressor_language_alias.py
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.10, headroom from this branch.
Importing `headroom` pulls in the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I verified the coercion logic
with a dependency-free script (replicating the enum + helper) and left
the full pytest to CI.
- Exact command / steps: ran the common aliases and the canonical values
through both the old `CodeLanguage(value.lower())` construction and the
new `coerce_language()`.
- Observed result: the old construction raises `ValueError` on every
alias (the crash / silent-skip); the new helper maps them and never
raises:
```text
OK alias 'js': old raised ValueError -> new maps to javascript
OK alias 'ts': old raised ValueError -> new maps to typescript
OK alias 'py': old raised ValueError -> new maps to python
OK alias 'jsx': old raised ValueError -> new maps to javascript
OK alias 'node': old raised ValueError -> new maps to javascript
OK canonical values pass through
OK case-insensitive + trimmed
OK unknown -> UNKNOWN (no ValueError)
LANGUAGE COERCION VERIFIED
```
- Not tested: running a full mixed-content document with ` ```js `
fences through a booted compression pipeline (needs the heavy stack).
The unit tests exercise the coercion directly and the
`compress(language="js")` entry point. Full local `pytest` deferred to
CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — ran
lint + a standalone logic check; full pytest deferred to CI (local OOM,
disclosed above)
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- No new dependencies; a small lookup table plus a helper and a
call-site change.
- @JerrettDavis tagging you — this one silently disables code-aware
compression for the most common fence tags (`js`/`ts`/`py`), so it may
be worth a look when you have a moment.
2026-07-11 10:27:33 +05:30
|
|
|
],
|
|
|
|
|
)
|
|
|
|
|
def test_coerce_language_maps_common_aliases(alias, expected):
|
|
|
|
|
assert coerce_language(alias) == expected
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
|
|
|
"canonical",
|
feat(code): add PHP support to CodeAwareCompressor (#2423)
## Description
Adds PHP to `CodeAwareCompressor`, fixing #201. PHP was already
*detected* as code (Magika labels in `headroom/compression/detector.py`
include `php`, and the Rust `magika_detector.rs` lists it too) but there
was no PHP `LangConfig`, so PHP content silently passed through
uncompressed. This wires PHP through the tree-sitter compression path
following the C# pattern (the most recently added, fully functional
language — deliberately not the quarantined Perl path).
A secondary detection bug is fixed along the way: PHP's `$variables`
match Perl's prefilter regex, and the existing Perl-dominance guard in
`detect_language` returned `UNKNOWN` for PHP files. An explicit `<?php`
open tag — which no Perl source contains — now drops Perl from the
candidate set before that guard runs.
## Type of Change
- [ ] Bug fix
- [x] New feature
- [ ] Documentation update
- [ ] Refactor
- [ ] Other
## Changes Made
- `headroom/transforms/code_compressor.py`: `CodeLanguage.PHP` +
`phtml`/`php5`/`php7`/`php8` aliases; PHP `LangConfig` built from the
actual tree-sitter-php grammar (node names verified by parsing samples):
`namespace_use_declaration` imports,
`function_definition`/`method_declaration` functions,
`class_declaration`/`interface_declaration`/`trait_declaration` classes,
`enum_declaration` types, `declaration_list` class bodies,
`compound_statement` function bodies. `namespace_definition` maps to
`package_node` so statement-scoped `namespace App;` hoists ahead of the
`use` imports (required PHP ordering); the rare block-scoped `namespace
A { }` form takes the same path and is preserved verbatim — valid
output, just no compression inside the block. PHP prefilter regexes
added; supported-languages error message updated; `<?php`-tag Perl
disambiguation in `detect_language`.
- `headroom/transforms/content_detector.py`: `php` entry in
`_CODE_PATTERNS` so raw PHP classifies as `SOURCE_CODE` and reaches the
code-aware route.
- `tests/test_transforms/test_code_compressor.py`: new `TestPhpSupport`
mirroring `TestCSharpSupport` — signatures preserved / bodies elided,
`<?php` → `namespace` → `use` → declarations ordering, auto-detection
despite the Perl sigil overlap, alias coercion, malformed passthrough.
- `tests/test_code_compressor_language_alias.py`: `php` in the canonical
list, `phtml` in the alias table.
- `docs/content/docs/code-compression.mdx`: PHP added to the Tier 2
supported-languages row.
No new dependency: `tree-sitter-language-pack` (the existing `[code]`
extra) already ships the PHP grammar. No Rust changes needed.
## Testing
- [x] New unit tests added and passing
- [x] Full affected test suites pass locally
**Test Output**
```
$ python -m pytest tests/test_transforms/test_code_compressor.py tests/test_code_compressor_language_alias.py -q
============================= 120 passed in 7.81s =============================
$ python -m pytest tests/test_transforms/ -q
3 failed, 443 passed # the 3 failures (kompress ONNX thread caps, kompress size gate,
# text_crusher unicode parity) reproduce identically on a clean
# upstream/main checkout in this environment — pre-existing local
# ONNX runtime quirks, unrelated to this change
$ ruff check . (0.15.17, CI-pinned) → All checks passed! | ruff format --check → clean
$ mypy headroom/transforms/code_compressor.py headroom/transforms/content_detector.py --ignore-missing-imports
Success: no issues found in 2 source files
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.13, tree-sitter +
tree-sitter-language-pack (<1.0) installed, branch
`feat/201-php-code-compression` off `upstream/main`.
- Exact command / steps: parsed PHP samples (namespaced class w/
methods, block-scoped namespace, mixed HTML+PHP) with
`tree_sitter_language_pack.get_parser('php')` to verify every node name
used in the config; then ran `CodeAwareCompressor().compress(php_code,
language="php")` and `compress(php_code)` (auto-detection) on a 48-line
realistic service class.
- Observed result: explicit and auto-detected paths both return
`language=CodeLanguage.PHP`, `compression_ratio=0.64`,
`syntax_valid=True`; method bodies elided to `// [N lines omitted]`
while `<?php`, `namespace`, `use` lines, class header, and all
signatures are preserved verbatim in the original order. Before the
detection fix, auto-detection returned `UNKNOWN` (Perl prefilter
dominance) — reproduced and then verified fixed.
- Not tested: exotic PHP shapes (heredoc-heavy code, attributes `#[...]`
on methods, interleaved multi-`<?php ?>` HTML templates beyond the basic
mixed case); these fall back to verbatim preservation via the
uncaptured-node pass or malformed-passthrough, both of which are covered
by tests for the simple cases.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-08-01 00:54:13 +02:00
|
|
|
["python", "javascript", "typescript", "go", "rust", "java", "c", "cpp", "perl", "php"],
|
fix(transforms/code): coerce language aliases instead of raising (#1975)
## Description
`CodeAwareCompressor.compress()` picks the language for AST-based
compression like this
(`headroom/transforms/code_compressor.py`):
```python
if language:
detected_lang = CodeLanguage(language.lower()) # <-- raises on anything not an exact enum value
confidence = 1.0
elif self.config.language_hint:
detected_lang = CodeLanguage(self.config.language_hint.lower())
confidence = 1.0
else:
detected_lang, confidence = detect_language(code)
```
`CodeLanguage` only accepts
`python`/`javascript`/`typescript`/`go`/`rust`/`java`/`c`/`cpp`/`perl`.
The very common markdown fence tags and hints — `js`, `ts`, `py`, `jsx`,
`tsx`, `node`, `rs`,
`c++` — are **not** enum values, so `CodeLanguage("js")` raises
`ValueError`. That construction
is *above* the method's own `try/except`, so:
- **Direct callers** — `CodeAwareCompressor().compress(code,
language="js")` and the module-level
`compress_code(code, language="js")` — crash with an uncaught
`ValueError`.
- **In the router (mixed content):** `split_into_sections` extracts the
raw fence tag
(`_CODE_FENCE_PATTERN` captures `\w*`, e.g. `js`) into
`ContentSection.language`, and that string
is passed straight into `compress(...)`. The `ValueError` is swallowed
by the outer `try/except`
in the strategy dispatch, so a ` ```js ` / ` ```ts ` / ` ```py ` block
silently **skips
code-aware compression** even when `enable_code_aware=True`, falling
back to the generic path.
So the three most common web/scripting languages, written with their
usual fence tags, never get
the structure-aware compressor.
Closes: no issue filed — found while auditing the code-compression
language path.
## Fix
Add a `coerce_language()` helper that maps common aliases/fence tags to
the canonical
`CodeLanguage` and returns `CodeLanguage.UNKNOWN` (never raises) for
anything unrecognized.
`compress()` now coerces the hint and, when the result is `UNKNOWN`,
falls back to
content-based `detect_language(code)` instead of constructing the enum
directly:
```python
if language:
detected_lang = coerce_language(language)
if detected_lang == CodeLanguage.UNKNOWN:
detected_lang, confidence = detect_language(code)
else:
confidence = 1.0
```
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/transforms/code_compressor.py`: add `_LANGUAGE_ALIASES` and
`coerce_language()`; use them in `compress()` for both the `language`
argument and `config.language_hint`, with a content-detection fallback
on `UNKNOWN`.
- `tests/test_code_compressor_language_alias.py`: cover alias mapping,
canonical passthrough, case/whitespace handling, unknown-returns-UNKNOWN
(no `ValueError`), and that `compress(language="js")` no longer raises.
## Testing
- [x] New regression tests added
(`tests/test_code_compressor_language_alias.py`)
- [x] Linting passes (`ruff check`) and formatting is clean (`ruff
format --check`)
- [ ] Full `pytest` deferred to CI (local-OOM reason below).
```text
$ uv run ruff check headroom/transforms/code_compressor.py tests/test_code_compressor_language_alias.py
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.10, headroom from this branch.
Importing `headroom` pulls in the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I verified the coercion logic
with a dependency-free script (replicating the enum + helper) and left
the full pytest to CI.
- Exact command / steps: ran the common aliases and the canonical values
through both the old `CodeLanguage(value.lower())` construction and the
new `coerce_language()`.
- Observed result: the old construction raises `ValueError` on every
alias (the crash / silent-skip); the new helper maps them and never
raises:
```text
OK alias 'js': old raised ValueError -> new maps to javascript
OK alias 'ts': old raised ValueError -> new maps to typescript
OK alias 'py': old raised ValueError -> new maps to python
OK alias 'jsx': old raised ValueError -> new maps to javascript
OK alias 'node': old raised ValueError -> new maps to javascript
OK canonical values pass through
OK case-insensitive + trimmed
OK unknown -> UNKNOWN (no ValueError)
LANGUAGE COERCION VERIFIED
```
- Not tested: running a full mixed-content document with ` ```js `
fences through a booted compression pipeline (needs the heavy stack).
The unit tests exercise the coercion directly and the
`compress(language="js")` entry point. Full local `pytest` deferred to
CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — ran
lint + a standalone logic check; full pytest deferred to CI (local OOM,
disclosed above)
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- No new dependencies; a small lookup table plus a helper and a
call-site change.
- @JerrettDavis tagging you — this one silently disables code-aware
compression for the most common fence tags (`js`/`ts`/`py`), so it may
be worth a look when you have a moment.
2026-07-11 10:27:33 +05:30
|
|
|
)
|
|
|
|
|
def test_coerce_language_accepts_canonical_values(canonical):
|
|
|
|
|
assert coerce_language(canonical) == CodeLanguage(canonical)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_coerce_language_is_case_insensitive_and_trims():
|
|
|
|
|
assert coerce_language(" JS ") == CodeLanguage.JAVASCRIPT
|
|
|
|
|
assert coerce_language("Python") == CodeLanguage.PYTHON
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@pytest.mark.parametrize("value", ["", " ", "not-a-language", "brainfuck", "yaml"])
|
|
|
|
|
def test_coerce_language_unknown_returns_unknown_not_valueerror(value):
|
|
|
|
|
# The whole point: never raise, so an unrecognized fence tag can fall back
|
|
|
|
|
# to content detection instead of crashing / being swallowed.
|
|
|
|
|
assert coerce_language(value) == CodeLanguage.UNKNOWN
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_compress_with_alias_language_does_not_raise():
|
|
|
|
|
"""The direct API path must not raise on a common alias."""
|
|
|
|
|
from headroom.transforms.code_compressor import CodeAwareCompressor
|
|
|
|
|
|
|
|
|
|
code = "function add(a, b) {\n return a + b;\n}\n"
|
|
|
|
|
compressor = CodeAwareCompressor()
|
|
|
|
|
# Before the fix this raised ValueError: 'js' is not a valid CodeLanguage.
|
|
|
|
|
result = compressor.compress(code, language="js")
|
|
|
|
|
assert result is not None
|
|
|
|
|
assert result.compressed is not None
|