headroom/tests/test_code_compressor_language_alias.py

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

71 lines
2.6 KiB
Python
Raw Normal View History

fix(transforms/code): coerce language aliases instead of raising (#1975) ## Description `CodeAwareCompressor.compress()` picks the language for AST-based compression like this (`headroom/transforms/code_compressor.py`): ```python if language: detected_lang = CodeLanguage(language.lower()) # <-- raises on anything not an exact enum value confidence = 1.0 elif self.config.language_hint: detected_lang = CodeLanguage(self.config.language_hint.lower()) confidence = 1.0 else: detected_lang, confidence = detect_language(code) ``` `CodeLanguage` only accepts `python`/`javascript`/`typescript`/`go`/`rust`/`java`/`c`/`cpp`/`perl`. The very common markdown fence tags and hints — `js`, `ts`, `py`, `jsx`, `tsx`, `node`, `rs`, `c++` — are **not** enum values, so `CodeLanguage("js")` raises `ValueError`. That construction is *above* the method's own `try/except`, so: - **Direct callers** — `CodeAwareCompressor().compress(code, language="js")` and the module-level `compress_code(code, language="js")` — crash with an uncaught `ValueError`. - **In the router (mixed content):** `split_into_sections` extracts the raw fence tag (`_CODE_FENCE_PATTERN` captures `\w*`, e.g. `js`) into `ContentSection.language`, and that string is passed straight into `compress(...)`. The `ValueError` is swallowed by the outer `try/except` in the strategy dispatch, so a ` ```js ` / ` ```ts ` / ` ```py ` block silently **skips code-aware compression** even when `enable_code_aware=True`, falling back to the generic path. So the three most common web/scripting languages, written with their usual fence tags, never get the structure-aware compressor. Closes: no issue filed — found while auditing the code-compression language path. ## Fix Add a `coerce_language()` helper that maps common aliases/fence tags to the canonical `CodeLanguage` and returns `CodeLanguage.UNKNOWN` (never raises) for anything unrecognized. `compress()` now coerces the hint and, when the result is `UNKNOWN`, falls back to content-based `detect_language(code)` instead of constructing the enum directly: ```python if language: detected_lang = coerce_language(language) if detected_lang == CodeLanguage.UNKNOWN: detected_lang, confidence = detect_language(code) else: confidence = 1.0 ``` ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/transforms/code_compressor.py`: add `_LANGUAGE_ALIASES` and `coerce_language()`; use them in `compress()` for both the `language` argument and `config.language_hint`, with a content-detection fallback on `UNKNOWN`. - `tests/test_code_compressor_language_alias.py`: cover alias mapping, canonical passthrough, case/whitespace handling, unknown-returns-UNKNOWN (no `ValueError`), and that `compress(language="js")` no longer raises. ## Testing - [x] New regression tests added (`tests/test_code_compressor_language_alias.py`) - [x] Linting passes (`ruff check`) and formatting is clean (`ruff format --check`) - [ ] Full `pytest` deferred to CI (local-OOM reason below). ```text $ uv run ruff check headroom/transforms/code_compressor.py tests/test_code_compressor_language_alias.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.10, headroom from this branch. Importing `headroom` pulls in the torch/transformers stack and a full `pytest` gets OOM-killed on this box, so I verified the coercion logic with a dependency-free script (replicating the enum + helper) and left the full pytest to CI. - Exact command / steps: ran the common aliases and the canonical values through both the old `CodeLanguage(value.lower())` construction and the new `coerce_language()`. - Observed result: the old construction raises `ValueError` on every alias (the crash / silent-skip); the new helper maps them and never raises: ```text OK alias 'js': old raised ValueError -> new maps to javascript OK alias 'ts': old raised ValueError -> new maps to typescript OK alias 'py': old raised ValueError -> new maps to python OK alias 'jsx': old raised ValueError -> new maps to javascript OK alias 'node': old raised ValueError -> new maps to javascript OK canonical values pass through OK case-insensitive + trimmed OK unknown -> UNKNOWN (no ValueError) LANGUAGE COERCION VERIFIED ``` - Not tested: running a full mixed-content document with ` ```js ` fences through a booted compression pipeline (needs the heavy stack). The unit tests exercise the coercion directly and the `compress(language="js")` entry point. Full local `pytest` deferred to CI (OOM, per above). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes — ran lint + a standalone logic check; full pytest deferred to CI (local OOM, disclosed above) - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - No new dependencies; a small lookup table plus a helper and a call-site change. - @JerrettDavis tagging you — this one silently disables code-aware compression for the most common fence tags (`js`/`ts`/`py`), so it may be worth a look when you have a moment.
2026-07-11 10:27:33 +05:30
"""Regression tests for language-hint / fence-tag coercion in code_compressor.
`CodeAwareCompressor.compress(code, language=...)` used to build the language
with `CodeLanguage(language.lower())`, which raises `ValueError` for anything
that is not an exact enum value. Common markdown fence tags and hints `js`,
`ts`, `py` are not enum values, so:
* direct callers (`compress(code, language="js")`) crashed, and
* inside the router the ValueError was swallowed, so ` ```js ` / ` ```ts ` /
` ```py ` fenced blocks silently skipped code-aware compression.
`coerce_language` maps aliases to the canonical language and returns UNKNOWN
(never raises) for unrecognized tags, letting the caller fall back to
content-based detection.
"""
import pytest
from headroom.transforms.code_compressor import CodeLanguage, coerce_language
@pytest.mark.parametrize(
"alias,expected",
[
("js", CodeLanguage.JAVASCRIPT),
("jsx", CodeLanguage.JAVASCRIPT),
("node", CodeLanguage.JAVASCRIPT),
("ts", CodeLanguage.TYPESCRIPT),
("tsx", CodeLanguage.TYPESCRIPT),
("py", CodeLanguage.PYTHON),
("python3", CodeLanguage.PYTHON),
("golang", CodeLanguage.GO),
("rs", CodeLanguage.RUST),
("c++", CodeLanguage.CPP),
],
)
def test_coerce_language_maps_common_aliases(alias, expected):
assert coerce_language(alias) == expected
@pytest.mark.parametrize(
"canonical",
["python", "javascript", "typescript", "go", "rust", "java", "c", "cpp", "perl"],
)
def test_coerce_language_accepts_canonical_values(canonical):
assert coerce_language(canonical) == CodeLanguage(canonical)
def test_coerce_language_is_case_insensitive_and_trims():
assert coerce_language(" JS ") == CodeLanguage.JAVASCRIPT
assert coerce_language("Python") == CodeLanguage.PYTHON
@pytest.mark.parametrize("value", ["", " ", "not-a-language", "brainfuck", "yaml"])
def test_coerce_language_unknown_returns_unknown_not_valueerror(value):
# The whole point: never raise, so an unrecognized fence tag can fall back
# to content detection instead of crashing / being swallowed.
assert coerce_language(value) == CodeLanguage.UNKNOWN
def test_compress_with_alias_language_does_not_raise():
"""The direct API path must not raise on a common alias."""
from headroom.transforms.code_compressor import CodeAwareCompressor
code = "function add(a, b) {\n return a + b;\n}\n"
compressor = CodeAwareCompressor()
# Before the fix this raised ValueError: 'js' is not a valid CodeLanguage.
result = compressor.compress(code, language="js")
assert result is not None
assert result.compressed is not None