akadmin/headroom
1
0
Fork 0
mirror of https://github.com/headroomlabs-ai/headroom.git synced 2026-08-27 14:17:10 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
2556 commits 410 branches 169 tags 142 MiB
Commit graph

13 commits

Author SHA1 Message Date
Shengbo_Wang
1de35e775f
fix(code): parse-probe tree-sitter availability in code_handler (#1231) (#1300) 
## Description

`_check_tree_sitter()` in
`headroom/compression/handlers/code_handler.py` only verified that
`tree_sitter_language_pack` could be imported. When tree-sitter core and
the language pack are built against different ABIs, the import succeeds
but `parser.language = get_language(...)` raises at request time,
silently falling back to the generic text compressor — with no warning,
while the banner still reports code-aware as enabled.

#1299 fixed the same class of bug in `transforms/code_compressor.py`.
This PR is the defensive follow-up tracked by #1231: it applies the same
parse probe to the compression **structure handler** so both code-aware
paths are consistent.

Closes #1231

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Replace the import-only probe in `code_handler._check_tree_sitter()`
with a real parse probe: construct a `Parser`, assign a Python language,
and parse `b"x = 1\n"` — if any step fails, mark unavailable
- Log a WARNING when import succeeds but parsing fails, so the downgrade
is visible
- Add `TestAvailabilityProbe` covering the simulated ABI mismatch (->
False) and healthy install (-> True) cases

> Note: an earlier revision of this PR also touched
`transforms/code_compressor.py`, but that fix landed independently via
#1299. After rebasing onto current `main`, this PR is scoped to the
remaining `code_handler.py` gap only.

## Testing

- [x] Unit tests pass (`pytest
tests/test_compression/test_code_handler.py` -> 22 passed, 8 skipped)
- [x] Linting / formatting pass (`ruff check .`, `ruff format --check
.`)
- [x] New tests added for new functionality

### Real Behavior Proof

- Setup: Windows 11 (GBK locale), Python 3.10, tree-sitter NOT installed
- Before fix: `_check_tree_sitter()` returns `True` on a
partial/ABI-mismatched install (import succeeds), then silently degrades
to the text compressor at request time with no warning
- After fix: the probe parses a trivial snippet; an ABI mismatch is
caught at probe time, `_check_tree_sitter()` returns `False`, and a
WARNING is logged.
`TestAvailabilityProbe::test_abi_mismatch_returns_false` reproduces this
with a fake Parser whose `language` setter raises.

Signed-off-by: RTCartist <wangshengb@buaa.edu.cn>
 2026-07-09 14:06:29 -05:00
Tejas Chopra
840871cb96
fix(compression): repair entropy preservation + JSON-safe truncation fallback (#1536) 
## Description

Reported by [@JoaoMarcos44](https://github.com/JoaoMarcos44) via an
independent security audit — thanks for the careful, well-documented
report.

Fixes two confirmed findings from a June 2026 security audit of
`headroom/compression/` (the `UniversalCompressor` utility). Both are
real defects in shipped, public, tested code; note that this module is
**not** on the proxy hot path (the proxy uses `headroom/transforms/`),
so real-world blast radius is module-local rather than proxy-wide.

- **SEC-01 (entropy bypass):** `use_entropy_preservation` was a silent
no-op. `compress()` tokenized content at character level
(`list(content)`) and fed single-char tokens to `compute_entropy_mask`,
whose `min_token_length` guard skipped every one — so high-entropy
secrets (API keys, OAuth tokens, UUIDs, hashes) were never preserved
despite the feature being enabled.
- **SEC-02 (JSON corruption):** the `_simple_compress` truncation
fallback (used when Kompress is unavailable or raises) inserted a
separator containing raw newlines. When that fallback ran on a span
inside a JSON string value it produced invalid JSON (RFC 8259 §7),
crashing downstream `json.loads()`.

The other three audited items need no code change and were verified, not
assumed: SEC-03 (surrogate DoS) is already caught by the `try/except` in
`code_handler._extract_mask` and falls back to regex — non-reproducible
even with `tree_sitter_language_pack` installed; SEC-04 (prompt
injection) is out of a compressor's scope; SEC-05 (SQLite race) is a
misread (`CompressionStore` defaults to `InMemoryBackend`; the SQLite
backend uses WAL + busy_timeout + a lock).

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Add `compute_entropy_mask_for_content()` (`masks.py`): scores
whitespace-delimited words and maps high-entropy ones back to character
positions, returning a char-aligned mask. The existing token-level
`compute_entropy_mask` is left intact.
- Introduce `SECRET_ENTROPY_MIN_LENGTH = 20` as the default word-length
floor. Normalized Shannon entropy rates short-but-diverse words (e.g.
"detailed") nearly as high as a real secret, so a length floor is the
discriminator; 20 matches the entropy-detection floor used by secret
scanners (trufflehog, detect-secrets) and prevents over-preserving prose
(which would otherwise block legitimate compression).
- Wire the content-level entropy pass into
`UniversalCompressor.compress()` (scores `content`, not the char-level
`tokens`).
- Replace the `_simple_compress` separator `"\n...[compressed]...\n"`
with the control-char-free `" ...[compressed]... "`.
- Add regression tests at the mask level and end-to-end.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ ruff check headroom/compression/
All checks passed!

$ mypy headroom/compression/masks.py headroom/compression/universal.py
Success: no issues found in 2 source files

$ pytest tests/test_compression/test_masks.py tests/test_compression/test_universal.py \
         tests/test_compression/test_json_handler.py tests/test_compression/test_code_handler.py -q
======================= 111 passed, 2 warnings in 10.76s =======================
```

## Real Behavior Proof

- Environment: macOS, Python 3.12 in repo `.venv`;
`tree_sitter_language_pack` and Kompress present.
- Exact command / steps: reproduced each finding by calling
`UniversalCompressor.compress()` directly before/after the fix — SEC-01:
`compute_entropy_mask(list("k="+secret))` preserved 0 of N tokens
(inert); after fix `compute_entropy_mask_for_content` preserves the
secret's char range and the end-to-end test shows a 43-char secret
dropped with preservation off / kept with it on. SEC-02:
`compress(json.dumps({...long value...}), content_type=JSON)` with
`use_kompress=False` raised `JSONDecodeError` before the fix and
round-trips through `json.loads()` after.
- Observed result: SEC-01 entropy preservation now functions; SEC-02
output is valid JSON on both the Kompress and fallback paths; the
previously-failing `test_compression_reduces_tokens` passes again (no
over-preservation).
- Not tested: `tests/test_compression/test_evals.py` and
`test_llm_eval.py` (require external API/model access); the
proxy/transforms live path is unaffected since it does not import
`UniversalCompressor`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

CHANGELOG not updated (handled by the release tooling). The audit also
flagged SEC-03/04/05 — left unchanged by design, with verification
rationale in the Description.
 2026-06-28 10:39:02 -07:00
Veesh Goldman
f39858c233
feat(code): add Perl support to code-aware compressor (#1125) 
## Description

Adds Perl as a supported language for `CodeAwareCompressor` /
`CodeStructureHandler`. Function bodies are compressed while
`use`/`require` imports, `sub`/`method` signatures, and
`package`/`class`/`role` declarations are preserved — bringing Perl up
to parity with the other Tier-2 languages.

No new dependencies: the Perl grammar already ships in
`tree-sitter-language-pack` (already a Headroom dependency), so this is
pure configuration.

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `code_handler.py`: Perl entries in the four per-language tables —
`_STRUCTURAL_NODE_TYPES`, `_SIGNATURE_PATTERNS` (regex fallback),
`_LANGUAGE_MARKERS` (detection), `_IMPORT_PATTERNS`. The existing
`_CONTAINER_BODY_TYPES` already covers Perl's `block` body node, so no
change was needed there.
- `code_compressor.py`: `CodeLanguage.PERL` enum value, a data-driven
`LangConfig`, a `_LANGUAGE_PREFILTER` entry, and the supported-language
string in the parser error message.
- Node-type names (`subroutine_declaration_statement`,
`package_statement`, `signature`, `block`, …) are from the
`tree-sitter-perl/tree-sitter-perl` grammar (MIT).
- Tests: 1 detection test + 2 regex-path signature/import-preservation
tests.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_compression/test_code_handler.py -q
collected 28 items
tests/test_compression/test_code_handler.py ............................ [100%]
======================== 28 passed, 1 warning in 2.53s =========================

$ ruff check headroom/compression/handlers/code_handler.py headroom/transforms/code_compressor.py tests/test_compression/test_code_handler.py
All checks passed!

$ mypy headroom/compression/handlers/code_handler.py headroom/transforms/code_compressor.py
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: Python 3.12, `pip install -e ".[code]"`
(tree-sitter-language-pack installed, `is_tree_sitter_available() ==
True`).
- Exact command / steps: ran `CodeStructureHandler().get_mask(code,
language="perl")` on a real Perl module (package + two subs with
bodies).
- Observed result: detected as `perl`, parsed via the `tree-sitter` path
(not regex), and the preserved span was exactly the imports + package +
sub signatures, with both sub bodies marked compressible:

```text
tree-sitter available: True
parser: tree-sitter | detected: perl
--- PRESERVED (signatures/imports/structure) ---
use strict;use warnings;package Greeter;sub new sub greet
```

- Not tested: the full proxy/MCP server end-to-end path (out of scope —
this PR only touches the code compressor's language tables).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

- Docs/CHANGELOG left unchecked — happy to add a Perl line to either if
you'd like; I wasn't sure of your preferred location.
- *Disclosure: I maintain the upstream `tree-sitter-perl` grammar this
relies on. It's already a transitive dependency of Headroom via
`tree-sitter-language-pack` — this PR only adds config to use it, with
no dependency changes.*

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-22 18:47:41 -05:00
Ashish
6cdb846200
fix(compression): use thread-local tree-sitter parsers in code handler (#893) 
## Description

`CodeStructureHandler` cached tree-sitter parsers in a process-global
dict; the lock only guarded creation, while `parse()` ran unlocked on
any thread. tree-sitter `Parser` objects are pyo3 `unsendable` — using
one from a non-creator thread panics. The proxy invokes handlers from
executor pool threads, so a shared parser is an eventual crash. Same
class already fixed in `transforms/code_compressor.py` (#604). Stacked
on #892.

Closes # <!-- compression-handler review -->

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/compression/handlers/code_handler.py`: one parser per
(thread, language) via `threading.local()`, porting the pattern from
`transforms/code_compressor.py`.
- `tests/test_compression/test_code_handler.py`: regression test parsing
from a 4-worker thread pool, asserting every call stays on the
tree-sitter path.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_compression/ -q
94 passed, 8 skipped
```

## Real Behavior Proof

- Environment: local macOS, Python 3.11, tree-sitter-language-pack
1.8.1, branch `fix/code-threadlocal-parsers` (stacked on #892).
- Exact command / steps: `pytest tests/test_compression/ -q`.
- Observed result: 16 parses across a 4-worker pool all stay on the
tree-sitter path with no pyo3 panic; previously a shared parser would be
touched cross-thread.
- Not tested: Reproducing the original panic under production
concurrency (covered structurally by the thread-pool test).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A — library change. See Test Output.

## Additional Notes

Stacked on #892 — review the top commit until that merges. PR 5 of 7.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
 2026-06-15 16:38:08 -05:00
Ashish
615e1ed6f5
test(compression): fill code handler coverage gaps (#895) 
## Description

`CodeStructureHandler` had zero dedicated tests before this series —
which is exactly why the P0 bugs in #890/#892/#893 went unnoticed. This
fills the remaining coverage gaps beyond the per-fix regression tests.
Stacked on #893.

Closes # <!-- compression-handler review -->

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- `tests/test_compression/test_code_handler.py`: language detection
(python/go/rust + default fallback); regex-path signature/import
preservation across go/rust/typescript/javascript; regex confidence
value; empty/whitespace content; unknown language; mask-length
invariant.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_compression/test_code_handler.py -q
25 passed
```

## Real Behavior Proof

- Environment: local macOS, Python 3.11, tree-sitter-language-pack
1.8.1, branch `test/code-handler-coverage` (stacked on #893).
- Exact command / steps: `pytest
tests/test_compression/test_code_handler.py -q`.
- Observed result: 25 tests pass; tree-sitter classes skip cleanly when
the pack is absent, regex-path tests always run.
- Not tested: N/A — this PR is tests only.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A — tests only. See Test Output.

## Additional Notes

Stacked on #893 — review the top commit until that merges. PR 6 of 7.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-06-15 10:25:57 -05:00
Ashish
b1f700fc27
fix(compression): convert tree-sitter byte offsets to char offsets (#892) 
## Description

tree-sitter reports node positions as byte offsets into the UTF-8
encoding, but `CodeStructureHandler` builds a character-indexed mask.
Any multi-byte character (accents, emoji, CJK in
docstrings/comments/strings) shifted every subsequent span, preserving
the wrong characters and leaking signature bytes into bodies. Stacked on
#890.

Closes # <!-- compression-handler review -->

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/compression/handlers/code_handler.py`: remap spans through a
byte->char table before masking; pure-ASCII content (byte == char) skips
the conversion.
- `tests/test_compression/test_code_handler.py`: regression test with
`café münü 🎉` in a comment, asserting the following signature and body
are correctly aligned.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_compression/ -q
93 passed, 8 skipped
```

## Real Behavior Proof

- Environment: local macOS, Python 3.11, tree-sitter-language-pack
1.8.1, branch `fix/code-byte-char-offsets` (stacked on #890).
- Exact command / steps: `pytest tests/test_compression/ -q`.
- Observed result: With 9 extra UTF-8 bytes ahead of it, a function
signature is exactly preserved and its body stays compressible; before,
the offsets were shifted.
- Not tested: End-to-end through the live proxy pipeline.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A — library change. See Test Output.

## Additional Notes

Stacked on #890 — review the top commit until that merges. PR 4 of 7.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-06-15 10:24:21 -05:00
Ashish
65b0e8c58d
fix(compression): measure short-value threshold on payload, not token (#889) 
## Description

`JSONStructureHandler._should_preserve_token` compared `len(token.text)`
— which includes both quote characters — against
`short_value_threshold`. A value of exactly threshold length was
rejected: the documented "20-char threshold" was effectively 18 chars of
payload. Stacked on #887.

Closes # <!-- compression-handler review -->

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/compression/handlers/json_handler.py`: strip quotes once at
the top of the string-value branch and use the payload length for both
the short-value and entropy checks.
- `tests/test_compression/test_json_handler.py`: regression test for a
value of exactly `short_value_threshold` length.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_compression/test_json_handler.py -q
33 passed
```

## Real Behavior Proof

- Environment: local macOS, Python 3.11, branch
`fix/json-quote-threshold` (stacked on #887).
- Exact command / steps: `pytest
tests/test_compression/test_json_handler.py -q`.
- Observed result: A 20-char value is preserved at a 20-char threshold;
previously it was dropped due to the +2 quote miscount.
- Not tested: End-to-end through the live proxy pipeline.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A — library change. See Test Output.

## Additional Notes

Stacked on #887 — review the top commit until that merges. PR 2 of 7.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-06-15 10:23:26 -05:00
Ashish
16ed73bca6
fix(compression): keep container bodies compressible in code handler (#890) 
## Description

Two bugs in `CodeStructureHandler`'s tree-sitter path. (1) Container
nodes (class/impl/trait/decorated definitions) were marked structural
over their full span and `_spans_to_mask` never un-marks, so every
method body inside a class was preserved and compression silently
no-opped at confidence 0.95. (2) Discovered while testing:
`tree-sitter-language-pack >= 1.0` switched to a Rust binding (methods,
not attributes; `parse(str)`), so the handler raised `TypeError` on
every call and silently fell back to regex.

Closes # <!-- compression-handler review -->

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/compression/handlers/code_handler.py`: containers emit a
signature-only span (start to body start); recursion gives nested
functions their own signature/body split; decorated definitions emit no
whole-node span.
- `headroom/compression/handlers/code_handler.py`: small compat shim
supporting both the classic attribute API and the new Rust-binding
method API.
- `tests/test_compression/test_code_handler.py`: new file (the handler
had zero dedicated tests) covering class/decorated/impl body
compressibility, regex fallback, and a preservation-ratio bound.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_compression/ -q
92 passed, 8 skipped
```

## Real Behavior Proof

- Environment: local macOS, Python 3.11, tree-sitter-language-pack 1.8.1
installed, branch `fix/code-container-bodies`.
- Exact command / steps: `pytest tests/test_compression/ -q`.
- Observed result: Class method bodies are now compressible
(preservation ratio drops from ~1.0 to roughly the signature fraction);
the tree-sitter path runs instead of falling back to regex.
- Not tested: End-to-end through the live proxy pipeline.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A — library change. See Test Output.

## Additional Notes

PR 3 of 7; branched fresh from main (independent of #887/#889).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-06-13 21:31:32 -07:00
Ashish
d6f0f0f642
fix(compression): correct JSON array item counting and entropy gate (#887) 
## Description

Two bugs in `JSONStructureHandler` that jointly defeated the "keep first
N array items fully" design. (1) Every comma under `array_depth > 0` was
counted as an array item separator — including commas *between keys
inside objects* — so for arrays of objects the first record's own keys
exhausted `max_array_items_full` and dropped values belonging to item 0.
(2) Fixing that unmasked a second bug: self-normalized Shannon entropy
scores English prose at 0.90+, above the 0.85 "identifier" threshold, so
every long description was preserved as a fake high-entropy identifier.

Closes # <!-- found during a compression-handler review -->

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/compression/handlers/json_handler.py`: replace depth-keyed
comma counting with a container stack so only commas whose immediate
enclosing container is an array advance that array's item index.
- `headroom/compression/handlers/json_handler.py`: gate the entropy
preservation check on a no-spaces identifier signal, so UUIDs/hashes
still pass but prose compresses.
- `tests/test_compression/test_json_handler.py`: regression tests for
object-comma counting, items past the threshold, and prose-vs-identifier
entropy.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_compression/test_json_handler.py -q
32 passed
```

## Real Behavior Proof

- Environment: local macOS, Python 3.11, branch
`fix/json-array-item-count`.
- Exact command / steps: `pytest
tests/test_compression/test_json_handler.py -q` plus an empirical mask
dump on `[{"a":1,"b":2,...}]`.
- Observed result: Values inside array item 0 are now preserved; long
prose values compress while UUIDs are retained (prose scored
0.906-0.929, UUID 0.956 — the threshold alone could not separate them).
- Not tested: End-to-end through the live proxy pipeline (the handler is
not yet wired into the proxy hot path).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A — library/compression change with no UI. See Test Output.

## Additional Notes

First of a 7-PR compression-handler review series.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-06-13 21:24:12 -07:00
chopratejas
3290a3d582 Remove LLMLingua: Kompress is the sole text compressor 
LLMLingua was the original ML text compressor (BERT-based). Kompress
(ModernBERT, trained on 330K structured tool outputs) replaced it with
better compression quality and simpler architecture.

Removed across 35 files:
- Deleted headroom/transforms/llmlingua_compressor.py
- Deleted tests/test_transforms/test_llmlingua_compressor.py
- Deleted tests/test_proxy_llmlingua.py
- Removed all enable_llmlingua config, _get_llmlingua methods,
  LLMLingua fallback paths, LLMLINGUA strategy enum values
- Removed CLI flags, model configs, compression handler references
- Simplified ContentRouter: Kompress is primary and only text compressor
 2026-03-26 11:11:00 -07:00
chopratejas
0adc39ab7a Fix CI: guard starlette imports, asyncio.run(), deprecate datetime.utcnow() 
- Guard starlette imports in test_compress_api.py (skip ASGI tests without proxy deps)
- Replace asyncio.get_event_loop().run_until_complete() with asyncio.run() (Python 3.13)
- Replace datetime.utcnow() with datetime.now(timezone.utc).replace(tzinfo=None) everywhere
 2026-02-19 11:03:24 -08:00
chopratejas
64a747d66e Fix flaky JavaScript signature preservation test threshold 
Lower threshold from 70% to 60% - methods inside class bodies
may be compressed, which is expected behavior.
 2026-01-15 15:38:25 -08:00
chopratejas
31aa72c885 Add universal compression module with ML-based content detection 
- Add headroom.compression module with UniversalCompressor
- ML-based content detection using Magika (JSON, code, logs, text)
- Structure-preserving compression via handler protocol
- JSON handler: preserves keys, brackets, high-entropy values (UUIDs)
- Code handler: preserves imports, signatures, types (tree-sitter AST)
- Entropy-based preservation for identifiers and hashes
- CCR integration for reversible compression
- Comprehensive test suite with LLM eval tests
- Add docs/compression.md with full API documentation
 2026-01-15 15:26:14 -08:00