mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
840871cb96
## Description Reported by [@JoaoMarcos44](https://github.com/JoaoMarcos44) via an independent security audit — thanks for the careful, well-documented report. Fixes two confirmed findings from a June 2026 security audit of `headroom/compression/` (the `UniversalCompressor` utility). Both are real defects in shipped, public, tested code; note that this module is **not** on the proxy hot path (the proxy uses `headroom/transforms/`), so real-world blast radius is module-local rather than proxy-wide. - **SEC-01 (entropy bypass):** `use_entropy_preservation` was a silent no-op. `compress()` tokenized content at character level (`list(content)`) and fed single-char tokens to `compute_entropy_mask`, whose `min_token_length` guard skipped every one — so high-entropy secrets (API keys, OAuth tokens, UUIDs, hashes) were never preserved despite the feature being enabled. - **SEC-02 (JSON corruption):** the `_simple_compress` truncation fallback (used when Kompress is unavailable or raises) inserted a separator containing raw newlines. When that fallback ran on a span inside a JSON string value it produced invalid JSON (RFC 8259 §7), crashing downstream `json.loads()`. The other three audited items need no code change and were verified, not assumed: SEC-03 (surrogate DoS) is already caught by the `try/except` in `code_handler._extract_mask` and falls back to regex — non-reproducible even with `tree_sitter_language_pack` installed; SEC-04 (prompt injection) is out of a compressor's scope; SEC-05 (SQLite race) is a misread (`CompressionStore` defaults to `InMemoryBackend`; the SQLite backend uses WAL + busy_timeout + a lock). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Add `compute_entropy_mask_for_content()` (`masks.py`): scores whitespace-delimited words and maps high-entropy ones back to character positions, returning a char-aligned mask. The existing token-level `compute_entropy_mask` is left intact. - Introduce `SECRET_ENTROPY_MIN_LENGTH = 20` as the default word-length floor. Normalized Shannon entropy rates short-but-diverse words (e.g. "detailed") nearly as high as a real secret, so a length floor is the discriminator; 20 matches the entropy-detection floor used by secret scanners (trufflehog, detect-secrets) and prevents over-preserving prose (which would otherwise block legitimate compression). - Wire the content-level entropy pass into `UniversalCompressor.compress()` (scores `content`, not the char-level `tokens`). - Replace the `_simple_compress` separator `"\n...[compressed]...\n"` with the control-char-free `" ...[compressed]... "`. - Add regression tests at the mask level and end-to-end. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check headroom/compression/ All checks passed! $ mypy headroom/compression/masks.py headroom/compression/universal.py Success: no issues found in 2 source files $ pytest tests/test_compression/test_masks.py tests/test_compression/test_universal.py \ tests/test_compression/test_json_handler.py tests/test_compression/test_code_handler.py -q ======================= 111 passed, 2 warnings in 10.76s ======================= ``` ## Real Behavior Proof - Environment: macOS, Python 3.12 in repo `.venv`; `tree_sitter_language_pack` and Kompress present. - Exact command / steps: reproduced each finding by calling `UniversalCompressor.compress()` directly before/after the fix — SEC-01: `compute_entropy_mask(list("k="+secret))` preserved 0 of N tokens (inert); after fix `compute_entropy_mask_for_content` preserves the secret's char range and the end-to-end test shows a 43-char secret dropped with preservation off / kept with it on. SEC-02: `compress(json.dumps({...long value...}), content_type=JSON)` with `use_kompress=False` raised `JSONDecodeError` before the fix and round-trips through `json.loads()` after. - Observed result: SEC-01 entropy preservation now functions; SEC-02 output is valid JSON on both the Kompress and fallback paths; the previously-failing `test_compression_reduces_tokens` passes again (no over-preservation). - Not tested: `tests/test_compression/test_evals.py` and `test_llm_eval.py` (require external API/model access); the proxy/transforms live path is unaffected since it does not import `UniversalCompressor`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes CHANGELOG not updated (handled by the release tooling). The audit also flagged SEC-03/04/05 — left unchanged by design, with verification rationale in the Description. |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| test_code_handler.py | ||
| test_evals.py | ||
| test_json_handler.py | ||
| test_llm_eval.py | ||
| test_masks.py | ||
| test_universal.py | ||