## Description
<!-- Briefly explain the change and why it is needed. -->
Closes #
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
-
## Testing
<!-- Check what you actually ran, then paste the real command output
below. -->
- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
# Paste relevant command output or artifact links here
```
## Real Behavior Proof
- Environment:
- Exact command / steps:
- Observed result:
- Not tested:
## Review Readiness
- [ ] I have performed a self-review
- [ ] This PR is ready for human review
## Checklist
- [ ] My code follows the project's style guidelines
- [ ] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
Add screenshots to help explain your changes.
## Additional Notes
<!-- Mention any N/A checklist items, tradeoffs, follow-ups, or
maintainer context. -->
## Description
Headroom's `detect_content_type()` only recognizes content starting with
`[` as a `JSON array. Many web search tools (SerpAPI, Tavily, custom
backends) return space-separated JSON objects instead of a real array
like follows
```json
{"title": "Result 1", "url": "..."} {"title": "Result 2", "url": "..."} {"title": "Result 3", "url": "..."}
```
That shape is detected as `PLAIN_TEXT` (confidence 0.5), so SmartCrusher
never processes it and web-search results compress 0%.
Closes#1741
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `content_detector.py`: `_try_detect_json` now recognizes a run of ≥2
whitespace-separated (space- or newline-separated) JSON objects and
returns `JSON_ARRAY` with `metadata["concatenated"] = True`. The router
already falls back to the Python regex detector when the native detector
returns `PLAIN_TEXT` (`content_router.py`), so this fixes routing on the
default backend too.
- `content_detector.py`: added `normalize_concatenated_json()` (and a
`_decode_concatenated_json()` helper) that rewrites the space-separated
shape into a canonical `[{…}, {…}]` array string.
- `smart_crusher.py`: `SmartCrusher.crush()` normalizes concatenated
JSON to a real array before handing it to the Rust crusher, so it
actually compresses.
- The change is deliberately conservative: a single object stays
unclaimed (`_try_detect_json('{"id": 1}')` → `None`), and any non-JSON
token between objects disqualifies the run. Existing `[`-array detection
is unchanged.
- Added tests and a CHANGELOG entry.
## Testing
- [x] Unit tests pass (`pytest`) — affected suites (full suite has
network-dependent ML tests that can't run offline; see note)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ ruff check .
All checks passed!
$ pytest tests/test_transforms_content_detection.py -q
............ [100%]
12 passed
$ pytest tests/test_transforms_content_router.py \
tests/test_smart_crusher_toin_attachment.py \
tests/test_transforms_tabular.py -q
96 passed, 2 skipped
# + SmartCrusher passthrough tests in test_text_compressors.py: 2 passed
```
## Real Behavior Proof
- Environment: macOS 26.5, Python 3.12.11, editable source build (`uv
pip install -e .`) with the Rust `_core` compiled locally; default
detection backend (native Rust → Python-regex fallback on PLAIN_TEXT).
- Exact command / steps: ran a 100-object space-separated `web_search`
payload through `detect_content_type()` and
`ContentRouter().compress()`, before and after the patch (repro below).
- Observed result: detection flips `PLAIN_TEXT` (conf 0.5) →
`JSON_ARRAY` (conf 1.0) and SmartCrusher compression goes from 0.0% to
34.2% (10369 → 6819 bytes) on the identical payload.
- Not tested: the native Rust *detector* path in isolation (the fix
relies on the existing documented Python-regex fallback for
`PLAIN_TEXT`); separators other than whitespace
(comma-separated-without-brackets is intentionally not claimed).
Before:
```
detected : ContentType.PLAIN_TEXT conf 0.5
strategy : CompressionStrategy.SMART_CRUSHER
orig bytes: 10369
comp bytes: 10369
reduction : 0.0%
```
After:
```
detected : ContentType.JSON_ARRAY conf 1.0
strategy : CompressionStrategy.SMART_CRUSHER
orig bytes: 10369
comp bytes: 6819
reduction : 34.2%
```
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation (CHANGELOG)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
Co-authored-by: JD Davis <mxjerrett@gmail.com>
This test pinned the pre-3e.1 behavior on three lines that the new
KeywordDetector intentionally changes:
1. `'token'` was asserted to be in SECURITY_KEYWORDS. It was dropped
from the security set in 3e.1 because it false-positived on every
LLM-token reference in our own product. Updated to assert the new
set (`security|password|auth|secret`) and explicitly that `token`
is gone.
2. SECURITY_PATTERN was tested via "rotate the auth token" (matched
via the now-removed `token` keyword). Now tested via "rotate the
auth header" (matches via `auth`, which is the real security
signal) plus a negative assertion that LLM-metric strings no
longer fire.
3. ERROR_PATTERN test added an assertion that "Connection timeout"
now flags as an error (3e.1 fixed the keyword/regex drift).
PRIORITY_PATTERNS_TEXT indices were also corrected: the Rust-supplied
markdown_prefixes table is ordered `# `, `## `, `### `, `#### `, `**`,
`> ` (six prefixes) so the bold/blockquote assertions live at indices
6 and 7, not 3 and 4. New `# ` and `## ` checks pin the lower indices.
Each diverging assertion carries a `fixed_in_3e1` comment so the
audit trail stays clear.
`.gitattributes` declares `*.py text eol=lf` and `*.sh text eol=lf`, but
74 files (73 .py, 1 .sh) are stored in the index with CRLF line endings,
violating that contract. Every macOS/Linux clone reports these files as
"modified" on fresh checkout because git's diff engine sees the stored
bytes don't match the attribute contract, even though the working tree
and index match byte-for-byte.
Running `git add --renormalize .` rewrites each affected blob so the
stored form matches the attribute declaration. No semantic changes —
every affected file's diff is "N insertions, N deletions" with inserts
and deletes being the same lines modulo line endings.
Follow-up commit adds `.git-blame-ignore-revs` so `git blame` / GitHub
blame skip this mechanical commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
