The optional `query` parameter on headroom_retrieve routed retrieval
through CompressionStore.search(), which BM25-scored the items inside a
single cached blob and dropped everything below a 0.3 relevance floor.
On small per-blob corpora with conversational queries this returned an
empty result the large majority of the time, so the LLM saw "nothing
found" for content that was actually present — pushing users to turn
compression off entirely.
Retrieval is fundamentally a hash lookup (this already matches the Rust
proxy's CCR store, which is put/get only — "no BM25 search"). Remove the
query/search path end to end and always return the full original
content:
Core (Python proxy):
- tool schemas (anthropic/openai/google) drop the `query` property
- parse_tool_call returns the hash (str | None) instead of (hash, query)
- response handler, proxy POST/GET/tool-call handlers, the MCP retrieve
tool, and the streaming feedback recorders retrieve by hash only
- proactive context-tracker expansion always restores full content
- delete CompressionStore.search() and its BM25 machinery (the bm25
module stays — it is still used by relevance/)
- CCRToolCall.query, CCRToolResult.was_search, and
ExpansionRecommendation.expand_full/search_query are removed
Plugins (advertised a now-defunct query param to the LLM):
- hermes (Python), openclaw + opencode (TypeScript) retrieve tools drop
`query` from their schemas, signatures, request URLs, and tests
Benchmarks/docs:
- ccr_regression + adversarial benchmarks switch from store.search() to
full hash retrieval (search input-injection tests repurposed to the
hash, the only remaining input surface)
- wiki/ARCHITECTURE.md, wiki/ccr.md, docs/content/docs/ccr.mdx,
config.py and store docstrings updated to describe hash-only retrieval
Tests updated to assert full-content retrieval and guard the removed
surface; the full CCR/proxy/store/TOIN suite passes. ruff + mypy clean.
## Description
<!-- Briefly explain the change and why it is needed. -->
Closes #
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
-
## Testing
<!-- Check what you actually ran, then paste the real command output
below. -->
- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
# Paste relevant command output or artifact links here
```
## Real Behavior Proof
- Environment:
- Exact command / steps:
- Observed result:
- Not tested:
## Review Readiness
- [ ] I have performed a self-review
- [ ] This PR is ready for human review
## Checklist
- [ ] My code follows the project's style guidelines
- [ ] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
Add screenshots to help explain your changes.
## Additional Notes
<!-- Mention any N/A checklist items, tradeoffs, follow-ups, or
maintainer context. -->
## Description
A data-driven push for better compression savings without accuracy loss,
in four parts: expose and tune the Rust compressor knobs, harden the CCR
retrieval store, add traffic-audit tooling that sizes opportunities from
real transcripts, and introduce **read maturation** — a new,
live-validated mechanism that compresses Read outputs *before* they ever
enter the provider prefix cache.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
### 1. Rust compressor extraction
- Expose `lossless_min_savings_ratio` end-to-end and lower the default
0.30 → 0.15 (lockstep across Rust, PyO3, and both Python config classes)
so the lossless Table/CSV compaction path wins more often.
- Expose the `CompactConfig` heuristics (core-field fraction,
heterogeneity ratio, flatten cap, bucket bounds) through PyO3 + Python.
- `SearchCompressor` grouped-by-file output (`rg --heading` style — path
once per file instead of per match). Library default off; the proxy
enables it in token mode.
- Complete `factor_out_constants`: constant fields now emit once in a
`_constant_fields` sentinel with slim rows (defensive per-item value
match; default off).
- `ContentRouter` accepts a SmartCrusher config override and the
search-grouping knob.
### 2. CCR store hardening
- Session-scale TTL: 300s → 1800s (CCRConfig, CompressionEntry,
CompressionStore, Rust `DEFAULT_TTL` — lockstep).
- **SQLite is the default CCR backend** (`~/.headroom/ccr_store.db`,
WAL): survives proxy restarts and is shared across workers.
`HEADROOM_CCR_BACKEND=memory` opts out.
- Multi-worker safety: `busy_timeout`, and corruption detection narrowed
so transient `SQLITE_BUSY` errors can never trigger database deletion.
- Data-at-rest hygiene: `chmod 600` on db + sidecars, expired rows swept
at open.
- Retrieval-miss messages are actionable (re-read the file / re-run the
command).
### 3. Traffic audit tooling (measure before tuning)
- `headroom audit-reads`: sizes Read opportunities from local Claude
Code transcripts (read share, stale %, line-number overhead, context
residency, cache-death windows).
- `--simulate-maturation`: Mechanism B risk sizing (re-read rates,
never-touched-again share, quiesce coverage, at-risk edits).
- `--codex`: shell-read classifier for Codex transcripts (rtk-wrapper
aware, workdir resolution).
- Findings that shaped this PR (81 sessions): Reads are 67% of tool
bytes; median Read lingers 118 turns (~13x lifetime cost); a prototyped
repeat-Read dedup measured 0.1% and was **removed** rather than shipped
as dead code.
### 4. Read maturation (Mechanism B) — experimental, default OFF
- Activity-based: a fresh large Read is held **out** of the provider
cache (trailing breakpoint relocated before it), stays verbatim while
its file is active, and matures into a CCR-backed marker once the file
is quiet for `quiesce_turns` (default 5; `max_hold_turns` bounds busy
files).
- Only the final compressed form ever enters the cache — **no cached
byte is ever mutated**; matured markers replay byte-identically.
- Wired into the Anthropic handler behind `--read-maturation` /
`HEADROOM_READ_MATURATION=1`; session state rides on the prefix tracker;
advisory (can never fail a request).
- Live-validated against the Anthropic API: held content excluded from
cache_creation; after maturation the prior cached prefix still served —
the no-bust invariant holds end-to-end.
### 5. Rebase / CI fixups (this update)
- Rebased onto latest `main` (was 28 commits behind): picks up `ci: pass
CODECOV_TOKEN to coverage uploads (#968)`, which is what was turning the
4 test shards red — the tests themselves passed (1528) but the post-test
codecov upload exited non-zero on a protected branch.
- Resolved the duplicate `lossless_min_savings_ratio` that two
independent main/branch additions left in `SmartCrusherConfig` and the
Rust-config kwarg (import-time `SyntaxError` + mypy `no-redef`).
- Aligned CCR tests with the new defaults (SQLite backend, 1800s TTL)
across `test_ccr`, `test_adapter_hooks`, `test_compression_store`,
`test_proxy_ccr`, and the lossy row-drop bridge test.
## Testing
<!-- Check what you actually ran, then paste the real command output
below. -->
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_proxy_ccr.py tests/test_ccr.py tests/test_compression_store.py tests/test_adapter_hooks.py tests/test_ccr_row_drop_store_bridge.py -q
170 passed, 4 warnings in 42.49s
$ python -m pytest tests/test_audit_reads.py tests/test_audit_codex.py tests/test_read_maturation.py tests/test_transforms_content_router.py tests/test_smart_crusher_toin_attachment.py -q
83 passed
$ mypy headroom/
Success: no issues found in 365 source files
$ python -m compileall headroom/ -q
COMPILE-OK
# CI (run 27488990477, pre-rebase head): all 4 shards ran to completion —
# "1528 passed, 120 skipped, 4922 deselected"
# The red shards were the codecov upload step, not test failures; fixed by
# the #968 rebase above.
```
## Real Behavior Proof
- Environment: macOS (darwin), Python 3.12 venv; branch
`feat/compression-extraction` rebased onto `origin/main` (head
7cb0f43b); GitHub Actions CI run 27488990477 for the test shards
- Exact command / steps: rebased onto latest main (clean, 13 commits
replayed, 0 conflicts); ran the pytest suites and mypy above locally;
inspected CI shard logs to confirm the failure was the codecov upload,
not the test phase
- Observed result: 253 targeted tests pass locally; mypy clean on 365
files; CI test phase reports `1528 passed, 120 skipped`; the only red
step (codecov `upload-coverage` → "Token required because branch is
protected") is resolved by the rebased-in #968 CODECOV_TOKEN fix
- Not tested: the read-maturation live-API no-bust validation
(`tests/test_live/`) was not re-run in this rebase pass (requires
provider keys); it was validated when the feature first landed, and no
maturation code changed in the rebase — only CCR-default test assertions
and the duplicate-field resolution
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
CHANGELOG is generated by release-please from the conventional commits,
so the CHANGELOG box is intentionally left unchecked. "Manual testing
performed" is unchecked deliberately — see `Real Behavior Proof` → `Not
tested` for the exact boundary (the live-API maturation validation was
not re-run in this rebase pass).
### Follow-ups (tracked, not in this PR)
- Mechanism B provider extensions: OpenAI-family wiring (no breakpoint
hold — bounded near-tail bust) and the Codex runtime read-detector (the
audit classifier is the prototype).
- Pilot enablement playbook: run `audit-reads --simulate-maturation` on
target traffic → pick `quiesce_turns` → enable via env → watch cache hit
rate + `read_maturation:N` transform tags.
## Description
Make the CCR retrieval store TTL configurable so long-running agent jobs
can keep `headroom_retrieve` markers resolvable beyond the previous
hard-coded 300-second window.
Fixes#714
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Add `HEADROOM_CCR_TTL_SECONDS` for the global CCR `CompressionStore`
default TTL, with validation and fallback to 300 seconds.
- Expose the effective CCR TTL as `store.default_ttl_seconds` from
`/v1/retrieve/stats`.
- Distinguish missing vs expired CCR retrieval failures in
`/v1/retrieve`, `/v1/retrieve/{hash}`, tool-call handling, and CCR
response handling.
- Update CCR docs, README wording, and CHANGELOG for the user-facing TTL
behavior.
- Add regression tests for env-configured TTL, invalid env fallback,
explicit TTL precedence, stats exposure, and expired retrieval detail.
## Reproduction
Before this change, the proxy-global CCR store always used the default
300-second TTL when callers used `get_compression_store()` without an
explicit `default_ttl`. A long-running agent job could receive a
`<<ccr:...>>` marker and fail later with a generic not-found/expired
response after the fixed 5-minute retention window.
The new tests cover this by setting `HEADROOM_CCR_TTL_SECONDS`, creating
CCR entries through the same global store used by the proxy, and
asserting that stored entries and `/v1/retrieve/stats` use the
configured retention.
## Real behavior proof
Setup tested:
- macOS 15.7.2
- Python 3.13.9 via `uv run --frozen --extra dev --extra proxy`
- Local Headroom proxy subprocess: `python -m headroom.proxy.server`
- Proxy config: `HEADROOM_TELEMETRY=off`,
`HEADROOM_CACHE_ENABLED=false`, `HEADROOM_RATE_LIMIT_ENABLED=false`
- Provider/model: no live upstream provider needed; exercised local
`/v1/compress` -> `/v1/retrieve` CCR HTTP flow with model `gpt-4o`
Exact steps run after the patch:
1. Start a real Headroom proxy process with
`HEADROOM_CCR_TTL_SECONDS=7200`.
2. POST a 200-item tool-result payload to `/v1/compress`.
3. Extract the emitted `<<ccr:...>>` marker hash from the compressed
messages.
4. GET `/v1/retrieve/stats`.
5. POST `/v1/retrieve` with the marker hash.
6. Repeat with `HEADROOM_CCR_TTL_SECONDS=1`, wait 1.5 seconds, and
retrieve the same way to verify explicit expiration reporting.
Observed result:
```json
{
"long_ttl": {
"ccr_hash": "b473e632aa47",
"retrieve_status": 200,
"retrieved_content_has_result_199": true,
"stats_default_ttl_seconds": 7200,
"stats_entry_count": 1,
"ttl_seconds": 7200
},
"short_ttl_expired": {
"ccr_hash": "b473e632aa47",
"retrieve_detail": "Entry expired (CCR TTL: 1 seconds; age: 2 seconds)",
"retrieve_status": 404,
"stats_default_ttl_seconds": 1,
"stats_entry_count": 1,
"ttl_seconds": 1
}
}
```
What I did not test:
- A live OpenRouter/OpenAI/Anthropic upstream request.
- A durable/non-memory CCR backend.
- A literal 5+ minute wall-clock wait; the process E2E used `7200` to
prove configured retention and `1` second to prove expiration behavior
quickly.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
## Test Output
```bash
UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy pytest tests/test_compression_store.py tests/test_proxy_ccr.py tests/test_ccr_response_handler.py tests/test_ccr_response_handler_extra.py
# 152 passed, 2 warnings in 12.87s
UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy pytest tests/test_adapter_hooks.py tests/test_toin_feedback.py tests/test_toin_fixes.py
# 55 passed, 13 skipped, 1 warning in 0.53s
UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy ruff check .
# All checks passed!
UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy ruff format --check .
# 776 files already formatted
UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy mypy headroom
# Success: no issues found in 346 source files
```
Existing warnings observed in the targeted tests were unrelated to this
change:
- AnthropicProvider tiktoken approximation warning in proxy CCR tests.
- Deprecated `ToolIntelligenceNetwork.get_recommendation()` warning in
existing TOIN assertions.
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
Not applicable.
## Additional Notes
No new dependencies. Default behavior remains 300 seconds unless
`HEADROOM_CCR_TTL_SECONDS` is set.
Companion to the MD5→SHA-256 switch in 98d458f. The hash-pinning test
asserted `hashlib.md5(content.encode()).hexdigest()[:24]`; flip it to
the new function. Also expanded the failure message so the next
person debugging this knows why this gate exists and what they need
to verify if they're tempted to change the hash function again.
CI test (3.10/3.11/3.12/3.13) failed on this single assertion after
the SHA-256 switch; with the test updated, the rest of the
compression_store regression (76 tests) stays green.
- Update test_hash_uses_sha256_truncated → test_hash_uses_md5_truncated
to match the SHA256→MD5 change in compression_store.py
- Use errors="surrogatepass" in compute_hash to handle lone surrogates
in unicode content (fixes pre-existing UnicodeEncodeError)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bug fixes:
- Replace bare except handlers with specific exception types and logging
in proxy/server.py (6 instances for CCR, SSE parsing, cost tracking)
- Fix session_id filtering security bug in memory/backends/local.py
(sessions were not properly isolated in vector search)
New tests (344 total):
- test_ccr_batch_processor.py: 51 tests for batch result processing
- test_compression_store.py: 76 tests for compression cache
- test_log_compressor.py: 47 tests for log format detection/compression
- test_search_compressor.py: 48 tests for grep output compression
- test_integrations/langchain/: 122 tests for LangChain integration
(agents, memory, retriever, streaming)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
