The optional `query` parameter on headroom_retrieve routed retrieval
through CompressionStore.search(), which BM25-scored the items inside a
single cached blob and dropped everything below a 0.3 relevance floor.
On small per-blob corpora with conversational queries this returned an
empty result the large majority of the time, so the LLM saw "nothing
found" for content that was actually present — pushing users to turn
compression off entirely.
Retrieval is fundamentally a hash lookup (this already matches the Rust
proxy's CCR store, which is put/get only — "no BM25 search"). Remove the
query/search path end to end and always return the full original
content:
Core (Python proxy):
- tool schemas (anthropic/openai/google) drop the `query` property
- parse_tool_call returns the hash (str | None) instead of (hash, query)
- response handler, proxy POST/GET/tool-call handlers, the MCP retrieve
tool, and the streaming feedback recorders retrieve by hash only
- proactive context-tracker expansion always restores full content
- delete CompressionStore.search() and its BM25 machinery (the bm25
module stays — it is still used by relevance/)
- CCRToolCall.query, CCRToolResult.was_search, and
ExpansionRecommendation.expand_full/search_query are removed
Plugins (advertised a now-defunct query param to the LLM):
- hermes (Python), openclaw + opencode (TypeScript) retrieve tools drop
`query` from their schemas, signatures, request URLs, and tests
Benchmarks/docs:
- ccr_regression + adversarial benchmarks switch from store.search() to
full hash retrieval (search input-injection tests repurposed to the
hash, the only remaining input surface)
- wiki/ARCHITECTURE.md, wiki/ccr.md, docs/content/docs/ccr.mdx,
config.py and store docstrings updated to describe hash-only retrieval
Tests updated to assert full-content retrieval and guard the removed
surface; the full CCR/proxy/store/TOIN suite passes. ruff + mypy clean.
## Description
<!-- Briefly explain the change and why it is needed. -->
Closes #
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
-
## Testing
<!-- Check what you actually ran, then paste the real command output
below. -->
- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
# Paste relevant command output or artifact links here
```
## Real Behavior Proof
- Environment:
- Exact command / steps:
- Observed result:
- Not tested:
## Review Readiness
- [ ] I have performed a self-review
- [ ] This PR is ready for human review
## Checklist
- [ ] My code follows the project's style guidelines
- [ ] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
Add screenshots to help explain your changes.
## Additional Notes
<!-- Mention any N/A checklist items, tradeoffs, follow-ups, or
maintainer context. -->
Retire the request-time hint API. PR-B5 splits TOIN into two phases:
1. Observation: TOIN keeps recording compressions/retrievals at runtime,
but `get_recommendation()` is deprecated and now returns None.
2. Publish-then-load: the new `headroom.cli.toin_publish` CLI walks the
on-disk store and emits `recommendations.toml`. The Rust proxy reads
that file once at startup via `transforms::recommendations` and
exposes `get(auth_mode, model, structure_hash) -> Option<&Rec>`.
PR-F3 will wire the loader into the live-zone dispatcher.
Per-tenant aggregation: `_patterns` is now keyed by
`(auth_mode, model_family, sig_hash)` so PAYG/OAuth/subscription tenants
no longer share buckets. Callers that don't supply auth/model land in the
`("unknown", "unknown", sig_hash)` slot. Added `_make_pattern_key` helper
+ updated tests that previously indexed by raw `structure_hash`.
AuthMode is canonical in `transforms::live_zone`; `transforms::recommendations`
re-exports it (no duplicate enum). Live-zone enum gained `Unknown`,
`as_str()`, and `Hash` derive to serve recommendations callers without a
second source of truth.
Why: per-request hint calls coupled output to mutable TOIN state, breaking
prompt-cache stability across runs (P2-27, P5-56). Pulling advice into a
startup-published TOML keeps per-request output deterministic and lets the
deploy pipeline gate publication independently of proxy uptime.
Per-PR-B5 plan: REALIGNMENT/04-phase-B-live-zone.md.
Three bugs fixed:
1. observations counter: get_recommendation() now increments an
observations counter on the ToolPattern, so we can track whether
TOIN is being consulted during compression.
2. streaming CCR feedback: when the LLM generates headroom_retrieve
tool_use in a streaming response, the proxy now extracts the
hash/query and calls store.retrieve() to trigger the full feedback
chain (_log_retrieval → process_pending_feedback → toin.record_retrieval).
Previously, streaming responses (all Claude Code traffic) never
triggered TOIN feedback, leaving preserve_fields empty.
3. preserve_fields population: with retrievals now flowing to TOIN,
field_retrieval_frequency gets populated, which feeds into
_build_recommendation() → preserve_fields in CompressionHint.
SmartCrusher can now learn which specific fields to preserve.
Root cause: headroom_retrieve tool calls in streaming responses were
returned to the client but never recorded as retrieval events in TOIN.
The entire feedback loop was write-only — TOIN learned compression
patterns but never learned from retrieval signals.
Adds 15 new tests covering observations counter, retrieval recording,
preserve_fields population, and CCR feedback extraction.
