headroom/plugins/hermes/headroom_retrieve/__init__.py
Tejas Chopra c2fc4d3753
fix(ccr): make headroom_retrieve a hash-only full-content lookup (#1532)
The optional `query` parameter on headroom_retrieve routed retrieval
through CompressionStore.search(), which BM25-scored the items inside a
single cached blob and dropped everything below a 0.3 relevance floor.
On small per-blob corpora with conversational queries this returned an
empty result the large majority of the time, so the LLM saw "nothing
found" for content that was actually present — pushing users to turn
compression off entirely.

Retrieval is fundamentally a hash lookup (this already matches the Rust
proxy's CCR store, which is put/get only — "no BM25 search"). Remove the
query/search path end to end and always return the full original
content:

Core (Python proxy):
- tool schemas (anthropic/openai/google) drop the `query` property
- parse_tool_call returns the hash (str | None) instead of (hash, query)
- response handler, proxy POST/GET/tool-call handlers, the MCP retrieve
tool, and the streaming feedback recorders retrieve by hash only
- proactive context-tracker expansion always restores full content
- delete CompressionStore.search() and its BM25 machinery (the bm25
module stays — it is still used by relevance/)
- CCRToolCall.query, CCRToolResult.was_search, and
ExpansionRecommendation.expand_full/search_query are removed

Plugins (advertised a now-defunct query param to the LLM):
- hermes (Python), openclaw + opencode (TypeScript) retrieve tools drop
`query` from their schemas, signatures, request URLs, and tests

Benchmarks/docs:
- ccr_regression + adversarial benchmarks switch from store.search() to
full hash retrieval (search input-injection tests repurposed to the
hash, the only remaining input surface)
- wiki/ARCHITECTURE.md, wiki/ccr.md, docs/content/docs/ccr.mdx,
config.py and store docstrings updated to describe hash-only retrieval

Tests updated to assert full-content retrieval and guard the removed
surface; the full CCR/proxy/store/TOIN suite passes. ruff + mypy clean.

## Description

<!-- Briefly explain the change and why it is needed. -->

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- 

## Testing

<!-- Check what you actually ran, then paste the real command output
below. -->

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
# Paste relevant command output or artifact links here
```

## Real Behavior Proof

- Environment:
- Exact command / steps:
- Observed result:
- Not tested:

## Review Readiness

- [ ] I have performed a self-review
- [ ] This PR is ready for human review

## Checklist

- [ ] My code follows the project's style guidelines
- [ ] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

Add screenshots to help explain your changes.

## Additional Notes

<!-- Mention any N/A checklist items, tradeoffs, follow-ups, or
maintainer context. -->
2026-06-28 10:32:43 -07:00

94 lines
3.5 KiB
Python

"""Headroom CCR retrieve plugin.
The headroom proxy (127.0.0.1:8787) compresses large tool outputs in LLM
requests, replacing them with markers like ``[N items compressed ...
hash=abc123]`` or ``<<ccr:abc123>>``. This plugin gives Hermes a tool to fetch the original
uncompressed content back from the proxy's compression store, so compressed
markers are no longer a black box.
Storage is in-memory on the proxy side with a TTL — expired or
post-proxy-restart hashes return 404 and the tool reports that clearly.
"""
from __future__ import annotations
import httpx
from tools.registry import tool_error, tool_result
_PROXY_URL = "http://127.0.0.1:8787"
HEADROOM_RETRIEVE_SCHEMA = {
"name": "headroom_retrieve",
"description": (
"Retrieve the original uncompressed content behind a headroom "
"compression marker. Markers look like "
"'[N items compressed ... hash=abc123]' OR '<<ccr:abc123>>' OR "
"'<<ccr:abc123,base64,4.5KB>>'. They are NOT file paths — never try "
"to cat/read them. When you see one in a tool result or in "
"conversation history, call this tool with the hash (the hex string "
"after 'hash=' or 'ccr:') to read the full original content instead "
"of guessing or re-running the command. Retrieval is by hash and "
"always returns the complete original content. Content expires after "
"a TTL — if expired, re-run the original command instead."
),
"parameters": {
"type": "object",
"properties": {
"hash": {
"type": "string",
"description": "Hash from the compression marker, e.g. 'abc123' from '[... hash=abc123]' or '<<ccr:abc123>>'",
},
},
"required": ["hash"],
},
}
def _handle_headroom_retrieve(args: dict, **kw) -> str:
hash_key = str(args.get("hash") or "").strip()
# Tolerate the model passing the whole marker instead of the bare hash:
# '<<ccr:abc123,base64,4.5KB>>' / 'ccr:abc123' / 'hash=abc123' -> 'abc123'
hash_key = hash_key.strip("<>").removeprefix("ccr:").removeprefix("hash=")
hash_key = hash_key.split(",")[0].strip()
if not hash_key:
return tool_error(
"hash is required (from a '[... hash=abc123]' or '<<ccr:abc123>>' marker)"
)
payload: dict = {"hash": hash_key}
try:
resp = httpx.post(f"{_PROXY_URL}/v1/retrieve", json=payload, timeout=15)
except httpx.HTTPError as exc:
return tool_error(
f"headroom proxy unreachable at {_PROXY_URL} ({type(exc).__name__}). "
"The proxy may be down; re-run the original command to get the data."
)
if resp.status_code == 404:
return tool_error(
"Content not found: expired (TTL passed) or proxy restarted. "
"Re-run the original command to regenerate the data."
)
if resp.status_code != 200:
return tool_error(f"headroom proxy returned HTTP {resp.status_code}: {resp.text[:200]}")
data = resp.json()
return tool_result(
{
"original_content": data.get("original_content", ""),
"original_tokens": data.get("original_tokens"),
"tool_name": data.get("tool_name"),
}
)
def register(ctx) -> None:
"""Register the headroom_retrieve tool. Called by the plugin loader."""
ctx.register_tool(
name="headroom_retrieve",
toolset="headroom",
schema=HEADROOM_RETRIEVE_SCHEMA,
handler=_handle_headroom_retrieve,
emoji="🗜️",
)