mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
4 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Tejas Chopra
|
c2fc4d3753
|
fix(ccr): make headroom_retrieve a hash-only full-content lookup (#1532)
The optional `query` parameter on headroom_retrieve routed retrieval through CompressionStore.search(), which BM25-scored the items inside a single cached blob and dropped everything below a 0.3 relevance floor. On small per-blob corpora with conversational queries this returned an empty result the large majority of the time, so the LLM saw "nothing found" for content that was actually present — pushing users to turn compression off entirely. Retrieval is fundamentally a hash lookup (this already matches the Rust proxy's CCR store, which is put/get only — "no BM25 search"). Remove the query/search path end to end and always return the full original content: Core (Python proxy): - tool schemas (anthropic/openai/google) drop the `query` property - parse_tool_call returns the hash (str | None) instead of (hash, query) - response handler, proxy POST/GET/tool-call handlers, the MCP retrieve tool, and the streaming feedback recorders retrieve by hash only - proactive context-tracker expansion always restores full content - delete CompressionStore.search() and its BM25 machinery (the bm25 module stays — it is still used by relevance/) - CCRToolCall.query, CCRToolResult.was_search, and ExpansionRecommendation.expand_full/search_query are removed Plugins (advertised a now-defunct query param to the LLM): - hermes (Python), openclaw + opencode (TypeScript) retrieve tools drop `query` from their schemas, signatures, request URLs, and tests Benchmarks/docs: - ccr_regression + adversarial benchmarks switch from store.search() to full hash retrieval (search input-injection tests repurposed to the hash, the only remaining input surface) - wiki/ARCHITECTURE.md, wiki/ccr.md, docs/content/docs/ccr.mdx, config.py and store docstrings updated to describe hash-only retrieval Tests updated to assert full-content retrieval and guard the removed surface; the full CCR/proxy/store/TOIN suite passes. ruff + mypy clean. ## Description <!-- Briefly explain the change and why it is needed. --> Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # Paste relevant command output or artifact links here ``` ## Real Behavior Proof - Environment: - Exact command / steps: - Observed result: - Not tested: ## Review Readiness - [ ] I have performed a self-review - [ ] This PR is ready for human review ## Checklist - [ ] My code follows the project's style guidelines - [ ] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) Add screenshots to help explain your changes. ## Additional Notes <!-- Mention any N/A checklist items, tradeoffs, follow-ups, or maintainer context. --> |
||
|
Rod Boev
|
cabf666b34
|
fix(ccr): wrap proactive expansion injection in XML attribution tag (#1398)
## Description In multi-agent threads, Headroom injects the proactive context expansion block directly into the latest non-frozen user turn's first text block as plain bracketed text. When that turn contains `<peer_turn from="AgentX">...</peer_turn>` markup, the injected block lands adjacent to agent-attributed regions with no machine-readable boundary. LLMs, loggers, and attribution parsers cannot distinguish Headroom-injected context from content attributed to AgentX, causing misattribution or treatment of the block as user-authored prompt injection. Root cause: `format_expansions_for_context` in `headroom/headroom/ccr/context_tracker.py` (~line 550) returns plain text bounded only by human-readable brackets (`[Proactive Context Expansion...]` / `[End Proactive Expansion]`). No XML wrapper is added at the injection site either. This PR wraps the entire return value of `format_expansions_for_context` in `<headroom_proactive_expansion>` tags. The existing brackets are preserved inside for human readability; the outer tag gives downstream consumers a provenance boundary consistent with the `<peer_turn>` XML convention used in multi-agent turns. Closes #503 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/headroom/ccr/context_tracker.py`: restructured the tail of `format_expansions_for_context` to wrap the joined parts in `<headroom_proactive_expansion>...</headroom_proactive_expansion>`. Inner brackets are unchanged. Empty-input early return is unchanged. Payload body is sanitized to escape any stray `</headroom_proactive_expansion>` close tag in expansion content, preventing wrapper boundary ambiguity. - `tests/test_ccr_context_tracker.py`: added XML wrapper assertions to existing formatter tests; new standalone tests for wrapper structure, full injection chain identifiability, and close-tag escape robustness. - `CHANGELOG.md`: entry under `[Unreleased]` for the injection format change. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_ccr_context_tracker.py -x -q`) - [x] Linting passes (`uv run ruff check .`) - [ ] Type checking passes (`uv run mypy headroom`) — N/A: single-expression change, no new types - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text uv run pytest tests/test_ccr_context_tracker.py -x -q 41 passed in 2.41s ``` ## Real Behavior Proof - Environment: local, Python 3.11+, `uv sync --extra dev` - Exact command / steps: `uv run python -c "from headroom.ccr.context_tracker import ContextTracker; t = ContextTracker(); r = t.format_expansions_for_context([{'hash':'h1','type':'full','content':'ctx','item_count':1,'reason':'r'}]); print(r.startswith('<headroom_proactive_expansion>'))"` → `True` on head, `False` on base; `uv run pytest tests/test_ccr_context_tracker.py -x -q` → 41 passed - Observed result: return value now starts with `<headroom_proactive_expansion>` and ends with `</headroom_proactive_expansion>`; inner `[Proactive Context Expansion...]` and `[End Proactive Expansion]` brackets are present and not duplicated - Not tested: live multi-agent thread rendering with Anthropic API; downstream attribution parser behavior in production ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes The injection site (`AnthropicHandlerMixin._append_context_to_latest_non_frozen_user_turn` in `anthropic.py`) is unchanged. Existing tests that check for `"[Proactive Context Expansion" in formatted` continue to pass since the brackets are preserved inside the XML wrapper. The tag name `headroom_proactive_expansion` uses underscores (not hyphens) to match the `snake_case` convention used in the repo's other XML-like constructs. To prevent a stray `</headroom_proactive_expansion>` inside expansion content (e.g., code snippets) from breaking the wrapper boundary, the body is sanitized to `<\/headroom_proactive_expansion>` before wrapping; a test covers this edge case. --------- Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
chopratejas
|
1bc163f5bc |
fix(ccr): scope proactive expansion by workspace (cross-project leak)
Closes the cross-project context leak Jocelyn reported 2026-05-26:
working on a Ruby/Rails project (daphni-rails), an unrelated Python
file (an Ollama inference provider from project `tamag0`) was being
injected into context as "Proactive Context Expansion - relevant to
your query". Two completely different projects, two different
languages, two different working directories — but the same proxy
process was serving both, and the in-memory ContextTracker had no
workspace identity to filter on.
Root cause
----------
`self.ccr_context_tracker` is one instance per proxy process. Every
session, every project, every user shared the same `_contexts` dict.
`track_compression()` stored sample content with no provenance key;
`analyze_query()` ran lexical keyword overlap across the full dict
without filtering. Within the 5-minute age window, surface-level
token matches ("provider", "session", "oauth", generic code/test
structure) scored above the 0.3 relevance threshold, recommendations
came back, and execute_expansions() injected the full original
content into a foreign session.
Refuted: this is NOT a race condition (joce's hypothesis). It
reproduces single-threaded, one-request-at-a-time. Plain shared
mutable state.
Fix
---
Add a required `workspace_key` to the tracker API and filter on it
inside `analyze_query`:
1. `CompressedContext` gets a `workspace_key: str` field.
2. `track_compression(..., workspace_key=...)` is now keyword-only,
no default — fail-loud on missing.
3. `analyze_query(..., workspace_key=...)` is also keyword-only; an
empty workspace_key short-circuits to `[]` (fail-closed per
`feedback_no_silent_fallbacks`).
4. The loop at `analyze_query` skips any entry whose workspace_key
differs from the request's.
In the Anthropic proxy handler:
5. New `_resolve_ccr_workspace(request, body)` static helper uses the
memory subsystem's `ProjectResolver` so CCR and memory agree on
project identity. Tier order: x-headroom-project-id →
x-headroom-cwd → CLI override → cwd: line in system prompt.
6. Both track and analyze sites gate on `ccr_workspace_key` being
non-empty — turning off proactive expansion entirely when project
identity can't be resolved is the safest default (it's an
optimization, not correctness).
7. `format_expansions_for_context(expansions, workspace_label=...)`
was already wired (GH #462 Fix C); the call site now passes the
label so the injected block declares its provenance, symmetric
with the memory injection header.
Affected population
-------------------
- Default mode (no `--cache`): bug fixed.
- Cache mode: was never affected — proactive expansion short-
circuits in cache mode to preserve prefix stability.
Tests
-----
- 6 new workspace-scoping tests in `test_ccr_context_tracker.py`:
same-workspace match still works, cross-workspace silently
filtered, empty workspace_key fail-closes, two workspaces each
see only their own, workspace_label propagates to formatter, LRU
cross-workspace doesn't leak even with full tracker.
- 6 new `_resolve_ccr_workspace` resolver tests in
`test_proxy_handler_helpers.py`: explicit project-id wins, cwd
header → key+label, two cwds get distinct keys, no-signal
fail-closed, system-prompt cwd: fallback, malformed request
fail-closed.
- 32 existing tracker tests updated to pass `workspace_key="ws-test"`.
- 55/55 tests pass; ci-precheck green.
Defense-in-depth follow-up
--------------------------
The compression_store itself (`headroom/cache/compression_store.py`)
also lacks workspace scoping — a CCR `headroom_retrieve` call from
Project B for a hash created by Project A would succeed. The
practical attack surface is closed by this PR (hashes only reach
Project B's model via proactive expansion, now gated), but
defense-in-depth hardening of the store is worth a separate PR.
Filed as task #44.
|
||
|
chopratejas
|
d724f14022 |
v0.2.2: Add CCR Response Handler, Context Tracker, and restructure docs
Features: - CCR Response Handler: Automatically intercepts and handles headroom_retrieve tool calls - CCR Context Tracker: Multi-turn awareness with proactive expansion of relevant compressed content - New CCR demo script showing before/after flow Documentation: - Restructured README from 885 lines to 190 lines for better DevEx - Split detailed docs into focused guides: ccr.md, sdk.md, configuration.md, text-compression.md, llmlingua.md, metrics.md, errors.md - Updated docs/README.md index with all new documentation Tests: - Added comprehensive tests for Response Handler (32 tests) - Added comprehensive tests for Context Tracker (32 tests) - All 977 tests passing |