mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
10 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Abhay Singh
|
113894600c
|
fix(cache/ccr): don't evict a live entry on a duplicate store at capacity (#2082)
## Description
`CompressionStore.store` (`headroom/cache/compression_store.py`) runs
eviction **before** it knows
whether the incoming `hash_key` is new or a re-store of an
already-present key:
```python
with self._lock:
self._evict_if_needed() # <-- runs first
existing = self._backend.get(hash_key)
if existing is not None:
... # duplicate / collision: overwrite in place
self._stale_heap_entries += 1
self._backend.set(hash_key, entry)
```
When the store is full and the incoming key **already exists** (a
duplicate re-store),
`_evict_if_needed()` removes the oldest *distinct* entry to "make room"
— but then `set()` merely
overwrites the existing key in place, so no room was ever needed. Net
effect: `count` drops to
`max_entries - 1` and a **live, never-retrieved entry is destroyed**.
That entry's `<<ccr:...>>`
marker, still sitting in the conversation history, then resolves to a
404 on `/v1/retrieve`.
This is not a corner case: the CCR mirror bridge
(`_mirror_single_hash_to_python_store` in
`smart_crusher.py`) re-`store()`s the same `explicit_hash` every turn a
`<<ccr:…>>` marker is
re-encountered, and markers persist across turns — so a full store
silently deletes a live sibling
entry on each duplicate.
Concrete (with the repo's `max_entries=3` fixture): store c0, c1, c2
(full), then re-store c1
(same content ⇒ same hash). Eviction pops the oldest (c0), deletes it,
then c1 is overwritten in
place. Final state: {c1, c2}, count 2, and **c0 is gone** — its marker
is now unredeemable.
Closes: no issue filed — found while auditing the compression store.
## Fix
Decide novelty before evicting: only `_evict_if_needed()` for a
genuinely new key; a
duplicate/replace overwrites in place (no eviction). The
collision/duplicate logging and
stale-heap accounting are unchanged.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/cache/compression_store.py`: `store()` reads `existing`
first and only evicts when the key is new.
- `tests/test_compression_store.py`: add
`test_duplicate_store_at_capacity_does_not_evict` (re-store an existing
hash at capacity keeps all entries and count at `max_entries`).
## Testing
- [x] New regression test added (`tests/test_compression_store.py`)
- [x] Linting/formatting clean — run with the CI-pinned `ruff==0.15.17`
- [ ] Full `pytest` deferred to CI (local-OOM reason below).
```text
$ uvx ruff@0.15.17 check headroom/cache/compression_store.py tests/test_compression_store.py
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.10, headroom from this branch.
Importing `headroom` pulls in the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I verified the store/evict
logic with a dependency-free script (replicating the backend + eviction
heap) and left the full pytest to CI.
- Exact command / steps: filled a `max_entries=3` store with c0/c1/c2,
then re-stored c1 (duplicate), through the old (evict-first) and new
(check-first) logic; also confirmed a genuinely new key still evicts the
oldest.
- Observed result: the old logic drops c0 (count 2); the new keeps all
three; and a new key at capacity still evicts the oldest:
```text
OLD: after duplicate re-store of h1 -> keys=['h1', 'h2'] count=2
NEW: after duplicate re-store of h1 -> keys=['h0', 'h1', 'h2'] count=3
NEW still evicts oldest for a genuinely new key at capacity
DUPLICATE-STORE EVICTION FIX VERIFIED (old drops a live entry; new keeps it)
```
- Not tested: a full proxy CCR round-trip (needs the heavy stack). The
fix is confined to `store()` and the new test drives it directly with
the `max_entries=3` fixture. Existing eviction tests use distinct keys
and stay green. Full local `pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — ran
lint + a standalone logic check; full pytest deferred to CI (local OOM,
disclosed above)
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- Internal reordering only, no signature change; no call sites or
backend mocks break.
- @JerrettDavis tagging you — this silently drops a live CCR entry
(making its marker 404) whenever a duplicate hash is re-stored at
capacity, which the mirror bridge does routinely. Thanks.
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Tejas Chopra
|
c2fc4d3753
|
fix(ccr): make headroom_retrieve a hash-only full-content lookup (#1532)
The optional `query` parameter on headroom_retrieve routed retrieval through CompressionStore.search(), which BM25-scored the items inside a single cached blob and dropped everything below a 0.3 relevance floor. On small per-blob corpora with conversational queries this returned an empty result the large majority of the time, so the LLM saw "nothing found" for content that was actually present — pushing users to turn compression off entirely. Retrieval is fundamentally a hash lookup (this already matches the Rust proxy's CCR store, which is put/get only — "no BM25 search"). Remove the query/search path end to end and always return the full original content: Core (Python proxy): - tool schemas (anthropic/openai/google) drop the `query` property - parse_tool_call returns the hash (str | None) instead of (hash, query) - response handler, proxy POST/GET/tool-call handlers, the MCP retrieve tool, and the streaming feedback recorders retrieve by hash only - proactive context-tracker expansion always restores full content - delete CompressionStore.search() and its BM25 machinery (the bm25 module stays — it is still used by relevance/) - CCRToolCall.query, CCRToolResult.was_search, and ExpansionRecommendation.expand_full/search_query are removed Plugins (advertised a now-defunct query param to the LLM): - hermes (Python), openclaw + opencode (TypeScript) retrieve tools drop `query` from their schemas, signatures, request URLs, and tests Benchmarks/docs: - ccr_regression + adversarial benchmarks switch from store.search() to full hash retrieval (search input-injection tests repurposed to the hash, the only remaining input surface) - wiki/ARCHITECTURE.md, wiki/ccr.md, docs/content/docs/ccr.mdx, config.py and store docstrings updated to describe hash-only retrieval Tests updated to assert full-content retrieval and guard the removed surface; the full CCR/proxy/store/TOIN suite passes. ruff + mypy clean. ## Description <!-- Briefly explain the change and why it is needed. --> Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # Paste relevant command output or artifact links here ``` ## Real Behavior Proof - Environment: - Exact command / steps: - Observed result: - Not tested: ## Review Readiness - [ ] I have performed a self-review - [ ] This PR is ready for human review ## Checklist - [ ] My code follows the project's style guidelines - [ ] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) Add screenshots to help explain your changes. ## Additional Notes <!-- Mention any N/A checklist items, tradeoffs, follow-ups, or maintainer context. --> |
||
|
Tejas Chopra
|
b7be3814f1
|
feat: compression extraction — Rust knob exposure, CCR hardening, traffic audits (#818)
## Description
A data-driven push for better compression savings without accuracy loss,
in four parts: expose and tune the Rust compressor knobs, harden the CCR
retrieval store, add traffic-audit tooling that sizes opportunities from
real transcripts, and introduce **read maturation** — a new,
live-validated mechanism that compresses Read outputs *before* they ever
enter the provider prefix cache.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
### 1. Rust compressor extraction
- Expose `lossless_min_savings_ratio` end-to-end and lower the default
0.30 → 0.15 (lockstep across Rust, PyO3, and both Python config classes)
so the lossless Table/CSV compaction path wins more often.
- Expose the `CompactConfig` heuristics (core-field fraction,
heterogeneity ratio, flatten cap, bucket bounds) through PyO3 + Python.
- `SearchCompressor` grouped-by-file output (`rg --heading` style — path
once per file instead of per match). Library default off; the proxy
enables it in token mode.
- Complete `factor_out_constants`: constant fields now emit once in a
`_constant_fields` sentinel with slim rows (defensive per-item value
match; default off).
- `ContentRouter` accepts a SmartCrusher config override and the
search-grouping knob.
### 2. CCR store hardening
- Session-scale TTL: 300s → 1800s (CCRConfig, CompressionEntry,
CompressionStore, Rust `DEFAULT_TTL` — lockstep).
- **SQLite is the default CCR backend** (`~/.headroom/ccr_store.db`,
WAL): survives proxy restarts and is shared across workers.
`HEADROOM_CCR_BACKEND=memory` opts out.
- Multi-worker safety: `busy_timeout`, and corruption detection narrowed
so transient `SQLITE_BUSY` errors can never trigger database deletion.
- Data-at-rest hygiene: `chmod 600` on db + sidecars, expired rows swept
at open.
- Retrieval-miss messages are actionable (re-read the file / re-run the
command).
### 3. Traffic audit tooling (measure before tuning)
- `headroom audit-reads`: sizes Read opportunities from local Claude
Code transcripts (read share, stale %, line-number overhead, context
residency, cache-death windows).
- `--simulate-maturation`: Mechanism B risk sizing (re-read rates,
never-touched-again share, quiesce coverage, at-risk edits).
- `--codex`: shell-read classifier for Codex transcripts (rtk-wrapper
aware, workdir resolution).
- Findings that shaped this PR (81 sessions): Reads are 67% of tool
bytes; median Read lingers 118 turns (~13x lifetime cost); a prototyped
repeat-Read dedup measured 0.1% and was **removed** rather than shipped
as dead code.
### 4. Read maturation (Mechanism B) — experimental, default OFF
- Activity-based: a fresh large Read is held **out** of the provider
cache (trailing breakpoint relocated before it), stays verbatim while
its file is active, and matures into a CCR-backed marker once the file
is quiet for `quiesce_turns` (default 5; `max_hold_turns` bounds busy
files).
- Only the final compressed form ever enters the cache — **no cached
byte is ever mutated**; matured markers replay byte-identically.
- Wired into the Anthropic handler behind `--read-maturation` /
`HEADROOM_READ_MATURATION=1`; session state rides on the prefix tracker;
advisory (can never fail a request).
- Live-validated against the Anthropic API: held content excluded from
cache_creation; after maturation the prior cached prefix still served —
the no-bust invariant holds end-to-end.
### 5. Rebase / CI fixups (this update)
- Rebased onto latest `main` (was 28 commits behind): picks up `ci: pass
CODECOV_TOKEN to coverage uploads (#968)`, which is what was turning the
4 test shards red — the tests themselves passed (1528) but the post-test
codecov upload exited non-zero on a protected branch.
- Resolved the duplicate `lossless_min_savings_ratio` that two
independent main/branch additions left in `SmartCrusherConfig` and the
Rust-config kwarg (import-time `SyntaxError` + mypy `no-redef`).
- Aligned CCR tests with the new defaults (SQLite backend, 1800s TTL)
across `test_ccr`, `test_adapter_hooks`, `test_compression_store`,
`test_proxy_ccr`, and the lossy row-drop bridge test.
## Testing
<!-- Check what you actually ran, then paste the real command output
below. -->
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_proxy_ccr.py tests/test_ccr.py tests/test_compression_store.py tests/test_adapter_hooks.py tests/test_ccr_row_drop_store_bridge.py -q
170 passed, 4 warnings in 42.49s
$ python -m pytest tests/test_audit_reads.py tests/test_audit_codex.py tests/test_read_maturation.py tests/test_transforms_content_router.py tests/test_smart_crusher_toin_attachment.py -q
83 passed
$ mypy headroom/
Success: no issues found in 365 source files
$ python -m compileall headroom/ -q
COMPILE-OK
# CI (run 27488990477, pre-rebase head): all 4 shards ran to completion —
# "1528 passed, 120 skipped, 4922 deselected"
# The red shards were the codecov upload step, not test failures; fixed by
# the #968 rebase above.
```
## Real Behavior Proof
- Environment: macOS (darwin), Python 3.12 venv; branch
`feat/compression-extraction` rebased onto `origin/main` (head
|
||
|
Hc
|
2533f7703e
|
fix(ccr): make retrieval TTL configurable (#715)
## Description Make the CCR retrieval store TTL configurable so long-running agent jobs can keep `headroom_retrieve` markers resolvable beyond the previous hard-coded 300-second window. Fixes #714 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add `HEADROOM_CCR_TTL_SECONDS` for the global CCR `CompressionStore` default TTL, with validation and fallback to 300 seconds. - Expose the effective CCR TTL as `store.default_ttl_seconds` from `/v1/retrieve/stats`. - Distinguish missing vs expired CCR retrieval failures in `/v1/retrieve`, `/v1/retrieve/{hash}`, tool-call handling, and CCR response handling. - Update CCR docs, README wording, and CHANGELOG for the user-facing TTL behavior. - Add regression tests for env-configured TTL, invalid env fallback, explicit TTL precedence, stats exposure, and expired retrieval detail. ## Reproduction Before this change, the proxy-global CCR store always used the default 300-second TTL when callers used `get_compression_store()` without an explicit `default_ttl`. A long-running agent job could receive a `<<ccr:...>>` marker and fail later with a generic not-found/expired response after the fixed 5-minute retention window. The new tests cover this by setting `HEADROOM_CCR_TTL_SECONDS`, creating CCR entries through the same global store used by the proxy, and asserting that stored entries and `/v1/retrieve/stats` use the configured retention. ## Real behavior proof Setup tested: - macOS 15.7.2 - Python 3.13.9 via `uv run --frozen --extra dev --extra proxy` - Local Headroom proxy subprocess: `python -m headroom.proxy.server` - Proxy config: `HEADROOM_TELEMETRY=off`, `HEADROOM_CACHE_ENABLED=false`, `HEADROOM_RATE_LIMIT_ENABLED=false` - Provider/model: no live upstream provider needed; exercised local `/v1/compress` -> `/v1/retrieve` CCR HTTP flow with model `gpt-4o` Exact steps run after the patch: 1. Start a real Headroom proxy process with `HEADROOM_CCR_TTL_SECONDS=7200`. 2. POST a 200-item tool-result payload to `/v1/compress`. 3. Extract the emitted `<<ccr:...>>` marker hash from the compressed messages. 4. GET `/v1/retrieve/stats`. 5. POST `/v1/retrieve` with the marker hash. 6. Repeat with `HEADROOM_CCR_TTL_SECONDS=1`, wait 1.5 seconds, and retrieve the same way to verify explicit expiration reporting. Observed result: ```json { "long_ttl": { "ccr_hash": "b473e632aa47", "retrieve_status": 200, "retrieved_content_has_result_199": true, "stats_default_ttl_seconds": 7200, "stats_entry_count": 1, "ttl_seconds": 7200 }, "short_ttl_expired": { "ccr_hash": "b473e632aa47", "retrieve_detail": "Entry expired (CCR TTL: 1 seconds; age: 2 seconds)", "retrieve_status": 404, "stats_default_ttl_seconds": 1, "stats_entry_count": 1, "ttl_seconds": 1 } } ``` What I did not test: - A live OpenRouter/OpenAI/Anthropic upstream request. - A durable/non-memory CCR backend. - A literal 5+ minute wall-clock wait; the process E2E used `7200` to prove configured retention and `1` second to prove expiration behavior quickly. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ## Test Output ```bash UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy pytest tests/test_compression_store.py tests/test_proxy_ccr.py tests/test_ccr_response_handler.py tests/test_ccr_response_handler_extra.py # 152 passed, 2 warnings in 12.87s UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy pytest tests/test_adapter_hooks.py tests/test_toin_feedback.py tests/test_toin_fixes.py # 55 passed, 13 skipped, 1 warning in 0.53s UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy ruff check . # All checks passed! UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy ruff format --check . # 776 files already formatted UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy mypy headroom # Success: no issues found in 346 source files ``` Existing warnings observed in the targeted tests were unrelated to this change: - AnthropicProvider tiktoken approximation warning in proxy CCR tests. - Deprecated `ToolIntelligenceNetwork.get_recommendation()` warning in existing TOIN assertions. ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) Not applicable. ## Additional Notes No new dependencies. Default behavior remains 300 seconds unless `HEADROOM_CCR_TTL_SECONDS` is set. |
||
|
Tejas Chopra
|
62a1f23b88 | fix: log Codex ws cancellations safely | ||
|
Tejas Chopra
|
eaf5980b4a | fix: stabilize codex compression, stats, and proxy lifecycle | ||
|
chopratejas
|
914a34fbd8 |
test(crusher): update compression_store hash test to expect SHA-256[:24]
Companion to the MD5→SHA-256 switch in 98d458f. The hash-pinning test asserted `hashlib.md5(content.encode()).hexdigest()[:24]`; flip it to the new function. Also expanded the failure message so the next person debugging this knows why this gate exists and what they need to verify if they're tempted to change the hash function again. CI test (3.10/3.11/3.12/3.13) failed on this single assertion after the SHA-256 switch; with the test updated, the rest of the compression_store regression (76 tests) stays green. |
||
|
rrubayet321
|
2ae71fe44d | chore: add nosec B324 annotations to non-cryptographic MD5 usages and update temporary database path to use system temp directory | ||
|
chopratejas
|
cfd44b3f6a |
Fix test failures: update hash test for MD5, handle unicode surrogates
- Update test_hash_uses_sha256_truncated → test_hash_uses_md5_truncated to match the SHA256→MD5 change in compression_store.py - Use errors="surrogatepass" in compute_hash to handle lone surrogates in unicode content (fixes pre-existing UnicodeEncodeError) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
chopratejas
|
d3298368bf |
fix: improve error handling and add comprehensive test coverage
Bug fixes: - Replace bare except handlers with specific exception types and logging in proxy/server.py (6 instances for CCR, SSE parsing, cost tracking) - Fix session_id filtering security bug in memory/backends/local.py (sessions were not properly isolated in vector search) New tests (344 total): - test_ccr_batch_processor.py: 51 tests for batch result processing - test_compression_store.py: 76 tests for compression cache - test_log_compressor.py: 47 tests for log format detection/compression - test_search_compressor.py: 48 tests for grep output compression - test_integrations/langchain/: 122 tests for LangChain integration (agents, memory, retriever, streaming) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> |