mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
cf6367add4
## Description
`SemanticCache.put` can evict a perfectly good, unrelated entry when it
merely updates a key that is already cached.
The method runs its at-capacity eviction loop *before* it computes the
entry's key:
```python
self._cleanup_expired()
# Evict if at capacity
while len(self._cache) >= self.config.max_entries:
self._evict_oldest()
...
key = messages_hash or self._generate_key(query)
...
self._cache[key] = entry
```
So when the same key is stored again while the cache is full (a
duplicate store, or a retried request that produces the same
`messages_hash`), the loop fires because `len == max_entries`, evicts
the LRU-oldest *distinct* entry, and only then overwrites the existing
key in place. Writing to an already-present key does not grow the map,
so nothing needed to be evicted — but an unrelated live entry is now
gone, and the next `get` for it is a false miss.
Concretely, with `max_entries=2` and keys `[h1, h2]`, re-storing `h2`
evicts `h1`, leaving `[h2]` even though only two distinct keys were ever
stored.
The sibling `CompressionCache.store_compressed` gets this right: it
deletes the existing key first, inserts, and only then trims — so
re-storing a present key never drops an unrelated entry.
## Fix
Compute the key first, then run the eviction loop only while the key is
genuinely new:
```python
key = messages_hash or self._generate_key(query)
while key not in self._cache and len(self._cache) >= self.config.max_entries:
self._evict_oldest()
```
An in-place update of an existing key no longer evicts anything; adding
a new key still trims to make room exactly as before.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/cache/semantic.py`: move the cache-key computation above the
eviction loop and gate the loop on `key not in self._cache` so an
in-place update never evicts.
- `tests/test_cache/test_semantic.py`: add
`test_update_at_capacity_does_not_evict_unrelated_entry`.
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/cache/semantic.py tests/test_cache/test_semantic.py
All checks passed!
$ python -m py_compile headroom/cache/semantic.py tests/test_cache/test_semantic.py
OK
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and a full `pytest`
gets OOM-killed on this box, so I verified the eviction logic with a
dependency-free script that replicates the `OrderedDict` +
`_evict_oldest` (popitem last=False) behavior for the old vs new loop,
and left the full pytest to CI.
- Exact command / steps: with `max_entries=2`, store `h1` then `h2`,
then re-store the already-present `h2`, under both the old loop (evict
before key dedup) and the new loop (evict only when key is new).
- Observed result: old loop leaves `['h2']` and `get(h1)` returns `None`
(h1 wrongly evicted); new loop leaves `['h1', 'h2']` with `get(h1)`
intact and `h2` updated. The regression test asserts h1 survives and h2
reflects the update.
- Not tested: a live embedding-backed cache round-trip; full local
`pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" and "type checking" boxes are unchecked
because the full suite imports the ML stack, which I can't run in this
environment; the change is a localized reordering of two existing
statements plus a loop guard, verified by the standalone proof and the
new regression test for CI. This is a different defect from the earlier
messages-hash keying fix — that one was about which slot a request maps
to; this one is about eviction dropping a live entry on an in-place
update.
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|---|---|---|
| .. | ||
| __init__.py | ||
| test_anthropic.py | ||
| test_backends.py | ||
| test_base.py | ||
| test_client_integration.py | ||
| test_dynamic_detector.py | ||
| test_google.py | ||
| test_openai.py | ||
| test_prefix_tracker.py | ||
| test_registry.py | ||
| test_semantic.py | ||