mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
f8eaaeb26a
## Description
The semantic tier of the dynamic-content detector compares an
unnormalized dot product against a cosine threshold, so it flags almost
everything as dynamic and strips the static content it is supposed to
protect.
`SemanticDetector` pre-computes exemplar embeddings and, per sentence,
scores similarity with `np.dot` and compares to `semantic_threshold`:
```python
self._exemplar_embeddings = self._model.encode(self.DYNAMIC_EXEMPLARS, convert_to_numpy=True)
...
sentence_embeddings = self._model.encode(sentence_texts, convert_to_numpy=True)
similarities = np.dot(sentence_embeddings, self._exemplar_embeddings.T)
...
if max_sim < self.config.semantic_threshold: # semantic_threshold defaults to 0.7
continue
```
`sentence_transformers.encode(..., convert_to_numpy=True)` does **not**
normalize by default. So `np.dot` here is an inner product whose
magnitude scales with the embedding norms (typically ~5-15 for MiniLM),
not a cosine similarity in [0, 1]. Comparing that against
`semantic_threshold=0.7` (documented and configured as a 0-1 similarity)
is a scale mismatch: nearly every sentence clears the threshold, so the
semantic tier classifies almost all text as dynamic, moves it into
`dynamic_content`, and empties `static_content` — busting the very cache
the detector exists to protect.
A standalone repro: an unrelated sentence with a true cosine of ~0.1 to
an exemplar produces a raw dot of ~9.1 (well over 0.7); normalized, it
correctly scores ~0.09 and stays static.
The correct behavior is used by the in-repo siblings:
`prediction/feature_extractor.py` passes `normalize_embeddings=True`,
and `memory/adapters/embedders.py` L2-normalizes before dot-product
similarity. This detector did neither.
## Fix
Pass `normalize_embeddings=True` to both `encode` calls (exemplars in
`__init__` and sentences in `detect`). Both sides of the dot product are
then unit vectors, so `np.dot` is a true cosine similarity in [-1, 1],
comparable to `semantic_threshold`.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/cache/dynamic_detector.py`: add `normalize_embeddings=True`
to the exemplar encode (`__init__`) and the sentence encode (`detect`),
with comments explaining the cosine requirement.
- `tests/test_cache/test_dynamic_detector.py`: add
`TestSemanticDetectorNormalization` — a recording fake model asserts
both encode calls pass `normalize_embeddings=True` (via `object.__new__`
for `detect`, and a monkeypatched registry for `__init__`). No model
download needed.
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [x] Unit tests pass (`uv run --extra dev pytest
tests/test_cache/test_dynamic_detector.py::TestSemanticDetectorNormalization
-q`)
- [x] Linting passes (`uvx ruff@0.15.17 check
headroom/cache/dynamic_detector.py
tests/test_cache/test_dynamic_detector.py headroom/memory/factory.py`)
- [x] Type checking passes (`uvx mypy==1.20.2
headroom/memory/factory.py`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/cache/dynamic_detector.py tests/test_cache/test_dynamic_detector.py
All checks passed!
$ python -m py_compile headroom/cache/dynamic_detector.py tests/test_cache/test_dynamic_detector.py
OK
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`, numpy.
Importing `headroom` pulls in the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I verified the scale mismatch
with a dependency-free numpy script (no sentence-transformers), and left
the full pytest to CI.
- Exact command / steps: built a MiniLM-dimension exemplar direction and
a sentence direction with a true cosine of ~0.1 (genuinely not dynamic),
gave them realistic un-normalized magnitudes (~9 and ~11), and computed
the old `np.dot` of the raw vectors versus the new `np.dot` of the
normalized vectors, against the 0.7 threshold.
- Observed result: old raw dot ~9.1 (far above 0.7 -> the unrelated
sentence is wrongly flagged dynamic); new cosine ~0.09 (below 0.7 ->
correctly kept static), and always within [-1, 1]. The new tests assert
both encode calls pass `normalize_embeddings=True`.
- Not tested: a real sentence-transformers model end to end; full local
`pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
Merged current `main` to pick up the repository-wide mypy cache-key
annotation fix, then verified the focused regression locally. the change
adds one keyword argument to two `encode` calls, verified by the numpy
proof and the new fake-model tests for CI. The fix brings this detector
in line with the two sibling call sites that already normalize.
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|---|---|---|
| .. | ||
| __init__.py | ||
| test_anthropic.py | ||
| test_backends.py | ||
| test_base.py | ||
| test_client_integration.py | ||
| test_dynamic_detector.py | ||
| test_google.py | ||
| test_openai.py | ||
| test_prefix_tracker.py | ||
| test_registry.py | ||
| test_semantic.py | ||