## Description
`SQLiteMemoryStore.query` (`headroom/memory/adapters/sqlite.py`) builds
pagination like this:
```python
if filter.limit is not None:
query += " LIMIT ?"
params.append(filter.limit)
if filter.offset > 0:
query += " OFFSET ?"
params.append(filter.offset)
```
SQLite's grammar allows `OFFSET` **only** as part of a `LIMIT` clause.
So a `MemoryFilter` with
an offset but no limit produces `... ORDER BY created_at DESC OFFSET ?`,
which SQLite rejects:
```
sqlite3.OperationalError: near "OFFSET": syntax error
```
Both `offset` and `limit` are public `MemoryFilter` fields (`ports.py`:
`limit` defaults to
`None`, `offset` to `0`), so any caller paginating with an offset but no
explicit limit crashes.
Closes: no issue filed — found while auditing the memory store query
builder.
## Fix
When an offset is present without a limit, emit SQLite's unbounded
`LIMIT -1` so `OFFSET` is
grammatically valid:
```python
if filter.offset > 0:
if filter.limit is None:
query += " LIMIT -1"
query += " OFFSET ?"
params.append(filter.offset)
```
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/memory/adapters/sqlite.py`: emit `LIMIT -1` when paginating
with an offset but no limit.
- `tests/test_memory/test_hierarchical.py`: add
`test_query_offset_without_limit` (offset skips rows; offset past the
end returns `[]`; no crash).
## Testing
- [x] New regression test added
(`tests/test_memory/test_hierarchical.py`)
- [x] Linting/formatting clean — run with the CI-pinned `ruff==0.15.17`
- [ ] Full `pytest` deferred to CI (local-OOM reason below).
```text
$ uvx ruff@0.15.17 check headroom/memory/adapters/sqlite.py tests/test_memory/test_hierarchical.py
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.10, headroom from this branch.
Importing `headroom` pulls in the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I reproduced the exact SQL
against a real stdlib `sqlite3` in-memory DB (the store's query is pure
SQL) and left the full pytest to CI.
- Exact command / steps: built the same `ORDER BY ... [LIMIT] [OFFSET]`
query for `offset=2, limit=None` with the old and new logic and ran it
against a 5-row table.
- Observed result: the old builder raises the exact `OperationalError`;
the new builder skips `offset` rows and returns the rest, and
`LIMIT`-only / `LIMIT`+`OFFSET` still work:
```text
OLD offset-no-limit: OperationalError -> near "OFFSET": syntax error
NEW offset-no-limit: rows=[2, 1, 0]
SQLITE OFFSET-WITHOUT-LIMIT FIX VERIFIED (old crashes; new paginates)
```
- Not tested: the full `HierarchicalMemory` stack (needs the heavy
embedder). The new test drives `SQLiteMemoryStore.query` directly with
`save_batch` + `MemoryFilter`. Full local `pytest` deferred to CI (OOM,
per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — ran
lint + a standalone SQLite check; full pytest deferred to CI (local OOM,
disclosed above)
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- One-line grammar fix plus a test; no new dependencies.
- @JerrettDavis tagging you — a paginating caller (offset, no limit)
currently crashes the memory store query; quick one. Thanks!
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
## Description
Track successful native MCP `memory_search` retrievals in persistent
memory metadata. Returned memories now increment `access_count` and
update `last_accessed`, so MCP usage contributes to memory budget and
retention signals.
Closes#2061
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Add an atomic, deduplicated `MemoryStore.record_access` operation.
- Expose access recording through `HierarchicalMemory` and
`LocalBackend`, invalidating stale cache entries.
- Record only the final active memories actually returned by MCP search.
- Fail open if usage metadata cannot be written.
- Add SQLite and MCP regression coverage.
## Testing
- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
pytest tests/test_memory --ignore=tests/test_memory/test_learn_flag.py -q
368 passed, 142 skipped, 158 warnings in 3.28s
pytest tests/test_memory/test_hierarchical.py tests/test_memory/test_mcp_server.py tests/test_memory/test_factory.py -q
40 passed, 53 skipped, 158 warnings in 0.75s
```
## Real Behavior Proof
- Environment: macOS, Python 3.13, SQLite memory store.
- Exact command / steps: save two memories; call `record_access` with
duplicate IDs plus a missing ID; read both rows; call it again for one
row.
- Observed result: each existing memory increments once per call,
duplicates do not double-count, missing IDs are ignored, and
`last_accessed` advances to the supplied timestamp.
- Not tested: the full repository suite and
`tests/test_memory/test_learn_flag.py`; the source checkout does not
include the compiled `headroom._core` Rust extension. Ruff and mypy were
not available in the local development environment; CI remains
authoritative for those checks.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
Documentation and changelog changes are not included because this is an
internal retrieval-metadata correction with no user-facing configuration
change. Access tracking is intentionally fail-open so a metadata write
failure cannot suppress a valid memory search result.
---------
Co-authored-by: xuyidiao <xuyidiao@bytedance.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Make the ONNX + sqlite-vec memory path truly batched.
Batch ONNX embed_batch calls, batch sqlite-vec index/remove work under a single cached connection, and update MCP warm-up to use batch embed/save/index flows.
Add focused regression tests for ONNX batching, sqlite-vec single-connection batch behavior, and MCP warm-up batching.
Skip the MCP-specific test when optional MCP dependencies are not installed.
Refs #240
Enable ReadLifecycle by default so stale/superseded Read outputs are
automatically replaced with compact CCR markers — these are provably safe
to compress (file was edited or re-read).
Replace static compression thresholds with adaptive parameters that scale
with conversation length and context pressure:
- protect_recent_reads_fraction: protects the most-recent 50% of messages
from Read exclusion. Old Reads beyond this window become compressible,
preventing the "28 excluded Read/Glob, 0 tokens saved" problem.
- min_ratio_relaxed / min_ratio_aggressive: compression acceptance
threshold interpolates linearly with context pressure (tokens / model
limit). Low pressure → 0.85 (picky), high pressure → 0.65 (accept
anything helpful). Eliminates the fixed 0.9 gate that was rejecting
20+ messages per request.
Also adds --no-read-lifecycle CLI flag, and fixes a missing
pytest.importorskip guard for sentence-transformers in memory tests.
DiffCompressor:
- Parse unified diff format and compress by reducing context lines
- Preserve file headers and all +/- change lines
- Score hunks by relevance (error keywords, query matches)
- Add summary line: [N files, +X -Y lines]
- Expected 30-50% savings on typical git diffs
- Wire into content router for CompressionStrategy.DIFF
- 30 tests covering parsing, compression, edge cases
hnswlib SIGILL fix:
- Move hnswlib import from module level to lazy loading
- hnswlib crashes with SIGILL (Illegal Instruction) on CPUs
without AVX support, before Python can catch the error
- Now imports only when HNSWVectorIndex is actually used
- HNSW_AVAILABLE is checked lazily via __getattr__
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implement comprehensive memory system supporting:
- Local backend (SQLite + FTS5 + HNSW) for zero-dependency operation
- Mem0 backends (Neo4j + Qdrant) for production graph memory
- DirectMem0Adapter for optimized pre-extracted data (bypasses LLM)
- Memory extraction with facts, entities, and relationships
- Proxy integration with --memory flag for automatic memory injection
Key components:
- headroom/memory/backends/: LocalBackend, Mem0Backend, DirectMem0Adapter
- headroom/memory/system.py: MemorySystem with tool-based interface
- headroom/memory/extraction.py: Entity and relationship extraction
- headroom/proxy/memory_handler.py: Proxy integration layer
- headroom/prediction/feature_extractor.py: Content analysis features
Testing:
- 217 new memory system tests covering all backends
- LoCoMo evaluation framework for memory quality assessment
- Integration tests for proxy memory functionality
Also removes deprecated example files in favor of focused test coverage.
- Wrap hnswlib import in try/except in hnsw.py
- Export HNSW_AVAILABLE flag from adapters module
- Add helpful error message when HNSWVectorIndex is used without hnswlib
- Add @pytest.mark.skipif to HNSW test classes
hnswlib requires C++ compilation and may not be available on all
platforms or Python versions in CI environments.
