headroom/tests/test_memory
Gautam Sharma 0924755591
fix(memory): serialize MCP backend initialization (#2309)
## Description

The Memory MCP server previously assigned its backend before
asynchronous embedder and vector-index warm-up completed. A tool call
arriving
during the handshake could therefore receive a partially initialized
backend.

Backend initialization is now atomic and shared between concurrent
callers. The backend is published only after warm-up succeeds. Failed
candidates are closed and discarded so later calls can retry with a
fresh backend.

  ## Type of Change

  - [x] Bug fix
  - [ ] New feature
  - [ ] Breaking change
  - [ ] Documentation-only change
  - [ ] Refactoring

  ## Changes Made

- Keep the initializing backend local until warm-up completes
successfully.
- Share one initialization task between handshake and concurrent tool
calls.
  - Await the shared task before exposing the backend to tool handlers.
- Shield shared initialization from cancellation by an individual tool
caller.
  - Close failed or cancelled backend candidates.
  - Clear failed initialization state so subsequent calls can retry.
  - Retrieve and log background initialization failures.
- Add regression tests for handshake races, failure recovery, and
concurrent initialization.
  - Add an Unreleased changelog entry.

  ## Testing

  - [x] Added regression tests
  - [x] Focused test suite passes
  - [x] Ruff checks pass
  - [x] Mypy passes
  - [x] Changed files pass formatting checks
  - [ ] Entire repository test suite passes without baseline failures

  Commands and results:

- `uv run --extra dev --frozen pytest
tests/test_memory/test_mcp_server.py -q`
    - `12 passed`
  - `uv run --extra dev --frozen ruff check .`
    - Passed
- `uv run --extra dev --frozen ruff format --check
headroom/memory/mcp_server.py tests/test_memory/test_mcp_server.py`
    - Passed
  - `uv run --extra dev --frozen mypy headroom --ignore-missing-imports`
    - Success across 504 source files
  - `uv run --extra dev --frozen pytest -q`
    - `9363 passed, 565 skipped, 4 failed`
- The four failures are existing, unrelated failures outside the changed
code:
      - `test_l2_appends_transform_label`
      - `test_recovery_records_sockets_and_secures_both_backups`
      - `test_dashboard_uses_cached_stats_and_lazy_history_feed_polling`
      - `test_smart_crusher_log_fallback_runs_for_valid_json`

Repository-wide `ruff format --check .` also identifies pre-existing
formatting drift in the untouched
`headroom/proxy/handlers/anthropic.py`.

  ## Real Behavior Proof

  The regression tests exercise the affected lifecycle directly:

  1. Start backend initialization through the MCP handshake.
  2. Suspend warm-up before it completes.
  3. Issue a memory tool call and verify its handler is not invoked.
4. Release warm-up and verify the tool receives the initialized backend.
5. Force background initialization to fail and verify the candidate is
closed.
6. Issue another tool call and verify initialization retries with a
fresh backend.
7. Start two tool calls concurrently and verify only one backend is
constructed.

  Observed behavior:

  - Tool calls remain pending while handshake warm-up is incomplete.
  - A partially initialized backend never reaches a tool handler.
  - Failed candidates are closed and discarded.
  - A later tool call successfully retries initialization.
  - Concurrent calls share one initialization task and backend.

  Environment: macOS arm64, CPython 3.12.13.
  
Not tested: a live stdio MCP client using the real ONNX model and
database. The affected initialization lifecycle is covered with
  deterministic asynchronous regression tests.

  ## Review Readiness

  - [x] I have performed a self-review before requesting human review.
  - [x] This PR is ready for human review.

  ## Checklist

- [x] The implementation follows the repository’s existing style and
error-handling conventions.
- [x] Tests cover the reported race, concurrent initialization, and
failure recovery.
- [x] Failed initialization does not leave a partially published
backend.
  - [x] Failed backend candidates are closed before retry.
  - [x] No unrelated files or formatting changes are included.
  - [x] No temporary logging, debug code, or commented-out code remains.
  - [x] Public behavior changes are documented in the changelog.
- [x] The branch has been rebased from the intended base and is ready
for review.

  ## Additional Notes

The four full-suite failures listed above occur outside the changed
Memory MCP code and are unrelated to this PR. All tests covering the
  modified initialization lifecycle pass.

---------

Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
2026-07-16 14:38:53 -07:00
..
__init__.py Add persistent memory system with zero-latency inline extraction 2026-01-14 21:32:09 -08:00
conftest.py fix(ccr): skip Anthropic marker emission when tool injection is deferred (#1273) 2026-06-23 12:48:05 -05:00
test_budget.py Fix ruff lint errors in test files 2026-03-24 15:54:12 -07:00
test_core_operations.py fix(memory): remove a superseded memory from the search indexes (#2143) 2026-07-14 04:24:00 -04:00
test_easy.py test(memory): skip decorators on offline model misses (#2020) 2026-07-11 10:14:05 -05:00
test_embedder_mps_serialization.py fix(memory): cap local embedder CPU thread oversubscription (#198) (#1559) 2026-07-01 17:12:02 -05:00
test_embedder_thread_cap.py fix(memory): cap local embedder CPU thread oversubscription (#198) (#1559) 2026-07-01 17:12:02 -05:00
test_extraction.py Add hierarchical memory system with graph + vector storage 2026-01-26 21:58:47 -08:00
test_factory.py Add centralized ML model configuration 2026-02-01 23:47:42 -08:00
test_factory_embedder_cache.py fix(memory): key the embedder cache on ollama_base_url (#2109) 2026-07-13 10:54:16 -04:00
test_factory_external.py chore(memory): add EXTERNAL backend extension points 2026-04-20 16:42:10 -07:00
test_hierarchical.py fix(memory/sqlite): don't emit OFFSET without LIMIT in query (#2063) 2026-07-13 09:46:45 -04:00
test_hnsw_batch_capacity.py fix(memory): size HNSW index_batch resize off the id high-water mark (#2139) 2026-07-13 23:43:12 -04:00
test_learn_flag.py fix(traffic-learner): raise min-evidence default and make it configurable 2026-04-30 17:44:22 +09:00
test_local_backend_search.py fix(memory): filter inactive graph-expanded results (#2210) 2026-07-15 19:58:13 +00:00
test_mcp_server.py fix(memory): serialize MCP backend initialization (#2309) 2026-07-16 14:38:53 -07:00
test_qdrant_env.py feat(memory): resolve Qdrant connection from HEADROOM_QDRANT_* env vars (#31) 2026-04-24 22:16:16 -07:00
test_query_conditions.py fix(memory): apply turn_id scope filter even without agent_id (#2130) 2026-07-13 23:41:40 -04:00
test_skip_helpers.py fix(ccr): skip Anthropic marker emission when tool injection is deferred (#1273) 2026-06-23 12:48:05 -05:00
test_supersession_repair.py feat(memory): add explicit supersession repair (#2217) 2026-07-15 18:17:17 +00:00
test_traffic_learner.py feat: headroom wrap opencode / unwrap opencode CLI (#1105) 2026-06-22 11:07:12 -05:00
test_writers.py fix: harden learn path handling across platforms 2026-05-09 15:45:26 -07:00