## Description
`_parse_judge_response` in `headroom/evals/memory/judge.py` defaulted
the score to `3.0` whenever it couldn't find a parseable `Score:` line
in the judge's raw text. `before_after.py`'s `GroundTruthEvaluator`
treats `judge_score >= 3.0` as "contains ground truth" (`contains_gt =
judge_score >= 3.0`). Because `3.0` is exactly the pass threshold, any
judge response the parser couldn't understand (malformed output, missing
`Score:` line, a refusal, truncated text, etc.) silently counted as a
pass instead of surfacing as a scoring failure, biasing
BFCL/ground-truth eval accuracy upward with no visibility into how often
it happened.
The fix tracks whether a real score was actually parsed out of the
response. If nothing parseable was found, the score now defaults to
`0.0` (a hard fail, below the `>= 3.0` threshold) and a `logger.warning`
is emitted with the raw judge text so the failure is visible instead of
silent.
Refs #1890.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/evals/memory/judge.py`: `_parse_judge_response` now tracks
whether a `Score:` line was successfully parsed; on failure it defaults
to `0.0` instead of `3.0` and logs a warning with the raw response text.
- `tests/test_memory_eval.py`: added
`TestJudge.test_parse_judge_response_unparseable_defaults_to_failing_score`,
asserting an unparseable response scores below the `3.0` pass threshold.
## Testing
- [x] Unit tests pass (`uv run pytest tests/test_memory_eval.py -k
judge`)
- [x] Linting passes (`uv run ruff check headroom/evals/memory/judge.py
headroom/evals/runners/before_after.py tests/test_memory_eval.py && uv
run ruff format --check headroom/evals/memory/judge.py
headroom/evals/runners/before_after.py tests/test_memory_eval.py`)
- [ ] Type checking passes (`uv run mypy headroom`) — not run; not part
of this repo's local validation loop for this change
- [x] New tests added for new functionality when applicable
- [x] Manual testing performed
### Test Output
```text
tests\test_memory_eval.py ....... [ 77%]
tests\test_verbosity_learn.py .. [100%]
9 passed
$ uv run ruff check headroom/evals/memory/judge.py headroom/evals/runners/before_after.py tests/test_memory_eval.py && uv run ruff format --check headroom/evals/memory/judge.py headroom/evals/runners/before_after.py tests/test_memory_eval.py
3 files already formatted
```
## Real Behavior Proof
- Environment: Windows 11, Python (uv-managed venv), no LLM provider
calls needed — `_parse_judge_response` is a pure text-parsing function.
- Exact command / steps: checked out the pre-fix version of
`_parse_judge_response` (default `score = 3.0`) and ran the new
regression test against an unparseable response (`"The model's response
looks reasonable overall."`, no `Score:` line). Confirmed it failed with
`assert 3.0 < 3.0`. Restored the fix and reran — passes, with `score ==
0.0`.
- Observed result: pre-fix, an unparseable judge response scored `3.0`
and would have passed `contains_gt = judge_score >= 3.0` in
`before_after.py`. Post-fix, the same input scores `0.0`, fails the
threshold, and logs a warning naming the raw response text.
- Not tested: the live
`create_openai_judge`/`create_anthropic_judge`/`create_litellm_judge`
call paths (require provider API keys) and the end-to-end
`GroundTruthEvaluator.evaluate` flow in `before_after.py` — only the
pure parsing function and its documented contract with the `>= 3.0`
threshold were exercised.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
- CHANGELOG.md is intentionally left untouched — this repo's release
pipeline generates it from conventional commits.
- No user-facing docs describe the parse-failure default, so no
documentation changes were needed.
- Kept the change minimal and localized to the parsing function; didn't
touch `before_after.py`'s threshold or comments since its `>= 3.0`
semantics for successfully-parsed scores are unchanged and correct.
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Implement comprehensive memory system supporting:
- Local backend (SQLite + FTS5 + HNSW) for zero-dependency operation
- Mem0 backends (Neo4j + Qdrant) for production graph memory
- DirectMem0Adapter for optimized pre-extracted data (bypasses LLM)
- Memory extraction with facts, entities, and relationships
- Proxy integration with --memory flag for automatic memory injection
Key components:
- headroom/memory/backends/: LocalBackend, Mem0Backend, DirectMem0Adapter
- headroom/memory/system.py: MemorySystem with tool-based interface
- headroom/memory/extraction.py: Entity and relationship extraction
- headroom/proxy/memory_handler.py: Proxy integration layer
- headroom/prediction/feature_extractor.py: Content analysis features
Testing:
- 217 new memory system tests covering all backends
- LoCoMo evaluation framework for memory quality assessment
- Integration tests for proxy memory functionality
Also removes deprecated example files in favor of focused test coverage.
