2 commits

Author	SHA1	Message	Date
Rod Boev	42ebbc6cce	fix(evals): default unparseable judge scores below pass threshold (#1892) ... ## Description `_parse_judge_response` in `headroom/evals/memory/judge.py` defaulted the score to `3.0` whenever it couldn't find a parseable `Score:` line in the judge's raw text. `before_after.py`'s `GroundTruthEvaluator` treats `judge_score >= 3.0` as "contains ground truth" (`contains_gt = judge_score >= 3.0`). Because `3.0` is exactly the pass threshold, any judge response the parser couldn't understand (malformed output, missing `Score:` line, a refusal, truncated text, etc.) silently counted as a pass instead of surfacing as a scoring failure, biasing BFCL/ground-truth eval accuracy upward with no visibility into how often it happened. The fix tracks whether a real score was actually parsed out of the response. If nothing parseable was found, the score now defaults to `0.0` (a hard fail, below the `>= 3.0` threshold) and a `logger.warning` is emitted with the raw judge text so the failure is visible instead of silent. Refs #1890. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/evals/memory/judge.py`: `_parse_judge_response` now tracks whether a `Score:` line was successfully parsed; on failure it defaults to `0.0` instead of `3.0` and logs a warning with the raw response text. - `tests/test_memory_eval.py`: added `TestJudge.test_parse_judge_response_unparseable_defaults_to_failing_score`, asserting an unparseable response scores below the `3.0` pass threshold. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_memory_eval.py -k judge`) - [x] Linting passes (`uv run ruff check headroom/evals/memory/judge.py headroom/evals/runners/before_after.py tests/test_memory_eval.py && uv run ruff format --check headroom/evals/memory/judge.py headroom/evals/runners/before_after.py tests/test_memory_eval.py`) - [ ] Type checking passes (`uv run mypy headroom`) — not run; not part of this repo's local validation loop for this change - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text tests\test_memory_eval.py ....... [ 77%] tests\test_verbosity_learn.py .. [100%] 9 passed $ uv run ruff check headroom/evals/memory/judge.py headroom/evals/runners/before_after.py tests/test_memory_eval.py && uv run ruff format --check headroom/evals/memory/judge.py headroom/evals/runners/before_after.py tests/test_memory_eval.py 3 files already formatted ``` ## Real Behavior Proof - Environment: Windows 11, Python (uv-managed venv), no LLM provider calls needed — `_parse_judge_response` is a pure text-parsing function. - Exact command / steps: checked out the pre-fix version of `_parse_judge_response` (default `score = 3.0`) and ran the new regression test against an unparseable response (`"The model's response looks reasonable overall."`, no `Score:` line). Confirmed it failed with `assert 3.0 < 3.0`. Restored the fix and reran — passes, with `score == 0.0`. - Observed result: pre-fix, an unparseable judge response scored `3.0` and would have passed `contains_gt = judge_score >= 3.0` in `before_after.py`. Post-fix, the same input scores `0.0`, fails the threshold, and logs a warning naming the raw response text. - Not tested: the live `create_openai_judge`/`create_anthropic_judge`/`create_litellm_judge` call paths (require provider API keys) and the end-to-end `GroundTruthEvaluator.evaluate` flow in `before_after.py` — only the pure parsing function and its documented contract with the `>= 3.0` threshold were exercised. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG.md is intentionally left untouched — this repo's release pipeline generates it from conventional commits. - No user-facing docs describe the parse-failure default, so no documentation changes were needed. - Kept the change minimal and localized to the parsing function; didn't touch `before_after.py`'s threshold or comments since its `>= 3.0` semantics for successfully-parsed scores are unchanged and correct. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com>	2026-07-08 19:31:25 -07:00
chopratejas	da74341858	Add hierarchical memory system with graph + vector storage ... Implement comprehensive memory system supporting: - Local backend (SQLite + FTS5 + HNSW) for zero-dependency operation - Mem0 backends (Neo4j + Qdrant) for production graph memory - DirectMem0Adapter for optimized pre-extracted data (bypasses LLM) - Memory extraction with facts, entities, and relationships - Proxy integration with --memory flag for automatic memory injection Key components: - headroom/memory/backends/: LocalBackend, Mem0Backend, DirectMem0Adapter - headroom/memory/system.py: MemorySystem with tool-based interface - headroom/memory/extraction.py: Entity and relationship extraction - headroom/proxy/memory_handler.py: Proxy integration layer - headroom/prediction/feature_extractor.py: Content analysis features Testing: - 217 new memory system tests covering all backends - LoCoMo evaluation framework for memory quality assessment - Integration tests for proxy memory functionality Also removes deprecated example files in favor of focused test coverage.	2026-01-26 21:58:47 -08:00