fix(evals): default unparseable judge scores below pass threshold (#1892)

## Description

`_parse_judge_response` in `headroom/evals/memory/judge.py` defaulted
the score to `3.0` whenever it couldn't find a parseable `Score:` line
in the judge's raw text. `before_after.py`'s `GroundTruthEvaluator`
treats `judge_score >= 3.0` as "contains ground truth" (`contains_gt =
judge_score >= 3.0`). Because `3.0` is exactly the pass threshold, any
judge response the parser couldn't understand (malformed output, missing
`Score:` line, a refusal, truncated text, etc.) silently counted as a
pass instead of surfacing as a scoring failure, biasing
BFCL/ground-truth eval accuracy upward with no visibility into how often
it happened.

The fix tracks whether a real score was actually parsed out of the
response. If nothing parseable was found, the score now defaults to
`0.0` (a hard fail, below the `>= 3.0` threshold) and a `logger.warning`
is emitted with the raw judge text so the failure is visible instead of
silent.

Refs #1890.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/evals/memory/judge.py`: `_parse_judge_response` now tracks
whether a `Score:` line was successfully parsed; on failure it defaults
to `0.0` instead of `3.0` and logs a warning with the raw response text.
- `tests/test_memory_eval.py`: added
`TestJudge.test_parse_judge_response_unparseable_defaults_to_failing_score`,
asserting an unparseable response scores below the `3.0` pass threshold.

## Testing

- [x] Unit tests pass (`uv run pytest tests/test_memory_eval.py -k
judge`)
- [x] Linting passes (`uv run ruff check headroom/evals/memory/judge.py
headroom/evals/runners/before_after.py tests/test_memory_eval.py && uv
run ruff format --check headroom/evals/memory/judge.py
headroom/evals/runners/before_after.py tests/test_memory_eval.py`)
- [ ] Type checking passes (`uv run mypy headroom`) — not run; not part
of this repo's local validation loop for this change
- [x] New tests added for new functionality when applicable
- [x] Manual testing performed

### Test Output

```text
tests\test_memory_eval.py .......                                                                                [ 77%]
tests\test_verbosity_learn.py ..                                                                                 [100%]
9 passed

$ uv run ruff check headroom/evals/memory/judge.py headroom/evals/runners/before_after.py tests/test_memory_eval.py && uv run ruff format --check headroom/evals/memory/judge.py headroom/evals/runners/before_after.py tests/test_memory_eval.py
3 files already formatted
```

## Real Behavior Proof

- Environment: Windows 11, Python (uv-managed venv), no LLM provider
calls needed — `_parse_judge_response` is a pure text-parsing function.
- Exact command / steps: checked out the pre-fix version of
`_parse_judge_response` (default `score = 3.0`) and ran the new
regression test against an unparseable response (`"The model's response
looks reasonable overall."`, no `Score:` line). Confirmed it failed with
`assert 3.0 < 3.0`. Restored the fix and reran — passes, with `score ==
0.0`.
- Observed result: pre-fix, an unparseable judge response scored `3.0`
and would have passed `contains_gt = judge_score >= 3.0` in
`before_after.py`. Post-fix, the same input scores `0.0`, fails the
threshold, and logs a warning naming the raw response text.
- Not tested: the live
`create_openai_judge`/`create_anthropic_judge`/`create_litellm_judge`
call paths (require provider API keys) and the end-to-end
`GroundTruthEvaluator.evaluate` flow in `before_after.py` — only the
pure parsing function and its documented contract with the `>= 3.0`
threshold were exercised.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

- CHANGELOG.md is intentionally left untouched — this repo's release
pipeline generates it from conventional commits.
- No user-facing docs describe the parse-failure default, so no
documentation changes were needed.
- Kept the change minimal and localized to the parsing function; didn't
touch `before_after.py`'s threshold or comments since its `>= 3.0`
semantics for successfully-parsed scores are unchanged and correct.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>

headroom/evals/memory/judge.py

@@ -201,7 +201,8 @@ def _parse_judge_response(text: str) -> tuple[float, str]:
         Tuple of (score, reasoning).
     """
     reasoning = ""
-    score = 3.0  # Default to middle score if parsing fails
+    score: float | None = None
+    parsed = False
 
     lines = text.strip().split("\n")
 
@@ -222,9 +223,20 @@ def _parse_judge_response(text: str) -> tuple[float, str]:
                     score = float(match.group(1))
                     # Clamp to valid range
                     score = max(1.0, min(5.0, score))
+                    parsed = True
             except ValueError:
                 logger.warning(f"Could not parse score from: {score_text}")
 
+    if not parsed:
+        # Default to a failing score so unparseable judge output doesn't
+        # silently pass downstream `judge_score >= 3.0` checks.
+        logger.warning(
+            f"Could not parse a score from judge response, defaulting to 0.0 (fail): {text!r}"
+        )
+        score = 0.0
+
+    assert score is not None
+
     # If no explicit reasoning found, use the whole text
     if not reasoning:
         reasoning = text.strip()

tests/test_memory_eval.py

@@ -180,6 +180,19 @@ Score: 3.5"""
         score, _ = _parse_judge_response(response)
         assert score == 1.0
 
+    def test_parse_judge_response_unparseable_defaults_to_failing_score(self):
+        """Unparseable judge output must default below the pass threshold.
+
+        Regression test for #1890: a missing/garbled "Score:" line used to
+        default to 3.0, which is exactly the `judge_score >= 3.0` pass
+        threshold in before_after.py, silently marking unparseable judge
+        responses as passing.
+        """
+        response = "The model's response looks reasonable overall."
+
+        score, _ = _parse_judge_response(response)
+        assert score < 3.0
+
     def test_simple_judge_exact_match(self):
         """Test simple judge with exact match."""
         score, reasoning = simple_judge(