From 42ebbc6cce02a0fd5e0a6e614348d47f4099649a Mon Sep 17 00:00:00 2001 From: Rod Boev Date: Wed, 8 Jul 2026 22:31:25 -0400 Subject: [PATCH] fix(evals): default unparseable judge scores below pass threshold (#1892) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Description `_parse_judge_response` in `headroom/evals/memory/judge.py` defaulted the score to `3.0` whenever it couldn't find a parseable `Score:` line in the judge's raw text. `before_after.py`'s `GroundTruthEvaluator` treats `judge_score >= 3.0` as "contains ground truth" (`contains_gt = judge_score >= 3.0`). Because `3.0` is exactly the pass threshold, any judge response the parser couldn't understand (malformed output, missing `Score:` line, a refusal, truncated text, etc.) silently counted as a pass instead of surfacing as a scoring failure, biasing BFCL/ground-truth eval accuracy upward with no visibility into how often it happened. The fix tracks whether a real score was actually parsed out of the response. If nothing parseable was found, the score now defaults to `0.0` (a hard fail, below the `>= 3.0` threshold) and a `logger.warning` is emitted with the raw judge text so the failure is visible instead of silent. Refs #1890. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/evals/memory/judge.py`: `_parse_judge_response` now tracks whether a `Score:` line was successfully parsed; on failure it defaults to `0.0` instead of `3.0` and logs a warning with the raw response text. - `tests/test_memory_eval.py`: added `TestJudge.test_parse_judge_response_unparseable_defaults_to_failing_score`, asserting an unparseable response scores below the `3.0` pass threshold. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_memory_eval.py -k judge`) - [x] Linting passes (`uv run ruff check headroom/evals/memory/judge.py headroom/evals/runners/before_after.py tests/test_memory_eval.py && uv run ruff format --check headroom/evals/memory/judge.py headroom/evals/runners/before_after.py tests/test_memory_eval.py`) - [ ] Type checking passes (`uv run mypy headroom`) — not run; not part of this repo's local validation loop for this change - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text tests\test_memory_eval.py ....... [ 77%] tests\test_verbosity_learn.py .. [100%] 9 passed $ uv run ruff check headroom/evals/memory/judge.py headroom/evals/runners/before_after.py tests/test_memory_eval.py && uv run ruff format --check headroom/evals/memory/judge.py headroom/evals/runners/before_after.py tests/test_memory_eval.py 3 files already formatted ``` ## Real Behavior Proof - Environment: Windows 11, Python (uv-managed venv), no LLM provider calls needed — `_parse_judge_response` is a pure text-parsing function. - Exact command / steps: checked out the pre-fix version of `_parse_judge_response` (default `score = 3.0`) and ran the new regression test against an unparseable response (`"The model's response looks reasonable overall."`, no `Score:` line). Confirmed it failed with `assert 3.0 < 3.0`. Restored the fix and reran — passes, with `score == 0.0`. - Observed result: pre-fix, an unparseable judge response scored `3.0` and would have passed `contains_gt = judge_score >= 3.0` in `before_after.py`. Post-fix, the same input scores `0.0`, fails the threshold, and logs a warning naming the raw response text. - Not tested: the live `create_openai_judge`/`create_anthropic_judge`/`create_litellm_judge` call paths (require provider API keys) and the end-to-end `GroundTruthEvaluator.evaluate` flow in `before_after.py` — only the pure parsing function and its documented contract with the `>= 3.0` threshold were exercised. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG.md is intentionally left untouched — this repo's release pipeline generates it from conventional commits. - No user-facing docs describe the parse-failure default, so no documentation changes were needed. - Kept the change minimal and localized to the parsing function; didn't touch `before_after.py`'s threshold or comments since its `>= 3.0` semantics for successfully-parsed scores are unchanged and correct. --------- Co-authored-by: JerrettDavis --- headroom/evals/memory/judge.py | 14 +++++++++++++- tests/test_memory_eval.py | 13 +++++++++++++ 2 files changed, 26 insertions(+), 1 deletion(-) diff --git a/headroom/evals/memory/judge.py b/headroom/evals/memory/judge.py index be468e844..4e711d352 100644 --- a/headroom/evals/memory/judge.py +++ b/headroom/evals/memory/judge.py @@ -201,7 +201,8 @@ def _parse_judge_response(text: str) -> tuple[float, str]: Tuple of (score, reasoning). """ reasoning = "" - score = 3.0 # Default to middle score if parsing fails + score: float | None = None + parsed = False lines = text.strip().split("\n") @@ -222,9 +223,20 @@ def _parse_judge_response(text: str) -> tuple[float, str]: score = float(match.group(1)) # Clamp to valid range score = max(1.0, min(5.0, score)) + parsed = True except ValueError: logger.warning(f"Could not parse score from: {score_text}") + if not parsed: + # Default to a failing score so unparseable judge output doesn't + # silently pass downstream `judge_score >= 3.0` checks. + logger.warning( + f"Could not parse a score from judge response, defaulting to 0.0 (fail): {text!r}" + ) + score = 0.0 + + assert score is not None + # If no explicit reasoning found, use the whole text if not reasoning: reasoning = text.strip() diff --git a/tests/test_memory_eval.py b/tests/test_memory_eval.py index 0f57021da..d422cf1b0 100644 --- a/tests/test_memory_eval.py +++ b/tests/test_memory_eval.py @@ -180,6 +180,19 @@ Score: 3.5""" score, _ = _parse_judge_response(response) assert score == 1.0 + def test_parse_judge_response_unparseable_defaults_to_failing_score(self): + """Unparseable judge output must default below the pass threshold. + + Regression test for #1890: a missing/garbled "Score:" line used to + default to 3.0, which is exactly the `judge_score >= 3.0` pass + threshold in before_after.py, silently marking unparseable judge + responses as passing. + """ + response = "The model's response looks reasonable overall." + + score, _ = _parse_judge_response(response) + assert score < 3.0 + def test_simple_judge_exact_match(self): """Test simple judge with exact match.""" score, reasoning = simple_judge(