## Description
Follow-up to **#1187** (the offline fidelity gate). That gate is
hermetic and **structured-only** (JSON tool outputs via Rust
compressors) so it can block every PR with zero setup. This PR adds the
genuinely-uncovered piece: **prose answer-recall on a real dataset
(HotpotQA)** in the **model-allowed weekly job**, where compression
routes through Kompress (ModernBERT).
> **Stacked on #1187.** Until that merges, this PR's diff shows its
commit too; it reduces to just `c71cc0cb` once #1187 lands. Please
review/merge #1187 first.
Closes #
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- **`CompressionOnlyRunner.evaluate_dataset_recall(suite)`**: for each
QA case, compress the supporting `context` via the production routing
path (`ContentRouter`) and check the `ground_truth` answer survives
(`compute_information_recall`). Counts only **probeable** cases — answer
literally present in the context and non-trivial (skips `yes/no`,
too-short) — so the aggregate is meaningful rather than inflated by
un-measurable cases.
- **`.github/workflows/eval.yml`**: a non-blocking step in the existing
`weekly-suite` job (schedule/manual only) drives it with
`load_hotpotqa(n=50)`. Defensive: a dataset download or model failure
emits `:⚠️:` and `|| true`, never failing the job.
- **Hermetic unit test** (`tests/test_dataset_recall_runner.py`):
exercises the method with synthetic JSON-array contexts (SmartCrusher /
Rust — no model, no network), so it runs in the standard `[dev]` shard.
### Scope notes
- **Prose path only.** BFCL / tool-schema integrity is already covered
by the existing `evaluate_tool_schema_compaction` eval (which runs in
the PR smoke-test), so this targets the previously-uncovered prose
recall path. NQ is an easy further extension using the same method +
`load_natural_questions`.
- **Why weekly, not per-PR.** Real datasets need a network download +
the ModernBERT model. The `weekly-suite` job already installs `[all]`
and genuinely runs every Monday (verified: 5 consecutive successful
scheduled runs), so it's the correct home — keeping PR CI fast and
hermetic.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ HF_HUB_OFFLINE=1 python -m pytest tests/test_dataset_recall_runner.py -q
.. [100%]
2 passed in 0.20s
```
## Real Behavior Proof
- Environment: local checkout of `feat/weekly-dataset-recall`, `pip
install -e ".[dev]"`, `HF_HUB_OFFLINE=1` (proves the unit tests need no
model/network)
- Exact command / steps: `HF_HUB_OFFLINE=1 python -m pytest
tests/test_dataset_recall_runner.py -q` -> `6 passed in 0.36s`; coverage
JSON confirms the runner's per-case exception handler and both
`warm_kompress_model` outcomes are exercised
- Observed result: with a synthetic suite of 3 cases (one probeable
answer in an error row, one trivial `yes`, one absent answer),
`evaluate_dataset_recall` counts only the 1 probeable case (`passed=1`,
`accuracy_rate=1.0`, `benchmark="dataset_recall:synthetic"`); a
monkeypatched compressor crash records the error and counts the case
failed instead of aborting; the new weekly-suite YAML step parses via
`yaml.safe_load` and sits under the `schedule || workflow_dispatch`
guard
- Not tested: the live HotpotQA download + ModernBERT compression --
exercised only by the weekly job (or `workflow_dispatch`), by design
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
- CHANGELOG/version intentionally untouched: repo uses
**release-please**.
- The weekly job can be triggered on demand via **workflow_dispatch** to
see the HotpotQA recall numbers without waiting for Monday.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
Ashish