1 commit

Author	SHA1	Message	Date
Zhenjia ZHOU	f8915067f6	feat(evals): register multilingual multi-wiki-qa (zh/ja/ko) dataset (#1530) ... ## Description The eval framework's `DATASET_REGISTRY` (`headroom/evals/datasets.py`) only had English datasets (squad, hotpotqa, longbench, …), so the LLM-in-the-loop `BeforeAfterRunner` could not be pointed at Chinese/Japanese/Korean. This registers `alexandrainst/multi-wiki-qa` as `multi_wiki_qa` — the only HF-loadable dataset with **uniform zh/ja/ko** extractive QA: SQuAD-style, paper-guaranteed **verbatim-span** answers over full Wikipedia articles. It makes multilingual compression eval first-class in the framework. Pairs with #1527 (which fixes the CJK-broken F1 tokenization + token estimation the framework's metrics use), so registered CJK data feeds into CJK-correct metrics. ## Type of Change - [x] New feature (non-breaking change that adds functionality) ## Changes Made - `headroom/evals/datasets.py`: add `load_multi_wiki_qa(n, lang)` — mirrors `load_longbench` exactly (the `_check_datasets_installed()` guard, `EvalCase` construction, `EvalSuite` return); reads the verified schema `answers["text"][0]` (a verbatim substring of `context`). - Register it in `DATASET_REGISTRY` under a new `rag_multilingual` category (same 4-key shape as every other entry). - `tests/test_evals_multilingual.py`: offline registration/shape tests (the live HF load is exercised in Real Behavior Proof, matching the other loaders). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`/`format`) - [x] Type checking passes (`mypy headroom/evals/datasets.py`) - [x] New tests added for new functionality - [x] Manual testing performed (see Real Behavior Proof) ### Test Output ```text $ .venv/bin/python -m pytest tests/test_evals_multilingual.py 2 passed $ ruff check / ruff format --check headroom/evals/datasets.py # clean $ mypy headroom/evals/datasets.py # clean ``` ## Real Behavior Proof - Environment: macOS (Darwin 25.3.0), Python in a uv venv with the `[evals]` extra (`datasets`), branch `feat/evals-multilingual-dataset` off `main`. - Exact command / steps: called the new loader against the live dataset — `load_multi_wiki_qa(n=3, lang="ja")`. - Observed result: it returned an `EvalSuite` named `multi_wiki_qa_ja` with 3 `EvalCase`s, each with non-empty `id`/`context`/`query`/`ground_truth`; the `ground_truth` is a **verbatim substring** of its `context` (the property the answer-retention eval relies on); contexts are full-article length (~2,796 chars). Sample answer: `'1988年10月'`. ```text suite: multi_wiki_qa_ja cases: 3 case fields ok: True ground_truth verbatim-substring of context: True ctx len: 2796 | answer: '1988年10月' ``` - Not tested: an end-to-end `BeforeAfterRunner` run against a live LLM (that costs API calls and is out of scope for the loader); zh-cn/ko configs (same schema, verified present via the dataset's splits). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation — N/A (internal eval tooling) - [x] My changes generate no new warnings - [x] I have added tests that prove the feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A: `headroom/evals/` is internal dev tooling, not user-facing runtime ## Additional Notes - **No new dependency.** `multi-wiki-qa` loads through the existing `[evals]` `datasets` extra (guarded by `_check_datasets_installed()`); the dataset is fetched at run time and **never vendored** into the repo. - **License:** `alexandrainst/multi-wiki-qa` is CC-BY-NC-SA-4.0 (non-commercial) — consistent with how the repo already references externally-licensed datasets (SQuAD/LongBench) by id without committing their data. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-14 13:25:10 -04:00