mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-10 14:27:00 -04:00
f8915067f6
## Description The eval framework's `DATASET_REGISTRY` (`headroom/evals/datasets.py`) only had English datasets (squad, hotpotqa, longbench, …), so the LLM-in-the-loop `BeforeAfterRunner` could not be pointed at Chinese/Japanese/Korean. This registers `alexandrainst/multi-wiki-qa` as `multi_wiki_qa` — the only HF-loadable dataset with **uniform zh/ja/ko** extractive QA: SQuAD-style, paper-guaranteed **verbatim-span** answers over full Wikipedia articles. It makes multilingual compression eval first-class in the framework. Pairs with #1527 (which fixes the CJK-broken F1 tokenization + token estimation the framework's metrics use), so registered CJK data feeds into CJK-correct metrics. ## Type of Change - [x] New feature (non-breaking change that adds functionality) ## Changes Made - `headroom/evals/datasets.py`: add `load_multi_wiki_qa(n, lang)` — mirrors `load_longbench` exactly (the `_check_datasets_installed()` guard, `EvalCase` construction, `EvalSuite` return); reads the verified schema `answers["text"][0]` (a verbatim substring of `context`). - Register it in `DATASET_REGISTRY` under a new `rag_multilingual` category (same 4-key shape as every other entry). - `tests/test_evals_multilingual.py`: offline registration/shape tests (the live HF load is exercised in Real Behavior Proof, matching the other loaders). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`/`format`) - [x] Type checking passes (`mypy headroom/evals/datasets.py`) - [x] New tests added for new functionality - [x] Manual testing performed (see Real Behavior Proof) ### Test Output ```text $ .venv/bin/python -m pytest tests/test_evals_multilingual.py 2 passed $ ruff check / ruff format --check headroom/evals/datasets.py # clean $ mypy headroom/evals/datasets.py # clean ``` ## Real Behavior Proof - Environment: macOS (Darwin 25.3.0), Python in a uv venv with the `[evals]` extra (`datasets`), branch `feat/evals-multilingual-dataset` off `main`. - Exact command / steps: called the new loader against the live dataset — `load_multi_wiki_qa(n=3, lang="ja")`. - Observed result: it returned an `EvalSuite` named `multi_wiki_qa_ja` with 3 `EvalCase`s, each with non-empty `id`/`context`/`query`/`ground_truth`; the `ground_truth` is a **verbatim substring** of its `context` (the property the answer-retention eval relies on); contexts are full-article length (~2,796 chars). Sample answer: `'1988年10月'`. ```text suite: multi_wiki_qa_ja cases: 3 case fields ok: True ground_truth verbatim-substring of context: True ctx len: 2796 | answer: '1988年10月' ``` - Not tested: an end-to-end `BeforeAfterRunner` run against a live LLM (that costs API calls and is out of scope for the loader); zh-cn/ko configs (same schema, verified present via the dataset's splits). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation — N/A (internal eval tooling) - [x] My changes generate no new warnings - [x] I have added tests that prove the feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A: `headroom/evals/` is internal dev tooling, not user-facing runtime ## Additional Notes - **No new dependency.** `multi-wiki-qa` loads through the existing `[evals]` `datasets` extra (guarded by `_check_datasets_installed()`); the dataset is fetched at run time and **never vendored** into the repo. - **License:** `alexandrainst/multi-wiki-qa` is CC-BY-NC-SA-4.0 (non-commercial) — consistent with how the repo already references externally-licensed datasets (SQuAD/LongBench) by id without committing their data. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
29 lines
1.2 KiB
Python
29 lines
1.2 KiB
Python
"""multi-wiki-qa multilingual loader registration.
|
|
|
|
The eval framework's DATASET_REGISTRY only had English datasets, so the
|
|
LLM-in-the-loop runner could not be pointed at Chinese/Japanese/Korean. This
|
|
registers `alexandrainst/multi-wiki-qa` (verbatim-span answers over full
|
|
Wikipedia articles, uniform zh/ja/ko). The live HF load is exercised in the
|
|
PR's Real Behavior Proof, not here, to keep the test offline (mirrors the other
|
|
dataset loaders).
|
|
"""
|
|
|
|
from headroom.evals.datasets import DATASET_REGISTRY, load_multi_wiki_qa
|
|
|
|
|
|
def test_multi_wiki_qa_registered():
|
|
assert "multi_wiki_qa" in DATASET_REGISTRY
|
|
entry = DATASET_REGISTRY["multi_wiki_qa"]
|
|
assert entry["loader"] is load_multi_wiki_qa
|
|
assert entry["category"] == "rag_multilingual"
|
|
# same 4-key shape as every other registry entry
|
|
assert set(entry) == {"loader", "description", "category", "default_n"}
|
|
|
|
|
|
def test_multi_wiki_qa_default_lang_is_callable():
|
|
# signature is (n, lang) like the other loaders; default lang is a real config
|
|
import inspect
|
|
|
|
sig = inspect.signature(load_multi_wiki_qa)
|
|
assert list(sig.parameters) == ["n", "lang"]
|
|
assert sig.parameters["lang"].default in {"ja", "ko", "zh-cn"}
|