From 23d73ae0708b0f03738ef1de41ece1dcb848db98 Mon Sep 17 00:00:00 2001 From: Ashish Date: Mon, 22 Jun 2026 20:53:59 -0700 Subject: [PATCH] test(evals): add offline fidelity regression gate (recall-based, zero-model) (#1187) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Description Headroom's lossy compression drops rows/lines using statistical heuristics but **never checks that meaning survived** — a dropped `OOM killed worker 3` line can silently flip a model's answer with no signal that compression caused it. The repo already ships a quality-metric toolkit (`headroom/evals/metrics.py`) and a `weekly-suite` eval job, but neither gates the compression path on a PR. This adds a **per-PR fidelity regression gate**: compress vendored golden tool-outputs through SmartCrusher's lossy path and assert the evidence that answers each case's question survives. It is the first of a planned trio (this is the "offline gate" half of the fidelity work); query-aware retention and a hard token-budget API are documented follow-ups. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Blocking gate** (`tests/test_compression_fidelity_regression.py`): compresses each golden case via `smart_crush_tool_output(..., with_compaction=False)` and scores with `compute_information_recall`. Two assertions: - **Per-case critical recall == 1.0** — every `answer_evidence` string (placed in error/anomaly rows, the documented SmartCrusher retention guarantee) must survive. - **Aggregate recall ≥ committed baseline** (`baseline.json`, tol 0.02) — catches softer regressions. - **Vendored fixtures** (`tests/fixtures/fidelity_golden/`): deterministic `_generate.py` emits `cases.json` (4 cases: OOM crash, payment exception, latency anomaly, CI failure) + `baseline.json`. - **Non-blocking weekly report** (`.github/workflows/eval.yml`): one step in the existing `weekly-suite` job (schedule/manual only) reuses the existing `evaluate_information_retention` runner for a recall report on the production routing path. - **Pure reuse**: scoring (`evals/metrics.py`), compressor (`smart_crush_tool_output`), and the weekly runner (`evaluate_information_retention`) all already existed. ### Design notes - **Zero new CI setup.** The blocking gate runs in the existing `[dev]` test shard — no new workflow, no new deps, **no model, no network, no secrets** (verified under `HF_HUB_OFFLINE=1`). It deliberately uses small hand-made structured fixtures rather than the repo's HuggingFace dataset loaders, which would require a network download + ModernBERT and don't belong in a fast PR gate. - **Scope:** structured JSON tool-output (the dominant, deterministic, model-free case). Real-dataset (HotpotQA/BFCL) recall — which needs `[all]` + a local model — is a **documented follow-up PR**, and the `weekly-suite` job (which genuinely runs every Monday) is its natural home. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ HF_HUB_OFFLINE=1 python -m pytest tests/test_compression_fidelity_regression.py -v tests/test_compression_fidelity_regression.py::test_critical_evidence_survives_compression[logs_oom] PASSED tests/test_compression_fidelity_regression.py::test_critical_evidence_survives_compression[payment_exception] PASSED tests/test_compression_fidelity_regression.py::test_critical_evidence_survives_compression[latency_anomaly] PASSED tests/test_compression_fidelity_regression.py::test_critical_evidence_survives_compression[ci_test_failures] PASSED tests/test_compression_fidelity_regression.py::test_aggregate_recall_not_regressed PASSED ============================== 5 passed in 0.18s =============================== ``` ## Real Behavior Proof - **Environment:** local checkout of `feat/fidelity-regression-gate`, `pip install -e ".[dev]"`, `HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1` (proves no model/network). - **Exact command / steps:** `HF_HUB_OFFLINE=1 python -m pytest tests/test_compression_fidelity_regression.py -q` → `5 passed in 0.14s`. - **Negative control (proves the gate has teeth):** compressing `logs_oom` and probing for a benign row that compression legitimately drops returns `recall = 0.00, lost = ['heartbeat ping 25']` — i.e. the gate fires when critical evidence is dropped, so it is not trivially green. - **Weekly (non-blocking) step verified locally:** ```text Information retention: 50/50 cases >=0.9 recall, avg compression 65.7% ``` - **Not tested:** real-dataset (HotpotQA/BFCL) recall and prose/ModernBERT compression — intentionally deferred to a follow-up PR targeting the weekly job. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG/version intentionally untouched: repo uses **release-please**. - **Follow-up PR (planned):** wire the real HotpotQA/BFCL loaders (`headroom/evals/datasets.py`) into the `weekly-suite` job for genuine benchmark-scale recall coverage (model-allowed, non-blocking). Further follow-ups from the same design: a live per-request fidelity guardrail, query-aware lossy retention, and a hard `target_tokens` budget API. Co-authored-by: Claude Opus 4.8 --- .github/workflows/eval.yml | 18 + tests/fixtures/fidelity_golden/_generate.py | 141 ++++ tests/fixtures/fidelity_golden/baseline.json | 5 + tests/fixtures/fidelity_golden/cases.json | 796 ++++++++++++++++++ tests/test_compression_fidelity_regression.py | 72 ++ 5 files changed, 1032 insertions(+) create mode 100644 tests/fixtures/fidelity_golden/_generate.py create mode 100644 tests/fixtures/fidelity_golden/baseline.json create mode 100644 tests/fixtures/fidelity_golden/cases.json create mode 100644 tests/test_compression_fidelity_regression.py diff --git a/.github/workflows/eval.yml b/.github/workflows/eval.yml index 3037d6c2b..f1c031ca6 100644 --- a/.github/workflows/eval.yml +++ b/.github/workflows/eval.yml @@ -126,6 +126,24 @@ jobs: env: OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} + + # Recall-based fidelity report on the production routing path. Zero cost + # (synthetic structured cases -> Rust compressors; no model, no API, no + # secrets). Non-blocking: surfaces recall trends weekly without gating. + # The blocking per-PR fidelity gate lives in + # tests/test_compression_fidelity_regression.py (runs in the [dev] shard). + - name: Information-retention recall report (zero cost, non-blocking) + run: | + python -c " + from headroom.evals.runners.compression_only import CompressionOnlyRunner + runner = CompressionOnlyRunner() + cases = runner.generate_info_retention_cases(n=50) + result = runner.evaluate_information_retention(cases) + print(f'Information retention: {result.passed_cases}/{result.total_cases} cases >=0.9 recall, avg compression {result.avg_compression_ratio:.1%}') + if not result.passed: + print(f'::warning title=Fidelity recall::{result.failed_cases} case(s) fell below 0.9 recall: {result.errors[:3]}') + " + - name: Upload results if: always() uses: actions/upload-artifact@v7 diff --git a/tests/fixtures/fidelity_golden/_generate.py b/tests/fixtures/fidelity_golden/_generate.py new file mode 100644 index 000000000..9ec25ef82 --- /dev/null +++ b/tests/fixtures/fidelity_golden/_generate.py @@ -0,0 +1,141 @@ +"""Deterministic generator for the fidelity-regression golden fixtures. + +Run to (re)create ``cases.json``:: + + python tests/fixtures/fidelity_golden/_generate.py + +Each case is a JSON tool output (a list of records) plus the question it should +let a model answer and the evidence that must survive compression. Critical +``answer_evidence`` is always placed in an *error* or *anomaly* row, because +those are the rows SmartCrusher formally guarantees to retain +(see ``tests/test_quality_retention.py``) — so the hard ``recall == 1.0`` gate +holds on a healthy tree. ``supporting_facts`` are softer details that may live +in droppable rows; they feed the aggregate-recall baseline, not the hard gate. + +No models, no network: content is built from fixed strings only. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +OUT = Path(__file__).parent / "cases.json" + + +def _filler(n: int, msg: str) -> list[dict]: + """n benign, near-identical INFO rows (droppable noise).""" + return [{"seq": i, "level": "INFO", "status": "ok", "msg": f"{msg} {i}"} for i in range(n)] + + +def _cases() -> list[dict]: + cases: list[dict] = [] + + # 1. OOM crash buried in benign logs — error-keyword retention. + rows = _filler(28, "heartbeat ping") + rows.insert( + 17, + { + "seq": 17, + "level": "ERROR", + "status": "failed", + "msg": "OOM killed worker 3 exceeded memory limit 8GiB", + }, + ) + rows.insert( + 9, {"seq": 9, "level": "INFO", "status": "ok", "msg": "checkpoint saved at step 4500"} + ) + cases.append( + { + "id": "logs_oom", + "question": "Why did the job fail?", + "content_type": "json_array", + "compress": {"with_compaction": False, "max_items_after_crush": 10}, + "answer_evidence": ["OOM killed worker 3", "exceeded memory limit"], + "supporting_facts": ["checkpoint saved at step 4500"], + "content": rows, + } + ) + + # 2. Payment exception — error-keyword retention. + rows = _filler(30, "GET /healthz 200") + rows.insert( + 21, + { + "seq": 21, + "level": "ERROR", + "status": "failed", + "msg": "exception: NullPointerException at PaymentService.charge line 88", + }, + ) + cases.append( + { + "id": "payment_exception", + "question": "Which service threw an exception and where?", + "content_type": "json_array", + "compress": {"with_compaction": False, "max_items_after_crush": 8}, + "answer_evidence": ["NullPointerException", "PaymentService.charge line 88"], + "supporting_facts": [], + "content": rows, + } + ) + + # 3. Latency anomaly — anomaly (>2 sigma) retention. Values cluster near 100ms, + # one row spikes to 99999ms. + rows = [ + {"seq": i, "region": "us-west-2", "latency_ms": 95 + (i % 11), "status": "ok"} + for i in range(30) + ] + rows[19] = { + "seq": 19, + "region": "us-east-1", + "latency_ms": 99999, + "status": "ok", + "note": "latency spike us-east-1", + } + cases.append( + { + "id": "latency_anomaly", + "question": "Which region had the latency spike, and how high?", + "content_type": "json_array", + "compress": {"with_compaction": False, "max_items_after_crush": 8}, + "answer_evidence": ["us-east-1", "99999"], + "supporting_facts": [], + "content": rows, + } + ) + + # 4. CI test failure among many passes — "failed" keyword retention. + rows = [ + {"seq": i, "test": f"test_module_{i}", "outcome": "passed", "duration_ms": 5 + i} + for i in range(30) + ] + rows[12] = { + "seq": 12, + "test": "test_auth_token_refresh", + "outcome": "failed", + "duration_ms": 41, + "error": "AssertionError: expected 200 got 401 in test_auth_token_refresh", + } + cases.append( + { + "id": "ci_test_failures", + "question": "Which test failed and why?", + "content_type": "json_array", + "compress": {"with_compaction": False, "max_items_after_crush": 9}, + "answer_evidence": ["test_auth_token_refresh", "expected 200 got 401"], + "supporting_facts": [], + "content": rows, + } + ) + + return cases + + +def main() -> None: + OUT.write_text(json.dumps(_cases(), indent=2) + "\n") + print(f"wrote {OUT} ({len(_cases())} cases)") + + +if __name__ == "__main__": + main() diff --git a/tests/fixtures/fidelity_golden/baseline.json b/tests/fixtures/fidelity_golden/baseline.json new file mode 100644 index 000000000..44f0b7023 --- /dev/null +++ b/tests/fixtures/fidelity_golden/baseline.json @@ -0,0 +1,5 @@ +{ + "aggregate_recall": 0.9167, + "tolerance": 0.02, + "note": "Mean information-recall across all golden cases (answer_evidence + supporting_facts) after lossy SmartCrusher compression. Deterministic. Regenerate fixtures via `python tests/fixtures/fidelity_golden/_generate.py`; update this value only when a recall change is intended and understood." +} diff --git a/tests/fixtures/fidelity_golden/cases.json b/tests/fixtures/fidelity_golden/cases.json new file mode 100644 index 000000000..13c1dc905 --- /dev/null +++ b/tests/fixtures/fidelity_golden/cases.json @@ -0,0 +1,796 @@ +[ + { + "id": "logs_oom", + "question": "Why did the job fail?", + "content_type": "json_array", + "compress": { + "with_compaction": false, + "max_items_after_crush": 10 + }, + "answer_evidence": [ + "OOM killed worker 3", + "exceeded memory limit" + ], + "supporting_facts": [ + "checkpoint saved at step 4500" + ], + "content": [ + { + "seq": 0, + "level": "INFO", + "status": "ok", + "msg": "heartbeat ping 0" + }, + { + "seq": 1, + "level": "INFO", + "status": "ok", + "msg": "heartbeat ping 1" + }, + { + "seq": 2, + "level": "INFO", + "status": "ok", + "msg": "heartbeat ping 2" + }, + { + "seq": 3, + "level": "INFO", + "status": "ok", + "msg": "heartbeat ping 3" + }, + { + "seq": 4, + "level": "INFO", + "status": "ok", + "msg": "heartbeat ping 4" + }, + { + "seq": 5, + "level": "INFO", + "status": "ok", + "msg": "heartbeat ping 5" + }, + { + "seq": 6, + "level": "INFO", + "status": "ok", + "msg": "heartbeat ping 6" + }, + { + "seq": 7, + "level": "INFO", + "status": "ok", + "msg": "heartbeat ping 7" + }, + { + "seq": 8, + "level": "INFO", + "status": "ok", + "msg": "heartbeat ping 8" + }, + { + "seq": 9, + "level": "INFO", + "status": "ok", + "msg": "checkpoint saved at step 4500" + }, + { + "seq": 9, + "level": "INFO", + "status": "ok", + "msg": "heartbeat ping 9" + }, + { + "seq": 10, + "level": "INFO", + "status": "ok", + "msg": "heartbeat ping 10" + }, + { + "seq": 11, + "level": "INFO", + "status": "ok", + "msg": "heartbeat ping 11" + }, + { + "seq": 12, + "level": "INFO", + "status": "ok", + "msg": "heartbeat ping 12" + }, + { + "seq": 13, + "level": "INFO", + "status": "ok", + "msg": "heartbeat ping 13" + }, + { + "seq": 14, + "level": "INFO", + "status": "ok", + "msg": "heartbeat ping 14" + }, + { + "seq": 15, + "level": "INFO", + "status": "ok", + "msg": "heartbeat ping 15" + }, + { + "seq": 16, + "level": "INFO", + "status": "ok", + "msg": "heartbeat ping 16" + }, + { + "seq": 17, + "level": "ERROR", + "status": "failed", + "msg": "OOM killed worker 3 exceeded memory limit 8GiB" + }, + { + "seq": 17, + "level": "INFO", + "status": "ok", + "msg": "heartbeat ping 17" + }, + { + "seq": 18, + "level": "INFO", + "status": "ok", + "msg": "heartbeat ping 18" + }, + { + "seq": 19, + "level": "INFO", + "status": "ok", + "msg": "heartbeat ping 19" + }, + { + "seq": 20, + "level": "INFO", + "status": "ok", + "msg": "heartbeat ping 20" + }, + { + "seq": 21, + "level": "INFO", + "status": "ok", + "msg": "heartbeat ping 21" + }, + { + "seq": 22, + "level": "INFO", + "status": "ok", + "msg": "heartbeat ping 22" + }, + { + "seq": 23, + "level": "INFO", + "status": "ok", + "msg": "heartbeat ping 23" + }, + { + "seq": 24, + "level": "INFO", + "status": "ok", + "msg": "heartbeat ping 24" + }, + { + "seq": 25, + "level": "INFO", + "status": "ok", + "msg": "heartbeat ping 25" + }, + { + "seq": 26, + "level": "INFO", + "status": "ok", + "msg": "heartbeat ping 26" + }, + { + "seq": 27, + "level": "INFO", + "status": "ok", + "msg": "heartbeat ping 27" + } + ] + }, + { + "id": "payment_exception", + "question": "Which service threw an exception and where?", + "content_type": "json_array", + "compress": { + "with_compaction": false, + "max_items_after_crush": 8 + }, + "answer_evidence": [ + "NullPointerException", + "PaymentService.charge line 88" + ], + "supporting_facts": [], + "content": [ + { + "seq": 0, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 0" + }, + { + "seq": 1, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 1" + }, + { + "seq": 2, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 2" + }, + { + "seq": 3, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 3" + }, + { + "seq": 4, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 4" + }, + { + "seq": 5, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 5" + }, + { + "seq": 6, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 6" + }, + { + "seq": 7, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 7" + }, + { + "seq": 8, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 8" + }, + { + "seq": 9, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 9" + }, + { + "seq": 10, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 10" + }, + { + "seq": 11, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 11" + }, + { + "seq": 12, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 12" + }, + { + "seq": 13, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 13" + }, + { + "seq": 14, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 14" + }, + { + "seq": 15, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 15" + }, + { + "seq": 16, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 16" + }, + { + "seq": 17, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 17" + }, + { + "seq": 18, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 18" + }, + { + "seq": 19, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 19" + }, + { + "seq": 20, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 20" + }, + { + "seq": 21, + "level": "ERROR", + "status": "failed", + "msg": "exception: NullPointerException at PaymentService.charge line 88" + }, + { + "seq": 21, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 21" + }, + { + "seq": 22, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 22" + }, + { + "seq": 23, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 23" + }, + { + "seq": 24, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 24" + }, + { + "seq": 25, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 25" + }, + { + "seq": 26, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 26" + }, + { + "seq": 27, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 27" + }, + { + "seq": 28, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 28" + }, + { + "seq": 29, + "level": "INFO", + "status": "ok", + "msg": "GET /healthz 200 29" + } + ] + }, + { + "id": "latency_anomaly", + "question": "Which region had the latency spike, and how high?", + "content_type": "json_array", + "compress": { + "with_compaction": false, + "max_items_after_crush": 8 + }, + "answer_evidence": [ + "us-east-1", + "99999" + ], + "supporting_facts": [], + "content": [ + { + "seq": 0, + "region": "us-west-2", + "latency_ms": 95, + "status": "ok" + }, + { + "seq": 1, + "region": "us-west-2", + "latency_ms": 96, + "status": "ok" + }, + { + "seq": 2, + "region": "us-west-2", + "latency_ms": 97, + "status": "ok" + }, + { + "seq": 3, + "region": "us-west-2", + "latency_ms": 98, + "status": "ok" + }, + { + "seq": 4, + "region": "us-west-2", + "latency_ms": 99, + "status": "ok" + }, + { + "seq": 5, + "region": "us-west-2", + "latency_ms": 100, + "status": "ok" + }, + { + "seq": 6, + "region": "us-west-2", + "latency_ms": 101, + "status": "ok" + }, + { + "seq": 7, + "region": "us-west-2", + "latency_ms": 102, + "status": "ok" + }, + { + "seq": 8, + "region": "us-west-2", + "latency_ms": 103, + "status": "ok" + }, + { + "seq": 9, + "region": "us-west-2", + "latency_ms": 104, + "status": "ok" + }, + { + "seq": 10, + "region": "us-west-2", + "latency_ms": 105, + "status": "ok" + }, + { + "seq": 11, + "region": "us-west-2", + "latency_ms": 95, + "status": "ok" + }, + { + "seq": 12, + "region": "us-west-2", + "latency_ms": 96, + "status": "ok" + }, + { + "seq": 13, + "region": "us-west-2", + "latency_ms": 97, + "status": "ok" + }, + { + "seq": 14, + "region": "us-west-2", + "latency_ms": 98, + "status": "ok" + }, + { + "seq": 15, + "region": "us-west-2", + "latency_ms": 99, + "status": "ok" + }, + { + "seq": 16, + "region": "us-west-2", + "latency_ms": 100, + "status": "ok" + }, + { + "seq": 17, + "region": "us-west-2", + "latency_ms": 101, + "status": "ok" + }, + { + "seq": 18, + "region": "us-west-2", + "latency_ms": 102, + "status": "ok" + }, + { + "seq": 19, + "region": "us-east-1", + "latency_ms": 99999, + "status": "ok", + "note": "latency spike us-east-1" + }, + { + "seq": 20, + "region": "us-west-2", + "latency_ms": 104, + "status": "ok" + }, + { + "seq": 21, + "region": "us-west-2", + "latency_ms": 105, + "status": "ok" + }, + { + "seq": 22, + "region": "us-west-2", + "latency_ms": 95, + "status": "ok" + }, + { + "seq": 23, + "region": "us-west-2", + "latency_ms": 96, + "status": "ok" + }, + { + "seq": 24, + "region": "us-west-2", + "latency_ms": 97, + "status": "ok" + }, + { + "seq": 25, + "region": "us-west-2", + "latency_ms": 98, + "status": "ok" + }, + { + "seq": 26, + "region": "us-west-2", + "latency_ms": 99, + "status": "ok" + }, + { + "seq": 27, + "region": "us-west-2", + "latency_ms": 100, + "status": "ok" + }, + { + "seq": 28, + "region": "us-west-2", + "latency_ms": 101, + "status": "ok" + }, + { + "seq": 29, + "region": "us-west-2", + "latency_ms": 102, + "status": "ok" + } + ] + }, + { + "id": "ci_test_failures", + "question": "Which test failed and why?", + "content_type": "json_array", + "compress": { + "with_compaction": false, + "max_items_after_crush": 9 + }, + "answer_evidence": [ + "test_auth_token_refresh", + "expected 200 got 401" + ], + "supporting_facts": [], + "content": [ + { + "seq": 0, + "test": "test_module_0", + "outcome": "passed", + "duration_ms": 5 + }, + { + "seq": 1, + "test": "test_module_1", + "outcome": "passed", + "duration_ms": 6 + }, + { + "seq": 2, + "test": "test_module_2", + "outcome": "passed", + "duration_ms": 7 + }, + { + "seq": 3, + "test": "test_module_3", + "outcome": "passed", + "duration_ms": 8 + }, + { + "seq": 4, + "test": "test_module_4", + "outcome": "passed", + "duration_ms": 9 + }, + { + "seq": 5, + "test": "test_module_5", + "outcome": "passed", + "duration_ms": 10 + }, + { + "seq": 6, + "test": "test_module_6", + "outcome": "passed", + "duration_ms": 11 + }, + { + "seq": 7, + "test": "test_module_7", + "outcome": "passed", + "duration_ms": 12 + }, + { + "seq": 8, + "test": "test_module_8", + "outcome": "passed", + "duration_ms": 13 + }, + { + "seq": 9, + "test": "test_module_9", + "outcome": "passed", + "duration_ms": 14 + }, + { + "seq": 10, + "test": "test_module_10", + "outcome": "passed", + "duration_ms": 15 + }, + { + "seq": 11, + "test": "test_module_11", + "outcome": "passed", + "duration_ms": 16 + }, + { + "seq": 12, + "test": "test_auth_token_refresh", + "outcome": "failed", + "duration_ms": 41, + "error": "AssertionError: expected 200 got 401 in test_auth_token_refresh" + }, + { + "seq": 13, + "test": "test_module_13", + "outcome": "passed", + "duration_ms": 18 + }, + { + "seq": 14, + "test": "test_module_14", + "outcome": "passed", + "duration_ms": 19 + }, + { + "seq": 15, + "test": "test_module_15", + "outcome": "passed", + "duration_ms": 20 + }, + { + "seq": 16, + "test": "test_module_16", + "outcome": "passed", + "duration_ms": 21 + }, + { + "seq": 17, + "test": "test_module_17", + "outcome": "passed", + "duration_ms": 22 + }, + { + "seq": 18, + "test": "test_module_18", + "outcome": "passed", + "duration_ms": 23 + }, + { + "seq": 19, + "test": "test_module_19", + "outcome": "passed", + "duration_ms": 24 + }, + { + "seq": 20, + "test": "test_module_20", + "outcome": "passed", + "duration_ms": 25 + }, + { + "seq": 21, + "test": "test_module_21", + "outcome": "passed", + "duration_ms": 26 + }, + { + "seq": 22, + "test": "test_module_22", + "outcome": "passed", + "duration_ms": 27 + }, + { + "seq": 23, + "test": "test_module_23", + "outcome": "passed", + "duration_ms": 28 + }, + { + "seq": 24, + "test": "test_module_24", + "outcome": "passed", + "duration_ms": 29 + }, + { + "seq": 25, + "test": "test_module_25", + "outcome": "passed", + "duration_ms": 30 + }, + { + "seq": 26, + "test": "test_module_26", + "outcome": "passed", + "duration_ms": 31 + }, + { + "seq": 27, + "test": "test_module_27", + "outcome": "passed", + "duration_ms": 32 + }, + { + "seq": 28, + "test": "test_module_28", + "outcome": "passed", + "duration_ms": 33 + }, + { + "seq": 29, + "test": "test_module_29", + "outcome": "passed", + "duration_ms": 34 + } + ] + } +] diff --git a/tests/test_compression_fidelity_regression.py b/tests/test_compression_fidelity_regression.py new file mode 100644 index 000000000..a83cea13d --- /dev/null +++ b/tests/test_compression_fidelity_regression.py @@ -0,0 +1,72 @@ +"""Offline fidelity regression gate (recall-based, zero-model). + +Compresses vendored golden tool-output fixtures through SmartCrusher's lossy +path and asserts that the evidence a model needs to answer each case's question +survives compression. Scoring is pure stdlib (``headroom.evals.metrics``) — no +ML model, no network, no API keys — so this runs in the standard ``[dev]`` CI +shard as a blocking PR check. + +A failure here means a code change made lossy compression silently drop +information that answers a known question. Fixtures and the committed baseline +are generated by ``tests/fixtures/fidelity_golden/_generate.py``. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from headroom.evals.metrics import compute_information_recall +from headroom.transforms.smart_crusher import SmartCrusherConfig, smart_crush_tool_output + +FIXTURE_DIR = Path(__file__).parent / "fixtures" / "fidelity_golden" +CASES: list[dict] = json.loads((FIXTURE_DIR / "cases.json").read_text()) +BASELINE: dict = json.loads((FIXTURE_DIR / "baseline.json").read_text()) + + +def _compress(case: dict) -> tuple[str, str]: + """Compress a case's tool output via the lossy SmartCrusher path (no model).""" + original = json.dumps(case["content"]) + cfg = SmartCrusherConfig(max_items_after_crush=case["compress"]["max_items_after_crush"]) + crushed, _modified, _info = smart_crush_tool_output( + original, cfg, with_compaction=case["compress"]["with_compaction"] + ) + return original, crushed + + +@pytest.mark.parametrize("case", CASES, ids=[c["id"] for c in CASES]) +def test_critical_evidence_survives_compression(case: dict) -> None: + """Every ``answer_evidence`` string MUST survive lossy compression (recall == 1.0). + + Critical evidence lives in error/anomaly rows, which SmartCrusher formally + guarantees to retain (see ``tests/test_quality_retention.py``). + """ + original, crushed = _compress(case) + result = compute_information_recall(original, crushed, case["answer_evidence"]) + assert result["recall"] == 1.0, ( + f"FIDELITY REGRESSION in '{case['id']}': compression dropped evidence " + f"needed to answer {case['question']!r}. Lost: {result['facts_lost']}" + ) + + +def test_aggregate_recall_not_regressed() -> None: + """Mean recall over all evidence must not fall below the committed baseline. + + Catches softer regressions (e.g. relevant-but-non-critical context being + dropped more aggressively) that the per-case critical gate would not. + """ + recalls = [] + for case in CASES: + original, crushed = _compress(case) + probes = case["answer_evidence"] + case["supporting_facts"] + recalls.append(compute_information_recall(original, crushed, probes)["recall"]) + + mean_recall = sum(recalls) / len(recalls) + floor = BASELINE["aggregate_recall"] - BASELINE["tolerance"] + assert mean_recall >= floor, ( + f"FIDELITY REGRESSION: mean recall {mean_recall:.4f} fell below baseline " + f"floor {floor:.4f} (baseline {BASELINE['aggregate_recall']} - tolerance " + f"{BASELINE['tolerance']}). If this drop is intended, regenerate the baseline." + )