mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
23d73ae070
## Description Headroom's lossy compression drops rows/lines using statistical heuristics but **never checks that meaning survived** — a dropped `OOM killed worker 3` line can silently flip a model's answer with no signal that compression caused it. The repo already ships a quality-metric toolkit (`headroom/evals/metrics.py`) and a `weekly-suite` eval job, but neither gates the compression path on a PR. This adds a **per-PR fidelity regression gate**: compress vendored golden tool-outputs through SmartCrusher's lossy path and assert the evidence that answers each case's question survives. It is the first of a planned trio (this is the "offline gate" half of the fidelity work); query-aware retention and a hard token-budget API are documented follow-ups. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Blocking gate** (`tests/test_compression_fidelity_regression.py`): compresses each golden case via `smart_crush_tool_output(..., with_compaction=False)` and scores with `compute_information_recall`. Two assertions: - **Per-case critical recall == 1.0** — every `answer_evidence` string (placed in error/anomaly rows, the documented SmartCrusher retention guarantee) must survive. - **Aggregate recall ≥ committed baseline** (`baseline.json`, tol 0.02) — catches softer regressions. - **Vendored fixtures** (`tests/fixtures/fidelity_golden/`): deterministic `_generate.py` emits `cases.json` (4 cases: OOM crash, payment exception, latency anomaly, CI failure) + `baseline.json`. - **Non-blocking weekly report** (`.github/workflows/eval.yml`): one step in the existing `weekly-suite` job (schedule/manual only) reuses the existing `evaluate_information_retention` runner for a recall report on the production routing path. - **Pure reuse**: scoring (`evals/metrics.py`), compressor (`smart_crush_tool_output`), and the weekly runner (`evaluate_information_retention`) all already existed. ### Design notes - **Zero new CI setup.** The blocking gate runs in the existing `[dev]` test shard — no new workflow, no new deps, **no model, no network, no secrets** (verified under `HF_HUB_OFFLINE=1`). It deliberately uses small hand-made structured fixtures rather than the repo's HuggingFace dataset loaders, which would require a network download + ModernBERT and don't belong in a fast PR gate. - **Scope:** structured JSON tool-output (the dominant, deterministic, model-free case). Real-dataset (HotpotQA/BFCL) recall — which needs `[all]` + a local model — is a **documented follow-up PR**, and the `weekly-suite` job (which genuinely runs every Monday) is its natural home. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ HF_HUB_OFFLINE=1 python -m pytest tests/test_compression_fidelity_regression.py -v tests/test_compression_fidelity_regression.py::test_critical_evidence_survives_compression[logs_oom] PASSED tests/test_compression_fidelity_regression.py::test_critical_evidence_survives_compression[payment_exception] PASSED tests/test_compression_fidelity_regression.py::test_critical_evidence_survives_compression[latency_anomaly] PASSED tests/test_compression_fidelity_regression.py::test_critical_evidence_survives_compression[ci_test_failures] PASSED tests/test_compression_fidelity_regression.py::test_aggregate_recall_not_regressed PASSED ============================== 5 passed in 0.18s =============================== ``` ## Real Behavior Proof - **Environment:** local checkout of `feat/fidelity-regression-gate`, `pip install -e ".[dev]"`, `HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1` (proves no model/network). - **Exact command / steps:** `HF_HUB_OFFLINE=1 python -m pytest tests/test_compression_fidelity_regression.py -q` → `5 passed in 0.14s`. - **Negative control (proves the gate has teeth):** compressing `logs_oom` and probing for a benign row that compression legitimately drops returns `recall = 0.00, lost = ['heartbeat ping 25']` — i.e. the gate fires when critical evidence is dropped, so it is not trivially green. - **Weekly (non-blocking) step verified locally:** ```text Information retention: 50/50 cases >=0.9 recall, avg compression 65.7% ``` - **Not tested:** real-dataset (HotpotQA/BFCL) recall and prose/ModernBERT compression — intentionally deferred to a follow-up PR targeting the weekly job. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG/version intentionally untouched: repo uses **release-please**. - **Follow-up PR (planned):** wire the real HotpotQA/BFCL loaders (`headroom/evals/datasets.py`) into the `weekly-suite` job for genuine benchmark-scale recall coverage (model-allowed, non-blocking). Further follow-ups from the same design: a live per-request fidelity guardrail, query-aware lossy retention, and a hard `target_tokens` budget API. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
796 lines
16 KiB
JSON
796 lines
16 KiB
JSON
[
|
|
{
|
|
"id": "logs_oom",
|
|
"question": "Why did the job fail?",
|
|
"content_type": "json_array",
|
|
"compress": {
|
|
"with_compaction": false,
|
|
"max_items_after_crush": 10
|
|
},
|
|
"answer_evidence": [
|
|
"OOM killed worker 3",
|
|
"exceeded memory limit"
|
|
],
|
|
"supporting_facts": [
|
|
"checkpoint saved at step 4500"
|
|
],
|
|
"content": [
|
|
{
|
|
"seq": 0,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "heartbeat ping 0"
|
|
},
|
|
{
|
|
"seq": 1,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "heartbeat ping 1"
|
|
},
|
|
{
|
|
"seq": 2,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "heartbeat ping 2"
|
|
},
|
|
{
|
|
"seq": 3,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "heartbeat ping 3"
|
|
},
|
|
{
|
|
"seq": 4,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "heartbeat ping 4"
|
|
},
|
|
{
|
|
"seq": 5,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "heartbeat ping 5"
|
|
},
|
|
{
|
|
"seq": 6,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "heartbeat ping 6"
|
|
},
|
|
{
|
|
"seq": 7,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "heartbeat ping 7"
|
|
},
|
|
{
|
|
"seq": 8,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "heartbeat ping 8"
|
|
},
|
|
{
|
|
"seq": 9,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "checkpoint saved at step 4500"
|
|
},
|
|
{
|
|
"seq": 9,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "heartbeat ping 9"
|
|
},
|
|
{
|
|
"seq": 10,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "heartbeat ping 10"
|
|
},
|
|
{
|
|
"seq": 11,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "heartbeat ping 11"
|
|
},
|
|
{
|
|
"seq": 12,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "heartbeat ping 12"
|
|
},
|
|
{
|
|
"seq": 13,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "heartbeat ping 13"
|
|
},
|
|
{
|
|
"seq": 14,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "heartbeat ping 14"
|
|
},
|
|
{
|
|
"seq": 15,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "heartbeat ping 15"
|
|
},
|
|
{
|
|
"seq": 16,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "heartbeat ping 16"
|
|
},
|
|
{
|
|
"seq": 17,
|
|
"level": "ERROR",
|
|
"status": "failed",
|
|
"msg": "OOM killed worker 3 exceeded memory limit 8GiB"
|
|
},
|
|
{
|
|
"seq": 17,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "heartbeat ping 17"
|
|
},
|
|
{
|
|
"seq": 18,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "heartbeat ping 18"
|
|
},
|
|
{
|
|
"seq": 19,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "heartbeat ping 19"
|
|
},
|
|
{
|
|
"seq": 20,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "heartbeat ping 20"
|
|
},
|
|
{
|
|
"seq": 21,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "heartbeat ping 21"
|
|
},
|
|
{
|
|
"seq": 22,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "heartbeat ping 22"
|
|
},
|
|
{
|
|
"seq": 23,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "heartbeat ping 23"
|
|
},
|
|
{
|
|
"seq": 24,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "heartbeat ping 24"
|
|
},
|
|
{
|
|
"seq": 25,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "heartbeat ping 25"
|
|
},
|
|
{
|
|
"seq": 26,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "heartbeat ping 26"
|
|
},
|
|
{
|
|
"seq": 27,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "heartbeat ping 27"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "payment_exception",
|
|
"question": "Which service threw an exception and where?",
|
|
"content_type": "json_array",
|
|
"compress": {
|
|
"with_compaction": false,
|
|
"max_items_after_crush": 8
|
|
},
|
|
"answer_evidence": [
|
|
"NullPointerException",
|
|
"PaymentService.charge line 88"
|
|
],
|
|
"supporting_facts": [],
|
|
"content": [
|
|
{
|
|
"seq": 0,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 0"
|
|
},
|
|
{
|
|
"seq": 1,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 1"
|
|
},
|
|
{
|
|
"seq": 2,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 2"
|
|
},
|
|
{
|
|
"seq": 3,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 3"
|
|
},
|
|
{
|
|
"seq": 4,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 4"
|
|
},
|
|
{
|
|
"seq": 5,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 5"
|
|
},
|
|
{
|
|
"seq": 6,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 6"
|
|
},
|
|
{
|
|
"seq": 7,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 7"
|
|
},
|
|
{
|
|
"seq": 8,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 8"
|
|
},
|
|
{
|
|
"seq": 9,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 9"
|
|
},
|
|
{
|
|
"seq": 10,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 10"
|
|
},
|
|
{
|
|
"seq": 11,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 11"
|
|
},
|
|
{
|
|
"seq": 12,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 12"
|
|
},
|
|
{
|
|
"seq": 13,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 13"
|
|
},
|
|
{
|
|
"seq": 14,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 14"
|
|
},
|
|
{
|
|
"seq": 15,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 15"
|
|
},
|
|
{
|
|
"seq": 16,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 16"
|
|
},
|
|
{
|
|
"seq": 17,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 17"
|
|
},
|
|
{
|
|
"seq": 18,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 18"
|
|
},
|
|
{
|
|
"seq": 19,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 19"
|
|
},
|
|
{
|
|
"seq": 20,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 20"
|
|
},
|
|
{
|
|
"seq": 21,
|
|
"level": "ERROR",
|
|
"status": "failed",
|
|
"msg": "exception: NullPointerException at PaymentService.charge line 88"
|
|
},
|
|
{
|
|
"seq": 21,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 21"
|
|
},
|
|
{
|
|
"seq": 22,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 22"
|
|
},
|
|
{
|
|
"seq": 23,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 23"
|
|
},
|
|
{
|
|
"seq": 24,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 24"
|
|
},
|
|
{
|
|
"seq": 25,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 25"
|
|
},
|
|
{
|
|
"seq": 26,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 26"
|
|
},
|
|
{
|
|
"seq": 27,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 27"
|
|
},
|
|
{
|
|
"seq": 28,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 28"
|
|
},
|
|
{
|
|
"seq": 29,
|
|
"level": "INFO",
|
|
"status": "ok",
|
|
"msg": "GET /healthz 200 29"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "latency_anomaly",
|
|
"question": "Which region had the latency spike, and how high?",
|
|
"content_type": "json_array",
|
|
"compress": {
|
|
"with_compaction": false,
|
|
"max_items_after_crush": 8
|
|
},
|
|
"answer_evidence": [
|
|
"us-east-1",
|
|
"99999"
|
|
],
|
|
"supporting_facts": [],
|
|
"content": [
|
|
{
|
|
"seq": 0,
|
|
"region": "us-west-2",
|
|
"latency_ms": 95,
|
|
"status": "ok"
|
|
},
|
|
{
|
|
"seq": 1,
|
|
"region": "us-west-2",
|
|
"latency_ms": 96,
|
|
"status": "ok"
|
|
},
|
|
{
|
|
"seq": 2,
|
|
"region": "us-west-2",
|
|
"latency_ms": 97,
|
|
"status": "ok"
|
|
},
|
|
{
|
|
"seq": 3,
|
|
"region": "us-west-2",
|
|
"latency_ms": 98,
|
|
"status": "ok"
|
|
},
|
|
{
|
|
"seq": 4,
|
|
"region": "us-west-2",
|
|
"latency_ms": 99,
|
|
"status": "ok"
|
|
},
|
|
{
|
|
"seq": 5,
|
|
"region": "us-west-2",
|
|
"latency_ms": 100,
|
|
"status": "ok"
|
|
},
|
|
{
|
|
"seq": 6,
|
|
"region": "us-west-2",
|
|
"latency_ms": 101,
|
|
"status": "ok"
|
|
},
|
|
{
|
|
"seq": 7,
|
|
"region": "us-west-2",
|
|
"latency_ms": 102,
|
|
"status": "ok"
|
|
},
|
|
{
|
|
"seq": 8,
|
|
"region": "us-west-2",
|
|
"latency_ms": 103,
|
|
"status": "ok"
|
|
},
|
|
{
|
|
"seq": 9,
|
|
"region": "us-west-2",
|
|
"latency_ms": 104,
|
|
"status": "ok"
|
|
},
|
|
{
|
|
"seq": 10,
|
|
"region": "us-west-2",
|
|
"latency_ms": 105,
|
|
"status": "ok"
|
|
},
|
|
{
|
|
"seq": 11,
|
|
"region": "us-west-2",
|
|
"latency_ms": 95,
|
|
"status": "ok"
|
|
},
|
|
{
|
|
"seq": 12,
|
|
"region": "us-west-2",
|
|
"latency_ms": 96,
|
|
"status": "ok"
|
|
},
|
|
{
|
|
"seq": 13,
|
|
"region": "us-west-2",
|
|
"latency_ms": 97,
|
|
"status": "ok"
|
|
},
|
|
{
|
|
"seq": 14,
|
|
"region": "us-west-2",
|
|
"latency_ms": 98,
|
|
"status": "ok"
|
|
},
|
|
{
|
|
"seq": 15,
|
|
"region": "us-west-2",
|
|
"latency_ms": 99,
|
|
"status": "ok"
|
|
},
|
|
{
|
|
"seq": 16,
|
|
"region": "us-west-2",
|
|
"latency_ms": 100,
|
|
"status": "ok"
|
|
},
|
|
{
|
|
"seq": 17,
|
|
"region": "us-west-2",
|
|
"latency_ms": 101,
|
|
"status": "ok"
|
|
},
|
|
{
|
|
"seq": 18,
|
|
"region": "us-west-2",
|
|
"latency_ms": 102,
|
|
"status": "ok"
|
|
},
|
|
{
|
|
"seq": 19,
|
|
"region": "us-east-1",
|
|
"latency_ms": 99999,
|
|
"status": "ok",
|
|
"note": "latency spike us-east-1"
|
|
},
|
|
{
|
|
"seq": 20,
|
|
"region": "us-west-2",
|
|
"latency_ms": 104,
|
|
"status": "ok"
|
|
},
|
|
{
|
|
"seq": 21,
|
|
"region": "us-west-2",
|
|
"latency_ms": 105,
|
|
"status": "ok"
|
|
},
|
|
{
|
|
"seq": 22,
|
|
"region": "us-west-2",
|
|
"latency_ms": 95,
|
|
"status": "ok"
|
|
},
|
|
{
|
|
"seq": 23,
|
|
"region": "us-west-2",
|
|
"latency_ms": 96,
|
|
"status": "ok"
|
|
},
|
|
{
|
|
"seq": 24,
|
|
"region": "us-west-2",
|
|
"latency_ms": 97,
|
|
"status": "ok"
|
|
},
|
|
{
|
|
"seq": 25,
|
|
"region": "us-west-2",
|
|
"latency_ms": 98,
|
|
"status": "ok"
|
|
},
|
|
{
|
|
"seq": 26,
|
|
"region": "us-west-2",
|
|
"latency_ms": 99,
|
|
"status": "ok"
|
|
},
|
|
{
|
|
"seq": 27,
|
|
"region": "us-west-2",
|
|
"latency_ms": 100,
|
|
"status": "ok"
|
|
},
|
|
{
|
|
"seq": 28,
|
|
"region": "us-west-2",
|
|
"latency_ms": 101,
|
|
"status": "ok"
|
|
},
|
|
{
|
|
"seq": 29,
|
|
"region": "us-west-2",
|
|
"latency_ms": 102,
|
|
"status": "ok"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "ci_test_failures",
|
|
"question": "Which test failed and why?",
|
|
"content_type": "json_array",
|
|
"compress": {
|
|
"with_compaction": false,
|
|
"max_items_after_crush": 9
|
|
},
|
|
"answer_evidence": [
|
|
"test_auth_token_refresh",
|
|
"expected 200 got 401"
|
|
],
|
|
"supporting_facts": [],
|
|
"content": [
|
|
{
|
|
"seq": 0,
|
|
"test": "test_module_0",
|
|
"outcome": "passed",
|
|
"duration_ms": 5
|
|
},
|
|
{
|
|
"seq": 1,
|
|
"test": "test_module_1",
|
|
"outcome": "passed",
|
|
"duration_ms": 6
|
|
},
|
|
{
|
|
"seq": 2,
|
|
"test": "test_module_2",
|
|
"outcome": "passed",
|
|
"duration_ms": 7
|
|
},
|
|
{
|
|
"seq": 3,
|
|
"test": "test_module_3",
|
|
"outcome": "passed",
|
|
"duration_ms": 8
|
|
},
|
|
{
|
|
"seq": 4,
|
|
"test": "test_module_4",
|
|
"outcome": "passed",
|
|
"duration_ms": 9
|
|
},
|
|
{
|
|
"seq": 5,
|
|
"test": "test_module_5",
|
|
"outcome": "passed",
|
|
"duration_ms": 10
|
|
},
|
|
{
|
|
"seq": 6,
|
|
"test": "test_module_6",
|
|
"outcome": "passed",
|
|
"duration_ms": 11
|
|
},
|
|
{
|
|
"seq": 7,
|
|
"test": "test_module_7",
|
|
"outcome": "passed",
|
|
"duration_ms": 12
|
|
},
|
|
{
|
|
"seq": 8,
|
|
"test": "test_module_8",
|
|
"outcome": "passed",
|
|
"duration_ms": 13
|
|
},
|
|
{
|
|
"seq": 9,
|
|
"test": "test_module_9",
|
|
"outcome": "passed",
|
|
"duration_ms": 14
|
|
},
|
|
{
|
|
"seq": 10,
|
|
"test": "test_module_10",
|
|
"outcome": "passed",
|
|
"duration_ms": 15
|
|
},
|
|
{
|
|
"seq": 11,
|
|
"test": "test_module_11",
|
|
"outcome": "passed",
|
|
"duration_ms": 16
|
|
},
|
|
{
|
|
"seq": 12,
|
|
"test": "test_auth_token_refresh",
|
|
"outcome": "failed",
|
|
"duration_ms": 41,
|
|
"error": "AssertionError: expected 200 got 401 in test_auth_token_refresh"
|
|
},
|
|
{
|
|
"seq": 13,
|
|
"test": "test_module_13",
|
|
"outcome": "passed",
|
|
"duration_ms": 18
|
|
},
|
|
{
|
|
"seq": 14,
|
|
"test": "test_module_14",
|
|
"outcome": "passed",
|
|
"duration_ms": 19
|
|
},
|
|
{
|
|
"seq": 15,
|
|
"test": "test_module_15",
|
|
"outcome": "passed",
|
|
"duration_ms": 20
|
|
},
|
|
{
|
|
"seq": 16,
|
|
"test": "test_module_16",
|
|
"outcome": "passed",
|
|
"duration_ms": 21
|
|
},
|
|
{
|
|
"seq": 17,
|
|
"test": "test_module_17",
|
|
"outcome": "passed",
|
|
"duration_ms": 22
|
|
},
|
|
{
|
|
"seq": 18,
|
|
"test": "test_module_18",
|
|
"outcome": "passed",
|
|
"duration_ms": 23
|
|
},
|
|
{
|
|
"seq": 19,
|
|
"test": "test_module_19",
|
|
"outcome": "passed",
|
|
"duration_ms": 24
|
|
},
|
|
{
|
|
"seq": 20,
|
|
"test": "test_module_20",
|
|
"outcome": "passed",
|
|
"duration_ms": 25
|
|
},
|
|
{
|
|
"seq": 21,
|
|
"test": "test_module_21",
|
|
"outcome": "passed",
|
|
"duration_ms": 26
|
|
},
|
|
{
|
|
"seq": 22,
|
|
"test": "test_module_22",
|
|
"outcome": "passed",
|
|
"duration_ms": 27
|
|
},
|
|
{
|
|
"seq": 23,
|
|
"test": "test_module_23",
|
|
"outcome": "passed",
|
|
"duration_ms": 28
|
|
},
|
|
{
|
|
"seq": 24,
|
|
"test": "test_module_24",
|
|
"outcome": "passed",
|
|
"duration_ms": 29
|
|
},
|
|
{
|
|
"seq": 25,
|
|
"test": "test_module_25",
|
|
"outcome": "passed",
|
|
"duration_ms": 30
|
|
},
|
|
{
|
|
"seq": 26,
|
|
"test": "test_module_26",
|
|
"outcome": "passed",
|
|
"duration_ms": 31
|
|
},
|
|
{
|
|
"seq": 27,
|
|
"test": "test_module_27",
|
|
"outcome": "passed",
|
|
"duration_ms": 32
|
|
},
|
|
{
|
|
"seq": 28,
|
|
"test": "test_module_28",
|
|
"outcome": "passed",
|
|
"duration_ms": 33
|
|
},
|
|
{
|
|
"seq": 29,
|
|
"test": "test_module_29",
|
|
"outcome": "passed",
|
|
"duration_ms": 34
|
|
}
|
|
]
|
|
}
|
|
]
|