mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
a2b2ec5463
Evaluation Suite: - Tiered eval framework (Tier 1 ~$3/15min, Tier 2/3 for extended coverage) - 16 benchmarks across 3 tiers: GSM8K, TruthfulQA, MMLU, ARC, HumanEval, SQuAD v2, BFCL, Tool Outputs, CCR needle retention, HotpotQA, and more - Before/After runner with full proxy support (compression + CCR retrieval) - LLM-as-judge for ground-truth comparison (BFCL function calling) - Zero-cost compression-only runner (CCR needle retention, info retention) - Cost tracker with per-model pricing and budget enforcement - Report card generator (Markdown, JSON, HTML) - Suite CLI: python -m headroom.evals suite --tier 1 - Fix BFCL dataset loader for current HuggingFace schema - CI workflow: PR smoke test + weekly full Tier 1 Results: SQuAD 97%, BFCL 97%, Tool Outputs 100%, CCR 100% SmartCrusher: - Universal JSON crush for heterogeneous arrays - Fix mypy redefinition warning in _crush_string_array Other: - Latency benchmark suite with docs - Known limitations doc - Prompt comparison evaluator - Config updates for new features |
||
|---|---|---|
| .. | ||
| scenarios | ||
| __init__.py | ||
| adversarial_ccr_tests.py | ||
| agent_cost_benchmark.py | ||
| bench_latency.py | ||
| bench_relevance.py | ||
| bench_transforms.py | ||
| ccr_regression_benchmark.py | ||
| comprehensive_eval.py | ||
| compression_benchmark.py | ||
| conftest.py | ||
| dynamic_detector_benchmark.py | ||
| headroom_adversarial_benchmark.py | ||
| headroom_worst_case_benchmark.py | ||
| real_world_agent_benchmark.py | ||
| run_benchmarks.py | ||