chopratejas a2b2ec5463 Add OSS evaluation suite, universal JSON crush, latency benchmarks ... Evaluation Suite: - Tiered eval framework (Tier 1 ~$3/15min, Tier 2/3 for extended coverage) - 16 benchmarks across 3 tiers: GSM8K, TruthfulQA, MMLU, ARC, HumanEval, SQuAD v2, BFCL, Tool Outputs, CCR needle retention, HotpotQA, and more - Before/After runner with full proxy support (compression + CCR retrieval) - LLM-as-judge for ground-truth comparison (BFCL function calling) - Zero-cost compression-only runner (CCR needle retention, info retention) - Cost tracker with per-model pricing and budget enforcement - Report card generator (Markdown, JSON, HTML) - Suite CLI: python -m headroom.evals suite --tier 1 - Fix BFCL dataset loader for current HuggingFace schema - CI workflow: PR smoke test + weekly full Tier 1 Results: SQuAD 97%, BFCL 97%, Tool Outputs 100%, CCR 100% SmartCrusher: - Universal JSON crush for heterogeneous arrays - Fix mypy redefinition warning in _crush_string_array Other: - Latency benchmark suite with docs - Known limitations doc - Prompt comparison evaluator - Config updates for new features		2026-02-23 19:08:54 -08:00
..
ISSUE_TEMPLATE	Prepare for OSS release v0.2.0	2026-01-07 11:36:44 -08:00
workflows	Add OSS evaluation suite, universal JSON crush, latency benchmarks	2026-02-23 19:08:54 -08:00
FUNDING.yml	Prepare for OSS release v0.2.0	2026-01-07 11:36:44 -08:00
PULL_REQUEST_TEMPLATE.md	Prepare for OSS release v0.2.0	2026-01-07 11:36:44 -08:00