84f9871e30
## Description Adds the Phase 0 `agent-evals/` nested project for benchmarking coding-agent task accuracy with and without Headroom's proxy/compression path. The project is intentionally separate from the published `headroom-ai` package and provides the shared A/B framework used by the stacked Phase 1 PR #1040. ## Type of Change - [x] New feature (non-breaking change that adds functionality) - [x] Tests only - [x] Documentation update ## Changes Made - Added a three-arm experiment model for direct provider calls, Headroom passthrough, and Headroom compression. - Added the resumable orchestrator, run manifest/config models, JSON logging, and append-only journal handling. - Added savings capture from Headroom response headers plus scorecard reporting for resolved rate and savings. - Added unit tests and live-test markers for provider-key dependent validation. - Kept the benchmark project isolated from the product package and normal Headroom release wheel. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality ### Test Output ```text agent-evals Phase 0 validation from original PR: 74 unit tests passed ruff clean mypy clean CI on this PR: changes and commitlint pass; product CI jobs are skipped because this only changes the nested agent-evals project. ``` ## Real Behavior Proof - Environment: local agent-evals development environment with provider-key dependent live tests skipped unless credentials are present. - Exact command / steps: Ran the Phase 0 unit suite, ruff, and mypy for the nested `agent-evals` project; GitHub CI also ran the repository change detection and commitlint jobs for this PR. - Observed result: The Phase 0 framework tests passed locally, static checks were clean, and GitHub CI reported passing change detection/commitlint for the PR. - Not tested: live provider accuracy claims; those require provider keys and larger benchmark runs and are intentionally covered by live-marked tests and later stacked phases. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|---|---|---|
| .. | ||
| src/agent_evals | ||
| tests | ||
| .gitignore | ||
| Makefile | ||
| pyproject.toml | ||
| README.md | ||
agent-evals
End-to-end accuracy A/B framework for Headroom: run trusted coding-agent benchmarks WITH vs WITHOUT Headroom's context-compression proxy and produce a statistically defensible verdict — does compression preserve what the agent can solve, and how much does it save?
This is a self-contained nested project inside the headroom repo. It consumes Headroom
only as the system-under-test (via base_url); it is not part of the headroom-ai
wheel and is not wired into headroom's make ci-precheck.
The clean A/B (why a proxy helps)
Headroom sits in the request path, so the only variable between arms is base_url:
| Arm | base_url | Headroom mode | Isolates |
|---|---|---|---|
A0_DIRECT |
provider API | none | native agent score |
A1_PASSTHROUGH |
localhost:N |
--no-optimize |
proxy-hop cost only |
B_HEADROOM |
localhost:M |
--mode token |
compression cost (vs A1) |
Headline accuracy claim = B vs A1. A1 vs A0 is the transparency sanity check (≈0).
Method (in one line)
Agentic evals are noisy (single-run pass@1 swings several points even at temp 0), so we run paired, multi-run, non-inferiority experiments: same tasks through every arm, K runs each, and a TOST equivalence test on accuracy (win = savings up, accuracy within margin δ).
Phases
- Phase 0 (this PR) — foundation: 3-arm abstraction, resumable orchestrator, per-task savings capture, proxy-transparency check. Runnable with no benchmark deps and ~no spend.
- Phase 1 — Aider Polyglot end-to-end (first real accuracy + savings scorecard).
- Phase 2 — SWE-bench Verified via OpenHands (the headline), per-transform ablation.
See the design spec for the full architecture.
Develop
cd agent-evals
python -m venv .venv && source .venv/bin/activate
make install # pip install -e ".[dev,stats]"
make gate # ruff + mypy + pytest (the agent-evals push gate)
agent-evals show-config
Live tests (spawn a real proxy / hit upstreams) are -m live/-m real_llm and are skipped
unless provider keys are present in your environment. Keys come from your shell/.env; they
are never read from or written to source.