mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
1 commit
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
JD Davis
|
2dc7e4ab27
|
test: add fluent Headroom harness (#2650)
## Description Adds `headroom.testing`, a fluent, contractual test harness for building Headroom scenarios and suites that can be simulated locally, orchestrated, deployed through the proxy, and handed off to `headroom-bench` / `agent-evals` with bench-native manifests. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add `headroom.testing.Headroom` fluent scenario builder with provider/platform/configuration facets such as `WithBedrock`, `OnAppleSilicon`, `Configure`, `WithCompression`, `WithCCR`, `WithCache`, `WithPrefixFreeze`, `WithReadMaturation`, and `WithMemory`. - Add contractual coverage over the current `HeadroomConfig` and `ProxyConfig` dataclass surfaces, including full JSON-ready proxy deployment payloads. - Add no-key local simulations, scenario/suite orchestration, guarantee evaluation, deployment plans, and a local proxy lifecycle context manager. - Add `headroom-bench` handoff artifacts, including `agent_evals.models.RunManifest`-compatible JSON without taking a runtime dependency on `agent-evals`. - Add demonstration tests for providers, feature facets, manifests, suites, guarantees, deployment payloads, and the no-key simulation path. - Fix unversioned OTEL meter lookup typing so `mypy headroom` remains green on current `main`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text python -m ruff check . All checks passed! python -m mypy headroom headroom\proxy\server.py:1680: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] headroom\proxy\server.py:1691: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] Success: no issues found in 512 source files python -m pytest tests/test_cli/test_subprocess_utf8_encoding.py tests/test_testing_harness.py -q 24 passed, 1 warning in 4.68s ``` ## Real Behavior Proof - Environment: Windows, Python 3.13.13, branch `feat/headroom-test-harness` rebased on `headroomlabs-ai/main`. - Exact command / steps: built a `Headroom.WithOpenAI().WithCompression(mode="cache", kompress=False).Build()` scenario and entered `scenario.deploy_local(port=19192, timeout_s=20)`. - Observed result: proxy launched, `/readyz` succeeded, handle returned `http://127.0.0.1:19192`, `OPENAI_BASE_URL=http://127.0.0.1:19192/v1`, and context-manager teardown completed. - Exact command / steps: emitted `scenario.agent_evals_manifest(...).to_dict()` and validated it with the current cloned `headroom-bench` `agent_evals.models.RunManifest` pydantic model. - Observed result: validation succeeded with arms `a0_direct`, `a1_passthrough`, and `b_headroom` for provider `openai`. - Not tested: upstream-provider API calls requiring real OpenAI/Anthropic/Bedrock keys; phase-1 validation intentionally stays no-key/local. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A. ## Additional Notes The pytest warning shown above is the existing OpenAI pricing-data staleness warning from cost estimation. The harness does not call upstream providers during local simulation. |