mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
10 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Ashish
|
23d73ae070
|
test(evals): add offline fidelity regression gate (recall-based, zero-model) (#1187)
## Description Headroom's lossy compression drops rows/lines using statistical heuristics but **never checks that meaning survived** — a dropped `OOM killed worker 3` line can silently flip a model's answer with no signal that compression caused it. The repo already ships a quality-metric toolkit (`headroom/evals/metrics.py`) and a `weekly-suite` eval job, but neither gates the compression path on a PR. This adds a **per-PR fidelity regression gate**: compress vendored golden tool-outputs through SmartCrusher's lossy path and assert the evidence that answers each case's question survives. It is the first of a planned trio (this is the "offline gate" half of the fidelity work); query-aware retention and a hard token-budget API are documented follow-ups. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Blocking gate** (`tests/test_compression_fidelity_regression.py`): compresses each golden case via `smart_crush_tool_output(..., with_compaction=False)` and scores with `compute_information_recall`. Two assertions: - **Per-case critical recall == 1.0** — every `answer_evidence` string (placed in error/anomaly rows, the documented SmartCrusher retention guarantee) must survive. - **Aggregate recall ≥ committed baseline** (`baseline.json`, tol 0.02) — catches softer regressions. - **Vendored fixtures** (`tests/fixtures/fidelity_golden/`): deterministic `_generate.py` emits `cases.json` (4 cases: OOM crash, payment exception, latency anomaly, CI failure) + `baseline.json`. - **Non-blocking weekly report** (`.github/workflows/eval.yml`): one step in the existing `weekly-suite` job (schedule/manual only) reuses the existing `evaluate_information_retention` runner for a recall report on the production routing path. - **Pure reuse**: scoring (`evals/metrics.py`), compressor (`smart_crush_tool_output`), and the weekly runner (`evaluate_information_retention`) all already existed. ### Design notes - **Zero new CI setup.** The blocking gate runs in the existing `[dev]` test shard — no new workflow, no new deps, **no model, no network, no secrets** (verified under `HF_HUB_OFFLINE=1`). It deliberately uses small hand-made structured fixtures rather than the repo's HuggingFace dataset loaders, which would require a network download + ModernBERT and don't belong in a fast PR gate. - **Scope:** structured JSON tool-output (the dominant, deterministic, model-free case). Real-dataset (HotpotQA/BFCL) recall — which needs `[all]` + a local model — is a **documented follow-up PR**, and the `weekly-suite` job (which genuinely runs every Monday) is its natural home. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ HF_HUB_OFFLINE=1 python -m pytest tests/test_compression_fidelity_regression.py -v tests/test_compression_fidelity_regression.py::test_critical_evidence_survives_compression[logs_oom] PASSED tests/test_compression_fidelity_regression.py::test_critical_evidence_survives_compression[payment_exception] PASSED tests/test_compression_fidelity_regression.py::test_critical_evidence_survives_compression[latency_anomaly] PASSED tests/test_compression_fidelity_regression.py::test_critical_evidence_survives_compression[ci_test_failures] PASSED tests/test_compression_fidelity_regression.py::test_aggregate_recall_not_regressed PASSED ============================== 5 passed in 0.18s =============================== ``` ## Real Behavior Proof - **Environment:** local checkout of `feat/fidelity-regression-gate`, `pip install -e ".[dev]"`, `HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1` (proves no model/network). - **Exact command / steps:** `HF_HUB_OFFLINE=1 python -m pytest tests/test_compression_fidelity_regression.py -q` → `5 passed in 0.14s`. - **Negative control (proves the gate has teeth):** compressing `logs_oom` and probing for a benign row that compression legitimately drops returns `recall = 0.00, lost = ['heartbeat ping 25']` — i.e. the gate fires when critical evidence is dropped, so it is not trivially green. - **Weekly (non-blocking) step verified locally:** ```text Information retention: 50/50 cases >=0.9 recall, avg compression 65.7% ``` - **Not tested:** real-dataset (HotpotQA/BFCL) recall and prose/ModernBERT compression — intentionally deferred to a follow-up PR targeting the weekly job. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG/version intentionally untouched: repo uses **release-please**. - **Follow-up PR (planned):** wire the real HotpotQA/BFCL loaders (`headroom/evals/datasets.py`) into the `weekly-suite` job for genuine benchmark-scale recall coverage (model-allowed, non-blocking). Further follow-ups from the same design: a live per-request fidelity guardrail, query-aware lossy retention, and a hard `target_tokens` budget API. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
dependabot[bot]
|
e408012c2b
|
ci: bump dtolnay/rust-toolchain from 1.95.0 to 1.100.0 in the actions-minor-patch group (#849)
Bumps the actions-minor-patch group with 1 update:
[dtolnay/rust-toolchain](https://github.com/dtolnay/rust-toolchain).
Updates `dtolnay/rust-toolchain` from 1.95.0 to 1.100.0
<details>
<summary>Commits</summary>
<ul>
<li><a
href="
|
||
|
Ashish
|
53a08c63bf
|
feat(evals): add zero-cost tool schema compaction integrity eval (#817)
## Summary - Adds `evaluate_tool_schema_compaction()` and `generate_tool_schema_cases()` to `CompressionOnlyRunner` - Four built-in cases cover the property-name vs annotation-key distinction: `title`, `deprecated`, `readOnly`, and all four at once - Each case asserts: byte count shrinks (annotations stripped), all `must_preserve` property names survive in `properties`, no `required` entry points to a stripped key, root-level schema annotations (`$schema`, `title`) are dropped - Wires the new eval into `.github/workflows/eval.yml` alongside the existing CCR round-trip smoke step — runs on every PR touching `headroom/transforms/**`, `headroom/evals/**`, or `headroom/compress.py`, at zero API cost ## Motivation PR #785 fixed a bug where the compaction pass stripped property *names* that happened to match DROP_KEYS (e.g. a field literally called `title`). This eval encodes the invariant that fix established so future changes to the compaction logic can't silently regress it. ## Test plan - [ ] `pytest tests/test_evals_metrics.py::test_tool_schema_compaction_integrity` — all 4 cases pass, `total_tokens_saved > 0` - [ ] CI smoke step "Run tool schema compaction integrity eval (zero cost)" passes with no API key required ## Real behavior proof ``` $ pytest tests/test_evals_metrics.py::test_tool_schema_compaction_integrity -v PASSED [100%] 1 passed in 0.53s ``` Zero API calls, zero cost. Runs in under 1 second. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Patrick A
|
53d2342291
|
ci: speed up GitHub Actions — path filters, caching, timeouts, version upgrades (#620)
* ci: speed up GitHub Actions - path filters, caching, timeouts, version upgrades
Performance improvements:
- init-e2e.yml, wrap-e2e.yml: add path filters so e2e Docker builds only run when
e2e-related files change (saves ~10 min per irrelevant PR push)
- init-e2e.yml, wrap-e2e.yml: add concurrency groups to cancel superseded PR runs
- ci.yml: add pip caching to lint and build jobs
- ci.yml: cache actionlint + act binaries in workflow-validation (skip curl on hits)
- eval.yml: add pip caching to smoke-test and weekly-suite jobs
- docs.yml: add pip caching for mkdocs-material install
- rust.yml: replace cargo install --locked cargo-audit/deny with taiki-e/install-action
(prebuilt binaries; saves 2-5 min per audit run)
Bug fixes:
- docker.yml: fix actions/checkout@v6 -> @v4 (v6 does not exist; would break all
Docker builds on every release/PR touching docker paths)
Version upgrades:
- wagoid/commitlint-github-action: @v5 -> @v6
- devcontainers.yml: docker/setup-buildx-action@v3 -> @v4 (align with docker.yml)
Safety improvements:
- ci.yml: add timeout-minutes to all 13 jobs (changes, lint, build-wheel,
prefetch-model, test x4, test-extras, test-agno, commitlint, build,
workflow-validation, docker-native-e2e, windows-native-wrapper, macos-native-wrapper)
- docker.yml: add timeout-minutes to docker-build (75m), docker-manifest (20m),
promote-latest (10m)
- eval.yml: add timeout-minutes to smoke-test (30m); bump weekly-suite 60->90m
- rust.yml: add timeout-minutes to test (30m), wheels (45m), audit (20m)
Observed wall-clock impact on recent PRs:
- Init E2E and Wrap E2E were running on every single PR push regardless of content
- CI workflow was taking 12-17 min; path filters reduce unnecessary e2e runs to 0
* fix(ci): bust actionlint+act cache when workflow file changes
Static cache key 'ci-tools-actionlint-act-v1' never invalidated on
tool version updates. Switched to hashFiles('.github/workflows/ci.yml')
so the cache busts automatically whenever the download scripts are
updated to point at a newer release.
Flagged by adversarial review (Architecture + Testing/Reliability personas).
* fix(ci): add missing Dockerfile COPY paths to e2e path filters
e2e/init/Dockerfile and e2e/wrap/Dockerfile COPY files not covered
by the initial path filter set:
init-e2e: Cargo.toml, Cargo.lock, rust-toolchain.toml, uv.lock,
.claude-plugin, .github/plugin/**, plugins/headroom-agent-hooks/**
wrap-e2e: Cargo.toml, Cargo.lock, rust-toolchain.toml, uv.lock,
sdk/typescript/**, plugins/openclaw/**
Without these, a Rust toolchain bump or SDK change on a PR would
skip the e2e gate entirely, only catching it on the merge to main.
Flagged by adversarial review (Domain/Correctness persona).
* fix(devcontainer): upgrade uv to >=0.7.0 to parse uv.lock revision=3
* fix(devcontainer): set UV_SKIP_WHEEL_FILENAME_CHECK=1 in post-create.sh for gitpython wheel
* ci: bump actions/checkout and actions/setup-node to v5 (Node.js 20 EOL Jun 16)
* fix(devcontainer): export UV_SKIP_WHEEL_FILENAME_CHECK so uv run also skips wheel check
* ci: bump all GitHub Actions to latest versions (Node.js 24)
* fix(test): accept release-please-action v4 or v5 in workflow assertion
* fix(format): ruff format test_release_workflows.py
|
||
|
JerrettDavis
|
2fd48260f6 | ci: skip live evals without credentials | ||
|
chopratejas
|
a281de6bb0 |
fix(ci): skip smoke OpenAI eval cleanly when OPENAI_API_KEY secret unset
The smoke-test job in .github/workflows/eval.yml has been silently
broken on every PR that touches headroom/transforms/**, evals/**, or
compress.py. Root cause: the public chopratejas/headroom repo has zero
Actions secrets configured (`gh api repos/chopratejas/headroom/actions/secrets`
returns `{"total_count":0,"secrets":[]}`), so OPENAI_API_KEY resolves
to the empty string and `openai.OpenAI()` raises OpenAIError at client
init before any compression code runs.
Fix: gate the live OpenAI eval step on a non-empty key and emit a
GitHub `:⚠️:` annotation when skipped, so the skip is loud in
the run summary (per the project's "no silent fallbacks" rule). The
CCR round-trip step above remains the mandatory gate — it tests real
compression logic with zero external dependencies.
Operators who DO wire OPENAI_API_KEY as a repo secret get the live
eval as before.
Test plan:
- yaml syntax validated locally
- Will re-run on PR #400 push; expect smoke-test to pass with the
:⚠️: annotation visible in the run summary.
Refs: F2.1 (unblocks merge of #400)
|
||
|
chopratejas
|
2a91cbb4b4 |
refactor: single-wheel maturin build backend (fixes #355)
Eliminates the dual-package architecture that was the root cause of #355. `pip install headroom-ai` now produces ONE wheel containing both the Python source (headroom/*.py) and the compiled Rust extension (headroom/_core.so). No more separate `headroom-core-py` package, no more chicken-and-egg with PyPI publication, no more wheelhouse / PIP_FIND_LINKS / composite-action plumbing in CI. This is the canonical pattern used by cryptography, polars, ruff, pydantic-core, and other Rust-as-core Python packages. Honors the "Rust as core engine" direction. ## What changed - pyproject.toml: `[build-system]` swapped from hatchling to maturin. `[tool.hatch.*]` deleted; `[tool.maturin]` added pointing at `crates/headroom-py/Cargo.toml` for the cdylib. `python-source = "."` picks up the root `headroom/` package directly (dashboard HTML templates and other non-Python files included automatically). - crates/headroom-py/pyproject.toml: deleted. The crate is no longer a separate published package; its Cargo.toml stays as the cdylib build target invoked via `[tool.maturin] manifest-path`. - crates/headroom-py/python/: deleted (placeholder layout for the old separate package). ## CI updates - ci.yml: `test` / `test-extras` / `test-agno` jobs simplified — Rust toolchain set up before `pip install -e .` (which now invokes maturin via build-system). Removed the "build wheel + symlink .so" dance. `build` job swapped from `python -m build` (hatch) to `maturin build` + `maturin sdist`. - release.yml: collapsed dual-package matrix into one. New `build-wheels` matrix produces cross-platform wheels for cp310/11/12/13 × {linux x86_64, linux aarch64, macos x86_64, macos aarch64}. New `collect-dist` aggregator merges artifacts. publish-pypi consumes the merged dist. - init-native-e2e.yml: dropped windows-latest from the matrix — upstream `esaxx-rs` (/MT) and `ort-sys` (/MD) link with conflicting MSVC C runtime libraries, so the Rust extension cannot build for win_amd64 today. Tracked as a follow-up; not a blocker for Linux+macOS. - headroom-e2e-setup: composite action now sets up Rust toolchain + Swatinem/rust-cache before `pip install -e .[proxy]`. - eval.yml, publish.yml, rust.yml: same pattern — rust toolchain before install. rust.yml's wheels job builds from root pyproject.toml (no more `-m crates/headroom-py/Cargo.toml`). - e2e/init/Dockerfile, e2e/wrap/Dockerfile: install rust + maturin in the build stage; copy `crates/` + workspace `Cargo.toml/lock` so the install can build the extension. Dropped `HEADROOM_REQUIRE_RUST_CORE=false` from wrap-e2e — the image now ships the full Rust core. - Dockerfile (main): simplified — no more Layer 2/3 dance with `headroom-core-py` install + symlink. Single `uv pip install` builds + installs everything. - .devcontainer/Dockerfile: rust toolchain + libssl-dev + maturin added so `uv sync` builds the extension inside the devcontainer. ## Lockfile + script - uv.lock: regenerated. No `headroom-core-py` entries remain. - scripts/build_rust_extension.sh: simplified from a symlink-into-tree workaround to a thin wrapper around `pip install -e .`. The maturin build-backend handles placement automatically. ## Local validation (all green on macOS aarch64) 1. Clean venv `pip install -e .` → `from headroom._core import …` works. 2. `maturin build --release` → 13.8 MB wheel, 336 files including `headroom/_core.cpython-311-darwin.so` (32 MB cdylib) and `headroom/dashboard/templates/dashboard.html`. 3. `pip install <wheel>` in fresh venv → import works. 4. Wheel contents verified via `unzip -l`. 5. `pytest tests/test_transforms/test_diff_compressor.py` — 29 passed. 6. `pytest tests/test_relevance.py` — 30 passed. 7. `cargo build --workspace` + `cargo test --workspace` — all green. 8. `make ci-precheck` — 176 Python tests + Rust + commitlint green. ## Migration notes Users on `pip install headroom-ai` get the Rust core automatically (linux + macos wheels). sdist installs require rust toolchain available locally — pip will build via maturin. Closes #355 Supersedes #357 (workarounds-based fix abandoned in favor of architectural fix) |
||
|
chopratejas
|
beec0789ed |
fix(ci): pin dtolnay/rust-toolchain to 1.95.0 to match rust-toolchain.toml
The action was set to @stable, which installs whatever the latest stable is (1.95.0 right now). Then maturin invokes cargo, which reads rust-toolchain.toml and re-resolves to "1.95.0 + clippy + rustfmt". rustup treats stable and 1.95.0 as distinct toolchain identities and refuses the second install with: failed to install component 'clippy-preview-x86_64-unknown-linux-gnu', detected conflict: 'bin/cargo-clippy' This was intermittent across the matrix (only test (3.10) tripped on the most recent run; others got lucky on cache state). Pinning the action ref to 1.95.0 makes both sides ask for the exact same toolchain identity, so the second install is a no-op and the conflict can't fire. Bump procedure stays the same: when rust-toolchain.toml's channel changes, update these refs in lock-step. Plugin manifests auto-bumped 0.11.0 -> 0.13.2 by sync-plugin-versions hook (unrelated to the workflow fix). |
||
|
chopratejas
|
d6a00ee89c |
ci: fix smart_crusher branch CI failures + add make ci-precheck pre-push gate
Five gates broke on the 2026-04-27 push of the smart_crusher branch.
Each is fixed below; the second half adds a `make ci-precheck` target
(plus an installable git pre-push hook) so the same dance never happens
again.
Failures fixed:
1. cargo fmt — 22 files had formatting drift introduced over the
stage 3c.1 work. `cargo fmt --all` reformatted them; no semantic
changes. `cargo test --workspace` still green (388 + supporting).
2. wheels job (macOS x86_64) — `fastembed -> ort -> ort-sys` does not
publish prebuilt ONNX Runtime binaries for `x86_64-apple-darwin`.
Removed that target from `.github/workflows/rust.yml`'s wheels
matrix. Apple Silicon (`aarch64-apple-darwin`) covers macOS
distribution; Intel macOS users can build from source. The matrix
now has 2 targets: linux x86_64 + macOS aarch64.
3. test-extras (relevance.py) — `tests/test_relevance.py::TestSmartCrusherIntegration`
constructs a `SmartCrusher`, which hard-imports `headroom._core`
since the python implementation was retired in stage 3c.1b. The
test-extras job didn't build the rust extension. Added the same
`maturin build + symlink` block the main `test` job uses.
4. smoke-test (eval.yml) — same root cause:
`compression_only.evaluate_ccr_lossless` instantiates a SmartCrusher.
Same fix: build the rust extension before the smoke test runs.
5. commitlint — three rules tripped:
- `subject-case` rejects PascalCase identifiers in subjects, but
the project deliberately names classes (SmartCrusher, HfTokenizer,
ContentRouter, DiffCompressor) in commit subjects. Disabled.
- `footer-leading-blank` is a warning that the wagoid action turns
into a CI failure; lines like `Module: foo.rs` in our bodies
match the conventional footer pattern and trip it. Disabled.
- `type-enum` doesn't include `parity`, but the project ships
parity-test infrastructure as its own concern (separate from
`test:`); added `parity` to the allowed types.
Pre-push verification — the prevention half:
`make ci-precheck` runs all of the above CI gates locally:
- `ci-precheck-rust`: cargo fmt --check + clippy + test --workspace.
- `ci-precheck-python`: builds the rust extension via maturin, then
runs the smart_crusher-affected python test files (185 tests across
test_transforms/, test_relevance*, test_ccr, test_acceptance,
test_critical_fixes, test_quality_retention).
- `ci-precheck-commitlint`: `npx commitlint --from origin/main --to
HEAD` against the same config CI uses. Skipped silently if npx is
not on PATH (install Node 18+ to enable).
`make install-git-hooks` (or `scripts/install-git-hooks.sh`) installs
a git pre-push hook that runs `make ci-precheck` automatically.
Bypass with `--no-verify` only when truly needed.
When new CI gates land in `.github/workflows/`, mirror them into a
`make ci-precheck-*` target. The Makefile is the local mirror of the
CI configuration; keeping them in sync is a load-bearing invariant.
Verification: `make ci-precheck` runs green on this commit.
|
||
|
chopratejas
|
a2b2ec5463 |
Add OSS evaluation suite, universal JSON crush, latency benchmarks
Evaluation Suite: - Tiered eval framework (Tier 1 ~$3/15min, Tier 2/3 for extended coverage) - 16 benchmarks across 3 tiers: GSM8K, TruthfulQA, MMLU, ARC, HumanEval, SQuAD v2, BFCL, Tool Outputs, CCR needle retention, HotpotQA, and more - Before/After runner with full proxy support (compression + CCR retrieval) - LLM-as-judge for ground-truth comparison (BFCL function calling) - Zero-cost compression-only runner (CCR needle retention, info retention) - Cost tracker with per-model pricing and budget enforcement - Report card generator (Markdown, JSON, HTML) - Suite CLI: python -m headroom.evals suite --tier 1 - Fix BFCL dataset loader for current HuggingFace schema - CI workflow: PR smoke test + weekly full Tier 1 Results: SQuAD 97%, BFCL 97%, Tool Outputs 100%, CCR 100% SmartCrusher: - Universal JSON crush for heterogeneous arrays - Fix mypy redefinition warning in _crush_string_array Other: - Latency benchmark suite with docs - Known limitations doc - Prompt comparison evaluator - Config updates for new features |