mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
b51cda10d7
## Description Follow-up to #862. That PR's body described a **Session Probes** section in `headroom/evals/README.md`, but the file edit missed the commit (edited in the wrong checkout). This adds the missing 22-line docs-only section: the record-then-score workflow for `HEADROOM_PROBE_RECORD_DIR` + `headroom evals probes`, including the plaintext-recording privacy note. Refs #861 (session-probe eval harness — this README section was part of that feature's spec). ## Type of Change - [x] Documentation update ## Changes Made - Add a **Session Probes (real recorded sessions)** section to `headroom/evals/README.md` (+22 lines, no code change): the two-step record (`HEADROOM_PROBE_RECORD_DIR=… headroom proxy start`) then score (`headroom evals probes --recordings …`) workflow, the three probe dimensions (exact numerics, artifact trail, error evidence), the retained/recoverable/lost classification, retention bucketing by ratio + per-transform grouping, and the `--json-output` flag. - Includes the opt-in privacy note: recordings contain full conversation content in plaintext and stay on the local machine. ## Testing - [x] Documentation builds/renders correctly - [x] Linting passes (`ruff check .`) - [x] New and existing unit tests pass locally with my changes ### Test Output ```text $ git diff --stat upstream/main..HEAD headroom/evals/README.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) Docs-only change — no code paths touched. The commands and flags documented (HEADROOM_PROBE_RECORD_DIR, `headroom evals probes`, --recordings, --json-output) are the surface shipped and tested in #862. ``` ## Real Behavior Proof - Environment: local macOS, repo .venv, Python 3.11.9 - Exact command / steps: rendered the edited `headroom/evals/README.md` and cross-checked every documented flag/command against the implemented CLI from #862 (`headroom evals probes`, `HEADROOM_PROBE_RECORD_DIR`, `--recordings`, `--json-output`) - Observed result: the new section renders correctly and every command/flag it names exists in the shipped probe harness; no code paths are changed by this PR, so behavior is unchanged - Not tested: nothing additional — docs-only change with no executable surface of its own ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes Pure documentation backfill for #862; the feature itself (recorder + retention probes) already merged. PR body updated to satisfy the PR-governance template gate. |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| test_anthropic.py | ||
| test_backends.py | ||
| test_base.py | ||
| test_client_integration.py | ||
| test_dynamic_detector.py | ||
| test_google.py | ||
| test_openai.py | ||
| test_prefix_tracker.py | ||
| test_registry.py | ||
| test_semantic.py | ||