1 commit

Author	SHA1	Message	Date
Focused Instability	c2106cbdab	feat: probe-based retention scoring of recorded compression events (#862) ... Closes #861 ## What First piece of compression quality measurement on **real proxied sessions** (vs the existing public-benchmark evals): an opt-in recorder captures (original, compressed) message pairs at each compression event, and a deterministic offline prober scores what survived. **Recorder** (`headroom/proxy/probe_recorder.py`) - `CompressionEventRecorder` implements the existing `PipelineExtension` protocol, subscribed to `INPUT_COMPRESSED`. Registered ONLY when `HEADROOM_PROBE_RECORD_DIR` is set — off means not even constructed, zero request-path overhead. - One JSONL line per compression event that changed tokens: `{ts, request_id, provider, model, tokens_before, tokens_after, transforms_applied, original_messages, compressed_messages}`. One file per PID (no interleaving), directory mode 0700. - Fail-open everywhere: construction failure logs a warning and disables recording; runtime exceptions are already swallowed by `PipelineExtensionManager.emit`. - Enabling handler change: the two `INPUT_COMPRESSED` emit sites (anthropic + openai) add a read-only `original_messages` reference to event metadata. No copies, no behavior change for other consumers. **Probes** (`headroom/evals/session_probes.py` + `headroom evals probes` CLI) - Probe targets extracted from ORIGINAL tool-result content across three dimensions: **exact numerics** (number + key context, incl. JSON-quoted keys), **artifact trail** (paths, URLs, hex hashes, UUIDs), **error evidence** (lines matching the existing `is_error_content` heuristic). - Each target classified as **retained** (verbatim, or surviving a legitimate format conversion — punctuation-normalized match; numerics require key AND value to survive; error lines tolerate dropped JSON key prefixes), **recoverable** (absent but a CCR retrieval marker is present), or **lost** (gone with no retrieval path). - Report: aggregate retention per dimension, bucketed by compression ratio (the quality-per-ratio curve), and grouped per transform. `--json-output` for machine-readable results. - Fully offline: no LLM, no API key. The recording format is designed to feed an LLM-judge pass later (out of scope per #861). ## Tests 33 new tests (red before, green after): `tests/test_probe_recorder.py` (11 — event filtering, JSONL shape, env activation, fail-open on unusable path, 0700 dir mode) and `tests/test_session_probes.py` (22 — extraction per dimension incl. JSON-quoted numerics, retained/recoverable/lost classification, format-change survival for numerics and error lines, ratio bucketing, transform dedup, malformed-line skipping, report rendering/serialization). Both proxy lifecycle tests additionally assert the INPUT_COMPRESSED `original_messages` metadata contract end-to-end through the real anthropic/openai handlers, so a refactor cannot silently disable the recorder. Local runs: new tests + `tests/test_proxy_pipeline_lifecycle.py` + `tests/test_canonical_pipeline.py` + `tests/test_pipeline.py` — 45 passed. `ruff check` + `uvx ruff format --check` clean. ### Self-review hardening (second commit) - Hex artifact regex now requires at least one `a-f`, so bare decimal runs (timestamps, counters) no longer inflate the artifact dimension. - Inflation events (ratio > 1, the #847 territory) get an explicit `1.00+ (inflated)` ratio bucket instead of silently dropping out of the bucketed view. - `run_probes` streams recording files line by line instead of slurping them. - Documented honestly: marker recoverability is event-scoped (comparative metric, not absolute); recorder writes synchronously on the request path (diagnostic sessions, not always-on). - `headroom/evals/README.md` gained a Session Probes usage section. ## Real behavior proof **Setup:** macOS (Darwin 25.5), Python 3.11.9, this branch, real proxy (`python -m headroom.proxy.server --port 18994 --anthropic-api-url http://127.0.0.1:18995`) with a local mock Anthropic upstream (no key), `HEADROOM_PROBE_RECORD_DIR=/tmp/headroom-probe-proof/recordings`. **Steps:** POSTed three Anthropic-format conversations whose tool results carry large JSON arrays (220-row uniform logs, 3000-row logs, 800 heterogeneous events), each containing known numerics, paths, trace hashes, and one error line. **Observed** — recorder wrote `compression-events-<pid>.jsonl` (one line per event); `headroom evals probes --recordings ...`: ``` Probed 3 compression events Aggregate retention: numerics 97.7% retained, 2.3% recoverable, 0.0% lost (527 targets) artifacts 100.0% retained, 0.0% recoverable, 0.0% lost (6555 targets) errors 100.0% retained, 0.0% recoverable, 0.0% lost (3 targets) By compression ratio (tokens_after / tokens_before): ratio 0.50-0.75: numerics 100.0% retained (CSV compaction — lossless, correctly recognized) ratio 0.75-1.00: numerics 95.7% retained, 4.3% recoverable (SmartCrusher sampling — dropped values carried a CCR marker) ``` All three classifications exercised: verbatim/format-change retention on the CSV-compacted events, **recoverable** on the heterogeneous event where SmartCrusher sampled rows out behind a `Retrieve more: hash=` marker, and the injected error lines retained in every event (the error-protection gate held). The `lost` path is covered by unit tests. The first iteration of this proof exposed two real bugs — naive verbatim matching misreported lossless JSON→CSV compaction as 100% lost, and duplicated transform markers double-counted tallies — both fixed with regression tests. **Not tested live:** Gemini path (no `INPUT_COMPRESSED` emit parity — pre-existing, same gap as #819); LLM-judge scoring (out of scope per #861). ## Security Recordings contain full conversation content in plaintext: opt-in env var only, local disk only, dir mode 0700, documented in CLI help. ## Out of scope (per #861) LLM-judge dimensions (decisions/intent, next steps), ACON-style counterfactual replay, automatic rule revision. --------- Co-authored-by: Ash Rhodes <ashley.rhodes@king.com>	2026-06-11 13:02:36 -05:00