mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
c2106cbdab
Closes #861 ## What First piece of compression quality measurement on **real proxied sessions** (vs the existing public-benchmark evals): an opt-in recorder captures (original, compressed) message pairs at each compression event, and a deterministic offline prober scores what survived. **Recorder** (`headroom/proxy/probe_recorder.py`) - `CompressionEventRecorder` implements the existing `PipelineExtension` protocol, subscribed to `INPUT_COMPRESSED`. Registered ONLY when `HEADROOM_PROBE_RECORD_DIR` is set — off means not even constructed, zero request-path overhead. - One JSONL line per compression event that changed tokens: `{ts, request_id, provider, model, tokens_before, tokens_after, transforms_applied, original_messages, compressed_messages}`. One file per PID (no interleaving), directory mode 0700. - Fail-open everywhere: construction failure logs a warning and disables recording; runtime exceptions are already swallowed by `PipelineExtensionManager.emit`. - Enabling handler change: the two `INPUT_COMPRESSED` emit sites (anthropic + openai) add a read-only `original_messages` reference to event metadata. No copies, no behavior change for other consumers. **Probes** (`headroom/evals/session_probes.py` + `headroom evals probes` CLI) - Probe targets extracted from ORIGINAL tool-result content across three dimensions: **exact numerics** (number + key context, incl. JSON-quoted keys), **artifact trail** (paths, URLs, hex hashes, UUIDs), **error evidence** (lines matching the existing `is_error_content` heuristic). - Each target classified as **retained** (verbatim, or surviving a legitimate format conversion — punctuation-normalized match; numerics require key AND value to survive; error lines tolerate dropped JSON key prefixes), **recoverable** (absent but a CCR retrieval marker is present), or **lost** (gone with no retrieval path). - Report: aggregate retention per dimension, bucketed by compression ratio (the quality-per-ratio curve), and grouped per transform. `--json-output` for machine-readable results. - Fully offline: no LLM, no API key. The recording format is designed to feed an LLM-judge pass later (out of scope per #861). ## Tests 33 new tests (red before, green after): `tests/test_probe_recorder.py` (11 — event filtering, JSONL shape, env activation, fail-open on unusable path, 0700 dir mode) and `tests/test_session_probes.py` (22 — extraction per dimension incl. JSON-quoted numerics, retained/recoverable/lost classification, format-change survival for numerics and error lines, ratio bucketing, transform dedup, malformed-line skipping, report rendering/serialization). Both proxy lifecycle tests additionally assert the INPUT_COMPRESSED `original_messages` metadata contract end-to-end through the real anthropic/openai handlers, so a refactor cannot silently disable the recorder. Local runs: new tests + `tests/test_proxy_pipeline_lifecycle.py` + `tests/test_canonical_pipeline.py` + `tests/test_pipeline.py` — 45 passed. `ruff check` + `uvx ruff format --check` clean. ### Self-review hardening (second commit) - Hex artifact regex now requires at least one `a-f`, so bare decimal runs (timestamps, counters) no longer inflate the artifact dimension. - Inflation events (ratio > 1, the #847 territory) get an explicit `1.00+ (inflated)` ratio bucket instead of silently dropping out of the bucketed view. - `run_probes` streams recording files line by line instead of slurping them. - Documented honestly: marker recoverability is event-scoped (comparative metric, not absolute); recorder writes synchronously on the request path (diagnostic sessions, not always-on). - `headroom/evals/README.md` gained a Session Probes usage section. ## Real behavior proof **Setup:** macOS (Darwin 25.5), Python 3.11.9, this branch, real proxy (`python -m headroom.proxy.server --port 18994 --anthropic-api-url http://127.0.0.1:18995`) with a local mock Anthropic upstream (no key), `HEADROOM_PROBE_RECORD_DIR=/tmp/headroom-probe-proof/recordings`. **Steps:** POSTed three Anthropic-format conversations whose tool results carry large JSON arrays (220-row uniform logs, 3000-row logs, 800 heterogeneous events), each containing known numerics, paths, trace hashes, and one error line. **Observed** — recorder wrote `compression-events-<pid>.jsonl` (one line per event); `headroom evals probes --recordings ...`: ``` Probed 3 compression events Aggregate retention: numerics 97.7% retained, 2.3% recoverable, 0.0% lost (527 targets) artifacts 100.0% retained, 0.0% recoverable, 0.0% lost (6555 targets) errors 100.0% retained, 0.0% recoverable, 0.0% lost (3 targets) By compression ratio (tokens_after / tokens_before): ratio 0.50-0.75: numerics 100.0% retained (CSV compaction — lossless, correctly recognized) ratio 0.75-1.00: numerics 95.7% retained, 4.3% recoverable (SmartCrusher sampling — dropped values carried a CCR marker) ``` All three classifications exercised: verbatim/format-change retention on the CSV-compacted events, **recoverable** on the heterogeneous event where SmartCrusher sampled rows out behind a `Retrieve more: hash=` marker, and the injected error lines retained in every event (the error-protection gate held). The `lost` path is covered by unit tests. The first iteration of this proof exposed two real bugs — naive verbatim matching misreported lossless JSON→CSV compaction as 100% lost, and duplicated transform markers double-counted tallies — both fixed with regression tests. **Not tested live:** Gemini path (no `INPUT_COMPRESSED` emit parity — pre-existing, same gap as #819); LLM-judge scoring (out of scope per #861). ## Security Recordings contain full conversation content in plaintext: opt-in env var only, local disk only, dir mode 0700, documented in CLI help. ## Out of scope (per #861) LLM-judge dimensions (decisions/intent, next steps), ACON-style counterfactual replay, automatic rule revision. --------- Co-authored-by: Ash Rhodes <ashley.rhodes@king.com>
139 lines
4.2 KiB
Python
139 lines
4.2 KiB
Python
"""Tests for the opt-in compression event probe recorder."""
|
|
|
|
import json
|
|
import stat
|
|
|
|
from headroom.pipeline import PipelineEvent, PipelineStage
|
|
from headroom.proxy.probe_recorder import (
|
|
RECORD_DIR_ENV,
|
|
CompressionEventRecorder,
|
|
probe_recorder_from_env,
|
|
)
|
|
|
|
ORIGINAL = [
|
|
{
|
|
"role": "user",
|
|
"content": [{"type": "tool_result", "tool_use_id": "t1", "content": "retry_limit: 3"}],
|
|
}
|
|
]
|
|
COMPRESSED = [
|
|
{
|
|
"role": "user",
|
|
"content": [{"type": "tool_result", "tool_use_id": "t1", "content": "[compressed]"}],
|
|
}
|
|
]
|
|
|
|
|
|
def _metadata(**overrides):
|
|
metadata = {
|
|
"tokens_before": 100,
|
|
"tokens_after": 40,
|
|
"transforms_applied": ["smart_crusher"],
|
|
"original_messages": ORIGINAL,
|
|
}
|
|
metadata.update(overrides)
|
|
return metadata
|
|
|
|
|
|
def _event(stage=PipelineStage.INPUT_COMPRESSED, messages=COMPRESSED, metadata=None):
|
|
return PipelineEvent(
|
|
stage=stage,
|
|
operation="proxy.request",
|
|
request_id="req-1",
|
|
provider="anthropic",
|
|
model="claude-test",
|
|
messages=messages,
|
|
metadata=_metadata() if metadata is None else metadata,
|
|
)
|
|
|
|
|
|
class TestCompressionEventRecorder:
|
|
def test_records_compression_event(self, tmp_path):
|
|
recorder = CompressionEventRecorder(tmp_path)
|
|
|
|
recorder.on_pipeline_event(_event())
|
|
|
|
lines = recorder.path.read_text(encoding="utf-8").splitlines()
|
|
assert len(lines) == 1
|
|
record = json.loads(lines[0])
|
|
assert record["request_id"] == "req-1"
|
|
assert record["provider"] == "anthropic"
|
|
assert record["model"] == "claude-test"
|
|
assert record["tokens_before"] == 100
|
|
assert record["tokens_after"] == 40
|
|
assert record["transforms_applied"] == ["smart_crusher"]
|
|
assert record["original_messages"] == ORIGINAL
|
|
assert record["compressed_messages"] == COMPRESSED
|
|
assert record["ts"] > 0
|
|
|
|
def test_appends_one_line_per_event(self, tmp_path):
|
|
recorder = CompressionEventRecorder(tmp_path)
|
|
|
|
recorder.on_pipeline_event(_event())
|
|
recorder.on_pipeline_event(_event())
|
|
|
|
assert len(recorder.path.read_text(encoding="utf-8").splitlines()) == 2
|
|
|
|
def test_ignores_other_stages(self, tmp_path):
|
|
recorder = CompressionEventRecorder(tmp_path)
|
|
|
|
recorder.on_pipeline_event(_event(stage=PipelineStage.INPUT_ROUTED))
|
|
|
|
assert not recorder.path.exists()
|
|
|
|
def test_skips_without_original_messages(self, tmp_path):
|
|
recorder = CompressionEventRecorder(tmp_path)
|
|
|
|
recorder.on_pipeline_event(_event(metadata=_metadata(original_messages=None)))
|
|
|
|
assert not recorder.path.exists()
|
|
|
|
def test_skips_zero_token_delta(self, tmp_path):
|
|
recorder = CompressionEventRecorder(tmp_path)
|
|
|
|
recorder.on_pipeline_event(_event(metadata=_metadata(tokens_after=100)))
|
|
|
|
assert not recorder.path.exists()
|
|
|
|
def test_skips_without_compressed_messages(self, tmp_path):
|
|
recorder = CompressionEventRecorder(tmp_path)
|
|
|
|
recorder.on_pipeline_event(_event(messages=None))
|
|
|
|
assert not recorder.path.exists()
|
|
|
|
def test_record_dir_is_private(self, tmp_path):
|
|
record_dir = tmp_path / "recordings"
|
|
|
|
CompressionEventRecorder(record_dir)
|
|
|
|
mode = stat.S_IMODE(record_dir.stat().st_mode)
|
|
assert mode == 0o700
|
|
|
|
|
|
class TestProbeRecorderFromEnv:
|
|
def test_disabled_without_env(self, monkeypatch):
|
|
monkeypatch.delenv(RECORD_DIR_ENV, raising=False)
|
|
|
|
assert probe_recorder_from_env() is None
|
|
|
|
def test_disabled_with_blank_env(self, monkeypatch):
|
|
monkeypatch.setenv(RECORD_DIR_ENV, " ")
|
|
|
|
assert probe_recorder_from_env() is None
|
|
|
|
def test_enabled_with_env(self, tmp_path, monkeypatch):
|
|
record_dir = tmp_path / "recordings"
|
|
monkeypatch.setenv(RECORD_DIR_ENV, str(record_dir))
|
|
|
|
recorder = probe_recorder_from_env()
|
|
|
|
assert isinstance(recorder, CompressionEventRecorder)
|
|
assert record_dir.is_dir()
|
|
|
|
def test_fail_open_on_unusable_path(self, tmp_path, monkeypatch):
|
|
blocker = tmp_path / "not-a-dir"
|
|
blocker.write_text("file", encoding="utf-8")
|
|
monkeypatch.setenv(RECORD_DIR_ENV, str(blocker / "recordings"))
|
|
|
|
assert probe_recorder_from_env() is None
|