headroom/tests/test_verbosity_controller.py

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

99 lines
3.4 KiB
Python
Raw Permalink Normal View History

feat: output-token reduction — verbosity shaper, per-user learning, counterfactual savings (#965) ## Description Adds the first levers that reduce the tokens the model **writes back** (output), complementing Headroom's existing input compression. Output costs 5× input on Opus-class models and is full of waste (ceremony, restated code, deep "thinking" on routine steps). Two phases in one self-contained PR off `main`: the request-side output shaper, then per-user verbosity learning plus an honest counterfactual savings estimator and dashboard surfacing. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Output shaper** (`output_shaper.py`, opt-in `HEADROOM_OUTPUT_SHAPER=1`): cache-safe verbosity steering appended to the system-prompt tail (5 levels); effort routing that lowers `output_config.effort` on mechanical tool-result continuations; legacy `thinking.budget_tokens` clamp. Never injects effort where absent, never toggles `thinking.type`. - **`headroom learn --verbosity`**: mines Claude Code transcripts for behavioral signals (interrupts, length-adaptive fast-skips, echo ratio), recommends a verbosity level (heuristic + optional `--llm-judge`), and seeds the savings baseline. - **Counterfactual estimator** (`output_savings.py`): per-stratum synthetic-control (estimated) + A/B holdout (measured) with a propagated 95% CI; conversation-stable arm assignment for A/B validity and prefix-cache safety. - **AIMD verbosity controller** (`verbosity_controller.py`): additive-increase / fast-back-off state machine; live signal emission gated off by default. - **Wiring + surfaces**: shaper resolves the learned level; recording rides the existing `transforms_applied` channel through the outcome funnel (no `RequestOutcome` changes); `headroom output-savings` CLI; dashboard "Output Tokens Saved" card. - **Docs**: simple-words user guide + design doc with the counterfactual methodology. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_output_savings.py tests/test_output_savings_cli.py \ tests/test_verbosity_learn.py tests/test_verbosity_controller.py \ tests/test_output_shaper.py -q 94 passed in 0.54s $ pytest tests/test_request_outcome.py tests/test_handler_outcome_tag_invariant.py \ tests/test_proxy_dashboard_stats_cache.py -q 44 passed $ ruff format --check . 831 files already formatted $ mypy headroom --ignore-missing-imports Success: no issues found in 361 source files ``` ## Real Behavior Proof - Environment: macOS, Python 3.12 (`.venv`), `anthropic` 0.76, live API model `claude-opus-4-8`. - Exact command / steps: `HEADROOM_OUTPUT_SHAPER=1`; `headroom learn --verbosity --apply` (seeds level + baseline); `python scripts/eval_output_shaper.py A` (live before/after); simulate holdout traffic then `headroom output-savings`. - Observed result: code-review ask — baseline 1,750 output tokens → L2 1,354 (−22.7%) → L3 599 (−65.8%), same bugs found. `learn --verbosity` on 24 real sessions → 11% interrupt / 26% fast-skip → L3 (high confidence). Measured A/B path → 31.7% reduction (95% CI 27.7%–35.7%). 94 new tests + 44 existing outcome/dashboard tests green; ruff + mypy clean. - Not tested: live streaming-path recording exercised only via unit tests (the `transforms_applied` funnel is shared across paths); runtime AIMD signal emission is gated off by default and not exercised live. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes Output savings are counterfactual (we never observe what the model *would* have written), so the estimator separates **estimated** (vs a learned baseline) from **measured** (A/B holdout via `HEADROOM_OUTPUT_HOLDOUT`) and always reports a confidence band — never a single made-up number. CHANGELOG left unchecked (release-please manages it). Runtime AIMD self-tuning is intentionally a TODO (controller built/tested; live signal emission gated behind `HEADROOM_VERBOSITY_AUTOTUNE`).
2026-06-16 21:06:43 -07:00
"""Tests for headroom.proxy.verbosity_controller — the AIMD state machine."""
from __future__ import annotations
from headroom.proxy.verbosity_controller import (
ControllerState,
Signal,
VerbosityController,
load_state,
save_state,
)
CTRL = VerbosityController(floor=1, ceil=4, probe_threshold=3, cooldown_turns=5)
def _run(signals, start=2):
state = ControllerState(level=start)
for s in signals:
state = CTRL.observe(state, s)
return state
class TestAdditiveIncrease:
def test_steps_up_only_after_threshold(self):
state = _run([Signal.TOO_MUCH, Signal.TOO_MUCH])
assert state.level == 2 # not yet at threshold
assert state.up_streak == 2
def test_steps_up_at_threshold(self):
state = _run([Signal.TOO_MUCH] * 3)
assert state.level == 3
assert state.up_streak == 0 # reset after stepping
def test_neutral_breaks_the_streak(self):
state = _run([Signal.TOO_MUCH, Signal.TOO_MUCH, Signal.NEUTRAL, Signal.TOO_MUCH])
assert state.level == 2 # streak was broken; only 1 consecutive at end
assert state.up_streak == 1
def test_does_not_exceed_ceiling(self):
state = _run([Signal.TOO_MUCH] * 30, start=4)
assert state.level == 4
class TestMultiplicativeDecrease:
def test_too_little_backs_off_immediately(self):
state = _run([Signal.TOO_LITTLE], start=3)
assert state.level == 2
assert state.cooldown == 5
def test_does_not_go_below_floor(self):
state = _run([Signal.TOO_LITTLE] * 10, start=2)
assert state.level == 1
def test_cooldown_suppresses_reescalation(self):
# Back off, then immediately get TOO_MUCH pressure — must not re-terse
# until the cooldown elapses.
state = ControllerState(level=3)
state = CTRL.observe(state, Signal.TOO_LITTLE) # → level 2, cooldown 5
assert state.level == 2
for _ in range(3): # would normally step up at 3, but we're cooling down
state = CTRL.observe(state, Signal.TOO_MUCH)
assert state.level == 2 # held
def test_reescalates_after_cooldown_expires(self):
state = ControllerState(level=2, cooldown=5)
# 5 neutral turns drain the cooldown...
for _ in range(5):
state = CTRL.observe(state, Signal.NEUTRAL)
assert state.cooldown == 0
# ...then sustained pressure can step up again.
for _ in range(3):
state = CTRL.observe(state, Signal.TOO_MUCH)
assert state.level == 3
class TestPersistence:
def test_roundtrip(self, tmp_path):
path = tmp_path / "ctrl.json"
save_state(path, ControllerState(level=3, up_streak=2, cooldown=1))
state = load_state(path, default_level=2, floor=1, ceil=4)
assert state.level == 3
assert state.up_streak == 2
def test_missing_uses_default(self, tmp_path):
state = load_state(tmp_path / "nope.json", default_level=2, floor=1, ceil=4)
assert state.level == 2
def test_corrupt_uses_default(self, tmp_path):
p = tmp_path / "bad.json"
p.write_text("{broken")
state = load_state(p, default_level=3, floor=1, ceil=4)
assert state.level == 3
def test_loaded_level_clamped(self, tmp_path):
p = tmp_path / "ctrl.json"
save_state(p, ControllerState(level=9))
state = load_state(p, default_level=2, floor=1, ceil=4)
assert state.level == 4