headroom/tests/test_verbosity_controller.py
Tejas Chopra a99dc61424
feat: output-token reduction — verbosity shaper, per-user learning, counterfactual savings (#965)
## Description

Adds the first levers that reduce the tokens the model **writes back**
(output), complementing Headroom's existing input compression. Output
costs 5× input on Opus-class models and is full of waste (ceremony,
restated code, deep "thinking" on routine steps). Two phases in one
self-contained PR off `main`: the request-side output shaper, then
per-user verbosity learning plus an honest counterfactual savings
estimator and dashboard surfacing.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **Output shaper** (`output_shaper.py`, opt-in
`HEADROOM_OUTPUT_SHAPER=1`): cache-safe verbosity steering appended to
the system-prompt tail (5 levels); effort routing that lowers
`output_config.effort` on mechanical tool-result continuations; legacy
`thinking.budget_tokens` clamp. Never injects effort where absent, never
toggles `thinking.type`.
- **`headroom learn --verbosity`**: mines Claude Code transcripts for
behavioral signals (interrupts, length-adaptive fast-skips, echo ratio),
recommends a verbosity level (heuristic + optional `--llm-judge`), and
seeds the savings baseline.
- **Counterfactual estimator** (`output_savings.py`): per-stratum
synthetic-control (estimated) + A/B holdout (measured) with a propagated
95% CI; conversation-stable arm assignment for A/B validity and
prefix-cache safety.
- **AIMD verbosity controller** (`verbosity_controller.py`):
additive-increase / fast-back-off state machine; live signal emission
gated off by default.
- **Wiring + surfaces**: shaper resolves the learned level; recording
rides the existing `transforms_applied` channel through the outcome
funnel (no `RequestOutcome` changes); `headroom output-savings` CLI;
dashboard "Output Tokens Saved" card.
- **Docs**: simple-words user guide + design doc with the counterfactual
methodology.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_output_savings.py tests/test_output_savings_cli.py \
        tests/test_verbosity_learn.py tests/test_verbosity_controller.py \
        tests/test_output_shaper.py -q
94 passed in 0.54s

$ pytest tests/test_request_outcome.py tests/test_handler_outcome_tag_invariant.py \
        tests/test_proxy_dashboard_stats_cache.py -q
44 passed

$ ruff format --check .
831 files already formatted

$ mypy headroom --ignore-missing-imports
Success: no issues found in 361 source files
```

## Real Behavior Proof

- Environment: macOS, Python 3.12 (`.venv`), `anthropic` 0.76, live API
model `claude-opus-4-8`.
- Exact command / steps: `HEADROOM_OUTPUT_SHAPER=1`; `headroom learn
--verbosity --apply` (seeds level + baseline); `python
scripts/eval_output_shaper.py A` (live before/after); simulate holdout
traffic then `headroom output-savings`.
- Observed result: code-review ask — baseline 1,750 output tokens → L2
1,354 (−22.7%) → L3 599 (−65.8%), same bugs found. `learn --verbosity`
on 24 real sessions → 11% interrupt / 26% fast-skip → L3 (high
confidence). Measured A/B path → 31.7% reduction (95% CI 27.7%–35.7%).
94 new tests + 44 existing outcome/dashboard tests green; ruff + mypy
clean.
- Not tested: live streaming-path recording exercised only via unit
tests (the `transforms_applied` funnel is shared across paths); runtime
AIMD signal emission is gated off by default and not exercised live.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

Output savings are counterfactual (we never observe what the model
*would* have written), so the estimator separates **estimated** (vs a
learned baseline) from **measured** (A/B holdout via
`HEADROOM_OUTPUT_HOLDOUT`) and always reports a confidence band — never
a single made-up number. CHANGELOG left unchecked (release-please
manages it). Runtime AIMD self-tuning is intentionally a TODO
(controller built/tested; live signal emission gated behind
`HEADROOM_VERBOSITY_AUTOTUNE`).
2026-06-16 21:06:43 -07:00

98 lines
3.4 KiB
Python

"""Tests for headroom.proxy.verbosity_controller — the AIMD state machine."""
from __future__ import annotations
from headroom.proxy.verbosity_controller import (
ControllerState,
Signal,
VerbosityController,
load_state,
save_state,
)
CTRL = VerbosityController(floor=1, ceil=4, probe_threshold=3, cooldown_turns=5)
def _run(signals, start=2):
state = ControllerState(level=start)
for s in signals:
state = CTRL.observe(state, s)
return state
class TestAdditiveIncrease:
def test_steps_up_only_after_threshold(self):
state = _run([Signal.TOO_MUCH, Signal.TOO_MUCH])
assert state.level == 2 # not yet at threshold
assert state.up_streak == 2
def test_steps_up_at_threshold(self):
state = _run([Signal.TOO_MUCH] * 3)
assert state.level == 3
assert state.up_streak == 0 # reset after stepping
def test_neutral_breaks_the_streak(self):
state = _run([Signal.TOO_MUCH, Signal.TOO_MUCH, Signal.NEUTRAL, Signal.TOO_MUCH])
assert state.level == 2 # streak was broken; only 1 consecutive at end
assert state.up_streak == 1
def test_does_not_exceed_ceiling(self):
state = _run([Signal.TOO_MUCH] * 30, start=4)
assert state.level == 4
class TestMultiplicativeDecrease:
def test_too_little_backs_off_immediately(self):
state = _run([Signal.TOO_LITTLE], start=3)
assert state.level == 2
assert state.cooldown == 5
def test_does_not_go_below_floor(self):
state = _run([Signal.TOO_LITTLE] * 10, start=2)
assert state.level == 1
def test_cooldown_suppresses_reescalation(self):
# Back off, then immediately get TOO_MUCH pressure — must not re-terse
# until the cooldown elapses.
state = ControllerState(level=3)
state = CTRL.observe(state, Signal.TOO_LITTLE) # → level 2, cooldown 5
assert state.level == 2
for _ in range(3): # would normally step up at 3, but we're cooling down
state = CTRL.observe(state, Signal.TOO_MUCH)
assert state.level == 2 # held
def test_reescalates_after_cooldown_expires(self):
state = ControllerState(level=2, cooldown=5)
# 5 neutral turns drain the cooldown...
for _ in range(5):
state = CTRL.observe(state, Signal.NEUTRAL)
assert state.cooldown == 0
# ...then sustained pressure can step up again.
for _ in range(3):
state = CTRL.observe(state, Signal.TOO_MUCH)
assert state.level == 3
class TestPersistence:
def test_roundtrip(self, tmp_path):
path = tmp_path / "ctrl.json"
save_state(path, ControllerState(level=3, up_streak=2, cooldown=1))
state = load_state(path, default_level=2, floor=1, ceil=4)
assert state.level == 3
assert state.up_streak == 2
def test_missing_uses_default(self, tmp_path):
state = load_state(tmp_path / "nope.json", default_level=2, floor=1, ceil=4)
assert state.level == 2
def test_corrupt_uses_default(self, tmp_path):
p = tmp_path / "bad.json"
p.write_text("{broken")
state = load_state(p, default_level=3, floor=1, ceil=4)
assert state.level == 3
def test_loaded_level_clamped(self, tmp_path):
p = tmp_path / "ctrl.json"
save_state(p, ControllerState(level=9))
state = load_state(p, default_level=2, floor=1, ceil=4)
assert state.level == 4