mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
a99dc61424
## Description
Adds the first levers that reduce the tokens the model **writes back**
(output), complementing Headroom's existing input compression. Output
costs 5× input on Opus-class models and is full of waste (ceremony,
restated code, deep "thinking" on routine steps). Two phases in one
self-contained PR off `main`: the request-side output shaper, then
per-user verbosity learning plus an honest counterfactual savings
estimator and dashboard surfacing.
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- **Output shaper** (`output_shaper.py`, opt-in
`HEADROOM_OUTPUT_SHAPER=1`): cache-safe verbosity steering appended to
the system-prompt tail (5 levels); effort routing that lowers
`output_config.effort` on mechanical tool-result continuations; legacy
`thinking.budget_tokens` clamp. Never injects effort where absent, never
toggles `thinking.type`.
- **`headroom learn --verbosity`**: mines Claude Code transcripts for
behavioral signals (interrupts, length-adaptive fast-skips, echo ratio),
recommends a verbosity level (heuristic + optional `--llm-judge`), and
seeds the savings baseline.
- **Counterfactual estimator** (`output_savings.py`): per-stratum
synthetic-control (estimated) + A/B holdout (measured) with a propagated
95% CI; conversation-stable arm assignment for A/B validity and
prefix-cache safety.
- **AIMD verbosity controller** (`verbosity_controller.py`):
additive-increase / fast-back-off state machine; live signal emission
gated off by default.
- **Wiring + surfaces**: shaper resolves the learned level; recording
rides the existing `transforms_applied` channel through the outcome
funnel (no `RequestOutcome` changes); `headroom output-savings` CLI;
dashboard "Output Tokens Saved" card.
- **Docs**: simple-words user guide + design doc with the counterfactual
methodology.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_output_savings.py tests/test_output_savings_cli.py \
tests/test_verbosity_learn.py tests/test_verbosity_controller.py \
tests/test_output_shaper.py -q
94 passed in 0.54s
$ pytest tests/test_request_outcome.py tests/test_handler_outcome_tag_invariant.py \
tests/test_proxy_dashboard_stats_cache.py -q
44 passed
$ ruff format --check .
831 files already formatted
$ mypy headroom --ignore-missing-imports
Success: no issues found in 361 source files
```
## Real Behavior Proof
- Environment: macOS, Python 3.12 (`.venv`), `anthropic` 0.76, live API
model `claude-opus-4-8`.
- Exact command / steps: `HEADROOM_OUTPUT_SHAPER=1`; `headroom learn
--verbosity --apply` (seeds level + baseline); `python
scripts/eval_output_shaper.py A` (live before/after); simulate holdout
traffic then `headroom output-savings`.
- Observed result: code-review ask — baseline 1,750 output tokens → L2
1,354 (−22.7%) → L3 599 (−65.8%), same bugs found. `learn --verbosity`
on 24 real sessions → 11% interrupt / 26% fast-skip → L3 (high
confidence). Measured A/B path → 31.7% reduction (95% CI 27.7%–35.7%).
94 new tests + 44 existing outcome/dashboard tests green; ruff + mypy
clean.
- Not tested: live streaming-path recording exercised only via unit
tests (the `transforms_applied` funnel is shared across paths); runtime
AIMD signal emission is gated off by default and not exercised live.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
Output savings are counterfactual (we never observe what the model
*would* have written), so the estimator separates **estimated** (vs a
learned baseline) from **measured** (A/B holdout via
`HEADROOM_OUTPUT_HOLDOUT`) and always reports a confidence band — never
a single made-up number. CHANGELOG left unchecked (release-please
manages it). Runtime AIMD self-tuning is intentionally a TODO
(controller built/tested; live signal emission gated behind
`HEADROOM_VERBOSITY_AUTOTUNE`).
53 lines
1.9 KiB
Python
53 lines
1.9 KiB
Python
"""Smoke tests for the output-savings CLI and the outcome→ledger wiring."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
|
|
from click.testing import CliRunner
|
|
|
|
from headroom.cli.main import main
|
|
from headroom.proxy.output_savings import SavingsRecorder, stratum_label
|
|
|
|
|
|
def test_output_savings_empty(tmp_path, monkeypatch):
|
|
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(tmp_path))
|
|
result = CliRunner().invoke(main, ["output-savings"])
|
|
assert result.exit_code == 0
|
|
assert "No output-savings data" in result.output
|
|
|
|
|
|
def test_output_savings_reports_estimate(tmp_path, monkeypatch):
|
|
monkeypatch.setenv("HEADROOM_WORKSPACE_DIR", str(tmp_path))
|
|
# Seed a baseline + treatment observations directly via the ledger.
|
|
from headroom.proxy.output_savings import SavingsLedger
|
|
|
|
ledger = SavingsLedger()
|
|
for _ in range(50):
|
|
ledger.baseline.observe("opus|new_user_ask|s|tools", 1000)
|
|
for _ in range(30):
|
|
ledger.record("treatment", "opus|new_user_ask|s|tools", 700)
|
|
ledger.save(tmp_path / "output_savings.json")
|
|
|
|
result = CliRunner().invoke(main, ["output-savings"])
|
|
assert result.exit_code == 0
|
|
assert "ESTIMATED" in result.output
|
|
assert "Reduction:" in result.output
|
|
assert "30.0%" in result.output
|
|
|
|
|
|
def test_recorder_round_trips_via_labels(tmp_path):
|
|
path = tmp_path / "savings.json"
|
|
rec = SavingsRecorder(path, flush_every=1)
|
|
# Baseline so the estimate has something to compare against.
|
|
rec._ledger.baseline.observe("opus|new_user_ask|s|tools", 1000)
|
|
labels = ["compress:smartcrush", stratum_label("treatment", "opus|new_user_ask|s|tools")]
|
|
assert rec.record_from_labels(labels, output_tokens=600) is True
|
|
assert rec.record_from_labels(["no-shaper-label"], output_tokens=999) is False
|
|
est = rec.estimate()
|
|
assert est.n_requests == 1
|
|
assert est.tokens_saved == 400 # 1000 - 600
|
|
|
|
# Persisted to disk.
|
|
data = json.loads(path.read_text())
|
|
assert "treatment" in data
|