diff --git a/.gitignore b/.gitignore index 052a2b6d1..5a2e08f42 100644 --- a/.gitignore +++ b/.gitignore @@ -21,6 +21,7 @@ scripts/* !scripts/tests/ !scripts/README.md !scripts/repro_codex_replay.py +!scripts/eval_output_shaper.py !scripts/fixtures/ !scripts/fixtures/*.json !scripts/record_fixtures.py diff --git a/README.md b/README.md index d0896e91d..4f108b25b 100644 --- a/README.md +++ b/README.md @@ -52,6 +52,7 @@ Headroom compresses everything your AI agent reads — tool outputs, logs, RAG c - **MCP server** — `headroom_compress`, `headroom_retrieve`, `headroom_stats` for any MCP client - **Cross-agent memory** — shared store across Claude, Codex, Gemini, auto-dedup - **`headroom learn`** — mines failed sessions, writes corrections to `CLAUDE.md` / `AGENTS.md` +- **Output token reduction** — trims what the model *writes back* (not just what you send): drops ceremony/restated code and skips deep "thinking" on routine steps. See [Output token reduction](#output-token-reduction-cut-what-the-model-writes-back). - **Reversible (CCR)** — originals are cached for retrieval on demand ## How it works (30 seconds) @@ -123,6 +124,54 @@ Granular extras: `[proxy]`, `[mcp]`, `[ml]`, `[code]`, `[memory]`, `[relevance]` Reproduce: `python -m headroom.evals suite --tier 1` · [Full benchmarks & methodology](https://headroom-docs.vercel.app/docs/benchmarks) +## Output token reduction (cut what the model writes back) + +Everything above shrinks the prompt you **send**. But you also pay for every +token the model **writes back** — and on Opus-class models output costs 5× input. +A lot of that output is waste: "Great, let me…" preambles, re-printing code you +just showed it, and deep "thinking" on routine steps like reading a file. + +Headroom can trim that too, from the proxy, without you changing any code: + +- **Verbosity steering** — appends a short "be terse, don't restate context" + note to the end of the system prompt (so your prompt cache still hits). +- **Effort routing** — when a turn is just the model resuming after a tool result + (a file read, a passing test), it dials the model's thinking effort down. New + questions and errors keep full effort. + +Turn it on: + +```bash +export HEADROOM_OUTPUT_SHAPER=1 # off by default +headroom proxy --port 8787 +``` + +**Learn the right terseness for you.** People don't *say* how terse they want +answers — they *show* it (they interrupt long replies, or move on before they +could have read them). `headroom learn --verbosity` reads your past sessions and +picks the level automatically: + +```bash +headroom learn --verbosity # preview what it found (dry run) +headroom learn --verbosity --apply # save it; the proxy uses it from now on +``` + +**See how many output tokens you saved.** Output savings are *counterfactual* — +we never see what the model *would* have written — so Headroom reports an honest +**estimate with a confidence range**, never a made-up number: + +```bash +headroom output-savings +# Reduction: 31.7% (95% CI 27.7% … 35.7%) [estimated] +``` + +Want a *measured* number instead of an estimate? Leave 10% of conversations +unshaped as a control group: `export HEADROOM_OUTPUT_HOLDOUT=0.1`. The dashboard +shows an **Output Tokens Saved** card next to input compression, labelled +`measured` or `estimated` with the confidence band. + +→ Full write-up incl. the measurement methodology: [`docs/proposals/output-token-reduction.md`](docs/proposals/output-token-reduction.md) + Star History Chart diff --git a/docs/output-token-reduction-guide.md b/docs/output-token-reduction-guide.md new file mode 100644 index 000000000..9fe0de3d9 --- /dev/null +++ b/docs/output-token-reduction-guide.md @@ -0,0 +1,136 @@ +# Output Token Reduction — User Guide + +A plain-English guide to cutting the tokens the model **writes back**. + +## Why this exists + +Headroom normally shrinks the prompt you **send**. This feature shrinks what the +model **returns**. That matters because: + +- Output tokens cost **5× more** than input on Opus-class models. +- A lot of model output is waste: "Great, let me help with that…" intros, + re-printing code you already showed it, restating tool results, and long + internal "thinking" even on trivial steps. + +You don't change any code. It runs in the Headroom proxy. + +## Turn it on + +```bash +export HEADROOM_OUTPUT_SHAPER=1 # off by default +headroom proxy --port 8787 +``` + +That's it. Two things now happen on every request: + +1. **Verbosity steering** — a short "be terse, don't restate context" instruction + is added to the **end** of the system prompt. (The end, so your prompt cache + still works.) +2. **Effort routing** — if a turn is just the model continuing after a tool ran + (e.g. it read a file and there were no errors), Headroom turns the model's + "thinking effort" down for that one turn. Real questions and error-handling + turns keep full effort. + +## The verbosity dial (levels 0–4) + +| Level | What the model is told | Good for | +|------:|------------------------|----------| +| 0 | (off) | disable steering | +| 1 | Skip the intro/outro chit-chat | people who read everything | +| 2 | Also: don't restate code/output already on screen | **default** — safe | +| 3 | Also: conclusions only, skip the reasoning | people who skim | +| 4 | Bare minimum, fragments OK | maximum savings, terse | + +Set it by hand if you want: + +```bash +export HEADROOM_VERBOSITY_LEVEL=3 +``` + +Or — better — let Headroom learn it from your habits (next section). + +## Let Headroom pick the level for you + +People rarely *say* "be brief." They *show* it: they interrupt long answers, or +reply so fast they couldn't have read the whole thing. `headroom learn +--verbosity` reads your past sessions and picks a level from those signals. + +```bash +# Preview what it found (doesn't change anything) +headroom learn --verbosity + +# Save it — the proxy uses this level from now on +headroom learn --verbosity --apply +``` + +Example output: + +``` +Verbosity — headroom + Interrupts: 29 (11% of turns) ← push-back signal + Fast-skips: 31 / 119 long answers (26% unread) ← strongest signal + >> Recommended verbosity level: 3 (confidence: high) +``` + +Add `--llm-judge` to have an LLM double-check the level (needs an API key). + +## See how much you saved + +Here's the honest part. We **can't directly measure** output savings — we never +see what the model *would* have written without our nudge. So Headroom reports an +**estimate with a confidence range**, never a fake exact number: + +```bash +headroom output-savings +``` + +``` +Output-token reduction + Method: ESTIMATED (synthetic control) + Requests: 1,240 shaped + Saved: ~410,000 output tokens + Reduction: 28.0% (95% CI 24.1% … 31.9%) +``` + +- **ESTIMATED** = compared against a baseline of your past (unshaped) sessions. +- **MEASURED** = the gold standard, if you opt into a holdout (below). + +### Want a *measured* number? + +Leave a slice of traffic unshaped as a control group: + +```bash +export HEADROOM_OUTPUT_HOLDOUT=0.1 # 10% of conversations stay unshaped +``` + +Now `headroom output-savings` compares shaped vs unshaped directly and reports a +**measured** reduction. The trade-off: you give up the savings on that 10%. + +## On the dashboard + +Open `http://localhost:8787/dashboard`. Next to the input-compression card +you'll see an **Output Tokens Saved** card showing the token count, the percent, +a `measured`/`estimated` badge, and the confidence range. + +## FAQ + +**Will this make answers worse?** +At level 2 (default), no — in our tests the model finds the same bugs and writes +the same fixes; it just stops re-printing code and skipping the "let me…" intro. +Levels 3–4 are terser by design; that's why learning the level per user matters. + +**Does it break prompt caching?** +No. The steering text is added at the *end* of the system prompt and is +byte-stable, so your cached prefix is untouched. + +**Is it safe with extended thinking / tool loops?** +Yes. It never disables thinking outright (that can error), it only lowers effort +on routine turns, and it never adds settings the model doesn't support. + +**How do I turn it off?** +Unset `HEADROOM_OUTPUT_SHAPER` (or set it to `0`) and restart the proxy. You can +also send `x-headroom-bypass: true` on a request to skip it for that call. + +--- + +Deep dive (design + the counterfactual math): [`proposals/output-token-reduction.md`](proposals/output-token-reduction.md) diff --git a/docs/proposals/output-token-reduction.md b/docs/proposals/output-token-reduction.md new file mode 100644 index 000000000..c768e5f35 --- /dev/null +++ b/docs/proposals/output-token-reduction.md @@ -0,0 +1,461 @@ +# Output Token Reduction + +**Branches:** `feat/output-token-reduction` (Phase 1) → `feat/verbosity-learning-and-counterfactual` (Phase 2). +**Status:** Phase 1 (output shaper) and Phase 2 (`learn --verbosity`, AIMD controller, counterfactual estimator, dashboard) both implemented + tested. Runtime AIMD signal-capture is staged (controller built/tested, live emission off by default). + +See **§7** for the counterfactual measurement methodology (how we honestly report a number we can't directly observe). + +--- + +## 1. The problem in one line + +Headroom's entire transform pipeline compresses what goes **into** the model. +Nothing today touches what comes **out**. But output tokens are billed at +5× input on Opus-class models ($25 vs $5 per MTok on `claude-opus-4-8`), and in +agentic coding loops a large fraction of the bill is output: thinking tokens, +restated code, ceremony ("Great, let me…"), and full-file rewrites where a +10-line edit would do. + +**The constraint that shapes everything:** the proxy never generates output +tokens — the model does. Once a token is streamed it is already billed. So +every output-token lever is **request-side**: change what we ask for, cap what +we allow, or avoid the generation entirely. There is no post-hoc lever. + +That gives three lever families plus a learning loop: + +| Lever | Mechanism | Status | +|---|---|---| +| **Verbosity steering** | Append a terse-style instruction to the system-prompt tail | ✅ built | +| **Effort routing** | Lower `output_config.effort` on mechanical turns | ✅ built | +| **Thinking-budget clamp** | Clamp legacy `thinking.budget_tokens` on mechanical turns | ✅ built | +| **Per-user learned level** | Mine past sessions for the right verbosity per user (`learn --verbosity`) | ✅ built | +| **Counterfactual estimator** | Honestly estimate output tokens saved + dashboard surfacing | ✅ built | +| **Runtime AIMD auto-tune** | Adjust level live from interrupt / skip signals | 🟡 controller built/tested; live signal emission off by default | + +--- + +## 2. Phase 1 — the output shaper (built) + +### 2.1 What it is + +`headroom/proxy/output_shaper.py` — a request-body rewriter invoked in +`handle_anthropic_messages` after every other body mutation (so the turn +classifier sees the final message list) and gated behind the same +`x-headroom-bypass` header as compression. Opt-in via `HEADROOM_OUTPUT_SHAPER=1`. + +### 2.2 Lever A — verbosity steering + +A deterministic instruction block is appended to the **tail** of the system +prompt. Five levels, cumulative: + +- **L0** — off (touch nothing). +- **L1** — no ceremony: skip preamble/postamble, don't announce what you're about to do. +- **L2** — L1 + no echo: never restate code/diffs/tool output already in context; reference by path:line; don't narrate tool results. **(default)** +- **L3** — L2 + conclusions only, omit rationale unless asked, prefer smallest edit. +- **L4** — caveman: fragments, minimum tokens, nothing but the answer. + +**Why the tail, not the head.** Prompt caching is a prefix match — any byte +change ahead of a `cache_control` breakpoint invalidates everything after it. +Prepending steering text would bust the provider prefix cache and cost more +than it saves. Appending after the last system block leaves the cached prefix +byte-identical; only the small, byte-stable steering block is reprocessed. The +steering text is frozen per level and applied idempotently (sentinel-tagged), +so repeated requests keep an identical prefix and a level change replaces the +block in place rather than stacking. + +### 2.3 Lever B — effort routing + +In an agentic loop, most API calls are **mechanical continuations**: the last +message is a clean `tool_result` (a file read, a passing test) and the model is +just resuming. Harnesses like Claude Code pin `output_config.effort` at `xhigh` +for *every* turn, including these — and effort drives thinking depth, which +bills as output. The router lowers effort to `low` on mechanical turns only. + +Turn classification is **purely structural** — no content regexes, no keyword +lists (per the project's no-hardcoded-patterns rule): + +| Last user message contains… | Classification | Action | +|---|---|---| +| Any text / image / document block | `NEW_USER_ASK` | leave effort alone | +| Only `tool_result`, none `is_error` | `MECHANICAL_CONTINUATION` | lower effort → `low` | +| Any `tool_result` with `is_error: true` | `ERROR_CONTINUATION` | leave effort alone (model must reason about the failure) | +| Anything else | `UNKNOWN` | leave alone | + +### 2.4 Lever C — legacy thinking-budget clamp + +On older models still sending `thinking: {type: "enabled", budget_tokens: N}`, +the router clamps `N` to the API floor (1024) on mechanical turns. The `type` +field is **never** toggled. + +### 2.5 Safety rules (each prevents a concrete failure) + +1. **Never inject `output_config.effort` where the client didn't send it.** + Models without effort support return 400 on it. Lowering an + already-present value is always valid — its presence proves the target + model accepts the param. +2. **Never toggle `thinking.type`.** Disabling thinking while history carries + thinking blocks 400s on some models, and the toggle busts the messages + cache tier (per the caching invalidation hierarchy). +3. **Byte-stable, idempotent steering** — repeated requests keep an identical + prefix; cache stays warm. +4. **Respect `x-headroom-bypass`** — sub-agent calls that opt out of + compression also opt out of shaping. + +### 2.6 Configuration + +| Env var | Default | Meaning | +|---|---|---| +| `HEADROOM_OUTPUT_SHAPER` | off | master switch (`1`/`true`/`yes`) | +| `HEADROOM_VERBOSITY_LEVEL` | `2` | 0–4 (clamped) | +| `HEADROOM_EFFORT_ROUTER` | on | set `0` to disable effort routing | +| `HEADROOM_MECHANICAL_EFFORT` | `low` | floor effort for mechanical turns | + +### 2.7 Tests + +`tests/test_output_shaper.py` — 34 tests, all passing. Covers turn +classification (every block-type path), cache-safe steering (string→block +conversion, append-after-`cache_control`, idempotency, level change), +effort routing (lower / never-inject / untouched-on-non-mechanical / +configurable target), legacy budget clamp, and the env gate. Ruff + mypy clean. + +--- + +## 3. Live before/after results + +Measured against `claude-opus-4-8` via `scripts/eval_output_shaper.py`, +comparing the exact body a client sends vs. the body the proxy forwards. +`usage.output_tokens` includes thinking. + +### 3.1 Verbosity steering — complex code-review ask + +Prompt: "review this TTLCache, find every bug, show fixes." + +| Condition | Mean output tokens | Reduction | +|---|---|---| +| Baseline | ~1,800–1,930 | — | +| L2 (no ceremony, no echo) | ~1,180–1,470 | **−22% to −39%** | +| L3 (conclusions only) | ~670 | **−63%** | + +**Quality check — same bugs found, only redundancy removed.** The baseline +opens with a title + "Let me go through them," finds the bugs, **re-prints the +entire fixed class**, then adds a **summary table restating all six bugs**, +then a trailing notes section — the same information appears ~2.5×. L2 opens +directly with the findings, gives the same fixed code block once, and stops. +Both correctly identify the no-locking race, `popitem(last=True)` evicting the +wrong end, and the mutate-during-iteration `RuntimeError`. Nothing of substance +is lost at L2; L3 additionally drops rationale prose (only for users who don't +read explanations). + +### 3.2 Effort routing — agentic mechanical continuation + +Transcript ending in a clean `tool_result`, `effort: xhigh` the way Claude Code +sends it. + +| Condition | Mean output tokens | Reduction | +|---|---|---| +| Baseline (xhigh) | ~1,120 | — | +| Shaped (routed to low) | 793 | **−29%** | + +### 3.3 Honest framing + +Caveman (L4) would get ~60–70% but degrades the experience. The taxonomy-driven +default (L2 + effort routing on mechanical turns) gets a realistic **25–40%** +with the user barely noticing. The learning loop (Phase 2) finds where each +user sits between those poles. + +--- + +## 4. Phase 2 — learning the right level per user + +### 4.1 Why this is necessary + +The right verbosity is **per-user**, not global. A fixed `HEADROOM_VERBOSITY_LEVEL` +is a guess. We can do better: mine the user's own past sessions to infer what +they actually tolerate — exactly the philosophy of `headroom learn`, which +already reads `~/.claude/projects/*.jsonl` and turns history into learned +context. + +### 4.2 The key insight (validated on real data) + +I prototyped the signal extraction and ran it over **24 real sessions** for +this project. The finding that shapes the whole design: + +| Signal | This user's data | Carries signal? | +|---|---|---| +| Explicit "be brief" keywords | **1** of 210 human msgs | ❌ almost none | +| Explicit "explain more" keywords | **1** of 210 | ❌ almost none | +| **Interruptions** (user cuts Claude off) | **29** (~1 per 7 turns) | ✅ strong | +| **Fast-skips** (reply <30s after a >250-word answer) | **15 of 100 long outputs** | ✅ strong | +| Long-output frequency | 100 outputs >250 words | ✅ context | + +**Users almost never *say* what verbosity they want — they show it +behaviorally.** Keyword matching (the obvious approach) is nearly empty. The +behavioral signals are rich. The strongest is **fast-skip**: a reply arriving +faster than the answer could have been read is a direct measurement of +"generated tokens nobody consumed," computable from timestamps + lengths +already in the JSONL. + +For this user — 29 interrupts, 15% unread-long-output rate, zero "explain more" +requests — the data reads as a clear **L2, arguably L3** user. + +### 4.3 Design — `headroom learn --verbosity` + +Slots into the existing `learn` architecture. `headroom/learn/plugins/claude.py` +already parses the JSONL; `analyzer.py` already does cheap-extraction → digest → +LLM-returns-JSON. We add a verbosity analysis path: + +**Step 1 — structural pass (pure Python, no patterns).** Per session, compute: +- **interrupt rate** — `[Request interrupted by user…]` markers per human turn +- **fast-skip rate** — human reply latency vs. preceding assistant output length +- **long-output frequency** — share of assistant texts over N words +- **echo ratio** — n-gram overlap between assistant output and prior context (restated code) +- reply-latency distribution vs. output length + +All mechanical, all from data already on disk. (Prototype: +`scripts/verbosity_scan.py`, validated above.) + +**Step 2 — LLM judgment pass (mirrors today's `learn`).** Feed a digest of +human messages + the structural stats to the analyzer LLM with one question: +*"Does this user read long explanations, and when they push back, is it for +more detail or less?"* This replaces brittle keyword lists with judgment — +consistent with the no-hardcoded-patterns rule. Returns a level + confidence + +rationale as JSON. + +**Step 3 — output (this is what `--verbosity` produces).** See §4.4. + +**Step 4 — runtime AIMD auto-tune.** The offline pass sets the *starting* level; +a live loop tracks drift. An interrupt or fast-skip nudges the level up one; an +"explain"/"why" follow-up drops it back. Hysteresis (require 2–3 consistent +signals before moving) prevents oscillation. Like TCP congestion control: probe +toward terser, back off on a "too terse" signal. + +### 4.4 What `--verbosity` outputs — and how it helps + +The command produces three concrete artifacts: + +**(a) A human-readable report (stdout):** +``` +Verbosity analysis — /Users/tcms/demo/headroom (24 sessions, 210 turns) + + Interrupts: 29 (1 per 7.2 turns) ← strong "too much" signal + Fast-skips: 15 / 100 long outputs ← 15% of long answers unread + Explicit brevity: 1 explicit verbose: 1 ← behavioral, not stated + + LLM read: "User interrupts frequently and rarely reads long + explanations; pushes back for less, never more detail." + + Recommended verbosity level: 2 (confidence: high) + Estimated output-token reduction at L2: ~25–35% +``` + +**(b) A persisted setting** written to `~/.headroom/` (alongside the existing +savings tracker) — per-project verbosity level + confidence: +```json +{"project": "/Users/tcms/demo/headroom", + "verbosity_level": 2, "confidence": "high", + "signals": {"interrupt_rate": 0.138, "fast_skip_rate": 0.15}, + "learned_at": "2026-06-12T…"} +``` + +**(c) The shaper reads it as its default.** `OutputShaperSettings.from_env()` +gains a fallback: if `HEADROOM_VERBOSITY_LEVEL` is unset, load the learned +per-project level instead of the hardcoded `2`. So **the output of `--verbosity` +directly becomes the live verbosity the proxy applies** — no manual tuning. + +**How it helps, concretely:** +1. **Removes the guess.** Today you set `HEADROOM_VERBOSITY_LEVEL=2` by hand. + After `learn --verbosity`, the level is derived from *your* behavior — a + heavy-interrupter gets L3, a "read everything" user gets L1. +2. **Per-project, not global.** Your exploratory side-project and your + production repo can carry different levels. +3. **Justified, not magic.** The report shows the signals and the LLM's read, + so the recommendation is auditable (matches the dashboard philosophy of + showing directional data, not opaque scores). +4. **Seeds the runtime loop.** The learned level is the AIMD starting point; + live signals refine it without re-running the offline pass. + +### 4.5 Mapping signals → level (initial heuristic, LLM-refined) + +| Interrupt rate | Fast-skip rate | "explain more" present | → Level | +|---|---|---|---| +| low | low | yes | 1 | +| low–med | low–med | no | 2 | +| high | high | no | 3 | +| very high | very high | no | 4 (offer, don't auto-apply) | + +The LLM judgment pass adjusts this — the table is the prior, not the verdict. + +--- + +## 5. Files + +| File | Status | Purpose | +|---|---|---| +| `headroom/proxy/output_shaper.py` | ✅ built | the shaper (steering + effort routing) | +| `headroom/proxy/handlers/anthropic.py` | ✅ wired | invoke shaper after body mutations | +| `tests/test_output_shaper.py` | ✅ 34 passing | unit coverage | +| `scripts/eval_output_shaper.py` | ✅ built | live before/after eval | +| `scripts/verbosity_scan.py` | 🔬 prototype | session-mining signal extraction | +| `headroom/learn/plugins/claude.py` (+ analyzer) | 🔜 extend | `--verbosity` analysis path | +| `~/.headroom/verbosity.json` | 🔜 | persisted per-project learned level | + +--- + +## 6. Roadmap + +1. ✅ **Measure** — live eval establishes the realistic ceiling and baseline. +2. ✅ **Effort router** — biggest win per unit risk, fully mechanical, invisible. +3. ✅ **Verbosity ladder at fixed L2** — safe default, cache-safe tail injection. +4. 🔜 **`learn --verbosity`** — derive the per-user starting level from sessions. +5. 🔜 **Runtime AIMD auto-tune** — refine the level live from interrupt/skip signals. +6. 🔭 **Waste taxonomy on the dashboard** — echo ratio, ceremony ratio, + full-file-rewrite detection, as token counts (no dollar estimates). +7. 🔭 **Budget whispering** — tell the model its token budget per turn-type, + sized from the historical output distribution in SQLite. + +--- + +## 7. Counterfactual measurement — how we show a % we can't directly observe + +This is the hard part, and it deserves its own section. + +### 7.1 Why output savings are not directly measurable + +Input compression is a **pure function**: Headroom takes a request, shrinks it, +and can count `tokens_before` and `tokens_after` — both are observable on the +same request. Output is different. When the shaper makes a request terser, the +model emits N output tokens. We **never observe** what it *would* have emitted +without the steering. Only one side of the counterfactual ever happens. So a +flat "we saved 30%" is a guess dressed as a fact. + +The design rule that follows: **never report a single number as if it were +measured.** Separate what is genuinely measured from what is estimated, label +each, and always attach uncertainty. + +### 7.2 Three tiers of honesty + +**Tier 1 — Estimated (synthetic control).** Build a per-stratum baseline of +*unshaped* output tokens from session history that predates the shaper +(`learn --verbosity` does this in the same pass that picks the level). For each +shaped request, the expected unshaped output is the baseline mean for that +request's stratum. Aggregate estimate: + +``` +tokens_saved = Σ over shaped requests ( baseline_mean[stratum] − observed_output ) +``` + +Summed as **signed** deltas — never clamped per-request. Clamping each delta at +zero would throw away the cases where a shaped response happened to be *longer* +by chance, biasing the total upward. Over many requests the noise averages out; +the systematic effect remains. Reported with a propagated 95% CI (see §7.5) and +always labelled "estimated." + +**Tier 2 — Measured (A/B holdout).** Set `HEADROOM_OUTPUT_HOLDOUT=0.1` and 10% +of conversations are deliberately left **unshaped** as a control arm. Within +each stratum, `mean(control) − mean(treatment)` is an **unbiased causal +estimate** of the per-request saving. This is the only number we call +"measured." It self-corrects: if steering doesn't actually help on some +workload, the holdout reveals it. The cost is real (you forgo savings on 10% of +traffic), so it's opt-in; default holdout is 0 (estimate-only). + +**Tier 3 — Direct waste (no counterfactual at all).** Echo ratio — the n-gram +overlap between a response and the context it was given — is a property of a +single response. "32% of this output restated context already on screen" needs +no counterfactual; it's a measured fact about output we *did* see, and it's +exactly what the shaper targets. Surfaced in `learn --verbosity` as +`mean_echo_ratio`. + +### 7.3 Stratification — comparing like with like + +You can't compare a "fix this typo" response to "design a caching layer." The +estimator buckets every request by features observable **before** the response: + +``` +stratum = model_family | turn_kind | input_token_bucket | has_tools + = e.g. "opus | mechanical_continuation | xl | tools" +``` + +Coarse on purpose (~25–50 strata) so per-stratum baselines stay dense. The live +proxy computes the stratum the exact same way the offline baseline does, so +treatment requests line up with their baseline. Unseen strata fall back +hierarchically (drop `has_tools`, then the bucket, …, then the global mean). + +### 7.4 The two constraints that happen to align + +Holdout assignment is **conversation-stable** — a whole conversation is either +treatment or control, decided by hashing a conversation-stable key (model + +first user message). This matters for two independent reasons that point the +same way: + +1. **Measurement validity** — mixing shaped and unshaped turns within one + conversation would contaminate the comparison (the history itself differs). +2. **Cache safety** — flipping a conversation's verbosity mid-stream changes the + system-prompt tail, which busts the provider prefix cache. + +So the same rule (assign per conversation, never per turn) is forced by both +the statistics and the caching. Nice when constraints agree. + +### 7.5 The confidence interval + +For the estimated tier, uncertainty comes from two sources, both propagated: + +``` +Var(tokens_saved) ≈ Σ_s [ n_s · σ²_observed,s + n_s² · σ²_baseline,s / m_s ] + └─ spread of shaped outputs ─┘ └─ finite-baseline error ─┘ +``` + +where `n_s` is treatment count in stratum `s` and `m_s` the baseline sample +count. For the measured tier it's the standard difference-of-means variance +`σ²_c/n_c + σ²_t/n_t` per stratum. The 95% band is `point ± 1.96·√Var`, surfaced +everywhere the number is (CLI and dashboard), so the reader sees the precision, +not just a point estimate. + +Output-token counts are right-skewed (a few huge responses). Means are still the +right statistic for *totals* (you're billed on the sum), but the CI widens +honestly when a stratum is dominated by a few large responses — which is the +correct signal that the estimate is soft there. + +### 7.6 How it flows end to end + +``` +learn --verbosity --apply + ├─ writes verbosity.json (the level the shaper applies) + └─ seeds output_savings.json (the per-stratum baseline = synthetic control) + +proxy request (HEADROOM_OUTPUT_SHAPER=1) + ├─ assign_arm(conversation) → treatment | control (holdout) + ├─ stratum_key(request features) + ├─ treatment: shape body; control: leave unshaped + └─ tag (arm, stratum) onto transforms_applied ← rides existing plumbing + +response completes → emit_request_outcome (one funnel, all paths) + └─ recorder.record(arm, stratum, output_tokens) → output_savings.json + +headroom output-savings / dashboard "Output Tokens Saved" card + └─ best_estimate(): measured if a holdout exists, else estimated; with CI +``` + +The recording rides the existing `transforms_applied` label channel, so it +works for streaming, non-streaming, and backend paths with no change to +`RequestOutcome` or its construction sites. + +### 7.7 What the user sees + +- **CLI:** `headroom output-savings` → + `Reduction: 31.7% (95% CI 27.7% … 35.7%) [MEASURED, 400 shaped requests]` +- **Dashboard:** an "Output Tokens Saved" hero card next to input compression — + token count, percent, a `measured`/`estimated` badge, and the CI band. +- **No dollar estimates** on the output card (per project convention) — token + counts and directional percentages only. + +### 7.8 Honest limitations + +- Estimated-tier accuracy depends on the baseline matching current workload; if + your tasks drift, re-run `learn --verbosity` or turn on a small holdout. +- The baseline must come from *unshaped* history. If you learn from sessions + where the shaper was already active, the baseline is contaminated — the live + holdout is the clean path forward. +- Runtime AIMD upward-ratcheting is gated off by default: we can reliably detect + "too much output" (fast-skip timing, stream cancellation) but not yet "too + little" at runtime without content heuristics, so auto-escalation stays + behind `HEADROOM_VERBOSITY_AUTOTUNE` until both directions are trustworthy. diff --git a/headroom/cli/learn.py b/headroom/cli/learn.py index d063a2172..2b69d89c9 100644 --- a/headroom/cli/learn.py +++ b/headroom/cli/learn.py @@ -3,7 +3,7 @@ from __future__ import annotations from pathlib import Path -from typing import TYPE_CHECKING +from typing import TYPE_CHECKING, Any import click @@ -105,6 +105,21 @@ Use 'auto' (default) to scan all detected agents.""" help="Only scan top-level main sessions, skipping nested subagent/workflow " "transcripts (Claude Code). Default scans everything.", ) +@click.option( + "--verbosity", + "verbosity_mode", + is_flag=True, + default=False, + help="Learn the user's preferred OUTPUT verbosity from behavioral signals " + "(interrupts, fast-skips) instead of analyzing failures. Writes the level " + "the output shaper applies, and seeds the savings baseline. --apply persists.", +) +@click.option( + "--llm-judge", + is_flag=True, + default=False, + help="With --verbosity: let an LLM override the heuristic level (needs an API key).", +) def learn( project: Path | None, analyze_all: bool, @@ -113,6 +128,8 @@ def learn( model: str | None, workers: int | None, main_only: bool, + verbosity_mode: bool, + llm_judge: bool, ) -> None: """Learn from past tool call failures to prevent future ones. @@ -139,6 +156,19 @@ def learn( max_workers = workers if workers is not None else min(os.cpu_count() or 4, 8) + # Verbosity learning is a distinct flow: it mines behavioral signals (no + # failure analysis) and needs no LLM unless --llm-judge is set. + if verbosity_mode: + _run_verbosity( + project=project, + analyze_all=analyze_all, + apply=apply, + agent=agent, + llm_judge=llm_judge, + model=model, + ) + return + # Resolve model early to fail fast with a clear message try: resolved_model = model or _detect_default_model() @@ -271,3 +301,152 @@ def learn( f"Total: {total_projects} projects, {total_failures} failures, " f"{total_recommendations} recommendations" ) + + +def _make_llm_judge(model: str) -> Any: + """Build an LLM judge callable for verbosity, or None if unavailable. + + The judge gets the behavioral signals and returns (level, rationale). Kept + best-effort: any failure (no key, parse error) returns None so the caller + falls back to the heuristic. + """ + + def judge(signals: dict) -> tuple[int, str] | None: + try: + import json + + import litellm + except ImportError: + return None + prompt = ( + "You tune how terse an AI coding assistant should be for one user, " + "from their behavioral signals. Levels: 1=light (skip ceremony), " + "2=no ceremony+no echo, 3=conclusions only, 4=caveman/fragments. " + "Users who interrupt often and reply faster than an answer could be " + "read (fast-skip) want LESS output.\n\n" + f"Signals: {json.dumps(signals)}\n\n" + 'Return ONLY JSON: {"level": <1-4>, "rationale": ""}' + ) + try: + resp = litellm.completion( + model=model, + messages=[{"role": "user", "content": prompt}], + max_tokens=200, + ) + text = resp["choices"][0]["message"]["content"] + start, end = text.find("{"), text.rfind("}") + data = json.loads(text[start : end + 1]) + return int(data["level"]), str(data.get("rationale", "LLM judgment")) + except Exception: + return None + + return judge + + +def _run_verbosity( + *, + project: Path | None, + analyze_all: bool, + apply: bool, + agent: str, + llm_judge: bool, + model: str | None, +) -> None: + """Learn preferred output verbosity from session transcripts.""" + from ..learn.registry import auto_detect_plugins, get_plugin + from ..learn.verbosity import analyze + from ..paths import ensure_workspace_dir + from ..proxy.output_savings import SavingsLedger + + # Verbosity mining reads Claude Code transcripts; restrict to that plugin. + if agent == "auto": + plugins = [p for p in auto_detect_plugins() if p.name == "claude"] + if not plugins: + click.echo("Verbosity learning currently supports Claude Code transcripts only.") + return + plugin = plugins[0] + else: + plugin = get_plugin(agent) + if plugin.name != "claude": + click.echo("Verbosity learning currently supports Claude Code transcripts only.") + return + + all_projects = plugin.discover_projects() + if not all_projects: + click.echo("No Claude Code project data found.") + return + + if analyze_all: + targets = all_projects + elif project: + resolved = project.resolve() + targets = [p for p in all_projects if p.project_path == resolved] + else: + cwd = Path.cwd().resolve() + targets = [p for p in all_projects if p.project_path == cwd] + if not targets: + for parent in cwd.parents: + targets = [p for p in all_projects if p.project_path == parent] + if targets: + break + if not targets: + click.echo("No matching project. Try --all or --project .") + return + + judge = _make_llm_judge(model or "claude-sonnet-4-6") if llm_judge else None + + for proj in targets: + session_paths = sorted(proj.data_path.glob("*.jsonl")) + if not session_paths: + continue + profile, baseline = analyze(session_paths, str(proj.project_path), llm_judge=judge) + sig = profile.signals + + click.echo(f"\n{'=' * 60}") + click.echo(f"Verbosity — {proj.name}") + click.echo(f"Path: {proj.project_path}") + click.echo(f"{'=' * 60}") + click.echo( + f" Sessions: {sig.get('sessions')} human turns: {sig.get('human_msgs')} " + f"responses: {sig.get('asst_responses')}" + ) + click.echo( + f" Interrupts: {sig.get('interrupts')} " + f"({sig.get('interrupt_rate', 0):.0%} of turns) " + "← push-back signal" + ) + click.echo( + f" Fast-skips: {sig.get('fast_skips')} / {sig.get('skip_eligible')} long " + f"answers ({sig.get('fast_skip_rate', 0):.0%} unread) ← strongest signal" + ) + click.echo(f" Echo ratio: {sig.get('mean_echo_ratio', 0):.1%} of output restated context") + click.echo(f"\n Source: {profile.source}") + click.echo(f" {profile.rationale}") + click.echo( + f"\n >> Recommended verbosity level: {profile.level} " + f"(confidence: {profile.confidence})" + ) + + if apply: + ws = ensure_workspace_dir() + from datetime import datetime, timezone + + profile.learned_at = datetime.now(timezone.utc).isoformat() + profile.save(ws / "verbosity.json") + # Seed the savings baseline: replace baseline, preserve any live + # treatment/control already accumulated. + ledger_path = ws / "output_savings.json" + ledger = SavingsLedger.load(ledger_path) + ledger.baseline = baseline + ledger.save(ledger_path) + click.echo(f"\n [WROTE] {ws / 'verbosity.json'} (level {profile.level})") + click.echo( + f" [WROTE] {ledger_path} (baseline: {baseline.total_samples} samples, " + f"{len(baseline.strata)} strata)" + ) + click.echo( + "\n The output shaper now uses this level when " + "HEADROOM_OUTPUT_SHAPER=1 and HEADROOM_VERBOSITY_LEVEL is unset." + ) + else: + click.echo("\n Dry run — use --apply to persist the level and baseline.") diff --git a/headroom/cli/main.py b/headroom/cli/main.py index 472028357..861ea51d6 100644 --- a/headroom/cli/main.py +++ b/headroom/cli/main.py @@ -45,6 +45,7 @@ def _register_commands() -> None: install, # noqa: F401 learn, # noqa: F401 mcp, # noqa: F401 + output_savings, # noqa: F401 perf, # noqa: F401 proxy, # noqa: F401 tools, # noqa: F401 diff --git a/headroom/cli/output_savings.py b/headroom/cli/output_savings.py new file mode 100644 index 000000000..a341b2eea --- /dev/null +++ b/headroom/cli/output_savings.py @@ -0,0 +1,61 @@ +"""CLI: show counterfactual output-token reduction.""" + +from __future__ import annotations + +import click + +from .main import main + + +@main.command(name="output-savings") +def output_savings() -> None: + """Show estimated/measured output-token reduction from the shaper. + + Output tokens are counterfactual — we never see what the model *would* have + emitted unshaped. This reports the honest estimate: + + * "measured" — from an A/B holdout (set HEADROOM_OUTPUT_HOLDOUT>0), the + unbiased difference between unshaped and shaped arms. + * "estimated" — synthetic control: shaped output vs. the per-stratum + baseline learned by `headroom learn --verbosity`. + + Both are shown with a 95% confidence band so the uncertainty is explicit. + """ + from ..paths import workspace_dir + from ..proxy.output_savings import SavingsLedger + + path = workspace_dir() / "output_savings.json" + if not path.exists(): + click.echo("No output-savings data yet.") + click.echo("Run `headroom learn --verbosity --apply` to seed the baseline,") + click.echo("then enable the shaper (HEADROOM_OUTPUT_SHAPER=1) and send traffic.") + return + + ledger = SavingsLedger.load(path) + est = ledger.best_estimate() + + click.echo(f"\n{'=' * 56}") + click.echo("Output-token reduction") + click.echo(f"{'=' * 56}") + if est.n_requests == 0: + click.echo(" No shaped requests recorded yet.") + click.echo( + f" Baseline: {ledger.baseline.total_samples} samples, " + f"{len(ledger.baseline.strata)} strata." + ) + return + + label = "MEASURED (A/B holdout)" if est.kind == "measured" else "ESTIMATED (synthetic control)" + click.echo(f" Method: {label}") + click.echo(f" Requests: {est.n_requests:,} shaped") + click.echo(f" Baseline: {est.baseline_tokens:,.0f} output tokens expected") + click.echo(f" Saved: {est.tokens_saved:,.0f} output tokens") + click.echo( + f" Reduction: {est.pct:.1f}% (95% CI {est.ci_low_pct:.1f}% … {est.ci_high_pct:.1f}%)" + ) + if est.kind == "estimated": + click.echo( + "\n Note: estimated vs the learned baseline. For a measured number," + "\n set HEADROOM_OUTPUT_HOLDOUT=0.1 to leave 10% of conversations" + "\n unshaped as a control arm." + ) diff --git a/headroom/dashboard/templates/dashboard.html b/headroom/dashboard/templates/dashboard.html index fd6939845..1167d1968 100644 --- a/headroom/dashboard/templates/dashboard.html +++ b/headroom/dashboard/templates/dashboard.html @@ -178,8 +178,8 @@