6904d47a01
## Description A small class of env vars is read by the proxy **live, per request** — the output-shaper family (`HEADROOM_OUTPUT_SHAPER`, `HEADROOM_VERBOSITY_LEVEL`, `HEADROOM_EFFORT_ROUTER`, `HEADROOM_MECHANICAL_EFFORT`, `HEADROOM_VERBOSITY_AUTOTUNE`, `HEADROOM_OUTPUT_HOLDOUT`), or captured at import (`HEADROOM_INTERCEPT_READ_MIN_CHARS`). The proxy reads them from its own process environment, fixed at launch. But `headroom wrap` reuses an already-running proxy (it restarts only on startup-config drift), so a value exported *after* the proxy started silently no-op'd — e.g. `export HEADROOM_OUTPUT_SHAPER=1` had zero effect on a reused proxy on `:8787`. This PR makes those live knobs **hot-reloadable**: `headroom wrap` pushes them to the running proxy, which applies them in memory — no restart (a restart would cold-start the ML stack, drop in-flight requests, and lose CCR/router caches). _No linked issue._ ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/runtime_env.py` (new): single source of truth registering the live knobs + a thread-safe process-global override store. `getenv()` (override-then-env) is a drop-in for `os.environ.get`; behaviour is byte-identical when no override is set. - Readers rerouted through `runtime_env.getenv`: `output_shaper.py`, the anthropic holdout read, and the ast-grep threshold (now a live read, not an import-time constant). - Proxy: loopback-only `POST /admin/runtime-env` applies overrides in memory; `/health` → `config.runtime_env` surfaces the live values so reuse is observable. - `wrap`: after attaching to a proxy (all call sites), best-effort push of the session's **explicitly-set** knobs. No-ops if nothing is set, `--no-proxy`, the proxy is unreachable, or it predates the endpoint (404). Only explicitly-set knobs are pushed, so a session never clobbers another with a default it never asked for. - Docs: README + output-token-reduction guide document the global-override caveat. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_runtime_env.py -q 16 passed $ python -m pytest tests/test_runtime_env.py tests/test_output_shaper.py -q 50 passed $ ruff check headroom/proxy/runtime_env.py headroom/proxy/output_shaper.py headroom/proxy/handlers/anthropic.py headroom/proxy/interceptors/astgrep.py headroom/proxy/server.py headroom/cli/wrap.py All checks passed! $ mypy headroom/proxy/runtime_env.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: local macOS, Python 3.12 `.venv`, branch `fix/runtime-env-hot-reload` at the PR head. - Exact command / steps: ran the test suites above. The 16 new `test_runtime_env` tests exercise the registry/store, overrides reaching the shaper + the ast-grep threshold, the `POST /admin/runtime-env` apply + `/health` reflect + loopback-only 404 + 400-on-non-object, and the wrap push payload / no-op / error-swallow paths. - Observed result: 50 passed; ruff + mypy clean on the changed modules; an override set via the endpoint is read by `getenv()` at the shaper and surfaced in `/health` config. - Not tested: a literal two-terminal manual session (start a proxy, `headroom wrap` a second session, `export HEADROOM_OUTPUT_SHAPER=1`, confirm the reused proxy picks it up). The behaviour is covered by the endpoint + wrap-push integration tests, but was not exercised by hand here. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - **Inherent caveat (documented):** overrides are global to the proxy — one process serves every attached wrapper, so the last explicit setting wins. No mechanism (restart or hot-reload) can give two sessions on one shared proxy different output-shaper settings. - **Scope:** startup-captured settings (`HEADROOM_TARGET_RATIO` etc.) are intentionally out of scope — a fresh proxy already gets them and they ride the existing `/health` config channel. - **Merge blocker:** this branch is currently **CONFLICTING with `main`** and needs a rebase/merge before it can land. - CHANGELOG.md left unchanged — releases are managed by release-please from conventional commits.
5.2 KiB
Output Token Reduction — User Guide
A plain-English guide to cutting the tokens the model writes back.
Why this exists
Headroom normally shrinks the prompt you send. This feature shrinks what the model returns. That matters because:
- Output tokens cost 5× more than input on Opus-class models.
- A lot of model output is waste: "Great, let me help with that…" intros, re-printing code you already showed it, restating tool results, and long internal "thinking" even on trivial steps.
You don't change any code. It runs in the Headroom proxy.
Turn it on
export HEADROOM_OUTPUT_SHAPER=1 # off by default
headroom proxy --port 8787
If a proxy is already running (e.g.
headroom wrap claudeattaches to one on port 8787 instead of starting a fresh one), it reads this switch from the environment it was launched with — so exporting it afterwards wouldn't reach it.headroom wraphandles this for you: it hot-syncs your current output settings to the running proxy (loopbackPOST /admin/runtime-env), applied immediately with no restart. Set the variables before you runwrap. Because one proxy is shared by every session attached to it, these settings are global — the most recent explicit value wins.
That's it. Two things now happen on every request:
- Verbosity steering — a short "be terse, don't restate context" instruction is added to the end of the system prompt. (The end, so your prompt cache still works.)
- Effort routing — if a turn is just the model continuing after a tool ran (e.g. it read a file and there were no errors), Headroom turns the model's "thinking effort" down for that one turn. Real questions and error-handling turns keep full effort.
The verbosity dial (levels 0–4)
| Level | What the model is told | Good for |
|---|---|---|
| 0 | (off) | disable steering |
| 1 | Skip the intro/outro chit-chat | people who read everything |
| 2 | Also: don't restate code/output already on screen | default — safe |
| 3 | Also: conclusions only, skip the reasoning | people who skim |
| 4 | Bare minimum, fragments OK | maximum savings, terse |
Set it by hand if you want:
export HEADROOM_VERBOSITY_LEVEL=3
Or — better — let Headroom learn it from your habits (next section).
Let Headroom pick the level for you
People rarely say "be brief." They show it: they interrupt long answers, or
reply so fast they couldn't have read the whole thing. headroom learn --verbosity reads your past sessions and picks a level from those signals.
# Preview what it found (doesn't change anything)
headroom learn --verbosity
# Save it — the proxy uses this level from now on
headroom learn --verbosity --apply
Example output:
Verbosity — headroom
Interrupts: 29 (11% of turns) ← push-back signal
Fast-skips: 31 / 119 long answers (26% unread) ← strongest signal
>> Recommended verbosity level: 3 (confidence: high)
Add --llm-judge to have an LLM double-check the level (needs an API key).
See how much you saved
Here's the honest part. We can't directly measure output savings — we never see what the model would have written without our nudge. So Headroom reports an estimate with a confidence range, never a fake exact number:
headroom output-savings
Output-token reduction
Method: ESTIMATED (synthetic control)
Requests: 1,240 shaped
Saved: ~410,000 output tokens
Reduction: 28.0% (95% CI 24.1% … 31.9%)
- ESTIMATED = compared against a baseline of your past (unshaped) sessions.
- MEASURED = the gold standard, if you opt into a holdout (below).
Want a measured number?
Leave a slice of traffic unshaped as a control group:
export HEADROOM_OUTPUT_HOLDOUT=0.1 # 10% of conversations stay unshaped
Now headroom output-savings compares shaped vs unshaped directly and reports a
measured reduction. The trade-off: you give up the savings on that 10%.
On the dashboard
Open http://localhost:8787/dashboard. Next to the input-compression card
you'll see an Output Tokens Saved card showing the token count, the percent,
a measured/estimated badge, and the confidence range.
FAQ
Will this make answers worse? At level 2 (default), no — in our tests the model finds the same bugs and writes the same fixes; it just stops re-printing code and skipping the "let me…" intro. Levels 3–4 are terser by design; that's why learning the level per user matters.
Does it break prompt caching? No. The steering text is added at the end of the system prompt and is byte-stable, so your cached prefix is untouched.
Is it safe with extended thinking / tool loops? Yes. It never disables thinking outright (that can error), it only lowers effort on routine turns, and it never adds settings the model doesn't support.
How do I turn it off?
Unset HEADROOM_OUTPUT_SHAPER (or set it to 0) and restart the proxy. You can
also send x-headroom-bypass: true on a request to skip it for that call.
Deep dive (design + the counterfactual math): proposals/output-token-reduction.md