akadmin/headroom
1
0
Fork 0
mirror of https://github.com/headroomlabs-ai/headroom.git synced 2026-08-27 14:17:10 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
2630 commits 410 branches 169 tags 142 MiB
Commit graph

4 commits

Author SHA1 Message Date
Andrew McFague
0b0133b7fd
Wire OpenAI Responses output shaping (#1438) 
## Description

Wire output shaping for OpenAI Responses traffic across HTTP
`/v1/responses` and Codex WebSocket `response.create` frames. The change
adds provider-specific shaping for `instructions`, `reasoning.effort`,
and `text.verbosity` while keeping Anthropic request mutation separate.

Review follow-up: merged byte-faithful `/v1/responses` forwarding from
#1557 and marks shaped HTTP Responses payloads as `body_mutated=True`,
so retry forwarding sends the shaped body instead of the original raw
bytes.

## Type of Change

- [ ] Bug fix (non-breaking change fixes an issue)
- [x] New feature (non-breaking change adds functionality)
- [ ] Breaking change (fix or feature would cause existing functionality
change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Added OpenAI Responses output shaping for `instructions`,
`reasoning.effort`, and `text.verbosity`.
- Wired shaping into `/v1/responses` HTTP and Codex WebSocket
`response.create` paths.
- Preserved `x-headroom-bypass` and `HEADROOM_OUTPUT_HOLDOUT` behavior.
- Added output-shaper transform labels for verbosity, text verbosity,
reasoning effort, holdout control, and strata.
- Updated output-savings conversation keys for Responses payloads and WS
`response.create` envelopes.
- Counted WS frame payload tokens when assigning output-savings strata.
- Merged byte-faithful `/v1/responses` forwarding from #1557 and kept
shaped HTTP bodies on the mutated-forwarding path.
- Added tests for classification, shaping, holdout, bypass, labels, WS
strata, and byte-faithful forwarding compatibility.
- Updated `CHANGELOG.md` for OpenAI Responses output-shaping support.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --extra dev python -m pytest tests/test_openai_codex_ws_lifecycle.py tests/test_openai_responses_output_shaper.py tests/test_codex_responses_passthrough_bytes.py tests/test_output_shaper.py tests/test_output_savings.py -q
110 passed, 1 warning in 1.49s

$ uv run --extra dev ruff check headroom/proxy/handlers/openai.py tests/test_openai_responses_output_shaper.py tests/test_codex_responses_passthrough_bytes.py tests/test_openai_codex_ws_lifecycle.py tests/test_output_shaper.py tests/test_output_savings.py
All checks passed!

$ git diff --check
No whitespace errors.
```

## Real Behavior Proof

- Environment: local macOS checkout, branch
`output-shaper-openai-responses`.
- Exact command / steps: ran targeted pytest, ruff, and diff checks
listed above.
- Observed result: targeted tests passed with an existing FastAPI
TestClient deprecation warning; ruff passed; diff check passed.
- Not tested: full repository test suite, live OpenAI traffic, browser
dashboard rendering, full `mypy headroom`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows project's style guidelines
- [x] I performed self-review of my code
- [x] I commented my code, particularly in hard-to-understand areas
- [x] I made corresponding changes to documentation
- [x] My changes generate no new warnings
- [x] I added tests that prove my fix is effective or that my feature
works
- [x] New and existing unit tests pass locally with my changes
- [x] I updated `CHANGELOG.md` if applicable

## Screenshots

N/A

## Additional Notes

- Non-applicable Type Change items are left unchecked.
- The pytest warning comes from `fastapi.testclient` importing Starlette
TestClient and was not introduced by this change.
- `CHANGELOG.md` includes entries for OpenAI Responses output-shaping
support and byte-faithful `/v1/responses` forwarding compatibility.

---------

Co-authored-by: obchain <riteshnikhoriya94@gmail.com>
 2026-07-05 13:59:21 -07:00
gglucass
27a5468349
fix(learn): aggregate verbosity baselines across projects instead of overwriting (#1288) 
## Description

`headroom learn --verbosity --apply --all` was building the
output-shaper's savings baseline from only **one** project.
`_run_verbosity` wrote the savings ledger *inside* the per-project loop
(`ledger.baseline = baseline; ledger.save(...)`), so each project
replaced the previous baseline and only the last project processed
survived — frequently a near-empty one. The synthetic-control estimate
that `/stats` exposes (`savings.by_layer.output_shaping`) was then
computed against a tiny, unrepresentative sample.

This PR makes `--all` aggregate across every targeted project and write
the ledger **once**, after the loop.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `BaselineModel.merge()` / `_Accum.merge()`
(`headroom/proxy/output_savings.py`): fold one baseline into another.
The accumulators hold additive online stats (`n` / `sum` / `sumsq`), so
merging is element-wise and order-independent — identical to having
observed both corpora against a single model.
- `_run_verbosity` (`headroom/cli/learn.py`): accumulate a single
`BaselineModel` across all targeted projects and persist it once after
the loop, instead of overwriting per project. The applied verbosity
level now comes from the project with the most samples (strongest
signal) rather than whichever sorted last. Single-project runs are
unchanged (an aggregate of one). When no transcripts are found, it
prints a clear message and writes nothing.
- Tests: unit test for `BaselineModel.merge`; CLI test that `--all
--apply` across two projects aggregates both strata (totals summed, not
last-wins) and applies the busier project's level.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run pytest tests/test_output_savings.py tests/test_cli_learn.py tests/test_verbosity_learn.py -q
tests/test_output_savings.py ...............................             [ 54%]
tests/test_cli_learn.py ...........                                      [ 73%]
tests/test_verbosity_learn.py ...............                            [100%]
============================== 57 passed in 0.51s ==============================

$ uv run ruff check headroom/cli/learn.py headroom/proxy/output_savings.py
All checks passed!

$ uv run mypy headroom/cli/learn.py headroom/proxy/output_savings.py
Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: macOS, Python 3.12.13, this branch off `upstream/main`.
- Exact command / steps: `headroom learn --verbosity --apply --all` (run
across a multi-project transcript corpus), then inspect
`~/.headroom/output_savings.json` (`baseline.glob.n`); compared against
`headroom learn --verbosity --apply` for a single busy project.
- Symptom (pre-fix, installed build): `headroom learn --verbosity
--apply --all` across a multi-project transcript corpus wrote
`~/.headroom/output_savings.json` with `baseline.glob.n = 2` (the last
project processed was a near-empty `…/venv/bin` dir), while targeting a
single busy project gave `baseline.glob.n = 15658`.
- With this change: the new CLI test
(`test_verbosity_all_apply_aggregates_baselines_across_projects`) drives
`--all --apply` over two projects (3 samples + 1 sample) and asserts the
persisted ledger has `total_samples == 4` with both strata present, plus
the busier project's level applied.
- Observed result: aggregated baseline persisted once; both strata
retained; level taken from the higher-sample project.
- Not tested: re-running the patched `--all` end-to-end on a live
multi-project machine (covered instead by the unit merge-math test and
the faked-`analyze` CLI test).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A — CLI/behavioral change.

## Additional Notes

- No linked issue (`Closes #` left blank intentionally).
- Documentation checklist item is N/A — no user-facing docs describe the
per-project overwrite behavior.
- Level-selection note: for `--all`, the applied verbosity level is now
deterministic (most-samples project) instead of last-processed; this is
the intended improvement, not a behavior to preserve.

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
 2026-06-30 08:37:37 -05:00
Lucas Santos
c30ec4cda8
fix: surface output reduction without a restart, and explain $0.00 savings on Python 3.14 (#1296) 
## Description

I was running headroom through pipx on Python 3.14 and hit two issues.

The Proxy $ Saved tile was stuck at $0.00 even though tokens were
tracking fine. Pricing comes from litellm, and litellm does not install
on Python 3.14 because of a version lock, so there was just nothing to
price against. Rather than hardcode a price table that goes stale, I
added a `litellm_available` flag to `/stats` and the tile now tells you
to reinstall on 3.13 when pricing isn't there, like the output-shaper
tile already does.

The other one was Output Tokens Saved showing "—" after I turned on the
shaper. The recorder reads the learned baseline once at startup, so if
you run `learn --verbosity --apply` while the proxy is already up it
never gets picked up, and a later flush writes the empty baseline over
the one learn just saved. Now it re-reads the baseline before estimating
and before each flush, so it works without a restart.

Closes # N/A (no tracking issue)

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `output_savings.py`: re-read the baseline from disk before estimating
and before each flush, so a baseline learned while the proxy is running
takes effect (and a re-learn with the same sample count too).
- `server.py`: expose a `litellm_available` flag on `/stats`.
- `dashboard.html`: when savings are zero and litellm is missing, point
to reinstalling on 3.13 instead of showing $0.00.
- tests and docs (`test_output_savings.py`, README, metrics, CHANGELOG).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_output_savings.py -q
34 passed, 1 warning in 0.11s
```

## Real Behavior Proof

- Environment: macOS, Python 3.13 (litellm present) and 3.14 (litellm
absent), running this branch.
- Exact command / steps: record shaped traffic, write a baseline to the
same file while the recorder is live (no restart), then estimate and
flush.
- Observed result: the recorder goes from `available: False` to
`available: True` once the baseline is written mid-run, and keeps it
after a flush. Before this it stayed `False` and the flush reset the
baseline. Raw output:
  ```text
  shaper traffic recorded, baseline not learned yet -> available: False
  learn --apply wrote baseline while proxy up; restart NOT performed
after baseline write -> available: True | method: estimated | pct: 50.3
  baseline kept after flush -> disk samples: 4
  ```
- Not tested: I did not render the tile hint in a browser, I checked the
flag on `/stats` and read the template instead.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

Just a final note, ruff and mypy are clean on what I changed. The
repo-wide `ruff check .` and `mypy headroom` do report a few problems,
but they're in files I didn't touch and already exist on the base
commit, so I left them alone to keep this small. Happy to do a separate
cleanup PR.

---------

Co-authored-by: JD Davis <mxjerrett@gmail.com>
 2026-06-26 12:15:42 -05:00
Tejas Chopra
a99dc61424
feat: output-token reduction — verbosity shaper, per-user learning, counterfactual savings (#965) 
## Description

Adds the first levers that reduce the tokens the model **writes back**
(output), complementing Headroom's existing input compression. Output
costs 5× input on Opus-class models and is full of waste (ceremony,
restated code, deep "thinking" on routine steps). Two phases in one
self-contained PR off `main`: the request-side output shaper, then
per-user verbosity learning plus an honest counterfactual savings
estimator and dashboard surfacing.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **Output shaper** (`output_shaper.py`, opt-in
`HEADROOM_OUTPUT_SHAPER=1`): cache-safe verbosity steering appended to
the system-prompt tail (5 levels); effort routing that lowers
`output_config.effort` on mechanical tool-result continuations; legacy
`thinking.budget_tokens` clamp. Never injects effort where absent, never
toggles `thinking.type`.
- **`headroom learn --verbosity`**: mines Claude Code transcripts for
behavioral signals (interrupts, length-adaptive fast-skips, echo ratio),
recommends a verbosity level (heuristic + optional `--llm-judge`), and
seeds the savings baseline.
- **Counterfactual estimator** (`output_savings.py`): per-stratum
synthetic-control (estimated) + A/B holdout (measured) with a propagated
95% CI; conversation-stable arm assignment for A/B validity and
prefix-cache safety.
- **AIMD verbosity controller** (`verbosity_controller.py`):
additive-increase / fast-back-off state machine; live signal emission
gated off by default.
- **Wiring + surfaces**: shaper resolves the learned level; recording
rides the existing `transforms_applied` channel through the outcome
funnel (no `RequestOutcome` changes); `headroom output-savings` CLI;
dashboard "Output Tokens Saved" card.
- **Docs**: simple-words user guide + design doc with the counterfactual
methodology.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_output_savings.py tests/test_output_savings_cli.py \
        tests/test_verbosity_learn.py tests/test_verbosity_controller.py \
        tests/test_output_shaper.py -q
94 passed in 0.54s

$ pytest tests/test_request_outcome.py tests/test_handler_outcome_tag_invariant.py \
        tests/test_proxy_dashboard_stats_cache.py -q
44 passed

$ ruff format --check .
831 files already formatted

$ mypy headroom --ignore-missing-imports
Success: no issues found in 361 source files
```

## Real Behavior Proof

- Environment: macOS, Python 3.12 (`.venv`), `anthropic` 0.76, live API
model `claude-opus-4-8`.
- Exact command / steps: `HEADROOM_OUTPUT_SHAPER=1`; `headroom learn
--verbosity --apply` (seeds level + baseline); `python
scripts/eval_output_shaper.py A` (live before/after); simulate holdout
traffic then `headroom output-savings`.
- Observed result: code-review ask — baseline 1,750 output tokens → L2
1,354 (−22.7%) → L3 599 (−65.8%), same bugs found. `learn --verbosity`
on 24 real sessions → 11% interrupt / 26% fast-skip → L3 (high
confidence). Measured A/B path → 31.7% reduction (95% CI 27.7%–35.7%).
94 new tests + 44 existing outcome/dashboard tests green; ruff + mypy
clean.
- Not tested: live streaming-path recording exercised only via unit
tests (the `transforms_applied` funnel is shared across paths); runtime
AIMD signal emission is gated off by default and not exercised live.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

Output savings are counterfactual (we never observe what the model
*would* have written), so the estimator separates **estimated** (vs a
learned baseline) from **measured** (A/B holdout via
`HEADROOM_OUTPUT_HOLDOUT`) and always reports a confidence band — never
a single made-up number. CHANGELOG left unchecked (release-please
manages it). Runtime AIMD self-tuning is intentionally a TODO
(controller built/tested; live signal emission gated behind
`HEADROOM_VERBOSITY_AUTOTUNE`).
 2026-06-16 21:06:43 -07:00