akadmin/headroom
1
0
Fork 0
mirror of https://github.com/headroomlabs-ai/headroom.git synced 2026-08-27 14:17:10 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
2630 commits 410 branches 169 tags 142 MiB
Commit graph

3 commits

Author SHA1 Message Date
Tejas Chopra
0845b26ee6
fix(proxy/cost): record each request's savings exactly once (drop 3 double-counts) (#2545) 
## Description

An audit of savings accounting found three **double-count** bugs: the P0
outcome-funnel refactor centralized cost + PERF recording in
`emit_request_outcome`, but three pre-funnel emits were never removed,
so they fire a second time on their paths.

| Path | Stray emit | + Funnel | Effect |
|---|---|---|---|
| OpenAI chat direct, non-streaming | explicit
`cost_tracker.record_tokens` (`handlers/openai.py` ~4140) |
`outcome.py:418` | **2× spend / requests; budget period cost doubled** →
`check_budget` can block at half the real spend |
| OpenAI **Responses** buffered (Codex HTTP) | explicit `record_tokens`
(~5223) | `outcome.py:418` | same |
| Codex **WS** turns | explicit `PERF` log line (~7291) |
`outcome.py:482` | `headroom perf` **double-counts** saved + requests
every WS turn (analyzer sums per line, no dedup by request_id) |

All three are pure duplicates: the funnel's `cost_tracker.record_tokens`
is a **superset** of the explicit calls' args, and its PERF line uses
the **same per-turn deltas** (verified: `7246-7249` == the explicit
line's fields). The `/stats` headline was already correct
(SavingsTracker fires once, inside the funnel) — only cost/budget and
`headroom perf` were affected.

Closes #

## Type of Change
- [x] Bug fix (non-breaking)

## Changes Made
- Remove the explicit `cost_tracker.record_tokens` on the OpenAI chat
non-streaming path and the Responses buffered path — keep the
`cache_write`/`uncached` computation the funnel needs.
- Remove the duplicate WS PERF log line (+ its now-dead `_perf_*` locals
and the now-unused `_summarize_transforms` import).
- Add a regression test: cost is recorded exactly once on the
non-streaming chat path (was 2×).

## Testing
- [x] `ruff check` + `ruff format --check` clean; `mypy` clean
- [x] Regression + existing tests pass

### Test Output
```text
pytest tests/test_openai_chat_turn_hooks.py -q            → 6 passed  (incl. new double-count regression)
pytest tests/test_openai_responses_context_compaction.py  → 12 passed
pytest tests/test_openai_codex_ws_lifecycle.py + timings + savings_deferral → 38 passed
ruff/mypy → clean
```

## Real Behavior Proof
- **Verified by code trace**, not just tests: `grep
cost_tracker.record_tokens` across the handler now returns only the
funnel call (`outcome.py:418`); the explicit chat/Responses calls are
gone. The WS funnel outcome (`openai.py:7246-7249`) feeds
`outcome.py:482`'s PERF with the same deltas the deleted line used.
- **Not covered:** a related finding (OpenAI-chat *streaming* skips turn
hooks entirely, `openai.py:3484 "and not stream"`) is **intentionally
deferred** — that gate protects re-drive-requiring hooks (tool-router
deferral) which can't run mid-stream; a proper fix needs a per-hook
"safe-on-stream" capability flag, out of scope here.

## Checklist
- [x] Self-reviewed
- [x] No new warnings; tests pass locally
- [x] Did **not** edit `CHANGELOG.md`

## Additional Notes
This is the "sources" half of the savings audit. A companion PR will fix
the "sinks" half — tool-search/deferral savings are never aggregated
into `Metrics`, so the session summary, `cost.py` summary, `headroom
perf --json/csv`, and the `all_layers` total under-report them.
 2026-07-24 20:40:44 -07:00
Tejas Chopra
c371d5ad60
fix(proxy/perf): count turn-hook message folds in token accounting (#2520) 
## Description

Turn hooks (the `headroom.proxy.turn_hooks` seam used by proxy
extensions, e.g. the lossless-guard plugin) fold tool_result / message
content in `on_request`, which runs **after** the pipeline has already
computed `optimized_tokens`. The saving was recorded to `/stats` via
`record_compression`, but was invisible to the `PERF` log line and
`headroom perf` (both read the pipeline's `original → optimized` delta).
Net effect: a plugin that folded 463 tokens still logged `tok_saved=0`.

This makes the per-turn token accounting count the hook's fold too,
across all three handler paths.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **Anthropic Messages handler** (`/v1/messages`): re-count messages
right after `run_request_hooks`, regardless of whether the hook replaced
the list or mutated it in place. Attribute the fold as a `turn_hook`
transform. Only ever lowers `optimized_tokens`.
- **OpenAI Chat handler** (`handle_openai_chat`,
`/v1/chat/completions`): same re-count. The existing code re-counted
hook-modified *tools* but not the *message* fold — this closes that gap
and adds the `turn_hook` transform tag.
- **OpenAI Responses handler** (`_compress_openai_responses_payload`,
`/v1/responses`): the seam previously only wrote hook-modified *tools*
back — a folded/replaced `input` list was silently dropped and
uncounted. Now snapshot the message-items token count **before** the
hook (an in-place fold would corrupt a post-hook baseline), write back a
replaced list, and add the fold delta to `tokens_saved` (the same
channel the tool-schema savings already ride to `/stats` and `headroom
perf`).
- Key detail: the identity check `ctx.messages is not <orig>` is
insufficient — the lossless-guard plugin mutates messages **in place**,
so an identity-gated re-count misses it. The re-count runs
unconditionally whenever a hook ran.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ ruff check headroom/proxy/handlers/anthropic.py headroom/proxy/handlers/openai.py \
    tests/test_openai_chat_turn_hooks.py tests/test_openai_responses_context_compaction.py
All checks passed!

$ mypy headroom/proxy/handlers/anthropic.py headroom/proxy/handlers/openai.py
Success: no issues found in 2 source files

$ pytest tests/test_turn_hooks.py tests/test_openai_chat_turn_hooks.py \
    tests/test_openai_responses_context_compaction.py -q
tests/test_turn_hooks.py .........                                       [ 34%]
tests/test_openai_chat_turn_hooks.py .....                               [ 53%]
tests/test_openai_responses_context_compaction.py ............           [100%]
26 passed in 14.05s
```

New regression tests (each fails on the pre-fix code):
- `test_in_place_message_fold_is_counted` (chat path) — hook folds
message content in place; asserts `turn_hook` in `x-headroom-transforms`
and a recorded `tokens_saved > 0`.
- `test_responses_turn_hook_message_fold_is_applied_and_counted`
(Responses path) — hook folds a `function_call_output` in place; asserts
the outbound payload reflects the fold **and** `tokens_saved > 0`.

## Real Behavior Proof

- **Environment:** local proxy (`headroom proxy --port 8793
--proxy-extension lossless_guard`), `HEADROOM_LICENSE_DEV=1`,
`HEADROOM_PROTECT_TOOL_RESULTS=Bash` (so the fold is purely the plugin's
turn hook), model `claude-haiku-4-5`. Request carries a `gh --json`
object (folded to TOON) and a `docker pull` log.
- **Exact steps:** send the request → read the `PERF` line in
`~/.headroom/logs/proxy.log` and `GET /stats`.
- **Observed result:**
- Before this change: `PERF ... tok_before=607 tok_after=607 tok_saved=0
... transforms=none` while `/stats` reported `{"lossless_guard": 145}` —
i.e. the saving existed but perf showed nothing.
- After this change: `PERF ... tok_before=607 tok_after=484
tok_saved=123 ... transforms=turn_hook`, `/stats` still
`{"lossless_guard": 145}`. (`123` is the honest whole-request
`count_messages` delta; `145` is the per-content-string delta
`record_compression` measures — different scopes, both real and
positive.)
- **Not tested:** the OpenAI Chat and Responses paths were verified by
unit test, not a live client run — my live setup routes Claude Code
through the Anthropic handler only.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — N/A
(internal accounting; no public API/doc surface)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

Behavior is unchanged when no turn hook is registered
(`registered_turn_hooks() == []` → the re-count block is skipped), so
pure-OSS installs are byte-identical and unaffected. OSS's own pipeline
compression was already counted correctly (it runs before the hook);
this only surfaces the extension/turn-hook layer.
 2026-07-24 09:38:52 -07:00
Tejas Chopra
c9217856d3
Tejas/turn hooks extension (#1903) 
## Description

<!-- Briefly explain the change and why it is needed. -->

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- 

## Testing

<!-- Check what you actually ran, then paste the real command output
below. -->

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
# Paste relevant command output or artifact links here
```

## Real Behavior Proof

- Environment:
- Exact command / steps:
- Observed result:
- Not tested:

## Review Readiness

- [ ] I have performed a self-review
- [ ] This PR is ready for human review

## Checklist

- [ ] My code follows the project's style guidelines
- [ ] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

Add screenshots to help explain your changes.

## Additional Notes

<!-- Mention any N/A checklist items, tradeoffs, follow-ups, or
maintainer context. -->
 2026-07-09 07:49:06 -07:00