mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
9 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Suliman Abdulrazzaq
|
620028fa18
|
fix(proxy): emit request log timestamps in UTC
## Description `RequestLog.timestamp` was serialized with `datetime.now().isoformat()`, which omits timezone information. Browsers then interpret the value as local time, so requests from a UTC container can display negative ages in non-UTC dashboards. Closes #2910 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - Emit request-log timestamps from `datetime.now(timezone.utc)` so the ISO-8601 value includes `+00:00`. - Add a regression test that parses the emitted timestamp and requires a UTC offset. ## Testing - [x] New tests added for the regression - [x] `python -m compileall -q headroom/proxy/outcome.py tests/test_request_outcome.py` - [x] `git diff --check` - [ ] Unit tests pass (`pytest`) — the repository's Rust extension cannot build in this Windows environment because `link.exe` (MSVC) is unavailable; the focused test is included for CI. ### Test Output ```text python -m compileall -q headroom/proxy/outcome.py tests/test_request_outcome.py (pass) git diff --check (pass) uv run pytest tests/test_request_outcome.py -q blocked while building headroom-py: linker `link.exe` not found ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.11; proxy timestamps are generated in `headroom/proxy/outcome.py`. - Exact command / steps: traced the Recent Requests write path and added a timestamp assertion in `tests/test_request_outcome.py` (CI will run with the project's Rust toolchain). - Observed result: the production call now emits an ISO-8601 timestamp with `+00:00`; the regression assertion requires an offset-aware UTC value, preventing browser timezone skew. - Not tested: full pytest suite locally because the MSVC linker is unavailable. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review - [x] I have added tests that prove my fix is effective - [x] I did not edit `CHANGELOG.md` Signed-off-by: Suliman Abdulrazzaq <suliman9000a@gmail.com> |
||
|
Tejas Chopra
|
a033ac4176
|
fix(cost): send litellm the total prompt so --budget stops seeing $0 (#2757)
## Description **`--budget` has been silently inert on any cache-warm request** — which is the normal case in an agent session. This is a disabled control, not a metrics bug. `record_tokens` passed only the **uncached slice** as litellm's `prompt_tokens`. Measured, `litellm.cost_per_token` charges: ``` (prompt_tokens - cache_read - cache_creation) * input_rate + cache_read * read_rate + cache_creation * write_rate ``` So `prompt_tokens` is the **whole** prompt and litellm removes the cached parts itself. Handing it the uncached slice drives the input term **negative** as soon as anything is cached. `estimate_cost` ends with `float(total) if total > 0 else None`, so it returned `None`, no `CostEntry` was appended, and `check_budget()` saw **$0**. | model | 100k prompt, 80k cached — old call | booked | | --- | --- | --- | | `gpt-5` | **-$0.065000** | None | | `gpt-4o-mini` | **-$0.003000** | None | | `claude-sonnet-4-5` | **-$0.156000** | None | Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made Fixing the total alone would **over-charge OpenAI**, because two bugs are entangled here. OpenAI exposes no cache-write counter, so `_infer_openai_cache_write_tokens` uses the uncached portion as a write proxy. At two of the four inference sites (`openai.py:4304`, `openai.py:5413`) `uncached_input_tokens` is derived by subtracting **only** `cache_read`, so `cache_write_tokens` and `uncached_tokens` are **the same tokens**. Summing all three would double-count the prompt and charge a write premium OpenAI does not have. (The other two sites — `openai.py:3969` and `streaming.py:2024` — subtract both, so their buckets are genuinely disjoint; those are left alone.) - `cost.py` — `record_tokens` now passes `uncached + cache_read + cache_write` as the prompt total. - `cost.py` — new `cache_inferred: bool = False` parameter. When set, the inferred write is excluded from **both** the prompt total and the write premium. The default preserves behaviour for every provider that reports disjoint buckets. - `outcome.py` — plumbs `outcome.cache_inferred` through. The field already existed on `RequestOutcome` for the dashboard; it just never reached cost. - `handlers/openai.py` — sets `cache_inferred=True` at the two outcome sites whose buckets genuinely duplicate. ## Testing - [x] Unit tests pass (new file) - [x] Linting passes (`ruff check` + `format --check`, pinned 0.15.17) - [x] New tests added - [x] Manual testing performed ### Test Output ```text $ uvx ruff@0.15.17 check headroom/proxy/cost.py headroom/proxy/outcome.py headroom/proxy/handlers/openai.py All checks passed! $ uvx ruff@0.15.17 format --check <same three> 3 files already formatted $ pytest tests/test_cost_budget_total_prompt.py -q 5 passed in 0.26s ``` The 5 new tests assert on the **arguments handed to `estimate_cost`** rather than on dollar values, so they pin the contract that broke without depending on litellm's pricing tables — or on litellm being installed at all. ## Real Behavior Proof - **Environment:** macOS 26.4 arm64, isolated git worktree off `upstream/main`. **litellm's actual formula**, established by probe rather than assumed: ```text rates (claude-sonnet-4-5): input=3e-06 read=3e-07 write=3.75e-06 total=100k, 80k read, 5k write -> $0.087750 hypothesis (p-r)*ir + r*rr + w*wr = $0.102750 x hypothesis (p-r-w)*ir + r*rr + w*wr = $0.087750 <- matches ``` **Before → after:** ```text Anthropic, disjoint: uncached=900 read=48000 write=1500 OLD prompt=900 raw=-0.125775 booked=None <-- BUDGET BLIND NEW prompt=50400 raw= 0.022725 booked=0.022725 OpenAI, inferred write == uncached: uncached=20000 read=80000 write=20000 OLD prompt=20000 raw=-0.090000 booked=None <-- BUDGET BLIND NEW prompt=100000, write excluded raw= 0.035000 booked=0.035 truth=0.035000 ``` The OpenAI "after" equals the hand-computed truth `20,000*input + 80,000*read_rate` exactly. - **Not fully tested locally.** The cost/outcome suites show an **identical 10-failure set** on this branch and on clean `upstream/main` in the same throwaway env — all `ModuleNotFoundError: headroom._core`, the compiled Rust extension this machine cannot currently build. So they are environment-only, not regressions. One of them, `test_funnel_passes_canonical_record_tokens_shape`, covers the `record_tokens` call shape this PR changes, so **CI is the authoritative check for that one**. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] I did **not** edit `CHANGELOG.md` ## Context Found during a tokenizer-consistency audit that also turned up: HuggingFace-routed models counting a 6,000-char message as **2 tokens**, `gpt-5`/`o4-mini` falling to a char estimator, and the Kompress size gate missing its own cap by 24%. Those are separate PRs — different subsystems, different risk. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
nangsontay
|
184146b688
|
fix(savings): surface request growth the tok_saved clamp swallows (#2708)
## Description `tokens_saved` is clamped at zero, so a request the proxy forwards **larger** than it arrived is indistinguishable in the PERF line from one it simply could not compress. Both read `tok_saved=0`. That ambiguity hides real regressions. Anything that appends to the body after compression — proactive context expansion, memory injection — can outweigh the compression it sits on top of and still look like a neutral turn. On the session that prompted this, a request went from 55,161 tokens in to 57,845 out and reported `tok_saved=0`, for 19 consecutive turns, with nothing in the logs distinguishing it from a turn with nothing left to compress. This reports the swallowed amount as `tok_inflated`. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `RequestOutcome.tokens_inflated`: `max(0, optimized_tokens - original_tokens)`, derived from two counts the outcome already carries — **no new plumbing at any of the emit sites**. - Added `tok_inflated=` to the PERF log line, next to `tok_saved=`. Diagnostic only, deliberately. It does **not** feed `tokens_saved` or `attempted_input_tokens`, for two reasons: 1. `attempted_input_tokens = optimized_tokens + tokens_saved` is a *size*, not a signed delta. Letting the second term go negative makes it smaller than the bytes actually forwarded, corrupting the active-savings denominator. 2. Injection paths already book their own cost through the retrieval-drawback channel. A negative landing in `tokens_saved` as well would count the same loss twice. So the clamp stays and the hidden number surfaces beside it. Worth noting there is already a revert-on-inflation guard *before* compression's own inflation can escape (`anthropic.py`, "Optimization inflated tokens … reverting to original messages") — it is only growth added *after* that point which the clamp was silently absorbing. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_request_outcome.py tests/test_cli_perf_format.py -q ============================== 54 passed in 1.56s ============================== $ pytest tests/ -q -k "outcome or perf or savings or stats" ======= 510 passed, 33 skipped, 9674 deselected, 542 warnings in 40.37s ======== $ ruff check headroom/proxy/outcome.py tests/test_request_outcome.py All checks passed! $ ruff format --check headroom/proxy/outcome.py tests/test_request_outcome.py 2 files already formatted $ mypy headroom/proxy/outcome.py --ignore-missing-imports Success: no issues found in 1 source file ``` Four new tests pin the distinction that was missing: shrank (0), no-op compression (0, and `tok_saved` also 0 — the two cases that used to look identical), grew (reports 2684 while `tok_saved` stays 0), and that `attempted_input_tokens` / `savings_pct` keep their unsigned semantics. `tests/test_cli_perf_format.py` parses hand-written PERF fixtures by field name, so adding a field does not disturb it — verified green above. ## Real Behavior Proof ### The field catching a real inflating request - Environment: macOS 15 (arm64), Python 3.13.14. A proxy booted from this branch: `headroom proxy --mode token --backend anthropic --anthropic-api-url http://127.0.0.1:<stub>`, isolated `HOME` so the run could not touch a developer's live logs/store, `HF_HOME` pointed at cached kompress weights so the lossy+CCR-marker path is exercised and proactive expansion can actually arm. - Exact command / steps: three requests over one conversation through the real HTTP path (`x-headroom-cwd: /tmp/proof`, `user-agent: claude-code/1.4.2`): a user turn carrying the real `~/.claude/rules/*.md` text (~8.2k tokens); then `assistant` + a short user turn so that block becomes compressible and gets tracked as a CCR entry; then a follow-up whose leading text block shares vocabulary with it, so proactive expansion fires and appends the original — which is how a request ends up leaving larger than it arrived. PERF lines read from the isolated `~/.headroom/logs/proxy.log`. - Observed result: real PERF output from that run — ```text msgs=1 tok_before=8170 tok_after=9553 tok_saved=0 tok_inflated=1383 ... transforms=router:text_block:mixed msgs=3 tok_before=8184 tok_after=9567 tok_saved=0 tok_inflated=1383 ... transforms=router:text_block:mixed msgs=5 tok_before=8245 tok_after=11880 tok_saved=0 tok_inflated=3635 ... transforms=router:text_block:mixed ``` Correlated from the same run: `CCR Tracker: Proactively expanded f0cf4efb42373ec225f57725 (1417 items)`, and the stub upstream confirms the block reached the wire (`has_expansion_block: true`, forwarded body 54,130 B on the third request). Every one of those turns reports `tok_saved=0`. Before this change that is all the log said, and it is the same thing it says when there was simply nothing left to compress. `tok_inflated=3635` is the number that was missing. For contrast, the same scenario run against a build where the request genuinely shrinks reported `tok_before=8245 tok_after=7359 tok_saved=886` — that build predates this field, so it does not print `tok_inflated`; the point is only that the inflating and shrinking cases are the two states the field has to separate, and on `main` today both render as `tok_saved=0` whenever the growth path is taken. The `tok_inflated=0` case on a shrinking request is covered by unit test. ### Scale of what was hidden From a live `--mode token` proxy on Claude Code traffic across four rotated logs: **305 of 3,743 requests (8%)** had `tok_after > tok_before` while every one reported `tok_saved=0` — 192,829 tokens of growth rendered as "nothing to compress". The worst single session held +2,643/turn for 19 consecutive turns. - Not tested: the `headroom perf` CLI was not run against a real log file containing the new field — it parses by field name and `tests/test_cli_perf_format.py` is green, but that is test-level rather than end-to-end evidence. Streaming responses were not exercised (the stub replies non-streaming), so the streaming emit path carries the new field on the strength of sharing `emit_request_outcome` rather than by observation. No dashboard or Prometheus consumer was re-run. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — one added field on an existing log line; no user-facing surface. ## Additional Notes - Independent of #2706 and #2707 — verified `conflicts=0` via `git merge-tree`; mergeable in any order. - Related but deliberately out of scope: `main` has no producer for retrieval-cost accounting (`record_savings_event` takes no `kind`/`tokens_retrieved`, and nothing writes `tokens_retrieved` anywhere), so proactive expansion's cost is not booked into net savings at all. Adding that channel is a cross-cutting accounting change and belongs in its own PR; this one only makes the growth visible in the log. |
||
|
inix
|
4aac068814
|
fix(proxy/metrics): move the savings-ledger append off the event loop (#2439)
## Description `PrometheusMetrics.record_request` appends one durable JSONL event per compressed request. That append is synchronous: `open` + `fcntl.flock` + `write`, plus a full-file rewrite once the ledger passes 1 MB. It runs on the event loop, inside `self._lock`. `export()` takes that same lock and holds it for the entire Prometheus serialization, so a slow ledger write stops `/metrics` cold. In a repro run of 200 compressed requests, `/metrics` completed zero scrapes and the event loop never yielded once across 6.4 seconds. The append now runs in a thread, outside the lock. `savings_ledger` already takes its own `flock` across processes, so the metrics lock was never what made the write safe. Both halves are one change. Awaiting inside the lock would hold it for the whole write rather than just the syscall, which is worse than what is on main today. The file already documents this hazard against itself. `record_stage_timings` (`prometheus_metrics.py:867-874`) picks a plain `threading.Lock` over `self._lock` specifically because "the async lock is also held by `export()` during Prometheus scrapes." The ledger append was the pattern that docstring warns about. No filed issue for this one. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Move the `savings_ledger.record_savings_event` call in `record_request` out of `async with self._lock` and run it through `asyncio.to_thread`. The call site keeps its keyword arguments verbatim; `to_thread` forwards `**kwargs`, so no `functools.partial` wrapper is needed. - Keep the `await`. Callers still see the event on disk when `record_request` returns, which `tests/test_savings_ledger_before_forwarded.py` asserts synchronously. - Add `tests/test_savings_ledger_offload.py`: lock scope, event-loop responsiveness, durability on return, and both arms of the `tokens_saved > 0 and not stateless` gate. `savings_ledger.py` is untouched. It stays synchronous so the MCP `headroom_compress` caller in `ccr/mcp_server.py:789` does not have to change. Sizing the executor is left alone on purpose. `asyncio.to_thread` uses the default pool, which is the documented tool for blocking I/O and already the idiom here (`helpers.py:1297`, `server.py:1694`, `:3557`, `:3613`, `:4244`). The compression pools are sized `max(1, os.cpu_count())` for CPU-bound work, and `PrometheusMetrics` holds no reference to `HeadroomProxy` anyway, so reaching them would mean a new constructor parameter. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_savings_ledger_offload.py tests/test_savings_ledger.py tests/test_savings_ledger_before_forwarded.py -q ======================== 26 passed, 1 warning in 4.08s ========================= $ ruff check . && ruff format --check headroom/proxy/prometheus_metrics.py tests/test_savings_ledger_offload.py All checks passed! 2 files already formatted $ mypy headroom/proxy/prometheus_metrics.py Success: no issues found in 1 source file ``` Broader sweep across the blast radius, 145 test files matching savings / metrics / outcome / stats / proxy / handler / server / ledger / cost / prometheus, each run under a per-file wall-clock watchdog: ```text 138 files pass, 1470 tests passed 7 non-green: HANG tests/test_agent_savings.py HANG tests/test_ccr_mcp_server.py HANG tests/test_netcost_gate.py HANG tests/test_proxy_compress_endpoint.py HANG tests/test_proxy_mode_benchmark.py HANG tests/test_read_maturation_handler_nobust.py FAIL tests/test_proxy_copilot_auth_hooks.py::test_openai_passthrough_applies_copilot_auth Same 7 files re-run with headroom/proxy/prometheus_metrics.py reverted to |
||
|
Ashish
|
a14ab45cf0
|
fix(proxy): make budget enforcement actually work (#885)
## Description
`CostTracker._costs` was initialized but never written to, so
`get_period_cost()` always returned `0` and `check_budget()` always
returned "allowed" — the `--budget` flag was a silent no-op.
`_prune_old_costs()` was dead code with zero callers. This makes budget
enforcement actually work: requests are rejected once the configured
limit is reached.
Closes # <!-- no tracked issue; discovered during a proxy-pipeline audit
-->
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- **`headroom/proxy/cost.py`** — `record_tokens()` now computes the
request cost via `estimate_cost()` and appends it to `_costs`,
activating `_prune_old_costs()`. When a call site has no API usage
breakdown (cache/uncached all zero), `tokens_sent` is used as the input
count so input cost is not silently dropped. `COST_RETENTION_HOURS` 24 →
744 so retention covers the longest budget period (monthly sums from the
1st; 24h retention would have under-enforced monthly budgets).
- **`headroom/proxy/outcome.py`** — the request funnel passes
`output_tokens` through to `record_tokens()` so costs include output,
for all providers.
- **`headroom/cli/proxy.py`** — added `--budget-period
[hourly|daily|monthly]` (env `HEADROOM_BUDGET_PERIOD`); it existed in
`ProxyConfig` and the server entry point but was unreachable from the
main CLI. Fixed the `--budget` help text that wrongly said "resets at
midnight UTC".
- **`headroom/cli/main.py`** — minor registration/version plumbing.
- Tests: regression coverage for the full `record_tokens →
get_period_cost → check_budget` chain, the `tokens_sent` fallback, and
the `--budget-period` flag/env wiring.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_cost_tracker_counterfactual.py tests/test_request_outcome.py -q
40 passed
$ ruff check headroom/proxy/cost.py headroom/proxy/outcome.py headroom/cli/proxy.py
All checks passed!
$ mypy headroom/proxy/cost.py headroom/proxy/outcome.py headroom/cli/proxy.py --ignore-missing-imports
Success: no issues found
```
## Real Behavior Proof
- Environment: local macOS, Python 3.11, branch `fix/budget-enforcement`
at the PR head commit.
- Exact command / steps: `pytest
tests/test_cost_tracker_counterfactual.py::test_budget_enforced_after_recording_costs
-v` — sets `CostTracker(budget_limit_usd=0.0001)`, records ~$1.50 of
Sonnet input, then asserts `check_budget()` returns not-allowed with
`remaining == 0`.
- Observed result: budget is now enforced — `get_period_cost()` reflects
real spend and `check_budget()` rejects once the limit is exceeded (the
proxy returns HTTP 429 on that path). On `main` the same test fails
because `_costs` is never populated and `check_budget()` always returns
allowed.
- Not tested: live end-to-end rejection against a running proxy with
real upstream traffic; the running proxy needs a restart on this version
to pick up the fix.
```text
$ pytest tests/test_cost_tracker_counterfactual.py::test_budget_enforced_after_recording_costs \
tests/test_cost_tracker_counterfactual.py::test_budget_input_cost_counted_without_usage_breakdown -v
2 passed
```
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
N/A — CLI/backend change with no UI surface. See **Test Output** and
**Real Behavior Proof** above for terminal evidence.
## Additional Notes
- The `ci.yml` coverage-upload change originally added here (commit
`
|
||
|
JD Davis
|
3c77e52ce4
|
feat: add Vertex AI proxy routing (#793)
## Description Adds first-class GCP Vertex AI proxy routing for publisher REST endpoints so Vertex requests are forwarded to a configurable regional Vertex host instead of falling through to the generic OpenAI/Anthropic/Gemini passthrough selection. Fixes #792 ## Type of Change - [x] New feature (non-breaking change that adds functionality) - [x] Documentation update ## Changes Made - Added a `vertex` provider target with `VERTEX_TARGET_API_URL` and `--vertex-api-url` support. - Registered explicit Vertex publisher routes for Google `generateContent`, `streamGenerateContent`, `countTokens` and Anthropic publisher `rawPredict`, `streamRawPredict` passthrough. - Added startup banner/routing output for Vertex AI. - Added focused tests for provider target resolution, CLI/env config, banner output, and route delegation. - Added `wiki/vertex.md` with usage examples and Google Cloud source links. ## Sources - Vertex AI Gemini inference reference: https://docs.cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference - Google Cloud REST authentication: https://docs.cloud.google.com/docs/authentication/rest - Google Application Default Credentials: https://docs.cloud.google.com/docs/authentication/application-default-credentials ## Testing - [x] Linting passes (`python -m ruff check .`) - [x] New tests added for new functionality - [x] Focused unit tests pass - [ ] Full unit suite completed locally - [ ] Rust tests completed locally - [ ] Type checking passes locally ## Test Output ```text $ python -m ruff check . All checks passed! $ python -m pytest tests/test_provider_registry.py tests/test_provider_proxy_routes.py tests/test_cli_proxy_env.py tests/test_banner_upstream_targets.py -q 57 passed, 1 warning in 13.75s ``` Local limitations: - `python -m pytest tests scripts/tests -q` timed out after 1 hour on this Windows machine before completing. - `cargo test -p headroom-proxy --test integration_vertex_raw_predict` could not run because `cargo` is not installed on PATH in this environment. - The commit hook's `mypy` step fails locally on an existing Windows `fcntl` typing issue in `headroom/subscription/tracker.py`; `ruff`, `ruff-format`, and plugin-version hooks passed, and the commit was made with only `mypy` skipped. ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have made corresponding changes to the documentation - [x] I have added tests that prove the feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable |
||
|
chopratejas
|
694589fec4 |
refactor(proxy): collapse 3 stream finalizers onto RequestOutcome.from_stream
Three streaming finalizers — ``_finalize_stream_response``,
``_stream_response_bedrock``, ``_stream_openai_via_backend`` — each
duplicated the same set of body- and config-derived fields when
constructing a ``RequestOutcome``:
* ``attempted_input_tokens = optimized_tokens + tokens_saved``
* ``num_messages = len(body.get("messages", []))``
* ``request_messages`` conditional on ``config.log_full_messages``
* ``transforms_applied`` list → tuple (frozen-dataclass contract)
* ``tags or {}`` normalization
* ``turn_id`` via ``compute_turn_id``
The last one was a real bug. Only the Bedrock site computed
``turn_id`` — sites 1 and 3 silently dropped it, breaking the
dashboard's multi-turn-session grouping for every Anthropic-SSE and
OpenAI-via-backend request. The new ``RequestOutcome.from_stream``
classmethod computes it uniformly so the three finalizers cannot
drift apart on derivation logic again.
Each call site now hands ``from_stream`` the body + provider-specific
cache/timing fields and gets a fully-constructed outcome back. The
funnel call after it stays identical (``await
self._record_request_outcome(outcome)``).
|
||
|
chopratejas
|
96a674f1b9 |
feat(proxy): add client (harness) identification — per-harness analytics across every handler
Cashes in the RequestOutcome refactor with a typed-field surface that
gives EVERY handler per-harness visibility — Codex / Claude Code /
aider / Cursor / Zed / opencode / DROID / antigravity / etc. — for one
field-add across the contract.
The one-field-add proof
Headroom went from "what fraction of OUR requests come from which
harness?" being unanswerable (handlers logged ad-hoc User-Agent strings
in heterogeneous tag dicts at 18 sites, with 9 of 18 not even
populating them) to one structured ``client: str | None`` value on
every observation flowing through the funnel. No new bookkeeping at
call sites; every handler picks it up via a single
``classify_client(headers)`` call at request entry.
Implementation
* New ``CLIENT_UA_MAP`` + ``classify_client()`` in
``headroom/proxy/auth_mode.py``. Substring match against
User-Agent; ``X-Client`` header overrides UA. Returns ``str | None``
so ``None`` is the loud "unidentified" signal rather than a silent
empty bucket.
* New ``RequestOutcome.client: str | None = None`` field.
* Funnel updates (in ``outcome.py``):
- Appends ``client=X`` to the PERF log line ONLY when set, so
``headroom perf --client X`` parsing stays clean for
unidentified traffic (no bogus ``client=`` token).
- Copies ``client`` into ``RequestLog.tags["client"]`` so the
dashboard's existing tag-based filtering surfaces per-harness
slicing with zero new columns.
* Every handler that constructs a RequestOutcome now passes
``client=client`` — wired across streaming.py (3 finalizers,
with ``_finalize_stream_response`` gaining a new optional kwarg
since it doesn't have direct access to headers), anthropic.py
(6 sites), openai.py (8 sites including Codex WS), gemini.py
(2 emitting sites), batch.py (5 sites).
Harnesses recognised
Anthropic ecosystem: claude-code, claude-cli, claude-vscode,
anthropic-cli
OpenAI ecosystem: codex-cli
Editors: cursor, zed
AI coding harnesses: aider, droid, opencode, github-copilot
Other: antigravity (Google experimental)
Adding a new client is a one-line edit to ``CLIENT_UA_MAP``.
Tests
* 8 new tests in ``test_request_outcome.py`` covering:
- ``client`` field round-trips on the value type
- ``classify_client`` against every recognised UA prefix
- ``X-Client`` header override beats UA match
- ``None`` for unknown traffic (the loud signal)
- Funnel appends ``client=X`` to PERF when set
- Funnel OMITS ``client=`` from PERF when None (no bogus empty)
- Funnel stamps ``client`` into ``RequestLog.tags``
* All 228 existing tests still pass (full sweep across streaming,
cache, Codex, Anthropic, OpenAI, Gemini, batch, auth-mode).
* ruff + ruff-format + mypy clean.
What's now true that wasn't before
Once this lands, the dashboard can answer:
* "Show me cache hit rate by harness"
→ ``GROUP BY tags.client FROM request_log``
* "Which harness contributes the most cache writes?"
→ same
* "Per-harness savings ratio"
→ same
* ``headroom perf --client codex`` / ``--client claude-code``
→ analyzer filters PERF log lines on ``client=X`` token
Zero new bookkeeping in handlers. Zero changes to Prometheus label
cardinality (kept the client dimension out of Prometheus on purpose —
the tags route is the right surface). The "what's our traffic split
by harness?" question is now answerable in three places (PERF log,
RequestLog tags, dashboard widgets that already filter on tags)
without any per-provider work.
|
||
|
chopratejas
|
e898f68b89 |
refactor(proxy): introduce RequestOutcome funnel; collapse 3 streaming finalizers
P0 audit (docs/superpowers/specs/P0-proxy-pipeline-audit.md) catalogued **18 metrics.record_request call sites** across 4 handler files with **4 distinct argument shapes**: 9 of 18 omitted `cached=`, 7 of 18 omitted `attempted_input_tokens=` (= bug #454/#455's "headline 0%"), only 4 sites emitted a `PERF` log line (= bug #327's "msgs=0" sibling — Codex traffic invisible to `headroom perf`), and `cache_hit` was hardcoded `False` at 9 of 18 RequestLog sites. The cause was structural, not tactical: every site was independently deciding what "record this completed request" meant. This PR puts a single value type + a single function between the handlers and the metrics layer. Two new files: * `headroom/proxy/outcome.py` — `RequestOutcome` frozen dataclass. Captures everything we ever need to record about one completed request: identity, tokens, cache stats (per-TTL splits + inferred flag for OpenAI), timing, transforms, diagnostics. Provider-specific fields default to neutral values so non-Anthropic handlers don't have to know about 5m/1h splits, non-OpenAI handlers don't have to know about inferred writes, etc. Computed properties (`cache_hit`, `cache_hit_pct`, `savings_pct`) make "forgot to compute it" mistakes structurally impossible. * `HeadroomProxy._record_request_outcome` in `server.py` — the single funnel. Owns the four downstream effects in canonical order: 1. `metrics.record_request(...)` with the FULL kwarg set 2. `cost_tracker.record_tokens(...)` with `(model, tokens_saved, optimized_tokens)` positional + all cache kwargs 3. `logger.log(RequestLog(...))` with `cache_hit` correctly derived 4. structured `PERF` log line in the canonical key=value shape Migrated three streaming finalizers in this PR: * `_finalize_stream_response` (Anthropic native + OpenAI HTTP streaming) * `_stream_response_bedrock` (Bedrock-native Anthropic streaming) * `_stream_openai_via_backend` (OpenAI/Azure backend via LiteLLM/AnyLLM) All three previously had inline, drifted versions of the four-call sequence. Each is now ~70 fewer lines: build a `RequestOutcome` from local context, call `self._record_request_outcome(outcome)`. The prefix-tracker mutation (Anthropic-specific) stays outside the funnel — different concern. Six more migrations queued for follow-up PRs (handle_anthropic_messages 6 sites, handle_openai_chat, handle_openai_responses, handle_openai_ responses_ws 2 sites, handle_gemini_*, handle_databricks_invocations). Each is mechanical now. Tests * New: `tests/test_request_outcome.py` — 14 tests covering value-type contract (frozen, derived properties, neutral defaults) + funnel contract (full record_request kwargs, canonical record_tokens shape, derived cache_hit in RequestLog, PERF log key=value format, optional cost_tracker/logger). Bind the real production method via descriptor binding so the test exercises the real implementation, not a fork. * All 135 existing streaming/cache/Codex tests pass with zero regressions (`tests/test_backend_streaming_cache_metrics.py`, `test_proxy_streaming_request_logger.py`, `test_proxy_streaming_resilience.py`, `test_proxy_anthropic_cache_stability.py`, `test_openai_codex_*`, `test_responses_ws_pyo3_compression.py`, `test_anthropic_pre_upstream_backpressure.py`). * `mypy headroom/proxy/{outcome,server,handlers/streaming}.py` clean. * `ruff check` clean. Surface impact * −238 lines from `handlers/streaming.py` (deduplication). * +92 lines in `server.py` (the funnel — counted ONCE, not 18×). * +130 lines in new `outcome.py` (frozen dataclass + docstrings). * Net production code: ~−16 lines today, ~−500 lines after the remaining six migrations land. Forward design constraints (per docs/superpowers/specs/P0-proxy-pipeline-audit.md §7) * KISS: one value type, one function, no factory hierarchies. * No regex in routing — handlers stay provider-specific in their upstream contract. Output unification only. * No silent fallbacks — `cache_hit` is computed, not defaulted. `cache_inferred=True` is the loud signal when OpenAI write count came from `_infer_openai_cache_write_tokens`. * PERF format frozen so `headroom/perf/analyzer.py` keeps parsing cleanly; P3 follow-up replaces the free-text shape with a structured event. |