mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
27 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
JD Davis
|
8a1d38bc5d
|
fix(proxy): complete stateless Responses and buffered CCR lifecycle (#2997)
## Description
Consolidates the related OpenAI Responses ZDR/stateless continuation and
buffered CCR response-lifecycle corrections on current main. It
preserves client storage policy, makes Headroom-owned continuations
stateless across HTTP and WebSocket, and prevents buffered streaming
paths from committing a false HTTP 200 before the real upstream outcome
is known.
Closes #2675
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Preserves explicit and omitted Responses `store` policy instead of
forcing provider storage or disabling memory tools.
- Replays normalized input, replayable outputs, encrypted reasoning
content, and Headroom function outputs without `previous_response_id`.
- Applies the same stateless continuation policy to HTTP and WebSocket.
- Prevents transparent memory execution after client-visible WebSocket
output.
- Delays buffered CCR ASGI status/headers until the operation resolves
for Anthropic Messages and OpenAI Responses.
- Preserves real 429/5xx status and retry headers.
- Converts malformed non-JSON/non-SSE upstream 200 replies to a
sanitized 502 protocol error.
- Preserves valid JSON-to-SSE synthesis and existing SSE adaptation.
- Removes unreachable task cleanup left behind after replacing the old
keepalive polling loop with a direct awaited operation.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
118 passed across the changed HTTP/WS ZDR, lifecycle, and both-provider CCR suites
11526 tests collected with no collection errors
ruff check .: All checks passed
ruff format --check .: 1411 files already formatted
mypy headroom/proxy/handlers/anthropic.py headroom/proxy/handlers/openai.py:
Success: no issues found in 2 source files
```
Exact-head CI is entirely green on
`
|
||
|
JD Davis
|
039cd2431a
|
fix(proxy): preserve merged session and quarantine contracts (#2943)
## Description Forward-fixes two integration contracts exposed while auditing the large August 12 merge batch on `main`. The Codex WebSocket request-ID hardening correctly gave every emitted dashboard/feed row a unique ID, but it also changed the human-readable `PERF` prefix from the stable WebSocket session ID to that per-emission ID. That broke operator correlation and the contract documented by the original merge. This PR separates storage identity from log correlation: rows remain unique, while `PERF` lines remain grouped under the session ID. The same audit found two tokenizer quarantine tests still modeling the pre-time-cap behavior. Timeout debt no longer activates quarantine after its deadline expires. The tests now establish a live deadline and therefore continue to exercise the intended fail-open branch without weakening the production guard. ## Changes - Add an optional `RequestOutcome.perf_request_id` correlation field, defaulting to the existing `request_id` behavior for all current callers. - Set that field to the stable session ID for both per-turn and residual Codex WebSocket emissions. - Strengthen the lifecycle regression test to prove the unique feed-row ID is not used as the `PERF` prefix. - Update tokenizer quarantine tests to model an active, time-capped quarantine. This is a forward fix; it does not revert the unique WebSocket request IDs or the time-capped quarantine behavior. ## Merge-batch audit context - Audited 48 squash merges from ` |
||
|
Rod Boev
|
d02df10758
|
fix(proxy): give each Codex /v1/responses WS turn a unique request_id (#2164)
## Description
After any Codex traffic, the dashboard "Recent Requests" table goes
blank — including the unrelated Anthropic/Claude rows — even though the
proxy is actively handling and compressing Codex `/v1/responses`
WebSocket turns and aggregate counters keep moving. The feed isn't
stale; it is being wiped client-side.
Root cause: the Codex WebSocket handler
`OpenAIHandlerMixin.handle_openai_responses_ws`
(`headroom/proxy/handlers/openai.py`) mints a single `request_id` per
WebSocket **session** (`_next_request_id()` near the top of the handler)
and reuses it for every per-turn `RequestOutcome` in
`_record_ws_response_metrics`, the session-residual outcome, and the
session-summary `RequestLog`. Those all flow through
`emit_request_outcome` (`headroom/proxy/outcome.py`), which writes a
`RequestLog` per outcome into the request logger that backs
`/stats.recent_requests` and `/transformations/feed` — so one session
with N turns produces N+ feed rows sharing one `request_id`. The
dashboard renders that feed with `<template x-for="req in
(stats.recent_requests || [])" :key="req.request_id">`
(`headroom/dashboard/templates/dashboard.html:1298`); Alpine requires
unique `:key`s, so duplicate ids abort the entire `x-for` render and
blank the whole table. Anthropic/HTTP requests each get a unique
incrementing id from `_next_request_id()` and are unaffected — which is
why only Codex traffic triggers the blanking.
This PR gives each Codex WS feed emission a fresh unique id from the
same authoritative `_next_request_id()` counter (per-turn, residual, and
summary sites), restoring the "one unique id per feed row" invariant
that Anthropic already satisfies. With unique ids the Alpine `:key`s no
longer collide and the table renders Codex turns like any other request.
Feed-row counts, per-turn token and savings values, ordering, and
per-session metrics/cost bookkeeping are unchanged; the `[{session
request_id}]` log prefixes still use the session id so a session's log
lines stay greppable together.
Scope: this is the backend root-cause fix. Hardening the dashboard
`:key` against duplicate/`null` keys is a separate render-robustness
change and is deliberately left to a follow-up (`Refs #310`); once the
backend guarantees unique ids, the collision that blanks the table is
gone. The comment's secondary `savings_percent.toFixed(0)` concern is
already resolved on `main` (the row uses `formatOptionalPercent`).
Closes #310. The concrete duplicate-`request_id` diagnosis and the live
`/stats?cached=1` capture came from @sphynxttl's comment on the issue.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/handlers/openai.py`: in `handle_openai_responses_ws`,
mint a fresh `request_id` from `_next_request_id()` at each request-feed
emission — the per-turn `RequestOutcome` in
`_record_ws_response_metrics`, the session-residual `RequestOutcome`,
and the session-summary `RequestLog` — instead of reusing the one
session id. The per-turn id is minted after the existing all-deltas-≤0
early-return, so no-op turns still emit nothing. The `[{request_id}]`
PERF/log prefixes keep the session id for operator correlation.
- `tests/test_openai_codex_ws_lifecycle.py`: new tests driving a
two-turn Codex WS session through the `_FakeWebSocket`/`_FakeUpstream`
harness with a capturing request logger and an incrementing
`_next_request_id`, asserting distinct per-row `request_id`s without
relying on local repro artifacts, unchanged per-turn token/savings
values, no phantom row for a no-op turn, and session-prefixed logs.
- `CHANGELOG.md`: `Unreleased → Fixed` entry.
## Testing
- [x] Unit tests pass (`uv run pytest
tests/test_openai_codex_ws_lifecycle.py`)
- [x] Linting passes (`uv run ruff check .`)
- [ ] Type checking passes (`uv run mypy headroom`)
- [x] New tests added for new functionality when applicable
- [x] Manual testing performed
### Test Output
```text
$ uv run pytest tests/test_openai_codex_ws_lifecycle.py -q
............................. [100%]
29 passed in 1.69s
$ uv run ruff check .
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12 via `uv`, no live provider — the
WS handler is exercised through the in-process
`_FakeWebSocket`/`_FakeUpstream` harness that mirrors the production
wire shape.
- Exact command / steps: ran `uv run pytest
tests/test_openai_codex_ws_lifecycle.py::test_ws_multi_turn_request_ids_are_unique
-q` on this branch and `uv run pytest
tests/test_openai_codex_ws_lifecycle.py -q` for the focused file; on
`origin/main`, the new regression node is absent and the WS emit sites
still use `request_id=request_id` in
`headroom/proxy/handlers/openai.py`.
- Observed result: a two-turn Codex WS session now yields
`recent_requests` rows with unique `request_id`s, so the dashboard's
Alpine `:key` no longer collides; token/savings values and row counts
are unchanged; a no-op turn still emits no row. On `origin/main`, the
handler still reuses the session `request_id` at the WS feed emit sites.
- Not tested: live dashboard browser render of the fixed feed.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- Type checking (`mypy`) left unchecked: not run in this pass; the
change only swaps the source of an existing `request_id` string field.
- Non-goal (out of scope): hardening the dashboard `x-for` `:key`
against duplicate/`null` keys is a separate render-robustness fix for a
follow-up (`Refs #310`); this PR removes the source of the duplicates.
The comment's `savings_percent.toFixed(0)` concern is already fixed on
`main` (`formatOptionalPercent`).
- Prior art: an earlier change (issue #399 era) added the per-turn Codex
WS `RequestLog`/PERF emission but reused the session id; this PR makes
those ids unique.
|
||
|
Ashish Patel
|
4ec416df88
|
fix(proxy): stop discarding compressed Codex WS later-frame payloads (#2823)
## Description
`headroom perf` reports 0 tokens saved for Codex CLI sessions despite
real traffic being processed (confirmed via the reporter's live proxy
stats in the issue). Root cause: a misplaced `return` statement in the
Codex WS later-frame compression path silently discards every compressed
payload and skips all token/savings bookkeeping for it.
Closes #2819
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/handlers/openai.py`
(`_maybe_compress_response_create_frame`): PR #1579 (2026-07-16) moved a
`return (raw_after_store, ...)` statement to the same indentation as the
enclosing `except Exception:` block instead of inside it. That made the
`return` fire **unconditionally** after every later (2nd+)
`response.create` frame in a Codex WS session — success or failure —
always forwarding the original pre-compression frame upstream and
skipping the entire success-path code below it (correct
rewritten-payload return, `tokens_saved`,
`attempted_input_tokens_total`, `ws_frames_compressed`). Fixed by moving
the `return` back inside the `except` block, restoring the success path.
- `tests/test_openai_codex_ws_lifecycle.py`: new regression test
`test_ws_later_frame_compression_is_actually_forwarded` — mocks the
compressor to report `modified=True` with a distinct rewritten payload
on a later frame, asserts the rewritten payload (not the original) is
what's actually sent upstream.
## Testing
- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`) — not run locally, will confirm
via CI
- [ ] Type checking passes (`mypy headroom`) — not run locally, will
confirm via CI
- [x] New tests added for new functionality
- [x] Manual testing performed (see Real Behavior Proof)
### Test Output
```text
$ .venv/Scripts/python -m pytest tests/test_openai_codex_ws_lifecycle.py tests/test_codex_ws_per_frame_memory.py tests/test_codex_ws_savings_deferral.py tests/test_openai_codex_ws_timings.py -q
............................................... 48 passed in 5.34s
$ .venv/Scripts/python -m pytest tests/ -k "openai or codex" -q (wider sweep, unrelated dirs excluded)
968 passed, 3 failed, 77 skipped, 2 errors in 483.21s
```
The 3 failures
(`test_client_integration.py::test_auto_detect_openai_optimizer`,
`test_compress_route_tokenizer_by_model.py::...[deepseek/deepseek-v4]`,
`test_learn/test_integration.py::TestCodexIntegration::test_full_pipeline`)
reproduce identically on a clean, unmodified `main` — confirmed by
stashing this PR's changes and re-running. They're local-environment
issues (a live litellm 503, and this dev box's tool registry missing a
`Bash` entry), not caused by this change.
## Real Behavior Proof
- Environment: Windows 11, Python 3.14.5, local venv, `headroom._core`
rebuilt via `maturin develop --release` against current `main` to rule
out stale-build noise
- Exact command / steps: (1) `git blame` on the buggy block traced the
misplaced `return` to commit `
|
||
|
John Xu
|
1c50eca8b3
|
fix(proxy): skip Responses memory tools for ChatGPT auth (#1579)
## Description Fix ChatGPT/Codex session-auth Responses proxy handling so the ChatGPT backend always receives an explicit `store=false`, while keeping Responses memory tools limited to the regular API-key path where stored responses are supported. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Detect ChatGPT auth before Responses memory-tool injection and force `store=false` for ChatGPT-auth Responses payloads. - Skip Responses memory tools and transparent memory-tool continuation handling for ChatGPT auth across HTTP, WebSocket first frames, WebSocket follow-up `response.create` frames, and WS-to-HTTP fallback. - Preserve API-key behavior after the current main merge: API-key requests that explicitly set `store=false` skip Responses memory tools, while API-key requests that receive injected memory tools are forced to `store=true` for continuation support. - Address Copilot formatter comments by making `_allow_responses_memory_tools` call sites formatter-stable. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --extra dev ruff format --check headroom/proxy/handlers/openai.py 1 file already formatted $ uv run --extra dev ruff check headroom/proxy/handlers/openai.py tests/test_openai_codex_routing.py tests/test_openai_codex_ws_timings.py tests/test_ws_http_fallback.py All checks passed! $ uv run --extra dev python -m pytest -q tests/test_openai_codex_routing.py tests/test_openai_codex_ws_timings.py tests/test_ws_http_fallback.py 37 passed in 0.34s ``` ## Real Behavior Proof - Environment: Local checkout of `fix/codex-store-false-memory-tools` using `uv run --extra dev`. - Exact command / steps: Ran the focused formatter, lint, and pytest commands listed in `Testing`. - Observed result: Formatting is stable, lint passes, and the focused OpenAI/Codex routing and fallback tests pass. - Not tested: Full test suite, `mypy headroom`, and a fresh live ChatGPT backend probe after the formatter-only follow-up. The original PR validation recorded that valid ChatGPT subscription backend requests return `200` with `store=false`, while identical `store=true` or omitted `store` requests return `400 Store must be set to false`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A. ## Additional Notes - Post-deploy monitoring terms: `Responses: forced store=false for ChatGPT auth`, `WS Responses: forced store=false for ChatGPT auth`, `chatgpt_store_false`, `Memory: forced store=true for Responses memory tool continuation`, and upstream 400s containing `Store must be set to false`. - Expected healthy signals: ChatGPT-auth Responses requests keep `store=false` and no longer fail with `Store must be set to false`; API-key memory-tool flows still inject memory tools and can continue via `previous_response_id`. - Rollback trigger: any increase in ChatGPT-auth 400s, API-key memory-tool continuation failures, or missing memory tool injection on API-key Responses requests. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
yaowei
|
ac7ee4e0bf
|
fix(proxy): support Codex WS compatible gateways (#1281)
Adds opt-in compatibility for OpenAI-compatible WebSocket gateways used behind Codex /v1/responses. - HEADROOM_OPENAI_WS_FLATTEN_RESPONSE_CREATE=1 flattens Codex response.create frames before upstream send. - HEADROOM_OPENAI_WS_PROPAGATE_UPSTREAM_CLOSE=1 propagates upstream close code/reason back to the client. - Default behavior is unchanged. Tested: python -m pytest tests/test_openai_codex_ws_lifecycle.py -q 18 passed Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
nangsontay
|
e5b3a634df
|
fix(proxy): dedupe Codex WS request logging for accurate mixed-provider dashboards (#2189)
## Description Running Claude Code (Anthropic) and Codex (OpenAI) against the **same** Headroom proxy instance on one port produced incorrect, unstable dashboard data. The proxy core is provider-isolated and multi-provider-safe by design; the defect was in the observability layer. The Codex `/v1/responses` **WebSocket** handler was the only path in the proxy that wrote to the request logger by hand instead of through the unified `emit_request_outcome` funnel, and it did so twice per session close: the per-turn funnel record plus an unconditional cumulative session-summary `RequestLog`. This PR removes the duplicate summary log so Codex WS emits exactly one request log per turn, matching the HTTP provider paths. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Dropped the duplicate cumulative session-summary `RequestLog` in the Codex WS handler while preserving the per-turn `emit_request_outcome` path. - Preserved gated `request_messages` and `turn_id` on residual outcomes so dashboard telemetry keeps the useful attribution without double-counting tokens. - Ensured explicit `--anyllm-provider` wins over a leaked `HEADROOM_ANYLLM_PROVIDER` environment variable. - Registered retry delay settings that had drifted out of the settings registry. - Hardened tests against developer-shell `HEADROOM_*` / `ANTHROPIC_CUSTOM_HEADERS` leakage and stabilized several focused proxy/wrap test fixtures. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ .venv/bin/pytest tests/ -q -p no:cacheprovider 8529 passed, 537 skipped, 5831 warnings in 276.25s (0:04:36) $ .venv/bin/ruff check <touched files> All checks passed! ``` ## Real Behavior Proof - Environment: macOS (Darwin 25.4.0), Python 3.13.14, pytest 9.0.3, ruff via project venv, branch `fix/multi-provider-runtime`. - Exact command / steps: Ran the full test suite without pytest cache provider and Ruff on all touched files; used `git stash` to confirm the stale fake-config failures pre-existed this change. - Observed result: Full suite passed with no failures; Ruff passed; Codex WS now routes end-of-session logging through `emit_request_outcome`, emitting one request log per turn with the same accounting model as Anthropic HTTP turns. - Not tested: Live simultaneous Claude + Codex dashboard run. `mypy headroom` was not run to completion; a scoped run reported one pre-existing `settings_store.py:470` coercion error outside this diff. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A - server-side observability fix; no UI markup changed. ## Additional Notes - The proxy's multi-provider routing, header/auth isolation, and per-model cache keying are already correct and unchanged here; only the WS observability write path was double-counting. - Architectural assessment: `plans/reports/research-260714-0004-multi-provider-upstream-compression-report.md`; root-cause + resolution trail: `plans/reports/debug-assessment-260714-0011-dashboard-instability-mixed-claude-codex-report.md`. - No live simultaneous Claude + Codex dashboard run was performed; validation is from test coverage and code review of the WS logging path. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Rod Boev
|
551f473e04
|
fix(proxy): accept Codex websocket before upstream retries (#2203)
## Description Codex Desktop could abandon Headroom's local `/v1/responses` WebSocket handshake before Headroom's upstream retry strategy had a chance to recover. The ChatGPT-auth path waited for an upstream opening handshake with a minimum 30-second timeout before sending the local 101, while the reported Codex Desktop handshake expired after about 34 seconds. This change accepts validated ChatGPT-auth Codex WebSockets before opening the upstream connection, then keeps the existing upstream retries and HTTP fallback behind the established local session. API-key sessions retain connect-before-accept behavior so upstream `x-codex-*` headers can still be attached to their client-facing 101. The change is scoped to the pre-101 timing failure and does not address the separate large-context streaming investigation in #1944. Closes #2184 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Accept ChatGPT-auth Codex WebSocket clients before the upstream connect and retry loop. - Preserve API-key connect-before-accept ordering and upstream `x-codex-*` handshake-header forwarding. - Keep upstream retry, relay, usage-state refresh, and WebSocket-to-HTTP fallback behavior after the local 101. - Add a deterministic regression that blocks the first upstream opening handshake and proves the local acceptance deadline is independent of it. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_openai_codex_ws_lifecycle.py -q`) - [x] Linting passes (`uv run ruff check headroom/proxy/handlers/openai.py tests/test_openai_codex_ws_lifecycle.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text uv run pytest tests/test_openai_codex_ws_lifecycle.py -q 28 passed in 2.02s uv run ruff check headroom/proxy/handlers/openai.py tests/test_openai_codex_ws_lifecycle.py All checks passed! uv run ruff format headroom/proxy/handlers/openai.py tests/test_openai_codex_ws_lifecycle.py --check 2 files already formatted ``` ## Real Behavior Proof - Environment: Python 3.12, synced development worktree, local fake Codex client and upstream WebSocket, no live provider - Exact command / steps: Run `uv run pytest tests/test_openai_codex_ws_lifecycle.py::test_chatgpt_ws_accepts_before_stalled_upstream_connect -q`; the fake upstream blocks its first opening handshake while the client enforces a bounded local-accept deadline. - Observed result: `1 passed in 0.46s`; the ChatGPT-auth client receives its local 101 before the blocked upstream connect is released, and the handler continues into its existing upstream recovery path. - Not tested: live Codex Desktop pre-turn compaction against ChatGPT subscription infrastructure ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes `CHANGELOG.md` is unchanged because the release pipeline generates it from conventional commits. No user documentation changes are required; the handler comments and ordered-flow docstring are updated with the auth-mode-specific behavior. The broader #1944 large-context disconnect surface remains out of scope. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
iotron
|
5cece7bf58
|
fix(proxy): Strip Codex responses-lite marker from response.create frame body (#1820)
## Description Codex CLI mirrors the `X-OpenAI-Internal-Codex-Responses-Lite` request header into the `response.create` WS frame body under `client_metadata.ws_request_header_x_openai_internal_codex_responses_lite`. The existing fix strips the header itself (both the HTTP fallback path and the WS handshake path) but never touched this frame-body copy, so Headroom still forwarded it to `wss://chatgpt.com/backend-api/codex/responses` unmodified. Upstream rejects `gpt-5.x` models whenever that field is truthy, so every Codex-through-Headroom turn on gpt-5.5/5.4 failed with a 400, even on builds that already contain the header-strip fix. Closes #1523 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/handlers/openai.py`: add `_strip_codex_lite_metadata()`, which removes `client_metadata.ws_request_header_x_openai_internal_codex_responses_lite` from a `response.create` frame body (checks both the flat-body shape and the `{"response": {...}}` envelope shape Codex uses depending on call path). Fail-safe no-op on non-JSON payloads or when the key is absent. - Wired the helper into both WS-forwarder send sites in the same file: the initial `first_msg_raw` send and the steady-state `_client_to_upstream` relay send. - `tests/test_openai_codex_ws_lifecycle.py`: added `test_ws_first_frame_strips_codex_lite_metadata_mirror`, which sends a `response.create` frame carrying the mirror key through the handler and asserts the frame actually forwarded to the fake upstream has the key removed while sibling `client_metadata` fields (e.g. `thread_id`) survive. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check headroom/proxy/handlers/openai.py tests/test_openai_codex_ws_lifecycle.py All checks passed! $ mypy headroom/proxy/handlers/openai.py Success: no issues found (pre-existing unrelated notes in headroom/proxy/server.py only) $ pytest tests/test_openai_codex_ws_lifecycle.py tests/test_openai_codex_routing.py tests/test_codex_openai_contract_parity.py -q 52 passed, 23 warnings in 1.66s # Proof the new test actually catches the bug (checked out the parent commit's # openai.py, i.e. pre-fix, with the new test present): $ pytest tests/test_openai_codex_ws_lifecycle.py -q -k strips_codex_lite_metadata FAILED tests/test_openai_codex_ws_lifecycle.py::test_ws_first_frame_strips_codex_lite_metadata_mirror AssertionError: assert 'ws_request_header_x_openai_internal_codex_responses_lite' not in {'thread_id': 't_1', 'ws_request_header_x_openai_internal_codex_responses_lite': True} 1 failed, 26 deselected in 0.63s # Restored the fix -> same test passes (see "52 passed" run above). ``` ## Real Behavior Proof - Environment: macOS, Python 3.13, headroom-ai 0.30.0 (pip-installed copy patched identically to this diff), Codex CLI 0.142.5, ChatGPT Plus subscription auth, model `gpt-5.5`. - Exact command / steps: `headroom wrap codex` (proxy on 127.0.0.1:8787) → `codex exec "..."` with `model_provider = "headroom"` in `~/.codex/config.toml`. Also reproduced deterministically via a throwaway debug proxy instance with `HEADROOM_CODEX_WIRE_DEBUG=1` wire capture, isolated from the live account/session. - Observed result: pre-patch, deterministic `400 unsupported_value` / "This model is not supported when using X-OpenAI-Internal-Codex-Responses-Lite" on every single turn (reproduced repeatedly, across a header-only-stripped build). Post-patch (installed copy with this exact diff): clean full completions streamed to `response.completed` with zero error frames via wire capture, and — after restarting the live production proxy to load the patched module — the user confirmed `codex` working normally end-to-end through Headroom on `gpt-5.5` in real day-to-day use, not just the isolated repro. - Not tested: the HTTP (non-WS) fallback path for Codex — that path doesn't carry a `response.create` frame body in the same way, and the existing header-strip logic already covers it; no upstream 400s observed there in this investigation. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — backend WS proxy fix, no UI surface. ## Additional Notes - CHANGELOG.md / docs left untouched: this is a narrowly-scoped bugfix to existing (undocumented-at-user-level) WS forwarding internals; happy to add a CHANGELOG entry if maintainers want one. - Full root-cause writeup with wire-capture details is on the issue: https://github.com/headroomlabs-ai/headroom/issues/1523#issuecomment-4887989873 - Did not run the full repo-wide `pytest`/`mypy headroom` (whole package) — scoped to the modified file and the three most relevant existing test modules (`test_openai_codex_ws_lifecycle.py`, `test_openai_codex_routing.py`, `test_codex_openai_contract_parity.py`), all passing. Glad to run the full suite if a maintainer wants that in CI instead. Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: Tejas Chopra <chopratejas@gmail.com> |
||
|
Rudimar Ronsoni
|
4bd3ddfaa5
|
fix(opencode): use local MCP config (#1383)
## Description Fixes OpenCode Headroom MCP configuration across wrap, MCP install/status/uninstall, and persistent install docs/CLI. OpenCode was being configured to use a remote HTTP MCP endpoint at `/mcp`, but the Headroom proxy does not expose MCP there. The correct OpenCode configuration is a local stdio MCP server that runs `headroom mcp serve`. Closes #1380 ## Type of Change - [x] Bug fix - [ ] New feature - [ ] Breaking change - [x] Documentation update - [x] Tests ## Changes Made - Changed OpenCode MCP registration to emit `type: "local"` with `command: ["headroom", "mcp", "serve"]`. - Changed OpenCode MCP environment serialization from `env` to OpenCode's `environment` key, while still reading legacy `env` entries. - Removed generated remote `/mcp` entries from OpenCode wrap/runtime config. - Made `wrap opencode --no-mcp` skip persistent `mcp.headroom` injection. - Kept provider-only OpenCode config injection from writing MCP; MCP persistence is owned by the registrar path. - Made `headroom mcp status` and `headroom mcp uninstall` use the registrar lifecycle so OpenCode is covered. - Added `opencode` to persistent install `--target` choices. - Clarified OpenCode persistent install docs to use `--scope provider` for direct `opencode.json` edits. - Added regression coverage for registrar serialization, wrap behavior, runtime config, provider-scope install, MCP CLI lifecycle, and install target parsing. ## Testing - [x] `rtk .venv/bin/python -m pytest tests/test_mcp_registry tests/test_cli/test_mcp.py tests/test_cli/test_wrap_opencode.py tests/test_providers_opencode_config.py tests/test_providers_opencode_install.py tests/test_install -q` - [x] Result after absorbing #1381 overlap: `263 passed, 1 skipped` - [x] Targeted Ruff check passed for the changed Python/test files. - [x] Targeted Ruff format check passed for the changed Python/test files. - [x] Isolated HOME smoke tests with real `opencode mcp list --pure`. ## Real Behavior Proof - `headroom mcp install --agent opencode --proxy-url http://127.0.0.1:9000 --force` against an isolated HOME wrote a valid local OpenCode MCP entry with `environment.HEADROOM_PROXY_URL`. - `opencode mcp list --pure` against that isolated HOME connected to `headroom mcp serve`. - `headroom wrap opencode --prepare-only --no-rtk --no-serena --port 9001` wrote local MCP plus provider config. - `headroom wrap opencode --prepare-only --no-rtk --no-serena --no-mcp --port 9002` wrote provider config without `mcp.headroom`. - Generated runtime `OPENCODE_CONFIG_CONTENT` was accepted by `opencode mcp list --pure`; `include_mcp=False` reported no MCP servers. - `headroom mcp status` detected the isolated OpenCode config and read the custom proxy URL. - `headroom mcp uninstall` removed `mcp.headroom` from the isolated OpenCode config while leaving provider config intact. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review |
||
|
Rod Boev
|
d24a3f8425
|
fix(proxy): bound Codex WS compression fallback latency (#1802)
## Description Codex `/v1/responses` WebSocket frames could spend the full global compression timeout before falling through unchanged, then report only a generic `compression_exception` reason. That made a recoverable timeout look like an opaque compression failure and left Codex users waiting around 30 seconds for frames that did not produce useful compression. This keeps the existing compression executor, adds a Codex WS-specific compression timeout bound, and records timeout fallback distinctly from other compression exceptions. Closes #922. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Bound Codex Responses WebSocket frame compression with a WS-specific timeout. - Pass that timeout through the existing compression executor instead of adding a parallel executor path. - Record timeout passthrough with `compression_timeout` instead of the generic compression exception reason. - Preserve generic `compression_exception` for non-timeout failures. - Add coverage for first-frame timeout bounds, timeout reason logging, generic exception preservation, and later-frame failed metrics. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_openai_codex_ws_lifecycle.py tests/test_codex_ws_compression_scheduler.py tests/test_compression_observability.py -q`) - [x] Linting passes (`uv run ruff check headroom/proxy/handlers/openai.py tests/test_openai_codex_ws_lifecycle.py tests/test_codex_ws_compression_scheduler.py tests/test_compression_observability.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text uv run pytest tests/test_openai_codex_ws_lifecycle.py -q 22 passed in 1.38s uv run pytest tests/test_codex_ws_compression_scheduler.py tests/test_compression_observability.py -q 14 passed, 1 skipped in 3.63s uv run pytest tests/test_openai_codex_ws_lifecycle.py tests/test_codex_ws_compression_scheduler.py tests/test_compression_observability.py -q 36 passed, 1 skipped in 2.09s uv run ruff check headroom/proxy/handlers/openai.py tests/test_openai_codex_ws_lifecycle.py tests/test_codex_ws_compression_scheduler.py tests/test_compression_observability.py All checks passed! uv run ruff format headroom/proxy/handlers/openai.py tests/test_openai_codex_ws_lifecycle.py --check 2 files already formatted ``` ## Real Behavior Proof - Environment: Windows, Python through the project `uv` environment. - Exact command / steps: run the focused Codex WS timeout regressions with small monkeypatched timeout values. - Observed result: base uses the global timeout path or reports only generic `compression_exception`; head passes with Codex WS timeout bounded to the smaller WS cap, logs `compression_timeout` for timeout fallback, preserves `compression_exception` for non-timeout failures, and records failed metrics for later-frame timeout fallback. - Not tested: live Codex Desktop traffic against paid OpenAI credentials. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes No changelog entry is needed for this request-path bug fix. Type checking was not part of the focused local validation for this Python-only change. Live Codex Desktop validation is not included because the regression is covered at the handler boundary. |
||
|
Andrew McFague
|
0b0133b7fd
|
Wire OpenAI Responses output shaping (#1438)
## Description Wire output shaping for OpenAI Responses traffic across HTTP `/v1/responses` and Codex WebSocket `response.create` frames. The change adds provider-specific shaping for `instructions`, `reasoning.effort`, and `text.verbosity` while keeping Anthropic request mutation separate. Review follow-up: merged byte-faithful `/v1/responses` forwarding from #1557 and marks shaped HTTP Responses payloads as `body_mutated=True`, so retry forwarding sends the shaped body instead of the original raw bytes. ## Type of Change - [ ] Bug fix (non-breaking change fixes an issue) - [x] New feature (non-breaking change adds functionality) - [ ] Breaking change (fix or feature would cause existing functionality change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added OpenAI Responses output shaping for `instructions`, `reasoning.effort`, and `text.verbosity`. - Wired shaping into `/v1/responses` HTTP and Codex WebSocket `response.create` paths. - Preserved `x-headroom-bypass` and `HEADROOM_OUTPUT_HOLDOUT` behavior. - Added output-shaper transform labels for verbosity, text verbosity, reasoning effort, holdout control, and strata. - Updated output-savings conversation keys for Responses payloads and WS `response.create` envelopes. - Counted WS frame payload tokens when assigning output-savings strata. - Merged byte-faithful `/v1/responses` forwarding from #1557 and kept shaped HTTP bodies on the mutated-forwarding path. - Added tests for classification, shaping, holdout, bypass, labels, WS strata, and byte-faithful forwarding compatibility. - Updated `CHANGELOG.md` for OpenAI Responses output-shaping support. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --extra dev python -m pytest tests/test_openai_codex_ws_lifecycle.py tests/test_openai_responses_output_shaper.py tests/test_codex_responses_passthrough_bytes.py tests/test_output_shaper.py tests/test_output_savings.py -q 110 passed, 1 warning in 1.49s $ uv run --extra dev ruff check headroom/proxy/handlers/openai.py tests/test_openai_responses_output_shaper.py tests/test_codex_responses_passthrough_bytes.py tests/test_openai_codex_ws_lifecycle.py tests/test_output_shaper.py tests/test_output_savings.py All checks passed! $ git diff --check No whitespace errors. ``` ## Real Behavior Proof - Environment: local macOS checkout, branch `output-shaper-openai-responses`. - Exact command / steps: ran targeted pytest, ruff, and diff checks listed above. - Observed result: targeted tests passed with an existing FastAPI TestClient deprecation warning; ruff passed; diff check passed. - Not tested: full repository test suite, live OpenAI traffic, browser dashboard rendering, full `mypy headroom`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows project's style guidelines - [x] I performed self-review of my code - [x] I commented my code, particularly in hard-to-understand areas - [x] I made corresponding changes to documentation - [x] My changes generate no new warnings - [x] I added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I updated `CHANGELOG.md` if applicable ## Screenshots N/A ## Additional Notes - Non-applicable Type Change items are left unchecked. - The pytest warning comes from `fastapi.testclient` importing Starlette TestClient and was not introduced by this change. - `CHANGELOG.md` includes entries for OpenAI Responses output-shaping support and byte-faithful `/v1/responses` forwarding compatibility. --------- Co-authored-by: obchain <riteshnikhoriya94@gmail.com> |
||
|
Rod Boev
|
5d3803a21c
|
fix(proxy): strip Codex lite header from OpenAI WebSockets (#1543)
## Description Codex WebSocket traffic through Headroom can forward `X-OpenAI-Internal-Codex-Responses-Lite` upstream. OpenAI tightened enforcement of that header on 2026-06-26 for `gpt-5.5`, `gpt-5.4`, and `gpt-5.4-mini`, so the same Codex setup can fail through Headroom with `unsupported_value` while succeeding when Headroom is bypassed. The OpenAI Responses WS handler strips only `x-headroom-*` internal headers today, so this Codex client header survives into both the direct upstream WebSocket connect and the WS HTTP fallback path. This change strips `X-OpenAI-Internal-Codex-Responses-Lite` from the upstream header copy inside `handle_openai_responses_ws` after routing resolution and before the upstream request is sent. `_ws_http_fallback(...)` reuses that same header dict, so the fallback path inherits the fix without a second guard. `headroom/proxy/helpers.py` stays unchanged; the shared helper contract remains `x-headroom-*` only. Closes #1525 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add a narrow case-insensitive strip for `X-OpenAI-Internal-Codex-Responses-Lite` in `headroom/proxy/handlers/openai.py` after `_resolve_codex_routing_headers(...)` and before `websockets.connect(...)`. - Keep `_strip_internal_headers(...)` in `headroom/proxy/helpers.py` unchanged so the documented `x-headroom-*` stripping scope does not widen. - Extend `tests/test_openai_codex_ws_lifecycle.py` to capture `additional_headers`, prove the direct WS leak on base, prove the fix on head, prove `_ws_http_fallback(...)` inherits sanitized headers, and prove adjacent non-lite headers still survive. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_openai_codex_ws_lifecycle.py -k "codex_responses_lite or without_codex_lite" -v`) - [x] Linting passes (`uv run ruff check headroom/proxy/handlers/openai.py tests/test_openai_codex_ws_lifecycle.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text Base proof before the fix: uv run pytest tests/test_openai_codex_ws_lifecycle.py -k "test_ws_codex_responses_lite_header_is_not_forwarded_upstream" -v FAILED tests/test_openai_codex_ws_lifecycle.py::test_ws_codex_responses_lite_header_is_not_forwarded_upstream AssertionError: assert 'X-OpenAI-Internal-Codex-Responses-Lite' not in { 'authorization': 'Bearer test', 'X-OpenAI-Internal-Codex-Responses-Lite': 'true', 'X-OpenAI-Debug': 'keep-me', 'ChatGPT-Account-ID': 'acct-123', 'x-client': 'codex', 'OpenAI-Beta': 'responses_websockets=2026-02-06' } Focused regression proof after the fix: uv run pytest tests/test_openai_codex_ws_lifecycle.py -k "codex_responses_lite or without_codex_lite" -v tests/test_openai_codex_ws_lifecycle.py::test_ws_codex_responses_lite_header_is_not_forwarded_upstream PASSED [ 33%] tests/test_openai_codex_ws_lifecycle.py::test_ws_codex_responses_lite_header_is_not_forwarded_to_fallback PASSED [ 66%] tests/test_openai_codex_ws_lifecycle.py::test_ws_without_codex_lite_preserves_adjacent_headers_and_api_key_route PASSED [100%] ====================== 3 passed, 16 deselected in 0.62s ======================= uv run ruff check headroom/proxy/handlers/openai.py tests/test_openai_codex_ws_lifecycle.py All checks passed! ``` ## Real Behavior Proof - Environment: local pytest async lifecycle harness in `tests/test_openai_codex_ws_lifecycle.py`, no live provider required. - Exact command / steps: `uv run pytest tests/test_openai_codex_ws_lifecycle.py -k "codex_responses_lite or without_codex_lite" -v` - Observed result: before the fix, the new direct-leak test failed because `websockets.connect(..., additional_headers=...)` still contained `X-OpenAI-Internal-Codex-Responses-Lite`. After the fix, the focused rerun passed and proved that both direct WS connect and forced `_ws_http_fallback(...)` receive sanitized headers while adjacent non-lite headers still survive. - Not tested: live Codex traffic against OpenAI with real credentials, unless that is added during implementation. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes `uv run mypy headroom` is outside the focused proof for this small WS-path fix and may remain unchecked if the coding pass keeps the validation surface to targeted pytest plus ruff. `CHANGELOG.md` remains unchanged because the resolved repo config says Headroom's release pipeline generates changelog entries from conventional commits. |
||
|
gglucass
|
b0cd0329c7
|
fix(proxy): stamp X-Client: codex on Responses endpoint for unidentified callers (#1036)
## Description Codex Desktop (OpenAI's Codex GUI/IDE app) sends a `User-Agent` of the form `Codex Desktop/<ver> (...)`, which is not in `CLIENT_UA_MAP`, so `classify_client` returns `None`. On a compression timeout the backend only takes the codex fail-open path when the client classifies as `codex`; for an unidentified client it refuses with HTTP 413 (`compression_refused`), which Codex treats as a hard connection failure. This stamps `X-Client: codex` on requests to the Responses endpoint (`/v1/responses`) only when the caller does not otherwise classify. The stamp is scoped to the Responses endpoint and skipped for any caller that already classifies through a recognized user-agent or explicit `X-Client`, so non-Codex traffic is not relabeled. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Tests only ## Changes Made - Added `should_stamp_codex_client(path, headers)` in `headroom.proxy.auth_mode` for narrow Responses-endpoint client stamping. - Applied the stamp in HTTP middleware before downstream request classification. - Applied the same stamp in the Responses WebSocket handler, which bypasses HTTP middleware. - Added unit coverage for the stamp/skip matrix, including Codex Desktop, explicit clients, recognized user-agents, and WebSocket behavior. - Merged current `main` and kept both the new stamp coverage and the existing Codex WebSocket image-generation regression coverage. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Formatting passes (`ruff format --check`) - [x] New tests added for new functionality ### Test Output ```text python -m pytest tests/test_codex_client_stamp.py tests/test_auth_mode.py tests/test_openai_codex_ws_lifecycle.py -q 48 passed in 1.13s ruff check headroom/proxy/auth_mode.py headroom/proxy/server.py headroom/proxy/handlers/openai.py tests/test_codex_client_stamp.py tests/test_auth_mode.py tests/test_openai_codex_ws_lifecycle.py All checks passed! ruff format --check headroom/proxy/auth_mode.py headroom/proxy/server.py headroom/proxy/handlers/openai.py tests/test_codex_client_stamp.py tests/test_auth_mode.py tests/test_openai_codex_ws_lifecycle.py 6 files already formatted ``` ## Real Behavior Proof - Environment: local Windows 11 development checkout, Python 3.13.13, branch updated from `upstream/main`. - Exact command / steps: Ran the focused unit suite for the new client-stamp behavior and the overlapping Codex WebSocket lifecycle tests, then ran `ruff check` and `ruff format --check` on the changed modules and tests. - Observed result: The focused suite passed with 48 tests, lint passed, and formatting passed. The tests assert that unidentified `/v1/responses` callers classify as Codex after stamping while explicit or already-recognized clients are preserved. - Not tested: a live end-to-end Codex Desktop session through a running `headroom wrap codex` instance; verification is at the unit/integration boundary for classification and request routing. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Zhenjia ZHOU
|
7dbbb4077e
|
fix(proxy): keep codex image-generation WS turns alive through the relay (#1000)
## Description Image generation through the proxy fails. Driving Codex (`/v1/responses` over WebSocket) through Headroom, an image-generation turn never returns an image — the client retries (`Reconnecting… n/5`) and gives up, while the same prompt works when Codex talks to ChatGPT directly. Root cause: two independent defects on the upstream `websockets.connect()`, both specific to how image generation behaves on the wire: 1. **Pong deadline kills the silent render.** An image turn emits a single `response.image_generation_call.generating` event and then goes silent for 20–60s while the model renders (no data frames). The hard-coded `ping_timeout=20` treats that healthy-but-quiet connection as dead and tears it down as `upstream_error` mid-render, before the image is ready. 2. **1 MiB frame cap drops the image.** The finished image comes back inline as a single base64 frame that exceeds the `websockets` default `max_size=2**20` (1 MiB), raising `PayloadTooBig` exactly as the image lands. They compound: with only ping fixed, the session survives the silent phase (observed ~20s → ~54s) but then dies on the oversized image frame. Normal text/tool turns stream tokens continuously and stay well under 1 MiB, so neither defect affects them — which is why this only ever bit image generation. Closes: N/A (no tracking issue) ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/handlers/openai.py`: on the upstream `/v1/responses` connect, set `ping_timeout=None` (keep `ping_interval=20` for NAT keepalive) so a long silent render is not torn down on a missing pong, and `max_size=None` so the inline base64 image frame is accepted instead of raising `PayloadTooBig`. - `tests/test_openai_codex_ws_lifecycle.py`: add `test_ws_upstream_connect_allows_large_frames_and_no_pong_deadline`, which captures the upstream connect kwargs and pins `ping_timeout is None` / `max_size is None`. ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check . All checks passed! $ mypy headroom --ignore-missing-imports Success: no issues found in 358 source files $ pytest tests/test_openai_codex_ws_lifecycle.py -q collected 15 items tests/test_openai_codex_ws_lifecycle.py .............. [100%] ============================== 15 passed in 0.72s ============================== $ pytest tests/test_openai_codex_ws_lifecycle.py -k large_frames_and_no_pong -q collected 15 items / 14 deselected / 1 selected tests/test_openai_codex_ws_lifecycle.py . [100%] ======================= 1 passed, 14 deselected in 0.35s ======================= ``` End-to-end (managed Codex image generation through the running proxy): ```text # BEFORE fix: fails at ~20s (unpatched) / ~54s (ping-only) # WS /v1/responses completed (cause=upstream_error, # last_upstream_type=response.image_generation_call.generating) # -> client "Reconnecting… n/5", no image produced # AFTER fix: [codex] Image ready; stopping the turn. Saved image: /tmp/headroom-imagegen-test.png $ file /tmp/headroom-imagegen-test.png PNG image data, 1254 x 1254, 8-bit/color RGB, non-interlaced (908 KB) # proxy session count +1 -> the turn DID traverse the proxy and completed. ``` ## Real Behavior Proof - Environment: macOS, headroom 0.23.0 running as the Codex `model_provider` (proxy on `127.0.0.1:8787`), Codex CLI 0.139.0 driving a managed `/v1/responses` image-generation turn through the proxy. - Exact command / steps: trigger a Codex image-generation turn (gpt-image-2) with the proxy in front; observe the upstream `/v1/responses` WS session in `proxy.log` and whether a PNG is returned. - Observed result: before the change the session dies with `upstream_error` while `last_upstream_type=response.image_generation_call.generating` and no image is produced; after the change a valid 1254×1254 PNG is returned and the session traverses the proxy normally. - Not tested: the full `pytest` suite was not run locally — this machine has no Rust toolchain to rebuild the matching `_core` extension, so the complete suite (incl. the pyo3 tests) is left to CI. The affected `test_openai_codex_ws_lifecycle.py` module was run against the installed extension and passes 15/15; `ruff check .` and `mypy headroom` were run in full and pass. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A. ## Additional Notes - `ruff check .` (whole repo) and `mypy headroom --ignore-missing-imports` (358 source files) were run locally and pass. The only `pytest` not run locally is the full suite, because the Rust `_core` cannot be rebuilt here without a toolchain; the directly affected lifecycle module passes 15/15 and CI runs the rest. - Documentation / CHANGELOG left unchecked — this is a focused two-line behavioral fix on the upstream WS connect; happy to add a CHANGELOG entry if preferred. - `ping_timeout=None` keeps `ping_interval` for NAT keepalive; if you'd rather bound it, a generous finite value (e.g. 300s) would also fix the render case — happy to switch. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
gglucass
|
0ce68dedd7
|
fix(proxy): restore Codex usage headers on WS and streaming SSE transports (#577) (#794)
## Description Codex's subscription/rate-limit window (the `x-codex-*` headers) was being **stripped on every transport Codex actually uses**, so session/weekly usage never reached the Codex CLI's own `/status` display, Headroom `/stats`/dashboard, or any consumer that sniffs the client-facing handshake. This PR restores it on **both** the WebSocket and streaming-SSE paths — the two halves of #577 — in one place. Fixes #577 **Supersedes #582 and #590.** This PR incorporates #582's SSE fix (carried verbatim with a `Co-authored-by` trailer) and additionally forwards the window onto the client `101` on the WS path, which #582/#590's capture-only WS code cannot do. Both can be closed as superseded once this merges — GitHub closing keywords only auto-close issues (hence `Fixes #577` above), not PRs, so #582/#590 need a manual close. ### WebSocket (`gpt-5.4+`) OpenAI delivers `x-codex-*` **only** on the upstream WS handshake response, never in data frames. `handle_openai_responses_ws` accepted the client WS *before* it connected upstream and never read `upstream.response.headers`, so the window was dropped. This reorders the handler to **connect upstream first**, extract the `x-codex-*` subset, then **accept the client WS with those headers attached** to the `101`, and refresh the Python state for `/stats` parity. ### Streaming SSE (incorporated from #582, @m16khb) Codex CLI almost always streams. `streaming.py` neither captured `x-codex-*` into `CodexRateLimitState` nor forwarded it — the forwarded-header filter matched only the substring `"ratelimit"`, which `x-codex-*` does not contain. This calls `update_from_headers()` **before** the `>=400` early-return (so a streaming 429/5xx still refreshes the window, matching the non-streaming handlers) and widens the forward filter to pass `x-codex-*`. > Credit: the SSE fix is @m16khb's work from #582, carried here verbatim with a > `Co-authored-by` trailer so the maintainer gets a single PR covering both > transports. This supersedes #582/#590's **WS** capture (which only writes > `/stats`); the connect-before-accept reorder additionally forwards the window to > the client `101`, which capture-only cannot do. #590's optional snapshot > persistence is intentionally left out (separable; hot-path sync write; doesn't > help the `101`-sniff consumers). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `openai.py`: add `_extract_codex_handshake_headers()` (strictly `x-codex-*`, via `raw_items()` to avoid `MultipleValuesError`; never `set-cookie`/`authorization`). - `openai.py`: reorder `handle_openai_responses_ws` — connect-only retry loop runs before `accept()`; `accept(headers=...)` carries the forwarded window; first client frame read afterward. HTTP fallback preserved; it now also refreshes `/stats` from the HTTP response headers. - `streaming.py`: capture `x-codex-*` on all statuses + widen the forwarded-header filter (from #582). ### Diff-size note The bulk of the `openai.py` line count is **whitespace-only relocation**: the relay block dedents one level out of the old per-attempt `async with`. Logical change is ~290 lines. **Review with `?w=1`.** In API-key mode the handshake carries no `x-codex-*`, so the accept-header list is empty and the path behaves exactly as before — the fix only activates for ChatGPT-subscription auth. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed - WS: `test_ws_connect_happens_before_accept`, `test_ws_forwards_codex_headers_to_client_accept` (only `x-codex-*` forwarded; `set-cookie`/`authorization` excluded; `/stats` refreshed), `test_ws_connect_failure_falls_back_to_http`, `test_ws_first_frame_timeout_after_connect_closes_upstream`. - Fallback: `test_fallback_refreshes_codex_rate_limit_state`. - SSE: `test_codex_rate_limit_headers_captured_and_forwarded_in_streaming`, `test_codex_rate_limit_captured_on_streaming_429` (from #582). - Wire-level e2e: `tests/e2e_ws_codex_usage_headers.py` boots the real proxy + fake upstream + real `websockets` client and reads the client `101` — closes the gap the unit tests stub (that uvicorn/starlette actually write `accept(headers=...)`). ## Test Output ``` $ uv run pytest tests/test_proxy_streaming_ratelimit_headers.py \ tests/test_ws_http_fallback.py \ tests/test_openai_codex_ws_lifecycle.py \ tests/test_openai_codex_ws_timings.py \ tests/test_codex_rate_limits.py -q 63 passed in 0.83s $ .venv/bin/python tests/e2e_ws_codex_usage_headers.py [codex-hdr-e2e] client 101 headers: x-codex-primary-used-percent: 42 x-codex-primary-window-minutes: 300 x-codex-secondary-used-percent: 7 x-codex-secondary-window-minutes: 10080 [codex-hdr-e2e] /stats reflects codex window (primary-used=42) === CODEX-HDR E2E ALL GREEN === $ uv run ruff check . && uv run ruff format --check <touched files> All checks passed! ``` ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - **Why connect-before-accept (not capture-only).** Once `accept()` sends the `101`, headers can no longer be added; the `x-codex-*` window only exists after we connect upstream. Capturing into Python state (as #582/#590's WS code does) fixes `/stats` but not the Codex CLI's native display or any `101`-sniffing consumer — those need the headers *on the client handshake*, which requires the reorder. - **Security.** Forwarding is filtered strictly to `x-codex-*`; `set-cookie`, `authorization`, and all other upstream headers are never forwarded to the client (asserted by both the unit test and the e2e). 🤖 Generated with [Claude Code](https://claude.com/claude-code) ## Contract Schemas Per maintainer request: a JSON Schema (draft 2020-12) artifact enshrining the OpenAI interaction expectations this changeset relies on, so drift is detectable later. Committed following the repo's parity convention: - schema: `tests/parity/fixtures/codex_openai_contracts/codex-openai-interaction.schema.json` - test: `tests/test_codex_openai_contract_parity.py` binds the schema to the **live code** in both directions, so drift fails CI rather than living only in this description - every declared `x-codex-*` header must be consumed by `parse_codex_rate_limits`, and `_extract_codex_handshake_headers` must forward exactly the declared subset and never `set-cookie`/`authorization`. No new dependency (does not pull in `jsonschema`). It covers, as `$defs`: - `WSUpstreamHandshakeResponse` / `StreamingUpstreamResponseHeaders` - the upstream `x-codex-*` header family (full superset, with per-header wire pattern + the parsed semantic type) the WS and SSE captures read. Source of truth: `parse_codex_rate_limits`. - `ClientForwardedHandshakeHeaders` - the WS-101 **allow/deny** contract: only `x-codex-*` may be forwarded; `set-cookie`/`authorization` are explicitly forbidden (`propertyNames` + `not`). - `ClientForwardedStreamingHeaders` - the wider SSE forward set (`*ratelimit*` OR `x-codex*`). - `WSClientRequestFrame` / `WSRelayEvent` / `HTTPFallbackRequestBody` - the WS frame envelopes and the unwrapped HTTP-fallback POST body. - `CodexRateLimitStatsOutput` - the headroom `/stats` shape the parity tests assert. Validated with `jsonschema` (Draft202012 `check_schema` passes; positive instances from the e2e validate; negative instances - a leaked `set-cookie`, a fallback body still carrying a top-level `type` - are correctly rejected). <details> <summary><code>codex-openai-interaction.schema.json</code> (draft 2020-12)</summary> ```json { "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://github.com/chopratejas/headroom/contracts/codex-openai-interaction.schema.json", "title": "Codex <-> OpenAI interaction contracts (PR #794)", "description": "Enshrines the OpenAI interaction expectations this changeset depends on, so drift is detectable. Header values are transported as strings on the wire; the `x-headroom-parsed-type` annotation on each records the semantic type the parser (headroom/subscription/codex_rate_limits.py) coerces them to. Sources: codex_rate_limits.parse_codex_rate_limits (header family + gating), openai._extract_codex_handshake_headers (WS-101 forward filter), streaming.py (SSE forward filter).", "$defs": { "OpenAICodexWindowHeaders": { "title": "x-codex-*-{primary,secondary} window headers", "description": "A rolling rate-limit/subscription window. A window is materialized iff its `*-used-percent` header is present and numeric; `*-window-minutes` and `*-reset-at` are optional. `primary` and `secondary` are independent and either may be absent.", "type": "object", "properties": { "x-codex-primary-used-percent": { "type": "string", "pattern": "^\\d+(?:\\.\\d+)?$", "x-headroom-parsed-type": "float (0-100, NaN-guarded)", "description": "Percent of the primary window consumed. Gates creation of the primary window." }, "x-codex-primary-window-minutes": { "type": "string", "pattern": "^\\d+$", "x-headroom-parsed-type": "int", "description": "Primary window size in minutes." }, "x-codex-primary-reset-at": { "type": "string", "pattern": "^\\d+$", "x-headroom-parsed-type": "int (Unix epoch seconds)", "description": "Absolute reset time of the primary window." }, "x-codex-secondary-used-percent": { "type": "string", "pattern": "^\\d+(?:\\.\\d+)?$", "x-headroom-parsed-type": "float (0-100, NaN-guarded)", "description": "Percent of the secondary window consumed. Gates creation of the secondary window." }, "x-codex-secondary-window-minutes": { "type": "string", "pattern": "^\\d+$", "x-headroom-parsed-type": "int" }, "x-codex-secondary-reset-at": { "type": "string", "pattern": "^\\d+$", "x-headroom-parsed-type": "int (Unix epoch seconds)" } }, "additionalProperties": true }, "OpenAICodexCreditsHeaders": { "title": "x-codex-credits-* headers", "description": "OpenAI credits balance. A credits snapshot is materialized iff `x-codex-credits-has-credits` is present; `unlimited` defaults to false; `balance` is optional.", "type": "object", "properties": { "x-codex-credits-has-credits": { "type": "string", "pattern": "^(?:[Tt][Rr][Uu][Ee]|[Ff][Aa][Ll][Ss][Ee]|[01])$", "x-headroom-parsed-type": "bool (true|false|1|0, case-insensitive)", "description": "Gates creation of the credits snapshot." }, "x-codex-credits-unlimited": { "type": "string", "pattern": "^(?:[Tt][Rr][Uu][Ee]|[Ff][Aa][Ll][Ss][Ee]|[01])$", "x-headroom-parsed-type": "bool (defaults false when absent/unparseable)" }, "x-codex-credits-balance": { "type": "string", "x-headroom-parsed-type": "str (empty -> null)", "description": "Free-form server string, e.g. \"$5.00\"." } }, "additionalProperties": true }, "OpenAICodexMetaHeaders": { "title": "x-codex meta headers", "type": "object", "properties": { "x-codex-limit-name": { "type": "string", "x-headroom-parsed-type": "str (empty -> null)", "description": "Active limit/model label, e.g. \"gpt-5.2-codex-sonic\"." }, "x-codex-promo-message": { "type": "string", "x-headroom-parsed-type": "str (empty -> null)", "description": "Server announcement. Also gates snapshot creation when present." } }, "additionalProperties": true }, "OpenAICodexRateLimitHeaders": { "title": "Full x-codex-* header family OpenAI may emit", "description": "Superset of every x-codex-* header headroom reads. parse_codex_rate_limits returns a snapshot iff at least one of: a primary window, a secondary window, a credits snapshot, or a non-empty promo message is present; otherwise null (treated as a non-Codex response). All members are individually optional.", "type": "object", "allOf": [ { "$ref": "#/$defs/OpenAICodexWindowHeaders" }, { "$ref": "#/$defs/OpenAICodexCreditsHeaders" }, { "$ref": "#/$defs/OpenAICodexMetaHeaders" } ], "additionalProperties": true }, "WSUpstreamHandshakeResponse": { "title": "OpenAI WS handshake (101) response headers consumed by the WS fix", "description": "On the Codex WebSocket transport the x-codex-* window is delivered ONLY on the upstream handshake response (never in data frames). handle_openai_responses_ws reads upstream.response.headers here. This is the contract the connect-before-accept reorder depends on: if OpenAI ever moves these headers off the handshake (e.g. into a frame), the WS half of the fix goes stale.", "$ref": "#/$defs/OpenAICodexRateLimitHeaders" }, "StreamingUpstreamResponseHeaders": { "title": "OpenAI streaming/HTTP response headers consumed by the SSE fix", "description": "On the streaming SSE/HTTP transport the same x-codex-* headers ride the HTTP response. streaming.py captures them on ALL statuses (including >=400) via update_from_headers, and forwards a wider set to the client (see ClientForwardedStreamingHeaders).", "$ref": "#/$defs/OpenAICodexRateLimitHeaders" }, "ClientForwardedHandshakeHeaders": { "title": "Headers forwarded onto the CLIENT-facing WS 101 (allow/deny contract)", "description": "_extract_codex_handshake_headers forwards ONLY headers whose (lowercased) name starts with `x-codex-`. Every other upstream handshake header - notably set-cookie and authorization - MUST NOT appear on the client 101. Enforced by propertyNames below and asserted by the unit tests + tests/e2e_ws_codex_usage_headers.py.", "type": "object", "propertyNames": { "pattern": "^[Xx]-[Cc][Oo][Dd][Ee][Xx]-" }, "not": { "anyOf": [ { "required": ["set-cookie"] }, { "required": ["Set-Cookie"] }, { "required": ["authorization"] }, { "required": ["Authorization"] } ] }, "additionalProperties": { "type": "string" } }, "ClientForwardedStreamingHeaders": { "title": "Headers forwarded to the client on the streaming SSE path", "description": "streaming.py forwards a header iff `\"ratelimit\" in name.lower()` OR `name.lower().startswith(\"x-codex\")`. This is a SUPERSET of the WS allow-list: it additionally passes generic *ratelimit* headers (e.g. the Anthropic streaming path) which do not contain the x-codex prefix.", "type": "object", "propertyNames": { "pattern": "(?:[Rr][Aa][Tt][Ee][Ll][Ii][Mm][Ii][Tt])|^[Xx]-[Cc][Oo][Dd][Ee][Xx]" }, "additionalProperties": { "type": "string" } }, "WSClientRequestFrame": { "title": "Client -> proxy WS data frame (Responses API over WS)", "description": "Codex sends the request as a response.create envelope. The HTTP fallback unwraps `.response` for the POST body, forces stream=true, and strips any top-level `type`. A flattened variant (no envelope, fields at top level) is also tolerated by the fallback.", "type": "object", "properties": { "type": { "const": "response.create" }, "response": { "type": "object", "properties": { "model": { "type": "string", "description": "e.g. gpt-5.4" }, "input": { "description": "String prompt or Responses-API structured input array.", "type": ["string", "array"] }, "stream": { "type": "boolean" } }, "required": ["model"], "additionalProperties": true } }, "required": ["type", "response"], "additionalProperties": true }, "WSRelayEvent": { "title": "proxy -> client WS data frame (relayed Responses API event)", "description": "SSE `data:` payloads relayed verbatim as WS text frames. `[DONE]` sentinels are dropped (not relayed). Every relayed event is a JSON object carrying a `type`. response.completed additionally carries usage under `response.usage`. anyOf (not oneOf): an error event also satisfies the looser lifecycle shape, which is fine.", "anyOf": [ { "title": "lifecycle event", "type": "object", "properties": { "type": { "type": "string", "examples": [ "response.created", "response.output_item.added", "response.completed" ] }, "response": { "type": "object", "additionalProperties": true } }, "required": ["type"], "additionalProperties": true }, { "title": "error event", "type": "object", "properties": { "type": { "const": "error" }, "error": { "type": "object", "properties": { "message": { "type": "string" } }, "required": ["message"], "additionalProperties": true } }, "required": ["type", "error"], "additionalProperties": true } ] }, "HTTPFallbackRequestBody": { "title": "proxy -> OpenAI HTTP POST body on WS->HTTP fallback", "description": "Derived from WSClientRequestFrame: the inner `.response` object, with `stream` forced to true and any top-level `type` removed.", "type": "object", "properties": { "model": { "type": "string" }, "stream": { "const": true }, "input": { "type": ["string", "array"] } }, "required": ["model", "stream"], "not": { "required": ["type"] }, "additionalProperties": true }, "CodexRateLimitStatsOutput": { "title": "headroom /stats output for the codex tracker (CodexRateLimitSnapshot.to_dict)", "description": "Internal (headroom-emitted) shape produced from the headers above; the WS and SSE update_from_headers parity tests assert this is refreshed. Included so drift in our own surface is also caught.", "type": "object", "properties": { "limit_id": { "const": "codex" }, "limit_name": { "type": ["string", "null"] }, "primary": { "$ref": "#/$defs/CodexWindowDict" }, "secondary": { "$ref": "#/$defs/CodexWindowDict" }, "credits": { "oneOf": [ { "type": "null" }, { "type": "object", "properties": { "has_credits": { "type": "boolean" }, "unlimited": { "type": "boolean" }, "balance": { "type": ["string", "null"] } }, "required": ["has_credits", "unlimited", "balance"], "additionalProperties": false } ] }, "promo_message": { "type": ["string", "null"] }, "captured_at": { "type": "number", "description": "Unix epoch seconds (float)." } }, "required": ["limit_id", "limit_name", "primary", "secondary", "credits", "promo_message", "captured_at"], "additionalProperties": false }, "CodexWindowDict": { "oneOf": [ { "type": "null" }, { "type": "object", "properties": { "used_percent": { "type": "number" }, "window_minutes": { "type": ["integer", "null"] }, "window_label": { "type": "string", "description": "e.g. \"5h\", \"7d\"-style label; \"unknown\" when window_minutes is null." }, "resets_at": { "type": ["integer", "null"], "description": "Unix epoch seconds." }, "seconds_until_reset": { "type": ["integer", "null"] } }, "required": ["used_percent", "window_minutes", "window_label", "resets_at", "seconds_until_reset"], "additionalProperties": false } ] } } } ``` </details> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: m16khb <m16khb@gmail.com> |
||
|
chopratejas
|
993c9076f9 |
refactor(proxy): migrate Codex WS + OpenAI HTTP + batch handlers; delete Databricks
Completes the migration of every ``metrics.record_request`` call site in ``headroom/proxy/handlers/`` onto the canonical funnel. After this commit, **zero ad-hoc record_request calls remain** across the entire handler subtree. Every request — regardless of provider, harness, or transport — flows through ``emit_request_outcome``. Migrated sites (this commit): * **handle_openai_responses_ws** (Codex WS) — 2 sites: - per-turn record (per ``response.completed``) - session-end residual (leftover tokens not captured per-turn) Pre-refactor these sites emitted only metrics + cost_tracker — no RequestLog, no PERF — so Codex traffic was invisible to ``headroom perf`` and the recent-requests feed. Funnel restores all four effects uniformly per turn. (Closes the visibility half of what #471's sibling PR addressed for the scheduler half.) The explicit session-summary RequestLog at session-end stays as a separate explicit log entry — it's a session-cumulative summary, distinct from per-turn observations. * **handle_openai_chat** — 3 sites: - response-cache hit (uses ``from_response_cache=True``) - backend-routed (LiteLLM/AnyLLM) non-streaming success - direct OpenAI non-streaming success * **handle_openai_responses** HTTP (Codex HTTP transport) — 1 site * **handle_passthrough** (OpenAI passthrough endpoints) — 1 site * **batch.py** handlers — 5 sites: - handle_google_batch_create - handle_google_batch_passthrough (Files API forward) - handle_google_batch_passthrough (list/get/cancel) - handle_google_batch_results (CCR-processed) - handle_batch_create (OpenAI batches) All converge on the funnel. Several gain request_id allocation they didn't have before (passthrough sites previously emitted ``request_id=None`` in logs). **Deleted: handle_databricks_invocations + its route + test cases.** Databricks was a 57-line thin wrapper at openai.py that parsed JSON, injected the model from URL into body, and delegated to ``handle_openai_chat``. It enabled ``databricks serving-endpoints query <model> --profile HEADROOM`` direct CLI use. No evidence of active users (no docs, no issues, no mentions). Databricks-hosted models still work via the standard ``/v1/chat/completions`` surface; LiteLLM has its own Databricks support too. If a user complains, this PR is a 30-minute revert. Architectural note: also updated 2 more test dummies (``_DummyOpenAIHandler`` in routing + WS lifecycle tests) to bind ``_record_request_outcome`` via the free function ``emit_request_outcome`` — same pattern as ``_run_compression_in_executor``. Final migration tally (from P0 audit + extensions): * **18 audit sites** + **5 batch.py sites discovered during migration** = 23 sites migrated * **1 site deleted** (Databricks) * **0 sites remaining** anywhere under ``handlers/`` Surface impact (this commit): * openai.py: −168 LOC (315 deletions − 147 insertions) * batch.py: +35 LOC (124 ins − 89 del; mostly comments) * proxy_routes.py: −4 LOC (Databricks route gone) * tests: +11 LOC (dummy `_record_request_outcome` bindings, 2 sites) * Net: ~−126 LOC in production handler code Tests * All 157 existing streaming/cache/Codex/anthropic/openai/backpressure/ routes tests pass with zero regressions. * ruff + ruff-format + mypy clean. This brings the cumulative refactor delta (across all 3 commits on this branch) to: contract introduced (outcome.py + funnel): ~+200 LOC fixed cost handler migrations (streaming + anthropic + gemini + openai + batch + WS): ~−700 LOC Databricks deletion: −57 LOC ────────────────────────────────────────────── ───────── Net production code delta: ~−557 LOC Plus +474 LOC of test coverage (RequestOutcome unit tests + funnel contract assertions). And every handler now emits identical observable outputs per request: same metrics shape, same cost_tracker shape, same RequestLog shape, same PERF format. The wire is uniform. |
||
|
Tejas Chopra
|
03d12fc140 | fix: format Codex compression changes | ||
|
Tejas Chopra
|
e9cae0131b |
fix: expose compression latency bottlenecks
Add Codex WS unit-level timing and bounded parallel compression, clarify context-tool session savings, and avoid costly diff/log fallbacks to Kompress. |
||
|
Gili Tzabari
|
28a0e19deb | Merge origin/main into fix/codex-ws-dashboard-performance | ||
|
Gili Tzabari
|
4279a7f353 | fix: populate Codex WS dashboard performance metrics | ||
|
Gili Tzabari
|
160989c43e | fix(proxy): bound Codex Responses compression work | ||
|
Tejas Chopra
|
62a1f23b88 | fix: log Codex ws cancellations safely | ||
|
Tejas Chopra
|
eaf5980b4a | fix: stabilize codex compression, stats, and proxy lifecycle | ||
|
Tejas Chopra
|
2b331e297a | fix: add Codex wire debug and WS usage metrics | ||
|
Adryan Eka Vandra
|
b71b659e1b
|
fix(ci): restore repro harness test in dev installs
Add websockets to the dev extra so the repro harness smoke test can import its websocket client dependency in the CI test matrix. Also apply ruff formatting to the files the formatter check was rejecting so the 3.12 lint job passes. |
||
|
Adryan Eka Vandra
|
a31d81a426
|
feat(proxy): track Codex WS sessions and cancel relay tasks deterministically
Unit 3 of the Codex proxy resilience plan. Eliminates the "aged process
has leaked relay tasks" hypothesis by making every WS session explicitly
tracked and both relay tasks deterministically cancelled when either
exits.
- New headroom/proxy/ws_session_registry.py: dict-backed
WebSocketSessionRegistry + WSSessionHandle with register /
deregister / attach_tasks / snapshot. Deregister is idempotent and
clears relay-task references so coroutine frames are not retained
past session end.
- HeadroomProxy exposes proxy.ws_sessions so /debug/ws-sessions
(Unit 5) can read the live snapshot.
- handle_openai_responses_ws now registers on websocket.accept()
success and deregisters in the outermost finally so no leak can
survive handshake-phase, mid-stream, or upstream-error paths. The
session_id / termination_cause is threaded through both relay
halves and both sides raise asyncio.CancelledError cleanly.
- Replaced asyncio.gather(_client_to_upstream(), _upstream_to_client(),
return_exceptions=True) with explicit asyncio.create_task(...)
(named codex-ws-c2u-<sid> / codex-ws-u2c-<sid>) +
asyncio.wait(FIRST_COMPLETED) + cancel-and-await on the survivor.
Termination cause is classified as client_disconnect /
client_error / upstream_disconnect / upstream_error /
response_completed from which task completed first plus inline
error captures from the halves.
- Prometheus metrics: new active_ws_sessions and active_relay_tasks
gauges plus ws_session_duration_ms_{sum,count,max} histogram
bucketed by termination cause. Mirrors the Unit 2 stage_timing_*
shape.
Preserved: upstream WS retry loop, WS→HTTP fallback, memory-context
timeout, compression pipeline, Unit 2 stage timings. Memory-tool
execution inside _upstream_to_client still runs when the client task
exits first; however, if the client disconnects *before* the upstream
emits response.completed, pending memory writes in `pending_fcs` are
dropped (unchanged from prior behavior — a crashing upstream has the
same effect). Note: handle_openai_responses (HTTP, line ~800) is a
single-shot HTTP request; lifecycle tracking isn't added there
(scope boundary).
Tests:
- tests/test_ws_session_registry.py: 8 registry unit tests
(register/deregister idempotency, snapshot shape, attach merging,
reference release).
- tests/test_openai_codex_ws_lifecycle.py: 6 integration tests
using real relay tasks (only upstream WS endpoint mocked):
happy-path, failing-test-first "client disconnect cancels upstream
relay within 100 ms", upstream-closes-first, upstream-error mid-
stream, handshake-failure deregister, 50 concurrent sessions.
- Regression: test_openai_codex_ws_timings, test_openai_codex_routing,
test_proxy_codex_route_aliases, test_ws_memory_relay all pass.
- Tests pass under python -W error::RuntimeWarning (no "coroutine
was never awaited").
|