mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
## Description Codex's subscription/rate-limit window (the `x-codex-*` headers) was being **stripped on every transport Codex actually uses**, so session/weekly usage never reached the Codex CLI's own `/status` display, Headroom `/stats`/dashboard, or any consumer that sniffs the client-facing handshake. This PR restores it on **both** the WebSocket and streaming-SSE paths — the two halves of #577 — in one place. Fixes #577 **Supersedes #582 and #590.** This PR incorporates #582's SSE fix (carried verbatim with a `Co-authored-by` trailer) and additionally forwards the window onto the client `101` on the WS path, which #582/#590's capture-only WS code cannot do. Both can be closed as superseded once this merges — GitHub closing keywords only auto-close issues (hence `Fixes #577` above), not PRs, so #582/#590 need a manual close. ### WebSocket (`gpt-5.4+`) OpenAI delivers `x-codex-*` **only** on the upstream WS handshake response, never in data frames. `handle_openai_responses_ws` accepted the client WS *before* it connected upstream and never read `upstream.response.headers`, so the window was dropped. This reorders the handler to **connect upstream first**, extract the `x-codex-*` subset, then **accept the client WS with those headers attached** to the `101`, and refresh the Python state for `/stats` parity. ### Streaming SSE (incorporated from #582, @m16khb) Codex CLI almost always streams. `streaming.py` neither captured `x-codex-*` into `CodexRateLimitState` nor forwarded it — the forwarded-header filter matched only the substring `"ratelimit"`, which `x-codex-*` does not contain. This calls `update_from_headers()` **before** the `>=400` early-return (so a streaming 429/5xx still refreshes the window, matching the non-streaming handlers) and widens the forward filter to pass `x-codex-*`. > Credit: the SSE fix is @m16khb's work from #582, carried here verbatim with a > `Co-authored-by` trailer so the maintainer gets a single PR covering both > transports. This supersedes #582/#590's **WS** capture (which only writes > `/stats`); the connect-before-accept reorder additionally forwards the window to > the client `101`, which capture-only cannot do. #590's optional snapshot > persistence is intentionally left out (separable; hot-path sync write; doesn't > help the `101`-sniff consumers). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `openai.py`: add `_extract_codex_handshake_headers()` (strictly `x-codex-*`, via `raw_items()` to avoid `MultipleValuesError`; never `set-cookie`/`authorization`). - `openai.py`: reorder `handle_openai_responses_ws` — connect-only retry loop runs before `accept()`; `accept(headers=...)` carries the forwarded window; first client frame read afterward. HTTP fallback preserved; it now also refreshes `/stats` from the HTTP response headers. - `streaming.py`: capture `x-codex-*` on all statuses + widen the forwarded-header filter (from #582). ### Diff-size note The bulk of the `openai.py` line count is **whitespace-only relocation**: the relay block dedents one level out of the old per-attempt `async with`. Logical change is ~290 lines. **Review with `?w=1`.** In API-key mode the handshake carries no `x-codex-*`, so the accept-header list is empty and the path behaves exactly as before — the fix only activates for ChatGPT-subscription auth. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed - WS: `test_ws_connect_happens_before_accept`, `test_ws_forwards_codex_headers_to_client_accept` (only `x-codex-*` forwarded; `set-cookie`/`authorization` excluded; `/stats` refreshed), `test_ws_connect_failure_falls_back_to_http`, `test_ws_first_frame_timeout_after_connect_closes_upstream`. - Fallback: `test_fallback_refreshes_codex_rate_limit_state`. - SSE: `test_codex_rate_limit_headers_captured_and_forwarded_in_streaming`, `test_codex_rate_limit_captured_on_streaming_429` (from #582). - Wire-level e2e: `tests/e2e_ws_codex_usage_headers.py` boots the real proxy + fake upstream + real `websockets` client and reads the client `101` — closes the gap the unit tests stub (that uvicorn/starlette actually write `accept(headers=...)`). ## Test Output ``` $ uv run pytest tests/test_proxy_streaming_ratelimit_headers.py \ tests/test_ws_http_fallback.py \ tests/test_openai_codex_ws_lifecycle.py \ tests/test_openai_codex_ws_timings.py \ tests/test_codex_rate_limits.py -q 63 passed in 0.83s $ .venv/bin/python tests/e2e_ws_codex_usage_headers.py [codex-hdr-e2e] client 101 headers: x-codex-primary-used-percent: 42 x-codex-primary-window-minutes: 300 x-codex-secondary-used-percent: 7 x-codex-secondary-window-minutes: 10080 [codex-hdr-e2e] /stats reflects codex window (primary-used=42) === CODEX-HDR E2E ALL GREEN === $ uv run ruff check . && uv run ruff format --check <touched files> All checks passed! ``` ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - **Why connect-before-accept (not capture-only).** Once `accept()` sends the `101`, headers can no longer be added; the `x-codex-*` window only exists after we connect upstream. Capturing into Python state (as #582/#590's WS code does) fixes `/stats` but not the Codex CLI's native display or any `101`-sniffing consumer — those need the headers *on the client handshake*, which requires the reorder. - **Security.** Forwarding is filtered strictly to `x-codex-*`; `set-cookie`, `authorization`, and all other upstream headers are never forwarded to the client (asserted by both the unit test and the e2e). 🤖 Generated with [Claude Code](https://claude.com/claude-code) ## Contract Schemas Per maintainer request: a JSON Schema (draft 2020-12) artifact enshrining the OpenAI interaction expectations this changeset relies on, so drift is detectable later. Committed following the repo's parity convention: - schema: `tests/parity/fixtures/codex_openai_contracts/codex-openai-interaction.schema.json` - test: `tests/test_codex_openai_contract_parity.py` binds the schema to the **live code** in both directions, so drift fails CI rather than living only in this description - every declared `x-codex-*` header must be consumed by `parse_codex_rate_limits`, and `_extract_codex_handshake_headers` must forward exactly the declared subset and never `set-cookie`/`authorization`. No new dependency (does not pull in `jsonschema`). It covers, as `$defs`: - `WSUpstreamHandshakeResponse` / `StreamingUpstreamResponseHeaders` - the upstream `x-codex-*` header family (full superset, with per-header wire pattern + the parsed semantic type) the WS and SSE captures read. Source of truth: `parse_codex_rate_limits`. - `ClientForwardedHandshakeHeaders` - the WS-101 **allow/deny** contract: only `x-codex-*` may be forwarded; `set-cookie`/`authorization` are explicitly forbidden (`propertyNames` + `not`). - `ClientForwardedStreamingHeaders` - the wider SSE forward set (`*ratelimit*` OR `x-codex*`). - `WSClientRequestFrame` / `WSRelayEvent` / `HTTPFallbackRequestBody` - the WS frame envelopes and the unwrapped HTTP-fallback POST body. - `CodexRateLimitStatsOutput` - the headroom `/stats` shape the parity tests assert. Validated with `jsonschema` (Draft202012 `check_schema` passes; positive instances from the e2e validate; negative instances - a leaked `set-cookie`, a fallback body still carrying a top-level `type` - are correctly rejected). <details> <summary><code>codex-openai-interaction.schema.json</code> (draft 2020-12)</summary> ```json { "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://github.com/chopratejas/headroom/contracts/codex-openai-interaction.schema.json", "title": "Codex <-> OpenAI interaction contracts (PR #794)", "description": "Enshrines the OpenAI interaction expectations this changeset depends on, so drift is detectable. Header values are transported as strings on the wire; the `x-headroom-parsed-type` annotation on each records the semantic type the parser (headroom/subscription/codex_rate_limits.py) coerces them to. Sources: codex_rate_limits.parse_codex_rate_limits (header family + gating), openai._extract_codex_handshake_headers (WS-101 forward filter), streaming.py (SSE forward filter).", "$defs": { "OpenAICodexWindowHeaders": { "title": "x-codex-*-{primary,secondary} window headers", "description": "A rolling rate-limit/subscription window. A window is materialized iff its `*-used-percent` header is present and numeric; `*-window-minutes` and `*-reset-at` are optional. `primary` and `secondary` are independent and either may be absent.", "type": "object", "properties": { "x-codex-primary-used-percent": { "type": "string", "pattern": "^\\d+(?:\\.\\d+)?$", "x-headroom-parsed-type": "float (0-100, NaN-guarded)", "description": "Percent of the primary window consumed. Gates creation of the primary window." }, "x-codex-primary-window-minutes": { "type": "string", "pattern": "^\\d+$", "x-headroom-parsed-type": "int", "description": "Primary window size in minutes." }, "x-codex-primary-reset-at": { "type": "string", "pattern": "^\\d+$", "x-headroom-parsed-type": "int (Unix epoch seconds)", "description": "Absolute reset time of the primary window." }, "x-codex-secondary-used-percent": { "type": "string", "pattern": "^\\d+(?:\\.\\d+)?$", "x-headroom-parsed-type": "float (0-100, NaN-guarded)", "description": "Percent of the secondary window consumed. Gates creation of the secondary window." }, "x-codex-secondary-window-minutes": { "type": "string", "pattern": "^\\d+$", "x-headroom-parsed-type": "int" }, "x-codex-secondary-reset-at": { "type": "string", "pattern": "^\\d+$", "x-headroom-parsed-type": "int (Unix epoch seconds)" } }, "additionalProperties": true }, "OpenAICodexCreditsHeaders": { "title": "x-codex-credits-* headers", "description": "OpenAI credits balance. A credits snapshot is materialized iff `x-codex-credits-has-credits` is present; `unlimited` defaults to false; `balance` is optional.", "type": "object", "properties": { "x-codex-credits-has-credits": { "type": "string", "pattern": "^(?:[Tt][Rr][Uu][Ee]|[Ff][Aa][Ll][Ss][Ee]|[01])$", "x-headroom-parsed-type": "bool (true|false|1|0, case-insensitive)", "description": "Gates creation of the credits snapshot." }, "x-codex-credits-unlimited": { "type": "string", "pattern": "^(?:[Tt][Rr][Uu][Ee]|[Ff][Aa][Ll][Ss][Ee]|[01])$", "x-headroom-parsed-type": "bool (defaults false when absent/unparseable)" }, "x-codex-credits-balance": { "type": "string", "x-headroom-parsed-type": "str (empty -> null)", "description": "Free-form server string, e.g. \"$5.00\"." } }, "additionalProperties": true }, "OpenAICodexMetaHeaders": { "title": "x-codex meta headers", "type": "object", "properties": { "x-codex-limit-name": { "type": "string", "x-headroom-parsed-type": "str (empty -> null)", "description": "Active limit/model label, e.g. \"gpt-5.2-codex-sonic\"." }, "x-codex-promo-message": { "type": "string", "x-headroom-parsed-type": "str (empty -> null)", "description": "Server announcement. Also gates snapshot creation when present." } }, "additionalProperties": true }, "OpenAICodexRateLimitHeaders": { "title": "Full x-codex-* header family OpenAI may emit", "description": "Superset of every x-codex-* header headroom reads. parse_codex_rate_limits returns a snapshot iff at least one of: a primary window, a secondary window, a credits snapshot, or a non-empty promo message is present; otherwise null (treated as a non-Codex response). All members are individually optional.", "type": "object", "allOf": [ { "$ref": "#/$defs/OpenAICodexWindowHeaders" }, { "$ref": "#/$defs/OpenAICodexCreditsHeaders" }, { "$ref": "#/$defs/OpenAICodexMetaHeaders" } ], "additionalProperties": true }, "WSUpstreamHandshakeResponse": { "title": "OpenAI WS handshake (101) response headers consumed by the WS fix", "description": "On the Codex WebSocket transport the x-codex-* window is delivered ONLY on the upstream handshake response (never in data frames). handle_openai_responses_ws reads upstream.response.headers here. This is the contract the connect-before-accept reorder depends on: if OpenAI ever moves these headers off the handshake (e.g. into a frame), the WS half of the fix goes stale.", "$ref": "#/$defs/OpenAICodexRateLimitHeaders" }, "StreamingUpstreamResponseHeaders": { "title": "OpenAI streaming/HTTP response headers consumed by the SSE fix", "description": "On the streaming SSE/HTTP transport the same x-codex-* headers ride the HTTP response. streaming.py captures them on ALL statuses (including >=400) via update_from_headers, and forwards a wider set to the client (see ClientForwardedStreamingHeaders).", "$ref": "#/$defs/OpenAICodexRateLimitHeaders" }, "ClientForwardedHandshakeHeaders": { "title": "Headers forwarded onto the CLIENT-facing WS 101 (allow/deny contract)", "description": "_extract_codex_handshake_headers forwards ONLY headers whose (lowercased) name starts with `x-codex-`. Every other upstream handshake header - notably set-cookie and authorization - MUST NOT appear on the client 101. Enforced by propertyNames below and asserted by the unit tests + tests/e2e_ws_codex_usage_headers.py.", "type": "object", "propertyNames": { "pattern": "^[Xx]-[Cc][Oo][Dd][Ee][Xx]-" }, "not": { "anyOf": [ { "required": ["set-cookie"] }, { "required": ["Set-Cookie"] }, { "required": ["authorization"] }, { "required": ["Authorization"] } ] }, "additionalProperties": { "type": "string" } }, "ClientForwardedStreamingHeaders": { "title": "Headers forwarded to the client on the streaming SSE path", "description": "streaming.py forwards a header iff `\"ratelimit\" in name.lower()` OR `name.lower().startswith(\"x-codex\")`. This is a SUPERSET of the WS allow-list: it additionally passes generic *ratelimit* headers (e.g. the Anthropic streaming path) which do not contain the x-codex prefix.", "type": "object", "propertyNames": { "pattern": "(?:[Rr][Aa][Tt][Ee][Ll][Ii][Mm][Ii][Tt])|^[Xx]-[Cc][Oo][Dd][Ee][Xx]" }, "additionalProperties": { "type": "string" } }, "WSClientRequestFrame": { "title": "Client -> proxy WS data frame (Responses API over WS)", "description": "Codex sends the request as a response.create envelope. The HTTP fallback unwraps `.response` for the POST body, forces stream=true, and strips any top-level `type`. A flattened variant (no envelope, fields at top level) is also tolerated by the fallback.", "type": "object", "properties": { "type": { "const": "response.create" }, "response": { "type": "object", "properties": { "model": { "type": "string", "description": "e.g. gpt-5.4" }, "input": { "description": "String prompt or Responses-API structured input array.", "type": ["string", "array"] }, "stream": { "type": "boolean" } }, "required": ["model"], "additionalProperties": true } }, "required": ["type", "response"], "additionalProperties": true }, "WSRelayEvent": { "title": "proxy -> client WS data frame (relayed Responses API event)", "description": "SSE `data:` payloads relayed verbatim as WS text frames. `[DONE]` sentinels are dropped (not relayed). Every relayed event is a JSON object carrying a `type`. response.completed additionally carries usage under `response.usage`. anyOf (not oneOf): an error event also satisfies the looser lifecycle shape, which is fine.", "anyOf": [ { "title": "lifecycle event", "type": "object", "properties": { "type": { "type": "string", "examples": [ "response.created", "response.output_item.added", "response.completed" ] }, "response": { "type": "object", "additionalProperties": true } }, "required": ["type"], "additionalProperties": true }, { "title": "error event", "type": "object", "properties": { "type": { "const": "error" }, "error": { "type": "object", "properties": { "message": { "type": "string" } }, "required": ["message"], "additionalProperties": true } }, "required": ["type", "error"], "additionalProperties": true } ] }, "HTTPFallbackRequestBody": { "title": "proxy -> OpenAI HTTP POST body on WS->HTTP fallback", "description": "Derived from WSClientRequestFrame: the inner `.response` object, with `stream` forced to true and any top-level `type` removed.", "type": "object", "properties": { "model": { "type": "string" }, "stream": { "const": true }, "input": { "type": ["string", "array"] } }, "required": ["model", "stream"], "not": { "required": ["type"] }, "additionalProperties": true }, "CodexRateLimitStatsOutput": { "title": "headroom /stats output for the codex tracker (CodexRateLimitSnapshot.to_dict)", "description": "Internal (headroom-emitted) shape produced from the headers above; the WS and SSE update_from_headers parity tests assert this is refreshed. Included so drift in our own surface is also caught.", "type": "object", "properties": { "limit_id": { "const": "codex" }, "limit_name": { "type": ["string", "null"] }, "primary": { "$ref": "#/$defs/CodexWindowDict" }, "secondary": { "$ref": "#/$defs/CodexWindowDict" }, "credits": { "oneOf": [ { "type": "null" }, { "type": "object", "properties": { "has_credits": { "type": "boolean" }, "unlimited": { "type": "boolean" }, "balance": { "type": ["string", "null"] } }, "required": ["has_credits", "unlimited", "balance"], "additionalProperties": false } ] }, "promo_message": { "type": ["string", "null"] }, "captured_at": { "type": "number", "description": "Unix epoch seconds (float)." } }, "required": ["limit_id", "limit_name", "primary", "secondary", "credits", "promo_message", "captured_at"], "additionalProperties": false }, "CodexWindowDict": { "oneOf": [ { "type": "null" }, { "type": "object", "properties": { "used_percent": { "type": "number" }, "window_minutes": { "type": ["integer", "null"] }, "window_label": { "type": "string", "description": "e.g. \"5h\", \"7d\"-style label; \"unknown\" when window_minutes is null." }, "resets_at": { "type": ["integer", "null"], "description": "Unix epoch seconds." }, "seconds_until_reset": { "type": ["integer", "null"] } }, "required": ["used_percent", "window_minutes", "window_label", "resets_at", "seconds_until_reset"], "additionalProperties": false } ] } } } ``` </details> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: m16khb <m16khb@gmail.com>
817 lines
29 KiB
Python
817 lines
29 KiB
Python
"""Unit 3: WebSocket session lifecycle + deterministic relay cancellation.
|
|
|
|
These tests exercise the Codex WS handler with a fake upstream and a
|
|
fake client WebSocket so we can drive the relay halves through their
|
|
real code paths (not mocked) and assert on registry / task state.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
import json
|
|
import sys
|
|
from types import SimpleNamespace
|
|
from unittest.mock import MagicMock, patch
|
|
|
|
import pytest
|
|
|
|
from headroom.proxy.handlers.openai import OpenAIHandlerMixin
|
|
from headroom.proxy.helpers import COMPRESSION_TIMEOUT_SECONDS
|
|
from headroom.proxy.ws_session_registry import WebSocketSessionRegistry
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Test doubles
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class _DummyMetrics:
|
|
def __init__(self) -> None:
|
|
self.active_ws_sessions = 0
|
|
self.active_ws_sessions_max = 0
|
|
self.active_relay_tasks = 0
|
|
self.ws_session_durations: list[float] = []
|
|
self.stage_timings: list[tuple[str, dict[str, float]]] = []
|
|
self.termination_causes: list[str] = []
|
|
self.recorded_requests: list[dict] = []
|
|
|
|
async def record_request(self, **kwargs): # pragma: no cover
|
|
self.recorded_requests.append(dict(kwargs))
|
|
return None
|
|
|
|
async def record_stage_timings(self, path: str, timings: dict[str, float]) -> None:
|
|
self.stage_timings.append((path, dict(timings)))
|
|
|
|
def inc_active_ws_sessions(self) -> None:
|
|
self.active_ws_sessions += 1
|
|
self.active_ws_sessions_max = max(self.active_ws_sessions_max, self.active_ws_sessions)
|
|
|
|
def dec_active_ws_sessions(self) -> None:
|
|
self.active_ws_sessions = max(0, self.active_ws_sessions - 1)
|
|
|
|
def inc_active_relay_tasks(self, n: int = 1) -> None:
|
|
self.active_relay_tasks += n
|
|
|
|
def dec_active_relay_tasks(self, n: int = 1) -> None:
|
|
self.active_relay_tasks = max(0, self.active_relay_tasks - n)
|
|
|
|
def record_ws_session_duration(self, duration_ms: float, cause: str) -> None:
|
|
self.ws_session_durations.append(duration_ms)
|
|
self.termination_causes.append(cause)
|
|
|
|
|
|
class _DummyOpenAIHandler(OpenAIHandlerMixin):
|
|
OPENAI_API_URL = "https://api.openai.com"
|
|
|
|
def __init__(self, ws_sessions: WebSocketSessionRegistry | None = None) -> None:
|
|
self.rate_limiter = None
|
|
self.metrics = _DummyMetrics()
|
|
self.config = SimpleNamespace(
|
|
optimize=False,
|
|
retry_max_attempts=1,
|
|
retry_base_delay_ms=1,
|
|
retry_max_delay_ms=1,
|
|
connect_timeout_seconds=10,
|
|
)
|
|
self.usage_reporter = None
|
|
self.openai_provider = SimpleNamespace(get_context_limit=lambda model: 128_000)
|
|
self.openai_pipeline = SimpleNamespace(apply=MagicMock())
|
|
self.anthropic_backend = None
|
|
self.cost_tracker = None
|
|
self.memory_handler = None
|
|
self.ws_sessions = ws_sessions or WebSocketSessionRegistry()
|
|
self.compression_executor_calls = 0
|
|
self.compression_executor_timeouts: list[float] = []
|
|
|
|
async def _next_request_id(self) -> str:
|
|
return "req-lifecycle-test"
|
|
|
|
async def _run_compression_in_executor(self, fn, *, timeout: float):
|
|
self.compression_executor_calls += 1
|
|
self.compression_executor_timeouts.append(timeout)
|
|
return fn()
|
|
|
|
async def _record_request_outcome(self, outcome) -> None:
|
|
# Mirror of ``HeadroomProxy._record_request_outcome`` for the
|
|
# mixin tests. Delegates to the free funnel function so the
|
|
# wire shape is identical to production.
|
|
from headroom.proxy.outcome import emit_request_outcome
|
|
|
|
await emit_request_outcome(self, outcome)
|
|
|
|
|
|
class _FakeWebSocketDisconnect(Exception):
|
|
"""Mirrors the ``WebSocketDisconnect`` type-name check in the handler.
|
|
|
|
The production code identifies "normal client gone" by
|
|
``"WebSocketDisconnect" in type(e).__name__`` — so the fake exception
|
|
type name must start with ``WebSocketDisconnect``.
|
|
"""
|
|
|
|
|
|
# Force the type-name substring match in the handler.
|
|
_FakeWebSocketDisconnect.__name__ = "WebSocketDisconnect_Fake"
|
|
|
|
|
|
class _FakeWebSocket:
|
|
"""Scripted client WebSocket that can delay / disconnect mid-stream."""
|
|
|
|
def __init__(
|
|
self,
|
|
frames: list[str] | None = None,
|
|
*,
|
|
disconnect_after_n_sends: int | None = None,
|
|
hold_after_initial: bool = False,
|
|
call_log: list[str] | None = None,
|
|
) -> None:
|
|
self.headers = {"authorization": "Bearer test"}
|
|
self._frames = list(frames or [])
|
|
self._hold_after_initial = hold_after_initial
|
|
self._disconnect_after_n_sends = disconnect_after_n_sends
|
|
self.sent_text: list[str] = []
|
|
self.sent_bytes: list[bytes] = []
|
|
self.accepted_subprotocol: str | None = None
|
|
self.accepted_headers: list[tuple[bytes, bytes]] | None = None
|
|
self.closed = False
|
|
self.close_code: int | None = None
|
|
self._call_log = call_log
|
|
# "client" can trip this event to simulate mid-stream disconnect.
|
|
self._disconnect_event = asyncio.Event()
|
|
self.client = SimpleNamespace(host="127.0.0.1", port=12345)
|
|
|
|
async def accept(self, subprotocol=None, headers=None) -> None:
|
|
self.accepted_subprotocol = subprotocol
|
|
self.accepted_headers = list(headers) if headers is not None else None
|
|
if self._call_log is not None:
|
|
self._call_log.append("accept")
|
|
|
|
async def receive_text(self) -> str:
|
|
if self._frames:
|
|
return self._frames.pop(0)
|
|
if self._hold_after_initial:
|
|
# Wait for simulated client disconnect.
|
|
await self._disconnect_event.wait()
|
|
# Use an exception type whose name starts with ``WebSocketDisconnect``
|
|
# so the handler's ``type(e).__name__`` check classifies this as a
|
|
# normal client exit (not a ``client_error``).
|
|
raise _FakeWebSocketDisconnect("client closed")
|
|
|
|
async def send_text(self, text: str) -> None:
|
|
self.sent_text.append(text)
|
|
if (
|
|
self._disconnect_after_n_sends is not None
|
|
and len(self.sent_text) >= self._disconnect_after_n_sends
|
|
):
|
|
# Trigger the "client gone" signal the next receive_text will see.
|
|
self._disconnect_event.set()
|
|
|
|
async def send_bytes(self, data: bytes) -> None:
|
|
self.sent_bytes.append(data)
|
|
|
|
async def close(self, code: int | None = None, reason: str | None = None) -> None:
|
|
self.closed = True
|
|
self.close_code = code
|
|
|
|
def trigger_disconnect(self) -> None:
|
|
self._disconnect_event.set()
|
|
|
|
|
|
class _FakeHeaders:
|
|
"""Minimal stand-in for websockets' handshake ``Headers``.
|
|
|
|
Exposes both ``raw_items()`` (preferred by the production header
|
|
extractor to survive duplicate names like ``set-cookie``) and
|
|
``items()``.
|
|
"""
|
|
|
|
def __init__(self, pairs) -> None:
|
|
if isinstance(pairs, dict):
|
|
pairs = list(pairs.items())
|
|
self._pairs = [(str(k), str(v)) for k, v in pairs]
|
|
|
|
def raw_items(self):
|
|
return list(self._pairs)
|
|
|
|
def items(self):
|
|
return list(self._pairs)
|
|
|
|
|
|
class _FakeUpstream:
|
|
"""Upstream that streams scripted events then optionally blocks.
|
|
|
|
``hold_after_events`` makes the async iterator wait forever after the
|
|
scripted events are exhausted — that mirrors a real upstream that
|
|
keeps the connection open after a ``response.completed`` event. The
|
|
handler's ``_upstream_to_client`` will block on it, so the only way
|
|
the outer ``asyncio.wait`` can progress is via the client-side task
|
|
completing — which is exactly the cancel-partner path we want to
|
|
test.
|
|
"""
|
|
|
|
def __init__(
|
|
self,
|
|
events: list[str],
|
|
*,
|
|
hold_after_events: bool = False,
|
|
raise_mid_stream: Exception | None = None,
|
|
response_headers=None,
|
|
) -> None:
|
|
self._events = list(events)
|
|
self._hold_after_events = hold_after_events
|
|
self._raise_mid_stream = raise_mid_stream
|
|
self.sent: list[str] = []
|
|
self.closed = False
|
|
# Mirror websockets' ClientConnection.response.headers, which is the
|
|
# only place OpenAI delivers the Codex x-codex-* subscription window.
|
|
self.response = SimpleNamespace(headers=_FakeHeaders(response_headers or []))
|
|
|
|
async def __aenter__(self) -> _FakeUpstream:
|
|
return self
|
|
|
|
async def __aexit__(self, exc_type, exc, tb) -> None:
|
|
self.closed = True
|
|
|
|
async def send(self, payload: str) -> None:
|
|
self.sent.append(payload)
|
|
|
|
async def close(self) -> None:
|
|
self.closed = True
|
|
|
|
def __aiter__(self):
|
|
return self._iter()
|
|
|
|
async def _iter(self):
|
|
for ev in self._events:
|
|
yield ev
|
|
if self._raise_mid_stream is not None:
|
|
raise self._raise_mid_stream
|
|
if self._hold_after_events:
|
|
# Wait forever — until the task is cancelled by the handler.
|
|
await asyncio.Event().wait()
|
|
|
|
|
|
def _make_fake_websockets_module(
|
|
upstream: _FakeUpstream | None,
|
|
*,
|
|
call_log: list[str] | None = None,
|
|
connect_error: Exception | None = None,
|
|
):
|
|
"""Build a fake ``websockets`` module.
|
|
|
|
Production now does ``upstream = await websockets.connect(...)`` (then
|
|
``async with upstream``), so ``connect`` must return an awaitable that
|
|
resolves to the connection. ``connect_error`` makes the await raise to
|
|
simulate an upstream handshake failure.
|
|
"""
|
|
module = MagicMock()
|
|
|
|
async def _connect(*args, **kwargs):
|
|
if call_log is not None:
|
|
call_log.append("connect")
|
|
if connect_error is not None:
|
|
raise connect_error
|
|
return upstream
|
|
|
|
module.connect = _connect
|
|
module.Subprotocol = str
|
|
return module
|
|
|
|
|
|
def _first_frame() -> str:
|
|
return json.dumps(
|
|
{
|
|
"type": "response.create",
|
|
"response": {"model": "gpt-5.4", "input": "hi"},
|
|
}
|
|
)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Tests
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_ws_first_frame_compression_uses_bounded_executor():
|
|
"""Codex WS compression must not run synchronously on the event loop."""
|
|
upstream_events = [
|
|
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
|
|
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
|
|
]
|
|
upstream = _FakeUpstream(upstream_events)
|
|
fake_ws_mod = _make_fake_websockets_module(upstream)
|
|
|
|
client_ws = _FakeWebSocket(frames=[_first_frame()])
|
|
handler = _DummyOpenAIHandler()
|
|
handler.config.optimize = True
|
|
handler._compress_openai_responses_payload = MagicMock(
|
|
return_value=(
|
|
{"model": "gpt-5.4", "input": "hi"},
|
|
False,
|
|
0,
|
|
[],
|
|
"router_no_compression",
|
|
10,
|
|
10,
|
|
)
|
|
)
|
|
|
|
with patch.dict(sys.modules, {"websockets": fake_ws_mod}):
|
|
await handler.handle_openai_responses_ws(client_ws)
|
|
|
|
assert handler.compression_executor_calls == 1
|
|
assert handler.compression_executor_timeouts == [COMPRESSION_TIMEOUT_SECONDS]
|
|
handler._compress_openai_responses_payload.assert_called_once()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_happy_path_registry_empty_after_response_completed():
|
|
"""Normal session completes — both relay tasks done, registry empty."""
|
|
upstream_events = [
|
|
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
|
|
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
|
|
]
|
|
upstream = _FakeUpstream(upstream_events)
|
|
fake_ws_mod = _make_fake_websockets_module(upstream)
|
|
|
|
client_ws = _FakeWebSocket(frames=[_first_frame()])
|
|
handler = _DummyOpenAIHandler()
|
|
|
|
with patch.dict(sys.modules, {"websockets": fake_ws_mod}):
|
|
await handler.handle_openai_responses_ws(client_ws)
|
|
|
|
assert handler.ws_sessions.active_count() == 0
|
|
assert handler.metrics.active_ws_sessions == 0
|
|
# termination_cause captured
|
|
assert handler.metrics.termination_causes
|
|
# Either "response_completed" or "client_disconnect" — both are
|
|
# acceptable here depending on which relay half exited first; the
|
|
# important thing is we recorded one.
|
|
assert handler.metrics.termination_causes[-1] in {
|
|
"response_completed",
|
|
"client_disconnect",
|
|
"upstream_disconnect",
|
|
}
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_ws_session_metrics_include_response_completed_usage():
|
|
"""Codex WS sessions should report real upstream usage, not zero-token sessions."""
|
|
|
|
upstream_events = [
|
|
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
|
|
json.dumps(
|
|
{
|
|
"type": "response.completed",
|
|
"response": {
|
|
"id": "r_1",
|
|
"usage": {
|
|
"input_tokens": 100,
|
|
"input_tokens_details": {"cached_tokens": 75},
|
|
"output_tokens": 12,
|
|
},
|
|
},
|
|
}
|
|
),
|
|
]
|
|
upstream = _FakeUpstream(upstream_events)
|
|
fake_ws_mod = _make_fake_websockets_module(upstream)
|
|
|
|
client_ws = _FakeWebSocket(frames=[_first_frame()])
|
|
handler = _DummyOpenAIHandler()
|
|
|
|
with patch.dict(sys.modules, {"websockets": fake_ws_mod}):
|
|
await handler.handle_openai_responses_ws(client_ws)
|
|
|
|
assert handler.metrics.recorded_requests
|
|
recorded = handler.metrics.recorded_requests[-1]
|
|
assert recorded["input_tokens"] == 100
|
|
assert recorded["output_tokens"] == 12
|
|
assert recorded["cache_read_tokens"] == 75
|
|
assert recorded["cache_write_tokens"] == 25
|
|
assert recorded["uncached_input_tokens"] == 25
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_ws_session_metrics_include_dashboard_performance_timings():
|
|
"""Codex WS response metrics should feed the dashboard Performance tab."""
|
|
|
|
upstream_events = [
|
|
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
|
|
json.dumps(
|
|
{
|
|
"type": "response.completed",
|
|
"response": {
|
|
"id": "r_1",
|
|
"usage": {
|
|
"input_tokens": 100,
|
|
"input_tokens_details": {"cached_tokens": 75},
|
|
"output_tokens": 12,
|
|
},
|
|
},
|
|
}
|
|
),
|
|
]
|
|
upstream = _FakeUpstream(upstream_events)
|
|
fake_ws_mod = _make_fake_websockets_module(upstream)
|
|
|
|
client_ws = _FakeWebSocket(frames=[_first_frame()])
|
|
handler = _DummyOpenAIHandler()
|
|
handler.config.optimize = True
|
|
|
|
def _noop_compress(payload, *, model, request_id, timing=None):
|
|
if timing is not None:
|
|
timing["compression_live_unit_extraction"] = 2.0
|
|
timing["compression_unit_router_strategy_passthrough"] = 3.0
|
|
return payload, False, 0, [], "test_noop", 10, 10, 0
|
|
|
|
handler._compress_openai_responses_payload = _noop_compress # type: ignore[method-assign]
|
|
|
|
with patch.dict(sys.modules, {"websockets": fake_ws_mod}):
|
|
await handler.handle_openai_responses_ws(client_ws)
|
|
|
|
assert handler.metrics.recorded_requests
|
|
recorded = handler.metrics.recorded_requests[-1]
|
|
assert recorded["overhead_ms"] > 0
|
|
assert recorded["ttfb_ms"] > 0
|
|
assert recorded["pipeline_timing"]["codex_ws.compression"] > 0
|
|
assert recorded["pipeline_timing"]["codex_ws.upstream_first_event"] > 0
|
|
assert recorded["pipeline_timing"]["codex_ws.compression_preflight_serialization"] > 0
|
|
assert recorded["pipeline_timing"]["codex_ws.compression_executor_wait_run"] > 0
|
|
assert recorded["pipeline_timing"]["codex_ws.compression_live_unit_extraction"] == 2.0
|
|
assert (
|
|
recorded["pipeline_timing"]["codex_ws.compression_unit_router_strategy_passthrough"] == 3.0
|
|
)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_client_disconnect_cancels_upstream_relay_within_100ms():
|
|
"""**Failing-test-first** scenario from the plan.
|
|
|
|
When the client side exits (``receive_text`` raises
|
|
``WebSocketDisconnect``) while upstream is still open and iterating,
|
|
the upstream relay task must be cancelled and become ``done()``
|
|
quickly. The registry must report no active sessions afterwards.
|
|
"""
|
|
# Upstream keeps iterating forever after one event, forcing the
|
|
# upstream-to-client task to block on the iterator. The only way
|
|
# out is a cancel from the handler's orchestration.
|
|
upstream_events = [
|
|
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
|
|
]
|
|
upstream = _FakeUpstream(upstream_events, hold_after_events=True)
|
|
fake_ws_mod = _make_fake_websockets_module(upstream)
|
|
|
|
# Client has one initial frame, then disconnects after the server
|
|
# sends the first forwarded event to us.
|
|
client_ws = _FakeWebSocket(
|
|
frames=[_first_frame()],
|
|
hold_after_initial=True,
|
|
)
|
|
handler = _DummyOpenAIHandler()
|
|
|
|
# Trigger disconnect shortly after the handler accepts.
|
|
async def _trigger() -> None:
|
|
await asyncio.sleep(0.05)
|
|
client_ws.trigger_disconnect()
|
|
|
|
with patch.dict(sys.modules, {"websockets": fake_ws_mod}):
|
|
trigger_task = asyncio.create_task(_trigger())
|
|
try:
|
|
await asyncio.wait_for(
|
|
handler.handle_openai_responses_ws(client_ws),
|
|
timeout=2.0,
|
|
)
|
|
finally:
|
|
trigger_task.cancel()
|
|
try:
|
|
await trigger_task
|
|
except asyncio.CancelledError:
|
|
pass
|
|
|
|
# Registry must be empty — the finally block deregistered the session.
|
|
assert handler.ws_sessions.active_count() == 0, (
|
|
"session leaked — deregister did not run in outermost finally"
|
|
)
|
|
assert handler.metrics.active_ws_sessions == 0
|
|
# We recorded a session duration (came through deregister path).
|
|
assert handler.metrics.ws_session_durations, (
|
|
"record_ws_session_duration never fired — deregister path broken"
|
|
)
|
|
# And we tagged the cause. For a client-side exit it should be one
|
|
# of: client_disconnect, client_error, upstream_disconnect (if
|
|
# upstream iteration happened to end first in a race).
|
|
cause = handler.metrics.termination_causes[-1]
|
|
assert cause in {
|
|
"client_disconnect",
|
|
"client_error",
|
|
"upstream_disconnect",
|
|
}, f"unexpected cause: {cause}"
|
|
|
|
# No codex-ws-* named task should still be running.
|
|
leaked = [
|
|
t
|
|
for t in asyncio.all_tasks()
|
|
if (t.get_name() or "").startswith("codex-ws-") and not t.done()
|
|
]
|
|
assert leaked == [], f"relay tasks leaked: {[t.get_name() for t in leaked]}"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_upstream_closes_first_cancels_client_task():
|
|
"""Upstream iterator ends naturally; client task should be cancelled.
|
|
|
|
The client is set to block on ``receive_text`` indefinitely; only a
|
|
cancel from the handler's orchestration releases it.
|
|
"""
|
|
upstream_events = [
|
|
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
|
|
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
|
|
]
|
|
upstream = _FakeUpstream(upstream_events, hold_after_events=False)
|
|
fake_ws_mod = _make_fake_websockets_module(upstream)
|
|
|
|
client_ws = _FakeWebSocket(
|
|
frames=[_first_frame()],
|
|
hold_after_initial=True,
|
|
)
|
|
handler = _DummyOpenAIHandler()
|
|
|
|
with patch.dict(sys.modules, {"websockets": fake_ws_mod}):
|
|
await asyncio.wait_for(
|
|
handler.handle_openai_responses_ws(client_ws),
|
|
timeout=2.0,
|
|
)
|
|
|
|
assert handler.ws_sessions.active_count() == 0
|
|
# We must still have recorded exactly one session duration.
|
|
assert len(handler.metrics.ws_session_durations) == 1
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_upstream_error_mid_stream_classifies_as_upstream_error():
|
|
upstream_events = [
|
|
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
|
|
]
|
|
upstream = _FakeUpstream(
|
|
upstream_events,
|
|
raise_mid_stream=RuntimeError("boom from upstream"),
|
|
)
|
|
fake_ws_mod = _make_fake_websockets_module(upstream)
|
|
|
|
client_ws = _FakeWebSocket(
|
|
frames=[_first_frame()],
|
|
hold_after_initial=True,
|
|
)
|
|
handler = _DummyOpenAIHandler()
|
|
|
|
with patch.dict(sys.modules, {"websockets": fake_ws_mod}):
|
|
await asyncio.wait_for(
|
|
handler.handle_openai_responses_ws(client_ws),
|
|
timeout=2.0,
|
|
)
|
|
|
|
assert handler.ws_sessions.active_count() == 0
|
|
assert handler.metrics.termination_causes
|
|
assert handler.metrics.termination_causes[-1] == "upstream_error"
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_response_cancel_frame_is_logged_as_client_cancel_lifecycle():
|
|
"""A Codex Ctrl-C maps to response.cancel on the WS stream.
|
|
|
|
The proxy should relay it upstream and classify the lifecycle as a
|
|
client-side cancel when no response.completed event follows.
|
|
"""
|
|
cancel_frame = json.dumps({"type": "response.cancel", "response_id": "r_1"})
|
|
upstream = _FakeUpstream([], hold_after_events=True)
|
|
fake_ws_mod = _make_fake_websockets_module(upstream)
|
|
|
|
client_ws = _FakeWebSocket(
|
|
frames=[_first_frame(), cancel_frame],
|
|
hold_after_initial=True,
|
|
)
|
|
handler = _DummyOpenAIHandler()
|
|
|
|
async def _trigger() -> None:
|
|
await asyncio.sleep(0.05)
|
|
client_ws.trigger_disconnect()
|
|
|
|
with patch.dict(sys.modules, {"websockets": fake_ws_mod}):
|
|
trigger_task = asyncio.create_task(_trigger())
|
|
try:
|
|
await asyncio.wait_for(
|
|
handler.handle_openai_responses_ws(client_ws),
|
|
timeout=2.0,
|
|
)
|
|
finally:
|
|
trigger_task.cancel()
|
|
try:
|
|
await trigger_task
|
|
except asyncio.CancelledError:
|
|
pass
|
|
|
|
assert cancel_frame in upstream.sent
|
|
assert handler.metrics.termination_causes[-1] == "client_cancel"
|
|
assert handler.ws_sessions.active_count() == 0
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_upstream_connect_failure_still_deregisters_cleanly():
|
|
"""Handshake-phase leak must be impossible: if upstream connect
|
|
raises before relay tasks are created, the session is still
|
|
registered+deregistered cleanly (or never registered). Either way,
|
|
no leak.
|
|
"""
|
|
fake_ws_mod = _make_fake_websockets_module(None, connect_error=RuntimeError("upstream refused"))
|
|
|
|
client_ws = _FakeWebSocket(frames=[_first_frame()])
|
|
handler = _DummyOpenAIHandler()
|
|
|
|
async def _fallback(*args, **kwargs):
|
|
return None
|
|
|
|
handler._ws_http_fallback = _fallback # type: ignore[assignment]
|
|
|
|
with patch.dict(sys.modules, {"websockets": fake_ws_mod}):
|
|
await handler.handle_openai_responses_ws(client_ws)
|
|
|
|
assert handler.ws_sessions.active_count() == 0
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_ws_connect_failure_falls_back_to_http():
|
|
"""When every upstream connect attempt fails, the client is still
|
|
accepted (with no x-codex-* headers, since there is no upstream
|
|
window) and the request is served via the HTTP POST fallback with
|
|
the client's first frame. Preserves the pre-reorder WS-upgrade-
|
|
failure behaviour.
|
|
"""
|
|
fake_ws_mod = _make_fake_websockets_module(
|
|
None, connect_error=RuntimeError("HTTP 500 from upstream")
|
|
)
|
|
|
|
first = _first_frame()
|
|
client_ws = _FakeWebSocket(frames=[first])
|
|
handler = _DummyOpenAIHandler()
|
|
|
|
fallback_calls: list[tuple] = []
|
|
|
|
async def _fallback(websocket, body, first_msg_raw, upstream_headers, request_id):
|
|
fallback_calls.append((body, first_msg_raw))
|
|
|
|
handler._ws_http_fallback = _fallback # type: ignore[assignment]
|
|
|
|
with patch.dict(sys.modules, {"websockets": fake_ws_mod}):
|
|
await handler.handle_openai_responses_ws(client_ws)
|
|
|
|
# Client was accepted with no upstream window to forward.
|
|
assert client_ws.accepted_headers is None
|
|
# Fallback ran with the first frame.
|
|
assert len(fallback_calls) == 1
|
|
_body, _first_raw = fallback_calls[0]
|
|
assert _first_raw == first
|
|
assert _body == json.loads(first)
|
|
# Clean teardown.
|
|
assert handler.ws_sessions.active_count() == 0
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_ws_connect_happens_before_accept():
|
|
"""The upstream connect must complete before the client 101 is sent,
|
|
so OpenAI's x-codex-* handshake headers are available to attach.
|
|
"""
|
|
upstream_events = [
|
|
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
|
|
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
|
|
]
|
|
call_log: list[str] = []
|
|
upstream = _FakeUpstream(upstream_events)
|
|
fake_ws_mod = _make_fake_websockets_module(upstream, call_log=call_log)
|
|
|
|
client_ws = _FakeWebSocket(frames=[_first_frame()], call_log=call_log)
|
|
handler = _DummyOpenAIHandler()
|
|
|
|
with patch.dict(sys.modules, {"websockets": fake_ws_mod}):
|
|
await handler.handle_openai_responses_ws(client_ws)
|
|
|
|
assert "connect" in call_log and "accept" in call_log
|
|
assert call_log.index("connect") < call_log.index("accept"), (
|
|
f"connect must precede accept, got {call_log}"
|
|
)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_ws_forwards_codex_headers_to_client_accept():
|
|
"""OpenAI's x-codex-* subscription window from the upstream WS
|
|
handshake must be forwarded onto the client-facing 101 (and only
|
|
that subset — never set-cookie/authorization), and Python /stats
|
|
state must be refreshed.
|
|
"""
|
|
upstream_events = [
|
|
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
|
|
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
|
|
]
|
|
# Include duplicate set-cookie to ensure raw_items() is used (a plain
|
|
# dict-style .items() on real websockets Headers raises on dupes).
|
|
handshake_headers = [
|
|
("x-codex-primary-used-percent", "42"),
|
|
("X-Codex-Primary-Window-Minutes", "300"),
|
|
("set-cookie", "a=1"),
|
|
("set-cookie", "b=2"),
|
|
("authorization", "Bearer leak"),
|
|
]
|
|
upstream = _FakeUpstream(upstream_events, response_headers=handshake_headers)
|
|
fake_ws_mod = _make_fake_websockets_module(upstream)
|
|
|
|
client_ws = _FakeWebSocket(frames=[_first_frame()])
|
|
handler = _DummyOpenAIHandler()
|
|
|
|
captured: dict = {}
|
|
|
|
def _fake_state():
|
|
class _S:
|
|
def update_from_headers(self, headers):
|
|
captured.update(headers)
|
|
|
|
return _S()
|
|
|
|
with (
|
|
patch.dict(sys.modules, {"websockets": fake_ws_mod}),
|
|
patch(
|
|
"headroom.subscription.codex_rate_limits.get_codex_rate_limit_state",
|
|
_fake_state,
|
|
),
|
|
):
|
|
await handler.handle_openai_responses_ws(client_ws)
|
|
|
|
assert client_ws.accepted_headers is not None
|
|
names = {name.decode("latin-1").lower() for name, _ in client_ws.accepted_headers}
|
|
assert names == {"x-codex-primary-used-percent", "x-codex-primary-window-minutes"}
|
|
assert "set-cookie" not in names
|
|
assert "authorization" not in names
|
|
# Original-case names preserved on the wire.
|
|
sent = {name.decode("latin-1") for name, _ in client_ws.accepted_headers}
|
|
assert "X-Codex-Primary-Window-Minutes" in sent
|
|
# Python /stats state refreshed with the same x-codex-* subset.
|
|
assert captured == {
|
|
"x-codex-primary-used-percent": "42",
|
|
"X-Codex-Primary-Window-Minutes": "300",
|
|
}
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_ws_first_frame_timeout_after_connect_closes_upstream():
|
|
"""If the client never sends its first frame after we connected, the
|
|
upstream WS must be closed (no leak) and the session deregistered.
|
|
"""
|
|
upstream = _FakeUpstream([], hold_after_events=True)
|
|
fake_ws_mod = _make_fake_websockets_module(upstream)
|
|
|
|
# No frames + hold => receive_text blocks until disconnect; we force a
|
|
# short first-frame timeout so the handler hits the timeout branch.
|
|
client_ws = _FakeWebSocket(frames=[], hold_after_initial=True)
|
|
handler = _DummyOpenAIHandler()
|
|
|
|
with (
|
|
patch.dict(sys.modules, {"websockets": fake_ws_mod}),
|
|
patch(
|
|
"headroom.proxy.handlers.openai.WS_FIRST_FRAME_TIMEOUT_SECONDS",
|
|
0.05,
|
|
),
|
|
):
|
|
await asyncio.wait_for(
|
|
handler.handle_openai_responses_ws(client_ws),
|
|
timeout=2.0,
|
|
)
|
|
|
|
assert upstream.closed, "upstream not closed on first-frame timeout"
|
|
assert client_ws.closed and client_ws.close_code == 1001
|
|
assert handler.ws_sessions.active_count() == 0
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_many_concurrent_sessions_cleanly_drained():
|
|
"""50 concurrent sessions: all drain; registry and named tasks go to 0."""
|
|
upstream_events = [
|
|
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
|
|
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
|
|
]
|
|
|
|
async def run_one() -> None:
|
|
upstream = _FakeUpstream(list(upstream_events))
|
|
fake_ws_mod = _make_fake_websockets_module(upstream)
|
|
client_ws = _FakeWebSocket(frames=[_first_frame()])
|
|
handler = _DummyOpenAIHandler()
|
|
with patch.dict(sys.modules, {"websockets": fake_ws_mod}):
|
|
await handler.handle_openai_responses_ws(client_ws)
|
|
assert handler.ws_sessions.active_count() == 0
|
|
|
|
await asyncio.gather(*[run_one() for _ in range(50)])
|
|
|
|
# Global check: no codex-ws-* named task remains.
|
|
leaked = [
|
|
t
|
|
for t in asyncio.all_tasks()
|
|
if (t.get_name() or "").startswith("codex-ws-") and not t.done()
|
|
]
|
|
assert leaked == []
|