headroom/tests/test_openai_codex_ws_lifecycle.py
gglucass 0ce68dedd7
fix(proxy): restore Codex usage headers on WS and streaming SSE transports (#577) (#794)
## Description

Codex's subscription/rate-limit window (the `x-codex-*` headers) was
being
**stripped on every transport Codex actually uses**, so session/weekly
usage
never reached the Codex CLI's own `/status` display, Headroom
`/stats`/dashboard,
or any consumer that sniffs the client-facing handshake. This PR
restores it on
**both** the WebSocket and streaming-SSE paths — the two halves of #577
— in one
place.

Fixes #577

**Supersedes #582 and #590.** This PR incorporates #582's SSE fix
(carried verbatim
with a `Co-authored-by` trailer) and additionally forwards the window
onto the client
`101` on the WS path, which #582/#590's capture-only WS code cannot do.
Both can be
closed as superseded once this merges — GitHub closing keywords only
auto-close
issues (hence `Fixes #577` above), not PRs, so #582/#590 need a manual
close.

### WebSocket (`gpt-5.4+`)

OpenAI delivers `x-codex-*` **only** on the upstream WS handshake
response, never
in data frames. `handle_openai_responses_ws` accepted the client WS
*before* it
connected upstream and never read `upstream.response.headers`, so the
window was
dropped. This reorders the handler to **connect upstream first**,
extract the
`x-codex-*` subset, then **accept the client WS with those headers
attached** to
the `101`, and refresh the Python state for `/stats` parity.

### Streaming SSE (incorporated from #582, @m16khb)

Codex CLI almost always streams. `streaming.py` neither captured
`x-codex-*` into
`CodexRateLimitState` nor forwarded it — the forwarded-header filter
matched only
the substring `"ratelimit"`, which `x-codex-*` does not contain. This
calls
`update_from_headers()` **before** the `>=400` early-return (so a
streaming 429/5xx
still refreshes the window, matching the non-streaming handlers) and
widens the
forward filter to pass `x-codex-*`.

> Credit: the SSE fix is @m16khb's work from #582, carried here verbatim
with a
> `Co-authored-by` trailer so the maintainer gets a single PR covering
both
> transports. This supersedes #582/#590's **WS** capture (which only
writes
> `/stats`); the connect-before-accept reorder additionally forwards the
window to
> the client `101`, which capture-only cannot do. #590's optional
snapshot
> persistence is intentionally left out (separable; hot-path sync write;
doesn't
> help the `101`-sniff consumers).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `openai.py`: add `_extract_codex_handshake_headers()` (strictly
`x-codex-*`, via
`raw_items()` to avoid `MultipleValuesError`; never
`set-cookie`/`authorization`).
- `openai.py`: reorder `handle_openai_responses_ws` — connect-only retry
loop runs
before `accept()`; `accept(headers=...)` carries the forwarded window;
first
client frame read afterward. HTTP fallback preserved; it now also
refreshes
  `/stats` from the HTTP response headers.
- `streaming.py`: capture `x-codex-*` on all statuses + widen the
forwarded-header
  filter (from #582).

### Diff-size note

The bulk of the `openai.py` line count is **whitespace-only
relocation**: the relay
block dedents one level out of the old per-attempt `async with`. Logical
change is
~290 lines. **Review with `?w=1`.** In API-key mode the handshake
carries no
`x-codex-*`, so the accept-header list is empty and the path behaves
exactly as
before — the fix only activates for ChatGPT-subscription auth.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

- WS: `test_ws_connect_happens_before_accept`,
`test_ws_forwards_codex_headers_to_client_accept`
(only `x-codex-*` forwarded; `set-cookie`/`authorization` excluded;
`/stats` refreshed),
`test_ws_connect_failure_falls_back_to_http`,
`test_ws_first_frame_timeout_after_connect_closes_upstream`.
- Fallback: `test_fallback_refreshes_codex_rate_limit_state`.
- SSE:
`test_codex_rate_limit_headers_captured_and_forwarded_in_streaming`,
  `test_codex_rate_limit_captured_on_streaming_429` (from #582).
- Wire-level e2e: `tests/e2e_ws_codex_usage_headers.py` boots the real
proxy + fake
upstream + real `websockets` client and reads the client `101` — closes
the gap
the unit tests stub (that uvicorn/starlette actually write
`accept(headers=...)`).

## Test Output

```
$ uv run pytest tests/test_proxy_streaming_ratelimit_headers.py \
                tests/test_ws_http_fallback.py \
                tests/test_openai_codex_ws_lifecycle.py \
                tests/test_openai_codex_ws_timings.py \
                tests/test_codex_rate_limits.py -q
63 passed in 0.83s

$ .venv/bin/python tests/e2e_ws_codex_usage_headers.py
[codex-hdr-e2e] client 101 headers:
    x-codex-primary-used-percent: 42
    x-codex-primary-window-minutes: 300
    x-codex-secondary-used-percent: 7
    x-codex-secondary-window-minutes: 10080
[codex-hdr-e2e] /stats reflects codex window (primary-used=42)
=== CODEX-HDR E2E ALL GREEN ===

$ uv run ruff check . && uv run ruff format --check <touched files>
All checks passed!
```

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

- **Why connect-before-accept (not capture-only).** Once `accept()`
sends the `101`,
headers can no longer be added; the `x-codex-*` window only exists after
we connect
upstream. Capturing into Python state (as #582/#590's WS code does)
fixes `/stats`
but not the Codex CLI's native display or any `101`-sniffing consumer —
those need
  the headers *on the client handshake*, which requires the reorder.
- **Security.** Forwarding is filtered strictly to `x-codex-*`;
`set-cookie`,
`authorization`, and all other upstream headers are never forwarded to
the client
  (asserted by both the unit test and the e2e).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

## Contract Schemas

Per maintainer request: a JSON Schema (draft 2020-12) artifact
enshrining the OpenAI
interaction expectations this changeset relies on, so drift is
detectable later.

Committed following the repo's parity convention:
- schema:
`tests/parity/fixtures/codex_openai_contracts/codex-openai-interaction.schema.json`
- test: `tests/test_codex_openai_contract_parity.py` binds the schema to
the **live code**
in both directions, so drift fails CI rather than living only in this
description -
every declared `x-codex-*` header must be consumed by
`parse_codex_rate_limits`, and
`_extract_codex_handshake_headers` must forward exactly the declared
subset and never
`set-cookie`/`authorization`. No new dependency (does not pull in
`jsonschema`).

It covers, as `$defs`:

- `WSUpstreamHandshakeResponse` / `StreamingUpstreamResponseHeaders` -
the upstream
`x-codex-*` header family (full superset, with per-header wire pattern +
the parsed
semantic type) the WS and SSE captures read. Source of truth:
`parse_codex_rate_limits`.
- `ClientForwardedHandshakeHeaders` - the WS-101 **allow/deny**
contract: only
`x-codex-*` may be forwarded; `set-cookie`/`authorization` are
explicitly forbidden
  (`propertyNames` + `not`).
- `ClientForwardedStreamingHeaders` - the wider SSE forward set
(`*ratelimit*` OR `x-codex*`).
- `WSClientRequestFrame` / `WSRelayEvent` / `HTTPFallbackRequestBody` -
the WS frame
  envelopes and the unwrapped HTTP-fallback POST body.
- `CodexRateLimitStatsOutput` - the headroom `/stats` shape the parity
tests assert.

Validated with `jsonschema` (Draft202012 `check_schema` passes; positive
instances from
the e2e validate; negative instances - a leaked `set-cookie`, a fallback
body still
carrying a top-level `type` - are correctly rejected).

<details>
<summary><code>codex-openai-interaction.schema.json</code> (draft
2020-12)</summary>

```json
{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://github.com/chopratejas/headroom/contracts/codex-openai-interaction.schema.json",
  "title": "Codex <-> OpenAI interaction contracts (PR #794)",
  "description": "Enshrines the OpenAI interaction expectations this changeset depends on, so drift is detectable. Header values are transported as strings on the wire; the `x-headroom-parsed-type` annotation on each records the semantic type the parser (headroom/subscription/codex_rate_limits.py) coerces them to. Sources: codex_rate_limits.parse_codex_rate_limits (header family + gating), openai._extract_codex_handshake_headers (WS-101 forward filter), streaming.py (SSE forward filter).",
  "$defs": {
    "OpenAICodexWindowHeaders": {
      "title": "x-codex-*-{primary,secondary} window headers",
      "description": "A rolling rate-limit/subscription window. A window is materialized iff its `*-used-percent` header is present and numeric; `*-window-minutes` and `*-reset-at` are optional. `primary` and `secondary` are independent and either may be absent.",
      "type": "object",
      "properties": {
        "x-codex-primary-used-percent": {
          "type": "string",
          "pattern": "^\\d+(?:\\.\\d+)?$",
          "x-headroom-parsed-type": "float (0-100, NaN-guarded)",
          "description": "Percent of the primary window consumed. Gates creation of the primary window."
        },
        "x-codex-primary-window-minutes": {
          "type": "string",
          "pattern": "^\\d+$",
          "x-headroom-parsed-type": "int",
          "description": "Primary window size in minutes."
        },
        "x-codex-primary-reset-at": {
          "type": "string",
          "pattern": "^\\d+$",
          "x-headroom-parsed-type": "int (Unix epoch seconds)",
          "description": "Absolute reset time of the primary window."
        },
        "x-codex-secondary-used-percent": {
          "type": "string",
          "pattern": "^\\d+(?:\\.\\d+)?$",
          "x-headroom-parsed-type": "float (0-100, NaN-guarded)",
          "description": "Percent of the secondary window consumed. Gates creation of the secondary window."
        },
        "x-codex-secondary-window-minutes": {
          "type": "string",
          "pattern": "^\\d+$",
          "x-headroom-parsed-type": "int"
        },
        "x-codex-secondary-reset-at": {
          "type": "string",
          "pattern": "^\\d+$",
          "x-headroom-parsed-type": "int (Unix epoch seconds)"
        }
      },
      "additionalProperties": true
    },
    "OpenAICodexCreditsHeaders": {
      "title": "x-codex-credits-* headers",
      "description": "OpenAI credits balance. A credits snapshot is materialized iff `x-codex-credits-has-credits` is present; `unlimited` defaults to false; `balance` is optional.",
      "type": "object",
      "properties": {
        "x-codex-credits-has-credits": {
          "type": "string",
          "pattern": "^(?:[Tt][Rr][Uu][Ee]|[Ff][Aa][Ll][Ss][Ee]|[01])$",
          "x-headroom-parsed-type": "bool (true|false|1|0, case-insensitive)",
          "description": "Gates creation of the credits snapshot."
        },
        "x-codex-credits-unlimited": {
          "type": "string",
          "pattern": "^(?:[Tt][Rr][Uu][Ee]|[Ff][Aa][Ll][Ss][Ee]|[01])$",
          "x-headroom-parsed-type": "bool (defaults false when absent/unparseable)"
        },
        "x-codex-credits-balance": {
          "type": "string",
          "x-headroom-parsed-type": "str (empty -> null)",
          "description": "Free-form server string, e.g. \"$5.00\"."
        }
      },
      "additionalProperties": true
    },
    "OpenAICodexMetaHeaders": {
      "title": "x-codex meta headers",
      "type": "object",
      "properties": {
        "x-codex-limit-name": {
          "type": "string",
          "x-headroom-parsed-type": "str (empty -> null)",
          "description": "Active limit/model label, e.g. \"gpt-5.2-codex-sonic\"."
        },
        "x-codex-promo-message": {
          "type": "string",
          "x-headroom-parsed-type": "str (empty -> null)",
          "description": "Server announcement. Also gates snapshot creation when present."
        }
      },
      "additionalProperties": true
    },
    "OpenAICodexRateLimitHeaders": {
      "title": "Full x-codex-* header family OpenAI may emit",
      "description": "Superset of every x-codex-* header headroom reads. parse_codex_rate_limits returns a snapshot iff at least one of: a primary window, a secondary window, a credits snapshot, or a non-empty promo message is present; otherwise null (treated as a non-Codex response). All members are individually optional.",
      "type": "object",
      "allOf": [
        { "$ref": "#/$defs/OpenAICodexWindowHeaders" },
        { "$ref": "#/$defs/OpenAICodexCreditsHeaders" },
        { "$ref": "#/$defs/OpenAICodexMetaHeaders" }
      ],
      "additionalProperties": true
    },
    "WSUpstreamHandshakeResponse": {
      "title": "OpenAI WS handshake (101) response headers consumed by the WS fix",
      "description": "On the Codex WebSocket transport the x-codex-* window is delivered ONLY on the upstream handshake response (never in data frames). handle_openai_responses_ws reads upstream.response.headers here. This is the contract the connect-before-accept reorder depends on: if OpenAI ever moves these headers off the handshake (e.g. into a frame), the WS half of the fix goes stale.",
      "$ref": "#/$defs/OpenAICodexRateLimitHeaders"
    },
    "StreamingUpstreamResponseHeaders": {
      "title": "OpenAI streaming/HTTP response headers consumed by the SSE fix",
      "description": "On the streaming SSE/HTTP transport the same x-codex-* headers ride the HTTP response. streaming.py captures them on ALL statuses (including >=400) via update_from_headers, and forwards a wider set to the client (see ClientForwardedStreamingHeaders).",
      "$ref": "#/$defs/OpenAICodexRateLimitHeaders"
    },
    "ClientForwardedHandshakeHeaders": {
      "title": "Headers forwarded onto the CLIENT-facing WS 101 (allow/deny contract)",
      "description": "_extract_codex_handshake_headers forwards ONLY headers whose (lowercased) name starts with `x-codex-`. Every other upstream handshake header - notably set-cookie and authorization - MUST NOT appear on the client 101. Enforced by propertyNames below and asserted by the unit tests + tests/e2e_ws_codex_usage_headers.py.",
      "type": "object",
      "propertyNames": {
        "pattern": "^[Xx]-[Cc][Oo][Dd][Ee][Xx]-"
      },
      "not": {
        "anyOf": [
          { "required": ["set-cookie"] },
          { "required": ["Set-Cookie"] },
          { "required": ["authorization"] },
          { "required": ["Authorization"] }
        ]
      },
      "additionalProperties": { "type": "string" }
    },
    "ClientForwardedStreamingHeaders": {
      "title": "Headers forwarded to the client on the streaming SSE path",
      "description": "streaming.py forwards a header iff `\"ratelimit\" in name.lower()` OR `name.lower().startswith(\"x-codex\")`. This is a SUPERSET of the WS allow-list: it additionally passes generic *ratelimit* headers (e.g. the Anthropic streaming path) which do not contain the x-codex prefix.",
      "type": "object",
      "propertyNames": {
        "pattern": "(?:[Rr][Aa][Tt][Ee][Ll][Ii][Mm][Ii][Tt])|^[Xx]-[Cc][Oo][Dd][Ee][Xx]"
      },
      "additionalProperties": { "type": "string" }
    },
    "WSClientRequestFrame": {
      "title": "Client -> proxy WS data frame (Responses API over WS)",
      "description": "Codex sends the request as a response.create envelope. The HTTP fallback unwraps `.response` for the POST body, forces stream=true, and strips any top-level `type`. A flattened variant (no envelope, fields at top level) is also tolerated by the fallback.",
      "type": "object",
      "properties": {
        "type": { "const": "response.create" },
        "response": {
          "type": "object",
          "properties": {
            "model": { "type": "string", "description": "e.g. gpt-5.4" },
            "input": {
              "description": "String prompt or Responses-API structured input array.",
              "type": ["string", "array"]
            },
            "stream": { "type": "boolean" }
          },
          "required": ["model"],
          "additionalProperties": true
        }
      },
      "required": ["type", "response"],
      "additionalProperties": true
    },
    "WSRelayEvent": {
      "title": "proxy -> client WS data frame (relayed Responses API event)",
      "description": "SSE `data:` payloads relayed verbatim as WS text frames. `[DONE]` sentinels are dropped (not relayed). Every relayed event is a JSON object carrying a `type`. response.completed additionally carries usage under `response.usage`. anyOf (not oneOf): an error event also satisfies the looser lifecycle shape, which is fine.",
      "anyOf": [
        {
          "title": "lifecycle event",
          "type": "object",
          "properties": {
            "type": {
              "type": "string",
              "examples": [
                "response.created",
                "response.output_item.added",
                "response.completed"
              ]
            },
            "response": { "type": "object", "additionalProperties": true }
          },
          "required": ["type"],
          "additionalProperties": true
        },
        {
          "title": "error event",
          "type": "object",
          "properties": {
            "type": { "const": "error" },
            "error": {
              "type": "object",
              "properties": { "message": { "type": "string" } },
              "required": ["message"],
              "additionalProperties": true
            }
          },
          "required": ["type", "error"],
          "additionalProperties": true
        }
      ]
    },
    "HTTPFallbackRequestBody": {
      "title": "proxy -> OpenAI HTTP POST body on WS->HTTP fallback",
      "description": "Derived from WSClientRequestFrame: the inner `.response` object, with `stream` forced to true and any top-level `type` removed.",
      "type": "object",
      "properties": {
        "model": { "type": "string" },
        "stream": { "const": true },
        "input": { "type": ["string", "array"] }
      },
      "required": ["model", "stream"],
      "not": { "required": ["type"] },
      "additionalProperties": true
    },
    "CodexRateLimitStatsOutput": {
      "title": "headroom /stats output for the codex tracker (CodexRateLimitSnapshot.to_dict)",
      "description": "Internal (headroom-emitted) shape produced from the headers above; the WS and SSE update_from_headers parity tests assert this is refreshed. Included so drift in our own surface is also caught.",
      "type": "object",
      "properties": {
        "limit_id": { "const": "codex" },
        "limit_name": { "type": ["string", "null"] },
        "primary": { "$ref": "#/$defs/CodexWindowDict" },
        "secondary": { "$ref": "#/$defs/CodexWindowDict" },
        "credits": {
          "oneOf": [
            { "type": "null" },
            {
              "type": "object",
              "properties": {
                "has_credits": { "type": "boolean" },
                "unlimited": { "type": "boolean" },
                "balance": { "type": ["string", "null"] }
              },
              "required": ["has_credits", "unlimited", "balance"],
              "additionalProperties": false
            }
          ]
        },
        "promo_message": { "type": ["string", "null"] },
        "captured_at": { "type": "number", "description": "Unix epoch seconds (float)." }
      },
      "required": ["limit_id", "limit_name", "primary", "secondary", "credits", "promo_message", "captured_at"],
      "additionalProperties": false
    },
    "CodexWindowDict": {
      "oneOf": [
        { "type": "null" },
        {
          "type": "object",
          "properties": {
            "used_percent": { "type": "number" },
            "window_minutes": { "type": ["integer", "null"] },
            "window_label": { "type": "string", "description": "e.g. \"5h\", \"7d\"-style label; \"unknown\" when window_minutes is null." },
            "resets_at": { "type": ["integer", "null"], "description": "Unix epoch seconds." },
            "seconds_until_reset": { "type": ["integer", "null"] }
          },
          "required": ["used_percent", "window_minutes", "window_label", "resets_at", "seconds_until_reset"],
          "additionalProperties": false
        }
      ]
    }
  }
}
```

</details>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: m16khb <m16khb@gmail.com>
2026-06-09 15:55:53 -05:00

817 lines
29 KiB
Python

"""Unit 3: WebSocket session lifecycle + deterministic relay cancellation.
These tests exercise the Codex WS handler with a fake upstream and a
fake client WebSocket so we can drive the relay halves through their
real code paths (not mocked) and assert on registry / task state.
"""
from __future__ import annotations
import asyncio
import json
import sys
from types import SimpleNamespace
from unittest.mock import MagicMock, patch
import pytest
from headroom.proxy.handlers.openai import OpenAIHandlerMixin
from headroom.proxy.helpers import COMPRESSION_TIMEOUT_SECONDS
from headroom.proxy.ws_session_registry import WebSocketSessionRegistry
# ---------------------------------------------------------------------------
# Test doubles
# ---------------------------------------------------------------------------
class _DummyMetrics:
def __init__(self) -> None:
self.active_ws_sessions = 0
self.active_ws_sessions_max = 0
self.active_relay_tasks = 0
self.ws_session_durations: list[float] = []
self.stage_timings: list[tuple[str, dict[str, float]]] = []
self.termination_causes: list[str] = []
self.recorded_requests: list[dict] = []
async def record_request(self, **kwargs): # pragma: no cover
self.recorded_requests.append(dict(kwargs))
return None
async def record_stage_timings(self, path: str, timings: dict[str, float]) -> None:
self.stage_timings.append((path, dict(timings)))
def inc_active_ws_sessions(self) -> None:
self.active_ws_sessions += 1
self.active_ws_sessions_max = max(self.active_ws_sessions_max, self.active_ws_sessions)
def dec_active_ws_sessions(self) -> None:
self.active_ws_sessions = max(0, self.active_ws_sessions - 1)
def inc_active_relay_tasks(self, n: int = 1) -> None:
self.active_relay_tasks += n
def dec_active_relay_tasks(self, n: int = 1) -> None:
self.active_relay_tasks = max(0, self.active_relay_tasks - n)
def record_ws_session_duration(self, duration_ms: float, cause: str) -> None:
self.ws_session_durations.append(duration_ms)
self.termination_causes.append(cause)
class _DummyOpenAIHandler(OpenAIHandlerMixin):
OPENAI_API_URL = "https://api.openai.com"
def __init__(self, ws_sessions: WebSocketSessionRegistry | None = None) -> None:
self.rate_limiter = None
self.metrics = _DummyMetrics()
self.config = SimpleNamespace(
optimize=False,
retry_max_attempts=1,
retry_base_delay_ms=1,
retry_max_delay_ms=1,
connect_timeout_seconds=10,
)
self.usage_reporter = None
self.openai_provider = SimpleNamespace(get_context_limit=lambda model: 128_000)
self.openai_pipeline = SimpleNamespace(apply=MagicMock())
self.anthropic_backend = None
self.cost_tracker = None
self.memory_handler = None
self.ws_sessions = ws_sessions or WebSocketSessionRegistry()
self.compression_executor_calls = 0
self.compression_executor_timeouts: list[float] = []
async def _next_request_id(self) -> str:
return "req-lifecycle-test"
async def _run_compression_in_executor(self, fn, *, timeout: float):
self.compression_executor_calls += 1
self.compression_executor_timeouts.append(timeout)
return fn()
async def _record_request_outcome(self, outcome) -> None:
# Mirror of ``HeadroomProxy._record_request_outcome`` for the
# mixin tests. Delegates to the free funnel function so the
# wire shape is identical to production.
from headroom.proxy.outcome import emit_request_outcome
await emit_request_outcome(self, outcome)
class _FakeWebSocketDisconnect(Exception):
"""Mirrors the ``WebSocketDisconnect`` type-name check in the handler.
The production code identifies "normal client gone" by
``"WebSocketDisconnect" in type(e).__name__`` — so the fake exception
type name must start with ``WebSocketDisconnect``.
"""
# Force the type-name substring match in the handler.
_FakeWebSocketDisconnect.__name__ = "WebSocketDisconnect_Fake"
class _FakeWebSocket:
"""Scripted client WebSocket that can delay / disconnect mid-stream."""
def __init__(
self,
frames: list[str] | None = None,
*,
disconnect_after_n_sends: int | None = None,
hold_after_initial: bool = False,
call_log: list[str] | None = None,
) -> None:
self.headers = {"authorization": "Bearer test"}
self._frames = list(frames or [])
self._hold_after_initial = hold_after_initial
self._disconnect_after_n_sends = disconnect_after_n_sends
self.sent_text: list[str] = []
self.sent_bytes: list[bytes] = []
self.accepted_subprotocol: str | None = None
self.accepted_headers: list[tuple[bytes, bytes]] | None = None
self.closed = False
self.close_code: int | None = None
self._call_log = call_log
# "client" can trip this event to simulate mid-stream disconnect.
self._disconnect_event = asyncio.Event()
self.client = SimpleNamespace(host="127.0.0.1", port=12345)
async def accept(self, subprotocol=None, headers=None) -> None:
self.accepted_subprotocol = subprotocol
self.accepted_headers = list(headers) if headers is not None else None
if self._call_log is not None:
self._call_log.append("accept")
async def receive_text(self) -> str:
if self._frames:
return self._frames.pop(0)
if self._hold_after_initial:
# Wait for simulated client disconnect.
await self._disconnect_event.wait()
# Use an exception type whose name starts with ``WebSocketDisconnect``
# so the handler's ``type(e).__name__`` check classifies this as a
# normal client exit (not a ``client_error``).
raise _FakeWebSocketDisconnect("client closed")
async def send_text(self, text: str) -> None:
self.sent_text.append(text)
if (
self._disconnect_after_n_sends is not None
and len(self.sent_text) >= self._disconnect_after_n_sends
):
# Trigger the "client gone" signal the next receive_text will see.
self._disconnect_event.set()
async def send_bytes(self, data: bytes) -> None:
self.sent_bytes.append(data)
async def close(self, code: int | None = None, reason: str | None = None) -> None:
self.closed = True
self.close_code = code
def trigger_disconnect(self) -> None:
self._disconnect_event.set()
class _FakeHeaders:
"""Minimal stand-in for websockets' handshake ``Headers``.
Exposes both ``raw_items()`` (preferred by the production header
extractor to survive duplicate names like ``set-cookie``) and
``items()``.
"""
def __init__(self, pairs) -> None:
if isinstance(pairs, dict):
pairs = list(pairs.items())
self._pairs = [(str(k), str(v)) for k, v in pairs]
def raw_items(self):
return list(self._pairs)
def items(self):
return list(self._pairs)
class _FakeUpstream:
"""Upstream that streams scripted events then optionally blocks.
``hold_after_events`` makes the async iterator wait forever after the
scripted events are exhausted — that mirrors a real upstream that
keeps the connection open after a ``response.completed`` event. The
handler's ``_upstream_to_client`` will block on it, so the only way
the outer ``asyncio.wait`` can progress is via the client-side task
completing — which is exactly the cancel-partner path we want to
test.
"""
def __init__(
self,
events: list[str],
*,
hold_after_events: bool = False,
raise_mid_stream: Exception | None = None,
response_headers=None,
) -> None:
self._events = list(events)
self._hold_after_events = hold_after_events
self._raise_mid_stream = raise_mid_stream
self.sent: list[str] = []
self.closed = False
# Mirror websockets' ClientConnection.response.headers, which is the
# only place OpenAI delivers the Codex x-codex-* subscription window.
self.response = SimpleNamespace(headers=_FakeHeaders(response_headers or []))
async def __aenter__(self) -> _FakeUpstream:
return self
async def __aexit__(self, exc_type, exc, tb) -> None:
self.closed = True
async def send(self, payload: str) -> None:
self.sent.append(payload)
async def close(self) -> None:
self.closed = True
def __aiter__(self):
return self._iter()
async def _iter(self):
for ev in self._events:
yield ev
if self._raise_mid_stream is not None:
raise self._raise_mid_stream
if self._hold_after_events:
# Wait forever — until the task is cancelled by the handler.
await asyncio.Event().wait()
def _make_fake_websockets_module(
upstream: _FakeUpstream | None,
*,
call_log: list[str] | None = None,
connect_error: Exception | None = None,
):
"""Build a fake ``websockets`` module.
Production now does ``upstream = await websockets.connect(...)`` (then
``async with upstream``), so ``connect`` must return an awaitable that
resolves to the connection. ``connect_error`` makes the await raise to
simulate an upstream handshake failure.
"""
module = MagicMock()
async def _connect(*args, **kwargs):
if call_log is not None:
call_log.append("connect")
if connect_error is not None:
raise connect_error
return upstream
module.connect = _connect
module.Subprotocol = str
return module
def _first_frame() -> str:
return json.dumps(
{
"type": "response.create",
"response": {"model": "gpt-5.4", "input": "hi"},
}
)
# ---------------------------------------------------------------------------
# Tests
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_ws_first_frame_compression_uses_bounded_executor():
"""Codex WS compression must not run synchronously on the event loop."""
upstream_events = [
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
]
upstream = _FakeUpstream(upstream_events)
fake_ws_mod = _make_fake_websockets_module(upstream)
client_ws = _FakeWebSocket(frames=[_first_frame()])
handler = _DummyOpenAIHandler()
handler.config.optimize = True
handler._compress_openai_responses_payload = MagicMock(
return_value=(
{"model": "gpt-5.4", "input": "hi"},
False,
0,
[],
"router_no_compression",
10,
10,
)
)
with patch.dict(sys.modules, {"websockets": fake_ws_mod}):
await handler.handle_openai_responses_ws(client_ws)
assert handler.compression_executor_calls == 1
assert handler.compression_executor_timeouts == [COMPRESSION_TIMEOUT_SECONDS]
handler._compress_openai_responses_payload.assert_called_once()
@pytest.mark.asyncio
async def test_happy_path_registry_empty_after_response_completed():
"""Normal session completes — both relay tasks done, registry empty."""
upstream_events = [
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
]
upstream = _FakeUpstream(upstream_events)
fake_ws_mod = _make_fake_websockets_module(upstream)
client_ws = _FakeWebSocket(frames=[_first_frame()])
handler = _DummyOpenAIHandler()
with patch.dict(sys.modules, {"websockets": fake_ws_mod}):
await handler.handle_openai_responses_ws(client_ws)
assert handler.ws_sessions.active_count() == 0
assert handler.metrics.active_ws_sessions == 0
# termination_cause captured
assert handler.metrics.termination_causes
# Either "response_completed" or "client_disconnect" — both are
# acceptable here depending on which relay half exited first; the
# important thing is we recorded one.
assert handler.metrics.termination_causes[-1] in {
"response_completed",
"client_disconnect",
"upstream_disconnect",
}
@pytest.mark.asyncio
async def test_ws_session_metrics_include_response_completed_usage():
"""Codex WS sessions should report real upstream usage, not zero-token sessions."""
upstream_events = [
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
json.dumps(
{
"type": "response.completed",
"response": {
"id": "r_1",
"usage": {
"input_tokens": 100,
"input_tokens_details": {"cached_tokens": 75},
"output_tokens": 12,
},
},
}
),
]
upstream = _FakeUpstream(upstream_events)
fake_ws_mod = _make_fake_websockets_module(upstream)
client_ws = _FakeWebSocket(frames=[_first_frame()])
handler = _DummyOpenAIHandler()
with patch.dict(sys.modules, {"websockets": fake_ws_mod}):
await handler.handle_openai_responses_ws(client_ws)
assert handler.metrics.recorded_requests
recorded = handler.metrics.recorded_requests[-1]
assert recorded["input_tokens"] == 100
assert recorded["output_tokens"] == 12
assert recorded["cache_read_tokens"] == 75
assert recorded["cache_write_tokens"] == 25
assert recorded["uncached_input_tokens"] == 25
@pytest.mark.asyncio
async def test_ws_session_metrics_include_dashboard_performance_timings():
"""Codex WS response metrics should feed the dashboard Performance tab."""
upstream_events = [
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
json.dumps(
{
"type": "response.completed",
"response": {
"id": "r_1",
"usage": {
"input_tokens": 100,
"input_tokens_details": {"cached_tokens": 75},
"output_tokens": 12,
},
},
}
),
]
upstream = _FakeUpstream(upstream_events)
fake_ws_mod = _make_fake_websockets_module(upstream)
client_ws = _FakeWebSocket(frames=[_first_frame()])
handler = _DummyOpenAIHandler()
handler.config.optimize = True
def _noop_compress(payload, *, model, request_id, timing=None):
if timing is not None:
timing["compression_live_unit_extraction"] = 2.0
timing["compression_unit_router_strategy_passthrough"] = 3.0
return payload, False, 0, [], "test_noop", 10, 10, 0
handler._compress_openai_responses_payload = _noop_compress # type: ignore[method-assign]
with patch.dict(sys.modules, {"websockets": fake_ws_mod}):
await handler.handle_openai_responses_ws(client_ws)
assert handler.metrics.recorded_requests
recorded = handler.metrics.recorded_requests[-1]
assert recorded["overhead_ms"] > 0
assert recorded["ttfb_ms"] > 0
assert recorded["pipeline_timing"]["codex_ws.compression"] > 0
assert recorded["pipeline_timing"]["codex_ws.upstream_first_event"] > 0
assert recorded["pipeline_timing"]["codex_ws.compression_preflight_serialization"] > 0
assert recorded["pipeline_timing"]["codex_ws.compression_executor_wait_run"] > 0
assert recorded["pipeline_timing"]["codex_ws.compression_live_unit_extraction"] == 2.0
assert (
recorded["pipeline_timing"]["codex_ws.compression_unit_router_strategy_passthrough"] == 3.0
)
@pytest.mark.asyncio
async def test_client_disconnect_cancels_upstream_relay_within_100ms():
"""**Failing-test-first** scenario from the plan.
When the client side exits (``receive_text`` raises
``WebSocketDisconnect``) while upstream is still open and iterating,
the upstream relay task must be cancelled and become ``done()``
quickly. The registry must report no active sessions afterwards.
"""
# Upstream keeps iterating forever after one event, forcing the
# upstream-to-client task to block on the iterator. The only way
# out is a cancel from the handler's orchestration.
upstream_events = [
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
]
upstream = _FakeUpstream(upstream_events, hold_after_events=True)
fake_ws_mod = _make_fake_websockets_module(upstream)
# Client has one initial frame, then disconnects after the server
# sends the first forwarded event to us.
client_ws = _FakeWebSocket(
frames=[_first_frame()],
hold_after_initial=True,
)
handler = _DummyOpenAIHandler()
# Trigger disconnect shortly after the handler accepts.
async def _trigger() -> None:
await asyncio.sleep(0.05)
client_ws.trigger_disconnect()
with patch.dict(sys.modules, {"websockets": fake_ws_mod}):
trigger_task = asyncio.create_task(_trigger())
try:
await asyncio.wait_for(
handler.handle_openai_responses_ws(client_ws),
timeout=2.0,
)
finally:
trigger_task.cancel()
try:
await trigger_task
except asyncio.CancelledError:
pass
# Registry must be empty — the finally block deregistered the session.
assert handler.ws_sessions.active_count() == 0, (
"session leaked — deregister did not run in outermost finally"
)
assert handler.metrics.active_ws_sessions == 0
# We recorded a session duration (came through deregister path).
assert handler.metrics.ws_session_durations, (
"record_ws_session_duration never fired — deregister path broken"
)
# And we tagged the cause. For a client-side exit it should be one
# of: client_disconnect, client_error, upstream_disconnect (if
# upstream iteration happened to end first in a race).
cause = handler.metrics.termination_causes[-1]
assert cause in {
"client_disconnect",
"client_error",
"upstream_disconnect",
}, f"unexpected cause: {cause}"
# No codex-ws-* named task should still be running.
leaked = [
t
for t in asyncio.all_tasks()
if (t.get_name() or "").startswith("codex-ws-") and not t.done()
]
assert leaked == [], f"relay tasks leaked: {[t.get_name() for t in leaked]}"
@pytest.mark.asyncio
async def test_upstream_closes_first_cancels_client_task():
"""Upstream iterator ends naturally; client task should be cancelled.
The client is set to block on ``receive_text`` indefinitely; only a
cancel from the handler's orchestration releases it.
"""
upstream_events = [
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
]
upstream = _FakeUpstream(upstream_events, hold_after_events=False)
fake_ws_mod = _make_fake_websockets_module(upstream)
client_ws = _FakeWebSocket(
frames=[_first_frame()],
hold_after_initial=True,
)
handler = _DummyOpenAIHandler()
with patch.dict(sys.modules, {"websockets": fake_ws_mod}):
await asyncio.wait_for(
handler.handle_openai_responses_ws(client_ws),
timeout=2.0,
)
assert handler.ws_sessions.active_count() == 0
# We must still have recorded exactly one session duration.
assert len(handler.metrics.ws_session_durations) == 1
@pytest.mark.asyncio
async def test_upstream_error_mid_stream_classifies_as_upstream_error():
upstream_events = [
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
]
upstream = _FakeUpstream(
upstream_events,
raise_mid_stream=RuntimeError("boom from upstream"),
)
fake_ws_mod = _make_fake_websockets_module(upstream)
client_ws = _FakeWebSocket(
frames=[_first_frame()],
hold_after_initial=True,
)
handler = _DummyOpenAIHandler()
with patch.dict(sys.modules, {"websockets": fake_ws_mod}):
await asyncio.wait_for(
handler.handle_openai_responses_ws(client_ws),
timeout=2.0,
)
assert handler.ws_sessions.active_count() == 0
assert handler.metrics.termination_causes
assert handler.metrics.termination_causes[-1] == "upstream_error"
@pytest.mark.asyncio
async def test_response_cancel_frame_is_logged_as_client_cancel_lifecycle():
"""A Codex Ctrl-C maps to response.cancel on the WS stream.
The proxy should relay it upstream and classify the lifecycle as a
client-side cancel when no response.completed event follows.
"""
cancel_frame = json.dumps({"type": "response.cancel", "response_id": "r_1"})
upstream = _FakeUpstream([], hold_after_events=True)
fake_ws_mod = _make_fake_websockets_module(upstream)
client_ws = _FakeWebSocket(
frames=[_first_frame(), cancel_frame],
hold_after_initial=True,
)
handler = _DummyOpenAIHandler()
async def _trigger() -> None:
await asyncio.sleep(0.05)
client_ws.trigger_disconnect()
with patch.dict(sys.modules, {"websockets": fake_ws_mod}):
trigger_task = asyncio.create_task(_trigger())
try:
await asyncio.wait_for(
handler.handle_openai_responses_ws(client_ws),
timeout=2.0,
)
finally:
trigger_task.cancel()
try:
await trigger_task
except asyncio.CancelledError:
pass
assert cancel_frame in upstream.sent
assert handler.metrics.termination_causes[-1] == "client_cancel"
assert handler.ws_sessions.active_count() == 0
@pytest.mark.asyncio
async def test_upstream_connect_failure_still_deregisters_cleanly():
"""Handshake-phase leak must be impossible: if upstream connect
raises before relay tasks are created, the session is still
registered+deregistered cleanly (or never registered). Either way,
no leak.
"""
fake_ws_mod = _make_fake_websockets_module(None, connect_error=RuntimeError("upstream refused"))
client_ws = _FakeWebSocket(frames=[_first_frame()])
handler = _DummyOpenAIHandler()
async def _fallback(*args, **kwargs):
return None
handler._ws_http_fallback = _fallback # type: ignore[assignment]
with patch.dict(sys.modules, {"websockets": fake_ws_mod}):
await handler.handle_openai_responses_ws(client_ws)
assert handler.ws_sessions.active_count() == 0
@pytest.mark.asyncio
async def test_ws_connect_failure_falls_back_to_http():
"""When every upstream connect attempt fails, the client is still
accepted (with no x-codex-* headers, since there is no upstream
window) and the request is served via the HTTP POST fallback with
the client's first frame. Preserves the pre-reorder WS-upgrade-
failure behaviour.
"""
fake_ws_mod = _make_fake_websockets_module(
None, connect_error=RuntimeError("HTTP 500 from upstream")
)
first = _first_frame()
client_ws = _FakeWebSocket(frames=[first])
handler = _DummyOpenAIHandler()
fallback_calls: list[tuple] = []
async def _fallback(websocket, body, first_msg_raw, upstream_headers, request_id):
fallback_calls.append((body, first_msg_raw))
handler._ws_http_fallback = _fallback # type: ignore[assignment]
with patch.dict(sys.modules, {"websockets": fake_ws_mod}):
await handler.handle_openai_responses_ws(client_ws)
# Client was accepted with no upstream window to forward.
assert client_ws.accepted_headers is None
# Fallback ran with the first frame.
assert len(fallback_calls) == 1
_body, _first_raw = fallback_calls[0]
assert _first_raw == first
assert _body == json.loads(first)
# Clean teardown.
assert handler.ws_sessions.active_count() == 0
@pytest.mark.asyncio
async def test_ws_connect_happens_before_accept():
"""The upstream connect must complete before the client 101 is sent,
so OpenAI's x-codex-* handshake headers are available to attach.
"""
upstream_events = [
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
]
call_log: list[str] = []
upstream = _FakeUpstream(upstream_events)
fake_ws_mod = _make_fake_websockets_module(upstream, call_log=call_log)
client_ws = _FakeWebSocket(frames=[_first_frame()], call_log=call_log)
handler = _DummyOpenAIHandler()
with patch.dict(sys.modules, {"websockets": fake_ws_mod}):
await handler.handle_openai_responses_ws(client_ws)
assert "connect" in call_log and "accept" in call_log
assert call_log.index("connect") < call_log.index("accept"), (
f"connect must precede accept, got {call_log}"
)
@pytest.mark.asyncio
async def test_ws_forwards_codex_headers_to_client_accept():
"""OpenAI's x-codex-* subscription window from the upstream WS
handshake must be forwarded onto the client-facing 101 (and only
that subset — never set-cookie/authorization), and Python /stats
state must be refreshed.
"""
upstream_events = [
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
]
# Include duplicate set-cookie to ensure raw_items() is used (a plain
# dict-style .items() on real websockets Headers raises on dupes).
handshake_headers = [
("x-codex-primary-used-percent", "42"),
("X-Codex-Primary-Window-Minutes", "300"),
("set-cookie", "a=1"),
("set-cookie", "b=2"),
("authorization", "Bearer leak"),
]
upstream = _FakeUpstream(upstream_events, response_headers=handshake_headers)
fake_ws_mod = _make_fake_websockets_module(upstream)
client_ws = _FakeWebSocket(frames=[_first_frame()])
handler = _DummyOpenAIHandler()
captured: dict = {}
def _fake_state():
class _S:
def update_from_headers(self, headers):
captured.update(headers)
return _S()
with (
patch.dict(sys.modules, {"websockets": fake_ws_mod}),
patch(
"headroom.subscription.codex_rate_limits.get_codex_rate_limit_state",
_fake_state,
),
):
await handler.handle_openai_responses_ws(client_ws)
assert client_ws.accepted_headers is not None
names = {name.decode("latin-1").lower() for name, _ in client_ws.accepted_headers}
assert names == {"x-codex-primary-used-percent", "x-codex-primary-window-minutes"}
assert "set-cookie" not in names
assert "authorization" not in names
# Original-case names preserved on the wire.
sent = {name.decode("latin-1") for name, _ in client_ws.accepted_headers}
assert "X-Codex-Primary-Window-Minutes" in sent
# Python /stats state refreshed with the same x-codex-* subset.
assert captured == {
"x-codex-primary-used-percent": "42",
"X-Codex-Primary-Window-Minutes": "300",
}
@pytest.mark.asyncio
async def test_ws_first_frame_timeout_after_connect_closes_upstream():
"""If the client never sends its first frame after we connected, the
upstream WS must be closed (no leak) and the session deregistered.
"""
upstream = _FakeUpstream([], hold_after_events=True)
fake_ws_mod = _make_fake_websockets_module(upstream)
# No frames + hold => receive_text blocks until disconnect; we force a
# short first-frame timeout so the handler hits the timeout branch.
client_ws = _FakeWebSocket(frames=[], hold_after_initial=True)
handler = _DummyOpenAIHandler()
with (
patch.dict(sys.modules, {"websockets": fake_ws_mod}),
patch(
"headroom.proxy.handlers.openai.WS_FIRST_FRAME_TIMEOUT_SECONDS",
0.05,
),
):
await asyncio.wait_for(
handler.handle_openai_responses_ws(client_ws),
timeout=2.0,
)
assert upstream.closed, "upstream not closed on first-frame timeout"
assert client_ws.closed and client_ws.close_code == 1001
assert handler.ws_sessions.active_count() == 0
@pytest.mark.asyncio
async def test_many_concurrent_sessions_cleanly_drained():
"""50 concurrent sessions: all drain; registry and named tasks go to 0."""
upstream_events = [
json.dumps({"type": "response.created", "response": {"id": "r_1"}}),
json.dumps({"type": "response.completed", "response": {"id": "r_1"}}),
]
async def run_one() -> None:
upstream = _FakeUpstream(list(upstream_events))
fake_ws_mod = _make_fake_websockets_module(upstream)
client_ws = _FakeWebSocket(frames=[_first_frame()])
handler = _DummyOpenAIHandler()
with patch.dict(sys.modules, {"websockets": fake_ws_mod}):
await handler.handle_openai_responses_ws(client_ws)
assert handler.ws_sessions.active_count() == 0
await asyncio.gather(*[run_one() for _ in range(50)])
# Global check: no codex-ws-* named task remains.
leaked = [
t
for t in asyncio.all_tasks()
if (t.get_name() or "").startswith("codex-ws-") and not t.done()
]
assert leaked == []