## Description
Codex's subscription usage window (primary/secondary rate-limit gauges)
stopped populating for ChatGPT-OAuth sessions. This PR restores it by
polling Codex's dedicated usage endpoint instead of relying on response
headers that are no longer sent.
### Why the previous approach no longer works
The existing code populates `CodexRateLimitState` from `x-codex-*`
rate-limit headers captured on the `/v1/responses` WebSocket handshake
(`update_from_headers` at WS accept). That worked when OpenAI returned
`x-codex-primary-used-percent`, `x-codex-primary-window-minutes`, etc.
on the handshake response.
OpenAI has since stopped sending those headers on the ChatGPT WebSocket
handshake. I confirmed this by faithfully replaying a real Plus-account
handshake (both `prewarm` and regular `request_kind`): no `x-codex-*`
headers come back on either. This matches OpenAI's own move to a
dedicated usage endpoint (`GET /backend-api/codex/usage` in CodexApi
mode) and reports such as openai/codex#14728. So `update_from_headers`
now runs on every accept but finds nothing to parse, and the window
silently stays empty.
The headers aren't coming back, so there is nothing to fix in the
parsing path. The data now lives behind a request we have to make
ourselves.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `subscription/codex_rate_limits.py`:
- `parse_codex_usage_payload()` / `update_from_usage_payload()` — map
the `GET /backend-api/wham/usage` JSON (`rate_limit.primary_window` /
`secondary_window` with `used_percent`, `limit_window_seconds`,
`reset_at`; `credits`; `rate_limit_reached_type`) into the existing
`CodexRateLimitState`. `limit_window_seconds` is converted to
window-minutes with the same round-up codex-rs uses (`(secs + 59) //
60`).
- `maybe_schedule_usage_poll()` — fire-and-forget, throttled to one
request per 60s, scoped to ChatGPT sessions (requires both a Bearer
token and `ChatGPT-Account-Id`; API-key traffic is skipped). Uses an
in-flight guard so concurrent accepts don't stack polls. Endpoint is
overridable via `HEADROOM_CODEX_USAGE_URL`.
- `proxy/handlers/openai.py`:
- At the Codex WS accept site, after the now-usually-empty
`update_from_headers` block, schedule the usage poll. Wrapped in
`contextlib.suppress` and fully non-blocking so it can never delay or
fail the WebSocket accept.
The old header-capture path is intentionally left in place as a no-cost
fallback in case OpenAI restores the headers.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed (live `/wham/usage` replay against a Plus
account returned HTTP 200 with the expected schema; payload fixture in
tests mirrors that real shape)
## Test Output
```
$ uv run pytest tests/test_codex_rate_limits.py -q
........................................ [100%]
41 passed in 0.16s
$ uv run ruff check headroom/subscription/codex_rate_limits.py headroom/proxy/handlers/openai.py tests/test_codex_rate_limits.py
All checks passed!
$ uv run mypy headroom/subscription/codex_rate_limits.py
Success: no issues found in 1 source file
```
## Additional Notes
- New tests cover: full-payload mapping, window-minutes round-up,
credits balance kept only when `has_credits`, promo object vs string,
empty payload returns `None`, missing `used_percent` skipped,
header-gating (requires Bearer + account-id), poll throttling, and
no-event-loop safety.
- Scoping to `ChatGPT-Account-Id` keeps the poll off API-key traffic,
and the 60s throttle plus in-flight guard bound it to at most one
lightweight GET per minute per running proxy.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Description
Adds first-class GCP Vertex AI proxy routing for publisher REST
endpoints so Vertex requests are forwarded to a configurable regional
Vertex host instead of falling through to the generic
OpenAI/Anthropic/Gemini passthrough selection.
Fixes#792
## Type of Change
- [x] New feature (non-breaking change that adds functionality)
- [x] Documentation update
## Changes Made
- Added a `vertex` provider target with `VERTEX_TARGET_API_URL` and
`--vertex-api-url` support.
- Registered explicit Vertex publisher routes for Google
`generateContent`, `streamGenerateContent`, `countTokens` and Anthropic
publisher `rawPredict`, `streamRawPredict` passthrough.
- Added startup banner/routing output for Vertex AI.
- Added focused tests for provider target resolution, CLI/env config,
banner output, and route delegation.
- Added `wiki/vertex.md` with usage examples and Google Cloud source
links.
## Sources
- Vertex AI Gemini inference reference:
https://docs.cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference
- Google Cloud REST authentication:
https://docs.cloud.google.com/docs/authentication/rest
- Google Application Default Credentials:
https://docs.cloud.google.com/docs/authentication/application-default-credentials
## Testing
- [x] Linting passes (`python -m ruff check .`)
- [x] New tests added for new functionality
- [x] Focused unit tests pass
- [ ] Full unit suite completed locally
- [ ] Rust tests completed locally
- [ ] Type checking passes locally
## Test Output
```text
$ python -m ruff check .
All checks passed!
$ python -m pytest tests/test_provider_registry.py tests/test_provider_proxy_routes.py tests/test_cli_proxy_env.py tests/test_banner_upstream_targets.py -q
57 passed, 1 warning in 13.75s
```
Local limitations:
- `python -m pytest tests scripts/tests -q` timed out after 1 hour on
this Windows machine before completing.
- `cargo test -p headroom-proxy --test integration_vertex_raw_predict`
could not run because `cargo` is not installed on PATH in this
environment.
- The commit hook's `mypy` step fails locally on an existing Windows
`fcntl` typing issue in `headroom/subscription/tracker.py`; `ruff`,
`ruff-format`, and plugin-version hooks passed, and the commit was made
with only `mypy` skipped.
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have made corresponding changes to the documentation
- [x] I have added tests that prove the feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
Closes the cross-project context leak Jocelyn reported 2026-05-26:
working on a Ruby/Rails project (daphni-rails), an unrelated Python
file (an Ollama inference provider from project `tamag0`) was being
injected into context as "Proactive Context Expansion - relevant to
your query". Two completely different projects, two different
languages, two different working directories — but the same proxy
process was serving both, and the in-memory ContextTracker had no
workspace identity to filter on.
Root cause
----------
`self.ccr_context_tracker` is one instance per proxy process. Every
session, every project, every user shared the same `_contexts` dict.
`track_compression()` stored sample content with no provenance key;
`analyze_query()` ran lexical keyword overlap across the full dict
without filtering. Within the 5-minute age window, surface-level
token matches ("provider", "session", "oauth", generic code/test
structure) scored above the 0.3 relevance threshold, recommendations
came back, and execute_expansions() injected the full original
content into a foreign session.
Refuted: this is NOT a race condition (joce's hypothesis). It
reproduces single-threaded, one-request-at-a-time. Plain shared
mutable state.
Fix
---
Add a required `workspace_key` to the tracker API and filter on it
inside `analyze_query`:
1. `CompressedContext` gets a `workspace_key: str` field.
2. `track_compression(..., workspace_key=...)` is now keyword-only,
no default — fail-loud on missing.
3. `analyze_query(..., workspace_key=...)` is also keyword-only; an
empty workspace_key short-circuits to `[]` (fail-closed per
`feedback_no_silent_fallbacks`).
4. The loop at `analyze_query` skips any entry whose workspace_key
differs from the request's.
In the Anthropic proxy handler:
5. New `_resolve_ccr_workspace(request, body)` static helper uses the
memory subsystem's `ProjectResolver` so CCR and memory agree on
project identity. Tier order: x-headroom-project-id →
x-headroom-cwd → CLI override → cwd: line in system prompt.
6. Both track and analyze sites gate on `ccr_workspace_key` being
non-empty — turning off proactive expansion entirely when project
identity can't be resolved is the safest default (it's an
optimization, not correctness).
7. `format_expansions_for_context(expansions, workspace_label=...)`
was already wired (GH #462 Fix C); the call site now passes the
label so the injected block declares its provenance, symmetric
with the memory injection header.
Affected population
-------------------
- Default mode (no `--cache`): bug fixed.
- Cache mode: was never affected — proactive expansion short-
circuits in cache mode to preserve prefix stability.
Tests
-----
- 6 new workspace-scoping tests in `test_ccr_context_tracker.py`:
same-workspace match still works, cross-workspace silently
filtered, empty workspace_key fail-closes, two workspaces each
see only their own, workspace_label propagates to formatter, LRU
cross-workspace doesn't leak even with full tracker.
- 6 new `_resolve_ccr_workspace` resolver tests in
`test_proxy_handler_helpers.py`: explicit project-id wins, cwd
header → key+label, two cwds get distinct keys, no-signal
fail-closed, system-prompt cwd: fallback, malformed request
fail-closed.
- 32 existing tracker tests updated to pass `workspace_key="ws-test"`.
- 55/55 tests pass; ci-precheck green.
Defense-in-depth follow-up
--------------------------
The compression_store itself (`headroom/cache/compression_store.py`)
also lacks workspace scoping — a CCR `headroom_retrieve` call from
Project B for a hash created by Project A would succeed. The
practical attack surface is closed by this PR (hashes only reach
Project B's model via proactive expansion, now gated), but
defense-in-depth hardening of the store is worth a separate PR.
Filed as task #44.
Eliminates P0-2 universally. Every Python forwarder (server.py
`_retry_request`, handlers/streaming.py `_stream_response`,
handlers/openai.py `_ws_http_fallback`, handlers/batch.py `_batch_passthrough`
+ batch-create + Google batch passthrough, handlers/anthropic.py CCR
continuation + batch endpoint) now switches from `httpx ... json=body` to
`httpx ... content=raw_bytes`. The default httpx JSON encoder was
re-serializing every request with `, `/`: ` separators and `\\uXXXX` ASCII
escapes — collapsing Anthropic prompt-cache hit-rate.
Forwarder strategy:
- unmutated body → forward `await request.body()` verbatim;
- mutated body → re-serialize once via the new
`serialize_body_canonical(body) -> bytes` helper (compact separators,
`ensure_ascii=False`, dict insertion order preserved).
`HEADROOM_PROXY_PYTHON_FORWARDER_MODE` env var configures the mode:
- `byte_faithful` (default) — the new behavior;
- `legacy_json_kwarg` — explicit operator opt-in for emergency rollback.
Documented in `docs/content/docs/configuration.mdx`. NOT a fallback —
unknown values raise loudly per build constraint #4.
`BodyMutationTracker` accompanies each request through the handler so
transform sites mark the tracker (`memory_injection`,
`image_compression`, `compression_*`, `batch_compression`,
`ccr_continuation`, etc.). At forwarder dispatch we additionally compare
the final body dict against the parsed original bytes as a structural
safety net — any silent mutation we missed still triggers canonical
re-serialization.
A2 follow-up: `handlers/openai.py:534-540` (Chat Completions memory
injection) was prepending a system message; replaced with
`append_text_to_latest_user_chat_message`, the OpenAI Chat Completions
analog of `_append_context_to_latest_non_frozen_user_turn`. The cache
hot zone (system messages) is now sacrosanct on /v1/chat/completions
too. Honors `HEADROOM_MEMORY_INJECTION_MODE=disabled`.
Structured logging: every forwarder emits an `event=outbound_request`
log line with `forwarder`, `path`, `body_bytes`, `body_mutated`,
`mutation_reasons`, `source` (passthrough|canonical|legacy),
`request_id`. Never logs Authorization or full body.
`_read_request_json` factored to share `_read_request_body_bytes` with
new `read_request_json_with_bytes` so the anthropic handler can capture
both the parsed dict and the original (decompressed) bytes.
Tests:
- `tests/test_proxy_byte_faithful_forwarding.py` (28 tests):
SHA-256 byte-equality on /v1/messages and streaming, unicode
preservation, numeric precision, mutation-tracker invariants,
canonical-serializer properties, legacy-mode rollback, OpenAI
Chat memory routing.
- Existing test mocks updated to accept the new `**kwargs` on
`_retry_request` (no behavior change).
- `tests/test_proxy_handlers_batch.py` updated to read the captured
`content=` bytes (formerly `json=`).
- One A2 test corrected (`test_anthropic_tool_sort_and_context_append_helpers`)
to match the live-zone-tail semantics introduced by A2.
Constraints satisfied: configurable env var; no new regex / hardcodes;
no silent fallback (`legacy_json_kwarg` is operator opt-in);
performant (`prepare_outbound_body_bytes` is O(1) for passthrough);
elegant single-responsibility helpers; structured tracing logs.
`.gitattributes` declares `*.py text eol=lf` and `*.sh text eol=lf`, but
74 files (73 .py, 1 .sh) are stored in the index with CRLF line endings,
violating that contract. Every macOS/Linux clone reports these files as
"modified" on fresh checkout because git's diff engine sees the stored
bytes don't match the attribute contract, even though the working tree
and index match byte-for-byte.
Running `git add --renormalize .` rewrites each affected blob so the
stored form matches the attribute declaration. No semantic changes —
every affected file's diff is "N insertions, N deletions" with inserts
and deletes being the same lines modulo line endings.
Follow-up commit adds `.git-blame-ignore-revs` so `git blame` / GitHub
blame skip this mechanical commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
