Mirrors the pattern used by headroom.ccr_backend so memory store,
vector index, and text index backends can be registered via setuptools
entry points.
- EXTERNAL enum value on StoreBackend, VectorBackend, TextBackend
- Optional *_backend_name fields on MemoryConfig
- entry_points(group=...) lookup in _create_{store,vector_index,text_index}
- New test_factory_external.py (7 tests) covering load / missing-name /
unknown-name paths
Default behavior (SQLITE + AUTO + FTS5) unchanged.
Extension groups:
headroom.memory_store
headroom.memory_vector
headroom.memory_text
The openclaw plugin depends on headroom-ai, but during the release build
the version-sync script updates the dependency to the new version (e.g.
^0.6.7) which hasn't been published to npm yet. Fix by installing the
locally-packed SDK tarball first so the dependency is already satisfied
when npm install runs for remaining packages.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Tests for jitter_delay_ms helper, asyncio.timeout shim, release workflow
--allow-same-version flag validation, SIGKILL fallback on Windows, and
LatencyHistogram from the repro harness script.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
asyncio.timeout was added in Python 3.11, but the project supports >=3.10.
The test_repro_codex_replay_smoke test was crashing on Python 3.10 CI with
"AttributeError: module 'asyncio' has no attribute 'timeout'".
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove 101 trailing whitespace violations (W293), 2 unused imports
(F401), 1 import sort issue (I001) in anthropic handler and test file
- Apply ruff format to anthropic handler and oauth routing test
- Fix mypy no-any-return in jitter_delay_ms by adding explicit type
- Fix mypy attr-defined for signal.SIGKILL on Windows using getattr
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The version-sync.py script already sets the target version in package.json
before npm version runs. When npm version receives the same version that's
already in package.json, it exits with "Version not changed" (exit code 1),
breaking the build job. Adding --allow-same-version makes npm version a
no-op when the version matches, fixing the release pipeline.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add websockets to the dev extra so the repro harness smoke test can import
its websocket client dependency in the CI test matrix. Also apply ruff
formatting to the files the formatter check was rejecting so the 3.12 lint
job passes.
Upgrade of the local uv CLI rewrote the lock file with `revision = 3`
and renamed the metadata key `upload_time` -> `upload-time`. No actual
dependency version changes.
Commit 62d0a50 widened the outer try/except around
handle_anthropic_messages to guarantee semaphore release on every
path. Because fastapi.HTTPException inherits from Exception, the
existing catch-all at line 1614 began intercepting the 429
rate-limit and 429 budget raises (and any other HTTPException),
turning them into opaque 502 responses with no Retry-After.
Add an explicit 'except HTTPException: raise' ahead of the
Exception handler so FastAPI's own handler still produces the
correct status code and headers. The outer finally still runs,
so _finalize_pre_upstream() and the semaphore release are
unaffected.
Also tighten the contention test: it previously caught any
exception with 'except Exception: pass', which masked this
regression. The new assertions pin each scenario to its documented
contract (HTTPException(429) for rate-limiter/cost-tracker;
response object for security/cache) so a future regression of
this shape fails loudly.
Task.get_stack(limit=32) walks coroutine frames synchronously and
measurably stalls the event loop when called for 50+ relay tasks during
a reconnect storm. The /debug/tasks snapshot does not need this by
default — the perf-cheap fields (name, coro_qualname, age, done) are
enough for the common case of "which tasks are alive".
- collect_tasks: add with_stack_depth=False kwarg; default leaves
stack_depth=None per entry.
- /debug/tasks: read ?stack=true query parameter and forward it. Default
response is cheap; opt-in explicitly when human-debugging one snapshot.
- Test: verify default entries have stack_depth=None and that
?stack=true produces at least one integer depth.
- Route --json output cleanly: JSON goes to stdout (machine-readable),
human-readable summary goes to stderr. Without --json, the human
summary stays on stdout as before. Makes piping the harness into
jq / other tools trivial without losing operator visibility.
- Add distinct exit codes so CI and shell wrappers can branch on the
failure class: 0 success, 1 crash, 2 proxy_unreachable, 3 livez
threshold exceeded, 4 warmup failed, 130 SIGINT (already correct).
Smoke test updated to assert EXIT_PROXY_UNREACHABLE instead of 1.
- Replace flat 50-250ms jitter in the Anthropic-client retry loop with
exponential backoff + 50-150% jitter
(base=250ms, max=5000ms, attempt counter). Matches the proxy's own
jitter_delay_ms helper; inlined to keep the script free of proxy
package imports.
- Log the resolved Anthropic pre-upstream concurrency at startup so
operators can correlate pre_upstream_wait_ms lines with the configured
cap. Distinguishes auto-detected vs. explicit vs. unbounded.
- Remove the stage_timings, session_id, and stage_timings_path fields
from RequestLog. They were declared optional but never populated at
any of the three RequestLog construction sites — stage timings flow
exclusively through emit_stage_timings_log (structured log line) and
Prometheus histograms. Keeping unpopulated fields misled readers.
Option (A) from the review.
- Inline the one-liner collect_ws_sessions / collect_warmup pass-throughs
at their two callers in server.py; keep collect_tasks since it has
real logic. Unit tests were rewritten against the registry's own
snapshot()/to_dict() serializers.
- Reflow StageMeasurement class declaration across lines (E501 at 116 chars).
- Add `from __future__ import annotations` to loopback_guard.py to match
sibling modules.
- Promote `TerminationCause` from bare `str` alias to `Literal[...]` with
the full set of valid causes (including `client_timeout` from P1 Fix 5).
Assignment sites in handlers/openai.py now pick up static validation.
- Tighten `WSSessionHandle.relay_tasks` to `list[_TaskLike]` using the
existing Protocol; drop defensive `getattr` on task names.
The relay termination-cause classifier read exc = t.exception()
inside contextlib.suppress(Exception). For a cancelled task,
.exception() raises CancelledError (BaseException) — which is
NOT caught by Exception, but also isn't the exception we want.
Separately, for a cancelled task we want to surface CancelledError
so the downstream isinstance(exc, CancelledError) branches run.
Split the two cases: if t.cancelled() is True, synthesize a
CancelledError() explicitly. Otherwise read t.exception() under
contextlib.suppress(asyncio.InvalidStateError) (defensive — the
task should be done post-gather).
_emit_pre_upstream_stage_timings under-communicated the function's
primary action: releasing the Unit 4 pre-upstream semaphore. Emitting
stage timings is secondary bookkeeping. Rename to _finalize_pre_upstream
across all ~15 call sites and add a docstring that explicitly calls out
idempotency + the semaphore-release contract.
Parametrized tests for the 4 pre-upstream early-return paths that bail
out of handle_anthropic_messages before reaching the upstream call:
- rate_limiter denies (429)
- cost_tracker blocks (429)
- security scan blocks (403)
- cache hit (200 — skips upstream entirely)
Each test holds a Semaphore(1) across multiple calls to confirm the
handler restores the original _value on every path, catching any
regression that forgets to call _emit_pre_upstream_stage_timings()
before the HTTPException / early return.
Pre-existing asymmetry with SubscriptionTracker.stop(): copilot's stop()
did not call task.cancel() on the 5s wait_for timeout. Now that shield
is removed the task is cancellable and should be cancelled on timeout —
otherwise a wedged poll task leaks past stop().
record_stage_timings() acquired the global asyncio.Lock on every
request finalization — the same lock held by export() during Prometheus
scrapes, which does string-building while holding it. Under N concurrent
request finalizations + an active scrape, all N queued behind the scrape.
Move the stage-timing triple update (sum + count + max) onto a tiny
synchronous threading.Lock. This keeps the multi-field update consistent
for scrapers without contending on the async lock. export() now snapshots
the three dicts under the sync lock, then builds the metrics string from
the snapshots outside any lock scope.
External cancellation while _ensure_initialized is in flight raises
CancelledError (BaseException). The existing 'except asyncio.TimeoutError:'
branch does not catch it, and caller 'except Exception:' blocks don't
either, so it propagated as-is with _backend possibly still assigned.
Add an explicit 'except asyncio.CancelledError:' handler that nulls
_backend, clears _initialized, logs at info, and re-raises. Cancellation
is a shutdown signal, not an error to swallow — but the state must be
clean so a later retry starts fresh.
If asyncio.wait_for fires while LocalBackend._ensure_initialized() is
running, _init_backend_locked may have already assigned self._backend
(the LocalBackend constructor) before its own await raised or was
cancelled. Callers doing if self.memory_handler._backend: then see
a truthy-but-broken backend.
Null self._backend in the TimeoutError handler so the post-timeout
state is consistent: _initialized=False AND _backend=None.
On Linux dual-stack sockets (IPV6_V6ONLY=0 default), an IPv4 loopback
connection arrives as ::ffff:127.0.0.1 and was 404'd by the literal set
check — silent outage of /debug/* when the proxy binds to :: or 0.0.0.0.
Replace the set-membership check with ipaddress.ip_address(host).is_loopback,
special-casing 'localhost' (not an IP literal) and treating ValueError
(malformed input) as non-loopback. The None sentinel for TestClient / UDS
sockets is preserved.
Both SubscriptionTracker._poll_loop and _CopilotQuotaTracker._poll_loop
wrapped their stop-event wait in asyncio.shield() inside wait_for().
When wait_for timed out (every poll_interval_s), it cancelled its own
outer task but the shielded inner Event.wait kept running forever —
one leaked Task per poll interval per tracker.
Across the two default trackers (10s poll each) this leaks ~0.2
tasks/sec in steady state: ~17k/day, ~120k/week. Event-loop
scheduler cost grows linearly with task count, which eventually
starves /livez and new WS accepts on long-lived processes. This
matches the 'aged :8787 proxy degrades over hours/days' symptom
captured in wiki/plans/2026-04-17-codex-proxy-runtime-analysis.md.
Discovered by the /debug/tasks endpoint added in Unit 5 of the
codex-proxy-resilience plan — 60s of idle time on a fixed-version
fork now shows 0 Event.wait tasks vs 310 → 320 growth on the
unpatched build.
Drop the shield wrapper so wait_for can cleanly cancel the inner
wait when its timeout fires. The stop() contract is unaffected:
setting _stop_event still returns the wait normally before the
timeout, triggering the break.
Adds a regression test per tracker that starts the poll loop with
a 50ms interval, lets it run for ~6 cycles, stops it, and asserts
Event.wait task count does not grow beyond baseline + 1 (the one
legitimate in-flight waiter).
Some earlier tests in the suite replace sys.modules['websockets'] with
a stub. When the smoke test ran after those, uvicorn's
websockets-sansio backend tried to import websockets.server at connect
time, resolved through the stub, and failed with
'No module named websockets.server; websockets is not a package',
preventing the mock proxy from starting.
Add an autouse fixture that drops stub websockets.* entries from
sys.modules, re-imports the real package + websockets.asyncio.server,
and restores prior state on teardown. Keeps the smoke test
deterministic regardless of collection order.
uvicorn's auto-select imports websockets_impl.py which requires the
legacy API (websockets<14). With websockets 16.0 installed in the
local venv, auto-select errored before its fallback could trigger,
failing the smoke test with 'No module named websockets.legacy'.
Explicitly selecting the websockets-sansio backend (shipped with
uvicorn) works against websockets 16.0 and keeps the smoke test
representative of the real production WS path.
Unit 5 of the Codex-proxy resilience plan. Three always-on
/debug/* endpoints make "what is this process doing right now?"
a single curl away:
- GET /debug/tasks serializes asyncio.all_tasks() to name,
coro_qualname, stack_depth, done, and age_seconds (derived from
the WS session registry for Codex relay tasks, null otherwise).
Sorted by age desc. No frame locals, no request bodies, no
coroutine args.
- GET /debug/ws-sessions returns WebSocketSessionRegistry.snapshot().
- GET /debug/warmup returns WarmupRegistry.to_dict().
A FastAPI Depends(require_loopback) dependency 404s (not 403) any
non-loopback caller so debug routes stay invisible to external
scanners. Dependency chosen over middleware because the route set
is small and explicit Depends(...) makes the guard visible to
reviewers and easy to override in tests.
Unit 4 of the Codex proxy resilience plan. Cold-start Anthropic replay
storms (many concurrent large POSTs immediately after restart) could
occupy every event-loop slot and thread-pool worker with deep-copy /
compression / memory-context work before any upstream connect, starving
``/livez`` and new Codex WS opens.
Gate the pre-upstream phase of ``handle_anthropic_messages`` with an
``asyncio.Semaphore`` constructed once per process. The semaphore is
acquired right after stage-timer setup and released before streaming
response bytes back to the client — it never spans the whole response
lifetime. Release is also driven from ``_emit_pre_upstream_stage_timings``
(already called at every known exit point), with explicit releases on
rate-limit / budget HTTPException, cache hit, security block, and
Bedrock errors.
Config: ``ProxyConfig.anthropic_pre_upstream_concurrency`` (None =>
auto-compute ``max(2, min(8, os.cpu_count() or 4))``; 0 or negative =>
disables the semaphore for the Unit 6 counter-factual).
CLI: ``--anthropic-pre-upstream-concurrency``.
Env: ``HEADROOM_ANTHROPIC_PRE_UPSTREAM_CONCURRENCY``.
Precedence: CLI > env > auto-compute.
Observability: a new ``pre_upstream_wait`` stage is recorded on the
existing ``StageTimer`` and surfaces in ``/metrics`` and stage-timings
log lines. Waits > 100ms emit an INFO log line with request_id +
session_id + wait ms so queueing is visible under load.
Compression stays enabled. Codex WS, OpenAI HTTP, ``/livez``,
``/readyz``, ``/health``, ``/metrics``, ``/stats`` are unchanged.
Unit 3 of the Codex proxy resilience plan. Eliminates the "aged process
has leaked relay tasks" hypothesis by making every WS session explicitly
tracked and both relay tasks deterministically cancelled when either
exits.
- New headroom/proxy/ws_session_registry.py: dict-backed
WebSocketSessionRegistry + WSSessionHandle with register /
deregister / attach_tasks / snapshot. Deregister is idempotent and
clears relay-task references so coroutine frames are not retained
past session end.
- HeadroomProxy exposes proxy.ws_sessions so /debug/ws-sessions
(Unit 5) can read the live snapshot.
- handle_openai_responses_ws now registers on websocket.accept()
success and deregisters in the outermost finally so no leak can
survive handshake-phase, mid-stream, or upstream-error paths. The
session_id / termination_cause is threaded through both relay
halves and both sides raise asyncio.CancelledError cleanly.
- Replaced asyncio.gather(_client_to_upstream(), _upstream_to_client(),
return_exceptions=True) with explicit asyncio.create_task(...)
(named codex-ws-c2u-<sid> / codex-ws-u2c-<sid>) +
asyncio.wait(FIRST_COMPLETED) + cancel-and-await on the survivor.
Termination cause is classified as client_disconnect /
client_error / upstream_disconnect / upstream_error /
response_completed from which task completed first plus inline
error captures from the halves.
- Prometheus metrics: new active_ws_sessions and active_relay_tasks
gauges plus ws_session_duration_ms_{sum,count,max} histogram
bucketed by termination cause. Mirrors the Unit 2 stage_timing_*
shape.
Preserved: upstream WS retry loop, WS→HTTP fallback, memory-context
timeout, compression pipeline, Unit 2 stage timings. Memory-tool
execution inside _upstream_to_client still runs when the client task
exits first; however, if the client disconnects *before* the upstream
emits response.completed, pending memory writes in `pending_fcs` are
dropped (unchanged from prior behavior — a crashing upstream has the
same effect). Note: handle_openai_responses (HTTP, line ~800) is a
single-shot HTTP request; lifecycle tracking isn't added there
(scope boundary).
Tests:
- tests/test_ws_session_registry.py: 8 registry unit tests
(register/deregister idempotency, snapshot shape, attach merging,
reference release).
- tests/test_openai_codex_ws_lifecycle.py: 6 integration tests
using real relay tasks (only upstream WS endpoint mocked):
happy-path, failing-test-first "client disconnect cancels upstream
relay within 100 ms", upstream-closes-first, upstream-error mid-
stream, handshake-failure deregister, 50 concurrent sessions.
- Regression: test_openai_codex_ws_timings, test_openai_codex_routing,
test_proxy_codex_route_aliases, test_ws_memory_relay all pass.
- Tests pass under python -W error::RuntimeWarning (no "coroutine
was never awaited").
Unit 1 of the codex-proxy-resilience plan. Preload now iterates BOTH
the Anthropic and OpenAI transform pipelines, dedupes shared transforms
by id(), and merges the status dict into a typed WarmupRegistry so
/debug/warmup (Unit 5) and /readyz have one source of truth.
MemoryHandler._ensure_initialized gains an asyncio.Lock with a
double-checked pattern so concurrent first callers share one backend
init instead of racing. The body is wrapped in asyncio.wait_for with
STARTUP_INIT_TIMEOUT_SECONDS (default 30s) — on timeout _initialized
stays False so subsequent requests retry (fail-open contract).
Startup now also forces one embedder warm-up encode so the ONNX graph
is compiled synchronously at startup instead of lazily on the first
request. Best-effort: failures are logged, never raised.
Unit 2 of the Codex proxy resilience plan. Introduce a lightweight
StageTimer utility and thread it through handle_openai_responses_ws and
handle_anthropic_messages to capture per-stage durations
(accept/first_client_frame/upstream_connect/upstream_first_event/
memory_context/compression/total_session for Codex WS;
read_request_json/deep_copy/compression_first_stage/memory_context/
upstream_connect/upstream_first_byte/total_pre_upstream for Anthropic
HTTP).
Each request/session emits exactly one structured log line
(STAGE_TIMINGS ...) and records sum/count/max Prometheus histograms
under headroom_stage_timing_ms_{sum,count,max} keyed by
(path, stage). A new session_id UUID is generated per request/session
and paired with the existing request_id so multi-turn sessions are
correlatable.
Compression, WS retries, WS->HTTP fallback, and the memory-context
fail-open timeout are unchanged. RequestLog gains three optional
backward-compatible fields (stage_timings, session_id,
stage_timings_path).
Adds `headroom/proxy/extensions.py` — a generic entry-point hook under
the `headroom.proxy_extension` group. External packages register an
`install(app, config)` callable that runs once at proxy startup and is
free to add ASGI middleware, routes, mutate config, or raise to
fail-closed (e.g., license check failure aborts startup).
Why: Headroom OSS keeps a deliberately minimal surface, but several
high-value capabilities — PII redaction + tool-call vaulting, data-
residency routing, multi-tenant RBAC, compliance audit, vision/voice
privacy — only make sense as separately-distributed packages. A small,
stable plugin contract lets those live outside this repo without
requiring the OSS to know about them.
Contract:
[project.entry-points."headroom.proxy_extension"]
my_extension = "my_pkg.extension:install"
def install(app: FastAPI, config: ProxyConfig) -> None: ...
An extension that raises from install() is a deliberate fail-closed
signal and aborts startup. Entry-point load failures are logged and
skipped so one broken third-party package cannot take the proxy down.
Changes:
* New: headroom/proxy/extensions.py (~55 LOC)
* headroom/proxy/server.py: invoke install_all(app, config) in
create_app() immediately after CORS middleware registration
* pyproject.toml: register slow and real_llm pytest markers
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
