Commit graph

1553 commits

Author SHA1 Message Date
JerrettDavis
cc7ad822f4 test(dashboard): add playwright fixture and skip when not available
Add browser/page fixtures and importorskip guard so dashboard
E2E tests are skipped (not failed) when playwright is not installed.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-21 00:01:22 -05:00
JerrettDavis
0aae886f77 feat: add live transformations feed to dashboard
- New /transformations/feed endpoint returning message diffs
- Alpine.js drawer UI with virtual scrolling and auto-stream pause
- Live Feed button hidden when log_full_messages=false
- Added --log-messages CLI flag to enable full message logging
- Backend stores request/response messages when enabled

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-20 23:47:43 -05:00
JerrettDavis
8bf11d22ad ci: retry macos pytest install 2026-04-20 23:36:02 -05:00
JerrettDavis
f5dfdda253 ci: publish Python distributions to GitHub releases 2026-04-20 23:32:31 -05:00
Tejas Chopra
21d909a7ed
Merge pull request #210 from chopratejas/feat/bundled-cli-tools-and-interceptors
feat: bundle ast-grep/difftastic/scc + generic tool_result interceptor framework
2026-04-20 20:52:38 -07:00
JerrettDavis
c6c43f41f1 fix: skip commitlint for PR merge commits 2026-04-20 22:52:20 -05:00
JerrettDavis
ce405b0a5a Fix OpenClaw GPR package build 2026-04-20 22:44:11 -05:00
Tejas Chopra
8ba1ebd88e
Merge pull request #214 from adryanev/fix/proxy-followup-resilience-guards
Add timeout guards to Anthropic pre-upstream work
2026-04-20 20:26:41 -07:00
Tejas Chopra
16d460849b
Merge pull request #213 from JerrettDavis/ci-fork-release-publishing
fix: complete fork-friendly release publishing
2026-04-20 20:25:54 -07:00
chopratejas
504f20a1f7 fix(review-3): address third-pass PR #210 feedback
7 new "important" findings from the third review pass:

1. base.py:158 — JSONDecodeError on OpenAI tool_call arguments is now
   logged at DEBUG (was silently replaced with {}). Callers can now
   diagnose why range-key checks didn't fire on malformed calls.

2. base.py:234,248 — logger.warning for interceptor transform() and
   key() failures now passes `exc_info=True` so stack traces are
   preserved in logs.

3. base.py:306 — progressive disclosure now pre-seeds `fired` from the
   frozen prefix. A file first Read in the cached prefix no longer gets
   re-outlined when the model Reads it again in the mutable tail.
   apply_to_messages() now takes a `frozen_count` kwarg and handles the
   split internally; the Transform adapter simplifies.

4. astgrep.py:114 — broadened `except` around binaries.resolve() to
   catch the full BinaryError hierarchy plus KeyError + OSError. In
   offline mode this is the difference between a debug log and a
   warning on every single request.

5. astgrep.py:181 — chmod 0700 failure on the temp dir now logs at
   DEBUG. Silent swallow meant a multi-tenant host could leave
   untrusted content world-readable with no indication the hardening
   skipped.

6. cli/proxy.py:297 — explicit `--intercept-tool-results` now fails
   fast (`sys.exit(1)`) when the critical tool can't be installed.
   Previously it warned and started with non-functional interceptors.

7. pipeline.py:85 — interceptor gate now checks
   `HeadroomConfig.intercept_tool_results` first, env var second.
   Non-CLI callers (SDK, tests, embedded) can enable via config
   instead of having to touch os.environ. Added the config field with
   default False.

Plus a new test: `test_progressive_disclosure_respects_frozen_prefix_history`
proves a file Read in the frozen prefix blocks re-outline in the tail.

46 tests total, ruff + mypy clean.
2026-04-20 20:24:47 -07:00
Adryan Eka Vandra
7e53362e62
style: format files for CI 2026-04-21 09:49:26 +07:00
Adryan Eka Vandra
4e9348dec4
fix: add anthropic pre-upstream timeouts 2026-04-21 09:48:32 +07:00
chopratejas
a847387b09 chore: run ruff format on recently merged test files
Three test files merged in #196 via Pi/Codex route-aliases PR failed
ruff format --check, blocking CI on this PR. Auto-formatting them here
so the branch passes. No logic changes — only whitespace / trailing
comma normalization that ruff format applies.
2026-04-20 17:36:44 -07:00
chopratejas
78d935fcf9 fix: docker install fails with PermissionError in readonly cache dir
Two bugs collided to break the Docker-native install CI:

1. `ensure_tools()` ran unconditionally at every proxy startup, even when
   `--intercept-tool-results` was not passed. The feature is opt-in, so
   there's no reason to pay the binary-fetch cost (or risk a failure) when
   nothing will use them.

2. The fetch loop caught PlatformNotSupported / OfflineError /
   BinaryFetchError / Sha256Mismatch but not `PermissionError`. In
   containerized environments where the home dir / cache dir isn't
   writable, `binary_path.parent.mkdir()` raises PermissionError
   (subclass of OSError), which propagated out of ensure_tools() and
   crashed proxy startup.

Fixes:
- Move ensure_tools() inside the `if intercept_tool_results:` branch in
  cli/proxy.py so the base case never triggers a fetch.
- Catch OSError (covers PermissionError, ENOSPC, etc.) in ensure_tools()
  so sandboxed / readonly filesystems degrade to no-op instead of
  crashing. Interceptors fall back to pass-through when their tool isn't
  resolvable.

Adds regression test `test_ensure_tools_survives_readonly_cache_dir` that
points the cache at a chmod-0500 parent and asserts ensure_tools()
returns without raising.
2026-04-20 17:36:44 -07:00
chopratejas
e165a579f8 fix(review): address PR #210 feedback — race, query-params, SHA logging, frozen prefix, etc
Addresses all 24 inline comments across the two review passes.

**CRITICAL fixes:**
- binaries.py: PID-scoped partial-file name prevents concurrent `headroom proxy`
  starts from clobbering each other's downloads.
- binaries.py: strip URL query params before computing the download filename
  (was breaking archive-type detection for mirror URLs with `?token=...`).
- cli/tools.py: `--force` cleanup now logs failures and bumps exit_code
  instead of silently swallowing exceptions.

**HIGH fixes:**
- binaries.py: log at INFO when SHA256 is unpinned; expose `sha_pinned` in
  doctor's status output.
- proxy/interceptors/base.py: `_FAILURES` counter + `interceptor_failure_counts()`
  getter; incremented on every `matches()`/`transform()`/`key()` exception so
  dashboards can distinguish "nothing eligible" from "everything crashing".
- cli/proxy.py: validate critical tools resolved when
  `--intercept-tool-results` is set; warn (don't fail) if a dependency is
  missing.
- proxy/interceptors/base.py: compute `tokens_before` from the original
  messages via `count_messages()` instead of back-calculating from
  `tokens_after + sum(saved)` (which double-counted message-level overhead).
- proxy/interceptors/astgrep.py: write untrusted tool_output into a private
  mode-0700 `tempfile.mkdtemp()` directory, not directly into shared `/tmp`.
- proxy/interceptors/base.py: `ToolResultInterceptorTransform.apply()` now
  honors `frozen_message_count` — leading cached-prefix messages are passed
  through untouched to preserve provider prefix caches.

**MEDIUM fixes:**
- proxy/interceptors/base.py: pre-built O(1) tool_use index replaces the
  O(n²) per-tool-result linear scan.
- proxy/interceptors/base.py: broken `progressive_disclosure_key()` now
  skips the interceptor entirely rather than firing without key protection.
- proxy/interceptors/astgrep.py: distinguish ast-grep rc=1 (no matches) from
  rc>=2 (real errors — bad syntax, missing grammar, corrupt binary).
- proxy/interceptors/astgrep.py: count JSON parse failures; warn when all
  lines fail to parse (indicates version mismatch).
- binaries.py: musl detection falls back to checking `/lib/ld-musl-*.so.1`
  when `ldd` is absent (Alpine).
- proxy/interceptors/astgrep.py: use `tempfile.mkdtemp()` + `shutil.rmtree`
  instead of `NamedTemporaryFile(delete=False)`; cleans up on Windows.
- binaries.py: chmod failures on POSIX now log a warning (only swallow on
  Windows where .exe is implicitly executable).
- tests/test_binaries.py: `test_mirror_substitution` now uses
  `monkeypatch.setenv()` instead of raw `os.environ` manipulation.
- tools.json: add `linux-x86_64-musl` and `linux-aarch64-musl` entries for
  `difft`; document the shared-asset strategy for both tools.
- proxy/interceptors/astgrep.py: log a debug line when
  `progressive_disclosure_key()` returns None for a tool whose tool_input
  shape we don't recognize.
- proxy/interceptors/base.py: moved `import json` to module top (was inside
  `_find_tool_use` hot loop).
- binaries.py: fix bare `.gz` detection — now explicitly excludes
  `.tar.gz`/`.tgz` instead of relying on a brittle "no dots" heuristic.
- proxy/interceptors/astgrep.py: provenance comment on each `_RANGE_KEYS`
  entry so future maintainers know which tool defined which key.
- cli/tools.py: comment explaining os.execv's lack of Python finalizer
  cleanup.
- proxy/interceptors/base.py: `InterceptionResult` now `frozen=True`.

**Test gaps closed:**
- Interceptor failure isolation (transform() raises → request survives,
  counter increments).
- Broken key() skips interceptor entirely.
- Refuse-to-enlarge guard (rewrite larger than original → pass through).
- Orphaned tool_result (no matching tool_use) doesn't crash.
- ToolResultInterceptorTransform.apply() happy path + frozen_message_count.
- ensure_tools() partial failure (one tool fetch fails, others succeed,
  proxy still starts).
- Mirror URL with query params doesn't leak into download filename.

44 tests total; ruff + mypy clean.
2026-04-20 17:36:44 -07:00
chopratejas
7cd67ef06d fix: bundle ast-grep/difftastic/scc + generic tool_result interceptor framework
What this does, in plain terms:

Headroom's proxy now ships with three CLI tools (ast-grep, difftastic,
scc) that it can use to shrink tool_result payloads before they reach
the model. The goal is simple: when Claude Code (or Codex, Aider, etc.)
asks the model to reason about a big file or diff, we swap the verbose
output for a compact, same-meaning version. Fewer tokens per turn, same
answers, lower bill.

Today a single interceptor is wired: ast-grep on Read. When an agent
reads a large code file, the proxy replaces the file body with an
outline of its top-level functions/classes plus docstrings. In live
tests that cut prompt tokens 74–76% on both OpenAI and Anthropic,
same answer either way.

How it works:
- `pip install headroom-ai` now installs ast-grep via a PyPI wheel
  (core dep). difftastic and scc are fetched once at proxy startup
  from pinned upstream GitHub releases and cached per-user.
- A generic registry (`headroom/proxy/interceptors/`) lets us add more
  tool-aware rewrites in one file each: declare `matches()` and
  `transform()`, call `register()`, done. No proxy or metrics plumbing
  per tool.
- Safety rails built in: pass-through when a Read specifies a line
  range; second Read of the same file in a conversation returns full
  content (progressive disclosure); any failing interceptor logs and
  skips, never crashes a request.

Opt-in for now:
- Off by default while this ships. Turn on with
  `headroom proxy --intercept-tool-results` or
  `HEADROOM_INTERCEPT_ENABLED=1`, so we can measure before flipping
  defaults.

What users see after turning it on:
- First `headroom wrap claude` boot is ~5s longer (binaries fetched).
  Every subsequent run is cache-only.
- Existing `transforms_applied` field in metrics gets entries like
  `interceptor:ast-grep`, so savings show up in current dashboards
  and HTML reports with no UI change.

Other housekeeping in this PR:
- uv.lock moved to .gitignore — regenerated locally per environment.
- 35 unit + integration tests, ruff + mypy clean.
- Dead-code audit done: removed `binaries.run()`, `needs_filesystem`
  plumbing, unused `_kind` tuple elements, unused `tool_output`
  parameter, and the never-set HEADROOM_SKIP_TOOLS_BOOTSTRAP env.
2026-04-20 17:36:44 -07:00
JerrettDavis
c788b7d6c9 style: align proxy tests with current ruff formatter
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-20 19:19:43 -05:00
JerrettDavis
42f1d78612 fix: publish fork release assets and packages
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-20 18:50:23 -05:00
chopratejas
5391761fe6 chore(memory): add EXTERNAL backend extension points
Mirrors the pattern used by headroom.ccr_backend so memory store,
vector index, and text index backends can be registered via setuptools
entry points.

- EXTERNAL enum value on StoreBackend, VectorBackend, TextBackend
- Optional *_backend_name fields on MemoryConfig
- entry_points(group=...) lookup in _create_{store,vector_index,text_index}
- New test_factory_external.py (7 tests) covering load / missing-name /
  unknown-name paths

Default behavior (SQLITE + AUTO + FTS5) unchanged.

Extension groups:
  headroom.memory_store
  headroom.memory_vector
  headroom.memory_text
2026-04-20 16:42:10 -07:00
Kayzo
fbbde51db1 fix(onnx): reduce retained cpu memory 2026-04-20 22:31:48 +00:00
Kayzo
6690740218 chore(merge): resolve upstream main conflicts for pi codex branch 2026-04-20 21:50:33 +00:00
JerrettDavis
0d48d4cd58 test(pr208): add coverage for is_anthropic_auth, file logging, repro helpers, and wrap paths
Boosts patch coverage by testing uncovered code paths in the PR diff:
- is_anthropic_auth header detection (6 cases)
- _setup_file_logging with new _headroom_log_dir path + OSError handling
- repro_codex_replay: _http_to_ws_url, _classify_exit, format_summary, build_parser
- CodexWsStats.record_error and AnthropicHttpStats dataclasses
- LatencyHistogram percentile boundary conditions
- wrap.py _get_log_path using headroom.paths module

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-20 14:13:45 -05:00
JerrettDavis
eb83f403d7 test: add coverage for PR #208 changes
Tests for jitter_delay_ms helper, asyncio.timeout shim, release workflow
--allow-same-version flag validation, SIGKILL fallback on Windows, and
LatencyHistogram from the repro harness script.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-20 13:57:09 -05:00
JerrettDavis
6200ba3963 style: fix ruff lint/format and mypy errors for CI 3.12 pass
- Remove 101 trailing whitespace violations (W293), 2 unused imports
  (F401), 1 import sort issue (I001) in anthropic handler and test file
- Apply ruff format to anthropic handler and oauth routing test
- Fix mypy no-any-return in jitter_delay_ms by adding explicit type
- Fix mypy attr-defined for signal.SIGKILL on Windows using getattr

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-20 13:23:20 -05:00
Tejas Chopra
cd11acc3c4
Merge pull request #197 from adryanev/fix/responses-retries-keep-compression
fix(proxy): Codex reconnect-storm resilience — bounded pre-upstream + WS session tracking + stage timings
2026-04-20 09:50:37 -07:00
Tejas Chopra
eff161615f
Merge pull request #199 from JerrettDavis/fix/release-publish-artifacts
fix: publish docker and package artifacts in release flow
2026-04-20 09:50:11 -07:00
Adryan Eka Vandra
b71b659e1b
fix(ci): restore repro harness test in dev installs
Add websockets to the dev extra so the repro harness smoke test can import
its websocket client dependency in the CI test matrix. Also apply ruff
formatting to the files the formatter check was rejecting so the 3.12 lint
job passes.
2026-04-20 22:12:01 +07:00
Adryan Eka Vandra
6354105146
fix(proxy): let HTTPException propagate past Anthropic catch-all
Commit 62d0a50 widened the outer try/except around
handle_anthropic_messages to guarantee semaphore release on every
path. Because fastapi.HTTPException inherits from Exception, the
existing catch-all at line 1614 began intercepting the 429
rate-limit and 429 budget raises (and any other HTTPException),
turning them into opaque 502 responses with no Retry-After.

Add an explicit 'except HTTPException: raise' ahead of the
Exception handler so FastAPI's own handler still produces the
correct status code and headers. The outer finally still runs,
so _finalize_pre_upstream() and the semaphore release are
unaffected.

Also tighten the contention test: it previously caught any
exception with 'except Exception: pass', which masked this
regression. The new assertions pin each scenario to its documented
contract (HTTPException(429) for rate-limiter/cost-tracker;
response object for security/cache) so a future regression of
this shape fails loudly.
2026-04-20 22:09:57 +07:00
Adryan Eka Vandra
8bbf3afda0
perf(proxy): gate /debug/tasks stack-depth computation behind query param
Task.get_stack(limit=32) walks coroutine frames synchronously and
measurably stalls the event loop when called for 50+ relay tasks during
a reconnect storm. The /debug/tasks snapshot does not need this by
default — the perf-cheap fields (name, coro_qualname, age, done) are
enough for the common case of "which tasks are alive".

- collect_tasks: add with_stack_depth=False kwarg; default leaves
  stack_depth=None per entry.
- /debug/tasks: read ?stack=true query parameter and forward it. Default
  response is cheap; opt-in explicitly when human-debugging one snapshot.
- Test: verify default entries have stack_depth=None and that
  ?stack=true produces at least one integer depth.
2026-04-20 22:09:24 +07:00
Adryan Eka Vandra
c438268d8a
feat(scripts): improve repro harness CLI contract and backoff
- Route --json output cleanly: JSON goes to stdout (machine-readable),
  human-readable summary goes to stderr. Without --json, the human
  summary stays on stdout as before. Makes piping the harness into
  jq / other tools trivial without losing operator visibility.
- Add distinct exit codes so CI and shell wrappers can branch on the
  failure class: 0 success, 1 crash, 2 proxy_unreachable, 3 livez
  threshold exceeded, 4 warmup failed, 130 SIGINT (already correct).
  Smoke test updated to assert EXIT_PROXY_UNREACHABLE instead of 1.
- Replace flat 50-250ms jitter in the Anthropic-client retry loop with
  exponential backoff + 50-150% jitter
  (base=250ms, max=5000ms, attempt counter). Matches the proxy's own
  jitter_delay_ms helper; inlined to keep the script free of proxy
  package imports.
2026-04-20 22:09:23 +07:00
Adryan Eka Vandra
55f59fad77
chore(proxy): announce pre-upstream concurrency; clean dead RequestLog fields and debug helpers
- Log the resolved Anthropic pre-upstream concurrency at startup so
  operators can correlate pre_upstream_wait_ms lines with the configured
  cap. Distinguishes auto-detected vs. explicit vs. unbounded.
- Remove the stage_timings, session_id, and stage_timings_path fields
  from RequestLog. They were declared optional but never populated at
  any of the three RequestLog construction sites — stage timings flow
  exclusively through emit_stage_timings_log (structured log line) and
  Prometheus histograms. Keeping unpopulated fields misled readers.
  Option (A) from the review.
- Inline the one-liner collect_ws_sessions / collect_warmup pass-throughs
  at their two callers in server.py; keep collect_tasks since it has
  real logic. Unit tests were rewritten against the registry's own
  snapshot()/to_dict() serializers.
2026-04-20 22:09:23 +07:00
Adryan Eka Vandra
5312991f76
test(proxy): cover semaphore release on Anthropic early-exit paths
Parametrized tests for the 4 pre-upstream early-return paths that bail
out of handle_anthropic_messages before reaching the upstream call:
  - rate_limiter denies (429)
  - cost_tracker blocks (429)
  - security scan blocks (403)
  - cache hit (200 — skips upstream entirely)

Each test holds a Semaphore(1) across multiple calls to confirm the
handler restores the original _value on every path, catching any
regression that forgets to call _emit_pre_upstream_stage_timings()
before the HTTPException / early return.
2026-04-20 22:08:10 +07:00
Adryan Eka Vandra
a4345f1cce
perf(proxy): reduce lock contention on stage-timing metrics
record_stage_timings() acquired the global asyncio.Lock on every
request finalization — the same lock held by export() during Prometheus
scrapes, which does string-building while holding it. Under N concurrent
request finalizations + an active scrape, all N queued behind the scrape.

Move the stage-timing triple update (sum + count + max) onto a tiny
synchronous threading.Lock. This keeps the multi-field update consistent
for scrapers without contending on the async lock. export() now snapshots
the three dicts under the sync lock, then builds the metrics string from
the snapshots outside any lock scope.
2026-04-20 22:08:09 +07:00
Adryan Eka Vandra
196d8d69a7
fix(proxy): propagate CancelledError from memory init cleanly
External cancellation while _ensure_initialized is in flight raises
CancelledError (BaseException). The existing 'except asyncio.TimeoutError:'
branch does not catch it, and caller 'except Exception:' blocks don't
either, so it propagated as-is with _backend possibly still assigned.

Add an explicit 'except asyncio.CancelledError:' handler that nulls
_backend, clears _initialized, logs at info, and re-raises. Cancellation
is a shutdown signal, not an error to swallow — but the state must be
clean so a later retry starts fresh.
2026-04-20 22:08:09 +07:00
Adryan Eka Vandra
3f1f82d7e2
fix(proxy): reset memory backend to None on init timeout
If asyncio.wait_for fires while LocalBackend._ensure_initialized() is
running, _init_backend_locked may have already assigned self._backend
(the LocalBackend constructor) before its own await raised or was
cancelled. Callers doing if self.memory_handler._backend: then see
a truthy-but-broken backend.

Null self._backend in the TimeoutError handler so the post-timeout
state is consistent: _initialized=False AND _backend=None.
2026-04-20 22:08:09 +07:00
Adryan Eka Vandra
0e166c60d4
fix(proxy): accept IPv6-mapped IPv4 loopback in debug guard
On Linux dual-stack sockets (IPV6_V6ONLY=0 default), an IPv4 loopback
connection arrives as ::ffff:127.0.0.1 and was 404'd by the literal set
check — silent outage of /debug/* when the proxy binds to :: or 0.0.0.0.

Replace the set-membership check with ipaddress.ip_address(host).is_loopback,
special-casing 'localhost' (not an IP literal) and treating ValueError
(malformed input) as non-loopback. The None sentinel for TestClient / UDS
sockets is preserved.
2026-04-20 22:08:09 +07:00
Adryan Eka Vandra
548c69a1f9
fix(subscription): stop leaking Event.wait tasks from poll loops
Both SubscriptionTracker._poll_loop and _CopilotQuotaTracker._poll_loop
wrapped their stop-event wait in asyncio.shield() inside wait_for().
When wait_for timed out (every poll_interval_s), it cancelled its own
outer task but the shielded inner Event.wait kept running forever —
one leaked Task per poll interval per tracker.

Across the two default trackers (10s poll each) this leaks ~0.2
tasks/sec in steady state: ~17k/day, ~120k/week. Event-loop
scheduler cost grows linearly with task count, which eventually
starves /livez and new WS accepts on long-lived processes. This
matches the 'aged :8787 proxy degrades over hours/days' symptom
captured in wiki/plans/2026-04-17-codex-proxy-runtime-analysis.md.

Discovered by the /debug/tasks endpoint added in Unit 5 of the
codex-proxy-resilience plan — 60s of idle time on a fixed-version
fork now shows 0 Event.wait tasks vs 310 → 320 growth on the
unpatched build.

Drop the shield wrapper so wait_for can cleanly cancel the inner
wait when its timeout fires. The stop() contract is unaffected:
setting _stop_event still returns the wait normally before the
timeout, triggering the break.

Adds a regression test per tracker that starts the poll loop with
a 50ms interval, lets it run for ~6 cycles, stops it, and asserts
Event.wait task count does not grow beyond baseline + 1 (the one
legitimate in-flight waiter).
2026-04-20 22:02:41 +07:00
Adryan Eka Vandra
30e7b7c498
test(scripts): restore real websockets module before repro smoke test
Some earlier tests in the suite replace sys.modules['websockets'] with
a stub. When the smoke test ran after those, uvicorn's
websockets-sansio backend tried to import websockets.server at connect
time, resolved through the stub, and failed with
'No module named websockets.server; websockets is not a package',
preventing the mock proxy from starting.

Add an autouse fixture that drops stub websockets.* entries from
sys.modules, re-imports the real package + websockets.asyncio.server,
and restores prior state on teardown. Keeps the smoke test
deterministic regardless of collection order.
2026-04-20 22:02:40 +07:00
Adryan Eka Vandra
ea0d519677
fix(tests): pin repro smoke server to websockets-sansio backend
uvicorn's auto-select imports websockets_impl.py which requires the
legacy API (websockets<14). With websockets 16.0 installed in the
local venv, auto-select errored before its fallback could trigger,
failing the smoke test with 'No module named websockets.legacy'.

Explicitly selecting the websockets-sansio backend (shipped with
uvicorn) works against websockets 16.0 and keeps the smoke test
representative of the real production WS path.
2026-04-20 22:02:40 +07:00
Adryan Eka Vandra
c482d75a73
feat(scripts): add Codex proxy reconnect-storm repro harness
Reproducibly exercises the multi-agent Codex reconnect/retry storm
(origin §"Latest Correction") against a local proxy: concurrent Codex WS
sessions + parallel Anthropic /v1/messages replays, with /livez probed
every 250ms to detect event-loop starvation. Exits non-zero if /livez
p99 exceeds the configured threshold.

- scripts/repro_codex_replay.py — CLI harness (asyncio + websockets +
  httpx, no new pip deps).
- scripts/fixtures/{anthropic_replay_body,codex_response_create_frame}.json
  — hand-crafted, fully synthetic fixtures shaped like real traffic.
- scripts/README.md — "Reproducing the reconnect storm" section.
- tests/test_scripts/test_repro_codex_replay_smoke.py — FastAPI/uvicorn
  mock server; runs the harness end-to-end in < 3s.
- .gitignore — allowlist the new checked-in scripts (existing rule
  previously ignored everything under scripts/ except install.sh/.ps1).
2026-04-20 22:02:02 +07:00
Adryan Eka Vandra
a4010aaa1b
feat(proxy): add loopback-only debug introspection endpoints
Unit 5 of the Codex-proxy resilience plan. Three always-on
/debug/* endpoints make "what is this process doing right now?"
a single curl away:

- GET /debug/tasks serializes asyncio.all_tasks() to name,
  coro_qualname, stack_depth, done, and age_seconds (derived from
  the WS session registry for Codex relay tasks, null otherwise).
  Sorted by age desc. No frame locals, no request bodies, no
  coroutine args.
- GET /debug/ws-sessions returns WebSocketSessionRegistry.snapshot().
- GET /debug/warmup returns WarmupRegistry.to_dict().

A FastAPI Depends(require_loopback) dependency 404s (not 403) any
non-loopback caller so debug routes stay invisible to external
scanners. Dependency chosen over middleware because the route set
is small and explicit Depends(...) makes the guard visible to
reviewers and easy to override in tests.
2026-04-20 22:01:42 +07:00
Adryan Eka Vandra
1de753bd20
feat(proxy): bound pre-upstream Anthropic work with a semaphore
Unit 4 of the Codex proxy resilience plan. Cold-start Anthropic replay
storms (many concurrent large POSTs immediately after restart) could
occupy every event-loop slot and thread-pool worker with deep-copy /
compression / memory-context work before any upstream connect, starving
``/livez`` and new Codex WS opens.

Gate the pre-upstream phase of ``handle_anthropic_messages`` with an
``asyncio.Semaphore`` constructed once per process. The semaphore is
acquired right after stage-timer setup and released before streaming
response bytes back to the client — it never spans the whole response
lifetime. Release is also driven from ``_emit_pre_upstream_stage_timings``
(already called at every known exit point), with explicit releases on
rate-limit / budget HTTPException, cache hit, security block, and
Bedrock errors.

Config: ``ProxyConfig.anthropic_pre_upstream_concurrency`` (None =>
auto-compute ``max(2, min(8, os.cpu_count() or 4))``; 0 or negative =>
disables the semaphore for the Unit 6 counter-factual).
CLI: ``--anthropic-pre-upstream-concurrency``.
Env: ``HEADROOM_ANTHROPIC_PRE_UPSTREAM_CONCURRENCY``.
Precedence: CLI > env > auto-compute.

Observability: a new ``pre_upstream_wait`` stage is recorded on the
existing ``StageTimer`` and surfaces in ``/metrics`` and stage-timings
log lines. Waits > 100ms emit an INFO log line with request_id +
session_id + wait ms so queueing is visible under load.

Compression stays enabled. Codex WS, OpenAI HTTP, ``/livez``,
``/readyz``, ``/health``, ``/metrics``, ``/stats`` are unchanged.
2026-04-20 22:01:42 +07:00
Adryan Eka Vandra
a31d81a426
feat(proxy): track Codex WS sessions and cancel relay tasks deterministically
Unit 3 of the Codex proxy resilience plan. Eliminates the "aged process
has leaked relay tasks" hypothesis by making every WS session explicitly
tracked and both relay tasks deterministically cancelled when either
exits.

- New headroom/proxy/ws_session_registry.py: dict-backed
  WebSocketSessionRegistry + WSSessionHandle with register /
  deregister / attach_tasks / snapshot. Deregister is idempotent and
  clears relay-task references so coroutine frames are not retained
  past session end.
- HeadroomProxy exposes proxy.ws_sessions so /debug/ws-sessions
  (Unit 5) can read the live snapshot.
- handle_openai_responses_ws now registers on websocket.accept()
  success and deregisters in the outermost finally so no leak can
  survive handshake-phase, mid-stream, or upstream-error paths. The
  session_id / termination_cause is threaded through both relay
  halves and both sides raise asyncio.CancelledError cleanly.
- Replaced asyncio.gather(_client_to_upstream(), _upstream_to_client(),
  return_exceptions=True) with explicit asyncio.create_task(...)
  (named codex-ws-c2u-<sid> / codex-ws-u2c-<sid>) +
  asyncio.wait(FIRST_COMPLETED) + cancel-and-await on the survivor.
  Termination cause is classified as client_disconnect /
  client_error / upstream_disconnect / upstream_error /
  response_completed from which task completed first plus inline
  error captures from the halves.
- Prometheus metrics: new active_ws_sessions and active_relay_tasks
  gauges plus ws_session_duration_ms_{sum,count,max} histogram
  bucketed by termination cause. Mirrors the Unit 2 stage_timing_*
  shape.

Preserved: upstream WS retry loop, WS→HTTP fallback, memory-context
timeout, compression pipeline, Unit 2 stage timings. Memory-tool
execution inside _upstream_to_client still runs when the client task
exits first; however, if the client disconnects *before* the upstream
emits response.completed, pending memory writes in `pending_fcs` are
dropped (unchanged from prior behavior — a crashing upstream has the
same effect). Note: handle_openai_responses (HTTP, line ~800) is a
single-shot HTTP request; lifecycle tracking isn't added there
(scope boundary).

Tests:
- tests/test_ws_session_registry.py: 8 registry unit tests
  (register/deregister idempotency, snapshot shape, attach merging,
  reference release).
- tests/test_openai_codex_ws_lifecycle.py: 6 integration tests
  using real relay tasks (only upstream WS endpoint mocked):
  happy-path, failing-test-first "client disconnect cancels upstream
  relay within 100 ms", upstream-closes-first, upstream-error mid-
  stream, handshake-failure deregister, 50 concurrent sessions.
- Regression: test_openai_codex_ws_timings, test_openai_codex_routing,
  test_proxy_codex_route_aliases, test_ws_memory_relay all pass.
- Tests pass under python -W error::RuntimeWarning (no "coroutine
  was never awaited").
2026-04-20 22:01:41 +07:00
Adryan Eka Vandra
7e2b078b76
feat(proxy): share cold-start warmup across pipelines + lock memory init
Unit 1 of the codex-proxy-resilience plan. Preload now iterates BOTH
the Anthropic and OpenAI transform pipelines, dedupes shared transforms
by id(), and merges the status dict into a typed WarmupRegistry so
/debug/warmup (Unit 5) and /readyz have one source of truth.

MemoryHandler._ensure_initialized gains an asyncio.Lock with a
double-checked pattern so concurrent first callers share one backend
init instead of racing. The body is wrapped in asyncio.wait_for with
STARTUP_INIT_TIMEOUT_SECONDS (default 30s) — on timeout _initialized
stays False so subsequent requests retry (fail-open contract).

Startup now also forces one embedder warm-up encode so the ONNX graph
is compiled synchronously at startup instead of lazily on the first
request. Best-effort: failures are logged, never raised.
2026-04-20 22:01:41 +07:00
Adryan Eka Vandra
c2fee2f706
feat(proxy): add per-stage timings for Codex WS and Anthropic HTTP paths
Unit 2 of the Codex proxy resilience plan. Introduce a lightweight
StageTimer utility and thread it through handle_openai_responses_ws and
handle_anthropic_messages to capture per-stage durations
(accept/first_client_frame/upstream_connect/upstream_first_event/
memory_context/compression/total_session for Codex WS;
read_request_json/deep_copy/compression_first_stage/memory_context/
upstream_connect/upstream_first_byte/total_pre_upstream for Anthropic
HTTP).

Each request/session emits exactly one structured log line
(STAGE_TIMINGS ...) and records sum/count/max Prometheus histograms
under headroom_stage_timing_ms_{sum,count,max} keyed by
(path, stage). A new session_id UUID is generated per request/session
and paired with the existing request_id so multi-turn sessions are
correlatable.

Compression, WS retries, WS->HTTP fallback, and the memory-context
fail-open timeout are unchanged. RequestLog gains three optional
backward-compatible fields (stage_timings, session_id,
stage_timings_path).
2026-04-20 22:01:41 +07:00
Adryan Eka Vandra
b0422cd27a
fix(proxy): retry responses transport without bypassing compression 2026-04-20 22:01:39 +07:00
Tuan Dinh
ddb20d9fde
fix: support OAuth Bearer token routing for proxy endpoints
Bearer-only auth (no x-api-key) was misrouted at /v1/models and
catch-all endpoints because routing only checked x-api-key header.
Claude Code users authenticating via ANTHROPIC_AUTH_TOKEN (OAuth)
send Authorization: Bearer instead of x-api-key.

- Add is_anthropic_auth() helper detecting x-api-key, anthropic-version,
  or Bearer sk-ant-* tokens
- Fix /v1/models routing to use consolidated auth detection
- Fix catch-all /{path:path} routing to recognize Bearer tokens
- Use Bearer token prefix for rate-limit key when x-api-key absent
- Add 19 tests covering auth detection, routing, and rate-limit keys

Closes chopratejas/headroom#200
2026-04-19 12:50:09 +07:00
JerrettDavis
3ddc7ff33a fix: restore release and compress regressions
Fix workflow validation failures by wiring detect-version outputs into all
release publish jobs, renaming the GitHub Packages skip variable to a
valid Actions variable name, and adjusting the macOS PATH export for
actionlint.

Also make min_tokens_to_compress use token counting instead of whitespace
splits so compact JSON tool outputs still compress after merging the
latest main branch changes, and add a regression test for that path.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-18 16:01:57 -05:00
Kayzo
9f0a813004 fix: address PR review feedback 2026-04-17 23:09:12 +00:00
Kayzo
4bdc1482af feat: add Pi/Codex and Cloud Code Assist compatibility routes
- Adds /v1/codex/responses aliases for OpenAI Codex clients configured with /v1 base URLs
- Uses JWT-derived account routing for /v1/responses/* subpaths (compact, cancel, etc)
- Adds /v1internal:streamGenerateContent aliases for Cloud Code Assist / Antigravity
- Preserves upstream HTTP error status/body in StreamingResponse (fixes empty SSE drops)
2026-04-17 17:45:08 +00:00