diff --git a/dashboard-cache-ttl-main.png b/dashboard-cache-ttl-main.png new file mode 100644 index 000000000..3433b1c56 Binary files /dev/null and b/dashboard-cache-ttl-main.png differ diff --git a/docs/context-mode-integration-analysis.md b/docs/context-mode-integration-analysis.md new file mode 100644 index 000000000..fc21c7bf1 --- /dev/null +++ b/docs/context-mode-integration-analysis.md @@ -0,0 +1,378 @@ +# context-mode → Headroom: enterprise plugin & variant analysis + +Analysis date: 2026-07-29. Sources: `/Users/tcms/demo/context-mode` @ v1.0.169, `/Users/tcms/demo/headroom` @ main. + +--- + +## 1. Bottom line + +context-mode and Headroom attack the same cost problem at **two different layers**, and they do not +overlap where it matters: + +| | context-mode | Headroom | +|---|---|---| +| Interception point | agent **tool-call boundary** (host hooks + MCP) | model **API boundary** (proxy / SDK / MCP) | +| Position relative to context | **pre-context** — data never enters | **in-context** — data already entered, gets squeezed | +| Mechanism | admission control: block, redirect, sandbox, externalize | compression: crush, cache, retrieve | +| Touches the wire request | never | always | +| Loss | lossless (full content in FTS5, queryable) | lossy squeeze + hash rehydrate | + +Headroom's own realignment doc identifies its correct compression target as the **live zone**: +"latest user message content + latest `tool_result` + latest `function_call_output` + latest +`local_shell_call_output`" (`REALIGNMENT/00-overview.md`, Phase B). + +**That is precisely the payload context-mode intercepts one layer earlier.** Headroom Phase B is +building a Rust engine to compress the latest tool result *after* it hits the wire. context-mode +stops that tool result from being produced at all. These are complements, not competitors — and the +upstream position is strictly cheaper: nothing to compress, nothing to cache-invalidate, no +token-validation fallback needed. + +Three strategic unlocks, in order of value: + +1. **Cache safety.** Headroom's #1 identified bug class is prompt-cache busting from request + mutation (5 top-tier cache-killer bugs, `REALIGNMENT/00-overview.md`). context-mode has + *structurally zero* cache-bust risk because it never touches the request body. +2. **Subscription safety.** The realignment flags "fingerprint-class subscription-revocation + risks" from `X-Headroom-*` header leakage, `anthropic-beta` mutation and re-serialization on + OAuth/subscription CLIs. A hook-layer product carries none of this — it is invisible to the + upstream. This is a *deployable-where-the-proxy-can't-go* capability. +3. **Proxy-free deployment.** Headroom's value today requires being in the API path + (`127.0.0.1:8787`). Verified live this session: with the proxy down, `headroom_stats` returns all + zeros and `headroom_compress` no-ops. Enterprises that cannot reroute model traffic (TLS trust, + egress policy, subscription auth) currently get nothing. context-mode's hook+MCP model needs no + interposition. + +Zero references to context-mode exist in the Headroom tree today — clean slate. + +--- + +## 2. context-mode: portable IP inventory + +41,617 lines of TypeScript, 11 MCP tools, 18 host adapters, npm-distributed +(`context-mode@1.0.169`, 8 runtime deps, esbuild-bundled). + +Ranked by *how hard it would be for Headroom to rebuild*: + +### Tier 1 — genuinely hard, no Headroom equivalent + +**1. Cross-host hook adapter layer** — `src/adapters/**` (~10K LOC), `src/adapters/types.ts`, +`src/adapters/detect.ts` (737 lines), `configs/` (18 hosts). +Normalizes three incompatible paradigms — `json-stdio` (Claude Code, Gemini/Qwen, Copilot, Codex, +Kimi, Cursor, Kiro, Antigravity), `ts-plugin` (OpenCode, KiloCode, OpenClaw), `mcp-only` (Zed, Pi, +OMP) — behind one contract: normalized `PreToolUse` / `PostToolUse` / `PreCompact` / +`SessionStart` events, a `PlatformCapabilities` matrix, and a 5-way decision +(`allow | deny | modify | context | ask`). Per-host install, config-format, and self-heal machinery +included (`hooks/heal-partial-install.mjs`, `scripts/plugin-cache-integrity.mjs`). +*Why hard to rebuild:* the value is entirely in the accumulated per-host quirks. There is no spec to +implement against. + +**2. Tool-boundary policy engine** — `src/security.ts` (889 lines). +A real policy decision point, not a regex list: glob→regex compilation, chained-command splitting +(`&&`/`;`/`|` with escape awareness), subshell extraction, deny/ask pattern ingestion from host +settings files, project-boundary containment (`evaluateProjectContainment` — Issue #852: an approved +`ctx_execute_file` cannot escape the repo via a path the user couldn't see), and a +**shell-escape scanner** (`SHELL_ESCAPE_PATTERNS`, `extractShellCommands`) that detects +`execSync`/`subprocess`/etc. embedded inside sandboxed *non-shell* code and re-evaluates the escaped +command against policy. +*Why hard to rebuild:* this is the sandbox-escape prevention layer. Getting it wrong is a CVE. + +**3. Multi-language sandbox executor** — `src/executor.ts` (785), `src/runPool.ts`, +`src/exit-classify.ts`, `src/truncate.ts`. +12 languages, stdout-only egress, timeouts, background detach, output caps, exit classification. +Enforces the "Think in Code" contract: the agent programs the analysis, only the answer enters +context. + +**4. Lossless externalization store** — `src/store.ts` (2,071 lines). +Dual SQLite FTS5 index — a tokenized `chunks` table *plus* a `chunks_trigram` table for +substring/identifier search where BM25 tokenization fails on code — with a `vocabulary` table and +schema migration path. Auto-externalizes any output >100 KB into FTS5 and returns a pointer. +Nothing is discarded; the model queries on demand. + +### Tier 2 — valuable, but partially duplicated in Headroom + +**5. Counterfactual savings accounting** — `src/session/analytics.ts` (3,085 lines), +`src/session/project-attribution.ts`, `src/session/db.ts` (1,726). +`ContextSavings`, `ThinkInCodeComparison`, `RealBytesStats`, `MultiAdapterLifetimeStats`, +`enumerateAdapterDirs()`. Measures *what would have entered context but didn't* — a different and +harder quantity than Headroom's `savings_ledger.py`, which records actual compression deltas. +Session event ledger + `tool_calls` + resume + per-project attribution. + +**6. Multi-vendor pricing catalog** — `src/session/pricing.ts` + `model-prices.json`. +61 curated models × 4 rate buckets (input / output / cache-read / cache-write), refreshed from +litellm, unknown model → `null` rather than a silently wrong Claude rate. +**Overlaps `headroom/pricing/*` heavily. Do not port.** + +### Tier 3 — do not port + +Compression heuristics, memory/graph/relevance, telemetry transport, dashboard, install UX, +update-check. Headroom has all of these, more mature, and Phase B/H is actively consolidating them. + +--- + +## 3. Headroom's actual extension seams + +Verified entry-point groups (all `importlib.metadata`-discovered, all opt-in): + +| Seam | Group | Contract | Source | +|---|---|---|---| +| Proxy extension | `headroom.proxy_extension` | `install(app: FastAPI, config: ProxyConfig) -> None` | `headroom/proxy/extensions.py:52` | +| Pipeline extension | `headroom.pipeline_extension` | `on_pipeline_event(PipelineEvent) -> PipelineEvent \| None` over 11 stages | `headroom/pipeline.py:13,68` | +| Learn plugin | `headroom.learn_plugin` | — | `headroom/learn/registry.py:44` | +| Memory text store | `headroom.memory_text` | — | `headroom/memory/config.py:41`, `factory.py:57` | +| Memory vector store | `headroom.memory_vector` | — | `headroom/memory/config.py:34` | +| Memory store | `headroom.memory_store` | — | `headroom/memory/config.py:25` | +| CCR backend | `headroom.ccr_backend` | — | `headroom/cache/compression_store.py:981` | +| Compression hooks | (subclass, not entry point) | `pre_compress` / `compute_biases` / `post_compress` | `headroom/hooks.py:1-31` | + +Two things worth noting: + +- `headroom/proxy/extensions.py:32` states an explicit **stability contract**: changing + `install(app, config)` or the group name requires a deprecation cycle. This is a supported public + seam, not an accident. +- `headroom/hooks.py:16` says outright: *"Headroom SaaS implements position-aware compression and + cross-turn deduplication via these hooks."* The open-core split is already designed in. + +**The exemplar to copy:** `plugins/headroom-oauth2/` — own `pyproject.toml`, own `LICENSE`, own +`SPEC.md`, registers on `headroom.proxy_extension`, dormant until `--proxy-extension oauth2`, +all config via env, "zero core changes." That is the enterprise plugin template. + +**The precedent to copy:** `headroom/lean_ctx/installer.py` and `headroom/rtk/installer.py` — +Headroom already ships thin installers that adopt sibling products. `plugins/headroom-agent-hooks` +already installs startup hooks into Claude Code and Copilot CLI. The socket exists. + +**The gap:** Headroom has *no tool-boundary interception anywhere*. It sees `tool_use`/`tool_result` +only as message content after the fact (`headroom/parser.py`, `headroom/tokenizers/*`). Its +`PipelineStage` enum has no tool-result stage. Everything context-mode does is upstream of +Headroom's earliest hook. + +--- + +## 4. Proposed plugins & variants + +Ranked by value ÷ effort. + +### P1 — `headroom-recall`: FTS5+trigram lossless store as `headroom.memory_text` + +**What:** port `src/store.ts` behind the existing `headroom.memory_text` seam. + +**Why this first:** it is the smallest diff onto an *already-existing* contract, and it fixes a real +product limitation. Today `headroom_retrieve(hash)` requires you to *know the hash* — the tool +description literally says "hash comes from compression markers like `[N items compressed... hash=abc123]`". +With an FTS5-backed store you get `retrieve-by-query`: "what did that build log say about OOM" +instead of "paste hash abc123". The trigram index matters specifically because BM25 tokenization +loses identifiers and stack frames. + +Composes rather than replaces: `compress` → return squeezed text + hash → store the *original* in +FTS5 → rehydrate by hash **or** by query. Also a natural `headroom.ccr_backend` implementation — +the realignment wants "CCR hardens: persistent backend" (Phase B), and this is one. + +**Enterprise variant:** shared team store, retention/TTL policy, per-project scoping (context-mode +already has `project-attribution.ts`), audit of every retrieval. + +**Effort:** medium. Reimplement in Python/Rust against Headroom's memory interface, or ship the +node store as a sidecar. Do not port the MCP tool surface — only the store. + +### P2 — `headroom-admission`: tool-boundary admission control across 18 hosts + +**What:** context-mode's adapter + hook layer, distributed the way `plugins/openclaw` and +`plugins/opencode` already are (TS package under `plugins/`), reporting savings into Headroom's +`savings_ledger.py` JSONL and emitting Headroom pipeline events. + +**Why:** this is the strategic piece. It gives Headroom: +- a **pre-wire** enforcement point, upstream of Phase B's live-zone engine, with no cache-bust and + no token-validation fallback required; +- coverage of **18 agent hosts** — the realignment's Phase G wants to "extend wrap CLIs (cline, + continue, goose, openhands)"; this is that work already done, and then some; +- a deployment mode that works under **subscription auth**, where the proxy is a revocation risk. + +**Enterprise value — this is the DLP story Headroom cannot currently tell.** A `curl` inside a Bash +tool call never touches the proxy, so Headroom is blind to it. context-mode blocks +`curl`/`wget`/`WebFetch`/inline `fetch()`/`requests.get` at the tool boundary and forces network +egress through `ctx_fetch_and_index`. That converts a token-savings feature into an +**egress-control** feature — a different budget line and a different buyer. + +**Effort:** high, but it's mostly packaging + a reporting bridge, not a rewrite. Keep it TypeScript; +Phase H retires Python *proxy* code but explicitly preserves "CLI wrappers, RTK installer" — the +installer layer is the surviving Python, and it can shell out. + +### P3 — `headroom-policy` (Enterprise, license-gated): the PDP + +**What:** `src/security.ts` as a policy decision point, plus centrally-managed org rulesets. + +Two attach points: the hook layer from P2 (tool-level `allow/deny/ask`), and +`headroom.pipeline_extension` at `PRE_SEND` (prompt-level policy). Feeds `headroom/audit/`. + +**Enterprise features that only make sense paid:** central policy service, org-wide allow/deny +rulesets, project-boundary containment enforcement, shell-escape detection inside sandboxed code, +tamper-evident audit trail, per-team reporting. Gate it with the ELv2 license key (see §6). + +**Effort:** medium. The engine exists and is tested (`tests/security/`, `src/security.ts` 889 lines); +the work is the control plane. + +### P4 — `headroom-sandbox`: Think-in-Code execution + +**What:** `executor.ts` exposed as a Headroom MCP tool (`headroom_execute`), 12 languages, +stdout-only. + +**Why:** this is the mechanism behind context-mode's largest measured savings — +`ctx_execute_file` returns 98% savings across 315 KB of real fixtures (`BENCHMARK.md` Part 1), +versus 82% for index+search (Part 2). Programming the analysis beats compressing the output. + +Must ship *with* P3: the shell-escape scanner is what stops the sandbox being an escape hatch. + +**Effort:** medium-high. Runtime isolation is the hard part; `headroom` already has a `sandbox` extra +in `pyproject.toml` to build on. + +### P5 — `headroom-attribution`: counterfactual savings + per-project cost + +**What:** port the *methodology* from `session/analytics.ts` — `RealBytesStats`, +`ThinkInCodeComparison`, `enumerateAdapterDirs`, `project-attribution.ts` — into Headroom's +`savings_ledger` / `reporting` / `dashboard`. + +**Why:** Headroom measures compression deltas (what it squeezed). context-mode measures the +counterfactual (what never entered). Enterprise buyers want the second number, sliced by team and +repo. Do **not** port `pricing.ts` — `headroom/pricing/*` already does this with litellm resolution. + +**Merge, don't port.** `headroom/audit/reads.py` is already a counterfactual measurement tool over +the same Claude Code transcript corpus (see §8). It has the better mechanism taxonomy — identical +repeat, subset containment, write-readback, stale, line-number scaffolding, context residency, +cache-death windows. `analytics.ts` has the multi-host coverage and per-project attribution it +lacks. Combine the two rather than adding a third implementation. + +**Effort:** low-medium, mostly a metrics-definition merge. + +### Variants (packaging, not code) + +- **Headroom No-Proxy Edition** — P1+P2 only, zero API interposition. Sells to buyers who cannot + reroute model traffic and to every subscription-auth user. Removes the single biggest deployment + blocker Headroom has. +- **Headroom Admission Control (Enterprise)** — P2+P3+P4 with a central policy plane and fleet + enrollment across 18 hosts. Positioned as AI-agent DLP/governance, not token savings. +- **Headroom Fleet** — P5 + `enumerateAdapterDirs` for org-wide rollout state and cost reporting. + +--- + +## 5. Evidence base + +context-mode's `BENCHMARK.md`: 21 scenarios, 376 KB raw → 16.5 KB context, **96% overall**, all +fixtures captured from real tool invocations (Context7, Playwright, `gh`, vitest, tsc, nginx logs, +`git log`, analytics CSV) rather than synthetic. Honest about its weak cases — 13% on a 0.4 KB +Playwright network dump, and Part 2 openly explains why index+search only reaches 50-93% (it returns +exact code blocks rather than summaries, by design). + +Test suite: 125 tests across executor/store/MCP-integration/ecosystem, plus 45 test dirs in `tests/` +covering adapters, security, session, hooks, analytics. + +That's a defensible enough evidence base to reuse in Headroom's own materials, and the fixture corpus +itself is reusable for Headroom's `benchmarks/`. + +--- + +## 6. Blockers — resolve these before writing code + +**1. License incompatibility (hard blocker).** +context-mode is **Elastic License 2.0**, "Copyright 2026 Mert Koseoglu". Headroom is +**Apache-2.0**, "Copyright 2025 Headroom Contributors". + +- ELv2 code **cannot** be merged into the Apache-2.0 core. Not a technicality — it would relicense + Headroom's core. +- ELv2 forbids providing the software "to third parties as a hosted or managed service." That + directly constrains `headroom-managed/`. +- Different copyright holders means this needs an **IP arrangement between entities**, not an + engineering decision. + +The good news: Headroom's plugin architecture is exactly the boundary that makes this tractable. +A separate package with its own `pyproject.toml` and its own `LICENSE`, registered on an entry +point — the `plugins/headroom-oauth2/` shape — can carry ELv2 while core stays Apache-2.0. ELv2 is +also the *right* license for a license-key-gated enterprise tier; it explicitly contemplates one. + +Recommendation: any context-mode-derived code ships as separately-licensed plugin packages under +`plugins/`, never vendored into `headroom/`. Get the IP arrangement in writing first. + +**2. Realignment collision.** +Phases A–I are ~40 PRs / 8–13 weeks and include deleting ~25K LOC. Do not open a new integration +front mid-Phase-B. P1 (`headroom.memory_text` / `ccr_backend`) is the exception — it *serves* Phase +B's "CCR hardens: persistent backend" goal rather than competing with it. + +**3. Phase H direction.** +Python proxy code is being retired. Write nothing new in `headroom/proxy/`. Target the surviving +layers: installers, memory writers, CLI wrappers, and Rust. + +--- + +## 7. Sequencing + +| Order | Item | Gate | +|---|---|---| +| 0 | IP/licensing arrangement | before any code | +| 1 | P1 `headroom-recall` — FTS5 store on `memory_text`/`ccr_backend` | lands inside Phase B, serves it | +| 2 | P2 `headroom-admission` — 18-host hook layer under `plugins/` | after Phase A stabilizes | +| 3 | Variant: **No-Proxy Edition** = P1+P2 | as soon as P2 works on 3+ hosts | +| 4 | P3 `headroom-policy` (Enterprise, ELv2, key-gated) | after P2 | +| 5 | P4 `headroom-sandbox` | with P3, never before | +| 6 | P5 `headroom-attribution` | opportunistic | + +--- + +## 8. Follow-up verification + +All four items flagged as open in the first pass are now resolved. + +**`headroom-managed/` is the SaaS arm, and it is unlicensed.** +`headroom-managed/pyproject.toml`: `name = "headroom-managed"`, `description = "Headroom SaaS +Platform - Managed context window optimization"`, `version = 0.1.0`. It has `app/auth.py`, +`app/middleware/`, `app/routes/`, `app/services/`, `app/models.py`, alembic migrations, and a +`pilot/`. There is **no `license` field and no LICENSE file** — i.e. proprietary by default. + +This *sharpens* the §6 blocker rather than easing it. ELv2 forbids providing the software "to third +parties as a hosted or managed service." The product whose name is literally *Managed* is the one +place context-mode-derived code cannot go without an explicit commercial grant from the copyright +holder. Plan the plugin boundary so that `headroom-managed` consumes only Apache-2.0 core +interfaces, never ELv2 implementations. + +**`headroom/audit/reads.py` does not overlap P3 — and it independently validates the whole thesis.** +It is a *measurement* tool, not an audit trail: it streams Claude Code `*.jsonl` transcripts to size +"the addressable bytes for each Read compression mechanism... so defaults are set from traffic, not +theory." No policy, no tamper-evidence. P3's audit trail remains a gap. + +Two lines in its docstring are the most useful corroboration in either repo: + +- *"context residency — how many assistant turns each Read stays in context (the multiplier on its + prefix-cache read cost; **the case for compress-before-cache-entry**)"* — Headroom is already + arguing, from its own traffic, for moving earlier in the pipeline. context-mode is the terminus of + that argument: compress before **context** entry, not merely before cache entry. +- *"identical repeat — a dedup mechanism for this was prototyped and removed: it measured 0.1% of + Read bytes on real traffic."* — Headroom has already empirically established that + message-history-level dedup is worthless. The addressable bytes are at the tool boundary, not in + history. That is the same conclusion the realignment reached from the cache side, arrived at + independently from the traffic side. + +It *does* overlap **P5** — `audit/reads.py` and context-mode's `session/analytics.ts` are two +independent implementations of counterfactual measurement over the same transcript corpus. Merge +them rather than porting; `audit/reads.py` has the better mechanism taxonomy, `analytics.ts` has +multi-host coverage and per-project attribution. + +**No plugin-authoring docs exist.** `docs/` is a Next.js site (`app/`, `content/`, `components/`); +`wiki/` has nothing on extension authoring (only `macos-deployment.md` matched). `plugins/headroom-oauth2/SPEC.md` +remains the de-facto authoring reference — which means whichever plugin lands first sets the house +style. Worth writing the authoring doc as part of P1. + +**Headroom publishes no benchmark results.** `benchmarks/` is 29 runner scripts with no committed +results artifacts, so no like-for-like number exists to compare against context-mode's 96%. The +comparison has to be run. The harness is there and is unusually strong on exactly the axis that +matters: `prefix_cache_benchmark.py`, `cache_bust_trace_report.py`, `cache_validation_bundle.py`, +`synthetic_token_cache_bust_report.py`, `proxy_mode_benchmark.py`, `agent_cost_benchmark.py`, +`real_world_agent_benchmark.py`. Use it to *prove* the §1 cache-safety claim empirically rather than +asserting it — a measured "zero cache-bust events" result is the strongest possible artifact for the +No-Proxy Edition. + +**Bonus finding — the platform axes are orthogonal.** +`docs/platform-feature-matrix.json` (schema v1, updated 2026-07-06) tracks coverage across +`["linux", "macos", "windows"]` — Headroom's platform axis is **operating system**. context-mode's +platform axis is **agent host** (18 of them). Headroom tracks no host-coverage matrix at all. P2 +therefore fills a dimension that does not currently exist in Headroom's own feature accounting, +which also means it needs a second matrix rather than new rows in this one. + +*Process note:* six subagents were dispatched across this analysis and all six stalled at the +600-second watchdog; one reported "Bash is temporarily unavailable" before dying, so the failures +were tool-layer, not analytical. Every finding in this document was verified directly. diff --git a/headroom/backends/litellm.py b/headroom/backends/litellm.py index 8430789ce..60fedfc3a 100644 --- a/headroom/backends/litellm.py +++ b/headroom/backends/litellm.py @@ -413,6 +413,40 @@ PROVIDER_REGISTRY: dict[str, ProviderConfig] = { } +# How long an upstream call may go silent before we give up on it. +# +# WHY THIS EXISTS. There was no timeout here at all, so a request the upstream +# never answered blocked its caller forever. Observed 2026-08-07 under load: +# four agent workers sat on ESTABLISHED connections for 36+ minutes while this +# proxy answered /readyz in 0.11s. No error, no retry, no log line -- the +# client just stops. That is the worst shape a failure can take, because it is +# indistinguishable from slow work and no supervisor can tell the difference. +# +# A float, not an httpx.Timeout, on purpose: litellm expands a float into all +# four httpx phases, so for a STREAMING call this becomes the maximum gap +# BETWEEN CHUNKS rather than a cap on total generation time. A long answer +# streaming steadily is never cut off; a stalled one dies. That is the +# semantic we want, and it falls out of the simpler type. +# +# 600s is deliberately generous -- long enough that no healthy call is at +# risk, short enough that a hang surfaces within a coffee break instead of +# never. +UPSTREAM_TIMEOUT_ENV = "HEADROOM_UPSTREAM_TIMEOUT" +DEFAULT_UPSTREAM_TIMEOUT = 600.0 + + +def _upstream_timeout() -> float: + """Seconds. Never raises; a junk env value must not disable the timeout.""" + import os + + try: + v = float(os.getenv(UPSTREAM_TIMEOUT_ENV, DEFAULT_UPSTREAM_TIMEOUT)) + except (TypeError, ValueError): + return DEFAULT_UPSTREAM_TIMEOUT + # 0 or negative would mean "no timeout" to httpx, which is the bug. + return v if v > 0 else DEFAULT_UPSTREAM_TIMEOUT + + def get_provider_config(provider: str) -> ProviderConfig: """Get provider config, with fallback for unknown providers.""" if provider in PROVIDER_REGISTRY: @@ -925,6 +959,9 @@ class LiteLLMBackend(Backend): logger.debug(f"LiteLLM request: model={litellm_model}") # Make the call + # Bounded, always: an upstream that never answers must not + # block the caller forever. setdefault so an explicit value wins. + kwargs.setdefault("timeout", _upstream_timeout()) response = await acompletion(**kwargs) # Convert to Anthropic format @@ -1055,6 +1092,9 @@ class LiteLLMBackend(Backend): kwargs["stream_options"] = {"include_usage": True} # Stream content — blocks emitted dynamically based on response + # Bounded, always: an upstream that never answers must not + # block the caller forever. setdefault so an explicit value wins. + kwargs.setdefault("timeout", _upstream_timeout()) response = await acompletion(**kwargs) output_tokens = 0 current_block_index = -1 @@ -1283,6 +1323,9 @@ class LiteLLMBackend(Backend): logger.debug(f"LiteLLM OpenAI request: model={litellm_model}") # Make the call + # Bounded, always: an upstream that never answers must not + # block the caller forever. setdefault so an explicit value wins. + kwargs.setdefault("timeout", _upstream_timeout()) response = await acompletion(**kwargs) # Build the usage block. LiteLLM normalizes prompt-cache stats from @@ -1452,6 +1495,9 @@ class LiteLLMBackend(Backend): elif headers.get("x-api-key"): kwargs["api_key"] = headers["x-api-key"] + # Bounded, always: an upstream that never answers must not + # block the caller forever. setdefault so an explicit value wins. + kwargs.setdefault("timeout", _upstream_timeout()) response = await acompletion(**kwargs) async for chunk in response: diff --git a/headroom/proxy/handlers/anthropic.py b/headroom/proxy/handlers/anthropic.py index 26e9f68ee..8251b3590 100644 --- a/headroom/proxy/handlers/anthropic.py +++ b/headroom/proxy/handlers/anthropic.py @@ -2562,8 +2562,14 @@ class AnthropicHandlerMixin: # turn-hook fold (optimized_messages is post-hook). Runs unconditionally. try: _orig_snapshot = original_client_messages # noqa: F821 (bound at request start) - original_tokens = tokenizer.count_messages(_orig_snapshot) - optimized_tokens = tokenizer.count_messages(optimized_messages) + # Off the event loop (#2810): both passes are CPU-bound real BPE + # since #2543, and running them inline stalled every other + # in-flight request on the same process (~1s on a 2.3 MB body). + # Same tokenizer instance, so the reported values are unchanged. + original_tokens = await asyncio.to_thread(tokenizer.count_messages, _orig_snapshot) + optimized_tokens = await asyncio.to_thread( + tokenizer.count_messages, optimized_messages + ) # Fold the tool-schema/desc compaction delta into BOTH endpoints so # tok_before - tok_after == tok_saved stays coherent in the PERF line # (count_messages never sees tool bytes). Same shape as the OpenAI chat @@ -2657,7 +2663,11 @@ class AnthropicHandlerMixin: parsed_original = json.loads(original_body_bytes) if parsed_original != body: body_mutation_tracker.mark_mutated("structural_diff_vs_original") - except (json.JSONDecodeError, ValueError): + # MemoryError is not a ValueError, so a re-parse spike on 1M-context + # bodies used to escape and abort an otherwise-fine request (#2768). + # This block is a safety net; marking mutated is already the safe + # outcome (it forces canonical re-serialization). + except (json.JSONDecodeError, ValueError, MemoryError, RecursionError): body_mutation_tracker.mark_mutated("original_unparseable") if ( diff --git a/headroom/proxy/handlers/openai.py b/headroom/proxy/handlers/openai.py index a7d22449b..23751c65e 100644 --- a/headroom/proxy/handlers/openai.py +++ b/headroom/proxy/handlers/openai.py @@ -3721,8 +3721,11 @@ class OpenAIHandlerMixin: # (result.tokens_before), which mismatches optimized_tokens (provider tokenizer) # and yields impossible tok_after>tok_before. Recount original from the # pre-compression snapshot so the message delta is on one scale. - original_tokens = tokenizer.count_messages(original_client_messages) - optimized_tokens = tokenizer.count_messages(body["messages"]) + # Off the event loop (#2810): see the matching note in the Anthropic handler. + original_tokens = await asyncio.to_thread( + tokenizer.count_messages, original_client_messages + ) + optimized_tokens = await asyncio.to_thread(tokenizer.count_messages, body["messages"]) if tool_tokens_before_compaction > 0: try: tool_tokens_after_compaction = tokenizer.count_text(_json_debug_dumps(tools)) diff --git a/headroom/proxy/prometheus_metrics.py b/headroom/proxy/prometheus_metrics.py index fe477290b..889ed60be 100644 --- a/headroom/proxy/prometheus_metrics.py +++ b/headroom/proxy/prometheus_metrics.py @@ -417,14 +417,14 @@ class PrometheusMetrics: return total_input_tokens, total_input_cost_usd try: - cost_stats = self.cost_tracker.stats() + # totals() rather than stats(): identical numbers, without the + # 31-day cost-record walk that stats()["budget_basis"] performs and + # this caller throws away. See CostTracker.totals. + tracked_input_tokens, tracked_input_cost_usd = self.cost_tracker.totals() except Exception: logger.debug("Failed to read cost tracker totals for savings history", exc_info=True) return total_input_tokens, total_input_cost_usd - tracked_input_tokens = cost_stats.get("total_input_tokens") - tracked_input_cost_usd = cost_stats.get("total_input_cost_usd") - if tracked_input_tokens is not None: try: total_input_tokens = self._savings_tracker_input_tokens_offset + max( diff --git a/run-all-plugins.sh b/run-all-plugins.sh new file mode 100755 index 000000000..f8b9b2ec9 --- /dev/null +++ b/run-all-plugins.sh @@ -0,0 +1,171 @@ +#!/usr/bin/env bash +# ───────────────────────────────────────────────────────────────────────────── +# run-all-plugins.sh — install + configure + run the Headroom proxy with ALL 5 +# enterprise plugins, the coding savings-profile, and ML compression offloaded +# to the Kompress-v2 Modal endpoint. Then confirm everything loaded. +# +# Plugins : lossless_guard, skill_search, observability, tier_router, tool_search +# Extra : headroom-ai[sandbox] (torch-free proxy; ML offloaded to Modal) +# Profile : coding (HEADROOM_SAVINGS_PROFILE) + cache mode (prefix-cache safe) +# +# Secrets are SOURCED from ~/env.txt and ~/.headroom/plugins.env — never inlined. +# Re-runnable: install is skipped when already satisfied (FORCE_INSTALL=1 forces). +# ───────────────────────────────────────────────────────────────────────────── +set -euo pipefail + +HR=/Users/tcms/demo/headroom +VENV="$HR/.venv" +PORT="${HEADROOM_PORT:-8787}" +ENV_TXT="${ENV_TXT:-$HOME/env.txt}" +PLUGINS_ENV="$HOME/.headroom/plugins.env" +LOG="${HEADROOM_LOG:-$HOME/.headroom/logs/proxy-all-plugins.log}" +mkdir -p "$(dirname "$LOG")" + +# ── 1. venv ────────────────────────────────────────────────────────────────── +# `python`/`pip`/`uv` are broken system-wide on this box — always use the venv, +# and `python -m pip` (the .venv/bin/pip shim is broken too). +# shellcheck disable=SC1091 +source "$VENV/bin/activate" +PY="$VENV/bin/python" + +# ── 2. install (guarded) ────────────────────────────────────────────────────── +# headroom-ai[sandbox] pulls proxy,code,relevance,reports,otel,html,mcp,spreadsheet +# (all torch-free — heavy ML is offloaded to the Modal Kompress endpoint below). +# The 5 plugins install --no-deps so pip won't drag PyPI's headroom-ai over the +# local editable one; headroom-license is their shared Ed25519 verifier. +need_install=1 +if [ "${FORCE_INSTALL:-0}" != "1" ]; then + n=$("$PY" -c 'import opentelemetry; from headroom.proxy.extensions import discover; print(len(list(discover())))' 2>/dev/null || echo 0) + [ "$n" = "5" ] && need_install=0 +fi +if [ "$need_install" = "1" ]; then + avail=$(df -g "$HR" 2>/dev/null | awk 'NR==2{print $4}') + echo "▶ disk: ${avail:-?}Gi free before install" + if [ -n "$avail" ] && [ "$avail" -lt 2 ]; then + echo "!! <2Gi free — aborting before heavy install (free space, then re-run)"; exit 1 + fi + echo "▶ installing pip+maturin, then headroom-ai[sandbox] + license + 5 plugins (editable)…" + "$PY" -m pip install -U pip maturin + # litellm >=1.92 ships an sdist-only Rust bridge whose AWS-SDK crates need rustc>=1.94.1; + # the default rustup toolchain here is older (pip builds litellm in a temp dir that misses + # the repo's 1.95 pin), so pin to the last pure-Python wheel line (1.91.4). Satisfies + # headroom's litellm>=1.86.2,<2.0 and skips the Rust build entirely. + "$PY" -m pip install "litellm<1.92" + "$PY" -m pip install -e "${HR}[sandbox]" "litellm<1.92" + "$PY" -m pip install -e /Users/tcms/demo/headroom-license + for p in lossless-guard skill-search observability tier-router tool-search; do + "$PY" -m pip install -e "/Users/tcms/demo/headroom-${p}" --no-deps + done +else + echo "▶ install satisfied (5 extensions discovered) — skipping (FORCE_INSTALL=1 to force)" +fi + +# ── 3. secrets from ~/env.txt ───────────────────────────────────────────────── +# Provides: OPENAI_API_KEY, ANTHROPIC_API_KEY, FIREWORKS_API_KEY (upstream creds); +# LANGFUSE_{PUBLIC,SECRET}_KEY + LANGFUSE_BASE_URL (observability sink); +# HEADROOM_KOMPRESS_ENDPOINT + _TOKEN (Modal ML offload). +[ -f "$ENV_TXT" ] || { echo "!! $ENV_TXT not found"; exit 1; } +set -a; # shellcheck disable=SC1090 +source "$ENV_TXT"; set +a + +# ── 4. plugin license (Ed25519, offline, wildcard) ──────────────────────────── +# HEADROOM_LICENSE + HEADROOM_LICENSE_PUBKEY. This is SEPARATE from the OSS cloud +# key (HEADROOM_LICENSE_KEY) — the banner will still say "OSS (no license key)", +# but each plugin prints "license accepted". Fallback: skip verification entirely. +if [ -f "$PLUGINS_ENV" ]; then + set -a; # shellcheck disable=SC1090 + source "$PLUGINS_ENV"; set +a +else + echo "▶ $PLUGINS_ENV missing — using dev license bypass" + export HEADROOM_LICENSE_DEV=1 +fi + +# ── 5. Kompress ML offload → Modal ──────────────────────────────────────────── +# Setting HEADROOM_KOMPRESS_ENDPOINT (+_TOKEN) alone routes Kompress inference to +# the Modal endpoint (content_router._get_kompress_remote). No other flag needed; +# HEADROOM_COMPRESS_ALLOW_REMOTE is a different thing (remote upstreams, not this). +: "${HEADROOM_KOMPRESS_ENDPOINT:?must be set in $ENV_TXT}" +export HEADROOM_KOMPRESS_ENDPOINT_TOKEN="${HEADROOM_KOMPRESS_ENDPOINT_TOKEN:-}" + +# ── 6. observability sink → Langfuse + spend attribution ────────────────────── +# HEADROOM_LANGFUSE_ENABLED must be explicitly truthy (LANGFUSE_* creds come from +# env.txt). Traces (agent.turn / llm.turn spans with gen_ai.usage.cost) land in +# Langfuse. Spend is opt-in: HEADROOM_MODEL_PRICES is {model-substr:{in,out}} in +# USD per 1K tokens. (Per-request identity — org/team/user/session — is supplied +# by the CLIENT via x-headroom-* headers, not settable here.) +export HEADROOM_LANGFUSE_ENABLED=1 +export HEADROOM_LANGFUSE_SERVICE_NAME=headroom-proxy +export HEADROOM_MODEL_PRICES='{"claude-opus":{"in":0.015,"out":0.075},"claude-sonnet":{"in":0.003,"out":0.015},"gpt-5":{"in":0.00125,"out":0.01},"gpt-4":{"in":0.003,"out":0.012}}' +# Metrics (counters) need a separate OTLP endpoint — none in env.txt, so left off: +# export HEADROOM_OTEL_METRICS_ENABLED=1 HEADROOM_OTEL_METRICS_ENDPOINT=http://localhost:4318 + +# ── 7. tier_router ───────────────────────────────────────────────────────────── +# Only stamps service_tier on the wire (no token delta). OpenAI 'flex' is only +# auto-selected for models declared eligible here. Anthropic tiers are a no-op by +# default. Clients force a tier with x-headroom-tier / x-headroom-background: 1. +export HEADROOM_TIER_FLEX_MODELS="${HEADROOM_TIER_FLEX_MODELS:-gpt-5,gpt-4.1,o4-mini}" + +# ── 8. plugin tuning (defaults shown; override as needed) ───────────────────── +# skill_search fires on Anthropic w/ >=min skills; tool_search on synthetic-tier +# providers w/ >=min tools; lossless_guard lossy tier is opt-in (kept OFF). +export HEADROOM_SKILL_SEARCH_MIN_SKILLS="${HEADROOM_SKILL_SEARCH_MIN_SKILLS:-8}" +export HEADROOM_TOOL_SEARCH_MIN_TOOLS="${HEADROOM_TOOL_SEARCH_MIN_TOOLS:-5}" +# export HEADROOM_LOSSLESS_GUARD_LOSSY=1 # opt-in irreversible Bash-noise drop + +# ── 9. coding profile + mode ─────────────────────────────────────────────────── +# savings_profile=coding tunes the pipeline for coding-agent traffic; cache mode +# freezes prior turns to preserve the provider prefix-cache (what coding wants). +export HEADROOM_SAVINGS_PROFILE=coding + +# ── 10. run + confirm ────────────────────────────────────────────────────────── +cleanup() { [ -n "${PROXY_PID:-}" ] && kill "$PROXY_PID" 2>/dev/null || true; } +trap cleanup INT TERM EXIT + +echo "▶ starting proxy on :$PORT (profile=coding, mode=cache, all 5 extensions)…" +headroom proxy --port "$PORT" --mode cache --proxy-extension '*' > "$LOG" 2>&1 & +PROXY_PID=$! + +# wait for readiness (no foreground sleep on this harness) +curl -s --retry 40 --retry-delay 1 --retry-all-errors --max-time 60 \ + "http://127.0.0.1:$PORT/health" >/dev/null 2>&1 || true + +echo +echo "══════════════════ CONFIRMATION ══════════════════" +echo "── extensions loaded (from $LOG) ──" +grep -iE "Extensions:|license accepted|installed \(" "$LOG" | sed 's/^/ /' || true + +echo "── Modal Kompress endpoint reachable? ──" +code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 "$HEADROOM_KOMPRESS_ENDPOINT" || echo "unreachable") +echo " $HEADROOM_KOMPRESS_ENDPOINT -> HTTP $code (any response = up; offload runs on real traffic)" + +echo "── /stats surfaces (empty until traffic flows) ──" +curl -s --max-time 5 "http://127.0.0.1:$PORT/stats" | "$PY" -c ' +import sys,json +d=json.load(sys.stdin) +print(" extension_savings :", d.get("extension_savings")) +print(" by_layer :", list(d.get("savings",{}).get("by_layer",{}))) +print(" tokens_saved_by_strat:", d.get("tokens_saved_by_strategy")) +print(" otel.enabled :", d.get("otel",{}).get("enabled")) +print(" langfuse.enabled :", d.get("langfuse",{}).get("enabled")) +' 2>/dev/null || echo " (stats not ready)" + +cat < dashboard (compression layer) + /stats.tokens_saved_by_strategy + skill_search -> /stats.extension_savings (NOT dashboard) — Anthropic client, >=8 skills + tool_search -> /stats.extension_savings (NOT dashboard) — synthetic-tier client, >=5 tools + observability -> Langfuse UI (spans + gen_ai.usage.cost) — send x-headroom-org/user/session + tier_router -> service_tier on the wire / provider bill (no token delta) + +── drive traffic (two clients — they exercise different plugins) ── + Claude Code : ANTHROPIC_BASE_URL=http://localhost:$PORT claude # lossless_guard + skill_search + OpenAI/opencode: OPENAI_BASE_URL=http://localhost:$PORT/v1 # tool_search + Dashboard : headroom dashboard (http://127.0.0.1:$PORT/dashboard) + Raw stats : curl -s localhost:$PORT/stats | python3 -m json.tool + +Proxy is running (pid $PROXY_PID). Ctrl-C to stop. Logs: $LOG +═══════════════════════════════════════════════════ +EOF + +wait "$PROXY_PID" || true diff --git a/tests/test_litellm_upstream_timeout.py b/tests/test_litellm_upstream_timeout.py new file mode 100644 index 000000000..9d6918e46 --- /dev/null +++ b/tests/test_litellm_upstream_timeout.py @@ -0,0 +1,70 @@ +"""Every upstream call must be bounded. + +There was no timeout in this backend at all. Observed 2026-08-07 under load: +four agent workers blocked on ESTABLISHED connections for 36+ minutes while +the proxy answered /readyz in 0.11s. No error, no retry, no log line -- the +caller simply stops, forever, and that is indistinguishable from slow work. +""" + +from __future__ import annotations + +import ast +from pathlib import Path + +import pytest + +from headroom.backends.litellm import ( + DEFAULT_UPSTREAM_TIMEOUT, + UPSTREAM_TIMEOUT_ENV, + _upstream_timeout, +) + +_SRC = Path(__file__).resolve().parents[1] / "headroom" / "backends" / "litellm.py" + + +def test_every_acompletion_call_is_bounded(): + """A new dispatch path added without a timeout reintroduces the hang. + + Checked structurally rather than by mocking, because the failure mode is a + call site someone ADDS later -- which no mock of the existing paths sees. + """ + tree = ast.parse(_SRC.read_text()) + calls, guards = 0, 0 + for node in ast.walk(tree): + if not isinstance(node, ast.Call): + continue + fn = node.func + if isinstance(fn, ast.Name) and fn.id == "acompletion": + calls += 1 + if ( + isinstance(fn, ast.Attribute) + and fn.attr == "setdefault" + and node.args + and isinstance(node.args[0], ast.Constant) + and node.args[0].value == "timeout" + ): + guards += 1 + assert calls > 0, "no acompletion call sites found -- test is stale" + assert guards >= calls, ( + f"{calls} acompletion call site(s) but only {guards} timeout guard(s); " + "an unbounded upstream call blocks its caller forever" + ) + + +def test_a_junk_env_value_cannot_disable_the_timeout(monkeypatch): + """`0` means 'no timeout' to httpx, i.e. exactly the bug. So does junk.""" + for bad in ("", "0", "-1", "nonsense", "None"): + monkeypatch.setenv(UPSTREAM_TIMEOUT_ENV, bad) + assert _upstream_timeout() == DEFAULT_UPSTREAM_TIMEOUT, bad + + +def test_an_operator_can_still_tune_it(monkeypatch): + monkeypatch.setenv(UPSTREAM_TIMEOUT_ENV, "42.5") + assert _upstream_timeout() == pytest.approx(42.5) + + +def test_the_default_is_generous_enough_for_real_work(): + """Streaming: litellm expands a float across all httpx phases, so this is + the max gap BETWEEN CHUNKS, not a cap on total generation. A steady long + answer is never cut off.""" + assert 60.0 <= DEFAULT_UPSTREAM_TIMEOUT <= 1800.0 diff --git a/tests/test_proxy/test_anthropic_recount_and_reparse_safety.py b/tests/test_proxy/test_anthropic_recount_and_reparse_safety.py new file mode 100644 index 000000000..1e7f67fad --- /dev/null +++ b/tests/test_proxy/test_anthropic_recount_and_reparse_safety.py @@ -0,0 +1,158 @@ +"""Two request-path safety nets in ``handle_anthropic_messages``. + +1. #2810 — the consistency re-count runs ``count_messages`` twice. Both passes + are CPU-bound real BPE (since #2543) and used to run directly on the event + loop, stalling every other in-flight request on the process (~1s on a 2.3 MB + body). They must run off the loop. +2. #2768 — the byte-faithful forwarder's verification re-parse of the original + body is best-effort, but ``MemoryError`` is not a ``ValueError``, so on + 1M-context payloads it escaped and aborted an otherwise-fine request. The + block must never be able to fail the request. +""" + +from __future__ import annotations + +import asyncio +import json +from unittest.mock import AsyncMock, MagicMock + +import pytest + +fastapi = pytest.importorskip("fastapi") + +from fastapi.testclient import TestClient # noqa: E402 + +from headroom.proxy.server import ProxyConfig, create_app # noqa: E402 + +MESSAGES = "/v1/messages" +MODEL = "claude-sonnet-4-6" + +# Only ever present in the PRE-compression snapshot, never in the outbound body. +# Long enough to clear the handler's min-token floors. +SENTINEL = "presnapshot-sentinel " * 500 + + +def _config(**overrides) -> ProxyConfig: + base = { + "optimize": True, + "cache_enabled": False, + "rate_limit_enabled": False, + "cost_tracking_enabled": False, + "mode": "token", + } + base.update(overrides) + return ProxyConfig(**base) + + +def _upstream_200() -> MagicMock: + payload = { + "id": "msg_1", + "type": "message", + "role": "assistant", + "content": [{"type": "text", "text": "ok"}], + "model": MODEL, + "usage": {"input_tokens": 10, "output_tokens": 2}, + } + resp = MagicMock() + resp.status_code = 200 + resp.headers = {"content-type": "application/json"} + resp.content = json.dumps(payload).encode() + resp.text = json.dumps(payload) + resp.json.return_value = payload + return resp + + +def test_consistency_recount_runs_off_the_event_loop(monkeypatch): + """No ``count_messages`` pass over the pre-compression snapshot may run on + the loop thread. The snapshot is identified by SENTINEL, which the pipeline + strips, so this pins the re-count specifically: the already-offloaded count + at request start also sees the sentinel and passes either way, while the + two re-count passes ran inline before #2810 and would fail here. + """ + import headroom.tokenizers as tokenizers_mod + + seen: list[bool] = [] # one entry per snapshot count: True == ran on the loop + + # Patch the class, not the cached instance, so pytest restores it for us. + tokenizer_cls = type(tokenizers_mod.get_tokenizer(MODEL)) + real_count = tokenizer_cls.count_messages + + def counting(self, messages): # noqa: ANN001, ANN202 + if SENTINEL in json.dumps(messages, default=str): + try: + asyncio.get_running_loop() + except RuntimeError: + seen.append(False) # worker thread — no running loop here + else: + seen.append(True) # blocking the event loop + return real_count(self, messages) + + monkeypatch.setattr(tokenizer_cls, "count_messages", counting) + + def stripping_apply(**kwargs): # noqa: ANN003, ANN202 + """Return genuinely-changed messages with the sentinel removed.""" + from types import SimpleNamespace + + compressed = [{**m, "content": "compressed"} for m in kwargs["messages"]] + return SimpleNamespace( + messages=compressed, + transforms_applied=["test_strip"], + timing={}, + tokens_before=100, + tokens_after=80, + waste_signals=None, + ) + + app = create_app(_config()) + with TestClient(app) as client: + proxy = client.app.state.proxy + proxy.anthropic_pipeline.apply = MagicMock(side_effect=stripping_apply) + proxy._retry_request = AsyncMock(return_value=_upstream_200()) + r = client.post( + MESSAGES, + json={ + "model": MODEL, + "max_tokens": 16, + "messages": [{"role": "user", "content": SENTINEL}], + }, + ) + + assert r.status_code == 200, r.text + assert seen, "no count_messages pass saw the snapshot; test is not exercising #2810" + assert not any(seen), f"{sum(seen)}/{len(seen)} snapshot counts blocked the event loop" + + +def test_memoryerror_in_verification_reparse_does_not_abort_the_request(monkeypatch): + """A ``MemoryError`` from the best-effort original-body re-parse must be + swallowed (the safe fallback marks the body mutated, forcing canonical + re-serialization) rather than escaping and killing the request. + """ + import headroom.proxy.handlers.anthropic as anthropic_mod + + real_loads = json.loads + raised = {"n": 0} + + def exploding_loads(s, *args, **kwargs): # noqa: ANN001, ANN002, ANN003, ANN202 + # Only the verification re-parse passes the raw original body bytes. + if isinstance(s, (bytes, bytearray)) and b"reparse-bomb" in s: + raised["n"] += 1 + raise MemoryError("simulated re-parse spike") + return real_loads(s, *args, **kwargs) + + monkeypatch.setattr(anthropic_mod.json, "loads", exploding_loads) + + app = create_app(_config(optimize=False)) + with TestClient(app) as client: + proxy = client.app.state.proxy + proxy._retry_request = AsyncMock(return_value=_upstream_200()) + r = client.post( + MESSAGES, + json={ + "model": MODEL, + "max_tokens": 16, + "messages": [{"role": "user", "content": "reparse-bomb"}], + }, + ) + + assert raised["n"] > 0, "the verification re-parse never ran; test is not exercising #2768" + assert r.status_code == 200, r.text