From d76a7296df121365d74c415b8c702a3ad80abd30 Mon Sep 17 00:00:00 2001 From: Patrick A <141967+neogenix@users.noreply.github.com> Date: Thu, 11 Jun 2026 19:59:11 -0400 Subject: [PATCH] fix(proxy): make CCR multi-worker warning conditional on backend (#770) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Problem The multi-worker startup warning always mentioned CCR retrieval failures, even when the operator had already configured a cross-worker backend via `HEADROOM_CCR_BACKEND`. That's noise — if they've set `HEADROOM_CCR_BACKEND=sqlite` or `redis`, CCR fragmentation is already resolved. This was surfaced during review of #628 (now closed): the reviewer correctly noted that Python `CompressionStore` defaults to `InMemoryBackend`, which is per-process — each uvicorn worker has its own singleton, so CCR markers written on worker A are invisible to worker B unless a shared backend is configured. ## Changes ### `headroom/proxy/server.py` The `workers > 1` warning is now conditional on `HEADROOM_CCR_BACKEND`: - **Backend unset (default `InMemoryBackend`, per-process):** warning includes CCR retrieval failures and suggests `HEADROOM_CCR_BACKEND=sqlite` to use a shared cross-worker store. - **Backend configured (`sqlite`/`redis`):** warning covers only the remaining per-worker stores (compression cache, prefix tracker, TOIN, CostTracker) — CCR fragmentation is already resolved. ### `RUST_DEV.md` Updated the multi-worker fragmentation section: - Removed the incorrect parenthetical claiming this only applies when the operator *explicitly* chooses `CcrBackendConfig::InMemory` (Python defaults to InMemory) - Added Python `CompressionStore` as item 1 in the fragmented-state list, with a note that setting `HEADROOM_CCR_BACKEND=sqlite` resolves it - Restored TOIN to the fragmented list with a note that its file-backed snapshots do not make it coherently shared across workers - Updated "Detecting it in the wild" to document the conditional warning behaviour ## Files changed | File | Change | |---|---| | `headroom/proxy/server.py` | Conditional two-branch warning based on `HEADROOM_CCR_BACKEND` | | `RUST_DEV.md` | Accurate per-process description of Python `CompressionStore`; restored TOIN | | `CHANGELOG.md` | Entry under `[Unreleased]` | Co-authored-by: JD Davis --- CHANGELOG.md | 1 + RUST_DEV.md | 34 +++++++++++++++------------- headroom/proxy/server.py | 48 +++++++++++++++++++++++++--------------- 3 files changed, 50 insertions(+), 33 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7545d2140..3414bdb77 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -78,6 +78,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Bug Fixes * **codex:** respect `CODEX_HOME` when `headroom wrap codex` writes provider, MCP, memory, backup, and global `AGENTS.md` config, and warn when `unwrap codex` may be looking at the default Codex home because `CODEX_HOME` is unset. +* **proxy:** multi-worker CCR warning is now conditional on backend — when `HEADROOM_CCR_BACKEND` is unset (default `InMemoryBackend`, per-process), the startup warning includes CCR retrieval failures and suggests `HEADROOM_CCR_BACKEND=sqlite`; when a cross-worker backend is already configured, the warning covers only the remaining per-worker stores (compression cache, prefix tracker, TOIN, CostTracker). Updated `RUST_DEV.md` to accurately document Python `CompressionStore` as per-process by default. * **deps:** move `gunicorn` to `[proxy-prod]` extra with `sys_platform != 'win32'` guard; removed from `[proxy]` to avoid forcing a Unix-only package on dev, CI, and Windows users ([#537](https://github.com/chopratejas/headroom/pull/537)) * **startup:** suppress proxy startup log noise -- litellm banner, trafilatura parse errors, HuggingFace Hub unauthenticated warnings, tiktoken fallback warning, and httpx INFO lines from sentence_transformers HEAD checks. Affected files: `headroom/providers/litellm.py`, `headroom/transforms/html_extractor.py`, `headroom/memory/adapters/embedders.py`, `headroom/providers/anthropic.py`, `headroom/providers/registry.py`, `headroom/image/onnx_router.py`, `headroom/transforms/kompress_compressor.py`. diff --git a/RUST_DEV.md b/RUST_DEV.md index 45b9c78e7..f2abab522 100644 --- a/RUST_DEV.md +++ b/RUST_DEV.md @@ -329,29 +329,33 @@ in-memory. ### What goes wrong with the in-memory backend on `--workers N > 1` -(Historical context — applies only when the operator explicitly -chooses `CcrBackendConfig::InMemory`.) Each uvicorn worker is a -separate Python process. Each process holds its own copies of: +Each uvicorn worker is a separate Python process. The following state is +fragmented across workers: -1. **`InMemoryCcrStore`** — sharded `DashMap` mapping - `hash → original_content` for content the compressor replaced with - `<>` markers. -2. **`HeadroomProxy._compression_caches`** (`headroom/proxy/server.py:367`) - — per-session `CompressionCache` dict. +1. **Python `CompressionStore`** — defaults to `InMemoryBackend` (per-process) + when `HEADROOM_CCR_BACKEND` is unset. Each worker has its own singleton; CCR + markers written on worker A are invisible to worker B. Set + `HEADROOM_CCR_BACKEND=sqlite` to use a shared cross-worker store. +2. **`HeadroomProxy._compression_caches`** (`headroom/proxy/server.py`) + — per-session `CompressionCache` dict (instance var, always per-worker). 3. **`HeadroomProxy.session_tracker_store`** — per-session prefix-tracker - state derived from Anthropic's `cache_read_input_tokens` responses. -4. **TOIN learner state** — pattern statistics used to bias the compressor. + state derived from Anthropic's `cache_read_input_tokens` responses + (instance var, always per-worker). +4. **TOIN learner state** — writes snapshots to `~/.headroom/toin.json` but + keeps per-process in-memory state; pattern statistics on one worker are not + visible to others until the next disk flush. When uvicorn round-robins requests across workers, a session whose turn-1 landed on worker A may have turn-2 land on worker B. Worker B has zero knowledge of what worker A did, the `<>` marker resolves to `None`, and the model sees an opaque directive it can't act on. Switching to `SqliteCcrStore` (default) or `RedisCcrStore` resolves the -fragmentation directly. +CCR fragmentation; a sticky-session load balancer resolves all of them. ### Detecting it in the wild -The proxy emits a `WARNING`-level log line on startup if the configured -backend is `InMemoryCcrStore` AND `WEB_CONCURRENCY` / uvicorn -`--workers` is > 1, pointing operators at this section. The other two -backends never warn — they're the supported multi-worker paths. +The proxy emits a `WARNING`-level log line on startup when `--workers N > 1`. +When `HEADROOM_CCR_BACKEND` is unset (default InMemoryBackend), the warning +includes CCR retrieval failures and suggests setting `HEADROOM_CCR_BACKEND=sqlite`. +When a cross-worker backend is already configured, the warning covers only the +remaining per-worker stores (compression cache, prefix tracker, TOIN, CostTracker). diff --git a/headroom/proxy/server.py b/headroom/proxy/server.py index c85e3a1a2..8a6867950 100644 --- a/headroom/proxy/server.py +++ b/headroom/proxy/server.py @@ -3277,24 +3277,36 @@ def run_server( app_target: Any uvicorn_kwargs: dict[str, Any] = {} if workers > 1: - # CCR / compression-cache / prefix-tracker / TOIN state are all - # per-process. Round-robin across workers fragments these caches - # and produces silent retrieval failures for `Retrieve original: - # hash=X` markers and avoidable cache busts on the upstream - # provider. See the "Multi-worker deployment — CCR fragmentation" - # section in RUST_DEV.md for the full failure modes and the - # sticky-session workaround. - logger.warning( - "Headroom is running with workers=%d. The in-memory CCR store, " - "compression cache, prefix tracker, TOIN state, and CostTracker are all " - "per-process; multi-worker deployments produce silent retrieval " - "failures, avoidable cache busts, and an unstable dashboard 'Proxy $ Saved' " - "hero tile (each /stats poll hits a different worker's partial total) when " - "sessions land on different workers. Run --workers 1 (or place a " - "sticky-session load balancer in front of multiple --workers 1 processes). " - "See RUST_DEV.md → 'Multi-worker deployment — CCR fragmentation'.", - workers, - ) + # CompressionCache and PrefixTracker are always per-worker instance vars. + # Python CompressionStore defaults to InMemoryBackend (per-process), so + # CCR markers written on worker A are invisible to worker B unless a + # cross-worker backend is configured via HEADROOM_CCR_BACKEND. + # See RUST_DEV.md -> "Multi-worker deployment -- CCR fragmentation". + if os.environ.get("HEADROOM_CCR_BACKEND", "").strip(): + logger.warning( + "Headroom is running with workers=%d. Compression cache, " + "prefix tracker, TOIN state, and CostTracker are all per-process; " + "multi-worker deployments produce avoidable cache busts and an " + "unstable dashboard 'Proxy $ Saved' hero tile (each /stats poll " + "hits a different worker's partial total) when sessions land on " + "different workers. Run --workers 1 or place a sticky-session load " + "balancer in front of multiple --workers 1 processes. " + "See RUST_DEV.md -> 'Multi-worker deployment -- CCR fragmentation'.", + workers, + ) + else: + logger.warning( + "Headroom is running with workers=%d. The in-memory CCR store, " + "compression cache, prefix tracker, TOIN state, and CostTracker are all " + "per-process; multi-worker deployments produce silent CCR retrieval " + "failures, avoidable cache busts, and an unstable dashboard 'Proxy $ Saved' " + "hero tile (each /stats poll hits a different worker's partial total) when " + "sessions land on different workers. Set HEADROOM_CCR_BACKEND=sqlite for a " + "persistent cross-worker CCR store, run --workers 1, or place a " + "sticky-session load balancer in front of multiple --workers 1 processes. " + "See RUST_DEV.md -> 'Multi-worker deployment -- CCR fragmentation'.", + workers, + ) os.environ[_MULTI_WORKER_CONFIG_ENV] = json.dumps(_proxy_config_payload(config)) app_target = "headroom.proxy.server:create_app_from_env" uvicorn_kwargs["factory"] = True