fix(proxy): make CCR multi-worker warning conditional on backend (#770)

## Problem

The multi-worker startup warning always mentioned CCR retrieval
failures, even when the operator had already configured a cross-worker
backend via `HEADROOM_CCR_BACKEND`. That's noise — if they've set
`HEADROOM_CCR_BACKEND=sqlite` or `redis`, CCR fragmentation is already
resolved.

This was surfaced during review of #628 (now closed): the reviewer
correctly noted that Python `CompressionStore` defaults to
`InMemoryBackend`, which is per-process — each uvicorn worker has its
own singleton, so CCR markers written on worker A are invisible to
worker B unless a shared backend is configured.

## Changes

### `headroom/proxy/server.py`

The `workers > 1` warning is now conditional on `HEADROOM_CCR_BACKEND`:

- **Backend unset (default `InMemoryBackend`, per-process):** warning
includes CCR retrieval failures and suggests
`HEADROOM_CCR_BACKEND=sqlite` to use a shared cross-worker store.
- **Backend configured (`sqlite`/`redis`):** warning covers only the
remaining per-worker stores (compression cache, prefix tracker, TOIN,
CostTracker) — CCR fragmentation is already resolved.

### `RUST_DEV.md`

Updated the multi-worker fragmentation section:
- Removed the incorrect parenthetical claiming this only applies when
the operator *explicitly* chooses `CcrBackendConfig::InMemory` (Python
defaults to InMemory)
- Added Python `CompressionStore` as item 1 in the fragmented-state
list, with a note that setting `HEADROOM_CCR_BACKEND=sqlite` resolves it
- Restored TOIN to the fragmented list with a note that its file-backed
snapshots do not make it coherently shared across workers
- Updated "Detecting it in the wild" to document the conditional warning
behaviour

## Files changed

| File | Change |
|---|---|
| `headroom/proxy/server.py` | Conditional two-branch warning based on
`HEADROOM_CCR_BACKEND` |
| `RUST_DEV.md` | Accurate per-process description of Python
`CompressionStore`; restored TOIN |
| `CHANGELOG.md` | Entry under `[Unreleased]` |

Co-authored-by: JD Davis <mxjerrett@gmail.com>

CHANGELOG.md

@@ -78,6 +78,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Bug Fixes
 
 * **codex:** respect `CODEX_HOME` when `headroom wrap codex` writes provider, MCP, memory, backup, and global `AGENTS.md` config, and warn when `unwrap codex` may be looking at the default Codex home because `CODEX_HOME` is unset.
+* **proxy:** multi-worker CCR warning is now conditional on backend — when `HEADROOM_CCR_BACKEND` is unset (default `InMemoryBackend`, per-process), the startup warning includes CCR retrieval failures and suggests `HEADROOM_CCR_BACKEND=sqlite`; when a cross-worker backend is already configured, the warning covers only the remaining per-worker stores (compression cache, prefix tracker, TOIN, CostTracker). Updated `RUST_DEV.md` to accurately document Python `CompressionStore` as per-process by default.
 * **deps:** move `gunicorn` to `[proxy-prod]` extra with `sys_platform != 'win32'` guard; removed from `[proxy]` to avoid forcing a Unix-only package on dev, CI, and Windows users ([#537](https://github.com/chopratejas/headroom/pull/537))
 * **startup:** suppress proxy startup log noise -- litellm banner, trafilatura parse errors, HuggingFace Hub unauthenticated warnings, tiktoken fallback warning, and httpx INFO lines from sentence_transformers HEAD checks. Affected files: `headroom/providers/litellm.py`, `headroom/transforms/html_extractor.py`, `headroom/memory/adapters/embedders.py`, `headroom/providers/anthropic.py`, `headroom/providers/registry.py`, `headroom/image/onnx_router.py`, `headroom/transforms/kompress_compressor.py`.
 

RUST_DEV.md

@@ -329,29 +329,33 @@ in-memory.
 
 ### What goes wrong with the in-memory backend on `--workers N > 1`
 
-(Historical context — applies only when the operator explicitly
-chooses `CcrBackendConfig::InMemory`.) Each uvicorn worker is a
-separate Python process. Each process holds its own copies of:
+Each uvicorn worker is a separate Python process. The following state is
+fragmented across workers:
 
-1. **`InMemoryCcrStore`** — sharded `DashMap` mapping
-   `hash → original_content` for content the compressor replaced with
-   `<<ccr:HASH>>` markers.
-2. **`HeadroomProxy._compression_caches`** (`headroom/proxy/server.py:367`)
-   — per-session `CompressionCache` dict.
+1. **Python `CompressionStore`** — defaults to `InMemoryBackend` (per-process)
+   when `HEADROOM_CCR_BACKEND` is unset. Each worker has its own singleton; CCR
+   markers written on worker A are invisible to worker B. Set
+   `HEADROOM_CCR_BACKEND=sqlite` to use a shared cross-worker store.
+2. **`HeadroomProxy._compression_caches`** (`headroom/proxy/server.py`)
+   — per-session `CompressionCache` dict (instance var, always per-worker).
 3. **`HeadroomProxy.session_tracker_store`** — per-session prefix-tracker
-   state derived from Anthropic's `cache_read_input_tokens` responses.
-4. **TOIN learner state** — pattern statistics used to bias the compressor.
+   state derived from Anthropic's `cache_read_input_tokens` responses
+   (instance var, always per-worker).
+4. **TOIN learner state** — writes snapshots to `~/.headroom/toin.json` but
+   keeps per-process in-memory state; pattern statistics on one worker are not
+   visible to others until the next disk flush.
 
 When uvicorn round-robins requests across workers, a session whose
 turn-1 landed on worker A may have turn-2 land on worker B. Worker B has
 zero knowledge of what worker A did, the `<<ccr:HASH>>` marker resolves
 to `None`, and the model sees an opaque directive it can't act on.
 Switching to `SqliteCcrStore` (default) or `RedisCcrStore` resolves the
-fragmentation directly.
+CCR fragmentation; a sticky-session load balancer resolves all of them.
 
 ### Detecting it in the wild
 
-The proxy emits a `WARNING`-level log line on startup if the configured
-backend is `InMemoryCcrStore` AND `WEB_CONCURRENCY` / uvicorn
-`--workers` is > 1, pointing operators at this section. The other two
-backends never warn — they're the supported multi-worker paths.
+The proxy emits a `WARNING`-level log line on startup when `--workers N > 1`.
+When `HEADROOM_CCR_BACKEND` is unset (default InMemoryBackend), the warning
+includes CCR retrieval failures and suggests setting `HEADROOM_CCR_BACKEND=sqlite`.
+When a cross-worker backend is already configured, the warning covers only the
+remaining per-worker stores (compression cache, prefix tracker, TOIN, CostTracker).

headroom/proxy/server.py

@@ -3277,24 +3277,36 @@ def run_server(
     app_target: Any
     uvicorn_kwargs: dict[str, Any] = {}
     if workers > 1:
-        # CCR / compression-cache / prefix-tracker / TOIN state are all
-        # per-process. Round-robin across workers fragments these caches
-        # and produces silent retrieval failures for `Retrieve original:
-        # hash=X` markers and avoidable cache busts on the upstream
-        # provider. See the "Multi-worker deployment — CCR fragmentation"
-        # section in RUST_DEV.md for the full failure modes and the
-        # sticky-session workaround.
-        logger.warning(
-            "Headroom is running with workers=%d. The in-memory CCR store, "
-            "compression cache, prefix tracker, TOIN state, and CostTracker are all "
-            "per-process; multi-worker deployments produce silent retrieval "
-            "failures, avoidable cache busts, and an unstable dashboard 'Proxy $ Saved' "
-            "hero tile (each /stats poll hits a different worker's partial total) when "
-            "sessions land on different workers. Run --workers 1 (or place a "
-            "sticky-session load balancer in front of multiple --workers 1 processes). "
-            "See RUST_DEV.md → 'Multi-worker deployment — CCR fragmentation'.",
-            workers,
-        )
+        # CompressionCache and PrefixTracker are always per-worker instance vars.
+        # Python CompressionStore defaults to InMemoryBackend (per-process), so
+        # CCR markers written on worker A are invisible to worker B unless a
+        # cross-worker backend is configured via HEADROOM_CCR_BACKEND.
+        # See RUST_DEV.md -> "Multi-worker deployment -- CCR fragmentation".
+        if os.environ.get("HEADROOM_CCR_BACKEND", "").strip():
+            logger.warning(
+                "Headroom is running with workers=%d. Compression cache, "
+                "prefix tracker, TOIN state, and CostTracker are all per-process; "
+                "multi-worker deployments produce avoidable cache busts and an "
+                "unstable dashboard 'Proxy $ Saved' hero tile (each /stats poll "
+                "hits a different worker's partial total) when sessions land on "
+                "different workers. Run --workers 1 or place a sticky-session load "
+                "balancer in front of multiple --workers 1 processes. "
+                "See RUST_DEV.md -> 'Multi-worker deployment -- CCR fragmentation'.",
+                workers,
+            )
+        else:
+            logger.warning(
+                "Headroom is running with workers=%d. The in-memory CCR store, "
+                "compression cache, prefix tracker, TOIN state, and CostTracker are all "
+                "per-process; multi-worker deployments produce silent CCR retrieval "
+                "failures, avoidable cache busts, and an unstable dashboard 'Proxy $ Saved' "
+                "hero tile (each /stats poll hits a different worker's partial total) when "
+                "sessions land on different workers. Set HEADROOM_CCR_BACKEND=sqlite for a "
+                "persistent cross-worker CCR store, run --workers 1, or place a "
+                "sticky-session load balancer in front of multiple --workers 1 processes. "
+                "See RUST_DEV.md -> 'Multi-worker deployment -- CCR fragmentation'.",
+                workers,
+            )
         os.environ[_MULTI_WORKER_CONFIG_ENV] = json.dumps(_proxy_config_payload(config))
         app_target = "headroom.proxy.server:create_app_from_env"
         uvicorn_kwargs["factory"] = True