From 1deb947ac139f8525c0da2bbe86d6caacfc48a49 Mon Sep 17 00:00:00 2001 From: ahhdammm Date: Thu, 9 Jul 2026 15:29:40 -0700 Subject: [PATCH] fix(proxy): hoist ccr_workspace_key default so /v1/messages survives CCR-inject off (#1096) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary `handle_anthropic_messages` only assigns `ccr_workspace_key` / `ccr_workspace_label` **inside** the `if (ccr_inject_tool or ccr_inject_system_instructions) and not _bypass:` block (around `headroom/proxy/handlers/anthropic.py:1302`), but references `ccr_workspace_key` **unconditionally** in the proactive-expansion gate at `headroom/proxy/handlers/anthropic.py:1394-1397`: ```python if ( self.ccr_context_tracker and self.config.ccr_proactive_expansion and ccr_workspace_key # <-- unbound when the inject block was skipped ): ``` Running the proxy with `--no-ccr-inject-tool` and the default `ccr_inject_system_instructions=False` (a real, supported configuration) skips the assignment. With `ccr_context_tracking=True` and `ccr_proactive_expansion=True` (both defaulting to `True`), the gate is reached and raises `UnboundLocalError`, which FastAPI surfaces as HTTP 500 on **every** `/v1/messages` request. The Claude Code SDK retries ~10 times (`type=system/api_retry`) and then emits the upstream error as the assistant reply (`API Error: 500 Internal Server Error`), which looked exactly like an Anthropic outage from the agent side. Fix: hoist `ccr_workspace_key, ccr_workspace_label = None, None` to before the gated block. The downstream uses already treat a falsy key as "workspace unresolved" — `track_compression` short-circuits to the existing `elif self.ccr_context_tracker and not ccr_workspace_key:` log line, and the proactive-expansion gate stays closed via short-circuit `and`. Behavior with CCR inject enabled is byte-identical. The bug appears to have been introduced by #500 (workspace scoping). I traced it after my NanoClaw containers started returning `API Error: 500 Internal Server Error` for every scheduled run — `journalctl --user -u headroom` showed the traceback. ## Reproduction Failing test in `tests/test_anthropic_ccr_workspace_unbound.py` mirrors the deployment config: ```python config = ProxyConfig( ccr_inject_tool=False, # user passed --no-ccr-inject-tool ccr_inject_system_instructions=False, # default ccr_context_tracking=True, # default — installs the tracker ccr_proactive_expansion=True, # default — reaches the gate ... ) ``` Before the fix: ``` headroom/proxy/handlers/anthropic.py:1397: in handle_anthropic_messages and ccr_workspace_key ^^^^^^^^^^^^^^^^^ E UnboundLocalError: cannot access local variable 'ccr_workspace_key' where it is not associated with a value FAILED tests/test_anthropic_ccr_workspace_unbound.py::test_proactive_expansion_does_not_raise_when_ccr_inject_disabled ``` After the fix: ``` tests/test_anthropic_ccr_workspace_unbound.py . [100%] 1 passed ``` ## Real behavior proof **Setup tested on:** Ubuntu 24.04 on WSL2 (NUC15CRH), Python 3.12.3, `headroom-ai==0.25.0` venv at `/home/adam/headroom-env/`, service started by user-level systemd unit: ``` headroom proxy --host 0.0.0.0 --port 8787 --mode token \ --no-ccr-inject-tool --no-ccr-marker --no-telemetry --code-aware ``` Provider: Anthropic via direct `CLAUDE_CODE_OAUTH_TOKEN` injection from the calling container (NanoClaw / Claude Agent SDK on `claude-opus-4-8`). **Before the patch** — every request through the proxy 500ed: ``` $ curl -sS -m 5 -o /tmp/r -w "HTTP %{http_code}\n" -X POST http://localhost:8787/v1/messages \ -H "x-api-key: placeholder" -H "anthropic-version: 2023-06-01" \ -H "content-type: application/json" \ -d '{"model":"claude-opus-4-8","max_tokens":10,"messages":[{"role":"user","content":"hi"}]}' HTTP 500 $ head -c 40 /tmp/r Internal Server Error $ journalctl --user -u headroom -n 50 --no-pager | grep -A1 ccr_workspace_key | head and ccr_workspace_key ^^^^^^^^^^^^^^^^^ UnboundLocalError: cannot access local variable 'ccr_workspace_key' where it is not associated with a value ``` NanoClaw container logs showed the SDK's 10 `system/api_retry` events then surfacing `API Error: 500 Internal Server Error` as the assistant result. **After the patch** (applied in place to the installed file, service restarted): ``` $ systemctl --user restart headroom $ TOKEN=$(jq -r .claudeAiOauth.accessToken ~/.claude/.credentials.json) $ curl -sS -m 30 -o /tmp/r -w "HTTP %{http_code}\n" -X POST http://localhost:8787/v1/messages \ -H "Authorization: Bearer $TOKEN" -H "anthropic-version: 2023-06-01" \ -H "content-type: application/json" \ -d '{"model":"claude-opus-4-8","max_tokens":20,"messages":[{"role":"user","content":"reply with just the word pong"}]}' HTTP 429 $ cat /tmp/r {"type":"error","error":{"type":"rate_limit_error","message":"Error"},"request_id":"req_011Cc9PSZHi4QssEKLhZX5uq"} ``` The local 500 is gone — the proxy now forwards cleanly and surfaces upstream's real response (here a 429 because the retry storm had been hammering the account for hours; the shape of the response, and the presence of an `anthropic-request_id`, confirms the proxy is no longer crashing on its own code path). Then `journalctl --user -u headroom --since "5 min ago" | grep -iE 'unbound|traceback'` returned no new occurrences after the restart at 12:30 PDT. **What I did *not* test:** - The `_bypass=True` path (same fix protects it, but I did not exercise it end-to-end). - The CCR-inject-on path — relied on the existing `tests/test_proxy_anthropic_cache_stability.py` and `tests/test_proxy_system_prompt_immutable.py` suites passing (they do; ran `uv run pytest tests/test_proxy_anthropic_cache_stability.py tests/test_anthropic_pre_upstream_backpressure.py tests/test_anthropic_stage_timings.py tests/test_provider_proxy_routes.py tests/test_proxy_system_prompt_immutable.py` → 68 passed). ## Test plan - [x] `uv run pytest tests/test_anthropic_ccr_workspace_unbound.py` — fails on `main`, passes on this branch. - [x] `uv run pytest tests/test_proxy_anthropic_cache_stability.py tests/test_anthropic_pre_upstream_backpressure.py tests/test_anthropic_stage_timings.py tests/test_provider_proxy_routes.py tests/test_proxy_system_prompt_immutable.py` — 68 passed. - [x] `uv run ruff check` / `uv run ruff format --check` on modified files — clean. - [x] Live proxy verified against the configuration that reproduced the bug. Co-authored-by: Adam Barnum Co-authored-by: JerrettDavis --- headroom/proxy/handlers/anthropic.py | 8 +- tests/test_anthropic_ccr_workspace_unbound.py | 82 +++++++++++++++++++ 2 files changed, 89 insertions(+), 1 deletion(-) create mode 100644 tests/test_anthropic_ccr_workspace_unbound.py diff --git a/headroom/proxy/handlers/anthropic.py b/headroom/proxy/handlers/anthropic.py index d6820f680..212858534 100644 --- a/headroom/proxy/handlers/anthropic.py +++ b/headroom/proxy/handlers/anthropic.py @@ -1673,7 +1673,13 @@ class AnthropicHandlerMixin: ) except Exception: # advisory hint only — must never fail a request pass - ccr_workspace_key = ccr_workspace_label = None + # Initialize before the gated block so the proactive-expansion + # gate below (which references ``ccr_workspace_key`` regardless of + # the inject flags) does not raise ``UnboundLocalError`` when the + # block is skipped — e.g. ``--no-ccr-inject-tool`` with the default + # ``ccr_inject_system_instructions=False``, or when ``_bypass`` is + # set. The downstream uses already treat falsy as "unresolved". + ccr_workspace_key, ccr_workspace_label = None, None if ( self.config.ccr_inject_tool or self.config.ccr_inject_system_instructions ) and not _bypass: diff --git a/tests/test_anthropic_ccr_workspace_unbound.py b/tests/test_anthropic_ccr_workspace_unbound.py new file mode 100644 index 000000000..d1ee63f51 --- /dev/null +++ b/tests/test_anthropic_ccr_workspace_unbound.py @@ -0,0 +1,82 @@ +"""Regression: ``ccr_workspace_key`` UnboundLocalError when CCR inject is off. + +``handle_anthropic_messages`` previously assigned ``ccr_workspace_key`` / +``ccr_workspace_label`` only inside the ``if (ccr_inject_tool or +ccr_inject_system_instructions) and not _bypass:`` block, but referenced +``ccr_workspace_key`` unconditionally later (the proactive-expansion gate). When +the proxy is started with ``--no-ccr-inject-tool`` and +``ccr_inject_system_instructions`` left at its ``False`` default — a real, +user-supported configuration — the assignment block was skipped and the later +reference raised ``UnboundLocalError``. FastAPI translated that into HTTP 500 on +every ``/v1/messages`` request. + +Reproducer config matches the deployment that surfaced the bug: + + * ``ccr_inject_tool=False`` (user passed ``--no-ccr-inject-tool``) + * ``ccr_inject_system_instructions=False`` (default) + * ``ccr_context_tracking=True`` (default — installs the tracker) + * ``ccr_proactive_expansion=True`` (default — reaches the gate) +""" + +from __future__ import annotations + +import httpx +from fastapi.testclient import TestClient + +from headroom.proxy.server import ProxyConfig, create_app + + +def _client() -> TestClient: + config = ProxyConfig( + optimize=False, + cache_enabled=False, + rate_limit_enabled=False, + cost_tracking_enabled=False, + log_requests=False, + # The two flags that gate the assignment block: + ccr_inject_tool=False, + ccr_inject_system_instructions=False, + # Tracker + proactive expansion enabled (defaults) reach the unbound use: + ccr_context_tracking=True, + ccr_proactive_expansion=True, + image_optimize=False, + ) + return TestClient(create_app(config)) + + +def test_proactive_expansion_does_not_raise_when_ccr_inject_disabled() -> None: + with _client() as client: + proxy = client.app.state.proxy + # Sanity: the tracker is wired up (necessary for the bug to trigger). + assert proxy.ccr_context_tracker is not None + + async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001 + return httpx.Response( + 200, + json={ + "id": "msg_1", + "type": "message", + "role": "assistant", + "content": [{"type": "text", "text": "ok"}], + "usage": { + "input_tokens": 1, + "output_tokens": 1, + "cache_read_input_tokens": 0, + "cache_creation_input_tokens": 0, + }, + }, + ) + + proxy._retry_request = _fake_retry + + response = client.post( + "/v1/messages", + headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"}, + json={ + "model": "claude-sonnet-4-6", + "max_tokens": 16, + "messages": [{"role": "user", "content": "hi"}], + }, + ) + + assert response.status_code == 200, response.text