mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
1deb947ac1
## Summary
`handle_anthropic_messages` only assigns `ccr_workspace_key` /
`ccr_workspace_label` **inside** the
`if (ccr_inject_tool or ccr_inject_system_instructions) and not
_bypass:` block (around `headroom/proxy/handlers/anthropic.py:1302`),
but references `ccr_workspace_key` **unconditionally** in the
proactive-expansion gate at
`headroom/proxy/handlers/anthropic.py:1394-1397`:
```python
if (
self.ccr_context_tracker
and self.config.ccr_proactive_expansion
and ccr_workspace_key # <-- unbound when the inject block was skipped
):
```
Running the proxy with `--no-ccr-inject-tool` and the default
`ccr_inject_system_instructions=False` (a real, supported configuration)
skips the assignment. With `ccr_context_tracking=True` and
`ccr_proactive_expansion=True` (both defaulting to `True`), the gate is
reached and raises `UnboundLocalError`, which FastAPI surfaces as HTTP
500 on **every** `/v1/messages` request. The Claude Code SDK retries ~10
times (`type=system/api_retry`) and then emits the upstream error as the
assistant reply (`API Error: 500 Internal Server Error`), which looked
exactly like an Anthropic outage from the agent side.
Fix: hoist `ccr_workspace_key, ccr_workspace_label = None, None` to
before the gated block. The downstream uses already treat a falsy key as
"workspace unresolved" — `track_compression` short-circuits to the
existing `elif self.ccr_context_tracker and not ccr_workspace_key:` log
line, and the proactive-expansion gate stays closed via short-circuit
`and`. Behavior with CCR inject enabled is byte-identical.
The bug appears to have been introduced by #500 (workspace scoping). I
traced it after my NanoClaw containers started returning `API Error: 500
Internal Server Error` for every scheduled run — `journalctl --user -u
headroom` showed the traceback.
## Reproduction
Failing test in `tests/test_anthropic_ccr_workspace_unbound.py` mirrors
the deployment config:
```python
config = ProxyConfig(
ccr_inject_tool=False, # user passed --no-ccr-inject-tool
ccr_inject_system_instructions=False, # default
ccr_context_tracking=True, # default — installs the tracker
ccr_proactive_expansion=True, # default — reaches the gate
...
)
```
Before the fix:
```
headroom/proxy/handlers/anthropic.py:1397: in handle_anthropic_messages
and ccr_workspace_key
^^^^^^^^^^^^^^^^^
E UnboundLocalError: cannot access local variable 'ccr_workspace_key' where it is not associated with a value
FAILED tests/test_anthropic_ccr_workspace_unbound.py::test_proactive_expansion_does_not_raise_when_ccr_inject_disabled
```
After the fix:
```
tests/test_anthropic_ccr_workspace_unbound.py . [100%]
1 passed
```
## Real behavior proof
**Setup tested on:** Ubuntu 24.04 on WSL2 (NUC15CRH), Python 3.12.3,
`headroom-ai==0.25.0` venv at `/home/adam/headroom-env/`, service
started by user-level systemd unit:
```
headroom proxy --host 0.0.0.0 --port 8787 --mode token \
--no-ccr-inject-tool --no-ccr-marker --no-telemetry --code-aware
```
Provider: Anthropic via direct `CLAUDE_CODE_OAUTH_TOKEN` injection from
the calling container (NanoClaw / Claude Agent SDK on
`claude-opus-4-8`).
**Before the patch** — every request through the proxy 500ed:
```
$ curl -sS -m 5 -o /tmp/r -w "HTTP %{http_code}\n" -X POST http://localhost:8787/v1/messages \
-H "x-api-key: placeholder" -H "anthropic-version: 2023-06-01" \
-H "content-type: application/json" \
-d '{"model":"claude-opus-4-8","max_tokens":10,"messages":[{"role":"user","content":"hi"}]}'
HTTP 500
$ head -c 40 /tmp/r
Internal Server Error
$ journalctl --user -u headroom -n 50 --no-pager | grep -A1 ccr_workspace_key | head
and ccr_workspace_key
^^^^^^^^^^^^^^^^^
UnboundLocalError: cannot access local variable 'ccr_workspace_key' where it is not associated with a value
```
NanoClaw container logs showed the SDK's 10 `system/api_retry` events
then surfacing `API Error: 500 Internal Server Error` as the assistant
result.
**After the patch** (applied in place to the installed file, service
restarted):
```
$ systemctl --user restart headroom
$ TOKEN=$(jq -r .claudeAiOauth.accessToken ~/.claude/.credentials.json)
$ curl -sS -m 30 -o /tmp/r -w "HTTP %{http_code}\n" -X POST http://localhost:8787/v1/messages \
-H "Authorization: Bearer $TOKEN" -H "anthropic-version: 2023-06-01" \
-H "content-type: application/json" \
-d '{"model":"claude-opus-4-8","max_tokens":20,"messages":[{"role":"user","content":"reply with just the word pong"}]}'
HTTP 429
$ cat /tmp/r
{"type":"error","error":{"type":"rate_limit_error","message":"Error"},"request_id":"req_011Cc9PSZHi4QssEKLhZX5uq"}
```
The local 500 is gone — the proxy now forwards cleanly and surfaces
upstream's real response (here a 429 because the retry storm had been
hammering the account for hours; the shape of the response, and the
presence of an `anthropic-request_id`, confirms the proxy is no longer
crashing on its own code path).
Then `journalctl --user -u headroom --since "5 min ago" | grep -iE
'unbound|traceback'` returned no new occurrences after the restart at
12:30 PDT.
**What I did *not* test:**
- The `_bypass=True` path (same fix protects it, but I did not exercise
it end-to-end).
- The CCR-inject-on path — relied on the existing
`tests/test_proxy_anthropic_cache_stability.py` and
`tests/test_proxy_system_prompt_immutable.py` suites passing (they do;
ran `uv run pytest tests/test_proxy_anthropic_cache_stability.py
tests/test_anthropic_pre_upstream_backpressure.py
tests/test_anthropic_stage_timings.py
tests/test_provider_proxy_routes.py
tests/test_proxy_system_prompt_immutable.py` → 68 passed).
## Test plan
- [x] `uv run pytest tests/test_anthropic_ccr_workspace_unbound.py` —
fails on `main`, passes on this branch.
- [x] `uv run pytest tests/test_proxy_anthropic_cache_stability.py
tests/test_anthropic_pre_upstream_backpressure.py
tests/test_anthropic_stage_timings.py
tests/test_provider_proxy_routes.py
tests/test_proxy_system_prompt_immutable.py` — 68 passed.
- [x] `uv run ruff check` / `uv run ruff format --check` on modified
files — clean.
- [x] Live proxy verified against the configuration that reproduced the
bug.
Co-authored-by: Adam Barnum <adamleebarnum@gmail.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
82 lines
3.1 KiB
Python
82 lines
3.1 KiB
Python
"""Regression: ``ccr_workspace_key`` UnboundLocalError when CCR inject is off.
|
|
|
|
``handle_anthropic_messages`` previously assigned ``ccr_workspace_key`` /
|
|
``ccr_workspace_label`` only inside the ``if (ccr_inject_tool or
|
|
ccr_inject_system_instructions) and not _bypass:`` block, but referenced
|
|
``ccr_workspace_key`` unconditionally later (the proactive-expansion gate). When
|
|
the proxy is started with ``--no-ccr-inject-tool`` and
|
|
``ccr_inject_system_instructions`` left at its ``False`` default — a real,
|
|
user-supported configuration — the assignment block was skipped and the later
|
|
reference raised ``UnboundLocalError``. FastAPI translated that into HTTP 500 on
|
|
every ``/v1/messages`` request.
|
|
|
|
Reproducer config matches the deployment that surfaced the bug:
|
|
|
|
* ``ccr_inject_tool=False`` (user passed ``--no-ccr-inject-tool``)
|
|
* ``ccr_inject_system_instructions=False`` (default)
|
|
* ``ccr_context_tracking=True`` (default — installs the tracker)
|
|
* ``ccr_proactive_expansion=True`` (default — reaches the gate)
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import httpx
|
|
from fastapi.testclient import TestClient
|
|
|
|
from headroom.proxy.server import ProxyConfig, create_app
|
|
|
|
|
|
def _client() -> TestClient:
|
|
config = ProxyConfig(
|
|
optimize=False,
|
|
cache_enabled=False,
|
|
rate_limit_enabled=False,
|
|
cost_tracking_enabled=False,
|
|
log_requests=False,
|
|
# The two flags that gate the assignment block:
|
|
ccr_inject_tool=False,
|
|
ccr_inject_system_instructions=False,
|
|
# Tracker + proactive expansion enabled (defaults) reach the unbound use:
|
|
ccr_context_tracking=True,
|
|
ccr_proactive_expansion=True,
|
|
image_optimize=False,
|
|
)
|
|
return TestClient(create_app(config))
|
|
|
|
|
|
def test_proactive_expansion_does_not_raise_when_ccr_inject_disabled() -> None:
|
|
with _client() as client:
|
|
proxy = client.app.state.proxy
|
|
# Sanity: the tracker is wired up (necessary for the bug to trigger).
|
|
assert proxy.ccr_context_tracker is not None
|
|
|
|
async def _fake_retry(method, url, headers, body, stream=False, **kwargs): # noqa: ANN001
|
|
return httpx.Response(
|
|
200,
|
|
json={
|
|
"id": "msg_1",
|
|
"type": "message",
|
|
"role": "assistant",
|
|
"content": [{"type": "text", "text": "ok"}],
|
|
"usage": {
|
|
"input_tokens": 1,
|
|
"output_tokens": 1,
|
|
"cache_read_input_tokens": 0,
|
|
"cache_creation_input_tokens": 0,
|
|
},
|
|
},
|
|
)
|
|
|
|
proxy._retry_request = _fake_retry
|
|
|
|
response = client.post(
|
|
"/v1/messages",
|
|
headers={"x-api-key": "test-key", "anthropic-version": "2023-06-01"},
|
|
json={
|
|
"model": "claude-sonnet-4-6",
|
|
"max_tokens": 16,
|
|
"messages": [{"role": "user", "content": "hi"}],
|
|
},
|
|
)
|
|
|
|
assert response.status_code == 200, response.text
|