mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
fix(proxy): keep core tools and the client's ToolSearch resident for PascalCase clients (#2647)
## Description `_TOOL_SEARCH_CORE_TOOLS` is spelled in lowercase, but the membership test compared the raw tool name, so the core-tool exemption never fired for clients that send PascalCase names. For Claude Code (`Bash`, `Read`, `Edit`, `ToolSearch`) **every** tool in the request body was deferred. The damaging part is that Claude Code's own `ToolSearch` was deferred. It is the schema fetcher for tools the client keeps in its local registry and never sends in the body — `TaskCreate`, `TaskUpdate`, `TaskList`, `WebFetch`, `EnterPlanMode`, `Monitor`, `LSP`, `Cron*`, `SendMessage`. Hiding it makes all of them permanently uncallable: advertised to the model in a `<system-reminder>`, but no search can return their schemas, because the injected `tool_search_tool_regex` only indexes what is in the request body. Closes #2646 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Compare tool names against the core set case-insensitively in `inject_tool_search_deferral` (`helpers.py`). - Add `"toolsearch"` to `_TOOL_SEARCH_CORE_TOOLS` so a client's own schema-fetch tool is never deferred. - Apply the same case-insensitive comparison to `inject_tool_search_deferral_openai`, which had the identical exact-match bug (including against `_OPENAI_TOOL_SEARCH_RESIDENT_NAMES = {"terminal"}`). - Add 3 tests on the Anthropic path and 1 on the OpenAI path. Both source changes are required: case-folding alone does not help `ToolSearch` (it was not in the set), and adding it alone does not help `Bash`/`Read`/`Edit`. **The token saving is unchanged** — MCP tools are still deferred. This is not a request to disable the feature. Beyond the stranded tools, the old behaviour also meant (a) routine `Bash`/`Read`/`Edit` loops each paid a search round-trip, the exact cost the core set exists to avoid, and (b) zero resident *real* tools remained, silently violating the invariant documented on `inject_tool_search_deferral` — the injected search tool is typed and does not satisfy it — which risks an upstream 400. The existing assertion for that invariant passes today only because its fixture uses lowercase names. ## Testing - [x] Unit tests pass (`pytest`) — the two affected files; see scope note below - [ ] Linting passes (`ruff check .`) — see note - [ ] Type checking passes (`mypy headroom`) — could not run, see note - [x] New tests added for new functionality - [x] Manual testing performed `ruff check .` reports 4 findings repo-wide, **all pre-existing and unrelated** (`plugins/headroom-oauth2/`), confirmed identical on unmodified `main`. Zero findings in the three files this PR touches, and `ruff format --check` is clean on all three. Left unchecked because the repo-wide command does not exit 0. `mypy headroom` could not run in my environment (numpy stubs error out under the resolved Python version before checking begins). Not attempted further — CI should be the authority. ### Test Output ```text $ python -m pytest tests/test_issue_746_tool_search.py tests/test_openai_tool_search_deferral.py -q 65 passed, 1 warning in 0.70s # Baseline on those two files before this PR: 62 (36 + 26). # The 3 new Anthropic tests + 1 new OpenAI test bring it to 65. # Red before the source change (tests written first): tests/test_issue_746_tool_search.py::test_core_tools_match_case_insensitively FAILED AssertionError: Bash assert True is None where {'name': 'Bash', ..., 'defer_loading': True}.get('defer_loading') tests/test_issue_746_tool_search.py::test_client_tool_search_tool_is_never_deferred FAILED AssertionError: assert True is None where {'name': 'ToolSearch', ..., 'defer_loading': True}.get('defer_loading') tests/test_issue_746_tool_search.py::test_resident_real_tool_survives_pascal_case_surface FAILED assert any(not t.get("type") and not t.get("defer_loading") for t in out) assert False 3 failed, 36 deselected $ python -m ruff check headroom/proxy/helpers.py tests/test_issue_746_tool_search.py tests/test_openai_tool_search_deferral.py All checks passed! $ python -m ruff format --check <same three files> 3 files already formatted ``` ## Real Behavior Proof - Environment: headroom 0.32.1 installed / 0.32.0 source, Python 3.13, macOS 15 (Darwin 25.5.0), Claude Code 2.1.220 with `ENABLE_TOOL_SEARCH=true` and `ANTHROPIC_BASE_URL=http://localhost:8787`, first-party Anthropic upstream, `HEADROOM_TOOL_SEARCH` truthy - Exact command / steps: build a Claude Code tool surface and pass it through the injector — `names = ["Bash","Read","Write","Edit","Glob","Grep","ToolSearch"] + [f"mcp__srv__t{i}" for i in range(12)]`, `tools = [{"name": n, "description": n, "input_schema": {}} for n in names]`, then `inject_tool_search_deferral(tools)` and print which entries carry `defer_loading` - Observed result: before the fix `resident real tools: []` with `ToolSearch deferred: True` (every built-in deferred). After the fix `resident real tools: ['Bash','Edit','Glob','Grep','Read','ToolSearch','Write']` with all 12 `mcp__srv__t*` still deferred, so the saving is retained. This matches a live session: the proxy logged `router:tool_search_deferral:25tools:22182tok ... client=claude-code` and `tool_search_tool_regex` could resolve only `mcp__*` tools — `TaskCreate`/`WebFetch`/`EnterPlanMode` returned no match until `ToolSearch` was recovered by regex-searching for it and then calling `select:TaskCreate,...` - Not tested: the full pytest suite (164 modules fail collection with `ModuleNotFoundError: No module named 'headroom._core'` because my environment imports the package via `PYTHONPATH` without building the Rust extension; identical failure confirmed on unmodified `main`, so it is environmental). `mypy headroom` not runnable here. No end-to-end run against a live upstream through a rebuilt proxy — verification is at the function boundary plus the live-session log evidence above. The OpenAI Responses path is covered by unit test only, not exercised against a real gpt-5.4+ deployment. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes - **Documentation**: N/A — no user-facing surface changes; behaviour returns to what the existing comments and docstring already describe. - **"New and existing unit tests pass locally"**: left unchecked deliberately. The tests covering the changed symbols pass (65), but I cannot run the whole suite locally without the compiled `headroom._core`. Not claiming more than I verified. - **Scope**: the OpenAI-path fix rides along because it is the identical three-line comparison bug in the sibling function. Happy to split it into its own PR if you would rather keep this Anthropic-only. - **Deliberately not done**: I did not add a `client != "claude-code"` gate at `handlers/anthropic.py`, even though the feature's own comment block scopes it to non-Claude-Code clients and `client=claude-code` is already known there (it appears in the `transforms=` log line). Gating there would forfeit the ~22k tokens/request currently saved on Claude Code's eagerly-shipped MCP schemas; keeping the meta-tool resident preserves both the saving and reachability. Flagging in case you would prefer to gate as well. - **Adjacent blind spot, out of scope**: `claude_code_tool_search_inactive` already checks both the tools array *and* the `anthropic-beta` header, but the injector's early-return guard checks only the array. That is why a plain-function `ToolSearch` slips past it and the injection runs on a client that is already deferring. Co-authored-by: Fabien Culpo <fabien.culpo@dawex.com>
This commit is contained in:
parent
1588f5e041
commit
1d29738818
3 changed files with 78 additions and 4 deletions
|
|
@ -2857,6 +2857,11 @@ _TOOL_SEARCH_CORE_TOOLS = frozenset(
|
|||
"webfetch",
|
||||
"question",
|
||||
"skill",
|
||||
# A client's own tool-search/schema-fetch tool (Claude Code's ``ToolSearch``).
|
||||
# It resolves tools the client keeps in its local registry and never puts in
|
||||
# the request body (TaskCreate, WebFetch, …), so deferring it hides the only
|
||||
# tool that can load them and they become permanently unreachable.
|
||||
"toolsearch",
|
||||
}
|
||||
)
|
||||
_TOOL_SEARCH_DEFAULT_TYPE = "tool_search_tool_regex_20251119"
|
||||
|
|
@ -2900,8 +2905,18 @@ def inject_tool_search_deferral(
|
|||
last_resident_real: dict[str, Any] | None = None
|
||||
resident_has_cache_control = False
|
||||
|
||||
# Clients disagree on casing for the same tool: Claude Code sends ``Bash`` /
|
||||
# ``ToolSearch`` where opencode sends ``bash``. Compare case-insensitively so
|
||||
# the exemption applies to both — an exact match silently deferred *every*
|
||||
# tool for PascalCase clients, including their own tool-search tool.
|
||||
core_lower = {name.lower() for name in core_tools}
|
||||
|
||||
for tool in tools:
|
||||
if not isinstance(tool, dict) or tool.get("type") or tool.get("name") in core_tools:
|
||||
if (
|
||||
not isinstance(tool, dict)
|
||||
or tool.get("type")
|
||||
or str(tool.get("name") or "").lower() in core_lower
|
||||
):
|
||||
# Non-dict, server/typed tools (web_search, computer, …), and core
|
||||
# tools stay resident and unchanged.
|
||||
out.append(tool)
|
||||
|
|
@ -3011,6 +3026,11 @@ def inject_tool_search_deferral_openai(
|
|||
|
||||
out: list[Any] = [{"type": _OPENAI_TOOL_SEARCH_TYPE}]
|
||||
deferred = 0
|
||||
# Case-insensitive for the same reason as the Anthropic path above: the
|
||||
# resident-name sets are lowercase, clients are not required to be.
|
||||
resident_lower = {name.lower() for name in core_tools} | {
|
||||
name.lower() for name in _OPENAI_TOOL_SEARCH_RESIDENT_NAMES
|
||||
}
|
||||
for tool in tools:
|
||||
if not isinstance(tool, dict):
|
||||
out.append(tool)
|
||||
|
|
@ -3020,9 +3040,7 @@ def inject_tool_search_deferral_openai(
|
|||
# trained to search namespaces / MCP servers). Everything else — core
|
||||
# coding tools and other hosted tools — stays resident.
|
||||
deferrable = (
|
||||
ttype == "function"
|
||||
and tool.get("name") not in core_tools
|
||||
and tool.get("name") not in _OPENAI_TOOL_SEARCH_RESIDENT_NAMES
|
||||
ttype == "function" and str(tool.get("name") or "").lower() not in resident_lower
|
||||
) or ttype == "mcp"
|
||||
if deferrable and not tool.get("defer_loading"):
|
||||
new_tool = dict(tool)
|
||||
|
|
|
|||
|
|
@ -257,3 +257,45 @@ def test_non_dict_and_typed_tools_stay_resident() -> None:
|
|||
out = inject_tool_search_deferral(tools)
|
||||
typed = [t for t in out if t.get("type") == "web_search_20250305"]
|
||||
assert len(typed) == 1 and typed[0].get("defer_loading") is None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# PascalCase clients (Claude Code). The core-tool exemption is spelled in
|
||||
# lowercase, so an exact-match comparison never fired for Claude Code: every
|
||||
# tool was deferred, including Claude Code's own ``ToolSearch``.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _claude_code_tools() -> list[dict]:
|
||||
"""Claude Code's surface: PascalCase built-ins, its ToolSearch, MCP tools."""
|
||||
names = ["Bash", "Read", "Write", "Edit", "Glob", "Grep", "ToolSearch"] + [
|
||||
f"mcp__srv__t{i}" for i in range(12)
|
||||
]
|
||||
return [{"name": n, "description": n, "input_schema": {}} for n in names]
|
||||
|
||||
|
||||
def test_core_tools_match_case_insensitively() -> None:
|
||||
# Without a case-insensitive match, routine edit/read/run loops each pay a
|
||||
# search round-trip — the exact thing _TOOL_SEARCH_CORE_TOOLS exists to avoid.
|
||||
out = inject_tool_search_deferral(_claude_code_tools())
|
||||
by_name = {t.get("name"): t for t in out if "name" in t}
|
||||
for name in ("Bash", "Read", "Write", "Edit", "Glob", "Grep"):
|
||||
assert by_name[name].get("defer_loading") is None, name
|
||||
# MCP tools are still deferred — the token saving is preserved.
|
||||
assert by_name["mcp__srv__t0"].get("defer_loading") is True
|
||||
|
||||
|
||||
def test_client_tool_search_tool_is_never_deferred() -> None:
|
||||
# ToolSearch is the client's own schema fetcher for tools that never appear
|
||||
# in the request body (TaskCreate, WebFetch, …). Deferring it hides the only
|
||||
# tool that can load them, so they become permanently unreachable.
|
||||
out = inject_tool_search_deferral(_claude_code_tools())
|
||||
by_name = {t.get("name"): t for t in out if "name" in t}
|
||||
assert by_name["ToolSearch"].get("defer_loading") is None
|
||||
|
||||
|
||||
def test_resident_real_tool_survives_pascal_case_surface() -> None:
|
||||
# The injected search tool is typed and does not satisfy the invariant on its
|
||||
# own; Anthropic 400s when every real tool is deferred.
|
||||
out = inject_tool_search_deferral(_claude_code_tools())
|
||||
assert any(not t.get("type") and not t.get("defer_loading") for t in out)
|
||||
|
|
|
|||
|
|
@ -143,3 +143,17 @@ def test_noop_when_nothing_deferrable():
|
|||
|
||||
def test_noop_for_non_list():
|
||||
assert inject_tool_search_deferral_openai(None, "gpt-5.5") is None
|
||||
|
||||
|
||||
def test_resident_names_match_case_insensitively():
|
||||
# The resident-name sets are lowercase; clients are not required to be. An
|
||||
# exact match deferred every tool for a PascalCase client, including its own
|
||||
# tool-search tool. Mirrors the Anthropic-side fix.
|
||||
tools = [_fn(n) for n in ("Bash", "Read", "Edit", "Terminal", "ToolSearch")] + [
|
||||
_fn(f"slack_{i}") for i in range(10)
|
||||
]
|
||||
out = inject_tool_search_deferral_openai(tools, "gpt-5.5")
|
||||
by_name = {t.get("name"): t for t in out if "name" in t}
|
||||
for name in ("Bash", "Read", "Edit", "Terminal", "ToolSearch"):
|
||||
assert by_name[name].get("defer_loading") is None, name
|
||||
assert by_name["slack_0"].get("defer_loading") is True
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue