From 8ea87e7804abfbb55beaf869e50dcb66deab975a Mon Sep 17 00:00:00 2001 From: Copilot <198982749+Copilot@users.noreply.github.com> Date: Thu, 13 Aug 2026 21:15:37 -0500 Subject: [PATCH] fix: tool_search_tool_regex deferred and falsely resolved on direct-Anthropic path (#2971) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Description Direct Anthropic users could receive `400 Tool reference 'tool_search_tool_regex' not found in available tools` when Claude Code sent a typeless `tool_search_tool_regex` entry. Headroom treated it as an ordinary deferrable tool, injected a typed search tool with the same name, and later mistook that typed search mechanism for a valid target of the stale `tool_reference`. This change prevents the duplicate injection and repairs already-poisoned transcripts without stripping valid references to ordinary deferred tools. It addresses the first-party Anthropic regression reported in [PR #2539's follow-up](https://github.com/headroomlabs-ai/headroom/pull/2539#issuecomment-5280259642) and complements the history repair from #2805. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - recognize typeless, case-insensitive `tool_search_tool_*` names as an existing client tool-search surface and skip Headroom's duplicate injection - exclude typed Anthropic search mechanisms from the set of valid `tool_reference` targets - preserve valid regular deferred-tool references and the normal deferral path for similar non-reserved names - add a first-party Anthropic handler regression that proves the outbound tools remain unchanged and stale search bookkeeping is removed - rebase onto #2996, which prevents the native detector from hanging the full CI test shard ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text .venv/Scripts/python.exe -m pytest \ tests/test_issue_746_tool_search.py \ tests/test_anthropic_stage_timings.py \ tests/test_cache_control_ttl_order.py \ tests/test_cache_ttl_preserved.py \ tests/test_proxy/test_tool_search_repair_after_turn_hooks.py \ tests/test_transforms/test_detect_fallback_1123.py \ tests/test_transforms_content_detection.py \ tests/test_transforms_content_router.py \ -q --disable-warnings --maxfail=1 170 passed, 1 warning in 10.27s .venv/Scripts/ruff.exe check . All checks passed! .venv/Scripts/ruff.exe format --check . 1411 files already formatted pre-commit run mypy --all-files Success: no issues found in 519 source files git diff --check (no output) ``` ## Real Behavior Proof - Environment: Windows, Python 3.13.3, first-party Anthropic handler test with `HEADROOM_TOOL_SEARCH` at its default enabled setting - Exact command / steps: run the same helper-level payload against the pre-fix base and this branch, then run `test_anthropic_direct_path_repairs_typeless_tool_search_regression` through `handle_anthropic_messages()` with 20 ordinary tools, one typeless `tool_search_tool_regex`, and a stale self-reference - Observed result: before the fix, Headroom injected a second typed search tool, deferred the typeless client tool, and removed 0 stale blocks; on this branch, it skips duplicate injection, preserves the client tools array, and removes the paired `server_tool_use` and `tool_search_tool_result` blocks before forwarding - Not tested: a live request against a paid Anthropic account; the production handler's outbound body is captured before the network boundary instead ## Runtime Rollout Safety - Rollout-managed feature(s): Anthropic server-side tool-search deferral (`HEADROOM_TOOL_SEARCH`) - Minimum rollout channel: standard CI; narrow corrective change to an existing default-on path - Stable/default behavior changed: yes; reserved typeless client search tools now suppress duplicate injection, and typed search mechanisms no longer satisfy deferred-tool references - Kill switch / disable path: set `HEADROOM_TOOL_SEARCH=0` to disable new injection; history repair remains unconditional so existing poisoned sessions can recover - Unsafe override required: no - Qualification impact: no new rollout surface or configuration; focused handler, helper, cache-control, and hook-order regressions cover the affected path - Rollback path: revert this PR; operators can set `HEADROOM_TOOL_SEARCH=0` while rolling back ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — proxy request transformation only. ## Additional Notes - Documentation is not changed because this fixes internal request classification and transcript repair without adding a user-facing option or workflow. - Anthropic documents `tool_search_tool_regex` / `tool_search_tool_bm25` as server search mechanisms; deferred definitions, rather than the search mechanism itself, are the valid `tool_reference` targets. - Rebased onto #2996, which fixes the unrelated native-detector hang that timed out shard 4 on the prior merge commit. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: JerrettDavis <2610199+JerrettDavis@users.noreply.github.com> Co-authored-by: JerrettDavis --- headroom/proxy/helpers.py | 19 +++- tests/test_anthropic_stage_timings.py | 82 +++++++++++++++ tests/test_issue_746_tool_search.py | 138 ++++++++++++++++++++++++++ 3 files changed, 236 insertions(+), 3 deletions(-) diff --git a/headroom/proxy/helpers.py b/headroom/proxy/helpers.py index 02a2f3c77..d98b67be3 100644 --- a/headroom/proxy/helpers.py +++ b/headroom/proxy/helpers.py @@ -2858,8 +2858,9 @@ def inject_tool_search_deferral( if not isinstance(tools, list) or len(tools) < _TOOL_SEARCH_MIN_TOOLS: return tools for tool in tools: - if isinstance(tool, dict) and str(tool.get("type", "")).startswith( - _TOOL_SEARCH_TOOL_TYPE_PREFIX + if isinstance(tool, dict) and ( + str(tool.get("type", "")).startswith(_TOOL_SEARCH_TOOL_TYPE_PREFIX) + or str(tool.get("name") or "").lower().startswith(_TOOL_SEARCH_TOOL_TYPE_PREFIX) ): return tools # client already uses tool search — leave it alone @@ -2974,7 +2975,19 @@ def strip_unsupported_tool_search_blocks(messages: Any, tools: Any) -> tuple[Any return messages, 0 tool_list = tools if isinstance(tools, list) else [] - available = {str(t["name"]) for t in tool_list if isinstance(t, dict) and t.get("name")} + # Typed search tools (type starts with "tool_search_tool_") are the search + # mechanism itself — they are never the target of a tool_reference lookup. + # Excluding them from `available` ensures that a stale history entry that + # references "tool_search_tool_regex" (from a turn where inject deferred a + # typeless client tool with that name) is correctly dropped rather than + # falsely kept because the injected typed search tool shares the same name. + available = { + str(t["name"]) + for t in tool_list + if isinstance(t, dict) + and t.get("name") + and not str(t.get("type") or "").startswith(_TOOL_SEARCH_TOOL_TYPE_PREFIX) + } has_search_tool = any( isinstance(t, dict) and str(t.get("type", "")).startswith(_TOOL_SEARCH_TOOL_TYPE_PREFIX) for t in tool_list diff --git a/tests/test_anthropic_stage_timings.py b/tests/test_anthropic_stage_timings.py index 56019d096..db8a2d3a1 100644 --- a/tests/test_anthropic_stage_timings.py +++ b/tests/test_anthropic_stage_timings.py @@ -337,6 +337,88 @@ def test_anthropic_third_party_upstream_strips_tool_search_tools(): ) +def test_anthropic_direct_path_repairs_typeless_tool_search_regression(): + """Do not double-inject a typeless search tool; heal its stale history.""" + tools = [ + { + "name": f"mcp_tool_{index}", + "description": f"tool {index}", + "input_schema": {"type": "object", "properties": {}}, + } + for index in range(20) + ] + tools.append( + { + "name": "tool_search_tool_regex", + "description": "client-provided tool search", + "input_schema": {"type": "object", "properties": {}}, + } + ) + messages = [ + {"role": "user", "content": "search for a tool"}, + { + "role": "assistant", + "content": [ + { + "type": "server_tool_use", + "id": "srvtoolu_regex", + "name": "tool_search_tool_regex", + "input": {"pattern": "regex"}, + }, + { + "type": "tool_search_tool_result", + "tool_use_id": "srvtoolu_regex", + "content": { + "type": "tool_search_tool_search_result", + "tool_references": [ + { + "type": "tool_reference", + "tool_name": "tool_search_tool_regex", + } + ], + }, + }, + ], + }, + {"role": "user", "content": "continue"}, + ] + request = _build_request( + { + "model": "claude-sonnet-4-6", + "max_tokens": 100, + "messages": messages, + "tools": tools, + }, + {"authorization": "Bearer sk-ant-api-test"}, + ) + handler = _DummyAnthropicHandler() + + import headroom.tokenizers as _tk + + orig_get = _tk.get_tokenizer + _tk.get_tokenizer = lambda model: _DummyTokenizer() + try: + response = anyio.run(handler.handle_anthropic_messages, request) + finally: + _tk.get_tokenizer = orig_get + + assert response.status_code == 200 + _, _, _, forwarded_body = handler.captured + # The client-owned typeless entry suppresses Headroom's typed search-tool + # injection, and the tools array remains byte-for-byte equivalent. + assert forwarded_body["tools"] == tools + assert not any(tool.get("type") for tool in forwarded_body["tools"]) + # The stale server-side round trip is removed before Anthropic validates it. + block_types = [ + block.get("type") + for message in forwarded_body["messages"] + for block in message.get("content", []) + if isinstance(message.get("content"), list) and isinstance(block, dict) + ] + assert "server_tool_use" not in block_types + assert "tool_search_tool_result" not in block_types + + def test_anthropic_http_invalid_body_still_emits_stage_timings(stage_log_capture): async def receive(): # Invalid JSON — produces ``ValueError`` from ``_read_request_json``. diff --git a/tests/test_issue_746_tool_search.py b/tests/test_issue_746_tool_search.py index 76c31d0ae..4b31cb7f8 100644 --- a/tests/test_issue_746_tool_search.py +++ b/tests/test_issue_746_tool_search.py @@ -471,3 +471,141 @@ def test_repair_strips_search_history_when_only_the_tool_is_missing() -> None: _poisoned_transcript(), [{"name": "AskUserQuestion", "input_schema": {}}] ) assert removed == 2 + + +# --------------------------------------------------------------------------- +# Regression tests for the direct-Anthropic regression reported in PR #2539 +# comment #5280259642: "Tool reference 'tool_search_tool_regex' not found in +# available tools". +# +# Root cause: when a client sends ``tool_search_tool_regex`` as a *typeless* +# tool, ``inject_tool_search_deferral`` would (a) not early-exit because the +# guard only checked ``type``, and (b) defer the tool. Anthropic then found +# the deferred copy via the server-side search and stored the tool's name in a +# ``tool_reference`` entry. On subsequent requests where the typed injected +# search tool was present, ``strip_unsupported_tool_search_blocks`` incorrectly +# treated the injected search tool's *name* as proof the reference was +# resolvable — but the typed server tool is not a valid deferred-tool target, so +# Anthropic rejected the request with 400. +# +# The two-part fix: +# 1. ``inject_tool_search_deferral`` early-exit also fires on a name-prefix +# match, preventing double-injection when the client carries a typeless +# ``tool_search_tool_*`` entry. +# 2. ``strip_unsupported_tool_search_blocks`` excludes typed search tools +# from the ``available`` set — they are the search mechanism, not targets. +# --------------------------------------------------------------------------- + + +def _transcript_with_search_tool_regex_reference() -> list[dict]: + """Transcript where the search found 'tool_search_tool_regex' itself. + + This happens when inject_tool_search_deferral defers a typeless client tool + named 'tool_search_tool_regex': Anthropic finds it and stores it as a + tool_reference. On subsequent requests the repair must drop the block + rather than falsely keep it because the typed injected search-tool shares + the same name. + """ + return [ + {"role": "user", "content": [{"type": "text", "text": "search for a tool"}]}, + { + "role": "assistant", + "content": [ + { + "type": "server_tool_use", + "id": "srvtoolu_REGEX", + "name": _TOOL_SEARCH_DEFAULT_NAME, + "input": {"pattern": "regex"}, + }, + { + "type": "tool_search_tool_result", + "tool_use_id": "srvtoolu_REGEX", + "content": { + "type": "tool_search_tool_search_result", + "tool_references": [ + { + "type": "tool_reference", + # The search found the deferred 'tool_search_tool_regex' + # typeless tool — this is the broken reference. + "tool_name": _TOOL_SEARCH_DEFAULT_NAME, + } + ], + }, + }, + ], + }, + ] + + +@pytest.mark.parametrize( + "name", + [_TOOL_SEARCH_DEFAULT_NAME, "TOOL_SEARCH_TOOL_BM25"], +) +def test_inject_deferral_exits_early_on_typeless_tool_search_name(name: str) -> None: + # A client that sends tool_search_tool_regex without a ``type`` field should + # be treated as already using tool search (name-prefix guard), so Headroom + # must not inject a second search tool on top of it. + typeless_search = {"name": name, "input_schema": {}} + tools = _tools(20) + [typeless_search] + result = inject_tool_search_deferral(tools) + assert result is tools # no injection + + +def test_inject_deferral_does_not_false_match_similar_typeless_tool_name() -> None: + # Keep ordinary tools whose names merely resemble the reserved prefix on the + # normal deferral path; the trailing underscore is part of the match. + tools = _tools(20) + [{"name": "tool_search_toolbox", "input_schema": {}}] + result = inject_tool_search_deferral(tools) + assert result is not tools + by_name = {tool.get("name"): tool for tool in result} + assert by_name["tool_search_toolbox"]["defer_loading"] is True + + +def test_repair_drops_search_tool_self_reference_when_inject_ran() -> None: + # Regression for PR #2539 comment #5280259642. + # + # Scenario: inject ran on a previous turn (has_search_tool=True because the + # typed search tool is present), but the transcript's tool_reference names + # 'tool_search_tool_regex' — the search tool itself. The typed injected + # search tool must NOT count as a valid reference target; the block must be + # dropped so Anthropic never sees an unresolvable tool_reference. + transcript = _transcript_with_search_tool_regex_reference() + # tools array after inject: typed search tool + regular deferred tools + tools = [ + _SEARCH_TOOL, # typed search tool — must NOT be in 'available' + {"name": "Bash", "input_schema": {}}, + {"name": "mcp_tool_x", "input_schema": {}, "defer_loading": True}, + ] + messages, removed = strip_unsupported_tool_search_blocks(transcript, tools) + assert removed == 2 # server_tool_use + tool_search_tool_result both dropped + # The assistant turn is entirely stripped (only search blocks were present). + assert len(messages) == 1 + assert messages[0]["role"] == "user" + + +def test_repair_noop_when_referenced_tool_is_regular_deferred_tool() -> None: + # Baseline: when the transcript references a normal deferred tool (not the + # search tool itself) and that tool is in the current tools array, the block + # must be kept — no false-positive stripping from the typed-search exclusion. + transcript = _poisoned_transcript() # references "AskUserQuestion" + tools = [ + _SEARCH_TOOL, + {"name": "AskUserQuestion", "input_schema": {}, "defer_loading": True}, + ] + messages, removed = strip_unsupported_tool_search_blocks(transcript, tools) + assert removed == 0 + assert messages is transcript + + +def test_repair_drops_when_referenced_tool_absent_despite_search_tool_present() -> None: + # The referenced tool is NOT in the current tools array even though the + # typed search tool is present (e.g. a compact request with a different tool + # subset). The block must be dropped. + transcript = _poisoned_transcript() # references "AskUserQuestion" + tools = [ + _SEARCH_TOOL, + {"name": "Bash", "input_schema": {}}, + # AskUserQuestion intentionally absent + ] + messages, removed = strip_unsupported_tool_search_blocks(transcript, tools) + assert removed == 2