fix: tool_search_tool_regex deferred and falsely resolved on direct-Anthropic path (#2971)

## Description

Direct Anthropic users could receive `400 Tool reference
'tool_search_tool_regex' not found in available tools` when Claude Code
sent a typeless `tool_search_tool_regex` entry. Headroom treated it as
an ordinary deferrable tool, injected a typed search tool with the same
name, and later mistook that typed search mechanism for a valid target
of the stale `tool_reference`.

This change prevents the duplicate injection and repairs
already-poisoned transcripts without stripping valid references to
ordinary deferred tools. It addresses the first-party Anthropic
regression reported in [PR #2539's
follow-up](https://github.com/headroomlabs-ai/headroom/pull/2539#issuecomment-5280259642)
and complements the history repair from #2805.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- recognize typeless, case-insensitive `tool_search_tool_*` names as an
existing client tool-search surface and skip Headroom's duplicate
injection
- exclude typed Anthropic search mechanisms from the set of valid
`tool_reference` targets
- preserve valid regular deferred-tool references and the normal
deferral path for similar non-reserved names
- add a first-party Anthropic handler regression that proves the
outbound tools remain unchanged and stale search bookkeeping is removed
- rebase onto #2996, which prevents the native detector from hanging the
full CI test shard

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
.venv/Scripts/python.exe -m pytest \
  tests/test_issue_746_tool_search.py \
  tests/test_anthropic_stage_timings.py \
  tests/test_cache_control_ttl_order.py \
  tests/test_cache_ttl_preserved.py \
  tests/test_proxy/test_tool_search_repair_after_turn_hooks.py \
  tests/test_transforms/test_detect_fallback_1123.py \
  tests/test_transforms_content_detection.py \
  tests/test_transforms_content_router.py \
  -q --disable-warnings --maxfail=1
170 passed, 1 warning in 10.27s

.venv/Scripts/ruff.exe check .
All checks passed!

.venv/Scripts/ruff.exe format --check .
1411 files already formatted

pre-commit run mypy --all-files
Success: no issues found in 519 source files

git diff --check
(no output)
```

## Real Behavior Proof

- Environment: Windows, Python 3.13.3, first-party Anthropic handler
test with `HEADROOM_TOOL_SEARCH` at its default enabled setting
- Exact command / steps: run the same helper-level payload against the
pre-fix base and this branch, then run
`test_anthropic_direct_path_repairs_typeless_tool_search_regression`
through `handle_anthropic_messages()` with 20 ordinary tools, one
typeless `tool_search_tool_regex`, and a stale self-reference
- Observed result: before the fix, Headroom injected a second typed
search tool, deferred the typeless client tool, and removed 0 stale
blocks; on this branch, it skips duplicate injection, preserves the
client tools array, and removes the paired `server_tool_use` and
`tool_search_tool_result` blocks before forwarding
- Not tested: a live request against a paid Anthropic account; the
production handler's outbound body is captured before the network
boundary instead

## Runtime Rollout Safety

- Rollout-managed feature(s): Anthropic server-side tool-search deferral
(`HEADROOM_TOOL_SEARCH`)
- Minimum rollout channel: standard CI; narrow corrective change to an
existing default-on path
- Stable/default behavior changed: yes; reserved typeless client search
tools now suppress duplicate injection, and typed search mechanisms no
longer satisfy deferred-tool references
- Kill switch / disable path: set `HEADROOM_TOOL_SEARCH=0` to disable
new injection; history repair remains unconditional so existing poisoned
sessions can recover
- Unsafe override required: no
- Qualification impact: no new rollout surface or configuration; focused
handler, helper, cache-control, and hook-order regressions cover the
affected path
- Rollback path: revert this PR; operators can set
`HEADROOM_TOOL_SEARCH=0` while rolling back

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A — proxy request transformation only.

## Additional Notes

- Documentation is not changed because this fixes internal request
classification and transcript repair without adding a user-facing option
or workflow.
- Anthropic documents `tool_search_tool_regex` / `tool_search_tool_bm25`
as server search mechanisms; deferred definitions, rather than the
search mechanism itself, are the valid `tool_reference` targets.
- Rebased onto #2996, which fixes the unrelated native-detector hang
that timed out shard 4 on the prior merge commit.

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JerrettDavis <2610199+JerrettDavis@users.noreply.github.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>

headroom/proxy/helpers.py

@@ -2858,8 +2858,9 @@ def inject_tool_search_deferral(
     if not isinstance(tools, list) or len(tools) < _TOOL_SEARCH_MIN_TOOLS:
         return tools
     for tool in tools:
-        if isinstance(tool, dict) and str(tool.get("type", "")).startswith(
+        if isinstance(tool, dict) and (
-            _TOOL_SEARCH_TOOL_TYPE_PREFIX
+            str(tool.get("type", "")).startswith(_TOOL_SEARCH_TOOL_TYPE_PREFIX)
+            or str(tool.get("name") or "").lower().startswith(_TOOL_SEARCH_TOOL_TYPE_PREFIX)
         ):
             return tools  # client already uses tool search — leave it alone
 
@@ -2974,7 +2975,19 @@ def strip_unsupported_tool_search_blocks(messages: Any, tools: Any) -> tuple[Any
         return messages, 0
 
     tool_list = tools if isinstance(tools, list) else []
-    available = {str(t["name"]) for t in tool_list if isinstance(t, dict) and t.get("name")}
+    # Typed search tools (type starts with "tool_search_tool_") are the search
+    # mechanism itself — they are never the target of a tool_reference lookup.
+    # Excluding them from `available` ensures that a stale history entry that
+    # references "tool_search_tool_regex" (from a turn where inject deferred a
+    # typeless client tool with that name) is correctly dropped rather than
+    # falsely kept because the injected typed search tool shares the same name.
+    available = {
+        str(t["name"])
+        for t in tool_list
+        if isinstance(t, dict)
+        and t.get("name")
+        and not str(t.get("type") or "").startswith(_TOOL_SEARCH_TOOL_TYPE_PREFIX)
+    }
     has_search_tool = any(
         isinstance(t, dict) and str(t.get("type", "")).startswith(_TOOL_SEARCH_TOOL_TYPE_PREFIX)
         for t in tool_list

tests/test_anthropic_stage_timings.py

@@ -337,6 +337,88 @@ def test_anthropic_third_party_upstream_strips_tool_search_tools():
     )
 
 
+def test_anthropic_direct_path_repairs_typeless_tool_search_regression():
+    """Do not double-inject a typeless search tool; heal its stale history."""
+    tools = [
+        {
+            "name": f"mcp_tool_{index}",
+            "description": f"tool {index}",
+            "input_schema": {"type": "object", "properties": {}},
+        }
+        for index in range(20)
+    ]
+    tools.append(
+        {
+            "name": "tool_search_tool_regex",
+            "description": "client-provided tool search",
+            "input_schema": {"type": "object", "properties": {}},
+        }
+    )
+    messages = [
+        {"role": "user", "content": "search for a tool"},
+        {
+            "role": "assistant",
+            "content": [
+                {
+                    "type": "server_tool_use",
+                    "id": "srvtoolu_regex",
+                    "name": "tool_search_tool_regex",
+                    "input": {"pattern": "regex"},
+                },
+                {
+                    "type": "tool_search_tool_result",
+                    "tool_use_id": "srvtoolu_regex",
+                    "content": {
+                        "type": "tool_search_tool_search_result",
+                        "tool_references": [
+                            {
+                                "type": "tool_reference",
+                                "tool_name": "tool_search_tool_regex",
+                            }
+                        ],
+                    },
+                },
+            ],
+        },
+        {"role": "user", "content": "continue"},
+    ]
+    request = _build_request(
+        {
+            "model": "claude-sonnet-4-6",
+            "max_tokens": 100,
+            "messages": messages,
+            "tools": tools,
+        },
+        {"authorization": "Bearer sk-ant-api-test"},
+    )
+    handler = _DummyAnthropicHandler()
+
+    import headroom.tokenizers as _tk
+
+    orig_get = _tk.get_tokenizer
+    _tk.get_tokenizer = lambda model: _DummyTokenizer()
+    try:
+        response = anyio.run(handler.handle_anthropic_messages, request)
+    finally:
+        _tk.get_tokenizer = orig_get
+
+    assert response.status_code == 200
+    _, _, _, forwarded_body = handler.captured
+    # The client-owned typeless entry suppresses Headroom's typed search-tool
+    # injection, and the tools array remains byte-for-byte equivalent.
+    assert forwarded_body["tools"] == tools
+    assert not any(tool.get("type") for tool in forwarded_body["tools"])
+    # The stale server-side round trip is removed before Anthropic validates it.
+    block_types = [
+        block.get("type")
+        for message in forwarded_body["messages"]
+        for block in message.get("content", [])
+        if isinstance(message.get("content"), list) and isinstance(block, dict)
+    ]
+    assert "server_tool_use" not in block_types
+    assert "tool_search_tool_result" not in block_types
+
+
 def test_anthropic_http_invalid_body_still_emits_stage_timings(stage_log_capture):
     async def receive():
         # Invalid JSON — produces ``ValueError`` from ``_read_request_json``.

tests/test_issue_746_tool_search.py

@@ -471,3 +471,141 @@ def test_repair_strips_search_history_when_only_the_tool_is_missing() -> None:
         _poisoned_transcript(), [{"name": "AskUserQuestion", "input_schema": {}}]
     )
     assert removed == 2
+
+
+# ---------------------------------------------------------------------------
+# Regression tests for the direct-Anthropic regression reported in PR #2539
+# comment #5280259642: "Tool reference 'tool_search_tool_regex' not found in
+# available tools".
+#
+# Root cause: when a client sends ``tool_search_tool_regex`` as a *typeless*
+# tool, ``inject_tool_search_deferral`` would (a) not early-exit because the
+# guard only checked ``type``, and (b) defer the tool.  Anthropic then found
+# the deferred copy via the server-side search and stored the tool's name in a
+# ``tool_reference`` entry.  On subsequent requests where the typed injected
+# search tool was present, ``strip_unsupported_tool_search_blocks`` incorrectly
+# treated the injected search tool's *name* as proof the reference was
+# resolvable — but the typed server tool is not a valid deferred-tool target, so
+# Anthropic rejected the request with 400.
+#
+# The two-part fix:
+#   1. ``inject_tool_search_deferral`` early-exit also fires on a name-prefix
+#      match, preventing double-injection when the client carries a typeless
+#      ``tool_search_tool_*`` entry.
+#   2. ``strip_unsupported_tool_search_blocks`` excludes typed search tools
+#      from the ``available`` set — they are the search mechanism, not targets.
+# ---------------------------------------------------------------------------
+
+
+def _transcript_with_search_tool_regex_reference() -> list[dict]:
+    """Transcript where the search found 'tool_search_tool_regex' itself.
+
+    This happens when inject_tool_search_deferral defers a typeless client tool
+    named 'tool_search_tool_regex': Anthropic finds it and stores it as a
+    tool_reference.  On subsequent requests the repair must drop the block
+    rather than falsely keep it because the typed injected search-tool shares
+    the same name.
+    """
+    return [
+        {"role": "user", "content": [{"type": "text", "text": "search for a tool"}]},
+        {
+            "role": "assistant",
+            "content": [
+                {
+                    "type": "server_tool_use",
+                    "id": "srvtoolu_REGEX",
+                    "name": _TOOL_SEARCH_DEFAULT_NAME,
+                    "input": {"pattern": "regex"},
+                },
+                {
+                    "type": "tool_search_tool_result",
+                    "tool_use_id": "srvtoolu_REGEX",
+                    "content": {
+                        "type": "tool_search_tool_search_result",
+                        "tool_references": [
+                            {
+                                "type": "tool_reference",
+                                # The search found the deferred 'tool_search_tool_regex'
+                                # typeless tool — this is the broken reference.
+                                "tool_name": _TOOL_SEARCH_DEFAULT_NAME,
+                            }
+                        ],
+                    },
+                },
+            ],
+        },
+    ]
+
+
+@pytest.mark.parametrize(
+    "name",
+    [_TOOL_SEARCH_DEFAULT_NAME, "TOOL_SEARCH_TOOL_BM25"],
+)
+def test_inject_deferral_exits_early_on_typeless_tool_search_name(name: str) -> None:
+    # A client that sends tool_search_tool_regex without a ``type`` field should
+    # be treated as already using tool search (name-prefix guard), so Headroom
+    # must not inject a second search tool on top of it.
+    typeless_search = {"name": name, "input_schema": {}}
+    tools = _tools(20) + [typeless_search]
+    result = inject_tool_search_deferral(tools)
+    assert result is tools  # no injection
+
+
+def test_inject_deferral_does_not_false_match_similar_typeless_tool_name() -> None:
+    # Keep ordinary tools whose names merely resemble the reserved prefix on the
+    # normal deferral path; the trailing underscore is part of the match.
+    tools = _tools(20) + [{"name": "tool_search_toolbox", "input_schema": {}}]
+    result = inject_tool_search_deferral(tools)
+    assert result is not tools
+    by_name = {tool.get("name"): tool for tool in result}
+    assert by_name["tool_search_toolbox"]["defer_loading"] is True
+
+
+def test_repair_drops_search_tool_self_reference_when_inject_ran() -> None:
+    # Regression for PR #2539 comment #5280259642.
+    #
+    # Scenario: inject ran on a previous turn (has_search_tool=True because the
+    # typed search tool is present), but the transcript's tool_reference names
+    # 'tool_search_tool_regex' — the search tool itself.  The typed injected
+    # search tool must NOT count as a valid reference target; the block must be
+    # dropped so Anthropic never sees an unresolvable tool_reference.
+    transcript = _transcript_with_search_tool_regex_reference()
+    # tools array after inject: typed search tool + regular deferred tools
+    tools = [
+        _SEARCH_TOOL,  # typed search tool — must NOT be in 'available'
+        {"name": "Bash", "input_schema": {}},
+        {"name": "mcp_tool_x", "input_schema": {}, "defer_loading": True},
+    ]
+    messages, removed = strip_unsupported_tool_search_blocks(transcript, tools)
+    assert removed == 2  # server_tool_use + tool_search_tool_result both dropped
+    # The assistant turn is entirely stripped (only search blocks were present).
+    assert len(messages) == 1
+    assert messages[0]["role"] == "user"
+
+
+def test_repair_noop_when_referenced_tool_is_regular_deferred_tool() -> None:
+    # Baseline: when the transcript references a normal deferred tool (not the
+    # search tool itself) and that tool is in the current tools array, the block
+    # must be kept — no false-positive stripping from the typed-search exclusion.
+    transcript = _poisoned_transcript()  # references "AskUserQuestion"
+    tools = [
+        _SEARCH_TOOL,
+        {"name": "AskUserQuestion", "input_schema": {}, "defer_loading": True},
+    ]
+    messages, removed = strip_unsupported_tool_search_blocks(transcript, tools)
+    assert removed == 0
+    assert messages is transcript
+
+
+def test_repair_drops_when_referenced_tool_absent_despite_search_tool_present() -> None:
+    # The referenced tool is NOT in the current tools array even though the
+    # typed search tool is present (e.g. a compact request with a different tool
+    # subset).  The block must be dropped.
+    transcript = _poisoned_transcript()  # references "AskUserQuestion"
+    tools = [
+        _SEARCH_TOOL,
+        {"name": "Bash", "input_schema": {}},
+        # AskUserQuestion intentionally absent
+    ]
+    messages, removed = strip_unsupported_tool_search_blocks(transcript, tools)
+    assert removed == 2