headroom/tests/test_hermes_tool_call_unwrap.py
pgjh a97b82413b
fix(proxy): unwrap Hermes tool_call bridge in tool name map (#2717)
## Description

Hermes Agent (NousResearch/hermes-agent) loads on-demand ("deferred")
tools via a `tool_search` → `tool_describe` → `tool_call` indirection.
On the wire, the emitted tool call is named **`tool_call`**, and the
REAL tool name lives inside the arguments payload:

```json
{
  "id": "call_abc123",
  "type": "function",
  "function": {
    "name": "tool_call",
    "arguments": "{\"name\": \"read_file\", \"arguments\": {\"path\": \"/etc/hostname\"}}"
  }
}
```

`ContentRouter._build_tool_name_map` only reads
`tool_calls[].function.name`, so it maps `tool_call` → `"tool_call"`
instead of the real tool name.

**Consequence**: `HEADROOM_EXCLUDE_TOOLS` /
`HEADROOM_PROTECT_TOOL_RESULTS` silently no-op for **ALL** deferred
tools (`read_file`, `write_file`, `search_files`, `mcp__*`,
`headroom_retrieve`, etc.). Their outputs get lossy-compressed even when
explicitly whitelisted, and `headroom_retrieve` falls into an endless
re-compression loop (`<<ccr:hash>>` → retrieve → re-compress →
`original_tokens: 0`).

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

Add an `unwrap_tool_call_name(name, arguments)` helper in `config.py`
(beside the existing `_tool_name_aliases`) and apply it at the **3
sites** where the tool_call_id → tool_name map is built:

1. **OpenAI chat path** — `content_router.py` `_build_tool_name_map`
(`tool_calls[].function`)
2. **Anthropic path** — `content_router.py` `_build_tool_name_map`
(`tool_use` blocks)
3. **Responses API path** — `openai.py`
`_compress_openai_responses_live_text_units_with_router`
(`function_call` items)

The helper:
- Passes non-wrapper names through unchanged
- Parses the arguments payload (JSON string or dict) and extracts the
inner `name`
- Fails open (returns the wrapper name) on malformed/unparseable
payloads — safe for all other clients

After unwrap, `is_tool_excluded()`/protect-list matching sees the real
tool name, so whitelists work for deferred tools exactly as they already
did for classic tools.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] New tests added for new functionality
- [x] Manual testing performed

`tests/test_hermes_tool_call_unwrap.py` — **14 tests**, all pass:

- Helper unit tests: passthrough, None/bad-JSON/missing-name fail-open,
unwrap (web_search, read_file, mcp__*, dict-args form)
- Whitelist activation: unwrapped name + `is_tool_excluded` against
`DEFAULT_EXCLUDE_TOOLS`
- Integration: `_build_tool_name_map` with OpenAI-format `tool_call`
wrapper and Anthropic-format `tool_use` wrapper
- Regression guard: documents that `tool_call` itself is NOT in
`DEFAULT_EXCLUDE_TOOLS` (the pre-fix failure mode)

### Test Output

```text
$ uv run pytest tests/test_hermes_tool_call_unwrap.py tests/test_content_router_exclude_tools.py tests/test_config.py -q
56 passed in 1.24s

$ uv run ruff check .
All checks passed!

$ uv run ruff format --check .
All checks passed!
```

## Real Behavior Proof

- Environment: Ubuntu 24.04 (kernel 7.0.0-28), Python 3.11.15, headroom
built from source at `v0.33.0-5-g6d5516dc` via `uv sync` (maturin), Rust
1.95.0 toolchain. Headroom proxy 0.34.0-dev running as a systemd user
service, `HEADROOM_PROTECT_TOOL_RESULTS=read_file,headroom_retrieve`,
mode=token. Upstream: local new-api-compatible gateway (deepseek-v4-pro
/ GLM-5.2).
- Exact command / steps: Client = Hermes Agent (fresh session via
`hermes chat -q --provider newapi2`, traffic routed through the headroom
proxy at 127.0.0.1:8788). Trigger a deferred `read_file` tool call and
observe `/stats` → `recent_requests`.
- Observed result: After fix, live traffic evidence shows request
`hr_1785683282_000016` (GLM-5.2 session) with `transforms_applied:
["router:excluded:tool", "openai:chat:tool_schema_compaction"]` — the
deferred `read_file` tool was recognized and excluded (whitelist hit),
21918 → 17218 input tokens. Before the fix this request showed no
`router:excluded:tool` for deferred tools — they were compressed. E2E
unit-level proof (`test_build_tool_name_map_exclusion_after_unwrap` +
standalone pipeline run): a `tool_call`-wrapped `read_file` output
(~9KB, 100 lines) passed through `ContentRouter.apply()` **verbatim**
(byte-identical), while a non-whitelisted tool (Bash) in the same
pipeline was compressed — proving the whitelist now works and the
pipeline still compresses normally.
- Not tested: Anthropic native clients (Claude Code) and Responses-API
clients (Codex) end-to-end — the patch sites for those paths are covered
by unit tests only. No new dependencies; no public API changes.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

No documentation changes needed (no public API change; behavior is
internal to the proxy tool-name map). Follow-up: end-to-end verification
with Anthropic/Responses-API clients once maintainers can run the proxy
CI on those paths.

Co-authored-by: pgjh <pgjh@users.noreply.github.com>
2026-08-04 22:16:46 -05:00

191 lines
6.5 KiB
Python

"""Tests for Hermes deferred-tool (`tool_call` wrapper) unwrapping.
Hermes Agent loads on-demand tools via a `tool_search`/`tool_describe`/
`tool_call` indirection: on the wire the emitted tool call is named
`tool_call` and the REAL tool name lives in the arguments payload
(`{"name": "...", "arguments": {...}}`). Tool exclusion / protect lists
match on the real name, so `_build_tool_name_map` must unwrap the bridge
or whitelists silently no-op for all deferred tools.
These tests pin the `unwrap_tool_call_name` helper and its integration
into `ContentRouter._build_tool_name_map` (OpenAI + Anthropic paths).
"""
from __future__ import annotations
from headroom.config import (
DEFAULT_EXCLUDE_TOOLS,
is_tool_excluded,
unwrap_tool_call_name,
)
from headroom.transforms.content_router import ContentRouter, ContentRouterConfig
# ---------------------------------------------------------------------------
# Helper unit tests
# ---------------------------------------------------------------------------
def test_unwrap_passthrough_plain_name() -> None:
assert unwrap_tool_call_name("read_file", '{"path": "/x"}') == "read_file"
def test_unwrap_passthrough_none_arguments() -> None:
assert unwrap_tool_call_name("tool_call", None) == "tool_call"
def test_unwrap_passthrough_bad_json() -> None:
assert unwrap_tool_call_name("tool_call", "bad json") == "tool_call"
def test_unwrap_passthrough_missing_name_key() -> None:
assert unwrap_tool_call_name("tool_call", '{"no_name": true}') == "tool_call"
def test_unwrap_passthrough_empty_name() -> None:
assert unwrap_tool_call_name("", None) == ""
def test_unwrap_web_search() -> None:
assert (
unwrap_tool_call_name("tool_call", '{"name": "web_search", "arguments": {}}')
== "web_search"
)
def test_unwrap_read_file() -> None:
assert (
unwrap_tool_call_name("tool_call", '{"name": "read_file", "arguments": {"path": "/x"}}')
== "read_file"
)
def test_unwrap_mcp_tool() -> None:
assert (
unwrap_tool_call_name(
"tool_call", '{"name": "mcp__codebase_memory__search", "arguments": {}}'
)
== "mcp__codebase_memory__search"
)
def test_unwrap_dict_arguments_form() -> None:
"""Arguments may arrive as a dict (not JSON string) on some paths."""
assert (
unwrap_tool_call_name("tool_call", {"name": "search_files", "arguments": {"pattern": "x"}})
== "search_files"
)
def test_unwrap_whitelist_activation() -> None:
"""Unwrapped names must activate the DEFAULT_EXCLUDE_TOOLS whitelist."""
assert is_tool_excluded("web_search", DEFAULT_EXCLUDE_TOOLS) is True
unwrapped = unwrap_tool_call_name("tool_call", '{"name": "web_search", "arguments": {}}')
assert is_tool_excluded(unwrapped, DEFAULT_EXCLUDE_TOOLS) is True
# ---------------------------------------------------------------------------
# _build_tool_name_map integration tests
# ---------------------------------------------------------------------------
def _router(exclude_tools: set[str] | None = None) -> ContentRouter:
config = ContentRouterConfig(
min_section_tokens=10,
enable_kompress=False,
exclude_tools=exclude_tools,
)
return ContentRouter(config)
def test_build_tool_name_map_openai_wrapped() -> None:
"""OpenAI-format assistant tool_calls with Hermes tool_call wrapper."""
messages = [
{
"role": "assistant",
"content": None,
"tool_calls": [
{
"id": "call_wrapped_1",
"type": "function",
"function": {
"name": "tool_call",
"arguments": '{"name": "read_file", "arguments": {"path": "/x"}}',
},
},
{
"id": "call_plain_2",
"type": "function",
"function": {"name": "web_search", "arguments": '{"query": "q"}'},
},
],
}
]
router = _router()
mapping = router._build_tool_name_map(messages)
assert mapping["call_wrapped_1"] == "read_file", (
"wrapped tool_call must map to the real tool name"
)
assert mapping["call_plain_2"] == "web_search", "plain tool names must pass through unchanged"
def test_build_tool_name_map_anthropic_wrapped() -> None:
"""Anthropic-format tool_use blocks with Hermes tool_call wrapper."""
messages = [
{
"role": "assistant",
"content": [
{
"type": "tool_use",
"id": "toolu_wrapped_1",
"name": "tool_call",
"input": {"name": "headroom_retrieve", "arguments": {"hash": "abc"}},
},
{
"type": "tool_use",
"id": "toolu_plain_2",
"name": "Read",
"input": {"file_path": "/x"},
},
],
}
]
router = _router()
mapping = router._build_tool_name_map(messages)
assert mapping["toolu_wrapped_1"] == "headroom_retrieve", (
"wrapped tool_call must map to the real tool name"
)
assert mapping["toolu_plain_2"] == "Read", "plain tool names must pass through unchanged"
def test_build_tool_name_map_wrapped_not_excluded_before_unwrap() -> None:
"""Sanity: without unwrapping, a wrapped read_file is NOT excluded.
This documents the failure mode the fix addresses: `tool_call` is not in
DEFAULT_EXCLUDE_TOOLS, so a whitelist match would never fire.
"""
assert is_tool_excluded("tool_call", DEFAULT_EXCLUDE_TOOLS) is False
assert is_tool_excluded("read_file", DEFAULT_EXCLUDE_TOOLS) is False
def test_build_tool_name_map_exclusion_after_unwrap() -> None:
"""Unwrapped names feed is_tool_excluded for whitelist decisions."""
router = _router(exclude_tools=set(DEFAULT_EXCLUDE_TOOLS) | {"read_file"})
messages = [
{
"role": "assistant",
"content": None,
"tool_calls": [
{
"id": "call_rf_1",
"type": "function",
"function": {
"name": "tool_call",
"arguments": '{"name": "read_file", "arguments": {"path": "/x"}}',
},
}
],
}
]
mapping = router._build_tool_name_map(messages)
assert mapping["call_rf_1"] == "read_file"
assert is_tool_excluded(mapping["call_rf_1"], router.config.exclude_tools or set()) is True