diff --git a/CHANGELOG.md b/CHANGELOG.md index b0b5bbd6b..f161cf555 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -102,6 +102,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Bug Fixes +* **tokenizers:** stop `TiktokenCounter.count_messages` from exploding on non-text content blocks. Its multi-part branch handled only `text` and OpenAI `image_url`; every other shape (Anthropic `image`/`tool_result`/`tool_use`, Strands blocks) fell through to `count_text(str(part))`, which json-stringified the base64 payload and tokenized it as text — a 1MB image counted as ~330K phantom tokens (~218x overcount in a standalone repro), corrupting every downstream budgeting/compression decision for multimodal OpenAI-model requests. Unknown block shapes now delegate to the base `_count_content_parts`, which prices images/documents by a bounded estimate (the overcount that helper already exists to prevent). * **install:** don't let a host env export override the manifest in persistent-docker deployments. `build_runtime_command` emitted the manifest's pinned `--env NAME=VALUE` pairs and then, for every host var matching a passthrough prefix, a bare `--env NAME`. Docker resolves duplicate `--env` last-wins, so a stale host export (e.g. `HEADROOM_BACKEND=anyllm`) that shared a passthrough prefix with a pinned manifest value (`HEADROOM_BACKEND=anthropic`) was appended after it and silently won, diverging the container from its deployment config. The bare passthrough is now skipped for any name the manifest already pins. * **memory:** honor explicit `store=false` on OpenAI `/v1/responses` requests by skipping Headroom memory-tool injection that depends on stored-response continuations. Memory context injection stays available, and requests no longer get rewritten to `store=true` behind the client's back ([#1944](https://github.com/headroomlabs-ai/headroom/issues/1944)). * **proxy/batch:** stop corrupting Google `batchGenerateContent` requests whose contents interleave text turns with text-less entries (functionCall/functionResponse/images). The batch handler restored preserved (non-text) entries with the raw-index loop that #836 replaced everywhere else — indexing the shorter `optimized_contents` (text-less entries produce no message) by the original `contents[]` index, which overwrites the wrong entry and drops any preserved entry whose original index is past the optimized length. A request like `[user text, model functionCall, user functionResponse, model text]` was forwarded to Google as two entries: the model's answer overwritten by the functionCall and the functionResponse dropped. The batch handler now uses the shared `_rebuild_gemini_contents` interleaving helper, so all entries survive in order. diff --git a/headroom/tokenizers/tiktoken_counter.py b/headroom/tokenizers/tiktoken_counter.py index 1a093e2df..728a098cf 100644 --- a/headroom/tokenizers/tiktoken_counter.py +++ b/headroom/tokenizers/tiktoken_counter.py @@ -282,7 +282,15 @@ class TiktokenCounter(BaseTokenizer): else: total += 170 # Base for high detail else: - total += self.count_text(str(part)) + # Any other block shape (Anthropic + # image/tool_result/tool_use, Strands blocks) + # is priced by the base handler, which uses a + # bounded per-image/document estimate. Stringifying + # it here would json-serialize a base64 blob and + # count it as text — a 1MB image becomes ~330K + # phantom tokens (the exact overcount base.py + # _count_content_parts exists to prevent). + total += self._count_content_parts([part]) elif isinstance(part, str): total += self.count_text(part) elif key == "role": diff --git a/tests/test_tokenizers.py b/tests/test_tokenizers.py index 31714e8be..b0b6bcfe9 100644 --- a/tests/test_tokenizers.py +++ b/tests/test_tokenizers.py @@ -103,6 +103,43 @@ class TestTiktokenCounter: count = counter.count_messages(messages) assert count > 0 + def test_count_messages_image_block_is_not_stringified(self): + """An Anthropic-style image block must be priced as an image, not text. + + Over the wire the image arrives as a base64 string inside list content. + The old count_messages else-branch stringified any non-text/non-image_url + part and tokenized it as text, so a 1MB image counted as ~330K phantom + tokens. The base handler prices image blocks by a bounded estimate, so the + count must stay small regardless of the base64 payload size. + """ + import base64 + + counter = TiktokenCounter() + blob = base64.b64encode(b"\x89PNG\r\n\x1a\n" + b"\x00" * 200_000).decode() + messages = [ + { + "role": "user", + "content": [ + {"type": "text", "text": "What is in this screenshot?"}, + { + "type": "image", + "source": { + "type": "base64", + "media_type": "image/png", + "data": blob, + }, + }, + ], + } + ] + + count = counter.count_messages(messages) + + # The base64 blob alone would be tens of thousands of text tokens; a + # bounded image estimate keeps the whole message well under that. + assert count < 5000, count + assert count < len(blob) // 10 + def test_encode_decode_roundtrip(self): """Test encode/decode roundtrip.""" counter = TiktokenCounter()