mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-10 14:27:00 -04:00
06add9e9d8
## Description Each token counter in `headroom/providers/` had grown its own shortened content-block walker, handling only the shapes its provider was expected to send. Everything else fell through and contributed **nothing**. Measured on one 6,800-char block, via `count_messages` of a single-block message — so 7–8 is message overhead alone: | block type | OpenAI ctr | Anthropic ctr | |---|---|---| | `text` (control) | 3409 | 3748 | | `tool_result` | **8** | 3748 | | `thinking` | **8** | **7** | | `document` | **8** | **7** | | `mcp_tool_result` | **8** | **7** | | `output_text` | **8** | **7** | | `refusal` | **8** | **7** | Two things make this worse than a coverage gap: 1. **Each counter zeroed blocks from its own provider.** `output_text` and `refusal` are OpenAI Responses shapes; `thinking` and `document` are Anthropic's. 2. **These are the counters the live pipelines use.** `proxy/server.py` builds them with `AnthropicProvider` / `OpenAIProvider`, so this is the main request path — not an edge case. #2743 fixed this for `/v1/compress` only, by routing that route to the registry tokenizers, whose `BaseTokenizer` walker is complete. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made Rather than add a **fifth** partial walker, the counters now delegate to the audited one: - `tokenizers/base.py` — new `count_content_blocks(parts, count_text_fn)` plus a thin `_DelegatingBlockCounter` adapter, since the provider counters are not `BaseTokenizer` subclasses. `BaseTokenizer` itself is untouched. - `providers/openai.py`, `providers/anthropic.py`, `providers/openai_compatible.py` — list-content branches delegate. **Why delegate instead of adding a `count_text(str(block))` catch-all:** that would serialize a base64 blob and price it as text. `tiktoken_counter.py` already documents the failure — a 1MB image becomes ~330K phantom tokens. The shared walker gives media a pixel/byte-based estimate. **Scope:** the three counters that accumulate token counts. `google.py` and `cohere.py` extract a *text string* first and count that, so the same defect there needs a differently-shaped fix — left as a follow-up. ## Testing - [x] Unit tests pass - [x] Linting passes (`ruff check` + `format --check`, pinned 0.15.17) - [x] New tests added - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_provider_counter_content_blocks.py -q 12 passed in 0.52s $ uvx ruff@0.15.17 check headroom/ tests/... --exclude headroom/dashboard/templates All checks passed! ``` **After the fix**, every shape lands within ~1% of the equivalent plain text, and media stays bounded: ```text block OpenAI Anthropic text (control) 3409 3748 tool_result 3409 3748 thinking 3419 3759 document 3423 3763 mcp_tool_result 3422 3762 output_text 3420 3760 refusal 3421 3761 image b64 200KB 1608 1607 <- pixel estimate, not ~50K as text ``` ## Real Behavior Proof — including a regression I caught **This change flipped an existing test**, and I only found it because every suite was run against clean `upstream/main` in the same environment with the failure sets diffed: ```text before the test rewrite: upstream/main : 1 failed, 104 passed this branch : 2 failed, 103 passed <- regression diff : + test_openai_compatible_token_counter_ignores_unhandled_content_shapes ``` That test asserted `content: [{"type": "image"}, 123] == 8` — i.e. it **pinned the defect**, that unhandled shapes contribute nothing. Rewritten as `..._prices_declared_media`: a declared image is now priced (1608) while a bare int is still correctly ignored (8), with the rationale in the docstring. ```text after the rewrite: upstream/main : 1 failed, 104 passed, 10 skipped, 25 errors this branch : 1 failed, 104 passed, 10 skipped, 25 errors failure sets : IDENTICAL ``` - **Pre-existing, not from this change:** the 1 failure and all 25 errors. The errors are all in `test_compress_route_tokenizer_by_model.py`, whose loopback `TestClient` fixture this throwaway env cannot satisfy. - **Environment note:** `content_router` and several suites need the compiled `headroom._core`, which isn't in a fresh worktree (gitignored, built in-place). I copied the built `.so` in to run these and removed it before committing. - **Not tested:** no live provider call, so the *absolute* accuracy of the 1600 image estimate against a real Anthropic/OpenAI bill is unverified — it is the value `BaseTokenizer` already used, and this PR only changes which blocks reach it. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] I did **not** edit `CHANGELOG.md` ## Related Fourth PR from one tokenizer-consistency audit: #2757 (litellm total prompt / `--budget`), #2758 (HuggingFace chat templates, `gpt-5`, gateway-wrapped names), #2759 (router token units). Plus #2756, which splits the local/provider token scales in `RequestOutcome`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| test_anthropic.py | ||
| test_cohere.py | ||
| test_deepseek.py | ||
| test_openai.py | ||
| test_universal.py | ||