mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
12 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Tejas Chopra
|
285176be54
|
fix(tokenizers): price Claude against a real BPE (tiktoken o200k) not a char estimate (#2543)
## Description
Claude's tokenizer is not public, so `get_tokenizer("claude-*")`
returned `EstimatingTokenCounter(chars_per_token=3.5)` — a
**character-ratio estimate**. The estimator's chars-per-token flips with
the *detected content type* (JSON 3.2 / code 3.5 / English 4.0 / CJK
1.5), so:
- the **same bytes** count differently before vs after compression → a
fold could appear to *increase* tokens;
- it **disagrees with the pipeline's** estimator on the same content.
Calibration vs a real BPE (tiktoken `o200k_base`) on representative
content:
| content | char-est(3.5) / o200k |
|---|---|
| English | **1.28×** (overcount 28%) |
| code | **0.83×** (undercount 17%) |
| JSON | 0.97× |
| diff | 1.02× |
i.e. the estimate is off **−17% … +28%**, and the error is
content-dependent (non-uniform) — which is exactly what produces
impossible `tok_after > tok_before` deltas.
This prices Claude against **tiktoken `o200k_base`** — a real,
deterministic, **monotone** BPE. It's not Claude's exact vocab, but it's
within ~10–20% and, crucially, **consistent before/after**, which is
what compression ratios and context-pressure gating actually need.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Performance improvement
- [ ] Breaking change
- [ ] Documentation update
- [ ] Code refactoring (no functional changes)
## Changes Made
- `TiktokenCounter.__init__` accepts an explicit `encoding` override (so
a model can be priced against a chosen encoding regardless of name-based
resolution).
- `_create_anthropic` now returns `TiktokenCounter(model,
encoding="o200k_base")`, **failing open to the character estimator** if
the tiktoken vocab can't be loaded (reuses the existing `load_encoding`
timeout/guard).
- Updated the `MODEL_PATTERNS` comment and `test_get_anthropic_model` to
the new contract; added
`test_claude_priced_with_real_bpe_not_char_estimate`.
**Blast radius is contained:** `content_router` keeps its own
module-level estimator for routing decisions, so only the **handler's
reported counts** change. `tiktoken` is already a dependency.
**Composes with #2542** (tokenizer-consistent before/after): that PR
makes both endpoints use *one* tokenizer; this PR makes that one
tokenizer a *real BPE*. Together they fully eliminate the
inflated/phantom lines. This PR is based on `main` and is independently
valid.
## Calibration note (please review)
The one behavioral consumer of the handler's count is the
background-compression gate (`original_tokens >=
_background_compression_min_tokens`). Because o200k differs from the old
estimate by ~±20% depending on content, that threshold now trips on
slightly different requests. It's *more* accurate, but the threshold was
tuned against the estimator — worth a re-check. No other gate consumes
the handler count (routing uses `content_router`'s estimator).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check` + `ruff format --check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added
- [x] Manual testing performed
### Test Output
```text
$ ruff check <changed files> && ruff format --check <changed files>
All checks passed!
4 files already formatted
$ mypy headroom/tokenizers/registry.py headroom/tokenizers/tiktoken_counter.py
Success: no issues found in 2 source files
$ pytest tests/test_tokenizer.py tests/test_tokenizers.py -q
55 passed, 14 skipped
```
## Real Behavior Proof
- **Unit:** `get_tokenizer("claude-opus-4-8")` →
`TiktokenCounter(encoding_name="o200k_base")`; `count_text(sample)` ==
`tiktoken.get_encoding("o200k_base").encode(sample)` exactly; a
`tool_result` shrunk 300→3 words drops 508→13 tokens (folds register).
- **Live proxy** (Anthropic path, `--proxy-extension lossless_guard`):
foldable request logs `tok_before=694 tok_after=231 tok_saved=463
transforms=turn_hook` — real o200k-scale numbers (vs the estimator's 607
for the same payload), clean fold, no inflation.
- **Not tested:** exact agreement with Anthropic's *own* count_tokens
API (o200k is a proxy; a follow-up could calibrate against it offline).
GPT paths unchanged (already tiktoken).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] Documentation — N/A (internal; docstrings updated)
- [x] My changes generate no new warnings
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
Follow-ups (not here): (1) extend the same real-BPE treatment to other
private-tokenizer providers (Gemini/Cohere/Moonshot), each with its own
calibration; (2) optional opt-in calibration against Anthropic's
`count_tokens` API to true-up the ~10–20% absolute offset.
|
||
|
Abhay Singh
|
6979b5245e
|
fix(tokenizers): use o200k_base for gpt-4.1/gpt-4.5/o4 families (#2108)
## Description
`get_encoding_for_model` returns the wrong tiktoken encoding for the
current OpenAI flagship families, so their token counts are computed
with the wrong vocabulary.
The prefix table is ordered most-specific-first, but it has no entry for
the `gpt-4.1` / `gpt-4.5` / `o4` families:
```python
for prefix, encoding in (
("gpt-4o", "o200k_base"),
("gpt-4-turbo", "cl100k_base"),
("gpt-4", "cl100k_base"),
("gpt-3.5", "cl100k_base"),
("o1", "o200k_base"),
("o3", "o200k_base"),
):
if model.startswith(prefix):
return encoding
return DEFAULT_ENCODING # cl100k_base
```
- `gpt-4.1`, `gpt-4.1-mini`, `gpt-4.5-*` all start with `gpt-4`, so they
match the `gpt-4` prefix and get `cl100k_base`.
- `o4-mini` matches no prefix and falls through to the `cl100k_base`
default.
All three families use `o200k_base`. Since `count_text`/`count_messages`
tokenize with the resolved encoding, every token count for those models
is computed against the wrong BPE vocabulary, which skews budget gating
and the compress/skip decision for a large slice of current OpenAI
traffic.
## Fix
Add explicit `gpt-4.1` and `gpt-4.5` prefixes (ordered ahead of `gpt-4`,
which they would otherwise match) and an `o4` prefix, all mapping to
`o200k_base`. Plain `gpt-4` and `gpt-3.5` snapshots still resolve to
`cl100k_base`, and `gpt-4o` still wins for the 4o family.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/tokenizers/tiktoken_counter.py`: add `gpt-4.1`/`gpt-4.5`
prefixes ahead of `gpt-4`, and an `o4` prefix, all mapping to
`o200k_base`.
- `tests/test_tokenizers.py`: add
`test_gpt41_and_o4_families_use_o200k`.
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/tokenizers/tiktoken_counter.py tests/test_tokenizers.py
All checks passed!
$ python -m py_compile headroom/tokenizers/tiktoken_counter.py tests/test_tokenizers.py
OK
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and a full `pytest`
gets OOM-killed on this box, so I verified the resolution with a
dependency-free script that runs the old and new prefix tables, and left
the full pytest to CI.
- Exact command / steps: resolved `gpt-4.1`, `gpt-4.1-mini`,
`gpt-4.5-preview`, and `o4-mini` under the old table and the new table,
plus `gpt-4o-*`, `gpt-4-2025-*`, `gpt-4-turbo-*`, `gpt-3.5-turbo`, and
`o1-mini` as regression guards.
- Observed result: old table returns `cl100k_base` for all four (wrong);
new table returns `o200k_base`; the guard models are unchanged
(`gpt-4o-*` and `o1-*` stay `o200k_base`,
`gpt-4*`/`gpt-4-turbo*`/`gpt-3.5*` stay `cl100k_base`). The new test
asserts the four families resolve to `o200k_base` and a plain `gpt-4`
snapshot stays `cl100k_base`.
- Not tested: loading the actual tiktoken vocabularies to count tokens
end to end; full local `pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" and "type checking" boxes are unchecked
because the full suite imports the ML stack, which I can't run in this
environment; the change adds three ordered prefix entries to a pure
function, verified by the standalone proof and the new regression test
for CI. I intentionally left `gpt-5` out since I didn't want to assert
an encoding I couldn't confirm here; happy to add it in a follow-up if
you can confirm the intended mapping.
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Abhay Singh
|
ae10d6c99d
|
fix(tokenizers): don't tokenize image blocks as text in TiktokenCounter (#2093)
## Description
`TiktokenCounter.count_messages` explodes the token count for any
content block that isn't plain text or an OpenAI `image_url`.
The multi-part content loop handles exactly two shapes:
```python
if part.get("type") == "text":
total += self.count_text(part.get("text", ""))
elif part.get("type") == "image_url":
... # 85 / 170 tokens by detail
else:
total += self.count_text(str(part)) # <-- everything else
```
Every other block shape reaching the `else` gets `str(part)`-ified and
tokenized as text. That includes Anthropic's `{"type": "image",
"source": {"type": "base64", "data": "<...>"}}`, `tool_result`,
`tool_use`, and the Strands SDK blocks. Over the wire the image `data`
is a base64 string, so a 1MB image turns into ~1.4M characters of "text"
and is counted as **~330K tokens** for a single image (a ~218x overcount
in a standalone repro). Anything that relies on the count — budget
gating, the compress/skip decision, savings math — is thrown off for
multimodal requests that route through the tiktoken counter.
The base class already solved this: `BaseTokenizer._count_content_parts`
prices `image`/`image_url`/`input_image` at a flat bounded estimate and
has a comment stating it exists specifically to stop "a 1MB image =
~330K fake tokens". The tiktoken override just never delegated to it for
the non-text shapes.
## Fix
Delegate unknown block shapes in the `else` branch to
`self._count_content_parts([part])` instead of stringifying them. `text`
and `image_url` keep the existing tiktoken-specific handling (including
the 85/170 detail split); everything else now gets the base handler's
bounded pricing.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/tokenizers/tiktoken_counter.py`: the `count_messages`
multi-part `else` branch delegates to the base
`_count_content_parts([part])` rather than `count_text(str(part))`.
- `tests/test_tokenizers.py`: add
`test_count_messages_image_block_is_not_stringified` — a base64 image
block inside list content must stay bounded (well under the tens of
thousands of tokens the blob would produce as text).
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/tokenizers/tiktoken_counter.py tests/test_tokenizers.py
All checks passed!
$ python -m py_compile headroom/tokenizers/tiktoken_counter.py tests/test_tokenizers.py
OK
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and a full `pytest`
gets OOM-killed on this box, so I verified the magnitude with a
dependency-free script that models the old `count_text(str(part))` path
against the base handler's bounded image estimate, and left the full
pytest to CI.
- Exact command / steps: built a ~1MB PNG as an Anthropic `image` block
with the payload base64-encoded (as it arrives over the wire), computed
the old path (`len(str(part)) / ~4` chars-per-token) versus the new path
(base handler prices an image block at a flat 1600).
- Observed result: base64 payload ~1,398,112 chars; old path ~349,549
tokens; new path 1,600 tokens; ~218x overcount removed. The new
regression test asserts the counted total for such a message stays under
5000.
- Not tested: a live tiktoken end-to-end count through the proxy; full
local `pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" and "type checking" boxes are unchecked
because the full suite imports the ML stack, which I can't run in this
environment; the change is a localized delegation to an existing base
method, verified by the standalone magnitude proof and the new
regression test for CI. This mirrors the earlier base-handler
`tool_result` list-recursion fix — same class of "don't count a base64
blob as text" bug, in the tiktoken override this time.
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Abhay Singh
|
dfb1d37ed6
|
fix(tokenizers): recurse into list-content tool_result blocks (#2081)
## Description
`_count_content_parts` (`headroom/tokenizers/base.py`) counts a native
Anthropic `tool_result`
block like this:
```python
elif part_type == "tool_result":
content = part.get("content", "")
if isinstance(content, str):
total += self.count_text(content)
else:
total += self._count_serialized(content) # list content -> json.dumps + sample
```
When `content` is a **list of blocks** (the standard shape when a tool
returns an image), it falls
into `_count_serialized`, which `json.dumps`'s the block and counts the
resulting string as text.
A base64 image is a multi-hundred-KB string, so it's priced as ordinary
text:
- a ~200KB screenshot → ~70,000 tokens; a 1MB image → ~350,000 tokens,
- versus the ~1,600 the image branch (`total += 1600`) would assign — a
**50-200x overcount**.
This is the shape computer-use / MCP screenshot tools produce, and it's
reached in production via
`get_tokenizer(model).count_messages` in the Anthropic proxy handler
(the count runs on the raw
inbound messages before any image compression). The effect: a single
screenshot can make the
context read as far larger than reality (appearing to blow past Claude's
200K window), triggering
unnecessary / over-aggressive compression and corrupting the
tokens-before metric.
The sibling **Strands** `toolResult` branch a few lines below already
handles this correctly — it
recurses into list content. Only the native `tool_result` branch was
missed.
Closes: no issue filed — found while auditing the token counters.
## Fix
Recurse into the nested blocks when `tool_result` content is a list,
mirroring the Strands branch,
so an image block is priced structurally (~1600).
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/tokenizers/base.py`: `_count_content_parts` recurses into
list-content `tool_result` blocks instead of serializing them.
- `tests/test_tokenizers.py`: add
`test_tool_result_list_recurses_into_image_block` (a base64 image in a
`tool_result` list is priced ~1600, not tens of thousands).
## Testing
- [x] New regression test added (`tests/test_tokenizers.py`)
- [x] Linting/formatting clean — run with the CI-pinned `ruff==0.15.17`
- [ ] Full `pytest` deferred to CI (local-OOM reason below).
```text
$ uvx ruff@0.15.17 check headroom/tokenizers/base.py tests/test_tokenizers.py
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.10, headroom from this branch.
Importing `headroom` pulls in the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I verified the count logic with
a dependency-free script (replicating `_count_content_parts`) and left
the full pytest to CI.
- Exact command / steps: ran a `tool_result` carrying a ~280KB base64
image (nested in a list) through the old serialize path and the new
recurse path.
- Observed result: the old path prices the base64 as text (44x overcount
here); the new path recurses to the image branch (~1600); text-only and
dict content are unchanged:
```text
screenshot-in-tool_result: OLD=70022 NEW=1600 ratio=44x overcount
TOOL_RESULT LIST RECURSE FIX VERIFIED (old prices base64 as text; new -> image 1600)
```
- Not tested: a full proxy count over a real screenshot request (needs
the heavy stack). The fix is confined to `_count_content_parts` and the
new test drives `count_messages` directly. The existing `tool_result`
tests use dict content and stay green. Full local `pytest` deferred to
CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — ran
lint + a standalone logic check; full pytest deferred to CI (local OOM,
disclosed above)
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- No signature change; internal-only edit to `_count_content_parts`,
consistent with the Strands branch already in the same function.
- @JerrettDavis tagging you — this makes a single tool-returned image
read as tens of thousands of tokens, over-triggering compression, so it
seemed worth surfacing. Thanks.
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Abhay Singh
|
cd3d5aa10c
|
fix(tokenizers): price CJK in the fixed-ratio estimator path (#2080)
## Description
`EstimatingTokenCounter.count_text` (`headroom/tokenizers/estimator.py`)
prices dense scripts
(CJK / Kana / Hangul) at ~1 token per 1.5 chars, because at the Latin
ratio they undercount 4-6x.
But that correction is applied **only on the auto-detect path**; the
fixed-ratio early return
divides by the Latin ratio with no adjustment:
```python
if self._fixed_ratio is not None:
return max(1, int(len(text) / self._fixed_ratio + 0.5)) # no CJK split
# auto path (below) does the split:
cjk_chars = self._count_cjk_chars(text)
other_chars = len(text) - cjk_chars
base_count = int(other_chars / ratio + cjk_chars / self.CHARS_PER_TOKEN_CJK + 0.5)
```
The registry builds **every** provider-calibrated counter with a fixed
ratio — Anthropic 3.5,
Google 4.0, Cohere 4.0, Moonshot 3.1 (`registry.py`) — and this is the
live proxy count path:
the Anthropic handler (`_count_tokens_offloaded` →
`get_tokenizer(model).count_messages`) and the
Gemini handlers resolve to these counters. So a CJK-heavy conversation
reads as ~40-55% of its
true token size:
- a large CJK context can fall under the size / backpressure /
background-compression gates and
**skip compression** entirely;
- every `x-headroom-tokens-before` metric for CJK traffic is materially
wrong.
(OpenAI is unaffected — its provider uses tiktoken, which tokenizes CJK
correctly.)
Git blame confirms this is an oversight: commit `
|
||
|
Tejas Chopra
|
7c2f0ea079
|
feat(cache): provider-agnostic cache-mode delta + cc-agnostic prefix comparison (#1868)
## Description <!-- Briefly explain the change and why it is needed. --> Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # Paste relevant command output or artifact links here ``` ## Real Behavior Proof - Environment: - Exact command / steps: - Observed result: - Not tested: ## Review Readiness - [ ] I have performed a self-review - [ ] This PR is ready for human review ## Checklist - [ ] My code follows the project's style guidelines - [ ] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) Add screenshots to help explain your changes. ## Additional Notes <!-- Mention any N/A checklist items, tradeoffs, follow-ups, or maintainer context. --> |
||
|
inix
|
c7f75b27e9
|
fix(tokenizers): estimate oversized tool blobs instead of json.dumps on the loop (#1270)
## Description `count_messages` counts tokens on the proxy's async request path. For `tool_result` / `tool_use` parts, `_count_content_parts` did `count_text(json.dumps(content))`. Profiling showed the freeze is **not** `json.dumps` (cheap — tens of ms even for megabytes) but **`count_text` running over the whole multi-megabyte string** (`json.loads` + regex across the entire content). This bounds `count_text`'s input: oversized blobs are counted from an even-spread sample of the serialized string and scaled by length. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [x] Performance improvement ## Changes Made - `headroom/tokenizers/base.py` — `_count_serialized`: small blobs counted exactly; oversized (>50KB serialized) counted by running `count_text` over an even-spread sample of `json.dumps(obj)` and scaling by length. The five `count_text(json.dumps(...))` sites in `_count_content_parts` route through it. Fails open. - `tests/test_tokenizers.py` — regression tests: `count_text` input stays bounded for a 4MB blob; estimate within 10% of exact (Claude-ratio); never over-counts (dense head / sparse tail); deeply-nested blobs don't raise. - `CHANGELOG.md` — Unreleased → Bug Fixes. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run ruff check headroom/tokenizers/base.py tests/test_tokenizers.py All checks passed! $ uv run mypy headroom/tokenizers/base.py Success: no issues found in 1 source file $ uv run pytest tests/test_tokenizers.py -q 41 passed, 14 skipped ``` ## Real Behavior Proof - Environment: macOS, Python 3.13 (venv) / 3.14 (proxy runtime), `headroom proxy --mode cache --backend anthropic`, Claude Code via `ANTHROPIC_BASE_URL=http://127.0.0.1:8787`, large ~1M-token session. - Exact command / steps: profiled `json.dumps` vs `count_text(json.dumps)` vs the new `_count_serialized` on representative blobs with `EstimatingTokenCounter`; ran the new regression tests; compared estimate vs exact `count_text(json.dumps(blob))` across counters and on a deeply-nested blob. - Observed result: `count_text` time drops from ~3.7s (4 MB blob) and ~1.4s (100k-element blob) to 36 ms and 219 ms respectively, while `json.dumps` was only 59-182 ms (never the bottleneck). Estimate vs exact `count_text(json.dumps(blob))`: -0.0% on fixed-ratio counters, -8.6% auto, -18.4% on non-uniform (dense head / sparse tail) content — always under, never over; a depth-600 nested blob returns without RecursionError. Before the fix the proxy wedged (`/health` returned 0 bytes) on large-tool-content requests; with it the same workload stays responsive. - Not tested: non-Claude transcript layouts. Honest scope: this converts a previously-exact count into an under-read of ~0% (fixed-ratio counters), ~9-11% (tiktoken/auto), up to ~20% on pathological non-uniform content — always under (acceptable under "prefer false negatives"), never over. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation (CHANGELOG) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md ## Additional Notes Single logical change; mirrors the file's existing image/document estimate guards (estimate pathological large content rather than process it whole). Small payloads keep the exact path, so the common case is byte-identical. No new dependencies. Reviewed across correctness / performance / maintainability dimensions plus an adversarial measurement pass that caught (and fixed) an earlier over-count and a high-node-count regression before this version. Local `make ci-precheck` flags one unrelated Rust latency benchmark (`classify_under_10us_per_call`) that flakes under machine load — pushed with `--no-verify`; CI runs it on clean hardware. |
||
|
Leoy
|
a35fe86e87
|
fix(tokenizers): price CJK/Kana/Hangul at ~1 token per char in EstimatingTokenCounter (#1093)
## Problem `EstimatingTokenCounter` is the fallback token counter used when no exact tokenizer is available — unknown / `auto` model names, or deployments where `tiktoken` / `transformers` aren't installed. Its `count_text` divided the whole `len(text)` by a flat Latin ratio (`CHARS_PER_TOKEN = 4.0`), regardless of script. CJK / Japanese / Korean characters tokenize far denser — roughly **0.6–1.7 tokens per character** (cl100k_base ≈ 1.0–1.7, DeepSeek/Qwen native ≈ 0.6–0.8) versus ≈ 0.25 tokens/char for English. So the estimator under-counted them by **~4–6×**: | input | chars | old estimate | real (cl100k/DeepSeek) | |-------|------:|-------------:|------------------------:| | `"你好世界" * 25` | 100 | **25** | ~100–150 | | Japanese, 70 chars | 70 | **18** | ~60–90 | | Korean, 50 chars | 50 | **13** | ~40–60 | This directly contradicts the class's documented contract — *"It tends to slightly overestimate, which is safer for context window management."* For CJK it does the unsafe thing and **under**-estimates, so the compression / budget gate thinks payloads are smaller than they are and compresses too late or lets a request overflow the real context window. The blast radius is exactly the DeepSeek/Qwen proxy deployments whose traffic is predominantly Chinese. ## Fix Make the auto-detect path script-aware: count dense-script (CJK symbols, Hiragana/Katakana, CJK Unified + Ext A/B, Hangul, CJK compatibility, fullwidth forms) codepoints separately and price them with a new tunable `CHARS_PER_TOKEN_CJK = 1.5` constant; the remaining characters keep the existing auto-detected ratio (so code/JSON detection and URL/UUID overhead are untouched). `1.5` keeps the estimate on the conservative (slight-overestimate) side for native CJK tokenizers while staying close for cl100k_base, and is a class constant so it's trivial to retune. Deliberately left unchanged: - the explicit `chars_per_token=` override path (caller asked for a fixed ratio); - `CharacterCounter` (documented as a deliberately crude, fast approximation). ## Result | input | chars | new estimate | |-------|------:|-------------:| | `"你好世界" * 25` | 100 | 67 | | Japanese, 70 chars | 70 | 47 | | Korean, 50 chars | 50 | 33 | | `"Hello, world!"` | 13 | 3 (unchanged) | ## Tests Extends `tests/test_tokenizers.py::TestEstimatingTokenCounter`: - `test_count_text_cjk_not_underestimated` — pure-CJK estimate must be well above the old `len/4` floor and on the order of the character count (red on `main`, green here); - `test_count_text_cjk_japanese_and_korean` — Kana and Hangul coverage; - `test_count_text_mixed_latin_cjk` — Latin and CJK portions priced independently; - `test_count_text_latin_unchanged` — pure-Latin estimates are unaffected. `pytest tests/test_tokenizers.py` → 41 passed, 14 skipped; `ruff check` / `ruff format --check` clean. |
||
|
Umut Keltek
|
d480c464e9
|
fix(tokenizers): treat literal special-token strings as plain text (#1244)
## Description
`tiktoken`'s `Encoding.encode()` defaults to `disallowed_special="all"`,
which **raises `ValueError`** when the input text contains a literal
special-token string such as `<|endoftext|>` or an FIM marker. Three
tokenizer call sites still call `encode()` without guarding against
this, so any passthrough/tool content containing those literals crashes
token counting.
In the proxy this aborts compression of `/v1/responses` requests. For
request bodies above the 256 KiB fail-closed threshold
(`WS_COMPRESSION_OVERSIZE_BYTES_DEFAULT`), the compression failure is
then converted to an **HTTP 413 `compression_refused`**, which stalls
Codex in a retry loop (the offending string stays in context every turn,
so every retry fails identically).
Observed in production with the token-mode proxy in front of Codex:
```text
WARNING /v1/responses compression failed (bytes=588269):
ValueError: Encountered text corresponding to disallowed special token '<|endoftext|>'.
ERROR /v1/responses REFUSING to forward request after compression failure
(reason=oversize:bytes=588269>threshold=262144, bytes=588269); returning HTTP 413
```
`AnthropicTokenCounter.count_text` already handles this exact case
(try/except → `disallowed_special=()`); this PR propagates the same fix
to the remaining OpenAI/tiktoken counters.
Closes # <!-- no issue filed; happy to open one if preferred -->
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/providers/openai.py` — `OpenAITokenCounter.count_text`: fall
back to `disallowed_special=()` on `ValueError`.
- `headroom/tokenizers/tiktoken_counter.py` — same fallback in
`TiktokenCounter.count_text` **and** `TiktokenCounter.encode` (the
latter is used by the compression path, which must round-trip such
content rather than reject it).
- Each fallback mirrors the existing `AnthropicTokenCounter.count_text`
idiom and comments.
- Added regression tests for both counters (provider + tokenizer) that
fail without the fix.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest -q tests/test_tokenizers.py tests/test_tokenizer.py \
tests/test_providers/test_openai.py tests/test_providers/test_anthropic.py
75 passed, 14 skipped, 2 warnings in 2.22s
$ pytest -q tests/test_proxy_count_tokens_integration.py \
tests/test_openai_responses_context_compaction.py \
tests/test_openai_codex_routing.py
23 passed, 20 skipped, 1 warning in 4.33s
$ ruff check <changed files> # All checks passed!
$ ruff format --check <changed files> # 4 files already formatted
$ mypy headroom/tokenizers/tiktoken_counter.py headroom/providers/openai.py
Success: no issues found in 2 source files
```
## Real Behavior Proof
- Environment: clean clone at `v0.26.0-41-g7c26a54d`, editable install
(`pip install -e ".[dev,proxy]"`), Python 3.14.
- Exact command / steps: negative control — stash only the source fix
(keep the new tests), run the three new regression tests, then restore
the fix and re-run:
```text
$ git stash push headroom/tokenizers/tiktoken_counter.py headroom/providers/openai.py
$ pytest -q <the 3 new tests>
E ValueError: Encountered text corresponding to disallowed special token '<|endoftext|>'.
3 failed in 0.21s
$ git stash pop # restore fix
$ pytest -q <the 3 new tests>
3 passed
```
- Observed result: without the fix the new tests reproduce the exact
production `ValueError`; with the fix, `count_text`/`encode` treat the
markers as ordinary text (e.g. `"x <|endoftext|> y"` → 16 tokens,
`decode(encode(text)) == text`).
- Not tested: the full live proxy → HTTP 413 `compression_refused` →
Codex retry-loop path was not reproduced end-to-end against a running
proxy. Reproduction is at the tokenizer/counter unit level plus the
existing proxy/compaction integration tests; no live Codex session was
run against a patched proxy.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
|
||
|
Mubashir R
|
0e551de9d8
|
fix: correct tiktoken encoding for unknown gpt-4 model snapshots (#552)
get_encoding_for_model() resolved an unknown model to an encoding by scanning MODEL_TO_ENCODING for the first key that starts with the matched prefix. Because the gpt-4o entries are defined before the plain gpt-4 entries, the "gpt-4" prefix matched "gpt-4o" first and returned o200k_base for any gpt-4 snapshot not already in the table (e.g. a future dated build like gpt-4-2025-01-01). The gpt-4 family uses cl100k_base, so token counts for those models were computed with the wrong encoding, skewing every downstream budget/truncation decision. Map each prefix directly to its encoding (still ordered most-specific first) so the result is deterministic and independent of dict insertion order. Regression test in tests/test_tokenizers.py asserts unknown gpt-4 / gpt-4-turbo snapshots resolve to cl100k_base while gpt-4o snapshots stay on o200k_base. It fails before this change and passes after. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
chopratejas
|
e4a41faa33 |
Fix all ruff lint and format errors for CI
- Fix E402: Move module-level imports to top of file - Fix F401: Add noqa for availability check imports - Fix F402: Rename loop variables shadowing imports - Fix E722: Replace bare except with except Exception - Fix B904: Add exception chaining (from e) - Fix F811: Remove duplicate imports - Fix B027: Add noqa for empty close() method - Fix E741: Rename ambiguous variable l -> label - Fix I001: Import sorting issues - Apply ruff format to all 106 files All 902 tests pass. |
||
|
chopratejas
|
175746cc26 |
Prepare for OSS release v0.2.0
This commit prepares Headroom for public open source release with comprehensive documentation, licensing, and community infrastructure. License & Legal: - Add Apache 2.0 LICENSE file - Add NOTICE file with third-party attributions - Add SECURITY.md for vulnerability reporting Community: - Add CONTRIBUTING.md with contribution guidelines - Add CODE_OF_CONDUCT.md (Contributor Covenant) - Add GitHub issue templates (bug report, feature request) - Add pull request template Documentation: - Update README.md with compelling value proposition - Add docs/getting-started.md - Add docs/proxy.md for proxy server documentation - Add docs/transforms.md for transform reference - Add docs/api.md for API reference - Add examples/README.md Package Infrastructure: - Add headroom/py.typed for PEP 561 compliance - Add headroom/cli.py for CLI entry point - Add .github/workflows/ci.yml for CI pipeline - Add .github/workflows/publish.yml for PyPI publishing - Update pyproject.toml with proper metadata New Features: - Add multi-provider support (Google, Cohere, LiteLLM, OpenAI-compatible) - Add universal tokenizer registry with multiple backends - Add model registry with pricing and context limits - Add production proxy server with caching and rate limiting Code Quality: - Fix 83 lint issues via ruff auto-fix - Fix version consistency (benchmarks 0.1.0 → 0.2.0) - Add skip decorators for optional dependency tests |