headroom/headroom/providers/anthropic.py
Abhay Singh 3ed8f76019
fix(providers): don't crash on a non-object HEADROOM_MODEL_LIMITS / models.json (#3089)
## Description

`_load_custom_model_config` in both `headroom/providers/anthropic.py`
and `headroom/providers/openai.py` loads the operator's custom model
configuration from `HEADROOM_MODEL_LIMITS` (a JSON string or a file
path) and `~/.headroom/models.json`, then reads it with
`loaded.get(...)`:

```python
loaded = json.loads(env_config)          # or json.load(f)
anthropic_config = loaded.get("anthropic", loaded)
```

The `try` guards only `except (json.JSONDecodeError, OSError)`. When the
value is **valid JSON but not an object** (a JSON array, number, string,
bool, or `null`), `json.loads` succeeds and returns a non-dict, so
`loaded.get(...)` raises `AttributeError` — which is *not* one of the
caught types. Instead of the intended warn-and-fall-back-to-defaults, a
misconfigured `HEADROOM_MODEL_LIMITS` (e.g.
`HEADROOM_MODEL_LIMITS='[1,2,3]'` or `'"gpt-4"'`) crashes provider
initialization. The same gap exists in the `models.json` branch of both
providers.

## Fix

After each load, validate `isinstance(loaded, dict)` and raise
`ValueError` with a clear message, and broaden the handler from `except
(json.JSONDecodeError, OSError)` to `except (ValueError, OSError)`.
`json.JSONDecodeError` is a subclass of `ValueError`, so this strictly
supersets the previous handling: every previously-caught malformed value
still warns and falls back, and a valid-JSON-but-non-object value now
does too, instead of crashing.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/providers/anthropic.py` and `headroom/providers/openai.py`
(`_load_custom_model_config`): add an `isinstance(loaded, dict)` guard
(raising `ValueError`) after the env-var load and after the
`models.json` load, and change both `except` clauses to `(ValueError,
OSError)`.
- `tests/test_provider_model_fallback.py`: added parametrized
`test_non_object_env_var_falls_back_to_defaults` (array / string /
number / bool / null) for both providers, and
`test_non_object_config_file_falls_back_to_defaults` for a non-object
`models.json`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added

### Test Output

```text
tests/test_provider_model_fallback.py  44 passed
# uvx ruff@0.15.22 check  -> All checks passed!
# uvx mypy@1.20.2 headroom/providers/anthropic.py headroom/providers/openai.py -> Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.15.22 and mypy 1.20.2 via uvx.
- Exact command / steps: reverted both providers and ran the new
regressions to capture the bug (`python -m pytest
tests/test_provider_model_fallback.py::TestAnthropicConfigLoading::test_non_object_env_var_falls_back_to_defaults
tests/test_provider_model_fallback.py::TestOpenAIConfigLoading::test_non_object_env_var_falls_back_to_defaults
tests/test_provider_model_fallback.py::TestAnthropicConfigLoading::test_non_object_config_file_falls_back_to_defaults`
-> 11 failed with `AttributeError` on `loaded.get` across the
array/string/number/bool/null shapes); restored the fix; re-ran the full
file (`python -m pytest tests/test_provider_model_fallback.py` -> 44
passed); then `uvx ruff@0.15.22 format`, `uvx ruff@0.15.22 check`, and
`uvx mypy@1.20.2` on both providers.
- Observed result: before the fix, `HEADROOM_MODEL_LIMITS='[1,2,3]'` (or
`'"gpt-4"'`, `'42'`, `'true'`, `'null'`) raised `AttributeError` out of
`_load_custom_model_config`; after the fix the same values log a warning
and the loader returns the default `{"context_limits": {}, "pricing":
{}[, "encodings": {}]}`, and a well-formed object config is unchanged.
- Not tested: a live proxy boot with a corrupt `HEADROOM_MODEL_LIMITS`
(the loader is exercised directly, which is the exact function provider
init calls).

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This is defensive parsing in the
provider model-config loader, not a rollout-channel-gated runtime
feature.
- Minimum rollout channel: N/A (no rollout-managed behavior).
- Stable/default behavior changed: only for a previously-crashing input.
A non-object `HEADROOM_MODEL_LIMITS` / `models.json` now warns and uses
built-in defaults instead of raising. Well-formed object configs are
parsed exactly as before.
- Kill switch / disable path: N/A — remove or correct the malformed
config value to load custom limits.
- Unsafe override required: no.
- Qualification impact: a corrupt or mistyped model-limits value
degrades to built-in defaults with a warning rather than failing
provider init.
- Rollback path: revert this PR; the loader returns to catching only
`json.JSONDecodeError`/`OSError`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`: it is generated by
release-please from my Conventional Commit PR title

## Additional Notes

Both providers carry the same loader shape, so the guard and the widened
`except` are applied identically to keep them in sync. The message names
the offending source (`HEADROOM_MODEL_LIMITS` vs the resolved
config-file path) so the warning is actionable.
2026-08-17 20:18:58 -07:00

815 lines
32 KiB
Python
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

"""Anthropic provider implementation for Headroom SDK.
Token counting uses Anthropic's official Token Count API when a client
is provided. This gives accurate counts for all content types including
JSON, non-English text, and tool definitions.
Usage:
from anthropic import Anthropic
from headroom import AnthropicProvider
client = Anthropic() # Uses ANTHROPIC_API_KEY env var
provider = AnthropicProvider(client=client) # Accurate counting via API
# Or without client (uses tiktoken approximation - less accurate)
provider = AnthropicProvider() # Warning: approximate counting
"""
import importlib.util
import json
import logging
import os
import re
import warnings
from typing import Any, cast
from headroom import paths as _paths
from headroom.pricing.litellm_pricing import estimate_cost_from_tokens
from headroom.tokenizers.base import (
TokenCountCache,
coerce_countable_text,
count_content_blocks,
)
from .base import Provider, TokenCounter
LITELLM_AVAILABLE = importlib.util.find_spec("litellm") is not None
def _get_litellm_clients() -> tuple[Any | None, Any | None]:
"""Import LiteLLM only when pricing/context metadata is needed."""
if not LITELLM_AVAILABLE:
return None, None
try:
import litellm
litellm.suppress_debug_info = True
litellm.set_verbose = False
from litellm import get_model_info as litellm_get_model_info
except ImportError:
return None, None
return litellm, litellm_get_model_info
logger = logging.getLogger(__name__)
# Warning flags
_FALLBACK_WARNING_SHOWN = False
_UNKNOWN_MODEL_WARNINGS: set[str] = set()
_ANSI_ESCAPE_RE = re.compile(r"\x1b(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])")
_DANGLING_ANSI_STYLE_SUFFIX_RE = re.compile(r"(?:\[[0-9;]*m\])+$")
def sanitize_anthropic_model_id(model: str) -> str:
"""Return an Anthropic model id without terminal styling artifacts."""
cleaned = _ANSI_ESCAPE_RE.sub("", str(model)).strip()
return _DANGLING_ANSI_STYLE_SUFFIX_RE.sub("", cleaned)
# `[1m]` is not only an ANSI artifact: Claude Code appends it to a model id to
# request the 1M context tier, and only then sends the `context-1m` beta header
# (#1158). Upstream rejects the suffix, so `sanitize_anthropic_model_id()` must
# keep stripping it before forwarding (#2027) — but the tier it encodes has to
# be read off the id *before* that happens, or a 1M request gets budgeted as if
# it were the base model's window.
_CONTEXT_1M_SUFFIX_RE = re.compile(r"(?:\[1m\])+$")
CONTEXT_1M_TOKENS = 1_000_000
def has_context_1m_suffix(model: str) -> bool:
"""Return True if ``model`` carries Claude Code's ``[1m]`` 1M-tier marker."""
return bool(_CONTEXT_1M_SUFFIX_RE.search(_ANSI_ESCAPE_RE.sub("", str(model)).strip()))
def sanitize_anthropic_model_metadata(value: Any) -> Any:
"""Strip model-id styling artifacts from Anthropic model metadata payloads."""
if isinstance(value, list):
return [sanitize_anthropic_model_metadata(item) for item in value]
if not isinstance(value, dict):
return value
cleaned: dict[str, Any] = {}
for key, item in value.items():
if key in {"id", "model"} and isinstance(item, str):
cleaned[key] = sanitize_anthropic_model_id(item)
else:
cleaned[key] = sanitize_anthropic_model_metadata(item)
return cleaned
# Anthropic model context limits
# All Claude 3+ models have 200K context
ANTHROPIC_CONTEXT_LIMITS: dict[str, int] = {
# Claude Fable 5 - 1M context
"claude-fable-5": 1000000,
# Claude Opus 4.8 - 1M context
"claude-opus-4-8": 1000000,
# Claude 4.7 (Opus 4.7) - 1M context
"claude-opus-4-7": 1000000,
# Claude 4.6 (Opus 4.6) - 1M context
"claude-opus-4-6": 1000000,
# Claude 4.5 (Opus 4.5)
"claude-opus-4-5-20251101": 200000,
# Claude Sonnet 5 - 1M context
"claude-sonnet-5": 1000000,
# Claude Sonnet 4.6 - 1M context window
"claude-sonnet-4-6": 1000000,
# Claude Sonnet 4.5
"claude-sonnet-4-5": 200000,
# Claude 4 (Sonnet 4, Haiku 4)
"claude-sonnet-4-20250514": 200000,
"claude-haiku-4-5-20251001": 200000,
# Claude 3.5
"claude-3-5-sonnet-20241022": 200000,
"claude-3-5-sonnet-latest": 200000,
"claude-3-5-haiku-20241022": 200000,
"claude-3-5-haiku-latest": 200000,
# Claude 3
"claude-3-opus-20240229": 200000,
"claude-3-opus-latest": 200000,
"claude-3-sonnet-20240229": 200000,
"claude-3-haiku-20240307": 200000,
# Claude 2
"claude-2.1": 200000,
"claude-2.0": 100000,
"claude-instant-1.2": 100000,
}
# Fallback pricing - LiteLLM is preferred source
# NOTE: These are ESTIMATES. Always verify against actual Anthropic billing.
# Last updated: 2026-07-04
ANTHROPIC_PRICING: dict[str, dict[str, float]] = {
# Claude Fable 5 (anthropic.com/pricing): $10 in / $50 out, cache read $1.
"claude-fable-5": {"input": 10.00, "output": 50.00, "cached_input": 1.00},
# Claude Opus 4.8 — current Opus tier: $5 in / $25 out, cache read $0.50.
"claude-opus-4-8": {"input": 5.00, "output": 25.00, "cached_input": 0.50},
# Claude 4.7 (current Opus tier)
"claude-opus-4-7": {"input": 5.00, "output": 25.00, "cached_input": 0.50},
# Claude 4.6 (current Opus tier)
"claude-opus-4-6": {"input": 5.00, "output": 25.00, "cached_input": 0.50},
# Claude 4.5 (current Opus tier — same rates as 4.64.8)
"claude-opus-4-5-20251101": {"input": 5.00, "output": 25.00, "cached_input": 0.50},
# Claude Sonnet 5 / 4.6 / 4.5 (current Sonnet tier): $3 in / $15 out, cache read $0.30
"claude-sonnet-5": {"input": 3.00, "output": 15.00, "cached_input": 0.30},
"claude-sonnet-4-6": {"input": 3.00, "output": 15.00, "cached_input": 0.30},
"claude-sonnet-4-5": {"input": 3.00, "output": 15.00, "cached_input": 0.30},
# Claude 4 (Sonnet/Haiku tier pricing)
"claude-sonnet-4-20250514": {"input": 3.00, "output": 15.00, "cached_input": 0.30},
"claude-haiku-4-5-20251001": {"input": 1.00, "output": 5.00, "cached_input": 0.10},
# Claude 3.5
"claude-3-5-sonnet-20241022": {"input": 3.00, "output": 15.00, "cached_input": 0.30},
"claude-3-5-sonnet-latest": {"input": 3.00, "output": 15.00, "cached_input": 0.30},
"claude-3-5-haiku-20241022": {"input": 0.80, "output": 4.00, "cached_input": 0.08},
"claude-3-5-haiku-latest": {"input": 0.80, "output": 4.00, "cached_input": 0.08},
# Claude 3
"claude-3-opus-20240229": {"input": 15.00, "output": 75.00, "cached_input": 1.50},
"claude-3-opus-latest": {"input": 15.00, "output": 75.00, "cached_input": 1.50},
"claude-3-sonnet-20240229": {"input": 3.00, "output": 15.00, "cached_input": 0.30},
"claude-3-haiku-20240307": {"input": 0.25, "output": 1.25, "cached_input": 0.03},
}
# Anthropic's long-context premium. On models that reach 1M over a 200K base,
# a prompt above 200K re-prices the *entire* request -- input, output and cache
# alike -- rather than only the tokens past the threshold. Multipliers are
# derived from LiteLLM's `*_above_200k_tokens` fields ($3->$6 in, $15->$22.50
# out, $0.30->$0.60 cache read).
#
# Only the Sonnet 4 / 4.5 family is tiered: Opus, and Sonnet 4.6 onward, are
# flat-rated across their whole window. This is the same population that needs
# the `[1m]` suffix to reach 1M at all, so a session that fills the window this
# unlocks is billed at these rates.
_LONG_CONTEXT_THRESHOLD = 200_000
_LONG_CONTEXT_PREMIUM: dict[str, float] = {"input": 2.0, "output": 1.5, "cached_input": 2.0}
_LONG_CONTEXT_TIERED_MODELS = (
"claude-sonnet-4-5",
"claude-sonnet-4-20250514",
"claude-4-sonnet-20250514",
)
def _apply_long_context_premium(
model: str, pricing: dict[str, float], input_tokens: int
) -> dict[str, float]:
"""Return ``pricing`` scaled by the long-context premium where it applies.
Used only on the manual fallback path; the LiteLLM path already applies the
published above-threshold rates itself.
"""
if input_tokens <= _LONG_CONTEXT_THRESHOLD:
return pricing
if not any(model.startswith(tiered) for tiered in _LONG_CONTEXT_TIERED_MODELS):
return pricing
return {key: rate * _LONG_CONTEXT_PREMIUM.get(key, 1.0) for key, rate in pricing.items()}
# Default limits for pattern-based inference
# Used when a model isn't in the explicit list but matches a known pattern
_PATTERN_DEFAULTS = {
"opus": {"context": 200000, "pricing": {"input": 5.00, "output": 25.00, "cached_input": 0.50}},
"sonnet": {
"context": 200000,
"pricing": {"input": 3.00, "output": 15.00, "cached_input": 0.30},
},
"haiku": {"context": 200000, "pricing": {"input": 0.80, "output": 4.00, "cached_input": 0.08}},
}
# Fallback for completely unknown Claude models
_UNKNOWN_CLAUDE_DEFAULT = {
"context": 200000, # Safe assumption for Claude 3+
"pricing": {"input": 3.00, "output": 15.00, "cached_input": 0.30}, # Sonnet-tier pricing
}
# DeepSeek fallback pricing for --anthropic-api-url deepseek routing
_DEEPSEEK_FALLBACK_PRICING: dict[str, dict[str, float]] = {
"deepseek-v4-flash": {"input": 0.14, "output": 0.28, "cached_input": 0.0028},
"deepseek-v4-pro": {"input": 0.435, "output": 0.87, "cached_input": 0.003625},
}
def _get_deepseek_pricing(model: str) -> dict[str, float] | None:
"""Get fallback pricing for a DeepSeek model.
Used when the Anthropic provider encounters a deepseek-* model name
(via --anthropic-api-url pointing at DeepSeek's Anthropic-compatible
endpoint) and LiteLLM is unavailable.
Args:
model: The model name to look up.
Returns:
Pricing dict with input/output/cached_input keys, or None.
"""
# Direct match
if model in _DEEPSEEK_FALLBACK_PRICING:
return cast(dict[str, float], _DEEPSEEK_FALLBACK_PRICING[model])
# Partial match
for known_model, prices in _DEEPSEEK_FALLBACK_PRICING.items():
if model in known_model or known_model in model:
return cast(dict[str, float], prices)
return None
def _load_custom_model_config() -> dict[str, Any]:
"""Load custom model configuration from environment or config file.
Checks (in order):
1. HEADROOM_MODEL_LIMITS environment variable (JSON string or file path)
2. ~/.headroom/models.json config file
Returns:
Dict with 'context_limits' and 'pricing' keys.
"""
config: dict[str, Any] = {"context_limits": {}, "pricing": {}}
# Check environment variable
env_config = os.environ.get("HEADROOM_MODEL_LIMITS", "")
if env_config:
try:
# Check if it's a file path
if os.path.isfile(env_config):
with open(env_config, encoding="utf-8") as f:
loaded = json.load(f)
else:
# Try to parse as JSON string
loaded = json.loads(env_config)
if not isinstance(loaded, dict):
raise ValueError(
f"HEADROOM_MODEL_LIMITS must be a JSON object, got {type(loaded).__name__}"
)
# Check for anthropic-specific config, fall back to root level
anthropic_config = loaded.get("anthropic", loaded)
if "context_limits" in anthropic_config:
config["context_limits"].update(anthropic_config["context_limits"])
if "pricing" in anthropic_config:
config["pricing"].update(anthropic_config["pricing"])
logger.debug(f"Loaded custom model config from HEADROOM_MODEL_LIMITS: {loaded}")
except (ValueError, OSError) as e:
# ValueError covers json.JSONDecodeError (a subclass) and the
# non-object guard above, so a malformed value warns and falls back
# to defaults instead of crashing provider init.
logger.warning(f"Failed to load HEADROOM_MODEL_LIMITS: {e}")
# Check config file. Prefer the canonical config-dir location, then fall
# back to the legacy workspace-root location for backward compatibility.
config_file = _paths.models_config_path()
if not config_file.exists():
legacy_models = _paths.workspace_dir() / "models.json"
if legacy_models.exists():
config_file = legacy_models
if config_file.exists():
try:
with open(config_file, encoding="utf-8") as f:
loaded = json.load(f)
if not isinstance(loaded, dict):
raise ValueError(f"{config_file} must contain a JSON object")
# Only load anthropic-specific config
anthropic_config = loaded.get("anthropic", loaded)
if "context_limits" in anthropic_config:
# Don't override env var settings
for model, limit in anthropic_config["context_limits"].items():
if model not in config["context_limits"]:
config["context_limits"][model] = limit
if "pricing" in anthropic_config:
for model, pricing in anthropic_config["pricing"].items():
if model not in config["pricing"]:
config["pricing"][model] = pricing
logger.debug(f"Loaded custom model config from {config_file}")
except (ValueError, OSError) as e:
logger.warning(f"Failed to load {config_file}: {e}")
return config
def _infer_model_tier(model: str) -> str | None:
"""Infer the model tier (opus/sonnet/haiku) from model name.
Uses pattern matching to handle future model releases.
"""
model_lower = model.lower()
# Check for tier keywords in model name
if "opus" in model_lower:
return "opus"
elif "sonnet" in model_lower:
return "sonnet"
elif "haiku" in model_lower:
return "haiku"
return None
class AnthropicTokenCounter(TokenCounter):
"""Token counter for Anthropic models.
When an Anthropic client is provided, uses the official Token Count API
(/v1/messages/count_tokens) for accurate counting. This handles:
- JSON-heavy tool payloads
- Non-English text
- Tool definitions and structured content
Falls back to tiktoken approximation only when no client is available.
"""
def __init__(self, model: str, client: Any = None, warn: bool = True):
"""Initialize token counter.
Args:
model: Anthropic model name.
client: Optional anthropic.Anthropic client for API-based counting.
If not provided, falls back to tiktoken approximation.
warn: If False, suppresses the no-client UserWarning (useful for
internal proxy usage where approximation is intentional).
"""
global _FALLBACK_WARNING_SHOWN
self.model = model
self._client = client
self._encoding: Any = None
self._count_cache = TokenCountCache()
self._use_api = client is not None
if not self._use_api and warn and not _FALLBACK_WARNING_SHOWN:
warnings.warn(
"AnthropicProvider: No client provided, using tiktoken approximation. "
"For accurate counting, pass an Anthropic client: "
"AnthropicProvider(client=Anthropic())",
UserWarning,
stacklevel=4,
)
_FALLBACK_WARNING_SHOWN = True
# Load tiktoken as fallback — bounded, so a stalled vocab download can't
# hang token counting inside a request (tiktoken's downloader has no
# network timeout); on timeout we estimate by characters instead (GH #956).
try:
from headroom.tokenizers.tiktoken_counter import (
TiktokenLoadError,
load_encoding,
)
self._encoding = load_encoding("cl100k_base")
except TiktokenLoadError:
self._encoding = None # count_text() falls back to a character estimate
except ImportError:
if not self._use_api:
warnings.warn(
"tiktoken not installed - token counting will be very approximate. "
"Install tiktoken or provide an Anthropic client.",
UserWarning,
stacklevel=4,
)
def count_text(self, text: str) -> int:
"""Count tokens in text.
Note: For single text strings, uses tiktoken approximation even when
API is available (API only supports full message counting).
"""
if not text:
return 0
cached = self._count_cache.get(text)
if cached is not None:
return cached
count = self._count_text_uncached(text)
self._count_cache.put(text, count)
return count
def _count_text_uncached(self, text: str) -> int:
if self._encoding:
# tiktoken with ~1.1x multiplier for Claude
try:
base_count = len(self._encoding.encode(text))
except ValueError:
# Real tool output can legitimately contain strings that look like
# tiktoken special tokens (for example FIM markers in code spans).
# Treat them as ordinary text for estimation instead of failing.
base_count = len(self._encoding.encode(text, disallowed_special=()))
return int(base_count * 1.1)
# Character-based fallback
return max(1, len(text) // 3)
def count_message(self, message: dict[str, Any]) -> int:
"""Count tokens in a single message.
Uses API if available, otherwise falls back to estimation.
"""
if self._use_api:
return self._count_message_via_api(message)
return self._count_message_estimated(message)
def _count_message_via_api(self, message: dict[str, Any]) -> int:
"""Count tokens using Anthropic Token Count API."""
try:
# Convert to Anthropic message format if needed
messages = [self._normalize_message(message)]
response = self._client.messages.count_tokens(
model=self.model,
messages=messages,
)
return int(response.input_tokens)
except Exception:
# Fall back to estimation on API error
return self._count_message_estimated(message)
def _count_message_estimated(self, message: dict[str, Any]) -> int:
"""Estimate token count without API."""
tokens = 4 # Role overhead
content = message.get("content")
if isinstance(content, str):
tokens += self.count_text(content)
elif isinstance(content, list):
# Delegate to the audited shared walker instead of a partial
# per-provider one. Each provider counter had grown its own
# shortened branch list, so every modern block priced at ~0:
# measured on a 6,800-char block this returned 8 tokens for
# tool_result, thinking, document, mcp_tool_result — and for
# output_text / refusal, which are OpenAI's OWN Responses shapes.
# The shared walker is also image-safe: a 200KB base64 image gets
# a pixel-based 1600, not the ~50K phantom text tokens a naive
# str(block) catch-all would produce.
tokens += count_content_blocks(content, self.count_text)
# OpenAI format tool calls. Guard the value, not just the key: an
# OpenAI-format assistant message often carries `tool_calls: null` on a
# no-tool turn, and `for ... in None` would raise TypeError.
for tool_call in message.get("tool_calls") or []:
if isinstance(tool_call, dict):
func = tool_call.get("function") or {}
tokens += self.count_text(coerce_countable_text(func.get("name")))
tokens += self.count_text(coerce_countable_text(func.get("arguments")))
return tokens
def _normalize_message(self, message: dict[str, Any]) -> dict[str, Any]:
"""Normalize message to Anthropic format."""
role = message.get("role", "user")
# Map OpenAI roles to Anthropic
if role == "system":
# System messages need special handling - count as user for API
return {"role": "user", "content": message.get("content", "")}
elif role == "tool":
# Tool results in OpenAI format
return {
"role": "user",
"content": [
{
"type": "tool_result",
"tool_use_id": message.get("tool_call_id", ""),
"content": message.get("content", ""),
}
],
}
return {"role": role, "content": message.get("content", "")}
def count_messages(self, messages: list[dict[str, Any]]) -> int:
"""Count tokens in a list of messages.
Uses the Token Count API for accurate counting when available.
"""
if self._use_api:
return self._count_messages_via_api(messages)
return self._count_messages_estimated(messages)
def _count_messages_via_api(self, messages: list[dict[str, Any]]) -> int:
"""Count tokens using Anthropic Token Count API."""
try:
# Separate system message (Anthropic handles it differently)
system_content = None
api_messages = []
for msg in messages:
if msg.get("role") == "system":
system_content = msg.get("content", "")
else:
api_messages.append(self._normalize_message(msg))
# Ensure we have at least one message
if not api_messages:
api_messages = [{"role": "user", "content": ""}]
kwargs: dict[str, Any] = {
"model": self.model,
"messages": api_messages,
}
if system_content:
kwargs["system"] = system_content
response = self._client.messages.count_tokens(**kwargs)
return int(response.input_tokens)
except Exception as e:
# Fall back to estimation on API error
warnings.warn(
f"Token Count API failed ({e}), using estimation", UserWarning, stacklevel=3
)
return self._count_messages_estimated(messages)
def _count_messages_estimated(self, messages: list[dict[str, Any]]) -> int:
"""Estimate token count without API."""
total = sum(self._count_message_estimated(msg) for msg in messages)
return total + 3 # Base overhead
class AnthropicProvider(Provider):
"""Provider implementation for Anthropic Claude models.
For accurate token counting, provide an Anthropic client:
from anthropic import Anthropic
provider = AnthropicProvider(client=Anthropic())
This uses Anthropic's official Token Count API which accurately handles:
- JSON-heavy tool payloads
- Non-English text
- Long system prompts
- Tool definitions and structured content
Without a client, falls back to tiktoken approximation (less accurate).
Custom Model Configuration:
You can configure custom models via environment variable or config file:
1. Environment variable (JSON string):
export HEADROOM_MODEL_LIMITS='{"context_limits": {"my-model": 200000}}'
2. Environment variable (file path):
export HEADROOM_MODEL_LIMITS=/path/to/models.json
3. Config file (~/.headroom/models.json):
{
"anthropic": {
"context_limits": {"my-model": 200000},
"pricing": {"my-model": {"input": 3.0, "output": 15.0}}
}
}
"""
def __init__(
self,
client: Any = None,
context_limits: dict[str, int] | None = None,
warn: bool = True,
):
"""Initialize Anthropic provider.
Args:
client: Optional anthropic.Anthropic client for accurate token counting.
If not provided, uses tiktoken approximation.
context_limits: Optional override for model context limits.
warn: If False, suppresses the no-client UserWarning. Set to False
in contexts where tiktoken approximation is intentional (e.g.
the internal proxy pipeline provider).
Example:
from anthropic import Anthropic
provider = AnthropicProvider(client=Anthropic())
"""
self._client = client
self._warn = warn
self._token_counters: dict[str, AnthropicTokenCounter] = {}
# Build context limits: defaults -> config file -> env var -> explicit
self._context_limits = {**ANTHROPIC_CONTEXT_LIMITS}
self._pricing = {**ANTHROPIC_PRICING}
# Load from config file and env var
custom_config = _load_custom_model_config()
self._context_limits.update(custom_config["context_limits"])
self._pricing.update(custom_config["pricing"])
# Explicit overrides take precedence
if context_limits:
self._context_limits.update(context_limits)
@property
def name(self) -> str:
return "anthropic"
def get_token_counter(self, model: str) -> TokenCounter:
"""Get token counter for a model.
If a client was provided to the provider, uses the Token Count API.
Otherwise falls back to tiktoken approximation.
"""
model = sanitize_anthropic_model_id(model)
if model not in self._token_counters:
self._token_counters[model] = AnthropicTokenCounter(
model=model,
client=self._client,
warn=self._warn,
)
return self._token_counters[model]
def get_context_limit(self, model: str) -> int:
"""Get context window limit for a model.
Resolution order:
1. Explicit context_limits passed to constructor
2. HEADROOM_MODEL_LIMITS environment variable
3. ~/.headroom/models.json config file
4. LiteLLM model info (if available)
5. Built-in ANTHROPIC_CONTEXT_LIMITS
6. Pattern-based inference (opus/sonnet/haiku)
7. Default fallback (200K for any Claude model)
A ``[1m]`` suffix raises the result to at least 1M: the caller asked for
the 1M tier and Claude Code sent the `context-1m` beta header, so the
real upstream window is 1M even when the base model's default is 200K.
Never raises an exception - uses sensible defaults for unknown models.
"""
if has_context_1m_suffix(model):
# Recursion terminates: the sanitized id has no `[1m]` left.
base = self.get_context_limit(sanitize_anthropic_model_id(model))
return max(base, CONTEXT_1M_TOKENS)
model = sanitize_anthropic_model_id(model)
# Check explicit and loaded limits
if model in self._context_limits:
return self._context_limits[model]
# Check for partial matches (e.g., "claude-3-5-sonnet" matches "claude-3-5-sonnet-20241022")
for known_model, limit in self._context_limits.items():
if model in known_model or known_model in model:
return limit
# Try LiteLLM for context limit
_, litellm_get_model_info = _get_litellm_clients()
if litellm_get_model_info is not None:
try:
info = litellm_get_model_info(model)
if info:
if "max_input_tokens" in info and info["max_input_tokens"] is not None:
limit = int(info["max_input_tokens"])
self._context_limits[model] = limit
return limit
if "max_tokens" in info and info["max_tokens"] is not None:
limit = int(info["max_tokens"])
self._context_limits[model] = limit
return limit
except Exception as e:
logger.debug(f"LiteLLM get_model_info failed for {model}: {e}")
# Pattern-based inference for new models
tier = _infer_model_tier(model)
if tier and tier in _PATTERN_DEFAULTS:
limit = cast(int, _PATTERN_DEFAULTS[tier]["context"])
self._warn_unknown_model(model, limit, f"inferred from '{tier}' tier")
# Cache for future calls
self._context_limits[model] = limit
return limit
# Fallback for unknown Claude models
if model.startswith("claude"):
limit = cast(int, _UNKNOWN_CLAUDE_DEFAULT["context"])
self._warn_unknown_model(model, limit, "using default Claude limit")
self._context_limits[model] = limit
return limit
# Non-Claude model - use conservative default
limit = 128000
self._warn_unknown_model(model, limit, "unknown provider, using conservative default")
self._context_limits[model] = limit
return limit
def _warn_unknown_model(self, model: str, limit: int, reason: str) -> None:
"""Warn about unknown model (once per model)."""
global _UNKNOWN_MODEL_WARNINGS
if model not in _UNKNOWN_MODEL_WARNINGS:
_UNKNOWN_MODEL_WARNINGS.add(model)
logger.warning(
f"Unknown Anthropic model '{model}': {reason} ({limit:,} tokens). "
f"To configure explicitly, set HEADROOM_MODEL_LIMITS env var or "
f"add to ~/.headroom/models.json"
)
def supports_model(self, model: str) -> bool:
"""Check if this provider supports the given model."""
model = sanitize_anthropic_model_id(model)
if model in self._context_limits:
return True
# Check prefix matches - support all Claude models
return model.startswith("claude")
def estimate_cost(
self,
input_tokens: int,
output_tokens: int,
model: str,
cached_tokens: int = 0,
) -> float | None:
"""Estimate cost for a request.
Tries LiteLLM first for up-to-date pricing, falls back to manual pricing.
Both paths apply Anthropic's long-context premium: on the Sonnet 4 / 4.5
family a prompt over 200K re-prices the whole request (see
``_LONG_CONTEXT_PREMIUM``).
"""
model = sanitize_anthropic_model_id(model)
# LiteLLM knows per-model cache and long-context rates, so let it price
# the whole request rather than rebuilding the rate card here.
cost = estimate_cost_from_tokens(
model,
input_tokens=input_tokens,
output_tokens=output_tokens,
cached_tokens=cached_tokens,
)
if cost is not None:
return cost
# Fall back to manual pricing
pricing = self._get_pricing(model)
if not pricing:
return None
rates = _apply_long_context_premium(model, pricing, input_tokens)
# Calculate cost
non_cached_input = input_tokens - cached_tokens
cost = (
(non_cached_input / 1_000_000) * rates["input"]
+ (cached_tokens / 1_000_000) * rates.get("cached_input", rates["input"])
+ (output_tokens / 1_000_000) * rates["output"]
)
return cost
def _get_pricing(self, model: str) -> dict[str, float] | None:
"""Get pricing for a model with fallback logic."""
model = sanitize_anthropic_model_id(model)
# Direct match
if model in self._pricing:
return self._pricing[model]
# Partial match
for known_model, prices in self._pricing.items():
if model in known_model or known_model in model:
return prices
# Pattern-based inference
tier = _infer_model_tier(model)
if tier and tier in _PATTERN_DEFAULTS:
return cast(dict[str, float], _PATTERN_DEFAULTS[tier]["pricing"])
# Default for unknown Claude models
if model.startswith("claude"):
return cast(dict[str, float], _UNKNOWN_CLAUDE_DEFAULT["pricing"])
# DeepSeek model fallback (via --anthropic-api-url)
if model.startswith("deepseek"):
return _get_deepseek_pricing(model)
return None