akadmin/headroom
1
0
Fork 0
mirror of https://github.com/headroomlabs-ai/headroom.git synced 2026-08-27 14:17:10 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
2630 commits 410 branches 169 tags 142 MiB
Commit graph

6 commits

Author SHA1 Message Date
Abhay Singh
3ed8f76019
fix(providers): don't crash on a non-object HEADROOM_MODEL_LIMITS / models.json (#3089) 
## Description

`_load_custom_model_config` in both `headroom/providers/anthropic.py`
and `headroom/providers/openai.py` loads the operator's custom model
configuration from `HEADROOM_MODEL_LIMITS` (a JSON string or a file
path) and `~/.headroom/models.json`, then reads it with
`loaded.get(...)`:

```python
loaded = json.loads(env_config)          # or json.load(f)
anthropic_config = loaded.get("anthropic", loaded)
```

The `try` guards only `except (json.JSONDecodeError, OSError)`. When the
value is **valid JSON but not an object** (a JSON array, number, string,
bool, or `null`), `json.loads` succeeds and returns a non-dict, so
`loaded.get(...)` raises `AttributeError` — which is *not* one of the
caught types. Instead of the intended warn-and-fall-back-to-defaults, a
misconfigured `HEADROOM_MODEL_LIMITS` (e.g.
`HEADROOM_MODEL_LIMITS='[1,2,3]'` or `'"gpt-4"'`) crashes provider
initialization. The same gap exists in the `models.json` branch of both
providers.

## Fix

After each load, validate `isinstance(loaded, dict)` and raise
`ValueError` with a clear message, and broaden the handler from `except
(json.JSONDecodeError, OSError)` to `except (ValueError, OSError)`.
`json.JSONDecodeError` is a subclass of `ValueError`, so this strictly
supersets the previous handling: every previously-caught malformed value
still warns and falls back, and a valid-JSON-but-non-object value now
does too, instead of crashing.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/providers/anthropic.py` and `headroom/providers/openai.py`
(`_load_custom_model_config`): add an `isinstance(loaded, dict)` guard
(raising `ValueError`) after the env-var load and after the
`models.json` load, and change both `except` clauses to `(ValueError,
OSError)`.
- `tests/test_provider_model_fallback.py`: added parametrized
`test_non_object_env_var_falls_back_to_defaults` (array / string /
number / bool / null) for both providers, and
`test_non_object_config_file_falls_back_to_defaults` for a non-object
`models.json`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added

### Test Output

```text
tests/test_provider_model_fallback.py  44 passed
# uvx ruff@0.15.22 check  -> All checks passed!
# uvx mypy@1.20.2 headroom/providers/anthropic.py headroom/providers/openai.py -> Success: no issues found in 2 source files
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.15.22 and mypy 1.20.2 via uvx.
- Exact command / steps: reverted both providers and ran the new
regressions to capture the bug (`python -m pytest
tests/test_provider_model_fallback.py::TestAnthropicConfigLoading::test_non_object_env_var_falls_back_to_defaults
tests/test_provider_model_fallback.py::TestOpenAIConfigLoading::test_non_object_env_var_falls_back_to_defaults
tests/test_provider_model_fallback.py::TestAnthropicConfigLoading::test_non_object_config_file_falls_back_to_defaults`
-> 11 failed with `AttributeError` on `loaded.get` across the
array/string/number/bool/null shapes); restored the fix; re-ran the full
file (`python -m pytest tests/test_provider_model_fallback.py` -> 44
passed); then `uvx ruff@0.15.22 format`, `uvx ruff@0.15.22 check`, and
`uvx mypy@1.20.2` on both providers.
- Observed result: before the fix, `HEADROOM_MODEL_LIMITS='[1,2,3]'` (or
`'"gpt-4"'`, `'42'`, `'true'`, `'null'`) raised `AttributeError` out of
`_load_custom_model_config`; after the fix the same values log a warning
and the loader returns the default `{"context_limits": {}, "pricing":
{}[, "encodings": {}]}`, and a well-formed object config is unchanged.
- Not tested: a live proxy boot with a corrupt `HEADROOM_MODEL_LIMITS`
(the loader is exercised directly, which is the exact function provider
init calls).

## Runtime Rollout Safety

- Rollout-managed feature(s): none. This is defensive parsing in the
provider model-config loader, not a rollout-channel-gated runtime
feature.
- Minimum rollout channel: N/A (no rollout-managed behavior).
- Stable/default behavior changed: only for a previously-crashing input.
A non-object `HEADROOM_MODEL_LIMITS` / `models.json` now warns and uses
built-in defaults instead of raising. Well-formed object configs are
parsed exactly as before.
- Kill switch / disable path: N/A — remove or correct the malformed
config value to load custom limits.
- Unsafe override required: no.
- Qualification impact: a corrupt or mistyped model-limits value
degrades to built-in defaults with a warning rather than failing
provider init.
- Rollback path: revert this PR; the loader returns to catching only
`json.JSONDecodeError`/`OSError`.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`: it is generated by
release-please from my Conventional Commit PR title

## Additional Notes

Both providers carry the same loader shape, so the guard and the widened
`except` are applied identically to keep them in sync. The message names
the offending source (`HEADROOM_MODEL_LIMITS` vs the resolved
config-file path) so the warning is actionable.
 2026-08-17 20:18:58 -07:00
Tejas Chopra
0cb72f45b2
fix(providers): stop a shorter model family shadowing a longer one (#2762) 
## Description

> **Stacked on #2761** — that PR splits `_lookup_encoding_name` out of
`_get_encoding_name_for_model`, which this one builds on. Please merge
#2761 first; the diff here will shrink to just this commit afterwards.

`_MODEL_ENCODINGS` and `_CONTEXT_LIMITS` are matched by prefix,
iterating in **plain dict order** — so the first *inserted* prefix wins
rather than the most specific one. `gpt-4.1` matched the `gpt-4` entry:

| model | resolved | actual | |
|---|---|---|---|
| `gpt-4.1` | 8192 | 1,047,576 | **128× under** |
| `gpt-4.1-mini` | 8192 | 1,047,576 | **128× under** |
| `gpt-4.1-nano` | 8192 | 1,047,576 | **128× under** |
| `gpt-4-32k-0613` | 8192 | 32,768 | 4× under |
| `gpt-5` / `-mini` / `-nano` | 128,000 | 400,000 | fell to
unknown-model default |
| `o4-mini` | 128,000 | 200,000 | fell to unknown-model default |

A 128× under-estimate matters because the context limit is what tells
the proxy how much headroom is left: it treats a 1M-context model as
nearly full and compresses accordingly.

The same shadowing picked the **wrong encoding** — `gpt-4.1` got
`cl100k_base` instead of `o200k_base`. Measured cost of that:

```text
                cl100k    o200k    error
python code        420      420    +0.0%
json blob          555      555    +0.0%
logs               580      580    +0.0%
english            201      201    +0.0%
CJK                600      450   +33.3%
```

So the encoding half is narrow but real — it only bites CJK content,
which the repo already treats as a case worth testing
(`tests/test_evals_cjk_tokenization.py`).

**Scope honestly:** `get_context_limit` consults LiteLLM *before* this
table, so the limit half only surfaces where LiteLLM is absent or does
not know the model. That is not hypothetical — the `litellm` dependency
carries a `python_version < '3.14'` marker (`pyproject.toml:56`), so
**any install on Python 3.14+ has no LiteLLM** and this table is
load-bearing. The encoding half never had a LiteLLM fallback and was
always wrong.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- Both prefix loops now iterate `sorted(..., key=len, reverse=True)` —
longest prefix wins. This is the root-cause fix: it also protects the
*next* model added to these tables.
- Added the missing families: `gpt-4.1` (+`-mini`/`-nano`), `gpt-5`
(+`-mini`/`-nano`), `o4-mini` to both tables.
- Left `supports_model`'s prefix loop alone — it only returns a bool, so
order cannot change its answer.

Not touched: `_PRICING`. `gpt-4.1`/`gpt-5` also fall through to the
GPT-4o pricing tier, which skews cost reporting, but that is a separate
concern with its own verification burden (published rates, staleness
window) and does not belong in a tokenizer-correctness fix.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check` + `ruff format`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality

### Test Output

12 of 24 new cases fail without the fix; the 12 that pass are the "must
not regress" rows (`gpt-4`, `gpt-4-turbo`, `gpt-4o`, `o3`,
`gpt-3.5-turbo`) — included precisely so the longest-prefix change can't
quietly move them:

```text
$ git stash push headroom/providers/openai.py && pytest tests/test_openai_model_table_resolution.py -q
FAILED ...::test_context_limit_prefers_the_most_specific_prefix[gpt-4.1-mini-1047576]
FAILED ...::test_context_limit_prefers_the_most_specific_prefix[gpt-4.1-nano-1047576]
FAILED ...::test_context_limit_prefers_the_most_specific_prefix[gpt-4.1-2025-04-14-1047576]
FAILED ...::test_context_limit_prefers_the_most_specific_prefix[gpt-4-32k-0613-32768]
FAILED ...::test_context_limit_prefers_the_most_specific_prefix[gpt-5-400000]
FAILED ...::test_context_limit_prefers_the_most_specific_prefix[gpt-5-mini-400000]
FAILED ...::test_context_limit_prefers_the_most_specific_prefix[o4-mini-200000]
FAILED ...::test_encoding_prefers_the_most_specific_prefix[gpt-4.1-o200k_base]
FAILED ...::test_encoding_prefers_the_most_specific_prefix[gpt-4.1-mini-o200k_base]
FAILED ...::test_encoding_prefers_the_most_specific_prefix[gpt-4.1-2025-04-14-o200k_base]
FAILED ...::test_cjk_is_not_over_counted_for_gpt_41
12 failed, 12 passed in 0.70s

$ git stash pop && pytest tests/test_openai_model_table_resolution.py -q
24 passed in 0.43s
```

Regression check — 119 suites touching openai / cost / savings / token /
compress / outcome / budget, this branch vs clean `main` in the same
environment, comparing failure *sets*:

```text
branch : 5 failed, 1486 passed, 86 skipped in 111.01s
main   : 5 failed, 1453 passed, 86 skipped in 132.07s

NEW failures introduced: (none)

pre-existing on both:
  test_bundled_tools_savings.py::test_compressed_payload_preserves_answer_anthropic
  test_compress_route_tokenizer_by_model.py::...[deepseek/deepseek-v4]   (needs `transformers`)
  test_compress_route_tokenizer_by_model.py::...[deepseek/deepseek-v4]   (needs `transformers`)
  test_image_compressor_singleton_reuse.py::test_onnx_router_is_built_once_and_cached
  test_openai_streaming_backend.py::...test_litellm_vertex_streaming_preserves_max_tokens_and_vendor_fields
```

```text
$ ruff check headroom/providers/openai.py tests/test_openai_model_table_resolution.py
All checks passed!
$ mypy headroom/providers/openai.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- **Environment:** macOS, Python 3.13.7, isolated worktree at
`upstream/main` (`ad56dd38`), no `litellm` installed (matching a Python
3.14+ install, where the dep marker excludes it).
- **Exact command / steps:** resolve context limit + encoding for each
model against published OpenAI values, before and after.
- **Observed result:**

```text
before                              after
model               limit  enc      model               limit      enc
gpt-4.1              8192  cl100k   gpt-4.1           1047576  o200k_base
gpt-4.1-mini         8192  cl100k   gpt-4.1-mini      1047576  o200k_base
gpt-4.1-nano         8192  cl100k   gpt-4.1-nano      1047576  o200k_base
gpt-4.1-2025-04-14   8192  cl100k   gpt-4.1-2025-04-14 1047576 o200k_base
gpt-4-32k-0613       8192  cl100k   gpt-4-32k-0613      32768  cl100k_base
gpt-5              128000  o200k    gpt-5              400000  o200k_base
o4-mini            128000  o200k    o4-mini            200000  o200k_base

unchanged: gpt-4=8192/cl100k, gpt-4-turbo=128000/cl100k,
           gpt-4o=128000/o200k, o3=200000, gpt-3.5-turbo=16385/cl100k
```
 2026-08-04 00:34:38 -07:00
Kenneth Wong
e84ca980cf
feat(anthropic): add Claude 5 family pricing & align current rates (#1767) 
## Summary

Adds Claude 5 generation metadata and aligns the Anthropic fallback
pricing / context-limit tables in `headroom/providers/anthropic.py` with
current [Anthropic
pricing](https://platform.claude.com/docs/en/about-claude/pricing)
(verified 2026-07-04). Several Claude 4.x entries carried stale rates,
the new Claude 5 models (Fable 5, Opus 4.8, Sonnet 5) had no metadata,
and the generic fallback tests disagreed with the provider tests.

Supersedes #1485 (rebased onto latest `main`, squashed to one commit,
extended with Sonnet 5 / Fable 5 and the requested fallback-test fixes).

## Changes

All values `$ / MTok`; `cached_input` = prompt-cache read = 0.1× input.

| Tier | Model | Before | After | Context |
|---|---|---|---|---|
| Fable | `claude-fable-5` | — *(new)* | $10 / $50 / $1.00 | **1M** |
| Opus | `claude-opus-4-8` | — *(new)* | $5 / $25 / $0.50 | **1M** |
| Opus | `claude-opus-4-7` | $15 / $75 / $1.50 | $5 / $25 / $0.50 | 1M |
| Opus | `claude-opus-4-6` | $15 / $75 / $1.50 | $5 / $25 / $0.50 | 1M |
| Opus | `claude-opus-4-5-20251101` | $15 / $75 / $1.50 | $5 / $25 /
$0.50 | 200K |
| Sonnet | `claude-sonnet-5` | — *(new)* | $3 / $15 / $0.30 | **1M** |
| Sonnet | `claude-sonnet-4-6` | — *(new)* | $3 / $15 / $0.30 | **1M** |
| Sonnet | `claude-sonnet-4-5` | — *(new)* | $3 / $15 / $0.30 | 200K |
| Haiku | `claude-haiku-4-5-20251001` | $0.80 / $4 / $0.08 *(3.5 rates)*
| $1 / $5 / $0.10 | 200K |

`claude-sonnet-4-20250514` and all Claude 3.x / 3.5.x entries were
already correct — left unchanged.

The **1M-context** entries (Fable 5, Opus 4.8, Sonnet 5, Sonnet 4.6) are
functional, not cosmetic: they ship in the long-context tier, and
without explicit entries the `sonnet` / `opus` pattern defaults would
report 200K.

Sonnet 5 is pinned to the **standard** Sonnet tier ($3 / $15 / $0.30);
Anthropic's introductory rate ($2 / $10 through Aug 31 2026) is
intentionally not encoded to avoid a time-dependent fixture.

## Fallback-model tests (addresses review on #1485)

`_PATTERN_DEFAULTS["opus"]` is aligned to the current Opus tier ($5 /
$25 / $0.50) so the generic fallback suite and the provider-specific
suite agree:

- `test_pricing_for_known_models` — Opus 4.5 pins $5 / $25 / $0.50
- `test_pattern_based_inference_opus` — unknown-opus fallback now $5 /
$25
- `test_cost_estimation_for_new_models` — fixture estimate corrected
$22.5 → $7.5
- `test_pattern_based_inference_sonnet` — retargeted to
`claude-sonnet-6-*` (the old `claude-sonnet-5-*` probe now
prefix-matches the real `claude-sonnet-5` key)

New provider coverage:

- `test_get_context_limit_claude_5_family` — Fable 5 / Opus 4.8 / Sonnet
5 all 1M
- `test_pricing_claude_5_family` — exact rate table for the 3 new models

Full suite: **48 passed**.

## Source

https://platform.claude.com/docs/en/about-claude/pricing
 2026-07-03 12:09:35 -07:00
julienguarino
17ecad9d89
fix(gemini): resolve Google model capabilities through ModelRegistry (#1276) 
## Description

Google model capability lookup was still tied to static provider tables
for support checks and context limits. That made plausible future Gemini
model ids fail token counting or context lookup even when they clearly
belonged to the Google provider family.

This change adds a tolerant `ModelRegistry.resolve()` runtime lookup
path and routes the Google provider through it. Exact built-in registry
matches still win first, LiteLLM pricing metadata can supply live limits
when available, and provider-scoped family fallbacks cover future Gemini
ids without letting Google claim unrelated models.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Added `ModelRegistry.resolve()` as a tolerant runtime capability
resolver.
- Added provider-scoped Google/Gemini family fallbacks for plausible
future model ids.
- Added support for LiteLLM-style `gemini/gemini-...` model ids in
provider inference and family fallback matching.
- Updated `GoogleProvider.supports_model()` and
`GoogleProvider.get_context_limit()` to use the shared model registry
path.
- Added regression tests for future Gemini ids, legacy Gemini context
limits, and unrelated model rejection.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
uv run --no-project --with pytest --with opentelemetry-api --with pydantic --with tiktoken --with litellm --with click --with rich python -B -m pytest tests/test_provider_model_fallback.py tests/test_models.py

65 passed

uv run --no-project --with ruff ruff check headroom/models/registry.py headroom/providers/google.py tests/test_provider_model_fallback.py tests/test_models.py

All checks passed!

uv run --no-project --with ruff ruff format --check headroom/models/registry.py headroom/providers/google.py tests/test_provider_model_fallback.py tests/test_models.py

4 files already formatted
```

## Real Behavior Proof

- Environment: macOS arm64 local checkout, Python 3.13 virtualenv for
editable install; deployed smoke test in a Cloud Run staging service
using an earlier commit from this fork branch before the review
follow-up.
- Exact command / steps: installed `headroom-ai[langchain]` from the
fork branch in the staging service, triggered long-context requests that
activate Headroom's LangChain compression path, then checked Cloud Run
logs after 2026-06-22 12:20 Europe/Paris.
- Observed result: Headroom initialized successfully, compressed
conversation memory (`23255 -> 5618 chars`), and no logs matched the
previous model-resolution failure signatures (`not recognized as a
Google model`, `Unknown context limit`).
- Not tested: staging was not rerun after the `gemini/gemini-...` review
follow-up; that prefix path is covered by local regression tests. Full
repository `uv run pytest` on local macOS is currently blocked by a
native `maturin`/`esaxx-rs` compile failure (`fatal error: 'cstdint'
file not found`). Type checking was not run.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A

## Additional Notes

- Documentation changes are not included because this is a runtime
compatibility fix with no public API or user-facing configuration
change.
- Full local test execution should be retried in CI or a Linux
environment where the native Rust extension build is healthy.

Co-authored-by: Julien Guarino <julien.guarino@fashiondata.io>
 2026-06-26 23:31:56 -05:00
JerrettDavis
d00c6739e1 Fix CI regressions for cache benchmark work 2026-04-04 22:33:44 -05:00
chopratejas
905c229251 Add AST-based code compression and custom model configuration 
CodeAwareCompressor:
- Tree-sitter based AST parsing for Python, JS, TS, Go, Rust, Java, C, C++
- Preserves imports, signatures, type annotations, error handlers
- Guarantees syntactically valid output
- Uses tree-sitter-language-pack for broad language support

ContentRouter:
- Intelligent compression orchestrator
- Auto-routes content to optimal compressor based on type detection
- Source hint support for high-confidence routing

Custom Model Configuration:
- HEADROOM_MODEL_LIMITS env var and ~/.headroom/models.json support
- Pattern-based inference for unknown models (opus/sonnet/haiku tiers)
- Support for Claude 4.5, Claude 4, o3, o3-mini
- Graceful fallback - never crashes on unknown models
 2026-01-14 13:46:55 -08:00