akadmin/headroom
1
0
Fork 0
mirror of https://github.com/headroomlabs-ai/headroom.git synced 2026-08-27 14:17:10 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
2630 commits 410 branches 169 tags 142 MiB
Commit graph

3 commits

Author SHA1 Message Date
Yossi Ovadia
eb5b5e4198
fix: Vertex model pricing shows $0.00 for versioned model names and vertex:anthropic provider (#2517) 
## Description

Two bugs cause `$0.00` cost display for Vertex AI users in headroom's
dashboard:

1. **Model name resolution** — Vertex appends `@YYYYMMDD` version tags
at runtime (e.g. `claude-haiku-4-5@20251001`). LiteLLM's database stores
bare names without version suffixes, so every versioned model missed the
lookup.

2. **Prefix cache savings** — the provider match checks `provider ==
"anthropic"` but Vertex traffic is tagged `provider ==
"vertex:anthropic"`, so cache read savings computed as $0.00. This bug
is **not** addressed by #2516.

Fixes #2515

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/pricing/litellm_model_resolution.py`: strip `@YYYYMMDD`
suffix before lookup; add `vertex_ai/` to `MODEL_PREFIX_RULES` for
Claude models; apply prefix rules to both original and bare names
- `headroom/proxy/cost.py`: extend provider match to include
`vertex:anthropic` alongside `anthropic` for prefix cache savings
- `tests/test_pricing_litellm_model_resolution.py`: 4 new tests covering
suffix stripping, versioned model resolution, pricing lookup, and
end-to-end resolve

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_pricing_litellm_model_resolution.py -v
collected 10 items

tests/test_pricing_litellm_model_resolution.py::test_prefix_rule_matches_case_insensitively PASSED
tests/test_pricing_litellm_model_resolution.py::test_resolution_candidates_try_bare_then_matching_prefix_then_alias PASSED
tests/test_pricing_litellm_model_resolution.py::test_pricing_lookup_candidates_include_provider_prefixes_and_aliases PASSED
tests/test_pricing_litellm_model_resolution.py::test_retired_claude_3_sonnet_aliases_to_sonnet_tier_not_haiku PASSED
tests/test_pricing_litellm_model_resolution.py::test_resolve_litellm_model_name_returns_first_known_candidate PASSED
tests/test_pricing_litellm_model_resolution.py::test_resolve_litellm_model_name_returns_original_when_unknown PASSED
tests/test_pricing_litellm_model_resolution.py::test_strip_vertex_version_suffix PASSED
tests/test_pricing_litellm_model_resolution.py::test_resolution_candidates_vertex_versioned_models PASSED
tests/test_pricing_litellm_model_resolution.py::test_pricing_lookup_candidates_vertex_versioned_models PASSED
tests/test_pricing_litellm_model_resolution.py::test_vertex_versioned_model_resolves_to_known_key PASSED

10 passed in 1.23s
```

## Real Behavior Proof

- Environment: macOS arm64, Python 3.11.13, headroom 0.32.1, Claude Code
2.1.211, `CLAUDE_CODE_USE_VERTEX=1`, persistent local proxy
- Exact command / steps: `python3 -c "from
headroom.pricing.litellm_model_resolution import resolution_candidates;
import litellm; m='claude-haiku-4-5@20251001'; [print(c,
litellm.model_cost.get(c,{}).get('input_cost_per_token',0)*1e6) for c in
resolution_candidates(m)]"`
- Observed result: before fix all versioned Vertex models returned
$0.00; after fix `claude-haiku-4-5@20251001`→$1.00/MTok,
`claude-opus-4@20250514`→$15.00/MTok, dashboard "Prefix Cache Impact"
shows Net savings $6.31 (was $0.00). Screenshots in issue #2515.
- Not tested: non-Vertex paths (direct Anthropic, Bedrock, OpenAI) —
changes are additive and guarded by `vertex:anthropic` provider check

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes

---------

Co-authored-by: JD Davis <jd@jds-macbook-air.tail2a279.ts.net>
 2026-08-11 23:03:09 -05:00
Abhay Singh
6137967083
fix(pricing): alias retired claude-3-sonnet to Sonnet-tier price, not Haiku (#2095) 
## Description

The `MODEL_ALIASES` fallback prices the retired Claude 3 Sonnet as
Claude 3 Haiku — a different, ~12x cheaper tier.

`MODEL_ALIASES` maps models that LiteLLM's cost DB no longer knows about
to a current key "that has equivalent pricing" (per the module comment).
The two Claude 3.5 Sonnet entries follow that rule — both map to
`claude-sonnet-4-20250514`, which is the same `$3 / $15` per-1M tier.
But the Claude 3 Sonnet entry was:

```python
"claude-3-sonnet-20240229": "claude-3-haiku-20240307",
```

`claude-3-sonnet-20240229` was a Sonnet-tier model at `$3.00 / $15.00`
per 1M (input/output). `claude-3-haiku-20240307` is `$0.25 / $1.25`. So
whenever LiteLLM lacks the retired Sonnet key and resolution falls
through to this alias (via `resolution_candidates` /
`pricing_lookup_candidates`), every cost and savings figure for that
model is understated **~12x on both input and output**. That's the
opposite of the "equivalent pricing" the alias table promises, and it
silently biases dashboards/ledger numbers for anyone still routing that
model.

## Fix

Alias the retired Claude 3 Sonnet to `claude-sonnet-4-20250514` — the
same-price ($3/$15) target the sibling retired-Sonnet aliases already
use — so the fallback preserves the tier instead of downgrading it.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/pricing/litellm_model_resolution.py`: change the
`claude-3-sonnet-20240229` alias target from `claude-3-haiku-20240307`
to `claude-sonnet-4-20250514`, with a comment explaining the tier.
- `tests/test_pricing_litellm_model_resolution.py`: add
`test_retired_claude_3_sonnet_aliases_to_sonnet_tier_not_haiku`.
- `CHANGELOG.md`: Bug Fixes entry.

## Testing

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uvx ruff@0.15.17 check headroom/pricing/litellm_model_resolution.py tests/test_pricing_litellm_model_resolution.py
All checks passed!
$ python -m py_compile headroom/pricing/litellm_model_resolution.py tests/test_pricing_litellm_model_resolution.py
OK
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and a full `pytest`
gets OOM-killed on this box, so I checked the tier delta with a
dependency-free script against the public list prices and left the full
pytest to CI.
- Exact command / steps: compared the old alias target
(`claude-3-haiku-20240307`, $0.25/$1.25) against the new one
(`claude-sonnet-4-20250514`, $3.00/$15.00), which matches the retired
Claude 3 Sonnet's own $3/$15 tier.
- Observed result: the Haiku target underpriced input 12x ($3.00 /
$0.25) and output 12x ($15.00 / $1.25). The new regression test asserts
the alias contains no `haiku` and equals the same-tier target used by
the other retired-Sonnet aliases.
- Not tested: an end-to-end resolution through a live LiteLLM cost DB
(the alias only fires when LiteLLM lacks the retired key); full local
`pytest` deferred to CI (OOM, per above).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

The "unit tests pass locally" and "type checking" boxes are unchecked
because the full suite imports the ML stack, which I can't run in this
environment; this is a one-line data fix in a pure module, verified by
the tier-delta proof and the new regression test for CI. Reachability is
bounded — the alias only matters when LiteLLM's cost DB doesn't already
know the retired model — but when it does fire the price is off by a
full tier.

Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
 2026-07-13 10:50:33 -04:00
JD Davis
4210d6e609
refactor(pricing): isolate litellm model resolution (#1936) 
## Description
Extracts LiteLLM model-name resolution rules into a pure pricing-domain
module. `litellm_pricing.py` now acts as the adapter that asks LiteLLM
whether candidate keys exist, while `litellm_model_resolution.py` owns
prefix rules, alias rules, lookup candidate ordering, and deterministic
resolution.

Closes #

## Type of Change
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made
- Added `headroom.pricing.litellm_model_resolution` with explicit prefix
rules, alias rules, pricing lookup candidates, and a pure resolver
function.
- Simplified `headroom.pricing.litellm_pricing` to delegate model-name
selection to the pure resolver while keeping its public API and cache
behavior intact.
- Added focused tests for candidate ordering, case-insensitive MiniMax
matching, aliases, and unknown-model fallback.

## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output
```text
python -m pytest tests/test_pricing_litellm_model_resolution.py tests/test_pricing_litellm.py tests/test_proxy_streaming_resilience.py::TestModelResolutionCaching -q
============================= 22 passed in 2.18s =============================

python -m ruff check headroom/pricing/litellm_model_resolution.py headroom/pricing/litellm_pricing.py tests/test_pricing_litellm_model_resolution.py tests/test_pricing_litellm.py tests/test_proxy_streaming_resilience.py
All checks passed!

python -m mypy headroom/pricing/litellm_model_resolution.py headroom/pricing/litellm_pricing.py
Success: no issues found in 2 source files

python -m compileall -q headroom\pricing\litellm_model_resolution.py headroom\pricing\litellm_pricing.py
# no output; exited 0

git commit -m "refactor(pricing): isolate litellm model resolution"
Sync plugin versions.....................................................Passed
check for merge conflicts................................................Passed
ruff.....................................................................Passed
ruff-format..............................................................Passed
mypy.....................................................................Passed
```

## Real Behavior Proof
- Environment: Windows PowerShell, Python 3.13.13, branch
`jd/pricing-model-resolution` based on `headroomlabs/main`.
- Exact command / steps: Ran pure resolver tests, LiteLLM pricing
adapter tests, model-resolution caching tests, focused ruff, targeted
mypy, compileall, and commit hooks.
- Observed result: Existing pricing behavior and cache behavior passed
while model resolution is now isolated and directly testable.
- Not tested: Full pytest suite and live LiteLLM network or package
update behavior beyond the local installed dependency/fakes.

## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)
N/A

## Additional Notes
Documentation and CHANGELOG updates are N/A for this internal refactor.
Full pytest was not run; validation is focused on
pricing/model-resolution behavior touched by this slice.
 2026-07-10 17:35:31 -05:00