akadmin/headroom
1
0
Fork 0
mirror of https://github.com/headroomlabs-ai/headroom.git synced 2026-08-27 14:17:10 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
2630 commits 410 branches 169 tags 142 MiB
Commit graph

2 commits

Author SHA1 Message Date
Abhay Singh
b976378c3e
test(pricing): stop asserting DeepSeek pricing freshness on wall-clock time (#2428) 
## Description

The shared `test` job is currently failing on every open PR because of a
wall-clock time-bomb in the DeepSeek pricing tests, not because of any
code change.


`tests/test_providers/test_deepseek.py::TestDeepSeekPricingModule::test_registry_staleness_and_source_url`
asserted:

```python
assert not registry.is_stale()
```

`PricingRegistry.is_stale()` returns `(date.today() - last_updated) >
timedelta(days=30)`. The DeepSeek registry ships `LAST_UPDATED =
date(2026, 6, 19)`, so this assertion holds only while the current date
stays within 30 days of that constant. Once it lapses, the test fails on
time alone, turning the `test` shard red for every unrelated PR in the
repo. It is failing right now (31 days past `LAST_UPDATED`).

This is not testing code behavior: it only checks that the machine's
clock is within 30 days of a hardcoded date. The sibling Anthropic and
OpenAI registries are 560 days old and make no such assertion, so
DeepSeek is the odd one out here rather than a deliberate freshness
gate.

## Fix

Drop the freshness assertion and keep the meaningful `source_url` check,
renaming the test to `test_registry_source_url` to match what it now
verifies.

The staleness mechanism stays fully and time-independently covered by
`tests/test_pricing.py::test_registry_staleness_and_warning`, which
builds registries with `date.today() - timedelta(days=30)` (asserts not
stale) and `date.today() - timedelta(days=31)` (asserts stale) plus the
warning text. So this removes a fragile environmental assertion without
reducing real coverage, and aligns DeepSeek with the Anthropic/OpenAI
registries.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `tests/test_providers/test_deepseek.py`: remove the
wall-clock-dependent `assert not registry.is_stale()`, keep the
`source_url` assertion, rename the test to `test_registry_source_url`,
and add a comment explaining why freshness is not asserted here.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uvx ruff@0.15.17 check tests/test_providers/test_deepseek.py
All checks passed!
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, project venv (`uv sync --extra
proxy`), `uvx ruff@0.15.17`.
- Exact command / steps: with the current date at 31 days past
`LAST_UPDATED`, ran the registry's `is_stale()` and the fixed test body
against the real modules, plus the mechanism test from
`tests/test_pricing.py`.
- Observed result: `get_deepseek_registry().is_stale()` is `True` on the
current date (which is exactly what broke the old assertion); the fixed
`test_registry_source_url` body passes regardless of the date; and
`test_registry_staleness_and_warning` still passes, so the staleness
mechanism remains covered.
- Not tested: a live DeepSeek pricing fetch (out of scope; pricing
values are unchanged).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
 2026-07-19 22:14:34 -07:00
Ali
0e6d922f88
feat(pricing): add DeepSeek V4 model pricing (deepseek-v4-flash, deepseek-v4-pro) (#1168) 
## Description

Adds pricing support for DeepSeek V4 models (`deepseek-v4-flash` and
`deepseek-v4-pro`) when routing Headroom through `--anthropic-api-url
https://api.deepseek.com/anthropic`. The vendored LiteLLM pricing
database predates DeepSeek V4, so cost estimation silently returned
`None` for these models.

## Type of Change

- [x] New feature (non-breaking change that adds functionality)

## Changes Made

- **`headroom/pricing/deepseek_prices.py`** — New pricing data module
with `ModelPricing` dataclass entries for both V4 models, following the
pattern of `anthropic_prices.py`
- **`headroom/pricing/__init__.py`** — Exports `DEEPSEEK_PRICES`,
`get_deepseek_registry()`, `DEEPSEEK_LAST_UPDATED`
- **`headroom/pricing/litellm_pricing.py`** — Runtime injection of
DeepSeek V4 pricing into `litellm.model_cost`, plus `deepseek-` prefix
added to `resolve_litellm_model()` provider prefix list
- **`headroom/providers/anthropic.py`** — DeepSeek fallback in
`_get_pricing()` when model starts with `deepseek-` and LiteLLM is
unavailable
- **`crates/headroom-proxy/data/model_prices_and_context_window.json`**
— Vendored JSON entries (bare + provider-prefixed) for Rust-side context
window lookups
- **`tests/test_providers/test_deepseek.py`** — 20 tests across 3 test
classes (pricing data, LiteLLM injection, Anthropic fallback)
- **`tests/test_pricing.py`** — Added DeepSeek export validation
alongside existing OpenAI/Anthropic assertions

## Testing

- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```
========================= 137 passed, 8 warnings in 8.47s =========================
```

## Real Behavior Proof

- Environment: Windows 10, Python 3.12, litellm 1.60+
- Exact command / steps: `python -c "from headroom.proxy.cost import
CostTracker; t = CostTracker();
print(t.estimate_cost('deepseek-v4-flash', input_tokens=1000000,
output_tokens=1000000))"`
- Observed result: `$0.4200` (0.14 input + 0.28 output per 1M tokens)
- Not tested: Live DeepSeek API routing via `--anthropic-api-url`
(requires API key and Docker deployment)

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

The 90% cache discount heuristic in `AnthropicProvider.estimate_cost()`
(line 680) is a pre-existing pattern. DeepSeek V4 has much deeper cache
discounts (98-99%), but the LiteLLM path currently falls through to the
manual fallback which uses correct cached prices. A future improvement
could prefer `cache_read_input_token_cost` from model info over the
hardcoded `* 0.1` heuristic.

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-06-24 09:44:27 -05:00