mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
fix(dashboard): price proxy savings without litellm (#1728)
## Description The dashboard's main `Proxy $ Saved` tile can stay at `$0` on Python 3.14 because the durable proxy savings tracker records `0.0` whenever LiteLLM is unavailable or cannot price a model. The token counters keep moving, but `proxy_savings.json` stores zero-dollar `compression_savings_usd` and `total_input_cost_usd` values for new entries, so `/stats` and the dashboard read a permanent zero for those rows. This fixes the proxy savings pricing authority so positive token deltas use LiteLLM list pricing when available and fall back to the existing Headroom savings fallback when exact pricing is unavailable. Existing historical rows keep their stored write-time values; this changes new savings entries going forward. Closes #1718. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added `DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN = 3.0 / 1_000_000` constant to `headroom/proxy/savings_tracker.py`. - Fixed `_estimate_compression_savings_usd()`: removed the early `litellm is None` zero-return; changed missing-pricing path from `return 0.0` to `raise RuntimeError`; fallback `except` now returns `tokens_saved * DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN` instead of `0.0`. - Fixed `_estimate_input_cost_usd()`: moved `use_breakdown` computation before the `litellm is None` guard; introduced `chargeable_tokens` which equals the breakdown sum when a breakdown exists, or `input_tokens` otherwise; both the `litellm is None` path and the `except Exception` path now use `chargeable_tokens` to avoid double-counting when breakdown tokens and `input_tokens` are both provided; exact LiteLLM cache metadata remains authoritative when present. - Added focused regression coverage in `tests/test_proxy_savings_history.py` for the LiteLLM-unavailable path, exact-price preservation, and the historical no-backfill boundary. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_proxy_savings_history.py tests/test_savings_ledger.py -q`) - [x] Linting passes (`uv run ruff check headroom/proxy/savings_tracker.py tests/test_proxy_savings_history.py tests/test_savings_ledger.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text Pytest command: uv run pytest tests/test_proxy_savings_history.py tests/test_savings_ledger.py -q Run through: conhost --headless cmd /v:on /c ============================= test session starts ============================= platform win32 -- Python 3.12.13, pytest-9.0.3, pluggy-1.6.0 rootdir: D:\Repos\headroom-pr-1718-fallback-savings-cost-zero configfile: pyproject.toml plugins: anyio-4.12.1, langsmith-0.9.3, asyncio-1.3.0, cov-7.0.0 asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 37 items tests\test_proxy_savings_history.py ...................... [ 59%] tests\test_savings_ledger.py ............ss. [100%] ============================== warnings summary =============================== tests/test_savings_ledger.py::test_proxy_record_request_appends_ledger_event D:\Repos\headroom-pr-1718-fallback-savings-cost-zero\.venv\Lib\site-packages\fastapi\testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. from starlette.testclient import TestClient as TestClient # noqa -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ================== 35 passed, 2 skipped, 1 warning in 16.47s ================== Ruff command: uv run ruff check headroom/proxy/savings_tracker.py tests/test_proxy_savings_history.py tests/test_savings_ledger.py Run through: conhost --headless cmd /v:on /c All checks passed! ``` ## Real Behavior Proof - Environment: Python proxy savings tracker with LiteLLM forced unavailable (`LITELLM_AVAILABLE=False`, `litellm=None`), using a temporary `proxy_savings.json`. - Exact command / steps: run `uv run pytest tests/test_proxy_savings_history.py tests/test_savings_ledger.py -q`, then inspect `test_fallback_request_pricing_stays_nonzero_with_litellm_unavailable_and_preserves_historic_zeros` and `test_fallback_input_cost_uses_breakdown_sum_not_input_tokens_when_litellm_unavailable`, which load a pre-existing file with zero-dollar historical rows, call `record_request()` with LiteLLM unavailable, and call `_estimate_input_cost_usd()` with both `input_tokens` and a nonzero breakdown. - Observed result: new lifetime, display-session, project, and history entries receive nonzero fallback-priced dollar values while the original zero-dollar history row remains unchanged, and the fallback input-cost path prices only the breakdown sum instead of `input_tokens + breakdown_sum`. - `test_litellm_resolution_and_savings_estimation_fallbacks` verifies that `_estimate_compression_savings_usd` and `_estimate_input_cost_usd` return fallback amounts (not `0.0`) for all three paths: LiteLLM available but metadata missing, LiteLLM available but pricing lookup raises, and `LITELLM_AVAILABLE=False`. - `test_input_cost_counts_cache_reads_when_uncached_input_is_zero` verifies that a fully prefix-cached request (`input_tokens=0, cache_read_tokens=1000`) prices the cache reads at the provider cache rate, not zero. - `test_fallback_input_cost_uses_breakdown_sum_not_input_tokens_when_litellm_unavailable` verifies that when LiteLLM is unavailable and both `input_tokens` and a nonzero cache breakdown are supplied, the fallback prices only the breakdown sum and not `input_tokens + breakdown_sum`, preventing double-counting. - `tests/test_savings_ledger.py` still passes locally, proving the sibling ledger consumer stays compatible with the helper fallback change. - Not tested: live provider traffic and historical backfill. Existing zero-dollar rows remain stored as they were written. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes ## Additional Notes `CHANGELOG.md` is unchanged because changelog generation is release-managed. The subscription contribution panel still has a separate USD wiring mismatch; this PR fixes the dashboard-facing `proxy_savings.json` path named in the latest issue follow-up and keeps historical backfill out of scope.
This commit is contained in:
Rod Boev
GitHub
parent
728b33088b
commit
188e382b44
2 changed files with 169 additions and 18 deletions
|
|
@ -34,6 +34,7 @@ PROJECT_NAME_MAX_LENGTH = 128
|
|||
DEFAULT_MAX_HISTORY_AGE_DAYS = 365
|
||||
DEFAULT_MAX_RESPONSE_HISTORY_POINTS = 500
|
||||
DEFAULT_DISPLAY_SESSION_INACTIVITY_MINUTES = 60
|
||||
DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN = 3.0 / 1_000_000
|
||||
|
||||
LITELLM_AVAILABLE = importlib.util.find_spec("litellm") is not None
|
||||
litellm: Any | None = None
|
||||
|
|
@ -189,18 +190,20 @@ def _resolve_litellm_model(model: str) -> str:
|
|||
def _estimate_compression_savings_usd(model: str, tokens_saved: int) -> float:
|
||||
"""Estimate compression savings in USD from saved input tokens."""
|
||||
litellm = _get_litellm_module()
|
||||
if tokens_saved <= 0 or litellm is None:
|
||||
if tokens_saved <= 0:
|
||||
return 0.0
|
||||
if litellm is None:
|
||||
return float(tokens_saved) * float(DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN)
|
||||
|
||||
try:
|
||||
resolved = _resolve_litellm_model(model)
|
||||
info = litellm.model_cost.get(resolved, {})
|
||||
input_cost_per_token = info.get("input_cost_per_token")
|
||||
if not input_cost_per_token:
|
||||
return 0.0
|
||||
raise RuntimeError("input cost unavailable")
|
||||
return float(tokens_saved) * float(input_cost_per_token)
|
||||
except Exception:
|
||||
return 0.0
|
||||
return float(tokens_saved) * float(DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN)
|
||||
|
||||
|
||||
def _estimate_input_cost_usd(
|
||||
|
|
@ -220,23 +223,31 @@ def _estimate_input_cost_usd(
|
|||
cache_read = _coerce_int(cache_read_tokens)
|
||||
cache_write = _coerce_int(cache_write_tokens)
|
||||
uncached = _coerce_int(uncached_input_tokens)
|
||||
litellm = _get_litellm_module()
|
||||
# Gate on tokens actually sent. Providers like Anthropic report cache
|
||||
# reads/writes separately from `input_tokens` (the uncached portion), so a
|
||||
# fully prefix-cached request has input_tokens == 0 while cache_read > 0.
|
||||
# Bailing on `input_tokens <= 0` alone dropped the real cache-read cost,
|
||||
# leaving days with compression savings but zero recorded spend.
|
||||
if total_input_tokens + cache_read + cache_write + uncached <= 0 or litellm is None:
|
||||
|
||||
# Prefer the breakdown when callers supply segmented token counts.
|
||||
# Never add `input_tokens` on top of the breakdown to avoid double-counting.
|
||||
use_breakdown = (cache_read + cache_write + uncached) > 0
|
||||
chargeable_tokens = (
|
||||
(cache_read + cache_write + uncached) if use_breakdown else total_input_tokens
|
||||
)
|
||||
if chargeable_tokens <= 0:
|
||||
return 0.0
|
||||
|
||||
litellm = _get_litellm_module()
|
||||
# Keep exact provider pricing authoritative when available.
|
||||
# `litellm` can be present but lack an entry for the resolved model,
|
||||
# in which case we fall back to a blended rate instead of zeroing usage.
|
||||
if litellm is None:
|
||||
return float(chargeable_tokens) * float(DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN)
|
||||
|
||||
try:
|
||||
resolved = _resolve_litellm_model(model)
|
||||
info = litellm.model_cost.get(resolved, {})
|
||||
input_cost_per_token = info.get("input_cost_per_token")
|
||||
if not input_cost_per_token:
|
||||
return 0.0
|
||||
raise RuntimeError("input cost unavailable")
|
||||
|
||||
if cache_read + cache_write + uncached > 0:
|
||||
if use_breakdown:
|
||||
cache_read_cost = info.get(
|
||||
"cache_read_input_token_cost",
|
||||
input_cost_per_token,
|
||||
|
|
@ -253,7 +264,7 @@ def _estimate_input_cost_usd(
|
|||
|
||||
return float(total_input_tokens) * float(input_cost_per_token)
|
||||
except Exception:
|
||||
return 0.0
|
||||
return float(chargeable_tokens) * float(DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN)
|
||||
|
||||
|
||||
def _normalize_history_entry(entry: Any) -> dict[str, Any] | None:
|
||||
|
|
|
|||
|
|
@ -327,8 +327,12 @@ def test_litellm_resolution_and_savings_estimation_fallbacks(monkeypatch):
|
|||
) == pytest.approx(0.2)
|
||||
|
||||
fake_litellm.model_cost = {}
|
||||
assert savings_tracker_module._estimate_compression_savings_usd("gpt-4o", 100) == 0.0
|
||||
assert savings_tracker_module._estimate_input_cost_usd("gpt-4o", 100) == 0.0
|
||||
assert savings_tracker_module._estimate_compression_savings_usd("gpt-4o", 100) == pytest.approx(
|
||||
100 * savings_tracker_module.DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN
|
||||
)
|
||||
assert savings_tracker_module._estimate_input_cost_usd("gpt-4o", 100) == pytest.approx(
|
||||
100 * savings_tracker_module.DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN
|
||||
)
|
||||
|
||||
monkeypatch.setattr(
|
||||
fake_litellm,
|
||||
|
|
@ -336,11 +340,115 @@ def test_litellm_resolution_and_savings_estimation_fallbacks(monkeypatch):
|
|||
lambda **kwargs: (_ for _ in ()).throw(RuntimeError("boom")),
|
||||
)
|
||||
assert savings_tracker_module._resolve_litellm_model("mystery-model") == "mystery-model"
|
||||
assert savings_tracker_module._estimate_compression_savings_usd("mystery-model", 100) == 0.0
|
||||
assert savings_tracker_module._estimate_compression_savings_usd(
|
||||
"mystery-model", 100
|
||||
) == pytest.approx(100 * savings_tracker_module.DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN)
|
||||
assert savings_tracker_module._estimate_input_cost_usd("mystery-model", 100) == pytest.approx(
|
||||
100 * savings_tracker_module.DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN
|
||||
)
|
||||
# Explicitly force the unavailable path for the whole tracker.
|
||||
monkeypatch.setattr(savings_tracker_module, "LITELLM_AVAILABLE", False)
|
||||
assert savings_tracker_module._estimate_compression_savings_usd("gpt-4o", 100) == pytest.approx(
|
||||
100 * savings_tracker_module.DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN
|
||||
)
|
||||
assert savings_tracker_module._estimate_input_cost_usd("gpt-4o", 100) == pytest.approx(
|
||||
100 * savings_tracker_module.DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN
|
||||
)
|
||||
|
||||
|
||||
def test_fallback_request_pricing_stays_nonzero_with_litellm_unavailable_and_preserves_historic_zeros(
|
||||
tmp_path, monkeypatch
|
||||
):
|
||||
# Legacy proxy_savings rows can legitimately store zero-dollar values.
|
||||
savings_path = tmp_path / "proxy_savings.json"
|
||||
savings_path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"schema_version": 3,
|
||||
"lifetime": {
|
||||
"requests": 1,
|
||||
"tokens_saved": 10,
|
||||
"compression_savings_usd": 0.0,
|
||||
"total_input_tokens": 120,
|
||||
"total_input_cost_usd": 0.0,
|
||||
},
|
||||
"display_session": {},
|
||||
"history": [
|
||||
{
|
||||
"timestamp": "2026-03-27T09:00:00Z",
|
||||
"provider": "openai",
|
||||
"model": "gpt-4o",
|
||||
"total_tokens_saved": 10,
|
||||
"compression_savings_usd": 0.0,
|
||||
"total_input_tokens": 120,
|
||||
"total_input_cost_usd": 0.0,
|
||||
}
|
||||
],
|
||||
"projects": {
|
||||
"fallback-demo": {
|
||||
"requests": 1,
|
||||
"tokens_saved": 10,
|
||||
"compression_savings_usd": 0.0,
|
||||
"total_input_tokens": 120,
|
||||
"total_input_cost_usd": 0.0,
|
||||
"last_activity_at": "2026-03-27T09:00:00Z",
|
||||
}
|
||||
},
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
tracker = SavingsTracker(path=str(savings_path))
|
||||
initial_snapshot = tracker.snapshot()
|
||||
assert initial_snapshot["lifetime"]["compression_savings_usd"] == 0.0
|
||||
assert initial_snapshot["display_session"]["compression_savings_usd"] == 0.0
|
||||
assert initial_snapshot["projects"]["fallback-demo"]["compression_savings_usd"] == 0.0
|
||||
assert initial_snapshot["history"][-1]["compression_savings_usd"] == 0.0
|
||||
|
||||
monkeypatch.setattr(savings_tracker_module, "LITELLM_AVAILABLE", False)
|
||||
assert savings_tracker_module._estimate_compression_savings_usd("gpt-4o", 100) == 0.0
|
||||
assert savings_tracker_module._estimate_input_cost_usd("gpt-4o", 100) == 0.0
|
||||
monkeypatch.setattr(savings_tracker_module, "litellm", None)
|
||||
assert tracker.record_request(
|
||||
model="gpt-4o",
|
||||
input_tokens=100,
|
||||
tokens_saved=50,
|
||||
project="fallback-demo",
|
||||
timestamp="2026-03-27T09:10:00Z",
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
savings_tracker_module,
|
||||
"_utc_now",
|
||||
lambda: datetime(2026, 3, 27, 9, 10, 30, tzinfo=timezone.utc),
|
||||
)
|
||||
|
||||
snapshot = tracker.snapshot()
|
||||
expected_savings_fallback = 50 * savings_tracker_module.DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN
|
||||
expected_input_fallback = 100 * savings_tracker_module.DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN
|
||||
assert snapshot["lifetime"]["compression_savings_usd"] == pytest.approx(
|
||||
expected_savings_fallback
|
||||
)
|
||||
assert snapshot["lifetime"]["total_input_cost_usd"] == pytest.approx(expected_input_fallback)
|
||||
assert snapshot["display_session"]["compression_savings_usd"] == pytest.approx(
|
||||
expected_savings_fallback
|
||||
)
|
||||
assert snapshot["display_session"]["total_input_cost_usd"] == pytest.approx(
|
||||
expected_input_fallback
|
||||
)
|
||||
assert snapshot["projects"]["fallback-demo"]["compression_savings_usd"] == pytest.approx(
|
||||
expected_savings_fallback
|
||||
)
|
||||
assert snapshot["projects"]["fallback-demo"]["total_input_cost_usd"] == pytest.approx(
|
||||
expected_input_fallback
|
||||
)
|
||||
assert snapshot["history"][-1]["compression_savings_usd"] == pytest.approx(
|
||||
expected_savings_fallback
|
||||
)
|
||||
|
||||
persisted = json.loads(savings_path.read_text(encoding="utf-8"))
|
||||
assert persisted["history"][0]["compression_savings_usd"] == 0.0
|
||||
assert persisted["history"][-1]["compression_savings_usd"] == pytest.approx(
|
||||
expected_savings_fallback
|
||||
)
|
||||
|
||||
|
||||
def test_input_cost_counts_cache_reads_when_uncached_input_is_zero(monkeypatch):
|
||||
|
|
@ -374,6 +482,38 @@ def test_input_cost_counts_cache_reads_when_uncached_input_is_zero(monkeypatch):
|
|||
assert cost == pytest.approx(0.3)
|
||||
|
||||
|
||||
def test_fallback_input_cost_uses_breakdown_sum_not_input_tokens_when_litellm_unavailable(
|
||||
monkeypatch,
|
||||
):
|
||||
# Regression: when both `input_tokens` and a nonzero cache breakdown are
|
||||
# present and LiteLLM is unavailable, the fallback must price only the
|
||||
# breakdown sum — never input_tokens + breakdown_sum — to avoid
|
||||
# double-counting the tokens that the breakdown already covers.
|
||||
monkeypatch.setattr(savings_tracker_module, "LITELLM_AVAILABLE", False)
|
||||
monkeypatch.setattr(savings_tracker_module, "litellm", None)
|
||||
|
||||
input_tokens = 1000
|
||||
cache_read = 200
|
||||
cache_write = 100
|
||||
uncached = 300
|
||||
breakdown_sum = cache_read + cache_write + uncached # 600
|
||||
|
||||
result = savings_tracker_module._estimate_input_cost_usd(
|
||||
"gpt-4o",
|
||||
input_tokens,
|
||||
cache_read_tokens=cache_read,
|
||||
cache_write_tokens=cache_write,
|
||||
uncached_input_tokens=uncached,
|
||||
)
|
||||
|
||||
fallback_rate = savings_tracker_module.DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN
|
||||
expected = breakdown_sum * fallback_rate
|
||||
double_counted = (input_tokens + breakdown_sum) * fallback_rate
|
||||
|
||||
assert result == pytest.approx(expected)
|
||||
assert result != pytest.approx(double_counted)
|
||||
|
||||
|
||||
def test_display_session_rolls_after_inactivity_and_counts_zero_savings_requests(
|
||||
tmp_path, monkeypatch
|
||||
):
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue