From 75d786117a15b55ca31daa2b2f07b123994d74d1 Mon Sep 17 00:00:00 2001 From: Dima Solodukha Date: Sat, 11 Jul 2026 02:12:32 +0300 Subject: [PATCH] fix(proxy): cache_savings_usd silently zeroes when litellm is unavailable (#2005) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Description On any install where `litellm` cannot be imported, `SavingsTracker`'s `lifetime.cache_savings_usd` and `display_session.cache_savings_usd` stay pinned at exactly `0.0` forever — while `cache_read_tokens` accumulates correctly and `total_input_cost_usd` stays nonzero, so the tracker looks alive and the zero is easy to miss. This hits every Python 3.14 install out of the box: the project's own dependency spec is `litellm>=1.86.2,<2.0 ; python_full_version < '3.14'`, so on 3.14 `LITELLM_AVAILABLE` is `False` and cache savings silently read as $0. Observed in the wild with 45.8M lifetime `cache_read_tokens` and `cache_savings_usd: 0.0` in `proxy_savings.json`. Root cause: `_estimate_cache_savings_usd` is the only one of the three USD estimators with no fallback when litellm is missing — `_estimate_input_cost_usd` and `_estimate_compression_savings_usd` both fall back to `DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN`, while `_estimate_cache_savings_usd` returns `0.0`. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/savings_tracker.py`: when litellm is unavailable, `_estimate_cache_savings_usd` now estimates at `DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN` per cache-read token — mirroring the fallback its two sibling estimators already use (approximate over zero). Unknown-model behaviour with litellm present is unchanged (still fails open to `0.0`). - `tests/test_proxy_savings_history.py`: new regression test `test_cache_savings_usd_falls_back_when_litellm_unavailable` (unit + through `SavingsTracker.record_request`); `test_cache_savings_edge_cases_zero_and_unpriced` now pins a fake litellm price table so it keeps testing the unpriced-model path on every environment (without litellm installed it would otherwise exercise the fallback path instead). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_proxy_savings_history.py -k "cache_savings" -q ========================= 3 passed, 1 warning in 0.34s ========================= $ ruff check headroom/proxy/savings_tracker.py tests/test_proxy_savings_history.py All checks passed! $ ruff format --check headroom/proxy/savings_tracker.py tests/test_proxy_savings_history.py 2 files already formatted $ mypy headroom/proxy/savings_tracker.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: macOS, Python 3.14.6 venv, headroom installed editable — litellm absent (excluded by the project's own `python_full_version < '3.14'` marker), i.e. the exact environment the bug ships in. - Exact command / steps: ran the one-liner below twice in that venv — once with `headroom/proxy/savings_tracker.py` checked out from main (`d2170b19`), once from this branch — output pasted verbatim: ```text $ python -c 'import headroom.proxy.savings_tracker as st; print("litellm importable:", st._get_litellm_module() is not None); print("cache_savings_usd for 1M cache-read tokens:", st._estimate_cache_savings_usd("claude-sonnet-4-6", 1_000_000))' # on main (d2170b19): litellm importable: False cache_savings_usd for 1M cache-read tokens: 0.0 # on this branch: litellm importable: False cache_savings_usd for 1M cache-read tokens: 3.0 ``` - Observed result: with litellm missing, main reports $0 saved for 1M cache-read tokens; this branch reports the blended-rate estimate ($3.00 at the default fallback rate), consistent with what `_estimate_input_cost_usd` already does for input cost. - Not tested: proxy end-to-end on Python < 3.14 with litellm installed (that path is unchanged — the litellm branch of the function is untouched, covered by the existing `test_cache_savings_usd_uses_litellm_discount_delta`). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - Documentation / CHANGELOG: no user-facing docs describe the per-function fallback behaviour, and I didn't find a maintained CHANGELOG.md at the repo root — happy to add an entry if there's a preferred place. - Full `tests/test_proxy_savings_history.py` run in my venv shows 7 failures that are identical on current main (missing optional dashboard deps in my environment, unrelated to this change); every test touching this change passes. - `mypy headroom/proxy/savings_tracker.py` also prints a pre-existing `pyproject.toml: note: unused section(s)` notice unrelated to this diff. --- headroom/proxy/savings_tracker.py | 12 +++++++--- tests/test_proxy_savings_history.py | 37 ++++++++++++++++++++++++++++- 2 files changed, 45 insertions(+), 4 deletions(-) diff --git a/headroom/proxy/savings_tracker.py b/headroom/proxy/savings_tracker.py index 92852bce2..32bda91dc 100644 --- a/headroom/proxy/savings_tracker.py +++ b/headroom/proxy/savings_tracker.py @@ -218,16 +218,22 @@ def _estimate_cache_savings_usd(model: str, cache_read_tokens: int) -> float: """Estimate cache-read savings in USD — the discount delta vs list price. Cache reads bill at the provider's discounted rate, so the saving per token - is ``input_cost_per_token - cache_read_input_token_cost``. Unknown models or - an unavailable litellm price as 0.0 (fail open); tokens still accumulate. + is ``input_cost_per_token - cache_read_input_token_cost``. Unknown models + price as 0.0 (fail open); tokens still accumulate. An unavailable litellm + falls back to ``DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN``, matching + ``_estimate_input_cost_usd``/``_estimate_compression_savings_usd`` — otherwise + cache_savings_usd silently reads as $0 forever on any install without + litellm (e.g. Python 3.14, where headroom's own dependency spec excludes it). Deliberately diverges from ``proxy/cost.py``'s session-scoped provider multipliers (``_CACHE_ECONOMICS``): this lifetime figure follows the per-model litellm pricing the rest of this module already uses. """ litellm = _get_litellm_module() - if cache_read_tokens <= 0 or litellm is None: + if cache_read_tokens <= 0: return 0.0 + if litellm is None: + return float(cache_read_tokens) * float(DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN) try: resolved = _resolve_litellm_model(model) diff --git a/tests/test_proxy_savings_history.py b/tests/test_proxy_savings_history.py index 7e353d85a..117fd1ee6 100644 --- a/tests/test_proxy_savings_history.py +++ b/tests/test_proxy_savings_history.py @@ -1689,7 +1689,13 @@ def test_active_display_session_without_cache_fields_reloads_safely(tmp_path, mo assert session["requests"] == 2 -def test_cache_savings_edge_cases_zero_and_unpriced(tmp_path): +def test_cache_savings_edge_cases_zero_and_unpriced(tmp_path, monkeypatch): + # Pin a litellm whose price table doesn't know the model, so this stays a + # test of the unpriced-model path on every environment — on installs + # without litellm (e.g. Python 3.14) the blended-rate fallback would + # otherwise kick in and produce a nonzero estimate. + fake_litellm = SimpleNamespace(model_cost={}) + monkeypatch.setattr(savings_tracker_module, "_get_litellm_module", lambda: fake_litellm) path = tmp_path / "proxy_savings.json" tracker = SavingsTracker(path=str(path)) @@ -1785,6 +1791,35 @@ def test_cache_savings_usd_uses_litellm_discount_delta(tmp_path, monkeypatch): assert tracker.snapshot()["lifetime"]["cache_savings_usd"] == pytest.approx(2.7) +def test_cache_savings_usd_falls_back_when_litellm_unavailable(tmp_path, monkeypatch): + # Regression: on any install without litellm (e.g. Python 3.14, where + # headroom-ai's own dependency spec excludes it), cache_savings_usd must + # use the same DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN estimate that + # _estimate_input_cost_usd already falls back to — not silently read as + # $0 forever while cache_read_tokens and total_input_cost_usd keep + # accumulating normally. + monkeypatch.setattr(savings_tracker_module, "LITELLM_AVAILABLE", False) + monkeypatch.setattr(savings_tracker_module, "litellm", None) + + fallback_rate = savings_tracker_module.DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN + assert savings_tracker_module._estimate_cache_savings_usd( + "claude-sonnet-4-6", 1_000_000 + ) == pytest.approx(1_000_000 * fallback_rate) + + tracker = SavingsTracker(path=str(tmp_path / "proxy_savings.json")) + tracker.record_request( + model="claude-sonnet-4-6", + input_tokens=1_000, + tokens_saved=0, + cache_read_tokens=1_000_000, + timestamp="2026-07-02T00:00:00Z", + ) + snapshot = tracker.snapshot() + assert snapshot["lifetime"]["cache_read_tokens"] == 1_000_000 + assert snapshot["lifetime"]["cache_savings_usd"] > 0.0 + assert snapshot["lifetime"]["cache_savings_usd"] == pytest.approx(1_000_000 * fallback_rate) + + def test_non_finite_state_values_coerce_to_defaults(tmp_path): path = tmp_path / "proxy_savings.json" # json accepts bare Infinity/NaN literals; a corrupted file must not crash