fix(savings): don't fabricate output savings for a free (zero-priced) model (#2298)

## Description

`_estimate_output_savings_usd` reports phantom output-shaping savings
for a model whose output price is legitimately `0.0`.

It reads the per-token output price from litellm and treats a falsy
value as "unavailable":

```python
output_cost_per_token = info.get("output_cost_per_token")
if not output_cost_per_token:
    raise RuntimeError("output cost unavailable")
return float(tokens_saved) * float(output_cost_per_token)
```

`if not output_cost_per_token` is `True` for both a **missing** price
(`None`) *and* a real **`0.0`** (a free / local / vendored-at-zero
model). So for a free model it raises, hits the `except`, and bills the
saved output tokens at `DEFAULT_FALLBACK_OUTPUT_COST_PER_TOKEN` ($15/M)
— fabricating output savings for a model that costs nothing.

This is the exact bug that `_estimate_compression_savings_usd` was
already fixed for (it now uses `if input_cost_per_token is None:`, with
a comment explaining that `if not ...` "treated a real 0.0 as
unavailable and billed the $3/M fallback — phantom savings").
`_estimate_input_cost_usd` carries the same fix.
`_estimate_output_savings_usd` is the one that was missed.

## Fix

Fall back only when the price is truly missing:

```python
if output_cost_per_token is None:
    raise RuntimeError("output cost unavailable")
```

A `0.0` price now correctly yields `$0` output savings; a missing price
still falls back to the estimate; a real price is unchanged.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/savings_tracker.py`: `_estimate_output_savings_usd`
falls back on `output_cost_per_token is None` instead of `not
output_cost_per_token`.
- `tests/test_savings_tracker_zero_price.py`: new tests (free → $0,
unknown → fallback, paid → real price), alongside the existing
compression/input-cost zero-price tests.
- `CHANGELOG.md`: Bug Fixes entry.

## Testing

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uvx ruff@0.15.17 check headroom/proxy/savings_tracker.py tests/test_savings_tracker_zero_price.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/proxy/savings_tracker.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17` / `uvx
mypy@1.20.2`. A full `pytest` OOM-kills this box (ML stack import), so I
reproduced the estimator with a dependency-free script and left the full
pytest to CI.
- Exact command / steps: ran 1,000,000 saved output tokens through the
OLD `if not ...` and NEW `is None` logic for a free model
(`output_cost_per_token = 0.0`), a paid model, and an unknown model
(`None`).
- Observed result: OLD bills the free model at the $15/M fallback
(`$0.015` phantom savings); NEW returns `$0.00`. The paid model is
unchanged; the unknown model still falls back under both.
- Not tested: a live proxy run pricing a free model end-to-end; full
local `pytest` deferred to CI (OOM).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

The "unit tests pass locally" box is unchecked because the full suite
imports the ML stack, which I can't run here. The new tests reuse the
`_fake_litellm` harness in `tests/test_savings_tracker_zero_price.py`
(the same file that pins the compression/input-cost zero-price
behavior), so they run under the normal CI pytest job; behaviour is
additionally verified by the standalone proof above.

---------

Co-authored-by: Tejas Chopra <chopratejas@gmail.com>

headroom/proxy/savings_tracker.py

@@ -238,7 +238,12 @@ def _estimate_output_savings_usd(model: str, tokens_saved: int) -> float:
         resolved = _resolve_litellm_model(model)
         info = litellm.model_cost.get(resolved, {})
         output_cost_per_token = info.get("output_cost_per_token")
-        if not output_cost_per_token:
+        # Distinguish "price unknown" (missing key -> fall back to the estimate)
+        # from a model that is legitimately free (output_cost_per_token == 0.0).
+        # `if not ...` treated a real 0.0 as unavailable and billed the fallback
+        # rate -> phantom output savings for a model that costs nothing. Mirrors
+        # the fix already applied to `_estimate_compression_savings_usd`.
+        if output_cost_per_token is None:
             raise RuntimeError("output cost unavailable")
         return float(tokens_saved) * float(output_cost_per_token)
     except Exception:

tests/test_savings_tracker_zero_price.py

@@ -15,8 +15,10 @@ import types
 from headroom.proxy import savings_tracker as st
 from headroom.proxy.savings_tracker import (
     DEFAULT_FALLBACK_INPUT_COST_PER_TOKEN,
+    DEFAULT_FALLBACK_OUTPUT_COST_PER_TOKEN,
     _estimate_compression_savings_usd,
     _estimate_input_cost_usd,
+    _estimate_output_savings_usd,
 )
 
 
@@ -61,3 +63,30 @@ def test_input_cost_zero_for_free_model(monkeypatch):
         lambda: _fake_litellm({"free-model": {"input_cost_per_token": 0.0}}),
     )
     assert _estimate_input_cost_usd("free-model", 500_000) == 0.0
+
+
+def test_output_savings_zero_for_free_model(monkeypatch):
+    # output_cost_per_token == 0.0 (free model) must yield $0, not the fallback.
+    monkeypatch.setattr(
+        st,
+        "_get_litellm_module",
+        lambda: _fake_litellm({"free-model": {"output_cost_per_token": 0.0}}),
+    )
+    assert _estimate_output_savings_usd("free-model", 1_000_000) == 0.0
+
+
+def test_output_savings_falls_back_for_unknown_model(monkeypatch):
+    # Model absent from litellm → output_cost_per_token is None → fall back.
+    monkeypatch.setattr(st, "_get_litellm_module", lambda: _fake_litellm({}))
+    got = _estimate_output_savings_usd("unknown-model", 1_000_000)
+    assert got == 1_000_000 * DEFAULT_FALLBACK_OUTPUT_COST_PER_TOKEN
+
+
+def test_output_savings_uses_real_price_for_paid_model(monkeypatch):
+    price = 15.0 / 1_000_000
+    monkeypatch.setattr(
+        st,
+        "_get_litellm_module",
+        lambda: _fake_litellm({"paid-model": {"output_cost_per_token": price}}),
+    )
+    assert _estimate_output_savings_usd("paid-model", 1_000_000) == 1_000_000 * price