akadmin/headroom
1
0
Fork 0
mirror of https://github.com/headroomlabs-ai/headroom.git synced 2026-08-27 14:17:10 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
2630 commits 410 branches 169 tags 142 MiB
Commit graph

3 commits

Author SHA1 Message Date
Abhay Singh
fb17156bfa
fix(savings): don't bill free models at the $3/M fallback in the ledger (#2147) 
## Description

The durable savings ledger records phantom cost-avoided for free
(0-priced) models, billing them at the `$3/M` blended fallback.

`estimate_cost_usd` prices a known model via
`_estimate_compression_savings_usd`, but gates the result on `> 0`:

```python
if model and model != UNKNOWN:
    priced = _estimate_compression_savings_usd(model, tokens_saved)
    if priced > 0:                    # <-- the bug
        return round(priced, 6)
return round(float(tokens_saved) * float(fallback_rate), 6)   # ~$3/M
```

`_estimate_compression_savings_usd` deliberately distinguishes three
cases: a litellm-priced model (`> 0`), a model litellm can't price
(returns the blended fallback itself), and a model that is *legitimately
free* — litellm has an entry with `input_cost_per_token == 0.0`, so it
returns `tokens_saved * 0.0 == 0.0`. Its own comment calls this out:
"`if not ...` treated a real 0.0 as unavailable and billed the $3/M
fallback — phantom savings for a model that costs nothing."

The ledger's `if priced > 0` re-introduces exactly that defect: a free
model's `0.0` is treated as "unpriced" and the code falls through to the
`$3/M` fallback. Every saved token on a free/local/promo model is then
written into the durable JSONL ledger — and surfaced by `headroom
savings` / `aggregate_savings` — as cost-avoided that never existed.

## Fix

Trust `_estimate_compression_savings_usd`'s return verbatim for known
models. It already returns the blended fallback for models litellm can't
price and `0.0` for free ones, so the `> 0` gate is not needed — and is
the source of the double-fallback. The `UNKNOWN`/empty-model path still
uses `fallback_rate` as before.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/savings_ledger.py`: `estimate_cost_usd` returns
`_estimate_compression_savings_usd(...)` unconditionally for known
models instead of gating on `> 0`.
- `tests/test_savings_ledger.py`: add
`test_free_model_is_not_billed_at_fallback` (free model → $0) and
`test_priced_model_uses_litellm_estimate` (priced model → estimate),
both monkeypatching the helper.
- `CHANGELOG.md`: Bug Fixes entry.

## Testing

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uvx ruff@0.15.17 check headroom/savings_ledger.py tests/test_savings_ledger.py
All checks passed!
$ python -m py_compile headroom/savings_ledger.py tests/test_savings_ledger.py
OK
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17`. Importing
`headroom` pulls in the torch/transformers stack and a full `pytest`
gets OOM-killed on this box, so I verified the pricing with a
dependency-free script that replicates the gate and the helper's three
cases, and left the full pytest to CI.
- Exact command / steps: priced 1,000,000 saved tokens for a free model,
a priced model, a litellm-unknown named model, and the explicit
`UNKNOWN` sentinel, under the old (`> 0` gate) and new (unconditional)
logic.
- Observed result: the old logic bills the free model `$3.00` (phantom);
the new logic bills `$0.00`. The priced model (`$2.00`), the
litellm-unknown fallback (`$3.00`), and the `UNKNOWN`-path
`fallback_rate` are unchanged. The new tests assert the free-model `$0`
and the priced-model estimate via a monkeypatched helper.
- Not tested: a live litellm lookup for a real free model; full local
`pytest` deferred to CI (OOM, per above).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

The "unit tests pass locally" and "type checking" boxes are unchecked
because the full suite imports the ML stack, which I can't run in this
environment; the change removes a `> 0` gate in a pure pricing function,
verified by the standalone proof and the new tests. This is the same
category as the earlier zero-price-model fix, but at a distinct,
still-buggy call site (the durable ledger) — the earlier fix landed
inside `_estimate_compression_savings_usd`.

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
 2026-07-14 12:19:30 -04:00
Ztkent
b3a559ba56
fix(savings): cap ledger retention at 30 days (#1985) 
## Description
The durable savings ledger (`headroom savings`) retained up to 365 days
of history with an unbounded-sounding "All time" window. Long-lived
installs accumulate an ever-growing `~/.headroom/savings_events.jsonl`,
and `--days` had no upper bound so a caller could request an arbitrarily
large lookback. This caps retention at 30 days everywhere it's read,
shrinks the compaction threshold to match, and renames the "All time"
window to reflect what it actually is now: `Last 30 days`.

## Type of Change
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made
- `headroom/savings_ledger.py`: `DEFAULT_RETENTION_DAYS` 365 → 30; add
`MAX_RETENTION_DAYS = 30` and hard-clamp the lookback inside
`aggregate_savings` so no caller (CLI or programmatic) can read back
further than 30 days, regardless of the `retention_days` argument passed
in.
- `headroom/savings_ledger.py`: report window `all_time` →
`last_30_days` (the bucket is exactly 30-day-bounded now, so it doubles
as the lifetime view too). `_COMPACT_SIZE_BYTES` 8 MiB → 1 MiB, since a
30-day-bounded ledger should never need to grow large.
- `headroom/cli/savings.py`: `--days` is now `click.IntRange(min=1,
max=30)` (was unbounded); help text states the max. Window label `"All
time"` → `"Last 30 days"`, and the label column width bumped 11 → 12 so
the longer label stays aligned with the other rows' progress bars.
- `tests/test_savings_ledger.py`: updated window-label assertions; added
a hard-cap regression test (`retention_days=365` passed explicitly still
excludes a 60-day-old event) and a `--days` range-rejection test
(31/60/365 all rejected).

## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ ruff check headroom/savings_ledger.py headroom/cli/savings.py tests/test_savings_ledger.py
All checks passed!

$ ruff format --check headroom/savings_ledger.py headroom/cli/savings.py tests/test_savings_ledger.py
3 files already formatted

$ mypy headroom --ignore-missing-imports
Success: no issues found in 409 source files

$ pytest tests/test_savings_ledger.py -q
............ss....                                                     [100%]
16 passed, 2 skipped in 6.11s
```

(ruff `0.15.17`, mypy `1.20.2` — pinned to match
`.github/workflows/ci.yml`'s `lint` job. Full multi-shard suite left to
CI; ran the full touched-module suite locally.)

## Real Behavior Proof
- Environment: macOS (Darwin 25.5.0), Python 3.13.14, local `uv` venv;
branch built and installed via `uv tool install --force`.
- Exact command / steps: ran `headroom savings` against a ledger holding
multiple models' events (claude-opus-4-8, claude-sonnet-5,
claude-haiku-4-5) recorded across the retention window, then ran
`headroom savings --days 60` to exercise the new upper bound.
- Observed result: all three windows (Today / Last 7 days / Last 30
days) populate and are each bounded to at most 30 days; cost-avoided
breaks down per model; `--days 60` is rejected by the new `1..30` range
instead of silently accepted.
- Not tested: Windows/macOS native-wrapper e2e jobs — left to CI.

```text
$ headroom savings

Today        █████░░░░░░░░░░░  33.8%  saved 8,702,348 / 25,781,326 tokens  $25.5830
Last 7 days  ██████░░░░░░░░░░  36.3%  saved 11,289,737 / 31,072,254 tokens  $34.8287
Last 30 days ██████░░░░░░░░░░  38.2%  saved 14,449,516 / 37,821,634 tokens  $48.5385

Cost avoided per model:
  claude-opus-4-8          $33.0494
  claude-sonnet-5          $15.2989
  claude-haiku-4-5-20251001 $0.1902

$ headroom savings --days 60
Usage: headroom savings [OPTIONS]
Try 'headroom savings --help' for help.

Error: Invalid value for '--days': 60 is not in the range 1<=x<=30.
```

- Not tested: Windows/macOS native-wrapper e2e jobs — left to CI.

## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)
N/A — CLI text output only, see Real Behavior Proof above.
 2026-07-11 00:02:23 -05:00
Ztkent
978ffa0a6a
feat(savings): durable savings ledger + headroom savings command (#1127) 
## Description

Adds a durable, cross-process savings ledger and a `headroom savings`
CLI that shows cost avoided plus Today / Last 7 days / All time
breakdowns by model and client. Unlike `headroom_stats` (a per-session,
in-memory snapshot), the ledger is on disk and survives proxy and agent
restarts, and is safe across the many MCP processes Headroom spawns.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Add `headroom/savings_ledger.py`: append-only, `fcntl`-locked JSONL
ledger at `~/.headroom/savings_events.jsonl`, safe across concurrent
writers (main MCP server, each subagent, and the proxy), aggregated on
read so totals survive restarts.
- litellm list pricing for known models; blended `$3/1M` input-token
fallback for `model="unknown"` (MCP compressions do not know the
upstream model). Self-pruning: events past the 365-day retention window
are dropped on read and the file is compacted once large.
- Add `headroom savings` CLI (`headroom/cli/savings.py`) with `--json`,
`--days N`, and `--reset` flags.
- Proxy client attribution: `record_request` accepts `client` and
threads `outcome.client` into the ledger, so proxy events record the
real harness (claude-code, codex, cursor, …) from the existing
`classify_client()` detection, falling back to `"proxy"` only when
unidentified.
- MCP compress hook records the client (from `clientInfo.name`) and
tokens saved after each `headroom_compress`; `HEADROOM_MCP_CLIENT` /
`HEADROOM_MCP_MODEL` env overrides.
- Add the `savings_events_path()` helper +
`HEADROOM_SAVINGS_EVENTS_PATH` env in `headroom/paths.py`, the docs page
`docs/content/docs/savings.mdx`, and 15 tests.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

The single warning is a pre-existing, repo-wide
`StarletteDeprecationWarning` from
`fastapi.testclient` (the venv has `httpx`, not `httpx2`); it is
unrelated to this
change and fires in every proxy test that spins up a `TestClient`.

```text
$ .venv/bin/python -m pytest tests/test_savings_ledger.py -q
...............                                                          [100%]
15 passed, 1 warning in 5.17s
# warning: fastapi/testclient.py StarletteDeprecationWarning (httpx vs httpx2) — third-party, pre-existing

$ .venv/bin/ruff check headroom/savings_ledger.py headroom/cli/savings.py \
    headroom/ccr/mcp_server.py headroom/proxy/prometheus_metrics.py \
    headroom/proxy/outcome.py headroom/paths.py tests/test_savings_ledger.py
All checks passed!

$ .venv/bin/mypy headroom/savings_ledger.py headroom/cli/savings.py \
    headroom/ccr/mcp_server.py headroom/proxy/prometheus_metrics.py \
    headroom/proxy/outcome.py headroom/paths.py
Success: no issues found in 6 source files
```

## Real Behavior Proof

- Environment: macOS (Darwin 25.5.0), Python 3.13.13, editable install
of this branch, proxy running on :8787
- Exact command / steps: route live agent + proxy traffic through
Headroom, then run `headroom savings`
- Observed result: distinct Today / Last 7 days / All time windows with
per-model and per-client breakdowns, as below
- Not tested: Windows runtime (no `fcntl`; the ledger falls back to
best-effort append)

```text
Today       ██░░░░░░░░░░░░░░  11.3%  saved 472,870 / 4,193,288 tokens  $1.5920
Last 7 days ██░░░░░░░░░░░░░░  11.9%  saved 505,170 / 4,244,288 tokens  $1.7385
All time    ██░░░░░░░░░░░░░░  13.0%  saved 566,170 / 4,339,288 tokens  $1.9815

Cost avoided per model:
  claude-sonnet-4-6        $1.2200
  claude-opus-4-8          $0.6685
  gpt-5.5                  $0.0840
  claude-haiku-4-5         $0.0090

Savings by client:
  claude-code              60 calls · 524,970 tokens saved
  cursor                   2 calls · 16,800 tokens saved
  codex                    3 calls · 24,400 tokens saved
```

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

The one pytest warning is a third-party `StarletteDeprecationWarning`
from `fastapi.testclient` (pre-existing, repo-wide); not introduced
here. CHANGELOG.md not updated.
 2026-06-22 18:47:57 -05:00