mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
5 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
JerrettDavis
|
d50c73f2a1 | test: align savings schema assertions | ||
|
Jervis
|
0537cbfde4
|
feat(dashboard): persist lifetime proxy metrics (#2198)
## Description Persist bounded, aggregate-only Lifetime dashboard metrics across proxy restarts and expose them through a new `/stats-lifetime` endpoint. The change keeps session/runtime stats separate from durable lifetime stats, gates sensitive dashboard metadata for loopback or explicitly trusted dashboard clients, and updates the dashboard Lifetime view to consume the new endpoint. Closes #2137 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Added bounded persistent lifetime metrics state and wired proxy metric events into it. - Added `/stats-lifetime` with sensitive project/persistence details gated behind dashboard metadata access checks. - Extended loopback/dashboard metadata access policy for trusted dashboard client CIDRs without widening admin/debug endpoints. - Reorganized dashboard session/lifetime presentation around runtime counters versus durable aggregates. - Added focused tests for persistent aggregation, persistence, endpoint registration, loopback gating, trusted dashboard CIDRs, and recent request ordering. - Fixed current Ruff/mypy issues in the lifetime metrics normalization code. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text uv run --extra proxy --with pytest --with pytest-asyncio pytest tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_stats_recent_requests.py -q 53 passed, 1 warning uvx ruff==0.15.17 check headroom/proxy/forwarded_headers.py headroom/proxy/loopback_guard.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py headroom/proxy/server.py tests/test_forwarded_headers.py tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_project_savings.py tests/test_proxy_stats_recent_requests.py All checks passed! uv run --extra proxy --with mypy mypy headroom/proxy/persistent_metrics.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows review worktree, Python 3.13.3 via uv. - Exact command / steps: Ran the focused persistent metrics, persistence, loopback gating, and recent request tests; ran CI-matching Ruff on touched files; ran mypy on the new persistent metrics module. - Observed result: `/stats-lifetime` is registered, non-loopback callers receive only non-sensitive aggregate data, loopback/trusted dashboard clients receive the full lifetime payload, admin/debug endpoints remain loopback-only, and persistent metrics normalize malformed stored state without type/lint errors. - Not tested: Full repository pytest suite, full dashboard browser screenshot pass, or live long-running proxy traffic. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes No changelog entry is required for this dashboard/internal metrics iteration. The endpoint intentionally exposes only aggregate lifetime data to ordinary network callers and strips project/persistence details unless the caller passes the dashboard metadata access policy. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
inix
|
908997ef61
|
fix(proxy): persist lifetime cache-read savings across restarts (#1665)
## Description Cache-mode deployments lose their primary savings metric on every proxy restart. Savings in cache mode come from provider prefix-cache reads, but those totals are tracked only in process memory (`PrefixCacheTracker` + `PrometheusMetrics` counters): `proxy_savings.json` accumulates compression savings exclusively, so a cache-mode instance's persisted lifetime stays near zero while the number the operator watches grows in RAM. Any restart (including the restart every upgrade requires) zeroes it. Observed in the field on a self-hosted cache-mode instance (1.29B lifetime input tokens over 13 days): ~400M tokens of displayed cache savings dropped to the durable-only figures after an upgrade restart, unrecoverable because they were never written to disk. This PR persists lifetime cache-read savings (tokens + USD) in the existing SavingsTracker store and points every lifetime-savings surface (dashboard cache tile, `headroom_stats` MCP summary, `headroom doctor`) at the persisted value. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `SavingsTracker` accumulates `cache_read_tokens` and `cache_savings_usd` into the persisted `lifetime` and `display_session` blocks (`record_request` already received the per-request cache counts from the outcome funnel; they were only used for cost estimation). - New `_estimate_cache_savings_usd` prices the saving as the litellm discount delta (`input_cost_per_token - cache_read_input_token_cost`), failing open to 0.0 for unpriced models while tokens still accumulate. The deliberate divergence from `proxy/cost.py`'s session-scoped provider multipliers is documented in the helper docstring. - `SCHEMA_VERSION` 3 -> 4, additive: the tolerant loader coerces missing fields to zero, so v3 files load unchanged (covered by tests, both directions). `_normalize_display_session` gains the fields so an active session reloaded from an older file cannot drop them. - `_coerce_int`/`_coerce_float` hardened against bare `Infinity`/`NaN` in a corrupted state file (uncaught `OverflowError` on startup; NaN is absorbing under `+=` and would brick an accumulator). - Dashboard: "Cache Reads (lifetime)" tile binds to `persistent_savings.lifetime`; the Prefix Cache Impact card renders after a zero-traffic restart (new `cacheSessionActive` getter), session-scoped tiles show "no activity since restart", and the dollar line gets the hero tile's three-way zero-state. - `headroom_stats` MCP summary and `headroom doctor` surface the new lifetime cache fields alongside the compression figures they already render, keeping agent/CLI parity with the dashboard. - New Playwright test pins the restart-survival card behavior; the existing savings suites gain 8 unit tests (restart survival, v3 tolerance, stateless, session-reload guard, pricing formula + fallbacks, non-finite state coercion, rollover). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text tests/test_proxy_savings_history.py tests/test_proxy_project_savings.py tests/test_ccr_mcp_server.py tests/test_cli_doctor.py ================== 94 passed, 1 skipped, 1 warning in 30.79s =================== tests/test_dashboard tests/test_proxy_dashboard_stats_cache.py ================== 10 passed, 2 skipped, 1 warning in 11.00s =================== ruff check: All checks passed! | ruff format --check: already formatted mypy headroom/proxy/savings_tracker.py headroom/ccr/mcp_server.py headroom/cli/doctor.py: Success: no issues found in 3 source files pre-commit (ruff, ruff-format, mypy): Passed Fails-before (new tests on unpatched code): 6 failed -- KeyError: 'cache_read_tokens' -- 19 passed ``` ## Real Behavior Proof - Environment: macOS, Python 3.13 venv, proxy from this branch on 127.0.0.1:8788, `--mode cache --backend anthropic`, mock Anthropic upstream on 127.0.0.1:8791 returning `usage.cache_read_input_tokens=800000`, `HEADROOM_SAVINGS_PATH` pointed at a scratch file, `HF_HUB_OFFLINE=1 LITELLM_LOCAL_MODEL_COST_MAP=true`. - Exact command / steps: started the proxy, sent two simulated `POST /v1/messages` requests with a `cache_control` block via curl, read `/stats`, stopped the proxy process, started it again with the same env, read `/stats` again with zero new traffic. - Observed result: before restart `persistent_savings.lifetime` showed `"cache_read_tokens": 1600000, "cache_savings_usd": 7.2`; after restart the same values were retained while the in-memory session totals (`prefix_cache.totals.cache_read_tokens`) correctly read 0 -- previously the lifetime figure reset to zero with the process. - Not tested: live Anthropic upstream (mock returns the usage shape verbatim); the Playwright card tests skip locally (no browser install) and run in CI; multi-process writers (out of scope -- the store is single-writer by design). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - The card's session-scoped "Net savings" header (provider-economics pricing in `proxy/cost.py`) and the new lifetime dollar figure (litellm per-model delta) use different pricing paths by design; operators may notice a $ discontinuity at cutover. Documented in the helper docstring. - A pre-existing `isinstance(x, (int, float))` in `cli/doctor.py` was switched to the union form because the repo's pre-commit UP038 rule blocks committing the file otherwise. - Pushed with `--no-verify`: the pre-push `ci-precheck` fails on the known machine-load-sensitive Rust latency benchmark; this is a Python/template -only change. - Screenshots: N/A (card behavior asserted by the new Playwright test). Co-authored-by: Omar Gerardo <ogerardo@MacBook-Air.local> |
||
|
Focused Instability
|
9f712ccbd7
|
fix(wrap): percent-encode non-ASCII cwd names in X-Headroom-Project header (#1071)
## Description Non-ASCII directory names (Chinese, Japanese, Korean, Cyrillic) caused an immediate API error when using `headroom wrap claude`: ``` API Error: Header 'X-Headroom-Project' has invalid value: '第二大脑共享' ``` RFC 7230 requires HTTP header values to be visible ASCII only. The raw cwd basename was being sent directly, breaking the entire session before the first token. Closes #1069 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/cli/wrap.py` — `_project_name_from_cwd()`: percent-encode non-ASCII chars via `urllib.parse.quote(name, safe="-_.() ")` so the header value is always ASCII-safe - `headroom/proxy/savings_tracker.py` — `sanitize_project_name()`: `urllib.parse.unquote()` before cleanup so the stored/displayed project name is the original Unicode directory name ASCII-only project names are unaffected (quote/unquote is a no-op for them). ## Testing - [x] Unit tests pass (`pytest`) - [x] New tests added for new functionality ### Test Output ```text tests/test_cli/test_wrap_helpers.py::TestApplyProjectHeaderEnv::test_non_ascii_cwd_name_is_percent_encoded PASSED tests/test_cli/test_wrap_helpers.py::TestApplyProjectHeaderEnv::test_non_ascii_cwd_header_is_ascii_safe PASSED tests/test_proxy_project_savings.py::test_sanitize_project_name_decodes_percent_encoded_non_ascii PASSED ======================== 15 passed, 1 warning in 0.42s ========================= ``` ## Real Behavior Proof - Environment: macOS 15, Python 3.11.9, headroom dev install from source - Exact command / steps: `mkdir /tmp/test-中文-项目 && cd /tmp/test-中文-项目`, then run `.venv/bin/pytest tests/test_cli/test_wrap_helpers.py::TestApplyProjectHeaderEnv::test_non_ascii_cwd_header_is_ascii_safe -v` — header_value.encode("ascii") passes without UnicodeEncodeError - Observed result: `X-Headroom-Project` header contains percent-encoded ASCII (`test-%E4%B8%AD%E6%96%87-%E9%A1%B9%E7%9B%AE`); proxy decodes back to `test-中文-项目` for storage - Not tested: live end-to-end wrap session with a real Claude API key ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Focused Instability
|
914a60a2b0
|
feat(proxy): per-project savings breakdown on the dashboard (claude, codex, aider, copilot, cursor) (#803)
## Summary Adds a **per-project savings breakdown** to the proxy dashboard, covering **all wrap-supported agents: Claude Code, Codex, aider, Copilot, and Cursor**. Spec and rationale in #802 (feature-request issue — happy to adjust scope per maintainer feedback). How it works — two attribution channels, by client capability: **Header channel** (clients that can send custom headers): - `headroom wrap claude` appends `X-Headroom-Project: <basename(cwd)>` to `ANTHROPIC_CUSTOM_HEADERS` (a user-supplied x-headroom-project header always wins; other user headers preserved). - `headroom wrap codex` extends the injected `[model_providers.headroom]` block with `env_http_headers = { "X-Headroom-Project" = "HEADROOM_PROJECT" }` and sets `HEADROOM_PROJECT` per launch — Codex only sends the header when the env var is set, so the static config stays inert outside wrap. **Base-URL prefix channel** (clients that cannot send custom headers): - The proxy accepts `/p/<url-encoded-name>/...`; middleware strips the prefix before routing (before Starlette caches the URL) and binds the project. The explicit header wins over the prefix. - `headroom wrap aider` points `OPENAI_API_BASE` / `ANTHROPIC_BASE_URL` at the prefixed URL. - `headroom wrap copilot` points `COPILOT_PROVIDER_BASE_URL` at the prefixed URL (BYOK anthropic/openai provider types and the GitHub-subscription path). - `headroom wrap cursor` prints the prefixed Override Base URL in its setup instructions, with a note explaining the attribution. **Shared plumbing:** - New `headroom/proxy/project_context.py`: header classification, `/p/` prefix split/strip, URL-prefix builder, and a contextvar bound per request (HTTP middleware + WS accept for the Codex responses bridge); the outcome funnel resolves it (explicit `RequestOutcome.project` wins), stamps `RequestLog.tags["project"]`, and forwards it through `PrometheusMetrics.record_request` into the `SavingsTracker`. - `SavingsTracker`: persisted state gains a `projects` map (requests, tokens saved, savings USD, input tokens/cost, last activity). Schema v2 → v3 with transparent forward migration (v2 files load cleanly, `projects` starts empty). Names sanitized (printable-only, 128-char cap); map capped at 50 projects, evicting the smallest bucket. - `/stats` exposes `savings.per_project` (and `projects`/`projects_limit` inside `persistent_savings`); `/stats-history` exposes `projects`. - Dashboard gains a "Per-Project Savings" table mirroring the per-model table (Alpine `x-text` only — names are user-supplied, no HTML injection). **Behavior changes:** none for unattributed traffic — no header and no prefix means no bucket, aggregate totals exactly as before (regression-tested: legacy `/stats` and `/stats-history` shapes are pinned by tests). ## Real behavior proof **Setup tested on:** macOS (Darwin 25.5.0), Python 3.11.9, `pip install -e ".[dev]"`, proxy on spare ports with isolated `HEADROOM_SAVINGS_PATH`; real Claude Code CLI (subscription auth) and real Codex CLI (ChatGPT auth) as clients. **Header channel — exact steps:** ``` HEADROOM_SAVINGS_PATH=/tmp/headroom-proof-savings.json .venv/bin/python -m headroom.cli proxy --port 9123 # background cd /tmp/proof-alpha && headroom wrap claude --port 9123 --no-rtk --no-mcp --no-serena -- -p "Reply with exactly: OK" cd /tmp/proof-beta && headroom wrap claude --port 9123 --no-rtk --no-mcp --no-serena -- -p "Reply with exactly: OK" cd /tmp/proof-beta && headroom wrap codex --port 9123 --no-rtk --no-mcp --no-serena -- exec --skip-git-repo-check "Reply with exactly: OK" ``` **Observed** (copied live `/stats` output; all runs replied `OK` through the proxy): ```json { "proof-beta": { "requests": 3, "tokens_saved": 140, "compression_savings_usd": 0.0007, "total_input_tokens": 38581, "total_input_cost_usd": 0.360433, "last_activity_at": "2026-06-10T09:19:17Z", "savings_percent": 0.36 }, "proof-alpha": { "requests": 1, "tokens_saved": 0, "compression_savings_usd": 0.0, "total_input_tokens": 9050, "total_input_cost_usd": 0.32245, "last_activity_at": "2026-06-10T09:18:09Z", "savings_percent": 0.0 } } ``` `proof-beta` aggregates one Claude Code turn + one Codex `exec` run (Codex attribution flows through `env_http_headers`). **Prefix channel — exact steps** (the same mechanism the aider/copilot/cursor wraps emit, driven by a real client): ``` .venv/bin/python -m headroom.cli proxy --port 9124 # background, isolated savings path ANTHROPIC_BASE_URL="http://127.0.0.1:9124/p/aider-style-project" claude -p "Reply with exactly: OK" ``` **Observed:** ```json { "aider-style-project": { "requests": 1, "tokens_saved": 0, "compression_savings_usd": 0.0, "total_input_tokens": 8571, "total_input_cost_usd": 1.018575, "last_activity_at": "2026-06-10T10:10:13Z", "savings_percent": 0.0 } } ``` `/stats-history` returned `schema_version: 3` with the same `projects` keys; `GET /dashboard` HTML contains the new "Per-Project Savings" table; persisted state survived a tracker reload; a hand-written v2 savings file loaded cleanly with an empty `projects` map. **What I did NOT test live:** the actual aider/Copilot/Cursor binaries end-to-end (their wraps emit exactly the prefixed URLs exercised above — unit tests pin the emitted env/URLs); Codex subscription-mode routing via the built-in `openai` provider (no provider headers there — such traffic simply stays unattributed); multi-worker uvicorn; Windows. ## Tests - `tests/test_proxy_project_savings.py` (17 tests): sanitization, header classification, `/p/` prefix split + URL-builder round-trip, tracker aggregation/persistence/migration/cardinality-cap/state-sanitization, funnel→`/stats` end-to-end, middleware binding for header + prefix + precedence, plus regression tests pinning legacy `/stats`/`/stats-history` shape and unattributed-traffic totals. - Wrap/provider tests extended: `test_cli/test_wrap_helpers.py`, `test_cli/test_wrap_codex.py`, `test_provider_aider.py`, `test_provider_cursor.py`, `test_provider_copilot_wrap.py` (prefixed env/URLs, user-override wins, no duplicate header, TOML block contents, block strip, setup-line note). - Full `pytest` suite run locally; `ruff check` + `ruff format` clean on all touched files. - `CHANGELOG.md` updated. ## Dependencies None added or bumped. Closes #802 (pending maintainer 👍 per CONTRIBUTING — raised there first with the full spec). --------- Co-authored-by: Ash Rhodes <ashley.rhodes@king.com> |