mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
0537cbfde4
## Description Persist bounded, aggregate-only Lifetime dashboard metrics across proxy restarts and expose them through a new `/stats-lifetime` endpoint. The change keeps session/runtime stats separate from durable lifetime stats, gates sensitive dashboard metadata for loopback or explicitly trusted dashboard clients, and updates the dashboard Lifetime view to consume the new endpoint. Closes #2137 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Added bounded persistent lifetime metrics state and wired proxy metric events into it. - Added `/stats-lifetime` with sensitive project/persistence details gated behind dashboard metadata access checks. - Extended loopback/dashboard metadata access policy for trusted dashboard client CIDRs without widening admin/debug endpoints. - Reorganized dashboard session/lifetime presentation around runtime counters versus durable aggregates. - Added focused tests for persistent aggregation, persistence, endpoint registration, loopback gating, trusted dashboard CIDRs, and recent request ordering. - Fixed current Ruff/mypy issues in the lifetime metrics normalization code. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text uv run --extra proxy --with pytest --with pytest-asyncio pytest tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_stats_recent_requests.py -q 53 passed, 1 warning uvx ruff==0.15.17 check headroom/proxy/forwarded_headers.py headroom/proxy/loopback_guard.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py headroom/proxy/server.py tests/test_forwarded_headers.py tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_project_savings.py tests/test_proxy_stats_recent_requests.py All checks passed! uv run --extra proxy --with mypy mypy headroom/proxy/persistent_metrics.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows review worktree, Python 3.13.3 via uv. - Exact command / steps: Ran the focused persistent metrics, persistence, loopback gating, and recent request tests; ran CI-matching Ruff on touched files; ran mypy on the new persistent metrics module. - Observed result: `/stats-lifetime` is registered, non-loopback callers receive only non-sensitive aggregate data, loopback/trusted dashboard clients receive the full lifetime payload, admin/debug endpoints remain loopback-only, and persistent metrics normalize malformed stored state without type/lint errors. - Not tested: Full repository pytest suite, full dashboard browser screenshot pass, or live long-running proxy traffic. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes No changelog entry is required for this dashboard/internal metrics iteration. The endpoint intentionally exposes only aggregate lifetime data to ordinary network callers and strips project/persistence details unless the caller passes the dashboard metadata access policy. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
49 lines
1.9 KiB
Python
49 lines
1.9 KiB
Python
"""Tests forwarding existing proxy metric events to Lifetime storage."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
|
|
from headroom.proxy.prometheus_metrics import PrometheusMetrics
|
|
from headroom.proxy.savings_tracker import SavingsTracker
|
|
|
|
|
|
def test_runtime_metric_events_feed_lifetime_without_resetting_runtime_counters(tmp_path) -> None:
|
|
tracker = SavingsTracker(path=str(tmp_path / "proxy_savings.json"), save_flush_every=25)
|
|
metrics = PrometheusMetrics(savings_tracker=tracker)
|
|
|
|
metrics.record_stack("codex")
|
|
asyncio.run(
|
|
metrics.record_request(
|
|
provider="anthropic",
|
|
model="claude-test",
|
|
input_tokens=10,
|
|
output_tokens=3,
|
|
tokens_saved=2,
|
|
latency_ms=1,
|
|
cached=True,
|
|
attempted_input_tokens=12,
|
|
cache_read_tokens=5,
|
|
cache_write_1h_tokens=2,
|
|
waste_signals={"repetition": 4},
|
|
)
|
|
)
|
|
asyncio.run(metrics.record_failed(provider="anthropic", model="claude-test"))
|
|
asyncio.run(metrics.record_rate_limited(provider="anthropic", model="claude-test"))
|
|
asyncio.run(metrics.record_cache_bust(tokens_lost=7))
|
|
asyncio.run(metrics.record_cache_miss_attribution("anthropic", "prefix_change"))
|
|
|
|
lifetime = tracker.lifetime_response()
|
|
|
|
assert lifetime["requests"]["total"] == 1
|
|
assert lifetime["requests"]["cached"] == 1
|
|
assert lifetime["requests"]["failed"] == 1
|
|
assert lifetime["requests"]["rate_limited"] == 1
|
|
assert lifetime["requests"]["by_provider"] == {"anthropic": 1}
|
|
assert lifetime["requests"]["by_stack"] == {"codex": 1}
|
|
assert lifetime["tokens"]["output"] == 3
|
|
assert lifetime["tokens"]["attempted_input"] == 12
|
|
assert lifetime["prefix_cache"]["bust_tokens"] == 7
|
|
assert lifetime["prefix_cache"]["misses_by_reason"] == {"prefix_change": 1}
|
|
assert lifetime["waste_signals"] == {"repetition": 4}
|
|
assert metrics.requests_total == 1
|