mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
fix(dashboard): include RTK stats in the historical tab (#1324)
## Description Restart the proxy, open the dashboard, go to the Historical tab and the RTK stats are gone. The Session tab shows them fine, Historical just doesn't have them. The reason is where the two tabs get their numbers. The Session tab calls `_get_context_tool_stats()` live, which reads RTK's own stats file. The Historical tab calls `history_response()`, which only contains the persisted proxy-compression data. RTK savings are never written into that savings JSON, they live in the RTK tool's separate stats file, so after a restart Historical has nothing to show for them. The fix makes `/stats-history` do the same thing `/stats` already does: pull the live RTK stats with `_get_context_tool_stats()` and attach them to the history response under a `cli_filtering` key (with `tool`, `label`, `lifetime` and `session`). The Historical tab then renders an RTK card from `historyStats.cli_filtering.lifetime.tokens_saved`. A few notes: 1. The card is hidden when `cli_filtering` is null, so setups without RTK look exactly as they do today. No empty card, no errors. 2. Reading the RTK stats is best-effort: if `_get_context_tool_stats()` raises (missing file, parse error, IO), `cli_filtering` falls back to null and the Historical tab stays available rather than returning a 500. 3. Nothing about how RTK stats are stored changed, we just read them on the history endpoint too, so there's no migration. Closes #1177 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/server.py`: the `/stats-history` handler now attaches live RTK stats under `cli_filtering`, the same source `/stats` uses, wrapped in best-effort error handling; the endpoint docstring documents the curated shape. - `headroom/dashboard/templates/dashboard.html`: add an RTK card to the Historical tab, hidden when there's no RTK data. - `tests/test_proxy_savings_history.py`: `test_stats_history_includes_cli_filtering`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run --extra dev python -m pytest tests/test_proxy_savings_history.py -q passed ruff: All checks passed! mypy: Success: no issues found ``` ## Real Behavior Proof - Environment: macOS, Python 3.13, this branch. - Exact command / steps: `uv run --extra dev python -m pytest tests/test_proxy_savings_history.py::test_stats_history_includes_cli_filtering`. The test hits `/stats-history` and asserts the payload carries `cli_filtering` with the RTK numbers the Historical tab reads. - Observed result: the `/stats-history` response now carries `cli_filtering` (`tool`/`label`/`lifetime`/`session`), the field the Historical tab was missing after a restart. `ruff` and `mypy` are clean on the changed files. - Not tested: I did not click through the rendered dashboard after a real restart, and this repo's test suite needs the native `_core` extension built (CI builds it), so the assertion runs in CI. The data the tab consumes is covered by the test, and the card is gated on that data being present. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes
This commit is contained in:
Lucas Santos
GitHub
parent
fa05ebc849
commit
35939c3536
5 changed files with 135 additions and 6 deletions
|
|
@ -34,6 +34,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
|
||||
### Bug Fixes
|
||||
|
||||
* **dashboard:** include RTK stats in the Historical tab; `/stats-history` now attaches live RTK/CLI-filtering stats the same way the Session tab does, so they survive a proxy restart ([#1177](https://github.com/chopratejas/headroom/issues/1177)).
|
||||
* **proxy:** stop discarding a finished compression on very large requests. After the transform pipeline completed, a telemetry-only waste-signal re-parse of the *original* messages ran on the critical path; on huge Claude Code transcripts (~400k tokens) that parse could exceed the Anthropic compression timeout, so the proxy failed open and forwarded the uncompressed request despite "Pipeline complete" logging real savings (`tokens_saved: 0`, `transforms_applied: []`, ~31s latency). Waste-signal detection is now skipped above `MAX_WASTE_SIGNAL_DETECTION_TOKENS` (100k) so the compression result stays on the critical path ([#296](https://github.com/chopratejas/headroom/issues/296)).
|
||||
* **codex:** retag existing Codex threads when `headroom init` injects the `headroom` provider, so Codex Desktop history stays visible. Codex filters its sidebar/search by the active `model_provider`; the init path set `model_provider = "headroom"` without retagging, so existing native `openai` threads disappeared from the menu (data was never deleted, only hidden). `_ensure_codex_provider` now reconciles thread tags openai→headroom, matching what the install and `wrap` paths already do; `headroom unwrap codex` handles the revert direction ([#961](https://github.com/chopratejas/headroom/issues/961)).
|
||||
* **install:** stop duplicating the container ENTRYPOINT in the `persistent-docker` runtime command. The published image already runs `headroom proxy` as its ENTRYPOINT, but `build_runtime_command` re-added `headroom proxy` after the image name, so the container ran `headroom proxy headroom proxy --host 0.0.0.0 …` and Click aborted with "Got unexpected extra arguments (headroom proxy)" — the deployment never became ready and rollback left nothing running. The runtime command now appends only the proxy flags ([#833](https://github.com/chopratejas/headroom/issues/833)).
|
||||
|
|
|
|||
|
|
@ -1399,6 +1399,18 @@
|
|||
</div>
|
||||
<div class="mt-2 text-xs text-gray-500">Based on persisted weekly buckets</div>
|
||||
</div>
|
||||
|
||||
<template x-if="historyStats.cli_filtering">
|
||||
<div class="bg-surface rounded-lg p-4 border border-border">
|
||||
<div class="text-xs text-gray-500 uppercase tracking-wide mb-1"
|
||||
x-text="(historyStats.cli_filtering?.label || cliFilteringLabel) + ' Lifetime Saved'"></div>
|
||||
<div class="flex items-baseline gap-2">
|
||||
<span class="text-3xl font-light tabular-nums text-cyan-400"
|
||||
x-text="formatNumber(historyStats.cli_filtering?.lifetime?.tokens_saved || 0)"></span>
|
||||
</div>
|
||||
<div class="mt-2 text-xs text-gray-500">CLI output filtering (lifetime)</div>
|
||||
</div>
|
||||
</template>
|
||||
</div>
|
||||
|
||||
<template x-if="hasHistoricalData">
|
||||
|
|
|
|||
|
|
@ -3178,7 +3178,15 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
|||
series: Literal["history", "hourly", "daily", "weekly", "monthly"] = "history",
|
||||
history_mode: Literal["compact", "full", "none"] = "compact",
|
||||
):
|
||||
"""Get durable proxy compression history plus display-session state."""
|
||||
"""Get durable proxy compression history plus display-session state.
|
||||
|
||||
The JSON payload also carries a ``cli_filtering`` key with live RTK
|
||||
stats. This is a curated subset (``tool``, ``label``, ``lifetime``,
|
||||
``session``) tailored to the Historical tab, not the full
|
||||
``_get_context_tool_stats()`` payload that ``/stats`` exposes. It is
|
||||
``None`` when RTK is absent or its stats cannot be read, so the tab
|
||||
simply hides the card rather than erroring.
|
||||
"""
|
||||
if format == "csv":
|
||||
filename = f"headroom-stats-history-{series}.csv"
|
||||
return Response(
|
||||
|
|
@ -3187,7 +3195,30 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
|
|||
headers={"Content-Disposition": f'attachment; filename="{filename}"'},
|
||||
)
|
||||
|
||||
return proxy.metrics.savings_tracker.history_response(history_mode=history_mode)
|
||||
history = proxy.metrics.savings_tracker.history_response(history_mode=history_mode)
|
||||
|
||||
# Augment with live RTK/cli-filtering lifetime stats so the Historical
|
||||
# tab can display them. These live in the context-tool's own stats file
|
||||
# and survive proxy restarts — exactly what the Historical tab needs.
|
||||
# Best-effort: if the RTK stats file can't be read (missing, parse error,
|
||||
# IO), fall back to None so the Historical tab stays available instead of
|
||||
# 500ing. The tab hides the card when cli_filtering is None.
|
||||
try:
|
||||
cli_stats = await asyncio.to_thread(_get_context_tool_stats)
|
||||
except Exception:
|
||||
logger.debug("stats-history: RTK stats unavailable", exc_info=True)
|
||||
cli_stats = None
|
||||
if cli_stats:
|
||||
history["cli_filtering"] = {
|
||||
"tool": str(cli_stats.get("tool", "rtk")),
|
||||
"label": str(cli_stats.get("label", "RTK")),
|
||||
"lifetime": cli_stats.get("lifetime", {}),
|
||||
"session": cli_stats.get("session", {}),
|
||||
}
|
||||
else:
|
||||
history["cli_filtering"] = None
|
||||
|
||||
return history
|
||||
|
||||
@app.get("/transformations/feed", dependencies=[Depends(_require_loopback)])
|
||||
async def transformations_feed(limit: int = 20):
|
||||
|
|
|
|||
|
|
@ -954,3 +954,45 @@ def test_dashboard_includes_history_toggle_and_endpoint(tmp_path, monkeypatch):
|
|||
assert "historyModelSourceSeriesLabel + ' buckets'" in html
|
||||
# Non-top-5 breakdown rows swap into the last chart slot when selected.
|
||||
assert "topModels[topModels.length - 1] = selected;" in html
|
||||
|
||||
|
||||
def test_stats_history_includes_cli_filtering(tmp_path, monkeypatch):
|
||||
"""The /stats-history response must include cli_filtering (RTK) lifetime stats.
|
||||
|
||||
Before this fix the endpoint returned only proxy compression data; after a
|
||||
restart the Historical tab showed no RTK savings at all.
|
||||
"""
|
||||
pytest.importorskip("fastapi")
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
import headroom.proxy.server as server
|
||||
from headroom.proxy.server import ProxyConfig, create_app
|
||||
|
||||
savings_path = tmp_path / "proxy_savings.json"
|
||||
monkeypatch.setenv("HEADROOM_SAVINGS_PATH", str(savings_path))
|
||||
|
||||
_rtk_lifetime_payload = {
|
||||
"tool": "rtk",
|
||||
"label": "RTK",
|
||||
"tokens_saved": 999,
|
||||
"session": {"tokens_saved": 200, "commands": 5},
|
||||
"lifetime": {"tokens_saved": 999, "commands": 42},
|
||||
}
|
||||
monkeypatch.setattr(server, "_get_context_tool_stats", lambda: _rtk_lifetime_payload)
|
||||
|
||||
config = ProxyConfig(
|
||||
cache_enabled=False,
|
||||
rate_limit_enabled=False,
|
||||
log_requests=False,
|
||||
)
|
||||
|
||||
with TestClient(create_app(config)) as client:
|
||||
response = client.get("/stats-history")
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
|
||||
assert "cli_filtering" in data, "Historical /stats-history must include cli_filtering"
|
||||
assert data["cli_filtering"] is not None
|
||||
assert data["cli_filtering"]["tool"] == "rtk"
|
||||
assert data["cli_filtering"]["label"] == "RTK"
|
||||
assert data["cli_filtering"]["lifetime"]["tokens_saved"] == 999
|
||||
|
|
|
|||
51
uv.lock
generated
51
uv.lock
generated
|
|
@ -972,6 +972,15 @@ wheels = [
|
|||
{ url = "https://files.pythonhosted.org/packages/55/e2/2537ebcff11c1ee1ff17d8d0b6f4db75873e3b0fb32c2d4a2ee31ecb310a/docstring_parser-0.17.0-py3-none-any.whl", hash = "sha256:cf2569abd23dce8099b300f9b4fa8191e9582dda731fd533daf54c4551658708", size = 36896, upload-time = "2025-07-21T07:35:00.684Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "et-xmlfile"
|
||||
version = "2.0.0"
|
||||
source = { registry = "https://pypi.org/simple/" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/d3/38/af70d7ab1ae9d4da450eeec1fa3918940a5fafb9055e934af8d6eb0c2313/et_xmlfile-2.0.0.tar.gz", hash = "sha256:dab3f4764309081ce75662649be815c4c9081e88f0837825f90fd28317d4da54", size = 17234, upload-time = "2024-10-25T17:25:40.039Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/c1/8b/5fe2cc11fee489817272089c4203e679c63b570a5aaeb18d852ae3cbba6a/et_xmlfile-2.0.0-py3-none-any.whl", hash = "sha256:7a91720bc756843502c3b7504c77b8fe44217c85c537d85037f0f536151b2caa", size = 18059, upload-time = "2024-10-25T17:25:39.051Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "evaluate"
|
||||
version = "0.4.6"
|
||||
|
|
@ -1449,7 +1458,7 @@ wheels = [
|
|||
|
||||
[[package]]
|
||||
name = "headroom-ai"
|
||||
version = "0.26.0"
|
||||
version = "0.27.0"
|
||||
source = { editable = "." }
|
||||
dependencies = [
|
||||
{ name = "ast-grep-cli" },
|
||||
|
|
@ -1483,6 +1492,7 @@ all = [
|
|||
{ name = "onnxruntime", version = "1.23.2", source = { registry = "https://pypi.org/simple/" }, marker = "python_full_version < '3.14'" },
|
||||
{ name = "onnxruntime", version = "1.26.0", source = { registry = "https://pypi.org/simple/" }, marker = "python_full_version >= '3.14'" },
|
||||
{ name = "openai" },
|
||||
{ name = "openpyxl" },
|
||||
{ name = "opentelemetry-exporter-otlp-proto-http" },
|
||||
{ name = "opentelemetry-sdk" },
|
||||
{ name = "pillow" },
|
||||
|
|
@ -1496,10 +1506,12 @@ all = [
|
|||
{ name = "torch" },
|
||||
{ name = "trafilatura" },
|
||||
{ name = "transformers" },
|
||||
{ name = "tree-sitter" },
|
||||
{ name = "tree-sitter-language-pack" },
|
||||
{ name = "uvicorn" },
|
||||
{ name = "watchdog" },
|
||||
{ name = "websockets" },
|
||||
{ name = "xlrd" },
|
||||
{ name = "zstandard" },
|
||||
]
|
||||
anyllm = [
|
||||
|
|
@ -1514,6 +1526,7 @@ benchmark = [
|
|||
{ name = "openai" },
|
||||
]
|
||||
code = [
|
||||
{ name = "tree-sitter" },
|
||||
{ name = "tree-sitter-language-pack" },
|
||||
]
|
||||
dev = [
|
||||
|
|
@ -1528,6 +1541,7 @@ dev = [
|
|||
{ name = "numpy", version = "2.4.1", source = { registry = "https://pypi.org/simple/" }, marker = "python_full_version >= '3.11'" },
|
||||
{ name = "ollama" },
|
||||
{ name = "openai" },
|
||||
{ name = "openpyxl" },
|
||||
{ name = "opentelemetry-exporter-otlp-proto-http" },
|
||||
{ name = "opentelemetry-sdk" },
|
||||
{ name = "pre-commit" },
|
||||
|
|
@ -1631,6 +1645,10 @@ relevance = [
|
|||
reports = [
|
||||
{ name = "jinja2" },
|
||||
]
|
||||
spreadsheet = [
|
||||
{ name = "openpyxl" },
|
||||
{ name = "xlrd" },
|
||||
]
|
||||
strands = [
|
||||
{ name = "strands-agents" },
|
||||
]
|
||||
|
|
@ -1667,7 +1685,7 @@ requires-dist = [
|
|||
{ name = "fastembed", marker = "extra == 'relevance'", specifier = ">=0.4.0" },
|
||||
{ name = "gunicorn", marker = "sys_platform != 'win32' and extra == 'proxy-prod'", specifier = ">=21.0.0" },
|
||||
{ name = "headroom-ai", extras = ["proxy"], marker = "extra == 'proxy-prod'" },
|
||||
{ name = "headroom-ai", extras = ["proxy", "code", "ml", "memory", "relevance", "image", "reports", "otel", "evals", "voice", "html", "benchmark", "mcp"], marker = "extra == 'all'" },
|
||||
{ name = "headroom-ai", extras = ["proxy", "code", "ml", "memory", "relevance", "image", "reports", "otel", "evals", "voice", "html", "benchmark", "mcp", "spreadsheet"], marker = "extra == 'all'" },
|
||||
{ name = "headroom-ai", extras = ["voice"], marker = "extra == 'voice-train'" },
|
||||
{ name = "hnswlib", marker = "extra == 'dev'", specifier = ">=0.8.0" },
|
||||
{ name = "hnswlib", marker = "extra == 'memory'", specifier = ">=0.8.0" },
|
||||
|
|
@ -1699,6 +1717,8 @@ requires-dist = [
|
|||
{ name = "openai", marker = "extra == 'dev'", specifier = ">=1.0.0" },
|
||||
{ name = "openai", marker = "extra == 'evals'", specifier = ">=1.0.0" },
|
||||
{ name = "openai", marker = "extra == 'proxy'", specifier = ">=2.14.0" },
|
||||
{ name = "openpyxl", marker = "extra == 'dev'", specifier = ">=3.1.0" },
|
||||
{ name = "openpyxl", marker = "extra == 'spreadsheet'", specifier = ">=3.1.0" },
|
||||
{ name = "opentelemetry-api", specifier = ">=1.24.0" },
|
||||
{ name = "opentelemetry-exporter-otlp-proto-http", marker = "extra == 'dev'", specifier = ">=1.24.0" },
|
||||
{ name = "opentelemetry-exporter-otlp-proto-http", marker = "extra == 'otel'", specifier = ">=1.24.0" },
|
||||
|
|
@ -1734,15 +1754,17 @@ requires-dist = [
|
|||
{ name = "transformers", marker = "extra == 'ml'", specifier = ">=4.30.0,<6.0" },
|
||||
{ name = "transformers", marker = "extra == 'proxy'", specifier = ">=4.30.0,<6.0" },
|
||||
{ name = "transformers", marker = "extra == 'voice'", specifier = ">=4.30.0,<6.0" },
|
||||
{ name = "tree-sitter-language-pack", marker = "extra == 'code'", specifier = ">=0.10.0" },
|
||||
{ name = "tree-sitter", marker = "extra == 'code'", specifier = ">=0.25.2,<0.26" },
|
||||
{ name = "tree-sitter-language-pack", marker = "extra == 'code'", specifier = ">=0.10.0,<1.0" },
|
||||
{ name = "uvicorn", marker = "extra == 'dev'", specifier = ">=0.23.0,<1.0" },
|
||||
{ name = "uvicorn", marker = "extra == 'proxy'", specifier = ">=0.23.0,<1.0" },
|
||||
{ name = "watchdog", marker = "extra == 'proxy'", specifier = ">=4.0.0" },
|
||||
{ name = "websockets", marker = "extra == 'dev'", specifier = ">=13.0" },
|
||||
{ name = "websockets", marker = "extra == 'proxy'", specifier = ">=13.0" },
|
||||
{ name = "xlrd", marker = "extra == 'spreadsheet'", specifier = ">=2.0.1" },
|
||||
{ name = "zstandard", marker = "extra == 'proxy'", specifier = ">=0.20.0" },
|
||||
]
|
||||
provides-extras = ["proxy", "proxy-prod", "code", "ml", "memory", "memory-stack", "pytorch-mps", "relevance", "image", "reports", "otel", "anyllm", "langchain", "agno", "strands", "mcp", "voice", "voice-train", "evals", "bedrock", "html", "benchmark", "dev", "all"]
|
||||
provides-extras = ["proxy", "proxy-prod", "code", "ml", "memory", "memory-stack", "pytorch-mps", "relevance", "image", "reports", "spreadsheet", "otel", "anyllm", "langchain", "agno", "strands", "mcp", "voice", "voice-train", "evals", "bedrock", "html", "benchmark", "dev", "all"]
|
||||
|
||||
[[package]]
|
||||
name = "hf-xet"
|
||||
|
|
@ -3558,6 +3580,18 @@ wheels = [
|
|||
{ url = "https://files.pythonhosted.org/packages/e9/a5/1be1516390333ff9be3a9cb648c9f33df79d5096e5884b5df71a588af463/opencv_python-4.13.0.92-cp37-abi3-win_amd64.whl", hash = "sha256:423d934c9fafb91aad38edf26efb46da91ffbc05f3f59c4b0c72e699720706f5", size = 40212062, upload-time = "2026-02-05T07:02:12.724Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "openpyxl"
|
||||
version = "3.1.5"
|
||||
source = { registry = "https://pypi.org/simple/" }
|
||||
dependencies = [
|
||||
{ name = "et-xmlfile" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/3d/f9/88d94a75de065ea32619465d2f77b29a0469500e99012523b91cc4141cd1/openpyxl-3.1.5.tar.gz", hash = "sha256:cf0e3cf56142039133628b5acffe8ef0c12bc902d2aadd3e0fe5878dc08d1050", size = 186464, upload-time = "2024-06-28T14:03:44.161Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/c0/da/977ded879c29cbd04de313843e76868e6e13408a94ed6b987245dc7c8506/openpyxl-3.1.5-py2.py3-none-any.whl", hash = "sha256:5282c12b107bffeef825f4617dc029afaf41d0ea60823bbb665ef3079dc79de2", size = 250910, upload-time = "2024-06-28T14:03:41.161Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "openresponses-types"
|
||||
version = "2.3.0.post1"
|
||||
|
|
@ -6607,6 +6641,15 @@ wheels = [
|
|||
{ url = "https://files.pythonhosted.org/packages/1f/f6/a933bd70f98e9cf3e08167fc5cd7aaaca49147e48411c0bd5ae701bb2194/wrapt-1.17.3-py3-none-any.whl", hash = "sha256:7171ae35d2c33d326ac19dd8facb1e82e5fd04ef8c6c0e394d7af55a55051c22", size = 23591, upload-time = "2025-08-12T05:53:20.674Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "xlrd"
|
||||
version = "2.0.2"
|
||||
source = { registry = "https://pypi.org/simple/" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/07/5a/377161c2d3538d1990d7af382c79f3b2372e880b65de21b01b1a2b78691e/xlrd-2.0.2.tar.gz", hash = "sha256:08b5e25de58f21ce71dc7db3b3b8106c1fa776f3024c54e45b45b374e89234c9", size = 100167, upload-time = "2025-06-14T08:46:39.039Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/1a/62/c8d562e7766786ba6587d09c5a8ba9f718ed3fa8af7f4553e8f91c36f302/xlrd-2.0.2-py2.py3-none-any.whl", hash = "sha256:ea762c3d29f4cca48d82df517b6d89fbce4db3107f9d78713e48cd321d5c9aa9", size = 96555, upload-time = "2025-06-14T08:46:37.766Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "xxhash"
|
||||
version = "3.6.0"
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue