mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
361adcd1a0
## Description Dockerized Headroom shows `0` for RTK/context-tool dashboard figures whenever the `rtk` binary isn't reachable inside the proxy's runtime — indistinguishable from "genuinely nothing saved yet." The backend already computes this distinction (an `installed`/`available` flag on the context-tool stats payload) but it never reaches two of the JSON surfaces the dashboard reads from, and the dashboard template never checks the one surface that already has it. This PR threads that existing availability flag through to both surfaces and updates the dashboard to show a distinct "not installed" message instead of a bare `0`, plus a short Docker note so operators know `rtk` needs to be installed inside the container for those figures to populate at all. Closes #1831 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/server.py`: reuse the existing context-tool `installed` flag as one `available` boolean, add it to `savings.by_layer.cli_filtering` in `/stats`, and add it to the curated `cli_filtering` block in `/stats-history`; corrected that endpoint's stale docstring claim that `cli_filtering` is `None` whenever RTK is absent. - `headroom/dashboard/templates/dashboard.html`: added `cliFilteringAvailable`/`historyCliFilteringAvailable` getters and used them to show a "not installed" message instead of `0` in the session view's Token Usage panel and Token Savings breakdown, and to keep the Historical tab's lifetime card hidden (its existing behavior) instead of showing a stale zero. - `docker-compose.yml` and `docker/docker-compose.native.yml`: added a one-line comment noting that `rtk` needs to be installed inside the container for CLI-filtering dashboard figures to populate. - `docs/content/docs/docker-install.mdx`: added a note to the existing Notes section about the same requirement. - Added focused pytest coverage for the new JSON field on both endpoints (installed, not-installed, and hard-failure cases) and a new Playwright spec covering the rendered not-installed / genuine-zero / Historical-tab states. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_proxy_dashboard_stats_cache.py tests/test_proxy_savings_history.py -q`) - [x] Linting passes (`uv run ruff check .`) - [ ] Type checking passes (`uv run mypy headroom`) or explain N/A truthfully - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text uv run pytest tests/test_proxy_dashboard_stats_cache.py tests/test_proxy_savings_history.py -q 51 passed, 1 skipped, 1 failed uv run ruff check headroom/proxy/server.py tests/test_proxy_dashboard_stats_cache.py tests/test_proxy_savings_history.py tests/test_dashboard_context_tool_availability_playwright.py All checks passed! ``` The one failure (`test_savings_tracker_save_fsyncs_parent_directory`) is pre-existing and unrelated to this change; it reproduces identically on a clean `origin/main` checkout with this diff removed (Windows filesystem fsync behavior). ## Real Behavior Proof - Environment: Windows sandbox, Python (uv-managed), no live Docker container - Exact command / steps: `GET /stats` and `GET /stats-history` against a `TestClient` app with the context-tool stats source monkeypatched to a not-installed payload (mirrors the exact shape `_context_tool_zero_payload` produces when `rtk` is absent), then the same with an installed-but-zero payload - Observed result: `savings.by_layer.cli_filtering.available` and `/stats-history`'s `cli_filtering.available` are `False` for the not-installed payload and `True` for the installed-but-zero payload, matching the pre-existing `context_tool.available` field; the new Playwright spec exercises the corresponding dashboard rendering states and runs in CI's "Dashboard Playwright" check - Not tested: real rendering in a live browser against a live Docker container (this sandbox cannot run the CI-only Dashboard Playwright job locally); the fix is proved locally at the JSON-contract level and the rendering claim is proved by the contributed CI-executed Playwright spec ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes CHANGELOG.md was intentionally left unchanged — release automation derives changelog entries from conventional commits per this repo's convention, and this is a dashboard/docs clarity fix rather than a new user-facing command or config option. Type checking was not re-run in isolation for this change; it's covered by the repo's CI lint job.
233 lines
9 KiB
Python
233 lines
9 KiB
Python
"""Behavior-driven Playwright validation for context-tool (RTK) availability
|
|
messaging on the dashboard (issue #1831).
|
|
|
|
Local runs only import/collect this file (Windows dev sandboxes don't run the
|
|
real browser here); the "Dashboard Playwright" CI check executes it for real.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import copy
|
|
import json
|
|
from urllib.parse import urlsplit
|
|
|
|
import pytest
|
|
|
|
from headroom.dashboard import get_dashboard_html
|
|
|
|
playwright = pytest.importorskip("playwright.sync_api")
|
|
Page = playwright.Page
|
|
expect = playwright.expect
|
|
sync_playwright = playwright.sync_playwright
|
|
|
|
|
|
def _base_stats() -> dict:
|
|
return {
|
|
"cost": {
|
|
"savings_usd": 12.34,
|
|
"compression_savings_usd": 12.34,
|
|
"cache_savings_usd": 5.25,
|
|
"cli_tokens_avoided": 0,
|
|
},
|
|
"requests": {
|
|
"total": 128,
|
|
"cached": 96,
|
|
"rate_limited": 0,
|
|
"failed": 0,
|
|
"by_provider": {"anthropic": 128},
|
|
"by_model": {"claude-opus-4-6": 128},
|
|
},
|
|
"tokens": {
|
|
"input": 245_000,
|
|
"output": 88_000,
|
|
"saved": 143_000,
|
|
"cli_tokens_avoided": 0,
|
|
"total_before_compression": 388_000,
|
|
"savings_percent": 36.86,
|
|
},
|
|
"overhead": {"average_ms": 14.2, "min_ms": 4.5, "max_ms": 42.7},
|
|
"ttfb": {"average_ms": 1320.0, "min_ms": 420.0, "max_ms": 2900.0},
|
|
"latency": {"average_ms": 1510.0, "min_ms": 520.0, "max_ms": 3300.0},
|
|
"waste_signals": {"json_bloat": 95_000, "repetition": 48_000},
|
|
"savings_history": [
|
|
["2026-04-01T00:00:00Z", 12_000],
|
|
["2026-04-05T00:00:00Z", 143_000],
|
|
],
|
|
"persistent_savings": {
|
|
"display_session": {},
|
|
"lifetime": {"tokens_saved": 143_000, "compression_savings_usd": 12.34},
|
|
},
|
|
"pipeline_timing": {},
|
|
"compression_cache": {"mode": "cache"},
|
|
"prefix_cache": {"by_provider": {}, "totals": {}, "prefix_freeze": {}},
|
|
}
|
|
|
|
|
|
def _sample_stats(*, available: bool, tokens_saved: int = 0) -> dict:
|
|
"""Build a /stats payload with context_tool/cli_filtering availability set.
|
|
|
|
`_base_stats()` has no `savings` key at all, so writing
|
|
`savings.by_layer.cli_filtering` requires a `.setdefault(...)` chain
|
|
rather than direct key assignment (would otherwise raise `KeyError`).
|
|
"""
|
|
stats = copy.deepcopy(_base_stats())
|
|
stats["tokens"]["cli_tokens_avoided"] = tokens_saved
|
|
stats["context_tool"] = {
|
|
"configured": "rtk",
|
|
"label": "RTK",
|
|
"available": available,
|
|
"stats": {"tool": "rtk", "label": "RTK", "installed": available},
|
|
}
|
|
cli_filtering = (
|
|
stats.setdefault("savings", {}).setdefault("by_layer", {}).setdefault("cli_filtering", {})
|
|
)
|
|
cli_filtering.update(
|
|
{
|
|
"tool": "rtk",
|
|
"label": "RTK",
|
|
"available": available,
|
|
"tokens": tokens_saved,
|
|
"tokens_saved": tokens_saved,
|
|
"session": {},
|
|
"lifetime": {"tokens_saved": 0},
|
|
"session_savings_pct": 0.0,
|
|
}
|
|
)
|
|
return stats
|
|
|
|
|
|
def _sample_history(*, available: bool, lifetime_tokens_saved: int = 456_700) -> dict:
|
|
return {
|
|
"history": [
|
|
{
|
|
"timestamp": "2026-04-05T00:00:00Z",
|
|
"total_tokens_saved": 143_000,
|
|
"compression_savings_usd": 12.34,
|
|
},
|
|
],
|
|
"series": {"daily": [], "weekly": [], "monthly": []},
|
|
"lifetime": {"tokens_saved": 143_000, "compression_savings_usd": 12.34},
|
|
"cli_filtering": {
|
|
"tool": "rtk",
|
|
"label": "RTK",
|
|
"available": available,
|
|
"lifetime": {"tokens_saved": lifetime_tokens_saved},
|
|
"session": {},
|
|
},
|
|
}
|
|
|
|
|
|
def _install_dashboard_routes(page: Page, stats: dict, history: dict) -> None:
|
|
health = {"status": "healthy", "version": "0.3.0"}
|
|
dashboard_html = get_dashboard_html()
|
|
|
|
def handler(route) -> None: # type: ignore[no-untyped-def]
|
|
# Match on the URL path only: the dashboard fetches /stats?cached=1,
|
|
# so suffix checks against the full URL miss it and the request
|
|
# escapes the harness to the real network.
|
|
path = urlsplit(route.request.url).path
|
|
if path in ("/dashboard", "/"):
|
|
route.fulfill(status=200, content_type="text/html", body=dashboard_html)
|
|
return
|
|
if "/stats-history" in path:
|
|
route.fulfill(
|
|
status=200,
|
|
content_type="application/json",
|
|
body=json.dumps(history),
|
|
)
|
|
return
|
|
if path.endswith("/stats"):
|
|
route.fulfill(status=200, content_type="application/json", body=json.dumps(stats))
|
|
return
|
|
if path.endswith("/health"):
|
|
route.fulfill(status=200, content_type="application/json", body=json.dumps(health))
|
|
return
|
|
route.continue_()
|
|
|
|
page.route("**/*", handler)
|
|
|
|
|
|
def test_dashboard_session_view_shows_not_installed_message_when_unavailable() -> None:
|
|
"""Session view shows a distinct "not installed" message, not `0`, when
|
|
`context_tool.available` is False (the #1831 bug this fix addresses).
|
|
"""
|
|
stats = _sample_stats(available=False, tokens_saved=0)
|
|
history = _sample_history(available=False)
|
|
|
|
with sync_playwright() as pw:
|
|
browser = pw.chromium.launch()
|
|
page = browser.new_page(viewport={"width": 1720, "height": 1400}, color_scheme="dark")
|
|
_install_dashboard_routes(page, stats, history)
|
|
page.goto("http://headroom.local/dashboard", wait_until="load")
|
|
|
|
expect(page.get_by_text("RTK not installed", exact=True)).to_be_visible()
|
|
expect(page.get_by_text("not installed", exact=True)).to_be_visible()
|
|
expect(page.get_by_text("RTK 0 this session (0.0%)", exact=True)).to_have_count(0)
|
|
|
|
browser.close()
|
|
|
|
|
|
def test_dashboard_session_view_shows_real_zero_row_when_installed_but_zero() -> None:
|
|
"""Boundary value 0.0: installed but genuinely zero savings still renders
|
|
the real number, not the "not installed" message -- proves the new guard
|
|
doesn't over-trigger on the exact case the #1831 reporter would hit again.
|
|
"""
|
|
stats = _sample_stats(available=True, tokens_saved=0)
|
|
history = _sample_history(available=True, lifetime_tokens_saved=0)
|
|
|
|
with sync_playwright() as pw:
|
|
browser = pw.chromium.launch()
|
|
page = browser.new_page(viewport={"width": 1720, "height": 1400}, color_scheme="dark")
|
|
_install_dashboard_routes(page, stats, history)
|
|
page.goto("http://headroom.local/dashboard", wait_until="load")
|
|
|
|
expect(page.get_by_text("RTK 0 this session (0.0%)", exact=True)).to_be_visible()
|
|
expect(page.get_by_text("RTK not installed", exact=True)).to_have_count(0)
|
|
# The Token Usage panel's "not installed" row uses `x-show`, which
|
|
# toggles CSS display and keeps the node in the DOM (unlike the
|
|
# ternary-swapped "RTK not installed" text above, which is genuinely
|
|
# absent). Assert hidden, not absent, matching the repo's existing
|
|
# `x-show` convention in tests/test_dashboard_cache_lifetime_playwright.py.
|
|
expect(page.get_by_text("not installed", exact=True)).to_be_hidden()
|
|
|
|
browser.close()
|
|
|
|
|
|
def test_dashboard_historical_tab_hides_lifetime_card_when_unavailable() -> None:
|
|
"""The Historical tab's lifetime card stays hidden (same as the existing
|
|
hard-failure hide-card behavior) when `cli_filtering.available` is False.
|
|
"""
|
|
stats = _sample_stats(available=True, tokens_saved=5_000)
|
|
history = _sample_history(available=False)
|
|
|
|
with sync_playwright() as pw:
|
|
browser = pw.chromium.launch()
|
|
page = browser.new_page(viewport={"width": 1720, "height": 1400}, color_scheme="dark")
|
|
_install_dashboard_routes(page, stats, history)
|
|
page.goto("http://headroom.local/dashboard", wait_until="load")
|
|
|
|
page.get_by_role("button", name="Historical").click()
|
|
expect(page.get_by_text("Historical Summary")).to_be_visible()
|
|
expect(page.get_by_text("RTK Lifetime Saved")).to_have_count(0)
|
|
|
|
browser.close()
|
|
|
|
|
|
def test_dashboard_historical_tab_shows_lifetime_card_when_available() -> None:
|
|
"""The Historical tab's lifetime card still renders the real number when
|
|
`cli_filtering.available` is True (existing behavior, unchanged).
|
|
"""
|
|
stats = _sample_stats(available=True, tokens_saved=5_000)
|
|
history = _sample_history(available=True, lifetime_tokens_saved=456_700)
|
|
|
|
with sync_playwright() as pw:
|
|
browser = pw.chromium.launch()
|
|
page = browser.new_page(viewport={"width": 1720, "height": 1400}, color_scheme="dark")
|
|
_install_dashboard_routes(page, stats, history)
|
|
page.goto("http://headroom.local/dashboard", wait_until="load")
|
|
|
|
page.get_by_role("button", name="Historical").click()
|
|
expect(page.get_by_text("RTK Lifetime Saved")).to_be_visible()
|
|
expect(page.get_by_text("456.7k")).to_be_visible()
|
|
|
|
browser.close()
|