10 commits

Author	SHA1	Message	Date
Rod Boev	88f935a1eb	fix(dashboard): deduplicate repeated savings metrics (#1804) ... ## Description The session dashboard repeats the same savings and performance numbers in adjacent places. `proxy_compression_saved` appears in several captions and detail rows, and average overhead and TTFB appear both in the hero area and again in Performance without adding new context. This narrows the non-hero dashboard presentation so repeated session metrics have one visible home plus decomposition where it adds information. It leaves `/stats`, savings math, cache attribution, and the hero proxy savings card unchanged. Refs #960 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Removed redundant non-hero session-view captions that restated proxy-compression token counts without adding a new dimension. - Kept canonical homes for proxy compression and token usage details. - Preserved Performance range context while avoiding adjacent restatement of hero averages. - Added a static dashboard regression for repeated session metrics. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_proxy_dashboard_stats_cache.py -q`) - [x] Linting passes (`uv run ruff check tests/test_proxy_dashboard_stats_cache.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_proxy_dashboard_stats_cache.py -q 12 passed, 1 skipped, 1 warning in 19.24s $ uv run ruff check tests/test_proxy_dashboard_stats_cache.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows, Python environment from `uv sync --extra dev`, browserless dashboard HTML inspection. - Exact command / steps: load `get_dashboard_html()` in the focused dashboard stats test and assert removed duplicate captions stay removed while canonical metric owners remain present. - Observed result: session-view repeated savings and performance labels no longer duplicate the same numbers without context. - Not tested: full browser screenshot and history-view de-duplication. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes No `CHANGELOG.md` edit: this repo generates changelog entries from conventional commits. This intentionally avoids the hero proxy savings card already covered by #927 and #1649, and it does not fold provider cache discount into Headroom-value savings.	2026-07-05 16:00:25 -07:00
gglucass	2fe19c39e4	feat(stats): surface Codex WS compression counters in /stats summary (#1680) ... ## Description Codex rides a long-lived WebSocket `/responses` connection. WS units are compressed and counted into the `codex_ws_*` metrics immediately, but turn-level records — the ones that feed `tokens_saved_total` and therefore the `/stats` `summary` block — only land when a `response.completed` frame carries usage tokens. A user watching `summary.api_requests` / `summary.compression` during an active Codex WS session sees frozen counters and concludes Headroom isn't working, even though the `codex_ws` stats section is advancing. (Reported by a Headroom Desktop user who cross-checked `/stats` against a healthy proxy and confirmed-correct Codex routing.) This PR surfaces the live per-unit counters inside `summary` so WS-only sessions are visible at a glance: ```json "codex_ws": {"units_total": 12, "units_modified": 9, "tokens_saved": 4321} ``` The block is deliberately **not** summed into `compression.total_tokens_removed`: turns that did record already contributed the same savings to `tokens_saved_total`, and the recorded-vs-unrecorded split is not tracked globally, so folding the unit sums into the totals would double-count. Additive visibility, not a second ledger. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/cost.py`: `build_session_summary` emits a `summary.codex_ws` block (`units_total`, `units_modified`, `tokens_saved`) sourced from the live per-unit metrics; only present when `codex_ws_units_total > 0`, so non-Codex sessions keep the existing summary shape. `getattr` defaults keep older/partial metrics objects working. - `tests/test_proxy_dashboard_stats_cache.py`: new `test_session_summary_surfaces_codex_ws_counters`; extended `test_session_summary_uses_generic_cli_filtering_keys` to assert the block is absent when counters are missing. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --extra dev pytest tests/test_proxy_dashboard_stats_cache.py =================== 11 passed, 1 skipped, 1 warning in 3.47s =================== $ uv run --extra dev pytest tests/test_compression_observability.py tests/test_proxy_healthchecks.py tests/test_pr208_changes.py ======================== 72 passed, 1 warning in 32.18s ======================== $ uv run --extra dev mypy headroom/proxy/cost.py Success: no issues found in 1 source file $ ruff check headroom/proxy/cost.py tests/test_proxy_dashboard_stats_cache.py All checks passed! ``` ## Real Behavior Proof - Environment: macOS 15 (Darwin 24.6.0), Python 3.10 venv via `uv`, branch `fix/stats-summary-codex-ws` @ upstream main - Exact command / steps: called `build_session_summary` with metrics carrying `codex_ws_units_total=12`, `codex_ws_units_modified_total=9`, `codex_ws_unit_tokens_saved_sum=4321` (same shape `create_app` passes at `/stats`), printed `summary["codex_ws"]` - Observed result: `{"units_total": 12, "units_modified": 9, "tokens_saved": 4321}`; with counters absent, `"codex_ws" not in summary` - Not tested: end-to-end `/stats` against a live Codex WS session on this build (the installed desktop bundle runs 0.28.0, which predates this branch); unit path is identical since `/stats` calls `build_session_summary` with the live metrics object ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - Documentation / CHANGELOG unchecked: `/stats` response fields aren't documented per-key, and CHANGELOG did not appear to track additive stats fields — happy to add either if maintainers want it. - Follow-up candidate (out of scope here): fold WS savings into the compression *totals* correctly by tracking a `codex_ws_tokens_saved_recorded_total` at turn-record time, so the unrecorded remainder could be added without double-counting. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-07-02 14:25:24 -07:00
Tejas Chopra	93627471b7	fix(perf): surface RTK/CLI context-tool savings in perf and the session card (#1433) ... ## Description `headroom perf` read only `proxy.log` compression records, so RTK's savings — which live in RTK's own lifetime counter and never land in `proxy.log` — were **invisible**: perf reported "token savings" while silently dropping the entire CLI-filtering layer. The dashboard **Session** card likewise showed only the session-delta (≈0 right after a proxy restart), with no scope label and no lifetime figure. This surfaces RTK lifetime savings in `headroom perf` (text + JSON) and clarifies the dashboard Session card. It complements #1324 (which added RTK to the Historical tab) by covering the two surfaces #1324 didn't: `perf` and the live Session card. Closes # N/A — complements #1324; no standalone issue. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/perf/analyzer.py`: `format_report` and `build_perf_summary` now attach RTK/CLI context-tool **lifetime** savings, sourced best-effort from `_get_context_tool_stats().lifetime` (the same source `/stats` and #1324 use). Lifetime — not session — is the right scope for a one-shot CLI, since the proxy-session baseline `/stats` subtracts is meaningless out of process. Omitted entirely when no tool is installed or its stats can't be read, so the report degrades to proxy-only rather than erroring. - `headroom/dashboard/templates/dashboard.html`: the Session card now labels the RTK number **"this session"**, uses the real `session_savings_pct` (via a new `cliFilteringSessionPctDisplay` getter) instead of an ad-hoc share, and shows **lifetime** alongside it (new `cliFilteringLifetime` getter + row, hidden when 0). - `tests/test_perf_cli_filtering.py` (new): perf surfaces RTK in text + JSON; omits cleanly when the tool is absent. - `tests/test_rtk_session_savings.py` (new): exercises the real `_get_context_tool_stats()` plumbing to pin that session RTK savings are the **delta from the startup baseline**, and session `savings_pct` is derived from that delta — not RTK's lifetime-diluted average. - `tests/test_proxy_dashboard_stats_cache.py`: updated the Session-card label assertion and added one for the new lifetime row. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check headroom/perf/analyzer.py tests/test_perf_cli_filtering.py tests/test_rtk_session_savings.py tests/test_proxy_dashboard_stats_cache.py All checks passed! $ mypy headroom/perf/analyzer.py mypy: No issues found $ python -m pytest tests/test_perf_cli_filtering.py tests/test_rtk_session_savings.py tests/test_proxy_dashboard_stats_cache.py tests/test_owned_asset_encoding.py -q 17 passed, 1 skipped in 15.66s ``` ## Real Behavior Proof - Environment: macOS (darwin), Python 3.12, branch `fix/rtk-savings-perf-dashboard`, RTK v0.28.2. - Exact command / steps: `headroom perf` and `headroom perf --format json`. - Observed result: the text report now includes a section `RTK CLI Filtering (lifetime, all-time) — Tokens saved: 26,867,610 (68.8%), Commands: 8,023`, and the JSON output carries `"cli_filtering": {"tool":"rtk","label":"RTK","tokens_saved":26867610,"commands":8023,"savings_pct":68.8}`. Before this change, both omitted RTK entirely (perf's "Total saved" was proxy-compression only). The dashboard template renders the new "this session" / "lifetime" RTK rows (verified via `get_dashboard_html()` + substring test). - Not tested: live dashboard browser click-through (template loads and the new strings are asserted by the substring test); CSV output of `perf` (per-model table only, by design). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG: left to Release Please (the conventional `fix(perf):` commit generates the entry on merge), matching how the existing "Bug Fixes" entries are produced. - Follow-up: #1403 (`fix/rtk-savings-scope-regression`) bundles unrelated kompress must-keep work (overlaps #1400/#1419) and only documents the scope `%` invariant in the abstract. The real, code-exercising session-delta regression now lives here (`test_rtk_session_savings.py`), so #1403 can be split — route the kompress bits to #1400/#1419 and drop the rest.	2026-06-25 21:13:36 -07:00
Shlok Tiwari	0d89c674cd	feat: measure and surface token throughput (tokens/sec) through the proxy (#983) ... ## Description This PR implements measuring and surfacing token throughput (tokens/second) through the proxy in the `headroom perf` CLI/analyzer and the dashboard UI. It tracks multiple throughput metrics—Input (wall-clock/active), Compression, Forward, and Generation throughput—supporting both rolling percentiles (p50/p95) and current (last 5 minutes) metrics. Closes #959 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Proxy Instrumentation (`headroom/proxy/outcome.py`)**: Added `total_ms`, `tok_out`, and `ttfb_ms` to the structured `PERF` logging payload. - **Log Parsing & Computations (`headroom/perf/analyzer.py`)**: Updated log parsing to read `STAGE_TIMINGS` and correlation fields from `PERF`, computing active/wall-clock throughputs for input, compression, forward, and generation stages. - **API Exposing (`headroom/proxy/server.py`)**: Exposes calculated rolling throughput percentiles and last-5-minute averages under the `throughput` field in `/stats`. - **Dashboard UI Layout (`headroom/dashboard/templates/dashboard.html`)**: Refactored the dashboard grid layout from 3 columns to 4 columns to house the new throughput hero card showing real-time token performance. - **Verification Tests (`tests/test_cli_perf_format.py`)**: Added test coverage specifically targeting token throughput log parser extraction, stage correlation, math correctness, and edge-case handling (empty fields, division by zero). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $env:PYTHONPATH="c:\Users\hp\Desktop\Headroom_oss"; .venv\Scripts\pytest tests/test_cli_perf_format.py ============================= test session starts ============================= platform win32 -- Python 3.11.15, pytest-9.1.0, pluggy-1.6.0 -- C:\Users\hp\Desktop\Headroom_oss\.venv\Scripts\python.exe cachedir: .pytest_cache rootdir: C:\Users\hp\Desktop\Headroom_oss configfile: pyproject.toml plugins: anyio-4.13.0 collecting ... collected 14 items tests/test_cli_perf_format.py::test_build_perf_summary_totals_and_pct PASSED [ 7%] tests/test_cli_perf_format.py::test_build_perf_summary_by_model_and_transform PASSED [ 14%] tests/test_cli_perf_format.py::test_build_perf_summary_empty_report_no_zero_division PASSED [ 21%] tests/test_cli_perf_format.py::test_perf_records_as_dicts_roundtrips_fields PASSED [ 28%] tests/test_cli_perf_format.py::test_perf_json_format PASSED [ 35%] tests/test_cli_perf_format.py::test_perf_json_raw_is_array PASSED [ 42%] tests/test_cli_perf_format.py::test_perf_json_raw_preserves_client_field PASSED [ 50%] tests/test_cli_perf_format.py::test_parse_perf_line_preserves_client_field PASSED [ 57%] tests/test_cli_perf_format.py::test_perf_csv_by_model PASSED [ 64%] tests/test_cli_perf_format.py::test_perf_csv_raw_per_record PASSED [ 71%] tests/test_cli_perf_format.py::test_perf_text_default_unchanged PASSED [ 78%] tests/test_cli_perf_format.py::test_perf_rejects_unknown_format PASSED [ 85%] tests/test_cli_perf_format.py::test_parse_perf_line_preserves_blank_client_field PASSED [ 92%] tests/test_cli_perf_format.py::test_throughput_parsing_and_calculations PASSED [100%] ============================== warnings summary =============================== .venv\Lib\site-packages\_pytest\config\__init__.py:1464 C:\Users\hp\Desktop\Headroom_oss\.venv\Lib\site-packages\_pytest\config\__init__.py:1464: PytestConfigWarning: Unknown config option: asyncio_mode self._warn_or_fail_if_strict(f"Unknown config option: {key}\n") .venv\Lib\site-packages\opentelemetry\util\_importlib_metadata.py:32 C:\Users\hp\Desktop\Headroom_oss\.venv\Lib\site-packages\opentelemetry\util\_importlib_metadata.py:32: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select. return EntryPoints(ep for group_eps in eps.values() for ep in group_eps) -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ======================= 14 passed, 2 warnings in 4.77s ======================== ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.11.15 - Exact command / steps: Run the pytest suite against the newly created token throughput parsing routines: `$env:PYTHONPATH="c:\Users\hp\Desktop\Headroom_oss"; .venv\Scripts\pytest tests/test_cli_perf_format.py` - Observed result: The suite executes 14 tests successfully, including the newly added `test_throughput_parsing_and_calculations` verification test verifying mathematical precision and fallback logic. - Not tested: None (all metrics are fully covered by unit tests in `test_cli_perf_format.py`) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - Backwards compatibility: Older log outputs lacking `tok_out` or `ttfb_ms` parse cleanly and fallback defaults prevent parser crashes. --------- Co-authored-by: Antigravity Agent <agent@antigravity.local>	2026-06-17 09:42:38 -05:00
skblue	b70fccbe17	fix(proxy): read RTK gain stats globally by default (#957) ... ## Description Closes #900. The proxy now reads RTK lifetime savings with global scope by default. This matches shared daemon deployments where the proxy process cwd is often `$HOME` or a service directory, while RTK savings are accumulated across the operator's projects. `HEADROOM_RTK_GAIN_SCOPE=project` keeps the old `rtk gain --project` behavior for operators who explicitly want the proxy process working directory as the scope. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Default RTK stats subprocess command to `rtk gain --format json` - Add `HEADROOM_RTK_GAIN_SCOPE=project` to opt into `rtk gain --project --format json` - Keep fallback/synthetic-zero payload `scope` aligned with the queried scope - Deduplicate context-tool zero payload construction - Document the new RTK gain scope environment variable ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text uv run --extra dev python -m pytest tests/test_proxy_dashboard_stats_cache.py tests/test_subscription_tracker_rtk_wired.py -q ============================= test session starts ============================== platform darwin -- Python 3.14.5, pytest-9.0.3, pluggy-1.6.0 rootdir: /Users/joshuasiu/vibe/temp/headroom configfile: pyproject.toml plugins: anyio-4.12.1, langsmith-0.8.0, asyncio-1.3.0, cov-7.0.0 asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 26 items tests/test_proxy_dashboard_stats_cache.py .......... [ 38%] tests/test_subscription_tracker_rtk_wired.py ................ [100%] ============================== 26 passed in 0.40s ============================== uv run --extra dev python -m pytest tests/test_proxy_stats_recent_requests.py tests/test_proxy_healthchecks.py -q ============================= test session starts ============================== platform darwin -- Python 3.14.5, pytest-9.0.3, pluggy-1.6.0 rootdir: /Users/joshuasiu/vibe/temp/headroom configfile: pyproject.toml plugins: anyio-4.12.1, langsmith-0.8.0, asyncio-1.3.0, cov-7.0.0 asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function collected 15 items tests/test_proxy_stats_recent_requests.py ... [ 20%] tests/test_proxy_healthchecks.py ............ [100%] ============================= 15 passed in 10.41s ============================== uv run --extra dev ruff check headroom/proxy/helpers.py tests/test_proxy_dashboard_stats_cache.py tests/test_subscription_tracker_rtk_wired.py All checks passed! uv run --extra dev ruff format --check headroom/proxy/helpers.py tests/test_proxy_dashboard_stats_cache.py tests/test_subscription_tracker_rtk_wired.py 3 files already formatted uv run --extra dev mypy headroom headroom/proxy/server.py:1141: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] headroom/proxy/server.py:1211: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] headroom/proxy/server.py:1215: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] Success: no issues found in 358 source files ``` ## Real Behavior Proof - Environment: macOS, Python 3.14.5 via `uv run --extra dev` - Exact command / steps: mocked RTK subprocess argv in unit tests - Observed result: default command is `rtk gain --format json`; project scope command is `rtk gain --project --format json`; invalid scope logs `event=rtk_gain_scope_invalid` and falls back to global - Not tested: full repository pytest suite ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes The unchecked comment and changelog items are not applicable for this scoped proxy stats fix.	2026-06-13 21:21:38 -07:00
Tejas Chopra	e9cae0131b	fix: expose compression latency bottlenecks ... Add Codex WS unit-level timing and bounded parallel compression, clarify context-tool session savings, and avoid costly diff/log fallbacks to Kompress.	2026-05-12 13:34:08 -07:00
Gili Tzabari	4b061792b2	feat: add lean-ctx context tool support	2026-05-11 17:54:17 -04:00
Tejas Chopra	eaf5980b4a	fix: stabilize codex compression, stats, and proxy lifecycle	2026-05-09 13:47:53 -07:00
Tejas Chopra	6eec38767b	Combine rtk and compression savings in dashboard	2026-05-08 14:46:54 -07:00
Kayzo	2b1ab269ca	fix: cache dashboard stats snapshots	2026-04-22 09:30:19 +00:00