feat: add dashboard agent usage stats (#814)
## Description
Add a clear dashboard view for per-agent token usage so end users can
see Cursor, Claude, Codex, and other detected clients with before/after
token counts, tokens saved, and savings percentages. The stats API now
exposes a stable `agent_usage` object that the dashboard renders near
the top of the session view.
Fixes #
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
### New Files
**Tests:**
- `tests/test_dashboard_agent_usage.py` — Covers agent classification,
exact per-request aggregation, and aggregate fallback behavior.
### Modified Files
- `headroom/proxy/server.py` — Adds per-agent usage aggregation to
`/stats` with before tokens, after tokens, output tokens, saved tokens,
savings percentage, source, providers, and models.
- `headroom/dashboard/templates/dashboard.html` — Adds a prominent Agent
Usage panel with totals, coverage status, per-agent token-flow bars,
request counts, before/after tokens, saved tokens, and share of savings.
## Testing
- [x] Unit tests pass: `.venv312/bin/pytest
tests/test_dashboard_agent_usage.py`
- [x] Linting passes: `.venv312/bin/ruff check headroom/proxy/server.py
tests/test_dashboard_agent_usage.py`
- [x] Diff whitespace check passes: `git diff --check
origin/main...HEAD`
- [x] Dashboard smoke render: local proxy on `127.0.0.1:8790`, captured
Chrome headless screenshot of `/dashboard`
- [x] New tests added for new functionality
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing relevant unit tests pass locally with my changes
- [ ] I have made corresponding changes to the documentation
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
The agent usage panel uses exact request-log data when available. If
detailed request logs are empty, it falls back to aggregate
provider/model request counts and labels the coverage as aggregate
fallback so users are not misled.
2026-06-12 22:12:22 +03:00
|
|
|
from __future__ import annotations
|
|
|
|
|
|
|
|
|
|
import pytest
|
|
|
|
|
from fastapi.testclient import TestClient
|
|
|
|
|
|
|
|
|
|
from headroom.proxy.models import ProxyConfig
|
|
|
|
|
from headroom.proxy.server import create_app
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
class FakeRequestLogger:
|
|
|
|
|
def __init__(self) -> None:
|
|
|
|
|
self._logs: list[dict[str, object]] = []
|
|
|
|
|
|
|
|
|
|
@property
|
|
|
|
|
def logs(self) -> list[dict[str, object]]:
|
|
|
|
|
return self._logs
|
|
|
|
|
|
|
|
|
|
@logs.setter
|
|
|
|
|
def logs(self, value: list[dict[str, object]]) -> None:
|
|
|
|
|
self._logs = value
|
|
|
|
|
|
|
|
|
|
def get_recent(self, limit: int) -> list[dict[str, object]]:
|
|
|
|
|
return self._logs[-limit:]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
class FakeLogEntry(dict[str, object]):
|
|
|
|
|
def __getattr__(self, name: str) -> object:
|
|
|
|
|
return self.get(name)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_stats_refreshes_recent_requests_when_cached() -> None:
|
|
|
|
|
app = create_app(
|
|
|
|
|
ProxyConfig(
|
|
|
|
|
optimize=False,
|
|
|
|
|
cache_enabled=False,
|
|
|
|
|
rate_limit_enabled=False,
|
|
|
|
|
cost_tracking_enabled=False,
|
|
|
|
|
log_requests=False,
|
|
|
|
|
ccr_inject_tool=False,
|
|
|
|
|
ccr_handle_responses=False,
|
|
|
|
|
ccr_context_tracking=False,
|
|
|
|
|
http2=False,
|
|
|
|
|
)
|
|
|
|
|
)
|
|
|
|
|
logger = FakeRequestLogger()
|
|
|
|
|
app.state.proxy.logger = logger
|
|
|
|
|
|
|
|
|
|
first_log = FakeLogEntry(
|
|
|
|
|
{
|
|
|
|
|
"timestamp": "2026-06-11T10:00:00Z",
|
|
|
|
|
"provider": "openai",
|
|
|
|
|
"model": "gpt-4.1",
|
|
|
|
|
"input_tokens_original": 100,
|
|
|
|
|
"input_tokens_optimized": 60,
|
|
|
|
|
"tokens_saved": 40,
|
|
|
|
|
"savings_percent": 40.0,
|
|
|
|
|
}
|
|
|
|
|
)
|
|
|
|
|
second_log = FakeLogEntry(
|
|
|
|
|
{
|
|
|
|
|
"timestamp": "2026-06-11T10:01:00Z",
|
|
|
|
|
"provider": "anthropic",
|
|
|
|
|
"model": "claude-sonnet",
|
|
|
|
|
"input_tokens_original": 200,
|
|
|
|
|
"input_tokens_optimized": 120,
|
|
|
|
|
"tokens_saved": 80,
|
|
|
|
|
"savings_percent": 40.0,
|
|
|
|
|
}
|
|
|
|
|
)
|
|
|
|
|
|
fix(proxy): scope CORS to loopback + gate operator/content endpoints (#1226)
## Description
Locks down the proxy's browser- and network-facing attack surface, which
matters most under a `--host 0.0.0.0` bind (the Docker default). The
wildcard CORS policy (`allow_origins=["*"]` + `allow_credentials=True`)
let any web page the user had open read the proxy's content endpoints —
`/v1/retrieve` returns raw, uncompressed tool outputs (source, secrets)
— via a cross-origin fetch to `127.0.0.1` (CWE-346). Several operator
endpoints additionally leaked sensitive data or allowed unauthenticated
state mutation to any network-reachable client.
This PR scopes CORS to loopback origins and extends the project's
existing `require_loopback` trust boundary (already used for `/admin/*`
and `/debug/*`) to the remaining exposed endpoints.
Closes #863.
Supersedes #864 and #758 — see "Additional Notes".
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- **CORS**: replaced `allow_origins=["*"]` + `allow_credentials=True`
with a port-agnostic loopback origin regex
(`https?://(localhost|127\.0\.0\.1|\[::1\])(:\d+)?`),
`allow_credentials=False`, and methods/headers narrowed to `GET/POST` +
`Content-Type/Authorization`. `HEADROOM_CORS_ORIGINS` (comma-separated)
pins an explicit allowlist for Docker/remote dashboards; `*` opts back
into the old wildcard.
- **`/transformations/feed`** and **`/cache/clear`** gated behind
`require_loopback` → 404 for non-loopback callers. The feed returns full
prompt/completion bodies when `log_full_messages` is on; `/cache/clear`
is unauthenticated state mutation (cache-eviction DoS / cost
amplification).
- **`/health`**: the `config` block (upstream API URLs, savings profile)
is now served only to loopback callers; network callers get the
`/readyz`-shape body (status/checks). `/livez` and `/readyz` remain
unauthenticated probes for orchestration.
- **`/stats`**: `recent_requests` / `request_logs` (per-request ids,
providers, models, errors) and `config` are served only to loopback
callers; aggregate counters stay public for remote monitoring.
- Added `_request_is_loopback()` helper mirroring `require_loopback`'s
two-gate check (loopback peer IP + loopback `Host` header, the
DNS-rebinding defence) but degrading the payload instead of returning
404, so monitors keep the non-sensitive fields.
- Tests: new `tests/test_proxy_cors.py` and
`tests/test_proxy_loopback_gating.py`; updated 4 existing tests that
assert the now-loopback-only data to use loopback clients.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ ruff check headroom/proxy/server.py tests/test_proxy_cors.py \
tests/test_proxy_loopback_gating.py tests/test_proxy_healthchecks.py \
tests/test_proxy_stats_recent_requests.py tests/test_proxy/test_transformations_feed.py
All checks passed!
$ mypy headroom --ignore-missing-imports
Success: no issues found in 380 source files
$ pytest tests/test_proxy_cors.py tests/test_proxy_loopback_gating.py \
tests/test_proxy/test_transformations_feed.py tests/test_proxy_healthchecks.py \
tests/test_proxy_stats_recent_requests.py tests/test_proxy_dashboard_stats_cache.py \
tests/test_proxy_compression_executor.py tests/test_header_isolation.py -q
======================== 82 passed, 1 skipped in 17.75s ========================
```
## Real Behavior Proof
- Environment: macOS (darwin 25.4.0), Python 3.12 venv; FastAPI
`TestClient` driving the real `create_app()` ASGI app
- Exact command / steps: issued requests as a non-loopback caller
(`client.host=testclient`) vs a loopback caller
(`base_url=http://127.0.0.1`, `client=("127.0.0.1", 9999)`), plus CORS
preflights with varying `Origin` headers
- Observed result: CORS — `http://evil.com` → no
`access-control-allow-origin`; `http://localhost:8787` and
`http://localhost:9000` → echoed (loopback allowed on any port);
`access-control-allow-credentials` → absent. `/cache/clear` and
`/transformations/feed` → 404 (network) / 200 (loopback). `/health`
`config` block present for loopback only. `/stats` `recent_requests`
present for loopback only, while the aggregate `tokens` block stays
present for network callers.
- Not tested: a live `headroom proxy` process bound on `0.0.0.0` reached
from a second host (simulated via ASGI peer/Host instead); end-to-end
browser DNS-rebinding (covered by the `Host`-header gate and its unit
test)
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
N/A — proxy/middleware change; behavior is captured under "Real Behavior
Proof".
## Additional Notes
**Supersedes two stale PRs that target the same issue but have drifted
from `main`:**
- **#864** (`fix(proxy): scope CORS to localhost`, @gabiudrescu) —
correct instinct and the source of the tighter `GET/POST` +
`Content-Type/Authorization` scoping kept here, but it derived the
allowlist from the `HEADROOM_PORT` env var (wrong when `--port` is
passed as a CLI flag), carried ~40 lines of unrelated punctuation churn,
and is ~125 commits behind `main`. The port-agnostic regex used here
resolves the reviewer's port concern.
- **#758** (`security: adversarial review`, @neogenix) — bundled these
same application-layer fixes with a large CI/CD + Docker supply-chain
pass. It is a ~160-commit-behind draft whose `server.py` no longer
merges cleanly (`main` independently adopted the same `require_loopback`
pattern). The application-layer fixes are rebased onto current `main`
here; the CI/Docker/supply-chain hardening from #758 is still valuable
and would be welcome as a separate, rebased PR.
Thanks to @gabiudrescu and @neogenix for the original analysis (#863).
**Deliberate scope / follow-ups (not in this PR):**
- `/stats` aggregate counters and the basic `/health` body remain
readable on a `0.0.0.0` bind by design, so remote monitoring keeps
working. Full lock-down is a one-line `Depends(require_loopback)` each
if preferred.
- The `/v1/retrieve*` family stays network-reachable; it can't be
loopback-gated without breaking legitimate remote/containerized agents
and needs auth instead — tracked separately.
- `ruff check .` is scoped to changed paths above because the dashboard
HTML template trips ruff's `invalid-syntax` (a known repo
false-positive); `mypy` is run over the full `headroom` package.
2026-06-21 00:50:55 -07:00
|
|
|
# Loopback client/Host: recent_requests is served only to loopback callers.
|
|
|
|
|
with TestClient(app, base_url="http://127.0.0.1", client=("127.0.0.1", 12345)) as client:
|
feat: add dashboard agent usage stats (#814)
## Description
Add a clear dashboard view for per-agent token usage so end users can
see Cursor, Claude, Codex, and other detected clients with before/after
token counts, tokens saved, and savings percentages. The stats API now
exposes a stable `agent_usage` object that the dashboard renders near
the top of the session view.
Fixes #
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
### New Files
**Tests:**
- `tests/test_dashboard_agent_usage.py` — Covers agent classification,
exact per-request aggregation, and aggregate fallback behavior.
### Modified Files
- `headroom/proxy/server.py` — Adds per-agent usage aggregation to
`/stats` with before tokens, after tokens, output tokens, saved tokens,
savings percentage, source, providers, and models.
- `headroom/dashboard/templates/dashboard.html` — Adds a prominent Agent
Usage panel with totals, coverage status, per-agent token-flow bars,
request counts, before/after tokens, saved tokens, and share of savings.
## Testing
- [x] Unit tests pass: `.venv312/bin/pytest
tests/test_dashboard_agent_usage.py`
- [x] Linting passes: `.venv312/bin/ruff check headroom/proxy/server.py
tests/test_dashboard_agent_usage.py`
- [x] Diff whitespace check passes: `git diff --check
origin/main...HEAD`
- [x] Dashboard smoke render: local proxy on `127.0.0.1:8790`, captured
Chrome headless screenshot of `/dashboard`
- [x] New tests added for new functionality
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing relevant unit tests pass locally with my changes
- [ ] I have made corresponding changes to the documentation
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
The agent usage panel uses exact request-log data when available. If
detailed request logs are empty, it falls back to aggregate
provider/model request counts and labels the coverage as aggregate
fallback so users are not misled.
2026-06-12 22:12:22 +03:00
|
|
|
logger.logs = [first_log]
|
|
|
|
|
first_response = client.get("/stats?cached=1")
|
|
|
|
|
assert first_response.status_code == 200
|
|
|
|
|
assert first_response.json()["recent_requests"][-1]["model"] == "gpt-4.1"
|
|
|
|
|
|
|
|
|
|
logger.logs = [first_log, second_log]
|
|
|
|
|
second_response = client.get("/stats?cached=1")
|
|
|
|
|
assert second_response.status_code == 200
|
|
|
|
|
second_payload = second_response.json()
|
|
|
|
|
|
feat(dashboard): persist lifetime proxy metrics (#2198)
## Description
Persist bounded, aggregate-only Lifetime dashboard metrics across proxy
restarts and expose them through a new `/stats-lifetime` endpoint. The
change keeps session/runtime stats separate from durable lifetime stats,
gates sensitive dashboard metadata for loopback or explicitly trusted
dashboard clients, and updates the dashboard Lifetime view to consume
the new endpoint.
Closes #2137
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)
## Changes Made
- Added bounded persistent lifetime metrics state and wired proxy metric
events into it.
- Added `/stats-lifetime` with sensitive project/persistence details
gated behind dashboard metadata access checks.
- Extended loopback/dashboard metadata access policy for trusted
dashboard client CIDRs without widening admin/debug endpoints.
- Reorganized dashboard session/lifetime presentation around runtime
counters versus durable aggregates.
- Added focused tests for persistent aggregation, persistence, endpoint
registration, loopback gating, trusted dashboard CIDRs, and recent
request ordering.
- Fixed current Ruff/mypy issues in the lifetime metrics normalization
code.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
uv run --extra proxy --with pytest --with pytest-asyncio pytest tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_stats_recent_requests.py -q
53 passed, 1 warning
uvx ruff==0.15.17 check headroom/proxy/forwarded_headers.py headroom/proxy/loopback_guard.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py headroom/proxy/server.py tests/test_forwarded_headers.py tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_project_savings.py tests/test_proxy_stats_recent_requests.py
All checks passed!
uv run --extra proxy --with mypy mypy headroom/proxy/persistent_metrics.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows review worktree, Python 3.13.3 via uv.
- Exact command / steps: Ran the focused persistent metrics,
persistence, loopback gating, and recent request tests; ran CI-matching
Ruff on touched files; ran mypy on the new persistent metrics module.
- Observed result: `/stats-lifetime` is registered, non-loopback callers
receive only non-sensitive aggregate data, loopback/trusted dashboard
clients receive the full lifetime payload, admin/debug endpoints remain
loopback-only, and persistent metrics normalize malformed stored state
without type/lint errors.
- Not tested: Full repository pytest suite, full dashboard browser
screenshot pass, or live long-running proxy traffic.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
N/A
## Additional Notes
No changelog entry is required for this dashboard/internal metrics
iteration. The endpoint intentionally exposes only aggregate lifetime
data to ordinary network callers and strips project/persistence details
unless the caller passes the dashboard metadata access policy.
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-16 02:18:13 +08:00
|
|
|
assert second_payload["recent_requests"][0]["model"] == "claude-sonnet"
|
feat: add dashboard agent usage stats (#814)
## Description
Add a clear dashboard view for per-agent token usage so end users can
see Cursor, Claude, Codex, and other detected clients with before/after
token counts, tokens saved, and savings percentages. The stats API now
exposes a stable `agent_usage` object that the dashboard renders near
the top of the session view.
Fixes #
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
### New Files
**Tests:**
- `tests/test_dashboard_agent_usage.py` — Covers agent classification,
exact per-request aggregation, and aggregate fallback behavior.
### Modified Files
- `headroom/proxy/server.py` — Adds per-agent usage aggregation to
`/stats` with before tokens, after tokens, output tokens, saved tokens,
savings percentage, source, providers, and models.
- `headroom/dashboard/templates/dashboard.html` — Adds a prominent Agent
Usage panel with totals, coverage status, per-agent token-flow bars,
request counts, before/after tokens, saved tokens, and share of savings.
## Testing
- [x] Unit tests pass: `.venv312/bin/pytest
tests/test_dashboard_agent_usage.py`
- [x] Linting passes: `.venv312/bin/ruff check headroom/proxy/server.py
tests/test_dashboard_agent_usage.py`
- [x] Diff whitespace check passes: `git diff --check
origin/main...HEAD`
- [x] Dashboard smoke render: local proxy on `127.0.0.1:8790`, captured
Chrome headless screenshot of `/dashboard`
- [x] New tests added for new functionality
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing relevant unit tests pass locally with my changes
- [ ] I have made corresponding changes to the documentation
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
The agent usage panel uses exact request-log data when available. If
detailed request logs are empty, it falls back to aggregate
provider/model request counts and labels the coverage as aggregate
fallback so users are not misled.
2026-06-12 22:12:22 +03:00
|
|
|
assert second_payload["request_logs"][-1]["model"] == "claude-sonnet"
|
|
|
|
|
|
|
|
|
|
|
2026-07-13 09:09:51 -05:00
|
|
|
def test_stats_recent_requests_includes_token_incomplete_requests() -> None:
|
|
|
|
|
app = create_app(
|
|
|
|
|
ProxyConfig(
|
|
|
|
|
optimize=False,
|
|
|
|
|
cache_enabled=False,
|
|
|
|
|
rate_limit_enabled=False,
|
|
|
|
|
cost_tracking_enabled=False,
|
|
|
|
|
log_requests=False,
|
|
|
|
|
ccr_inject_tool=False,
|
|
|
|
|
ccr_handle_responses=False,
|
|
|
|
|
ccr_context_tracking=False,
|
|
|
|
|
http2=False,
|
|
|
|
|
)
|
|
|
|
|
)
|
|
|
|
|
logger = FakeRequestLogger()
|
|
|
|
|
app.state.proxy.logger = logger
|
|
|
|
|
|
|
|
|
|
logger.logs = [
|
|
|
|
|
FakeLogEntry(
|
|
|
|
|
{
|
|
|
|
|
"request_id": "req-haiku-1",
|
|
|
|
|
"timestamp": "2026-07-09T10:00:00Z",
|
|
|
|
|
"provider": "anthropic",
|
|
|
|
|
"model": "claude-haiku",
|
|
|
|
|
"transforms_applied": [],
|
|
|
|
|
}
|
|
|
|
|
),
|
|
|
|
|
FakeLogEntry(
|
|
|
|
|
{
|
|
|
|
|
"request_id": "req-haiku-2",
|
|
|
|
|
"timestamp": "2026-07-09T10:01:00Z",
|
|
|
|
|
"provider": "anthropic",
|
|
|
|
|
"model": "claude-haiku",
|
|
|
|
|
"input_tokens_original": None,
|
|
|
|
|
"input_tokens_optimized": None,
|
|
|
|
|
"output_tokens": None,
|
|
|
|
|
"tokens_saved": 0,
|
|
|
|
|
"savings_percent": 0.0,
|
|
|
|
|
"transforms_applied": [],
|
|
|
|
|
}
|
|
|
|
|
),
|
|
|
|
|
FakeLogEntry(
|
|
|
|
|
{
|
|
|
|
|
"request_id": "req-sonnet-1",
|
|
|
|
|
"timestamp": "2026-07-09T10:02:00Z",
|
|
|
|
|
"provider": "anthropic",
|
|
|
|
|
"model": "claude-sonnet",
|
|
|
|
|
"input_tokens_original": 200,
|
|
|
|
|
"input_tokens_optimized": 120,
|
|
|
|
|
"output_tokens": 40,
|
|
|
|
|
"tokens_saved": 80,
|
|
|
|
|
"savings_percent": 40.0,
|
|
|
|
|
"transforms_applied": ["smart_crusher"],
|
|
|
|
|
}
|
|
|
|
|
),
|
|
|
|
|
]
|
|
|
|
|
|
|
|
|
|
with TestClient(app, base_url="http://127.0.0.1", client=("127.0.0.1", 12345)) as client:
|
|
|
|
|
response = client.get("/stats")
|
|
|
|
|
|
|
|
|
|
assert response.status_code == 200
|
|
|
|
|
payload = response.json()
|
|
|
|
|
assert [req["model"] for req in payload["recent_requests"]] == [
|
feat(dashboard): persist lifetime proxy metrics (#2198)
## Description
Persist bounded, aggregate-only Lifetime dashboard metrics across proxy
restarts and expose them through a new `/stats-lifetime` endpoint. The
change keeps session/runtime stats separate from durable lifetime stats,
gates sensitive dashboard metadata for loopback or explicitly trusted
dashboard clients, and updates the dashboard Lifetime view to consume
the new endpoint.
Closes #2137
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)
## Changes Made
- Added bounded persistent lifetime metrics state and wired proxy metric
events into it.
- Added `/stats-lifetime` with sensitive project/persistence details
gated behind dashboard metadata access checks.
- Extended loopback/dashboard metadata access policy for trusted
dashboard client CIDRs without widening admin/debug endpoints.
- Reorganized dashboard session/lifetime presentation around runtime
counters versus durable aggregates.
- Added focused tests for persistent aggregation, persistence, endpoint
registration, loopback gating, trusted dashboard CIDRs, and recent
request ordering.
- Fixed current Ruff/mypy issues in the lifetime metrics normalization
code.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
uv run --extra proxy --with pytest --with pytest-asyncio pytest tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_stats_recent_requests.py -q
53 passed, 1 warning
uvx ruff==0.15.17 check headroom/proxy/forwarded_headers.py headroom/proxy/loopback_guard.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py headroom/proxy/server.py tests/test_forwarded_headers.py tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_project_savings.py tests/test_proxy_stats_recent_requests.py
All checks passed!
uv run --extra proxy --with mypy mypy headroom/proxy/persistent_metrics.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows review worktree, Python 3.13.3 via uv.
- Exact command / steps: Ran the focused persistent metrics,
persistence, loopback gating, and recent request tests; ran CI-matching
Ruff on touched files; ran mypy on the new persistent metrics module.
- Observed result: `/stats-lifetime` is registered, non-loopback callers
receive only non-sensitive aggregate data, loopback/trusted dashboard
clients receive the full lifetime payload, admin/debug endpoints remain
loopback-only, and persistent metrics normalize malformed stored state
without type/lint errors.
- Not tested: Full repository pytest suite, full dashboard browser
screenshot pass, or live long-running proxy traffic.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
N/A
## Additional Notes
No changelog entry is required for this dashboard/internal metrics
iteration. The endpoint intentionally exposes only aggregate lifetime
data to ordinary network callers and strips project/persistence details
unless the caller passes the dashboard metadata access policy.
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-16 02:18:13 +08:00
|
|
|
"claude-sonnet",
|
2026-07-13 09:09:51 -05:00
|
|
|
"claude-haiku",
|
|
|
|
|
"claude-haiku",
|
|
|
|
|
]
|
feat(dashboard): persist lifetime proxy metrics (#2198)
## Description
Persist bounded, aggregate-only Lifetime dashboard metrics across proxy
restarts and expose them through a new `/stats-lifetime` endpoint. The
change keeps session/runtime stats separate from durable lifetime stats,
gates sensitive dashboard metadata for loopback or explicitly trusted
dashboard clients, and updates the dashboard Lifetime view to consume
the new endpoint.
Closes #2137
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)
## Changes Made
- Added bounded persistent lifetime metrics state and wired proxy metric
events into it.
- Added `/stats-lifetime` with sensitive project/persistence details
gated behind dashboard metadata access checks.
- Extended loopback/dashboard metadata access policy for trusted
dashboard client CIDRs without widening admin/debug endpoints.
- Reorganized dashboard session/lifetime presentation around runtime
counters versus durable aggregates.
- Added focused tests for persistent aggregation, persistence, endpoint
registration, loopback gating, trusted dashboard CIDRs, and recent
request ordering.
- Fixed current Ruff/mypy issues in the lifetime metrics normalization
code.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
uv run --extra proxy --with pytest --with pytest-asyncio pytest tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_stats_recent_requests.py -q
53 passed, 1 warning
uvx ruff==0.15.17 check headroom/proxy/forwarded_headers.py headroom/proxy/loopback_guard.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py headroom/proxy/server.py tests/test_forwarded_headers.py tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_project_savings.py tests/test_proxy_stats_recent_requests.py
All checks passed!
uv run --extra proxy --with mypy mypy headroom/proxy/persistent_metrics.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows review worktree, Python 3.13.3 via uv.
- Exact command / steps: Ran the focused persistent metrics,
persistence, loopback gating, and recent request tests; ran CI-matching
Ruff on touched files; ran mypy on the new persistent metrics module.
- Observed result: `/stats-lifetime` is registered, non-loopback callers
receive only non-sensitive aggregate data, loopback/trusted dashboard
clients receive the full lifetime payload, admin/debug endpoints remain
loopback-only, and persistent metrics normalize malformed stored state
without type/lint errors.
- Not tested: Full repository pytest suite, full dashboard browser
screenshot pass, or live long-running proxy traffic.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
N/A
## Additional Notes
No changelog entry is required for this dashboard/internal metrics
iteration. The endpoint intentionally exposes only aggregate lifetime
data to ordinary network callers and strips project/persistence details
unless the caller passes the dashboard metadata access policy.
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-16 02:18:13 +08:00
|
|
|
assert payload["recent_requests"][0]["token_accounting_status"] == "complete"
|
|
|
|
|
assert payload["recent_requests"][0]["has_exact_tokens"] is True
|
2026-07-13 09:09:51 -05:00
|
|
|
assert payload["recent_requests"][1]["output_tokens"] is None
|
|
|
|
|
assert payload["recent_requests"][1]["token_accounting_status"] == "partial"
|
|
|
|
|
assert payload["recent_requests"][1]["tokens_saved"] == 0
|
feat(dashboard): persist lifetime proxy metrics (#2198)
## Description
Persist bounded, aggregate-only Lifetime dashboard metrics across proxy
restarts and expose them through a new `/stats-lifetime` endpoint. The
change keeps session/runtime stats separate from durable lifetime stats,
gates sensitive dashboard metadata for loopback or explicitly trusted
dashboard clients, and updates the dashboard Lifetime view to consume
the new endpoint.
Closes #2137
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)
## Changes Made
- Added bounded persistent lifetime metrics state and wired proxy metric
events into it.
- Added `/stats-lifetime` with sensitive project/persistence details
gated behind dashboard metadata access checks.
- Extended loopback/dashboard metadata access policy for trusted
dashboard client CIDRs without widening admin/debug endpoints.
- Reorganized dashboard session/lifetime presentation around runtime
counters versus durable aggregates.
- Added focused tests for persistent aggregation, persistence, endpoint
registration, loopback gating, trusted dashboard CIDRs, and recent
request ordering.
- Fixed current Ruff/mypy issues in the lifetime metrics normalization
code.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
uv run --extra proxy --with pytest --with pytest-asyncio pytest tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_stats_recent_requests.py -q
53 passed, 1 warning
uvx ruff==0.15.17 check headroom/proxy/forwarded_headers.py headroom/proxy/loopback_guard.py headroom/proxy/persistent_metrics.py headroom/proxy/savings_tracker.py headroom/proxy/server.py tests/test_forwarded_headers.py tests/test_persistent_metrics.py tests/test_persistent_metrics_integration.py tests/test_persistent_metrics_persistence.py tests/test_proxy_loopback_gating.py tests/test_proxy_project_savings.py tests/test_proxy_stats_recent_requests.py
All checks passed!
uv run --extra proxy --with mypy mypy headroom/proxy/persistent_metrics.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows review worktree, Python 3.13.3 via uv.
- Exact command / steps: Ran the focused persistent metrics,
persistence, loopback gating, and recent request tests; ran CI-matching
Ruff on touched files; ran mypy on the new persistent metrics module.
- Observed result: `/stats-lifetime` is registered, non-loopback callers
receive only non-sensitive aggregate data, loopback/trusted dashboard
clients receive the full lifetime payload, admin/debug endpoints remain
loopback-only, and persistent metrics normalize malformed stored state
without type/lint errors.
- Not tested: Full repository pytest suite, full dashboard browser
screenshot pass, or live long-running proxy traffic.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
N/A
## Additional Notes
No changelog entry is required for this dashboard/internal metrics
iteration. The endpoint intentionally exposes only aggregate lifetime
data to ordinary network callers and strips project/persistence details
unless the caller passes the dashboard metadata access policy.
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-16 02:18:13 +08:00
|
|
|
assert payload["recent_requests"][2]["input_tokens_optimized"] is None
|
|
|
|
|
assert payload["recent_requests"][2]["token_accounting_status"] == "missing"
|
|
|
|
|
assert payload["recent_requests"][2]["has_exact_tokens"] is False
|
2026-07-13 09:09:51 -05:00
|
|
|
assert payload["summary"]["uncompressed_requests"]["unknown_token_accounting"] == 2
|
|
|
|
|
assert payload["request_logs"][-1]["model"] == "claude-sonnet"
|
|
|
|
|
|
|
|
|
|
|
feat: add dashboard agent usage stats (#814)
## Description
Add a clear dashboard view for per-agent token usage so end users can
see Cursor, Claude, Codex, and other detected clients with before/after
token counts, tokens saved, and savings percentages. The stats API now
exposes a stable `agent_usage` object that the dashboard renders near
the top of the session view.
Fixes #
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
### New Files
**Tests:**
- `tests/test_dashboard_agent_usage.py` — Covers agent classification,
exact per-request aggregation, and aggregate fallback behavior.
### Modified Files
- `headroom/proxy/server.py` — Adds per-agent usage aggregation to
`/stats` with before tokens, after tokens, output tokens, saved tokens,
savings percentage, source, providers, and models.
- `headroom/dashboard/templates/dashboard.html` — Adds a prominent Agent
Usage panel with totals, coverage status, per-agent token-flow bars,
request counts, before/after tokens, saved tokens, and share of savings.
## Testing
- [x] Unit tests pass: `.venv312/bin/pytest
tests/test_dashboard_agent_usage.py`
- [x] Linting passes: `.venv312/bin/ruff check headroom/proxy/server.py
tests/test_dashboard_agent_usage.py`
- [x] Diff whitespace check passes: `git diff --check
origin/main...HEAD`
- [x] Dashboard smoke render: local proxy on `127.0.0.1:8790`, captured
Chrome headless screenshot of `/dashboard`
- [x] New tests added for new functionality
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing relevant unit tests pass locally with my changes
- [ ] I have made corresponding changes to the documentation
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
The agent usage panel uses exact request-log data when available. If
detailed request logs are empty, it falls back to aggregate
provider/model request counts and labels the coverage as aggregate
fallback so users are not misled.
2026-06-12 22:12:22 +03:00
|
|
|
def test_agent_usage_totals_use_proxy_only_savings(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
|
|
|
monkeypatch.setenv("HEADROOM_REQUIRE_RUST_CORE", "false")
|
|
|
|
|
app = create_app(
|
|
|
|
|
ProxyConfig(
|
|
|
|
|
optimize=False,
|
|
|
|
|
cache_enabled=False,
|
|
|
|
|
rate_limit_enabled=False,
|
|
|
|
|
cost_tracking_enabled=False,
|
|
|
|
|
log_requests=False,
|
|
|
|
|
ccr_inject_tool=False,
|
|
|
|
|
ccr_handle_responses=False,
|
|
|
|
|
ccr_context_tracking=False,
|
|
|
|
|
http2=False,
|
|
|
|
|
)
|
|
|
|
|
)
|
|
|
|
|
logger = FakeRequestLogger()
|
|
|
|
|
app.state.proxy.logger = logger
|
|
|
|
|
|
|
|
|
|
logger.logs = [
|
|
|
|
|
FakeLogEntry(
|
|
|
|
|
{
|
|
|
|
|
"timestamp": "2026-06-11T10:00:00Z",
|
|
|
|
|
"provider": "openai",
|
|
|
|
|
"model": "gpt-5.2-codex",
|
|
|
|
|
"tags": {"client": "codex"},
|
|
|
|
|
"input_tokens_original": 1000,
|
|
|
|
|
"input_tokens_optimized": 900,
|
|
|
|
|
"output_tokens": 50,
|
|
|
|
|
"tokens_saved": 100,
|
|
|
|
|
"savings_percent": 10.0,
|
|
|
|
|
}
|
|
|
|
|
)
|
|
|
|
|
]
|
|
|
|
|
|
|
|
|
|
with TestClient(app) as client:
|
|
|
|
|
proxy = client.app.state.proxy
|
|
|
|
|
proxy.metrics.tokens_input_total = 900
|
|
|
|
|
proxy.metrics.tokens_saved_total = 100
|
|
|
|
|
proxy.metrics.tokens_output_total = 50
|
|
|
|
|
|
|
|
|
|
response = client.get("/stats")
|
|
|
|
|
|
|
|
|
|
assert response.status_code == 200
|
|
|
|
|
payload = response.json()
|
|
|
|
|
|
fix: remove rtk and lean-ctx CLI context tools (#2677)
## Description
Removes both third-party CLI context tools — **rtk** and **lean-ctx** —
and with them the context-tool selector itself. Headroom no longer
downloads, installs or configures either one, and there is no
replacement.
The previous pass (#2344) gated only three entry points inside
`headroom/cli/wrap.py`. That left the feature reachable in practice:
| Gap | Effect |
|---|---|
| `scripts/install.sh:1544`, `install.ps1:1681` | Ran `rtk init --global
--auto-patch` from bash/PowerShell, **bypassing the Python gate
entirely** — `curl \| sh` still wrote a Claude Code `PreToolUse` hook
regardless of `HEADROOM_RTK` |
| `wrap.py` `_setup_context_tool_for_agent` | **`wrap openhands` was
broken by default**: `rtk_required=True` met a gate returning `None` →
`SystemExit(1)`. Invisible because all 8 openhands tests patched
`_ensure_rtk_binary` to a fake path |
| `proxy/helpers.py`, `subscription/tracker.py` | Proxy shelled out to
`rtk gain` from `/stats`, the dashboard and `headroom perf`; the tracker
polled it per contribution (`_RTK_WIRING_DEFAULT = "enabled"`) |
| No cleanup path | Nothing removed artifacts an earlier default had
installed, so a machine that once ran the old default kept rtk in the
loop forever (#1669, #1955) |
Also worth noting: the rtk binary download had **no SHA or signature
verification** — only `rtk --version` as a smoke test.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)
## Changes Made
**Removed** — `headroom/rtk/` and `headroom/lean_ctx/` packages,
`headroom/cli/wrap_rtk_metrics.py`, `_selected_context_tool` /
`_setup_context_tool_for_agent` / `_VALID_CONTEXT_TOOLS`, the `--rtk` /
`--no-rtk` / `--no-project-rtk` / `--keep-rtk` flags across all 18 wrap
subcommands, `HEADROOM_RTK*`, the proxy-side `rtk gain` polling, the
dashboard CLI-filtering panel (rows + all 8 `cliFiltering*` Alpine
getters), `paths.rtk_path()` / `lean_ctx_path()`, the SDK path helpers,
`benchmarks/rtk_loop_learn_eval.py`, and the `headroom/rtk/**` CI path
filters.
**Fails loudly, not silently** — `--context-tool` / `--no-context-tool`
/ `HEADROOM_CONTEXT_TOOL` are kept solely to error out. They live in
shell profiles, aliases and CI jobs, and accepting them as a no-op would
read as Headroom having quietly stopped working. The installers reject
them too, which matters more than it looks: their arg parsers forward
the first unknown flag **and everything after it** to the wrapped tool,
so a leftover `--no-rtk` would have silently swallowed a following
`--port` and then been ignored downstream.
**New `headroom/context_tool_cleanup.py`** — deleting the code cannot
help a machine that already ran the old default, since the hooks,
binaries and injected guidance are durable on disk.
`purge_context_tool_artifacts()` runs once per `wrap`/`unwrap` and
removes the registered hook entries, the generated hook scripts, the
Headroom-managed `~/.local/bin` symlinks, the vendored
`~/.headroom/bin/{rtk,lean-ctx}` binaries, the `lean-ctx` MCP server
entry and the marker-fenced instruction blocks. Deliberately
conservative: idempotent, **skips** a malformed config rather than
overwriting it, and only unlinks a symlink resolving inside Headroom's
own bin dir so a user's own build is untouched. It reports on
**stderr**, because `wrap/unwrap openclaw --prepare-only` emit
machine-readable JSON on stdout as their entire contract. Skipped for
`wrap selfheal` (runs from a SessionStart hook; must not race Claude
Code's writer for `~/.claude.json`) and for `--help`, which must stay
read-only.
**Client-config hardening** (discovered while investigating a "corrupted
Serena settings file" report) — `wrap.py` reset a settings file to `{}`
when an existing file would not parse, then wrote that back. One
hand-edited typo or a transient `EACCES`/`EINTR` on a valid file
destroyed the user's `permissions`, `env` and `hooks`, on **every
`headroom wrap claude`**. It now refuses to write. Separately,
`fsutil.write_text` is now atomic (temp file + `fsync` + `os.replace`),
fixing all 14 non-atomic client-config writes at once; it follows
symlinks rather than replacing them (dotfile managers) and preserves an
existing file's mode.
**Deliberately kept** — `rtk` stays in the wrapper-peel list in
`transforms/content_router.py`. It sits beside `sudo`/`env`/`timeout` as
shell-command grammar, so `rtk cat f` is still classified as a file read
for anyone running their own rtk install, which the purge intentionally
leaves alone.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ ruff check headroom/ tests/ e2e/ --exclude headroom/dashboard/templates
All checks passed!
$ ruff format --check headroom/ tests/ e2e/ --exclude headroom/dashboard/templates
1255 files already formatted
$ mypy headroom/
Success: no issues found in 508 source files
$ pytest tests/test_context_tool_cleanup.py -q
11 passed
$ pytest tests/test_fsutil.py -q
12 passed
$ pytest tests/test_cli/test_wrap_codex.py -q # 89 tests
89 passed in 431.68s
$ pytest tests/test_cli/test_wrap_opencode.py -q
39 passed in 257.46s
$ pytest tests/test_cli/test_wrap_helpers.py -q
45 passed
$ pytest tests/test_paths.py -q
75 passed
$ pytest tests/test_cli/test_unwrap_claude.py -q
14 passed
$ pytest tests/test_proxy_savings_history.py -q
39 passed
$ pytest tests/test_cli/test_wrap_copilot.py -q
27 passed
$ pytest tests/test_cli/test_wrap_zcode.py -q
20 passed
$ pytest tests/test_subscription_tracker.py -q
9 passed
$ pytest tests/test_proxy_dashboard_stats_cache.py -q
5 passed, 1 skipped
```
Repo-wide grep for 14 removed symbols (`headroom.rtk`,
`headroom.lean_ctx`, `_ensure_rtk_binary`, `_selected_context_tool`,
`_get_context_tool_stats`, `rtk_path`, `lean_ctx_path`,
`wrap_rtk_metrics`, `HEADROOM_RTK`, `cli_tokens_avoided`,
`tokens_saved_rtk`, …) across `*.py`, `*.ts`, `*.sh`, `*.ps1`, `*.yml`,
`*.html`: **zero hits**.
Notable test changes: `test_wrap_openhands.py` no longer patches
`_ensure_rtk_binary` and asserts `wrap openhands --prepare-only` exits 0
unpatched — the regression that was previously masked.
`test_wrap_continue.py` and `test_wrap_hintfile_agents.py` were removed
(every test drove RTK instruction injection). A new
`test_subscription_tracker.py::test_load_state_written_before_cli_context_tools_were_removed`
proves a pre-removal `subscription_state.json` still loads.
## Real Behavior Proof
- **Environment:** macOS 15.4 (darwin 25.4.0), Python 3.12.6, Headroom @
this branch, real `~/.headroom` and `~/.claude` on the dev machine.
- **Exact command / steps and observed result:**
```text
# 1. Retired flag fails loudly instead of silently no-op'ing
$ headroom wrap codex --prepare-only --context-tool rtk
Error: CLI context tools (rtk, lean-ctx) have been removed from Headroom: they
rewrote shell commands through a third-party binary Headroom no longer manages.
Drop --context-tool / --no-context-tool and unset HEADROOM_CONTEXT_TOOL;
`headroom wrap` uninstalls what they left behind on first run.
$ HEADROOM_CONTEXT_TOOL=lean-ctx headroom wrap codex --prepare-only
Error: CLI context tools (rtk, lean-ctx) have been removed from Headroom: ...
# 2. install.sh rejects the retired flags (extracted parse_wrap_args harness)
['--no-rtk', '--port', '9999'] rc=1 ERROR: CLI context tools ... Drop --no-rtk
['--context-tool=rtk'] rc=1 ERROR: CLI context tools ... Drop --context-tool
$ bash -n scripts/install.sh # syntax OK
# 3. Purge ran against the real machine, which had all the orphaned artifacts
$ python -c "from headroom.context_tool_cleanup import purge_context_tool_artifacts; ..."
removed ~/.headroom/bin/lean-ctx (51 MB)
removed ~/.headroom/bin/rtk (7.7 MB)
removed ~/.local/bin/rtk (symlink into ~/.headroom/bin)
removed ~/.claude/hooks/rtk-rewrite.sh
removed 8 lean-ctx-* hook scripts
# ~/.claude.json afterwards: 90 top-level keys, 19 projects, mcpServers unchanged
# → ~59 MB reclaimed, no unrelated key touched
# 4. stdout stays machine-readable while the purge reports (planted a fake artifact)
$ headroom wrap openclaw --prepare-only --gateway-provider-id codex >out 2>err
$ cat out
{"enabled":true,"config":{"proxyPort":8787,...}} # parses as JSON
$ cat err
Retired CLI context tool cleanup: removed /Users/tcms/.headroom/bin/rtk
# 5. --help is inert (planted artifact survives), a real run purges
$ headroom wrap codex --help → artifact survived: CORRECT
$ headroom wrap openclaw --prepare-only → purged: CORRECT
# 6. MCP purge dry-run against a copy of the real 82 KB ~/.claude.json
top-level keys 90 -> 90; projects 19 -> 19; LOST keys: none
all content outside mcpServers byte-identical: True
```
Dashboard rendered via the Playwright test after the panel removal:
"Token Savings" shows only `Proxy 0 (0.0%)` / `Of total wire: 36.86%`,
and "Token Usage" reads Before Compression → Proxy Removed → After
Compression with no "Filtered (this session)" row. Nothing below the
removed panel broke.
- **Not tested:** Windows and Linux (macOS only) — `install.ps1` is
verified by brace-balance and inspection, not executed, since no `pwsh`
is available locally. The wrap e2e suite (`e2e/wrap/run.py`) was updated
but not run; it needs the Docker e2e image. `serena project index`
interaction is exercised in the stacked base PR.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
**Stacked on #2676** (`tejas/serena-config-bootstrap`) — please merge
that first; this PR's base should then be retargeted to `main`, or it
will read as containing that fix too.
**Breaking-change migration for users:**
- Drop `--rtk`, `--no-rtk`, `--no-project-rtk`, `--keep-rtk`,
`--context-tool`, `--no-context-tool` from any alias, script or CI job,
and unset `HEADROOM_RTK*` / `HEADROOM_CONTEXT_TOOL`. They now error
rather than being ignored, so the failure is immediate and
self-explaining.
- Previously-installed artifacts are purged automatically on the next
`wrap`/`unwrap`; no manual cleanup needed.
- `headroom perf --json` no longer carries a `cli_filtering` key, and
`/stats` no longer returns a `context_tool` section.
**Docs:** `docs/rtk-architecture.md` deleted; RTK/lean-ctx removed from
`README.md`,
`docs/content/docs/{configuration,opencode,grok-build,docker-install,filesystem-contract}.mdx`,
`docs/observability.md` and the matching `wiki/` pages.
`REALIGNMENT/09-phase-G-rtk-observability.md` is marked SUPERSEDED
rather than deleted, to keep the planning record.
**Follow-ups not in scope:** `_emit_wrap_interrupted` was deleted as
dead code — its only caller was the `except KeyboardInterrupt` guarding
the binary download, so with no download there is nothing slow left to
interrupt.
2026-07-30 22:59:41 -07:00
|
|
|
assert payload["tokens"]["saved"] == 100
|
feat: add dashboard agent usage stats (#814)
## Description
Add a clear dashboard view for per-agent token usage so end users can
see Cursor, Claude, Codex, and other detected clients with before/after
token counts, tokens saved, and savings percentages. The stats API now
exposes a stable `agent_usage` object that the dashboard renders near
the top of the session view.
Fixes #
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
### New Files
**Tests:**
- `tests/test_dashboard_agent_usage.py` — Covers agent classification,
exact per-request aggregation, and aggregate fallback behavior.
### Modified Files
- `headroom/proxy/server.py` — Adds per-agent usage aggregation to
`/stats` with before tokens, after tokens, output tokens, saved tokens,
savings percentage, source, providers, and models.
- `headroom/dashboard/templates/dashboard.html` — Adds a prominent Agent
Usage panel with totals, coverage status, per-agent token-flow bars,
request counts, before/after tokens, saved tokens, and share of savings.
## Testing
- [x] Unit tests pass: `.venv312/bin/pytest
tests/test_dashboard_agent_usage.py`
- [x] Linting passes: `.venv312/bin/ruff check headroom/proxy/server.py
tests/test_dashboard_agent_usage.py`
- [x] Diff whitespace check passes: `git diff --check
origin/main...HEAD`
- [x] Dashboard smoke render: local proxy on `127.0.0.1:8790`, captured
Chrome headless screenshot of `/dashboard`
- [x] New tests added for new functionality
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing relevant unit tests pass locally with my changes
- [ ] I have made corresponding changes to the documentation
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
The agent usage panel uses exact request-log data when available. If
detailed request logs are empty, it falls back to aggregate
provider/model request counts and labels the coverage as aggregate
fallback so users are not misled.
2026-06-12 22:12:22 +03:00
|
|
|
assert payload["agent_usage"]["totals"]["before_tokens"] == 1000
|
|
|
|
|
assert payload["agent_usage"]["totals"]["tokens_saved"] == 100
|
|
|
|
|
assert payload["agent_usage"]["totals"]["savings_percent"] == 10.0
|
|
|
|
|
assert payload["agent_usage"]["agents"][0]["share_of_saved_percent"] == 100.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def test_stats_preserves_default_smart_crusher_compaction_state() -> None:
|
|
|
|
|
config = ProxyConfig(
|
|
|
|
|
optimize=False,
|
|
|
|
|
cache_enabled=False,
|
|
|
|
|
rate_limit_enabled=False,
|
|
|
|
|
cost_tracking_enabled=False,
|
|
|
|
|
)
|
fix(proxy): scope CORS to loopback + gate operator/content endpoints (#1226)
## Description
Locks down the proxy's browser- and network-facing attack surface, which
matters most under a `--host 0.0.0.0` bind (the Docker default). The
wildcard CORS policy (`allow_origins=["*"]` + `allow_credentials=True`)
let any web page the user had open read the proxy's content endpoints —
`/v1/retrieve` returns raw, uncompressed tool outputs (source, secrets)
— via a cross-origin fetch to `127.0.0.1` (CWE-346). Several operator
endpoints additionally leaked sensitive data or allowed unauthenticated
state mutation to any network-reachable client.
This PR scopes CORS to loopback origins and extends the project's
existing `require_loopback` trust boundary (already used for `/admin/*`
and `/debug/*`) to the remaining exposed endpoints.
Closes #863.
Supersedes #864 and #758 — see "Additional Notes".
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- **CORS**: replaced `allow_origins=["*"]` + `allow_credentials=True`
with a port-agnostic loopback origin regex
(`https?://(localhost|127\.0\.0\.1|\[::1\])(:\d+)?`),
`allow_credentials=False`, and methods/headers narrowed to `GET/POST` +
`Content-Type/Authorization`. `HEADROOM_CORS_ORIGINS` (comma-separated)
pins an explicit allowlist for Docker/remote dashboards; `*` opts back
into the old wildcard.
- **`/transformations/feed`** and **`/cache/clear`** gated behind
`require_loopback` → 404 for non-loopback callers. The feed returns full
prompt/completion bodies when `log_full_messages` is on; `/cache/clear`
is unauthenticated state mutation (cache-eviction DoS / cost
amplification).
- **`/health`**: the `config` block (upstream API URLs, savings profile)
is now served only to loopback callers; network callers get the
`/readyz`-shape body (status/checks). `/livez` and `/readyz` remain
unauthenticated probes for orchestration.
- **`/stats`**: `recent_requests` / `request_logs` (per-request ids,
providers, models, errors) and `config` are served only to loopback
callers; aggregate counters stay public for remote monitoring.
- Added `_request_is_loopback()` helper mirroring `require_loopback`'s
two-gate check (loopback peer IP + loopback `Host` header, the
DNS-rebinding defence) but degrading the payload instead of returning
404, so monitors keep the non-sensitive fields.
- Tests: new `tests/test_proxy_cors.py` and
`tests/test_proxy_loopback_gating.py`; updated 4 existing tests that
assert the now-loopback-only data to use loopback clients.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ ruff check headroom/proxy/server.py tests/test_proxy_cors.py \
tests/test_proxy_loopback_gating.py tests/test_proxy_healthchecks.py \
tests/test_proxy_stats_recent_requests.py tests/test_proxy/test_transformations_feed.py
All checks passed!
$ mypy headroom --ignore-missing-imports
Success: no issues found in 380 source files
$ pytest tests/test_proxy_cors.py tests/test_proxy_loopback_gating.py \
tests/test_proxy/test_transformations_feed.py tests/test_proxy_healthchecks.py \
tests/test_proxy_stats_recent_requests.py tests/test_proxy_dashboard_stats_cache.py \
tests/test_proxy_compression_executor.py tests/test_header_isolation.py -q
======================== 82 passed, 1 skipped in 17.75s ========================
```
## Real Behavior Proof
- Environment: macOS (darwin 25.4.0), Python 3.12 venv; FastAPI
`TestClient` driving the real `create_app()` ASGI app
- Exact command / steps: issued requests as a non-loopback caller
(`client.host=testclient`) vs a loopback caller
(`base_url=http://127.0.0.1`, `client=("127.0.0.1", 9999)`), plus CORS
preflights with varying `Origin` headers
- Observed result: CORS — `http://evil.com` → no
`access-control-allow-origin`; `http://localhost:8787` and
`http://localhost:9000` → echoed (loopback allowed on any port);
`access-control-allow-credentials` → absent. `/cache/clear` and
`/transformations/feed` → 404 (network) / 200 (loopback). `/health`
`config` block present for loopback only. `/stats` `recent_requests`
present for loopback only, while the aggregate `tokens` block stays
present for network callers.
- Not tested: a live `headroom proxy` process bound on `0.0.0.0` reached
from a second host (simulated via ASGI peer/Host instead); end-to-end
browser DNS-rebinding (covered by the `Host`-header gate and its unit
test)
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
N/A — proxy/middleware change; behavior is captured under "Real Behavior
Proof".
## Additional Notes
**Supersedes two stale PRs that target the same issue but have drifted
from `main`:**
- **#864** (`fix(proxy): scope CORS to localhost`, @gabiudrescu) —
correct instinct and the source of the tighter `GET/POST` +
`Content-Type/Authorization` scoping kept here, but it derived the
allowlist from the `HEADROOM_PORT` env var (wrong when `--port` is
passed as a CLI flag), carried ~40 lines of unrelated punctuation churn,
and is ~125 commits behind `main`. The port-agnostic regex used here
resolves the reviewer's port concern.
- **#758** (`security: adversarial review`, @neogenix) — bundled these
same application-layer fixes with a large CI/CD + Docker supply-chain
pass. It is a ~160-commit-behind draft whose `server.py` no longer
merges cleanly (`main` independently adopted the same `require_loopback`
pattern). The application-layer fixes are rebased onto current `main`
here; the CI/Docker/supply-chain hardening from #758 is still valuable
and would be welcome as a separate, rebased PR.
Thanks to @gabiudrescu and @neogenix for the original analysis (#863).
**Deliberate scope / follow-ups (not in this PR):**
- `/stats` aggregate counters and the basic `/health` body remain
readable on a `0.0.0.0` bind by design, so remote monitoring keeps
working. Full lock-down is a one-line `Depends(require_loopback)` each
if preferred.
- The `/v1/retrieve*` family stays network-reachable; it can't be
loopback-gated without breaking legitimate remote/containerized agents
and needs auth instead — tracked separately.
- `ruff check .` is scoped to changed paths above because the dashboard
HTML template trips ruff's `invalid-syntax` (a known repo
false-positive); `mypy` is run over the full `headroom` package.
2026-06-21 00:50:55 -07:00
|
|
|
# Loopback client/Host: the `config` block is served only to loopback callers.
|
|
|
|
|
client = TestClient(
|
|
|
|
|
create_app(config), base_url="http://127.0.0.1", client=("127.0.0.1", 12345)
|
|
|
|
|
)
|
feat: add dashboard agent usage stats (#814)
## Description
Add a clear dashboard view for per-agent token usage so end users can
see Cursor, Claude, Codex, and other detected clients with before/after
token counts, tokens saved, and savings percentages. The stats API now
exposes a stable `agent_usage` object that the dashboard renders near
the top of the session view.
Fixes #
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
### New Files
**Tests:**
- `tests/test_dashboard_agent_usage.py` — Covers agent classification,
exact per-request aggregation, and aggregate fallback behavior.
### Modified Files
- `headroom/proxy/server.py` — Adds per-agent usage aggregation to
`/stats` with before tokens, after tokens, output tokens, saved tokens,
savings percentage, source, providers, and models.
- `headroom/dashboard/templates/dashboard.html` — Adds a prominent Agent
Usage panel with totals, coverage status, per-agent token-flow bars,
request counts, before/after tokens, saved tokens, and share of savings.
## Testing
- [x] Unit tests pass: `.venv312/bin/pytest
tests/test_dashboard_agent_usage.py`
- [x] Linting passes: `.venv312/bin/ruff check headroom/proxy/server.py
tests/test_dashboard_agent_usage.py`
- [x] Diff whitespace check passes: `git diff --check
origin/main...HEAD`
- [x] Dashboard smoke render: local proxy on `127.0.0.1:8790`, captured
Chrome headless screenshot of `/dashboard`
- [x] New tests added for new functionality
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing relevant unit tests pass locally with my changes
- [ ] I have made corresponding changes to the documentation
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
The agent usage panel uses exact request-log data when available. If
detailed request logs are empty, it falls back to aggregate
provider/model request counts and labels the coverage as aggregate
fallback so users are not misled.
2026-06-12 22:12:22 +03:00
|
|
|
|
|
|
|
|
response = client.get("/stats")
|
|
|
|
|
|
|
|
|
|
assert response.status_code == 200
|
|
|
|
|
assert response.json()["config"]["smart_crusher_with_compaction"] is None
|