akadmin/headroom
1
0
Fork 0
mirror of https://github.com/headroomlabs-ai/headroom.git synced 2026-08-27 14:17:10 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
2630 commits 410 branches 169 tags 142 MiB
Commit graph

6 commits

Author SHA1 Message Date
gglucass
6c9f41e08c
perf(perf): skip rotated logs outside the requested window (#3081) 
## Description

`parse_log_files(last_n_hours=N)` reads every `proxy.log*` file in full
— line by line, applying the PERF / STAGE_TIMINGS / ROUTER regexes to
each — and only then filters records against the cutoff. The cost of a
windowed query is O(retained log history), not O(window).

`/stats` is the hot caller. `_build_stats_payload` recomputes throughput
over `last_n_hours=1.0` behind a 10s cache TTL, so anything polling the
endpoint re-reads and re-regexes the entire rotated set every 10 seconds
for an answer that lives in the tail of the newest file or two.

Rotation caps the log directory at 10 MB × 5 backups
(`proxy/helpers.py`), so this is a bounded ~60 MB rather than an
unbounded leak. But it is a fixed tax that ramps up as a user's logs
fill toward that ceiling and then stays there — on a machine that has
reached the cap it is ~0.43s of pure waste on every stats rebuild.

The fix: skip any file whose mtime predates the cutoff. The logs are
append-only, so a file untouched since before the window cannot contain
a record inside it. `--hours 0` ("all data") still reads everything.

## Type of Change

- [x] Performance improvement

## Changes Made

- `parse_log_files` prunes rotated files by mtime before opening them;
files are `stat`'d once and the value reused for the ordering
(previously `stat`'d once per file anyway, as the sort key).
- A file that rotates away between `glob` and `stat` is skipped instead
of raising `OSError`.
- New `PerfReport.log_files_skipped` so coverage reporting stays honest
— `log_files_read` on its own would silently understate how much log
exists on disk. Defaulted, so existing callers are unaffected.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

Both new tests were confirmed to fail against unpatched `main`. The
windowed one fails on behavior (`total_lines_parsed`: `assert 2 == 1`),
not merely on the new field — the assertion order is deliberate, since a
read-then-filter implementation produces the same records and only
differs in work done.

### Test Output

```text
$ uv run --frozen --extra dev pytest tests/test_cli_perf_format.py \
    tests/test_proxy_dashboard_stats_cache.py tests/test_agent_savings.py -q
59 passed, 1 skipped, 1 warning in 3.36s

$ uvx ruff check headroom/perf/analyzer.py tests/test_cli_perf_format.py
All checks passed!

$ uvx ruff format --check headroom/perf/analyzer.py tests/test_cli_perf_format.py
2 files already formatted

$ uv run --frozen --extra dev mypy headroom/perf/analyzer.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: macOS 15 (arm64), Python 3.10.18, headroom-ai at
6d2254df, against a real `~/.headroom/logs` holding 54 MB across six
rotations (`proxy.log` + `.1`–`.5`) from a proxy that had been running
for weeks.
- Exact command / steps: pointed `analyzer.LOG_DIR` at the live log
directory and timed `parse_log_files(last_n_hours=1.0)` three times,
taking the median; ran it once on this branch and once with
`headroom/perf/analyzer.py` stashed back to `main`.
- Observed result: main = 0.426s median, 6 files read, 246,819 lines
parsed. This branch = 0.141s median, 2 files read, 4 skipped, 48,147
lines parsed. 3.0x faster, 80% fewer lines parsed, identical throughput
figure. The two files still read are the live log and one rotation that
had been written inside the last hour, which is correct.
- Not tested: Windows and Linux (the mtime semantics used here are
POSIX-standard and `pathlib` handles both, but I ran only macOS). No
benchmark on a log directory below the rotation ceiling — the win there
is proportionally smaller by construction, since there is less stale
history to skip.

## Runtime Rollout Safety

- Rollout-managed feature(s): none — this is a pure read-path
optimization inside the perf log parser.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: no. Windowed queries return the same
records; only the work to produce them changes. `--hours 0` is
untouched.
- Kill switch / disable path: n/a — revert the commit. There is no flag
because there is no behavior to toggle.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: single-commit revert; `PerfReport.log_files_skipped` is
a defaulted field, so no persisted or serialized data depends on it.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
 2026-08-17 20:20:56 -07:00
akothari-godaddy
7ddcbcb616
perf: surface optimization overhead diagnostics (#1212) 
## Summary
- add overhead diagnostics to perf JSON output
- report optimization p50/p95/p99, slow request percentage, per-stage
totals/percentiles, and top slow requests
- update text report and recommendations to point at the slowest stage
and HEADROOM_COMPRESSION_TIMEOUT_SECONDS when optimization is
consistently slow

## Verification
- python -m py_compile headroom/perf/analyzer.py
tests/test_cli_perf_format.py
- pytest tests/test_cli_perf_format.py could not run locally because
pytest is not installed in this Python environment
 2026-07-15 21:04:21 +00:00
Shlok Tiwari
0d89c674cd
feat: measure and surface token throughput (tokens/sec) through the proxy (#983) 
## Description

This PR implements measuring and surfacing token throughput
(tokens/second) through the proxy in the `headroom perf` CLI/analyzer
and the dashboard UI. It tracks multiple throughput metrics—Input
(wall-clock/active), Compression, Forward, and Generation
throughput—supporting both rolling percentiles (p50/p95) and current
(last 5 minutes) metrics.

Closes #959

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **Proxy Instrumentation (`headroom/proxy/outcome.py`)**: Added
`total_ms`, `tok_out`, and `ttfb_ms` to the structured `PERF` logging
payload.
- **Log Parsing & Computations (`headroom/perf/analyzer.py`)**: Updated
log parsing to read `STAGE_TIMINGS` and correlation fields from `PERF`,
computing active/wall-clock throughputs for input, compression, forward,
and generation stages.
- **API Exposing (`headroom/proxy/server.py`)**: Exposes calculated
rolling throughput percentiles and last-5-minute averages under the
`throughput` field in `/stats`.
- **Dashboard UI Layout
(`headroom/dashboard/templates/dashboard.html`)**: Refactored the
dashboard grid layout from 3 columns to 4 columns to house the new
throughput hero card showing real-time token performance.
- **Verification Tests (`tests/test_cli_perf_format.py`)**: Added test
coverage specifically targeting token throughput log parser extraction,
stage correlation, math correctness, and edge-case handling (empty
fields, division by zero).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$env:PYTHONPATH="c:\Users\hp\Desktop\Headroom_oss"; .venv\Scripts\pytest tests/test_cli_perf_format.py
============================= test session starts =============================
platform win32 -- Python 3.11.15, pytest-9.1.0, pluggy-1.6.0 -- C:\Users\hp\Desktop\Headroom_oss\.venv\Scripts\python.exe
cachedir: .pytest_cache
rootdir: C:\Users\hp\Desktop\Headroom_oss
configfile: pyproject.toml
plugins: anyio-4.13.0
collecting ... collected 14 items

tests/test_cli_perf_format.py::test_build_perf_summary_totals_and_pct PASSED [  7%]
tests/test_cli_perf_format.py::test_build_perf_summary_by_model_and_transform PASSED [ 14%]
tests/test_cli_perf_format.py::test_build_perf_summary_empty_report_no_zero_division PASSED [ 21%]
tests/test_cli_perf_format.py::test_perf_records_as_dicts_roundtrips_fields PASSED [ 28%]
tests/test_cli_perf_format.py::test_perf_json_format PASSED              [ 35%]
tests/test_cli_perf_format.py::test_perf_json_raw_is_array PASSED        [ 42%]
tests/test_cli_perf_format.py::test_perf_json_raw_preserves_client_field PASSED [ 50%]
tests/test_cli_perf_format.py::test_parse_perf_line_preserves_client_field PASSED [ 57%]
tests/test_cli_perf_format.py::test_perf_csv_by_model PASSED             [ 64%]
tests/test_cli_perf_format.py::test_perf_csv_raw_per_record PASSED       [ 71%]
tests/test_cli_perf_format.py::test_perf_text_default_unchanged PASSED   [ 78%]
tests/test_cli_perf_format.py::test_perf_rejects_unknown_format PASSED   [ 85%]
tests/test_cli_perf_format.py::test_parse_perf_line_preserves_blank_client_field PASSED [ 92%]
tests/test_cli_perf_format.py::test_throughput_parsing_and_calculations PASSED [100%]

============================== warnings summary ===============================
.venv\Lib\site-packages\_pytest\config\__init__.py:1464
  C:\Users\hp\Desktop\Headroom_oss\.venv\Lib\site-packages\_pytest\config\__init__.py:1464: PytestConfigWarning: Unknown config option: asyncio_mode
  
    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

.venv\Lib\site-packages\opentelemetry\util\_importlib_metadata.py:32
  C:\Users\hp\Desktop\Headroom_oss\.venv\Lib\site-packages\opentelemetry\util\_importlib_metadata.py:32: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.
    return EntryPoints(ep for group_eps in eps.values() for ep in group_eps)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================= 14 passed, 2 warnings in 4.77s ========================
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.11.15
- Exact command / steps: Run the pytest suite against the newly created
token throughput parsing routines:
`$env:PYTHONPATH="c:\Users\hp\Desktop\Headroom_oss";
.venv\Scripts\pytest tests/test_cli_perf_format.py`
- Observed result: The suite executes 14 tests successfully, including
the newly added `test_throughput_parsing_and_calculations` verification
test verifying mathematical precision and fallback logic.
- Not tested: None (all metrics are fully covered by unit tests in
`test_cli_perf_format.py`)

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

- Backwards compatibility: Older log outputs lacking `tok_out` or
`ttfb_ms` parse cleanly and fallback defaults prevent parser crashes.

---------

Co-authored-by: Antigravity Agent <agent@antigravity.local>
 2026-06-17 09:42:38 -05:00
Michael Sam
6d3f39f213
feat: add dashboard agent usage stats (#814) 
## Description

Add a clear dashboard view for per-agent token usage so end users can
see Cursor, Claude, Codex, and other detected clients with before/after
token counts, tokens saved, and savings percentages. The stats API now
exposes a stable `agent_usage` object that the dashboard renders near
the top of the session view.

Fixes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

### New Files

**Tests:**
- `tests/test_dashboard_agent_usage.py` — Covers agent classification,
exact per-request aggregation, and aggregate fallback behavior.

### Modified Files

- `headroom/proxy/server.py` — Adds per-agent usage aggregation to
`/stats` with before tokens, after tokens, output tokens, saved tokens,
savings percentage, source, providers, and models.
- `headroom/dashboard/templates/dashboard.html` — Adds a prominent Agent
Usage panel with totals, coverage status, per-agent token-flow bars,
request counts, before/after tokens, saved tokens, and share of savings.

## Testing

- [x] Unit tests pass: `.venv312/bin/pytest
tests/test_dashboard_agent_usage.py`
- [x] Linting passes: `.venv312/bin/ruff check headroom/proxy/server.py
tests/test_dashboard_agent_usage.py`
- [x] Diff whitespace check passes: `git diff --check
origin/main...HEAD`
- [x] Dashboard smoke render: local proxy on `127.0.0.1:8790`, captured
Chrome headless screenshot of `/dashboard`
- [x] New tests added for new functionality

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing relevant unit tests pass locally with my changes
- [ ] I have made corresponding changes to the documentation
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

The agent usage panel uses exact request-log data when available. If
detailed request logs are empty, it falls back to aggregate
provider/model request counts and labels the coverage as aggregate
fallback so users are not misled.
 2026-06-12 14:12:22 -05:00
Michael Sam
d2cdab268d
feat(proxy): add agent-90 savings profile (#830) 
## Summary
- add an `agent-90` savings profile with cross-agent proxy env exports
- wire the profile into proxy/router runtime kwargs, including
force-Kompress routing and a smaller read-protection window
- expose effective savings-profile config in `/stats` and add focused
regression coverage

## Type of change
- [x] feat (non-breaking change which adds functionality)
- [ ] fix (non-breaking change which fixes an issue)
- [ ] docs
- [ ] test/CI-only
- [ ] refactor-only

## Testing
- [x] `python3 -m py_compile headroom/agent_savings.py
headroom/cli/agent_savings.py headroom/cli/main.py headroom/cli/proxy.py
headroom/proxy/models.py headroom/proxy/server.py
headroom/transforms/content_router.py tests/test_agent_savings.py
tests/test_proxy_healthchecks.py tests/test_cli/test_wrap_persistent.py
tests/test_transforms/test_content_router.py`
- [x] `git diff --check`
- [x] manual smoke: `agent-savings --profile agent-90 --format json`
returns `HEADROOM_TARGET_RATIO=0.10`
- [x] manual smoke:
`proxy_pipeline_kwargs(ProxyConfig(savings_profile="agent-90"))` enables
`force_kompress`, system/user compression, and
`read_protection_window=2`
- [x] manual smoke: Anthropic-style `tool_result` routes through
Kompress with `target_ratio=0.10`
- [ ] `pytest` suite not run: pytest is not installed in the available
local Python environments

## Notes
This keeps agent-90 as an opt-in profile. Existing defaults remain
unchanged unless `HEADROOM_SAVINGS_PROFILE=agent-90` or
`ProxyConfig(savings_profile="agent-90")` is set.
 2026-06-11 18:58:06 -05:00
Kumario
9fe4886cf6
feat(perf): add --format {text,json,csv} to headroom perf (#648) 
* feat(perf): add structured summary/record builders to analyzer

parse_log_files() already returns a fully-structured PerfReport, but
the only way to read it was the colored text report. Add reusable
machine-readable views so CI guards, dashboards, and agent harnesses
can consume perf data without scraping ANSI text:

- build_perf_summary(report) -> dict with the aggregated KPIs
  (savings_pct, cache_hit_pct, by_model, by_transform, ...), mirroring
  format_report() numbers exactly.
- perf_records_as_dicts(report) -> per-record list for --raw output.
- PERF_RECORD_FIELDS: shared column order for CSV/raw consumers.

Pure additions; no behaviour change to existing callers. Part of #595.

* feat(perf): add --format {text,json,csv} to headroom perf

Adds a machine-readable output path to the perf command (issue #595):

- --format json: aggregated summary (default) or, with --raw, a JSON
  array of per-record dicts.
- --format csv: per-model breakdown (default) or, with --raw, one row
  per PERF record using the shared PERF_RECORD_FIELDS column order.
- --format text (default): unchanged human-readable report.

Enables CI guards (jq '.savings_pct < 70'), dashboards, and agent
wrappers to consume perf data without scraping ANSI text.

Closes #595.

* test(perf): cover --format json/csv and structured builders

Unit tests for build_perf_summary (totals, savings/cache pct,
by_model/by_transform, empty-report zero-division guard) and
perf_records_as_dicts, plus CliRunner integration tests for
--format json, json --raw, csv, csv --raw, the unchanged text
default, and rejection of an unknown format. Part of #595.

* fix(perf): rename transform loop var to satisfy mypy

The structured-summary builder reused `recs` for both the per-model
(list[PerfRecord]) and per-transform (list[TransformRecord]) groupings, so
mypy flagged the second assignment as an incompatible-type reuse
(analyzer.py:704). Rename the transform loop variable to `t_recs` so each
loop keeps a single element type. No behavior change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Kumario1 <ramsakal.ipec@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-06 22:43:10 -07:00