mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
10251b65ca
## Description Sync the docs with the code after the live-zone realignment. The `IntelligentContextManager` (ICM), `RollingWindow`, and scoring modules were deleted in PR #350 (May 2026), but the README and benchmark docstrings still advertised them as live, and an example still imported the deleted module (broken on run). This fixes the README + benchmarks and removes the dead example. I validated the README against the code with three parallel static-analysis sub-agents (features/architecture, CLI/extras/wrap-matrix, public API/integrations). Most of the README checked out accurate; only the items below were stale/wrong. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - README: removed the `IntelligentContext` bullet and `IntelligentContext / RollingWindow` from the transforms list (both deleted in PR #350). - README: standardized `Kompress-base` -> `Kompress-v2-base` to match the HF model id `chopratejas/kompress-v2-base` and the existing badges (diagram re-aligned). - README: corrected the CodeCompressor language list to match the `CodeLanguage` enum (added TS, C, Perl). - README: softened the unanchored "6 algorithms" tagline to "content-aware compressors". - README: Cortex Code is library-mode only — there is no `headroom wrap cortex`, so the compatibility-matrix row no longer shows a wrap checkmark. - Deleted `examples/test_intelligent_context_toin_ccr.py` — it imported the deleted `IntelligentContextManager` (ImportError on run) and is unreferenced. - Removed stale `RollingWindow` mentions from benchmark docstrings/comments (`benchmarks/__init__.py`, `bench_transforms.py`, `bench_latency.py`, `scenarios/conversations.py`); the accurate PR-B1 retirement comment is kept. ## Testing - [ ] Unit tests pass (`pytest`) — N/A, docs/docstring + example deletion only - [x] Linting passes — `ruff check` clean on all changed benchmark files - [ ] Type checking passes — N/A (no type-relevant changes) - [ ] New tests added — N/A - [x] Manual testing performed — see Real Behavior Proof ### Test Output ```text $ ruff check benchmarks/__init__.py benchmarks/bench_transforms.py benchmarks/bench_latency.py benchmarks/scenarios/conversations.py All checks passed! # stale refs remaining in README/benchmarks (excluding accurate retirement notes): $ grep -rn "IntelligentContext|RollingWindow|Kompress-base" README.md benchmarks/ | grep -v retire (only benchmarks/bench_transforms.py:362 — the accurate PR-B1 retirement comment) # deleted example is unreferenced anywhere: $ grep -rn "test_intelligent_context_toin_ccr" --include=*.md --include=*.yml --include=*.py . (no hits) ``` ## Real Behavior Proof - Environment: macOS (darwin, arm64), Python 3.12 `.venv`, ruff 0.14.x, repo at branch `docs/sync-readme-with-code` off latest `main`. - Exact command / steps: (1) three parallel sub-agents grep/Read-validated README claims vs `headroom/`, `pyproject.toml`, `sdk/typescript/`; (2) directly verified each flagged mismatch (`CodeLanguage` enum, `HF_MODEL_ID`, absence of `IntelligentContext`/`RollingWindow` classes); (3) confirmed the example imports a deleted module and is unreferenced; (4) `ruff check` on changed benchmark files; (5) re-grepped README + benchmarks for any remaining stale refs. - Observed result: README and benchmark docstrings now match the code; the only surviving `RollingWindow` string is the accurate retirement comment; the broken example is removed; ruff passes; the ASCII architecture diagram still aligns after the `Kompress-v2-base` rename. - Not tested: rendering of the README on GitHub/PyPI (text-only change); the separate `docs/content/` and `wiki/` doc sets (see Additional Notes — out of scope for this PR). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective — N/A (docs/example cleanup) - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A (Release Please auto-generates from the conventional commit) ## Additional Notes **Larger related finding (NOT in this PR):** the published docs site (`docs/content/docs/*.mdx`) and the `wiki/*.md` set still document `IntelligentContextManager`, `RollingWindow`, `RollingWindowConfig`, `IntelligentContextConfig`, and `ScoringWeights` as live API — with `from headroom import RollingWindow` / `from headroom.transforms import IntelligentContextManager` code examples that would `ImportError`. It is half-migrated (a couple of `.mdx` files already note "removed in 0.9.x" while neighbors still teach it as current). This is ~15 files and the fixes require rewriting examples to the live-zone model, not just deletions — recommended as a focused follow-up PR rather than bundling it here.
150 lines
5.8 KiB
Text
150 lines
5.8 KiB
Text
---
|
|
title: Benchmarks
|
|
description: Compression performance, accuracy preservation, latency overhead, and real-world production telemetry from 250+ Headroom proxy instances.
|
|
---
|
|
|
|
Headroom's core promise: compress context without losing accuracy. This page covers compression benchmarks, accuracy evaluations, latency overhead, and production telemetry.
|
|
|
|
## Compression Performance
|
|
|
|
Tested on Apple M-series (CPU), Headroom v0.5.18. Each test runs `compress()` on realistic tool outputs.
|
|
|
|
| Content Type | Original | Compressed | Saved | Ratio | Latency |
|
|
|---|---|---|---|---|---|
|
|
| JSON array (100 items) | 3,163 | 297 | 2,866 | **90.6%** | 1ms |
|
|
| JSON array (500 items) | 9,526 | 1,614 | 7,912 | **83.1%** | 2ms |
|
|
| Shell output (200 lines) | 3,238 | 469 | 2,769 | **85.5%** | 1ms |
|
|
| Build log (200 lines) | 2,412 | 148 | 2,264 | **93.9%** | 1ms |
|
|
| grep results (150 hits) | 2,624 | 2,624 | 0 | 0.0% | <1ms |
|
|
| Python source (~480 lines) | 2,958 | 2,958 | 0 | 0.0% | <1ms |
|
|
| **Total** | **23,921** | **8,110** | **15,811** | **66.1%** | **5ms** |
|
|
|
|
<Callout type="info" title="Zero compression is intentional">
|
|
grep results and Python source show 0% compression. These are already compact structured formats. SmartCrusher only compresses JSON arrays; code passes through to preserve correctness.
|
|
</Callout>
|
|
|
|
## Accuracy Benchmarks
|
|
|
|
### HTML Extraction
|
|
|
|
**Dataset**: Scrapinghub Article Extraction Benchmark (181 HTML pages with ground truth)
|
|
|
|
| Metric | Value |
|
|
|---|---|
|
|
| **F1 Score** | 0.919 |
|
|
| **Precision** | 0.879 |
|
|
| **Recall** | 0.982 |
|
|
| **Compression** | 94.9% |
|
|
|
|
For LLM applications, recall is critical -- 98.2% means nearly all article content is preserved. The slight precision drop (some extra content) does not hurt LLM accuracy.
|
|
|
|
### JSON Compression (SmartCrusher)
|
|
|
|
**Test**: 100 production log entries with critical error at position 67. Task: find the error, error code, resolution, and affected count.
|
|
|
|
| Metric | Baseline | Headroom |
|
|
|---|---|---|
|
|
| Input tokens | 10,144 | 1,260 |
|
|
| Correct answers | 4/4 | **4/4** |
|
|
| Compression | -- | **87.6%** |
|
|
|
|
SmartCrusher preserves first N items (schema), last N items (recency), all anomalies (errors, warnings), and statistical distribution.
|
|
|
|
### QA Accuracy Preservation
|
|
|
|
| Metric | Original HTML | Extracted | Delta |
|
|
|---|---|---|---|
|
|
| F1 Score | 0.85 | 0.87 | +0.02 |
|
|
| Exact Match | 60% | 62% | +2% |
|
|
|
|
<Callout type="info" title="Extraction can improve accuracy">
|
|
Removing HTML noise sometimes helps LLMs focus on relevant content, leading to slightly higher scores on extraction benchmarks.
|
|
</Callout>
|
|
|
|
## Latency Overhead
|
|
|
|
### SDK Compression Latency
|
|
|
|
Measured per-scenario on Apple M-series (CPU):
|
|
|
|
| Scenario | Tokens In | Tokens Out | Saved | p50 (ms) | p95 (ms) |
|
|
|----------|-----------|------------|-------|----------|----------|
|
|
| JSON: Search Results (100 items) | 10.2K | 1.5K | 8.7K | 189 | 231 |
|
|
| JSON: Search Results (500 items) | 50.2K | 1.5K | 48.7K | 943 | 955 |
|
|
| JSON: Search Results (1K items) | 100.5K | 1.5K | 99.0K | 2,012 | 2,198 |
|
|
| JSON: API Responses (500 items) | 38.9K | 1.1K | 37.8K | 743 | 776 |
|
|
| JSON: Database Rows (1K rows) | 43.7K | 605 | 43.1K | 961 | 1,104 |
|
|
| JSON: String Array (100 strings) | 1.1K | 231 | 820 | 15 | 15 |
|
|
| JSON: String Array (500 strings) | 4.9K | 233 | 4.6K | 72 | 80 |
|
|
| JSON: Number Array (200 numbers) | 1.2K | 192 | 1.1K | 31 | 62 |
|
|
| JSON: Mixed Array (250 items) | 2.3K | 368 | 1.9K | 38 | 40 |
|
|
|
|
### Cost-Benefit Analysis
|
|
|
|
Net latency benefit = LLM time saved from fewer tokens minus compression overhead (at Claude Sonnet pricing, $3.0/MTok):
|
|
|
|
| Scenario | Compress (ms) | LLM Saved (ms) | Net Benefit | Savings per 1K Requests |
|
|
|----------|---------------|-----------------|-------------|------------------------|
|
|
| JSON: Search Results (100 items) | 189 | 261 | **+72ms** | $26 |
|
|
| JSON: Search Results (500 items) | 943 | 1,461 | **+518ms** | $146 |
|
|
| JSON: Search Results (1K items) | 2,012 | 2,969 | **+957ms** | $297 |
|
|
| JSON: API Responses (500 items) | 743 | 1,134 | **+391ms** | $113 |
|
|
| JSON: Database Rows (1K rows) | 961 | 1,292 | **+331ms** | $129 |
|
|
|
|
Compression pays for itself in latency for 11 of 12 tested scenarios against Claude Sonnet. Slower and more expensive models (Opus) benefit even more.
|
|
|
|
### Pipeline Step Timing
|
|
|
|
| Step | Median | P90 | Description |
|
|
|------|--------|-----|-------------|
|
|
| `pipeline_total` | 16.9ms | 289ms | Full compression pipeline |
|
|
| `content_router` | 11.7ms | 259ms | Content detection + routing |
|
|
| `smart_crusher` | 50.1ms | 50ms | JSON array compression |
|
|
| `text_compressor` | 32.0ms | 576ms | Text compression (Kompress ONNX) |
|
|
| `initial_token_count` | 2.9ms | 16ms | Token counting (tiktoken) |
|
|
|
|
ContentRouter accounts for 91--98% of pipeline cost on average. CacheAligner is sub-millisecond.
|
|
|
|
## Production Telemetry
|
|
|
|
Real-world data from **50,000+ proxy sessions** across 250+ unique instances (March--April 2026). Collected via anonymous telemetry (opt-in: `HEADROOM_TELEMETRY=on`; telemetry is off by default).
|
|
|
|
### Proxy Overhead
|
|
|
|
| Percentile | Latency |
|
|
|---|---|
|
|
| **Median (P50)** | **52ms** |
|
|
| P90 | 309ms |
|
|
| P99 | 4,172ms |
|
|
| Mean | 161ms |
|
|
|
|
The median 52ms overhead is negligible compared to LLM inference time (typically 2--10 seconds).
|
|
|
|
### Compression Rate
|
|
|
|
| Percentile | Compression |
|
|
|---|---|
|
|
| P25 | 4.8% |
|
|
| **Median** | **4.8%** |
|
|
| P75 | 6.9% |
|
|
| Mean | 11.3% |
|
|
|
|
Median compression is modest because many requests are short conversational turns. Heavy tool-use sessions (file reads, shell output) see 40--80% compression.
|
|
|
|
### Fleet Summary
|
|
|
|
| Metric | Value |
|
|
|---|---|
|
|
| Clean instances | 249 |
|
|
| Total tokens saved | 1.4 billion |
|
|
| Total savings | ~$4,000 |
|
|
| OS distribution | Linux 57%, macOS 38%, Windows 5% |
|
|
|
|
## Reproducing Results
|
|
|
|
```bash
|
|
git clone https://github.com/chopratejas/headroom.git
|
|
cd headroom
|
|
pip install -e ".[evals,html]"
|
|
pytest tests/test_evals/ -v -s
|
|
```
|