Mirror of headroomlabs-ai/headroom (AI context compression proxy)
Find a file
gglucass 841663da16
fix(proxy): make Kompress eager preload cache-only so a cold cache can't block startup (#783)
## Description

`ContentRouter.eager_load_compressors()` runs a network
`hf_hub_download` of the Kompress ONNX model on the **blocking
startup/lifespan path**, before the proxy binds its port. On a cold
cache this is unsafe:

- the download can hang long enough to blow the supervisor's bind
timeout, or
- a native crash in the download/ML stack (an **uncatchable `Fatal
Python error: Aborted` / SIGABRT**) kills the interpreter before it ever
`listen()`s.

Either way the supervisor sees "proxy never opened its port" and gives
up. We observed this in the field from the desktop app (process aborted
during `eager_load_compressors -> _load_kompress_onnx ->
hf_hub_download` of `onnx/kompress-int8.onnx`, while the only Python
thread was parked in the HuggingFace download file-lock; the abort came
from a native thread, so `try/except` at the call site cannot catch it).

The eager preload is a latency optimization and must never be able to
block — or kill — startup. This change makes startup preload
**cache-only**: if the model isn't already cached, we defer the download
to first use (off the startup path) and bind the port normally. Warm
starts are unchanged.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `onnx_runtime.hf_hub_download_local_first(...)`: added `allow_network`
(default `True`). When `False`, a cache miss re-raises the local-lookup
error instead of falling back to a network download.
- `kompress_compressor`: added `allow_download` (default `True`)
threaded through `preload()` -> `_load_kompress()` ->
`_load_kompress_onnx()` / `_load_kompress_pytorch()` and the ModernBERT
tokenizer load. Added `KompressModelNotCached`, raised when a cache-only
load misses. Auto-mode no longer falls back to a PyTorch network
download on a cache-only miss — it propagates so the caller can defer.
- `content_router.eager_load_compressors()`: calls
`preload(allow_download=False)`. On `KompressModelNotCached` it logs and
reports the component as `"deferred"` (a status
`warmup.merge_transform_status` already handles gracefully) instead of
letting a cold download run on the startup path.

Default (first-request) loading behavior and warm-start preload are
unchanged.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

New tests in `tests/test_kompress_preload_deferral.py` cover: cache-only
`hf_hub_download_local_first` never hits the network; default still
falls back; cache-only ONNX load raises `KompressModelNotCached`;
auto-mode does **not** trigger a PyTorch download on a cache-only miss;
and `eager_load_compressors` reports `deferred` (cold) / `enabled`
(warm). Existing `_load_kompress` dispatch tests updated for the new
keyword-only param.

> Note on environment: I do not have a clean reproduction of the native
SIGABRT itself (it depends on a specific machine's HF download/ML native
stack), so the "Manual testing performed" box is left unchecked. The
tests target the structural fix — that startup preload can no longer
perform a network download — which is the precondition for the crash.

## Test Output

```
$ uv run pytest -v tests/test_kompress_preload_deferral.py
tests/test_kompress_preload_deferral.py::test_local_first_no_network_when_disallowed PASSED
tests/test_kompress_preload_deferral.py::test_local_first_falls_back_to_network_by_default PASSED
tests/test_kompress_preload_deferral.py::test_load_kompress_onnx_cache_miss_raises_not_cached PASSED
tests/test_kompress_preload_deferral.py::test_load_kompress_auto_does_not_pytorch_download_on_cache_miss PASSED
tests/test_kompress_preload_deferral.py::test_eager_load_defers_when_model_not_cached PASSED
tests/test_kompress_preload_deferral.py::test_eager_load_enabled_when_model_cached PASSED
6 passed in 4.82s

$ uv run pytest tests/test_transforms/test_kompress_compressor.py tests/test_transforms_content_router.py tests/test_onnx_runtime.py tests/test_proxy_warmup.py
63 passed

$ uv run ruff check <changed files>            # All checks passed!
$ uv run mypy headroom/onnx_runtime.py headroom/transforms/kompress_compressor.py headroom/transforms/content_router.py
Success: no issues found
```

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable (auto-generated from
conventional commits)

## Additional Notes

This contains the cold-start case. A native crash in onnxruntime
*session init* (as opposed to the download) on first request would still
be a separate issue; it is not what was observed here (the abort was
during the HF download), and isolating it would be a larger, separate
change.

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-11 12:53:03 -05:00
.claude-plugin ci(release): align manifest + pyproject + package.json to 0.22.3 2026-05-25 18:41:38 -07:00
.devcontainer fix(transforms): use thread-local tree-sitter parsers to prevent pyo3 Unsendable panic (#604) 2026-06-10 18:30:00 -05:00
.github ci: limit commitlint to pull requests (#843) 2026-06-10 22:39:47 -05:00
benchmarks fix: B2 — live-zone block dispatcher skeleton 2026-05-02 12:45:43 -07:00
crates ci: bump pyo3 from 0.22.6 to 0.24.1 in the cargo group across 1 directory (#270) 2026-06-10 23:01:33 -05:00
docker feat: add differential network capture harness (#761) 2026-06-08 22:18:31 -07:00
docs fix(ccr): make retrieval TTL configurable (#715) 2026-06-10 23:20:46 -05:00
e2e feat(proxy): per-project savings breakdown on the dashboard (claude, codex, aider, copilot, cursor) (#803) 2026-06-10 21:04:45 -05:00
examples docs: fix env var discrepancies across markdown files 2026-05-29 15:04:13 +05:30
headroom fix(proxy): make Kompress eager preload cache-only so a cold cache can't block startup (#783) 2026-06-11 12:53:03 -05:00
plugins feat: add OAuth2 client-credentials upstream-auth proxy extension (#778) (#784) 2026-06-11 11:42:25 -05:00
REALIGNMENT docs: add Realignment plan (40 PRs, 9 phases) 2026-05-01 23:34:46 -07:00
scripts fix(proxy): lazy-import server to avoid fastapi crash (#442) 2026-06-10 12:44:23 -05:00
sdk/typescript chore: release main (#607) 2026-06-08 11:21:23 -07:00
sql feat(telemetry): add headroom_stack and install_mode identity fields 2026-04-17 17:12:38 +02:00
tests fix(proxy): make Kompress eager preload cache-only so a cold cache can't block startup (#783) 2026-06-11 12:53:03 -05:00
wiki feat(kompress): warn on unrecognized HEADROOM_KOMPRESS_BACKEND + document backend selection (#204) 2026-06-10 20:13:17 -05:00
.actrc feat: add act testing config, fix gitignore, make workflow production-ready 2026-04-15 20:28:29 -05:00
.actrc.local.example feat: add act testing config, fix gitignore, make workflow production-ready 2026-04-15 20:28:29 -05:00
.changelog.md fix: use /tmp for changelog artifact to avoid . file matching issues 2026-04-15 22:54:34 -05:00
.commitlintrc.json ci: fix smart_crusher branch CI failures + add make ci-precheck pre-push gate 2026-04-27 11:13:47 -07:00
.dockerignore chore: normalize line endings in init diffs 2026-04-21 20:17:14 -05:00
.env.act.example feat: add act testing config, fix gitignore, make workflow production-ready 2026-04-15 20:28:29 -05:00
.env.example fix(security): patch loopback guard, retry None raise, blocking subprocess, and cache stats race 2026-06-04 16:09:53 -04:00
.git-blame-ignore-revs chore: add .git-blame-ignore-revs 2026-04-24 15:35:29 +02:00
.gitattributes chore: enforce LF checkout for Python files 2026-04-23 08:43:03 -05:00
.gitguardian.yaml fix(security): allowlist GitGuardian-flagged test fixtures 2026-05-02 18:33:22 -07:00
.gitignore feat: switch Kompress default to kompress-v2-base with weight-only int8 ONNX (#799) 2026-06-09 23:28:40 -07:00
.pre-commit-config.yaml fix(proxy): lazy-import server to avoid fastapi crash (#442) 2026-06-10 12:44:23 -05:00
.release-please-config.json ci(release-please): set versioned PR title pattern to fix tagging jam 2026-06-04 09:56:35 -07:00
.release-please-manifest.json chore: release main (#607) 2026-06-08 11:21:23 -07:00
Cargo.lock ci: bump pyo3 from 0.22.6 to 0.24.1 in the cargo group across 1 directory (#270) 2026-06-10 23:01:33 -05:00
Cargo.toml ci: bump pyo3 from 0.22.6 to 0.24.1 in the cargo group across 1 directory (#270) 2026-06-10 23:01:33 -05:00
CHANGELOG.md fix(learn): claude-cli streams output with idle timeout (#373) 2026-06-11 11:55:19 -05:00
claude_analysis_ttl.py chore: add cache TTL cost analysis script 2026-05-13 10:49:17 -07:00
CODE_OF_CONDUCT.md Prepare for OSS release v0.2.0 2026-01-07 11:36:44 -08:00
codecov.yml ci: scope codecov patch coverage 2026-04-22 00:05:47 -05:00
CONTRIBUTING.md chore: wire pre-commit ruff hooks into make install-git-hooks (#786) 2026-06-09 23:09:22 -05:00
deny.toml feat(rust): scaffold workspace + parity harness (phase-0) 2026-04-24 13:39:48 -07:00
docker-bake.hcl refactor(docker): migrate to bake with multi-variant distroless images 2026-04-04 21:51:37 +05:30
docker-compose.yml chore(deps): loosen over-pinned constraints and add upper bounds (#538) 2026-06-08 22:06:24 -08:00
Dockerfile chore(deps): loosen over-pinned constraints and add upper bounds (#538) 2026-06-08 22:06:24 -08:00
ENTERPRISE.md docs: add enterprise.md 2026-06-04 15:36:47 -07:00
Headroom-2.gif Add demo GIF to README 2026-01-20 18:57:17 -08:00
headroom-savings.png docs: rewrite README for clarity and highlight Kompress-base, leaderboard, RTK 2026-04-18 09:36:49 -07:00
headroom_learn.gif docs: add headroom learn demo GIF to README 2026-03-07 17:56:17 -08:00
HeadroomDemo-Fast.gif Replace demo GIF with HeadroomDemo-Fast.gif 2026-04-10 15:05:25 -07:00
LICENSE Prepare for OSS release v0.2.0 2026-01-07 11:36:44 -08:00
llms.txt docs: fix stale API references, retired class imports, and incorrect examples 2026-06-02 19:19:19 -04:00
Makefile fix: A0 — fail-loud rust core deployment smoke test 2026-05-02 17:52:37 -07:00
mkdocs.yml feat: add Vertex AI proxy routing (#793) 2026-06-09 23:05:30 -07:00
NOTICE Prepare for OSS release v0.2.0 2026-01-07 11:36:44 -08:00
PR.md fix: restore release and compress regressions 2026-04-18 16:01:57 -05:00
pyproject.toml fix(transforms): use thread-local tree-sitter parsers to prevent pyo3 Unsendable panic (#604) 2026-06-10 18:30:00 -05:00
README.md docs(install): document corporate SSL-inspection workaround (#735) (#775) 2026-06-11 11:01:52 -05:00
rust-toolchain.toml fix(rust): clippy 1.95 unnecessary_sort_by + pin toolchain 2026-04-27 12:11:49 -07:00
RUST_DEV.md docs: fix env var discrepancies across markdown files 2026-05-29 15:04:13 +05:30
SECURITY.md Prepare for OSS release v0.2.0 2026-01-07 11:36:44 -08:00
TESTING-copilot-subscription.md fix(copilot): restore generic endpoint for non-subscription OAuth (#610) (#612) 2026-06-04 16:27:54 -07:00
uv.lock ci: bump the uv group across 1 directory with 17 updates (#832) 2026-06-10 18:20:43 -05:00

  ██╗  ██╗███████╗ █████╗ ██████╗ ██████╗  ██████╗  ██████╗ ███╗   ███╗
  ██║  ██║██╔════╝██╔══██╗██╔══██╗██╔══██╗██╔═══██╗██╔═══██╗████╗ ████║
  ███████║█████╗  ███████║██║  ██║██████╔╝██║   ██║██║   ██║██╔████╔██║
  ██╔══██║██╔══╝  ██╔══██║██║  ██║██╔══██╗██║   ██║██║   ██║██║╚██╔╝██║
  ██║  ██║███████╗██║  ██║██████╔╝██║  ██║╚██████╔╝╚██████╔╝██║ ╚═╝ ██║
  ╚═╝  ╚═╝╚══════╝╚═╝  ╚═╝╚═════╝ ╚═╝  ╚═╝ ╚═════╝  ╚═════╝ ╚═╝     ╚═╝
                  The context compression layer for AI agents

6095% fewer tokens · library · proxy · MCP · 6 algorithms · local-first · reversible

CI codecov PyPI npm Model: Kompress-v2-base License: Apache 2.0 Docs

Docs · Install · Proof · Agents · Discord · llms.txt · Enterprise

AI agents / LLMs: read /llms.txt here, or fetch the live index / full docs blob.


chopratejas%2Fheadroom | Trendshift

Headroom compresses everything your AI agent reads — tool outputs, logs, RAG chunks, files, and conversation history — before it reaches the LLM. Same answers, fraction of the tokens.

Headroom in action
Live: 10,144 → 1,260 tokens — same FATAL found.

What it does

  • Librarycompress(messages) in Python or TypeScript, inline in any app
  • Proxyheadroom proxy --port 8787, zero code changes, any language
  • Agent wrapheadroom wrap claude|codex|cursor|aider|copilot in one command
  • MCP serverheadroom_compress, headroom_retrieve, headroom_stats for any MCP client
  • Cross-agent memory — shared store across Claude, Codex, Gemini, auto-dedup
  • headroom learn — mines failed sessions, writes corrections to CLAUDE.md / AGENTS.md
  • Reversible (CCR) — originals are cached for retrieval on demand

How it works (30 seconds)

 Your agent / app
   (Claude Code, Cursor, Codex, LangChain, Agno, Strands, your own code…)
        │   prompts · tool outputs · logs · RAG results · files
        ▼
    ┌────────────────────────────────────────────────────┐
    │  Headroom   (runs locally — your data stays here)  │
    │  ────────────────────────────────────────────────  │
    │  CacheAligner  →  ContentRouter  →  CCR            │
    │                    ├─ SmartCrusher   (JSON)        │
    │                    ├─ CodeCompressor (AST)         │
    │                    └─ Kompress-base  (text, HF)    │
    │                                                    │
    │  Cross-agent memory  ·  headroom learn  ·  MCP     │
    └────────────────────────────────────────────────────┘
        │   compressed prompt  +  retrieval tool
        ▼
 LLM provider  (Anthropic · OpenAI · Bedrock · …)
  • ContentRouter — detects content type, selects the right compressor
  • SmartCrusher / CodeCompressor / Kompress-base — compress JSON, AST, or prose
  • CacheAligner — stabilizes prefixes so provider KV caches actually hit
  • CCR — stores originals locally; LLM calls headroom_retrieve if it needs them

Architecture · CCR reversible compression · Kompress-v2-base model card

Get started (60 seconds)

# 1 — Install
pip install "headroom-ai[all]"          # Python
npm install headroom-ai                 # Node / TypeScript

# 2 — Pick your mode
headroom wrap claude                    # wrap a coding agent
headroom proxy --port 8787              # drop-in proxy, zero code changes
# or: from headroom import compress      # inline library

# 3 — See the savings
headroom perf

Granular extras: [proxy], [mcp], [ml], [code], [memory], [relevance], [image], [agno], [langchain], [evals]. Requires Python 3.10+.

Proof

Savings on real agent workloads:

Workload Before After Savings
Code search (100 results) 17,765 1,408 92%
SRE incident debugging 65,694 5,118 92%
GitHub issue triage 54,174 14,761 73%
Codebase exploration 78,502 41,254 47%

Accuracy preserved on standard benchmarks:

Benchmark Category N Baseline Headroom Delta
GSM8K Math 100 0.870 0.870 ±0.000
TruthfulQA Factual 100 0.530 0.560 +0.030
SQuAD v2 QA 100 97% 19% compression
BFCL Tools 100 97% 32% compression

Reproduce: python -m headroom.evals suite --tier 1 · Full benchmarks & methodology

Star History Chart

Agent compatibility matrix

Agent headroom wrap Notes
Claude Code --memory · --code-graph
Codex shares memory with Claude
Cursor prints config — paste once
Aider starts proxy + launches
Copilot CLI starts proxy + launches
OpenClaw installs as ContextEngine plugin

Any OpenAI-compatible client works via headroom proxy. MCP-native: headroom mcp install.

GitHub Copilot CLI subscription mode

Headroom can route GitHub Copilot CLI subscription traffic through the local proxy:

headroom wrap copilot --subscription -- --model gpt-4o

This lets Headroom intercept OpenAI-compatible Copilot CLI requests and apply the same proxy compression pipeline before forwarding to GitHub Copilot's hosted API. The wrapper resolves the account-specific Copilot API endpoint and prints it as COPILOT_PROVIDER_API_URL=... during launch.

Platform support note: macOS auth reuse via Copilot CLI Keychain storage has been smoke-tested. Windows Credential Manager, Linux Secret Service / secret-tool, and Docker/CI token-injection paths are implemented or planned as auth-discovery paths, but still need real OS validation before they should be considered fully vetted. For Docker and CI, prefer passing an explicit GITHUB_COPILOT_TOKEN or GITHUB_COPILOT_GITHUB_TOKEN rather than relying on host keychain access.

When to use · When to skip

Great fit if you…

  • run AI coding agents daily and want savings without changing your code
  • work across multiple agents and want shared memory
  • need reversible compression — originals are retrievable via CCR within the configured TTL

Skip it if you…

  • only use a single provider's native compaction and don't need cross-agent memory
  • work in a sandboxed environment where local processes can't run
Integrations — drop Headroom into any stack
Your setup Hook in with
Any Python app compress(messages, model=…)
Any TypeScript app await compress(messages, { model })
Anthropic / OpenAI SDK withHeadroom(new Anthropic()) · withHeadroom(new OpenAI())
Vercel AI SDK wrapLanguageModel({ model, middleware: headroomMiddleware() })
LiteLLM litellm.callbacks = [HeadroomCallback()]
LangChain HeadroomChatModel(your_llm)
Agno HeadroomAgnoModel(your_model)
Strands Strands guide
ASGI apps app.add_middleware(CompressionMiddleware)
Multi-agent SharedContext().put / .get
MCP clients headroom mcp install
What's inside
  • SmartCrusher — universal JSON: arrays of dicts, nested objects, mixed types.
  • CodeCompressor — AST-aware for Python, JS, Go, Rust, Java, C++.
  • Kompress-base — our HuggingFace model, trained on agentic traces.
  • Image compression — 4090% reduction via trained ML router.
  • CacheAligner — stabilizes prefixes so Anthropic/OpenAI KV caches actually hit.
  • IntelligentContext — score-based context fitting with learned importance.
  • CCR — reversible compression; LLM retrieves originals on demand.
  • Cross-agent memory — shared store, agent provenance, auto-dedup.
  • SharedContext — compressed context passing across multi-agent workflows.
  • headroom learn — plugin-based failure mining for Claude, Codex, Gemini.
Pipeline internals

Headroom exposes one stable request lifecycle across compress(), the SDK, and the proxy:

SetupPre-StartPost-StartInput ReceivedInput CachedInput RoutedInput CompressedInput RememberedPre-SendPost-SendResponse Received

  • Transforms do the work: CacheAligner, ContentRouter, SmartCrusher, CodeCompressor, Kompress-base, IntelligentContext / RollingWindow.
  • Pipeline extensions observe or customize lifecycle stages via on_pipeline_event(...).
  • Compression hooks sit alongside the canonical lifecycle as an additional extension seam.
  • Proxy extensions remain the server/app integration seam for ASGI middleware, routes, and startup policy.

Provider and tool-specific behavior lives under headroom/providers/ so core orchestration stays focused on lifecycle, sequencing, and policy.

  • CLI/tool slices: headroom/providers/claude, copilot, codex, openclaw
  • Provider runtime slices: headroom/providers/claude, gemini, plus shared backend/runtime dispatch in headroom/providers/registry.py
  • Core files stay orchestration-first: wrap.py, client.py, cli/proxy.py, and proxy/server.py delegate provider-specific env shaping, API target normalization, backend selection, and transport dispatch.

Install

pip install "headroom-ai[all]"          # Python, everything
npm install headroom-ai                 # TypeScript / Node
docker pull ghcr.io/chopratejas/headroom:latest

Granular extras: [proxy], [mcp], [ml] (Kompress-base), [code], [memory], [relevance], [image], [agno], [langchain], [evals]. Requires Python 3.10+.

Using pipx? Choose a supported interpreter explicitly:

pipx install --python python3.13 "headroom-ai[all]"

Installation guide — Docker tags, persistent service, PowerShell, devcontainers.

Corporate / SSL-inspection environments

If pip install "headroom-ai[all]" fails with CERTIFICATE_VERIFY_FAILED (unable to get local issuer certificate), your network uses SSL inspection — a MITM proxy presenting a company-issued CA. The build backend (maturin) downloads rustup over a connection your TLS stack doesn't trust. Install Rust first so the build doesn't fetch it:

# macOS / Linux
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh && rustup default stable
# Windows
winget install Rustlang.Rustup && rustup default stable

Restart your shell, then pip install "headroom-ai[all]". A prebuilt wheel avoids the Rust build entirely where available: pip install --only-binary headroom-ai headroom-ai.

Two runtime assets are fetched over TLS; if they are blocked, trust your corporate CA via REQUESTS_CA_BUNDLE / SSL_CERT_FILE / CURL_CA_BUNDLE:

  • cdn.pyke.io — the ONNX Runtime for the Rust core. Alternatively pre-provide it with ORT_STRATEGY=system and ORT_LIB_LOCATION=/path/to/onnxruntime.
  • huggingface.co — the kompress-base compression model. Pre-download it and run with HF_HUB_OFFLINE=1, or set HF_ENDPOINT to a trusted mirror.

Running with compression disabled (pure gateway) requires neither asset.

headroom learn

headroom learn in action

headroom learn — mines failed sessions, writes corrections to CLAUDE.md / AGENTS.md / GEMINI.md.

Documentation

Start here Go deeper
Quickstart Architecture
Proxy How compression works
MCP tools CCR — reversible compression
Memory Cache optimization
Failure learning Benchmarks
Configuration Limitations

Compared to

Headroom runs locally, covers every content type, works with every major framework, and is reversible.

Scope Deploy Local Reversible
Headroom All context — tools, RAG, logs, files, history Proxy · library · middleware · MCP Yes Yes
RTK CLI command outputs CLI wrapper Yes No
lean-ctx CLI commands, MCP tools, editor rules CLI wrapper · MCP Yes No
Compresr, Token Co. Text sent to their API Hosted API call No No
OpenAI Compaction Conversation history Provider-native No No

Attribution. Headroom ships with the excellent RTK binary for shell-output rewriting — git show --short, scoped ls, summarized installers. Huge thanks to the RTK team; their tool is a first-class part of our stack, and Headroom compresses everything downstream of it. Headroom can also use lean-ctx as the selected CLI context tool; set HEADROOM_CONTEXT_TOOL=lean-ctx before running headroom wrap ....

Contributing

git clone https://github.com/chopratejas/headroom.git && cd headroom
pip install -e ".[dev]" && pytest

Devcontainers in .devcontainer/ (default + memory-stack with Qdrant & Neo4j). See CONTRIBUTING.md.

Community

License

Apache 2.0 — see LICENSE.