mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
3 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Tejas Chopra
|
224578e80b
|
fix(kompress): reject artifacts that fail at run, and prefetch model files at startup (#2740)
## Description Three cold-start / robustness gaps found while debugging a user report of **0.12% savings across 722 requests** (49.8M input tokens, 60,920 saved). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made ### 1. The artifact fallback was unreachable for run-time failures `_create_onnx_session` tries `int8-wo` → `fp32` → `int8`, and its docstring describes exactly this scenario — but it only skipped a candidate when `InferenceSession(...)` **construction** threw. The int8 weight-only artifact carries `MatMulNBits` with `bits=8`. ORT's CPU kernel only handles 8-bit through the prepacked MLAS path, so a build or ISA without an 8-bit `SQNBitGemm` kernel falls into `ComputeBUnpacked`, which hard-asserts `nbits_ == 4`. That raises on `session.run()` **after** construction succeeded — so the fp32 candidate was never reached and ML compression was dead for the process lifetime. The reported log has 207 consecutive failures over three days. A two-token `_smoke_run` inside the existing candidate loop makes the fallback fire. `onnxruntime>=1.16.0` is unpinned, so which side of this an install lands on is a lottery. ### 2. A broken model cost an inference on every request, forever The per-request handler logged a `WARNING` and passed through with no latch — 207 identical lines that read as noise rather than "ML compression is dead". Now latches to passthrough after **3 consecutive** failures (any success resets the count) with one actionable `ERROR` naming the artifact override. ### 3. The model download began on the first request, not at startup #2001 was right to move Kompress off the startup path — on RHEL/CentOS 7-family hosts, entering cached native init before the port binds segfaults in `libarrow`/jemalloc with no Python traceback (#1908), which no `try/except` can catch. **This PR does not touch that.** But #2001 left the ~4-minute *download* on the first request, with every request in that window silently uncompressed behind one "model not ready" warning. Downloading is separable from loading. `prefetch_kompress_artifacts` resolves the files over plain `huggingface_hub` HTTP and never constructs an `InferenceSession` or imports `transformers`, so startup can prefetch bytes without touching the boundary #1908 crashes on. Native load stays deferred, status stays `deferred`, and a test asserts no session is constructed during prefetch. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ .venv/bin/ruff check headroom/ tests/test_kompress_failsafe.py tests/test_kompress_preload_deferral.py --exclude headroom/dashboard/templates All checks passed! $ .venv/bin/mypy headroom/ Success: no issues found in 508 source files $ python -m pytest tests/test_kompress_failsafe.py tests/test_kompress_preload_deferral.py \ tests/test_kompress_request_nonblocking.py tests/test_force_kompress_all.py \ tests/test_kompress_must_keep.py tests/test_proxy_disable_kompress.py \ tests/test_proxy_per_provider_kompress.py tests/test_proxy_warmup.py \ tests/test_proxy_eager_preload_bind.py -q 95 passed in 10.51s ``` ## Real Behavior Proof - **Environment:** macOS 26.4 arm64, Python 3.12.6, onnxruntime 1.21.1, repo `.venv`. **(1) Fallback chain, against the real HF repo:** ```text WARNING ONNX artifact 'onnx/kompress-int8-wo.onnx' from chopratejas/kompress-v2-base is unusable (... nbits_ == 4 was false ...); trying next candidate SESSION OK -> ['input_ids', 'attention_mask'] SMOKE RUN OK on the selected artifact ``` Also confirmed the default artifact really is 8-bit, by loading the cached blob: `{'bits': [8], 'block_size': [128]}`. **(2) Files-only prefetch, with `InferenceSession` patched to raise:** ```text INFO Kompress: prefetching model artifacts for chopratejas/kompress-v2-base ... prefetch ok=True in 0.08s, no session constructed ``` - **Not tested / important caveat:** the user's exact failure **cannot be reproduced on this machine**. On ORT 1.21.1 arm64 the int8-wo artifact fails at *construction* (`matmul_nbits.cc:115`), which the pre-existing load-only fallback already caught. Their build fails at *execution* (`matmul_nbits.cc:442`, `ComputeBUnpacked`). So the run-time path is pinned with a fake ORT session that constructs fine and then rejects `run()` — a mechanism test, not a reproduction of their build. Confirming the fix on their host needs their `onnxruntime` version. - **Not tested:** no RHEL/CentOS 7 host available to re-verify #1908 non-regression; the argument is structural (prefetch never constructs a session) and asserted by `test_prefetch_never_constructs_a_session_or_imports_transformers`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Rod Boev
|
5bd2266f16
|
fix(kompress): raise the default execution-slot wait (#2456)
## Description Concurrent Kompress requests currently fail open after a 25 ms execution-slot wait even though ordinary ONNX inference can hold the single slot for hundreds of milliseconds. This raises the existing default wait to 3000 ms while retaining concurrency one, the `HEADROOM_KOMPRESS_EXECUTION_TIMEOUT_MS` override, the tighter acquire and request budgets, and passthrough after a genuine timeout. The reproduction and validated 3000 ms setting come from https://github.com/headroomlabs-ai/headroom/issues/2451 Closes #2451 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Raise the default Kompress execution-slot wait from 25 ms to 3000 ms. - Start the Kompress request deadline at call entry and carry it through single-item acquire, single-to-batch delegation, and sequential-fallback lineage. - Cap the raised execution-slot wait by that live request deadline on both single-item and batch acquire paths. - Keep the per-backend default concurrency at one and preserve all tighter time budgets. - Add queued single-item, batch, request-deadline, carried-deadline lineage, and router-watchdog lifecycle regressions at the same owner layer that currently fails. - Preserve the explicit short-timeout fail-open path. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_kompress_failsafe.py tests/test_kompress_request_nonblocking.py tests/test_content_router_single_item_deadline.py tests/test_transforms/test_kompress_deadline.py -v`) - [x] Linting passes (`uv run ruff check headroom/transforms/kompress_compressor.py tests/test_kompress_failsafe.py tests/test_content_router_single_item_deadline.py tests/test_transforms/test_kompress_deadline.py`) - [x] Formatting passes (`uv run ruff format headroom/transforms/kompress_compressor.py tests/test_kompress_failsafe.py tests/test_content_router_single_item_deadline.py tests/test_transforms/test_kompress_deadline.py --check`) - [x] New regression tests prove the saturation fix - [ ] Manual testing performed ### Test Output ```text uv run pytest tests/test_kompress_failsafe.py tests/test_kompress_request_nonblocking.py tests/test_content_router_single_item_deadline.py tests/test_transforms/test_kompress_deadline.py -v 37 passed in 4.22s uv run ruff check headroom/transforms/kompress_compressor.py tests/test_kompress_failsafe.py tests/test_content_router_single_item_deadline.py tests/test_transforms/test_kompress_deadline.py All checks passed! uv run ruff format headroom/transforms/kompress_compressor.py tests/test_kompress_failsafe.py tests/test_content_router_single_item_deadline.py tests/test_transforms/test_kompress_deadline.py --check 4 files already formatted ``` ## Real Behavior Proof - Environment: worktree Python environment from `uv sync --extra dev`, focused pytest with real Python threads and `threading.BoundedSemaphore` - Exact command / steps: hold the sole execution slot with the environment override unset, start queued single-item and batch compression workers, wait until each worker proves it reached a blocked acquire on the shared execution semaphore, release the slot, rerun the explicit 1 ms timeout preservation case, then set `HEADROOM_COMPRESSION_DEADLINE_MS=10` and repeat the held-slot single-item and batch acquires plus a router single-cache-miss run whose Kompress load sleeps past the request deadline. - Observed result: The queued single-item and batch workers each proved a real blocked acquire before release, then acquired after release and compressed, while `HEADROOM_KOMPRESS_EXECUTION_TIMEOUT_MS=1` still passed through promptly, the 10 ms request deadline capped the raised default wait so both held-slot paths failed open before 200 ms without reaching model inference, the single-to-batch and sequential-fallback lineage regressions proved later branches inherit the original request start instead of resetting it, and the router lifecycle proof showed the carried deadline now allows slow Kompress load to start but still expires before model inference after the outer request has already failed open. - Not tested: live ONNX proxy savings under sustained concurrent load ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes `CHANGELOG.md` stays unchanged because the release pipeline generates changelog entries from conventional commits. The fail-open path from #1430 stays intact; this change stops it from firing spuriously under ordinary queueing. |
||
|
Tejas Chopra
|
36202f4d0b
|
fix(windows): unwedge compression on degraded ONNX runtimes (every request timing out at 30s, 0% savings) (#822)
## Summary Multiple Windows users reported (via Discord, on v0.23.0, `pip install "headroom-ai[all]"`) that the proxy delivers **zero compression** and adds **+30s latency to every request**: `Optimization failed: TimeoutError:` with `compression_first_stage ≈ 30000ms` on every optimization attempt, for the lifetime of the process. Log analysis showed the wedge starts at the **first message eligible for real compression** (earlier requests succeed because everything is skipped/excluded) — and never recovers, even though the Kompress model loaded successfully at startup. ### Root cause chain 1. `create_cpu_session_options` disabled ONNX Runtime's CPU memory arena on **all** platforms. On Windows this is catastrophic: every `Run()` falls back to per-node `VirtualAlloc`/free, slowing ModernBERT inference by 2–3 orders of magnitude (onnxruntime#11627). One reporter's perf summary showed max optimization overhead of **200,369ms** (~13 chunks × ~15s) — slow, not deadlocked. 2. The first slow inference outlives the proxy's 30s compression-stage timeout. `asyncio.wait_for` abandons the future but **cannot kill the executor thread**, which keeps holding the Kompress `BoundedSemaphore(1)`. 3. Every later compression blocks on an **unbounded** `semaphore.acquire()`, times out at exactly 30s, and leaks another thread — permanently wedging the proxy until restart. Two adjacent Windows bugs found in the same logs are fixed too: `subprocess.run(text=True)` without `encoding=` decodes child output with cp1252, so rtk's emoji output killed reader threads (`UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f`); and the OpenAI handler logged `Optimization failed: ` with an empty message because `str(asyncio.TimeoutError())` is empty. ### Fixes - **`onnx_runtime.py`** — keep the CPU arena at ORT's default on Windows; Linux/macOS keep the legacy low-RSS behavior (arena disabled) bit-for-bit. New `HEADROOM_ONNX_CPU_ARENA` env overrides either way. All ONNX sessions (Kompress, image router, memory embedders) share this helper, so one fix covers them all. - **`kompress_compressor.py`** — three layers of wedge-proofing, each fail-safing to passthrough instead of blocking: - bounded semaphore acquire (`HEADROOM_KOMPRESS_ACQUIRE_TIMEOUT_SECONDS`, default 5s) - wall-clock budget per compress/compress_batch call (`HEADROOM_KOMPRESS_TIME_BUDGET_SECONDS`, default 20s — under the 30s stage timeout, so Kompress gives up before the request is abandoned). Batch bail never emits a partially-covered text. - preload canary (`HEADROOM_KOMPRESS_CANARY_SECONDS`, default 5s, one retry to forgive cold-start warmup): machines that can never finish inference inside the stage timeout get ML compression disabled up front with one actionable warning, instead of a guaranteed 30s timeout per request. - Setting any knob `<= 0` disables that guard (restores legacy behavior). First give-up logs at WARNING with remediation hints; repeats drop to DEBUG. - **`proxy/helpers.py`, `interceptors/astgrep.py`** — `encoding="utf-8", errors="replace"` on rtk/lean-ctx/ast-grep subprocess calls. - **`handlers/openai.py`** — failure log now includes request id + exception type, matching the Anthropic handler. ### Non-Windows perf - Session options on Linux/macOS are unchanged (pinned by tests). - The only new hot-path cost is one `time.monotonic()` + a bounded acquire per chunk: micro-benchmarked at sub-microsecond (bounded acquire measured marginally *faster* than the old context-manager acquire), vs 50–500ms of inference per chunk. - Real-model smoke run on macOS: identical compression output (ratio 0.262 on a 1020-word sample), canary passes, budget/acquire give-up paths verified against the real ONNX stack by forcing tiny env values. Related (same symptom, different root cause — **not** addressed here): #810 tracks the blocked-tiktoken-download hang, which produces the same per-request 30s `TimeoutError` signature. The bounded-acquire/budget changes in this PR limit the blast radius of Kompress-side slowness only. ## Validation - `.venv/bin/ruff check headroom/ tests/...` — clean - `.venv/bin/ruff format --check` — clean (355 files) - `.venv/bin/mypy` on all five changed source files — no issues - `python -m pytest tests/test_onnx_runtime.py tests/test_kompress_failsafe.py tests/test_subprocess_encoding.py` — 25 passed (new coverage: arena platform matrix + env overrides, stuck-semaphore passthrough for compress and batch, budget bail incl. mid-batch no-data-loss, canary trip/pass/retry/disable/error-safety, UTF-8 subprocess kwargs) - `python -m pytest tests/test_transforms_content_router.py tests/test_proxy_handler_helpers.py tests/test_codex_ws_compression_scheduler.py tests/test_proxy_warmup.py tests/test_proxy_pipeline_lifecycle.py` — 52 passed (existing suites for touched areas) <!-- headroom-maintainer-template-completion:start --> ## Description This PR prepares `fix(windows): unwedge compression on degraded ONNX runtimes (every request timing out at 30s, 0% savings)` for review by documenting the intended change, validation evidence, and remaining merge-readiness context. Linked issues: None declared. ## Type of Change - [x] Bug fix - [ ] New feature - [ ] Documentation - [ ] Refactor - [ ] Tests only ## Changes Made - Commit: fix(windows): unwedge compression on degraded ONNX runtimes - Commit: fix(kompress): run preload canary off the startup path - Touches `headroom/onnx_runtime.py` - Touches `headroom/proxy/handlers/openai.py` - Touches `headroom/proxy/helpers.py` - Touches `headroom/proxy/interceptors/astgrep.py` - Touches `headroom/transforms/kompress_compressor.py` - Touches `tests/test_kompress_failsafe.py` ## Testing - [x] GitHub checks reviewed - [x] Metadata/template validation - [ ] Local functional testing ### Test Output ```text gh pr view 822 --repo chopratejas/headroom --json statusCheckRollup - CI / changes: SUCCESS - CodeQL / Analyze (actions): SUCCESS - Evaluation Suite / smoke-test: SUCCESS - Init E2E / docker-init-e2e: SUCCESS - PR Governance / label: SUCCESS - Wrap E2E / docker-wrap-e2e: SUCCESS - CodeQL / Analyze (c-cpp): SUCCESS - CodeQL / Analyze (javascript-typescript): SUCCESS - CodeQL / Analyze (python): SUCCESS - CodeQL / Analyze (rust): SUCCESS - Evaluation Suite / weekly-suite: SKIPPED - CI / commitlint: SUCCESS ``` ## Real Behavior Proof - Environment: GitHub PR metadata and checks for `chopratejas/headroom` PR #822. - Exact command / steps: Reviewed PR title, commits, changed files, linked issues, labels, and check rollup; appended this maintainer template completion block without replacing the author's original description. - Observed result: PR body now contains all required governance sections, checked readiness fields, and a non-placeholder validation evidence block. - Not tested: This pass updated PR metadata only; code validation remains represented by the linked GitHub checks and any author-provided evidence above. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review <!-- headroom-maintainer-template-completion:end --> --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |