2 commits

Author	SHA1	Message	Date
Rod Boev	15ac650d40	fix(proxy): fail open when kompress saturation would exhaust pre-upstream budget (#1430) ... ## Description Concurrent Anthropic `/v1/messages` traffic can still exhaust Headroom's pre-upstream budget because Kompress ONNX execution waits on the request critical path. When Kompress saturates, requests eventually fail with `503 pre-upstream queue saturated` even though compression can safely degrade to passthrough. This PR makes Kompress saturation fail open on the Anthropic hot path, so requests continue uncompressed when compression capacity is under pressure. It keeps the executor and stage-timing evidence intact, and it preserves blocking model-load validation so runtime pressure does not silently skip the validation path. Closes #1025 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - add a bounded execution-slot acquire path so Anthropic requests fail open to passthrough when Kompress saturation would consume the pre-upstream budget - preserve explicit execution-timeout counters and Anthropic passthrough/stage-timing observability instead of hiding the pressure path - keep `_validate_pytorch_device()` on blocking acquire semantics so model-load validation still waits for capacity instead of failing open - make the blocking validation acquire explicit to `mypy` without changing runtime behavior - extend focused regressions for pre-upstream backpressure, Kompress saturation, execution-skip observability, and validation waiting - align the CLI timeout help text and `ProxyConfig` comment with the fail-open runtime behavior - update `CHANGELOG.md` for the proxy runtime fix ## Testing - [x] Unit tests pass (`uv run pytest tests/test_anthropic_pre_upstream_backpressure.py tests/test_proxy_compression_executor.py tests/test_kompress_request_nonblocking.py`) - [x] Linting passes (`uv run ruff check tests/test_anthropic_pre_upstream_backpressure.py` and `uv run ruff format tests/test_anthropic_pre_upstream_backpressure.py --check`) - [x] Type checking passes (`uv run mypy headroom --ignore-missing-imports`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text Focused local validation passed: - uv run pytest tests/test_anthropic_pre_upstream_backpressure.py tests/test_proxy_compression_executor.py tests/test_kompress_request_nonblocking.py -x -v 37 passed, 1 warning in 12.01s - uv run ruff check headroom/proxy/handlers/anthropic.py headroom/transforms/kompress_compressor.py tests/test_anthropic_pre_upstream_backpressure.py tests/test_proxy_compression_executor.py tests/test_kompress_request_nonblocking.py All checks passed! - uv run ruff format headroom/proxy/handlers/anthropic.py headroom/transforms/kompress_compressor.py tests/test_anthropic_pre_upstream_backpressure.py tests/test_proxy_compression_executor.py tests/test_kompress_request_nonblocking.py --check 5 files already formatted - uv run mypy headroom --ignore-missing-imports Success: no issues found in 398 source files Base-branch proof on origin/main (fa05ebc849): - test_acquire_timeout_degrades_to_passthrough fails because the handler still returns 503 - test_saturation_fail_open_does_not_hang_request fails because get_kompress_execution_stats() does not exist - test_compression_executor_skip_signal_remains_visible passes on base too, so it stays as compatibility coverage rather than the failing-then-passing proof for this fix Review-follow-up validation passed after aligning the timeout wording with fail-open behavior: - uv run pytest tests/test_anthropic_pre_upstream_backpressure.py -x -v 20 passed, 1 warning in 1.38s - uv run ruff check headroom/cli/proxy.py headroom/proxy/models.py headroom/proxy/handlers/anthropic.py headroom/transforms/kompress_compressor.py tests/test_anthropic_pre_upstream_backpressure.py tests/test_proxy_compression_executor.py tests/test_kompress_request_nonblocking.py All checks passed! ``` ## Real Behavior Proof - Environment: local Anthropic pre-upstream and Kompress execution regression harnesses covering the `/v1/messages` hot path - Exact command / steps: run the focused pytest command above on `origin/main` and on this branch, including the semaphore-saturation path in `test_saturation_fail_open_does_not_hang_request` and the validation-slot hold in `test_validation_probe_waits_for_execution_slot` - Observed result: the reviewed head no longer returns `503` on the pre-upstream pressure path, request-thread Kompress saturation degrades to passthrough while incrementing execution timeout stats, and model-load validation still waits for capacity instead of failing open - Not tested: wrap/install fallout mentioned in the original issue ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new type-check or lint warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - Scoped to the runtime queue-pressure fault only; the issue's wrap/unwrap and deployment complaints stay out of this PR. - `test_compression_executor_skip_signal_remains_visible` remains in the suite to prove the skip signal stays visible, but it is compatibility coverage rather than the failing-then-passing regression for the bug fix. - Local validation included `uv run mypy headroom --ignore-missing-imports` after the explicit validation-acquire narrowing was added for CI parity. - Attribution: the issue body isolated the hot-path ONNX compression stall and the pre-upstream saturation symptom that this PR fixes.	2026-06-30 13:41:22 -05:00
xspawnnn	3fc2a78a5e	fix(kompress): never block the request path on the cold-cache model download (#1161) ... Closes #1146. ## Problem On a cold cache, the first request that reaches the Kompress deep compressor triggers an inline `hf_hub_download` of the 274 MB `chopratejas/kompress-v2-base` ONNX model **on the request thread**. That download races the proxy's compression budget (`HEADROOM_COMPRESSION_TIMEOUT_SECONDS`, default 30s — the `compression_first_stage` timeout): the fetch is cancelled mid-transfer, **nothing finalizes in the HF cache**, and the request fails open (uncompressed). Because the partial blob never lands, every subsequent request repeats the same ~30s hang + fail-open, so the deep compressor never actually becomes available through the proxy. This is a **distinct root cause from #946** (which concerns the timeout itself). Here the model must simply never be fetched synchronously on a latency-sensitive request. ## Fix Make the request path cache-only and move the one-time download off-thread. **`kompress_compressor.py`** - `compress(..., allow_download=False)` — new keyword (default `True`, so the direct API and `compress_batch` are unchanged) that resolves the model cache-only; on a cold cache it raises `KompressModelNotCached` and passes through instead of blocking on the network. - `is_ready()` — lockless cache-membership check, safe to call on the hot path. - `ensure_background_download(model_id, device)` — starts at most one daemon thread per model to pull the artifact down out of band (a finished/failed thread is replaced, so a transient failure can be retried by a later request). The compression timeout does not bound this thread. **`content_router.py`** — gate the deep path on readiness: - not ready → return passthrough immediately and kick off the background download; - ready → `compress(allow_download=False)` (cache-only, no network on the request thread). Net effect: the cold-cache deep path returns in ~0 ms (passthrough) instead of hanging ~30 s; the model downloads once in the background; subsequent requests transparently use the deep compressor once it is cached. ## Verification Clean install of `headroom-ai==0.26.0` (main `@9f7f3ad` + this patch), HF cache empty: - **Cold-cache router request**: returns in **56 ms**, passthrough (output == input), fetch handed to a background daemon thread — vs. the pre-fix inline hang. - **A/B on the same `compress()` call** with a slow-fetch stand-in: pre-fix (`allow_download=True`) blocked **24.00 s** on the request thread; post-fix (`allow_download=False`) returned **59 ms**. - **4 new regression tests** in `tests/test_kompress_request_nonblocking.py` (cache-only passthrough; one-thread-per-model background download; router skips deep path when not ready; router stays cache-only when ready) — `4 passed`. ## Files - `headroom/transforms/kompress_compressor.py` — non-blocking load + cache-only `compress` - `headroom/transforms/content_router.py` — gate deep path on `is_ready()`; background fetch when cold - `tests/test_kompress_request_nonblocking.py` — regression coverage --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-19 11:33:05 -05:00