2 commits

Author	SHA1	Message	Date
Sebastian Schkudlara	517bf992cf	fix(proxy): quarantine compression while timed-out workers run (#2292) ... ## Description A request-side `asyncio.wait_for()` timeout stops waiting, but it cannot preempt an executor thread that already started. The proxy counted those late workers and still admitted more compression, so repeated slow calls could consume the whole compression pool and charge every request another full timeout. This change tracks running post-timeout workers as timeout debt and quarantines request-path compression while that debt is non-zero. New attempts raise `CompressionQuarantinedError` before executor admission, using an `asyncio.TimeoutError` subclass so Python 3.10 handlers apply the existing compression-failure policy. Quarantine clears automatically after all known timed-out workers genuinely exit. Mitigates #946 and #810. It does not attempt to kill the first running thread; Python cannot safely preempt it. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Track started, finished, timed-out, and debt-recorded state under the existing compression metrics lock. - Reject new compression before enqueue while timed-out workers remain; clear quarantine on the final worker exit. - Preserve queued-timeout behavior: work cancelled before worker start does not activate quarantine; a cancellation/start race is conservatively tracked as running debt. - Add `/health` and `/stats` runtime fields for quarantine state, worker debt, activations, and skips. - Add `headroom_compression_quarantine_total{event="activated|skipped"}` Prometheus counters. - Add regression, recovery, queue-race, runtime-payload, export, reset, and Python 3.10 exception-class coverage. - Update `CHANGELOG.md`; no dependency or lockfile changes. ## Reproduction On base commit `718c8dc5`, I applied only the new regression test and ran: ```bash .venv/bin/pytest -q \ tests/test_proxy_compression_executor.py::test_timeout_quarantines_new_work_until_timed_out_worker_finishes ``` The first worker timed out but remained blocked. The second callable entered the executor instead of being rejected: ```text FAILED: DID NOT RAISE TimeoutError ``` ## Testing - [x] Affected unit tests pass - [x] Linting passes (`ruff check .`) - [x] Changed-file type checking passes - [x] New tests added for the fix - [x] Manual testing performed ### Test Output ```text Python 3.10.20 $ .venv/bin/pytest -q -m 'not slow' \ tests/test_proxy_compression_executor.py \ tests/test_prometheus_obs_counters.py \ tests/test_proxy/test_compression_failure_action.py \ tests/test_proxy/test_compression_timeout_config.py \ tests/test_anthropic_pre_upstream_backpressure.py \ tests/test_openai_codex_ws_lifecycle.py \ tests/test_codex_ws_compression_scheduler.py \ tests/test_gemini_compression_offload.py \ tests/test_proxy_handlers_batch.py \ tests/test_tokenizer_count_offload.py \ tests/test_cold_start_fast_pass.py 125 passed, 1 skipped, 1 deselected, 1 warning in 11.06s $ .venv/bin/ruff check . All checks passed! $ .venv/bin/ruff format --check . 1310 files already formatted $ .venv/bin/mypy headroom/proxy/server.py headroom/proxy/prometheus_metrics.py Success: no issues found in 2 source files $ git diff --check # no output ``` The warning is the existing Starlette `TestClient`/`httpx` deprecation warning. ## Real Behavior Proof - Environment: macOS 15.7.4 x86_64, Python 3.10.20, `compression_max_workers=2`, direct proxy executor path, no external provider/model. - Exact command / steps: instantiate the proxy; run a blocking compression callable with a 50 ms timeout; immediately attempt a second callable and time the rejection; release the first worker; wait for debt to reach zero; run the second callable again; export Prometheus metrics. - Observed result: the first request timed out at 51.182 ms; the second attempt was rejected in 0.014 ms and its callable never started; debt was 1 while quarantined, returned to 0 after release, and compression then resumed normally. ```json { "after_release": { "activations_total": 1, "leaked_threads_total": 1, "quarantine_active": false, "skips_total": 1, "timed_out_workers": 0 }, "bypass_elapsed_ms": 0.014, "bypass_error": "compression quarantined: 1 timed-out worker(s) still running", "during_quarantine": { "quarantine_active": true, "timed_out_workers": 1 }, "first_timeout_elapsed_ms": 51.182, "prometheus": [ "headroom_compression_quarantine_total{event=\"activated\"} 1", "headroom_compression_quarantine_total{event=\"skipped\"} 1" ], "resumed_result": "resumed", "second_callable_started_during_quarantine": false } ``` - Not tested: a live external model/provider; forced termination of a permanently wedged native worker; the marked slow native scheduler benchmark. A broad non-slow run collected 9,859 selected tests but was stopped at `tests/test_adversarial_grid.py::TestRunGrid::test_grid_shape_and_schema` after a macOS process sample showed the pre-existing native `_core.abi3.so` semaphore stall (`_dispatch_semaphore_wait_slow` → `semaphore_wait_trap`). The affected executor/handler slice above completed cleanly. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented hard-to-understand concurrency paths - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove the fix is effective - [x] New and affected existing unit tests pass locally - [x] I have updated the CHANGELOG.md --------- Co-authored-by: Tejas Chopra <chopratejas@gmail.com>	2026-07-16 14:30:06 -07:00
Rocker Zhang	d7283387ac	feat(metrics): export compression-failed and kompress size-gate counters (#1569) ... ## What Two Prometheus counters that make previously-invisible compression behavior measurable on `/metrics`. - `headroom_compression_failed_total{reason=timeout|error}` — incremented at both Anthropic fail-open sites (single-message and batch-create), where an optimization exception forwards the request uncompressed. Before this, ratio could bleed at these sites with nothing in `/metrics`; only a response header recorded the single-message case. The timeout/error split separates "compression budget too tight" from "real bug". - `headroom_kompress_size_gate_total{outcome=within|exceeded}` — the size gate (#1171) routes oversized blocks off ModernBERT. The within/exceeded split proves whether the gate ever fires on real traffic. `within` counts a gate pass, not whether ML compression then ran. ## How Both reuse the existing `PrometheusMetrics` singleton and the established `defaultdict(int)` counter + text-exposition pattern. The handler records via `self.metrics`; `content_router` records through the existing `CompressionObserver` hook to avoid an import cycle. Cleared in `reset_runtime`; exposition blocks are emitted only when non-empty. ## Verification - `tests/test_prometheus_obs_counters.py` (6 tests): per-reason/outcome bucketing, empty-string default buckets, exposition format, conditional absence until recorded, and reset clearing. All green. - Counter increments and well-formed exposition (HELP/TYPE balanced, labels escaped) confirmed by direct exercise; gate `within`/`exceeded` shown mutually exclusive across the eligible-block call sites. Single commit, rebased on current `main`. Addresses #1567.	2026-07-14 13:25:24 -04:00