mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
4 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Abhay Singh
|
c5a08d22e0
|
fix(proxy): time-cap the compression timeout-debt quarantine (#2360) (#2412)
## Description Fixes #2360. The proxy runs compression on a bounded thread-pool executor with a per-request deadline. Because Python cannot preempt a worker after its `asyncio.wait_for` times out, the code quarantines new compression while a timed-out worker is still running (`_compression_timed_out_in_flight > 0`), to avoid piling more work onto a saturating executor. The gap: that counter only decrements when the worker finally exits. A worker that **never returns** — a hung or pathological compression of a large frame — keeps the counter above zero forever, so the quarantine stays open permanently and every subsequent compression raises `CompressionQuarantinedError`. On Codex WS this is exactly what #2360 reports: one 5s timeout, then Token Savings pinned at ~0% with no recovery, even though the machine is fine. The "parity with direct upstream" nature of the accounting was correct; the only missing piece is an upper bound on how long a single stuck worker may hold the quarantine. ## Fix Add a time cap on the quarantine: - A deadline (`_compression_quarantine_deadline`) is (re)armed on every fresh timeout, to `now + HEADROOM_COMPRESSION_QUARANTINE_MAX_SECONDS` (default **60s**). - The gate quarantines only while `timed_out_in_flight > 0` **and** `now < deadline`. Once the deadline lapses with no new timeouts, the worker is presumed leaked/abandoned and compression resumes. The release is counted once (a `"released"` quarantine metric + a warning), and the deadline is cleared so it is not re-counted on every later request. - The bounded executor still caps thread growth, and any new timeout re-arms the quarantine, so ongoing genuine slowness keeps quarantining while a single hung worker cannot pin it forever. This preserves the original protection (a burst of slow compressions still quarantines) while guaranteeing recovery. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/server.py`: add `_compression_quarantine_deadline` / `_compression_quarantine_max_seconds` (from `HEADROOM_COMPRESSION_QUARANTINE_MAX_SECONDS`, default 60s) and `_compression_quarantine_releases`; arm the deadline when timeout debt is recorded; release the quarantine (once) in the gate when the deadline lapses. - `tests/test_platform_stabilization_functional.py`: add a test that a standing timed-out worker quarantines within the cap and releases (running compression again, counted once) past it. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uvx ruff@0.15.17 check headroom/proxy/server.py tests/test_platform_stabilization_functional.py All checks passed! $ uvx mypy@1.20.2 --ignore-missing-imports headroom/proxy/server.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17` / `uvx mypy@1.20.2`. A full `pytest` here imports the ML stack and OOMs this box, so I modeled the gate/deadline state machine with a dependency-free script and left the added `create_app` test to CI. - Exact command / steps: simulated a standing timed-out worker, then exercised the gate at times within the cap, past the cap, and after a fresh timeout, plus a normal worker exit. - Observed result: within the cap the gate quarantines (raises); past the cap it releases exactly once and then lets compression run; a new timeout re-arms the quarantine; a normal worker exit clears the debt. Matching the added handler test (`_run_compression_in_executor` raises `CompressionQuarantinedError` within the cap and returns the callable's result past it, with `_compression_quarantine_releases == 1`). - Not tested: a live Codex WS session hanging a real worker; the added test drives `_run_compression_in_executor` directly with the quarantine state set. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes The default cap (60s) is deliberately well above a normal slow-but-completing compression so the original saturation protection is unchanged in practice; it only ever fires for a worker that has run far past its deadline. Tunable via `HEADROOM_COMPRESSION_QUARANTINE_MAX_SECONDS`. The `"released"` quarantine metric and a one-time warning make the recovery observable. The "unit tests pass locally" box is unchecked because the added `create_app` test imports the ML stack (OOM on this box); it runs under the normal CI job, and the state machine is verified by the standalone proof above. |
||
|
Alex Sun
|
685ebe457d
|
fix(ccr): report embedded hashes from compress endpoint (#717)
## Description
Fixes `/v1/compress` so its `ccr_hashes` response includes retrievable
CCR hashes embedded in compressed message content, including row-drop
and recursive JSON markers that may not be present in
`TransformResult.markers_inserted`.
The original PR also changed query-based JSON row search. Current `main`
intentionally made CCR retrieval a hash-only, full-content lookup in
#1532, so that obsolete half is not restored. This reconciliation
preserves the reporting bug fix without reversing the current retrieval
contract.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that causes existing functionality
to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- extract 12–24 character CCR hashes from `Retrieve more`, `Retrieve
original`, and `<<ccr:...>>` markers
- scan both transform marker metadata and nested rendered message values
- preserve stable encounter order and deduplicate case-insensitively
- exclude non-retrieval transform metadata such as tool digests and
stable-prefix hashes
- return the normalized hashes from `/v1/compress`
- add helper-level and endpoint-level regression coverage
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual behavior inspection performed
### Test Output
```text
$ uv run --extra dev pytest tests/test_proxy_compress_endpoint.py -q
50 passed, 1 warning in 6.54s
$ uv run --extra dev ruff check headroom/proxy/handlers/openai.py tests/test_proxy_compress_endpoint.py
All checks passed!
$ uv run --extra dev ruff format --check headroom/proxy/handlers/openai.py tests/test_proxy_compress_endpoint.py
2 files already formatted
$ git diff --check
# no output
```
## Real Behavior Proof
- Environment: macOS, Python 3.12.13, current `main` at `
|
||
|
Tejas Chopra
|
f2c48e26c6
|
feat(compress): reach the lossless provider seam on the general path and default /v1/compress to marker-free output (#2691)
## Description Two related changes to the compression seams, plus the review fixes for both. Supersedes #2661 and #2662, which are closed in favour of this branch — the fixes are inseparable from the code they fix, so reviewing them together is cheaper than landing two PRs and patching them afterwards. **1. A registered lossless provider now competes on the general path.** The `headroom.transforms.lossless_provider` seam was only ever consulted from `_lossless_compact_excluded`, gated on `DEFAULT_EXCLUDE_TOOLS` (`config.py:216` — `Read/Grep/Glob/Write/Edit/WebSearch/WebFetch`). Gateway traffic carries the caller's own tool names — LiteLLM's `headroom` guardrail (https://docs.litellm.ai/docs/proxy/headroom) posts requests containing tools like `search_docs` / `run_ci` / `fetch_rows` — so a registered provider was structurally unreachable for every gateway/sidecar deployment. The seam existed; nothing could get to it. **2. `POST /v1/compress` is marker-free by default.** A CCR marker is only useful to a caller that also injects the `headroom_retrieve` tool AND can reach `/v1/retrieve`. Neither holds here: tool injection lives in the provider request handlers (`handlers/anthropic.py:1894`), never in `handle_compress`; and every `/v1/retrieve*` route is `Depends(_require_loopback)` (`server.py:4422, 4470, 4749, 4781`) with no remote opt-in — `HEADROOM_COMPRESS_ALLOW_REMOTE` drops the loopback dependency on `/v1/compress` only. So a gateway forwards a `Retrieve more: hash=…` pointer the model cannot follow, and the proxy pays a CCR store write nobody reads. `config.mode="ccr"` opts back in. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [x] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made ### Seam — `content_router.py`, `lossless_provider.py` - `_lossless_first` (STAGE 0, every block on every path) consults `get_lossless_provider()` and keeps whichever output is smaller. **Strict no-op when no provider is registered**, which is the default; and because it is best-of rather than authoritative, a provider can never do worse than the built-in folds. - Malformed provider output can no longer escape. Every shape check runs inside the `try`: result must be `None`, or a 2-element tuple/list of two `str`. Anything else is ignored at debug level. (Previously the unpack sat outside the `try`, so a 3-tuple raised `ValueError` up through `TransformPipeline.apply`, which re-raises.) - Empty / whitespace-only candidates are rejected rather than silently replacing block content. - **Providers are never offered diff content.** Diff folding is subtractive with no inverse check and a reflowed hunk breaks `git apply` — the same reason the built-in `diff` fold is restricted at `content_router.py:2481`. - The third-party `kind` label is sanitised against `^[a-z0-9_]{1,32}$` before reaching `transforms_applied` and the per-strategy metric dicts, so a caller-controlled string cannot explode Prometheus label cardinality. `fullmatch`, not `match`: `$` also matches before a trailing newline, which would put a newline in a label. - `set_lossless_provider(provider, *, verifier=None)` — in lossless-only mode, where STAGE 0's output is final and there is no marker to recover from, a registered verifier must confirm the fold or the candidate is dropped. No verifier registered = today's behaviour. `provider=None` clears both. - The provider is invoked once per block, not twice (`_has_lossless_fold` probes `_lossless_first` and discards the result, then STAGE 0 recomputes). Bounded memo, wholesale clear on overflow, no lock — a race costs one redundant fold. The memo keys on the provider registration generation, so registering or clearing a provider after a block was already folded takes effect. - The seam docstring now records that the provider runs on the general path and inside the parallel compression pool, so it must be thread-safe as well as deterministic. ### Route — `handlers/openai.py`, `server.py` - `_derived_compress_pipeline(key, **overrides)` replaces the copy-pasted pipeline-derivation block; `_no_ccr_pipeline` (the new default) and `_lossy_inline_pipeline` both use it. - The default pipeline is **built at startup** and included in `_eager_preload_transforms`, so a fresh pod does not pay ContentRouter construction and compressor load on its first request, inside the compression-executor budget. - An unrecognised `config.mode` returns 400 naming the valid values instead of silently falling back to the default. - Claude-family model names resolve their context limit from the Anthropic provider. Real divergence: `bedrock/anthropic.claude-3-5-sonnet` is 200000 there and 128000 on the OpenAI provider. The tokenizer still comes from the OpenAI pipeline's provider — a separate, larger change, noted in a comment. - Documents why the derived router deliberately does **not** share the base router's compression cache: keys do not encode CCR-marker mode, so sharing would leak marker-laden entries into the marker-free path. ### Behavior change A `/v1/compress` caller that relied on default markers now gets none. The only in-tree caller that can resolve them is the TypeScript SDK (`sdk/typescript/src/client.ts:398 retrieve`, `:422 handleToolCall`); it needs `config: {"mode": "ccr"}` to keep today's behaviour, and landing that SDK default in the same release would leave only gateway callers — for whom markers were never resolvable — seeing a difference. No `HEADROOM_COMPRESS_DEFAULT_MODE` compat env deliberately: a flag nobody sets becomes permanent debt, and the wire-level `mode` already covers the one caller that needs it. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_lossless_first_dispatch.py tests/test_lossless_excluded_compaction.py \ tests/test_lossless_mode.py tests/test_lossless_then_lossy.py tests/test_lossless_diff_fold_guard.py \ tests/test_bash_search_lossless_fold.py tests/test_proxy_compress_endpoint.py \ tests/test_ccr_row_drop_store_bridge.py tests/test_gateway_sidecar_ports.py tests/test_compress_api.py \ tests/test_platform_stabilization_functional.py tests/test_proxy_eager_preload_bind.py \ tests/test_proxy_warmup.py tests/test_router_registry_smartcrusher.py -q 189 passed in 32.04s $ ruff check <all 9 changed files> All checks passed! $ ruff format --check <all 9 changed files> 9 files already formatted $ mypy headroom/transforms/content_router.py headroom/transforms/lossless_provider.py \ headroom/proxy/handlers/openai.py headroom/proxy/server.py Success: no issues found in 4 source files ``` New tests cover, one concern each: every malformed provider shape; empty and whitespace-only results; diff content never reaching a provider (call-recording); `kind` sanitisation including the trailing-newline case; the verifier accepting / rejecting / raising; clearing a provider clearing its verifier; single provider invocation per block; memo invalidation on re-registration; unknown and valid `mode` values; the default pipeline existing before any request; and Claude vs OpenAI context-limit resolution with `token_budget` precedence preserved. ## Real Behavior Proof - **Environment:** macOS arm64, Python 3.12.6, proxy built from `_proxy_config_from_env()` with the default `coding` savings profile; Kompress both disabled and offloaded to a remote `kompress-v2-base` endpoint; `HEADROOM_COMPRESSION_TIMEOUT_SECONDS=300`. - **Steps:** `POST /v1/compress` over `TestClient` with OpenAI-shaped payloads under non-excluded tool names (`run_ci`, `list_files`, `code_search`, `fetch_rows`) — a CI log with ANSI escapes and repeated lines, a 160-path listing, a 150-line grep dump, a 150-row JSON array; plus a second payload with a RAG user blob, a 200-row JSON tool result and a 300-line log. Ran with and without a provider registered via `set_lossless_provider`. - **Observed — seam reachability:** | Kompress | no provider registered | provider registered | |---|---|---| | off | 19,284 → 10,265 tokens (46.8%) | 19,284 → **7,879 (59.1%)** | | on (remote) | 19,284 → 9,366 tokens (51.4%) | 19,284 → **7,146 (62.9%)** | Before this change the right-hand column was identical to the left — the registered provider was never called on this payload. - **Observed — marker-free default costs nothing:** | Config | tokens | saved | |---|---|---| | markers on (previous default) | 37,791 → 24,415 | 35.4% | | markers off (new default) | 37,791 → 24,415 | **35.4% — identical** | | `mode="lossy_inline"` | 37,791 → 25,129 | 33.5% | | `--lossless` | 37,791 → 35,100 | 7.1% | - **Observed — memo staleness, before the fix:** registering a provider that folds a grep block to 5 bytes left the block at its 1496-byte built-in fold, and clearing a provider kept serving the provider's output. Both correct after keying on the registration generation. - **Observed — context limit:** `bedrock/anthropic.claude-3-5-sonnet` resolves 200000 via the Anthropic provider, 128000 via the OpenAI provider. - **Not tested:** `tests/test_transforms_content_router.py` was not run — it does not complete on this machine, wedging on its 5th test while that test passes in 5.7s alone. Verified pre-existing before this work: with the diff stashed, the clean tree stalled at the identical test, and it stalled the same way under `HF_HUB_OFFLINE=1 HEADROOM_OFFLINE=1`. The machine was also out of disk at the time, which may be the real cause rather than the suspected native-detector deadlock (#575) — worth a separate issue either way. CI should be the arbiter here. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes Two pre-existing test files needed adjusting, both direct consequences rather than scope creep: - `test_platform_stabilization_functional.py::test_v1_compress_success_reports_actual_metrics` patches `openai_pipeline.apply`, which the default mode no longer routes through. **This test was already failing on the marker-free-default commit** before any of the fixes — my original test selection missed it. It now patches the pipeline the route actually uses. - `test_proxy_eager_preload_bind.py` substitutes fake pipelines to control exactly what the preload walks; the eager build injected the real derived router's statuses into an exact-equality assertion. Its shared helper now clears the derived cache, preserving each test's intent without weakening an assertion. Docs unchecked — follow-ups worth doing in the same release: document `config.mode` values in `docs/content/docs/litellm.mdx` and `wiki/proxy.md`; the TS SDK `mode:"ccr"` default; and an operational note that `COMPRESSION_TIMEOUT_SECONDS` defaults to 30 (`helpers.py:687`) while a remote ML endpoint makes one sequential call per unit — on a large payload it trips the executor timeout and the handler fails open, returning `compression_skipped: true` with `tokens_before: 0`, which reads as "nothing to save" rather than "we gave up". Those zeroed counters are misleading and worth a separate fix. |
||
|
JD Davis
|
1d2b76e72e
|
fix: harden persistent install startup (#1851)
## Description Hardens persistent install startup and proxy compression behavior for issue #1843. Repeated `headroom install start` / scheduled ensure calls no longer spawn duplicate runtimes by default, and `/v1/compress` now fails open on compression timeout instead of returning a 503. The PR also adds a machine-readable platform feature matrix and app-level stabilization tests for health, compression functionality, timeout behavior, and matrix evidence. Refs #1843 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Wrapped direct persistent deployment starts with the existing profile-local runtime start lock. - Made `headroom install start` idempotent when the deployment is already healthy. - Added wedged-runtime handling: if a PID is running but `/readyz` does not recover inside the grace window, stop it before starting again. - Kept `install agent ensure` inside the already-held lock while delegating to the shared start helper. - Changed `/v1/compress` timeout behavior from `503 compression_timeout` to fail-open `200` with original messages, `compression_skipped: true`, and `skip_reason: compression_timeout`. - Added `tests/test_platform_stabilization_functional.py` covering real FastAPI health/compression routes, successful compression metrics, timeout fail-open speed, and a real JSON tool payload that reduces tokens. - Added `docs/platform-feature-matrix.json` and `docs/platform-stabilization.md` for Linux/macOS/Windows hardening coverage and known gaps. - Strengthened matrix tests so cited local test/workflow paths must exist. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # Clean source tree without copied Rust extension: functional module is skipped locally, as CI copies _core from the built wheel. > python -m pytest tests/test_install tests/test_cli/test_install_cli.py tests/test_platform_feature_matrix.py tests/test_platform_stabilization_functional.py -q collected 120 items / 1 skipped 119 passed, 2 skipped in 17.08s > python -m ruff check headroom/proxy/handlers/openai.py tests/test_platform_stabilization_functional.py tests/test_platform_feature_matrix.py All checks passed! # Local Windows compiled-core proof: > python -m maturin build --profile ci --out dist-local Built wheel for abi3 Python >= 3.10 to dist-local\headroom_ai-0.29.0-cp310-abi3-win_amd64.whl # Copied _core.pyd from the wheel into headroom/ for local route execution, then: > python -m pytest tests/test_platform_stabilization_functional.py -q collected 4 items 4 passed in 6.71s > python -m pytest tests/test_install tests/test_cli/test_install_cli.py tests/test_platform_feature_matrix.py -q collected 120 items 119 passed, 1 skipped in 17.16s Commit hooks: Sync plugin versions.....................................................Passed check for merge conflicts................................................Passed ruff.....................................................................Passed ruff-format..............................................................Passed mypy.....................................................................Passed ``` ## Real Behavior Proof - Environment: Windows 11, PowerShell, Python 3.13.13, worktree `C:\git\headroom-stabilization` on branch `jd/cross-platform-stabilization`. - Exact command / steps: built the Windows wheel with `maturin`, extracted `_core.pyd`, ran the new FastAPI route tests and install/matrix tests listed above, then removed generated artifacts before committing. - Observed result: direct start paths now no-op when healthy, skip spawning when the start lock is contended, and stop a wedged runtime before restart. `/v1/compress` now returns original messages quickly on timeout instead of a 503. The real JSON tool-payload smoke test returns `tokens_before > tokens_after`, `tokens_saved > 0`, `compression_ratio < 1.0`, and non-empty transforms through the public route. - Not tested: full native Windows persistent process e2e remains blocked by the upstream CRT/wheel issue already documented in workflows and in the matrix. No real OS service was installed locally; service manager behavior is covered by argument-level unit tests. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes CHANGELOG is not updated because this is an unreleased hardening/test/documentation pass. The platform matrix intentionally records partial/blocked Windows/macOS e2e gaps instead of claiming full coverage where the repo cannot currently run it. |