mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
5 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Radhakrishnan Pachyappan
|
89493714d2
|
fix(health): label kompress as degraded/optional when not yet loaded (#2865)
## Description `/readyz` reports kompress as `"status": "unhealthy"` while the top-level payload simultaneously reports `"status": "healthy"` and `"ready": true`. This is a visible contradiction — kompress is intentionally excluded from the aggregate readiness gate, but it still receives the harshest label when it hasn't finished loading. This PR is a superset of #2829: it makes the same `degraded` status change **and** adds an `"optional": true` field to the component dict so API consumers can distinguish optional components from gating ones without parsing the `status` string. Fixes #2813. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/server.py` — `_component_health()` accepts `optional: bool = False`; when `optional=True` and not-ready, status is `"degraded"` instead of `"unhealthy"`; `"optional": True` is added to the returned dict so callers can identify optional components without parsing the status string. Kompress call passes `optional=True`. - `tests/test_proxy_health.py` — All 11 kompress assertion dicts updated: `"status": "degraded"` for not-ready cases and `"optional": True` for all kompress cases (covering disabled/healthy/degraded states in the full parametrized matrix). ## Schema diff **Before** (kompress not yet loaded): ```json { "enabled": true, "ready": false, "status": "unhealthy", "backend": null } ``` **After**: ```json { "enabled": true, "ready": false, "status": "degraded", "optional": true, "backend": null } ``` The `"optional": true` field is additive — existing consumers that only check `status` are unaffected. The field gives consumers a stable machine-readable signal without requiring them to enumerate which component names are optional. ## Testing - [x] Unit tests pass (`pytest`) — CI only; `headroom._core` (compiled Rust extension) is not available locally, blocking direct `pytest tests/test_proxy_health.py` locally. All tests that don't import through `headroom.proxy.server → headroom.transforms → headroom._core` run locally. - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality (existing tests updated to cover the new status value and the new `"optional"` field) - [ ] Manual testing performed ### Test Output ``` $ uv run ruff check headroom/proxy/server.py tests/test_proxy_health.py All checks passed! $ uv run mypy headroom/proxy/server.py Success: no issues found in 1 source file ``` Full test suite (`tests/test_proxy_health.py`) is verified by CI; local run blocked by missing `headroom._core` native extension. ## Real Behavior Proof - Environment: local dev checkout, Windows 11, Python 3.14.3 - Ruff + mypy pass locally on both changed files (see Test Output above) - `tests/test_proxy_health.py` test suite requires `headroom._core` (compiled Rust extension not available locally) — CI run covers this - Diff is a mechanical expansion of the same `optional` flag already approved in #2829's head, plus the additive `"optional": true` response field ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title --------- Signed-off-by: Radhakrishnan Pachyappan <radhakrishnan.p@op.tech> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
Parideboy
|
3a27c4dacb
|
fix(proxy/debug): reconcile Kompress warmup state in /debug/warmup (#2711)
## Description
`/debug/warmup` serialized the warmup registry verbatim, so a Kompress
slot left at the startup snapshot kept reporting `{"status": "null",
"info": {"source_status": "deferred"}}` forever — even while the ONNX
model was loaded and actively compressing.
`/health` and `/readyz` already fix this: #2402 added
`_reconcile_kompress_health()`, which promotes the slot from live
runtime state. The debug route never called it, so its answer depended
on whether a health probe happened to run first. That is the half of
#2624 still reproducing on `main`.
Second defect: `WarmupSlot.mark_loaded()` only *updates* `info`, so the
startup-planted `source_status: "deferred"` survived promotion and the
slot serialized as the self-contradictory `{"status": "loaded", "info":
{"source_status": "deferred", "backend": "onnx"}}`.
Closes #2624
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/server.py`: call the existing
`_reconcile_kompress_health()` in the `/debug/warmup` route before
serializing the registry. The reconciler never instantiates a compressor
and never calls `preload()` / `ensure_background_load()` / `compress()`
— it only reads `is_ready()` / `ready_backend()` on an already resident
instance, or falls back to the module-level ONNX cache — so the endpoint
stays side-effect free and idempotent.
- `headroom/proxy/server.py`: stamp `source_status="runtime"` at both
`mark_loaded()` promotion sites in `_reconcile_kompress_health()` (the
resident-compressor path and the `_kompress_cache` fallback),
overwriting the stale startup marker.
- `tests/test_proxy_debug_endpoints.py`: three regression tests plus a
read-only compressor stub whose `preload` / `ensure_background_load`
raise, so a future change that makes the debug route trigger a load
fails loudly.
- `tests/test_proxy_health.py`: assert the promoted slot's
`info["source_status"] == "runtime"`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ pytest tests/test_proxy_debug_endpoints.py tests/test_proxy_health.py tests/test_proxy_warmup.py -q
tests\test_proxy_debug_endpoints.py ............................. [ 52%]
tests\test_proxy_health.py ................. [ 83%]
tests\test_proxy_warmup.py ......... [100%]
============================= 55 passed in 36.56s =============================
$ ruff check headroom/proxy/server.py tests/test_proxy_debug_endpoints.py tests/test_proxy_health.py
All checks passed!
$ ruff format --check headroom/proxy/server.py tests/test_proxy_debug_endpoints.py tests/test_proxy_health.py
3 files already formatted
$ mypy headroom --ignore-missing-imports
Success: no issues found in 506 source files
```
The three new tests were confirmed to be genuine regression tests: with
the `server.py` change reverted and the tests kept, all three fail.
```text
$ git stash push -- headroom/proxy/server.py && pytest tests/test_proxy_debug_endpoints.py -q -k kompress
FAILED tests/test_proxy_debug_endpoints.py::test_debug_warmup_promotes_deferred_kompress_after_runtime_load
FAILED tests/test_proxy_debug_endpoints.py::test_debug_warmup_keeps_pending_kompress_null
FAILED tests/test_proxy_debug_endpoints.py::test_debug_warmup_never_starts_kompress_loading
====================== 3 failed, 26 deselected in 3.98s =======================
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.13.11, pytest 9.1.1, ruff 0.15.x,
mypy 1.20.2, branch based on `main` at
|
||
|
AxelRay
|
d50cfabedc
|
fix(proxy): report deferred Kompress status and promote health from cache (#2564)
## Description When Kompress preload is deferred until first request, startup still logs "not installed" even if ML deps are present. After the model later loads into the module cache, /readyz and /health can keep reporting kompress as unhealthy because reconcile only inspected attached compressor instances. This PR reports deferred startup accurately and promotes health from the live module cache once the model is ready, without starting loads from health checks. Closes #2560 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Treat eager-status `deferred` as installed-but-deferred at proxy startup and log that state instead of "not installed". - Promote `/readyz` and `/health` Kompress readiness from the module-level model cache when attached compressors are missing or not ready. - Keep health inspection free of lazy getters and download side effects. - Add regressions for deferred startup logging and cache-based health promotion. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ PYTHONPATH=/tmp/headroom-2561 python -m pytest tests/test_proxy_health.py tests/test_proxy_eager_preload_bind.py -q -o addopts= 21 passed, 1 warning in 2.73s $ ruff format --check headroom/proxy/server.py tests/test_proxy_health.py tests/test_proxy_eager_preload_bind.py 3 files already formatted $ ruff check headroom/proxy/server.py tests/test_proxy_health.py tests/test_proxy_eager_preload_bind.py All checks passed! ``` ## Real Behavior Proof - Environment: Linux VPS, Python 3.11 venv with headroom-ai 0.32.1 wheel for `_core`, checked out main + this branch overlayed for source under test - Exact command / steps: `PYTHONPATH=/tmp/headroom-2561 python -m pytest tests/test_proxy_health.py tests/test_proxy_eager_preload_bind.py -q -o addopts=`; `ruff format --check` and `ruff check` on the three changed files - Observed result: 21 focused tests passed, including deferred startup log regression and module-cache health promotion; ruff format/check clean - Not tested: live multi-request proxy with real ONNX model download on this host; install-status follow-up mentioned in the issue comment ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes - Scoped to Kompress status reporting only. The separate `headroom install status` ownership probe in the issue comment is left for a follow-up. Co-authored-by: axelray-dev <axelray-dev@users.noreply.github.com> |
||
|
Rod Boev
|
54526bc858
|
fix(proxy): promote Kompress health after runtime load (#2402)
## Description
`/readyz` can keep reporting Kompress as `{"ready": false, "status":
"unhealthy", "backend": null}` after the live compressor has already
become ready. Startup intentionally records Kompress as `deferred`
without loading the model, `WarmupRegistry.merge_transform_status()`
stores that only as metadata, and the health check later serializes the
stale warmup slot instead of the live runtime compressor state. The
request path can already see the real readiness signal through
`KompressCompressor.is_ready()`, but nothing promotes the health surface
after startup.
This change keeps startup behavior untouched and reconciles Kompress
health from the live compressor right before `/readyz` serializes
component state. It adds side-effect-free runtime backend accessors for
local and remote Kompress implementations, promotes the warmup slot only
when the runtime compressor is ready, preserves loaded state on
transient inspection failures, and keeps Kompress excluded from
aggregate readiness.
Closes #2386
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/kompress_compressor.py`: add a side-effect-free
`ready_backend()` accessor that returns the cached backend for the
current model or `None`.
- `headroom/transforms/kompress_remote.py`: add `ready_backend()`
returning `"remote"` for the always-ready remote adapter.
- `headroom/proxy/server.py`: derive Kompress health from the live
enabled `ContentRouter` instances, promote the warmup slot only when
runtime readiness is real, respect per-provider re-enable overrides, and
preserve loaded state on transient inspection failures.
- `tests/test_proxy_health.py`: add focused regression, override,
pending, remote, no-instantiation, disabled, fail-open, and
aggregate-readiness coverage.
- `tests/test_kompress_preload_deferral.py`: keep startup-deferral proof
current if a helper needs the new accessor surface.
## Testing
- [x] Unit tests pass (`uv run pytest tests/test_proxy_health.py
tests/test_kompress_preload_deferral.py
tests/test_kompress_request_nonblocking.py -q`)
- [x] Linting passes (`uv run ruff check headroom/proxy/server.py
headroom/transforms/kompress_compressor.py
headroom/transforms/kompress_remote.py tests/test_proxy_health.py
tests/test_kompress_preload_deferral.py`)
- [x] Formatting passes (`uv run ruff format headroom/proxy/server.py
headroom/transforms/kompress_compressor.py
headroom/transforms/kompress_remote.py tests/test_proxy_health.py
tests/test_kompress_preload_deferral.py --check`)
- [x] New tests added for new functionality when applicable
- [x] Manual testing performed
### Test Output
```text
$ uv run pytest tests/test_proxy_health.py tests/test_kompress_preload_deferral.py tests/test_kompress_request_nonblocking.py -q
................................ [100%]
32 passed, 1 warning in 2.06s
$ uv run ruff check headroom/proxy/server.py headroom/transforms/kompress_compressor.py headroom/transforms/kompress_remote.py tests/test_proxy_health.py tests/test_kompress_preload_deferral.py
All checks passed!
$ uv run ruff format headroom/proxy/server.py headroom/transforms/kompress_compressor.py headroom/transforms/kompress_remote.py tests/test_proxy_health.py tests/test_kompress_preload_deferral.py --check
4 files already formatted
```
## Real Behavior Proof
- Environment: Windows host, local FastAPI test app with the same
`HeadroomProxy`, `WarmupRegistry`, and `/readyz` route used in
production
- Exact command / steps: run `uv run pytest tests/test_proxy_health.py
tests/test_kompress_preload_deferral.py
tests/test_kompress_request_nonblocking.py -q`, covering a deferred
startup slot, a pending resident compressor, a global-disable plus
`disable_kompress_anthropic=False` override, and a router whose lazy
getters would raise if health instantiated them
- Observed result: deferred runtime readiness promotes to `{"enabled":
true, "ready": true, "status": "healthy", "backend": "onnx"}`, a pending
resident compressor stays `{"ready": false, "backend": null}`, a
per-provider override re-enables health even when the global flag is
off, and the health path never instantiates Kompress
- Not tested: live remote Kompress endpoint behavior beyond the local
remote-adapter contract
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have made corresponding changes to the documentation if needed
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
## Additional Notes
- `CHANGELOG.md` stays untouched because Headroom generates changelog
entries from conventional commits.
- Kompress remains a soft component already excluded from aggregate
readiness. This PR fixes only the per-component health report.
- The health path must remain read-only; it must not call `preload()`,
`ensure_background_load()`, `compress()`, or any network or model I/O.
|
||
|
GUOHAO LIU
|
f1663ea557
|
fix(health): exclude kompress from aggregate readiness + adversarial PBT (#2066)
## Description Kompress's model-not-ready state (e.g. after fresh install before first compression cycle) was being incorrectly reported as a proxy-wide failure in the aggregate readiness endpoint, because the health check treated it the same as a hard failure. This PR: 1. Excludes kompress from the aggregate readiness check (Closes #1842) 2. Adds adversarial + PBT tests to verify the exclusion behavior 3. Surfaces model-not-ready state to operators via dedicated log ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/health.py`: exclude kompress from aggregate readiness check - `headroom/proxy/checks.py`: surface kompress model-not-ready state in health log - `tests/test_proxy_health.py`: add adversarial + PBT tests for kompress exclusion - `tests/test_proxy_health.py`: add model-not-ready edge case coverage ## Testing - [x] Unit tests pass - [x] Linting passes - [x] Adversarial edge cases covered ### Test Output ```text $ uv run pytest tests/test_proxy_health.py -x -q -v (adversarial + PBT tests pass) ``` ## Real Behavior Proof - Environment: Linux, headroom main - Exact command / steps: `uv run pytest tests/test_proxy_health.py -x -q` - Observed result: All tests pass including new adversarial/PBT coverage - Not tested: End-to-end with live kompress instance in model-not-ready state ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review --------- Co-authored-by: lennney <lennney@users.noreply.github.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> Co-authored-by: Tejas Chopra <chopratejas@gmail.com> |