fix: harden persistent install startup (#1851)
## Description
Hardens persistent install startup and proxy compression behavior for
issue #1843. Repeated `headroom install start` / scheduled ensure calls
no longer spawn duplicate runtimes by default, and `/v1/compress` now
fails open on compression timeout instead of returning a 503. The PR
also adds a machine-readable platform feature matrix and app-level
stabilization tests for health, compression functionality, timeout
behavior, and matrix evidence.
Refs #1843
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Wrapped direct persistent deployment starts with the existing
profile-local runtime start lock.
- Made `headroom install start` idempotent when the deployment is
already healthy.
- Added wedged-runtime handling: if a PID is running but `/readyz` does
not recover inside the grace window, stop it before starting again.
- Kept `install agent ensure` inside the already-held lock while
delegating to the shared start helper.
- Changed `/v1/compress` timeout behavior from `503 compression_timeout`
to fail-open `200` with original messages, `compression_skipped: true`,
and `skip_reason: compression_timeout`.
- Added `tests/test_platform_stabilization_functional.py` covering real
FastAPI health/compression routes, successful compression metrics,
timeout fail-open speed, and a real JSON tool payload that reduces
tokens.
- Added `docs/platform-feature-matrix.json` and
`docs/platform-stabilization.md` for Linux/macOS/Windows hardening
coverage and known gaps.
- Strengthened matrix tests so cited local test/workflow paths must
exist.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
# Clean source tree without copied Rust extension: functional module is skipped locally, as CI copies _core from the built wheel.
> python -m pytest tests/test_install tests/test_cli/test_install_cli.py tests/test_platform_feature_matrix.py tests/test_platform_stabilization_functional.py -q
collected 120 items / 1 skipped
119 passed, 2 skipped in 17.08s
> python -m ruff check headroom/proxy/handlers/openai.py tests/test_platform_stabilization_functional.py tests/test_platform_feature_matrix.py
All checks passed!
# Local Windows compiled-core proof:
> python -m maturin build --profile ci --out dist-local
Built wheel for abi3 Python >= 3.10 to dist-local\headroom_ai-0.29.0-cp310-abi3-win_amd64.whl
# Copied _core.pyd from the wheel into headroom/ for local route execution, then:
> python -m pytest tests/test_platform_stabilization_functional.py -q
collected 4 items
4 passed in 6.71s
> python -m pytest tests/test_install tests/test_cli/test_install_cli.py tests/test_platform_feature_matrix.py -q
collected 120 items
119 passed, 1 skipped in 17.16s
Commit hooks:
Sync plugin versions.....................................................Passed
check for merge conflicts................................................Passed
ruff.....................................................................Passed
ruff-format..............................................................Passed
mypy.....................................................................Passed
```
## Real Behavior Proof
- Environment: Windows 11, PowerShell, Python 3.13.13, worktree
`C:\git\headroom-stabilization` on branch
`jd/cross-platform-stabilization`.
- Exact command / steps: built the Windows wheel with `maturin`,
extracted `_core.pyd`, ran the new FastAPI route tests and
install/matrix tests listed above, then removed generated artifacts
before committing.
- Observed result: direct start paths now no-op when healthy, skip
spawning when the start lock is contended, and stop a wedged runtime
before restart. `/v1/compress` now returns original messages quickly on
timeout instead of a 503. The real JSON tool-payload smoke test returns
`tokens_before > tokens_after`, `tokens_saved > 0`, `compression_ratio <
1.0`, and non-empty transforms through the public route.
- Not tested: full native Windows persistent process e2e remains blocked
by the upstream CRT/wheel issue already documented in workflows and in
the matrix. No real OS service was installed locally; service manager
behavior is covered by argument-level unit tests.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
N/A
## Additional Notes
CHANGELOG is not updated because this is an unreleased
hardening/test/documentation pass. The platform matrix intentionally
records partial/blocked Windows/macOS e2e gaps instead of claiming full
coverage where the repo cannot currently run it.
2026-07-10 04:40:34 +00:00
|
|
|
# Platform Stabilization Matrix
|
|
|
|
|
|
|
|
|
|
This matrix is the hardening source of truth for install, startup, runtime, cache, and compression behavior across Linux, macOS, and Windows. The machine-readable matrix lives in `docs/platform-feature-matrix.json`; CI tests validate its shape so every feature has explicit platform status and test evidence.
|
|
|
|
|
|
|
|
|
|
## Status Values
|
|
|
|
|
|
|
|
|
|
- `covered`: unit, integration, or native e2e coverage exists for the platform.
|
|
|
|
|
- `partial`: coverage exists, but a named gap remains.
|
|
|
|
|
- `gap`: no meaningful coverage exists yet.
|
|
|
|
|
- `blocked`: coverage is intentionally excluded by a known external blocker.
|
|
|
|
|
|
|
|
|
|
## Current Priorities
|
|
|
|
|
|
|
|
|
|
1. Keep install/setup/run idempotent. Persistent starts must not create duplicate proxy instances unless a future explicit opt-in exists.
|
|
|
|
|
2. Keep compression fail-open. Cold model downloads, saturated executors, or timeout paths must pass through unchanged traffic instead of hanging agents.
|
|
|
|
|
3. Keep cache behavior visible. CCR persistence and TOIN skip recommendations must have restart and served-recommendation coverage.
|
|
|
|
|
4. Keep Windows honest. Windows-specific tests should run locally and in CI whenever they do not require the currently blocked native wheel build.
|
|
|
|
|
|
|
|
|
|
## Issue 1843 Coverage Map
|
|
|
|
|
|
|
|
|
|
- Windows service quoting: `install_windows_service`
|
|
|
|
|
- Duplicate startup processes: `single_instance_start`
|
|
|
|
|
- Slow compression and hangs: `compression_fail_open`
|
|
|
|
|
- CCR restart persistence: `ccr_persistence`
|
|
|
|
|
- TOIN recommendation wiring: `toin_skip_recommendations`
|
|
|
|
|
- Install/setup/run sanity: `install_apply_python`, `init_cli`, `wrap_prepare_only`
|
|
|
|
|
|
|
|
|
|
When adding or closing a hardening item, update `docs/platform-feature-matrix.json` in the same PR as the implementation or test change.
|