mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
18 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
3488f8d4b5
|
fix(install): use --userns=keep-id under Podman so bind-mount writes don't fail (#2846)
## Description
`build_runtime_command` unconditionally adds `--user <uid>:<gid>` on
non-Windows hosts:
```python
# headroom/install/runtime.py
if not _is_windows():
getuid = getattr(os, "getuid", None)
getgid = getattr(os, "getgid", None)
if callable(getuid) and callable(getgid):
command.extend(["--user", f"{getuid()}:{getgid()}"])
```
That is correct for Docker, where container UIDs equal host UIDs, but
wrong for rootless Podman, where the host user is already mapped to
container UID 0 and the `/etc/subuid` range is mapped to container UIDs
1 and above. Passing `--user $(id -u):$(id -g)` therefore selects a
container UID backed by a subordinate host UID that owns nothing. The
bind-mounted `~/.headroom` appears inside the container as `root:root`
and is unwritable, so every write fails:
```text
PermissionError: [Errno 13] Permission denied: '/tmp/headroom-home/.headroom/memories'
event=proxy_inbound_request_aborted path=/v1/messages reason=PermissionError
```
The proxy still starts and reports healthy, so the failure only surfaces
once a request touches a write path. As the reporter confirmed,
`--userns=keep-id` (or omitting `--user`) fixes it.
The fix detects Podman and uses `--userns=keep-id` instead of `--user`,
which maps the host user to the same UID inside the container and keeps
the bind mounts writable. Docker still gets `--user`, unchanged.
Detection is subprocess-free: it resolves the `docker` binary and checks
its real name for the common `docker -> podman` symlink shim (e.g. NixOS
`/run/current-system/sw/bin/docker -> podman`), with an explicit
`HEADROOM_CONTAINER_RUNTIME` (`podman` / `docker`) override for setups
the symlink heuristic cannot see, such as a wrapper script.
Fixes #2804
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/install/runtime.py`: added `_container_runtime_is_podman()`
(env override, then a `docker`-binary realpath basename check, no
subprocess). In `build_runtime_command`, when Podman is detected the
command uses `--userns=keep-id` instead of `--user <uid>:<gid>`.
- `tests/test_install/test_runtime.py`: pinned the existing docker test
to the Docker path via `HEADROOM_CONTAINER_RUNTIME=docker` and asserted
`--userns=keep-id` is absent there; added
`test_build_runtime_command_podman_uses_keep_id_not_user` asserting the
Podman path drops `--user` and adds `--userns=keep-id`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
# Fail-before (source fix stashed, new test kept):
tests/test_install/test_runtime.py::test_build_runtime_command_podman_uses_keep_id_not_user FAILED
assert "--userns=keep-id" in command
AssertionError: assert '--userns=keep-id' in ['docker', 'run', '--rm', ...]
# Pass-after (fix applied):
tests/test_install/test_runtime.py 26 passed
# Broader install suite (excluding the pre-existing env-specific PowerShell installer test):
tests/test_install/ 142 passed, 1 skipped
# uvx ruff@0.15.17 check -> All checks passed!
# uvx mypy@1.20.2 headroom/install/runtime.py -> Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, project venv, pytest 9.1.1,
ruff 0.15.17 and mypy 1.20.2 via uvx.
- Exact command / steps: confirmed `build_runtime_command` adds `--user`
unconditionally on non-Windows, then drove both runtimes
deterministically via the `HEADROOM_CONTAINER_RUNTIME` override.
Fail-before with `git stash push headroom/install/runtime.py` and
`python -m pytest tests/test_install/test_runtime.py -k
podman_uses_keep_id` (the command still carries `--user`, no keep-id),
pass-after with `git stash pop` and rerunning the file (26 passed).
- Observed result: with Podman detected the docker command now contains
`--userns=keep-id` and no `--user`/`1000:1001`, matching the
`--userns=keep-id` invocation the reporter verified writes successfully;
with Docker it is unchanged (`--user 1000:1001`, no keep-id).
- Not tested: a live rootless-Podman deployment writing to a bind mount
(no Podman in this environment). The command construction is verified
directly, and `--userns=keep-id` is the documented, reporter-confirmed
switch for the rootless-Podman ID-mapping.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`: it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
Detection is intentionally subprocess-free and conservative: it only
diverges from today's behavior when the `docker` binary literally
resolves to a `podman`-named target, or when
`HEADROOM_CONTAINER_RUNTIME` is set. Real Docker installs are untouched.
The override also gives a clean escape hatch in both directions if a
given host's symlink layout hides the runtime. This is the `--user` half
of the persistent-docker + Podman issues; the separate host-memory-path
problem (#2803) is addressed in its own PR.
|
||
|
|
045f3dfe6f
|
fix(install): use CREATE_NO_WINDOW instead of DETACHED_PROCESS on Windows (#2527)
## Description On Windows, the detached agent process spawned by `install hook ensure` (and the `install restart` self-spawn) pops up a visible black console window repeatedly, because `DETACHED_PROCESS` makes `CREATE_NO_WINDOW` a no-op per the Win32 process-creation-flags docs. #2521 ## Type of Change - [x] Bug fix (non-breaking change which fixes an issue) ## Changes Made - `headroom/install/runtime.py`: `start_detached_agent()` now uses `CREATE_NO_WINDOW` instead of `DETACHED_PROCESS`, combined with `CREATE_NEW_PROCESS_GROUP` (unchanged detach/isolation semantics, window hidden). - `headroom/install/runtime.py`: `_spawn_detached_restart()` now also sets `CREATE_NO_WINDOW` on Windows (previously had no `creationflags` at all on that platform). - `tests/test_install/test_runtime.py`: updated the Windows branch of `test_start_detached_agent_and_run_foreground` to assert the actual `creationflags` value passed to `Popen`, instead of just monkeypatching an unused `DETACHED_PROCESS` attribute. ## Testing - [x] Added/updated tests - [x] Ran full local test suite ``` $ python -m pytest tests/test_install -q ======================= 137 passed, 1 skipped in 48.68s ======================= $ ruff check headroom/install/runtime.py tests/test_install/test_runtime.py All checks passed! $ ruff format --check headroom/install/runtime.py tests/test_install/test_runtime.py 2 files already formatted $ mypy headroom --ignore-missing-imports Success: no issues found in 506 source files ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13.11, headroom repo local checkout - Exact command / steps: `python -m pytest tests/test_install/test_runtime.py -q`, plus manual read of `subprocess` Windows creation-flag semantics (`DETACHED_PROCESS` + child console allocation vs `CREATE_NO_WINDOW`) - Observed result: all 25 tests in `test_runtime.py` pass, including the updated assertion that `creationflags == CREATE_NO_WINDOW | CREATE_NEW_PROCESS_GROUP` on the Windows code path - Not tested: did not reproduce the original visible-console-popup repro end-to-end via live Claude Code hook invocation (no environment with the full hook-triggered respawn loop set up in this session); relying on the Win32 docs and the reporter's own local verification of the same flag swap ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
560ffae103
|
feat(deploy): Add turnkey deploy command (#1404)
## Description Adds `headroom deploy` as the turnkey, zero-config local deployment entrypoint. The command chooses the most capable deployment path it can verify on the current host, configures detected tools through the existing persistent-install machinery, starts the proxy, and preserves the existing rollback behavior if an update fails. The selection order favors performance first: NVIDIA Docker GPU passthrough when `nvidia-smi` and Docker's NVIDIA runtime are available, then plain Docker, then native scheduled recovery, then a detached Python runtime fallback. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [x] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - Added the top-level `headroom deploy` command and reused the existing install manifest/apply/start/rollback path. - Added conservative runtime selection for GPU Docker, plain Docker, native schedulers, and detached Python fallback. - Added Docker runtime support for manifest-driven `--gpus all` passthrough. - Added tests for Docker selection, GPU Docker selection, detached fallback, GPU command rendering, and subprocess wrapper compliance. - Updated README and persistent-install docs to present the turnkey deployment flow and performance-first GPU behavior. - Allowed documented `opencode` targets through `headroom install apply --target`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Formatting passes (`ruff format --check`) - [x] Type checking passes in local pre-commit and CI - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text uv run --with pytest --with pytest-asyncio python -m pytest tests/test_cli/test_subprocess_utf8_encoding.py tests/test_cli/test_install_cli.py tests/test_install/test_runtime.py::test_build_runtime_command_for_docker_includes_gpu_passthrough tests/test_install/test_planner.py -q 47 passed in 1.57s uvx --from ruff==0.15.17 ruff check headroom/cli/install.py headroom/install/runtime.py tests/test_cli/test_install_cli.py tests/test_install/test_runtime.py --output-format concise All checks passed! uvx --from ruff==0.15.17 ruff format --check headroom/cli/install.py headroom/install/runtime.py tests/test_cli/test_install_cli.py tests/test_install/test_runtime.py 4 files already formatted ``` GitHub checks are green on the current head. ## Real Behavior Proof - Environment: Windows local worktree `C:\git\headroom`, Python 3.13 via `uv`; GitHub Actions Ubuntu/macOS/Windows runners for full PR CI. - Exact command / steps: Ran the focused deploy/install tests above, checked the touched Python files with the CI-pinned Ruff version, and confirmed the current PR head is mergeable with green GitHub checks. - Observed result: The deploy command, runtime selection, Docker GPU command rendering, install CLI behavior, and subprocess encoding coverage all pass locally; the branch is no longer conflicted. - Not tested: Actual RTX 4090 hardware passthrough on a physical NVIDIA workstation; the PR tests conservative detection and Docker command rendering without requiring GPU hardware in CI. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A - CLI/runtime behavior only. ## Additional Notes CHANGELOG update is not included because this is an unreleased feature PR and the repository's release tooling owns release notes from conventional commits. |
||
|
|
e5b3a634df
|
fix(proxy): dedupe Codex WS request logging for accurate mixed-provider dashboards (#2189)
## Description Running Claude Code (Anthropic) and Codex (OpenAI) against the **same** Headroom proxy instance on one port produced incorrect, unstable dashboard data. The proxy core is provider-isolated and multi-provider-safe by design; the defect was in the observability layer. The Codex `/v1/responses` **WebSocket** handler was the only path in the proxy that wrote to the request logger by hand instead of through the unified `emit_request_outcome` funnel, and it did so twice per session close: the per-turn funnel record plus an unconditional cumulative session-summary `RequestLog`. This PR removes the duplicate summary log so Codex WS emits exactly one request log per turn, matching the HTTP provider paths. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Dropped the duplicate cumulative session-summary `RequestLog` in the Codex WS handler while preserving the per-turn `emit_request_outcome` path. - Preserved gated `request_messages` and `turn_id` on residual outcomes so dashboard telemetry keeps the useful attribution without double-counting tokens. - Ensured explicit `--anyllm-provider` wins over a leaked `HEADROOM_ANYLLM_PROVIDER` environment variable. - Registered retry delay settings that had drifted out of the settings registry. - Hardened tests against developer-shell `HEADROOM_*` / `ANTHROPIC_CUSTOM_HEADERS` leakage and stabilized several focused proxy/wrap test fixtures. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ .venv/bin/pytest tests/ -q -p no:cacheprovider 8529 passed, 537 skipped, 5831 warnings in 276.25s (0:04:36) $ .venv/bin/ruff check <touched files> All checks passed! ``` ## Real Behavior Proof - Environment: macOS (Darwin 25.4.0), Python 3.13.14, pytest 9.0.3, ruff via project venv, branch `fix/multi-provider-runtime`. - Exact command / steps: Ran the full test suite without pytest cache provider and Ruff on all touched files; used `git stash` to confirm the stale fake-config failures pre-existed this change. - Observed result: Full suite passed with no failures; Ruff passed; Codex WS now routes end-of-session logging through `emit_request_outcome`, emitting one request log per turn with the same accounting model as Anthropic HTTP turns. - Not tested: Live simultaneous Claude + Codex dashboard run. `mypy headroom` was not run to completion; a scoped run reported one pre-existing `settings_store.py:470` coercion error outside this diff. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A - server-side observability fix; no UI markup changed. ## Additional Notes - The proxy's multi-provider routing, header/auth isolation, and per-model cache keying are already correct and unchanged here; only the WS observability write path was double-counting. - Architectural assessment: `plans/reports/research-260714-0004-multi-provider-upstream-compression-report.md`; root-cause + resolution trail: `plans/reports/debug-assessment-260714-0011-dashboard-instability-mixed-claude-codex-report.md`. - No live simultaneous Claude + Codex dashboard run was performed; validation is from test coverage and code review of the WS logging path. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
|
c3db8e47f8
|
fix(install): default docker image to headroomlabs-ai GHCR registry (#1867) (#2039)
## Description
The GitHub repository was transferred from `chopratejas/headroom` to
`headroomlabs-ai/headroom`. GitHub 301-redirects transferred repos for
web and git operations, but **GitHub Container Registry (GHCR) does
not** — the old package `ghcr.io/chopratejas/headroom` is now orphaned
and frozen (its `latest` tag stopped advancing at `0.27.0`), while CI
publishes new images to `ghcr.io/headroomlabs-ai/headroom` (the workflow
derives the path from `${{ github.repository }}`).
Headroom's install tooling still defaulted to the dead path, so
`headroom install apply --preset persistent-docker`, `headroom init`,
and the standalone install scripts all pulled a stale `0.27.0` image
instead of the current release.
This changes every docker-image **default** to
`ghcr.io/headroomlabs-ai/headroom:latest`.
Closes #1867
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/install/models.py` — `InstallManifest.image` default.
- `headroom/cli/install.py` — `--image` Click option default.
- `headroom/cli/init.py` — two `InstallManifest(...)` image args.
- `scripts/install.sh` / `scripts/install.ps1` — `IMAGE_DEFAULT` /
`$ImageDefault` plus the `--image` help-text default.
- `docker/docker-compose.native.yml` — image default in both services.
- `tests/test_install/test_planner.py` (4) and
`tests/test_install/test_runtime.py` (5) — updated the assertions that
pinned the old image (including `assert "<image>" in command`), so they
now verify the corrected registry threads through the planner and docker
runtime command.
Scope note: `github.com/chopratejas/...` links and the plugin
marketplace slug are intentionally **not** changed — GitHub redirects
those, so they still work. Only the genuinely-dead GHCR image references
are touched. No new dependency, no new abstraction.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ uv run pytest tests/test_install/ -q
99 passed, 1 skipped in 48.34s
$ uv run ruff check headroom/install/models.py headroom/cli/install.py headroom/cli/init.py
All checks passed!
$ grep -rn "ghcr.io/chopratejas/headroom" headroom/ scripts/ docker/ tests/
# (no source matches — every default now points at headroomlabs-ai)
```
The install-test assertions pinned the old image, so they fail against
the old defaults and pass after the fix — they are the regression guard.
## Real Behavior Proof
- **Environment:** Windows 11, Python 3.13.5, headroom installed from
this branch (editable, via `uv`).
- **Exact command / steps:** The dead-registry claim is verifiable at
the registry level, independent of a release:
```text
docker pull ghcr.io/chopratejas/headroom:latest # old default → 0.27.0
(frozen / orphaned)
docker pull ghcr.io/headroomlabs-ai/headroom:latest # new default →
current release
```
And the install pipeline now emits the correct image (covered by
`tests/test_install/test_runtime.py`, which asserts the resolved docker
command contains `ghcr.io/headroomlabs-ai/headroom:latest`).
- **Observed result:** `grep` confirms no `ghcr.io/chopratejas/headroom`
default remains in code, scripts, compose, or tests;
`tests/test_install/` is green (99 passed) with the corrected image
asserted end-to-end through planner → runtime command.
- **Not tested:** A live `docker pull` of both tags on this specific
machine (no local Docker daemon guaranteed) — the registry difference is
reproducible by anyone running the two `docker pull` commands above; and
an end-to-end `headroom install apply --preset persistent-docker`
against a real Docker host.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
N/A
## Additional Notes
- Documentation checklist item is N/A — no user-facing docs surface
beyond the CHANGELOG entry.
- `mypy` left unchecked — not run as part of this verification; the
change is a string-default swap with no type-level surface.
- The `--image` help text and `docker-compose.native.yml` were included
so every user-facing default is consistent; only the dead GHCR image
string was changed.
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
|
b097ef3e25
|
fix(install): don't let host env override the manifest in persistent-docker (#2090)
## Description In persistent-docker deployments a stale host env var can silently override the value the deployment manifest pinned for the container. `build_runtime_command` builds the `docker run` argv in two passes: 1. It emits the manifest's pinned env as `--env NAME=VALUE` (from `base_env` plus the deployment env). 2. It then walks `os.environ` and, for every name matching a `PASSTHROUGH_ENV_PREFIXES` prefix, appends a bare `--env NAME` so the host value is forwarded into the container. A manifest-pinned name and a host-exported name can collide when they share a passthrough prefix. `HEADROOM_BACKEND` is the clearest case: the manifest pins `--env HEADROOM_BACKEND=anthropic` in pass 1, and pass 2 also matches the `HEADROOM_` prefix and appends a bare `--env HEADROOM_BACKEND`. Docker resolves duplicate `--env` flags last-wins, and the bare passthrough comes last, so a stale host export `HEADROOM_BACKEND=anyllm` wins and the container runs a different backend than its deployment config says. `start_persistent_docker` runs the resulting command through `subprocess.run` with the parent process environment, so whatever the operator happened to have exported leaks in and overrides the manifest. The fix skips the bare passthrough for any name the manifest already pins, so the pinned value stands while unrelated host secrets (API keys and so on) are still passed through as before. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/install/runtime.py`: skip the bare `--env NAME` passthrough when `NAME` is already pinned by the manifest (`and name not in runtime_env`). - `tests/test_install/test_runtime.py`: add `test_build_runtime_command_docker_manifest_env_beats_host_passthrough`, which exports a conflicting `HEADROOM_BACKEND` and asserts the command keeps the manifest value and emits no bare passthrough for it. - `CHANGELOG.md`: Bug Fixes entry. ## Testing - [ ] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uvx ruff@0.15.17 format headroom/install/runtime.py tests/test_install/test_runtime.py 2 files left unchanged $ uvx ruff@0.15.17 check headroom/install/runtime.py tests/test_install/test_runtime.py All checks passed! $ python -m py_compile headroom/install/runtime.py tests/test_install/test_runtime.py OK ``` ## Real Behavior Proof - Environment: local checkout, Python 3.11, `uvx ruff@0.15.17`. - Exact command / steps: ran a standalone script that reproduces the two-pass argv build and models Docker's duplicate `--env` last-wins resolution, with the manifest pinning `HEADROOM_BACKEND=anthropic` and the host exporting `HEADROOM_BACKEND=anyllm`. - Observed result: the old build resolves the effective `HEADROOM_BACKEND` to the host value `anyllm` (bare passthrough wins); the new build keeps the manifest value `anthropic` and emits no bare `HEADROOM_BACKEND` token, while a non-pinned passthrough (`ANTHROPIC_API_KEY`) is still forwarded. - Not tested: I did not run the full `pytest` suite locally because it pulls in the ML stack; the new regression test is left for CI. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes The "unit tests pass locally" and "type checking" boxes are unchecked because the full suite imports the ML dependencies, which I can't run in this environment; the change is a pure function over `build_runtime_command`, verified by the standalone proof above and covered by the new regression test for CI. Co-authored-by: Tejas Chopra <chopratejas@gmail.com> |
||
|
|
1d2b76e72e
|
fix: harden persistent install startup (#1851)
## Description Hardens persistent install startup and proxy compression behavior for issue #1843. Repeated `headroom install start` / scheduled ensure calls no longer spawn duplicate runtimes by default, and `/v1/compress` now fails open on compression timeout instead of returning a 503. The PR also adds a machine-readable platform feature matrix and app-level stabilization tests for health, compression functionality, timeout behavior, and matrix evidence. Refs #1843 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Wrapped direct persistent deployment starts with the existing profile-local runtime start lock. - Made `headroom install start` idempotent when the deployment is already healthy. - Added wedged-runtime handling: if a PID is running but `/readyz` does not recover inside the grace window, stop it before starting again. - Kept `install agent ensure` inside the already-held lock while delegating to the shared start helper. - Changed `/v1/compress` timeout behavior from `503 compression_timeout` to fail-open `200` with original messages, `compression_skipped: true`, and `skip_reason: compression_timeout`. - Added `tests/test_platform_stabilization_functional.py` covering real FastAPI health/compression routes, successful compression metrics, timeout fail-open speed, and a real JSON tool payload that reduces tokens. - Added `docs/platform-feature-matrix.json` and `docs/platform-stabilization.md` for Linux/macOS/Windows hardening coverage and known gaps. - Strengthened matrix tests so cited local test/workflow paths must exist. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # Clean source tree without copied Rust extension: functional module is skipped locally, as CI copies _core from the built wheel. > python -m pytest tests/test_install tests/test_cli/test_install_cli.py tests/test_platform_feature_matrix.py tests/test_platform_stabilization_functional.py -q collected 120 items / 1 skipped 119 passed, 2 skipped in 17.08s > python -m ruff check headroom/proxy/handlers/openai.py tests/test_platform_stabilization_functional.py tests/test_platform_feature_matrix.py All checks passed! # Local Windows compiled-core proof: > python -m maturin build --profile ci --out dist-local Built wheel for abi3 Python >= 3.10 to dist-local\headroom_ai-0.29.0-cp310-abi3-win_amd64.whl # Copied _core.pyd from the wheel into headroom/ for local route execution, then: > python -m pytest tests/test_platform_stabilization_functional.py -q collected 4 items 4 passed in 6.71s > python -m pytest tests/test_install tests/test_cli/test_install_cli.py tests/test_platform_feature_matrix.py -q collected 120 items 119 passed, 1 skipped in 17.16s Commit hooks: Sync plugin versions.....................................................Passed check for merge conflicts................................................Passed ruff.....................................................................Passed ruff-format..............................................................Passed mypy.....................................................................Passed ``` ## Real Behavior Proof - Environment: Windows 11, PowerShell, Python 3.13.13, worktree `C:\git\headroom-stabilization` on branch `jd/cross-platform-stabilization`. - Exact command / steps: built the Windows wheel with `maturin`, extracted `_core.pyd`, ran the new FastAPI route tests and install/matrix tests listed above, then removed generated artifacts before committing. - Observed result: direct start paths now no-op when healthy, skip spawning when the start lock is contended, and stop a wedged runtime before restart. `/v1/compress` now returns original messages quickly on timeout instead of a 503. The real JSON tool-payload smoke test returns `tokens_before > tokens_after`, `tokens_saved > 0`, `compression_ratio < 1.0`, and non-empty transforms through the public route. - Not tested: full native Windows persistent process e2e remains blocked by the upstream CRT/wheel issue already documented in workflows and in the matrix. No real OS service was installed locally; service manager behavior is covered by argument-level unit tests. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes CHANGELOG is not updated because this is an unreleased hardening/test/documentation pass. The platform matrix intentionally records partial/blocked Windows/macOS e2e gaps instead of claiming full coverage where the repo cannot currently run it. |
||
|
|
816cb85fa8
|
fix(install): close parent log fd in start_detached_agent (#1576)
## Description
`start_detached_agent()` opens the agent log file and hands it to
`subprocess.Popen` as `stdout`/`stderr`, then returns the process
**without
closing the parent's copy of the file descriptor**. The child inherits
the fd
and writes to it, but the parent keeps its own copy open forever.
The result: every `headroom install start` leaks one file descriptor in
the
parent, and the leaked handle pins the log file open so it can't be
rotated.
On a tight `ulimit` or inside a container, repeated starts can walk
straight
into the fd limit.
```python
# headroom/install/runtime.py — before
log_file = open(log_file_path, "a", encoding="utf-8", errors="replace")
kwargs = {"stdout": log_file, "stderr": log_file, ...}
return subprocess.Popen(command, **kwargs) # parent's log_file never closed
```
The fix closes the parent's copy in a `try/finally` right after `Popen`
returns:
```python
try:
proc = subprocess.Popen(command, **kwargs)
finally:
# The child has inherited the log file descriptor, so the parent's
# copy is dead weight. Closing it (even when Popen raises) avoids
# leaking one fd per `headroom install start` and lets the log file
# be rotated.
log_file.close()
return proc
```
The `finally` is deliberate: it also covers the case where `Popen`
itself
raises (bad executable, fork failure), which would otherwise leak the
just-opened handle. This matches the `with open(...)` pattern already
used by
`run_foreground()` a few lines above.
Closes #1554
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/install/runtime.py`: close the parent's log file descriptor
in a `try/finally` after `subprocess.Popen` in `start_detached_agent()`,
so it is released on the normal path and when `Popen` raises.
- `tests/test_install/test_runtime.py`: add two regression tests — one
for a normal start, one for `Popen` raising — asserting the parent's log
handle is closed afterwards.
- `CHANGELOG.md`: Bug Fixes entry under Unreleased.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
Before the fix (`runtime.py` reverted to its parent commit, new tests
kept) —
the assertion inspects the *actual* log file handle and finds it still
open:
```text
E AssertionError: assert False is True
E + where False = <_io.TextIOWrapper name='...\deploy\demo\runner.log' mode='a' encoding='utf-8'>.closed
FAILED tests/test_install/test_runtime.py::test_start_detached_agent_closes_parent_log_fd
FAILED tests/test_install/test_runtime.py::test_start_detached_agent_closes_log_fd_when_popen_raises
============================== 2 failed in 0.43s ==============================
```
After the fix:
```text
tests\test_install\test_runtime.py ................
====================== 16 passed, 1 deselected in 0.35s =======================
```
(The one deselected test,
`test_runtime_start_lock_blocks_another_process`, is a
pre-existing failure on my Windows box — it fails identically on a clean
checkout of `main` and is unrelated to this change.)
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, headroom built from this
branch (`uv sync --extra dev`).
- Exact command / steps: ran the two regression tests, which drive the
real `start_detached_agent` code path — real `open()`, real
`Popen(stdout=...)`, real (or missing) `close()`. Only
`subprocess.Popen` is stubbed so the test never launches an actual
detached agent; the fd-lifecycle bug lives entirely in how the parent
handles its own handle, and that runs for real.
- Observed result: the log file handle the parent passed to `Popen` is
`.closed == False` before the fix and `.closed == True` after — for both
the normal path and the `Popen`-raises path (output above).
- Not tested: I intentionally did not spin up many real detached agents
to watch the OS fd table grow — on Windows that means flashing console
windows and isn't a clean signal anyway. The handle-state assertion on
the real file object is the deterministic equivalent. Did not run the
full `mypy headroom` pass (one-line lifecycle change, no new types).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- Single logical change, no new dependencies, default behavior otherwise
unchanged.
- Rebased/merged latest `main` to clear a `CHANGELOG.md` conflict.
- @chopratejas this is a sibling of #1555 (the `wrap.py` readiness-loop
handle leak you've got filed). I scoped this PR to #1554 only; happy to
follow up on #1555 separately if you'd like.
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
|
6b227b9c90
|
fix(install): use Windows-safe PID liveness probe in runtime_status (#1544) (#1560)
## Description `headroom install status` crashed with `OSError: [WinError 87] The parameter is incorrect` on Windows and, worse, tore down the live proxy it was only meant to inspect. `runtime_status()` probed liveness with a bare `os.kill(pid, 0)` guarded only by `except OSError`. Against a detached Windows agent (`DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP`), that call raises WinError 87, which CPython surfaces as a `SystemError` — not an `OSError` — so it escaped the handler, crashed status, and left the deployment dead (PID file removed, port 8787 freed). This mirrors the `os.kill`/`SystemError` fix PR #1315 applied to `cli/wrap.py`. Closes #1544 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added a shared Windows-safe `headroom._subprocess.pid_alive()` helper: rejects non-positive PIDs, prefers `psutil.pid_exists()`, and treats `SystemError` (WinError 87) as "not alive". - `install/runtime.py` `runtime_status()` now delegates to `pid_alive()` instead of an unguarded `os.kill(pid, 0)`. - `install/runtime.py` `stop_runtime()` now also catches `SystemError` to avoid the same crash class on shutdown. - `cli/wrap.py` `_pid_alive()` now delegates to the shared helper, so the marker-cleanup path and the install/runtime status path share one liveness probe (the shared helper the issue asked for). - Added regression tests for the helper and `runtime_status`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ ruff check . All checks passed! $ ruff format --check headroom/_subprocess.py headroom/install/runtime.py headroom/cli/wrap.py tests/test_install/test_runtime.py tests/test_pid_alive.py 5 files already formatted $ mypy headroom --ignore-missing-imports (exit 0) $ pytest tests/test_pid_alive.py tests/test_install/test_runtime.py tests/test_cli/test_wrap_helpers.py tests/test_cli/test_wrap_persistent.py \ --deselect "tests/test_install/test_runtime.py::test_runtime_start_lock_blocks_another_process" 89 passed ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13, ruff 0.15.17 / mypy 1.20.2 / pytest 9.1.0, psutil 7.2.2, branch `fix/1544-windows-pid-liveness`. - Exact command / steps: ran the four checks above; the new `tests/test_pid_alive.py` injects a `SystemError` (simulated WinError 87) and a stubbed `psutil` to drive both code paths, and `test_runtime_status_*` exercise `runtime_status()` end to end with a PID file present. - Observed result: `runtime_status` returns `"running"` for a live PID without sending any signal (asserted), returns `"stopped"` instead of crashing when the probe raises `SystemError`, and the helper only ever passes signal `0`. All 89 targeted tests pass; ruff/format/mypy clean. - Not tested: the full `headroom install apply --preset persistent-task` detached-agent reproduction against a live proxy was not run end to end; it is instead covered by the deterministic `SystemError`/WinError-87 injection regression tests. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - One pre-existing test, `test_runtime_start_lock_blocks_another_process`, fails on my local Windows checkout **before** these changes too (it asserts cross-process file-lock blocking and depends on `HOME` semantics that differ on Windows). It is unrelated to this fix and is deselected above; it passes on the Linux CI runners. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
feedead077
|
fix(install): stop duplicating ENTRYPOINT in persistent-docker runtime command (#833) (#1348)
## Description `headroom install apply --preset persistent-docker` pulls the image, starts the container, then fails after ~45s with "Deployment 'default' did not become ready after start." The rollback removes the container and manifest, leaving nothing running and no logs. Root cause: the published image already bakes the proxy invocation into its ENTRYPOINT (`Dockerfile`: `ENTRYPOINT ["headroom", "proxy"]`), but `build_runtime_command()` in `headroom/install/runtime.py` re-added `headroom proxy` after the image name. Docker concatenates ENTRYPOINT + args, so the container ran `headroom proxy headroom proxy --host 0.0.0.0 ...` and Click aborted with `Got unexpected extra arguments (headroom proxy)`. The runtime command now appends only the proxy flags after the image name, substituting the all-interface container bind host for the host pair carried in `proxy_args`. Closes #833 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/install/runtime.py`: drop the duplicated `headroom proxy` from the docker `build_runtime_command` output; append only `--host <bind> *proxy_args[2:]`. Extracted `CONTAINER_BIND_HOST` and `_PROXY_ARGS_HOST_PAIR_LEN` named constants. - `tests/test_install/test_runtime.py`: new regression test asserting the args appended after the image name never re-add the `headroom proxy` ENTRYPOINT. - `CHANGELOG.md`: Unreleased → Bug Fixes entry. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_install/ -q 91 passed, 1 skipped in 5.48s $ uv run ruff check headroom/install/runtime.py tests/test_install/test_runtime.py All checks passed! $ uv run mypy headroom/install/runtime.py Success: no issues found in 1 source file ``` #### RED → GREEN proof RED — new test with the prod fix reverted (test kept): ```text E AssertionError: container args re-add the ENTRYPOINT — got ['headroom', 'proxy', '--host', '0.0.0.0', '--port', '8787', '--backend', 'anthropic'] FAILED tests/test_install/test_runtime.py::test_build_runtime_command_for_docker_does_not_duplicate_entrypoint 1 failed in 0.17s ``` GREEN — with the fix applied: ```text tests/test_install/test_runtime.py::test_build_runtime_command_for_docker_does_not_duplicate_entrypoint 1 passed in 0.11s ``` ## Real Behavior Proof - Environment: Linux, Python 3.13, headroom @ this branch. - Exact command / steps: reproduce the exact concatenation Docker performs (ENTRYPOINT `headroom proxy` + the buggy CMD `headroom proxy --host 0.0.0.0 --port 8787`): ```text $ headroom proxy headroom proxy --host 0.0.0.0 --port 8787 Usage: headroom proxy [OPTIONS] Try 'headroom proxy --help' for help. Error: Got unexpected extra arguments (headroom proxy) ``` This is the exact error from the issue. After the fix, `build_runtime_command` appends only the flags after the image name: ```text args after image: ['--host', '0.0.0.0', '--port', '8787', '--backend', 'anthropic'] ``` so the container runs `headroom proxy --host 0.0.0.0 --port 8787 --backend anthropic` (ENTRYPOINT + flags) and Click accepts it. - Observed result: pre-fix Click aborts with the unexpected-arguments error (container crash-loops); post-fix the command line is valid. - Not tested: pulling and running the real `ghcr.io` image end-to-end (requires the published image + Docker host); the failure is fully determined by the generated argv, which is covered above and by the unit test. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes Scope is limited to the docker runtime command construction. The Python (`runtime_kind=python`) path was already correct and is unchanged. Screenshots N/A (CLI-only change). Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
9252d852c5
|
fix(init): guard persistent task startup (#616)
## Description Prevent `headroom init` hooks from spawning duplicate persistent-task runners while a proxy is still starting. Fixes #615 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Problem `_ensure_profile_running()` checked readiness for only one second and then launched `start_detached_agent()` whenever the proxy was not ready yet. When Claude/Codex hooks fired close together, each hook could race through that path and spawn another detached persistent-task runner. ## Changes Made - Add a profile-local, nonblocking runtime start lock around init hook startup. - Re-check readiness after acquiring the lock so late-arriving hooks do not start a duplicate runner. - If a runtime is already alive, wait up to 15 seconds for readiness before stopping and restarting it. - Add regression tests for lock contention, slow startup, and cross-process lock behavior. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ## Test Output ``` UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy pytest tests/test_cli/test_init_cli.py tests/test_cli/test_install_cli.py tests/test_cli/test_wrap_persistent.py tests/test_install/test_runtime.py # 89 passed in 0.61s UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy ruff check . # All checks passed! UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy ruff format --check . # 775 files already formatted UV_SKIP_WHEEL_FILENAME_CHECK=1 uv run --frozen --extra dev --extra proxy mypy headroom --ignore-missing-imports # Success: no issues found in 346 source files ``` Manual sandbox check: ``` # before this change: 3 ensure calls spawned 3 detached starts # after this change: 3 ensure calls spawned 1 detached start while the runtime was still starting ``` ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes Docs and CHANGELOG were left unchanged because this is a small runtime bug fix with no user-facing CLI/API change. |
||
|
|
efd2ac1ca4 |
chore: renormalize line endings to LF
`.gitattributes` declares `*.py text eol=lf` and `*.sh text eol=lf`, but 74 files (73 .py, 1 .sh) are stored in the index with CRLF line endings, violating that contract. Every macOS/Linux clone reports these files as "modified" on fresh checkout because git's diff engine sees the stored bytes don't match the attribute contract, even though the working tree and index match byte-for-byte. Running `git add --renormalize .` rewrites each affected blob so the stored form matches the attribute declaration. No semantic changes — every affected file's diff is "N insertions, N deletions" with inserts and deletes being the same lines modulo line endings. Follow-up commit adds `.git-blame-ignore-revs` so `git blame` / GitHub blame skip this mechanical commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
d4574f4ae2 |
test: avoid global platform leaks in install tests
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
|
38bf3e639c |
test: expand coverage across helper slices
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
|
4a87753713 |
feat(docker): forward HEADROOM_WORKSPACE_DIR and HEADROOM_CONFIG_DIR into containers
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
bd242fc62d |
test: expand persistent install coverage
Add focused regression coverage for install, runtime, provider, state, health, supervisor, and persistent wrap flows so the new persistent deployment surfaces are exercised more thoroughly in CI. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
|
865bef2216 |
fix: harden persistent install wrappers and review gaps
Align Docker-native wrapper help and runtime behavior with the Python install contract, including persistent deployment metadata, baked install-image defaults, and explicit unsupported wrap targets. Harden the Python persistent-install path with profile validation, safer provider-scope handling, Windows environment restoration, runtime parity improvements, and rollback-safe apply/update behavior. Update README, Docker install docs, CI, and focused regressions to cover the Windows BOM failure, wrapper parity, compose coverage, and Docker-native wrap behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
|
21896a095c |
feat: add persistent install lifecycle management
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |