mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
fix(wrap): isolate proxy stdio from proxy.log on Windows (#1191)
## Description Fix the Windows `proxy.log` rollover storm by separating wrap-managed subprocess stdio from the proxy's rotating runtime log. `headroom/cli/wrap.py` currently opens `~/.headroom/logs/proxy.log` and hands that file handle to the proxy subprocess, while `headroom/proxy/helpers.py` also rotates that same path at 10 MB with five backups. On Windows, the inherited stdio handle prevents the rename in `RotatingFileHandler.doRollover()`, which matches the repeated `WinError 32` traceback loop documented in `#1184`. This change keeps `proxy.log` as the canonical rotating runtime log and moves wrap-managed stdio into a dedicated sibling file so rollover can succeed without losing startup diagnostics. Closes #1184 The reproduction and split-fix sketch in https://github.com/chopratejas/headroom/issues/1184 materially shaped the chosen scope; this PR follows that root-cause split rather than changing the proxy's rotation policy. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - redirect wrap-managed proxy subprocess `stdout` and `stderr` into a dedicated sibling log instead of `proxy.log` - keep `proxy.log` as the success-path `Logs:` target and the sole rotating runtime log owned by the proxy - read startup-failure tails from the dedicated stdio log so early crashes remain debuggable - add focused regression coverage around `_start_proxy()` and document the behavior change in `CHANGELOG.md` ## Testing - [x] Unit tests pass (`uv run pytest tests/test_cli_proxy_env.py`) - [x] Linting passes (`uv run ruff check headroom/cli/wrap.py tests/test_cli_proxy_env.py`) - [x] Formatting checks pass (`uv run ruff format headroom/cli/wrap.py tests/test_cli_proxy_env.py --check`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text uv sync --extra dev uv run pytest tests/test_cli_proxy_env.py # Result: 46 passed in 2.79s uv run ruff check headroom/cli/wrap.py tests/test_cli_proxy_env.py # Result: All checks passed! uv run ruff format headroom/cli/wrap.py tests/test_cli_proxy_env.py --check # Result: 2 files already formatted ``` ## Real Behavior Proof - Environment: Windows, Python 3.12.13, local worktree with no live provider dependency. - Exact command / steps: `uv run pytest tests/test_cli_proxy_env.py -k "start_proxy_redirects_subprocess_stdio_to_standalone_log or start_proxy_tail_reads_standalone_stdio_log_on_process_exit or start_proxy_passes_resolved_copilot_api_url_to_proxy" -q` - Observed result: `3 passed, 43 deselected in 0.37s`; the regression slice proves `_start_proxy()` now routes subprocess `stdout` and `stderr` to `proxy-stdio.log`, still reports `Logs: .../proxy.log` to the user, reads startup-failure tails from `proxy-stdio.log`, and preserves Copilot target URL/token env wiring. - Not tested: a live Windows rollover reproduction with a real proxy process writing enough output to rotate `proxy.log`; `uv run mypy headroom`; the repo-wide suite beyond the focused regression and lint checks. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) Not applicable, the proof is command and log behavior rather than a visual change. ## Additional Notes The intended scope stayed narrow: isolate wrap-managed stdio from `proxy.log`, keep runtime logging semantics unchanged, and avoid widening into proxy-side logging policy changes unless the wrap-only fix proves insufficient during implementation.
This commit is contained in:
parent
c7295cad1d
commit
959ab0de47
3 changed files with 67 additions and 9 deletions
|
|
@ -36,6 +36,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
* **ccr:** make retrieval store TTL configurable with `HEADROOM_CCR_TTL_SECONDS`, expose the effective TTL in `/v1/retrieve/stats`, and distinguish expired retrievals from missing hashes.
|
||||
* **proxy:** add native Bedrock `/model/{id}/converse-stream` route and forward it through the existing streaming EventStream/SSE pipeline.
|
||||
* **wrap (codex):** fix `headroom wrap codex` producing a `config.toml` with duplicate top-level `model_provider` / `openai_base_url` keys (TOML-spec error) when the user had already configured their own provider. The injector now rewrites pre-existing top-level `model_provider` and `openai_base_url` lines in place — the previous value is kept in a `# was: …` trailing comment — instead of unconditionally prepending a duplicate, so `codex` can start against the proxy. The pre-wrap snapshot mechanism continues to byte-for-byte restore the original file on `headroom unwrap codex`.
|
||||
* **wrap:** isolate wrapped proxy subprocess stdout/stderr into `proxy-stdio.log`, so `proxy.log` remains the canonical rotating runtime log and Windows rollover failures from `RotatingFileHandler` are no longer blocked by wrapper stdio handles ([#1184](https://github.com/chopratejas/headroom/issues/1184)).
|
||||
|
||||
|
||||
## [0.27.0](https://github.com/chopratejas/headroom/compare/v0.26.0...v0.27.0) (2026-06-22)
|
||||
|
|
|
|||
|
|
@ -328,6 +328,11 @@ def _get_log_path() -> Path:
|
|||
return log_dir / "proxy.log"
|
||||
|
||||
|
||||
def _get_proxy_stdio_log_path() -> Path:
|
||||
"""Get path for dedicated proxy stdio capture."""
|
||||
return _get_log_path().with_name("proxy-stdio.log")
|
||||
|
||||
|
||||
def _start_proxy(
|
||||
port: int,
|
||||
*,
|
||||
|
|
@ -344,9 +349,9 @@ def _start_proxy(
|
|||
) -> subprocess.Popen:
|
||||
"""Start Headroom proxy as a background subprocess.
|
||||
|
||||
Logs are written to ~/.headroom/logs/proxy.log to avoid pipe buffer
|
||||
deadlocks (macOS pipe buffer is ~64KB — a busy proxy fills it quickly,
|
||||
blocking the process).
|
||||
Stdout and stderr are written to a dedicated sibling file, usually
|
||||
`~/.headroom/logs/proxy-stdio.log`, to avoid pipe deadlock risk without
|
||||
competing with the rotating `proxy.log` runtime log.
|
||||
"""
|
||||
cmd = [sys.executable, "-m", "headroom.cli", "proxy", "--port", str(port)]
|
||||
|
||||
|
|
@ -388,7 +393,8 @@ def _start_proxy(
|
|||
|
||||
timeout_seconds = _resolve_wrap_proxy_timeout_seconds()
|
||||
log_path = _get_log_path()
|
||||
log_file = open(log_path, "a") # noqa: SIM115
|
||||
stdio_log_path = _get_proxy_stdio_log_path()
|
||||
stdio_log_file = open(stdio_log_path, "a") # noqa: SIM115
|
||||
|
||||
# Ensure proxy subprocess uses UTF-8 (Windows defaults to cp1252)
|
||||
proxy_env = os.environ.copy()
|
||||
|
|
@ -418,8 +424,8 @@ def _start_proxy(
|
|||
|
||||
proc = subprocess.Popen(
|
||||
cmd,
|
||||
stdout=log_file,
|
||||
stderr=log_file,
|
||||
stdout=stdio_log_file,
|
||||
stderr=stdio_log_file,
|
||||
env=proxy_env,
|
||||
start_new_session=os.name == "posix",
|
||||
)
|
||||
|
|
@ -431,19 +437,20 @@ def _start_proxy(
|
|||
time.sleep(1)
|
||||
if _check_proxy(port):
|
||||
click.echo(f" Logs: {log_path}")
|
||||
stdio_log_file.close()
|
||||
return proc
|
||||
# Check if process died
|
||||
if proc.poll() is not None:
|
||||
log_file.close()
|
||||
stdio_log_file.close()
|
||||
# Read last few lines of log for error context
|
||||
try:
|
||||
tail = log_path.read_text()[-500:]
|
||||
tail = stdio_log_path.read_text()[-500:]
|
||||
except Exception:
|
||||
tail = "(no log output)"
|
||||
raise RuntimeError(f"Proxy exited with code {proc.returncode}: {tail}")
|
||||
|
||||
proc.kill()
|
||||
log_file.close()
|
||||
stdio_log_file.close()
|
||||
raise RuntimeError(
|
||||
f"Proxy failed to start on port {port} within {timeout_seconds} seconds. "
|
||||
f"Set {_WRAP_PROXY_TIMEOUT_ENV} to a larger number of seconds for slow startup."
|
||||
|
|
|
|||
|
|
@ -106,6 +106,32 @@ class TestCLIWrapProxyTimeout:
|
|||
assert env["GITHUB_COPILOT_API_URL"] == "https://copilot-api.acme.ghe.com"
|
||||
assert env["GITHUB_COPILOT_API_TOKEN"] == "copilot-api-token"
|
||||
|
||||
def test_start_proxy_redirects_subprocess_stdio_to_standalone_log(self, monkeypatch, tmp_path):
|
||||
fake_proc = _FakeProxyProcess()
|
||||
captured: dict[str, object] = {}
|
||||
logs: list[str] = []
|
||||
|
||||
monkeypatch.delenv(wrap_mod._WRAP_PROXY_TIMEOUT_ENV, raising=False)
|
||||
monkeypatch.setattr(wrap_mod, "_get_log_path", lambda: tmp_path / "proxy.log")
|
||||
monkeypatch.setattr(wrap_mod, "_check_proxy", lambda _port: True)
|
||||
monkeypatch.setattr(wrap_mod.time, "sleep", lambda _seconds: None)
|
||||
monkeypatch.setattr(wrap_mod, "_ml_wrap_extras_detected", lambda: False)
|
||||
monkeypatch.setattr(wrap_mod.click, "echo", lambda message: logs.append(str(message)))
|
||||
|
||||
def fake_popen(*args, **kwargs): # noqa: ANN002, ANN003
|
||||
captured["kwargs"] = kwargs
|
||||
return fake_proc
|
||||
|
||||
monkeypatch.setattr(wrap_mod.subprocess, "Popen", fake_popen)
|
||||
|
||||
proc = wrap_mod._start_proxy(8787, agent_type="codex")
|
||||
|
||||
assert proc is fake_proc
|
||||
assert captured["kwargs"]["stdout"] is captured["kwargs"]["stderr"]
|
||||
assert captured["kwargs"]["stdout"].name == str(tmp_path / "proxy-stdio.log")
|
||||
assert captured["kwargs"]["stdout"].name != str(tmp_path / "proxy.log")
|
||||
assert f" Logs: {tmp_path / 'proxy.log'}" in logs
|
||||
|
||||
def test_env_timeout_allows_slow_start_proxy_to_succeed(self, monkeypatch, tmp_path):
|
||||
fake_proc = _FakeProxyProcess()
|
||||
sleeps = []
|
||||
|
|
@ -129,6 +155,30 @@ class TestCLIWrapProxyTimeout:
|
|||
assert sleeps == [1, 1, 1, 1]
|
||||
assert fake_proc.killed is False
|
||||
|
||||
def test_start_proxy_tail_reads_standalone_stdio_log_on_process_exit(
|
||||
self, monkeypatch, tmp_path
|
||||
):
|
||||
fake_proc = _FakeProxyProcess()
|
||||
fake_proc.returncode = 1
|
||||
fake_proc.poll = lambda: fake_proc.returncode
|
||||
|
||||
monkeypatch.setenv(wrap_mod._WRAP_PROXY_TIMEOUT_ENV, "2")
|
||||
monkeypatch.setattr(wrap_mod, "_get_log_path", lambda: tmp_path / "proxy.log")
|
||||
monkeypatch.setattr(wrap_mod, "_check_proxy", lambda _port: False)
|
||||
monkeypatch.setattr(wrap_mod.time, "sleep", lambda _seconds: None)
|
||||
monkeypatch.setattr(wrap_mod.subprocess, "Popen", lambda *args, **kwargs: fake_proc)
|
||||
|
||||
(tmp_path / "proxy.log").write_text("canonical runtime log output")
|
||||
(tmp_path / "proxy-stdio.log").write_text("proxy stdio startup output")
|
||||
|
||||
with pytest.raises(RuntimeError) as excinfo:
|
||||
wrap_mod._start_proxy(8787, agent_type="codex")
|
||||
|
||||
message = str(excinfo.value)
|
||||
assert "Proxy exited with code 1" in message
|
||||
assert "proxy stdio startup output" in message
|
||||
assert "canonical runtime log output" not in message
|
||||
|
||||
def test_timeout_error_names_configured_timeout_and_env_var(self, monkeypatch, tmp_path):
|
||||
fake_proc = _FakeProxyProcess()
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue