headroom/tests/test_wrap_concurrent_settings.py

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

255 lines
10 KiB
Python
Raw Normal View History

fix(wrap): stop concurrent wrap sessions clobbering settings.local.json (#3232) ## Description Several `headroom wrap` sessions in one project each write the proxy URL into `.claude/settings.local.json` and restore it on exit. That read-modify-write was unsynchronised. The write itself is atomic so the file never tears, but the updates were still lost against each other: - **Live sessions were silently unrouted.** The first session to exit deleted the key while its siblings were still running. They kept working, but their traffic stopped going through the proxy — no error, no warning, no savings. - **A dead proxy was written back into the project.** A session that started second captured the *first* session's proxy URL as "the original", so its exit restored a URL pointing at a port that was already gone. Every later session in that project then failed to connect. - **SIGTERM/SIGHUP never ran the restore at all.** `cleanup` was registered as the handler, but a Python signal handler that returns normally does not unwind the stack — under PEP 475 the interrupted `waitpid` is simply retried. The `finally` block that restores `settings.local.json` never ran, while the handler had already terminated the proxy underneath a child that was still alive. Closes #3205 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - **`_wrap_settings_lock`** — an exclusive OS lock (flock / `msvcrt.locking`) held across the settings read-modify-write. A workspace that cannot hold lock state degrades to the previous behaviour rather than failing, matching `_proxy_start_lock`. - **`.headroom_wrap_owners.json`** — a sidecar recording, per env key, the true pre-wrap `original` plus the live sessions holding it. The first writer records the original; later writers inherit it and are flagged `inherited`, so no session restores a value it did not observe first-hand. A session exits without restoring while a sibling still holds the key. Dead holders are pruned with the same conservative PID+identity liveness the proxy-client markers use, so a SIGKILLed session cannot wedge the key. - **`unwrap` passes `force=True`** — unwrap is the user explicitly asking for their settings back, so it drops every claim instead of deferring to a live sibling and silently printing success while leaving the proxy URL in the file. - **The #2221 self-heal passes `dead_ports`** — a wrapper process can outlive its proxy (proxy alone SIGKILLed). Its claim would otherwise veto the self-heal and leave `ANTHROPIC_BASE_URL` pointing at a port just proven dead. - **`_rehome_wrap_marker`** — the wrap marker has one slot, won by the last writer. When that writer exits while a sibling still owns the key, the marker is rewritten to describe the survivor (carrying the record's true original), so the survivor keeps its #2221 self-heal record instead of being left with a marker describing a dead process. - **`_exit_on_signal`** replaces `cleanup` as the SIGTERM/SIGHUP handler. Raising `SystemExit` unwinds, so the settings restore actually runs and cleanup happens exactly once from `finally`. - **`_proxy_start_lock` now shares `_locked_file`** with the new settings lock rather than carrying a second verbatim copy of the platform branches. ## Testing - [x] Unit tests pass (`pytest`) — full suite, 11518 passed / 588 skipped - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed `tests/test_wrap_concurrent_settings.py` (14 tests) covers: a sibling exit leaving survivors routed, the last session out restoring the true original, a pre-existing user URL surviving the whole cycle, three sessions in every exit order, a crashed session not wedging the key, forced unwrap past a live session, a holder that outlived its proxy not vetoing the self-heal, marker rehoming, and the signal-handler unwind. ### Test Output ```text $ uv run pytest tests/test_cli/test_unwrap_claude.py tests/test_cli/test_wrap_claude_base_url.py \ tests/test_cli/test_wrap_claude_finally_unbound.py tests/test_cli/test_wrap_claude_vertex_proxy_env.py \ tests/test_cli/test_wrap_claude.py tests/test_cli/test_wrap_dead_marker_selfheal.py \ tests/test_cli/test_wrap_helpers.py tests/test_cli/test_wrap_stale_marker.py \ tests/test_cli/test_wrap_persistent.py tests/test_wrap_concurrent_settings.py tests/test_cli_doctor.py -q tests/test_wrap_concurrent_settings.py .............. [ 72%] tests/test_cli_doctor.py ............................................... [ 89%] ............................... [100%] ============================= 285 passed in 3.01s ============================== $ uv run pytest tests/ -q ======== 11518 passed, 588 skipped, 6036 warnings in 1831.34s (0:30:31) ======== $ uv run ruff check . All checks passed! $ uv run mypy headroom Success: no issues found in 527 source files ``` ## Real Behavior Proof - **Environment:** macOS 15 (Darwin 25.4.0), Python 3.12.13, repo venv, Claude provider path (`ANTHROPIC_BASE_URL` in `.claude/settings.local.json`). - **Exact command / steps:** a script spawning **two real OS processes** — no mocks, real PIDs, real files — that call the same `_write_claude_wrap_base_url` / `_restore_claude_wrap_base_url` helpers `wrap claude` uses. The project starts with a real user gateway already set. Session A (port 8787) starts, session B (port 8788) starts 0.7s later, A exits while B is still running, then B exits. Run identically on `main` and on this branch. **Before (on `main`) — both bugs visible:** ```text start : {"ANTHROPIC_BASE_URL": "https://my-gateway.example.com"} session port=8787 started, remembers previous='https://my-gateway.example.com' session port=8788 started, remembers previous='http://127.0.0.1:8787' both sessions running : {"ANTHROPIC_BASE_URL": "http://127.0.0.1:8788"} session port=8787 exited after FIRST session exits : {"ANTHROPIC_BASE_URL": "https://my-gateway.example.com"} session port=8788 exited after LAST session exits : {"ANTHROPIC_BASE_URL": "http://127.0.0.1:8787"} ``` Session B is still running, but after A exits the proxy URL is gone from under it — B is unrouted with no error. And the final state is `http://127.0.0.1:8787`: a dead proxy left permanently in the user's project, with their real gateway lost. **After (this branch):** ```text start : {"ANTHROPIC_BASE_URL": "https://my-gateway.example.com"} session port=8787 started, remembers previous='https://my-gateway.example.com' session port=8788 started, remembers previous='http://127.0.0.1:8787' both sessions running : {"ANTHROPIC_BASE_URL": "http://127.0.0.1:8788"} session port=8787 exited after FIRST session exits : {"ANTHROPIC_BASE_URL": "http://127.0.0.1:8788"} session port=8788 exited after LAST session exits : {"ANTHROPIC_BASE_URL": "https://my-gateway.example.com"} ``` B stays routed after A exits, and the last session out restores the user's real gateway. - **Observed result:** matches the intent on both counts — no unrouting, no dead proxy residue, user's pre-existing URL preserved. - **Not tested:** Windows (`msvcrt.locking`) — the lock and dead-holder pruning are exercised on POSIX only; the Windows branch is the same code path `_proxy_start_lock` has shipped with. No live end-to-end run against a real Anthropic endpoint with two concurrent `claude` CLIs; the proof above drives the same helpers out of two real processes instead. Foundry/Vertex key variants are covered by unit tests, not by a live run. Real SIGTERM/SIGHUP delivery to a running `wrap claude` was not exercised end to end — the handler's unwind is covered by a unit test, and full signal delivery would need a spawned and killed subprocess, which the existing #1768 test also declined to do. ## Runtime Rollout Safety - **Rollout-managed feature(s):** none — this is an unconditional correctness fix on the wrap settings path. - **Minimum rollout channel:** n/a. - **Stable/default behavior changed:** yes, three ways. (1) A wrap session exiting while a sibling holds the key now leaves the key in place instead of removing it. (2) SIGTERM/SIGHUP now unwinds, so the child CLI is terminated by `subprocess.run`'s cleanup rather than being left running against a torn-down proxy. (3) Two new sidecar files appear next to `settings.local.json`: `.headroom_wrap_owners.json` (removed when the last holder exits) and `.headroom_wrap_settings.lock` (retained by design — deleting a live lock file creates an inode-replacement race). - **Kill switch / disable path:** none. A workspace where the lock file cannot be created degrades to the previous unsynchronised behaviour automatically. - **Unsafe override required:** no. - **Qualification impact:** none beyond the wrap settings path. - **Rollback path:** revert the commit; the sidecar files are ignored by older versions and can be deleted safely. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes - The ownership record is keyed per env key, so `ANTHROPIC_BASE_URL`, the Foundry/Vertex variants and the tool-search entry are tracked independently. - Documentation: the behaviour is documented in the helper docstrings rather than user-facing docs — the sidecar files are internal state a user never configures. - Follow-up worth considering: `.headroom_wrap_settings.lock` is intentionally never deleted (matching `_proxy_start_lock`'s retention rationale), so it stays in `.claude/` after `unwrap`. Removing it safely needs a separate think about the inode-replacement race. Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-23 22:36:17 -07:00
"""Concurrent `headroom wrap` sessions sharing one project's settings (#3205).
`wrap claude` writes ANTHROPIC_BASE_URL into `.claude/settings.local.json` and
restores it on exit. Several sessions in one project run that read-modify-write
concurrently. The write is atomic so the file never tears, but the updates were
still lost against each other:
* the first session's exit deleted the key while the others were still
running -- they silently stopped routing through the proxy, kept working,
and lost every byte of compression with no error anywhere; and
* a session that started second remembered the *first* session's proxy URL as
"the original", so its exit wrote a dead proxy back into the file, which
every later session in that project then failed to connect to.
"""
from __future__ import annotations
import json
from pathlib import Path
from unittest import mock
import pytest
from headroom.cli import wrap as W
@pytest.fixture
def settings(tmp_path: Path) -> Path:
path = tmp_path / ".claude" / "settings.local.json"
path.parent.mkdir(parents=True)
path.write_text(json.dumps({"env": {"FOO": "bar"}}), encoding="utf-8")
return path
def _env(path: Path) -> dict:
return json.loads(path.read_text(encoding="utf-8")).get("env", {}) if path.exists() else {}
class _Sessions:
"""Drive several wrap sessions with distinct, controllable PIDs."""
def __init__(self, *pids: int) -> None:
self.live = set(pids)
def __enter__(self) -> _Sessions:
self._patches = [
mock.patch.object(W, "_pid_alive", lambda pid: pid in self.live),
mock.patch.object(W, "_identity_mismatch", lambda *a: False),
]
for p in self._patches:
p.start()
return self
def __exit__(self, *exc: object) -> None:
for p in self._patches:
p.stop()
def launch(self, pid: int, url: str, path: Path, port: int | None = None) -> str | None:
with mock.patch("os.getpid", lambda: pid):
return W._write_claude_wrap_base_url(url, settings_path=path, port=port)
def exit(self, pid: int, previous: str | None, path: Path) -> None:
self.live.discard(pid)
with mock.patch("os.getpid", lambda: pid):
W._restore_claude_wrap_base_url(previous, settings_path=path)
def crash(self, pid: int) -> None:
"""Vanish without running cleanup (SIGKILL, hard reboot)."""
self.live.discard(pid)
def test_first_session_exiting_leaves_the_others_routed(settings: Path) -> None:
"""The reported symptom: sessions silently stop routing when a sibling exits."""
with _Sessions(1001, 1002) as s:
a = s.launch(1001, "http://127.0.0.1:8787", settings)
s.launch(1002, "http://127.0.0.1:8788", settings)
s.exit(1001, a, settings)
assert "ANTHROPIC_BASE_URL" in _env(settings), "surviving session was unrouted"
def test_last_session_out_restores_the_true_original(settings: Path) -> None:
"""A later session must not restore an earlier session's dead proxy URL."""
with _Sessions(1001, 1002) as s:
a = s.launch(1001, "http://127.0.0.1:8787", settings)
b = s.launch(1002, "http://127.0.0.1:8788", settings)
s.exit(1001, a, settings)
s.exit(1002, b, settings)
assert _env(settings) == {"FOO": "bar"}, "stale proxy URL left behind"
def test_a_pre_existing_user_base_url_survives_the_whole_cycle(settings: Path) -> None:
"""A URL the project already had is restored, not deleted."""
settings.write_text(
json.dumps({"env": {"ANTHROPIC_BASE_URL": "http://user-proxy:1234"}}), encoding="utf-8"
)
with _Sessions(1001, 1002) as s:
a = s.launch(1001, "http://127.0.0.1:8787", settings)
b = s.launch(1002, "http://127.0.0.1:8788", settings)
s.exit(1001, a, settings)
s.exit(1002, b, settings)
assert _env(settings)["ANTHROPIC_BASE_URL"] == "http://user-proxy:1234"
def test_three_sessions_any_exit_order(settings: Path) -> None:
for order in ([1001, 1002, 1003], [1003, 1001, 1002], [1002, 1003, 1001]):
settings.write_text(json.dumps({"env": {"FOO": "bar"}}), encoding="utf-8")
with _Sessions(*order) as s:
prev = {
pid: s.launch(pid, f"http://127.0.0.1:{8787 + i}", settings)
for i, pid in enumerate(order)
}
for pid in order[:-1]:
s.exit(pid, prev[pid], settings)
assert "ANTHROPIC_BASE_URL" in _env(settings), f"unrouted early in {order}"
s.exit(order[-1], prev[order[-1]], settings)
assert _env(settings) == {"FOO": "bar"}, f"residue after {order}"
def test_a_crashed_session_does_not_wedge_the_key(settings: Path) -> None:
"""A SIGKILLed session never releases; its claim must be pruned as dead."""
with _Sessions(1001, 1002) as s:
s.launch(1001, "http://127.0.0.1:8787", settings)
b = s.launch(1002, "http://127.0.0.1:8788", settings)
s.crash(1001)
s.exit(1002, b, settings)
assert _env(settings) == {"FOO": "bar"}
assert not W._wrap_owners_path(settings).exists()
def test_single_session_behaviour_is_unchanged(settings: Path) -> None:
with _Sessions(1001) as s:
a = s.launch(1001, "http://127.0.0.1:8787", settings)
assert _env(settings)["ANTHROPIC_BASE_URL"] == "http://127.0.0.1:8787"
s.exit(1001, a, settings)
assert _env(settings) == {"FOO": "bar"}
def test_restore_without_an_owner_record_still_honours_the_caller(settings: Path) -> None:
"""unwrap and legacy sessions pass the previous value directly."""
settings.write_text(
json.dumps({"env": {"ANTHROPIC_BASE_URL": "http://127.0.0.1:8787"}}), encoding="utf-8"
)
assert not W._wrap_owners_path(settings).exists()
W._restore_claude_wrap_base_url("http://legacy:9999", settings_path=settings)
assert _env(settings)["ANTHROPIC_BASE_URL"] == "http://legacy:9999"
def test_tool_search_key_is_tracked_independently(settings: Path) -> None:
"""Ownership is per key -- the tool-search entry has the same race."""
with _Sessions(1001, 1002) as s:
with mock.patch("os.getpid", lambda: 1001):
a = W._write_claude_wrap_tool_search("auto", settings_path=settings)
with mock.patch("os.getpid", lambda: 1002):
W._write_claude_wrap_tool_search("auto", settings_path=settings)
s.live.discard(1001)
with mock.patch("os.getpid", lambda: 1001):
W._restore_claude_wrap_tool_search(a, settings_path=settings)
assert W._TOOL_SEARCH_ENV in _env(settings), "surviving session lost tool-search"
def test_exit_on_signal_unwinds_so_finally_can_run() -> None:
"""`cleanup` as the handler never unwound; the settings restore never ran."""
with pytest.raises(SystemExit) as excinfo:
W._exit_on_signal(15, None)
assert excinfo.value.code == 143
def test_unwrap_forces_the_restore_past_a_live_session(settings: Path) -> None:
"""`unwrap` is the user asking for their settings back -- it must not no-op.
Deferring to a live sibling is right for a session exiting on its own, but
unwrap deferring means the command prints success while leaving the proxy
URL in the file.
"""
with _Sessions(1001) as s:
s.launch(1001, "http://127.0.0.1:8787", settings)
with mock.patch("os.getpid", lambda: 2002):
W._restore_claude_wrap_base_url(None, settings_path=settings, force=True)
assert _env(settings) == {"FOO": "bar"}, "unwrap left the proxy URL behind"
assert not W._wrap_owners_path(settings).exists(), "unwrap left ownership state behind"
def test_unwrap_restores_the_true_original_not_the_marker_value(settings: Path) -> None:
"""A caller with no claim of its own trusts the record over its marker.
The single-slot marker is won by the *last* writer, whose `previous` is the
first session's proxy URL -- restoring that is the #3205 bug via unwrap.
"""
settings.write_text(
json.dumps({"env": {"ANTHROPIC_BASE_URL": "http://user-proxy:1234"}}), encoding="utf-8"
)
with _Sessions(1001, 1002) as s:
s.launch(1001, "http://127.0.0.1:8787", settings, port=8787)
s.launch(1002, "http://127.0.0.1:8788", settings, port=8788)
with mock.patch("os.getpid", lambda: 2002):
W._restore_claude_wrap_base_url(
"http://127.0.0.1:8787", settings_path=settings, force=True
)
assert _env(settings)["ANTHROPIC_BASE_URL"] == "http://user-proxy:1234"
def test_a_holder_that_outlived_its_proxy_cannot_veto_the_selfheal(settings: Path) -> None:
"""#2221: a wrapper PID can outlive its proxy; its claim must not block."""
with _Sessions(1001) as s:
s.launch(1001, "http://127.0.0.1:8787", settings, port=8787)
# PID 1001 is still alive, but port 8787 has been proven dead.
W._restore_claude_wrap_base_url(None, settings_path=settings, dead_ports=frozenset({8787}))
assert _env(settings) == {"FOO": "bar"}, "dead proxy URL survived the self-heal"
def test_exiting_session_hands_its_marker_to_a_survivor(settings: Path) -> None:
"""The marker has one slot; the leaver must not strand or hijack it."""
with _Sessions(1001, 1002) as s:
s.launch(1001, "http://127.0.0.1:8787", settings, port=8787)
b = s.launch(1002, "http://127.0.0.1:8788", settings, port=8788)
marker = W._read_wrap_marker(settings)
assert marker is not None and marker["pid"] == 1002, "last writer owns the marker"
s.exit(1002, b, settings)
marker = W._read_wrap_marker(settings)
assert marker is not None, "survivor lost its #2221 self-heal record"
assert marker["pid"] == 1001, "marker still describes the exited session"
assert marker["port"] == 8787
assert marker["previous"] is None, "marker must carry the true original"
def test_the_founding_session_still_honours_an_explicit_previous(settings: Path) -> None:
"""A sole writer observed the pre-wrap value first-hand; do not override it."""
with _Sessions(1001) as s:
s.launch(1001, "http://127.0.0.1:8787", settings)
s.exit(1001, "https://existing-gateway.example.com/v1", settings)
assert _env(settings)["ANTHROPIC_BASE_URL"] == "https://existing-gateway.example.com/v1"