headroom/tests/test_wrap_concurrent_settings.py
Tejas Chopra f27f235032
fix(wrap): stop concurrent wrap sessions clobbering settings.local.json (#3232)
## Description

Several `headroom wrap` sessions in one project each write the proxy URL
into
`.claude/settings.local.json` and restore it on exit. That
read-modify-write was
unsynchronised. The write itself is atomic so the file never tears, but
the
updates were still lost against each other:

- **Live sessions were silently unrouted.** The first session to exit
deleted the
key while its siblings were still running. They kept working, but their
traffic
  stopped going through the proxy — no error, no warning, no savings.
- **A dead proxy was written back into the project.** A session that
started
second captured the *first* session's proxy URL as "the original", so
its exit
restored a URL pointing at a port that was already gone. Every later
session in
  that project then failed to connect.
- **SIGTERM/SIGHUP never ran the restore at all.** `cleanup` was
registered as the
handler, but a Python signal handler that returns normally does not
unwind the
stack — under PEP 475 the interrupted `waitpid` is simply retried. The
`finally`
block that restores `settings.local.json` never ran, while the handler
had
  already terminated the proxy underneath a child that was still alive.

Closes #3205

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- **`_wrap_settings_lock`** — an exclusive OS lock (flock /
`msvcrt.locking`) held
across the settings read-modify-write. A workspace that cannot hold lock
state
  degrades to the previous behaviour rather than failing, matching
  `_proxy_start_lock`.
- **`.headroom_wrap_owners.json`** — a sidecar recording, per env key,
the true
pre-wrap `original` plus the live sessions holding it. The first writer
records
the original; later writers inherit it and are flagged `inherited`, so
no
session restores a value it did not observe first-hand. A session exits
without
restoring while a sibling still holds the key. Dead holders are pruned
with the
same conservative PID+identity liveness the proxy-client markers use, so
a
  SIGKILLed session cannot wedge the key.
- **`unwrap` passes `force=True`** — unwrap is the user explicitly
asking for
their settings back, so it drops every claim instead of deferring to a
live
sibling and silently printing success while leaving the proxy URL in the
file.
- **The #2221 self-heal passes `dead_ports`** — a wrapper process can
outlive its
proxy (proxy alone SIGKILLed). Its claim would otherwise veto the
self-heal and
  leave `ANTHROPIC_BASE_URL` pointing at a port just proven dead.
- **`_rehome_wrap_marker`** — the wrap marker has one slot, won by the
last
writer. When that writer exits while a sibling still owns the key, the
marker is
rewritten to describe the survivor (carrying the record's true
original), so the
survivor keeps its #2221 self-heal record instead of being left with a
marker
  describing a dead process.
- **`_exit_on_signal`** replaces `cleanup` as the SIGTERM/SIGHUP
handler. Raising
`SystemExit` unwinds, so the settings restore actually runs and cleanup
happens
  exactly once from `finally`.
- **`_proxy_start_lock` now shares `_locked_file`** with the new
settings lock
  rather than carrying a second verbatim copy of the platform branches.

## Testing

- [x] Unit tests pass (`pytest`) — full suite, 11518 passed / 588
skipped
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

`tests/test_wrap_concurrent_settings.py` (14 tests) covers: a sibling
exit leaving
survivors routed, the last session out restoring the true original, a
pre-existing
user URL surviving the whole cycle, three sessions in every exit order,
a crashed
session not wedging the key, forced unwrap past a live session, a holder
that
outlived its proxy not vetoing the self-heal, marker rehoming, and the
signal-handler unwind.

### Test Output

```text
$ uv run pytest tests/test_cli/test_unwrap_claude.py tests/test_cli/test_wrap_claude_base_url.py \
    tests/test_cli/test_wrap_claude_finally_unbound.py tests/test_cli/test_wrap_claude_vertex_proxy_env.py \
    tests/test_cli/test_wrap_claude.py tests/test_cli/test_wrap_dead_marker_selfheal.py \
    tests/test_cli/test_wrap_helpers.py tests/test_cli/test_wrap_stale_marker.py \
    tests/test_cli/test_wrap_persistent.py tests/test_wrap_concurrent_settings.py tests/test_cli_doctor.py -q

tests/test_wrap_concurrent_settings.py ..............                    [ 72%]
tests/test_cli_doctor.py ............................................... [ 89%]
...............................                                          [100%]

============================= 285 passed in 3.01s ==============================

$ uv run pytest tests/ -q
======== 11518 passed, 588 skipped, 6036 warnings in 1831.34s (0:30:31) ========

$ uv run ruff check .
All checks passed!

$ uv run mypy headroom
Success: no issues found in 527 source files
```

## Real Behavior Proof

- **Environment:** macOS 15 (Darwin 25.4.0), Python 3.12.13, repo venv,
Claude
  provider path (`ANTHROPIC_BASE_URL` in `.claude/settings.local.json`).
- **Exact command / steps:** a script spawning **two real OS processes**
— no
  mocks, real PIDs, real files — that call the same
`_write_claude_wrap_base_url` / `_restore_claude_wrap_base_url` helpers
`wrap claude` uses. The project starts with a real user gateway already
set.
Session A (port 8787) starts, session B (port 8788) starts 0.7s later, A
exits
while B is still running, then B exits. Run identically on `main` and on
this
  branch.

**Before (on `main`) — both bugs visible:**

```text
start                       : {"ANTHROPIC_BASE_URL": "https://my-gateway.example.com"}
  session port=8787 started, remembers previous='https://my-gateway.example.com'
  session port=8788 started, remembers previous='http://127.0.0.1:8787'
both sessions running       : {"ANTHROPIC_BASE_URL": "http://127.0.0.1:8788"}
  session port=8787 exited
after FIRST session exits   : {"ANTHROPIC_BASE_URL": "https://my-gateway.example.com"}
  session port=8788 exited
after LAST session exits    : {"ANTHROPIC_BASE_URL": "http://127.0.0.1:8787"}
```

Session B is still running, but after A exits the proxy URL is gone from
under it
— B is unrouted with no error. And the final state is
`http://127.0.0.1:8787`: a
dead proxy left permanently in the user's project, with their real
gateway lost.

**After (this branch):**

```text
start                       : {"ANTHROPIC_BASE_URL": "https://my-gateway.example.com"}
  session port=8787 started, remembers previous='https://my-gateway.example.com'
  session port=8788 started, remembers previous='http://127.0.0.1:8787'
both sessions running       : {"ANTHROPIC_BASE_URL": "http://127.0.0.1:8788"}
  session port=8787 exited
after FIRST session exits   : {"ANTHROPIC_BASE_URL": "http://127.0.0.1:8788"}
  session port=8788 exited
after LAST session exits    : {"ANTHROPIC_BASE_URL": "https://my-gateway.example.com"}
```

B stays routed after A exits, and the last session out restores the
user's real
gateway.

- **Observed result:** matches the intent on both counts — no unrouting,
no dead
  proxy residue, user's pre-existing URL preserved.
- **Not tested:** Windows (`msvcrt.locking`) — the lock and dead-holder
pruning
  are exercised on POSIX only; the Windows branch is the same code path
`_proxy_start_lock` has shipped with. No live end-to-end run against a
real
Anthropic endpoint with two concurrent `claude` CLIs; the proof above
drives the
same helpers out of two real processes instead. Foundry/Vertex key
variants are
covered by unit tests, not by a live run. Real SIGTERM/SIGHUP delivery
to a
running `wrap claude` was not exercised end to end — the handler's
unwind is
covered by a unit test, and full signal delivery would need a spawned
and
  killed subprocess, which the existing #1768 test also declined to do.

## Runtime Rollout Safety

- **Rollout-managed feature(s):** none — this is an unconditional
correctness fix
  on the wrap settings path.
- **Minimum rollout channel:** n/a.
- **Stable/default behavior changed:** yes, three ways. (1) A wrap
session exiting
while a sibling holds the key now leaves the key in place instead of
removing
  it. (2) SIGTERM/SIGHUP now unwinds, so the child CLI is terminated by
  `subprocess.run`'s cleanup rather than being left running against a
  torn-down proxy. (3) Two new sidecar files appear next to
`settings.local.json`: `.headroom_wrap_owners.json` (removed when the
last
holder exits) and `.headroom_wrap_settings.lock` (retained by design —
deleting
  a live lock file creates an inode-replacement race).
- **Kill switch / disable path:** none. A workspace where the lock file
cannot be
created degrades to the previous unsynchronised behaviour automatically.
- **Unsafe override required:** no.
- **Qualification impact:** none beyond the wrap settings path.
- **Rollback path:** revert the commit; the sidecar files are ignored by
older
  versions and can be deleted safely.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- The ownership record is keyed per env key, so `ANTHROPIC_BASE_URL`,
the
Foundry/Vertex variants and the tool-search entry are tracked
independently.
- Documentation: the behaviour is documented in the helper docstrings
rather than
user-facing docs — the sidecar files are internal state a user never
configures.
- Follow-up worth considering: `.headroom_wrap_settings.lock` is
intentionally
never deleted (matching `_proxy_start_lock`'s retention rationale), so
it stays
in `.claude/` after `unwrap`. Removing it safely needs a separate think
about
  the inode-replacement race.

Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-23 22:36:17 -07:00

254 lines
10 KiB
Python

"""Concurrent `headroom wrap` sessions sharing one project's settings (#3205).
`wrap claude` writes ANTHROPIC_BASE_URL into `.claude/settings.local.json` and
restores it on exit. Several sessions in one project run that read-modify-write
concurrently. The write is atomic so the file never tears, but the updates were
still lost against each other:
* the first session's exit deleted the key while the others were still
running -- they silently stopped routing through the proxy, kept working,
and lost every byte of compression with no error anywhere; and
* a session that started second remembered the *first* session's proxy URL as
"the original", so its exit wrote a dead proxy back into the file, which
every later session in that project then failed to connect to.
"""
from __future__ import annotations
import json
from pathlib import Path
from unittest import mock
import pytest
from headroom.cli import wrap as W
@pytest.fixture
def settings(tmp_path: Path) -> Path:
path = tmp_path / ".claude" / "settings.local.json"
path.parent.mkdir(parents=True)
path.write_text(json.dumps({"env": {"FOO": "bar"}}), encoding="utf-8")
return path
def _env(path: Path) -> dict:
return json.loads(path.read_text(encoding="utf-8")).get("env", {}) if path.exists() else {}
class _Sessions:
"""Drive several wrap sessions with distinct, controllable PIDs."""
def __init__(self, *pids: int) -> None:
self.live = set(pids)
def __enter__(self) -> _Sessions:
self._patches = [
mock.patch.object(W, "_pid_alive", lambda pid: pid in self.live),
mock.patch.object(W, "_identity_mismatch", lambda *a: False),
]
for p in self._patches:
p.start()
return self
def __exit__(self, *exc: object) -> None:
for p in self._patches:
p.stop()
def launch(self, pid: int, url: str, path: Path, port: int | None = None) -> str | None:
with mock.patch("os.getpid", lambda: pid):
return W._write_claude_wrap_base_url(url, settings_path=path, port=port)
def exit(self, pid: int, previous: str | None, path: Path) -> None:
self.live.discard(pid)
with mock.patch("os.getpid", lambda: pid):
W._restore_claude_wrap_base_url(previous, settings_path=path)
def crash(self, pid: int) -> None:
"""Vanish without running cleanup (SIGKILL, hard reboot)."""
self.live.discard(pid)
def test_first_session_exiting_leaves_the_others_routed(settings: Path) -> None:
"""The reported symptom: sessions silently stop routing when a sibling exits."""
with _Sessions(1001, 1002) as s:
a = s.launch(1001, "http://127.0.0.1:8787", settings)
s.launch(1002, "http://127.0.0.1:8788", settings)
s.exit(1001, a, settings)
assert "ANTHROPIC_BASE_URL" in _env(settings), "surviving session was unrouted"
def test_last_session_out_restores_the_true_original(settings: Path) -> None:
"""A later session must not restore an earlier session's dead proxy URL."""
with _Sessions(1001, 1002) as s:
a = s.launch(1001, "http://127.0.0.1:8787", settings)
b = s.launch(1002, "http://127.0.0.1:8788", settings)
s.exit(1001, a, settings)
s.exit(1002, b, settings)
assert _env(settings) == {"FOO": "bar"}, "stale proxy URL left behind"
def test_a_pre_existing_user_base_url_survives_the_whole_cycle(settings: Path) -> None:
"""A URL the project already had is restored, not deleted."""
settings.write_text(
json.dumps({"env": {"ANTHROPIC_BASE_URL": "http://user-proxy:1234"}}), encoding="utf-8"
)
with _Sessions(1001, 1002) as s:
a = s.launch(1001, "http://127.0.0.1:8787", settings)
b = s.launch(1002, "http://127.0.0.1:8788", settings)
s.exit(1001, a, settings)
s.exit(1002, b, settings)
assert _env(settings)["ANTHROPIC_BASE_URL"] == "http://user-proxy:1234"
def test_three_sessions_any_exit_order(settings: Path) -> None:
for order in ([1001, 1002, 1003], [1003, 1001, 1002], [1002, 1003, 1001]):
settings.write_text(json.dumps({"env": {"FOO": "bar"}}), encoding="utf-8")
with _Sessions(*order) as s:
prev = {
pid: s.launch(pid, f"http://127.0.0.1:{8787 + i}", settings)
for i, pid in enumerate(order)
}
for pid in order[:-1]:
s.exit(pid, prev[pid], settings)
assert "ANTHROPIC_BASE_URL" in _env(settings), f"unrouted early in {order}"
s.exit(order[-1], prev[order[-1]], settings)
assert _env(settings) == {"FOO": "bar"}, f"residue after {order}"
def test_a_crashed_session_does_not_wedge_the_key(settings: Path) -> None:
"""A SIGKILLed session never releases; its claim must be pruned as dead."""
with _Sessions(1001, 1002) as s:
s.launch(1001, "http://127.0.0.1:8787", settings)
b = s.launch(1002, "http://127.0.0.1:8788", settings)
s.crash(1001)
s.exit(1002, b, settings)
assert _env(settings) == {"FOO": "bar"}
assert not W._wrap_owners_path(settings).exists()
def test_single_session_behaviour_is_unchanged(settings: Path) -> None:
with _Sessions(1001) as s:
a = s.launch(1001, "http://127.0.0.1:8787", settings)
assert _env(settings)["ANTHROPIC_BASE_URL"] == "http://127.0.0.1:8787"
s.exit(1001, a, settings)
assert _env(settings) == {"FOO": "bar"}
def test_restore_without_an_owner_record_still_honours_the_caller(settings: Path) -> None:
"""unwrap and legacy sessions pass the previous value directly."""
settings.write_text(
json.dumps({"env": {"ANTHROPIC_BASE_URL": "http://127.0.0.1:8787"}}), encoding="utf-8"
)
assert not W._wrap_owners_path(settings).exists()
W._restore_claude_wrap_base_url("http://legacy:9999", settings_path=settings)
assert _env(settings)["ANTHROPIC_BASE_URL"] == "http://legacy:9999"
def test_tool_search_key_is_tracked_independently(settings: Path) -> None:
"""Ownership is per key -- the tool-search entry has the same race."""
with _Sessions(1001, 1002) as s:
with mock.patch("os.getpid", lambda: 1001):
a = W._write_claude_wrap_tool_search("auto", settings_path=settings)
with mock.patch("os.getpid", lambda: 1002):
W._write_claude_wrap_tool_search("auto", settings_path=settings)
s.live.discard(1001)
with mock.patch("os.getpid", lambda: 1001):
W._restore_claude_wrap_tool_search(a, settings_path=settings)
assert W._TOOL_SEARCH_ENV in _env(settings), "surviving session lost tool-search"
def test_exit_on_signal_unwinds_so_finally_can_run() -> None:
"""`cleanup` as the handler never unwound; the settings restore never ran."""
with pytest.raises(SystemExit) as excinfo:
W._exit_on_signal(15, None)
assert excinfo.value.code == 143
def test_unwrap_forces_the_restore_past_a_live_session(settings: Path) -> None:
"""`unwrap` is the user asking for their settings back -- it must not no-op.
Deferring to a live sibling is right for a session exiting on its own, but
unwrap deferring means the command prints success while leaving the proxy
URL in the file.
"""
with _Sessions(1001) as s:
s.launch(1001, "http://127.0.0.1:8787", settings)
with mock.patch("os.getpid", lambda: 2002):
W._restore_claude_wrap_base_url(None, settings_path=settings, force=True)
assert _env(settings) == {"FOO": "bar"}, "unwrap left the proxy URL behind"
assert not W._wrap_owners_path(settings).exists(), "unwrap left ownership state behind"
def test_unwrap_restores_the_true_original_not_the_marker_value(settings: Path) -> None:
"""A caller with no claim of its own trusts the record over its marker.
The single-slot marker is won by the *last* writer, whose `previous` is the
first session's proxy URL -- restoring that is the #3205 bug via unwrap.
"""
settings.write_text(
json.dumps({"env": {"ANTHROPIC_BASE_URL": "http://user-proxy:1234"}}), encoding="utf-8"
)
with _Sessions(1001, 1002) as s:
s.launch(1001, "http://127.0.0.1:8787", settings, port=8787)
s.launch(1002, "http://127.0.0.1:8788", settings, port=8788)
with mock.patch("os.getpid", lambda: 2002):
W._restore_claude_wrap_base_url(
"http://127.0.0.1:8787", settings_path=settings, force=True
)
assert _env(settings)["ANTHROPIC_BASE_URL"] == "http://user-proxy:1234"
def test_a_holder_that_outlived_its_proxy_cannot_veto_the_selfheal(settings: Path) -> None:
"""#2221: a wrapper PID can outlive its proxy; its claim must not block."""
with _Sessions(1001) as s:
s.launch(1001, "http://127.0.0.1:8787", settings, port=8787)
# PID 1001 is still alive, but port 8787 has been proven dead.
W._restore_claude_wrap_base_url(None, settings_path=settings, dead_ports=frozenset({8787}))
assert _env(settings) == {"FOO": "bar"}, "dead proxy URL survived the self-heal"
def test_exiting_session_hands_its_marker_to_a_survivor(settings: Path) -> None:
"""The marker has one slot; the leaver must not strand or hijack it."""
with _Sessions(1001, 1002) as s:
s.launch(1001, "http://127.0.0.1:8787", settings, port=8787)
b = s.launch(1002, "http://127.0.0.1:8788", settings, port=8788)
marker = W._read_wrap_marker(settings)
assert marker is not None and marker["pid"] == 1002, "last writer owns the marker"
s.exit(1002, b, settings)
marker = W._read_wrap_marker(settings)
assert marker is not None, "survivor lost its #2221 self-heal record"
assert marker["pid"] == 1001, "marker still describes the exited session"
assert marker["port"] == 8787
assert marker["previous"] is None, "marker must carry the true original"
def test_the_founding_session_still_honours_an_explicit_previous(settings: Path) -> None:
"""A sole writer observed the pre-wrap value first-hand; do not override it."""
with _Sessions(1001) as s:
s.launch(1001, "http://127.0.0.1:8787", settings)
s.exit(1001, "https://existing-gateway.example.com/v1", settings)
assert _env(settings)["ANTHROPIC_BASE_URL"] == "https://existing-gateway.example.com/v1"