headroom/tests/test_proxy/test_settings_fresh_process_precedence.py
nangsontay e5b3a634df
fix(proxy): dedupe Codex WS request logging for accurate mixed-provider dashboards (#2189)
## Description

Running Claude Code (Anthropic) and Codex (OpenAI) against the **same**
Headroom proxy instance on one port produced incorrect, unstable
dashboard data. The proxy core is provider-isolated and
multi-provider-safe by design; the defect was in the observability
layer. The Codex `/v1/responses` **WebSocket** handler was the only path
in the proxy that wrote to the request logger by hand instead of through
the unified `emit_request_outcome` funnel, and it did so twice per
session close: the per-turn funnel record plus an unconditional
cumulative session-summary `RequestLog`. This PR removes the duplicate
summary log so Codex WS emits exactly one request log per turn, matching
the HTTP provider paths.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Dropped the duplicate cumulative session-summary `RequestLog` in the
Codex WS handler while preserving the per-turn `emit_request_outcome`
path.
- Preserved gated `request_messages` and `turn_id` on residual outcomes
so dashboard telemetry keeps the useful attribution without
double-counting tokens.
- Ensured explicit `--anyllm-provider` wins over a leaked
`HEADROOM_ANYLLM_PROVIDER` environment variable.
- Registered retry delay settings that had drifted out of the settings
registry.
- Hardened tests against developer-shell `HEADROOM_*` /
`ANTHROPIC_CUSTOM_HEADERS` leakage and stabilized several focused
proxy/wrap test fixtures.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ .venv/bin/pytest tests/ -q -p no:cacheprovider
8529 passed, 537 skipped, 5831 warnings in 276.25s (0:04:36)

$ .venv/bin/ruff check <touched files>
All checks passed!
```

## Real Behavior Proof

- Environment: macOS (Darwin 25.4.0), Python 3.13.14, pytest 9.0.3, ruff
via project venv, branch `fix/multi-provider-runtime`.
- Exact command / steps: Ran the full test suite without pytest cache
provider and Ruff on all touched files; used `git stash` to confirm the
stale fake-config failures pre-existed this change.
- Observed result: Full suite passed with no failures; Ruff passed;
Codex WS now routes end-of-session logging through
`emit_request_outcome`, emitting one request log per turn with the same
accounting model as Anthropic HTTP turns.
- Not tested: Live simultaneous Claude + Codex dashboard run. `mypy
headroom` was not run to completion; a scoped run reported one
pre-existing `settings_store.py:470` coercion error outside this diff.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A - server-side observability fix; no UI markup changed.

## Additional Notes

- The proxy's multi-provider routing, header/auth isolation, and
per-model cache keying are already correct and unchanged here; only the
WS observability write path was double-counting.
- Architectural assessment:
`plans/reports/research-260714-0004-multi-provider-upstream-compression-report.md`;
root-cause + resolution trail:
`plans/reports/debug-assessment-260714-0011-dashboard-instability-mixed-claude-codex-report.md`.
- No live simultaneous Claude + Codex dashboard run was performed;
validation is from test coverage and code review of the WS logging path.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-07-15 18:18:34 +00:00

57 lines
1.9 KiB
Python

"""Prove env > file > default precedence in a genuinely separate process.
A mocked ``restart_current_deployment`` proves dispatch only -- it can't
prove settings actually take effect the way a real restarted proxy would
pick them up. This spawns a real subprocess that imports
``headroom.settings_store`` cold and reports what it observes, closing the
gap a Codex red-team pass flagged in the original (mock-only) test plan.
"""
import json
import os
import subprocess
import sys
import pytest
from headroom import paths, settings_store
@pytest.fixture
def workspace(tmp_path, monkeypatch):
"""Point the workspace dir (settings.json) at an isolated tmp dir."""
monkeypatch.setenv(paths.HEADROOM_WORKSPACE_DIR_ENV, str(tmp_path))
monkeypatch.delenv(paths.HEADROOM_SETTINGS_PATH_ENV, raising=False)
return tmp_path
def test_env_beats_file_beats_default_in_subprocess(workspace, monkeypatch):
for field in settings_store.SETTINGS:
monkeypatch.delenv(field.env, raising=False)
settings_store.save({"target_ratio": 0.3, "rpm": 20})
script = (
"import os, json\n"
"from headroom import settings_store\n"
"settings_store.apply_to_environ(settings_store.load())\n"
"print(json.dumps({'rpm': os.environ.get('HEADROOM_RPM'), "
"'target_ratio': os.environ.get('HEADROOM_TARGET_RATIO')}))\n"
)
env = dict(os.environ)
env["HEADROOM_WORKSPACE_DIR"] = str(workspace)
env["HEADROOM_RPM"] = "999" # explicit export: must win over the file's 20
env.pop(
"HEADROOM_TARGET_RATIO", None
) # not exported: file's 0.3 must win over the code default
result = subprocess.run(
[sys.executable, "-c", script],
capture_output=True,
text=True,
env=env,
timeout=30,
)
assert result.returncode == 0, result.stderr
out = json.loads(result.stdout)
assert out["rpm"] == "999"
assert out["target_ratio"] == "0.3"