mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
## Description Running Claude Code (Anthropic) and Codex (OpenAI) against the **same** Headroom proxy instance on one port produced incorrect, unstable dashboard data. The proxy core is provider-isolated and multi-provider-safe by design; the defect was in the observability layer. The Codex `/v1/responses` **WebSocket** handler was the only path in the proxy that wrote to the request logger by hand instead of through the unified `emit_request_outcome` funnel, and it did so twice per session close: the per-turn funnel record plus an unconditional cumulative session-summary `RequestLog`. This PR removes the duplicate summary log so Codex WS emits exactly one request log per turn, matching the HTTP provider paths. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Dropped the duplicate cumulative session-summary `RequestLog` in the Codex WS handler while preserving the per-turn `emit_request_outcome` path. - Preserved gated `request_messages` and `turn_id` on residual outcomes so dashboard telemetry keeps the useful attribution without double-counting tokens. - Ensured explicit `--anyllm-provider` wins over a leaked `HEADROOM_ANYLLM_PROVIDER` environment variable. - Registered retry delay settings that had drifted out of the settings registry. - Hardened tests against developer-shell `HEADROOM_*` / `ANTHROPIC_CUSTOM_HEADERS` leakage and stabilized several focused proxy/wrap test fixtures. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ .venv/bin/pytest tests/ -q -p no:cacheprovider 8529 passed, 537 skipped, 5831 warnings in 276.25s (0:04:36) $ .venv/bin/ruff check <touched files> All checks passed! ``` ## Real Behavior Proof - Environment: macOS (Darwin 25.4.0), Python 3.13.14, pytest 9.0.3, ruff via project venv, branch `fix/multi-provider-runtime`. - Exact command / steps: Ran the full test suite without pytest cache provider and Ruff on all touched files; used `git stash` to confirm the stale fake-config failures pre-existed this change. - Observed result: Full suite passed with no failures; Ruff passed; Codex WS now routes end-of-session logging through `emit_request_outcome`, emitting one request log per turn with the same accounting model as Anthropic HTTP turns. - Not tested: Live simultaneous Claude + Codex dashboard run. `mypy headroom` was not run to completion; a scoped run reported one pre-existing `settings_store.py:470` coercion error outside this diff. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A - server-side observability fix; no UI markup changed. ## Additional Notes - The proxy's multi-provider routing, header/auth isolation, and per-model cache keying are already correct and unchanged here; only the WS observability write path was double-counting. - Architectural assessment: `plans/reports/research-260714-0004-multi-provider-upstream-compression-report.md`; root-cause + resolution trail: `plans/reports/debug-assessment-260714-0011-dashboard-instability-mixed-claude-codex-report.md`. - No live simultaneous Claude + Codex dashboard run was performed; validation is from test coverage and code review of the WS logging path. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
57 lines
1.9 KiB
Python
57 lines
1.9 KiB
Python
"""Prove env > file > default precedence in a genuinely separate process.
|
|
|
|
A mocked ``restart_current_deployment`` proves dispatch only -- it can't
|
|
prove settings actually take effect the way a real restarted proxy would
|
|
pick them up. This spawns a real subprocess that imports
|
|
``headroom.settings_store`` cold and reports what it observes, closing the
|
|
gap a Codex red-team pass flagged in the original (mock-only) test plan.
|
|
"""
|
|
|
|
import json
|
|
import os
|
|
import subprocess
|
|
import sys
|
|
|
|
import pytest
|
|
|
|
from headroom import paths, settings_store
|
|
|
|
|
|
@pytest.fixture
|
|
def workspace(tmp_path, monkeypatch):
|
|
"""Point the workspace dir (settings.json) at an isolated tmp dir."""
|
|
monkeypatch.setenv(paths.HEADROOM_WORKSPACE_DIR_ENV, str(tmp_path))
|
|
monkeypatch.delenv(paths.HEADROOM_SETTINGS_PATH_ENV, raising=False)
|
|
return tmp_path
|
|
|
|
|
|
def test_env_beats_file_beats_default_in_subprocess(workspace, monkeypatch):
|
|
for field in settings_store.SETTINGS:
|
|
monkeypatch.delenv(field.env, raising=False)
|
|
settings_store.save({"target_ratio": 0.3, "rpm": 20})
|
|
|
|
script = (
|
|
"import os, json\n"
|
|
"from headroom import settings_store\n"
|
|
"settings_store.apply_to_environ(settings_store.load())\n"
|
|
"print(json.dumps({'rpm': os.environ.get('HEADROOM_RPM'), "
|
|
"'target_ratio': os.environ.get('HEADROOM_TARGET_RATIO')}))\n"
|
|
)
|
|
env = dict(os.environ)
|
|
env["HEADROOM_WORKSPACE_DIR"] = str(workspace)
|
|
env["HEADROOM_RPM"] = "999" # explicit export: must win over the file's 20
|
|
env.pop(
|
|
"HEADROOM_TARGET_RATIO", None
|
|
) # not exported: file's 0.3 must win over the code default
|
|
|
|
result = subprocess.run(
|
|
[sys.executable, "-c", script],
|
|
capture_output=True,
|
|
text=True,
|
|
env=env,
|
|
timeout=30,
|
|
)
|
|
assert result.returncode == 0, result.stderr
|
|
out = json.loads(result.stdout)
|
|
assert out["rpm"] == "999"
|
|
assert out["target_ratio"] == "0.3"
|