mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
fix(proxy): return 502, not 200, when upstream connect retries are exhausted (#3083)
## Description When every connect retry to the upstream API fails, `_stream_response_inner` synthesizes its own SSE error response (added in #1639, so an h2 `StreamReset` wouldn't surface as an unhandled 502). It was built without a `status_code`, so Starlette defaulted it to **200**. A 200 carrying a lone `event: error` frame and no `message_start` is indistinguishable, to every Anthropic/OpenAI SDK, from a successful stream that produced no events. Claude Code reports: ``` API Error: API returned an empty or malformed response (HTTP 200) - check for a proxy or gateway intercepting the request ``` The client also cannot recover, because 200 is not a retryable status. **It does not self-heal.** Compression fails open on timeout, so the proxy forwards the full uncompressed body; the client retries, re-sends the same oversized payload, hits the same transport failure, and gets another 200. The session is stuck until the client is pointed away from the proxy. Related — same *symptom*, different root cause, so this closes none of them: #3040, #3055, #3019, #2952 (CCR buffered-stream conversion), #3071, #3017. Worth noting that #3040 ("first messages succeed, fails after several turns", closed `NOT_PLANNED`) matches this failure's shape exactly. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [x] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) Marked breaking because the status code on this path changes 200 to 502. See **Runtime Rollout Safety**. ## Changes Made - `handlers/streaming.py` — the synthesized transport-error response now returns **502**. The structured SSE body is unchanged for clients that read it. No body byte has been forwarded at that point, so the status line is still ours to set. - `prometheus_metrics.py` — new `headroom_upstream_connection_errors_total{provider}`. This path forwards no upstream status, so there was nothing to attribute the failure to in `/metrics`; it survived only as a log line. Mirrors `record_compression_failed` and takes the same `_obs_counter_lock`. - `server.py` — `HEADROOM_LOG_LEVEL` for uvicorn's level, previously hardcoded to `"warning"` with no env var and no CLI flag. Default unchanged. An unrecognized value warns and falls back rather than raising (uvicorn raises `KeyError` on unknown levels). - `docs/content/docs/proxy.mdx` — documents the new env var in the Observability table. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed `test_stream_reset_exhaustion_yields_sse_error_not_crash` asserted the SSE body but never the status — which is how the 200 survived. Added a test that pins the status specifically, a happy-path guard, and coverage for the counter and the env-var resolver. ### Test Output ```text $ python -m pytest tests/test_h2_stream_reset_retry.py tests/test_prometheus_obs_counters.py tests/test_uvicorn_log_level_env.py -q 29 passed in 5.26s $ python -m ruff check . All checks passed! $ python -m ruff format --check . 1506 files already formatted $ python -m mypy headroom/proxy/handlers/streaming.py headroom/proxy/prometheus_metrics.py headroom/proxy/server.py Success: no issues found in 3 source files # Fails before the fix (status_code=502 line removed, nothing else changed): $ python -m pytest tests/test_h2_stream_reset_retry.py -k status_is_not_200 assert result.status_code == 502 E assert 200 == 502 FAILED tests/test_h2_stream_reset_retry.py::test_stream_reset_exhaustion_status_is_not_200 1 failed, 5 deselected in 1.28s ``` Broader regression run (181 passed): `test_h2_stream_reset_retry`, `test_prometheus_obs_counters`, `test_uvicorn_log_level_env`, `test_prometheus_label_escaping`, `test_observability_metrics`, `test_prometheus_stage_timing_concurrency`, `test_proxy_streaming_ratelimit_headers`, `test_proxy_retry_429`, `test_proxy_byte_faithful_forwarding`, `test_ws_http_fallback`, `test_mid_turn_steering`, `test_proxy_anthropic_cache_stability`. ## Real Behavior Proof - Environment: Windows 11, Python 3.13.15, headroom @ this branch. Genuine `create_app()` FastAPI app under real uvicorn — no mocks, no TestClient. Upstream pinned to `http://127.0.0.1:59999` (a closed port), so every connect attempt is a real TCP refusal, producing a real `httpx.ConnectError` (an `httpx.TransportError`) into the branch under test. `retry_max_attempts=2`. - Exact command / steps: boot the real app with `HEADROOM_LOG_LEVEL=info` and `ProxyConfig(anthropic_api_url="http://127.0.0.1:59999")`, POST a `stream:true` request to `/v1/messages`, then scrape `/metrics`. Verbatim commands below. - Observed result: `HTTP_STATUS=502` (previously 200), structured SSE error body intact, `headroom_upstream_connection_errors_total{provider="anthropic"} 1`, and a uvicorn access line present only because `HEADROOM_LOG_LEVEL=info` was honored. Verbatim output below. - Not tested: the h2 `StreamReset` variant specifically — reproduced via `ConnectError`, a sibling `httpx.TransportError` travelling the identical code path (the existing `test_stream_reset_exhaustion_*` tests cover `RemoteProtocolError` at unit level). Not exercised against the OpenAI, Gemini, or Bedrock streaming handlers, which have their own error paths. No load or concurrency testing. Commands run after the patch: ```bash # boot the real app with a dead upstream and the new env var set HEADROOM_LOG_LEVEL=info python run_proxy_proof.py # ProxyConfig(anthropic_api_url="http://127.0.0.1:59999") curl -s -o resp.txt -w "HTTP_STATUS=%{http_code}\ncontent_type=%{content_type}\n" \ http://127.0.0.1:8799/v1/messages \ -H "content-type: application/json" \ -H "x-api-key: proof-key" \ -H "anthropic-version: 2023-06-01" \ -d @request.json # {"model":"claude-opus-5","max_tokens":64,"stream":true,"messages":[...]} ``` After-fix evidence: ```text PROOF: HEADROOM_LOG_LEVEL='info' -> uvicorn log_level='info' PROOF: upstream pinned to http://127.0.0.1:59999 (closed port) HTTP_STATUS=502 content_type=text/event-stream; charset=utf-8 event: error data: {"type": "error", "error": {"type": "connection_error", "message": "Failed to connect to upstream API: All connection attempts failed"}} ``` ```text $ curl -s http://127.0.0.1:8799/metrics | grep upstream_connection_errors # HELP headroom_upstream_connection_errors_total Exhausted-retries upstream transport failures by provider; the proxy answered 502 itself because no upstream response arrived # TYPE headroom_upstream_connection_errors_total counter headroom_upstream_connection_errors_total{provider="anthropic"} 1 ``` ```text # uvicorn access log — present only because HEADROOM_LOG_LEVEL=info was honored: INFO: 127.0.0.1:62472 - "POST /v1/messages HTTP/1.1" 502 Bad Gateway INFO: 127.0.0.1:62479 - "GET /metrics HTTP/1.1" 200 OK ``` All three changes are exercised end to end: the status is 502, the structured body survives, the counter increments, and the env var takes effect. Separately, this ran against a real deployment: the fix is live on a self-hosted proxy at `0.35.1-alpha.3` (Azure Container Apps, Cloudflare in front), where the original HTTP 200 was first observed against `0.35.1-alpha.1`. ## Runtime Rollout Safety - Rollout-managed feature(s): none — unconditional bug fix, no flag. - Minimum rollout channel: n/a — ships with the change. - Stable/default behavior changed: yes. This path returns 502 instead of 200. `HEADROOM_LOG_LEVEL` and the new counter both default to current behavior (`warning`; the counter is absent from `/metrics` until the first occurrence). - Kill switch / disable path: none. Happy to add an env guard if you would prefer it staged, though a 200 on this path is never correct. - Unsafe override required: no. - Qualification impact: any client treating the synthesized 200 as success now sees a 5xx. That is the fix — such a client was silently accepting a truncated response. Retry-on-5xx logic in the Anthropic and OpenAI SDKs will now retry a transient transport failure, which is the intended behavior. - Rollback path: revert the commit; single and self-contained. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A — terminal output above. ## Additional Notes **Scope.** Three changes in one PR, against the "one logical change" guidance. They share a single root cause: this bug was only findable by reading `/metrics`, because the failing path emitted no status, no counter, and (see below) no usable log line. The counter and the env var are the observability that should have made it a five-minute diagnosis instead of a forensic exercise. Happy to split the `HEADROOM_LOG_LEVEL` change into its own PR if you would rather keep the fix minimal — just say so. **Related defect, filed separately as #3087.** While producing the proof above I found that the proxy's own `logger.error("Connection error to upstream API: ...")` never reaches stdout: that run produced **zero** `headroom.proxy` logger lines, only uvicorn's own. Root cause is `_setup_file_logging()` setting `propagate = False` on the `headroom` logger (`helpers.py:1536`), which sends every application record to `~/.headroom/logs/proxy.log` and nowhere else — invisible in any container, where stdout is the log channel. That is precisely why this PR adds a counter rather than trusting a log line. Not fixed here: the right remedy is a maintainer call, so it is written up in #3087 with a repro rather than folded into this PR. **No dependency changes.** The dead-upstream harness used for the proof above is ~25 lines (`ProxyConfig(anthropic_api_url="http://127.0.0.1:59999")` + `uvicorn.run(create_app(config))`); happy to contribute it as an e2e test if that is useful. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Serge ARADJ
GitHub
parent
b88b9078d8
commit
a3d9424de9
7 changed files with 197 additions and 4 deletions
|
|
@ -284,6 +284,7 @@ Rewrite the upstream model per request — for example, send small, tool-free ca
|
|||
| `--telemetry` / `HEADROOM_TELEMETRY` | off | **Local-only** usage stats for your own `/stats`, `/metrics`, and dashboard. Nothing leaves the machine. |
|
||||
| `--log-file` / `HEADROOM_LOG_FILE` | none | JSONL request/response log. |
|
||||
| `--log-messages` | `false` | Include full message bodies in the log (may contain sensitive data). |
|
||||
| `HEADROOM_LOG_LEVEL` | `warning` | uvicorn's log level (`critical`, `error`, `warning`, `info`, `debug`, `trace`). Raise to `info` for the per-request access log when diagnosing a deployed proxy. An unrecognized value warns and falls back to `warning`. |
|
||||
| `HEADROOM_OTEL_METRICS_ENABLED` | `false` | Export OpenTelemetry metrics (`HEADROOM_OTEL_METRICS_ENDPOINT`, …). See [OTLP export](/docs/metrics#opentelemetry-otlp-export). |
|
||||
| `HEADROOM_LANGFUSE_ENABLED` | `false` | Emit Langfuse traces (`LANGFUSE_PUBLIC_KEY` / `LANGFUSE_SECRET_KEY`). |
|
||||
|
||||
|
|
|
|||
|
|
@ -1340,11 +1340,22 @@ class StreamingMixin:
|
|||
if upstream_response is None:
|
||||
raise last_connect_error or RuntimeError("upstream connection did not start")
|
||||
# Retries exhausted (or a transport failure escaped the loop): emit a
|
||||
# clean SSE error instead of letting an h2 StreamReset bubble up as a
|
||||
# 502. Covers ConnectError/timeouts and Local/RemoteProtocolError. (#1639)
|
||||
# clean SSE error instead of letting an h2 StreamReset bubble up as an
|
||||
# unhandled 502. Covers ConnectError/timeouts and Local/RemoteProtocol-
|
||||
# Error. (#1639)
|
||||
#
|
||||
# The status MUST stay non-2xx. No body byte has been forwarded yet, so
|
||||
# the status line is still ours to set, and a 200 here is
|
||||
# indistinguishable — to every Anthropic/OpenAI SDK — from a successful
|
||||
# stream that produced no events: the client reports "empty or malformed
|
||||
# response (HTTP 200)" and cannot retry, because 200 is not retryable.
|
||||
# 502 keeps the structured SSE body for clients that read it while
|
||||
# letting SDK retry logic treat a transient upstream transport failure
|
||||
# as what it is.
|
||||
except httpx.TransportError as e:
|
||||
error_msg = str(e) or repr(e)
|
||||
logger.error(f"[{request_id}] Connection error to upstream API: {error_msg}")
|
||||
self.metrics.record_upstream_connection_error(provider)
|
||||
|
||||
async def _error_gen():
|
||||
error_event = {
|
||||
|
|
@ -1357,7 +1368,11 @@ class StreamingMixin:
|
|||
yield f"event: error\ndata: {json.dumps(error_event)}\n\n".encode()
|
||||
|
||||
self._cleanup_mid_turn_stream(session_key)
|
||||
return StreamingResponse(_error_gen(), media_type="text/event-stream")
|
||||
return StreamingResponse(
|
||||
_error_gen(),
|
||||
status_code=502,
|
||||
media_type="text/event-stream",
|
||||
)
|
||||
|
||||
# Capture Codex rate-limit window data from the upstream response
|
||||
# headers, for *every* status. Codex (gpt-5.x) almost always streams, so
|
||||
|
|
|
|||
|
|
@ -152,6 +152,14 @@ class PrometheusMetrics:
|
|||
# whether the compression budget is too tight vs. a real bug.
|
||||
self.compression_failed_by_reason: dict[str, int] = defaultdict(int)
|
||||
|
||||
# Upstream transport failures on the streaming path, keyed by provider.
|
||||
# Raised when every connect retry is exhausted and the proxy synthesizes
|
||||
# its own error response instead of forwarding an upstream status. That
|
||||
# path emits no upstream status code to attribute the failure to, so
|
||||
# without this counter it is invisible in metrics and survives only as a
|
||||
# log line.
|
||||
self.upstream_connection_errors_by_provider: dict[str, int] = defaultdict(int)
|
||||
|
||||
# Kompress size-gate outcomes, keyed by outcome ("within",
|
||||
# "exceeded"). The gate routes oversized blocks away from ML
|
||||
# compression (ContentRouter._kompress_max_tokens, #1171). This
|
||||
|
|
@ -355,6 +363,7 @@ class PrometheusMetrics:
|
|||
self.savings_by_source.clear()
|
||||
with self._obs_counter_lock:
|
||||
self.compression_failed_by_reason.clear()
|
||||
self.upstream_connection_errors_by_provider.clear()
|
||||
self.kompress_size_gate_by_outcome.clear()
|
||||
self.compression_quarantine_by_event.clear()
|
||||
|
||||
|
|
@ -568,6 +577,18 @@ class PrometheusMetrics:
|
|||
with self._obs_counter_lock:
|
||||
self.compression_failed_by_reason[reason or "error"] += 1
|
||||
|
||||
def record_upstream_connection_error(self, provider: str) -> None:
|
||||
"""Record one exhausted-retries upstream transport failure.
|
||||
|
||||
Called from the streaming handler's ``httpx.TransportError`` fallback
|
||||
(handlers/streaming.py), where the proxy synthesizes its own 502
|
||||
because no upstream response ever arrived. Guarded by
|
||||
``_obs_counter_lock`` for the same reason as
|
||||
``record_compression_failed``.
|
||||
"""
|
||||
with self._obs_counter_lock:
|
||||
self.upstream_connection_errors_by_provider[provider or "unknown"] += 1
|
||||
|
||||
def record_kompress_size_gate(self, outcome: str) -> None:
|
||||
"""Record one kompress size-gate decision, bucketed by ``outcome``.
|
||||
|
||||
|
|
@ -1404,9 +1425,23 @@ class PrometheusMetrics:
|
|||
# then format outside it (see _obs_counter_lock).
|
||||
with self._obs_counter_lock:
|
||||
compression_failed = dict(self.compression_failed_by_reason)
|
||||
upstream_conn_errors = dict(self.upstream_connection_errors_by_provider)
|
||||
kompress_size_gate = dict(self.kompress_size_gate_by_outcome)
|
||||
compression_quarantine = dict(self.compression_quarantine_by_event)
|
||||
|
||||
if upstream_conn_errors:
|
||||
lines.extend(
|
||||
[
|
||||
"# HELP headroom_upstream_connection_errors_total Exhausted-retries upstream transport failures by provider; the proxy answered 502 itself because no upstream response arrived",
|
||||
"# TYPE headroom_upstream_connection_errors_total counter",
|
||||
]
|
||||
)
|
||||
for prov, count in upstream_conn_errors.items():
|
||||
lines.append(
|
||||
f'headroom_upstream_connection_errors_total{{provider="{_escape_label_value(prov)}"}} {count}'
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
if compression_failed:
|
||||
lines.extend(
|
||||
[
|
||||
|
|
|
|||
|
|
@ -5485,11 +5485,18 @@ def run_server(
|
|||
else:
|
||||
app_target = create_app(config)
|
||||
|
||||
# "warning" keeps the default terminal quiet (uvicorn's access log is one line
|
||||
# per request). It was previously hardcoded, which left deployed proxies with
|
||||
# no way to turn request logging on: operators diagnosing a production
|
||||
# incident could not see uvicorn's view of the traffic at all, with no env var
|
||||
# and no CLI flag to change it. Overridable now; the default is unchanged.
|
||||
uvicorn_log_level = _resolve_uvicorn_log_level()
|
||||
|
||||
uvicorn.run(
|
||||
app_target,
|
||||
host=config.host,
|
||||
port=config.port,
|
||||
log_level="warning",
|
||||
log_level=uvicorn_log_level,
|
||||
workers=workers if workers > 1 else None, # None = single process (default)
|
||||
limit_concurrency=limit_concurrency,
|
||||
# Defense-in-depth: the loopback guard for /debug/* endpoints trusts
|
||||
|
|
@ -5556,6 +5563,30 @@ def _get_env_str(name: str, default: str) -> str:
|
|||
return os.environ.get(name, default)
|
||||
|
||||
|
||||
# uvicorn rejects anything outside this set with a KeyError during startup, so an
|
||||
# operator typo in HEADROOM_LOG_LEVEL must not be able to stop the proxy booting.
|
||||
_UVICORN_LOG_LEVELS = frozenset({"critical", "error", "warning", "info", "debug", "trace"})
|
||||
_UVICORN_LOG_LEVEL_DEFAULT = "warning"
|
||||
|
||||
|
||||
def _resolve_uvicorn_log_level() -> str:
|
||||
"""Resolve uvicorn's log level from ``HEADROOM_LOG_LEVEL``.
|
||||
|
||||
Falls back to the previous hardcoded default on an unset or unrecognized
|
||||
value, warning loudly rather than failing the boot.
|
||||
"""
|
||||
raw = _get_env_str("HEADROOM_LOG_LEVEL", _UVICORN_LOG_LEVEL_DEFAULT).strip().lower()
|
||||
if raw in _UVICORN_LOG_LEVELS:
|
||||
return raw
|
||||
logger.warning(
|
||||
"Ignoring unrecognized HEADROOM_LOG_LEVEL=%r; using %r. Valid values: %s",
|
||||
raw,
|
||||
_UVICORN_LOG_LEVEL_DEFAULT,
|
||||
", ".join(sorted(_UVICORN_LOG_LEVELS)),
|
||||
)
|
||||
return _UVICORN_LOG_LEVEL_DEFAULT
|
||||
|
||||
|
||||
def _parse_exclude_tools(cli_excludes: str | None) -> set[str]:
|
||||
"""Parse extra never-compress tool names from CLI args and env var.
|
||||
|
||||
|
|
|
|||
|
|
@ -30,6 +30,7 @@ def _mock_proxy():
|
|||
proxy._config.retry_max_delay_ms = 0
|
||||
proxy.config = proxy._config
|
||||
proxy.memory_handler = None
|
||||
proxy.metrics = MagicMock()
|
||||
proxy._parse_sse_usage_from_buffer = MagicMock(return_value=None)
|
||||
proxy._finalize_stream_response = AsyncMock(return_value=None)
|
||||
return proxy
|
||||
|
|
@ -142,3 +143,37 @@ async def test_stream_reset_exhaustion_yields_sse_error_not_crash():
|
|||
assert proxy.http_client.send.await_count == 2
|
||||
assert b"event: error" in body
|
||||
assert b"connection_error" in body
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_stream_reset_exhaustion_status_is_not_200():
|
||||
"""The synthesized error response must not claim success.
|
||||
|
||||
A 200 here is indistinguishable, to every Anthropic/OpenAI SDK, from a
|
||||
successful stream that produced no events — the client reports "empty or
|
||||
malformed response (HTTP 200)" and cannot retry, because 200 is not a
|
||||
retryable status. This asserts the status line specifically: the body
|
||||
assertions above passed for the entire time the status was 200.
|
||||
"""
|
||||
proxy = _mock_proxy()
|
||||
proxy.http_client.build_request = MagicMock(return_value=MagicMock())
|
||||
proxy.http_client.send = AsyncMock(side_effect=httpx.RemoteProtocolError("reset"))
|
||||
|
||||
result = await _run_stream(proxy)
|
||||
|
||||
assert result.status_code == 502
|
||||
proxy.metrics.record_upstream_connection_error.assert_called_once()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_successful_stream_still_returns_200():
|
||||
"""Guard the happy path against the 502 change above."""
|
||||
proxy = _mock_proxy()
|
||||
good = _good_stream_response([b'event: message_stop\ndata: {"type":"message_stop"}\n\n'])
|
||||
proxy.http_client.build_request = MagicMock(return_value=MagicMock())
|
||||
proxy.http_client.send = AsyncMock(return_value=good)
|
||||
|
||||
result = await _run_stream(proxy)
|
||||
|
||||
assert result.status_code == 200
|
||||
proxy.metrics.record_upstream_connection_error.assert_not_called()
|
||||
|
|
|
|||
|
|
@ -8,6 +8,8 @@ Covers the related counters added to ``PrometheusMetrics``:
|
|||
via the observer hook, split into "exceeded" vs "within".
|
||||
* ``headroom_compression_quarantine_total{event}`` — records quarantine
|
||||
activation and immediate executor skips while a timed-out worker remains.
|
||||
* ``headroom_upstream_connection_errors_total{provider}`` — recorded when the
|
||||
streaming path exhausts its connect retries and answers 502 itself.
|
||||
|
||||
Imports only the metrics module so the test stays free of heavy ML deps.
|
||||
"""
|
||||
|
|
@ -41,6 +43,35 @@ def test_record_compression_failed_empty_reason_defaults_to_error() -> None:
|
|||
assert metrics.compression_failed_by_reason["error"] == 1
|
||||
|
||||
|
||||
def test_record_upstream_connection_error_buckets_by_provider() -> None:
|
||||
metrics = PrometheusMetrics()
|
||||
|
||||
metrics.record_upstream_connection_error("anthropic")
|
||||
metrics.record_upstream_connection_error("openai")
|
||||
metrics.record_upstream_connection_error("openai")
|
||||
|
||||
assert metrics.upstream_connection_errors_by_provider["anthropic"] == 1
|
||||
assert metrics.upstream_connection_errors_by_provider["openai"] == 2
|
||||
|
||||
|
||||
def test_record_upstream_connection_error_empty_provider_defaults_to_unknown() -> None:
|
||||
metrics = PrometheusMetrics()
|
||||
|
||||
metrics.record_upstream_connection_error("")
|
||||
|
||||
assert metrics.upstream_connection_errors_by_provider["unknown"] == 1
|
||||
|
||||
|
||||
async def test_upstream_connection_errors_exported_in_prometheus_text() -> None:
|
||||
metrics = PrometheusMetrics()
|
||||
metrics.record_upstream_connection_error("anthropic")
|
||||
|
||||
text = await metrics.export()
|
||||
|
||||
assert "# TYPE headroom_upstream_connection_errors_total counter" in text
|
||||
assert 'headroom_upstream_connection_errors_total{provider="anthropic"} 1' in text
|
||||
|
||||
|
||||
def test_record_kompress_size_gate_buckets_by_outcome() -> None:
|
||||
metrics = PrometheusMetrics()
|
||||
|
||||
|
|
|
|||
45
tests/test_uvicorn_log_level_env.py
Normal file
45
tests/test_uvicorn_log_level_env.py
Normal file
|
|
@ -0,0 +1,45 @@
|
|||
"""HEADROOM_LOG_LEVEL resolution for uvicorn's log level.
|
||||
|
||||
uvicorn's level was previously hardcoded to "warning", so a deployed proxy had
|
||||
no way to turn request logging on — no env var, no CLI flag. These tests pin the
|
||||
override and, more importantly, pin that an operator typo cannot stop the proxy
|
||||
booting (uvicorn raises on an unknown level).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from headroom.proxy.server import _resolve_uvicorn_log_level
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clear_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.delenv("HEADROOM_LOG_LEVEL", raising=False)
|
||||
|
||||
|
||||
def test_defaults_to_warning_when_unset() -> None:
|
||||
assert _resolve_uvicorn_log_level() == "warning"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("level", ["critical", "error", "warning", "info", "debug", "trace"])
|
||||
def test_accepts_every_uvicorn_level(level: str, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setenv("HEADROOM_LOG_LEVEL", level)
|
||||
|
||||
assert _resolve_uvicorn_log_level() == level
|
||||
|
||||
|
||||
def test_normalizes_case_and_whitespace(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setenv("HEADROOM_LOG_LEVEL", " INFO \n")
|
||||
|
||||
assert _resolve_uvicorn_log_level() == "info"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("bad", ["verbose", "WARN", "", "10"])
|
||||
def test_unrecognized_value_falls_back_instead_of_raising(
|
||||
bad: str, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""A typo must degrade to the old default, never fail the boot."""
|
||||
monkeypatch.setenv("HEADROOM_LOG_LEVEL", bad)
|
||||
|
||||
assert _resolve_uvicorn_log_level() == "warning"
|
||||
Loading…
Add table
Add a link
Reference in a new issue