akadmin/headroom
1
0
Fork 0
mirror of https://github.com/headroomlabs-ai/headroom.git synced 2026-08-27 14:17:10 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions

feat(proxy): show real upstream provider on dashboard for OpenAI-compatible endpoints (#1594)

Browse source 
## Description

When the proxy runs against a custom OpenAI-compatible endpoint via
`--openai-api-url` (OpenRouter, Groq, Together, Azure OpenAI, …), the
dashboard always showed the provider as **OpenAI**, because the OpenAI
handler records every request with `provider="openai"`.

This detects well-known upstreams from the `--openai-api-url` host and
adds a `--provider-name` override that takes precedence (the issue's
option 3). The label is resolved only where the dashboard/stats payload
is built — the internal provider key stays `openai`, so pricing and
request formatting are unaffected.

| Upstream URL | Provider shown |
|--------------|----------------|
| `https://api.openai.com/v1` | OpenAI |
| `https://openrouter.ai/api/v1` | OpenRouter |
| `https://api.groq.com/openai/v1` | Groq |
| `https://api.together.xyz/v1` | Together AI |
| `https://<resource>.openai.azure.com/` | Azure OpenAI |

Unknown hosts keep the `openai` label unless `--provider-name` is set.

Closes #1533

## Type of Change

- [x] New feature (non-breaking change that adds functionality)

## Changes Made

- `helpers.py`: `classify_openai_upstream()` (host → display name) +
`resolve_display_provider()` (precedence: `--provider-name` > host
detection > raw provider; only relabels `openai`).
- `models.py`: `ProxyConfig.provider_name`.
- `cli/proxy.py`: `--provider-name` flag, threaded into `ProxyConfig`.
- `server.py`: relabel at the four dashboard/stats display sites (recent
requests, transformations feed, `requests.by_provider`, agent-usage
breakdown) via the resolver / `_remap_provider_counts`. Stored logs and
metrics keys are untouched.
- `docs/content/docs/proxy.mdx`: document `--provider-name`.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] New tests added

### Test Output

```text
$ pytest tests/test_provider_display_classification.py tests/test_dashboard_agent_usage.py -q
16 passed
13 passed

$ ruff check headroom/proxy/helpers.py headroom/proxy/server.py headroom/proxy/models.py headroom/cli/proxy.py
All checks passed!
```

## Real Behavior Proof

- Environment: repo branch `feat/1533-upstream-provider-classify` @
HEAD, local `.venv` (Python 3)
- Exact command / steps: ran the helpers directly from the venv —
`python -c "from headroom.proxy.helpers import classify_openai_upstream,
resolve_display_provider;
print(classify_openai_upstream('https://openrouter.ai/api/v1'));
print(resolve_display_provider('openai',
openai_api_url='https://openrouter.ai/api/v1'));
print(resolve_display_provider('openai',
openai_api_url='https://openrouter.ai/api/v1', provider_name='Groq'));
print(resolve_display_provider('anthropic'))"`
- Observed result: host detection relabels `openai` → `OpenRouter`,
`--provider-name` overrides detection (`Groq`), and the `anthropic`
label (plus the `openai` pricing key) is unchanged. Full output below:
  ```text
  classify openrouter           -> OpenRouter
  resolve openai+openrouter url -> OpenRouter
  override provider-name        -> Groq
  anthropic untouched           -> anthropic
  ```
- Not tested: live dashboard render against a real OpenRouter key (the
payload-builder logic is covered by the unit tests above).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
This commit is contained in:
Manmit Singh 2026-07-16 00:39:56 +05:30 • committed by GitHub
parent 7a5d8a7ace
commit 996c1174a8
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
7 changed files with 193 additions and 6383 deletions
Download patch file Download diff file Expand all files Collapse all files

1
docs/content/docs/proxy.mdx
View file

@ -45,6 +45,7 @@ Telemetry is **off by default** (opt-in). Opt in with `HEADROOM_TELEMETRY=on` or
| `--log-messages` | `false` | Store full request/response content for the live feed |
| `--budget` | None | Daily budget limit in USD |
| `--openai-api-url` | `https://api.openai.com` | Custom OpenAI API URL |
| `--provider-name` | Detected from `--openai-api-url` | Display name for the OpenAI-compatible upstream on the dashboard (e.g. `OpenRouter`). Well-known hosts (OpenRouter, Groq, Together, Azure OpenAI, …) are detected automatically; this overrides them. Routing and pricing are unaffected. |
| `--anthropic-api-url` | Anthropic default | Custom Anthropic API URL |
| `--gemini-api-url` | Gemini default | Custom Gemini API URL |
| `--backend` | `anthropic` | Backend: `anthropic`, `bedrock`, `openrouter`, `anyllm`, or `litellm-<provider>` |

11
headroom/cli/proxy.py
View file

@ -803,6 +803,15 @@ def dashboard(port: int, no_open: bool) -> None:
default=None,
help="Custom OpenAI API URL for passthrough endpoints (env: OPENAI_TARGET_API_URL)",
)
@click.option(
"--provider-name",
default=None,
help=(
"Display name for the OpenAI-compatible upstream shown on the dashboard "
"(e.g. 'OpenRouter'). Overrides hostname detection from --openai-api-url. "
"Internal routing and pricing are unaffected."
),
)
@click.option(
"--gemini-api-url",
default=None,
@ -966,6 +975,7 @@ def proxy(
anthropic_extra_headers: str | None,
openai_extra_headers: str | None,
openai_api_url: str | None,
provider_name: str | None,
gemini_api_url: str | None,
cloudcode_api_url: str | None,
vertex_api_url: str | None,
@ -1155,6 +1165,7 @@ def proxy(
anthropic_extra_headers=resolved_anthropic_extra_headers,
openai_extra_headers=resolved_openai_extra_headers,
openai_api_url=provider_api_overrides.openai,
provider_name=provider_name,
gemini_api_url=provider_api_overrides.gemini,
cloudcode_api_url=provider_api_overrides.cloudcode,
vertex_api_url=provider_api_overrides.vertex,

6379
headroom/cli/wrap.py.orig
View file

File diff suppressed because it is too large Load diff

62
headroom/proxy/helpers.py
View file

@ -555,6 +555,68 @@ def get_sse_event_max_bytes() -> int:
)
# Well-known OpenAI-compatible upstreams, matched by host against the
# configured ``--openai-api-url``. Used only to label the dashboard/stats
# display provider — the internal provider key stays ``openai`` so pricing
# and request formatting are unaffected (issue #1533).
_OPENAI_COMPATIBLE_HOSTS: tuple[tuple[str, str], ...] = (
("openrouter.ai", "OpenRouter"),
("api.groq.com", "Groq"),
("api.together.xyz", "Together AI"),
("api.fireworks.ai", "Fireworks AI"),
("api.deepseek.com", "DeepSeek"),
("api.mistral.ai", "Mistral"),
("api.perplexity.ai", "Perplexity"),
("openai.azure.com", "Azure OpenAI"),
("api.openai.com", "OpenAI"),
)
def classify_openai_upstream(url: str | None) -> str | None:
"""Map a custom ``--openai-api-url`` to a well-known provider display name.
Matches the URL host against :data:`_OPENAI_COMPATIBLE_HOSTS` (exact or
subdomain). Returns ``None`` when no URL is set or the host is unrecognized
(callers then fall back to an explicit ``--provider-name`` or the raw
``openai`` label).
"""
if not url:
return None
from urllib.parse import urlparse
try:
host = (urlparse(url).hostname or "").lower()
except (ValueError, TypeError):
return None
if not host:
return None
for needle, name in _OPENAI_COMPATIBLE_HOSTS:
if host == needle or host.endswith("." + needle):
return name
return None
def resolve_display_provider(
raw_provider: str | None,
*,
openai_api_url: str | None = None,
provider_name: str | None = None,
) -> str:
"""Resolve the dashboard display provider for a logged request.
Only requests whose internal provider is ``openai`` are reclassified;
Anthropic/Bedrock/Gemini keep their own labels. This affects the display
label only — pricing and request formatting still key on ``openai``.
Precedence: explicit ``--provider-name`` > host detection > raw provider.
"""
raw = (raw_provider or "").strip()
if raw.lower() != "openai":
return raw or "unknown"
if provider_name:
return provider_name
return classify_openai_upstream(openai_api_url) or raw
# Body-too-large status code (PR-A8 / P5-59). Default 413 (RFC 7231 §6.5.11).
# Configurable via HEADROOM_PROXY_BODY_TOO_LARGE_STATUS for operators who need
# to override (no expected production use; documentation knob).

4
headroom/proxy/models.py
View file

@ -124,6 +124,10 @@ class ProxyConfig:
port: int = 8787
anthropic_api_url: str | None = None # Custom Anthropic API URL override
openai_api_url: str | None = None # Custom OpenAI API URL override
# Display label for the OpenAI-compatible upstream (dashboard/stats only).
# Overrides hostname detection from ``openai_api_url``; the internal
# provider stays ``openai`` so pricing/format keys are unaffected.
provider_name: str | None = None
gemini_api_url: str | None = None # Custom Gemini API URL override
cloudcode_api_url: str | None = None # Custom Cloud Code Assist API URL override
vertex_api_url: str | None = None # Custom Vertex AI regional API URL override

34
headroom/proxy/server.py
View file

@ -141,6 +141,7 @@ from headroom.proxy.helpers import (
initialize_context_tool_session_baseline,
is_anthropic_auth, # noqa: F401
jitter_delay_ms,
resolve_display_provider,
retry_after_ms,
)
from headroom.proxy.loop_callback_failure_policy import is_known_websocket_callback_failure
@ -291,6 +292,23 @@ def _classify_agent_from_log(entry: dict[str, Any]) -> tuple[str, str, str]:
return "unknown", _agent_label("unknown"), "unknown"
def _remap_provider_counts(counts: dict[str, int], config: ProxyConfig) -> dict[str, int]:
"""Relabel ``openai`` upstream counts with the configured display provider.
Display only — the stored metrics key stays ``openai`` (issue #1533).
Collisions (none today) are summed defensively.
"""
out: dict[str, int] = {}
for provider, count in counts.items():
display = resolve_display_provider(
provider,
openai_api_url=config.openai_api_url,
provider_name=config.provider_name,
)
out[display] = out.get(display, 0) + int(count)
return out
def _build_agent_usage_summary(
logs: list[dict[str, Any]],
*,
@ -3235,7 +3253,11 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
{
"request_id": log.get("request_id"),
"timestamp": log.get("timestamp"),
"provider": log.get("provider"),
"provider": resolve_display_provider(
log.get("provider"),
openai_api_url=proxy.config.openai_api_url,
provider_name=proxy.config.provider_name,
),
"model": log.get("model"),
"input_tokens_original": _recent_request_optional_number(
log, "input_tokens_original"
@ -3465,7 +3487,7 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
tool_schema_requests += 1
agent_usage = _build_agent_usage_summary(
recent_request_logs,
requests_by_provider=dict(m.requests_by_provider),
requests_by_provider=_remap_provider_counts(dict(m.requests_by_provider), proxy.config),
requests_by_model=dict(m.requests_by_model),
global_before_tokens=proxy_total_before_compression,
global_after_tokens=m.tokens_input_total,
@ -3584,7 +3606,7 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
"cached": m.requests_cached,
"rate_limited": m.requests_rate_limited,
"failed": m.requests_failed,
"by_provider": dict(m.requests_by_provider),
"by_provider": _remap_provider_counts(dict(m.requests_by_provider), proxy.config),
"by_model": dict(m.requests_by_model),
"by_stack": dict(m.requests_by_stack),
},
@ -4020,7 +4042,11 @@ def create_app(config: ProxyConfig | None = None) -> FastAPI:
{
"request_id": log.get("request_id"),
"timestamp": log.get("timestamp"),
"provider": log.get("provider"),
"provider": resolve_display_provider(
log.get("provider"),
openai_api_url=proxy.config.openai_api_url,
provider_name=proxy.config.provider_name,
),
"model": log.get("model"),
"input_tokens_original": log.get("input_tokens_original"),
"input_tokens_optimized": log.get("input_tokens_optimized"),

85
tests/test_provider_display_classification.py Normal file
View file

@ -0,0 +1,85 @@
"""Upstream-provider display classification for OpenAI-compatible endpoints.
Covers issue #1533: when ``--openai-api-url`` points at a non-OpenAI upstream
(OpenRouter, Groq, …), the dashboard should show that provider instead of
always "openai". The internal provider key stays ``openai`` so pricing and
request formatting are unaffected — only the display label changes.
"""
from __future__ import annotations
import pytest
from headroom.proxy.helpers import (
classify_openai_upstream,
resolve_display_provider,
)
from headroom.proxy.models import ProxyConfig
from headroom.proxy.server import _remap_provider_counts
@pytest.mark.parametrize(
"url,expected",
[
("https://api.openai.com/v1", "OpenAI"),
("https://openrouter.ai/api/v1", "OpenRouter"),
("https://api.groq.com/openai/v1", "Groq"),
("https://api.together.xyz/v1", "Together AI"),
("https://my-resource.openai.azure.com/", "Azure OpenAI"),
("https://api.deepseek.com/v1", "DeepSeek"),
],
)
def test_classify_known_hosts(url: str, expected: str) -> None:
assert classify_openai_upstream(url) == expected
@pytest.mark.parametrize("url", [None, "", "not-a-url", "https://api.mycorp.internal/v1"])
def test_classify_unknown_or_missing_returns_none(url: str | None) -> None:
assert classify_openai_upstream(url) is None
def test_resolve_detects_from_url() -> None:
assert (
resolve_display_provider("openai", openai_api_url="https://openrouter.ai/api/v1")
== "OpenRouter"
)
def test_resolve_explicit_name_wins_over_detection() -> None:
assert (
resolve_display_provider(
"openai",
openai_api_url="https://openrouter.ai/api/v1",
provider_name="Internal Gateway",
)
== "Internal Gateway"
)
def test_resolve_non_openai_provider_untouched() -> None:
# Anthropic/Bedrock/etc. keep their own label regardless of openai_api_url.
assert (
resolve_display_provider("anthropic", openai_api_url="https://openrouter.ai/api/v1")
== "anthropic"
)
def test_resolve_plain_openai_unchanged() -> None:
assert resolve_display_provider("openai") == "openai"
# Unknown custom host with no override falls back to the raw label.
assert (
resolve_display_provider("openai", openai_api_url="https://api.mycorp.internal/v1")
== "openai"
)
def test_remap_provider_counts_relabels_only_openai() -> None:
config = ProxyConfig(openai_api_url="https://openrouter.ai/api/v1")
counts = {"openai": 7, "anthropic": 3}
assert _remap_provider_counts(counts, config) == {"OpenRouter": 7, "anthropic": 3}
def test_remap_provider_counts_noop_without_custom_upstream() -> None:
config = ProxyConfig()
counts = {"openai": 5}
assert _remap_provider_counts(counts, config) == {"openai": 5}