mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
d6a1af40d5
## Description OpenAI-format `POST /v1/chat/completions` requests routed through `--backend litellm-vertex` fail when the client includes `max_tokens`. The proxy currently runs its direct-OpenAI compatibility shim before backend dispatch, renames `max_tokens` to `max_completion_tokens`, then the LiteLLM path no longer recognizes that field as standard and sweeps it into `extra_body`. Vertex rejects the resulting request with `extra_body: Extra inputs are not permitted`. This change scopes the rename shim to the direct OpenAI path only. Backend-routed chat requests now keep `max_tokens`, which LiteLLM already forwards correctly for the Vertex Anthropic path. Direct GPT-5 and o-series compatibility stays unchanged. Closes #2392. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Thread a backend-owned translation flag into `_normalize_openai_max_tokens`. - Skip the legacy-to-completion-token rename on backend-routed OpenAI chat requests. - Keep the direct OpenAI compatibility path covered with a backend-owned translation no-op test. - Add buffered and streaming handler-level regressions for the exact `litellm-vertex` request shape, proving the request survives the `/v1/chat/completions` normalization boundary with vendor fields intact. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py -q`) - [x] Linting passes (`uv run ruff check headroom/proxy/handlers/openai.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py -q ......sss............ [100%] 20 passed, 3 skipped, 1 warning in 42.13s $ uv run ruff check headroom/proxy/handlers/openai.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py All checks passed! $ uv run ruff format headroom/proxy/handlers/openai.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py --check 5 files already formatted ``` ## Real Behavior Proof - Environment: Windows, synced Headroom development environment, mocked LiteLLM provider boundary, no paid GCP credentials required - Exact command / steps: run `uv run pytest tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py -q`, using the issue payload shape `{"model":"claude-sonnet-4-6","max_tokens":32,"messages":[{"role":"user","content":"hi"}],"chat_template_kwargs":{"enable_thinking":false}}` through `POST /v1/chat/completions` - Observed result: buffered and streaming `litellm-vertex` requests keep `max_tokens` as a named backend kwarg, preserve `chat_template_kwargs` in `extra_body`, omit `max_completion_tokens` from `extra_body`, and return success through the handler boundary. Direct-path normalization still renames legacy `max_tokens`. - Not tested: live Vertex AI request ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [ ] My code follows the project's style guidelines - [ ] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - `CHANGELOG.md`: N/A, the release pipeline generates it from the conventional-commit subject. - Scope is intentionally narrow: this fixes the exact backend-routed `max_tokens` failure and does not broaden `extra_body` hardening for unrelated OpenAI fields.
57 lines
2.1 KiB
Python
57 lines
2.1 KiB
Python
"""OpenAI chat-path compatibility shim: max_tokens -> max_completion_tokens.
|
|
|
|
GPT-5 / o-series chat models reject the legacy ``max_tokens`` and require
|
|
``max_completion_tokens`` ("Unsupported parameter: 'max_tokens' is not supported
|
|
with this model. Use 'max_completion_tokens' instead."). openai-compatible
|
|
clients (opencode, older SDKs) still send ``max_tokens``, so the proxy — which
|
|
already owns the outbound request body — translates it.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from headroom.proxy.handlers.openai import _normalize_openai_max_tokens
|
|
|
|
|
|
def test_renames_legacy_max_tokens():
|
|
body = {"model": "gpt-5.3-chat-latest", "max_tokens": 256, "messages": []}
|
|
_normalize_openai_max_tokens(body, backend_owns_translation=False)
|
|
assert "max_tokens" not in body
|
|
assert body["max_completion_tokens"] == 256
|
|
|
|
|
|
def test_backend_owned_translation_preserves_max_tokens():
|
|
body = {"model": "claude-sonnet-4-6", "max_tokens": 32, "messages": []}
|
|
_normalize_openai_max_tokens(body, backend_owns_translation=True)
|
|
assert body == {"model": "claude-sonnet-4-6", "max_tokens": 32, "messages": []}
|
|
|
|
|
|
def test_preserves_existing_max_completion_tokens_and_drops_legacy():
|
|
body = {"max_tokens": 256, "max_completion_tokens": 100}
|
|
_normalize_openai_max_tokens(body)
|
|
assert "max_tokens" not in body
|
|
assert body["max_completion_tokens"] == 100 # explicit value wins
|
|
|
|
|
|
def test_noop_when_only_max_completion_tokens():
|
|
body = {"max_completion_tokens": 128}
|
|
_normalize_openai_max_tokens(body)
|
|
assert body == {"max_completion_tokens": 128}
|
|
|
|
|
|
def test_noop_when_neither_present():
|
|
body = {"model": "gpt-4o", "messages": []}
|
|
_normalize_openai_max_tokens(body)
|
|
assert "max_completion_tokens" not in body
|
|
assert "max_tokens" not in body
|
|
|
|
|
|
def test_null_max_tokens_is_dropped_without_setting_completion():
|
|
body = {"max_tokens": None}
|
|
_normalize_openai_max_tokens(body)
|
|
assert "max_tokens" not in body
|
|
assert body.get("max_completion_tokens") is None
|
|
|
|
|
|
def test_non_dict_is_safe():
|
|
_normalize_openai_max_tokens(None) # type: ignore[arg-type]
|
|
_normalize_openai_max_tokens("nope") # type: ignore[arg-type]
|