mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
2 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Rod Boev
|
d6a1af40d5
|
fix(proxy): skip max_tokens rename for backend-routed openai chat (#2401)
## Description OpenAI-format `POST /v1/chat/completions` requests routed through `--backend litellm-vertex` fail when the client includes `max_tokens`. The proxy currently runs its direct-OpenAI compatibility shim before backend dispatch, renames `max_tokens` to `max_completion_tokens`, then the LiteLLM path no longer recognizes that field as standard and sweeps it into `extra_body`. Vertex rejects the resulting request with `extra_body: Extra inputs are not permitted`. This change scopes the rename shim to the direct OpenAI path only. Backend-routed chat requests now keep `max_tokens`, which LiteLLM already forwards correctly for the Vertex Anthropic path. Direct GPT-5 and o-series compatibility stays unchanged. Closes #2392. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Thread a backend-owned translation flag into `_normalize_openai_max_tokens`. - Skip the legacy-to-completion-token rename on backend-routed OpenAI chat requests. - Keep the direct OpenAI compatibility path covered with a backend-owned translation no-op test. - Add buffered and streaming handler-level regressions for the exact `litellm-vertex` request shape, proving the request survives the `/v1/chat/completions` normalization boundary with vendor fields intact. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py -q`) - [x] Linting passes (`uv run ruff check headroom/proxy/handlers/openai.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py -q ......sss............ [100%] 20 passed, 3 skipped, 1 warning in 42.13s $ uv run ruff check headroom/proxy/handlers/openai.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py All checks passed! $ uv run ruff format headroom/proxy/handlers/openai.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py --check 5 files already formatted ``` ## Real Behavior Proof - Environment: Windows, synced Headroom development environment, mocked LiteLLM provider boundary, no paid GCP credentials required - Exact command / steps: run `uv run pytest tests/test_proxy/test_openai_backend_path.py tests/test_openai_streaming_backend.py tests/test_openai_max_completion_tokens.py tests/test_litellm_openai_passthrough.py -q`, using the issue payload shape `{"model":"claude-sonnet-4-6","max_tokens":32,"messages":[{"role":"user","content":"hi"}],"chat_template_kwargs":{"enable_thinking":false}}` through `POST /v1/chat/completions` - Observed result: buffered and streaming `litellm-vertex` requests keep `max_tokens` as a named backend kwarg, preserve `chat_template_kwargs` in `extra_body`, omit `max_completion_tokens` from `extra_body`, and return success through the handler boundary. Direct-path normalization still renames legacy `max_tokens`. - Not tested: live Vertex AI request ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [ ] My code follows the project's style guidelines - [ ] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - `CHANGELOG.md`: N/A, the release pipeline generates it from the conventional-commit subject. - Scope is intentionally narrow: this fixes the exact backend-routed `max_tokens` failure and does not broaden `extra_body` hardening for unrelated OpenAI fields. |
||
|
Tejas Chopra
|
285808b90e
|
fix(proxy/openai): translate max_tokens -> max_completion_tokens on chat path (#1774)
## Description GPT-5 / o-series chat models reject the legacy `max_tokens` — `AI_APICallError: Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.` — while gpt-4o/4.1 accept `max_completion_tokens` too. openai-compatible clients (opencode via `@ai-sdk/openai-compatible`, older SDKs) still send `max_tokens`, so requests for GPT-5 models fail at the proxy's OpenAI upstream. This is a blocker for any such client pointed at a GPT-5 model through Headroom. The proxy already owns the outbound `/v1/chat/completions` body (it rewrites `messages` to compress them), so translate the token param there: rename `max_tokens` → `max_completion_tokens` when the newer form isn't already set, then drop the rejected legacy key. One-way, safe for current OpenAI models; no-op when the client already sends `max_completion_tokens`. The Responses path (`max_output_tokens`) is unaffected. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - New `_normalize_openai_max_tokens(body)` helper + call in `handle_openai_chat` after body finalization, before upstream forward. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added ### Test Output ```text tests/test_openai_max_completion_tokens.py .... 6 passed ruff check ... All checks passed! mypy headroom/proxy/handlers/openai.py ... Success: no issues found ``` ## Real Behavior Proof - Environment: local worktree, Python 3.12. - Exact command / steps: reproduced live — opencode (`@ai-sdk/openai-compatible` → Headroom proxy) targeting `gpt-5.3-chat-latest` failed with `Unsupported parameter: 'max_tokens' ... Use 'max_completion_tokens'` in the DEBUG stream log. The shim renames the param on the outbound body. - Observed result: unit tests confirm the rename/drop/no-op cases. - Not tested: full live opencode completion (its headless `run` stalls for unrelated reasons in this env — separate from this param fix). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes Discovered while debugging why opencode wouldn't run through the proxy: three layered blockers — (1) missing `models` map in the injected provider config [PR #1716], (2) no `apiKey` in the injected config / HTTP path doesn't inject `OPENAI_API_KEY` like the WS path does, (3) this `max_tokens` vs `max_completion_tokens` mismatch. This PR addresses (3). |