akadmin/headroom
1
0
Fork 0
mirror of https://github.com/headroomlabs-ai/headroom.git synced 2026-08-27 14:17:10 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
2696 commits 410 branches 169 tags 142 MiB
Commit graph

3 commits

Author SHA1 Message Date
Abhay Singh
43a7b578a1
fix(backends): don't crash the OpenAI->Anthropic converter on empty choices (#2484) 
## Description

`_to_anthropic_response` in both backends converts a non-streaming
OpenAI-shape response to Anthropic shape and indexes the first choice
directly:

```python
# headroom/backends/litellm.py
choice = litellm_response.choices[0]
# headroom/backends/anyllm.py
choice = response.choices[0]
```

A non-streaming upstream response can be HTTP 200 with an **empty**
`choices` list: Azure OpenAI content filtering does exactly this, and
any OpenAI-compatible gateway can return a usage-only / filtered turn
the same way. With `choices: []`, `choices[0]` raises `IndexError`,
which surfaces as a 500 for the request instead of a normal (if empty)
turn.

This is an intra-file asymmetry: the streaming siblings in the same two
files already guard it (`if not chunk.choices: continue` / `if
hasattr(chunk, "choices") and chunk.choices:`), and
`headroom/proxy/handlers/openai.py` documents the exact hazard in
`_apply_stream_usage_option`: "the common `chunk.choices[0].delta`
pattern then raises IndexError" on a usage-only `choices: []` chunk. The
non-streaming converters just never got the same guard.

## Fix

Return a valid empty assistant turn (`content: []`, `stop_reason:
"end_turn"`, usage still mapped) when `choices` is empty, before
indexing. The client gets a clean empty response instead of a 500,
matching how the streaming path already tolerates the same shape.
Non-empty responses are unchanged.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/backends/litellm.py`: empty-`choices` guard at the top of
`_to_anthropic_response`, returning an empty assistant turn with mapped
usage.
- `headroom/backends/anyllm.py`: same guard in its
`_to_anthropic_response`.
- `tests/test_litellm_nonstream_cache_usage.py`,
`tests/test_backend_anyllm.py`: regressions passing an empty-`choices`
response through each converter and asserting an empty turn instead of
IndexError.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_litellm_nonstream_cache_usage.py::test_to_anthropic_response_empty_choices_returns_empty_turn tests/test_backend_anyllm.py::test_to_anthropic_response_empty_choices_returns_empty_turn -q
2 passed

# with the fix reverted, both fail with
# IndexError: list index out of range

$ uvx ruff@0.15.17 check headroom/backends/litellm.py headroom/backends/anyllm.py tests/test_backend_anyllm.py tests/test_litellm_nonstream_cache_usage.py
All checks passed!
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/backends/litellm.py headroom/backends/anyllm.py
Success: no issues found in 2 source files
```

Note: `tests/test_backend_anyllm.py` has 7 `@pytest.mark.asyncio` tests
that fail locally because pytest-asyncio is not configured in this
environment (`Unknown config option: asyncio_mode`); they are unrelated
to this change and pass in CI. The two new tests here are synchronous
and pass locally.

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, project venv (`uv sync --extra
proxy`), `uvx ruff@0.15.17` / `uvx mypy@1.20.2`, pytest in the venv.
- Exact command / steps: built a response stand-in with `choices=[]` and
a usage object, called `LiteLLMBackend._to_anthropic_response` (on a
bare `object.__new__` instance) and
`AnyLLMBackend._to_anthropic_response` (via the file's fake-backend
fixture); then reverted both backend files and re-ran.
- Observed result: with the fix each converter returns `{type: message,
role: assistant, content: [], stop_reason: end_turn, usage: {...}}` with
the input/output token counts mapped; with the fix reverted both raise
`IndexError: list index out of range`. Ran against the actual modules
via the two test files.
- Not tested: a live Azure OpenAI content-filtered response routed
through the backend end to end.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
 2026-07-22 06:08:10 -07:00
Abhay Singh
44a174fef4
fix(backends/litellm): guard None completion_tokens in usage mapping (#2322) 
## Description

`_anthropic_usage_from_litellm` maps a LiteLLM `Usage` object to the
Anthropic response shape on the buffered (non-streaming) backend path.
Every numeric field is `None`-guarded with `int(... or 0)` except
`output_tokens`:

```python
cache_read = int(getattr(litellm_usage, "cache_read_input_tokens", 0) or 0)
cache_write = int(getattr(litellm_usage, "cache_creation_input_tokens", 0) or 0)
...
prompt_tokens = int(getattr(litellm_usage, "prompt_tokens", 0) or 0)
usage: dict[str, Any] = {
    "input_tokens": max(prompt_tokens - cache_read - cache_write, 0),
    "output_tokens": getattr(litellm_usage, "completion_tokens", 0),   # <-- no guard
}
```

The `getattr(..., 0)` default only fires when the attribute is
**absent**. LiteLLM's `Usage` is a pydantic model that always carries
`completion_tokens`, so the default never applies; when a provider
leaves the value `None`, `output_tokens` becomes `None`.

That `None` then propagates:

- `LiteLLMBackend.complete_message` builds the Anthropic-shaped body
with `"usage": usage`.
- The buffered anthropic-backend handler reads `output_tokens =
usage.get("output_tokens", 0)` (again, a present key returns its `None`
value, not the default) and passes it to
`RequestOutcome(output_tokens=...)`, whose field is declared `int`.
- The outcome-recording path does arithmetic on it, e.g. Prometheus
`self.tokens_output_total += output_tokens`, which raises `TypeError:
unsupported operand type(s) for +=: 'int' and 'NoneType'`.

So a provider that returns usage with a `None` completion count breaks
metrics recording for that request on any `--backend litellm` /
Bedrock/Vertex deployment.

## Fix

Guard the field the same way as its three siblings, so the mapping
always emits an `int`:

```python
"output_tokens": int(getattr(litellm_usage, "completion_tokens", 0) or 0),
```

No change for the normal case (an integer count passes through
unchanged); only a `None` (or absent) value now becomes `0` instead of
`None`.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/backends/litellm.py`: `None`-guard `output_tokens` in
`_anthropic_usage_from_litellm`.
- `tests/test_litellm_nonstream_cache_usage.py`: add
`test_output_tokens_none_coerced_to_zero` asserting a `None` completion
count maps to `int` `0`.
- `CHANGELOG.md`: Bug Fixes entry.

## Testing

- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
$ uvx ruff@0.15.17 check headroom/backends/litellm.py tests/test_litellm_nonstream_cache_usage.py
All checks passed!
$ uvx ruff@0.15.17 format --check headroom/backends/litellm.py tests/test_litellm_nonstream_cache_usage.py
2 files already formatted
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/backends/litellm.py
Success: no issues found in 1 source file
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17` / `uvx
mypy@1.20.2`. A full `pytest` OOMs this box (ML-stack import), so I
reproduced the field logic with a dependency-free script and left the
full pytest to CI.
- Exact command / steps: modeled the OLD (`getattr(..., 0)`) and NEW
(`int(getattr(..., 0) or 0)`) field derivations for a usage object with
`completion_tokens=None`, an integer, and the attribute absent, then
simulated the downstream `total += output_tokens`.
- Observed result: OLD produced `None` for the `None` case and the
downstream `+=` raised `TypeError`; NEW produced `0`/`7`/`0`
respectively and the `+=` succeeded. The added unit test asserts
`usage["output_tokens"] == 0` and `isinstance(..., int)`.
- Not tested: a live LiteLLM/Bedrock request that returns a `None`
completion count; the added test drives `_anthropic_usage_from_litellm`
directly with a `SimpleNamespace`, matching the existing tests in this
file.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable

## Additional Notes

The "unit tests pass locally" box is unchecked because a local pytest
run imports the ML stack and OOMs this box; the added test uses the same
`SimpleNamespace`-driven, dependency-light pattern as the neighbouring
tests in `test_litellm_nonstream_cache_usage.py` and runs under the
normal CI pytest job, and the behavior is corroborated by the standalone
proof above.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
 2026-07-17 12:11:38 -07:00
Dima Solodukha
d604e86904
fix(litellm): surface Bedrock cache token usage in non-streaming responses (#1848) 
## Description

Non-streaming `complete_message()` builds the Anthropic-shape usage from
`prompt_tokens`/`completion_tokens` only. LiteLLM's `prompt_tokens`
includes cached tokens, so when Bedrock prompt caching is active a
non-streaming client sees `input_tokens` equal to the full prompt and no
cache fields. That looks identical to the cache being broken (#1345),
and the savings tracker never credits the hits. The streaming and OpenAI
paths already map these fields.

Related: #1390 — that PR makes the markers reach Bedrock; this one makes
the result visible in non-streaming responses.

Closes # (contributes to #1345 together with #1390; not closing it
alone)

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Extract `_anthropic_usage_from_litellm()` in
`headroom/backends/litellm.py`: maps `cache_read_input_tokens` /
`cache_creation_input_tokens` (with `prompt_tokens_details` fallback)
into the Anthropic-shape usage and reports `input_tokens` without the
cached portion, matching what Anthropic returns.
- Use it in `complete_message()` instead of the inline
`prompt_tokens`/`completion_tokens` dict.
- Add `tests/test_litellm_nonstream_cache_usage.py` (5 cases: plain
usage, cache read, cache write, `prompt_tokens_details` fallback,
negative clamp).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest tests/test_litellm_nonstream_cache_usage.py -q
5 passed, 1 warning in 2.13s

$ ruff check headroom/backends/litellm.py tests/test_litellm_nonstream_cache_usage.py
All checks passed!
```

mypy not run locally: my environment fails on unrelated numpy stubs
(`numpy/__init__.pyi: Type statement is only supported in Python
3.12+`); relying on CI for the mypy gate.

## Real Behavior Proof

- Environment: real AWS Bedrock, us-east-1,
`us.anthropic.claude-sonnet-4-5-20250929-v1:0`, headroom-ai 0.30.0 with
this patch, Python 3.13.
- Exact command / steps: `headroom proxy --backend bedrock
--bedrock-region us-east-1 --mode cache --port 8787`, then three
identical non-streaming `POST /v1/messages` with a 1,226-token system
block marked `cache_control: {"type": "ephemeral"}` (fresh salted
prefix), with the conversion fix from #1390 applied so markers reach
Bedrock.
- Observed result: before this patch usage reported `input_tokens=1213`
with no cache fields on every call; after — call 1: `input_tokens=11,
cache_creation_input_tokens=1226`; calls 2–3: `input_tokens=11,
cache_read_input_tokens=1226`. Matches a direct-to-Bedrock baseline
(boto3 `invoke_model` with the same payload).
- Not tested: streaming path (unchanged by this PR), non-Bedrock LiteLLM
providers (mapping is provider-agnostic: fields are absent → behavior
identical to before).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

N/A — token counts are in Real Behavior Proof above.

## Additional Notes

Documentation and CHANGELOG unchecked: single-function bugfix, no
user-facing docs describe the non-streaming usage fields; happy to add a
CHANGELOG entry if maintainers want one.

---------

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
 2026-07-09 12:44:36 -05:00