mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
f64aac9733
## Description
The non-streaming Gemini/Vertex handler reads token counts straight from
the response's `usageMetadata`:
```python
try:
usage = resp_json.get("usageMetadata", {})
total_input_tokens = usage.get("promptTokenCount", optimized_tokens)
output_tokens = usage.get("candidatesTokenCount", 0)
cache_read_tokens = usage.get("cachedContentTokenCount", 0)
except (...):
...
uncached_input_tokens = max(0, total_input_tokens - cache_read_tokens) # OUTSIDE the try
```
`.get(key, default)` only falls back when the key is **absent**. When
`usageMetadata` carries a key with a **null** value — which Gemini can
do on a safety-blocked turn that produced no candidates — `.get` returns
`None`. That `None` then reaches:
- `max(0, total_input_tokens - cache_read_tokens)` (a `None - int` →
`TypeError`), and
- `RequestOutcome(output_tokens=...)`, whose field is `int` and which
the metrics recorder increments (`tokens_output_total += output_tokens`
→ `TypeError`).
Both run on the success (non-`except`) path, so a single such response
crashes the request and its outcome recording. The Gemini streaming path
already guards these with a `_usage_int` helper; the non-streaming path
(two sites) did not.
## Fix
Coerce the three counts with `int(... or fallback)`, matching the
streaming `_usage_int` guard and the LiteLLM usage mappings:
```python
total_input_tokens = int(usage.get("promptTokenCount", optimized_tokens) or optimized_tokens)
output_tokens = int(usage.get("candidatesTokenCount", 0) or 0)
cache_read_tokens = int(usage.get("cachedContentTokenCount", 0) or 0)
```
No change for a normal integer usage; only a `None` (or absent) value
now becomes the fallback/0. Applied to both non-streaming
usage-extraction sites in `handlers/gemini.py`.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/proxy/handlers/gemini.py`: `int(... or fallback)`-guard
`promptTokenCount` / `candidatesTokenCount` / `cachedContentTokenCount`
at both non-streaming usage sites.
- `tests/test_proxy/test_gemini_savings_profile.py`: add a regression
driving a `generateContent` request whose
`usageMetadata.candidatesTokenCount` is `null`, asserting a 200, an
`int` `output_tokens == 0`, and `uncached_input_tokens == 20` (the
`max(0, …)` no longer raises).
- `CHANGELOG.md`: Bug Fixes entry.
## Testing
- [ ] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ uvx ruff@0.15.17 check headroom/proxy/handlers/gemini.py tests/test_proxy/test_gemini_savings_profile.py
All checks passed!
$ uvx ruff@0.15.17 format --check headroom/proxy/handlers/gemini.py tests/test_proxy/test_gemini_savings_profile.py
2 files already formatted
$ uvx mypy@1.20.2 --ignore-missing-imports headroom/proxy/handlers/gemini.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12, `uvx ruff@0.15.17` / `uvx
mypy@1.20.2`. A full `pytest` OOMs this box (ML-stack import), so I
reproduced the extraction with a dependency-free script and left the
full pytest to CI.
- Exact command / steps: modeled the OLD (bare `.get`) and NEW (`int(...
or fallback)`) derivations for a blocked response
(`candidatesTokenCount: null`, valid prompt count), a null
`promptTokenCount`, a normal response, and an absent-usage response.
- Observed result: OLD raised `TypeError` at `max(0, None - …)` for a
null prompt count and left `output_tokens = None` (which crashes the
int-typed outcome/metrics recorder) for a null candidate count; NEW
produced `(20, 0)` for the blocked case, `(15, 0)` for the null-prompt
case (the `optimized_tokens` fallback), `(60, 30)` for a normal
response, and the fallbacks for absent usage. The added
`create_app`/`TestClient` test drives the handler end to end and asserts
a 200 with `int` outcome counts.
- Not tested: a live Gemini safety-blocked response; the added test uses
a mocked `_retry_request` returning a `usageMetadata` with a null count,
matching the existing Gemini test harness in this file.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
The "unit tests pass locally" box is unchecked because a local pytest
run imports the ML stack and OOMs this box; the added test reuses the
existing `create_app`/`TestClient` + mocked-`_retry_request` harness in
`test_gemini_savings_profile.py` and runs under the normal CI pytest
job, and the behavior is corroborated by the standalone proof above.
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|---|---|---|
| .. | ||
| test_anthropic_buffered_timeout.py | ||
| test_anthropic_ccr_deferred_injection.py | ||
| test_anthropic_ccr_raise.py | ||
| test_anthropic_streaming_ccr_retrieve.py | ||
| test_anthropic_upstream_header.py | ||
| test_background_compression.py | ||
| test_bedrock_passthrough.py | ||
| test_cc_switch_reconciler.py | ||
| test_ccr_frozen_prefix_coupling.py | ||
| test_compression_failure_action.py | ||
| test_compression_timeout_config.py | ||
| test_compute_turn_id.py | ||
| test_gemini_savings_profile.py | ||
| test_header_safe_transforms.py | ||
| test_mcp_stats_aggregation.py | ||
| test_model_router.py | ||
| test_model_router_wiring.py | ||
| test_openai_backend_path.py | ||
| test_openai_chat_savings_profile.py | ||
| test_openai_responses_ccr.py | ||
| test_openai_stream_usage_option.py | ||
| test_openai_transport_path_prefix.py | ||
| test_openai_upstream_header.py | ||
| test_phase3_byte_identity.py | ||
| test_request_logger.py | ||
| test_settings_fresh_process_precedence.py | ||
| test_settings_store.py | ||
| test_transformations_feed.py | ||