Commit graph

1 commit

Author SHA1 Message Date
chopratejas
3e97a0cb73 fix(proxy): record cache reads/writes on backend-routed streaming (#327)
Two regressions surfaced as "Cache write: 0" in `headroom perf` and the
dashboard for every backend-routed streaming request (e.g. SvenMeyer's
DROID CLI > headroom > Azure GPT-5.5 setup):

* `_stream_openai_via_backend` parsed only `completion_tokens` and never
  read `prompt_tokens_details.cached_tokens` from the upstream usage
  frame. It also emitted no PERF log line at all, so `headroom perf`
  couldn't even count the request to report numbers. Now buffers SSE
  bytes, drains via `_parse_sse_usage_from_buffer(provider="openai")`,
  infers writes via `_infer_openai_cache_write_tokens` (only when the
  upstream actually reported usage — mirrors `_extract_responses_usage`),
  threads cache values into `record_request`, `cost_tracker.record_tokens`,
  the RequestLog, and a real PERF log line.

* `_stream_response_bedrock` hardcoded `cache_read=0 cache_write=0
  cache_hit_pct=0` in its PERF line regardless of what `message_start.usage`
  reported. Extended `stream_state` with `cache_read_input_tokens` and
  `cache_creation_input_tokens` (plus 5m/1h TTL buckets), captures them
  from `message_start`, threads through `record_request(cached=...)`,
  `cost_tracker.record_tokens(...)`, and `RequestLog(cache_hit=...)`.

Tests: four new tests in `test_backend_streaming_cache_metrics.py` cover
both paths plus a source-level regression guard against the hardcoded
zero string reappearing.
2026-05-14 11:01:57 -07:00