Commit graph

2 commits

Author SHA1 Message Date
chopratejas
e7008f64d7 Count Strands reasoningContent, image, document, video tokens (#111 follow-up)
reasoningContent: exact counting via count_text() — pure text, no estimation
image: decode with Pillow for (w*h)/750 formula, fallback by byte size
document: ~1500 tokens/page heuristic (3KB/page of PDF)
video: ~1000 tokens/frame heuristic (30KB/frame)

Text content (reasoning, text, toolResult) uses exact tokenization.
Binary content (image, document, video) uses provider formula or
size-based estimates — accurate counting requires content extraction
that only the provider can do.

14 tests covering all Strands content block types.
2026-04-08 17:31:00 -07:00
chopratejas
8f18468f3b Fix Strands SDK content block tokenization (#111)
Strands SDK sends content blocks without a 'type' field:
  {"text": "..."} instead of {"type": "text", "text": "..."}
  {"toolUse": {...}} instead of {"type": "tool_use", ...}
  {"toolResult": {...}} instead of {"type": "tool_result", ...}

The tokenizer's _count_content_parts() only matched on type field,
causing Strands blocks to fall through to json.dumps estimation.
Now explicitly handles Strands text, toolUse, and toolResult formats
with proper recursive counting for nested toolResult content.

8 new tests covering Strands text blocks, tool blocks, and mixed formats.
2026-04-07 14:19:15 -07:00