Root fix: compute_optimal_k() now scales k with content diversity using
the SimHash uniqueness ratio already computed in the function.
diversity ~1.0 → keep 100% of items (all unique, dropping any loses info)
diversity ~0.5 → keep ~65%
diversity ~0.0 → keep ~30% (same as before for repetitive data)
No hardcoded RAG detection. No field name heuristics. Pure statistics —
works for any JSON array regardless of source (Pinecone, Chroma, Weaviate,
LangChain, custom APIs).
When all items are kept (high diversity), SmartCrusher tries to compress
text WITHIN each item's long string fields using Kompress (if available).
Falls back gracefully when Kompress is not installed.
Before: 12 unique RAG chunks → kept 2, dropped 10 (0/6 key concepts)
After: 12 unique RAG chunks → kept 12, compressed within (6/6 concepts)
Also adds tests/test_adaptive_sizer.py (16 tests covering high/low/moderate
diversity, knee interactions, bias, caps).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
