- Add --no-intelligent-context to disable IntelligentContextManager
- Add --no-intelligent-scoring to disable multi-factor scoring
- Add --no-compress-first to disable compression before dropping
- Update README to reflect IntelligentContext as the default context manager
- Add headroom.evals module with 12+ dataset loaders (HotpotQA, SQuAD,
Natural Questions, TriviaQA, MS MARCO, LongBench, NarrativeQA, BFCL,
ToolBench, CodeSearchNet, HumanEval, built-in tool outputs)
- Add before/after evaluation runner that compares LLM responses with
original vs compressed context
- Add metrics: F1 score, semantic similarity, exact match, ground truth
- Add CLI: python -m headroom.evals quick|benchmark|list|report
- Add [evals] extra to pyproject.toml for pip install headroom-ai[evals]
Fix ContentRouter to use LLMLingua for plain text compression:
- Route TEXT strategy through LLMLingua instead of heuristic TextCompressor
- Adjust LLMLingua compression rates for better accuracy (0.5 vs 0.25)
- HotpotQA now achieves 95% accuracy with 44% compression
Update documentation with evaluation framework section
Fix test isolation in test_toin.py (TOIN singleton persistence)
Previously, HeadroomAgnoModel called wrapped_model.response() which ran the
tool execution loop internally. This meant tool outputs (often 60k+ chars)
were never optimized - only the initial messages were compressed.
The fix delegates response() to the inherited Model.response(), which calls
self.invoke() for each API call. Our invoke() override optimizes messages
before delegating to wrapped_model.invoke(), ensuring tool outputs are
compressed on every API request.
Results from multi_tool_agent_test.py with Claude Sonnet:
- Tokens before optimization: 25,713
- Tokens after optimization: 6,100
- Tokens saved: 19,613 (76.3%)
- Both baseline and optimized found all critical information
Also adds:
- multi_tool_agent_test.py: Real function calling test with 4 tools
- multi_tool_compression_test.py: Direct compression test
- README update with multi-tool agent test results
- Replace marketing claims with actual API test results
- Add needle-in-haystack test: critical error at position 67/100
- Show real JSON compression: 100 entries → 6 entries (93.9% reduction)
- Verified with Claude Sonnet: 87.6% fewer tokens, 4/4 correct answers
- Add example scripts for reproducing the tests
Features:
- with_fast_memory(): Zero-latency inline extraction (Letta-style)
- Memory extracted as part of LLM response, no extra API calls
- Semantic retrieval with local embeddings (sub-50ms)
- with_memory(): Background extraction for non-blocking memory
- SQLite + FTS5 storage with vector similarity search
- Multi-user isolation by user_id
Memory enables temporal compression - extract key facts instead of
carrying full conversation history (4000 tokens → 50 tokens).
Includes:
- Comprehensive test suite (71 new tests)
- Documentation (docs/memory.md)
- Benchmark examples comparing approaches
- E2E test with LLM-as-judge evaluation
CodeAwareCompressor:
- Tree-sitter based AST parsing for Python, JS, TS, Go, Rust, Java, C, C++
- Preserves imports, signatures, type annotations, error handlers
- Guarantees syntactically valid output
- Uses tree-sitter-language-pack for broad language support
ContentRouter:
- Intelligent compression orchestrator
- Auto-routes content to optimal compressor based on type detection
- Source hint support for high-confidence routing
Custom Model Configuration:
- HEADROOM_MODEL_LIMITS env var and ~/.headroom/models.json support
- Pattern-based inference for unknown models (opus/sonnet/haiku tiers)
- Support for Claude 4.5, Claude 4, o3, o3-mini
- Graceful fallback - never crashes on unknown models
Integrate Microsoft's LLMLingua-2 ML-based compression as an opt-in
feature for the proxy server, with excellent developer experience.
Features:
- New CLI flags: --llmlingua, --llmlingua-device, --llmlingua-rate
- ProxyConfig options: llmlingua_enabled, llmlingua_device, llmlingua_target_rate
- Smart startup hints when llmlingua is available but not enabled
- Helpful error messages when enabled but not installed
- LLMLinguaCompressor inserted before RollingWindow in pipeline
Why opt-in:
- Heavy dependencies (~2GB torch, transformers)
- 10-30s cold start for model loading
- ~1GB RAM when loaded
- Default proxy stays lightweight (<5ms overhead)
Tests:
- 26 new tests in test_proxy_llmlingua.py covering config, setup,
banner status, CLI args, DevEx messages, and edge cases
Documentation:
- Updated README.md with proxy integration section
- Updated docs/proxy.md with LLMLingua CLI options
- Updated docs/transforms.md with LLMLinguaCompressor reference
- Updated docs/ARCHITECTURE.md with pipeline and file structure
- Updated CHANGELOG.md with new feature
Add standalone text compression utilities that applications can use
explicitly for non-JSON content:
- SearchCompressor: for grep/ripgrep output (file:line:content format)
- LogCompressor: for build/test logs (pytest, npm, cargo output)
- TextCompressor: for generic plain text with anchor preservation
- detect_content_type: content type detection for routing decisions
Design decision: Text compression is OPT-IN, not automatic. SmartCrusher
continues to compress JSON automatically (structure-preserving, safe),
but passes non-JSON through unchanged. Applications decide when and how
to compress text content based on their specific needs.
This prevents lossy text compression from being applied automatically,
which could lose important context in coding tasks (e.g., root cause
errors in logs, critical matches in search results).
Includes 22 tests covering content detection, compression utilities,
and SmartCrusher pass-through behavior.
- Renamed package from 'headroom' to 'headroom-ai' (PyPI name conflict)
- Fixed numpy/jinja2 imports to be lazy (core install no longer crashes)
- Fixed SQLite default path (now uses temp directory)
- Fixed f-string {tool} crash in proxy server
- Updated README with correct package name and examples
- Added quickstart and troubleshooting docs
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
