Improve proxy scalability for high-concurrency scenarios with multiple agents:
- Add connection pool configuration (max_connections=500, max_keepalive=100)
- Enable HTTP/2 multiplexing by default for better throughput
- Add multi-worker support via --workers flag for multi-core scaling
- Add --limit-concurrency flag for backpressure control
- Reuse httpx client for CCR continuations instead of creating new clients
- Add httpx[http2] dependency for HTTP/2 support
- Add comprehensive test suite for scalability features
New CLI flags:
--max-connections Max connections to upstream APIs (default: 500)
--max-keepalive Max keepalive connections (default: 100)
--no-http2 Disable HTTP/2 (enabled by default)
--workers Number of worker processes (default: 1)
--limit-concurrency Max concurrent connections before 503 (default: 1000)
- Implement streaming memory tool detection and execution for Anthropic API
- Buffer SSE response to detect tool_use blocks, execute tools, and stream continuation
- Add helpful error detection and messaging for subscription credential restrictions
- Add startup note when memory tools enabled warning about API key requirement
- Move hnswlib to core dependencies for memory system
- Update CLI to show memory tool/context status on startup
- Add hnswlib>=0.8.0 to dev dependencies so CI can run HNSW tests
- Add new 'memory' extras group for users who want vector search
- Include 'memory' in 'all' extras
- Add headroom.evals module with 12+ dataset loaders (HotpotQA, SQuAD,
Natural Questions, TriviaQA, MS MARCO, LongBench, NarrativeQA, BFCL,
ToolBench, CodeSearchNet, HumanEval, built-in tool outputs)
- Add before/after evaluation runner that compares LLM responses with
original vs compressed context
- Add metrics: F1 score, semantic similarity, exact match, ground truth
- Add CLI: python -m headroom.evals quick|benchmark|list|report
- Add [evals] extra to pyproject.toml for pip install headroom-ai[evals]
Fix ContentRouter to use LLMLingua for plain text compression:
- Route TEXT strategy through LLMLingua instead of heuristic TextCompressor
- Adjust LLMLingua compression rates for better accuracy (0.5 vs 0.25)
- HotpotQA now achieves 95% accuracy with 44% compression
Update documentation with evaluation framework section
Fix test isolation in test_toin.py (TOIN singleton persistence)
- Add .pre-commit-config.yaml with ruff and ruff-format hooks
- Add pre-commit to dev dependencies
- Ignore UP038 rule (tuple syntax in isinstance is clearer)
Run `uv run pre-commit install` to enable hooks locally.
- Fix HeadroomAgnoModel to properly extend agno.models.base.Model as a dataclass
- Implement required abstract methods (invoke, ainvoke, invoke_stream, ainvoke_stream)
- Add type: ignore comments for method signature overrides
- Fix mypy errors in telemetry/models.py and telemetry/toin.py
- Add real Ollama integration tests for both Agno and LangChain (no API keys needed)
- Add ollama and langchain-ollama to dev dependencies for local testing
- Update existing tests to use new HeadroomAgnoModel API
- Add detailed breakdowns by provider and model to /stats
- Include compression, telemetry, and feedback loop statistics
- Add latency.average_ms metric
- Add real-world agent benchmark with MCP tool patterns
- Add worst-case and adversarial benchmarks for edge cases
- Bump version to 0.2.12
Features:
- Add field-level learning to TOIN from retrieved items
- CompressionStore now passes retrieved_items to TOIN for learning
- Add FieldSemantics class for tracking field usage patterns
Test improvements:
- Add TestCacheOptimizerInvocation to verify optimizer is actually invoked
- Add TestSemanticCacheIntegration to verify cache hit returns without API call
- Add TestSessionStatsTracking to verify session stats are tracked
- Add TestEndToEndTOINIntegration for full CCR cycle with TOIN
- Add critical field_semantics assertions to catch feedback loop bugs
Fixes:
- Remove unused imports and variables (ruff linting)
Bump version to 0.2.11
- HeadroomAgnoModel: Drop-in wrapper for any Agno model with automatic
context optimization
- HeadroomPreHook/HeadroomPostHook: Agent-level hooks for tracking
optimization metrics across tool calls
- Provider detection for Agno models (OpenAI, Anthropic, Google, etc.)
- Full test coverage for model wrapper and hooks
- Add litellm as a core dependency for accessing its community-maintained
model pricing database (2,425+ models across all major providers)
- Create headroom/pricing/litellm_pricing.py with simple wrapper functions
- Update ModelRegistry.estimate_cost() to fetch pricing from LiteLLM
- Remove hardcoded pricing fields from ModelInfo dataclass
- Update tests to reflect new pricing source
Features:
- with_fast_memory(): Zero-latency inline extraction (Letta-style)
- Memory extracted as part of LLM response, no extra API calls
- Semantic retrieval with local embeddings (sub-50ms)
- with_memory(): Background extraction for non-blocking memory
- SQLite + FTS5 storage with vector similarity search
- Multi-user isolation by user_id
Memory enables temporal compression - extract key facts instead of
carrying full conversation history (4000 tokens → 50 tokens).
Includes:
- Comprehensive test suite (71 new tests)
- Documentation (docs/memory.md)
- Benchmark examples comparing approaches
- E2E test with LLM-as-judge evaluation
CodeAwareCompressor:
- Tree-sitter based AST parsing for Python, JS, TS, Go, Rust, Java, C, C++
- Preserves imports, signatures, type annotations, error handlers
- Guarantees syntactically valid output
- Uses tree-sitter-language-pack for broad language support
ContentRouter:
- Intelligent compression orchestrator
- Auto-routes content to optimal compressor based on type detection
- Source hint support for high-confidence routing
Custom Model Configuration:
- HEADROOM_MODEL_LIMITS env var and ~/.headroom/models.json support
- Pattern-based inference for unknown models (opus/sonnet/haiku tiers)
- Support for Claude 4.5, Claude 4, o3, o3-mini
- Graceful fallback - never crashes on unknown models
Integrate Microsoft's LLMLingua-2 ML-based compression as an opt-in
feature for the proxy server, with excellent developer experience.
Features:
- New CLI flags: --llmlingua, --llmlingua-device, --llmlingua-rate
- ProxyConfig options: llmlingua_enabled, llmlingua_device, llmlingua_target_rate
- Smart startup hints when llmlingua is available but not enabled
- Helpful error messages when enabled but not installed
- LLMLinguaCompressor inserted before RollingWindow in pipeline
Why opt-in:
- Heavy dependencies (~2GB torch, transformers)
- 10-30s cold start for model loading
- ~1GB RAM when loaded
- Default proxy stays lightweight (<5ms overhead)
Tests:
- 26 new tests in test_proxy_llmlingua.py covering config, setup,
banner status, CLI args, DevEx messages, and edge cases
Documentation:
- Updated README.md with proxy integration section
- Updated docs/proxy.md with LLMLingua CLI options
- Updated docs/transforms.md with LLMLinguaCompressor reference
- Updated docs/ARCHITECTURE.md with pipeline and file structure
- Updated CHANGELOG.md with new feature
- Fix 34 mypy errors across 17 files with type annotations and casts
- Add type: ignore comments for legitimate dynamic patterns
- Handle None operands with (value or 0) pattern
- Cast return values to proper types (int, float, str, bool)
- Add EstimatingTokenCounter imports where needed
- Use getattr() for potentially missing attributes
- Fix flaky test_paraphrase_match with more distinct semantic examples
- Add mlx to mypy ignore list (broken third-party stubs)
- Renamed package from 'headroom' to 'headroom-ai' (PyPI name conflict)
- Fixed numpy/jinja2 imports to be lazy (core install no longer crashes)
- Fixed SQLite default path (now uses temp directory)
- Fixed f-string {tool} crash in proxy server
- Updated README with correct package name and examples
- Added quickstart and troubleshooting docs
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
