Commit graph

23 commits

Author SHA1 Message Date
chopratejas
0161cdb386 chore(transforms): retire dead text_compressor module (Phase 3e.3)
`headroom/transforms/text_compressor.py` was a regex-line-sampling
fallback that nothing in the runtime called. ContentRouter routes
`CompressionStrategy.TEXT` straight to the Kompress ML compressor at
`content_router.py:1046` — the comment there literally says 'Prefer
Kompress ML compressor for text'. The Python file was orphaned but
still imported by its own test class, making it look live in the 3e
queue.

Drops the 3e.3 port from the queue: there's nothing to port.

# Removed
* `headroom/transforms/text_compressor.py` (255 LOC, unused)
* `tests/test_text_compressors.py::TestTextCompressor` (3 tests)
* `text_compressor` mention in `error_detection.py` shim docstring
* `text_compressor` mention in `test_signals_keyword_parity.py` docstring
* `TextCompressor` mention in `bench_latency.py` scenario comment

# Kept (defensive)
The legacy marker regex in `ccr/tool_injection.py:213` stays — it
parses an even older TextCompressor output format (pre-2026), is
purely defensive, and removal buys nothing. Test references to that
format in `test_ccr_tool_injection.py` document the regex contract
and stay too.

# Test plan
* `make ci-precheck` clean
* `tests/test_text_compressors.py` 19 passes (was 22, dropped 3)
2026-04-29 21:14:36 -07:00
rrubayet321
2ae71fe44d chore: add nosec B324 annotations to non-cryptographic MD5 usages and update temporary database path to use system temp directory 2026-04-07 13:07:26 +06:00
JerrettDavis
6f701033a1 test: align anthropic cache stability fixtures
Sync the Anthropic cache stability test double with the prefix tracker contract used by the handler.

Format the benchmark scripts that were failing ruff format --check in CI.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-06 23:26:51 -05:00
JerrettDavis
8e4d7759de Harden cache validation reporting and TTL analysis 2026-04-06 20:11:21 -05:00
JerrettDavis
ca7384402b Fix review regressions in compare, wrap, and Docker 2026-04-05 17:12:48 -05:00
JerrettDavis
5ffa77f4a3 Harden anthropic cache-mode replay stability 2026-04-05 16:11:39 -05:00
JerrettDavis
d00c6739e1 Fix CI regressions for cache benchmark work 2026-04-04 22:33:44 -05:00
JerrettDavis
96fd3d9652 Add Anthropic cache-mode delta replay 2026-04-04 22:23:10 -05:00
JerrettDavis
f2a32f9721 Harden Anthropic cache mode stability 2026-04-04 22:06:52 -05:00
JerrettDavis
1206c0ede2 Improve Claude cache simulation accounting 2026-04-04 21:54:45 -05:00
JerrettDavis
09829be0f4 Add memory-conscious Claude session benchmark harness 2026-04-04 21:19:40 -05:00
JerrettDavis
83b730f2b9 Fix CI lint/format failures after proxy mode hardening 2026-04-04 14:42:32 -05:00
JerrettDavis
54419ad8b8 Rebrand proxy modes to token/cache and harden cache-mode stability 2026-04-04 14:32:07 -05:00
chopratejas
3290a3d582 Remove LLMLingua: Kompress is the sole text compressor
LLMLingua was the original ML text compressor (BERT-based). Kompress
(ModernBERT, trained on 330K structured tool outputs) replaced it with
better compression quality and simpler architecture.

Removed across 35 files:
- Deleted headroom/transforms/llmlingua_compressor.py
- Deleted tests/test_transforms/test_llmlingua_compressor.py
- Deleted tests/test_proxy_llmlingua.py
- Removed all enable_llmlingua config, _get_llmlingua methods,
  LLMLingua fallback paths, LLMLINGUA strategy enum values
- Removed CLI flags, model configs, compression handler references
- Simplified ContentRouter: Kompress is primary and only text compressor
2026-03-26 11:11:00 -07:00
chopratejas
1474355f1c Format prefix_cache_benchmark.py for ruff 0.15 compatibility 2026-03-12 22:03:38 -07:00
chopratejas
8f438b0674 Introducing headroom wrap
- headroom wrap claude is the simplest way to start up claude
- It will also install rtk-ai locally
- rtk-ai is a cli wrapper that can save ~90% tokens for CLI calls made by Claude Code
2026-03-11 23:01:42 -07:00
chopratejas
a2b2ec5463 Add OSS evaluation suite, universal JSON crush, latency benchmarks
Evaluation Suite:
- Tiered eval framework (Tier 1 ~$3/15min, Tier 2/3 for extended coverage)
- 16 benchmarks across 3 tiers: GSM8K, TruthfulQA, MMLU, ARC, HumanEval,
  SQuAD v2, BFCL, Tool Outputs, CCR needle retention, HotpotQA, and more
- Before/After runner with full proxy support (compression + CCR retrieval)
- LLM-as-judge for ground-truth comparison (BFCL function calling)
- Zero-cost compression-only runner (CCR needle retention, info retention)
- Cost tracker with per-model pricing and budget enforcement
- Report card generator (Markdown, JSON, HTML)
- Suite CLI: python -m headroom.evals suite --tier 1
- Fix BFCL dataset loader for current HuggingFace schema
- CI workflow: PR smoke test + weekly full Tier 1

Results: SQuAD 97%, BFCL 97%, Tool Outputs 100%, CCR 100%

SmartCrusher:
- Universal JSON crush for heterogeneous arrays
- Fix mypy redefinition warning in _crush_string_array

Other:
- Latency benchmark suite with docs
- Known limitations doc
- Prompt comparison evaluator
- Config updates for new features
2026-02-23 19:08:54 -08:00
chopratejas
bd2d447c26 Add quality retention eval and fix linting for Python 3.12
- Add quality_retention_eval.py for needle-in-haystack testing to verify
  intelligent compression retains critical information (100% retention achieved)
- Add intelligent_context_integration_test.py for comprehensive pipeline testing
- Add test_progressive_summarizer.py with 36 tests for ProgressiveSummarizer
- Add HeadroomConfig parameter to HeadroomClient for direct config injection
- Update pipeline.py with IntelligentContextManager wiring and logging
- Fix all ruff linting issues and format for Python 3.12 compatibility
- Add comprehensive_eval.py benchmark for multi-scenario evaluation
- Add real_data_demo.py for production-scale volume testing
- Add reasoning agent test examples (groq, debug)
2026-01-19 21:52:18 -08:00
chopratejas
77248cd1a3 Enhance /stats endpoint and add comprehensive benchmarks
- Add detailed breakdowns by provider and model to /stats
- Include compression, telemetry, and feedback loop statistics
- Add latency.average_ms metric
- Add real-world agent benchmark with MCP tool patterns
- Add worst-case and adversarial benchmarks for edge cases
- Bump version to 0.2.12
2026-01-17 22:49:04 -08:00
chopratejas
e4a41faa33 Fix all ruff lint and format errors for CI
- Fix E402: Move module-level imports to top of file
- Fix F401: Add noqa for availability check imports
- Fix F402: Rename loop variables shadowing imports
- Fix E722: Replace bare except with except Exception
- Fix B904: Add exception chaining (from e)
- Fix F811: Remove duplicate imports
- Fix B027: Add noqa for empty close() method
- Fix E741: Rename ambiguous variable l -> label
- Fix I001: Import sorting issues
- Apply ruff format to all 106 files

All 902 tests pass.
2026-01-10 15:33:44 -08:00
chopratejas
7a05808e0f Add cache optimization module with scalable dynamic content detection
Implements a comprehensive cache optimization layer for LLM providers:

- Provider-specific optimizers (Anthropic, OpenAI, Google) with distinct
  caching strategies: explicit breakpoints, prefix stabilization, and
  CachedContent API respectively

- Scalable dynamic content detector using three strategies:
  1. Structural detection: "Label: value" patterns (language-agnostic)
  2. Entropy-based detection: high-entropy strings (IDs, tokens, hashes)
  3. Universal patterns: ISO 8601, UUIDs, JWTs, hex hashes

- NO hardcoded locale-specific patterns (no month names, etc.)

- Semantic caching layer with LRU eviction and TTL support

- Plugin registry for provider selection and custom optimizers

- 131 tests, real-world benchmarks showing 20-55% compression at <0.3ms
2026-01-07 14:07:49 -08:00
chopratejas
175746cc26 Prepare for OSS release v0.2.0
This commit prepares Headroom for public open source release with
comprehensive documentation, licensing, and community infrastructure.

License & Legal:
- Add Apache 2.0 LICENSE file
- Add NOTICE file with third-party attributions
- Add SECURITY.md for vulnerability reporting

Community:
- Add CONTRIBUTING.md with contribution guidelines
- Add CODE_OF_CONDUCT.md (Contributor Covenant)
- Add GitHub issue templates (bug report, feature request)
- Add pull request template

Documentation:
- Update README.md with compelling value proposition
- Add docs/getting-started.md
- Add docs/proxy.md for proxy server documentation
- Add docs/transforms.md for transform reference
- Add docs/api.md for API reference
- Add examples/README.md

Package Infrastructure:
- Add headroom/py.typed for PEP 561 compliance
- Add headroom/cli.py for CLI entry point
- Add .github/workflows/ci.yml for CI pipeline
- Add .github/workflows/publish.yml for PyPI publishing
- Update pyproject.toml with proper metadata

New Features:
- Add multi-provider support (Google, Cohere, LiteLLM, OpenAI-compatible)
- Add universal tokenizer registry with multiple backends
- Add model registry with pricing and context limits
- Add production proxy server with caching and rate limiting

Code Quality:
- Fix 83 lint issues via ruff auto-fix
- Fix version consistency (benchmarks 0.1.0 → 0.2.0)
- Add skip decorators for optional dependency tests
2026-01-07 11:36:44 -08:00
chopratejas
9c7d4512d6 Initial commit: Headroom SDK - LLM context optimization toolkit
A comprehensive SDK for optimizing LLM context windows, reducing token
usage while preserving critical information for AI agents.

Core Features:
- SmartCrusher: Statistical compression of tool outputs (70-85% reduction)
- CacheAligner: Prefix optimization for prompt cache hits
- RollingWindow: Intelligent context window management
- BM25/Hybrid relevance scoring for smart item selection

Integrations:
- OpenAI and Anthropic provider support
- LangChain integration (ChatModel, Callbacks, Runnable)
- MCP (Model Context Protocol) integration for tool compression

Test Coverage:
- 372 tests passing across all modules
- 35 performance benchmarks
- Real-world agent evaluations with 88% token savings

Key Components:
- headroom/transforms/: Core compression transforms
- headroom/providers/: OpenAI and Anthropic support
- headroom/integrations/: LangChain and MCP integrations
- headroom/relevance/: BM25 and hybrid scoring
- headroom/pricing/: Model pricing registry
- benchmarks/: Performance benchmark suite
- examples/: Usage examples and demos
2026-01-06 23:16:58 -08:00