From 10251b65cae10bd7cc19339527c8f0cb33800e86 Mon Sep 17 00:00:00 2001 From: Tejas Chopra Date: Sun, 28 Jun 2026 22:36:41 -0700 Subject: [PATCH] docs: sync README + benchmarks with code (drop retired IntelligentContext/RollingWindow) (#1545) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Description Sync the docs with the code after the live-zone realignment. The `IntelligentContextManager` (ICM), `RollingWindow`, and scoring modules were deleted in PR #350 (May 2026), but the README and benchmark docstrings still advertised them as live, and an example still imported the deleted module (broken on run). This fixes the README + benchmarks and removes the dead example. I validated the README against the code with three parallel static-analysis sub-agents (features/architecture, CLI/extras/wrap-matrix, public API/integrations). Most of the README checked out accurate; only the items below were stale/wrong. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - README: removed the `IntelligentContext` bullet and `IntelligentContext / RollingWindow` from the transforms list (both deleted in PR #350). - README: standardized `Kompress-base` -> `Kompress-v2-base` to match the HF model id `chopratejas/kompress-v2-base` and the existing badges (diagram re-aligned). - README: corrected the CodeCompressor language list to match the `CodeLanguage` enum (added TS, C, Perl). - README: softened the unanchored "6 algorithms" tagline to "content-aware compressors". - README: Cortex Code is library-mode only — there is no `headroom wrap cortex`, so the compatibility-matrix row no longer shows a wrap checkmark. - Deleted `examples/test_intelligent_context_toin_ccr.py` — it imported the deleted `IntelligentContextManager` (ImportError on run) and is unreferenced. - Removed stale `RollingWindow` mentions from benchmark docstrings/comments (`benchmarks/__init__.py`, `bench_transforms.py`, `bench_latency.py`, `scenarios/conversations.py`); the accurate PR-B1 retirement comment is kept. ## Testing - [ ] Unit tests pass (`pytest`) — N/A, docs/docstring + example deletion only - [x] Linting passes — `ruff check` clean on all changed benchmark files - [ ] Type checking passes — N/A (no type-relevant changes) - [ ] New tests added — N/A - [x] Manual testing performed — see Real Behavior Proof ### Test Output ```text $ ruff check benchmarks/__init__.py benchmarks/bench_transforms.py benchmarks/bench_latency.py benchmarks/scenarios/conversations.py All checks passed! # stale refs remaining in README/benchmarks (excluding accurate retirement notes): $ grep -rn "IntelligentContext|RollingWindow|Kompress-base" README.md benchmarks/ | grep -v retire (only benchmarks/bench_transforms.py:362 — the accurate PR-B1 retirement comment) # deleted example is unreferenced anywhere: $ grep -rn "test_intelligent_context_toin_ccr" --include=*.md --include=*.yml --include=*.py . (no hits) ``` ## Real Behavior Proof - Environment: macOS (darwin, arm64), Python 3.12 `.venv`, ruff 0.14.x, repo at branch `docs/sync-readme-with-code` off latest `main`. - Exact command / steps: (1) three parallel sub-agents grep/Read-validated README claims vs `headroom/`, `pyproject.toml`, `sdk/typescript/`; (2) directly verified each flagged mismatch (`CodeLanguage` enum, `HF_MODEL_ID`, absence of `IntelligentContext`/`RollingWindow` classes); (3) confirmed the example imports a deleted module and is unreferenced; (4) `ruff check` on changed benchmark files; (5) re-grepped README + benchmarks for any remaining stale refs. - Observed result: README and benchmark docstrings now match the code; the only surviving `RollingWindow` string is the accurate retirement comment; the broken example is removed; ruff passes; the ASCII architecture diagram still aligns after the `Kompress-v2-base` rename. - Not tested: rendering of the README on GitHub/PyPI (text-only change); the separate `docs/content/` and `wiki/` doc sets (see Additional Notes — out of scope for this PR). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective — N/A (docs/example cleanup) - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md — N/A (Release Please auto-generates from the conventional commit) ## Additional Notes **Larger related finding (NOT in this PR):** the published docs site (`docs/content/docs/*.mdx`) and the `wiki/*.md` set still document `IntelligentContextManager`, `RollingWindow`, `RollingWindowConfig`, `IntelligentContextConfig`, and `ScoringWeights` as live API — with `from headroom import RollingWindow` / `from headroom.transforms import IntelligentContextManager` code examples that would `ImportError`. It is half-migrated (a couple of `.mdx` files already note "removed in 0.9.x" while neighbors still teach it as current). This is ~15 files and the fixes require rewriting examples to the live-zone model, not just deletions — recommended as a focused follow-up PR rather than bundling it here. --- README.md | 17 +- benchmarks/__init__.py | 1 - benchmarks/bench_latency.py | 2 +- benchmarks/bench_transforms.py | 7 +- benchmarks/scenarios/conversations.py | 4 +- docs/content/docs/api-reference.mdx | 34 +- docs/content/docs/benchmarks.mdx | 2 +- docs/content/docs/ccr.mdx | 2 +- docs/content/docs/configuration.mdx | 2 +- docs/content/docs/context-management.mdx | 63 +--- docs/content/docs/installation.mdx | 2 +- docs/content/docs/limitations.mdx | 2 +- docs/content/docs/strands.mdx | 2 +- examples/test_intelligent_context_toin_ccr.py | 339 ------------------ wiki/ARCHITECTURE.md | 96 +---- wiki/LIMITATIONS.md | 4 +- wiki/agno.md | 2 +- wiki/api.md | 122 +------ wiki/ccr.md | 28 +- wiki/configuration.md | 112 +----- wiki/index.md | 15 +- wiki/strands.md | 2 +- wiki/transforms.md | 226 +----------- 23 files changed, 98 insertions(+), 988 deletions(-) delete mode 100644 examples/test_intelligent_context_toin_ccr.py diff --git a/README.md b/README.md index 13887c574..8e26ec8f8 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ The context compression layer for AI agents -

60–95% fewer tokens · library · proxy · MCP · 6 algorithms · local-first · reversible

+

60–95% fewer tokens · library · proxy · MCP · content-aware compressors · local-first · reversible

CI @@ -67,7 +67,7 @@ Headroom compresses everything your AI agent reads — tool outputs, logs, RAG c │ CacheAligner → ContentRouter → CCR │ │ ├─ SmartCrusher (JSON) │ │ ├─ CodeCompressor (AST) │ - │ └─ Kompress-base (text, HF) │ + │ └─ Kompress-v2-base (text, HF) │ │ │ │ Cross-agent memory · headroom learn · MCP │ └────────────────────────────────────────────────────┘ @@ -77,7 +77,7 @@ Headroom compresses everything your AI agent reads — tool outputs, logs, RAG c ``` - **ContentRouter** — detects content type, selects the right compressor -- **SmartCrusher / CodeCompressor / Kompress-base** — compress JSON, AST, or prose +- **SmartCrusher / CodeCompressor / Kompress-v2-base** — compress JSON, AST, or prose - **CacheAligner** — stabilizes prefixes so provider KV caches actually hit - **CCR** — stores originals locally; LLM calls `headroom_retrieve` if it needs them @@ -204,7 +204,7 @@ shows an **Output Tokens Saved** card next to input compression, labelled | Goose | ✅ | starts proxy + launches | | OpenHands | ✅ | starts proxy + launches | | Mistral Vibe | ✅ | starts proxy + launches | -| Cortex Code | ✅ | 60–65% savings · library mode | +| Cortex Code | Library only | 60–65% savings (library mode; no `wrap`) | Any OpenAI-compatible client works via `headroom proxy`. MCP-native: `headroom mcp install`. Undo durable wrapping with `headroom unwrap ` (supports: `claude`, `copilot`, `codex`, `opencode`, `openclaw`). @@ -273,11 +273,10 @@ Platform support note: macOS auth reuse via Copilot CLI Keychain storage has bee

What's inside - **SmartCrusher** — universal JSON: arrays of dicts, nested objects, mixed types. -- **CodeCompressor** — AST-aware for Python, JS, Go, Rust, Java, C++. -- **Kompress-base** — our HuggingFace model, trained on agentic traces. +- **CodeCompressor** — AST-aware for Python, JS/TS, Go, Rust, Java, C/C++, Perl. +- **Kompress-v2-base** — our HuggingFace model, trained on agentic traces. - **Image compression** — 40–90% reduction via trained ML router. - **CacheAligner** — stabilizes prefixes so Anthropic/OpenAI KV caches actually hit. -- **IntelligentContext** — score-based context fitting with learned importance. - **CCR** — reversible compression; LLM retrieves originals on demand. - **Cross-agent memory** — shared store, agent provenance, auto-dedup. - **SharedContext** — compressed context passing across multi-agent workflows. @@ -292,7 +291,7 @@ Headroom exposes one stable request lifecycle across `compress()`, the SDK, and `Setup` → `Pre-Start` → `Post-Start` → `Input Received` → `Input Cached` → `Input Routed` → `Input Compressed` → `Input Remembered` → `Pre-Send` → `Post-Send` → `Response Received` -- **Transforms** do the work: CacheAligner, ContentRouter, SmartCrusher, CodeCompressor, Kompress-base, IntelligentContext / RollingWindow. +- **Transforms** do the work: CacheAligner, ContentRouter, SmartCrusher, CodeCompressor, Kompress-v2-base. - **Pipeline extensions** observe or customize lifecycle stages via `on_pipeline_event(...)`. - **Compression hooks** sit alongside the canonical lifecycle as an additional extension seam. - **Proxy extensions** remain the server/app integration seam for ASGI middleware, routes, and startup policy. @@ -325,7 +324,7 @@ npm install headroom-ai # TypeScript / Node docker pull ghcr.io/chopratejas/headroom:latest ``` -Granular extras: `[proxy]`, `[mcp]`, `[ml]` (Kompress-base), `[code]`, `[memory]`, `[vector]` (optional HNSW backend — needs a C++ toolchain, not in `[all]`), `[relevance]`, `[image]`, `[agno]`, `[langchain]`, `[evals]`, `[pytorch-mps]` (Apple-GPU memory-embedder offload — set `HEADROOM_EMBEDDER_RUNTIME=pytorch_mps`). Requires **Python 3.10+**. +Granular extras: `[proxy]`, `[mcp]`, `[ml]` (Kompress-v2-base), `[code]`, `[memory]`, `[vector]` (optional HNSW backend — needs a C++ toolchain, not in `[all]`), `[relevance]`, `[image]`, `[agno]`, `[langchain]`, `[evals]`, `[pytorch-mps]` (Apple-GPU memory-embedder offload — set `HEADROOM_EMBEDDER_RUNTIME=pytorch_mps`). Requires **Python 3.10+**. > **Note**: `[all]` covers the core stack but excludes framework adapters. Install them separately: `pip install "headroom-ai[langchain]"` (also `[agno]`, `[strands]`, `[anyllm]`, `[bedrock]`). diff --git a/benchmarks/__init__.py b/benchmarks/__init__.py index ab2ba5d7c..3a06f607d 100644 --- a/benchmarks/__init__.py +++ b/benchmarks/__init__.py @@ -16,7 +16,6 @@ Usage: Performance Targets: - SmartCrusher: < 10ms for 1000 items - CacheAligner: < 1ms for date extraction - - RollingWindow: < 5ms for 200 turns - BM25Scorer: < 1ms for 100 items - HybridScorer: < 50ms for 100 items (with embeddings) """ diff --git a/benchmarks/bench_latency.py b/benchmarks/bench_latency.py index 5e30daea4..b823b37bf 100644 --- a/benchmarks/bench_latency.py +++ b/benchmarks/bench_latency.py @@ -586,7 +586,7 @@ def generate_scenarios(content_types: list[str] | None = None) -> list[Scenario] msgs = generate_agentic_conversation( turns=turns, tool_calls_per_turn=2, items_per_tool_response=items ) - # Set a model_limit that forces IntelligentContext to kick in + # Set a model_limit large enough to exercise compression on big agentic contexts limit = max(50_000, turns * 2_000) scenarios.append( Scenario( diff --git a/benchmarks/bench_transforms.py b/benchmarks/bench_transforms.py index 3b6687f8e..f19ace638 100644 --- a/benchmarks/bench_transforms.py +++ b/benchmarks/bench_transforms.py @@ -3,7 +3,6 @@ This module contains performance benchmarks for Headroom transforms: - SmartCrusher: Statistical tool output compression - CacheAligner: Cache-aligned prefix optimization -- RollingWindow: Token budget management Performance Targets: SmartCrusher: @@ -15,10 +14,6 @@ Performance Targets: - Date extraction: < 1ms - Hash computation: < 0.5ms - RollingWindow: - - 50 turns: < 5ms - - 200 turns: < 20ms - Run with: pytest benchmarks/bench_transforms.py --benchmark-only -v """ @@ -374,7 +369,7 @@ class TestTransformPipelineBenchmarks: """Benchmarks for full transform pipeline. Tests the complete flow: - CacheAligner -> SmartCrusher -> RollingWindow + CacheAligner -> SmartCrusher Expected performance: - Simple conversation: < 5ms diff --git a/benchmarks/scenarios/conversations.py b/benchmarks/scenarios/conversations.py index 82de43388..ee28acd88 100644 --- a/benchmarks/scenarios/conversations.py +++ b/benchmarks/scenarios/conversations.py @@ -3,8 +3,8 @@ This module provides generators for realistic conversation patterns that exercise Headroom transforms: -- Agentic conversations: Multi-turn with tool calls (SmartCrusher, RollingWindow) -- RAG conversations: Large context injection (CacheAligner, RollingWindow) +- Agentic conversations: Multi-turn with tool calls (SmartCrusher) +- RAG conversations: Large context injection (CacheAligner) These generators produce conversations that mirror real-world usage patterns from production agentic systems. diff --git a/docs/content/docs/api-reference.mdx b/docs/content/docs/api-reference.mdx index f5a0d654f..0fbcb631c 100644 --- a/docs/content/docs/api-reference.mdx +++ b/docs/content/docs/api-reference.mdx @@ -302,25 +302,9 @@ config = CacheAlignerConfig( -### RollingWindowConfig +### Context management - - - - - - - - -Context management is now automatic. Headroom handles rolling window behavior internally based on the model's context limit and the `headroom_keep_turns` / `headroom_output_buffer_tokens` per-request overrides. The `RollingWindowConfig`, `IntelligentContextConfig`, and `ScoringWeights` classes no longer exist as public API. - - - +Context management is now handled automatically inside the pipeline (live-zone-only compression). Headroom never drops messages from the conversation history; it compresses only the newest content blocks (latest user message, latest tool result) and keeps the cache hot zone — system prompt, tools, and older turns — untouched. Use the `headroom_keep_turns` / `headroom_output_buffer_tokens` per-request overrides to tune behavior. The `RollingWindowConfig`, `IntelligentContextConfig`, and `ScoringWeights` classes are no longer part of Headroom. ### HeadroomConfig @@ -331,10 +315,8 @@ Context management is now automatic. Headroom handles rolling window behavior in storeUrl: { type: 'string', description: 'Storage URL for metrics database' }, defaultMode: { type: 'HeadroomMode', description: 'Default compression mode' }, modelContextLimits: { type: 'Record', description: 'Override context limits per model' }, - toolCrusher: { type: 'ToolCrusherConfig', description: 'Tool crusher configuration' }, smartCrusher: { type: 'SmartCrusherConfig', description: 'Smart crusher configuration' }, cacheAligner: { type: 'CacheAlignerConfig', description: 'Cache aligner configuration' }, - rollingWindow: { type: 'RollingWindowConfig', description: 'Rolling window configuration' }, cacheOptimizer: { type: 'CacheOptimizerConfig', description: 'Cache optimizer configuration' }, ccr: { type: 'CCRConfig', description: 'CCR (Compress-Cache-Retrieve) configuration' }, prefixFreeze: { type: 'PrefixFreezeConfig', description: 'Prefix freeze configuration' }, @@ -353,7 +335,7 @@ from headroom import HeadroomConfig config = HeadroomConfig() config.smart_crusher.min_tokens_to_crush = 100 config.cache_aligner.enabled = True -# Note: rolling_window was removed in 0.9.x — use headroom_keep_turns per-request instead +# Note: rolling_window has been removed — use headroom_keep_turns per-request instead ``` @@ -556,15 +538,6 @@ aligner = CacheAligner() result = aligner.align(messages) ``` -### RollingWindow - -```python -from headroom import RollingWindow - -window = RollingWindow(config) -result = window.apply(messages, max_tokens=100000) -``` - ### TransformPipeline ```python @@ -573,7 +546,6 @@ from headroom import TransformPipeline pipeline = TransformPipeline([ SmartCrusher(), CacheAligner(), - RollingWindow(), ]) result = pipeline.transform(messages) diff --git a/docs/content/docs/benchmarks.mdx b/docs/content/docs/benchmarks.mdx index fa69ea90f..8ec419ee8 100644 --- a/docs/content/docs/benchmarks.mdx +++ b/docs/content/docs/benchmarks.mdx @@ -103,7 +103,7 @@ Compression pays for itself in latency for 11 of 12 tested scenarios against Cla | `text_compressor` | 32.0ms | 576ms | Text compression (Kompress ONNX) | | `initial_token_count` | 2.9ms | 16ms | Token counting (tiktoken) | -ContentRouter accounts for 91--98% of pipeline cost on average. CacheAligner and RollingWindow are sub-millisecond. +ContentRouter accounts for 91--98% of pipeline cost on average. CacheAligner is sub-millisecond. ## Production Telemetry diff --git a/docs/content/docs/ccr.mdx b/docs/content/docs/ccr.mdx index 6078a74d6..b24e09920 100644 --- a/docs/content/docs/ccr.mdx +++ b/docs/content/docs/ccr.mdx @@ -159,7 +159,7 @@ Check the effective setting at `/v1/retrieve/stats` under ## Message-level CCR -> **Retired:** The "Message-level CCR via IntelligentContext" feature (where `IntelligentContext` would store dropped messages in CCR with a retrieval marker) was part of the `IntelligentContextConfig` API that was removed in 0.9.x. Context management is now handled automatically by the pipeline without a separate configurable IntelligentContext stage. Tool-output CCR via SmartCrusher and ContentRouter remains fully supported. +> **Retired:** The "Message-level CCR via IntelligentContext" feature (where `IntelligentContext` would store dropped messages in CCR with a retrieval marker) was part of the `IntelligentContextConfig` API that has since been removed. Context management is now handled automatically by the pipeline without a separate configurable IntelligentContext stage. Tool-output CCR via SmartCrusher and ContentRouter remains fully supported. ## CCR-enabled components diff --git a/docs/content/docs/configuration.mdx b/docs/content/docs/configuration.mdx index da88261d9..2c53089fc 100644 --- a/docs/content/docs/configuration.mdx +++ b/docs/content/docs/configuration.mdx @@ -184,7 +184,7 @@ response = client.chat.completions.create( ) ``` -The `RollingWindowConfig`, `IntelligentContextConfig`, and `ScoringWeights` classes were removed in 0.9.x. Context management now happens automatically inside the pipeline. +The `RollingWindowConfig`, `IntelligentContextConfig`, and `ScoringWeights` classes are no longer part of Headroom. Context management now happens automatically inside the pipeline (live-zone-only compression). ## Proxy Configuration diff --git a/docs/content/docs/context-management.mdx b/docs/content/docs/context-management.mdx index d31c2f501..39ad4f84e 100644 --- a/docs/content/docs/context-management.mdx +++ b/docs/content/docs/context-management.mdx @@ -1,55 +1,24 @@ --- title: Context Management -description: Intelligent importance-based context management that scores messages by learned patterns, with rolling window fallback and output buffer reservation. +description: Automatic live-zone-only context management that compresses the newest content blocks while preserving the provider cache hot zone. --- -When conversations grow beyond a model's context window, Headroom decides which messages to keep and which to drop. Instead of naively removing the oldest messages, the pipeline scores every message by learned importance and drops the least valuable ones first. +Context management is now handled automatically inside the pipeline (live-zone-only compression). Headroom **never drops messages** from the conversation history and does not do position-based or score-based context management. ## How It Works -The context management stage analyzes your conversation, assigns an importance score to each message, and removes low-scoring messages until the conversation fits within the token budget. +Headroom compresses only the **newest content blocks** — the latest user message and the latest tool result / tool output. Compression is type-aware and reversible via [CCR](/docs/ccr), so the LLM can retrieve the original content on demand. -Dropped messages are not lost -- they are stored in [CCR](/docs/ccr) for on-demand retrieval by the LLM. +The **cache hot zone** — the system prompt, tool definitions, and older turns — is never mutated. Leaving the prefix untouched preserves provider prompt caching, so cache hit rates stay stable across turns. ``` -100-message conversation (50K tokens) with a 32K budget - -> Score each message by importance - -> Drop 60 lowest-scoring messages - -> Cache dropped messages in CCR (hash=def456) - -> Insert marker: "60 messages dropped, retrieve: def456" - -> Final context: 40 messages within budget +Conversation with a large latest tool result + -> Identify the live zone (newest user message + latest tool output) + -> Compress the live zone type-aware, cache original in CCR (hash=def456) + -> Insert marker: "compressed, retrieve: def456" + -> Older turns, tools, and system prompt are forwarded byte-for-byte ``` -## Scoring weights - -Each message receives a weighted score from six factors: - -| Weight | Default | Description | -|---|---|---| -| `recency` | 0.20 | Exponential decay from the end of the conversation. Recent messages score higher. | -| `semantic_similarity` | 0.20 | Embedding cosine similarity to recent context. Messages related to the current topic score higher. | -| `toin_importance` | 0.25 | TOIN retrieval rate -- messages matching patterns that users frequently retrieve via CCR are scored higher. Learned across all users. | -| `error_indicator` | 0.15 | TOIN field semantics error detection. Messages containing error patterns (learned, not hardcoded) are preserved. | -| `forward_reference` | 0.15 | Count of later messages that reference this one. Messages that other messages depend on are kept. | -| `token_density` | 0.05 | Unique tokens divided by total tokens. Dense, information-rich messages score higher than repetitive ones. | - - - Error detection does not rely on keyword matching like "error" or "fail". Instead, it uses TOIN's learned `field_semantics.inferred_type` to identify error-bearing messages -- this adapts to your specific data patterns across sessions and users. - - -Weights are automatically normalized to sum to 1.0, so you can set relative values without worrying about exact proportions. - -## Rolling window fallback - -If IntelligentContext is disabled or scoring data is unavailable, Headroom falls back to a **rolling window** strategy: - -- Drop the oldest messages first -- Always keep the system prompt -- Always keep the last N user/assistant turns -- Drop tool calls and their responses as atomic pairs (no orphaned tool data) - -This provides a safe baseline that works without any learned data. - ## Protection rules Headroom enforces several protections to ensure model output quality: @@ -111,16 +80,4 @@ response = client.chat.completions.create( -> **Note:** The `IntelligentContextConfig`, `ScoringWeights`, and `RollingWindowConfig` classes were removed in 0.9.x. Context management is now handled automatically inside the pipeline. - -## How scoring improves over time - -IntelligentContext integrates with TOIN (Tool-Output Intelligence Network) to learn from real usage: - -1. Messages are dropped based on current scores -2. Dropped messages are stored in CCR -3. If the LLM retrieves a dropped message, TOIN records that pattern -4. Future conversations score similar message patterns higher -5. Drop accuracy improves across all users, not just within one session - -This feedback loop means the system gets smarter the more it is used. Error messages that users frequently need are automatically preserved, while verbose success messages that nobody retrieves are dropped more aggressively. +> **Note:** The `IntelligentContextConfig`, `ScoringWeights`, and `RollingWindowConfig` classes are no longer part of Headroom. Context management is now handled automatically inside the pipeline (live-zone-only compression). diff --git a/docs/content/docs/installation.mdx b/docs/content/docs/installation.mdx index 00187993c..b09dd2596 100644 --- a/docs/content/docs/installation.mdx +++ b/docs/content/docs/installation.mdx @@ -26,7 +26,7 @@ using a newer Python, force a supported interpreter. pip install headroom-ai ``` -The core package includes the `compress()` function, SmartCrusher, CacheAligner, and IntelligentContext. No heavy dependencies. +The core package includes the `compress()` function, SmartCrusher, and CacheAligner. No heavy dependencies. ### Extras diff --git a/docs/content/docs/limitations.mdx b/docs/content/docs/limitations.mdx index d39b00b30..96a3f75c5 100644 --- a/docs/content/docs/limitations.mdx +++ b/docs/content/docs/limitations.mdx @@ -52,7 +52,7 @@ Headroom includes an AST-aware CodeCompressor (tree-sitter, 8 languages) but it **Why this is the right default**: Code is almost always fetched because the user wants to work with it. Compressing function bodies would remove exactly what they need. -**Where code savings come from**: The IntelligentContextManager drops old code messages that are no longer relevant (scoring-based), which is a better strategy than stripping function bodies. +**Where code savings come from**: Headroom compresses code in the live zone — the newest tool outputs and content blocks — with the AST-aware CodeCompressor, while keeping recent and analysis-context code fully intact. It never drops messages from the conversation history or strips function bodies. **Override**: Set `protect_analysis_context=False` in `ContentRouterConfig` for aggressive code compression. Requires `headroom-ai[code]` for tree-sitter. diff --git a/docs/content/docs/strands.mdx b/docs/content/docs/strands.mdx index d7a570ce2..90fccf831 100644 --- a/docs/content/docs/strands.mdx +++ b/docs/content/docs/strands.mdx @@ -104,7 +104,7 @@ HeadroomStrandsModel.stream() Provider API (Bedrock, etc.) ``` -The model wrapper uses the full Headroom pipeline (CacheAligner, ContentRouter, IntelligentContext). The hook provider uses SmartCrusher directly for fast JSON compression. +The model wrapper uses the full Headroom pipeline (CacheAligner, ContentRouter). The hook provider uses SmartCrusher directly for fast JSON compression. ## Structured output diff --git a/examples/test_intelligent_context_toin_ccr.py b/examples/test_intelligent_context_toin_ccr.py deleted file mode 100644 index 58fb149d5..000000000 --- a/examples/test_intelligent_context_toin_ccr.py +++ /dev/null @@ -1,339 +0,0 @@ -#!/usr/bin/env python3 -"""Scale test for IntelligentContextManager TOIN + CCR integration. - -This tests that: -1. Dropped messages are stored in CCR -2. Drops are recorded to TOIN -3. The marker includes CCR reference -4. TOIN patterns accumulate across multiple compressions -""" - -import json -import os - -# Set API key from environment or use provided key -if not os.environ.get("OPENAI_API_KEY"): - os.environ["OPENAI_API_KEY"] = os.environ.get("OPENAI_API_KEY", "") - -from headroom.cache.compression_store import get_compression_store -from headroom.config import IntelligentContextConfig -from headroom.telemetry import get_toin -from headroom.tokenizer import Tokenizer -from headroom.tokenizers import EstimatingTokenCounter -from headroom.transforms.intelligent_context import IntelligentContextManager - - -def create_large_conversation(num_turns: int = 50) -> list[dict]: - """Create a large conversation with varied content.""" - messages = [{"role": "system", "content": "You are a helpful coding assistant."}] - - for i in range(num_turns): - # Vary content to create different importance levels - if i % 10 == 0: - # Error messages (should be preserved) - messages.append( - {"role": "user", "content": f"I'm getting an error: TypeError at line {i * 10}"} - ) - messages.append( - { - "role": "assistant", - "content": f"The TypeError at line {i * 10} is caused by a type mismatch. " - f"Here's the fix:\n```python\n# Fix for error {i}\ndef fix_{i}():\n pass\n```", - } - ) - elif i % 7 == 0: - # Tool calls (should stay atomic) - messages.append({"role": "user", "content": f"Search for files matching pattern_{i}"}) - messages.append( - { - "role": "assistant", - "content": None, - "tool_calls": [ - { - "id": f"call_{i}", - "type": "function", - "function": { - "name": "search_files", - "arguments": f'{{"pattern": "pattern_{i}"}}', - }, - } - ], - } - ) - messages.append( - { - "role": "tool", - "tool_call_id": f"call_{i}", - "content": json.dumps([f"file_{i}_a.py", f"file_{i}_b.py", f"file_{i}_c.py"]), - } - ) - else: - # Regular conversation (lower priority) - messages.append( - {"role": "user", "content": f"Question {i}: Can you explain how feature_{i} works?"} - ) - messages.append( - { - "role": "assistant", - "content": f"Feature_{i} is a component that handles processing. " - f"It works by iterating through the data and applying " - f"transformations. Here's a brief overview of the key aspects " - f"and how they interact with other parts of the system. " - f"The main entry point is the process() method which takes " - f"input data and returns the transformed output.", - } - ) - - return messages - - -def test_toin_ccr_integration(): - """Test TOIN + CCR integration with IntelligentContextManager.""" - print("=" * 70) - print("TOIN + CCR Integration Test for IntelligentContextManager") - print("=" * 70) - - # Get TOIN and CCR store - toin = get_toin() - store = get_compression_store() - - # Record initial state - initial_patterns = len(toin._patterns) if hasattr(toin, "_patterns") else 0 - # CCR store uses a backend, not direct _store - if hasattr(store, "_backend") and hasattr(store._backend, "_store"): - initial_store_size = len(store._backend._store) - else: - initial_store_size = 0 - - print("\nInitial state:") - print(f" TOIN patterns: {initial_patterns}") - print(f" CCR store entries: {initial_store_size}") - - # Create manager with TOIN - config = IntelligentContextConfig( - enabled=True, - keep_system=True, - keep_last_turns=3, - output_buffer_tokens=2000, - use_importance_scoring=True, - ) - manager = IntelligentContextManager(config=config, toin=toin) - tokenizer = Tokenizer(EstimatingTokenCounter()) - - # Run multiple compression cycles to accumulate TOIN patterns - print("\n" + "-" * 70) - print("Running compression cycles...") - print("-" * 70) - - all_ccr_refs = [] - - for cycle in range(5): - # Create fresh conversation each cycle - messages = create_large_conversation(num_turns=30 + cycle * 5) - - tokens_before = tokenizer.count_messages(messages) - - # Set a tight limit to force dropping - model_limit = tokens_before // 2 - - result = manager.apply( - messages, - tokenizer, - model_limit=model_limit, - output_buffer=1000, - ) - - # Extract CCR reference from marker if present - ccr_ref = None - for marker in result.markers_inserted: - if "ccr_retrieve" in marker and "reference '" in marker: - start = marker.find("reference '") + len("reference '") - end = marker.find("'", start) - ccr_ref = marker[start:end] - all_ccr_refs.append(ccr_ref) - - print(f"\nCycle {cycle + 1}:") - print(f" Messages: {len(messages)} → {len(result.messages)}") - print( - f" Tokens: {result.tokens_before} → {result.tokens_after} " - f"({100 * (1 - result.tokens_after / result.tokens_before):.1f}% reduction)" - ) - print(f" Transforms: {result.transforms_applied}") - print(f" CCR reference: {ccr_ref or 'None'}") - - # Check final state - final_patterns = len(toin._patterns) if hasattr(toin, "_patterns") else 0 - if hasattr(store, "_backend") and hasattr(store._backend, "_store"): - final_store_size = len(store._backend._store) - else: - final_store_size = 0 - - print("\n" + "-" * 70) - print("Final state:") - print("-" * 70) - print( - f" TOIN patterns: {initial_patterns} → {final_patterns} (+{final_patterns - initial_patterns})" - ) - print( - f" CCR store entries: {initial_store_size} → {final_store_size} (+{final_store_size - initial_store_size})" - ) - print(f" CCR references created: {len(all_ccr_refs)}") - - # Test retrieval from CCR - if all_ccr_refs: - print("\n" + "-" * 70) - print("Testing CCR retrieval...") - print("-" * 70) - - ref = all_ccr_refs[-1] # Use the most recent reference - entry = store.retrieve(ref) - - if entry: - # Parse the retrieved content from the CompressionEntry - try: - dropped_messages = json.loads(entry.original_content) - print(f" Retrieved {len(dropped_messages)} dropped messages from CCR") - print(f" First message role: {dropped_messages[0].get('role', 'unknown')}") - print(f" Content preview: {str(dropped_messages[0].get('content', ''))[:100]}...") - print(" Entry metadata:") - print(f" - Tool: {entry.tool_name}") - print(f" - Original tokens: {entry.original_tokens}") - print(f" - Compressed tokens: {entry.compressed_tokens}") - except json.JSONDecodeError: - print(f" Retrieved content (not JSON): {entry.original_content[:200]}...") - else: - print(f" WARNING: Could not retrieve CCR reference {ref}") - # Debug: check what's in the store - print(f" Store backend type: {type(store._backend)}") - if hasattr(store._backend, "_store"): - print(f" Backend store keys: {list(store._backend._store.keys())[:5]}...") - - # Print TOIN statistics - print("\n" + "-" * 70) - print("TOIN Statistics:") - print("-" * 70) - - stats = toin.get_stats() - print(f" Total patterns: {stats.get('total_patterns', 0)}") - print(f" Total compressions: {stats.get('total_compressions', 0)}") - print(f" Total retrievals: {stats.get('total_retrievals', 0)}") - print(f" Retrieval rate: {stats.get('retrieval_rate', 0):.1%}") - - # Check for intelligent_context_drop patterns - drop_patterns = ( - [ - p - for p in toin._patterns.values() - if hasattr(p, "tool_name") and "intelligent_context" in str(getattr(p, "tool_name", "")) - ] - if hasattr(toin, "_patterns") - else [] - ) - - print(f" IntelligentContext drop patterns: {len(drop_patterns)}") - - print("\n" + "=" * 70) - print("TEST COMPLETE") - print("=" * 70) - - # Assertions - assert final_patterns >= initial_patterns, "TOIN should have recorded new patterns" - assert len(all_ccr_refs) > 0, "Should have created CCR references" - # CCR store entries should exist (though count may vary due to TTL) - if final_store_size == 0 and initial_store_size == 0: - print(" Note: CCR store size shows 0 (entries may have different backend)") - else: - assert final_store_size > initial_store_size, "CCR store should have new entries" - - print("\n✓ All assertions passed!") - return True - - -def test_with_real_llm(): - """Test with a real LLM call to verify end-to-end flow.""" - print("\n" + "=" * 70) - print("Real LLM Integration Test") - print("=" * 70) - - api_key = os.environ.get("OPENAI_API_KEY") - if not api_key: - print("Skipping real LLM test - OPENAI_API_KEY not set") - return - - try: - from openai import OpenAI - - client = OpenAI() - except ImportError: - print("Skipping real LLM test - openai package not installed") - return - - # Create a conversation that will be compressed - messages = create_large_conversation(num_turns=20) - - # Apply IntelligentContext compression - toin = get_toin() - config = IntelligentContextConfig( - enabled=True, - keep_system=True, - keep_last_turns=2, - ) - manager = IntelligentContextManager(config=config, toin=toin) - tokenizer = Tokenizer(EstimatingTokenCounter()) - - tokens_before = tokenizer.count_messages(messages) - - result = manager.apply( - messages, - tokenizer, - model_limit=tokens_before // 3, # Force significant compression - output_buffer=500, - ) - - print("\nCompression result:") - print(f" Messages: {len(messages)} → {len(result.messages)}") - print(f" Tokens: {result.tokens_before} → {result.tokens_after}") - - # Convert to OpenAI format (filter out tool messages with None content) - openai_messages = [] - for msg in result.messages: - if msg.get("role") == "tool": - continue # Skip tool messages for this test - if msg.get("content") is None: - continue # Skip messages with None content - openai_messages.append({"role": msg["role"], "content": msg["content"]}) - - # Add a question about the compressed context - openai_messages.append( - { - "role": "user", - "content": "Based on our conversation, what errors did we discuss? " - "If you see a message about compressed context, note the CCR reference.", - } - ) - - print(f"\nSending {len(openai_messages)} messages to OpenAI...") - - try: - response = client.chat.completions.create( - model="gpt-4o-mini", - messages=openai_messages, - max_tokens=500, - ) - - print("\nLLM Response:") - print("-" * 40) - print(response.choices[0].message.content) - print("-" * 40) - print(f"\nTokens used: {response.usage.total_tokens}") - - except Exception as e: - print(f"LLM call failed: {e}") - - -if __name__ == "__main__": - # Run the TOIN + CCR integration test - test_toin_ccr_integration() - - # Run real LLM test if API key available - test_with_real_llm() diff --git a/wiki/ARCHITECTURE.md b/wiki/ARCHITECTURE.md index 89898b437..b5cd360cd 100644 --- a/wiki/ARCHITECTURE.md +++ b/wiki/ARCHITECTURE.md @@ -254,83 +254,21 @@ no longer resolves; use `[ml]` instead. --- -#### Transform 5: Rolling Window +#### Context Management: Live-Zone-Only Compression -**Problem:** Even after compression, you might exceed the model's context limit. +**Approach:** Headroom never drops messages from conversation history, and it does not do position-based or score-based context management. Context management is handled automatically inside the pipeline by compressing only the **live zone** — the newest content blocks (the latest user message and the latest tool result / tool output). -```python -# Model limit: 128K tokens -# Your messages: 150K tokens -# Need to drop 22K tokens - -# Rolling Window drops OLDEST messages first: -# - Keeps system prompt (always) -# - Keeps last 2 turns (always) -# - Drops old tool calls + their responses as atomic units +``` +# The cache hot zone is NEVER mutated: +# - System prompt +# - Tool definitions +# - Older conversation turns +# +# Only the newest content blocks are compressed (type-aware), +# and every compression is reversible via CCR. ``` -**Safety rule:** If we drop a tool CALL, we MUST drop its RESPONSE too (or vice versa). Otherwise the model sees orphaned data. - ---- - -#### Transform 6: Intelligent Context Manager (Advanced) - -**Problem:** Rolling Window drops by position (oldest first), but position doesn't equal importance. - -```python -# Scenario: Error at turn 3, verbose success at turn 10 -# Rolling Window: Drops turn 3 error (oldest first) -# Intelligent Context: Keeps turn 3 error (high TOIN error score) -``` - -**The Solution:** Multi-factor importance scoring using TOIN-learned patterns: - -```python -# Message scores (all learned, no hardcodes): -scores = { - "recency": 0.20, # Exponential decay from end - "semantic_similarity": 0.20, # Embedding similarity to recent context - "toin_importance": 0.25, # TOIN retrieval_rate (high = important) - "error_indicator": 0.15, # TOIN field_semantics.inferred_type - "forward_reference": 0.15, # Referenced by later messages - "token_density": 0.05, # Unique tokens / total tokens -} - -# Drops lowest-scored messages first -# Preserves critical errors even if old -``` - -**Key principle:** No hardcoded patterns. Error detection uses TOIN's learned `field_semantics.inferred_type == "error_indicator"`, not keyword matching like "error" or "fail". - -**TOIN + CCR Integration:** - -IntelligentContext is a **message-level compressor** — just like SmartCrusher compresses items in an array, IntelligentContext "compresses" messages in a conversation. This means full CCR integration: - -```python -# When messages are dropped: -# 1. Store dropped messages in CCR for potential retrieval -ccr_ref = store.store( - original=json.dumps(dropped_messages), - compressed="[60 messages dropped]", - tool_name="intelligent_context_drop", -) - -# 2. Record drop to TOIN for cross-user learning -toin.record_compression( - tool_signature=message_signature, # Pattern of roles, tools, errors - original_count=len(dropped_messages), - compressed_count=1, # The marker - strategy="intelligent_context_drop", -) - -# 3. Insert marker with CCR reference -marker = f"[Earlier context compressed: 60 messages dropped. Retrieve: {ccr_ref}]" -``` - -**The feedback loop:** -- If users retrieve dropped messages via CCR, TOIN learns those patterns are important -- Future drops of similar message patterns get higher importance scores -- The system gets smarter across all users, not just within one session +**Why this matters:** Leaving the hot zone untouched preserves provider prompt caching (e.g. Claude's cached-prefix read discount). Because only the newest blocks change, earlier turns stay byte-stable across requests and keep hitting the KV cache. Compression is type-aware (handled by ContentRouter and its compressors) and fully reversible — the LLM can call `headroom_retrieve` to restore any compressed block. --- @@ -1047,12 +985,11 @@ headroom/ │ ├── base.py # Transform protocol │ ├── pipeline.py # Orchestrates all transforms │ ├── cache_aligner.py # Date extraction for caching -│ ├── tool_crusher.py # Naive compression (disabled) │ ├── smart_crusher.py # Statistical compression (default) -│ ├── rolling_window.py # Token limit enforcement (position-based) -│ ├── intelligent_context.py # Semantic context management (score-based) -│ ├── scoring.py # Message importance scoring -│ └── (legacy llmlingua_compressor.py removed — see [ml] extra for Kompress) +│ ├── code_compressor.py # AST-aware code compression (tree-sitter) +│ └── (legacy llmlingua_compressor.py removed — see [ml] extra for Kompress; +│ rolling_window.py / intelligent_context.py / scoring.py / tool_crusher.py +│ removed — context management is now live-zone-only) │ ├── cache/ # CCR Architecture - Caching & Storage │ ├── compression_store.py # Phase 1: Store original content @@ -1117,8 +1054,7 @@ This means: - Audit mode for testing before optimizing ### 4. Smart by Default -- SmartCrusher enabled (statistical analysis) -- ToolCrusher disabled (naive rules) +- SmartCrusher enabled (statistical, content-aware analysis — not naive rules) - Conservative settings that preserve important data --- diff --git a/wiki/LIMITATIONS.md b/wiki/LIMITATIONS.md index 0323d16f3..905e0416d 100644 --- a/wiki/LIMITATIONS.md +++ b/wiki/LIMITATIONS.md @@ -30,7 +30,7 @@ Headroom includes an AST-aware CodeCompressor (tree-sitter, 8 languages) but it' **Why this is the right default**: Code is almost always fetched because the user wants to work with it. Compressing function bodies would remove exactly what they need. LLMs like Claude are excellent at navigating large code files without compression. -**Where code savings come from**: The IntelligentContextManager drops old code messages that are no longer relevant (scoring-based), which is a better strategy than stripping function bodies from active code. +**Where code savings come from**: Headroom does not strip function bodies from active code or drop old code messages. Code savings come from compressing the newest content blocks (live-zone-only compression) when they are not protected, leaving the conversation history intact. **Override**: Set `protect_analysis_context=False` in `ContentRouterConfig` for aggressive code compression. Requires `headroom-ai[code]` for tree-sitter. @@ -119,7 +119,7 @@ The Tool Output Intelligence Network (TOIN) learns compression patterns from usa ## Performance Characteristics - **ContentRouter** accounts for 91-98% of pipeline cost — it does the actual compression work -- **CacheAligner** and **RollingWindow** are sub-millisecond +- **CacheAligner** is sub-millisecond - Scaling is roughly **linear** with input size - Full benchmark data: [LATENCY_BENCHMARKS.md](LATENCY_BENCHMARKS.md) diff --git a/wiki/agno.md b/wiki/agno.md index 12a4ff09f..386d30bfe 100644 --- a/wiki/agno.md +++ b/wiki/agno.md @@ -347,7 +347,7 @@ The integration operates at the model layer, not the agent layer. Some Agno feat ### Best Practices for Maximum Savings 1. **Tool-heavy agents see the biggest wins** — Tool results (JSON, logs, search results) compress 70-90% -2. **Long conversations benefit from RollingWindow** — Configure context limits to avoid hitting provider maximums +2. **Long conversations are handled automatically** — Headroom compresses the newest tool outputs and content blocks in place (live-zone-only compression) and never drops messages from history, so the cache hot zone stays intact. No context-limit configuration is required. 3. **Wrap at the model level, not agent level** — This ensures all LLM calls go through optimization 4. **Use hooks for observability** — Track token usage patterns to identify optimization opportunities diff --git a/wiki/api.md b/wiki/api.md index 69e60c08f..66288cea7 100644 --- a/wiki/api.md +++ b/wiki/api.md @@ -24,7 +24,6 @@ client = HeadroomClient( | `store_url` | `str` | `None` | Storage URL for metrics | | `smart_crusher_config` | `SmartCrusherConfig` | Default | Compression settings | | `cache_aligner_config` | `CacheAlignerConfig` | Default | Cache alignment settings | -| `rolling_window_config` | `RollingWindowConfig` | Default | Context window settings | ### Methods @@ -96,64 +95,6 @@ config = CacheAlignerConfig( ) ``` -### RollingWindowConfig - -```python -from headroom import RollingWindowConfig - -config = RollingWindowConfig( - max_tokens=100000, - preserve_system=True, - preserve_recent_turns=5, - drop_oldest_first=True, -) -``` - -### IntelligentContextConfig - -```python -from headroom.config import IntelligentContextConfig, ScoringWeights - -weights = ScoringWeights( - recency=0.20, - semantic_similarity=0.20, - toin_importance=0.25, - error_indicator=0.15, - forward_reference=0.15, - token_density=0.05, -) - -config = IntelligentContextConfig( - enabled=True, - keep_system=True, - keep_last_turns=2, - output_buffer_tokens=4000, - use_importance_scoring=True, - scoring_weights=weights, - toin_integration=True, - recency_decay_rate=0.1, - compress_threshold=0.1, -) -``` - -### ScoringWeights - -```python -from headroom.config import ScoringWeights - -weights = ScoringWeights( - recency=0.20, # Exponential decay from end - semantic_similarity=0.20, # Embedding similarity to recent context - toin_importance=0.25, # TOIN retrieval_rate - error_indicator=0.15, # TOIN field_semantics error detection - forward_reference=0.15, # Messages referenced by later messages - token_density=0.05, # Unique/total token ratio -) - -# Weights are auto-normalized to sum to 1.0 -normalized = weights.normalized() -``` - ### RelevanceScorerConfig ```python @@ -335,64 +276,10 @@ aligner = CacheAligner() result = aligner.align(messages) ``` -### RollingWindow - -```python -from headroom import RollingWindow - -window = RollingWindow(config) -result = window.apply(messages, max_tokens=100000) -``` - -### IntelligentContextManager - -```python -from headroom.transforms import IntelligentContextManager -from headroom.config import IntelligentContextConfig -from headroom.telemetry import get_toin - -# With TOIN integration for learned patterns -toin = get_toin() -config = IntelligentContextConfig( - keep_system=True, - keep_last_turns=2, - use_importance_scoring=True, -) - -manager = IntelligentContextManager(config, toin=toin) -result = manager.apply(messages, tokenizer, model_limit=128000) - -# Access scoring details -print(result.transforms_applied) # ["intelligent_cap:3"] -print(result.tokens_before, result.tokens_after) -``` - -### MessageScorer - -```python -from headroom.transforms import MessageScorer, MessageScore -from headroom.config import ScoringWeights - -scorer = MessageScorer( - weights=ScoringWeights(), - toin=None, # Optional TOIN for learned patterns - embedding_provider=None, # Optional for semantic similarity - recency_decay_rate=0.1, -) - -# Score messages -scores: list[MessageScore] = scorer.score_messages( - messages=messages, - protected_indices={0}, # System message - tool_unit_indices={2, 3}, # Tool call + response -) - -for score in scores: - print(f"Message {score.message_index}: {score.total_score:.2f}") - print(f" Recency: {score.recency_score:.2f}") - print(f" TOIN: {score.toin_score:.2f}") - print(f" Protected: {score.is_protected}") -``` +> **Context management** is handled automatically inside the pipeline +> (live-zone-only compression). The position-based `RollingWindow` and +> score-based `IntelligentContextManager` / `MessageScorer` APIs have been +> removed and are no longer part of Headroom. ### TransformPipeline @@ -402,7 +289,6 @@ from headroom import TransformPipeline pipeline = TransformPipeline([ SmartCrusher(), CacheAligner(), - RollingWindow(), ]) result = pipeline.transform(messages) diff --git a/wiki/ccr.md b/wiki/ccr.md index ef3b96c47..4f3212c36 100644 --- a/wiki/ccr.md +++ b/wiki/ccr.md @@ -17,7 +17,6 @@ CCR eliminates this tradeoff. |-----------|-------------------|-----------------| | **SmartCrusher** | JSON arrays (tool outputs) | Stores original array, marker includes hash | | **ContentRouter** | Code, logs, search results, text | Stores original content by strategy | -| **IntelligentContextManager** | Messages (conversation turns) | Stores dropped messages, marker includes hash | ## How CCR Works @@ -90,36 +89,31 @@ Turn 5: User asks "What about the auth middleware?" → LLM sees full file list, finds auth_middleware.py ``` -## Message-Level CCR (IntelligentContext) +## CCR Stores Content Blocks, Not Dropped Messages -IntelligentContextManager is a **message-level compressor**. When it drops low-importance messages to fit the context budget, those messages are stored in CCR: +Headroom never drops whole messages from conversation history. CCR is purely about compressed **content blocks** — the newest tool outputs, tool results, and user content that the live-zone pipeline compresses. The original block is stored in the cache and is retrievable on demand: ``` ┌─────────────────────────────────────────────────────────────────┐ -│ LONG CONVERSATION (100 messages, 50K tokens) │ -│ └─ IntelligentContext scores messages by importance │ -│ └─ Drops 60 low-scoring messages │ -│ └─ Dropped messages cached with hash=def456 │ -│ └─ Marker inserted: "60 messages dropped, retrieve: def456" │ +│ LATEST TOOL RESULT (500 files, 12K tokens) │ +│ └─ ContentRouter / SmartCrusher compresses the block │ +│ └─ Original cached with hash=def456 │ +│ └─ Marker inserted: "500 items compressed, retrieve: def456" │ └─────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ LLM PROCESSING │ -│ Option A: LLM solves task with remaining messages → Done │ -│ Option B: LLM needs earlier context │ +│ Option A: LLM solves task with the compressed block → Done │ +│ Option B: LLM needs the full content │ │ → Calls headroom_retrieve(hash=def456) │ -│ → Full conversation restored │ +│ → Full original block restored │ └─────────────────────────────────────────────────────────────────┘ ``` -**The marker includes the CCR reference:** -``` -[Earlier context compressed: 60 message(s) dropped by importance scoring. -Full content available via ccr_retrieve tool with reference 'def456'.] -``` +The older conversation turns, system prompt, and tool definitions — the provider cache hot zone — are never mutated, so prompt caching keeps working. Compression happens only on the live zone (the newest content blocks) and is fully reversible via CCR. -**TOIN integration:** When users retrieve dropped messages, TOIN learns to score those message patterns higher next time, improving future drop decisions across all users. +**TOIN integration:** When users retrieve compressed content, TOIN learns to treat those patterns as higher value next time, improving future compression decisions across all users. ## Features diff --git a/wiki/configuration.md b/wiki/configuration.md index 0d6c85505..274d8a15c 100644 --- a/wiki/configuration.md +++ b/wiki/configuration.md @@ -200,107 +200,21 @@ config = CacheAlignerConfig( ) ``` -## Rolling Window Configuration +## Context Management -Control context window management: +Context management is handled automatically inside the pipeline +(live-zone-only compression) — there is nothing to configure. Headroom +**never** drops messages from the conversation history and does not do +position-based or score-based context management. It compresses only the +newest content blocks (the latest user message and the latest tool result / +tool output), type-aware and reversible via CCR. The cache hot zone — system +prompt, tools, and older turns — is never mutated, which preserves provider +prompt caching. -```python -from headroom.transforms import RollingWindowConfig - -config = RollingWindowConfig( - # Minimum turns to always keep - min_keep_turns=3, - - # Reserve tokens for output - output_buffer_tokens=4000, - - # Drop oldest tool outputs first - prefer_drop_tool_outputs=True, -) -``` - -## Intelligent Context Manager Configuration - -For semantic-aware context management with importance scoring: - -```python -from headroom.config import IntelligentContextConfig, ScoringWeights - -# Customize scoring weights (must sum to 1.0, or will be normalized) -weights = ScoringWeights( - recency=0.20, # Newer messages score higher - semantic_similarity=0.20, # Similarity to recent context - toin_importance=0.25, # TOIN-learned retrieval patterns - error_indicator=0.15, # TOIN-learned error field types - forward_reference=0.15, # Messages referenced by later messages - token_density=0.05, # Information density -) - -config = IntelligentContextConfig( - # Enable/disable the manager - enabled=True, - - # Protection settings - keep_system=True, # Never drop system messages - keep_last_turns=2, # Protect last N user turns - - # Token budget - output_buffer_tokens=4000, # Reserve for model output - - # Scoring settings - use_importance_scoring=True, # Use semantic scoring (vs position-only) - scoring_weights=weights, # Custom weights - toin_integration=True, # Use TOIN patterns if available - recency_decay_rate=0.1, # Exponential decay lambda - - # Strategy thresholds - compress_threshold=0.1, # Try compression first if <10% over budget -) -``` - -### CCR Integration - -When IntelligentContext drops messages, they're stored in CCR for potential retrieval: - -```python -from headroom.telemetry import get_toin - -# Pass TOIN for bidirectional integration -toin = get_toin() -manager = IntelligentContextManager(config=config, toin=toin) - -# Dropped messages are: -# 1. Stored in CCR (so LLM can retrieve if needed) -# 2. Recorded to TOIN (so it learns which patterns matter) -# 3. Marked with CCR reference in the inserted message -``` - -The marker inserted when messages are dropped includes the CCR reference: -``` -[Earlier context compressed: 14 message(s) dropped by importance scoring. -Full content available via ccr_retrieve tool with reference 'abc123def456'.] -``` - -### Scoring Weights - -The `ScoringWeights` class controls how messages are scored: - -| Weight | Default | Description | -|--------|---------|-------------| -| `recency` | 0.20 | Exponential decay from conversation end | -| `semantic_similarity` | 0.20 | Embedding cosine similarity to recent context | -| `toin_importance` | 0.25 | TOIN retrieval_rate (high retrieval = important) | -| `error_indicator` | 0.15 | TOIN field_semantics error detection | -| `forward_reference` | 0.15 | Count of later messages referencing this one | -| `token_density` | 0.05 | Unique tokens / total tokens | - -Weights are automatically normalized to sum to 1.0: - -```python -weights = ScoringWeights(recency=1.0, toin_importance=1.0) -normalized = weights.normalized() -# recency=0.5, toin_importance=0.5, others=0.0 -``` +> The earlier `RollingWindowConfig`, `IntelligentContextConfig`, and +> `ScoringWeights` configuration classes (and the position-/score-based +> context managers they configured) have been removed and are no longer part +> of Headroom. ## Environment Variables diff --git a/wiki/index.md b/wiki/index.md index b83f0d782..b63f16911 100644 --- a/wiki/index.md +++ b/wiki/index.md @@ -236,24 +236,23 @@ headroom wrap openclaw ## How It Works -Headroom runs a three-stage pipeline on every request: +Headroom runs a two-stage pipeline on every request: ```mermaid graph LR A[Your Prompt] --> B[CacheAligner] B --> C[ContentRouter] - C --> D[IntelligentContext] - D --> E[LLM Provider] + C --> E[LLM Provider] C -->|JSON| F[SmartCrusher] C -->|Code| G[CodeCompressor] C -->|Text| H[Kompress] C -->|Logs| I[LogCompressor] - F --> D - G --> D - H --> D - I --> D + F --> E + G --> E + H --> E + I --> E ``` **Stage 1: CacheAligner** — Stabilizes message prefixes so the provider's KV cache actually hits. Claude offers a 90% read discount on cached prefixes; CacheAligner makes that work. @@ -270,7 +269,7 @@ graph LR | Git diffs | **DiffCompressor** | Preserves change hunks, drops unchanged context. | | HTML | **HTMLExtractor** | Strips markup, extracts readable content. | -**Stage 3: IntelligentContext** — If the conversation still exceeds the model's context limit, scores each message by importance (recency, references, density) and drops the lowest-value ones. +Context management is handled automatically inside the pipeline (live-zone-only compression): Headroom compresses only the newest content blocks (the latest user message and tool results) and never drops messages from history. The system prompt, tool definitions, and older turns — the provider cache hot zone — are left untouched so prompt caching keeps working. **Nothing is lost.** Compressed content goes into the CCR store (Compress-Cache-Retrieve). The LLM gets a `headroom_retrieve` tool and can fetch full originals when it needs more detail. diff --git a/wiki/strands.md b/wiki/strands.md index 171852a39..585340c35 100644 --- a/wiki/strands.md +++ b/wiki/strands.md @@ -160,7 +160,7 @@ HeadroomStrandsModel.stream() Provider API (Bedrock, etc.) ``` -The model wrapper uses Headroom's full pipeline (CacheAligner → ContentRouter → IntelligentContext). The hook provider uses SmartCrusher directly for fast JSON compression of individual tool results. +The model wrapper uses Headroom's full pipeline (CacheAligner → ContentRouter). The hook provider uses SmartCrusher directly for fast JSON compression of individual tool results. --- diff --git a/wiki/transforms.md b/wiki/transforms.md index 4c9d93493..30ea74f7f 100644 --- a/wiki/transforms.md +++ b/wiki/transforms.md @@ -110,218 +110,19 @@ config = CacheAlignerConfig( --- -## RollingWindow +## Context management -Context management within token limits. +Context management is handled automatically inside the pipeline +(live-zone-only compression). Headroom **never** drops messages from the +conversation history and does not do position-based or score-based context +management. It compresses only the newest content blocks (the latest user +message and the latest tool result / tool output), type-aware and reversible +via CCR. The cache hot zone — system prompt, tools, and older turns — is never +mutated, which preserves provider prompt caching. -### The Problem - -Long conversations exceed context limits. Naive truncation breaks tool calls: - -``` -[tool_call: search] # Kept -[tool_result: ...] # Dropped = orphaned call! -``` - -### The Solution - -RollingWindow drops complete tool units, preserving pairs: - -```python -from headroom import RollingWindow - -window = RollingWindow(config) -result = window.apply(messages, max_tokens=100000) - -# Guarantees: -# 1. Tool calls paired with results -# 2. System prompt preserved -# 3. Recent turns kept -# 4. Oldest tool outputs dropped first -``` - -### Configuration - -```python -from headroom import RollingWindowConfig - -config = RollingWindowConfig( - max_tokens=100000, # Target token limit - preserve_system=True, # Always keep system prompt - preserve_recent_turns=5, # Keep last 5 user/assistant turns - drop_oldest_first=True, # Remove oldest tool outputs -) -``` - -### Drop Priority - -1. **Oldest tool outputs** - First to go -2. **Old assistant messages** - Summary preserved -3. **Old user messages** - Only if necessary -4. **Never dropped**: System prompt, recent turns, active tool pairs - -> **Note:** For more intelligent context management based on semantic importance rather than just position, see [IntelligentContextManager](#intelligentcontextmanager) below. - ---- - -## IntelligentContextManager - -Semantic-aware context management with TOIN-learned importance scoring. - -### The Problem - -RollingWindow drops messages by position (oldest first), but position doesn't equal importance: - -- An error message from turn 3 might be critical -- A verbose success response from turn 10 might be expendable -- Messages referenced by later turns should be preserved - -### The Solution - -IntelligentContextManager uses multi-factor importance scoring: - -```python -from headroom.transforms import IntelligentContextManager, IntelligentContextConfig - -manager = IntelligentContextManager(config) -result = manager.apply(messages, tokenizer, model_limit=128000) - -# Guarantees: -# 1. System messages never dropped (configurable) -# 2. Last N turns always protected -# 3. Tool calls/responses dropped atomically -# 4. Drops by importance score, not just position -``` - -### How Scoring Works - -Messages are scored on multiple factors (all learned, no hardcodes): - -| Factor | Weight | Description | -|--------|--------|-------------| -| Recency | 20% | Exponential decay from conversation end | -| Semantic Similarity | 20% | Embedding similarity to recent context | -| TOIN Importance | 25% | Learned from retrieval patterns | -| Error Indicators | 15% | TOIN-learned error field detection | -| Forward References | 15% | Messages referenced by later messages | -| Token Density | 5% | Information density (unique/total tokens) | - -**Key principle:** No hardcoded patterns. Error detection uses TOIN's `field_semantics.inferred_type == "error_indicator"`, not keyword matching. - -### Configuration - -```python -from headroom.transforms import IntelligentContextManager -from headroom.config import IntelligentContextConfig, ScoringWeights - -# Custom scoring weights -weights = ScoringWeights( - recency=0.20, - semantic_similarity=0.20, - toin_importance=0.25, - error_indicator=0.15, - forward_reference=0.15, - token_density=0.05, -) - -config = IntelligentContextConfig( - enabled=True, - keep_system=True, # Never drop system messages - keep_last_turns=2, # Protect last N user turns - output_buffer_tokens=4000, # Reserve for model output - use_importance_scoring=True, # Enable semantic scoring - scoring_weights=weights, # Custom weights - toin_integration=True, # Use TOIN patterns - recency_decay_rate=0.1, # Exponential decay lambda - compress_threshold=0.1, # Try compression first if <10% over -) - -manager = IntelligentContextManager(config) -``` - -### Strategy Selection - -Based on how much over budget you are: - -| Overage | Strategy | Action | -|---------|----------|--------| -| Under budget | NONE | No action needed | -| < 10% over | COMPRESS_FIRST | Try deeper compression | -| >= 10% over | DROP_BY_SCORE | Drop lowest-scored messages | - -### TOIN + CCR Integration - -IntelligentContextManager is a **message-level compressor**. Just like SmartCrusher compresses items in a JSON array, IntelligentContext "compresses" messages in a conversation by dropping low-value ones. - -**Bidirectional TOIN integration:** - -1. **Scoring uses TOIN patterns**: Learned retrieval rates and field semantics inform importance scores -2. **Drops are recorded to TOIN**: When messages are dropped, TOIN learns the pattern -3. **CCR stores originals**: Dropped messages are stored in CCR for potential retrieval -4. **Retrievals feed back to TOIN**: If users retrieve dropped messages, TOIN learns to score those patterns higher - -```python -from headroom.telemetry import get_toin - -toin = get_toin() -manager = IntelligentContextManager(config, toin=toin) - -# TOIN provides (for scoring): -# - retrieval_rate: How often this message pattern is retrieved (high = important) -# - field_semantics: Learned field types (error_indicator, identifier, etc.) -# - commonly_retrieved_fields: Fields that users frequently need - -# TOIN receives (from drops): -# - Message pattern signatures (role counts, has_tools, has_errors) -# - Token counts (original vs marker size) -# - Retrieval feedback when users access CCR -``` - -**What this means:** -- When you drop a message pattern and users frequently retrieve it, TOIN learns to score it higher next time -- When you drop a pattern and no one retrieves it, that confirms it was safe to drop -- The feedback loop improves drop decisions across all users, not just in one session - -### Example: Before vs After - -**RollingWindow (position-based):** -``` -Messages: [sys, user1, asst1, user2, asst2_error, user3, asst3, user4, asst4] -Over budget by 3 messages. -Drops: user1, asst1, user2 (oldest first) -Result: Loses context, keeps verbose asst3 -``` - -**IntelligentContextManager (score-based):** -``` -Messages scored: - - asst2_error: 0.85 (TOIN learned error indicator) - - asst1: 0.45 (old, low density) - - asst3: 0.40 (verbose, low unique tokens) - -Drops: asst1, asst3, user1 (lowest scores) -Result: Preserves critical error message -``` - -### Backwards Compatibility - -Convert from RollingWindowConfig: - -```python -from headroom.config import IntelligentContextConfig, RollingWindowConfig - -rolling_config = RollingWindowConfig( - max_tokens=100000, - preserve_system=True, - preserve_recent_turns=3, -) - -# Convert to intelligent context config -intelligent_config = IntelligentContextConfig( - keep_system=rolling_config.preserve_system, - keep_last_turns=rolling_config.preserve_recent_turns, -) -``` +> The earlier position-based `RollingWindow` and score-based +> `IntelligentContextManager` transforms have been removed and are no longer +> part of Headroom. --- @@ -528,12 +329,11 @@ This enables the feedback loop where compression decisions improve based on actu Combine transforms for optimal results. ```python -from headroom import TransformPipeline, SmartCrusher, CacheAligner, RollingWindow +from headroom import TransformPipeline, SmartCrusher, CacheAligner pipeline = TransformPipeline([ SmartCrusher(), # First: compress tool outputs CacheAligner(), # Then: stabilize prefix - RollingWindow(), # Finally: fit in context ]) result = pipeline.transform(messages) @@ -551,13 +351,11 @@ The earlier hand-assembled `TransformPipeline([..., LLMLinguaCompressor(), ...]) | 1 | CacheAligner | Stabilize prefix for caching | | 2 | SmartCrusher | Compress JSON tool outputs | | 3 | Kompress (ML) | ML compression on remaining text (optional, `[ml]` extra) | -| 4 | RollingWindow | Enforce token limits (always last) | **Why this order?** - CacheAligner first to maximize prefix stability - SmartCrusher handles JSON arrays efficiently - Kompress compresses remaining long text -- RollingWindow truncates only if still over limit ---