mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
7 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Focused Instability
|
f9285766dd
|
feat: attribute reread waste to over-compression via marker check (#901)
## Description Fixes #899. The `reread` signal (#853/#854) counts re-served tool results but cannot answer the question that motivated it: **did Headroom cause the re-read?** A re-read after an intact first serve is agent behavior; a re-read after Headroom markerized the first serve is over-compression cost. This PR splits the signal so the actionable part is visible. Request-local, no store lookups: the client resends full history each turn and the pipeline recompresses it deterministically, so the current request already holds the evidence. `TransformPipeline.apply` passes `current_messages` into `parse_messages(compressed_messages=...)`. For each counted reread group, if the transformed copy of the **first serve** carries a CCR retrieval marker and its original text is gone, the group's counted repeats go into `reread_compressed_tokens`. Lossless reshaping (no marker) is deliberately not attributed. Closes #899. ## Type of Change - [x] New feature (non-breaking change that adds functionality) ## Changes Made - `parser.py`: `parse_messages` gains an optional `compressed_messages` param; the content-hash reread loop accumulates per-group `counted_tokens` and attributes them to `reread_compressed_tokens` when the first serve's transformed copy carries a CCR marker (`CCR_RETRIEVAL_MARKER_RE`, kept local to avoid a transforms import cycle). - `transforms/pipeline.py`: pass `current_messages` (post-transform copy) into the existing waste-detection `parse_messages` call. - `config.py`: new `reread_compressed_tokens` WasteSignals field; `dashboard.html` + `reporting/generator.py` surface it. - Tests: `tests/test_reread_attribution.py` + WasteSignals contract update. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] New tests added for new functionality - [x] New and existing unit tests pass locally with my changes ### Test Output ```text $ pytest tests/test_reread_attribution.py tests/test_parser.py tests/test_gemini_function_response_waste.py tests/test_codex_responses_waste_signals.py -q 122 passed in 1.50s $ pytest tests/ -k "waste or pipeline or reporting or config or reread" -q 348 passed, 33 skipped, 6010 deselected # (1 unrelated env-dependent failure: test_proxy_gemini_native_integration::test_generation_config — 404, reproduces on main without these changes; needs a Gemini key locally) $ ruff check headroom/parser.py headroom/transforms/pipeline.py All checks passed! ``` ## Real Behavior Proof - Environment: local macOS, repo .venv, Python 3.11.9 - Exact command / steps: rebased onto current main to resolve conflicts with #909 (merged), then ran the reread + parser + waste suites above - Observed result: a reread whose first serve is markerized attributes to `reread_compressed_tokens`; an intact first serve and a lossless (no-marker) reshape do not. #909's re-issued-call detection (same call, different bytes) continues to count and dedup correctly alongside it — all 122 targeted tests pass. - Not tested: live proxy traffic; the one gemini-native route test above (environmental 404, not introduced here). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes **Rebased onto current main after #909 merged.** #909 added a re-issued-call reread pass *after* the original content-hash loop this PR modifies — the conflict was textual/adjacent, not a re-architecture. Resolution preserves #909's `counted_results` dedup contract and leaves its new pass unchanged; #901's attribution stays scoped to the content-hash groups it was reviewed against (attributing #909's call-key pass too would be a separate follow-up). The diff differs from the prior approval only by this reshape — worth a quick re-glance. |
||
|
Focused Instability
|
5f1d88ad27
|
feat: detect re-served tool results as over-compression waste signal (#854)
Closes #853 ## What Adds a `reread` waste signal: identical `tool_result` content appearing at more than one message position means the agent re-fetched something already in context — the dominant failure signature of over-compression (Manus context-engineering; JetBrains "Complexity Trap", arXiv:2508.21433). Per-request savings can't see this cost; this signal makes it visible. - `WasteSignals.reread_tokens` — new field, in `total()`, exported as `"reread"` in `to_dict()`. - `parse_messages()` groups `tool_result` blocks by their **existing** `content_hash` and counts every repeat beyond the first serve. No new hashing or tokenization; one O(blocks) dict pass. - `REREAD_MIN_TOKENS = 50` guard: short outputs ("ok", empty diffs) legitimately repeat and are skipped. Duplicates within a single message (same `source_index`) are not counted. - Works across all formats the parser already normalizes to `tool_result` blocks: OpenAI `role=tool`, Anthropic `tool_result`, Strands/Bedrock `toolResult` (#813/#815). - Flows through existing generic plumbing with zero handler changes: pipeline → `RequestOutcome.waste_signals` → Prometheus `headroom_waste_signal_tokens_total{signal="reread"}` → dashboard "Waste Detected" panel. Dashboard gains label/color entries for the new key. ## Tests 7 new tests in `tests/test_parser.py::TestRereadDetection` (red before, green after): OpenAI + Anthropic format detection, repeat-counting semantics (first serve free), single-occurrence, short-duplicate guard, same-message guard, `total()`/`to_dict()` participation. Updated 2 exact-shape assertions in `tests/test_config.py`. Local runs: `tests/test_parser.py` (72 passed), `tests/test_config.py` + outcome/reporting/observability/storage/proxy-hooks suites (190 passed), `tests/test_canonical_pipeline.py` + `tests/test_proxy_pipeline_lifecycle.py` (11 passed). `ruff check` + `ruff format --check` clean. ## Real behavior proof **Setup:** macOS (Darwin 25.5), Python 3.11.9, this branch, real proxy server (`python -m headroom.proxy.server --port 18970 --anthropic-api-url http://127.0.0.1:18971`) with a local mock Anthropic upstream returning a canned `/v1/messages` response (no real key needed). **Steps:** POSTed an Anthropic-format conversation to the live proxy: agent fetches a 14 KB JSON log array via `get_logs` tool, then fetches the identical content again under a different `tool_use_id` (the re-read). **Observed result** — `curl http://127.0.0.1:18970/metrics` after the request: ``` # HELP headroom_waste_signal_tokens_total Tokens attributed to detected waste signals # TYPE headroom_waste_signal_tokens_total counter headroom_waste_signal_tokens_total{signal="json_bloat"} 9858 headroom_waste_signal_tokens_total{signal="reread"} 4935 ``` `reread` = 4935 tokens, exactly the second serve of the ~4.9k-token tool result (json_bloat counts both occurrences ≈ 2×). `/stats` shows the same: `"waste_signals": {"json_bloat": 9858, "reread": 4935, ...}` — which is what the dashboard panel renders. Also verified the negative path live: a conversation whose tool results contain non-compressible plain code text produced no waste-signal entries (the pipeline only attributes waste when compression actually engaged, unchanged behavior). **Not tested:** Gemini `functionResponse` path (parser doesn't produce `tool_result` blocks for it — pre-existing gap tracked in #819); dashboard rendering only verified via the `/stats` payload the panel binds to, not a browser screenshot. ## Out of scope (per #853) Tool-call argument matching, compression-marker attribution, tokens-per-task metric, cache hit-rate panel. --------- Co-authored-by: Ash Rhodes <ashley.rhodes@king.com> |
||
|
chopratejas
|
967b0db439 |
fix: B1 — retire ICM, RollingWindow, scoring, relevance + dependents
Phase B step 1 of the live-zone-only realignment. Removes ~10K LOC of
"drop messages from history" machinery that became unreachable after
PR-A1 made `/v1/messages` a passthrough on the proxy. Live-zone-only
compression (PR-B2..B7) operates on content blocks within messages;
message-list mutation no longer happens in the pipeline.
Python deletes:
- headroom/transforms/intelligent_context.py (1077 LOC)
- headroom/transforms/rolling_window.py (395 LOC)
- headroom/transforms/progressive_summarizer.py (508 LOC)
- headroom/transforms/scoring.py (459 LOC)
- headroom/transforms/tool_crusher.py (338 LOC)
- 5 corresponding tests/test_transforms/* and tests/test_proxy_intelligent_context.py
Rust deletes:
- crates/headroom-core/src/context/* (manager, config, workspace,
candidate, ccr_drop, strategy/, mod) + safety.rs replaced
- crates/headroom-core/src/scoring/* (mod, score, scorer, traits, weights)
- MessageScorerComparator from crates/headroom-parity (PR #338/#343
becomes deletable; sunk cost stays sunk)
- 13 message_scorer fixtures + record_message_scorer.py
Rust adds (move + rewrite):
- crates/headroom-core/src/transforms/safety.rs — `tool_pair_indices`
preserves the OpenAI/Anthropic tool_use ↔ tool_result pairing rule
the live-zone dispatcher (PR-B2) needs. No IcmConfig dependency.
Surface refactors:
- HeadroomConfig: drop `tool_crusher`, `rolling_window`,
`intelligent_context` fields; hoist `output_buffer_tokens` to top
level (used by client.py).
- ProxyConfig: drop `intelligent_context*` fields.
- `headroom wrap` proxy server: retire IntelligentContextManager
and RollingWindow imports + branch; pipeline is CacheAligner →
ContentRouter (smart_routing) or CacheAligner → SmartCrusher
(legacy).
- CLI: drop `--no-intelligent-context`, `--no-intelligent-scoring`,
`--no-compress-first` flags.
- LangChain memory integration: rename `_apply_rolling_window` →
`_apply_compression`, drop RollingWindowConfig dep. Threshold is
now advisory — B6 will rework the contract.
- TransformPipeline.create_pipeline now takes only cache_aligner_config.
- headroom/__init__.py + headroom/transforms/__init__.py: strip
exports of deleted symbols.
Bug fixes uncovered by full pytest sweep:
- providers/copilot/wrap.py: `environ or os.environ` collapsed
empty-dict to falsy → callers passing `environ={}` accidentally
pulled from os.environ. Use `environ if environ is not None else
os.environ`.
Test correctness fixes:
- _DummyAnthropicHandler._retry_request gains **_kwargs to match
the real handler signature post-A8.
- test_ws_http_fallback extracts JSON from `content=` (post-A3
byte-faithful) rather than the obsolete `json=` kwarg.
- test_ccr_response_handler_extra fixture joins SSE events with
`\n\n` per spec (post-A8 byte-buffer parser requirement).
- test_proxy_responses_phase_preservation: capture via direct
handler attached to the named logger, so the assertion is
order-independent (proxy `_setup_file_logging` flips
`headroom.propagate=False` once any earlier test triggers it).
- conftest.py autouse fixture resets `headroom.propagate=True`
before each test as a defensive measure for the same pollution.
- test_wrap_copilot_translated_backend_still_requires_byok:
monkeypatch.delenv every provider key so the BYOK error
actually fires.
- test_native_installers: skip when system bash < 4.3 (macOS ships 3.2).
- TestGeminiEmbedContent / TestGeminiBatchEmbedContents:
pytest.mark.skip — proxy currently has no :embedContent route;
feature gap, not regression.
Acceptance:
- cargo build --workspace + cargo clippy + cargo fmt --check: green.
- cargo test --workspace --exclude headroom-py: 777 passed.
- pytest: 4892 passed, 240 skipped, 0 failed.
- git grep returns only intentional comments referencing the deletion.
Per-PR-B1 plan: REALIGNMENT/04-phase-B-live-zone.md.
|
||
|
Tejas Chopra
|
08d81f2e2c |
fix: dashboard metrics, TTFB tracking, eager LLMLingua loading, and multi-provider consistency
Dashboard was showing wildly incorrect metrics (99.5% savings, 3ms overhead) due to using Anthropic API's non-cached input_tokens instead of optimized_tokens, and dividing overhead by total request count instead of optimized-only count. Key fixes: - Use optimized_tokens (what we sent) for dashboard aggregation, not API's input_tokens which excludes cached portion - Track overhead_count separately from latency_count for correct averages - Add TTFB (time to first byte) measurement, replace full stream latency in UI - Eager-load LLMLingua model at proxy startup (eliminates 5.9s first-request delay) - Simplify CostTracker to token-based accounting with counterfactual cost display - Add two-tier compression cache to ContentRouter (skip set + result cache) - Fix compression pinning to detect both CCR and ReadLifecycle markers - Clamp tokens_saved to max(0, ...) across all provider paths - Add per-transform timing instrumentation to pipeline - Guard against over-aggressive code compression (<5% ratio) - Fix ReadLifecycle partial read supersede logic (_read_covers range check) - Disable CacheAligner and compress_superseded by default - Fix all pre-existing mypy errors (CompressionCache return types) - Fix test mocks to accept **kwargs for cache token parameters |
||
|
chopratejas
|
e4a41faa33 |
Fix all ruff lint and format errors for CI
- Fix E402: Move module-level imports to top of file - Fix F401: Add noqa for availability check imports - Fix F402: Rename loop variables shadowing imports - Fix E722: Replace bare except with except Exception - Fix B904: Add exception chaining (from e) - Fix F811: Remove duplicate imports - Fix B027: Add noqa for empty close() method - Fix E741: Rename ambiguous variable l -> label - Fix I001: Import sorting issues - Apply ruff format to all 106 files All 902 tests pass. |
||
|
chopratejas
|
c1feb60595 |
feat: Add CCR architecture, TOIN telemetry, and DevEx improvements
## Core Features ### Compress-Cache-Retrieve (CCR) Architecture - Implement reversible compression with automatic retrieval support - Add CompressionStore for caching original content with TTL-based eviction - Add CompressionFeedback for learning from retrieval patterns - Implement tool injection for LLM retrieval capability - Add MCP server support for CCR operations - Track retrieval rates to dynamically adjust compression aggressiveness ### Tool Output Intelligence Network (TOIN) - Implement cross-session pattern learning for tool compression - Add ToolSignature for structural hashing of tool outputs - Track compression success rates per strategy (top_n, sample, truncate, etc.) - Implement privacy-preserving telemetry with SHA256 hashing - Add persistent storage with JSON file backend - Support network-effect learning across tool types ### SmartCrusher Enhancements - Add crushability analysis with variance/uniqueness detection - Implement statistical anomaly detection for outlier preservation - Add relevance-based item prioritization using BM25 scoring - Support multiple compression strategies with quality retention - Add change point detection for time-series data - Implement constant factoring for homogeneous datasets ## Developer Experience Improvements ### Exception Hierarchy - Add HeadroomError base class for all custom exceptions - Add specific exceptions: ConfigurationError, ProviderError, StorageError, CompressionError, TokenizationError, CacheError, ValidationError, TransformError ### Client Enhancements - Add validate_setup() for configuration verification - Add get_stats() for in-memory session metrics without DB query - Track session statistics (requests, tokens saved, cache hits) ### Logging Infrastructure - Add structured logging to TransformPipeline with token savings - Add logging to RollingWindow for dropped message tracking - Add logging to ToolCrusher for compression events - Add logging to CacheAligner for cache hit/miss detection - Add logging to SmartCrusher for strategy selection ## Bug Fixes (from deep analysis) ### Critical Fixes - Fix eviction heap memory leak with stale entry tracking - Fix hash collision detection in compression store - Fix strategy truncation desync in TOIN - Fix non-deterministic set truncation with sorted iteration - Fix race conditions in lazy initialization with proper locking - Fix user count double-counting in TOIN metrics ### High Priority Fixes - Fix unbounded strategy_success_rates growth with LRU eviction - Fix mutable pattern references with defensive copying - Fix lock held during file I/O with copy-then-write pattern - Fix state divergence on eviction with success event recording - Fix TOIN skip check order for CPU efficiency - Fix preserve_fields type mismatch (set vs list) - Fix prioritize_indices exceeding max_items limit - Fix instance ID collision risk (32-bit to 64-bit hash) ## Testing - Add comprehensive test suites for CCR, TOIN, and telemetry - Add crushability detection tests - Add quality retention tests for compression - Add integration tests for cross-component data flow - All 902 tests passing |
||
|
chopratejas
|
9c7d4512d6 |
Initial commit: Headroom SDK - LLM context optimization toolkit
A comprehensive SDK for optimizing LLM context windows, reducing token usage while preserving critical information for AI agents. Core Features: - SmartCrusher: Statistical compression of tool outputs (70-85% reduction) - CacheAligner: Prefix optimization for prompt cache hits - RollingWindow: Intelligent context window management - BM25/Hybrid relevance scoring for smart item selection Integrations: - OpenAI and Anthropic provider support - LangChain integration (ChatModel, Callbacks, Runnable) - MCP (Model Context Protocol) integration for tool compression Test Coverage: - 372 tests passing across all modules - 35 performance benchmarks - Real-world agent evaluations with 88% token savings Key Components: - headroom/transforms/: Core compression transforms - headroom/providers/: OpenAI and Anthropic support - headroom/integrations/: LangChain and MCP integrations - headroom/relevance/: BM25 and hybrid scoring - headroom/pricing/: Model pricing registry - benchmarks/: Performance benchmark suite - examples/: Usage examples and demos |