Commit graph

4 commits

Author SHA1 Message Date
chopratejas
e190544c77 fix: B2 — live-zone block dispatcher skeleton
Phase B step 2 of the live-zone-only realignment. Replaces PR-A1's
unconditional "passthrough" stub with a real dispatcher that
inspects the Anthropic /v1/messages body, identifies the live zone
(latest user message at index >= frozen_message_count), and routes
each block to a per-type compressor. PR-B2 wires every per-type
compressor to a no-op, so the dispatcher returns
LiveZoneOutcome::NoChange on every call — bytes-in == bytes-out.
PR-B3+ replaces the no-ops with SmartCrusher, Log, Search, Diff,
and Code compressors.

Adds:
- crates/headroom-core/src/transforms/live_zone.rs — public API:
  - `compress_live_zone(body, frozen_message_count, AuthMode)`
  - `LiveZoneOutcome::{NoChange, Modified}`
  - `CompressionManifest` with per-block outcomes (message_index,
    block_index, block_type, BlockAction).
  - `BlockAction::{NoOpSkeleton, Excluded { reason }}`. The
    HOT_ZONE_BLOCK_TYPES list (`tool_use`, `thinking`,
    `redacted_thinking`, `compaction`) excludes blocks even when
    they appear in the latest user message.
  - `AuthMode::{Payg, OAuth, Subscription}` — accepted but unused
    in B2; PR-F2 wires the auth-mode gate.
  - 12 unit tests pin: empty messages, no messages field, invalid
    JSON, latest user message selection, frozen_count respect,
    hot-zone block exclusion, string-shaped content, no user msg
    in live zone, AuthMode no-op, NoChange contract, manifest
    counters, frozen-count clamping.

- crates/headroom-proxy/src/compression/live_zone_anthropic.rs —
  new entry point. `compress_anthropic_request` parses the body,
  resolves frozen_count via `resolve_frozen_count` (PR-A4 helper),
  dispatches via `compress_live_zone`, and returns
  `Outcome::NoCompression` on PR-B2 success / `Outcome::Passthrough
  { reason: NotJson | NoMessages | ModeOff }` on body-shape /
  policy issues. Six unit tests pin: mode_off short-circuit, no
  messages field, invalid JSON, valid body NoCompression,
  empty body, cache_control disabled.

Modifies:
- compression/mod.rs — re-exports `compress_anthropic_request` from
  `live_zone_anthropic` instead of `anthropic`. The old anthropic
  module is reduced to the `resolve_frozen_count` helper only
  (not deleted, because its CacheControlAutoFrozen-policy gate is
  reused).
- proxy.rs — passes `state.config.cache_control_auto_frozen` into
  the dispatcher. Drops the obsolete "live_zone reserved for
  Phase B" warning that PR-A1 emitted on every request.
- compression/anthropic.rs — pruned to the resolve_frozen_count
  helper plus its tests. The PR-A1 passthrough stub
  `compress_anthropic_request` is gone (live_zone_anthropic owns
  the name now).
- config.rs — `compression_mode` doc updated to reflect the wired
  dispatcher (no longer "reserved for Phase B").
- tests/integration_compression.rs — `compression_decision_logged`
  pins the new log contract (`decision="no_change"`,
  `reason="no_op_skeleton_pr_b2"`, plus manifest fields
  `frozen_message_count`, `messages_total`, `live_zone_blocks`).
  Asserts the obsolete Phase A warning is NOT emitted.
- proxy.rs no longer imports CompressionMode (only used inside the
  retired warning).

Benchmark cleanup (B1 leftovers that surfaced now):
- benchmarks/proxy_mode_benchmark.py + claude_session_mode_benchmark.py:
  drop `intelligent_context=False` arg from ProxyConfig (the field
  was retired in B1; tests/test_proxy_mode_benchmark.py and
  tests/test_claude_session_mode_benchmark.py imported these
  factories and started failing).
- benchmarks/bench_transforms.py: delete TestRollingWindowBenchmarks
  class; rewire TestTransformPipelineBenchmarks fixture without
  RollingWindow.
- benchmarks/conftest.py: drop rolling_window_config fixture.
- benchmarks/run_benchmarks.py: drop the `window` suite + table
  rows referencing RollingWindow.

Cache-safety invariant:
- PR-B2 dispatcher never mutates body bytes (no-op skeleton). The
  proxy forwards the original buffered bytes byte-equal. Phase A's
  SHA-256 fixtures pin this.
- `passthrough_mode_live_zone_currently_passthrough_byte_equal_sha256`
  retitled comment to reflect the dispatcher being live but
  no-op.

Acceptance:
- cargo build --workspace + clippy + fmt: green.
- cargo test --workspace --exclude headroom-py: all green
  (777 + 12 new live_zone + 6 new live_zone_anthropic tests).
- pytest: 4678 passed, 240 skipped, 0 failed.
- Anthropic decision log includes manifest fields per the
  observability contract documented in
  REALIGNMENT/02-architecture.md.

Per-PR-B2 plan: REALIGNMENT/04-phase-B-live-zone.md.
2026-05-02 12:45:43 -07:00
chopratejas
a2b2ec5463 Add OSS evaluation suite, universal JSON crush, latency benchmarks
Evaluation Suite:
- Tiered eval framework (Tier 1 ~$3/15min, Tier 2/3 for extended coverage)
- 16 benchmarks across 3 tiers: GSM8K, TruthfulQA, MMLU, ARC, HumanEval,
  SQuAD v2, BFCL, Tool Outputs, CCR needle retention, HotpotQA, and more
- Before/After runner with full proxy support (compression + CCR retrieval)
- LLM-as-judge for ground-truth comparison (BFCL function calling)
- Zero-cost compression-only runner (CCR needle retention, info retention)
- Cost tracker with per-model pricing and budget enforcement
- Report card generator (Markdown, JSON, HTML)
- Suite CLI: python -m headroom.evals suite --tier 1
- Fix BFCL dataset loader for current HuggingFace schema
- CI workflow: PR smoke test + weekly full Tier 1

Results: SQuAD 97%, BFCL 97%, Tool Outputs 100%, CCR 100%

SmartCrusher:
- Universal JSON crush for heterogeneous arrays
- Fix mypy redefinition warning in _crush_string_array

Other:
- Latency benchmark suite with docs
- Known limitations doc
- Prompt comparison evaluator
- Config updates for new features
2026-02-23 19:08:54 -08:00
chopratejas
e4a41faa33 Fix all ruff lint and format errors for CI
- Fix E402: Move module-level imports to top of file
- Fix F401: Add noqa for availability check imports
- Fix F402: Rename loop variables shadowing imports
- Fix E722: Replace bare except with except Exception
- Fix B904: Add exception chaining (from e)
- Fix F811: Remove duplicate imports
- Fix B027: Add noqa for empty close() method
- Fix E741: Rename ambiguous variable l -> label
- Fix I001: Import sorting issues
- Apply ruff format to all 106 files

All 902 tests pass.
2026-01-10 15:33:44 -08:00
chopratejas
9c7d4512d6 Initial commit: Headroom SDK - LLM context optimization toolkit
A comprehensive SDK for optimizing LLM context windows, reducing token
usage while preserving critical information for AI agents.

Core Features:
- SmartCrusher: Statistical compression of tool outputs (70-85% reduction)
- CacheAligner: Prefix optimization for prompt cache hits
- RollingWindow: Intelligent context window management
- BM25/Hybrid relevance scoring for smart item selection

Integrations:
- OpenAI and Anthropic provider support
- LangChain integration (ChatModel, Callbacks, Runnable)
- MCP (Model Context Protocol) integration for tool compression

Test Coverage:
- 372 tests passing across all modules
- 35 performance benchmarks
- Real-world agent evaluations with 88% token savings

Key Components:
- headroom/transforms/: Core compression transforms
- headroom/providers/: OpenAI and Anthropic support
- headroom/integrations/: LangChain and MCP integrations
- headroom/relevance/: BM25 and hybrid scoring
- headroom/pricing/: Model pricing registry
- benchmarks/: Performance benchmark suite
- examples/: Usage examples and demos
2026-01-06 23:16:58 -08:00