mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
01a423a316
Replaces PR1's lossless/lossy split with ReformatTransform (pack denser, no info lost) and OffloadTransform (drop bytes, CCR-stash original via required cache_key). With CCR every transform is information-preserving end-to-end, so the lossless/lossy distinction misnamed the architecture. OffloadTransform carries a cheap, structural estimate_bloat() method scoped to its domain — generic byte-redundancy heuristics miss domain semantics. The orchestrator runs reformat phase + per-offload bloat estimation in parallel via rayon::join + par_iter, then runs offload iff bloat clears threshold OR reformat underwhelmed. Transforms shipped: REFORMATS (lossless): - JsonMinifier: serde_json round-trip whitespace stripping. - LogTemplate: Drain-inspired order-preserving template miner. Collapses consecutive runs of same-template lines into [Template Tn: ...] (Nx) + variant table. Win comes from emitting the constant-token prefix once instead of N times. Lossless: every original line reconstructible from template + variants. OFFLOADS (drop bytes, stash original via CCR): - LogOffload: wraps existing LogCompressor; bloat = repetition x uniqueness_weight + dilution x priority_dilution_weight. - DiffOffload: wraps existing DiffCompressor; bloat = context-to- change ratio. Bug-fix-on-port — persists original under the cache_key the parity-bound DiffCompressor mints (closes a leak). - DiffNoise: drops lockfile hunks (Cargo.lock, package-lock.json, yarn.lock, etc., suffix list configurable in TOML) and whitespace-only hunks. Stashes original via CCR for retrieval. Search offload exists but is not in default re-exports — modern agents (Claude Code, Codex) use scoped rg/grep, the marginal value didn't justify default registration. Reach via the explicit module path if opting in. JSON Offload is intentionally absent from this PR — already lives at SmartCrusher; Phase 3g PR3 wraps it in the OffloadTransform contract. Thresholds and weights live in config/pipeline.toml, embedded via include_str!; PipelineConfig::from_toml_str loads runtime overrides. 98 new pipeline tests; full headroom-core suite (714) and workspace tests green; cargo fmt clean. No regex, per project convention. |
||
|---|---|---|
| .. | ||
| act | ||
| actions/headroom-e2e-setup | ||
| ISSUE_TEMPLATE | ||
| plugin | ||
| workflows | ||
| dependabot.yml | ||
| FUNDING.yml | ||
| PULL_REQUEST_TEMPLATE.md | ||