Stage 3c.2 PR4. Restores Python's CCR-Dropped semantics on the lossy
path (the cornerstone reversibility guarantee that the port had
silently dropped) and flips the OSS default to lossless-first with a
configurable savings threshold.
# The user-visible behavior
Default `SmartCrusher::new()` now runs:
1. Try lossless compaction.
2. If savings >= `lossless_min_savings_ratio` (default 0.30), ship
it — `compacted` populated, `ccr_hash = None`, nothing dropped.
3. Otherwise fall through to the lossy path — drop rows AND
populate `ccr_hash` so the runtime can cache the full original
for tool-call retrieval.
**No data is ever lost.** "Lossy" means "compressed view inline; full
payload retrievable via CCR cache" — same semantics as Python's
SmartCrusher with CCR enabled. The runtime (PyO3 bridge / proxy
server) owns the cache; this crate computes the hash and emits a
marker so the prompt knows where to look.
# What changed
- `SmartCrusherConfig.lossless_min_savings_ratio: f64` (default 0.30).
Single configurable knob — Enterprise overrides as needed. Below
the threshold, lossless declines and lossy + CCR runs.
- `SmartCrusher::new(cfg)` flips to include the compaction stage by
default. `SmartCrusher::without_compaction(cfg)` is the explicit
opt-out for callers / fixtures that depend on pre-PR4 behavior.
- `crush_array` rewritten:
- Lossless-first dispatch with savings-ratio gate
- Lossy path now hashes the full original (12-char SHA-256 prefix)
and emits a CCR-Dropped marker in `dropped_summary` whenever
rows are dropped
- `ccr_hash` field populated whenever rows were dropped
- `process_value` substitutes the compacted string into the JSON
tree when lossless wins, so `crush()` output reflects the win
- PyO3 bridge: `SmartCrusher.without_compaction()` static method;
`SmartCrusherConfig` exposes the new `lossless_min_savings_ratio`
field; Python `SmartCrusher` wrapper accepts `with_compaction=True`
(default) and routes to the right Rust constructor.
- Parity harness: legacy 17 fixtures use `without_compaction()` so
byte-equal coverage of the lossy path is preserved.
# Tests
- Rust: 281/281 smart_crusher unit tests pass (was 277). Six new
tests cover: lossless wins above threshold, lossy falls through
below threshold, CCR hash deterministic + input-dependent, lossy
without compaction emits CCR, passthrough paths don't emit CCR,
without_compaction yields no compacted field.
- Python parity: 21/21 (legacy fixtures via without_compaction).
- Python lossless default smoke: 3/3 new tests in
test_smart_crusher_lossless_default.py.
- Python retention: 21/21 (updated to opt into the lossy path
explicitly since their semantics target row-level retention).
- make ci-precheck green.
Modules:
crates/headroom-core/src/transforms/smart_crusher/{config,crusher}.rs
crates/headroom-parity/src/lib.rs
crates/headroom-py/src/lib.rs
headroom/transforms/smart_crusher.py
tests/test_quality_retention.py
tests/test_transforms/test_smart_crusher_{lossless_default,rust_parity}.py
