mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
da7716a95a
Closes four gaps in the Rust SmartCrusher pipeline that, together,
wire CCR storage end-to-end so the LLM can actually retrieve dropped
data:
1. CCR-Dropped marker is now injected into process_value's lossy-path
output as a sentinel object {"_ccr_dropped": "<<ccr:HASH N_rows_offloaded>>"}
appended to the kept-items array. Previously the store held the
original but no pointer reached the prompt -- the retrieval contract
was data-on-server, no-way-to-ask. Sentinel-as-object preserves the
array-of-dicts shape so downstream iteration with x.get(...) keeps
working.
2. Walker / process_value drift removed. process_value gains a
Value::String arm that handles stringified-JSON containers (parse,
recurse, re-encode) and opaque blobs (CCR marker + store) -- same
semantics walker.rs has always had, now reachable from the main
crush() pipeline.
3. Opaque-string CCR now stores originals. DocumentCompactor gains an
Option<Arc<dyn CcrStore>> field; emit_opaque_ccr_marker calls
store.put when one is configured. Same hash regardless of store
presence -- runtime contract is stable across configurations.
Same wiring is shared between walker.rs and process_value via the
extracted helper.
5. PyO3 surface adds SmartCrusher.compact_document_json(doc_json) ->
compacted-json string. Routes through the crusher's existing CCR
store, so ccr_get resolves both row-drop and opaque-string hashes.
Tests:
- 5 new Rust integration tests in ccr_roundtrip.rs (marker visibility,
nested-array marker, opaque-string roundtrip, stringified-JSON
recursion, walker-with-store)
- 4 new Python tests covering the marker visible-to-LLM contract via
both the native PyO3 surface and the Python shim
- 5 legacy parity fixtures re-recorded (dict_array_*, duplicate_dicts_40)
-- their lossy outputs now carry the sentinel; Rust + Python both
match the new bytes (parity-run smart_crusher: 17/17)
|
||
|---|---|---|
| .. | ||
| dict_array_30_b0ba1d26fd03.json | ||
| dict_array_30_bias_high_dc3b36e60560.json | ||
| dict_array_30_bias_low_a850512901bd.json | ||
| dict_array_100_sequential_b91d1fff5ba3.json | ||
| duplicate_dicts_40_40a0670dec17.json | ||
| empty_array_41ecf20098c5.json | ||
| mixed_array_3f929019b62f.json | ||
| nested_3deep_with_array_bb5036c153aa.json | ||
| nested_object_with_array_b66d43299ea1.json | ||
| non_json_passthrough_4322abf677f3.json | ||
| nulls_and_bools_06d6fd6adcab.json | ||
| number_array_40_changepoint_8cd416c24d53.json | ||
| short_array_passthrough_aeb16224b61f.json | ||
| small_object_passthrough_801afa36a32c.json | ||
| string_array_25_6fe4a307de81.json | ||
| time_series_50_25cd28df5a50.json | ||
| unicode_dict_array_4820ed6c0c9a.json | ||