mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
3e348f327f
## Description CCR entries could end up holding a `<<ccr:...>>` marker — or nothing at all — where the original bytes belonged, so `headroom_retrieve(hash)` answered with the very placeholder the caller was trying to resolve. For a base64/credential field that is permanent, silent data loss: the inner marker's hash is the only handle on the real payload, and it disappears from anywhere the model can see. Four sites, one root cause — **a compressed intermediate (or nothing) was stored in place of the source**, the same defect class as #1209 (tag placeholders persisted as originals). Closes #2694 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **`compaction/walker.rs`** — `walk_array` compacted through the store-LESS `compact()`. Opaque cells inside a compacted table got a marker whose payload was **never written**, so retrieval 404'd forever. Now uses `compact_with_store` so the emitted hash resolves. - **`compaction/classifier.rs`** — nothing stopped an already-marked string from being offloaded a second time, which stashed the MARKER as the new entry's "original". Marker-bearing text is our own output, not source content, so it is never classified opaque. One guard at the choke point both the walker and the table compactor share. - **`smart_crusher/crusher.rs`** — on the prose-hook path the row-drop marker hashed and stored rows whose leaves were **already** rewritten (prose compressed, blobs marker-substituted), so retrieving dropped rows returned compressed output. Now hashes and stashes the pre-processing array via `crush_array_with_source`. - **`content_router.py`** — compression pinning matched only `Retrieve more: hash=` / `Retrieve original: hash=`, **not** `<<ccr:`, so opaque-blob output was readmitted to the compressor on a later turn — the path that feeds the corruption above. Consolidated into `_is_already_compressed()` and applied at all three pinning sites. - **`cache/compression_store.py`** — store-level guard: refuse to persist a *bare* marker as `original_content` and log at ERROR, so a future producer regression surfaces loudly instead of silently converting "retrievable" into "gone". Deliberately narrow — originals may legally *contain* markers (nested offloads); only a bare marker is rejected. - **Regression tests** — `test_nested_table_markers_resolve_to_source_bytes` (asserts payloads are verbatim-retrievable, not merely that a marker was emitted) and `test_already_marked_content_is_not_re_offloaded`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ cargo build -p headroom-core Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 37s $ cargo test -p headroom-core --lib smart_crusher test result: ok. 329 passed; 0 failed; 0 ignored; 0 measured; 583 filtered out; finished in 0.19s $ python -m pytest tests/test_transforms/test_smart_crusher_ccr_roundtrip.py -q 16 passed in 0.68s $ python -m pytest tests/test_ccr_row_drop_store_bridge.py tests/test_ccr_tool_injection.py -q 50 passed in 12.52s $ python -m pytest tests/test_compression_store.py tests/test_lossless_mode.py -q 100 passed in 13.14s $ ruff check headroom/transforms/content_router.py headroom/cache/compression_store.py \ tests/test_transforms/test_smart_crusher_ccr_roundtrip.py All checks passed! $ mypy headroom/transforms/content_router.py headroom/cache/compression_store.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - **Environment:** macOS (Darwin 25.4.0), Python 3.12.6, headroom-ai 0.33.0 editable, `HEADROOM_CCR_BACKEND=memory`, Rust extension rebuilt via `maturin develop --release`. - **Exact command / steps:** compact a nested document — 5 rows whose `detail` field is a stringified sub-array of 6 base64 blobs (1600 B each) — then, for every `<<ccr:HASH>>` marker in the output, call `ccr_get(HASH)` and check the payload is the verbatim source rather than a marker. ```python inner = [{"k": f"key{i}", "v": i, "tok": blob(1200)} for i in range(6)] doc = {"rows": [{"id": i, "detail": json.dumps(inner), "note": "x"} for i in range(5)]} out = SmartCrusher().compact_document_json(json.dumps(doc)) for h in re.findall(r"<<ccr:([0-9a-f]+)", out): payload = crusher.ccr_get(h) # must be real bytes, not a marker ``` - **Observed result — BEFORE (on `main`):** all six payloads collapsed into a single dead marker. The rendered sub-table was re-classified opaque (`html`, because `<<` reads as a tag), offloaded again, and its payload never stored — so the six inner hashes were erased from the visible text *and* the outer hash resolved to nothing. ```text {"rows":"[5]{detail:string,id:int,note:string} <<ccr:3fb1d44933da,html,289B>>,0,x <<ccr:3fb1d44933da,html,289B>>,1,x ..."} 3fb1d44933da -> RUST MISS # unrecoverable — 6 × 1600 B gone ``` - **Observed result — AFTER (this branch):** the sub-table stays inline, each blob keeps its own marker, and every marker resolves to verbatim source. ```text {"rows":"[5]{detail:string,id:int,note:string} \"[6]{k:string,tok:string,v:int} key0,\"\"<<ccr:955b1fed2ef7,base64,1.6KB>>\"\",0 ..."} 6ad5846997f4: resolves, len=1600, is-verbatim-source=True 78a0bd9364a7: resolves, len=1600, is-verbatim-source=True 955b1fed2ef7: resolves, len=1600, is-verbatim-source=True a0cef69da7f0: resolves, len=1600, is-verbatim-source=True dfcde5e940c0: resolves, len=1600, is-verbatim-source=True e57c4e0a3ce8: resolves, len=1600, is-verbatim-source=True RESULT: PASS — every marker resolves to real source bytes ``` ## Notes for reviewers - The issue also reports **function words dropped from retained prose** (`is`, `a`, `the`) and **interleaved log output corrupting `headroom doctor`'s table borders**. Those are separate defects on different paths (extractive prose compression and log-handler buffering respectively) and are **not** addressed here — this PR is scoped to the CCR store/retrieve corruption. They should be tracked separately; the prose one overlaps #2586. - The `crusher.rs` prose-hook fix is on the Rust pipeline (`json_offload`) rather than the Python proxy path, but it is the same store-the-intermediate bug and was cheap to close while in the file. |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| test_code_compressor.py | ||
| test_code_compressor_cjk.py | ||
| test_content_router.py | ||
| test_detect_fallback_1123.py | ||
| test_diff_compressor.py | ||
| test_diff_compressor_rust_parity.py | ||
| test_html_extractor.py | ||
| test_kompress_compressor.py | ||
| test_kompress_deadline.py | ||
| test_kompress_remote.py | ||
| test_kompress_size_gate.py | ||
| test_ort_dylib.py | ||
| test_pipeline_waste_signal_limit.py | ||
| test_read_lifecycle.py | ||
| test_smart_crusher_attribution.py | ||
| test_smart_crusher_audit_safe.py | ||
| test_smart_crusher_bugs.py | ||
| test_smart_crusher_ccr_retrieve_exemption.py | ||
| test_smart_crusher_ccr_roundtrip.py | ||
| test_smart_crusher_lossless_default.py | ||
| test_smart_crusher_rust_parity.py | ||
| test_tag_protector.py | ||
| test_text_crusher.py | ||
| test_text_crusher_cjk_eval.py | ||
| test_text_crusher_parity.py | ||
| test_text_crusher_routing.py | ||
| test_tree_sitter_thread_safety.py | ||