mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
c7295cad1d
## Description
SmartCrusher's `lossless:table` compaction path emits opaque-blob CCR
markers
(`<<ccr:HASH,KIND,SIZE>>`) but never wrote the original payload to the
CCR
store. As a result `GET /v1/retrieve/{hash}` and the `headroom_retrieve`
tool
return **404** for those hashes. The opaque-*string* path
(`walker::emit_opaque_ccr_marker`) already stores its payload; the table
compactor diverged simply because no store was threaded into it.
Closes #1083
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `compaction/compactor.rs`: add `compact_with_store(items, cfg, store)`
and a
private `compact_inner`; thread `Option<&Arc<dyn CcrStore>>` through
`build_homogeneous_table` → `build_row` → `cell_from_value` and the
recursive
bucket/nested calls. In the `Opaque` branch, `store.put(&hash, payload)`
under
the **same** `hash_opaque` value that becomes the marker hash (mirrors
`walker::emit_opaque_ccr_marker`). Public `compact` is unchanged — it
delegates
with `None`.
- `compaction/mod.rs`: add `CompactionStage::run_with_store`; `run` is
unchanged.
- `crusher.rs`: the lossless branch now calls
`stage.run_with_store(items, self.ccr_store.as_ref())` instead of
`stage.run(items)`.
- Two new unit tests in `compactor.rs` (see below).
The IR (and therefore the rendered marker text) is identical whether or
not a
store is supplied — the store only gains the write that should already
have
happened, so existing output stays byte-for-byte the same.
## Testing
<!-- Check what you actually ran, then paste the real command output
below. -->
- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
> Note: this change is in the Rust core (`crates/headroom-core`), so the
> Python-specific checks above are N/A. The Rust equivalents were run:
### Test Output
```text
$ cargo test -p headroom-core --lib compaction
test result: ok. 70 passed; 0 failed; 0 ignored; 0 measured; 766 filtered out; finished in 0.01s
$ cargo fmt -p headroom-core -- --check
# clean (exit 0)
```
New tests:
- `opaque_payload_is_stored_under_marker_hash` — after
`compact_with_store`, the
original blob is retrievable via `store.get(marker_hash)`, and the
stored key
equals `hash_opaque(payload)` (locks the key↔marker contract).
- `store_presence_does_not_change_the_ir` — `compact` and
`compact_with_store`
produce identical IR; only the store write is added.
(The full `cargo test -p headroom-core --lib` run has 18 pre-existing
failures,
all in `transforms::magika_detector` — they require the ONNX
runtime/model and
are unrelated to this change. All 70 compaction + crusher tests pass.)
## Real Behavior Proof
- Environment: Windows, Rust 1.95.0, `cargo test -p headroom-core` (no
live proxy).
- Exact command / steps: build a 2-item array with a long opaque-blob
field →
`compact_with_store(&items, &cfg, Some(&InMemoryCcrStore))` → read the
`OpaqueRef.ccr_hash` from the IR → `store.get(ccr_hash)`.
- Observed result: before the fix the store is empty (retrieval would
404);
after the fix `store.get(ccr_hash) == Some(original_payload)` and the
marker
hash is unchanged.
- Not tested: end-to-end through a running proxy / a real `GET
/v1/retrieve/{hash}`
HTTP round-trip. Verified at the unit level that the store now receives
the
payload under the exact marker hash, which is the write that was
missing.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
- Docs/CHANGELOG checklist items are N/A — this is an internal
correctness fix
with no user-facing API change.
- Scope is intentionally minimal: public `compact`/`run` signatures are
preserved (delegating with `None`), so all existing callers and the 68
in-crate compaction tests are unaffected. Only the lossless
`crush_array`
branch opts into the store-threading via `run_with_store`.
|
||
|---|---|---|
| .. | ||
| headroom-core | ||
| headroom-parity | ||
| headroom-proxy | ||
| headroom-py | ||