Three orthogonal hot-path fixes targeting concurrent-request throughput.
Each is independently bench-measured below; the proxy hot path benefits
from all three at once.
== 1. PyO3 GIL release on heavy compute ==
PyO3 methods (crush, smart_crush_content, crush_array_json,
compact_document_json, compress, compress_with_stats) used to hold the
GIL across the entire Rust call. Result: a 100ms compress() blocked
EVERY other Python thread for 100ms — multi-worker uvicorn deployments
serialized through SmartCrusher.
Wrap each compute call in `py.allow_threads(|| ...)`. Inputs (`&str`
from Python) are copied to owned `String` first because PyO3 ties them
to the GIL hold. PyDict construction stays on the GIL side.
Measured: 4 Python threads each running 20 crushes:
before (GIL held): ~3.3s wall (serialized — equivalent to 4×0.83s)
after (allow_threads): 826ms wall (4.01x speedup, perfect parallel)
== 2. CcrStore: Mutex<HashMap> -> DashMap-backed sharded ==
Single Mutex was the dominant bottleneck under multi-worker load — every
put/get serialized through one lock. Replace with DashMap (sharded
concurrent map, lock-free reads within a shard) plus a separate
small Mutex<VecDeque> for FIFO insertion-order eviction. Reads of
distinct keys never contend; writes only contend during the brief
order-queue push or capacity-sweep.
A/B bench (200 mixed put/get ops × N threads, in benches/ccr_store.rs):
Threads | DashMap Legacy Mutex Speedup
-------------------------------------------
1 | 63 µs 71 µs 1.13x
2 | 98 µs 194 µs 2.0x
4 | 178 µs 707 µs 4.0x
8 | 342 µs 1267 µs 3.7x
Legacy degrades ~linearly with thread count; DashMap stays near-flat
per-thread. Real multi-worker scaling.
== 3. Single-serialize the lossy CCR payload ==
The lossy `crush_array` path used to serialize the full array TWICE:
once in `hash_array_for_ccr` (allocates `Value::Array(items.to_vec())`,
deep-clones every Value subtree, then serializes), and a second time
in the store-write site. For a 50-item dict array that's ~MB of
allocator pressure per crushed array.
Introduce `canonical_array_json` (serializes `&[Value]` directly — same
bytes as `Value::Array(items.to_vec())` but no wrapper allocation +
no tree clone), call it ONCE per lossy path, then both hash and store
from those same bytes. Hash-format stable — all 17 parity fixtures
match byte-for-byte.
== Tests ==
- 8 ccr.rs unit tests including a new concurrent-stress test (8 threads
× 200 puts/gets, every key readable afterwards)
- 14 ccr_roundtrip integration tests stay green
- parity-run smart_crusher: 17/17 fixtures match
- 479 lib + 14 integration + 185 Python tests all pass
- New benches/ccr_store.rs runs the A/B and is committed for regression
visibility
== Dependencies added ==
- dashmap v6 (mature, widely-used in tokio/linkerd ecosystem)
The PR8 marker injection appends a sentinel object
{"_ccr_dropped": "<<ccr:HASH N_rows_offloaded>>"} to the kept-items
array on the lossy path so the LLM sees the retrieval pointer in the
prompt. Tests that iterate compressed arrays via subscript access
(e["level"], r["status"], i["labels"], m["text"]) hit KeyError on
the sentinel because it doesn't share the record schema.
Same root cause as the test_quality_retention fixes in PR8 -- these
integration tests were left out of that pass.
Ship a public helper headroom.transforms.smart_crusher.strip_ccr_sentinels
so tests can use it cleanly: `for e in strip_ccr_sentinels(entries):`
and production callers iterating compressed output get a single
canonical filter instead of inlining the _ccr_dropped check.
The 7 previously-failing tests in PR #292 CI now pass:
- langchain test_100_percent_errors_preserved_logs
- langchain test_errors_preserved_with_many_errors
- langchain test_search_results_with_query_term
- mcp test_all_log_errors_preserved
- mcp test_slack_significant_compression_with_content
- mcp test_database_error_status_preserved
- mcp test_github_bugs_partial_preservation
753 tests across the integration + transforms + retention suites pass
locally. Plugin manifests auto-bumped 0.13.3 -> 0.13.4 by the
sync-plugin-versions hook (unrelated to this fix).
Closes four gaps in the Rust SmartCrusher pipeline that, together,
wire CCR storage end-to-end so the LLM can actually retrieve dropped
data:
1. CCR-Dropped marker is now injected into process_value's lossy-path
output as a sentinel object {"_ccr_dropped": "<<ccr:HASH N_rows_offloaded>>"}
appended to the kept-items array. Previously the store held the
original but no pointer reached the prompt -- the retrieval contract
was data-on-server, no-way-to-ask. Sentinel-as-object preserves the
array-of-dicts shape so downstream iteration with x.get(...) keeps
working.
2. Walker / process_value drift removed. process_value gains a
Value::String arm that handles stringified-JSON containers (parse,
recurse, re-encode) and opaque blobs (CCR marker + store) -- same
semantics walker.rs has always had, now reachable from the main
crush() pipeline.
3. Opaque-string CCR now stores originals. DocumentCompactor gains an
Option<Arc<dyn CcrStore>> field; emit_opaque_ccr_marker calls
store.put when one is configured. Same hash regardless of store
presence -- runtime contract is stable across configurations.
Same wiring is shared between walker.rs and process_value via the
extracted helper.
5. PyO3 surface adds SmartCrusher.compact_document_json(doc_json) ->
compacted-json string. Routes through the crusher's existing CCR
store, so ccr_get resolves both row-drop and opaque-string hashes.
Tests:
- 5 new Rust integration tests in ccr_roundtrip.rs (marker visibility,
nested-array marker, opaque-string roundtrip, stringified-JSON
recursion, walker-with-store)
- 4 new Python tests covering the marker visible-to-LLM contract via
both the native PyO3 surface and the Python shim
- 5 legacy parity fixtures re-recorded (dict_array_*, duplicate_dicts_40)
-- their lossy outputs now carry the sentinel; Rust + Python both
match the new bytes (parity-run smart_crusher: 17/17)
The action was set to @stable, which installs whatever the latest
stable is (1.95.0 right now). Then maturin invokes cargo, which reads
rust-toolchain.toml and re-resolves to "1.95.0 + clippy + rustfmt".
rustup treats stable and 1.95.0 as distinct toolchain identities and
refuses the second install with:
failed to install component 'clippy-preview-x86_64-unknown-linux-gnu',
detected conflict: 'bin/cargo-clippy'
This was intermittent across the matrix (only test (3.10) tripped on
the most recent run; others got lucky on cache state). Pinning the
action ref to 1.95.0 makes both sides ask for the exact same toolchain
identity, so the second install is a no-op and the conflict can't fire.
Bump procedure stays the same: when rust-toolchain.toml's channel
changes, update these refs in lock-step.
Plugin manifests auto-bumped 0.11.0 -> 0.13.2 by sync-plugin-versions
hook (unrelated to the workflow fix).
The action was set to @stable, which installs whatever the latest
stable is (1.95.0 right now). Then maturin invokes cargo, which reads
rust-toolchain.toml and re-resolves to "1.95.0 + clippy + rustfmt".
rustup treats stable and 1.95.0 as distinct toolchain identities and
refuses the second install with:
failed to install component 'clippy-preview-x86_64-unknown-linux-gnu',
detected conflict: 'bin/cargo-clippy'
This was intermittent across the matrix (only test (3.10) tripped on
the most recent run; others got lucky on cache state). Pinning the
action ref to 1.95.0 makes both sides ask for the exact same toolchain
identity, so the second install is a no-op and the conflict can't fire.
Bump procedure stays the same: when rust-toolchain.toml's channel
changes, update these refs in lock-step.
Plugin manifests auto-bumped 0.11.0 -> 0.13.2 by sync-plugin-versions
hook (unrelated to the workflow fix).
CcrStore trait + InMemoryCcrStore (1000 entries, 5-min TTL, FIFO
eviction, idempotent re-store) live at the crate root. SmartCrusher's
lossy crush_array path now actually stashes the full original [items]
canonical-JSON into the configured store keyed by the same ccr_hash it
embeds in the prompt marker -- closing the no-data-loss contract that
was previously hash-only.
PyO3 surface:
- crusher.crush_array_json(items_json) -> dict with ccr_hash + kept items
- crusher.ccr_get(hash) -> Optional[str] for retrieval
- crusher.ccr_len() -> int for telemetry
Python shim passes both through. Default constructors enable the store
(matches Python's CCR-enabled default); without_compaction() also gets
it because CCR is a contract, not an opt-in extra.
Tests proving compress -> store -> retrieve -> reconstruct:
- 7 unit tests in ccr.rs (put/get/eviction/expiry)
- 9 Rust integration tests (crates/headroom-core/tests/ccr_roundtrip.rs)
- 10 Python tests including 4 explicit before/after element-equality
assertions through both the native PyO3 surface and the Python shim
Plugin manifest versions auto-bumped by the sync-plugin-versions
pre-commit hook (unrelated to CCR but co-resident in the working tree).
Stage 3c.2 PR5. Closes the gap between the public `crush()` API and
the standalone `DocumentCompactor` walker. Augments
`SmartCrusher::process_value`'s String branch to mirror the walker's
two String cases:
1. Stringified-JSON containers: parse, recurse via process_value,
re-emit. The wrapping field stays a string but its contents are
processed end-to-end. Special-cases the lossless-compaction
path (when recursion returns Value::String) to avoid double-
JSON-encoding.
2. Opaque blobs (long base64 / HTML / long-text strings):
substitute with `<<ccr:HASH,KIND,SIZE>>` markers — same format
as walker.rs and PR4's lossy CCR-Dropped markers, so downstream
consumers can pattern-match regardless of which path emitted.
Type chore so the package version doesn't bump.
# Why this matters
After PR4, calling `SmartCrusher::new()` and then `crush(json_blob)`
gets lossless-first compaction on top-level arrays. But for tool
outputs that wrap a JSON-encoded payload INSIDE a string field, or
contain opaque blobs (base64-encoded files, HTML chunks), the public
API was a no-op. The walker handled these but only via its own
entry point; users calling `crush()` never hit it.
PR5 brings walker semantics into the public path. Same vision the
user described early in this stage:
> JSON within JSON, opaque payloads, multi-layered
now works for `crush()` callers without any extra setup.
# Implementation (~80 lines)
- `process_string` method on SmartCrusher: dispatches to JSON-recurse
/ CCR-substitute / passthrough.
- `try_parse_json_container_str`: cheap parse-only-containers helper.
- `ccr_marker_for_string` + `opaque_kind_label` + `humanize_bytes`:
marker formatting matching walker.rs byte-for-byte.
Reuses every PR2 primitive — no new traits, no new IR, no new
abstractions.
# Tests (6 new)
- Short string passthrough (no false positives)
- Stringified-JSON array recurses (50 items inside a string field)
- Opaque base64 blob → CCR marker substitution
- Top-level plain text passthrough (crush(plain_text) unchanged)
- Short JSON-looking strings unchanged (no false-positive opaque)
- Helper parses only containers, not bare scalars
303/303 smart_crusher tests pass (was 297 → +6 PR5).
185/185 Python tests. make ci-precheck green.
Module: crates/headroom-core/src/transforms/smart_crusher/crusher.rs
Two more tests broke after PR4's lossless-first default flip — same
root cause as the langchain/MCP fixes (#287's first patch):
- tests/test_proxy_ccr.py — TestEndToEndTOINIntegration asserts
CCR-cache state after compression. Lossless wins on the test
fixture and skips CCR entirely (nothing dropped). Pin to lossy
via with_compaction=False so the cache assertion holds.
- tests/test_text_compressors.py — TestSmartCrusherTextIntegration
asserts JSON-array shape round-trip. Lossless substitutes a
CSV+schema string. Pin to lossy + JSON shape via
with_compaction=False. Lossless coverage exists separately in
test_smart_crusher_lossless_default.py.
Same pattern, same fix. CI run that surfaced these:
actions/runs/25025876328
PR4 flipped the OSS default to lossless-first. The MCP server and
LangChain eval tests assert wire-format and row-level retention
properties that belong to the lossy path; the lossless path
substitutes a CSV+schema STRING in place of arrays, which is great
for LLM prompts but wire-incompatible with consumers that iterate
the JSON.
Pin both call sites to the lossy + CCR-Dropped path via
`with_compaction=False`. Same retention semantics as Python's
pre-PR4 SmartCrusher behavior — full payload still cached via CCR
for tool retrieval; nothing is lost.
Modules:
- headroom/integrations/mcp/server.py — runtime MCP wrapper
- tests/test_integrations/langchain/test_evals.py — eval fixture
CI run that surfaced these: actions/runs/25025161868
Stage 3c.2 PR3a-redux. The original PR3a (#286) was merged on GitHub
on 2026-04-27 but its content never reached main — its parent merge
(#285 PR2) was squash-merged, which drops commits stacked on top of
the source branch. The walker module disappeared along with the
intermediate commits.
This PR re-lands `walker.rs` and updates the `compaction/mod.rs`
re-exports — same content as commit 4db4f46 from the lost branch.
No new functionality, no behavior change for existing crusher paths.
# What the walker does (recap)
Recursive descent over any JSON value:
match value {
Object(m) => recurse into each field
Array(xs) => recurse into items, then try TabularCompactor on the array
String(s) => parse-as-JSON-and-recurse / CCR-substitute / leave
scalar => unchanged
}
Compactable spots become inline strings holding the rendered bytes.
The wrapping JSON structure is preserved.
# Why chore, not feat
This is a re-land of previously-merged-but-lost code. No new feature
shipped — fixing a regression caused by the stacked-PR squash-merge
accident. Using `chore` keeps the version from bumping for what is
effectively a parity restoration.
# Tests
- 13 walker unit tests pass.
- 462/462 headroom-core lib tests overall.
- 17/17 SmartCrusher parity fixtures byte-equal (untouched).
- 185/185 Python tests pass.
- make ci-precheck green.
Module: crates/headroom-core/src/transforms/smart_crusher/compaction/walker.rs
Stage 3c.2 PR4. Restores Python's CCR-Dropped semantics on the lossy
path (the cornerstone reversibility guarantee that the port had
silently dropped) and flips the OSS default to lossless-first with a
configurable savings threshold.
# The user-visible behavior
Default `SmartCrusher::new()` now runs:
1. Try lossless compaction.
2. If savings >= `lossless_min_savings_ratio` (default 0.30), ship
it — `compacted` populated, `ccr_hash = None`, nothing dropped.
3. Otherwise fall through to the lossy path — drop rows AND
populate `ccr_hash` so the runtime can cache the full original
for tool-call retrieval.
**No data is ever lost.** "Lossy" means "compressed view inline; full
payload retrievable via CCR cache" — same semantics as Python's
SmartCrusher with CCR enabled. The runtime (PyO3 bridge / proxy
server) owns the cache; this crate computes the hash and emits a
marker so the prompt knows where to look.
# What changed
- `SmartCrusherConfig.lossless_min_savings_ratio: f64` (default 0.30).
Single configurable knob — Enterprise overrides as needed. Below
the threshold, lossless declines and lossy + CCR runs.
- `SmartCrusher::new(cfg)` flips to include the compaction stage by
default. `SmartCrusher::without_compaction(cfg)` is the explicit
opt-out for callers / fixtures that depend on pre-PR4 behavior.
- `crush_array` rewritten:
- Lossless-first dispatch with savings-ratio gate
- Lossy path now hashes the full original (12-char SHA-256 prefix)
and emits a CCR-Dropped marker in `dropped_summary` whenever
rows are dropped
- `ccr_hash` field populated whenever rows were dropped
- `process_value` substitutes the compacted string into the JSON
tree when lossless wins, so `crush()` output reflects the win
- PyO3 bridge: `SmartCrusher.without_compaction()` static method;
`SmartCrusherConfig` exposes the new `lossless_min_savings_ratio`
field; Python `SmartCrusher` wrapper accepts `with_compaction=True`
(default) and routes to the right Rust constructor.
- Parity harness: legacy 17 fixtures use `without_compaction()` so
byte-equal coverage of the lossy path is preserved.
# Tests
- Rust: 281/281 smart_crusher unit tests pass (was 277). Six new
tests cover: lossless wins above threshold, lossy falls through
below threshold, CCR hash deterministic + input-dependent, lossy
without compaction emits CCR, passthrough paths don't emit CCR,
without_compaction yields no compacted field.
- Python parity: 21/21 (legacy fixtures via without_compaction).
- Python lossless default smoke: 3/3 new tests in
test_smart_crusher_lossless_default.py.
- Python retention: 21/21 (updated to opt into the lossy path
explicitly since their semantics target row-level retention).
- make ci-precheck green.
Modules:
crates/headroom-core/src/transforms/smart_crusher/{config,crusher}.rs
crates/headroom-parity/src/lib.rs
crates/headroom-py/src/lib.rs
headroom/transforms/smart_crusher.py
tests/test_quality_retention.py
tests/test_transforms/test_smart_crusher_{lossless_default,rust_parity}.py
Stage 3c.2 PR2. Adds an opt-in compaction stage that runs BEFORE the
existing lossy pipeline. When configured, it tries to losslessly
re-shape arrays of objects into a recursive Compaction IR and renders
that to bytes via a pluggable Formatter trait. When not configured
(default OSS), behavior is byte-equal with the pre-PR2 path — all 17
SmartCrusher parity fixtures stay green.
# What lands
- Recursive Compaction IR (`compaction/ir.rs`): Table / Buckets /
OpaqueRef / Untouched. CellValue can hold a nested Compaction so
multi-level cases (stringified-JSON inside cells, heterogeneous
arrays bucketed by discriminator, opaque blobs CCR-substituted)
share one tree shape.
- Cell classifier (`compaction/classifier.rs`): per-cell decision —
Scalar / JsonObject / JsonArray / StringifiedJson(parsed) /
Opaque(kind). Conservative: in doubt, return Scalar.
- TabularCompactor (`compaction/compactor.rs`): array → IR. Handles
uniform-nested flattening into dotted columns ("meta.region",
"meta.tier"), stringified-JSON parsing + recursion, opaque-blob
CCR-substitution (12-char SHA-256 prefix), and heterogeneous
bucketing by discriminator. Falls through to a sparse Table when
no clean discriminator exists, so we always do better than the
lossy path for object arrays.
- Formatter trait (`compaction/formatter.rs`) + two impls:
- JsonFormatter: structured JSON for debugging / programmatic use.
- CsvSchemaFormatter: [N]{col:type,col:type} declaration + CSV
rows. Steals TOON's row-count-and-shape declaration without
adopting TOON's bespoke escaping. CSV is the format LLMs are
strongest at — every model has seen millions of examples in
training. >30% smaller than raw JSON serialization on tabular
fixtures.
- Wiring (`crusher.rs`, `builder.rs`): SmartCrusher gains an optional
compaction stage. Builder methods with_compaction(stage) and
with_default_compaction() opt in. CrushArrayResult gets two new
fields (compacted, compaction_kind) populated only when the stage
runs. strategy_info becomes compaction kind when compaction won.
# Why this design
- Three-trait extension surface preserved. PR1 added Constraint /
Observer / Scorer; PR2 adds Formatter as the fourth pluggable
seam. Enterprise plug-ins land cleanly without forking core.
- Empty default builder rule held. SmartCrusherBuilder::new() still
produces a no-compaction crusher. with_default_compaction() is
the explicit OSS preset. No silent fallbacks.
- Recursive IR was the unlock. A flat table-of-scalars IR would have
collapsed the moment a cell held nested JSON. Making
CellValue::Nested hold another Compaction made stringified-JSON
parsing + heterogeneous bucketing + opaque substitution all share
one renderer pass.
- CCR substitution for opaque cells. Strings classified as
base64/HTML/long-opaque become structured markers keyed by 12-char
SHA-256 prefix. The full bytes round-trip via the CCR store (PyO3
bridge owns actual storage; this PR emits the marker and computes
the hash).
# Tests
- 60 new unit tests across IR / classifier / compactor / formatter /
wiring (448 total in headroom-core, was 388).
- 17/17 SmartCrusher parity fixtures byte-equal — default-config
path completely unchanged.
- 21/21 Python parity tests pass via PyO3 bridge.
- make ci-precheck green: ruff, mypy, cargo fmt/clippy/test
(1.95.0), commitlint.
# Deferred to follow-up PRs
- ToonFormatter (small; ship after eval harness compares formats)
- Diff/code detection in cells → routes to DiffCompressor /
CodeCompressor (coupled to ContentRouter Phase 4)
- Budget-aware row dropping (Constraint-respecting) when rendered
size exceeds budget
- Format A/B eval harness
- ContentRouter unification (Phase 4)
Modules: crates/headroom-core/src/transforms/smart_crusher/compaction/*, builder.rs, crusher.rs, mod.rs
Stage 3c.2 PR1 — the public extension surface that lets Enterprise
crates plug richer components into SmartCrusher without forking. Three
traits, one builder, behavior-equivalent on every parity fixture.
The three traits:
- Scorer (re-exported from `crate::relevance::RelevanceScorer`).
Already a trait; OSS HybridScorer (BM25 + fastembed). Enterprise
point: per-tenant Loop-trained scorer.
- Constraint (new in `traits.rs`). `must_keep(items, item_strings)
-> Vec<usize>` — indices the allocator must keep regardless of
saliency. OSS defaults: `KeepErrorsConstraint`,
`KeepStructuralOutliersConstraint` — thin wrappers around the
existing `detect_error_items_for_preservation` and
`detect_structural_outliers` functions. Enterprise point:
BusinessRuleConstraint, RegulatoryConstraint::HIPAA, and so on.
- Observer (new in `traits.rs`). `on_event(&CrushEvent)` fires once
per top-level `crush()` call with strategy + sizes + elapsed_ns.
OSS default: TracingObserver — writes to the `tracing` crate at
debug, zero-cost when filtered out. Enterprise point:
AuditObserver, MetricsObserver, LoopTrainingObserver.
The builder (`builder.rs`):
`SmartCrusherBuilder::new(config)` starts EMPTY (no scorer, no
constraints, no observers — explicit composition; "no silent
fallbacks" applied to the API surface). Methods stack:
with_scorer, add_constraint, add_default_oss_constraints (appends
KeepErrors + KeepStructuralOutliers), add_observer,
with_default_oss_setup (HybridScorer + default constraints +
TracingObserver in one call).
`SmartCrusher::new(config)` is preserved as the OSS default factory
(equivalent to `SmartCrusher::builder(config).with_default_oss_setup
.build()`). Every existing caller (proxy, content_router,
integrations, evals) continues to work unchanged.
Internal refactor:
`SmartCrusherPlanner` now holds `&[Box<dyn Constraint>]` and
iterates the configured constraints via a new
`apply_constraints(items, item_strings, keep)` method. Replaces four
hardcoded `detect_structural_outliers` +
`detect_error_items_for_preservation` call sites in the four plan
methods. With the OSS default constraint stack the must-keep set is
byte-identical to pre-PR1 — verified by all 17 parity fixtures.
`SmartCrusher` gained two fields: `constraints: Vec<Box<dyn
Constraint>>` and `observers: Vec<Box<dyn Observer>>`. New
`from_parts` constructor (#[doc(hidden)]) is the builder's exit
point.
What did NOT change in this PR:
- The internal planning algorithm (lossless tabular, saliency
scoring, structured markers — those are PR 2/3/4).
- The string/number/object/mixed-array crusher paths in
`crushers.rs` and the `prioritize_indices` helper in
`orchestration.rs` — they still call the detection functions
directly. Path B from the design doc: dict-array path is the
primary value plugin point; lifting the leaf compressors can come
later if customers ask.
Tests:
15 new tests across `traits.rs`, `constraints.rs`, `observer.rs`,
`builder.rs`. Coverage: each constraint trait method called and
pinned (errors flagged, structural outliers detected, item_strings
cache parity, empty-array safety); builder empty-build path,
default-OSS-stack append, add_constraint order preservation,
with_default_oss_setup yields expected counts, observer fires
end-to-end on a real crush; TracingObserver name stable, on_event
doesn't panic.
Verification:
- cargo test --workspace: 403 passed (was 388, +15 new), 0 failed.
- parity: 17/17 byte-equal for smart_crusher.
- make ci-precheck: green.
Stage 3c.2 PR sequence:
- PR 1 (this commit): three traits + builder.
- PR 2 (next): improvement A — TabularCompactor.
- PR 3: improvement B — saliency scoring + structured allocator.
- PR 4: improvement C — structured marker formatter.
- PR 5: ENT-A — `headroom-enterprise` scaffold.
The cosign signing step passed bake metadata via env var:
env:
BAKE_META: ${{ steps.bake.outputs.metadata }}
run: echo "$BAKE_META" | jq ...
For large bake targets (code-nonroot, runtime-code-nonroot) the
metadata JSON is large enough that combined argv+env at bash spawn
exceeds Linux ARG_MAX (~128 KiB on ubuntu-latest), so bash dies with
E2BIG before the script even runs.
Switch to writing metadata into a heredoc-backed temp file, then read
it via jq file input. Heredocs put the JSON in the script body itself,
which bash reads from a temp file (no ARG_MAX limit), bypassing the
env-size ceiling entirely.
Module: .github/workflows/docker.yml
Two-part fix for the CI break that followed the previous push.
Lint fix:
`crates/headroom-core/src/transforms/smart_crusher/analyzer.rs:779` used
`pairs.sort_by(|a, b| b.1.cmp(&a.1))`. Clippy 1.95 introduced
`unnecessary_sort_by` and prefers `sort_by_key(|b|
std::cmp::Reverse(b.1))` for the same desc-sort by a single field.
Replaced. Functionally identical (both stable sorts on the count field
descending; first-occurrence tie order preserved either way).
Drift fix:
The same lint passed locally on rust 1.92 / clippy 0.1.92 because the
dev box hadn't been updated since December. CI runners use
`dtolnay/rust-toolchain@stable` which always grabs the latest stable
(1.95 today), so any lint added between local's last update and CI's
runner image breaks the pre-push gate. Two ways to close this:
1. Pin a specific Rust version in `rust-toolchain.toml`.
2. Run `rustup update stable` before every push.
Option 1 is more robust — `rust-toolchain.toml` is auto-respected by
`cargo`, so any local check (or CI check using the same project) syncs
to the pinned version automatically. Pinned to `1.95.0`. Bump procedure
documented in the file: edit channel, `rustup update`, `make
ci-precheck`, fix any new lints, commit.
Verification:
- `rustup` synced the dev box to 1.95.0 on the next `cargo` invocation.
- `cargo clippy --workspace -- -D warnings` clean on 1.95.
- `make ci-precheck` green (388 rust + 185 python + commitlint).
Five gates broke on the 2026-04-27 push of the smart_crusher branch.
Each is fixed below; the second half adds a `make ci-precheck` target
(plus an installable git pre-push hook) so the same dance never happens
again.
Failures fixed:
1. cargo fmt — 22 files had formatting drift introduced over the
stage 3c.1 work. `cargo fmt --all` reformatted them; no semantic
changes. `cargo test --workspace` still green (388 + supporting).
2. wheels job (macOS x86_64) — `fastembed -> ort -> ort-sys` does not
publish prebuilt ONNX Runtime binaries for `x86_64-apple-darwin`.
Removed that target from `.github/workflows/rust.yml`'s wheels
matrix. Apple Silicon (`aarch64-apple-darwin`) covers macOS
distribution; Intel macOS users can build from source. The matrix
now has 2 targets: linux x86_64 + macOS aarch64.
3. test-extras (relevance.py) — `tests/test_relevance.py::TestSmartCrusherIntegration`
constructs a `SmartCrusher`, which hard-imports `headroom._core`
since the python implementation was retired in stage 3c.1b. The
test-extras job didn't build the rust extension. Added the same
`maturin build + symlink` block the main `test` job uses.
4. smoke-test (eval.yml) — same root cause:
`compression_only.evaluate_ccr_lossless` instantiates a SmartCrusher.
Same fix: build the rust extension before the smoke test runs.
5. commitlint — three rules tripped:
- `subject-case` rejects PascalCase identifiers in subjects, but
the project deliberately names classes (SmartCrusher, HfTokenizer,
ContentRouter, DiffCompressor) in commit subjects. Disabled.
- `footer-leading-blank` is a warning that the wagoid action turns
into a CI failure; lines like `Module: foo.rs` in our bodies
match the conventional footer pattern and trip it. Disabled.
- `type-enum` doesn't include `parity`, but the project ships
parity-test infrastructure as its own concern (separate from
`test:`); added `parity` to the allowed types.
Pre-push verification — the prevention half:
`make ci-precheck` runs all of the above CI gates locally:
- `ci-precheck-rust`: cargo fmt --check + clippy + test --workspace.
- `ci-precheck-python`: builds the rust extension via maturin, then
runs the smart_crusher-affected python test files (185 tests across
test_transforms/, test_relevance*, test_ccr, test_acceptance,
test_critical_fixes, test_quality_retention).
- `ci-precheck-commitlint`: `npx commitlint --from origin/main --to
HEAD` against the same config CI uses. Skipped silently if npx is
not on PATH (install Node 18+ to enable).
`make install-git-hooks` (or `scripts/install-git-hooks.sh`) installs
a git pre-push hook that runs `make ci-precheck` automatically.
Bypass with `--no-verify` only when truly needed.
When new CI gates land in `.github/workflows/`, mirror them into a
`make ci-precheck-*` target. The Makefile is the local mirror of the
CI configuration; keeping them in sync is a load-bearing invariant.
Verification: `make ci-precheck` runs green on this commit.
Stage 3c.1b step 2 + cleanup. The python `SmartCrusher` (3669 lines)
is replaced by a thin pyo3-backed shim (~290 lines) that delegates
every byte to `headroom._core.SmartCrusher` (built from
`crates/headroom-py`, landed in the previous commit). There is no
python implementation and no env-var fallback — the wheel is a hard
import.
Why now: parity was already proven across 17 fixtures + the python-
side bridge test (1+17 in `test_smart_crusher_rust_parity.py`).
Keeping a shadow python impl behind a flag is a permanent maintenance
cost with no operational benefit. Stage 3c.1b deletes ~3380 lines of
python parser/scorer/analyzer/orchestrator code; the rust crate has
its own coverage (388 unit tests + property tests in headroom-core).
Surface preserved (drop-in for every production caller):
- `headroom.transforms.smart_crusher.SmartCrusher` — same class name,
same `__init__(config, relevance_config, scorer, ccr_config)`
signature (the latter three are accepted for source-compat and
silently dropped — rust port keeps those subsystems disabled in
Stage 3c.1, they re-attach in Stage 3c.2).
- `SmartCrusherConfig` and `CrushResult` dataclasses kept as python
dataclasses (callers use `asdict()` / dataclass matching on them).
- `crush(content, query, bias)`, `_smart_crush_content(content, ...)`,
`apply(messages, tokenizer, **kwargs)`, and
`_extract_context_from_messages(messages)` all preserved.
- `smart_crush_tool_output(content, config, ccr_config)` thin wrapper.
The transform-protocol `apply()` orchestration stays python (message
walking, digest-marker insertion, token counting); only the per-
message compression call delegates to rust.
Removed:
- Python parser / planner / scorer / analyzer / classifier (~3380 lines).
- Internal helpers `_classify_array`, `_detect_sequential_pattern`,
`_detect_rare_status_values`, `_detect_items_by_learned_semantics`,
`_percentile_linear`, `_compute_k_split`, `_crush_number_array`,
`_process_value`, etc. — rust crate has parallel coverage.
- `SmartAnalyzer`, `ArrayType`, `CompressionStrategy`,
`extract_query_anchors` — internals; not used by any production
caller (only tests probed them).
Tests deleted (probed deleted internals — same precedent as Stage 3b):
- `tests/test_transforms/test_smart_crusher.py` (40 tests)
- `tests/test_transforms/test_universal_json_crush.py` (45)
- `tests/test_transforms/test_anchor_selector.py` (49)
- `tests/test_toin_field_learning.py` (21)
- `tests/test_crushability.py` (20)
Tests trimmed (removed methods/classes that probe deferred subsystems
— scorer injection, CCR marker injection, TOIN feedback recording —
all of which re-attach in Stage 3c.2):
- `tests/test_transforms/test_smart_crusher_bugs.py`:
TestNumberArraySchemaPreservation, TestStage3c1BugFixes.
- `tests/test_relevance.py`: 2 scorer-injection tests.
- `tests/test_ccr.py`: TestSmartCrusherCCRIntegration class +
test_custom_marker_template.
- `tests/test_toin_integration.py`: TestTOINIntegration +
TestStoreToTOINHash classes.
- `tests/test_critical_fixes.py`: TestSmartCrusherTOINIntegration +
test_full_feedback_loop.
- `tests/test_acceptance.py::TestQueryAnchorExtraction`: dropped the
`extract_query_anchors` probe; kept the end-to-end "Alice
preserved" assertion.
Bug fixes from Stage 3c.1 (#1 percentile linear interp, #2 zero-
padded sequential, #3 rare-status pareto, #4 k-split overshoot) are
pinned by the rust crate and the parity fixtures
(`tests/parity/fixtures/smart_crusher/`).
Tests:
- 517 passed in the smart_crusher-adjacent file set
(test_transforms/, test_relevance*, test_ccr, test_toin_integration,
test_quality_retention, test_acceptance, test_critical_fixes).
- 18 in `test_smart_crusher_rust_parity.py` (1 sanity + 17 fixtures).
- 388 rust unit tests still green.
One stale-error-message regex in `test_relevance_extra.py` updated
from "requires sentence-transformers" → "requires fastembed".
Stage 3c.1b step 1: expose `SmartCrusherConfig`, `CrushResult`, and
`SmartCrusher` to Python via `headroom._core`. The Python shim that
delegates to it (replacing the 3669-line Python implementation) lands
in the next commit; this commit just builds the bridge and a
fixture-replay test that pins it.
Surface:
- `headroom._core.SmartCrusherConfig(**fields)` — every field of the
Rust `SmartCrusherConfig` exposed as a kwarg with matching default.
- `headroom._core.CrushResult` — read-only mirror of the Rust struct
with `compressed`, `original`, `was_modified`, `strategy` getters.
- `headroom._core.SmartCrusher(config=None)` — constructor accepts
only `config`; the Python shim drops `relevance_config`, `scorer`,
and `ccr_config` since Stage 3c.1 keeps those subsystems disabled.
- `crush(content, query="", bias=1.0)` and `smart_crush_content(...)`
methods mirror the Python signatures.
Verification:
- All 17 recorded parity fixtures byte-equal between Python and the
PyO3 bridge (`tests/test_transforms/test_smart_crusher_rust_parity.py`,
18 tests pass — 1 fixture-count sanity + 17 fixtures).
- The Rust-side `cargo run -p headroom-parity --bin parity-run --
run --only smart_crusher` was already 17/17 green.
The two tests catch different regression classes:
- Rust-only test: catches drift in the Rust port's logic.
- Python bridge test: catches PyO3 input/output translation bugs.
Adds the SmartCrusher half of the Rust-vs-Python parity harness. Path A
from the Stage 3c.1 plan: record fixtures from Python (with real
fastembed embeddings + the post-bug-fix code), drive the Rust port over
the same inputs, and assert byte-equal output on every recorded
scenario.
What's in:
- `tests/parity/record_smart_crusher.py`: standalone recorder for
`SmartCrusher.crush(content, query, bias)`. The generic recorder
framework only captures one positional, so this script writes its
own JSON envelope `{input: {content, query, bias}, config, output}`.
17 scenarios cover the planning paths exercised by ContentRouter
in production: passthrough, smart_sample, top_n, time-series,
duplicates, unicode (`ensure_ascii=False`), nested-3-deep, empties,
bias above and below 1.0.
- `crates/headroom-parity/src/lib.rs`: `SmartCrusherComparator`
reconstructs `SmartCrusherConfig` from the fixture's config block,
runs Rust `SmartCrusher::crush()`, emits the same JSON shape Python
serialized.
- `crates/headroom-parity/examples/diff_fixture.rs`: diagnostic CLI
that prints expected-vs-actual for one fixture (used during the
iteration that found the serializer bug below).
Serializer fix — found by the harness:
SmartCrusher uses `safe_json_dumps` (compact `(",", ":")` separators
+ `ensure_ascii=False`) for the wire bytes. The Rust port was using
`python_json_dumps` (default Python: `(", ", ": ")` + `ensure_ascii=
True`), which is the right choice for hashing but wrong for the
output. Refactored `anchor_selector.rs` to take a small
`JsonFmt { sort_keys, compact, ensure_ascii }` config so the three
flavors share one writer, added `python_safe_json_dumps`, and
switched `_smart_crush_content` to call it. All three flavors now
have byte-exact tests.
Two unit tests in `crusher.rs` were pinning the old (wrong) format
and have been re-pinned to the compact form.
Cross-language status:
- All 17 empty-query fixtures: byte-equal.
- Embedding-driven (non-empty-query) fixtures deferred until the
~0.0002 numeric drift between Python `onnxruntime` and Rust `ort`
is resolved (or until we accept the drift via a tolerance — none of
the 17 fixtures exercises a borderline relevance_threshold call,
and downstream code only branches at the 0.3 threshold).
Tests: cargo test --workspace (388 + supporting) green.
Replace `sentence-transformers` (PyTorch-backed) with `fastembed` (ONNX-
backed) so Python and Rust call into the same library + same model for
relevance scoring. Both sides run BAAI/bge-small-en-v1.5 (33M params,
384 dims, ~30 MB int8-quantized ONNX) auto-downloaded from HF Hub.
Cross-language verification on ('authentication failed for user',
'login error'): Python=0.7505, Rust=0.7507, delta ~0.0002 — well below
the relevance_threshold (0.3) buffer SmartCrusher uses for keep/drop
decisions, so the two implementations agree on every observable
SmartCrusher output. (True byte-equal would require both to load the
identical ONNX weights file — Rust's `fastembed` crate and Python's
`fastembed` package can pick different upstream artifacts; deferred.)
Why fastembed:
- removes torch from the relevance/ path (Phase 6: drop torch from
Python).
- ~2-3x faster than sentence-transformers' all-MiniLM-L6-v2 for the
same input shape.
- bge-small-en-v1.5 outranks all-MiniLM-L6-v2 on MTEB by ~6 points.
- self-contained: no longer reads ML_MODEL_DEFAULTS.sentence_transformer
from utils config.
Replace the embedding scorer stub with a real fastembed-rs
implementation. Same library + same model as the Python side will
use after the next commit, giving byte-equal embeddings on identical
inputs.
Cargo.toml: fastembed = "5". Default features pull in `ort` (ONNX
Runtime) with auto-download of the runtime binary at build time
(~21s additional first-build); model weights (BAAI/bge-small-en-v1.5,
~30 MB int8-quantized ONNX) auto-download from HuggingFace Hub on
first use.
embedding.rs:
- EmbeddingScorer wraps Option<Mutex<TextEmbedding>>. Mutex required
because TextEmbedding::embed needs &mut self (single-threaded ONNX
session); concurrent callers serialize on the lock, fine for the
SmartCrusher hot path where inference dominates lock contention.
- EmbeddingScorer::try_new() — explicit construction with HF Hub
download. Returns Result; surface errors to callers.
- EmbeddingScorer::try_new_with_model(EmbeddingModel) — bring your
own model from fastembed's catalog.
- EmbeddingScorer::default() — STUB only (model=None,
is_available()=false). Mirrors Python's "sentence-transformers
not installed" branch byte-for-byte. To get a real scorer, call
try_new() and pass via HybridScorer::with_scorers().
Why default() is a stub: with auto-load Default, model availability
would depend on whether HF Hub cache has the file — non-deterministic
in tests. Explicit try_new() keeps Default cheap and predictable.
cosine_similarity:
- f32 vec inputs (fastembed returns Vec<Vec<f32>>).
- Clamped to [0, 1] (mirrors Python _cosine_similarity — only
positive similarity matters for relevance).
- Defensive: zero vectors / mismatched dims → 0.0.
score / score_batch:
- Empty input / unavailable model → empty score with explanatory
reason.
- Batch encodes items + context in one model call (Python parity:
amortizes model dispatch).
- Inference failures degrade gracefully with empty scores rather
than panicking.
Tests:
- 5 cosine-similarity unit tests (offline).
- 3 unavailable-scorer tests (model=None path).
- 3 model-backed integration tests gated on RUN_FASTEMBED_TESTS=1
(semantic-match-outranks-unrelated, batch-shape, model-loads).
- All 388 headroom-core tests pass without RUN_FASTEMBED_TESTS;
with it set, the gated 3 also pass.
Net: 388 unit tests, clippy clean. HybridScorer's BM25-fallback path
remains correct (default embedding scorer reports unavailable).
Stage 3c.1 next: switch Python's relevance/embedding.py to fastembed
PyPI package + record parity fixtures with real embeddings on both
sides.
Lockstep fixes for the four known bugs in headroom/transforms/smart_crusher.py
plus the field-iteration ordering parity fix. Both languages now agree
byte-for-byte on the affected code paths — prerequisite for parity
fixtures landing next.
Bug #1 — percentile off-by-one (Python line 2844 + Rust crushers.rs)
Replaces integer-division indexing with linear-interpolation
percentile (numpy "linear" method). New _percentile_linear helper
shared by both languages: index = q * (n - 1), interpolate between
floor and ceil.
Bug #2 — zero-padded string IDs misclassified as sequential
Track had_non_string_numeric flag; if every parseable value came
from a string (no actual int/float), return False (categorical, not
sequential). Pre-fix: int("001") loses zero-padding and fakes a
sequential pattern.
Bug #3 — rare-status detection cardinality cap
Cardinality cap raised from 10 to 50. Single-dominant check
replaced with Pareto top-K: smallest K such that top-K covers >=80%
of items. If K <= 5, items NOT in top-K are outliers. Catches
bimodal distributions like 60×INFO + 25×WARN + 15 distinct error
codes.
Bug #4 — k-split overshoot when k_total=1
Clamp after the floored fractions: k_first=min(k_first, k_total),
k_last=min(k_last, max(0, k_total - k_first)). No-op for the
common case k_total >= 2.
Field iteration ordering (Python line 1049)
`for key in all_keys` → `for key in sorted(all_keys)`. Set
iteration is non-deterministic across PYTHONHASHSEED; downstream
short-circuits in _select_strategy and _detect_pattern would pick
different fields between runs. Rust uses BTreeMap (sorted ASCII);
sorting Python locks both languages to the same iteration order.
Verification:
- 56 Python tests pass (51 existing + 5 new lockstep tests under
TestStage3c1BugFixes class).
- 382 Rust tests pass (rust bug #1 documentation test replaced
with two new "fixed behavior" tests).
- Clippy clean.
Status: all four bugs are now fixed in BOTH languages. Parity
fixtures can be recorded against post-fix Python and asserted
byte-equal against Rust. That's the next commit.
Lands the top-level SmartCrusher orchestrator. Owns config,
anchor_selector, scorer, and analyzer. Three Python entry points
ported:
- _execute_plan (line 3617) → SmartCrusher::execute_plan
- _crush_array (line 2400) → SmartCrusher::crush_array
- _crush_mixed_array (line 2914) → SmartCrusher::crush_mixed_array
Module: smart_crusher/crusher.rs (~430 lines).
execute_plan:
- Trivial sort-by-index, clone, return. Mirrors Python verbatim.
- Schema-preserving: each kept item is unchanged.
crush_array:
- Computes adaptive_k from compute_optimal_k(item_strings, bias).
- Tier-1 boundary: n <= adaptive_k → "none:adaptive_at_limit"
passthrough.
- analyzer.analyze_array → ArrayAnalysis.
- crushability gate: SKIP path returns "skip:<reason>" with original
items unchanged.
- planner.create_plan → CompressionPlan.
- execute_plan → final items.
- Returns CrushArrayResult { items, strategy_info, ccr_hash,
dropped_summary }.
crush_mixed_array:
- n <= 8 → "mixed:passthrough".
- Group items by JSON type (dict / str / number / list / null /
bool). First-occurrence order preserved across groups via
GroupBuckets helper (mirrors Python's dict insertion-order
iteration).
- Small groups (< min_items_to_analyze): keep all items.
- Dict group: recurse into self.crush_array; survivors matched
back to original indices via canonical-JSON serialization
(anchor_selector::python_json_dumps_sort_keys).
- Str group: dispatch to crush_string_array; survivors matched
via &str equality.
- Number group: inline first/last + outlier (>variance_threshold σ)
detection — mirrors Python's _crush_mixed_array number arm
(no summary prefix).
- list / bool / null / other: keep all.
Stubbed paths (matches Python's "subsystems disabled" behavior
byte-for-byte; parity fixtures will be recorded with these off):
- TOIN: never produces a recommendation. effective_max_items =
adaptive_k. No preserve_fields, no strategy/level override.
- Feedback: never produces hints.
- CCR: never caches; ccr_hash = None.
- Telemetry: no-op.
- _compress_text_within_items: passthrough (text compression has
its own port pipeline).
- summarize_dropped_items: empty string.
The TOIN/CCR/feedback/telemetry integration ports happen later
(Stage 3c.2 follow-ups). The current state is a complete like-for-
like port of SmartCrusher's CORE compression decisions.
Net: 12 new crusher tests, 375 total in headroom-core, clippy clean.
Pipeline coverage tested:
- execute_plan: empty / sorted / out-of-bounds index handling.
- crush_array: adaptive_at_limit, skip path, low-uniqueness
compression, error-item preservation.
- crush_mixed_array: passthrough at threshold, group-and-compress
dicts, lists/nulls keep-all.
Next commit: SmartCrusher::crush(content, query, bias) top-level
that parses JSON, classifies array type, dispatches to the right
crusher, serializes back. Then Python lockstep + parity fixtures
+ PyO3 bridge.
Direct port of three Python methods from smart_crusher.py:
- _deduplicate_indices_by_content (line 1721) → deduplicate_indices_by_content
- _fill_remaining_slots (line 1794) → fill_remaining_slots
- _prioritize_indices (line 1891) → prioritize_indices
Used by every _plan_* method (next commit) to clean up index sets
before they become the final keep_indices.
Key design:
- All three operate on BTreeSet<usize> so iteration is sorted and
deterministic.
- Item content hashes use anchor_selector::compute_item_hash, which
serializes via Python-compatible json.dumps(sort_keys=True) and
truncates md5 to 16 hex chars. Ensures Rust and Python collapse the
same items to the same hash.
- Non-dict items (rare in the dict-array context) hash via str()
fallback to mirror Python's behavior.
prioritize_indices pipeline:
1. Dedup pass (when config.dedup_identical_items, default true).
2. Fill pass — top up to effective_max with diverse uniques if
under-budget.
3. Already <= effective_max? Return.
4. Otherwise keep ALL critical items (error keywords + structural
outliers + numeric anomalies) — Python's "quality guarantee" path
that may exceed effective_max when criticals dominate.
5. Add first-3 / last-2 anchors if room.
6. Fill remaining with non-critical kept indices in ascending order.
TOIN field-semantics: Python's _detect_items_by_learned_semantics
path is stubbed (returns empty set). Mirrors Python's "no
field_semantics provided" branch exactly. When TOIN is ported the
parameter slots back in without changing call sites.
Numeric-anomaly detection: helper numeric_anomaly_indices walks
analysis.field_stats and flags items >variance_threshold σ from
the per-field mean — same formula the analyzer uses for crushability
signal counting.
Net: 12 new orchestration tests, 350 total in headroom-core, clippy
clean.
Next commit: planning layer (_create_plan + four _plan_* methods)
wires these helpers + the relevance scorer + AnchorSelector into
strategy-specific keep_indices builders.
Two more pieces of the relevance ladder, plus the create_scorer factory.
embedding.rs — STUB
- Mirrors Python's "sentence-transformers not installed" branch
byte-for-byte. is_available() returns false; score() and
score_batch() return RelevanceScore::empty.
- Real ONNX-backed implementation lands in a follow-up commit
(tracked as task #22). When it does, this module needs zero
changes at any call site — flipping is_available() to true is
enough.
- Public surface: EmbeddingScorer::default() / ::new(model_name).
Default model name pinned to sentence-transformers/all-MiniLM-L6-v2.
hybrid.rs — full port (with stub embedding under the hood)
- Adaptive alpha tuning per Python hybrid.py:115-151. Regex pattern
detection on the context: UUIDs (alpha >= 0.85), 2+ numeric IDs
(>=0.75), single numeric ID (>=0.65), hostname/email (>=0.6).
Clamped to [0.3, 0.9].
- Email regex pinned to Python's literal pattern including the
[A-Z|a-z] typo (`|` inside `[...]` is a literal pipe, not an
alternation). Pinned for byte-equal parity with Python source.
- Graceful BM25 fallback when embedding.is_available() == false:
- Items with any matched term get score >= 0.3.
- Items with 2+ matched terms get +0.2 (capped at 1.0).
- Reason prefixed with "Hybrid (BM25 only, boosted): ".
Mirrors Python score()/score_batch() boost rules exactly.
- When real ONNX scorer lands, the hybrid path automatically uses
alpha-weighted fusion: combined = alpha*BM25 + (1-alpha)*Emb.
- score_batch is amortized: BM25 batches once, embedding batches
once, alpha computed once.
mod.rs — factory
- create_scorer(tier) returns Box<dyn RelevanceScorer>. Mirrors
Python's create_scorer factory. "embedding" tier returns Err with
the same shape as Python's RuntimeError until the ONNX backend
flips on.
Net: 39 relevance tests, 338 total in headroom-core, clippy clean.
SmartCrusher's planning layer (next commit) can now wire in
HybridScorer and behave parity-equal with Python deployments that
don't have sentence-transformers installed. ONNX embeddings remain
on the runway.
Direct port of headroom/relevance/base.py (RelevanceScore + RelevanceScorer
trait) and headroom/relevance/bm25.py (BM25 keyword scorer).
base.rs:
- RelevanceScore: clamps score to [0,1] via new() (Python __post_init__).
Plus empty(reason) helper for "no match" cases.
- RelevanceScorer trait: required score(item, context). Default
score_batch falls back to per-item dispatch; concrete scorers override
for vectorized/amortized batch impls.
- default_batch_score: free function fall-back for tests.
- is_available(): true by default; ONNX scorer (next commit) overrides.
bm25.rs:
- BM25Scorer with k1=1.5, b=0.75, max_score=10.0 defaults — pinned to
Python.
- Tokenization via single regex with three alternatives (UUID first,
then 4+ digit numeric ID, then alphanumeric+underscore). Order
matters — UUIDs would otherwise be split into 8/4/4/4/12 hex pieces
by the alphanumeric arm. Caught by tokenize_uuid_as_single_token test.
- BM25 formula matches Python including the simplified single-doc IDF
(constant ln(2)) and length-normalization parameters.
- Long-token bonus: +0.3 when any matched term has length >= 8. Boosts
UUID/long-ID matches above the keyword baseline.
- Optimized score_batch pre-tokenizes the context once and computes
avg_doc_len across the batch (matches Python's amortization).
- Reasons match Python's surface: "BM25: no term matches" /
"BM25: matched 'X'" / "BM25: matched N terms (a, b, c...)" for
single calls, "BM25: N terms" for batch.
Iteration determinism: Python iterates query Counter in dict
insertion order. We sort keys alphabetically before scoring so
matched_terms ordering is deterministic across runs (the BM25 score
itself is order-independent — sum is commutative). Python's order is
non-deterministic across PYTHONHASHSEED, so this is parity-safe and
arguably better behavior.
Net: 20 BM25 + base tests, 319 total in headroom-core, clippy clean.
Next commit: ONNX-backed embedding scorer (sentence-transformers via
the ort crate + tokenizers crate).
Direct port of headroom/transforms/anchor_selector.py (770 lines) plus
AnchorConfig from headroom/config.py. Used by SmartCrusher's planning
layer to allocate "anchor slots" — positions kept purely for their
location in the array (front/middle/back) before relevance scoring
fills in the rest.
Public surface:
- AnchorConfig: 16-field config struct with defaults pinned to Python.
- DataPattern: SearchResults / Logs / TimeSeries / Generic. from_string
accepts case-insensitive strings; unknown strings fall through to
Generic. Named from_string (not from_str) to avoid colliding with
std::str::FromStr.
- AnchorStrategy: FrontHeavy / BackHeavy / Balanced / Distributed.
- AnchorWeights: front/middle/back distribution + normalize().
- AnchorSelector: stateless selector with .select_anchors() entry point.
Supporting helpers (all parity-pinned):
- calculate_information_score: weighted blend (0.4 uniqueness +
0.3 length + 0.3 structural). Clamped [0,1].
- calculate_value_uniqueness / _length_score / _structural_uniqueness:
per-factor scoring.
- compute_item_hash: md5(json.dumps(item, sort_keys=True))[:16] —
byte-equal with Python.
Critical helper for hash parity: python_json_dumps_sort_keys.
Python's json.dumps default uses (', ', ': ') separators (with
spaces) and ensure_ascii=True (\\uXXXX escapes for non-ASCII,
surrogate pairs for codepoints > U+FFFF). serde_json's compact
default doesn't match. The custom serializer here pins both:
json_dumps_basic {"a": 2, "b": 1}
json_dumps_non_ascii_escaped {"k": "caf\\u00e9"}
json_dumps_emoji_uses_surrogate_pair {"k": "\\ud83d\\ude00"}
compute_item_hash_matches_python_basic → 8aacdb17187e6acf
compute_item_hash_matches_python_with_unicode → 6761da28ed7eb489
Mismatching the format would silently change which items count as
duplicates during region-based anchor selection, so this lays the
foundation for byte-equal parity in the planning layer.
Net: 31 new tests in anchor_selector (299 total in headroom-core),
clippy clean.
Next commits in scope: relevance scorer (BM25 → embeddings via ONNX
→ hybrid), then planning + execution + orchestration layers, then
Python lockstep bug fixes + parity fixtures.
Python's _compute_k_split (smart_crusher.py:2722) computes:
k_first = max(1, round(k_total * first_fraction))
k_last = max(1, round(k_total * last_fraction))
For k_total=1, both round() results are 0 and both max(1, …) return
1, giving k_first + k_last = 2 > k_total = 1. The crusher then keeps
2 items when adaptive sizing said 1 — violating
max_items_after_crush whenever k_total floors to 1.
Fix: after the floored fractions, clamp:
k_first = min(k_first, k_total)
k_last = min(k_last, k_total - k_first)
For k_total >= 2 (the common case) the clamp is a no-op — Python and
Rust agree byte-for-byte. For k_total <= 1 (the previously buggy
edge), Rust now keeps ≤ k_total items.
This is a one-sided fix: Rust correct, Python overshoots. The Python
fix lands in commit 7 alongside parity fixtures (matching the
already-applied pattern for bugs #2 and #3, both of which are also
fixed in Rust now and pending in Python at commit 7).
Real-world reachability: every crusher early-returns "passthrough" on
n <= 8 before compute_k_split is even called, so the only path that
reaches k_total=1 in production is via direct calls to compute_k_split
or compute_optimal_k. Defensive parity remains worth pinning.
Two new tests:
- bug4_k_split_no_overshoot_when_k_total_is_one (the fix itself)
- bug4_k_split_no_overshoot_when_k_total_is_two (boundary not regressed)
Net: 268 unit tests passing, clippy clean, parity harness intact.
Bug status:
#1 percentile off-by-one — Rust faithful (port-as-is), both fixed at commit 7
#2 sequential pattern int(zero-padded) — Rust fixed, Python at commit 7
#3 rare-status cardinality cap — Rust fixed, Python at commit 7
#4 k-split overshoot — Rust fixed (this commit), Python at commit 7
Three crushers from headroom/transforms/smart_crusher.py ported.
Each takes a SmartCrusherConfig + bias and returns
(crushed_items, strategy_string). All schema-preserving — output is
items/values from the original; no generated text.
What's in:
1. compute_k_split (smart_crusher.py:2693)
Wraps adaptive_sizer::compute_optimal_k. Splits k_total into
first/last/importance via config.first_fraction / last_fraction.
Uses f64::round_ties_even() (Rust 1.77+) to match Python's
banker's-rounding round() — important for off-by-one parity on
.5-edged k computations.
2. crush_string_array (smart_crusher.py:2727)
Adaptive K via Kneedle. Mandatory-keep: error-keyword strings +
length-anomaly strings (>variance_threshold σ from mean length).
Boundary-keep: first K_first + last K_last. Stride-based diverse
fill with content-dedup. Output preserves original array order
(BTreeSet iteration). Strategy includes dedup= and errors= counts
when nonzero.
3. crush_number_array (smart_crusher.py:2810) — CARRIES BUG #1
Statistics-driven (mean/median/stdev/p25/p75). Outliers flagged
at variance_threshold σ. Change-points via window-mean comparison
(config.preserve_change_points + n>10 gates). Strategy string
embeds full stats summary via format_g (Python's :.4g approximation).
BUG #1 — percentile off-by-one — ported AS-IS:
sorted_finite[len/4] / sorted_finite[3*len/4]. Cosmetic
(strategy-string only). Test bug1_percentile_off_by_one_documented
pins the buggy index choice; commit 7 fixes both languages and
regenerates fixtures.
4. crush_object (smart_crusher.py:3015)
Token-budget gate (config.min_tokens_to_crush=200). Three
passthrough exits: n<=8, total tokens too low, k_total>=n. Always
keeps: error-keyword values + small values (<=12 tokens via
len/4 + len/4 + 2 heuristic). Boundary keys + stride fill with
Python's recompute-each-iter cap (mirrored faithfully — slower
but parity-true). Output preserves key insertion order via
serde_json/preserve_order's IndexMap.
Supporting helpers in stats_math.rs:
- median(values) — Python statistics.median (mean-of-middles for
even, total_cmp sort for NaN determinism).
- format_g(x) — approximate Python f"{x:.4g}" (4 sig figs,
scientific outside [-4, 4) exponent range, trailing-zero strip,
explicit-sign 2-digit exponent). Pinned by 5 fixed-output tests.
Field iteration order: key/object iteration uses BTreeMap-sorted (in
analyzer) and IndexMap-insertion-order (in serde_json::Map for
crush_object). The Python sorted-key fix scheduled for commit 7 also
covers crush_object's iteration paths.
Net: 266 unit tests passing in headroom-core, clippy clean (MSRV 1.80),
parity harness intact (4/4 diff_compressor).
Next commit: planning + execution layer (_create_plan, _execute_plan,
plan-builder methods) with BUG #4 fix (k-split overshoot).
Direct port of headroom/transforms/adaptive_sizer.py. This is the
prerequisite that smart_crusher's array crushers (string, number,
object, mixed, dict) all depend on for "how many items to keep" via
information-saturation detection.
Six pieces, each with verified-against-Python tests:
1. simhash(text) -> u64
Character 4-gram window. Hash each gram with MD5; take first 8 bytes
as big-endian u64 (matches Python's int(hexdigest()[:16], 16)).
Per-bit weighted voting; final fingerprint is bit j set iff votes[j]>0.
Iterates by Unicode codepoints (chars), not bytes — matches Python str
slicing. Pinned values verified for "", "a", "abcd", "hello",
"hello world", "café" (UTF-8 multibyte), and a longer sentence.
2. hamming_distance(a, b) -> u32
XOR + count_ones. Trivial.
3. count_unique_simhash(items, threshold) -> usize
Greedy clustering of fingerprints by Hamming distance. First-fit
matches Python's append-only iteration order.
4. compute_unique_bigram_curve(items) -> Vec<usize>
Whitespace-split lowercase words. Single-word items contribute
(word, ""); empty-string items contribute ("", ""). Returns
cumulative unique-bigram count after each item.
5. find_knee(curve) -> Option<usize>
Kneedle in normalized [0,1] space. Returns knee_idx + 1 (matching
Python's "include up to and including" semantics) when max_diff
exceeds 0.05; else None. Flat curves return Some(1).
6. compute_optimal_k(items, bias, min_k, max_k) -> usize
Three-tier orchestrator: fast path (n<=8 → n, or unique<=3),
Kneedle on bigram curve with diversity floor when ratio>0.7, no-knee
fallback to keep_fraction = 0.3 + 0.7*diversity, bias multiplier,
zlib validation, final clamp.
7. validate_with_zlib(items, k, max_k, tolerance)
Compares full vs subset compression ratios. ratio_diff > tolerance
→ bump k by 20%. Uses flate2 default backend (miniz_oxide); per-byte
length drift vs CPython libz absorbed by the 15% tolerance. If
parity fixtures flake, swap to flate2 features = ["zlib"] for
byte-equal output to system libz.
A counterintuitive case I verified: 20 identical short lines with k=5
DOES trigger the 20% bump (subset compresses LESS efficiently per byte
than the full text — zlib has less context). Pinned in
validate_zlib_bumps_k_when_subset_undercompresses; confirmed Python
returns the same 6.
Net: 33 new tests in adaptive_sizer (242 total in headroom-core),
clippy clean, parity harness intact.
Next commit: three array crushers (string, number, object) + bug #1
(percentile off-by-one) using compute_optimal_k.
Self-review of commit c26d225 found one real divergence from Python:
Python's `_analyze_field` numeric block is wrapped in
`try/except (OverflowError, ValueError)`. On overflow (e.g., variance
of `[1e200, -1e200]`), the except branch resets:
stats.min_val = None
stats.max_val = None
stats.mean_val = None
stats.variance = 0 # ← int literal, not None
stats.change_points = []
Rust was silently propagating Inf/NaN. Two-part fix:
1. `stats_math.rs::{mean,sample_variance,sample_stdev}` now return None
when the result is non-finite. This mirrors Python's exception path
for the overflow-on-extreme-floats case.
2. `analyzer.rs::analyze_field` numeric branch now detects any non-
finite stat (or None from helpers) and resets the entire group
atomically. Variance is reset to `Some(0.0)` to match Python's
`variance = 0` literal — downstream truthiness checks
(`if stats.variance:` and `(variance or 0) > 0`) treat 0 the same
as None, but the FieldStats serialization shape will matter for
the parity fixtures landing in commit 7.
Three new tests pin the new behavior:
- stats_math::mean_non_finite_overflow_returns_none
- stats_math::sample_variance_non_finite_returns_none
- stats_math::sample_stdev_non_finite_returns_none
- analyzer::analyze_field_numeric_overflow_resets_all_stats_to_none
Other findings from the review were documented as either deferred to
Stage 3c.2 (bool-as-numeric anomalies, exotic float formatting) or
pinned to Stage 3c.1 commit 7 (Python sorted-key iteration alongside
fixture regeneration). Each is exotic enough that real fixtures won't
trip it before the Python source fixes land.
Net: 147 unit tests passing in smart_crusher, clippy clean.
Stage 3c.1 commit 2 of ~7. Ports the standalone analyzer helpers from
Python `smart_crusher.py:484-748` and applies the BUG #3 fix to
`detect_rare_status_values` in Rust. The matching Python fix lands
later in this PR (before parity fixtures are recorded).
# What's added
- `error_keywords.rs` — pinned set of 12 error keywords ported from
`headroom/transforms/error_detection.py:18-33`, with three tests
pinning length, casing, and exact membership so accidental edits
surface in CI.
- `field_detect.rs` —
- `detect_id_field_statistically` (Python `smart_crusher.py:484-530`):
high-uniqueness field heuristic with UUID-format and entropy paths
for strings, sequential and high-range paths for numerics. 5 tests.
- `detect_score_field_statistically` (Python lines 533-603): bounded-
range numeric with descending-sort signal. Sequential rejection
matches the Python intent (IDs are sequential, scores aren't). The
Python `[-1, 1]` chained-comparison precedence is faithfully
preserved (`min_val >= -1.0 && max_val <= 1.0`). 7 tests including
confidence-cap and unbounded-range rejection.
- `outliers.rs` —
- `detect_structural_outliers` (Python lines 606-650): rare-field +
rare-status detection. Returns ascending-sorted deduplicated indices
via BTreeSet (Python uses `list(set(...))` with non-deterministic
order — pinning sorted order makes parity fixtures stable).
- `detect_rare_status_values` (Python lines 653-701) **with BUG #3
fix** — see below.
- `detect_error_items_for_preservation` (Python lines 711-748): scans
item JSON for any of the 12 ERROR_KEYWORDS. 6 tests.
# BUG #3 fix — `detect_rare_status_values`
Python's original guard at line 674
`if not (2 <= len(unique_values) <= 10): continue`
caps cardinality at 10, so error-code domains with >10 codes are
skipped entirely. A rare error like "ERR_TIMEOUT" appearing 1% of the
time is missed when the field has 50+ distinct codes.
The fix replaces the cap-and-dominance approach with a Pareto check:
1. Cardinality cap raised to **50** (above which the field is almost
certainly an ID/free-form column, not a status enum).
2. Sort value frequencies descending. Find the smallest K such that
top-K covers >=80% of items.
3. If K <= 5, remaining values are "rare" → items containing them
are outliers.
This unifies both cases the original algorithm partially handled:
- Low cardinality + dominant: 95×ok + 5 errors → top-1 covers 95% →
4 rare values flagged. Same as before.
- Higher cardinality + bimodal: 60×info + 25×warn + 15 distinct rare
errors → top-2 covers 85% → 15 rare values flagged. **New** —
pre-fix this returned 0.
- Uniform distribution: 50 distinct values, 1 each → top-K never
reaches 80% with K <= 5 → skip. Correctly identifies as
non-categorical.
Three regression tests pin each case. The matching Python fix will
land later in this PR before parity fixtures are recorded so the
fixtures byte-match.
# What's NOT in this commit
`detect_items_by_learned_semantics` (Python lines 751-829) is deferred
to a later commit because it depends on the TOIN `FieldSemantics`
type, which isn't ported yet. That helper integrates learned cross-
session compression patterns and isn't on the critical path for the
parity-port milestone.
The next commit lays down the `SmartAnalyzer` struct itself
(`analyze_array`, `_analyze_field`, `_detect_change_points`,
`_detect_pattern`, `_detect_temporal_field`, `analyze_crushability`,
`_select_strategy`, `_estimate_reduction`).
# Build / test
- `cargo build -p headroom-core` clean.
- `cargo clippy -p headroom-core -- -D warnings` clean.
- 167 unit tests in `headroom-core` (was 135).
- `cargo run -p headroom-parity run` — diff_compressor 27/27 still
matched.
Code review (`/code-review` on commit `d219bee`) caught one critical
bug, two important parity gaps, and a few quality nits. Fixed all of
them; all 135 unit tests pass; diff_compressor parity harness
unaffected (27/27 still matched).
# Critical fix — `hash_field_name` truncation length
Rust truncated SHA-256 to **16** hex chars; Python uses **8** (per
`smart_crusher.py:177`: `hashlib.sha256(...).hexdigest()[:8]`). 16-char
hashes would never collide with TOIN's 8-char `preserve_fields`,
silently disabling the entire `use_feedback_hints` cache lookup path.
Fix: `hex[..8]` instead of `hex[..16]`. Three pinning tests re-verified
against actual Python reference output. Doc comment now warns
explicitly that the length must match Python or TOIN lookups silently miss.
# Important fix — `python_int_parse` mirrors Python's `int()` semantics
`statistics.rs::detect_sequential_pattern` previously called
`s.parse::<i64>()`. Python's `int()` differs in three ways that affect
realistic payloads:
- strips ASCII whitespace (Rust's `parse` rejects)
- accepts leading `+` (Rust accepts; same)
- accepts PEP 515 underscores like `"3_000"` (Rust rejects)
A field with `[" 1 ", " 2 ", " 3 ", "4", "5"]` would parse all five
in Python (sequential = True) but only one in Rust (`nums.len() < 5`
→ False). Silent parity break.
Fix: new private `python_int_parse` helper that strips whitespace,
handles underscore separators, and rejects edge cases Python rejects.
Six new tests pin the behavior.
# Important fix — `python_repr` for `item_matches_anchors`
Python compares anchors via `anchor in str(item).lower()`. We were
using `serde_json::to_string(&item).to_lowercase()`, which differs in
three ways that affect substring matching:
- quote chars (`'` vs `"`)
- bool/null literals (`True`/`False`/`None` vs `true`/`false`/`null`)
- spacing (`key: value, ...` vs `key:value,...`)
Anchor `"none"` would match Python form but not JSON. Inverse for
`"null"`. Real divergence.
Fix: new private `python_repr` walks `serde_json::Value` and emits
Python-equivalent form. Plus enable `serde_json/preserve_order` at
workspace level so `Value::Object` preserves JSON parse order
(matching Python `dict` since 3.7).
# Suggestion fixes
- Classifier comment for `[True, False, 1] -> MIXED_ARRAY` now walks
both Python and Rust paths step by step.
- `ArrayAnalysis::field_stats` doc notes the BTreeMap vs Python-dict
order nuance for the analyzer port to resolve.
- Added regression tests for "all unparseable strings", "single int
among strings", fractional-step sequential, and the email-typo
pattern.
# Build / test
- `cargo build -p headroom-core` clean.
- `cargo clippy -p headroom-core -- -D warnings` clean.
- 135 unit tests in `headroom-core`, all passing (was 55).
- `cargo run -p headroom-parity run` — diff_compressor 27/27 still matched.
Stage 3c.1 — like-for-like Rust port of `headroom/transforms/smart_crusher.py`.
This commit lays the foundation: module layout, configuration, foundational
data types, and the simpler helpers (classification, hashing, anchors,
basic statistics). Subsequent commits add the analyzer, crushers, plan
execution, and the orchestrator.
# What's in this commit
`crates/headroom-core/src/transforms/smart_crusher/`:
- `mod.rs` — module entry, public re-exports, port narrative.
- `classifier.rs` — `classify_array` / `ArrayType` (dict/string/number/
bool/nested/mixed/empty). Direct port of `_classify_array`.
- `config.rs` — `SmartCrusherConfig` with defaults pinned to Python
byte-for-byte.
- `hashing.rs` — `hash_field_name` (SHA-256 truncated to 16 hex chars),
matches `hashlib.sha256(name.encode()).hexdigest()[:16]` exactly.
- `statistics.rs` — `is_uuid_format`, `calculate_string_entropy`,
`detect_sequential_pattern` (with **BUG #2 fix** — see below).
- `anchors.rs` — `extract_query_anchors`, `item_matches_anchors`. Five
regex patterns ported via `std::sync::LazyLock`.
- `types.rs` — `CompressionStrategy`, `FieldStats`, `CrushabilityAnalysis`,
`ArrayAnalysis`, `CompressionPlan`, `CrushResult`. Field-by-field
mirror of the Python @dataclasses so the PyO3 bridge in 3c.1b can
reconstruct them without manual translators.
# Bug #2 fixed in this commit (Python fix lands later in same PR)
`smart_crusher.py:444-448` — `_detect_sequential_pattern` calls
`int(string_value)` and silently strips zero-padding, so padded string
IDs like `["001", "002", ..., "100"]` get misclassified as a sequential
numeric pattern. Fix: track whether each parsed numeric value
originated as a string. If EVERY parsed value was a string, refuse to
flag as sequential. Mixed numeric+string fields still detect
correctly because the unambiguous numerics dominate. Test:
`bug2_zero_padded_strings_no_longer_misclassified`.
# What's NOT in this commit (subsequent commits)
- `SmartAnalyzer` — `analyze_array`, `_analyze_field`, `_detect_change_points`,
`_detect_pattern`, `_detect_temporal_field`, `analyze_crushability`,
`_select_strategy`, `_estimate_reduction`.
- The five array crushers (`_crush_array`, `_crush_string_array`,
`_crush_number_array`, `_crush_mixed_array`, `_crush_object`).
- Planning (`_compute_k_split`, `_create_plan`, `_plan_*` family).
- Orchestration (`_prioritize_indices`, `_deduplicate_indices_by_content`,
`_fill_remaining_slots`).
- `SmartCrusher` orchestrator class itself.
- Parity harness fixtures.
- The remaining 3 Python bug fixes (#1, #3, #4) — landed alongside the
code paths they affect.
# Build / test
- `cargo build -p headroom-core` — clean.
- `cargo clippy -p headroom-core -- -D warnings` — clean.
- 55 new unit tests across the 6 new files, all passing.
Architectural improvements (lossless-first, unified saliency score,
structured CCR markers) are deferred to Stage 3c.2 — see design doc at
`~/Desktop/SmartCrusher-Architecture-Improvements.md`.
The previous version of this step did `python -c "import headroom._core"`
to find the wheel's installed `.so` path. That failed in CI:
ModuleNotFoundError: No module named 'headroom._core'
— exactly the chicken-and-egg this step exists to fix. The editable
install (`pip install -e .`) puts the in-tree `headroom/` source dir
ahead of site-packages on `sys.path`. So `import headroom` finds the
in-tree dir (which doesn't yet have the `.so`), then
`import headroom._core` fails to find the submodule. The symlink we're
about to create is what makes the import work — but we can't import
to discover the symlink target before creating it.
Locate the `.so` via filesystem instead: read site-packages from
`site.getsitepackages()[0]`, glob for `_core.cpython-*.so` under
`<site-packages>/headroom/`, and symlink that into the in-tree dir.
The smoke test (`from headroom._core import DiffCompressor`) runs
*after* the symlink and confirms end-to-end resolution.
Also added `set -euo pipefail` and a sanity check with `ls -la` of the
site-packages dir if the glob comes up empty, so future failures
diagnose themselves.
`maturin develop` requires a virtualenv (it errors with "Couldn't find a
virtualenv or conda environment"). CI's setup-python provides a bare
system Python without a venv, so the dev script's build path doesn't
work there.
This switches CI to build a release wheel via `maturin build` and
install it with `pip install --force-reinstall --no-deps`. Then symlink
the installed `.so` into the in-tree `headroom/` package so the
editable install resolves `import headroom._core` past the source-dir
shadowing of site-packages.
`scripts/build_rust_extension.sh` is unchanged — it stays optimized for
local dev (where there IS a venv).
