PR #396's X2 dry-run caught a wheel-import failure on the manylinux_2_28
floor matrix entry (both x86_64 and aarch64). Same class as #355:
ImportError: ... undefined symbol: __libc_single_threaded
`__libc_single_threaded` is a single-byte char added in glibc 2.32.
Newer libstdc++ (gcc 11+) reads it inside `__cxa_thread_atexit_impl`
to elide locking on the single-threaded fast path. ORT prebuilt static
archives compiled with gcc-14.2.1 against glibc-2.38+ headers bake in
the reference. Users with glibc < 2.32 hit ImportError on
`import headroom._core`.
Latent since the ORT artifact bump that started using gcc 14. X1 is
the gate that catches it at release time; X2 caught it at PR time —
exactly as designed.
Fix:
1. glibc_compat.c adds Section B: `char __libc_single_threaded = 0;`
Setting to 0 (multi-threaded) is safe; libstdc++ takes the locked
slow path. Setting to 1 would race in any multithreaded Rust wheel.
2. build.rs adds `-Wl,-u,__libc_single_threaded` so the shim's archive
members are pulled regardless of scan order.
3. audit_wheel_glibc_symbols.py POST_FLOOR_SYMBOLS adds the new
symbol — verified locally: the audit now rejects the failing
PR #396 wheel with the right message.
The previous hot-fix (#363) addressed npm artifact downloads and added
openssl-devel installs in the manylinux container, but the wheel build
still fails on three of four matrix entries with three distinct errors:
1. ubuntu-x86_64 with `manylinux: auto` resolved to manylinux2014
(CentOS 7 / OpenSSL 1.0.2k). `openssl-sys 0.9` requires OpenSSL
1.1.0+ — "different version of OpenSSL was found".
2. ubuntu-aarch64 cross-compiles via `aarch64-unknown-linux-gnu-gcc`
from an x86_64 manylinux container. The `yum install openssl-devel`
we added installs x86_64 headers; `/usr/aarch64-unknown-linux-gnu/
include/` has no OpenSSL — "openssl/opensslv.h: No such file or
directory".
3. macos-15-intel fails on `ort-sys` (transitive via the ML compression
backend), which has no prebuilt ONNX Runtime binaries for
`x86_64-apple-darwin`. Unrelated to OpenSSL; an upstream limitation.
Why the workspace pulls openssl-sys at all: `hf-hub` (transitive via
`fastembed`) hard-codes `native-tls` as a default feature. Cargo's
feature unification then enables openssl-sys for the whole workspace
despite our `reqwest`/`tokio-tungstenite`/`tokio-rustls` preferences.
# Fix 1: vendored OpenSSL
Add `openssl = { version = "0.10", features = ["vendored"] }` to
`crates/headroom-proxy/Cargo.toml`. The `vendored` feature compiles
OpenSSL from source as part of the cargo build — works on every target
uniformly. Local build verified: cargo now pulls
`openssl-src v300.6.0+3.6.2` and compiles it. ~30s extra one-time
build cost.
The `openssl/vendored` feature DEFEATS `OPENSSL_DIR`. We therefore
remove the previous hot-fix's "Install OpenSSL (macOS)" step that
exported `OPENSSL_DIR` — leaving it would silently regress to the
system-OpenSSL path that broke originally.
# Fix 2: pin manylinux floor to 2_28
Change x86_64-unknown-linux-gnu from `manylinux: auto` to
`manylinux: 2_28` (matching aarch64 + the e2e Dockerfiles). This
isn't strictly required with vendored OpenSSL — the floor is now
glibc 2.28 / AlmaLinux 8 which has modern toolchain — but it removes
the CentOS-7 surface entirely and matches our runtime container
target.
# Fix 3: drop x86_64-apple-darwin from the matrix
`ort-sys 2.0.0-rc.12` has no prebuilt ONNX Runtime binaries for that
target. Building ORT from source would add CMake + ~5 minutes per
build. Apple Silicon macOS (`aarch64-apple-darwin`) is fully covered;
Intel-mac users install from the platform-independent sdist this
matrix also produces.
Tracked as a follow-up: switch the ML backend to `ort-tract` or
upstream a request for x86_64 macOS prebuilts.
# before-script-linux: keep perl-IPC-Cmd, drop openssl-devel
OpenSSL's vendored `Configure` script needs `IPC::Cmd` (without it
the build fails with "Can't locate IPC/Cmd.pm"). System
openssl-devel is no longer needed.
# Tests
4 new regression tests gate this:
- `test_headroom_proxy_vendors_openssl`
- `test_build_wheels_installs_perl_ipc_cmd_for_vendored_openssl`
- `test_build_wheels_does_not_set_openssl_dir`
- `test_build_wheels_matrix_excludes_intel_macos`
Plus the previous 7. All 11 release-workflow tests pass.
`make ci-precheck` PASSED. Local `cargo build --release -p headroom-py`
green.
Eliminates the dual-package architecture that was the root cause of #355.
`pip install headroom-ai` now produces ONE wheel containing both the Python
source (headroom/*.py) and the compiled Rust extension (headroom/_core.so).
No more separate `headroom-core-py` package, no more chicken-and-egg with
PyPI publication, no more wheelhouse / PIP_FIND_LINKS / composite-action
plumbing in CI.
This is the canonical pattern used by cryptography, polars, ruff,
pydantic-core, and other Rust-as-core Python packages. Honors the
"Rust as core engine" direction.
## What changed
- pyproject.toml: `[build-system]` swapped from hatchling to maturin.
`[tool.hatch.*]` deleted; `[tool.maturin]` added pointing at
`crates/headroom-py/Cargo.toml` for the cdylib. `python-source = "."`
picks up the root `headroom/` package directly (dashboard HTML
templates and other non-Python files included automatically).
- crates/headroom-py/pyproject.toml: deleted. The crate is no longer a
separate published package; its Cargo.toml stays as the cdylib build
target invoked via `[tool.maturin] manifest-path`.
- crates/headroom-py/python/: deleted (placeholder layout for the old
separate package).
## CI updates
- ci.yml: `test` / `test-extras` / `test-agno` jobs simplified — Rust
toolchain set up before `pip install -e .` (which now invokes maturin
via build-system). Removed the "build wheel + symlink .so" dance.
`build` job swapped from `python -m build` (hatch) to
`maturin build` + `maturin sdist`.
- release.yml: collapsed dual-package matrix into one. New `build-wheels`
matrix produces cross-platform wheels for cp310/11/12/13 ×
{linux x86_64, linux aarch64, macos x86_64, macos aarch64}. New
`collect-dist` aggregator merges artifacts. publish-pypi consumes the
merged dist.
- init-native-e2e.yml: dropped windows-latest from the matrix —
upstream `esaxx-rs` (/MT) and `ort-sys` (/MD) link with conflicting
MSVC C runtime libraries, so the Rust extension cannot build for
win_amd64 today. Tracked as a follow-up; not a blocker for Linux+macOS.
- headroom-e2e-setup: composite action now sets up Rust toolchain +
Swatinem/rust-cache before `pip install -e .[proxy]`.
- eval.yml, publish.yml, rust.yml: same pattern — rust toolchain before
install. rust.yml's wheels job builds from root pyproject.toml (no
more `-m crates/headroom-py/Cargo.toml`).
- e2e/init/Dockerfile, e2e/wrap/Dockerfile: install rust + maturin in
the build stage; copy `crates/` + workspace `Cargo.toml/lock` so the
install can build the extension. Dropped `HEADROOM_REQUIRE_RUST_CORE=false`
from wrap-e2e — the image now ships the full Rust core.
- Dockerfile (main): simplified — no more Layer 2/3 dance with
`headroom-core-py` install + symlink. Single `uv pip install` builds
+ installs everything.
- .devcontainer/Dockerfile: rust toolchain + libssl-dev + maturin
added so `uv sync` builds the extension inside the devcontainer.
## Lockfile + script
- uv.lock: regenerated. No `headroom-core-py` entries remain.
- scripts/build_rust_extension.sh: simplified from a symlink-into-tree
workaround to a thin wrapper around `pip install -e .`. The maturin
build-backend handles placement automatically.
## Local validation (all green on macOS aarch64)
1. Clean venv `pip install -e .` → `from headroom._core import …` works.
2. `maturin build --release` → 13.8 MB wheel, 336 files including
`headroom/_core.cpython-311-darwin.so` (32 MB cdylib) and
`headroom/dashboard/templates/dashboard.html`.
3. `pip install <wheel>` in fresh venv → import works.
4. Wheel contents verified via `unzip -l`.
5. `pytest tests/test_transforms/test_diff_compressor.py` — 29 passed.
6. `pytest tests/test_relevance.py` — 30 passed.
7. `cargo build --workspace` + `cargo test --workspace` — all green.
8. `make ci-precheck` — 176 Python tests + Rust + commitlint green.
## Migration notes
Users on `pip install headroom-ai` get the Rust core automatically
(linux + macos wheels). sdist installs require rust toolchain available
locally — pip will build via maturin.
Closes#355
Supersedes #357 (workarounds-based fix abandoned in favor of
architectural fix)
Eliminate P3-33 / P3-34. Wraps every per-block compression in
the live-zone dispatcher with two new gates:
1. Per-content-type byte thresholds — pinned as `const` at the top
of `live_zone.rs` so the table is grep-able and reviewable in
one place. No magic numbers anywhere in the dispatch logic; a
`threshold_for(ContentType)` helper returns the value. Below
threshold → no compressor invoked, recorded as
`BlockAction::BelowByteThreshold { content_type, byte_count,
threshold_bytes }`. Thresholds:
- JSON-array tool_results: 1 KiB
- Build / log output: 512 B
- Search-result blocks: 1 KiB
- Git-diff blocks: 1 KiB
- Source code: 2 KiB (pinned for the future
Rust code-compressor port)
- Plain text: 5 KiB (pinned for Kompress wiring)
- HTML: 5 KiB (no compressor today)
2. Tokenizer-validated rejection — the byte-length proxy
(`compressed_bytes >= original_bytes`) is replaced with a
token-count check using `headroom_core::tokenizer::get_tokenizer`.
The dispatcher creates one tokenizer per request (model-aware
via the new `model: &str` parameter to
`compress_anthropic_live_zone`) and counts both the original
and compressed text. When `compressed_tokens >= original_tokens`
the candidate is rejected and the original bytes are kept.
`BlockAction::Compressed` and `BlockAction::RejectedNotSmaller`
gain `original_tokens` and `compressed_tokens` fields so the
proxy can log token-savings (the currency that actually matters
for prompt cache + provider billing) instead of bytes.
The proxy `live_zone_anthropic.rs` extracts `body["model"]` (or
falls back to `DEFAULT_MODEL = "claude-3-5-sonnet-20241022"` when
the field is missing — the chars-per-token estimator is calibrated
for the Claude family at 3.5 cpt) and threads it through. The
`Compressed` outcome now reports token counts from the manifest,
not byte counts, so the existing
`tokens_before / tokens_after` plumbing is suddenly accurate.
Tests added:
- `live_zone_thresholds.rs::below_threshold_no_compression_attempted`
— 200 B JSON array → `BelowByteThreshold` and `NoChange`.
- `live_zone_thresholds.rs::above_threshold_compression_attempted`
— 10 KB JSON array → byte-threshold gate clears and a compressor
runs (either `Compressed` or `RejectedNotSmaller`).
- `live_zone_token_validation.rs::compressed_more_tokens_falls_back`
— pathological input must not produce `Compressed` with
`compressed_tokens >= original_tokens`.
- `live_zone_token_validation.rs::compressed_fewer_tokens_accepted`
— well-formed JSON array of dicts → `Compressed` with strict
token shrinkage.
- Property test `live_zone_compression_token_count_non_increasing`
— for any well-formed body generated by `proptest`, the
dispatcher's emitted body has token-count <= input's token-count.
Pins the central PR-B4 invariant: the dispatcher never inflates
tokens.
Existing 12 unit tests in `live_zone.rs` and 6 integration tests
in `tests/live_zone_dispatch.rs` updated for the new field shape
and the `model` parameter; all pass. The diff-routing test's
fixture grew to 1.3 KiB so it clears the new GitDiff threshold
gate, exercising the dispatch path rather than short-circuiting.
Per-PR-B4 plan: REALIGNMENT/04-phase-B-live-zone.md.
PR-A6 of the Phase A cache-safety lockdown. Eliminates P5-50 and preps
P0-6 (memory tool injection toggling).
Two cache-killer patterns the merge + tracker defeat:
1. Mid-session mutation: when memory was enabled the proxy did an
ad-hoc concat of `context-management-2025-06-27` onto the client
value (anthropic.py:1244-1248). The order varied with the client
value, breaking byte-stable headers across turns.
2. Token drop-out across turns: clients (Claude Code, Codex CLI) MAY
drop a beta token between turn N and turn N+1 even when the proxy
mutated turn N to add it. The cache hot zone is positional, so the
next turn's prefix bytes hash differently and the prefix-cache
read misses.
Changes
-------
`headroom/proxy/helpers.py`
* `merge_anthropic_beta` / `merge_openai_beta`: pure, deterministic,
order-preserving merge. Client tokens first (in their original
order), then Headroom-required tokens (in the order passed). Dedupe
is case-insensitive but preserves the original casing of the first
occurrence. No regex.
* `SessionBetaTracker`: bounded LRU keyed by (provider, session_id),
unioning client tokens with previously-seen tokens. OrderedDict
LRU; threading.RLock for thread safety (mirrors the
CompressionCache pattern from compression_cache.py).
* `get_session_beta_tracker` / `_reset_session_beta_tracker_for_test`
process-wide singleton with test reset.
* `log_beta_header_merge`: structured log per cache-affecting merge.
* Env-var knobs (NO HARDCODES):
- HEADROOM_BETA_HEADER_STICKY=enabled|disabled (default enabled).
- HEADROOM_BETA_TRACKER_MAX_SESSIONS (default 1000).
`headroom/proxy/handlers/anthropic.py`
* After `compute_session_id` (line ~744): record client
`anthropic-beta` against the session tracker, write the sticky
value back into `headers` if changed. Order matters: sticky-merge
FIRST so memory-injection has the canonical baseline.
* Memory-injection site (line ~1244): replace the ad-hoc concat with
`merge_anthropic_beta(headers["anthropic-beta"], required_tokens)`.
`headroom/proxy/handlers/openai.py`
* Chat-completions (line ~360): record/merge `openai-beta`.
* /v1/responses HTTP (line ~1213): compute `_responses_session_id`
and record/merge `openai-beta`.
* /v1/responses WS (line ~1711): replace the ad-hoc absent-only
inject with `merge_openai_beta(sticky, ["responses_websockets=
2026-02-06"])`. Replaces any case-variants of the existing key.
Tests
-----
`tests/test_anthropic_beta_session_sticky.py` (26 tests):
* Pure helper: empty inputs, only-client, only-headroom, ordering,
dedupe casing, deterministic memory-injection order, no-double-
inject when token already present.
* Tracker: sticky-on across turns even when client drops, casing
preservation, provider namespace independence, LRU eviction at
max_sessions, env-var validation (loud failures), thread safety
under 16-thread concurrent access, blank-input rejection.
`tests/test_openai_beta_session_sticky.py` (17 tests):
* Mirror of the anthropic suite for `OpenAI-Beta`.
* Plus WS-specific coverage: sticky-then-merge of
`responses_websockets=2026-02-06` against client baseline.
`tests/test_openai_codex_routing.py`
* Add `session_tracker_store` stub to `_DummyOpenAIHandler` so the
routing tests still exercise the responses HTTP handler now that
it computes a session_id for beta-merge.
Notes
-----
Build constraints honored:
* Configurable: HEADROOM_BETA_HEADER_STICKY,
HEADROOM_BETA_TRACKER_MAX_SESSIONS.
* No regex, no hardcodes (env-var bounds), no fallbacks (disabled
mode is operator opt-in for diagnostics, loud failures on invalid
values).
* Structured tracing log via `log_beta_header_merge`.
Acceptance:
* 43 new tests pass.
* `cargo test --workspace` green (no Rust changes).
* `make ci-precheck` green.
Adds an opt-in compression interceptor that buffers Anthropic
/v1/messages requests, runs IntelligentContextManager over the
messages array, and forwards the (possibly trimmed) body upstream.
All other paths, methods, and content-types stay on the original
streaming passthrough — so existing operators see zero change.
Behaviour gates ALL must be true to buffer + compress:
- --compression flag (or HEADROOM_PROXY_COMPRESSION=1)
- method == POST
- path == /v1/messages
- Content-Type: application/json
- ICM constructed successfully at startup
Falls through to streaming on any failure: parse, missing fields,
unknown model, body-too-large. Compression must never break a
request — that's the safety contract.
Model context windows come from a vendored LiteLLM snapshot at
crates/headroom-proxy/data/model_prices_and_context_window.json
parsed once into an OnceLock<HashMap>. Refresh via
scripts/refresh_model_limits.sh. Rationale documented inline:
hardcoded tables silently rot; LiteLLM is the canonical source
the entire LLM-tooling ecosystem relies on.
New tests:
- 16 unit tests across compression::{anthropic, icm, model_limits}
- 5 integration tests: off-passthrough, on-short-passthrough,
on-oversized-trim, on-non-json-skip, on-non-llm-path-skip
Verification:
- cargo test --workspace -> 884 passed, 0 failed
- cargo clippy --workspace -- -D warnings -> clean
- cargo fmt --check -> clean
Replaces PR1's lossless/lossy split with ReformatTransform (pack
denser, no info lost) and OffloadTransform (drop bytes, CCR-stash
original via required cache_key). With CCR every transform is
information-preserving end-to-end, so the lossless/lossy distinction
misnamed the architecture.
OffloadTransform carries a cheap, structural estimate_bloat() method
scoped to its domain — generic byte-redundancy heuristics miss domain
semantics. The orchestrator runs reformat phase + per-offload bloat
estimation in parallel via rayon::join + par_iter, then runs offload
iff bloat clears threshold OR reformat underwhelmed.
Transforms shipped:
REFORMATS (lossless):
- JsonMinifier: serde_json round-trip whitespace stripping.
- LogTemplate: Drain-inspired order-preserving template miner.
Collapses consecutive runs of same-template lines into
[Template Tn: ...] (Nx) + variant table. Win comes from emitting
the constant-token prefix once instead of N times. Lossless: every
original line reconstructible from template + variants.
OFFLOADS (drop bytes, stash original via CCR):
- LogOffload: wraps existing LogCompressor; bloat = repetition x
uniqueness_weight + dilution x priority_dilution_weight.
- DiffOffload: wraps existing DiffCompressor; bloat = context-to-
change ratio. Bug-fix-on-port — persists original under the
cache_key the parity-bound DiffCompressor mints (closes a leak).
- DiffNoise: drops lockfile hunks (Cargo.lock, package-lock.json,
yarn.lock, etc., suffix list configurable in TOML) and
whitespace-only hunks. Stashes original via CCR for retrieval.
Search offload exists but is not in default re-exports — modern
agents (Claude Code, Codex) use scoped rg/grep, the marginal value
didn't justify default registration. Reach via the explicit module
path if opting in.
JSON Offload is intentionally absent from this PR — already lives at
SmartCrusher; Phase 3g PR3 wraps it in the OffloadTransform contract.
Thresholds and weights live in config/pipeline.toml, embedded via
include_str!; PipelineConfig::from_toml_str loads runtime overrides.
98 new pipeline tests; full headroom-core suite (714) and workspace
tests green; cargo fmt clean. No regex, per project convention.
Ports `headroom.transforms.log_compressor` to Rust. The biggest-by-
impact remaining compressor port: build/test logs are where the
10-50x compression wins live.
* Stack-trace state machine: per-flavor dispatcher (Python Traceback,
JS, Java, Rust error, Go); each flavor has its own termination
rule. Python terminated on any blank line, dropping mid-trace
lines from chained-exception traces.
* Conservative dedupe: preserves message prefix (everything before
first `:` or `=`); only trailing region is tokenised. Python's
blanket normalisation collapsed segfaults at different addresses.
* Loud CCR failures: `tracing::warn!` + `logger.warning` instead of
bare `except: pass`.
* `LogLevel::FAIL` documented as cosmetic-equivalent to ERROR.
Same shape as search_compressor port. Rust `LogCompressor`
orchestrates format detect -> classify -> score -> select ->
format -> CCR. Inline static-table format detector (YAGNI),
aho-corasick level classifier with word-boundary post-filter
(`signals::keyword_detector` technique), hand-rolled per-flavor
stack-trace state machine. `signals::LineImportanceDetector` NOT
consumed -- log levels are structural, not prose-style importance.
`headroom.transforms.log_compressor` becomes a thin shim:
`compress()` delegates to Rust end-to-end; internal helpers
preserved for the existing 50-test surface. Two existing tests
updated for new dedupe semantics + new compress orchestration.
* 17 Rust unit tests
* 50 Python tests pass
* `make ci-precheck` clean
Establish `crates/headroom-core/src/signals/` as a top-level module
holding cross-cutting detection traits. Phase 3e.1 ports
`error_detection.py` to a `LineImportanceDetector` trait + a
`Tiered<T>` combinator + a single concrete `KeywordDetector` impl
backed by aho-corasick. Three traits at three granularities are
sketched (line / blob / item); only line-importance is implemented
today.
Two bug fixes from the Python source bake into both the Rust impl
and the Python regex shim:
1. `ERROR_KEYWORDS` listed `timeout|abort|denied|rejected` but
`ERROR_PATTERN` regex omitted them. Lines like `"Connection
timeout"` were silently neutral despite the keyword being canonical.
Both surfaces now flag them.
2. `SECURITY_KEYWORDS` carried `token`, which false-positived on
every reference to LLM tokens (`input_tokens`, `tokens_saved`, ...)
in our own product. Dropped from the security set.
The Python `error_detection.py` shim now reflects keyword data out of
Rust via `keyword_registry_snapshot()` and recompiles the legacy
`re.Pattern` objects on the fly. Existing callers (text_compressor,
search_compressor, intelligent_context) continue to import the same
names with no source changes; caller migration to the trait API
happens in their own port PRs.
The trait architecture is the seam where a future ML detector slots
in without touching `KeywordDetector` or any caller. The canonical
extension is documented in `signals/README.md` as a classifier head
on the existing `bge-small-en-v1.5` embedder loaded by
`relevance::EmbeddingScorer` -- 384-dim -> 4-class softmax,
~1.5 KB head, ~1 ms inference, no extra model file. Two alternatives
(distilled tinyBERT in ONNX, logistic regression on lexical
features) are kept open in case BGE-head underfits.
Per the no-silent-fallbacks rule: only `KeywordDetector` lands as a
concrete impl. No NoOp, no MockDetector, no stub-ML -- those will
arrive with their real implementations.
Phase 3g (Compression Pipeline Formalization, issue #315) is queued
as the cross-cutting follow-up that will make lossless-then-lossy-
then-CCR ordering an explicit, observable architecture rather than
implicit per-compressor logic. Trait shapes there will reuse the
signals primitive landed in this PR.
Re-lands two audit fixes that were marked "merged" on GitHub but never
reached main: squash-merging the parent stack changed its commit SHA,
which silently dropped the contents of the stacked PRs (#301, #305).
Single PR this time — no stacking risk.
What lands:
1. **`enable_ccr_marker` Rust gate** — new field on `SmartCrusherConfig`
(default `true`). `crush_array` checks it before emitting the
`<<ccr:HASH>>` marker text and the CCR store write. PyO3 surface +
parity-fixture tolerance updated; recorded fixtures predate the
field and inherit the `true` default.
2. **Python shim collapses both flags to the gate** — both
`ccr_config.enabled=False` and `ccr_config.inject_retrieval_marker
=False` now flip the Rust gate off. Storing a payload nothing in
the prompt can reference is pointless, and storing under
`enabled=False` would be a surprise side effect the user
explicitly opted out of.
3. **Custom `scorer` / `relevance_config` fails loud** — replaces the
prior WARNING-and-drop. Silently dropping a user-supplied scorer
is a textbook silent fallback. `NotImplementedError` instead.
Verified zero production callers pass these args; full plumbing
arrives with Stage-3c.2's relevance-crate Python bridge.
Tests:
- 2 new Rust unit tests in `crusher.rs::tests`
- 6 new Python tests in `test_smart_crusher_toin_attachment.py`
(3 CCR marker-knob behaviors + 3 scorer fail-loud)
- Removed `test_ccr_inject_marker_false_logs_warning` (the WARNING is
gone now that the flag is honored)
- `make ci-precheck` green; eval suite + observability tests run
twice consecutively to verify no TOIN file pollution leaks into the
regular+coverage double-run on Python 3.11
RUST_DEV.md audit table reflects both gaps closed.
The PR8 marker injection appends a sentinel object
{"_ccr_dropped": "<<ccr:HASH N_rows_offloaded>>"} to the kept-items
array on the lossy path so the LLM sees the retrieval pointer in the
prompt. Tests that iterate compressed arrays via subscript access
(e["level"], r["status"], i["labels"], m["text"]) hit KeyError on
the sentinel because it doesn't share the record schema.
Same root cause as the test_quality_retention fixes in PR8 -- these
integration tests were left out of that pass.
Ship a public helper headroom.transforms.smart_crusher.strip_ccr_sentinels
so tests can use it cleanly: `for e in strip_ccr_sentinels(entries):`
and production callers iterating compressed output get a single
canonical filter instead of inlining the _ccr_dropped check.
The 7 previously-failing tests in PR #292 CI now pass:
- langchain test_100_percent_errors_preserved_logs
- langchain test_errors_preserved_with_many_errors
- langchain test_search_results_with_query_term
- mcp test_all_log_errors_preserved
- mcp test_slack_significant_compression_with_content
- mcp test_database_error_status_preserved
- mcp test_github_bugs_partial_preservation
753 tests across the integration + transforms + retention suites pass
locally. Plugin manifests auto-bumped 0.13.3 -> 0.13.4 by the
sync-plugin-versions hook (unrelated to this fix).
The action was set to @stable, which installs whatever the latest
stable is (1.95.0 right now). Then maturin invokes cargo, which reads
rust-toolchain.toml and re-resolves to "1.95.0 + clippy + rustfmt".
rustup treats stable and 1.95.0 as distinct toolchain identities and
refuses the second install with:
failed to install component 'clippy-preview-x86_64-unknown-linux-gnu',
detected conflict: 'bin/cargo-clippy'
This was intermittent across the matrix (only test (3.10) tripped on
the most recent run; others got lucky on cache state). Pinning the
action ref to 1.95.0 makes both sides ask for the exact same toolchain
identity, so the second install is a no-op and the conflict can't fire.
Bump procedure stays the same: when rust-toolchain.toml's channel
changes, update these refs in lock-step.
Plugin manifests auto-bumped 0.11.0 -> 0.13.2 by sync-plugin-versions
hook (unrelated to the workflow fix).
Stage 3c.2 PR2. Adds an opt-in compaction stage that runs BEFORE the
existing lossy pipeline. When configured, it tries to losslessly
re-shape arrays of objects into a recursive Compaction IR and renders
that to bytes via a pluggable Formatter trait. When not configured
(default OSS), behavior is byte-equal with the pre-PR2 path — all 17
SmartCrusher parity fixtures stay green.
# What lands
- Recursive Compaction IR (`compaction/ir.rs`): Table / Buckets /
OpaqueRef / Untouched. CellValue can hold a nested Compaction so
multi-level cases (stringified-JSON inside cells, heterogeneous
arrays bucketed by discriminator, opaque blobs CCR-substituted)
share one tree shape.
- Cell classifier (`compaction/classifier.rs`): per-cell decision —
Scalar / JsonObject / JsonArray / StringifiedJson(parsed) /
Opaque(kind). Conservative: in doubt, return Scalar.
- TabularCompactor (`compaction/compactor.rs`): array → IR. Handles
uniform-nested flattening into dotted columns ("meta.region",
"meta.tier"), stringified-JSON parsing + recursion, opaque-blob
CCR-substitution (12-char SHA-256 prefix), and heterogeneous
bucketing by discriminator. Falls through to a sparse Table when
no clean discriminator exists, so we always do better than the
lossy path for object arrays.
- Formatter trait (`compaction/formatter.rs`) + two impls:
- JsonFormatter: structured JSON for debugging / programmatic use.
- CsvSchemaFormatter: [N]{col:type,col:type} declaration + CSV
rows. Steals TOON's row-count-and-shape declaration without
adopting TOON's bespoke escaping. CSV is the format LLMs are
strongest at — every model has seen millions of examples in
training. >30% smaller than raw JSON serialization on tabular
fixtures.
- Wiring (`crusher.rs`, `builder.rs`): SmartCrusher gains an optional
compaction stage. Builder methods with_compaction(stage) and
with_default_compaction() opt in. CrushArrayResult gets two new
fields (compacted, compaction_kind) populated only when the stage
runs. strategy_info becomes compaction kind when compaction won.
# Why this design
- Three-trait extension surface preserved. PR1 added Constraint /
Observer / Scorer; PR2 adds Formatter as the fourth pluggable
seam. Enterprise plug-ins land cleanly without forking core.
- Empty default builder rule held. SmartCrusherBuilder::new() still
produces a no-compaction crusher. with_default_compaction() is
the explicit OSS preset. No silent fallbacks.
- Recursive IR was the unlock. A flat table-of-scalars IR would have
collapsed the moment a cell held nested JSON. Making
CellValue::Nested hold another Compaction made stringified-JSON
parsing + heterogeneous bucketing + opaque substitution all share
one renderer pass.
- CCR substitution for opaque cells. Strings classified as
base64/HTML/long-opaque become structured markers keyed by 12-char
SHA-256 prefix. The full bytes round-trip via the CCR store (PyO3
bridge owns actual storage; this PR emits the marker and computes
the hash).
# Tests
- 60 new unit tests across IR / classifier / compactor / formatter /
wiring (448 total in headroom-core, was 388).
- 17/17 SmartCrusher parity fixtures byte-equal — default-config
path completely unchanged.
- 21/21 Python parity tests pass via PyO3 bridge.
- make ci-precheck green: ruff, mypy, cargo fmt/clippy/test
(1.95.0), commitlint.
# Deferred to follow-up PRs
- ToonFormatter (small; ship after eval harness compares formats)
- Diff/code detection in cells → routes to DiffCompressor /
CodeCompressor (coupled to ContentRouter Phase 4)
- Budget-aware row dropping (Constraint-respecting) when rendered
size exceeds budget
- Format A/B eval harness
- ContentRouter unification (Phase 4)
Modules: crates/headroom-core/src/transforms/smart_crusher/compaction/*, builder.rs, crusher.rs, mod.rs
The cosign signing step passed bake metadata via env var:
env:
BAKE_META: ${{ steps.bake.outputs.metadata }}
run: echo "$BAKE_META" | jq ...
For large bake targets (code-nonroot, runtime-code-nonroot) the
metadata JSON is large enough that combined argv+env at bash spawn
exceeds Linux ARG_MAX (~128 KiB on ubuntu-latest), so bash dies with
E2BIG before the script even runs.
Switch to writing metadata into a heredoc-backed temp file, then read
it via jq file input. Heredocs put the JSON in the script body itself,
which bash reads from a temp file (no ARG_MAX limit), bypassing the
env-size ceiling entirely.
Module: .github/workflows/docker.yml
