Replaces the ICM-based interceptor on /v1/messages and adds OpenAI
/v1/chat/completions and /v1/responses arms. The proxy now follows
exactly one rule, applied per-request:
Find the last user message. Freeze every byte before it. In the
new turn, walk for tool outputs and route them through the
CompressionPipeline (Diff/Log/Json offloads + JsonMinifier/
LogTemplate reformats). Everything else passes through.
Why this design:
- Provider prefix caches (Anthropic cache_control, OpenAI auto
prefix, OpenAI prompt_cache_key) are positional. We never modify
prefix bytes, so cache hits are preserved 100%.
- User text, system prompts, assistant text are never touched —
accuracy preserved.
- Tool outputs (file reads, search results, build logs, diffs)
are where the tokens are. The CompressionPipeline already in
headroom-core compresses them with CCR backup.
What this PR removes:
- ICM (IntelligentContextManager) is no longer wired into the
proxy. It's still in headroom-core for future use, but the
proxy crate doesn't import it. ICM message-dropping risks
cache busts that token mode rules out by construction.
What this PR adds (compression module rewrite):
- compression/walker.rs — compress_blob() shared helper
- compression/pipeline.rs — build CompressionPipeline + CcrStore
once at startup
- compression/anthropic.rs — token-mode walker (tool_result blocks)
- compression/openai.rs — token-mode walker (role:tool messages)
- compression/responses.rs — token-mode walker (function_call_output
items; previous_response_id skips)
- compression/mod.rs — endpoint classifier + dispatch
Per-shape rules documented per file. Highlights:
- Anthropic: walk tool_result blocks (string OR list); compress
text inside list-shaped blocks; preserve images
- OpenAI chat: only role:tool with string content
- OpenAI responses: only function_call_output with string output;
reasoning items, *_call items, image_generation_call all
preserved verbatim; previous_response_id triggers full
passthrough (server holds the conversation)
Tests:
- Unit (32 in compression module + 3 walker tests)
- Mock integration (8 tests, including prefix-byte-identical
assertion across Anthropic / OpenAI / Responses)
- Real OpenAI e2e (3 tests, gated on HEADROOM_E2E=1):
* /v1/chat/completions tool message: 70 prompt tokens received
(compressed from a ~120KB / ~30K-token raw log payload)
* /v1/responses function_call_output: 68 input tokens
(same payload, same compression)
* Prefix cache preservation: 3,456 of 3,518 tokens cached
on the second turn (98%) — proves byte-stable prefix
Verification:
- cargo test --workspace -> 904 passed, 0 failed
- cargo clippy --workspace --all-targets -- -D warnings -> clean
- cargo fmt --check -> clean
- HEADROOM_E2E=1 cargo test --test e2e_token_mode -> 3/3 pass
- make ci-precheck -> green
Adds an opt-in compression interceptor that buffers Anthropic
/v1/messages requests, runs IntelligentContextManager over the
messages array, and forwards the (possibly trimmed) body upstream.
All other paths, methods, and content-types stay on the original
streaming passthrough — so existing operators see zero change.
Behaviour gates ALL must be true to buffer + compress:
- --compression flag (or HEADROOM_PROXY_COMPRESSION=1)
- method == POST
- path == /v1/messages
- Content-Type: application/json
- ICM constructed successfully at startup
Falls through to streaming on any failure: parse, missing fields,
unknown model, body-too-large. Compression must never break a
request — that's the safety contract.
Model context windows come from a vendored LiteLLM snapshot at
crates/headroom-proxy/data/model_prices_and_context_window.json
parsed once into an OnceLock<HashMap>. Refresh via
scripts/refresh_model_limits.sh. Rationale documented inline:
hardcoded tables silently rot; LiteLLM is the canonical source
the entire LLM-tooling ecosystem relies on.
New tests:
- 16 unit tests across compression::{anthropic, icm, model_limits}
- 5 integration tests: off-passthrough, on-short-passthrough,
on-oversized-trim, on-non-json-skip, on-non-llm-path-skip
Verification:
- cargo test --workspace -> 884 passed, 0 failed
- cargo clippy --workspace -- -D warnings -> clean
- cargo fmt --check -> clean
The @1.95.0 git ref of dtolnay/rust-toolchain shipped action code
that errors on ubuntu-latest with:
failed to install component: 'clippy-preview-x86_64-unknown-linux-gnu',
detected conflict: 'bin/cargo-clippy'
The runner's pre-installed Rust ships cargo-clippy at $HOME/.cargo/bin,
and the older action code's rustup invocation hits a path conflict
when adding the clippy-preview component for 1.95.0.
The @stable ref of the action has the fix; pass toolchain: 1.95.0 as
input so the version stays pinned. rust-toolchain.toml continues to
be the source of truth for the version (used by cargo's
auto-detection); this keeps the action's install in sync.
Replaces PR1's lossless/lossy split with ReformatTransform (pack
denser, no info lost) and OffloadTransform (drop bytes, CCR-stash
original via required cache_key). With CCR every transform is
information-preserving end-to-end, so the lossless/lossy distinction
misnamed the architecture.
OffloadTransform carries a cheap, structural estimate_bloat() method
scoped to its domain — generic byte-redundancy heuristics miss domain
semantics. The orchestrator runs reformat phase + per-offload bloat
estimation in parallel via rayon::join + par_iter, then runs offload
iff bloat clears threshold OR reformat underwhelmed.
Transforms shipped:
REFORMATS (lossless):
- JsonMinifier: serde_json round-trip whitespace stripping.
- LogTemplate: Drain-inspired order-preserving template miner.
Collapses consecutive runs of same-template lines into
[Template Tn: ...] (Nx) + variant table. Win comes from emitting
the constant-token prefix once instead of N times. Lossless: every
original line reconstructible from template + variants.
OFFLOADS (drop bytes, stash original via CCR):
- LogOffload: wraps existing LogCompressor; bloat = repetition x
uniqueness_weight + dilution x priority_dilution_weight.
- DiffOffload: wraps existing DiffCompressor; bloat = context-to-
change ratio. Bug-fix-on-port — persists original under the
cache_key the parity-bound DiffCompressor mints (closes a leak).
- DiffNoise: drops lockfile hunks (Cargo.lock, package-lock.json,
yarn.lock, etc., suffix list configurable in TOML) and
whitespace-only hunks. Stashes original via CCR for retrieval.
Search offload exists but is not in default re-exports — modern
agents (Claude Code, Codex) use scoped rg/grep, the marginal value
didn't justify default registration. Reach via the explicit module
path if opting in.
JSON Offload is intentionally absent from this PR — already lives at
SmartCrusher; Phase 3g PR3 wraps it in the OffloadTransform contract.
Thresholds and weights live in config/pipeline.toml, embedded via
include_str!; PipelineConfig::from_toml_str loads runtime overrides.
98 new pipeline tests; full headroom-core suite (714) and workspace
tests green; cargo fmt clean. No regex, per project convention.
Ports `headroom.transforms.log_compressor` to Rust. The biggest-by-
impact remaining compressor port: build/test logs are where the
10-50x compression wins live.
* Stack-trace state machine: per-flavor dispatcher (Python Traceback,
JS, Java, Rust error, Go); each flavor has its own termination
rule. Python terminated on any blank line, dropping mid-trace
lines from chained-exception traces.
* Conservative dedupe: preserves message prefix (everything before
first `:` or `=`); only trailing region is tokenised. Python's
blanket normalisation collapsed segfaults at different addresses.
* Loud CCR failures: `tracing::warn!` + `logger.warning` instead of
bare `except: pass`.
* `LogLevel::FAIL` documented as cosmetic-equivalent to ERROR.
Same shape as search_compressor port. Rust `LogCompressor`
orchestrates format detect -> classify -> score -> select ->
format -> CCR. Inline static-table format detector (YAGNI),
aho-corasick level classifier with word-boundary post-filter
(`signals::keyword_detector` technique), hand-rolled per-flavor
stack-trace state machine. `signals::LineImportanceDetector` NOT
consumed -- log levels are structural, not prose-style importance.
`headroom.transforms.log_compressor` becomes a thin shim:
`compress()` delegates to Rust end-to-end; internal helpers
preserved for the existing 50-test surface. Two existing tests
updated for new dedupe semantics + new compress orchestration.
* 17 Rust unit tests
* 50 Python tests pass
* `make ci-precheck` clean
Establish `crates/headroom-core/src/signals/` as a top-level module
holding cross-cutting detection traits. Phase 3e.1 ports
`error_detection.py` to a `LineImportanceDetector` trait + a
`Tiered<T>` combinator + a single concrete `KeywordDetector` impl
backed by aho-corasick. Three traits at three granularities are
sketched (line / blob / item); only line-importance is implemented
today.
Two bug fixes from the Python source bake into both the Rust impl
and the Python regex shim:
1. `ERROR_KEYWORDS` listed `timeout|abort|denied|rejected` but
`ERROR_PATTERN` regex omitted them. Lines like `"Connection
timeout"` were silently neutral despite the keyword being canonical.
Both surfaces now flag them.
2. `SECURITY_KEYWORDS` carried `token`, which false-positived on
every reference to LLM tokens (`input_tokens`, `tokens_saved`, ...)
in our own product. Dropped from the security set.
The Python `error_detection.py` shim now reflects keyword data out of
Rust via `keyword_registry_snapshot()` and recompiles the legacy
`re.Pattern` objects on the fly. Existing callers (text_compressor,
search_compressor, intelligent_context) continue to import the same
names with no source changes; caller migration to the trait API
happens in their own port PRs.
The trait architecture is the seam where a future ML detector slots
in without touching `KeywordDetector` or any caller. The canonical
extension is documented in `signals/README.md` as a classifier head
on the existing `bge-small-en-v1.5` embedder loaded by
`relevance::EmbeddingScorer` -- 384-dim -> 4-class softmax,
~1.5 KB head, ~1 ms inference, no extra model file. Two alternatives
(distilled tinyBERT in ONNX, logistic regression on lexical
features) are kept open in case BGE-head underfits.
Per the no-silent-fallbacks rule: only `KeywordDetector` lands as a
concrete impl. No NoOp, no MockDetector, no stub-ML -- those will
arrive with their real implementations.
Phase 3g (Compression Pipeline Formalization, issue #315) is queued
as the cross-cutting follow-up that will make lossless-then-lossy-
then-CCR ordering an explicit, observable architecture rather than
implicit per-compressor logic. Trait shapes there will reuse the
signals primitive landed in this PR.
Re-lands two audit fixes that were marked "merged" on GitHub but never
reached main: squash-merging the parent stack changed its commit SHA,
which silently dropped the contents of the stacked PRs (#301, #305).
Single PR this time — no stacking risk.
What lands:
1. **`enable_ccr_marker` Rust gate** — new field on `SmartCrusherConfig`
(default `true`). `crush_array` checks it before emitting the
`<<ccr:HASH>>` marker text and the CCR store write. PyO3 surface +
parity-fixture tolerance updated; recorded fixtures predate the
field and inherit the `true` default.
2. **Python shim collapses both flags to the gate** — both
`ccr_config.enabled=False` and `ccr_config.inject_retrieval_marker
=False` now flip the Rust gate off. Storing a payload nothing in
the prompt can reference is pointless, and storing under
`enabled=False` would be a surprise side effect the user
explicitly opted out of.
3. **Custom `scorer` / `relevance_config` fails loud** — replaces the
prior WARNING-and-drop. Silently dropping a user-supplied scorer
is a textbook silent fallback. `NotImplementedError` instead.
Verified zero production callers pass these args; full plumbing
arrives with Stage-3c.2's relevance-crate Python bridge.
Tests:
- 2 new Rust unit tests in `crusher.rs::tests`
- 6 new Python tests in `test_smart_crusher_toin_attachment.py`
(3 CCR marker-knob behaviors + 3 scorer fail-loud)
- Removed `test_ccr_inject_marker_false_logs_warning` (the WARNING is
gone now that the flag is honored)
- `make ci-precheck` green; eval suite + observability tests run
twice consecutively to verify no TOIN file pollution leaks into the
regular+coverage double-run on Python 3.11
RUST_DEV.md audit table reflects both gaps closed.
The PR8 marker injection appends a sentinel object
{"_ccr_dropped": "<<ccr:HASH N_rows_offloaded>>"} to the kept-items
array on the lossy path so the LLM sees the retrieval pointer in the
prompt. Tests that iterate compressed arrays via subscript access
(e["level"], r["status"], i["labels"], m["text"]) hit KeyError on
the sentinel because it doesn't share the record schema.
Same root cause as the test_quality_retention fixes in PR8 -- these
integration tests were left out of that pass.
Ship a public helper headroom.transforms.smart_crusher.strip_ccr_sentinels
so tests can use it cleanly: `for e in strip_ccr_sentinels(entries):`
and production callers iterating compressed output get a single
canonical filter instead of inlining the _ccr_dropped check.
The 7 previously-failing tests in PR #292 CI now pass:
- langchain test_100_percent_errors_preserved_logs
- langchain test_errors_preserved_with_many_errors
- langchain test_search_results_with_query_term
- mcp test_all_log_errors_preserved
- mcp test_slack_significant_compression_with_content
- mcp test_database_error_status_preserved
- mcp test_github_bugs_partial_preservation
753 tests across the integration + transforms + retention suites pass
locally. Plugin manifests auto-bumped 0.13.3 -> 0.13.4 by the
sync-plugin-versions hook (unrelated to this fix).
The action was set to @stable, which installs whatever the latest
stable is (1.95.0 right now). Then maturin invokes cargo, which reads
rust-toolchain.toml and re-resolves to "1.95.0 + clippy + rustfmt".
rustup treats stable and 1.95.0 as distinct toolchain identities and
refuses the second install with:
failed to install component 'clippy-preview-x86_64-unknown-linux-gnu',
detected conflict: 'bin/cargo-clippy'
This was intermittent across the matrix (only test (3.10) tripped on
the most recent run; others got lucky on cache state). Pinning the
action ref to 1.95.0 makes both sides ask for the exact same toolchain
identity, so the second install is a no-op and the conflict can't fire.
Bump procedure stays the same: when rust-toolchain.toml's channel
changes, update these refs in lock-step.
Plugin manifests auto-bumped 0.11.0 -> 0.13.2 by sync-plugin-versions
hook (unrelated to the workflow fix).
Stage 3c.2 PR2. Adds an opt-in compaction stage that runs BEFORE the
existing lossy pipeline. When configured, it tries to losslessly
re-shape arrays of objects into a recursive Compaction IR and renders
that to bytes via a pluggable Formatter trait. When not configured
(default OSS), behavior is byte-equal with the pre-PR2 path — all 17
SmartCrusher parity fixtures stay green.
# What lands
- Recursive Compaction IR (`compaction/ir.rs`): Table / Buckets /
OpaqueRef / Untouched. CellValue can hold a nested Compaction so
multi-level cases (stringified-JSON inside cells, heterogeneous
arrays bucketed by discriminator, opaque blobs CCR-substituted)
share one tree shape.
- Cell classifier (`compaction/classifier.rs`): per-cell decision —
Scalar / JsonObject / JsonArray / StringifiedJson(parsed) /
Opaque(kind). Conservative: in doubt, return Scalar.
- TabularCompactor (`compaction/compactor.rs`): array → IR. Handles
uniform-nested flattening into dotted columns ("meta.region",
"meta.tier"), stringified-JSON parsing + recursion, opaque-blob
CCR-substitution (12-char SHA-256 prefix), and heterogeneous
bucketing by discriminator. Falls through to a sparse Table when
no clean discriminator exists, so we always do better than the
lossy path for object arrays.
- Formatter trait (`compaction/formatter.rs`) + two impls:
- JsonFormatter: structured JSON for debugging / programmatic use.
- CsvSchemaFormatter: [N]{col:type,col:type} declaration + CSV
rows. Steals TOON's row-count-and-shape declaration without
adopting TOON's bespoke escaping. CSV is the format LLMs are
strongest at — every model has seen millions of examples in
training. >30% smaller than raw JSON serialization on tabular
fixtures.
- Wiring (`crusher.rs`, `builder.rs`): SmartCrusher gains an optional
compaction stage. Builder methods with_compaction(stage) and
with_default_compaction() opt in. CrushArrayResult gets two new
fields (compacted, compaction_kind) populated only when the stage
runs. strategy_info becomes compaction kind when compaction won.
# Why this design
- Three-trait extension surface preserved. PR1 added Constraint /
Observer / Scorer; PR2 adds Formatter as the fourth pluggable
seam. Enterprise plug-ins land cleanly without forking core.
- Empty default builder rule held. SmartCrusherBuilder::new() still
produces a no-compaction crusher. with_default_compaction() is
the explicit OSS preset. No silent fallbacks.
- Recursive IR was the unlock. A flat table-of-scalars IR would have
collapsed the moment a cell held nested JSON. Making
CellValue::Nested hold another Compaction made stringified-JSON
parsing + heterogeneous bucketing + opaque substitution all share
one renderer pass.
- CCR substitution for opaque cells. Strings classified as
base64/HTML/long-opaque become structured markers keyed by 12-char
SHA-256 prefix. The full bytes round-trip via the CCR store (PyO3
bridge owns actual storage; this PR emits the marker and computes
the hash).
# Tests
- 60 new unit tests across IR / classifier / compactor / formatter /
wiring (448 total in headroom-core, was 388).
- 17/17 SmartCrusher parity fixtures byte-equal — default-config
path completely unchanged.
- 21/21 Python parity tests pass via PyO3 bridge.
- make ci-precheck green: ruff, mypy, cargo fmt/clippy/test
(1.95.0), commitlint.
# Deferred to follow-up PRs
- ToonFormatter (small; ship after eval harness compares formats)
- Diff/code detection in cells → routes to DiffCompressor /
CodeCompressor (coupled to ContentRouter Phase 4)
- Budget-aware row dropping (Constraint-respecting) when rendered
size exceeds budget
- Format A/B eval harness
- ContentRouter unification (Phase 4)
Modules: crates/headroom-core/src/transforms/smart_crusher/compaction/*, builder.rs, crusher.rs, mod.rs
The cosign signing step passed bake metadata via env var:
env:
BAKE_META: ${{ steps.bake.outputs.metadata }}
run: echo "$BAKE_META" | jq ...
For large bake targets (code-nonroot, runtime-code-nonroot) the
metadata JSON is large enough that combined argv+env at bash spawn
exceeds Linux ARG_MAX (~128 KiB on ubuntu-latest), so bash dies with
E2BIG before the script even runs.
Switch to writing metadata into a heredoc-backed temp file, then read
it via jq file input. Heredocs put the JSON in the script body itself,
which bash reads from a temp file (no ARG_MAX limit), bypassing the
env-size ceiling entirely.
Module: .github/workflows/docker.yml
Five gates broke on the 2026-04-27 push of the smart_crusher branch.
Each is fixed below; the second half adds a `make ci-precheck` target
(plus an installable git pre-push hook) so the same dance never happens
again.
Failures fixed:
1. cargo fmt — 22 files had formatting drift introduced over the
stage 3c.1 work. `cargo fmt --all` reformatted them; no semantic
changes. `cargo test --workspace` still green (388 + supporting).
2. wheels job (macOS x86_64) — `fastembed -> ort -> ort-sys` does not
publish prebuilt ONNX Runtime binaries for `x86_64-apple-darwin`.
Removed that target from `.github/workflows/rust.yml`'s wheels
matrix. Apple Silicon (`aarch64-apple-darwin`) covers macOS
distribution; Intel macOS users can build from source. The matrix
now has 2 targets: linux x86_64 + macOS aarch64.
3. test-extras (relevance.py) — `tests/test_relevance.py::TestSmartCrusherIntegration`
constructs a `SmartCrusher`, which hard-imports `headroom._core`
since the python implementation was retired in stage 3c.1b. The
test-extras job didn't build the rust extension. Added the same
`maturin build + symlink` block the main `test` job uses.
4. smoke-test (eval.yml) — same root cause:
`compression_only.evaluate_ccr_lossless` instantiates a SmartCrusher.
Same fix: build the rust extension before the smoke test runs.
5. commitlint — three rules tripped:
- `subject-case` rejects PascalCase identifiers in subjects, but
the project deliberately names classes (SmartCrusher, HfTokenizer,
ContentRouter, DiffCompressor) in commit subjects. Disabled.
- `footer-leading-blank` is a warning that the wagoid action turns
into a CI failure; lines like `Module: foo.rs` in our bodies
match the conventional footer pattern and trip it. Disabled.
- `type-enum` doesn't include `parity`, but the project ships
parity-test infrastructure as its own concern (separate from
`test:`); added `parity` to the allowed types.
Pre-push verification — the prevention half:
`make ci-precheck` runs all of the above CI gates locally:
- `ci-precheck-rust`: cargo fmt --check + clippy + test --workspace.
- `ci-precheck-python`: builds the rust extension via maturin, then
runs the smart_crusher-affected python test files (185 tests across
test_transforms/, test_relevance*, test_ccr, test_acceptance,
test_critical_fixes, test_quality_retention).
- `ci-precheck-commitlint`: `npx commitlint --from origin/main --to
HEAD` against the same config CI uses. Skipped silently if npx is
not on PATH (install Node 18+ to enable).
`make install-git-hooks` (or `scripts/install-git-hooks.sh`) installs
a git pre-push hook that runs `make ci-precheck` automatically.
Bypass with `--no-verify` only when truly needed.
When new CI gates land in `.github/workflows/`, mirror them into a
`make ci-precheck-*` target. The Makefile is the local mirror of the
CI configuration; keeping them in sync is a load-bearing invariant.
Verification: `make ci-precheck` runs green on this commit.
Code review (`/code-review` on commit `d219bee`) caught one critical
bug, two important parity gaps, and a few quality nits. Fixed all of
them; all 135 unit tests pass; diff_compressor parity harness
unaffected (27/27 still matched).
# Critical fix — `hash_field_name` truncation length
Rust truncated SHA-256 to **16** hex chars; Python uses **8** (per
`smart_crusher.py:177`: `hashlib.sha256(...).hexdigest()[:8]`). 16-char
hashes would never collide with TOIN's 8-char `preserve_fields`,
silently disabling the entire `use_feedback_hints` cache lookup path.
Fix: `hex[..8]` instead of `hex[..16]`. Three pinning tests re-verified
against actual Python reference output. Doc comment now warns
explicitly that the length must match Python or TOIN lookups silently miss.
# Important fix — `python_int_parse` mirrors Python's `int()` semantics
`statistics.rs::detect_sequential_pattern` previously called
`s.parse::<i64>()`. Python's `int()` differs in three ways that affect
realistic payloads:
- strips ASCII whitespace (Rust's `parse` rejects)
- accepts leading `+` (Rust accepts; same)
- accepts PEP 515 underscores like `"3_000"` (Rust rejects)
A field with `[" 1 ", " 2 ", " 3 ", "4", "5"]` would parse all five
in Python (sequential = True) but only one in Rust (`nums.len() < 5`
→ False). Silent parity break.
Fix: new private `python_int_parse` helper that strips whitespace,
handles underscore separators, and rejects edge cases Python rejects.
Six new tests pin the behavior.
# Important fix — `python_repr` for `item_matches_anchors`
Python compares anchors via `anchor in str(item).lower()`. We were
using `serde_json::to_string(&item).to_lowercase()`, which differs in
three ways that affect substring matching:
- quote chars (`'` vs `"`)
- bool/null literals (`True`/`False`/`None` vs `true`/`false`/`null`)
- spacing (`key: value, ...` vs `key:value,...`)
Anchor `"none"` would match Python form but not JSON. Inverse for
`"null"`. Real divergence.
Fix: new private `python_repr` walks `serde_json::Value` and emits
Python-equivalent form. Plus enable `serde_json/preserve_order` at
workspace level so `Value::Object` preserves JSON parse order
(matching Python `dict` since 3.7).
# Suggestion fixes
- Classifier comment for `[True, False, 1] -> MIXED_ARRAY` now walks
both Python and Rust paths step by step.
- `ArrayAnalysis::field_stats` doc notes the BTreeMap vs Python-dict
order nuance for the analyzer port to resolve.
- Added regression tests for "all unparseable strings", "single int
among strings", fractional-step sequential, and the email-typo
pattern.
# Build / test
- `cargo build -p headroom-core` clean.
- `cargo clippy -p headroom-core -- -D warnings` clean.
- 135 unit tests in `headroom-core`, all passing (was 55).
- `cargo run -p headroom-parity run` — diff_compressor 27/27 still matched.
Stage 3c.1 — like-for-like Rust port of `headroom/transforms/smart_crusher.py`.
This commit lays the foundation: module layout, configuration, foundational
data types, and the simpler helpers (classification, hashing, anchors,
basic statistics). Subsequent commits add the analyzer, crushers, plan
execution, and the orchestrator.
# What's in this commit
`crates/headroom-core/src/transforms/smart_crusher/`:
- `mod.rs` — module entry, public re-exports, port narrative.
- `classifier.rs` — `classify_array` / `ArrayType` (dict/string/number/
bool/nested/mixed/empty). Direct port of `_classify_array`.
- `config.rs` — `SmartCrusherConfig` with defaults pinned to Python
byte-for-byte.
- `hashing.rs` — `hash_field_name` (SHA-256 truncated to 16 hex chars),
matches `hashlib.sha256(name.encode()).hexdigest()[:16]` exactly.
- `statistics.rs` — `is_uuid_format`, `calculate_string_entropy`,
`detect_sequential_pattern` (with **BUG #2 fix** — see below).
- `anchors.rs` — `extract_query_anchors`, `item_matches_anchors`. Five
regex patterns ported via `std::sync::LazyLock`.
- `types.rs` — `CompressionStrategy`, `FieldStats`, `CrushabilityAnalysis`,
`ArrayAnalysis`, `CompressionPlan`, `CrushResult`. Field-by-field
mirror of the Python @dataclasses so the PyO3 bridge in 3c.1b can
reconstruct them without manual translators.
# Bug #2 fixed in this commit (Python fix lands later in same PR)
`smart_crusher.py:444-448` — `_detect_sequential_pattern` calls
`int(string_value)` and silently strips zero-padding, so padded string
IDs like `["001", "002", ..., "100"]` get misclassified as a sequential
numeric pattern. Fix: track whether each parsed numeric value
originated as a string. If EVERY parsed value was a string, refuse to
flag as sequential. Mixed numeric+string fields still detect
correctly because the unambiguous numerics dominate. Test:
`bug2_zero_padded_strings_no_longer_misclassified`.
# What's NOT in this commit (subsequent commits)
- `SmartAnalyzer` — `analyze_array`, `_analyze_field`, `_detect_change_points`,
`_detect_pattern`, `_detect_temporal_field`, `analyze_crushability`,
`_select_strategy`, `_estimate_reduction`.
- The five array crushers (`_crush_array`, `_crush_string_array`,
`_crush_number_array`, `_crush_mixed_array`, `_crush_object`).
- Planning (`_compute_k_split`, `_create_plan`, `_plan_*` family).
- Orchestration (`_prioritize_indices`, `_deduplicate_indices_by_content`,
`_fill_remaining_slots`).
- `SmartCrusher` orchestrator class itself.
- Parity harness fixtures.
- The remaining 3 Python bug fixes (#1, #3, #4) — landed alongside the
code paths they affect.
# Build / test
- `cargo build -p headroom-core` — clean.
- `cargo clippy -p headroom-core -- -D warnings` — clean.
- 55 new unit tests across the 6 new files, all passing.
Architectural improvements (lossless-first, unified saliency score,
structured CCR markers) are deferred to Stage 3c.2 — see design doc at
`~/Desktop/SmartCrusher-Architecture-Improvements.md`.
The previous version of this step did `python -c "import headroom._core"`
to find the wheel's installed `.so` path. That failed in CI:
ModuleNotFoundError: No module named 'headroom._core'
— exactly the chicken-and-egg this step exists to fix. The editable
install (`pip install -e .`) puts the in-tree `headroom/` source dir
ahead of site-packages on `sys.path`. So `import headroom` finds the
in-tree dir (which doesn't yet have the `.so`), then
`import headroom._core` fails to find the submodule. The symlink we're
about to create is what makes the import work — but we can't import
to discover the symlink target before creating it.
Locate the `.so` via filesystem instead: read site-packages from
`site.getsitepackages()[0]`, glob for `_core.cpython-*.so` under
`<site-packages>/headroom/`, and symlink that into the in-tree dir.
The smoke test (`from headroom._core import DiffCompressor`) runs
*after* the symlink and confirms end-to-end resolution.
Also added `set -euo pipefail` and a sanity check with `ls -la` of the
site-packages dir if the glob comes up empty, so future failures
diagnose themselves.
`maturin develop` requires a virtualenv (it errors with "Couldn't find a
virtualenv or conda environment"). CI's setup-python provides a bare
system Python without a venv, so the dev script's build path doesn't
work there.
This switches CI to build a release wheel via `maturin build` and
install it with `pip install --force-reinstall --no-deps`. Then symlink
the installed `.so` into the in-tree `headroom/` package so the
editable install resolves `import headroom._core` past the source-dir
shadowing of site-packages.
`scripts/build_rust_extension.sh` is unchanged — it stays optimized for
local dev (where there IS a venv).
The python `DiffCompressor` was retired in this PR's stage-3b commit;
the public class now delegates to `headroom._core` (built from
`crates/headroom-py`). Without the wheel installed in CI, every test
that constructs a `DiffCompressor` fails with `ModuleNotFoundError:
No module named 'headroom._core'`.
This adds a build step to the main `test` job in `ci.yml` that:
1. Installs the stable Rust toolchain (`dtolnay/rust-toolchain@stable`).
2. Caches the cargo registry and build output (`Swatinem/rust-cache@v2`).
3. Installs maturin.
4. Runs `scripts/build_rust_extension.sh`, which calls `maturin develop`
and symlinks the built `.so` into the in-tree `headroom/` package
so the editable install resolves `import headroom._core`.
Only the main `test` job needs this — `test-extras` and `test-agno`
run narrow subsets that don't construct `DiffCompressor`. The existing
`rust.yml` workflow continues to handle wheel builds for distribution
and `cargo test` for the Rust workspace.
GitHub Actions deprecated the macos-13 runner label. The validate-workflows
actionlint step in CI fails because macos-13 is no longer in the available
labels list. macos-15-intel is the current x86_64 macOS runner.
(Bumped from macos-14 to macos-15 for arm64 was unnecessary; macos-14 is
still valid and we keep it for cache-warmth.)
cargo fmt --check failed in CI: import order in proxy.rs (cfg(test)
attributes before/after non-attr imports) and a few line-wrapping
nits in e2e_real.rs. Ran cargo fmt --all to fix.
maturin-action@v1 does not have a 'manifest-path' input — the action
warned 'Unexpected input(s) manifest-path' and proceeded to invoke
maturin from the repo root, which sees the workspace Cargo.toml with
no [package] section and bails. Move -m crates/headroom-py/Cargo.toml
back inside the 'args' string.
Adds 0.10.7-ab46594 (root) and 0.10.7-<variant>-<sha> (variants) so
images can be referenced by an exact version+commit pair without
relying on the moving variant or :latest tags.
CodeQL alert #61 (CWE-275, actions/missing-workflow-permissions):
add explicit `permissions: contents: read` to the rust workflow root.
Defaults the GITHUB_TOKEN to read-only across all jobs, so even if the
repo policy changes, this workflow stays at least-privilege. No job in
this workflow needs write — wheels/audit/parity all read-only.
Add real end-to-end test suite at tests/e2e_real.rs gated behind
HEADROOM_E2E=1. Spawns the actual Python Headroom proxy as a subprocess,
runs the Rust proxy in-process in front of it, and exercises:
- health endpoints across the full chain
- Anthropic non-streaming (real API call)
- Anthropic streaming SSE (real API call) with chunk-level validation
- OpenAI non-streaming (real API call)
- X-Request-Id generation and pass-through
Adds tokio-process feature for Command/Child usage. Loads .env at the
repo root for API keys (does not log values). Tests skip cleanly when
HEADROOM_E2E is unset, so cargo test stays fast.
When the docker workflow is triggered directly by release.published
(rather than via workflow_call from the Release parent), inputs.enable_ref_tags
is null and produced an empty enable= attribute that the metadata-action
rejected. Default to true on non-release triggers and skip ref/pr tags
on release events where they don't apply anyway.
- Replace full-sha image tags with type=sha,format=short (7-char) so the
primary package versions list stops accumulating long sha-only entries.
- Route cosign signatures into a sibling GHCR package via
COSIGN_REPOSITORY=<image>-signatures, so the main image's package
version list stays clean. GHCR does not yet implement the OCI 1.1
Distribution Referrers API (community discussion #163029, June 2025),
so legacy signature mode is used here -- OCI 1.1 mode would force the
signature manifest's subject into the same repo as the image and
override COSIGN_REPOSITORY. Verifiers must export the same
COSIGN_REPOSITORY value when running 'cosign verify'.
- Add a promote-latest job that runs after the variant matrix and
re-pushes the :latest tag pointing at the root image with a unique
index annotation. This forces a fresh manifest digest, generating a
new GHCR package version with current timestamp so :latest sits at
the top of the version listing instead of whichever variant happened
to finish last.
maturin>=1.5 requires -m to point to Cargo.toml, not pyproject.toml.
Fixes wheel build job failure in CI (all three matrix targets).
Also switches to manifest-path: action param for cleaner workflow syntax.
Applies same fix to Makefile build-wheel and develop targets.
Two fixes for the init-native-e2e matrix surfaced on PR #256:
1. Composite action installed `headroom` without extras, but
`headroom/cli/__init__.py` eagerly imports `proxy.server` (via
`cli/proxy.py`), which requires `fastapi`. All 6 POSIX jobs hit
`ModuleNotFoundError: No module named 'fastapi'` before `init` ran.
Fix: install `-e .[proxy]` to match the Docker e2e image.
2. On Windows, shims are `.cmd` files and Git Bash's `which` cannot
resolve them (exact-match only). Python's `shutil.which` (used by
`headroom init`) honors PATHEXT and finds the shim fine, but the
pre-flight `which` step failed first. Fix: use `Get-Command` via
`pwsh` for the Windows verification step.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Existing Docker init-e2e runs on ubuntu only. Platform-specific bugs
(Windows path separators in written hook commands, PowerShell-vs-bash
matcher strings, macOS keychain prompts, shutil.which PATHEXT quirks)
slip past it. Add a matrix workflow that drops a noop shim for each
target agent and runs ``headroom init -g <target>`` on each of the
three supported OSes, then asserts the settings file was written to
the platform-correct location.
Matrix: [ubuntu-latest, macos-latest, windows-latest] x [claude,
codex, copilot]. ``openclaw`` is excluded because it delegates to
``headroom wrap openclaw`` which needs a real OpenClaw CLI and can't
be stubbed with a noop shim; the Docker suite already covers its
negative path.
Common setup (Python install, editable headroom install, shim drop,
PATH wiring) is factored into a composite action at
.github/actions/headroom-e2e-setup so follow-up per-command workflows
(install-native-e2e, wrap-native-e2e) can be near-copies that only
supply their matrix and assertion blocks. The composite action uses
the cross-platform shim scripts from e2e/_lib/make_shim.{sh,ps1} that
landed with the harness refactor.
Scoped trigger: pull_request touching init code OR the harness, plus
pushes to main and manual dispatch. This avoids burning CI minutes on
every push to unrelated feature branches while still gating every PR
that could regress init behavior.
Not verified locally: Windows runner behavior. Reviewer should watch
the first matrix run on PR.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The sync-plugin-versions pre-commit hook recomputes plugin semver from
git history + conventional-commits bump rules. Adding the feat(init)
-v/--verbose commit triggers a minor bump (0.11.4 -> 0.12.0). Land
that bump as its own chore so subsequent test/ci commits on this
branch aren't flagged as drift by the hook.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Running the repo's sync-plugin-versions pre-commit hook updates
.claude-plugin/marketplace.json, .github/plugin/marketplace.json, and
the two headroom-agent-hooks plugin.json manifests to the release
semver computed from git tags (0.11.4 at time of branch). Landing this
first keeps subsequent commits on this branch from tripping the
hook's auto-fix path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`headroom wrap codex` injects a `model_provider = "headroom"` block
plus a `[model_providers.headroom]` table into `~/.codex/config.toml`
so Codex routes both HTTP and WebSocket traffic through the proxy. The
matching `unwrap codex` subcommand did not exist, so the injected
block stayed in `config.toml` forever — the moment the proxy stopped,
Codex (CLI and macOS app) started erroring with
`Missing environment variable: OPENAI_API_KEY`, and users had to hand-
edit the file to recover.
Fix:
* `_inject_codex_provider_config` now snapshots the pre-wrap file to
`~/.codex/config.toml.headroom-backup` before the first modification
and leaves that snapshot untouched on subsequent wrap runs. The
injection is also rewritten to use two self-contained marker-
delimited blocks (top-level key and provider table) so stripping
them never consumes user content that sits between them.
* `_inject_memory_mcp_config` takes the same snapshot, so
`wrap codex --memory` without a full provider injection is still
fully reversible.
* New `_restore_codex_provider_config` helper and `unwrap codex`
click command:
* backup present → restore byte-for-byte and delete the backup;
* backup absent but Headroom block present → strip the block and
keep surrounding user content;
* config contained only Headroom content → remove the file so
Codex falls back to defaults;
* nothing to undo → safe no-op.
Codex is the only wrap target that modifies a persistent user config
file: claude/aider/cursor/copilot all go through env vars or project-
scoped files only, so this bug was unique to Codex.
Tests:
* `tests/test_cli/test_wrap_codex.py` adds 20 new cases covering the
strip/snapshot helpers directly, round-trip idempotency of
wrap → wrap → unwrap, handling of malformed prior configs, and
end-to-end CliRunner invocations of `headroom wrap codex
--prepare-only` / `headroom unwrap codex` against a temp `$HOME`.
* All 153 existing `tests/test_cli/` tests continue to pass.
Plugin manifest versions were re-synced from `pyproject.toml` (0.11.2)
by the `sync-plugin-versions` pre-commit hook; the previous values
(0.10.3) had drifted.
Reported by @raenaryl in Discord on 0.6.3; confirmed still broken on
current `main` (0.11.x).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
