feat(rust): port Kompress ML prose compressor to Rust (parity-only) (#1153)

Adds crates/headroom-core/src/transforms/kompress.rs (672 lines): the Kompress
ML prose compressor ported to Rust, running the ModernBERT tokenizer plus the
kompress-v2-base ONNX model through ort with a cache-only loader that never
touches the network.

Parity-only. Nothing calls it: the only references outside the module are the
pub mod / pub use declarations in transforms/mod.rs. live_zone.rs still carries
TODO(PR-B4), so PlainText dispatch remains a no-op and Python continues to serve
prose compression. The pyo3 bridge is untouched and no Python source changes, so
the new engine is unreachable from the shipped package. #1155 wires it up.

Ships 21 recorded parity fixtures, a KompressComparator in headroom-parity, and
scripts/record_kompress_fixtures.py.

Verified byte-identical to the recorded Python output:

  [kompress] total=21 matched=21 skipped=0 diffed=0

That required ONNX Runtime >= 1.24 (see #2591) — below it ort deadlocks instead
of erroring, which is why these fixtures had never been run. In CI the model is
absent from the HF cache, so the comparator errors and the fixtures report
Skipped rather than hanging.

Also gates the module behind the ml feature, matching magika_detector: kompress.rs
uses ort, which is optional = true, so an unconditional pub mod broke
cargo check --no-default-features (the static-musl path). CI does not catch that
class of break because cargo test --workspace only builds default features.

.gitignore

@@ -38,6 +38,7 @@ scripts/*
 !scripts/audit_wheel_glibc_symbols.py
 !scripts/replay_codex_ws_load.py
 !scripts/export_kompress_v2_onnx.py
+!scripts/record_kompress_fixtures.py
 
 # Rust / Cargo build artifacts
 /target/

crates/headroom-core/Cargo.toml

@@ -137,7 +137,6 @@ redis = { version = "0.27", optional = true, default-features = false }
 # classifier live with the other Phase B/F policy primitives without
 # cycling through the proxy crate. Tiny crate (no I/O, just types).
 http = "1"
-
 # Load ONNX Runtime dynamically on every platform. The alternative,
 # `ort-download-binaries-*`, statically links Microsoft's prebuilt ORT:
 # on Windows it emits DirectML link libs (`DXCORE`, `DXGI`, `D3D12`,
@@ -153,7 +152,10 @@ fastembed = { version = "5", default-features = false, optional = true, features
     "ort-load-dynamic",
     "image-models",
 ] }
-ort = { version = "2.0.0-rc.12", default-features = false, optional = true, features = ["load-dynamic"] }
+# Direct dependency for Kompress inference (`ort::session::Session` /
+# `ort::value::Tensor`). Keep this pinned to the lock entry and optional so
+# `default-features = false` consumers can still build without ONNX Runtime.
+ort = { version = "=2.0.0-rc.12", default-features = false, optional = true, features = ["load-dynamic"] }
 
 [features]
 # `ml` is ON by default, so a stock build is byte-for-byte what it was before:

crates/headroom-core/src/transforms/kompress.rs

@@ -0,0 +1,672 @@
+//! Kompress — Rust port of `headroom.transforms.kompress_compressor`.
+//!
+//! A ModernBERT token compressor for prose / plain-text tool outputs.
+//! Where SmartCrusher/Log/Search/Diff are deterministic structural
+//! compressors, Kompress is an **ML** compressor: it runs the trained
+//! `chopratejas/kompress-v2-base` model (a fine-tune of
+//! `answerdotai/ModernBERT-base` with a token keep/discard head + a
+//! span-importance CNN head, exported to ONNX) and keeps only the words
+//! the model scores as salient.
+//!
+//! # Model layering
+//!
+//! - **Inference weights:** `chopratejas/kompress-v2-base` — the ONNX
+//!   artifact (`onnx/kompress-int8-wo.onnx`, weight-only int8 via the
+//!   `com.microsoft` `MatMulNBits` contrib op; falls through to
+//!   `onnx/kompress-fp32.onnx` then `onnx/kompress-int8.onnx`). This is
+//!   *the model behind text compression*.
+//! - **Tokenizer:** `answerdotai/ModernBERT-base`'s `tokenizer.json`.
+//!   Kompress is a fine-tune of ModernBERT and reuses its exact vocab,
+//!   so the kompress repo ships no tokenizer of its own.
+//!
+//! # ONNX contract
+//!
+//! Inputs `input_ids` + `attention_mask` (both `int64`, shape
+//! `[batch, seq]`); output `final_scores` (`f32`, shape `[batch, seq]`)
+//! — per-token salience in `[0, 1]` with the dual-head logic baked into
+//! the graph. Keep decision is `score > 0.5`.
+//!
+//! # Compression path (mirrors the Python ONNX/proxy path exactly)
+//!
+//! 1. `words = content.split_whitespace()`. If `< 10` words → passthrough.
+//! 2. For each `chunk_words`-sized (default 350) window of words:
+//!    tokenize with the word list as **pre-tokenized** input
+//!    (`is_split_into_words=True` in `transformers`), truncating to 512
+//!    tokens; recover `input_ids` / `attention_mask` / `word_ids`.
+//! 3. Run ONNX → `final_scores`. Reduce to **max score per word**.
+//! 4. Keep word `w` (global index `w + chunk_start`) when its max score
+//!    exceeds the threshold (default 0.5), or, when `target_ratio` is
+//!    set, when it is in the top-`ratio` fraction by score.
+//! 5. Emit the kept words, in original order, joined by single spaces.
+//!
+//! # Parity
+//!
+//! Byte-exact against the Python reference on the ONNX path: tokenizer
+//! `input_ids`/`word_ids` reproduce `transformers` exactly, ONNX scores
+//! match to ~1e-6 (far below the 0.5 threshold), and the kept-word set +
+//! joined output match byte-for-byte. See
+//! `tests/parity/fixtures/kompress/` and `KompressComparator` in
+//! `crates/headroom-parity`.
+//!
+//! # CCR
+//!
+//! This engine returns the compressed string only. CCR offload of the
+//! dropped words (so the model can retrieve the original on demand) is
+//! handled by the live-zone dispatcher via [`crate::ccr::CcrStore`],
+//! exactly as for the Search/Log/Diff compressors — not inside this
+//! engine. The Python reference's inline `[N items compressed... hash=]`
+//! marker is intentionally **not** reproduced; the Rust side uses the
+//! canonical `<<ccr:HASH>>` marker convention.
+
+use std::collections::{BTreeSet, HashMap};
+use std::path::{Path, PathBuf};
+use std::sync::Mutex;
+
+use ort::session::Session;
+use ort::value::Tensor;
+use thiserror::Error;
+use tokenizers::tokenizer::TruncationParams;
+use tokenizers::{EncodeInput, InputSequence, Tokenizer};
+
+// ─── Tunable defaults (parity-pinned to kompress-v2-base) ───────────────
+
+/// HuggingFace repo holding the trained ONNX weights.
+pub const DEFAULT_MODEL_ID: &str = "chopratejas/kompress-v2-base";
+/// HuggingFace repo holding the tokenizer Kompress reuses.
+pub const DEFAULT_TOKENIZER_REPO: &str = "answerdotai/ModernBERT-base";
+/// Words per inference chunk. Coupled to the model's training window;
+/// kompress-v2-base was trained for 350.
+pub const DEFAULT_CHUNK_WORDS: usize = 350;
+/// Keep a word when its max per-token score exceeds this. Matches the
+/// ONNX `get_keep_mask` hard-coded `> 0.5`.
+pub const DEFAULT_SCORE_THRESHOLD: f32 = 0.5;
+/// Inputs shorter than this many words pass through untouched — too
+/// little signal for the model and the per-call cost dominates.
+pub const MIN_WORDS: usize = 10;
+/// Max ModernBERT sequence length per chunk (truncation bound).
+pub const MAX_SEQ_LEN: usize = 512;
+
+/// ONNX artifact candidates, tried in order. The first is a fp32 model whose
+/// input shape is frozen to a static `[1, MAX_SEQ_LEN]` — required by the
+/// OpenVINO **NPU** EP, which cannot compile dynamic `seq` (it hangs during
+/// graph compilation on the dynamic-shape variants). When a static model is
+/// loaded, `score_chunk` right-pads each chunk to its fixed length (detected
+/// via [`detect_static_seq`]); dynamic models take the chunk's natural length
+/// and pay no padding cost. The static model is absent from a vanilla install
+/// (it is generated separately for NPU deployments), so this entry is simply
+/// skipped on CPU/GPU. The remaining variants are the dynamic fall-throughs:
+/// weight-only int8 (smallest; `MatMulNBits`, unsupported on NPU), then dynamic
+/// fp32 (lossless reference), then the v1-era dynamic int8. A candidate is
+/// skipped on download miss or on session-load failure.
+pub const ONNX_CANDIDATES: &[&str] = &[
+    "onnx/kompress-fp32-static512.onnx",
+    "onnx/kompress-int8-wo.onnx",
+    "onnx/kompress-fp32.onnx",
+    "onnx/kompress-int8.onnx",
+];
+
+// ─── Types ──────────────────────────────────────────────────────────────
+
+/// Configuration for [`Kompress`]. Field defaults match kompress-v2-base;
+/// domain-specific models override `model_id` + `chunk_words` together.
+#[derive(Debug, Clone)]
+pub struct KompressConfig {
+    pub model_id: String,
+    pub tokenizer_repo: String,
+    pub chunk_words: usize,
+    pub score_threshold: f32,
+    pub min_words: usize,
+}
+
+impl Default for KompressConfig {
+    fn default() -> Self {
+        Self {
+            model_id: DEFAULT_MODEL_ID.to_string(),
+            tokenizer_repo: DEFAULT_TOKENIZER_REPO.to_string(),
+            chunk_words: DEFAULT_CHUNK_WORDS,
+            score_threshold: DEFAULT_SCORE_THRESHOLD,
+            min_words: MIN_WORDS,
+        }
+    }
+}
+
+/// Result of a Kompress compression. Mirrors the Python `KompressResult`
+/// fields that the proxy path populates (CCR `cache_key` is owned by the
+/// dispatcher, not this engine).
+#[derive(Debug, Clone, PartialEq)]
+pub struct KompressResult {
+    pub compressed: String,
+    pub original: String,
+    /// Whitespace-split word count of the input.
+    pub original_tokens: usize,
+    /// Word count of the output.
+    pub compressed_tokens: usize,
+    /// `compressed_tokens / original_tokens`, computed in f64 to match the
+    /// Python reference's `float` division bit-for-bit.
+    pub compression_ratio: f64,
+    pub model_used: String,
+}
+
+impl KompressResult {
+    /// Words dropped (never negative).
+    pub fn tokens_saved(&self) -> usize {
+        self.original_tokens.saturating_sub(self.compressed_tokens)
+    }
+
+    /// True when nothing was compressed (output == input word stream).
+    pub fn is_passthrough(&self) -> bool {
+        self.compressed_tokens == self.original_tokens
+    }
+}
+
+#[derive(Debug, Error)]
+pub enum KompressError {
+    #[error("failed to load tokenizer for `{repo}`: {source}")]
+    Tokenizer {
+        repo: String,
+        #[source]
+        source: Box<dyn std::error::Error + Send + Sync>,
+    },
+    #[error("failed to download `{repo}` from HuggingFace Hub: {source}")]
+    Hub {
+        repo: String,
+        #[source]
+        source: Box<dyn std::error::Error + Send + Sync>,
+    },
+    #[error("no loadable ONNX artifact in `{model_id}` (tried {tried:?}): {source}")]
+    Onnx {
+        model_id: String,
+        tried: Vec<String>,
+        #[source]
+        source: Box<dyn std::error::Error + Send + Sync>,
+    },
+}
+
+// ─── Compressor ─────────────────────────────────────────────────────────
+
+/// A loaded Kompress model + tokenizer. Construct once (model load is
+/// expensive) and share; `compress` takes `&self`.
+///
+/// ONNX inference is serialized behind a `Mutex` — matching the Python
+/// reference, which caps ONNX execution to one concurrent call (the CPU
+/// provider does not parallelize the batch dimension for this model).
+pub struct Kompress {
+    config: KompressConfig,
+    tokenizer: Tokenizer,
+    session: Mutex<Session>,
+    /// `Some(n)` when the loaded ONNX has a **fixed** sequence dimension (a
+    /// static `[1, n]` input), in which case `score_chunk` right-pads every
+    /// chunk to `n`. `None` for the usual dynamic-`seq` models, which take the
+    /// chunk's natural length. Detected from the session's `input_ids` shape
+    /// (see [`detect_static_seq`]). The static path exists for execution
+    /// providers that cannot compile dynamic shapes (OpenVINO NPU); masked
+    /// padding leaves the real-token scores unchanged, so output is identical.
+    static_seq: Option<usize>,
+}
+
+impl std::fmt::Debug for Kompress {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        f.debug_struct("Kompress")
+            .field("config", &self.config)
+            .finish_non_exhaustive()
+    }
+}
+
+/// Inspect a built session's `input_ids` input: return `Some(n)` if its
+/// sequence dimension is a fixed `n > 0` (a static-shape model), else `None`
+/// (dynamic `seq`). ONNX inputs are `[batch, seq]`; a dynamic dim is reported
+/// as `-1` by ONNX Runtime.
+fn detect_static_seq(session: &Session) -> Option<usize> {
+    let outlet = session.inputs().iter().find(|o| o.name() == "input_ids")?;
+    let seq = *outlet.dtype().tensor_shape()?.get(1)?;
+    (seq > 0).then_some(seq as usize)
+}
+
+impl Kompress {
+    /// Wrap built artifacts into a `Kompress`, detecting whether the loaded
+    /// model has a static sequence length (so `score_chunk` knows to pad).
+    fn assemble(config: KompressConfig, tokenizer: Tokenizer, session: Session) -> Self {
+        let static_seq = detect_static_seq(&session);
+        Self {
+            config,
+            tokenizer,
+            session: Mutex::new(session),
+            static_seq,
+        }
+    }
+
+    /// Build from local artifact paths — no network. Used by tests and
+    /// the parity harness against the on-disk HuggingFace cache.
+    pub fn from_files(
+        tokenizer_path: impl AsRef<Path>,
+        onnx_path: impl AsRef<Path>,
+        config: KompressConfig,
+    ) -> Result<Self, KompressError> {
+        let tokenizer = load_tokenizer(tokenizer_path.as_ref(), &config.tokenizer_repo)?;
+        let session = build_session(onnx_path.as_ref()).map_err(|e| KompressError::Onnx {
+            model_id: config.model_id.clone(),
+            tried: vec![onnx_path.as_ref().display().to_string()],
+            source: e,
+        })?;
+        Ok(Self::assemble(config, tokenizer, session))
+    }
+
+    /// Build by resolving artifacts from the HuggingFace Hub (cache-first,
+    /// downloading on miss). Blocking — call off the hot path. Tries the
+    /// [`ONNX_CANDIDATES`] in order.
+    pub fn from_pretrained(config: KompressConfig) -> Result<Self, KompressError> {
+        let api = hf_hub::api::sync::Api::new().map_err(|e| KompressError::Hub {
+            repo: config.model_id.clone(),
+            source: Box::new(e),
+        })?;
+
+        let tok_path = api
+            .model(config.tokenizer_repo.clone())
+            .get("tokenizer.json")
+            .map_err(|e| KompressError::Hub {
+                repo: config.tokenizer_repo.clone(),
+                source: Box::new(e),
+            })?;
+        let tokenizer = load_tokenizer(&tok_path, &config.tokenizer_repo)?;
+
+        let model_api = api.model(config.model_id.clone());
+        let mut last_err: Option<Box<dyn std::error::Error + Send + Sync>> = None;
+        let mut tried: Vec<String> = Vec::new();
+        for candidate in ONNX_CANDIDATES {
+            tried.push((*candidate).to_string());
+            let onnx_path: PathBuf = match model_api.get(candidate) {
+                Ok(p) => p,
+                Err(e) => {
+                    last_err = Some(Box::new(e));
+                    continue;
+                }
+            };
+            match build_session(&onnx_path) {
+                Ok(session) => {
+                    return Ok(Self::assemble(config, tokenizer, session));
+                }
+                Err(e) => {
+                    last_err = Some(e);
+                    continue;
+                }
+            }
+        }
+        Err(KompressError::Onnx {
+            model_id: config.model_id.clone(),
+            tried,
+            source: last_err.unwrap_or_else(|| "no ONNX candidates configured".to_string().into()),
+        })
+    }
+
+    /// Cache-only construction: resolve the tokenizer + ONNX artifact from
+    /// the local HuggingFace cache **without ever hitting the network**, and
+    /// return `Ok(None)` when they are not present (or no candidate loads).
+    ///
+    /// This is the Rust mirror of the Python reference's
+    /// `allow_download=False` path (`KompressModelNotCached` → defer): it lets
+    /// the live-zone dispatcher attempt a load on a hot path without risking a
+    /// blocking 261 MB download. When the model isn't cached the caller passes
+    /// plain text through untouched, exactly as Python does when Kompress is
+    /// unavailable.
+    pub fn from_cache(config: KompressConfig) -> Result<Option<Self>, KompressError> {
+        let Some(tok_path) = hf_cache_file(&config.tokenizer_repo, &["tokenizer.json"]) else {
+            // Diagnostic: a `None` here is the #1 cause of a silent
+            // `kompress_ready=false`. Name the repo + the roots searched so
+            // operators don't have to guess between "not downloaded" and
+            // "present but unreadable" (e.g. HF symlinks over `\\wsl$`, which
+            // native Windows can't follow — `path.exists()` returns false on
+            // the unresolved symlink). Cache-only, so this is a defer, not an
+            // error: the caller passes plain text through.
+            tracing::warn!(
+                event = "kompress_cache_miss",
+                stage = "tokenizer",
+                tokenizer_repo = %config.tokenizer_repo,
+                searched_roots = ?hf_hub_roots(),
+                "Kompress deferred: tokenizer.json not found in HF cache \
+                 (not downloaded, or present but unreadable — e.g. HF symlinks \
+                 over \\\\wsl$ which native Windows cannot follow)"
+            );
+            return Ok(None);
+        };
+        let tokenizer = load_tokenizer(&tok_path, &config.tokenizer_repo)?;
+        let mut found_onnx = false;
+        for candidate in ONNX_CANDIDATES {
+            let rel: Vec<&str> = candidate.split('/').collect();
+            let Some(onnx_path) = hf_cache_file(&config.model_id, &rel) else {
+                continue;
+            };
+            found_onnx = true;
+            match build_session(&onnx_path) {
+                Ok(session) => {
+                    return Ok(Some(Self::assemble(config, tokenizer, session)));
+                }
+                Err(e) => {
+                    // The ONNX file is present but the session would not
+                    // build — e.g. the active ORT execution provider rejects
+                    // the graph (OpenVINO/NPU cannot compile the int8
+                    // weight-only `MatMulNBits` op). Loudly surface it and try
+                    // the next candidate (fp32) rather than die silently.
+                    tracing::warn!(
+                        event = "kompress_session_build_failed",
+                        candidate = %candidate,
+                        onnx_path = %onnx_path.display(),
+                        error = %e,
+                        "Kompress: ONNX found but session build failed; trying next candidate"
+                    );
+                }
+            }
+        }
+        tracing::warn!(
+            event = "kompress_cache_miss",
+            stage = "onnx",
+            model_id = %config.model_id,
+            candidates = ?ONNX_CANDIDATES,
+            any_onnx_found = found_onnx,
+            searched_roots = ?hf_hub_roots(),
+            "Kompress deferred: no usable ONNX session \
+             (no candidate file in cache, or every candidate failed to build)"
+        );
+        Ok(None)
+    }
+
+    /// Model-decides compression (the proxy path): keep words scoring
+    /// above `config.score_threshold`.
+    pub fn compress(&self, content: &str) -> KompressResult {
+        self.compress_inner(content, None)
+    }
+
+    /// Forced-ratio compression: keep the top `target_ratio` fraction of
+    /// words by score (at least one). `None` defers to the threshold path.
+    /// The proxy never sets this — only the user-facing API does.
+    pub fn compress_with_ratio(&self, content: &str, target_ratio: Option<f64>) -> KompressResult {
+        self.compress_inner(content, target_ratio)
+    }
+
+    fn compress_inner(&self, content: &str, target_ratio: Option<f64>) -> KompressResult {
+        let words: Vec<&str> = content.split_whitespace().collect();
+        let n_words = words.len();
+        if n_words < self.config.min_words {
+            return self.passthrough(content, n_words);
+        }
+
+        let mut kept_ids: BTreeSet<usize> = BTreeSet::new();
+        let mut chunk_start = 0usize;
+        while chunk_start < n_words {
+            let end = (chunk_start + self.config.chunk_words).min(n_words);
+            match self.score_chunk(&words[chunk_start..end]) {
+                Ok(word_scores) => {
+                    self.select_words(&word_scores, chunk_start, target_ratio, &mut kept_ids);
+                }
+                Err(_) => {
+                    // A chunk that fails inference is treated as
+                    // "nothing salient here" — matches the Python
+                    // reference's per-call passthrough-on-error.
+                    return self.passthrough(content, n_words);
+                }
+            }
+            chunk_start += self.config.chunk_words;
+        }
+
+        if kept_ids.is_empty() {
+            return self.passthrough(content, n_words);
+        }
+
+        let compressed_words: Vec<&str> = kept_ids
+            .iter()
+            .filter(|&&w| w < n_words)
+            .map(|&w| words[w])
+            .collect();
+        let compressed_tokens = compressed_words.len();
+        let compressed = compressed_words.join(" ");
+        let compression_ratio = if n_words == 0 {
+            1.0
+        } else {
+            compressed_tokens as f64 / n_words as f64
+        };
+        KompressResult {
+            compressed,
+            original: content.to_string(),
+            original_tokens: n_words,
+            compressed_tokens,
+            compression_ratio,
+            model_used: self.config.model_id.clone(),
+        }
+    }
+
+    /// Tokenize one chunk of words and return the **max score per word**
+    /// (`word_index -> score`). `word_index` is local to the chunk.
+    fn score_chunk(
+        &self,
+        chunk_words: &[&str],
+    ) -> Result<HashMap<usize, f32>, Box<dyn std::error::Error + Send + Sync>> {
+        let seq_in: Vec<&str> = chunk_words.to_vec();
+        let encoding = self
+            .tokenizer
+            .encode(EncodeInput::Single(InputSequence::from(seq_in)), true)?;
+        let ids: Vec<i64> = encoding.get_ids().iter().map(|&x| x as i64).collect();
+        let attn: Vec<i64> = encoding
+            .get_attention_mask()
+            .iter()
+            .map(|&x| x as i64)
+            .collect();
+        let word_ids = encoding.get_word_ids();
+        let mut ids = ids;
+        let mut attn = attn;
+
+        // Static-shape models (e.g. the OpenVINO NPU build, which cannot
+        // compile a dynamic `seq`) require a fixed `[1, static_seq]` input, so
+        // right-pad every chunk to that length. Real tokens occupy
+        // `0..real_seq`; the tail is padding with `attention_mask = 0`, which
+        // masks those positions out of self-attention — the scores at real
+        // positions are identical to an unpadded run, so keep/discard decisions
+        // (hence parity) are unchanged. The tokenizer truncates to
+        // `MAX_SEQ_LEN`, so the chunk never exceeds a `static_seq` of that size.
+        // Dynamic models (`static_seq == None`) take the chunk's natural length
+        // and pay no padding cost — the default for CPU/GPU.
+        let seq = match self.static_seq {
+            Some(n) => {
+                debug_assert!(ids.len() <= n);
+                ids.resize(n, 0);
+                attn.resize(n, 0);
+                n
+            }
+            None => ids.len(),
+        };
+
+        let input_ids = Tensor::from_array(([1usize, seq], ids))?;
+        let attention_mask = Tensor::from_array(([1usize, seq], attn))?;
+
+        let scores: Vec<f32> = {
+            let mut session = self
+                .session
+                .lock()
+                .unwrap_or_else(|poisoned| poisoned.into_inner());
+            let outputs = session.run(ort::inputs![
+                "input_ids" => input_ids,
+                "attention_mask" => attention_mask
+            ])?;
+            let (_shape, data) = outputs["final_scores"].try_extract_tensor::<f32>()?;
+            data.to_vec()
+        };
+
+        let mut word_scores: HashMap<usize, f32> = HashMap::new();
+        for (idx, wid) in word_ids.iter().enumerate() {
+            let Some(w) = wid else { continue };
+            let Some(&s) = scores.get(idx) else { continue };
+            let entry = word_scores.entry(*w as usize).or_insert(f32::MIN);
+            if s > *entry {
+                *entry = s;
+            }
+        }
+        Ok(word_scores)
+    }
+
+    /// Apply the threshold or top-k rule to one chunk's per-word scores,
+    /// inserting kept **global** word indices into `kept_ids`.
+    fn select_words(
+        &self,
+        word_scores: &HashMap<usize, f32>,
+        chunk_start: usize,
+        target_ratio: Option<f64>,
+        kept_ids: &mut BTreeSet<usize>,
+    ) {
+        if word_scores.is_empty() {
+            return;
+        }
+        match target_ratio {
+            Some(ratio) => {
+                // Stable top-k: iterate words in ascending index order so
+                // equal scores break toward the lower word index — this
+                // matches CPython's stable `sorted()` over the
+                // insertion-ordered score dict (tokens emitted in word
+                // order).
+                let mut ordered: Vec<(usize, f32)> =
+                    word_scores.iter().map(|(&w, &s)| (w, s)).collect();
+                ordered.sort_by_key(|&(w, _)| w);
+                ordered.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal));
+                let num_keep = ((ordered.len() as f64 * ratio) as usize).max(1);
+                for &(w, _) in ordered.iter().take(num_keep) {
+                    kept_ids.insert(w + chunk_start);
+                }
+            }
+            None => {
+                for (&w, &s) in word_scores {
+                    if s > self.config.score_threshold {
+                        kept_ids.insert(w + chunk_start);
+                    }
+                }
+            }
+        }
+    }
+
+    fn passthrough(&self, content: &str, n_words: usize) -> KompressResult {
+        KompressResult {
+            compressed: content.to_string(),
+            original: content.to_string(),
+            original_tokens: n_words,
+            compressed_tokens: n_words,
+            compression_ratio: 1.0,
+            model_used: self.config.model_id.clone(),
+        }
+    }
+
+    /// Expose the active config (read-only).
+    pub fn config(&self) -> &KompressConfig {
+        &self.config
+    }
+}
+
+// ─── Loading helpers ────────────────────────────────────────────────────
+
+fn load_tokenizer(path: &Path, repo: &str) -> Result<Tokenizer, KompressError> {
+    let mut tokenizer = Tokenizer::from_file(path).map_err(|e| KompressError::Tokenizer {
+        repo: repo.to_string(),
+        source: e,
+    })?;
+    // Match the Python reference: truncation=True, max_length=512.
+    tokenizer
+        .with_truncation(Some(TruncationParams {
+            max_length: MAX_SEQ_LEN,
+            ..Default::default()
+        }))
+        .map_err(|e| KompressError::Tokenizer {
+            repo: repo.to_string(),
+            source: e,
+        })?;
+    Ok(tokenizer)
+}
+
+fn build_session(path: &Path) -> Result<Session, Box<dyn std::error::Error + Send + Sync>> {
+    let session = Session::builder()?.commit_from_file(path)?;
+    Ok(session)
+}
+
+/// Resolve `rel` (e.g. `["tokenizer.json"]` or `["onnx", "kompress-int8-wo.onnx"]`)
+/// inside the local HuggingFace cache for `repo` (`"owner/name"`), searching
+/// every snapshot under every candidate cache root. Returns `None` if not
+/// present — never touches the network.
+fn hf_cache_file(repo: &str, rel: &[&str]) -> Option<PathBuf> {
+    let repo_dir = format!("models--{}", repo.replace('/', "--"));
+    for hub in hf_hub_roots() {
+        let snapshots = hub.join(&repo_dir).join("snapshots");
+        let Ok(entries) = std::fs::read_dir(&snapshots) else {
+            continue;
+        };
+        for snap in entries.flatten() {
+            let mut cand = snap.path();
+            for part in rel {
+                cand = cand.join(part);
+            }
+            if cand.exists() {
+                return Some(cand);
+            }
+        }
+    }
+    None
+}
+
+/// HuggingFace hub cache roots in resolution precedence. Cross-platform so
+/// the cache-only loader works on Windows (native `headroom-proxy.exe`) as
+/// well as Linux: `HF_HUB_CACHE` (the hub dir directly) → `HF_HOME/hub` →
+/// `{HOME|USERPROFILE}/.cache/huggingface/hub`. `HOME` is the unix home; on
+/// Windows the process sees `USERPROFILE` (and often no `HOME`), so both are
+/// tried. Honoring `HF_HOME` also lets a Windows proxy point at a WSL cache.
+fn hf_hub_roots() -> Vec<PathBuf> {
+    let mut roots = Vec::new();
+    let push_env = |roots: &mut Vec<PathBuf>, var: &str, suffix: &[&str]| {
+        if let Ok(v) = std::env::var(var) {
+            if !v.is_empty() {
+                let mut p = PathBuf::from(v);
+                for s in suffix {
+                    p = p.join(s);
+                }
+                roots.push(p);
+            }
+        }
+    };
+    push_env(&mut roots, "HF_HUB_CACHE", &[]);
+    push_env(&mut roots, "HF_HOME", &["hub"]);
+    push_env(&mut roots, "HOME", &[".cache", "huggingface", "hub"]);
+    push_env(&mut roots, "USERPROFILE", &[".cache", "huggingface", "hub"]);
+    roots
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn config_defaults_match_kompress_v2_base() {
+        let c = KompressConfig::default();
+        assert_eq!(c.model_id, "chopratejas/kompress-v2-base");
+        assert_eq!(c.tokenizer_repo, "answerdotai/ModernBERT-base");
+        assert_eq!(c.chunk_words, 350);
+        assert_eq!(c.score_threshold, 0.5);
+        assert_eq!(c.min_words, 10);
+    }
+
+    #[test]
+    fn result_helpers() {
+        let r = KompressResult {
+            compressed: "a b".into(),
+            original: "a b c d".into(),
+            original_tokens: 4,
+            compressed_tokens: 2,
+            compression_ratio: 0.5,
+            model_used: DEFAULT_MODEL_ID.into(),
+        };
+        assert_eq!(r.tokens_saved(), 2);
+        assert!(!r.is_passthrough());
+
+        let p = KompressResult {
+            compressed: "a b".into(),
+            original: "a b".into(),
+            original_tokens: 2,
+            compressed_tokens: 2,
+            compression_ratio: 1.0,
+            model_used: DEFAULT_MODEL_ID.into(),
+        };
+        assert_eq!(p.tokens_saved(), 0);
+        assert!(p.is_passthrough());
+    }
+}

crates/headroom-core/src/transforms/mod.rs

@@ -20,6 +20,8 @@ pub mod anchor_selector;
 pub mod content_detector;
 pub mod detection;
 pub mod diff_compressor;
+#[cfg(feature = "ml")]
+pub mod kompress;
 pub mod live_zone;
 pub mod log_compressor;
 #[cfg(feature = "ml")]
@@ -40,6 +42,11 @@ pub use detection::detect;
 pub use diff_compressor::{
     DiffCompressionResult, DiffCompressor, DiffCompressorConfig, DiffCompressorStats,
 };
+#[cfg(feature = "ml")]
+pub use kompress::{
+    Kompress, KompressConfig, KompressError, KompressResult, DEFAULT_MODEL_ID,
+    DEFAULT_TOKENIZER_REPO,
+};
 pub use live_zone::{
     compress_anthropic_live_zone, compress_openai_chat_live_zone,
     compress_openai_responses_live_zone, summarize_openai_responses_no_change_reason, AuthMode,

crates/headroom-core/tests/kompress_parity.rs

@@ -0,0 +1,131 @@
+//! Byte-parity integration test for the Kompress Rust port.
+//!
+//! Runs the production [`Kompress`] engine against the trace fixtures
+//! recorded from the Python reference (`tests/parity/fixtures/kompress/`)
+//! and asserts the compressed output matches byte-for-byte.
+//!
+//! Model-gated: if the ModernBERT tokenizer + kompress-v2-base ONNX
+//! artifact are not present in the local HuggingFace cache (e.g. CI with
+//! no network / no preloaded model), the test SKIPS rather than fails —
+//! mirroring the parity harness's "stub → Skipped" tolerance. Run it
+//! locally after `python scripts/record_kompress_trace.py` to get the
+//! real assertion.
+
+use std::fs;
+use std::path::{Path, PathBuf};
+
+use headroom_core::transforms::kompress::{Kompress, KompressConfig};
+use serde_json::Value;
+
+fn hf_cache_file(repo_dir: &str, rel: &[&str]) -> Option<PathBuf> {
+    let home = std::env::var("HOME").ok()?;
+    let snapshots = Path::new(&home)
+        .join(".cache/huggingface/hub")
+        .join(repo_dir)
+        .join("snapshots");
+    for snap in fs::read_dir(snapshots).ok()?.filter_map(|e| e.ok()) {
+        let mut cand = snap.path();
+        for part in rel {
+            cand = cand.join(part);
+        }
+        if cand.exists() {
+            return Some(cand);
+        }
+    }
+    None
+}
+
+#[test]
+fn kompress_matches_python_fixtures_byte_for_byte() {
+    let tok = hf_cache_file("models--answerdotai--ModernBERT-base", &["tokenizer.json"]);
+    let onnx = hf_cache_file(
+        "models--chopratejas--kompress-v2-base",
+        &["onnx", "kompress-int8-wo.onnx"],
+    );
+    let (tok, onnx) = match (tok, onnx) {
+        (Some(t), Some(o)) => (t, o),
+        _ => {
+            eprintln!(
+                "SKIP: kompress model/tokenizer not in HF cache; \
+                 run `python scripts/record_kompress_trace.py` first"
+            );
+            return;
+        }
+    };
+
+    let fixtures_dir =
+        Path::new(env!("CARGO_MANIFEST_DIR")).join("../../tests/parity/fixtures/kompress");
+    if !fixtures_dir.exists() {
+        eprintln!("SKIP: fixtures dir {} missing", fixtures_dir.display());
+        return;
+    }
+
+    let kompress = Kompress::from_files(&tok, &onnx, KompressConfig::default())
+        .expect("load kompress from local files");
+
+    let mut checked = 0usize;
+    let mut paths: Vec<PathBuf> = fs::read_dir(&fixtures_dir)
+        .expect("read fixtures dir")
+        .filter_map(|e| e.ok())
+        .map(|e| e.path())
+        .filter(|p| {
+            p.extension().map(|x| x == "json").unwrap_or(false)
+                && p.file_name()
+                    .map(|n| n != "_manifest.json")
+                    .unwrap_or(false)
+        })
+        .collect();
+    paths.sort();
+
+    for path in paths {
+        let fx: Value = serde_json::from_str(&fs::read_to_string(&path).unwrap()).unwrap();
+        let name = path.file_name().unwrap().to_string_lossy().to_string();
+        // Standard parity fixture: {transform, input, config, output}.
+        let content = fx["input"].as_str().expect("fixture.input string");
+        let out = &fx["output"];
+        let exp_compressed = out["compressed"].as_str().expect("output.compressed");
+        let exp_ratio = out["compression_ratio"]
+            .as_f64()
+            .expect("output.compression_ratio");
+
+        let result = kompress.compress(content);
+
+        assert_eq!(
+            result.compressed, exp_compressed,
+            "[{name}] compressed output diverged from Python reference"
+        );
+        assert!(
+            (result.compression_ratio - exp_ratio).abs() < 1e-6,
+            "[{name}] ratio {} != python {}",
+            result.compression_ratio,
+            exp_ratio
+        );
+        checked += 1;
+    }
+
+    assert!(checked > 0, "no kompress fixtures were checked");
+    eprintln!("kompress parity: {checked} fixtures matched byte-for-byte");
+}
+
+#[test]
+fn short_input_passes_through() {
+    // Pure-logic check — no model needed. Fewer than MIN_WORDS words must
+    // pass through unchanged regardless of model availability... but the
+    // engine needs a model to construct. Guard on cache like the main test.
+    let tok = hf_cache_file("models--answerdotai--ModernBERT-base", &["tokenizer.json"]);
+    let onnx = hf_cache_file(
+        "models--chopratejas--kompress-v2-base",
+        &["onnx", "kompress-int8-wo.onnx"],
+    );
+    let (Some(tok), Some(onnx)) = (tok, onnx) else {
+        eprintln!("SKIP: model not cached");
+        return;
+    };
+    let kompress = Kompress::from_files(&tok, &onnx, KompressConfig::default()).unwrap();
+
+    let short = "only a few words here";
+    let r = kompress.compress(short);
+    assert!(r.is_passthrough());
+    assert_eq!(r.compressed, short);
+    assert_eq!(r.compression_ratio, 1.0);
+}

crates/headroom-parity/src/lib.rs

@@ -614,6 +614,105 @@ impl TransformComparator for TextCrusherComparator {
     }
 }
 
+/// Kompress comparator — runs the ML prose compressor against fixtures
+/// recorded from the Python reference.
+///
+/// Unlike the deterministic compressors, Kompress needs the
+/// `kompress-v2-base` ONNX model + ModernBERT tokenizer. The comparator
+/// resolves them **from the local HuggingFace cache only** (never the
+/// network) and lazily loads once. When the artifacts are absent — CI
+/// with no preloaded model — `run` returns `Err`, so the harness marks
+/// every kompress fixture `Skipped` rather than failing. Record + run
+/// locally (after `python scripts/record_fixtures.py`) for the real
+/// byte-parity assertion.
+///
+/// Fixtures are recorded with `enable_ccr=False` so the output is the
+/// pure joined kept-word stream (the Rust engine never emits the Python
+/// inline CCR marker; live-zone CCR uses the `<<ccr:>>` convention).
+pub struct KompressComparator {
+    model: std::sync::OnceLock<Option<headroom_core::transforms::kompress::Kompress>>,
+}
+
+impl Default for KompressComparator {
+    fn default() -> Self {
+        Self {
+            model: std::sync::OnceLock::new(),
+        }
+    }
+}
+
+impl KompressComparator {
+    pub fn new() -> Self {
+        Self::default()
+    }
+
+    fn hf_cache_file(repo_dir: &str, rel: &[&str]) -> Option<PathBuf> {
+        let home = std::env::var("HOME").ok()?;
+        let snapshots = Path::new(&home)
+            .join(".cache/huggingface/hub")
+            .join(repo_dir)
+            .join("snapshots");
+        for snap in fs::read_dir(snapshots).ok()?.filter_map(|e| e.ok()) {
+            let mut cand = snap.path();
+            for part in rel {
+                cand = cand.join(part);
+            }
+            if cand.exists() {
+                return Some(cand);
+            }
+        }
+        None
+    }
+
+    fn model(&self) -> Option<&headroom_core::transforms::kompress::Kompress> {
+        self.model
+            .get_or_init(|| {
+                use headroom_core::transforms::kompress::{Kompress, KompressConfig};
+                let tok = Self::hf_cache_file(
+                    "models--answerdotai--ModernBERT-base",
+                    &["tokenizer.json"],
+                )?;
+                let onnx = Self::hf_cache_file(
+                    "models--chopratejas--kompress-v2-base",
+                    &["onnx", "kompress-int8-wo.onnx"],
+                )?;
+                Kompress::from_files(&tok, &onnx, KompressConfig::default()).ok()
+            })
+            .as_ref()
+    }
+}
+
+impl TransformComparator for KompressComparator {
+    fn name(&self) -> &str {
+        "kompress"
+    }
+
+    fn run(
+        &self,
+        input: &serde_json::Value,
+        _config: &serde_json::Value,
+    ) -> Result<serde_json::Value> {
+        let content = input
+            .as_str()
+            .context("kompress fixture input must be a JSON string")?;
+        let model = self
+            .model()
+            .context("kompress model/tokenizer not in local HF cache (fixture skipped)")?;
+        let r = model.compress(content);
+        Ok(serde_json::json!({
+            "compressed": r.compressed,
+            "original": r.original,
+            "original_tokens": r.original_tokens,
+            "compressed_tokens": r.compressed_tokens,
+            "compression_ratio": r.compression_ratio,
+            // Engine never emits CCR markers; dispatcher owns CCR. Python
+            // fixtures are recorded with enable_ccr=False so cache_key is null.
+            "cache_key": serde_json::Value::Null,
+            "model_used": r.model_used,
+        }))
+    }
+}
+
 /// Every built-in comparator, in a stable order.
 pub fn builtin_comparators() -> Vec<Box<dyn TransformComparator>> {
     vec![
@@ -625,6 +724,7 @@ pub fn builtin_comparators() -> Vec<Box<dyn TransformComparator>> {
         Box::new(SmartCrusherComparator),
         Box::new(ContentDetectorComparator),
         Box::new(TextCrusherComparator),
+        Box::new(KompressComparator::new()),
     ]
 }
 

scripts/record_kompress_fixtures.py

@@ -0,0 +1,47 @@
+#!/usr/bin/env python3
+"""Record standard parity fixtures for the Kompress transform only.
+
+Drives the Python `KompressCompressor` (enable_ccr=False) over the shared
+`_varied_kompress_inputs()` workload while `record_all()` has the compress
+method patched, so only `tests/parity/fixtures/kompress/` is (re)written —
+no churn to other transforms' fixtures.
+
+Run after the model is cached:
+    python scripts/record_kompress_fixtures.py
+"""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+REPO = Path(__file__).resolve().parent.parent
+sys.path.insert(0, str(REPO))
+
+
+def main() -> int:
+    from tests.parity.recorder import _varied_kompress_inputs, record_all
+
+    statuses = record_all()
+    if not statuses.get("kompress", "").startswith("patched"):
+        print(f"kompress not patched: {statuses.get('kompress')}", file=sys.stderr)
+        return 1
+
+    from headroom.transforms.kompress_compressor import (
+        KompressCompressor,
+        KompressConfig,
+    )
+
+    kc = KompressCompressor(KompressConfig(enable_ccr=False))
+    inputs = _varied_kompress_inputs()
+    for s in inputs:
+        kc.compress(s)
+
+    out_dir = REPO / "tests" / "parity" / "fixtures" / "kompress"
+    n = len(list(out_dir.glob("*.json")))
+    print(f"recorded {n} kompress fixtures from {len(inputs)} inputs -> {out_dir}", file=sys.stderr)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

tests/parity/fixtures/kompress/0321ef8700c1fd25.json

@@ -0,0 +1,22 @@
+{
+  "config": {
+    "chunk_words": 350,
+    "device": "auto",
+    "enable_ccr": false,
+    "model_id": "chopratejas/kompress-v2-base",
+    "score_threshold": 0.5
+  },
+  "input": "ERROR 2026-06-18 connection refused to upstream after 3 retries; WARN falling back to cache; INFO request completed in 421ms status 200; DEBUG tracing span closed for request abc123; ERROR 2026-06-18 connection refused to upstream after 3 retries; WARN falling back to cache; INFO request completed in 421ms status 200; DEBUG tracing span closed for request abc123; ERROR 2026-06-18 connection refused to upstream after 3 retries; WARN falling back to cache; INFO request completed in 421ms status 200; DEBUG tracing span closed for request abc123; ERROR 2026-06-18 connection refused to upstream after 3 retries; WARN falling back to cache; INFO request completed in 421ms status 200; DEBUG tracing span closed for request abc123; ERROR 2026-06-18 connection refused to upstream after 3 retries; WARN falling back to cache; INFO request completed in 421ms status 200; DEBUG tracing span closed for request abc123; ERROR 2026-06-18 connection refused to upstream after 3 retries; WARN falling back to cache; INFO request completed in 421ms status 200; DEBUG tracing span closed for request abc123; ",
+  "input_sha256": "0321ef8700c1fd257ca9f5a435eeb71b17c46b0086e8e30697ab2473dd232fea",
+  "output": {
+    "cache_key": null,
+    "compressed": "ERROR 2026-06-18 connection refused upstream after 3 retries; WARN falling back to cache; INFO request completed in 421ms status 200; DEBUG tracing span closed for request abc123; ERROR 2026-06-18 connection refused upstream after 3 retries; WARN falling back to cache; INFO request completed in 421ms status 200; DEBUG tracing span closed for request abc123; ERROR 2026-06-18 connection refused upstream after 3 retries; WARN falling back to cache; INFO request completed in 421ms status 200; DEBUG tracing span closed request abc123; ERROR 2026-06-18 connection refused upstream after 3 retries; WARN falling back to cache; INFO request completed in 421ms status 200; DEBUG tracing span closed request abc123; ERROR 2026-06-18 connection refused upstream after 3 retries; WARN falling back to cache; INFO request completed in 421ms status 200; DEBUG tracing span closed for request abc123; ERROR 2026-06-18 connection refused upstream after 3 retries; WARN falling back to cache; INFO request completed in 421ms status 200; DEBUG tracing span closed for request abc123;",
+    "compressed_tokens": 160,
+    "compression_ratio": 0.9523809523809523,
+    "model_used": "chopratejas/kompress-v2-base",
+    "original": "ERROR 2026-06-18 connection refused to upstream after 3 retries; WARN falling back to cache; INFO request completed in 421ms status 200; DEBUG tracing span closed for request abc123; ERROR 2026-06-18 connection refused to upstream after 3 retries; WARN falling back to cache; INFO request completed in 421ms status 200; DEBUG tracing span closed for request abc123; ERROR 2026-06-18 connection refused to upstream after 3 retries; WARN falling back to cache; INFO request completed in 421ms status 200; DEBUG tracing span closed for request abc123; ERROR 2026-06-18 connection refused to upstream after 3 retries; WARN falling back to cache; INFO request completed in 421ms status 200; DEBUG tracing span closed for request abc123; ERROR 2026-06-18 connection refused to upstream after 3 retries; WARN falling back to cache; INFO request completed in 421ms status 200; DEBUG tracing span closed for request abc123; ERROR 2026-06-18 connection refused to upstream after 3 retries; WARN falling back to cache; INFO request completed in 421ms status 200; DEBUG tracing span closed for request abc123; ",
+    "original_tokens": 168
+  },
+  "recorded_at": "2026-06-18T23:18:25.091568+00:00",
+  "transform": "kompress"
+}

tests/parity/fixtures/kompress/1586e806db77fd58.json

@@ -0,0 +1,22 @@
+{
+  "config": {
+    "chunk_words": 350,
+    "device": "auto",
+    "enable_ccr": false,
+    "model_id": "chopratejas/kompress-v2-base",
+    "score_threshold": 0.5
+  },
+  "input": "Installation requires Python three point ten or newer along with the optional machine learning extra which pulls onnxruntime and transformers for the token compression model used by the proxy at request time. Installation requires Python three point ten or newer along with the optional machine learning extra which pulls onnxruntime and transformers for the token compression model used by the proxy at request time. Installation requires Python three point ten or newer along with the optional machine learning extra which pulls onnxruntime and transformers for the token compression model used by the proxy at request time. ",
+  "input_sha256": "1586e806db77fd58225e7c1f03619de5f67cc9412ebc8b7ff92fde93cd64bdd0",
+  "output": {
+    "cache_key": null,
+    "compressed": "Installation requires Python three point ten or newer along with optional machine learning extra pulls onnxruntime transformers for token compression model used by proxy at request time. Installation requires Python three point ten or newer along with optional machine learning extra pulls onnxruntime transformers for token compression model used by proxy at request time. Installation requires Python three point ten or newer along with optional machine learning extra pulls onnxruntime transformers for token compression model used by proxy at request time.",
+    "compressed_tokens": 81,
+    "compression_ratio": 0.84375,
+    "model_used": "chopratejas/kompress-v2-base",
+    "original": "Installation requires Python three point ten or newer along with the optional machine learning extra which pulls onnxruntime and transformers for the token compression model used by the proxy at request time. Installation requires Python three point ten or newer along with the optional machine learning extra which pulls onnxruntime and transformers for the token compression model used by the proxy at request time. Installation requires Python three point ten or newer along with the optional machine learning extra which pulls onnxruntime and transformers for the token compression model used by the proxy at request time. ",
+    "original_tokens": 96
+  },
+  "recorded_at": "2026-06-18T23:18:31.734574+00:00",
+  "transform": "kompress"
+}

tests/parity/fixtures/kompress/1730a6bdcfa35f6d.json

@@ -0,0 +1,22 @@
+{
+  "config": {
+    "chunk_words": 350,
+    "device": "auto",
+    "enable_ccr": false,
+    "model_id": "chopratejas/kompress-v2-base",
+    "score_threshold": 0.5
+  },
+  "input": "Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. ",
+  "input_sha256": "1730a6bdcfa35f6d19faf40b0e3765bdb1d4b54a5e413aa0ed741e638ebe4749",
+  "output": {
+    "cache_key": null,
+    "compressed": "Meanwhile diligent engineer reviews compression output carefully ensure most salient tokens survive while redundant filler discarded without losing essential meaning. quick brown fox jumps over lazy dog. quick brown fox jumps over lazy dog. quick brown fox jumps over lazy dog. quick brown fox jumps over lazy dog. quick brown fox jumps over lazy dog.",
+    "compressed_tokens": 55,
+    "compression_ratio": 0.7746478873239436,
+    "model_used": "chopratejas/kompress-v2-base",
+    "original": "Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. ",
+    "original_tokens": 71
+  },
+  "recorded_at": "2026-06-18T23:18:31.240602+00:00",
+  "transform": "kompress"
+}

tests/parity/fixtures/kompress/2a4d2bed53060038.json

@@ -0,0 +1,22 @@
+{
+  "config": {
+    "chunk_words": 350,
+    "device": "auto",
+    "enable_ccr": false,
+    "model_id": "chopratejas/kompress-v2-base",
+    "score_threshold": 0.5
+  },
+  "input": "Traceback (most recent call last): File main.py line 42 in handler raise ValueError invalid token; the request could not be processed because the upstream returned an unexpected payload shape repeatedly. Traceback (most recent call last): File main.py line 42 in handler raise ValueError invalid token; the request could not be processed because the upstream returned an unexpected payload shape repeatedly. Traceback (most recent call last): File main.py line 42 in handler raise ValueError invalid token; the request could not be processed because the upstream returned an unexpected payload shape repeatedly. ",
+  "input_sha256": "2a4d2bed53060038a62e1d9c9fe4151a3b66ebf8a98e228599291ef934d53273",
+  "output": {
+    "cache_key": null,
+    "compressed": "Traceback (most recent call last): File main.py line 42 in handler raise ValueError invalid token; request could not be processed because upstream returned an unexpected payload shape repeatedly. Traceback (most recent call last): File main.py line 42 in handler raise ValueError invalid token; request could not be processed because upstream returned an unexpected payload shape repeatedly. Traceback (most recent call last): File main.py line 42 in handler raise ValueError invalid token; request could not be processed because upstream returned an unexpected payload shape repeatedly.",
+    "compressed_tokens": 84,
+    "compression_ratio": 0.9333333333333333,
+    "model_used": "chopratejas/kompress-v2-base",
+    "original": "Traceback (most recent call last): File main.py line 42 in handler raise ValueError invalid token; the request could not be processed because the upstream returned an unexpected payload shape repeatedly. Traceback (most recent call last): File main.py line 42 in handler raise ValueError invalid token; the request could not be processed because the upstream returned an unexpected payload shape repeatedly. Traceback (most recent call last): File main.py line 42 in handler raise ValueError invalid token; the request could not be processed because the upstream returned an unexpected payload shape repeatedly. ",
+    "original_tokens": 90
+  },
+  "recorded_at": "2026-06-18T23:18:25.316100+00:00",
+  "transform": "kompress"
+}

tests/parity/fixtures/kompress/4547d4fba3cb8234.json

@@ -0,0 +1,22 @@
+{
+  "config": {
+    "chunk_words": 350,
+    "device": "auto",
+    "enable_ccr": false,
+    "model_id": "chopratejas/kompress-v2-base",
+    "score_threshold": 0.5
+  },
+  "input": "just a handful of words below the threshold now",
+  "input_sha256": "4547d4fba3cb8234b17cb8c0c5196cc9c983da011c0e025fa99e2dc22d5e00ed",
+  "output": {
+    "cache_key": null,
+    "compressed": "just a handful of words below the threshold now",
+    "compressed_tokens": 9,
+    "compression_ratio": 1.0,
+    "model_used": "chopratejas/kompress-v2-base",
+    "original": "just a handful of words below the threshold now",
+    "original_tokens": 9
+  },
+  "recorded_at": "2026-06-18T23:18:19.224369+00:00",
+  "transform": "kompress"
+}

tests/parity/fixtures/kompress/45d9e4200abaee4f.json

@@ -0,0 +1,22 @@
+{
+  "config": {
+    "chunk_words": 350,
+    "device": "auto",
+    "enable_ccr": false,
+    "model_id": "chopratejas/kompress-v2-base",
+    "score_threshold": 0.5
+  },
+  "input": "Summarize the following document while preserving every named entity and numeric figure so the reader can reconstruct the key facts later. Summarize the following document while preserving every named entity and numeric figure so the reader can reconstruct the key facts later. Summarize the following document while preserving every named entity and numeric figure so the reader can reconstruct the key facts later. Summarize the following document while preserving every named entity and numeric figure so the reader can reconstruct the key facts later. ",
+  "input_sha256": "45d9e4200abaee4fbaa71006890e11acc3eacdcfa50856f5cac953e19b6be643",
+  "output": {
+    "cache_key": null,
+    "compressed": "Summarize following document while preserving every named entity and numeric figure so reader can reconstruct key facts later. Summarize following document while preserving every named entity and numeric figure so reader can reconstruct key facts later. Summarize following document while preserving every named entity and numeric figure so reader can reconstruct key facts later. Summarize following document while preserving every named entity and numeric figure so reader can reconstruct key facts later.",
+    "compressed_tokens": 72,
+    "compression_ratio": 0.8571428571428571,
+    "model_used": "chopratejas/kompress-v2-base",
+    "original": "Summarize the following document while preserving every named entity and numeric figure so the reader can reconstruct the key facts later. Summarize the following document while preserving every named entity and numeric figure so the reader can reconstruct the key facts later. Summarize the following document while preserving every named entity and numeric figure so the reader can reconstruct the key facts later. Summarize the following document while preserving every named entity and numeric figure so the reader can reconstruct the key facts later. ",
+    "original_tokens": 84
+  },
+  "recorded_at": "2026-06-18T23:18:31.416183+00:00",
+  "transform": "kompress"
+}

tests/parity/fixtures/kompress/717ac5c1b5a49c4e.json

@@ -0,0 +1,22 @@
+{
+  "config": {
+    "chunk_words": 350,
+    "device": "auto",
+    "enable_ccr": false,
+    "model_id": "chopratejas/kompress-v2-base",
+    "score_threshold": 0.5
+  },
+  "input": "The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. ",
+  "input_sha256": "717ac5c1b5a49c4ebcccc7d177c54b21d4f249c4b1e7e1f8696a69eab07b5f9b",
+  "output": {
+    "cache_key": null,
+    "compressed": "quick brown fox jumps over lazy dog. quick brown fox jumps over lazy dog. quick brown fox jumps over lazy dog. quick brown fox jumps over lazy dog. quick brown fox jumps over lazy dog. quick brown fox jumps over lazy dog. Meanwhile diligent engineer reviews compression output carefully ensure most salient tokens survive while redundant filler is discarded without losing essential meaning.",
+    "compressed_tokens": 63,
+    "compression_ratio": 0.7875,
+    "model_used": "chopratejas/kompress-v2-base",
+    "original": "The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. ",
+    "original_tokens": 80
+  },
+  "recorded_at": "2026-06-18T23:18:24.191529+00:00",
+  "transform": "kompress"
+}

tests/parity/fixtures/kompress/74fde229e9465c81.json

@@ -0,0 +1,22 @@
+{
+  "config": {
+    "chunk_words": 350,
+    "device": "auto",
+    "enable_ccr": false,
+    "model_id": "chopratejas/kompress-v2-base",
+    "score_threshold": 0.5
+  },
+  "input": "Once the migration completes the application reads its runtime feature flags from the host mounted configuration file and hot reloads them on change without requiring a full restart of the running service process. Once the migration completes the application reads its runtime feature flags from the host mounted configuration file and hot reloads them on change without requiring a full restart of the running service process. Once the migration completes the application reads its runtime feature flags from the host mounted configuration file and hot reloads them on change without requiring a full restart of the running service process. ",
+  "input_sha256": "74fde229e9465c817fdb3c259add19c4e20baa74730872dbd325edbc5688cbb4",
+  "output": {
+    "cache_key": null,
+    "compressed": "Once migration completes application reads its runtime feature flags from host mounted configuration file and hot reloads them on change without requiring a full restart running service process. Once migration completes application reads its runtime feature flags from host mounted configuration file hot reloads them on change without requiring a full restart running service process. Once migration completes application reads its runtime feature flags host mounted configuration file and hot reloads them on change without requiring full restart running service process.",
+    "compressed_tokens": 81,
+    "compression_ratio": 0.8181818181818182,
+    "model_used": "chopratejas/kompress-v2-base",
+    "original": "Once the migration completes the application reads its runtime feature flags from the host mounted configuration file and hot reloads them on change without requiring a full restart of the running service process. Once the migration completes the application reads its runtime feature flags from the host mounted configuration file and hot reloads them on change without requiring a full restart of the running service process. Once the migration completes the application reads its runtime feature flags from the host mounted configuration file and hot reloads them on change without requiring a full restart of the running service process. ",
+    "original_tokens": 99
+  },
+  "recorded_at": "2026-06-18T23:18:31.939337+00:00",
+  "transform": "kompress"
+}

tests/parity/fixtures/kompress/8417b9b5a308609c.json

@@ -0,0 +1,22 @@
+{
+  "config": {
+    "chunk_words": 350,
+    "device": "auto",
+    "enable_ccr": false,
+    "model_id": "chopratejas/kompress-v2-base",
+    "score_threshold": 0.5
+  },
+  "input": "Configure the service by setting config.timeout = 30 and config.retries = 5 then restart /etc/service/daemon to apply; verify via /health endpoint returning 200 within the configured start_period window of fifteen seconds. Configure the service by setting config.timeout = 30 and config.retries = 5 then restart /etc/service/daemon to apply; verify via /health endpoint returning 200 within the configured start_period window of fifteen seconds. Configure the service by setting config.timeout = 30 and config.retries = 5 then restart /etc/service/daemon to apply; verify via /health endpoint returning 200 within the configured start_period window of fifteen seconds. ",
+  "input_sha256": "8417b9b5a308609ce3c482d6925ca0c01fc9c33dc4f918427faecb528acc8385",
+  "output": {
+    "cache_key": null,
+    "compressed": "Configure service by setting config.timeout = 30 and config.retries = 5 then restart /etc/service/daemon to apply; verify via /health endpoint returning 200 within configured start_period window fifteen seconds. Configure service by setting config.timeout = 30 config.retries = 5 then restart /etc/service/daemon to apply; verify via /health endpoint returning 200 within configured start_period window fifteen seconds. Configure service by setting config.timeout = 30 config.retries = 5 then restart /etc/service/daemon to apply; verify via /health endpoint returning 200 within configured start_period window fifteen seconds.",
+    "compressed_tokens": 82,
+    "compression_ratio": 0.8817204301075269,
+    "model_used": "chopratejas/kompress-v2-base",
+    "original": "Configure the service by setting config.timeout = 30 and config.retries = 5 then restart /etc/service/daemon to apply; verify via /health endpoint returning 200 within the configured start_period window of fifteen seconds. Configure the service by setting config.timeout = 30 and config.retries = 5 then restart /etc/service/daemon to apply; verify via /health endpoint returning 200 within the configured start_period window of fifteen seconds. Configure the service by setting config.timeout = 30 and config.retries = 5 then restart /etc/service/daemon to apply; verify via /health endpoint returning 200 within the configured start_period window of fifteen seconds. ",
+    "original_tokens": 93
+  },
+  "recorded_at": "2026-06-18T23:18:25.692106+00:00",
+  "transform": "kompress"
+}

tests/parity/fixtures/kompress/96fbac155a26ca59.json

@@ -0,0 +1,22 @@
+{
+  "config": {
+    "chunk_words": 350,
+    "device": "auto",
+    "enable_ccr": false,
+    "model_id": "chopratejas/kompress-v2-base",
+    "score_threshold": 0.5
+  },
+  "input": "only nine words here so it passes through cleanly",
+  "input_sha256": "96fbac155a26ca59e2f81c6fd1ac0e45c11ad41c87225ac73d881c652be18b60",
+  "output": {
+    "cache_key": null,
+    "compressed": "only nine words here so it passes through cleanly",
+    "compressed_tokens": 9,
+    "compression_ratio": 1.0,
+    "model_used": "chopratejas/kompress-v2-base",
+    "original": "only nine words here so it passes through cleanly",
+    "original_tokens": 9
+  },
+  "recorded_at": "2026-06-18T23:18:19.224096+00:00",
+  "transform": "kompress"
+}

tests/parity/fixtures/kompress/9a0fba0115477c8b.json

@@ -0,0 +1,22 @@
+{
+  "config": {
+    "chunk_words": 350,
+    "device": "auto",
+    "enable_ccr": false,
+    "model_id": "chopratejas/kompress-v2-base",
+    "score_threshold": 0.5
+  },
+  "input": "The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. ",
+  "input_sha256": "9a0fba0115477c8b17d30763a33c036f22a9d8bd4e503a82a60cae9f39fd377d",
+  "output": {
+    "cache_key": null,
+    "compressed": "The quick brown fox jumps over lazy dog. quick brown fox jumps over lazy dog. quick brown fox jumps over the lazy dog.",
+    "compressed_tokens": 23,
+    "compression_ratio": 0.8518518518518519,
+    "model_used": "chopratejas/kompress-v2-base",
+    "original": "The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. ",
+    "original_tokens": 27
+  },
+  "recorded_at": "2026-06-18T23:18:24.045187+00:00",
+  "transform": "kompress"
+}

tests/parity/fixtures/kompress/9a4c0b3a49037be2.json

@@ -0,0 +1,22 @@
+{
+  "config": {
+    "chunk_words": 350,
+    "device": "auto",
+    "enable_ccr": false,
+    "model_id": "chopratejas/kompress-v2-base",
+    "score_threshold": 0.5
+  },
+  "input": "The committee reviewed the quarterly report and concluded that revenue grew twelve percent while operating costs declined by four percent year over year across all major regional markets surveyed. The committee reviewed the quarterly report and concluded that revenue grew twelve percent while operating costs declined by four percent year over year across all major regional markets surveyed. The committee reviewed the quarterly report and concluded that revenue grew twelve percent while operating costs declined by four percent year over year across all major regional markets surveyed. The committee reviewed the quarterly report and concluded that revenue grew twelve percent while operating costs declined by four percent year over year across all major regional markets surveyed. ",
+  "input_sha256": "9a4c0b3a49037be270b7493d9844736f27108e55da49a09979446c100d9a8116",
+  "output": {
+    "cache_key": null,
+    "compressed": "committee reviewed quarterly report concluded revenue grew twelve percent while operating costs declined by four percent year over year across all major regional markets surveyed. committee reviewed quarterly report concluded revenue grew twelve percent while operating costs declined by four percent year over year across all major regional markets surveyed. committee reviewed quarterly report concluded revenue grew twelve percent while operating costs declined by four percent year over year across all major regional markets surveyed. committee reviewed quarterly report concluded revenue grew twelve percent while operating costs declined by four percent year over year across all major regional markets surveyed.",
+    "compressed_tokens": 100,
+    "compression_ratio": 0.8620689655172413,
+    "model_used": "chopratejas/kompress-v2-base",
+    "original": "The committee reviewed the quarterly report and concluded that revenue grew twelve percent while operating costs declined by four percent year over year across all major regional markets surveyed. The committee reviewed the quarterly report and concluded that revenue grew twelve percent while operating costs declined by four percent year over year across all major regional markets surveyed. The committee reviewed the quarterly report and concluded that revenue grew twelve percent while operating costs declined by four percent year over year across all major regional markets surveyed. The committee reviewed the quarterly report and concluded that revenue grew twelve percent while operating costs declined by four percent year over year across all major regional markets surveyed. ",
+    "original_tokens": 116
+  },
+  "recorded_at": "2026-06-18T23:18:31.596517+00:00",
+  "transform": "kompress"
+}

tests/parity/fixtures/kompress/9ef7db920a3d84e9.json

@@ -0,0 +1,22 @@
+{
+  "config": {
+    "chunk_words": 350,
+    "device": "auto",
+    "enable_ccr": false,
+    "model_id": "chopratejas/kompress-v2-base",
+    "score_threshold": 0.5
+  },
+  "input": "tokens   with\tirregular\n\nwhitespace   runs   and\ttabs   scattered throughout the body of the text repeatedly again and again here throughout the body of the text repeatedly again and again here throughout the body of the text repeatedly again and again here throughout the body of the text repeatedly again and again here throughout the body of the text repeatedly again and again here ",
+  "input_sha256": "9ef7db920a3d84e9c49aab459c9aa98b5eb4e41a58aab6ec4464b3270b07334c",
+  "output": {
+    "cache_key": null,
+    "compressed": "tokens irregular whitespace runs tabs scattered throughout body text repeatedly again again here throughout body text repeatedly again again here throughout body text repeatedly again again here throughout body text repeatedly again again here throughout body text repeatedly again again here",
+    "compressed_tokens": 41,
+    "compression_ratio": 0.6507936507936508,
+    "model_used": "chopratejas/kompress-v2-base",
+    "original": "tokens   with\tirregular\n\nwhitespace   runs   and\ttabs   scattered throughout the body of the text repeatedly again and again here throughout the body of the text repeatedly again and again here throughout the body of the text repeatedly again and again here throughout the body of the text repeatedly again and again here throughout the body of the text repeatedly again and again here ",
+    "original_tokens": 63
+  },
+  "recorded_at": "2026-06-18T23:18:25.473143+00:00",
+  "transform": "kompress"
+}

tests/parity/fixtures/kompress/a901398a6751d2d2.json

@@ -0,0 +1,22 @@
+{
+  "config": {
+    "chunk_words": 350,
+    "device": "auto",
+    "enable_ccr": false,
+    "model_id": "chopratejas/kompress-v2-base",
+    "score_threshold": 0.5
+  },
+  "input": "The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. ",
+  "input_sha256": "a901398a6751d2d2fd3bd22de4f7962cd5e1bdbc8d84f085a5b8a283c6f629fa",
+  "output": {
+    "cache_key": null,
+    "compressed": "quick brown fox jumps over lazy dog. quick brown fox jumps over lazy dog. quick brown fox jumps over lazy dog. quick brown fox jumps over lazy dog. Meanwhile diligent engineer reviews compression output carefully ensure most salient tokens survive while redundant filler is discarded without losing essential meaning. Meanwhile diligent engineer reviews compression output carefully ensure most salient tokens survive while redundant filler is discarded without losing essential meaning.",
+    "compressed_tokens": 70,
+    "compression_ratio": 0.7954545454545454,
+    "model_used": "chopratejas/kompress-v2-base",
+    "original": "The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. ",
+    "original_tokens": 88
+  },
+  "recorded_at": "2026-06-18T23:18:24.515863+00:00",
+  "transform": "kompress"
+}

tests/parity/fixtures/kompress/a92c8ac9342234f5.json

@@ -0,0 +1,22 @@
+{
+  "config": {
+    "chunk_words": 350,
+    "device": "auto",
+    "enable_ccr": false,
+    "model_id": "chopratejas/kompress-v2-base",
+    "score_threshold": 0.5
+  },
+  "input": "ERROR 2026-06-18 connection refused to upstream after 3 retries; WARN falling back to cache; INFO request completed in 421ms status 200; DEBUG tracing span closed for request abc123; ERROR 2026-06-18 connection refused to upstream after 3 retries; WARN falling back to cache; INFO request completed in 421ms status 200; DEBUG tracing span closed for request abc123; ERROR 2026-06-18 connection refused to upstream after 3 retries; WARN falling back to cache; INFO request completed in 421ms status 200; DEBUG tracing span closed for request abc123; ",
+  "input_sha256": "a92c8ac9342234f5b88932a9b51db8ddaf634e0429f4d6d69f5d763ea8e0abb7",
+  "output": {
+    "cache_key": null,
+    "compressed": "ERROR 2026-06-18 connection refused upstream after 3 retries; WARN falling back cache; INFO request completed in 421ms status 200; DEBUG tracing span closed request abc123; ERROR 2026-06-18 connection refused upstream after 3 retries; WARN falling back cache; INFO request completed in 421ms status 200; DEBUG tracing span closed request abc123; ERROR 2026-06-18 connection refused upstream after 3 retries; WARN falling back to cache; INFO request completed in 421ms status 200; DEBUG tracing span closed request abc123;",
+    "compressed_tokens": 76,
+    "compression_ratio": 0.9047619047619048,
+    "model_used": "chopratejas/kompress-v2-base",
+    "original": "ERROR 2026-06-18 connection refused to upstream after 3 retries; WARN falling back to cache; INFO request completed in 421ms status 200; DEBUG tracing span closed for request abc123; ERROR 2026-06-18 connection refused to upstream after 3 retries; WARN falling back to cache; INFO request completed in 421ms status 200; DEBUG tracing span closed for request abc123; ERROR 2026-06-18 connection refused to upstream after 3 retries; WARN falling back to cache; INFO request completed in 421ms status 200; DEBUG tracing span closed for request abc123; ",
+    "original_tokens": 84
+  },
+  "recorded_at": "2026-06-18T23:18:24.713787+00:00",
+  "transform": "kompress"
+}

tests/parity/fixtures/kompress/b4fc3e13ca821a10.json

@@ -0,0 +1,22 @@
+{
+  "config": {
+    "chunk_words": 350,
+    "device": "auto",
+    "enable_ccr": false,
+    "model_id": "chopratejas/kompress-v2-base",
+    "score_threshold": 0.5
+  },
+  "input": "The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. ",
+  "input_sha256": "b4fc3e13ca821a109371cbcd98e4b7920c9b46d90d8ab367e7927c4c5f5fbd96",
+  "output": {
+    "cache_key": null,
+    "compressed": "quick brown fox jumps over lazy dog. Meanwhile diligent engineer reviews compression output carefully ensure most salient tokens survive redundant filler is discarded without losing essential meaning. quick brown fox jumps over lazy dog. Meanwhile diligent engineer reviews compression output carefully ensure most salient tokens survive redundant filler is discarded without losing essential meaning. quick brown fox jumps over lazy dog. Meanwhile diligent engineer reviews compression output carefully ensure most salient tokens survive redundant filler is discarded without losing essential meaning. quick brown fox jumps over lazy dog. Meanwhile diligent engineer reviews compression output carefully ensure most salient tokens survive redundant filler is discarded without losing essential meaning. quick brown fox jumps over lazy dog. Meanwhile diligent engineer reviews compression output carefully ensure most salient tokens survive redundant filler is discarded without losing essential meaning. quick brown fox jumps over lazy dog. Meanwhile diligent engineer reviews compression output carefully ensure most salient tokens survive redundant filler is discarded without losing essential meaning. quick brown fox jumps over lazy dog. Meanwhile diligent engineer reviews compression output carefully ensure most salient tokens survive redundant filler is discarded without losing essential meaning. quick brown fox jumps over lazy dog. Meanwhile diligent engineer reviews compression output carefully ensure most salient tokens survive redundant filler is discarded without losing essential meaning. quick brown fox jumps over lazy dog. Meanwhile diligent engineer reviews compression output carefully ensure most salient tokens survive redundant filler is discarded without losing essential meaning. quick brown fox jumps over lazy dog. Meanwhile diligent engineer reviews compression output carefully ensure most salient tokens survive while redundant filler quick brown fox jumps over lazy dog. Meanwhile diligent engineer reviews compression output carefully ensure most salient tokens survive redundant filler is discarded without losing essential meaning. quick brown fox jumps over lazy dog. Meanwhile diligent engineer reviews compression output carefully ensure most salient tokens survive redundant filler is discarded without losing essential meaning. quick brown fox jumps over lazy dog. Meanwhile diligent engineer reviews compression output carefully ensure most salient tokens survive redundant filler is discarded without losing essential meaning. quick brown fox jumps over lazy dog. Meanwhile diligent engineer reviews compression output carefully ensure most salient tokens survive redundant filler is discarded without losing essential meaning. quick brown fox jumps over lazy dog. Meanwhile diligent engineer reviews compression output carefully ensure most salient tokens survive redundant filler is discarded without losing essential meaning. quick brown fox jumps over lazy dog. Meanwhile diligent engineer reviews compression output carefully ensure most salient tokens survive redundant filler is discarded without losing essential meaning. quick brown fox jumps over lazy dog. Meanwhile diligent engineer reviews compression output carefully ensure most salient tokens survive redundant filler is discarded without losing essential meaning. quick brown fox jumps over lazy dog. Meanwhile diligent engineer reviews compression output carefully ensure most salient tokens survive redundant filler is discarded without losing essential meaning. quick brown fox jumps over lazy dog. Meanwhile diligent engineer reviews compression output carefully ensure most salient tokens survive redundant filler is discarded without losing essential meaning. quick brown fox jumps over lazy dog. Meanwhile diligent engineer reviews compression output carefully ensure most salient tokens survive while redundant filler",
+    "compressed_tokens": 530,
+    "compression_ratio": 0.7571428571428571,
+    "model_used": "chopratejas/kompress-v2-base",
+    "original": "The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. The quick brown fox jumps over the lazy dog. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. ",
+    "original_tokens": 700
+  },
+  "recorded_at": "2026-06-18T23:18:27.216565+00:00",
+  "transform": "kompress"
+}

tests/parity/fixtures/kompress/d9d334193be8ddeb.json

@@ -0,0 +1,22 @@
+{
+  "config": {
+    "chunk_words": 350,
+    "device": "auto",
+    "enable_ccr": false,
+    "model_id": "chopratejas/kompress-v2-base",
+    "score_threshold": 0.5
+  },
+  "input": "Performance benchmarks on the reference hardware show the int8 weight only model matching the full precision baseline within one tenth of a percent on the held out evaluation split across five hundred samples. Performance benchmarks on the reference hardware show the int8 weight only model matching the full precision baseline within one tenth of a percent on the held out evaluation split across five hundred samples. Performance benchmarks on the reference hardware show the int8 weight only model matching the full precision baseline within one tenth of a percent on the held out evaluation split across five hundred samples. ",
+  "input_sha256": "d9d334193be8ddeb11c1dfa5c88b2ee0c14822bb8576d806e8fa0b19a09443c3",
+  "output": {
+    "cache_key": null,
+    "compressed": "Performance benchmarks on reference hardware show int8 weight only model matching full precision baseline within one tenth percent on held out evaluation split across five hundred samples. Performance benchmarks on reference hardware show int8 weight only model matching full precision baseline within one tenth percent on held out evaluation split across five hundred samples. Performance benchmarks on reference hardware show int8 weight only model matching full precision baseline within one tenth percent on held out evaluation split across five hundred samples.",
+    "compressed_tokens": 81,
+    "compression_ratio": 0.8181818181818182,
+    "model_used": "chopratejas/kompress-v2-base",
+    "original": "Performance benchmarks on the reference hardware show the int8 weight only model matching the full precision baseline within one tenth of a percent on the held out evaluation split across five hundred samples. Performance benchmarks on the reference hardware show the int8 weight only model matching the full precision baseline within one tenth of a percent on the held out evaluation split across five hundred samples. Performance benchmarks on the reference hardware show the int8 weight only model matching the full precision baseline within one tenth of a percent on the held out evaluation split across five hundred samples. ",
+    "original_tokens": 99
+  },
+  "recorded_at": "2026-06-18T23:18:32.081771+00:00",
+  "transform": "kompress"
+}

tests/parity/fixtures/kompress/e3387b34a2e2f5e0.json

@@ -0,0 +1,22 @@
+{
+  "config": {
+    "chunk_words": 350,
+    "device": "auto",
+    "enable_ccr": false,
+    "model_id": "chopratejas/kompress-v2-base",
+    "score_threshold": 0.5
+  },
+  "input": "tiny input",
+  "input_sha256": "e3387b34a2e2f5e0bef68eb40e6696537a8573863aeb81c67827a15543b4515a",
+  "output": {
+    "cache_key": null,
+    "compressed": "tiny input",
+    "compressed_tokens": 2,
+    "compression_ratio": 1.0,
+    "model_used": "chopratejas/kompress-v2-base",
+    "original": "tiny input",
+    "original_tokens": 2
+  },
+  "recorded_at": "2026-06-18T23:18:19.224283+00:00",
+  "transform": "kompress"
+}

tests/parity/fixtures/kompress/e63c879a46621bdb.json

@@ -0,0 +1,22 @@
+{
+  "config": {
+    "chunk_words": 350,
+    "device": "auto",
+    "enable_ccr": false,
+    "model_id": "chopratejas/kompress-v2-base",
+    "score_threshold": 0.5
+  },
+  "input": "Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. ",
+  "input_sha256": "e63c879a46621bdb85e0063a716c136f8b11c05433ac315c86c9e536894ed004",
+  "output": {
+    "cache_key": null,
+    "compressed": "Meanwhile diligent engineer reviews compression output carefully ensure most salient tokens survive while redundant filler is discarded without losing essential meaning. Meanwhile diligent engineer reviews compression output carefully ensure most salient tokens survive while redundant filler is discarded without losing essential meaning.",
+    "compressed_tokens": 42,
+    "compression_ratio": 0.8076923076923077,
+    "model_used": "chopratejas/kompress-v2-base",
+    "original": "Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. Meanwhile the diligent engineer reviews the compression output carefully to ensure the most salient tokens survive while redundant filler is discarded without losing the essential meaning. ",
+    "original_tokens": 52
+  },
+  "recorded_at": "2026-06-18T23:18:24.364083+00:00",
+  "transform": "kompress"
+}

tests/parity/recorder.py

@@ -280,6 +280,20 @@ def record_all(root: Path | None = None) -> dict[str, str]:
     except Exception as e:
         statuses["content_detector"] = f"blocked:{e.__class__.__name__}:{e}"
 
+    # --- kompress ----------------------------------------------------------
+    # ML prose compressor. Requires onnxruntime + the kompress-v2-base model
+    # cached locally; "blocked" (soft) when either is missing so recording
+    # still succeeds for the deterministic transforms. The workload driver
+    # constructs it with enable_ccr=False so the recorded output is the pure
+    # joined kept-word stream (deterministic, store-independent).
+    try:
+        from headroom.transforms.kompress_compressor import KompressCompressor
+
+        _wrap_method(KompressCompressor, "compress", "kompress", root=root)
+        statuses["kompress"] = "patched"
+    except Exception as e:
+        statuses["kompress"] = f"blocked:{e.__class__.__name__}:{e}"
+
     return statuses
 
 
@@ -750,6 +764,7 @@ def run_default_workload(root: Path | None = None) -> dict[str, int]:
         "cache_aligner": 0,
         "ccr": 0,
         "content_detector": 0,
+        "kompress": 0,
     }
 
     # log_compressor
@@ -839,9 +854,92 @@ def run_default_workload(root: Path | None = None) -> dict[str, int]:
     except Exception as e:
         LOG.warning("content_detector workload failed: %s", e)
 
+    # kompress — ML prose compressor. enable_ccr=False so the recorded
+    # `compressed` is the deterministic joined kept-word stream. Soft-fails
+    # when onnxruntime / the model are unavailable (the Rust comparator
+    # likewise skips when the model is not cached).
+    try:
+        from headroom.transforms.kompress_compressor import (
+            KompressCompressor,
+            KompressConfig,
+        )
+
+        kc = KompressCompressor(KompressConfig(enable_ccr=False))
+        for s in _varied_kompress_inputs():
+            kc.compress(s)
+            counts["kompress"] += 1
+    except Exception as e:
+        LOG.warning("kompress workload failed: %s", e)
+
     return counts
 
 
+def _varied_kompress_inputs() -> list[str]:
+    """≥20 varied prose/log/mixed inputs for the Kompress ML compressor.
+
+    Spans short passthrough (<10 words), single-chunk prose, multi-chunk
+    bodies (>350 words), whitespace-irregular text, and log/error-like
+    content so the recorded fixtures cover chunking, max-score-per-word
+    reduction, the >0.5 threshold, and the passthrough short-circuit.
+    """
+    fox = "The quick brown fox jumps over the lazy dog. "
+    engineer = (
+        "Meanwhile the diligent engineer reviews the compression output "
+        "carefully to ensure the most salient tokens survive while redundant "
+        "filler is discarded without losing the essential meaning. "
+    )
+    log = (
+        "ERROR 2026-06-18 connection refused to upstream after 3 retries; "
+        "WARN falling back to cache; INFO request completed in 421ms status 200; "
+        "DEBUG tracing span closed for request abc123; "
+    )
+    inputs: list[str] = [
+        # short → passthrough
+        "only nine words here so it passes through cleanly",
+        "tiny input",
+        "just a handful of words below the threshold now",
+        # single-chunk prose of growing length
+        fox * 3,
+        fox * 6 + engineer,
+        engineer * 2,
+        fox * 4 + engineer * 2,
+        # log / error-like
+        log * 3,
+        log * 6,
+        "Traceback (most recent call last): File main.py line 42 in handler "
+        "raise ValueError invalid token; the request could not be processed "
+        "because the upstream returned an unexpected payload shape repeatedly. " * 3,
+        # whitespace-irregular
+        "tokens   with\tirregular\n\nwhitespace   runs   and\ttabs   scattered "
+        + "throughout the body of the text repeatedly again and again here " * 5,
+        # mixed prose + paths + assignments
+        "Configure the service by setting config.timeout = 30 and config.retries = 5 "
+        "then restart /etc/service/daemon to apply; verify via /health endpoint "
+        "returning 200 within the configured start_period window of fifteen seconds. " * 3,
+        # multi-chunk (>350 words)
+        (fox + engineer) * 20,
+        (log + engineer) * 18,
+        fox * 120,
+        # medium prose variations
+        engineer + fox * 5,
+        "Summarize the following document while preserving every named entity "
+        "and numeric figure so the reader can reconstruct the key facts later. " * 4,
+        "The committee reviewed the quarterly report and concluded that revenue "
+        "grew twelve percent while operating costs declined by four percent "
+        "year over year across all major regional markets surveyed. " * 4,
+        "Installation requires Python three point ten or newer along with the "
+        "optional machine learning extra which pulls onnxruntime and transformers "
+        "for the token compression model used by the proxy at request time. " * 3,
+        "Once the migration completes the application reads its runtime feature "
+        "flags from the host mounted configuration file and hot reloads them on "
+        "change without requiring a full restart of the running service process. " * 3,
+        "Performance benchmarks on the reference hardware show the int8 weight "
+        "only model matching the full precision baseline within one tenth of a "
+        "percent on the held out evaluation split across five hundred samples. " * 3,
+    ]
+    return inputs
+
+
 __all__ = [
     "record",
     "record_all",