`headroom/transforms/tag_protector.py` was a regex-driven scan-and-
replace loop that ran on every kompress call from ContentRouter
(`content_router.py:1089`). The Python implementation had five real
bugs we now fix in the port — the most consequential being a
`str.replace(.., .., 1)` first-occurrence-replace bug that silently
collapsed two identical custom-tag blocks in the same input to a
single placeholder + a stray duplicate of the second block.
# Bug fixes (each pinned by a `fixed_in_3e4` test)
* **#1: O(n²) on nested custom tags.** Python's `while changed` loop
restarted a full regex scan after every replacement. Rust walks
once in linear time on input length.
* **#2: First-occurrence replace bug.** `result.replace(orig, ph, 1)`
replaces the FIRST textual match, not the matched offset. Two
identical custom-tag blocks collapsed to one placeholder + a stray
duplicate of the second block. The Rust walker stitches output by
offset so distinct blocks always get distinct placeholders.
* **#3: Silent 50-iteration cap.** Python had a hard `max_iterations
= 50` safety limit that quietly truncated tag protection on deeply
nested input. The Rust walker is bounded by input length only.
* **#4: Self-closing pass duplicate-replace risk.** Python ran a
second loop with the same `replace_first` bug for self-closers.
Rust handles self-closers in the same single pass.
* **#5: Placeholder collision.** If the input contained a literal
`{{HEADROOM_TAG_…}}` substring, Python silently let the collision
break restoration. Rust salts the prefix and reports it in stats.
# Architecture
Two-phase walker:
* Phase 1 (`identify_spans`): linear scan over input bytes, hand-
rolled tag-open / tag-close lexer (no regex). Maintains a stack of
open custom tags; on a matching close, collapses the inner span
into a single `Span { start, end, Block }`. Self-closing custom
tags become `Span { ..., SelfClosing }` immediately. Marker-only
mode (`compress_tagged_content=true`) emits Open/CloseMarker spans
instead. Orphan opens stay un-protected (matches Python behavior).
Orphan closes are emitted verbatim and counted in stats.
* Phase 2 (`emit_output`): walks `text` once, splicing placeholders
for span ranges and copying everything else verbatim. Offset-based,
never `str.replace`.
PyO3 surface: `protect_tags`, `restore_tags`, `is_html_tag`,
`known_html_tag_names`. The Python shim retires the regex internals
and re-exports `KNOWN_HTML_TAGS` (rebuilt from the Rust list) +
`_is_html_tag` for backwards compat with `content_router.py` and the
existing test surface.
# Test plan
* 25 Rust unit tests including 4 `fixed_in_3e4_*` bug-fix tests
* 27 Python tests (23 existing + 4 new `fixed_in_3e4` parity tests)
* 5 integration tests in `test_tag_protection_integration.py` pass
* `make ci-precheck` clean
`headroom/transforms/query_echo.py` was already disabled in all three
proxy handlers (`anthropic.py`, `openai.py`, `gemini.py`) — each
carried the same comment: 'disabled — hurts prefix caching in long
conversations. The echo changes every turn, invalidating the cached
prefix.' That call was right: the echo's per-turn variability would
bust the Anthropic/OpenAI/Gemini prompt cache and cost more in TTFT
than the recall benefit ever paid back. The module was orphaned but
still living in the tree, with its own 70-test file pinning the
disabled behavior.
# Removed
* `headroom/transforms/query_echo.py` (123 LOC)
* `tests/test_query_echo.py` (whole file)
* The three 'disabled — hurts prefix caching' comment blocks in the
proxy handlers (the rationale lives in this commit message and the
PR description; no need to leave dead-code breadcrumbs in the hot
path).
# Kept
* `headroom.utils.extract_user_query` is a different function with
the same name and is still used elsewhere — untouched.
# Test plan
* `make ci-precheck` clean
* No remaining references to `query_echo`/`QueryEcho`/`Query Echo`/
`inject_query_echo` in the tree.
`headroom/transforms/text_compressor.py` was a regex-line-sampling
fallback that nothing in the runtime called. ContentRouter routes
`CompressionStrategy.TEXT` straight to the Kompress ML compressor at
`content_router.py:1046` — the comment there literally says 'Prefer
Kompress ML compressor for text'. The Python file was orphaned but
still imported by its own test class, making it look live in the 3e
queue.
Drops the 3e.3 port from the queue: there's nothing to port.
# Removed
* `headroom/transforms/text_compressor.py` (255 LOC, unused)
* `tests/test_text_compressors.py::TestTextCompressor` (3 tests)
* `text_compressor` mention in `error_detection.py` shim docstring
* `text_compressor` mention in `test_signals_keyword_parity.py` docstring
* `TextCompressor` mention in `bench_latency.py` scenario comment
# Kept (defensive)
The legacy marker regex in `ccr/tool_injection.py:213` stays — it
parses an even older TextCompressor output format (pre-2026), is
purely defensive, and removal buys nothing. Test references to that
format in `test_ccr_tool_injection.py` document the regex contract
and stay too.
# Test plan
* `make ci-precheck` clean
* `tests/test_text_compressors.py` 19 passes (was 22, dropped 3)
Ports `headroom.transforms.log_compressor` to Rust. The biggest-by-
impact remaining compressor port: build/test logs are where the
10-50x compression wins live.
* Stack-trace state machine: per-flavor dispatcher (Python Traceback,
JS, Java, Rust error, Go); each flavor has its own termination
rule. Python terminated on any blank line, dropping mid-trace
lines from chained-exception traces.
* Conservative dedupe: preserves message prefix (everything before
first `:` or `=`); only trailing region is tokenised. Python's
blanket normalisation collapsed segfaults at different addresses.
* Loud CCR failures: `tracing::warn!` + `logger.warning` instead of
bare `except: pass`.
* `LogLevel::FAIL` documented as cosmetic-equivalent to ERROR.
Same shape as search_compressor port. Rust `LogCompressor`
orchestrates format detect -> classify -> score -> select ->
format -> CCR. Inline static-table format detector (YAGNI),
aho-corasick level classifier with word-boundary post-filter
(`signals::keyword_detector` technique), hand-rolled per-flavor
stack-trace state machine. `signals::LineImportanceDetector` NOT
consumed -- log levels are structural, not prose-style importance.
`headroom.transforms.log_compressor` becomes a thin shim:
`compress()` delegates to Rust end-to-end; internal helpers
preserved for the existing 50-test surface. Two existing tests
updated for new dedupe semantics + new compress orchestration.
* 17 Rust unit tests
* 50 Python tests pass
* `make ci-precheck` clean
The 12 langchain integration evals generate fixture data via
`random.choice`/`random.randint` without seeding. SmartCrusher's
anchor selection consumes the same global random state, so a handful
of unseeded inputs (~1% of seed values) skip the first/last anchor
preservation and the eval flakes — surfaced on PR #319 CI even though
this PR doesn't touch SmartCrusher.
Confirmed pre-existing: identical 5/500 seed failures on `main`
@ `cf3877d`. The fix is the smallest one that doesn't paper over the
underlying selector behavior — seed `random` per-test via an autouse
fixture so dataset generation is reproducible.
Ports `headroom.transforms.search_compressor` to Rust as the first
consumer of the `signals::LineImportanceDetector` trait shipped in
Phase 3e.1.
The Python regex registry (`_GREP_PATTERN`/`_RG_CONTEXT_PATTERN`)
silently misparsed two real-world inputs. The hand-rolled Rust parser
fixes both:
* **Windows paths.** `^([^:]+):(\d+):(.*)$` captured only the drive
letter from `C:\Users\foo\bar.py:42:line`, then the `\d+` group
failed on `\`. Result: every Windows-formatted line was silently
dropped from `file_matches`. The Rust parser detects the drive
prefix and starts the line-number scan after the drive colon.
* **Filenames with `-`.** `_RG_CONTEXT_PATTERN`'s `[^:-]+` excluded
dashes from the path, so legitimate names like
`pre-commit-config.yaml-42-line` parsed wrong. The Rust parser
anchors on the *line-number marker* (`<sep>\d+<sep>`), so paths
can contain dashes freely.
Two further hardening changes:
* CCR storage failures are loud (Python silently swallowed them).
* Per-file dedup is `O(n log n)` via `BTreeSet<(line_no, content_hash)>`
(Python used linear `match not in file_selected`, worst-case
quadratic for big files).
The Rust `SearchCompressor` owns a `Box<dyn LineImportanceDetector>`
defaulting to `KeywordDetector`. Priority scoring routes through the
trait instead of a hardcoded regex list, so a future BGE classifier
head (per the trait extension docs) can take over without touching
the compressor.
Sidecar `SearchCompressorStats` captures lines unparsed, files
dropped by `max_files`, matches dropped by per-file vs global caps,
and the CCR skip reason -- diagnostics Python never emitted.
`headroom.transforms.search_compressor` is now a thin shim that
delegates `compress()` to Rust end-to-end (so the parser bug fixes
land in production), and keeps the legacy `_parse_search_results`
helper routed through the same Rust parser. The other internal
helpers (`_score_matches`, `_select_matches`, `_format_output`)
stay Python -- they're heavily covered by existing direct-call tests
and Rust scoring is byte-equivalent.
The 4 public dataclasses are unchanged. Tests that monkeypatched the
old internal `_store_in_ccr` helper are updated to exercise the new
`_persist_to_python_ccr` boundary instead.
* 16 Rust unit tests (parser, scoring, selection, CCR round-trip) +
3 explicit `fixed_in_3e2` markers for the bug-fix lines
* 53 Python tests (existing suite intact; 2 updated for new shape)
* `make ci-precheck` clean
Stacks on PR #317 (signals trait module).
This test pinned the pre-3e.1 behavior on three lines that the new
KeywordDetector intentionally changes:
1. `'token'` was asserted to be in SECURITY_KEYWORDS. It was dropped
from the security set in 3e.1 because it false-positived on every
LLM-token reference in our own product. Updated to assert the new
set (`security|password|auth|secret`) and explicitly that `token`
is gone.
2. SECURITY_PATTERN was tested via "rotate the auth token" (matched
via the now-removed `token` keyword). Now tested via "rotate the
auth header" (matches via `auth`, which is the real security
signal) plus a negative assertion that LLM-metric strings no
longer fire.
3. ERROR_PATTERN test added an assertion that "Connection timeout"
now flags as an error (3e.1 fixed the keyword/regex drift).
PRIORITY_PATTERNS_TEXT indices were also corrected: the Rust-supplied
markdown_prefixes table is ordered `# `, `## `, `### `, `#### `, `**`,
`> ` (six prefixes) so the bold/blockquote assertions live at indices
6 and 7, not 3 and 4. New `# ` and `## ` checks pin the lower indices.
Each diverging assertion carries a `fixed_in_3e1` comment so the
audit trail stays clear.
Establish `crates/headroom-core/src/signals/` as a top-level module
holding cross-cutting detection traits. Phase 3e.1 ports
`error_detection.py` to a `LineImportanceDetector` trait + a
`Tiered<T>` combinator + a single concrete `KeywordDetector` impl
backed by aho-corasick. Three traits at three granularities are
sketched (line / blob / item); only line-importance is implemented
today.
Two bug fixes from the Python source bake into both the Rust impl
and the Python regex shim:
1. `ERROR_KEYWORDS` listed `timeout|abort|denied|rejected` but
`ERROR_PATTERN` regex omitted them. Lines like `"Connection
timeout"` were silently neutral despite the keyword being canonical.
Both surfaces now flag them.
2. `SECURITY_KEYWORDS` carried `token`, which false-positived on
every reference to LLM tokens (`input_tokens`, `tokens_saved`, ...)
in our own product. Dropped from the security set.
The Python `error_detection.py` shim now reflects keyword data out of
Rust via `keyword_registry_snapshot()` and recompiles the legacy
`re.Pattern` objects on the fly. Existing callers (text_compressor,
search_compressor, intelligent_context) continue to import the same
names with no source changes; caller migration to the trait API
happens in their own port PRs.
The trait architecture is the seam where a future ML detector slots
in without touching `KeywordDetector` or any caller. The canonical
extension is documented in `signals/README.md` as a classifier head
on the existing `bge-small-en-v1.5` embedder loaded by
`relevance::EmbeddingScorer` -- 384-dim -> 4-class softmax,
~1.5 KB head, ~1 ms inference, no extra model file. Two alternatives
(distilled tinyBERT in ONNX, logistic regression on lexical
features) are kept open in case BGE-head underfits.
Per the no-silent-fallbacks rule: only `KeywordDetector` lands as a
concrete impl. No NoOp, no MockDetector, no stub-ML -- those will
arrive with their real implementations.
Phase 3g (Compression Pipeline Formalization, issue #315) is queued
as the cross-cutting follow-up that will make lossless-then-lossy-
then-CCR ordering an explicit, observable architecture rather than
implicit per-compressor logic. Trait shapes there will reuse the
signals primitive landed in this PR.
The PrometheusMetrics `compressions_by_strategy` and
`tokens_saved_by_strategy` counters (PR #302) were tracked in process
but never exported, because the Prometheus->Supabase pipeline treats
each metric name as a separate column. The data clock for the
code_compressor port-vs-retire decision needed actual production
visibility, not just CI assertions.
Add both dicts to the `/stats` endpoint and nest them under
`pipeline_timing._strategies` in the beacon Supabase payload. The
existing `pipeline_timing` column is JSONB and absorbs the nested
shape -- zero schema change.
`_build_pipeline_timing` is extracted from the inline payload-builder
so the projection has its own unit tests, including a guard that the
`_strategies` sub-key is only emitted when at least one counter is
non-empty.
The Prometheus scrape still does NOT carry these metrics; the existing
`test_prometheus_export_does_not_leak_per_strategy_metrics` guard
remains in place.
Replaces the dispatch-path detection with the locked Stage-3d chain:
Tier 1: magika_detect() (PR3)
Tier 2: unidiff::is_diff() (PR4)
Tier 3: PlainText fall-through
The regex `content_detector` is no longer on the production path —
it stays in the tree as a comparison oracle (and for any direct
caller); a future PR retires it entirely.
What lands:
- `crates/headroom-core/src/transforms/detection.rs`: new `detect()`
function that chains the two tiers. Tier-1 errors log at WARN
level and continue to Tier 2 (the chain's *next* tier IS the
legitimate fallback for magika failure; treating tier-1 error as
hard-fail would block all detection on transient ONNX issues).
- 12 unit tests covering: empty, JSON, source code, HTML, standard
git diff, naked hunk (Tier 2 catch), prose, grep search results
(locked-design behavior change), build log, YAML, Rust source,
determinism across repeated calls.
- PyO3 binding `detect_content_type` now calls the chain. Synthesizes
the legacy `DetectionResult` shape (confidence=1.0, empty metadata)
since the chain doesn't surface a probabilistic score and no
production caller reads metadata from the binding today.
- Python `headroom/transforms/content_router.py`: `_detect_content`
now delegates to `headroom._core.detect_content_type`. The Python-
side `_get_magika_detector` + regex fallback is retired (single
detection surface; no parallel paths). Test for the helper rewritten
to monkeypatch the Rust binding instead of the old Python paths.
Behavior changes (per locked design):
- `SearchResults` and `BuildOutput` ContentTypes route to PlainText
(or SourceCode if magika happens to label it code-like) rather
than to specialized strategies. No regex tier on the Rust side,
per `project_rust_content_detection_arch.md`. If proxy benchmarks
show real loss on grep/build outputs, we add focused detectors
later — not preemptively.
Stacked on PR4 (unidiff). When PR4 squash-merges, this PR rebases
trivially against main.
Tests:
- 12 new Rust unit tests in `transforms::detection::tests`
- 43 Python content_router tests (was 42; old monkeypatch test
rewritten in place, not duplicated)
- `make ci-precheck` green
The previous _ListHandler approach attached a handler to the
headroom.proxy logger and worked locally on Python 3.14, but failed in
CI on Python 3.10-3.13 — the warning record never reached the handler.
Root cause is unclear (possibly cross-test logger state), but the
handler-attachment path is brittle for a single-warning assertion.
Replace it with unittest.mock.patch.object on the handler module's
logger.warning. This is invariant to logging hierarchy, propagation
flags, and per-test logger mutations — we directly observe the call
that the production code makes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Issue #296 reports compression appearing to complete successfully then
being discarded after a 30s timeout. The diagnostic gap that makes the
report hard to root-cause:
1. The pipeline's "Pipeline complete: ..." and "Pipeline: freezing
first ..." log lines have no request_id, so under concurrent load
the reporter cannot tell whether the success log is from the
timing-out request or a sibling request.
2. The handler's catch-all "Optimization failed: {e}" passes only
str(e), which is empty for asyncio.TimeoutError — the report shows
"Optimization failed:" with nothing after the colon.
This change is observability-only:
- pipeline.apply now reads request_id from kwargs and prefixes its two
INFO log lines with [request_id] when present.
- The four anthropic_pipeline.apply call sites in the Anthropic handler
(3 in handle_anthropic_messages, 1 in handle_anthropic_batch_create)
pass request_id through.
- The catch-all warning becomes
"[{request_id}] Optimization failed: {type(e).__name__}: {e}" so
TimeoutError is distinguishable from real exceptions in bug reports.
No behavior change. Adds two tests covering both diagnostics.
This is intentionally not a fix for #296 — the underlying timeout still
needs reproduction at 367k+ token transcripts. With these diagnostics in
place, the next bug report will be able to confirm whether the failing
request actually reached the pipeline.
Refs #296
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Re-lands two audit fixes that were marked "merged" on GitHub but never
reached main: squash-merging the parent stack changed its commit SHA,
which silently dropped the contents of the stacked PRs (#301, #305).
Single PR this time — no stacking risk.
What lands:
1. **`enable_ccr_marker` Rust gate** — new field on `SmartCrusherConfig`
(default `true`). `crush_array` checks it before emitting the
`<<ccr:HASH>>` marker text and the CCR store write. PyO3 surface +
parity-fixture tolerance updated; recorded fixtures predate the
field and inherit the `true` default.
2. **Python shim collapses both flags to the gate** — both
`ccr_config.enabled=False` and `ccr_config.inject_retrieval_marker
=False` now flip the Rust gate off. Storing a payload nothing in
the prompt can reference is pointless, and storing under
`enabled=False` would be a surprise side effect the user
explicitly opted out of.
3. **Custom `scorer` / `relevance_config` fails loud** — replaces the
prior WARNING-and-drop. Silently dropping a user-supplied scorer
is a textbook silent fallback. `NotImplementedError` instead.
Verified zero production callers pass these args; full plumbing
arrives with Stage-3c.2's relevance-crate Python bridge.
Tests:
- 2 new Rust unit tests in `crusher.rs::tests`
- 6 new Python tests in `test_smart_crusher_toin_attachment.py`
(3 CCR marker-knob behaviors + 3 scorer fail-loud)
- Removed `test_ccr_inject_marker_false_logs_warning` (the WARNING is
gone now that the flag is honored)
- `make ci-precheck` green; eval suite + observability tests run
twice consecutively to verify no TOIN file pollution leaks into the
regular+coverage double-run on Python 3.11
RUST_DEV.md audit table reflects both gaps closed.
The new SmartCrusher.apply() tests added in this PR feed the global
TOIN learning store (default path `~/.headroom/toin.json`). On Python
3.11, CI runs the suite twice (regular + coverage); a pattern written
in the first pass changes which rows the lossy sampler keeps in the
second pass and breaks
`tests/test_integrations/langchain/test_evals.py::TestRelevancePreservation::test_first_last_items_always_preserved`.
Fix: an `isolated_toin` fixture that points `HEADROOM_TOIN_PATH` at a
tempdir for the test and resets the singleton on enter and exit. Same
isolation pattern PR #300 already uses for `test_smart_crusher_toin_attachment.py`.
Move _setup_file_logging() from module import to create_app() so
importing headroom.proxy.server in tests or library contexts no longer
silently attaches a RotatingFileHandler to the user's live proxy.log.
This was discovered via PR #303: running the test suite produced
"Optimization failed: TimeoutError" entries in ~/.headroom/logs/proxy.log
because TestClient instantiations from test_proxy_anthropic_compression_diagnostics.py
inherited the module-level handler.
Add a regression test that imports the server module in a subprocess
and asserts no RotatingFileHandler is attached to the headroom logger.
Refs #303 (review thread).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
`caplog` flakes under the full suite — earlier tests can leave
`logging.disable()` or root-handler state that suppresses records on
the named logger before pytest's caplog filter sees them. The test
passed in isolation but failed deterministically once the larger
suite preceded it (CI run 25070639932 across all four Python
versions, ~4500 tests in).
Fix: monkeypatch `headroom.transforms.smart_crusher.logger.warning`
directly. Bypasses every level/disable/handler concern; we only
assert the constructor *called* `logger.warning` with the expected
flag name in the message — which is exactly the contract this test
guards.
Adds a `CompressionObserver` Protocol (`headroom.transforms.observability`)
and wires `ContentRouter` and `SmartCrusher` to call it once per real
compression event. `PrometheusMetrics` implements the protocol and
accumulates per-strategy counters (`compressions_by_strategy`,
`tokens_saved_by_strategy`).
Why: the TOIN→SmartCrusher silent disconnect was invisible for three
weeks because no signal distinguished by strategy. With per-strategy
counters in place, the next regression of that shape fails the test
suite the day it lands instead of waiting on a manual audit.
Counters live as in-process state only — deliberately NOT exported via
the Prometheus scrape or OTel surface. The metric→Supabase pipeline
treats each metric name as a column, and we cannot add new columns.
CI-level observability via `tests/test_compression_observability.py` is
sufficient to catch silent regressions; production export waits on a
non-column-adding pipeline.
The Stage 3c.1b retirement of the Python SmartCrusher silently disconnected
three subsystems. The audit on 2026-04-28 caught them; this commit fixes
what's fixable today and labels the rest visibly.
## TOIN learning loop — fixed
Before: `ContentRouter._record_to_toin` skipped SmartCrusher on the
assumption SmartCrusher recorded its own TOIN events. The retired Python
class did. The Rust port doesn't know about TOIN. Net result: the
highest-traffic compression strategy stopped fueling the learning loop,
silently.
Fix: shim's `crush()` and `_smart_crush_content()` now call
`toin.record_compression()` after a real compression. Filtered on
`strategy != "passthrough"` because the Rust port flips
`was_modified=True` from JSON whitespace re-canonicalization. Best
effort: TOIN failures are logged at debug level and never break
compression.
Token estimates use `len(json) // 4` (the rule the retired Python used)
because the router doesn't pass a tokenizer down to this layer and
re-tokenizing here would dominate the recording cost.
7 tests in `tests/test_smart_crusher_toin_attachment.py`:
- crush() records on real compression, doesn't on passthrough
- structurally-similar inputs land on the same pattern
- _smart_crush_content() records (legacy apply() path)
- TOIN errors don't break compression
- non-JSON input doesn't record
- inject_retrieval_marker=False emits a WARNING
## CCR marker emission knob — labelled, not yet fixed
`ccr_config.inject_retrieval_marker=False` is not honored — the Rust
port emits `<<ccr:HASH N_rows_offloaded>>` markers in `dropped_summary`
unconditionally. Today the production default has the flag True so no
one is hitting the gap, but the silent-disconnect was a real
regression. Shim now logs a WARNING when callers pass `False` so the
mismatch is visible. Fix needs a Rust-side gate; tracked in
`RUST_DEV.md`.
## Custom relevance scorer — labelled, not yet fixed
`relevance_config` and `scorer` constructor args are accepted for
source compatibility but the Rust default `HybridScorer` always
runs. Shim was logging this at debug; bumped to WARNING. Tracked.
## RUST_DEV.md
New "Known regressions in retired-Python components" section with a
table per retired component. The intent is that this section gets
updated whenever a regression closes (or a new one is found), so the
duplicate-codebase tax stays visible instead of decaying into folklore.
The Anthropic handler's CCR injector path applied a frozen_message_count
guard to system instruction injection but not to tool injection. When
Kompress fired for the first time in a session, the tools array was
mutated unconditionally, invalidating Anthropic's prefix cache and
dropping cache_read_input_tokens to zero on calls where ~48K tokens
were previously being cached.
Mirror the existing inject_system_instructions guard for inject_tool:
when frozen_message_count > 0, defer tool injection so the warm prefix
stays intact.
Adds test_ccr_tool_injection_disabled_when_prefix_frozen as a direct
companion to the existing test_ccr_system_instruction_injection_
disabled_when_prefix_frozen.
Fixes#294
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Faithful port of `headroom/transforms/content_detector.py` into
`headroom-core`. Same regex patterns, dispatch order, confidence
formulas, and line-count caps; lockstep with Python via 21 recorded
parity fixtures (every dispatch branch exercised).
- crates/headroom-core/src/transforms/content_detector.rs: regex-only
detector (no ML); ContentType/DetectionResult mirror Python's enum +
dataclass surface; metadata uses serde_json::Map for clean PyO3
bridging.
- Tie-break in code detection: track scores in first-match insertion
order (matches Python dict iteration semantics on `max()` ties).
- TypeScript second pattern is start-anchored — Python's
`pattern.match(line)` is start-anchored, but the regex crate's
`is_match` is unanchored, so the literal `:` prefix is required for
parity.
- crates/headroom-parity: ContentDetectorComparator + universal f64
normalization in `compare_fixture` (serde_json's lossy parse vs
full-precision serialize creates a 1-ULP asymmetry that broke
comparison; round-trip the actual through to_string/from_str).
- crates/headroom-py: detect_content_type/is_json_array_of_dicts and
PyDetectionResult exposed via PyO3 with GIL released during scan.
- tests/parity/recorder.py: new `_wrap_function` for free-function
recording; content_detector hook + 21 varied inputs covering JSON
arrays, diffs, HTML, search, build/log, six languages, and fallbacks.
Sets up PR2 (ContentRouter scaffold) to call into Rust ContentDetector
in-process via the bridge.
The PR8 marker injection appends a sentinel object
{"_ccr_dropped": "<<ccr:HASH N_rows_offloaded>>"} to the kept-items
array on the lossy path so the LLM sees the retrieval pointer in the
prompt. Tests that iterate compressed arrays via subscript access
(e["level"], r["status"], i["labels"], m["text"]) hit KeyError on
the sentinel because it doesn't share the record schema.
Same root cause as the test_quality_retention fixes in PR8 -- these
integration tests were left out of that pass.
Ship a public helper headroom.transforms.smart_crusher.strip_ccr_sentinels
so tests can use it cleanly: `for e in strip_ccr_sentinels(entries):`
and production callers iterating compressed output get a single
canonical filter instead of inlining the _ccr_dropped check.
The 7 previously-failing tests in PR #292 CI now pass:
- langchain test_100_percent_errors_preserved_logs
- langchain test_errors_preserved_with_many_errors
- langchain test_search_results_with_query_term
- mcp test_all_log_errors_preserved
- mcp test_slack_significant_compression_with_content
- mcp test_database_error_status_preserved
- mcp test_github_bugs_partial_preservation
753 tests across the integration + transforms + retention suites pass
locally. Plugin manifests auto-bumped 0.13.3 -> 0.13.4 by the
sync-plugin-versions hook (unrelated to this fix).
Closes four gaps in the Rust SmartCrusher pipeline that, together,
wire CCR storage end-to-end so the LLM can actually retrieve dropped
data:
1. CCR-Dropped marker is now injected into process_value's lossy-path
output as a sentinel object {"_ccr_dropped": "<<ccr:HASH N_rows_offloaded>>"}
appended to the kept-items array. Previously the store held the
original but no pointer reached the prompt -- the retrieval contract
was data-on-server, no-way-to-ask. Sentinel-as-object preserves the
array-of-dicts shape so downstream iteration with x.get(...) keeps
working.
2. Walker / process_value drift removed. process_value gains a
Value::String arm that handles stringified-JSON containers (parse,
recurse, re-encode) and opaque blobs (CCR marker + store) -- same
semantics walker.rs has always had, now reachable from the main
crush() pipeline.
3. Opaque-string CCR now stores originals. DocumentCompactor gains an
Option<Arc<dyn CcrStore>> field; emit_opaque_ccr_marker calls
store.put when one is configured. Same hash regardless of store
presence -- runtime contract is stable across configurations.
Same wiring is shared between walker.rs and process_value via the
extracted helper.
5. PyO3 surface adds SmartCrusher.compact_document_json(doc_json) ->
compacted-json string. Routes through the crusher's existing CCR
store, so ccr_get resolves both row-drop and opaque-string hashes.
Tests:
- 5 new Rust integration tests in ccr_roundtrip.rs (marker visibility,
nested-array marker, opaque-string roundtrip, stringified-JSON
recursion, walker-with-store)
- 4 new Python tests covering the marker visible-to-LLM contract via
both the native PyO3 surface and the Python shim
- 5 legacy parity fixtures re-recorded (dict_array_*, duplicate_dicts_40)
-- their lossy outputs now carry the sentinel; Rust + Python both
match the new bytes (parity-run smart_crusher: 17/17)
CcrStore trait + InMemoryCcrStore (1000 entries, 5-min TTL, FIFO
eviction, idempotent re-store) live at the crate root. SmartCrusher's
lossy crush_array path now actually stashes the full original [items]
canonical-JSON into the configured store keyed by the same ccr_hash it
embeds in the prompt marker -- closing the no-data-loss contract that
was previously hash-only.
PyO3 surface:
- crusher.crush_array_json(items_json) -> dict with ccr_hash + kept items
- crusher.ccr_get(hash) -> Optional[str] for retrieval
- crusher.ccr_len() -> int for telemetry
Python shim passes both through. Default constructors enable the store
(matches Python's CCR-enabled default); without_compaction() also gets
it because CCR is a contract, not an opt-in extra.
Tests proving compress -> store -> retrieve -> reconstruct:
- 7 unit tests in ccr.rs (put/get/eviction/expiry)
- 9 Rust integration tests (crates/headroom-core/tests/ccr_roundtrip.rs)
- 10 Python tests including 4 explicit before/after element-equality
assertions through both the native PyO3 surface and the Python shim
Plugin manifest versions auto-bumped by the sync-plugin-versions
pre-commit hook (unrelated to CCR but co-resident in the working tree).
Two more tests broke after PR4's lossless-first default flip — same
root cause as the langchain/MCP fixes (#287's first patch):
- tests/test_proxy_ccr.py — TestEndToEndTOINIntegration asserts
CCR-cache state after compression. Lossless wins on the test
fixture and skips CCR entirely (nothing dropped). Pin to lossy
via with_compaction=False so the cache assertion holds.
- tests/test_text_compressors.py — TestSmartCrusherTextIntegration
asserts JSON-array shape round-trip. Lossless substitutes a
CSV+schema string. Pin to lossy + JSON shape via
with_compaction=False. Lossless coverage exists separately in
test_smart_crusher_lossless_default.py.
Same pattern, same fix. CI run that surfaced these:
actions/runs/25025876328
PR4 flipped the OSS default to lossless-first. The MCP server and
LangChain eval tests assert wire-format and row-level retention
properties that belong to the lossy path; the lossless path
substitutes a CSV+schema STRING in place of arrays, which is great
for LLM prompts but wire-incompatible with consumers that iterate
the JSON.
Pin both call sites to the lossy + CCR-Dropped path via
`with_compaction=False`. Same retention semantics as Python's
pre-PR4 SmartCrusher behavior — full payload still cached via CCR
for tool retrieval; nothing is lost.
Modules:
- headroom/integrations/mcp/server.py — runtime MCP wrapper
- tests/test_integrations/langchain/test_evals.py — eval fixture
CI run that surfaced these: actions/runs/25025161868
Stage 3c.2 PR4. Restores Python's CCR-Dropped semantics on the lossy
path (the cornerstone reversibility guarantee that the port had
silently dropped) and flips the OSS default to lossless-first with a
configurable savings threshold.
# The user-visible behavior
Default `SmartCrusher::new()` now runs:
1. Try lossless compaction.
2. If savings >= `lossless_min_savings_ratio` (default 0.30), ship
it — `compacted` populated, `ccr_hash = None`, nothing dropped.
3. Otherwise fall through to the lossy path — drop rows AND
populate `ccr_hash` so the runtime can cache the full original
for tool-call retrieval.
**No data is ever lost.** "Lossy" means "compressed view inline; full
payload retrievable via CCR cache" — same semantics as Python's
SmartCrusher with CCR enabled. The runtime (PyO3 bridge / proxy
server) owns the cache; this crate computes the hash and emits a
marker so the prompt knows where to look.
# What changed
- `SmartCrusherConfig.lossless_min_savings_ratio: f64` (default 0.30).
Single configurable knob — Enterprise overrides as needed. Below
the threshold, lossless declines and lossy + CCR runs.
- `SmartCrusher::new(cfg)` flips to include the compaction stage by
default. `SmartCrusher::without_compaction(cfg)` is the explicit
opt-out for callers / fixtures that depend on pre-PR4 behavior.
- `crush_array` rewritten:
- Lossless-first dispatch with savings-ratio gate
- Lossy path now hashes the full original (12-char SHA-256 prefix)
and emits a CCR-Dropped marker in `dropped_summary` whenever
rows are dropped
- `ccr_hash` field populated whenever rows were dropped
- `process_value` substitutes the compacted string into the JSON
tree when lossless wins, so `crush()` output reflects the win
- PyO3 bridge: `SmartCrusher.without_compaction()` static method;
`SmartCrusherConfig` exposes the new `lossless_min_savings_ratio`
field; Python `SmartCrusher` wrapper accepts `with_compaction=True`
(default) and routes to the right Rust constructor.
- Parity harness: legacy 17 fixtures use `without_compaction()` so
byte-equal coverage of the lossy path is preserved.
# Tests
- Rust: 281/281 smart_crusher unit tests pass (was 277). Six new
tests cover: lossless wins above threshold, lossy falls through
below threshold, CCR hash deterministic + input-dependent, lossy
without compaction emits CCR, passthrough paths don't emit CCR,
without_compaction yields no compacted field.
- Python parity: 21/21 (legacy fixtures via without_compaction).
- Python lossless default smoke: 3/3 new tests in
test_smart_crusher_lossless_default.py.
- Python retention: 21/21 (updated to opt into the lossy path
explicitly since their semantics target row-level retention).
- make ci-precheck green.
Modules:
crates/headroom-core/src/transforms/smart_crusher/{config,crusher}.rs
crates/headroom-parity/src/lib.rs
crates/headroom-py/src/lib.rs
headroom/transforms/smart_crusher.py
tests/test_quality_retention.py
tests/test_transforms/test_smart_crusher_{lossless_default,rust_parity}.py
Stage 3c.1b step 2 + cleanup. The python `SmartCrusher` (3669 lines)
is replaced by a thin pyo3-backed shim (~290 lines) that delegates
every byte to `headroom._core.SmartCrusher` (built from
`crates/headroom-py`, landed in the previous commit). There is no
python implementation and no env-var fallback — the wheel is a hard
import.
Why now: parity was already proven across 17 fixtures + the python-
side bridge test (1+17 in `test_smart_crusher_rust_parity.py`).
Keeping a shadow python impl behind a flag is a permanent maintenance
cost with no operational benefit. Stage 3c.1b deletes ~3380 lines of
python parser/scorer/analyzer/orchestrator code; the rust crate has
its own coverage (388 unit tests + property tests in headroom-core).
Surface preserved (drop-in for every production caller):
- `headroom.transforms.smart_crusher.SmartCrusher` — same class name,
same `__init__(config, relevance_config, scorer, ccr_config)`
signature (the latter three are accepted for source-compat and
silently dropped — rust port keeps those subsystems disabled in
Stage 3c.1, they re-attach in Stage 3c.2).
- `SmartCrusherConfig` and `CrushResult` dataclasses kept as python
dataclasses (callers use `asdict()` / dataclass matching on them).
- `crush(content, query, bias)`, `_smart_crush_content(content, ...)`,
`apply(messages, tokenizer, **kwargs)`, and
`_extract_context_from_messages(messages)` all preserved.
- `smart_crush_tool_output(content, config, ccr_config)` thin wrapper.
The transform-protocol `apply()` orchestration stays python (message
walking, digest-marker insertion, token counting); only the per-
message compression call delegates to rust.
Removed:
- Python parser / planner / scorer / analyzer / classifier (~3380 lines).
- Internal helpers `_classify_array`, `_detect_sequential_pattern`,
`_detect_rare_status_values`, `_detect_items_by_learned_semantics`,
`_percentile_linear`, `_compute_k_split`, `_crush_number_array`,
`_process_value`, etc. — rust crate has parallel coverage.
- `SmartAnalyzer`, `ArrayType`, `CompressionStrategy`,
`extract_query_anchors` — internals; not used by any production
caller (only tests probed them).
Tests deleted (probed deleted internals — same precedent as Stage 3b):
- `tests/test_transforms/test_smart_crusher.py` (40 tests)
- `tests/test_transforms/test_universal_json_crush.py` (45)
- `tests/test_transforms/test_anchor_selector.py` (49)
- `tests/test_toin_field_learning.py` (21)
- `tests/test_crushability.py` (20)
Tests trimmed (removed methods/classes that probe deferred subsystems
— scorer injection, CCR marker injection, TOIN feedback recording —
all of which re-attach in Stage 3c.2):
- `tests/test_transforms/test_smart_crusher_bugs.py`:
TestNumberArraySchemaPreservation, TestStage3c1BugFixes.
- `tests/test_relevance.py`: 2 scorer-injection tests.
- `tests/test_ccr.py`: TestSmartCrusherCCRIntegration class +
test_custom_marker_template.
- `tests/test_toin_integration.py`: TestTOINIntegration +
TestStoreToTOINHash classes.
- `tests/test_critical_fixes.py`: TestSmartCrusherTOINIntegration +
test_full_feedback_loop.
- `tests/test_acceptance.py::TestQueryAnchorExtraction`: dropped the
`extract_query_anchors` probe; kept the end-to-end "Alice
preserved" assertion.
Bug fixes from Stage 3c.1 (#1 percentile linear interp, #2 zero-
padded sequential, #3 rare-status pareto, #4 k-split overshoot) are
pinned by the rust crate and the parity fixtures
(`tests/parity/fixtures/smart_crusher/`).
Tests:
- 517 passed in the smart_crusher-adjacent file set
(test_transforms/, test_relevance*, test_ccr, test_toin_integration,
test_quality_retention, test_acceptance, test_critical_fixes).
- 18 in `test_smart_crusher_rust_parity.py` (1 sanity + 17 fixtures).
- 388 rust unit tests still green.
One stale-error-message regex in `test_relevance_extra.py` updated
from "requires sentence-transformers" → "requires fastembed".
Stage 3c.1b step 1: expose `SmartCrusherConfig`, `CrushResult`, and
`SmartCrusher` to Python via `headroom._core`. The Python shim that
delegates to it (replacing the 3669-line Python implementation) lands
in the next commit; this commit just builds the bridge and a
fixture-replay test that pins it.
Surface:
- `headroom._core.SmartCrusherConfig(**fields)` — every field of the
Rust `SmartCrusherConfig` exposed as a kwarg with matching default.
- `headroom._core.CrushResult` — read-only mirror of the Rust struct
with `compressed`, `original`, `was_modified`, `strategy` getters.
- `headroom._core.SmartCrusher(config=None)` — constructor accepts
only `config`; the Python shim drops `relevance_config`, `scorer`,
and `ccr_config` since Stage 3c.1 keeps those subsystems disabled.
- `crush(content, query="", bias=1.0)` and `smart_crush_content(...)`
methods mirror the Python signatures.
Verification:
- All 17 recorded parity fixtures byte-equal between Python and the
PyO3 bridge (`tests/test_transforms/test_smart_crusher_rust_parity.py`,
18 tests pass — 1 fixture-count sanity + 17 fixtures).
- The Rust-side `cargo run -p headroom-parity --bin parity-run --
run --only smart_crusher` was already 17/17 green.
The two tests catch different regression classes:
- Rust-only test: catches drift in the Rust port's logic.
- Python bridge test: catches PyO3 input/output translation bugs.
Adds the SmartCrusher half of the Rust-vs-Python parity harness. Path A
from the Stage 3c.1 plan: record fixtures from Python (with real
fastembed embeddings + the post-bug-fix code), drive the Rust port over
the same inputs, and assert byte-equal output on every recorded
scenario.
What's in:
- `tests/parity/record_smart_crusher.py`: standalone recorder for
`SmartCrusher.crush(content, query, bias)`. The generic recorder
framework only captures one positional, so this script writes its
own JSON envelope `{input: {content, query, bias}, config, output}`.
17 scenarios cover the planning paths exercised by ContentRouter
in production: passthrough, smart_sample, top_n, time-series,
duplicates, unicode (`ensure_ascii=False`), nested-3-deep, empties,
bias above and below 1.0.
- `crates/headroom-parity/src/lib.rs`: `SmartCrusherComparator`
reconstructs `SmartCrusherConfig` from the fixture's config block,
runs Rust `SmartCrusher::crush()`, emits the same JSON shape Python
serialized.
- `crates/headroom-parity/examples/diff_fixture.rs`: diagnostic CLI
that prints expected-vs-actual for one fixture (used during the
iteration that found the serializer bug below).
Serializer fix — found by the harness:
SmartCrusher uses `safe_json_dumps` (compact `(",", ":")` separators
+ `ensure_ascii=False`) for the wire bytes. The Rust port was using
`python_json_dumps` (default Python: `(", ", ": ")` + `ensure_ascii=
True`), which is the right choice for hashing but wrong for the
output. Refactored `anchor_selector.rs` to take a small
`JsonFmt { sort_keys, compact, ensure_ascii }` config so the three
flavors share one writer, added `python_safe_json_dumps`, and
switched `_smart_crush_content` to call it. All three flavors now
have byte-exact tests.
Two unit tests in `crusher.rs` were pinning the old (wrong) format
and have been re-pinned to the compact form.
Cross-language status:
- All 17 empty-query fixtures: byte-equal.
- Embedding-driven (non-empty-query) fixtures deferred until the
~0.0002 numeric drift between Python `onnxruntime` and Rust `ort`
is resolved (or until we accept the drift via a tolerance — none of
the 17 fixtures exercises a borderline relevance_threshold call,
and downstream code only branches at the 0.3 threshold).
Tests: cargo test --workspace (388 + supporting) green.
Lockstep fixes for the four known bugs in headroom/transforms/smart_crusher.py
plus the field-iteration ordering parity fix. Both languages now agree
byte-for-byte on the affected code paths — prerequisite for parity
fixtures landing next.
Bug #1 — percentile off-by-one (Python line 2844 + Rust crushers.rs)
Replaces integer-division indexing with linear-interpolation
percentile (numpy "linear" method). New _percentile_linear helper
shared by both languages: index = q * (n - 1), interpolate between
floor and ceil.
Bug #2 — zero-padded string IDs misclassified as sequential
Track had_non_string_numeric flag; if every parseable value came
from a string (no actual int/float), return False (categorical, not
sequential). Pre-fix: int("001") loses zero-padding and fakes a
sequential pattern.
Bug #3 — rare-status detection cardinality cap
Cardinality cap raised from 10 to 50. Single-dominant check
replaced with Pareto top-K: smallest K such that top-K covers >=80%
of items. If K <= 5, items NOT in top-K are outliers. Catches
bimodal distributions like 60×INFO + 25×WARN + 15 distinct error
codes.
Bug #4 — k-split overshoot when k_total=1
Clamp after the floored fractions: k_first=min(k_first, k_total),
k_last=min(k_last, max(0, k_total - k_first)). No-op for the
common case k_total >= 2.
Field iteration ordering (Python line 1049)
`for key in all_keys` → `for key in sorted(all_keys)`. Set
iteration is non-deterministic across PYTHONHASHSEED; downstream
short-circuits in _select_strategy and _detect_pattern would pick
different fields between runs. Rust uses BTreeMap (sorted ASCII);
sorting Python locks both languages to the same iteration order.
Verification:
- 56 Python tests pass (51 existing + 5 new lockstep tests under
TestStage3c1BugFixes class).
- 382 Rust tests pass (rust bug #1 documentation test replaced
with two new "fixed behavior" tests).
- Clippy clean.
Status: all four bugs are now fixed in BOTH languages. Parity
fixtures can be recorded against post-fix Python and asserted
byte-equal against Rust. That's the next commit.
# The bug
Several test modules and two production modules loaded the project `.env`
at *import time*. During pytest collection (where every test module is
imported once), this populated `os.environ` with API keys from `.env`.
The skipif guards in `test_proxy_passthrough_integration.py` (and
others) evaluate at collection time:
@pytest.mark.skipif(not os.environ.get("OPENAI_API_KEY"), reason="...")
If the polluter module was collected *before* the guard, the guard saw
the leaked key, decided not to skip, and the integration tests ran
live against a fake key and failed. In a fresh local-dev venv with
`.env` + full `[dev]` extras, this manifested as ~16 spurious test
failures plus a misleading test runtime of 6+ minutes (live HTTP).
# Why now
CI does not see this (no `.env`). It only manifests when:
1. `litellm` (and friends) are installed — they run `dotenv.load_dotenv()`
on import, populating `os.environ` from `.env`.
2. A `.env` file with real API keys exists locally.
Until the venv was provisioned with the full `[dev]` extras during
recent test work, `pytest.importorskip("litellm")` and
`from headroom.pricing import litellm_pricing` both silently no-op'd
(via try/except ImportError → `LITELLM_AVAILABLE=False`), so the leak
never triggered. With litellm now installed, the latent bug surfaced.
# The fix — three patterns
1. **Production modules** (`headroom/pricing/litellm_pricing.py`,
`headroom/backends/litellm.py`): wrap the eager `import litellm` with
a snapshot/restore of `os.environ`. Any keys litellm's bundled
`python-dotenv` adds during import are deleted immediately. The
module is fully imported and cached in `sys.modules` so subsequent
imports hit the cache without re-running the side effect.
2. **Test modules using `pytest.importorskip("litellm")`**
(`test_backend_bugs.py`, `test_bedrock_region.py`,
`test_cost_tracker_counterfactual.py`): replace with
`tests._dotenv.importorskip_no_env_leak("litellm")`, which does the
same snapshot/restore around `importlib.import_module`.
3. **Test modules that intentionally need `.env` values for skipif
guards** (`test_compression_summary_*.py`, `test_query_echo.py`,
`test_cost_tracker_counterfactual.py`, `test_memory_usage_integration.py`,
`test_bundled_tools_savings.py`): replace module-level
`os.environ.setdefault(...)` / `dotenv.load_dotenv()` with
`tests._dotenv.load_env_overrides()` (returns a local dict — does
NOT mutate `os.environ`) plus `autouse_apply_env(...)` (function-
scoped fixture that applies via `monkeypatch.setenv`, auto-cleaned
at teardown). The skipif still works because
`ANTHROPIC_KEY = os.environ.get(...) or _env_overrides.get(...)`
reads from the local dict as fallback.
# Helper module
New `tests/_dotenv.py` exposes:
- `load_env_overrides() -> dict[str, str]` — read `.env` into a dict.
- `autouse_apply_env(overrides) -> fixture` — function-scoped autouse
fixture that applies via `monkeypatch.setenv`.
- `importorskip_no_env_leak(module) -> module` — drop-in
`pytest.importorskip` substitute that quarantines env mutations.
# Results
Local full-suite (excluding live-LLM and live-feed tests):
- Before: 46 failed, 4830 passed, 387s
- After: 2 failed, 4672 passed, 134s
The remaining 2 failures are unrelated environment-dependent tests
(missing `PIL` / Docker daemon).
The python `DiffCompressor` is replaced by a thin pyo3-backed shim that
delegates to `headroom._core.DiffCompressor`. There is no python
implementation and no env-var fallback — the wheel is a hard import.
Why now: opt-in defaults don't drive python retirement. Byte-equal
parity was already proven across 27 fixtures (stage 3a); keeping a
shadow python impl behind a flag is a permanent maintenance cost with
no operational benefit. Stage 3b deletes ~700 lines of python parser /
scorer / formatter code; the rust crate has its own coverage.
Surface preserved:
- `headroom.transforms.diff_compressor.DiffCompressor` — same class
name, same `__init__`, same `compress(content, context)` shape.
Returns python `DiffCompressionResult` dataclasses so call sites that
destructure with `asdict()` work unchanged.
- `DiffCompressorConfig` and `DiffCompressionResult` dataclasses kept.
- Sidecar `compress_with_stats(...)` exposes the rust-only
`DiffCompressorStats` (per-file hunk drops, context lines trimmed,
file_mode normalizations) for observability.
Removed:
- Python parser / scorer / formatter (~700 lines).
- Internal `DiffHunk` / `DiffFile` parser dataclasses (rust crate has
parallel coverage).
- 12 tests that probed `_parse_diff` / `_score_hunks` or the deleted
parser dataclasses. The 29 public-API tests in `test_diff_compressor.py`
remain and now exercise the rust backend through the same import path.
- `HEADROOM_DIFF_COMPRESSOR_BACKEND` env var — no longer meaningful.
Build:
- `scripts/build_rust_extension.sh` runs `maturin develop` and symlinks
the built `.so` into `headroom/` so `import headroom._core` resolves
past the in-tree package shadowing the maturin overlay.
- `.gitignore` excludes the symlinks and allowlists the build script.
Tests:
- 544 transforms pass; 33 rust-vs-fixture parity tests guard the pyo3
bridge against regressions (`tests/test_transforms/test_diff_compressor_rust_parity.py`).
- Mypy clean.
User audit caught three gaps that prevented DiffCompressor from being
invoked even when the input was a real diff. These complement the four
emit-time bugs fixed in the previous commit — those fixes only kick in
once DiffCompressor receives the input. Without these gap fixes, real
merge-commit diffs and `git log -p` outputs with long commit messages
were misrouted away from DiffCompressor entirely.
# The three gaps (each fixed in Python; gap 3 also fixed in Rust)
1. Detector scan window was hardcoded to first 50 lines.
`_try_detect_diff` in content_detector.py only inspected
`content.split("\n")[:50]`. `git log -p` outputs commonly have
commit messages longer than 50 lines (releases, squashed commits,
bots), pushing the `diff --git` header out of the detection window.
Result: input was returned with `content_type=PLAIN_TEXT` and routed
to the text compressor, never reaching DiffCompressor. Fix: window
widened to 500 lines.
2. Detector regex didn't recognize merge-commit headers.
`_DIFF_HEADER_PATTERN` matched `diff --git`, `--- a/`, and the
regular `@@ -A,B +C,D @@` hunk header. Merge-commit diffs from
`git log -p` use `diff --combined <path>`, `diff --cc <path>`, and
combined-diff hunk headers `@@@+`. The shared `--- a/` line still
triggered the detector with low confidence, but only barely. Fix:
extended the regex to recognize all four merge-shaped header forms.
3. DiffCompressor parser only matched `^diff --git`.
Even after fixing detection, the parser's `_DIFF_GIT_PATTERN`
wouldn't match `diff --combined` or `diff --cc`, so merge diffs
reached DiffCompressor and were treated as one giant pre-diff blob —
passed through unchanged after the previous PR's pre-diff
preservation fix. Fix: added `_DIFF_COMBINED_PATTERN` and
`_DIFF_CC_PATTERN`; `_parse_diff` starts a new file section on any
of the three header forms. Mirrored in Rust as `is_diff_header`
helper that checks all three regexes.
# Why this matters end-to-end
DiffCompressor's value comes from being routed to. Detection +
parser-level coverage are upstream of the compressor — without them,
the compressor never sees the input. The previous PR's four bug fixes
(rename, combined-diff hunks, no-newline marker, pre-diff content) are
correct and necessary, but for merge commits and long-preamble diffs,
they were only firing on the rare cases where the detector misclicked
into DiffCompressor anyway. With these three gaps closed, the
ContentRouter→DiffCompressor pipeline actually engages on:
- `git log -p` outputs of any commit-message length
- Merge-commit diffs (`diff --combined`, `diff --cc`)
- Combined-diff snippets (`@@@`+ hunk-only inputs)
# New fixtures (3 added to the existing 24)
- `066bc82…` — `diff --combined` merge diff (3-way)
- `5d950a94…` — `diff --cc` merge diff (alternate form)
- `66c86f64…` — long pre-diff content (60-line commit message)
followed by a rename diff (exercises detector scan widening +
pre-diff preservation in tandem)
Parity: total=27 matched=27 skipped=0 diffed=0.
# Tests
- Python: 4 new tests across 2 new test classes —
`TestRoutingGapMergeDiffs` (combined / cc parser) and
`TestRoutingGapDetectorScanWindow` (long preamble detection +
combined-diff regex recognition).
- Rust: 2 new tests covering combined / cc parser sections.
# Verification
- 27/27 parity fixtures byte-equal.
- Python: 41/41 tests pass (was 37).
- Rust: 18/18 transforms tests; 62/62 workspace; 5/5 proptests.
- `cargo fmt --check` clean; `cargo clippy --workspace --all-targets
-- -D warnings` clean.
Audit caught four bugs that the byte-equal parity harness can't catch on
its own — both Python and Rust were faithfully emitting the buggy output.
Fixed in lockstep so parity is maintained while the underlying behavior
is now correct on inputs the existing 20 fixtures didn't exercise.
# The four bugs (each fixed in both Python and Rust)
1. Renames silently dropped from output. Parser captured `is_renamed=True`
but the emitter never emitted ANY rename markers. Output of a rename
looked exactly like a plain modification of the old path. Fix: capture
`rename from` / `rename to` / `similarity index N%` / `dissimilarity
index N%` / `copy from` / `copy to` lines in a new `rename_lines` field
on `DiffFile`; emit them after `diff --git` in canonical git ordering.
2. Combined diff hunks (`@@@`) silently dropped. Hunk-header regex only
matched `@@`, so 3-way merge hunks had `current_hunk` never set and
ALL their content fell through to the no-op branch. Fix in Python:
regex switched to `^(@@+) ... \1` (backreferences match any number of
`@`s on each side). Fix in Rust: alternation over `@@`, `@@@`, `@@@@`
since `regex` is RE2-based and rejects backreferences. n>3 octopus
merges still fall through; rare in practice.
3. `\ No newline at end of file` markers can be context-trimmed away.
Treated as ordinary "other" lines — if more than `max_context_lines`
from a `+`/`-` change, dropped. Round-trip-breaking for patches; can
change whether the trailing line has a newline. Fix: in
`_reduce_context`, force-add any line starting with `\` to the keep
set regardless of distance.
4. Pre-diff content silently dropped. Anything before the first `diff
--git` — commit messages from `git log -p`, email headers from `git
format-patch`, fork-and-rebase metadata — was discarded. Fix:
`_parse_diff` now returns `(pre_diff_lines, files)`; `format_output`
prepends pre-diff content verbatim when present.
# Hidden parity bug found during the work
`_compress_files` constructed a fresh `DiffFile` from the parsed one but
only copied a subset of the fields by name. The new `rename_lines` and
`original_*_line` fields were silently dropped here, so the parser
populated them correctly but the emitter saw an empty `rename_lines`
list. Caught by writing a real test instead of a smoke test — the smoke
test passed because it hit the no-diff-found short-circuit, not the
parser/emitter pipeline. Constructor now copies all fields explicitly.
# Parity status
- Existing 20 fixtures: still byte-equal between fixed Python and fixed
Rust. None of them exercised the buggy paths.
- 4 NEW fixtures recorded against fixed Python, exercising each bug-fix
path: rename, 3-way combined diff, `\ No newline` marker far from
changes, pre-diff commit headers. All 4 byte-equal between Python and
Rust.
- Parity harness: total=24 matched=24 skipped=0 diffed=0.
# Observability
Some normalizations remain parity-bound (file mode `100644` hardcode,
`Binary files differ` simplification). Those are surfaced in
`DiffCompressorStats::file_mode_normalizations` /
`binary_files_simplified` (Rust) and via `logger.warning` (Python's new
`_log_loss_signals` helper, called once per compress).
# Tests
- Python: 4 new test classes (11 tests) covering rename markers,
combined diffs, no-newline preservation, pre-diff content. Edge case:
no pre-diff content must NOT add a leading blank line.
- Rust: 4 new `bugfix_*` unit tests with the same scenarios.
- Existing Python tests calling `_parse_diff` directly were updated for
the new `(pre_diff, files)` tuple return.
# Verification
- Python: 37/37 tests pass (was 26).
- Rust: 16/16 transforms tests; 60/60 workspace unit tests; 5/5
proptests; 1/1 doctest.
- Parity: 24/24 byte-equal.
- `cargo fmt --check` clean; `cargo clippy --workspace --all-targets
-- -D warnings` clean.
The `qdrant-env-vars` change in d3c37d7 (PR #266) added `qdrant_url` and
`qdrant_api_key` keys to the kwargs that `MemoryHandler` passes into
`DirectMem0Adapter.__init__`. The corresponding assertion in
`test_ensure_initialized_fast_paths_and_qdrant_variants` was missed in that
PR and has been failing on `main` ever since. Surfacing here because it
fails on every PR's CI; not caused by the Rust tokenizer work this branch
adds.
The two new keys are both `None` when the corresponding `HEADROOM_QDRANT_*`
env vars are unset, which is the case in this test.
