The bounded compression executor introduced in this PR moved every
handler's compression call from `asyncio.wait_for(asyncio.to_thread(...))`
to `self._run_compression_in_executor(...)`, which lives on
`HeadroomProxy` (server.py) and is inherited by handler mixins at
runtime.
The test's `_DummyOpenAIHandler` only inherits `OpenAIHandlerMixin`,
not `HeadroomProxy`, so it lacks the method. The Responses API
compression path caught the AttributeError and silently fell back —
which made `test_handle_openai_responses_stream_keeps_compression`
fail with `apply.call_count == 0`.
Add a synchronous stub that just invokes the callable; tests don't
need real thread-pool semantics.
Three audit follow-ups from issue #327's deep-dive review.
C1 — CompressionCache concurrency lock
======================================
`CompressionCache` instances are shared per `session_id` and accessed from
async-dispatched threadpool workers. Pre-fix, concurrent requests for the
same session raced on `_cache`, `_stable_hashes`, `_first_seen`, and
`_total_tokens_saved` with no synchronization. Observable failures:
* Lost-update on `_total_tokens_saved` (read-modify-write).
* `RuntimeError: OrderedDict mutated during iteration` from `apply_cached`
when a concurrent `store_compressed` evicts during the walk.
* Lost stable-hash records — next-turn compute_frozen_count reads
inconsistent state.
May also explain part of SvenMeyer's `_cache: 0 entries / 1003 misses`
observation: the cache was being clobbered concurrently.
Added `threading.RLock` guarding all mutating methods. `RLock` (not `Lock`)
so future code can call locked methods from inside another locked method
without self-deadlock. Also locked `HeadroomProxy._compression_caches`
dict-of-caches access via a separate `_compression_caches_lock` so two
concurrent calls for the same session_id can't each create distinct
CompressionCache objects (which would split the cache state between them).
The `/stats` endpoint snapshots the cache list under the dict lock before
iterating to avoid eviction-during-iteration.
C2 — Multi-worker CCR fragmentation: documented + startup warning
=================================================================
The in-memory `InMemoryCcrStore` (Rust), `_compression_caches` (Python),
`session_tracker_store` (Python), and TOIN learner state are ALL
per-process. Multi-worker uvicorn round-robins requests across workers,
so a session whose turn-1 lands on worker A may have turn-2 land on
worker B. Worker B has zero knowledge of A's CCR markers, replay cache,
or prefix-cache state. Result: `Retrieve original: hash=X` markers stay
in-context as opaque directives, every fresh tool_result is recompressed
from scratch, and `frozen_message_count=0` causes Anthropic prefix-cache
busts on every cross-worker turn.
Added a "Multi-worker deployment — CCR fragmentation" section in
`RUST_DEV.md` documenting the failure modes, the supported configuration
(`--workers 1`), and the sticky-session workaround for horizontal scale.
The proxy emits a `WARNING`-level log line on startup if `workers > 1` is
detected, pointing at the doc section.
C3 — Bounded compression executor with cancel-aware metrics
===========================================================
`asyncio.wait_for(asyncio.to_thread(pipeline.apply), timeout=...)`
cancellation does NOT propagate into the threadpool worker that's running
Rust code. Once the worker has picked up the task,
`concurrent.futures.Future.cancel()` returns False and the thread runs to
completion. Stuck threads accumulated invisibly on asyncio's default
executor, contending with unrelated `to_thread` callers (file IO, etc.).
Replaced all 7 `asyncio.to_thread` call sites for `pipeline.apply()`
across `proxy/handlers/anthropic.py` (3) and `proxy/handlers/openai.py` (4)
with a new `HeadroomProxy._run_compression_in_executor(fn, *, timeout)`
helper that:
1. Submits to a dedicated bounded `ThreadPoolExecutor` named
`headroom-compress` (configurable via
`ProxyConfig.compression_max_workers`; defaults to
`min(32, (cpu_count or 1) * 4)`).
2. Increments `_compression_in_flight` (gauge) when work starts and
decrements when work completes; tracks `_compression_in_flight_max`
as a high-water mark.
3. Detects "leaked threads" by comparing wall-clock elapsed against the
timeout in the worker's `finally` block. Increments
`_compression_leaked_threads` when a worker finishes after its
asyncio future was cancelled. Operators can see the leaked-thread
rate climbing in `/stats runtime.compression_executor` BEFORE the
pool fills up.
Tests
=====
* `TestCompressionCacheConcurrency` (3 tests) — many threads
store_compressed / apply_cached / update_from_result on a single
CompressionCache; assert no exceptions, no lost updates, no partial
state.
* `test_get_compression_cache_returns_same_instance_under_contention` —
32 concurrent `_get_compression_cache(same_id)` calls return the
identical instance (would split pre-lock).
* `test_proxy_compression_executor.py` (8 tests) — pool size respects
config, in-flight gauge tracks running compressions, high-water mark
is monotonic, timeout propagates to awaiter, leaked-thread counter
increments on post-deadline completion, `/stats` surfaces all three
gauges.
Verification
============
* All 123 targeted regression tests pass.
* `make ci-precheck` clean.
* No `Co-Authored-By` trailer; conventional `fix:` prefix; no
`--no-verify`.
The Anthropic token-mode handler walked past prefix_tracker.frozen_message_count
whenever an upcoming tool_result's content-hash matched comp_cache._stable_hashes
or should_defer_compression returned True. That conflated content equality with
positional cache membership.
Anthropic's prefix cache is POSITIONAL: bytes 0..K cached, anything past K is
fresh. _stable_hashes is content-keyed and grows unbounded. In long Claude Code
sessions where tool_result content rhymes across turns (repeated system prompts,
repeated file reads, repeated tool descriptions), the walker advanced
frozen_message_count to len(messages) on every turn and the pipeline produced
transforms_applied=[] on 73% of requests in user SvenMeyer's reported session
(headroom-stats-2026-05-01.json: 74 of 101 eligible requests "prefix_frozen") —
even after the prior fix in 44944fb. The 15 requests that did compress averaged
21%, proving compression itself works when reached.
Fix: delete the walker. The freeze boundary is now
frozen_message_count = min(
prefix_tracker.frozen_message_count, # positional ground truth
comp_cache.compute_frozen_count(messages), # local cache lower bound
)
compute_frozen_count's use of _stable_hashes can only LOWER the freeze via the
min clamp, never raise it past prefix_tracker's value. For any position in the
gap [compute_frozen_count, prefix_tracker.frozen_count], recompressing produces
byte-stable output (compression is deterministic on input content), so
Anthropic's prefix cache stays valid.
Cross-handler verification:
* OpenAI handler (proxy/handlers/openai.py:358-382) does not have this walker
— uses only compute_frozen_count. Codex routes through OpenAI handler. Both
unaffected.
* Streaming and non-streaming both invoke anthropic_pipeline.apply() before the
upstream call. One fix covers both paths.
* Cache mode (is_cache_mode) takes the _extract_cache_stable_delta path and is
independent of the walker. Unaffected.
Tests: six new regression tests lock down the post-fix invariants — clamp to
min(prefix_tracker, compute_frozen_count); fresh tool_result whose hash matches
old _stable_hashes entry is not frozen; frozen prefix byte-stable across the
pipeline; 10-turn session produces non-empty compression suffix every turn;
streaming and non-streaming compute identical frozen_message_count; OpenAI
handler never calls the walker functions. Plus scripts/smoke_issue_327.py
(gated by RUN_LIVE_API=1) drives a 10-turn conversation against
api.anthropic.com in both shapes (string + list-of-blocks) and both modes
(streaming + non-streaming).
ci-precheck clean. 191 tests pass.
Follow-ups (separate PRs):
* Fix _cache perpetually empty (anthropic.py result.messages != working_messages
comparison rarely fires in token mode).
* Cap _stable_hashes with bounded LRU + 1h TTL — hygiene only after the freeze
gate is removed.
* List-shape tool_result content gates at content_router.py:1975 and
intelligent_context.py:657 (cluster A from the audit).
Direct port of `headroom.transforms.scoring.MessageScorer` (459 LOC).
Foundation piece for the IntelligentContext port (PR-B onward).
What's wired:
- Deterministic factors fully ported: recency (exp-decay), forward
references (tool_call_id graph), token density (unique/total).
- External-dep factors gated behind traits: `EmbeddingProvider` and
`ToinProvider`. No concrete impls yet — both default to neutral
values matching Python's `embedding_provider=None` / `toin=None`.
PR-A1 wires fastembed; PR-A2 plugs in a PyO3 ToinProvider.
- ScoringWeights + MessageScore with serde + BTreeMap-ordered
breakdown for stable JSON.
Parity:
- 13 fixtures recorded from Python, byte-equal under the comparator.
- Floats rounded to 5 decimals on both sides — absorbs f32-vs-f64
drift in the weighted sum without masking real bugs.
Drive-by: re-fix three pre-existing clippy errors in
smart_crusher/crusher.rs that re-emerged with new test additions
(field_reassign_with_default + dead hash_array_for_ccr).
test_anthropic_hooks_do_not_break_extract_user_query_lookup mocks
pipeline.apply to return tokens_after=40 and a tiny compressed
message. The pre-Bug-3 proxy trusted the mock's tokens_after and
emitted x-headroom-tokens-after: 40. After issue #327 Bug 3 the
proxy recounts optimized_tokens from result.messages with its
own tokenizer (the mocked "compressed" string counts to 11), so the
header asserted against the wrong tokenizer's number.
Compute the expected value from the same tokenizer the proxy uses
(get_tokenizer("claude-sonnet-4-6")) and assert the recounted
header matches that. Add a tokens_before > tokens_after invariant
so the spirit of the test (compression actually reduced bytes) is
preserved without coupling to a specific tokenizer's calibration.
Three bugs combined to drive end-to-end compression on the Anthropic
backend to ~0% in token mode (the default). User report #327 saw a
~9× drop in dashboard savings from one day to the next on Claude
Code traffic; the dashboard headline was technically correct but the
underlying compression genuinely was not running. After this change
the same Claude Code-shape multi-turn conversation goes from
14987 → 14371 tokens at the request boundary on turn 1 and only
recompresses the freshest tool_result on subsequent turns, with the
prior turns frozen byte-identical to preserve the upstream prefix
cache.
Bug 1 — IntelligentContextManager inner ContentRouter has no observer
PR #302 (commit cf979958, 2026-04-28) wired CompressionObserver onto
the outer ContentRouter in proxy/server.py and onto SmartCrusher.
The inner ContentRouter constructed lazily inside
IntelligentContextManager._get_content_router (added Jan 18, 2026
in 57b2de5 alongside the COMPRESS_FIRST strategy) was missed. That
inner router handles the bulk of Claude Code's tool_result-block
compression, so per-strategy counters surfaced by PR #314 in v0.15.0
showed compressions_by_strategy={"text": 6} while
summary.compression.total_tokens_removed=1.3M — math-impossible.
Fix: add observer= parameter to IntelligentContextManager.__init__,
forward it to the inner ContentRouter at intelligent_context.py:525,
and pass observer=self.metrics from proxy/server.py.
Bug 2 — TTL deferral marks every fresh tool_result as stable
should_defer_compression in compression_cache.py returned True on
first-sight (added 2026-04-07 in commit 22dad13 with the intent of
batching first-time compressions near the 5-min cache TTL boundary
to trade many small busts for one). The token-mode walker at
anthropic.py:766-787 walks every message past frozen_message_count,
calls should_defer_compression on each fresh tool_result, gets True,
and advances ttl_frozen += 1 — every iteration. Result:
frozen_message_count grows to len(messages), the pipeline freezes
the entire request, and nothing reaches a real compressor.
The defer-first-sight rationale assumes recurring content within
TTL. Real Claude Code traffic produces unique content per turn, so
"defer until next sight" defers forever. Compressing fresh content
on first sight does not bust any prefix cache because Anthropic has
not cached that byte position yet — it's a cache write either way.
Fix: should_defer_compression returns False on first-sight (record
the timestamp; compress now). Subsequent sightings within TTL still
defer (batch window preserved for genuinely repeating content).
Updated tests in test_compression_cache.py to assert the corrected
semantics and verify _first_seen is recorded on first call.
Bug 3 — cross-tokenizer comparison in token-mode inflation guard
anthropic.py:634 sets original_tokens = tokenizer.count_messages(...)
using the proxy-side EstimatingTokenCounter. The token-mode branch
at line 816 set optimized_tokens = result.tokens_after from
pipeline, which uses the provider-side AnthropicProvider tiktoken
estimator. The two tokenizers disagree by ~25% on the same payload.
The inflation guard at line 901
(if optimized_tokens > original_tokens: revert to originals) treats
those two numbers as comparable. After a real 12% compression the
provider-tokenizer figure was still higher than the proxy-tokenizer
baseline, so the guard fired, optimized_messages was reset to the
original input, transforms_applied was emptied, and tokens_saved
went to 0. The dashboard showed no compression even when the
pipeline successfully compressed.
Fix: recount optimized_tokens with the proxy tokenizer right after
the pipeline returns, so the guard compares apples-to-apples. The
recount cost is a few ms on a 50K-token request and is dwarfed by
upstream call latency.
Verification
* 80 targeted tests across test_compression_cache,
test_compression_observability, test_proxy_anthropic_cache_stability,
test_proxy_intelligent_context pass.
* make ci-precheck clean.
* End-to-end real-API run against api.anthropic.com via local proxy:
- Turn 1 fresh: 14987 → 14371 (4.1%) on a 3-tool-round payload;
smart_crusher and diff strategies fired with non-zero savings.
- Turn 2 (turn 1 history + 1 new tool_result): 23161 → 21928 (5.3%);
only the new tool_result compressed; older turns marked
router:protected:user_message; Anthropic returned
cache_creation_input_tokens > 0 confirming the prefix was not
busted.
Two new regression tests in test_compression_observability lock down
the inner ContentRouter observer wiring so a future copy of Bug 1
fails the suite the day it lands.
CI flagged two issues on the rebased branch:
1. ruff format --check failed on server.py and test_traffic_learner.py
after the rebase; line-collapse / trailing-whitespace nits.
2. Codecov reported 80% patch coverage with 20 lines missing in the
matcher helpers — mostly branches not exercised by the high-level
tests (empty Levenshtein inputs, source-prefix Bash parsing, env-var
skip, equal-string short-circuit in binary match, the substantive-
token path that beats the edit-distance gate, error_recovery patterns
with non-canonical content in _drop_contradictions).
Adds 16 targeted unit tests for those branches and applies ruff format.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The recovery matchers paired any failed and successful tool call within
a 5-call window with no semantic check that the pair was actually a
retry. This produced confidently-wrong rules like:
File `state.rs` does not exist. The correct path is `lib.rs`.
…where the user simply read two unrelated files in the same directory.
Across sessions the same user can also typo in opposite directions,
producing directly contradictory rules side by side.
This commit adds three structural checks:
1. Read recovery: require the failed and successful basenames to be
identical or close in Levenshtein distance. Rejects the "same dir,
different file" case that was the most common noise source.
2. Bash recovery: require both commands to share a binary (allowing
path-prefixed variants and short prefix-versions like
`python` ↔ `python3`) AND either have low normalized edit distance
or share a substantive non-flag token. Rejects pairs that share only
the binary name but differ in every meaningful argument.
3. Contradiction filter on flush: detect A→B and B→A pairs in
error_recovery patterns and drop both. They almost always indicate
opposite-direction typos in different sessions, not stable advice.
Also: stash failed_path in metadata so the contradiction filter and
downstream consumers can reason about pairs without parsing content.
Tests: 13 new tests covering the heuristics directly. Existing tests
exercising legitimate recoveries (`python`→`python3`, `ruff`→`.venv/bin/ruff`,
`pip install`→success) continue to pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Asserts that stop()'s final flush_to_file does not bypass the evidence
threshold. Earlier behavior collapsed the gate to 1 at shutdown,
persisting every singleton pattern. This guards against that change
sneaking back in.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The traffic learner was emitting one-shot error_recovery patterns that
contradicted each other and bloated MEMORY.md with low-signal noise. Two
issues drove this:
1. The shutdown flush bypassed the evidence gate: the in-memory
_min_evidence was set to 2, but on stop() the gate dropped to 1, so
every singleton pattern got persisted at session end. This is the
opposite of how evidence thresholding should work — singletons are
the least trustworthy patterns, not the most.
2. The default min_evidence of 2 is too low to filter noise from the
matchers, which pair up failed/successful tool calls within a small
sliding window without a strong semantic check that the calls are
actually related.
Changes:
- Raise default min_evidence from 2 to 5 in TrafficLearner.
- Remove the shutdown-relaxation in flush_to_files; require
self._min_evidence at all times, including on stop().
- Add traffic_learning_min_evidence to ProxyConfig (default 5).
- Add --min-evidence CLI flag with HEADROOM_MIN_EVIDENCE envvar so
users and embedded clients (desktop apps, plugins) can tune the
threshold without source changes.
- Thread the config value through HeadroomProxy into TrafficLearner.
- Tests: cover default propagation and custom value flow.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`headroom/transforms/tag_protector.py` was a regex-driven scan-and-
replace loop that ran on every kompress call from ContentRouter
(`content_router.py:1089`). The Python implementation had five real
bugs we now fix in the port — the most consequential being a
`str.replace(.., .., 1)` first-occurrence-replace bug that silently
collapsed two identical custom-tag blocks in the same input to a
single placeholder + a stray duplicate of the second block.
# Bug fixes (each pinned by a `fixed_in_3e4` test)
* **#1: O(n²) on nested custom tags.** Python's `while changed` loop
restarted a full regex scan after every replacement. Rust walks
once in linear time on input length.
* **#2: First-occurrence replace bug.** `result.replace(orig, ph, 1)`
replaces the FIRST textual match, not the matched offset. Two
identical custom-tag blocks collapsed to one placeholder + a stray
duplicate of the second block. The Rust walker stitches output by
offset so distinct blocks always get distinct placeholders.
* **#3: Silent 50-iteration cap.** Python had a hard `max_iterations
= 50` safety limit that quietly truncated tag protection on deeply
nested input. The Rust walker is bounded by input length only.
* **#4: Self-closing pass duplicate-replace risk.** Python ran a
second loop with the same `replace_first` bug for self-closers.
Rust handles self-closers in the same single pass.
* **#5: Placeholder collision.** If the input contained a literal
`{{HEADROOM_TAG_…}}` substring, Python silently let the collision
break restoration. Rust salts the prefix and reports it in stats.
# Architecture
Two-phase walker:
* Phase 1 (`identify_spans`): linear scan over input bytes, hand-
rolled tag-open / tag-close lexer (no regex). Maintains a stack of
open custom tags; on a matching close, collapses the inner span
into a single `Span { start, end, Block }`. Self-closing custom
tags become `Span { ..., SelfClosing }` immediately. Marker-only
mode (`compress_tagged_content=true`) emits Open/CloseMarker spans
instead. Orphan opens stay un-protected (matches Python behavior).
Orphan closes are emitted verbatim and counted in stats.
* Phase 2 (`emit_output`): walks `text` once, splicing placeholders
for span ranges and copying everything else verbatim. Offset-based,
never `str.replace`.
PyO3 surface: `protect_tags`, `restore_tags`, `is_html_tag`,
`known_html_tag_names`. The Python shim retires the regex internals
and re-exports `KNOWN_HTML_TAGS` (rebuilt from the Rust list) +
`_is_html_tag` for backwards compat with `content_router.py` and the
existing test surface.
# Test plan
* 25 Rust unit tests including 4 `fixed_in_3e4_*` bug-fix tests
* 27 Python tests (23 existing + 4 new `fixed_in_3e4` parity tests)
* 5 integration tests in `test_tag_protection_integration.py` pass
* `make ci-precheck` clean
`headroom/transforms/query_echo.py` was already disabled in all three
proxy handlers (`anthropic.py`, `openai.py`, `gemini.py`) — each
carried the same comment: 'disabled — hurts prefix caching in long
conversations. The echo changes every turn, invalidating the cached
prefix.' That call was right: the echo's per-turn variability would
bust the Anthropic/OpenAI/Gemini prompt cache and cost more in TTFT
than the recall benefit ever paid back. The module was orphaned but
still living in the tree, with its own 70-test file pinning the
disabled behavior.
# Removed
* `headroom/transforms/query_echo.py` (123 LOC)
* `tests/test_query_echo.py` (whole file)
* The three 'disabled — hurts prefix caching' comment blocks in the
proxy handlers (the rationale lives in this commit message and the
PR description; no need to leave dead-code breadcrumbs in the hot
path).
# Kept
* `headroom.utils.extract_user_query` is a different function with
the same name and is still used elsewhere — untouched.
# Test plan
* `make ci-precheck` clean
* No remaining references to `query_echo`/`QueryEcho`/`Query Echo`/
`inject_query_echo` in the tree.
`headroom/transforms/text_compressor.py` was a regex-line-sampling
fallback that nothing in the runtime called. ContentRouter routes
`CompressionStrategy.TEXT` straight to the Kompress ML compressor at
`content_router.py:1046` — the comment there literally says 'Prefer
Kompress ML compressor for text'. The Python file was orphaned but
still imported by its own test class, making it look live in the 3e
queue.
Drops the 3e.3 port from the queue: there's nothing to port.
# Removed
* `headroom/transforms/text_compressor.py` (255 LOC, unused)
* `tests/test_text_compressors.py::TestTextCompressor` (3 tests)
* `text_compressor` mention in `error_detection.py` shim docstring
* `text_compressor` mention in `test_signals_keyword_parity.py` docstring
* `TextCompressor` mention in `bench_latency.py` scenario comment
# Kept (defensive)
The legacy marker regex in `ccr/tool_injection.py:213` stays — it
parses an even older TextCompressor output format (pre-2026), is
purely defensive, and removal buys nothing. Test references to that
format in `test_ccr_tool_injection.py` document the regex contract
and stay too.
# Test plan
* `make ci-precheck` clean
* `tests/test_text_compressors.py` 19 passes (was 22, dropped 3)
Ports `headroom.transforms.log_compressor` to Rust. The biggest-by-
impact remaining compressor port: build/test logs are where the
10-50x compression wins live.
* Stack-trace state machine: per-flavor dispatcher (Python Traceback,
JS, Java, Rust error, Go); each flavor has its own termination
rule. Python terminated on any blank line, dropping mid-trace
lines from chained-exception traces.
* Conservative dedupe: preserves message prefix (everything before
first `:` or `=`); only trailing region is tokenised. Python's
blanket normalisation collapsed segfaults at different addresses.
* Loud CCR failures: `tracing::warn!` + `logger.warning` instead of
bare `except: pass`.
* `LogLevel::FAIL` documented as cosmetic-equivalent to ERROR.
Same shape as search_compressor port. Rust `LogCompressor`
orchestrates format detect -> classify -> score -> select ->
format -> CCR. Inline static-table format detector (YAGNI),
aho-corasick level classifier with word-boundary post-filter
(`signals::keyword_detector` technique), hand-rolled per-flavor
stack-trace state machine. `signals::LineImportanceDetector` NOT
consumed -- log levels are structural, not prose-style importance.
`headroom.transforms.log_compressor` becomes a thin shim:
`compress()` delegates to Rust end-to-end; internal helpers
preserved for the existing 50-test surface. Two existing tests
updated for new dedupe semantics + new compress orchestration.
* 17 Rust unit tests
* 50 Python tests pass
* `make ci-precheck` clean
The 12 langchain integration evals generate fixture data via
`random.choice`/`random.randint` without seeding. SmartCrusher's
anchor selection consumes the same global random state, so a handful
of unseeded inputs (~1% of seed values) skip the first/last anchor
preservation and the eval flakes — surfaced on PR #319 CI even though
this PR doesn't touch SmartCrusher.
Confirmed pre-existing: identical 5/500 seed failures on `main`
@ `cf3877d`. The fix is the smallest one that doesn't paper over the
underlying selector behavior — seed `random` per-test via an autouse
fixture so dataset generation is reproducible.
Ports `headroom.transforms.search_compressor` to Rust as the first
consumer of the `signals::LineImportanceDetector` trait shipped in
Phase 3e.1.
The Python regex registry (`_GREP_PATTERN`/`_RG_CONTEXT_PATTERN`)
silently misparsed two real-world inputs. The hand-rolled Rust parser
fixes both:
* **Windows paths.** `^([^:]+):(\d+):(.*)$` captured only the drive
letter from `C:\Users\foo\bar.py:42:line`, then the `\d+` group
failed on `\`. Result: every Windows-formatted line was silently
dropped from `file_matches`. The Rust parser detects the drive
prefix and starts the line-number scan after the drive colon.
* **Filenames with `-`.** `_RG_CONTEXT_PATTERN`'s `[^:-]+` excluded
dashes from the path, so legitimate names like
`pre-commit-config.yaml-42-line` parsed wrong. The Rust parser
anchors on the *line-number marker* (`<sep>\d+<sep>`), so paths
can contain dashes freely.
Two further hardening changes:
* CCR storage failures are loud (Python silently swallowed them).
* Per-file dedup is `O(n log n)` via `BTreeSet<(line_no, content_hash)>`
(Python used linear `match not in file_selected`, worst-case
quadratic for big files).
The Rust `SearchCompressor` owns a `Box<dyn LineImportanceDetector>`
defaulting to `KeywordDetector`. Priority scoring routes through the
trait instead of a hardcoded regex list, so a future BGE classifier
head (per the trait extension docs) can take over without touching
the compressor.
Sidecar `SearchCompressorStats` captures lines unparsed, files
dropped by `max_files`, matches dropped by per-file vs global caps,
and the CCR skip reason -- diagnostics Python never emitted.
`headroom.transforms.search_compressor` is now a thin shim that
delegates `compress()` to Rust end-to-end (so the parser bug fixes
land in production), and keeps the legacy `_parse_search_results`
helper routed through the same Rust parser. The other internal
helpers (`_score_matches`, `_select_matches`, `_format_output`)
stay Python -- they're heavily covered by existing direct-call tests
and Rust scoring is byte-equivalent.
The 4 public dataclasses are unchanged. Tests that monkeypatched the
old internal `_store_in_ccr` helper are updated to exercise the new
`_persist_to_python_ccr` boundary instead.
* 16 Rust unit tests (parser, scoring, selection, CCR round-trip) +
3 explicit `fixed_in_3e2` markers for the bug-fix lines
* 53 Python tests (existing suite intact; 2 updated for new shape)
* `make ci-precheck` clean
Stacks on PR #317 (signals trait module).
This test pinned the pre-3e.1 behavior on three lines that the new
KeywordDetector intentionally changes:
1. `'token'` was asserted to be in SECURITY_KEYWORDS. It was dropped
from the security set in 3e.1 because it false-positived on every
LLM-token reference in our own product. Updated to assert the new
set (`security|password|auth|secret`) and explicitly that `token`
is gone.
2. SECURITY_PATTERN was tested via "rotate the auth token" (matched
via the now-removed `token` keyword). Now tested via "rotate the
auth header" (matches via `auth`, which is the real security
signal) plus a negative assertion that LLM-metric strings no
longer fire.
3. ERROR_PATTERN test added an assertion that "Connection timeout"
now flags as an error (3e.1 fixed the keyword/regex drift).
PRIORITY_PATTERNS_TEXT indices were also corrected: the Rust-supplied
markdown_prefixes table is ordered `# `, `## `, `### `, `#### `, `**`,
`> ` (six prefixes) so the bold/blockquote assertions live at indices
6 and 7, not 3 and 4. New `# ` and `## ` checks pin the lower indices.
Each diverging assertion carries a `fixed_in_3e1` comment so the
audit trail stays clear.
Establish `crates/headroom-core/src/signals/` as a top-level module
holding cross-cutting detection traits. Phase 3e.1 ports
`error_detection.py` to a `LineImportanceDetector` trait + a
`Tiered<T>` combinator + a single concrete `KeywordDetector` impl
backed by aho-corasick. Three traits at three granularities are
sketched (line / blob / item); only line-importance is implemented
today.
Two bug fixes from the Python source bake into both the Rust impl
and the Python regex shim:
1. `ERROR_KEYWORDS` listed `timeout|abort|denied|rejected` but
`ERROR_PATTERN` regex omitted them. Lines like `"Connection
timeout"` were silently neutral despite the keyword being canonical.
Both surfaces now flag them.
2. `SECURITY_KEYWORDS` carried `token`, which false-positived on
every reference to LLM tokens (`input_tokens`, `tokens_saved`, ...)
in our own product. Dropped from the security set.
The Python `error_detection.py` shim now reflects keyword data out of
Rust via `keyword_registry_snapshot()` and recompiles the legacy
`re.Pattern` objects on the fly. Existing callers (text_compressor,
search_compressor, intelligent_context) continue to import the same
names with no source changes; caller migration to the trait API
happens in their own port PRs.
The trait architecture is the seam where a future ML detector slots
in without touching `KeywordDetector` or any caller. The canonical
extension is documented in `signals/README.md` as a classifier head
on the existing `bge-small-en-v1.5` embedder loaded by
`relevance::EmbeddingScorer` -- 384-dim -> 4-class softmax,
~1.5 KB head, ~1 ms inference, no extra model file. Two alternatives
(distilled tinyBERT in ONNX, logistic regression on lexical
features) are kept open in case BGE-head underfits.
Per the no-silent-fallbacks rule: only `KeywordDetector` lands as a
concrete impl. No NoOp, no MockDetector, no stub-ML -- those will
arrive with their real implementations.
Phase 3g (Compression Pipeline Formalization, issue #315) is queued
as the cross-cutting follow-up that will make lossless-then-lossy-
then-CCR ordering an explicit, observable architecture rather than
implicit per-compressor logic. Trait shapes there will reuse the
signals primitive landed in this PR.
The PrometheusMetrics `compressions_by_strategy` and
`tokens_saved_by_strategy` counters (PR #302) were tracked in process
but never exported, because the Prometheus->Supabase pipeline treats
each metric name as a separate column. The data clock for the
code_compressor port-vs-retire decision needed actual production
visibility, not just CI assertions.
Add both dicts to the `/stats` endpoint and nest them under
`pipeline_timing._strategies` in the beacon Supabase payload. The
existing `pipeline_timing` column is JSONB and absorbs the nested
shape -- zero schema change.
`_build_pipeline_timing` is extracted from the inline payload-builder
so the projection has its own unit tests, including a guard that the
`_strategies` sub-key is only emitted when at least one counter is
non-empty.
The Prometheus scrape still does NOT carry these metrics; the existing
`test_prometheus_export_does_not_leak_per_strategy_metrics` guard
remains in place.
Replaces the dispatch-path detection with the locked Stage-3d chain:
Tier 1: magika_detect() (PR3)
Tier 2: unidiff::is_diff() (PR4)
Tier 3: PlainText fall-through
The regex `content_detector` is no longer on the production path —
it stays in the tree as a comparison oracle (and for any direct
caller); a future PR retires it entirely.
What lands:
- `crates/headroom-core/src/transforms/detection.rs`: new `detect()`
function that chains the two tiers. Tier-1 errors log at WARN
level and continue to Tier 2 (the chain's *next* tier IS the
legitimate fallback for magika failure; treating tier-1 error as
hard-fail would block all detection on transient ONNX issues).
- 12 unit tests covering: empty, JSON, source code, HTML, standard
git diff, naked hunk (Tier 2 catch), prose, grep search results
(locked-design behavior change), build log, YAML, Rust source,
determinism across repeated calls.
- PyO3 binding `detect_content_type` now calls the chain. Synthesizes
the legacy `DetectionResult` shape (confidence=1.0, empty metadata)
since the chain doesn't surface a probabilistic score and no
production caller reads metadata from the binding today.
- Python `headroom/transforms/content_router.py`: `_detect_content`
now delegates to `headroom._core.detect_content_type`. The Python-
side `_get_magika_detector` + regex fallback is retired (single
detection surface; no parallel paths). Test for the helper rewritten
to monkeypatch the Rust binding instead of the old Python paths.
Behavior changes (per locked design):
- `SearchResults` and `BuildOutput` ContentTypes route to PlainText
(or SourceCode if magika happens to label it code-like) rather
than to specialized strategies. No regex tier on the Rust side,
per `project_rust_content_detection_arch.md`. If proxy benchmarks
show real loss on grep/build outputs, we add focused detectors
later — not preemptively.
Stacked on PR4 (unidiff). When PR4 squash-merges, this PR rebases
trivially against main.
Tests:
- 12 new Rust unit tests in `transforms::detection::tests`
- 43 Python content_router tests (was 42; old monkeypatch test
rewritten in place, not duplicated)
- `make ci-precheck` green
The previous _ListHandler approach attached a handler to the
headroom.proxy logger and worked locally on Python 3.14, but failed in
CI on Python 3.10-3.13 — the warning record never reached the handler.
Root cause is unclear (possibly cross-test logger state), but the
handler-attachment path is brittle for a single-warning assertion.
Replace it with unittest.mock.patch.object on the handler module's
logger.warning. This is invariant to logging hierarchy, propagation
flags, and per-test logger mutations — we directly observe the call
that the production code makes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Issue #296 reports compression appearing to complete successfully then
being discarded after a 30s timeout. The diagnostic gap that makes the
report hard to root-cause:
1. The pipeline's "Pipeline complete: ..." and "Pipeline: freezing
first ..." log lines have no request_id, so under concurrent load
the reporter cannot tell whether the success log is from the
timing-out request or a sibling request.
2. The handler's catch-all "Optimization failed: {e}" passes only
str(e), which is empty for asyncio.TimeoutError — the report shows
"Optimization failed:" with nothing after the colon.
This change is observability-only:
- pipeline.apply now reads request_id from kwargs and prefixes its two
INFO log lines with [request_id] when present.
- The four anthropic_pipeline.apply call sites in the Anthropic handler
(3 in handle_anthropic_messages, 1 in handle_anthropic_batch_create)
pass request_id through.
- The catch-all warning becomes
"[{request_id}] Optimization failed: {type(e).__name__}: {e}" so
TimeoutError is distinguishable from real exceptions in bug reports.
No behavior change. Adds two tests covering both diagnostics.
This is intentionally not a fix for #296 — the underlying timeout still
needs reproduction at 367k+ token transcripts. With these diagnostics in
place, the next bug report will be able to confirm whether the failing
request actually reached the pipeline.
Refs #296
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Re-lands two audit fixes that were marked "merged" on GitHub but never
reached main: squash-merging the parent stack changed its commit SHA,
which silently dropped the contents of the stacked PRs (#301, #305).
Single PR this time — no stacking risk.
What lands:
1. **`enable_ccr_marker` Rust gate** — new field on `SmartCrusherConfig`
(default `true`). `crush_array` checks it before emitting the
`<<ccr:HASH>>` marker text and the CCR store write. PyO3 surface +
parity-fixture tolerance updated; recorded fixtures predate the
field and inherit the `true` default.
2. **Python shim collapses both flags to the gate** — both
`ccr_config.enabled=False` and `ccr_config.inject_retrieval_marker
=False` now flip the Rust gate off. Storing a payload nothing in
the prompt can reference is pointless, and storing under
`enabled=False` would be a surprise side effect the user
explicitly opted out of.
3. **Custom `scorer` / `relevance_config` fails loud** — replaces the
prior WARNING-and-drop. Silently dropping a user-supplied scorer
is a textbook silent fallback. `NotImplementedError` instead.
Verified zero production callers pass these args; full plumbing
arrives with Stage-3c.2's relevance-crate Python bridge.
Tests:
- 2 new Rust unit tests in `crusher.rs::tests`
- 6 new Python tests in `test_smart_crusher_toin_attachment.py`
(3 CCR marker-knob behaviors + 3 scorer fail-loud)
- Removed `test_ccr_inject_marker_false_logs_warning` (the WARNING is
gone now that the flag is honored)
- `make ci-precheck` green; eval suite + observability tests run
twice consecutively to verify no TOIN file pollution leaks into the
regular+coverage double-run on Python 3.11
RUST_DEV.md audit table reflects both gaps closed.
The new SmartCrusher.apply() tests added in this PR feed the global
TOIN learning store (default path `~/.headroom/toin.json`). On Python
3.11, CI runs the suite twice (regular + coverage); a pattern written
in the first pass changes which rows the lossy sampler keeps in the
second pass and breaks
`tests/test_integrations/langchain/test_evals.py::TestRelevancePreservation::test_first_last_items_always_preserved`.
Fix: an `isolated_toin` fixture that points `HEADROOM_TOIN_PATH` at a
tempdir for the test and resets the singleton on enter and exit. Same
isolation pattern PR #300 already uses for `test_smart_crusher_toin_attachment.py`.
Move _setup_file_logging() from module import to create_app() so
importing headroom.proxy.server in tests or library contexts no longer
silently attaches a RotatingFileHandler to the user's live proxy.log.
This was discovered via PR #303: running the test suite produced
"Optimization failed: TimeoutError" entries in ~/.headroom/logs/proxy.log
because TestClient instantiations from test_proxy_anthropic_compression_diagnostics.py
inherited the module-level handler.
Add a regression test that imports the server module in a subprocess
and asserts no RotatingFileHandler is attached to the headroom logger.
Refs #303 (review thread).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
`caplog` flakes under the full suite — earlier tests can leave
`logging.disable()` or root-handler state that suppresses records on
the named logger before pytest's caplog filter sees them. The test
passed in isolation but failed deterministically once the larger
suite preceded it (CI run 25070639932 across all four Python
versions, ~4500 tests in).
Fix: monkeypatch `headroom.transforms.smart_crusher.logger.warning`
directly. Bypasses every level/disable/handler concern; we only
assert the constructor *called* `logger.warning` with the expected
flag name in the message — which is exactly the contract this test
guards.
Adds a `CompressionObserver` Protocol (`headroom.transforms.observability`)
and wires `ContentRouter` and `SmartCrusher` to call it once per real
compression event. `PrometheusMetrics` implements the protocol and
accumulates per-strategy counters (`compressions_by_strategy`,
`tokens_saved_by_strategy`).
Why: the TOIN→SmartCrusher silent disconnect was invisible for three
weeks because no signal distinguished by strategy. With per-strategy
counters in place, the next regression of that shape fails the test
suite the day it lands instead of waiting on a manual audit.
Counters live as in-process state only — deliberately NOT exported via
the Prometheus scrape or OTel surface. The metric→Supabase pipeline
treats each metric name as a column, and we cannot add new columns.
CI-level observability via `tests/test_compression_observability.py` is
sufficient to catch silent regressions; production export waits on a
non-column-adding pipeline.
The Stage 3c.1b retirement of the Python SmartCrusher silently disconnected
three subsystems. The audit on 2026-04-28 caught them; this commit fixes
what's fixable today and labels the rest visibly.
## TOIN learning loop — fixed
Before: `ContentRouter._record_to_toin` skipped SmartCrusher on the
assumption SmartCrusher recorded its own TOIN events. The retired Python
class did. The Rust port doesn't know about TOIN. Net result: the
highest-traffic compression strategy stopped fueling the learning loop,
silently.
Fix: shim's `crush()` and `_smart_crush_content()` now call
`toin.record_compression()` after a real compression. Filtered on
`strategy != "passthrough"` because the Rust port flips
`was_modified=True` from JSON whitespace re-canonicalization. Best
effort: TOIN failures are logged at debug level and never break
compression.
Token estimates use `len(json) // 4` (the rule the retired Python used)
because the router doesn't pass a tokenizer down to this layer and
re-tokenizing here would dominate the recording cost.
7 tests in `tests/test_smart_crusher_toin_attachment.py`:
- crush() records on real compression, doesn't on passthrough
- structurally-similar inputs land on the same pattern
- _smart_crush_content() records (legacy apply() path)
- TOIN errors don't break compression
- non-JSON input doesn't record
- inject_retrieval_marker=False emits a WARNING
## CCR marker emission knob — labelled, not yet fixed
`ccr_config.inject_retrieval_marker=False` is not honored — the Rust
port emits `<<ccr:HASH N_rows_offloaded>>` markers in `dropped_summary`
unconditionally. Today the production default has the flag True so no
one is hitting the gap, but the silent-disconnect was a real
regression. Shim now logs a WARNING when callers pass `False` so the
mismatch is visible. Fix needs a Rust-side gate; tracked in
`RUST_DEV.md`.
## Custom relevance scorer — labelled, not yet fixed
`relevance_config` and `scorer` constructor args are accepted for
source compatibility but the Rust default `HybridScorer` always
runs. Shim was logging this at debug; bumped to WARNING. Tracked.
## RUST_DEV.md
New "Known regressions in retired-Python components" section with a
table per retired component. The intent is that this section gets
updated whenever a regression closes (or a new one is found), so the
duplicate-codebase tax stays visible instead of decaying into folklore.
The Anthropic handler's CCR injector path applied a frozen_message_count
guard to system instruction injection but not to tool injection. When
Kompress fired for the first time in a session, the tools array was
mutated unconditionally, invalidating Anthropic's prefix cache and
dropping cache_read_input_tokens to zero on calls where ~48K tokens
were previously being cached.
Mirror the existing inject_system_instructions guard for inject_tool:
when frozen_message_count > 0, defer tool injection so the warm prefix
stays intact.
Adds test_ccr_tool_injection_disabled_when_prefix_frozen as a direct
companion to the existing test_ccr_system_instruction_injection_
disabled_when_prefix_frozen.
Fixes#294
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Faithful port of `headroom/transforms/content_detector.py` into
`headroom-core`. Same regex patterns, dispatch order, confidence
formulas, and line-count caps; lockstep with Python via 21 recorded
parity fixtures (every dispatch branch exercised).
- crates/headroom-core/src/transforms/content_detector.rs: regex-only
detector (no ML); ContentType/DetectionResult mirror Python's enum +
dataclass surface; metadata uses serde_json::Map for clean PyO3
bridging.
- Tie-break in code detection: track scores in first-match insertion
order (matches Python dict iteration semantics on `max()` ties).
- TypeScript second pattern is start-anchored — Python's
`pattern.match(line)` is start-anchored, but the regex crate's
`is_match` is unanchored, so the literal `:` prefix is required for
parity.
- crates/headroom-parity: ContentDetectorComparator + universal f64
normalization in `compare_fixture` (serde_json's lossy parse vs
full-precision serialize creates a 1-ULP asymmetry that broke
comparison; round-trip the actual through to_string/from_str).
- crates/headroom-py: detect_content_type/is_json_array_of_dicts and
PyDetectionResult exposed via PyO3 with GIL released during scan.
- tests/parity/recorder.py: new `_wrap_function` for free-function
recording; content_detector hook + 21 varied inputs covering JSON
arrays, diffs, HTML, search, build/log, six languages, and fallbacks.
Sets up PR2 (ContentRouter scaffold) to call into Rust ContentDetector
in-process via the bridge.
The PR8 marker injection appends a sentinel object
{"_ccr_dropped": "<<ccr:HASH N_rows_offloaded>>"} to the kept-items
array on the lossy path so the LLM sees the retrieval pointer in the
prompt. Tests that iterate compressed arrays via subscript access
(e["level"], r["status"], i["labels"], m["text"]) hit KeyError on
the sentinel because it doesn't share the record schema.
Same root cause as the test_quality_retention fixes in PR8 -- these
integration tests were left out of that pass.
Ship a public helper headroom.transforms.smart_crusher.strip_ccr_sentinels
so tests can use it cleanly: `for e in strip_ccr_sentinels(entries):`
and production callers iterating compressed output get a single
canonical filter instead of inlining the _ccr_dropped check.
The 7 previously-failing tests in PR #292 CI now pass:
- langchain test_100_percent_errors_preserved_logs
- langchain test_errors_preserved_with_many_errors
- langchain test_search_results_with_query_term
- mcp test_all_log_errors_preserved
- mcp test_slack_significant_compression_with_content
- mcp test_database_error_status_preserved
- mcp test_github_bugs_partial_preservation
753 tests across the integration + transforms + retention suites pass
locally. Plugin manifests auto-bumped 0.13.3 -> 0.13.4 by the
sync-plugin-versions hook (unrelated to this fix).
Closes four gaps in the Rust SmartCrusher pipeline that, together,
wire CCR storage end-to-end so the LLM can actually retrieve dropped
data:
1. CCR-Dropped marker is now injected into process_value's lossy-path
output as a sentinel object {"_ccr_dropped": "<<ccr:HASH N_rows_offloaded>>"}
appended to the kept-items array. Previously the store held the
original but no pointer reached the prompt -- the retrieval contract
was data-on-server, no-way-to-ask. Sentinel-as-object preserves the
array-of-dicts shape so downstream iteration with x.get(...) keeps
working.
2. Walker / process_value drift removed. process_value gains a
Value::String arm that handles stringified-JSON containers (parse,
recurse, re-encode) and opaque blobs (CCR marker + store) -- same
semantics walker.rs has always had, now reachable from the main
crush() pipeline.
3. Opaque-string CCR now stores originals. DocumentCompactor gains an
Option<Arc<dyn CcrStore>> field; emit_opaque_ccr_marker calls
store.put when one is configured. Same hash regardless of store
presence -- runtime contract is stable across configurations.
Same wiring is shared between walker.rs and process_value via the
extracted helper.
5. PyO3 surface adds SmartCrusher.compact_document_json(doc_json) ->
compacted-json string. Routes through the crusher's existing CCR
store, so ccr_get resolves both row-drop and opaque-string hashes.
Tests:
- 5 new Rust integration tests in ccr_roundtrip.rs (marker visibility,
nested-array marker, opaque-string roundtrip, stringified-JSON
recursion, walker-with-store)
- 4 new Python tests covering the marker visible-to-LLM contract via
both the native PyO3 surface and the Python shim
- 5 legacy parity fixtures re-recorded (dict_array_*, duplicate_dicts_40)
-- their lossy outputs now carry the sentinel; Rust + Python both
match the new bytes (parity-run smart_crusher: 17/17)
