Commit graph

3 commits

Author SHA1 Message Date
Tejas Chopra
446ec26003
feat(transforms): dispatch kompress/text via the compressor registry + forward question (#2411)
## What
Completes the if/elif → registry migration in the content router:
**KOMPRESS and TEXT** now dispatch through the `kompress` built-in
adapter (`_registry_compress`), like every other strategy. Also **fixes
a latent bug** in `_invoke_kompress` that dropped the QA-aware
`question` argument (hardcoded `None`) — `question` now rides
`CompressInput.config['question']` and is forwarded into
`_try_ml_compressor`, so QA-aware compression content is preserved.

## Intentionally NOT byte-identical (one approved change)
The sole behavior change is the KOMPRESS/TEXT **token metric**: reported
`compressed_tokens` is now `_estimate_tokens(output.content)` — the
router's calibrated estimate, consistent with `original_tokens` and
every other registry-dispatched strategy — instead of the Kompress
model's own tuple count. **Compressed content is preserved byte-for-byte
in all paths.**

## Decision-impact analysis (traced every reader of `compressed_tokens`)
No content, routing, keep/drop, fallback, or lossless-then-lossy
decision reads this metric for KOMPRESS/TEXT: they're not in
`fallback_eligible_strategy` nor `{SEARCH,LOG,HTML}`, and the
STAGE-0/general layering calls `_try_ml_compressor` directly (unchanged,
already forwards `question`). The only downstream value-reader is
`_record_to_toin`'s skip gate (`original_tokens <= compressed_tokens`) —
**telemetry/learning only**, never affects returned content or routing,
and arguably more correct now (both sides on the same `_estimate_tokens`
scale). Consciously accepted.

## Tests
Rewrote the PR-C2 deferral-pinning tests →
registry-dispatch-matches-direct (content matches the direct
`_try_ml_compressor(..., question)` call; token assertion switched
`==<model count>` → `==_estimate_tokens(output)`, the only assertion
change, solely due to the approved metric switch). Added a
QA-differential test (question changes content) + an adapter-level
`question`-forwarding test. Offline suite: 96 passed; ruff 0.15.17 +
mypy clean.

**Note:** the full content-router CI suite may require further test
updates for any test that exercises the real KOMPRESS/TEXT branch and
asserts the returned count equals the model's tuple `compressed_tokens`
— those should switch to `_estimate_tokens(output)`. (The broad
content_router/compression selection wasn't run locally — it needs
ONNX/HF.)

After this, the router's per-strategy dispatch is fully
registry-resolved.
2026-07-18 22:03:53 -07:00
Tejas Chopra
7ebda67ef6
feat(transforms): add compressed signal + dispatch code_aware/html/diff via registry (#2400)
## What
Third increment of the adapter phase (builds on #2391/#2399). Adds a
`compressed: bool` field to `CompressOutput` and uses it to flip the
**fallback/passthrough** strategies — CODE_AWARE and HTML (and DIFF
where clean) — to registry-resolved dispatch, byte-identically.

## The contract addition (the enabling piece)
`CompressOutput.compressed: bool = True` — lets a compressor signal
**passthrough** (did-not-compress, `content` is the original unchanged)
vs a real result. This is what the router's `None`-driven
fallback/passthrough branches needed to move to the registry without
changing behavior. Default `True`, so existing and external compressors
are unaffected.

## How (byte-identical)
A new `_registry_compress` helper returns the `CompressOutput` (or
`None` when the built-in is unavailable, preserving the `_get_*` guard's
passthrough). The flipped branches map that back to their historical
`compressed is None` semantics:
- **CODE_AWARE:** a passthrough (`not output.compressed` / `None`) sets
local `compressed = None`, so the existing `_try_ml_compressor` Kompress
fallback + `lossless_then_lossy` no-shrink retry +
`strategy`/`strategy_chain` mutations run **verbatim**.
- **HTML:** a `None`/passthrough falls through to the bottom passthrough
exactly as before (`strategy_chain == [html, passthrough]`).

## Deferred
SMART_CRUSHER, KOMPRESS, TEXT, PASSTHROUGH — the
SmartCrusher→Kompress→Log fallback chain + the ML boundary — are the
next (final) increment, left byte-for-byte here. Reversibility gate,
external dispatch (#2388), default behavior unchanged. No new
config/env.

## Testing
`tests/test_router_registry_dispatch.py` +
`tests/test_builtin_compressor_adapters.py` extended: differential tests
for CODE_AWARE (success AND None→Kompress-fallback with matching
`strategy_chain`, ML mocked), HTML (success AND None→`[html,
passthrough]`), and the adapter `compressed=False`-on-None mapping.
Offline suite: 88 passed; ruff + mypy clean. The full content-router
suite in CI is the authoritative byte-identical gate.
2026-07-18 16:50:05 -07:00
Tejas Chopra
981616c60e
feat(transforms): make built-in compressors real Compressor implementations (adapters) (#2391)
## What
Turns each built-in registry entry into a working `Compressor` (the
`compressor_registry` contract): `compress(CompressInput) ->
CompressOutput` delegates to the same underlying built-in method the
content router already invokes in `_apply_strategy_to_content`, reached
through the router's own `_get_*` getter so config flows through
identically. Token counts use the router's `_estimate_tokens`;
`lossless` mirrors the descriptor; `recoverable` is `{}` (built-ins
persist CCR recovery to the store as a side effect, not via their return
value).

Adapted: `smart_crusher, code_aware, search, log, tabular, config, html,
kompress`.

## Behavior change
**None — additive by construction.** Dispatch, the `_get_*` getters,
fallback chains, the reversibility gate, and config are all unchanged.
The router still dispatches built-ins via its existing if/elif and never
routes a request through the registry;
`_resolve_active_external_compressors` filters built-in entries out of
the opt-in external-dispatch path *by type* (the class name
`_BuiltinCompressorEntry` is load-bearing). A default request is
byte-identical: `_active_external_compressors == []`, external dispatch
is an inert guard, and adapters are reachable only via
`compressor_registry.get()/active()`.

## `image` — documented passthrough (not a guess)
`ImageCompressor.compress(messages)` operates on image blocks inside
message dicts, not `str` content, and isn't on the
`_apply_strategy_to_content` path, so there's no faithful `str→str`
delegation. Its adapter is a documented non-raising passthrough rather
than a fabricated one.

## Testing
`tests/test_builtin_compressor_adapters.py` — differential tests
asserting each adapter's output matches the built-in's direct output
(JSON→smart_crusher, CSV→tabular, log lines→log, grep→search,
config→config, Python→code_aware, HTML→html); kompress is mocked (no ML
inference); every registry entry has a working non-raising `compress`.
Updated the obsolete guard test in `test_compressor_selection.py`.
Offline suite: 72 passed; ruff + mypy clean. (Broad
content-router/compression suite deferred to CI — it needs HF-Hub/ONNX
model loads.)

This is PR-A of the adapter phase (built-ins become Compressor
implementations); flipping the router's dispatch to registry-resolved is
the follow-up. Builds on #2370/#2371/#2373/#2388.
2026-07-18 12:59:49 -07:00