headroom/tests/test_content_router_detection_dedup.py

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

97 lines
4 KiB
Python
Raw Normal View History

perf(content_router): dedupe content detection (#2419) ## Description ContentRouter ran the native content detector two to three times on identical content, on the hottest path in the proxy (every compressed message, every request). This cuts it to once. `_detect_content` isn't cheap and isn't memoized. It strips a detection envelope, runs the Rust/Magika ONNX classifier, then several regex passes. `compress()` ran it once for debug logging that's off by default, then `_determine_strategy()` recomputed it (plus `is_mixed_content`) on the same content. That's twice per `compress()`, and three times on the `apply()` cache-miss path. Closes: N/A (no filed issue, surfaced by an internal contribution-backlog audit). ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `compress()` computes `is_mixed_content` and `_detect_content` once, then threads both into `_determine_strategy` through new optional params (`mixed`, `detection`). - `_determine_strategy` uses the passed values when present, and computes them itself when they're `None`. Its one private caller changes. Any other caller keeps working. - Added `tests/test_content_router_detection_dedup.py`. One test asserts `compress()` detects exactly once (it fails before the fix at `assert 2 == 1`). The other asserts the threaded result routes the same as the recomputed one across content types. - Updated two existing `_determine_strategy` test doubles to take the new kwargs. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_content_router_detection_dedup.py tests/test_transforms_content_router.py \ tests/test_transforms/test_content_router.py tests/test_transforms_content_detection.py -q 135 passed in 8.98s $ pytest tests/test_transforms/ tests/test_content_router_*.py tests/test_router_*.py \ tests/test_lossless_excluded_compaction.py -q 423 passed, 62 skipped in 54.93s $ ruff check . All checks passed! $ mypy headroom Success: no issues found in 505 source files ``` ## Real Behavior Proof - Environment: macOS (Darwin), Python 3.13, headroom worktree on this branch off `upstream/main`, `HF_HUB_OFFLINE=1 LITELLM_LOCAL_MODEL_COST_MAP=true`. A counter wraps the real `_detect_content` and delegates to it, so real routing and compression run. - Exact command / steps: run the real router over one representative message and count `_detect_content` calls on the fixed tree, then `git stash` the source and count again on the unfixed tree. Covered `router.compress(blob)` and `router.apply([tool_msg])`. - Observed result: `compress()` dropped from 2 detection calls to 1, and `apply()` dropped from 3 to 2, on the same input with the same routing strategy (`text`) and the same output. The once-only test flips from `assert 2 == 1` before to passing after. - Not tested: production Magika ONNX timing. This dev env has no onnxruntime, so the detector ran its regex fallback tier, which makes the saved cost a floor, not a ceiling. I also scoped out the Tier B extension (threading the `apply()` Pass-1 detection into `compress()`) on purpose. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes Scope is the default routing path. `force_kompress` already uses the cheaper regex detector, so it never paid the redundant native cost. `_compress_mixed` re-detects per split section, but that's different content (sub-sections), so it's out of scope. The `apply()` Pass-1 detection stays. It gates the `is_code` protection check for every message, including cache hits that never reach `compress()`. Threading it into `compress()` would widen a shared task tuple and change the public `compress()` signature, all for a cache-miss-only save, so I left it as a possible follow-up. Doc checklist item is N/A (internal perf dedup, no user-facing docs change). This is a Python-only change, so the first push will use `--no-verify` for the known `ci-precheck` Rust-latency bench flake (`classify_under_10us_per_call`), which runs clean in CI.
2026-07-19 23:50:46 +08:00
"""Regression guards for the content-detection dedup on the router hot path.
``ContentRouter.compress`` used to run the native ``_detect_content`` twice on
identical content: once for (default-off) debug logging, then again inside
``_determine_strategy``. The native Rust/Magika pass is the router's hottest
per-message cost, so ``compress`` now runs it once and threads the result into
``_determine_strategy``. These tests pin the call count at one and prove the
threaded result routes identically to the recomputed one.
"""
from __future__ import annotations
import pytest
import headroom.transforms.content_router as content_router_module
from headroom.transforms.content_detector import ContentType, DetectionResult
from headroom.transforms.content_router import (
CompressionStrategy,
ContentRouter,
ContentRouterConfig,
RouterCompressionResult,
)
@pytest.fixture(autouse=True)
def _reset_detect_module_state(monkeypatch: pytest.MonkeyPatch) -> None:
# The native-detector circuit breaker (#575) is process-wide; keep it from
# leaking across tests (mirrors tests/test_transforms_content_router.py).
monkeypatch.setattr(content_router_module, "_detect_native_unhealthy", False)
monkeypatch.setattr(content_router_module, "_detect_backend_warned", False)
monkeypatch.setattr(content_router_module, "_detect_panic_warned", False)
def _count_detects(monkeypatch: pytest.MonkeyPatch, result: DetectionResult) -> dict[str, int]:
"""Replace the module-level ``_detect_content`` with a deterministic counter.
Both call sites (``compress`` and ``_determine_strategy``) resolve the name
from the module namespace at call time, so a single patch counts every call.
"""
calls = {"n": 0}
def _counting(content: str) -> DetectionResult:
calls["n"] += 1
return result
monkeypatch.setattr(content_router_module, "_detect_content", _counting)
return calls
def test_compress_runs_detection_once(monkeypatch: pytest.MonkeyPatch) -> None:
# Before the dedup this was 2 — compress ran _detect_content for debug
# logging, then _determine_strategy ran it again on identical bytes. The
# fix threads the single result through, so it must now be exactly 1.
router = ContentRouter(ContentRouterConfig(prefer_code_aware_for_code=False))
calls = _count_detects(monkeypatch, DetectionResult(ContentType.PLAIN_TEXT, 1.0, {}))
monkeypatch.setattr(content_router_module, "is_mixed_content", lambda content: False)
# Stub the downstream compressor — this test isolates detection, not output.
monkeypatch.setattr(
router,
"_compress_pure",
lambda *a, **k: RouterCompressionResult(
compressed="x", original="x", strategy_used=CompressionStrategy.TEXT
),
)
router.compress("a representative plain-text blob that is comfortably non-empty")
assert calls["n"] == 1
@pytest.mark.parametrize(
("mixed", "detected"),
[
(False, ContentType.SOURCE_CODE),
(False, ContentType.JSON_ARRAY),
(False, ContentType.BUILD_OUTPUT),
(False, ContentType.PLAIN_TEXT),
(True, ContentType.SOURCE_CODE), # mixed regex hit, native says code -> trust native
(True, ContentType.PLAIN_TEXT), # genuinely mixed
],
)
def test_determine_strategy_threaded_matches_recomputed(
monkeypatch: pytest.MonkeyPatch, mixed: bool, detected: ContentType
) -> None:
# Threading precomputed (mixed, detection) must pick the SAME strategy as
# letting _determine_strategy recompute them: the dedup changes cost, not
# routing.
router = ContentRouter(ContentRouterConfig(prefer_code_aware_for_code=False))
detection = DetectionResult(detected, 1.0, {})
monkeypatch.setattr(content_router_module, "is_mixed_content", lambda content: mixed)
monkeypatch.setattr(content_router_module, "_detect_content", lambda content: detection)
recomputed = router._determine_strategy("payload")
threaded = router._determine_strategy("payload", mixed=mixed, detection=detection)
assert threaded is recomputed