headroom/tests/test_transforms/test_detect_fallback_1123.py
JD Davis a708c0571e
fix(ci): prevent native detector from hanging test shards (#2996)
## Description

CI shard 4 was not merely slow: after thousands of fast tests it parked
indefinitely inside `headroom._core.detect_content_type` at 0% CPU. The
router watchdogged only the first native call and then permanently
trusted direct calls via `_detect_native_verified`. Earlier suite
activity can change ORT/native state after that first success, making a
later call deadlock until GitHub cancels the job.

This keeps every native call bounded by the existing watchdog, activates
the process-wide pure-Python circuit breaker after a timeout, restores
the test-job ceiling to 30 minutes, and removes a separate wall-clock
scheduler assertion that generated false shard-1 failures despite the
structural regression guards passing.

No issue is auto-closed by this infrastructure repair.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Removed the unsafe process-lifetime `_detect_native_verified` fast
path.
- Kept every native detection call behind the existing bounded watchdog.
- Preserved the process-wide fallback circuit breaker so only the first
wedged call consumes the watchdog budget.
- Added a success-then-hang regression test.
- Isolated native circuit-breaker state in fallback exception tests.
- Restored the CI test timeout from the temporary 90-minute diagnostic
ceiling to 30 minutes.
- Replaced the Codex scheduler's noise-sensitive p99/p50 assertion with
its meaningful absolute regression ceiling while retaining source-level
guards against the removed semaphore and nested executor.
- Corrected import order and formatting defects inherited from current
main so the synthetic merge commit passes repository-wide lint.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
Exact local shard-4 command with coverage:
2723 passed, 172 skipped, 8661 deselected in 108.40s

Focused detector/router suite:
62 passed

Codex scheduler suite:
3 passed, 1 skipped

ruff check .
All checks passed!

ruff format --check .
1411 files already formatted

mypy headroom/transforms/content_router.py
Success: no issues found in 1 source file
```

Exact-head GitHub CI on `28f284c7a1` is
entirely green. Test jobs 1–4, test-extras, test-agno, build, wheel,
lint, CodeQL, dependency audit, secret scan, smoke, governance, and
conflict checks all passed. Remaining skips are path-filtered jobs not
applicable to this diff.

## Real Behavior Proof

- Environment: macOS arm64/Python 3.13 locally; GitHub-hosted
Ubuntu/Python 3.12 using the production CI workflow and prebuilt wheel.
- Exact command / steps: reproduced `pytest tests scripts/tests --splits
4 --group 4 ...` hanging in native detection; sampled the parked
process; reran with `pytest-timeout` to locate `_rust_detect`; applied
the correction; reran the exact shard locally and all four CI shards
remotely.
- Observed result: local shard 4 completed in 1:48. GitHub shard 4's
pytest step completed in 5:45 and its full job in 8:06 under the
restored 30-minute ceiling. All four shards passed on the same head.
- Not tested: deliberately wedging a real production ORT runtime outside
the deterministic mocked regression; the watchdog behavior is covered
with a native-call fake that succeeds once and then never returns.

## Runtime Rollout Safety

- Rollout-managed feature(s): native content detection watchdog and
fallback only.
- Minimum rollout channel: normal patch release; no staged feature flag
required.
- Stable/default behavior changed: every native detection call remains
watchdog-bounded instead of only the first successful call.
- Kill switch / disable path: `HEADROOM_DETECT_BACKEND=python` bypasses
native detection; `HEADROOM_DETECT_TIMEOUT_SECS` controls the watchdog
budget.
- Unsafe override required: none.
- Qualification impact: full Python CI matrix must remain green; exact
shard-4 completion is the primary qualification evidence.
- Rollback path: human revert of this PR if bounded calls cause an
unexpected regression; setting the Python backend provides an immediate
operational fallback without code rollback.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation — inline
lifecycle documentation and PR operational notes; no user-facing docs
change is needed
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

Not applicable; no UI change.

## Additional Notes

Human review only. No merge or auto-merge action has been configured.
The branch includes current main and preserves the MCP SDK compatibility
cap `mcp>=1.28.1,<2.0.0`.
2026-08-13 20:47:55 -05:00

86 lines
3.3 KiB
Python

"""Native (Rust) content-detector failures must degrade to the pure-Python
detector instead of propagating out as an HTTP 500. Regression test for #1123."""
from __future__ import annotations
import asyncio
import pytest
import headroom._ort as ort_runtime
from headroom.transforms import content_router as cr
# Patch the native detector via its string target ("headroom._core.detect_content_type")
# rather than a module alias captured at import time. content_router._detect_content does a
# fresh `from headroom._core import detect_content_type` on every call, and other tests pop
# headroom._core out of sys.modules (e.g. test_rust_core_smoke), which rebuilds the module
# object. A captured alias would then go stale and the patch would miss the live module —
# the control-flow tests would silently run the real detector and never see the exception.
@pytest.fixture(autouse=True)
def _compatible_mock_native_runtime(monkeypatch: pytest.MonkeyPatch) -> None:
"""Keep mocked native calls reachable regardless of prior test state."""
monkeypatch.setattr(ort_runtime, "rust_ort_runtime_compatible", lambda: True)
monkeypatch.setattr(cr, "_detect_native_unhealthy", False)
def test_falls_back_on_rust_exception(monkeypatch):
"""An ordinary exception from the native detector degrades to regex."""
def _boom(_content):
raise RuntimeError("simulated native failure")
monkeypatch.setenv("HEADROOM_DETECT_BACKEND", "rust")
monkeypatch.setattr("headroom._core.detect_content_type", _boom)
monkeypatch.setattr(cr, "_detect_panic_warned", False, raising=False)
# Must not raise; returns a usable detection result from the regex path.
result = cr._detect_content('{"a": 1, "b": [1, 2, 3]}')
assert result is not None
assert result.content_type is not None
def test_falls_back_on_baseexception_panic(monkeypatch):
"""A BaseException-derived panic (like pyo3's PanicException) is caught too."""
class FakePanic(BaseException):
pass
def _panic(_content):
raise FakePanic("simulated pyo3 panic")
monkeypatch.setenv("HEADROOM_DETECT_BACKEND", "rust")
monkeypatch.setattr("headroom._core.detect_content_type", _panic)
monkeypatch.setattr(cr, "_detect_panic_warned", False, raising=False)
result = cr._detect_content("some plain text content here")
assert result is not None
def test_control_flow_exceptions_propagate(monkeypatch):
"""KeyboardInterrupt/SystemExit must not be swallowed by the fallback."""
def _interrupt(_content):
raise KeyboardInterrupt
monkeypatch.setenv("HEADROOM_DETECT_BACKEND", "rust")
monkeypatch.setattr("headroom._core.detect_content_type", _interrupt)
monkeypatch.setattr(cr, "_detect_panic_warned", False, raising=False)
with pytest.raises(KeyboardInterrupt):
cr._detect_content("content")
def test_cancelled_error_propagates(monkeypatch):
"""asyncio.CancelledError must propagate, not be swallowed as a fallback."""
def _cancel(_content):
raise asyncio.CancelledError()
monkeypatch.setenv("HEADROOM_DETECT_BACKEND", "rust")
monkeypatch.setattr("headroom._core.detect_content_type", _cancel)
monkeypatch.setattr(cr, "_detect_panic_warned", False, raising=False)
with pytest.raises(asyncio.CancelledError):
cr._detect_content("content")