mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
23 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
JD Davis
|
a708c0571e
|
fix(ci): prevent native detector from hanging test shards (#2996)
## Description
CI shard 4 was not merely slow: after thousands of fast tests it parked
indefinitely inside `headroom._core.detect_content_type` at 0% CPU. The
router watchdogged only the first native call and then permanently
trusted direct calls via `_detect_native_verified`. Earlier suite
activity can change ORT/native state after that first success, making a
later call deadlock until GitHub cancels the job.
This keeps every native call bounded by the existing watchdog, activates
the process-wide pure-Python circuit breaker after a timeout, restores
the test-job ceiling to 30 minutes, and removes a separate wall-clock
scheduler assertion that generated false shard-1 failures despite the
structural regression guards passing.
No issue is auto-closed by this infrastructure repair.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Removed the unsafe process-lifetime `_detect_native_verified` fast
path.
- Kept every native detection call behind the existing bounded watchdog.
- Preserved the process-wide fallback circuit breaker so only the first
wedged call consumes the watchdog budget.
- Added a success-then-hang regression test.
- Isolated native circuit-breaker state in fallback exception tests.
- Restored the CI test timeout from the temporary 90-minute diagnostic
ceiling to 30 minutes.
- Replaced the Codex scheduler's noise-sensitive p99/p50 assertion with
its meaningful absolute regression ceiling while retaining source-level
guards against the removed semaphore and nested executor.
- Corrected import order and formatting defects inherited from current
main so the synthetic merge commit passes repository-wide lint.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
Exact local shard-4 command with coverage:
2723 passed, 172 skipped, 8661 deselected in 108.40s
Focused detector/router suite:
62 passed
Codex scheduler suite:
3 passed, 1 skipped
ruff check .
All checks passed!
ruff format --check .
1411 files already formatted
mypy headroom/transforms/content_router.py
Success: no issues found in 1 source file
```
Exact-head GitHub CI on `
|
||
|
JD Davis
|
a3fe5cb65b
|
fix(onnx): enforce Rust API-24 runtime compatibility (#2979)
## Description Rust fastembed enables ORT C API 24, but the Python dependency allowed ONNX Runtime 1.23.2. Entering ort's initializer with that library deadlocks permanently instead of returning an error. Align dependency resolution where compatible wheels exist and preflight native detection where they do not. Closes #2960 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Require ONNX Runtime 1.24+ for Python 3.11+ in the proxy and voice extras. - Keep the available pre-1.24 runtime on Python 3.10 for Python ONNX consumers. - Refuse to auto-pin an incompatible runtime into the Rust extension. - Bypass native detection immediately when API 24 is unavailable, preserving Python fallback without a five-second watchdog delay or stuck native thread. - Add dependency, pinning, override, and router regression coverage. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest -q tests/test_transforms/test_ort_dylib.py tests/test_onnx_dependency_contract.py tests/test_onnx_runtime.py tests/test_transforms/test_content_router.py 88 passed in 9.31s $ uv run ruff check headroom/_ort.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py tests/test_onnx_dependency_contract.py All checks passed! ``` ## Real Behavior Proof - Environment: macOS arm64; Python 3.13.14 and uv-managed Python 3.10.20. - Exact command / steps: run the issue's direct `headroom._core.detect_content_type` call in a subprocess with a 12-second timeout on Python 3.13; run `_detect_content` on Python 3.10 after resolving the proxy extra. - Observed result: Python 3.13 resolves ORT 1.26.0 and native detection returns `json_array`; Python 3.10 resolves ORT 1.23.2, leaves `ORT_DYLIB_PATH` unset, reports compatibility false, and immediately returns the Python `json_array` fallback. - Not tested: Linux-specific shared-object execution locally; CI's existing Linux Rust job already preflights ORT 1.24+ and exercises native tests. ## Runtime Rollout Safety - Rollout-managed feature(s): Native Rust content detection. - Minimum rollout channel: Stable/default; this is a deadlock prevention guard. - Stable/default behavior changed: Python 3.11+ installs a compatible ORT; Python 3.10 skips incompatible native detection. - Kill switch / disable path: `HEADROOM_DETECT_BACKEND=python` remains available; an explicit `ORT_DYLIB_PATH` remains an operator override. - Unsafe override required: No. - Qualification impact: Native detection stays enabled only with API-24-compatible ORT. - Rollback path: Revert this PR, which restores the old watchdog-only degradation. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable. ## Additional Notes The large lockfile diff is dependency resolution: Python 3.10 keeps ORT 1.23.2 while 3.11+ resolves 1.26.0. The functional Python change is intentionally small and keeps explicit `ORT_DYLIB_PATH` overrides working. |
||
|
michaeltarleton
|
677e09735a
|
fix(transforms): stop ContentRouter recompressing headroom_retrieve results (#2654)
## Description `ContentRouter` (the transform actually registered in the default/proxy compression pipeline -- see `transforms/pipeline.py`) recompresses the output of its own `headroom_retrieve` tool. That tool's entire contract is returning already-retrieved, original content verbatim; recompressing it produces a new `<<ccr:hash>>` marker the caller can never redeem -- an unresolvable retrieval loop. `SmartCrusher` already has a guard against this exact failure mode (#1077), but only on its `apply()` entry point. `ContentRouter` calls the lower-level `SmartCrusher.crush()` directly, bypassing that guard entirely, since `crush()` takes a raw content string with no tool identity at all. Closes #1077 (reopens the same failure mode ContentRouter's own call path, which #1077's original fix did not cover). ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `transforms/content_router.py`: adds an unconditional guard to all three of the places `ContentRouter` can hand a `headroom_retrieve` result to compression: the OpenAI-shape `role:"tool"`/legacy `role:"function"` string-content loop, the Anthropic-shape `tool_result` block loop, and a third, distinct shape -- top-level `{"type": "text"}` blocks under a `role:"tool"`/`"function"` message that never go through a `tool_result` wrapper (a real, already-tested wire shape in this codebase; see `test_tool_role_text_blocks_compressed_by_default`). All three use `is_tool_excluded()` (not a bare comparison) because MCP-served tools appear here under their qualified form, e.g. `mcp__headroom__headroom_retrieve`. Legacy `role:"function"` messages carry no call id in that shape, so the tool name is read directly off the message's `name` field instead of through the id-keyed `tool_name_map`. - Hoisted the per-iteration `is_tool_excluded(..., ("headroom_retrieve",))` calls into a single precomputed `ccr_retrieve_tool_ids` set, computed once alongside the existing `excluded_tool_ids` set, rather than recomputing aliases on every message/block. - `config.py`: adds `"headroom_retrieve"` to `DEFAULT_EXCLUDE_TOOLS` and `DEFAULT_VERBATIM_EXCLUDE_TOOLS` -- this also covers a third path (cross-turn message dedup, `_cross_turn_dedup_messages`) that consults the same frozensets and has no dedicated guard of its own. Also hardens `_tool_name_aliases()` against a non-string tool name (pre-existing fragility, not introduced by this PR, but shares the same call path) by returning no aliases instead of crashing on `.lower()`. - Documentation: updated `ContentRouterConfig.exclude_tools`'s field comment (was stale -- didn't mention this override is unconditional even when a caller explicitly empties `exclude_tools`), and added a comment on `DEFAULT_VERBATIM_EXCLUDE_TOOLS` noting all three real consumers. - Kept `"headroom_retrieve"` as a literal string (matching every other entry in those frozensets) rather than importing the existing `CCR_TOOL_NAME` constant from `ccr.tool_injection` into `content_router.py` -- that module is imported eagerly by `pipeline.py` (unlike `smart_crusher.py`, which imports the same constant lazily), so pulling in `headroom.ccr` there would add a new eager-import edge to a hot module for a one-line DRY win. Happy to change this if a maintainer prefers the constant. **Known, accepted tradeoff:** `is_tool_excluded()`'s alias matching strips any `mcp__<server>__` prefix before comparing, so a third-party MCP server exposing a tool literally named `headroom_retrieve` would also match. Narrowing this to headroom's own server specifically would need a bespoke check inconsistent with how every other excluded-tool entry is matched in this codebase; given how specific the name is, the collision risk is accepted rather than special-cased. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_transforms/ tests/test_transforms_content_router.py -q 1 failed, 420 passed, 62 skipped in 12.50s FAILED tests/test_transforms/test_kompress_compressor.py::...test_onnx_session_options_read_thread_caps (pre-existing, unrelated to this diff -- confirmed via `git stash` that it fails identically against unmodified upstream/main; an ONNX thread-cap assertion, not a compression-routing test) $ uv run ruff check headroom/config.py headroom/transforms/content_router.py \ tests/test_transforms/test_content_router_ccr_retrieve_exemption.py \ tests/test_transforms_content_router.py tests/test_transforms/test_content_router.py All checks passed! $ uv run ruff format --check <same files> 5 files already formatted $ uv run mypy headroom/config.py headroom/transforms/content_router.py Success: no issues found in 2 source files ``` - `tests/test_transforms/test_content_router_ccr_retrieve_exemption.py`: 10 tests -- MCP-qualified name (Anthropic + OpenAI shape), bare name, unconditional-even-with- `exclude_tools=frozenset()`, negative control (normal tools still compressed, asserted via the absence of the `router:excluded:ccr_retrieve` marker), the top-level-text-block shape, legacy `role:"function"`, litellm list-form content nested in a `tool_result` block, mixed retrieve+normal blocks in one turn, and a content well below the compression floor (proving the guard is size-independent). - `tests/test_transforms/test_content_router.py`: `test_anthropic_mcp_bare_tool_alias_exclude_tools` (#1822) updated to assert the new, stronger byte-verbatim guarantee for `headroom_retrieve` specifically; `test_anthropic_mcp_bare_tool_alias_exclude_tools_generic` added to keep the original #1822 general-mechanism coverage (bare-alias matching for an arbitrary, non-exempt tool). - `tests/test_transforms_content_router.py`: updated 10 pre-existing `_process_content_blocks()` unit tests for the new `ccr_retrieve_tool_ids` parameter (all pass empty sets -- none of those tests involve `headroom_retrieve`). - Verified the local installed package copy (a separate, drifted internal version) with a standalone repro script exercising the two new shapes directly against `ContentRouter.apply()` -- both correctly report `router:excluded:ccr_retrieve`. ## Real Behavior Proof - Environment: Windows 11, Python 3.13.7, `uv sync --extra dev` on this branch. - Exact command / steps: standalone repro building an assistant `tool_use` for `mcp__headroom__headroom_retrieve` paired with a large-JSON `tool_result`, through `ContentRouter().apply()`; repeated for the top-level-text-block and legacy-`function`-role shapes. - Observed result: unpatched (Anthropic `tool_result` shape, `git stash` to `upstream/main`), the retrieve output was rewritten 3680 -> 1861 bytes (mangled into a compact tabular form); patched (this branch), it is forwarded 3680 -> 3680 bytes byte-identical, no `<<ccr:` marker present. The two additional shapes fixed in this PR's second commit -- top-level text block under `role:"tool"`, and legacy OpenAI `role:"function"` -- both report `excluded=True` (protected) against this branch, where they reported `excluded=False` (recompressed) before the second commit. - Not tested: the actual `headroom mcp serve` + `headroom wrap` proxy end-to-end over a live Anthropic API call (would need API credentials); the OpenAI-chat-completions `CompressionUnit` path (out of scope, see #1176 below); the opt-in `ToolResultInterceptorTransform` path. ## Relationship to other issues/PRs - Issue #1077 (closed) is this exact bug; PR #1323 fixed it only for `SmartCrusher.apply()`'s own call path (the "legacy" pipeline path, per `smart_crusher.py`'s own comment), not `ContentRouter`, which is what the default/proxy pipeline actually uses. - Open PR #1176 addresses an adjacent, non-overlapping gap: the `CompressionUnit`-based OpenAI chat-completions path (`router.compress()` calls in `transforms/compression_units.py`/`compression_batches.py`), which has no tool-identity context at all and needs its own capture/restore mechanism. This PR does not touch that path. - Filed #2656 as a follow-up: code review on this PR found the same bug class still reachable through `SmartCrusher.apply()`'s own bare-name guard (not alias-aware, so it misses the MCP-qualified form) and through two unguarded direct `.crush()` calls in the LangGraph and Strands integrations. Both are pre-existing, narrower/separate call paths from `ContentRouter`'s primary proxy pipeline, so tracking them separately keeps this PR reviewable as one logical change. - Also not covered by this PR (flagging rather than silently omitting): `proxy/system_compaction.py`'s `router.compress(text, context="")` call, and the opt-in `ToolResultInterceptorTransform` (`HEADROOM_INTERCEPT_ENABLED=1`) -- neither was checked for CCR-awareness. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` -- it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A -- this is a backend compression-routing fix with no UI surface. ## Additional Notes This PR is two commits: the first commit added the initial two-loop guard; a second commit followed after code review found the guard was incomplete for two additional wire shapes (top-level text blocks, legacy `role:"function"`) and added the missing test coverage plus a few cleanup items (deduplicated guard logic, comment accuracy, a pre-existing non-string-tool-name fragility). See `Changes Made` above for the full list. Filed #2656 for the remaining out-of-scope gaps found during that same review. --------- Co-authored-by: Michael Tarleton <mtarleton@istation.com> |
||
|
inix
|
9b016f2b64
|
perf(content_router): dedupe content detection (#2419)
## Description
ContentRouter ran the native content detector two to three times on
identical content, on the hottest path in the proxy (every compressed
message, every request). This cuts it to once.
`_detect_content` isn't cheap and isn't memoized. It strips a detection
envelope, runs the Rust/Magika ONNX classifier, then several regex
passes. `compress()` ran it once for debug logging that's off by
default, then `_determine_strategy()` recomputed it (plus
`is_mixed_content`) on the same content. That's twice per `compress()`,
and three times on the `apply()` cache-miss path.
Closes: N/A (no filed issue, surfaced by an internal
contribution-backlog audit).
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `compress()` computes `is_mixed_content` and `_detect_content` once,
then threads both into `_determine_strategy` through new optional params
(`mixed`, `detection`).
- `_determine_strategy` uses the passed values when present, and
computes them itself when they're `None`. Its one private caller
changes. Any other caller keeps working.
- Added `tests/test_content_router_detection_dedup.py`. One test asserts
`compress()` detects exactly once (it fails before the fix at `assert 2
== 1`). The other asserts the threaded result routes the same as the
recomputed one across content types.
- Updated two existing `_determine_strategy` test doubles to take the
new kwargs.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_content_router_detection_dedup.py tests/test_transforms_content_router.py \
tests/test_transforms/test_content_router.py tests/test_transforms_content_detection.py -q
135 passed in 8.98s
$ pytest tests/test_transforms/ tests/test_content_router_*.py tests/test_router_*.py \
tests/test_lossless_excluded_compaction.py -q
423 passed, 62 skipped in 54.93s
$ ruff check .
All checks passed!
$ mypy headroom
Success: no issues found in 505 source files
```
## Real Behavior Proof
- Environment: macOS (Darwin), Python 3.13, headroom worktree on this
branch off `upstream/main`, `HF_HUB_OFFLINE=1
LITELLM_LOCAL_MODEL_COST_MAP=true`. A counter wraps the real
`_detect_content` and delegates to it, so real routing and compression
run.
- Exact command / steps: run the real router over one representative
message and count `_detect_content` calls on the fixed tree, then `git
stash` the source and count again on the unfixed tree. Covered
`router.compress(blob)` and `router.apply([tool_msg])`.
- Observed result: `compress()` dropped from 2 detection calls to 1, and
`apply()` dropped from 3 to 2, on the same input with the same routing
strategy (`text`) and the same output. The once-only test flips from
`assert 2 == 1` before to passing after.
- Not tested: production Magika ONNX timing. This dev env has no
onnxruntime, so the detector ran its regex fallback tier, which makes
the saved cost a floor, not a ceiling. I also scoped out the Tier B
extension (threading the `apply()` Pass-1 detection into `compress()`)
on purpose.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A
## Additional Notes
Scope is the default routing path. `force_kompress` already uses the
cheaper regex detector, so it never paid the redundant native cost.
`_compress_mixed` re-detects per split section, but that's different
content (sub-sections), so it's out of scope.
The `apply()` Pass-1 detection stays. It gates the `is_code` protection
check for every message, including cache hits that never reach
`compress()`. Threading it into `compress()` would widen a shared task
tuple and change the public `compress()` signature, all for a
cache-miss-only save, so I left it as a possible follow-up.
Doc checklist item is N/A (internal perf dedup, no user-facing docs
change). This is a Python-only change, so the first push will use
`--no-verify` for the known `ci-precheck` Rust-latency bench flake
(`classify_under_10us_per_call`), which runs clean in CI.
|
||
|
Tejas Chopra
|
1d79e70f95
|
fix(tests): repair three main-branch test failures (#2306)
## Description `main` CI is red on three independent test failures. All three are **test-side** bugs (stale cache, semantic merge conflict, stale mock) — no product code regressed. Each test passed in isolation but failed on `main`, and each also blocks the `chore: release main` PR (#1923). Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - **`test_l2_appends_transform_label`** — `tool_desc_max_chars()` memoises into a module global. An earlier test in shard 1 reads it with the env unset, pinning the cache to `0`, so this test's `setenv("HEADROOM_TOOL_DESC_MAX_CHARS=20")` was swallowed (`assert 0 == 20`). Reset the cache before reading and after, mirroring the sibling `test_l2_skips_label_when_disabled`. - **`test_dashboard_uses_cached_stats_and_lazy_history_feed_polling`** — semantic merge conflict: #2198 (persist lifetime metrics) intentionally retired the session-card `Filtered (lifetime)` row and moved CLI-filtering lifetime into the history tab as `Lifetime Saved`, while the assertion from #1433 still checked the old string. Assert the current `Lifetime Saved` label. - **`test_smart_crusher_log_fallback_runs_for_valid_json`** — stale mock: #1857 made token counting whitespace-aware, so the router now rates the JSON above the naive `len(content.split())==8` the no-op kompress mock reported, making it look like a saving and short-circuiting before the Log fallback. Mock now reports `_estimate_tokens(content)` to match the router. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) ### Test Output ```text $ pytest tests/test_anthropic_compaction_transforms.py \ tests/test_proxy_dashboard_stats_cache.py \ tests/test_transforms_content_router.py -q 78 passed, 1 skipped in 12.14s $ ruff check <the three files> All checks passed! $ ruff format --check <the three files> 3 files already formatted ``` ## Real Behavior Proof - Environment: local `.venv`, Python 3.12.6, pytest 9.0.2 (same three tests that fail on the `main` CI shards 1/3/4). - Exact command / steps: ran the three previously-failing tests by node id — all pass. Reproduced the shard-isolation failure for #1 by calling `tool_desc_max_chars()` with the env unset (cache → 0) before the test, confirmed the reset makes it pass. - Observed result: 3/3 target tests pass; 78 passed / 1 skipped across the three full files. - Not tested: full suite (unchanged product code); CI shards will re-run on this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes `mypy headroom` (the CI-enforced scope) is unaffected — these edits touch only `tests/`, which CI does not type-check. Once this lands on `main`, the `chore: release main` PR (#1923) drops to just the `test_root_server_json_matches_builder` failure, which is the release version-bump `server.json` regen (not a code bug). |
||
|
monkeygold
|
02c77640a9
|
fix(transforms): guard Log fallback against invalid JSON + fix MIXED false-positive on source code (#1347)
## Summary Three related fixes in the content router/detector, addressing data-loss and misrouting bugs found via chaotic audit: - **SMART_CRUSHER → Log fallback guard (#1306):** Truncated/invalid JSON tool outputs were tagged `json_array` by the native magika detector (classifies by shape, not parseability), routed to SmartCrusher (no-op), Kompress (no-op), then collapsed by LogCompressor to a single CCR-retrieval marker — **99.9% data loss** when CCR retrieval isn't configured. A JSON-validity guard (`_content_is_valid_json`) now skips the Log fallback for content that fails `json.loads`; valid JSON arrays still reach it (LogCompressor is a no-op on them). - **MIXED false-positive on source code:** `is_mixed_content` regex heuristics misclassify Python with dict/list literals (`{`, `[` at line start → `has_json_blocks`) + docstrings (`has_prose`) as MIXED, wasting 1–1.4s latency with 0% compression. When the native detector confidently says `SOURCE_CODE` (confidence ≥ 0.8), `_determine_strategy` now trusts it over the regex heuristics. - **PASSTHROUGH for code when CodeAware disabled:** When `prefer_code_aware_for_code=False` (default), source code now uses `PASSTHROUGH` instead of `KOMPRESS`, honouring the config's "let code pass through unmangled" intent. KOMPRESS can destroy code semantics (98% compression, 11% fact recall on large blobs). - **RecursionError hardening:** Caught in both `_try_detect_json` and `_content_is_valid_json` so deeply nested JSON (`[[[[...]]]]` with 10k+ levels) no longer crashes the detector/router — also serves as a DoS mitigation. #### Test plan - [x] `tests/test_transforms_content_router.py` — 36 passed (8 new tests) - [x] `tests/test_transforms_content_detection.py` — 9 passed - [x] `tests/test_cache_aligner_detector_only.py` — 22 passed - [x] `tests/test_compression_decision.py`, `test_compression_policy.py`, `test_compress_api.py`, `test_compression_safety_rails.py` — 137 passed, 5 skipped - [x] `ruff check` on changed files — all checks passed - [x] `mypy` on changed files — no issues found New tests cover: - Invalid JSON skips Log fallback (content preserved verbatim) - Valid JSON arrays still reach Log fallback - MIXED false-positive overridden by high-confidence SOURCE_CODE detection - Low-confidence SOURCE_CODE does NOT override MIXED (safety) - Genuine mixed content (PLAIN_TEXT detection) still uses MIXED - PASSTHROUGH preserves code verbatim, never invokes Kompress - CodeAware explicitly enabled still uses CODE_AWARE #### Risks / rollback - Behaviour change: code blobs previously routed through MIXED→KOMPRESS now use PASSTHROUGH. This is the documented intent of `prefer_code_aware_for_code=False`; if a deployment relied on the accidental KOMPRESS compression of code, set `prefer_code_aware_for_code=True` to restore CODE_AWARE. - The JSON-validity guard adds one `json.loads` call in the narrow "no savings" fallback path only — negligible overhead. - Revert is a single-commit revert; no schema/migration changes. Generated with [Devin](https://devin.ai) Co-authored-by: monkeygold <monkeygold@users.noreply.github.com> Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
vscunha
|
05932d7165
|
fix(proxy): compress OpenCode tool schemas and embedded JSON (#1535)
## Description
Fixes two remaining OpenCode/OpenAI Chat compression gaps after `main`
incorporated the original savings-profile threading and user
content-block work from this PR.
OpenCode requests can still report very low savings when most input
tokens live in verbose `tools` schemas rather than messages. They can
also route poorly when a short instruction wraps a valid JSON block but
does not satisfy the existing long-prose heuristic.
Closes #1534
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that causes existing functionality
to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Compact OpenAI Chat Completions `tools` schemas whenever request
compression is active, reusing the existing OpenAI Responses schema
compactor. The outbound tool invocation shape is preserved while
non-semantic annotations such as `$schema`, `title`, and `examples` are
removed.
- Include the tool-schema token delta in Headroom's savings accounting
and expose `openai:chat:tool_schema_compaction` in the applied
transforms.
- Detect valid JSON blocks surrounded by prose or log text as mixed
content, so short OpenCode instructions route through mixed/SmartCrusher
handling instead of falling through or producing a no-op.
- Adapt the mixed-content change to the new
`headroom.transforms.mixed_content` module introduced on `main` by
#1939.
## Why the Focus Changed
The original headline fix—threading savings-profile kwargs into
`/v1/chat/completions`—is now already present on `main`, as is the user
content-block opt-in behavior. Those duplicate changes were removed
during the merge.
The branch also no longer changes developer/system role protection or
forced-Kompress semantics. It follows `main` for both, so the earlier
instruction-role safety concern is outside the current diff.
The resulting PR is limited to two OpenCode-specific compression gaps
that remain reproducible on current `main`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check` on changed files)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for the fixed behavior
- [ ] Manual live-upstream testing performed after the latest rebase
### Test Output
```text
59 passed, 1 warning in 83.53s
All checks passed! # ruff check
4 files already formatted # ruff format --check
python -m py_compile: passed
git diff --check: passed
```
Focused test coverage includes:
- OpenAI Chat tool-schema compaction, transform reporting, outbound
schema shape, and positive token savings.
- Embedded JSON mixed-content detection, SmartCrusher routing, positive
savings, and preservation of a critical sentinel value.
- Current `main` regressions for savings-profile threading, user content
blocks, turn hooks, and forced-Kompress behavior.
## Real Behavior Proof
- Environment: Linux ARM64, Python 3.13.12, current `main` at `
|
||
|
Rocker Zhang
|
a069979466
|
fix(content_router): pin FREEZE_BLOCK_DECISION verdict to stop cache-write churn (#1620)
## Description The per-block freeze decision was inert. On the cached-block re-check, a later tighter `min_ratio` (context pressure rises within a session) could downgrade an earlier "compress" verdict to skip, restore the original block, and bust the prefix cache — a self-inflicted cache-write churn that costs the very tokens compression saved. This pins the decision instead. A frozen "compress" verdict re-accepts (`accept_threshold = 1.0`) rather than re-running the per-turn `min_ratio` gate, so a block that was accepted stays accepted. First-sighting still uses the live `min_ratio` gate (`accept_threshold = min_ratio`): the freeze only pins past accepts, it never loosens the first decision (that would be a silent ratio bet). Gated behind `HEADROOM_FREEZE_BLOCK_DECISION`, default off, byte-identical to today when unset. Composes with the #1307 reversibility guard on the compress path: a frozen accept still defers to the lossy-unrecoverable skip. Closes #1619 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `content_router.py`: on the cached-block hit path and the first-sighting path, compute `accept_threshold` (1.0 when a "compress" verdict is frozen for the block, else the live `min_ratio`) and gate accept on it; record the pin when the legacy re-check would have downgraded. - Frozen verdicts are stored per content-block key and only ever hold "compress" (a "skip" never warms the result cache). - No change when `HEADROOM_FREEZE_BLOCK_DECISION` is unset. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom` — no new errors) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest tests/test_transforms_content_router.py -q 38 passed in 0.23s # 11 freeze/pin/churn cases + 27 existing; run against a real 0.28.0 _core. # - test_freeze_off_is_byte_identical_flapping_baseline: freeze-off == baseline byte-for-byte # - test_freeze_on_pins_compress_verdict_across_turns: pin fires; downgrade prevented $ ruff format --check . && ruff check . -> clean ``` ## Real Behavior Proof - Environment: isolated git worktree on latest `main`, real `_core.abi3.so` built for this tree via maturin (not a stale symlink), scratch venv. - Exact command: `pytest tests/test_transforms_content_router.py -q` - Observed: freeze-off path is byte-identical to the flapping baseline; with freeze on, the verdict is pinned across turns (pin-count assertion passes) so the block is not downgraded/restored and no cache-write churn occurs. - Not tested: end-to-end proxy A/B token-savings delta (follow-up; feature ships default-off). ## Screenshots (if applicable) N/A ## Review Readiness Ready for review. Default-off, composes with #1307, self-contained to the block-decision path. ## Checklist - [x] My code follows the style guidelines of this project - [x] I have performed a self-review of my code - [x] I have commented my code in hard-to-understand areas - [x] Documentation intentionally deferred until the default-off approach is confirmed - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes - [x] CHANGELOG intentionally deferred until the default-off approach is confirmed ## Additional Notes Neighbour of #625 (prefix-stability). Docs/CHANGELOG intentionally deferred until the approach is confirmed. The end-to-end A/B is a follow-up; the churn-prevention is proven at the router unit level here. Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Dima Solodukha
|
4e30dde2ac
|
fix(router): compact JSON evades compression via whitespace token counting (#1857)
## Description
`ContentRouter` counts section tokens with `len(content.split())`. On
compact machine-generated JSON — the default output of
`json.dumps(separators=(",", ":"))`, `JSON.stringify`, and boto3 — there
are no spaces, so a large payload counts as ~1 "token". Every section
compression ratio then computes as ~1.0 and the `min_ratio` acceptance
gate silently rejects the compressor's real output: the router logs
`router:noop` while SmartCrusher separately logs `was_modified=true`.
Compression effectively no-ops on the most common agent payload type
(tool results returning JSON), on every provider.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Add `_estimate_tokens(text)` — a size-proportional estimate
(`len(text) // 4`, floored at 1), monotone in content size for any
format.
- Replace the decision-relevant `len(...split())` counts in
`ContentRouter` (section original/compressed token counts feeding the
ratio gates, plus the debug estimates) with `_estimate_tokens(...)`.
- Add `tests/test_content_router_compact_json.py`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_content_router_compact_json.py -q
2 passed, 1 warning
$ ruff check headroom/transforms/content_router.py tests/test_content_router_compact_json.py
All checks passed!
$ mypy headroom/transforms/content_router.py --ignore-missing-imports
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: `ContentRouter` invoked directly on a 150-item
ECS-service JSON tool_result, Python 3.13, estimator tokenizer.
- Exact command / steps: run the same payload two ways — compact
(`json.dumps(..., separators=(",", ":"))`) and the identical data with
spaces (`separators=(", ", ": ")`) — through
`ContentRouter(ContentRouterConfig(skip_user_messages=False))`.
- Observed result: before this change, compact JSON saved 0.0%
(`router:noop`) while the identical data with spaces saved 43.3%
(`router:tool_result:smart_crusher`) — same data, same compressor, only
whitespace differed. After this change, compact JSON compresses
equivalently to the spaced form.
- Not tested: no behavior change expected for content that already
tokenizes with whitespace (prose, code); those counts move from
word-count to chars/4 but the ratio comparison is self-consistent (both
sides use the same estimator).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
Scope kept deliberately narrow: only the counts that feed
compression-acceptance decisions are changed. Non-decision `.split()`
uses elsewhere are left alone. Happy to add a CHANGELOG entry if you'd
like one.
---------
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
Tejas Chopra
|
0d18ef26f4
|
fix(transforms/content-router): route grep/log output away from HTML extractor (#1719)
## Description Follow-up to #1717 (envelope-aware detection). Even when the tool-output envelope is unwrapped, the native (magika) detector still tags dense `grep`/`rg` output and build logs as **HTML** — file paths and `</>`/brackets read as markup. Those then get routed to the HTML article-extractor, which is lossy for that content (it strips the code and identifiers the lines carry). When the structural log/search detectors positively claim the payload, override the HTML verdict: build output / tracebacks → LOG (checked first), `path:line` grep output → SEARCH. It **reuses the existing `_try_detect_log` / `_try_detect_search` detectors**, so no new pattern or regex is introduced, and it only ever reconsiders an HTML verdict — every other detection is untouched. Per-content and deterministic (no cross-turn state), so prefix caching is unaffected. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `_detect_content()`: when the native detector returns `HTML`, re-check with `_try_detect_log` then `_try_detect_search` and return their verdict when they claim the payload (`headroom/transforms/content_router.py`). - Regression test. ## Testing - [x] Unit tests pass - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality ### Test Output ```text $ pytest tests/test_transforms_content_router.py tests/test_transforms_content_detection.py -q 46 passed in 0.94s $ ruff check headroom/transforms/content_router.py tests/test_transforms_content_router.py All checks passed! $ mypy headroom/transforms/content_router.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: local, Python 3.12.6, native `headroom._core` detect backend. - With the native detector forced to `html`: `grep`-over-`.html`-template output detects as `SEARCH_RESULTS`, a build/error log as `BUILD_OUTPUT`, and a genuine HTML article as `HTML` (override does not fire). - Verified directly that raw magika returns `html` for realistic `grep`-over-HTML output, and that this change reroutes it to `search`. - Not tested: end-to-end proxy request replay. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes - Builds on #1717; the two changes live in the same `_detect_content` function (both prevent tool output from being misrouted to the HTML extractor). - Pre-existing mypy findings in `tests/test_transforms_content_router.py` are unrelated and left as-is; `mypy headroom` is clean and the added test is typed. |
||
|
Tejas Chopra
|
a85a04be87
|
fix(transforms/content-router): detect on inner tool-output payload (#1717)
## Description
Coding-agent harnesses wrap each tool result in an envelope such as
`<returncode>0</returncode>\n<output>…</output>` (also `<stdout>`,
`<stderr>`,
`<tool_result>`, `<result>`). The native content detector read those
wrapper
tags as markup and classified the whole payload as HTML/XML — so source
code,
grep results, and logs were misrouted to the HTML article-extractor,
which
blanks or corrupts them (dropping identifiers and route converters).
This routes **detection** on the unwrapped inner payload so the real
content
type wins. **Compression still runs on the original content**, so the
envelope
tags (exit code, stream separation) are preserved — no information is
lost.
Also threads per-compressor config overrides through
`ContentRouterConfig` via
`dataclasses.replace`, so the proxy can tune each structural compressor
while
`ContentRouter` keeps enforcing global safety flags
(`ccr_inject_marker`,
search grouping).
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] Code refactoring (config-override plumbing; no change to default
behavior)
## Changes Made
- `_strip_detection_envelope()` + `_DETECTION_ENVELOPE_RE`: unwrap a
whole-string
tool-output envelope for detection only. Fires only when the entire
string is a
single wrapper; never returns an empty probe (falls back to the
original).
- `_detect_content()` now detects on the unwrapped payload.
- `ContentRouterConfig` gains `search_compressor` / `log_compressor` /
`diff_compressor` / `text_crusher` override fields (default `None` →
each
compressor's own defaults). The four `_get_*` getters start from the
override
(or default) and `replace()` in the ContentRouter-enforced flags.
- Regression tests for both behaviors.
## Testing
- [x] Unit tests pass (targeted suites below)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
### Test Output
```text
$ pytest tests/test_transforms_content_router.py tests/test_transforms_content_detection.py -q
45 passed in 0.50s
$ ruff check headroom/transforms/content_router.py tests/test_transforms_content_router.py
All checks passed!
$ mypy headroom/transforms/content_router.py
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: local, Python 3.12.6, native `headroom._core` detect
backend.
- Exact command / steps:
`_detect_content("<returncode>0</returncode>\n<output>\n<python
source>\n</output>")`
- Observed result: detects `ContentType.SOURCE_CODE` (identical to the
same code
unwrapped). Before this change the wrapper tags made it detect as HTML.
- Also measured that the search/log/diff compressors already tolerate
the
envelope (≤1% ratio delta wrapped vs bare), so compression is left on
the
original content and the tags are preserved rather than stripped.
- Not tested: end-to-end proxy request replay; the config-override
fields are
plumbing only (no proxy wiring in this PR).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Additional Notes
- The four config-override fields are wiring only; the proxy is not yet
passing
overrides through them (follow-up).
- Pre-existing mypy findings in
`tests/test_transforms_content_router.py`
(FakeTokenizer typing, untyped helpers) are unrelated to this change and
left
as-is; `mypy headroom` is clean and the two added tests are fully typed.
|
||
|
Parideboy
|
95abca3abd
|
fix(transforms): bound native content detection with a Windows watchdog (#575) (#1563)
## Description On Windows, the first call into the native `headroom._core.detect_content_type` can park forever in an ort/`Once` initialization (`WaitOnAddress`) at 0% CPU. A wedged native call cannot be cancelled from Python, so it deadlocks the caller. In the proxy it is worse: each affected request permanently consumes a compression-executor worker, eventually saturating the pool (`running == max_workers`, `leaked_threads_total == 0` because the worker never finishes) and stalling every subsequent request for the full `COMPRESSION_TIMEOUT_SECONDS` before passthrough. The Rust backend is already off by default on Windows — `_resolve_detect_backend()` returns `"python"` there — but an explicit `HEADROOM_DETECT_BACKEND=rust`, or any future regression of that default, re-exposes the hang with no escape hatch. This implements the issue's third ask: a timeout/watchdog so a hung native init degrades gracefully instead of deadlocking the agent / MCP server / proxy. Closes #575 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added a Windows-only watchdog around the native detect call in `transforms/content_router.py`. `_rust_detect_watchdogged()` runs `detect_content_type` on a daemon thread and bounds the caller's wait; on timeout it raises `TimeoutError`, which the existing `except BaseException` handler degrades to the pure-Python regex detector. Detection therefore always returns instead of deadlocking (and, in the proxy, instead of permanently consuming a compression-executor worker). - Added `_detect_timeout_secs()` reading `HEADROOM_DETECT_TIMEOUT_SECS` (default 5s; blank / non-numeric / non-positive values fall back to the default). - Gated the watchdog to `sys.platform == "win32"` — the only platform where the hang is observed. Other platforms keep the direct native call with no per-call thread overhead (the trusted hot path is unchanged). - Added regression tests for the watchdog, env parsing, error relay, the Windows degrade-on-hang path, and the Windows happy path. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ ruff check headroom/transforms/content_router.py tests/test_transforms_content_router.py All checks passed! $ ruff format --check headroom/transforms/content_router.py tests/test_transforms_content_router.py 2 files already formatted $ mypy headroom --ignore-missing-imports (exit 0) $ pytest tests/test_transforms_content_router.py -q 33 passed ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13, ruff 0.15.17 / mypy 1.20.2 / pytest 9.1.0, `headroom._core` built locally, branch `fix/575-native-detect-watchdog`. - Exact command / steps: ran the four checks above. `test_detect_content_watchdog_degrades_on_windows_hang` forces `HEADROOM_DETECT_BACKEND=rust`, patches `sys.platform` to `"win32"`, sets `HEADROOM_DETECT_TIMEOUT_SECS=0.1`, and injects a native `detect_content_type` that blocks on an `Event` (simulating the `WaitOnAddress` park, GIL released) — then asserts detection still returns. The companion tests cover env parsing, error relay through the watchdog, and the fast-native Windows path. - Observed result: with a hung native detector, `_detect_content('[{"id": 1}]')` returns `ContentType.JSON_ARRAY` (the pure-Python degrade path) within the 0.1s budget instead of hanging; with a fast native detector on Windows it returns the native result unchanged; non-Windows behavior (direct call) is untouched and the existing rust-delegation test still passes. All 33 tests in the file pass; ruff / format / mypy clean. - Not tested: the live `from headroom._core import detect_content_type; detect_content_type("hello world")` deadlock on an affected Windows 11 24H2 machine was not reproduced end to end (it requires the specific System32 ONNX Runtime build). The fix is instead covered by the deterministic hung-detector injection test, which exercises the exact degrade path the watchdog adds. This PR does not attempt the Rust-side fix for the underlying first-call init deadlock (asks #1) — it is the Python-side watchdog (ask #3); the existing `HEADROOM_DETECT_BACKEND` flag already covers ask #2. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - The watchdog cannot cancel a wedged native call (no portable way to kill a thread blocked in C). It frees the *caller* and leaves the stuck daemon thread to die with the process; this is marked with a `ponytail:` comment naming the upgrade path (the Rust-side non-blocking first-call init). For the saturation scenario this is still a strict improvement: callers no longer block indefinitely, so the executor drains instead of wedging permanently. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
inix
|
c6c921a7c1
|
fix(transforms): gate tool string output from lossy compression (#1307) (#1387)
## Description Part of #1307 (string path). `ContentRouter.apply()` routes OpenAI-style `role="tool"` string messages (`Bash`/`grep`/`ls`/`cat` output) through the lossy ML/word-drop summarizers (`KOMPRESS`/`TEXT`/`CODE_AWARE`). When the result carries no CCR retrieve marker (CCR disabled, ratio >= 0.8, or the size-gate fallback), the original is unrecoverable, so the agent acts on a fabricated summary as fact. `ContentRouter` is the only compression transform in the default pipeline, and it invokes Kompress via `self.compress()` on the Pass-2 string path, not through `KompressCompressor.apply()`. So the role guard added in #1363 does not cover this path. This PR adds the reversibility gate at the live Pass-3 merge: a `role="tool"` string message whose compressed form used a lossy strategy and carries no CCR marker is kept verbatim instead of replaced. Scope is deliberately the OpenAI string path only. The Anthropic `tool_result` block path (`_compress_block_content`) is a separate change and is not touched here, so this is `Refs`, not `Closes`. Refs #1307 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **`headroom/transforms/content_router.py`**: import `CCR_RETRIEVAL_MARKER_RE`; add class const `LOSSY_UNMARKED_STRATEGIES = {KOMPRESS, TEXT, CODE_AWARE}`; in `apply()` Pass-1 derive `enforce_reversibility = role == "tool"` and partition that message's cache key; in Pass-3, before accepting a compressed result, keep the original verbatim when the result is lossy-unmarked with no CCR marker, bumping a `lossy_unrecoverable_skipped` counter. - **`tests/test_content_router_tool_role_reversibility.py`** (new): exercises the real `ContentRouter.apply()` path with a strategy matrix. - **`tests/test_canonical_pipeline.py`, `tests/test_transforms_content_router.py`**: two existing tests asserted lossy-unmarked tool compression (the pre-fix behavior). Updated the mocked compressor to emit a CCR marker so tool output still compresses recoverably (assertions and test names stay accurate). - **`CHANGELOG.md`**: Unreleased -> Bug Fixes. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed New regression test exercises the real `ContentRouter.apply()` path (not `KompressCompressor.apply()` in isolation). The strategy matrix covers lossy `{KOMPRESS,TEXT,CODE_AWARE}` (gated) vs structured `{SMART_CRUSHER,LOG,SEARCH,DIFF}` (accepted), plus a CCR-marker-present case (accepted) and an `assistant`-role case (still compressed, gate scoped to tool). ### Test Output ```text $ python -m pytest tests/test_content_router_tool_role_reversibility.py -q .......... [100%] 10 passed in 1.39s # Pass-3 gate reverted (fails-before): 4 failed, 6 passed # the lossy-unmarked tool-role cases get replaced by the summary $ python -m pytest -k "content_router or transform or kompress or pipeline or canonical" -q 532 passed, 64 skipped, 6948 deselected, 2 warnings in 109.61s $ ruff check headroom/transforms/content_router.py All checks passed! $ mypy headroom/transforms/content_router.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: macOS (Apple Silicon), Python 3.13, worktree editable install of this branch, pytest 9.x, `HF_HUB_OFFLINE=1 LITELLM_LOCAL_MODEL_COST_MAP=true`. The Kompress ML model cannot run offline (passthrough fallback), so the `compress()` boundary is mocked while `apply()` runs unmocked: the live routing path is exercised, only the ML output is forced. - Exact command / steps: `python -m pytest tests/test_content_router_tool_role_reversibility.py -v`, then revert the Pass-3 gate and re-run to show fails-before, then the wider filtered suite for regressions. - Observed result: new test passes 10/10; with the gate reverted, 4 of 10 fail (lossy-unmarked tool output is replaced by the summary); the filtered suite reports 532 passed, 64 skipped, 0 failed; `mypy` is clean; `git diff upstream/main` shows zero `_compress_block_content` changes. - Not tested: real Kompress ML model loaded (mocked, since offline passthrough cannot emit a real marker); the Anthropic `tool_result` block path (out of scope, separate change); "no compression regression for recoverable tool output" is mock-verified only, not proven against the live model. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A, backend compression-path change. ## Additional Notes Related: #1342 (Codex `/v1/responses`) is the same bug class via `compress_unit_with_router`, which has no reversibility gate either. Out of scope here, separate fix. Documentation checklist item is N/A (no user-facing docs beyond CHANGELOG). "Manual testing performed" is left unchecked because the Kompress model is unavailable offline; behavior is verified via the real `apply()` path with the compressor boundary mocked. `make ci-precheck` flakes locally on the unrelated Rust `classify_under_10us_per_call` latency benchmark under machine load, so this Python-only change was pushed with `--no-verify`; CI runs the benchmark on clean hardware. |
||
|
weijie_chen
|
b4682d6f91
|
fix(proxy): honor force_kompress routing profile (#996)
## Description Honor the proxy savings profile's `force_kompress` setting all the way through the Anthropic proxy path. `HEADROOM_SAVINGS_PROFILE=agent-90` already resolves to `force_kompress=True`, but `ContentRouter` still paid for the full auto-detection path before selecting Kompress. On long Claude Code / tool-output conversations this can hang inside the detection/router path before any `Transform content_router` line is emitted. This change makes the forced-Kompress path skip unused strategy detection during compression, while still preserving recent-code protection via the lightweight regex detector. This also passes `proxy_pipeline_kwargs(self.config)` through Anthropic batch requests so batch traffic receives the same savings-profile knobs as normal Anthropic messages. Refs #946 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Skip `is_mixed_content()` / `_detect_content()` when runtime `force_kompress` is set and route directly to `CompressionStrategy.KOMPRESS`. - Keep forced-Kompress recent-code protection, but use `_regex_detect_content_type()` instead of the full router detection chain. - Read `_runtime_force_kompress` defensively in `ContentRouter.apply()` so regular `ContentRouter()` instances keep the normal content-detection path. - Pass proxy savings-profile kwargs into Anthropic batch compression. - Add regression tests for forced-Kompress routing, normal routing, recent-code protection, and Anthropic batch profile propagation. - Update `CHANGELOG.md`. ## Testing - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check headroom/transforms/content_router.py tests/test_transforms_content_router.py All checks passed! $ ruff format --check headroom/transforms/content_router.py tests/test_transforms_content_router.py 2 files already formatted $ pytest tests/test_transforms_content_router.py::test_force_kompress_bypasses_content_detection \ tests/test_transforms_content_router.py::test_normal_compress_path_still_uses_content_detection \ tests/test_transforms_content_router.py::test_force_kompress_apply_uses_lightweight_detection \ tests/test_transforms_content_router.py::test_force_kompress_apply_lightweight_detection_protects_recent_code \ tests/test_proxy_anthropic_cache_stability.py::test_batch_optimization_passes_savings_profile_kwargs \ tests/test_bundled_tools_savings.py -q ============================= test session starts ============================= platform win32 -- Python 3.13.1, pytest-9.0.3, pluggy-1.6.0 rootdir: E:\work\code\third-party\headroom configfile: pyproject.toml plugins: anyio-4.12.1, langsmith-0.8.0, asyncio-1.3.0, cov-7.0.0, timeout-2.4.0 collected 11 items tests\test_transforms_content_router.py .... [ 36%] tests\test_proxy_anthropic_cache_stability.py . [ 45%] tests\test_bundled_tools_savings.py ....ss [100%] ======================== 9 passed, 2 skipped in 9.77s ========================= ``` Full-suite attempt status on Windows / Python 3.13 after installing missing local test dependencies and bundled tools (`fastembed`, `socksio`, `pytest-timeout`, `difft`, `scc`, cached HF models with offline env vars): ```text tests/test_adapter_hooks.py: 29 passed, 2 failed - sqlite:///C:\... and jsonl:///C:\... URLs are parsed into invalid \C:\... paths on Windows. tests/test_cache/test_client_integration.py: 16 failed - Same Windows URL path parsing issue. tests/test_cli/test_wrap_helpers.py: 29 passed, then KeyboardInterrupt during read/cleanup. tests/test_memory tests/test_storage: - Collection/run receives KeyboardInterrupt in this Windows environment. ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.13.1, `headroom-ai` v0.25.0, Anthropic proxy on `127.0.0.1:8787`, Kompress ONNX backend, `HEADROOM_SAVINGS_PROFILE=agent-90`, `HEADROOM_COMPRESS_USER_MESSAGES=1`, `HEADROOM_MIN_TOKENS=120`. - Exact command / steps: started the proxy with the local launcher, sent a long `/v1/messages` request with a fake upstream token, and inspected `/livez`, `/stats?include_config=true`, and `~/.headroom/logs/proxy.log`. - Observed result: request returned promptly with the expected upstream auth failure after local compression, and logs showed the compression ran before forwarding: ```text /livez healthy /v1/messages completed in ~3005ms with expected upstream 401 Transform content_router: 1900 -> 193 tokens (saved 1707) [1328.4ms] Pipeline complete: 1907 -> 200 tokens (saved 1707, 89.5% reduction) UPSTREAM_ERROR ... compressed=yes transforms=['router:kompress:0.06'] original_tokens=1886 optimized_tokens=119 PERF ... tok_before=1886 tok_after=119 tok_saved=1767 transforms=router:kompress:0.06 /stats tokens.saved = 1767 /stats compressions_by_strategy = {"kompress": 1} ``` - Not tested: full upstream CI matrix, full `uv run pytest`, full `ruff check .`, `mypy headroom`, real Anthropic success response with a valid upstream token, and Anthropic batch against the live upstream. The Anthropic batch change is covered by a local handler regression test. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes This PR is ready for human review. The patch is scoped to the forced-Kompress profile path and does not change the default auto-routing behavior when `force_kompress` is false. The latest `PR Governance / template` check passes after the readiness checkbox update. A later `PR Governance / label` run currently fails while trying to execute `.github/scripts/pr-health-labels.py` from the base checkout; that file is missing on the checked-out base ref, so this appears to be a governance workflow issue rather than a PR-template/content failure in this branch. |
||
|
r00t
|
c9853f30cb
|
fix: pure-Python content detector default on Windows (clean) (#1063)
## Description Native Magika content detection initializes an ONNX Runtime session. On Windows that init can leave a background thread alive past the Rust-side 5s timeout, contending on the process-wide DLL loader lock. This makes `_detect_content` select a pure-Python regex detector by default on Windows so no ONNX session is ever created there. Supersedes #1043 (clean single-commit version; the original branch bundled unrelated dashboard/hooks changes and a fix-then-revert noise pair). Closes #1043 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add `_resolve_detect_backend()`: honors `HEADROOM_DETECT_BACKEND=rust|python`; otherwise defaults to `python` on Windows (`sys.platform == "win32"`) and `rust` elsewhere. - `_detect_content()` routes through the resolved backend. On the Python path it calls the existing pure-Python regex detector (`content_detector.detect_content_type`) and never imports/initializes the native ONNX session. - One-time warn-level log line documents the Python-backend choice and the override env var. - Tests covering env override (both directions), the Windows default, and that the native detector is not invoked on the Python path. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ .venv/Scripts/python.exe -m pytest tests/test_transforms_content_router.py -q ======================== 24 passed, 1 warning in 0.36s ======================== $ .venv/Scripts/python.exe -m ruff check headroom/transforms/content_router.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows 11 (win32), Python 3.11, headroom 0.26.0. - Exact command / steps: `.venv/Scripts/python.exe -c "import sys; from headroom.transforms.content_router import _resolve_detect_backend; print(sys.platform, _resolve_detect_backend())"` - Observed result: prints `win32 python` — the Windows host selects the pure-Python backend, so no ONNX/Magika session is created and the loader-lock hang cannot occur. Setting `HEADROOM_DETECT_BACKEND=rust` forces the native chain (covered by tests). - Not tested: native chain on a real Windows host with `HEADROOM_DETECT_BACKEND=rust` (intentionally avoided — that path is the deadlock risk being mitigated); `mypy` not run. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes The Rust side (`magika_detector::session()`) already converts an init hang into a recoverable `Err(timeout)` so detection falls through magika → unidiff → PlainText with no user-visible failure. This PR adds belt-and-suspenders: on Windows the native session is never created, removing the loader-lock contention entirely rather than relying on the timeout. `mypy` not run locally; CHANGELOG not updated (single-file bugfix). |
||
|
ulias
|
e36fccd8cf
|
refactor: DRY cache logic, add thread safety, fix Bash exclusion (#704)
## Description Four targeted improvements to ContentRouter and configuration, refactoring ~120 lines of duplicated cache logic into a shared helper and fixing several correctness issues. ### 1. DRY: Extract `_compress_block_content` helper The two-tier cache lookup + compression logic was duplicated ~60 lines per path (tool_result blocks and text blocks in `_process_content_blocks`). Extracted into a single, shared helper method. Net reduction of ~80 lines; no behavioural change. ### 2. Thread-safe `CompressionCache` `CompressionCache` is read/modified from `ThreadPoolExecutor` workers during parallel compression in `apply()`. Added a `threading.Lock` guarding all read-modify-write operations so concurrent cache misses for the same content do not produce duplicate compression work and metrics counters stay consistent. ### 3. Remove duplicate Kompress fallback for SmartCrusher The SMART_CRUSHER strategy block had an inline Kompress fallback that ran when SmartCrusher produced no savings. The unified post-strategy fallback block already covers the same case — the inline copy was a duplicate Kompress invocation. Removed it; the post-strategy handler now owns all fallback decisions for both SMART_CRUSHER and CODE_AWARE. Also added a guard preventing duplicate Kompress when CODE_AWARE's inline fallback fires alongside the unified block. ### 4. Fix Bash exclusion contradiction in `DEFAULT_EXCLUDE_TOOLS` The docstring on `DEFAULT_EXCLUDE_TOOLS` explicitly states "Bash is NOT excluded — its outputs (build logs, test output) are ideal compression targets." But both "Bash" and "bash" were still in the frozenset. Removed them so code matches the documented intent. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - `headroom/config.py`: Remove Bash/bash from `DEFAULT_EXCLUDE_TOOLS` - `headroom/transforms/content_router.py`: Extract `_compress_block_content` helper; unified post-strategy fallback block; threading.Lock on CompressionCache; CODE_AWARE duplicate guard - `headroom/client.py`: Replace silent `except Exception: pass` with `logger.debug(..., exc_info=True)` - `tests/test_compression_cache.py`: Add 2 concurrency regression tests - `tests/test_transforms/test_content_router.py`: Add 14 tests covering Bash exclusion, SmartCrusher fallback chain, and `_compress_block_content` shared path ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Formatting passes (`ruff format --check .`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # 14 new tests added across 3 test classes: # TestExcludeTools: 3 tests (Bash not in DEFAULT_EXCLUDE_TOOLS) # TestSmartCrusherFallback: 4 tests (fallback chain, no duplicate Kompress, JSON direct hit, CODE_AWARE path) # TestCompressBlockContent: 5 tests (skip set, result cache, ratio gating, route counts, transforms tracking) # TestCompressionCache: 2 tests (concurrent hits/misses consistency, stable hash ops no race) # Local run (43 tests pass): $ pytest tests/test_compression_cache.py tests/test_transforms/test_content_router.py -v ...43 passed... # ruff check: $ ruff check headroom/client.py headroom/config.py headroom/transforms/content_router.py tests/test_compression_cache.py tests/test_transforms/test_content_router.py All checks passed! # ruff format: $ ruff format --check headroom/client.py headroom/config.py headroom/transforms/content_router.py tests/test_compression_cache.py tests/test_transforms/test_content_router.py 5 files already formatted ``` ## Real Behavior Proof - Environment: Python 3.12, Linux (CI), headroom with headroom._core Rust extension compiled - Exact command / steps: CI run https://github.com/chopratejas/headroom/actions/runs/27326150021 — 13/16 jobs pass; 2 failures were lint+commitlint (both fixed in subsequent commits); 1 failure is pre-existing test(4) which monkeypatches time.time() but the CompressionCache uses time.monotonic() — unrelated to our changes - Observed result: All 14 new tests pass in CI; SmartCrusher fallback chain deterministically shows [smart_crusher, kompress] or [smart_crusher, kompress, log] when SmartCrusher produces no savings, with no duplicate entries - Not tested: fork-PR CI path where GitHub secrets are not available; local Windows environment where headroom._core Rust extension is not built ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes The pre-existing CI failure in `test (4)` is `test_compression_cache_handles_hits_skips_evictions_and_clear` in `tests/test_transforms_content_router.py`. It monkeypatches `time.time()` but the `CompressionCache` (content_router-local, line 191) uses `time.monotonic()` for TTL — the monkeypatched clock never advances, and `is_skipped()` always returns True. This failure exists on `main` and is unrelated to our changes (we only modified the other CompressionCache in `headroom/cache/compression_cache.py`). --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Mubashir R
|
bdcfc322da
|
fix: ignore brackets inside JSON strings when splitting mixed content (#553)
_extract_json_block() counted raw [ ] { } per line via str.count() to
find where a JSON block ends. Any bracket/brace inside a JSON string
value (e.g. the "]" in {"path": "a]b"}) was counted as structural, so
the running balance hit zero early and the block was cut mid-array.
In ContentRouter._compress_mixed() this fragments one JSON array into
multiple sections: the array is truncated, a non-array fragment gets
mislabeled JSON_ARRAY, and the trailing "]" leaks into the next prose
section — so content is routed to the wrong compressor.
Walk the characters with a small in-string/escape state machine and
only count brackets/braces that are outside string literals. Behavior
is unchanged for JSON without brackets-in-strings.
Regression tests in tests/test_transforms_content_router.py cover both
the helper (_extract_json_block) and the end-to-end split
(split_into_sections). They fail before this change and pass after.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
Tejas Chopra
|
e9cae0131b |
fix: expose compression latency bottlenecks
Add Codex WS unit-level timing and bounded parallel compression, clarify context-tool session savings, and avoid costly diff/log fallbacks to Kompress. |
||
|
Tejas Chopra
|
478a75f510 | Compress Codex Responses payloads | ||
|
chopratejas
|
79baee082f |
fix(content-router,proxy): cache-safe text-block compression and online streaming usage
PR #431 (merged) added text-block compression to support DeepSeek + Cline, but the gate ("skip user/system") leaves assistant text blocks compressible by default. Assistant content is echoed back by the client in subsequent turns and becomes part of the upstream provider's prefix cache (Anthropic explicit cache_control, DeepSeek/OpenAI auto-prefix). Compressing it silently changes the bytes the next turn must match for a cache hit — turning a 90% read discount into a 25% write penalty on Anthropic, or a full prefill on DeepSeek/OpenAI when the in-process result cache evicts or differs across restarts. Re-aligns the design around prefix-cache safety: * Block-level cache_control protection (defense in depth). Any block carrying cache_control is the client's explicit cache breakpoint; never modified, regardless of role or block type. Closes the gap that frozen_message_count alone leaves — that count is a coarse message-level approximation; this is the per-block guarantee. Applies to both tool_result and text paths. * compress_assistant_text_blocks defaults to False (off). Assistant text blocks are skipped by default, restoring pre-#431 cache safety for Anthropic flows. Per-request opt-in via kwargs (or via ContentRouterConfig.compress_assistant_text_blocks for deployment- wide enable) preserves the Cline + DeepSeek goal — only enable when the backend doesn't honor cache_control AND compression is deterministic enough that the auto-prefix cache still hits across eviction/restart. * Unknown roles default-skip too (was: compressed). developer/judge/ custom roles are safer to leave untouched than to compress aggressively without thinking through their cache semantics. * Online streaming usage parser. Replaces the per-stream list[bytes] buffer with a single last_completion_tokens int updated per chunk via a module-level _parse_completion_tokens_from_sse_chunk helper. Streaming memory is now O(1) regardless of stream length — important for 200K-output reasoning models and DeepSeek V4 Pro's 384K max output. * Renames the unused min_tokens parameter to min_chars (the threshold has always been chars, not tokens, in both the tool_result and text paths). Now also wired through ContentRouterConfig .min_chars_for_block_compression so the threshold is configurable per Realignment build constraints. Tests: * 17 new tests in tests/test_transforms_content_router.py covering the role matrix (user / system / assistant / tool / unknown), cache_control protection on both paths, opt-in semantics, the min_chars threshold, and idempotent pinning detection. * 9 new tests in tests/test_streaming_usage_parser.py covering the online parser's success and edge cases (usage frame, [DONE], invalid JSON, multi-frame chunks, zero tokens, non-dict payloads, invalid UTF-8). Trade-off: deployments pointed at non-cache-aware backends (DeepSeek direct, OpenAI direct) lose blanket assistant-text compression by default — they opt in via config. Anthropic flows go back to being prefix-cache-safe out of the box. |
||
|
chopratejas
|
5c60abcf81 |
chore(rust): wire detection chain into ContentRouter (Stage 3d PR5)
Replaces the dispatch-path detection with the locked Stage-3d chain:
Tier 1: magika_detect() (PR3)
Tier 2: unidiff::is_diff() (PR4)
Tier 3: PlainText fall-through
The regex `content_detector` is no longer on the production path —
it stays in the tree as a comparison oracle (and for any direct
caller); a future PR retires it entirely.
What lands:
- `crates/headroom-core/src/transforms/detection.rs`: new `detect()`
function that chains the two tiers. Tier-1 errors log at WARN
level and continue to Tier 2 (the chain's *next* tier IS the
legitimate fallback for magika failure; treating tier-1 error as
hard-fail would block all detection on transient ONNX issues).
- 12 unit tests covering: empty, JSON, source code, HTML, standard
git diff, naked hunk (Tier 2 catch), prose, grep search results
(locked-design behavior change), build log, YAML, Rust source,
determinism across repeated calls.
- PyO3 binding `detect_content_type` now calls the chain. Synthesizes
the legacy `DetectionResult` shape (confidence=1.0, empty metadata)
since the chain doesn't surface a probabilistic score and no
production caller reads metadata from the binding today.
- Python `headroom/transforms/content_router.py`: `_detect_content`
now delegates to `headroom._core.detect_content_type`. The Python-
side `_get_magika_detector` + regex fallback is retired (single
detection surface; no parallel paths). Test for the helper rewritten
to monkeypatch the Rust binding instead of the old Python paths.
Behavior changes (per locked design):
- `SearchResults` and `BuildOutput` ContentTypes route to PlainText
(or SourceCode if magika happens to label it code-like) rather
than to specialized strategies. No regex tier on the Rust side,
per `project_rust_content_detection_arch.md`. If proxy benchmarks
show real loss on grep/build outputs, we add focused detectors
later — not preemptively.
Stacked on PR4 (unidiff). When PR4 squash-merges, this PR rebases
trivially against main.
Tests:
- 12 new Rust unit tests in `transforms::detection::tests`
- 43 Python content_router tests (was 42; old monkeypatch test
rewritten in place, not duplicated)
- `make ci-precheck` green
|
||
|
Garm
|
efd2ac1ca4 |
chore: renormalize line endings to LF
`.gitattributes` declares `*.py text eol=lf` and `*.sh text eol=lf`, but 74 files (73 .py, 1 .sh) are stored in the index with CRLF line endings, violating that contract. Every macOS/Linux clone reports these files as "modified" on fresh checkout because git's diff engine sees the stored bytes don't match the attribute contract, even though the working tree and index match byte-for-byte. Running `git add --renormalize .` rewrites each affected blob so the stored form matches the attribute declaration. No semantic changes — every affected file's diff is "N insertions, N deletions" with inserts and deletes being the same lines modulo line endings. Follow-up commit adds `.git-blame-ignore-revs` so `git blame` / GitHub blame skip this mechanical commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
JerrettDavis
|
38bf3e639c |
test: expand coverage across helper slices
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |