mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
91 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Rod Boev
|
dbbef4bd41
|
fix(code-compressor): recover valid Python rewrites after local syntax rejection (#2202)
## Description A compile-invalid Python definition rewrite currently makes `CodeAwareCompressor` discard every otherwise valid rewrite in the file and return the original source at 0 percent reduction. The existing whole-file safety guard stays in place, while a Python-only recovery replay now preserves the rejected definition and keeps independent valid compression. The recovery reuses the current Python validation authority in `ast.parse()` plus `compile()`, runs only after the first assembled module already fails `_verify_syntax()`, and stays out of non-Python paths. Closes #1233 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added a Python-only recovery replay after the first assembled module fails syntax validation. - Preserved only the invalid function or class rewrite while allowing independent valid definitions to remain compressed. - Kept the existing whole-file syntax guard and original-source fallback as the terminal safety check. - Added focused invalid-node, valid-modern-syntax, and fail-safe coverage. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_python_invalid_node_falls_back_locally tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_python_recovery_does_not_block_valid_modern_syntax tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_python_recovery_still_returns_original_when_all_candidates_invalid tests/test_transforms/test_code_compressor.py::TestEdgeCases::test_syntax_errors_in_input tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_ast_preserves_structure_for_python -v`) - [x] Linting passes (`uv run ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text uv run pytest tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_python_invalid_node_falls_back_locally tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_python_recovery_does_not_block_valid_modern_syntax tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_python_recovery_still_returns_original_when_all_candidates_invalid tests/test_transforms/test_code_compressor.py::TestEdgeCases::test_syntax_errors_in_input tests/test_transforms/test_code_compressor.py::TestRealASTRuns::test_ast_preserves_structure_for_python -v 5 passed in 0.34s uv run ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py All checks passed! uv run ruff format headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py --check 2 files already formatted ``` ## Real Behavior Proof - Environment: Windows, synced `uv` environment with `dev` and `code` extras installed - Exact command / steps: run the focused invalid-node regression through public `compress(..., language="python")` - Observed result: `1 passed in 0.19s`; the invalid candidate stays original, the neighboring valid candidate remains compressed, and `headroom-PR-TARGET-1233-PROOF.md` records the base `ratio=1.0` whole-file rollback against the fixed head behavior. - Not tested: the stale future-import mismatch discussed in the old issue comment, already covered on current main ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - The stale future-import comment on #1233 is not the live slice here; current main already validates Python with `compile()` and already covers that ordering case. - This fix keeps the existing whole-file fail-safe and does not broaden into cross-language recovery or new syntax models. - `CHANGELOG.md` remains unchanged because Headroom generates release notes from conventional commits. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Parideboy
|
c46cd8f950
|
fix(core): load ONNX Runtime dynamically so headroom._core imports on non-AVX2 x86-64 (#1715)
## Description
`import headroom._core` dies with SIGILL (`Illegal instruction`) on
x86-64 CPUs without AVX2 (Pentium N4200, Celeron N4500, AMD FX 8350 —
all reported on the issue). The repo sets no `RUSTFLAGS`/`target-cpu`
anywhere, so first-party Rust code is baseline x86-64; the AVX2 code
comes from Microsoft's prebuilt ONNX Runtime, statically linked into the
extension by fastembed's `ort-download-binaries-rustls-tls` feature on
non-Windows targets. Because it is statically linked, its code is mapped
and initialized when the extension module loads — **before** the runtime
AVX2 guard from #1162 can run, which is why that fix helped Magika init
but not the import-time crash.
Fix, mirroring what Windows already does for its own reasons (DirectML
link libs): build with `ort-load-dynamic` on every platform, so ONNX
Runtime is only `dlopen`'d at first use, where the #1162 AVX2 guard
falls back to the non-ONNX detection tiers on unsupported CPUs. Since
both target blocks became identical, they are collapsed into one
platform-independent `fastembed` dependency.
To keep Magika/fastembed working out of the box on Linux/macOS, the
existing `ORT_DYLIB_PATH` auto-pin (`headroom/_ort.py`, previously
Windows-only) now resolves the pip `onnxruntime` package's shared
library on all platforms (`onnxruntime.dll` / `libonnxruntime.so*` /
`libonnxruntime*.dylib`). The pip `onnxruntime` CPU wheels use runtime
CPU dispatch, so they also work on pre-AVX2 machines — non-AVX2 users
get working ML detection instead of a crash. Without the `onnxruntime`
package, ML detection degrades gracefully to the non-ONNX tiers exactly
as it already does on Windows.
Fixes #1278
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `crates/headroom-core/Cargo.toml`: replaced the per-target `fastembed`
blocks (`ort-download-binaries-rustls-tls` on non-Windows,
`ort-load-dynamic` on Windows) with a single platform-independent
dependency on `ort-load-dynamic`, with a comment documenting both the
DirectML and the AVX2/#1278 rationale.
- `Cargo.lock`: regenerated — `ort-sys` drops its static-download
dependencies (`hmac-sha256`, `lzma-rust2`, `ureq`); no version bumps.
- `headroom/_ort.py`: `ORT_DYLIB_PATH` auto-pin extended from
Windows-only to all platforms via a small `_find_dylib` helper that
resolves the platform's shared-library name inside the pip `onnxruntime`
package.
- `tests/test_transforms/test_ort_dylib.py`: replaced the obsolete
`test_noop_on_non_windows` with Linux (versioned `.so`) and macOS
(`.dylib`) pin tests; module docstring updated.
- `docs/content/docs/configuration.mdx`: `ORT_DYLIB_PATH` row updated
from Windows-only wording to the cross-platform behavior.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ cargo fmt --all -- --check && cargo clippy --workspace -- -D warnings && cargo test -p headroom-core --lib
clean
test result: 844 passed; 0 failed; 1 ignored
$ python -m pytest tests/test_transforms/test_ort_dylib.py -q
8 passed
$ ruff check headroom/_ort.py tests/test_transforms/test_ort_dylib.py
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11 (AVX2-capable — the SIGILL itself is not
reproducible on this machine), Python 3.13, Rust 1.95.0, local checkout
branched from `upstream/main` (
|
||
|
Dima Solodukha
|
4e30dde2ac
|
fix(router): compact JSON evades compression via whitespace token counting (#1857)
## Description
`ContentRouter` counts section tokens with `len(content.split())`. On
compact machine-generated JSON — the default output of
`json.dumps(separators=(",", ":"))`, `JSON.stringify`, and boto3 — there
are no spaces, so a large payload counts as ~1 "token". Every section
compression ratio then computes as ~1.0 and the `min_ratio` acceptance
gate silently rejects the compressor's real output: the router logs
`router:noop` while SmartCrusher separately logs `was_modified=true`.
Compression effectively no-ops on the most common agent payload type
(tool results returning JSON), on every provider.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Add `_estimate_tokens(text)` — a size-proportional estimate
(`len(text) // 4`, floored at 1), monotone in content size for any
format.
- Replace the decision-relevant `len(...split())` counts in
`ContentRouter` (section original/compressed token counts feeding the
ratio gates, plus the debug estimates) with `_estimate_tokens(...)`.
- Add `tests/test_content_router_compact_json.py`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_content_router_compact_json.py -q
2 passed, 1 warning
$ ruff check headroom/transforms/content_router.py tests/test_content_router_compact_json.py
All checks passed!
$ mypy headroom/transforms/content_router.py --ignore-missing-imports
Success: no issues found in 1 source file
```
## Real Behavior Proof
- Environment: `ContentRouter` invoked directly on a 150-item
ECS-service JSON tool_result, Python 3.13, estimator tokenizer.
- Exact command / steps: run the same payload two ways — compact
(`json.dumps(..., separators=(",", ":"))`) and the identical data with
spaces (`separators=(", ", ": ")`) — through
`ContentRouter(ContentRouterConfig(skip_user_messages=False))`.
- Observed result: before this change, compact JSON saved 0.0%
(`router:noop`) while the identical data with spaces saved 43.3%
(`router:tool_result:smart_crusher`) — same data, same compressor, only
whitespace differed. After this change, compact JSON compresses
equivalently to the spaced form.
- Not tested: no behavior change expected for content that already
tokenizes with whitespace (prose, code); those counts move from
word-count to chars/4 but the ratio comparison is self-consistent (both
sides use the same estimator).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
Scope kept deliberately narrow: only the counts that feed
compression-acceptance decisions are changed. Non-decision `.split()`
uses elsewhere are left alone. Happy to add a CHANGELOG entry if you'd
like one.
---------
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
gglucass
|
fd9ddaa238
|
fix(proxy): cold-start fast pass — defer only Kompress, not the whole pipeline (#2073)
## Description Since #1850, the freeze path forwards a session's provider-cached prefix byte-identical — so a session is permanently locked to whatever form its cold start put in the provider cache. That fix is correct (it stopped token-mode cache busting measured at +41% cost), but it interacts badly with off-path background compression (#1171): when `HEADROOM_BACKGROUND_COMPRESSION=1` defers a cold-start-large request (frozen=0, ≥50k tokens), the ENTIRE pipeline is deferred and the raw transcript is forwarded, cached, and frozen. The background job's results can never be applied afterward (doing so would rewrite the frozen prefix), so the session forfeits its compression savings for its lifetime. Field data (same day, same session, A/B across a version boundary): ~15k tokens/turn saved when the cold start compressed synchronously vs 0/turn forever when it deferred. Notably, the recurring savings came from `read_lifecycle` stale-read drops completing in ~300ms — deferral throws away sub-second lossless wins to avoid a 30s Kompress pass. Only the Kompress ML stage can blow the request budget (the #1171 cascade). This PR splits the two: - The deferral branch now runs the pipeline synchronously with a new `skip_kompress=True` per-call kwarg — everything except the ML stage — under a bounded budget, and forwards the pruned form. The provider caches (and #1850 freezes) the *compressed* transcript, so the cheap savings persist for the session's lifetime. - The full pipeline (Kompress included) still goes to the background job, unchanged, keyed against the original messages so its content-hash results remain reusable at future cache-miss boundaries. - Fail-open: on fast-pass timeout or error, the request forwards uncompressed exactly as before this change. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/content_router.py`: new per-call `skip_kompress` runtime kwarg (follows the existing `_runtime_force_kompress` pattern). Gates only the Kompress deep-path call site; units routed there take the identical fallback used when the model isn't ready. Wins over `force_kompress`. - `headroom/proxy/helpers.py`: `COLD_START_FAST_PASS_TIMEOUT_SECONDS` (env `HEADROOM_COLD_START_FAST_PASS_TIMEOUT_SECONDS`, default 10s), documented next to `COMPRESSION_TIMEOUT_SECONDS`. - `headroom/proxy/handlers/anthropic.py`: the background-deferral branch runs the fast pass synchronously, stores its result in the session `CompressionCache`, forwards the pruned messages, and tags `deferred:kompress_background` (or `deferred:dropped` when the enqueue was dropped). On failure it constructs the same `_DeferredCompressionResult` as before. The Anthropic handler is the only deferral site (OpenAI/Gemini handlers don't defer). - `tests/test_transforms/test_content_router.py`: `skip_kompress` never invokes the ML stage and wins over `force_kompress` (mirrors the existing `force_kompress` test). - `tests/test_cold_start_fast_pass.py`: handler-level tests — exactly one synchronous `skip_kompress=True` pass, the background job runs the full pipeline, the forwarded body carries the fast-pass form, fast-pass results land in the compression cache; and the fail-open path (executor timeout → original messages forwarded, background job still queued). - `CHANGELOG.md`: Bug Fixes entry. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev pytest tests/test_cold_start_fast_pass.py -v tests/test_cold_start_fast_pass.py::test_cold_start_runs_fast_pass_and_defers_only_kompress PASSED tests/test_cold_start_fast_pass.py::test_fast_pass_failure_falls_back_to_full_deferral PASSED ============================== 2 passed in 0.28s =============================== $ uv run --frozen --extra dev pytest tests/test_proxy/test_background_compression.py \ tests/test_proxy/test_phase3_byte_identity.py tests/test_anthropic_stage_timings.py \ tests/test_transforms/test_content_router.py tests/test_anthropic_pre_upstream_backpressure.py ======================== 90 passed, 1 warning in 10.47s ======================== $ uv run --frozen --extra dev mypy headroom/proxy/handlers/anthropic.py headroom/proxy/helpers.py headroom/transforms/content_router.py Success: no issues found in 3 source files $ ruff check <changed files> && ruff format --check <changed files> All checks passed! / 5 files already formatted ``` ## Real Behavior Proof - Environment: macOS 15 (darwin 24.6.0), Python 3.10 via `uv run --frozen --extra dev`; field logs from a production desktop deployment (Python 3.12, `HEADROOM_MODE=token`, `HEADROOM_BACKGROUND_COMPRESSION=1`, subscription auth policy). - Exact command / steps: compared per-request PERF log lines for the same Claude Code session served by 0.30.0-lineage (sync cold start) vs 0.31.0-lineage (deferred cold start) on the same day. - Observed result: deferred-cold-start sessions log `tok_saved=0` on every subsequent turn with `Pipeline: freezing first 281/284 messages`; sync-cold-start sessions log `tok_saved=15526-18791` per turn with `read_lifecycle:stale` transforms at `opt_ms≈300`. - Not tested: this patch has not run against a live proxy yet (behavior verified at the handler-test level); `ruff`/`mypy` scoped to changed files. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — proxy pipeline change, no UI. ## Additional Notes - Companion to #2057 (nested tool_result image token counting) and #2058 (new-content-relative savings rate) — all three came out of the same investigation into near-zero reported savings on long 1M-context Claude Code sessions. - Deliberate scope cuts: the OpenAI/Gemini handlers don't have a deferral branch, so nothing to change there; the background job is left keyed to original messages (not the fast-pass output) so its cached results match client-resent bytes at future cache-miss boundaries. - Timeout leak caveat is documented in code: a fast-pass timeout briefly leaks an executor worker, but without the ML stage the pass is bounded by routing + statistical crushers (observed 5-8s worst case on multi-M-token counted transcripts). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Tejas Chopra <chopratejas@gmail.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Tejas Chopra
|
b6eb7a7613
|
feat(kompress): optional remote compression endpoint (HEADROOM_KOMPRESS_ENDPOINT) (#2171)
## Description Adds an **opt-in remote Kompress backend** so the proxy can offload Kompress ML inference to a hosted `/compress` endpoint instead of loading the ONNX model in-process. This lets Headroom run as a lean proxy in a sandbox installed with only `[proxy]` deps while the model runs elsewhere. The feature is purely additive: with `HEADROOM_KOMPRESS_ENDPOINT` unset, behavior remains the existing in-process Kompress path. ## Type of Change - [x] New feature (non-breaking change that adds functionality) ## Changes Made - `headroom/transforms/kompress_remote.py`: adds `RemoteKompressCompressor`, a `KompressCompressor`-compatible HTTP client that posts to `/compress`, sends optional bearer auth, skips network for tiny inputs, and fails open on HTTP/network/malformed-response errors. - `headroom/transforms/kompress_compressor.py`: extracts `store_kompress_in_ccr()` so the remote client reuses the same proxy-local CCR marker/storage policy without importing the ML model. - `headroom/transforms/content_router.py`: selects the remote compressor when `HEADROOM_KOMPRESS_ENDPOINT` is set, while `"disabled"` still wins and the unset path remains local Kompress. - `tests/test_transforms/test_kompress_remote.py`: covers mocked remote success, auth/header/request behavior, tiny-input no-call behavior, HTTP fail-open, malformed-success fail-open, and router env selection. ## Testing - [x] Unit tests pass (`uv run --extra dev pytest tests/test_transforms/test_kompress_remote.py -q`) - [x] Linting passes (`uvx ruff@0.15.17 check headroom/transforms/kompress_remote.py headroom/transforms/kompress_compressor.py headroom/transforms/content_router.py tests/test_transforms/test_kompress_remote.py`) - [x] Formatting passes (`uvx ruff@0.15.17 format --check headroom/transforms/kompress_remote.py headroom/transforms/kompress_compressor.py headroom/transforms/content_router.py tests/test_transforms/test_kompress_remote.py`) - [x] Type checking passes (`uv run --extra dev mypy headroom/transforms/kompress_remote.py headroom/transforms/kompress_compressor.py headroom/transforms/content_router.py`) - [x] New tests added for new functionality - [x] Manual testing performed by the author against a live endpoint ## Real Behavior Proof - Environment: Windows 11 review worktree, Python 3.13.3 for mocked tests; author also manually tested against a Modal deployment of `chopratejas/kompress-v2-base`. - Exact command / steps: ran the focused mocked endpoint test file plus lint/format/mypy on the changed modules. - Observed result: remote success maps endpoint response into `KompressResult`; short inputs do not call the network; 503 responses and malformed 200 responses return the original content; router selects the remote compressor only when the env var is set. - Not tested: full `pytest` suite; production concurrency/latency under load; endpoints other than the author's Modal reference deployment. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation — follow-up README flag section - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - The endpoint/deploy artifact (`modal_serve.py`) lives in the separate `kompress` repo; this PR is only the client-side flag. - The endpoint is intentionally stateless for CCR. Original-content storage and retrieval markers remain proxy-local. - Design note: this capability is intentionally in OSS as an opt-in flag. The same flag serves self-hosted endpoints and, later, a hosted endpoint. --------- Co-authored-by: JerrettDavis <mxjerrett@gmail.com> |
||
|
Ingmar Krusch
|
3d0e59e518
|
fix(content-router): protect_tool_results must not be weakened by profile-derived read_protection_window (#2105)
## Description `ContentRouter.apply()` computes `read_protection_window` from `protect_recent_reads_fraction`, where `0.0` (the sentinel `--protect-tool-results` sets, per #1374's documented contract) means "protect all excluded-tool output regardless of conversation depth." The method then unconditionally overwrote that window with a per-request `read_protection_window` kwarg whenever one was present. `proxy_pipeline_kwargs()` supplies that kwarg on every request from the active `AgentSavingsProfile.protect_recent` (the default `coding` profile sets `protect_recent=2`), so in practice only the last 2 messages ever kept read-protection regardless of `--protect-tool-results` — older excluded-tool output (`Read`, `Glob`, `Grep`, `Write`, `Edit` results) silently fell through to lossy Kompress compression. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/content_router.py`: the runtime `read_protection_window` kwarg may now only *narrow* the window when `self.config.protect_recent_reads_fraction > 0`. It can no longer override the `0.0` ("protect everything") sentinel that `--protect-tool-results` sets. - `tests/test_content_router_exclude_tools.py`: regression coverage that `--protect-tool-results`-equivalent config (`protect_recent_reads_fraction=0.0`) stays fully protected even when a savings-profile kwarg would otherwise shrink the window. - `tests/test_transforms/test_content_router.py`: unit coverage of the precedence logic itself (kwarg narrows when fraction > 0, kwarg is ignored when fraction == 0.0). - `CHANGELOG.md`: added an `### Bug Fixes` entry under `Unreleased`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_content_router_exclude_tools.py tests/test_transforms/test_content_router.py -q ============================= test session starts ============================== platform darwin -- Python 3.13.13, pytest-9.0.3, pluggy-1.6.0 collected 64 items tests/test_content_router_exclude_tools.py ...... [ 9%] tests/test_transforms/test_content_router.py ........................... [ 51%] ............................... [100%] ============================== 64 passed in 2.77s ============================== $ uv run ruff check headroom/transforms/content_router.py tests/test_content_router_exclude_tools.py tests/test_transforms/test_content_router.py All checks passed! $ uv run mypy headroom/transforms/content_router.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - **Environment:** personal fork deployed as a real proxy (macOS launchd service, `headroom install apply`) with `--backend bedrock --mode token --code-aware --protect-tool-results Bash`, `HEADROOM_SAVINGS_PROFILE=coding` (library default `protect_recent=2`), fronting a live Claude Code session. - **Exact command / steps:** in a long-running Claude Code session against this deployment, `Read` a source file, continue the conversation past 2 more assistant turns (so the file's `Read` result ages past the profile's `protect_recent=2` window), then have the agent re-read or reference the same file. - **Observed result:** before the fix, the aged `Read` output for a plain (non-code) file came back as `[N items compressed to M. Retrieve more: hash=...]` despite `--protect-tool-results` being set and `Read` sitting in `DEFAULT_EXCLUDE_TOOLS` — confirmed by direct proxy log inspection (`content_router.py`'s override silently winning over the `0.0` sentinel) and by byte-diffing the installed pipx package against this same fork's git source to rule out a stale build. After applying the fix, the same sequence leaves the aged `Read` output intact (no compression marker) — verified via `pytest` regression tests plus a fresh live-session check post-deploy. - **Not tested:** this deployment has since switched to `--mode cache` (upstream's tested/benchmarked default for the `coding` profile as of `68676daa`), where the whole `read_protection_window` mechanism this bug lives in is structurally unreachable for anything inside the frozen prefix — so the precedence fix in this PR is primarily relevant to `token`-mode deployments (or any deployment where cache mode's frozen-prefix boundary hasn't yet advanced past the affected message). It has not been independently re-verified live under `--mode token` after the most recent rebase onto `main` (only the automated test suite was rerun post-rebase). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — backend logic change, no UI surface. ## Additional Notes - "I have made corresponding changes to the documentation" is unchecked: no file in `docs/`, `README.md`, or `CONTRIBUTING.md` documents `read_protection_window`, `protect_recent_reads_fraction`, or `--protect-tool-results` precedence at all, so there was no existing section to update, and no new section was added either. This is arguably a pre-existing documentation gap this PR doesn't close. - No linked issue number: this was found via independent investigation of a personal deployment, not filed as a `headroomlabs-ai/headroom` issue first. Co-authored-by: Ingmar Krusch <ingmar.krusch@gmail.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Tejas Chopra <chopratejas@gmail.com> |
||
|
Yevhen Koval
|
ec55ddcfb3
|
feat(transforms): first-class C# support in CodeAwareCompressor (#1926)
Refs #1664 ## Description First-class C# support in `CodeAwareCompressor` via the tree-sitter `csharp` grammar, at parity with Java/C++/Rust: `using` directives, namespace headers, and type/member signatures preserved verbatim; method/constructor/destructor/operator/local-function bodies compressed; malformed input passes through unchanged. **No new dependencies** — the grammar ships inside the already-pinned `tree-sitter-language-pack==0.13.0` (resolves only as `"csharp"`; `c_sharp`/`cs` raise `LookupError`). Spec and maintainer go-ahead in the issue. Closes #1664 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `CodeLanguage.CSHARP` + full `_LANG_CONFIGS` entry; `_LANGUAGE_PREFILTER` and `content_detector` patterns chosen to be C#-distinctive (so Java doesn't mis-tag). - New data-driven `LangConfig` fields (pattern of #1334's `class_body_node_types`): `container_node_types` — block-scoped `namespace { }` routed through class compression so members compress without the wrapper being re-emitted verbatim; `opaque_node_types` — `#if`…`#endif` wrappers preserved verbatim without recursion (recursing + wrapper re-emit duplicated whole files, up to ~1.9x input on real repos); `#if` blocks wrapping only usings are emitted with the imports so they stay ahead of type declarations. - Shared-path fixes surfaced by real C# repos, each guarded and covered by a fail-before test: keep an Allman `{` on its own line in class reconstruction (K&R path byte-for-byte unchanged; Allman Java now compresses instead of falling back); line-based child extraction no longer swallows the following line for nodes ending at column 0 (C# `#region`/`#endregion` span their trailing newline — the over-slice duplicated the next member's signature or the closing brace); uncaptured top-level nodes preceding the first captured node (license banners, `#region License`) are emitted first instead of relocated below the code (tree-sitter-c-sharp rejects top-level `#region` after a type declaration, so relocation forfeited compression for the whole file). - `TestCSharpSupport` (8 tests) + a C# case in the parametrized member-container test; CHANGELOG entry. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_transforms/test_code_compressor.py -q 2 failed, 78 passed, 1 warning, 4 errors # the 2 failures / 4 errors reproduce # identically on main in the same env # (network-dependent tokenizer setup) Fail-before: with both changed sources reverted to main, the new C#-scoped selection reports "10 failed, 5 passed" (the 5 other languages keep passing); on the branch: "15 passed". $ ruff check headroom/transforms/code_compressor.py headroom/transforms/content_detector.py tests/test_transforms/test_code_compressor.py All checks passed! $ ruff format --check <same files> 3 files already formatted ``` ## Real Behavior Proof - Environment: Linux 6.8.0 aarch64, Python 3.10.12; `uv run --no-project --with "tree-sitter-language-pack==0.13.0" --with "tree-sitter>=0.25.2,<0.26" --with "pydantic>=2.0.0"`; real `CodeAwareCompressor` (`CodeCompressorConfig(enable_ccr=False)`, otherwise defaults), no mocks. - Exact command / steps: cloned two real .NET repos at depth 1 (`github.com/JamesNK/Newtonsoft.Json @ 4f73e74`, `github.com/App-vNext/Polly @ 7a1d10f`), ran `python proof_csharp.py <repo>` over every `.cs` file (chars/4 token estimate; tiktoken BPE download unavailable in my sandbox). Script in the collapsed section below. - Observed result: 16.1% tokens saved on Newtonsoft.Json (945/945 syntax-valid), 37.8% on Polly (797/797 syntax-valid), zero content duplication; full output: ```text repo: Newtonsoft.Json (945 .cs files) tokens before: 1,777,691 after: 1,490,629 saved: 287,062 (16.1%) files compressed: 479 pass-through: 466 inflated(>before): 19 syntax_valid: 945/945 latency ms P50: 0.7 P95: 18.7 P99: 44.1 max: 255.0 mean: 3.5 repo: Polly (797 .cs files) tokens before: 1,100,523 after: 684,303 saved: 416,220 (37.8%) files compressed: 693 pass-through: 104 inflated(>before): 15 syntax_valid: 797/797 latency ms P50: 0.8 P95: 11.6 P99: 28.9 max: 74.1 mean: 2.4 ``` After rebasing onto current `main` (which touched the same transform files via #1906/#1747/#1668) I re-ran the Polly proof on the rebased tree: 37.8% saved, 797/797 syntax-valid, P99 28.5ms — unchanged. Signatures/properties verbatim, bodies elided with call summaries, `using` order and preproc balance intact; residual "inflated" files are +2…+209 chars of assembly blank lines, not duplicated content. Newtonsoft is the adversarial case (multi-targeting: heavy `#if`, `#region`, Allman) — its conditional regions stay verbatim by design. Latency at parity with Java (<50ms P99; max is the pre-existing symbol-analysis cost on ~1800+-line files, shared with other languages). - Not tested: proxy end-to-end path with C# through `ContentRouter` (tested the `CodeAwareCompressor` API directly); CCR retrieval round-trips (`enable_ccr=False` in proof runs); exact tiktoken counts (chars/4 estimate — relative ratios are tokenizer-independent); Windows/macOS; full native `uv run pytest` with the Rust extension (ran the complete `test_code_compressor.py` in a lightweight venv; its 2 failures/4 errors reproduce identically on `main`); `mypy`. <details> <summary>proof_csharp.py (reproducible)</summary> ```python """Real behavior proof: run the real CodeAwareCompressor over a .NET repo.""" import pathlib import statistics import sys import time from headroom.transforms.code_compressor import ( CodeAwareCompressor, CodeCompressorConfig, ) try: import tiktoken ENC = tiktoken.get_encoding("cl100k_base") def toks(s: str) -> int: return len(ENC.encode(s, disallowed_special=())) except Exception: def toks(s: str) -> int: return len(s) // 4 target = pathlib.Path(sys.argv[1]) comp = CodeAwareCompressor(CodeCompressorConfig(enable_ccr=False)) tot_before = tot_after = 0 n_files = n_compressed = n_valid = n_passthrough = n_inflated = 0 times_ms: list[float] = [] for f in sorted(target.rglob("*.cs")): try: code = f.read_text(encoding="utf-8-sig", errors="replace") except OSError: continue t0 = time.perf_counter() r = comp.compress(code, language="csharp") times_ms.append((time.perf_counter() - t0) * 1000) n_files += 1 b, a = toks(code), toks(r.compressed) tot_before += b tot_after += a if r.compressed == code: n_passthrough += 1 else: n_compressed += 1 if r.syntax_valid: n_valid += 1 if a > b: n_inflated += 1 times_ms.sort() p = lambda q: times_ms[min(int(len(times_ms) * q), len(times_ms) - 1)] print(f"repo: {target.name} ({n_files} .cs files)") print(f" tokens before: {tot_before:,} after: {tot_after:,} saved: {tot_before - tot_after:,} ({(1 - tot_after / tot_before) * 100:.1f}%)") print(f" files compressed: {n_compressed} pass-through: {n_passthrough} inflated(>before): {n_inflated}") print(f" syntax_valid: {n_valid}/{n_files}") print(f" latency ms P50: {p(0.50):.1f} P95: {p(0.95):.1f} P99: {p(0.99):.1f} max: {times_ms[-1]:.1f} mean: {statistics.mean(times_ms):.1f}") ``` </details> ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A — terminal evidence above. ## Additional Notes - Dependency justification: none added, none bumped; the `csharp` grammar is inside the already-pinned `tree-sitter-language-pack==0.13.0` wheel; `uv.lock` untouched. - Architecture: malformed input passes through byte-identical; every risky construct prefers the false negative (verbatim) over corruption; invalid reassembly falls back to the original via the existing validation gate (observed live); no new imports at module load; P99 <50ms on both proof repos. - Known v1 limitations (deliberate false negatives, possible follow-ups): expression-bodied members and property accessor bodies stay verbatim; declarations inside `#if` regions stay verbatim. - Related pre-existing finding, out of scope: C/C++ exhibit the same `#if`-wrapper duplication on `main` (an `#if`-wrapped C++ class is emitted twice, ratio 1.62). Happy to file separately. - `mypy` unchecked above because I did not run it in my environment. |
||
|
Hafiz Ismail
|
adf8fed9bd
|
fix(code): stop TS export duplication + comment displacement (#1906)
## Description `CodeAwareCompressor` (AST-based code compression, `headroom/transforms/code_compressor.py`) had two bugs in its structure-reassembly path, found while investigating a reported Go brace-duplication issue (the Go bug itself — `statement_list` row-range swallowing a block's closing brace — was already fixed on `main` in #1668; this PR fixes what was *actually* still broken): 1. **TS/JS `export` keyword duplication.** `export function foo() {}` / `export class Foo {}` compressed to `export export function foo() {}` — invalid syntax, silently discarded by `_verify_syntax`'s fallback (the caller never sees an error, compression just quietly no-ops). Root cause: `_compress_function_ast` / `_compress_class_ast` slice a node's source by **line**, not by byte offset, deliberately — to preserve leading indentation for definitions nested inside classes. But when a node shares its *first* line with a preceding sibling (the `export` keyword is a sibling of the function inside tree-sitter's `export_statement` node, not part of the function node itself), that line-based slice pulled the sibling's text in too. The `export_statement` handler then re-prepended the same `export` text on top, producing the duplicate. 2. **Doc-comment displacement (all languages).** A `/** ... */` or `//` doc comment directly above a top-level function/class/type got detached from its declaration during AST extraction and re-emitted in one cluster at the very end of the compressed output, instead of staying attached to what it documents. Root cause: doc comments are top-level *siblings* of the declaration they document, not children of it — the extractor didn't attach them to anything, so they fell through to a "leftover top-level code" bucket that gets flushed as a single block after all functions. Also tightens `test_actual_go_compression`, which — per its own comment — was written to *tolerate* the Go bug (`compression_ratio may be 1.0 if compression produces invalid syntax`) rather than catch it. Since the underlying Go bug is already fixed on `main`, this now asserts real compression (`compression_ratio < 1.0`), matching its JS/Python siblings. Closes #1905 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made Two commits: the fix itself, then the tests that prove it — bisectable independently, both pass the full suite on their own. **Commit 1 — `fix(code):`** - `headroom/transforms/code_compressor.py`: add `_get_node_lines()` — line-based node slicing that still preserves indentation, but trims a preceding sibling's text from the first line when that prefix isn't pure whitespace (i.e. an `export` keyword sharing the line), so callers that re-add the sibling text themselves don't get a duplicate; used by `_compress_function_ast` and `_compress_class_ast`. - `headroom/transforms/code_compressor.py`: add `_get_leading_comment_text()` — walks a node's `prev_sibling` chain to collect contiguous doc-comment nodes immediately above it (no blank line in between) and returns them for the caller to prepend, also marking their byte ranges as captured so they aren't independently swept into the leftover top-level-code bucket; wired into every capture branch in `_extract_structure` (package, import, export statement, decorator, function, class, type). - `CHANGELOG.md`: added an entry under `### Fixed`. **Commit 2 — `test(code):`** - `tests/test_transforms/test_code_compressor.py`: `test_actual_go_compression` now asserts `compression_ratio < 1.0` instead of tolerating a 1.0 fallback. - `tests/test_code_aware_brace_comment_regressions.py` (new): 4 regression tests — TS `export` not duplicated + valid syntax, TS doc comments stay attached, Go doc comments stay attached, and a real-TS-compression parity test matching the existing JS/Python/Go "actual compression" tests. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py tests/test_code_aware_brace_comment_regressions.py All checks passed! $ ruff format --check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py tests/test_code_aware_brace_comment_regressions.py 3 files already formatted $ pytest tests/test_transforms/test_code_compressor.py tests/test_code_aware_regressions.py tests/test_code_aware_brace_comment_regressions.py -q 83 passed in 6.23s $ pytest -q # full suite 7912 passed, 5 failed, 442 skipped in 417.43s (0:06:57) # The 5 failures are pre-existing and unrelated: confirmed to fail identically # with this PR's changes stashed out (clean upstream/main checkout). # - test_wrap_marker_is_stale_when_pid_reused (PID-reuse detection, env-specific) # - test_read_cached_oauth_token_falls_back_to_gh_cli (leaks real local `gh` credentials) # - test_rtk_reader_returns_none_on_nonzero_exit / test_lean_ctx_reader_returns_none_on_failure_and_logs # (pass in isolation; fail only in full-suite order — pre-existing test-pollution, unrelated to code_compressor.py) # - test_parser_usable_in_thread_pool (test itself passes a str to parser.parse(), # which tree-sitter's binding has always required as bytes — a pre-existing test # bug unrelated to this change; separate fix in progress on another branch) $ mypy headroom Success: no issues found in 408 source files ``` ## Real Behavior Proof - Environment: macOS (Darwin 24.6.0), Python 3.14.5, headroom-ai dev checkout built via `uv sync --extra dev` + `maturin develop -m crates/headroom-py/Cargo.toml` (real `headroom._core` build, not mocked), `tree-sitter==0.25.2` / `tree-sitter-language-pack` per the pinned `[code]` extra. - Exact command / steps: ran `CodeAwareCompressor(CodeCompressorConfig(min_tokens_for_compression=10, enable_ccr=False)).compress(open("sdk/typescript/src/client.ts").read(), language="typescript")` identically against `git stash`-ed (pre-fix) and current (post-fix) trees; full snippet and additional samples below. - Observed result: `client.ts` (real 20KB SDK file in this repo) went from `compression_ratio=1.0` with a silent fallback (`export export class HeadroomClient` in the raw AST attempt, invalid syntax) to `compression_ratio=0.942`, `syntax_valid=True` — real compression, no duplication; full before/after table below. - Not tested: real-world repos beyond this repo's own SDK sample and the bundled benchmark fixture — broader corpus testing may follow as a comment on this PR. **Exact command, full snippet:** ```python from headroom.transforms.code_compressor import CodeAwareCompressor, CodeCompressorConfig compressor = CodeAwareCompressor(CodeCompressorConfig(min_tokens_for_compression=10, enable_ccr=False)) with open("sdk/typescript/src/client.ts") as f: code = f.read() result = compressor.compress(code, language="typescript") ``` **Observed result, before vs. after, real code:** | Sample | Before (main) | After (this fix) | |---|---|---| | `sdk/typescript/src/client.ts` (real 20KB SDK file, this repo) | `compression_ratio=1.0`, silent fallback — `export export class HeadroomClient` in the raw AST attempt, invalid syntax | `compression_ratio=0.942`, `syntax_valid=True` — real compression, no duplication | | TS fixture exercising both bugs (exported fn/class + doc comments) | `compression_ratio=1.0`, silent fallback | `compression_ratio=0.993`, `syntax_valid=True` | | `middleware/ratelimit.go` (bundled benchmark sample) | `compression_ratio=0.862`, `syntax_valid=True` — unaffected (Go bug already fixed on `main` by #1668) | `compression_ratio=0.862`, `syntax_valid=True` — unchanged, confirms no regression | | `generate_go_code(3)` (existing test fixture) | `compression_ratio=0.498` | `compression_ratio=0.498` — unchanged, confirms no regression | On code shaped to actually exercise elision (function bodies long enough to exceed `max_body_lines=5`), TypeScript compresses in line with other languages once the correctness bug stops blocking it entirely: | Language | Compression savings (synthetic fixture, ~10-line function bodies) | |---|---| | Python | 64.4% | | Go | 52.3% | | TypeScript | 49.0% | | JavaScript | 42.8% | (`client.ts`'s real-world 5.8% savings is lower than the synthetic TypeScript number above because most of its methods are ≤5 lines — under the elision threshold regardless of language — not because of a language-specific limitation.) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes The Go brace-duplication bug that motivated this investigation was already fixed on `main` (#1668, merged before this branch was based) — confirmed via the minimal repro and `ratelimit.go`, both compress cleanly with no duplicated braces. This PR fixes what was still actually broken: the TS/JS `export`-duplication bug and the doc-comment displacement bug (both present across languages), found empirically while verifying the original bug report against the current `main`. --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
Rod Boev
|
bb112dd176
|
feat(compression): add audit-safe mode with protected pattern matching (#1899)
## Description `SmartCrusher.crush_array_json` (`headroom/transforms/smart_crusher.py`) selects rows to keep using statistical signals such as variance, structural anomaly, and position. It has no concept of "this row is audit/compliance-relevant and must stay visible in the prompt." A rare row, such as a leakage flag, compliance marker, or non-standard failure line, can be sampled out like any routine row, or moved behind an opaque `<<ccr:HASH ...>>` retrieval marker the model has no reason to ask for. In audit, SRE, and quant-falsification workloads, rare rows are frequently the most important evidence, so silent disappearance is a real safety issue rather than only a lossy-compression tradeoff. This adds an opt-in `audit_safe` mode to `SmartCrusher`: - `SmartCrusherConfig(audit_safe=True, protected_patterns=[...], fail_closed_on_protected_loss=True)` - Rows are scanned for pattern matches, string or regex, against each row's canonical JSON text before compression runs. - After compression, any protected row missing from the output is spliced back in verbatim, whether it was dropped by the statistical selector or left only behind a CCR marker. - A verification pass re-counts protected-row survivors after splicing. If the count is still short, the crusher fails closed and returns the original, uncompressed content instead of shipping a result with fewer protected matches than the input had. Setting `fail_closed_on_protected_loss=False` ships the best-effort spliced result with a logged warning instead. Protection applies on both `crush_array_json`, the dict-shaped API used by direct callers and the CCR retrieval flow, and `_smart_crush_content`, the tuple-shaped API `apply()` actually calls for every compressed tool/tool_result message. It is live on the real tool-output compression path. Scope: this covers JSON-array-shaped content routed through `SmartCrusher`, the common case for tool outputs such as API results, log lines, and DB rows returned as JSON. Raw CSV/plain-text content compressed by other transforms, including Kompress and log/tabular compressors, is out of scope for this PR; `protected_patterns` only has row structure to match against when the content is or renders to a JSON array. Closes #1705 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/smart_crusher.py`: added `audit_safe`, `protected_patterns`, and `fail_closed_on_protected_loss` fields to `SmartCrusherConfig`; these stay Python-side and do not reach the Rust config because this is post-processing around existing Rust-backed compression. - Added `_compile_protected_patterns`, `_canon`, `_row_matches_protected`, `_scan_protected_rows`, and `_splice_missing_protected` as the shared scan/match/splice primitives. - Added `_apply_audit_safe_protection` for dict-shaped `crush_array_json` results and `_apply_audit_safe_protection_to_content` for tuple-shaped `_smart_crush_content` / `apply()` results. Both splice missing protected rows back in, then verify and fail closed or warn on residual loss. - Wired both `crush_array_json` and `_smart_crush_content` to scan for protected rows before compression and apply protection after. - `CHANGELOG.md`: added an `Unreleased / Features` entry. - Default `audit_safe=False`, so existing callers keep current behavior. A regression test compares a configured-but-disabled crusher's output byte-for-byte against an unconfigured one. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_transforms/test_smart_crusher_audit_safe.py`) - [x] Linting passes (`uv run ruff check .`) - [x] Type checking passes (`uv run mypy headroom/transforms/smart_crusher.py`) - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/ -k "(smart_crusher or crush or audit) and not test_optimizer_not_called_in_audit_mode" --no-header -q ... tests\test_transforms\test_smart_crusher_audit_safe.py ........... [ 65%] ... 169 passed, 10 skipped, 8176 deselected, 1 warning in 21.84s $ uv run ruff check . All checks passed! $ uv run mypy headroom/transforms/smart_crusher.py Success: no issues found in 1 source file ``` `-k` excludes `test_optimizer_not_called_in_audit_mode` (`tests/test_cache/test_client_integration.py`), a pre-existing, unrelated Windows temp-path failure in SQLite storage init that reproduces identically on a clean `origin/main` checkout with none of this PR's changes applied; it matched the `-k audit` filter by name coincidence only. ## Real Behavior Proof - Environment: Windows, Python 3.12 via uv-managed venv, `headroom._core` built locally via `maturin` / cargo 1.95.0, no LLM provider needed because this is pure transform-layer behavior. - Exact command / steps: Ran `uv run pytest tests/ -k "(smart_crusher or crush or audit) and not test_optimizer_not_called_in_audit_mode" --no-header -q`, `uv run ruff check .`, and `uv run mypy headroom/transforms/smart_crusher.py`; also exercised the audit-safe tests that build a 62-row JSON array with two `AUDIT_FLAG` rows, run it through `SmartCrusher(SmartCrusherConfig(audit_safe=True, protected_patterns=["AUDIT_FLAG"]), with_compaction=False)` via both `crush_array_json` and `Transform.apply()` over a synthetic tool message, parse the compressed output back to JSON, and drive the splice/verify/fail-closed helper paths with engineered row-drop and forced-mismatch scenarios. - Observed result: Protected rows are present in the compressed output in every tested scenario; the fail-closed branch returns the original content byte-for-byte with `strategy_info == "audit_safe:fail_closed"` when verification detects residual loss; `audit_safe=False` produces output byte-identical to a crusher with no audit-safe configuration. - Not tested: Raw CSV/plain-text tool output compressed via non-SmartCrusher transforms, including Kompress and log/tabular compressors, is out of scope. Top-level `headroom.compress()` / `CompressConfig` wiring for `audit_safe` and `protected_patterns` is a natural follow-up and is not included here. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes No user-facing docs were updated because I did not find an existing `SmartCrusherConfig` field reference doc to extend. The top-level `compress()` / `CompressConfig` wiring mentioned in "Not tested" is a reasonable immediate follow-up if this mechanism is the right shape. |
||
|
Vinay Gupta
|
140d6e4f96
|
fix(router): honor MCP aliases in excluded tools (#1822) (#1863)
## Description Normalize MCP tool-name aliases in the shared exclusion matcher so Anthropic/custom-agent names like `mcp_Server_tool` match the documented `mcp__*` glob and bare tool exclusions such as `headroom_retrieve`. Closes #1822 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added MCP alias matching for `mcp__server__tool`, `mcp_Server_tool`, and the bare wrapped tool name. - Added Anthropic `tool_use` / `tool_result` regressions for custom-agent MCP names and bare `headroom_retrieve` exclusions. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ .venv/bin/python -m pytest tests/test_transforms/test_content_router.py -q 57 passed, 1 warning in 0.95s $ .venv/bin/python -m ruff check . All checks passed! $ .venv/bin/python -m ruff format --check . 1058 files already formatted $ .venv/bin/python -m mypy headroom --ignore-missing-imports Success: no issues found in 407 source files ``` ## Real Behavior Proof - Environment: macOS, Python 3.13.5 local venv with editable headroom build. - Exact command / steps: Added #1822 regressions, ran the focused tests before the fix, then reran after adding MCP aliases. - Observed result: Before the fix, custom-agent MCP tool results were compressed instead of excluded; after the fix, the full content-router test file passes and excluded MCP results stay on the lossless excluded path. - Not tested: Full repository test suite locally; GitHub CI passed the full PR matrix. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review - [x] Principal engineer agent approved - [x] Senior developer agent approved ## Checklist - [x] My code follows the project style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes Review agents approved the scoped MCP exclusion-alias fix. One non-blocking review note: #1822 also mentions TOIN/prefix-cache symptoms, while this PR specifically fixes the custom-agent MCP exclusion name-resolution path. |
||
|
Zhenjia ZHOU
|
b38315cf72
|
fix(code-compressor): CJK-aware relevance-query symbol matching (#1747)
## Description
`CodeAwareCompressor` gives a code symbol a relevance "context boost"
when the query names it. The query tokenizer in
`_analyze_symbol_importance` used an ASCII-only delimiter class, so a
CJK query (no spaces, CJK punctuation) collapsed into one blob and never
matched an ASCII symbol name; the substring fallback was also gated
behind `len(name) > 3`, dropping short ASCII names glued to CJK.
This extracts the query tokenization + matching into two pure helpers,
adds CJK/full-width punctuation as delimiters, and relaxes the `len>3`
guard only for CJK queries. ASCII/English behavior is byte-identical.
`code_compressor` is pure-Python (no Rust twin, no parity fixtures).
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/transforms/code_compressor.py`: add `_query_context_tokens`
(CJK/full-width punctuation + ideographic space as delimiters) and
`_symbol_in_context` (substring `len>3` guard relaxed only for CJK
queries), used by `_analyze_symbol_importance`.
- `tests/test_transforms/test_code_compressor_cjk.py`: pure-function
tests (CJK isolation, short-name relaxation, English-unchanged, empty).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for new functionality
- [x] Manual testing performed (see Real Behavior Proof)
### Test Output
```text
$ .venv/bin/python -m pytest tests/test_transforms/test_code_compressor_cjk.py
6 passed
$ .venv/bin/python -m pytest tests/test_transforms/test_code_compressor.py
35 passed, 39 skipped # no regression (skips need the [code] tree-sitter extra)
$ ruff check / mypy headroom/transforms/code_compressor.py # clean
```
## Real Behavior Proof
- Environment: macOS (Darwin), Python in a uv venv, branch
`feat/code-compressor-cjk-relevance` off `main`.
- Exact command / steps: called the extracted helpers directly on CJK
and ASCII queries.
- Observed result: `_query_context_tokens("请重点保留（parse_config）的解析配置")`
isolates `parse_config` as its own token (before: the whole query was
one blob, so the exact-match boost never fired);
`_symbol_in_context("db", ...)` now matches a short ASCII name glued to
a CJK query (before: dropped by the `len>3` guard). English is unchanged
— for `"keep the database helper"`, `_symbol_in_context("db", ...)`
still returns `False` (no spurious short substring match). All 6 new
tests pass; the existing 35 `code_compressor` tests are unchanged.
- Not tested: end-to-end `compress()` (needs the `[code]` tree-sitter
extra); the fix is at the pure query-matching layer and is verified
there.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — N/A
(internal compressor behavior)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md — N/A: internal relevance-scoring
fix, no user-facing surface change
## Additional Notes
- Scope note: a pure-CJK query that names the function only by a Chinese
description (no ASCII token anywhere) still cannot match an ASCII symbol
name — cross-script query matching remains out of scope.
|
||
|
Rob Francis
|
32ce99e4b4
|
fix(build): enable Intel macOS pip installs via ort-load-dynamic (#1538)
## Description
Fix `pip install headroom-ai` on Intel Mac (`x86_64-apple-darwin`).
Source installs failed because `ort-sys 2.0.0-rc.12` (transitive via
`fastembed`) does not ship prebuilt ONNX Runtime binaries for that
target, causing maturin/cargo to exit during the wheel build.
This PR mirrors the existing Windows fix: build the Rust core with
`ort-load-dynamic`, pin `ORT_DYLIB_PATH` to the pip `onnxruntime` native
library at import time, and publish Intel macOS wheels from CI.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `crates/headroom-core/Cargo.toml`: use `ort-load-dynamic` for
`x86_64-apple-darwin` instead of `ort-download-binaries-rustls-tls`.
- `headroom/_ort.py`: extend the ORT dylib pin hook to Intel macOS
(`darwin` + `x86_64`), resolving `libonnxruntime*.dylib` from the pip
`onnxruntime` package.
- `.github/workflows/release.yml` and `.github/workflows/rust.yml`: add
`macos-15-intel` / `x86_64-apple-darwin` wheel matrix entries.
- `tests/test_release_workflows.py` and
`tests/test_transforms/test_ort_dylib.py`: update/add coverage for the
new target and dylib pin behavior.
- `README.md`: note that prebuilt wheels are published for Intel macOS.
## Testing
- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_transforms/test_ort_dylib.py \
tests/test_release_workflows.py::test_fastembed_uses_dynamic_ort_on_windows \
tests/test_release_workflows.py::test_build_wheels_matrix_includes_intel_macos_with_dynamic_ort -q
.......................... [100%]
10 passed in 0.18s
$ maturin build --release -o /tmp/headroom-dist
📦 Built wheel for abi3 Python ≥ 3.10 to /tmp/headroom-dist/headroom_ai-0.27.0-cp310-abi3-macosx_10_12_x86_64.whl
$ python3.11 -m venv /tmp/hr-venv && /tmp/hr-venv/bin/pip install .
Successfully built headroom-ai
Successfully installed headroom-ai-0.27.0
$ cd /tmp && /tmp/hr-venv/bin/python -c "import headroom; import headroom._core; print('ok')"
version 0.27.0
_core ok
```
## Real Behavior Proof
- Environment: macOS `x86_64-apple-darwin`, Python 3.11.5, Rust 1.95.0
- Exact command / steps: Reproduced the reported failure with `pip
install headroom-ai` (sdist build dies in `ort-sys` for
`x86_64-apple-darwin`); after this patch ran `maturin build --release`,
then `pip install .` in a clean venv, then `python -c "import
headroom._core"`.
- Observed result: Before fix, cargo/maturin exit 101 on missing ORT
prebuilts; after fix, wheel build succeeds and `headroom._core` imports
cleanly (`version 0.27.0`, `_core ok`).
- Not tested: `macos-15-intel` GitHub Actions wheel matrix row (will be
validated by CI after merge).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
- ML features (magika detection, fastembed embeddings) still require
`onnxruntime` at runtime on Intel Mac. Users should install
`headroom-ai[proxy]` or `pip install onnxruntime`; `_ort.py` auto-pins
`ORT_DYLIB_PATH` when that package is present.
- Apple Silicon (`aarch64-apple-darwin`) behavior is unchanged: it
continues to bundle ORT via `ort-download-binaries-rustls-tls`.
- Lint/mypy not re-run locally in this pass; targeted pytest +
maturin/pip install proof covers the changed surface.
---------
Co-authored-by: Bor <you@example.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
|
||
|
Rod Boev
|
838c5234a8
|
fix(transforms): normalize diff compressor context (#1801)
## Description Unified diff content could skip compression when the router reached the DIFF strategy with no question context. `DiffCompressor.compress()` defaulted omitted context to an empty string, but explicit `None` still crossed into the Rust boundary and raised before any compression result could be produced. The router also had a DEBUG-only crash path because it measured `len(context)` before DIFF dispatch. This normalizes `None` at the router entry and at the DIFF wrapper boundary so direct and routed diff compression both send a string context to Rust. Closes #1798. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Normalize `None` context to `""` before router debug logging and compression dispatch. - Normalize `None` context to `""` again before calling the Rust diff compressor. - Add regressions for explicit `None`, omitted context, non-empty context preservation, and DEBUG-enabled router DIFF dispatch. - Keep DIFF fallback behavior unchanged so patch-shaped content is not routed through a lossy fallback. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_transforms/test_diff_compressor.py tests/test_transforms/test_content_router.py -q`) - [x] Linting passes (`uv run ruff check headroom/transforms/diff_compressor.py headroom/transforms/content_router.py tests/test_transforms/test_diff_compressor.py tests/test_transforms/test_content_router.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text uv run pytest tests/test_transforms/test_diff_compressor.py tests/test_transforms/test_content_router.py -q 86 passed in 3.09s uv run pytest tests/test_transforms/test_content_router.py -q 55 passed in 2.84s uv run ruff check headroom/transforms/diff_compressor.py headroom/transforms/content_router.py tests/test_transforms/test_diff_compressor.py tests/test_transforms/test_content_router.py All checks passed! ``` ## Real Behavior Proof - Environment: Windows, Python through the project `uv` environment. - Exact command / steps: run the new DIFF context regressions against base and head. - Observed result: base fails explicit `None` at the fake Rust boundary with `AssertionError: Rust diff compressor received None context`; head passes explicit `None`, omitted context, non-empty context, and DEBUG-enabled router dispatch. - Not tested: native Rust internals beyond the Python wrapper boundary. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes No changelog entry is needed for this narrow wrapper and router bug fix. Type checking was not part of the focused local validation for this Python-only change. |
||
|
Tejas Chopra
|
f0670404ce
|
feat(content-router): lossless-excluded compaction (grep/log/json) + enable in coding/general personas (#1762)
Builds on the now-merged personas (#1732). Two pieces: ### 1. Lossless compaction for EXCLUDED tool output Excluded tools (Read/Grep/Glob/Write/Edit) stay out of *lossy* compression, but their output is compacted by detected shape: | shape | transform | guarantee | |---|---|---| | grep (SEARCH) | ripgrep --heading fold | **byte-lossless** (`search_unheading` recovers) | | log (BUILD_OUTPUT) | ANSI strip + run-collapse | **byte-lossless** modulo non-semantic ANSI | | json | whitespace-minify | **data-lossless** (`json.loads` equal), NOT byte-exact | Source code + glob path-lists → verbatim. grep gated on `_try_detect_search` (the general/Magika classifier calls grep-over-code SOURCE_CODE and would miss it). Off by default (`compact_excluded_lossless`). ### 2. Enable it in the coding/general personas `compact_excluded_lossless=True` on the coding + general profiles, threaded via `proxy_env` + `proxy_pipeline_kwargs` + a per-request `ContentRouter.apply` override. So `HEADROOM_SAVINGS_PROFILE=coding` auto-folds excluded grep/log/json. ## Why The coding persona was getting ~2.5% on OpenCode because its dominant traffic (Grep/Read) is excluded, and RTK (shell-only, lossy) never sees OpenCode's *native* tools. This recovers those savings losslessly. ## Measured (end-to-end via coding-persona kwargs, real `rg` output) 41,589 → 26,562 chars (**−36%**), `router:excluded:lossless_search`, byte-recoverable. ## Accuracy grep/log = byte-lossless → edit-safe. json = data-lossless (edit-caveat for read-then-edit-JSON, documented). Read of source code → untouched (tested). 47 tests (personas + all three tiers + persona-enablement + end-to-end). ruff + mypy clean. **No personas duplication** — rebased onto main after #1732 landed. Supersedes #1755. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Tejas Chopra
|
eea667a720
|
feat(transforms): adaptive Otsu KEEP/DROP threshold (+ land relevance split on main) (#1726)
## Description Lands the prompt-conditioned relevance split **on `main`** and makes its KEEP/DROP threshold **adaptive**. Context: the Stage B work (#1722) was merged into the feature branch `tejas/proxy-lossless-mode` rather than `main`, so `relevance_split.py` never reached `main`. This PR cherry-picks that work onto `main` and adds the adaptive threshold on top, in three commits: 1. Prompt-conditioned KEEP/DROP tail split (Stage B) — segment LOG/SEARCH output into records, score each against the request's information need (user prompt + triggering tool-call args) via `headroom/relevance/`, keep relevant records verbatim, Kompress the low-relevance tail. Mode-agnostic (marker-free in lossless, retrieval-marker in CCR). 2. On by default with hot-path rails — background embedding-model pre-warm (BM25 until warm, never blocks a request) + optional `relevance_max_records` cap (default 0 = no cap). 3. **Adaptive Otsu threshold** (this PR's new work) — see below. ### Adaptive threshold The keep/drop cut is no longer a fixed constant. For each output we compute the natural relevant/irrelevant break in *its own* score distribution via **Otsu's method** (parameter-free — candidate cuts are the data's own values, no bins or magic numbers), floored by `relevance.relevance_threshold` so absolutely irrelevant records are never kept verbatim. The bar therefore moves with the content + prompt: a highly-relevant output keeps its top cluster and compresses the merely-moderate tail; a mostly-irrelevant output drops almost everything. All-equal scores fall back to the floor. Toggle via `relevance_adaptive_threshold` (default `True`). Closes # ## Type of Change - [x] New feature (non-breaking change that adds functionality) ## Changes Made - `relevance_split.py`: `adaptive_threshold()` + `_otsu_threshold()`; `plan_relevance_split(..., adaptive=True)` uses the adaptive cut, floored by `threshold`. - `content_router.py`: `relevance_adaptive_threshold` config (default `True`), threaded into the split. (Plus the Stage B split + default-on rails from the cherry-picked commits.) - `tests/test_relevance_split.py`: adaptive-threshold cases (bimodal split, floored, all-equal, moves-with-distribution) on top of the Stage B suite. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality ### Test Output ```text $ pytest tests/test_relevance_split.py tests/test_transforms_content_router.py tests/test_lossless_mode.py -q 80 passed, 1 warning in 3.41s $ ruff check headroom/transforms/relevance_split.py headroom/transforms/content_router.py tests/test_relevance_split.py All checks passed! $ ruff format --check <changed files> 3 files already formatted $ mypy headroom/transforms/relevance_split.py headroom/transforms/content_router.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - **Environment:** local, Python 3.12.6. - **Steps:** `adaptive_threshold()` exercised directly on synthetic score distributions; `plan_relevance_split(adaptive=True)` and the real `ContentRouter._apply_strategy_to_content` path driven with a deterministic scorer + Kompress-tail stub (offline). - **Observed:** - Bimodal scores `[0.92, 0.88, 0.12, 0.05]` → cut lands in the valley (`0.12 < t < 0.88`), keeping the high cluster. - Mostly-irrelevant `[0.30, 0.28, 0.05, 0.03]` → cut floored at `0.25`. - All-equal scores → floor. - Higher-scoring distribution yields a higher cut than a lower one (bar adapts). - Router split still fires in both lossless and CCR mode; DIFF stays pure lossless; disabling the flag is byte-identical. - **Not tested:** live embedding model warm/latency at scale; end-to-end `/v1/retrieve` resolution of the CCR tail marker (marker plumbing itself is covered upstream). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes - Supersedes the orphaned #1722 merge (which landed on the feature branch, not `main`); this PR is the canonical path onto `main`. - **Follow-ups discussed:** TEXT-strategy extension (relevance split for plain prose, currently whole-block Kompress); batch multiple DROP runs into one Kompress call; eval of savings/fidelity on live traffic. - N/A: CHANGELOG (feature not yet released). |
||
|
Kiryu Tsukimiya
|
9157173018
|
fix(read-lifecycle): persist STALE Read originals in the CCR store (#1488)
## Description
`read_lifecycle` emits STALE/SUPERSEDED Read markers containing
`Retrieve original: hash=...`, but `headroom_retrieve(hash)` 404s on
every such marker — the original content is never actually stored.
Affects the default config (`read_lifecycle=on`, `compress_stale=on`)
and the common Claude Code flow: read a file, edit it, then want the
prior content back.
**Root cause:** `ContentRouter.transform` instantiated
`ReadLifecycleManager` with
`compression_store=kwargs.get("compression_store")`, but no caller ever
sets that kwarg. `self.store` was always `None`, so `read_lifecycle.py`
emitted the marker with a SHA-256 hash but skipped the
`store.store(...)` call. Every other compressor (SmartCrusher, Kompress,
search/log/diff/code) resolves its store directly via
`get_compression_store()`.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/transforms/content_router.py`: inject a CCR store into
`ReadLifecycleManager` via an explicit `is None` check + guarded
`get_compression_store()` import (matches `smart_crusher.py`'s pattern).
Falls back to marker-only when the module is absent in stripped builds.
- `headroom/transforms/read_lifecycle.py`: wrap `store.store(...)` in
`try/except` with a precomputed fallback hash so a transient backend
failure can't break `compress()` (mirrors `read_maturation.py`). Pass
`explicit_hash=ccr_hash` to avoid double SHA-256 and keep marker/store
key in lockstep.
- `tests/test_transforms/test_read_lifecycle.py`: regression test
(`TestContentRouterIntegration`) that drives `headroom.compress()` and
asserts the STALE marker's hash resolves in the global CCR store.
## Testing
- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`) — not run locally
- [ ] Type checking passes (`mypy headroom`) — not run locally
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ HEADROOM_CCR_BACKEND=memory .venv/bin/python -m pytest tests/test_transforms/test_read_lifecycle.py -v
============================== 23 passed in 0.43s ==============================
```
## Real Behavior Proof
- Environment: Python 3.13, headroom-ai dev install (`uv sync --extra
dev`), `HEADROOM_CCR_BACKEND=memory`, Linux x86_64.
- Exact command / steps: Run `headroom.compress()` on a synthetic STALE
conversation (Read then Edit of the same file):
```python
from headroom import compress
result = compress([
{"role": "assistant", "content": [{"type": "tool_use", "id": "t1",
"name": "Read",
"input": {"file_path": "/tmp/foo.txt"}}]},
{"role": "user", "content": [{"type": "tool_result", "tool_use_id":
"t1",
"content": "source line\n" * 500}]},
{"role": "assistant", "content": [{"type": "tool_use", "id": "t2",
"name": "Edit",
"input": {"file_path": "/tmp/foo.txt"}}]},
{"role": "user", "content": [{"type": "tool_result", "tool_use_id":
"t2",
"content": "edited"}]},
], model="claude-sonnet-4-5-20250929")
```
then `get_compression_store().retrieve(<hash-from-marker>)`.
- Observed result: post-fix `retrieve(hash)` returns HIT (`tool=Read`,
`strategy=read_lifecycle:stale`); pre-fix it returned MISS (the bug).
Full log:
```text
transforms_applied: ['read_lifecycle:stale:/tmp/foo.txt',
'router:excluded:tool', 'router:excluded:tool']
hashes from markers: ['3fbd603ecf1bcf50a86650d2']
store backend: InMemoryBackend
retrieve(3fbd603ecf1bcf50a86650d2) -> HIT tool=Read
strategy=read_lifecycle:stale
```
- Not tested: SQLite backend persistence across processes; Rust `_core`
extension code path; OpenAI / Gemini providers; Claude Code live (proxy
+ MCP server end-to-end).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation — N/A
(internal fix, no public API change)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable — leaving to
maintainers' convention
## Additional Notes
- No existing issue. #389 describes the same symptom class with a
different root cause (SmartCrusher row-drop CCR bridge); it explicitly
lists `read_lifecycle.py` as a producer that populates the store — this
PR makes that claim true.
- Commits: `
|
||
|
Tejas Chopra
|
5771a8020e
|
fix(deps): remediate dependency CVEs and publish SBOM (#1509)
## Description
Supply-chain hardening: takes the **shipped** dependency surface from
**26 known CVEs to 0**. `pip install headroom-ai[all]` now resolves with
no known vulnerabilities (verified with Anchore syft + grype). Also
publishes a checked-in SBOM package (`sbom/`) so any user — especially
pilots running their own security review — can verify what's inside and
that we track it.
This addresses the Dependabot alerts on `main` (9 high / 4 moderate / 7
low at time of writing).
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
**Rust**
- `pyo3` 0.24 → 0.29 (GHSA-36hh-v3qg-5jq4 High, GHSA-chgr-c6px-7xpp
Med). Migrated `Python::allow_threads` → `Python::detach` (10 sites) and
added `from_py_object` to the `Clone`-deriving `#[pyclass]` types (both
required by the 0.25+ API).
- `pyo3-log` 0.12 → 0.13; `lru` 0.12 → 0.18 (GHSA-rhfx-m35p-ff5j).
**Python**
- `torch` → 2.12.1, `mem0ai` → 2.x.
- Floor-pinned transitive CVE deps via `[tool.uv]
constraint-dependencies`: `pygments>=2.20.0`,
`pydantic-settings>=2.14.2`, `gitpython>=3.1.50`, `langsmith>=0.9.0`.
- **Removed `benchmark` from the `[all]` aggregate** so the default
install is CVE-free. `lm-eval` is invoked as an external subprocess
(`python -m lm_eval`) and never imported, so it is not a true runtime
dep — it remains available via the opt-in `[benchmark]` extra. See
[Accepted Risks](#additional-notes).
**npm (build/test tooling — never shipped in the
wheel/container/published SDK)**
- `esbuild` override `>=0.28.1` in `sdk/typescript` + `plugins/openclaw`
(GHSA-g7r4-m6w7-qqqr).
- `docs/`: `@anthropic-ai/sdk` → `^0.106.0` (GHSA-p7fg-763f-g4gf),
`postcss` override to force Next.js's bundled copy ≥8.5.10
(GHSA-qx2v-qp2m-jg93); regenerated a stale `bun.lock` that carried a
**Critical** vitest/vite.
**CI**
- Pinned `pypa/gh-action-pypi-publish` `@release/v1` → `@v1.13.0`
(GHSA-vxmw-7h4f-hqxh) in `release.yml` + `publish.yml`.
**SBOM**
- New `sbom/` directory: CycloneDX 1.7 + SPDX 2.3 SBOMs, grype scan
evidence, 330-package license inventory, and a regeneration guide.
## Testing
- [ ] Unit tests pass (`pytest`) — N/A, no Python source changed
(deps/config only)
- [x] Linting passes — `cargo fmt --check` + `cargo clippy` clean on the
changed crate; 0 `.py` files changed so `ruff`/`mypy` scope is
unaffected
- [x] Type checking passes — `cargo check --workspace` (0 errors)
- [ ] New tests added — N/A (dependency bumps; covered by existing
suites)
- [x] Manual testing performed — see Real Behavior Proof
### Test Output
```text
# headroom-ai[all] product surface — the number that matters
$ grype sbom:sbom/headroom-sbom-all-extra.cdx.json
No vulnerabilities found
# full repo scan (universal lock incl. opt-in [benchmark] + dev)
$ grype sbom:sbom/headroom-sbom.cdx.json
NAME INSTALLED TYPE VULNERABILITY SEVERITY
sqlitedict 2.1.0 python GHSA-g4r7-86gm-pgqc High # [benchmark]-only, unpatchable, accepted
nltk 3.9.4 python GHSA-p4gq-832x-fm9v High # [benchmark]-only, unpatchable, accepted
# pyo3 0.29 migration — extension builds + imports + runs
$ cargo check --workspace
Finished `dev` profile [unoptimized + debuginfo] target(s)
$ maturin develop && python -c "from headroom._core import DiffCompressor, SmartCrusher; ..."
extension OK — detach + from_py_object paths exercised
# lru 0.18 — eviction path
$ cargo test -p headroom-proxy --lib drift
14 passed, 213 filtered out
# per-ecosystem npm audits
$ (cd sdk/typescript && npm audit) -> found 0 vulnerabilities
$ (cd plugins/openclaw && npm audit) -> found 0 vulnerabilities
$ (cd docs && npm audit && bun audit) -> found 0 vulnerabilities / No vulnerabilities found
```
## Real Behavior Proof
- Environment: macOS (darwin 25.4.0, arm64), Python 3.12 `.venv`, Rust
1.95 toolchain, syft 1.46.0, grype 0.115.0, bun 1.3.14, maturin 1.13.3.
- Exact command / steps: (1) `uv export --extra all --no-dev
--no-emit-project | syft → grype` for the product surface; (2) `cargo
check --workspace` + `maturin develop` + extension import/compress smoke
test; (3) `cargo test -p headroom-proxy --lib drift`; (4) `cargo fmt
--check` + `cargo clippy -p headroom-py`; (5) `npm audit` in
sdk/openclaw/docs + `bun audit` in docs.
- Observed result: `headroom-ai[all]` resolution scans clean — "No
vulnerabilities found" (179 pkgs); full/prod SBOM shows only the 2
documented accepted CVEs; pyo3 0.29 extension imports and runs (detach +
from_py_object paths exercised); drift tests 14/14 pass; cargo fmt +
clippy clean; all npm/bun audits report 0.
- Not tested: full `pytest` suite (no Python source changed);
release-profile wheel build (used dev-profile `maturin develop` for the
import proof — the extension is semantically identical).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
(`sbom/README.md`)
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective — N/A
(dependency bumps; existing suites + scans cover it)
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md — N/A (Release Please
auto-generates from the conventional commit)
## Additional Notes
**Accepted risks (the 2 residual CVEs).** Both originate solely from the
EleutherAI `lm-evaluation-harness` under the **opt-in `[benchmark]`
extra**, which Headroom invokes as a subprocess (never imports):
- `sqlitedict` CVE-2024-35515 (High) — pickle deserialization; package
abandoned (last release 2021), **no upstream fix exists**.
- `nltk` CVE-2026-54293 (High) — path traversal in `nltk.data.load()`;
affects ≤3.9.4 (current latest), **no patched release**.
Neither is in `[all]`, the published wheel, or the container. They are
documented in `sbom/README.md` and will be picked up automatically once
upstream ships fixes.
**Release/CHANGELOG:** N/A items above are because this is a
dependency/security PR with no Python source changes; CHANGELOG is
Release-Please-managed via the conventional commit message.
|
||
|
Lucas Santos
|
43494ff526
|
fix(proxy): stop re-compressing headroom_retrieve output and emitting unredeemable markers (#1323)
## Description Two related CCR problems that both end in unreadable content. The first one (#1077) is an infinite loop. Any tool output over ~500 bytes gets replaced with a `<<ccr:hash>>` marker, and you call `headroom_retrieve` to get the original back. But the proxy then compresses the *retrieve response too*, so what comes back is a brand new marker. Retrieve that one and you get another marker. The second one (#1006), the proxy makes two independent decisions per request: SmartCrusher compresses, and the `headroom_retrieve` tool gets injected. The injection is deferred when there's a frozen message prefix (`frozen_message_count > 0`), but compression keeps running anyway. So the agent receives `[... compressed to N. Retrieve more: hash=...]` markers with no `headroom_retrieve` tool to redeem them. For #1077, SmartCrusher now skips `headroom_retrieve` results. Before crushing a tool message (OpenAI `role=tool`) or tool-result block (Anthropic `type=tool_result`), it checks whether that tool id maps to the CCR tool, and if so leaves it alone. Retrieved content stays readable. For #1006, compression and injection are no longer decided in isolation. The injection decision is extracted into `should_inject_ccr_tool`, which the Anthropic handler calls: when injection was deferred because of a frozen prefix but compression just emitted new markers, it injects the tool anyway, so a marker is never handed to an agent that can't act on it. The existing session-sticky dedup means sessions that already have the tool don't get it re-injected and don't lose their cache. Closes #1077 Closes #1006 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/smart_crusher.py`: exempt `headroom_retrieve` results from compression on both the OpenAI `role=tool` and Anthropic `type=tool_result` paths. - `headroom/proxy/helpers.py`: add `should_inject_ccr_tool`, the deferral-plus-override decision the handler used to inline, so the #1006 behaviour is testable at the decision point. - `headroom/proxy/handlers/anthropic.py`: call `should_inject_ccr_tool` to couple injection with compression; rename the misleading `frozen_prefix=` log key to `frozen_message_count=`. - `tests/test_transforms/test_smart_crusher_ccr_retrieve_exemption.py` and `tests/test_proxy/test_ccr_frozen_prefix_coupling.py`: new tests; the frozen-prefix test now drives `should_inject_ccr_tool` so it would fail if the override were removed. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ uv run --extra dev python -m pytest tests/test_proxy/test_ccr_frozen_prefix_coupling.py tests/test_transforms/test_smart_crusher_ccr_retrieve_exemption.py -q 5 passed, 1 skipped ruff: All checks passed! mypy: Success: no issues found ``` The SmartCrusher test skips locally because the Rust extension `.so` is built for a different OS, the same skip the existing SmartCrusher tests take locally. It runs in CI where the extension is built. ## Real Behavior Proof - Environment: macOS, Python 3.13, this branch. - Exact command / steps: `uv run --extra dev python -m pytest tests/test_proxy/test_ccr_frozen_prefix_coupling.py tests/test_transforms/test_smart_crusher_ccr_retrieve_exemption.py -q`. The frozen-prefix test calls `should_inject_ccr_tool` (the function the Anthropic handler now uses) with a frozen prefix and freshly emitted markers, then drives `apply_session_sticky_ccr_tool` end to end and asserts `headroom_retrieve` lands in the outbound tools. The exemption test runs a `headroom_retrieve` tool result through SmartCrusher on both the OpenAI and Anthropic shapes. - Observed result: 5 passed, 1 skipped. The retrieve tool is injected even under a frozen prefix once markers exist, and is not injected when no markers were emitted. Removing the handler override flips `should_inject_ccr_tool` and fails the test. - Not tested: a full live proxy session. The behaviours are covered at the decision, transform, and handler-call level by the new tests. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes This one touches compression gating, so it's worth a careful read on the injection coupling, that's the part where a wrong call would re-introduce data loss. 1. Tool results with no id mapping still compress, marked with `# ponytail:` comments. Only ids we can positively identify as the CCR tool are exempted. 2. The injection coupling keys off `injector.has_compressed_content`, so the tool only shows up when there's actually something to retrieve. --------- Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
Ben Younes
|
90734b691a
|
fix(proxy): keep large compression results on the critical path (#296) (#1352)
## Description In Anthropic token mode, compression appears to complete in the transform pipeline (`proxy.log` shows `Pipeline complete: ... saved N tokens`), but ~30s later the proxy times out in `compression_first_stage` and forwards the **original** uncompressed request — so `/stats` and `recent_requests` show `tokens_saved: 0`, `savings_percent: 0.0`, `transforms_applied: []`, `optimization_latency_ms: ~31,000`. It starts once a compacted Claude Code transcript grows to ~367k–425k input tokens. Root cause: after the pipeline finishes, `TransformPipeline.apply` runs a **telemetry-only** waste-signal re-parse of the *original* messages (`parse_messages`) on the critical path. On a several-hundred-thousand-token transcript that diagnostic parse can take tens of seconds and blow the Anthropic compression timeout — so the already-computed compression result is discarded and the proxy fails open with the original request. Fix: skip waste-signal detection above `MAX_WASTE_SIGNAL_DETECTION_TOKENS` (100k). The diagnostic never changes the compression result, so skipping it on huge requests keeps the result on the critical path. Smaller requests are unaffected. (The earlier diagnostics PRs #303/#304 — both merged — added the `request_id`/exception-type logging that made this root cause visible. This is the focused follow-up fix.) Closes #296 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/pipeline.py`: gate waste-signal detection on `tokens_before <= waste_signal_token_limit` (default `MAX_WASTE_SIGNAL_DETECTION_TOKENS = 100_000`, overridable via kwarg); above the limit, log a debug line and skip. Extracted the "saved enough" predicate and a named `_MIN_TOKENS_SAVED_FOR_WASTE_SIGNALS` constant (was a bare `100`). - `tests/test_transforms/test_pipeline_waste_signal_limit.py`: new regression test — above the limit the waste-signal parse is skipped and the compression result is preserved; below the limit it still runs. - `CHANGELOG.md`: Unreleased → Bug Fixes entry. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run pytest tests/test_canonical_pipeline.py tests/test_proxy_anthropic_compression_diagnostics.py tests/test_transforms/test_pipeline_waste_signal_limit.py -q 12 passed in 35.97s $ uv run ruff check headroom/transforms/pipeline.py tests/test_transforms/test_pipeline_waste_signal_limit.py All checks passed! $ uv run mypy headroom/transforms/pipeline.py Success: no issues found in 1 source file ``` #### TDD verification (RED → GREEN) RED — new test with the prod fix reverted (waste-signal detection still runs on the large request): ```text E AssertionError: waste-signal parse must be skipped above the limit assert True is False 1 failed, 1 passed in 0.17s ``` (The 1 passing on red is the below-limit no-regression guard.) GREEN — with the fix applied: ```text 2 passed in 0.12s ``` ## Real Behavior Proof - Environment: Linux, Python 3.13, headroom @ this branch. - Exact command / steps: drive `TransformPipeline.apply` with a stub transform that compresses and a tracked `parse_messages`, sizing the request above vs below the limit: - `tokens_before=200_000`, limit `100_000` → `parse_messages` is **not** called; the result still carries `transforms_applied=['test:shrink']` and `tokens_after < tokens_before` (pre-fix: `parse_messages` ran, which is the slow step the timeout killed, discarding this result). - `tokens_before=10_000`, limit `100_000` → `parse_messages` **is** called (diagnostic preserved for normal requests). - Observed result: above the limit the compression result reaches the caller without the diagnostic parse that caused the timeout; below the limit behavior is unchanged. - Not tested: the live multi-hundred-k-token Claude Code session against Anthropic that originally tripped the wall-clock timeout (needs a real large transcript + provider); the causal chain (slow `parse_messages` on the critical path → timeout → discard) is covered deterministically by the unit test. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes The limit is overridable per-call via the `waste_signal_token_limit` kwarg, so callers that want the diagnostic on larger requests can opt back in. Waste-signal data is telemetry only (OTel metrics) — it never affects the compressed output sent upstream. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
Vinay Gupta
|
82384022bd
|
fix(code): slice tree-sitter byte offsets as UTF-8 (#1332)
## Description CodeAwareCompressor was slicing Python strings with tree-sitter `start_byte` / `end_byte` offsets directly. That works for ASCII-only files, but it corrupts slices after non-ASCII source text such as CJK characters or emoji because tree-sitter offsets are UTF-8 byte offsets while Python string indexes are character offsets. This caused code-aware compression to produce invalid intermediate Python and then safely fall back to the original file, resulting in 0% compression on affected files. Closes #1319 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added `_slice_code_bytes()` in `headroom/transforms/code_compressor.py` to slice source text using UTF-8 byte offsets. - Updated `_get_node_text()` to use byte-safe slicing. - Routed the other direct tree-sitter byte-offset slices through the same helper. - Added regression tests in `tests/test_transforms/test_code_compressor.py`: - `test_get_node_text_uses_utf8_byte_offsets` - `test_ast_compresses_python_after_non_ascii_source` ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ .venv/bin/python -m pytest tests/test_transforms/test_code_compressor.py 68 passed, 1 warning $ .venv/bin/python -m ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py All checks passed! $ .venv/bin/python -m ruff format --check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py 2 files already formatted $ git diff --check # no output $ /tmp/headroom-1319-venv/bin/python -m mypy headroom headroom/proxy/server.py:1186: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] headroom/proxy/server.py:1257: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] headroom/proxy/server.py:1261: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] pyproject.toml: note: unused section(s): module = ['mlx.*'] Success: no issues found in 394 source files ``` ## Real Behavior Proof - Environment: macOS arm64, Python 3.14.5, tree-sitter 0.25.2, tree-sitter-language-pack 0.13.0 - Exact command / steps: On `main`, ran a local reproducer with a Python source string containing a CJK docstring before a second function; called `_get_node_text()` on the second tree-sitter function node; ran a full `CodeAwareCompressor.compress(...)` repro with non-ASCII module text before an import and a compressible function; re-ran both repros on this branch. - Observed result: Before fix, `_get_node_text()` returned the wrong slice (`'nd():\n return 2\n'` instead of `'def second():\n return 2'`) and full compression fell back to the original file with `compression_ratio: 1.0`; after fix, `_get_node_text()` returns the full expected function slice and full compression succeeds with `compression_ratio < 1.0`, `syntax_valid: True`, and does not return the original. - Not tested: Full repository test suite; live proxy/provider integrations; Windows/Linux platform-specific behavior. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes - Documentation was not updated because this is an internal bug fix with no user-facing API or behavior change beyond restoring intended compression. - `CHANGELOG.md` was not updated because the fix is narrow and issue-scoped; maintainers can advise if they want a changelog entry. - The fix is intentionally small and targeted: it only changes how tree-sitter byte offsets are converted back into Python source text, without changing compression heuristics or language behavior. |
||
|
Vinay Gupta
|
c35af858ea
|
fix(code): compress class member containers (#1334)
## Description CodeAwareCompressor used the same `body_node_types` config to find both executable function bodies and class/impl member containers. That works when those AST nodes happen to match, but it misses member containers such as Java `class_body`, C++ `field_declaration_list`, and Rust `declaration_list`, so class methods were returned essentially uncompressed. This adds an optional `class_body_node_types` override for class/impl member containers and uses it only in class compression. It also skips anonymous punctuation tokens while reconstructing class bodies and keeps same-line C++ class semicolons attached to the compressed class declaration. Closes #1318 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added `LangConfig.class_body_node_types` for languages whose class/impl member container differs from executable method-body nodes. - Configured class member containers for JavaScript, TypeScript, Java, C++, and Rust. - Updated `_compress_class_ast` to use class-member containers, skip anonymous punctuation children, and preserve C++ `};` output without creating stray top-level semicolons. - Added regression coverage proving class/impl methods compress for JavaScript, TypeScript, Java, C++, and Rust. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ PYTHONPATH=. /tmp/headroom-1319-venv/bin/python -m pytest tests/test_transforms/test_code_compressor.py -q collected 71 items tests/test_transforms/test_code_compressor.py .......................... [ 36%] ............................................. [100%] 71 passed, 1 warning in 0.36s $ /tmp/headroom-1319-venv/bin/python -m ruff check . All checks passed! $ /tmp/headroom-1319-venv/bin/python -m ruff format --check . 965 files already formatted $ PYTHONPATH=. /tmp/headroom-1319-venv/bin/python -m mypy headroom headroom/proxy/server.py:1186: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] headroom/proxy/server.py:1257: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] headroom/proxy/server.py:1261: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked] pyproject.toml: note: unused section(s): module = ['mlx.*'] Success: no issues found in 394 source files ``` ## Real Behavior Proof - Environment: macOS arm64, Python 3.14.5, branch `fix-code-compressor-class-members`, tree-sitter grammar pack installed in `/tmp/headroom-1319-venv`, repo imported with `PYTHONPATH=.`. - Exact command / steps: Reproduced class-method compression with `CodeAwareCompressor(CodeCompressorConfig(enable_ccr=False, min_tokens_for_compression=1, max_body_lines=1))` for Java/C++/Rust before the fix, then reran the pytest/ruff/mypy commands listed above after the patch. - Observed result: Java/C++/Rust class methods now compress below 1.0 while `syntax_valid` remains true; C++ output preserves `};`; regression coverage also verifies JavaScript/TypeScript class member containers. - Not tested: Full repository pytest suite; local `uv run` editable builds are blocked on this machine by native C++ header failures in optional/native dependencies (`hnswlib` / Rust `esaxx-rs`), so validation used a lightweight venv with `PYTHONPATH=.`. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes Documentation and CHANGELOG updates are not applicable for this narrow bug fix. The pytest warning shown above is from running without `pytest-asyncio` in the lightweight verification venv (`asyncio_mode` config is unknown there); it is unrelated to this change. |
||
|
Parafee41
|
cbd361de2a
|
fix(code): validate Python compressed syntax (#1302)
## Description Fix a Python code-compression validity gap from #1233 where tree-sitter parsing could mark compressed output as syntactically valid even when Python compile-time syntax rules reject it. This keeps `from __future__ import ...` statements in the import-preservation bucket so they stay before executable definitions, and adds Python `compile(..., "exec")` verification after `ast.parse`. It also keeps the earlier conservative class-method decorator indentation hardening from this branch. Refs #1233. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Treat Python `future_import_statement` nodes as preserved imports. - Verify Python compressed output with both `ast.parse` and `compile(..., "exec")`. - Preserve original source-line indentation for decorators attached to class methods. - Add a regression fixture covering `from __future__ import annotations`, class decorators, property decorators, async methods, and `match` statements. - Add a direct regression assertion that future imports stay before executable definitions. - Document the user-visible fix in `CHANGELOG.md`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ /tmp/headroom-issue-1233-venv/bin/python -m pytest tests/test_transforms/test_code_compressor.py::TestTreeSitterIntegration::test_python_future_import_stays_at_module_start -q 1 passed, 1 warning $ /tmp/headroom-issue-1233-venv/bin/python -m pytest tests/test_transforms/test_code_compressor.py -q 61 passed, 1 warning $ uv run --extra dev ruff check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py All checks passed! $ uv run --extra dev ruff format --check headroom/transforms/code_compressor.py tests/test_transforms/test_code_compressor.py 2 files already formatted $ git diff --check # no output ``` ## Real Behavior Proof - Environment: macOS, Python 3.11.14, local checkout with `[code]` dependencies installed in `/tmp/headroom-issue-1233-venv`. - Exact command / steps: added `test_python_future_import_stays_at_module_start`, ran it before the fix to confirm the compressed output failure, then reran the focused test and full `tests/test_transforms/test_code_compressor.py` after the patch. - Observed result: before this patch, the regression fixture produced compressed Python with `from __future__ import annotations` after class/function definitions. `result.syntax_valid` was `True`, but `compile(result.compressed, "<test>", "exec")` failed with `SyntaxError: from __future__ imports must occur at the beginning of the file`. After this patch, the focused regression and full code-compressor test file pass locally, and the regression now directly asserts that the future import appears before executable definitions. - Not tested: full repository pytest, `mypy headroom`, and a broad corpus run over third-party source files. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) N/A ## Additional Notes This PR is now scoped to the stable compile-time failure path in #1233. The broader syntax-failure rate from the issue may still need corpus-level follow-up. Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
Nadia Ujovich
|
7c93c50c2c
|
Marker-free lossless_only mode + gate opaque-blob CCR markers behind enable_ccr_marker (#1129)
## Description `enable_ccr_marker` only gated the **row-drop sentinel** path. The **opaque-blob** path still emitted `<<ccr:HASH,kind,size>>` markers unconditionally whenever a string cell exceeded `opaque_min_bytes` (256), so **no configuration could produce a fully marker-free prompt**. Any `<<ccr:>>` marker is a promise that the full payload lives in the CCR store and must be fetched back via a retrieval tool call — there was no way to get compression without that round-trip dependency. **Rebased on #1130 (merged).** That PR fixed the opaque-blob gate at the classifier (`ClassifyConfig.emit_opaque_markers`, driven by `enable_ccr_marker`) and closed #1091. This branch originally carried its own equivalent gating commit; that commit is now **redundant and has been dropped** — `classifier.rs` here is identical to upstream. What remains is the **net-new** work that is **not** in #1130: - **Strict `lossless_only` mode** — keeps lossless tabular compaction, but routes every path that would need a CCR marker (row-drop sentinel **and** opaque-blob offload) to leave content uncompacted instead, so output is always marker-free **and** byte-recoverable. - **Python parity** — `lossless_only` exposed across both config dataclasses, a `SmartCrusher` kwarg, and a per-call `crush(..., lossless_only=)` override. - **`HEADROOM_LOSSLESS_ONLY` env var** — wires the mode through to the proxy runtime so real agents can use it. The #1130 opaque gate is consumed here through a single centralized helper (`opaque_markers_enabled() = enable_ccr_marker && !lossless_only`) used by **all four** `ClassifyConfig` construction sites. ## Type of Change - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - **`feat(smart_crusher)`** — Add `lossless_only` (default `false`): keeps lossless tabular compaction but routes every marker-requiring path (row-drop sentinel + opaque-blob offload) to leave content uncompacted instead. Exposed across the Rust core, PyO3 bridge, both Python config dataclasses, a `SmartCrusher` kwarg, a per-call `crush(..., lossless_only=)` override, and `smart_crush_tool_output`. Includes a `debug_assert` documenting the load-bearing invariant (a `lossless_only` crusher must never reach the CCR store write). - **`refactor(smart_crusher)`** — Extract `SmartCrusherConfig::opaque_markers_enabled()` as the single source of truth for `enable_ccr_marker && !lossless_only`, consumed by **all four** `ClassifyConfig` sites: the compaction-stage builder, `with_compaction_format`, the top-level `process_string` path (Rust core), and the PyO3 `compact_document_json` document-compactor path. No site derives the gate inline anymore, so they cannot drift. - **`feat(proxy)`** — Expose the mode via `HEADROOM_LOSSLESS_ONLY`: `ContentRouterConfig.smart_crusher_lossless_only` → `_get_smart_crusher`; the proxy reads the env var and sets it on the live router config. Previously reachable only via the Python API, never through the proxy. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check` on changed files) - [ ] Type checking passes (`mypy headroom`) — not run (see Additional Notes) - [x] New tests added for new functionality - [x] Manual testing performed (proxy env-var seam, end-to-end — see Real Behavior Proof) ### Test Output ```text ### RUST (cargo test -p headroom-core --lib smart_crusher) test result: ok. 323 passed; 0 failed; 0 ignored; 0 measured; 516 filtered out ### PYTEST (test_smart_crusher_bugs.py + test_agent_savings.py + test_smart_crusher_toin_attachment.py) 45 passed ### RUFF (changed files) All checks passed! ### FMT + CLIPPY (cargo fmt --check && cargo clippy --workspace --lib) clean — no warnings ``` New/affected tests: `enable_ccr_marker_false_suppresses_opaque_markers`, `lossless_only_leaves_array_uncompacted_instead_of_dropping`, `lossless_only_inlines_opaque_blobs_when_table_ships`, `lossless_only_never_writes_to_ccr_store` (Rust); `TestLosslessOnlyMode`, `test_router_lossless_only_flag_reaches_crusher`, `test_router_lossless_only_defaults_off` (Python). Coexists green with #1130's `long_string_stays_scalar_when_opaque_markers_disabled` (Rust) and `test_smart_crusher_toin_attachment.py` (Python). The Python `TestOpaqueMarkerGate` from the dropped gating commit was removed as redundant with #1130's coverage. ## Real Behavior Proof ### Proxy env-var seam — end-to-end (this revision) The one path with no automated coverage was `server.py` reading `HEADROOM_LOSSLESS_ONLY` from the environment and threading it into the live router config. Verified end-to-end by instantiating the **real** `HeadroomProxy`, pulling the `ContentRouter` out of its pipeline, and crushing a 50-row array with >256B opaque cells through the real Rust crusher: | | `HEADROOM_LOSSLESS_ONLY=1` | env unset (default) | |---|---|---| | `crusher._lossless_only` | **True** | **False** | | output contains `<<ccr:` | **No** (marker-free) | Yes (normal lossy) | | byte-recoverable (round-trips to original JSON) | **Yes** | No (rows offloaded) | This confirms the full chain `os.environ["HEADROOM_LOSSLESS_ONLY"]` → `server.py` → `ContentRouterConfig.smart_crusher_lossless_only` → `content_router.py` → `crusher_config.lossless_only` → Rust crusher. The default column proves strict mode genuinely changes behavior (not a no-op) and that the default path is unchanged. ### Prior live-traffic run - Environment: Headroom proxy in front of a real agent (Hermes) routed to NVIDIA NIM (OpenAI-compatible upstream). Isolated config dir; `OPENAI_TARGET_API_URL=https://integrate.api.nvidia.com/v1`, `HEADROOM_LOSSLESS_ONLY=1`. Confirmed with `ss` that all LLM traffic flowed agent → proxy → upstream with no direct bypass. - Exact command / steps: Start the proxy with `python -m headroom.proxy.server --host 127.0.0.1 --port 8787`; point the agent's LLM base_url at `http://127.0.0.1:8787/v1`; run a real chat plus a `search_files`-style task; read `/stats` and `/v1/retrieve/stats`; then toggle `HEADROOM_LOSSLESS_ONLY` and repeat for the comparison. - Observed result: With 150K+ tokens of real traffic processed, `lossless_only` kept the CCR store empty (`entry_count: 0`) and emitted zero markers. A synthetic before/after with opaque (>256B) cells produced 12 `<<ccr:>>` markers in default mode and 0 under `lossless_only`, with output round-tripping to the original JSON structure. - Not tested: A live `lossless_only`-vs-markers contrast on real agent traffic. The SmartCrusher offload path never engaged on this agent's tool outputs (`total_compressions: 0`; CCR store stayed at `entry_count: 0` even after a broad codebase search), and compression stayed marginal (~0.2–0.4%) in both modes. The agent's tool results don't match the crushable-array profile the offload paths target, so the marker path is never exercised in that integration. Why SmartCrusher barely engages with this agent's outputs is a separate integration question (output format / routing / size thresholds), out of scope for this change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation — N/A (config docstrings updated in-tree; no separate docs) - [x] My changes generate no new warnings - [x] I have added tests that prove my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable — N/A ## Additional Notes - Rebased on top of merged #1130; the now-redundant opaque-blob gating commit was dropped, so this PR is purely the `lossless_only` feature + proxy wiring on top of #1130's gate. - `mypy headroom` was not run in this environment; happy to add the result if CI requires it. - Default behavior is fully preserved: `enable_ccr_marker` defaults to `true`, `lossless_only` defaults to `false`, and `HEADROOM_LOSSLESS_ONLY` unset is a no-op. |
||
|
Zhenjia ZHOU
|
6c68ff4e9f
|
perf(compression): take large cold-start contexts off the synchronous kompress path (#1171) (#1298)
## Description On a cold-start large context, kompress (ModernBERT ONNX) runs **synchronously on the request thread** — ~200–300s for ~1M tokens. It blows the 30s compression budget, leaks a non-preemptible worker, and cascades (executor saturation → queue timeouts on healthy requests); on timeout the request is forwarded **uncompressed** after eating 30s. This adds four layered, **default-off, fail-open** mitigations so the request path is never blocked on ML compression. Closes #1171 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Phase 0 — size gate** (`HEADROOM_KOMPRESS_MAX_TOKENS`, default 50000): route oversized text away from ModernBERT (→ LogCompressor / TextCrusher / passthrough) at the single `_try_ml_compressor` boundary. - **Phase 1 — cooperative deadline** (`HEADROOM_COMPRESSION_DEADLINE_MS`, default 20000): any kompress run self-terminates at the next chunk boundary past the budget, keeping the unprocessed tail verbatim. - **Phase 2 — TextCrusher** (`HEADROOM_TEXT_CRUSHER`): a new **native Rust** extractive prose compressor in `crates/headroom-core/src/transforms/text_crusher/`, exposed via PyO3 as `headroom._core.TextCrusher` with a thin Python wrapper. It **reuses the shared `crate::relevance::BM25Scorer`** rather than reimplementing BM25, and ships record/replay parity fixtures (mirroring the SmartCrusher Rust-core + Python-shim pattern). - **Phase 3 — off-path compression** (`HEADROOM_BACKGROUND_COMPRESSION`): forward uncompressed immediately and compress in a per-process background drain; a byte-identical cache hit on a later turn means the request never blocks on ML. - Benchmark (`benchmarks/text_crusher_quality_eval.py`), CHANGELOG entry, and docstrings documenting the fail-open limits. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`, new modules) - [x] New tests added for new functionality - [ ] Manual testing performed (Phase 0/1 gate-fire + deadline observed on real traffic in earlier iterations; Phase 3 off-path is unit- + byte-identity-tested, not yet live-validated) ### Test Output ```text $ pytest tests/test_transforms/ tests/test_cache/ \ tests/test_proxy/test_background_compression.py tests/test_proxy/test_phase3_byte_identity.py -q 501 passed, 37 skipped in 40.33s $ cargo test -p headroom-core --lib text_crusher test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 834 filtered out $ ruff check <changed files> All checks passed! $ mypy headroom/proxy/background_compression.py headroom/transforms/text_crusher.py Success: no issues found in 2 source files ``` New coverage: size-gate incl. the strategy-dispatch funnel (KOMPRESS + TEXT); partial-run deadline (chunk-0 compressed + chunk-1 verbatim tail); BackgroundCompressor (dedup / queue-full / fail-open); Phase 3 byte-identity round-trip; TextCrusher unit + parity. ## Real Behavior Proof - Environment: macOS, local dev — `uv` venv, Rust `_core` built via `uv pip install -e .`. - Exact command / steps: the `pytest` / `cargo test` / `ruff` / `mypy` commands shown under Test Output; quality eval `python benchmarks/text_crusher_quality_eval.py /tmp/squad_dev.json`. - Observed result: 501 Python + 3 Rust tests pass; ruff + mypy clean on changed/new modules. Quality eval: TextCrusher keeps ~94% of buried SQuAD answers at 30% size vs ~36% truncate/random; self-contained speed run ~333k words in ~76ms (one O(n) pass) — sub-second where ModernBERT takes minutes (fast-vs-slow contrast, not a same-input run). - Not tested: Phase 3 off-path on live traffic; multi-worker (per-process by design — see Additional Notes). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - **All four features are off by default and fail-open** — with the env flags unset the paths are no-ops for realistic inputs; on any error the request is forwarded (compressed if possible, else verbatim), never dropped. A full background queue / duplicate key surfaces as `deferred:dropped`. - **Known limits (documented in `background_compression.py`):** Phase 3 is per-process, in-memory, and token-mode-only — these are **lost-savings, never lost-correctness**, and consistent with the project's existing per-process compression cache + sticky-session multi-worker model. The startup multi-worker warning now names off-path background compression. - Phase 2 reuses the existing BM25 scorer; reuse did not improve answer-retention over a Python prototype (query-awareness dominates) — its value is the Rust speed + repo-conventional Rust-core/Python-shim shape. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Parideboy
|
3ccdad6c67
|
Pin ORT dylib on Windows; init Python logging (#1010)
## Description On Windows, headroom's Rust core resolves `onnxruntime.dll` at runtime via `ort-load-dynamic`. Without an explicit `ORT_DYLIB_PATH`, the bare DLL search can land on `C:\Windows\System32\onnxruntime.dll`, the Windows ML OS component, and `Session::new()` can deadlock instead of returning an error. Since a hang is not an `Err`, the tiered fallback cannot engage until the proxy-level timeout fires. This PR pins `ORT_DYLIB_PATH` to the pip-installed `onnxruntime` DLL at import time, and wires Rust `tracing` events into Python logging so the proxy log surfaces these failures when they occur. Closes #928 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added `headroom/_ort.py` with a Windows-only, idempotent `ensure_ort_dylib_pinned()` resolver that respects an existing `ORT_DYLIB_PATH`. - Call the pin from `headroom/__init__.py` before importing `_core` consumers. - Log the effective ORT dylib path from the content router startup path on Windows. - Enable Rust tracing-to-log compatibility and initialize `pyo3-log` in the `_core` module. - Add timeout diagnostics in the Magika detector with the effective `ORT_DYLIB_PATH`. - Document `ORT_DYLIB_PATH` and `HEADROOM_MAGIKA_INIT_TIMEOUT_SECS`. - Add unit coverage for the resolver behavior. ## Testing - [x] Unit tests pass (`python -m pytest tests/test_transforms/test_ort_dylib.py -q`) - [x] Linting passes (`ruff check headroom/_ort.py headroom/__init__.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py`) - [x] Formatting passes (`ruff format --check headroom/_ort.py headroom/__init__.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_transforms/test_ort_dylib.py -q 7 passed in 0.19s $ ruff check headroom/_ort.py headroom/__init__.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py All checks passed! $ ruff format --check headroom/_ort.py headroom/__init__.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py 4 files already formatted $ cargo check -p headroom-py cargo: The term 'cargo' is not recognized as a name of a cmdlet, function, script file, or executable program. ``` ## Real Behavior Proof - Environment: Windows 11 24H2, Python 3.13, RTX 4080 - Exact command / steps: `python -c "import headroom; from headroom._core import detect_content_type as d; print(d(open('headroom/compress.py').read()).content_type)"` - Observed result: `source_code` in 301ms, clean exit, `Magika: ENABLED` in proxy log - Not tested: macOS/Linux manual runtime behavior; `_ort.py` is a no-op outside Windows, and CI covers cross-platform build/test behavior. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable (N/A: repo uses release-please) ## Additional Notes The branch was rebased onto current `main` and the commit subject was updated to satisfy commitlint. Local Rust verification could not be run on this Windows machine because `cargo` is not installed; GitHub CI should be treated as the Rust build verification for the `pyo3-log` dependency and workspace lockfile changes. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Rocker Zhang
|
5e0bb69725
|
fix(code): verify a real parse in tree-sitter availability check (#1231) (#1299)
## Description `is_tree_sitter_available()` / `_check_tree_sitter_available()` in `headroom/transforms/code_compressor.py` return `True` based on importing `tree_sitter_language_pack` alone, without ever constructing a parser or attempting a parse. When the installed pack/parser combination is ABI-incompatible, `get_parser`/`parse` raises at runtime; the caller catches it and silently falls back to the lossy text compressor, while the availability flag and startup banner still report code-aware as on. This is the defensive half that the `<1.0` pin in #1234 does not cover: if that cap is ever lifted, the availability signal silently lies again. Follow-up to #1231. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Make `_check_tree_sitter_available()` construct a parser and parse a tiny snippet, returning `True` only if it yields a real `module` AST instead of trusting an import. - Add `_tree_sitter_importable()` for the cheap import-only probe, and use it to guard parser construction so the real-parse check cannot recurse. - Add tests asserting the check is `False` when parsing raises and `True` on a real parse, plus that AST compression runs for python/rust without falling back. ## Testing - [ ] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # pytest tests/test_transforms/test_code_compressor.py -> passed locally (tree-sitter-language-pack 0.13.0) # ruff check . and ruff format --check . pass locally on the rebased branch. # Full pytest suite / mypy not run locally; left to CI. ``` ## Real Behavior Proof - Environment: local repo on tree-sitter-language-pack 0.13.0, tree-sitter 0.25.2, Python 3.12, Linux - Exact command / steps: call `is_tree_sitter_available()`, then run `pytest tests/test_transforms/test_code_compressor.py` - Observed result: with a working pack the probe parses and returns `True` (code-aware runs, strategy `CODE_AWARE` rather than the kompress fallback); the new `test_check_tree_sitter_available_false_when_parse_broken` confirms that when parsing raises the check now returns `False` instead of the old import-only `True`, so the lossy fallback is no longer entered silently. - Not tested: reproducing the specific ABI-incompatible 1.x pack combo against a live install (covered instead by a mocked broken parse in the test) ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable |
||
|
Parideboy
|
a00fb6761e
|
fix(router): degrade to pure-Python detection on native panic (#1123) (#1260)
## Description When the native (Rust) content detector panicked, the pyo3 `PanicException` (a `BaseException`, not `Exception`) escaped `_detect_content` and surfaced as an HTTP 500 instead of degrading. This catches `BaseException` (excluding control-flow exceptions) around the native call and falls back to the pure-Python regex detector, logging a single warning. Closes #1123 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - `headroom/transforms/content_router.py`: wrapped the native detect call in `_detect_content` so any `BaseException` (except `KeyboardInterrupt`/`SystemExit`/`GeneratorExit`) degrades to `_regex_detect_content_type`, warning once via a module-level `_detect_panic_warned` flag. - `tests/test_transforms/test_detect_fallback_1123.py`: new regression tests for RuntimeError fallback, BaseException-panic fallback, and KeyboardInterrupt propagation. ## Testing - [x] Unit tests pass (`pytest`) - [x] New tests added for new functionality ### Test Output ```text $ pytest tests/test_transforms/test_detect_fallback_1123.py tests/test_transforms/test_content_router.py -q 54 passed ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, pytest-asyncio 1.4.0 - Exact command / steps: Monkeypatched the native detector to raise RuntimeError, a BaseException-derived fake panic, and KeyboardInterrupt, then called `_detect_content`. - Observed result: RuntimeError and the BaseException panic both degrade to a valid regex detection result; KeyboardInterrupt still propagates. 54 tests pass. - Not tested: Could not reproduce a real pyo3 panic in this build (`pyo3_runtime` is not importable here), so the fallback is exercised via simulated exceptions. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Parideboy
|
a2159c0b66
|
feat(proxy): support glob patterns in exclude_tools (#870) (#1259)
## Description `exclude_tools` only matched tool names exactly, so users could not exclude families of tools (for example all `mcp__*`). This adds glob-pattern support via a shared `is_tool_excluded` helper used by both the content router and the OpenAI handler, keeping exact/case-insensitive matching intact. Closes #870 ## Type of Change - [x] New feature (non-breaking change that adds functionality) ## Changes Made - `headroom/config.py`: added `is_tool_excluded(name, exclude_tools)` helper that keeps exact/case-insensitive matching and adds `fnmatch` glob support. - `headroom/transforms/content_router.py` and `headroom/proxy/handlers/openai.py`: routed tool-exclusion checks through the shared helper. - `headroom/proxy/server.py`: documented glob support in the `--exclude-tools` CLI help and `_parse_exclude_tools` docstring. - `tests/test_transforms/test_content_router.py`: added `test_glob_exclude_tools` and `test_is_tool_excluded_helper`. ## Testing - [x] Unit tests pass (`pytest`) - [x] New tests added for new functionality ### Test Output ```text $ pytest tests/test_transforms/test_content_router.py -q 53 passed $ pytest tests/ -k "exclude or config" -q 59 passed ``` ## Real Behavior Proof - Environment: Windows 11, Python 3.12, pytest-asyncio 1.4.0 - Exact command / steps: Ran the content-router suite and the exclude/config-focused tests after adding the helper and glob support. - Observed result: 53 content-router tests pass (including the two new glob tests) and 59 exclude/config tests pass; glob patterns like `mcp__*` now exclude matching tools while exact names still work. - Not tested: Did not exercise glob exclusion against a live MCP server end to end. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
xspawnnn
|
3fc2a78a5e
|
fix(kompress): never block the request path on the cold-cache model download (#1161)
Closes #1146.
## Problem
On a cold cache, the first request that reaches the Kompress deep
compressor triggers an inline `hf_hub_download` of the 274 MB
`chopratejas/kompress-v2-base` ONNX model **on the request thread**.
That download races the proxy's compression budget
(`HEADROOM_COMPRESSION_TIMEOUT_SECONDS`, default 30s — the
`compression_first_stage` timeout): the fetch is cancelled mid-transfer,
**nothing finalizes in the HF cache**, and the request fails open
(uncompressed). Because the partial blob never lands, every subsequent
request repeats the same ~30s hang + fail-open, so the deep compressor
never actually becomes available through the proxy.
This is a **distinct root cause from #946** (which concerns the timeout
itself). Here the model must simply never be fetched synchronously on a
latency-sensitive request.
## Fix
Make the request path cache-only and move the one-time download
off-thread.
**`kompress_compressor.py`**
- `compress(..., allow_download=False)` — new keyword (default `True`,
so the direct API and `compress_batch` are unchanged) that resolves the
model cache-only; on a cold cache it raises `KompressModelNotCached` and
passes through instead of blocking on the network.
- `is_ready()` — lockless cache-membership check, safe to call on the
hot path.
- `ensure_background_download(model_id, device)` — starts at most one
daemon thread per model to pull the artifact down out of band (a
finished/failed thread is replaced, so a transient failure can be
retried by a later request). The compression timeout does not bound this
thread.
**`content_router.py`** — gate the deep path on readiness:
- not ready → return passthrough immediately and kick off the background
download;
- ready → `compress(allow_download=False)` (cache-only, no network on
the request thread).
Net effect: the cold-cache deep path returns in ~0 ms (passthrough)
instead of hanging ~30 s; the model downloads once in the background;
subsequent requests transparently use the deep compressor once it is
cached.
## Verification
Clean install of `headroom-ai==0.26.0` (main `@
|
||
|
ulias
|
e36fccd8cf
|
refactor: DRY cache logic, add thread safety, fix Bash exclusion (#704)
## Description Four targeted improvements to ContentRouter and configuration, refactoring ~120 lines of duplicated cache logic into a shared helper and fixing several correctness issues. ### 1. DRY: Extract `_compress_block_content` helper The two-tier cache lookup + compression logic was duplicated ~60 lines per path (tool_result blocks and text blocks in `_process_content_blocks`). Extracted into a single, shared helper method. Net reduction of ~80 lines; no behavioural change. ### 2. Thread-safe `CompressionCache` `CompressionCache` is read/modified from `ThreadPoolExecutor` workers during parallel compression in `apply()`. Added a `threading.Lock` guarding all read-modify-write operations so concurrent cache misses for the same content do not produce duplicate compression work and metrics counters stay consistent. ### 3. Remove duplicate Kompress fallback for SmartCrusher The SMART_CRUSHER strategy block had an inline Kompress fallback that ran when SmartCrusher produced no savings. The unified post-strategy fallback block already covers the same case — the inline copy was a duplicate Kompress invocation. Removed it; the post-strategy handler now owns all fallback decisions for both SMART_CRUSHER and CODE_AWARE. Also added a guard preventing duplicate Kompress when CODE_AWARE's inline fallback fires alongside the unified block. ### 4. Fix Bash exclusion contradiction in `DEFAULT_EXCLUDE_TOOLS` The docstring on `DEFAULT_EXCLUDE_TOOLS` explicitly states "Bash is NOT excluded — its outputs (build logs, test output) are ideal compression targets." But both "Bash" and "bash" were still in the frozenset. Removed them so code matches the documented intent. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - `headroom/config.py`: Remove Bash/bash from `DEFAULT_EXCLUDE_TOOLS` - `headroom/transforms/content_router.py`: Extract `_compress_block_content` helper; unified post-strategy fallback block; threading.Lock on CompressionCache; CODE_AWARE duplicate guard - `headroom/client.py`: Replace silent `except Exception: pass` with `logger.debug(..., exc_info=True)` - `tests/test_compression_cache.py`: Add 2 concurrency regression tests - `tests/test_transforms/test_content_router.py`: Add 14 tests covering Bash exclusion, SmartCrusher fallback chain, and `_compress_block_content` shared path ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Formatting passes (`ruff format --check .`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # 14 new tests added across 3 test classes: # TestExcludeTools: 3 tests (Bash not in DEFAULT_EXCLUDE_TOOLS) # TestSmartCrusherFallback: 4 tests (fallback chain, no duplicate Kompress, JSON direct hit, CODE_AWARE path) # TestCompressBlockContent: 5 tests (skip set, result cache, ratio gating, route counts, transforms tracking) # TestCompressionCache: 2 tests (concurrent hits/misses consistency, stable hash ops no race) # Local run (43 tests pass): $ pytest tests/test_compression_cache.py tests/test_transforms/test_content_router.py -v ...43 passed... # ruff check: $ ruff check headroom/client.py headroom/config.py headroom/transforms/content_router.py tests/test_compression_cache.py tests/test_transforms/test_content_router.py All checks passed! # ruff format: $ ruff format --check headroom/client.py headroom/config.py headroom/transforms/content_router.py tests/test_compression_cache.py tests/test_transforms/test_content_router.py 5 files already formatted ``` ## Real Behavior Proof - Environment: Python 3.12, Linux (CI), headroom with headroom._core Rust extension compiled - Exact command / steps: CI run https://github.com/chopratejas/headroom/actions/runs/27326150021 — 13/16 jobs pass; 2 failures were lint+commitlint (both fixed in subsequent commits); 1 failure is pre-existing test(4) which monkeypatches time.time() but the CompressionCache uses time.monotonic() — unrelated to our changes - Observed result: All 14 new tests pass in CI; SmartCrusher fallback chain deterministically shows [smart_crusher, kompress] or [smart_crusher, kompress, log] when SmartCrusher produces no savings, with no duplicate entries - Not tested: fork-PR CI path where GitHub secrets are not available; local Windows environment where headroom._core Rust extension is not built ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes The pre-existing CI failure in `test (4)` is `test_compression_cache_handles_hits_skips_evictions_and_clear` in `tests/test_transforms_content_router.py`. It monkeypatches `time.time()` but the `CompressionCache` (content_router-local, line 191) uses `time.monotonic()` for TTL — the monkeypatched clock never advances, and `is_skipped()` always returns True. This failure exists on `main` and is unrelated to our changes (we only modified the other CompressionCache in `headroom/cache/compression_cache.py`). --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Michael Sam
|
d2cdab268d
|
feat(proxy): add agent-90 savings profile (#830)
## Summary - add an `agent-90` savings profile with cross-agent proxy env exports - wire the profile into proxy/router runtime kwargs, including force-Kompress routing and a smaller read-protection window - expose effective savings-profile config in `/stats` and add focused regression coverage ## Type of change - [x] feat (non-breaking change which adds functionality) - [ ] fix (non-breaking change which fixes an issue) - [ ] docs - [ ] test/CI-only - [ ] refactor-only ## Testing - [x] `python3 -m py_compile headroom/agent_savings.py headroom/cli/agent_savings.py headroom/cli/main.py headroom/cli/proxy.py headroom/proxy/models.py headroom/proxy/server.py headroom/transforms/content_router.py tests/test_agent_savings.py tests/test_proxy_healthchecks.py tests/test_cli/test_wrap_persistent.py tests/test_transforms/test_content_router.py` - [x] `git diff --check` - [x] manual smoke: `agent-savings --profile agent-90 --format json` returns `HEADROOM_TARGET_RATIO=0.10` - [x] manual smoke: `proxy_pipeline_kwargs(ProxyConfig(savings_profile="agent-90"))` enables `force_kompress`, system/user compression, and `read_protection_window=2` - [x] manual smoke: Anthropic-style `tool_result` routes through Kompress with `target_ratio=0.10` - [ ] `pytest` suite not run: pytest is not installed in the available local Python environments ## Notes This keeps agent-90 as an opt-in profile. Existing defaults remain unchanged unless `HEADROOM_SAVINGS_PROFILE=agent-90` or `ProxyConfig(savings_profile="agent-90")` is set. |
||
|
gglucass
|
841663da16
|
fix(proxy): make Kompress eager preload cache-only so a cold cache can't block startup (#783)
## Description `ContentRouter.eager_load_compressors()` runs a network `hf_hub_download` of the Kompress ONNX model on the **blocking startup/lifespan path**, before the proxy binds its port. On a cold cache this is unsafe: - the download can hang long enough to blow the supervisor's bind timeout, or - a native crash in the download/ML stack (an **uncatchable `Fatal Python error: Aborted` / SIGABRT**) kills the interpreter before it ever `listen()`s. Either way the supervisor sees "proxy never opened its port" and gives up. We observed this in the field from the desktop app (process aborted during `eager_load_compressors -> _load_kompress_onnx -> hf_hub_download` of `onnx/kompress-int8.onnx`, while the only Python thread was parked in the HuggingFace download file-lock; the abort came from a native thread, so `try/except` at the call site cannot catch it). The eager preload is a latency optimization and must never be able to block — or kill — startup. This change makes startup preload **cache-only**: if the model isn't already cached, we defer the download to first use (off the startup path) and bind the port normally. Warm starts are unchanged. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `onnx_runtime.hf_hub_download_local_first(...)`: added `allow_network` (default `True`). When `False`, a cache miss re-raises the local-lookup error instead of falling back to a network download. - `kompress_compressor`: added `allow_download` (default `True`) threaded through `preload()` -> `_load_kompress()` -> `_load_kompress_onnx()` / `_load_kompress_pytorch()` and the ModernBERT tokenizer load. Added `KompressModelNotCached`, raised when a cache-only load misses. Auto-mode no longer falls back to a PyTorch network download on a cache-only miss — it propagates so the caller can defer. - `content_router.eager_load_compressors()`: calls `preload(allow_download=False)`. On `KompressModelNotCached` it logs and reports the component as `"deferred"` (a status `warmup.merge_transform_status` already handles gracefully) instead of letting a cold download run on the startup path. Default (first-request) loading behavior and warm-start preload are unchanged. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed New tests in `tests/test_kompress_preload_deferral.py` cover: cache-only `hf_hub_download_local_first` never hits the network; default still falls back; cache-only ONNX load raises `KompressModelNotCached`; auto-mode does **not** trigger a PyTorch download on a cache-only miss; and `eager_load_compressors` reports `deferred` (cold) / `enabled` (warm). Existing `_load_kompress` dispatch tests updated for the new keyword-only param. > Note on environment: I do not have a clean reproduction of the native SIGABRT itself (it depends on a specific machine's HF download/ML native stack), so the "Manual testing performed" box is left unchecked. The tests target the structural fix — that startup preload can no longer perform a network download — which is the precondition for the crash. ## Test Output ``` $ uv run pytest -v tests/test_kompress_preload_deferral.py tests/test_kompress_preload_deferral.py::test_local_first_no_network_when_disallowed PASSED tests/test_kompress_preload_deferral.py::test_local_first_falls_back_to_network_by_default PASSED tests/test_kompress_preload_deferral.py::test_load_kompress_onnx_cache_miss_raises_not_cached PASSED tests/test_kompress_preload_deferral.py::test_load_kompress_auto_does_not_pytorch_download_on_cache_miss PASSED tests/test_kompress_preload_deferral.py::test_eager_load_defers_when_model_not_cached PASSED tests/test_kompress_preload_deferral.py::test_eager_load_enabled_when_model_cached PASSED 6 passed in 4.82s $ uv run pytest tests/test_transforms/test_kompress_compressor.py tests/test_transforms_content_router.py tests/test_onnx_runtime.py tests/test_proxy_warmup.py 63 passed $ uv run ruff check <changed files> # All checks passed! $ uv run mypy headroom/onnx_runtime.py headroom/transforms/kompress_compressor.py headroom/transforms/content_router.py Success: no issues found ``` ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable (auto-generated from conventional commits) ## Additional Notes This contains the cold-start case. A native crash in onnxruntime *session init* (as opposed to the download) on first request would still be a separate issue; it is not what was observed here (the abort was during the HF download), and isolating it would be a larger, separate change. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
gglucass
|
8f374263d3
|
feat(transforms): attribute read_lifecycle + smart_crush tags (#249)
## What Enrich the `transforms_applied` tags emitted by `ReadLifecycleManager` and `SmartCrusher` so each tag carries the specific target it acted on, instead of being an opaque counter: - `read_lifecycle:<state>` -> `read_lifecycle:<state>:<file_path>` - `smart_crush:<n>` -> `smart_crush:<n>:<tool1,tool2,...>` (tool names resolved from the assistant's `tool_calls` / `tool_use` metadata; falls back to `smart_crush:<n>` when no name resolves) Downstream UIs can then show *what* a compression acted on (which file was a stale read, which tools had their output crushed), not just that it happened. ## Note on the rebase The original revision targeted `ToolCrusher` / `tool_crush:<n>`. That transform has since been retired and replaced by the Rust-backed `SmartCrusher` (which emits `smart_crush:<n>`), so the tool-name attribution moved to `smart_crusher.py`. The `read_lifecycle` half is unchanged. ## Response-header compatibility `x-headroom-transforms` is built as `",".join(transforms_applied)`. A tag containing a comma (tool-name lists; file paths) would make that header ambiguous to split back into tags. To keep the header backward compatible, `header_safe_transforms` (`headroom/proxy/cost.py`) collapses the enriched tags back to their legacy counter shape **for the header only** -- the full enriched detail still flows through the structured `transforms_applied` list (dashboards, request logs, activity feed). Applied at all three header sites (openai / anthropic / gemini handlers). Paths containing `:` survive in `transforms_applied` because consumers bound their split to 3 parts. ## Tests - `tests/test_transforms/test_read_lifecycle.py` -- OpenAI + Anthropic tag shape, colon-in-path preservation - `tests/test_transforms/test_smart_crusher_attribution.py` -- OpenAI + Anthropic tool-name resolution, dedup, no-name fallback, id/name-missing skips - `tests/test_proxy/test_header_safe_transforms.py` -- header normalization keeps the joined header unambiguous (incl. comma-in-path) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
mbachaud
|
6367d0b722
|
feat(kompress): warn on unrecognized HEADROOM_KOMPRESS_BACKEND + document backend selection (#204)
## Summary
This PR was originally \"HEADROOM_KOMPRESS_BACKEND env + GPU/MPS
auto-detect\" (for #202). While it sat, main independently shipped the
backend-selection env var in
|
||
|
Patrick A
|
2ad300aff8
|
fix(transforms): use thread-local tree-sitter parsers to prevent pyo3 Unsendable panic (#604)
## Problem
pyo3 marks `_native::Parser` as `#[pyclass(unsendable)]`, which causes a
hard thread-assertion panic when a parser created on one thread is
accessed from another:
```
thread '<unnamed>' panicked at pyo3-0.28.3/src/impl_/pyclass.rs:1055:9:
assertion `left == right` failed: _native::Parser is unsendable, but sent to another thread
left: ThreadId(2)
right: ThreadId(1)
```
The prior implementation stored parsers in a module-level `dict[str,
Any]` (`_tree_sitter_languages`). When `_run_compression_in_executor`
dispatched compression work to a `ThreadPoolExecutor`, pool workers
grabbed parsers from that shared dict that were originally created on
the main asyncio thread and panicked.
This produces a 500 on every request where code compression is attempted
via a pool thread.
## Fix
Replace the global dict with `threading.local()` so each thread creates
and owns its own parser instances. No cross-thread parser access is
possible.
```python
# before
_tree_sitter_languages: dict[str, Any] = {} # shared — crosses threads
# after
_tree_sitter_local = threading.local() # per-thread — isolated
```
`is_tree_sitter_loaded()` and `unload_tree_sitter()` updated to operate
on the current thread's local cache (semantics unchanged for
single-threaded callers).
## Tests
9 regression tests added in
`tests/test_transforms/test_tree_sitter_thread_safety.py`:
- Thread isolation: two threads get distinct parser instances
- Within-thread reuse: same thread gets the same cached instance
- Thread pool: parsers usable from `ThreadPoolExecutor` workers without
panic
- Concurrent workers: each distinct pool thread owns a unique parser
- `is_tree_sitter_loaded` / `unload_tree_sitter` lifecycle
Also adds a `filterwarnings` entry for
`PytestUnraisableExceptionWarning`: pyo3 emits this when short-lived
test threads drop parsers at teardown; it does not occur in production
where pool threads are long-lived.
## Relation to #564
PR #564 proposes the same `threading.local()` approach but was blocked
on missing tests (`CHANGES_REQUESTED`). This PR includes the full test
suite.
|
||
|
Tejas Chopra
|
fc0cba7b48 | fix: format Kompress tests for ruff | ||
|
Tejas Chopra
|
a2ea9648a4 | fix: add Kompress backend and thread controls | ||
|
chopratejas
|
6aacd4805a |
fix: A9 — tag protector discards wrap on placeholder loss
When a placeholder is lost during compression, restore_tags now
discards the wrap rather than appending the original tag at the
trailing edge of the output. The old "append" fallback emitted
malformed XML — an opening tag with no body and no closing tag —
on ~350 production requests over 9 days. Per the proxy log
findings, the corruption pattern was `compressed-stuff <tag>`,
which downstream models interpret as a truncated message.
Concrete changes:
* `crates/headroom-core/src/transforms/tag_protector.rs`:
- `restore_tags` no longer accumulates `tail_appends`. Lost
placeholders are silently dropped from the output bytes.
- New `restore_tags_with_request_id` entry point threads an
optional request id into the structured ERROR log so the
proxy layer can wire request context end-to-end. PyO3 binding
keeps the existing 2-arg signature (no Python caller has a
request id today).
- `tag_lost_warn` is replaced by `tag_lost_error`. Severity
moves from WARN to ERROR with structured fields
(`event=tag_protector_placeholder_lost`, `tag_preview`,
`compressed_length`, `action=discarded_wrap`, optional
`request_id`) so operators can alert on the corruption rather
than have it disappear into a WARN line.
- `parse_tag_at` gained a bounds check after consuming a
leading '/' — proptest discovered an OOB on input `</`.
- The old `restore_lost_placeholder_appended` test (which
pinned the broken behavior) is replaced with three positive
tests: wrap-discard, idempotence on full loss, and
partial-loss-keeps-present-drops-lost.
- New proptest suite enforces three invariants over arbitrary
inputs: no introduced asymmetry, idempotence on full
placeholder loss, and no orphan-byte injection.
* `headroom/transforms/tag_protector.py`: docstring updated
to document the discard-wrap semantics — the prior text
("appended on the trailing edge") is now incorrect.
* `tests/test_tag_protector_invariant.py` (new): Python-side
invariant suite that exercises the same three properties
end-to-end through the public Python API. Uses a deterministic
seeded random walk (no `hypothesis` dependency) so CI is stable
and reproducible.
* `tests/test_transforms/test_tag_protector.py`: replaces the
broken-behavior test with the new wrap-discard semantics.
Per-finding-#3: ~/Desktop/HEADROOM_PROXY_LOG_FINDINGS_2026_05_03.md.
|
||
|
chopratejas
|
967b0db439 |
fix: B1 — retire ICM, RollingWindow, scoring, relevance + dependents
Phase B step 1 of the live-zone-only realignment. Removes ~10K LOC of
"drop messages from history" machinery that became unreachable after
PR-A1 made `/v1/messages` a passthrough on the proxy. Live-zone-only
compression (PR-B2..B7) operates on content blocks within messages;
message-list mutation no longer happens in the pipeline.
Python deletes:
- headroom/transforms/intelligent_context.py (1077 LOC)
- headroom/transforms/rolling_window.py (395 LOC)
- headroom/transforms/progressive_summarizer.py (508 LOC)
- headroom/transforms/scoring.py (459 LOC)
- headroom/transforms/tool_crusher.py (338 LOC)
- 5 corresponding tests/test_transforms/* and tests/test_proxy_intelligent_context.py
Rust deletes:
- crates/headroom-core/src/context/* (manager, config, workspace,
candidate, ccr_drop, strategy/, mod) + safety.rs replaced
- crates/headroom-core/src/scoring/* (mod, score, scorer, traits, weights)
- MessageScorerComparator from crates/headroom-parity (PR #338/#343
becomes deletable; sunk cost stays sunk)
- 13 message_scorer fixtures + record_message_scorer.py
Rust adds (move + rewrite):
- crates/headroom-core/src/transforms/safety.rs — `tool_pair_indices`
preserves the OpenAI/Anthropic tool_use ↔ tool_result pairing rule
the live-zone dispatcher (PR-B2) needs. No IcmConfig dependency.
Surface refactors:
- HeadroomConfig: drop `tool_crusher`, `rolling_window`,
`intelligent_context` fields; hoist `output_buffer_tokens` to top
level (used by client.py).
- ProxyConfig: drop `intelligent_context*` fields.
- `headroom wrap` proxy server: retire IntelligentContextManager
and RollingWindow imports + branch; pipeline is CacheAligner →
ContentRouter (smart_routing) or CacheAligner → SmartCrusher
(legacy).
- CLI: drop `--no-intelligent-context`, `--no-intelligent-scoring`,
`--no-compress-first` flags.
- LangChain memory integration: rename `_apply_rolling_window` →
`_apply_compression`, drop RollingWindowConfig dep. Threshold is
now advisory — B6 will rework the contract.
- TransformPipeline.create_pipeline now takes only cache_aligner_config.
- headroom/__init__.py + headroom/transforms/__init__.py: strip
exports of deleted symbols.
Bug fixes uncovered by full pytest sweep:
- providers/copilot/wrap.py: `environ or os.environ` collapsed
empty-dict to falsy → callers passing `environ={}` accidentally
pulled from os.environ. Use `environ if environ is not None else
os.environ`.
Test correctness fixes:
- _DummyAnthropicHandler._retry_request gains **_kwargs to match
the real handler signature post-A8.
- test_ws_http_fallback extracts JSON from `content=` (post-A3
byte-faithful) rather than the obsolete `json=` kwarg.
- test_ccr_response_handler_extra fixture joins SSE events with
`\n\n` per spec (post-A8 byte-buffer parser requirement).
- test_proxy_responses_phase_preservation: capture via direct
handler attached to the named logger, so the assertion is
order-independent (proxy `_setup_file_logging` flips
`headroom.propagate=False` once any earlier test triggers it).
- conftest.py autouse fixture resets `headroom.propagate=True`
before each test as a defensive measure for the same pollution.
- test_wrap_copilot_translated_backend_still_requires_byok:
monkeypatch.delenv every provider key so the BYOK error
actually fires.
- test_native_installers: skip when system bash < 4.3 (macOS ships 3.2).
- TestGeminiEmbedContent / TestGeminiBatchEmbedContents:
pytest.mark.skip — proxy currently has no :embedContent route;
feature gap, not regression.
Acceptance:
- cargo build --workspace + cargo clippy + cargo fmt --check: green.
- cargo test --workspace --exclude headroom-py: 777 passed.
- pytest: 4892 passed, 240 skipped, 0 failed.
- git grep returns only intentional comments referencing the deletion.
Per-PR-B1 plan: REALIGNMENT/04-phase-B-live-zone.md.
|
||
|
chopratejas
|
704fb2f19d |
fix: A2 — system prompt immutable; memory routes to live-zone tail; cache_aligner detector-only
P0-1: Delete `_inject_system_context` from `proxy/server.py`. Memory context now routes exclusively to the first text block of the latest non-frozen user message via `_append_context_to_latest_non_frozen_user_turn` (promoted to the canonical default in handlers/anthropic.py). Mirror applied to OpenAI Responses API at handlers/openai.py: `body["instructions"]` is no longer mutated; memory context appends to the latest user item in `body["input"]`. P2-23: Replace `headroom/transforms/cache_aligner.py` with a detector-only implementation. The legacy rewrite path (~400 LOC) is removed. The volatile- content detector uses no regex — UUIDs via `uuid.UUID`, ISO 8601 via `datetime.fromisoformat`, JWT shape via base64url segment-count check, hex hashes via length + `int(token, 16)` validation. Volatile findings surface through `cache_metrics`/`warnings`/`logger.warning`; the prompt is never mutated. Configurability: new env var `HEADROOM_MEMORY_INJECTION_MODE` with values `live_zone_tail` (default) and `disabled`. No `system_prompt` value — that path is permanently retired. Structured logs: every memory injection emits `event=memory_injection` with `decision`, `bytes_injected`, `query_hash` (BLAKE2b, never raw query), `session_id`, `request_id`. Auth is never logged. Tests: - Add `tests/test_proxy_system_prompt_immutable.py` (7 tests). - Add `tests/test_cache_aligner_detector_only.py` (20 tests). - Replace `tests/test_transforms/test_cache_aligner.py` (rewrite-path tests, 58 cases) with detector-only behavior. - Update `tests/test_acceptance.py::TestDateTrap` to pin the new detector-only contract. Acceptance: - `git grep -n "_inject_system_context\|_inject_to_system_or_instructions" headroom/` returns nothing. - `git grep -n "import re\|from re import" headroom/transforms/cache_aligner.py` returns nothing. - Targeted suite (`test_proxy_system_prompt_immutable.py`, `test_cache_aligner_detector_only.py`, `test_proxy_anthropic_cache_stability.py`, `test_acceptance.py::TestDateTrap`, `test_memory*.py`, `test_cli/`) green. |
||
|
chopratejas
|
c9aaba3f5b |
feat(rust): port tag_protector to Rust + 5 bug fixes (Phase 3e.4)
`headroom/transforms/tag_protector.py` was a regex-driven scan-and-
replace loop that ran on every kompress call from ContentRouter
(`content_router.py:1089`). The Python implementation had five real
bugs we now fix in the port — the most consequential being a
`str.replace(.., .., 1)` first-occurrence-replace bug that silently
collapsed two identical custom-tag blocks in the same input to a
single placeholder + a stray duplicate of the second block.
# Bug fixes (each pinned by a `fixed_in_3e4` test)
* **#1: O(n²) on nested custom tags.** Python's `while changed` loop
restarted a full regex scan after every replacement. Rust walks
once in linear time on input length.
* **#2: First-occurrence replace bug.** `result.replace(orig, ph, 1)`
replaces the FIRST textual match, not the matched offset. Two
identical custom-tag blocks collapsed to one placeholder + a stray
duplicate of the second block. The Rust walker stitches output by
offset so distinct blocks always get distinct placeholders.
* **#3: Silent 50-iteration cap.** Python had a hard `max_iterations
= 50` safety limit that quietly truncated tag protection on deeply
nested input. The Rust walker is bounded by input length only.
* **#4: Self-closing pass duplicate-replace risk.** Python ran a
second loop with the same `replace_first` bug for self-closers.
Rust handles self-closers in the same single pass.
* **#5: Placeholder collision.** If the input contained a literal
`{{HEADROOM_TAG_…}}` substring, Python silently let the collision
break restoration. Rust salts the prefix and reports it in stats.
# Architecture
Two-phase walker:
* Phase 1 (`identify_spans`): linear scan over input bytes, hand-
rolled tag-open / tag-close lexer (no regex). Maintains a stack of
open custom tags; on a matching close, collapses the inner span
into a single `Span { start, end, Block }`. Self-closing custom
tags become `Span { ..., SelfClosing }` immediately. Marker-only
mode (`compress_tagged_content=true`) emits Open/CloseMarker spans
instead. Orphan opens stay un-protected (matches Python behavior).
Orphan closes are emitted verbatim and counted in stats.
* Phase 2 (`emit_output`): walks `text` once, splicing placeholders
for span ranges and copying everything else verbatim. Offset-based,
never `str.replace`.
PyO3 surface: `protect_tags`, `restore_tags`, `is_html_tag`,
`known_html_tag_names`. The Python shim retires the regex internals
and re-exports `KNOWN_HTML_TAGS` (rebuilt from the Rust list) +
`_is_html_tag` for backwards compat with `content_router.py` and the
existing test surface.
# Test plan
* 25 Rust unit tests including 4 `fixed_in_3e4_*` bug-fix tests
* 27 Python tests (23 existing + 4 new `fixed_in_3e4` parity tests)
* 5 integration tests in `test_tag_protection_integration.py` pass
* `make ci-precheck` clean
|
||
|
chopratejas
|
da7716a95a |
chore(rust): SmartCrusher CCR marker injection + walker unification
Closes four gaps in the Rust SmartCrusher pipeline that, together,
wire CCR storage end-to-end so the LLM can actually retrieve dropped
data:
1. CCR-Dropped marker is now injected into process_value's lossy-path
output as a sentinel object {"_ccr_dropped": "<<ccr:HASH N_rows_offloaded>>"}
appended to the kept-items array. Previously the store held the
original but no pointer reached the prompt -- the retrieval contract
was data-on-server, no-way-to-ask. Sentinel-as-object preserves the
array-of-dicts shape so downstream iteration with x.get(...) keeps
working.
2. Walker / process_value drift removed. process_value gains a
Value::String arm that handles stringified-JSON containers (parse,
recurse, re-encode) and opaque blobs (CCR marker + store) -- same
semantics walker.rs has always had, now reachable from the main
crush() pipeline.
3. Opaque-string CCR now stores originals. DocumentCompactor gains an
Option<Arc<dyn CcrStore>> field; emit_opaque_ccr_marker calls
store.put when one is configured. Same hash regardless of store
presence -- runtime contract is stable across configurations.
Same wiring is shared between walker.rs and process_value via the
extracted helper.
5. PyO3 surface adds SmartCrusher.compact_document_json(doc_json) ->
compacted-json string. Routes through the crusher's existing CCR
store, so ccr_get resolves both row-drop and opaque-string hashes.
Tests:
- 5 new Rust integration tests in ccr_roundtrip.rs (marker visibility,
nested-array marker, opaque-string roundtrip, stringified-JSON
recursion, walker-with-store)
- 4 new Python tests covering the marker visible-to-LLM contract via
both the native PyO3 surface and the Python shim
- 5 legacy parity fixtures re-recorded (dict_array_*, duplicate_dicts_40)
-- their lossy outputs now carry the sentinel; Rust + Python both
match the new bytes (parity-run smart_crusher: 17/17)
|
||
|
chopratejas
|
22c8fec4c1 |
chore(rust): SmartCrusher CCR storage layer + roundtrip verification
CcrStore trait + InMemoryCcrStore (1000 entries, 5-min TTL, FIFO eviction, idempotent re-store) live at the crate root. SmartCrusher's lossy crush_array path now actually stashes the full original [items] canonical-JSON into the configured store keyed by the same ccr_hash it embeds in the prompt marker -- closing the no-data-loss contract that was previously hash-only. PyO3 surface: - crusher.crush_array_json(items_json) -> dict with ccr_hash + kept items - crusher.ccr_get(hash) -> Optional[str] for retrieval - crusher.ccr_len() -> int for telemetry Python shim passes both through. Default constructors enable the store (matches Python's CCR-enabled default); without_compaction() also gets it because CCR is a contract, not an opt-in extra. Tests proving compress -> store -> retrieve -> reconstruct: - 7 unit tests in ccr.rs (put/get/eviction/expiry) - 9 Rust integration tests (crates/headroom-core/tests/ccr_roundtrip.rs) - 10 Python tests including 4 explicit before/after element-equality assertions through both the native PyO3 surface and the Python shim Plugin manifest versions auto-bumped by the sync-plugin-versions pre-commit hook (unrelated to CCR but co-resident in the working tree). |
||
|
chopratejas
|
1601591900 |
feat(rust): SmartCrusher PR4 — lossless-first default + CCR-Dropped restoration
Stage 3c.2 PR4. Restores Python's CCR-Dropped semantics on the lossy
path (the cornerstone reversibility guarantee that the port had
silently dropped) and flips the OSS default to lossless-first with a
configurable savings threshold.
# The user-visible behavior
Default `SmartCrusher::new()` now runs:
1. Try lossless compaction.
2. If savings >= `lossless_min_savings_ratio` (default 0.30), ship
it — `compacted` populated, `ccr_hash = None`, nothing dropped.
3. Otherwise fall through to the lossy path — drop rows AND
populate `ccr_hash` so the runtime can cache the full original
for tool-call retrieval.
**No data is ever lost.** "Lossy" means "compressed view inline; full
payload retrievable via CCR cache" — same semantics as Python's
SmartCrusher with CCR enabled. The runtime (PyO3 bridge / proxy
server) owns the cache; this crate computes the hash and emits a
marker so the prompt knows where to look.
# What changed
- `SmartCrusherConfig.lossless_min_savings_ratio: f64` (default 0.30).
Single configurable knob — Enterprise overrides as needed. Below
the threshold, lossless declines and lossy + CCR runs.
- `SmartCrusher::new(cfg)` flips to include the compaction stage by
default. `SmartCrusher::without_compaction(cfg)` is the explicit
opt-out for callers / fixtures that depend on pre-PR4 behavior.
- `crush_array` rewritten:
- Lossless-first dispatch with savings-ratio gate
- Lossy path now hashes the full original (12-char SHA-256 prefix)
and emits a CCR-Dropped marker in `dropped_summary` whenever
rows are dropped
- `ccr_hash` field populated whenever rows were dropped
- `process_value` substitutes the compacted string into the JSON
tree when lossless wins, so `crush()` output reflects the win
- PyO3 bridge: `SmartCrusher.without_compaction()` static method;
`SmartCrusherConfig` exposes the new `lossless_min_savings_ratio`
field; Python `SmartCrusher` wrapper accepts `with_compaction=True`
(default) and routes to the right Rust constructor.
- Parity harness: legacy 17 fixtures use `without_compaction()` so
byte-equal coverage of the lossy path is preserved.
# Tests
- Rust: 281/281 smart_crusher unit tests pass (was 277). Six new
tests cover: lossless wins above threshold, lossy falls through
below threshold, CCR hash deterministic + input-dependent, lossy
without compaction emits CCR, passthrough paths don't emit CCR,
without_compaction yields no compacted field.
- Python parity: 21/21 (legacy fixtures via without_compaction).
- Python lossless default smoke: 3/3 new tests in
test_smart_crusher_lossless_default.py.
- Python retention: 21/21 (updated to opt into the lossy path
explicitly since their semantics target row-level retention).
- make ci-precheck green.
Modules:
crates/headroom-core/src/transforms/smart_crusher/{config,crusher}.rs
crates/headroom-parity/src/lib.rs
crates/headroom-py/src/lib.rs
headroom/transforms/smart_crusher.py
tests/test_quality_retention.py
tests/test_transforms/test_smart_crusher_{lossless_default,rust_parity}.py
|
||
|
chopratejas
|
c765c53bf8 |
feat(rust): retire python smart_crusher, ship rust-only via pyo3
Stage 3c.1b step 2 + cleanup. The python `SmartCrusher` (3669 lines) is replaced by a thin pyo3-backed shim (~290 lines) that delegates every byte to `headroom._core.SmartCrusher` (built from `crates/headroom-py`, landed in the previous commit). There is no python implementation and no env-var fallback — the wheel is a hard import. Why now: parity was already proven across 17 fixtures + the python- side bridge test (1+17 in `test_smart_crusher_rust_parity.py`). Keeping a shadow python impl behind a flag is a permanent maintenance cost with no operational benefit. Stage 3c.1b deletes ~3380 lines of python parser/scorer/analyzer/orchestrator code; the rust crate has its own coverage (388 unit tests + property tests in headroom-core). Surface preserved (drop-in for every production caller): - `headroom.transforms.smart_crusher.SmartCrusher` — same class name, same `__init__(config, relevance_config, scorer, ccr_config)` signature (the latter three are accepted for source-compat and silently dropped — rust port keeps those subsystems disabled in Stage 3c.1, they re-attach in Stage 3c.2). - `SmartCrusherConfig` and `CrushResult` dataclasses kept as python dataclasses (callers use `asdict()` / dataclass matching on them). - `crush(content, query, bias)`, `_smart_crush_content(content, ...)`, `apply(messages, tokenizer, **kwargs)`, and `_extract_context_from_messages(messages)` all preserved. - `smart_crush_tool_output(content, config, ccr_config)` thin wrapper. The transform-protocol `apply()` orchestration stays python (message walking, digest-marker insertion, token counting); only the per- message compression call delegates to rust. Removed: - Python parser / planner / scorer / analyzer / classifier (~3380 lines). - Internal helpers `_classify_array`, `_detect_sequential_pattern`, `_detect_rare_status_values`, `_detect_items_by_learned_semantics`, `_percentile_linear`, `_compute_k_split`, `_crush_number_array`, `_process_value`, etc. — rust crate has parallel coverage. - `SmartAnalyzer`, `ArrayType`, `CompressionStrategy`, `extract_query_anchors` — internals; not used by any production caller (only tests probed them). Tests deleted (probed deleted internals — same precedent as Stage 3b): - `tests/test_transforms/test_smart_crusher.py` (40 tests) - `tests/test_transforms/test_universal_json_crush.py` (45) - `tests/test_transforms/test_anchor_selector.py` (49) - `tests/test_toin_field_learning.py` (21) - `tests/test_crushability.py` (20) Tests trimmed (removed methods/classes that probe deferred subsystems — scorer injection, CCR marker injection, TOIN feedback recording — all of which re-attach in Stage 3c.2): - `tests/test_transforms/test_smart_crusher_bugs.py`: TestNumberArraySchemaPreservation, TestStage3c1BugFixes. - `tests/test_relevance.py`: 2 scorer-injection tests. - `tests/test_ccr.py`: TestSmartCrusherCCRIntegration class + test_custom_marker_template. - `tests/test_toin_integration.py`: TestTOINIntegration + TestStoreToTOINHash classes. - `tests/test_critical_fixes.py`: TestSmartCrusherTOINIntegration + test_full_feedback_loop. - `tests/test_acceptance.py::TestQueryAnchorExtraction`: dropped the `extract_query_anchors` probe; kept the end-to-end "Alice preserved" assertion. Bug fixes from Stage 3c.1 (#1 percentile linear interp, #2 zero- padded sequential, #3 rare-status pareto, #4 k-split overshoot) are pinned by the rust crate and the parity fixtures (`tests/parity/fixtures/smart_crusher/`). Tests: - 517 passed in the smart_crusher-adjacent file set (test_transforms/, test_relevance*, test_ccr, test_toin_integration, test_quality_retention, test_acceptance, test_critical_fixes). - 18 in `test_smart_crusher_rust_parity.py` (1 sanity + 17 fixtures). - 388 rust unit tests still green. One stale-error-message regex in `test_relevance_extra.py` updated from "requires sentence-transformers" → "requires fastembed". |
||
|
chopratejas
|
5328d87b1e |
feat(rust): pyo3 bridge for SmartCrusher
Stage 3c.1b step 1: expose `SmartCrusherConfig`, `CrushResult`, and `SmartCrusher` to Python via `headroom._core`. The Python shim that delegates to it (replacing the 3669-line Python implementation) lands in the next commit; this commit just builds the bridge and a fixture-replay test that pins it. Surface: - `headroom._core.SmartCrusherConfig(**fields)` — every field of the Rust `SmartCrusherConfig` exposed as a kwarg with matching default. - `headroom._core.CrushResult` — read-only mirror of the Rust struct with `compressed`, `original`, `was_modified`, `strategy` getters. - `headroom._core.SmartCrusher(config=None)` — constructor accepts only `config`; the Python shim drops `relevance_config`, `scorer`, and `ccr_config` since Stage 3c.1 keeps those subsystems disabled. - `crush(content, query="", bias=1.0)` and `smart_crush_content(...)` methods mirror the Python signatures. Verification: - All 17 recorded parity fixtures byte-equal between Python and the PyO3 bridge (`tests/test_transforms/test_smart_crusher_rust_parity.py`, 18 tests pass — 1 fixture-count sanity + 17 fixtures). - The Rust-side `cargo run -p headroom-parity --bin parity-run -- run --only smart_crusher` was already 17/17 green. The two tests catch different regression classes: - Rust-only test: catches drift in the Rust port's logic. - Python bridge test: catches PyO3 input/output translation bugs. |
||
|
chopratejas
|
c829dfa539 |
fix(python+rust): smart_crusher bugs #1, #2, #3, #4 + sorted iteration
Lockstep fixes for the four known bugs in headroom/transforms/smart_crusher.py plus the field-iteration ordering parity fix. Both languages now agree byte-for-byte on the affected code paths — prerequisite for parity fixtures landing next. Bug #1 — percentile off-by-one (Python line 2844 + Rust crushers.rs) Replaces integer-division indexing with linear-interpolation percentile (numpy "linear" method). New _percentile_linear helper shared by both languages: index = q * (n - 1), interpolate between floor and ceil. Bug #2 — zero-padded string IDs misclassified as sequential Track had_non_string_numeric flag; if every parseable value came from a string (no actual int/float), return False (categorical, not sequential). Pre-fix: int("001") loses zero-padding and fakes a sequential pattern. Bug #3 — rare-status detection cardinality cap Cardinality cap raised from 10 to 50. Single-dominant check replaced with Pareto top-K: smallest K such that top-K covers >=80% of items. If K <= 5, items NOT in top-K are outliers. Catches bimodal distributions like 60×INFO + 25×WARN + 15 distinct error codes. Bug #4 — k-split overshoot when k_total=1 Clamp after the floored fractions: k_first=min(k_first, k_total), k_last=min(k_last, max(0, k_total - k_first)). No-op for the common case k_total >= 2. Field iteration ordering (Python line 1049) `for key in all_keys` → `for key in sorted(all_keys)`. Set iteration is non-deterministic across PYTHONHASHSEED; downstream short-circuits in _select_strategy and _detect_pattern would pick different fields between runs. Rust uses BTreeMap (sorted ASCII); sorting Python locks both languages to the same iteration order. Verification: - 56 Python tests pass (51 existing + 5 new lockstep tests under TestStage3c1BugFixes class). - 382 Rust tests pass (rust bug #1 documentation test replaced with two new "fixed behavior" tests). - Clippy clean. Status: all four bugs are now fixed in BOTH languages. Parity fixtures can be recorded against post-fix Python and asserted byte-equal against Rust. That's the next commit. |
||
|
chopratejas
|
f5f465418b |
feat(rust): retire python diff_compressor, ship rust-only via pyo3
The python `DiffCompressor` is replaced by a thin pyo3-backed shim that delegates to `headroom._core.DiffCompressor`. There is no python implementation and no env-var fallback — the wheel is a hard import. Why now: opt-in defaults don't drive python retirement. Byte-equal parity was already proven across 27 fixtures (stage 3a); keeping a shadow python impl behind a flag is a permanent maintenance cost with no operational benefit. Stage 3b deletes ~700 lines of python parser / scorer / formatter code; the rust crate has its own coverage. Surface preserved: - `headroom.transforms.diff_compressor.DiffCompressor` — same class name, same `__init__`, same `compress(content, context)` shape. Returns python `DiffCompressionResult` dataclasses so call sites that destructure with `asdict()` work unchanged. - `DiffCompressorConfig` and `DiffCompressionResult` dataclasses kept. - Sidecar `compress_with_stats(...)` exposes the rust-only `DiffCompressorStats` (per-file hunk drops, context lines trimmed, file_mode normalizations) for observability. Removed: - Python parser / scorer / formatter (~700 lines). - Internal `DiffHunk` / `DiffFile` parser dataclasses (rust crate has parallel coverage). - 12 tests that probed `_parse_diff` / `_score_hunks` or the deleted parser dataclasses. The 29 public-API tests in `test_diff_compressor.py` remain and now exercise the rust backend through the same import path. - `HEADROOM_DIFF_COMPRESSOR_BACKEND` env var — no longer meaningful. Build: - `scripts/build_rust_extension.sh` runs `maturin develop` and symlinks the built `.so` into `headroom/` so `import headroom._core` resolves past the in-tree package shadowing the maturin overlay. - `.gitignore` excludes the symlinks and allowlists the build script. Tests: - 544 transforms pass; 33 rust-vs-fixture parity tests guard the pyo3 bridge against regressions (`tests/test_transforms/test_diff_compressor_rust_parity.py`). - Mypy clean. |
||
|
chopratejas
|
48c13245c5 |
fix(diff): close ContentRouter routing gaps for merge diffs and long preambles
User audit caught three gaps that prevented DiffCompressor from being
invoked even when the input was a real diff. These complement the four
emit-time bugs fixed in the previous commit — those fixes only kick in
once DiffCompressor receives the input. Without these gap fixes, real
merge-commit diffs and `git log -p` outputs with long commit messages
were misrouted away from DiffCompressor entirely.
# The three gaps (each fixed in Python; gap 3 also fixed in Rust)
1. Detector scan window was hardcoded to first 50 lines.
`_try_detect_diff` in content_detector.py only inspected
`content.split("\n")[:50]`. `git log -p` outputs commonly have
commit messages longer than 50 lines (releases, squashed commits,
bots), pushing the `diff --git` header out of the detection window.
Result: input was returned with `content_type=PLAIN_TEXT` and routed
to the text compressor, never reaching DiffCompressor. Fix: window
widened to 500 lines.
2. Detector regex didn't recognize merge-commit headers.
`_DIFF_HEADER_PATTERN` matched `diff --git`, `--- a/`, and the
regular `@@ -A,B +C,D @@` hunk header. Merge-commit diffs from
`git log -p` use `diff --combined <path>`, `diff --cc <path>`, and
combined-diff hunk headers `@@@+`. The shared `--- a/` line still
triggered the detector with low confidence, but only barely. Fix:
extended the regex to recognize all four merge-shaped header forms.
3. DiffCompressor parser only matched `^diff --git`.
Even after fixing detection, the parser's `_DIFF_GIT_PATTERN`
wouldn't match `diff --combined` or `diff --cc`, so merge diffs
reached DiffCompressor and were treated as one giant pre-diff blob —
passed through unchanged after the previous PR's pre-diff
preservation fix. Fix: added `_DIFF_COMBINED_PATTERN` and
`_DIFF_CC_PATTERN`; `_parse_diff` starts a new file section on any
of the three header forms. Mirrored in Rust as `is_diff_header`
helper that checks all three regexes.
# Why this matters end-to-end
DiffCompressor's value comes from being routed to. Detection +
parser-level coverage are upstream of the compressor — without them,
the compressor never sees the input. The previous PR's four bug fixes
(rename, combined-diff hunks, no-newline marker, pre-diff content) are
correct and necessary, but for merge commits and long-preamble diffs,
they were only firing on the rare cases where the detector misclicked
into DiffCompressor anyway. With these three gaps closed, the
ContentRouter→DiffCompressor pipeline actually engages on:
- `git log -p` outputs of any commit-message length
- Merge-commit diffs (`diff --combined`, `diff --cc`)
- Combined-diff snippets (`@@@`+ hunk-only inputs)
# New fixtures (3 added to the existing 24)
- `066bc82…` — `diff --combined` merge diff (3-way)
- `5d950a94…` — `diff --cc` merge diff (alternate form)
- `66c86f64…` — long pre-diff content (60-line commit message)
followed by a rename diff (exercises detector scan widening +
pre-diff preservation in tandem)
Parity: total=27 matched=27 skipped=0 diffed=0.
# Tests
- Python: 4 new tests across 2 new test classes —
`TestRoutingGapMergeDiffs` (combined / cc parser) and
`TestRoutingGapDetectorScanWindow` (long preamble detection +
combined-diff regex recognition).
- Rust: 2 new tests covering combined / cc parser sections.
# Verification
- 27/27 parity fixtures byte-equal.
- Python: 41/41 tests pass (was 37).
- Rust: 18/18 transforms tests; 62/62 workspace; 5/5 proptests.
- `cargo fmt --check` clean; `cargo clippy --workspace --all-targets
-- -D warnings` clean.
|
||
|
chopratejas
|
6d47a0cd00 |
fix(diff_compressor): four silent information-loss paths in Python AND Rust
Audit caught four bugs that the byte-equal parity harness can't catch on its own — both Python and Rust were faithfully emitting the buggy output. Fixed in lockstep so parity is maintained while the underlying behavior is now correct on inputs the existing 20 fixtures didn't exercise. # The four bugs (each fixed in both Python and Rust) 1. Renames silently dropped from output. Parser captured `is_renamed=True` but the emitter never emitted ANY rename markers. Output of a rename looked exactly like a plain modification of the old path. Fix: capture `rename from` / `rename to` / `similarity index N%` / `dissimilarity index N%` / `copy from` / `copy to` lines in a new `rename_lines` field on `DiffFile`; emit them after `diff --git` in canonical git ordering. 2. Combined diff hunks (`@@@`) silently dropped. Hunk-header regex only matched `@@`, so 3-way merge hunks had `current_hunk` never set and ALL their content fell through to the no-op branch. Fix in Python: regex switched to `^(@@+) ... \1` (backreferences match any number of `@`s on each side). Fix in Rust: alternation over `@@`, `@@@`, `@@@@` since `regex` is RE2-based and rejects backreferences. n>3 octopus merges still fall through; rare in practice. 3. `\ No newline at end of file` markers can be context-trimmed away. Treated as ordinary "other" lines — if more than `max_context_lines` from a `+`/`-` change, dropped. Round-trip-breaking for patches; can change whether the trailing line has a newline. Fix: in `_reduce_context`, force-add any line starting with `\` to the keep set regardless of distance. 4. Pre-diff content silently dropped. Anything before the first `diff --git` — commit messages from `git log -p`, email headers from `git format-patch`, fork-and-rebase metadata — was discarded. Fix: `_parse_diff` now returns `(pre_diff_lines, files)`; `format_output` prepends pre-diff content verbatim when present. # Hidden parity bug found during the work `_compress_files` constructed a fresh `DiffFile` from the parsed one but only copied a subset of the fields by name. The new `rename_lines` and `original_*_line` fields were silently dropped here, so the parser populated them correctly but the emitter saw an empty `rename_lines` list. Caught by writing a real test instead of a smoke test — the smoke test passed because it hit the no-diff-found short-circuit, not the parser/emitter pipeline. Constructor now copies all fields explicitly. # Parity status - Existing 20 fixtures: still byte-equal between fixed Python and fixed Rust. None of them exercised the buggy paths. - 4 NEW fixtures recorded against fixed Python, exercising each bug-fix path: rename, 3-way combined diff, `\ No newline` marker far from changes, pre-diff commit headers. All 4 byte-equal between Python and Rust. - Parity harness: total=24 matched=24 skipped=0 diffed=0. # Observability Some normalizations remain parity-bound (file mode `100644` hardcode, `Binary files differ` simplification). Those are surfaced in `DiffCompressorStats::file_mode_normalizations` / `binary_files_simplified` (Rust) and via `logger.warning` (Python's new `_log_loss_signals` helper, called once per compress). # Tests - Python: 4 new test classes (11 tests) covering rename markers, combined diffs, no-newline preservation, pre-diff content. Edge case: no pre-diff content must NOT add a leading blank line. - Rust: 4 new `bugfix_*` unit tests with the same scenarios. - Existing Python tests calling `_parse_diff` directly were updated for the new `(pre_diff, files)` tuple return. # Verification - Python: 37/37 tests pass (was 26). - Rust: 16/16 transforms tests; 60/60 workspace unit tests; 5/5 proptests; 1/1 doctest. - Parity: 24/24 byte-equal. - `cargo fmt --check` clean; `cargo clippy --workspace --all-targets -- -D warnings` clean. |