mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
35 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Ashish
|
6469fcd018
|
feat(transforms): language-aware stack-trace collapse for Go/Rust/.NET/Java/Node (#1791)
## Description
Stack-trace handling covered only Python tracebacks and a generic ` at
symbol(` pattern. Go panics and Rust panics flowed to prose compression;
.NET traces were unrecognized; Java chained exceptions split into
separate traces at every `Caused by:` (so later chain heads fell off the
`max_stack_traces` cliff); and oversized traces were blindly
head-truncated — keeping runtime scheduler noise while dropping the app
frames and chain heads an agent actually needs.
This PR adds language-aware trace flavors (Go, Rust, .NET, Java chains,
Node async) to the Rust core and both Python mirrors, and replaces blind
truncation with a runtime-frame collapse: message lines, chain heads,
the trace head, and app-code frames survive; contiguous runtime/stdlib
frames fold into `[... N frames collapsed]` markers. A 147-line Go panic
dump compresses to 19 lines with the panic message, signal line, and app
frame intact.
Note one intentional behavior change: now that Go/Rust panics are
*detected*, panics ≤8KB in tool outputs gain the existing error-output
protection (`protect_error_outputs`) they previously missed — small
panics stay verbatim, exactly like small Python tracebacks already do.
## Type of Change
- [x] New feature (non-breaking change which adds functionality)
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring
## Changes Made
- `crates/headroom-core/src/transforms/log_compressor.rs`:
- New `TraceFlavor::GoPanic` (`panic:` / `fatal error:` / `goroutine N
[state]:` openers, tab-indented `.go:` frame lines, `created by` /
call-line continuation, blank-separated goroutine blocks) and
`TraceFlavor::DotNet` (`Unhandled exception.`, `at … ) in file:line N`
frames — checked before Java since those frames also satisfy the Java
shape — plus `--->` inner-exception heads and `--- End of` separators).
- Renamed the misnamed `Go` flavor to `RustBacktrace` (its `is_go_frame`
matched `N: 0x<hex>` — the Rust backtrace shape) and gave it real
openers (`thread '…' panicked at`, `stack backtrace:`); the free-text
panic-message line after the opener stays in the trace (`terminates` now
receives `lines_so_far`).
- Java: continues across `Caused by:` / `Suppressed:` / `... N more`,
and `is_java_at_frame` admits `/` so JPMS module frames (`at
java.base/…`) pass the opener re-check — without this, modern JDK traces
fragmented at the parse cap into ≤20-line groups.
- Frame collapse (`collapse_trace_frames`): for traces over
`stack_trace_max_lines`, keeps message/chain-head lines, first
`trace_head_frames` frames, up to `trace_app_frames` app frames; runtime
frames (prefix + path marker tables per language) fold into `[... N
frames collapsed]` markers that occupy the run's first line slot and
carry score 0.8 so the global cap doesn't drop them first. Collapsed
frame indices are excluded from the context-line pass (otherwise ±3
context re-added them), and the parse cap re-opens on continuation lines
so selection sees one contiguous trace. New config:
`collapse_runtime_frames=true`, `trace_head_frames=3`,
`trace_app_frames=5`; new sidecar stat `runtime_frames_collapsed`.
- `crates/headroom-py/src/lib.rs`: the three new knobs on the
`LogCompressorConfig` PyO3 signature.
- `headroom/transforms/log_compressor.py`: dataclass fields +
constructor pass-through; `_parse_lines` opener patterns mirrored per
the documented contract.
- `headroom/transforms/content_detector.py`: `_LOG_PATTERNS` additions
(Go panic/goroutine/frame lines, Rust panic/backtrace/numbered frames,
.NET, Java chain heads, Node `at async`); JS/Java `at` pattern admits
JPMS module paths.
- `tests/test_transforms_stack_traces.py` (new, 10 tests) + 7 new Rust
unit tests (flavor open/continue/terminate, chain grouping, collapse
keeps chain heads/app frames, collapse-off comparison, small traces
untouched).
## Testing
- [x] Added new tests for the changes
- [x] All existing tests pass
### Test Output
```
$ cargo test -p headroom-core
928 passed; 3 ignored
$ python -m pytest tests/test_transforms_stack_traces.py tests/test_log_compressor.py \
tests/test_transforms_log_compressor.py tests/test_transforms_content_detection.py \
tests/test_transforms_content_router.py tests/test_transforms/test_content_router.py \
tests/test_lossless_mode.py tests/test_compression_fidelity_regression.py -q
191 passed
```
## Real Behavior Proof
- Environment: macOS arm64, Python 3.13, repo main @
|
||
|
Tejas Chopra
|
5771a8020e
|
fix(deps): remediate dependency CVEs and publish SBOM (#1509)
## Description
Supply-chain hardening: takes the **shipped** dependency surface from
**26 known CVEs to 0**. `pip install headroom-ai[all]` now resolves with
no known vulnerabilities (verified with Anchore syft + grype). Also
publishes a checked-in SBOM package (`sbom/`) so any user — especially
pilots running their own security review — can verify what's inside and
that we track it.
This addresses the Dependabot alerts on `main` (9 high / 4 moderate / 7
low at time of writing).
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
**Rust**
- `pyo3` 0.24 → 0.29 (GHSA-36hh-v3qg-5jq4 High, GHSA-chgr-c6px-7xpp
Med). Migrated `Python::allow_threads` → `Python::detach` (10 sites) and
added `from_py_object` to the `Clone`-deriving `#[pyclass]` types (both
required by the 0.25+ API).
- `pyo3-log` 0.12 → 0.13; `lru` 0.12 → 0.18 (GHSA-rhfx-m35p-ff5j).
**Python**
- `torch` → 2.12.1, `mem0ai` → 2.x.
- Floor-pinned transitive CVE deps via `[tool.uv]
constraint-dependencies`: `pygments>=2.20.0`,
`pydantic-settings>=2.14.2`, `gitpython>=3.1.50`, `langsmith>=0.9.0`.
- **Removed `benchmark` from the `[all]` aggregate** so the default
install is CVE-free. `lm-eval` is invoked as an external subprocess
(`python -m lm_eval`) and never imported, so it is not a true runtime
dep — it remains available via the opt-in `[benchmark]` extra. See
[Accepted Risks](#additional-notes).
**npm (build/test tooling — never shipped in the
wheel/container/published SDK)**
- `esbuild` override `>=0.28.1` in `sdk/typescript` + `plugins/openclaw`
(GHSA-g7r4-m6w7-qqqr).
- `docs/`: `@anthropic-ai/sdk` → `^0.106.0` (GHSA-p7fg-763f-g4gf),
`postcss` override to force Next.js's bundled copy ≥8.5.10
(GHSA-qx2v-qp2m-jg93); regenerated a stale `bun.lock` that carried a
**Critical** vitest/vite.
**CI**
- Pinned `pypa/gh-action-pypi-publish` `@release/v1` → `@v1.13.0`
(GHSA-vxmw-7h4f-hqxh) in `release.yml` + `publish.yml`.
**SBOM**
- New `sbom/` directory: CycloneDX 1.7 + SPDX 2.3 SBOMs, grype scan
evidence, 330-package license inventory, and a regeneration guide.
## Testing
- [ ] Unit tests pass (`pytest`) — N/A, no Python source changed
(deps/config only)
- [x] Linting passes — `cargo fmt --check` + `cargo clippy` clean on the
changed crate; 0 `.py` files changed so `ruff`/`mypy` scope is
unaffected
- [x] Type checking passes — `cargo check --workspace` (0 errors)
- [ ] New tests added — N/A (dependency bumps; covered by existing
suites)
- [x] Manual testing performed — see Real Behavior Proof
### Test Output
```text
# headroom-ai[all] product surface — the number that matters
$ grype sbom:sbom/headroom-sbom-all-extra.cdx.json
No vulnerabilities found
# full repo scan (universal lock incl. opt-in [benchmark] + dev)
$ grype sbom:sbom/headroom-sbom.cdx.json
NAME INSTALLED TYPE VULNERABILITY SEVERITY
sqlitedict 2.1.0 python GHSA-g4r7-86gm-pgqc High # [benchmark]-only, unpatchable, accepted
nltk 3.9.4 python GHSA-p4gq-832x-fm9v High # [benchmark]-only, unpatchable, accepted
# pyo3 0.29 migration — extension builds + imports + runs
$ cargo check --workspace
Finished `dev` profile [unoptimized + debuginfo] target(s)
$ maturin develop && python -c "from headroom._core import DiffCompressor, SmartCrusher; ..."
extension OK — detach + from_py_object paths exercised
# lru 0.18 — eviction path
$ cargo test -p headroom-proxy --lib drift
14 passed, 213 filtered out
# per-ecosystem npm audits
$ (cd sdk/typescript && npm audit) -> found 0 vulnerabilities
$ (cd plugins/openclaw && npm audit) -> found 0 vulnerabilities
$ (cd docs && npm audit && bun audit) -> found 0 vulnerabilities / No vulnerabilities found
```
## Real Behavior Proof
- Environment: macOS (darwin 25.4.0, arm64), Python 3.12 `.venv`, Rust
1.95 toolchain, syft 1.46.0, grype 0.115.0, bun 1.3.14, maturin 1.13.3.
- Exact command / steps: (1) `uv export --extra all --no-dev
--no-emit-project | syft → grype` for the product surface; (2) `cargo
check --workspace` + `maturin develop` + extension import/compress smoke
test; (3) `cargo test -p headroom-proxy --lib drift`; (4) `cargo fmt
--check` + `cargo clippy -p headroom-py`; (5) `npm audit` in
sdk/openclaw/docs + `bun audit` in docs.
- Observed result: `headroom-ai[all]` resolution scans clean — "No
vulnerabilities found" (179 pkgs); full/prod SBOM shows only the 2
documented accepted CVEs; pyo3 0.29 extension imports and runs (detach +
from_py_object paths exercised); drift tests 14/14 pass; cargo fmt +
clippy clean; all npm/bun audits report 0.
- Not tested: full `pytest` suite (no Python source changed);
release-profile wheel build (used dev-profile `maturin develop` for the
import proof — the extension is semantically identical).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
(`sbom/README.md`)
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective — N/A
(dependency bumps; existing suites + scans cover it)
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md — N/A (Release Please
auto-generates from the conventional commit)
## Additional Notes
**Accepted risks (the 2 residual CVEs).** Both originate solely from the
EleutherAI `lm-evaluation-harness` under the **opt-in `[benchmark]`
extra**, which Headroom invokes as a subprocess (never imports):
- `sqlitedict` CVE-2024-35515 (High) — pickle deserialization; package
abandoned (last release 2021), **no upstream fix exists**.
- `nltk` CVE-2026-54293 (High) — path traversal in `nltk.data.load()`;
affects ≤3.9.4 (current latest), **no patched release**.
Neither is in `[all]`, the published wheel, or the container. They are
documented in `sbom/README.md` and will be picked up automatically once
upstream ships fixes.
**Release/CHANGELOG:** N/A items above are because this is a
dependency/security PR with no Python source changes; CHANGELOG is
Release-Please-managed via the conventional commit message.
|
||
|
Nadia Ujovich
|
7c93c50c2c
|
Marker-free lossless_only mode + gate opaque-blob CCR markers behind enable_ccr_marker (#1129)
## Description `enable_ccr_marker` only gated the **row-drop sentinel** path. The **opaque-blob** path still emitted `<<ccr:HASH,kind,size>>` markers unconditionally whenever a string cell exceeded `opaque_min_bytes` (256), so **no configuration could produce a fully marker-free prompt**. Any `<<ccr:>>` marker is a promise that the full payload lives in the CCR store and must be fetched back via a retrieval tool call — there was no way to get compression without that round-trip dependency. **Rebased on #1130 (merged).** That PR fixed the opaque-blob gate at the classifier (`ClassifyConfig.emit_opaque_markers`, driven by `enable_ccr_marker`) and closed #1091. This branch originally carried its own equivalent gating commit; that commit is now **redundant and has been dropped** — `classifier.rs` here is identical to upstream. What remains is the **net-new** work that is **not** in #1130: - **Strict `lossless_only` mode** — keeps lossless tabular compaction, but routes every path that would need a CCR marker (row-drop sentinel **and** opaque-blob offload) to leave content uncompacted instead, so output is always marker-free **and** byte-recoverable. - **Python parity** — `lossless_only` exposed across both config dataclasses, a `SmartCrusher` kwarg, and a per-call `crush(..., lossless_only=)` override. - **`HEADROOM_LOSSLESS_ONLY` env var** — wires the mode through to the proxy runtime so real agents can use it. The #1130 opaque gate is consumed here through a single centralized helper (`opaque_markers_enabled() = enable_ccr_marker && !lossless_only`) used by **all four** `ClassifyConfig` construction sites. ## Type of Change - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - **`feat(smart_crusher)`** — Add `lossless_only` (default `false`): keeps lossless tabular compaction but routes every marker-requiring path (row-drop sentinel + opaque-blob offload) to leave content uncompacted instead. Exposed across the Rust core, PyO3 bridge, both Python config dataclasses, a `SmartCrusher` kwarg, a per-call `crush(..., lossless_only=)` override, and `smart_crush_tool_output`. Includes a `debug_assert` documenting the load-bearing invariant (a `lossless_only` crusher must never reach the CCR store write). - **`refactor(smart_crusher)`** — Extract `SmartCrusherConfig::opaque_markers_enabled()` as the single source of truth for `enable_ccr_marker && !lossless_only`, consumed by **all four** `ClassifyConfig` sites: the compaction-stage builder, `with_compaction_format`, the top-level `process_string` path (Rust core), and the PyO3 `compact_document_json` document-compactor path. No site derives the gate inline anymore, so they cannot drift. - **`feat(proxy)`** — Expose the mode via `HEADROOM_LOSSLESS_ONLY`: `ContentRouterConfig.smart_crusher_lossless_only` → `_get_smart_crusher`; the proxy reads the env var and sets it on the live router config. Previously reachable only via the Python API, never through the proxy. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check` on changed files) - [ ] Type checking passes (`mypy headroom`) — not run (see Additional Notes) - [x] New tests added for new functionality - [x] Manual testing performed (proxy env-var seam, end-to-end — see Real Behavior Proof) ### Test Output ```text ### RUST (cargo test -p headroom-core --lib smart_crusher) test result: ok. 323 passed; 0 failed; 0 ignored; 0 measured; 516 filtered out ### PYTEST (test_smart_crusher_bugs.py + test_agent_savings.py + test_smart_crusher_toin_attachment.py) 45 passed ### RUFF (changed files) All checks passed! ### FMT + CLIPPY (cargo fmt --check && cargo clippy --workspace --lib) clean — no warnings ``` New/affected tests: `enable_ccr_marker_false_suppresses_opaque_markers`, `lossless_only_leaves_array_uncompacted_instead_of_dropping`, `lossless_only_inlines_opaque_blobs_when_table_ships`, `lossless_only_never_writes_to_ccr_store` (Rust); `TestLosslessOnlyMode`, `test_router_lossless_only_flag_reaches_crusher`, `test_router_lossless_only_defaults_off` (Python). Coexists green with #1130's `long_string_stays_scalar_when_opaque_markers_disabled` (Rust) and `test_smart_crusher_toin_attachment.py` (Python). The Python `TestOpaqueMarkerGate` from the dropped gating commit was removed as redundant with #1130's coverage. ## Real Behavior Proof ### Proxy env-var seam — end-to-end (this revision) The one path with no automated coverage was `server.py` reading `HEADROOM_LOSSLESS_ONLY` from the environment and threading it into the live router config. Verified end-to-end by instantiating the **real** `HeadroomProxy`, pulling the `ContentRouter` out of its pipeline, and crushing a 50-row array with >256B opaque cells through the real Rust crusher: | | `HEADROOM_LOSSLESS_ONLY=1` | env unset (default) | |---|---|---| | `crusher._lossless_only` | **True** | **False** | | output contains `<<ccr:` | **No** (marker-free) | Yes (normal lossy) | | byte-recoverable (round-trips to original JSON) | **Yes** | No (rows offloaded) | This confirms the full chain `os.environ["HEADROOM_LOSSLESS_ONLY"]` → `server.py` → `ContentRouterConfig.smart_crusher_lossless_only` → `content_router.py` → `crusher_config.lossless_only` → Rust crusher. The default column proves strict mode genuinely changes behavior (not a no-op) and that the default path is unchanged. ### Prior live-traffic run - Environment: Headroom proxy in front of a real agent (Hermes) routed to NVIDIA NIM (OpenAI-compatible upstream). Isolated config dir; `OPENAI_TARGET_API_URL=https://integrate.api.nvidia.com/v1`, `HEADROOM_LOSSLESS_ONLY=1`. Confirmed with `ss` that all LLM traffic flowed agent → proxy → upstream with no direct bypass. - Exact command / steps: Start the proxy with `python -m headroom.proxy.server --host 127.0.0.1 --port 8787`; point the agent's LLM base_url at `http://127.0.0.1:8787/v1`; run a real chat plus a `search_files`-style task; read `/stats` and `/v1/retrieve/stats`; then toggle `HEADROOM_LOSSLESS_ONLY` and repeat for the comparison. - Observed result: With 150K+ tokens of real traffic processed, `lossless_only` kept the CCR store empty (`entry_count: 0`) and emitted zero markers. A synthetic before/after with opaque (>256B) cells produced 12 `<<ccr:>>` markers in default mode and 0 under `lossless_only`, with output round-tripping to the original JSON structure. - Not tested: A live `lossless_only`-vs-markers contrast on real agent traffic. The SmartCrusher offload path never engaged on this agent's tool outputs (`total_compressions: 0`; CCR store stayed at `entry_count: 0` even after a broad codebase search), and compression stayed marginal (~0.2–0.4%) in both modes. The agent's tool results don't match the crushable-array profile the offload paths target, so the marker path is never exercised in that integration. Why SmartCrusher barely engages with this agent's outputs is a separate integration question (output format / routing / size thresholds), out of scope for this change. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation — N/A (config docstrings updated in-tree; no separate docs) - [x] My changes generate no new warnings - [x] I have added tests that prove my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable — N/A ## Additional Notes - Rebased on top of merged #1130; the now-redundant opaque-blob gating commit was dropped, so this PR is purely the `lossless_only` feature + proxy wiring on top of #1130's gate. - `mypy headroom` was not run in this environment; happy to add the result if CI requires it. - Default behavior is fully preserved: `enable_ccr_marker` defaults to `true`, `lossless_only` defaults to `false`, and `HEADROOM_LOSSLESS_ONLY` unset is a no-op. |
||
|
Zhenjia ZHOU
|
6c68ff4e9f
|
perf(compression): take large cold-start contexts off the synchronous kompress path (#1171) (#1298)
## Description On a cold-start large context, kompress (ModernBERT ONNX) runs **synchronously on the request thread** — ~200–300s for ~1M tokens. It blows the 30s compression budget, leaks a non-preemptible worker, and cascades (executor saturation → queue timeouts on healthy requests); on timeout the request is forwarded **uncompressed** after eating 30s. This adds four layered, **default-off, fail-open** mitigations so the request path is never blocked on ML compression. Closes #1171 ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [x] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Phase 0 — size gate** (`HEADROOM_KOMPRESS_MAX_TOKENS`, default 50000): route oversized text away from ModernBERT (→ LogCompressor / TextCrusher / passthrough) at the single `_try_ml_compressor` boundary. - **Phase 1 — cooperative deadline** (`HEADROOM_COMPRESSION_DEADLINE_MS`, default 20000): any kompress run self-terminates at the next chunk boundary past the budget, keeping the unprocessed tail verbatim. - **Phase 2 — TextCrusher** (`HEADROOM_TEXT_CRUSHER`): a new **native Rust** extractive prose compressor in `crates/headroom-core/src/transforms/text_crusher/`, exposed via PyO3 as `headroom._core.TextCrusher` with a thin Python wrapper. It **reuses the shared `crate::relevance::BM25Scorer`** rather than reimplementing BM25, and ships record/replay parity fixtures (mirroring the SmartCrusher Rust-core + Python-shim pattern). - **Phase 3 — off-path compression** (`HEADROOM_BACKGROUND_COMPRESSION`): forward uncompressed immediately and compress in a per-process background drain; a byte-identical cache hit on a later turn means the request never blocks on ML. - Benchmark (`benchmarks/text_crusher_quality_eval.py`), CHANGELOG entry, and docstrings documenting the fail-open limits. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`, new modules) - [x] New tests added for new functionality - [ ] Manual testing performed (Phase 0/1 gate-fire + deadline observed on real traffic in earlier iterations; Phase 3 off-path is unit- + byte-identity-tested, not yet live-validated) ### Test Output ```text $ pytest tests/test_transforms/ tests/test_cache/ \ tests/test_proxy/test_background_compression.py tests/test_proxy/test_phase3_byte_identity.py -q 501 passed, 37 skipped in 40.33s $ cargo test -p headroom-core --lib text_crusher test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 834 filtered out $ ruff check <changed files> All checks passed! $ mypy headroom/proxy/background_compression.py headroom/transforms/text_crusher.py Success: no issues found in 2 source files ``` New coverage: size-gate incl. the strategy-dispatch funnel (KOMPRESS + TEXT); partial-run deadline (chunk-0 compressed + chunk-1 verbatim tail); BackgroundCompressor (dedup / queue-full / fail-open); Phase 3 byte-identity round-trip; TextCrusher unit + parity. ## Real Behavior Proof - Environment: macOS, local dev — `uv` venv, Rust `_core` built via `uv pip install -e .`. - Exact command / steps: the `pytest` / `cargo test` / `ruff` / `mypy` commands shown under Test Output; quality eval `python benchmarks/text_crusher_quality_eval.py /tmp/squad_dev.json`. - Observed result: 501 Python + 3 Rust tests pass; ruff + mypy clean on changed/new modules. Quality eval: TextCrusher keeps ~94% of buried SQuAD answers at 30% size vs ~36% truncate/random; self-contained speed run ~333k words in ~76ms (one O(n) pass) — sub-second where ModernBERT takes minutes (fast-vs-slow contrast, not a same-input run). - Not tested: Phase 3 off-path on live traffic; multi-worker (per-process by design — see Additional Notes). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes - **All four features are off by default and fail-open** — with the env flags unset the paths are no-ops for realistic inputs; on any error the request is forwarded (compressed if possible, else verbatim), never dropped. A full background queue / duplicate key surfaces as `deferred:dropped`. - **Known limits (documented in `background_compression.py`):** Phase 3 is per-process, in-memory, and token-mode-only — these are **lost-savings, never lost-correctness**, and consistent with the project's existing per-process compression cache + sticky-session multi-worker model. The startup multi-worker warning now names off-path background compression. - Phase 2 reuses the existing BM25 scorer; reuse did not improve answer-retention over a Python prototype (query-awareness dominates) — its value is the Rust speed + repo-conventional Rust-core/Python-shim shape. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Parideboy
|
3ccdad6c67
|
Pin ORT dylib on Windows; init Python logging (#1010)
## Description On Windows, headroom's Rust core resolves `onnxruntime.dll` at runtime via `ort-load-dynamic`. Without an explicit `ORT_DYLIB_PATH`, the bare DLL search can land on `C:\Windows\System32\onnxruntime.dll`, the Windows ML OS component, and `Session::new()` can deadlock instead of returning an error. Since a hang is not an `Err`, the tiered fallback cannot engage until the proxy-level timeout fires. This PR pins `ORT_DYLIB_PATH` to the pip-installed `onnxruntime` DLL at import time, and wires Rust `tracing` events into Python logging so the proxy log surfaces these failures when they occur. Closes #928 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Added `headroom/_ort.py` with a Windows-only, idempotent `ensure_ort_dylib_pinned()` resolver that respects an existing `ORT_DYLIB_PATH`. - Call the pin from `headroom/__init__.py` before importing `_core` consumers. - Log the effective ORT dylib path from the content router startup path on Windows. - Enable Rust tracing-to-log compatibility and initialize `pyo3-log` in the `_core` module. - Add timeout diagnostics in the Magika detector with the effective `ORT_DYLIB_PATH`. - Document `ORT_DYLIB_PATH` and `HEADROOM_MAGIKA_INIT_TIMEOUT_SECS`. - Add unit coverage for the resolver behavior. ## Testing - [x] Unit tests pass (`python -m pytest tests/test_transforms/test_ort_dylib.py -q`) - [x] Linting passes (`ruff check headroom/_ort.py headroom/__init__.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py`) - [x] Formatting passes (`ruff format --check headroom/_ort.py headroom/__init__.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_transforms/test_ort_dylib.py -q 7 passed in 0.19s $ ruff check headroom/_ort.py headroom/__init__.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py All checks passed! $ ruff format --check headroom/_ort.py headroom/__init__.py headroom/transforms/content_router.py tests/test_transforms/test_ort_dylib.py 4 files already formatted $ cargo check -p headroom-py cargo: The term 'cargo' is not recognized as a name of a cmdlet, function, script file, or executable program. ``` ## Real Behavior Proof - Environment: Windows 11 24H2, Python 3.13, RTX 4080 - Exact command / steps: `python -c "import headroom; from headroom._core import detect_content_type as d; print(d(open('headroom/compress.py').read()).content_type)"` - Observed result: `source_code` in 301ms, clean exit, `Magika: ENABLED` in proxy log - Not tested: macOS/Linux manual runtime behavior; `_ort.py` is a no-op outside Windows, and CI covers cross-platform build/test behavior. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable (N/A: repo uses release-please) ## Additional Notes The branch was rebased onto current `main` and the commit subject was updated to satisfy commitlint. Local Rust verification could not be run on this Windows machine because `cargo` is not installed; GitHub CI should be treated as the Rust build verification for the `pyo3-log` dependency and workspace lockfile changes. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Tejas Chopra
|
b7be3814f1
|
feat: compression extraction — Rust knob exposure, CCR hardening, traffic audits (#818)
## Description
A data-driven push for better compression savings without accuracy loss,
in four parts: expose and tune the Rust compressor knobs, harden the CCR
retrieval store, add traffic-audit tooling that sizes opportunities from
real transcripts, and introduce **read maturation** — a new,
live-validated mechanism that compresses Read outputs *before* they ever
enter the provider prefix cache.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [x] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
### 1. Rust compressor extraction
- Expose `lossless_min_savings_ratio` end-to-end and lower the default
0.30 → 0.15 (lockstep across Rust, PyO3, and both Python config classes)
so the lossless Table/CSV compaction path wins more often.
- Expose the `CompactConfig` heuristics (core-field fraction,
heterogeneity ratio, flatten cap, bucket bounds) through PyO3 + Python.
- `SearchCompressor` grouped-by-file output (`rg --heading` style — path
once per file instead of per match). Library default off; the proxy
enables it in token mode.
- Complete `factor_out_constants`: constant fields now emit once in a
`_constant_fields` sentinel with slim rows (defensive per-item value
match; default off).
- `ContentRouter` accepts a SmartCrusher config override and the
search-grouping knob.
### 2. CCR store hardening
- Session-scale TTL: 300s → 1800s (CCRConfig, CompressionEntry,
CompressionStore, Rust `DEFAULT_TTL` — lockstep).
- **SQLite is the default CCR backend** (`~/.headroom/ccr_store.db`,
WAL): survives proxy restarts and is shared across workers.
`HEADROOM_CCR_BACKEND=memory` opts out.
- Multi-worker safety: `busy_timeout`, and corruption detection narrowed
so transient `SQLITE_BUSY` errors can never trigger database deletion.
- Data-at-rest hygiene: `chmod 600` on db + sidecars, expired rows swept
at open.
- Retrieval-miss messages are actionable (re-read the file / re-run the
command).
### 3. Traffic audit tooling (measure before tuning)
- `headroom audit-reads`: sizes Read opportunities from local Claude
Code transcripts (read share, stale %, line-number overhead, context
residency, cache-death windows).
- `--simulate-maturation`: Mechanism B risk sizing (re-read rates,
never-touched-again share, quiesce coverage, at-risk edits).
- `--codex`: shell-read classifier for Codex transcripts (rtk-wrapper
aware, workdir resolution).
- Findings that shaped this PR (81 sessions): Reads are 67% of tool
bytes; median Read lingers 118 turns (~13x lifetime cost); a prototyped
repeat-Read dedup measured 0.1% and was **removed** rather than shipped
as dead code.
### 4. Read maturation (Mechanism B) — experimental, default OFF
- Activity-based: a fresh large Read is held **out** of the provider
cache (trailing breakpoint relocated before it), stays verbatim while
its file is active, and matures into a CCR-backed marker once the file
is quiet for `quiesce_turns` (default 5; `max_hold_turns` bounds busy
files).
- Only the final compressed form ever enters the cache — **no cached
byte is ever mutated**; matured markers replay byte-identically.
- Wired into the Anthropic handler behind `--read-maturation` /
`HEADROOM_READ_MATURATION=1`; session state rides on the prefix tracker;
advisory (can never fail a request).
- Live-validated against the Anthropic API: held content excluded from
cache_creation; after maturation the prior cached prefix still served —
the no-bust invariant holds end-to-end.
### 5. Rebase / CI fixups (this update)
- Rebased onto latest `main` (was 28 commits behind): picks up `ci: pass
CODECOV_TOKEN to coverage uploads (#968)`, which is what was turning the
4 test shards red — the tests themselves passed (1528) but the post-test
codecov upload exited non-zero on a protected branch.
- Resolved the duplicate `lossless_min_savings_ratio` that two
independent main/branch additions left in `SmartCrusherConfig` and the
Rust-config kwarg (import-time `SyntaxError` + mypy `no-redef`).
- Aligned CCR tests with the new defaults (SQLite backend, 1800s TTL)
across `test_ccr`, `test_adapter_hooks`, `test_compression_store`,
`test_proxy_ccr`, and the lossy row-drop bridge test.
## Testing
<!-- Check what you actually ran, then paste the real command output
below. -->
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_proxy_ccr.py tests/test_ccr.py tests/test_compression_store.py tests/test_adapter_hooks.py tests/test_ccr_row_drop_store_bridge.py -q
170 passed, 4 warnings in 42.49s
$ python -m pytest tests/test_audit_reads.py tests/test_audit_codex.py tests/test_read_maturation.py tests/test_transforms_content_router.py tests/test_smart_crusher_toin_attachment.py -q
83 passed
$ mypy headroom/
Success: no issues found in 365 source files
$ python -m compileall headroom/ -q
COMPILE-OK
# CI (run 27488990477, pre-rebase head): all 4 shards ran to completion —
# "1528 passed, 120 skipped, 4922 deselected"
# The red shards were the codecov upload step, not test failures; fixed by
# the #968 rebase above.
```
## Real Behavior Proof
- Environment: macOS (darwin), Python 3.12 venv; branch
`feat/compression-extraction` rebased onto `origin/main` (head
|
||
|
Focused Instability
|
06b2625b17
|
feat: gated Markdown-KV compaction formatter (serialization-aware output) (#859)
Closes #858. ## What Adds an opt-in **Markdown-KV** renderer to the lossless-first compaction stage, plus the plumbing to pick a compaction formatter by name. Default behavior is unchanged (`csv-schema`). Format-comprehension benchmarks show models retrieve values from Markdown-KV substantially more reliably than from CSV (~60.7% vs ~44.3%) — token-cheapest is not the same as most comprehensible. This makes the trade-off selectable per workload. ## How - **`MarkdownKvFormatter`** (`compaction/formatter.rs`): keeps the `[N]{cols}` declaration line, renders each row as a Markdown list item with `key: value` lines. - Missing cells omitted entirely (the KV advantage over positional CSV). - Strings ambiguous on a line (newlines, leading/trailing whitespace, empty) render JSON-quoted; everything else raw — commas and quotes need no escaping. - Nested cells inline compact JSON; opaque cells keep the fixed `<<ccr:HASH,KIND,SIZE>>` marker contract shared by all formatters. - **`CompactionStage::from_format_name`** maps `"csv-schema" | "json" | "markdown-kv"` to presets. - **Core**: `SmartCrusher::with_compaction_format(config, name)` — standard OSS composition with the named formatter. - **PyO3 bridge**: `SmartCrusher.with_compaction_format(config, format_name)` staticmethod; `ValueError` on unknown names (loud, no silent fallback). - **Python**: `SmartCrusher(compaction_format=...)` kwarg, falling back to the `HEADROOM_COMPACTION_FORMAT` env var, default `"csv-schema"`. ## Safety - **Default-off**: the default constructor path still calls the Rust `new()` constructor, so byte-parity coverage stays on the exact production codepath. A test asserts default output is byte-identical to an explicit `csv-schema` opt-in. - The existing `lossless_min_savings_ratio` gate (0.30) still applies. Markdown-KV repeats field names per row, so it clears the gate less often than CSV and falls through to the lossy path — we never inline a "lossless" rendering that isn't actually smaller. - CCR marker format unchanged across formatters; downstream retrieval pattern-matching keeps working. - No user/assistant content dropped — the formatter is a pure rendering of the same Compaction IR. ## Tests - Rust: 10 new unit tests in `compaction/formatter.rs` (table/buckets rendering, missing-cell omission, string quoting, CCR markers, drop summary, byte-size sanity vs raw JSON). `cargo test -p headroom-core`: 894 passed. Clippy + fmt clean. - Python: `tests/test_compaction_markdown_kv.py` (10 tests) — bridge rendering end-to-end, name→preset parity with the default constructor, kwarg/env knob precedence, loud failure on unknown names, default-output-unchanged guarantee. Existing smart_crusher suite: 38 passed. --------- Co-authored-by: Ash Rhodes <ashley.rhodes@king.com> |
||
|
dependabot[bot]
|
4ff7b4426d
|
ci: bump pyo3 from 0.22.6 to 0.24.1 in the cargo group across 1 directory (#270)
Bumps the cargo group with 1 update in the / directory: [pyo3](https://github.com/pyo3/pyo3). Updates `pyo3` from 0.22.6 to 0.24.1 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/pyo3/pyo3/releases">pyo3's releases</a>.</em></p> <blockquote> <h2>PyO3 0.24.1</h2> <p>This release is a security fix for the <code>PyString::from_object</code> method, which passed <code>&str</code> data to the Python C API without checking for a terminating nul byte. All historical PyO3 versions are affected, and we recommend you upgrade if you are using <code>PyString::from_object</code>. Thank you to <a href="https://github.com/vthib"><code>@vthib</code></a> for the report and <a href="https://github.com/Dr-Emann"><code>@Dr-Emann</code></a> for the fix. A RUSTSEC advisory will be published shortly.</p> <p>Aside from the security fix, this release contains a number of other non-breaking additions:</p> <ul> <li>An <code>abi3-py313</code> feature to support compiling with the Python 3.13 stable ABI.</li> <li><code>PyAnyMethods::getattr_opt</code> to get optional attributes without paying the cost of a Python exception when the attribute in question does not exist.</li> <li>Constructor for <code>PyInt::new</code>.</li> <li><code>with_critical_section2</code> for locking two objects at the same time on the free-threaded build.</li> <li>Fix for a PyO3 0.24.0 regression with <code>Option<&str></code> and <code>Option<&T></code> (where <code>T: PyClass</code>) function arguments no longer being permitted</li> </ul> <p>There are also a few other small bug fixes for edge cases, mostly related to compile errors from PyO3's macro code.</p> <p>Thank you to the following contributors for the improvements:</p> <p><a href="https://github.com/bschoenmaeckers"><code>@bschoenmaeckers</code></a> <a href="https://github.com/davidhewitt"><code>@davidhewitt</code></a> <a href="https://github.com/Dr-Emann"><code>@Dr-Emann</code></a> <a href="https://github.com/emmagordon"><code>@emmagordon</code></a> <a href="https://github.com/epontan"><code>@epontan</code></a> <a href="https://github.com/Icxolu"><code>@Icxolu</code></a> <a href="https://github.com/IvanIsCoding"><code>@IvanIsCoding</code></a> <a href="https://github.com/jelmer"><code>@jelmer</code></a> <a href="https://github.com/jonaspleyer"><code>@jonaspleyer</code></a> <a href="https://github.com/ngoldbaum"><code>@ngoldbaum</code></a> <a href="https://github.com/Owen-CH-Leung"><code>@Owen-CH-Leung</code></a> <a href="https://github.com/Tpt"><code>@Tpt</code></a> <a href="https://github.com/Trolldemorted"><code>@Trolldemorted</code></a> <a href="https://github.com/XuehaiPan"><code>@XuehaiPan</code></a></p> <h2>PyO3 0.24.0</h2> <p>This release is an incremental improvement of refinements and optimizations following the new APIs established in PyO3's last few releases.</p> <p>Support for <code>jiff</code> datetime conversions have been added, and also UUID conversions.</p> <p>The <code>FromPyObject</code> derive macro has gained new <code>#[pyo3(default = ...)]</code> and <code>#[pyo3(rename_all = ...)]</code> options, and the <code>IntoPyObject</code> derive macro has gained a new <code>#[pyo3(into_py_with = ...)]</code> option.</p> <p>PyO3 will now pass positional arguments to Python functions using the "vectorcall" protocol in many cases, which should be an optimization over the previous behaviour (of creating a Python tuple of positional arguments).</p> <p>Many methods on iterators of Python collections have been optimized.</p> <p>There are also many other incremental improvements, bug fixes and smaller features.</p> <p>Thank you to everyone who contributed code, documentation, design ideas, bug reports, and feedback. The following contributors' commits are included in this release:</p> <p><a href="https://github.com/0x676e67"><code>@0x676e67</code></a> <a href="https://github.com/alex"><code>@alex</code></a> <a href="https://github.com/arielb1"><code>@arielb1</code></a> <a href="https://github.com/bschoenmaeckers"><code>@bschoenmaeckers</code></a> <a href="https://github.com/davidhewitt"><code>@davidhewitt</code></a></p> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/PyO3/pyo3/blob/main/CHANGELOG.md">pyo3's changelog</a>.</em></p> <blockquote> <h2>[0.24.1] - 2025-03-31</h2> <h3>Added</h3> <ul> <li>Add <code>abi3-py313</code> feature. <a href="https://redirect.github.com/PyO3/pyo3/pull/4969">#4969</a></li> <li>Add <code>PyAnyMethods::getattr_opt</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4978">#4978</a></li> <li>Add <code>PyInt::new</code> constructor for all supported number types (i32, u32, i64, u64, isize, usize). <a href="https://redirect.github.com/PyO3/pyo3/pull/4984">#4984</a></li> <li>Add <code>pyo3::sync::with_critical_section2</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4992">#4992</a></li> <li>Implement <code>PyCallArgs</code> for <code>Borrowed<'_, 'py, PyTuple></code>, <code>&Bound<'py, PyTuple></code>, and <code>&Py<PyTuple></code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/5013">#5013</a></li> </ul> <h3>Fixed</h3> <ul> <li>Fix <code>is_type_of</code> for native types not using same specialized check as <code>is_type_of_bound</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4981">#4981</a></li> <li>Fix <code>Probe</code> class naming issue with <code>#[pymethods]</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4988">#4988</a></li> <li>Fix compile failure with required <code>#[pyfunction]</code> arguments taking <code>Option<&str></code> and <code>Option<&T></code> (for <code>#[pyclass]</code> types). <a href="https://redirect.github.com/PyO3/pyo3/pull/5002">#5002</a></li> <li>Fix <code>PyString::from_object</code> causing of bounds reads with <code>encoding</code> and <code>errors</code> parameters which are not nul-terminated. <a href="https://redirect.github.com/PyO3/pyo3/pull/5008">#5008</a></li> <li>Fix compile error when additional options follow after <code>crate</code> for <code>#[pyfunction]</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/5015">#5015</a></li> </ul> <h2>[0.24.0] - 2025-03-09</h2> <h3>Packaging</h3> <ul> <li>Add supported CPython/PyPy versions to cargo package metadata. <a href="https://redirect.github.com/PyO3/pyo3/pull/4756">#4756</a></li> <li>Bump <code>target-lexicon</code> dependency to 0.13. <a href="https://redirect.github.com/PyO3/pyo3/pull/4822">#4822</a></li> <li>Add optional <code>jiff</code> dependency to add conversions for <code>jiff</code> datetime types. <a href="https://redirect.github.com/PyO3/pyo3/pull/4823">#4823</a></li> <li>Add optional <code>uuid</code> dependency to add conversions for <code>uuid::Uuid</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4864">#4864</a></li> <li>Bump minimum supported <code>inventory</code> version to 0.3.5. <a href="https://redirect.github.com/PyO3/pyo3/pull/4954">#4954</a></li> </ul> <h3>Added</h3> <ul> <li>Add <code>PyIterator::send</code> method to allow sending values into a python generator. <a href="https://redirect.github.com/PyO3/pyo3/pull/4746">#4746</a></li> <li>Add <code>PyCallArgs</code> trait for passing arguments into the Python calling protocol. This enabled using a faster calling convention for certain types, improving performance. <a href="https://redirect.github.com/PyO3/pyo3/pull/4768">#4768</a></li> <li>Add <code>#[pyo3(default = ...']</code> option for <code>#[derive(FromPyObject)]</code> to set a default value for extracted fields of named structs. <a href="https://redirect.github.com/PyO3/pyo3/pull/4829">#4829</a></li> <li>Add <code>#[pyo3(into_py_with = ...)]</code> option for <code>#[derive(IntoPyObject, IntoPyObjectRef)]</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4850">#4850</a></li> <li>Add FFI definitions <code>PyThreadState_GetFrame</code> and <code>PyFrame_GetBack</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4866">#4866</a></li> <li>Optimize <code>last</code> for <code>BoundListIterator</code>, <code>BoundTupleIterator</code> and <code>BorrowedTupleIterator</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4878">#4878</a></li> <li>Optimize <code>Iterator::count()</code> for <code>PyDict</code>, <code>PyList</code>, <code>PyTuple</code> & <code>PySet</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4878">#4878</a></li> <li>Optimize <code>nth</code>, <code>nth_back</code>, <code>advance_by</code> and <code>advance_back_by</code> for <code>BoundTupleIterator</code> <a href="https://redirect.github.com/PyO3/pyo3/pull/4897">#4897</a></li> <li>Add support for <code>types.GenericAlias</code> as <code>pyo3::types::PyGenericAlias</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4917">#4917</a></li> <li>Add <code>MutextExt</code> trait to help avoid deadlocks with the GIL while locking a <code>std::sync::Mutex</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4934">#4934</a></li> <li>Add <code>#[pyo3(rename_all = "...")]</code> option for <code>#[derive(FromPyObject)]</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4941">#4941</a></li> </ul> <h3>Changed</h3> <ul> <li>Optimize <code>nth</code>, <code>nth_back</code>, <code>advance_by</code> and <code>advance_back_by</code> for <code>BoundListIterator</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4810">#4810</a></li> <li>Use <code>DerefToPyAny</code> in blanket implementations of <code>From<Py<T>></code> and <code>From<Bound<'py, T>></code> for <code>PyObject</code>. <a href="https://redirect.github.com/PyO3/pyo3/pull/4593">#4593</a></li> <li>Map <code>io::ErrorKind::IsADirectory</code>/<code>NotADirectory</code> to the corresponding Python exception on Rust 1.83+. <a href="https://redirect.github.com/PyO3/pyo3/pull/4747">#4747</a></li> <li><code>PyAnyMethods::call</code> and friends now require <code>PyCallArgs</code> for their positional arguments. <a href="https://redirect.github.com/PyO3/pyo3/pull/4768">#4768</a></li> <li>Expose FFI definitions for <code>PyObject_Vectorcall(Method)</code> on the stable abi on 3.12+. <a href="https://redirect.github.com/PyO3/pyo3/pull/4853">#4853</a></li> <li><code>#[pyo3(from_py_with = ...)]</code> now take a path rather than a string literal <a href="https://redirect.github.com/PyO3/pyo3/pull/4860">#4860</a></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
||
|
Tejas Chopra
|
eaf5980b4a | fix: stabilize codex compression, stats, and proxy lifecycle | ||
|
chopratejas
|
89f7b6c2dd |
fix: complete /v1/responses compression telemetry, multi-frame WS, frozen-count cap
Builds on PR #406 (HTTP /v1/responses PyO3) and PR #410 (WS first-frame PyO3) which landed the binding for `compress_openai_responses_live_zone`. This change closes the remaining gaps so every (provider × endpoint × auth-mode × streaming) combination compresses AND surfaces in the dashboard. Telemetry: extend the PyO3 binding return tuple from `(bytes, modified)` to `(bytes, modified, tokens_saved, transforms_applied)` by adding `CompressionManifest::tokens_saved()` and `transforms_applied()` accessors on the existing manifest. The Python proxy populates request-log telemetry from the binding output instead of recounting tokens. Updates the existing 2-tuple call sites in HTTP and WS first-frame, plus the unpacks in tests. WebSocket multi-frame compression: subscription Codex users keep a long-lived WS open and send multiple `response.create` events per session. PR #410 only compressed the first frame; subsequent frames went raw. Added `_maybe_compress_response_create_frame` closure inside `_client_to_upstream` that runs the same Rust dispatcher on every client→upstream `response.create` text frame, passes other event types (response.cancel, session.update, etc.) through unchanged, and accumulates `tokens_saved` / `transforms_applied` / `ws_frames_compressed` counters across the session. Pre-existing dashboard gap: `streaming.py` and `anthropic.py` write `RequestLog` entries; the non-streaming OpenAI HTTP and WS handlers did not. Result: /transformations/feed was invisible for every Codex turn and every Cline / OpenClaude / Aider turn. Added the same wiring in `handle_openai_chat` (non-streaming), `handle_openai_responses` (non-streaming HTTP), and `handle_openai_responses_ws` (session-end). All three populate `auth_mode` + `endpoint` tags so the dashboard can break compression activity down by client class (PAYG / OAuth / Subscription) and surface (`chat_completions` / `responses_http` / `responses_ws`). The WS metric record is now unconditional — was previously gated on `tokens_saved > 0`, so first-frame no-changes never registered. compute_frozen_count over-freeze for prose-format clients: `compute_frozen_count` walked until it found an unstable `tool_result` / `role: "tool"` block. Cline / OpenClaude / Aider — clients that embed tool calls as XML inside plain text — never produce such a boundary, so the function returned `len(messages)` and the pipeline froze 100% of messages including the brand-new user turn. Live zone empty → `Transform content_router: 16414 → 16414 tokens (saved 0)`. Reported on Discord 2026-05-07 with Cline+DeepSeek. Fix: cap at `max(0, len(messages) - 1)`. Updates 3 existing test assertions whose expected values encoded the old over-freeze. Adds 6 new prose-format invariant tests. CodeQL "clear-text logging of sensitive information" fix: `tests/e2e_real_compression.py` previously stored API keys in local variables in the same scope as diagnostic prints, which CodeQL flagged via data-flow analysis. Refactored to read keys from `os.environ` inside the request helper — the credentials never enter the runner's main scope, so the taint flow never reaches the print. End-to-end verification with real keys (.env): /v1/messages (PAYG, non-stream) tok 14109 → 969 saved 13140 /v1/messages (PAYG, stream) tok 14109 → 969 saved 13140 /v1/chat/completions (PAYG, non-stream) tok 18460 → 1374 saved 17086 /v1/chat/completions (PAYG, stream) tok 18460 → 1374 saved 17086 (cache_hit=100%) /v1/responses HTTP (PAYG, non-stream) bytes 50138 → 597 saved 18391 /v1/responses WS (frame 1) bytes 46429 → 488 saved 16791 /v1/responses WS (frame 2 multi) bytes 46429 → 488 saved 16791 /v1/responses WS (response.cancel) passthrough untouched Tests: cargo workspace + pytest (4846 pass, 0 fail), make ci-precheck passed, two E2E scripts (multi-turn HTTP, WS fake-upstream) all green. |
||
|
chopratejas
|
c48735d029 |
fix(core): expose compress_openai_responses_live_zone via PyO3 (hot-fix c1/2)
PR-C5 (May 3) retired the Python `/v1/responses` compression pipeline with the comment "Rust handles item-aware compression natively" — but the standalone `crates/headroom-proxy` binary that was supposed to do that compression is not deployed by the CLI today (`headroom proxy` and `headroom wrap codex` both run only the Python proxy via uvicorn). Result: every `/v1/responses` request since v0.20.16 has been forwarded uncompressed. Codex CLI is the flagship consumer of this endpoint; this is the regression users have been reporting. Closes Bug 1 of the Codex regression by exposing the existing `headroom_core::transforms::compress_openai_responses_live_zone` as a PyO3 binding so the Python proxy can call the live-zone dispatcher in-process. The `headroom._core` extension is already loaded at proxy startup (PR-A0 verifies), so adding one more callable is mechanical. Why PyO3 inline (Layer 1) vs originally-intended two-process chain (Layer 2): the inline call requires zero deployment changes — the wheel already ships `headroom._core`. Layer 2 (build + ship the standalone `headroom-proxy` binary, teach CLI to spawn both processes) is the right long-term move; Layer 1 restores v0.5.21 functional behaviour today. # Returns `(body, modified)`. On change → `(new_body_bytes, True)`; on passthrough → `(input_bytes, False)`. # Failure mode Never raises. The dispatcher's `LiveZoneError` cases (body not JSON, no input array) are passthrough conditions matching the Rust proxy's `compress_openai_responses_request` contract. # Tests 14 new tests in `tests/test_responses_pyo3_compression.py`: binding exposed, passthrough cases, every F1 AuthMode variant, empty-model default, no-raise on garbage bytes. |
||
|
chopratejas
|
17d6207bf5 |
fix(crusher): shim __libc_single_threaded for glibc < 2.32 + extend audit
PR #396's X2 dry-run caught a wheel-import failure on the manylinux_2_28 floor matrix entry (both x86_64 and aarch64). Same class as #355: ImportError: ... undefined symbol: __libc_single_threaded `__libc_single_threaded` is a single-byte char added in glibc 2.32. Newer libstdc++ (gcc 11+) reads it inside `__cxa_thread_atexit_impl` to elide locking on the single-threaded fast path. ORT prebuilt static archives compiled with gcc-14.2.1 against glibc-2.38+ headers bake in the reference. Users with glibc < 2.32 hit ImportError on `import headroom._core`. Latent since the ORT artifact bump that started using gcc 14. X1 is the gate that catches it at release time; X2 caught it at PR time — exactly as designed. Fix: 1. glibc_compat.c adds Section B: `char __libc_single_threaded = 0;` Setting to 0 (multi-threaded) is safe; libstdc++ takes the locked slow path. Setting to 1 would race in any multithreaded Rust wheel. 2. build.rs adds `-Wl,-u,__libc_single_threaded` so the shim's archive members are pulled regardless of scan order. 3. audit_wheel_glibc_symbols.py POST_FLOOR_SYMBOLS adds the new symbol — verified locally: the audit now rejects the failing PR #396 wheel with the right message. |
||
|
chopratejas
|
820e66cae6 |
fix(ci): force-link glibc shim with -Wl,-u so aarch64 wheel includes it
PR #385's shim works on x86_64 wheel build but FAILS audit on aarch64 in run 25358313722: FAIL: headroom_ai-0.20.27-cp310-cp310-manylinux_2_28_aarch64.whl references symbols above its glibc floor: __isoc23_strtoll (no version tag, introduced in glibc 2.38) Diagnosis: cargo's link order on aarch64 happens to place our shim's static archive BEFORE the ORT prebuilt archives. When the linker scans our archive, no UND `__isoc23_*` exists yet (ORT hasn't been scanned), so our shim's `.o` is dropped (no symbol to satisfy). ORT scans next, registers UND, but our archive isn't rescanned. Result: `_core.so` still has UND `__isoc23_*` symbols and the audit rightly rejects the wheel. On x86_64 the order happened to be the opposite (ORT first → UND registered → our archive scans next → satisfies → pulled in). Order is implementation-defined and clearly arch-dependent. Fix: emit `cargo:rustc-link-arg=-Wl,-u,<sym>` for each `__isoc23_*` symbol in `build.rs`. `-u <sym>` (a.k.a. `--undefined`) tells the linker to treat the symbol as undefined at the START of linking, which forces any archive defining it to be scanned and its members pulled in regardless of relative archive order. Shim is now uniformly linked on both x86_64 and aarch64. Documented inline in `build.rs`. Standard workaround for the static-library-link-order problem when the consumer scans after the provider. |
||
|
chopratejas
|
6b15acc3b5 |
fix(ci): glibc shim — drop alias attribute, forward-declare strtol
PR #384 introduced glibc_compat.c using __attribute__((weak, alias("strtol"))) which fails to compile because GCC requires the alias TARGET to live in the same translation unit. strtol is in libc.so.6, not the .c file. Result: clippy fails on every Linux CI job for every PR + main: glibc_compat.c: error: '__isoc23_strtol' aliased to undefined symbol 'strtol' Two-line architectural change: (1) drop the alias attribute, give each __isoc23_* function a plain body that calls the older strtol family; (2) forward-declare the older prototypes ourselves instead of #include <stdlib.h>, otherwise GCC's __REDIRECT_NTH(strtol -> __isoc23_strtol) would silently rewrite our delegation into an infinite recursion. Symbol-resolution semantics unchanged: on glibc 2.38+, libc's strong __isoc23_strtoll preempts ours via global-scope-first lookup; on glibc < 2.38, ours wins. Either way the symbol resolves and import succeeds. Both traps documented inline in glibc_compat.c so a future refactor doesn't reintroduce them. PR #384's commit message overstated the validation: I tested the audit script against the broken wheel but did NOT compile the shim itself before merging. Adding the X1 smoke-import gate (separate PR) is what would have caught this. |
||
|
chopratejas
|
e2146724af |
fix: ship glibc 2.38 compat shim + wheel symbol audit (closes #355)
Issue #355: published headroom_ai-*-manylinux_2_28_*.whl fails to import on Ubuntu 22.04, Debian 11/12, Conda envs with libc < 2.38: 'ImportError: undefined symbol: __isoc23_strtoll'. Root cause: ORT prebuilt artifacts (downloaded via fastembed's ort-download-binaries-rustls-tls feature) are compiled with gcc-14.2.1 on a glibc-2.38+ host and reference __isoc23_strtoll. Our manylinux build host has glibc 2.38 so the link succeeds; end users with older glibc don't. Two-part fix: (1) crates/headroom-py/glibc_compat.c provides weak-alias definitions for __isoc23_strtol/strtoll/strtoul/strtoull delegating to the older strtol* family, compiled by build.rs on Linux/glibc only. The dynamic linker prefers glibc's strong symbol when present (>= 2.38) and falls back to ours when not (< 2.38). (2) scripts/audit_wheel_glibc_symbols.py is a release.yml gate that runs objdump -T on every Linux wheel and rejects any UND symbol whose required glibc version exceeds the wheel's manylinux floor. Validated: the audit correctly rejects the actually-broken v0.20.26 wheel with a precise diagnostic. The shim itself is a tiny static link with zero runtime cost. Regression tests in tests/test_release_workflows.py pin the shim's load-bearing pieces (.c file, build.rs trigger, [build-dependencies] cc dep) and the audit invocation in release.yml. Future drift fails at PR time, not in the next release. This is the same bug class as PR #371 (rustls-everywhere). Each instance gets fixed; the audit gate now catches the *class* — any future static linkage that introduces a post-floor symbol gets blocked before publish. |
||
|
chopratejas
|
6f2c0a8400 |
fix(ci): rustls-everywhere — eliminate openssl-sys from build tree
# Root cause of the wheel-build cascade We have shipped 5 release-pipeline hot-fixes in 12 hours, each addressing a different symptom of the same architectural problem: 1. PR #363 — npm artifact downloads + tried `yum openssl-devel` 2. PR #367 — vendored OpenSSL in `headroom-proxy` + dropped Intel mac 3. PR #369 — Debian-cross perl install (`perl` not `libipc-cmd-perl`) 4. PR #370 — moved `openssl/vendored` from headroom-proxy to headroom-py 5. (this PR) — ELIMINATE OpenSSL entirely Each fix exposed a different missing system package or feature flag in a different build surface (manylinux x86_64 vs aarch64-cross-Debian vs macOS Intel vs e2e/wrap Dockerfile vs e2e/init Dockerfile vs main Dockerfile vs devcontainer). We were playing whack-a-mole because every Cargo dep change to the OpenSSL surface required matching system-package updates in 6+ different Dockerfiles and workflows, and the PR-level CI didn't exercise all of them. # Why this PR is the structural fix `fastembed` exposes clean rustls feature flags: - `hf-hub-rustls-tls` (replaces default `hf-hub-native-tls`) - `ort-download-binaries-rustls-tls` (replaces default `…native-tls`) By disabling fastembed's default features and enabling the rustls variants explicitly, we remove `native-tls` (and therefore `openssl-sys`, `openssl`, `openssl-src`, perl modules, OpenSSL build-time deps, vendored OpenSSL ~30s build cost) from the entire workspace dep tree. Verified locally: $ cargo tree -p headroom-py -i openssl-sys error: package ID specification `openssl-sys` did not match any packages $ cargo tree -p headroom-py -i native-tls error: package ID specification `native-tls` did not match any packages $ cargo build --release -p headroom-py Finished `release` profile [optimized] target(s) in 25.57s (Down from 1m+ with vendored OpenSSL.) # Cleanups enabled by this change - crates/headroom-py/Cargo.toml — dropped the `openssl/vendored` workaround from PR #370. - crates/headroom-proxy/Cargo.toml — same dep removed. - e2e/wrap/Dockerfile — dropped `yum install openssl-devel pkgconfig perl-IPC-Cmd`. Comment retained explaining why. - e2e/init/Dockerfile — same. - Dockerfile (main) — dropped `pkg-config libssl-dev` from apt-get. - .devcontainer/Dockerfile — dropped `pkg-config libssl-dev`. - .github/workflows/release.yml — removed the entire before-script-linux block (perl install probe + multi-package-manager dispatch + fail-loud assertion). No longer needed. # Regression gate Three new structural tests in tests/test_release_workflows.py: - test_no_openssl_sys_in_wheel_build_tree — runs `cargo tree -p <crate> -i openssl-sys` for headroom-py / headroom-proxy / headroom-core. If openssl-sys reappears (a future native-tls enabler creeping in via a new dep), this fails AT PR TIME with an actionable message. - test_no_native_tls_in_wheel_build_tree — same shape, native-tls is the proximate cause. - test_fastembed_uses_rustls_features — checks the Cargo.toml so a future "let me bump fastembed and forget the features" doesn't silently re-introduce OpenSSL. Plus two cleanup gates: - test_dockerfiles_no_longer_install_openssl_devel - test_release_yml_does_not_install_openssl_or_perl_for_wheels All 13 release-workflow tests pass. `make ci-precheck` PASSED. # What this teaches us about rollouts (per user's ultrathink ask) The 5-fix cascade exposed three meta-problems: 1. PR checks don't block merges. PR #370 had docker-init-e2e, docker-wrap-e2e, docker-native-e2e all FAILED yet got merged. Branch protection should require these checks. Operator action needed (cannot fix in code). 2. Local validation is misleading. `cargo build -p headroom-py` from the workspace root used the workspace lockfile and looked green; CI did fresh resolution against headroom-py's manifest alone where the feature wasn't enabled. Lesson: verify structural invariants with `cargo tree -e features` before trusting that a build "works." 3. 6+ build surfaces with independent system-dep state. Every Cargo change required matching updates in 6 places. The structural answer (this PR) is to NOT depend on system OpenSSL at all. Where structural fixes are not possible, the answer is a single shared scripts/install-rust-build-deps.sh — but with this PR there's nothing left to install. |
||
|
chopratejas
|
a5c7f6fed9 |
fix(ci): vendored OpenSSL must live in headroom-py, not headroom-proxy
PR #367 added `openssl = { features = ["vendored"] }` to
`crates/headroom-proxy/Cargo.toml`, expecting Cargo's feature
unification to propagate the vendored feature throughout the
workspace. PR #369 unblocked aarch64 by fixing the perl install.
The next release run on `
|
||
|
chopratejas
|
2a91cbb4b4 |
refactor: single-wheel maturin build backend (fixes #355)
Eliminates the dual-package architecture that was the root cause of #355. `pip install headroom-ai` now produces ONE wheel containing both the Python source (headroom/*.py) and the compiled Rust extension (headroom/_core.so). No more separate `headroom-core-py` package, no more chicken-and-egg with PyPI publication, no more wheelhouse / PIP_FIND_LINKS / composite-action plumbing in CI. This is the canonical pattern used by cryptography, polars, ruff, pydantic-core, and other Rust-as-core Python packages. Honors the "Rust as core engine" direction. ## What changed - pyproject.toml: `[build-system]` swapped from hatchling to maturin. `[tool.hatch.*]` deleted; `[tool.maturin]` added pointing at `crates/headroom-py/Cargo.toml` for the cdylib. `python-source = "."` picks up the root `headroom/` package directly (dashboard HTML templates and other non-Python files included automatically). - crates/headroom-py/pyproject.toml: deleted. The crate is no longer a separate published package; its Cargo.toml stays as the cdylib build target invoked via `[tool.maturin] manifest-path`. - crates/headroom-py/python/: deleted (placeholder layout for the old separate package). ## CI updates - ci.yml: `test` / `test-extras` / `test-agno` jobs simplified — Rust toolchain set up before `pip install -e .` (which now invokes maturin via build-system). Removed the "build wheel + symlink .so" dance. `build` job swapped from `python -m build` (hatch) to `maturin build` + `maturin sdist`. - release.yml: collapsed dual-package matrix into one. New `build-wheels` matrix produces cross-platform wheels for cp310/11/12/13 × {linux x86_64, linux aarch64, macos x86_64, macos aarch64}. New `collect-dist` aggregator merges artifacts. publish-pypi consumes the merged dist. - init-native-e2e.yml: dropped windows-latest from the matrix — upstream `esaxx-rs` (/MT) and `ort-sys` (/MD) link with conflicting MSVC C runtime libraries, so the Rust extension cannot build for win_amd64 today. Tracked as a follow-up; not a blocker for Linux+macOS. - headroom-e2e-setup: composite action now sets up Rust toolchain + Swatinem/rust-cache before `pip install -e .[proxy]`. - eval.yml, publish.yml, rust.yml: same pattern — rust toolchain before install. rust.yml's wheels job builds from root pyproject.toml (no more `-m crates/headroom-py/Cargo.toml`). - e2e/init/Dockerfile, e2e/wrap/Dockerfile: install rust + maturin in the build stage; copy `crates/` + workspace `Cargo.toml/lock` so the install can build the extension. Dropped `HEADROOM_REQUIRE_RUST_CORE=false` from wrap-e2e — the image now ships the full Rust core. - Dockerfile (main): simplified — no more Layer 2/3 dance with `headroom-core-py` install + symlink. Single `uv pip install` builds + installs everything. - .devcontainer/Dockerfile: rust toolchain + libssl-dev + maturin added so `uv sync` builds the extension inside the devcontainer. ## Lockfile + script - uv.lock: regenerated. No `headroom-core-py` entries remain. - scripts/build_rust_extension.sh: simplified from a symlink-into-tree workaround to a thin wrapper around `pip install -e .`. The maturin build-backend handles placement automatically. ## Local validation (all green on macOS aarch64) 1. Clean venv `pip install -e .` → `from headroom._core import …` works. 2. `maturin build --release` → 13.8 MB wheel, 336 files including `headroom/_core.cpython-311-darwin.so` (32 MB cdylib) and `headroom/dashboard/templates/dashboard.html`. 3. `pip install <wheel>` in fresh venv → import works. 4. Wheel contents verified via `unzip -l`. 5. `pytest tests/test_transforms/test_diff_compressor.py` — 29 passed. 6. `pytest tests/test_relevance.py` — 30 passed. 7. `cargo build --workspace` + `cargo test --workspace` — all green. 8. `make ci-precheck` — 176 Python tests + Rust + commitlint green. ## Migration notes Users on `pip install headroom-ai` get the Rust core automatically (linux + macos wheels). sdist installs require rust toolchain available locally — pip will build via maturin. Closes #355 Supersedes #357 (workarounds-based fix abandoned in favor of architectural fix) |
||
|
chopratejas
|
c9aaba3f5b |
feat(rust): port tag_protector to Rust + 5 bug fixes (Phase 3e.4)
`headroom/transforms/tag_protector.py` was a regex-driven scan-and-
replace loop that ran on every kompress call from ContentRouter
(`content_router.py:1089`). The Python implementation had five real
bugs we now fix in the port — the most consequential being a
`str.replace(.., .., 1)` first-occurrence-replace bug that silently
collapsed two identical custom-tag blocks in the same input to a
single placeholder + a stray duplicate of the second block.
# Bug fixes (each pinned by a `fixed_in_3e4` test)
* **#1: O(n²) on nested custom tags.** Python's `while changed` loop
restarted a full regex scan after every replacement. Rust walks
once in linear time on input length.
* **#2: First-occurrence replace bug.** `result.replace(orig, ph, 1)`
replaces the FIRST textual match, not the matched offset. Two
identical custom-tag blocks collapsed to one placeholder + a stray
duplicate of the second block. The Rust walker stitches output by
offset so distinct blocks always get distinct placeholders.
* **#3: Silent 50-iteration cap.** Python had a hard `max_iterations
= 50` safety limit that quietly truncated tag protection on deeply
nested input. The Rust walker is bounded by input length only.
* **#4: Self-closing pass duplicate-replace risk.** Python ran a
second loop with the same `replace_first` bug for self-closers.
Rust handles self-closers in the same single pass.
* **#5: Placeholder collision.** If the input contained a literal
`{{HEADROOM_TAG_…}}` substring, Python silently let the collision
break restoration. Rust salts the prefix and reports it in stats.
# Architecture
Two-phase walker:
* Phase 1 (`identify_spans`): linear scan over input bytes, hand-
rolled tag-open / tag-close lexer (no regex). Maintains a stack of
open custom tags; on a matching close, collapses the inner span
into a single `Span { start, end, Block }`. Self-closing custom
tags become `Span { ..., SelfClosing }` immediately. Marker-only
mode (`compress_tagged_content=true`) emits Open/CloseMarker spans
instead. Orphan opens stay un-protected (matches Python behavior).
Orphan closes are emitted verbatim and counted in stats.
* Phase 2 (`emit_output`): walks `text` once, splicing placeholders
for span ranges and copying everything else verbatim. Offset-based,
never `str.replace`.
PyO3 surface: `protect_tags`, `restore_tags`, `is_html_tag`,
`known_html_tag_names`. The Python shim retires the regex internals
and re-exports `KNOWN_HTML_TAGS` (rebuilt from the Rust list) +
`_is_html_tag` for backwards compat with `content_router.py` and the
existing test surface.
# Test plan
* 25 Rust unit tests including 4 `fixed_in_3e4_*` bug-fix tests
* 27 Python tests (23 existing + 4 new `fixed_in_3e4` parity tests)
* 5 integration tests in `test_tag_protection_integration.py` pass
* `make ci-precheck` clean
|
||
|
chopratejas
|
45720301e5 |
feat(rust): port log_compressor to Rust + bug fixes (Phase 3e.5)
Ports `headroom.transforms.log_compressor` to Rust. The biggest-by- impact remaining compressor port: build/test logs are where the 10-50x compression wins live. * Stack-trace state machine: per-flavor dispatcher (Python Traceback, JS, Java, Rust error, Go); each flavor has its own termination rule. Python terminated on any blank line, dropping mid-trace lines from chained-exception traces. * Conservative dedupe: preserves message prefix (everything before first `:` or `=`); only trailing region is tokenised. Python's blanket normalisation collapsed segfaults at different addresses. * Loud CCR failures: `tracing::warn!` + `logger.warning` instead of bare `except: pass`. * `LogLevel::FAIL` documented as cosmetic-equivalent to ERROR. Same shape as search_compressor port. Rust `LogCompressor` orchestrates format detect -> classify -> score -> select -> format -> CCR. Inline static-table format detector (YAGNI), aho-corasick level classifier with word-boundary post-filter (`signals::keyword_detector` technique), hand-rolled per-flavor stack-trace state machine. `signals::LineImportanceDetector` NOT consumed -- log levels are structural, not prose-style importance. `headroom.transforms.log_compressor` becomes a thin shim: `compress()` delegates to Rust end-to-end; internal helpers preserved for the existing 50-test surface. Two existing tests updated for new dedupe semantics + new compress orchestration. * 17 Rust unit tests * 50 Python tests pass * `make ci-precheck` clean |
||
|
chopratejas
|
4d799d5264 |
feat(rust): port search_compressor to Rust + signals trait consumer (Phase 3e.2)
Ports `headroom.transforms.search_compressor` to Rust as the first consumer of the `signals::LineImportanceDetector` trait shipped in Phase 3e.1. The Python regex registry (`_GREP_PATTERN`/`_RG_CONTEXT_PATTERN`) silently misparsed two real-world inputs. The hand-rolled Rust parser fixes both: * **Windows paths.** `^([^:]+):(\d+):(.*)$` captured only the drive letter from `C:\Users\foo\bar.py:42:line`, then the `\d+` group failed on `\`. Result: every Windows-formatted line was silently dropped from `file_matches`. The Rust parser detects the drive prefix and starts the line-number scan after the drive colon. * **Filenames with `-`.** `_RG_CONTEXT_PATTERN`'s `[^:-]+` excluded dashes from the path, so legitimate names like `pre-commit-config.yaml-42-line` parsed wrong. The Rust parser anchors on the *line-number marker* (`<sep>\d+<sep>`), so paths can contain dashes freely. Two further hardening changes: * CCR storage failures are loud (Python silently swallowed them). * Per-file dedup is `O(n log n)` via `BTreeSet<(line_no, content_hash)>` (Python used linear `match not in file_selected`, worst-case quadratic for big files). The Rust `SearchCompressor` owns a `Box<dyn LineImportanceDetector>` defaulting to `KeywordDetector`. Priority scoring routes through the trait instead of a hardcoded regex list, so a future BGE classifier head (per the trait extension docs) can take over without touching the compressor. Sidecar `SearchCompressorStats` captures lines unparsed, files dropped by `max_files`, matches dropped by per-file vs global caps, and the CCR skip reason -- diagnostics Python never emitted. `headroom.transforms.search_compressor` is now a thin shim that delegates `compress()` to Rust end-to-end (so the parser bug fixes land in production), and keeps the legacy `_parse_search_results` helper routed through the same Rust parser. The other internal helpers (`_score_matches`, `_select_matches`, `_format_output`) stay Python -- they're heavily covered by existing direct-call tests and Rust scoring is byte-equivalent. The 4 public dataclasses are unchanged. Tests that monkeypatched the old internal `_store_in_ccr` helper are updated to exercise the new `_persist_to_python_ccr` boundary instead. * 16 Rust unit tests (parser, scoring, selection, CCR round-trip) + 3 explicit `fixed_in_3e2` markers for the bug-fix lines * 53 Python tests (existing suite intact; 2 updated for new shape) * `make ci-precheck` clean Stacks on PR #317 (signals trait module). |
||
|
Tejas Chopra
|
cf3877de38
|
Merge pull request #317 from chopratejas/rust-stage-3e-1-signals
feat(rust): signals trait module + KeywordDetector (Phase 3e.1) |
||
|
chopratejas
|
12c2665531 |
feat(rust): signals trait module + KeywordDetector (Phase 3e.1)
Establish `crates/headroom-core/src/signals/` as a top-level module holding cross-cutting detection traits. Phase 3e.1 ports `error_detection.py` to a `LineImportanceDetector` trait + a `Tiered<T>` combinator + a single concrete `KeywordDetector` impl backed by aho-corasick. Three traits at three granularities are sketched (line / blob / item); only line-importance is implemented today. Two bug fixes from the Python source bake into both the Rust impl and the Python regex shim: 1. `ERROR_KEYWORDS` listed `timeout|abort|denied|rejected` but `ERROR_PATTERN` regex omitted them. Lines like `"Connection timeout"` were silently neutral despite the keyword being canonical. Both surfaces now flag them. 2. `SECURITY_KEYWORDS` carried `token`, which false-positived on every reference to LLM tokens (`input_tokens`, `tokens_saved`, ...) in our own product. Dropped from the security set. The Python `error_detection.py` shim now reflects keyword data out of Rust via `keyword_registry_snapshot()` and recompiles the legacy `re.Pattern` objects on the fly. Existing callers (text_compressor, search_compressor, intelligent_context) continue to import the same names with no source changes; caller migration to the trait API happens in their own port PRs. The trait architecture is the seam where a future ML detector slots in without touching `KeywordDetector` or any caller. The canonical extension is documented in `signals/README.md` as a classifier head on the existing `bge-small-en-v1.5` embedder loaded by `relevance::EmbeddingScorer` -- 384-dim -> 4-class softmax, ~1.5 KB head, ~1 ms inference, no extra model file. Two alternatives (distilled tinyBERT in ONNX, logistic regression on lexical features) are kept open in case BGE-head underfits. Per the no-silent-fallbacks rule: only `KeywordDetector` lands as a concrete impl. No NoOp, no MockDetector, no stub-ML -- those will arrive with their real implementations. Phase 3g (Compression Pipeline Formalization, issue #315) is queued as the cross-cutting follow-up that will make lossless-then-lossy- then-CCR ordering an explicit, observable architecture rather than implicit per-compressor logic. Trait shapes there will reuse the signals primitive landed in this PR. |
||
|
chopratejas
|
5c60abcf81 |
chore(rust): wire detection chain into ContentRouter (Stage 3d PR5)
Replaces the dispatch-path detection with the locked Stage-3d chain:
Tier 1: magika_detect() (PR3)
Tier 2: unidiff::is_diff() (PR4)
Tier 3: PlainText fall-through
The regex `content_detector` is no longer on the production path —
it stays in the tree as a comparison oracle (and for any direct
caller); a future PR retires it entirely.
What lands:
- `crates/headroom-core/src/transforms/detection.rs`: new `detect()`
function that chains the two tiers. Tier-1 errors log at WARN
level and continue to Tier 2 (the chain's *next* tier IS the
legitimate fallback for magika failure; treating tier-1 error as
hard-fail would block all detection on transient ONNX issues).
- 12 unit tests covering: empty, JSON, source code, HTML, standard
git diff, naked hunk (Tier 2 catch), prose, grep search results
(locked-design behavior change), build log, YAML, Rust source,
determinism across repeated calls.
- PyO3 binding `detect_content_type` now calls the chain. Synthesizes
the legacy `DetectionResult` shape (confidence=1.0, empty metadata)
since the chain doesn't surface a probabilistic score and no
production caller reads metadata from the binding today.
- Python `headroom/transforms/content_router.py`: `_detect_content`
now delegates to `headroom._core.detect_content_type`. The Python-
side `_get_magika_detector` + regex fallback is retired (single
detection surface; no parallel paths). Test for the helper rewritten
to monkeypatch the Rust binding instead of the old Python paths.
Behavior changes (per locked design):
- `SearchResults` and `BuildOutput` ContentTypes route to PlainText
(or SourceCode if magika happens to label it code-like) rather
than to specialized strategies. No regex tier on the Rust side,
per `project_rust_content_detection_arch.md`. If proxy benchmarks
show real loss on grep/build outputs, we add focused detectors
later — not preemptively.
Stacked on PR4 (unidiff). When PR4 squash-merges, this PR rebases
trivially against main.
Tests:
- 12 new Rust unit tests in `transforms::detection::tests`
- 43 Python content_router tests (was 42; old monkeypatch test
rewritten in place, not duplicated)
- `make ci-precheck` green
|
||
|
chopratejas
|
3f8de4e117 |
fix(smart_crusher): re-land orphaned audit close-out — CCR knob + scorer fail-loud
Re-lands two audit fixes that were marked "merged" on GitHub but never reached main: squash-merging the parent stack changed its commit SHA, which silently dropped the contents of the stacked PRs (#301, #305). Single PR this time — no stacking risk. What lands: 1. **`enable_ccr_marker` Rust gate** — new field on `SmartCrusherConfig` (default `true`). `crush_array` checks it before emitting the `<<ccr:HASH>>` marker text and the CCR store write. PyO3 surface + parity-fixture tolerance updated; recorded fixtures predate the field and inherit the `true` default. 2. **Python shim collapses both flags to the gate** — both `ccr_config.enabled=False` and `ccr_config.inject_retrieval_marker =False` now flip the Rust gate off. Storing a payload nothing in the prompt can reference is pointless, and storing under `enabled=False` would be a surprise side effect the user explicitly opted out of. 3. **Custom `scorer` / `relevance_config` fails loud** — replaces the prior WARNING-and-drop. Silently dropping a user-supplied scorer is a textbook silent fallback. `NotImplementedError` instead. Verified zero production callers pass these args; full plumbing arrives with Stage-3c.2's relevance-crate Python bridge. Tests: - 2 new Rust unit tests in `crusher.rs::tests` - 6 new Python tests in `test_smart_crusher_toin_attachment.py` (3 CCR marker-knob behaviors + 3 scorer fail-loud) - Removed `test_ccr_inject_marker_false_logs_warning` (the WARNING is gone now that the flag is honored) - `make ci-precheck` green; eval suite + observability tests run twice consecutively to verify no TOIN file pollution leaks into the regular+coverage double-run on Python 3.11 RUST_DEV.md audit table reflects both gaps closed. |
||
|
Tejas Chopra
|
035fa02c19
|
Merge pull request #295 from chopratejas/rust-stage-3d-pr1-content-detector
chore(rust): port ContentDetector to Rust + parity + PyO3 bridge |
||
|
chopratejas
|
625290cfbc |
chore(rust): port ContentDetector to Rust + parity harness + PyO3 bridge
Faithful port of `headroom/transforms/content_detector.py` into `headroom-core`. Same regex patterns, dispatch order, confidence formulas, and line-count caps; lockstep with Python via 21 recorded parity fixtures (every dispatch branch exercised). - crates/headroom-core/src/transforms/content_detector.rs: regex-only detector (no ML); ContentType/DetectionResult mirror Python's enum + dataclass surface; metadata uses serde_json::Map for clean PyO3 bridging. - Tie-break in code detection: track scores in first-match insertion order (matches Python dict iteration semantics on `max()` ties). - TypeScript second pattern is start-anchored — Python's `pattern.match(line)` is start-anchored, but the regex crate's `is_match` is unanchored, so the literal `:` prefix is required for parity. - crates/headroom-parity: ContentDetectorComparator + universal f64 normalization in `compare_fixture` (serde_json's lossy parse vs full-precision serialize creates a 1-ULP asymmetry that broke comparison; round-trip the actual through to_string/from_str). - crates/headroom-py: detect_content_type/is_json_array_of_dicts and PyDetectionResult exposed via PyO3 with GIL released during scan. - tests/parity/recorder.py: new `_wrap_function` for free-function recording; content_detector hook + 21 varied inputs covering JSON arrays, diffs, HTML, search, build/log, six languages, and fallbacks. Sets up PR2 (ContentRouter scaffold) to call into Rust ContentDetector in-process via the bridge. |
||
|
chopratejas
|
29aadb1054 |
perf(rust): tier-1 multi-worker wins — GIL release, sharded CCR store, single-serialize CCR write
Three orthogonal hot-path fixes targeting concurrent-request throughput.
Each is independently bench-measured below; the proxy hot path benefits
from all three at once.
== 1. PyO3 GIL release on heavy compute ==
PyO3 methods (crush, smart_crush_content, crush_array_json,
compact_document_json, compress, compress_with_stats) used to hold the
GIL across the entire Rust call. Result: a 100ms compress() blocked
EVERY other Python thread for 100ms — multi-worker uvicorn deployments
serialized through SmartCrusher.
Wrap each compute call in `py.allow_threads(|| ...)`. Inputs (`&str`
from Python) are copied to owned `String` first because PyO3 ties them
to the GIL hold. PyDict construction stays on the GIL side.
Measured: 4 Python threads each running 20 crushes:
before (GIL held): ~3.3s wall (serialized — equivalent to 4×0.83s)
after (allow_threads): 826ms wall (4.01x speedup, perfect parallel)
== 2. CcrStore: Mutex<HashMap> -> DashMap-backed sharded ==
Single Mutex was the dominant bottleneck under multi-worker load — every
put/get serialized through one lock. Replace with DashMap (sharded
concurrent map, lock-free reads within a shard) plus a separate
small Mutex<VecDeque> for FIFO insertion-order eviction. Reads of
distinct keys never contend; writes only contend during the brief
order-queue push or capacity-sweep.
A/B bench (200 mixed put/get ops × N threads, in benches/ccr_store.rs):
Threads | DashMap Legacy Mutex Speedup
-------------------------------------------
1 | 63 µs 71 µs 1.13x
2 | 98 µs 194 µs 2.0x
4 | 178 µs 707 µs 4.0x
8 | 342 µs 1267 µs 3.7x
Legacy degrades ~linearly with thread count; DashMap stays near-flat
per-thread. Real multi-worker scaling.
== 3. Single-serialize the lossy CCR payload ==
The lossy `crush_array` path used to serialize the full array TWICE:
once in `hash_array_for_ccr` (allocates `Value::Array(items.to_vec())`,
deep-clones every Value subtree, then serializes), and a second time
in the store-write site. For a 50-item dict array that's ~MB of
allocator pressure per crushed array.
Introduce `canonical_array_json` (serializes `&[Value]` directly — same
bytes as `Value::Array(items.to_vec())` but no wrapper allocation +
no tree clone), call it ONCE per lossy path, then both hash and store
from those same bytes. Hash-format stable — all 17 parity fixtures
match byte-for-byte.
== Tests ==
- 8 ccr.rs unit tests including a new concurrent-stress test (8 threads
× 200 puts/gets, every key readable afterwards)
- 14 ccr_roundtrip integration tests stay green
- parity-run smart_crusher: 17/17 fixtures match
- 479 lib + 14 integration + 185 Python tests all pass
- New benches/ccr_store.rs runs the A/B and is committed for regression
visibility
== Dependencies added ==
- dashmap v6 (mature, widely-used in tokio/linkerd ecosystem)
|
||
|
chopratejas
|
da7716a95a |
chore(rust): SmartCrusher CCR marker injection + walker unification
Closes four gaps in the Rust SmartCrusher pipeline that, together,
wire CCR storage end-to-end so the LLM can actually retrieve dropped
data:
1. CCR-Dropped marker is now injected into process_value's lossy-path
output as a sentinel object {"_ccr_dropped": "<<ccr:HASH N_rows_offloaded>>"}
appended to the kept-items array. Previously the store held the
original but no pointer reached the prompt -- the retrieval contract
was data-on-server, no-way-to-ask. Sentinel-as-object preserves the
array-of-dicts shape so downstream iteration with x.get(...) keeps
working.
2. Walker / process_value drift removed. process_value gains a
Value::String arm that handles stringified-JSON containers (parse,
recurse, re-encode) and opaque blobs (CCR marker + store) -- same
semantics walker.rs has always had, now reachable from the main
crush() pipeline.
3. Opaque-string CCR now stores originals. DocumentCompactor gains an
Option<Arc<dyn CcrStore>> field; emit_opaque_ccr_marker calls
store.put when one is configured. Same hash regardless of store
presence -- runtime contract is stable across configurations.
Same wiring is shared between walker.rs and process_value via the
extracted helper.
5. PyO3 surface adds SmartCrusher.compact_document_json(doc_json) ->
compacted-json string. Routes through the crusher's existing CCR
store, so ccr_get resolves both row-drop and opaque-string hashes.
Tests:
- 5 new Rust integration tests in ccr_roundtrip.rs (marker visibility,
nested-array marker, opaque-string roundtrip, stringified-JSON
recursion, walker-with-store)
- 4 new Python tests covering the marker visible-to-LLM contract via
both the native PyO3 surface and the Python shim
- 5 legacy parity fixtures re-recorded (dict_array_*, duplicate_dicts_40)
-- their lossy outputs now carry the sentinel; Rust + Python both
match the new bytes (parity-run smart_crusher: 17/17)
|
||
|
chopratejas
|
22c8fec4c1 |
chore(rust): SmartCrusher CCR storage layer + roundtrip verification
CcrStore trait + InMemoryCcrStore (1000 entries, 5-min TTL, FIFO eviction, idempotent re-store) live at the crate root. SmartCrusher's lossy crush_array path now actually stashes the full original [items] canonical-JSON into the configured store keyed by the same ccr_hash it embeds in the prompt marker -- closing the no-data-loss contract that was previously hash-only. PyO3 surface: - crusher.crush_array_json(items_json) -> dict with ccr_hash + kept items - crusher.ccr_get(hash) -> Optional[str] for retrieval - crusher.ccr_len() -> int for telemetry Python shim passes both through. Default constructors enable the store (matches Python's CCR-enabled default); without_compaction() also gets it because CCR is a contract, not an opt-in extra. Tests proving compress -> store -> retrieve -> reconstruct: - 7 unit tests in ccr.rs (put/get/eviction/expiry) - 9 Rust integration tests (crates/headroom-core/tests/ccr_roundtrip.rs) - 10 Python tests including 4 explicit before/after element-equality assertions through both the native PyO3 surface and the Python shim Plugin manifest versions auto-bumped by the sync-plugin-versions pre-commit hook (unrelated to CCR but co-resident in the working tree). |
||
|
chopratejas
|
1601591900 |
feat(rust): SmartCrusher PR4 — lossless-first default + CCR-Dropped restoration
Stage 3c.2 PR4. Restores Python's CCR-Dropped semantics on the lossy
path (the cornerstone reversibility guarantee that the port had
silently dropped) and flips the OSS default to lossless-first with a
configurable savings threshold.
# The user-visible behavior
Default `SmartCrusher::new()` now runs:
1. Try lossless compaction.
2. If savings >= `lossless_min_savings_ratio` (default 0.30), ship
it — `compacted` populated, `ccr_hash = None`, nothing dropped.
3. Otherwise fall through to the lossy path — drop rows AND
populate `ccr_hash` so the runtime can cache the full original
for tool-call retrieval.
**No data is ever lost.** "Lossy" means "compressed view inline; full
payload retrievable via CCR cache" — same semantics as Python's
SmartCrusher with CCR enabled. The runtime (PyO3 bridge / proxy
server) owns the cache; this crate computes the hash and emits a
marker so the prompt knows where to look.
# What changed
- `SmartCrusherConfig.lossless_min_savings_ratio: f64` (default 0.30).
Single configurable knob — Enterprise overrides as needed. Below
the threshold, lossless declines and lossy + CCR runs.
- `SmartCrusher::new(cfg)` flips to include the compaction stage by
default. `SmartCrusher::without_compaction(cfg)` is the explicit
opt-out for callers / fixtures that depend on pre-PR4 behavior.
- `crush_array` rewritten:
- Lossless-first dispatch with savings-ratio gate
- Lossy path now hashes the full original (12-char SHA-256 prefix)
and emits a CCR-Dropped marker in `dropped_summary` whenever
rows are dropped
- `ccr_hash` field populated whenever rows were dropped
- `process_value` substitutes the compacted string into the JSON
tree when lossless wins, so `crush()` output reflects the win
- PyO3 bridge: `SmartCrusher.without_compaction()` static method;
`SmartCrusherConfig` exposes the new `lossless_min_savings_ratio`
field; Python `SmartCrusher` wrapper accepts `with_compaction=True`
(default) and routes to the right Rust constructor.
- Parity harness: legacy 17 fixtures use `without_compaction()` so
byte-equal coverage of the lossy path is preserved.
# Tests
- Rust: 281/281 smart_crusher unit tests pass (was 277). Six new
tests cover: lossless wins above threshold, lossy falls through
below threshold, CCR hash deterministic + input-dependent, lossy
without compaction emits CCR, passthrough paths don't emit CCR,
without_compaction yields no compacted field.
- Python parity: 21/21 (legacy fixtures via without_compaction).
- Python lossless default smoke: 3/3 new tests in
test_smart_crusher_lossless_default.py.
- Python retention: 21/21 (updated to opt into the lossy path
explicitly since their semantics target row-level retention).
- make ci-precheck green.
Modules:
crates/headroom-core/src/transforms/smart_crusher/{config,crusher}.rs
crates/headroom-parity/src/lib.rs
crates/headroom-py/src/lib.rs
headroom/transforms/smart_crusher.py
tests/test_quality_retention.py
tests/test_transforms/test_smart_crusher_{lossless_default,rust_parity}.py
|
||
|
chopratejas
|
d6a00ee89c |
ci: fix smart_crusher branch CI failures + add make ci-precheck pre-push gate
Five gates broke on the 2026-04-27 push of the smart_crusher branch.
Each is fixed below; the second half adds a `make ci-precheck` target
(plus an installable git pre-push hook) so the same dance never happens
again.
Failures fixed:
1. cargo fmt — 22 files had formatting drift introduced over the
stage 3c.1 work. `cargo fmt --all` reformatted them; no semantic
changes. `cargo test --workspace` still green (388 + supporting).
2. wheels job (macOS x86_64) — `fastembed -> ort -> ort-sys` does not
publish prebuilt ONNX Runtime binaries for `x86_64-apple-darwin`.
Removed that target from `.github/workflows/rust.yml`'s wheels
matrix. Apple Silicon (`aarch64-apple-darwin`) covers macOS
distribution; Intel macOS users can build from source. The matrix
now has 2 targets: linux x86_64 + macOS aarch64.
3. test-extras (relevance.py) — `tests/test_relevance.py::TestSmartCrusherIntegration`
constructs a `SmartCrusher`, which hard-imports `headroom._core`
since the python implementation was retired in stage 3c.1b. The
test-extras job didn't build the rust extension. Added the same
`maturin build + symlink` block the main `test` job uses.
4. smoke-test (eval.yml) — same root cause:
`compression_only.evaluate_ccr_lossless` instantiates a SmartCrusher.
Same fix: build the rust extension before the smoke test runs.
5. commitlint — three rules tripped:
- `subject-case` rejects PascalCase identifiers in subjects, but
the project deliberately names classes (SmartCrusher, HfTokenizer,
ContentRouter, DiffCompressor) in commit subjects. Disabled.
- `footer-leading-blank` is a warning that the wagoid action turns
into a CI failure; lines like `Module: foo.rs` in our bodies
match the conventional footer pattern and trip it. Disabled.
- `type-enum` doesn't include `parity`, but the project ships
parity-test infrastructure as its own concern (separate from
`test:`); added `parity` to the allowed types.
Pre-push verification — the prevention half:
`make ci-precheck` runs all of the above CI gates locally:
- `ci-precheck-rust`: cargo fmt --check + clippy + test --workspace.
- `ci-precheck-python`: builds the rust extension via maturin, then
runs the smart_crusher-affected python test files (185 tests across
test_transforms/, test_relevance*, test_ccr, test_acceptance,
test_critical_fixes, test_quality_retention).
- `ci-precheck-commitlint`: `npx commitlint --from origin/main --to
HEAD` against the same config CI uses. Skipped silently if npx is
not on PATH (install Node 18+ to enable).
`make install-git-hooks` (or `scripts/install-git-hooks.sh`) installs
a git pre-push hook that runs `make ci-precheck` automatically.
Bypass with `--no-verify` only when truly needed.
When new CI gates land in `.github/workflows/`, mirror them into a
`make ci-precheck-*` target. The Makefile is the local mirror of the
CI configuration; keeping them in sync is a load-bearing invariant.
Verification: `make ci-precheck` runs green on this commit.
|
||
|
chopratejas
|
5328d87b1e |
feat(rust): pyo3 bridge for SmartCrusher
Stage 3c.1b step 1: expose `SmartCrusherConfig`, `CrushResult`, and `SmartCrusher` to Python via `headroom._core`. The Python shim that delegates to it (replacing the 3669-line Python implementation) lands in the next commit; this commit just builds the bridge and a fixture-replay test that pins it. Surface: - `headroom._core.SmartCrusherConfig(**fields)` — every field of the Rust `SmartCrusherConfig` exposed as a kwarg with matching default. - `headroom._core.CrushResult` — read-only mirror of the Rust struct with `compressed`, `original`, `was_modified`, `strategy` getters. - `headroom._core.SmartCrusher(config=None)` — constructor accepts only `config`; the Python shim drops `relevance_config`, `scorer`, and `ccr_config` since Stage 3c.1 keeps those subsystems disabled. - `crush(content, query="", bias=1.0)` and `smart_crush_content(...)` methods mirror the Python signatures. Verification: - All 17 recorded parity fixtures byte-equal between Python and the PyO3 bridge (`tests/test_transforms/test_smart_crusher_rust_parity.py`, 18 tests pass — 1 fixture-count sanity + 17 fixtures). - The Rust-side `cargo run -p headroom-parity --bin parity-run -- run --only smart_crusher` was already 17/17 green. The two tests catch different regression classes: - Rust-only test: catches drift in the Rust port's logic. - Python bridge test: catches PyO3 input/output translation bugs. |
||
|
chopratejas
|
f5f465418b |
feat(rust): retire python diff_compressor, ship rust-only via pyo3
The python `DiffCompressor` is replaced by a thin pyo3-backed shim that delegates to `headroom._core.DiffCompressor`. There is no python implementation and no env-var fallback — the wheel is a hard import. Why now: opt-in defaults don't drive python retirement. Byte-equal parity was already proven across 27 fixtures (stage 3a); keeping a shadow python impl behind a flag is a permanent maintenance cost with no operational benefit. Stage 3b deletes ~700 lines of python parser / scorer / formatter code; the rust crate has its own coverage. Surface preserved: - `headroom.transforms.diff_compressor.DiffCompressor` — same class name, same `__init__`, same `compress(content, context)` shape. Returns python `DiffCompressionResult` dataclasses so call sites that destructure with `asdict()` work unchanged. - `DiffCompressorConfig` and `DiffCompressionResult` dataclasses kept. - Sidecar `compress_with_stats(...)` exposes the rust-only `DiffCompressorStats` (per-file hunk drops, context lines trimmed, file_mode normalizations) for observability. Removed: - Python parser / scorer / formatter (~700 lines). - Internal `DiffHunk` / `DiffFile` parser dataclasses (rust crate has parallel coverage). - 12 tests that probed `_parse_diff` / `_score_hunks` or the deleted parser dataclasses. The 29 public-API tests in `test_diff_compressor.py` remain and now exercise the rust backend through the same import path. - `HEADROOM_DIFF_COMPRESSOR_BACKEND` env var — no longer meaningful. Build: - `scripts/build_rust_extension.sh` runs `maturin develop` and symlinks the built `.so` into `headroom/` so `import headroom._core` resolves past the in-tree package shadowing the maturin overlay. - `.gitignore` excludes the symlinks and allowlists the build script. Tests: - 544 transforms pass; 33 rust-vs-fixture parity tests guard the pyo3 bridge against regressions (`tests/test_transforms/test_diff_compressor_rust_parity.py`). - Mypy clean. |
||
|
chopratejas
|
0414cb70e4 |
feat(rust): scaffold workspace + parity harness (phase-0)
Bootstrap the Rust port of Headroom. Additive only — no existing Python code modified. Ships the workshop, not the widgets. Layout Cargo.toml (workspace) + rust-toolchain.toml crates/headroom-core — transform library, stub only crates/headroom-proxy — axum binary, /healthz only crates/headroom-py — PyO3 cdylib, exposes headroom._core.hello() crates/headroom-parity — Rust-vs-Python oracle harness + parity-run CLI Tooling Makefile: test, test-parity, bench, build-proxy, build-wheel, fmt, lint .github/workflows/rust.yml: test, wheels (linux/mac), audit, parity-nightly deny.toml for cargo-deny Parity corpus tests/parity/recorder.py + scripts/record_fixtures.py 125 recorded fixtures across 5 leaf transforms (ccr, tokenizer, log_compressor, diff_compressor, cache_aligner) Docs RUST_DEV.md — developer setup and workspace reference docs/spec/022-rust-migration.md — migration plan and stage breakdown .gitignore: whitelist scripts/record_fixtures.py; ignore target/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |